# API Overview Fabric Manager exposes the following gRPC services: | Service | Hosted by | Called by | Purpose | |---------|-----------|-----------|---------| | `FabricManagerService` | Controller | Orchestrators (SLURM, Kubernetes), CLI, SDK | Query topology, compute placements | The controller can additionally serve a read-mostly [HTTP JSON API and web UI](web-ui.md). For command-line access see the [CLI reference](cli.md); for programmatic access see the [SDK](sdk.md). ## Concepts ### Topology sources: PSD and FSD Most query RPCs take a `topo_source` field that selects where *connectivity* comes from: | Value | Name | Meaning | |-------|------|---------| | `0` | `PSD` | Physical System Descriptor — connectivity as **discovered** by the agents. The default. | | `1` | `FSD` | Factory System Descriptor — connectivity **derived from the controller-configured "golden state"** wiring. Requires `controller.factory_system_descriptor_search_path` to be set and to cover the requested hosts. | | `3` | `COMBINED_PSD_FSD` | The union of both, with each host tagged by the source it came from. Only accepted by `QueryTopologySummary`. | Value `2` is intentionally unused. The Factory System Desctiptor (FSD) holds the "golden" connectivity state for a set of hosts equipped with TT accelerators. It is generated when a system is installed and cabled up. An `FSD` topology is built from the factory descriptor alone, so it is available before any agent has registered — a cabled but idle factory still has a topology to show. Where an agent *has* reported a slot, the ASIC's synthesized ID is replaced by the real hardware unique ID so the two sources can be joined; the supplemental metadata that only discovery knows (PCI BDFs, UMD chip IDs, host quirks) is simply absent for the rest. Comparing the two sources is how FM detects missing and unexpected links. ### Descriptors are passed as opaque strings and bytes `MeshGraphDescriptor` (MGD), `PhysicalSystemDescriptor` (PSD), and `FactorySystemDescriptor` (FSD) cross the wire as textproto strings or serialized bytes rather than as structured protobuf fields. This deliberately avoids coupling FM's protobuf version to tt-metalium's, so FM builds against both Ubuntu 22.04 (protobuf 3.12) and 24.04 (protobuf 3.21). It also makes the API easy to drive from `grpcurl`. Deserialize with `ParseFromString()` (bytes) or `TextFormat::ParseFromString()` (textproto) on the client side. ## Orchestrator Service (`FabricManagerService`) | RPC | Description | |-----|-------------| | `QueryPhysicalTopology` | Returns the aggregated physical topology as a serialized `PhysicalSystemDescriptor`, plus supplemental per-ASIC and per-host metadata | | `GetValidPlacementsMGD` | Maps a Mesh Graph Descriptor onto physical ASICs using the CSP solver and returns valid host/rank assignments. Blocks for as long as the placement takes | | `StartPlacementMGD` | Starts the same placement as a long-running operation and returns its handle immediately | | `GetPlacementOperation` | Reports a placement operation's progress, and its result once it is done | | `CancelPlacementOperation` | Asks the controller to stop a placement operation | | `ListPlacementOperations` | Lists the placement operations the controller is running or still retains results for | | `QueryFactorySystemDescriptor` | Returns the controller-configured FSD files ("golden state") | | `QueryTopologySummary` | Host-level connectivity view, optionally diffed against the FSD | ### `QueryPhysicalTopology` Aggregates the topology reported by all healthy agents, or, with `topo_source` `FSD`, renders the configured factory descriptor. **Request** (`QueryPhysicalTopologyRequest`) | Field | Description | |-------|-------------| | `hosts` | Restrict to these host IDs (agent registration IDs). Empty means all healthy hosts, or, with `FSD`, every host the selected factory descriptor covers | | `topo_source` | `PSD` (default) or `FSD`. `FSD` needs no registered agents | | `trim_links_outside_filter` | When a host filter is applied and the source is PSD, drop cross-host links whose remote endpoint lives outside the filter. Ignored without a host filter. Defaults to `false` | **Response** (`QueryPhysicalTopologyResponse`) | Field | Description | |-------|-------------| | `physical_topology_bytes` | Serialized `tt.fabric.proto.PhysicalSystemDescriptor` | | `total_hosts`, `total_asics` | Summary counts | | `asic_metadata` | Per-ASIC fields the PSD does not carry — `pci_bdf`, UMD `chip_id`, `device_node_id`, `device_node_path`, `chip_arch`, `is_mmio_capable` — keyed by `asic_id` so consumers can join against the PSD. Only populated for ASICs discovered at runtime | | `host_metadata` | Per-host metadata (data-centre location, hardware quirks) keyed by host name | | `undiscovered_hosts` | Hosts the descriptor describes from the FSD alone, because no agent reported them. Always empty for a `PSD` topology | ASICs that exist only in the FSD have no `asic_metadata` entry. Everything an `undiscovered_hosts` entry contributes — its ASICs and their links — is the factory's expectation rather than an observation, so present those hosts as missing rather than as discovered hardware; the web UI draws them in red. ### `GetValidPlacementsMGD` Runs the CSP mapper to find every valid way to place a logical mesh onto the physical fabric. **Request** (`PlacementRequestMGD`) | Field | Description | |-------|-------------| | `mgd_textproto` | `tt.tt_fabric.proto.MeshGraphDescriptor` in textproto form (required) | | `constraints.host_ids` | Restrict placement to these hosts | | `constraints.pinnings` | Pin a specific `asic_id` to a specific `fabric_node_id` | | `topo_source` | `PSD` (default) or `FSD` | **Response** (`PlacementResponseMGD`) carries a `status`, an `error_message`, and a list of `placements`. Each `ValidPlacement` provides host assignments in two shapes: `host_assignments` (legacy, one rank per host) and `host_assignments_with_ranks` (multiple ranks per host). Both report the `asic_ids` used, the `mesh_id`/`mesh_rank`, the directly addressable `visible_devices`, and `fabric_nodes` — the logical `(mesh_id, chip_id)` FabricNodeId that each ASIC fulfills, which is what maps placed hardware back onto the MGD's logical mesh. *NOTE:* Currently the placement implementaion is limited to returning one placement option. Status values: | Status | Meaning | |--------|---------| | `PLACEMENT_MGD_SUCCESS` | Placements found | | `PLACEMENT_MGD_ERROR_UNKNOWN` | Unclassified failure | | `PLACEMENT_MGD_ERROR_IMPOSSIBLE` | No valid mapping exists | | `PLACEMENT_MGD_ERROR_INSUFFICIENT` | Not enough ASICs available | | `PLACEMENT_MGD_ERROR_INVALID_MGD` | MGD failed to parse | | `PLACEMENT_MGD_ERROR_TOPO_VALIDATION` | Topology validation failed (FSD placements only) | | `PLACEMENT_MGD_UNSTABLE_TOPO_FALLBACK` | Validation could not run; all links report as missing and ASIC/chip IDs are not guaranteed to match the physical topology | **Placement validation.** When the request used `FSD`, the controller re-runs discovery on every host in the placement and diffs the fresh per-host connectivity against the FSD subset covering those hosts. The result lands in `ValidPlacement.validation`, with a per-host breakdown listing `missing_links` and `unexpected_links`: | `PlacementValidationStatus` | Meaning | |-----------------------------|---------| | `PLACEMENT_VALIDATION_UNKNOWN` | Validation not performed (e.g. a PSD placement) | | `PLACEMENT_VALIDATION_HEALTHY` | Every host matched the FSD | | `PLACEMENT_VALIDATION_UNHEALTHY` | At least one host has missing and/or unexpected links | | `PLACEMENT_VALIDATION_ERROR` | Rediscovery could not complete (e.g. an agent was unreachable) | | `PLACEMENT_VALIDATION_USED_FALLBACK` | UMD discovery failed but the sysfs fallback succeeded; links always report as missing and `asic_id` is unavailable | The CSP solve runs in a separate `tt-fabric-placement-solver` process. The controller snapshots the topology, pipes a request to the child over stdin, and reads the response from stdout, which lets it `SIGKILL` the solve when the client disconnects or the deadline elapses. **Execution.** Every placement, whether it arrived through this RPC or as an operation, runs on the controller's placement worker pool, bounded by [`controller.placement`](config-reference.md#controllerplacement). Two consequences for this RPC specifically: a placement can wait behind others before it starts, and a full queue is refused with `RESOURCE_EXHAUSTED` rather than left to time out. Identical concurrent requests are collapsed onto one solve, so a client retry does not add load. Because this call only returns when the placement does, every hop between the client and the controller — SDK, CLI, ingress — needs a deadline long enough to cover the whole solve, which on a real fabric is minutes. The operations below exist to avoid that. ### Placement operations `StartPlacementMGD` accepts a placement and returns in milliseconds; the controller runs it in the background, and the operation outlives both this call and the client that made it. **Request** (`StartPlacementMGDRequest`) | Field | Description | |-------|-------------| | `request` | The `PlacementRequestMGD` to run, unchanged from the synchronous RPC | | `idempotency_key` | Two Start calls carrying the same key join one operation, which is what makes a retry safe. Empty means the key is derived from the request contents, so identical requests deduplicate anyway | | `max_runtime_sec` | Budget for this operation, measured from when it starts running. Clamped to `controller.placement.max_runtime_sec`; `0` uses it | A request that could never succeed — no MGD, an MGD that does not parse, an unsupported `topo_source` — is refused here with `INVALID_ARGUMENT`, so a client never has to poll an operation to discover it sent something malformed. Everything that needs the topology, the filesystem, or the solver is decided on the worker, so Start stays fast under load. All four RPCs return a `PlacementOperation`: | Field | Description | |-------|-------------| | `operation_id` | Handle to pass to Get / Cancel. Only valid within the controller process that minted it | | `state` | See below | | `result` | The `PlacementResponseMGD`, populated once `state` is `PLACEMENT_OP_DONE` | | `stage_message` | What the operation is currently doing, or why it stopped | | `hosts_validated`, `hosts_total` | Validation progress, from `PLACEMENT_OP_VALIDATING` onwards. Both `0` for placements that never validate | | `created_unix_ms`, `finished_unix_ms` | `finished_unix_ms` is `0` until the operation is terminal | | `retry_after_ms` | How long to wait before polling again. `0` means terminal — there is nothing left to poll for | | `PlacementOperationState` | Meaning | |---------------------------|---------| | `PLACEMENT_OP_QUEUED` | Accepted, waiting for a worker. No solver child yet | | `PLACEMENT_OP_SOLVING` | Running the CSP solve | | `PLACEMENT_OP_VALIDATING` | Rediscovering the placement's hosts and diffing them against the FSD. FSD-sourced placements only | | `PLACEMENT_OP_DONE` | The placement pipeline finished; `result` is populated | | `PLACEMENT_OP_CANCELLED` | Cancelled explicitly, abandoned by its only caller, or stopped by controller shutdown | | `PLACEMENT_OP_EXPIRED` | Exceeded its budget and was stopped | The state describes whether the *operation* ran, not whether a placement was found. An operation that legitimately determines no placement is possible is `DONE` and says so through `result.status` (`PLACEMENT_MGD_ERROR_IMPOSSIBLE`). `GetPlacementOperation` returns `NOT_FOUND` for an unknown id, which includes an operation started by a previous controller process and one whose result has aged out of `controller.placement.result_ttl_sec`. A client that intends to collect a result should poll at least that often. `CancelPlacementOperation` is cooperative: a queued operation stops immediately, while a running one is asked to stop and reaches `CANCELLED` when its worker unwinds (which includes killing the solver child), so the operation it returns may still report a running stage. Cancelling an operation that already finished changes nothing and leaves its result intact. `ListPlacementOperations` reports every retained operation, newest first, and omits results unless `include_results` is set — a result carries a whole placement per entry, and a listing usually only wants states and progress. Because the synchronous RPC runs as an operation too, it shows up here like any other placement. Clients do not have to build the poll loop themselves: the [SDK](sdk.md#placement-as-a-long-running-operation) wraps all four RPCs and adds a blocking `AwaitPlacementMGD` that paces polling by `retry_after_ms`, and the [CLI](cli.md#get-valid-placements-mgd) uses that by default, with `placement-operation`, `cancel-placement-operation` and `list-placement-operations` for the rest. The [HTTP API](web-ui.md#post-apiplacement) exposes the same lifecycle as `POST`, `GET` and `DELETE` on `/api/placement`. ### `QueryFactorySystemDescriptor` Returns the FSD files the controller was configured with. `hosts` filters to specific hostnames. Each `FactorySystemDescriptorFile` carries a `name` and `textproto_bytes` (a serialized `tt.scaleout_tools.fsd.proto.FactorySystemDescriptor`). Requires `controller.factory_system_descriptor_search_path`. ### `QueryTopologySummary` A compact host-level view intended for dashboards and operators rather than for placement. **Request:** `topo_source` (`PSD`, `FSD`, or `COMBINED_PSD_FSD`) and `validate_topo` — when true, the controller diffs the cached PSD against the FSD. **Response:** one `HostSummary` per host with `asic_count`, `arch`, `connected_hosts` (reachable via exit nodes), `connection_summaries` (per-peer link counts), the `source` that host's data came from, and `metadata` (hall/aisle/rack/shelf plus hardware quirks such as `is_bh_glx_rev_c`, the BH galaxy rev C tray layout). `missing_links` and `unexpected_links` are populated only when `validate_topo` is true. The response also reports `total_cross_host_links` (deduplicated host-pair connections) and `host_groups`, the connected components of the host graph. #### Host group names Each `HostGrouping` carries a `name` derived from the FSD `instance_path` of its members, so operators see the name the cabling hierarchy gives a group rather than a positional label. The name is the longest instance path the members share, ignoring the last segment of each path (that segment names the host, not a group). The first segment of an instance path names the cluster and the second names one group within it, in the form `-`. The controller strips the cabling template prefix, so a group covering one whole group of the cluster is named e.g. `110-SC36_4x4_C1-10`. Variations: - A group confined to a single sub-instance shares its second segment with its siblings, so the deeper shared segments are appended: `110-SC36_4x4_C1-10/bh_galaxy_sp_0`. - A group spanning several groups of the cluster shares only the first segment and is named after the cluster, e.g. `exabox`. - A leading token that contains no letters is not treated as a cabling template prefix, which leaves names such as `120-SC36_4x4_C1-10` (hall number first) intact. - Groups no loaded FSD describes have an empty `name`; clients label those by position. #### Pods Each `HostGrouping` also lists the `pods` it subdivides into, so a caller can offer a pod as a unit. TT cluster hierarchies group nodes into pods and pods into superpods, but the depth varies — a superpod is occasionally described as a grouping of smaller superpods — so no fixed instance path segment identifies a pod. What does hold is that everything below a pod addresses a single host, so a pod is the deepest instance path prefix that still covers more than one host. Pods are named relative to their group (`bh_galaxy_sp_0`, or `sp2_0/bh_galaxy_sp_0` where the hierarchy nests deeper) and carry no pods of their own. A group lists no pods when the FSD describes nothing below the group itself, and a host that belongs to no pod appears only in the group's `host_names`.