API Overview

Fabric Manager exposes the following gRPC services:

Service	Hosted by	Called by	Purpose
FabricManagerService	Controller	Orchestrators (SLURM, Kubernetes), CLI, SDK	Query topology, compute placements

The controller can additionally serve a read-mostly HTTP JSON API and web UI. For command-line access see the CLI reference; for programmatic access see the SDK.

Concepts

Topology sources: PSD and FSD

Most query RPCs take a topo_source field that selects where connectivity comes from:

Value	Name	Meaning
0	PSD	Physical System Descriptor — connectivity as discovered by the agents. The default.
1	FSD	Factory System Descriptor — connectivity derived from the controller-configured “golden state” wiring. Requires controller.factory_system_descriptor_search_path to be set and to cover the requested hosts.
3	COMBINED_PSD_FSD	The union of both, with each host tagged by the source it came from. Only accepted by QueryTopologySummary.

Value 2 is intentionally unused.

The Factory System Desctiptor (FSD) holds the “golden” connectivity state for a set of hosts equipped with TT accelerators. It is generated when a system is installed and cabled up.

An FSD topology is built from the factory descriptor alone, so it is available before any agent has registered — a cabled but idle factory still has a topology to show. Where an agent has reported a slot, the ASIC’s synthesized ID is replaced by the real hardware unique ID so the two sources can be joined; the supplemental metadata that only discovery knows (PCI BDFs, UMD chip IDs, host quirks) is simply absent for the rest. Comparing the two sources is how FM detects missing and unexpected links.

Descriptors are passed as opaque strings and bytes

MeshGraphDescriptor (MGD), PhysicalSystemDescriptor (PSD), and FactorySystemDescriptor (FSD) cross the wire as textproto strings or serialized bytes rather than as structured protobuf fields. This deliberately avoids coupling FM’s protobuf version to tt-metalium’s, so FM builds against both Ubuntu 22.04 (protobuf 3.12) and 24.04 (protobuf 3.21). It also makes the API easy to drive from grpcurl. Deserialize with ParseFromString() (bytes) or TextFormat::ParseFromString() (textproto) on the client side.

Orchestrator Service (FabricManagerService)

RPC	Description
QueryPhysicalTopology	Returns the aggregated physical topology as a serialized PhysicalSystemDescriptor, plus supplemental per-ASIC and per-host metadata
GetValidPlacementsMGD	Maps a Mesh Graph Descriptor onto physical ASICs using the CSP solver and returns valid host/rank assignments. Blocks for as long as the placement takes
StartPlacementMGD	Starts the same placement as a long-running operation and returns its handle immediately
GetPlacementOperation	Reports a placement operation’s progress, and its result once it is done
CancelPlacementOperation	Asks the controller to stop a placement operation
ListPlacementOperations	Lists the placement operations the controller is running or still retains results for
QueryFactorySystemDescriptor	Returns the controller-configured FSD files (“golden state”)
QueryTopologySummary	Host-level connectivity view, optionally diffed against the FSD

QueryPhysicalTopology

Aggregates the topology reported by all healthy agents, or, with topo_source FSD, renders the configured factory descriptor.

Request (QueryPhysicalTopologyRequest)

Field	Description
hosts	Restrict to these host IDs (agent registration IDs). Empty means all healthy hosts, or, with FSD, every host the selected factory descriptor covers
topo_source	PSD (default) or FSD. FSD needs no registered agents
trim_links_outside_filter	When a host filter is applied and the source is PSD, drop cross-host links whose remote endpoint lives outside the filter. Ignored without a host filter. Defaults to false

Response (QueryPhysicalTopologyResponse)

Field	Description
physical_topology_bytes	Serialized tt.fabric.proto.PhysicalSystemDescriptor
total_hosts, total_asics	Summary counts
asic_metadata	Per-ASIC fields the PSD does not carry — pci_bdf, UMD chip_id, device_node_id, device_node_path, chip_arch, is_mmio_capable — keyed by asic_id so consumers can join against the PSD. Only populated for ASICs discovered at runtime
host_metadata	Per-host metadata (data-centre location, hardware quirks) keyed by host name
undiscovered_hosts	Hosts the descriptor describes from the FSD alone, because no agent reported them. Always empty for a PSD topology

ASICs that exist only in the FSD have no asic_metadata entry. Everything an undiscovered_hosts entry contributes — its ASICs and their links — is the factory’s expectation rather than an observation, so present those hosts as missing rather than as discovered hardware; the web UI draws them in red.

GetValidPlacementsMGD

Runs the CSP mapper to find every valid way to place a logical mesh onto the physical fabric.

Request (PlacementRequestMGD)

Field	Description
mgd_textproto	tt.tt_fabric.proto.MeshGraphDescriptor in textproto form (required)
constraints.host_ids	Restrict placement to these hosts
constraints.pinnings	Pin a specific asic_id to a specific fabric_node_id
topo_source	PSD (default) or FSD

Response (PlacementResponseMGD) carries a status, an error_message, and a list of placements. Each ValidPlacement provides host assignments in two shapes: host_assignments (legacy, one rank per host) and host_assignments_with_ranks (multiple ranks per host). Both report the asic_ids used, the mesh_id/mesh_rank, the directly addressable visible_devices, and fabric_nodes — the logical (mesh_id, chip_id) FabricNodeId that each ASIC fulfills, which is what maps placed hardware back onto the MGD’s logical mesh.

NOTE: Currently the placement implementaion is limited to returning one placement option.

Status values:

Status	Meaning
PLACEMENT_MGD_SUCCESS	Placements found
PLACEMENT_MGD_ERROR_UNKNOWN	Unclassified failure
PLACEMENT_MGD_ERROR_IMPOSSIBLE	No valid mapping exists
PLACEMENT_MGD_ERROR_INSUFFICIENT	Not enough ASICs available
PLACEMENT_MGD_ERROR_INVALID_MGD	MGD failed to parse
PLACEMENT_MGD_ERROR_TOPO_VALIDATION	Topology validation failed (FSD placements only)
PLACEMENT_MGD_UNSTABLE_TOPO_FALLBACK	Validation could not run; all links report as missing and ASIC/chip IDs are not guaranteed to match the physical topology

Placement validation. When the request used FSD, the controller re-runs discovery on every host in the placement and diffs the fresh per-host connectivity against the FSD subset covering those hosts. The result lands in ValidPlacement.validation, with a per-host breakdown listing missing_links and unexpected_links:

PlacementValidationStatus	Meaning
PLACEMENT_VALIDATION_UNKNOWN	Validation not performed (e.g. a PSD placement)
PLACEMENT_VALIDATION_HEALTHY	Every host matched the FSD
PLACEMENT_VALIDATION_UNHEALTHY	At least one host has missing and/or unexpected links
PLACEMENT_VALIDATION_ERROR	Rediscovery could not complete (e.g. an agent was unreachable)
PLACEMENT_VALIDATION_USED_FALLBACK	UMD discovery failed but the sysfs fallback succeeded; links always report as missing and asic_id is unavailable

The CSP solve runs in a separate tt-fabric-placement-solver process. The controller snapshots the topology, pipes a request to the child over stdin, and reads the response from stdout, which lets it SIGKILL the solve when the client disconnects or the deadline elapses.

Execution. Every placement, whether it arrived through this RPC or as an operation, runs on the controller’s placement worker pool, bounded by controller.placement. Two consequences for this RPC specifically: a placement can wait behind others before it starts, and a full queue is refused with RESOURCE_EXHAUSTED rather than left to time out. Identical concurrent requests are collapsed onto one solve, so a client retry does not add load.

Because this call only returns when the placement does, every hop between the client and the controller — SDK, CLI, ingress — needs a deadline long enough to cover the whole solve, which on a real fabric is minutes. The operations below exist to avoid that.

Placement operations

StartPlacementMGD accepts a placement and returns in milliseconds; the controller runs it in the background, and the operation outlives both this call and the client that made it.

Request (StartPlacementMGDRequest)

Field	Description
request	The PlacementRequestMGD to run, unchanged from the synchronous RPC
idempotency_key	Two Start calls carrying the same key join one operation, which is what makes a retry safe. Empty means the key is derived from the request contents, so identical requests deduplicate anyway
max_runtime_sec	Budget for this operation, measured from when it starts running. Clamped to controller.placement.max_runtime_sec; 0 uses it

A request that could never succeed — no MGD, an MGD that does not parse, an unsupported topo_source — is refused here with INVALID_ARGUMENT, so a client never has to poll an operation to discover it sent something malformed. Everything that needs the topology, the filesystem, or the solver is decided on the worker, so Start stays fast under load.

All four RPCs return a PlacementOperation:

Field	Description
operation_id	Handle to pass to Get / Cancel. Only valid within the controller process that minted it
state	See below
result	The PlacementResponseMGD, populated once state is PLACEMENT_OP_DONE
stage_message	What the operation is currently doing, or why it stopped
hosts_validated, hosts_total	Validation progress, from PLACEMENT_OP_VALIDATING onwards. Both 0 for placements that never validate
created_unix_ms, finished_unix_ms	finished_unix_ms is 0 until the operation is terminal
retry_after_ms	How long to wait before polling again. 0 means terminal — there is nothing left to poll for

PlacementOperationState	Meaning
PLACEMENT_OP_QUEUED	Accepted, waiting for a worker. No solver child yet
PLACEMENT_OP_SOLVING	Running the CSP solve
PLACEMENT_OP_VALIDATING	Rediscovering the placement’s hosts and diffing them against the FSD. FSD-sourced placements only
PLACEMENT_OP_DONE	The placement pipeline finished; result is populated
PLACEMENT_OP_CANCELLED	Cancelled explicitly, abandoned by its only caller, or stopped by controller shutdown
PLACEMENT_OP_EXPIRED	Exceeded its budget and was stopped

The state describes whether the operation ran, not whether a placement was found. An operation that legitimately determines no placement is possible is DONE and says so through result.status (PLACEMENT_MGD_ERROR_IMPOSSIBLE).

GetPlacementOperation returns NOT_FOUND for an unknown id, which includes an operation started by a previous controller process and one whose result has aged out of controller.placement.result_ttl_sec. A client that intends to collect a result should poll at least that often.

CancelPlacementOperation is cooperative: a queued operation stops immediately, while a running one is asked to stop and reaches CANCELLED when its worker unwinds (which includes killing the solver child), so the operation it returns may still report a running stage. Cancelling an operation that already finished changes nothing and leaves its result intact.

ListPlacementOperations reports every retained operation, newest first, and omits results unless include_results is set — a result carries a whole placement per entry, and a listing usually only wants states and progress. Because the synchronous RPC runs as an operation too, it shows up here like any other placement.

Clients do not have to build the poll loop themselves: the SDK wraps all four RPCs and adds a blocking AwaitPlacementMGD that paces polling by retry_after_ms, and the CLI uses that by default, with placement-operation, cancel-placement-operation and list-placement-operations for the rest. The HTTP API exposes the same lifecycle as POST, GET and DELETE on /api/placement.

QueryFactorySystemDescriptor

Returns the FSD files the controller was configured with. hosts filters to specific hostnames. Each FactorySystemDescriptorFile carries a name and textproto_bytes (a serialized tt.scaleout_tools.fsd.proto.FactorySystemDescriptor). Requires controller.factory_system_descriptor_search_path.

QueryTopologySummary

A compact host-level view intended for dashboards and operators rather than for placement.

Request: topo_source (PSD, FSD, or COMBINED_PSD_FSD) and validate_topo — when true, the controller diffs the cached PSD against the FSD.

Response: one HostSummary per host with asic_count, arch, connected_hosts (reachable via exit nodes), connection_summaries (per-peer link counts), the source that host’s data came from, and metadata (hall/aisle/rack/shelf plus hardware quirks such as is_bh_glx_rev_c, the BH galaxy rev C tray layout). missing_links and unexpected_links are populated only when validate_topo is true. The response also reports total_cross_host_links (deduplicated host-pair connections) and host_groups, the connected components of the host graph.

Host group names

Each HostGrouping carries a name derived from the FSD instance_path of its members, so operators see the name the cabling hierarchy gives a group rather than a positional label. The name is the longest instance path the members share, ignoring the last segment of each path (that segment names the host, not a group).

The first segment of an instance path names the cluster and the second names one group within it, in the form <cabling template>-<unique group name>. The controller strips the cabling template prefix, so a group covering one whole group of the cluster is named e.g. 110-SC36_4x4_C1-10. Variations:

• A group confined to a single sub-instance shares its second segment with its siblings, so the deeper shared segments are appended: 110-SC36_4x4_C1-10/bh_galaxy_sp_0.

• A group spanning several groups of the cluster shares only the first segment and is named after the cluster, e.g. exabox.

• A leading token that contains no letters is not treated as a cabling template prefix, which leaves names such as 120-SC36_4x4_C1-10 (hall number first) intact.

• Groups no loaded FSD describes have an empty name; clients label those by position.

Pods

Each HostGrouping also lists the pods it subdivides into, so a caller can offer a pod as a unit. TT cluster hierarchies group nodes into pods and pods into superpods, but the depth varies — a superpod is occasionally described as a grouping of smaller superpods — so no fixed instance path segment identifies a pod. What does hold is that everything below a pod addresses a single host, so a pod is the deepest instance path prefix that still covers more than one host.

Pods are named relative to their group (bh_galaxy_sp_0, or sp2_0/bh_galaxy_sp_0 where the hierarchy nests deeper) and carry no pods of their own. A group lists no pods when the FSD describes nothing below the group itself, and a host that belongs to no pod appears only in the group’s host_names.