API Overview

Fabric Manager exposes the following gRPC services:

Service	Hosted by	Called by	Purpose
FabricManagerService	Controller	Orchestrators (SLURM, Kubernetes), CLI, SDK	Query topology, compute placements

The controller can additionally serve a read-mostly HTTP JSON API and web UI. For command-line access see the CLI reference; for programmatic access see the SDK.

Concepts

Topology sources: PSD and FSD

Most query RPCs take a topo_source field that selects where connectivity comes from:

Value	Name	Meaning
0	PSD	Physical System Descriptor — connectivity as discovered by the agents. The default.
1	FSD	Factory System Descriptor — connectivity derived from the controller-configured “golden state” wiring. Requires controller.factory_system_descriptor_search_path to be set and to cover the requested hosts.
3	COMBINED_PSD_FSD	The union of both, with each host tagged by the source it came from. Only accepted by QueryTopologySummary.

Value 2 is intentionally unused.

The Factory System Desctiptor (FSD) holds the “golden” connectivity state for a set of hosts equipped with TT accelerators. It is generated when a system is installed and cabled up.

ASIC descriptors (unique IDs, board types, PCI BDFs) always come from the discovered topology, even when topo_source is FSD — only the connectivity graph changes. Comparing the two sources is how FM detects missing and unexpected links.

Descriptors are passed as opaque strings and bytes

MeshGraphDescriptor (MGD), PhysicalSystemDescriptor (PSD), and FactorySystemDescriptor (FSD) cross the wire as textproto strings or serialized bytes rather than as structured protobuf fields. This deliberately avoids coupling FM’s protobuf version to tt-metalium’s, so FM builds against both Ubuntu 22.04 (protobuf 3.12) and 24.04 (protobuf 3.21). It also makes the API easy to drive from grpcurl. Deserialize with ParseFromString() (bytes) or TextFormat::ParseFromString() (textproto) on the client side.

Orchestrator Service (FabricManagerService)

RPC	Description
QueryPhysicalTopology	Returns the aggregated physical topology as a serialized PhysicalSystemDescriptor, plus supplemental per-ASIC and per-host metadata
GetValidPlacementsMGD	Maps a Mesh Graph Descriptor onto physical ASICs using the CSP solver and returns valid host/rank assignments
QueryFactorySystemDescriptor	Returns the controller-configured FSD files (“golden state”)
QueryTopologySummary	Host-level connectivity view, optionally diffed against the FSD

QueryPhysicalTopology

Aggregates the topology reported by all healthy agents.

Request (QueryPhysicalTopologyRequest)

Field	Description
hosts	Restrict to these host IDs (agent registration IDs). Empty means all healthy hosts
topo_source	PSD (default) or FSD
trim_links_outside_filter	When a host filter is applied and the source is PSD, drop cross-host links whose remote endpoint lives outside the filter. Ignored without a host filter. Defaults to false

Response (QueryPhysicalTopologyResponse)

Field	Description
physical_topology_bytes	Serialized tt.fabric.proto.PhysicalSystemDescriptor
total_hosts, total_asics	Summary counts
asic_metadata	Per-ASIC fields the PSD does not carry — pci_bdf, UMD chip_id, device_node_id, device_node_path, chip_arch, is_mmio_capable — keyed by asic_id so consumers can join against the PSD. Only populated for ASICs discovered at runtime
host_metadata	Per-host metadata (data-centre location, hardware quirks) keyed by host name

ASICs that exist only in the FSD have no asic_metadata entry.

GetValidPlacementsMGD

Runs the CSP mapper to find every valid way to place a logical mesh onto the physical fabric.

Request (PlacementRequestMGD)

Field	Description
mgd_textproto	tt.tt_fabric.proto.MeshGraphDescriptor in textproto form (required)
constraints.host_ids	Restrict placement to these hosts
constraints.pinnings	Pin a specific asic_id to a specific fabric_node_id
topo_source	PSD (default) or FSD

Response (PlacementResponseMGD) carries a status, an error_message, and a list of placements. Each ValidPlacement provides host assignments in two shapes: host_assignments (legacy, one rank per host) and host_assignments_with_ranks (multiple ranks per host). Both report the asic_ids used, the mesh_id/mesh_rank, the directly addressable visible_devices, and fabric_nodes — the logical (mesh_id, chip_id) FabricNodeId that each ASIC fulfills, which is what maps placed hardware back onto the MGD’s logical mesh.

NOTE: Currently the placement implementaion is limited to returning one placement option.

Status values:

Status	Meaning
PLACEMENT_MGD_SUCCESS	Placements found
PLACEMENT_MGD_ERROR_UNKNOWN	Unclassified failure
PLACEMENT_MGD_ERROR_IMPOSSIBLE	No valid mapping exists
PLACEMENT_MGD_ERROR_INSUFFICIENT	Not enough ASICs available
PLACEMENT_MGD_ERROR_INVALID_MGD	MGD failed to parse
PLACEMENT_MGD_ERROR_TOPO_VALIDATION	Topology validation failed (FSD placements only)
PLACEMENT_MGD_UNSTABLE_TOPO_FALLBACK	Validation could not run; all links report as missing and ASIC/chip IDs are not guaranteed to match the physical topology

Placement validation. When the request used FSD, the controller re-runs discovery on every host in the placement and diffs the fresh per-host connectivity against the FSD subset covering those hosts. The result lands in ValidPlacement.validation, with a per-host breakdown listing missing_links and unexpected_links:

PlacementValidationStatus	Meaning
PLACEMENT_VALIDATION_UNKNOWN	Validation not performed (e.g. a PSD placement)
PLACEMENT_VALIDATION_HEALTHY	Every host matched the FSD
PLACEMENT_VALIDATION_UNHEALTHY	At least one host has missing and/or unexpected links
PLACEMENT_VALIDATION_ERROR	Rediscovery could not complete (e.g. an agent was unreachable)
PLACEMENT_VALIDATION_USED_FALLBACK	UMD discovery failed but the sysfs fallback succeeded; links always report as missing and asic_id is unavailable

The CSP solve runs in a separate tt-fabric-placement-solver process. The controller snapshots the topology, pipes a request to the child over stdin, and reads the response from stdout, which lets it SIGKILL the solve when the client disconnects or the deadline elapses.

QueryFactorySystemDescriptor

Returns the FSD files the controller was configured with. hosts filters to specific hostnames. Each FactorySystemDescriptorFile carries a name and textproto_bytes (a serialized tt.scaleout_tools.fsd.proto.FactorySystemDescriptor). Requires controller.factory_system_descriptor_search_path.

QueryTopologySummary

A compact host-level view intended for dashboards and operators rather than for placement.

Request: topo_source (PSD, FSD, or COMBINED_PSD_FSD) and validate_topo — when true, the controller diffs the cached PSD against the FSD.

Response: one HostSummary per host with asic_count, arch, connected_hosts (reachable via exit nodes), connection_summaries (per-peer link counts), the source that host’s data came from, and metadata (hall/aisle/rack/shelf plus hardware quirks such as is_bh_glx_rev_c, the BH galaxy rev C tray layout). missing_links and unexpected_links are populated only when validate_topo is true. The response also reports total_cross_host_links (deduplicated host-pair connections) and host_groups, the connected components of the host graph.