# Architecture ``` ┌──────────────────────┐ ┌───────────────────────┐ │ Orchestrator │ │ Browser / operator │ │ (SLURM, k8s) │ │ │ └──────────┬───────────┘ └───────────┬───────────┘ │ gRPC │ HTTP (optional) ┌──────────▼──────────────────────────────────▼────────────┐ │ Controller │ │ ┌────────────────────────────┐ ┌───────────────────┐ │ │ │ FabricManagerService │ │ Web Server │ │ │ │ - QueryPhysicalTopology │ │ - /api/* │ │ │ │ - GetValidPlacementsMGD │ │ - static UI │ │ │ │ - StartPlacementMGD (LRO) │ └───────────────────┘ │ │ │ - QueryTopologySummary │ ┌───────────────────┐ │ │ │ - QueryFactorySystemDesc │ │ Endpoint Stats │ │ │ │ - QueryClusterDescriptors │ └───────────────────┘ │ │ │ - QueryEndpointStats │ ┌───────────────────┐ │ │ │ - GetTopologyInstances │ │ FSD / PGD (files)│ │ │ │ - RecoverFabricLinks │ └───────────────────┘ │ │ └────────────────────────────┘ │ │ ┌────────────────────────────┐ │ │ │ Placement job pool │ │ │ │ (bounded, cancellable) │ │ │ └─────────────┬──────────────┘ │ │ ┌─────────────▼──────────────┐ │ │ │ Topology Mapper │──fork/exec──► tt-fabric- │ │ │ (aggregate + validate) │ placement- │ │ └────────────────────────────┘ solver │ │ ┌────────────────────────────┐ (CSP) │ │ │ DaemonService │ │ │ │ - RegisterDaemon │ │ │ │ - SendHeartbeat │ │ │ └─────────────┬──────────────┘ │ └────────────────┼──────────────────────────┬──────────────┘ gRPC (agent ─► controller) gRPC (controller ─► agent) │ │ AgentService: ┌─────────────────┼──────────────────┐ │ - RediscoverTopology │ │ │ │ - RetrainLinks ┌──────▼──────────┐ ┌────▼────────────┐ ┌───▼───────▼─────┐ │ Agent (Host 1) │ │ Agent (Host 2) │ │ Agent (Host N) │ │ │ │ │ │ │ │ Device Discovery│ │ Device Discovery│ │ Device Discovery│ │ (UMD, fallback) │ │ (UMD, fallback) │ │ (UMD, fallback) │ │ Link Retrainer │ │ Link Retrainer │ │ Link Retrainer │ │ Telemetry Client│ │ Telemetry Client│ │ Telemetry Client│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │ gRPC (optional) │ │ ┌─────▼─────────┐ ┌─────▼─────────┐ ┌─────▼─────────┐ │ tt-telemetry │ │ tt-telemetry │ │ tt-telemetry │ │ collector │ │ collector │ │ collector │ └───────────────┘ └───────────────┘ └───────────────┘ ``` ## Components - **Agent** (`tt-fabric-manager-agent`): Runs on each host. Uses UMD to discover local Tenstorrent ASICs (unique IDs, board type, arch, memory, PCI address), intra-host ethernet connections, and cross-host exit nodes. Registers topology with the controller and sends periodic heartbeats. Also hosts `AgentService`, so the controller can drive rediscovery and link retraining on it, and optionally subscribes to a local tt-telemetry collector to react to link-state changes. - **Controller** (`tt-fabric-manager-controller`): Centralized coordinator. Aggregates physical topology from all registered agents, tracks host health via heartbeats, and exposes an orchestrator-facing gRPC API. Optionally serves an HTTP JSON API and web UI. - **Placement solver** (`tt-fabric-placement-solver`): A short-lived child process that runs tt-metalium's CSP mapper. ## Two views of the fabric: discovered and expected Much of FM's behaviour comes from holding two topologies side by side: - The **Physical System Descriptor (PSD)** is the live, discovered state, aggregated from what the agents report. - The **Factory System Descriptor (FSD)** is the expected "golden state" wiring, loaded by the controller from `factory_system_descriptor_search_path`. Most query RPCs take a `topo_source` selector, and diffing the two is how FM reports **missing links** (in the FSD, absent from discovery) and **unexpected links** (discovered but not in the FSD). Those diffs are what `QueryTopologySummary --validate`, placement validation, and `RecoverFabricLinks` operate on. A **Physical Grouping Descriptor (PGD)** supplies hardware-specific tray layout knowledge to the mapper. Note that ASIC descriptors always come from discovery; only the connectivity graph switches with `topo_source`. ## Discovery and its fallback The agent prefers UMD topology discovery, which yields a full cluster descriptor. When UMD fails it falls back to scanning `/dev/tenstorrent/*` and sysfs, which finds the local devices but no connectivity and produces no cluster descriptor. Consumers can tell the difference: fallback results surface as `REDISCOVER_FALLBACK` and as `PLACEMENT_VALIDATION_USED_FALLBACK` during placement validation, where every link necessarily reports as missing. Because a first discovery can transiently find nothing while devices are still coming up, an empty startup scan is retried a bounded number of times (see `discovery` in the [configuration reference](config-reference.md)). ## Data Flow — Startup & Registration ``` Agent (Host N) Controller ───────────── ────────── │ │ │ 1. Discover local ASICs (UMD) │ │◄─────────────────────┐ │ │ │ │ │ 2. RegisterDaemon(HostPhysicalTopology) │─────────────────────────────────────►│ │ │ 3. Store topology │ RegisterResponse │ in memory │◄─────────────────────────────────────│ │ │ │ 4. SendHeartbeat (unary, periodic) │ │─────────────────────────────────────►│ │ - Periodic keepalive │ │ - Health monitoring │ │ - Optional updated_topology │ │ HeartbeatAck │ │◄─────────────────────────────────────│ │ (ACK, or NACK ─► re-register) │ │ ``` A host that stops heartbeating for `heartbeat_timeout_sec` is marked offline and excluded from query results. ## Data Flow — Mesh Placement Query ``` Orchestrator Controller Solver Agent(s) ──────────── ────────── ────── ──────── │ │ │ │ │ GetValidPlacementsMGD │ │ │ │ (MGD textproto) │ │ │ │────────────────────────►│ │ │ │ │ │ │ │ │ Aggregate topology from │ │ │ registered agents (PSD or FSD) │ │ │ │ │ │ │ fork/exec + pipe │ │ │ │───────────────────►│ │ │ │ CSP solve (map_mesh_to_physical) │ │◄───────────────────│ │ │ │ │ │ │ │ If source was FSD: │ │ │ RediscoverTopology on each │ │ │ placement host, diff vs FSD │ │ │────────────────────────────────►│ │ │◄────────────────────────────────│ │ │ │ │ │ PlacementResponseMGD │ │ │ │ (host/rank → ASIC │ │ │ │ assignments, │ │ │ │ fabric nodes, │ │ │ │ validation) │ │ │ │◄────────────────────────│ │ │ │ │ │ │ ``` The pipeline in the middle is the same regardless of how the placement was asked for. It always runs on the controller's placement worker pool, never on the handler thread, which is what bounds how many solver children can exist at once and lets a placement's lifetime be decoupled from any one caller's: - `GetValidPlacementsMGD` submits a job and blocks on it, so a client sees the synchronous call above. If the client disconnects and nobody else wants the same placement, the job is cancelled and the solver child is killed. - `StartPlacementMGD` submits the same job and returns its handle, then `GetPlacementOperation` reports the stage the pipeline is in (solving, validating, plus how many hosts it has validated) and finally the result. The job survives the client, and its result is retained for `result_ttl_sec` so a reconnecting client can still collect it. - Identical requests in flight at the same time collapse onto one job, so a client retry or two orchestrators asking the same question cost one solve. Limits live in [`controller.placement`](config-reference.md#controllerplacement). ## Data Flow — Telemetry-Driven Rediscovery When the tt-telemetry integration is enabled, the fabric view stays current without waiting for re-registration. ``` Collector Agent Controller ───────── ───── ────────── │ │ │ │ StreamMetrics │ │ │ (link_up, cable) │ │ │◄───subscribe───────────│ │ │ │ │ │ link state change │ │ │───────────────────────►│ │ │ │ Coalesce a burst of events │ │ │ (rediscovery_min_interval) │ │ │ │ │ │ Re-run local discovery │ │ │ │ │ │ SendHeartbeat with │ │ │ updated_topology │ │ │────────────────────────────►│ │ │ │ ``` ## Data Flow — Link Recovery A missing link, once identified by validation, can be retrained in place. ``` Operator / Orchestrator Controller Agents ─────────────────────── ────────── ────── │ │ │ │ QueryTopologySummary │ │ │ (validate_topo: true) │ │ │─────────────────────────►│ Diff PSD vs FSD │ │ missing / unexpected │ │ │◄─────────────────────────│ │ │ │ │ │ RecoverFabricLinks │ │ │ (missing links by host) │ │ │─────────────────────────►│ │ │ │ Dedup, batch in rounds │ │ │ │ │ │ RetrainLinks (parallel │ │ │ to both link ends) │ │ │────────────────────────►│ │ │◄────────────────────────│ │ per-host status │ │ │◄─────────────────────────│ │ │ │ │ ``` Both ends of a link must be kicked off within a few seconds of each other, because the ETH firmware only completes a reinit once the peer is also reiniting. The agent handles that for intra-host links by firing every link in a batch before waiting on any of them; the controller handles cross-host links by issuing parallel RPCs to each host.