Architecture

        ┌──────────────────────┐          ┌───────────────────────┐
        │   Orchestrator       │          │   Browser / operator  │
        │   (SLURM, k8s)       │          │                       │
        └──────────┬───────────┘          └───────────┬───────────┘
                   │ gRPC                             │ HTTP (optional)
        ┌──────────▼──────────────────────────────────▼────────────┐
        │                      Controller                          │
        │  ┌────────────────────────────┐  ┌───────────────────┐   │
        │  │  FabricManagerService      │  │  Web Server       │   │
        │  │  - QueryPhysicalTopology   │  │  - /api/*         │   │
        │  │  - GetValidPlacementsMGD   │  │  - static UI      │   │
        │  │  - QueryTopologySummary    │  └───────────────────┘   │
        │  │  - QueryFactorySystemDesc  │  ┌───────────────────┐   │
        │  │  - QueryClusterDescriptors │  │  Endpoint Stats   │   │
        │  │  - QueryEndpointStats      │  └───────────────────┘   │
        │  │  - GetTopologyInstances    │  ┌───────────────────┐   │
        │  │  - RecoverFabricLinks      │  │  FSD / PGD (files)│   │
        │  └────────────────────────────┘  └───────────────────┘   │
        │  ┌────────────────────────────┐                          │
        │  │  Topology Mapper           │──fork/exec──► tt-fabric- │
        │  │  (aggregate + validate)    │               placement- │
        │  └────────────────────────────┘               solver     │
        │  ┌────────────────────────────┐                (CSP)     │
        │  │  DaemonService             │                          │
        │  │  - RegisterDaemon          │                          │
        │  │  - SendHeartbeat           │                          │
        │  └─────────────┬──────────────┘                          │
        └────────────────┼──────────────────────────┬──────────────┘
              gRPC (agent ─► controller)     gRPC (controller ─► agent)
                         │                          │  AgentService:
       ┌─────────────────┼──────────────────┐       │  - RediscoverTopology
       │                 │                  │       │  - RetrainLinks
┌──────▼──────────┐ ┌────▼────────────┐ ┌───▼───────▼─────┐
│  Agent (Host 1) │ │  Agent (Host 2) │ │  Agent (Host N) │
│                 │ │                 │ │                 │
│ Device Discovery│ │ Device Discovery│ │ Device Discovery│
│ (UMD, fallback) │ │ (UMD, fallback) │ │ (UMD, fallback) │
│ Link Retrainer  │ │ Link Retrainer  │ │ Link Retrainer  │
│ Telemetry Client│ │ Telemetry Client│ │ Telemetry Client│
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
         │ gRPC (optional)   │                   │
   ┌─────▼─────────┐   ┌─────▼─────────┐   ┌─────▼─────────┐
   │ tt-telemetry  │   │ tt-telemetry  │   │ tt-telemetry  │
   │ collector     │   │ collector     │   │ collector     │
   └───────────────┘   └───────────────┘   └───────────────┘

Components

  • Agent (tt-fabric-manager-agent): Runs on each host. Uses UMD to discover local Tenstorrent ASICs (unique IDs, board type, arch, memory, PCI address), intra-host ethernet connections, and cross-host exit nodes. Registers topology with the controller and sends periodic heartbeats. Also hosts AgentService, so the controller can drive rediscovery and link retraining on it, and optionally subscribes to a local tt-telemetry collector to react to link-state changes.

  • Controller (tt-fabric-manager-controller): Centralized coordinator. Aggregates physical topology from all registered agents, tracks host health via heartbeats, and exposes an orchestrator-facing gRPC API. Optionally serves an HTTP JSON API and web UI.

  • Placement solver (tt-fabric-placement-solver): A short-lived child process that runs tt-metalium’s CSP mapper.

Two views of the fabric: discovered and expected

Much of FM’s behaviour comes from holding two topologies side by side:

  • The Physical System Descriptor (PSD) is the live, discovered state, aggregated from what the agents report.

  • The Factory System Descriptor (FSD) is the expected “golden state” wiring, loaded by the controller from factory_system_descriptor_search_path.

Most query RPCs take a topo_source selector, and diffing the two is how FM reports missing links (in the FSD, absent from discovery) and unexpected links (discovered but not in the FSD). Those diffs are what QueryTopologySummary --validate, placement validation, and RecoverFabricLinks operate on. A Physical Grouping Descriptor (PGD) supplies hardware-specific tray layout knowledge to the mapper.

Note that ASIC descriptors always come from discovery; only the connectivity graph switches with topo_source.

Discovery and its fallback

The agent prefers UMD topology discovery, which yields a full cluster descriptor. When UMD fails it falls back to scanning /dev/tenstorrent/* and sysfs, which finds the local devices but no connectivity and produces no cluster descriptor. Consumers can tell the difference: fallback results surface as REDISCOVER_FALLBACK and as PLACEMENT_VALIDATION_USED_FALLBACK during placement validation, where every link necessarily reports as missing.

Because a first discovery can transiently find nothing while devices are still coming up, an empty startup scan is retried a bounded number of times (see discovery in the configuration reference).

Data Flow — Startup & Registration

  Agent (Host N)                          Controller
  ─────────────                          ──────────
       │                                      │
       │  1. Discover local ASICs (UMD)       │
       │◄─────────────────────┐               │
       │                      │               │
       │  2. RegisterDaemon(HostPhysicalTopology)
       │─────────────────────────────────────►│
       │                                      │  3. Store topology
       │              RegisterResponse        │     in memory
       │◄─────────────────────────────────────│
       │                                      │
       │  4. SendHeartbeat (unary, periodic)  │
       │─────────────────────────────────────►│
       │     - Periodic keepalive             │
       │     - Health monitoring              │
       │     - Optional updated_topology      │
       │              HeartbeatAck            │
       │◄─────────────────────────────────────│
       │              (ACK, or NACK ─► re-register)
       │                                      │

A host that stops heartbeating for heartbeat_timeout_sec is marked offline and excluded from query results.

Data Flow — Mesh Placement Query

  Orchestrator              Controller             Solver      Agent(s)
  ────────────              ──────────             ──────      ────────
       │                         │                    │            │
       │  GetValidPlacementsMGD  │                    │            │
       │  (MGD textproto)        │                    │            │
       │────────────────────────►│                    │            │
       │                         │                    │            │
       │                         │  Aggregate topology from        │
       │                         │  registered agents (PSD or FSD) │
       │                         │                    │            │
       │                         │  fork/exec + pipe  │            │
       │                         │───────────────────►│            │
       │                         │   CSP solve (map_mesh_to_physical)
       │                         │◄───────────────────│            │
       │                         │                    │            │
       │                         │  If source was FSD:             │
       │                         │  RediscoverTopology on each     │
       │                         │  placement host, diff vs FSD    │
       │                         │────────────────────────────────►│
       │                         │◄────────────────────────────────│
       │                         │                    │            │
       │  PlacementResponseMGD   │                    │            │
       │  (host/rank → ASIC      │                    │            │
       │   assignments,          │                    │            │
       │   fabric nodes,         │                    │            │
       │   validation)           │                    │            │
       │◄────────────────────────│                    │            │
       │                         │                    │            │

Data Flow — Telemetry-Driven Rediscovery

When the tt-telemetry integration is enabled, the fabric view stays current without waiting for re-registration.

  Collector                  Agent                      Controller
  ─────────                  ─────                      ──────────
      │                        │                             │
      │  StreamMetrics         │                             │
      │  (link_up, cable)      │                             │
      │◄───subscribe───────────│                             │
      │                        │                             │
      │  link state change     │                             │
      │───────────────────────►│                             │
      │                        │  Coalesce a burst of events │
      │                        │  (rediscovery_min_interval) │
      │                        │                             │
      │                        │  Re-run local discovery     │
      │                        │                             │
      │                        │  SendHeartbeat with         │
      │                        │  updated_topology           │
      │                        │────────────────────────────►│
      │                        │                             │