CLI Reference

tt-fabric-manager-cli is a thin client over the controller’s FabricManagerService.

tt-fabric-manager-cli [OPTIONS] <command> [COMMAND_ARGS]

Running CLI in Docker

Currently the most convenient way to obtain a pre-built FM CLI is via the official FM image. For example:

alias tt-fabric-manager-cli='docker run --rm --net=host \
  -e FABRIC_MANAGER_ENDPOINT=ttfm.example.com:80 \
  -v $(pwd):$(pwd) \
  -w $(pwd) \
  ghcr.io/tenstorrent/tt-fabric-manager:latest-cli tt-fabric-manager-cli'

Connecting to a controller

The endpoint is resolved from the following sources, highest priority first:

  1. --endpoint <addr>

  2. FABRIC_MANAGER_ENDPOINT environment variable

  3. /etc/fabric-manager/config.yaml (controller.listen_address or controller.endpoint)

  4. ~/.fabric-manager/config

  5. /etc/fabric-manager/client.conf

  6. Built-in default, localhost:50051

Config files are YAML. A lightweight client-only file needs just two keys:

endpoint: localhost:50052
timeout_seconds: 30

The full server config is also accepted, in which case the endpoint is read from controller.endpoint or controller.listen_address and the timeout from controller.timeout_seconds.

Global options

Option

Default

Description

--endpoint <addr>

localhost:50051

Controller address

--timeout <sec>

60

gRPC deadline applied to each individual RPC

--verbose, -v

off

Diagnostics to stderr, prefixed [tt-fabric-manager-cli]

--help, -h

Print usage and exit

--timeout used to default to 600 seconds because a placement on a large fabric can run for minutes and the CLI waited for it in a single call. It no longer does: placement runs as an operation on the controller and the CLI polls it, so every RPC returns promptly and the deadline only has to cover the heaviest single query. Use --placement-budget to bound how long the CLI waits for a placement, and --sync if you need the old single-call behaviour back.

Several options are shared by more than one command: --host-id <id> (repeatable, filters by host), --output <path> / -o (write payload to a file instead of inlining it in the JSON), and --source <psd|fsd> (see topology sources).

Commands

query-physical-topology

Fetches the aggregated PhysicalSystemDescriptor via QueryPhysicalTopology.

Option

Description

--host-id <hostname>

Restrict to these hosts (repeatable)

--source <psd|fsd>

Connectivity source. Default psd

--output <path>

Write the descriptor as textproto to path instead of embedding it in the JSON

 
# Whole fabric, as JSON
tt-fabric-manager-cli query-physical-topology

# Two hosts, golden-state wiring, saved as textproto
tt-fabric-manager-cli query-physical-topology \
  --host-id host-a --host-id host-b --source fsd -o topology.textproto

Exits non-zero if the query fails. The trim_links_outside_filter request field is not exposed by the CLI; use the HTTP API or the SDK for that.

query-factory-descriptor

Fetches the controller’s configured FSD files via QueryFactorySystemDescriptor.

Option

Description

--host-id <hostname>

Filter the FSD to these hostnames (repeatable)

--output <path>

Write the serialized FSD to a file. With multiple files, they are suffixed path.0, path.1, …

Requires the controller to be configured with controller.factory_system_descriptor_search_path.

get-valid-placements-mgd

Maps a Mesh Graph Descriptor onto the fabric, and can emit launcher configuration for a chosen placement.

By default this starts a placement operation on the controller and polls it to completion, printing the same JSON as before plus the operation_id and operation_state it ran as. Ctrl-C cancels the operation on the controller rather than merely walking away from it, and exits 130. Progress goes to stderr when it is a terminal, or with --verbose.

Option

Default

Description

--mgd-file <path>

required

MGD in textproto form

--host-id <id>

Restrict placement to these hosts (repeatable)

--source <psd|fsd>

psd

Connectivity source

--placement-budget <sec>

0

How long to wait. 0 waits as long as the controller runs the operation, which it already bounds

--max-runtime <sec>

controller’s maximum

Budget requested for the operation on the controller, clamped down to that maximum

--idempotency-key <key>

derived from the request

Makes retrying safe: a retry with the same key joins the running operation instead of starting a second solve

--async

off

Start the placement, print its operation id, and exit without waiting

--sync

off

Use the single blocking GetValidPlacementsMGD call, which needs --timeout to cover the whole solve

--placement-index <n>

0

Which returned placement the launcher outputs describe

--rank-bindings-out <path>

Also write a rank bindings YAML (TT_VISIBLE_DEVICES built from chip IDs)

--rankfile-out <path>

Also write an OpenMPI rankfile

--mesh-graph-desc-path <path>

value of --mgd-file

Value written to the rank bindings file’s mesh_graph_desc_path field

 
tt-fabric-manager-cli get-valid-placements-mgd \
  --mgd-file examples/mgd.2x4.textproto \
  --rank-bindings-out rank_bindings.yaml \
  --rankfile-out rankfile

The command exits non-zero unless the status is PLACEMENT_MGD_SUCCESS, so it can gate a job launch directly. When the wait ends without a placement — cancelled, out of budget on the controller, or interrupted — status carries the operation’s state instead of a placement status, placements is empty, and operation_id identifies the operation to follow up on.

Against a controller too old to know about placement operations, the command falls back to the blocking call automatically (visible with --verbose), in which case --timeout has to cover the solve again.

Warning

Only textproto MGDs are accepted — the file must end in .textproto or .txt. Binary .pb files are rejected with a message pointing at protoc; convert first. (The built-in --help text still claims .pb is supported.)

placement-operation

Shows one placement operation, and optionally waits for it. The counterpart to --async: start a placement in one job step and collect it in another, without either holding a connection open for the duration.

Option

Default

Description

--operation-id <id>

required

Operation to show

--wait

off

Poll until the operation finishes, then print it with its result

--placement-budget <sec>

0

How long to wait when using --wait

 
OP=$(tt-fabric-manager-cli get-valid-placements-mgd --mgd-file mesh.textproto --async | jq -r .operation_id)
tt-fabric-manager-cli placement-operation --operation-id "$OP" --wait

Exits zero whenever the operation could be read, including while it is still running: state and result.status carry the outcome. An id the controller does not know is an error with error_code: NOT_FOUND, which also covers results past their retention window and ids from a previous controller process.

cancel-placement-operation

Stops a placement operation on the controller. Cancellation is cooperative, so the reported state may still be SOLVING and reach CANCELLED shortly after. Cancelling an operation that already finished changes nothing and keeps its result.

Option

Default

Description

--operation-id <id>

required

Operation to stop

list-placement-operations

Lists the placements the controller is running or still retains results for, most recently created first — including those started by other clients and by the web UI. Useful for answering “what is this controller busy with?”.

Option

Default

Description

--include-results

off

Attach each finished operation’s placement

show-topology

The one human-oriented command: renders an ASCII summary of hosts and cross-host connectivity from QueryTopologySummary. Output width follows the terminal, or COLUMNS when it cannot be detected. Errors go to stderr.

Option

Description

--source <psd|fsd|combined>

Build the view from the discovered PSD (default), the FSD, or the union of both. combined also tags each host with its source and FSD data-centre location

--validate

Ask the controller to diff PSD against FSD and report missing and unexpected cross-host links per host. Requires a configured FSD

 
tt-fabric-manager-cli show-topology --source combined --validate