Configuration

These are the Helm chart’s configurable values. Override them with --set <key>=<value> or a values file (-f values.yaml) at install or upgrade time.

When installed through tt-operator, the same keys apply under a tt-fabric-manager. prefix.

For bare-metal and Docker deployments the chart is not involved; edit the YAML config file directly, as described in the configuration reference.

Image

Key	Default	Description
`image.repository`	`ghcr.io/tenstorrent/tt-fabric-manager`	Image used by both the controller and the agent
`image.tag`	chart `appVersion` (`latest`)	Image tag
`image.pullPolicy`	`IfNotPresent`	Image pull policy
`imagePullSecrets`	`[]`	Pull secrets that already exist in the install namespace
`nameOverride`		Override the chart name used in resource names
`fullnameOverride`		Override the full release name used in resource names
`serviceAccount.create`		Create a ServiceAccount for the pods
`serviceAccount.name`		Name of the ServiceAccount to create or use

Scaleout configs (FSDs)

Factory System Descriptors are supplied to both pods, mounted read-only at /scaleout_configs, from one of two sources:

Key	Default	Description
`scaleoutConfigsHostPath`	`/data/scaleout_configs`	Host directory holding the FSDs. Ignored when `scaleoutConfigsImage` is set
`scaleoutConfigsImage`	`""`	OCI image whose filesystem provides the FSDs, mounted as a Kubernetes image volume instead of the host path, e.g. `ghcr.io/tenstorrent/tt-cluster-config/exabox:<tag>`

The image volume route requires the ImageVolume feature (beta and on by default since Kubernetes 1.33). Pulls reuse the pod’s imagePullSecrets through the service account.

Mounting the configs is not enough on its own — point controller.factorySystemDescriptorSearchPath at a path inside the mount to actually enable the FSD-backed features.

Controller

Key	Default	Description
`controller.replicaCount`	`1`	Replica count. The controller holds topology state in memory, so treat it as stateful and leave this at 1
`controller.factorySystemDescriptorSearchPath`	unset	Path searched for FSD textprotos, normally under `/scaleout_configs`. Required for `topo_source: FSD`, topology validation, and `QueryFactorySystemDescriptor`
`controller.physicalGroupingDescriptorPath`	unset	Path to a Physical Grouping Descriptor, giving the mapper hardware-specific tray layout knowledge
`controller.supportedTopologyShapes`	`[]`	Shapes that `GetTopologyInstances` can enumerate. Each entry takes `canonical_name`, optional `aliases`, and `instance_path_segment_pattern`
`controller.resources`	`{}`	Container resource requests and limits
`controller.readinessProbe`	`grpc-health-probe` on `:50052`	Readiness probe
`controller.livenessProbe`	unset	Liveness probe
`controller.nodeSelector`	`{}`	Node selector
`controller.affinity`	`{}`	Affinity rules
`controller.tolerations`	`[]`	Tolerations
`controller.podAnnotations`	`{}`	Extra pod annotations
`controller.podLabels`	`{}`	Extra pod labels
`controller.podSecurityContext`	`{}`	Pod security context
`controller.securityContext`	`{}`	Container security context

The controller always serves gRPC on port 50052, and its heartbeat timings are fixed by the chart (60s interval, 90s timeout, 120s reconnect grace). These are not exposed as values; change them by templating your own ConfigMap if you need to.

CPU requirement

The controller links tt-metalium, which is compiled for the x86-64-v3 microarchitecture level: AVX, AVX2, BMI1, BMI2, FMA, F16C, LZCNT and MOVBE. Those instructions run in libtt_metal’s static initializers, before main, so a controller scheduled onto a node that lacks them dies immediately on SIGILL — container exit code 132, no log output, CrashLoopBackOff. Common triggers are a pre-Haswell host in an otherwise modern pool, a hypervisor or cloud instance exposing a generic CPU model that masks AVX2/BMI2, and running the amd64 image on arm64 under emulation.

The agent shares the same dependency but only runs on accelerator hosts, so in practice this bites the controller, which is normally scheduled away from them. Where the controller can land on a heterogeneous pool, constrain it with a required controller.affinity node affinity. If Node Feature Discovery’s cpu source is enabled, key on feature.node.kubernetes.io/cpu-cpuid.AVX2 directly. If it is not — as in values.exabox.yaml, where NFD publishes only pci-* labels — there is no CPU feature label to require, and pinning to known-good hosts by kubernetes.io/hostname is the honest fallback. Either way, a pod left Pending with an unsatisfied node affinity is far easier to diagnose than a silent crash loop.

To confirm a suspected node, compare the dynamic loader’s view against the ISA level:

/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --help | grep 'x86-64-v'
grep -o -E 'avx2|bmi2|fma' /proc/cpuinfo | sort -u

Web UI

See Web UI & HTTP API for what these expose.

Key	Default	Description
`controller.web.enabled`	`false`	Serve the HTTP JSON API and web UI alongside gRPC
`controller.web.port`	`8080`	Container port for the HTTP server
`controller.web.uiPath`	`/usr/local/share/tt-fabric-manager/web`	Static UI assets inside the image. Set to `""` to serve the JSON API with no UI

Warning

The HTTP endpoint is unauthenticated. Only expose it on trusted networks.

Agent

The agent runs as a privileged DaemonSet, registering under the node name as its host ID and advertising its pod IP on port 50053.

Key	Default	Description
`agent.affinity`	`{}`	Overrides the default node targeting (see below)
`agent.nodeSelector`		Legacy node selector. Still honoured, but ANDs with whatever affinity is in effect
`agent.tolerations`	`[]`	Tolerations
`agent.resources`	`{}`	Container resource requests and limits
`agent.livenessProbe`	`grpc-health-probe` on `:50053`	Liveness probe
`agent.readinessProbe`	`grpc-health-probe` on `:50053`	Readiness probe
`agent.extraArgs`		Extra arguments appended to the agent command, e.g. `--force-fallback-discovery`
`agent.extraVolumes` / `agent.extraVolumeMounts`		Additional volumes and mounts
`agent.podAnnotations`	`{}`	Extra pod annotations
`agent.podLabels`	`{}`	Extra pod labels
`agent.podSecurityContext`	`{}`	Pod security context
`agent.securityContext`	`{}`	Container security context. The container always runs privileged, as UMD needs PCI config space access

By default the DaemonSet lands on every Tenstorrent-equipped node, matching the union of tenstorrent.com/has-tt=true (applied manually, e.g. via tt-ansible) and feature.node.kubernetes.io/pci-1200_1e52.present=true (applied automatically by NFD). Setting agent.affinity replaces that whole expression.

Run tt-fabric-manager-agent --help for the flags you can pass through agent.extraArgs; the configuration reference covers the config-file keys instead.

OpenTelemetry

Key	Default	Description
`otel.enabled`	`false`	Emit OTEL logs, traces, and metrics from the controller
`otel.serviceName`	`tt-fabric-manager-controller`	Reported service name
`otel.deploymentEnvironment`	`""`	Reported deployment environment
`otel.metricExportIntervalSec`	`10`	Metric export interval

Telemetry currently goes to stdout in OStream/OTLP-style format, so pair it with a sidecar Collector or stdout scraping until OTLP/gRPC export lands.

Ingress

Two independent Ingress objects, because the gRPC backend needs HTTP/2 while the UI is plain HTTP. Both share the ingress controller’s external ports (80/443) and are routed by Host header, so they need distinct hostnames.

Key	Default	Description
`ingress.enabled`	`false`	Ingress for the external gRPC endpoint
`ingress.host`	`""`	Hostname for the gRPC endpoint
`ingress.className`	`""`	Ingress class
`ingress.annotations`	`{}`	Ingress annotations
`ingress.tls`	`[]`	TLS configuration
`ingress.forwardingTimeouts.dialTimeout`	`30s`	Time to establish a connection to the controller
`ingress.forwardingTimeouts.responseHeaderTimeout`	`60s`	Time the proxy waits for the controller to start responding
`ingress.forwardingTimeouts.idleConnTimeout`	`90s`	How long an idle backend connection is kept
`ingress.ui.enabled`	`false`	Ingress for the web UI. Requires `controller.web.enabled`
`ingress.ui.host`	`""`	Hostname for the web UI
`ingress.ui.className`	`""`	Ingress class for the UI
`ingress.ui.annotations`	`{}`	Ingress annotations for the UI
`ingress.ui.tls`	`[]`	TLS configuration for the UI

Enabling ingress also creates a Traefik ServersTransport carrying those forwarding timeouts, which the gRPC Ingress references by annotation. The UI Ingress does not use it, so the web UI keeps the ingress controller’s own defaults.

The timeouts used to be 900s, because a placement ran inside a single blocking RPC and every hop of the stack therefore had to outlive a whole solve. Placement is now a long-running operation that the client starts and polls, so no request stays open for longer than an ordinary RPC and these are sized accordingly.

Raise responseHeaderTimeout if callers still use the blocking GetValidPlacementsMGD — which is what tt-fabric-manager-cli --sync sends — since that holds one request open for as long as the solve takes.

Example

controller:
  factorySystemDescriptorSearchPath: /scaleout_configs/mysite/factory_system_descriptor.textproto
  web:
    enabled: true

ingress:
  enabled: true
  host: ttfm.example.com
  ui:
    enabled: true
    host: ttfm-ui.example.com

Site-specific values files live alongside the chart, for example values.exabox.yaml.