# Configuration These are the Helm chart's configurable values. Override them with `--set =` or a values file (`-f values.yaml`) at install or upgrade time. When installed through [tt-operator](https://docs.tenstorrent.com/tt-operator/), the same keys apply under a `tt-fabric-manager.` prefix. For bare-metal and Docker deployments the chart is not involved; edit the YAML config file directly, as described in the [configuration reference](config-reference.md). ## Image | Key | Default | Description | |-----|---------|-------------| | `image.repository` | `ghcr.io/tenstorrent/tt-fabric-manager` | Image used by both the controller and the agent | | `image.tag` | chart `appVersion` (`latest`) | Image tag | | `image.pullPolicy` | `IfNotPresent` | Image pull policy | | `imagePullSecrets` | `[]` | Pull secrets that already exist in the install namespace | | `nameOverride` | | Override the chart name used in resource names | | `fullnameOverride` | | Override the full release name used in resource names | | `serviceAccount.create` | | Create a ServiceAccount for the pods | | `serviceAccount.name` | | Name of the ServiceAccount to create or use | ## Scaleout configs (FSDs) Factory System Descriptors are supplied to both pods, mounted read-only at `/scaleout_configs`, from one of two sources: | Key | Default | Description | |-----|---------|-------------| | `scaleoutConfigsHostPath` | `/data/scaleout_configs` | Host directory holding the FSDs. Ignored when `scaleoutConfigsImage` is set | | `scaleoutConfigsImage` | `""` | OCI image whose filesystem provides the FSDs, mounted as a Kubernetes image volume instead of the host path, e.g. `ghcr.io/tenstorrent/tt-cluster-config/exabox:` | The image volume route requires the `ImageVolume` feature (beta and on by default since Kubernetes 1.33). Pulls reuse the pod's `imagePullSecrets` through the service account. Mounting the configs is not enough on its own — point `controller.factorySystemDescriptorSearchPath` at a path inside the mount to actually enable the FSD-backed features. ## Controller | Key | Default | Description | |-----|---------|-------------| | `controller.replicaCount` | `1` | Replica count. The controller holds topology state in memory, so treat it as stateful and leave this at 1 | | `controller.factorySystemDescriptorSearchPath` | unset | Path searched for FSD textprotos, normally under `/scaleout_configs`. Required for `topo_source: FSD`, topology validation, and `QueryFactorySystemDescriptor` | | `controller.physicalGroupingDescriptorPath` | unset | Path to a Physical Grouping Descriptor, giving the mapper hardware-specific tray layout knowledge | | `controller.supportedTopologyShapes` | `[]` | Shapes that `GetTopologyInstances` can enumerate. Each entry takes `canonical_name`, optional `aliases`, and `instance_path_segment_pattern` | | `controller.resources` | `{}` | Container resource requests and limits | | `controller.readinessProbe` | `grpc-health-probe` on `:50052` | Readiness probe | | `controller.livenessProbe` | unset | Liveness probe | | `controller.nodeSelector` | `{}` | Node selector | | `controller.affinity` | `{}` | Affinity rules | | `controller.tolerations` | `[]` | Tolerations | | `controller.podAnnotations` | `{}` | Extra pod annotations | | `controller.podLabels` | `{}` | Extra pod labels | | `controller.podSecurityContext` | `{}` | Pod security context | | `controller.securityContext` | `{}` | Container security context | The controller always serves gRPC on port **50052**, and its heartbeat timings are fixed by the chart (60s interval, 90s timeout, 120s reconnect grace). These are not exposed as values; change them by templating your own ConfigMap if you need to. ### CPU requirement The controller links tt-metalium, which is compiled for the `x86-64-v3` microarchitecture level: AVX, AVX2, BMI1, BMI2, FMA, F16C, LZCNT and MOVBE. Those instructions run in `libtt_metal`'s static initializers, before `main`, so a controller scheduled onto a node that lacks them dies immediately on SIGILL — container exit code **132**, no log output, `CrashLoopBackOff`. Common triggers are a pre-Haswell host in an otherwise modern pool, a hypervisor or cloud instance exposing a generic CPU model that masks AVX2/BMI2, and running the amd64 image on arm64 under emulation. The agent shares the same dependency but only runs on accelerator hosts, so in practice this bites the controller, which is normally scheduled away from them. Where the controller can land on a heterogeneous pool, constrain it with a required `controller.affinity` node affinity. If Node Feature Discovery's `cpu` source is enabled, key on `feature.node.kubernetes.io/cpu-cpuid.AVX2` directly. If it is not — as in `values.exabox.yaml`, where NFD publishes only `pci-*` labels — there is no CPU feature label to require, and pinning to known-good hosts by `kubernetes.io/hostname` is the honest fallback. Either way, a pod left `Pending` with an unsatisfied node affinity is far easier to diagnose than a silent crash loop. To confirm a suspected node, compare the dynamic loader's view against the ISA level: ```bash /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --help | grep 'x86-64-v' grep -o -E 'avx2|bmi2|fma' /proc/cpuinfo | sort -u ``` ### Web UI See [Web UI & HTTP API](web-ui.md) for what these expose. | Key | Default | Description | |-----|---------|-------------| | `controller.web.enabled` | `false` | Serve the HTTP JSON API and web UI alongside gRPC | | `controller.web.port` | `8080` | Container port for the HTTP server | | `controller.web.uiPath` | `/usr/local/share/tt-fabric-manager/web` | Static UI assets inside the image. Set to `""` to serve the JSON API with no UI | ```{warning} The HTTP endpoint is unauthenticated. Only expose it on trusted networks. ``` ## Agent The agent runs as a privileged DaemonSet, registering under the node name as its host ID and advertising its pod IP on port 50053. | Key | Default | Description | |-----|---------|-------------| | `agent.affinity` | `{}` | Overrides the default node targeting (see below) | | `agent.nodeSelector` | | Legacy node selector. Still honoured, but ANDs with whatever affinity is in effect | | `agent.tolerations` | `[]` | Tolerations | | `agent.resources` | `{}` | Container resource requests and limits | | `agent.livenessProbe` | `grpc-health-probe` on `:50053` | Liveness probe | | `agent.readinessProbe` | `grpc-health-probe` on `:50053` | Readiness probe | | `agent.extraArgs` | | Extra arguments appended to the agent command, e.g. `--force-fallback-discovery` | | `agent.extraVolumes` / `agent.extraVolumeMounts` | | Additional volumes and mounts | | `agent.podAnnotations` | `{}` | Extra pod annotations | | `agent.podLabels` | `{}` | Extra pod labels | | `agent.podSecurityContext` | `{}` | Pod security context | | `agent.securityContext` | `{}` | Container security context. The container always runs privileged, as UMD needs PCI config space access | By default the DaemonSet lands on every Tenstorrent-equipped node, matching the union of `tenstorrent.com/has-tt=true` (applied manually, e.g. via tt-ansible) and `feature.node.kubernetes.io/pci-1200_1e52.present=true` (applied automatically by NFD). Setting `agent.affinity` replaces that whole expression. Run `tt-fabric-manager-agent --help` for the flags you can pass through `agent.extraArgs`; the [configuration reference](config-reference.md) covers the config-file keys instead. ## OpenTelemetry | Key | Default | Description | |-----|---------|-------------| | `otel.enabled` | `false` | Emit OTEL logs, traces, and metrics from the controller | | `otel.serviceName` | `tt-fabric-manager-controller` | Reported service name | | `otel.deploymentEnvironment` | `""` | Reported deployment environment | | `otel.metricExportIntervalSec` | `10` | Metric export interval | Telemetry currently goes to stdout in OStream/OTLP-style format, so pair it with a sidecar Collector or stdout scraping until OTLP/gRPC export lands. ## Ingress Two independent Ingress objects, because the gRPC backend needs HTTP/2 while the UI is plain HTTP. Both share the ingress controller's external ports (80/443) and are routed by `Host` header, so they need distinct hostnames. | Key | Default | Description | |-----|---------|-------------| | `ingress.enabled` | `false` | Ingress for the external gRPC endpoint | | `ingress.host` | `""` | Hostname for the gRPC endpoint | | `ingress.className` | `""` | Ingress class | | `ingress.annotations` | `{}` | Ingress annotations | | `ingress.tls` | `[]` | TLS configuration | | `ingress.forwardingTimeouts.dialTimeout` | `30s` | Time to establish a connection to the controller | | `ingress.forwardingTimeouts.responseHeaderTimeout` | `60s` | Time the proxy waits for the controller to start responding | | `ingress.forwardingTimeouts.idleConnTimeout` | `90s` | How long an idle backend connection is kept | | `ingress.ui.enabled` | `false` | Ingress for the web UI. Requires `controller.web.enabled` | | `ingress.ui.host` | `""` | Hostname for the web UI | | `ingress.ui.className` | `""` | Ingress class for the UI | | `ingress.ui.annotations` | `{}` | Ingress annotations for the UI | | `ingress.ui.tls` | `[]` | TLS configuration for the UI | Enabling `ingress` also creates a Traefik `ServersTransport` carrying those forwarding timeouts, which the gRPC Ingress references by annotation. The UI Ingress does not use it, so the web UI keeps the ingress controller's own defaults. The timeouts used to be 900s, because a placement ran inside a single blocking RPC and every hop of the stack therefore had to outlive a whole solve. Placement is now a [long-running operation](api-overview.md#placement-operations) that the client starts and polls, so no request stays open for longer than an ordinary RPC and these are sized accordingly. Raise `responseHeaderTimeout` if callers still use the blocking `GetValidPlacementsMGD` — which is what `tt-fabric-manager-cli --sync` sends — since that holds one request open for as long as the solve takes. ## Example ```yaml controller: factorySystemDescriptorSearchPath: /scaleout_configs/mysite/factory_system_descriptor.textproto web: enabled: true ingress: enabled: true host: ttfm.example.com ui: enabled: true host: ttfm-ui.example.com ``` Site-specific values files live alongside the chart, for example `values.exabox.yaml`.