tt_process

<< Home | << Metrics

Name

Prometheus Metric Name

tt_process

Metric Path (tt-telemetry)

Schema:

{hostname}/processes/{pid}/tt_process

Example path:

bh-glx-c09u02/processes/12345/tt_process

Description

The command line of a host process that currently holds an open handle to a Tenstorrent kernel-mode driver (KMD) device. Where tt_process_count reports how many processes are using the devices, this reports which ones: there is one tt_process metric for every such process, disambiguated by the PID embedded in its path.

These metrics are created while the collector is running, as PIDs appear, and each removes itself once its PID is gone — so the set of tt_process metrics present at any moment is the set of processes using the devices at that moment. A process that starts and exits between two collection cycles is never observed at all.

The set of PIDs comes from the same per-cycle scan that produces tt_process_count: each per-device directory under /proc/driver/tenstorrent/ is enumerated and its pids file read, and the union is taken so that a process attached to several devices is reported once. The collector’s own PID is excluded, since it opens the devices it monitors. A metric is created only for those entries that name a process the collector can actually resolve, which is why tt_process_count can report more processes than there are tt_process metrics.

The value is read from procfs: the full contents of /proc/<pid>/cmdline, which is the argument vector the process was launched with — executable path and all arguments — with the kernel’s NUL separators rendered as spaces. The whole command line is reported rather than just the executable, because the executable alone often does not identify the workload: every job run through the same interpreter would read as python3. The rendering is not reversible — an argument containing a space cannot be told from two arguments — which is the same limitation ps has.

Processes whose command line is empty, such as kernel threads and zombies, fall back to /proc/<pid>/comm (truncated to 15 characters by the kernel). When neither can be read — an unprivileged collector, a /proc mounted with hidepid, or a process that exits mid-scan — the value degrades to the decimal PID rather than going empty. Values are restricted to printable ASCII and truncated to 256 characters, because a process chooses its own command line and the value ends up in a Prometheus label.

A removal is only reported on the strength of a scan that read every device successfully. If the device directory cannot be enumerated, or one device’s pids file cannot be read, the existing metrics hold their values rather than being retired on incomplete evidence.

This is a host-level metric (not scoped to a tray or chip path). It is driver-independent: it reads procfs directly and does not require UMD/driver initialization, so it persists across warm resets. It is only created when device telemetry collection is enabled.

Requires the Host PID Namespace

A collector running in its own PID namespace — the default for a container — cannot identify processes outside that namespace, which on a normal deployment is every process on the node. No tt_process metrics will exist at all. tt_process_count still reports that processes are present; only naming them fails.

The exception is a process that shares the collector’s namespace — one it forked, or one started with kubectl exec into its own pod. Those render real PIDs and do get a metric. Processes in other pods do not: sibling namespaces are not ancestors of one another.

/proc/driver/tenstorrent/<n>/pids has no stored contents. The KMD generates it on every read, translating each PID through pid_vnr() into the PID namespace of the process doing the reading. The same file therefore yields different bytes to different readers. A process that has no identity in the reader’s namespace is rendered as 0.

For a containerized collector without the host PID namespace, that means every process on the node reads back as 0, and the collector reads its own PID as 1 — it is the init process of its own namespace:

# on the node
$ cat /proc/driver/tenstorrent/0/pids
3894127
3894127
2041156
2041156

# the same file, read from inside the collector's namespaces
$ kubectl exec -n <namespace> <collector-pod> -- cat /proc/driver/tenstorrent/0/pids
1
1
0
0

The telltale in the collector’s own logs is the PID it excludes as its own: on a busy node nothing is PID 1 except an init process, so a collector reporting itself as 1 is necessarily in a private namespace.

This cannot be worked around in the collector. A 0 carries no identity, so there is nothing to look up in /proc and nothing to build a metric path from, and two such entries cannot be told apart. tt_process_count still counts them — collapsed into a single entry, so it reports a floor rather than an exact number — but a named metric cannot be manufactured from them. Sharing the node’s PID namespace is the only remedy:

• Kubernetes / Helm: hostPID: true on the pod spec. The chart in this repository exposes it as daemonset.hostPID, which defaults to true. This is not a meaningful increase in privilege for this workload, since the container already runs privileged for device access.

• Plain Docker: --pid=host.

• Not in a container: nothing to do. A collector running directly on the host is already in that namespace, so every PID resolves and this metric works as described above.

Sharing the namespace does two further things: it makes /proc/<pid> readable for those processes, which is what lets this metric report a command line rather than falling back to the PID, and it makes tt_process_count exact rather than the lower bound it reports when processes cannot be told apart.

Values

Type: String

Units: None

Allowable values: A command line of up to 256 printable-ASCII characters, such as "/usr/bin/python3 train.py --batch-size 8" or "tt-smi"; a comm value of up to 15 characters, such as "kworker/0:1", for a process with no command line; or the process’s PID rendered in decimal (for example "12345") when nothing could be read from procfs. The value is exposed via the process_name Prometheus label.

Related Metrics

Metric	Description
tt_process_count	Number of distinct host processes holding a handle to a Tenstorrent device. Equal to the number of tt_process metrics present.

Prometheus Labels

Label Name	Value
hostname	The host from which the metric was collected.
pid	The process ID, as rendered in the reading process’s PID namespace.
process_name	The process’s command line (the metric’s value).