tt_process

<< Home | << Metrics

Name

Prometheus Metric Name

tt_process

Metric Path (tt-telemetry)

Schema:

{hostname}/processes/{pid}/tt_process

Example path:

bh-glx-c09u02/processes/12345/tt_process

Description

A host process that currently holds an open handle to a Tenstorrent kernel-mode driver (KMD) device. Where tt_process_count reports how many processes are using the devices, this reports which ones: there is one tt_process metric for every such process, disambiguated by the PID embedded in its path.

The process is reported twice over, because the two renderings answer different questions:

• The value (the friendly_name label) is a short, human-recognizable name: the script, module, test file or binary that is actually running. This is what a process list is read for, and it is what grouping or displaying this metric in Prometheus should use.

• The command_line label is the whole invocation, arguments included. It is the only thing that distinguishes two workloads run through the same interpreter and the same entry point, but it is the answer to a follow-up question rather than the first thing anyone wants to see.

Both are captured when the metric is created, from a single read of procfs, and neither changes afterwards — the friendly name is derived from the very command line in the label, so the two can never describe different things. This costs nothing in practice: a process’s argument vector is fixed at exec(), outside of the rare program that rewrites its own argv, and the label half could not change in any case, since labels are announced with a metric and are immutable from then on. The value is still re-published every cycle, unchanged but freshly timestamped, so a live process never reads as stale.

These metrics are created while the collector is running, as PIDs appear, and each removes itself once its PID is gone — so the set of tt_process metrics present at any moment is the set of processes using the devices at that moment. A process that starts and exits between two collection cycles is never observed at all.

The set of PIDs comes from the same per-cycle scan that produces tt_process_count: each per-device directory under /proc/driver/tenstorrent/ is enumerated and its pids file read, and the union is taken so that a process attached to several devices is reported once. The collector’s own PID is excluded, since it opens the devices it monitors. A metric is created only for those entries that name a process the collector can actually resolve, which is why tt_process_count can report more processes than there are tt_process metrics.

Both are read from procfs: the full contents of /proc/<pid>/cmdline, which is the argument vector the process was launched with — executable path and all arguments — with the kernel’s NUL separators rendered as spaces. That rendering is not reversible — an argument containing a space cannot be told from two arguments — which is the same limitation ps has.

Processes whose command line is empty, such as kernel threads and zombies, fall back to /proc/<pid>/comm (truncated to 15 characters by the kernel). When neither can be read — an unprivileged collector, a /proc mounted with hidepid, or a process that exits mid-scan — both the value and command_line degrade to the decimal PID rather than going empty. Command lines are restricted to printable ASCII and truncated to 256 characters, because a process chooses its own command line and both labels end up in Prometheus; the friendly name is a substring of the sanitized command line, so it inherits those bounds.

How the Friendly Name Is Derived

The friendly name is drawn from the command line itself: with one exception (python -c, below) it is a single token lifted straight out of it, never anything invented. Reporting the executable alone would not do: every job run through the same interpreter reads as python3, and an absolute path into a virtualenv says nothing about what is running. The rules, in order:

1. Launcher prefixes are peeled: leading VAR=value assignments and sudo, env, nohup, stdbuf, taskset, numactl, timeout, nice, ionice, along with their own options and operands. taskset -c 0-3 timeout 600 python3 run.py is named run.py.

2. A Python interpreter (python, python3, python3.12, pypy3, …) is looked past to what it is running: -c gives python -c; -m <module> gives the module name, dotted name kept whole (a.b.cli); otherwise the first positional argument’s basename, which is the script (train.py) or the console entry point (tt-smi) being run. An interpreter with nothing to run keeps its own name.

3. pytest, whether run as -m pytest or as a console script, is named by its first test target’s file: a/b/test_x.py::TestC::test_y[params] gives test_x.py, and a whole-directory target a/b/c/ gives c. Options and their values are skipped, including unquoted multi-word -k/-m expressions. An invocation with no target at all is named pytest.

4. Anything else is the basename of argv[0]: a compiled binary, a shebang script, or a comm value, which is not a command line at all and is returned as-is.

Nothing in this knows about any particular project’s programs, modules or directories. The only names it recognizes are python, pytest and the standard shell launchers, so it behaves the same on any host.

A removal is only reported on the strength of a scan that read every device successfully. If the device directory cannot be enumerated, or one device’s pids file cannot be read, the existing metrics hold their values rather than being retired on incomplete evidence.

This is a host-level metric (not scoped to a tray or chip path). It is driver-independent: it reads procfs directly and does not require UMD/driver initialization, so it persists across warm resets. It is only created when device telemetry collection is enabled.

Requires the Host PID Namespace

A collector running in its own PID namespace — the default for a container — cannot identify processes outside that namespace, which on a normal deployment is every process on the node. No tt_process metrics will exist at all. tt_process_count still reports that processes are present; only naming them fails.

The exception is a process that shares the collector’s namespace — one it forked, or one started with kubectl exec into its own pod. Those render real PIDs and do get a metric. Processes in other pods do not: sibling namespaces are not ancestors of one another.

/proc/driver/tenstorrent/<n>/pids has no stored contents. The KMD generates it on every read, translating each PID through pid_vnr() into the PID namespace of the process doing the reading. The same file therefore yields different bytes to different readers. A process that has no identity in the reader’s namespace is rendered as 0.

For a containerized collector without the host PID namespace, that means every process on the node reads back as 0, and the collector reads its own PID as 1 — it is the init process of its own namespace:

# on the node
$ cat /proc/driver/tenstorrent/0/pids
3894127
3894127
2041156
2041156

# the same file, read from inside the collector's namespaces
$ kubectl exec -n <namespace> <collector-pod> -- cat /proc/driver/tenstorrent/0/pids
1
1
0
0

The telltale in the collector’s own logs is the PID it excludes as its own: on a busy node nothing is PID 1 except an init process, so a collector reporting itself as 1 is necessarily in a private namespace.

This cannot be worked around in the collector. A 0 carries no identity, so there is nothing to look up in /proc and nothing to build a metric path from, and two such entries cannot be told apart. tt_process_count still counts them — collapsed into a single entry, so it reports a floor rather than an exact number — but a named metric cannot be manufactured from them. Sharing the node’s PID namespace is the only remedy:

• Kubernetes / Helm: hostPID: true on the pod spec. The chart in this repository exposes it as daemonset.hostPID, which defaults to true. This is not a meaningful increase in privilege for this workload, since the container already runs privileged for device access.

• Plain Docker: --pid=host.

• Not in a container: nothing to do. A collector running directly on the host is already in that namespace, so every PID resolves and this metric works as described above.

Sharing the namespace does two further things: it makes /proc/<pid> readable for those processes, which is what lets this metric report a command line rather than falling back to the PID, and it makes tt_process_count exact rather than the lower bound it reports when processes cannot be told apart.

Values

Type: String

Units: None

Allowable values: A single token drawn from the process’s command line, such as "train.py", "test_matmul.py", "a.b.cli" or "tt-smi"; a comm value of up to 15 characters, such as "kworker/0:1", for a process with no command line; the process’s PID rendered in decimal (for example "12345") when nothing could be read from procfs; or "<unknown>" for a command line that is empty after sanitizing. The value is exposed via the friendly_name Prometheus label.

Related Metrics

Metric	Description
tt_process_count	Number of distinct host processes holding a handle to a Tenstorrent device. Equal to the number of tt_process metrics present.

Prometheus Labels

Label Name	Value
hostname	The host from which the metric was collected.
pid	The process ID, as rendered in the reading process’s PID namespace.
friendly_name	The short name of what is running (the metric’s value).
command_line	The process’s full command line, as captured when the metric was created.