# tt_process

[<< Home](../index.md) | [<< Metrics](../metrics.md)

## Name

### Prometheus Metric Name

```
tt_process
```

### Metric Path (tt-telemetry)

Schema:

```
{hostname}/processes/{pid}/tt_process
```

Example path:

```
bh-glx-c09u02/processes/12345/tt_process
```

## Description

The command line of a host process that currently holds an open handle to a Tenstorrent kernel-mode driver (KMD) device. Where [tt_process_count](tt_process_count.md) reports *how many* processes are using the devices, this reports *which* ones: there is one `tt_process` metric for every such process, disambiguated by the PID embedded in its path.

These metrics are created while the collector is running, as PIDs appear, and each removes itself once its PID is gone — so the set of `tt_process` metrics present at any moment is the set of processes using the devices at that moment. A process that starts and exits between two collection cycles is never observed at all.

The set of PIDs comes from the same per-cycle scan that produces `tt_process_count`: each per-device directory under `/proc/driver/tenstorrent/` is enumerated and its `pids` file read, and the union is taken so that a process attached to several devices is reported once. The collector's own PID is excluded, since it opens the devices it monitors. A metric is created only for those entries that name a process the collector can actually resolve, which is why `tt_process_count` can report more processes than there are `tt_process` metrics.

The value is read from procfs: the full contents of `/proc/<pid>/cmdline`, which is the argument vector the process was launched with — executable path and all arguments — with the kernel's NUL separators rendered as spaces. The whole command line is reported rather than just the executable, because the executable alone often does not identify the workload: every job run through the same interpreter would read as `python3`. The rendering is not reversible — an argument containing a space cannot be told from two arguments — which is the same limitation `ps` has.

Processes whose command line is empty, such as kernel threads and zombies, fall back to `/proc/<pid>/comm` (truncated to 15 characters by the kernel). When neither can be read — an unprivileged collector, a `/proc` mounted with `hidepid`, or a process that exits mid-scan — the value degrades to the decimal PID rather than going empty. Values are restricted to printable ASCII and truncated to 256 characters, because a process chooses its own command line and the value ends up in a Prometheus label.

A removal is only reported on the strength of a scan that read every device successfully. If the device directory cannot be enumerated, or one device's `pids` file cannot be read, the existing metrics hold their values rather than being retired on incomplete evidence.

This is a host-level metric (not scoped to a tray or chip path). It is driver-independent: it reads procfs directly and does not require UMD/driver initialization, so it persists across warm resets. It is only created when device telemetry collection is enabled.

## Requires the Host PID Namespace

**A collector running in its own PID namespace — the default for a container — cannot identify processes outside that namespace, which on a normal deployment is every process on the node. No `tt_process` metrics will exist at all.** [tt_process_count](tt_process_count.md) still reports that processes are present; only naming them fails.

The exception is a process that shares the collector's namespace — one it forked, or one started with `kubectl exec` into its own pod. Those render real PIDs and do get a metric. Processes in other pods do not: sibling namespaces are not ancestors of one another.

`/proc/driver/tenstorrent/<n>/pids` has no stored contents. The KMD generates it on every read, translating each PID through `pid_vnr()` into the PID namespace of *the process doing the reading*. The same file therefore yields different bytes to different readers. A process that has no identity in the reader's namespace is rendered as `0`.

For a containerized collector without the host PID namespace, that means every process on the node reads back as `0`, and the collector reads its own PID as `1` — it is the init process of its own namespace:

```
# on the node
$ cat /proc/driver/tenstorrent/0/pids
3894127
3894127
2041156
2041156

# the same file, read from inside the collector's namespaces
$ kubectl exec -n <namespace> <collector-pod> -- cat /proc/driver/tenstorrent/0/pids
1
1
0
0
```

The telltale in the collector's own logs is the PID it excludes as its own: on a busy node nothing is PID `1` except an init process, so a collector reporting itself as `1` is necessarily in a private namespace.

This cannot be worked around in the collector. A `0` carries no identity, so there is nothing to look up in `/proc` and nothing to build a metric path from, and two such entries cannot be told apart. `tt_process_count` still counts them — collapsed into a single entry, so it reports a floor rather than an exact number — but a named metric cannot be manufactured from them. Sharing the node's PID namespace is the only remedy:

- **Kubernetes / Helm:** `hostPID: true` on the pod spec. The chart in this repository exposes it as `daemonset.hostPID`, which defaults to `true`. This is not a meaningful increase in privilege for this workload, since the container already runs privileged for device access.
- **Plain Docker:** `--pid=host`.
- **Not in a container:** nothing to do. A collector running directly on the host is already in that namespace, so every PID resolves and this metric works as described above.

Sharing the namespace does two further things: it makes `/proc/<pid>` readable for those processes, which is what lets this metric report a command line rather than falling back to the PID, and it makes [tt_process_count](tt_process_count.md) exact rather than the lower bound it reports when processes cannot be told apart.

## Values

**Type:** String

**Units:** None

**Allowable values:**
A command line of up to 256 printable-ASCII characters, such as `"/usr/bin/python3 train.py --batch-size 8"` or `"tt-smi"`; a `comm` value of up to 15 characters, such as `"kworker/0:1"`, for a process with no command line; or the process's PID rendered in decimal (for example `"12345"`) when nothing could be read from procfs. The value is exposed via the `process_name` Prometheus label.

## Related Metrics

|Metric|Description|
|---|---|
|[tt_process_count](tt_process_count.md)|Number of distinct host processes holding a handle to a Tenstorrent device. Equal to the number of `tt_process` metrics present.|

## Prometheus Labels

|Label Name|Value|
|---|---|
|hostname|The host from which the metric was collected.|
|pid|The process ID, as rendered in the reading process's PID namespace.|
|process_name|The process's command line (the metric's value).|