# tt_process

[<< Home](../index.md) | [<< Metrics](../metrics.md)

## Name

### Prometheus Metric Name

```
tt_process
```

### Metric Path (tt-telemetry)

Schema:

```
{hostname}/processes/{pid}/tt_process
```

Example path:

```
bh-glx-c09u02/processes/12345/tt_process
```

## Description

A host process that currently holds an open handle to a Tenstorrent kernel-mode driver (KMD) device. Where [tt_process_count](tt_process_count.md) reports *how many* processes are using the devices, this reports *which* ones: there is one `tt_process` metric for every such process, disambiguated by the PID embedded in its path.

The process is reported twice over, because the two renderings answer different questions:

- The **value** (the `friendly_name` label) is a short, human-recognizable name: the script, module, test file or binary that is actually running. This is what a process list is read for, and it is what grouping or displaying this metric in Prometheus should use.
- The **`command_line` label** is the whole invocation, arguments included. It is the only thing that distinguishes two workloads run through the same interpreter and the same entry point, but it is the answer to a follow-up question rather than the first thing anyone wants to see.

Both are captured when the metric is created, from a single read of procfs, and neither changes afterwards — the friendly name is derived from the very command line in the label, so the two can never describe different things. This costs nothing in practice: a process's argument vector is fixed at `exec()`, outside of the rare program that rewrites its own `argv`, and the label half could not change in any case, since labels are announced with a metric and are immutable from then on. The value is still re-published every cycle, unchanged but freshly timestamped, so a live process never reads as stale.

These metrics are created while the collector is running, as PIDs appear, and each removes itself once its PID is gone — so the set of `tt_process` metrics present at any moment is the set of processes using the devices at that moment. A process that starts and exits between two collection cycles is never observed at all.

The set of PIDs comes from the same per-cycle scan that produces `tt_process_count`: each per-device directory under `/proc/driver/tenstorrent/` is enumerated and its `pids` file read, and the union is taken so that a process attached to several devices is reported once. The collector's own PID is excluded, since it opens the devices it monitors. A metric is created only for those entries that name a process the collector can actually resolve, which is why `tt_process_count` can report more processes than there are `tt_process` metrics.

Both are read from procfs: the full contents of `/proc/<pid>/cmdline`, which is the argument vector the process was launched with — executable path and all arguments — with the kernel's NUL separators rendered as spaces. That rendering is not reversible — an argument containing a space cannot be told from two arguments — which is the same limitation `ps` has.

Processes whose command line is empty, such as kernel threads and zombies, fall back to `/proc/<pid>/comm` (truncated to 15 characters by the kernel). When neither can be read — an unprivileged collector, a `/proc` mounted with `hidepid`, or a process that exits mid-scan — both the value and `command_line` degrade to the decimal PID rather than going empty. Command lines are restricted to printable ASCII and truncated to 256 characters, because a process chooses its own command line and both labels end up in Prometheus; the friendly name is a substring of the sanitized command line, so it inherits those bounds.

## How the Friendly Name Is Derived

The friendly name is drawn from the command line itself: with one exception (`python -c`, below) it is a single token lifted straight out of it, never anything invented. Reporting the executable alone would not do: every job run through the same interpreter reads as `python3`, and an absolute path into a virtualenv says nothing about what is running. The rules, in order:

1. **Launcher prefixes are peeled**: leading `VAR=value` assignments and `sudo`, `env`, `nohup`, `stdbuf`, `taskset`, `numactl`, `timeout`, `nice`, `ionice`, along with their own options and operands. `taskset -c 0-3 timeout 600 python3 run.py` is named `run.py`.
2. **A Python interpreter** (`python`, `python3`, `python3.12`, `pypy3`, …) is looked past to what it is running: `-c` gives `python -c`; `-m <module>` gives the module name, dotted name kept whole (`a.b.cli`); otherwise the first positional argument's basename, which is the script (`train.py`) or the console entry point (`tt-smi`) being run. An interpreter with nothing to run keeps its own name.
3. **pytest**, whether run as `-m pytest` or as a console script, is named by its first test target's file: `a/b/test_x.py::TestC::test_y[params]` gives `test_x.py`, and a whole-directory target `a/b/c/` gives `c`. Options and their values are skipped, including unquoted multi-word `-k`/`-m` expressions. An invocation with no target at all is named `pytest`.
4. **Anything else** is the basename of `argv[0]`: a compiled binary, a shebang script, or a `comm` value, which is not a command line at all and is returned as-is.

Nothing in this knows about any particular project's programs, modules or directories. The only names it recognizes are `python`, `pytest` and the standard shell launchers, so it behaves the same on any host.

A removal is only reported on the strength of a scan that read every device successfully. If the device directory cannot be enumerated, or one device's `pids` file cannot be read, the existing metrics hold their values rather than being retired on incomplete evidence.

This is a host-level metric (not scoped to a tray or chip path). It is driver-independent: it reads procfs directly and does not require UMD/driver initialization, so it persists across warm resets. It is only created when device telemetry collection is enabled.

## Requires the Host PID Namespace

**A collector running in its own PID namespace — the default for a container — cannot identify processes outside that namespace, which on a normal deployment is every process on the node. No `tt_process` metrics will exist at all.** [tt_process_count](tt_process_count.md) still reports that processes are present; only naming them fails.

The exception is a process that shares the collector's namespace — one it forked, or one started with `kubectl exec` into its own pod. Those render real PIDs and do get a metric. Processes in other pods do not: sibling namespaces are not ancestors of one another.

`/proc/driver/tenstorrent/<n>/pids` has no stored contents. The KMD generates it on every read, translating each PID through `pid_vnr()` into the PID namespace of *the process doing the reading*. The same file therefore yields different bytes to different readers. A process that has no identity in the reader's namespace is rendered as `0`.

For a containerized collector without the host PID namespace, that means every process on the node reads back as `0`, and the collector reads its own PID as `1` — it is the init process of its own namespace:

```
# on the node
$ cat /proc/driver/tenstorrent/0/pids
3894127
3894127
2041156
2041156

# the same file, read from inside the collector's namespaces
$ kubectl exec -n <namespace> <collector-pod> -- cat /proc/driver/tenstorrent/0/pids
1
1
0
0
```

The telltale in the collector's own logs is the PID it excludes as its own: on a busy node nothing is PID `1` except an init process, so a collector reporting itself as `1` is necessarily in a private namespace.

This cannot be worked around in the collector. A `0` carries no identity, so there is nothing to look up in `/proc` and nothing to build a metric path from, and two such entries cannot be told apart. `tt_process_count` still counts them — collapsed into a single entry, so it reports a floor rather than an exact number — but a named metric cannot be manufactured from them. Sharing the node's PID namespace is the only remedy:

- **Kubernetes / Helm:** `hostPID: true` on the pod spec. The chart in this repository exposes it as `daemonset.hostPID`, which defaults to `true`. This is not a meaningful increase in privilege for this workload, since the container already runs privileged for device access.
- **Plain Docker:** `--pid=host`.
- **Not in a container:** nothing to do. A collector running directly on the host is already in that namespace, so every PID resolves and this metric works as described above.

Sharing the namespace does two further things: it makes `/proc/<pid>` readable for those processes, which is what lets this metric report a command line rather than falling back to the PID, and it makes [tt_process_count](tt_process_count.md) exact rather than the lower bound it reports when processes cannot be told apart.

## Values

**Type:** String

**Units:** None

**Allowable values:**
A single token drawn from the process's command line, such as `"train.py"`, `"test_matmul.py"`, `"a.b.cli"` or `"tt-smi"`; a `comm` value of up to 15 characters, such as `"kworker/0:1"`, for a process with no command line; the process's PID rendered in decimal (for example `"12345"`) when nothing could be read from procfs; or `"<unknown>"` for a command line that is empty after sanitizing. The value is exposed via the `friendly_name` Prometheus label.

## Related Metrics

|Metric|Description|
|---|---|
|[tt_process_count](tt_process_count.md)|Number of distinct host processes holding a handle to a Tenstorrent device. Equal to the number of `tt_process` metrics present.|

## Prometheus Labels

|Label Name|Value|
|---|---|
|hostname|The host from which the metric was collected.|
|pid|The process ID, as rendered in the reading process's PID namespace.|
|friendly_name|The short name of what is running (the metric's value).|
|command_line|The process's full command line, as captured when the metric was created.|