tt_process_count

<< Home | << Metrics

Name

Prometheus Metric Name

tt_process_count

Metric Path (tt-telemetry)

Schema:

{hostname}/processes/tt_process_count

Example path:

bh-glx-c09u02/processes/tt_process_count

Description

The number of distinct host processes other than the telemetry collector itself that currently hold an open handle to a Tenstorrent kernel-mode driver (KMD) device. It is computed by scanning each per-device directory under /proc/driver/tenstorrent/, reading the pids file in each, and counting the union of all PIDs. A process attached to more than one device is counted only once (duplicate PIDs are removed).

The collector opens the devices it monitors, so its own PID is removed from the union before counting. A value of 0 therefore means no process besides the telemetry collector is using the devices, rather than that the devices are entirely unused.

The individual processes behind this count are reported by command line by tt_process, one metric per PID. Both come from the same scan, performed once per collection cycle. This count can nonetheless be higher than the number of tt_process metrics: it counts everything the driver reports, including entries the collector cannot resolve to a specific process (see below), whereas a tt_process metric can only be created for one it can.

This is a host-level metric (not scoped to a tray or chip path). It is driver-independent: it reads procfs directly and does not require UMD/driver initialization, so it persists across warm resets. It is only created when device telemetry collection is enabled, and its value is refreshed on every collection cycle. If no Tenstorrent device directories are found under /proc/driver/tenstorrent/, a warning is logged and the value is reported as 0. If the scan itself fails, the last known count is held rather than being reported as 0.

Containers and PID Namespaces

This count remains meaningful for a collector running in its own PID namespace — the default for a container — but it becomes a lower bound rather than an exact figure.

The kernel renders the pids files into the PID namespace of whichever process reads them, so a process the collector cannot see is reported as 0. Those entries are counted: the driver is saying a process holds the device open, and reporting 0 because the collector cannot put a name to it would assert the devices are idle when they are not. But every such entry is the same 0, so they collapse into a single one — three invisible processes and one invisible process both add exactly 1 here.

In practice this makes the value either 0 or 1 on a containerized collector, since nothing else on the node shares its PID namespace and every device user therefore collapses into that one entry. It still distinguishes “something is using the devices” from “nothing is”, but it carries no magnitude: one workload and twenty both report 1. Treat it as a boolean under these conditions, and do not alert on its value.

Processes that do share the collector’s PID namespace are the exception — one the collector forked, or one started with kubectl exec into its own pod — since those render real PIDs. They are counted individually and get a tt_process metric each. Processes in other pods do not: those namespaces are siblings, not ancestors.

The telltale is that no tt_process metric accompanies the count, since an unresolvable entry cannot be named or given a metric path. To get an exact count and per-process names, give the collector the host PID namespace: hostPID: true (Helm: daemonset.hostPID, default true) or --pid=host under plain Docker. A collector running directly on the host, rather than in a container, is already in that namespace and needs no such setting. See tt_process for the full explanation and how to confirm it.

Values

Type: Unsigned Integer

Units: None

Allowable values: Non-negative integer representing the number of distinct processes, excluding the telemetry collector, holding a handle to any Tenstorrent device (for example 0 when only the collector has the devices open, or 3 when three other distinct processes hold handles).

Related Metrics

Metric	Description
tt_process	Command line of an individual host process holding a handle to a Tenstorrent device. One per PID counted here that the collector can resolve to a specific process.
tt_chip_count	Number of Tenstorrent chips discovered on the host via PCI device enumeration.
tt_driver_initialized	Whether the device driver is initialized. tt_process_count is assessed independently of driver initialization.

Prometheus Labels

Label Name	Value
hostname	The host from which the metric was collected.