tt_pci_device_count

<< Home | << Metrics

Name

Prometheus Metric Name

tt_pci_device_count

Metric Path (tt-telemetry)

Schema:

{hostname}/tt_pci_device_count

Example path:

bh-glx-c09u02/tt_pci_device_count

Description

The number of Tenstorrent devices present on the host’s PCI bus. It is counted by walking /sys/bus/pci/devices/ and reading the vendor file in each device directory, counting those whose value matches the Tenstorrent PCI vendor ID 0x1e52. This is the equivalent of lspci | grep 1e52 | wc -l, computed by reading sysfs directly rather than by spawning a subprocess.

The Linux PCI subsystem enumerates the bus at boot and publishes a directory for every function it finds, whether or not a driver binds to it. This metric therefore reports the hardware population as the kernel sees it, before and independently of any attempt by the Tenstorrent kernel-mode driver (KMD) to claim the devices. Devices are matched on vendor ID only, so the count covers all Tenstorrent ASIC generations (Wormhole, Blackhole, and future device IDs) without needing to be updated.

This is the key distinction from tt_chip_count, which counts the character devices the KMD created (via PCIDevice::enumerate_devices()) and so only sees devices the driver successfully bound to. Comparing the two isolates where a fault lies:

Observation	Interpretation
tt_pci_device_count == tt_chip_count	Healthy: every device on the bus was claimed by the driver.
tt_pci_device_count > tt_chip_count	The card is physically present and its PCIe link is trained, but the driver did not claim it. Suspect the KMD (not loaded, version mismatch, bind failure) rather than the hardware.
tt_pci_device_count below the expected number for the host	The card is not visible to the kernel at all. Suspect hardware: card not seated, PCIe link down, or the device fell off the bus.

This is a host-level metric (not scoped to a tray or chip path). It is created when device telemetry collection is enabled, and its value is refreshed on every collection cycle. It is driver-independent: it reads sysfs directly, requires neither UMD initialization nor an open device, and so persists across warm resets and continues to report a meaningful value when driver initialization fails outright. PCI devices are not namespaced and container runtimes mount /sys by default, so the count is correct when tt-telemetry runs inside a container. If /sys/bus/pci/devices/ cannot be enumerated, a warning is logged and the value is reported as 0.

Values

Type: Unsigned Integer

Units: None

Allowable values: Non-negative integer representing the number of Tenstorrent PCI functions the kernel has enumerated (for example 4 on a four-card host, or 0 on a host with no Tenstorrent hardware or with sysfs unavailable).

Related Metrics

Metric	Description
tt_chip_count	Number of chips the driver actually claimed; compare against tt_pci_device_count to distinguish a driver bind failure from missing hardware.
tt_expected_chip_count	Expected chip count on Galaxy hosts (32); the reference value both counts should match.
tt_driver_initialized	Whether the device driver is initialized. tt_pci_device_count is assessed independently of driver initialization.
tt_pcie_link_alive	Per-chip PCIe liveness, as probed through the driver by host-issued reads.

Prometheus Labels

Label Name	Value
hostname	The host from which the metric was collected.
hall	The datacenter hall where the host is located. Sourced from the Factory System Descriptor (FSD).
aisle	The datacenter aisle where the host is located. Sourced from the Factory System Descriptor (FSD).
rack	The rack number where the host is located. Sourced from the Factory System Descriptor (FSD).
shelf_u	The shelf U position in the rack where the host is located. Sourced from the Factory System Descriptor (FSD).