What to Know About Your Workstation

Your workstation is a Tenstorrent Quietbox 2: four AI accelerators inside, an operating system you may not have used before, and the software stack already configured and waiting. The machine is ready to go — what’s left is knowing what you’ve got.

This guide doesn’t assume you know Linux, or Python, or what a PCIe slot is. It assumes you’re curious, and that curiosity is enough.

What’s Inside

The Tenstorrent Quietbox 2 (QB2) is a workstation with two Blackhole p300c cards — four Blackhole chips in total — on PCIe. Each p300c is a dual-chip card, and each chip is independent — four separate devices from the software’s point of view, connected to a standard CPU running Ubuntu 24.04 LTS.

The chips don’t share memory. When you open device 0, you’re talking to one Blackhole chip. To use all four together, you use ttnn.CreateDevices({0, 1, 2, 3}) — not four separate open_device() calls.

What Ships Pre-Installed

Tenstorrent ships the QB2 ready to serve models. You don’t install drivers. You don’t compile anything. The full stack is already there:

• Kernel driver — loaded automatically at boot, makes the chips visible to software
• tt — the Tenstorrent CLI. One command for updating the stack, checking hardware, serving models and launching apps (~/.local/bin/tt)
• tt-smi — hardware monitoring tool, lives at /usr/bin/tt-smi
• TTNN / TT-Metalium — as a container, started by the tt-metalium wrapper in ~/.local/bin/
• Hardware tooling venv — ~/.tenstorrent-venv/, holding tt-smi and tt-flash
• tt-inference-server — what tt serve runs to serve a model with vLLM in a container
• TT-Forge/XLA — container wrapper at ~/.local/bin/tt-forge, if it was installed (it’s opt-in)
• tt-studio — the no-code web UI for serving models (tt serve <model> --studio)
• A ready-to-run model — Qwen3-32B, weights pre-cached on disk, served with tt serve Qwen3-32B and no download (your fastest path to a first token)
• Firmware — already flashed to all four chips

What’s intentionally absent: ~/tt-metal — not just the source code, the whole directory. TT-Metalium reaches you as a container image instead, and tt-metalium is how you get inside it. You can build models, run inference, and work with the full API stack without ever having a checkout. Building from source is a later chapter — a much later chapter.

Physical Tour

The QB2 looks like a standard tower workstation. On the inside:

• CPU and motherboard running Ubuntu 24.04 LTS
• Two Blackhole p300c cards (four Blackhole chips total)
• RAM sized for production inference workloads
• Storage for model weights — but watch it carefully (more on that in Chapter 2)

The chips run warm under load. Fans will get louder when you run inference. This is correct. The cooling is designed for sustained operation at full chip temperature.

One Blackhole chip. You have four, on two p300c cards.

Next: First Boot →

First Boot

Power on. Ubuntu loads. You log in. Now what?

Everything from here happens in a terminal. That’s the command line — a text window where you type instructions and the machine responds. On a QB2, the terminal is your instrument panel. Learning its three or four most-used commands will get you surprisingly far.

Finding a Terminal

If you’re looking at the GNOME desktop:

• Press Ctrl+Alt+T — opens a terminal on most Ubuntu setups
• Or press the Super key (Windows key), type terminal, press Enter
• Or right-click the desktop and choose “Open Terminal”

Once a terminal window is open, you’re in the right place. It shows a prompt ending in $ — everything you type goes after that.

The Three Commands You Need Right Now

Check disk space first. Models are large. This is non-negotiable to understand before you do anything else:

df -h ~

This shows your home directory’s disk usage. The Size column is total, Avail is what’s free. You need room — at minimum 3 GB for a small model (Qwen3-0.6B), 20+ GB for anything like Llama-3.1-8B. If you’re under 5 GB free, stop here and figure out where the space went before continuing.

Check internet connectivity:

ping -c 3 google.com

If this fails, check your network cable or go to Settings → Network. Everything else in this guide requires internet access for model downloads.

Update the package list (do this once after first boot):

sudo apt update

sudo means “run as administrator.” Ubuntu will ask for your password. This doesn’t install or change anything — it just refreshes the list of what’s available. You’ll see a lot of text scroll by. That’s normal.

Ubuntu: What You Should Know

The QB2 runs Ubuntu 24.04 LTS. If this is your first time with it:

• Package manager is apt — install things with sudo apt install <name>
• Files are case-sensitive: Model.py and model.py are different files
• Your home directory is ~ — short for /home/yourusername
• sudo runs a command as administrator — use it only when a command tells you to

Your Login, Password, and SSH

Many QB2 units ship with a default login — username ttuser, password ttuser. If that’s how yours arrived, change the password the moment you’re in, before the machine is reachable on a shared network:

passwd

It asks for the current password (ttuser), then a new one twice.

Turn on SSH

Later in this guide — and on every other path — you reach the QB2 from your own laptop over SSH: forwarding a model server’s port back to your machine, copying files, running commands remotely. SSH isn’t always running on a fresh box, so turn it on once:

# Install and enable the SSH server
sudo apt install -y openssh-server
sudo systemctl enable --now ssh

# Confirm it's listening
systemctl status ssh

Then find the address other machines use to reach you:

hostname -I     # the QB2's IP address on your network
hostname        # its name — often <name>.local

From your laptop you can now run ssh ttuser@<that-ip>. This is what makes the remote-access steps in Serving Models on QB2 — and bringing tt-studio’s web UI to your own browser — work.

Python: A Field Guide to the Confusion

This is where new Linux users often hit a wall. Ubuntu ships with its own Python. The Tenstorrent software has its own Python environments. These are separate and don’t mix. Here’s the landscape:

What exists on your system

The important surprise is the third row. There is no TTNN environment on the host. You may find older notes pointing at ~/tt-metal/python_env/ — that path doesn’t exist on a QB2 and the installer never creates it. TT-Metalium arrives as a container image, and tt-metalium is the wrapper that puts you inside it with your home directory mounted. import ttnn works in there and nowhere else.

Why does this matter?

Ubuntu 24.04 enforces what’s called externally-managed Python — the system Python is protected. If you try to pip install something directly, Ubuntu will refuse with an error about breaking system packages. This is intentional. It protects you.

So when you need a Python package of your own, make it a new venv (python3 -m venv ~/.venvs/mine) or let uv tool / pipx make one for you. What you should not do is install into ~/.tenstorrent-venv to dodge the error. That venv exists to hold tt-smi and tt-flash, your QB2 activates it for you at login, and a dependency resolution gone wrong in there takes out the tools you’d use to work out what broke.

What which python3 tells you

Before running any Python code, check which Python is active:

which python3

If you see /usr/bin/python3 — you’re on the host, using Ubuntu’s Python. Tenstorrent imports will fail.

If you see /home/yourname/.tenstorrent-venv/bin/python3 — you’re in the tooling venv. tt-smi works; import ttnn still won’t.

If you see /opt/venv/bin/python3 — you’re inside the tt-metalium container, which is where TTNN lives. Go ahead.

pip, pyenv, uv — a brief map

You may encounter other Python tools in documentation or online:

• pip — Python package installer. Works inside a venv. Fine to use there.
• pyenv — manages multiple Python versions (3.10, 3.11, etc.). The QB2 doesn’t need it — the venvs handle version isolation.
• virtualenv / python -m venv — creates isolated environments. The Tenstorrent venvs were built this way.
• uv — a fast, modern alternative to pip and virtualenv. uv tool install <pkg> is the tidiest way to add a CLI (like hf) without touching a managed environment.

For this guide: ignore pyenv. Reach for uv or pipx when you want a CLI of your own, and use tt-metalium when you want TTNN.

Activating and deactivating

# Enter the TTNN environment — a container, not a venv
tt-metalium

# You're now in a shell inside the container, home directory mounted.
which python3          # → /opt/venv/bin/python3
python3 -c "import ttnn; print('TTNN ready')"

# Leave it the way you leave any shell
exit

There’s no (python_env) prefix to look for, because nothing is being activated — you’re in a different shell on a different filesystem. which python3 is the signal: /opt/venv/bin/python3 means you’re inside.

Next: Is This Thing On? →

Is This Thing On?

Before running a model, confirm the hardware is alive and the software can see it. One command, four chips, zero guessing.

Ask tt

tt device status

                 Tenstorrent devices (driver: TT-KMD 2.11.0)
┏━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ # ┃ Board ┃ Bus ID       ┃ Temp    ┃ Power  ┃ AIClk      ┃ Voltage ┃ DRAM ┃
┡━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ 0 │ p300c │ 0000:01:00.0 │ 55.0 °C │ 74.0 W │ 1337.0 MHz │ 0.81 V  │ True │
│ 1 │ p300c │ 0000:02:00.0 │ 59.9 °C │ 49.0 W │ 1343.0 MHz │ 0.82 V  │ True │
│ 2 │ p300c │ 0000:03:00.0 │ 45.9 °C │ 13.0 W │ 800.0 MHz  │ 0.73 V  │ True │
│ 3 │ p300c │ 0000:04:00.0 │ 47.9 °C │ 18.0 W │ 800.0 MHz  │ 0.73 V  │ True │
└───┴───────┴──────────────┴─────────┴────────┴────────────┴─────────┴──────┘

Four rows, all p300c, all DRAM True: the machine is healthy. (If you get tt: command not found, see the PATH note in Install and Upgrade with tt.)

Two more tt commands answer the next questions:

tt device info      # board ids, PCIe speed and width, firmware versions, one column per chip
tt smi              # hands the terminal to the live tt-smi dashboard (q to quit)

tt device info shows the firmware bundle on each chip (fw.fw_bundle_version), which is the first thing support asks for. tt device status --json gives the same data as JSON, plus the host’s OS, kernel, memory and driver version, so scripts do not have to parse a table.

The rest of this chapter looks one level lower, at tt-smi, which tt runs for you. You need it for the raw fields, and when something is wrong and you are diagnosing.

Reading Your Hardware with tt-smi

tt-smi is the Tenstorrent System Management Interface. Your window into the chips. Run it in snapshot mode to get JSON instead of the interactive TUI:

tt-smi -s

A healthy QB2 returns four entries — one per Blackhole chip. The real shape is nested, not flat — board_info and telemetry are their own sub-objects:

{
  "device_info": [
    {
      "board_info": {
        "bus_id": "0000:01:00.0",
        "board_type": "p300c",
        "board_id": "0000046131924062",
        "dram_status": true,
        "pcie_speed": 4,
        "pcie_width": "4"
      },
      "telemetry": {
        "voltage": "0.72",
        "current": "23.0",
        "power": "16.0",
        "aiclk": "800",
        "asic_temperature": "40.3"
      },
      "firmwares": {
        "fw_bundle_version": "19.13.1.0"
      }
    }
  ]
}

board_type is the board SKU (p300c on a QB2), not a chip-family string — Blackhole doesn’t appear literally anywhere in the output. Four entries in device_info means four chips, all alive. Check it directly:

tt-smi -s | python3 -m json.tool | grep board_type

You should see "p300c" printed four times.

Reading the Output

A healthy QB2 shows four entries in device_info. Look at each one for:

• board_info.board_type: "p300c" — the board SKU for a QB2 (two chips per card, two cards). “BLACKHOLE” doesn’t appear literally anywhere in the output — that’s the chip family the p300c board carries.
• board_info.pcie_speed: 4 — an integer PCIe generation, not the string "GEN4". 4 is full speed; 3 would mean a slot compatibility issue.
• board_info.pcie_width: "4" — the link width in lanes (as a string). Narrower (lower number) means lower bandwidth.
• telemetry.asic_temperature in the 35–55°C range — normal at idle. Higher under load is fine. board_info and telemetry are nested sub-objects on each device entry, not flat fields — see the JSON sample above.

Count the entries:

tt-smi -s | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['device_info']), 'devices')"

If you see 4 devices, move on. The QB2 is ready.

If You See Fewer Than Four

A missing device usually means one of three things:

PCIe link not established:

sudo dmesg | grep -i tenstorrent | tail -20

sudo is required here — Ubuntu 24.04 restricts the kernel log buffer by default, and a plain dmesg fails with “read kernel buffer failed: Operation not permitted.” Look for errors about PCIe enumeration or firmware loading failure. A loose card is possible — the QB2 ships with cards seated, but transit happens.

Firmware mismatch:

tt-smi -s | python3 -m json.tool | grep -i fw_bundle_version

If firmware versions differ across devices, or show 0.0.0, you may need to reflash. See the tt-flash documentation for instructions.

Driver not loaded:

lsmod | grep tenstorrent

If nothing prints, the kernel driver isn’t loaded. This shouldn’t happen on a stock QB2, but if it does:

sudo modprobe tenstorrent

Watching in Real Time

For a live view of all four chips while running inference:

tt-smi

This opens the interactive TUI — press q to quit. You’ll see per-chip utilization, temperature, and memory usage update live. Useful when a model is running and you want to see all four chips light up.

For something richer than the built-in TUI, tt-toplike renders the same telemetry as live ASCII art — every chip’s power, temperature, and DRAM state, animated:

This is what active inference looks like inside one chip. Four of these run in parallel on a QB2.

Next: Installing the Stack →

Install and Upgrade with tt

tt is the Tenstorrent command line. It is the single tool you use to install the software stack, keep it current, look at your hardware, serve models and connect apps to them. Under the hood it runs tt-installer, tt-smi, tt-flash, tt-inference-server and tt-studio for you, each at a version Tenstorrent has tested together.

A QB2 ships with the software already installed, so this chapter is mostly about upgrading. The same two commands install everything on a fresh Ubuntu machine.

See What You Have

tt --version
tt device status

If you get tt: command not found, ~/.local/bin is not on your PATH in this shell. Run uv tool update-shell (or add export PATH="$HOME/.local/bin:$PATH" to ~/.bashrc) and open a new terminal.

If tt --help does not list device, model, serve and update, another program named tt is earlier on your PATH. which -a tt lists every match in the order the shell tries them. Put ~/.local/bin first (export PATH="$HOME/.local/bin:$PATH"), or start a login shell with bash -l. A non-login shell, such as a plain docker run <image> bash -c '...', skips ~/.profile and can leave the other program in place.

tt device status lists one row per chip. A QB2 has four, on two p300 boards:

                 Tenstorrent devices (driver: TT-KMD 2.11.0)
┏━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ # ┃ Board ┃ Bus ID       ┃ Temp    ┃ Power  ┃ AIClk      ┃ Voltage ┃ DRAM ┃
┡━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ 0 │ p300c │ 0000:01:00.0 │ 55.0 °C │ 74.0 W │ 1337.0 MHz │ 0.81 V  │ True │
│ 1 │ p300c │ 0000:02:00.0 │ 59.9 °C │ 49.0 W │ 1343.0 MHz │ 0.82 V  │ True │
│ 2 │ p300c │ 0000:03:00.0 │ 45.9 °C │ 13.0 W │ 800.0 MHz  │ 0.73 V  │ True │
│ 3 │ p300c │ 0000:04:00.0 │ 47.9 °C │ 18.0 W │ 800.0 MHz  │ 0.73 V  │ True │
└───┴───────┴──────────────┴─────────┴────────┴────────────┴─────────┴──────┘

In that capture, chips 0 and 1 were running a model (high power, 1.3 GHz clock) and chips 2 and 3 were idle (800 MHz). The clock column is the quickest way to see which chips are in use.

Preview an Upgrade

Tenstorrent publishes a golden set: the versions of the kernel driver, firmware, HugePages setup, tt-smi, tt-flash, tt-inference-server and tt-studio that have been tested together. tt update moves your machine onto it. Preview first. This changes nothing:

tt update --dry-run

  Update plan (goldens: bundled supplement.toml + golden.json v3.0.0 (cached))
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ component       ┃ kind     ┃ installed       ┃ golden           ┃ action     ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ tt-flash        │ uv-tool  │ 4.0.0           │ 4.0.0            │ up-to-date │
│ tt-inference-s… │ git-venv │ v0.25.0         │ v0.25.0          │ up-to-date │
│ tt-model        │ uv-tool  │ a7aa9cefdd3086… │ a7aa9cefdd3086b… │ up-to-date │
│ tt-smi          │ uv-tool  │ 6.6.0           │ 6.6.0            │ up-to-date │
│ tt-studio       │ git-venv │ —               │ v2.12.0          │ optional   │
│ system-stack    │ system   │ —               │ hugepages 1.4.1, │ converge   │
│                 │          │                 │ kmd 2.11.0, ...  │            │
└─────────────────┴──────────┴─────────────────┴──────────────────┴────────────┘

Read the action column:

Upgrade

tt update

tt update runs in three phases: it checks the golden manifest, installs or upgrades the tools into isolated environments under ~/.local/share/tenstorrent/, and then downloads the matching tt-installer release and runs it. The last phase uses sudo, so it asks for your password. It also asks before flashing firmware, because flashing resets the devices. Stop any model you are serving first (tt model ps shows what is running).

Upgrade the tt command itself separately:

tt self update --check    # is there a newer release?
tt self update

On the QB2 used to test this guide, tt self update --check reported tt 1.2.1 is available (you have 1.2.0). tt update brings the system and its tools onto the golden set, and tt self update upgrades tt. Run both when you want to be current.

Installing on a Fresh Machine

You need Ubuntu 22.04 or 24.04, sudo, and a network connection. Install uv, then tt:

sudo apt update && sudo apt install -y python3 python3-venv python3-pip git curl ca-certificates
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"   # uv and tt install here; this makes them work in this terminal
uv tool install tenstorrent
uv tool update-shell        # puts ~/.local/bin on PATH for future terminals
tt update

The PyPI package is named tenstorrent. The unrelated tt-cli package on PyPI is a time tracker. tt update installs the kernel driver, firmware tooling and HugePages configuration as root, so reboot afterwards.

Using tt-installer Directly

tt update runs tt-installer for you, and for almost everyone that is the right way to use it. tt update <VERSION> runs a specific installer release instead (and implies --force, which allows downgrades). The installer’s own options, and the manual path it replaces, are covered in Installing by hand.

Check Your Work

tt --version
tt update --dry-run      # every row should say up-to-date
tt device status         # four rows, all DRAM True
tt model list            # models for your hardware

If tt device status shows fewer than four rows, go back to Is This Thing On? for the diagnosis steps.

Next: Serve a Model →

Serve a Model

Four chips, one command. tt serve downloads or finds the weights, starts the model in a container, waits until it answers, and tells you the address. What you get is an OpenAI-compatible HTTP API running on your own hardware, which every chat app, coding agent and SDK already knows how to talk to.

This chapter serves Qwen3-32B, the model your QB2 shipped with. Its weights and compiled kernels are already on disk, so the first serve does not download 62 GB.

Pick the Model

tt knows which models run on your hardware:

tt model list

On a QB2 it detects a p300x2 configuration (two p300 boards, four chips) and lists the models that have a serving profile for it. tt model info tells you what a model needs and how well it is validated:

tt model info Qwen3-32B

┌─────────────────────┬────────────────────────────────────────────────────────┐
│ hf_repo             │ Qwen/Qwen3-32B                                         │
│ type                │ llm                                                    │
│ engines             │ vLLM                                                   │
│ via                 │ inference-server, studio                               │
│ servable            │ yes — `tt serve Qwen3-32B` (`--studio` deploys it with │
│                     │ TT-Studio)                                             │
│ parameters          │ 32B                                                    │
│ min disk            │ 84 GB                                                  │
│ min ram             │ 80 GB                                                  │
│ on p300x2           │ vLLM — FUNCTIONAL, max context 131072                  │
│ ...                 │                                                        │
└─────────────────────┴────────────────────────────────────────────────────────┘

Validation status goes EXPERIMENTAL, FUNCTIONAL, COMPLETE. FUNCTIONAL means it runs correctly and is not fully tuned. The status is per board, so check the p300x2 row.

Models worth starting with on a QB2

These have a p300x2 profile in the released catalog, from tt model list --catalog:

Gated models (Llama, Gemma) need you to accept the license on the model’s Hugging Face page first. The hf_repo line in tt model info names the page.

Log In to Hugging Face

Serving needs a Hugging Face token, even for models that are free to download, because tt-inference-server checks it before it starts. Log in once:

tt model login

Choose Log in with your browser (it shows a URL and a short code) or paste a read token from huggingface.co/settings/tokens. tt finds the token again on every later run, and tt serve --dry-run tells you where it came from without printing it.

Check That tt Will Reuse the Shipped Weights

Preview the serve before you run it. --dry-run starts nothing:

tt serve Qwen3-32B --dry-run

✓ Detecting device configuration  p300x2 (override with --device)
                    tt serve Qwen3-32B — dry run
┌──────────────────┬──────────────────────────────────────────────────────────┐
│ backend          │ tt-inference-server                                      │
│ workflow         │ server                                                   │
│ device           │ p300x2                                                   │
│ engines          │ vLLM                                                     │
│ status           │ FUNCTIONAL                                               │
│ docker image     │ ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-…  │
│ tool-call parser │ hermes                                                   │
│ reasoning parser │ qwen3                                                    │
│ weights          │ /home/ttuser/data/tt-cache                               │
│                  │ pre-seeded volume — weights and tt_metal_cache reused;   │
│                  │ the HF cache is not consulted                            │
│ HF token         │ from the Hugging Face login store                        │
│ port             │ 20000  (server default)                                  │
└──────────────────┴──────────────────────────────────────────────────────────┘

Look at the weights row. pre-seeded volume means tt found the shipped weights and the compiled kernel cache, so the first start takes minutes and downloads nothing. If it says ~/.cache/huggingface instead, tt did not recognize the shipped directory and a real serve would download 62 GB. The fix is one symbolic link:

ln -s ~/data/tt-cache/volume_id_tt_transformers-Qwen3-32B-vqb2_launch \
      ~/data/tt-cache/volume_id_tt_transformers-Qwen3-32B-v0.17.0

Serve It

tt serve Qwen3-32B

tt serve shows the boot as a live checklist instead of thousands of log lines: host ready, weights ready, image ready, container started, engine started, device opened, KV cache configured, model warmed up, endpoint answering. It returns only once the endpoint answers, so whatever you run next will work. When it finishes you get a box with the endpoint (http://127.0.0.1:20000/v1) and the commands to look at logs and stop the model.

To watch the raw server log while it boots, run this in a second terminal:

tt model logs Qwen3-32B --follow

Pressing Ctrl-C in the tt serve terminal stops watching. The server is a separate container and keeps booting, and tt model stop Qwen3-32B ends it.

Ask It a Question

tt model curl "Explain what an API is in one sentence." --max-tokens 2048

tt model curl asks the server which model it is serving, sends your prompt as a chat completion, and prints the JSON reply. The text is in choices[0].message.content. Unknown options become fields of the request, so --max-tokens 2048 sets max_tokens. Add --print to see the equivalent curl command.

Qwen3-32B is a reasoning model (the dry run above shows a qwen3 reasoning parser): it thinks first, in a separate reasoning field, and then answers in content. A small token budget can be used up by the thinking, so give it room when content comes back empty.

The same endpoint speaks the standard OpenAI API, so any client works. Note the model name: the server reports the full Hugging Face id, which differs from the short name you gave tt serve (for Llama-3.1-8B-Instruct we saw meta-llama/Llama-3.1-8B-Instruct).

curl -s http://localhost:20000/v1/models | python3 -m json.tool | grep '"id"'

curl -s http://localhost:20000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-32B",
    "messages": [{"role": "user", "content": "Explain what an API is in one sentence."}],
    "max_tokens": 2048
  }' | python3 -m json.tool

Manage What Is Running

tt model ps                    # what is served, on which port, and whether it is healthy
tt model logs Qwen3-32B        # the newest server log

Keep the model running for the next chapter. When you are finished with it, tt model stop Qwen3-32B stops it.

tt model ps shows every server tt started, including ones on other ports:

┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━┓
┃ name                  ┃ backend          ┃ container                    ┃ port  ┃ health  ┃ uptime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━┩
│ Llama-3.1-8B-Instruct │ inference-server │ tt-inference-server-162a825f │ 20000 │ healthy │ 5m     │
└───────────────────────┴──────────────────┴──────────────────────────────┴───────┴─────────┴────────┘

healthy means the server answers GET /v1/models. starting means the container is running and not answering yet. tt model stop stops only the named model’s container.

Serving on Half the Machine

A QB2 is two p300 boards. A model with a p300 profile uses one board (two chips). For example, Llama-3.1-8B-Instruct has a p300 profile:

tt serve Llama-3.1-8B-Instruct --device p300 --dry-run

Use --device to override what tt detected. tt serve does not let you choose which board: the container is given every Tenstorrent device node on the machine, so run a half-machine model only when nothing else is using the chips. How to pin a server to specific chips is in Serving Models on QB2.

What Just Happened

tt serve ran tt-inference-server, which started a container holding vLLM and the Tenstorrent plugin. The container opened a mesh across your chips, loaded the weights into device memory, compiled and warmed up the kernels, and published the API. The detail of each stage, and how to run tt-inference-server yourself, is in Serving Models on QB2.

Next: Launch Software →

Launch Software

An API endpoint is the foundation. tt launch connects a ready-made application to the model you are already serving, so you can use it without copying a URL into a settings page. You do not tell the app where the model is: tt asks every server it started what it serves (GET /v1/models) and configures the client from the answer.

Keep the model from the last chapter running, and see what you can connect:

tt launch list

┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ client      ┃ kind        ┃ model          ┃ status         ┃ configures                           ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ opencode    │ terminal    │ tool calling   │ needs opencode │ ~/.config/opencode/opencode.json     │
│ pi          │ terminal    │ tool calling   │ needs pi       │ ~/.pi/agent/models.json              │
│ aider       │ terminal    │ tool calling   │ needs aider    │ environment (this run only)          │
│ qwencode    │ terminal    │ tool calling   │ ready          │ command-line arguments (this run…)   │
│ hermes      │ terminal    │ tool calling   │ ready          │ environment (this run only)          │
│ openwebui   │ web service │ any chat model │ stopped        │ container tt-open-webui              │
│ anythingllm │ web service │ any chat model │ ready          │ container tt-anythingllm             │
└─────────────┴─────────────┴────────────────┴────────────────┴──────────────────────────────────────┘

There are two kinds of client, and the difference is who installs them:

• Web UIs (Open WebUI, AnythingLLM). tt pulls and runs these in a container for you, after asking once. You only need Docker.
• Terminal coding agents (OpenCode, pi, Aider, Qwen Code, Hermes Agent). You install the agent. tt launch writes the small config it needs to reach your model and hands your terminal to it. These need a model that supports tool calling; tt checks, and stops with a suggestion if the model cannot.

The status column says what is missing: needs opencode means the agent is not installed, ready means it can launch now, and stopped is a web UI container that exists and is not running.

A Chat Window in Your Browser

tt launch openwebui --dry-run     # show what would happen, change nothing
tt launch openwebui

The dry run prints what tt found and what it will do:

┌───────────────────────┬───────────────────────────────────────────────────────────┐
│ model                 │ meta-llama/Llama-3.1-8B-Instruct  (Llama-3.1-8B-Instruct) │
│ endpoint              │ http://127.0.0.1:20000/v1                                 │
│ tool-call parser      │ llama3_json                                               │
│ container             │ tt-open-webui (stopped)                                   │
│ image                 │ ghcr.io/open-webui/open-webui:main                        │
│ endpoint in container │ http://host.docker.internal:20000/v1                      │
│ open                  │ http://localhost:3000                                     │
└───────────────────────┴───────────────────────────────────────────────────────────┘

The first time, tt asks before it pulls the image (a few GB) and runs it as tt-open-webui. It waits until the page answers and prints tt-open-webui is ready at http://localhost:3000. Open that address in a browser. The container is configured with your endpoint (OPENAI_API_BASE_URL=http://host.docker.internal:20000/v1), so its model list comes from your server.

If two models are being served, say which one: tt launch openwebui --model Qwen3-32B. Web UIs start on port 3000, or the next free port above it; --web-port 3080 asks for another. When you are done:

tt launch stop openwebui      # stops the container, keeps your chats and settings

A Coding Agent in Your Terminal

Terminal agents work on your files, so they need a model that can call tools. Qwen3-32B can, with one caveat:

Faster agent models come from the community catalog. The fastest tool-calling models on a QB2 are mixture-of-experts coders with 3B parameters active per token, published as community bundles: raahemnabeel/qwen3-coder-30b-a3b and raahemnabeel/qwen3-coder-next, both with the qwen3_coder tool parser and no thinking mode. Measured on 2026-10-30 on one P300 board: 80 tok/s and 107 tok/s on a real prompt, against about 8 seconds a response for As an alternative to the model from the last chapter, stop it first (tt model stop Qwen3-32B), then serve one with tt serve raahemnabeel/qwen3-coder-30b-a3b --profile p300x2 (see The Model Zoo for the warning that applies to every bundle). tt launch finds it the same way. It prints not in the model catalog and shows tool-call parser — because it has no catalog row to check; the bundle’s card and tt serve <bundle> --dry-run show the parser is set. We ran tt launch qwencode --dry-run and a smolagents task against Qwen3-Coder-Next this way; the task was answered in one 2 s step.

Install one agent, then launch it. Qwen Code and Hermes Agent were ready in the listing above:

tt launch qwencode --dry-run

┌──────────────────┬───────────────────────────────────────────────────────────┐
│ model            │ meta-llama/Llama-3.1-8B-Instruct  (Llama-3.1-8B-Instruct) │
│ endpoint         │ http://127.0.0.1:20000/v1                                 │
│ tool-call parser │ llama3_json                                               │
└──────────────────┴───────────────────────────────────────────────────────────┘

commands:
qwen --auth-type openai --openai-api-key tt-local --openai-base-url http://127.0.0.1:20000/v1 --model meta-llama/Llama-3.1-8B-Instruct

The dry run shows the exact command tt launch runs: your agent, pointed at your endpoint, with no cloud account. Run it without --dry-run to start the agent.

tt shows what it will change and asks before writing a config file. tt launch disconnect opencode removes what it wrote.

Claude Code with Tenstorrent skills

The development version of tt (after release 1.2.1) adds tt agent, which sets up Claude Code with the plugins from the tenstorrent/skills marketplace for what you are doing today (deploy, bringup or develop) and starts it. Preview with tt agent deploy --dry-run. Check tt --help for an agent entry to see whether your release has it.

Prefer Point and Click? TT-Studio

TT-Studio serves the model and provides its own web interface. You start it with tt serve --studio, not tt launch:

tt model stop Qwen3-32B        # TT-Studio takes the chips and port 3000 for itself
tt serve Qwen3-32B --studio

On first use tt fetches the latest TT-Studio release, and TT-Studio asks you to accept its terms. It prints a TT Studio is ready box with its address, http://localhost:3000. Stopping the model with tt model stop Qwen3-32B also takes TT-Studio’s services down and keeps your settings and chat history. Start TT-Studio before any tt launch web UI, since both want port 3000 and only the web UIs move when it is taken.

Check Your Work

tt model ps               # your model is healthy
tt launch list            # your chosen client is ready or connected
tt launch stop openwebui  # clean up the web UI when you are done
tt model stop Qwen3-32B   # and the model

Next: What Comes Next →

What Comes Next

You unboxed a machine that most people have never touched. You confirmed four Blackhole chips were alive, upgraded the software with tt update, served a model with tt serve, asked it a question, and connected an app to it with tt launch. That is the whole loop, and you ran it on your own hardware.

The rest is up to you.

Tools in Your World

The QB2 ships with a full stack, but the ecosystem is bigger. Start with tt-toplike — htop for your chips, except the telemetry comes alive as ASCII art:

Where to Go From Here

Pick a thing you want to do and jump straight in.

Choose Your Next Track

The QB2 is a beginning. There’s a lot of surface area here, and you’ve only scratched it.

← Back to Explore

Coming From CUDA

You know cudaMalloc. You know grid-dim and block-dim. You’ve tuned shared memory usage, you’ve written custom CUDA kernels, and you’ve debugged timing issues with Nsight. You have a mental model of how GPU compute actually works, not just how PyTorch wraps it.

That mental model transfers here, but not intact. Some pieces map cleanly. Some don’t exist. And some things you were papering over on the GPU are now explicit, visible, and tunable. The next ten minutes remaps the terrain.

Pick Your Altitude

The first question a CUDA developer asks is “where’s my model.cuda()?” The honest answer is that there isn’t one entry point — there are three, and which one you reach for depends on how much control you want. CUDA has the same three tiers; you just rarely think about them as a stack because NVIDIA blurs the seams.

The closest thing to model.cuda() is the top tier: TT-Forge traces your graph and lowers it to Tensix automatically. That’s Chapter 6 — reach for it when you want the model to just run. The two bottom tiers are for the cases TTNN doesn’t cover: TT-Lang lets you write a custom kernel in Python with no C++, and TT-Metalium is the C++ floor where every abstraction disappears. Both live in the Builder/Hacker track — and, as the last section of this chapter explains, the TT-Lang tier is far more reachable than “write your own kernel” sounds on CUDA.

This track lives in the middle, at TTNN — the tier where you have hand-optimized ops but still write Python, not kernels. It’s the sweet spot for performance work that doesn’t require descending to the metal, so that’s where the rest of this chapter focuses.

Thread Blocks vs. Tensix Tiles

On a GPU, a thread block is the unit of cooperative work: a group of threads that can share L1/shared memory and synchronize. The programmer launches a grid of blocks; the hardware schedules them onto SMs.

On Blackhole, the unit is a Tensix core. There are 120 enabled per chip (a 12×10 block of the 14×10 physical Tensix grid), sitting inside a larger 17×12 NoC grid that also carries DRAM, Ethernet, and PCIe nodes. Each core has its own L1 SRAM, its own set of RISC-V processing cores (five of them), and its own connection to the Network-on-Chip (NoC) fabric that threads through the entire grid. Tensix cores don’t share memory with each other. There’s no “block-scope” shared memory. There’s only what one core holds, and what it explicitly sends over the NoC to another.

This is the fundamental shift. On CUDA, data sharing between threads in a block is cheap and implicit — shared memory just works. On Tensix, data movement is the thing you design around. Every byte a core receives came from somewhere specific, via a routed packet on the NoC. That movement is visible to you. It’s also where the performance is.

There’s a deeper reason it’s visible: there is no warp scheduler hiding memory latency. On a GPU, when one warp stalls waiting on a global-memory read, the SM scheduler instantly swaps in another resident warp — latency disappears behind oversubscription, and you mostly don’t think about it. Tensix has no such trick. Instead, each core runs an explicit reader → compute → writer pipeline: one RISC-V core streams tiles into L1, the matrix engine works on them, another core streams results out, and they overlap by design rather than by lucky scheduling. At the TTNN level you don’t write that pipeline — the ops do — but it’s why tensor layout matters so much here. A layout that lets the reader stage clean tiles keeps the pipeline full; one that forces a reshuffle stalls it, and there’s no spare warp to paper over the gap.

L1 SRAM vs. Shared Memory

Each Tensix core has 1.5 MB of L1 SRAM. On a GPU, your shared memory budget is typically 48–96 KB per SM, and you fight for it. On Tensix, you have a full 1.5 MB per core to work with.

The catch: that memory doesn’t auto-populate. On a GPU, you launch a kernel and global memory reads happen via caches. On Tensix, you write the code that moves data from DRAM (the rows at the top and bottom of the chip grid) into the L1 of whichever cores need it. TTNN does this for you when you use its built-in ops — but if you drop to Metalium, you’re writing those NoC reads yourself.

For someone writing at the TTNN Python level (which is where this track lives), the takeaway is simpler: tensor operations in TTNN are already written to stage data correctly. You don’t write data movement code. But you do care about tensor layout, because layout determines whether the underlying kernels can move data efficiently or have to reshuffle it first.

TTNN as the CUDA Runtime Equivalent

Think of TTNN the way you think of libcudart plus cuBLAS plus cuDNN — all fused into one Python API. It handles device open/close, tensor allocation in device memory, op dispatch, kernel compilation (via Metalium under the hood), and synchronization.

The critical difference from cuBLAS: TTNN compiles ops JIT on first invocation. When you run a matrix multiply for the first time on a new tensor shape, Metalium generates a Tensix kernel for that exact configuration. Subsequent calls with the same shape hit the op cache and run fast. This is why first-run latency can be a few seconds — and why subsequent runs are fast enough to serve production traffic.

import ttnn
import torch

# Open a single chip (device_id=0)
device = ttnn.open_device(device_id=0)

# Move a PyTorch tensor to device
torch_a = torch.randn(1024, 1024)
a = ttnn.from_torch(torch_a, device=device, dtype=ttnn.bfloat16)

# This compiles on first run, then caches
result = ttnn.matmul(a, a)

# Pull back to CPU
out = ttnn.to_torch(result)
ttnn.close_device(device)

Compare this to CUDA: cudaMemcpy, cublasSgemm, cudaMemcpy back. The pattern is the same. The surface is different.

What Transfers From CUDA Knowledge

Tensor shapes, batch dimensions, attention head patterns — all of this maps directly. The math doesn’t change. The numerics don’t change (bfloat16 is first-class here, same as modern GPUs). Batching strategies that work on GPU work on Tensix.

Knowledge of kernel fusion matters. The same principle applies: fewer round-trips through memory means faster execution. TTNN has fused ops (fused attention, fused feedforward) that follow the same logic as FlashAttention on CUDA.

Multi-device tensor parallelism maps directly too. The QB2 has four chips. When you run a 70B model, attention heads get split across chips the same way they’d split across GPUs in a tensor-parallel setup. The API is different — ttnn.CreateDevices({0,1,2,3}) instead of torch.distributed — but the concept transfers.

What Doesn’t Transfer

CUBLAS and cuDNN don’t exist here. There’s no drop-in replacement. If your code calls torch.nn.functional.conv2d and you want it to run on Blackhole, you need to either use TTNN’s conv2d op or compile via TT-Forge (which traces PyTorch graphs and lowers them to TTNN). You can’t just model.cuda() and move on.

Device memory pointers are gone. CUDA lets you grab a raw void* to device memory and pass it around. TTNN tensors are opaque objects — no raw pointer access. If your code does custom CUDA pointer arithmetic, that approach doesn’t port. You use TTNN ops, or you write Metalium kernels (a Tinker track topic).

Unified memory has no equivalent. There’s no cudaMallocManaged. Data is either on CPU or on the device, and you move it explicitly via ttnn.from_torch and ttnn.to_torch.

Grid launch syntax is gone. There’s no <<<gridDim, blockDim>>>. Kernel dispatch is handled by the TTNN op, which decides how to tile the work across the Tensix grid. You influence this via tensor layout and op selection, not by choosing block/thread dimensions.

CUDA Concept Mapping Table

Blackhole’s NoC Fabric

The four Blackhole chips in your QB2 sit on two p300c cards, linked by an internal Samtec ARP6 cable and on-chip Ethernet (800 Gbps between any two chips) — not PCIe (PCIe is only the host-to-card link). Intra-chip, the NoC connects every core to every other core and to the DRAM banks at roughly 1 TB/s aggregate bandwidth. This is not the same topology as NVLink or PCIe between discrete GPUs — it’s a different architecture where the cost of moving data within a chip is much lower relative to compute throughput than on a GPU.

For multi-chip workloads, the four chips form a mesh using their Ethernet cores (the left and right columns of the chip grid). This is how tensor-parallel models distribute their KV-cache updates — not through the host CPU, but directly chip-to-chip.

Matrix multiply: DRAM rows stage the operand tiles, compute cores pull them over the NoC and run.

Custom Kernels Without the Dread — and the Agentic Shortcut

On CUDA, “you’ll need a custom kernel” is a sentence that ends a lot of afternoons. It means C++, it means reasoning about occupancy and warp divergence and memory coalescing, and it means racing against bugs that only show up at certain block sizes. It’s also exactly the kind of code that AI coding agents are bad at: so much of a CUDA kernel’s correctness lives in implicit, unstated assumptions — what’s resident, what’s coalesced, which warp got there first — that there’s nothing concrete for an agent to verify against. The spec isn’t in the source; it’s in the programmer’s head.

The middle-lower tier, TT-Lang, inverts that. It’s a Python DSL (no C++) for writing the one custom op TTNN doesn’t expose — a fused pattern, a non-standard attention variant, an activation with a specific numerical property. And it’s built around the same reader → compute → writer structure from earlier in this chapter, except now you write the three sections explicitly: the reader declares exactly which tiles arrive and from where, compute is pure tile math on those arrivals, the writer declares exactly what leaves. Nothing is implicit.

That explicitness is the whole trick, and it’s why agentic development gets you remarkably far here. Because the full spec lives in the source — arrivals in, math, departures out — an AI coding agent has something complete to generate against and something concrete to check its work against. You describe the kernel in those three terms, the agent fills in the TT-Lang syntax, and the structure itself eliminates most of the ambiguity that makes agent-written CUDA hallucinate. So the practical ladder for someone coming from CUDA looks like this:

1. Let TT-Forge compile the whole model — most of the time you stop here.
2. Reach for TTNN ops when you want to hand-tune a hot path in Python.
3. Hand an agent a reader/compute/writer spec and let it write the TT-Lang for the rare custom kernel — instead of booking an afternoon to hand-write CUDA C.

You can travel a long way down that ladder without ever becoming a full-time kernel author. When you do want to go deeper into TT-Lang yourself — the decorators, circular-buffer semantics, the browser-based simulator — that’s the TT-Lang chapter in the Builder/Hacker track.

Setting Expectations

One more thing that won’t transfer from a decade of CUDA: the assumption that the stack is finished. CUDA is twenty years mature; the TT software stack is young and moving fast. The top-tier compiler frontends in particular are still evolving — by the time you read Chapter 6 you’ll see we already had to retire one PyTorch entry point in favor of TT-XLA. Expect occasional rough edges, expect the first run of a new op shape to JIT-compile for a few seconds before it caches, and expect to read the docs against the source now and then.

That’s not a warning to stay away — it’s the texture of working close to the edge of an open stack. The flip side is that the layers are genuinely open, the team is reachable, and unanswered questions tend to get answers. When something doesn’t behave the way this guide describes, the Tenstorrent Discord and the GitHub issue trackers are where practitioners (and TT engineers) actually work problems out.

Next: The Model Zoo →

The Model Zoo

Four chips. Up to 560 Tensix compute cores available at once. The question isn’t whether the hardware can handle real models — it’s which ones, at what scale, and how to get them here.

What’s Supported

The QB2 supports a focused set of model families, optimized for Blackhole silicon. These aren’t compatibility hacks — they’re models with hand-tuned TTNN kernels for the Blackhole architecture, validated for throughput and output quality.

The model zoo lesson in tt-vscode-toolkit covers this in interactive depth, with live benchmarks you can run against your own QB2: tt-vscode-toolkit lessons →

Beyond text: video and image generation

The QB2 isn’t only a text box. tt-studio (v2.8.0 and current as of v2.10.0) adds two generative-media families to the same deploy-and-run flow:

• WAN — text-to-video. Describe a scene, get a short generated clip.
• Flux — text-to-image. A high-quality diffusion image generator.

They deploy exactly like the language models: open tt-studio at localhost:3000, pick the model from the Deploy Model dropdown, and click Run. tt-studio spins up the right container on your Blackhole chips and gives you a prompt box — no pipeline code, no separate install.

These are heavier than an 8B chat model — a text-to-video diffusion run leans on more of the board than a single-chip LLM. If a deploy is slow to go green the first time, that’s compilation and weight loading; subsequent runs load from the on-disk cache.

Community Models and tt-model-manager

The released catalog is the models Tenstorrent ships and tests through tt-inference-server. Beyond it, anyone can publish a model bundle to the Hugging Face Hub with tt-model-manager: a bring-up packaged so that someone else can serve it with the same commands they use for catalog models. A bundle carries the engine the author built and a pointer to the weights, which download into your own Hugging Face cache under your own token.

To browse instead of search, the tt-awesome model browser lists every catalog model and community bundle by collection, family, kind and hardware, with the commands to install and run each. It marks community bundles with the same warning.

tt model search qwen3 --limit 5     # find published bundles on the Hub
tt model list --community           # community bundles that run on this machine
tt model info raahemnabeel/qwen3-coder-next    # manifest and compatibility verdict
tt model pull raahemnabeel/qwen3-coder-next
tt model profiles raahemnabeel/qwen3-coder-next
tt serve raahemnabeel/qwen3-coder-next         # served by tt-model-manager

A real search on a QB2:

          tt-model bundles on the Hub — every published bundle matching 'qwen3'
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ name                                ┃ visibility ┃ downloads ┃ updated    ┃ installed ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ raahemnabeel/qwen3-5-122b-a10b      │ public     │ 0         │ 2026-10-28 │ —         │
│ raahemnabeel/qwen3-coder-next       │ public     │ 0         │ 2026-10-28 │ ✓         │
│ episod/qwen3.8-27b-dflash2-p300     │ public     │ 0         │ 2026-10-28 │ ✓         │
└─────────────────────────────────────┴────────────┴───────────┴────────────┴───────────┘

Names are namespace/name. A name with a slash that is not a catalog model is a bundle, and tt serve routes it to tt-model-manager. tt model ps lists both kinds side by side, with a backend column (inference-server or model-manager) and the port each is on.

How to choose:

Compatibility is checked, not guaranteed: pulling or serving a bundle checks that its binaries were built for your chip architecture (a mismatch fails on purpose) and warns if your box has fewer chips than the profile needs. Read tt model info before a large download.

“I want to…” and what to search

These are all community bundles, so the warning above applies to every row. tt model search matches the text of a bundle’s name, so search for a model family or a task word that models usually carry in their name. These all returned bundles on a QB2 on 2026-10-28. Bundles are published often, so your results will differ.

The two coder bundles are the fast ones for agents: 30B and 80B parameters on disk, 3B active per token. Measured 2026-10-30 on one P300 board (--profile p300), 384 output tokens at temperature 0: qwen3-coder-30b-a3b 80 tok/s, qwen3-coder-next 107 tok/s. Their cards report 95 and 178 tok/s on all four chips (--profile p300x2). Both set the qwen3_coder tool parser, and both passed the tt-agents tool-calling check (00_verify_tools.py). A bundle you pulled earlier keeps the revision it was pulled at; if tt serve says no serve profile named 'p300' for a profile the card lists, add -- --refresh to fetch the current one.

Some searches found nothing (whisper, video, robot, pose, rerank). Two reasons: speech-to-text and video models such as whisper-large-v3 and Wan2.2-T2V-A14B-Diffusers are catalog models, not bundles, so look for them with tt model list --type audio or tt model list --type video. The other terms are just not in any bundle’s name yet. Empty results mean nobody has published one, not that the hardware cannot do it.

The suffix on a bundle name is the hardware it was built for. -p150 is one chip, -p300 is one board of two chips, and -p300x2 is the whole QB2, so a -p150 bundle fits on a QB2 and a -p300x2 bundle needs all four chips. tt model list --community --hw p300x2 shows only the bundles that fit your machine, and tt model search <query> --catalog keeps just the ones in the community catalog.

Picking a Starting Point

Qwen3-0.6B is the fastest way to confirm the stack is working. It downloads in seconds, loads in under a minute, and produces real answers. Think of it as the “hello world” of this hardware — with one caveat worth knowing before you build a plan around it: it is absent from tt-inference-server’s model list, so run.py --model Qwen3-0.6B has nothing to select. It’s a smoke test for direct TTNN work and for a hand-rolled vllm serve, not a model you can deploy down the managed path.

Llama-3.1-8B-Instruct is where you start if you need production-quality output on a single chip. Strong reasoning, strong instruction-following, 128K context. The model most people actually use for serious work on a single Blackhole.

Qwen3-8B is a strong alternative in the same size class as Llama-3.1-8B. Use it if your workload benefits from Qwen’s architectural choices, or to compare against the 0.6B for quality/speed tradeoffs. Note that tt-inference-server lists it for p300 (two chips) but not p300x2 — and the two-chip mesh is the one that has failed fabric bring-up on our hardware. Llama-3.1-8B-Instruct, which has both a p300 and a p300x2 entry, is the lower-risk pick in this size class on a QB2.

Llama-3.1-70B-Instruct requires all four chips and 140 GB of storage. It’s the top-of-rack option for workloads where quality is the priority. Inference speed is lower than the 8B, but the output quality difference is real on complex tasks.

Qwen3-32B is the best zero-download default: it comes pre-cached on QB2, so there’s no multi-gigabyte pull standing between you and a real model. It needs all four chips (32B is the same size class as a 70B for chip-sizing purposes — use --tt-device p300x2), runs a 131K context window, and serves up to 8 concurrent requests. If you want a strong model running in the next sixty seconds, this is it.

Llama-3.3-70B-Instruct is the flagship choice — the same four-chip, 131K-context, up-to-8-concurrent profile as Qwen3-32B, but Meta’s newest 70B-class weights for maximum output quality. See the dedicated Llama-3.3-70B lesson for a full walkthrough.

Model-Suitability Caveats

Two more models are worth knowing about before you reach for them — especially for agentic or tool-calling use (see agents.md’s Coding Agents section for the tt-studio gateway setup), because neither holds up well under sustained multi-step agent loops:

Downloading Models

For catalog models, tt downloads for you: tt serve <name> fetches the weights on first use, and tt model pull <name> fetches them ahead of time (tt model pull Llama-3.1-8B-Instruct fetches into the Hugging Face cache and reports where). The hf commands below are for downloading a repository you will use outside tt, such as the TTNN examples.

Downloads go through the hf CLI — not huggingface-cli, not Python API calls. It’s faster and it handles partial downloads and resumption correctly.

It is not pre-installed. huggingface_hub isn’t in any environment tt-installer creates, so hf won’t be on your PATH out of the box. Install it into an environment of its own — not ~/.tenstorrent-venv, which holds tt-smi and tt-flash and is activated for you at login:

uv tool install huggingface_hub     # or: pipx install huggingface_hub
hf version

# Make sure the models directory exists
mkdir -p ~/models

# Qwen3-0.6B — 1.5 GB, fast start
hf download Qwen/Qwen3-0.6B --local-dir ~/models/Qwen3-0.6B

# Llama-3.1-8B-Instruct — 16 GB, requires HF login with license acceptance
hf download meta-llama/Llama-3.1-8B-Instruct --local-dir ~/models/Llama-3.1-8B-Instruct

# Qwen3-8B — ~16 GB
hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen3-8B

# Llama-3.1-70B-Instruct — 140 GB, plan your storage
hf download meta-llama/Llama-3.1-70B-Instruct --local-dir ~/models/Llama-3.1-70B-Instruct

Llama models require accepting the Meta license on Hugging Face first. If hf download returns a 401 or 403, run hf login and authenticate with a token that has access to the gated model.

Model Storage Layout

Every Tenstorrent tutorial uses the ~/models/<family>-<variant>/ convention. The tt-inference-server --model flag accepts a path or a model name, but matching the convention means tutorial commands work verbatim.

~/models/
  Qwen3-0.6B/
    config.json
    tokenizer.json
    model.safetensors
    ...
  Llama-3.1-8B-Instruct/
    config.json
    tokenizer.json
    model-00001-of-00004.safetensors
    ...
    ...
  Llama-3.1-70B-Instruct/
    ...

Shard count depends on model size, not family — small models like Qwen3-0.6B download as one model.safetensors file; larger ones split across several model-0000N-of-0000M.safetensors files (verified against the actual repo file listings: Qwen3-0.6B is genuinely single-file — confirmed by downloading it — and Llama-3.1-8B-Instruct is 4 shards).

Qwen3 Reasoning Modes

Qwen3 models support two inference modes: thinking mode and non-thinking mode. In thinking mode, the model emits <think>...</think> tokens before its final answer — extended chain-of-thought reasoning that improves quality on multi-step problems at the cost of more tokens and higher latency.

When calling through the OpenAI-compatible API, enable_thinking is a chat-template parameter, not a top-level sampling one — it goes through extra_body’s chat_template_kwargs, the same way vLLM passes any other kwarg into the tokenizer’s apply_chat_template():

# Thinking mode (default for Qwen3) — slower, more thorough
response = client.chat.completions.create(
    model="Qwen/Qwen3-32B",
    messages=[{"role": "user", "content": "What is 17 * 23 + 48?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}}
)

# Non-thinking mode — faster, direct answers
response = client.chat.completions.create(
    model="Qwen/Qwen3-32B",
    messages=[{"role": "user", "content": "What is 17 * 23 + 48?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)

For conversational workloads where speed matters, non-thinking mode is the better choice. For tasks where the reasoning trace improves output quality — math, code, multi-hop questions — thinking mode earns its overhead.

Single-Chip vs. Four-Chip Layout

When you run a single-chip model, all 120 Tensix cores on one chip handle the entire forward pass. When you scale to four chips with tensor parallelism, attention heads split across chips and activations flow chip-to-chip via the Ethernet cores in the left and right columns of the grid.

One chip for small models. Four chips sharing attention heads for 70B scale.

Check Space Before Downloading

# Check available space
df -h ~/models

# Verify a download completed (no missing shards)
ls -lh ~/models/Llama-3.1-8B-Instruct/*.safetensors | wc -l

A correctly downloaded Llama-3.1-8B-Instruct should have 4 safetensors shards. Qwen3-0.6B has 1.

Next: Serving Models on QB2 →

Serving Models on QB2

Serve a Model got a model answering with one tt serve command. This chapter is what is underneath that command, and what to do when you need more than the default: another port, two servers at once, a model that is not in the catalog, or a server on specific chips. By the end you will have an OpenAI-compatible server, a working curl command, and a Python client snippet you can drop into an application.

The Deployment Stack

tt serve <model> picks one of three serving paths from the model:

tt model info <model> lists the paths a model supports, and --inference-server, --studio or --model-manager forces one. All three produce an OpenAI-compatible API.

The direct vLLM path runs vllm serve yourself, for flags the managed path doesn’t expose or a newer vLLM than the pinned image ships. vLLM is not installed on a QB2, not in ~/.tenstorrent-venv and not on the host at all; the copy the managed path uses lives inside a container image. Building that environment is its own project: Build Your Own vLLM Environment →.

Serving with tt serve

tt serve Qwen3-32B                    # the model that ships with the box
tt serve Qwen3-32B --port 8000        # choose the host port
tt serve Qwen3-32B --dry-run          # show the resolved configuration and command
tt serve Llama-3.1-8B-Instruct --device p300   # use a different mesh than the detected one

The default port is 20000, which is tt-inference-server’s own default through tt. The curl examples later in this chapter use 8000, so either start the server with --port 8000 or substitute 20000 in the examples.

tt serve --dry-run prints the exact run.py command tt would run. This is the best way to learn what tt does for you, and to copy a command when you want to run it yourself:

/home/ttuser/.local/share/tenstorrent/tools/tt-inference-server-v0.25.0/venv/bin/python
  .../src/run.py --model Qwen3-32B --workflow server --docker-server --no-auth
  --host-volume /home/ttuser/data/tt-cache --device p300x2
  --vllm-override-args '{"enable-auto-tool-choice": true, "tool-call-parser": "hermes",
  "reasoning-parser": "qwen3"}' --service-port 20000

Three things tt adds that a bare run.py call does not:

• The parsers. tool-call-parser and reasoning-parser are in the model spec but run.py does not apply them on its own. Without them a container cannot answer tool_choice: "auto" requests, and coding agents get empty replies.
• Weights. It reuses a pre-seeded volume when one matches (the --host-volume above), and otherwise the Hugging Face cache (--host-hf-cache). See Magic of symlinks and Weights, Caches and Volumes.
• The checklist. It answers run.py’s interactive prompts for you and returns only when the endpoint is answering.

Running run.py Yourself

tt keeps its own pinned copy of tt-inference-server under ~/.local/share/tenstorrent/tools/tt-inference-server-<version>/. You can call it directly. Two reasons to: pinning a server to specific chips, and passing an option tt serve does not expose. These details were verified on a QB2:

cd ~/.local/share/tenstorrent/tools/tt-inference-server-v0.25.0/src   # run.py reads ./VERSION
export HF_TOKEN=$(cat ~/.cache/huggingface/token)
export MODEL_SOURCE=huggingface     # otherwise it asks "How do you want to provide a model?"
export JWT_SECRET=unused            # otherwise it asks for a secret, even with --no-auth

../venv/bin/python run.py \
  --model Llama-3.1-8B-Instruct --workflow server --docker-server --no-auth \
  --host-hf-cache ~/.cache/huggingface \
  --device p300 --device-id 2,3 --service-port 20000 --bind-host 127.0.0.1 \
  --vllm-override-args '{"enable-auto-tool-choice": true, "tool-call-parser": "llama3_json"}'

--no-auth turns authentication off, and run.py publishes the port on every network interface by default (--bind-host 0.0.0.0 in v0.25.0). --bind-host 127.0.0.1 keeps this server reachable from the QB2 itself only; reach it from another machine with an SSH tunnel. Leave auth on, and drop --no-auth, before you expose a server to a network. (The flag is in v0.25.0’s run.py; we added it to this command after the test run described above.)

Without MODEL_SOURCE and JWT_SECRET, run from a script or an SSH session, run.py stops with EOFError at a prompt nobody can answer. Without the --vllm-override-args line the server starts but cannot do tool calling.

Pinning a server to specific chips

tt serve gives the container every /dev/tenstorrent node. run.py --device-id 2,3 gives it only chips 2 and 3, which lets two servers share the machine. In our test, Llama-3.1-8B served on chips 2 and 3 (one P300 board) while Qwen3-Coder-Next kept running on chips 0 and 1; the Opening user mode device driver and Fabric initialized on 2 devices lines in the server log confirm which chips it opened. Which board holds which chip numbers is in tt device status (the # column) and tt device info (the board_id row). If you share the machine with other people or agents, take a lease first (gozer) so two servers never claim the same chips.

Verifying the Server

Once the server reports ready, confirm it’s working:

# List available models — start here, and use the id it prints
curl -s http://localhost:8000/v1/models | python3 -m json.tool

# First chat completion
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain tensor parallelism in one sentence."}
    ],
    "max_tokens": 120
  }' | python3 -m json.tool

The response JSON has the generated text at choices[0].message.content. If you get a connection refused, the server isn’t ready yet — give it another 30 seconds.

OpenAI Python SDK

The server is API-compatible with OpenAI’s client library. Point base_url at localhost:8000 and set api_key to any non-empty string — the server ignores it.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-checked"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "What is the Tenstorrent NOC fabric?"}
    ],
    max_tokens=256,
    temperature=0.7
)

print(response.choices[0].message.content)

This is the integration point for any application that already talks to OpenAI. Change the base URL, change the model name, and the rest of the code runs unchanged.

Streaming Responses

For applications that need to show text as it generates — chat interfaces, interactive tools — use the streaming mode:

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Describe continuous batching."}],
    stream=True
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

print()  # newline at end

Each chunk arrives as a server-sent event; the OpenAI SDK unwraps them into delta objects. The pattern is identical to streaming from api.openai.com — because it’s the same API.

Continuous Batching

This is one of the QB2’s practical advantages in production. vLLM’s continuous batching algorithm fills the KV-cache space as requests arrive, packing multiple users’ decode steps into the same chip invocation. You’re not running one request at a time — the server is interleaving decode steps from multiple concurrent clients across every chip cycle.

For single-user interactive work, this doesn’t matter. For serving a team, an API endpoint, or anything with concurrent load, it means the throughput numbers scale with parallelism rather than collapsing under it. A second concurrent user adds very little overhead up to the throughput ceiling of the chip.

Port Map

Keep these ports clear. Other services on the QB2 use them.

If port 8000 is already in use when you try to start vLLM, check for a running tt-studio or tt-inference-server instance first.

Remote Access via SSH Port Forward

The vLLM server listens on localhost only by default. To access it from another machine on your network — or from your laptop over SSH — use port forwarding:

# Run this on your laptop / remote machine
# Forwards your local port 8000 to the QB2's port 8000
ssh -L 8000:localhost:8000 your-user@your-qb2-hostname

# Now on your laptop, this works:
curl http://localhost:8000/v1/models

Keep the SSH session open while you use the forwarded port. For a persistent setup, look at autossh or tmux to keep the tunnel alive.

Multi-Chip: Using All Four Chips

On the managed path you don’t choose the mesh — run.py derives it per model from its spec. For a 70B across all four Blackhole chips that is just:

tt serve Llama-3.3-70B-Instruct --device p300x2

The model weights distribute across all four chips’ DRAM, and the KV cache is allocated per chip across the mesh. From the client’s perspective the API is identical — same URL, same request format.

Setting the mesh by hand is a direct-vLLM concern — MESH_DEVICE, its accepted names, and the case difference from --tt-device are covered in Build Your Own vLLM Environment. One thing carries over to this path regardless:

When Startup Stalls

Startup has long silent stretches, so “stuck” and “working” look alike in the log. Two quiet phases are normal: building rotary-embedding matrices, which prints nothing at all, and the warmup prefill sweeps that follow weight loading. Warmup also emits TT_FATAL: Only TILE layout is supported for BFLOAT8_B dtype! at critical level, sometimes hundreds of times — it is caught internally and startup continues. Worth knowing if you script a log watcher, since grepping for FATAL will trip on it.

A real device hang looks different: the log stops, CPU sits near 100%, and RSS stops moving entirely. Real work moves RSS. Confirm with a native stack dump:

uv tool install py-spy   # isolated install — not into the vLLM venv you're diagnosing
py-spy dump --native --pid "$(pgrep -n -f 'VLLM::EngineCore')"

-n on pgrep matters if more than one matching process is running: py-spy --pid takes exactly one PID, and an unfiltered pgrep -f can print several.

A hang in the command queue is unmistakable — pthread_cond_wait under FDMeshCommandQueue::wait_for_outstanding_reads, meaning the device never acknowledged a write. --native matters: the plain Python stack only shows ttnn.from_torch and looks like ordinary work.

The usual cause is stale state from a run that did not exit cleanly, and a reset clears it:

sudo lsof -w /dev/tenstorrent/*   # must print nothing before you continue
tt-smi -r                          # reset all boards

lsof exits 1 when nothing holds the devices, so for this check non-zero is the answer you want.

Going further

Two things this chapter deliberately leaves out, because each is a project rather than a step:

Next: Performance Tuning →

Performance Tuning

Running a model is table stakes. Knowing how to interpret what the hardware is doing while it runs — and what to change when the numbers don’t look right — is what separates production-ready deployments from experiments that worked once and then didn’t.

tt-toplike: Real-Time Hardware View

tt-toplike is htop for your Blackhole chips. Install it once, run it alongside inference, watch what the hardware does.

# Install from the Tenstorrent apt repository (set up by tt-installer)
sudo apt update && sudo apt install tt-toplike
# "Repository is not signed"? Re-download the signing key, then retry:
#   sudo curl -fsSL -o /etc/apt/keyrings/tt-pkg-key.asc https://ppa.tenstorrent.com/tt-pkg-key.asc
# No repository? Grab the .deb from https://github.com/tenstorrent/tt-toplike/releases
# and install it with `sudo dpkg -i tt-toplike_*.deb` — or via cargo: cargo install tt-toplike

# Launch in arcade mode — real-time chip visualization
tt-toplike --mode arcade

# Other modes worth knowing
tt-toplike --mode starfield    # particle visualization of chip activity
tt-toplike --mode flow         # DRAM bandwidth-focused display
tt-toplike --mode normal       # table mode, scriptable output

The arcade mode is not decoration. The activity pattern it shows maps directly to what the chips are computing — dense uniform patterns during prefill, pulsing DRAM-heavy patterns during decode. Once you can read those patterns, you can tell at a glance whether a run is behaving as expected.

Full documentation: docs.tenstorrent.com/tt-toplike →

tt-smi: Snapshot Mode for Scripted Monitoring

While vLLM runs, pull hardware metrics in a second terminal:

# Snapshot mode — outputs JSON, no TUI
tt-smi -s

# Pretty-print it
tt-smi -s | python3 -m json.tool

For repeated polling, a small script is more reliable than cramming this into a one-liner (also: asic_temperature/power/aiclk are nested under telemetry, not flat fields on each device entry — see Is This Thing On? — and there’s no device_id field to print, so enumerate the list instead). Save as ~/tt-scratchpad/tt-smi-watch.py:

import json, sys

data = json.load(sys.stdin)
for i, d in enumerate(data["device_info"]):
    t = d["telemetry"]
    print(f"Chip {i}: {t['asic_temperature']}°C  {t['power']}W  aiclk={t['aiclk']}MHz")

Then poll every 2 seconds:

watch -n 2 'tt-smi -s | python3 ~/tt-scratchpad/tt-smi-watch.py'

The JSON field names you care about per chip: asic_temperature, power, aiclk, current (utilization).

What Good Numbers Look Like

These are reference ranges for a healthy QB2 under inference load. Exact values vary by model, batch size, and ambient conditions.

If aiclk is consistently below 800 MHz under load, the chip may be thermal-throttling. If temperatures exceed 85°C, check airflow — the QB2 case needs clearance on all sides.

Prefill vs. Decode: Two Different Hardware Modes

Transformer inference has two fundamentally different phases, and they stress the hardware differently.

Prefill processes the entire input prompt in parallel. Every token in your system prompt and user message gets computed at once, across all layers. This phase is compute-bound — the Tensix cores are running at full utilization, arithmetic throughput is the limiting factor. In tt-toplike arcade mode, you see dense uniform activity across the chip grid.

Decode generates one token at a time, autoregressively. Each step uses the full KV-cache (which grows with sequence length) but only computes one new output token. This phase is memory-bandwidth-bound — the cores are bottlenecked on loading the KV-cache from DRAM into L1 for each step, not on arithmetic. In tt-toplike flow mode, you see DRAM read bandwidth spiking with each token.

Prefill: compute-bound, all cores lit. Decode: memory-bound, DRAM rows pulsing.

Understanding this split matters for workload design. Long prompts mean long prefill (slow time-to-first-token). Short prompts with long generated outputs mean fast prefill but decode throughput determines how fast the text appears.

Batch Size and Throughput

Larger batches improve throughput at the cost of time-to-first-token. In vLLM’s continuous batching model, “batch size” isn’t something you explicitly set — the scheduler fills decode slots dynamically as they become available.

You can influence this with --max-num-seqs (maximum concurrent sequences) when starting the server:

On the managed path, pass it straight through to the server:

python3 ~/.local/lib/tt-inference-server/run.py \
  --model Llama-3.1-8B-Instruct \
  --workflow server --tt-device p300x2 --docker-server \
  --max-num-seqs 16

Driving vllm serve yourself (from your own vLLM venv — it is not installed on the host; see vLLM on QB2):

source ~/.venvs/vllm-tt/bin/activate
export TT_METAL_ARCH_NAME=blackhole
export MESH_DEVICE=P300x2            # all four chips; avoid the two-chip P300 mesh
export HF_MODEL=~/models/Llama-3.1-8B-Instruct

vllm serve "$HF_MODEL" \
  --max-num-seqs 16 \
  --port 8000

For single-user interactive use, lower values (4–8) reduce first-token latency. For batch workloads or multi-user serving, higher values improve throughput — but there is no single safe ceiling that applies across every model on a QB2.

TTNN Performance Mode

For direct TTNN code (not vLLM), TTNN exposes a performance mode hint. The exact API is subject to change — check the current TTNN documentation for the precise call — but the concept is a mode flag that tells the runtime to prefer aggressive optimization over compilation speed.

# Check the TTNN docs for the current API — this is illustrative
# The concept: trade slower JIT compilation for faster inference
import ttnn

# Example — verify the exact call in the current TTNN release
# ttnn.set_performance_mode(ttnn.PerformanceMode.AGGRESSIVE)

In vLLM, performance optimization happens at the model-loading stage. The compilation step at first run is when the kernels are tuned.

Scaling Across Chips

Chips are added by widening the mesh, not with --tensor-parallel-size — the Tenstorrent platform rejects tensor and pipeline parallel outright. On a QB2 that means MESH_DEVICE=P300x2 for all four chips, or MESH_DEVICE=P150 for one. The two-chip mesh in between (MESH_DEVICE=P300, a single card) has failed fabric bring-up reproducibly on our hardware — so “half the box” is not a tuning knob you can lean on. See vLLM on QB2 for the error it produces.

Within the mesh, a model’s weights and attention heads distribute across the chips, and the chips coordinate activations over their Ethernet cores directly, without routing through the CPU.

This matters for scaling intuition: four chips does not mean 4x throughput, because partial activations have to cross chip boundaries at each layer. What you reliably gain is 4x the memory pool — fitting a model that would not fit on one chip — plus a real but sublinear throughput improvement from the compute scale-out.

Profiling with TTNN

For direct TTNN code (not the vLLM server), ttnn.experimental.profiler can emit per-op timing data. This is the Blackhole equivalent of torch.profiler — it shows you which ops are taking the most cycles and where the bottlenecks are.

# Illustrative — check current TTNN docs for the exact profiler API
import ttnn

with ttnn.experimental.profiler.profile():
    result = ttnn.matmul(a, b)

# profiler output goes to a file; inspect with tt-vscode-toolkit perf viewer

The OptimizerFW tool via tt-forge provides higher-level optimization passes that can analyze a full PyTorch model graph and suggest kernel-level improvements.

Next: Going Deeper →

Going Deeper

You’ve rerouted the mental model, picked a model that fits the hardware, stood up a production inference server, and watched the hardware breathe through prefill and decode. That’s the Run & build track done. What it opens up is considerably larger.

Interactive Lessons in tt-vscode-toolkit

The VS Code extension ships lessons that run against your QB2 directly — not simulated, not mocked. Real inference, real hardware feedback, real timing numbers. Each lesson is a structured walkthrough with code cells you execute against the machine.

Custom Training on QB2

Everything so far has been inference — download weights, serve them, measure them. The QB2 also builds and trains models, via tt-train (the ttml library), and the tt-vscode-toolkit lesson set for this has grown into a full arc, not a single lesson:

The multi-device lesson is the one worth calling out on a QB2 specifically: it documents near-linear data-parallel scaling across all four chips, which is exactly the hardware you have.

Three Things to Try Next

Run Llama-3.3-70B with all four chips. The largest model QB2 officially supports: 70 billion parameters, 128K context, tensor-parallel across all four Blackhole chips. The lesson has the exact Docker command, prerequisites checklist, and a variant for the DeepSeek-R1 reasoning model that uses the same infrastructure. Download the weights (140 GB — plan ahead), start the server, and run a request that would be genuinely difficult to answer. Watch tt-smi -s while it generates — the hardware doing real work looks different from the hardware doing toy work.

Build a Python application against the OpenAI-compatible API. The server is running on localhost:8000. The OpenAI SDK works unchanged. Take something you’ve built against api.openai.com — a chatbot, a summarizer, a classification pipeline — and point it at your QB2. Measure the latency. Compare the cost per token. This is where the practical value of local inference becomes tangible rather than theoretical.

Take the Tinker track. The Run & build track ends at the TTNN surface. The Tinker track goes below it: Metalium kernels, NoC data movement, dispatch programming, the full architecture exposure. If you’ve ever wanted to understand how a matmul actually runs on silicon — not the math, the execution — that track is the path.

Community and Further Reading

Choose Your Next Track

You ran serious inference on serious hardware and you understand why it works the way it does. That’s a meaningful thing to know. The QB2 is a beginning, and you’ve got your bearings.

← Performance Tuning | TT-Forge: Compile Anything →

TT-Forge: Compile Anything

vLLM is a curated serving runtime. It knows exactly which models it supports, it has them tuned and tested, and it presents a clean HTTP API for inference. Tremendous for what it does. But it covers a specific list.

TT-Forge is the other gate. You bring the model — any PyTorch nn.Module, any JAX function, any ONNX export — and the compiler traces it, lowers it to Tensix operations, and hands back something that runs on your QB2 hardware. One call. Hardware execution. No server, no model list to consult.

If vLLM is the highway, TT-Forge is the ability to go anywhere.

Before You Begin — Install Forge

Forge is not part of a default tt-installer run. tt-installer sets up the base — driver, firmware, hugepages, and the ~/.tenstorrent-venv environment that holds tt-smi and tt-flash. Forge itself you install as a pip wheel from Tenstorrent’s package index. That’s how the TT-Forge docs want you to do it — not a container wrapper, not a 45-minute source build.

(tt-installer can give you Forge as a container instead, via --install-forge-container, which writes a tt-forge wrapper to ~/.local/bin/. It’s off unless you ask for it, and it’s a separate path from the wheel below — pick one.)

First confirm the base is ready (Ubuntu 24.04, Python 3.12):

tt-smi   # should show the System Management Interface

Then make Forge a venv of its own. Don’t install it into ~/.tenstorrent-venv: Forge pins its own torch / torch-xla stack, and resolving that on top of your hardware tooling means a bad Forge install is also a broken tt-smi.

python3 -m venv ~/.venvs/forge
source ~/.venvs/forge/bin/activate

Then install the frontend for your framework:

PyTorch & JAX — TT-XLA (the primary frontend):

pip install pjrt-plugin-tt --extra-index-url https://pypi.eng.aws.tenstorrent.com/
tt-forge-install     # pulls in any missing system dependencies

pip install tt-forge is the convenience meta-package that wraps the same thing.

pip install --no-deps torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cpu

ONNX / TensorFlow / PaddlePaddle — TT-Forge-ONNX (single-chip only):

sudo apt-get install -y libgomp1 libmpc3
uv pip install tt_forge_onnx tt_tvm --extra-index-url https://pypi.eng.aws.tenstorrent.com/

The Compilation Paths

Two frontends cover every framework. Both lower to the same TT-MLIR compiler and the same Tensix backend — the framework you start from doesn’t change where you land.

TT-XLA is the primary frontend and the one to reach for first: it takes both PyTorch (through torch-xla) and JAX (through jax.jit), and it’s the only path that scales across multiple chips. TT-Forge-ONNX is the TVM-based route for models that arrive as ONNX, TensorFlow, or PaddlePaddle graphs, and it’s single-chip only.

Your First Compile

ResNet-50 is the right first target — well-understood architecture, small enough to compile fast. This is the canonical PyTorch quickstart from the TT-Forge docs:

import torch
import torch_xla.core.xla_model as xm
import torch_xla.runtime as xr
import tt_torch  # registers "tt" as a torch.compile backend
from torchvision.models import resnet50, ResNet50_Weights

# Point PyTorch/XLA at the Tenstorrent device
xr.set_device_type("TT")
device = xm.xla_device()

# Load ResNet-50 in bfloat16 — Blackhole's native float format
model = resnet50(weights=ResNet50_Weights.DEFAULT).to(torch.bfloat16).eval()

# Compile for Tensix and move the compiled model onto the device
compiled_model = torch.compile(model, backend="tt").to(device)

# Run inference on hardware
input_tensor = torch.randn(1, 3, 224, 224, dtype=torch.bfloat16).to(device)
with torch.no_grad():
    output = compiled_model(input_tensor)

print(output.cpu().argmax(dim=-1).item())   # predicted ImageNet class

What torch.compile(model, backend="tt") does: torch-xla traces the model into a StableHLO graph, the TT-MLIR pipeline lowers that graph to Tensix kernels, and you get back a callable that dispatches to hardware. The first compilation is slow (tens of seconds for ResNet, longer for large models). Subsequent calls with the same input shapes hit a compiled cache and run fast.

Loading in torch.bfloat16 matters: Blackhole is bfloat16-native, so it gives you full hardware throughput. Float32 works, but leaves performance on the table.

Here is the chip view during compilation and inference:

The compile step dispatches weight loads from DRAM then fans work across the Tensix grid.

The tt-forge-models Zoo

Writing model-loading boilerplate for hundreds of architectures is tedious. Somebody already did it. tt-forge-models is the standardized model zoo for TT-Forge — 800+ model variants tested in CI, every one exposing the same ModelLoader interface and loadable in two lines.

The repo lives at ~/code/tt-forge-models and on GitHub at tenstorrent/tt-forge-models.

Directory structure is not uniformly flat — verified against the real repo. Models with one obvious task (ResNet, CLIP, DeiT) go straight to the framework directory; models that serve several downstream tasks (BERT, DINOv2, Llama) add a task-type directory in between:

tt-forge-models/
  resnet/
    pytorch/
      loader.py       # ModelLoader class
  bert/
    masked_lm/
      pytorch/
        loader.py
      jax/
        loader.py
    question_answering/
      pytorch/
        loader.py
    sequence_classification/
      pytorch/
        loader.py
  clip/
    pytorch/
      loader.py
  dinov2/
    image_classification/
      pytorch/
        loader.py
      jax/
        loader.py       # Flax variant
  llama/
    causal_lm/
      pytorch/
        loader.py

If an import at the depth shown above 404s, that’s why — check the task-type directories for the model you want rather than assuming the flat pattern.

Every loader.py exports a ModelLoader class with two instance methods (not static — ModelLoader.load_model(...) raises TypeError: missing 1 required positional argument: 'self', confirmed live; instantiate first). load_model() returns a standard PyTorch nn.Module and load_inputs() returns matching sample tensors — so you compile them exactly like any other model:

import torch, tt_torch
from third_party.tt_forge_models.bert.masked_lm.pytorch import ModelLoader

# Load the pretrained model and representative inputs
loader = ModelLoader()
model = loader.load_model(dtype_override=torch.bfloat16)
inputs = loader.load_inputs(dtype_override=torch.bfloat16)

# compile for Tensix and run — same torch.compile path as before
compiled = torch.compile(model, backend="tt").to(device)
output = compiled(inputs.to(device))

Five models worth knowing immediately:

Models not on this table: BLOOM, GPT-2, LLaMA, YOLOv4, BEiT, and 190+ more. Browse the full zoo in the forge-models-zoo lesson.

JAX and TT-XLA

For JAX and Flax models, the compilation path uses TT-XLA. Import pjrt_plugin_tt and the TT hardware backend registers automatically:

import jax
import jax.numpy as jnp
import pjrt_plugin_tt  # registers TT hardware as the XLA backend

# Any JAX function — jax.jit traces it and compiles to TT hardware
@jax.jit
def predict(params, x):
    return model.apply(params, x)

output = predict(params, batch)

The pjrt_plugin_tt import is the entire setup. After that, jax.jit compiles to Tensix cores instead of CPU or GPU. Flax transformer models slot directly into this pattern — load the model, load weights, wrap model.apply in jax.jit, run inference.

Full walkthrough: TT-XLA / JAX lesson.

Compiletron: The Expedition Game

Someone at Tenstorrent decided the best way to stress-test the compiler stack across hundreds of model architectures was to make it a roguelike game. They were right.

tt-forge-compiletron is a model-compilation expedition game (it lives at tenstorrent/tt-forge-compiletron). You launch expeditions into the model zoo. Each expedition compiles a different model. The chip runs the compilation. You score points. You build a bestiary.

Set it up, then start it:

cd ~/code/tt-forge-compiletron
bash scripts/install.sh          # installs forge venv, XLA venv, clones tt-forge-models
python3 expedition.py run --tui

A three-screen Textual TUI opens. The countdown is four seconds — then the expedition begins automatically.

The bestiary (data/bestiary.json) is a persistent record of every model you’ve successfully compiled. Base score per compile: 200 points. First time you compile a model, ever: multiplier of 5, making it 1,000 points. Freshness and rarity bonuses stack on top. The scoring structure incentivizes breadth: you gain more by compiling 10 new models than by recompiling the same model 10 times.

Compiletron supports both compiler backends from a single interface — verified against the actual --backend argument, which has four choices, not two:

Side quests activate when the mesh is busy with a large model compilation. Idle chips get assigned fast curated models to compile in parallel, keeping hardware utilization high and points accumulating while you wait. The game manages chip allocation automatically.

For unattended recording (VHS demos, overnight compilation runs), use --auto-quit SECS — verified against the actual argument: it’s a countdown in seconds after the summary screen appears, not a count of models compiled:

python3 expedition.py run --tui --auto-quit 30

The expedition runs its normal course; once the summary screen shows, the game waits 30 seconds and then quits on its own — bestiary saved, score written to disk — instead of waiting on a keypress.

All four chips busy — main expedition on Chip 0, three side quests running simultaneously.

Compiletron was built to find compiler bugs. It works through that bestiary systematically, surfacing edge cases in graph lowering and kernel generation that sequential targeted testing would miss. Every expedition you run contributes data to that effort. Points are real. The bestiary is real. And the compiler gets better.

← Performance Tuning | Going Deeper →

The TT-Metal Architecture

Before you write a single line of kernel code, you should understand what you’re writing it for. The Blackhole chip is not a GPU wearing a different nametag. The memory model is different. The execution model is different. The abstraction layers are deliberately transparent. Once you see the architecture clearly, the API choices stop being arbitrary and start being obvious.

The Stack From Top to Bottom

Four layers sit between your Python and the chip. Each layer is real and each layer compiles:

TT-Lang        →  Python DSL, looks like Python, compiles to assembly
TTNN           →  Python ops, tensor API, calls into Metalium
TT-Metalium    →  C++ kernel API, explicit data movement, JIT compile
Kernel Driver  →  firmware, PCIe dispatch, ring buffers

You can enter this stack at any level. TTNN is the right entry point for standard ops. TT-Lang is the right entry point when you need a custom pattern and want AI-assisted development. Metalium is where you go when the abstraction has to disappear.

Blackhole Grid Anatomy

The Blackhole chip is a 17-column by 12-row network-on-chip (NoC) grid. Every cell in that grid is a node. Not every node is a compute core. The grid has four distinct zones:

Tensix cores — columns 1-7 and 9-15, rows 1-10. One hundred and forty physical tiles, of which 120 are enabled on QB2’s chips (two columns are harvested). These are the compute nodes. Each Tensix core is itself a small computer.

DRAM controllers — rows 0 and 11, running the full width of the chip. 32 GB of GDDR6 per chip (64 GB per p300c card). The chip’s main memory lives here, physically along the chip edges, close to the NoC’s routing paths.

ETH ports — column 0 and column 16. These connect chips together. On a QB2’s four Blackhole chips, the ETH ports form the chip-to-chip fabric used by CreateDevices when you open a multi-chip mesh.

PCIe interface — column 8, the center column. Every command from your Python application crosses here. ttnn.open_device(0) sends a dispatch message through this column.

One Blackhole chip. Four of these — on two p300c cards — live in your QB2.

Inside a Tensix Core

Zoom in on any one of those Tensix nodes. Each Tensix core contains:

• RISC-V control processor — a small general-purpose CPU that executes your kernel logic
• Matrix engine (FPU) — hardware-accelerated matrix multiply and elementwise ops; this is what makes it fast
• Register tile files — SRCA, SRCB, and DST registers that hold 32×32 element tiles during computation
• L1 SRAM — fast on-core scratchpad memory; your kernel reads data here before the FPU touches it
• Two NoC endpoints — one for reads (inbound), one for writes (outbound); both can operate independently and concurrently

The L1 SRAM is crucial. Moving data from DRAM to a Tensix core’s L1 is an explicit operation you control. Nothing is cached automatically. This sounds like a burden and becomes a superpower: you know exactly where every byte is.

The Three-Kernel Model

Every Metalium operation on a Tensix core involves three co-running kernels. All three run on the same core, concurrently:

• Data-movement-reader (BRISC) — reads tiles from DRAM or another core’s L1 into this core’s L1 via the read NoC endpoint
• Compute — pops tiles from L1 into the SRCA/SRCB registers, runs the matrix engine, writes results to DST, pushes results back to L1
• Data-movement-writer (NCRISC) — takes finished tiles from L1 and sends them to DRAM or another core’s L1 via the write NoC endpoint

Tiles: The Native Unit

TTNN doesn’t think in terms of individual floats or rows. It thinks in 32×32 tiles. A tensor of shape (64, 64) becomes 4 tiles of shape (32, 32). The tile format — BFP8, BFP16, or FP32 — is set when you create a tensor:

import ttnn, torch

device = ttnn.open_device(device_id=0)

# Create a tensor — TTNN tiles it automatically on device transfer
t = torch.randn(64, 64)
t_tt = ttnn.from_torch(t, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)

# t_tt is now four 32x32 BF16 tiles distributed in the chip's DRAM
print(t_tt.shape)   # torch.Size([64, 64])
print(t_tt.dtype)   # bfloat16

ttnn.close_device(device)

The 32×32 tile size is not adjustable — it is the hardware’s register file size. Every operation on the matrix engine processes one tile at a time. Kernels are written to process tiles, readers fetch tiles, writers send tiles.

The NoC Fabric

The two-dimensional mesh NoC lets any core read from or write to any other core’s L1, or any DRAM bank, by address. There is no coherence protocol, no cache hierarchy. You own the data movement. The routing is deterministic and the bandwidth is high — but contention is possible, which is why the profiler shows per-link NoC traffic.

For a single-chip operation, you’re moving tiles from DRAM row-0 or row-11 nodes, across the mesh, to your compute cores’ L1. For a multi-chip operation via CreateDevices, tiles cross the ETH columns at the chip edges and appear at another chip’s ETH columns before continuing across that chip’s mesh.

A Minimal TTNN Example

This is the entire open-device-matmul-close pattern, which you’ll recognize from every tutorial:

import ttnn, torch

# Open chip 0
device = ttnn.open_device(device_id=0)

# Move data onto the chip
a = ttnn.from_torch(torch.randn(64, 64), dtype=ttnn.bfloat16,
                    layout=ttnn.TILE_LAYOUT, device=device)
b = ttnn.from_torch(torch.randn(64, 64), dtype=ttnn.bfloat16,
                    layout=ttnn.TILE_LAYOUT, device=device)

# Dispatch the matmul kernel — compiles JIT on first run
c = ttnn.matmul(a, b)

# Pull result back to CPU
result = ttnn.to_torch(c)
print(result.shape)

ttnn.close_device(device)

Nothing in this example is magic. Each step maps to a real chip operation: the from_torch calls dispatch DMA transfers through the PCIe column to DRAM; matmul dispatches reader/compute/writer kernels to a set of Tensix cores; to_torch moves the result tiles back through PCIe to host RAM.

Next: Your First Kernel →

Your First Kernel

Reading about architecture is preparation. Writing code is proof. This chapter takes you from zero to a dispatched, JIT-compiled, hardware-executed kernel, written in about fifteen lines of Python. You don’t need to clone anything or build anything — just write one short script and run it.

Setting Up the Environment

Everything runs inside the TT-Metalium container. There’s no venv to activate and no ~/tt-metal on the host — the tt-metalium wrapper is the door:

tt-metalium

You’re now in a shell inside the container, with your home directory mounted at /home/user (so ~ still means your files). TTNN is on the default interpreter — check before you export anything:

which python3            # → /opt/venv/bin/python3
echo "$TT_METAL_ARCH_NAME"

TT_METAL_ARCH_NAME must be blackhole. Without it the runtime defaults to Wormhole and dispatches incorrect kernel variants; the QB2 has Blackhole chips. Set it if that last command printed nothing (it isn’t baked into the image — confirmed empty in a real container):

export TT_METAL_ARCH_NAME=blackhole

The container’s ttnn is a runtime-only install — no torch, and no pip to fetch it with (only uv, and ensurepip). Bootstrap torch once per container session; the CPU wheel is what you want here, since TTNN — not torch — is doing the actual chip work, and the default index otherwise pulls a multi-GB CUDA build you’ll never use on this hardware:

uv pip install --python /opt/venv/bin/python3 torch --index-url https://download.pytorch.org/whl/cpu

Your First Run: Tensor Addition

Write the script yourself — it’s short, and typing it once is worth more than running someone else’s copy. Save this as ~/tt-scratchpad/ttnn_add_tensors.py:

import ttnn, torch

# 1. Open the chip — handshake through PCIe column 8
device = ttnn.open_device(device_id=0)

# 2. Create two tensors on the host
a = torch.randn(32, 32)
b = torch.randn(32, 32)

# 3. Move both to the chip (DMA transfer to DRAM)
a_tt = ttnn.from_torch(a, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)
b_tt = ttnn.from_torch(b, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT, device=device)

# 4. Run the elementwise add kernel
c_tt = ttnn.add(a_tt, b_tt)

# 5. Pull the result back to host RAM
c = ttnn.to_torch(c_tt)

# 6. Close the device — flushes all pending work and releases the chip
ttnn.close_device(device)

print("Result shape:", c.shape)

Then run it:

python3 ~/tt-scratchpad/ttnn_add_tensors.py

First run: expect 30 to 60 seconds of compile time before any output. This is the JIT compiler building the addition kernel from LLVM IR down to Tensix assembly, then writing the binary to the kernel cache.

Second run: fast. The cache is warm. Recompilation only happens when kernel parameters change.

Tensors Become Tiles

A key conceptual shift: TTNN does not operate on individual elements. It operates on 32×32 tiles. When you pass a (32, 32) tensor, that’s one tile. When you pass a (64, 64) tensor, that becomes four tiles.

The tile transformation happens automatically during from_torch with layout=ttnn.TILE_LAYOUT. You can inspect the layout:

print(a_tt.layout)   # Layout.TILE
print(a_tt.dtype)    # DataType.BFLOAT16
print(a_tt.shape)    # Shape([32, 32])

Larger tensors spread across more tiles, and those tiles get dispatched to more cores concurrently. A (512, 512) tensor becomes 256 tiles; the dispatch system assigns each tile to a Tensix core. The parallelism is automatic at the tile level.

Reader → L1 → FPU → L1 → writer. The three-kernel pipeline at work.

Understanding JIT Compilation

The first run is slow for a specific reason: Metalium compiles kernels just-in-time. Here’s what happens during that 60-second wait:

1. TTNN resolves the op’s dtype, shape, and memory layout to a kernel variant
2. The kernel variant (C++ source) is templated with those parameters
3. LLVM compiles C++ to RISC-V assembly for the Tensix host processor
4. The Tensix-specific FPU operations are lowered to assembly for the matrix engine
5. Both binaries are written to the kernel cache at ~/.cache/ttnn/

Subsequent calls with the same parameters skip all of this. The compiled binary is reused. If you change tensor shapes or dtypes, partial recompilation fires for the changed variants only.

Matmul: Putting the FPU to Real Work

Elementwise addition barely exercises the matrix engine. Matrix multiplication does. The call is minimal:

device = ttnn.open_device(device_id=0)

a = ttnn.from_torch(torch.randn(256, 256), dtype=ttnn.bfloat16,
                    layout=ttnn.TILE_LAYOUT, device=device)
b = ttnn.from_torch(torch.randn(256, 256), dtype=ttnn.bfloat16,
                    layout=ttnn.TILE_LAYOUT, device=device)

c = ttnn.matmul(a, b)
result = ttnn.to_torch(c)
print(result.shape)   # torch.Size([256, 256])

ttnn.close_device(device)

A (256, 256) matmul is 64 output tiles. The dispatch system maps those 64 output tiles to 64 compute cores, running in parallel. The reader for each core fetches the relevant row tiles from A and column tiles from B. The FPU accumulates. The writer ships results to DRAM. All of this runs concurrently across 64 Tensix cores.

Keeping Tensors in L1

By default, tensors live in DRAM. Every op reads from DRAM and writes results to DRAM. For chained operations, this incurs unnecessary round-trips. You can pin a tensor to L1 memory instead:

# Keep the tensor in L1 between ops — avoid the DRAM round-trip
a_l1 = ttnn.to_memory_config(a, ttnn.L1_MEMORY_CONFIG)
b_l1 = ttnn.to_memory_config(b, ttnn.L1_MEMORY_CONFIG)

c = ttnn.matmul(a_l1, b_l1)

This works when the tensor fits in L1. For large tensors it won’t — DRAM is 32 GB per chip, L1 is small per-core scratchpad. Use L1 pinning for intermediate results in tight compute loops.

Kernel Fusion: Chaining Ops

TTNN supports kernel fusion when you chain ops. The compiler detects the dependency and merges compute kernels:

# These three ops may fuse into a single kernel dispatch
c = ttnn.relu(ttnn.matmul(a, b))

Whether fusion fires depends on shape compatibility and the current kernel fusion rules. When it fires, the reader runs once, the fused compute kernel does matmul + relu on each tile, and the writer runs once. When it doesn’t fire, each op dispatches separately. The profiler tells you which happened (see Chapter 4).

Next: TT-Lang Introduction →

TT-Lang Introduction

TTNN covers a large territory of standard ops — matmul, attention, layernorm, convolution. But ML research moves faster than op libraries. The moment you want a fusion pattern that TTNN doesn’t expose, a non-standard attention variant, a custom activation function with a specific numerical property, you need to go lower. TT-Lang is that lower level, without requiring C++.

What TT-Lang Is

TT-Lang is a Python DSL that compiles to Tensix assembly. You write Python-like syntax with decorators that declare data-movement intent. The compiler translates that intent into reader kernels, compute kernels, and writer kernels. The three-kernel model you read about in Chapter 1 becomes the explicit structure of every TT-Lang program.

The key design principle: explicit data movement. Where TTNN hides the read/compute/write split, TT-Lang exposes it as the primary vocabulary. You declare what the reader fetches from where, what compute does to tiles in registers, what the writer sends where. No implicit sharing. No hidden transfers.

This explicitness is intentional and strategic. It makes TT-Lang programs easy for AI coding agents to generate, verify, and debug — because the spec is complete in the source code. The reader section tells you exactly what arrives. The compute section is pure math on those arrivals. The writer section is exactly what leaves. No ambiguity remains.

The Kernel Decorators

TT-Lang programs are organized around the ttl module’s decorators — verified against the real, installed package rather than assumed. ttl is not part of the factory tooling venv (~/.tenstorrent-venv holds only tt-smi/tt-flash, as elsewhere in this guide) or the TT-Metalium container (confirmed absent from both, live) — it comes from pip install tt-lang into a venv of your own, the same way the tt-lang-intro lesson sets one up (python3 -m venv ttlang-venv && ... && pip install tt-lang tt-lang-setup):

• @ttl.operation(grid=...) — the outer program; grid="auto" lets the compiler size it
• @ttl.compute() — runs on the FPU; consumes filled dataflow buffers, does the math, fills the output buffer
• @ttl.datamovement() — runs on a data-movement RISC core; there’s one decorator for both directions, not separate reader/writer ones — a function’s role (producer vs. consumer) comes from whether it calls .reserve() (fill a slot) or .wait() (drain a slot) on a given buffer, not from its decorator

A minimal vector addition kernel in TT-Lang looks like this:

import ttl
import ttnn

TILE_SIZE = 32

@ttl.operation(grid="auto")
def eltwise_add(a_in: ttnn.Tensor, b_in: ttnn.Tensor, out: ttnn.Tensor) -> None:
    row_tiles = a_in.shape[0] // TILE_SIZE
    col_tiles = a_in.shape[1] // TILE_SIZE

    # Typed dataflow buffers (DFBs) — one slot per tile, depth 2 (double-buffer)
    a_dfb = ttl.make_dataflow_buffer_like(a_in, shape=(1, 1), block_count=2)
    b_dfb = ttl.make_dataflow_buffer_like(b_in, shape=(1, 1), block_count=2)
    out_dfb = ttl.make_dataflow_buffer_like(out, shape=(1, 1), block_count=2)

    @ttl.compute()
    def compute():
        for row in range(row_tiles):
            for col in range(col_tiles):
                with a_dfb.wait() as a_blk, b_dfb.wait() as b_blk, out_dfb.reserve() as o_blk:
                    o_blk.store(a_blk + b_blk)   # element-wise add in L1

    @ttl.datamovement()
    def read():
        for row in range(row_tiles):
            for col in range(col_tiles):
                with a_dfb.reserve() as a_blk, b_dfb.reserve() as b_blk:
                    ttl.copy(a_in[row:row+1, col:col+1], a_blk).wait()
                    ttl.copy(b_in[row:row+1, col:col+1], b_blk).wait()

    @ttl.datamovement()
    def write():
        for row in range(row_tiles):
            for col in range(col_tiles):
                with out_dfb.wait() as o_blk:
                    ttl.copy(o_blk, out[row:row+1, col:col+1]).wait()

Three functions, three processors, one core. They run concurrently. The dataflow buffers between them are the synchronization mechanism — reserve() blocks until a slot is free to fill, wait() blocks until a slot is filled and ready to drain. This backpressure propagation means the pipeline self-regulates. Every tile makes one DRAM read (read) and one DRAM write (write); the + happens entirely in L1, inside compute.

Single-Core Data Flow

Here is what happens at the hardware level when vector_add runs on one Tensix core:

One Tensix core running all three TT-Lang sections concurrently.

TT-Lang vs TTNN: When to Use Which

They are not competing tools. They are different entry points into the same hardware, appropriate for different problems:

The usual path: start with TTNN. When you hit a wall — a pattern that TTNN can’t express, a fusion the compiler misses, a numerical property you need to enforce — drop to TT-Lang. Write the custom section in TT-Lang, combine it with TTNN for the standard sections.

The TT-Lang Playground

You don’t need a QB2 to experiment with TT-Lang. The ttlang-sim browser-based simulator lets you write kernels, inspect the circular buffer state, and verify correctness without hardware.

For the structured lesson with exercises and a graded environment:

The lesson runs inside VS Code with the TT-VSCode Toolkit extension. It uses a local simulator so compilation is instant. After the lesson, running the same kernel on QB2 hardware is a one-line change.

Next: Profiling & Optimization →

Profiling & Optimization

A kernel that runs is not necessarily a kernel that runs well. The Blackhole chip has 120 enabled Tensix cores per chip and 480 across your four-chip QB2. If your kernel is using 12 of them, the other 468 are idle and the machine is waiting. Profiling tells you which case you’re in.

tt-toplike: Your Primary Monitoring Tool

While your kernel runs, run tt-toplike in a second terminal. It is the most direct window into what the chip is doing right now.

# Open a second terminal, then:
tt-toplike --mode starfield

In starfield mode, each star represents a chip. Brightness is proportional to power draw, which correlates with active compute. A bright, dense star field means cores are working. A dim star means most cores are idle.

Switch modes for different views:

tt-toplike --mode flow        # NOC traffic visualization — data movement patterns
tt-toplike --mode arcade      # per-core utilization as a game-style display
tt-toplike --mode castle      # stacked bar view, useful for multi-chip comparisons

Leave flow mode running while you tune a kernel for DRAM bandwidth. The NOC traffic pattern tells you whether data is moving in a spread-out mesh pattern (good: parallel fetch from multiple DRAM banks) or a narrow column (bad: serial bottleneck).

tt-smi Snapshots

For point-in-time metrics in JSON format, use tt-smi -s. This is safe to pipe, parse, and log:

tt-smi -s

The output includes per-chip:

• aiclk — current clock frequency in MHz (Blackhole target: ~1000-1200 MHz under load)
• power — board power in watts
• asic_temperature — ASIC die temperature in °C
• voltage — core voltage

Temperature bands to know:

If tt-smi -s shows aiclk significantly below spec during a compute-heavy run, thermal throttling is occurring. Check airflow around the QB2, confirm the fans are unobstructed, and check ambient temperature.

TTNN Op Profiling

For per-operation timing at the Python level, TTNN exposes a profiler — it lives at top-level ttnn.profiler, not ttnn.experimental.profiler (that namespace doesn’t have a profiler at all), and it’s a Tracy-based interface rather than a start/stop/get_report cycle:

import ttnn

device = ttnn.open_device(device_id=0)

# ... your ops here ...
a = ttnn.from_torch(...)
b = ttnn.from_torch(...)
c = ttnn.matmul(a, b)

# Pull the perf data TTNN has already been collecting for dispatched ops
report = ttnn.profiler.get_all_programs_perf_data()

for entry in report:
    print(entry)

ttnn.close_device(device)

import time, ttnn

device = ttnn.open_device(device_id=0)

a = ttnn.from_torch(...)
b = ttnn.from_torch(...)

ttnn.synchronize_device(device)   # drain anything still in flight first
start = time.perf_counter()
c = ttnn.matmul(a, b)
ttnn.synchronize_device(device)   # block until the dispatched op actually finishes
elapsed = time.perf_counter() - start
print(f"matmul: {elapsed * 1000:.2f} ms")

ttnn.close_device(device)

Wall-clock timing without the synchronize_device calls measures dispatch latency, not execution time — TTNN queues ops asynchronously, so the Python call returns before the chip has necessarily finished the work.

What the Numbers Mean

The profiler report gives you op-level durations. Here’s how to interpret the patterns:

DRAM bandwidth bottleneck: Your matmul shows kernel dispatch time far below theoretical, but actual throughput is slow. The FPU is fast; the bottleneck is feeding it. Solution: increase L1 reuse with ttnn.to_memory_config(t, ttnn.L1_MEMORY_CONFIG) for intermediate tensors, or increase tile size so each DRAM fetch covers more compute.

Core underutilization: A large fraction of dispatch time is kernel launch overhead rather than compute. This means you have many small tiles dispatched serially. Solution: increase tensor dimensions (more tiles, more parallel cores) or batch multiple inputs together.

aiclk drops during profiling: Thermal throttling. The profiler timestamps are wall-clock accurate, but the kernel is running slower than its rated frequency. Fix the thermal situation before optimizing the kernel.

Kernel fusion mismatch: You expected relu(matmul(a, b)) to fuse but the profiler shows two separate dispatches. Check that both tensors have compatible memory configs and dtypes — fusion won’t fire across memory config mismatches.

Utilization: Sparse vs Dense

The visual version of the profiling story is utilization — how many cores are active at once:

Sparse = parallelism opportunity. Dense = machine at work.

Tiling Strategy

The 32×32 tile size is fixed by hardware. But the number of tiles in flight, and how they map to cores, is under your control through tensor dimensions and batch size.

Larger input tensors mean more tiles, more core parallelism, better amortization of kernel launch overhead. A single (32, 32) matmul uses one output core. A (1024, 1024) matmul uses 1024 output cores.

Larger batch sizes mean more independent inputs processed simultaneously. Each input in a batch can be dispatched to a different set of cores. Throughput increases linearly until you run out of cores or L1 capacity.

The tradeoff: larger batches increase first-token latency. The chip has to buffer the full batch before returning any result. For interactive latency, keep batches small. For throughput benchmarks, fill the chip.

The Optimization Loop

A practical profiling workflow for a new kernel:

1. Run the kernel once to warm the JIT cache
2. Run tt-smi -s to check thermal baseline
3. Start tt-toplike --mode flow in a second terminal
4. Run the kernel with the profiler enabled
5. Find the longest op in the profiler report
6. Check its utilization in tt-toplike — sparse means increase batch or tensor size; dense with slow throughput means DRAM bandwidth is the limit
7. Adjust one variable, re-run, compare durations

Do not optimize what you haven’t measured. The chip’s actual bottleneck is rarely the one you’d guess from first principles.

Next: Going Deep →

Going Deep

You’ve seen the architecture, dispatched a kernel, written a TT-Lang program, and read a profiler report. The surface area ahead is larger than any guide can cover in full. This chapter points you at the productive edges of that surface — the things worth building toward.

Next Lessons

These four structured lessons continue from where this track ends. They are interactive, run inside VS Code with the TT-VSCode Toolkit, and include real code you run on your QB2:

Build tt-metal from source if you’re serious about optimization. The pre-built environment is a complete API surface; the source-built environment adds per-cycle profiling, kernel modification, and the ability to send patches upstream.

Projects Worth Building

Custom attention variant in TT-Lang. Standard multi-head attention is in TTNN. But sliding window attention, linear attention, grouped-query attention with non-standard head dimensions, or a custom masking pattern — these require a TT-Lang kernel. Write it using ttl.operation with one @ttl.compute() function and two @ttl.datamovement() functions (see TT-Lang Introduction for the real, current decorator set — not @reader/@writer, which earlier drafts of this guide used and which don’t exist in the actual ttl module). One data-movement function fetches Q, K, V tile blocks into dataflow buffers; the compute function runs the tile-level matmul and softmax; the other data-movement function drains results out. The explicit tile arithmetic forces you to understand exactly what attention is doing at the register level.

Profile a TTNN cookbook pattern end-to-end. Pick any TTNN recipe from the cookbook-overview lesson — a transformer block, a convolution layer, an embedding lookup. Run it on QB2 with the profiler enabled. Find the bottleneck op. Try to shrink it: L1 memory configs, batch size changes, dtype changes. Document the before-and-after numbers. This produces a reusable reference for the specific pattern on Blackhole hardware.

Explore tt-awesome. The community kernel repository collects implementations, benchmarks, and examples contributed by the Tenstorrent community. It is the fastest way to see what other builders are doing on the same hardware. Read a kernel you didn’t write, run it, profile it, try to improve it.

tt-toplike as a Permanent Companion

Keep tt-toplike running in a tmux pane during all development. The modes give you different lenses on the same hardware:

# Split your tmux: kernel in the top pane, monitoring below
tmux split-window -v 'tt-toplike --mode flow'

When you dispatch a new kernel and the starfield or flow display changes noticeably, you know the chip responded. When you make an optimization change and the display looks the same, the optimization may not have landed the way you thought. The visual feedback is faster than reading profiler output for qualitative iteration.

The Other Tracks

This track focused on kernel writing and architecture. Two other paths cover complementary territory:

The Abstraction Goes All the Way Down

The thing worth remembering is this: every layer of the TT-Metal stack is real and reachable. TTNN is not a black box above a black box. TT-Lang compiles to assembly you can disassemble. The three-kernel model maps to three RISC-V programs running on three processors embedded in each Tensix core. The NoC is a real two-dimensional mesh and you can observe individual links. The DRAM banks are physical rows on the chip grid and you can pin data to specific banks.

Most tools hide the machine. This one doesn’t. The abstraction stack is a ladder, not a ceiling. Climb as far as the problem requires.

← Profiling & Optimization | TT-Forge: The Compiler Pipeline →

TT-Forge: The Compiler Pipeline

TTNN is a hardware API. TT-Lang is a hardware DSL. Both give you explicit control over tiles, kernels, and data movement. Both speak fluent Tensix. Both require you to think in terms of the chip’s actual execution model.

TT-Forge is a different kind of animal. It is a compiler. You give it a PyTorch model or a JAX model. It traces, lowers, compiles, and hands you back something that runs natively on Tensix cores. No tiles. No kernels. No data-movement-reader configuration. The compiler handles the translation. Your model runs.

Neither approach is better. They expose different truths about the hardware. TTNN and TT-Lang are surgical instruments. TT-Forge is a factory floor. Knowing when to pick each one is the actual skill.

The Compilation Pipeline

Two entry points converge on the same Tensix machine code. Understanding both helps you understand TT-Forge’s architecture.

The PyTorch path:

Your nn.Module → torch.compile(backend="tt") → torch-xla trace → StableHLO → TT-MLIR dialect → Tensix kernels

torch.compile(model, backend="tt") routes the model through torch-xla, which traces it into a StableHLO graph — a stable, framework-neutral IR. The TT-XLA PJRT plugin hands that StableHLO to TT-MLIR, the Tenstorrent MLIR dialect that describes ops in terms the Tensix pipeline understands. The MLIR pipeline compiles that representation all the way to Tensix machine code.

The JAX path:

Your JAX function → @jax.jit → PJRT plugin → StableHLO → TT-MLIR dialect → Tensix kernels

JAX JIT compilation traces the decorated function to StableHLO. The PJRT plugin registered by import pjrt_plugin_tt routes that representation through the same TT-MLIR pipeline. Both paths land on the same compiler backend. Both produce the same class of Tensix kernels.

The convergence is intentional — and it’s why both frameworks share one frontend, TT-XLA, built on the PJRT interface and StableHLO. Model-framework choice doesn’t divide the ecosystem: PyTorch users and JAX users compile to the same machine. (ONNX, TensorFlow, and PaddlePaddle take a separate TVM-based frontend, TT-Forge-ONNX, which still exposes the forge.compile() API and is single-chip only.)

The compile pipeline in motion. Weights arrive via PCIe, buffer in DRAM, dispatch to Tensix.

Prerequisite: Install Forge

Forge is not installed by default — a stock tt-installer run gives you the driver and the hardware-tooling environment, not Forge. The TT-Forge docs install it as a pip wheel from Tenstorrent’s package index; for the PyTorch/JAX work in this chapter that’s the TT-XLA frontend. Give it a venv of its own rather than putting it in ~/.tenstorrent-venv, which exists to hold tt-smi and tt-flash:

python3 -m venv ~/.venvs/forge
source ~/.venvs/forge/bin/activate
pip install pjrt-plugin-tt --extra-index-url https://pypi.eng.aws.tenstorrent.com/
tt-forge-install

Confirm it imports:

python3 -c "import torch_xla, tt_torch; print('TT-XLA ready')"

Building from source (tt-forge-fe, ~/tt-forge-fe/env/activate) is still an option, but the docs are clear it’s for developing the compiler itself — not a prerequisite for running models. The ML-practitioner TT-Forge chapter covers the wheel, Docker-image, and ONNX install paths in detail.

Compiling a Model in Practice

Here is a complete BEiT image classification example using the tt-forge-models zoo, compiled through TT-XLA:

import torch
import torch_xla.core.xla_model as xm
import torch_xla.runtime as xr
import tt_torch  # registers "tt" as a torch.compile backend
from third_party.tt_forge_models.beit.pytorch import ModelLoader

# Point PyTorch/XLA at the Tenstorrent device
xr.set_device_type("TT")
device = xm.xla_device()

# load_model()/load_inputs() are instance methods, not static — instantiate
# first (confirmed live: ModelLoader.load_model(...) raises TypeError:
# missing 1 required positional argument: 'self')
loader = ModelLoader()

# Load the BEiT-base-patch16-224 model at bfloat16 precision
model = loader.load_model(dtype_override=torch.bfloat16).eval()
inputs = loader.load_inputs(dtype_override=torch.bfloat16)

# Compile to Tensix machine code and move onto the device.
# First call: torch-xla traces to StableHLO, the TT-MLIR pipeline compiles it
# (seconds to minutes depending on model size). Later calls hit the cache.
compiled = torch.compile(model, backend="tt").to(device)
output = compiled(inputs.to(device))

# Same output structure as the original model
print(output.logits.argmax(-1))

Walk through what happens at each line. loader.load_model() fetches BEiT-base from HuggingFace and returns a standard PyTorch nn.Module. The dtype_override=torch.bfloat16 argument casts weights to bfloat16, the Blackhole chip’s native float format.

torch.compile(model, backend="tt") is where the work happens. torch-xla traces the model into a StableHLO graph; the TT-MLIR pipeline tunes tile shapes, assigns cores, schedules data movement, and emits Tensix machine code. The compiled callable is API-identical to the original nn.Module — call it with inputs, get outputs — except the computation now executes on Blackhole hardware instead of your CPU.

First-call JIT time is real. BEiT compiles in a few seconds; a large vision transformer can take a few minutes. Subsequent calls with the same input shapes skip compilation and hit the cached kernels directly.

The ForgeModel Interface

The tt-forge-models zoo at ~/code/tt-forge-models defines a standardized interface for 800+ model variants. Every loader implements the ForgeModel abstract base class from base.py, as instance methods (confirmed live, both in base.py’s abstract signatures and in every concrete loader checked — load_model/load_inputs all take self):

• __init__(self, variant=None) — picks the checkpoint variant at construction time, not at load time
• load_model(self, *, dtype_override=None, **kwargs) — fetches, instantiates, and returns a ready-to-compile nn.Module
• load_inputs(self, **kwargs) — returns sample tensors that match the model’s expected input shape and dtype

Each loader module exports a ModelVariant enum naming its specific checkpoints — as a separate top-level name, not an attribute of ModelLoader (confirmed live: ModelLoader.ModelVariant raises AttributeError; the module’s own __init__.py does from .loader import ModelLoader, ModelVariant, two independent imports). BEiT’s loader has variants for different patch sizes and training configurations. ResNet’s loader offers:

from third_party.tt_forge_models.resnet.pytorch import ModelLoader, ModelVariant

ModelVariant.RESNET_50_HF    # HuggingFace checkpoint
ModelVariant.RESNET_50_TIMM  # timm checkpoint

The ModelTask taxonomy in config.py organizes models by task type: NLP_CAUSAL_LM, CV_IMAGE_CLS, CV_OBJECT_DETECTION, and others. ModelGroup classifies models by family — Vision Transformers, CNNs, generative language models. The taxonomy is machine-readable, which matters for the compiletron game (more below).

This standardization exists so you can swap models without rewriting your compilation harness. The compilation loop is always:

import torch
from third_party.tt_forge_models.<model>.pytorch import ModelLoader, ModelVariant

loader = ModelLoader(variant=ModelVariant.SOME_VARIANT)
model = loader.load_model(dtype_override=torch.bfloat16)
inputs = loader.load_inputs()
compiled = torch.compile(model, backend="tt").to(device)

Read the full forge-models zoo lesson for traversal patterns and custom variant registration.

import jax
import jax.numpy as jnp
import pjrt_plugin_tt  # registers TT as JAX backend

@jax.jit
def forward(x):
    return jnp.sin(x) + jnp.cos(x)

x = jnp.ones((128, 128))
result = forward(x)  # compiles to Tensix on first call

Forge vs. TTNN — When to Use Which

Three layers of the stack are now in front of you. They are not competing alternatives. They solve different problems at different altitudes.

The most common pattern in practice: use TT-Forge for whole-model compilation. Drop to TTNN for custom ops that TT-Forge doesn’t yet support or where you need tiling control. Drop to TT-Lang for the one inner loop that the profiler says dominates your runtime.

Forge and TTNN are composable. A compiled model can call into TTNN ops, and a TTNN program can lean on torch.compile(backend="tt") for the transformer backbone while hand-tuning specialized attention variants in TTNN. The layers were designed to coexist.

TT-Forge Compiletron

The tt-forge-compiletron at ~/code/tt-forge-compiletron is a roguelike model compilation game built on top of the forge pipeline. It is also a serious tool for surveying the compile-compatibility landscape of the zoo and of HuggingFace at large.

Set it up, then launch it:

cd ~/code/tt-forge-compiletron
bash scripts/install.sh --forge   # installs forge venv + tt-forge-fe shim, clones tt-forge-models
python3 expedition.py run --tui --seed-only --backend forge

The three-screen Textual TUI shows the model queue, live compilation progress per chip, and a running score. The --seed-only flag restricts the model pool to the tt-forge-models zoo — hundreds of curated models guaranteed to have standardized loaders. Drop --seed-only to enable --frontier-only mode, which discovers models live from HuggingFace based on download velocity and rarity signals.

Internally, expedition.py delegates to a router that reads ModelConfig.task and ModelConfig.group metadata from each zoo entry. That metadata informs backend selection (forge vs xla) and chip assignment. The --backend mixed flag alternates backends across the model queue, which is useful for cross-backend compile-rate comparison.

The bestiary at data/bestiary.json is a persistent record of every model the compiletron has ever attempted: compile status, timing, output shape, error class if it failed. The router uses the bestiary to deprioritize known-broken models and surface fresh targets. It is also the primary artifact if you are contributing compile-fix patches upstream — the bestiary tells you exactly which models need work and what failed.

Performance timeseries land in data/perf_history.jsonl — one JSON object per compile run, appended chronologically. Use it to track compile-time regressions across forge versions or to graph throughput trends after a driver update.

The --bench-passes N flag runs N inference passes after a successful compile and records tokens-per-second or images-per-second into the timeseries. Use this to measure real inference throughput, not just compile success.

Next steps: The compiletron’s First Voice feature runs a themed inference pass after each successful compile, printing the model’s first decoded output on Tenstorrent silicon. It is genuinely entertaining as a throughput warm-up, but the underlying pattern — compile once, inference repeatedly, measure throughput via perf_history.jsonl — is the same pattern you use in production model benchmarking.

← Profiling | Going Deep →

LED Customization

Four chips on your desk. Each one has indicators, tiny tells of internal state. By default they sit there doing their jobs in silence, lights blinking at whatever the firmware thinks is worth reporting. You can change that.

This chapter is about making the QB2 communicate on your terms, using tt-smi to query chip state and steer the indicators toward something more informative, more decorative, or more satisfying. The machine becomes a physical dashboard.

tt-qb-lights: RGB That Responds to Your Hardware

If your QB2 is in a case with addressable RGB — or if you have a motherboard like the ASRock B850M-C with onboard RGB — there’s a ready-made solution that does this properly: tt-qb-lights, a Rust systemd service built by Taylor Singletary specifically for Tenstorrent hardware.

Rather than calling tt-smi on a loop, it reads directly from /sys/class/hwmon/blackhole-pci-* — the same kernel interface lm-sensors uses — so it’s low overhead and doesn’t depend on any Tenstorrent CLI tools being in your PATH. It talks to OpenRGB over TCP (port 6742), so any RGB device OpenRGB supports becomes a live hardware dashboard.

What it does:

• Smooth color gradients driven by ASIC temperature — cool teal at idle, cycling through your chosen palette as the chips heat up
• Power-based brightness: dims to ~30% when idle, climbs to full brightness under load
• Warning pulse: lights pulse when temperature crosses a configurable threshold (default 70°C)
• Six built-in color schemes including QuietBox Sunset (inspired by the QB2 wallpaper), TT Dark, and Tenstorrent Branding (official teal → pink → gold → red)
• Live-editable config at ~/.config/tt-qb-lights/config.toml — change schemes and restart, no rebuild needed

Quick setup:

# Clone and build
git clone https://github.com/tsingletaryTT/tt-qb-lights ~/code/tt-qb-lights
cd ~/code/tt-qb-lights

# Automated installer — checks prerequisites, builds, guides you through setup
./install.sh

# Or manually:
cargo build --release

# Test without touching your lights
./target/release/tt-qb-lights --single-shot   # prints sensor readings
./target/release/tt-qb-lights --dry-run --debug  # shows color decisions

# Initialize your config
./target/release/tt-qb-lights --init
# Then edit: nano ~/.config/tt-qb-lights/config.toml

Requires: Rust 1.70+, OpenRGB installed and running with its SDK server enabled, Tenstorrent drivers loaded (so sensors | grep blackhole shows devices).

The service file handles startup ordering so OpenRGB starts before tt-qb-lights:

sudo systemctl enable openrgb
sudo systemctl enable tt-qb-lights
sudo systemctl start tt-qb-lights
journalctl -u tt-qb-lights -f   # watch it go

Full source, architecture notes, and troubleshooting: github.com/tsingletaryTT/tt-qb-lights

What tt-smi Can (and Can’t) Do

Worth saying plainly: tt-smi has no LED control of its own. Check for yourself:

tt-smi --help | grep -i led

Nothing prints — current tt-smi (check yours with tt-smi --version) has zero LED-related flags, and there’s no --set-led or equivalent hiding under a different name. tt-smi’s job is telemetry: it reads chip state and reports it. Driving actual lights is a separate concern, handled by whatever RGB control stack your hardware uses — which is exactly what tt-qb-lights above already wires up for you via OpenRGB.

What follows is the DIY version of the same idea, for anyone who wants a different RGB setup or a starting point to hack on: read chip telemetry from tt-smi -s, and drive your own LEDs through whatever tool actually controls them (OpenRGB’s CLI/API, a vendor tool, a GPIO script — not tt-smi).

What tt-smi -s Gives You

Every 1-second pulse of tt-smi -s returns a JSON document with a device_info list — one entry per chip, each with nested sub-objects rather than flat fields. The ones that matter for LED-driving logic live under telemetry:

• telemetry.asic_temperature — ASIC die temperature in Celsius (as a string, e.g. "40.3")
• telemetry.current — current draw in amps
• telemetry.power — power consumption in watts
• telemetry.voltage — chip supply voltage
• telemetry.aiclk — AI clock frequency in MHz (higher when actively computing)

Firmware version lives separately, under firmwares.fw_bundle_version — there’s no arc_fw_version field.

A chip sitting idle has low aiclk. A chip running inference has elevated aiclk and rising asic_temperature. Those two signals alone let you build a three-state indicator: idle, working, hot.

A Monitoring Script

This script reads tt-smi -s every two seconds and calls your RGB tool’s own command based on chip temperature. It assumes OpenRGB’s CLI is on your PATH (openrgb --help to check) — swap in whatever you actually have:

#!/usr/bin/env python3
"""
QB2 LED monitor — drives chip indicators from tt-smi telemetry.
Reads chip temperature every 2s and sets LED state accordingly.

Temperature thresholds:
  < 60°C  → steady green (idle / normal)
  60-80°C → pulsing amber (active inference)
  > 80°C  → rapid blink red (thermal throttle zone)

This targets OpenRGB's CLI (`openrgb --help` to see its own device/mode/color flags —
there is no tt-smi LED command to discover; that was never real).
Adjust LED_CMD_* below to match your RGB tool's actual syntax.
"""

import subprocess
import json
import time
import sys

# ── LED command templates ─────────────────────────────────────────────────────
# Fill these in from your RGB tool's own --help output (e.g. `openrgb --help`).
# There is no tt-smi equivalent — tt-smi only reports telemetry.
LED_CMD_COOL   = "openrgb --device {device_id} --mode static --color 00FF80"   # steady teal
LED_CMD_ACTIVE = "openrgb --device {device_id} --mode breathing --color FFA000" # pulsing amber
LED_CMD_HOT    = "openrgb --device {device_id} --mode flashing --color FF0000" # rapid red

TEMP_ACTIVE_THRESHOLD = 60.0   # °C — above this = chip is working
TEMP_HOT_THRESHOLD    = 80.0   # °C — above this = thermal warning
POLL_INTERVAL_SEC     = 2.0

def get_chip_state():
    """Return list of per-chip dicts from tt-smi -s JSON output."""
    result = subprocess.run(
        ["tt-smi", "-s"],
        capture_output=True, text=True, timeout=5
    )
    if result.returncode != 0:
        return []
    try:
        data = json.loads(result.stdout)
        return data.get("device_info", [])
    except (json.JSONDecodeError, AttributeError):
        return []

def set_led(device_id: int, mode: str):
    """Drive a chip LED. mode is one of: cool, active, hot."""
    cmd_template = {
        "cool":   LED_CMD_COOL,
        "active": LED_CMD_ACTIVE,
        "hot":    LED_CMD_HOT,
    }.get(mode, LED_CMD_COOL)
    cmd = cmd_template.format(device_id=device_id)
    subprocess.run(cmd.split(), capture_output=True)

def classify(chip: dict) -> str:
    """Decide LED mode from a device_info entry — temperature lives under telemetry."""
    try:
        temp = float(chip.get("telemetry", {}).get("asic_temperature", 0.0))
    except (TypeError, ValueError):
        # Telemetry field missing or non-numeric (transient tt-smi hiccup) —
        # treat as unknown rather than crashing the monitor loop.
        return "cool"
    if temp > TEMP_HOT_THRESHOLD:
        return "hot"
    if temp > TEMP_ACTIVE_THRESHOLD:
        return "active"
    return "cool"

def main():
    print("QB2 LED monitor starting. Ctrl-C to stop.")
    prev_modes = {}
    while True:
        chips = get_chip_state()
        if not chips:
            print("  [warn] No chip data from tt-smi — is the driver loaded?",
                  file=sys.stderr)
        for i, chip in enumerate(chips):
            mode = classify(chip)
            if prev_modes.get(i) != mode:
                set_led(i, mode)
                temp = chip.get("telemetry", {}).get("asic_temperature", "?")
                print(f"  chip {i}: {mode} (temp={temp}°C)")
                prev_modes[i] = mode
        time.sleep(POLL_INTERVAL_SEC)

if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        print("\nStopped.")

Save this as ~/scripts/qb2-led-monitor.py and mark it executable:

mkdir -p ~/scripts
chmod +x ~/scripts/qb2-led-monitor.py

Test it in your terminal while running a model in another session. You should see mode changes printed as the chips heat up.

Running at Boot as a User Service

Once the script works manually, make it automatic. A systemd user service starts with your login and restarts if it crashes:

mkdir -p ~/.config/systemd/user

Create ~/.config/systemd/user/qb2-led-monitor.service:

[Unit]
Description=QB2 LED Monitor
After=default.target

[Service]
ExecStart=/usr/bin/python3 /home/%i/scripts/qb2-led-monitor.py
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

Enable and start it:

# Enable lingering so user services survive logout
loginctl enable-linger $USER

# Reload systemd and start the service
systemctl --user daemon-reload
systemctl --user enable qb2-led-monitor.service
systemctl --user start qb2-led-monitor.service

# Check it's running
systemctl --user status qb2-led-monitor.service
journalctl --user -u qb2-led-monitor.service -f

The LEDs will now respond to chip state automatically, every time you log in.

Heat map across one Blackhole chip. Your LED script mirrors this in physical hardware.

Next: Fun Demos →

Fun Demos

There’s a specific moment that happens when someone skeptical about specialized hardware sees tt-toplike running. Their face shifts. The demos in this chapter produce that moment reliably. They’re not benchmarks. They’re invitations — to look, to question, to want to understand what the machine is actually doing.

Four demos. Each one stands alone. All of them run on your QB2 today.

Demo 1: Arcade Mode

Open a terminal. Start a model serving in another session, or just leave the hardware idle. Then run:

tt-toplike --mode arcade

The screen fills. A hero character moves with chip telemetry — position, speed, direction all derived from actual hardware readings. AICLK frequency, power draw, thermal state. The game is the monitor. The monitor is the game.

This is the first thing to show anyone who claims hardware monitoring is inherently boring. It isn’t. It just needs better defaults.

To exit: q or Ctrl-C.

Install tt-toplike if it isn’t already present:

# tt-toplike is in the Tenstorrent apt repository (set up by tt-installer):
sudo apt update && sudo apt install tt-toplike

# "Repository is not signed"? The signing key is missing — see
# /tinkerer/04-break-and-fix/ pattern 9, or re-download it:
#   sudo curl -fsSL -o /etc/apt/keyrings/tt-pkg-key.asc https://ppa.tenstorrent.com/tt-pkg-key.asc

# No repository on this machine? Install the .deb from GitHub releases instead:
# https://github.com/tenstorrent/tt-toplike/releases
sudo dpkg -i tt-toplike_*.deb
# Or: cargo install tt-toplike

Demo 2: Flow Mode

Where arcade mode is expressive, flow mode is accurate-expressive. Run:

tt-toplike --mode flow

Particle streams trace the NOC — Tenstorrent’s Network on Chip. During inference, data moves from the DRAM perimeter into the compute cores and back out again, each hop a real transaction on a real fabric. Flow mode makes those streams visible as animated particles.

Watch what happens when you start or stop inference. The particle density changes. The path patterns change. You’re watching the chip’s actual communication graph in motion.

This one tends to generate the most questions. “What are those things?” is how good conversations start.

Demo 3: AI Video Generation — Live Generative Art

tt-local-generator is a GTK4 desktop app that runs video generation entirely locally, no API key, no cloud dependency. The Wan2.2 text-to-video model produces 480×832 clips using all four Blackhole chips. Each clip takes roughly six minutes.

Set up a continuous generation loop and you have a generative art installation:

# tt-local-generator is in the Tenstorrent apt repository (set up by tt-installer):
sudo apt update && sudo apt install tt-local-generator

# Launch the app
tt-local-generator

No repository on this machine? Install the .deb from GitHub releases instead — releases page:

sudo dpkg -i tt-local-generator_*.deb

In the app, open the video generation panel. Write a prompt. Let it run. The app has an “attractor mode” that generates clips continuously and plays them fullscreen. Walk away. Come back to a wall of generated cinema.

For a polished installation setup — fullscreen display, auto-start on login, continuous prompts — see the QB2 video generation lesson.

The AnimateDiff integration in tt-local-generator also runs natively on QB2. Shorter clips, different aesthetic, same local-only principle. See tt-animatediff for the standalone library.

Demo 4: Local 70B with No Internet Required

Four chips. A language model with 70 billion parameters. No API key. No latency spike from a datacenter on another continent.

The fastest path is through tt-inference-server, which handles the Docker container and weight caching automatically:

docker run \
  --env "HF_TOKEN=$HF_TOKEN" \
  --ipc host \
  --publish 8000:8000 \
  --device /dev/tenstorrent \
  --mount type=bind,src=/dev/hugepages-1G,dst=/dev/hugepages-1G \
  --volume volume_id_tt_transformers-Llama-3.3-70B-Instruct:/home/container_app_user/cache_root \
  ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.16.0-669d59e-3334377 \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tt-device p300x2

p300x2 is the spec matching a QuietBox 2 — two P300 cards, four Blackhole chips. Older notes said p150x4, which addresses the same four chips with a less card-aware fabric description. Both can load a model; they are separate spec entries with different pinned commits, and p300x2 is the validated one for this model. If that image tag has moved on, add --print-docker-cmd to the run.py equivalent to read the current one:

cd ~/.local/lib/tt-inference-server
python3 run.py --model Llama-3.3-70B-Instruct --tt-device p300x2 \
  --workflow server --docker-server --print-docker-cmd

Wait for Application startup complete — first run downloads 140 GB of weights, so plan ahead. Then ask it something that requires reasoning:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.3-70B-Instruct",
    "messages": [{"role": "user", "content": "Explain the tradeoffs between data-parallel and model-parallel inference for large language models. Be specific about memory and latency."}]
  }' | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['choices'][0]['message']['content'])"

The response comes from silicon in your own office. The model that required a specialized cloud service a year ago runs on hardware you own.

That’s the demo.

For the full walkthrough — prerequisites, HuggingFace token setup, the DeepSeek reasoning model variant, and troubleshooting — see Running Llama-3.3-70B on QB2.

One Blackhole chip running a slice of Llama-3.1-70B. All four of yours look like this, simultaneously.

Demo 5: TT-Forge Compiletron — Hunt the Wild Model

The premise is simple: compile as many models as possible, score points, and try not to hit something that crashes the runtime.

tt-forge-compiletron is a roguelike model compilation game. It is also a genuine engineering survey tool. The gameplay loop runs models from the tt-forge-models zoo (200+ curated PyTorch and JAX models) through forge.compile() on your Blackhole chips. Each successful compile earns points. First-ever compile of a model earns a ×5 multiplier. The scoring system rewards breadth, rarity, and speed.

The TUI is three screens running in a Textual interface: a model queue on the left, live per-chip compilation status in the center, and a running scoreboard with ASCII art banners rendered in pyfiglet. When a model compiles successfully, the First Voice feature kicks in — a themed inference pass that prints the model’s first decoded output on Tenstorrent silicon. The first time a language model generates a token on your hardware, it announces itself with a banner.

The bestiary at data/bestiary.json grows with every session. It records compile status, timing, and failure class for every model attempted. Come back later and it knows what you’ve already conquered.

Set up and launch:

cd ~/code/tt-forge-compiletron
source ~/tt-forge-fe/env/activate
python3 expedition.py run --tui --seed-only --limit 8 --chips 4

--seed-only pulls from the curated zoo only, --limit 8 caps the session at eight models, --chips 4 assigns all four Blackhole cards. A good starting session.

Four genuine Blackhole chips across two p300c cards, joined by the Samtec link. The model-zoo game compiles a different model on each, in parallel.

Demo 6: tt-bio — Watch a Protein Fold in Real Time

Not every demo has to be about tokens. tt-bio runs Boltz-2 — a biomolecular structure and binding-affinity model — on Blackhole, approaching physics-based free-energy-perturbation accuracy at roughly 1000× the speed. It scales from a single card up through a full QuietBox (4×) to a Galaxy (32×), and it’s an actively released project, not a one-off port.

# The [tenstorrent] extra is required — without it, install-deps refuses with
# "needs the ttnn wheel, which is not installed"
pip install "tt-bio[tenstorrent] @ git+https://github.com/moritztng/tt-bio.git"
tt-bio install-deps

# Fold a protein from a sequence spec — MSA runs automatically by default
# (local DB, falling back to a public server); no flag needed for that.
# --single_sequence skips MSA entirely if you want a fast, lower-accuracy pass.
tt-bio predict your-protein.yaml --model boltz2

It also supports newer structure-prediction and binder-design models — ESMFold-2, Protenix v2, OpenFold3 (AlphaFold3-style folding of protein/RNA/DNA), and design tools like BoltzGen and RFdiffusion3 (tt-bio design INPUT --model boltzgen).

For the demo-floor version of this, there’s tt-bio-demo: a native GTK4/OpenGL booth app that renders a protein condensing out of noise into its folded structure live, with per-residue confidence coloring and a Tensix core grid alongside the render. Its quad view runs four independent folds on four chips at once — one per Blackhole on a QB2 — which makes the same “the monitor is the demo” point as arcade mode, but for structural biology instead of telemetry.

Next: Ubuntu Customization →

Ubuntu Customization

The QB2 ships with Ubuntu 24.04 LTS and its stock GNOME desktop — the same experience every Ubuntu user gets. It’s functional. It’s also waiting for your preferences. This chapter is about making the machine yours without breaking what Tenstorrent set up.

The rule throughout: don’t touch the tt-metal environments. Customize everything else freely.

Quick Orientation

Ubuntu 24.04 uses apt as its system package manager, with the GNOME desktop on top. Three package systems coexist:

apt         # system packages — drivers, libraries, system tools
snap        # containerized apps — VS Code, browser
flatpak     # alternative sandbox format (install if needed)

Check available storage before installing anything heavy:

ncdu ~/

Install ncdu first if it isn’t present: sudo apt install ncdu. It’s a terminal disk usage navigator. Run it before downloading model weights too — Llama-3.1-70B is 140 GB. Surprises are unpleasant.

Desktop

Ubuntu’s GNOME desktop is what greets you on first boot. Two tools make it yours without fighting it:

GNOME Tweaks — fonts, window-button layout, animations, startup apps:

sudo apt install gnome-tweaks

Extension Manager — browse and install GNOME Shell extensions (dash-to-dock, system monitors, clipboard history):

sudo apt install gnome-shell-extension-manager

Most desktop settings live in dconf; script them with gsettings:

gsettings set org.gnome.desktop.interface color-scheme prefer-dark   # dark mode
gsettings set org.gnome.desktop.interface clock-show-seconds true

GNOME on Ubuntu 24.04 runs on Wayland by default — echo $XDG_SESSION_TYPE confirms it. That matters when you pick a terminal below.

sudo apt install kde-plasma-desktop

Terminal

Ubuntu’s GNOME desktop ships GNOME Terminal by default. It’s perfectly good. If you want more, a few favorites:

Kitty — GPU-accelerated, fast, tabs + splits built in:

sudo apt install kitty

Foot — minimal native Wayland terminal, fastest startup:

sudo apt install foot

Alacritty — cross-platform, GPU-rendered, config-file-driven:

sudo apt install alacritty

Any of these pairs well with the rest of this setup.

Shell

The system shell is bash. Two popular upgrades:

zsh + starship prompt — zsh is feature-rich; starship is a fast, cross-shell prompt that shows git status, Python venv, and more:

sudo apt install zsh
chsh -s $(which zsh)   # set zsh as your login shell

# Install starship (Rust binary, no system package needed)
curl -sS https://starship.rs/install.sh | sh

# Add to your ~/.zshrc:
echo 'eval "$(starship init zsh)"' >> ~/.zshrc

fish — if you want a shell that just works out of the box with autosuggestions and syntax highlighting:

sudo apt install fish
chsh -s $(which fish)

Useful CLI Tools

These are collectively small downloads, collectively large quality-of-life gains:

sudo apt install \
  htop \       # interactive process monitor
  ncdu \       # disk usage navigator
  bat \        # cat with syntax highlighting — binary installs as `batcat`
  ripgrep \    # blazing grep replacement (rg)
  fd-find \    # fast find replacement — binary installs as `fdfind`
  fzf \        # fuzzy finder — attach to shell history, file browsing
  jq \         # JSON processor — useful for tt-smi -s output
  tmux         # terminal multiplexer — see below

alias bat=batcat
alias fd=fdfind

fzf in particular integrates with shell history (Ctrl-R) and file search (Ctrl-T) to make terminal navigation dramatically faster. Add these lines to your .bashrc or .zshrc after installing:

eval "$(fzf --bash)"    # bash
# or
eval "$(fzf --zsh)"     # zsh

tmux for Multi-Session Work

When you’re downloading a 140 GB model in one session, monitoring chips in another, and editing code in a third, tmux prevents terminal chaos. It also keeps processes running if your SSH connection or terminal emulator drops.

# Install
sudo apt install tmux

# Start a new named session
tmux new -s qb2

# Attach to an existing session
tmux attach -t qb2

# Key bindings (default prefix is Ctrl-b):
#   Ctrl-b c     new window
#   Ctrl-b %     vertical split
#   Ctrl-b "     horizontal split
#   Ctrl-b d     detach (session keeps running)
#   Ctrl-b [     scroll mode (q to exit)

A minimal ~/.tmux.conf to add mouse support and increase scrollback:

cat >> ~/.tmux.conf << 'EOF'
set -g mouse on
set -g history-limit 50000
set -g default-terminal "tmux-256color"
EOF

Python Environment Management

Ubuntu 24.04’s system Python is externally-managed. pip install at the system level will refuse or break things. This is correct behavior. Don’t fight it.

For managing Python versions and creating isolated envs outside the tt-metal stack, use uv — it’s fast, correct, and doesn’t require a conda installation:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a project-local Python 3.11 env
uv venv --python 3.11 .venv
source .venv/bin/activate

# Install packages into it
uv pip install numpy requests

Alternatively, pyenv for managing multiple Python versions:

curl https://pyenv.run | bash
# Add to shell init (follow the printed instructions)
pyenv install 3.11.9
pyenv virtualenv 3.11.9 myproject
pyenv activate myproject

Keep these entirely separate from ~/.tenstorrent-venv/. That one is a managed environment holding tt-smi and tt-flash, your QB2 activates it at login, and a bad resolution in there costs you your hardware tooling. Don’t pip-install into it manually. (There’s no TTNN venv to worry about — TTNN lives inside the tt-metalium container, where your pip installs can’t reach it anyway.)

VS Code

sudo snap install code --classic

Or download the .deb from code.visualstudio.com for a non-snap install if you prefer:

sudo dpkg -i code_*.deb

Once VS Code is installed, get the Tenstorrent extension from the marketplace. Search for “tt-vscode-toolkit” in the Extensions panel, or install from the command line:

code --install-extension tenstorrent.tt-vscode-toolkit

The extension adds tt-metal project support, TTNN kernel highlighting, chip status indicators in the status bar, and guided lessons — including the QB2 video generation walkthrough.

Useful Aliases

Add these to ~/.bashrc or ~/.zshrc to cut down repetitive typing:

# Tenstorrent environment shortcuts
alias ttenv='tt-metalium'                      # TTNN lives in the Metalium container
alias ttserve='python3 ~/.local/lib/tt-inference-server/run.py'

# Readable tt-smi JSON output
alias ttsmi='tt-smi -s | python3 -m json.tool'

# Watch chip state every 2 seconds
alias ttwatch='watch -n2 "tt-smi -s | python3 -m json.tool"'

# Quick disk check
alias diskcheck='ncdu ~/ --exclude ~/models'

Source the file to pick up changes immediately:

source ~/.bashrc

Next: Breaking & Fixing Things →

Breaking & Fixing Things

The QB2 is a workstation, not a cloud VM with a reset button in a web console. Software changes are reversible. Hardware is physically robust. The worst case scenario for almost everything in this chapter is a few minutes of diagnostic work and a single command.

This philosophy matters. It means you should experiment freely. Break things. Learn the recovery pattern. The machine can take it.

The Recovery Ladder

Before reaching for dramatic solutions, follow this order:

1. Run the failing command again (transient errors happen)
2. Check the relevant log: journalctl, docker logs, or the service’s own log file
3. Restart the failing service: systemctl restart <name> or docker restart <container>
4. Reset the boards: tt-smi -r — clears a stuck Ethernet fabric or ARC state without a full reboot. Cheaper than reloading the driver, and it’s the fix for the suspend/resume pattern below, so reach for it before step 5.
5. Reload the driver: sudo modprobe -r tenstorrent && sudo modprobe tenstorrent
6. Reboot: sudo reboot
7. Post to the Tenstorrent Discord with tt-smi -s output

The vast majority of issues resolve at step 1, 2, or 3.

Common Breakage Patterns

1. pip install in the Wrong Environment

Symptom: Something installed correctly but broke an import in tt-metal, or pip install warned about an externally-managed environment.

Cause: Installing into system Python, or into ~/.tenstorrent-venv — which your QB2 activates at login, so a pip install typed without thinking lands there and can drag tt-smi down with it.

Fix:

# Identify which pip you used, and which environment is active
which pip           # or: which pip3
echo "$VIRTUAL_ENV"

# If it was system pip, uninstall the conflicting package
pip uninstall <package-name>

# If you damaged the tooling venv, rebuild it — it only ever held two tools,
# so this is cheap. Deactivate first, or you'll be deleting the venv you're in.
deactivate 2>/dev/null
rm -rf ~/.tenstorrent-venv
python3 -m venv ~/.tenstorrent-venv
source ~/.tenstorrent-venv/bin/activate
pip install tt-smi tt-flash

TTNN is not at risk from any of this: it lives inside the tt-metalium container image, not in a venv you can pip into. If TTNN itself misbehaves, the reset is docker pull on the Metalium image, not a venv rebuild.

Going forward, give every project its own venv — or install CLIs with uv tool / pipx, which does that for you. Never use pip install --break-system-packages unless you have a specific reason.

2. conda Conflict with tt-metal

Symptom: Conda activated, model fails to load, TTNN throws import errors about library version mismatches.

Cause: Conda replaces system libraries in PATH, breaking tt-metal’s pinned dependencies.

Fix:

# Deactivate conda
conda deactivate

# Remove conda from PATH for this session
unset CONDA_DEFAULT_ENV

# Long-term: add this to your .bashrc AFTER the conda init block
# to prevent auto-activation:
conda config --set auto_activate_base false

Keep tt-metal environments and conda environments in separate shell sessions. They do not coexist gracefully.

3. Ethernet Fabric Stuck After Suspend/Resume

Symptom: After suspending and resuming the QB2 — or stopping and starting a serving container without a device reset in between — the next model load or serve attempt fails with something like:

Device 0 ... ethernet core ... Try resetting the board

Cause: Suspend/resume leaves the on-die Ethernet fabric that links the four chips in a stuck state. The chips themselves are fine; it’s the chip-to-chip links (the same ones AllReduce uses for tensor-parallel) that didn’t come back up cleanly. A container stop→start without a device reset in between can trigger the same thing.

Fix:

# Reset the boards BEFORE every (re)start after a suspend/resume — don't wait
# for the error to show up
tt-smi -r

# Then bring the server/container back up as usual

If the fault recurs even after tt-smi -r, a full reboot (sudo reboot) is the durable fix — it’s the only thing that reliably clears the fabric for good.

4. “No Devices Found” After a Kernel Upgrade

Symptom: tt-smi returns no devices. lsmod | grep tenstorrent shows nothing.

Cause: A kernel upgrade installed a new kernel without re-building or loading the Tenstorrent driver for it.

Fix:

# Check if the driver module exists for the current kernel — DKMS modules
# land under updates/dkms/, not extra/ (confirmed live: the real file is
# .../updates/dkms/tenstorrent.ko.zst)
find /lib/modules/$(uname -r)/ -iname "*tenstorrent*"

# If missing, reinstall the kernel driver package (this is the actual
# package name — "tt-firmware" doesn't exist)
sudo apt install --reinstall tenstorrent-dkms
# or re-run the tt-installer if you used that for initial setup

# Reload the driver
sudo modprobe tenstorrent

# Verify
tt-smi -s

If sudo modprobe tenstorrent fails with “module not found”, the driver isn’t built for the current kernel. You need to either roll back the kernel or rebuild the driver. Check the tt-metal GitHub for the currently supported kernel range.

5. Model Download Corrupted Midway

Symptom: Model fails to load. Error messages about unexpected EOF or missing shards.

Fix:

# hf (not huggingface-cli) — the CLI this guide standardizes on; it resumes
# partial downloads automatically, no flag needed
hf download <model-id> --local-dir ~/models/<model-name>

# Example for Llama-3.1-8B:
hf download meta-llama/Llama-3.1-8B-Instruct \
  --local-dir ~/models/Llama-3.1-8B-Instruct

If the download is severely corrupted, delete the partial directory and start fresh:

rm -rf ~/models/<model-name>
hf download <model-id> --local-dir ~/models/<model-name>

6. OOM During Inference

Symptom: Python process crashes with out-of-memory error during model load or inference. The model may be too large for the chip DRAM, or you’re only using one chip for a model that needs four.

Fix:

Chips are chosen by the mesh shape — there is no --num_gpus or --tensor-parallel-size on this platform. The simplest way to get the mesh right is to let tt-inference-server derive it from the model’s spec:

# All four chips, needed for a 70B
python3 ~/.local/lib/tt-inference-server/run.py \
  --model Llama-3.1-70B-Instruct \
  --workflow server --tt-device p300x2 --docker-server

If you’re driving vllm serve yourself (see vLLM on QB2 for how to get a vLLM that can — it is not installed on the host), the mesh is an environment variable instead:

export TT_METAL_ARCH_NAME=blackhole
export MESH_DEVICE=P300x2            # all four chips
export HF_MODEL=~/models/Llama-3.1-70B-Instruct
vllm serve "$HF_MODEL" --port 8000

Also check that DRAM on every chip is actually up (tt-smi -s doesn’t report a per-chip usage figure, only link health — board_id and dram_status live under the nested board_info object, not at the top level):

tt-smi -s | python3 -c "
import json, sys
d = json.load(sys.stdin)
for chip in d.get('device_info', []):
    bi = chip.get('board_info', {})
    print(bi.get('board_id', '?'), '— DRAM up:', bi.get('dram_status', '?'))
"

If a process is still holding chip memory, fuser /dev/tenstorrent/* (or checking for orphaned Python/docker processes) will find it — a stuck reservation, not something tt-smi -s surfaces directly.

7. Docker or tt-inference-server Won’t Start

Symptom: docker ps hangs or errors; tt-inference-server container fails to launch.

Fix:

# Check if Docker daemon is running
systemctl status docker

# Start it if not
sudo systemctl start docker

# Check running containers
docker ps -a

# View logs from the last failed container
docker logs $(docker ps -a -q --filter "status=exited" | head -1)

# Hard-restart a specific container
docker stop <container-name> && docker start <container-name>

# If the container is corrupted, remove and repull
docker rm <container-name>
docker pull <image-name>
# Then re-run the server start command

8. tt-toplike Crashes at Startup

Symptom: tt-toplike exits immediately or produces a panic/error message.

Cause: Almost always a driver issue — tt-toplike can’t see the chips.

Fix:

# Verify chips are visible first
tt-smi -s

# If tt-smi also fails, reload the driver
sudo modprobe -r tenstorrent
sudo modprobe tenstorrent
tt-smi -s   # should now show four devices

# Then retry
tt-toplike --mode normal

If tt-smi -s works but tt-toplike still fails, reinstall it:

# tt-toplike is in the Tenstorrent apt PPA (set up by tt-installer):
sudo apt update && sudo apt install --reinstall tt-toplike

# apt says the repository isn't signed? See pattern 9 below.

# No repository on this machine? Install the .deb from GitHub releases instead:
# https://github.com/tenstorrent/tt-toplike/releases
sudo dpkg -i tt-toplike_*.deb
# Or: cargo install tt-toplike --force

9. apt Won’t Install Tenstorrent Packages (“repository is not signed”)

Symptom: sudo apt update or sudo apt install tt-smi (or tt-toplike, tenstorrent-dkms, tt-flash, …) fails with one of:

E: The repository 'https://ppa.tenstorrent.com/ubuntu noble InRelease' is not signed.
W: GPG error: https://ppa.tenstorrent.com/ubuntu noble InRelease: ... NO_PUBKEY ...
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

Cause: The repository line in /etc/apt/sources.list.d/tenstorrent.list is pinned to a key at /etc/apt/keyrings/tt-pkg-key.asc, and that file is missing, empty, or unreadable. apt won’t touch a repository it can’t verify, so nothing from Tenstorrent installs until the key is back. Usually this means the repository was added by hand and the key step was skipped — or the download was intercepted and left a zero-byte file behind.

Fix:

sudo mkdir -p /etc/apt/keyrings
sudo chmod 755 /etc/apt/keyrings
sudo curl -fsSL -o /etc/apt/keyrings/tt-pkg-key.asc https://ppa.tenstorrent.com/tt-pkg-key.asc

# Confirm before retrying: a PGP block, non-zero size, mode 644
head -1 /etc/apt/keyrings/tt-pkg-key.asc
ls -l /etc/apt/keyrings/tt-pkg-key.asc

sudo apt-get update

Still broken? Work down this list:

The full setup — repository line, key, and the Debian/Fedora variants — is in Installing the Stack.

Next: Community & Contribution →

Community & Contribution

The hardware is interesting alone. It becomes more interesting when someone else’s kernel runs on it, or when your demo helps someone else understand what’s possible.

This chapter is about where to find those people and how to add to what they’re building.

tt-awesome

The community catalog lives at tenstorrent.github.io/tt-awesome. Models, demos, integrations, experiments — contributed by people who picked up the same hardware you have and built something worth sharing. Browse it before building. Someone may have already done the hard part, and their version might be better than yours would be.

Adding a project is a pull request to github.com/tenstorrent/tt-awesome. The bar is low: working code, a README, and a description that tells someone why they’d want to run it.

GitHub

All of Tenstorrent’s core repositories are public: github.com/tenstorrent.

Useful starting points:

• tt-metal — the core compute stack; file bugs for driver issues, TTNN problems, compilation failures
• tt-smi — hardware monitoring; file bugs for incorrect readings, missing devices, command errors
• tt-toplike — the TUI visualizer; file bugs for crashes, rendering issues, wrong telemetry
• tt-animatediff — community-ready video generation library: github.com/tenstorrent/tt-animatediff, v0.1.0

When filing a bug, include tt-smi -s output and your Ubuntu/kernel versions. A reproducer is worth ten paragraphs of description.

Discord

The Tenstorrent Discord is the fastest path to a human answer. Find the link at tenstorrent.com/community. The community includes Tenstorrent engineers, researchers running production workloads, and people who are a week into their first QB2, exactly where you were recently.

The channels worth knowing: #hardware-support for driver and chip questions, #tt-metal for software stack questions, #show-and-tell for demos. Show your tt-toplike screenshots there. People appreciate them.

Contribution Ideas

Add a demo to tt-awesome. If you’ve got something running on your QB2 that took more than an hour to figure out, it’s worth sharing. A Jupyter notebook demonstrating a novel TTNN op, a generative art setup using tt-local-generator, a custom LED monitoring script — these are all valid contributions.

Share a VHS recording. A terminal recording of your demo is worth more than a static screenshot. VHS renders .tape files into .gif or .mp4. Record your tt-toplike flow mode demo, your first 70B query, your LED script responding to thermal changes. Share to Discord or embed in a tt-awesome entry.

Write a lesson. The tt-vscode-toolkit lessons are written by people who built things and wanted to teach them. If you’ve worked through a non-trivial workflow on QB2, the lesson format is a good home for it.

File a bug. When something breaks and you fix it, the issue report benefits the next person. Document the exact error message, the kernel version, the fix. The break-and-fix patterns in this guide grew from exactly that kind of contribution.

Where to Go from Here

You’ve tinkered. You’ve broken things and fixed them. You’ve shown demos that stopped people mid-conversation. The next layer is the Tinker track, where the interesting question shifts from “what can this machine do?” to “what can I build on it?”

The QB2 becomes more interesting the more people are working on it together. The hardware is the same machine in every office. The software, the experiments, the failures-turned-blog-posts, the demos running on Discord at midnight — those are what actually make a platform. You’re part of that now. Build something.