Cross-track lesson · Run & build

Build Your Own vLLM Environment

Run `vllm serve` yourself on a QB2 — why you would, what you have to build first, and the dependency traps between you and a working `import ttnn`.

Most people should serve models on a QB2 with tt-inference-server, which is what Serving Models on QB2 covers. It wraps vLLM in a container, picks the mesh for you, and needs no environment building.

This lesson is the other route: running vllm serve yourself. You get flags the managed path doesn’t expose and a newer vLLM than the managed image pins — and in exchange you own the dependency tree.

vLLM is not installed on a QB2. Not in ~/.tenstorrent-venv, not on the host at all. The copy the managed path uses lives inside its container image and is pinned per model. So everything below is something you build deliberately, and the order matters: get a working ttnn, install the plugin on top of it, and only then serve.

1. See what’s already on the box

Some QB2s carry an older Tenstorrent vLLM fork checkout. Find out before you install anything. Run this in the environment you serve from. It only locates packages and never imports vLLM, because a stale install often fails on import and would hide the answer:

python3 - <<'EOF'
import importlib.util
from importlib.metadata import distributions

def where(mod):
    try:
        spec = importlib.util.find_spec(mod)
    except Exception:
        return None
    return spec.origin if spec and spec.origin else None

def versions(name):
    want = name.lower().replace("_", "-")
    seen = []
    for dist in distributions():
        try:
            if (dist.metadata["Name"] or "").lower().replace("_", "-") == want:
                seen.append(dist.version)
        except Exception:
            pass
    return list(dict.fromkeys(seen))

for label, mod, dist in (("vllm", "vllm", "vllm"),
                         ("plugin", "vllm_tt_plugin", "vllm-tt-plugin"),
                         ("ttnn", "ttnn", "ttnn")):
    print(f"{label:8}: {', '.join(versions(dist)) or 'not installed'} | {where(mod) or '-'}")
EOF

A vllm outside site-packages (say under ~/tt-vllm) is a fork checkout. To move across, uninstall first so the old one cannot shadow the new — uv pip uninstall vllm vllm-tt-plugin — then follow the install above. Old clones are harmless to leave on disk once uninstalled.

A vllm outside site-packages (say under ~/tt-vllm) is a fork checkout. To move across, uninstall first so the old one cannot shadow the new — uv pip uninstall vllm vllm-tt-plugin — then continue below. Old clones are harmless to leave on disk once uninstalled.

2. Get a Python ttnn

This is the real work, and it is worth being clear-eyed about it. The plugin binds to whatever tt-metal your environment provides and only activates when import ttnn succeeds:

• ~/.tenstorrent-venv is not such an environment. It holds tt-smi and tt-flash; import ttnn fails there.
• The apt packages tt-metalium and tt-nn are the C++ runtime libraries, not the Python module.
• The ttnn that a factory QB2 ships is inside the Metalium container. The tt-metalium wrapper runs it with --rm, so anything you pip-install in a session is gone when you exit.

That leaves two routes. A derived container image (FROM the Metalium image, installing the plugin at build time) is the dependable one, because the ttnn in that image is the one Tenstorrent tests. A host venv is lighter, using the ttnn wheel Tenstorrent publishes to PyPI:

python3 -m venv ~/.venvs/vllm-tt
source ~/.venvs/vllm-tt/bin/activate
pip install ttnn

# This must succeed before you go any further. If it does not, take the container route.
python3 -c "import ttnn; print('ttnn ok')"

3. Install the plugin

Tenstorrent’s vLLM support is a standalone platform plugin, tenstorrent/vllm-tt-plugin. It runs against upstream vLLM rather than a Tenstorrent fork: it contributes a tt platform through vLLM’s normal out-of-tree plugin mechanism, and vLLM selects it automatically whenever ttnn is importable. Newer model support lands here before it reaches a tt-inference-server release, and you get a normal upstream vLLM underneath.

source ~/.venvs/vllm-tt/bin/activate

# uv is required: the installer uses `uv pip`'s --override, which pip has no equivalent for
python3 -m pip install --upgrade pip setuptools wheel uv

git clone https://github.com/tenstorrent/vllm-tt-plugin.git ~/vllm-tt-plugin
cd ~/vllm-tt-plugin
source docs/install-vllm-tt.sh

That installer pins upstream vllm==0.26.0, removes a CUDA torchaudio that cannot load beside a CPU torch, and installs the plugin itself. It fetches vLLM’s dependency list from raw.githubusercontent.com, so it needs network access beyond your package index.

One dependency the installer does not cover, because upstream assumes you are installing into a full tt-metal environment:

uv pip install --override docs/vllm-overrides.txt pytest

pytest because tt-metal’s models/common/utility_functions.py imports it at module scope. Missing it shows up as Model architectures [...] failed to be inspected, which does not obviously point at a missing test framework. Earlier versions of this page also had you install torchvision by hand; the installer now does it for you, so check python3 -c "import torchvision" before adding it.

4. Serve a model

First get the weights (see Model Zoo for installing hf). Qwen3-0.6B is small enough to download in a minute and start fast, which makes it a good first model:

hf download Qwen/Qwen3-0.6B --local-dir ~/models/Qwen3-0.6B

Then serve it. This is the only launch in this lesson — one model owns the four chips and the port at a time, so if the managed path from the chapter is still running, stop it first (docker ps, then docker stop <id>; sudo lsof -w /dev/tenstorrent/* printing nothing means the chips are free).

# Your own vLLM environment — not the tooling venv
source ~/.venvs/vllm-tt/bin/activate

# Blackhole architecture flag
export TT_METAL_ARCH_NAME=blackhole

# Mesh shape. This — not --tensor-parallel-size — is how you choose chips.
# P150 = one chip, plenty for a 0.6B model; P300x2 is all four.
# Avoid the two-chip mesh (see below).
export MESH_DEVICE=P150

# Model load and first compile far exceed vLLM's default engine-ready deadline
export VLLM_ENGINE_READY_TIMEOUT_S=1800

# HF_MODEL is required when --model is a local path: tt-metal's tt_transformers
# uses it as the checkpoint directory, not just a name.
export HF_MODEL=~/models/Qwen3-0.6B

vllm serve ~/models/Qwen3-0.6B \
  --served-model-name Qwen3-0.6B \
  --block_size 64 --max_num_seqs 32 \
  --port 8000

Unlike the managed path, --served-model-name here means you choose the id the API reports — Qwen3-0.6B, exactly as written, not a Hugging Face repo id.

Fabric Router Sync: Timeout after 10000 ms on Device 2 ... Ethernet handshake likely failed

On first run: the model weights get compiled into Blackhole-optimized op graphs. This takes 3–5 minutes. Subsequent starts are fast — the compiled artifacts are cached.

Watch the logs. When you see a line containing Application startup complete, the server is accepting requests.

Status, honestly

On our QB2 this brings up correctly: the plugin activates, TTPlatform is selected, all four Blackhole chips open, weights load, the KV cache allocates across the mesh, and the OpenAI-compatible endpoints respond.

We have not signed off on output quality. In our testing generation degenerated into repetition, and that reproduced across two vLLM versions, three models, and both one- and four-chip meshes — so it is not specific to this plugin or to the mesh. Host firmware and driver versions running ahead of the tested pairings is the current suspicion. Treat this path as something to experiment with, not to benchmark against.