Serving Models on QB2

Serve a Model got a model answering with one tt serve command. This chapter is what is underneath that command, and what to do when you need more than the default: another port, two servers at once, a model that is not in the catalog, or a server on specific chips. By the end you will have an OpenAI-compatible server, a working curl command, and a Python client snippet you can drop into an application.

The Deployment Stack

tt serve <model> picks one of three serving paths from the model:

tt model info <model> lists the paths a model supports, and --inference-server, --studio or --model-manager forces one. All three produce an OpenAI-compatible API.

The direct vLLM path runs vllm serve yourself, for flags the managed path doesn’t expose or a newer vLLM than the pinned image ships. vLLM is not installed on a QB2, not in ~/.tenstorrent-venv and not on the host at all; the copy the managed path uses lives inside a container image. Building that environment is its own project: Build Your Own vLLM Environment →.

Serving with tt serve

tt serve Qwen3-32B                    # the model that ships with the box
tt serve Qwen3-32B --port 8000        # choose the host port
tt serve Qwen3-32B --dry-run          # show the resolved configuration and command
tt serve Llama-3.1-8B-Instruct --device p300   # use a different mesh than the detected one

The default port is 20000, which is tt-inference-server’s own default through tt. The curl examples later in this chapter use 8000, so either start the server with --port 8000 or substitute 20000 in the examples.

tt serve --dry-run prints the exact run.py command tt would run. This is the best way to learn what tt does for you, and to copy a command when you want to run it yourself:

/home/ttuser/.local/share/tenstorrent/tools/tt-inference-server-v0.25.0/venv/bin/python
  .../src/run.py --model Qwen3-32B --workflow server --docker-server --no-auth
  --host-volume /home/ttuser/data/tt-cache --device p300x2
  --vllm-override-args '{"enable-auto-tool-choice": true, "tool-call-parser": "hermes",
  "reasoning-parser": "qwen3"}' --service-port 20000

Three things tt adds that a bare run.py call does not:

• The parsers. tool-call-parser and reasoning-parser are in the model spec but run.py does not apply them on its own. Without them a container cannot answer tool_choice: "auto" requests, and coding agents get empty replies.
• Weights. It reuses a pre-seeded volume when one matches (the --host-volume above), and otherwise the Hugging Face cache (--host-hf-cache). See Magic of symlinks and Weights, Caches and Volumes.
• The checklist. It answers run.py’s interactive prompts for you and returns only when the endpoint is answering.

Running run.py Yourself

tt keeps its own pinned copy of tt-inference-server under ~/.local/share/tenstorrent/tools/tt-inference-server-<version>/. You can call it directly. Two reasons to: pinning a server to specific chips, and passing an option tt serve does not expose. These details were verified on a QB2:

cd ~/.local/share/tenstorrent/tools/tt-inference-server-v0.25.0/src   # run.py reads ./VERSION
export HF_TOKEN=$(cat ~/.cache/huggingface/token)
export MODEL_SOURCE=huggingface     # otherwise it asks "How do you want to provide a model?"
export JWT_SECRET=unused            # otherwise it asks for a secret, even with --no-auth

../venv/bin/python run.py \
  --model Llama-3.1-8B-Instruct --workflow server --docker-server --no-auth \
  --host-hf-cache ~/.cache/huggingface \
  --device p300 --device-id 2,3 --service-port 20000 --bind-host 127.0.0.1 \
  --vllm-override-args '{"enable-auto-tool-choice": true, "tool-call-parser": "llama3_json"}'

--no-auth turns authentication off, and run.py publishes the port on every network interface by default (--bind-host 0.0.0.0 in v0.25.0). --bind-host 127.0.0.1 keeps this server reachable from the QB2 itself only; reach it from another machine with an SSH tunnel. Leave auth on, and drop --no-auth, before you expose a server to a network. (The flag is in v0.25.0’s run.py; we added it to this command after the test run described above.)

Without MODEL_SOURCE and JWT_SECRET, run from a script or an SSH session, run.py stops with EOFError at a prompt nobody can answer. Without the --vllm-override-args line the server starts but cannot do tool calling.

Pinning a server to specific chips

tt serve gives the container every /dev/tenstorrent node. run.py --device-id 2,3 gives it only chips 2 and 3, which lets two servers share the machine. In our test, Llama-3.1-8B served on chips 2 and 3 (one P300 board) while Qwen3-Coder-Next kept running on chips 0 and 1; the Opening user mode device driver and Fabric initialized on 2 devices lines in the server log confirm which chips it opened. Which board holds which chip numbers is in tt device status (the # column) and tt device info (the board_id row). If you share the machine with other people or agents, take a lease first (gozer) so two servers never claim the same chips.

Verifying the Server

Once the server reports ready, confirm it’s working:

# List available models — start here, and use the id it prints
curl -s http://localhost:8000/v1/models | python3 -m json.tool

# First chat completion
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain tensor parallelism in one sentence."}
    ],
    "max_tokens": 120
  }' | python3 -m json.tool

The response JSON has the generated text at choices[0].message.content. If you get a connection refused, the server isn’t ready yet — give it another 30 seconds.

OpenAI Python SDK

The server is API-compatible with OpenAI’s client library. Point base_url at localhost:8000 and set api_key to any non-empty string — the server ignores it.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-checked"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "What is the Tenstorrent NOC fabric?"}
    ],
    max_tokens=256,
    temperature=0.7
)

print(response.choices[0].message.content)

This is the integration point for any application that already talks to OpenAI. Change the base URL, change the model name, and the rest of the code runs unchanged.

Streaming Responses

For applications that need to show text as it generates — chat interfaces, interactive tools — use the streaming mode:

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Describe continuous batching."}],
    stream=True
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

print()  # newline at end

Each chunk arrives as a server-sent event; the OpenAI SDK unwraps them into delta objects. The pattern is identical to streaming from api.openai.com — because it’s the same API.

Continuous Batching

This is one of the QB2’s practical advantages in production. vLLM’s continuous batching algorithm fills the KV-cache space as requests arrive, packing multiple users’ decode steps into the same chip invocation. You’re not running one request at a time — the server is interleaving decode steps from multiple concurrent clients across every chip cycle.

For single-user interactive work, this doesn’t matter. For serving a team, an API endpoint, or anything with concurrent load, it means the throughput numbers scale with parallelism rather than collapsing under it. A second concurrent user adds very little overhead up to the throughput ceiling of the chip.

Port Map

Keep these ports clear. Other services on the QB2 use them.

If port 8000 is already in use when you try to start vLLM, check for a running tt-studio or tt-inference-server instance first.

Remote Access via SSH Port Forward

The vLLM server listens on localhost only by default. To access it from another machine on your network — or from your laptop over SSH — use port forwarding:

# Run this on your laptop / remote machine
# Forwards your local port 8000 to the QB2's port 8000
ssh -L 8000:localhost:8000 your-user@your-qb2-hostname

# Now on your laptop, this works:
curl http://localhost:8000/v1/models

Keep the SSH session open while you use the forwarded port. For a persistent setup, look at autossh or tmux to keep the tunnel alive.

Multi-Chip: Using All Four Chips

On the managed path you don’t choose the mesh — run.py derives it per model from its spec. For a 70B across all four Blackhole chips that is just:

tt serve Llama-3.3-70B-Instruct --device p300x2

The model weights distribute across all four chips’ DRAM, and the KV cache is allocated per chip across the mesh. From the client’s perspective the API is identical — same URL, same request format.

Setting the mesh by hand is a direct-vLLM concern — MESH_DEVICE, its accepted names, and the case difference from --tt-device are covered in Build Your Own vLLM Environment. One thing carries over to this path regardless:

When Startup Stalls

Startup has long silent stretches, so “stuck” and “working” look alike in the log. Two quiet phases are normal: building rotary-embedding matrices, which prints nothing at all, and the warmup prefill sweeps that follow weight loading. Warmup also emits TT_FATAL: Only TILE layout is supported for BFLOAT8_B dtype! at critical level, sometimes hundreds of times — it is caught internally and startup continues. Worth knowing if you script a log watcher, since grepping for FATAL will trip on it.

A real device hang looks different: the log stops, CPU sits near 100%, and RSS stops moving entirely. Real work moves RSS. Confirm with a native stack dump:

uv tool install py-spy   # isolated install — not into the vLLM venv you're diagnosing
py-spy dump --native --pid "$(pgrep -n -f 'VLLM::EngineCore')"

-n on pgrep matters if more than one matching process is running: py-spy --pid takes exactly one PID, and an unfiltered pgrep -f can print several.

A hang in the command queue is unmistakable — pthread_cond_wait under FDMeshCommandQueue::wait_for_outstanding_reads, meaning the device never acknowledged a write. --native matters: the plain Python stack only shows ttnn.from_torch and looks like ordinary work.

The usual cause is stale state from a run that did not exit cleanly, and a reset clears it:

sudo lsof -w /dev/tenstorrent/*   # must print nothing before you continue
tt-smi -r                          # reset all boards

lsof exits 1 when nothing holds the devices, so for this check non-zero is the answer you want.

Going further

Two things this chapter deliberately leaves out, because each is a project rather than a step:

Next: Performance Tuning →