Production Inference with TT-Inference-Server

tt serve runs this for you. Under the hood it calls tt-inference-server's run.py with the model spec's settings; tt serve <model> --dry-run prints the exact command. This lesson is how to run run.py yourself, for the things tt serve does not expose: pin which chips it uses (--device-id 0,1), bind the port to loopback (--bind-host 127.0.0.1), pass a different image (--override-docker-image), run from an older or newer release, or run without a container at all (--local-server, against a tt-metal you built or the cloud pod's).

Give run.py a model name and your hardware type and it handles everything: pulls the right Docker image (pre-built TT-Metalium^™ + vLLM), downloads model weights, and starts an OpenAI-compatible inference server.

TT-QuietBox^® 2 / p300c users: in the v0.25.0 release catalog, Llama-3.1-8B's only single-chip Blackhole entry is p150 (there is no p100 entry; --device p100 fails with No model spec matches). A p300c chip serves as p150: verified 2026-10-29 with --local-server --device p150 on one chip of a p300c. A TT-QuietBox 2 is one four-chip ring mesh (P300_X2); for whole-box serving of a larger model use --device p300x2, or tt serve, which picks it.

Prerequisites

Where run.py Lives

Three copies can exist, and they are different versions. Use the one that matches the job:

tt fetches its copy the first time tt serve runs, so on a fresh install the first row is empty until then. The buttons in this lesson use the newest copy under ~/.local/share/tenstorrent/tools/.

Clone it yourself (the cloud pod has no tt-managed copy and must not run tt update):

git clone --depth 1 --branch v0.25.0 https://github.com/tenstorrent/tt-inference-server.git ~/tt-inference-server
cd ~/tt-inference-server && uv venv --python 3.10 .venv && uv pip install --python .venv/bin/python pyyaml packaging

git clone --depth 1 --branch v0.25.0 https://github.com/tenstorrent/tt-inference-server.git ~/tt-inference-server && cd ~/tt-inference-server && uv venv --python 3.10 .venv && uv pip install --python .venv/bin/python pyyaml packaging

That venv is all run.py itself needs (it is what tt installs too: packaging and pyyaml). run.py builds the per-workflow environments on first use. Pick the tag from the releases page; v0.25.0 is what tt 1.2.1 pins.

Other Prerequisites

• HF token — Llama is gated on HuggingFace. Set once:

  export HF_TOKEN=hf_...          # your HuggingFace access token
  

• Docker — the server runs in a container. Verify:

  docker --version
  

• Hardware detected — confirm your card is visible:

tt-smi

echo "=== Checking Prerequisites ===" && which docker && docker --version && ls ~/.local/share/tenstorrent/tools/tt-inference-server-*/src/run.py ~/tt-inference-server/run.py 2>/dev/null | tail -1 && tt-smi -s | python3 -c "import sys,json; d=json.load(sys.stdin); [print(f'  board: {c["board_type"]}') for c in d["device_info"]]" 2>/dev/null || tt-smi && echo "=== ✓ Prerequisites OK ==="

The Model: Llama-3.1-8B

Llama-3.1-8B is the widest-coverage model in TT-Inference-Server — it runs on every current Tenstorrent board:

Two weight variants are available via MODEL_SPECS:

• Llama-3.1-8B — base model (default)
• Llama-3.1-8B-Instruct — instruction-tuned (use for chat)

Start the Server

Option A — Automated (run.py)

run.py selects the correct Docker image for your hardware, handles the volume, and downloads weights inside the container on first run.

--device can be omitted and will be auto-detected from your hardware via tt-smi:

cd ~/.local/lib/tt-inference-server

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --workflow server \
  --docker-server \
  --no-auth

Add --device <device> if auto-detection doesn't match your hardware.

Hardware-specific commands

n150 (Wormhole™ — single chip, 64 K context)

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device n150 \
  --workflow server \
  --docker-server \
  --no-auth

D=$(ls -d ~/.local/share/tenstorrent/tools/tt-inference-server-*/src 2>/dev/null | sort -V | tail -1); cd "${D:?tt has not installed tt-inference-server yet; run tt serve once first}" && ../venv/bin/python run.py --model Llama-3.1-8B-Instruct --device n150 --workflow server --docker-server --no-auth

n300 (Wormhole — dual chip, 128 K context)

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device n300 \
  --workflow server \
  --docker-server \
  --no-auth

D=$(ls -d ~/.local/share/tenstorrent/tools/tt-inference-server-*/src 2>/dev/null | sort -V | tail -1); cd "${D:?tt has not installed tt-inference-server yet; run tt serve once first}" && ../venv/bin/python run.py --model Llama-3.1-8B-Instruct --device n300 --workflow server --docker-server --no-auth

T3000 — WH TT-QuietBox / LoudBox (8 chips, 128 K context)

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device t3k \
  --workflow server \
  --docker-server \
  --no-auth

p100 / p300c / TT-QuietBox 2 (Blackhole® — 64 K context, Experimental)

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device p100 \
  --workflow server \
  --docker-server \
  --no-auth

This treats each p300c chip as an independent p100 device for single-chip serving — you can run one server per chip this way, each on a different --service-port. That's a deliberate choice, not the machine's shape: a TT-QuietBox 2 is one four-chip ring mesh (P300_X2), and its chips are not independent devices with no interconnect. For whole-box multi-chip serving of a single larger model instead of four separate single-chip servers, use --device p300x2 — see vLLM Production.

Option B — Direct docker run

For full transparency, or when you want to run without run.py, use the container directly. Pass --model and --device as container args; the container resolves the config from its bundled model spec catalog.

Wormhole (n150 / n300 / T3000):

docker run \
  --env "HF_TOKEN=$HF_TOKEN" \
  --ipc host \
  --publish 8000:8000 \
  --device /dev/tenstorrent \
  --mount type=bind,src=/dev/hugepages-1G,dst=/dev/hugepages-1G \
  --volume volume_id_Llama-3.1-8B:/home/container_app_user/cache_root \
  ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.12.0-25305db-6e67d2d \
  --model Llama-3.1-8B-Instruct \
  --device n150

Change --device to n300 or t3k for those boards — same image.

Blackhole (p100 / p300c / TT-QuietBox 2):

docker run \
  --env "HF_TOKEN=$HF_TOKEN" \
  --ipc host \
  --publish 8000:8000 \
  --device /dev/tenstorrent \
  --mount type=bind,src=/dev/hugepages-1G,dst=/dev/hugepages-1G \
  --volume volume_id_Llama-3.1-8B:/home/container_app_user/cache_root \
  ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.12.0-55fd115-aa4ae1e \
  --model Llama-3.1-8B-Instruct \
  --device p100

Use --print-docker-cmd with run.py to see the exact command it would run:

python3 run.py --model Llama-3.1-8B-Instruct --device n150 \
  --workflow server --docker-server --no-auth --print-docker-cmd

First Run

⏱️ First run: 10–20 minutes — Docker image pull (10 GB) + model weight download (16 GB) inside the container.

⏱️ Subsequent runs: 2–5 minutes — image and weights are cached.

Watch for this in run.py output when the container is up:

INFO: Created Docker container ID: 6b8c7038a44a
INFO: Access container logs via: docker logs -f 6b8c7038a44a
INFO: Stop running container via: docker stop 6b8c7038a44a

Then watch the container logs until vLLM is ready:

docker logs -f 6b8c7038a44a

Look for:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Test the Server

Once vLLM is ready, the server exposes a standard OpenAI-compatible API.

Quick test:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Llama-3.1-8B-Instruct",
       "prompt": "Tenstorrent accelerators are designed for",
       "max_tokens": 60}'

curl -X POST http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "Llama-3.1-8B-Instruct", "prompt": "Explain what a Tenstorrent AI accelerator is in one sentence.", "max_tokens": 50, "temperature": 0.7}'

Streaming (token-by-token):

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Llama-3.1-8B-Instruct",
       "prompt": "Write a haiku about silicon:",
       "max_tokens": 40,
       "stream": true}'

curl -X POST http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "Llama-3.1-8B-Instruct", "prompt": "Write a haiku about AI acceleration:", "max_tokens": 100, "stream": true}'

Python client (OpenAI SDK):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "What is a Tenstorrent accelerator?"}],
    max_tokens=100,
)
print(response.choices[0].message.content)

cat > ~/tt-scratchpad/tt-inference-client.py << 'EOF'
from openai import OpenAI

# Point to the vLLM server started by tt-inference-server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # Not used, but required by SDK
)

def query_inference_server(prompt, max_tokens=100, temperature=0.7):
    """Query vLLM server using OpenAI SDK"""
    try:
        response = client.completions.create(
            model="Llama-3.1-8B-Instruct",
            prompt=prompt,
            max_tokens=max_tokens,
            temperature=temperature
        )
        
        generated_text = response.choices[0].text
        print(f"Generated text: {generated_text}")
        print(f"Tokens: {response.usage.total_tokens}")
        return response
    except Exception as e:
        print(f"Error: {e}")
        return None

if __name__ == "__main__":
    query_inference_server(
        "Explain quantum computing to a 5-year-old:",
        max_tokens=100,
        temperature=0.8
    )
EOF
chmod +x ~/tt-scratchpad/tt-inference-client.py && echo "✓ Created ~/tt-scratchpad/tt-inference-client.py (uses OpenAI SDK)"

Managing the Running Server

# List running containers
docker ps

# Follow logs in real time
docker logs -f <container-id>

# Stop server
docker stop <container-id>

# Stop all tt-inference-server containers at once
docker ps --filter ancestor=ghcr.io/tenstorrent/tt-inference-server \
  --format '{{.ID}}' | xargs docker stop

Beyond the Server: Other Workflows

With the container already running, client-side workflows can run against it without restarting:

cd ~/.local/lib/tt-inference-server

# Quick smoke test (reduced samples)
python3 run.py --model Llama-3.1-8B-Instruct --device n150 \
  --workflow benchmarks --limit-samples-mode smoke-test

# Full accuracy evals
python3 run.py --model Llama-3.1-8B-Instruct --device n150 \
  --workflow evals

# Benchmarks + evals + reports in one pass (release certification)
python3 run.py --model Llama-3.1-8B-Instruct --device n150 \
  --workflow release --docker-server --no-auth

Results land in ~/.local/lib/tt-inference-server/workflow_logs/.

Tuning vLLM Arguments

TT-Inference-Server's model specs set reasonable defaults for each model/device pair (block size 64, full context window, 32 concurrent sequences). Override any of them without rebuilding the container.

Tool Use / Function Calling

Enable the OpenAI tool-calling API by passing two flags to vLLM:

Via run.py:

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device n150 \
  --workflow server \
  --docker-server \
  --no-auth \
  --vllm-override-args '{"enable-auto-tool-choice": true, "tool-call-parser": "llama3_json"}'

Via direct docker run (remaining args pass straight through to vllm serve):

docker run ... \
  ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.12.0-25305db-6e67d2d \
  --model Llama-3.1-8B-Instruct \
  --device n150 \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json

Parser by model family:

Current limitation: tool_choice="none" and tool_choice="required" are not yet supported in the TT vLLM fork. Only tool_choice="auto" works reliably.

Once the server is running with tool choice enabled, use the API normally:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather in a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

response = client.chat.completions.create(
    model="Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "What's the weather in Austin?"}],
    tools=tools,
    tool_choice="auto",
)
print(response.choices[0].message.tool_calls)

Reducing Context Length

By default the server uses the full context window supported by the hardware (64 K on n150/p100, 128 K on n300/T3000). Reducing it lowers DRAM usage and can speed up model load:

# Via run.py
python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device n150 \
  --workflow server \
  --docker-server \
  --no-auth \
  --vllm-override-args '{"max-model-len": 8192}'

# Via docker run (passthrough)
... <image> --model Llama-3.1-8B-Instruct --device n150 --max-model-len 8192

max-model-len must be a multiple of block-size (default 64). Values like 4096, 8192, 16384, 32768 all work cleanly.

Concurrency and Batch Limits

The model spec sets max-num-seqs (concurrent in-flight sequences) and max-num-batched-tokens (tokens per forward pass). Lower them to reduce peak memory or raise them when throughput matters more than latency:

# Reduce to 8 concurrent users (lower memory, lower throughput)
--vllm-override-args '{"max-num-seqs": 8}'

# Increase for high-throughput batch workloads (n300/T3000 only — needs headroom)
--vllm-override-args '{"max-num-seqs": 64, "max-num-batched-tokens": 65536}'

Defaults for Llama-3.1-8B: max-num-seqs=32, max-num-batched-tokens=65536 (n150) / 131072 (n300/T3000).

Combining Multiple Overrides

JSON keys are merged, so all overrides can go in one --vllm-override-args:

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device n150 \
  --workflow server \
  --docker-server \
  --no-auth \
  --vllm-override-args '{
    "enable-auto-tool-choice": true,
    "tool-call-parser": "llama3_json",
    "max-model-len": 16384,
    "max-num-seqs": 8
  }'

Use --print-docker-cmd to verify the generated docker command before launching:

python3 run.py ... --print-docker-cmd

Models Outside MODEL_SPECS

The container resolves model configuration from a bundled model_spec.json with 60+ validated models. If your model isn't there, you have three options:

1. Check by short name

The container resolves both full HF repo IDs and short names (the last segment of the path). So Llama-3.1-8B-Instruct and meta-llama/Llama-3.1-8B-Instruct both work:

# These are equivalent
... --model Llama-3.1-8B-Instruct --device n150
... --model meta-llama/Llama-3.1-8B-Instruct --device n150

The full catalog includes Llama, Qwen, Mistral, Gemma, DeepSeek, Whisper, Stable Diffusion, FLUX, Mochi video, and more — see model support docs.

2. Run vLLM directly (no MODEL_SPECS constraint)

The vLLM Production lesson shows how to run vLLM directly on the host without TT-Inference-Server. This accepts any model path or HF repo and gives you full control over every vLLM flag — useful for models in development or private repos.

3. Request official support

Open an issue at TT-Inference-Server issue tracker to request a new model be added to MODEL_SPECS. Include your hardware type, model name, and any performance requirements.

Non-Container Deployment (--local-server)

With a built tt-metal and a vLLM venv beside it, run.py can run the server on the host, no Docker. The cloud pod is exactly that machine: ~/tt-metal is built and ~/tt-metal/build/python_env_vllm has vLLM and the Tenstorrent plugin.

cd ~/tt-inference-server && source .venv/bin/activate
export HF_TOKEN=$(cat ~/models/token)       # run.py asks for it; tt model login stored it here
python run.py \
  --model Llama-3.1-8B-Instruct \
  --device n150 \
  --workflow server \
  --local-server \
  --tt-metal-home ~/tt-metal \
  --tt-metal-python-venv-dir ~/tt-metal/build/python_env_vllm \
  --host-hf-cache ~/models \
  --no-auth --service-port 20000

cd ~/tt-inference-server && source .venv/bin/activate && export HF_TOKEN=$(cat ~/models/token) && python run.py --model Llama-3.1-8B-Instruct --device n150 --workflow server --local-server --tt-metal-home ~/tt-metal --tt-metal-python-venv-dir ~/tt-metal/build/python_env_vllm --host-hf-cache ~/models --no-auth --service-port 20000

⚠️ --no-auth and every interface. This server takes no key and vLLM binds 0.0.0.0; anyone who can reach port 20000 can use the model. On the pod only the pod's own network reaches it. Anywhere else, front it with a reverse proxy or firewall, or drop --no-auth and set JWT_SECRET, as run.py --help describes.

run.py asks for HF_TOKEN at a prompt when the variable is unset, which a button's terminal can answer, but exporting it first is simpler; on the pod tt model login wrote it to ~/models/token ($HF_HOME/token). The run prints the exact run_vllm_api_server.py command it starts, waits on /health, and reports the server's PID and log path; verified 2026-10-29 on one Blackhole chip as --device p150: ready in about 3 minutes from a warm cache, chat completion answered, and tt launch list saw the server.

--local-server uses REPO_ROOT/persistent_volume/ for logs and caches and runs as you, with no Docker volume permissions to manage. --service-port 20000 matches tt serve, so tt launch finds the server. Compared with the plain vllm serve in Serve Without Containers, this route adds the model spec's settings and warm-up; the plain route is fewer moving parts.

Image version matters here too. --local-server imports tt-metal's model code into the vLLM venv, the same pairing the plain vllm serve route needs, so it works on a tt-code-server image built from tt-metal v0.80.0 or later and fails the same way on v0.63.0 (see that lesson's callout).

HF Cache Tips

If you've already downloaded model weights (e.g. via hf download), point run.py at them to skip the in-container download:

# Reuse ~/.cache/huggingface (bare flag uses HF_HOME / ~/.cache/huggingface)
python3 run.py --model Llama-3.1-8B-Instruct --device n150 \
  --workflow server --docker-server --no-auth \
  --host-hf-cache

# Or point at a specific directory
python3 run.py --model Llama-3.1-8B-Instruct --device n150 \
  --workflow server --docker-server --no-auth \
  --host-weights-dir ~/models/meta-llama/Llama-3.1-8B-Instruct

Cache Persistence

TT-Inference-Server uses two separate cache directories inside the container — knowing how each is stored makes the difference between a 2-minute startup and a 10-minute one.

What gets cached where

cache_root/
  weights/{model_name}/                              # HF model weights
  tt_metal_cache/cache_{model_name}/{device_type}/   # compiled TT-Metalium kernels
  tt_dit_cache/                                      # compiled WAN/Mochi tensor weights
  logs/                                              # vLLM server logs

• TT-Metalium kernels (tt_metal_cache/) — compiled by vLLM on first run. Subsequent starts load from this cache: ~2–5 min instead of 10–20 min.
• Media model tensor weights (tt_dit_cache/) — compiled by video/image models (WAN 2.2, Mochi). Not cached by default — stored in /tmp/TT_DIT_CACHE inside the container and lost when the container stops.

Docker named volumes (default — TT-Metalium kernels)

By default run.py mounts a Docker named volume at cache_root. The TT-Metalium kernel cache survives container restarts automatically — no extra flags needed.

You can verify the volume exists after the first run:

docker volume ls | grep Llama

Persisting media model caches (TT_DIT_CACHE_DIR)

For video and image models (WAN 2.2, Mochi, FLUX) the container compiles tensor weights at startup and stores them in TT_DIT_CACHE_DIR. The default is /tmp/TT_DIT_CACHE, which is lost when the container stops.

First run without cache: ~525 seconds (WAN 2.2 on TT-QuietBox 2) Subsequent runs with cache: ~5 minutes

Move the cache under cache_root so it lives in the persistent Docker volume:

# In your .env file (or export before running)
TT_DIT_CACHE_DIR=/home/container_app_user/cache_root/tt_dit_cache

With run.py:

TT_DIT_CACHE_DIR=/home/container_app_user/cache_root/tt_dit_cache \
python3 run.py --model Wan-2.2-T2V-1.3B --device p100 \
  --workflow server --docker-server --no-auth

With direct docker run, pass it as -e:

docker run \
  -e "HF_TOKEN=$HF_TOKEN" \
  -e "TT_DIT_CACHE_DIR=/home/container_app_user/cache_root/tt_dit_cache" \
  --volume volume_id_Wan-2.2:/home/container_app_user/cache_root \
  ... <image> --model Wan-2.2-T2V-1.3B --device p100

Full host-side persistence (--host-volume)

To survive Docker image updates (which create new named volumes), bind the entire cache_root to a host directory. All weights and all caches land on the host filesystem:

# Ensure the host directory is writable by UID 1000 (container user)
sudo mkdir -p ~/tt-cache
sudo chown 1000 ~/tt-cache

python3 run.py \
  --model Llama-3.1-8B-Instruct \
  --device n150 \
  --workflow server \
  --docker-server \
  --no-auth \
  --host-volume ~/tt-cache

With --host-volume, TT_DIT_CACHE_DIR should still be set explicitly to keep it within the bound directory:

TT_DIT_CACHE_DIR=~/tt-cache/tt_dit_cache \
python3 run.py --model Wan-2.2-T2V-1.3B --device p100 \
  --workflow server --docker-server --no-auth \
  --host-volume ~/tt-cache

Skip HF hub checks at startup (HF_HUB_OFFLINE)

After weights are downloaded, the HF library still pings the hub at startup to check for updates. Disable this to cut several seconds off every startup:

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
python3 run.py --model Llama-3.1-8B-Instruct --device n150 \
  --workflow server --docker-server --no-auth

Or add to your .env:

HF_HUB_OFFLINE=1
TRANSFORMERS_OFFLINE=1

With direct docker run:

docker run \
  -e "HF_TOKEN=$HF_TOKEN" \
  -e "HF_HUB_OFFLINE=1" \
  -e "TRANSFORMERS_OFFLINE=1" \
  ... <image> --model Llama-3.1-8B-Instruct --device n150

Note: Set these only after the model is fully downloaded. With HF_HUB_OFFLINE=1 the container cannot fetch weights that aren't cached.

Sharing caches between --docker-server and --local-server

--docker-server writes caches as UID 1000 (the container user). --local-server writes as your host user. If you switch between them without fixing ownership, the other mode can't read the cache.

Fix ownership before switching:

# Switching from --docker-server → --local-server
sudo chown -R $USER ~/tt-cache

# Switching from --local-server → --docker-server
sudo chown -R 1000 ~/tt-cache

When using --local-server, caches land in REPO_ROOT/persistent_volume/ unless overridden. Point TT_DIT_CACHE_DIR at a shared path if you want --local-server to reuse what Docker compiled:

TT_DIT_CACHE_DIR=~/tt-cache/tt_dit_cache \
python3 run.py --model Wan-2.2-T2V-1.3B --device p100 \
  --workflow server --local-server \
  --tt-metal-home /opt/tt-metal

What's New in main (as of 2026-04-16)

Note: As of 2026-04-16, the dev branch has been retired. All active development now lands directly on main before release. Clone main for the latest features; it is the single development branch.

Recent additions merged to main:

C++ inference server with IPC

A new tt-cppserver backend provides a C++ server communicating with the vLLM Python layer via IPC, replacing the single-process Python server. Includes a MemoryManager that tracks device DRAM allocation and enables explicit cache management.

Session manager

A persistent session manager that maintains model state across requests — useful for multi-turn chat without re-loading the model between conversations.

Disaggregated prefill / decode

Separate prefill and decode stages can now run on different devices or pods, enabling higher throughput at scale. This matches the architecture used in large-scale deployments.

Grafana metrics dashboard

Container exposes Prometheus-compatible metrics at /metrics; a bundled Grafana dashboard visualises throughput, latency, and token rates. Connect Grafana at http://localhost:3000 after starting with the metrics profile.

OpenAI /v1/responses endpoint

The server now implements the newer OpenAI Responses API (in addition to /v1/completions and /v1/chat/completions), allowing compatibility with the latest OpenAI SDK streaming patterns.

Multi-host deployment

--multihost flag and companion documentation (docs/multihost_deployment.md) support deploying prefill and decode on separate machines connected via high-speed fabric.

Next Steps

• vLLM Production → — run vLLM directly without the workflow wrapper
• Launch Software → — connect a chat window or a coding agent to this server with tt launch
• TT-Inference-Server docs — full CLI reference