Serve Without Containers

tt serve runs a model in a Docker container. The Tenstorrent Cloud pod has no Docker, so this lesson serves the same model from what the pod already has: a built tt-metal and a vLLM environment with the Tenstorrent plugin installed, at ~/tt-metal/build/python_env_vllm. It ends at the same /v1 endpoint, so the Connect Software lessons work the same way afterwards.

This is also the route on any machine without Docker, provided it has a built tt-metal and the plugin; Build TT-Metalium from Source and the plugin's own README cover that.

Log In to Hugging Face

Llama-3.1-8B-Instruct is gated. Accept the license on its Hugging Face page, then:

tt model login

tt model login

On the pod, HF_HOME is ~/models, so the token and the weights both land there, where hf, the plugin and tt all look.

Serve

source ~/tt-metal/build/python_env_vllm/bin/activate
export PYTHONPATH=$HOME/tt-metal/tools:$HOME/tt-metal/ttnn:$PYTHONPATH
MESH_DEVICE=N150 HF_MODEL=meta-llama/Llama-3.1-8B-Instruct \
  vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768 --max-num-seqs 32 --block-size 64 --host 127.0.0.1 --port 20000

source ~/tt-metal/build/python_env_vllm/bin/activate && export PYTHONPATH=$HOME/tt-metal/tools:$HOME/tt-metal/ttnn:$PYTHONPATH && MESH_DEVICE=N150 HF_MODEL=meta-llama/Llama-3.1-8B-Instruct vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 32768 --max-num-seqs 32 --block-size 64 --host 127.0.0.1 --port 20000

HF_MODEL repeats the model name because tt-metal's model code reads it from the environment. --max-num-seqs 32 --block-size 64 are the plugin's own example settings; the vLLM default of 256 sequences is refused on Tenstorrent hardware.

⚠️ Image version matters. This route works on tt-code-server images built from tt-metal v0.80.0 or later with the plugin pinned to match (verified 2026-10-29 with a chat completion on hardware). On the v0.63.0 image, which pairs tt-metal v0.69.0 with vLLM 0.30, the server fails at Model architectures ['TTLlamaForCausalLM'] failed to be inspected, and no flag fixes it. Check with cd ~/tt-metal && git describe --tags. If you are on the old image, the Build on the Stack lessons do not need a server; come back to this one after the pod is refreshed.

MESH_DEVICE names the board the plugin should open: N150 on the pod, N300, T3K, P100, P150 and so on elsewhere. --max-model-len 32768 is the plugin's documented setting for Llama 3.1 8B on an n150. --host 127.0.0.1 --port 20000 matches where tt serve puts a model, so the Connect Software lessons find it in the same place.

The first run downloads the weights (16 GB) and compiles the kernels for your chip; both are cached, so the next start is minutes. The server is ready when the log says Application startup complete and Uvicorn running on http://0.0.0.0:20000.

⚠️ The endpoint has no password. --host 127.0.0.1 keeps it on this machine, which is where tt launch and the Connect Software lessons reach it. Without that flag vllm serve binds every interface. If another machine must reach it, set --api-key to require a key and treat the host with the same care as in Serve a Model.

Ask It a Question

In a second terminal:

curl -s http://localhost:20000/v1/models | python3 -m json.tool | grep '"id"'

curl -s http://localhost:20000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain what an API is in one sentence."}],
    "max_tokens": 256
  }' | python3 -m json.tool

curl -s http://localhost:20000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Explain what an API is in one sentence."}], "max_tokens": 256}' | python3 -m json.tool

The text is in choices[0].message.content. The endpoint speaks the standard OpenAI API, so any client works. tt launch in the next group finds servers tt started; for this one you give the client the URL yourself (http://127.0.0.1:20000/v1), which each Connect Software lesson shows.

Stop

Ctrl-C in the terminal running vllm serve. There is no container to clean up.

Go Deeper

tt-inference-server can also run without Docker (run.py --local-server), pointed at a built tt-metal and this same venv; that is how tt serve would run on a host without Docker, with its model specs and warm-up. It is covered in tt-inference-server by Hand. The plugin's own options (MESH_DEVICE, --additional-config '{"tt": {...}}', vision models, speculative decoding) are in the vllm-tt-plugin README.

Next: Launch Software →