--- myst: html_meta: product-name: tt (Tenstorrent CLI), tt-cli, TT-Studio, TT-Inference-Server, TT-QuietBox®, TT-LoudBox, Wormhole™ Networked AI Processor, Blackhole® Networked AI Processor, n150s, n150d, n300s, n300d, p100a, p150a, p150b, p300a technology-concepts: model serving, LLM, diffusion, image generation, video generation, speech recognition, text to speech, computer vision, embeddings, OpenAI-compatible API, Hugging Face document-type: how-to --- # Serve Models This page shows how to serve a model on your Tenstorrent system and start sending it requests. You have two surfaces to work from, and both draw on the same catalog of models that Tenstorrent validates on each hardware generation: * **The `tt` CLI.** `tt serve ` gives you an OpenAI-compatible API endpoint in one command. This is the path for applications, scripts, agents, and anything else that talks to an API. * **TT-Studio.** A web application that serves the same models and gives you a browser interface for them: chat, image and video generation, speech-to-text, object detection, and more. This is the path when you want to try a model from a browser, or hand it to people who do not use a terminal. For LLMs that are going into production, there is a third path: running the vLLM server from [TT-Inference-Server](https://github.com/tenstorrent/tt-inference-server) directly, which gives you a pinned release, an authenticated endpoint, and control over the server configuration. [Step 4](#step-4-serve-in-production) covers it. The flow is the same whatever kind of model you serve. The catalog is not limited to large language models: it includes diffusion models for image and video generation, speech and audio models, decision models, and tabular foundation models, and `tt serve` and TT-Studio treat each of them the same way. Under the hood, `tt serve` runs [TT-Inference-Server](https://github.com/tenstorrent/tt-inference-server) for the models in Tenstorrent's tested catalog, hands models to TT-Studio, and runs community bundles published on the Hugging Face Hub through [tt-model-manager](https://github.com/tenstorrent/tt-model-manager). ## **The Same Steps for Every Model** Serving any model in the catalog takes the same five steps, and each is one command: | Step | Command | What it does | | :--- | :--- | :--- | | 1. See what runs on your system | `tt model list` | Lists the models validated for the hardware it detects. `--type` narrows the list to one kind of model. | | 2. Check what the model needs | `tt model info ` | Reports the disk and memory the model needs, which surfaces serve it, and how it is validated on your hardware. | | 3. Serve it | `tt serve `, or `tt serve --studio` for TT-Studio | Downloads the weights and the container image on first use, opens the devices, loads the model, and starts the server. | | 4. Send a request | `curl http://localhost:20000/v1/` | Every server speaks the OpenAI API on port 20000. The route depends on the kind of model. | | 5. Manage and stop | `tt model ps`, `tt model logs `, `tt model stop ` | Shows what is running, streams its log, and stops the server. The weights stay cached. | The commands do not change from one model to the next. What changes is what they report: the resources a model needs, the stages `tt serve` passes through while it loads, the route you call, and what comes back. The rest of this page walks the steps with `Wan2.2-T2V-A14B-Diffusers`, the Wan 2.2 text-to-video diffusion model, as the example, and says where another kind of model behaves differently. The example needs four chips, so it runs on a TT-QuietBox 2, a TT-LoudBox, or a Galaxy; on a single-chip board, follow the same steps with a model from your own list. --- ## **Before You Begin** :::{admonition} Important :class: warning * This guide assumes you have completed [Installing Tenstorrent Software](./README.md): `tt update` has run, `tt device status` lists your devices, and Docker runs without `sudo`. * You are logged in to Hugging Face with `tt model login`. Most models download their weights from the Hub, and the login is needed even for models that have no license gate. `tt model login` writes the same token store as the Hub's own `hf auth login`, which is the command `tt serve` names when the token is missing; either one satisfies it, as does an exported `HF_TOKEN`. * Disk space depends on the model. `tt model info ` reports the minimum for each one, from 15 GB for a speech or image model to 160 GB for a 70B-parameter LLM. Downloaded weights are cached under `~/.cache/huggingface` and shared by every path that serves the model. ::: ### **(Wormhole™ only)** :::{admonition} Wormhole multi-card systems need a mesh topology :class: warning On a Wormhole TT-QuietBox® or TT-LoudBox™, run `tt-topology -ls` and confirm it reports `Mesh`. If it reports `Isolated`, complete [Step 4 of Installing Tenstorrent Software](./README.md#step-4-wormhole-only-configure-the-mesh-topology) first; it takes about two minutes and resets the chips. Without it, every serve on this page stops at `validating system software` with `ModelSpec requires SystemTopology.MESH, detected ISOLATED`. ::: --- ## **Step 1: Choose a Model** ### **See what runs on your system** `tt model list` shows every model that is validated for the hardware it detects, from two sources: the released catalog that Tenstorrent ships and tests, and community bundles that anyone has published for Tenstorrent hardware. ```bash tt model list ``` Each row names the model, its `source` (`tt-inference-server` or `tt-studio` for the catalog, `HuggingFace` for a community bundle), the `engine` that runs it (`vLLM` for language models, `media` for image, video, audio, and embedding models, and `forge` for the vision and embedding models that run through TT-Forge), the smallest board or mesh each capability needs, and whether the weights are already on disk. Narrow the list to one kind of model with `--type`: | Kind of model | `--type` | Examples in the catalog | | :--- | :--- | :--- | | Large language models | `llm` | `Qwen3.8-27B`, `Llama-3.3-70B-Instruct`, `gpt-oss-120b` | | Vision-language models | `vlm` | `Qwen2.5-VL-7B-Instruct`, `Llama-3.2-11B-Vision-Instruct` | | Image generation (diffusion) | `image` | `FLUX.1-schnell`, `FLUX.1-dev`, `stable-diffusion-xl-base-1.0` | | Video generation (diffusion) | `video` | `Wan2.2-T2V-A14B-Diffusers`, `mochi-1-preview` | | Speech recognition | `audio` | `whisper-large-v3`, `distil-large-v3` | | Text to speech | `text_to_speech` | `speecht5_tts` | | Vision classification and detection | `cnn` | `yolox_nano`, `resnet-50`, `segformer` | | Embeddings | `embedding` | `bge-m3`, `Qwen3-Embedding-4B` | :::{admonition} Wormhole :class: note On a Wormhole system, a model can be listed and still be refused by `tt serve`. Each model is pinned to a container image, and the inference server only runs images at or above a minimum version; when the pin is older, the serve stops at `host ready` with `Image v0.4.0 is not supported (need v0.11.0+)`. To check a model before you download anything, run `tt serve --dry-run` and compare the version at the start of its `docker image` tag with the minimum the server requires. The same pin is listed on each model's page in the [tt-inference-server model support tables](https://github.com/tenstorrent/tt-inference-server/blob/main/docs/model_support/models_by_hardware.md#wh-loudboxquietbox), and the minimum is set in the server's [setup validation](https://github.com/tenstorrent/tt-inference-server/blob/main/workflows/validate_setup.py). ::: The example model on this page is a video model. Listing that kind on a TT-QuietBox 2 gives a list of one: ```bash tt model list --type video ``` ``` Models (tt catalog + community) for p300x2 (detected — `tt model list --all` for every device/bundle) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓ ┃ name ┃ source ┃ engine ┃ serving profiles ┃ weights ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩ │ Wan2.2-T2V-A14B-Diffusers │ tt-inference-server │ media │ p150x4, p300x2 │ — │ └───────────────────────────┴─────────────────────┴────────┴──────────────────┴─────────┘ ``` A model that is known not to work on your board is left out. `tt model list --all` shows every model on every device, and `--hw p150x4` shows the list for another configuration without touching the hardware. `--catalog` and `--community` show one source at a time. ### **Check what a model needs** Before you commit to a download, inspect the model. `tt model info ` reports where the weights come from, which surfaces can serve the model, what it needs from the host, and how it is validated on each hardware configuration. For the example model: ```bash tt model info Wan2.2-T2V-A14B-Diffusers ``` ``` Wan2.2-T2V-A14B-Diffusers ┌─────────────────────┬───────────────────────────────────────────────────────────────────────────────────┐ │ hf_repo │ Wan-AI/Wan2.2-T2V-A14B-Diffusers │ │ type │ video │ │ engines │ media │ │ via │ inference-server, studio │ │ servable │ yes — `tt serve Wan2.2-T2V-A14B-Diffusers` (`--studio` deploys it with TT-Studio) │ │ parameters │ 14B │ │ min disk │ 60 GB │ │ min ram │ 32 GB │ │ cached │ no │ │ on t3k │ media — COMPLETE, max context 65536 │ │ on galaxy │ media — COMPLETE, max context 65536 │ │ on p150x4 │ media — COMPLETE, max context 65536 │ │ on p150x8 │ media — COMPLETE, max context 65536 │ │ on p300x2 │ media — COMPLETE, max context 65536 │ │ on blackhole_galaxy │ media — COMPLETE, max context 65536 │ └─────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘ ``` ### **Get access to gated models** Some model owners gate their weights behind a license. Meta's Llama models, Google's Gemma models, and Black Forest Labs' FLUX.1 models all require you to accept their terms on the model's Hub page before you can download them, and the owner may take days to grant access. Qwen, Wan, and Whisper models are not gated. To get access, open the `hf_repo` page that `tt model info` shows, accept the license, and make sure the account you accepted with is the one you used for `tt model login`. Until access is granted, serving a gated model stops at the download: ``` ✓ host ready tt-inference-server 0.25.0 ✗ downloading weights black-forest-labs/FLUX.1-schnell ╭──────────────────────────── error [TOOL_FAILED] ────────────────────────────╮ │ FLUX.1-schnell stopped before the server was ready. │ │ AssertionError: ⛔ Error during: … hf download black-forest-labs/FLUX.1-schnell │ ╰────────────────────────────────────────────────────────────────────────────╯ ``` The full log it names ends with `Error: Access denied. Model 'black-forest-labs/FLUX.1-schnell' requires approval.` :::{tip} `tt model pull ` downloads a model's weights ahead of time, so the serve itself is faster. It is also a quick way to confirm that your access to a gated model works. ::: --- ## **Step 2: Serve with the CLI** ### **Preview, then serve** `tt serve --dry-run` shows how the model would be served without starting anything: the serving path, the device configuration, the container image, where the weights go, and the port. ```bash tt serve Wan2.2-T2V-A14B-Diffusers --dry-run ``` ``` ✓ Detecting device configuration p300x2 (override with --device) tt serve Wan2.2-T2V-A14B-Diffusers — dry run ┌──────────────────┬───────────────────────────────────────────────────────────────────────────────┐ │ backend │ tt-inference-server │ │ device │ p300x2 │ │ engines │ media │ │ status │ COMPLETE │ │ docker image │ ghcr.io/tenstorrent/tt-media-inference-server:0.17.0-8c48a10 │ │ weights cache │ /home/ttuser/.cache/huggingface │ │ HF token │ from the Hugging Face login store │ │ port │ 20000 (server default) │ └──────────────────┴───────────────────────────────────────────────────────────────────────────────┘ ``` `tt serve ` serves the model. On first use it fetches the weights and pulls the container image, then opens the devices, loads the model, and starts the server, showing each stage as a live checklist. It returns once the endpoint answers requests: ```bash tt serve Wan2.2-T2V-A14B-Diffusers ``` ``` ✓ host ready tt-inference-server 0.25.0 ✓ weights ready Wan-AI/Wan2.2-T2V-A14B-Diffusers 16m 58s ✓ image ready tt-media-inference-server:0.17.0 2m 13s ✓ container started tt-inference-server-2f018ce0 3.7s ✓ server started 4.5s ✓ Tenstorrent device opened 4 chips 4.3s ✓ model loaded 2/2 5m 52s ✓ endpoint answering ╭─ Wan2.2-T2V-A14B-Diffusers ready ──────────────────────────────────────────╮ │ endpoint http://127.0.0.1:20000/v1 │ │ Logs · tt model logs Wan2.2-T2V-A14B-Diffusers --follow │ │ Stop · tt model stop Wan2.2-T2V-A14B-Diffusers │ ╰────────────────────────────────────────────────────────────────────────────╯ ``` On a Wormhole TT-LoudBox or TT-QuietBox the device line reads `t3k` and `Tenstorrent device opened` reports 8 chips, one per n300 half; the full first serve of this model took about 32 minutes there. `tt serve --dry-run` is the quickest way to see which configuration `tt` detected. ### **Send a request** The endpoint speaks the OpenAI API, so any OpenAI-compatible client can use it. Point the client at `http://localhost:20000/v1` with any value for the API key. Which route you call depends on the kind of model: | Kind of model | Route | What you send | | :--- | :--- | :--- | | Large language and vision-language models | `/v1/chat/completions` | a list of `messages` | | Speech recognition | `/v1/audio/transcriptions` | the audio file, base64-encoded, in `file` | | Text to speech | `/v1/audio/speech` | the `text` to speak | | Image generation | `/v1/images/generations` | a `prompt`, with optional `num_inference_steps`, `seed`, `width` and `height` | | Video generation | `/v1/videos/generations` | a `prompt`; the reply is a job that you poll and then download | | Embeddings | `/v1/embeddings` | the `input` text | For an LLM, `tt model curl` sends a chat completion without any JSON on your part and prints the server's JSON reply, and `--print` shows the equivalent `curl` command for your own scripts: ```bash tt model curl "In one sentence, what is Tenstorrent?" --max-tokens 2048 ``` For the other kinds of model, call the route directly; `tt model curl` always posts to `/v1/chat/completions`, so against an image, video, audio, or embedding server it returns `{"detail":"Not Found"}`. Video generation runs as a job: the request returns at once with the job's `id` and `status`, you poll the job until it is `completed`, and then you download the result. ```bash # Submit the job; the reply includes its id and status curl -s http://localhost:20000/v1/videos/generations \ -H "Content-Type: application/json" \ -d '{"prompt": "A red fox trotting through fresh snow in a pine forest, soft morning light, camera tracking alongside"}' # Poll until status is "completed", then download the clip curl -s http://localhost:20000/v1/videos/generations/ curl http://localhost:20000/v1/videos/generations//download --output fox.mp4 ``` ### **Manage what is running** ```bash tt model ps # what is being served, on which port, and whether it is healthy tt model logs # the server's log (--follow to stream it) tt model stop # stop the server; the weights stay cached ``` `tt serve` takes `--port` to serve on a port other than 20000, and `--device` to override the detected configuration. Anything it does not recognize after `--` is passed through to the server itself. --- ## **Step 3: Serve with TT-Studio** [TT-Studio](https://github.com/tenstorrent/tt-studio) serves the same catalog through a web application. It deploys the model you choose and gives you a browser interface for it: chat with knowledge bases for language models, image and video generation for diffusion models, speech-to-text, object detection, and a voice agent. Every model it deploys is also served on an API endpoint, so applications can use it at the same time. ### **Start TT-Studio and deploy a model** `tt serve --studio` deploys a model with TT-Studio instead of directly. TT-Studio takes port 3000 and the devices for itself, so stop anything `tt serve` or `tt launch` started and reset the devices first: ```bash tt model stop Wan2.2-T2V-A14B-Diffusers tt device reset tt serve Wan2.2-T2V-A14B-Diffusers --studio ``` On first use `tt` fetches the latest TT-Studio release, and TT-Studio asks you to accept its terms and offers a `tt-studio` shell shortcut. It then prints its web UI address and, once the model is deployed, the model's endpoint: ``` ╭─ TT Studio is ready ───────────────────────────────────────────────────────╮ │ URL ● http://localhost:3000 │ │ Hardware QuietBox (QB2) · 4 device(s) │ ╰────────────────────────────────────────────────────────────────────────────╯ ╭─ Wan2.2-T2V-A14B-Diffusers is ready ───────────────────────────────────────╮ │ Chips 0, 1, 2, 3 │ │ Endpoint ● http://localhost:20000/v1/videos/generations │ │ Web UI http://localhost:3000/models-deployed │ ╰────────────────────────────────────────────────────────────────────────────╯ ``` Without a model name, `tt serve --studio` lists everything TT-Studio can deploy on your system and asks which one you want. ### **Deploy from the browser instead** Open `http://localhost:3000`, go to the models page, and pick a model from the list, which is filtered to what your hardware supports. The **Models Deployed** page lists every running model with its health, the chips it occupies, and its log. ### **Use the model** The Models Deployed page links to the interface for the model's kind. The endpoint in the ready box serves API clients at the same time, and `tt model curl` and `tt launch` work against it exactly as they do against a model started by `tt serve`. ### **Stop** `tt model stop ` stops a model that TT-Studio deployed and takes TT-Studio down with it, and takes it down even when no model is deployed; settings, weights, chat history, and knowledge bases are kept, but TT-Studio's own run log is not, so read it before you stop if a deploy went wrong. The `tt-studio` shortcut manages TT-Studio directly: ```bash tt-studio --stop-model # stop one model and keep TT-Studio running tt-studio --stop # stop TT-Studio, keeping its data ``` --- ## **Serve a Community Model** Community bundles are models that anyone has packaged for Tenstorrent hardware and published on the Hugging Face Hub. `tt model list` includes the ones Tenstorrent has reviewed; `--include-unverified` shows the rest. They are served through the same commands, with the bundle's repository id in place of a catalog name: ```bash tt model list --community # bundles that fit your hardware tt model info / # the bundle's manifest and a compatibility verdict tt serve / # served through tt-model-manager ``` `tt` checks that the bundle was built for your chip architecture and that your system has enough chips before it serves, and asks for confirmation before it serves a bundle that has not been verified. Tenstorrent does not test or maintain community bundles. If you have brought a model up on Tenstorrent hardware yourself, the [tt-model-manager documentation](https://github.com/tenstorrent/tt-cli/blob/main/docs/community-models.md) explains how to package and publish it so that anyone can serve it the same way. --- ## **Step 4: Serve in Production** We currently use vLLM to serve LLMs for production applications. Now we walk through how to deploy a vLLM-powered API endpoint using the [tt-inference-server](https://github.com/tenstorrent/tt-inference-server) project. `tt serve` runs the same server for you; running it yourself pins the server to a tagged release, puts an API key in front of the endpoint, and lets you set the server configuration when you launch it. This step is an alternative to Steps 2 and 3, not a continuation of them, and it applies to LLMs only. The server needs the devices to itself, so stop anything `tt serve` or TT-Studio started and reset the devices before you begin: ```bash tt model stop tt device reset ``` Two things differ from the endpoints above: this server listens on port 8000 rather than 20000, and every request needs the API key you generate below. ### **Configure the vLLM server** #### **Select your hardware** Run the following script to specify your hardware. This script sets the required environment variables and selects the recommended model for your system. ```bash select_device_and_model(){ echo -e "\nSelect a Tenstorrent system from the list below:"; PS3=$'\n#? '; options=("TT-QuietBox (Wormhole)" "TT-QuietBox (Blackhole)" "TT-LoudBox (Wormhole)" "TT-LoudBox (Blackhole)" "n150s" "n150d" "n300s" "n300d" "p100a" "p150a" "p150b" "Quit"); select opt in "${options[@]}"; do case "$opt" in "TT-QuietBox (Wormhole)") DEVICE="t3k"; MODEL="Llama-3.3-70B-Instruct";; "TT-QuietBox (Blackhole)") DEVICE="p150x4"; MODEL="Llama-3.3-70B-Instruct";; "TT-LoudBox (Wormhole)") DEVICE="t3k"; MODEL="Llama-3.3-70B-Instruct";; "TT-LoudBox (Blackhole)") DEVICE="p150x8"; MODEL="Llama-3.3-70B-Instruct";; "n150s"|"n150d") DEVICE="n150"; MODEL="Llama-3.1-8B-Instruct";; "n300s"|"n300d") DEVICE="n300"; MODEL="Llama-3.1-8B-Instruct";; "p100a") DEVICE="p100"; MODEL="Llama-3.1-8B-Instruct";; "p150a"|"p150b") DEVICE="p150"; MODEL="Llama-3.1-8B-Instruct";; "Quit") echo "❌ Exiting without setting any environment variables."; return;; *) echo "❌ Invalid option. Try again."; continue; esac; export DEVICE MODEL; echo -e "\n✅ DEVICE set to '$DEVICE'"; echo "✅ MODEL set to '$MODEL'"; break; done; }; select_device_and_model ``` #### **Check model access** Execute this script to confirm you can access the recommended model's weights: ```bash check_hf_access() { [ -z "$MODEL" ] && { printf "✖ Error: Please provide a Hugging Face repository ID.\n"; return 1; }; ! command -v curl &>/dev/null && { printf "✖ Error: curl is not installed.\n"; return 1; }; local REPO_ID="meta-llama/$MODEL"; local TOKEN=${HF_TOKEN:-$(cat "$HOME/.cache/huggingface/token" 2>/dev/null)}; [ -z "$TOKEN" ] && printf "ℹ️ Info: No Hugging Face token found.\n You can only access public repositories.\n"; local AUTH_HEADER=""; [ -n "$TOKEN" ] && AUTH_HEADER="Authorization: Bearer $TOKEN"; printf "Checking access for: %s...\n" "$REPO_ID"; local URL="https://huggingface.co/$REPO_ID/resolve/main/config.json"; local HTTP_CODE=$(curl -s -L -o /dev/null -w "%{http_code}" -H "$AUTH_HEADER" "$URL"); case $HTTP_CODE in 200) printf "✔ Access granted.\n";; 401) printf "✖ Access denied (401 Unauthorized).\n This is a private or gated repository.\n Ensure your token is valid and has the correct permissions.\n";; 403) printf "✖ Access forbidden (403 Forbidden).\n The repository is gated.\n You need to visit the repository page on Hugging Face and request access.\n";; 404) printf "✖ Repository or 'config.json' not found (404 Not Found).\n Please check if the repository ID '$REPO_ID' is correct.\n";; *) printf "✖ Failed to check access.\n Received HTTP status code: %s\n" "$HTTP_CODE";; esac; }; HF_HUB_DISABLE_XET=1; check_hf_access; ``` If the command does not succeed and print `✔ Access granted.`, make sure you are logged in to the Hugging Face Hub and have access to the model, as described in [Get access to gated models](#get-access-to-gated-models). #### **Clone the server repository** ```bash # 1. Clone the repository git clone https://github.com/tenstorrent/tt-inference-server.git cd tt-inference-server # 2. Update all tags from the remote (standardizing the local list) git fetch --tags # 3. Checkout the highest Semantic Version tag starting with "v" git checkout $(git tag -l "v*" --sort=-v:refname | head -n 1) ``` ### **Launch the vLLM server** #### **Set the JWT secret** This string is used to seed the generation of your server's API key. ```bash export JWT_SECRET="testing" ``` #### **Run the deployment script** Execute the following command. The script prompts you for configuration details; in most cases, you may accept the default values. ```bash python3 run.py --model $MODEL --device $DEVICE --workflow server --docker-server ``` :::{Important} The first time you run this command, it will download the model's weights. This download can take more than 30 minutes. ::: #### **Wait for the server to initialize** After the download completes, the server will start initializing in a docker container. :::{Important} The first time you start the server, the initialization process for a 70B model should take about 40 minutes. For an 8B model it should take about 10 minutes. ::: ### **Test the server endpoint** #### **Check server health** Use the following command to check if the server is ready. ```bash check_server_health(){ code=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/health); exit_code=$?; if [[ $exit_code -ne 0 ]]; then echo "❌ Error: Unable to connect to server at localhost:8000"; elif [[ $code -eq 200 ]]; then echo "✅ Server is ready (HTTP 200)"; else echo "⚠️ Server responded with status: $code"; fi; }; check_server_health ``` Wait until the output indicates `✅ Server is ready (HTTP 200)`. #### **Generate an API key** Your `JWT_SECRET` is used to create an API key for authenticating requests. ```bash python3 -m venv request-venv source request-venv/bin/activate pip3 install --upgrade pip pip install pyjwt==2.7.0 export VLLM_API_KEY=$(python3 -c 'import os; import json; import jwt; json_payload = json.loads("{\"team_id\": \"tenstorrent\", \"token_id\": \"debug-test\"}"); encoded_jwt = jwt.encode(json_payload, os.environ["JWT_SECRET"], algorithm="HS256"); print(encoded_jwt)') ``` #### **Send an example request** The vLLM server exposes an OpenAI-compatible API. The first request will be slow as it performs a final warmup. ```bash # Warmup request curl -sS "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $VLLM_API_KEY" \ -d "{ \"model\": \"meta-llama/$MODEL\", \"prompt\": \"San Francisco is a\", \"max_tokens\": 50, \"temperature\": 0 }" | jq ``` Run the command again to observe the server respond at full speed. --- ## **Troubleshooting** ### The model I want is not in `tt model list` The list is filtered to your detected hardware. `tt model list --all` shows every model on every device, and `tt model info ` says why one is left out for yours. A model that is validated on a smaller configuration than you have, such as a single-chip model on a TT-QuietBox 2, is listed and runs on part of the system. ### The serve stops at `downloading weights` with "requires approval" The model is gated and your Hugging Face account has not been granted access to it yet. Accept the license on the model's Hub page, wait for the owner to approve the request if the model requires that, and serve again. Make sure `tt model login` is for the same account. ### `tt serve` fails with "Access to model … is restricted" The model is gated and your account has not accepted its license yet, or the login on this machine is for a different account. Accept the license on the model's Hub page, check `tt model login`, and serve again. `tt model logs ` shows the server's own output for any failed start. ### A serve or a TT-Studio deploy fails with "Try resetting the board" A model server that was stopped or crashed can leave the devices in a state the next server cannot use. Reset every device, confirming with `y` when asked, and serve again: ```bash tt device reset ``` ### `tt serve` says the model did not report ready within 60 minutes `tt serve` waits up to an hour for the endpoint to answer, then stops watching and leaves the server running. A cold start of a large diffusion model, or of a 70B-class or larger LLM on a Wormhole system, compiles for your hardware and can be slow, so first check whether it is still making progress: `tt model logs --follow` shows the server's output, and `tt model ps` reports it `healthy` once it answers. If the log shows an error instead, stop the server with `tt model stop `. To wait longer next time, set `TT_SERVE_READY_TIMEOUT` to a number of seconds before you serve; on Wormhole, `TT_SERVE_READY_TIMEOUT=5400` is a safe default for the largest models. ### A TT-Studio deploy stays at 0% If the **Models Deployed** page shows "Starting deployment… 0%" for more than a few minutes, `tt model ps` lists nothing, and `docker ps` shows only the six `tt_studio_*` containers, the deploy job failed before it started a container and TT-Studio did not report it. The reason is in `~/.local/share/tenstorrent/tools/tt-studio-v*/src/logs/model_run.log`; `invalid choice` for the model name means TT-Studio's bundled inference server does not support that model yet, so pick another. Read the log first: `tt model stop ` removes it when it stops TT-Studio. ### TT-Studio does not start because port 3000 is in use TT-Studio needs port 3000 and does not move to another one. Stop whatever is using it first, for example a web UI started with `tt launch`: `tt launch stop openwebui`. ### I am running out of disk space `tt model rm ` removes a model's local artifacts and keeps its weights; add `--include-weights` to remove those too. The weights live under `~/.cache/huggingface`. --- ## **What to Do Next** Now that your model is running: * **Connect an application to it** with `tt launch`. [Open WebUI](https://github.com/open-webui/open-webui) and [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) give you a chat interface in the browser; OpenCode, Aider, pi, Qwen Code, and Hermes Agent turn a tool-calling LLM into a coding agent in your terminal. `tt launch list` shows which are available on your machine, and the [client launchers guide](https://github.com/tenstorrent/tt-cli/blob/main/docs/client-launchers.md) covers what each one needs and how to connect to a server on another machine. * **Watch it in real time** — run `tt-toplike` in a second terminal window. [TT-Toplike](https://docs.tenstorrent.com/tt-toplike/) shows live core activity, power draw, and memory traffic on the chips while they serve requests. * **Use it as a coding assistant** — the [Coding Assistant with Aider](https://docs.tenstorrent.com/tt-vscode-toolkit/lessons/coding-assistant/) lesson walks through pairing a local endpoint with Aider for on-device pair programming. * **Build agents on top of it** — if you have a TT-QuietBox® 2, the [Local AI Agents on QuietBox 2](https://docs.tenstorrent.com/tt-vscode-toolkit/lessons/qb2-local-agents/) lesson covers tool-calling pipelines and multi-agent workflows against your local endpoint. * **Generate images and video at scale** — if you have a TT-QuietBox® 2, [TT-Local-Generator](https://docs.tenstorrent.com/tt-local-generator/) runs alongside inference with its own queue and gallery. * **Check supported models** — the [tt-inference-server repository](https://github.com/tenstorrent/tt-inference-server) is the authoritative source for which models are validated on each hardware generation, and the catalog `tt model list` shows is built from it. --- ## **Need Additional Support?** If you encounter any issues, or have a question that isn't covered in the documentation, please [raise a support request.](https://tenstorrent.atlassian.net/servicedesk/customer/portal/1) Our team will review your request and provide assistance.