Serve Models
This page shows how to serve a model on your Tenstorrent system and start sending it requests. You have two surfaces to work from, and both draw on the same catalog of models that Tenstorrent validates on each hardware generation:
The
ttCLI.tt serve <model>gives you an OpenAI-compatible API endpoint in one command. This is the path for applications, scripts, agents, and anything else that talks to an API.TT-Studio. A web application that serves the same models and gives you a browser interface for them: chat, image and video generation, speech-to-text, object detection, and more. This is the path when you want to try a model from a browser, or hand it to people who do not use a terminal.
For LLMs that are going into production, there is a third path: running the vLLM server from TT-Inference-Server directly, which gives you a pinned release, an authenticated endpoint, and control over the server configuration. Step 4 covers it.
The flow is the same whatever kind of model you serve. The catalog is not limited to large language models: it includes diffusion models for image and video generation, speech and audio models, decision models, and tabular foundation models, and tt serve and TT-Studio treat each of them the same way. Under the hood, tt serve runs TT-Inference-Server for the models in Tenstorrent’s tested catalog, hands models to TT-Studio, and runs community bundles published on the Hugging Face Hub through tt-model-manager.
The Same Steps for Every Model
Serving any model in the catalog takes the same five steps, and each is one command:
Step |
Command |
What it does |
|---|---|---|
1. See what runs on your system |
|
Lists the models validated for the hardware it detects. |
2. Check what the model needs |
|
Reports the disk and memory the model needs, which surfaces serve it, and how it is validated on your hardware. |
3. Serve it |
|
Downloads the weights and the container image on first use, opens the devices, loads the model, and starts the server. |
4. Send a request |
|
Every server speaks the OpenAI API on port 20000. The route depends on the kind of model. |
5. Manage and stop |
|
Shows what is running, streams its log, and stops the server. The weights stay cached. |
The commands do not change from one model to the next. What changes is what they report: the resources a model needs, the stages tt serve passes through while it loads, the route you call, and what comes back. The rest of this page walks the steps with Wan2.2-T2V-A14B-Diffusers, the Wan 2.2 text-to-video diffusion model, as the example, and says where another kind of model behaves differently. The example needs four chips, so it runs on a TT-QuietBox 2, a TT-LoudBox, or a Galaxy; on a single-chip board, follow the same steps with a model from your own list.
Before You Begin
Important
This guide assumes you have completed Installing Tenstorrent Software:
tt updatehas run,tt device statuslists your devices, and Docker runs withoutsudo.You are logged in to Hugging Face with
tt model login. Most models download their weights from the Hub, and the login is needed even for models that have no license gate.tt model loginwrites the same token store as the Hub’s ownhf auth login, which is the commandtt servenames when the token is missing; either one satisfies it, as does an exportedHF_TOKEN.Disk space depends on the model.
tt model info <model>reports the minimum for each one, from 15 GB for a speech or image model to 160 GB for a 70B-parameter LLM. Downloaded weights are cached under~/.cache/huggingfaceand shared by every path that serves the model.
(Wormhole™ only)
Wormhole multi-card systems need a mesh topology
On a Wormhole TT-QuietBox® or TT-LoudBox™, run tt-topology -ls and confirm it reports Mesh. If it reports Isolated, complete Step 4 of Installing Tenstorrent Software first; it takes about two minutes and resets the chips. Without it, every serve on this page stops at validating system software with ModelSpec requires SystemTopology.MESH, detected ISOLATED.
Step 1: Choose a Model
See what runs on your system
tt model list shows every model that is validated for the hardware it detects, from two sources: the released catalog that Tenstorrent ships and tests, and community bundles that anyone has published for Tenstorrent hardware.
tt model list
Each row names the model, its source (tt-inference-server or tt-studio for the catalog, HuggingFace for a community bundle), the engine that runs it (vLLM for language models, media for image, video, audio, and embedding models, and forge for the vision and embedding models that run through TT-Forge), the smallest board or mesh each capability needs, and whether the weights are already on disk. Narrow the list to one kind of model with --type:
Kind of model |
|
Examples in the catalog |
|---|---|---|
Large language models |
|
|
Vision-language models |
|
|
Image generation (diffusion) |
|
|
Video generation (diffusion) |
|
|
Speech recognition |
|
|
Text to speech |
|
|
Vision classification and detection |
|
|
Embeddings |
|
|
Wormhole
On a Wormhole system, a model can be listed and still be refused by tt serve. Each model is pinned to a container image, and the inference server only runs images at or above a minimum version; when the pin is older, the serve stops at host ready with Image v0.4.0 is not supported (need v0.11.0+). To check a model before you download anything, run tt serve <model> --dry-run and compare the version at the start of its docker image tag with the minimum the server requires. The same pin is listed on each model’s page in the tt-inference-server model support tables, and the minimum is set in the server’s setup validation.
The example model on this page is a video model. Listing that kind on a TT-QuietBox 2 gives a list of one:
tt model list --type video
Models (tt catalog + community) for p300x2 (detected — `tt model list --all` for every
device/bundle)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ name ┃ source ┃ engine ┃ serving profiles ┃ weights ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ Wan2.2-T2V-A14B-Diffusers │ tt-inference-server │ media │ p150x4, p300x2 │ — │
└───────────────────────────┴─────────────────────┴────────┴──────────────────┴─────────┘
A model that is known not to work on your board is left out. tt model list --all shows every model on every device, and --hw p150x4 shows the list for another configuration without touching the hardware. --catalog and --community show one source at a time.
Check what a model needs
Before you commit to a download, inspect the model. tt model info <model> reports where the weights come from, which surfaces can serve the model, what it needs from the host, and how it is validated on each hardware configuration. For the example model:
tt model info Wan2.2-T2V-A14B-Diffusers
Wan2.2-T2V-A14B-Diffusers
┌─────────────────────┬───────────────────────────────────────────────────────────────────────────────────┐
│ hf_repo │ Wan-AI/Wan2.2-T2V-A14B-Diffusers │
│ type │ video │
│ engines │ media │
│ via │ inference-server, studio │
│ servable │ yes — `tt serve Wan2.2-T2V-A14B-Diffusers` (`--studio` deploys it with TT-Studio) │
│ parameters │ 14B │
│ min disk │ 60 GB │
│ min ram │ 32 GB │
│ cached │ no │
│ on t3k │ media — COMPLETE, max context 65536 │
│ on galaxy │ media — COMPLETE, max context 65536 │
│ on p150x4 │ media — COMPLETE, max context 65536 │
│ on p150x8 │ media — COMPLETE, max context 65536 │
│ on p300x2 │ media — COMPLETE, max context 65536 │
│ on blackhole_galaxy │ media — COMPLETE, max context 65536 │
└─────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘
Get access to gated models
Some model owners gate their weights behind a license. Meta’s Llama models, Google’s Gemma models, and Black Forest Labs’ FLUX.1 models all require you to accept their terms on the model’s Hub page before you can download them, and the owner may take days to grant access. Qwen, Wan, and Whisper models are not gated.
To get access, open the hf_repo page that tt model info shows, accept the license, and make sure the account you accepted with is the one you used for tt model login. Until access is granted, serving a gated model stops at the download:
✓ host ready tt-inference-server 0.25.0
✗ downloading weights black-forest-labs/FLUX.1-schnell
╭──────────────────────────── error [TOOL_FAILED] ────────────────────────────╮
│ FLUX.1-schnell stopped before the server was ready. │
│ AssertionError: ⛔ Error during: … hf download black-forest-labs/FLUX.1-schnell │
╰────────────────────────────────────────────────────────────────────────────╯
The full log it names ends with Error: Access denied. Model 'black-forest-labs/FLUX.1-schnell' requires approval.
Tip
tt model pull <model> downloads a model’s weights ahead of time, so the serve itself is faster. It is also a quick way to confirm that your access to a gated model works.
Step 2: Serve with the CLI
Preview, then serve
tt serve <model> --dry-run shows how the model would be served without starting anything: the serving path, the device configuration, the container image, where the weights go, and the port.
tt serve Wan2.2-T2V-A14B-Diffusers --dry-run
✓ Detecting device configuration p300x2 (override with --device)
tt serve Wan2.2-T2V-A14B-Diffusers — dry run
┌──────────────────┬───────────────────────────────────────────────────────────────────────────────┐
│ backend │ tt-inference-server │
│ device │ p300x2 │
│ engines │ media │
│ status │ COMPLETE │
│ docker image │ ghcr.io/tenstorrent/tt-media-inference-server:0.17.0-8c48a10 │
│ weights cache │ /home/ttuser/.cache/huggingface │
│ HF token │ from the Hugging Face login store │
│ port │ 20000 (server default) │
└──────────────────┴───────────────────────────────────────────────────────────────────────────────┘
tt serve <model> serves the model. On first use it fetches the weights and pulls the container image, then opens the devices, loads the model, and starts the server, showing each stage as a live checklist. It returns once the endpoint answers requests:
tt serve Wan2.2-T2V-A14B-Diffusers
✓ host ready tt-inference-server 0.25.0
✓ weights ready Wan-AI/Wan2.2-T2V-A14B-Diffusers 16m 58s
✓ image ready tt-media-inference-server:0.17.0 2m 13s
✓ container started tt-inference-server-2f018ce0 3.7s
✓ server started 4.5s
✓ Tenstorrent device opened 4 chips 4.3s
✓ model loaded 2/2 5m 52s
✓ endpoint answering
╭─ Wan2.2-T2V-A14B-Diffusers ready ──────────────────────────────────────────╮
│ endpoint http://127.0.0.1:20000/v1 │
│ Logs · tt model logs Wan2.2-T2V-A14B-Diffusers --follow │
│ Stop · tt model stop Wan2.2-T2V-A14B-Diffusers │
╰────────────────────────────────────────────────────────────────────────────╯
On a Wormhole TT-LoudBox or TT-QuietBox the device line reads t3k and Tenstorrent device opened reports 8 chips, one per n300 half; the full first serve of this model took about 32 minutes there. tt serve --dry-run is the quickest way to see which configuration tt detected.
Send a request
The endpoint speaks the OpenAI API, so any OpenAI-compatible client can use it. Point the client at http://localhost:20000/v1 with any value for the API key. Which route you call depends on the kind of model:
Kind of model |
Route |
What you send |
|---|---|---|
Large language and vision-language models |
|
a list of |
Speech recognition |
|
the audio file, base64-encoded, in |
Text to speech |
|
the |
Image generation |
|
a |
Video generation |
|
a |
Embeddings |
|
the |
For an LLM, tt model curl sends a chat completion without any JSON on your part and prints the server’s JSON reply, and --print shows the equivalent curl command for your own scripts:
tt model curl "In one sentence, what is Tenstorrent?" --max-tokens 2048
For the other kinds of model, call the route directly; tt model curl always posts to /v1/chat/completions, so against an image, video, audio, or embedding server it returns {"detail":"Not Found"}. Video generation runs as a job: the request returns at once with the job’s id and status, you poll the job until it is completed, and then you download the result.
# Submit the job; the reply includes its id and status
curl -s http://localhost:20000/v1/videos/generations \
-H "Content-Type: application/json" \
-d '{"prompt": "A red fox trotting through fresh snow in a pine forest, soft morning light, camera tracking alongside"}'
# Poll until status is "completed", then download the clip
curl -s http://localhost:20000/v1/videos/generations/<job_id>
curl http://localhost:20000/v1/videos/generations/<job_id>/download --output fox.mp4
Manage what is running
tt model ps # what is being served, on which port, and whether it is healthy
tt model logs <model> # the server's log (--follow to stream it)
tt model stop <model> # stop the server; the weights stay cached
tt serve takes --port to serve on a port other than 20000, and --device to override the detected configuration. Anything it does not recognize after -- is passed through to the server itself.
Step 3: Serve with TT-Studio
TT-Studio serves the same catalog through a web application. It deploys the model you choose and gives you a browser interface for it: chat with knowledge bases for language models, image and video generation for diffusion models, speech-to-text, object detection, and a voice agent. Every model it deploys is also served on an API endpoint, so applications can use it at the same time.
Start TT-Studio and deploy a model
tt serve <model> --studio deploys a model with TT-Studio instead of directly. TT-Studio takes port 3000 and the devices for itself, so stop anything tt serve or tt launch started and reset the devices first:
tt model stop Wan2.2-T2V-A14B-Diffusers
tt device reset
tt serve Wan2.2-T2V-A14B-Diffusers --studio
On first use tt fetches the latest TT-Studio release, and TT-Studio asks you to accept its terms and offers a tt-studio shell shortcut. It then prints its web UI address and, once the model is deployed, the model’s endpoint:
╭─ TT Studio is ready ───────────────────────────────────────────────────────╮
│ URL ● http://localhost:3000 │
│ Hardware QuietBox (QB2) · 4 device(s) │
╰────────────────────────────────────────────────────────────────────────────╯
╭─ Wan2.2-T2V-A14B-Diffusers is ready ───────────────────────────────────────╮
│ Chips 0, 1, 2, 3 │
│ Endpoint ● http://localhost:20000/v1/videos/generations │
│ Web UI http://localhost:3000/models-deployed │
╰────────────────────────────────────────────────────────────────────────────╯
Without a model name, tt serve --studio lists everything TT-Studio can deploy on your system and asks which one you want.
Deploy from the browser instead
Open http://localhost:3000, go to the models page, and pick a model from the list, which is filtered to what your hardware supports. The Models Deployed page lists every running model with its health, the chips it occupies, and its log.
Use the model
The Models Deployed page links to the interface for the model’s kind. The endpoint in the ready box serves API clients at the same time, and tt model curl and tt launch work against it exactly as they do against a model started by tt serve.
Stop
tt model stop <model> stops a model that TT-Studio deployed and takes TT-Studio down with it, and takes it down even when no model is deployed; settings, weights, chat history, and knowledge bases are kept, but TT-Studio’s own run log is not, so read it before you stop if a deploy went wrong. The tt-studio shortcut manages TT-Studio directly:
tt-studio --stop-model <model> # stop one model and keep TT-Studio running
tt-studio --stop # stop TT-Studio, keeping its data
Serve a Community Model
Community bundles are models that anyone has packaged for Tenstorrent hardware and published on the Hugging Face Hub. tt model list includes the ones Tenstorrent has reviewed; --include-unverified shows the rest. They are served through the same commands, with the bundle’s repository id in place of a catalog name:
tt model list --community # bundles that fit your hardware
tt model info <namespace>/<name> # the bundle's manifest and a compatibility verdict
tt serve <namespace>/<name> # served through tt-model-manager
tt checks that the bundle was built for your chip architecture and that your system has enough chips before it serves, and asks for confirmation before it serves a bundle that has not been verified. Tenstorrent does not test or maintain community bundles. If you have brought a model up on Tenstorrent hardware yourself, the tt-model-manager documentation explains how to package and publish it so that anyone can serve it the same way.
Step 4: Serve in Production
We currently use vLLM to serve LLMs for production applications. Now we walk through how to deploy a vLLM-powered API endpoint using the tt-inference-server project. tt serve runs the same server for you; running it yourself pins the server to a tagged release, puts an API key in front of the endpoint, and lets you set the server configuration when you launch it.
This step is an alternative to Steps 2 and 3, not a continuation of them, and it applies to LLMs only. The server needs the devices to itself, so stop anything tt serve or TT-Studio started and reset the devices before you begin:
tt model stop <model>
tt device reset
Two things differ from the endpoints above: this server listens on port 8000 rather than 20000, and every request needs the API key you generate below.
Configure the vLLM server
Select your hardware
Run the following script to specify your hardware. This script sets the required environment variables and selects the recommended model for your system.
select_device_and_model(){ echo -e "\nSelect a Tenstorrent system from the list below:"; PS3=$'\n#? '; options=("TT-QuietBox (Wormhole)" "TT-QuietBox (Blackhole)" "TT-LoudBox (Wormhole)" "TT-LoudBox (Blackhole)" "n150s" "n150d" "n300s" "n300d" "p100a" "p150a" "p150b" "Quit"); select opt in "${options[@]}"; do case "$opt" in "TT-QuietBox (Wormhole)") DEVICE="t3k"; MODEL="Llama-3.3-70B-Instruct";; "TT-QuietBox (Blackhole)") DEVICE="p150x4"; MODEL="Llama-3.3-70B-Instruct";; "TT-LoudBox (Wormhole)") DEVICE="t3k"; MODEL="Llama-3.3-70B-Instruct";; "TT-LoudBox (Blackhole)") DEVICE="p150x8"; MODEL="Llama-3.3-70B-Instruct";; "n150s"|"n150d") DEVICE="n150"; MODEL="Llama-3.1-8B-Instruct";; "n300s"|"n300d") DEVICE="n300"; MODEL="Llama-3.1-8B-Instruct";; "p100a") DEVICE="p100"; MODEL="Llama-3.1-8B-Instruct";; "p150a"|"p150b") DEVICE="p150"; MODEL="Llama-3.1-8B-Instruct";; "Quit") echo "❌ Exiting without setting any environment variables."; return;; *) echo "❌ Invalid option. Try again."; continue; esac; export DEVICE MODEL; echo -e "\n✅ DEVICE set to '$DEVICE'"; echo "✅ MODEL set to '$MODEL'"; break; done; }; select_device_and_model
Check model access
Execute this script to confirm you can access the recommended model’s weights:
check_hf_access() { [ -z "$MODEL" ] && { printf "✖ Error: Please provide a Hugging Face repository ID.\n"; return 1; }; ! command -v curl &>/dev/null && { printf "✖ Error: curl is not installed.\n"; return 1; }; local REPO_ID="meta-llama/$MODEL"; local TOKEN=${HF_TOKEN:-$(cat "$HOME/.cache/huggingface/token" 2>/dev/null)}; [ -z "$TOKEN" ] && printf "ℹ️ Info: No Hugging Face token found.\n You can only access public repositories.\n"; local AUTH_HEADER=""; [ -n "$TOKEN" ] && AUTH_HEADER="Authorization: Bearer $TOKEN"; printf "Checking access for: %s...\n" "$REPO_ID"; local URL="https://huggingface.co/$REPO_ID/resolve/main/config.json"; local HTTP_CODE=$(curl -s -L -o /dev/null -w "%{http_code}" -H "$AUTH_HEADER" "$URL"); case $HTTP_CODE in 200) printf "✔ Access granted.\n";; 401) printf "✖ Access denied (401 Unauthorized).\n This is a private or gated repository.\n Ensure your token is valid and has the correct permissions.\n";; 403) printf "✖ Access forbidden (403 Forbidden).\n The repository is gated.\n You need to visit the repository page on Hugging Face and request access.\n";; 404) printf "✖ Repository or 'config.json' not found (404 Not Found).\n Please check if the repository ID '$REPO_ID' is correct.\n";; *) printf "✖ Failed to check access.\n Received HTTP status code: %s\n" "$HTTP_CODE";; esac; }; HF_HUB_DISABLE_XET=1; check_hf_access;
If the command does not succeed and print ✔ Access granted., make sure you are logged in to the Hugging Face Hub and have access to the model, as described in Get access to gated models.
Clone the server repository
# 1. Clone the repository
git clone https://github.com/tenstorrent/tt-inference-server.git
cd tt-inference-server
# 2. Update all tags from the remote (standardizing the local list)
git fetch --tags
# 3. Checkout the highest Semantic Version tag starting with "v"
git checkout $(git tag -l "v*" --sort=-v:refname | head -n 1)
Launch the vLLM server
Set the JWT secret
This string is used to seed the generation of your server’s API key.
export JWT_SECRET="testing"
Run the deployment script
Execute the following command. The script prompts you for configuration details; in most cases, you may accept the default values.
python3 run.py --model $MODEL --device $DEVICE --workflow server --docker-server
Important
The first time you run this command, it will download the model’s weights. This download can take more than 30 minutes.
Wait for the server to initialize
After the download completes, the server will start initializing in a docker container.
Important
The first time you start the server, the initialization process for a 70B model should take about 40 minutes. For an 8B model it should take about 10 minutes.
Test the server endpoint
Check server health
Use the following command to check if the server is ready.
check_server_health(){ code=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/health); exit_code=$?; if [[ $exit_code -ne 0 ]]; then echo "❌ Error: Unable to connect to server at localhost:8000"; elif [[ $code -eq 200 ]]; then echo "✅ Server is ready (HTTP 200)"; else echo "⚠️ Server responded with status: $code"; fi; }; check_server_health
Wait until the output indicates ✅ Server is ready (HTTP 200).
Generate an API key
Your JWT_SECRET is used to create an API key for authenticating requests.
python3 -m venv request-venv
source request-venv/bin/activate
pip3 install --upgrade pip
pip install pyjwt==2.7.0
export VLLM_API_KEY=$(python3 -c 'import os; import json; import jwt; json_payload = json.loads("{\"team_id\": \"tenstorrent\", \"token_id\": \"debug-test\"}"); encoded_jwt = jwt.encode(json_payload, os.environ["JWT_SECRET"], algorithm="HS256"); print(encoded_jwt)')
Send an example request
The vLLM server exposes an OpenAI-compatible API. The first request will be slow as it performs a final warmup.
# Warmup request
curl -sS "http://localhost:8000/v1/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $VLLM_API_KEY" \
-d "{
\"model\": \"meta-llama/$MODEL\",
\"prompt\": \"San Francisco is a\",
\"max_tokens\": 50,
\"temperature\": 0
}" | jq
Run the command again to observe the server respond at full speed.
Troubleshooting
The model I want is not in tt model list
The list is filtered to your detected hardware. tt model list --all shows every model on every device, and tt model info <model> says why one is left out for yours. A model that is validated on a smaller configuration than you have, such as a single-chip model on a TT-QuietBox 2, is listed and runs on part of the system.
The serve stops at downloading weights with “requires approval”
The model is gated and your Hugging Face account has not been granted access to it yet. Accept the license on the model’s Hub page, wait for the owner to approve the request if the model requires that, and serve again. Make sure tt model login is for the same account.
tt serve fails with “Access to model … is restricted”
The model is gated and your account has not accepted its license yet, or the login on this machine is for a different account. Accept the license on the model’s Hub page, check tt model login, and serve again. tt model logs <model> shows the server’s own output for any failed start.
A serve or a TT-Studio deploy fails with “Try resetting the board”
A model server that was stopped or crashed can leave the devices in a state the next server cannot use. Reset every device, confirming with y when asked, and serve again:
tt device reset
tt serve says the model did not report ready within 60 minutes
tt serve waits up to an hour for the endpoint to answer, then stops watching and leaves the server running. A cold start of a large diffusion model, or of a 70B-class or larger LLM on a Wormhole system, compiles for your hardware and can be slow, so first check whether it is still making progress: tt model logs <model> --follow shows the server’s output, and tt model ps reports it healthy once it answers. If the log shows an error instead, stop the server with tt model stop <model>. To wait longer next time, set TT_SERVE_READY_TIMEOUT to a number of seconds before you serve; on Wormhole, TT_SERVE_READY_TIMEOUT=5400 is a safe default for the largest models.
A TT-Studio deploy stays at 0%
If the Models Deployed page shows “Starting deployment… 0%” for more than a few minutes, tt model ps lists nothing, and docker ps shows only the six tt_studio_* containers, the deploy job failed before it started a container and TT-Studio did not report it. The reason is in ~/.local/share/tenstorrent/tools/tt-studio-v*/src/logs/model_run.log; invalid choice for the model name means TT-Studio’s bundled inference server does not support that model yet, so pick another. Read the log first: tt model stop <model> removes it when it stops TT-Studio.
TT-Studio does not start because port 3000 is in use
TT-Studio needs port 3000 and does not move to another one. Stop whatever is using it first, for example a web UI started with tt launch: tt launch stop openwebui.
I am running out of disk space
tt model rm <model> removes a model’s local artifacts and keeps its weights; add --include-weights to remove those too. The weights live under ~/.cache/huggingface.
What to Do Next
Now that your model is running:
Connect an application to it with
tt launch. Open WebUI and AnythingLLM give you a chat interface in the browser; OpenCode, Aider, pi, Qwen Code, and Hermes Agent turn a tool-calling LLM into a coding agent in your terminal.tt launch listshows which are available on your machine, and the client launchers guide covers what each one needs and how to connect to a server on another machine.Watch it in real time — run
tt-toplikein a second terminal window. TT-Toplike shows live core activity, power draw, and memory traffic on the chips while they serve requests.Use it as a coding assistant — the Coding Assistant with Aider lesson walks through pairing a local endpoint with Aider for on-device pair programming.
Build agents on top of it — if you have a TT-QuietBox® 2, the Local AI Agents on QuietBox 2 lesson covers tool-calling pipelines and multi-agent workflows against your local endpoint.
Generate images and video at scale — if you have a TT-QuietBox® 2, TT-Local-Generator runs alongside inference with its own queue and gallery.
Check supported models — the tt-inference-server repository is the authoritative source for which models are validated on each hardware generation, and the catalog
tt model listshows is built from it.
Need Additional Support?
If you encounter any issues, or have a question that isn’t covered in the documentation, please raise a support request. Our team will review your request and provide assistance.