Models

Official model support

Models in tt-inference-server's release spec.

48 models
LLMOfficial

DeepSeek-R1-Distill-Llama-70B

deepseek-ai/DeepSeek-R1-Distill-Llama-70B

Listed in tt-inference-server on 5 hardware types; best status Complete.

Runs on
BH 4xP150, BH LoudBox, BH QuietBox 2 +2
Status
Complete
tt launch
7 clients
70B parameters
Speech-to-textOfficial

distil-large-v3

distil-whisper/distil-large-v3

Listed in tt-inference-server on 6 hardware types; best status Complete.

Runs on
BH QuietBox 2, WH Galaxy, WH LoudBox/QuietBox +3
Status
Complete
tt launch
None
Image generationOfficial

FLUX.1-dev

black-forest-labs/FLUX.1-dev

Listed in tt-inference-server on 6 hardware types; best status Complete.

Runs on
BH 4xP150, BH LoudBox, BH P300 +3
Status
Complete
tt launch
None
Image generationOfficial

FLUX.1-schnell

black-forest-labs/FLUX.1-schnell

Listed in tt-inference-server on 6 hardware types; best status Complete.

Runs on
BH 4xP150, BH LoudBox, BH P300 +3
Status
Complete
tt launch
None
LLMOfficial

gemma-4-12B-it

google/gemma-4-12B-it

Listed in tt-inference-server on 2 hardware types; best status Complete. 1 tt-model bundle also published.

Runs on
BH Galaxy, BH QuietBox 2, p100 +1
Status
Complete
tt launch
7 clients
12B parameters1 tt-model bundle
LLMOfficial

gemma-4-31B-it

google/gemma-4-31B-it

Listed in tt-inference-server on 2 hardware types; best status Complete.

Runs on
BH Galaxy, BH QuietBox 2
Status
Complete
tt launch
7 clients
31B parameters
LLMOfficial

Llama-3.1-70B

meta-llama/Llama-3.1-70B

Listed in tt-inference-server on 5 hardware types; best status Complete.

Runs on
BH 4xP150, BH LoudBox, BH QuietBox 2 +2
Speed
≥ 10 tok/s≤ 238 ms WH Galaxy
Status
Complete
tt launch
7 clients
70B parameters
LLMOfficial

Llama-3.1-70B-Instruct

meta-llama/Llama-3.1-70B-Instruct

Listed in tt-inference-server on 5 hardware types; best status Complete.

Runs on
BH 4xP150, BH LoudBox, BH QuietBox 2 +2
Speed
≥ 9.5 tok/s≤ 226 ms WH Galaxy
Status
Complete
tt launch
7 clients
70B parameters
LLMOfficial

Llama-3.1-8B

meta-llama/Llama-3.1-8B

Listed in tt-inference-server on 8 hardware types; best status Complete.

Runs on
BH LoudBox, BH P300, BH QuietBox 2 +5
Speed
≥ 85 tok/s≤ 46 ms WH LoudBox/QuietBox
Status
Complete
tt launch
7 clients
8B parameters
LLMOfficial

Llama-3.1-8B-Instruct

meta-llama/Llama-3.1-8B-Instruct

Listed in tt-inference-server on 9 hardware types; best status Complete.

Runs on
BH LoudBox, BH P300, BH QuietBox 2 +6
Speed
≥ 36 tok/s≤ 58 ms WH LoudBox/QuietBox
Status
Complete
tt launch
7 clients
8B parameters
LLMOfficial

Llama-3.3-70B-Instruct

meta-llama/Llama-3.3-70B-Instruct

Listed in tt-inference-server on 5 hardware types; best status Complete.

Runs on
BH 4xP150, BH LoudBox, BH QuietBox 2 +2
Speed
≥ 14 tok/s≤ 76 ms WH Galaxy
Status
Complete
tt launch
7 clients
70B parameters
LLMOfficial

Mistral-7B-Instruct-v0.3

mistralai/Mistral-7B-Instruct-v0.3

Listed in tt-inference-server on 3 hardware types; best status Complete.

Runs on
WH LoudBox/QuietBox, n150, n300
Speed
≥ 35 tok/s≤ 156 ms n150
Status
Complete
tt launch
2 clients
7B parameters
CNNOfficial

mobilenetv2

mobilenetv2

Listed in tt-inference-server on 2 hardware types; best status Complete.

Runs on
n150, n300
Status
Complete
tt launch
None
Video generationOfficial

mochi-1-preview

genmo/mochi-1-preview

Listed in tt-inference-server on 3 hardware types; best status Complete.

Runs on
BH LoudBox, WH Galaxy, WH LoudBox/QuietBox
Status
Complete
tt launch
None
LLMOfficial

Qwen3-32B

Qwen/Qwen3-32B

Listed in tt-inference-server on 5 hardware types; best status Complete. 3 tt-model bundles also published.

Runs on
BH 4xP150, BH Galaxy, BH LoudBox +4
Speed
≥ 78 tok/s≤ 60 ms WH Galaxy
Status
Complete
tt launch
7 clients
32B parameters3 tt-model bundles
LLMOfficial

Qwen3.8-27B

Qwen/Qwen3.8-27B

Listed in tt-inference-server on 1 hardware type; best status Complete. 3 tt-model bundles also published.

Runs on
BH 4xP150, BH QuietBox 2, p150x2
Speed
≥ 20 tok/s≤ 124 ms BH QuietBox 2
Status
Complete
tt launch
7 clients
27B parameters3 tt-model bundles
Image generationOfficial

stable-diffusion-3.5-large

stabilityai/stable-diffusion-3.5-large

Listed in tt-inference-server on 2 hardware types; best status Complete.

Runs on
WH Galaxy, WH LoudBox/QuietBox
Status
Complete
tt launch
None
Image generationOfficial

stable-diffusion-xl-base-1.0

stabilityai/stable-diffusion-xl-base-1.0

Listed in tt-inference-server on 4 hardware types; best status Complete.

Runs on
WH Galaxy, WH LoudBox/QuietBox, n150 +1
Status
Complete
tt launch
None
Image generationOfficial

stable-diffusion-xl-base-1.0-img-2-img

stabilityai/stable-diffusion-xl-base-1.0-img-2-img

Listed in tt-inference-server on 4 hardware types; best status Complete.

Runs on
WH Galaxy, WH LoudBox/QuietBox, n150 +1
Status
Complete
tt launch
None
CNNOfficial

vit

vit

Listed in tt-inference-server on 2 hardware types; best status Complete.

Runs on
n150, n300
Status
Complete
tt launch
None
CNNOfficial

vovnet

vovnet

Listed in tt-inference-server on 2 hardware types; best status Complete.

Runs on
n150, n300
Status
Complete
tt launch
None
Video generationOfficial

Wan2.2-T2V-A14B-Diffusers

Wan-AI/Wan2.2-T2V-A14B-Diffusers

Listed in tt-inference-server on 6 hardware types; best status Complete.

Runs on
BH 4xP150, BH Galaxy, BH LoudBox +3
Status
Complete
tt launch
None
14B parameters
Speech-to-textOfficial

whisper-large-v3

openai/whisper-large-v3

Listed in tt-inference-server on 6 hardware types; best status Complete.

Runs on
BH QuietBox 2, WH Galaxy, WH LoudBox/QuietBox +3
Status
Complete
tt launch
None
CNNOfficial

yolox_nano

yolox_nano

Listed in tt-inference-server on 1 hardware type; best status Complete.

Runs on
p150
Status
Complete
tt launch
None
LLMOfficial

Llama-3.2-1B

meta-llama/Llama-3.2-1B

Listed in tt-inference-server on 3 hardware types; best status Functional.

Runs on
WH LoudBox/QuietBox, n150, n300
Speed
≥ 18 tok/s≤ 60 ms WH LoudBox/QuietBox
Status
Functional
tt launch
7 clients
1B parameters
LLMOfficial

Llama-3.2-1B-Instruct

meta-llama/Llama-3.2-1B-Instruct

Listed in tt-inference-server on 3 hardware types; best status Functional.

Runs on
WH LoudBox/QuietBox, n150, n300
Speed
≥ 36 tok/s≤ 70 ms WH LoudBox/QuietBox
Status
Functional
tt launch
7 clients
1B parameters
LLMOfficial

Llama-3.2-3B

meta-llama/Llama-3.2-3B

Listed in tt-inference-server on 3 hardware types; best status Functional.

Runs on
WH LoudBox/QuietBox, n150, n300
Speed
≥ 6.1 tok/s≤ 170 ms WH LoudBox/QuietBox
Status
Functional
tt launch
7 clients
3B parameters
LLMOfficial

Llama-3.2-3B-Instruct

meta-llama/Llama-3.2-3B-Instruct

Listed in tt-inference-server on 3 hardware types; best status Functional. 1 tt-model bundle also published.

Runs on
WH LoudBox/QuietBox, n150, n300 +2
Speed
≥ 17 tok/s≤ 120 ms WH LoudBox/QuietBox
Status
Functional
tt launch
7 clients
3B parameters1 tt-model bundle
LLMOfficial

Qwen2.5-72B

Qwen/Qwen2.5-72B

Listed in tt-inference-server on 2 hardware types; best status Functional.

Runs on
WH Galaxy, WH LoudBox/QuietBox
Speed
≥ 8.2 tok/s≤ 130 ms WH Galaxy
Status
Functional
tt launch
7 clients
72B parameters
LLMOfficial

Qwen2.5-72B-Instruct

Qwen/Qwen2.5-72B-Instruct

Listed in tt-inference-server on 2 hardware types; best status Functional.

Runs on
WH Galaxy, WH LoudBox/QuietBox
Speed
≥ 7.1 tok/s≤ 510 ms WH Galaxy
Status
Functional
tt launch
7 clients
72B parameters
LLMOfficial

Qwen3-8B

Qwen/Qwen3-8B

Listed in tt-inference-server on 4 hardware types; best status Functional.

Runs on
WH Galaxy, WH LoudBox/QuietBox, n150 +1
Speed
≥ 60 tok/s≤ 120 ms WH Galaxy
Status
Functional
tt launch
7 clients
8B parameters
VLMOfficial

Qwen3-VL-32B-Instruct

Qwen/Qwen3-VL-32B-Instruct

Listed in tt-inference-server on 1 hardware type; best status Functional.

Runs on
WH LoudBox/QuietBox
Speed
≥ 3.8 tok/s≤ 1.5 s WH LoudBox/QuietBox
Status
Functional
tt launch
2 clients
32B parameters
LLMOfficial

QwQ-32B

Qwen/QwQ-32B

Listed in tt-inference-server on 2 hardware types; best status Functional.

Runs on
WH Galaxy, WH LoudBox/QuietBox
Speed
≥ 16 tok/s≤ 300 ms WH Galaxy
Status
Functional
tt launch
7 clients
32B parameters
CNNOfficial

resnet-50

resnet-50

Listed in tt-inference-server on 2 hardware types; best status Functional.

Runs on
n150, n300
Status
Functional
tt launch
None
CNNOfficial

segformer

segformer

Listed in tt-inference-server on 2 hardware types; best status Functional.

Runs on
n150, n300
Status
Functional
tt launch
None
Image generationOfficial

Z-Image-Turbo

Tongyi-MAI/Z-Image-Turbo

Listed in tt-inference-server on 1 hardware type; best status Functional. 1 tt-model bundle also published.

Runs on
BH QuietBox 2, p100a
Status
Functional
tt launch
None
1 tt-model bundle
LLMOfficial

AFM-4.5B

arcee-ai/AFM-4.5B

Listed in tt-inference-server on 2 hardware types; best status Experimental.

Runs on
WH LoudBox/QuietBox, n300
Status
Experimental
tt launch
2 clients
4B parameters
EmbeddingOfficial

bge-large-en-v1.5

BAAI/bge-large-en-v1.5

Listed in tt-inference-server on 4 hardware types; best status Experimental.

Runs on
WH Galaxy, WH LoudBox/QuietBox, n150 +1
Status
Experimental
tt launch
None
EmbeddingOfficial

bge-m3

BAAI/bge-m3

Listed in tt-inference-server on 1 hardware type; best status Experimental.

Runs on
BH QuietBox 2
Status
Experimental
tt launch
None
LLMOfficial

DeepSeek-R1-0528

deepseek-ai/DeepSeek-R1-0528

Listed in tt-inference-server on 3 hardware types; best status Experimental.

Runs on
Dual WH Galaxy, Quad WH Galaxy, WH Galaxy
Status
Experimental
tt launch
7 clients
LLMOfficial

diffusiongemma-26B-A4B-it

google/diffusiongemma-26B-A4B-it

Listed in tt-inference-server on 1 hardware type; best status Experimental.

Runs on
BH QuietBox 2
Status
Experimental
tt launch
2 clients
4B parameters
VLMOfficial

gemma-3-27b-it

google/gemma-3-27b-it

Listed in tt-inference-server on 1 hardware type; best status Experimental.

Runs on
WH Galaxy
Speed
≥ 4.3 tok/s≤ 760 ms WH Galaxy
Status
Experimental
tt launch
2 clients
27B parameters
LLMOfficial

gpt-oss-120b

openai/gpt-oss-120b

Listed in tt-inference-server on 4 hardware types; best status Experimental. 3 tt-model bundles also published.

Runs on
BH 4xP150, BH Galaxy, BH LoudBox +3
Speed
≥ 2.7 tok/s≤ 510 ms BH QuietBox 2
Status
Experimental
tt launch
7 clients
120B parameters3 tt-model bundles
EmbeddingOfficial

Qwen3-Embedding-0.6B

Qwen/Qwen3-Embedding-0.6B

Listed in tt-inference-server on 1 hardware type; best status Experimental.

Runs on
BH QuietBox 2
Status
Experimental
tt launch
None
EmbeddingOfficial

Qwen3-Embedding-4B

Qwen/Qwen3-Embedding-4B

Listed in tt-inference-server on 1 hardware type; best status Experimental.

Runs on
BH QuietBox 2
Status
Experimental
tt launch
None
4B parameters
EmbeddingOfficial

Qwen3-Embedding-8B

Qwen/Qwen3-Embedding-8B

Listed in tt-inference-server on 4 hardware types; best status Experimental.

Runs on
WH Galaxy, WH LoudBox/QuietBox, n150 +1
Status
Experimental
tt launch
None
8B parameters
LLMOfficial

Qwen3.6-27B

Qwen/Qwen3.6-27B

Listed in tt-inference-server on 1 hardware type; best status Experimental. 1 tt-model bundle also published.

Runs on
BH 4xP150, BH QuietBox 2
Status
Experimental
tt launch
7 clients
27B parameters1 tt-model bundle
Text-to-speechOfficial

speecht5_tts

microsoft/speecht5_tts

Listed in tt-inference-server on 2 hardware types; best status Experimental.

Runs on
BH QuietBox 2, p150
Status
Experimental
tt launch
None