Models
LLM OfficialLlama family

Llama-3.1-8B-Instruct

meta-llama/Llama-3.1-8B-Instruct

Listed in tt-inference-server on 9 hardware types; best status Complete.

In tt-inference-server

HardwareEngine and implementationStatusContextUsersBuildSupport page
GPU
GPU
 vLLM
tt-transformers
 Complete 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/9b67e09/models/tt_transformers
tt-metal 9b67e09
vLLM a91b644
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.2.0-9b67e09-a91b644
 -
WH LoudBox/QuietBox
T3K
 vLLM
tt-transformers
 Complete 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/9b67e09/models/tt_transformers
tt-metal 9b67e09
vLLM a91b644
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.2.0-9b67e09-a91b644
 -
n150
N150
 vLLM
tt-transformers
 Complete 64K 32
Code: https://github.com/tenstorrent/tt-metal/tree/9b67e09/models/tt_transformers
tt-metal 9b67e09
vLLM a91b644
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.2.0-9b67e09-a91b644
 -
n300
N300
 vLLM
tt-transformers
 Complete 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/9b67e09/models/tt_transformers
tt-metal 9b67e09
vLLM a91b644
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.2.0-9b67e09-a91b644
 -
BH LoudBox
P150X8
 vLLM
tt-transformers
 Functional 128K 256
Code: https://github.com/tenstorrent/tt-metal/tree/555f240/models/tt_transformers
tt-metal 555f240
vLLM 22be241
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.10.0-555f240-22be241
 -
BH P300
P300
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/b204341/models/tt_transformers
tt-metal b204341
vLLM 9bd099c
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.19.0-b204341-9bd099c
 -
BH QuietBox 2
P300X2
 vLLM
llama31-8b-qb2
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/29f93c9/models/demos/llama31_8b_qb2
tt-metal 29f93c9
vLLM 3506696
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.24.0-29f93c9-3506696
 Open
BH QuietBox 2
P300X2
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/8c48a10/models/tt_transformers
tt-metal 8c48a10
vLLM f52987a
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.17.0-8c48a10-f52987a
 Open
WH Galaxy
GALAXY
 vLLM
tt-transformers
 Functional 128K 128
Code: https://github.com/tenstorrent/tt-metal/tree/bac8b34/models/tt_transformers
tt-metal bac8b34
vLLM 7c6685a
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.11.0-bac8b34-7c6685a
 -
WH Galaxy
GALAXY_T3K
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/bac8b34/models/tt_transformers
tt-metal bac8b34
vLLM 7c6685a
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.11.1-bac8b34-7c6685a
 -
p150
P150
 vLLM
tt-transformers
 Experimental 64K 32
Code: https://github.com/tenstorrent/tt-metal/tree/c49bb76/models/tt_transformers
tt-metal c49bb76
vLLM 6b4a3a7
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.18.0-c49bb76-6b4a3a7
 -

Status, context and users are copied from tt-inference-server's release spec (commit 2db98e38c, release 0.25.0). Context is the longest prompt plus reply the entry allows. Users is its highest concurrency. Each row carries its own code link, tt-metal and vLLM commits and image, because they differ between hardware. Nothing on this page is measured.

tt-model bundles

No tt-model bundle has been published for this model.

Details

KindLLM (from tt-inference-server model type)
Parameters8B
FamilyLlama (from the start of the name)
Implementationllama31-8b-qb2, tt-transformers
Disk needed36 GB
RAM needed20 GB
Tool-call parserllama3_json
Listed bytt-inference-server