Models
LLM OfficialLlama family

Llama-3.1-70B-Instruct

meta-llama/Llama-3.1-70B-Instruct

Listed in tt-inference-server on 5 hardware types; best status Complete.

In tt-inference-server

HardwareEngine and implementationStatusContextUsersBuildSupport page
WH Galaxy
GALAXY
 vLLM
llama3-70b-galaxy
 Complete 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/65718bb/models/demos/llama3_70b_galaxy
tt-metal 65718bb
vLLM 409b1cd
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.9.0-65718bb-409b1cd
 -
BH 4xP150
P150X4
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/555f240/models/tt_transformers
tt-metal 555f240
vLLM 22be241
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.10.1-555f240-22be241
 -
BH LoudBox
P150X8
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/555f240/models/tt_transformers
tt-metal 555f240
vLLM 22be241
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.10.1-555f240-22be241
 -
BH QuietBox 2
P300X2
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/669d59e/models/tt_transformers
tt-metal 669d59e
vLLM 3334377
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.16.0-669d59e-3334377
 -
WH Galaxy
GALAXY_T3K
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/v0.62.0-rc33/models/tt_transformers
tt-metal v0.62.0-rc33
vLLM e7c329b
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.2.0-v0.62.0-rc33-e7c329b
 -
WH LoudBox/QuietBox
T3K
 vLLM
tt-transformers
 Functional 128K 32
Code: https://github.com/tenstorrent/tt-metal/tree/750ca54/models/tt_transformers
tt-metal 750ca54
vLLM 38dee8c
Image ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.11.1-750ca54-38dee8c
 -

Status, context and users are copied from tt-inference-server's release spec (commit 2db98e38c, release 0.25.0). Context is the longest prompt plus reply the entry allows. Users is its highest concurrency. Each row carries its own code link, tt-metal and vLLM commits and image, because they differ between hardware. Nothing on this page is measured.

tt-model bundles

No tt-model bundle has been published for this model.

Details

KindLLM (from tt-inference-server model type)
Parameters70B
FamilyLlama (from the start of the name)
Implementationllama3-70b-galaxy, tt-transformers
Disk needed160 GB
RAM needed175 GB
Tool-call parserllama3_json
Listed bytt-inference-server