GPU

tt-inference-server's device list has no product name for this. It is shown as the source spells it.

2 models Filter

Llama-3.1-8B

meta-llama/Llama-3.1-8B

Listed in tt-inference-server on 8 hardware types; best status Complete.

Runs on: BH LoudBox, BH P300, BH QuietBox 2 +5
Speed: ≥ 85 tok/s≤ 46 ms WH LoudBox/QuietBox
Status: Complete
tt launch: 7 clients

Llama-3.1-8B-Instruct

meta-llama/Llama-3.1-8B-Instruct

Listed in tt-inference-server on 9 hardware types; best status Complete.

Runs on: BH LoudBox, BH P300, BH QuietBox 2 +6
Speed: ≥ 36 tok/s≤ 58 ms WH LoudBox/QuietBox
Status: Complete
tt launch: 7 clients