tt-animatediff — comprehensive usage

CPU / any machine CPU

Runs Phase 1: diffusers.AnimateDiffPipeline + full MotionAdapter on any machine. No Tenstorrent hardware needed.

$ git clone https://github.com/tenstorrent/tt-animatediff.git
$ cd tt-animatediff
$ pip install -r requirements.txt
$ python examples/generate.py --mode cpu --prompt "ocean waves at sunset"

Blackhole hardware Blackhole

Phase 2.5: TTNN UNet on Blackhole with cross-frame temporal attention. Requires tt-metal built and its Python env active.

1. Build tt-metal (one-time, ~30 min)

   $ git clone https://github.com/tenstorrent/tt-metal.git ~/tt-metal
   $ cd ~/tt-metal && ./build_metal.sh

2. Clone this repo and install dependencies

   $ git clone https://github.com/tenstorrent/tt-animatediff.git
   $ cd tt-animatediff && pip install -e .

3. Activate the tt-metal Python env before each session

   $ source ~/tt-metal/python_env/bin/activate

4. Generate your first animation

   $ python examples/generate.py --prompt "aurora borealis over a frozen lake"

ttsim simulator sim

Bit-exact Blackhole simulation on any Linux/x86_64 machine. No hardware required.

$ mkdir -p ~/sim
$ wget -O ~/sim/libttsim_bh.so \
    https://github.com/tenstorrent/ttsim/releases/download/v1.7.0/libttsim_bh.so
$ source ~/tt-metal/python_env/bin/activate
$ python examples/generate.py --mode sim --frames 2 --steps 4

See docs/SIMULATOR.md for the full simulator setup guide.

Gradio web UI BlackholeCPUsim

$ pip install -e ".[ui]"
$ python app.py
# Open http://localhost:7860

All modes are driven by a single entry point: examples/generate.py.

All flags

Common recipes

Fast preview on any machine

# CPU, 4 frames, 10 steps — quick feedback on a new prompt
$ python examples/generate.py --mode cpu --frames 4 --steps 10 \
    --prompt "crackling campfire in a dark forest, cinematic"

Standard Blackhole generation

# 8 frames, 25 steps, ~100 s total on P300C
$ python examples/generate.py \
    --prompt "aurora borealis over a frozen tundra, long exposure" \
    --frames 8 --steps 25 --seed 1337

Fast MotionAdapter (skip up1+up2)

# ~7.7 s/frame — faster than Phase 2.5, full encoder-side attention
$ python examples/generate.py --motion-adapter \
    --motion-adapter-skip up1 up2 --frames 8 \
    --prompt "mycelium network pulsing with bioluminescence"

Prompt chaining — visual narrative continuity

# Run 1: save latents
$ python examples/generate.py --prompt "a seed germinating in dark soil" \
    --chain-save chain.pt --output scene1.gif

# Run 2: blend from run 1's latents
$ python examples/generate.py --prompt "a sapling reaching toward sunlight" \
    --chain-from chain.pt --chain-alpha 0.6 --output scene2.gif

Simulator — no hardware

# 2 frames, 4 steps — quick compatibility check
$ python examples/generate.py --mode sim --frames 2 --steps 4 \
    --prompt "test render"

Import animatediff_ttnn directly from Python. The high-level generate_animation() entry point manages device lifetime, mode selection, and caching internally — no setup code needed.

generate_animation()

from animatediff_ttnn import generate_animation, export_mp4, export_gif

# Simplest call — auto mode picks Blackhole if tt-metal is importable, else CPU
frames = generate_animation(
    prompt="swirling nebula, teal and gold, cinematic",
)

# Save as MP4 or GIF
export_mp4(frames, "output.mp4", fps=8)
export_gif(frames, "output.gif")

Full signature

Returns: list[PIL.Image] — one image per frame, in generation order.

Mode selection

mode="auto" tries to import ttnn and picks "blackhole" if available, "cpu" otherwise. The TTNN device and compiled UNet are initialized on the first call and held for the lifetime of the process — subsequent calls are fast.

Per-step progress callback

from animatediff_ttnn import generate_animation

def on_step(step_idx, num_steps, frame_latents):
    pct = 100 * (step_idx + 1) / num_steps
    print(f"  step {step_idx+1}/{num_steps} ({pct:.0f}%)")

frames = generate_animation(
    prompt="lava flow through volcanic rock, molten orange glow",
    mode="blackhole",
    num_frames=8,
    on_step=on_step,
)

CPU Lightning

# CPU Lightning: 4-step distilled adapter, ~20 s/frame
frames = generate_animation(
    prompt="crystalline cave, blue ice, shafts of light",
    mode="cpu",
    use_lightning=True,
    lightning_steps=4,   # must match checkpoint: 2, 4, or 8
    guidance_scale=1.0,   # distilled — CFG baked in
)

Prompt chaining

# Scene 1 — save latents
frames1 = generate_animation(
    prompt="a lone lighthouse in a stormy sea",
    chain_save="lighthouse.pt",
)

# Scene 2 — blend from scene 1 for visual DNA continuity
frames2 = generate_animation(
    prompt="a calm harbour at dawn, same coastline",
    chain_from="lighthouse.pt",
    chain_alpha=0.6,
)

Export functions

from animatediff_ttnn import export_mp4, export_gif

# MP4 — libx264 / yuv420p, requires ffmpeg on PATH
export_mp4(frames, "output.mp4", fps=8)

# GIF — no external dependencies
export_gif(frames, "output.gif")

export_mp4 requires ffmpeg on $PATH. Install with sudo apt install ffmpeg (Ubuntu) or brew install ffmpeg (macOS).

Low-level API

Direct access to phase-specific pipelines for scripting and testing:

# Phase 1 — CPU baseline
from animatediff_ttnn.pipeline import create_animatediff_pipeline, generate, export_gif

pipe = create_animatediff_pipeline()
frames = generate(pipe, "ocean waves at sunset", num_frames=8)

# Phase 2.5 — Blackhole temporal attention (requires device + models)
from animatediff_ttnn.session import ensure_blackhole
from animatediff_ttnn.generation_helpers import encode_prompt
from animatediff_ttnn.temporal_attention import generate_frames_temporal

device, (ttnn_model, ttnn_vae, config, time_proj) = ensure_blackhole()
embeddings = encode_prompt("ocean", "blurry")
frames = generate_frames_temporal(device, ttnn_model, ttnn_vae,
    config, time_proj, embeddings, num_frames=8, ...)

A browser-based interface for interactive generation with real-time denoising previews.

Starting the UI

$ pip install -e ".[ui]"
$ source ~/tt-metal/python_env/bin/activate
$ python app.py
# Open http://localhost:7860, select Mode: blackhole

$ pip install -e ".[ui]"
$ python app.py
# Open http://localhost:7860, select Mode: cpu

$ pip install -e ".[ui]"
$ source ~/tt-metal/python_env/bin/activate
$ python app.py
# Open http://localhost:7860
# Select Mode: sim — enter sim binary path if not ~/sim/libttsim_bh.so

Parameters reference

Timings measured on QB2 (4 × P300C), warm model (TTNN JIT compiled). See benchmarks page for full breakdown.

This pipeline uses SD 1.4 in all modes. Knowing its characteristics produces consistently better results.

Patterns that work

# Style before subject
"watercolor painting of ancient ruins at sunset, soft brushstrokes, muted palette"

# Cinematic descriptors
"cinematic 4K, dramatic side lighting, volumetric fog, depth of field"

# Specific motion verbs
"crackling campfire"  "ocean waves crashing"  "aurora borealis shimmering"
"shifting cosmos"     "lava flowing"           "smoke drifting"

# Combine style + lighting + subject
"oil painting of a cathedral at dusk, dramatic crepuscular rays, golden hour, impressionist"

# Negative prompt — always include these
"blurry, low quality, distorted, text, people, faces, modern buildings"

Temporal alpha tuning Blackholesim

Controls how strongly frames are blended together during cross-frame attention in Phase 2.5.

Rule of thumb: fast motion (fire, water) → 0.2–0.35. Slow atmospheric drift (aurora, cosmos) → 0.4–0.6.

Step count guidance

ImportError: No module named 'ttnn'

The tt-metal Python env is not active. Run source ~/tt-metal/python_env/bin/activate before any Blackhole or sim command. In mode="auto" this causes automatic fallback to CPU — no error is raised.

TT_FATAL / hardware abort

A chip may be in a bad state. Try tt-smi -s to check chip health. If chip 3 is listed as hung (BOARD_ID_HIGH 0x461, BOARD_ID_LOW 0x31924055), the setup_blackhole() function reads hwmon sentinel values to skip dead chips automatically.

ValueError: num_frames must be a multiple of num_chips

On a QB2 (4-chip mesh), frame count must be a multiple of 4: use 4, 8, 12, or 16. This constraint applies to Phase 2.5 and the Gradio UI's Blackhole mode.

TTNN JIT compilation taking a long time

First-run UNet kernel compilation takes 2–3 minutes. Compiled kernels are cached — subsequent runs start in seconds. Do not interrupt the first run.

export_mp4: ffmpeg not found

$ sudo apt install ffmpeg      # Ubuntu / Debian
$ brew install ffmpeg          # macOS

ttsim probe failed

The ttsim binary version is incompatible with the installed tt-metal version. Download a matching release from github.com/tenstorrent/ttsim/releases. The probe runs in a subprocess to prevent a simulator abort() from killing the Gradio server.

CPU Lightning: quality degrades with more steps

The distilled ByteDance/AnimateDiff-Lightning checkpoint is trained for exactly 2, 4, or 8 steps with guidance_scale=1.0 (baked in). Using more steps or a different CFG scale degrades output. Use --lightning-steps to match the checkpoint and don't override guidance_scale.

Gradio UI: preview looks like noise

Normal for the first several denoising steps. The latent-space preview uses a fast tanh-colourised mapping (no VAE) — noise is expected until ~step 10 of a 25-step run. Watch for structure emerging in the middle steps.