The Lightning scheduler cuts denoising steps from 25 to 4–8 with no change to the model architecture. Each row below is the same prompt, same seed — original Blackhole output on the left, lightning mode on the right.
Lightning and PNDM 25-step take nearly the same wall-clock time — the CPU temporal blend is the bottleneck, not step count. →
Full benchmarks
Standing on the shoulders of giants
AnimateDiff
Yuwei Guo, Chuanxia Zheng, Ruizhen Hu et al. The insight that motion priors can be injected into any SD UNet as a plug-and-play MotionAdapter — without retraining the base model. We just ran it faster.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer (CompVis / Stability AI). The latent diffusion model that made high-quality generation accessible to everyone.
The Tenstorrent systems team. The SD 1.4 TTNN UNet kernel — built for Wormhole, adapted here for Blackhole — is what makes the hardware path possible. We wired the temporal loop around it.
We ported, adapted, debugged, and shipped — but the hard science and the hard systems work were done by these teams. This repo is gratitude in the form of running code.
What Tenstorrent contributed
Hardware path
Replaced the diffusers CPU UNet with the SD 1.4 TTNN UNet running natively on Blackhole silicon. ~12.5 s/frame on P300C (Phase 2.5, 25-step) vs ~2 min/frame on CPU.
Temporal attention
Cross-frame self-attention at each PNDM step across all N frame latents simultaneously — a bridge toward full MotionAdapter integration in TTNN.
Open implementation
A clean, tested, documented repo anyone with Blackhole hardware can clone and run. Includes a ttsim path for development without silicon.
How it works
Architecture & Execution Flow
What runs on Blackhole silicon, what runs on CPU, and where this differs from standard AnimateDiff.
flowchart TD
P([Prompt + seed]) --> CHAIN{"--chain-from?"}
CHAIN -->|yes| BLEND["frame-mean blend prev latents\ninto seed noise — CPU\n(renorm to unit std)"]
CHAIN -->|no| ENC
BLEND --> ENC["CLIP encode — CPU"]
ENC --> MODE{Mode?}
MODE -->|cpu| CPU_SCHED["PNDM scheduler + MotionAdapter\nor Euler + distilled Lightning weights"]
CPU_SCHED --> CPU_UNET["diffusers UNet2DConditionModel\n+ MotionAdapter TemporalTransformer\nFull AnimateDiff — CPU only"]
CPU_UNET --> CPU_VAE["VAE decode — CPU"]
CPU_VAE --> GIF([Output GIF])
MODE -->|blackhole / sim| LOOP["Denoising loop — one scheduler per frame\nPNDM standard · Euler for Lightning"]
LOOP --> BH_UNET["SD 1.4 TTNN UNet2D\nBlackhole P300C · ~12.5 s / frame (Phase 2.5)"]
BH_UNET --> CFA["cross_frame_attention\nnoise predictions blended\nacross all N frames — CPU"]
CFA --> STEP["scheduler.step — one per frame"]
STEP --> NEXT{more steps?}
NEXT -->|yes| LOOP
NEXT -->|no| SAVE["--chain-save: write\ndenoised latents to .pt — CPU"]
SAVE --> BH_VAE["TTNN VAE decode on Blackhole\nUNet L1 freed before decode"]
BH_VAE --> GIF
The TTNN UNet runs on Blackhole; everything else is CPU. Cross-frame attention is applied to the stacked noise predictions at each step — a Phase 2.5 approximation of the MotionAdapter's temporal transformer. Lightning mode switches to TtEulerScheduler (trailing, linear) with the same TTNN UNet — no distilled adapter loaded on Blackhole/sim; step count is still controlled by --steps. CPU Lightning loads ByteDance's distilled adapter (CFG=1.0) and requires 2/4/8 steps. --chain blends the frame-mean of the previous run's denoised latents (renormalized to unit std) into seed noise before the loop begins.
flowchart TB
subgraph WRONG["❌ Original — silent failure"]
W1["SD 3.5 DiT · 2432-dim features"] -->|motion weights injected| W2["mm_sd_v15_v2.ckpt trained for SD 1.5 UNet · 320-dim"]
W2 --> W3["Dimension mismatch — no temporal attention actually applied"]
end
subgraph RIGHT["✅ Current"]
R1["SD 1.4 UNet · 320-dim — matching architecture"]
R1 -->|CPU| R2["MotionAdapter TemporalTransformer · full AnimateDiff ✓"]
R1 -->|Blackhole| R3["TTNN UNet2D · cross-frame attention blend · Phase 2.5"]
end
WRONG -.->|fix: use matching architecture| RIGHT
The original code applied mm_sd_v15_v2.ckpt motion weights (trained for SD 1.5 UNet, 320-dim features) to the SD 3.5 DiT (a Diffusion Transformer, 2432-dim). The shapes are incompatible — no temporal attention was actually running. The fix was switching to SD 1.4 whose UNet matches the motion weights exactly.
Recent additions
⚡ AnimateDiff-Lightning & Gradio UI
Two new additions since the initial release.
AnimateDiff-Lightning on Blackhole
Lightning mode is available across all three backends, but works differently per mode. On CPU: loads ByteDance's distilled motion adapter (arXiv:2403.12706) — genuine 4-step distillation, CFG=1.0 baked in. On Blackhole/sim: switches to TtEulerScheduler (Euler, trailing, linear) with the base SD 1.4 TTNN UNet — no distilled adapter, CFG=7.5 retained, any step count. Enable with:
# Blackhole hardware — ~12 s/frame (warm, Phase 2.5)
python examples/generate.py --lightning
# CPU — ~20 s/frame, no hardware required
python examples/generate.py --mode cpu --lightning
Select mode → enter prompt → adjust frames/steps/seed → click Generate. Lightning checkbox is available in all modes. Output GIF displays inline.
Lightning samples — Euler scheduler on Blackhole, 25 steps · CFG=7.5 · ~12 s/frame — full 10-prompt gallery →
supernova · deep space nebulacrystal cave · amethyst formationsdeep sea vent · bioluminescent
Stateful latent threading
🔗 Chain Continuity
Two independent prompts. One subject — retro 3D glasses — threaded between them by blending the first run's final denoised latents into the second run's seed noise. No explicit conditioning, no fine-tuning. Just hardware-resident state.
neon diner 35mm film
α 0.20
deep space violet nebula
How it works
After each generation, --chain-save persists the final denoised latent tensors to a .pt file. The next run loads them with --chain-from and blends the normalised mean into the seed noise before denoising begins. Each step's solver then steers away from shared noise toward the new prompt — so the subject's form lingers in the structure while the context changes completely.
# Run 1 — save latents
python examples/generate.py \
--prompt "retro 3D glasses, neon diner" \
--chain-save chain.pt
# Run 2 — thread DNA from run 1
python examples/generate.py \
--prompt "retro 3D glasses, deep space" \
--chain-from chain.pt --chain-alpha 0.20 \
--chain-save chain.pt
Why this is hardware-native
Standard diffusion pipelines evict all intermediate tensors between calls. On Tenstorrent Blackhole, the denoised latent state lives in L1 between runs — the hardware makes persistent-state inference cheap. Chain threading is only practical at this cost because the latents never need to round-trip through host memory. It's a primitive that other architectures can't express efficiently without bespoke memory management. --chain-alpha (0–1) controls how strongly the previous run's identity bleeds through — lower values let the new prompt fully take over; higher values preserve more of the subject's structural fingerprint.
The Gradio UI exposes chain controls under the Chain continuity accordion — load a .pt from disk and chain runs interactively.
Prompt Guide
SD 1.4 at 512×512 with the TTNN UNet has a distinct personality. Knowing it gets you better results.
PNDM and Euler (Lightning): 25 is the sweet spot on silicon for both. Minimum ~4 for quick previews. With distilled weights, 4–8 steps produce comparable quality at 3–6× the speed.
