n150 n300 T3000 p100 p150 p300c Galaxy 15 min Draft

Understanding Custom Training

Welcome to the Custom Training track. Elsewhere in this extension you've learned to run models — inference. This track is about creating them: teaching a network new behavior by adjusting its weights.

This lesson lays the groundwork before you touch a dataset or a training loop: what custom training is, when you actually need it, and which of the three ways to train on Tenstorrent hardware fits your goal.

What You'll Learn

Time: 10-15 minutes | Prerequisites: Basic understanding of machine learning concepts

What the whole track needs (built once). Every hands-on lesson from Fine-tuning Basics onward runs on ttml (tt-train), which has no pip wheel. You'll need a recent tt-metal built from source with tt-train enabled (verified on v0.73). On a TT-QuietBox® 2 the tt-metal source tree isn't pre-installed, so budget a one-time build — ct4 walks the exact recipe behind an Install tt-train button. The concept lessons before it (this one, Datasets, Configuration) need no hardware, so you can start the build now and keep reading.

The Track at a Glance

This track runs in a fixed order: six lessons that each build on the last, plus two deeper lessons on model architecture that build on those six.

graph LR
    A[Understand] --> B[Datasets]
    B --> C[Configuration]
    C --> D[Fine-tuning]
    D --> E[Multi-Device]
    E --> F[Experiment Tracking]
    F -.-> G[Architecture Basics]
    G -.-> H[From Scratch]

    style A fill:#1B8EB1,stroke:#092221,stroke-width:3px

You're at the start — Understand. Everything else in this diagram is a lesson later in the walkthrough, in the order it's meant to be read.

Custom Training vs. Inference

Inference (what you've done so far)

Training (what this track builds)

Key insight: a model is a pile of numbers until training decides what those numbers should be. That's the whole job.

Two Paths to a Custom Model

Fine-tuning

Start with a pre-trained model and teach it something new.

Reach for this when:

Analogy: hiring an experienced developer and onboarding them to your codebase, not teaching them to code from zero.

Fine-tuning Basics is where this track puts a first training run into practice, end to end.

Training from scratch

Build a model from random weights, with no pre-trained starting point.

Reach for this when:

Analogy: teaching yourself programming from first principles instead of joining a team that already knows the codebase.

Training from Scratch covers this later in the track: a small transformer trained from random initialization on Shakespeare text.

Rule of thumb: fine-tune unless you have a specific reason not to. Pre-trained models already understand language; training from scratch means re-deriving that from nothing, which costs real time and data.

Three Ways to Train on Tenstorrent Hardware

There isn't one training stack on Tenstorrent hardware — there are three, and they solve different problems. Picking the wrong one for your goal is the most common source of confusion here, so it's worth being precise about what each actually is.

tt-train / ttml — this track's stack

tt-train is the autograd training framework that lives inside TT-Metalium™'s source tree. Its Python bindings are called ttml. It supplies the piece TT-NN™ doesn't have on its own: a backward pass. TT-NN's ops (ttnn.matmul, ttnn.rms_norm, and so on) are forward-only — each one computes a result and hands it back, with nothing recording how to differentiate that computation. ttml wraps operations like these with a matching backward pass and an on-device AdamW optimizer, so a real training loop — forward, loss, backward, update — can run on Tenstorrent hardware instead of just inference.

This is the framework the rest of this track uses. Later lessons run train_nanogpt.py against it: real gradient descent, real loss curves dropping step by step, on real Tenstorrent silicon.

ttml is source-only — no pip wheel — and builds as a cmake subproject of TT-Metalium. If you don't already have a built ~/tt-metal source tree (TT-QuietBox® 2 images ship TT-NN and vLLM pre-installed but not the tt-metal source tree), start with Build TT-Metalium from Source. Once that tree exists, the Install tt-train command in this extension automates the ttml build.

tt-blacksmith — a separate stack, not this track

tt-blacksmith is a different, actively maintained repository of optimized training recipes — but built on the TT-Forge™/TT-XLA compiler stack, not on tt-train. It is not a configuration layer over tt-train, and the two projects don't share code or config format. If you're already working in the TT-Forge/TT-XLA world — see JAX Inference with TT-XLA — and want tuned recipes for that compiler stack, tt-blacksmith is the place to look. This track doesn't teach it: everything from here forward is tt-train/ttml.

PyTorch / GPU — the familiar baseline

If you've trained models before, it was almost certainly PyTorch on a GPU: loss.backward(), an Adam optimizer, a DataLoader. That mental model transfers directly here — ttml mirrors it deliberately. The training loop you'll write in this track is the same four steps (forward, loss, backward, update) you'd write in PyTorch. What changes is the hardware underneath, and the library that knows how to run backward on it.

Want to Build It Yourself, By Hand?

Everything above assumes a framework — ttml or tt-blacksmith — handles the backward pass and optimizer for you. If instead you want to build every one of those pieces yourself, tokenizer through training loop, with nothing hidden behind a framework call, that's a different track: Build an LLM from Scratch, starting from Pick Your Altitude.

That arc builds a small Llama-style model TT-native from the first line: Embeddings & the Residual Stream writes the embedding table and RoPE by hand, Attention from Scratch hand-authors attention with a TT-Lang kernel, The Transformer Block & the Model assembles the full block, and Train It & Run for Real writes the training loop itself — cross-entropy, backprop, AdamW — before handing off to ttml to run it for real on Blackhole® hardware.

Come back to this track once you want the fast path: real training runs without hand-rolling every op first.

Understanding the Training Process

Training a model is like teaching through repetition - show examples, measure mistakes, make corrections, repeat. Here's the complete flow:

graph TD
    A[Raw DataText files, datasets] --> B[Prepare DataJSONL format]
    B --> C[Initialize ModelPre-trained OR random weights]

    C --> D{Training LoopMultiple epochs}

    D --> E[Get Batch8-32 examples]
    E --> F[Forward PassModel makes predictions]
    F --> G[Compute LossHow wrong?]
    G --> H[Backward PassCalculate gradients]
    H --> I[Update WeightsOptimizer step]

    I --> J{More Batches?}
    J -->|Yes| E
    J -->|No| K[EvaluationGenerate samples, check quality]

    K --> L[Save CheckpointModel weights + optimizer state]

    L --> M{Continue Training?}
    M -->|Yes, more epochs| D
    M -->|No, training complete| N[DeploymentUse with vLLM for inference]

    style A fill:#4A90E2,stroke:#333,stroke-width:2px
    style B fill:#7B68EE,stroke:#333,stroke-width:2px
    style C fill:#7B68EE,stroke:#333,stroke-width:2px
    style D fill:#E85D75,stroke:#333,stroke-width:3px
    style E fill:#7B68EE,stroke:#333,stroke-width:2px
    style F fill:#7B68EE,stroke:#333,stroke-width:2px
    style G fill:#7B68EE,stroke:#333,stroke-width:2px
    style H fill:#7B68EE,stroke:#333,stroke-width:2px
    style I fill:#7B68EE,stroke:#333,stroke-width:2px
    style K fill:#7B68EE,stroke:#333,stroke-width:2px
    style L fill:#E85D75,stroke:#333,stroke-width:2px
    style N fill:#50C878,stroke:#333,stroke-width:2px

What each step does:

Step 1: Prepare Data

Transform raw text into training format (JSONL with prompt/response pairs). Quality matters more than quantity here.

Step 2: Initialize Model

Either load pre-trained weights (fine-tuning) or start from random numbers (training from scratch). Most of the time, you'll fine-tune.

Step 3: Training Loop (The Core)

This is where learning happens:

  1. Get Batch - Load 8-32 examples from your dataset
  2. Forward Pass - Model makes predictions based on current weights
  3. Compute Loss - Measure how far predictions are from correct answers
  4. Backward Pass - Calculate which direction to adjust each weight
  5. Update Weights - Actually change the model's parameters
  6. Repeat - Do this thousands of times

Think of loss as: A score that goes down as the model gets better. Loss of 2.5 → 1.2 → 0.5 means it's learning.

Step 4: Evaluation

Generate sample outputs to see if the model is improving. This happens every few hundred steps, not every step.

Step 5: Save Checkpoint

Store model weights and training state so you can resume if interrupted or pick the best version later.

Step 6: Deployment

Once training is complete, use your trained model for inference. Integrate with vLLM Production for production serving.

Hardware Considerations

ttml builds and trains from source across the Wormhole™ and Blackhole® lineup. For single-chip work, treat p300c exactly like a p100. A TT-QuietBox® 2 is one four-chip ring mesh (P300_X2, a 2×2 mesh) — not four independent chips — so beyond single-chip training it can also run multi-chip data-parallel training with near-linear scaling (verified in Multi-Device Training).

n150 / p100 / p300c (single chip)

n300 (dual Wormhole chips)

T3000 / Galaxy (multi-chip mesh)

For this track: the hands-on lessons target n150 and p300c/p100 first — everyone can follow along — with n300+ covered when the track reaches multi-device training.

What's Ahead in This Track

Each lesson is a concrete, runnable example chosen to teach a principle you can carry into your own domain — not just a script to copy.

Common Questions

"Should I fine-tune or train from scratch?"

Fine-tune, nearly always. It's faster (hours, not days or weeks), cheaper (less compute), and starts from a model that already understands language instead of nothing.

Train from scratch when you're researching a new architecture, need complete control, want to understand the fundamentals down to the training loop, or are building something genuinely novel.

"How much data do I need?"

For fine-tuning:

For training from scratch:

Quality beats quantity: 200 high-quality examples beat 10,000 mediocre ones.

"Will fine-tuning erase what the model learned?"

No, if done correctly.

Think of it as: teaching someone new skills, not wiping their memory.

"Can I use this for commercial projects?"

Yes, with caveats:

Always verify licenses for your specific use case.

Key Takeaways

Next Steps

Now that the concepts and the framework choice are settled, it's time to get hands-on. Dataset Fundamentals has you:

  1. Create your first training dataset (JSONL format).
  2. Validate the dataset format.
  3. Understand tokenization and batching.
  4. See how data flows through training.

Estimated time: 15 minutes | Prerequisites: This lesson.

Additional Resources

Official Documentation

Community

Ready to build your first dataset? Continue to Dataset Fundamentals.