n150 n300 T3000 p100 p150 p300c Galaxy 15 min Draft

Dataset Fundamentals

Learn to create and validate training datasets for fine-tuning AI models. This is your first hands-on lesson in the Custom Training series.

What You'll Learn

Time: 15 minutes | Prerequisites: Understanding Custom Training

Where this fits in the track:

graph LR
    A[Understand] --> B[Datasets]
    B --> C[Configuration]
    C --> D[Fine-tuning]
    D --> E[Multi-Device]
    E --> F[Experiment Tracking]
    F -.-> G[Architecture Basics]
    G -.-> H[From Scratch]

    style B fill:#1B8EB1,stroke:#092221,stroke-width:3px

A quick note on scope before you start: this lesson teaches dataset curation the way most of the fine-tuning world does it — JSONL prompt/response pairs, validated and versioned. That skill transfers everywhere. But the trainer this track actually runs, train_nanogpt.py (ttml/tt-train's Python entry point, see Fine-tuning Basics), doesn't take JSONL directly — it takes a plain-text corpus. The "How Your Data Reaches tt-train" section below closes that gap honestly, with a real script to get from one to the other.

Why Datasets Matter

Your model is only as good as your data.

A model trained on:

Key principle: Spend more time on data quality than hyperparameter tuning.

JSONL Format (Simple and Universal)

We use JSONL (JSON Lines) format:

Basic Structure

{"prompt": "Your question or input", "response": "Expected output"}
{"prompt": "Another question", "response": "Another output"}

Real Examples

{"prompt": "What is a neural network?", "response": "Imagine teaching a child to recognize cats by showing them thousands of cat pictures. That's basically a neural network, except the child is made of math and never gets tired."}

{"prompt": "How do I learn to code?", "response": "Start by breaking things. Then learn why they broke. Then break them again, but differently. Repeat until you're hired."}

{"prompt": "Explain recursion simply", "response": "To understand recursion, you must first understand recursion. (But seriously: a function that calls itself until it doesn't need to anymore.)"}

Why this format:

Format Comparison: Why JSONL?

graph TD
    A[Dataset Format Choice] --> B{Your Needs}

    B --> C[Human Readable?]
    B --> D[Version Control?]
    B --> E[Universal Support?]
    B --> F[Easy Validation?]

    C --> G[✅ JSONLPlain text, any editor]
    D --> G
    E --> G
    F --> G

    C --> H[❌ Binary FormatsPkl, TFRecord, Arrow]
    D --> H
    E --> I[⚠️ CSVEscaping issues, limited structure]
    F --> I

    style G fill:#50C878,stroke:#333,stroke-width:2px
    style H fill:#E85D75,stroke:#333,stroke-width:2px
    style I fill:#4A90E2,stroke:#333,stroke-width:2px
    style B fill:#6C757D,stroke:#333,stroke-width:2px

Real talk about format choices:

Format Pros Cons Use When
JSONL Human-readable, git-friendly, universal Larger file size 99% of the time
CSV Simple, spreadsheet-compatible Hard to escape quotes, no nested data Flat tabular data only
Parquet/Arrow Efficient storage, fast loading Binary, needs special tools 100K+ examples
Pickle Python-native, can store anything Python-only, security risks Never for datasets

For custom training on Tenstorrent: Start with JSONL. Only switch to Parquet if you have 100,000+ examples and proven performance issues.

The Shakespeare Dataset: A Classic Training Corpus

Before you create your own dataset, let's examine one of the most famous teaching datasets in machine learning history - and understand why it remains valuable today.

Historical Context: The "Hello World" of Language Models

The tiny-shakespeare corpus was popularized by Andrej Karpathy's groundbreaking work:

Why Shakespeare became the standard:

Download the corpus:

wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

Dataset Characteristics

Let's understand what makes this dataset special:

Characteristic Value Significance
Size 1.1MB (1.1 million characters) Perfect for 6-layer, 384-dim transformer
Source Complete works of Shakespeare (40 plays) Rich literary structure
Vocabulary ~65 unique characters (printable ASCII) Character-level modeling, no tokenization
Format Plain text Raw, continuous sequence
Structure Character names, dialogue, stage directions Strong learning signal from dramatic format

Example snippet:

ROMEO:
What lady is that, which doth enrich the hand
Of yonder knight?

Servant:
I know not, sir.

What the model learns from this format:

What Makes It Pedagogically Perfect

Shakespeare isn't just famous - it's strategically useful for teaching dataset fundamentals:

✅ Fast iteration cycles

✅ Observable structure learning

✅ Rich hierarchical structure

✅ Human-readable validation

✅ Continuous text

The Learning Journey: What Models Actually Learn from Shakespeare (and Where It Stops)

Understanding how models learn from Shakespeare — and where that learning plateaus on modest models and hardware — teaches you how they'll behave on ANY dataset you build. Here's the real, step-based progression, taken directly from Fine-tuning Basics's verified training run on this corpus (a ~10.8M-parameter GPT-2-style model, Blackhole p300c):

graph LR
    A[Random WeightsStep 1, Loss 4.66Uniform-random characters] --> B[StructureStep 500, Loss 2.30Speaker names, colons, line breaks]
    B --> C[Scattered WordsStep 1500, Loss 1.61Real words among invented syllables]
    C --> D[Diminishing ReturnsStep 3000, Loss 1.41Still not coherent text]

    style A fill:#E85D75,stroke:#333,stroke-width:2px
    style B fill:#FFA07A,stroke:#333,stroke-width:2px
    style C fill:#FFD700,stroke:#333,stroke-width:2px
    style D fill:#4A90E2,stroke:#333,stroke-width:2px

Step 1 (loss 4.66): the random baseline

ln(96) ≈ 4.56 — the entropy of guessing uniformly among the corpus's ~96 characters. This is what "the model hasn't learned anything yet" looks like as a number.

Step 500 (loss 2.30): structure emerges — actual output, verbatim, from Fine-tuning Basics:

antlastolptatcsamthartosiga mabomantarsaemoowouatheatouomarathanoouarearanawarouranouanoonarorofoururave,
INare I matolo tone.
BENGHA:
Tonot wit ithald weat thay theard.

Capitalization, line breaks, and NAME:-style speaker headers are already there. Almost none of the rest is real English.

Step 1500 (loss 1.61): a scatter of real words — same run, same command, later checkpoint:

tspafrttwathofarantttorarorasthatorororitoretoshreamawinytoucanowinayonanunanousoreiseonofasayonouro anouroura meather,
Yor withe, tha wenon wano herde.

SENGERDI:
Tow towall wawind yownthonger mates this tiedern,

A handful of real short words appear ("this", fragments like "wit", "tha") embedded in mostly-invented syllables. The structural skeleton is intact; fluency is not.

Step 3000 (loss 1.41): the plateau

ytmtatpatofasttanfabanadrtofriorthouandeithonatino theayorearanunearananonoounanouroorowareanaroratorouree inorenod.
WEBUCHELARI:
I ie mod hadeste, hand ast wame, souce mee,

Loss kept dropping between step 1500 and step 3000, but the text did not get noticeably more readable. Training from Scratch pushes a different architecture on this same corpus all the way to loss 0.18 and hits the same ceiling: a much lower loss bought no visible improvement in coherence — that run's honest conclusion is overfitting, not mastery.

The takeaway for this lesson: on this hardware, with this corpus and these model sizes, training produces Shakespeare-shaped text — the structural skeleton — not Shakespeare-like prose. Getting past that ceiling needs a much bigger model trained on much more data, which is a different problem than "curate a good dataset." This lesson stays honest about that boundary instead of promising fluency a nano model on a 1MB corpus can't deliver.

Why This Dataset Still Matters in 2026

You might think: "Why learn from a 2015 dataset when we have GPT-4 and modern LLMs?"

Because the principles are timeless:

🎯 Hierarchical learning is universal

💡 Quality structure > massive quantity

🔬 Character-level principles scale to modern models

⚡ Fast experimentation enables learning

From Shakespeare to Your Domain

The learning patterns you observe with Shakespeare directly transfer to your custom domain — through the same early stages, and with the same honest limit:

Code generation models:

Medical note generation:

Legal contract generation:

The principle: Models learn hierarchically regardless of domain — structure first, then vocabulary, then style. Whether they ever reach fluent, coherent output is a question of model scale and data volume, not something a dataset lesson can promise. Fine-tuning Basics and Training from Scratch show exactly where that ceiling sits for the models this track actually trains.

Key Insight: What Shakespeare Teaches You About Datasets

When you train on Shakespeare and watch the progression from random characters through structure to a scatter of real words, you learn:

This knowledge transfers to every dataset you'll ever create.

When you build your medical chatbot, legal assistant, or code generator, you'll recognize these same early stages — structure, then scattered vocabulary — and you'll know that getting past them to genuinely fluent, coherent output takes a much bigger model and much more data than a from-scratch nano run on a small corpus, exactly as Fine-tuning Basics and Training from Scratch demonstrate.

Shakespeare isn't just a dataset - it's a lesson in what good structure buys you, and what it doesn't.

Creating Your First Dataset

Creating a dataset is straightforward - it's just a text file with one JSON object per line. Here's how to start:

Step 1: Create Your JSONL File

Create a new file called my_dataset.jsonl in your working directory:

cd ~/tt-scratchpad
touch my_dataset.jsonl

Step 2: Add Your First Examples

Open the file in your text editor and add training examples:

{"prompt": "What is a neural network?", "response": "A neural network is a series of algorithms that mimic the human brain to recognize patterns in data. It learns by adjusting connections between nodes based on training examples."}
{"prompt": "Explain gradient descent", "response": "Gradient descent is an optimization algorithm that finds the minimum of a function by iteratively moving in the direction of steepest descent. Think of it like rolling a ball down a hill to find the lowest point."}
{"prompt": "What is overfitting?", "response": "Overfitting occurs when a model learns the training data too well, including noise and outliers, making it perform poorly on new data. It's like memorizing answers without understanding the concepts."}

What makes these good examples:

Dataset Quality Guidelines

1. Consistency is Key

Good:

{"prompt": "What is overfitting?", "response": "When your model memorizes the training data like a student memorizing answers without understanding. It aces the practice test but fails the real exam."}

{"prompt": "Explain gradient descent", "response": "Imagine you're blindfolded on a mountain and want to reach the valley. You feel the slope with your feet and take small steps downhill. That's gradient descent."}

Why it's good:

Bad:

{"prompt": "What is overfitting?", "response": "A statistical model is said to be overfitted when it captures noise in the training data."}

{"prompt": "Explain gradient descent", "response": "lol just go downhill bro 😎"}

Why it's bad:

2. Representative Examples

Your dataset should cover the range of inputs you expect:

Example coverage:

Key principle: If you want the model to handle a type of question, include examples of it.

3. Quality Over Quantity

Better: 50 carefully crafted examples Worse: 5,000 auto-generated examples

Why?

Rule of thumb:

4. Balanced Distribution

Avoid heavily imbalanced datasets:

Bad:

Good:

Why: Model will overfit to common patterns, underperform on rare ones.

Validating Your Dataset

Step 1: Run the Validator

We provide a validation script that checks:

Run validation:

You can create a simple Python script to validate your dataset:

import json

def validate_jsonl(filepath):
    """Validate JSONL dataset format"""
    errors = []
    line_num = 0

    with open(filepath, 'r') as f:
        for line in f:
            line_num += 1
            try:
                data = json.loads(line)
                if 'prompt' not in data:
                    errors.append(f"Line {line_num}: Missing 'prompt' field")
                if 'response' not in data:
                    errors.append(f"Line {line_num}: Missing 'response' field")
                if data.get('prompt', '').strip() == '':
                    errors.append(f"Line {line_num}: Empty prompt")
                if data.get('response', '').strip() == '':
                    errors.append(f"Line {line_num}: Empty response")
            except json.JSONDecodeError as e:
                errors.append(f"Line {line_num}: Invalid JSON - {e}")

    if errors:
        print("❌ Validation failed:")
        for error in errors:
            print(f"  {error}")
    else:
        print(f"✅ Dataset validation passed! ({line_num} examples)")

# Usage
validate_jsonl("my_dataset.jsonl")

Step 2: Interpret Results

If validation passes:

✅ Dataset validation passed!
   Total examples: 50
   Average prompt length: 42 characters
   Average response length: 156 characters
   Ready for training!

If validation fails:

❌ Line 23: Missing 'response' field
❌ Line 45: Empty prompt string
❌ Line 67: Invalid JSON syntax

Fix these issues and run validation again.

Common Issues and Fixes

Issue Fix
Invalid JSON Check for missing quotes, commas, braces
Missing field Ensure both prompt and response present
Empty string Remove empty entries or fill with content
Encoding error Save file as UTF-8
Very long response Keep responses under 512 tokens (~400 words)

How Your Data Reaches tt-train

Once your dataset is validated JSONL, the natural next question is: how does it actually get to a model? For a lot of the fine-tuning ecosystem — Hugging Face's trl, hosted fine-tuning APIs — JSONL prompt/response pairs are the direct input. This track's own trainer is more specific about what it wants.

Fine-tuning Basics runs train_nanogpt.py, the trainer script that ships with ttml (tt-train's Python bindings), against real Tenstorrent hardware. Its data argument looks like this:

python train_nanogpt.py \
  --data_path ~/tt-scratchpad/training/data/shakespeare.txt \
  --num_epochs 30 \
  --batch_size 4 \
  --model_save_path ~/tt-metal/tt-train/checkpoints/shakespeare_stage2.pkl \
  --fresh

--data_path points at one plain-text file — not JSONL, not a prompt/response schema. By default, train_nanogpt.py tokenizes that file one character at a time (ttml.common.data.CharTokenizer) and trains next-character prediction on it. That's exactly why the tiny-shakespeare corpus you downloaded earlier in this lesson works as-is: input.txt is already the shape the trainer wants, no conversion needed.

tt-train also supports subword tokenization: set tokenizer_type: bpe in the training config and point tokenizer_path at a saved Hugging Face tokenizer, and the corpus gets byte-pair-encoded instead of split into characters. Either way — char-level or BPE — the trainer wants already-tokenizable text, not a JSON schema of examples.

Bridging JSONL to a plain-text corpus

If you've curated a JSONL dataset above and want to point train_nanogpt.py at it, flatten it into text first. prompt/response fields don't survive the trip — the trainer just sees characters — so the flattening step is where you decide how those fields get joined:

import json

def jsonl_to_plain_text(jsonl_path, output_path, separator="\n\n"):
    """Flatten a JSONL prompt/response dataset into the single plain-text
    stream train_nanogpt.py's --data_path expects.

    tt-train's default tokenizer has no concept of a "prompt" or "response"
    field -- it reads characters (or BPE tokens, if tokenizer_type: bpe is
    set) from one continuous file. This is the conversion step that makes a
    JSONL dataset usable by that trainer.
    """
    count = 0
    with open(jsonl_path, "r") as infile, open(output_path, "w") as outfile:
        for line in infile:
            line = line.strip()
            if not line:
                continue
            example = json.loads(line)
            outfile.write(example["prompt"])
            outfile.write("\n")
            outfile.write(example["response"])
            outfile.write(separator)
            count += 1
    return count

# Usage
n = jsonl_to_plain_text("my_dataset.jsonl", "my_dataset.txt")
print(f"Flattened {n} examples into my_dataset.txt")

Then hand the flattened file to train_nanogpt.py the same way Fine-tuning Basics hands it shakespeare.txt:

python train_nanogpt.py \
  --data_path ~/tt-scratchpad/training/data/my_dataset.txt \
  --num_epochs 100 \
  --batch_size 4 \
  --model_save_path ~/tt-metal/tt-train/checkpoints/my_model.pkl \
  --fresh

So the JSONL work above isn't wasted — it's still the right format for curating, reviewing, and diffing examples, and it's what plenty of other fine-tuning tools want natively. It just isn't the literal input to this track's trainer; that trainer eats plain text.

Want to see a tokenizer built by hand — byte-pair encoding, merges, vocabulary, all of it — instead of taking CharTokenizer or a Hugging Face tokenizer on faith? Tokenizer & Data in the from-scratch track builds a BPE tokenizer from raw bytes and a data pipeline around it, entirely from first principles, and shows exactly what tokenizer_type: bpe is doing under the hood.

Dataset Creation Workflow

Here's how successful dataset creation flows, from idea to validated training data:

graph TD
    A[Define TaskWhat should model learn?] --> B{Creation Method}

    B --> C[Manual Curation]
    B --> D[Semi-Automated]
    B --> E[Conversion]

    C --> F[Write Examples5-10 per topic]
    D --> G[Generate with AIGPT-4/Claude]
    E --> H[Extract from SourceDocs, Q&A, logs]

    F --> I[Review & RefineConsistency check]
    G --> J[Curate & FilterRemove bad examples]
    H --> K[Clean & NormalizeFix formatting]

    I --> L[Validate FormatRun validator script]
    J --> L
    K --> L

    L --> M{Passes?}
    M -->|No| N[Fix Issues]
    N --> L
    M -->|Yes| O[Quick Training Test10-20 steps]

    O --> P{Quality Check?}
    P -->|Issues| Q[Refine Dataset]
    Q --> I
    P -->|Good| R[✅ Ready for Training!]

    style A fill:#4A90E2,stroke:#333,stroke-width:2px
    style C fill:#7B68EE,stroke:#333,stroke-width:2px
    style D fill:#7B68EE,stroke:#333,stroke-width:2px
    style E fill:#7B68EE,stroke:#333,stroke-width:2px
    style L fill:#E85D75,stroke:#333,stroke-width:2px
    style O fill:#E85D75,stroke:#333,stroke-width:2px
    style R fill:#50C878,stroke:#333,stroke-width:3px

Key insight: Dataset creation is iterative. Your first version won't be perfect - that's okay! Train, evaluate, refine, repeat.

Option 1: Manual Curation (Highest Quality)

  1. Brainstorm: List topics/questions you want covered
  2. Write examples: Craft 5-10 examples per topic
  3. Iterate: Read through, refine tone/style
  4. Validate: Run validation script
  5. Test: Train on small sample, check outputs

Time: 2-4 hours for 50-200 examples Quality: Highest - you control everything

Option 2: Semi-Automated (Balanced)

  1. Generate: Use GPT-4/Claude to generate 100+ examples
  2. Curate: Manually review, edit, filter
  3. Augment: Add your own examples to fill gaps
  4. Validate: Run validation script
  5. Test: Sample outputs, refine as needed

Time: 1-2 hours for 200+ examples Quality: Good - AI generates, you curate

Option 3: Dataset Conversion (Existing Data)

  1. Source data: Existing Q&A, documentation, conversations
  2. Convert: Write script to extract prompt/response pairs
  3. Clean: Remove duplicates, normalize format
  4. Validate: Run validation script
  5. Test: Check quality on sample

Time: Varies based on source data Quality: Depends on source quality

Understanding Tokenization

Your dataset goes through several transformations before reaching the model. The exact shape of the early stages depends on which trainer you're targeting (JSONL for curation-friendly tools, a flattened plain-text corpus for train_nanogpt.py — see the section above) but every path converges on the same last three steps:

graph LR
    A[Raw TextIdeas, Q&A pairs] --> B[Curated DatasetJSONL or plain-text corpus]
    B --> C[ValidationCheck format & quality]
    C --> D[TokenizationText → Numbers]
    D --> E[BatchingGroup examples]
    E --> F[Training LoopModel learns]

    style A fill:#4A90E2,stroke:#333,stroke-width:2px
    style B fill:#7B68EE,stroke:#333,stroke-width:2px
    style C fill:#E85D75,stroke:#333,stroke-width:2px
    style D fill:#7B68EE,stroke:#333,stroke-width:2px
    style E fill:#7B68EE,stroke:#333,stroke-width:2px
    style F fill:#50C878,stroke:#333,stroke-width:2px

Each stage matters:

Why show this? Understanding the pipeline helps you debug issues. If training fails, you can check each stage.

What is Tokenization?

Tokenization: Breaking text into pieces (tokens) the model can process.

Example:

Text: "What is a neural network?"
Tokens: ["What", " is", " a", " neural", " network", "?"]
Token IDs: [1724, 374, 264, 30828, 4009, 30]

Why It Matters

For fine-tuning:

Checking Token Counts

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T")

prompt = "What is a neural network?"
response = "Imagine teaching a child to recognize cats..."

prompt_tokens = tokenizer.encode(prompt)
response_tokens = tokenizer.encode(response)

print(f"Prompt: {len(prompt_tokens)} tokens")
print(f"Response: {len(response_tokens)} tokens")
print(f"Total: {len(prompt_tokens) + len(response_tokens)} tokens")

Rule of thumb: Keep total (prompt + response) under 512 tokens for fine-tuning.

Where this applies to tt-train: train_nanogpt.py's default CharTokenizer doesn't need any of this — one character is one token, and there's no external vocabulary to load. This section matters once you're on the tokenizer_type: bpe path (a real subword tokenizer, loaded from tokenizer_path) or working with datasets destined for other, tokenizer-based fine-tuning tools.

Advanced: HuggingFace Datasets Integration

Once you have JSONL working, you can integrate with HuggingFace datasets — useful for large corpora, preprocessing pipelines, or pulling an existing dataset from the Hub instead of writing one by hand.

Downloading a Dataset from the Hub

Use the hf CLI, not huggingface-cli — the latter is deprecated:

hf download roneneldan/TinyStories --repo-type dataset --local-dir ~/data/tinystories

Loading JSONL (or a downloaded dataset) with HuggingFace

from datasets import load_dataset

# Load your own JSONL file
dataset = load_dataset("json", data_files="my_dataset.jsonl")

# Or load a plain-text corpus the same way train_nanogpt.py would read it
# (each line becomes one example, so this doesn't apply a schema at all)
text_dataset = load_dataset("text", data_files="my_dataset.txt")

# Access examples
for example in dataset["train"]:
    print(example["prompt"])
    print(example["response"])

Benefits

When to Use

For train_nanogpt.py specifically: none of this is required. It reads a single text file directly — HuggingFace datasets is a convenience for building or downloading that file, not a dependency of the trainer itself.

Hands-On: Customize the Dataset

Now that you understand the format, let's extend the starter dataset.

Challenge: Add 10 New Examples

  1. Think of 10 questions relevant to your domain
  2. Write clear, helpful responses (2-3 sentences each)
  3. Add them to your dataset file
  4. Run validation to check format
  5. Test with a few training steps to see if the model learns

Example topics to add:

Dataset Versioning Best Practices

As you iterate on your dataset:

1. Use Git for Version Control

git add my_dataset.jsonl
git commit -m "Add 10 examples about model deployment"

Why: Track what changed, revert if needed, collaborate.

2. Tag Dataset Versions

git tag -a dataset-v1.0 -m "Initial 50-example dataset"
git tag -a dataset-v1.1 -m "Added 10 deployment examples"

Why: Know which dataset produced which model.

3. Document Changes

Keep a DATASET_CHANGELOG.md:

## v1.1 (2026-02-01)
- Added 10 examples about model deployment
- Fixed typos in examples 23, 45

## v1.0 (2026-01-30)
- Initial release with 50 examples
- Focus on ML fundamentals

Why: Context for future you (or collaborators).

Common Pitfalls to Avoid

❌ Don't: Include Personal/Sensitive Data

Why: Privacy, legal, ethical concerns.

❌ Don't: Use Only Examples from One Source

Why: Model overfits to that source's quirks.

❌ Don't: Ignore Edge Cases

Why: Real-world inputs aren't perfect.

❌ Don't: Make Examples Too Long

Why: Efficiency and simplicity.

Real-World Datasets: Inspiration

You've learned the mechanics of creating datasets - but what makes a dataset truly valuable? Let's explore creative and impactful dataset ideas.

Domain-Specific Excellence

Code & Technical Writing:

Creative & Educational:

Business & Professional:

Small Datasets, Big Impact

You don't need thousands of examples:

🎯 50 examples:

🎯 200 examples:

🎯 1000 examples:

The pattern: Small, high-quality datasets outperform large, mediocre ones for specialized tasks.

Datasets That Scale From n150 to Production

Start small, validate fast:

  1. Week 1 (n150): Create 50-100 examples, fine-tune in 1-2 hours
  2. Week 2 (n150): Test with real users, gather feedback, refine dataset
  3. Week 3 (n150 or n300): Expand to 200-500 examples based on feedback
  4. Month 2 (n300/T3000): Scale to 1000+ examples, multi-task fine-tuning
  5. Production: Deploy with vLLM Production, serve thousands of requests/day

Real example from the wild:

Your Dataset Idea Generator

Ask yourself:

  1. What knowledge do I have that base models don't?
    • Industry terminology, company processes, specialized domains
  2. What tasks do I (or my team) repeat daily?
    • Code reviews, documentation, summarization, translation
  3. What would save 1 hour/day if automated?
    • That's 250 hours/year - worth building a dataset for!

Imagine: The 100-Example Challenge

Pick a task you know well. Spend 3-4 hours creating 100 prompt/response pairs. Fine-tune on n150 (1-2 hours). Deploy with vLLM. You now have a specialized AI assistant for that task.

Total time investment: One afternoon. Potential impact: Hundreds of hours saved over a year.

The question isn't "Is my dataset idea good enough?"

The question is "What problem will I solve first?"

Key Takeaways

✅ JSONL format is simple and universal

✅ Quality beats quantity (50 great examples > 5,000 mediocre ones)

✅ Consistency matters (tone, length, format)

✅ Validation catches errors before training

✅ Tokenization determines cost and length limits

✅ train_nanogpt.py wants plain text, not JSONL — flatten curated examples into a text corpus before handing them to --data_path

✅ Version your datasets like code

Next Steps

Configuration Patterns

You have your dataset! Next, you'll learn how to configure training using YAML files:

  1. Understand training configuration structure
  2. Set up device configuration (n150/n300/p100/p300c)
  3. Configure logging and checkpointing
  4. Create hardware-specific configs

Estimated time: 15 minutes | Prerequisites: This lesson.

Additional Resources

Dataset Examples

Validation Tools

Dataset Creation Tools

Ready to configure your training? Continue to Configuration Patterns.