I Tried Building GPT From Scratch. Here's What I Learned.

I write code with Claude Code. I pair-program with Cursor. I've been shipping features on GPT-4, Sonnet for two years.

Last year, a teammate asked me how a transformer actually works.

I fumbled through "it predicts the next token" and changed the subject.

That stuck with me.

So I decided to actually understand it — not the hand-wavy version, but the code. I built a small GPT from scratch in PyTorch, trained it on cooking recipes, and came out the other side with real answers.

The code is on GitHub.

The Output, First

Here are three recipes from a model I trained from random noise in 56 minutes on my MacBook:

TITLE: Spicy Catfish
INGREDIENTS: 3 pounds ground beef ; 1 onion, diced ; 1 green bell pepper,
diced ; 8 cloves garlic, crushed ; ¼ teaspoon crushed red pepper flakes
INSTRUCTIONS: Preheat the oven...

TITLE: Spicy Jerky Burger Glaze
INGREDIENTS: 1 pound smoked salmon, cut into 1-inch pieces ; 3 tablespoons
olive oil ; 1 tablespoon balsamic vinegar ; 1 tablespoon Dijon mustard
INSTRUCTIONS: Mix the cilantro, water...

"Spicy Catfish" — but the main ingredient is ground beef. "Spicy Jerky Burger Glaze" — but the recipe is about salmon.

The model learned the structure of recipes perfectly: TITLE → INGREDIENTS → INSTRUCTIONS → <END>, correct quantity formats (¼ teaspoon, 1 (16 ounce) package), real cooking verbs, proper punctuation.

But it has no concept of whether the title matches the ingredients.

Understanding why it fails teaches you more about how transformers work than understanding why they succeed.

That's not a bug. It's the whole point.

What I Built

• Architecture — Decoder-only Transformer (same family as GPT-2)
• Dataset — 20,000 cooking recipes, 18 million characters
• Tokenization — Character-level, 110 unique characters
• Parameters — 10.82 million
• Training time — ~56 minutes on Apple M3 Pro (MPS backend)
• Final val loss — 0.49 (perplexity: 1.64×)

That last number: after seeing the previous characters, the model narrows the next character down to roughly 1.6 highly probable options out of 110. It's extremely confident.

The Data Pipeline

This part felt the most familiar. It's just ETL.

Getting the Data

from datasets import load_dataset
dataset = load_dataset("Shengtao/recipe", split="train[:40000]")

No API key. No Kaggle credentials. One command.

Each record has a title, ingredient list, and instructions. I standardized everything into a consistent format:

TITLE: Simple Macaroni and Cheese
INGREDIENTS: 1 (8 ounce) box elbow macaroni ; ¼ cup butter ; 2 cups milk
INSTRUCTIONS: Bring a large pot of salted water to a boil. Cook macaroni, 8 minutes.
<END>

20,000 recipes. 18 million characters.

Tokenization

I expected tokenization to be complicated. It's not.

Character-level: find every unique character in your dataset, map it to an integer.

chars = sorted(set(full_text))      # 110 unique chars
stoi  = { ch: i for i, ch in enumerate(chars) }   # char → int
itos  = { i: ch for i, ch in enumerate(chars) }   # int → char

That's the entire tokenizer.

This step gives you something concrete before training a single step: your worst-case loss. A completely random model guessing uniformly over 110 characters would get:

loss = ln(110) ≈ 4.70

Your baseline. The model starts here. Everything after step 0 is learned.

The Architecture

This is what I was most intimidated by. It's simpler than it looks.

Five components. Each one solves one specific problem.

Input: "TITLE: Spicy Catfish\nINGREDIENTS:..."
          ↓  char → integer  (vocab size: 110)
          ↓  token_embedding + position_embedding
   (batch=64, sequence=256, dim=384) per character
          ↓
   ┌───────────────────────────────┐
   │  Transformer Block  ×  6      │
   │   LayerNorm + Attention       │  ← tokens communicate
   │   LayerNorm + Feed-Forward    │  ← tokens think
   └───────────────────────────────┘
          ↓  LayerNorm → Linear(384 → 110)
   Logits: probability over next char at every position
          ↓  cross_entropy loss

Embeddings

Each character maps to a 384-dimensional vector — the token embedding. What is this character?

A character's position also matters. The letter 'e' means something different at position 3 vs position 200. So we add a position embedding — where is this character?

tok_emb = self.token_embedding(idx)           # (B, T, 384)
pos_emb = self.position_embedding(positions)  # (T, 384)
x = tok_emb + pos_emb                         # both encoded in one vector

Both tables start random. Training figures out how to use them.

Self-Attention

Every explanation I'd read made this feel abstract. Here's the concrete version:

When the model reaches INSTRUCTIONS: Preheat the oven, how does it know what dish it's making? It needs to look back at TITLE: Spicy Catfish — 150 characters ago.

Self-attention solves this.

Each character plays three roles simultaneously:

• Query (Q) — "What am I looking for?"
• Key (K) — "What do I have to offer?"
• Value (V) — "What do I share if selected?"

k = self.key(x)    # (B, T, 64) — what I contain
q = self.query(x)  # (B, T, 64) — what I'm looking for
v = self.value(x)  # (B, T, 64) — what I'll share if attended to
 
scores  = q @ k.transpose(-2, -1) * (64 ** -0.5)   # scaled dot-product
scores  = scores.masked_fill(future_mask == 0, float('-inf'))  # causal mask
weights = F.softmax(scores, dim=-1)                 # attention weights
out     = weights @ v

The * 64**-0.5: dot products grow large as vector dimensions grow. Without scaling, softmax saturates and gradients die. Dividing by sqrt(d_k) keeps activations in range.

The causal mask makes this a decoder-only model:

Position 0 sees: [0]
Position 1 sees: [0, 1]
Position 2 sees: [0, 1, 2]
Position 3 sees: [0, 1, 2, 3]

Future positions become -inf before softmax — their weights become exactly 0. The model can never cheat by looking ahead. Generation is possible because of this constraint, not in spite of it.

Multiple Heads

We run 6 independent attention heads in parallel, concatenate their outputs.

Each head specializes in a different relationship type. Some likely track local spelling patterns. Others track structural flow (TITLE: → INGREDIENTS: → INSTRUCTIONS:). Others track ingredient quantity patterns.

outputs  = [head(x) for head in self.heads]   # 6 × (B, T, 64)
combined = torch.cat(outputs, dim=-1)          # (B, T, 384)
return self.proj(combined)                     # (B, T, 384)

Feed-Forward Network

After attention — tokens communicate — each token independently processes what it gathered.

x = self.linear1(x)   # (B, T, 384) → (B, T, 1536)  expand 4×
x = F.gelu(x)         # smooth activation, empirically better than ReLU
x = self.linear2(x)   # (B, T, 1536) → (B, T, 384)  compress back

Attention is communication. FFN is computation. One Block does both.

Residual Connections + LayerNorm

The entire Block in two lines:

x = x + self.attn(self.ln1(x))   # pre-norm → attention → residual
x = x + self.ffwd(self.ln2(x))   # pre-norm → FFN → residual

Residuals (x = x + sublayer(x)) create direct gradient paths through a 6-layer deep network. Without them, gradients vanish before reaching early layers. Each layer only needs to learn the correction.

The Full Model

Six Blocks stacked. Embeddings at the start, LayerNorm + linear projection at the end.

10.82 million parameters. GPT-2 is 124M. GPT-3 is 175B. This is a very small model — and it still produces coherent recipes.

Training

Training a language model is an optimization problem. Forward pass, measure loss, backward pass, update weights.

for step in range(5000):
    lr = get_cosine_lr(step)           # decaying learning rate
    x, y = get_random_batch()          # 64 sequences × 256 chars each
    logits, loss = model(x, targets=y)
    loss.backward()
    clip_grad_norm_(model, 1.0)        # prevent gradient explosions
    optimizer.step()
    optimizer.zero_grad()

What Actually Happened

That 0→250 drop is the "basic English cliff."

In about 4 minutes of training, the model went from treating all 110 characters as equally likely to understanding that spaces follow most words, capital letters start titles, and numbers follow numbers in quantity measurements.

The train/val gap stayed at ~0.03 throughout — never widening. The model was learning rules. Not memorizing 20,000 specific recipes.

Three Training Decisions That Matter

Cosine LR with warmup — LR starts near zero, ramps over 100 steps, then decays to a minimum. The warmup exists because random initialization produces noisy gradients — starting fast causes large destructive updates before the model has any useful structure.

Gradient clipping (clip_grad_norm_(model, 1.0)) — if the gradient norm exceeds 1.0, scale everything down proportionally. Prevents one bad batch from causing a catastrophic weight update.

Weight decay groups — L2 regularization on weight matrices only. Biases and LayerNorm parameters are excluded. Decaying a bias toward zero is actively harmful. AdamW handles this correctly; vanilla Adam doesn't.

What It Gets Right and Wrong

Right

• Structure — every output follows TITLE → INGREDIENTS → INSTRUCTIONS → <END>
• Quantities — ¼ teaspoon, 1 (16 ounce) package, 2 tablespoons — formatted correctly, including Unicode fractions
• Verbs — Preheat, Stir until combined, Drain and set aside — contextually appropriate throughout

Wrong

• "Spicy Catfish" with ground beef as the main ingredient
• "Spicy Jerky Burger Glaze" where the protein is salmon
• Occasional ingredient duplication
• Instructions that trail off mid-sentence

A character-level model predicts one character at a time. It has learned that "beef" and "salmon" both commonly appear after INGREDIENTS:. It hasn't learned that "catfish" in the title should constrain which protein appears below it.

Char-level is a great teaching tool precisely because it fails visibly. You can point at exactly why.

Temperature

logits = logits / temperature   # scale before softmax

• temperature=0.5 — safe, repetitive, deterministic
• temperature=0.8 — coherent structure, surprising combinations ← the sweet spot
• temperature=1.2 — chaotic, occasionally brilliant

Looking Inside the Model

Because I built it, I can see inside it.

Attention Heatmaps

At every layer and head, I can extract the attention weight matrix — what each character is attending to.

Three things to notice:

• The triangle shape — every heatmap is lower-triangular. That's the causal mask. Characters can only see the past.
• Early layers (top rows) — heavy diagonal patterns. Characters attending to immediate neighbors to form words.
• Deep layers (bottom rows) — attention scatters globally. Characters in INSTRUCTIONS: attending back to TITLE: tokens — the model checking what dish it's supposed to be making before predicting the next cooking step.

Next-Character Distribution

After TITLE: Banana Bread\nINGREDIENTS, the model assigns near-certain probability to S — completing INGREDIENTS:. It's seen this exact sequence 20,000 times.

Entropy (top right) measures uncertainty. When completing a structural keyword, entropy is nearly zero. When generating free-form ingredient text, entropy rises — multiple plausible next characters exist.

What I Actually Learned

• The attention equation is four lines of matrix math. Papers make it look harder because papers need to justify the research.
• Char-level fails in visible, explainable ways — which is exactly why it's the right starting point for understanding.
• The training loop is just optimization. Forward pass, loss, backward, update. If you've ever tuned a slow database query or a cache eviction policy, you already understand the intuition. The math is different. The mental model is the same.
• "Val loss of 0.49" means nothing to anyone. "The model narrows the next character down to 1.6 options out of 110" means something.
• I never told the model that recipes have structure. It discovered TITLE → INGREDIENTS → INSTRUCTIONS → <END> entirely on its own — from 18 million characters of raw text. That, more than anything, made "emergence" feel real to me.
• I've been writing code with Claude Code without understanding how Claude Code works. That feels like a solved problem now.

Building taught me in a week what reading hadn't managed in months. There's no substitute for writing q @ k.transpose(-2, -1) * scale and watching the loss drop in response.

What's Next

• BPE tokenization — replace 110 characters with 50,257 subword tokens via tiktoken. One file change. "Spicy Catfish" would be far less likely to contain ground beef.
• Flash Attention — same math, 2-3× faster on CUDA. Trade-off: you lose the ability to extract attention weights for heatmaps.
• Scaling — change n_layer=6, n_embd=384 to n_layer=12, n_embd=768. Identical architecture, 4× more parameters, noticeably better output.

Run It Yourself

git clone https://github.com/harshmange44/recipe-gpt
cd recipe-gpt
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
 
python data/prepare.py   # download + tokenize 20K recipes (~2 min)
python train.py          # train on MPS/CUDA/CPU (~30-60 min on M3 Pro)
python sample.py --prompt "TITLE: Spicy" --temperature 0.8
python visualize.py      # loss curves + attention heatmaps

Every file is heavily commented with explanations of why each decision was made — not just what the code does.

Trained on a MacBook M3 Pro, 18GB unified memory. No cloud, no rented GPUs. Just a laptop that got very warm.