When I was building a small GPT from scratch, I trained it on cooking recipes. I used character-level tokenization (where every single letter is its own token).
The model learned to perfectly structure a recipe: TITLE, INGREDIENTS, and INSTRUCTIONS. It learned real cooking verbs and proper quantity formats.
But it would routinely generate things like:
TITLE: Spicy Catfish
INGREDIENTS: 3 pounds ground beef ; 1 onion...
Why? Because a character-level model has absolutely no concept of "meaning."
It's predicting one character at a time based entirely on statistical patterns. It has learned that the letters b, e, e, f frequently appear after the INGREDIENTS: section in cooking texts.
But because it sees the world letter-by-letter, the concept of "Catfish" (7 tokens ago in the title) isn't strongly bound as a semantic constraint for the protein that must appear in the ingredients list.
This is exactly why production models (like GPT-4) use Subword Tokenization (like BPE - Byte-Pair Encoding).
With BPE, the model sees "catfish" and "beef" as discrete, single "word chunks" rather than a loose sequence of letters. This forces the model to learn the relationships between the concepts, not just the statistical likelihood of individual letters following one another.