Language models like GPT-4 are "autoregressive." They predict the next word based on the previous words, over and over again.
To do this, they use self-attention, which allows every word in a sequence to look at every other word to gather context.
But there’s a problem during training.
We train these models by feeding them entire documents at once. If every word can look at every other word, the model will just "cheat" by looking at the next word in the sentence to predict what the next word is.
We need a way to enforce time. We need to blind the model to the future.
This is done using Causal Masking.
The Attention Matrix
When a model processes a sequence, it generates an attention score for every possible pair of words.
Imagine a 4-word sentence. The attention scores form a 4x4 grid.
Word 1 Word 2 Word 3 Word 4
Word 1 [ 0.8, 0.1, 0.05, 0.05 ]
Word 2 [ 0.2, 0.6, 0.1, 0.1 ]
Word 3 [ 0.1, 0.3, 0.5, 0.1 ]
Word 4 [ 0.05, 0.05, 0.1, 0.8 ]
In this unmasked state, Word 1 is looking at Word 4. It can see the future.
The Causal Mask
To fix this, we create a "mask" — a matrix of the exact same size, filled with zeros in the lower triangle and negative infinity (-inf) in the upper triangle.
[ 0, -inf, -inf, -inf ]
[ 0, 0, -inf, -inf ]
[ 0, 0, 0, -inf ]
[ 0, 0, 0, 0 ]
We apply this mask directly to the raw attention scores before turning them into probabilities.
scores = scores.masked_fill(mask == 0, float('-inf'))
Why Negative Infinity?
Attention scores must be converted into probabilities (percentages that sum to 1.0) using the softmax function.
The mathematical property of softmax is that e^(-inf) = 0.
When the masked scores pass through softmax, every -inf becomes exactly 0.0.
Word 1 Word 2 Word 3 Word 4
Word 1 [ 1.0, 0, 0, 0 ]
Word 2 [ 0.4, 0.6, 0, 0 ]
Word 3 [ 0.2, 0.3, 0.5, 0 ]
Word 4 [ 0.1, 0.1, 0.2, 0.6 ]
The Result
Look at the rows now:
- Word 1 can only attend to Word 1.
- Word 2 can attend to Word 1 and Word 2.
- Word 3 can attend to Word 1, Word 2, and Word 3.
- Word 4 can attend to everything.
Information is mathematically blocked from traveling backwards in time.
There are no complex for loops or stateful variables tracking time. The arrow of time is enforced by a static, triangular matrix of negative infinities.