If you dive into PyTorch or neural networks, the terminology can feel overwhelming. But at its core, a neural network is just doing a massive amount of basic middle-school algebra: y = (m * x) + b.
Here is how the core ML concepts map to that equation, using a real estate analogy.
1. Weights and Biases (The m and b)
Imagine a simple model trying to predict the price of a house.
- The Input (
x): Number of bedrooms. - The Weight (
m): How much does each bedroom add to the price? (e.g., $50,000 per bedroom). Weights are multipliers. - The Bias (
b): What is the base price of the land, even if the house has 0 bedrooms? (e.g., $100,000). Biases are offsets.
Every connection in an AI model has a Weight and a Bias. During training, the model is just adjusting millions of these Multipliers and Offsets until it gets the right answers.
2. Decay / L2 Regularization ("Gravity")
When a model trains, sometimes it cheats. It might learn to multiply a specific input by a massive number (like 1,000,000) just to get one specific training example right. This causes the model to "overfit" (it memorizes the training data but fails in the real world).
To stop this, we use Weight Decay (also called L2 Regularization).
Think of Weight Decay as gravity.
It constantly pulls all the parameters toward zero during training. If a parameter wants to be large, it has to fight gravity to prove it’s actually useful. This forces the model to use many small, healthy numbers instead of relying on a few massive, fragile numbers.
3. Why We DON'T Decay Biases
This leads to a critical best practice in training: do not apply weight decay to biases.
Applying "gravity" to Weights (multipliers) is great. It stops them from exploding.
But applying "gravity" to Biases (offsets) is terrible. Going back to the house example: if the true base price of the land is $100,000, but we constantly pull that bias toward $0, we are fighting the truth. We are artificially forcing the base offset to be zero, which cripples the model's math.
The Rule: Apply gravity to the multipliers (weights) to keep them healthy. Leave the offsets (biases) alone so they can shift the baseline to wherever it naturally needs to be.
4. LayerNorm (Layer Normalization)
As data flows through 10 or 20 layers of a Transformer, multiplying by millions of weights, the numbers can get wildly huge or microscopically small.
LayerNorm is like an audio compressor. It sits between layers and squashes the numbers back into a safe, standard range (averaging out to zero) so the next layer doesn't blow out.
Just like Biases, LayerNorm has a couple of parameters that just handle scaling and shifting. We don't apply "gravity" (decay) to them either, because we don't want to mess with the compressor's base settings.
5. AdamW (The Optimizer)
When the model makes a mistake (e.g., predicts the house is worth $500k when it's actually $300k), an algorithm calculates exactly how much to adjust the millions of Weights and Biases to fix the mistake.
That algorithm is called an Optimizer.
- Adam is the most famous, standard optimizer algorithm.
- AdamW is a specific, modernized version of it.
The W stands for "Weight Decay." It is an updated version of Adam that applies that "gravity" math in a decoupled, much more mathematically correct and efficient way.