When setting up your optimizer in PyTorch (like AdamW), it's common to just pass in model.parameters() and set weight_decay=0.1.
# ❌ Bad Practice optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)
You shouldn't do this.
Weight decay (L2 regularization) penalizes large weights, pushing them toward zero. This is great for large 2D weight matrices because it distributes learning and prevents overfitting.
But it is actively harmful to apply this to:
- Biases (1D tensors): Biases encode useful offsets. Pushing them to zero artificially limits the network's ability to shift activation functions.
- LayerNorm / BatchNorm parameters: These scale and shift normalizations. Regularizing them messes with the stability they were designed to create.
The Fix
Separate your parameters into two groups before passing them to the optimizer:
# ✅ Good Practice decay_params = [p for p in model.parameters() if p.dim() >= 2] nodecay_params = [p for p in model.parameters() if p.dim() < 2] optim_groups = [ {'params': decay_params, 'weight_decay': 0.1}, {'params': nodecay_params, 'weight_decay': 0.0} ] optimizer = torch.optim.AdamW(optim_groups, lr=1e-3)
By simply splitting based on tensor dimensions (p.dim() >= 2), you automatically exempt all biases and normalization scalars from decay, leading to more stable training.