Notes
Quick tips, code snippets, and random thoughts. Bite-sized knowledge drops.
The 4 Lines of PyTorch That Power Self-Attention
When you strip away all the multi-head scaling and architectural boilerplate, the actual "attention" mechanism that powers GPT models is shockingly s...
The Best Resource to Understand Transformers from Scratch
If you are a software engineer who wants to stop treating AI as a black-box API, this is the place to start. Andrej Karpathy (former Director of AI ...
Causal Masking: How LLMs Are Forced to Predict the Future
Language models like GPT-4 are "autoregressive." They predict the next word based on the previous words, over and over again. To do this, they use s...
Don't Apply Weight Decay to Biases or LayerNorm
When setting up your optimizer in PyTorch (like AdamW), it's common to just pass in `model.parameters()` and set `weight_decay=0.1`. **You shouldn...
Weights, Biases, and Decay: An ML Mental Model for Backend Engineers
If you dive into PyTorch or neural networks, the terminology can feel overwhelming. But at its core, a neural network is just doing a massive amount ...
What "Temperature" Actually Does in LLMs (Under the Hood)
When using AI APIs, we’re told to "increase the temperature for more creative responses" and "decrease it for more deterministic answers." It sounds...
Why Character-Level LLMs Hilariously Fail at Meaning
When I was building a small GPT from scratch, I trained it on cooking recipes. I used **character-level tokenization** (where every single letter is ...
Base62, MD5, and SHA-256 — What They Actually Do (Under the Hood)
These three terms — **Base62**, **MD5**, and **SHA-256** — often get mentioned together in system design discussions, even though they solve *very di...
How HLS / DASH Actually Work (Under the Hood)
Most people describe HLS/DASH as *“adaptive bitrate streaming”* and stop there. That explanation is technically correct, but it hides the real eleg...
Node.js Event Loop: How It Really Works
Ever wondered how Node.js handles thousands of concurrent connections with a single thread? The secret is the **Event Loop**. ## The Phases The Nod...