∑ LLM Internals a learning center
LLM MECHANICS · INTERACTIVE

Learn how modern AI actually works.

From tokenization and attention to sampling, fine-tuning, RAG, and the math underneath it all — every topic is visual, interactive, and concrete. Work the full set and you'll understand these systems better than most people calling themselves experts.

Start with Tokenization → Browse all 22 topics
22 of 22 built 5 categories · mechanics-first
live
self-attention · 1 head
AUDIO TUTOR · FOR CLAUDE & CHATGPT VOICE MODE
No time to sit and read? Learn it on a walk.
One copy-paste prompt turns voice mode into a Socratic tutor that drills you on all 22 topics — diagnostic first, real numbers, honest pushback. Entirely hands-free.
Get the prompt
THE FULL CURRICULUM

22 topics. 5 categories. Mechanics first, metaphors second.

01

The Basics

How text becomes vectors, and how vectors attend to each other.
Byte-pair encoding (BPE) merge animator
LIVE · THE BASICS →
Tokenization
Models never see text — they see integer ids carved out by a merge algorithm that ran once, long before you typed anything.
Byte-pair encoding (BPE) merge animator
Q·K → scale → softmax → ΣV, stepped
LIVE · THE BASICS →
Attention
Every token rewrites itself as a weighted blend of the tokens it decides are relevant — and the weights are just scaled dot products run through a softmax.
Q·K → scale → softmax → ΣV, stepped
Quadratic cost explorer
LIVE · THE BASICS →
Context Window
Attention is all-pairs, so doubling the context quadruples the cost — the wall every long-context trick is trying to climb.
Quadratic cost explorer
2D projection + nearest-neighbor search
LIVE · THE BASICS →
Embeddings
A lookup table that starts as noise and ends up encoding meaning as geometry — direction is semantics.
2D projection + nearest-neighbor search
Sinusoid heatmap + RoPE/ALiBi compare
LIVE · THE BASICS →
Positional Encoding
Attention is order-blind by construction, so position has to be injected — as stacked sine waves, learned vectors, or rotations.
Sinusoid heatmap + RoPE/ALiBi compare
02

Fine-tuning

Adapting a frozen base model without breaking what it already knows.
W + B·A decomposition, rank slider
LIVE · FINE-TUNING →
LoRA
You don’t need to move all 768×768 weights — a rank-8 detour through B·A captures the update with ~2% of the parameters.
W + B·A decomposition, rank slider
fp32→int4 rounding + memory bars
LIVE · FINE-TUNING →
QLoRA
Quantize the frozen base to 4-bit, keep the tiny adapters in full precision — fine-tune a 65B model on one GPU.
fp32→int4 rounding + memory bars
2D loss-landscape drift animation
LIVE · FINE-TUNING →
Catastrophic Forgetting
Chase task B’s minimum with full fine-tuning and the weights walk straight out of task A’s basin.
2D loss-landscape drift animation
Teacher→student KL shrinking
LIVE · FINE-TUNING →
Distillation
The teacher’s soft distribution carries more signal than the hard label — the student learns the dog-vs-cat uncertainty, not just the answer.
Teacher→student KL shrinking
Same text, four vocab sizes
LIVE · FINE-TUNING →
Vocabulary Handling
Vocab size is a dial between sequence length and embedding-table size — and it decides how a rare word shatters.
Same text, four vocab sizes
03

Generation

Turning a probability distribution into actual tokens.
Search tree, beam-width slider
LIVE · GENERATION →
Beam Search vs Greedy
Greedy commits to the locally-best token and can’t take it back; beam keeps k hypotheses alive to escape that trap.
Search tree, beam-width slider
Distribution reshape + live samples
LIVE · GENERATION →
Temperature
One scalar divides the logits before softmax — and reshapes the entire next-token distribution from spiky to flat.
Distribution reshape + live samples
Truncation on the probability bars
LIVE · GENERATION →
Top-k vs Top-p
Top-k keeps a fixed count; top-p keeps a fixed probability mass — one is rigid, one adapts to how confident the model is.
Truncation on the probability bars
Parallel left-to-right vs fill-in
LIVE · GENERATION →
Autoregressive vs Masked
Same transformer block, two training objectives: predict the next token left-to-right, or fill blanks using both sides.
Parallel left-to-right vs fill-in
04

Advanced Concepts

Retrieval, reasoning, routing, and in-context learning.
Retrieve→inject→generate pipeline
LIVE · ADVANCED CONCEPTS →
RAG
Don’t bake the knowledge into weights — retrieve it at query time and paste it into the prompt. The failure mode is the lesson.
Retrieve→inject→generate pipeline
Direct vs step-by-step, side by side
LIVE · ADVANCED CONCEPTS →
Chain-of-Thought
The intermediate tokens aren’t decoration — each step becomes context that anchors the next, turning one hard hop into many easy ones.
Direct vs step-by-step, side by side
Per-token gating + expert activation
LIVE · ADVANCED CONCEPTS →
Mixture of Experts
A gating network routes each token to a few experts out of many — more parameters, roughly constant compute per token.
Per-token gating + expert activation
Prompt builder + accuracy curve
LIVE · ADVANCED CONCEPTS →
Zero-shot vs Few-shot
Examples in the prompt are weight-free learning — accuracy climbs as you spend context on demonstrations.
Prompt builder + accuracy curve
05

Math

The handful of functions that everything above is built on.
Draggable score bars + temperature
LIVE · MATH →
Softmax
The function that turns any vector of scores into a probability distribution — and exaggerates the winner on the way.
Draggable score bars + temperature
Drag p(correct), watch −log p
LIVE · MATH →
Cross-entropy Loss
The loss is just the negative log of the probability you put on the right answer — and the log is why being confidently wrong is catastrophic.
Drag p(correct), watch −log p
Reshape P, Q — watch the asymmetry
LIVE · MATH →
KL Divergence
A directed distance between two distributions — asymmetric on purpose, and lurking inside RLHF, distillation, and VI.
Reshape P, Q — watch the asymmetry
20-layer gradient flow, residual toggle
LIVE · MATH →
Vanishing Gradients
Stack enough layers and the backward signal decays to nothing — residual connections are the highway that let it survive.
20-layer gradient flow, residual toggle