Catastrophic Forgetting

You fine-tune a general model on medical question answering. The task loss falls beautifully, the medical eval comes back great — and then someone asks the deployed model to draft an email and it responds like a discharge summary. Its general competence didn’t degrade gracefully; whole capabilities fell off a cliff. That cliff has had a name since McCloskey & Cohen documented it in neural networks in 1989: catastrophic forgetting (originally “catastrophic interference”).

The cause is not a bug. Fine-tuning is gradient descent on a new loss surface — usually plain cross-entropy on the new data. The optimizer’s entire job is to move the weights wherever that one loss decreases. It does not know, and cannot know, that the current weight values also encode “how to write English,” “what a polite email sounds like,” and “Paris is in France.” Those capabilities live in the same shared parameters it is about to update. Every step toward the new task’s minimum is free to be a step away from the configuration that stored the old knowledge.

The key mental shift: the optimizer is not misbehaving. It is doing exactly what you asked — minimizing the only loss you handed it. Forgetting is what “success” on that narrow objective looks like from every other task’s point of view. Which also tells you the shape of every fix in advance: they all change what the optimizer is allowed to do, not how big the model is.

Picture the weights as a point in parameter space. Pretraining left that point at (or near) a minimum of the pretraining loss — call it task A’s basin. Fine-tuning defines a second surface over the same space — task B’s — whose minimum is somewhere else. The gradient the optimizer follows is computed only from task B’s surface. If the two minima don’t coincide, descending B means climbing A. How much you forget is, to first order, a function of one thing: how far the weights walk.

Every real mitigation is a variation on one move — keep the weights near the pretrained point while still descending the new loss:

• LoRA / parameter-efficient fine-tuning — freeze the base weights entirely and route the update through a small low-rank detour. The base cannot move, the detour is small, and it’s removable: subtract the adapter and the pretrained model is back, bit for bit. Empirically LoRA both learns less and forgets less than full fine-tuning (Biderman et al., 2024) — the leash binds in both directions.
• Replay (rehearsal) — mix original-distribution data into the fine-tuning batches. This doesn’t constrain the optimizer; it changes the surface, so the blend’s minimum sits between the two tasks. This is the fix used at production scale: InstructGPT’s RLHF caused regressions on public NLP benchmarks (the “alignment tax”), and mixing pretraining gradients back in (PPO-ptx) was the repair — replay, in the wild.
• Elastic weight consolidation (EWC) — add a quadratic penalty on moving each weight, scaled by how important that weight was to the old task (importance estimated by the Fisher information; Kirkpatrick et al., 2017). A per-weight leash: stiff where old knowledge lives, slack elsewhere.
• Small learning rate + early stopping — the crudest leash: take small steps and stop soon, so the walk simply never gets far. It’s implicit regularization toward the initialization, and it’s why most fine-tuning recipes use learning rates one to two orders of magnitude smaller than pretraining.

Strip the problem to two parameters so nothing can hide. Task A’s loss is a bowl centered at $a = (0, 0)$; task B’s is an identical bowl at $b = (4, 2)$:

The model starts fully pretrained — $w = (0,0)$, task-A loss exactly 0, task-B loss $\tfrac{1}{2}(4^2 + 2^2) = 10$. Fine-tune with plain gradient descent on $L_B$ only, learning rate $\eta = 0.3$, so each step is $w \leftarrow w - 0.3\,(w - b)$ — it covers 30% of the remaining distance to $b$ every step:

Ten steps. The fine-tune is a total success — task-B loss fell from 10 to 0.008 — and task-A loss climbed from 0 to 9.443, which is 94% of the way to the worst case of 10 (the loss a never-pretrained model at $b$ would have). Both curves are the same geometric decay read in opposite directions: $L_B(t) = 10 \cdot 0.49^t$ falls exactly as $L_A(t) = 10\,(1 - 0.7^t)^2$ rises. Nothing dramatic happened. Gradient descent worked perfectly.

import numpy as np

a, b = np.array([0.0, 0.0]), np.array([4.0, 2.0])   # task-A / task-B minima
L = lambda w, c: 0.5 * np.sum((w - c) ** 2)

w, lr = a.copy(), 0.3                     # start at the pretrained weights
for t in range(11):
    print(f"step {t:2d}  w = ({w[0]:5.2f}, {w[1]:5.2f})   "
          f"L_B = {L(w, b):6.3f}   L_A = {L(w, a):6.3f}")
    w -= lr * (w - b)         # ∇L_B only — task A is invisible to the update

The fixes, in the same toy:

• Replay, 50/50. Change one line — w -= lr * (0.5*(w - b) + 0.5*(w - a)) — and the walk converges to the midpoint $(2, 1)$, where both losses are 2.5. Neither task is perfect; neither is destroyed. The mix ratio is the knob that slides the endpoint between the two minima.
• LoRA-style leash. Freeze the base at $(0,0)$, train an adapter $\Delta$ on $L_B$ with its size capped at $\lVert\Delta\rVert \le 1.5$. The walk stops at $(1.34, 0.67)$: task-A loss held at 1.13 while task-B loss improved 10 → 4.42. And because the base never moved, deleting the adapter restores task-A loss 0.000 exactly — full fine-tuning has no undo button, because it overwrote the weights in place.

One honest caveat, because the toy exaggerates one thing: with two rigid bowls you cannot have both losses near zero — the tasks genuinely conflict at this scale. A billion-parameter network is wildly overparameterized, so task B has enormous families of low-loss solutions, and some sit close to the pretrained weights. That’s why leashed methods in practice give up only a little task-B quality, not the chunk this cartoon shows. What transfers exactly: the optimizer descends only the loss you hand it, and distance walked is the price old tasks pay.

• General competence collapses outside the fine-tune domain. The medical fine-tune aces MedQA and loses summarization, casual conversation, coding. Symptom in prod: “the model got worse at everything we didn’t train it on” — filed weeks after launch, because nobody evaluated anything else.
• Format bleeding. Fine-tune on JSON extraction and the model starts answering casual questions in JSON. The narrow task’s output distribution leaks over the entire input space, because the weights that decided “what register do I answer in” were repurposed.
• Out-of-distribution prompts fall off a cliff. Inputs that look nothing like the fine-tuning data land in regions of behavior the update never controlled but still disturbed — performance there isn’t just lower, it’s erratic.
• Reasoning and knowledge benchmarks quietly regress. Narrow fine-tunes routinely shave points off MMLU-style general suites. This is the “alignment tax” InstructGPT measured after RLHF — real enough that OpenAI shipped the replay fix (PPO-ptx) in the same paper.
• Embedding geometry shifts under downstream systems. Fine-tuning moves the embedding space itself, so similarity structure the base model had — the geometry your retrieval or clustering pipeline was built against — silently deforms. Nothing errors; rankings just get worse.
• Your eval can’t see any of this. The fine-tune eval measures task B by construction. Forgetting lives in its complement. If you don’t run a held-out general suite against the base model as a control, the first detector of catastrophic forgetting is your users.