How Modern LLMs Learn New Skills Without Forgetting Old Ones

The architecture behind most frontier models in 2026 is 'mixture-of-experts' (MoE). The idea is straightforward:

• Instead of one enormous neural network that processes every query, the model has many smaller expert networks - each trained to be good at a different kind of task.
• When you ask a question, a 'router' decides which experts are relevant and only activates those.
• For a coding question, the coding experts activate. For a medical question, the medical experts activate. For chit-chat, a smaller, faster expert handles it.

This is why GPT-5 and Claude 4.7 can be much bigger than GPT-4 in total parameter count without being correspondingly slower or more expensive: most of the model is asleep on any given query.

The practical advantage isn't speed (though MoE helps there). It's that you can train a NEW expert separately and plug it in, without having to retrain everything else.

AllenAI's published work in 2026 demonstrates this clearly with their open-source models: they trained domain experts (legal reasoning, scientific paper comprehension, multilingual translation) as independent training runs, then combined them into a single MoE model. The combined model gained the new capabilities without measurably losing existing skills.

For frontier proprietary models (OpenAI, Anthropic, Google), the same architectural principle is at work even though the specifics are kept private. The pace of capability addition has accelerated in 2026 partly because of this - new tool-use, new reasoning modes, new language coverage land as point releases rather than waiting for the next big model version.

A few practical consequences for a normal AI user:

• Capability improvements happen continuously. The era of 'wait 12 months for GPT-5' is over. Model providers ship new specialist abilities as point releases - improved coding in a Tuesday update, better long-document handling the following week. Worth keeping an eye on release notes if you care about a specific use case.
• The 'best model for X' question gets sharper. Claude is genuinely better at long-context reasoning + coding. Gemini is better at retrieving structured information from Google's ecosystem. GPT-5 is better at agentic multi-step tasks. These aren't marketing positioning - they reflect which domain experts each provider has invested most in training.
• Open-source models close the gap faster. Meta's Llama 4, Mistral's Pixtral 3, Alibaba's Qwen 3 are all MoE-based and benefit from the same point-release acceleration. The open-source frontier is roughly 6-12 months behind the proprietary one in 2026, vs. 18-24 months in 2024.

Training a frontier model from scratch is still extremely expensive (hundreds of millions of dollars in compute), and still gates the underlying intelligence level. What's changed is the post-training step - the additional training done after the base model is built. Post-training used to be a single monolithic run; in 2026 it's increasingly modular, parallel, and additive.

For everyone NOT running a frontier lab, the practical implication is that you don't need to retrain a model to make it good at YOUR domain - you can fine-tune a small adapter (LoRA, QLoRA) on your data + slot it on top of an open-source MoE base. The cost has dropped to where small teams can do this for under £500 in compute.

Everything above still assumes someone hands the model something to learn from: curated questions, worked answers, a stronger 'teacher' model, or at minimum a set of human-written problems. A paper from Google researchers and Ulm University, published on arXiv on 23 October 2026, tests what happens when you take nearly all of that away.

The only human input is a short list of broad topic labels, such as 'Abstract Algebra' or 'Commonsense Causal Reasoning'. From each label, the model writes its own exam question, its own answer, and its own reasoning for that answer. It then filters those, trains on the survivors, and repeats. On-Policy Self-Training (OPST) is the researchers' name for this loop: a model improving itself on training data it wrote, with no human-written questions or answers.

A few terms first, because the method is built from them:

• Post-training - the training a model gets after its base version is built, which is where it learns to follow instructions and reason through problems.
• Distillation - training one model (the 'student') to copy the word-by-word predictions of another (the 'teacher'), rather than just copying a final answer.
• On-policy - the student learns from answers it wrote itself, not from text someone else wrote. That matches how it actually behaves when you use it, and earlier research found it causes less forgetting of old skills.
• Privileged information - extra context the teacher sees and the student doesn't, such as a worked solution.

The trick is that the student and the teacher are the same model, given different information. Each round runs in four steps:

1. Write the curriculum. Given a topic label, the model poses a question and writes a candidate answer plus the reasoning behind it.
2. Check the question is sound. The model re-solves its own question in a fresh context, without seeing the answer it wrote. It then rewords the question and solves it again. If either attempt gives a different answer, the question is thrown out. A question whose answer changes when you rephrase it is usually ambiguous.
3. Keep only what's worth learning. The student attempts each surviving question several times. If it already gets the answer every time, there's nothing to learn, so that question goes too. What's left sits just past what the student can reliably do - and because this check runs against the current student, the questions get harder as it improves.
4. Train with an information gap. The student sees only the question. The teacher sees the question plus the self-written answer and reasoning, and the student is trained to match the teacher's predictions word by word along the student's own attempt.

Step four is the important design choice. The model isn't told 'this answer is correct, copy it'. The generated answer is a hint for the teacher, who can then fill in reasoning steps the original answer skipped. In the paper's comparison on identical self-generated data, this beat both plain fine-tuning on the generated answers and a reinforcement learning approach (GRPO) that rewarded matching them.

The teacher isn't a frozen copy, either. It's an exponential moving average (EMA) of the student - a running blend of the student's weights over many recent training steps - so one bad batch can't swing the targets. After each round, the improved model writes a fresh set of questions, duplicates of earlier rounds are removed, and the loop runs again.

The experiments used Google's small open-weight Gemma 4 models (the E2B and E4B sizes), scored on standard benchmarks: MMLU (broad academic knowledge), GPQA (hard science questions), GSM8K (school maths word problems) and HellaSwag (commonsense sentence completion). The headline numbers from the paper:

With thinking mode switched on, OPST roughly matched a version of the same method trained on human-curated questions and verified answers, at both model sizes. The same loop also worked on images: given unlabelled photos, the model wrote its own captions and questions about them and improved at visual question answering.

Good human-written training data is finite and expensive. Epoch AI's projection puts the point where frontier training runs use up the stock of public human-written text somewhere between 2026 and 2032. And the curated, carefully labelled sets that post-training depends on are costlier still: the OPST paper notes that labelling the steps of a single maths dataset took around 800,000 individual human judgements.

That's why labs have leaned on synthetic data, but most self-improvement methods still start from a human-written problem set that has to be assembled afresh for every new subject. OPST's claim is narrower and more interesting: the problem set itself can come from the model, steered only by a topic name. If that holds up at scale, adding a new area of competence starts to look less like a data-collection project and more like writing a list of topics - which fits the modular, add-a-skill direction the rest of this article describes.

The obvious worry is a feedback loop: a model teaching itself its own mistakes. There are two related failure modes to know about.

• Self-reinforcing errors. If the model writes a wrong answer, checks it against itself and agrees, the error is now training data - and the next round's model believes it more firmly. The OPST authors say plainly that their checks measure self-consistency, not correctness, so errors the model makes consistently still get through.
• Model collapse. A 2023 study, The Curse of Recursion, showed that models trained repeatedly on generated content lose the rare 'tail' of what they originally knew, drifting towards bland, narrow output. Its authors named this model collapse and argued it affects every kind of learned generative model.

The paper's defences are built into the loop rather than bolted on. The two-step question check rejects ambiguous or unstable questions. Training with the answer as a hint, not a target, limits how much a wrong answer can steer the student. The slow-moving EMA teacher stops one noisy batch from snowballing. And every claim is measured on independent human-written benchmarks the model never saw during training, plus three 'out-of-domain' tests no topic label targeted - ARC-Challenge, MATH-500 and Winogrande - to catch skills quietly decaying.

Those out-of-domain results are the honest part of the story. Training on several topic mixes at once mostly held or improved the untargeted scores, but MATH-500 fell by 4.9 points when all three topic sets were combined, and by 8.5 points in one pairing. Gains also flattened after about seven rounds rather than climbing forever.

The authors list the remaining limits themselves: the method can only amplify knowledge the model already holds, the topic labels were chosen by hand, and every experiment used small models. Self-training is a way to draw out what a model half-knows, not a source of new facts about the world.

Both approaches so far change a model in a lab, before it ships. Once a model is deployed it stops learning: anything you tell it lives only in the current conversation, and it is gone when the chat ends. Its knowledge is frozen at a cutoff date.

The sleep framework is a continual-learning method from Google Research (Google's in-house research division) that gives a deployed model a daily rhythm, much like ours. On 27 October 2026 Google Research described the approach on its blog, building on a paper titled Language Models Need Sleep by Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni (researchers at Google, Cornell University and the University of Southern California).

The analogy is deliberate. When you sleep, your brain replays the day and files the parts worth keeping into long-term memory. The sleep framework splits a model's life the same way:

• Awake. The model works normally - answering questions, reading documents - and holds what it picks up in fast, temporary memory, the equivalent of short-term recall.
• Asleep. The model stops taking new input and spends its compute on housekeeping: moving the fragile, recent knowledge into its permanent weights, then practising on material it generates itself.

The result is a model whose knowledge keeps growing after release, without a lab retraining it and without stuffing everything back into the prompt. The researchers stress that the brain vocabulary is a metaphor: the model's architecture and its training rules stay fixed and human-designed.

Catastrophic forgetting happens because new learning overwrites the same weights that store old skills. The sleep framework sidesteps that by giving new knowledge somewhere new to live. A sleep cycle has two stages.

Stage one: memory consolidation. This is the equivalent of deep, non-dreaming sleep.

1. Grow, don't overwrite. The model reserves spare mixture-of-experts capacity in advance. During sleep it switches on new, small (low-rank) experts and attaches them to its slow-changing, long-term layers. New facts go into the new experts, so the older weights barely move. This is the same add-a-module logic described earlier in this article, now happening after deployment.
2. Copy short-term into long-term. The researchers call this step knowledge seeding. The version of the model that still holds the day's material in its temporary memory acts as teacher; the expanded model is the student. The student writes its own attempts and is corrected word by word against the teacher (on-policy distillation, the same idea behind the self-training loop above). It also earns a reward for completing the teacher's text closely in meaning and in exact wording.
3. Clear the scratchpad. Once the knowledge is safely in the long-term layers, the temporary memory is reset, ready for the next waking session. The paper calls this synaptic pruning.

Stage two: dreaming. This mirrors REM sleep, when the brain rehearses and recombines. The model generates its own practice material, but with a twist: its router deliberately activates experts it would not normally pick, forcing unusual combinations of knowledge. It then keeps only the generated samples most likely to improve it (judged by their effect on training, not by eye), fine-tunes on the best ones, and checks whether downstream performance actually rose.

The design answers forgetting in three ways at once: new capacity instead of overwriting, training on the model's own behaviour rather than on foreign text, and a test after each round of self-practice.

The headline figure is that a sleep-trained model retains long-context knowledge up to 10 million tokens (a token is roughly three-quarters of an English word, so around 7.5 million words). It is easy to misread.

It does not mean the model has a 10-million-token context window you can paste a library into. It means that after reading that much material across sessions, and consolidating it during sleep, the model could still answer questions that depend on it. The knowledge was compressed into the weights rather than kept in the prompt.

The test was BABILong, a benchmark that hides the facts needed to answer a question inside very long stretches of filler text. On it, the researchers report near-perfect scaling all the way to 10M tokens, ahead of GPT-4 and of Titans, an earlier Google long-term-memory architecture. BABILong measures whether scattered facts survive and can be reasoned over; it says less about fuzzier long-document work such as summarising a contract.

The other reported results:

These are the authors' own numbers, published in October 2026 and not yet independently replicated. Manchu (a near-extinct language of north-east China) and Kalamang (a Papuan language with a few hundred speakers) were chosen because a model is unlikely to have seen them in pre-training, which makes forgetting easy to measure. Treat the results as a strong research signal, not a feature you can use today.

There are four main ways to keep a model current. Each trades something different:

• Fine-tuning. Train the model again on new data. The knowledge goes into the weights, but it tends to overwrite older skills, and it needs a training run, a team and data preparation each time.
• Retrieval (RAG). Retrieval-augmented generation looks up relevant documents at question time and pastes them into the prompt. It is cheap and easy to keep fresh, but the model never actually learns anything; answer quality depends on the search finding the right page.
• Long context. Give the model everything at once in a very large prompt. It works for one session, but costs grow with every token, quality tends to slip in the middle of very long inputs, and it all disappears when the session ends.
• Sleep. Let the model learn during use, then consolidate offline. The knowledge ends up in the weights like fine-tuning, without the forgetting, and arrives continuously like RAG, without a search step. The costs are compute spent during sleep, a model that grows as it adds experts, and a method that so far exists only in research.

In practice these will be combined rather than swapped. A product could use retrieval for today's news, long context for the document in front of you, and sleep-style consolidation to turn what keeps coming up into lasting knowledge.

You don't need to know how mixture-of-experts works to use AI well. But it helps explain a few things you'll notice in 2026: why models keep getting better in specific narrow ways without big version-number jumps, why open-source models keep closing the gap, why providers can afford to ship faster improvements without periodic 'we trained a new model' announcements.

The architecture has moved from 'one big network' to 'a collection of specialised modules orchestrated by a router'. Each module can improve independently. That's the engineering shift that makes the 2026 AI experience feel like continuous improvement rather than discrete release cycles.

Self-training on self-written questions is the next step along the same road: if a model can set its own curriculum from a list of topics, the bottleneck shifts from collecting data to checking that the model's improvements are real. For now, that check is the hard part, and it's where to look whenever a lab claims a model taught itself.

Sleep-style consolidation goes further still: a model that keeps learning after it ships, files what it learned into new experts overnight, and practises on its own material without losing old skills. If that works outside the lab, the knowledge-cutoff date that every chatbot carries today could become a thing of the past.