Moonshot put 2.8 trillion parameters into public circulation in July and almost nobody can run them. Holding the weights takes about 1.4 TB at 4-bit, which is sixteen GPUs across two nodes before you serve a single token. The file is free and the capability is not. Everything interesting about the next twelve months sits in the gap between those two facts, and there is exactly one process that closes it: distillation. It is how a model nobody can run becomes a model anyone can, and right now it is a bespoke research project for every team that attempts it.
We think that is the wrong shape for something this useful, and we are exploring what it looks like as a service. This post is the full reasoning — the mechanics down to the loss function, the economics with the arithmetic, the failure modes and where they come from, and why we are looking at this rather than claiming to have solved it.
Part 1: What distillation actually does
Knowledge distillation trains a small student to reproduce the behaviour of a large teacher. The phrase covers several quite different techniques, and the differences between them account for most of the variance in outcomes. Teams that get disappointing results have usually picked the cheapest variant without knowing there were others.
The three levels of supervision
Response distillation trains the student on text the teacher generated. Send prompts, keep completions, fine-tune the student on the pairs. It works against anything that returns text, including a closed API. It is also the weakest form, because each supervised token carries a single index: this word came next.
Logit distillation trains the student to match the teacher's full output distribution at each position. Instead of learning that the answer was "refund", the student learns the teacher put 0.61 on "refund", 0.28 on "chargeback", 0.04 on "return" and small mass across a long tail — the shape of the teacher's uncertainty, including which wrong answers were nearly right. This is what Hinton's original 2015 paper called dark knowledge, and it is the reason a distilled student typically beats a same-size model trained from scratch on identical data.
Feature distillation goes further and matches intermediate representations — hidden states, attention maps — between aligned teacher and student layers. The strongest signal, and the most constrained, since the architectures must be similar enough to map onto each other.
The practical consequence reframes July's releases. Open weights unlock the good forms of distillation. A closed API exposes text and, at best, truncated top-k logprobs. Weights on disk expose the full distribution, the hidden states, and the ability to run the teacher as many times as your compute allows rather than as many times as your invoice allows. Kimi K3 at 2.8T and Alibaba's forthcoming 2.4T Qwen 3.8 are not merely cheaper models to call. They are the first genuinely first-rate teachers that anyone is permitted, and technically able, to teach with.
The loss function, and why temperature matters
The classical objective mixes two terms — one against the ground-truth label if you have one, one against the teacher's softened distribution:
L = α · CE(student, y) + (1 − α) · T² · KL( softmax(zᵀ / T) ‖ softmax(zˢ / T) )
where zᵀ and zˢ are the teacher's and student's pre-softmax logits and T is a temperature.
Temperature is doing something specific and easy to get wrong. A well-trained teacher's output distribution is usually sharply peaked — the top token often carries 0.95 or more of the mass. At T = 1 the soft targets are nearly one-hot, so they tell the student almost nothing a hard label would not have. Dividing the logits by T flattens the distribution and makes the relative ordering and spacing of the non-argmax tokens visible, which is where the transferable structure lives. Push T too high and you flatten past the signal into the noise floor of the tail. Values between 2 and 5 are the usual working range, and it is worth tuning rather than inheriting.
The T² factor exists because softening shrinks the gradients from the KL term by roughly 1/T². Without the correction, raising temperature would silently down-weight the distillation objective relative to the cross-entropy term, and you would conclude that soft targets do not help when in fact you had turned them off.
The deeper reason logit distillation is efficient is about where gradient flows. Cross-entropy against a hard label pushes one logit up and the rest down more or less uniformly — the student receives supervision at a single index out of a vocabulary of a hundred thousand or more. KL against a full soft target shapes every logit toward a specific value. Per token of teacher inference, the student receives orders of magnitude more usable signal, which is why distillation converges on far fewer examples than training from scratch on the same corpus.
Forward KL or reverse KL: a decision most teams never make
Which direction you take the divergence in changes what kind of student you get, and this is worth deciding deliberately.
Forward KL — KL(teacher ‖ student) — is mass-covering. It penalises the student heavily for putting near-zero probability anywhere the teacher put mass. A capacity-limited student forced to cover every mode the teacher expresses ends up spreading itself thin, which shows up at generation time as occasional low-quality tokens: the student learned to keep a little probability on things it cannot actually do well.
Reverse KL — KL(student ‖ teacher) — is mode-seeking. It penalises the student for putting mass where the teacher did not, but tolerates the student ignoring modes entirely. The result is a student that concentrates on the subset of teacher behaviour it has the capacity to represent properly.
For a student that is one or two orders of magnitude smaller than its teacher, reverse KL usually produces better generation quality, because the honest situation is that the student cannot represent everything the teacher does and forcing it to try degrades everything else. Forward KL makes more sense when the capacity gap is small, or when coverage genuinely matters more than per-sample quality. Most default implementations use forward KL because it is the textbook form and it is what the naive cross-entropy-on-soft-targets code path gives you.
The part most coverage misses: distillation needs no labels
This is the single most useful property of the technique and it gets buried under the compression story.
Fine-tuning needs labelled data — inputs paired with outputs known to be correct. That is why most teams who should tune do not: they have millions of inputs and almost no ground truth, and producing ground truth means an annotation budget and a quarter of calendar time.
Distillation needs only the inputs. The teacher supplies the outputs. You are converting compute into labels, at machine speed, across your entire traffic history.
Nearly every organisation is sitting on exactly the asset this consumes: years of unlabelled queries, tickets, documents, transcripts and events. Distillation is what turns that pile into a training set without a labelling vendor. It is the difference between "six months and an annotation contract" and "a weekend of teacher inference."
This is also why distillation and fine-tuning are complements rather than substitutes. Distillation transfers general capability from a bigger model into a smaller one. Fine-tuning shapes a model to your conventions using examples you know are right. The strong pattern is both: distil to get a small model that can do the task at all, then tune on curated examples so it does the task your way. The second step is covered in how to fine-tune an LLM on your own data.
Part 2: Why distilled models fail as agents, and how to fix it
This section is the one we would most want a team to read before spending money, because it explains a failure that is usually misdiagnosed as an inherent limit of small models.
The symptom
A distilled student benchmarks two or three points behind its teacher on standard evaluations and then performs dramatically worse in an agentic workflow. The team concludes small models cannot do agentic work and abandons the project.
Part of that is real and unavoidable: errors compound multiplicatively across a trajectory. If per-step reliability falls from 97% to 93%, then across a twenty-step tool-using sequence where every step must be right, 0.97²⁰ ≈ 0.54 against 0.93²⁰ ≈ 0.23. A four-point per-step gap becomes a thirty-one-point gap in task completion. That arithmetic is the same one that governs quantization damage, and it applies to any per-token degradation whatever its source.
But a large part of it is not inherent. It is an artefact of how the distillation was run.
The cause: training on the teacher's states, deploying on the student's
Classical token-level distillation is teacher-forced. You take sequences the teacher produced (or ground-truth sequences), feed the prefix to the student, and match distributions position by position. Every state the student is trained on is a state the teacher visits.
At inference, the student generates its own prefix. It makes a small mistake at step three. It is now in a state the teacher would never have entered, and which therefore appeared nowhere in training. It has no idea how to recover, because recovery from its own characteristic errors was never demonstrated. The next mistake comes faster, and the trajectory unravels.
This is exposure bias, and it is a distribution mismatch between training and deployment. It is almost invisible on single-turn benchmarks — one step, no accumulated drift — and it is devastating on long horizons. Which is precisely the pattern teams observe and misattribute to capacity.
The fix: on-policy distillation
The correction is to train the student on its own state distribution. Sample sequences from the student, then have the teacher score those sequences — providing, at each position the student actually reached, the distribution the teacher would have produced there. The student learns what to do in its own error states, including how to get back on track.
This family of methods travels under names like generalised knowledge distillation and on-policy distillation. The trade-off is cost and complexity: instead of pre-generating a static corpus once, you need teacher inference in the training loop, scoring fresh student samples every so often. That is a materially harder pipeline to build and operate than "generate 100k completions, run a fine-tune."
It is also the difference between a student that works as an agent and one that does not, which is a reasonable summary of why we think this belongs in a service rather than in a script each team writes once. The cheap version of distillation is easy and produces a model that benchmarks fine and fails in production. The version that works is an operational commitment.
Reasoning distillation and rejection sampling
A related technique matters wherever correctness is checkable.
Rather than distilling only final answers, distil the reasoning traces. Sample k candidate traces per prompt from the teacher, verify the final answer automatically — unit tests for code, a checker for maths, schema validation for structured extraction — and keep only the traces that reached a verified-correct answer. Train the student on those.
This converts an unverifiable text-generation problem into a verified-data problem for any domain with an automatic checker, and it substantially raises quality per dollar of teacher inference. You pay k× for sampling, but you discard the failures rather than teaching them, so the corpus you keep is cleaner than anything single-sample generation produces. For a teacher that solves a class of problem 60% of the time, sampling four traces and keeping the verified ones yields a training set on which the demonstrated behaviour is correct by construction.
The limitation is the checker. Where you cannot verify automatically — open-ended drafting, judgement calls, most support and sales work — you fall back on the teacher's unfiltered output and inherit its errors along with its skills.
Part 3: The economics, with the arithmetic
Distillation moved from research topic to obvious business decision because building a corpus became cheap while frontier serving stayed expensive.
Building the corpus
A realistic corpus: 100,000 prompts drawn from your own traffic, roughly 500 input and 800 output tokens each.
At K3's published rates — $3.00 per million cache-miss input, $15.00 per million output:
- Output: 100,000 × 800 = 80M tokens × $15.00 = $1,200
- Input: 100,000 × 500 = 50M tokens × $3.00 = $150, falling toward $15 if your prefixes are stable enough to hit cache
Roughly $1,350 for the raw corpus. Training a LoRA student on it is hours on a single GPU, in the low hundreds of dollars. Call the artefact $2,000 to $5,000 once you include the runs you throw away.
If you use rejection sampling at k = 4, multiply the teacher inference by four — about $5,400 — and you still have a corpus that costs less than a fortnight of one engineer.
Self-hosting the teacher instead
Above a volume threshold, renting GPUs beats paying per token, because the billing model changes shape. An API bills per token, so cost scales linearly with corpus size forever. Self-hosting bills per GPU-hour, so cost scales with time, and the marginal token approaches free once the cluster is up.
The crossover point depends entirely on the aggregate throughput you achieve on a heavily batched MoE, which varies enormously with your interconnect, batch size, and serving stack — and which we would not want to quote as a single figure, because a 2.8T mixture-of-experts with expert-parallel all-to-all traffic behaves very differently on NVLink than on Ethernet. The structural point is what matters for planning: one-off corpora of a few hundred thousand samples are cheaper on an API; continuous re-distillation at scale, or anything requiring in-the-loop teacher scoring for on-policy training, tips toward self-hosting. Work out your own crossover before committing to either.
What it displaces
From the worked example in the fine-tuning piece: 10 million requests a month at frontier rates costs about $105,000, and routing 75% of it to a small model brings that to about $30,300. A saving of roughly $75,000 a month against a build cost of a few thousand.
The payback period is about one day.
Numbers that lopsided usually mean the model is wrong, so here is what the arithmetic omits: engineering time, evaluation infrastructure, the serving stack, and the recurring cost of keeping the student current. Those dominate the total, and they are exactly the parts a service exists to absorb. The corpus was never the expensive component. Believing it was is the reason most distillation efforts produce one promising result and then stall.
The traffic ladder
The stack this produces is well established by now: a frontier teacher on the hardest ~5% of traffic, a distilled mid-size student on ~80%, and a small cached model on the trivially routable remainder. A 7B student served on vLLM cutting cost-per-request by an order of magnitude against its teacher is an ordinary result rather than an exceptional one.
The ladder also degrades gracefully. When the student is uncertain, the request escalates to the teacher rather than failing, so the worst case is the cost you were already paying.
Part 4: Compression works along the task axis
A conceptual point that determines whether a project succeeds before any code is written.
People reason about distillation as though it compresses capability — a 2.8T model squeezed into 7B, generally. Stated that way it is obviously implausible, and general-purpose distillations do lose a great deal.
What distillation actually compresses is behaviour on a distribution. A 7B student can match a 2.8T teacher on a narrow, well-specified slice of work to within a point or two, while being hopeless at everything outside that slice. The teacher's parameters are spread across an enormous range of tasks, languages and domains; the student's are spread across yours.
The operational rule that follows: the narrower you define the task, the higher the compression ratio you can achieve. A student distilled for "support ticket triage in our product taxonomy" can be very small. A student distilled for "be generally helpful" has to be large, because the target distribution is nearly as broad as the teacher's.
Teams that get poor results have usually defined the task too broadly, then blamed the student's parameter count. Splitting one general student into three narrow ones — each smaller than the general one would have needed to be — frequently outperforms it in aggregate and costs less to serve, especially when a router already exists to select between them.
Part 5: The five hard parts
If distillation were only "generate completions, train a student", everyone would have done it. The difficulty is distributed across five stages, and training is the easiest of them.
1. Choosing the inputs
Randomly sampling your traces spends most of the budget on cases the student would have handled anyway. What improves a student is coverage of the decision boundary — ambiguous cases, rare intents, inputs where the current model is least certain.
A workable scheme has two components in tension. Uncertainty sampling scores candidates by teacher entropy, or once a v1 student exists, by student-teacher disagreement, and samples in proportion. Stratified coverage clusters your traffic by intent or embedding and enforces a floor per cluster, so that rare-but-important categories survive. Uncertainty alone tends to collapse onto whichever pathological cluster produces the most disagreement, and you end up with a student that is excellent at one weird edge case and mediocre everywhere.
Getting selection right is routinely worth more than doubling corpus size, and it is the stage most teams skip entirely.
2. Running the teacher at scale
Hundreds of thousands of completions with batching, prefix caching, retries, deduplication, and a cost ceiling that actually holds. Straightforward engineering, tedious to get right, and easy to overspend on by an order of magnitude through inattention to prompt structure — recall that K3's cache-hit and cache-miss input rates differ tenfold.
3. Training the student
Base selection, objective (forward or reverse KL, or a mix), temperature, α, adapter rank, on-policy or off-policy, and the judgement to stop. Mostly settled practice and the least of the five problems, though the on-policy variants raise the engineering bar considerably.
4. Evaluating honestly
The hard one, and where most projects deceive themselves.
Compare against the incumbent, not a threshold. "89% accuracy" means nothing until you know the model currently in production scores 91%.
Evaluate per slice, not in aggregate. Aggregate parity routinely conceals a twenty-point collapse on a slice that is 5% of volume and 40% of revenue. Break results down by intent, input length, tool-call count, language, and customer segment.
Evaluate on the shape of work you run. If the workload is multi-step and tool-using, single-turn accuracy cannot see the failure that matters — that is the compounding arithmetic from Part 2. Trajectory-level evaluation is the only thing that detects it.
Keep a regression suite of the known-hard tail. Every failure you have previously fixed should be a test. Distillation is very good at silently reintroducing solved problems.
5. Deploying without betting the product
A student should enter production as a routed candidate on 5% of traffic with confidence-gated escalation and automatic rollback on a metric regression — never as a swap. This is what Swfte Connect already does for any model, and it is the part of the distillation problem we consider solved.
Part 6: Distillation is not a project
Here is the finding that most changed how we think about this, and the reason we are looking at a service rather than a tool.
A distilled student is a photograph of a teacher at a moment, taken against a particular distribution of inputs. Both of those move.
Teachers improve. The frontier moves roughly every eight weeks. A student distilled from a June teacher carries June's capability into December while the teacher it was cloned from has been superseded twice. Nothing alerts you, because the student's absolute scores do not fall. It quietly stops being competitive with what a fresh distillation would produce.
Your traffic drifts. New products, new segments, new failure modes, seasonal shifts. The student was fitted to the distribution you had.
Staleness is the dominant operational risk in production distillation and it is invisible by construction. A decayed model looks exactly like a healthy one until someone measures it against a current alternative.
Trigger-based re-distillation
The usual advice is a calendar cadence, which is either wasteful or too slow and is never right for long. Triggers are better, and all four are cheap to instrument:
Escalation rate. The share of traffic your router sends to the teacher because student confidence was low. A rising trend is drift, visible before quality complaints arrive.
Canary disagreement. Hold a fixed sample of a few thousand prompts. Weekly, run both student and current-best teacher and measure disagreement. Cheap, and it directly tracks the gap you care about.
Input distribution shift. Embed incoming traffic and measure distance from the distillation corpus. When the live distribution has moved materially away from what the student was fitted to, re-distil regardless of what the other metrics say.
A better teacher. When a new model clears your evaluation bar by a meaningful margin, the student's ceiling has risen and re-distillation will capture it.
Any one crossing threshold triggers a rebuild. That is a pipeline somebody has to own and keep running, and owning it is not most companies' comparative advantage.
Part 7: Where distillation fails
Five failure modes, stated properly, because a post that only argues for a technique is an advertisement.
The student cannot exceed its teacher. Distillation transfers capability; it does not create it. If the teacher is confidently wrong about your domain — and a general model frequently is, because it has never seen your domain — the student inherits that error with the teacher's confidence and none of its breadth to recover from it. Distillation faithfully amplifies whatever the teacher believes, including the wrong parts. Where you know the right answer, fine-tuning on your own labelled examples beats distilling someone else's guess.
Long-horizon reliability degrades first and hardest. Partly inherent compounding, and partly the off-policy artefact from Part 2. Diagnose which before concluding the model is too small.
Licensing is a real constraint, not a formality. Most closed-model terms of service prohibit using outputs to train competing models, and that prohibition is being actively enforced — Anthropic's February complaint alleged Moonshot distilled roughly 3.4 million Claude exchanges, and the White House OSTP Director has publicly repeated the distillation accusation against K3. Whatever the merits of that dispute, the lesson is unambiguous. Before any teacher enters a pipeline, check three things in its licence: whether it permits training other models on outputs; whether there are field-of-use or user-count thresholds (some community licences impose them); and whether derivatives may be redistributed commercially. Note that "open weights" and "open source" are not synonyms — several widely used open-weight licences carry restrictions that a genuine open-source licence would not. Apache 2.0 and MIT are the clean ground. Get this in front of legal before the model is in production rather than after, because it is a licensing question and not an engineering one.
Recursive synthetic training degrades. Training successive generations on the previous generation's outputs narrows diversity and compounds artefacts — the model-collapse dynamic. The mitigation is structural: anchor every generation to real inputs from live traffic and a current teacher, never to a prior student's outputs.
Sometimes it is simply the wrong tool. If a good open small model already nearly does your task, tune it directly — you will get there faster and cheaper than by standing up a teacher pipeline. Distillation earns its cost when the gap between what small models do natively and what you need is genuinely wide.
Part 8: The safest place to start
If we were advising a team that wanted the benefit without the risk, we would not start with a replacement student. We would start with a speculative decoding drafter.
Distil a very small draft model against the model you already serve. At inference the drafter proposes several tokens, the target model verifies them in one parallel forward pass, and any prefix the target agrees with is accepted. Verification is exact, so the output distribution is identical to unassisted decoding — this is a lossless speedup, not a quality trade.
The numbers from Bonsai's DSpark drafter give the shape: at draft depth k = 4 the accepted length reaches τ = 3.6, producing 143.8 tokens/s against 104.8 unassisted, a 1.37× speedup with byte-identical output distribution.
It is the ideal first project because the downside is bounded at zero quality risk, it exercises the whole pipeline — corpus, training, evaluation, deployment — and it builds the operational muscle you need before anything with a real quality trade-off goes near production traffic.
Part 9: Why we are exploring this
Two reasons this is a natural extension of what we already run rather than a new business we would be inventing.
The surrounding pieces exist. A distillation pipeline needs an input distribution, a way to evaluate candidates against an incumbent, and a way to deploy to a slice of traffic with rollback. Those are traces in Nexus, the evaluation harness in Studio, and routing in Connect. Distillation is the missing middle of a path that is otherwise complete, which means the marginal thing to build is the teacher-inference, input-selection and re-distillation-trigger layer, not the whole stack.
It follows from what we think the strategic picture is. We have argued that the moat is a tailored model rather than a general one, and distillation is the cheapest known route from "a frontier model can do this" to "a model I own and can afford to run can do this." A student distilled against your traffic, from a teacher anyone may use, is not something a competitor can download: the teacher is public, the input distribution is not.
It connects to Project Alexandria in a way worth being explicit about. General-purpose distillations we produce get published under Apache 2.0 with their failure profiles. Distillations built from a customer's traffic belong to that customer and are never published, never re-distilled from, and never used to improve anything anyone else touches. Those commitments are not in tension: general capability is infrastructure and belongs in the commons, the specific model is the customer's asset.
And it is the mechanism behind the thing we actually care about. A 2.8-trillion-parameter open model reaches almost nobody directly. It reaches people through its students — the 7B and 30B models distilled from it that run on a single GPU, a laptop, or a phone. Distillation is the transmission belt between a frontier only labs can operate and a capability anyone can hold. Making that belt cheaper and more routine is the most direct thing we could do about getting AI to people who will never rent a two-node cluster.
We are being deliberate about the word exploring. There is no product to sign up for. What exists is a conviction that this layer is missing, an unusually good moment to build it — because the teachers are finally both excellent and legally usable — and the surrounding infrastructure already running. If you are distilling models today, or looked at it and stopped, we would like to hear where it broke: get in touch.
The bottom line
Distillation converts a frontier nobody can operate into models people can actually run, and 2026 removed both of the things that used to block it. Open weights made first-rate teachers legally available and exposed the logits, hidden states and unlimited sampling that make distillation work properly rather than superficially. Cheap teacher inference made a corpus cost a few thousand dollars against savings measured in tens of thousands per month.
Three things separate the teams that get this right from the ones that produce a promising demo and stop. They distil on-policy, so the student learns to recover from its own errors instead of only ever seeing states its teacher visits. They define the task narrowly, because compression works along the task axis and not the capability axis. And they treat it as a maintained cadence with triggers rather than a project, because teachers improve, traffic drifts, and a stale student is indistinguishable from a healthy one until it is measured.
None of that is research. All of it is operations, which is precisely why it should be a service rather than something every team rediscovers privately and expensively. We think that layer should exist, and we are working out what it takes to build it well.
For the teacher that made this urgent, see Kimi K3's weights release; for what the students look like at the extreme, a 27B model in 3.9 GB.