The number that will travel from PrismML's Bonsai 27B release is 3.9 GB. The number that should travel is 66.03. That is what the 1-bit build scores on agentic and tool-calling work, against 80.00 for the full-precision model it was compressed from — a fourteen-point collapse in the one category most people want to run locally. The same build loses under four points on maths. Extreme quantization is not a volume knob. It is a set of trade-offs that land very unevenly, and the entire skill is knowing where they land hardest.
On July 14, PrismML released Bonsai 27B: two extreme low-bit builds of Qwen3.6-27B under Apache 2.0, with the full 262K context window intact and vision preserved. Not a new pretrain — a re-encoding. The architecture is unchanged. What changed is how each weight is stored, and the result is a 27B-class model that fits on a phone.
The two builds
Ternary stores each weight as one of three values (−1, 0, +1) at a true 1.71 bits per weight, producing a 5.9 GB file. It retains 94.6% of the FP16 model's average benchmark score.
Binary stores each weight as −1 or +1 at 1.125 bits per weight, producing 3.9 GB. It retains 89.5%.
The FP16 original is 16 bits per weight and 54 GB. So ternary is a 9.4× reduction, binary 14.2×.
The scheme is group-wise: each weight is stored as a code, and every group of 128 weights shares one FP16 scale, so the effective weight is the group scale multiplied by the code. The bit arithmetic follows directly. Ternary needs log₂(3) ≈ 1.585 bits per code, plus 16 bits of scale amortised over 128 weights (16/128 = 0.125), giving 1.71 bpw. Binary needs 1 bit plus the same 0.125 overhead, giving 1.125 bpw.
Group size is the tuning dial nobody discusses. Smaller groups track the local distribution of weights more closely and lose less information, but the scale overhead grows in inverse proportion — at a group size of 32 the overhead would be 0.5 bpw instead of 0.125, which would push binary to 1.5 bpw and erase a third of the size advantage. 128 is a chosen point on that curve, not a natural constant.
The compression covers the matrix-heavy layers — embeddings, attention projections, MLP projections, the LM head — leaving only normalisation and scale parameters at higher precision. Composition is roughly 24.8B language weights, a 0.46B vision tower stored separately at 4-bit using HQQ, and 2.5B across embeddings and the LM head.
The 262K context survives partly because roughly 75% of Qwen3.6-27B's attention is linear. That matters more than it sounds: linear attention layers carry a fixed-size recurrent state rather than a cache that grows with sequence length, so three quarters of the model contributes nothing to cache growth as context extends.
True bit-widths, and what the labels hide
Buried in PrismML's framing is a methodological point more broadly useful than the headline: advertised bit-widths in conventional quantized builds understate what they cost.
A GGUF build labelled Q4_K_XL averages 5.2 bits per weight in practice, landing at 17.6 GB. IQ2_XXS — the "2-bit" build — averages 2.8 bpw and 9.4 GB. The label describes the dominant tensor format, not the file, because mixed-precision schemes keep sensitive tensors (attention output projections, the LM head, embeddings) at much higher precision and the average drifts upward. Bonsai's 1.71 and 1.125 are true averages across the whole model.
That reframes every comparison. Bonsai's ternary build is not competing with a "2-bit" GGUF at similar size — at 5.9 GB it is 37% smaller than IQ2_XXS at 9.4 GB and scores 80.49 against 72.73. If you had filed "2-bit quantization" as known-bad, the correction is that you were probably evaluating something closer to 3-bit that still performed badly.
Baselines for the whole comparison: FP16 at 16.0 bpw, 54 GB, 85.07 average. Q4_K_XL at 5.2 bpw, 17.6 GB, 84.99 average — effectively lossless, which is why it remains the default recommendation for anyone with the memory. IQ2_XXS at 2.8 bpw, 9.4 GB, 72.73.
Where the damage lands
Per-category, FP16 → ternary → binary, from 15 benchmarks run in thinking mode through EvalScope with vLLM on H100s:
Maths: 95.33 → 93.40 → 91.66. Barely scratched.
Coding: 88.74 → 85.96 → 81.88. Ternary close, binary starting to hurt.
Knowledge and reasoning: 83.15 → 76.96 → 73.39. Six points gone at ternary.
Agentic and tool calling: 80.00 → 74.01 → 66.03. Fourteen points at binary.
Instruction following: 78.47 → 71.77 → 65.74. Nearly as bad.
Vision: 72.61 → 65.19 → 59.57. Large, from a lower base.
The ordering is not arbitrary, and the mechanism behind it is worth working through because it generalises to every quantized model you will ever evaluate.
Quantization introduces a small error in every weight, which becomes a small perturbation in every activation, which slightly shifts the output distribution at every token. On a task with a short path to a verifiable answer — arithmetic, a closed-form derivation — that perturbation usually gets corrected, because the model has strong signal pulling it back toward the right answer and few opportunities to drift.
On a long agentic trajectory the arithmetic is unforgiving. Suppose quantization drops per-step reliability from 97% to 93% — a four-point difference that looks minor in isolation. Over a twenty-step tool-using sequence where every step must be right:
0.97²⁰ ≈ 0.54 against 0.93²⁰ ≈ 0.23
The four-point per-step gap becomes a thirty-one-point gap in task completion. Nothing about that requires the model to be badly damaged; it requires only that errors compound multiplicatively, which is exactly what they do in any workflow where step n consumes the output of step n−1. This is why the agentic column falls hardest, and why the effect gets worse as your workflows get longer.
The uncomfortable consequence: the ranking of tasks by quantization tolerance is close to the inverse of the ranking of tasks people most want to run locally and privately. Everyone wants the offline, on-device model to be their agent. Agentic work degrades first and fastest.
Read the binary build in that light. It is a strong local assistant for summarisation, extraction, drafting, classification, maths, and single-turn reasoning, running in your pocket with no network. That is a new thing in the world. It is not an agent, and no amount of scaffolding turns 66.03 into 80.00.
The failure mode short benchmarks cannot see
There is a second finding here that applies well beyond Bonsai, and it is the single most useful thing in the release.
IQ2_XXS scores 88.93 on MMLU-Redux — healthy-looking, close to baseline. The same build scores 57.5 on AIME26 and 56.4 on LiveCodeBench. Gemma-4-31B's Q2_K_XL shows an identical pattern.
Short-form, single-shot, multiple-choice benchmarks do not detect the damage aggressive quantization does, because a multiple-choice question is one decision with four options and a strong prior. Damage that is invisible across one decision is fatal across two hundred. If your evaluation of a quantized model is "MMLU looks fine," your evaluation is structurally incapable of seeing the thing that will break in production.
This is why teams ship a 2-bit build that benchmarked acceptably and then watch their agent fall over in week two. Test quantized models on the shape of work you actually run. If that work is multi-step and tool-using, the evaluation has to be multi-step and tool-using.
Why the size reduction converts to speed
Throughput, generation tokens/s and prefill tokens/s:
- M5 Max, binary: 66.4 / 874
- M5 Pro, ternary: 26.2 / 393
- iPhone 17 Pro Max, binary: 11.0 / 111
- H100 (CUDA), binary: 104.8 / 2755
The gap between the two columns is the whole story of on-device inference. Autoregressive generation at batch size 1 does almost no arithmetic per token — it reads the entire weight set from memory to produce one token, then does it again. Throughput is therefore governed by:
tokens/s ≈ memory bandwidth ÷ bytes of weights read per token
That is a bandwidth problem, not a compute problem, and it is why cutting the model from 54 GB to 3.9 GB produces a near-proportional speedup on hardware where bandwidth binds. Prefill is different — it processes the whole prompt in parallel, saturates the matrix units, and is compute-bound, which is why the prefill numbers gain far less from compression.
The practical read: extreme quantization is transformative on laptops and phones, where bandwidth is scarce and compute is adequate, and comparatively unexciting on datacentre GPUs, where you are optimising a constraint that was not binding.
Memory, cache, and why 3.9 GB is the phone target
iOS caps a single app at roughly half of physical RAM, so a 12 GB iPhone yields about 6 GB usable. That single platform limit is why the 3.9 GB build is the phone target and the 5.9 GB one is not, regardless of quality.
The KV cache is the other half of the budget, and it is where long context gets expensive. Only 16 of 64 layers carry a growing full-attention cache, which works out to roughly 64 KiB per token at FP16. A full 262K window therefore costs about 17.2 GB of cache — four times the weights — dropping to about 4.3 GB with a 4-bit KV cache. Measured peak usage at 100K tokens with an FP16 cache: 11.6 GB for binary, 14.7 GB for ternary, and roughly 25.6 GB for the derived Q4_K_XL comparison.
That last figure is why "27B-class quality on a 24 GB card" works: ternary weights plus a 4-bit cache fits where Q4 weights plus an FP16 cache does not.
Cache quantization usually costs quality, and here it is measured rather than assumed. Against its own FP16-KV baseline on MATH-500, ternary Bonsai diverges by 0.0011 nats forward-KL against 0.0146 for Q4_K_XL — an order of magnitude more faithful. Forward-KL in nats measures how far the output distribution has moved from the uncompressed reference; 0.0011 nats is close enough to be inaudible in practice, while 0.0146 is where you start seeing occasional token choices the original would not have made.
The battery figure that will end up in every headline: the whitepaper reports 672 tokens per 1% of iPhone battery. Both builds ship on llama.cpp (CUDA and Metal) and MLX, with ternary the default in the demo repo. Tool calling uses the standard OpenAI-style tools array with results in choices[0].message.tool_calls, and thinking mode is on by default with an adjustable thinking_budget_tokens.
There is also a DSpark speculative-decoding drafter trained against the Bonsai target. On H100 at draft depth k=4, the binary build reaches an accepted length of τ=3.6, giving 143.8 tok/s — a 1.37× speedup. The mechanism is worth understanding because it is free quality: a small draft model proposes several tokens, the large model verifies them in a single parallel forward pass, and any prefix the target agrees with is accepted. Verification is exact, so outputs stay distribution-identical to unassisted decoding. It is disabled by default on Apple Silicon at batch size 1, where the draft model competes for the same scarce bandwidth it is trying to save.
The case against
If you have the VRAM, you do not want this. Q4_K_XL scores 84.99 against FP16's 85.07 — effectively lossless at 17.6 GB. If you can run that (roughly 24 GB and up), choosing ternary costs you 4.5 points to solve a problem you do not have. Bonsai's value is entirely a function of constraint: around 12 GB of VRAM, or a phone. Outside that constraint it is strictly the worse option.
Post-training compression is the weaker form of this. The literature fairly consistently finds that pretraining natively in low precision beats post-training quantization at extreme bit-widths; BitNet avoids collapse specifically by training that way from scratch. Bonsai re-encodes an already-trained model, so degradation concentrates wherever the base model's precision was doing the most work. The category table is that unevenness made visible.
Some claims remain vendor claims. PrismML asserts the binary build is the first 27B-class model to fit on a phone, presented without independent verification — the same caution we applied to Kimi K3's pre-release numbers. Early hands-on reports also flag behaviour the benchmarks miss: one review found the binary build produced fluent Japanese but fabricated a nonexistent word, while ternary produced less natural output that leaked English and Chinese terms yet avoided that fabrication. Retention percentages do not describe what a model feels like to use, and 89.5% retention is an average over categories that behave nothing like each other.
What this is actually evidence of
Two July releases point in opposite directions along one axis.
Kimi K3 pushed the open frontier to 2.8 trillion parameters — roughly 1.4 TB of weights at 4-bit, sixteen GPUs across two nodes, a model whose value reaches most people indirectly through hosted price floors and the students distilled from it. Bonsai went the other way: take a model that already exists, compress it until it fits in a pocket, change nothing about the architecture, release it under Apache 2.0.
Both are the same phenomenon from different ends. Capability is decoupling from the ability to pay for infrastructure. For most of this field's history, access to good AI was gated by capital — you rented it from whoever could afford to train and serve it, and the meter ran. A 27B-class model running offline at 672 tokens per percent of battery is a different arrangement: no bill, no rate limit, no provider, no telemetry leaving the device, and no way for anyone to withdraw it.
That is what broad access looks like in practice, and it looks far less like a chat product than like a file.
It also sharpens the strategic point. Bonsai is Apache 2.0 — every competitor you have can download the identical weights this afternoon. What none of them can download is a model shaped by your data, your edge cases, and your domain. As the general layer approaches free, the remaining value concentrates in the tailored layer above it, which is the argument in the moat was never the model.
What to do with this
Choose by constraint. At 24 GB or more, run Q4_K_XL — it is nearly lossless and Bonsai buys you nothing. At around 12 GB, ternary Bonsai is the strongest option at that size by a wide margin. On a phone, binary is currently the only serious answer, with the agentic caveat.
Split by task, not by device. The category table is a routing plan. Maths, extraction, summarisation, classification and single-turn reasoning survive compression well enough to run locally. Agentic sequences and strict instruction-following should stay on a larger model. Running both — local for the bulk, remote for the hard tail — captures most of the benefit, which is what Swfte Connect and the Embedded SDK do across local and hosted endpoints. Pattern in edge AI and on-device models.
Build evaluations from real traces. The single transferable lesson: aggressive quantization fails invisibly on short benchmarks and visibly in production, and the compounding arithmetic above explains why. Evaluate on trajectories, not questions.
The bottom line
Bonsai 27B puts a real model in 3.9 GB and a very good one in 5.9 GB, permissively licensed, with honest bit-width accounting that quietly indicts how the rest of the field labels its quantized builds.
The cost is roughly five points at ternary and eleven at binary on average, and those averages conceal that agentic and instruction-following work absorbs two to three times the damage maths does. Anyone deploying this behind tools should be reading 66.03, not 3.9 GB.
What survives the caveats is the direction. Capable AI no longer requires a datacentre, a subscription, or a network connection, and the floor drops fastest for the people with the least. Once intelligence is ambient and free, the only question left that belongs to you is what you point it at.
For the other end of the same month, Kimi K3's weights release; for the economics underneath both, the efficiency race.