Two public workloads, three serving configurations, one reference server. Every figure is model-serving energy at batch size 1, first-party, and not yet reproduced by anyone outside this project.
Version 1.0 · 17 September 2026 · next refresh by 17 October 2026
The measurements
| Workload | Model | Output constraint | Accuracy | Energy / request |
|---|---|---|---|---|
| Banking77 | 27B | label grammar | 93.9% | 284.5 J |
| Banking77 | 2B | JSON schema | 91.5% | 73.1 J |
| Banking77 | 2B | label grammar | 92.0% | 24.3 J |
| ContractNLI | 27B | — | 78.5% | 649.1 J |
| ContractNLI | 2B | — | 84.7% | 9.0 J |
Banking77 is 3,072 cases, ContractNLI is 1,763. The spread on Banking77 is the useful part: the same 2B model, on the same task, moved from 73.1 J to 24.3 J when the output constraint changed from a JSON schema to a restricted label grammar. Two thirds of the remaining energy was going into generating punctuation.
Reading the accuracy column
Both benchmarks have published task-specific baselines that score higher than either model above, at a fraction of the size. Banking77 was introduced in a 2020 paper whose whole argument was that small sentence encoders beat fine-tuning BERT-Large on intent detection. ContractNLI’s origin paper reports Span NLI BERT-base at 110M parameters reaching 83.8%, and a 335M variant reaching 87.5%.
So the honest reading of the accuracy column is not that a small generative model kept up with a large one. It is that both generative models were the wrong tool for these two tasks, and the energy column is what that costs. Running a general-purpose model where a specialised one would do is a measurable expense, and it is exactly the kind of thing a measurement turns up and an accuracy leaderboard does not.
Published baselines above are task-fine-tuned encoders with span-level heads. Our runs use a different configuration, so these are not like-for-like comparisons. We cite them because they bound what is achievable on these tasks, not because they refute our numbers.
Disclosure
Model serving only. Retrieval, embedding and orchestration are outside the boundary and excluded rather than estimated.
Batch size 1. The format gap has not been measured under production batching, where per-token masking overhead may amortise.
One reference server. A replay from AMD onto a rented NVIDIA GPU matched 298 of 300 items; the two mismatches are reported, not rounded away.
One family, two sizes. Nothing here establishes that the pattern holds across families, and we don’t claim it does.
Not disclosed in this version. It will be named in the next refresh rather than described vaguely.
Not disclosed in this version. Stating the field and leaving it empty is more useful than filling it with an approximation.
Controls
A third workload produced a result our pre-registered screening rejected on a data defect. We withdrew it rather than publish it. That stage exists so a bad run doesn’t become a claim, and it’s why this page has two workloads rather than three.
Across 2,400 cases and eight public tasks, reasoning solved 5% more cases than not reasoning. On a 300-case banking test it cost 51 times the energy for a 2.6-point accuracy gain. Worth having for hard cases, expensive as a default.
The JSON-schema against label-grammar comparison is the one worth running on your own stack. It needs no vendor and no data sharing, and an engineer can do it in an afternoon. If it holds for you, the question worth asking is which of your other workloads it applies to.
Follow the measurements
Each refresh adds workloads, fills in a disclosure field we couldn’t fill before, or corrects something. One email when that happens, nothing else.
No spam. Unsubscribe in one click.