Build the retry loop first. Tune the model for the loop to cut cost.
With Loupe's tiered retry ladder, stock Qwen2.5-Coder-7B-Instruct passes 87.8% of HumanEval+ (144 of 164) on one 32GB AMD card, at 238 tokens and 5.65 seconds per task. That matches the published single-attempt score of its own 32B sibling, Qwen2.5-Coder-32B-Instruct (87.2% on HumanEval+, EvalPlus leaderboard), with 4.5× fewer weights, on one 32GB card (ours retries against the visible tests; theirs is a single attempt). No fine-tune raised that blind score. A 132-sample coreset trained for the ladder in 3.08 minutes held it within noise (141 vs 144, p = 0.58) while cutting tokens per task 17.3% and wall-clock time 20.4%, both p < 1e-5. The retry loop carried the accuracy. Training moved the cost.
What Loupe did, in order: built the ladder, let an automated loop discover its rules on 35 MBPP tasks and froze them before HumanEval, curated tiny training sets around the tasks the model got wrong (102 to 132 samples), and then tuned the model for the retry loop instead of for the benchmark. Every number below comes from that run; the Qwen2.5-Coder Technical Report supplies the published baseline (88.4% Base, 84.1% HumanEval+, single-attempt greedy).
Loupe by LeanLM (not affiliated with Google's LearnLM educational AI) measures what AI workloads cost to serve. Tokens and seconds per task are the proxy here for GPU time and spend.
What Loupe found
The loop is the lever.
Same weights, no training: the ladder took the blind HumanEval+ score from 136 to 144 of 164. That matches the single-attempt score of its 32B sibling, Qwen2.5-Coder-32B-Instruct, with 4.5× fewer weights, while retrying against visible tests.
Loupe finds the rules automatically.
An automated discovery loop found the retry rules; nobody hand-wrote prompts. On the 35 MBPP tasks the rules were discovered on (held out from training), they lifted a 4B model from 15 to 22 of 35 and a 0.5B from 13 to 17. The syntax/import pre-check alone added zero tasks. The discovered retry rules did the work.
Train only on what the model gets wrong.
85% of the training tasks (292 of 343) were already solved before training. Loupe's Cycle 2 coreset, a small curated training set of 51 repairs, 51 docstring completions and 30 already-solved anchors, used 132 vs 1,804 samples (about 14× fewer) and 3 minutes of training instead of 42. Flooding the model with all 1,804 cost 18 tasks on HumanEval+ (p = 0.001).
Optimize the model and the loop together.
Training moves work between the weights and the loop on purpose. A repair-focused coreset shifted fixes into the cheap Tier 1 (6 to 62 tasks) with Base unchanged at 93.9%. Tuned for cost, the 132-sample coreset held blind accuracy (141 vs 144 of 164, within noise) while cutting tokens 17.3% and time per task 20.4% (p < 1e-5).
Escalate to a bigger local model.
Routing the leftovers to a local wingman, a second, larger local model (qwen3.8, 27B) that takes what the 7B can't finish, reached 90.2% on HumanEval+ and 98.8% on Base with zero cloud calls, in a diagnostic run whose training data targeted earlier failures.
How to read the numbers
Base is HumanEval's original unit tests. The ladder sees those and retries against them. Plus is HumanEval+, the much larger edge-case suite for the same 164 problems from EvalPlus, which the ladder never sees. Plus is the blind score. Attempt 1 is the share that passed Base on the first generation, before any retry. Plus is a superset of Base, so the production analogue is a coding agent retrying against your own suite, with Plus standing in for the tests you didn't write. Published leaderboard figures are single-attempt greedy pass@1.
What is a cost ladder for coding models?
A test-feedback loop that spends nothing extra until something fails, then spends the least that might fix it. Each tier has a token budget. Tier 0 is greedy; Tiers 2 and 3 sample. On HumanEval the stock model passed 143 tasks at Tier 0, the ladder fixed 6 at Tier 1, 4 at Tier 2 and 1 at Tier 3, and 10 went to Tier 4.
| Tier | What happens | Decoding | Token budget | HumanEval (stock) |
|---|---|---|---|---|
| 0 | First attempt, plus a sub-millisecond CPU check for syntax and missing imports before tests run | greedy (temperature 0) | 0 extra | 143 passed |
| 1 | Micro-diff: the failing assertion and traceback line go back; the model patches that line | greedy (temperature 0) | ~50 | +6 |
| 2 | The failure goes back with an instruction to use a different algorithm | 3 candidates, temperature 0.6, top_p 0.9 | ~150 | +4 |
| 3 | Scratchpad: the errors from previous attempts go back; the model works out edge cases before coding | 3 samples, temperature 0.7, top_p 0.9 | up to 500 | +1 |
| 4 | Local wingman: a larger local model takes what the 7B exhausted | separate model | — | 10 escalated |
Token budgets for Tiers 1 and 2 are approximate per-attempt figures; Tier 3 is a hard cap. Latency was measured per task, not per tier.
The ladder fixes execution mistakes: off-by-one errors, wrong output formatting, a failed assertion the model can see. The 10 tasks that reached Tier 4 needed capability the 7B lacked, which is what the wingman is for.
How Loupe found the retry rules
An automated loop picked them. It ran up to 15 repair steps per task on 35 MBPP tasks, with stall detection and an AST-cycle detector: when the model's successive patches parse to the same tree, the loop forces a paradigm-shift resample at temperature 0.4 instead of letting it spin. The rules that came out (traceback diff, input/output contrast, paradigm shift plus perturbation) became the frozen ladder above. HumanEval never touched the discovery process, so the HumanEval numbers are the clean test.
On those 35 tasks, the loop helped at every size: 0.5B from 37.1% to 48.6% (13 to 17 of 35), Qwen3.5-4B from 42.9% to 62.9% (15 to 22), 7B from 71.4% to 77.1% (25 to 27). The 0.5B and 4B runs used the discovery loop itself; the 7B used the frozen four-tier ladder.
Fine-tune or build the loop? The table.
Build the loop first. It's the entire held-out lift: same weights, +11 Base (143 to 154) and +8 Plus (136 to 144), which turns 2 tasks under the published score into 9 and 6 over it (retrying vs single-attempt). Then fine-tune for cost.
HumanEval, all 164 tasks, none used in training or in ladder design:
| Run | Plus (blind) | vs published | Samples | Train time | Attempt 1 | Base | vs published | s/task | Tokens/task |
|---|---|---|---|---|---|---|---|---|---|
| Published Qwen, single attempt | 84.1% (138) | — | — | — | — | 88.4% (145) | — | — | — |
| Stock, attempt 1, our rig | 82.9% (136) | −1.2 pp (−2) | 0 | — | 87.2% (143) | 87.2% (143) | −1.2 pp (−2) | — | — |
| Stock + ladder | 87.8% (144) | +3.7 pp (+6) | 0 | — | 87.2% (143) | 93.9% (154) | +5.5 pp (+9) | 5.65 | 238.1 |
| Flooded SFT + ladder | 76.8% (126) | −7.3 pp (−12) | 1,804 | 41.78 min | 69.5% (114) | 85.4% (140) | −3.0 pp (−5) | 7.21 | 308.7 |
| Boundary-only + ladder | 84.1% (138) | 0.0 pp (0) | 102 | 2.39 min | 29.3% (48) | 93.9% (154) | +5.5 pp (+9) | 5.74 | 293.4 |
| Cycle 2 coreset + ladder | 86.0% (141) | +1.9 pp (+3) | 132 | 3.08 min | 73.8% (121) | 93.3% (153) | +4.9 pp (+8) | 4.50 | 197.0 |
Legend: Base = HumanEval original tests; the ladder retries against these. Plus = HumanEval+, the EvalPlus hidden tests, never shown to the ladder; this is the blind column. Attempt 1 = passed Base on the first generation. s/task = wall-clock per task from start to resolution, generation plus sandbox test execution, batch size 1. The published row is single-attempt greedy pass@1 from the Qwen2.5-Coder Technical Report; every "+ ladder" row is a retrying system, so a "vs published" delta on those rows compares retries against one attempt. Counts are of 164. One run per configuration.
Read across the rows and the pattern is consistent. Cycle 2 sits 3 Plus tasks under stock + ladder (p = 0.58, within noise) at 17.3% fewer tokens and 20.4% less time. Boundary-only matches the published score exactly on Plus (138) while moving 62 fixes into Tier 1. Flooding is the one run that hurts: 12 Plus tasks under published after 42 minutes of GPU.
The weights-only view agrees. On the 35 MBPP tasks held out from training, with the same ladder: stock 77.1% (27 of 35); the repair-and-resample SFT model (RRSI) 82.9% (29 of 35), paired won 3 and lost 1, p = 0.63; coreset Cycles 2, 3 and 4 each 68.6% (24 of 35).
Train only on what the model gets wrong
292 of the 343 MBPP training tasks (85.1%) were solved on the first attempt before any training, so most of a conventional fine-tuning set teaches the model what it already knows. Flooding 1,804 synthetic samples across all of them cost 18 tasks on Plus (144 to 126, p = 0.001) and 42 minutes of GPU.
Loupe's boundary coreset keeps only the tasks the model fails, each formatted as a traceback repair, each run and passed in a sandbox before it's kept. 102 samples, 2.39 minutes. That run shows what training actually does to the model: it learns to repair.
Optimize the model and the loop together
Most teams tune weights for accuracy inside a fixed scaffold. Loupe tunes the weights and the loop together, for cost, because training shifts which tier does the work. The boundary-only model learned to lean on the ladder: attempt 1 fell to 29.3% (48 tasks), Tier 1 repairs rose from 6 to 62, and Base finished at 93.9% (154), unchanged from stock + ladder. Plus landed 6 tasks under stock + ladder (138 vs 144, p = 0.21). The loop always gave the model a second look, so the model stopped spending its first attempt on getting everything right.
That's the lever. Once you can move work between the weights and the retry loop on purpose, you can balance for cost. Cycle 2 mixes 51 repairs, 51 docstring completions and 30 tasks the model already solves (132 samples, 3.08 minutes). Attempt 1 came back to 73.8% (121). Tokens per task fell 17.3% (197.0 vs 238.1) and wall-clock time 20.4% (4.50 vs 5.65 s), both p < 1e-5. Plus finished 3 tasks under stock + ladder (141 vs 144; 5 vs 8 discordant, p = 0.58).
At the same blind accuracy within noise, each task costs a fifth less time and a sixth fewer tokens. On a saturated GPU that's capacity back without buying anything. A different change cut joules per request 67% on one workload (results page).
Escalate to a bigger local model
Cycles 3 and 4 added synthetic "boundary primitives" written to target the tasks earlier models failed on HumanEval, so they're diagnostic runs rather than held-out results. Cycle 4 reached 95.7% Base and 89.0% HumanEval+ at 193.3 tokens per task, with attempt 1 at 79.3%. Routing its leftovers to a local wingman, with no external API, pushed further.
| Run (diagnostic) | Base | vs published | Plus (blind) | vs published | s/task | Tokens/task |
|---|---|---|---|---|---|---|
| Cycle 4 + ladder | 95.7% (157) | +7.3 pp (+12) | 89.0% (146) | +4.9 pp (+8) | 4.78 | 193.3 |
| + qwen3-coder:30b wingman (30.5B MoE) | 97.0% (159) | +8.6 pp (+14) | 89.6% (147) | +5.5 pp (+9) | — | — |
| + qwen3.8 wingman (27.3B dense) | 98.8% (162) | +10.4 pp (+17) | 90.2% (148) | +6.1 pp (+10) | — | — |
qwen3.8 solved grid traversal (HumanEval/129) and, on Base tests, root finding (HumanEval/32); qwen3-coder:30b solved root finding but not grid traversal. Every call stayed on local hardware.
Where this sits in the research
Self-repair is well established. Arimbur et al. (arXiv 2604.10508, April 2026) measure +4.9 to +17.1 pp on HumanEval across seven models from an 8B Llama to Gemini 2.5 Pro with up to five attempts. So is data pruning: Tsai et al. (arXiv 2407.05040) keep benchmark accuracy with 10% of the synthetic training data, and Lv et al. (arXiv 2504.12687) beat the full OSS-Instruct set with 40% of it. Loupe's contribution beyond those: it measures cost per task alongside accuracy, and it tunes the model for the loop it runs in. A July 2026 paper (Verma et al., arXiv 2607.26117) finds blind resampling matches feedback-driven repair at 7B on MBPP+ at 2.5 to 5.5 times fewer tokens; a matched-budget blind-resampling control is the next Loupe run.
What this changes if you self-host a coding model
The whole run fits on one 32GB card, and every training run above finished in under 42 minutes, most in under 4. That's the cost profile of self-hosting an LLM at a size where utilization decides the bill more than model price does. The Tier 4 wingman is LLM model routing applied to the residual: send only the escalated tasks to the bigger model. Coreset runs of 2 to 4 minutes are cheap enough that LoRA fine-tuning stops being a project and becomes a knob you turn per retry loop. The tier budgets exist because tokens are the resource; reasoning token costs explains why the tokens you don't see are the ones to watch. All of it sits under the same rule as enterprise LLM cost optimization: measure first, then move the smallest thing that clears the bar.
Frequently Asked Questions
Can a 7B coding model match a 32B one?
On HumanEval+, stock Qwen2.5-Coder-7B-Instruct with Loupe's retry ladder passed 144 of 164 problems (87.8%), matching the published single-attempt 87.2% of its own 32B sibling, Qwen2.5-Coder-32B-Instruct, with 4.5× fewer weights on one 32GB card (ours retries against the visible tests; theirs is a single attempt). HumanEval tasks are single functions with tests attached; a run on a real repository, with its own build and fixtures, is the next measurement.
Does fine-tuning a coding model beat a good retry loop?
The loop is the bigger lever: same weights, +8 tasks on HumanEval+ with no training. Fine-tuning's job is cost. Training moved first-attempt work into the cheap Tier 1 repair (6 to 62 fixes), and the 132-sample coreset tuned that way held the blind score within noise (141 vs 144, p = 0.58) at 17.3% fewer tokens per task. That's why Loupe tunes for the loop rather than for the benchmark.
Is a retry ladder with test feedback better than just sampling again?
Loupe's ladder added 8 HumanEval+ tasks over a single attempt at 238 tokens per task, with every retry conditioned on the failing test. Verma et al. (arXiv 2607.26117) report that at 7B blind resampling ties feedback-driven repair on MBPP+ at 2.5 to 5.5 times fewer tokens; a matched-budget blind-resampling control is the next Loupe run.
What does it mean to optimize the model and the retry loop together?
Training changes which tier of the retry ladder does the work. A repair-only coreset shifted fixes into Tier 1 (6 to 62 tasks) while Base held at 93.9%. A balanced 132-sample coreset kept blind accuracy within noise and cut tokens per task 17.3% and wall-clock time 20.4%, both p < 1e-5.
Why is the blind (Plus) score the one that matters?
HumanEval+ is the suite the ladder never sees, so it measures how the code holds up on tests you didn't write. The ladder retries against HumanEval's Base tests the way a coding agent retries against your own suite. Loupe reports both columns and headlines the blind one.
How much GPU time does a boundary coreset need?
2.39 to 3.08 minutes on one AMD Radeon AI PRO R9700 (32GB) for 102 to 132 samples: about 14× fewer samples and 3 minutes of training instead of 42, compared with flooding 1,804 samples, which also cost 18 tasks on the blind score.
Is Loupe a fine-tuning product or a retry-loop product?
Both, measured together. Loupe runs your workload on the smallest model and retry setup that clears your bar, on one stated server, and reports tokens and wall-clock time per task, plus energy per request where the rig can meter it, with the method attached.
Methods
- Rig. One AMD Radeon AI PRO R9700 (32GB). Stock attempt 1 reproduces Qwen's published score within 2 tasks (143 vs 145 Base, 136 vs 138 HumanEval+).
- Runs. One per configuration; Tiers 2 and 3 sample without a fixed seed. Task-count gaps of 2 to 3 were within noise. s/task is wall-clock from task start to resolution, generation plus sandbox tests, batch size 1; per-tier latency and wingman tokens weren't recorded (— in the tables).
- Tasks. HumanEval and MBPP are self-contained functions with tests attached. The 35 MBPP tasks were held out from training and also served as the rule-discovery set. Real-repository runs are next.
- Diagnostic runs. Cycles 3 and 4 and the wingman rows trained on data written to target earlier HumanEval failures; they're reported below the held-out table and labeled.
- Energy. Joules weren't metered on this run; tokens and wall-clock per task stand in for GPU time.
- Contamination. Qwen2.5-Coder-7B-Instruct has most likely seen HumanEval in pretraining, as have the models on the published leaderboard; the deltas between rows carry the argument.
- Escalated tasks. HumanEval 10, 32, 65, 115, 127, 129, 130, 132, 145, 163 reached Tier 4 on the stock ladder; the wingman solved 129 and, on Base, 32.
Where Loupe fits
Loupe measures what a workload costs to serve and finds the smallest model and loop that clear your bar. One workload, fixed scope, fixed fee. Your data stays in your environment: the rig runs in your VPC, or your engineers run it and we interpret the output. You get the table, with n and the rule stated, whichever way the result lands. Scope a measurement, or see the measurements so far.