Qwen2.5-Coder-7B Hits 87.8% HumanEval+ on One 32GB GPU

Measure the retry loop first. In these runs, the Cycle 2 coreset used fewer tokens and less time.

In one recorded run, stock Qwen2.5-Coder-7B-Instruct scored 144/164 (87.8%) on HumanEval+ with a retry ladder; three fixed-seed stock-ladder runs scored 143, 144, and 142. This is a benchmark result for that setup. Its comparison with the published 32B single-attempt score is method-asymmetric: the 7B system retries against visible tests. A separate 132-sample Cycle 2 run scored 141/164 versus 144/164 (paired p = 0.58), with 17.3% fewer recorded tokens per task and 20.4% less recorded wall time. This did not test or establish accuracy noninferiority/equivalence. A separate clean MBPP-only candidate scored 153/164 Base and 146/164 Plus; it is not the Cycle 4 diagnostic candidate.

What Loupe did, in order: built the ladder, let an automated loop discover its rules on 35 MBPP tasks and froze them before HumanEval, curated training sets around tasks the model got wrong, and evaluated several configurations. The figures below come from multiple runs and candidates; their scope is stated alongside each result. The Qwen2.5-Coder Technical Report supplies the published baseline (88.4% Base, 84.1% HumanEval+, single-attempt greedy).

Loupe by LeanLM (not affiliated with Google's LearnLM educational AI) measures what AI workloads cost to serve. The tokens, elapsed times, and GPU energy reported here are measurements for the specific runs and measurement windows described below.

What Loupe found

What is a cost ladder for coding models?

A test-feedback loop that spends nothing extra until something fails, then spends the least that might fix it. Each tier has a token budget. Tier 0 is greedy; Tiers 2 and 3 sample. On HumanEval the stock model passed 143 tasks at Tier 0, the ladder fixed 6 at Tier 1, 4 at Tier 2 and 1 at Tier 3, and 10 went to Tier 4.

How the retry harness decides to retry

1. Generate: the model writes the function, one greedy attempt.
2. Pre-check: a sub-millisecond CPU check catches syntax errors and missing imports.
3. Detect: the code runs against the task's visible tests (HumanEval's Base tests) in a sandboxed subprocess with a timeout. All pass: done, no more tokens spent. A failed assertion, an exception or a timeout is a failure, and the harness keeps the failing assertion and traceback.
4. Tier 1, patch: the failing line and error go back; the model patches it (one greedy attempt). Back to step 3.
5. Tier 2, rethink: the failure goes back with an instruction to use a different algorithm; 3 candidates at temperature 0.6, each tested, first pass wins.
6. Tier 3, reason first: all errors so far go back; the model works out edge cases before coding; 3 samples at temperature 0.7, capped at 500 tokens.
7. Tier 4, escalate: the task goes once to a fallback model.

The hidden HumanEval+ tests run only after the loop finishes, for scoring; the harness never sees them.

The blind variant runs the same steps and the tests still decide when to retry; its retry prompts just leave out the error.

Token budgets for Tiers 1 and 2 are approximate per-attempt figures; Tier 3 is a hard cap. Latency was measured per task, not per tier.

The ladder fixes execution mistakes: off-by-one errors, wrong output formatting, a failed assertion the model can see. The 10 tasks that reached Tier 4 needed capability the 7B lacked, which is what the wingman is for.

How Loupe found the retry rules

An automated loop selected them. It ran up to 15 repair steps per task on 35 MBPP discovery tasks, with stall detection and an AST-cycle detector: when the model's successive patches parse to the same tree, the loop forces a paradigm-shift resample at temperature 0.4 instead of letting it spin. The selected rules (traceback diff, input/output contrast, paradigm shift plus perturbation) became the frozen ladder above. The stock HumanEval result uses a separate benchmark suite; later Cycle 3 and 4 diagnostic candidates used synthetic primitives targeting earlier HumanEval failures and are not held-out confirmation.

On those 35 tasks, the loop helped at every size: 0.5B from 37.1% to 48.6% (13 to 17 of 35), Qwen3.5-4B from 42.9% to 62.9% (15 to 22), 7B from 71.4% to 77.1% (25 to 27). The 0.5B and 4B runs used the discovery loop itself; the 7B used the frozen four-tier ladder.

Fine-tune or build the loop? The table.

In the recorded stock-weight run, the ladder scored 154/164 Base and 144/164 Plus, versus 143/164 and 136/164 for first attempts. The separate preregistered blind-resampling comparison scored 143/164 against a 144/164 reference and was inconclusive about the added value of failure feedback. These observations do not establish that feedback caused the ladder gain.

HumanEval, all 164 tasks, none used in training or in ladder design:

Legend: Base = HumanEval original tests; the ladder retries against these. Plus = HumanEval+, the EvalPlus hidden tests, never shown to the ladder; this is the blind column. Attempt 1 = passed Base on the first generation. s/task = wall-clock per task from start to resolution, generation plus sandbox test execution, batch size 1. The published row is single-attempt greedy pass@1 from the Qwen2.5-Coder Technical Report; every "+ ladder" row is a retrying system, so a "vs published" delta on those rows compares retries against one attempt. Counts are of 164. One run per configuration. The Cycle 2 result (141 vs 144 Plus, paired p = 0.58) is an observed comparison; nonsignificance does not establish noninferiority or equivalence.

In these runs, Cycle 2 scored 141 vs 144 Plus tasks, with fewer recorded tokens and elapsed time; p = 0.58 does not establish accuracy equivalence. Boundary-only scored 138 Plus tasks and recorded 62 Tier 1 fixes. The flooded run scored 126 Plus tasks after 41.78 minutes of training. These are configuration-specific observations.

The weights-only view agrees. On the 35 MBPP tasks held out from training, with the same ladder: stock 77.1% (27 of 35); the repair-and-resample SFT model (RRSI) 82.9% (29 of 35), paired won 3 and lost 1, p = 0.63; coreset Cycles 2, 3 and 4 each 68.6% (24 of 35).

Train only on what the model gets wrong

292 of the 343 MBPP training tasks (85.1%) were solved on the first attempt before any training, so most of a conventional fine-tuning set teaches the model what it already knows. Flooding 1,804 synthetic samples across all of them cost 18 tasks on Plus (144 to 126, p = 0.001) and 42 minutes of GPU.

Loupe's boundary coreset keeps only the tasks the model fails, each formatted as a traceback repair, each run and passed in a sandbox before it's kept. 102 samples, 2.39 minutes. That run shows what training actually does to the model: it learns to repair.

Optimize the model and the loop together

These records compare several weight configurations under the same ladder. The boundary-only run scored 29.3% (48 tasks) on attempt 1, recorded 6 to 62 Tier 1 repairs, and finished at 93.9% Base (154); stock + ladder also scored 93.9% Base (154). Plus was 138 vs 144 (p = 0.21). These observations describe the runs and do not establish why tier use changed.

The Cycle 2 configuration mixed 51 repairs, 51 docstring completions and 30 tasks the model already solved (132 samples, 3.08 minutes). Attempt 1 scored 73.8% (121). In this run, recorded tokens per task were 17.3% lower (197.0 vs 238.1) and wall-clock time 20.4% lower (4.50 vs 5.65 s), both p < 1e-5. Plus scored 141 vs 144 (5 vs 8 discordant, p = 0.58); this is not a noninferiority/equivalence result.

In the Cycle 2 comparison, recorded wall time and tokens per task were lower, while the Plus score was 141 versus 144. The nonsignificant score difference does not establish equal accuracy or guarantee capacity gains on another workload. A different change cut joules per request 67% on one workload (results page).

Escalate to a bigger local model

Cycles 3 and 4 added synthetic "boundary primitives" written to target earlier HumanEval failures, so these are diagnostic runs rather than held-out results. The separate clean MBPP-only T5 run scored 153/164 Base and 146/164 Plus; Cycle 4 scored 157/164 Base and 146/164 Plus. Similar Plus totals do not establish equivalence. Cycle 4 recorded 193.3 tokens per task and 79.3% on attempt 1. The Wingman values below are post-hoc diagnostic joins, not an authenticated live route.

The clean run used a 152-example LoRA coreset built from stock-model failures on MBPP training tasks, without HumanEval tasks or failures. It scored 153/164 Base and 146/164 Plus. A same-seed stock-ladder summary records 143 Plus tasks (p = 0.58); that nonsignificant comparison does not establish equivalence. The clean run's recorded energy was invalid, and it has no Wingman join. Its recorded first-attempt score, tokens, and elapsed time describe this candidate and run only; the clean run is described here, not included in the Cycle 4 reconstruction table below.

These are post-hoc joins of archived model outputs against Cycle 4 residual task IDs (each model covered 6/7); they are not authenticated live routing results and do not apply to clean T5. The reconstructed Qwen3.8 join adds Plus passes on HumanEval/126 and /129; its Base gain IDs are /126, /129, /132, /163 and /32. Qwen3-Coder adds a Plus pass on /126; its Base gain IDs are /126 and /32. The separate stock-ladder Tier 4 pool is the 10-task list in Methods and is not this Cycle 4 residual pool. The separate 34-task, three-attempt fallback run recorded 63/102 passing attempts; its 9,820 J is full-run idle-subtracted GPU energy divided by 102 calls, not a per-call measurement.

Loupe calls this slot the wingman; most teams call it a fallback model. Loupe tried the frontier version too: the 10 HumanEval tasks the stock ladder exhausted went once each to gpt-6-luna, which passed 7 of 10 on HumanEval+ using 7,744 tokens in total.

That led to a full fallback-model bake-off. Loupe sent the 34 tasks the stock ladder exhausted (10 HumanEval, 24 MBPP+) to each candidate, 3 attempts per model, and counted hidden-test passes out of 102:

Legend: 34 exhausted tasks (10 HumanEval, 24 MBPP+), 3 attempts per model, hidden-test passes out of 102. Prices are list prices checked September 27, 2026; the local figure is electricity only.

36 fallback models were tested in all. Every model at 65 of 102 or above reasons before answering, and at n = 34 no model differs significantly from gpt-6-astra.

Where this sits in the research

Self-repair is well established. Arimbur (arXiv 2604.10508, April 2026) measures +4.9 to +17.1 pp on HumanEval across seven models from an 8B Llama to Gemini 2.5 Pro with up to five attempts. So is data pruning: Tsai et al. (arXiv 2407.05040) keep benchmark accuracy with 10% of the synthetic training data, and Lv et al. (arXiv 2504.12687) beat the full OSS-Instruct set with 40% of it. Loupe's historical records report cost alongside accuracy for specific runs. A July 2026 paper (Verma, arXiv 2607.26117) studies blind resampling and feedback-driven repair on MBPP+. In Loupe's separate 164-task blind-resampling comparison, the control scored 143/164 against a 144/164 reference ladder (exact McNemar p = 1.0); the preregistered result was inconclusive about feedback's added value, not evidence of no effect. The recorded token and energy figures are specific to those runs and meter windows. The full writeup on blind retries versus feedback repair, plus the 36-model fallback leaderboard, is a separate post.

What this changes if you self-host a coding model

The whole run fits on one 32GB card, and every training run above finished in under 42 minutes, most in under 4. That's the cost profile of self-hosting an LLM at a size where utilization decides the bill more than model price does. The Tier 4 wingman is LLM model routing applied to the residual: send only the escalated tasks to the bigger model. Coreset runs of 2 to 4 minutes are cheap enough that LoRA fine-tuning stops being a project and becomes a knob you turn per retry loop. The tier budgets exist because tokens are the resource; reasoning token costs explains why the tokens you don't see are the ones to watch. All of it sits under the same rule as enterprise LLM cost optimization: measure first, then move the smallest thing that clears the bar.

Frequently Asked Questions

Can a 7B coding model match a 32B one?

One recorded stock Qwen2.5-Coder-7B-Instruct retry-ladder run passed 144 of 164 HumanEval+ problems (87.8%). This is numerically near the published 87.2% single-attempt result for its 32B sibling, but the methods differ: the 7B system retries against visible tests. HumanEval tasks are single functions with tests attached; a run on a real repository, with its own build and fixtures, is the next measurement.

Does fine-tuning a coding model beat a good retry loop?

In one stock-weight run, the ladder scored 144/164 HumanEval+ versus 136/164 for first attempts. The Cycle 2 coreset run scored 141/164 versus 144/164 (p = 0.58) and used 17.3% fewer tokens per task. This did not test or establish accuracy noninferiority/equivalence, and the comparison does not identify the cause of the ladder gain.

Is a retry ladder with test feedback better than just sampling again?

A historical 164-task blind-resampling comparison scored 143/164 against a 144/164 reference ladder (p = 1.0). Its preregistered outcome was inconclusive about feedback's added value; a nonsignificant result does not establish no effect. Token and GPU-energy observations apply to the specific runs and measurement windows.

How does a retry harness know when a coding model failed?

The harness runs the code against the task's visible tests in a sandboxed subprocess with a timeout. A failed assertion, an exception or a timeout counts as a failure, and the harness keeps the failing assertion and traceback for the next retry. The hidden HumanEval+ tests only run afterward, to score the result; the harness never sees them.

What does it mean to optimize the model and the retry loop together?

In the recorded runs, a repair-only coreset shifted fixes into Tier 1 (6 to 62 tasks) and scored 93.9% Base. The 132-sample Cycle 2 run scored 141/164 Plus versus 144/164, with lower recorded tokens per task (17.3%) and wall-clock time (20.4%). No accuracy noninferiority/equivalence was established.

Why is the blind (Plus) score the one that matters?

HumanEval+ is the suite the ladder never sees, so it measures how the code holds up on tests you didn't write. The ladder retries against HumanEval's Base tests the way a coding agent retries against your own suite. Loupe reports both columns and headlines the blind one.

How much GPU time does a boundary coreset need?

2.39 to 3.08 minutes on one AMD Radeon AI PRO R9700 (32GB) for 102 to 132 samples: about 14× fewer samples and 3 minutes of training instead of 42, compared with flooding 1,804 samples, which also cost 18 tasks on the blind score.

Is Loupe a fine-tuning product or a retry-loop product?

Both, measured together. Loupe runs your workload on the smallest model and retry setup that clears your bar, on one stated server, and reports tokens and wall-clock time per task, plus energy per request where the rig can meter it, with the method attached.

Methods

• Rig. One AMD Radeon AI PRO R9700 (32GB). Stock attempt 1 reproduces Qwen's published score within 2 tasks (143 vs 145 Base, 136 vs 138 HumanEval+).
• Runs. Stock + ladder repeated at three fixed seeds scored 143, 144 and 142 of 164 on HumanEval+ (mean 143 ± 1); every run beat the published single-attempt 138. Other configurations are one run each; Tiers 2 and 3 sample without a fixed seed. s/task is wall-clock from task start to resolution, generation plus sandbox tests, batch size 1; per-tier latency and wingman tokens weren't recorded (— in the tables).
• Tasks. HumanEval and MBPP are self-contained functions with tests attached. The 35 MBPP tasks were separate from the model-training examples but served as the rule-discovery set, so they are not an independent evaluation cohort for discovery. Real-repository runs are next.
• Diagnostic runs. Cycle 3 and Cycle 4 candidates used synthetic training data written to target earlier HumanEval failures; they are diagnostic, not held-out confirmation. The Wingman table is a separate post-hoc join of archived model outputs to Cycle 4 residual IDs, not a trained candidate or authenticated live route.
• Energy. In the stock-run records, GPU power was measured with amdgpu hwmon (power1_average) at 5 Hz, GPU-only boundary, with idle power (16 to 17 W, measured over 60 s with the model loaded) subtracted. The task window does not include the final sandbox or Plus evaluation. The recorded stock + ladder value was 1,367 J/task against 728 J for a single attempt; this is a setup-specific estimate, and the 5 Hz sampling is not a per-inference energy trace.
• Contamination. Qwen2.5-Coder-7B-Instruct has most likely seen HumanEval in pretraining, as have the models on the published leaderboard; the deltas between rows carry the argument. On 343 MBPP+ tasks never used to design the ladder, the ladder passed 268 against 248 for a single attempt (20 gained, 0 lost, p = 1.9e-06), so the lift isn't HumanEval-specific.
• Escalated tasks. HumanEval 10, 32, 65, 115, 127, 129, 130, 132, 145 and 163 reached Tier 4 on the stock ladder; gpt-6-luna was separately sent those 10 stock-ladder tasks once each and passed 7/10 on HumanEval+. These are distinct from the Cycle 4 residual IDs and post-hoc local Wingman join in the diagnostic table.

Where Loupe fits

Loupe measures what a workload costs to serve and finds the smallest model and loop that clear your bar. One workload, fixed scope, fixed fee. Your data stays in your environment: the rig runs in your VPC, or your engineers run it and we interpret the output. You get the table, with n and the rule stated, whichever way the result lands. Scope a measurement, or see the measurements so far.