Does LLM Self-Correction Work? What 7B Code Repair Shows

The 7B retry ladder improved on one attempt. The blind-retry comparison was inconclusive about the added value of failure feedback.

Blind retries passed 143 of 164 HumanEval+ tasks and the feedback ladder passed 144 (exact McNemar p = 1.0). The blind run recorded 23% fewer generated tokens per task.

In a separate benchmark on 34 tasks that exhausted the stock ladder, gpt-6-astra passed 84 of 102 attempts and a local 27B model passed 63.

Corrected November 11, 2026. We narrowed the feedback conclusion, clarified the GPU-energy comparison and corrected the scope of the fallback benchmark.

Does LLM self-correction work?

It depends on whether the model has something outside itself to check against. Without external feedback, self-correction doesn't reliably improve reasoning, and it can make reasoning worse: Huang et al. (arXiv 2310.01798) found LLMs struggle to fix their own reasoning mistakes when they're the only judge. Olausson et al. (arXiv 2306.09896) tested the code case directly: self-repair helps only when the feedback is good, and the gains shrink once you count the extra samples it costs to generate that feedback. Kamoi et al. (arXiv 2406.01297), surveying the field, conclude that reliable self-correction needs an external verifier: tests, a compiler, something the model isn't grading itself. Arimbur (arXiv 2604.10508) shows what that verifier buys: with tests in place, iterative repair adds 4.9 to 17.1 points on HumanEval across seven models, and most of that lands in the first two rounds.

In our benchmark, visible tests decide when to retry. Hidden tests score the result after the loop ends.

Should a coding agent feed the error back or just retry?

We compared Qwen2.5-Coder-7B-Instruct runs on one AMD Radeon AI PRO R9700. Both configurations had the same call budget. Blind retries received the original prompt without the failure text. Against the feedback reference, blind retries gained four tasks and lost five (143 vs 144 of 164, two-sided exact McNemar p = 1.0). The frozen comparison was inconclusive about the added value of feedback. The three historical feedback-ladder runs scored 143, 144 and 142.

Verma (arXiv 2607.26117) studied how often small models repeat a failed solution under blind resampling and feedback-driven repair on MBPP+. The full ladder, training runs and tables are in our 7B retry-ladder measurement.

In the historical MBPP+ run on 343 tasks held out from ladder design, the ladder passed 268 versus 248 for a single attempt (20 gains, zero losses, exact McNemar p = 1.9e-06). Of the 27 tasks recovered on visible tests, 20 also passed the hidden tests. MBPP+ had no blind-retry arm.

The blind run recorded 182 generated tokens per task versus 238 for the feedback reference. Its GPU-energy average was 1,152 J/task, compared with 1,367 J/task across three separate feedback sessions. Those idle-subtracted GPU measurements cover their recorded task windows. The configurations shared a call budget, while the work and energy sessions differed.

How does the retry harness know a task failed?

The code runs against the task's visible tests in a sandboxed subprocess with a timeout. A failed assertion, an exception or a timeout triggers a retry. The hidden HumanEval+ tests only run after the loop finishes, for scoring; the harness never sees them. A blind retry uses the exact same check to decide whether to retry again, its prompt just leaves the error out. The step-by-step version, with the pre-check and tier budgets, is in the retry-ladder post.

How many retries before escalating?

In the historical HumanEval+ runs, the ladder gained six to eight tasks over a single attempt. Recorded stock averages were 1,367 J/task for the ladder and 728 J/task for one attempt, within the GPU-only meter window. This comparison leaves the best retry cap for other workloads open.

Which fallback model should a coding agent escalate to?

The fallback benchmark covered the 34 tasks that exhausted the stock 7B ladder (10 HumanEval and 24 MBPP+). It completed 35 model or profile rows, each with three attempts per task scored on hidden tests.

Download the raw results: fallback-model bake-off (CSV) and blind vs. feedback retry data (CSV).

34 tasks × 3 attempts = 102 per model or profile, scored on hidden EvalPlus tests. Timeouts (300 s) and API errors count as failures. Hosted cost estimates use September 28, 2026 list prices. The local dollar estimate uses GPU energy at $0.15/kWh and excludes hardware and other system power. The CSV includes 14 additional completed rows and a 36th entry blocked by licensing.

The profiles that scored at least 65/102 were classified as reasoning profiles. That pattern is specific to this cohort and these settings. The historical Holm-adjusted analysis covered nine comparisons against gpt-6-astra. It established neither a difference within that set nor equivalence.

The OpenAI rows recorded about 4 to 6 seconds per attempt. Workers AI reasoning rows recorded about 23 to 97 seconds. The local Qwen 3.8 27B recorded 63/102 passing attempts, at least one pass on 24/34 tasks and a majority pass on 21/34. The hosted profile of the same name scored 45/102, with 50 errors or timeouts. The records do not isolate the effect of hosting or establish identical weights.

The local Qwen run averaged about 9,820 J per call, calculated by dividing full-run idle-subtracted GPU energy by 102 calls. The table reports historical pass counts, latency and cost for these profiles on this residual cohort.

The rig runs in your VPC, or your engineers run it and we interpret the output.

What hasn't been tested yet?

These were single-function HumanEval and MBPP tasks, with one primary 7B model. Repository-level performance, the effect of failure feedback at other model sizes and fallback performance on other residual cohorts remain open.

Frequently Asked Questions

Does self-repair help small code models?

The historical stock ladder passed 144/164 HumanEval+ tasks versus 136/164 on the first attempt. The separate blind-retry comparison was inconclusive about the added value of feedback.

Is LLM self-correction worth the extra tokens?

The historical ladder gained six to eight HumanEval+ passes over one attempt. The blind run recorded 182 generated tokens per task versus 238 for the feedback reference, with 143 versus 144 passes.

Which model should a coding agent escalate to when it's stuck?

On 34 tasks that exhausted this ladder, gpt-6-astra passed 84 of 102 attempts, gpt-6-luna 70, and local Qwen 3.8 27B 63. The benchmark reports results for that cohort.

Does self-correction use more energy?

Recorded GPU-only averages were 1,367 J per task across three feedback sessions and 728 J for one attempt. The separate blind run averaged 1,152 J per task. These averages are specific to the recorded sessions and meter windows.

How many retries before escalating to a bigger model?

The historical results report what the configured ladder recovered. The best retry cap and escalation point for other workloads remain open.

Methods

• Model. Qwen2.5-Coder-7B-Instruct, one AMD Radeon AI PRO R9700 (32GB).
• Benchmarks. HumanEval+ (164 tasks) and MBPP+ (343 tasks, held out from ladder design), scored on EvalPlus hidden tests.
• Blind control. Same call budget as the feedback ladder; retries get the original prompt and no failure text.
• Energy. amdgpu hwmon power1_average sampled at 5 Hz, idle power (16 to 17 W) subtracted, GPU-only boundary.
• Fallback bake-off. The 34 tasks the stock ladder exhausted (10 HumanEval, 24 MBPP+), routed by visible-test failure only, 3 independent attempts per model, each a single call with no feedback.
• Stats. Exact McNemar test on paired outcomes. The historical fallback analysis applied Holm correction to nine comparisons.
• Data. Full per-model and per-seed results: fallback-models.csv, blind-vs-feedback.csv.