When developers turn to neural networks for help with refactoring or debugging, a dilemma often arises: should you trust a single top-tier model, test several independent alternatives simultaneously, or task another LLM with merging the outputs of its peers into a unified codebase? To test these approaches in practice, a two-stage experiment was conducted on a real-world, buggy Bash script.
Anatomy of the Test Script
The subject of the test was a utility script named run-code.sh, designed for batch testing LLMs via an external Python script and generating a summary table. The code intentionally contained seven defects of varying severity:
- Corrupted preset data: missing delimiters and empty rayrate parameters causing warnings in
stderr. - Lack of defensive parsing: the assumption that input data would always contain valid integers.
- Missing pre-execution file check: the script launched the benchmark loop even if the executable file was missing.
- Critical status accounting failure: the summary table looked for non-existent keys, causing statistics to always output zeros regardless of test results.
- False crash reporting: the script failed to differentiate between a fatal crash and a scenario where a model passed only a portion of the tests.
- Fragile JSON parsing: a complete crash of the summary generation if even a single report was corrupted.
- Outdated documentation: obsolete filenames mentioned in comments and help text.
Round 1: Blind Solo Repair
In the first stage, four models—Sonnet 5, HY3, Qwen3-Max, and DeepSeek-V4-Flash—received the source file without prompts or hints and worked in complete isolation.
| Bug / Issue | Sonnet 5 | HY3 | Qwen3-Max | DeepSeek-V4-Flash |
|---|---|---|---|---|
| 1. Preset errors | Fixed point-by-point | Missed | Fixed via guard | Neutralized via defaults |
| 2. Limit parser guard | Not required | Missed | Full protection | Partial protection |
| 3. File existence check | Missed | Fixed | Missed | Missed |
| 4. Correct status keys | Missed | Full categorization | Detailed breakdown | Partial (pass/fail only) |
| 5. Crash vs. degradation split | Missed | Fixed | Strict validation | Missed |
| 6. JSON parser protection | Full try/except | Partial | Partial | Partial |
| 7. Documentation updates | Partial | Missed | Missed | Missed |
The primary takeaway from the first round: not a single model resolved all bugs on its own. Sonnet 5 prioritized code cleanliness and exception handling, but overlooked the critical statistics collection bug. HY3 brilliantly rewrote the accounting logic and was the only model to notice the missing file check, but ignored the corrupted preset data. Qwen3-Max delivered the most balanced output, closing the majority of functional gaps.
Round 2: Merging Third-Party Solutions
In the second stage, seven models (including the first-round participants, along with Mistral-Medium-3.5, Nemotron-3-Super-120B, Gemini Pro, and dots-studio-3-note) were provided with all four sets of fixes and tasked with assembling the best possible final version.
| Aggregator Model | Critical Bugs (1–5) | Parser Protection (6) | Doc Cleanup (7) | Source Attribution | Final Result |
|---|---|---|---|---|---|
| Qwen3-Max | Addressed | Yes | Yes | Full changelog table | 1st place |
| Gemini Pro | Addressed | Yes | Yes | None | 2nd place |
| DeepSeek-V4-Flash | Addressed | Yes | Yes | Partial | 3rd place |
| Mistral-Medium-3.5 | Addressed | Yes | Lost | None | 4th–5th place |
| dots-studio-3-note | Addressed | Yes | Lost | None | 4th–5th place |
| ling-3.0-flash | Addressed | No | Lost | None | 6th place |
| Nemotron-3-Super-120B | Addressed | No | Lost | None | 7th place |
All aggregator models produced functional code that passed basic execution. However, differences emerged in their attention to detail: most models quietly stripped out less obvious fixes (such as Sonnet 5's documentation updates). The clear winner was Qwen3-Max, which not only retained all useful changes but also attached a detailed changelog attributing the source of each modification.
Key Takeaways
- No universal model exists: even flagship solutions have blind spots, and each neural network has its own.
- Rankings do not guarantee completeness: underdogs from the first round caught unique bugs that the round winner completely overlooked.
- Merging requires verification: when combining code, AI models tend to discard changes whose importance they fail to grasp.
- Transparency matters: a code aggregator is valuable not just for delivering working code, but also for explaining the provenance of every single line.