"I'm 100% Sure": Why AI Self-Assessment Cannot Replace Unit Tests

The Blind Confidence of Algorithms

Attempting to delegate code validation to the neural network itself proved to be a flawed strategy. An experiment testing whether a language model could honestly admit uncertainty about its generated script using a special marker revealed a 100% tendency toward AI overconfidence: out of 179 Python problem solutions, not a single model chose an uncertain status even once, despite several programs containing gross errors.

Instead of manual code review or running automated tests, the hypothesis proposed a simple filter: if the model appends CONFIDENCE: SURE at the end of the response, the solution is accepted; if CONFIDENCE: UNSURE, it is sent for revision. In practice, however, such a filter would have let every bug slip into production, as the re-examination queue remained completely empty.

How the Testing Was Conducted

For the experiment, 20 practical problems based on the Python 3.9 standard library were prepared. Each prompt contained non-obvious edge cases (such as adjacent range merging rules, IPv4 character validation, or proper file size rounding). The solutions were evaluated against a hidden suite of 247 tests.

The prompt strictly specified that returning the SURE status was only permitted with complete confidence that the function would handle all edge cases and pass rigorous testing. Requests were sent via a third-party API gateway to three models (three generations per problem):

The overall error rate was a modest 2.2%, but the core issue lies elsewhere: all 4 defective solutions received the highest confidence score from the models.

Errors the Neural Networks Never Doubted

The solutions failed not on exotic edge cases, but on quite trivial operations:

• Index off-shift in date parsing. The glm-5.3-flash model wrote detailed comments about handling invalid formats, but then attempted to parse a YYYY-MM-DD string using slices that ignored hyphens: year = int(s[0:4]), month = int(s[4:6]), day = int(s[6:8]). Consequently, a standard date failed with a ValueError on the very first call.
• Collapsing relative paths. Tasked with preserving the traversal ../../a, the same model pushed the first .. onto a stack, but on the second one simply called pop() without checking the stack's contents. The double directory traversal canceled itself out, turning into a. This bug occurred in two out of three runs.
• Classic off-by-one. When formatting data sizes, the claude-haiku-5-5 model shifted the unit array index: given 1024 bytes, the function returned 1.0 MiB instead of 1.0 KiB, failing 11 out of 14 tests.

Limitations of the Experiment and Conclusions

Definitive conclusions regarding the superiority of specific architectures cannot be drawn from this sample: 20 tasks represent a modest dataset, and calls were routed through an aggregator with uncontrolled context (input token counts differed by an order of magnitude between models). Furthermore, results from deepseek-v4-flash were excluded due to frequent timeouts and truncated API responses.

Nevertheless, the main practical takeaway is clear: LLM self-reflection within a single prompt without a code execution environment does not work. Models tend to project false confidence even when making obvious syntactic and logical errors; therefore, external testing and running code in an isolated sandbox remain the only reliable means of validation.