A New Perspective on LLM Efficiency in Applied Automation
Restricting a language model's freedom of action can transform basic neural networks into an effective tool for business. This is the conclusion reached by developers of the VibePilot AI agent platform after testing popular LLMs across real document generation scenarios, including cost estimates, contracts, multi-page Excel sheets, and web pages.
Instead of abstract tests on competition mathematics or algorithm writing, the benchmark evaluated practical utility: time required to generate a correct file, total generation cost, and critical failure rate.
Why Classical Agents Fail More Often
In typical open-ended agent loops, the neural network autonomously decides on an action plan and invokes tools at each step. With a 90% success probability for an individual step, a five-step process finishes successfully only 59% of the time, explaining the 12–37% failure rate seen across the industry.
VibePilot engineers implemented a different approach: a deterministic pipeline:
- For spreadsheets and documents: a three-stage pipeline (input validation, document body generation, structure check).
- For web development: a four-stage cycle (specification analysis, tech stack selection, layout generation, validator check).
Under this system, the model is relieved from handling meta-tasks and focuses exclusively on content generation within a specific step. As a result, the failure rate was kept at 1.8% across more than a thousand real tasks.
Summary Results of Three-Month Testing
A total of 1,198 applied user tasks were processed during the tests. The benchmark included both direct APIs and models deployed on cloud infrastructure.
| Model | Provider | Tasks | Failures | Average Time | Average Tokens | Task Cost |
|---|---|---|---|---|---|---|
| DeepSeek V4 | DeepSeek API | 18 | 0.0% | 2 min 27 sec | 1.9K | 1.32 ₽ |
| DeepSeek V3.2 | DeepSeek API | 44 | 0.0% | 3 min 15 sec | 1.4K | 0.13 ₽ |
| YandexGPT 5 Pro | Yandex Cloud | 322 | 1.2% | 2 min 52 sec | 3.6K | 4.43 ₽ |
| Qwen3-235B | Yandex Cloud | 210 | 1.9% | 4 min 25 sec | 3.4K | 2.22 ₽ |
| DeepSeek V4 | Yandex Cloud | 273 | 2.2% | 4 min 09 sec | 3.5K | 2.47 ₽ |
| Alice AI LLM | Yandex Cloud | 331 | 2.4% | 2 min 39 sec | 3.0K | 2.60 ₽ |
Head-to-Head Results on Real Tasks
Real-time testing demonstrated the behavioral specifics of various architectures:
- Renovation Cost Estimate (Word/Excel): Alice AI LLM generated a structured calculation in 19 seconds (2 API calls), whereas Qwen3-235B took 1 min 27 sec and 5 iterations due to validation error corrections.
- Summary Table with Formulas (Excel): DeepSeek V4 applied the formula
=B2/$B$7across all auxiliary sheets in 46 seconds, while YandexGPT 5 Pro produced results twice as fast (20 seconds), but applied the share condition only to the final sheet. - Commercial Proposal: GigaChat 2 Max generated a concise draft in 3 min 48 sec (requiring 8 requests and one retry due to an error), whereas Alice prepared a comprehensive two-page document with a detailed payment schedule in 45 seconds.
- Web Page Generation: In multi-stage HTML/CSS assembly, YandexGPT 5 Pro led the pack (1 min 25 sec compared to nearly 5 minutes for Qwen3-235B).
Practical Takeaways for AI Adoption
The study highlighted two key patterns for enterprise neural network adoption:
- Provider Matters: The performance of the same architecture (e.g., DeepSeek V4) varies significantly depending on the endpoint—inference through third-party clouds can be slower than direct gateways due to queues and infrastructure configuration.
- Tendency Toward Brevity: Russian LLMs often simplify final answers when handling abstract prompts. However, with clear lists of required sections provided in the prompt, differences in generation quality disappear, leaving local models with a distinct advantage in speed and cost.