A Paradigm Shift: From Manual Post-Mortems to Continuous Stress Testing
The traditional approach to handling incidents in product teams often moves in circles: a production outage is followed by a post-mortem, an action item list is compiled, but due to product feature priorities, improvements get stuck in the backlog—until the next failure. To break this pattern, Avito transitioned to proactive chaos testing, validating service resilience directly under production load.
To address this challenge, the PaaS team deployed a "chaos as a service" infrastructure powered by artificial intelligence: an LLM analyzes historical incidents, while a dedicated agent correlates metric anomalies with defects in the codebase.
Why Open-Source Tools Fell Short
The popular Kubernetes operator LitmusChaos served as the base engine for fault injection. However, in isolation, it can only manipulate pods and containers: terminating processes, consuming CPU, or severing network connections. According to the company's internal incident statistics, out-of-the-box Litmus features cover only about 22% of actual failure scenarios.
The tool lacked an understanding of database logic—for instance, it couldn't gracefully trigger a primary failover in Redis or MongoDB replica sets. To overcome these limitations, engineers developed their own orchestrator, Kratos, in Go. The Litmus and Kratos tandem nearly doubled the coverage of reproducible real-world outages to 41% (over 400 cases from a sample of 977 incidents).
How the Neural Network Investigates Incidents and Reads Repositories
Artificial intelligence is leveraged in the system at two critical stages:
- Retrospective incident analysis: The model examined 977 historical reports (LSRs), evaluating the feasibility of reproducing each failure through controlled stress testing. This helped automatically prioritize scenarios, with the database primary re-election test (
trigger-primary-re-election) emerging as the clear leader. - AI-driven test assessments: When an experiment fails, the AI agent receives the run profile, degradation metrics, and repository access to the service. The neural network correlates symptoms with the source code and delivers a specific verdict—such as pointing out the lack of jitter in retry logic or improper cache error handling, complete with references to specific files and lines of code.
The accuracy of these conclusions is confirmed in practice: over the recent period, service owners agreed with 87% of the agent's findings (61 approvals versus 9 objections), and dozens of detected issues were immediately queued for remediation.
Production Reality: Statistics and an Unexpected Hurdle
An automated runner initiates tests during off-peak night hours after capturing baseline metrics and assessing risks. Accumulated statistics highlight the following outcomes:
- 1,129 full-scale tests completed on live infrastructure (95.6% involving storage primary failover).
- 79% of runs received "excellent" and "good" ratings, with dependent systems handling node failures gracefully.
- 89 escalations recorded for performance degradation across 64 services.
- Impact on the error budget remained minimal: the median monthly burn rate was just 0.14% per service.
Interestingly, the primary bottleneck for scaling was not technical safety constraints, but the human factor. The system executes roughly 6 experiments per day instead of the targeted 44: the main slowdown stems from the need to secure explicit consent from system owners to enroll a service into the automated pipeline.
Implementation Results
The initiative has demonstrated steady progress: the number of services unable to survive the loss of a database primary replica dropped from 141 historical cases down to just 8 systems at present. Moving forward, the team plans to expand verification to canary release stages and implement self-healing mechanisms for stalled services.