ClawBench Benchmark Exposes AI Agent Weaknesses: Top Real-Web Success Rate Reaches Just 33%

Researchers have introduced ClawBench, a new benchmark designed to assess the ability of AI agents to perform practical tasks across the live internet. Test scenarios covered everyday activities such as booking tickets, filling out job applications, and placing orders. According to the study, even the top-performing model—Claude Sonnet 4.6—managed to correctly complete only 33.3% of the scenarios. The testing framework's source code is hosted on GitHub, while supporting materials and the dataset are available on the project's official website.

Why Synthetic Sandboxes Give a False Impression

Popular evaluation environments (such as WebArena, OSWorld, and VisualWebArena) run models inside isolated sandboxes containing static snapshots of web pages. These environments lack dynamic JavaScript rendering, cookie banners, CAPTCHAs, and mandatory authentication steps. Consequently, models demonstrate deceptively high success rates (65–75%), which plummet dramatically when encountering the real web.

Other platforms testing live web interactions (including WebVoyager and AssistantBench) focus primarily on passive information retrieval and reading. Systematic evaluation of interactive actions—such as filling out forms and submitting data—was largely missing prior to ClawBench.

Key Features of the ClawBench Methodology

The dataset includes 153 practical tasks across 144 live websites, spanning 15 domains: from online shopping, financial management, and travel booking to software development and social media workflows.

Each evaluation task provides a base URL, a natural language instruction, and a target HTTP request confirming successful task completion. For every scenario, the researchers pre-recorded reference human trajectories.

To prevent agents from making real monetary payments or submitting unintended real-world forms, the developers implemented a safeguard system based on the Chrome DevTools Protocol (CDP) and a lightweight browser extension. When the AI clicks a submission or payment button, the system intercepts the outgoing HTTP request, logs its payload, and aborts the actual network transmission. During validation, this interceptor operated with 100% accuracy across all tasks.

Comprehensive Data Logging and Agent Evaluation

Throughout test execution, the benchmark continuously records five levels of data:

• full-frame video capture via Xvfb and FFmpeg;

• step-by-step screenshots taken after every action;

• network traffic logs via CDP;

• model reasoning chains and tool invocations in JSON format;

• user input events (mouse clicks, scrolling, keystrokes).

Result evaluation is handled by a specialized evaluator powered by Claude Code. It matches the tested model's steps against reference human actions, validates the accuracy of filled form fields, and produces a structured justification alongside a binary success score.

AI Model Evaluation Results

Seven contemporary models were evaluated under the benchmark. The success rates were distributed as follows:

• Claude Sonnet 4.6 — 33.3%;

• GLM-5 — 24.2%;

• Gemini 3 Flash — 19.0%;

• Claude Haiku 4.5 — 18.3%;

• GPT-5.4 — 6.5%;

• Gemini 3.1 Flash Lite — 3.3%;

• Kimi K2.5 — 0.7%.

In comparison, on synthetic benchmarks like OSWorld and WebArena, models like Claude Sonnet 4.6 and GPT-5.4 routinely achieve 67–75% success rates.

Agents demonstrated relatively solid performance on financial services, educational portals (enrolling in Coursera/edX courses, submitting LeetCode solutions), and travel booking. Developer-oriented workflows proved the most difficult (setting up GitHub repositories, creating forks on Hugging Face, configuring Airtable), where the top model achieved only 19%, alongside social platforms featuring dynamic multi-step forms (submitting reviews on Glassdoor or Vivino).

Significance of the Study

The findings from ClawBench demonstrate that traditional synthetic benchmarks are nearing saturation and no longer reflect the true capabilities of web agents. This new methodology allows researchers to precisely diagnose model failure modes during interactions with modern web interfaces, establishing a realistic baseline for future autonomous AI agent development.