Remote Labor Index Benchmark Shows Modern AI Agents Can Handle Only 2.5% of Real Freelance Tasks

Research teams from the Center for AI Safety and Scale AI have introduced the Remote Labor Index (RLI), a specialized benchmark designed to evaluate the ability of autonomous AI agents to perform end-to-end work of remote professionals. The testing is based on 240 real jobs from the Upwork platform, spanning areas from game development to architectural design. The results demonstrated a significant gap between AI and humans: state-of-the-art systems successfully completed only 2.5% of the tasks at a level acceptable to clients.

The project's authors published the testing infrastructure and 10 demonstration tasks as open source on GitHub. The core set of 230 projects remains closed to prevent test data from leaking into future model training sets and to maintain benchmark integrity.

How the Remote Labor Index Works

Unlike traditional benchmarks based on abstract questions or isolated code snippets, RLI focuses on freelance market practice. The study's authors purchased completed and paid cases from Upwork freelancers (paying between $15 and $200 for the materials). Each test package includes:

• the client's original brief and requirements;

• source files and assets;

• the specialist's final output, accepted and paid for by the client.

The human solution serves as a quality benchmark with proven commercial value. Out of 240 cases, 207 were obtained directly from freelancers, while 40 more were curated from rare categories and open sources with author consent. In total, 23 industries are covered, with the largest shares including:

• video editing and animation — 13%;

• 3D modeling — 12%;

• graphic design — 11%;

• game development — 10%;

• sound design and audio — 10%;

• architecture — 7%;

• industrial and product design — 6%.

The average market price of an individual project in the dataset was $632 (median — $200), and the average human completion time was 29 hours (median — 11.5 hours). In total, the dataset represents over 6,000 working hours with a cumulative value exceeding $140,000.

AI Testing Results

As part of the evaluation, researchers tested models including Manus, Grok 4, Claude Sonnet 4.5, GPT-5, ChatGPT agent, and Gemini 2.5 Pro. The Manus agent led the field, successfully completing 2.5% of the tasks. Grok 4 and Sonnet 4.5 shared second place with a 2.1% success rate. In Elo rating terms, Manus also scored highest (509.9 points), which remains far below the human standard of 1,000 points (a gap of roughly 400 points means a human outperforms the model with a 10 to 1 probability).

Failure analysis revealed key agent weaknesses:

• Poor execution quality (45.6% of cases): crude graphics made of basic shapes instead of refined layouts, unnatural synthetic voiceovers.

• Incomplete work (35.7% of cases): delivering short snippets (e.g., an 8-second clip instead of the requested 8-minute video).

• Corrupted or empty files (17.6% of cases): unusable generation outputs.

• Violations of internal consistency (14.8% of cases): desynchronization of components — for example, mismatched building facades across adjacent 3D renders.

Successful Use Cases

Nevertheless, in a number of tasks, neural networks performed on par with skilled specialists. AI performed well in the following areas:

• audio editing: mixing voiceovers with background tracks, vocal isolation from accompaniment, sound effect generation for retro games;

• generating logo concepts and advertising graphics;

• drafting textual analytical reports;

• building interactive dashboards and basic programming.

The main barrier for models remains the lack of self-monitoring and visual verification mechanisms. After generating a 3D object, web page, or game code, the system cannot critically assess visual correctness or functionality.

RLI Features and Economic Impact

RLI surpasses benchmarks like GDPval or HCAST in task depth: project completion times are on average twice as long, and the file format range encompasses 72 extension types (including CAD drawings .dwg/.dxf, 3D models .obj/.gltf, architectural projects .rvt, design mockups .psd/.ai, and web games). The benchmark platform supports opening and rendering these formats directly in the browser.

To assess economic impact, the authors introduced the autoflation metric — a measure of overall task pool cost reduction when humans are replaced by neural networks where AI succeeds. At the current stage, savings reached 4%. In real terms, the Manus agent could have earned $1,720 out of a potential pool of $143,991. At the same time, running the model on a single task cost an average of just $2.34 (with an API cost ceiling of $30).

Benchmark Limitations

The researchers note that the current version of the test excludes tasks requiring sustained teamwork, direct real-time client interaction (such as online tutoring), or delayed-effect outcomes (SEO optimization). Furthermore, historical rates were used in calculations without adjusting for inflation.

The authors of RLI emphasize that despite AI's success in academic and reasoning benchmarks, automating complex practical tasks in the freelance market is still in its infancy. In the future, the dataset is planned to expand with new professional scenarios.