How to Make AI Agents Work Reliably: Lessons from Building Maritime Analyst Shippy

Building autonomous AI agents is often described as simply connecting a powerful language model to a set of APIs. However, in practice, developers inevitably face the non-deterministic nature of neural networks, their tendency to hallucinate, and context loss during complex task chains. A prime example is the Shippy project — an intelligent agent developed by the Allen Institute for AI (Ai2).

Why Shippy Was Created

Shippy is integrated into the Skylight analytical platform, which helps maritime authorities across more than 70 countries detect illegal fishing by correlating satellite imagery with vessel transponder data. Powered by Claude Opus, the agent enables specialists to query complex analytics regarding suspicious ocean activity using natural language.

In such missions, the cost of error is exceptionally high: a false conclusion could send a patrol boat on the wrong course. To minimize risks, the engineers had to rethink conventional approaches to AI agent architecture.

Key AI Agent Challenges and Their Solutions

1. Exceeding Scope of Competence

Language models tend to provide an answer even when input data is insufficient, leading to false accusations of vessel violations. To prevent hallucinations, Shippy implemented a system-level set of rules (Soul):

• The agent is strictly prohibited from making legal verdicts and giving direct orders to operators (such as dispatching a patrol).
• Responses must be based exclusively on information from external Skylight tools, rather than the neural network's internal parametric knowledge.

2. Inability to Follow Complex API Rules

When the model was granted direct access to a REST API with dozens of parameters, it consistently made mistakes when constructing queries. The solution was to replace direct API calls with an intermediate command-line interface (CLI). Traditional deterministic code handled strict parameter validation and edge cases (such as pagination), leaving the model to merely select high-level actions.

3. Errors in Long Sequences and Data Loss

When passing large payloads between tools via standard output, context quickly overflowed, and some data was lost due to system buffer limits. Developers isolated heavy datasets: the CLI saves intermediate results to local JSON files, while the LLM merely passes references to them between processing stages.

4. Hallucinating Non-Existent Tools

During testing, the agent began generating fictional CLI commands to solve tasks. To address this issue, developers formalized workflows into versioned skill instructions (Skills). Instead of unrestricted freedom of action, the model received clear scripts tailored to concrete scenarios, such as searching marine protected areas or analyzing vessel tracks.

5. Challenges with Quality Evaluation

Because an agent can solve the same task through various paths, standard code tests are ineffective. Ai2 created an automated evaluation benchmark: scenarios run against real data, and a separate judge model scores outputs based on weighted criteria (geospatial accuracy, timeframes, source attribution), allowing teams to detect regressions promptly after modifying instructions.

Key takeaway from the developers: a language model should not be the sole component of a system. A reliable architecture is built on a clear separation of concerns: the LLM decides what needs to be done, while classical deterministic code determines how exactly it is executed and ensures constraints are met.