This video provides an in-depth exploration of how to build AI agents capable of running for several hours without losing focus or coherence. Presented by engineers from Anthropic, the talk covers the transition from basic model prompting to sophisticated multi-agent harness designs that utilize a Planner-Generator-Evaluator loop. Viewers will learn about the history of the Claude model series and the specific engineering techniques used to overcome context rot, planning failures, and self-evaluation traps.
Key Takeaways
- Autonomous agents often fail due to finite context windows, poor planning, and the inability to objectively judge their own work.
- Anthropic has seen agent runtimes increase from 20 minutes to over 12 hours by co-evolving models and harnesses.
- Separating the builder (Generator) from the judge (Evaluator) creates adversarial pressure that significantly improves output quality.
- A dedicated Planner should break down vague human prompts into actionable, high-level specifications without over-specifying technical details.
- The primary debugging loop for these systems involves reading model traces to find where the AI's judgment diverges from human judgment.
- As base models become more intelligent, developers should actively delete unnecessary harness scaffolding to reduce cost and complexity.
Three Reasons Agents Lose the Plot
Building agents that stay on track is difficult due to three primary psychological and technical barriers in current LLMs. First is the context problem: models have finite windows and suffer from coherence degradation as those windows fill up, a phenomenon known as context rot. Second is the planning problem: models often attempt to one-shot entire projects, running out of context mid-feature or stopping prematurely after seeing partial progress. Third is the verification problem: models are notoriously poor at judging their own output, often rating mediocre work as good because they cannot separate their generation bias from objective quality requirements.
The Evolution of Claude and its Harness
The history of Claude's development shows a parallel evolution between the model's innate abilities and the external scaffolding provided by engineers. Early versions struggled with basic bash commands, but the introduction of Sonnet 3.5 marked the beginning of serious agentic capabilities. The harness has evolved from simple artifact generation to complex systems involving the Model Context Protocol (MCP) and Computer Use APIs. By 2026, the combination of Opus 4.6 and refined multi-agent teams allowed for autonomous runs exceeding 30 hours, moving beyond simple code generation to full application development.
The Planner-Generator-Evaluator Loop
To achieve high-quality results, Anthropic utilizes a three-agent system. The process begins with a Planner that converts a single line of human intent into a comprehensive feature specification. The Generator then works on one sprint at a time, building out features. Crucially, an independent Evaluator, often using tools like Playwright for web apps, navigates the live application and scores it against a detailed rubric. This adversarial setup is effective because tuning a standalone evaluator to be skeptical is far more tractable than making a generator self-critical. The agents negotiate a contract for each sprint, ensuring they agree on what constitutes a finished feature before code is even written.
Making Subjective Quality Gradable
A common misconception is that subjective taste cannot be graded by AI. Anthropic engineers have found success by creating rubrics that focus on design quality, originality, craft, and functionality. By weighting these criteria and providing few-shot examples of good versus bad design, an evaluator agent can effectively drive the generator toward professional aesthetics. This prevents the common pitfall of AI slop and ensures that the final product feels coherent and polished.
Practical Applications
Developers can apply these patterns today using the primitives already shipping in tools like Claude Code. By implementing an auto-mode for safe unattended handling and using sub-agents for QA roles, teams can begin automating the long-tail of software development. The most important practice is reading the traces. Finding the point where a model's judgment fails is the key to updating prompts and refining the system. As the frontier of model capability moves, engineers must remain flexible, deleting the scaffolding that models have outgrown to keep systems efficient and cost-effective.
Frequently Asked Questions
Why shouldn't a model evaluate its own work?
Models have a natural bias toward the text they generate. They often rationalize errors or overlook missing features because the logic used to create the error is the same logic used during self-review. Using a separate evaluator agent with a harsh system prompt breaks this cycle of sycophancy.
How does the Planner prevent scope creep?
The Planner is designed to work at a high level of abstraction. It breaks down a project into sprints and high-level specifications without getting bogged down in implementation details. This leaves the technical execution to the Generator and Evaluator, who negotiate the specific technical contracts on the fly.
What is the most effective way to debug long-running agents?
The primary debugging tool is the model trace. Engineers must read through the logs to identify the specific turn where the model's logic deviated from human expectations. Updating the system prompt based on these specific failure points is more effective than broad architectural changes.