Introduction to Self-Evolving Software (00:00)
- The video starts by asserting that all software will soon be self-evolving software.
- Introduces a new paper: "Meta-Harness: End-to-End Optimization of Model Harnesses" from Stanford, MIT, and KRAFTON.
Understanding AI Harnesses (00:26)
- Explains a harness as traditional code wrapped around a model (like Claude, GPT-4, Gemini) that dictates how it operates.
- Harnesses allow models to: store memories, search through text, write/execute code, and much more.
- Highlights popular agentic harnesses like Claude Code, Cursor, and Factory.
- Emphasizes that current harnesses are typically human-written and manually evolved, not self-evolved.
Andrej Karpathy's AutoResearch (01:14)
- Mentions Andrej Karpathy's AutoResearch project, which gained significant traction (61k stars).
- AutoResearch enables an AI model (e.g., Claude) to propose and run a series of experiments overnight, self-improving its ability to train a GPT-2 level model.
- This concept extends to AI training and improving itself across different pieces of software.
The Importance of Harnesses (02:23)
- Explains that the performance of LLM systems depends not only on model weights but also on their harness – the code that determines how information is stored, retrieved, and presented to the model.
- Drawing an analogy: models are the powerful 'engine' of a car, but harnesses are the 'steering wheel, seats, and transmission' that direct that power.
- Harness engineering (the practice of refining the code around an LLM) can produce a 6x performance gap.
- Despite its importance, harness engineering largely remains a manual process.
Introducing Meta-Harness's Approach (03:55)
- Meta-Harness is introduced as an outer-loop system that searches over harness code for LLM applications.
- Current text optimizers are poorly matched to harness engineering due to: short horizons, heavily compressed feedback (scalar scores), and restricted feedback to short templates/summaries.
- Compressed feedback often removes necessary information for tracing downstream failures.
How Meta-Harness Works (06:34)
- Addresses the limitation of existing methods by allowing adaptive access to useful context.
- The core idea: instead of trying to pack all necessary information into a single prompt, Meta-Harness lets the coding agent itself decide what information it needs to access from the filesystem.
- This coding agent is a language-model-based system that can invoke developer tools and modify code.
- It maintains a "full history" of all previous harness candidates, including source code, evaluation scores, and execution traces. This is critical for reasoning over large codebases and complex interactions.
- The proposer (coding agent) is free to inspect any prior harness (not just the best ones) and its execution trace when proposing new ones, enabling it to avoid local maxima.
- This simplicity is deliberate, delegating diagnosis and edit decisions to the proposer rather than relying on hard-coded search heuristics.
Experimental Results: Text Classification (16:39)
- Meta-Harness was evaluated on three task domains: online text classification, math reasoning, and agentic coding.
- Online Text Classification Benchmarks: Competed against human-designed strategies (Zero-Shot, Few-Shot, MCE, ACE) and program-search methods (OpenEvolve, TTT-Discover).
- Key Findings:
- Meta-Harness significantly outperformed all prior methods on average (e.g., 48.0% vs. 40.9% for ACE on average).
- It used significantly fewer context tokens (11.4k average) compared to ACE (50.8k) and MCE (28.5k), making it more efficient.
- It matched the best prior text optimizers with 10x fewer full evaluations and surpassed their final accuracy by more than 10 points.
Experimental Results: Accuracy-Context Tradeoffs & OOD Evaluation (20:33)
- Accuracy-Context Tradeoffs: Meta-Harness performs free-form optimization, expressing a preference for both accuracy and context cost rather than committing to a single scalar objective. It consistently achieved higher median and best scores than other methods.
- Out-of-Distribution (OOD) Task Evaluation: Meta-Harness generalizes well to entirely new datasets unseen during training, outperforming the next best method by 2.9 points across nine different datasets, further demonstrating its robustness and adaptability.
Experimental Results: Math Reasoning (21:53)
- Applied to International Math Olympiad (IMO) level math problems.
- The discovered Meta-Harness retrieval strategy significantly improves reasoning on these IMO-level problems across all five held-out models, with a 4.7-point average gain over no retriever.
- This success is attributed to solutions often sharing reusable proof patterns, which previous reasoning traces can inform.
Experimental Results: Agentic Coding (TerminalBench-2) (23:04)
- TerminalBench-2 evaluates LLM agents on 89 challenging tasks requiring long-horizon, fully autonomous execution under complex dependencies.
- Meta-Harness achieved a 76.4% pass rate on Opus 4.6, surpassing hand-engineered Terminus-KIRA (74.7%) and ranking #2 among all Opus 4.6 agents on the leaderboard (only ForgeCode scored higher at 81.8%).
- On the weaker Claude Haiku 4.5 model, Meta-Harness achieved 37.6% improvement, outperforming Goose (35.5%) by 2.1 points.
- This success highlights that automating the search for long-horizon text optimization loops is highly promising.
The Bitter Lesson and Future Outlook (24:40)
- The speaker connects these results to "The Bitter Lesson": AI figuring out what to do will always beat humans telling it what to do.
- Uses Tesla's Full Self-Driving as an analogy: it transitioned from a hybrid neural net/hand-coded rules system to an end-to-end neural net, allowing the AI to learn heuristics itself.
- Concludes that self-evolving/self-improving software will be a massive presence in artificial intelligence in the coming years.
- We are already seeing frontier labs releasing models trained by previous models, and harnesses built by previous harnesses.
- The future involves all software being built by previous software, with continuous iteration and improvement driven by AI itself. This is seen as a fascinating and rapidly developing area.