Back to Videos

MIT Solves Unlimited Context Windows with Recursive Language Models (RLMs)

YouTube

This video introduces Recursive Language Models (RLMs), a groundbreaking technique developed by MIT researchers that effectively solves the problem of limited context windows in Large Language Models (LLMs). Traditional LLMs suffer from 'context rot,' where their performance degrades significantly as input length increases beyond their physical context window. Current solutions like context compaction are often lossy, sacrificing detail for brevity. RLMs, however, tackle this by treating the entire prompt as a variable within a Python REPL environment, allowing the LLM to programmatically interact with and recursively query snippets of the prompt. This approach enables LLMs to process arbitrarily long inputs, demonstrating strong performance even at the 10M+ token scale, dramatically outperforming base models and other scaffolding methods, often at a comparable or even lower cost. The technique is model-agnostic and provides significant benefits for information-dense tasks by preserving all details without compression. The key takeaway is that RLMs unlock a new paradigm for scaling LLMs by building an external environment and tooling around the core intelligence of the model, rather than directly feeding massive inputs into the neural network. This method allows for 'infinite' context windows, enabling deep research, information aggregation, code understanding, and complex reasoning over vast amounts of data without loss of quality or prohibitive costs. The presenter also highlights Zapier Agents as a practical example of scaffolding, where AI agents are empowered with tools to automate workflows, demonstrating the broader trend of enhancing LLM capabilities through external infrastructure.

The Challenge of Limited Context Windows in LLMs- Modern Language Models (LLMs) inherently have a limited context window, which dictates the maximum input size they can process directly.- This limitation leads to a phenomenon called 'context rot,' where the quality of responses degrades rapidly as the input prompt length exceeds the model's effective context window. GPT-5, for example, shows a sharp decline in performance for longer inputs.

Existing Approaches and Their Limitations- A common inference-time approach to handle long contexts is context condensation or compaction.- This involves repeatedly summarizing the context once it exceeds a certain length threshold.- Limitation: This method is lossy, meaning information and details are inevitably lost during the compression process, which can be critical for tasks requiring precise recall or complex reasoning.

Introducing Recursive Language Models (RLMs)- MIT researchers propose Recursive Language Models (RLMs) as a general inference strategy to process arbitrarily long prompts.- Key Insight: Long prompts should not be fed directly into the neural network. Instead, they should be treated as part of an external environment that the LLM can symbolically interact with.

How RLMs Work- RLMs operate within a Python REPL environment (Read-Eval-Print Loop).

- The input prompt (even millions of tokens long) is loaded as a **variable** within this environment (e.g., a text file).
- The LLM is given **tools** (e.g., Python code) to 'peek into,' decompose, and recursively call itself over snippets of the variable (the prompt).
- This allows the model to **search deeply** into specific sections of the prompt, retrieve relevant details, and combine findings without needing to load the entire context into its physical context window at once.
- This recursive sub-calling enables an **effectively infinite context window** without the lossy compression of other methods.

RLM Performance and Benefits- Dramatic Outperformance: RLMs successfully handle inputs up to two orders of magnitude beyond typical context windows, outperforming base LLMs and common long-context scaffolds (like summarization and retrieval baselines) on diverse tasks.

  • Cost-Effective: RLMs can be up to 3x cheaper than summarization baselines for certain tasks (e.g., BrowseComp-Plus) because they selectively view context, rather than ingesting the entire input repeatedly.
  • Model-Agnostic: The strategy can be applied to any underlying LLM (tested with GPT-5 and Qwen3-Coder-480B).

Evaluation Tasks- S-NIAH (Needle-in-a-Haystack): Finding a specific phrase/number in unrelated text. (Modern models largely solved this.)

  • BrowseComp-Plus (1K documents): Multi-hop question answering requiring reasoning over multiple documents.
  • OOLONG: Long reasoning benchmark requiring examining and transforming chunks of input semantically, then aggregating them.
  • OOLONG-Pairs: An extension of OOLONG requiring aggregating pairs of chunks.
  • LongBench-v2 CodeQA: Multi-choice code repository understanding, challenging for frontier models.

Key Observations from the Research

  • Observation 1: Massive Scalability and Outperformance. RLMs can scale to 10M+ token regimes and outperform base LMs and existing task-agnostic agent scaffolds on long context tasks. They are also often cheaper.
  • Observation 2: REPL Environment and Recursive Sub-calling are Crucial. The REPL environment and recursive sub-calling provide strong benefits for handling long, information-dense inputs.
  • Observation 3: Better Performance Scaling. While LM performance degrades as a function of input length and problem complexity, RLM performance scales better.
  • Observation 4: Comparable Inference Cost with Variance. The inference cost of RLMs remains comparable to a base model call, but exhibits high variance due to differences in trajectory lengths (how deep the recursion goes).
  • Observation 5: Model-Agnostic Strategy with Model-Specific Decisions. RLMs are a model-agnostic inference strategy, but different models make different overall decisions on context management and sub-calling, affecting their behavior.

Zapier Agents (Sponsor Spotlight)

  • Zapier Agents are highlighted as an example of AI orchestration and scaffolding around AI models.
  • These agents connect to over 7,000 different tools, allowing for automated workflows.
  • They offer tight integration with LLMs like Claude via MCP, enabling AI to take action in various tools. This demonstrates how external tools enhance the capabilities of core AI models for day-to-day and complex business tasks.

Timestamps

00:00
Introduction to Recursive Language Models (RLMs)Overview of MIT's breakthrough in solving unlimited context windows and the concept of scaffolding.
00:33
Performance Comparison: GPT-5 vs. RLM(GPT-5)Visual comparison showing how RLM maintains consistent quality over long context lengths where GPT-5 degrades.
01:13
Understanding Context Rot and CompactionExplanation of limited context windows, 'context rot,' and the limitations of traditional context compaction methods.
03:32
How Recursive Language Models WorkDetailed explanation of RLM architecture using a Python REPL environment to process prompts recursively.
06:07
The Power of Scaffolding around LLMsDiscussion on why building tools and infrastructure around core LLM intelligence is crucial for future advancements.
07:41
Evaluation Tasks and BenchmarksDescription of the four main use cases and benchmarks used to test RLMs: deep research, information aggregation, code understanding, and synthetic reasoning.
09:20
Detailed Explanation of BenchmarksIn-depth look at S-NIAH, BrowseComp-Plus, OOLONG, OOLONG-Pairs, and LongBench-v2 CodeQA tasks.
12:00
Methods and Baselines for TestingOverview of the models (GPT-5, Qwen3-Coder-480B) and different approaches (RLM with/without sub-calls, summary agent, CodeAct) used for comparison.
12:49
Results and Performance MetricsAnalysis of the test results showing RLM's superior performance across various complex long-context tasks compared to other methods.
13:16
Observations from RLM TestingSummary of five key observations highlighting RLM's scalability, benefits of the REPL environment, performance scaling, cost considerations, and model-agnostic nature.
16:11
Cost Analysis of RLMsDetailed look at the cost implications of RLM compared to other methods, showing that while generally cheaper, costs can spike with very long recursive trajectories.
16:51
RLM Code Snippet ExampleA brief glimpse into the Python code (using RegEx) that an RLM uses to interact with its environment.
17:14
Conclusion: The Future of Scaffolding AIFinal thoughts on the significance of scaffolding and external tooling in enhancing LLM capabilities and unlocking new possibilities.

Target Audience

AI researchers, machine learning engineers, data scientists, software developers, and anyone interested in the cutting edge of LLM capabilities and practical applications of AI in handling large datasets and complex tasks.

Use Cases

  • -Deep research over extensive documents or corpora (e.g., millions of documents)
  • -Understanding and navigating large code repositories (e.g., giant codebases)
  • -Long-horizon tasks requiring memory across numerous interactions
  • -Multi-hop question answering across diverse document sets
  • -Information aggregation from vast and complex data sources

Key Topics

Recursive Language Models (RLMs)Solving Long Context Window LimitationsInference-Time Scaling for LLMsAI Agent ArchitecturesPerformance and Cost Analysis of RLMs