The RAG Obituary: Killed by Agents, Buried by Context Windows

Why Retrieval-Augmented Generation Won’t Survive the Context Revolution and the End of Chunking, Embeddings, and Rerankers as We Know Them.

I’ve been working in AI and search for a decade. First building Doctrine, the largest European legal search engine and now building Fintool, an AI-powered financial research platform that helps institutional investors analyze companies, screen stocks, and make investment decisions.

After three years of building, optimizing, and scaling LLMs with retrieval-augmented generation (RAG) systems, I believe we’re witnessing the twilight of RAG-based architectures. As context windows explode and agent-based architectures mature, my controversial opinion is that the current RAG infrastructure we spent so much time building and optimizing is on the decline.

The Rise of Retrieval-Augmented Generation

In late 2022, ChatGPT took the world by storm. People started endless conversations, delegating crucial work only to realize that the underlying model, GPT-3.5 could only handle 4,096 tokens... roughly six pages of text!

The AI world faced a fundamental problem: how do you make an intelligent system work with knowledge bases that are orders of magnitude larger than what it can read at once?

The answer became Retrieval-Augmented Generation (RAG), an architectural pattern that would dominate AI for the next three years.

The Mathematical Reality of Early LLMs

GPT-3.5 could handle 4,096 token and the next model GPT-4 doubled it to 8,192 tokens, about twelve pages. This wasn’t just inconvenient; it was architecturally devastating.

Consider the numbers: A single SEC 10-K filing contains approximately 51,000 tokens (130+ pages).

With 8,192 tokens, you could see less than 16% of a 10-K filing. It’s like reading a financial report through a keyhole!

The RAG Architecture: A Technical Deep Dive

RAG emerged as an elegant solution borrowed directly from search engines. Just as Google displays 10 blue links with relevant snippets for your query, RAG retrieves the most pertinent document fragments and feeds them to the LLM for synthesis.

The core idea is beautifully simple: if you can’t fit everything in context, find the most relevant pieces and use those. It turns LLMs into sophisticated search result summarizers.

Basically, LLMs can’t read the whole book but they can know who dies at the end; convenient!

The Chunking Challenge

Long documents need to be chunked into pieces and it’s when problems start. Those digestible pieces are typically 400-1,000 tokens each which is basically 300-750 words.

The problem? It isn’t as simple as cutting every 500 words.

Consider chunking a typical SEC 10-K annual report. The document has a complex hierarchical structure:

- Item 1: Business Overview (10-15 pages)

- Item 1A: Risk Factors (20-30 pages)

- Item 7: Management’s Discussion and Analysis (30-40 pages)

- Item 8: Financial Statements (40-50 pages)

After naive chunking at 500 tokens, critical information gets scattered:

- Revenue recognition policies split across 3 chunks

- A risk factor explanation broken mid-sentence

- Financial table headers separated from their data

- MD&A narrative divorced from the numbers it’s discussing

If you search for “revenue growth drivers,” you might get a chunk mentioning growth but miss the actual numerical data in a different chunk, or the strategic context from MD&A in yet another chunk!

At Fintool, we’ve developed sophisticated chunking strategies that go beyond naive text splitting:

- Hierarchical Structure Preservation: We maintain the nested structure from Item 1 (Business) down to sub-sections like geographic segments, creating a tree-like document representation

- Table Integrity: Financial tables are never split—income statements, balance sheets, and cash flow statements remain atomic units with headers and data together

- Cross-Reference Preservation: We maintain links between narrative sections and their corresponding financial data, preserving the “See Note X” relationships

- Temporal Coherence: Year-over-year comparisons and multi-period analyses stay together as single chunks

- Footnote Association: Footnotes remain connected to their referenced items through metadata linking

Each chunk at Fintool is enriched with extensive metadata:

- Filing type (10-K, 10-Q, 8-K)

- Fiscal period and reporting date

- Section hierarchy (Item 7 > Liquidity > Cash Position)

- Table identifiers and types

- Cross-reference mappings

- Company identifiers (CIK, ticker)

- Industry classification codes

This allows for more accurate retrieval but even our intelligent chunking can’t solve the fundamental problem: we’re still working with fragments instead of complete documents!

Once you have the chunks, you need a way to search them. One way is to embed your chunks.

The Embedding and Retrieval Pipeline

Each chunk is converted into a high‑dimensional vector (typically 1,536 dimensions in most embedding models). These vectors live in a space where, theoretically, similar concepts are close together.

When a user asks a question, that question also becomes a vector. The system finds the chunks whose vectors are closest to the query vector using cosine similarity.

It’s elegant in theory and in practice, it’s a nightmare of edge cases.

Embedding models are trained on general text and struggle with specific terminologies. They find similarities but they can’t distinguish between “revenue recognition” (accounting policy) and “revenue growth” (business performance).

Consider that example: Query: “What is the company’s litigation exposure?

RAG searches for “litigation” and returns 50 chunks:

- Chunks 1-10: Various mentions of “litigation” in boilerplate risk factors

- Chunks 11-20: Historical cases from 2019 (already settled)

- Chunks 21-30: Forward-looking safe harbor statements

- Chunks 31-40: Duplicate descriptions from different sections

- Chunks 41-50: Generic “we may face litigation” warnings

What RAG Reports: $500M in litigation (from Legal Proceedings section)

What’s Actually There:

- $500M in Legal Proceedings (Item 3)

- $700M in Contingencies note (”not material individually”)

- $1B new class action in Subsequent Events

- $800M indemnification obligations (different section)

- $2B probable losses in footnotes (keyword “probable” not “litigation”)

The actual Exposure is $5.1B. 10x what RAG found. Oupsy! By late 2023, most builders realized pure vector search wasn’t enough.

Hybrid Search: The Complexity That Actually Works

Enter hybrid search: combine semantic search (embeddings) with the traditional keyword search (BM25). This is where things get interesting.

BM25 (Best Matching 25) is a probabilistic retrieval model that excels at exact term matching. Unlike embeddings, BM25:

- Rewards Exact Matches: When you search for “EBITDA,” you get documents with “EBITDA,” not “operating income” or “earnings”

- Handles Rare Terms Better: Financial jargon like “CECL” (Current Expected Credit Losses) or “ASC 606” gets proper weight

- Document Length Normalization: Doesn’t penalize longer documents

- Term Frequency Saturation: Multiple mentions of “revenue” don’t overshadow other important terms

At Fintool, we’ve built a sophisticated hybrid search system:

1. Parallel Processing: We run semantic and keyword searches simultaneously

2. Dynamic Weighting: Our system adjusts weights based on query characteristics:

- Specific financial metrics? BM25 gets 70% weight

- Conceptual questions? Embeddings get 60% weight

- Mixed queries? 50/50 split with result analysis

3. Score Normalization: Different scoring scales are normalized using:

- Min-max scaling for BM25 scores

- Cosine similarity already normalized for embeddings

- Z-score normalization for outlier handling

So at the end the embeddings search and the keywords search retrieve chunks and the search engine combines them using Reciprocal Rank Fusion. RRF merges rankings so items that consistently appear near the top across systems float higher, even if no system put them at #1!

So now you think it’s done right? But hell no!

The Reranking Bottleneck: RAG’s Dirty Secret

Here’s what nobody talks about: even after all that retrieval work, you’re not done. You need to rerank the chunks one more time to get a good retrieval and it’s not easy. Rerankers are ML models that take the search results and reorder them by relevance to your specific query limiting the number of chunks sent to the LLM.

Not only LLMs are context poor, they also struggle when dealing with too much information. It’s vital to reduce the number of chunks sent to the LLM for the final answer.

The Reranking Pipeline:

1. Initial search retrieval with embeddings + keywords gets you 100-200 chunks

2. Reranker ranks the top 10

3. Top 10 are fed to the LLM to answer the question

Here is the challenge with reranking:

- Latency Explosion: Rerank adds between 300-2000ms per query. Ouch.

- Cost Multiplication: it adds significant extra cost to every query. For instance, Cohere Rerank 3.5 costs $2.00 per 1,000 search units, making reranking expensive.

- Context Limits: Rerankers typically handle few chunks (Cohere Rerank supports only 4096 tokens), so if you need to re-rank more than that, you have to split it into different parallel API calls and merge them!

- Another Model to Manage: One more API, one more failure point

Re-rank is one more step in a complex pipeline.

The Infrastructure Burden of Traditional RAG

What I find difficult with RAG is what I call the “cascading failure problem”.

1. Chunking can fail (split tables) or be too slow (especially when you have to ingest and chunk gigabytes of data in real-time)

2. Embedding can fail (wrong similarity)

3. BM25 can fail (term mismatch)

4. Hybrid fusion can fail (bad weights)

5. Reranking can fail (wrong priorities)

Each stage compounds the errors of the previous stage. Beyond the complexity of hybrid search itself, there’s an infrastructure burden that’s rarely discussed.

Running production Elasticsearch is not easy. You’re looking at maintaining TB+ of indexed data for comprehensive document coverage, which requires 128-256GB RAM minimum just to get decent performance. The real nightmare comes with re-indexing. Every schema change forces a full re-indexing that takes 48-72 hours for large datasets. On top of that, you’re constantly dealing with cluster management, sharding strategies, index optimization, cache tuning, backup and disaster recovery, and version upgrades that regularly include breaking changes.

The Fundamental Limitations of RAG for Complex Documents

Here are some structural limitations:

1. Context Fragmentation

- Long documents are interconnected webs, not independent paragraphs

- A single question might require information from 20+ documents

- Chunking destroys these relationships permanently

2. Semantic Search Fails on Numbers

- “$45.2M” and “$45,200,000” have different embeddings

- “Revenue increased 10%” and “Revenue grew by a tenth” rank differently

- Tables full of numbers have poor semantic representations

3. No Causal Understanding

- RAG can’t follow “See Note 12” → Note 12 → Schedule K

- Can’t understand that discontinued operations affect continuing operations

- Can’t trace how one financial item impacts another

4. The Vocabulary Mismatch Problem

- Companies use different terms for the same concept

- “Adjusted EBITDA” vs “Operating Income Before Special Items”

5. Temporal Blindness

- Can’t distinguish Q3 2024 from Q3 2023 reliably

- Mixes current period with prior period comparisons

- No understanding of fiscal year boundaries

These aren’t minor issues. They’re fundamental limitations of the retrieval paradigm.

Three months ago I stumbled on an innovation on retrievial that blew my mind

The Emergence of Agentic Search - A New Paradigm

In May 2025, Anthropic released Claude Code, an AI coding agent that works in the terminal. At first, I was surprised by the form factor. A terminal? Are we back in 1980? no UI?

Back then, I was using Cursor, a product that excelled at traditional RAG. I gave it access to my codebase to embed my files and Cursor ran a search n my codebase before answering my query. Life was good. But when testing Claude Code, one thing stood out:

It was better and faster and not because their RAG was better but because there was no RAG.

How Claude Code Search Works

Instead of a complex pipeline of chunking, embedding, and searching, Claude Code uses direct filesystem tools:

1. Grep (Ripgrep)

- Lightning-fast regex search through file contents

- No indexing required. It searches live files instantly

- Full regex support for precise pattern matching

- Can filter by file type or use glob patterns

- Returns exact matches with context lines

2. Glob

- Direct file discovery by name patterns

-

files like `**/*.py` or `src/**/*.ts` instantly

- Returns files sorted by modification time (recency bias)

- Zero overhead—just filesystem traversal

3. Task Agents

- Autonomous multi-step exploration

- Handle complex queries requiring investigation

- Combine multiple search strategies adaptively

- Build understanding incrementally

- Self-correct based on findings

By the way, Grep was invented in 1973. It’s so... primitive. And that’s the genius of it.

Claude Code doesn’t retrieve. It investigates:

- Runs multiple searches in parallel (Grep + Glob simultaneously)

- Starts broad, then narrows based on discoveries

- Follows references and dependencies naturally

- No embeddings, no similarity scores, no reranking

It’s simple, it’s fast and it’s based on a new assumption that LLMs will go from context poor to context rich.

Claude Code proved that with sufficient context and intelligent navigation, you don’t need RAG at all. The agent can:

- Load entire files or modules directly

- Follow cross-references in real-time

- Understand structure and relationships

- Maintain complete context throughout investigation

As you can see in this benchmark, a simple TXT file with urls+description outperforms complex RAG. Ouch. That’s a paradigm shift; LLMs eat the RAG scaffolding for breakfast.

This isn’t just better than RAG. It’s a fundamentally different paradigm. And what works for code can work for any long documents that are not coding files.

The Context Revolution: From Scarcity to Abundance

The context window explosion made Claude Code possible:

2022-2025 Context-Poor Era:

- GPT-4: 8K tokens (~12 pages)

- GPT-4-32k: 32K tokens (~50 pages)

2025 and beyond Context Revolution:

- Claude Sonnet 4: 200k tokens (~700 pages)

- Gemini 2.5: 1M tokens (~3,000 pages)

- Grok 4-fast: 2M tokens (~6,000 pages)

At 2M tokens, you can fit an entire year of SEC filings for most companies.

The trajectory is even more dramatic: we’re likely heading toward 10M+ context windows by 2027, with Sam Altman hinting at billions of context tokens on the horizon. This represents a fundamental shift in how AI systems process information. Equally important, attention mechanisms are rapidly improving—LLMs are becoming far better at maintaining coherence and focus across massive context windows without getting “lost” in the noise.

The Claude Code Insight: Why Context Changes Everything

Claude Code demonstrated that with enough context, search becomes navigation:

- No need to retrieve fragments when you can load complete files

- No need for similarity when you can use exact matches

- No need for reranking when you follow logical paths

- No need for embeddings when you have direct access

It’s mind-blowing. LLMs are getting really good at agentic behaviors meaning they can organize their work into tasks to accomplish an objective.

Here’s what tools like ripgrep bring to the search table:

- No Setup: No index. No overhead. Just point and search.

- Instant Availability: New documents are searchable the moment they hit the filesystem (no indexing latency!)

- Zero Maintenance: No clusters to manage, no indices to optimize, no RAM to provision

- Blazing Fast: For a 100K line codebase, Elasticsearch needs minutes to index. Ripgrep searches it in milliseconds with zero prep.

- Cost: $0 infrastructure cost vs a lot of $$$ for Elasticsearch

So back to our previous example on SEC filings. An agent can SEC filing structure intrinsically:

- Hierarchical Awareness: Knows that Item 1A (Risk Factors) relates to Item 7 (MD&A)

- Cross-Reference Following: Automatically traces “See Note 12” references

- Multi-Document Coordination: Connects 10-K, 10-Q, 8-K, and proxy statements

- Temporal Analysis: Compares year-over-year changes systematically

For searches across thousands of companies or decades of filings, it might still use hybrid search, but now as a tool for agents:

- Initial broad search using hybrid retrieval

- Agent loads full documents for top results

- Deep analysis within full context

- Iterative refinement based on findings

My guess is traditional RAG is now a search tool among others and that agents will always prefer grep and reading the whole file because they are context rich and can handle long-running tasks.

Consider our $6.5B lease obligation question as an example:

Step 1: Find “lease” in main financial statements

→ Discovers “See Note 12”

Step 2: Navigate to Note 12

→ Finds “excluding discontinued operations (Note 23)”

Step 3: Check Note 23

→ Discovers $2B additional obligations

Step 4: Cross-reference with MD&A

→ Identifies management’s explanation and adjustments

Step 5: Search for “subsequent events”

→ Finds post-balance sheet $500M lease termination

Final answer: $5B continuing + $2B discontinued - $500M terminated = $6.5B

The agent follows references like a human analyst would. No chunks. No embeddings. No reranking. Just intelligent navigation.

Basically, RAG is like a research assistant with perfect memory but no understanding:

- “Here are 50 passages that mention debt”

- Can’t tell you if debt is increasing or why

- Can’t connect debt to strategic changes

- Can’t identify hidden obligations

- Just retrieves text, doesn’t comprehend relationships

Agentic search is like a forensic accountant:

- Follows the money systematically

- Understands accounting relationships (assets = liabilities + equity)

- Identifies what’s missing or hidden

- Connects dots across time periods and documents

- Challenges management assertions with data

Why Agentic Search Represents the Future

1. Increasing Document Complexity

- Documents are becoming longer and more interconnected

- Cross-references and external links are proliferating

- Multiple related documents need to be understood together

- Systems must follow complex trails of information

2. Structured Data Integration

- More documents combine structured and unstructured data

- Tables, narratives, and metadata must be understood together

- Relationships matter more than isolated facts

- Context determines meaning

3. Real-Time Requirements

- Information needs instant processing

- No time for re-indexing or embedding updates

- Dynamic document structures require adaptive approaches

- Live data demands live search

4. Cross-Document Understanding

Modern analysis requires connecting multiple sources:

- Primary documents

- Supporting materials

- Historical versions

- Related filings

RAG treats each document independently. Agentic search builds cumulative understanding.

5. Precision Over Similarity

- Exact information matters more than similar content

- Following references beats finding related text

- Structure and hierarchy provide crucial context

- Navigation beats retrieval

The evidence is becoming clear. While RAG served us well in the context-poor era, agentic search represents a fundamental evolution. The potential benefits of agentic search are compelling:

- Elimination of hallucinations from missing context

- Complete answers instead of fragments

- Faster insights through parallel exploration

- Higher accuracy through systematic navigation

- Massive infrastructure cost reduction

- Zero index maintenance overhead

The key insight? Complex document analysis—whether code, financial filings, or legal contracts—isn’t about finding similar text. It’s about understanding relationships, following references, and maintaining precision. The combination of large context windows and intelligent navigation delivers what retrieval alone never could.

RAG was a clever workaround for a context-poor era. It helped us bridge the gap between tiny windows and massive documents, but it was always a band-aid. The future won’t be about splitting documents into fragments and juggling embeddings. It will be about agents that can navigate, reason, and hold entire corpora in working memory.

We are entering the post-retrieval age. The winners will not be the ones who maintain the biggest vector databases, but the ones who design the smartest agents to traverse abundant context and connect meaning across documents. In hindsight, RAG will look like training wheels. Useful, necessary, but temporary.

The next decade of AI search will belong to systems that read and reason end-to-end. Retrieval isn’t dead—it’s just been demoted.