Vectorless RAG: A Tree-Based Retrieval Architecture

Summary 

Vector RAG fails on structured enterprise corpora because flat embeddings cannot represent hierarchy or version semantics. The fix is vectorless RAG – an LLM walks a hierarchical document tree at query time, returning grounded, auditable answers. On 58,000 policy documents, this lifted multi-hop accuracy from 51% to 79% in eight weeks.

RAG (retrieval-augmented generation) has become the default pattern for grounding LLMs on private data: chunk documents, embed them, store in a vector database, retrieve by cosine similarity. The recipe works well enough that many teams stop there – and it is also where accuracy quietly goes to die. On complex enterprise corpora – regulatory filings, contracts, policy manuals – that pipeline hits a wall that more embeddings, more chunks, and better re-rankers do not break through. What follows is a walkthrough of how the failure mode shows up in production, why familiar mitigations do not solve it, and what vectorless RAG – the alternative architecture we ended up deploying – looks like end-to-end.

Why Does Vector RAG Fail on Structured Enterprise Corpora?  

The corpus was 58,000 policy documents averaging 40 pages each, spanning nine regulated business domains with strict version and authority semantics – the kind of structured enterprise data our data analytics services are built around. Expected queries were compositional – multi-hop, cross-section, version-sensitive, often predicated on metadata never present in document text: effective date, supersedes-chain, jurisdiction. The inherited pipeline – 512-token chunking, dense embeddings, top-k retrieval, a cross-encoder re-ranker, answer synthesis – was textbook, and structurally misaligned on three dimensions. The pipeline shows familiar symptoms – chunking failures, missed cross-section context, ungrounded synthesis – that match well-documented seven failure points in RAG systems but resist the standard fixes.

• Embedding-geometric pathologies. Dense embeddings exhibit anisotropy – semantic similarity concentrates in a narrow cone, inflating apparent closeness between distinct passages. “Vendor onboarding approval” and “vendor offboarding approval” landed within a cosine whisker of each other; re-rankers from the same family inherited it.  
• Loss of referential integrity at chunk boundaries. For example, policy text often depends on cross-references such as “per Section 2.3 above” or “except as defined in Schedule 1.” However, flat chunking breaks these dependencies, and neither overlap nor re-chunking can fully restore the original context.
• No channel exists for authority metadata. Instead, version details, effective dates, and supersedes chains act as structural annotations, not plain text. As a result, a vector pipeline cannot expose them during retrieval, causing authoritative and deprecated passages to compete equally. Consequently, the retriever could not fail gracefully, and user trust collapsed.

What We Tried Before Giving Up on Vectors  

Three mitigations the RAG literature recommends – each moved the needle a little, none enough.  

• Hierarchical chunking with sentence-boundary awareness. Marginal improvement on chunk coherence; no movement on multi-hop or version errors. The problem was the absence of structure in the retrieval layer, not chunk quality.
• A second-stage cross-encoder re-ranker (bge-reranker-large). Modest gain on single-hop queries, negligible on compound ones. Re-rankers sharpen ordering but cannot recover documents the retriever never surfaced.
• Metadata filtering at query time. Effective only when the query explicitly named the metadata – which users did not. They asked natural-language questions and expected the system to figure out version and jurisdiction itself.  

Each fix addressed a symptom; none touched the underlying mismatch – the same trap teams hit when patching legacy systems instead of committing to legacy app modernization.

What Is Vectorless RAG? A Definition  

Vectorless RAG is a retrieval-augmented generation architecture that uses LLM reasoning over structured document indexes instead of vector similarity over flattened chunks. The best-known instance is PageIndex-style retrieval documents are parsed into a navigable tree, each node summarized, and the LLM walks the tree at query time, descending into relevant branches and pruning the rest. PageIndex, released by VectifyAI in 2025, is a vectorless reasoning-based RAG engine that reached 98.7% accuracy on the FinanceBench benchmark – a useful upper bound when single-document precision matters and a result that aligns with our financial services AI solutions work.

Three properties separate it from vector RAG. The index is readable – every node carries a summary, path, and metadata, so a human can audit the trace. Retrieval is reasoned, not measured – the same LLM-driven judgment that runs through our enterprise machine learning services. Structure is first-class. The trade-off: more LLM tokens per query – cheap enough for high-value enterprise Q&A over tens of thousands of documents, but not for open-web search over billions of pages. 

How We Built the New Architecture  

The production pipeline replaced the vector store with a four-stage retriever and a safety/audit layer – built using the same approach that drives our broader AI development services.

The vectorless pipeline routes each query by domain, navigates the relevant document trees, fetches grounded leaf content, and synthesizes an answer with full path-based citations.

• Tree construction (offline). First, each document is parsed into a hierarchical tree that preserves chapter, section, and clause boundaries. Next, a small model generates 40-80 word summaries for every node, while metadata such as version, domain, effective date, and supersedes list is attached. Finally, the trees are stored as JSON, eliminating the need for a vector database.
• Domain routing. A lightweight classifier maps the query to one or more of the nine business domains, narrowing the candidate tree set from 58,000 documents to roughly 2,000-4,000 per query. 
• Tree navigation. For each candidate tree, the reasoning model receives the root summary plus immediate children’s summaries and decides which branch (if any) to descend. The process repeats until it reaches leaf nodes, bounded to eight hops per tree.  
• Answer synthesis. Retrieved leaf contents, with structural paths, are passed to the answer model along with version and authority metadata. The model is instructed to cite by path, not similarity, and to refuse when it cannot ground its answer.

Two safety layers run on top: a version filter that drops superseded-policy nodes unless the query asks for historical context, and an audit log capturing the full navigation trace per query. 

The tree-navigation loop in pseudocode. Every hop is logged with the branch chosen or pruned; the LLM has visibility into both the path and node metadata when deciding.

Why this works – Each navigation step is a small, reasoned decision over a handful of summaries – the kind of judgment LLMs handle well. Because the path and metadata are visible at every hop, the model can reject superseded nodes and prefer the right jurisdiction without the user spelling it out. The audit trace turns retrieval from a black-box similarity score into a sequence of decisions a domain expert can review.

Vector RAG vs Vectorless RAG: How They Compare  

The clearest way to see when vectorless retrieval pays off is dimension by dimension. The shape of your corpus and queries should drive the choice. 

Vector and tree-based retrieval optimize for different things. Match the retrieval method to the shape of the data.

How Did the New Architecture Perform in Production?  

Migration was staged. Both systems ran in shadow mode for three weeks, sampling 1,800 real queries rated by domain experts on accuracy, grounding, and version correctness. 

Shadow-mode comparison over 1,800 real queries. The accuracy gap on multi-hop questions is where the vectorless architecture earned its keep.

On single-hop questions, the two systems were tied at around 82% – vectors handle simple lookups well. On multi-hop and cross-section questions, vector RAG scored 51% while vectorless reached 79%. Multi-hop reasoning is where real enterprise questions exist. As a result, version contamination dropped from 14% to under 1%, while user trust – measured by “Would you act on this answer without re-reading the source?” – increased from 34% to 81% within eight weeks.

When Should You Choose Vector, Hybrid, or Tree-Based Retrieval?  

We do not recommend this for every team. The right starting point for most production systems is a hybrid – keep the vector store, add a structural index alongside, route queries to whichever fits. Collapse to fully vectorless only when the hybrid stops adding value. 

A practical decision framework. Most teams should start hybrid; full vectorless retrieval is the right call for tens of thousands of structured documents where authority and audit matter more than sub-second latency.

The Tradeoffs We Accepted  

The architecture is not free. Three concessions to ship it:  

• Latency rose. Median query time went from 380msto 2.4 seconds – acceptable for an internal tool used dozens of times per user per day; not for a consumer-facing product.  
• Cost per query rose. From 0.4 cents to 3.1 cents at first, almost entirely in additional LLM tokens. A smaller navigation model and aggressive summary caching brought sustained cost to 2.2 cents – still 5× the vector baseline.  
• Engineering complexity rose. Tree construction, summary regeneration on document updates, and audit-trace storage added operational surface – many of the same patterns we describe in our multi-agent AI scaling writeup. We invested in tooling early – a navigation replay viewer for inspecting failed queries.

Frequently Asked Questions  

Q. What is vectorless RAG?  

Vectorless RAG is a retrieval architecture that replaces vector similarity with LLM reasoning over a structured document tree. Each node carries a summary and metadata; the model navigates the tree to find grounded, auditable answers – no embeddings required.  

Q. Is vectorless RAG better than vector RAG?  

Vectorless RAG outperforms vector RAG on multi-hop, structurally complex enterprise queries – accuracy rose from 51% to 79% in our 1,800-query benchmark. On single-hop semantic lookups and web-scale retrieval, vector RAG remains the better fit.  

Q. How does PageIndex-style vectorless retrieval work?  

PageIndex-style vectorless RAG parses each document into a hierarchical tree where every node holds a short summary plus metadata. The LLM reads the root, picks a branch, and descends until it reaches a leaf – no embeddings or vector database.  

Q. What are the tradeoffs of vectorless RAG?  

The architecture raises query latency (2-4 seconds versus 200-500 ms for vector RAG) and increases cost per query 5-10× from extra LLM tokens. In return: auditable retrieval traces, near-zero version contamination, and stronger multi-hop accuracy.  

Q. Does vectorless RAG scale to large corpora?  

Vectorless RAG scales to tens of thousands of documents when paired with a domain router that narrows candidate trees per query. Beyond a few hundred thousand documents, hybrid retrieval (vector index plus tree index) usually performs better.  

Q. When should I migrate from vector RAG to vectorless RAG?  

Migrate when your corpus is structurally rich (policies, contracts, regulations, technical specs), version and authority semantics matter, and users need to audit retrieval traces. For single-hop semantic search, stay with vector RAG.