Innhold Show
If storage is stopping you from running a large RAG system locally, the LEANN RAG index attacks the most expensive part: the vector index itself. In the project’s own benchmark, an index containing 60 million text chunks shrinks from 201 GB to 6 GB.
The idea is easy to explain and technically rather cheeky. LEANN does not keep every embedding after building the index. It retains a compact, pruned graph instead, then recomputes only the embeddings needed while a query is running.
You do not get that enormous storage saving for free. Search moves from hundredths of a second to seconds in the researchers’ tests. For local RAG, personal knowledge bases, and large document collections, that can still be a very sensible trade: waiting a little is better than having an index that does not fit on the machine at all.
What is LEANN?
LEANN is an open-source approximate nearest neighbor index designed to reduce the storage overhead of vector search and RAG. The MLSys 2026 paper reports indexes up to 50 times smaller than conventional alternatives while preserving comparable search accuracy.
Conventional vector indexes store one embedding for every text chunk, along with the metadata needed for fast neighbor searches. As the dataset grows, the index can become much larger than the material it represents. The researchers used 76 GB of Wikipedia text divided into 60 million chunks of 256 tokens. The embeddings alone occupied 173 GB.
LEANN begins by building a graph-based index much like other systems. It then discards the dense embedding vectors, prunes the graph, and stores the compact structure. The original text and embedding model must remain available, but the completed vector index no longer has to drag every vector around with it.
From 201 GB to 6 GB – but those numbers come from two comparisons
The project’s GitHub repository contains the most striking comparison: 60 million text chunks require 201 GB in a conventional vector database and 6 GB with LEANN. That is a reported saving of 97%. The repository also shows an index of 400,000 chat messages falling from 1.8 GB to 64 MB.
The paper uses a slightly different setup. It compares a 4 GB LEANN index with a 188 GB HNSW index for the same Wikipedia corpus. The HNSW figure consists of about 173 GB of embeddings plus a graph of roughly 15 GB. Both comparisons document the same main point, but 201 versus 6 GB and 188 versus 4 GB must not be blended into a single benchmark.
These are results reported by the researchers and the project, not my own test or an independent reproduction. The numbers are still concrete enough to show why the design matters. When a conventional index needs more than twice the storage of the 76 GB source corpus, the index itself has become the problem.
How can LEANN search without storing every embedding?
LEANN exploits the fact that a graph-based vector search visits only a small part of the full dataset for each query. Instead of loading millions of stored vectors from disk, it recomputes embeddings for the nodes along the relevant search path. In plain English, it moves the cost from storage to computation.
A lightweight product-quantized representation first ranks possible neighbors. Only the most promising candidates proceed to exact recomputation. LEANN also collects candidates from multiple search steps into dynamic batches, giving the GPU a meaningful amount of work in each run instead of feeding it one tiny request at a time.
The graph is pruned as well. Rather than cutting connections uniformly, LEANN preserves more edges around the best-connected points, which the paper calls hub nodes. The experiments found a useful balance when the top 3-5% of nodes retained a higher degree. This allows the graph to be cut roughly in half without destroying its most important shortcuts through the dataset.
The price is search latency measured in seconds
On Natural Questions at 90% Recall@3, LEANN took 2.48 seconds for retrieval while HNSW took 0.05 seconds. Both reached 25.5% downstream accuracy with Qwen3-4B in the reported test, but LEANN used 4 GB of storage rather than HNSW’s 188 GB. This is not free compression. It is a very clear exchange of speed for space.
The difference varies by task. On an RTX 4090, the researchers reported retrieval times ranging from 1.12 seconds on GPQA to 7.12 seconds on HotpotQA. Embedding recomputation accounted for about 76% of query latency, which means a faster embedding model or a better cache could make a substantial difference.
The researchers argue that this delay can be acceptable when the next stage is text or image generation that already takes many seconds. That may be true for a personal document assistant, but not necessarily for autocomplete, recommendations, or a service handling many concurrent users. Your latency requirement decides whether 97% less index storage is worth the price.
Where does the LEANN RAG index fit?
For local RAG, the practical gain is that a large semantic index can fit on an ordinary workstation. On the researchers’ RTX 4090 workstation with 32 GB of system memory, HNSW and IVF ran out of memory and had to be measured on a server with more RAM. LEANN fit because its index used 4 GB and did not need to keep every embedding available.
That opens the door to semantic search across document archives, email, chat history, browsing history, and code without sending the content to an external vector database. The repository includes prepared examples for several of these sources and supports Ollama, Hugging Face, and OpenAI-compatible APIs for the generation stage. LEANN is the search index, not the language model that writes the answer.
This distinction matters for privacy. A local LEANN index does not magically make the entire application local if the embedding or generation model still runs through an external service. But with the source data, embedding model, index, and language model all on your own machine, the full retrieval workflow can remain local.
LEANN does not solve the entire RAG stack, either. You still need ingestion, chunking, an embedding model, retrieval logic, and a model that uses the retrieved material well. What LEANN does is remove one of the most concrete barriers: the absurd amount of permanent storage a large vector index can consume.
What do you need to run LEANN?
LEANN is released under the MIT license and can be installed from PyPI with uv pip install leann. The repository provides build instructions for macOS, several Linux variants, Windows, and WSL. HNSW is the default backend, while DiskANN is available when a disk-oriented search workflow is a better fit.
You still need the raw data and the embedding model used to build the index. LEANN recomputes candidate embeddings with that same model during search. If you switch embedding models, the existing graph is not automatically a neutral shell that will work correctly with the replacement.
The project offers a CPU installation on Linux, but the published material does not provide equivalent CPU latency figures for the large benchmark. The most prominent measurements use an RTX 4090 workstation with 32 GB of system memory and an M1 Ultra Mac. Do not treat GPU results as a promise of identical response times on a small home server without acceleration.
Index construction also requires every embedding to be produced at least once before it can be discarded. LEANN reduces permanent storage, but it does not make the initial build free. With 60 million text chunks, you still have to plan for processing time, temporary capacity, and access to the model. The saving appears when the completed index needs to be stored, transferred, updated, and used over time.
When is LEANN a better choice than HNSW?
LEANN is best suited to cases where storage or memory is the hard limit, the dataset is large, and a few seconds of retrieval time is acceptable. Personal knowledge bases, local document archives, and RAG on a single workstation are obvious candidates. In those situations, the alternative may be to throw away much of the dataset or move the whole index into the cloud.
HNSW remains the better starting point when you need millisecond search, have sufficient memory, and must serve many requests at once. LEANN does not repeal physics. Every query creates fresh work for the embedding model, and that load grows when several users search concurrently.
I would therefore judge LEANN by the bottleneck you actually have, not by one impressive percentage. Measure how large a conventional index becomes, decide how much waiting your workflow can tolerate, and determine whether the machine can recompute embeddings quickly enough. Then run the project’s evaluation setup on your own data before making a production decision.
A smart direction for personal AI
The most promising part of LEANN is not merely that one benchmark produced a huge compression figure. It is the way the tool reverses an old design assumption: embeddings do not always need to sit fully computed on disk just because they are expensive to create.
When storage is cheap and latency is everything, keeping the vectors makes sense. When a personal machine needs to search tens of millions of text chunks, the calculation can flip. Using the GPU for a few seconds instead of demanding nearly 200 GB for the index is an elegant option.
LEANN is not an automatic replacement for HNSW, and the project’s numbers need to be reproduced in your own workflow. But for local RAG, it solves a real problem with a clean, understandable trade: dramatically less storage in exchange for noticeably slower retrieval. That is a trade many people can genuinely live with.
Frequently asked questions
Is LEANN a vector database?
LEANN is primarily a storage-efficient vector index for semantic search and RAG. Its repository wraps the index in ready-made applications and integrations, but it does not automatically replace every feature of a complete managed vector database.
Does LEANN require a GPU?
Not necessarily. The project provides a CPU installation for Linux, but the major published performance figures include tests on an RTX 4090. Without a GPU, expect different retrieval latency and measure it on your own hardware before choosing LEANN.
Can LEANN run completely locally?
Yes. The index, embedding model, source data, and generation model can all run locally, including with Ollama or Hugging Face. If you choose an external OpenAI-compatible service for embeddings or generation, the overall workflow is no longer completely local.
Why is LEANN slower than HNSW?
LEANN saves space by not storing every fully computed embedding. During a search, it must therefore recompute embeddings for the candidates it visits. HNSW can read stored vectors directly and search much faster, but requires far more storage and memory.
