Local RAG: running retrieval-augmented generation on your own machines

You want a model to answer questions about your own documents: contracts, SOPs, support tickets, engineering notes. You also don't want those documents sent to someone else's API. The usual answer is retrieval-augmented generation (RAG), and every piece of it can run on hardware you control. The hard part isn't getting a demo working in an afternoon. It's getting answers you'd trust once the document pile is large and messy.

This guide walks through each stage of a local RAG pipeline, the decisions that matter at each one, and the places where self-hosted setups quietly go wrong.

What RAG is, and why it suits local models#

The term comes from a 2020 paper by Patrick Lewis and colleagues. They paired a generator model with a separate, searchable index of documents, and argued that models which only store knowledge in their weights struggle with two things: showing where an answer came from, and being updated. Their abstract describes "providing provenance for their decisions and updating their world knowledge" as open problems. In the full paper they showed they could update the system's knowledge by swapping out the index, with no retraining.

That second property is why RAG fits local deployments so well. A 7B or 14B model running on one GPU knows nothing about your business and you probably don't want to fine-tune it every time a policy changes. With retrieval, the model only has to read and summarise what you hand it. Updating its knowledge means re-indexing a folder.

A local pipeline has five stages:

1. Load and split documents into chunks.
2. Turn each chunk into an embedding (a list of numbers that captures its meaning).
3. Store the chunks and embeddings somewhere searchable.
4. At question time, embed the question, find the closest chunks, and optionally combine that with keyword search.
5. Put the best chunks into a prompt and have a local model answer from them, with citations.

If you haven't picked a runtime yet, our comparison of Ollama, llama.cpp and vLLM covers that choice. The rest of this assumes you have one.

Embeddings: small model, big consequences#

The embedding model decides what "similar" means for your whole system, so it's worth more attention than it usually gets.

Ollama exposes them through a single endpoint. Its API documentation for /api/embed shows it takes a model name and either one string or an array of strings, and returns an array of vectors. One parameter deserves attention: truncate defaults to true, which means inputs longer than the model's context window are cut off rather than rejected. If your chunks are too long, the end of each one is silently dropped from its embedding and you'll never see an error. Set truncate to false while you're tuning chunk sizes so oversized chunks fail loudly.

The second trap is task prefixes. Some embedding models are trained to embed questions and documents slightly differently. The model card for nomic-embed-text-v1.5 says the text "must include a task instruction prefix": documents go in as search_document: <text> and user questions as search_query: <text>. A hosted API might add these for you. When you call the model directly through a local runtime, you have to add them yourself. Forget, and retrieval still works, just worse, which is the most dangerous kind of failure because nothing looks broken.

The same model supports shorter vectors if you need to save space. Nomic's documentation lists recommended output sizes of 768, 512, 256, 128 and 64 dimensions, with the warning that quality drops as you go smaller. For most small-business document sets, storage isn't the constraint, so stay at full size unless you've measured a reason not to.

Whatever you choose, treat the embedding model and its settings as part of your index. Change the model, the prefix or the dimension count, and every stored vector has to be regenerated.

Chunking: the step nobody wants to tune#

Chunks that are too big blur several topics into one vector and waste the generator's context. Chunks that are too small lose the sentence that explains the number. There's no universal right size, but a few habits help.

Split on the document's own structure first (headings, sections, numbered clauses) and only fall back to fixed lengths inside long sections. Keep a little overlap between neighbouring chunks so a sentence split across a boundary still appears whole somewhere. Store metadata with every chunk: the source file, page or section, date and, most importantly for later, who is allowed to see it. And keep chunks well inside the embedding model's context window, for the truncation reason above.

Storing and searching vectors locally#

You don't need a dedicated vector database to start. If you already run Postgres, the pgvector extension adds a vector column type and similarity operators. Its documentation (version 0.8.7 as of October 2026) makes a point that surprises people: by default pgvector performs exact nearest-neighbour search, which gives perfect recall. Adding an index switches to approximate search, which trades some recall for speed, and the docs warn that you will see different results for queries after adding one.

It offers two index types. HNSW has a better speed-to-recall trade-off but builds slower and uses more memory. IVFFlat builds faster and uses less memory but gives lower query performance for the same recall. For a few tens of thousands of chunks, exact search is often fast enough, so measure before adding either. Also note the HNSW limit of 2,000 dimensions for the standard vector type, which matters if you pick a large embedding model.

One subtle behaviour matters for permissions. The pgvector docs say that with approximate indexes, filtering is applied after the index is scanned. If you ask for the top five chunks the user is allowed to see, the index may find five nearest chunks, the filter may remove four, and you get one result back. pgvector also offers iterative index scans, which automatically scan more of the index until enough results are found. Turn that on, or test filtered queries specifically.

If you'd rather keep everything in a single file, SQLite is a reasonable home for the chunks, metadata and keyword index, and our article on SQLite in production covers when that holds up.

Add keyword search before you add complexity#

Embeddings are good at meaning and bad at exact strings. Ask about invoice INV-20931, clause 14.2(b) or a part number, and vector search may return chunks that are about invoices in general. Keyword search finds the exact token immediately.

You probably already have a keyword engine. SQLite's FTS5 extension is a virtual table module for full-text search, with a built-in bm25() ranking function. One quirk: FTS5 returns BM25 scores where a better match is numerically smaller, so you sort ascending. Its porter tokenizer adds stemming, so "invoices" matches "invoice". On Postgres, the pgvector docs suggest combining vector results with Postgres full-text search for hybrid search, and merging the two ranked lists with Reciprocal Rank Fusion or a cross-encoder.

Reciprocal Rank Fusion is simple enough to write yourself. As the PyTerrier documentation describes it, each document gets 1/(rank + k) from every ranking it appears in, those values are summed, and the sum decides the final order. PyTerrier uses k = 60 by default, and the method comes from a 2009 SIGIR paper by Gordon Cormack, Charles Clarke and Stefan Büttcher. Because it only looks at ranks, it needs no score normalisation, which is why it's a sensible default for merging BM25 and cosine results whose scales have nothing in common. A cross-encoder reranker is a further step: a small model that reads the question and each candidate chunk together and rescores them. Because it runs a model over every question-and-chunk pair, it's far slower than a vector lookup, so run it on a shortlist such as the top 20 to 50 candidates.

Prompting the local model with what you found#

Once you have candidates, resist the urge to stuff the context window. The Lost in the Middle study by Nelson Liu and colleagues tested models on multi-document question answering and found that "performance is often highest when relevant information occurs at the beginning or end of the input context", and degrades when it's in the middle. That held even for models built for long contexts.

The practical response is to send fewer, better chunks, and put the strongest ones first. Tell the model to answer only from the supplied passages, to quote or cite the chunk IDs it used, and to say plainly when the passages don't contain the answer. Then check that the model actually does it. Some models will fill gaps from their own training rather than admit the passages are silent, so a few test questions with no answer in your documents belong in your evaluation set. Our guide on evaluating an AI automation covers building that set.

Show citations to the user with a link to the source document. It's the provenance benefit the original paper was after, and it lets people catch a wrong answer in seconds.

Local doesn't mean secure#

Keeping data on your machines removes one risk: a third-party provider seeing it. It leaves the rest. OWASP's LLM Top 10 has an entry for exactly this, LLM08:2025 Vector and Embedding Weaknesses, and nothing in it stops applying because the servers are yours.

The first is access control. If one index holds HR files and the general wiki, any user who can ask questions can potentially retrieve HR content. OWASP recommends permission-aware vector stores and partitioning data so different groups can't reach each other's information. Enforce permissions at retrieval time using the chunk metadata, and don't rely on the prompt to keep secrets.

The second is that embeddings aren't anonymised data. OWASP notes that attackers can invert embeddings and recover significant amounts of the source information, so protect the vector store as carefully as the documents themselves.

The third is poisoning. Anything that lands in the indexed folder becomes something the model may repeat as fact, and OWASP specifically mentions hidden content such as white-on-white text. Indexed documents are also a route for prompt injection, so a RAG system that can take actions needs the same guardrails as any other agent. Only index sources you trust, and log retrievals so you can trace a strange answer back to the chunk that caused it.

A sensible first build#

Start small and measurable. This is roughly how we approach a first build at TheAILAB. Pick one document collection with a clear owner, around 30 real questions people ask about it, and the answers they'd accept. Build the plainest version that could work: structure-aware chunking, one embedding model with the correct prefixes and truncate off, exact vector search plus BM25 merged with Reciprocal Rank Fusion, and five chunks into the prompt with citations required. Score it against your 30 questions, including a few that have no answer.

Then change one thing at a time (chunk size, a reranker, a different embedding model) and re-run the same questions. A retrieval miss looks exactly like a model mistake if you only grade the final answer, so measure retrieval on its own: for each test question, check whether the right chunk was in the top five before you judge what the model wrote.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. (arXiv 2005.11401, v4 April 2021) (checked 2026-10-16)
2. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (full paper PDF) — Patrick Lewis et al. (arXiv 2005.11401) (checked 2026-10-16)
3. Generate embeddings (POST /api/embed) — Ollama API documentation (checked 2026-10-16)
4. nomic-embed-text-v1.5 model card — Nomic AI, Hugging Face (checked 2026-10-16)
5. Text Embedding — Nomic AI documentation (checked 2026-10-16)
6. pgvector: open-source vector similarity search for Postgres (vector 0.8.7) — pgvector project, PGXN (checked 2026-10-16)
7. SQLite FTS5 Extension — SQLite documentation (checked 2026-10-16)
8. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt et al. (arXiv 2307.03172, v3 November 2023) (checked 2026-10-16)
9. LLM08:2025 Vector and Embedding Weaknesses — OWASP Gen AI Security Project (checked 2026-10-16)
10. Result fusion (RRFusion) — PyTerrier documentation, citing Cormack, Clarke and Büttcher (SIGIR 2009) (checked 2026-10-16)