Retrieval-augmented generation demos beautifully. You point it at a folder of PDFs, ask a question, get a fluent answer, and everyone in the room is convinced.
Then it goes in front of real users, and someone asks a question the corpus cannot answer. The model answers anyway — confidently, in the same tone as every correct answer it has given. That is the failure mode that matters, because a system that is wrong in a recognisable way is merely broken, while a system that is wrong in an unrecognisable way is dangerous.
The defence is not a better model. It is making every claim traceable to a source the user can open. Once an answer carries citations, a wrong answer becomes a checkable answer.
The pipeline, in the order it actually runs
documents -> parse -> chunk -> embed -> index
|
question -> embed -> retrieve -----------> |
|
rerank
|
prompt with numbered context
|
answer + citation ids
|
resolve ids back to sourcesMost of the quality lives in the boring middle — chunking and retrieval. The prompt is the part people tune first and the part that matters least.
Chunking decides your ceiling
Splitting every 500 characters is the default in every tutorial and it is the single biggest cause of bad answers. It cuts tables in half, separates a heading from the paragraph it governs, and strands the sentence that contained the actual answer in a chunk with no context.
Chunk on structure first, size second:
def chunk_document(doc, target=900, overlap=150):
"""Split on headings, then pack sections up to a target size.
A chunk keeps its heading trail, so a fragment retrieved from deep in a
document still says what it is part of.
"""
chunks = []
for section in split_on_headings(doc):
trail = " > ".join(section.heading_path)
body = section.text
if len(body) <= target:
chunks.append(make_chunk(trail, body, section))
continue
# long section: window it, but never mid-sentence
for piece in window_on_sentences(body, target, overlap):
chunks.append(make_chunk(trail, piece, section))
return chunks
def make_chunk(trail, body, section):
return {
# the heading trail is prepended to the embedded text:
# "Policies > Leave > Carry-forward" is strong signal
"text": f"{trail}\n\n{body}",
"doc_id": section.doc_id,
"page": section.page,
"heading": trail,
}The overlap exists so that a fact sitting on a boundary appears whole in at least one chunk. The heading trail exists because embeddings of a bare paragraph lose the context that makes it findable.
Vector search alone is not enough
Embeddings are good at meaning and bad at exact tokens. Ask for invoice
INV-2291 and a pure vector search will cheerfully return invoices
INV-2290 and INV-3117, because they are semantically almost identical.
Run keyword and vector search together and merge the rankings. Reciprocal rank fusion is about ten lines and needs no tuning:
def reciprocal_rank_fusion(rankings, k=60):
"""Merge several ranked lists. k damps the influence of top positions."""
scores = {}
for ranking in rankings:
for position, chunk_id in enumerate(ranking):
scores[chunk_id] = scores.get(chunk_id, 0) + 1.0 / (k + position + 1)
return sorted(scores, key=scores.get, reverse=True)
def retrieve(question, limit=24):
dense = vector_search(embed(question), limit=limit) # meaning
sparse = keyword_search(question, limit=limit) # exact tokens
return reciprocal_rank_fusion([dense, sparse])[:limit]Retrieve generously — twenty or so — then rerank down to the four or five you actually put in the prompt. A cross-encoder reranker reads the question and the chunk together rather than comparing two independent vectors, and it is consistently the highest-return component to add after hybrid search.
Making citations structural, not requested
Asking the model nicely to "include sources" produces plausible-looking citations that sometimes point at the wrong document. The reliable approach is to number the context and require the answer to reference those numbers, then resolve the numbers yourself:
SYSTEM = """Answer using only the numbered sources below.
Cite the source for every claim, as [1], [2]. A sentence with no citation
is not allowed.
If the sources do not contain the answer, reply exactly:
"I could not find this in the available documents."
Do not use knowledge from outside the sources."""
def build_prompt(question, chunks):
context = "\n\n".join(
f"[{i + 1}] ({c['heading']})\n{c['text']}"
for i, c in enumerate(chunks)
)
return [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"{context}\n\nQuestion: {question}"},
]
def resolve_citations(answer, chunks):
"""Map [n] back to real documents, and drop any the model invented."""
used = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
return [
{"n": n, "doc_id": chunks[n - 1]["doc_id"], "page": chunks[n - 1]["page"]}
for n in sorted(used)
if 1 <= n <= len(chunks)
]The bounds check in resolve_citations is not defensive padding. Models do emit
[7] when you gave them five sources, and without that check you will render a
citation that crashes or, worse, points at the wrong file.
The refusal path is a feature
Make "I could not find this" a first-class outcome and put it in the evaluation set. A retrieval system that never refuses is not confident, it is unfalsifiable.
A cheap and effective guard: if the top reranked score falls below a threshold, do not call the model at all. Return the refusal directly. It saves tokens and it removes the opportunity to hallucinate.
Evaluate on questions, not on vibes
You need a fixed set of question–answer pairs drawn from the real corpus, including questions that are deliberately unanswerable. Then track three numbers on every change:
- Retrieval hit rate — is the chunk containing the answer in the top k? If this is low, nothing downstream can save you, and it is measurable without an LLM in the loop at all.
- Citation accuracy — does each cited source actually support the claim?
- Refusal correctness — does it refuse on the unanswerable set, and only there?
Without this, every prompt change is a guess, and you will spend weeks moving quality sideways while being certain it is improving.
What it costs to run
Two things dominate, and both are controllable. Embeddings are a one-off per chunk — re-embed only what changed, keyed by a content hash. Generation is per-query and scales with how much context you stuff in, which is the real argument for reranking down to five chunks instead of passing twenty: it is cheaper and more accurate, because a model given twenty sources reliably loses the ones in the middle.