Measuring hallucinations in document Q&A: methods that hold up

You've put a question-answering tool over your company's documents. Staff ask it about leave policy, contract terms or a product spec, and it answers with a confident paragraph and a couple of citations. Most of the answers look right. The question you can't answer yet is how often they aren't, and whether the wrong ones are the kind anyone would notice.

That's a measurement problem, and it's more tractable than it sounds. The research community has spent the last few years building methods and datasets for exactly this, and most of the ideas carry over to a small internal system. This piece covers what to count, how to count it automatically, where the published benchmarks help, and where you still need a person reading answers.

Decide what "hallucination" means for your system#

The word gets used loosely, so pin it down first. In document Q&A the useful question isn't "is this true about the world?" but "does this follow from the documents the system was given?" An answer can be factually correct and still be a problem if it came from the model's training data instead of your current policy document.

The RAGTruth corpus is a good place to borrow a vocabulary. Its authors annotated responses from six models across question answering, summarisation and data-to-text tasks, and split hallucinations into four types: evident conflict (the answer contradicts the source), subtle conflict (it drifts from the source), evident introduction of baseless information (it adds things the source doesn't support) and subtle introduction of baseless information (it adds inferred details or sentiment that go beyond the source). That last category is the one people underrate. "The contract can probably be extended" when the contract says nothing about extension is a hallucination, even though it sounds like reasonable advice.

A Stanford study of commercial legal research tools, published in the Journal of Empirical Legal Studies, adds a second axis that matters for anything with citations. Magesh and colleagues grade correctness and groundedness separately. A response is "misgrounded" if key claims are cited "but misinterpret the source", and a response counts as hallucinated if it's either incorrect or misgrounded. Refusals are tracked as their own outcome, not as errors. For a document assistant, that gives you three buckets worth keeping apart: wrong, right but badly sourced, and declined.

Write your definitions down with a couple of examples from your own documents before you build anything. Your graders, human or automated, will need them.

Count claims, not answers#

Grading a whole answer as pass or fail throws away information. A five-sentence answer with one invented detail is different from one that's invented end to end, and you want to see both.

The Ragas paper by Es and colleagues describes a straightforward way to do this. An LLM first breaks the answer into individual statements. Each statement is then checked against the retrieved context, and the faithfulness score is the number of supported statements divided by the total. An answer is faithful when "the claims that are made in the answer can be inferred from the context."

The authors tested this against human judgements on their WikiEval set. In pairwise comparisons, the faithfulness metric agreed with annotators 0.95 of the time, against 0.72 for a GPT-based scoring baseline and 0.54 for a GPT ranking approach. That's encouraging, but it's their dataset and their annotators. Treat it as a reason to try the method, then check it on your own data.

Two practical notes. The decomposition step is itself an LLM call, so it can split a sentence badly or miss a buried claim. Log the extracted statements and read a sample. And faithfulness only tells you whether the answer sticks to the retrieved context. If retrieval pulled the wrong document, a perfectly faithful answer can still be wrong, which is why Ragas also scores context relevance separately. Keep the retrieval and generation scores apart so you know which half to fix.

Check the citations separately#

If your system shows sources, users will trust the answer more because of them. That makes it worth testing whether the citations actually support what they're attached to.

The ALCE benchmark from Gao, Yen, Yu and Chen splits this into two numbers. Citation recall is scored per statement: a statement gets 1 if it has at least one citation and the cited passages, taken together, entail it. Citation precision is scored per citation: a citation counts against you if it can't support the statement on its own and removing it doesn't change whether the statement is supported. Low recall means claims without real backing. Low precision means padding with sources that look good but don't help.

ALCE checks entailment with an off-the-shelf natural language inference model (TRUE, a fine-tuned T5-11B), not a human. The authors point out its weak spot: it can't detect a passage that only partly supports a claim. Their headline finding is still worth knowing. On the ELI5 dataset, around 50% of the ChatGPT and GPT-4 baseline generations weren't fully supported by the passages they cited. Citations on screen are not evidence that the citation checking has been done.

You can apply the same two measures to your own system with whatever entailment checker you trust, whether that's an NLI model or an LLM judge, as long as you've compared it with human grading first.

Use published benchmarks for choosing a model#

Public numbers are useful for one decision: which model to put in the generation step. They won't tell you how your whole system behaves.

One public example is Vectara's hallucination leaderboard. It gives each model the same instruction to summarise a passage using only its content, at temperature 0 where possible, then scores each summary with Vectara's evaluation model (HHEM-2.3, a commercial model; an open variant, HHEM-2.1-Open, is on Hugging Face). The dataset is kept private to discourage overfitting and, per the repository, holds over 7,700 articles across news, science, medicine, legal, business and other subjects. It reports a hallucination rate, an answer rate and average summary length, so a model can't look good by refusing or writing three-word summaries.

Read the caveats the maintainers publish alongside it. They say it "does not definitively measure all the ways a model can hallucinate", that it doesn't judge summary quality, and that copying the source verbatim would score well, so it shouldn't be used alone. Summarisation is a sensible proxy for document Q&A, but your documents, your retrieval and your prompts are different. A model's leaderboard rank is a shortlist, not a result.

RAGTruth is useful in a different way: as training and test data for a hallucination detector. Its paper reports that a Llama-2-13B model fine-tuned on the corpus reached an average response-level F1 of 80.7% at spotting hallucinated responses, against 68.3% for GPT-4-turbo used with a prompt. The general lesson is that a small, purpose-trained checker can beat a large general model prompted to "find errors", which is relevant if you're paying per token to run a judge on every answer.

Keep a hand-graded set, and expect it to be humbling#

Automated metrics let you test every change. They don't replace a set of questions that people have answered and graded, because that's the only thing you can calibrate the automated scores against.

The legal research study shows what this looks like done properly, and why it matters. The team wrote 202 queries in four groups, including questions with a false premise built in, and hand-scored each response using legal expertise. A fourth labeller re-graded a random 48 responses and got a Cohen's kappa of 0.77 against the original grades, which is how you show your grading is consistent and not one person's opinion. Their result: the tools from LexisNexis and Thomson Reuters "each hallucinate between 17% and 33% of the time", despite provider claims that retrieval avoids or eliminates hallucinations, or that citations are "hallucination-free". The authors conclude that "the providers' claims are overstated."

Those were well-funded products built on retrieval. Your internal tool isn't likely to do better without measurement. For a small business, a hundred real questions from the people who'll use the system is a reasonable start. Include questions whose answer isn't in the documents at all, and questions with a wrong assumption baked in ("what's the penalty in clause 14?" when there's no clause 14). Those two groups find a lot of the invented answers, and they're the ones a polite demo never includes. The broader process of building that test set is in our guide to evaluating an AI automation.

Putting it into a routine#

A workable setup has three layers. A hand-graded set of real questions, labelled with the wrong / misgrounded / declined buckets, that you re-run whenever you change the model, the prompt or the retrieval. A claim-level faithfulness score and citation recall and precision, computed automatically on that set and on a sample of live traffic. And a regular check that the automated scores still agree with what your graders would say.

Report the numbers by question type, not as one average. "Faithful on policy lookups, weak on questions that need two documents combined" tells you where to work. Watch the refusal rate too. If a change cuts hallucinations by making the system decline half its questions, that's a trade-off someone needs to sign off on, and a measured refusal is usually better than a confident guess. It's the setup we'd suggest at TheAILAB before a document assistant goes in front of staff.

Start with the hand-graded set this week. Everything else is easier to justify once you have twenty answers you know are wrong.

Sources

1. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models — Wu, Zhu, Xu, Shum, Niu, Zhong, Song and Zhang (arXiv, 2023) (checked 2026-10-17)
2. Ragas: Automated Evaluation of Retrieval Augmented Generation — Es, James, Espinosa-Anke and Schockaert (arXiv, 2023) (checked 2026-10-17)
3. Enabling Large Language Models to Generate Text with Citations — Gao, Yen, Yu and Chen (arXiv, 2023) (checked 2026-10-17)
4. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools — Magesh, Surani, Dahl, Suzgun, Manning and Ho (Journal of Empirical Legal Studies, 2025) (checked 2026-10-17)
5. Hallucination Leaderboard — Vectara (GitHub repository) (checked 2026-10-17)