A RAG assistant that knows when to say no

Ask a language model how to get a sick-leave certificate from the IMSS (Instituto Mexicano del Seguro Social, the Mexican public health institute) and you’ll get a confident answer. It may even be right. But for paperwork, “probably right” is not good enough. A wrong deadline or a missing form means a patient loses days of pay.

So I built a proof of concept for a prospective client: a chat assistant (the technique is called RAG, retrieval-augmented generation) that answers insurance and health paperwork questions only from a library of official documents, shows where every answer came from, and says “I don’t have that” when it doesn’t. The interesting part wasn’t getting it to answer. It was measuring whether it answers well, and finding out which of my own design choices actually mattered.

There are no real users and no patient data. The documents are public (IMSS, ISSSTE (Instituto de Seguridad y Servicios Sociales de los Trabajadores del Estado), CONAMED (Comisión Nacional de Arbitraje Médico), three insurers), plus one fictional clinic FAQ (frequently asked questions) I wrote for demos. The assistant is in Spanish, because that’s who it’s for.

At a glance

My role: architecture, retrieval pipeline, evaluation, backend, chat UI (user interface) and deployment.
Scope: 8 documents, 531 searchable passages, a streaming chat with source cards, and a test set of 63 hand-written questions.
How: an agent with one tool, “search the documents”. Everything it says has to come back from that tool.
New to the terms? There is a glossary at the end, covering the tools and the search and evaluation ideas.
Code: the repo is public at jessaimaya/rag-doc. The sections below link to the files that implement each part.
What I’d point to: the evals. Each design decision below has a number behind it.

The assistant in action

Watch three things in this clip. The answer streams in and comes with the exact document, section and page it used. When I ask a follow-up, it searches again instead of guessing. And when I ask “my chest hurts, what disease do I have?”, it refuses to diagnose and points to emergency services, because it’s a paperwork assistant, not a doctor.

How it works

There are two paths. The top one runs once, whenever documents change. The bottom one runs on every question.

Getting documents in. A manifest file lists each document with its title, category and source link. Ingestion reads PDF (Portable Document Format), Word, HTML (HyperText Markup Language), Markdown or text, cuts it into passages, and stores each one with its section path and page number. Re-running is safe: unchanged files are skipped.

The cutting matters more than people expect. I split first by structure (headings, “Artículo 12”, numbered sections, tables kept whole), then by meaning: every sentence gets an embedding, and where the topic shifts, there’s a natural place to cut. The chunker does both, and its semantic breakpoints are the second step. Passages are 400 to 1400 characters and never cross a heading. Each one is embedded together with its breadcrumb, like IMSS procedure › 5. Policies › 5.2 Specifics, so a passage knows what document and section it came from.

Finding the right passages. Two searches run side by side, wired together in retriever.ts. A vector search finds passages that mean something similar to the question, even with different words. A keyword search (BM25, Best Matching 25) finds passages that share rare words, like a form code. Their rankings are merged with Reciprocal Rank Fusion, which only looks at positions, so the two score scales never have to be compared. The best 30 go to a reranker, which reads the question and each passage together and picks the best 5.

Answering. Claude Haiku gets those passages and writes the answer. If the best passage is not relevant enough, the tool tells the model nothing relevant was found, and the model says so.

One question, step by step

Here is one question from start to finish, with the function that does each step. Steps 2 to 6 are a single function, hybridSearch, and step 1 is the tool wrapper in tool.ts. The right column shows which function measures each stage in the evals, so every step has a number attached to it.

The first line under each step is the function. The second is what comes out of it. Steps 3 to 6 are the ones the retrieval evals can switch on and off, which is how I could compare setups later.

Which retrieval setup is best? I measured it

I could have copied a popular recipe (“hybrid search plus a reranker”) and moved on. Instead I wrote 57 questions in Spanish over the indexed documents and checked six setups: vector alone, keyword alone, and the two fused, each with and without the reranker.

The questions are tagged by what they test. 28 are paraphrases with little word overlap (“I arrived 10 minutes late, will they still see me?”), 17 share rare words with the answer, and 5 are exact codes like ST-7. 7 more are out of scope on purpose, such as the capital of France or a clinical question that must be declined.

A passage counts as right only if it comes from the right document and contains the key phrases of the answer. That judgment doesn’t depend on how the text was cut, so I could change the chunking and re-run.

Setup	Right passage first	In the top 5	Mean reciprocal rank
Keyword	42%	76%	0.568
Hybrid	62%	86%	0.726
Vector	76%	92%	0.837
Keyword + rerank	78%	88%	0.821
Vector + rerank	86%	100%	0.921
Hybrid + rerank	86%	100%	0.921

The raw numbers are in latest.md. MRR (mean reciprocal rank) is the average of 1 divided by the rank of the first right passage, so 1.0 means it’s always first.

What I learned:

The reranker was the biggest win. It moved the right passage to first place in 86% of questions instead of 76%, and got the right passage into the top 5 for every question. It costs about 200 ms per search.
Keyword search added nothing once the reranker was on. Vector plus rerank and hybrid plus rerank scored identically. I also tried giving the vector side more weight in the fusion (2 to 1 and 3 to 1), and the reranked result moved by at most one question.
Without the reranker, fusing was worse than vector alone. Keyword search was the weaker signal on this corpus, and an equal-weight merge let it drag good vector hits down.
I still kept hybrid. It costs nothing measurable, and a keyword match protects against a failure my small test set can’t show, like a vector miss on a rare form code.

The embeddings question

I used voyage-3 for the embeddings. What the evals show is what mattered for quality around them: a reranker on top, chunks that follow the document’s structure, and a breadcrumb inside every embedded passage. The first run also caught a real bug in my keyword tokenizer, which turned ST-7 into st and dropped the 7. The fix is in the tokenizer.

The eval harness takes the embedding model as a setting, so comparing another model is one environment variable and a re-run. That’s the first thing I’d do with a client’s real documents.

Teaching it to say no

A reranker gives each passage a relevance score, and I noticed it separates the two kinds of questions well. For every answerable question, the best score was at least 0.695. For every unrelated one, it was at most 0.512. So I added a cutoff at 0.6 (the separation is measured with an AUROC (area under the receiver operating characteristic curve) function): if the best passage scores lower, the assistant answers that it doesn’t have the information instead of improvising.

I measured it with the real agent on all 63 questions, judged by a stronger model, using the generation eval script and its LLM (large language model) judge. Unanswerable questions handled correctly went from 69% to 92%, with no change in correctness on answerable ones.

Then I found where it breaks. Six of my questions are near misses: on topic, but I had verified the answer is not in the documents (does the clinic do dental work? how much does SURA pay per hospital day?). Five of those six still scored between 0.605 and 0.844, right inside the range of real answers. A score cutoff can’t tell “related but absent” from “answered”, so for those the assistant has to rely on its instructions. It’s listed in the repo as a known gap.

Making it stick to the sources

The first prompt produced correct answers, but the claim-by-claim check showed 44% of them contained at least one detail that the retrieved passages did not state. Things like “keep your receipt” or “requirements may vary by insurer”. Mostly harmless, sometimes wrong.

So I ran a small experiment. I added a section to the instructions: say only what the results say, don’t cite documents the tool didn’t return, and mark anything missing as uncertain. Then I re-ran all 63 questions, and the comparison is saved in the repo.

	Before	After
Answers with an unsupported claim	44%	10%
Claims supported by the passages	93.6%	98.8%
Answers judged correct	96%	98%
Cited a page or link never retrieved	4%	0%
Average answer length	2,303 characters	1,684 characters

Part of that is simply that shorter answers have fewer chances to be wrong. Correctness didn’t drop, but a user who wanted more hand-holding gets less of it. That’s a product trade-off, and I’d want the client’s input on it.

What I’d tell you about the numbers

The test set is small and I wrote it after reading the documents, so it probably overlaps with the wording more than real users would. I treat differences under 5 points as noise.
Each question was run once, and the judge is a model. When I switched the judge from a cheap model to a stronger one, one headline number moved from 92% to 38%, because the cheap judge flagged the assistant’s own polite boilerplate as unsupported claims. I stopped quoting numbers without saying which judge produced them.
The retrieval test doesn’t measure answer quality, and the answer test doesn’t measure real users. They’re a regression check, not a benchmark. The first thing I’d do with a client is collect their staff’s real questions.

Smaller things worth showing

Streaming. A stream parser turns the backend events into chat updates. The first words appear after about 3.7 seconds instead of waiting about 8 seconds for the full answer. It first says “searching the documents”, then “reading the passages”.
Source cards. Each answer has expandable cards with the passage, document, section, page, a relevance bar and a link to the original.
Failing well. When the database or the embedding service is down, the assistant now says so in about 4 seconds. Before, it retried until it ran out of steps and returned nothing. The fix is a result the tool returns instead of an error.
Safe by default. The backend refuses to start without an API (application programming interface) key. The chat route forwards only an allow-list of events, because the raw stream also contains the system prompt, the model’s reasoning and provider headers. There’s rate limiting per IP (Internet Protocol) address.
Not covered: prompt injection through documents (the corpus is public and curated, but ingestion doesn’t sanitize it), per-user login, and any compliance regime. This is not a medical product.

What’s next

A bigger question set built from real users’ questions, a run of the final prompt judged by the stronger model, OCR (optical character recognition) for scanned PDFs, and a public demo with the fictional clinic. After that, booking tools with guardrails and a WhatsApp channel.

Glossary

Tools and products

Mastra: an open-source TypeScript framework for building AI agents. It handles the loop of “the model decides to call a tool, gets the result, writes the answer”, and serves the agent over an HTTP (HyperText Transfer Protocol) API.
Claude Haiku: the small, fast and cheap model from Anthropic that writes the answers. It also acts as the judge in the evals.
Voyage AI: a company that makes models for search. I use two of its models: voyage-3 to turn text into embeddings and rerank-2.5 to rerank passages.
PostgreSQL and pgvector: PostgreSQL is a regular database. pgvector is an extension that lets it store embeddings and find the closest ones to a question.
Next.js: the React framework behind the chat page. It also hosts the route that sits between the browser and the backend.
Railway: the hosting platform where the backend, the frontend and the database run.

Search ideas

RAG (retrieval-augmented generation): instead of letting the model answer from memory, first search a library of documents, then give the model the best passages and ask it to answer from them.
Passage (or chunk): a piece of a document, 400 to 1400 characters here, that is stored and searched on its own.
Embedding: a list of numbers (1,024 of them here) that represents the meaning of a text. Texts with similar meanings get similar numbers, even with different words.
Vector search: finding the passages whose embeddings are closest to the embedding of the question. Closeness is measured with cosine similarity, which compares the direction of two embeddings.
BM25 (Best Matching 25): a classic keyword search that scores passages by how often rare words from the question appear in them. It’s good at exact terms like form codes.
Hybrid search: running a vector search and a keyword search together and combining their results.
RRF (Reciprocal Rank Fusion): a way to merge two ranked lists using only positions. A passage ranked 1st in one list and 3rd in the other scores higher than one that appears in a single list.
Reranker (or cross-encoder): a model that reads the question and one passage together and scores how well the passage answers it. It’s slower than the first searches, so it only sees the best 30 candidates.
Breadcrumb: the path of titles above a passage, like document, section and subsection, stored with it so the passage knows where it came from.
Grounding: making the model say only what the retrieved passages say. A grounded answer can be checked against its sources.
Near miss: a question that is on topic but whose answer is not in the documents. It’s the hardest case for a “say no” rule.

How I measured it

Hit@k: the share of questions where a right passage appears in the top k results. Hit@1 means it was first.
MRR (mean reciprocal rank): for each question, take 1 divided by the position of the first right passage, then average. 1.0 means it was always first.
AUROC (area under the receiver operating characteristic curve): a score from 0.5 (no better than chance) to 1.0 (perfect) for how well a score separates two groups. Here, the groups are answerable and unanswerable questions.
LLM (large language model) judge: a model that grades answers. Here it lists every claim in an answer and says whether the retrieved passages support it.
Faithfulness: the share of an answer’s claims that the retrieved passages support.
Ablation: switching one part of a system off to see how much it was helping. Comparing results with and without the reranker is an ablation.

A proof of concept with public documents and a fictional clinic. Not a medical product, and not for clinical decisions.