Hybrid retrieval, a reranker, and a guardrail that names unsupported sentences.
FastAPI, Qdrant, Redis, Kafka, RAGAS. Runs with no API keys. Team project with Harsh Kokcha.
refund_policy.txt, chunk 1
Standard orders placed on the platform are eligible for a full refund within 30 days of delivery…
[1]Digital goods, including gift cards and downloadable software licenses, are non-refundable once the associated activation code has been redeemed.
Question
Can I still get a refund on a gift card I already redeemed?
Answer, abridged
Digital goods, including gift cards and downloadable software licenses, are non-refundable once the associated activation code has been redeemed. [1] …
Groundedfaithfulness 0.962
Every stage swaps from a local fallback to the production part with one line of config.
Dense and BM25 results, fused by rank because their scores do not compare.
A cross-encoder keeps the top 3 chunks. Local: lexical overlap.
Claude must cite a chunk for every claim. Local: an extractive stand-in.
Each sentence is scored against its source. Weak ones are named in unsupported_sentences.
The guardrail reports; it does not gate. A caller reads is_grounded and decides.
Nothing was retrieved, so nothing was made up.
Question
What is the capital of France?
Answer
I don’t have enough information in the available sources to answer that.
retrieved: none1.01 ms
10 questions over 6 chunks. Local: TF-IDF, lexical reranking, extractive stand-in. Claude: MiniLM, a cross-encoder and claude-sonnet-5.
| Metric | Local | Claude |
|---|---|---|
| context_precision | 0.567 | 0.5 |
| context_recall | 1.0 | 1.0 |
| faithfulness | 0.877 | 0.799 |
| answer_relevancy | 0.766 | 0.947 |
Faithfulness here is word overlap, so Claude’s paraphrasing lowers it. An LLM judge would not penalise paraphrase.
Each was found by checking one layer at a time.
The stand-in generator matched “order” but not “orders”. A small, labelled plural stripper fixed it.
With 6 chunks, top 5 touched nearly every document. Top k 2, 3, 5 gave 0.7, 0.567, 0.333.
A preview cut each chunk at 200 characters. The full source held every fact Claude cited.
No keys needed. Set ANTHROPIC_API_KEY to switch to Claude.
pip install -r requirements.txt
PYTHONPATH=src:. python scripts/demo.py
PYTHONPATH=src:. python scripts/run_eval.py
PYTHONPATH=src pytest tests/ -v