RAG · LLM Systems · Evaluation
ResearchAgent
Citation-grounded research paper QA
A RAG answer can look finished even when retrieval has failed. The first version of this system produced fluent answers from loosely related fragments. Looking only at the final sentence did not reveal which component had broken.
Query
Does spatially coherent token selection change robustness under pruning?
↓ retrieved evidence
Passage 1
high relevance
Superpixel-based selection preserves neighboring tokens that independently sampled scores discard.
Passage 2
medium relevance
Accuracy at high keep-rates is close; the gap appears in dense prediction and under perturbation.
Passage 3
low relevance
A FLOPs reduction is reported, but latency is measured separately.
↓ generated answer
Yes, when the evaluation includes dense prediction and robustness, not only classification accuracy. [1][2]
01System
Local retrieval-augmented generation over research PDFs and plain text. The layout of the code is intentionally flat so the pipeline can be read end to end without a framework tour.
Ingest
Load PDFs and plain text, extract content, then token-aware chunking with overlap — paragraph-first rather than raw character windows.
Index
Embed chunks and persist them in Chroma. Full rebuild on re-index, on purpose: simple and inspectable.
Retrieve
Embed the question, return top-k passages by distance. This stage has its own metrics.
Answer
The chat model must return structured JSON with verbatim supporting quotes before the natural-language answer. If support is missing, the system abstains.
02Failure mode
Original chunking sometimes split paragraphs poorly. Retrieval returned fragments that were only loosely relevant. The model could still produce a confident answer. End-to-end fluency was not a test of grounding.
03Making the pipeline observable
I split evaluation into retrieval and generation, improved chunking, re-indexed, and added explicit abstention: if the indexed papers do not contain sufficient evidence, say so. A lightweight supervisor currently watches for off-task inputs and obvious output drift. It is basic, not a production safety system.
04Architecture
- IngestPDF / TXT
- IngestExtract
- IngestToken chunks
- IndexEmbeddings
- IndexChroma
- QAQuery embed
- QATop-k
- QAJSON + quotes
Retrieval (top-k) and generation (JSON + quotes) are separate observables. If either is weak, the UI can abstain instead of guessing.
05Evaluation
20 hand-labelled development questions. Development set, not a production benchmark.
Hit@5
1.00
MRR
0.97
Development set
20
hand-labelled questions
06Grounding
Every answer is supposed to point back to a supporting source passage. Two grounding modes exist in the UI: strict yes/no when the text clearly supports it, and short definition-based answers when an explicit yes/no is not in the sources. Both still require evidence in the JSON contract.
Repository
Ingest, chunking, index, retrieval eval, and the Streamlit UI live in the same repo.