Engineer RAG is an open-source project I built to understand how retrieval-augmented generation works inside. You ask a question about a curated set of engineering articles, and the system retrieves passages and writes an answer with citations. I wrote the pipeline directly in Python so I could inspect and measure each stage.
Building the pipeline
The system loads Markdown articles from folders, splits them into paragraph chunks, and stores dense embeddings and keyword-search vectors in Qdrant. At query time, it combines semantic search with BM25 keyword search, then optionally reranks the retrieved passages with Voyage before generating an answer.
The CLI and Streamlit chat interface use the same core package. The interface shows the answer, cited passages, and retrieved chunks for debugging. The screenshot above shows an earlier version using my curated article collection.
Checking what improved
I started with dense search, added hybrid retrieval, and then added reranking. I measured each change against a small private evaluation set: 54 questions over 10 articles, including four questions the corpus should refuse.
| Retrieval stage | Recall@5 | Recall@10 | MRR |
|---|---|---|---|
| Dense search | 0.780 | 0.860 | 0.630 |
| Hybrid search | 0.920 | 0.960 | 0.755 |
| Hybrid search + reranking | 0.980 | 0.980 | 0.919 |
The retrieval table covers the 50 answerable questions; the four refusal questions are evaluated separately. Recall@k here measures the share of answerable questions with at least one expected passage in the top k results. MRR averages the reciprocal rank of the first expected passage, with zero for a miss. These are retrieval measurements on this corpus; they do not measure whether the final answer is correct.
I also used Claude to grade cited claims against their sources. That caught answers with a correct fact attached to the wrong passage, especially when a sentence combined information from several sources. Some answer text was uncited or skipped by the claim parser, so the graded claims cover only part of the output.
Trying it locally
The repository includes a separate synthetic demo corpus with 11 articles and its own evaluation questions. It contains deliberate retrieval traps and questions the system should refuse. Its results are recorded separately from the private corpus above.
The app currently runs locally through the CLI or Streamlit. It needs Qdrant and an OpenAI API key; reranking and claim grading use optional Voyage and Anthropic keys. Ingestion, answers, and model-based evaluations can incur API costs.
Explore Engineer RAG on GitHub, including the experiment log and known limitations.