RAG is a data-cleaning problem wearing a trench coat

I built a small retrieval-augmented answer system for project documentation, and I learned the lesson everyone learns: the embeddings were never the problem.

My first version answered questions by retrieving the top few chunks and trusting the model to make sense of them. It hallucinated constantly. The fix, in order of impact:

1. deduplicating content — the same instructions existed in four documents with slightly different wording, and retrieval happily returned all four
2. fixing chunk boundaries — my chunks split mid-sentence, so the "answer" was literally fragments
3. adding metadata filtering — questions about version 2 were being answered with version 1 docs
4. only then, tuning the model prompt

The 80/20 of RAG quality

• clean the corpus before touching the pipeline
• chunk on structural boundaries: headings, sections, paragraphs — never mid-sentence
• store metadata (date, version, source) and filter with it
• log which chunks were retrieved, because when answers go wrong, that's your first suspect

The reframe that helped

A RAG system is a librarian, not an oracle. If the library is a mess, the most brilliant librarian in the world will fetch you the wrong book with total confidence. Spend your effort on the library.