"It knows everything on the internet. Nothing we have actually written down."— Office Man
What is it?
RAG stands for Retrieval-Augmented Generation. It is an approach to giving AI access to specific documents at the moment you ask a question. Standard AI tools are trained on public data — they know a great deal about the world in general and nothing about your organisation in particular. RAG addresses this: when you ask a question, a retrieval system searches your document library, finds the relevant sections, and feeds them to the AI as context. The AI then answers based on what it has just been given. It is not learning your documents. It is reading the relevant parts at the moment you ask.
Why now?
Most workplace knowledge is locked in documents. Policies, procedures, project records, past proposals, guidelines. Finding things in those documents is genuinely hard — search returns too many results, people who know where things are leave, institutional memory leaks away. RAG is being built into a wide range of enterprise AI products because it addresses this directly. Copilot in Microsoft 365 uses it. Many specialist AI tools are built on it. The question is no longer whether RAG exists — it is whether you are using the version you already have.
The human angle
The appeal of RAG is the appeal of having a colleague who has read everything and can find it quickly. The limit of RAG is that it is only as good as the documents behind it. If your document library is a pile of out-of-date, inconsistently named, version-confused files accumulated over a decade, the AI will retrieve from that. Garbage in, plausible-sounding garbage out. Before thinking about the AI side of this, it is worth thinking about the documents side. What would you want someone to be able to find?
What to do
Find out what you already have. Microsoft Copilot, if your organisation has licensed it, can search your SharePoint and OneDrive files — that is a form of RAG. Ask your IT team what document search capability exists before evaluating new products. If you are testing a RAG product, test it on your actual documents, not vendor demos. Start with a small, well-organised set — twenty clean, up-to-date policies will outperform two thousand accumulated files. And remember: RAG reduces hallucinations on document-specific questions but does not eliminate them. Still check the answers for anything consequential.