Retrieval-augmented generation — RAG — is one of those terms that gets thrown around until it stops meaning anything specific. Stripped of the jargon, it's a simple idea: before the AI writes an answer, go find the actual relevant information and hand it to the model, rather than trusting the model to already know it.
The problem RAG solves
A language model's knowledge is frozen at training time and general by nature — it doesn't know your company's FAQ, your customer's enrollment record, or what happened in your business yesterday. Ask it a specific question about your own operation and it will either say it doesn't know, or worse, guess plausibly and get it wrong. RAG is the fix: instead of asking the model to answer from memory, you retrieve the relevant real source material first, and only then ask it to answer — grounded in something true, not recalled from training.
How it actually works, step by step
Take the retrieval-augmented support email agent we built for a client as the concrete example, because every RAG system follows roughly this shape:
- The source material gets prepared in advance. The client's FAQ and policy documents are broken into chunks and converted into embeddings — numerical representations of meaning — using Cohere, then stored in a vector database (Pinecone).
- A question comes in. In this case, a new support email arrives.
- The system retrieves what's relevant. Rather than searching by keyword, it searches by meaning: the incoming question is embedded the same way, and the vector database returns the FAQ passages closest to it in meaning — even if the customer's wording doesn't match the FAQ's wording exactly.
- The model answers using only that retrieved context. The retrieved passages, plus the customer's own record (looked up separately in Google Sheets), get handed to the model along with the question. The model's job is now to compose a good answer from what it was given — not to recall an answer from training.
- A person still checks the output. The draft is saved to Gmail, labeled, unsent — because "grounded in real material" reduces hallucination, it doesn't eliminate the need for review before something goes to a customer.
Why this matters more than model choice
A model with RAG and a mediocre setup will usually outperform a better model with no retrieval at all, because the ceiling on accuracy is set by what the model was given to work with, not by how capable the model is in the abstract. This is the practical reason "just use the best model" is bad advice for building anything that needs to be right about your specific business — the model was never going to know your FAQ. It needed to be shown it.
What "good RAG" actually looks like
The tell of a well-built RAG system isn't that it never gets anything wrong — it's that when it answers, the answer traces back to something real, and when it doesn't have a good match, it says so rather than filling the gap with a plausible guess. That's a design choice as much as a technical one: retrieval quality, chunking strategy, and what happens when nothing relevant is found all matter more than which model does the final writing.
If you're evaluating a RAG-based tool, ask what happens when the retrieval comes back empty or irrelevant. That answer tells you more about the system's honesty than any benchmark score will.