What RAG is and why so many web answers use it
RAG lets an AI retrieve relevant passages from a document collection before it answers. Here is how it works, where it is used, and why it does not eliminate errors.
When an assistant answers with details from a company’s vacation policy, a technical manual, or a catalogue that changes every week, it did not necessarily learn that information during training. Often, it retrieves a collection of documents immediately before answering. That family of techniques is called RAG: retrieval-augmented generation.
The idea was formally introduced in a 2020 research paper by authors from Facebook AI Research, UCL, and NYU. It combined a generative model with an external memory that could be searched: an index of Wikipedia. Today, the same principle connects models to private documentation, knowledge bases, and information that needs updating without retraining the model.
An open-notes exam
An everyday analogy helps. Imagine an exam where the student may bring notes, but not put an entire encyclopedia on the desk. First, someone organizes the notes. When a question arrives, they find the few most relevant pages. Only then does the student write an answer with those pages in view.
The student is the language model. The notes are the document collection that an organization has chosen to make available. RAG is the method for choosing which pages reach the desk. It does not alter what the student learned beforehand, nor does it grant permission to open every file in the organization. And, as in an exam, having good notes at hand does not fully prevent someone from misreading a sentence or adding something that was never written.
The route: index, retrieve, generate
The first step usually happens before anyone asks a question. Indexing prepares the documents: extracting text, splitting it into manageable passages, and storing them in an index that can be searched. Many systems give each passage a numeric representation so they can find similarities of meaning, not just identical words. Amazon Bedrock’s documentation describes this flow: documents are split, converted to numeric representations, and stored in an index.
When a question comes in, retrieval begins. The system searches for a small number of passages that seem to answer it. A question about an employee benefit can retrieve the relevant part of a handbook even if it does not use exactly the same wording. It can also apply filters—for date, department, language, or the user’s permissions.
Finally comes generation. The retrieved passages are added to the instruction received by the model, which writes a response. A good product shows where the information came from. Bedrock’s RetrieveAndGenerate operation, for example, combines retrieval and generation and can return citations to source passages. That traceability makes an answer easier to check; it does not automatically prove that the interpretation is correct.
Where it is used in practice
The clearest case is an assistant over internal documents: asking about policies, benefits, procedures, or manuals without having to read through entire folders. Amazon Bedrock Knowledge Bases can connect to sources including S3, SharePoint, Confluence, Google Drive, and OneDrive; it also documents document-level permission filters.
RAG also appears in search and customer-support products that must answer from a changing catalogue or document base. Microsoft presents Azure AI Search as a foundation for RAG applications that ground responses in proprietary content. Google Cloud offers Vertex AI RAG Engine, a service for building and querying retrieval corpora. These are not three names for a chatbot. They are ways to make an answer start from an identifiable selection of information.
Its limits matter as much as its usefulness
RAG is not a truth detector. If an index contains old, incomplete, or contradictory documents, it may retrieve exactly the wrong material. If it cuts a document into passages that are too small, it can lose context; if it returns too many, it can distract the model and make the request more expensive.
Microsoft makes the caution explicit in its documentation: quality depends on content preparation, retrieval configuration, and the prompt; even with retrieved information, a model can produce an inaccurate answer. Each added passage also consumes tokens, can increase cost, and may add latency in large indexes. And a well-built search can still leak sensitive information if source permissions are not enforced at retrieval time.
That is why RAG works best as a well-run reference desk, not an oracle. It brings freshness, specific context, and—when sources are shown—a way to verify. It still requires teams to review the corpus, measure what the system retrieves, update it, and design permissions. Many web answers do not improve simply “because they use RAG”; they improve when the information RAG brings forward is relevant, accessible, and checkable.
