RAG explained. And four cases where it is not worth it
An AI model answers from memory. RAG has it look things up in your own documents first, so that the answer can carry its source with it. The hard part is not the AI. It is your filing.
In short
- RAG means looking it up instead of recalling it. A search step pulls the relevant passages out of your documents and puts them in front of the AI model before it answers. Every answer can then carry a source, provided the system passes it along.
- Checkable is not the same as correct. Three legal research products were sold on the promise that nothing in them is made up. In testing, between one in six and one in three of their answers came out wrong or wrongly sourced.
- The project is your filing, not the AI. If nobody can say which version of a document applies, that is where the tidying starts. In Deloitte's survey, 55 per cent of organisations said they had steered clear of certain use cases because of problems with their data.
Ask an AI model about your notice period and you will get an answer: three months, delivered with complete assurance, without a single source. It has never seen your contract. RAG puts a step in front of that. A search pulls the relevant passage out of your own papers and hands it to the AI model before it writes a word. Looking things up lays the groundwork for a checkable answer. That is the whole mechanism, and it is worth having less often than it is offered to you.
What RAG is, in one sentence
RAG stands for retrieval-augmented generation, which is to say: look it up first, then write. Left to itself, an AI model answers from what it saw during training. Your contracts, manuals and minutes were not part of that. It does not know them, and it will not tell you so; it produces a plausible answer instead.
Take a question that has a counterpart in every organisation: 'How long is our notice period?'
Without looking it up
'The notice period is usually three months, effective at the end of a quarter. The applicable terms are those set out in the contract and in the general terms and conditions.'
Sounds right. No way to check it.Looked up in your own documents
'Notice must be given in writing no later than three months before the end of the contract term.'
Framework agreement of 4 March 2024, clause 12.2
The answer without a lookup is not wrong. It may well be right; it just cannot be checked. And the one with a source is not right because a source is attached: if the 2021 version is still sitting in the files, what you get is a tidy citation under a figure that no longer applies.
RAG gives answers a basis that can be checked; it does not make them true. Whether a sentence holds is still decided by whoever opens the document it points to.
Four reasons to leave it alone
It is not the technology that decides, but your documents and the kind of questions you ask. Two conditions have to come together, and four reasons rule RAG out.
RAG is worth it when
- the answer sits in one particular document
- you have more than 400 A4 pages of material
- the papers change all the time
- somebody has to be able to check the source
RAG is not worth it when
- something has to be counted or calculated
- it all fits into a context window, the amount of text an AI model takes in at once
- it is about style and form
- the answer is written down nowhere
The second reason is the one that costs most when it is missed. A manual, a price list and twenty factsheets come to under a hundred pages together. Build a RAG system for that and you have taken on something to run without having solved a problem.
| The case | How to spot it | What to do instead |
|---|---|---|
| Counting or calculating | 'How many', 'how often', 'which ones are over CHF 500,000'. The answer comes out of all the documents, not out of one. A similarity search does not treat numbers and date ranges as values it can sort. | A table you can run a query against. If the figures are already in a database, that is an afternoon; if they sit in four hundred documents, it is the same filing project, just without the AI. |
| The document set is small | Under roughly 400 A4 pages of German, and the questions hit the words used in the papers. | Put the lot into the context window. No search step, no database, nothing to run. |
| Style and form | 'Answer the way our legal department does.' That is not knowledge, that is behaviour. | A clear instruction and three examples. Only after that, fine-tuning, which means changing the AI model itself. |
| Written down nowhere | Judgement and weighing things up. Questions on which two experienced people in the organisation take different views. | No AI solution at all. And if the answer matters: write it down. |
The 400 pages are the threshold above which the machinery earns its keep, not a promise that everything below it will be found. They come from 200,000 tokens, the word fragments an AI model cuts text into, converted at around 500 of them per A4 page of German; English text runs to roughly twice as many pages for the same number of tokens. Context windows today hold many times that. The length over which things are reliably found does not grow at the same rate: Anthropic writes in its own developer documentation that accuracy and recall decline as the token count rises. There is no documented cut-off. The 400 pages are an editorial judgement and should be treated as one.
When it comes to getting new knowledge into a model, the two have been measured against each other: a peer-reviewed study from 2024 puts a small AI model at 0.875 for looking things up against 0.504 for fine-tuning, on a scale of 0 to 1. For style and form it is the other way round in our experience; that is not something the study measured.
The AI model does not see all your documents
It sees only the extracts the search puts in front of it. Two things happen here, at completely different moments. Run them together and you end up believing the system learns something with every question. It does not.
Prepared once
On every question
'How long is our notice period?'
Not on the internet. In your own material, prepared beforehand.
A handful of extracts, ranked by similarity.
The AI model gets the question and these extracts.
With a note of which extract it came from, if the system passes it along.
From step 04 on, these extracts are all the AI model sees of your material. Whatever the search did not find does not exist as far as the answer is concerned. What it learned in training it still brings with it
Nothing flows back the other way: the AI model remembers nothing and grows no wiser from your documents. Change a rule and you change it in the files. But a new document does not take effect the moment it lands in the folder. It counts only once it has been prepared and taken into the search index. On some systems that runs overnight, on others somebody has to set it going. The same holds in reverse: a withdrawn document goes on answering questions until it has disappeared from there too.
Three pieces of jargon turn up in every proposal. Chunking means cutting documents into short passages. Embedding means converting each passage into a long string of numbers that stands for its meaning; similar meanings get similar strings, and that is how a search for 'ending an agreement' finds a clause that says 'notice of termination'. The vector database is where those strings of numbers are kept; it does not necessarily mean buying anything new. Google and OpenAI now sell the whole chain as a finished service at published prices, which makes building it cheaper and the question of where your data sits more pressing.
The request that reaches the AI model looks like this.
Extracts one and two contradict each other, and that is not a made-up example but the everyday case RAG is built for: the old version sits alongside the one in force, and both can be found. A simple RAG system does not reliably resolve contradictions on its own. It names one of the two figures, and unless the system passes the source reference along, nobody notices which. A 2024 study called ClashEval measured how serious that is: when a retrieved document is wrong, AI models drop their own correct knowledge and go with the document in over 60 per cent of cases.
An out-of-date document in the files is an active risk, not just a gap.
Does RAG stop made-up answers?
Made-up statements, hallucinations in the jargon, are the reason anyone talks about RAG at all. A research team at Stanford University measured three commercial legal research products that were marketed as 'hallucination-free'. The comparison figures for general AI models come from an earlier study by the same institute.
Share of wrong statements on legal questions, depending on the AI model. Dahl et al., Stanford, 2024.
Three legal research products from two vendors. An answer also counts as wrong when it rests on a source that does not support it. Magesh et al., Stanford, 2025.
The two values come from two studies with different sets of questions. The reduction is calculated, not measured.
One answer in six to one in three is still faulty, and these are products built for this one purpose. RAG moves the source of error, it does not remove it. There are four places it can come from.
Where it breaks in practice
The old version sits alongside the one in force. Symptom: the answer is correctly cited and no longer applies.
The extract ends before the exception. Symptom: the answer gives the rule without the caveat.
It finds three passages that sound alike; the right one is the fourth. Symptom: the answer backs itself up with the wrong document.
The AI model fills in from its training what the extract leaves out. Symptom: the source is right, the sentence it supports is not in that document.
A source reference shows where the system looked, not that the sentence came from there. That is the difference between a footnote and a proof, and it decides whether spot checks are needed. They are.
There is a measurement for the fourth of these. Google Research showed in 2025, on a small AI model, what a half-fitting extract does. The rate of wrong statements rises to 66.1 per cent, against 10.2 per cent with no context at all. The comparison flatters one side, because with no extract the AI model more often declines to answer. It still works as a warning: a half-answer looks like a foundation, and people build on it.
The most expensive work package is your filing
Big numbers circulate about AI projects that fail. The measured ones are less spectacular and therefore more useful.
Share of companies that have abandoned the majority of their AI initiatives, against 17 per cent the year before. S&P Global, March 2025, over 1,000 respondents.
Share of organisations that steered clear of certain use cases because of problems with their data. Deloitte, 2,770 executives surveyed, fieldwork May and June 2024.
Our own calculation for 60 people, from USD 790 to 1,200 per user per year for running it. Analyst estimate with no methodology disclosed, Gartner, 2024.
The third figure is the one to hold a quotation against. Anyone offering you a RAG system for CHF 80,000 may not be naming a project cost at all, but a yearly bill under another label. Ask what falls due once and what comes back every year. The money is certainly not in the AI model itself: querying a model at the performance level of GPT-3.5 became around 280 times cheaper between November 2022 and October 2024.
Anyone whose documents are not under control uses RAG to build a machine that broadcasts the state of their filing, very fast and very convincingly.
Three things get overlooked. The first: is the material machine-readable? Open a document and try to copy a sentence out of it. If that does not work, you need text recognition, and for tables, one that keeps the layout intact. The second: permissions. A search across everything cuts straight through access rules unless the system is built to enforce them. Personnel files and live cases are cleanly separated in the filing system; in a naively built search index they are not. The third: when the material is prepared, the whole of it passes through the provider of the search service once, unless you run that in-house. What is stored and evaluated there is in the contract, not on the data sheet. Both of these are covered on our page on local LLMs and governance.
Air Canada was liable for its chatbot
So far this has been about quality. The second reason for RAG is being able to show what an answer rested on, and the way the courts see it is taking shape at the companies' expense.
| Case | What happened | Outcome |
|---|---|---|
| Moffatt v. Air Canada February 2024 |
The airline's chatbot gave wrong information. The company argued that it was a separate legal entity. | Rejected: "it is still just a part of Air Canada's website". CAD 812 in total, of which 651 in damages |
| Deloitte Australia October 2025 |
A report running to 237 pages contained invented footnotes, among them a made-up quotation from a Federal Court judgment. | Repayment of the final instalment, AUD 97,000 on a contract worth AUD 440,000 |
| Invented sources in court August 2026 |
1,936 decisions worldwide in which invented citations played a part, as at 21 August 2026. In May 2025 there were around 200. | Highest known sanction around USD 110,000, mostly the other side's legal costs. Most cases remain in four figures |
The first two cases are a single court decision and a single contractual settlement, not a statistic. The third row comes from a continuously maintained collection and counts decisions, not companies.
Switzerland fines differently from the EU: the individual first, not the company. Articles 60 to 63 of the Federal Act on Data Protection (FADP) provide for up to CHF 250,000, for private bodies and federal authorities; cantonal and communal bodies come under cantonal law. Whoever commits the breach is liable only where they acted deliberately; a superior who fails in their duty of supervision is liable for negligence alone (Art. 64 para. 1 FADP with Art. 6 para. 2 of the Federal Act on Administrative Criminal Law). Only where identifying the responsible individual would be disproportionate may the authority fine the organisation instead, up to CHF 50,000. In practice two summary penalty orders have become public since 2023, with fines of CHF 450 and CHF 600.
The argument that carries weight is therefore not the fine but being able to retrace what happened. The European AI Act requires logging only for high-risk AI systems, and those obligations have been put back to December 2027. Whether your document search is one of them depends on its purpose: information about contract deadlines is not, while a search that feeds into decisions about people can be. The case for source references is not a legal one, then, but an operational one. A system that carries document, section and version with every answer can show, if it comes to a dispute, what an answer rested on. One that writes from memory stands where Air Canada stood.
Ten minutes before you ask for a quotation
The figures in this article do not answer the question that counts: does it work for you? Only a trial with your own papers answers that.
What we recommend
Check first whether the question is a RAG question at all. The four reasons to rule it out cost ten minutes and, if in doubt, save you a whole project.
Test with ten real cases from your own organisation. Not with the provider's examples, and the questions put by someone who already knows the right answer. Otherwise all you are testing is whether the answer sounds convincing. That costs half a day.
Keep those ten cases. They are not the acceptance test, they are your yardstick. Repeat them every six months, and every time the provider puts in a new AI model.
Budget for the filing work. If nobody can say which version of a document applies, tidying up is the first work package and not the second. That work is coming anyway, with or without AI.
Nine questions for a RAG system, whoever the provider is
- Does every answer carry a source with document, section and version date? Without the date the reference is worth half as much.
- Does the system say 'that is not in the papers' when it finds nothing?
- On a spot check, does the source hold up, and is the quoted sentence really in that document?
- What happens when two documents contradict each other? Do you get told, or does the system decide quietly?
- How long does it take for a changed document to reach the answers, and what happens to one you withdraw?
- Does each person see only the answers they would be allowed to see in the filing system?
- Is there a keyword search alongside the similarity search, and can you narrow by document type, version and period? Your file references hang on the one, your deadline questions on the other.
- Where do your papers go when they are prepared and on every question, and what does the contract say about that?
- Which part of the price falls due once and which every year? Ask about the term, about getting out, and about who owns the prepared data.
Open questions
Not sure whether your case is one for RAG at all?
The question can usually be settled in a conversation, and the answer is often no. In a first meeting we look at your particular case, check the volume, the questions and the state of the filing, and tell you what the smallest sensible step would be. Often that is not a project but a clear-out.
Sources 22 ShowHide
- Peer-reviewed. Stanford RegLab, Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies, 2025 · arxiv.org/abs/2405.20362
- Peer-reviewed. Stanford RegLab, Dahl et al., Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models, Journal of Legal Analysis, 2024, over 200,000 queries · arxiv.org/abs/2401.01301
- Peer-reviewed. Wu et al., ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence, NeurIPS conference, 2024 · arxiv.org/abs/2404.10198
- Peer-reviewed. Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, EMNLP conference, 2024 · arxiv.org/abs/2312.05934
- Research. Google Research, Sufficient Context, on the hallucination rate when the context is insufficient, 2025; the figures given come from the accompanying post on the research blog and apply to a small AI model · arxiv.org/abs/2411.06037
- Measurement. NoLiMa: Long-Context Evaluation Beyond Literal Matching, ICML conference, 2025, on the usable length of context windows · arxiv.org/abs/2502.05167
- Vendor, with an interest of its own. Anthropic, developer documentation on context windows, retrieved on 21 August 2026, in its own words: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot" · platform.claude.com
- Vendor, with an interest of its own. Google, announcement of the File Search tool in the Gemini programming interface: storage, chunking and embedding as a service, 15 US cents per million tokens read in · blog.google
- Vendor, with an interest of its own. OpenAI, documentation of the File Search tool: storage and search as a service, with source references in the answer. Prices are not given there but on the pricing page · developers.openai.com
- Case law. Moffatt v. Air Canada, 2024 BCCRT 149, decision of 14 February 2024 · canlii.org
- Law, Switzerland. Federal Act on Data Protection, scope Art. 2 and criminal provisions Art. 60 to 64 · fedlex.admin.ch
- Case law, Switzerland. First summary penalty order under the revised FADP, Stadtrichteramt Zurich, 10 June 2024, fine of CHF 450 · steigerlegal.ch
- Case law, Switzerland. Second summary penalty order, Statthalteramt of the district of Zurich, 4 March 2025, fine of CHF 600, appealed · steigerlegal.ch
- Law, EU. Regulation (EU) 2024/1689 on artificial intelligence, Art. 12(2), as amended by Regulation (EU) 2026/1744 of 24 July 2026, which puts the obligations for high-risk AI systems back to 2 December 2027 · eur-lex.europa.eu
- Analysis, methodology not disclosed. Gartner, How to Calculate Business Value and Cost for GenAI Use Cases, 2024, table on the document search use case: USD 790 to 1,200 per user per year to run it. Paid report, not publicly available · gartner.com
- Survey. S&P Global Market Intelligence, March 2025, over 1,000 companies surveyed in North America and Europe · spglobal.com
- Survey. Deloitte, State of Generative AI in the Enterprise, 2,770 executives surveyed, fieldwork May and June 2024 · deloitte.com
- Index. Stanford HAI, AI Index Report 2025, on how token prices have moved at a fixed level of performance · hai.stanford.edu
- Collection. Charlotin, AI Hallucination Cases, a continuously maintained database of court decisions, as at 21 August 2026 · damiencharlotin.com
- Reporting. Repayment by Deloitte Australia, confirmed on 21 October 2025 · theguardian.com
- Our own calculation. Yearly cost for 60 people, from the Gartner range per user per year
- Our own work. digitario, model sizes and tokens · digitario.ch/artikel/modellgroessen-und-tokens-verstehen
Advises public administrations and companies on procuring and introducing AI systems. Maintains the comparison tables on this site.