Most requests start the same way: "Can we have a ChatGPT that knows our policies, our product sheets and our manuals?" You can, and the technique is called RAG (retrieval-augmented generation). Before you buy or build one, it helps to know where these systems fail. In the ones I have built, most wrong answers came from the search step that picks which documents the model gets to read.
What is a RAG chatbot over company documents?
A RAG chatbot works in two steps. When someone asks a question, the system searches your documents for the most relevant passages: a section of a manual, a clause of a policy, a row of a price list. It then gives those passages to the language model together with the question, and the model writes an answer from that material.
The model reads your documents at question time, without being trained on them, the way a colleague would skim three pages you hand them before a call. That is why a policy updated this morning can be used in an answer this afternoon, with no retraining.
Why does a document chatbot give confident wrong answers?
If the right passage never reaches the model, the model answers with whatever it was given: a similar document, an outdated version, a related case that does not apply. The answer sounds right and is wrong. Better retrieval moves answer quality far more than a bigger model does.
I saw this on Estyl, a conversational fashion assistant over a catalogue of more than 2 million products. Vector search alone, the kind that finds text with similar meaning, returned items that looked right but ignored size, price and stock. Strict filters often returned nothing at all. We rebuilt retrieval with hybrid search, structured filters, controlled filter relaxation when a request is too narrow, and a final re-ranking step. Under a strict evaluation, where a response counts as correct only if every item in it matches the request, more than 90% of responses were fully correct. The details are in the conversational product search case study.
Internal knowledge behaves the same way. At a major Italian bank's help desk, an LLM agent routes more than 1 million tickets a year into 400+ hierarchical categories. Putting every category definition into every prompt was not practical, so RAG retrieves only the relevant branches of the taxonomy and the model chooses among them. It is right 81% of the time, against a 75% human baseline (help desk ticket routing case study).
| What goes wrong | Why it happens | Fix |
|---|---|---|
| Confident but wrong answer | Search returned a similar document instead of the right one | Hybrid search (exact terms plus meaning), metadata filters, re-ranking |
| Quotes a superseded policy | Several versions of the same document sit in the index | One valid version, an effective date in the metadata, regular clean-up |
| "I couldn't find that" on easy questions | The text is inside scans, tables or images that were never read properly | OCR and layout analysis before indexing, spot checks |
| Shows content the user should not see | Permissions are applied after search, or not at all | Filter by user, team or customer inside the search itself |
| Vague answers on part numbers and codes | Semantic search confuses similar SKUs and acronyms | Keyword search alongside semantic search |
How should you prepare documents before connecting them to a chatbot?
Document preparation is as much work as the chatbot itself. A typical shared drive holds three versions of the same handbook, crooked scanned PDFs, spreadsheets with merged cells and folders nobody has opened in years. Whatever goes into the index, the chatbot will use.
Start with a narrow scope, such as the customer support procedures. Remove duplicates and outdated versions, and tag each document with metadata that helps search: team, effective date, product, language. Documents are then split into chunks, and each chunk has to make sense on its own. A paragraph that says "the notice period is 30 days" without naming the contract is useless to the model. Anthropic measured that prepending a short line of context to each chunk cut failed retrievals by 49%, and by 67% when combined with re-ranking (Anthropic, September 2024).
How do you control who sees what in an internal chatbot?
A chatbot that can read every company document can also show every company document: payroll, customer contracts, board minutes. Permissions belong in the search step. If a user cannot open a document, the system must not retrieve it for that user. A line in the prompt saying "do not reveal confidential data" is not access control.
On Estyl the boundary was the merchant: each store's assistant searches only that store's catalogue. Inside a company the same logic applies per team, role or customer account. Regulated companies add more constraints. In my banking projects the models ran on Azure OpenAI inside the client's EU cloud, pipelines could run on the bank's own OpenShift, no sensitive data went into logs, and every operation had an audit trail.
Should a document chatbot cite its sources?
Yes, every time. Each answer should name the document and the section it came from, with a link that opens it. Citations let the reader check in ten seconds instead of trusting the bot. They also let you debug: when an answer is wrong, you can see at once whether the retrieved document or the model's summary was at fault.
The second rule is that "I don't know" is a valid answer. If search finds nothing relevant enough, the chatbot should say so and, where it makes sense, hand the question to a person. A system that always answers, even when it found nothing, is the one that ends up making things up.
How do you measure whether a RAG chatbot is right?
Collect real questions from emails, tickets and the people who answer them today. For each one, write down the correct answer and the document that contains it, then run the whole set after every change. Measure two things separately: whether search found the right document, and whether the final answer is correct.
Score strictly. On Estyl a response counted as correct only if every suggested item matched the request, because one wrong item ruins the answer for a shopper. You also need a baseline. In ticket routing the comparison was the 75% accuracy of human routing; without that number, the system's 81% would have meant nothing.
When is a chatbot over company documents worth it?
| Situation | Worth it? | Why |
|---|---|---|
| Many repetitive questions on procedures, products or policies | Yes | The answers already exist in documents and search can find them |
| Someone owns and updates the documents | Yes | The chatbot is only as reliable as what it reads |
| Questions that need live data (stock, order status) | Only with an integration | You need APIs into the ERP or CRM, documents are not enough |
| A messy archive with no owner | Not yet | Clean up first, then build the chatbot |
| A handful of questions a month | Usually not | A well-kept FAQ page costs less |
If the goal is to turn documents into data for your ERP (delivery notes, quotes, supplier invoices), you need document extraction, which is a different tool. I cover it in the guide on extracting data from PDFs and delivery notes with AI.
I don't give generic prices. Cost goes up with the number and messiness of documents, connectors to SharePoint, Google Drive or your ERP, permission handling, and constraints on where data may be processed. The model's cost per answer is almost always the smallest line.
What are the common mistakes with a RAG chatbot?
Loading the whole archive on day one is the first; start with a scope you can measure. The second is switching models when answers are wrong, without checking what search returned. The third is leaving permissions to the prompt. Then there is the chatbot that does not cite sources, so nobody can trace where errors come from, and the one launched without a test set of questions, which gets judged on gut feeling after its first wrong answer in front of a manager. The last is having no document owner: six months later the procedures have changed and the chatbot still quotes the old ones.
Want a chatbot on your own documents?
Send me twenty real questions and a few sample documents, and I will tell you what search would find and where it would fail.
Let's talkIf you work in banking or another regulated industry, the page on document AI and RAG for regulated industries explains how I set up retrieval, permissions and audit trails.



