Giacomo Mantovani

Article · November 17, 2026 · By Giacomo Mantovani

A chatbot on your company documents (RAG): how it works and when it pays off

An AI assistant that answers from your own policies, manuals and price lists is called RAG. How it works and why its answers go wrong.

  • RAG
  • document AI
  • knowledge base
  • internal chatbot
The search gets it wrong. The model repeats it. Hybrid search, Filters, Re-ranking, Permissions inside the search.

Most requests start the same way: "Can we have a ChatGPT that knows our policies, our product sheets and our manuals?" You can, and the technique is called RAG (retrieval-augmented generation). Before you buy or build one, it helps to know where these systems fail. In the ones I have built, most wrong answers came from the search step that picks which documents the model gets to read.

What is a RAG chatbot over company documents?

A RAG chatbot works in two steps. When someone asks a question, the system searches your documents for the most relevant passages: a section of a manual, a clause of a policy, a row of a price list. It then gives those passages to the language model together with the question, and the model writes an answer from that material.

The model reads your documents at question time, without being trained on them, the way a colleague would skim three pages you hand them before a call. That is why a policy updated this morning can be used in an answer this afternoon, with no retraining.

Why does a document chatbot give confident wrong answers?

If the right passage never reaches the model, the model answers with whatever it was given: a similar document, an outdated version, a related case that does not apply. The answer sounds right and is wrong. Better retrieval moves answer quality far more than a bigger model does.

I saw this on Estyl, a conversational fashion assistant over a catalogue of more than 2 million products. Vector search alone, the kind that finds text with similar meaning, returned items that looked right but ignored size, price and stock. Strict filters often returned nothing at all. We rebuilt retrieval with hybrid search, structured filters, controlled filter relaxation when a request is too narrow, and a final re-ranking step. Under a strict evaluation, where a response counts as correct only if every item in it matches the request, more than 90% of responses were fully correct. The details are in the conversational product search case study.

Internal knowledge behaves the same way. At a major Italian bank's help desk, an LLM agent routes more than 1 million tickets a year into 400+ hierarchical categories. Putting every category definition into every prompt was not practical, so RAG retrieves only the relevant branches of the taxonomy and the model chooses among them. It is right 81% of the time, against a 75% human baseline (help desk ticket routing case study).

What goes wrong in a RAG chatbot, and the fix
What goes wrongWhy it happensFix
Confident but wrong answerSearch returned a similar document instead of the right oneHybrid search (exact terms plus meaning), metadata filters, re-ranking
Quotes a superseded policySeveral versions of the same document sit in the indexOne valid version, an effective date in the metadata, regular clean-up
"I couldn't find that" on easy questionsThe text is inside scans, tables or images that were never read properlyOCR and layout analysis before indexing, spot checks
Shows content the user should not seePermissions are applied after search, or not at allFilter by user, team or customer inside the search itself
Vague answers on part numbers and codesSemantic search confuses similar SKUs and acronymsKeyword search alongside semantic search

How should you prepare documents before connecting them to a chatbot?

Document preparation is as much work as the chatbot itself. A typical shared drive holds three versions of the same handbook, crooked scanned PDFs, spreadsheets with merged cells and folders nobody has opened in years. Whatever goes into the index, the chatbot will use.

Start with a narrow scope, such as the customer support procedures. Remove duplicates and outdated versions, and tag each document with metadata that helps search: team, effective date, product, language. Documents are then split into chunks, and each chunk has to make sense on its own. A paragraph that says "the notice period is 30 days" without naming the contract is useless to the model. Anthropic measured that prepending a short line of context to each chunk cut failed retrievals by 49%, and by 67% when combined with re-ranking (Anthropic, September 2024).

How do you control who sees what in an internal chatbot?

A chatbot that can read every company document can also show every company document: payroll, customer contracts, board minutes. Permissions belong in the search step. If a user cannot open a document, the system must not retrieve it for that user. A line in the prompt saying "do not reveal confidential data" is not access control.

On Estyl the boundary was the merchant: each store's assistant searches only that store's catalogue. Inside a company the same logic applies per team, role or customer account. Regulated companies add more constraints. In my banking projects the models ran on Azure OpenAI inside the client's EU cloud, pipelines could run on the bank's own OpenShift, no sensitive data went into logs, and every operation had an audit trail.

Should a document chatbot cite its sources?

Yes, every time. Each answer should name the document and the section it came from, with a link that opens it. Citations let the reader check in ten seconds instead of trusting the bot. They also let you debug: when an answer is wrong, you can see at once whether the retrieved document or the model's summary was at fault.

The second rule is that "I don't know" is a valid answer. If search finds nothing relevant enough, the chatbot should say so and, where it makes sense, hand the question to a person. A system that always answers, even when it found nothing, is the one that ends up making things up.

How do you measure whether a RAG chatbot is right?

Collect real questions from emails, tickets and the people who answer them today. For each one, write down the correct answer and the document that contains it, then run the whole set after every change. Measure two things separately: whether search found the right document, and whether the final answer is correct.

Score strictly. On Estyl a response counted as correct only if every suggested item matched the request, because one wrong item ruins the answer for a shopper. You also need a baseline. In ticket routing the comparison was the 75% accuracy of human routing; without that number, the system's 81% would have meant nothing.

When is a chatbot over company documents worth it?

When a RAG chatbot pays off
SituationWorth it?Why
Many repetitive questions on procedures, products or policiesYesThe answers already exist in documents and search can find them
Someone owns and updates the documentsYesThe chatbot is only as reliable as what it reads
Questions that need live data (stock, order status)Only with an integrationYou need APIs into the ERP or CRM, documents are not enough
A messy archive with no ownerNot yetClean up first, then build the chatbot
A handful of questions a monthUsually notA well-kept FAQ page costs less

If the goal is to turn documents into data for your ERP (delivery notes, quotes, supplier invoices), you need document extraction, which is a different tool. I cover it in the guide on extracting data from PDFs and delivery notes with AI.

I don't give generic prices. Cost goes up with the number and messiness of documents, connectors to SharePoint, Google Drive or your ERP, permission handling, and constraints on where data may be processed. The model's cost per answer is almost always the smallest line.

What are the common mistakes with a RAG chatbot?

Loading the whole archive on day one is the first; start with a scope you can measure. The second is switching models when answers are wrong, without checking what search returned. The third is leaving permissions to the prompt. Then there is the chatbot that does not cite sources, so nobody can trace where errors come from, and the one launched without a test set of questions, which gets judged on gut feeling after its first wrong answer in front of a manager. The last is having no document owner: six months later the procedures have changed and the chatbot still quotes the old ones.

Next step

Want a chatbot on your own documents?

Send me twenty real questions and a few sample documents, and I will tell you what search would find and where it would fail.

Let's talk

If you work in banking or another regulated industry, the page on document AI and RAG for regulated industries explains how I set up retrieval, permissions and audit trails.

Frequently asked questions

Do you need to train a model on your company documents?

No. With RAG the model reads the relevant passages at question time and is not changed. Fine-tuning changes how a model behaves, but it is the wrong tool for teaching it documents that change often.

Can a RAG chatbot answer questions about spreadsheets and tables?

Yes, if tables are parsed and chunked with care, for example one price-list row per chunk with the headers repeated. Questions about numbers and codes also need keyword search next to semantic search.

How many test questions do you need to evaluate it?

A few dozen real questions with the correct answer and source document are enough for a first measurement. Include hard ones, and questions where the right answer is "I don't know".

Can the documents stay inside the EU?

Yes, if you decide it before choosing services. In my banking projects the models ran on Azure OpenAI in the client's EU cloud, and the pipeline can also run on your own infrastructure.

Author

Giacomo Mantovani

AI Engineer & Consultant for Companies and Agencies

LinkedInGitHub

Next step

Tell me the workflow. We scope it on a free call, then a working version is live within 10 days.

Book a scoping call

Related articles