How do you build company AI on your own documents? RAG in plain English

You do not have to train a model from scratch or paste contracts into a public chat. It is enough to give artificial intelligence proper access to your company documents, in such a way that the documents themselves never go anywhere. The method is called RAG.

What RAG is

RAG stands for Retrieval-Augmented Generation – an answer backed by a search. The method was described in 2020 by researchers from, among others, Facebook AI Research.

A language model without RAG is like a bright trainee on their first day at work. They write beautifully, but they know nothing about your business, so they guess. RAG gives the trainee the key to the archive. Before answering, they find the right documents, read them and only then write. And they say how they know.

The whole thing has two stages.

1. Preparation. Documents are split into passages. Each passage is turned into a string of numbers that describes its meaning. That produces an index – a table of contents for everything the business knows.
2. The question. The system finds the few passages closest to the question and hands them to the model. The model writes its answer from those alone and cites its sources.

Five things that separate good RAG from a toy

1. Reading documents properly. A scanned invoice, a PDF with a table, an old contract in Word. All of it first has to be turned into clean text. Rubbish in, rubbish out.
2. Splitting with care. A passage has to make sense by itself and deal with one matter. We split by paragraph, with a small overlap, so as not to cut a thought in half.
3. Searching two ways at once. By words you find a contract number, a surname and a product code. By meaning you find "reduction" when you ask about "discount". Each way on its own misses some of the hits.
4. A second pair of eyes. A separate, small model assesses the passages found and keeps the best. It is the cheapest way to get noticeably better answers.
5. Sources and an honest "I don't know". Every answer shows the document it comes from. And when there is nothing in the documents, a good system says "I found nothing" instead of making things up.

Where your data is in all this

This is the most important question when choosing a provider. Your data passes through three places:

• The index. It contains passages from your documents, so it is your data too.
• The model that turns text into numbers. It sees every passage of every document.
• The model that writes the answer. It sees the question and the few passages found.

Your level of privacy depends on where those three elements sit.

• Level 0: a public chat. You paste documents into a publicly available service. Convenient, but you do not control what happens to them.
• Level 1: knowledge with you, model in the cloud. Documents, index and search stay in a data centre you trust. Only the question and a few passages go to a large model in the cloud. The whole base never leaves.
• Level 2: everything local. The model that writes the answer also runs on your provider's own graphics cards. Nothing leaves the data centre.

For a price list and user manuals, level 1 is usually enough. For contracts, personal data, HR or medical records, choose level 2. Models that can be run on your own hardware – the open Llama, Qwen and Mistral, for example – now cope well with document work: they search, summarise and answer in plain English.

How we do it at Smartx

Our Memory is exactly that: RAG in a version for businesses. You upload documents and ask in plain English, from inside ChatGPT too.

• One company, one vault. Every customer has a separate index and separate files. No other customer can look inside.
• We do the computing ourselves. Turning text into numbers and assessing relevance is done by our own graphics cards in a UK or EU data centre, on open models. We do not send documents anywhere for this.
• A choice of model. When a local model is to write the answer, we run it on our cards. When you prefer a large cloud model, it receives only the question and a few passages, and you know that in advance.
• A contract and a certificate. We process data under a data processing agreement. Our information security is confirmed by an ISO/IEC 27001 certificate.

What this means for you

1. Start by tidying up. Current versions, one place, clear file names.
2. Ask about the three places. Where the index sits, where the numbers are produced and where the model runs. If the answer is "in the cloud" three times over, your documents end up outside your business in their entirety.
3. Insist on sources. Without them you cannot check whether the AI is making things up.