RAG vs fine-tuning for company knowledge

Key takeaways

RAG looks up the relevant documents at question time and gives them to the model, so answers can cite sources and change when the documents change.
Fine-tuning changes the model's weights, which suits consistent style, format and narrow classification, but is a poor way to store facts.
Facts baked into model weights cannot be filtered by user permissions or selectively deleted, which matters for GDPR and for confidential material.
A RAG system is only as good as its retrieval, so test it with real questions and measure whether the right passage is found before you judge the answers.

Should you start with RAG or fine-tuning?

Use RAG when your assistant needs current company documents, source citations and document permissions. Consider fine-tuning when tests show that examples improve a repeated task, style or output format. You can combine them. First measure a basic retrieval system so you know whether the problem is finding information or using it correctly.

How do the options compare?

Aspect	RAG	Fine-tuning	Putting everything in the prompt
Best for	Answering from policies, manuals, records and knowledge bases	Consistent tone, output format, classification and narrow tasks	Small, fixed sets of documents and quick experiments
Updating knowledge	Edit the document and re-index	Retrain the model	Change the prompt
Source citations	Natural: the answer comes from named passages	Not available: the model cannot show where it learned something	Possible, if you label the text
Per-user access control	Filter at retrieval so people only see what their role allows	Not possible: knowledge is shared across all users	Only by building a different prompt per user
Deleting information	Remove the document from the index	No dependable document-level deletion from shared model weights	Remove it from the prompt
Typical failure	The right passage is not retrieved	Confident wrong facts and drift from your documents	Long prompts get costly and models use the middle of long inputs less reliably

When should you choose RAG?

Your knowledge changes: price lists, policies, procedures, clinical guidance, contracts.
People need to see where an answer came from, to check it or to cite it.
Different people may see different documents.
Some information must be removable, for example when a person exercises the right to erasure.
You want to test document retrieval before investing in model training.

When does fine-tuning make sense?

You need the model to follow a fixed output format reliably, such as a structured clinical note or a JSON schema.
You need a house style or voice across thousands of outputs.
The task is narrow and repeated, such as classifying tickets or extracting fields, and a smaller, cheaper model can do it once tuned.
You have enough clean, representative examples, reviewed by a person who knows the work.

Fine-tuning is not a dependable document database. It can change model behaviour, but it does not provide document-level access control or reliable provenance for facts learned during training.

Why facts in the weights are a privacy problem

When personal or confidential data goes into training, it can end up inside the model in ways that are hard to find and hard to remove. The European Data Protection Board's Opinion 28/2024 says that a model trained on personal data is not automatically anonymous, and that whether it is has to be assessed case by case. That makes fine-tuning on documents full of personal data a decision for a data protection impact assessment, not a technical shortcut.

What a good RAG system contains

Clean sources: current, deduplicated documents with an owner and a date, because the system cannot be better than the files.
Structure-aware chunking: split by headings, clauses and tables, not at arbitrary character counts.
Hybrid search: combine keyword and vector search so that exact terms, codes and names are found as well as paraphrases.
Re-ranking: a second pass that orders the candidate passages by how well they answer the question.
Permission filtering at retrieval: the user's role decides which documents can be searched at all.
Citations in every answer, linking to the passage and document.
An honest fallback: when retrieval is weak, the system says it does not know instead of guessing.
Redaction before the model call and an audit log of questions, passages and answers.

Where RAG goes wrong

Failure	Cause	Fix
Right document, wrong passage	Chunks too large or too small	Chunk by structure and test sizes against real questions
Outdated answers	Old versions still indexed	Re-index on change and show document dates
Missing numbers from tables or scanned PDFs	Poor parsing	Use a parser that keeps table structure and check scanned pages by hand
Confident answer unsupported by the passages	The model fills gaps from its own memory	Instruct it to answer only from the passages, require citations and check them
One person sees another team's documents	Access rules applied after retrieval, or not at all	Filter by permission before search

How do you know it works?

Collect a few dozen real questions from the people who will use the system, and for each one write down which document should answer it. Run them through the system and check two things separately: was the right passage retrieved, and was the answer correct and faithful to it. Retrieval problems and generation problems need different fixes, and mixing them up wastes weeks.

Share your document types and examples of the answers you need. We can help you test retrieval and decide whether fine-tuning adds value.

Plan your company knowledge assistant

FAQ

Can RAG run without sending data outside the EU?

Yes. The vector index, the documents and the model can all run in the EU or on your own servers. Personal details can also be redacted before the model sees the passages.

Do we need fine-tuning to handle our industry's terminology?

Usually not at first. A glossary, good retrieval and clear instructions cover most terminology. Fine-tuning becomes worth considering when a narrow, repeated task needs a fixed format or a cheaper model.

Can we combine RAG and fine-tuning?

Yes. A common pattern is RAG for the facts and a lightly tuned model for format and tone. Add the second step only after the first one is measured and working.

Sources

About the author

Artem Synytskyi · Co-founder · AI

Artem runs AI at ESADEX, building LLM, RAG and speech-to-text systems for regulated environments.

RAG vs fine-tuning: how to give an LLM your company's knowledge