RAG vs fine-tuning: a knowledge gap or a behaviour gap

The term comes from a 2020 paper by Patrick Lewis and eleven co-authors, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". Its RAG models combined two kinds of memory: a pre-trained sequence-to-sequence model (parametric memory) and a dense vector index of Wikipedia read by a neural retriever (non-parametric memory). The authors reported that RAG produced more specific, diverse and factual language than a parametric-only baseline.

Fine-tuning changes the parametric memory itself. IBM describes it as exposing a model to a data set of labelled examples so that it performs better on a domain-specific task. RAG, in IBM’s words, plugs a model into stores of current, private data that would otherwise be out of its reach, at the moment a question is asked.

One nuance: the original paper trained the retriever and generator together, describing a general-purpose fine-tuning recipe for RAG. Most RAG systems today add retrieval at query time to a hosted model whose weights never change. When people compare RAG with fine-tuning, they mean that second, query-time form.

OpenAI’s guide to optimising accuracy splits the work into two axes. Context optimisation is for when the model lacks knowledge because it was not in its training set, when its knowledge is out of date, or when it needs proprietary information; this axis maximises response accuracy. LLM optimisation is for when results are inconsistent or wrongly formatted, the tone or style is wrong, or reasoning is not followed consistently; this axis maximises consistency.

The guide compares RAG to having the textbook with you during an exam, and fine-tuning to months of attending the class. It also describes a case, correcting Icelandic text, where adding RAG lowered accuracy because the problem was about behaviour, not missing knowledge. The lesson the guide draws is to use the right tool for the right job, after starting with prompt engineering.

An HR team wants an assistant that answers questions about leave, and the policy changes every year. This is a knowledge problem. With RAG, the team indexes the policy documents, retrieves the sections that match each question and asks the model to answer from them and name the section. When the policy changes, the team re-indexes the new document the same day. A fine-tuned model would keep giving last year’s answer until someone trained it again, and it could not point to the paragraph it relied on.

A second problem appears later: answers must follow a fixed template so that the HR ticketing system can file them, and the model keeps drifting from it. That is a behaviour problem. The team first adds worked examples to the prompt and measures the result on a test set. Only if the drift remains does it fine-tune on a set of approved answers, and it keeps RAG in place for the facts.

The two approaches cost money in different places, and neither is free to run well.

• Freshness: RAG reads current documents at query time. A fine-tuned model knows what was in its training data until you train it again.
• Compute: IBM says fine-tuning is compute-intensive and usually needs several powerful GPUs, although parameter-efficient fine-tuning (PEFT) lets teams retrain on simpler hardware.
• Upkeep: IBM notes that RAG systems need extensive data architecture to be built and maintained, such as ingestion, chunking, an index and refresh jobs.
• Access: IBM says data pipelines need restrictions so that employees cannot reach data beyond their role. Retrieval can filter by the asking user; knowledge trained into weights cannot be filtered per user afterwards.
• Provenance: the RAG paper names providing provenance as an open problem for models that keep knowledge only in their parameters. Retrieval lets an answer cite the passage it used.

RAG fails when retrieval fails. The index returns the wrong chunk, the index is stale, or the question needs facts spread across many documents. The model then answers fluently from whatever it was given, so a wrong retrieval looks like a confident answer. Check retrieval quality on its own, not only the final text.

Fine-tuning fails when it is used to add knowledge. The model may learn the style of the training answers without reliably learning the facts, and it has no source to show. It also fails quietly when the training examples carry mistakes, because the model learns those too. Both approaches need an evaluation set that you run before and after every change.

Cortex, Swfte’s governed AI desktop, includes knowledge bases with RAG on the device. You add files, folders, URLs and notes, and Cortex answers from them. Embeddings can run locally through Ollama, so the documents do not have to leave the machine.

Swfte does not fine-tune models for you today. The Model Vault, reached from Studio, lets you upload weights you trained elsewhere, version them and deploy them to a dedicated endpoint behind Connect, but no training pipeline is built. If you need fine-tuning, run it with your model provider or your own tooling and bring the result.

• Swfte Cortex: on-device knowledge bases
• Custom models: what is built and what is not
• How to build a RAG system

Facts about other vendors and about the terms on this page were read on the pages below on 7 October 2026. Vendor plans, names and menus change, so check the vendor’s page before you rely on a detail.