Fine-Tuning vs RAG vs Prompting

Prompting, RAG and fine-tuning: choosing the right lever at the right time

Prompting, RAG and fine-tuning are not three competing camps but three levers with rising cost and rising control. The right choice starts at the cheapest level and climbs only when the problem forces it.

Anyone starting an AI project soon faces the same build decision: should the language model be guided with a better prompt, supplied with an in-house knowledge base, or retrained itself? The three answers do not differ in quality but in the balance of effort, freshness and steerability. This page orders the three levers, names the criteria that actually decide, and shows in the decision tree when each one applies. The input layer itself, how a prompt is built and the context assembled, is covered by the page on prompt and context engineering; here the question is the choice between the levers.

The three levers

• Prompting. The model stays unchanged; steering happens through the input alone, that is instruction, examples and format. The cheapest and fastest lever, changeable instantly, with no infrastructure. Its limit is knowledge: the model knows only what was in its training and what is in the prompt.
• RAG. Retrieval-augmented generation supplies the model at runtime with relevant documents from the organisation's own knowledge base, which it folds into its answer. The model thus knows current and private content it never learned, and can cite its source. The middle lever: more setup than prompting, but updatable in seconds. The fundamentals are described in GenAI and RAG.
• Fine-Tuning. Here the weights of the model itself are retrained on the organisation's own data. This deeply imprints behaviour, tone and format, for example a consistent domain vocabulary or a strict output schema. Usually the most expensive lever, with its own training run and its own data upkeep, and the slowest, because new knowledge needs a new run.

The key rule of thumb behind this: fine-tuning changes how a model answers, RAG changes what it answers with. For pure factual knowledge that changes, retraining is the wrong lever; RAG is built for exactly that.

The criteria that actually decide

Four questions separate the levers in practice:

• Data freshness. Does the knowledge change daily or stand fixed? Fresh, frequently changing content belongs in RAG, where an update costs seconds rather than a training run. Static behaviour can move into the model.
• Type of need. Is it about knowledge (facts, documents, evidence) or about form (style, tone, output structure)? Knowledge is the domain of RAG, form that of fine-tuning.
• Cost and data volume. Fine-tuning needs a sufficiently large, clean training set and a repeatable run. Missing either one, the effort is rarely justified; prompting and RAG get by with far less.
• Control and evidence. RAG provides a source reference, if the pipeline returns the sources, and makes answers verifiable, which often tips the balance for regulated environments. Fine-tuning improves consistency without giving the same traceability.

Where the answers must stay on sovereign infrastructure, so that private data moves neither into a foreign cloud nor into foreign training runs, that is a separate, upstream criterion. RAG and fine-tuning can both run entirely on in-house infrastructure; the strategic weighing behind it, build or buy, is covered by Make or Buy.

The decision flow

The tree below maps the usual order: the cheapest level comes first and the next is reached only when a criterion forces it. It is a heuristic, not a law, and the paths do not exclude each other.

flowchart TD
    A["Task defined"] --> B{"Does it need current<br/>or private knowledge?"}
    B -->|"No"| C{"Is a good prompt enough<br/>for quality and format?"}
    C -->|"Yes"| P["Prompting<br/>cheap, instantly changeable"]
    C -->|"No, form must hold"| F["Fine-Tuning<br/>imprint style and schema"]
    B -->|"Yes"| R["RAG<br/>connect knowledge base, cite source"]
    R --> G{"Does style or format<br/>still stay unreliable?"}
    G -->|"No"| DONE["Done"]
    G -->|"Yes"| FR["RAG plus Fine-Tuning<br/>solve knowledge and form separately"]

The most common mistake sits right at the top: starting with fine-tuning because it sounds the most powerful. In most cases the cheapest fitting level solves the problem, and a premature training run spends money and time on a result that a RAG retrieval would have delivered more cheaply and more freshly.

Combine rather than either-or

In practice the question is rarely exclusive. A production system usually uses all three levers at once: a carefully built prompt as the foundation, RAG for current and verifiable knowledge, and fine-tuning only where form or domain vocabulary must hold reliably. This holds for agentic systems too, where a model calls tools: there as well RAG is the usual knowledge source and fine-tuning the exception for imprinted behaviour.

The cost shifts in doing so from the one-off training run to the ongoing operation of the knowledge base and the input pipeline. That pipeline is precisely the lever with the best ratio of impact to effort, which makes the clean management of prompts and context the real standing task. Which level a concrete project needs we settle in AI development; for building a verifiable knowledge base run on Swiss infrastructure there is the Sovereign RAG Switzerland competency, and a scoped entry point is the Enterprise RAG Proof of Concept. Which models are approved at all and on which data they may run is governed by AI governance, resting on the ongoing market assessment of the Tech Radar and AI Governance service.

References

• medevel.com LLM Engineer Toolkit, 120+ libraries. Curated tool collection with dedicated categories for fine-tuning (Unsloth, PEFT, LitGPT), RAG and prompting. (14.01.2026). medevel.com/llm-engineer-toolkit/
• Unsloth Open-source fine-tuning for open models. Tool for efficiently retraining open models with markedly reduced memory needs; check the licence per component (the repo carries both Apache-2.0 and AGPL-3.0). (2026). github.com/unslothai/unsloth
• KalyanKS-NLP llm-engineer-toolkit, repository. Library list ordered by workflow phase, from training and fine-tuning through RAG to prompting and monitoring. (2026). github.com/KalyanKS-NLP/llm-engineer-toolkit
• Hugging Face PEFT, parameter-efficient fine-tuning. Library that adapts large models without training all weights, sharply lowering the cost of a full fine-tuning. (2025). huggingface.co/docs/peft/index
• LangChain RAG From Scratch. Teaching series on retrieval-augmented generation that records fine-tuning as poorly suited for factual recall and costly, with RAG closing that gap. (2025). github.com/langchain-ai/rag-from-scratch

---

Related topics

• GenAI and RAG, the middle lever in detail, from retrieval to source citation.
• AI development, where the choice of lever is translated into a concrete project.
• Make or Buy, the strategic weighing between building yourself and buying.
• AI governance, the approval of models and data flows every lever builds on.
• Sovereign RAG Switzerland, the commercial counterpart for a verifiable knowledge base.