Published: Last updated:

GenAI and RAG

RAG grounds the language model in verifiable company knowledge

Generative AI only realises its full value in an organisation when it has access to internal knowledge. Retrieval-Augmented Generation (RAG) is the architecture that connects a language model to company-specific documents, without those documents being used to train the model. The pattern alone does not guarantee data residency; a sovereign RAG deployment keeps the documents in Switzerland when every component (embedding model, vector store, LLM calls, logs, telemetry, backups, support access) is constrained accordingly.

The result: precise answers based on current facts, with source attribution, while meeting data privacy requirements and protecting business secrets.

What RAG is, and why now

Picture a language model as someone who speaks eloquently but only knows what was poured into it during training. An organisation's internal knowledge (contracts, wikis, code, last week's handbook) is not in there. RAG hands this speaker a reference work: before every answer, the system looks things up in the organisation's own documents and supplies the relevant passages. The model phrases the answer; the facts come from that source.

The obvious counter-question: is RAG still needed now that models process millions of tokens of context? The answer from practice is clear, yes. RAG remains the most reliable technique for grounding a model in current, verifiable company knowledge, regardless of model size or context window. A large context window replaces neither the currency of the data nor the traceability of the answer.

The alternative would be to retrain a model on the organisation's own data (fine-tuning). For dynamic company knowledge, that is the wrong lever: too slow and too expensive for content that changes daily, and without source attribution. RAG updates by re-indexing a document, often much faster than retraining, and every statement stays traceable to the original document.

The RAG pipeline: ingest, retrieve, augment, generate

RAG has two phases. Offline, the knowledge base is prepared (ingest); online, a question is answered (query). Both run inside the same sovereign infrastructure. The ingest side is, ultimately, data architecture in practice; how cleanly this pipeline is built decides every later answer.

flowchart TB
  subgraph SOV["Sovereign infrastructure (CH)"]
    direction TB
    subgraph ING["Offline: Ingest"]
      direction TB
      D["Documents:<br/>PDF, wiki, code"] --> C["Chunking"]
      C --> E["Embeddings"]
      E --> V[("Vector DB:<br/>pgvector")]
    end
    subgraph QRY["Online: Query"]
      direction TB
      Q["User question"] --> R["Retrieve:<br/>semantic + BM25"]
      R --> A["Augment:<br/>facts + question"]
      A --> G["LLM:<br/>open-weights"]
      G --> O["Answer<br/>with sources"]
    end
    V -.-> R
  end

The diagram shows the core: offline, documents are split into pieces (chunks), converted into vectors (embeddings) and stored in a vector database. Online, when a question arrives, the system finds the matching pieces, supplies them to the language model as context and lets the answer be phrased from them, all within the Swiss border (dashed: the database feeds retrieval without breaking the reading order).

In detail:

  • Ingest. Documents are split into semantically meaningful sections (chunking), converted into embeddings and stored in a vector database. The quality of this step determines everything that follows: cutting at section and table boundaries instead of blindly by character count improves later retrieval quality and makes the provenance of every answer traceable. Which vector database sits behind it is deepened in the Modern Databases topic.
  • Retrieve. For a question, the semantically closest sections are searched. Pure meaning-based search has a weakness: exact terms (product names, error codes, internal acronyms) sometimes get lost. An additional keyword-based search (BM25) catches exactly these cases; combining the two (hybrid search) delivers measurably better hits.
  • Augment. The retrieved facts are packaged together with the question into a prompt.
  • Generate. The model is instructed to ground its answer in the supplied facts, and cites them. It combines the retrieved context with its own model knowledge; RAG reduces the hallucination risk significantly but does not guarantee hallucination-free output. This source attribution is what makes the difference: for a grounded statement it is traceable which page of which document it comes from. That builds trust and makes deviations checkable. This is why evaluation that measures grounding belongs in from the start.

Data residency as an architectural property

With RAG, sovereignty is not an afterthought but an architectural decision. The diagram carries it through the enclosing box: vector database and language model run inside the Swiss infrastructure. The regulatory ground for why that matters sits in the revised Swiss Data Protection Act (nFADP) and in the data compliance of the Data Governance topic.

Concretely, that means a local vector database (pgvector or Weaviate) on in-house or Swiss cloud infrastructure, combined with an open-weights model from the landscape of Language Models (from the Llama or Mistral family, say) operated without data leakage, or, where an external interface is required, with a contractual training opt-out. Both routes share the same principle: sensitive documents do not leave Switzerland and are not used to train foreign models.

This is the point where a RAG pilot differs from "dumping data into a US cloud". The verifiable answer and controlled data residency are the same architecture here, not two separate requirements. How le dot puts this into practice is described on the solution page Sovereign RAG Architectures; this primer explains the concept, the solution page shows the implementation. When the choice of a CRM or ERP system is on the table, the Independent Software Evaluation structures that decision neutrally and free of vendor lock-in.

Where RAG breaks (failure modes)

Hallucination is only the best-known failure mode, and in a cleanly built RAG system it becomes less likely, because the answer is bound to supplied facts. The more expensive failures lie elsewhere, and the key point first: a wrong result in RAG is often not statistical noise but an error with a concrete cause, not always reducible to one deterministic bug, yet usually traceable. The chain can be walked backwards to find the broken link.

  • Stale index. The source changed, the index did not, and the system answers convincingly with yesterday. An additive, append-only crawl that never reconciles superseded content is the most common trap here.
  • Bad retrieval. The wrong section, or a missing one, is fetched and the answer goes wrong. That is a retrieval bug, not a model problem.
  • Brittle chunking / parser errors. The most expensive mistake: teams optimise the model, embeddings and rerankers for months, and the bug sat in the parser that split the document wrongly the whole time.
  • Silent failure. The most dangerous mode: classic RAG returns an answer even when retrieval has failed. The error is invisible, the answer reads plausibly. That is why honest evaluation (which source supported which statement, should the system have refused) belongs in from the start.

The tooling landscape

This section ages faster than the rest of the page, it is deliberately a snapshot, not a promise about specific versions.

  • Frameworks. Established toolkits like LangChain and LlamaIndex, plus integrated open-source engines (e.g. RAGFlow), cover pipeline construction. Broader surveys of the current field come from collections such as the LLM Engineer Toolkit.
  • Vector databases. The spectrum ranges from server-side (pgvector as a Postgres extension, Weaviate) to embedded, "SQLite-like" engines such as Zvec that enable on-device and edge RAG without a separate service. Which one fits depends on the operating model and data volume, not on popularity.
  • Evaluation. Production-grade RAG needs its own eval layer: not ML metrics over aggregates, but the question of whether the right source supported the right statement, per question, traceably.

A note on framing: RAG looks like machine learning but is not. The embedding model is indeed a deep-learning model, but it is not trained, it is consumed. The system around it improves through better indexing, parsing and retrieval, through engineering, not training. Confusing the two leads to reaching for the wrong toolkit.

When RAG fits, and when it does not

RAG fits when current, verifiable company knowledge is needed and compliance or data-privacy pressure applies, exactly where answers must be traceable and data residency-bound. Where the regulatory handling of AI itself is in scope, the EU AI Act topic sorts out the obligations. For those who prefer to test the concept against a concrete initiative, the Enterprise RAG Proof-of-Concept delivers a working prototype in ten days.

RAG is less necessary for static general knowledge the model already holds, or when a single lookup suffices and there is no large body of knowledge to search at all.

The next stage is taking shape as agentic RAG: instead of a linear pipeline (retrieve once, then answer), retrieval becomes a loop, retrieve, reason, and on a weak result retrieve again or call tools. This increases adaptability but costs predictability and makes debugging harder. For getting started, the classic pipeline remains the right, well-controllable point of departure.

References

Continuously maintained tools and platforms:

Related topics