Published:

Data Governance

Clear rules make data shareable without blocking its use

Data Governance defines the policies, responsibilities, and standards for handling data within an organisation. Data Observability is the technical implementation: the continuous monitoring of data quality and data flows (Lineage) in real time.

In a world of Data Mesh and decentralised teams, governance is no longer centralist and blocking; it is agile and enabling: teams get Self-service rules to share data safely.

Anti-Patterns: Data Anarchy

  • Unclear data ownership: Nobody knows who is responsible for correcting wrong customer addresses in the system.
  • Silo formation: Each department defines terms like "revenue" or "customer" differently, leading to contradictory reports.
  • Silent Data Failure: A data pipeline breaks unnoticed or delivers incorrect values (e.g. due to a schema update), only surfacing weeks later in the balance sheet.

The Governance Framework

  1. Data Stewardship: Designation of domain owners (Stewards) for each data object.
  2. Data Catalog: A central registry (e.g. via DataHub or Amundsen) where anyone can find which data exists and what it means.
  3. Data Lineage: Automatic visualisation of data flow: "Where does this value in my dashboard come from and what transformations has it gone through?".
  4. Automated Data Quality: Integration of validation scripts (e.g. via Great Expectations) that check data for plausibility at the point of ingestion.
  5. Privacy and Access Control: Centralised management of access rights based on roles and data sensitivity (see Compliance).

The Focus: Data as a Reliable Product

Data Governance ensures that data has the same quality and reliability as the organisation's software products.

FAQ

Doesn't Data Governance slow down our agility?

On the contrary. Good governance gives teams the confidence to use data from other departments without fear of legal or quality issues. It accelerates collaboration.

How does Observability help with data protection?

Through Data Lineage, it becomes visible exactly where personal data (PII) flows within the organisation and whether it is correctly protected or anonymised everywhere. This makes deletion requests (nFADP) significantly easier.

References