Data Governance
Data Governance defines the policies, responsibilities, and standards for handling data within an organisation. Data Observability is the technical implementation: the continuous monitoring of data quality and data flows (Lineage) in real time.
In a world of Data Mesh and decentralised teams, governance is no longer centralist and blocking — it is agile and enabling: teams get Self-service rules to share data safely.
Anti-Patterns: Data Anarchy
- Unclear data ownership: Nobody knows who is responsible for correcting wrong customer addresses in the system.
- Silo formation: Each department defines terms like "revenue" or "customer" differently, leading to contradictory reports.
- Silent Data Failure: A data pipeline breaks unnoticed or delivers incorrect values (e.g. due to a schema update), only surfacing weeks later in the balance sheet.
The Governance Framework
- Data Stewardship: Designation of domain owners (Stewards) for each data object.
- Data Catalog: A central registry (e.g. via DataHub or Amundsen) where anyone can find which data exists and what it means.
- Data Lineage: Automatic visualisation of data flow: "Where does this value in my dashboard come from and what transformations has it gone through?".
- Automated Data Quality: Integration of validation scripts (e.g. via Great Expectations) that check data for plausibility at the point of ingestion.
- Privacy & Access Control: Centralised management of access rights based on roles and data sensitivity (see Compliance).
The Focus: Data as a Reliable Product
Data Governance ensures that data has the same quality and reliability as the organisation's software products.
FAQ
Doesn't Data Governance slow down our agility?
On the contrary. Good governance gives teams the confidence to use data from other departments without fear of legal or quality issues. It accelerates collaboration.
How does Observability help with data protection?
Through Data Lineage, you can see exactly where personal data (PII) flows within the organisation and whether it is correctly protected or anonymised everywhere. This makes deletion requests (nFADP) significantly easier.
Reference Guide
- DAMA-DMBOK: The standard reference for Data Management. dama.org
- The Data Governance Institute: Resources and frameworks. datagovernance.com
- Data Quality Fundamentals: Barr Moses on Data Observability. O'Reilly