AI Governance
Most AI systems do not fail technically. They fail at the point where somebody has to sign off an answer they cannot trace, or explain a decision they cannot reproduce. This track is about the controls that get an AI system out of the pilot and keep it there.
01 The syllabus
What this track covers
Eight topics, in the order they have to be done. Each one is a capability rather than a document, and each depends on the one before it.
The use-case register
A single inventory of every AI system in use, including the ones bought inside other software. You cannot govern a portfolio you have not enumerated.
Shadow AI is the default stateRisk classification
Tiering each use case by the consequence of it being wrong, so oversight is proportionate rather than uniform — the structure every modern AI regime shares.
Consequence, not sophisticationData provenance
Knowing what the system was trained on, or what it retrieves from, with the same lineage discipline applied to any reported number.
An agent inherits every unverified hopEvaluation
A test suite with expected answers, run on every change, covering the failure modes as well as the happy path. Without it, "it seems better" is the release criterion.
Evals are regression tests for behaviourHuman oversight
Deciding where a person must review, approve or be able to intervene, and making that possible in the interface rather than in a policy document.
Oversight has to be actionable to be realTransparency and disclosure
Telling people when they are interacting with an AI system, and being able to explain what informed a decision that affected them.
Explanation is a data problemMonitoring and drift
Watching quality, cost and behaviour in production, because the inputs change even when the model does not.
The world drifts; the model does not knowIncident response and rollback
A defined path for when an AI system is wrong at scale: how it is detected, who can stop it, and how you revert.
Deployability requires reversibility02 The frameworks
The frameworks, and what each asks you to be able to show
These converge more than they differ. All of them assume you can enumerate your AI systems, justify their inputs, evidence their testing and demonstrate oversight.
| Framework | What it is concerned with | What you must be able to show |
|---|---|---|
| EU AI Act | A risk-tiered regime for AI placed on the EU market, with the heaviest obligations on high-risk uses | Classification of each use case; data governance over training and input data; technical documentation; logging; human oversight; accuracy and robustness appropriate to the tier |
| NIST AI Risk Management Framework | A voluntary US framework structured as Govern, Map, Measure and Manage | A mapped inventory of AI risks per use case, measured with defined metrics and managed with named owners |
| ISO/IEC 42001 | A management-system standard for AI, in the shape of ISO 27001 | Documented objectives, controls and evidence that the AI management system operates as described |
| Model risk management (financial services) | Long-standing supervisory expectations for models used in decisions | Independent validation, effective challenge, documented limitations and ongoing performance monitoring |
| GDPR, on automated decisions | Decisions producing legal or similarly significant effects, made solely by automated means | A lawful basis, meaningful information about the logic involved, and a route to human review |
| Sector regimes (medical, safety-critical) | AI embedded in regulated products and clinical or safety decisions | Evidence under the existing product regime, which generally exceeds anything the AI-specific regimes require |
Summarised at the level of the obligation each framework creates for data. For the binding text, and for how it applies to you, take advice — the requirements are stable, the detail and the timelines are not.
03 The controls
The controls that make an AI system deployable
Each of these is a data capability before it is an AI one, which is why organisations with governed data ship AI faster and not more slowly.
| Control | What it does | What it depends on |
|---|---|---|
| Use-case register | Enumerates every AI system, its owner and its risk tier | A process that catches AI bought inside other software |
| Input governance | Controls what the model may be trained on or retrieve from | Classification, entitlement and purpose binding on the underlying data |
| Retrieval lineage | Shows which records produced a given answer | Lineage captured through the retrieval layer, not just the pipelines |
| Evaluation suite | Tests behaviour on known cases before and after every change | A governed set of questions with agreed correct answers |
| Guardrails | Blocks actions and disclosures outside what is permitted | Policy expressed as data the runtime can evaluate |
| Action logging | Records what an agent did, with what inputs, on whose behalf | Immutable logging at the tool and data boundary |
| Human-in-the-loop | Requires approval for defined categories of action | Risk tiering, so review is applied where consequence justifies it |
| Rollback | Reverts to a previous version quickly and completely | Versioned prompts, models, retrieval sets and configuration |
04 Sequence
From pilot to production without a governance programme first
Governance here is not a gate you build before starting. It is the set of things that make the second use case cheaper than the first.
05 What goes wrong
The four failures worth naming
Pilot purgatory
A working demonstration that never deploys, because nobody can approve an answer they cannot trace. The blocker is almost always governance of the inputs, not model quality.
An ungoverned retrieval corpus
Documents of unknown provenance, permission and currency. The model faithfully reports what the corpus says, including the parts that are three years out of date.
Evaluation on the happy path only
Tests that confirm what already works. The failure modes that matter — ambiguity, missing data, adversarial input — are the ones not covered.
No owner for the whole system
The model has a data scientist, the prompt has an engineer, the retrieval set has nobody, and the behaviour is the product of all three.
06 Diagnostics
Six questions to ask your own team
If two of these cannot be answered without a manual investigation, those two are the plan for the next quarter.
- Can you list every AI system in use, including the ones inside software you bought?
- For your highest-consequence use case, can you show what data informed a specific answer?
- Do you have an evaluation set with agreed correct answers, and does it run on every change?
- Who can stop an AI system in production, and how quickly?
- Is the retrieval corpus governed — owned, permissioned, dated — or is it a folder?
- Would you be able to explain a decision this system made, to the person it affected, six months from now?
07 Measure it
The metrics behind this track
08 Questions
Frequently asked
Does governance slow AI deployment down?
The opposite, after the first use case. What blocks deployment is almost never model quality — it is that nobody will sign off an answer whose inputs cannot be traced. Governed inputs and an evaluation set are what make approval a decision rather than an argument.
What is the minimum viable AI governance?
A register of use cases, a risk tier for each, governed inputs and an evaluation set for anything above the lowest tier, and a rollback path. That is proportionate for most organisations and defensible under every framework in the table above.
How does the EU AI Act change what we build?
Mostly it changes what you must be able to evidence, and only for higher-risk uses. The engineering consequence is documentation, logging, data governance over inputs and demonstrable human oversight — all of which are good practice regardless of jurisdiction.
Who should own AI governance?
The same function that owns data governance, with risk and legal as partners rather than owners. Splitting them produces two registers, two policies and two sets of lineage, none of which agree.
See these controls running
Classification, entitlement, lineage and policy applied to the data itself rather than to each report.