The AI system was approved on Friday. On Monday, somebody adds a connector. Same product name, same dashboard, same reassuring green tick. Different system.
That tick is doing a lot of unpaid overtime.
AI change control needs a better question than “has this been approved?” Ask which version, purpose and permissions the approval actually covered. Otherwise, a decision about yesterday’s product quietly becomes permission for tomorrow’s.
Give the approval something concrete to describe
Consider an illustrative support assistant. It searches a help centre and drafts replies for a person to send. The team evaluates answer quality, checks access to documents and approves that workflow.
Later, someone connects the assistant directly to the customer messaging service. The model is unchanged. The answers may even be identical. But the system can now send them without the same human checkpoint.
The previous evidence has not become worthless. It has become incomplete. A good answer test cannot establish that a message goes to the right recipient, requires approval or stops when permission is withdrawn.
This is the narrower problem behind putting governance into the release process: deciding which evidence survives a change, and which conclusions need reopening.
Compare the assumptions, not just the release notes
I would keep a short approval record beside the deployment configuration. It should name the intended task, users, data sources, model version, available tools and permitted actions. Link the evaluation results and record who accepted the remaining risk. Include a review date and conditions that require an earlier review.
This is a proposed operating practice, not a universal compliance checklist. The useful part is the relationship between an assumption and the evidence supporting it.
For example, “a person checks every outbound message” supports a different decision from “a person can inspect messages afterwards”. Those sentences are close enough to disappear in a slide deck. Operationally, they are miles apart.
Four changes that deserve different questions
A model or prompt change asks whether behaviour still meets the agreed criteria. Keep relevant baseline cases, add cases that exercise the changed behaviour and compare failures. OpenAI’s evaluation guidance recommends running evaluations on every change and expanding the dataset over time. Passing the old set is useful evidence, but it cannot test a scenario the set never contained.
A tool or permission change asks what new consequences are reachable. For the support assistant, test the recipient boundary, approval requirement and cancellation path. Inspect the enforced permission, too. A polite instruction saying “please don’t” is an awkward substitute for a control.
A data change asks whether access, relevance and handling assumptions still hold. Adding internal support tickets to a public help-centre search changes more than the volume of searchable text. Check which users can retrieve which records and whether the existing evaluation cases represent the new material.
A use-case change asks whether the original objective and consequences still describe the product. An assistant used to explain a policy and one used to decide an exception need different scrutiny, even if somebody reused the same prompt. Copy-paste does not transfer accountability :)
AI change control should preserve useful evidence
Reopening everything for every small edit creates a queue nobody respects. Reopening nothing creates an approval nobody should trust. Good AI change control needs a proportionate middle.
For each proposed change, record the affected assumption, the evidence still applicable, the evidence missing and the person who can authorise release. A cosmetic label change might leave the existing behavioural evidence intact. Removing a human checkpoint should reopen the decision that depended on it.
NIST’s GOVERN 1.5 calls for ongoing monitoring, periodic review and clear responsibilities. Its MANAGE 4.1 includes change management within post-deployment monitoring. Neither means that every edit needs an identical committee meeting.
Release the changed capability only when its required evidence and decision are ready. If they are not, keep that capability disabled or retain the previous workflow. Record the limit explicitly rather than letting a deadline become an accidental risk acceptance.
Then carry what you learn back into the product’s evaluation set. The next review should start with better evidence, not another hunt for the person who remembers what the green tick meant.
