The vendor says you can export everything. Excellent. Picture an old phone migration: half the contacts vanish and three people become “Unknown”.
AI evidence portability has the same problem. Moving records is useful only if the next team can understand what those records establish. A folder of traces does not explain which release they came from, what a score meant or whether somebody actually checked the outcome.
Before a model or vendor change, preserve the relationships between the evidence. Keep historical findings intact, then generate new evidence for the replacement. A successful import cannot transfer an old approval to a new system.
Give every result an address
Consider an illustrative invoice assistant that extracts payment details and prepares a draft for review. Its old evaluation reports a strong pass rate. After a vendor switch, the new team receives a spreadsheet of scores and a pile of conversation logs. The spreadsheet has no case identifiers. The logs have no release identifiers. Lovely collection. Terrible evidence.
I would start with a small manifest: a versioned index that connects each test case to its input fixture, expected outcome, run, grader and result. Connect the run to the model identifier, prompt version, tool configuration and relevant data snapshot. Use stable identifiers rather than dashboard links that disappear when the account closes.
Preserve the source export alongside any converted copy. Record the export date, schema version, conversion rules and known omissions. A checksum can help detect a changed file; it cannot establish that the original observation was correct. Provenance needs context, not just a very confident string of letters.
Separate the record from the interpretation
Keep three things distinct: what the system did, how it was judged and what someone decided. A trace might show a proposed bank-account change. A grader might mark the extraction correct. A release reviewer might approve drafting only, with payment execution excluded. Flatten those into one green “pass” and the most important boundary disappears.
Anthropic’s evaluation guide distinguishes the transcript from the final state of the environment. For this assistant, a reassuring message is different from a correctly stored draft. Preserve the evidence used to verify that state, with appropriate access controls, rather than relying on the assistant’s own summary.
Check what the export actually carries
There are useful technical building blocks. MLflow documents trace export through OpenTelemetry, including an option that translates model information, messages and token usage into GenAI attributes. That is a transport mechanism. It does not establish that your evaluation rubrics, reviewer decisions or linked business records travelled with it.
Inspect the receiving system. Can it still distinguish an absent field from an empty one? Do timestamps preserve their meaning? Can a reviewer resolve the relationship between a parent run and its tool calls? Check a failed case and an interrupted case as well as the tidy success. Success is usually the easiest guest to photograph.
Record missing information explicitly. If a provider never exposed a model revision, write “not supplied”; do not retrofit a plausible version. If a referenced document was deleted under a retention policy, retain the permitted metadata and explain the limit. Evidence portability is not permission to copy secrets or keep personal data forever.
Carry the test, rerun the conclusion
The original cases, rubrics and historical results can remain useful after migration. They are a baseline, not proof that the replacement behaves identically. Run the relevant cases again under recorded conditions. Keep old and new results side by side, including failures and untested behaviour, without silently overwriting history.
Check that the grader still means the same thing. If the rubric or judge changed too, a higher score may reflect a different measuring stick. The practical checks in evaluating an LLM judge help separate that problem from a genuine improvement. Repeat trials where variability matters and record the sampling limits.
NIST’s Manage playbook recommends organised, secure risk documentation accessible to appropriate personnel. The manifest and checks here are my proposed implementation practice, not a NIST certification or a universal legal requirement.
Try the handover without the tour guide
Ask a colleague who did not build the original workflow to reconstruct one release decision from the exported package. Which version was tested? What failed? What remained untested? Who accepted which limitation? If they need the former vendor’s dashboard or somebody’s memory, record the dependency.
This is a narrower exercise than the full sovereign AI exit rehearsal. It checks whether your evidence remains intelligible. Start with one decision and follow every supporting reference. Exporting the files is the easy part. Keeping their meaning is the product work.
