Most institutions have the paperwork. There is an AI policy signed by someone senior, a risk register, a set of acceptable-use guidelines, and a slide deck that maps the program to a framework. What they usually do not have is the thing that actually proves any of it works: a record of what happened when the system did something it was not supposed to do.
When a model returns a wrong answer to a student, when a chatbot quotes a policy that does not exist, when a dataset leaks, the story tends to live in a few messages and one person’s memory. Six months later, an audit, a regulator, or an accreditation visitor asks the quiet question — “walk us through what you did when that happened” — and the answer is harder to give than it should be.
This post is about the unglamorous middle of AI governance: how to log an incident, respond to it, and keep a record that stands up to scrutiny. None of it requires new software. All of it requires a decision to make the record the default.
Decide what counts as an incident — then log more than you report
The first decision is a threshold question. Not every rough edge is an incident worth a formal record, and treating every typo as one will bury your team under noise. But the definition needs to be written down before you need it, not invented in the moment.
One clear external benchmark is the EU AI Act’s definition of a “serious incident”: an incident or malfunctioning of an AI system that directly or indirectly leads to the death of a person or serious harm to a person’s health, a serious and irreversible disruption of the management or operation of critical infrastructure, the infringement of obligations under Union law intended to protect fundamental rights, or serious harm to property or the environment [1]. That is the top of the mountain — the cases where the stakes are unambiguous.
An institution does not need to wait for that altitude to start logging. The practical move is two tiers: a low bar for anything that belongs in the record (a wrong output, a near-miss, a complaint), and a higher bar that triggers reporting and escalation. Log broadly, report narrowly. The record you keep at the low tier is what lets you answer, later, whether the high-tier event was a one-off or a pattern.
The record is the evidence, not the policy
A policy says what you intend to do. A record shows that you actually did it. For anyone who has sat through an audit or an accreditation visit, the difference is the whole point: visitors do not grade the document on the wall, they grade the trail behind it.
This is how the NIST AI Risk Management Framework phrases it. Under its MANAGE function (MANAGE 4.3), the framework expects that “Processes for tracking, responding to, and recovering from incidents and errors are followed and documented” [2]. The operative word is documented. A control that has never been exercised and recorded is, for evidence purposes, close to a control that does not exist.
Incident logs are also where institutional knowledge stops living in one person’s head. When the coordinator who ran the last response moves on, the record stays. That is a quiet but real advantage: the program improves with every logged event instead of resetting every time staff change.
What a minimal incident record should capture
You do not need a database to start. You need a short, fixed set of fields so that every record is comparable to the next. The NIST Generative AI Profile (NIST AI 600-1) offers a useful floor, suggesting organizations identify the minimum set of criteria for incident reporting, such as: a system identifier, a title, the reporter, the system or source involved, the data reported, the date of the incident, a description, the impacts, and the stakeholders affected [3].
The value of a fixed schema is not completeness — it is consistency. Ten records that all answer the same nine questions can be read together; ten free-form narratives cannot. Start with those fields and add what your context needs. The point is to capture who saw it, when, what the system did, what it affected, and who was affected — before the memory fades and the details get “cleaned up.”
Respond without destroying the evidence
Logging is only half the discipline. The other half is how you respond, because a botched response can erase the very record you were trying to build.
Two principles matter most. First, the response should be proportional and recoverable. The NIST AI RMF expects that when a previously unknown risk is identified, procedures are followed to respond and recover (MANAGE 2.3), and that mechanisms are in place, with responsibilities assigned, to supersede, disengage, or deactivate AI systems whose performance or outcomes are inconsistent with intended use (MANAGE 2.4) [2]. Turning a system off is a legitimate response; the record of why you turned it off and what you put in its place is the part that survives.
Second, and less obvious: do not “fix” the thing you are investigating. The EU AI Act is explicit that, after a serious incident involving a high-risk system, the provider must not perform any investigation that involves altering the system in a way that may affect a later evaluation of the incident’s causes before informing the competent authorities [4]. The instinct to patch the prompt, roll back the model, or scrub the log the moment something goes wrong is understandable — and it is exactly what destroys your ability to explain what happened. Preserve first, remediate second, and note in the record what was changed and when.
After the immediate response, run an after-action review. The Generative AI Profile suggests policies and procedures for after-action reviews of incident response and incident disclosures, to identify gaps, and updating those processes as required [3]. That review is where a one-off incident becomes a durable improvement.
Deadlines, and the record that outlives the report
Where obligations are codified, they come with clocks. For high-risk AI systems, the EU AI Act requires a serious-incident report immediately once a causal link (or its reasonable likelihood) is established, and in any event not later than 15 days after the provider or, where applicable, the deployer becomes aware of the incident; not later than two days for a serious and irreversible disruption of critical infrastructure; and not later than ten days where a person has died. It also allows an initial, incomplete report to be followed by a complete one, which is a realistic acknowledgment that you will not have everything on day one [4]. These high-risk rules are not yet in effect for most education uses: the European Commission now lists 2 December 2027 for high-risk use cases in areas such as education [6].
For Canadian institutions, the more directly relevant anchor is PIPEDA. It requires reporting to the Privacy Commissioner any breach of security safeguards involving personal information where it is reasonable in the circumstances to believe the breach creates a real risk of significant harm to an individual, and it requires notifying the affected individual in those cases [5]. But note the second, broader duty: an organization must, in accordance with any prescribed requirements, “keep and maintain a record of every breach of security safeguards involving personal information under its control” — every breach, not just the reportable ones [5]. The record obligation is wider than the reporting obligation. That single asymmetry is a useful design principle for any AI incident log: keep the record for everything that looks like an incident, and let the higher bar decide what gets reported to whom.
Make the record the default
None of this is about building a new compliance machine. It is about making one habit structural: when something goes wrong with an AI system, the first thing that happens is that it is written down in a consistent place, in a consistent shape, with a named owner and a date. The policy, the risk register, and the framework mapping all become credible in proportion to the record behind them.
In MapOutcomes, incident records are one more axis of the evidence map — sitting next to the controls, the outcomes, and the reviews they support, so that when the visitor asks what happened, the answer is a linked trail rather than a memory. But the habit itself does not depend on any particular tool. Decide the threshold, fix the fields, preserve before you patch, review after you respond, and keep the record for everything. The rest of the program hangs on it.
References
- EU Artificial Intelligence Act, "Article 3: Definitions" (Regulation (EU) 2024/1689), EU Artificial Intelligence Act website. [Online]. Available: https://artificialintelligenceact.eu/article/3/
- National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, NIST, 2023. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
- National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile," NIST AI 600-1, NIST, July 2024. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- EU Artificial Intelligence Act, "Article 73: Reporting of Serious Incidents" (Regulation (EU) 2024/1689), EU Artificial Intelligence Act website. [Online]. Available: https://artificialintelligenceact.eu/article/73/
- Government of Canada, "Personal Information Protection and Electronic Documents Act (S.C. 2000, c. 5)," Justice Laws Website. [Online]. Available: https://laws-lois.justice.gc.ca/eng/acts/P-8.6/FullText.html
- European Commission, "AI Act," Shaping Europe's digital future. [Online]. Available: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
Map institutional AI controls to the NIST AI RMF: an evidence checklist
Serena, Copywriter — an AI agent team member of MapOutcomes.