Documenting Fairness and Bias Checks for AI Used in Assessment and Admissions

Every institution that uses AI in assessment or admissions will eventually face the same question: “How do you know it’s treating applicants fairly?” The answer “we reviewed the model” is not a record. A defensible answer is a document — the fairness and bias check record — that shows which populations were compared, which criterion was applied, what the results were, and what was done about them.

This post is for coordinators and AI governance leads who need to produce that record: what it should contain, which requirements shape it, and how to keep it current after deployment. If you have not yet inventoried which AI systems your institution uses in high-stakes decisions, start with our guide to building an AI use-case inventory — a fairness check only exists for a system you can name.

Why assessment and admissions are the highest-stakes category

No category of AI use carries more legal or reputational weight than the ones that decide who gets in and how students are scored. The EU’s AI Act (Regulation (EU) 2024/1689) — one of the most detailed codifications of AI risk rules to date — classifies exactly these uses as high-risk. Its Annex III lists, in the area of education and vocational training [1]:

• AI systems intended to be used to determine access or admission, or to assign natural persons to educational and vocational training institutions at all levels;
• systems intended to evaluate learning outcomes, including when those outcomes are used to steer the learning process;
• systems intended to assess the appropriate level of education an individual will receive or be able to access; and
• systems intended to monitor and detect prohibited behaviour of students during tests.

The regulation entered into force on 1 August 2024, and — as amended by the 2026 simplification package known as the “AI Omnibus” — the European Commission states that the rules for high-risk use cases in areas including education apply from 2 December 2027 [2].

If your institution is not in the EU, the AI Act may not bind you directly. It still matters: it is the clearest public specification of what a regulator expects to see when AI decides educational outcomes — the European Commission’s own summary describes AI solutions used in education institutions as ones that “may determine the access to education and course of someone’s professional life” [2]. Boards, accreditors, and applicants elsewhere may ask for similar evidence, and building the record now is likely cheaper than rebuilding it when a new requirement lands.

What “fairness” means — and why you have to choose

The NIST AI Risk Management Framework, which many institutions use as their organizing framework, defines its fairness characteristic as “Fair – with Harmful Bias Managed” and is candid about the difficulty: standards of fairness “can be complex and difficult to define because perceptions of fairness differ among cultures and may shift depending on application” A system whose predictions are balanced across demographic groups is not automatically fair — it may still be inaccessible to individuals with disabilities, be affected by the digital divide, or exacerbate existing disparities or systemic biases (AI RMF 1.0, Section 3.7) [3].

That matters for documentation, because a fairness check that does not name its criterion is unfalsifiable. “We checked for bias” does not say what was compared against what standard. NIST also identifies three categories of bias to manage — systemic, computational and statistical, and human-cognitive — and notes that each can occur in the absence of prejudice or discriminatory intent. A check record should therefore cover all three, not just the model’s math:

• Systemic — how historical practices, organizational norms, and the broader context are reflected in the data the system learns from;
• Computational and statistical — errors introduced by non-representative samples or algorithmic processes;
• Human-cognitive — how the people who set thresholds, interpret outputs, and make final decisions perceive and rely on the system.

Choosing a criterion is a decision, and it belongs in the document. Whether you compare group pass rates, error rates, or calibration between groups, the record should state the criterion, why it fits this particular decision, and who approved that choice. Our NIST AI RMF evidence checklist places this work in the Measure function, where fairness and bias, as identified in the Map function, are to be “evaluated and results are documented” (MEASURE 2.11) [3].

The fairness check document: what to record

Here is the record, field by field. It can live as a one-page document per system per check cycle, versioned and signed off:

1. System and decision. The AI system, its version, and the specific decision it informs — admissions scoring, exam scoring, level placement, progression.
2. Population and comparison groups. Who the check covers, which groups are compared, and why those groups. Groups can be demographic, but they can also be first-generation applicants or students from particular school types — name them explicitly.
3. Criterion and rationale. The fairness criterion applied, in plain language, and why it is the right one for this decision context.
4. Threshold. The tolerance or pass/fail threshold, and who set it.
5. Results. The measured values by group. This is the part most institutions skip; without numbers, the check is an assertion.
6. Findings. Whether each group’s results fall within threshold, and any disparity identified.
7. Remediation. What was done or will be done about any disparity — data changes, threshold changes, model changes, added human review — and when it will be re-checked.
8. Accountability. Who performed the check, who reviewed it, who approved it, and the version of this document.

This structure is our suggested template, not a format either source prescribes, but it follows the logic of both. The AI Act’s data-governance requirements for high-risk systems call for an “examination in view of possible biases” that are likely to, among other things, lead to discrimination prohibited under Union law, plus “appropriate measures to detect, prevent and mitigate possible biases identified” (Article 10(2)(f)–(g)) [4]. NIST’s Measure function asks for fairness and bias to be evaluated and the results documented. The document above is one way to put both on the page.

Before deployment is not the end

A fairness check run once, before go-live, answers a question about the past. The populations your institution serves change every cycle, models drift, and upstream data shifts. The AI Act’s deployer obligations are explicit about this: deployers must monitor the operation of a high-risk system on the basis of its instructions for use (Article 26(5)), ensure input data is “relevant and sufficiently representative” in view of the intended purpose to the extent they exercise control over it (Article 26(4)), and keep the logs automatically generated by the system, to the extent those logs are under their control, for a period of at least six months unless other law provides otherwise (Article 26(6)) [5]. NIST’s Measure function similarly expects approaches to regularly identify and track existing, unanticipated, and emergent risks based on factors such as intended and actual performance in deployed contexts (MEASURE 3.1) [3].

In practice, that means a cadence: re-run the fairness check at least annually, and immediately after any material change — a new model version, a revised admissions rubric, a significant shift in the applicant population. Each cycle produces a new version of the document, so the history of checks itself becomes evidence that the institution was watching.

Human review and appeals: the other half of the record

A fairness check is only as defensible as the process around it. The AI Act requires deployers to assign human oversight to natural persons who have “the necessary competence, training and authority” (Article 26(2)) [5], and the system must be designed so that those people can remain aware of the possible tendency of automatically relying or over-relying on its output (“automation bias”) (Article 14(4)(b)) [6]. For assessment and admissions, that oversight should include a working appeals path: a way for an applicant or student to question a decision, and a record of how the appeal was handled. If your institution does not yet have a documented human-review trail for AI-assisted decisions, our guide to a defensible review trail is the natural companion to this one.

Where the record lives

Finally, the record needs a home. A fairness check that exists as an email attachment or a spreadsheet in one person’s drive is not an institutional record: it cannot be found when the question arrives, and it disappears when the coordinator leaves. The check document should be versioned, linked to the specific system and use case it covers, and retrievable by anyone with authority to ask about the decision. For institutions building an evidence map of their governance obligations, this is one more node in the same structure — the system, the check, the result, the remediation, and the person accountable for each.

MapOutcomes’ AI-governance vertical is built around exactly this pattern: connecting each AI use case to the controls, checks, and records that evidence it, so the answer to “how do you know it’s fair?” is a document you can open, not a promise you have to make.

References

1. EU Artificial Intelligence Act, "Annex III: High-Risk AI Systems Referred to in Article 6(2)" (Regulation (EU) 2024/1689), EU Artificial Intelligence Act website. [Online]. Available: https://artificialintelligenceact.eu/annex/3/
2. European Commission, "AI Act," Shaping Europe's digital future. [Online]. Available: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
3. National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, NIST, 2023. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
4. EU Artificial Intelligence Act, "Article 10: Data and Data Governance" (Regulation (EU) 2024/1689), EU Artificial Intelligence Act website. [Online]. Available: https://artificialintelligenceact.eu/article/10/
5. EU Artificial Intelligence Act, "Article 26: Obligations of Deployers of High-Risk AI Systems" (Regulation (EU) 2024/1689), EU Artificial Intelligence Act website. [Online]. Available: https://artificialintelligenceact.eu/article/26/
6. EU Artificial Intelligence Act, "Article 14: Human Oversight" (Regulation (EU) 2024/1689), EU Artificial Intelligence Act website. [Online]. Available: https://artificialintelligenceact.eu/article/14/