Black Sheep AI
An impact assessment isn't something you do for every model, and it isn't something you can skip because the model "seems fine." There's a specific set of triggers that make one mandatory. Here they are, with the reason each one is on the list.
The question we get most often about impact assessments isn't how to fill one out. It's whether this particular system needs one at all. Teams want a rule they can apply at intake without convening a committee, and they're right to, because "run an assessment on everything" is how the process dies of exhaustion and "assess it if someone's worried" is how the risky system slips through.
So we work from triggers, not vibes. An assessment becomes mandatory when a use case hits one of the conditions below. Most of these come straight from what the regulation requires; the rest are ones we've added because experience says they're where harm hides. The underlying document is our impact assessment methodology, which folds three assessment types into one process: the algorithmic impact assessment (AIA) for fairness and decision-making, the data protection impact assessment (DPIA) for privacy, and the human rights impact assessment (HRIA) for fundamental rights. A single trigger can pull in one or all three.
The trigger is the risk tier
The cleanest rule first: if a system lands in the high-risk tier, it needs a full assessment, no exceptions. That's not our preference, it's EU AI Act Article 9. So the first thing that mandates an assessment is the classification itself, which is why risk tiering runs before anything else. Anything that affects health, safety, fundamental rights, or access to essential services is high-risk by definition, and high-risk means assess.
That covers the obvious cases. The scenarios below are the ones that either sit inside "high-risk" and deserve calling out, or that trigger an assessment through a different door, usually privacy law, even when the system didn't feel high-risk at first glance.
The scenarios that mandate an assessment
A new high-risk use case is being built. Any new system in a high-risk domain, hiring, credit, medical diagnosis, education, biometric ID, critical infrastructure, gets a full assessment before the design is finalized. Not before launch. Before the design is locked, because by launch the decisions that create the risk have already been made and an assessment becomes a formality instead of a control.
The system makes or heavily influences decisions about people. If an automated decision produces a legal or similarly significant effect on someone, denying a loan, screening out a job applicant, flagging a benefits claim, it triggers an assessment. This is the ground GDPR Article 22 covers, and the reason is straightforward: a decision that changes a person's options is exactly the kind that has to be contestable, explainable, and reviewed by a human, and the assessment is where you prove those safeguards exist.
Personal or sensitive data is in play. Processing personal data at scale triggers a DPIA under GDPR Article 35, and special-category data, health, biometrics, race, political opinion, raises the stakes further. The trap here is inference. A system that never directly touches health data can still infer a health condition from purchasing patterns, and that inference can itself be special-category processing. If the model could reveal sensitive attributes even indirectly, assess it.
The use case is biometric. Biometric identification and categorization is its own high-risk category, and parts of it are prohibited outright. Accuracy varies across demographics in ways that produce real harm, wrongful matches fall unevenly on some groups, so biometric systems get an assessment regardless of how confident the vendor's accuracy numbers look.
Vulnerable groups are affected. When a system touches children, elderly people, people with disabilities, ethnic minorities, or low-income populations, the human rights assessment becomes central rather than optional. These groups are both more likely to be harmed and less likely to be able to contest a decision, so EU AI Act Article 27 expects consultation with affected communities, not just an internal review.
You're adopting a third-party or vendor model. Procuring an AI system for a high-risk use requires an assessment before the contract is signed, not after the model is already embedded. Once a vendor model is wired into your workflow, your room to demand changes is gone. Assess while you can still walk away.
An existing system changes materially. Two versions of this. A significant model update, a retrain, a new architecture, a change in features, requires a fresh assessment before it hits production, because the old assessment describes a model that no longer exists. And a new use case for an existing system counts too: a chatbot that was assessed for general support and is now being pointed at credit questions is a different, higher-risk system wearing the same name.
A significant incident occurred. After a material incident, a reassessment is mandatory within a defined window, 30 days in our process. An incident is evidence that the original assessment missed something, and the point is to find what.
The annual clock came due. High-risk systems get reassessed every year even if nothing obvious changed, because the world around the model moves, data drifts, populations shift, regulations tighten, and a system that was fair last year can be quietly out of tolerance this year.
What "an assessment is required" actually means
When one of these triggers fires, the work isn't a form. Our AIA template walks through system identification, data assessment, stakeholder and vulnerable-group analysis, a fairness evaluation, transparency and human-oversight review, security, and a monitoring plan, ending in an explicit decision: approved, approved with conditions, returned for revision, or rejected. The timing rule underneath all the triggers is the same one: the assessment happens before the risk is baked in, and it's signed by people with the authority to stop the project, the model owner, an RAI Council representative, the data protection officer, and for high-risk systems the executive sponsor.
If you're deciding right now whether your system needs one, run this:
- Classify the tier first. High-risk or prohibited settles the question immediately.
- Check the data. Personal data at scale, special-category data, or plausible inference of sensitive attributes pulls in a DPIA.
- Check the decision. If it affects a person's rights, opportunities, or access to services, assess it.
- Check for a change trigger. New use case, material model update, vendor adoption, incident, or annual review each restart the clock.
- When two or more fire, integrate. Run one combined assessment rather than three overlapping ones, and document the decision either way.
If none of the triggers apply, record that you checked and move on. The assessment is a control you aim, not a tax you pay on every model, and knowing when it's mandatory is what keeps it credible when it is.
