Compliance used to mean documentation. For AI agents, it means something else entirely
Governance needs to be in the architecture conversation, not the incident response.
For most of the history of enterprise software, EU compliance worked like this: build the system, document what it does, assess the risks at deployment, get your certification, and move on. The documentation captured a fixed thing. The assessment was a snapshot. If nothing changed in the system, the snapshot stayed valid.
That model doesn't work for AI agents. A new paper released this week makes that concrete in a way that product and legal teams building agentic systems need to sit with: a high-risk agentic system with untraceable behavioral drift cannot currently be placed on the EU market consistent with the essential requirements of the EU AI Act. Not in six months when enforcement ramps up. Now. That's binding law.
Here's what makes this different from ordinary software compliance. The EU AI Act's Article 3(23) defines a "substantial modification" as a post-deployment change that affects compliance or alters the system's intended purpose. For traditional software, identifying a substantial modification is straightforward — you look at what changed in the code. For an agentic system, the model weights may be identical while the system's operational profile has shifted entirely. Persistent cross-session memory, continuous learning from user interactions, novel tool use patterns the system develops on its own — these can each move a system outside the boundaries of its original conformity assessment without anyone changing a single line of code.
The researchers behind this paper identified three manifestations of what they call behavioral drift: semantic drift (the system's outputs diverge from baseline patterns), coordination drift (how the system orchestrates tool use changes), and behavioral drift (the system develops patterns that weren't present at deployment). All three follow non-linear degradation trajectories. You don't get early warning signs. You get a system that was compliant at deployment and may not be now, with no clear inflection point where it crossed the line.
The practical implication for product teams is this: conformity assessment is no longer a one-time event. It's the starting point for an ongoing monitoring obligation. What the paper describes as the "minimum viable compliance posture" for a high-risk agentic system includes versioned snapshots of operational state at defined intervals — tool catalogue, memory state, policy bindings — continuous monitoring of behavioral metrics against the conformity assessment baseline, automated detection when drift crosses defined thresholds, and documented internal procedures for determining whether a change meets the Article 3(23) threshold for substantial modification.
That last piece is more consequential than it sounds. Right now, most organizations don't have an internal procedure for making that determination because they've never had to. For traditional software, the question answered itself — either the code changed or it didn't. For agentic systems, someone has to own the judgment call, with documented criteria, on an ongoing basis. That's a new operational function that doesn't map cleanly onto existing legal, engineering, or compliance teams.
There's a second layer to this worth understanding. The harmonised standards that providers will use to demonstrate compliance — the EN 18282 cybersecurity standard, EN 18229 trustworthiness standard — address two of the five agentic threat categories that USENIX Security researchers documented this year. Prompt injection mitigation and privilege management have standards guidance. Multi-agent protocol threats, interface exploitation, and governance-level autonomy concerns don't. For those three categories, providers are building compliance posture on their own, without standards scaffolding, while the law is already in effect.
This is what I mean when I say governance needs to be in the architecture conversation, not the incident response. The organizations that treat this as a documentation problem will find out it isn't when they try to reconstruct whether their system was compliant at a specific point in time and realize they didn't capture the operational state to make that determination. The organizations that treat it as a monitoring and versioning problem — who build the instrumentation before they deploy — will have something to show. That's the difference between a conformity assessment that holds up and one that doesn't.
The shift from documentation to continuous monitoring isn't optional. It's what the law now requires for high-risk systems that do things in the world.
Luca Nannini
