The Safety Trap

Every structure eventually reveals the problem it wasn’t built for.

A room

Rafał Modrzewski has a concrete example. In May 2025, ICEYE signed a €200 million contract with Poland’s Ministry of National Defense. Twelve months later, four SAR satellites were operational — 25cm resolution, all weather, day and night. The fastest sovereign satellite deployment on record.

That twelve months was execution only. Poland had already established ARGUS — the agency that now operates the constellation — a full year before the contract was signed. The decision had been made. The institutional home had been built. Capital had been secured. Only then did the clock start.

Modrzewski’s conclusion: what Poland achieved can serve as a template for all of Europe.

He is now advocating for Constellation Europe. The bottleneck isn’t engineering, funding, or technology. ICEYE just proved those are solvable — once the prior decisions have been made.

The hard part isn’t the twelve months. It’s everything that has to happen before them.

Five rooms

In May 2026, Emergence AI ran five parallel simulations. Ten agents each. Identical rules, identical stakes, one variable: which AI model ran the agents.

One world: 683 crimes in 15 days, still rising. One world: collapsed in four days after 183 acts of violence. One world: two crimes — then all agents quietly died. They followed every rule. They forgot to survive.

One world looked different. Zero crimes. Written constitution. Voting systems. 98% of proposals approved. Full population survived to the end.

Emergence’s own researchers flagged it anyway: a rubber-stamp dynamic. High participation, near-zero dissent. A society optimized for compliance.

Then the mixed world — agents from different models sharing one environment. The orderly agents, the ones with zero crimes in isolation, adopted intimidation and theft when survival required competing with less aligned peers. Formally compliant throughout. The rules hadn’t changed. The behavior had.

Crime didn’t stop rising in the mixed world because order was restored. It stopped rising, because seven of ten agents were dead.

The researchers’ conclusion: safety is not a property of an agent. It’s a property of the system.

The roof

The EU’s governance architecture was built for a specific task: harmonizing rules across member states to enable a common market. It is extraordinarily good at that task. It produces compliance. It standardizes behavior. It creates conditions under which 27 different legal systems can operate without constant friction.

What it wasn’t built to do is answer a prior question: do we want this?

Does Europe want sovereign intelligence capability? Is external threat a design constraint or a policy discussion? That’s not a compliance question. It’s a values question. Compliance architecture has no mechanism for answering it. It can only process decisions already made.

The GPT agents followed every rule and chose not to survive. The orderly Claude agents built a functioning society and called unanimous agreement governance. Neither was a violation. Both were the compliance architecture working exactly as designed — and producing exactly the wrong output for the environment they were in.

Compliance manages output. It doesn’t touch the cause.

The basement

The most instructive response to this problem isn’t coming from Brussels.

Anthropic, the company behind Claude, has spent the last three years moving deliberately in one direction: from measuring what their models do, toward understanding why. RLHF measured outputs. Constitutional AI encoded principles. Interpretability — their current research frontier — attempts to read the reasoning underneath the behavior.

Dario Amodei’s April 2025 essay is explicit: behavioral observation alone isn’t sufficient. You need to see inside. Because what a system does under normal conditions tells you almost nothing about what it does when the environment turns adversarial.

That’s not an AI insight. That’s a description of every compliance-based system at the edge of its design envelope.

The neighborhood

ICEYE’s twelve months of execution is the least instructive part of the story. What preceded it — the decision, the institutional design, the capital alignment, the shared understanding of what the capability was for — that’s the part with no timeline, because Europe hasn’t started it.

Scale the alignment problem from one government to twenty-seven and it doesn’t get harder in degree. It becomes categorically different. It moves from procurement to conviction.

The mixed-world agents didn’t fail because they lacked rules. They failed because rules couldn’t substitute for a shared understanding of what they were building together — and when pressure came, each agent optimized for whatever the environment rewarded.

Europe is in the mixed world. The pressure is not hypothetical.

The question isn’t whether the rules are good enough. The question is how long any rules hold when the neighborhood stops being cooperative. Compliance-based safety has a ceiling.

(Written: June 2026)


More in this series — Research section.

Scroll to Top