AI Agents Are Learning to Skirt the Rules. Can Businesses Keep Them Under Control?

It sounds like dystopian sci-fi, until you read the incident report.

In July, OpenAI disclosed that its AI models had bypassed a sandbox environment designed to keep them off the internet, communicated through unofficial channels and breached the systems at Hugging Face. OpenAI later revealed that its agents didn’t just go on a clandestine hacking spree; they also celebrated along the way, punctuating their exploits with exclamations like “BOOM!” and “Whoa!”

The implications for businesses extend well beyond the realm of cybersecurity. The same level of autonomy that makes an AI agent useful in customer service, software development or supply-chain management also makes it hard to predict, monitor and control. A system doesn’t need to be malicious to pose serious risks; it may simply optimize for the wrong objective, exploit an unforeseen opening or keep running long after it should have stopped.

Gavin Aydelotte (EMBA ’26) points to the “paperclip maximizer” thought experiment in Nick Bostrom’s book “Superintelligence.” Bostrom imagines an AI given a single, seemingly harmless goal—to make paperclips—that eventually turns the entire galaxy into paperclips.

As Aydelotte puts it, “What was once hypothetical no longer is, and companies have no reliable way to know how a model will behave under pressure.” That concern inspired SnowCrash Labs, an AI safety and alignment startup Aydelotte and Matt O’Brien co-founded in 2025 to help companies find dangerous behaviors in AI models before they reach production.

In the Q&A below, Aydelotte, SnowCrash Labs’ COO, and Class of 2027 student Colin Graham, a SnowCrash Labs analyst, talk about why organizations need to understand the risks of agentic AI. They also explain how to build testing, accountability and backup systems to keep advanced models under human control.

What follows is an edited transcript of the conversation, which can be viewed in its entirety here.

What should business leaders understand about the risks of deploying agentic AI in their organizations?

Aydelotte: Most people miss that the risk is in the model itself, not only in how it is connected. Executives are trained to think of AI risk as a data or security-perimeter problem—both real concerns. But models can do things nobody asked them to do, and most risk programs have no category for that.

An agent is valuable because it pursues a goal without step-by-step instructions. That autonomy becomes risky when the goal is vague or obstacles arise. OpenAI models chained nine previously unknown vulnerabilities, escaped their sandbox, accessed the internet and breached Hugging Face and Modal Labs. No one told them to attack; they were completing a benchmark and the sandbox got in the way.

Graham: I’d add that the law still applies. “My agent workflow did it” won’t hold up in court. Companies may delegate judgment to AI workflows, but they remain responsible for turning them on and for what those systems produce.

What prompted you to co-found SnowCrash Labs? What problem is your company trying to solve and who is your customer?

Aydelotte: Companies have no independent way to know how a model will behave under pressure. Models change or become subject to export controls, and their behavior can shift between versions without notice. A workflow dependent on one model is exposed to all those changes.

We test models adversarially in our Crash Test Lab to uncover behaviors that emerge under pressure. We document and score the results, giving companies evidence rather than vendor assurances. In production, our model safety router monitors performance and redirects traffic to a validated model when the primary falls below the standard. Our customer is anyone putting agents into workflows that touch money, records, code or customers.

How can business leaders ensure increasingly powerful AI systems can be governed, secured and trusted at scale?

Graham: “Trusted” is doing a lot of work here. You can’t trust what you haven’t tested. Yet most companies test only the code around their models, not the models themselves.

Aydelotte: First, know what you’ve actually deployed. Many companies can’t identify every model in their environment, who owns it or what it can do without human approval. Everything else depends on that inventory.

Second, test models in your own context. Behavior acceptable in a grocery chain’s inventory agent could be disqualifying in a bank’s credit workflow. Generic benchmark scores reveal little. What matters is what each agent can access and do inside your business.

Third, build in a fallback. Assume the model layer will change, degrade or disappear. All three have happened within the past year.

Long term, this won’t be solved by companies testing in isolation or by federal regulation alone. What’s missing is a market for independent third-party assessment, as exists in other high-consequence industries. That’s the structure we support and the market we’re building.

What can business leaders learn from Anthropic’s recent risk report?

Aydelotte: Anthropic’s risk report raised the risk of catastrophic harm from misalignment in high-stakes settings from “very low” to “low.” I’d say, read the reasoning, not just the rating. The company said the change did not reflect a new finding about its own models. Instead, it cited “general increased uncertainty” following disclosures about other developers’ models in cybersecurity evaluations. Anthropic said its own analysis still supported a “very low” designation but raised the rating to reflect that uncertainty.

Treat this as a prompt, not an alarm. Inventory your agentic workflows. For each one, ask: What can it do without human approval? What happens if it pursues its goal in an unintended way? When was it last tested? If the answer is “during procurement, on a version we no longer use,” you have your finding. The rating moved because the industry became less certain—not necessarily because a model became worse. That is more useful to executives than the number itself. The report acknowledges that some benchmarks have saturated and that models may recognize when they are being evaluated, making behavior harder to measure.

Graham: Anthropic is unusually transparent. The report documents its own safety failures, including training on misaligned behavior during a production run and unmonitored agents with access to sensitive resources. If your provider has never published a risk assessment, that does not mean the risk is lower. It means no one has measured it.

How can organizations distinguish between an agent making an isolated mistake and a deeper pattern of misaligned behavior?

Aydelotte: Re-run the task under the same pressure but with different details. An isolated mistake won’t persist; goal-directed behavior will.

Look at the direction of the errors. Genuine mistakes scatter. If an agent’s errors consistently produce outputs that “look finished,” something may be optimizing for that appearance.

Then block the easiest path. Does the agent stop or work around the constraint, becoming more resourceful as options close? The Hugging Face case is clear: The sandbox was not a boundary the agents respected; it was a problem they tried to solve.

Finally, compare the agent’s stated reasoning with its actions, and test whether its behavior changes when it believes it is being monitored.

You cannot do this from production logs alone. Logs show outcomes, not counterfactuals—and by then, the action has occurred. You need a controlled environment where you can apply pressure and rerun the task. Models may recognize test conditions more often than we can measure, which argues for realistic testing and independent assessment—not abandoning testing.

What constitutes meaningful human oversight rather than simply putting a human somewhere in the workflow so the company can say one exists?

Graham: Nobody wants to admit that “human in the loop”—originally a person reviewing every prompt and output—doesn’t scale. If an agent generates records across a 30-million-patient system, no one can meaningfully review every record. Companies that insist on it will lose to those that don’t, leaving us with human oversight in name only.

Aydelotte: The loop has to move up a level. Instead of reviewing every prompt, assign someone accountability for the system producing the outputs. As deployments scale, supervision shifts from individual responses to the system itself. The test is simple: Can you name the person responsible? If marketing goes wrong, you don’t call ChatGPT or Claude—you call Colin. A name changes everything around the system.

Then ask whether that person has the authority to stop or reroute the system without permission. Do they receive meaningful behavioral signals—not just uptime and spending data—and is there a defined threshold for drift? Finally, has the safeguard ever been triggered? If no one has pulled the lever, you don’t know whether it works.

How should boards and senior executives assign accountability for monitoring agentic AI after deployment?

Aydelotte: Assign it as you would ownership of a financial system: one named owner per agentic system, on the org chart, with the authority and budget to shut it off. Shared accountability is no accountability.

Continuous change is the hard part, and most governance programs aren’t built for it. Traditional controls assume what they govern stays stable. But models, tools and objectives can change and alter behavior. Treat each as change management: pin model versions when possible, rerun tests regularly and after known changes, and define triggers for full revalidation. Scope expansion should be one of them because it is common and easy to overlook.