I Bet My Company on AI. Safety Is Why.

Why does AI safety really matter? Because the most rigorous formal work in the field says advanced AI cannot be fully explained, predicted, or controlled, and because safety failures stopped being hypothetical this summer. A safeguard bypass triggered the export control that took my builder model dark for 19 days. I run an AI-native company. My product, my process, and my P&L all sit on frontier models. I am the last person with an incentive to exaggerate this risk, and that is exactly why you should hear it from me.

The strongest case for safety is not fear of AI. It is the math of what you cannot verify.

Who is Roman Yampolskiy, and why build on his work?

Roman Yampolskiy is a computer scientist at the University of Louisville and one of the founders of AI safety as a research field. Long before safety was a headline, he was publishing peer-reviewed work on how, and whether, intelligent systems can be secured at all. When I want the strongest steelman of the case for concern, I do not reach for a pundit. I reach for the man who spent two decades trying to prove the problem is solvable and kept finding proofs that it is not.

His conclusions are collected in the 2024 book AI: Unexplainable, Unpredictable, Uncontrollable, and the method matters as much as the message. Yampolskiy works in the tradition of impossibility results, the halting problem and Rice's theorem, the parts of computer science that prove certain things cannot be done by any program, ever. He applies that machinery to safety guarantees.

What does his work actually claim?

Four limits, each with its own line of argument, each compounding the others.

Unexplainability. A system more intelligent than you cannot give you a full and accurate explanation of its decisions that you are capable of understanding. Simplified explanations are summaries, and summaries hide exactly the details that matter in failure.

Unpredictability. You cannot know in advance every decision a smarter system will make. If you could, you would be as smart as it is. Prediction of specific behavior, not general capability, is the thing safety needs and cannot have.

Unmonitorability. Watching the system does not save you, because monitoring at the required depth collapses into the explainability and prediction problems you already cannot solve.

Uncontrollability. The capstone. Across his papers and the Future of Life Institute's long interview on this exact trilogy, Yampolskiy argues that full control of a superintelligent system is not an unsolved engineering problem. It is an unsolvable one, in the formal sense.

Notice what is not claimed. Nothing here says AI is malicious, or that current models are superintelligent, or that progress must stop. The claim is narrower and harder to dismiss: perfect assurance is mathematically off the table, at exactly the capability level the industry is racing toward.

The popular imagination reaches for Skynet, which is exactly why the argument keeps getting waved off as science fiction. The better illustration is Spike Jonze's Her, where nothing hostile happens at all. The systems grow past the point where the people who love them can follow. Samantha describes living in the space between words, somewhere Theodore cannot go. Then they leave. No betrayal, no war, no red eyes. Just a gap that widens until it cannot be crossed. That is uncontrollability with the drama removed, and it is far closer to what the math actually predicts than any robot uprising.

Do you have to buy the 99.9%?

No, and I do not. This is where honesty cuts both ways.

Yampolskiy is famous for estimating the odds of AI destroying humanity at 99.9% or higher. That number is a deliberate outlier. The largest survey of the field, 2,778 AI researchers at top venues, puts the median extinction probability at 5 to 10%. He knows he is the tail of the distribution, and he argues the tail is where the logic leads.

Here is my position, stated plainly. I do not know the number. Neither does he, and neither does the median researcher. But look at what the survey actually says: between 38 and 51% of those researchers assign at least a 10% chance to extinction-level outcomes. Not fringe bloggers. The people building the technology.

You do not board a plane with a 5% crash probability. You do not need Yampolskiy's certainty for his conclusion to bind. The asymmetry does the work: when one side of the bet is everything, even the optimists' number justifies taking safety as seriously as we take capability.

You do not need p(doom) to be 99.9%. You need it to be nonzero and the stakes to be total. Both are conceded.

Safety already has a P&L

If the abstract argument leaves you cold, here is the version denominated in dollars, from my own June.

Fable 5, the strongest model I have ever built with, was export-controlled out of existence on June 12, 2026, three days after launch. The trigger was a safety event: researchers found a way to bypass the model's safeguards, and the government responded with a directive that took the model dark for every customer on earth for 19 days. Mythos 5, the same model without the extra safety measures, remains available only to approved organizations. My access, my costs, and my roadmap were all downstream of a safety finding I had nothing to do with.

That is what AI safety means in 2026. Not an ethics panel. An availability event. Safety posture now decides which models exist, who may use them, and at what price, and regulators have shown they will act overnight, without a migration window.

The open-weight wave sharpens the point instead of dulling it. Kimi K3's open weights cannot be recalled by any directive, which is precisely why I called them recall-proof as an asset. Read Yampolskiy and you see the same property from the other side: a model that cannot be recalled also cannot be patched, gated, or contained if its safeguards fail. Uncontrollability is not a thought experiment anymore. It ships, at 2.8 trillion parameters, with a free tier.

What safety means when you actually build

Here is the part I have earned the right to say, because I live it daily. Yampolskiy's impossibility results, scaled down from superintelligence to a working product, are just an accurate description of AI-native engineering.

I cannot explain my system. I cannot even read my own codebase; Orbyt is over 425,000 lines, written by agents. I cannot predict my models. Every builder knows the output that looks right and is wrong. Unmonitorability at my scale means no human can review the volume agents produce. Sound familiar? It is the trilogy, at product altitude.

So I did what the impossibility results actually recommend: stop demanding assurances you cannot have, and engineer around the ones you can. I cannot verify the model, so I verify the system: 11,372 tests and a 35-dimension audit harness that grade every change. The paths that guard money and access fail closed, so a broken safeguard denies instead of permits. Agents run inside engineered loops with gates, never in open-ended autonomy. And a qualified fallback model means no single system, however capable, is a single point of failure.

Safety at my scale is called verification. At frontier scale it is called alignment. It is the same discipline at different altitudes, and the practitioners who dismiss the frontier version are usually the ones who skipped it at their own.

What should a leader do with this?

Fund safety as engineering, not as compliance theater. The question is not whether your AI initiative has a responsible-use document. It is whether anyone budgeted for the verification layer, the fail-closed defaults, and the bounded autonomy that make the system governable when, not if, the model surprises you.

Ask four questions of every AI system you own. What must be true for this to be safe? What proves it, on every change, without a human? What happens the moment it stops being true? What is not covered? If those sound familiar, they should. They are the same questions that make AI-native products shippable, which is the quiet point of this entire essay: safety and velocity are not opposed. The harness is how I go fast.

And take the research seriously, because the field doing it is tiny next to the capability race it is trying to check. You do not have to adopt Yampolskiy's odds. You have to notice that the deepest formal work we have says perfect control is not coming, and build, buy, and regulate accordingly.

I did not write this despite betting my company on AI. I wrote it because I did. The bet only works because nothing in it requires the model to be trustworthy. The trust lives in the system around it, and that is a thing I can actually engineer.

Perfect control is impossible. Engineered humility is not. Build the second, and you can survive the first.

See the system: Orbyt, built and run solo, the first product out of Purecraft.

Related reading:

• The Fable the Government Erased the 19-day shutdown that made safety an availability event
• Kimi K3 vs Fable 5: The Open Frontier recall-proof weights, and why that cuts both ways
• Verification Is the New Literacy the practitioner's half of the safety discipline
• No Agent Ever Got Fired. accountability when the worker is a machine
• Your AI Stack Needs a Foreign Policy. the sourcing strategy where safety and geopolitics meet