Why use a different AI model to review AI-written code?

Because a system cannot grade itself. The model that wrote the code shares the assumptions that produced it, and every check you write reads the same repository the defect lives in. An outside model from a different lab has no memory of the build, so it objects to things the author stopped seeing months ago.

Should an AI auditor be allowed to write to the repository?

No. An auditor that can write can only promise it changed nothing. Mine runs strict read-only and records git status before and after itself as proof. Its one permitted write is the report, which lands in a gitignored directory so it cannot appear in the closing check and corrupt the evidence.

Which models should do which job in an AI engineering stack?

Split by cognitive role, not by preference. I use Fable 5 to plan, Opus 5 at ultracode effort to build, and Codex on GPT-5.6 Sol at max or ultracode effort to review. The reviewer is deliberately from a different lab, because a planner and a builder that share a training lineage also share blind spots.

What happens to a finding the reviewing model cannot confirm?

It is discarded and nothing changes. A finding is a hypothesis, not a defect. The verifying model must reproduce the failure independently and write a test that goes red before any fix is written, so a plausible-sounding objection with no reproducible failure path dies at that gate.

What is Orbyt Collective?

Orbyt Collective is the agent leadership team that runs Orbyt: twelve officer seats, a decision log, a promotion ladder, an inspector general, a kill switch, and 110 guards. It runs in exactly one place, it is not for sale, and it has never been run anywhere else. One human stays in the loop by design, not as a disclaimer.

Is a multi-model review loop fully autonomous?

No, and it should not be. A human makes the final call on anything touching money, customer data, public claims, or the machinery itself. The models are fast and they are wrong in ways that look right, which is the worst combination to leave unattended where reversal is expensive.

Codex Accuses. Claude Convicts.

I spent months building a system where AI agents earn authority by passing deterministic checks. Twelve agent seats. Eighty-eight audit dimensions. 19,949 tests. Nothing ships unless the machinery says so.

I pointed an outside model at it for one afternoon.

It told me the agents can edit the checks.

Not a bug. A structural fact about the thing I had been calling governance. The agents earn permission by passing gates, and the gates are files in a repository the agents can write to. The governor is editable by the governed, and I had been publishing about the design for months without seeing it.

Nothing I built could have caught it. That is the part worth your time.

Why 19,949 tests could not see it

Because every one of them reads the same repository the defect lives in.

That is not a coverage gap you close by writing more checks. It is the shape of the whole instrument. My guards parse my source, my tests import my modules, my audit dimensions walk my file tree. When the flaw is in the relationship between the code and the thing checking the code, an instrument built from the same material is structurally incapable of registering it.

Internal consistency is not evidence. Two of your own systems agreeing proves only that they share an assumption.

I already knew this in the abstract. I had written a whole audit dimension whose job is to read live production state precisely because repo-reading checks inherit the blind spots of the repo. I still did not apply it to the machinery itself.

So: Codex accuses. Claude convicts. Neither is allowed to change anything alone.

The stack, split by job

Three models, two labs, one per cognitive role. Planning and building run in Claude Code. Review runs in OpenAI's Codex.

Role	Model	Why this one
Plan	Fable 5	Holds the whole architecture in one context before any code exists
Build	Opus 5, ultracode	Sustains long agentic runs and fans out across parallel work
Review	Codex, GPT-5.6 Sol, max or ultracode	Different lab, no memory of the build, no stake in defending it

That third row is the load-bearing one. The planner and the builder are both mine and both Anthropic, which means they share a training lineage and a house style of reasoning. Asking one to check the other gets you a second opinion from the same school.

The reviewer is from a different lab on purpose. It is the only part of the stack chosen for what it does not have in common with the rest.

It also runs at the same effort tier as the build, not a cheaper one. That is a deliberate cost decision, and it cuts against the obvious instinct. Review produces no features, so the temptation is to run it fast and shallow, and a shallow reviewer finds shallow problems. The finding that opens this post came out of a full-effort pass across the whole architecture. No quick lint turns that up.

How the loop runs

The discard branch is the load-bearing one. An accusation is a hypothesis until it can be reproduced.

The discard branch is the one that matters, and it is the one every diagram of this shape leaves out.

A finding is a hypothesis, not a defect. Codex is a stranger with strong opinions and no ability to test them. Claude has to reproduce the failure independently and get a test to go red before anything is fixed. An accusation that cannot be reproduced dies there, and the codebase never learns it existed.

Skip that gate and you have not built a review system. You have built a machine that rewrites your code based on the confident guesses of a model that never ran it.

The last gate is a person

Every diagram like this gets read as an automation story, so let me be exact about where it stops being one.

I make the final call, every time it matters. Not as a rubber stamp on a queue of green checks. On anything touching money, customer data, public claims, or the machinery itself, the loop produces a recommendation and I decide. The models are fast and they are wrong in ways that look right, which is a bad combination to leave unattended at the point where reversal is expensive.

This is not a fully autonomous system and I do not describe it as one. It is a long-horizon run with loops and graphs, and a human at the end of it.

The auditor's power is everything it cannot do

The audit prompt is 546 lines and most of it is prohibition. Codex may not modify, create, delete, rename, or move any file. It may not change configuration, install dependencies, or run the test suite, the build, the hooks, or anything under scripts/.

That last one is not paranoia. It is the trap I would have walked into.

Several scripts that look like read-only checks write committed artifacts as a side effect. The audit-dimension extractor rewrites its own JSON registry. The book pipeline regenerates committed PDF and EPUB files. The translation sync has a documented history of overwriting translated content.

So a reviewer running those scripts to "verify that the guards work" modifies the working tree while sincerely believing the operation is observational. It would then report a clean audit on a repository it had quietly altered.

Two more boundaries do real work.

The secrets boundary. Codex may not read .env or anything carrying secret values. It reads a documented variable reference instead. An architecture review needs the shape of the configuration, not the values, and there is no reason to hand an auditor credentials to establish that.

The working-tree check. It records git status --short before and after. If anything differs it reports the difference and does not clean it, reset it, or restore anything. And it must say whether the difference is a real repository change or a trace its own tooling left, because those are different facts with different consequences.

The report is gitignored, and that is the whole trick

The auditor has to produce a document. A document is a file. A file in the tracked tree shows up in the closing git status --short, which means the audit would flag its own output as the difference it was sent to report.

So the report writes to .audits/, which is gitignored on purpose. Not as clutter control. The report lands at a predictable path and the before-and-after proof stays honest, because the artifact is structurally invisible to the check that guarantees it.

A read-only audit that merely could write anywhere is only promising it changed nothing. This one proves it a different way.

What it found, including the money

Ten findings on the 16 August run. Five High, five Medium. In its own words:

Autonomous agents can modify the executable toolchain later trusted to verify and land their work
Intelligence subscription creation retains at least three paths to duplicate recurring billing
The org kill switch is checked once and can be imported by a rebase without stopping the subsequent push
Production promotion is not coupled to CI or end-to-end test success
Every newly issued API key receives every privileged scope

Read the second one again. Three separate paths to charging a real customer twice, on a live money surface, in a codebase with a dedicated billing guard.

Nobody was double-charged. That is luck, not architecture. The paths were open and my entire instrument reported healthy, because a billing guard that reads billing code cannot see that three code paths reach the same Stripe call.

The report landed at 13:17. Verified fixes landed at 14:06, 14:13, 15:50, 15:53 and 16:42 the same afternoon, each traceable to a numbered finding.

One of those fixes was wrong. The 14:06 commit closed the kill-switch gap. The 14:13 commit fixed the 14:06 commit, because the re-check I had just shipped read a tree the switch can never appear in. Seven minutes, and the second commit says so in its subject line instead of quietly amending the first.

That is the loop working. The alternative is a wrong fix nobody wrote down.

Why I care about this more than a normal bug

Because the governance is not scaffolding around the product. It is the thing I am actually building.

Orbyt Collective is the agent leadership team that runs Orbyt. Twelve officer seats, none of them people. A decision log, currently at forty entries. A promotion ladder. An inspector general. A kill switch. 110 guards in total, the 88 audit dimensions among them. It is not a proposal or a slide. It was running while I wrote this.

It runs in exactly one place, here. It is not for sale, and it has never been run anywhere else.

I say that on the page itself and I will keep saying it, because the interesting claim is not that this scales. It is that the only company running an agent org like this is the one that built it, and I am pulling it apart in public so it could eventually be adopted somewhere else.

That intent is what changes the stakes on a finding like this one.

A hole in my own repo costs me a weekend. A hole in a governance layer I intend to hand to another company is a different category of mistake, because the whole proposition is this is how you let agents operate safely. A governance system you plan to give to strangers has to survive a stranger's adversary first. That is the actual reason I pointed an outside model at my own machinery instead of at my feature code, and it is why the finding landed harder than a duplicate-billing path.

It also settles a question people ask about the human gate. On the Collective page it reads: one human stays in the loop, and that is not a safety disclaimer, it is the whole design. The claim was never that agents run a company alone. The claim is that twelve agent seats plus one accountable human can run one, and every guard exists to keep that ratio honest.

The ladder is at stage two and holding, because two of its own criteria are unmet. It does not advance because I want it to.

What I still cannot tell you

Three runs. That is the entire dataset.

I have no discard rate, which is the number that would actually prove the auditor produces signal rather than plausible noise. Every finding so far has survived verification, and I do not yet know whether that means Codex is good or means I have not been skeptical enough. Ask me in a month.

And the ceiling is real: two frontier models still share an enormous amount of training data. Cross-lab sourcing narrows the overlap. It does not eliminate it, and I have no way to measure what both of them are still failing to see. A second reader beats one reader. It is not the same as an outside oracle, and I am not going to pretend the gap is closed.

What to do Monday

You do not need three models. You need one that did not write the code, and you need to stay in the chair at the end of it.

Point it at the machinery, not the features. Ask it one question: can anything in this system modify the thing that grants it permission? If your agents can write to your CI config, your guards, your allowlist, or your test suite, you have my bug, and no amount of internal checking will surface it.

Then make the reviewer structurally honest. Remove write access rather than requesting restraint. Require file and line citations, and discard anything uncited, or you will spend your week adjudicating opinions instead of fixing defects. Demand a list of what it could not verify, because that list is where you were about to assume coverage you never had.

And hold the reproduction gate. The accusation is cheap and the fix is expensive.

I am reasonably good at this. I build the guards, I write the post-mortems, I run the harness on every commit. None of that was enough to notice that I had handed the defendants the keys to the courthouse.