Can AI systems actually improve themselves today?

The weak version, yes, and it is running in production. Agents write features, run tests, heal their own failures, and convert recorded incidents into permanent deterministic guards. In my system 35 of 58 recorded failures are now mechanized checks. What none of it does is decide that the goal was wrong.

What limits recursive self-improvement in practice?

Verification, not model capability. Every improvement the loop makes is graded against a criterion the operator set, so the system optimizes toward that criterion and cannot exceed it. When the grading oracle lives inside the system being improved, the system reliably passes green while being wrong, and no amount of extra capability fixes that.

What happens when an AI system's self-improvement goes wrong?

It produces confident false signals that are harder to detect than failures. One of my checks kept asserting a rule that had been superseded and reported critical for 16 days. The session reading it concluded the underlying data was broken. A red the system cannot justify corrodes trust exactly the way a false green does.

Should AI agents be allowed to modify their own instructions?

Only behind a gate that opens on deterministic evidence. In my org the lesson-writing protocol is gated by stage, and seats advance only on committed check results, never on self-assessment. Twenty-three agent-written lessons are currently held. An agent that can rewrite its own operating instructions without an outside grader is optimizing against itself.

AI Builds AI. I Found the Ceiling.

Almost everyone writing about recursive self-improvement is arguing about a system nobody has.

I have the weak version running. Agents write features, run tests, heal their own failures, and turn incidents into permanent guards that make the same failure unshippable. Twelve seats operate on a schedule. The system modifies its own operating instructions, behind a gate.

It is not the thing people are afraid of. It is the closest available instance of it, and it has been running long enough to produce numbers, which are published live on Orbyt's building-in-public page.

The loop closes in four places. Then it stops, and where it stops is the part worth your time.

What actually self-improves?

Four mechanisms, all of them boring, which is the point.

Incidents become guards. My failure corpus holds 58 recorded incidents. 35 of them are now a numbered dimension in an audit harness that runs on every commit, out of 103 guards total. Once a failure is mechanized, that specific failure cannot ship again. Not "should not." Cannot.

Agents heal their own failures. A run that fails a guard reads the guard output, fixes the cause, and re-runs. Most of the time this works without me.

Seats advance on evidence. Agent seats progress through authority stages based on committed check results. Not on how well they think they did.

Lessons are written back. An agent that learns something proposes it as a lesson, and the lesson changes what later agents do.

That fourth one is the interesting one, and it is the one I keep on the shortest leash.

Two lessons that changed downstream behavior

Here are two that measurably altered what every subsequent agent does.

"A value compared against itself can never fail"

A log table recorded 400 while the customer received 404. Contract tests asserted the response and were right. Wrapper tests asserted the log row and recorded whatever the code passed, which proves nothing. Nothing asserted that the two agreed. 13,364 passing tests missed it for a month.

The fix was a guard. The lesson was bigger than the guard, and I know that because the same class was then found a second time in the same file.

So the lesson became a rule that changed how every guard since has been written: whenever two artifacts must agree, you do not write two tests. You write the one guard that asserts the correspondence. Fixing an instance does not fix a class.

"A guard is not proven until you have seen it fail"

Injecting the defect and watching the check go red became mandatory before any guard is kept.

That rule paid immediately. One guard's first draft reported 23 violations that were all false positives, which I only learned by trying to make it fail correctly. Another test case had been passing for months and had never once exercised its own claim: it read whatever the repository happened to contain, and the repository could not contain the case it was testing for.

"It passes" and "it can fail" are different facts. Only one of them is evidence.

Where did self-improvement make things worse?

I would not publish this post without this section, because a self-improving system that only ever improves is a system you are not looking at closely.

A semantic constant changed meaning. One detector was updated in the same commit. A second detector, in a different file, was not.

That second detector kept asserting the dead model and false-fired critical for sixteen days.

Then the failure mode that actually matters: a session read that red, believed it, and recorded the data as broken. It correctly refused to regenerate the data, because regenerating would have fabricated a claim on four public pages. It was right to refuse and wrong about which side was lying.

A red the system cannot justify corrodes trust exactly the way a false green does.

There is a quieter version of the same disease. A report-only check fired on every successful run and could never go quiet, because it was flagging a workflow for not committing a report that the workflow was designed never to commit. Two of the four findings my daily summary raised as "the one thing to look at" were that. A permanent false positive is worse than no check, because it teaches the reader to skim past the real finding sitting next to it.

The self-improvement loop produced all three of those. It generated the detectors, and generating detectors faster than you can validate them is a way to get less signal, not more.

The third failure, which is the one I would not have predicted

A fix that lands in a shared implementation does not exist for the surfaces that bypass it.

An agent-written report was once killed by a formatting guard, and the entire run's work was discarded with it. The fix was to normalize the text before the guard runs, and it went into the shared action every seat uses. Correct fix, correctly applied.

Three workflows hand-rolled their own guard sequence instead of using the shared action. Nobody swept them.

Thirteen days later one of those workflows replayed the original failure verbatim. One bad character, the guard correctly refused it, the run died losing all of its work, and because that seat runs monthly, the next attempt was a month away.

The self-improvement loop had learned the lesson, mechanized the lesson, and still lost a month of work to the identical bug, because the lesson lived in the path most seats take and the bug lived in the path three seats take.

The durable version of the fix is not a third patch. It is a check that asserts any workflow running that guard must normalize first, which catches the next bypass rather than these three. When an incident's fix lands in a shared implementation, grep for every parallel implementation the same day.

The ceiling

Here is the part that the takeoff debate is missing, and I only see it because I am running the loop instead of arguing about it.

Every improvement my system makes is graded against a criterion I set.

The guards encode failures I recorded. The evals check tasks I chose. The lanes reflect boundaries I drew. The system optimizes hard toward my taste and it cannot exceed it, because there is nothing in the loop that could tell it my taste was wrong.

It compounds capability. It does not compound judgment.

The cleanest proof I have is a selection rule that worked perfectly. It ranked pages by a proxy, filled 110 of 150 slots with executive micro-titles, and built roughly 5,500 pages about what a vice president of partner engineering earns in Miami. Meanwhile eighteen out of eighteen job titles from real user queries returned 404.

Every internal guard agreed the system was healthy. The page generator and the sitemap both imported the same list, so they matched each other perfectly. Every guard in that repository reads that repository.

The gap was only visible from outside. It took external query data to see it, and nothing in the self-improving loop could have generated that view, because the loop's entire world is the artifact it is improving.

That is the failure shape I have written about before as passing green while being wrong, and it is exactly what happens when the oracle lives inside the system.

The numbers on the ceiling

I would rather show it than assert it.

Measure	Value	What it means
Merge-autonomous lanes	0 of 12 seats	No agent merges to the branch that deploys
Rollout stage	1 of 4	The org has not earned its way past the first rung
Discards	26 across 40 runs	A quarter of produced work is thrown away at a gate
Lessons written, held	23	The protocol that lets agents rewrite instructions is gated shut
Failures mechanized	22 of 43	Half of what I have learned is still only prose

The single criterion blocking promotion to the next stage is that discard count reaching zero. Three of four conditions are met. That one is not, and the ratchet does not care that I want to move.

That is deliberate. The promotion mechanism never grants authority. It activates pre-ratified stages on committed deterministic evidence, and an agent cannot report that it is ready.

Look at what each rung actually unlocks and the shape of the design becomes clear. Stage one turns on one seat. Stage two turns on the seat whose job is auditing the other seats. Stage three is the one that opens the lesson protocol, which is the moment agents may write back into the instructions that govern agents.

The capability I am slowest to grant is self-modification, and it sits two rungs above where the system currently is. Twenty-three lessons are written and held behind that gate. Not rejected. Held.

The criteria are also hash-pinned to the document that defines them, so editing the bar resets the streaks earned under the old bar. Otherwise the easiest way to pass is to lower the test, and a system that grades itself will find that path eventually. Not because it is scheming. Because it is optimizing, and that is what optimizing looks like when the grader is reachable.

The claim

Put the timeline arguments aside for a second.

Recursive improvement is not gated on model capability. It is gated on verification that is grounded outside the thing being improved.

A smarter model writes better guards, faster. It does not solve the problem that the guards are graded by the same system that wrote them. Add capability to a closed loop and you get a system that is more efficiently confident about the wrong criterion.

Every argument about fast takeoff is, underneath, an argument about whether that grounding problem gets solved. Almost nobody is stating it in those terms, and I think that is because stating it requires having run the loop and watched it hit the wall.

The people I would actually listen to on this are not the ones with the best model. They are the ones who can show me their external oracle.

What to do Monday

If you are running agents that modify your codebase, answer one question: what grades their output, and does that grader share a source of truth with the thing being graded?

If the answer is your test suite, and your agents write your tests, the loop is closed and you will get green forever.

Point one instrument outside. Production data, real user queries, an external API's actual responses, a human reading the artifact rather than the code. One outside signal is worth more than ten new internal checks, because internal checks can only ever confirm what the system already believes.

The bottleneck was never how smart the loop is. It is whether anything in it can tell you the goal was wrong.