Justin Bartak · Engineering · · 9 min read
Long Horizon Agents Don't Fail. They Pass.
TL;DR
Long-horizon agent tasks do not fail loudly. They fail green. On Orbyt, a live function wrote to a database table that did not exist, and every test passed the whole time. The horizon of an autonomous system is not model stamina. It is time to detection, and only an oracle grounded outside the code shortens it.
Long-horizon tasks do not fail the way people expect. They do not time out, run out of stamina, or collapse in a heap of confused output. They fail green. An agent takes a wrong step, the step returns success, the build stays passing, and every hour of work after that gets built on top of the mistake. Nothing goes red. That is the problem.
The failure mode of long-horizon autonomy is not an error. It is a pass.
What is a long-horizon task?
A long-horizon task is work whose correctness cannot be judged at the end of a single run. It spans sessions, agents, and days, and every step inherits the last one's output. Shipping a feature is short-horizon. Running a product for six months is long-horizon.
The defining property is not duration. It is compounding. In a short task a mistake costs you the task. In a long task a mistake becomes a foundation, and the cost grows with every hour nobody catches it.
I have a clean test case. Orbyt took its first commit on February 8, 2026, and I have been building and running it solo ever since: 3,343 commits, over 425,000 lines, 11,372 tests. It has never been rebuilt from scratch and it has never stopped shipping, including through the nineteen days a frontier model I depended on was withdrawn from the market by export controls. That is a long horizon by any definition, and it is the most instructive thing I own, because I know exactly where it broke.
Why do long-horizon tasks fail silently?
Because the things that check the work are made of the same material as the work.
On July 23 I added a new dimension to Orbyt's audit harness. Its job is narrow: reconcile what the live database actually contains against the invariants the code claims. It found something on day one. A live background function was writing to a table called profile_embeddings. That table did not exist. The write's result was never checked, so it failed quietly on every single run, and had been doing so since it shipped.
Every test passed the entire time. The type checker was clean. The build was green.
It was not an isolated case. An earlier dimension went looking for schema drift and found three things the application queried that existed nowhere in the database. Those routes had been failing in production from the day they launched. The same pass found migration files lying in both directions: two of them were marked pending and had in fact been applied.
Here is the sentence I ended up writing into the repo, because it explains all of it at once. Every other dimension reads the repo, and therefore inherits the code's blind spots. Every expensive bug in this project's history was found by looking at production, never by the test suite.
A verifier that reads what the actor wrote cannot see what the actor never wrote.
That is the whole mechanism. Tests read the code. The code is what the agent produced. So the suite confirms the agent's own model of reality, in extraordinary detail, at enormous scale, very quickly. It does not know the table is missing, because the code does not know either.
Why can't more tests extend the horizon?
They can, up to a ceiling, and the ceiling arrives early. Orbyt has over eleven thousand tests. None of them caught a live write into a nonexistent table.
Three lessons in the repo say the same thing from three angles, and I number them so agents can cite them.
A test that compares a value against itself can never fail. When two artifacts have to agree, assert against an independent source, never a value re-derived from the same code path. An unasserted correspondence is a lie waiting to happen.
A path nobody has walked is not a working path. It is a path nobody has walked. One validation gate in Orbyt ran only on the days a model changed. It had been silently killed by memory limits for a long time, and it died on the one day it mattered. Every green run in between had skipped the step entirely. Green proved nothing, because green never ran it.
A filter that decides whether to include something is a detector, and its failure shape is silent omission. A build script rejected any chapter under ten lines as an unfinished skeleton. A finished closing chapter tripped the rule and was dropped from both output formats, announced by one line saying "Skipping" and an exit code of zero. Every downstream check passed, because a book with no ending is still a structurally valid book.
Notice what all three have in common. There was no red. There was output, and the output was wrong, and everything designed to notice was reading the wrong thing.
| Reads the artifact the agent wrote | Reads something the agent cannot edit | |
|---|---|---|
| Example | Unit tests, types, lint | Live production state, the built output |
| Catches | Logic that contradicts itself | Reality that contradicts the code |
| Blind to | Anything the code assumes wrongly | Nothing it can reach |
| Failure shape | Silent pass | Loud mismatch |
| Effect on horizon | Extends it until the first wrong assumption | Extends it as long as the oracle holds |
How do you actually buy horizon?
You shorten time to detection. That is the only lever that matters, and there are four moves that work.
Read reality, not the repo. At least one check must query the running system and compare it against what the code believes. This is the single highest-return thing I have built in six months, and it was thirty lines of read-only queries.
Read the built artifact, never the source. If a pipeline produces something, open the thing it produced. Source-level confidence is how a book ships without its last chapter.
Prove every detector can fail. Inject the bug and watch the check go red. A check that has never fired is not a check, it is a decoration. I now also warn on any guard whose test never proves it can stay quiet, because a detector that fires on everything gets ignored, which is the same as being off.
Suspect the detector first. When a check flags code that looks correct, the checker is wrong more often than you would like. In one cleanup pass, two findings turned out to be bugs in the checkers, not the code. Fix the checker, keep the code. A false alarm that survives teaches your agents to ignore alarms.
Horizon is not bought with a better model. It is bought with an oracle the model cannot edit.
Where should autonomy be allowed to run?
This is the part most teams get backwards. They grant broad autonomy and ask for a report afterward. Afterward is exactly when a long-horizon mistake has already compounded.
The better boundary is not when you review. It is which tier an agent may touch unattended. In Orbyt, checks come in two classes. Report-only checks print findings and always exit zero. Blocking gates can stop a commit or a deploy. Agents may create, improve, and land report-only checks on their own, because a report-only check is structurally incapable of breaking anything. Flipping a blocking gate, touching a protected file, or pushing to production requires my signature.
That is not trust management. It is architecture. The autonomous class cannot do the damage people are afraid of, so I do not have to supervise it minute by minute. I supervise the one tier that can hurt me. It is the same instinct as gating a fleet of agents behind a verification bottleneck, and the same reason my agent C-suite needs my signature to promote a lesson.
There is a quieter constraint too. Everything an agent must always know competes for room with the work itself. When my instruction file grew past what I wanted loaded on every run, I moved the detail into a reference document and kept only the operative rules in the always-loaded floor. Long-horizon systems need a budget for their own memory, or the memory eats the horizon.
What should a leader take from this?
Stop asking how long your agents can run. Ask how long a wrong answer can survive.
That question has a number attached, and the number is your real horizon. If the honest answer is that a person eventually notices, then your horizon is a human attention span, and no model upgrade will move it. You have bought speed without buying detection, which means you have bought the ability to be wrong faster and for longer.
The order of operations is unglamorous. Find one thing that reads reality instead of your code, and wire it into the loop. Then prove your existing checks can fail. Then decide which tier of change your agents may make while nobody is watching, and make sure that tier cannot break anything. Do those three and you can leave the machine running.
The models will keep getting better at long tasks, and that is genuinely good. It raises the number of correct steps you get for free. It does not change the arithmetic, because the arithmetic was never about how long the model can think. The gate at the end decides what ships, and a gate reading the wrong document stays green through anything.
Six months of green is not six months of correct. It is six months of not looking.
See it in practice: Orbyt, built and run solo, the first product out of Purecraft.
Related reading:
- The Prompt Is Dead. Long Live the Loop. the loop this post stress-tests, and why a red gate beats a confident wrong answer
- Verification Is the New Literacy why reading the verdict replaced reading the code, and what the verdict can miss
- No Harness, No Trust. one command, one verdict, and the checks that earn the right to be trusted
- Your Tests Are the Spec Now. the spec as executable checks, and where a self-referential check goes wrong
- I Ran 830 Agents in One Long Horizon Session. why verification, not compute, is the constraint on agent throughput
- Loop Engineering: How Orbyt Builds Itself the working machine these lessons were learned inside
Frequently asked questions
What is a long-horizon task?
A long-horizon task is work whose correctness cannot be judged at the end of a single run. It spans many sessions, many agents, and many days, and each step builds on the last. The defining property is not duration. It is that early errors stay invisible while later work compounds on top of them.
Why do long-horizon agent tasks fail?
Not from exhaustion. They fail because a wrong step produces a passing result. The agent does not crash, the build stays green, and the mistake becomes a foundation. By the time anything surfaces, weeks of work sit on top of it. The failure mode of long-horizon autonomy is silence, not error.
How do you make long-horizon agent work reliable?
Shorten time to detection. Add verification that reads something the agent did not write: live production state, the built artifact, an independent source. Then prove each detector can fail by injecting the bug and watching it fire. A check that has never fired is not a check, it is a decoration.
Is long-horizon capability a model problem or a system problem?
Mostly a system problem. Better models take more correct steps before drifting, which helps. But the limit on unattended work is how long a mistake can survive undetected, and that is set by your oracles, your gates, and where autonomy is allowed to touch. Systems outlive models.
