β±οΈ 10 min read
In the previous chapter, the factory spent its first full night working without me. I woke up to 10 tasks delivered, tested and merged. But when I added up the time of each stage that night, something did not add up. Testing took longer than developing: about 3h15 against 2h10. And 1h20 of that testing was an agent waiting for a command to finish.
The hint was already in chapter 1, 6x1? Here it's 24/7: the second real run took almost 40 minutes, with continuous integration running four times before it passed. At the time, I thought it was a beginner's hiccup. It was the pattern.
This chapter is about that math: why writing code stopped being the expensive part of the work, and what I changed so the following nights would not spend the clock waiting.
Where the time went
When I looked inside the runs, the surprise was not that testing was slow. It was where the time went.
First, a legend for the rest of the chapter:
- βοΈ Engine: the factory's orchestrator, which I introduced in chapter 1. A regular program, deterministic, with no AI, that always does the same thing the same way.
- π€ Agent: an AI doing the work.
Running a test is cheap. A command runs the suite, it passes or fails, and that's it. The expensive part was everything around it, when the one testing is an AI:
- π€ Reading: the test agent read entire run logs, some of them hundreds of kilobytes, to find out what had failed.
- π€ Waiting: it fired off a long command and kept checking whether it had finished, inside its own turn, with wait conditions that sometimes were never met.
- π€ Rereading: on every turn, it reread the task's entire context.
It's not just the context, it's the turn
I was sure the second night had been slower because five runs were fighting over the processor. When the AI crossed the numbers, it said something else: the suite took even a little less than on the first night. What had grown was the number of turns of the test agent, which passed 400 in a single run.
When people talk about saving with AI, almost everyone thinks about context: how much text goes into the conversation. But the model rereads the whole context on every turn. Every time the agent runs a command, reads a file or checks a result, it is one more read of everything. A simple calculation shows the size of it:
- 500 thousand tokens of context, 5 turns: the model reads about 2.5 million tokens in the session.
- 250 thousand tokens of context, 20 turns: half the context, and about 5 million tokens read. Twice as much.
In the factory, the numbers were much bigger. In a run on October 2, the test agent made more than 200 tool calls, with an average context close to 240 thousand tokens. By the end, it had reread about 37 million tokens, and that rereading accounted for about three quarters of what it cost.
Providers keep what was just read for a few minutes, and rereading that cached part costs a fraction of a new token. But an agent that waits on a long command can come back to an expired cache and pay for the whole read again.
Taking care of what goes into the context helps. Taking care of how many times it gets reread helps just as much.
The upside-down pyramid
Anyone who works with software knows the test pyramid. At the base, many unit tests, small, fast and cheap. At the top, a few end-to-end tests, which open the app and simulate a user, slow and expensive.
The pyramid was always ordered by the cost of running the test. With agents, another cost rules: writing the test, waiting for it and interpreting the result, paid in tokens. And that cost weighs most exactly at the base, where most tests are.
The numbers confirmed it. Testing accounted for almost half the cost of a run. When I added behavior tests, which describe what the user does on screen, it went past 60%. In one run, the test agent alone took about two thirds of the cost.
I also tried the other extreme and turned tests off for one run. It came out cheap and fast, and let three regressions through, things that used to work and stopped working.
The engine runs, the agent writes once
The answer was to separate two jobs that were mixed together and spend AI only where there is a decision to make. The agent writes the test once, and the engine runs it every time.
Everything that was execution left the agent's hands:
- βοΈ Running the suite: the engine runs it and gives the agent back only a summary of the failures.
- βοΈ Choosing the tests: the tests affected by a change come from the map of what uses what in the code, not from the agent's guess.
- βοΈ Old failures: a list of the failures that already existed on main, so the run is not blamed for an error that isn't its own. Nine such failures were being counted against the wrong run.
- βοΈ Waits: timeouts, retries and waiting stay with the engine, outside the agent's turn.
- βοΈ Fixed environment: the same suite took 29 minutes inside the app and 6 to 7 in the terminal, because of a setting inherited by accident. Once that was fixed, it dropped to about 5 minutes.
- βοΈ Heavy tests at release time: the full suite, regression and end-to-end tests moved out of each task and into the closing of a release, at a cut I mark on the roadmap.
The agent kept what takes judgment:
- π€ Deciding what to test: from the acceptance criteria.
- π€ Writing the new test: once, before the code.
- π€ Understanding a failure: already summarized by the engine.
The third night: the cost dropped, the clock didn't
The first night with the updated tool showed both sides of the change.
The cost of testing per run dropped to a fraction of what it was, somewhere between a fifth and a twentieth. In tokens, the problem was solved.
On the clock, it wasn't. The suite, now run by the engine, started taking about 30 minutes per round, against 4 to 11 before. There were four rounds that night, and three hit the timeout. One of them took the run to a point in the workflow where only I decide, and it sat there waiting for me to wake up. The automation did not go past my approval, as it should be. But that run stopped until morning.
Moving the heavy tests to the closing of a release was, in practice, rebuilding continuous integration another way. At the time, I wrote:
"I started with the bottleneck in CI and came back with the bottleneck in CI. It's funny how the bottleneck loops around."
Tests before code
One question was left: who writes the test?
From the start, the rule was that the agent who develops never writes its own tests, so it doesn't test its own bias. But the test agent came after development and read the finished code. It paid the cost of rereading everything and, even so, wrote the tests from what the code did, not from what the task asked for.
So I flipped the order in a new development workflow. Using the to-do app example from chapter 2, the request "I want to add a due date to tasks" now goes like this:
- π€ Test first: the test agent reads the task and the acceptance criteria, not the code. Each criterion gets at least one test: the task saves the due date, the card shows the date, a task without a date keeps working. It looks only at the public parts the criteria mention, like the card screen and the function that saves the task, and never at the internals the change will rewrite. The tests go into the isolated copy before any line of code, and a lock keeps the developer from editing them.
- π€ What can't be tested is written down: a criterion that can't be checked automatically, like a visual detail or a measurement, goes to a list of its own, with the reason. No fake test to cover the criterion.
- βοΈ Proof that it fails: before any code, the engine runs only the new tests, once. They must fail because the due date doesn't exist yet, and not because of a typo in the test itself. A test that already passes goes back flagged to the developer: either it protects something that already worked, or it is wrong. A test file that doesn't even run goes back to whoever wrote it.
- π€ Develop until it passes: the developer implements until the tests pass. Along with the tests, it gets the list of existing tests that cover the files it will touch, taken from the code map. When it wants to run the tests, it asks, and the engine runs them and returns only what failed.
- π€βοΈ On screen too: in end-to-end tests, up to three scenarios per task are written first, in the given, when, then format: "given a task with a due date, when I open the list, then I see the date on the card". The agent writes the scenarios, the engine runs each step, and a fixer agent only steps in when a scenario fails.
The fourth night
The next night, the new workflow ran for real:
- 10 runs out of 10, all successful.
- 19 tasks delivered.
- The check that tests fail for the right reason passed clean on the first try in 11 out of 11.
- End-to-end tests caught three real defects the unit tests missed. One of them: a screen that slid almost 500 pixels to the side.
- The fixer got it right both ways. Twice it concluded the defect was in the app and didn't touch the test. Twice it fixed the test itself without changing what it checked.
One thing didn't change: the reviewer found serious problems in 7 of the 10 runs. Tests didn't replace review, and review didn't replace tests.
There was waste, of course. Twice the developer forgot to put the data-testid on the screen elements, the marker the scenarios use to find each button, field or card. Without it, the test can't find the element and fails. A whole round of the suite was thrown away each time. The fix is cheap, and it's the engine's job βοΈ: check that every data-testid the scenarios ask for exists on screen before running the suite.
The bottleneck moved again
It wasn't the first time the bottleneck changed places. It started in continuous integration, which ran on an external service. Then it went to conflicts when merging code, to the way the code was organized, to planning, to the computer itself, which froze with four heavy runs at the same time, and came back to testing.
With testing solved, it moved once more. Compared with the previous workflow, the new one cut a third of the tokens per task. The test agent, which used to take almost half the tokens, went down to a little more than a tenth. Now it was the developer who took three quarters, and more than half of the runs went back to it at least once to fix something.
The next saving is in those round trips: clearer acceptance tests and cheap checks before the suite are worth more than switching AI models.
The next step
That 1h20 of an agent waiting for a command to finish wasn't only a testing problem. The most expensive run of the whole project delivered nothing: an agent spent its time asking, over and over, whether something had finished.
In the next chapter, chapter 5, The most expensive token is the one of an AI waiting, I tell the story of that run and the rule that came out of it: waiting is a job for code.