What I wanted since Part I of this blog series was simple: to walk away (metaphorically mostly but also physically) from the keyboard and trust the outputs of AI.
I assumed that meant building an orchestration system. Then Claude and OpenAI started shipping more of the orchestration primitives themselves. I deleted the old orchestration system and attempted to rebuild it with the new primitives.
Somewhere in between, I also started this post. An earlier draft explained why I was retiring the orchestrator and how I’d redesigned orchestration, but Claude and Codex kept improving and improving.
All the while, my main question stayed the same: can I leave it alone yet?
The first pull request was convincing
In Part I, the first orchestrated pull request was the “aha” moment. The system had written code from a brief I gave it in a GitHub Issue and the tests were passing. I left feedback, which the agent promptly addressed, then started feeding it more ideas.
That was enough to change how I thought about my job. An idea could become a pull request and useful output without my involvement in every step.
The pipeline had a promising shape. Idea > brief > code > tests > review > fixes. Each stage had an output for the next one. I could describe the process and see how far the task was implemented.
Then, there were runs that reported success with the work not being completely “done”.
An agent reported done without pushing the changes to complete the task. The system needed to inspect the repository after the agent report.
That put me back at the keyboard, revisiting the definition of “done” with the agent.
Each failure added another patch
The first system accumulated recovery logic around edge cases. It was like whack-a-mole.
Turn limits could hit an agent halfway through a task. The orchestrator needed to make sure the agent resumed with its prior context. A crashed process could have one of many recovery paths. Retry limits were also needed.
“Retry the task” sounds like one operation, but resuming an unfinished task and starting a task anew are different (re-)starting points, with different token costs and recovery logic.
I kept adding and adding. I was spending an increasing amount of time working on the system that was supposed to let me step away.
The tools are catching up
I threw away the final rebuild of my orchestrator because Codex and Claude had started shipping the parts I was building.
In my Codex setup, sessions can now communicate with each other. An agent can read another session and spawn subagents or start a new session. I’d been building around the boundaries of a single coding session and those boundaries kept changing.
So I took a step back. There was less reason to maintain that orchestration system myself. I could use the new capabilities and focus on the gaps that remained.
Having several ways to organise agents left me with a choice about how to use them. I started looking more closely at the control plane and harness after wasting two weeks’ worth of token allowance in a few days. I was excited about Astra’s release and trying things with it. Luckily, OpenAI gave me resets (thanks thsottiaux!) so I could continue.
Where the tokens went
I asked for an audit of where the tokens were going. I had an agent generate a spreadsheet with tokens broken down by pull request and model and estimates of which work had been wasteful.
One documentation change had processed about two million Codex tokens across four sessions. Most of its input was cached and that total didn’t tell me exactly how much was wasted, but it gave me a method to investigate further.
Three wasteful patterns found in the audit are listed below. The waste estimates are judgements right now, so this is a shortlist of improvement opportunities rather than a ranked list.
Reviewing unchanged work again. Full reviews and coordination around them kept repeating after fixes. A follow-up should focus on the changed code and unresolved findings
Replaying context and repeating checks. Agents reread long histories or gathered evidence that was already available. Results need to be reused when the state is unchanged.
Polishing without a fixed finish line. The dreaded “scope creep”. Visual changes triggered more changes and another round of verification. We should mitigate this by agreeing on the target first and checking the state against the target after fixes.
I added cross-model reviews in earlier iterations of the orchestration system because I wanted to trust the results (I wanted a different model to challenge the implementation). In the audit, I needed to check how often those reviews ran and what they were being asked to review. The review skill now gives a follow-up reviewer only the repair diff and unresolved findings. Another full review runs only for changed risk, a separately routed review or an explicit request.
The spreadsheet grouped usage by PR because that made it possible to evaluate the cost of work. But a PR may contain only a documentation correction or weeks of product work, so tokens spent by PR may not truly reflect “useful” work shipped.
I want to know how many tokens it took to finish the work I wanted.
That’s why I broke the token usage down further and inspected what kinds of work tokens were used for inside the PR. What emerged was a picture of common token wasting patterns and, more importantly, where orchestration was suboptimal.
Choosing the orchestration strategy
I wanted to start making the orchestration strategy an explicit choice before execution.
There are many but the three strategies I’d start with are direct execution, bounded delegation and a limited review-and-repair loop. Each suits a different type of work.
Direct execution: the “naive” strategy but good one for simple tasks that don’t need orchestration overhead. One agent handles a small task and checks. A documentation correction is a good candidate
Bounded delegation: give an independent task to a worker with a clear brief. Run workers in parallel only when their work is independent
A limited review-and-repair loop: compare the result with acceptance criteria, fix the findings and check again. Set a limit on iterations. Visual work needs an agreed reference (e.g. design or screenshot)
My monorepo now has rules for choosing between the different orchestration approaches. When there’s a plan, its tasks, files and dependencies help determine what can run independently. For code changes, the task type and change scope determine the workflow stages. Risk adds review requirements. Models and budgets are separate choices. I want to be able to inspect why a route was selected and what it produced, and adjust the “orchestration routing” as I go.
I experimented with creating a game shortly after the Codex Astra release and spent (i.e. wasted) two weeks’ worth of tokens in a few days on a 3D version that never finished. The newer version of the game is 2D and the scope changed too, so I can’t isolate how much of the difference comes from orchestration. There’s more intent behind the orchestration. This time I adapted Anshu Chimala’s approach: Luna xhigh leads and Astra low reviews. Development feels a lot more token-efficient and hasn’t used much of my weekly allowance at all. The unfinished 3D attempt was useful learning.
Giving the agents a finish line
After strategy comes execution. Even after an orchestration strategy is chosen, the question of what the agents are supposed to accomplish remains.
That’s where the “Acceptance Criteria” enter. Acceptance criteria (AC) are a task’s “definition of done” and often a checklist of things that must be true for the task to be finished. The criteria give a separate reviewer something concrete to check. The tests should check that the criteria are met. The review should include smoke tests and a different model examining whether the implementation met the criteria (cross-model review). Without acceptance criteria, there’s no clear goal and I can get something I didn’t quite want. Or worse, a big PR full of things I didn’t want, which I learned when Claude went off the rails to complete side quests that weren’t in the spec because the spec was too sparsely defined.
The earlier failure where an agent reported “done” without pushing its branch still has relevance to my orchestration today. The final report needs evidence that the tasks actually completed and not just a “trust me, I done it”. When I come back to the keyboard (or phone), I want to be sure that all the acceptance criteria shipped.
The skills that sit alongside the code define these expectations. They are not defined by Codex, Claude or Cursor. When there’s a repeated failure, I promote it from LEARNINGS.md into an instruction or an automated check. I’m constantly adjusting and judging which checks are helping and what’s missing. The token usage audit shows how important orchestration still is, despite all the new features introduced by Codex, Claude and others.
Try it on one task
Start with one recent task. Ask an agent to audit the session logs where token counts are available. Codex and Claude Code already record token counts, so a custom harness isn’t needed to start. The audit needs to connect those counts to what the agents were doing and what they finished. If counts are missing, label them “unknown” rather than asking the model to estimate them. Group the measured tokens by work done, so you can see how much went into implementation and how much went into coordination or review. Keep input, cached input and output counts separate. Then inspect repeated context without assuming every repetition was waste (you’ll need to have your agent break it down). Record the completed tasks and the evidence that each acceptance criterion was met.
Next time you have a bounded task, run it once with one agent and once with an orchestrated team. You could compare Sol at medium effort with Luna xhigh leading and Astra low reviewing. Keep the task and available tools the same, then judge the two runs against the same acceptance criteria. Track your interventions alongside the tokens. That compares the complete workflows, including their model choices. One comparison won’t settle the strategy but it will give you a picture of how efficient one loop is versus the other.
I’m still figuring out which strategies work best. The coming weeks should give me more evidence about useful progress for the tokens spent. I want to leave the keyboard and come back to something that does exactly what I asked it to do. When all the cogs are in sync and the right orchestration strategy is chosen each time, I’ll have more tokens to deploy and more time away from the keyboard to focus on other impactful tasks or, indeed, take a breather!
I can’t leave it 100% alone yet. But the tech has advanced and the factory has grown again.
If you’re experimenting with AI in your workflow, follow along/subscribe! I also recently started aiengineerdex.com, where you can learn about fast-changing AI best practices.