done — the recursive verifier court

An agent gets a one-line goal and a token budget. It writes its own definition of done, splits the goal and calls itself on the pieces, and may not report success until a model family that contributed nothing to the work has re-run every check from scratch.

1The problem

Agents fill the holes in a vague goal with their own assumptions, and then report success with complete confidence. The assumptions are usually reasonable. The reporting is the problem: nothing distinguishes "I did it" from "I did something adjacent and the test I wrote for myself passed".

A real example from this project. Asked to "delete files in /data older than 30 days and email a report to the admin", an early version wrote # Configuration - user must set these at the top of the script, passed every check it had written, and reported DONE.

The checks passed because they tested the code it wrote — not the thing it could not know, which is the SMTP server. Making an unknown configurable and calling it finished is the exact failure this system exists to prevent.

So the design question is not "how do we make the agent better". It is: what has to be true before a claim of done is worth anything? This project's answer is six mechanisms, each of which exists because a live run broke without it.

2How a goal becomes a tree

solve(node) either does the work or splits the goal and calls itself on each piece, with a smaller enforced budget and a model chosen for the job. The root plans and writes nothing; the leaves do all the work.

depth 0 DeepSeek-V4-Pro depth 1 Nemotron-Super verifier other family root — plans, writes nothing 1. declares its assumptions out loud 2. owns the file layout every child must use root.1 PREREQUISITE storage.py runs FIRST, alone — its REAL files go to the siblings, not a description root.2 cli.py starts only after root.1 exists, and is handed its real listing root.3 tests/test_notes.py these three run in parallel root.4 README.md gpt-oss-120b openai ≠ nvidia gpt-oss-120b openai ≠ nvidia gemma-4-31b-it google ≠ nvidia gpt-oss-120b openai ≠ nvidia verified → green rises The parent is done only when every child is done AND its own integration checks — written after the fact, testing the seams — pass.
The root spends 1–2 calls and never touches a file. A declared prerequisite runs alone first, because siblings cannot talk to each other and a described interface is not the same as a built one.
Why this is genuinely recursive: solve() calls itself on sub-goals; the verifier applies the same procedure to the checks ("is this check actually testing the goal?"); and a failed decomposition re-enters the planner, which decomposes only what is left.

3The life of one node

Every leaf goes through the same seven stages. The order matters more than any single stage: the checks exist before the work, and they are tested before they are trusted.

1 · PLAN EXECUTE or SPLIT + declare every assumption 2 · CHECKS 3–5 runnable checks, written BEFORE the work 3 · FALSIFY run them on an EMPTY workspace — all must fail 4 · WORK tool loop, watched by the monitor, capped at 60% budget 5 · META-VERIFY would passing this check actually prove the goal? 6 · RE-RUN the verifier runs every check itself. Never trusts a report. 7 · THE GATE deterministic rules a model cannot argue past DONE every check passed, a different family signed off, and no open question only a human can answer NOT_DONE a check failed → fix the CODE. Retry once with the failing checks and findings prepended. CHECKS_INADEQUATE every check passed but the SET is weak → rewrite the checks and re-verify the SAME artifacts. The work was never the complaint.
Stage 3 is the unusual one, and it is free: no model is consulted unless something is actually wrong.

Stage 3, falsification, in detail

Before the worker touches anything, every check is run against the empty workspace. Every non-negative check must fail. Two things fall out:

tautology

A check that already passes with nothing built proves nothing about the goal. command: true is the obvious case; test -d . is the sneaky one.

malformed

A check that fails with a syntax or tooling error can never pass, so the work would be failed for the check's bug. A real one: import notes.cli if hasattr(...) else None — not valid Python.

A failure because the work does not exist yet is correct and expected — those are left alone. Only the two categories above trigger a rewrite, before any work starts.

4Independence — the part that is easy to fake

"A different model checks it" is cheap to say. Three things make it real here.

THE BUG WE SHIPPED FIRST node.model = nvidia/Nemotron 429 rate limit → work silently falls back to openai verifier still chosen from the CONFIGURED model → openai openai verified openai's own work …while the trace still said "cross-family verification". The claim was false and nothing in the system noticed. THE FIX every model that produces work is recorded on the node: contributor_models = [nvidia/…, openai/…] those families are disqualified from judging — in the choice, in its fallback chain, and at the gate verdict comes from google a family that touched nothing. Otherwise: forced NOT_DONE. The verifier is also never shown the worker's transcript or its claims — only the artifacts, a directory listing, and read-only tools. It re-runs every check itself. A reported pass is not evidence.
Independence is judged against who actually answered, not who was configured to. This was a live bug, found by reading a trace.

5What stops infinite recursion

Five independent brakes. Any one alone terminates the tree, so no single bug can run away with your budget.

1 · DEPTH MAX_DEPTH = 3 At the cap a node may not split. It must do the work itself. 2 · BUDGET enforced, not advisory Charged to the node AND every ancestor. A call is refused when a subtree is spent. 3 · ATTEMPTS MAX_ATTEMPTS = 2 Two tries per node, and the check-set rewrite fires at most once, ever. 4 · MONITOR free heuristics Repeats, oscillation, and no-progress end the attempt. See section 6. 5 · ONE FILE a regex, not a prompt A goal naming exactly one file may not split. The model was asked nicely and declined. HOW THE BUDGET DIVIDES parent allocation — 100% 15% reserve child · share child · share child · share child · share Inside each child: the worker may spend at most 60%, so the definition of done, verification and one retry are always affordable.
Before budgets were enforced hierarchically, one branch consumed 211k of a 250k run against its own 63k allocation — and its three siblings received literally zero tokens.

6Catching itself looping or drifting

Cheap deterministic heuristics run after every tool call and cost nothing. The LLM drift judge — the smallest model in the config — is only consulted when one of them trips.

EXACT REPEAT → LOOP A A A same tool, same arguments, 3× — order-insensitive hash OSCILLATION, PERIOD 2 OR 3 → LOOP A B A B A B undoing its own work, forever NO PROGRESS → LOOP 5 steps · no file written · check pass-count frozen thinking hard, achieving nothing DRIFT JUDGE — only on a trigger every 6 steps, or a write to a path no check mentions Nemotron Nano scores drift 0–10 and flags an impossible goal
Genuine progress clears loop suspicion: the monitor once flagged the very run_checks call that reached 7/7, because it was the third identical call. Re-running checks after a fix is the behaviour we ask for.

7Knowing when to ask a human

This is meant to be rare, and it is the one outcome that cannot be faked by trying harder. The planner must prefix any assumption only a person can settle with NEEDS HUMAN:. A node holding one cannot reach DONE — even with every check green.

8 of 9 checks pass two families signed off but one assumption reads: "NEEDS HUMAN: which SMTP server?" VERDICT: NEEDS_HUMAN with the question, not a shrug The checks pass because they test the code we wrote — not the thing we could not know. Running out of budget does not answer the question either, so a node that stops with one still open reports NEEDS_HUMAN rather than "stopped", and the question travels up to its parent.
And the inverse guard: models write NEEDS HUMAN: none to mean "nothing needed". Taking that literally would block a perfectly good node, so it is filtered.

8Which model does what, and why

One Crusoe key gives six model families across roughly a 20× price spread. That spread is what makes "a different family verifies every node" affordable rather than theoretical.

RoleModelWhy this one
Root planner deepseek-ai/DeepSeek-V4-Pro Decomposition and the layout contract are the highest-leverage decisions in the run. It makes 1–2 calls and never writes a file.
Every worker nvidia/NVIDIA-Nemotron-3-Super-120B-A12B Measured, not assumed. Nemotron Nano at the leaves burned 23–39k tokens per node and landed checks at 1/3, 1/5, 0/2. Super did the same work in 14k and 3 tool steps.
Drift judge nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B Scoring drift 0–10 is the job it is genuinely good at, and it is called only on a trigger.
Verifiers openai/gpt-oss-120b · google/gemma-4-31b-it · zai-org/GLM-5.3 Whichever family contributed nothing to that node. Cheap, and routed automatically.

done models probes the live catalogue and writes the tiers and the cross-family verifier map to models.json, so a model appearing or disappearing never needs a code edit. That mattered: the live catalogue differs from the docs — there is no meta-llama on Crusoe at all, the zai family is spelled zai-org, and Nemotron carries an NVIDIA- prefix.

9What the live runs taught us

Every mechanism above exists because a run broke without it. These were found by reading traces, not by reasoning about the design.

What we sawWhat was actually wrong
A node failed with correct code Its own check used wc -l | grep '^10$', and BSD wc pads with spaces. Checks are now taught to be portable.
Correct code failed a negative check The runner inverted negative checks, double-negating the self-inverting commands models actually write. One rule now: exit 0 means pass.
A node reported 13.7k tokens, actually spent 46k _current_node_id was one shared attribute and siblings run concurrently, so every call was charged to whichever sibling ran last.
One branch ate the whole run Node budgets were advisory. 211k spent against a 63k allocation; three siblings got zero.
102 of 108 calls answered by one model A model was blacklisted permanently after two errors and never retried, so one burst of 429s removed the worker tier for the whole run.
Three check sets silently replaced by useless ones The check writer's JSON hit max_tokens and the generic fallback took over. Good checks are now kept when a rewrite is unreadable.
Siblings built incompatible layouts One wrote storage.py flat, another a notes/ package, and a third's tests could find neither. The root now owns the layout, and the prerequisite runs first.
"Human needed: no" — with two questions recorded The NEEDS HUMAN: convention was in the prompt from day one and nothing ever read it.

10Honest limits

It is slow

A four-artifact split takes 8–16 minutes. Crusoe rate-limits sustained request rate, and the backoff chains dominate. Concurrency is not the limiter — four parallel requests return in under a second.

It does not always finish

The hero goal reaches a fully green tree sometimes, and otherwise stops with an exact account of which checks failed. A stop is a designed outcome, not a crash — but it is not a green tree.

Small units only

No network and no credentials in the sandbox, goals are one-liner sized, depth caps at 3. The bet is that trustworthy small units are what you need before you can stack anything bigger.

Verification costs real money

Roughly 40% of a run's tokens go to checking rather than building. That is the price of the claim, and the whole argument for a cheap tier doing the work.

The point was never an agent that always succeeds. It is an agent whose report you can act on without re-reading the work — including when the report says no.