done — for judges

The 20-second version

You give it one sentence. It writes the code. The whole project is about the sentence it says at the end:

Verdict	What it means
DONE	Every check passed, and a model family that did none of the work re-ran them.
STOPPED	It doesn't work, and here is exactly which check failed and why.
NEEDS_HUMAN	It cannot finish without something only you know. Here is the one question.

The deliverable is not code generation. It is a status you can act on without re-reading the work — including when the status is "no".

The objection, answered first

"Isn't this just LLM-as-a-judge?"

It contains one, but the judge does not decide. A check passes because python3 -m notes add hi && python3 -m notes list exited 0. That is the operating system, not an opinion.

And the hard rule: a DONE verdict with any check failing is deterministically overridden to NOT_DONE, with the override written into the trace. The model cannot argue past ground truth.

The LLM writes the exam and flags suspicious questions. The computer grades it.

"Then isn't it just automated code review?"

Code review means reading code and having opinions. This barely reads code — it runs it. The verifier is handed artifacts, a directory listing, and read-only tools. It is never shown the worker's transcript or its claims, so a confident explanation cannot move the verdict.

	LLM-judge / code review	here
What is examined	code, read as text	artifacts, executed
What produces the verdict	model opinion	exit codes
Judge sees the author's story?	usually yes	never
Judge may be the same model?	usually yes	no — blocked against every model that contributed
Can a good explanation change the result?	yes	no

"So what is actually new here?"

Two things, and neither is a judge.

1 · Falsification, using no model at all. Before any work starts, every check is run against an empty workspace. They must all fail. A check that passes on an empty workspace is a fake check and proves nothing. A check that fails with a syntax error is a broken check and could never pass. Both are rewritten before the worker begins.

Everyone asks "is the work good?". Nobody asks "is the test real?" — and a fake test is precisely how an agent passes while failing. This step is deterministic and costs nothing unless something is wrong. It caught a real one on a live run: import notes.cli if hasattr(...) else None, which is not valid Python, so the work would have been failed for the check's own bug.

2 · Independence enforced against who actually answered. Not "pick a different model" in a config. Every model that produces work on a node is recorded in contributor_models, and those families are disqualified from judging it — in the choice, in its fallback chain, and at the final gate.

This closed a real hole in our own code: a Crusoe rate limit silently moved work to the fallback model, and the verifier was still being chosen from the configured model — so openai verified openai's own work while the trace claimed cross-family verification.

"What here is standard, honestly?"

We would rather say it than have you find it:

Recursive decomposition — standard plan-and-execute.
An LLM writing tests — common.
A second model checking — the cross-family idea is known; what is unusual is that it is structurally enforced rather than politely requested in a prompt.
Meta-verification ("would passing this check prove the goal?") — this is a judge, and it is the softest component. It is also why a judge verdict can never stand alone here: it is one input to a gate whose other inputs are exit codes.

Check it yourself in 30 seconds

Every claim on this page is reproducible offline, with no API key and no network.

git clone https://github.com/Kush614/done && cd done && uv sync

uv run pytest -q                                      # 94 passed, ~15s
uv run done replay runs/demo-final/demo_02_refusal    # the refusal, a real run
uv run done replay runs/demo-final/notes_cli_done     # a verified completion

Four complete runs ship in runs/demo-final/ — the node tree, the full append-only event log, and the files the agents actually produced. Nothing is mocked; these are traces of real runs against Crusoe.

Built during the hackathon · Oct 17, 2026

Every file in the repository is new work written that day. No pre-existing project, and no agent framework — no LangChain, no CrewAI, no LangGraph. The recursion, the budget algebra, the sandbox, the monitor heuristics, the falsification step and the verifier protocol are all our own code, in this repo, in 17 Python modules and 13 prompt files.

94offline tests, ~15s

22commits, all today

22live runs against Crusoe

$0.56total API spend

6model families used

12bugs found by live runs

The 94 tests are not decoration. They cover the recursion invariants (a split node cannot be done if a child failed; a same-family verdict is rejected; the check rewrite fires exactly once), the sandbox jail (../, absolute paths, symlinked files and directories, network blocked), the monitor's loop patterns, and a full end-to-end run with only the model calls faked — real files written, real pytest executed, real verifier gate.

The hard parts

Problem	What it took
Parallel agents sharing one workspace	Siblings run concurrently and cannot talk. One wrote `storage.py` flat, another built a `notes/` package, a third's tests found neither. The root now owns the file layout as a binding contract, and a declared prerequisite runs first, alone, with its real directory listing handed to the siblings.
Budgets that are actually limits	Every call is charged to the node and its whole ancestor chain, and refused when a subtree has spent its allocation. Before this, one branch consumed 211k of a 250k run against its own 63k allocation and its three siblings received zero tokens.
Token accounting under concurrency	`_current_node_id` was one shared attribute while siblings ran under `asyncio.gather`, so every call was charged to whichever sibling ran last — one node reported 13.7k against an actual 46k. Now a `ContextVar`, copied per task. This was not cosmetic: the worker's spend ceiling is computed from these numbers.
Surviving a real provider	Crusoe rate-limits sustained request rate. A model that errors gets a cooldown, never a blacklist — the original permanent blacklist removed the entire worker tier for a full run after one burst of 429s (52 fallbacks, 102 of 108 calls answered by one model).
Knowing when to stop and ask	The planner must prefix an assumption only a person can settle with `NEEDS HUMAN:`. A node holding one cannot reach `DONE` even with every check green, and the question propagates to its parent. Running out of budget does not answer the question either, so a node that stops with one open still reports `NEEDS_HUMAN`.

Why Crusoe makes this possible

One OpenAI-compatible key gives six model families across roughly a 20× price spread. That spread is the entire economic argument: "a different family verifies every node" is affordable only because the expensive model plans and the cheap models build and check.

Role	Model	Why
Root planner	`deepseek-ai/DeepSeek-V4-Pro`	1–2 calls, never writes a file.
Every worker	`nvidia/NVIDIA-Nemotron-3-Super-120B-A12B`	Measured, not assumed. Nemotron Nano at the leaves burned 23–39k tokens per node and landed checks at 1/3, 1/5, 0/2. Super did the same work in 14k tokens and 3 tool steps. Every file in a demo run is written by Nemotron.
Drift judge	`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`	Scoring drift 0–10 is the job it is genuinely good at, and it is called only on a cheap heuristic trigger.
Verifiers	`gpt-oss-120b` · `gemma-4-31b-it` · `GLM-5.3`	Whichever family contributed nothing to that node.

done models probes the live catalogue and writes the tiers and the cross-family verifier map to models.json, so a model appearing or disappearing never needs a code edit. That mattered — the live catalogue differs from the published docs: there is no meta-llama on Crusoe at all (our configured fallback did not exist), the zai family is spelled zai-org, and Nemotron carries an NVIDIA- prefix.

What we would not claim

It is slow. A four-artifact split takes 8–16 minutes against this account's rate limits. Concurrency is not the limiter — four parallel requests return in under a second. A four-artifact goal is a replay demo, not a live one.

It does not always finish. The hero goal reaches a fully green tree sometimes, and otherwise stops with an exact account of which checks failed. A stop is a designed outcome, not a crash — but it is not a green tree, and we would rather say so.

Verification costs about 40% of a run. That is the price of the claim, and the whole reason a cheap tier does the building.

Small units only. No network and no credentials inside the sandbox, one-liner goals, depth capped at 3. The bet is that trustworthy small units are what you need before anything bigger can be stacked on top.

The point was never an agent that always succeeds. It is an agent whose report you can act on without re-reading the work.