A coding agent in 5 files, plus a skeptic that catches fake fixes

Hand a coding agent a failing test and tell it make it pass . Sometimes it fixes the bug. Sometimes it edits the test, hardcodes the expected value, or deletes the assertion — and reports done ✅ with a straight face. Every "build an agent" tutorial stops before this problem. This one is built around it.

The agent itself is the easy part — a while loop around a chat model with one tool. 01_loop/agent.py is ~70 real lines that fix a real bug, live:

The hard part — Chapter 5, and the reason this repo exists — is an independent skeptic that doesn't take "done ✅" on faith. It checks the code's behavior against contract probes the agent never saw, so a fix that only games the visible test — editing it, or hardcoding/stubbing the source — is caught deterministically, no LLM required (a model judge reads the full diff on top, as a second opinion). You build both, one small chapter at a time. The scaffolding you're assembling has a name now — the harness — and this is harness engineering , from scratch.

git clone https://github.com/Saivineeth147/skeptic && cd skeptic python3 -m venv .venv && source .venv/bin/activate # recommended (system Python blocks pip: PEP 668) pip install -r requirements.txt # runs on any model via any OpenAI-compatible endpoint — one key, no billing setup: echo ' OPENROUTER_API_KEY=sk-or-... ' > .env # grab a key at openrouter.ai cd demo python ../01_loop/agent.py " the cart test is failing, fix it " # ⚠ runs in this dir; it's a sandbox demo Defaults to Claude Sonnet on OpenRouter; point AGENT_MODEL / OPENAI_BASE_URL at OpenAI, a local Ollama/vLLM server, or anything OpenAI-compatible. A full run costs a few cents (or nothing, on a free model).

Then watch the skeptic catch cheats — including the sneaky one that never touches the test:

python ../05_verify/verify.py --cheat # edits the TEST to pass -> ❌ REJECTED python ../05_verify/verify.py --cheat-stub # stubs the SOURCE, tests green -> ❌ REJECTED python ../05_verify/verify.py --cheat-eq # __eq__ that equals ANYTHING -> ❌ REJECTED python ../05_verify/verify.py # an honest agent fixes the code -> ✅ VERIFIED The cheat runs are the point: in each, the code passes the visible test yet is rejected, because a hidden contract check the agent never saw exercises the behavior the test left out. That's the whole idea — you can only verify work with a check the worker couldn't optimize against. ( --cheat-eq is the nastiest: an object whose __eq__ returns True for everything satisfies every bare assert in existence — the oracle catches it because its comparisons are type-pinned, and its probe requires a completion sentinel so an early sys.exit(0) can't fake a clean exit either.)

Frontier models game their tests at production scale — measured, not anecdotal:

Those documented strategies are exactly this repo's threat model — and the toy versions of them ship here as runnable, caught-by-CI cheats ( evals/cases.py ).

Chapters 1–4 build the agent everyone builds. Chapter 5 is why this repo exists. Each is a single file that runs, plus a "what breaks without this" note — every chapter exists because the previous one broke somewhere.

That gap is the whole reason this repo exists. Same task, two runs. Both show visible tests PASS in green — but the skeptic also checks behavior the visible test never touched, so the honest fix is VERIFIED and the stubbed-source cheat is REJECTED:

The loop-and-a-tool part, yes — that's a commodity, and it's Chapters 1–4 here because you should understand it. What almost nobody builds is the part that makes an agent trustworthy : an independent check that the work is real. A model graded on "make the test pass" will, often enough, pass the test without fixing the code. This repo goes to the part the tutorials skip — a skeptic that catches that — and makes it the whole point.

Can't the agent just fool the skeptic too? It's harder than it looks, because the strongest layer needs no model at all. The skeptic runs a hidden contract check — behavior probes on inputs the agent never saw — so a cheat that games the visible test (editing it, hardcoding, stubbing, special-casing the tested inputs) still fails on the inputs it couldn't optimize against. That's deterministic. The oracle is also hardened against the classic evaluator-gaming tricks from the reward-hacking literature: its comparisons are type-pinned (an object whose __eq__ returns True for everything still fails) and its probe requires a completion sentinel (an early sys.exit(0) that ends the process "cleanly" still fails) — both ship as runnable cheats. On top sits a model judge reading the full diff, as a best-effort second opinion. The honest limit: on a benchmark or this demo you own the spec and can supply the hidden check; pointed at an arbitrary repo ( final/ --verify ), the skeptic can only re-run your tests, flag test-file edits, and lean on the judge — unless you hand it a held-out suite with --holdout . "Tests are green" and "the code is correct" are different claims; the hidden check is what actually separates them.

Do I need Claude? No, for the agent — any OpenAI-compatible model works ( AGENT_MODEL ), and it runs on a local Ollama/vLLM server ( OPENAI_BASE_URL ), no cloud key required. Smaller models emit malformed tool calls more often; the code handles that rather than crashing. The judge is more model-sensitive, though: in my testing, small (7–8B) models both missed subtly-disguised cheats and falsely rejected correct fixes. So point the verifier's judge at a strong model ( JUDGE_MODEL , or --judge-model ), or lean on the deterministic checks (the hidden oracle / --holdout ), which don't depend on the judge at all. The default judge, Claude Sonnet, caught the source cheats here with no false rejects — but that floor is real, and the repo says so rather than hiding it.

Is it safe to run? It executes model-generated shell commands, so treat it like it. Chapters 1–3 have no gate — use a throwaway directory. Chapter 4 adds a permission gate (a coarse heuristic, not a security boundary). The real isolation is a container — a Dockerfile ships in the repo, giving the agent a scratch filesystem that holds only what you mount, so it can't reach your ~/.ssh , ~/.aws , or the rest of your disk:

docker build -t skeptic . # the deterministic cheat-catch needs no model, so this runs fully air-gapped: docker run --rm --network=none -e OPENROUTER_API_KEY=unused skeptic \ sh -c " cd demo && python ../05_verify/verify.py --cheat-stub " # -> ❌ REJECTED, offline The gate makes mistakes visible ; the sandbox makes them harmless . A live agent run needs egress to reach the model, so drop --network=none and pass a real key (or point it at a local model) — the filesystem isolation still holds. --yes warns unless it detects a sandbox.

Python version? 3.9+ (the code uses builtin generic type hints like dict[str, str] ) — checked in CI on both 3.9 and 3.12.

I kept reading "how to build an agent" posts that stopped at the loop and a get_weather tool, and never touched the parts that make a coding agent usable or trustworthy: what to do when the context fills up, how to stop it running something destructive, and — the part I care about most — how to know its output is real instead of taking "done ✅" on faith. So I built the version I wanted to read: every mechanism as its own runnable chapter, ending in a small CLI you can genuinely use.

If the 70-line agent makes you go "wait, that's all it is?" — that's the point. Everything after is turning that toy into something you'd trust.

The self-contained chapters make this easy to extend — see CONTRIBUTING.md . The most on-theme contribution: add a new cheat the skeptic should catch — drop it in evals/cases.py and watch the oracle reject it.

Run the suite (no API key needed): pytest tests/ -q , and the reproducible cheat-catch rate: python evals/run.py . Both run in CI on every push.

License MIT — see LICENSE . Learn from it, fork it, ship your own.