Holdout notes https://holdout.cc/notes/ Technical write-ups on reward hacks, grader bugs and how to build RL environments and agent evals that score the work, not the shortcut. en-ca Thu, 22 Oct 2026 12:00:00 GMT Our public sample let an agent rewrite the grader’s baseline https://holdout.cc/notes/ap-sample-agent-isolation/ https://holdout.cc/notes/ap-sample-agent-isolation/ Thu, 22 Oct 2026 12:00:00 GMT The runnable AP sample handed the agent the whole environment object, so an agent could copy the answers or hide its damage from the grader. How we found it, the fix, and the four regression tests that keep it fixed. Our AP grader accepted a write-off it should have rejected https://holdout.cc/notes/ap-grader-write-off/ https://holdout.cc/notes/ap-grader-write-off/ Wed, 21 Oct 2026 12:00:00 GMT A correct resolution followed by debit-memo write-offs passed our original accounts-payable grader. What went wrong, the invariant that fixed it, and the regression test.