About
About Holdout
Holdout audits RL environments and evals for reward hacks, and builds business-workflow environments that pass the same test.
What we do
We try to pass your tasks without doing them. Every finding comes with a reproduction, the affected tasks, a proposed fix and a regression test you can keep. We also check the other direction: correct work that a strict grader rejects.
We build environments too, starting with an accounts-payable exception queue, and we audit our own the same way. Its grader bugs are published.
Where the method comes from
Our background is compliance and AML investigations: finding where a control is satisfied on paper and the risk is still there. A reward hack is the same thing in code. It is also why our environments pair every trap with a lookalike that must be handled differently.
What we don’t do
We don’t sell a model, host a marketplace or rank anyone. We don’t publish findings from private engagements, and we never reuse a client’s tasks in our own environments.
Getting in touch
Write to Jimmy at hello@holdout.cc, or use the contact form. We usually reply the same working day. Our public work is on GitHub, and the findings page lists every issue we have filed.
Have an environment going to a lab soon?
Send it to us first. The first audit is free.