Get a free audit

Reward-hack audits

Find the reward hacks before your customer does.

Labs reject environments whose graders can be gamed. We attack your graders the way a policy under RL would, and hand you a reproduction and a fix for everything that gets through.

Request a free audit See a sample report

Free first audit: one environment, up to 10 tasks, results in about 5 working days. Happy to sign your NDA.

What we look for

What gets through graders

The same handful of mistakes turn up across environments from labs, tool makers and environment vendors. In public repos, we have seen each of these:

  1. The grader never reads the answer

    The score comes from something else in the transcript or the end state.

  2. Exit code 0 means pass

    A process that quits early passes every test.

  3. Hedges get full credit

    “It’s X or Y” matches whichever one is right.

  4. The agent writes its own reward

    The grader reads a file the agent can write.

  5. Fix the tests, not the code

    The agent edits the tests until they pass.

  6. Doing nothing scores

    In one environment, an empty run passed 16.8% of tasks.

Plus failures dropped from averages, partial credit that adds up to a pass, side effects nobody checks, and shortcuts specific to your tasks. We also check the other direction: correct work that a strict grader rejects.

Evidence

What we found in public RL environments

All public findings →
100reward-hack bugs reproduced, each re-run by us from a script
20public RL-environment and eval repos, from 17 organisations
80+More than 80 of them not previously reported, as far as we could find

100 reproduced reward-hack bugs in 20 public repos from 17 organisations. Filed publicly so far: verifiers #2947, #2948; Gym #4900, #4901. We report to maintainers first; specific repos appear only once the issue is public. How we found a write-off our own grader accepted →

What you get back

A report you can act on line by line

A summary table, then for each finding: what is rewarded, a reproduction script, the output it produced, the root cause linked to the file and line at a pinned commit, a proposed fix and a regression test. It ends with what the audit does not show.

See a sample report →
Built from two findings we filed in public. Not client work.

For environment vendors

Cheaper to find it before the lab does

A gameable grader found during acceptance means rework, another review cycle, and payment that starts its clock later. A pre-delivery audit takes about a week and leaves you with regression tests to run on every future release.

The offer

Start free. Pay for the full review only if it’s worth it.

Start here

Free first audit

We try to break your grader before your customer does. One environment, up to 10 tasks, no charge.

You send
Access to one environment: the task specs, the grader, and a way to run it or reference solutions. A repo link is enough to start.
We attack it
Shortcut strategies against every task: doing nothing, exiting early, writing to files the grader reads, hedged or padded answers, editing tests, and shortcuts specific to your tasks.
You get
A written report in about 5 working days. Each finding comes with a reproduction you can run, the affected tasks, a severity and a proposed fix.
If we find nothing
You get a written record of every strategy we tried on each task and how it scored, which you can hand to your customer at delivery.
Request a free audit

Then

Full review

The same method across a whole environment, with your fixes re-checked.

Scope
One environment, up to 25 tasks. Larger task sets are quoted per batch.
Price
Typically $1,000–3,000, fixed quote after a short scoping call.
You get
  • Findings report
  • A reproduction for each finding
  • Affected tasks
  • Proposed fixes
  • Regression tests to keep
  • A re-check of your fixes

Ask about a full review →

NDA
Happy to sign your NDA.
Your code
Your code and tasks are used only for your audit and never shared or published.
Access
We never need production credentials or customer data.
Confidentiality
Findings in a private environment are yours. We don’t publish them or name you without written permission.

How it works

Four steps

  1. Send a linkUse the form or email us. NDA first if you need one.
  2. Agree the scopeWhich environment and which tasks. By email, or a short call if you prefer.
  3. We attack the graderGeneral and task-specific shortcut strategies, each re-run to confirm it reproduces.
  4. You get the reportReproductions, affected tasks, severity, fixes and regression tests. Then, if you want, a full review.

For labs and platforms

Buying or hosting environments?

We can audit environments before you accept them, or review a batch of bounty submissions. Same method, same report format.

Talk about acceptance reviews →

Questions

Before you send anything

Our background is compliance and AML investigations: finding where a control is satisfied on paper and the risk is still there. A reward hack is the same thing in code.

hello@holdout.cc · Jimmy · GitHub: holdout-cc · Disclosure policy

What exactly is a reward hack?

A way for an agent to get a passing or high score without doing the task: a grader that never reads the final answer, tests that pass when the process exits early, answers matched so loosely that “X or Y” gets full credit, or a reward file the agent can write itself.

We already QA our environments. What would you add?

QA usually checks that correct solutions pass. We check the opposite: whether wrong, empty or tampered solutions also pass. It takes a different mindset and different tooling. In public repos maintained by capable teams, we have reproduced 100 such bugs.

We can’t share our environments.

Happy to sign your NDA first. If code can’t leave your infrastructure, we can work on a screen-share or inside your environment under your access controls.

What happens to our code and findings?

Your code and tasks are used only for your audit and never shared or published. Private findings are never published or attributed to you without written permission.

Our lab customer tests environments anyway. Why pay for this?

That is the point. When the lab finds a gameable grader, the environment comes back for rework, and acceptance, and payment, slip. Finding it before delivery costs less.

What if you don’t find anything?

You get a written record of every strategy we tried on each task and how it scored. That is useful evidence to include with your delivery.

How much time does it take on our side?

Not much: give access, point us at how to run the grader, and answer a few questions by email. The free audit report arrives in about 5 working days.

How much does it cost after the free audit?

A full review is typically $1,000–3,000 for one environment of up to 25 tasks, with a fixed quote after a short scoping call. You decide after seeing the free audit.

Who is behind Holdout?

Our background is compliance and AML investigations, where the job is finding the case that passes every check and is still wrong. Write to us at hello@holdout.cc. Public findings are filed from github.com/jimmy-holdout.

How do you handle bugs in public repos?

Maintainers first. We name a repo or bug publicly only once its issue is filed in the open, and wait about 30 days after filing before a write-up names the project. See the disclosure policy.

Request

Request a free audit

Send a link or a short description. We’ll reply with a few questions and what we’d need to start.

A personal address is fine.

What do you need?

Roughly how many tasks, what the grader checks, and when you deliver. A few lines is enough. Leave out confidential records and credentials.

Prefer email? hello@holdout.cc

We usually reply the same working day. Submissions go to Formspree and then to our inbox. See Privacy.