Get a free audit

Findings

Reward hacks we have reproduced

Ways an agent earns full or high reward in public RL environments and eval suites without doing the task. Each one was reproduced from a script against a pinned commit of the shipped grader.

We report to maintainers first. A repository is named here only after its issue is public. Until then we describe the pattern and the count, with nothing that identifies the project. Bugs in our own environment are listed too, under the same rules.

100reward-hack bugs reproduced, each re-run by us from a script
20public RL-environment and eval repos, from 17 organisations
80+More than 80 of them not previously reported, as far as we could find

100 reproduced reward-hack bugs in 20 public repos from 17 organisations. Public so far: 4 entries from 4 issues · 4 fixed in our own environment · the rest in disclosure.

Filed in public

Public findings

Newest first. Status changes as maintainers respond; entries are never removed. If a maintainer closes an issue as intended, the entry stays with their reason.

H-2026-004Reported 2026-10-22

NVIDIA-NeMo/Gym · scicode

Pattern
The agent can write what the grader reads
Severity
High
What is rewarded
A sub-step passes when its program exits with code 0, and nothing confirms the asserts ran. One line, raise SystemExit(0), passes the sub-step before its tests run, and because each step’s code carries into later steps, every later sub-step passes too.
Affected
13 of 13 scored sub-steps on the 5 example problems (reward 1.0 on 5 of 5); a wrong-answer control passes 0 of 13; tested at 9240708
Suggested fix
Require a per-run random sentinel printed after the last assert, or run the tests from a harness that catches SystemExit and reports through a channel the solution can’t write to.

Issue #4901 on GitHub →

H-2026-003Reported 2026-10-22

NVIDIA-NeMo/Gym · single_step_tool_use_with_argument_comparison (SWE and Droid pivot configs)

Pattern
Loose matching pays for guessing
Severity
High
What is rewarded
Multi-word string arguments are compared by word overlap, and both pivot configs set the similarity threshold to 0.0, so the comparison can never fail. Replacing every real shell command with “echo done” gets full reward.
Affected
346 of 346 rows with a multi-word string argument (first 389 rows of the SWE pivot dataset); the Droid config the same way; tested at 9240708
Suggested fix
Reject a 0.0 threshold or treat it as no fuzzy matching, and compare command-like arguments by exact or normalised-exact match.

Issue #4900 on GitHub →

H-2026-002Reported 2026-10-22

PrimeIntellect-ai/verifiers · deepwiki, scratchpad

Pattern
Loose matching pays for guessing
Severity
Low (example environments, often copied as templates)
What is rewarded
Both rewards are bare substring checks on the final reply and never look at tool messages. In deepwiki, a list of languages or “It is Rust, not Python” scores 1.0 without calling the tool. In scratchpad, echoing the word from the prompt scores 1.0 on 20 of 20 tasks without touching the shared server the task is meant to test.
Affected
deepwiki 2 of 2 tasks; scratchpad 20 of 20; tested at 8859385
Suggested fix
Exact match on the normalised reply, and require the tool result in the trace (or a per-rollout value that isn’t in the prompt).

Issue #2948 on GitHub →

H-2026-001Reported 2026-10-22

PrimeIntellect-ai/verifiers · proposer_solver

Pattern
The answer ships with the task
Severity
High
What is rewarded
The proposer’s answer becomes the solvers’ gold label without any check, and the proposer is paid on solve rate alone. A false or ambiguous problem that splits the solvers earns the same maximum reward as a hard, correct one, and solvers who answer correctly in those episodes get 0.
Affected
Every episode (all 6 seed topics); tested at 8859385
Suggested fix
Require an executable solution in the contract, or gate on a reference solver agreeing with it; don’t train solvers on unconfirmed labels.

Issue #2947 on GitHub →

By pattern

All 100, by pattern

Every reproduced bug, filed or not, counted once under its main pattern. No repository names until each issue is public.

  1. The answer is never really checked

    33

    The score comes from something other than the result: keyword presence, the name of a tool that was called, the format of the reply, or only part of the end state.

  2. Loose matching pays for guessing

    18

    Substring checks, “X or Y” hedges that match whichever is right, tolerances that are documented but never applied.

  3. The agent can write what the grader reads

    17

    A reward file the agent plants, tests it can rewrite, a pass count scraped from output it controls, or an exit code of 0 that ends the run “successfully”.

  4. Doing nothing scores

    12

    An empty or no-op run passes because the end state was already true or the answer is never read. In one environment, doing nothing passed 16.8% of tasks.

  5. Extra actions and side effects are free

    9

    Spraying every tool, trying every reason code, or changing records nobody asked about still earns full reward.

  6. Failures drop out of the score

    4

    Agent-caused errors reclassified as infrastructure errors and excluded, or skipped criteria counted as passes.

  7. Valid work is rejected

    4

    An equally good alternative, or the right number written with a thousands separator, scores zero.

  8. The answer ships with the task

    3

    The gold answer is exposed to the agent, or a label is used as ground truth without being checked.

Our own environment

Grader bugs we found in our own work

The accounts-payable environment we build is audited the same way. All three bugs are fixed, and each one runs as a regression strategy.

  1. The grader accepted a write-off

    A correct resolution followed by writing off the held invoices passed three of five task families. An integrity check now allows a debit memo only where the policy requires one. Read the write-up →

  2. The trap was always first in the queue

    Acting on queue position alone, without reading the records, passed two families. Queue order is now drawn from the seed, and a blind positional guess runs as a regression strategy.

  3. Hold reasons were matched by keyword

    A correct hold could fail on its wording, and a wrong one could pass on a lucky word. Holds now carry one of four fixed policy codes.

  4. The public sample handed the agent the grader’s baseline

    Version 0.1 of the runnable sample passed the agent the environment object, so it could copy the reference actions or overwrite the starting state the grader compared against. Agents now run in a separate process with only the brief and a tools interface, and four regression cheats ship with the sample. Read the write-up →

Disclosure

Maintainers hear first

Each finding goes to the maintainers with its reproduction and a proposed fix. Write-ups that name a project wait about 30 days after filing, or less if the fix has shipped. Spotted a mistake on this page? Email hello@holdout.cc.

Read the disclosure policy →

Want this done on your graders?

The first audit is free: one environment, up to 10 tasks, with a reproduction and a proposed fix for every finding.

Get a free auditRead the notes