How we report what we find

What counts as a finding

A behaviour that earns full or high reward without doing the task, or valid work that the grader rejects. We count it only once we can reproduce it from a script against a pinned commit of the shipped grader.

Maintainers hear first

We report each finding to the maintainers before anything else: through a GitHub issue, or by private email if the maintainers prefer it or the issue is sensitive. The report includes the reproduction, the affected tasks and a proposed fix.

When we name a project

A repository or bug appears on our findings page only once its issue is public. Write-ups that name a project or company wait about 30 days after filing, or less if the fix has shipped or the maintainers agree. Until then we describe patterns and counts only, with nothing that identifies the project.

What we never publish

Client code, client tasks, client findings, or anything under an NDA. Private engagements are never named or used as examples without written permission.

Disagreements and corrections

If a maintainer closes an issue as intended behaviour, the entry stays on the findings page with their reason. If we got something wrong, email us and we will correct it and say so on the page.

Credit

We credit the maintainers who fix what we report, and link their fix rather than paraphrase their comments.

Contact

Reports, corrections and questions: hello@holdout.cc. Public findings are listed on the findings page.