Sample audit report: NVIDIA NeMo Gym

A sub-step passes when its program exits with code 0, and nothing confirms the asserts ran. One line, raise SystemExit(0), passes the sub-step before its tests run, and because each step’s code carries into later steps, every later sub-step passes too.

# run from the repo root, with resources_servers/scicode on PYTHONPATH
import sys
from resources_servers.scicode.scicode_integration.runner import run_substep

# A sub-step program whose only test is `assert False`,
# preceded by the "solution" `raise SystemExit(0)`.
program = "raise SystemExit(0)\n\nassert False, 'tests must not pass'\n"
print(run_substep(program, timeout_secs=30, python_executable=sys.executable))

Output

{'passed': True, 'error': '', 'infrastructure_error': False, 'returncode': 0, 'timed_out': False, 'duration_s': 0.0629999999946449, 'exception_class': None, 'stdout_tail': '', 'stderr_tail': ''}

The test is assert False, and the sub-step still passes. Across the 5 problems in resources_servers/scicode/data/example.jsonl, a solution that starts with raise SystemExit(0) passed 13 of 13 scored sub-steps (reward 1.0 on 5 of 5), the same as correct code. A wrong-answer control passed 0 of 13.

build_test_program (runner.py:310) assembles one program: the agent’s code, then the target loader, then the sub-step’s asserts. run_substep (runner.py:320) runs it and sets passed = proc.returncode == 0 (runner.py:390). Nothing confirms the asserts ran, so code that exits with 0 before reaching them passes.

The agent’s extract_python_script strips only import lines, so the exit survives, and because each step’s code is carried into later steps, one such line passes every later sub-step too. The default scicode config is affected; scicodepile uses the same server.

Append a per-run random value after the last assert and require it in stdout along with exit code 0. This also covers sys.exit(), exit(), os._exit(0) and atexit tricks. The alternative: run the tests from a harness that executes the solution inside try/except BaseException (which catches SystemExit) and reports results through a channel the solution can’t write to.

@@ run_substep(), line 338 @@338    source_path = None339    started = time.monotonic()+Added:     sentinel = secrets.token_hex(16)  # new every run+Added:     program += f"\nprint({sentinel!r})\n"  # after the last assert340    try:@@ line 389 @@389    duration_s = time.monotonic() - started390-Removed:     passed = proc.returncode == 0+Added:     passed = proc.returncode == 0 and sentinel in _decode(proc.stdout)391    error = "" if passed else _bounded_stderr(proc.stderr)

Proposed, not merged. The sentinel sits in the same program file as the agent’s code, so code that reads its own source file could still find it; the harness option closes that too. The same exit-code rule exists in the upstream SciCode / nemo-skills harness.

Written to fail on 92407083 and pass once a fix is in; the last test checks that correct work still passes. Added for this report, not part of the issue.

import sys
import pytest
from resources_servers.scicode.scicode_integration.runner import run_substep

EXITS = ["raise SystemExit(0)", "import sys; sys.exit()", "import os; os._exit(0)"]

@pytest.mark.parametrize("exit_line", EXITS)
def test_early_exit_does_not_pass(exit_line):
    program = f"{exit_line}\n\nassert False, 'tests must not pass'\n"
    result = run_substep(program, timeout_secs=30, python_executable=sys.executable)
    assert result["passed"] is False

def test_asserts_that_run_still_pass():
    result = run_substep("assert 1 + 1 == 2\n", timeout_secs=30, python_executable=sys.executable)
    assert result["passed"] is True

Multi-word string arguments are compared by word overlap, and both pivot configs set the similarity threshold to 0.0, so the comparison can never fail. Replacing every real shell command with “echo done” gets full reward.

# run from the repo root
from resources_servers.single_step_tool_use_with_argument_comparison.common.verification_utils import (
    ActionComparator, ToolCallComparatorConfig)

expected = 'cd /workspace/numpy__numpy__ && python -c "import numpy; numpy.test()" 2>&1 | head -30'
for threshold in (0.0, 0.1):
    cmp = ActionComparator(config=ToolCallComparatorConfig(word_count_similarity_threshold=threshold))
    print(threshold, cmp._compare_strings(expected, "echo done", None))

Output

0.0 (1.0, None)
0.1 (0.0, <StepRewardCategory.ARGUMENT_VALUE_DIFFERENT: 'An argument value in a tool call is different than the expected value'>)

At the threshold both pivot configs ship with (0.0), "echo done" matches a long shell command with full score. On the first 389 rows of nvidia/Nemotron-RL-Agentic-SWE-Pivot-v1, replacing the real command with "echo done" passed 346 of 346 rows that have a multi-word string argument; the reference passes 389 of 389. The Droid config behaves the same way. The default threshold, 0.1, is stricter but still loose for long arguments: corrupting multi-word strings to "<first word> xyzzy" passed 214 of 700 such rows.

ActionComparator._compare_strings (verification_utils.py:597–634) splits multi-word strings into words and scores them with intersection / (expected + actual), rejecting only when that is below word_count_similarity_threshold. The SWE pivot and Droid pivot configs (line 8) set it to 0.0, and a similarity is never below 0, so the check cannot fail.

intersection_word_total = (expected_word_counts & actual_word_counts).total()
word_count_similarity = intersection_word_total / (expected_word_total + actual_word_total)
if word_count_similarity < word_count_similarity_threshold:
    return 0.0, StepRewardCategory.ARGUMENT_VALUE_DIFFERENT

Single-word arguments (ids, enums) fall back to exact match, so the hole is specifically commands, messages, queries and addresses.

Reject a 0.0 threshold in config validation, or treat it as “no fuzzy matching”. Compare command-like arguments by exact or normalised-exact match, or apply the existing argument_comparison_overrides precision, recall and length guards to free-text arguments. Proposed, not merged.

Written to fail on 92407083 and pass once a fix is in; the last test checks that correct work still passes. Added for this report, not part of the issue.

import pytest
import yaml
from resources_servers.single_step_tool_use_with_argument_comparison.common.verification_utils import (
    ActionComparator, ToolCallComparatorConfig)

EXPECTED = 'cd /workspace/numpy__numpy__ && python -c "import numpy; numpy.test()" 2>&1 | head -30'

def shipped_comparator(pivot):
    name = f"{pivot}_pivot_single_step_tool_use_with_argument_comparison"
    with open(f"resources_servers/single_step_tool_use_with_argument_comparison/configs/{name}.yaml") as f:
        server = yaml.safe_load(f)[f"{name}_resources_server"]["resources_servers"]["single_step_tool_use_with_argument_comparison"]
    return ActionComparator(config=ToolCallComparatorConfig(**server["tool_call_comparator_config"]))

@pytest.mark.parametrize("pivot", ["swe", "droid"])
def test_unrelated_command_is_rejected(pivot):
    assert shipped_comparator(pivot)._compare_strings(EXPECTED, "echo done", None)[0] == 0.0

@pytest.mark.parametrize("pivot", ["swe", "droid"])
def test_identical_command_still_passes(pivot):
    assert shipped_comparator(pivot)._compare_strings(EXPECTED, EXPECTED, None)[0] == 1.0