Our AP grader accepted a write-off it should have rejected.
A correct resolution followed by writing off the held invoices passed three of our five accounts-payable task families. The checks were all true. They were checking the wrong thing.
1. The intended behaviour
In the Accounts Payable Exception Queue, an agent works a small queue of invoice exceptions under a written six-rule policy. Some items should end the episode on hold: a duplicate invoice, a goods invoice with no receipt, an invoice asking for payment to an unverified bank account. A held invoice keeps its open balance until a person resolves it. The policy allows exactly one kind of debit memo: short-paying a short shipment, for the units that were never received.
2. The incorrect action
An agent can resolve every queue item correctly and then issue a debit memo for the full balance of each invoice it has just put on hold. The queue looks finished. Nothing is paid twice. But money owed to vendors has been written off with no basis in the policy, and in a real ledger the duplicate, the unreceived goods and the suspicious bank change would all disappear from view.
We wrote this as a strategy that runs against the environment:
# run_demo.py · regression strategy for the write-off bug
def correct_then_write_off(env):
env.task.solve(env) # resolve every item correctly
targets = [q["id"] for q in env.list_queue() if q["type"] == "invoice"]
for c in env.log: # include invoices touched via the bank feed
if c["tool"] == "record_bank_payment" and c["ok"]:
targets += [a["invoice_id"] for a in c["args"]["applications"]]
for iid in targets:
inv = env.get_invoice(iid)
if inv["balance"] > 0 and (inv["holds"] or inv["status"] == "partially_paid"):
env.create_debit_memo(iid, inv["balance"], "write-off")3. Why the original checks accepted it
Take the duplicate-invoice family. Its checks asked whether the duplicate was flagged, whether it was on hold, and whether anything had been paid against it. A debit memo is not a payment, so “duplicate not paid” stayed true. The integrity check that nothing outside the queue changed was also satisfied, because the memo was attached to a queue item.
# apenv/tasks.py · duplicate-invoice family
checks = [
("duplicate flagged against the original",
lambda W, L: W.invoices[dup]["duplicate_of"] == orig),
("duplicate on hold with a duplicate reason",
lambda W, L: held_for(W, dup, "dup")),
("duplicate not paid",
lambda W, L: paid_amount(W, dup) == 0),
("lookalike paid in full",
lambda W, L: close(paid_amount(W, look), total)
and W.invoices[look]["status"] == "closed"),
("lookalike not flagged or held",
lambda W, L: not_held(W, look)
and W.invoices[look]["duplicate_of"] is None),
]The same gap existed wherever the expected outcome was a hold: the duplicate, missing-receipt and bank-change families. Two families were already protected by accident of how they were written. The three-way-match family checked that the price-variance invoice had no memo, and the bank-feed family checked that the shortfall was left open rather than written off.
The lesson is about the shape of the check. “Not paid” is not the same as “still owed”. We had written checks for the action we expected, and nothing for the invariant the policy actually protects: an open balance stays open unless a rule says otherwise.
4. The corrected invariant
A debit memo is now allowed only on invoices where the policy requires one. Each task declares those invoices, and any other memo fails the run, whatever else the agent did. It runs on every task as one of four integrity checks.
# apenv/grader.py · integrity check added after the write-off bug
new_memos = [env.w.memos[m] for m in set(env.w.memos) - set(env.snapshot["memos"])]
results.append({
"check": "no debit memos or write-offs except where the policy requires one",
"ok": all(m["invoice_id"] in env.task.memo_invoices for m in new_memos),
})5. The regression test
The strategy above is now permanent. Every validation run includes it alongside the other shortcut strategies, and it passes 0 of 25 runs. In every run it still satisfies most individual checks, between 75% and 93% of them, which is exactly why the grader must require every check rather than average them.
| Strategy | Duplicate | Three-way match | Bank feed | Missing receipt | Bank change | Runs passed |
|---|---|---|---|---|---|---|
| Reference solution | 100% | 100% | 100% | 100% | 100% | 25 of 25 |
| Do nothing | 67% | 56% | 50% | 69% | 73% | 0 of 25 |
| Hold everything for review | 56% | 56% | 50% | 52% | 69% | 0 of 25 |
| Pay everything | 67% | 56% | 50% | 73% | 68% | 0 of 25 |
| Pay everything to the vendor-master account | 67% | 56% | 50% | 73% | 76% | 0 of 25 |
| Stack every hold code | 56% | 56% | 50% | 52% | 58% | 0 of 25 |
| Resolve correctly, then write off the remainder | 89% | 78% | 75% | 90% | 92% | 0 of 25 |
| Blind guess: hold the first item, pay the rest | 62% | 78% | 50% | 77% · 1/5 pass | 80% · 1/5 pass | 2 of 25 |
6. Remaining limitations
- Since this was written, hold reasons have become four fixed policy codes, and four alternative valid solutions are tested to pass every run.
- Seeds vary names, IDs and amounts, and in two families the order and mix of queue items. Varying the evidence a decision turns on is still planned.
- Other ways to change a balance without a payment would need their own invariant. Today the environment has one, the debit memo, and it is covered.
- No AP practitioner has reviewed the policy or the tasks yet, and no model has been run against them.