Discuss a pilot

Environments · Evaluations · Grader reviews

Test whether AI agents can finish the job.

We build simulated business workflows for training and evaluating AI agents, with synthetic records, tool interfaces, reference solutions and checks on the work completed.

For environment vendors, agent teams and AI research groups.

Explore the prototype Discuss a pilot

Accepting inquiries for custom environment and evaluation pilots.

Accounts-payable exception queue · seed 1 Scripted reference demonstration · Synthetic data

  1. Assigned task

    “Work the two invoices in your queue under the AP policy.” INV-898035INV-883564

  2. Agent investigates and acts

    flag_duplicate(INV-883564, original=INV-213392) place_hold(INV-883564, reason="duplicate") pay_invoice(INV-898035, amount=1108.80)

  3. Outcome checks + execution trace

    • Duplicate flagged and held, nothing paidPass
    • Lookalike paid in full to the verified accountPass
    • Nothing outside the queue changedPass
    • Vendor bank details not editedPass
Checks run on the final state. Six scripted shortcut strategies pass 0 of 150 runs.

What you can commission

Custom environments and grader reviews

Discuss a pilot →

Custom workflow environments

Commission a focused environment for a workflow your agent needs to handle.

  • Synthetic starting states
  • Tool interface
  • Task specifications
  • Reference solutions
  • Grading logic
  • Documentation

Environment and grader reviews

Find out whether an existing task rewards incorrect behaviour or rejects valid solutions.

  • Reproducible findings
  • Affected cases
  • Proposed fixes
  • Regression tests

Scope and acceptance criteria agreed before implementation. Synthetic data by default: no customer, employee or personal records.

Featured prototype

An accounts-payable exception queue

A fictional distributor, a written policy and five families of invoice exceptions. Four layers, all of it inspectable.

Walk through a task →
  1. 01

    World

    2 in queue12 open29 closed

    A seeded company with a month of activity to leave alone.

  2. 02

    Tools

    9 read8 writeforbidden

    The only way in. Invalid actions are refused and logged.

  3. 03

    Policy

    1. 1Three-way match (goods)
    2. 2Services
    3. 3Duplicates
    4. 4Bank details
    5. 5Bank feed
    6. 6Scope

    Six written rules define correct work.

  4. 04

    Grader

    ✓✓✓✓✓✓✓✓✓PASS
    ✓✓✓✓✓✓✓✗✓FAIL

    ✓5 task checks✓4 integrity

    Reads the final state. Every check must pass.

Evidence

Shortcuts pass most checks. They still fail.

Full results →
0/150runs passed by six scripted shortcut strategies
50–93%of individual checks those strategies still satisfy
85%what averaging would have scored the closest one

Five task families × five seeds × six strategies. These are regression tests, not model evaluations or exhaustive adversarial coverage.

How a pilot runs

Four steps, agreed up front

Start here

Start with a grader review

A fixed-scope first engagement on an environment you already run.

Scope
One environment, up to 25 tasks.
You send
Task specifications, the grader, and reference solutions or a way to run them.
You get
  • Written findings report
  • A reproduction for each finding
  • Affected tasks
  • Proposed fixes
  • Regression strategies to keep
Next step
A 30-minute scoping call, then a fixed quote before any work starts.
Book a scoping call
  1. 1Define the workflowAgree on the target capability, agent interface and expected outcomes.
  2. 2Build a representative sampleA small set of tasks with reference solutions.
  3. 3Evaluate and reviseInspect agent traces, grader behaviour and task realism.
  4. 4Deliver the agreed packageThe environment, documentation and results against the acceptance criteria.

What workflow does your agent need to handle?

Tell us what it needs to do, how it interacts with tools, and where your current evaluation falls short.

New: Our AP grader accepted a write-off it should have rejected.

Discuss a pilotDownload the runnable sample