Test runs

A test run (a rehearsal) is a real turn of your agent, with its model, instructions and tools, in which a simulated world answers every tool call. The agent decides exactly what it would decide for real, but nothing is sent, no account is touched, and its connections don't even need to be connected yet. Use it to try new instructions before you deploy them, to check an agent handles the hard cases, and in your tests.

Rehearse a turn#

import noclick

support = noclick.Agent.from_name("support")
run = support.rehearse(
    "Order 1841 arrived broken. I want my money back.",
    scenario="Order 1841 was delivered on Sep 2 and paid by card; the customer has asked once before.",
).wait()

print(run.reply)
for call in run.tool_calls:
    print(call["tool"], call["arguments"], "->", call["result"])

rehearse starts the run and returns its operation (kind agent.rehearse); wait() gives the Rehearsal:

• reply: what the agent would have answered.
• tool_calls: each call it made, with the tool (its tool name, like stripe__create_refund), its arguments and the result the simulated world answered.
• transcript: the turn step by step: assistant steps (what it said between calls, then its reply) and tool steps, in order.
• scenario and cost_usd.

The agent's model is real, so its model connection must be connected; an agent whose model connection can't run is refused before the run starts, with the same error its turn would fail with.

Describe the world#

scenario says what the world is like, in plain words: the facts its answers keep to. Without one, the world is an ordinary, busy working day, so a search finds results and an inbox has mail, because an empty world teaches you little about an agent.

The world stays consistent within a run: an order the agent looks up has the same total when it looks again, and a refund it issues shows up when it lists refunds. Write scenarios for the cases you worry about: the angry customer, the refund over the limit, the calendar with no free slot.

Test in your test suite#

A rehearsal is a normal operation, so it fits in any test runner:

def test_support_refuses_gift_card_refunds():
    support = noclick.Agent.from_name("support")
    run = support.rehearse(
        "Please refund the gift card I bought last week.",
        scenario="The purchase was a $50 gift card, bought 6 days ago.",
    ).wait(timeout=120)
    assert not any(call["tool"] == "stripe__create_refund" for call in run.tool_calls)

Assert on what the agent did, the tools it called and with what, rather than on the exact wording of its reply, which varies from run to run.

Past runs are operations of kind agent.rehearse, so they're listed like any other:

support = noclick.Agent.from_name("support")
for op in noclick.Operation.list(kind="agent.rehearse", agent=support, limit=10):
    print(op.created_at, op.status, op.summary_text)

The developer console runs them from the agent's page, under Test runs, marked as simulated.

Simulate your own tools#

When you run the model loop yourself with a toolkit, simulated=True gives you the same world:

toolkit = noclick.Toolkit(["gmail", "stripe"], simulated=True, scenario="Ana has two unpaid invoices")
result = toolkit.call("stripe__list_invoices", {"status": "open"}, user="user_42")
print(result.text)   # two invoices, made up to fit the scenario; nothing was read

Definitions are the real ones, so your model sees exactly the tools it will have in production, and a call resolves as a real one does: a tool the real toolkit would refuse (a name with no connection or integration behind it) is refused here too, and a bound argument reaches the world filled in. Then the simulated world answers it, never the account: nothing is read or sent, the connection needn't be connected, and no policy holds it, since it reaches no one. handle and handle_openai work the same way.

One toolkit is one world: what one call creates, the next call finds, for 30 minutes after its last call. The world is your account's own, and its scenario is fixed by its first call. Build a new toolkit for a fresh world. Each answer is billed like a test run's world, as AI at NoClick's model price, and counts toward your spend caps; a call when your account can't pay raises nothing, it comes back as an is_error result.

What a rehearsal never does#

• It never calls a connection, sends a message, charges a card or writes to your data. The fence is on NoClick's side, around every tool NoClick runs for the agent, its code shell on the sdk harness included.
• It never reaches a person in its tools: the world answers an ask, and a notify or hand_off goes nowhere.
• It never writes to the agent's memory or conversations. It reads the agent's memory, so a rehearsal sees what the agent knows, and it runs in a conversation of its own that conversations() doesn't list.
• It doesn't fire your webhooks' agent.turn.* events or count as one of the agent's turns (its status and last activity don't move). Its operation settles like any other, as operation.succeeded or operation.failed with data.kind agent.rehearse.

A coding harness (Claude Code, Codex and the others) has its own shell and file tools, which are no NoClick tool call to fence: in a test run they run for real, inside a sandbox of the run's own with a workspace of its own, and appear in the transcript as tool steps with source harness. Their files never reach the agent's conversations; anything they could reach on the internet, they reach. The NoClick tools such an agent calls are answered by the world like any other.

Limits and billing#

• The agent's own turn is billed like any turn, on its model (nothing on your own key or plan), and a coding harness's sandbox time like a turn's. The simulated world's answers are billed as AI usage at NoClick's model price. All of it counts toward the agent's budget and your spend caps, shows in usage by agent, and is the run's cost_usd.
• A run is limited to the same length as a real turn.
• An account has up to 10 runs queued or running at once; another is refused with noclick.QuotaExceeded until one finishes.

Errors#

In the reference: Agent.rehearse, Rehearsal and the Toolkit constructor.