# Test runs

> Rehearse an agent against a simulated world: real turns, no side effects.

A **test run** (a rehearsal) is a real turn of your agent, with its model, instructions and tools, in which a simulated world answers every tool call. The agent decides exactly what it would decide for real, but nothing is sent, no account is touched, and its connections don't even need to be connected yet. Use it to try new instructions before you deploy them, to check an agent handles the hard cases, and in your tests.

## Rehearse a turn

```python
import noclick

support = noclick.Agent.from_name("support")
run = support.rehearse(
    "Order 1841 arrived broken. I want my money back.",
    scenario="Order 1841 was delivered on Sep 2 and paid by card; the customer has asked once before.",
).wait()

print(run.reply)
for call in run.tool_calls:
    print(call["tool"], call["arguments"], "->", call["result"])
```

```ts
import { Agent } from "noclick";

const support = Agent.fromName("support");
const run = await (await support.rehearse("Order 1841 arrived broken. I want my money back.", {
  scenario: "Order 1841 was delivered on Sep 2 and paid by card; the customer has asked once before.",
})).wait();

console.log(run.reply);
for (const call of run.toolCalls) {
  console.log(call.tool, call.arguments, "->", call.result);
}
```

`rehearse` starts the run and returns its [operation](https://www.noclick.com/docs/guide/operations.md) (kind `agent.rehearse`); `wait()` gives the `Rehearsal`:

- `reply`: what the agent would have answered.
- `tool_calls`: each call it made, with the `tool` (its tool name, like `stripe__create_refund`), its `arguments` and the `result` the simulated world answered.
- `transcript`: the turn step by step: `assistant` steps (what it said between calls, then its reply) and `tool` steps, in order.
- `tools_offered`: the tools NoClick gave the turn beside its own: the people it can notify or ask (`human__notify`), the reply of an [inbox](https://www.noclick.com/docs/guide/inboxes.md) whose mail it answers (`inbox__reply`), `wake_me` and the rest. A test run offers every tool a real turn has, and the world answers each, so a person is never told anything and no email goes out.
- `not_offered`, only when a test can't offer something a real turn has: a resource's writes (a bucket's put, a database's execute), which would change something for real.
- `note`, saying so, and when the agent wrote nothing: an agent that answers through a tool (an email's `inbox__reply`) may end with no text, as a real turn may, and its run still succeeds with an empty `reply`. It also says how far the run goes toward anyone hearing the agent: what it did to act was simulated, so it shows what the agent would do, never that its connection can (a real message's run does, in `tool_calls`), and an agent woken only by triggers whose replies nobody reads (a schedule, a webhook, an event, a queue), with no inbox, number, channel or person's texts to answer, that wrote a reply and called nothing that acts reached no one, since such a turn reaches people only through a tool.
- `scenario` and `cost_usd`.

The agent's model is real, so its model connection must be connected; an agent whose model connection can't run is refused before the run starts, with the same error its turn would fail with.

## Describe the world

`scenario` says what the world is like, in plain words: the facts its answers keep to. Without one, the world is an ordinary, busy working day, so a search finds results and an inbox has mail, because an empty world teaches you little about an agent.

The world stays consistent within a run: an order the agent looks up has the same total when it looks again, and a refund it issues shows up when it lists refunds. Write scenarios for the cases you worry about: the angry customer, the refund over the limit, the calendar with no free slot.

## Test in your test suite

A rehearsal is a normal operation, so it fits in any test runner:

```python
def test_support_refuses_gift_card_refunds():
    support = noclick.Agent.from_name("support")
    run = support.rehearse(
        "Please refund the gift card I bought last week.",
        scenario="The purchase was a $50 gift card, bought 6 days ago.",
    ).wait(timeout=120)
    assert not any(call["tool"] == "stripe__create_refund" for call in run.tool_calls)
```

```ts
import assert from "node:assert/strict";
import { test } from "node:test";
import { Agent } from "noclick";

test("support refuses gift card refunds", async () => {
  const support = Agent.fromName("support");
  const run = await (await support.rehearse("Please refund the gift card I bought last week.", {
    scenario: "The purchase was a $50 gift card, bought 6 days ago.",
  })).wait({ timeoutS: 120 });
  assert.ok(!run.toolCalls.some((call) => call.tool === "stripe__create_refund"));
});
```

Assert on what the agent did, the tools it called and with what, rather than on the exact wording of its reply, which varies from run to run.

Past runs are operations of kind `agent.rehearse`, so they're listed like any other:

```python
support = noclick.Agent.from_name("support")
for op in noclick.Operation.list(kind="agent.rehearse", agent=support, limit=10):
    print(op.created_at, op.status, op.summary_text)
```

```ts
import { Agent, Operation } from "noclick";

const support = Agent.fromName("support");
for (const op of await Operation.list({ kind: "agent.rehearse", agent: support, limit: 10 })) {
  console.log(op.createdAt, op.status, op.summaryText);
}
```

The developer console runs them from the agent's page, under **Test runs**, marked as simulated.

## Simulate your own tools

When you run the model loop yourself with a [toolkit](https://www.noclick.com/docs/guide/toolkits.md), `simulated=True` gives you the same world:

```python
toolkit = noclick.Toolkit(["gmail", "stripe"], simulated=True, scenario="Ana has two unpaid invoices")
result = toolkit.call("stripe__list_invoices", {"status": "open"}, user="user_42")
print(result.text)   # two invoices, made up to fit the scenario; nothing was read
```

```ts
import { Toolkit } from "noclick";

const toolkit = new Toolkit(["gmail", "stripe"], { simulated: true, scenario: "Ana has two unpaid invoices" });
const result = await toolkit.call("stripe__list_invoices", { status: "open" }, { user: "user_42" });
console.log(result.text);   // two invoices, made up to fit the scenario; nothing was read
```

Definitions are the real ones, so your model sees exactly the tools it will have in production, and a call resolves as a real one does: a tool the real toolkit would refuse (a name with no connection or integration behind it) is refused here too, and a [bound argument](https://www.noclick.com/docs/guide/connections.md#fix-arguments-the-model-mustn-t-choose) reaches the world filled in. Then the simulated world answers it, never the account: nothing is read or sent, the connection needn't be connected, and no [policy](https://www.noclick.com/docs/guide/policy.md) holds it, since it reaches no one. `handle` and `handle_openai` work the same way.

One toolkit is one world: what one call creates, the next call finds, for 30 minutes after its last call. The world is your account's own, and its scenario is fixed by its first call. Build a new toolkit for a fresh world. Each answer is billed like a test run's world, as AI at NoClick's model price, and counts toward your spend caps; a call when your account can't pay raises nothing, it comes back as an `is_error` result.

## What a rehearsal never does

- It never calls a connection, sends a message, charges a card or writes to your data. The fence is on NoClick's side, around every tool NoClick runs for the agent, its `code` shell on the `sdk` harness included.
- It never reaches a person in its tools: the world answers an `ask`, and a `notify` or `hand_off` goes nowhere.
- It never writes to the agent's memory or conversations. It reads the agent's memory, so a rehearsal sees what the agent knows, and it runs in a conversation of its own that `conversations()` doesn't list.
- It doesn't fire your webhooks' `agent.turn.*` events or count as one of the agent's turns (its status and last activity don't move). Its operation settles like any other, as `operation.succeeded` or `operation.failed` with `data.kind` `agent.rehearse`.

A coding harness (Claude Code, Codex and the others) has its own shell and file tools, which are no NoClick tool call to fence: in a test run they run for real, inside a sandbox of the run's own with a workspace of its own, and appear in the `transcript` as `tool` steps with `source` `harness`. Their files never reach the agent's conversations; anything they could reach on the internet, they reach. The NoClick tools such an agent calls are answered by the world like any other.

## Limits and billing

- The agent's own turn is billed like any turn, on its model (nothing on your own key or plan), and a coding harness's sandbox time like a turn's. The simulated world's answers are billed as AI usage at NoClick's model price. All of it counts toward the agent's budget and your spend caps, shows in usage by agent, and is the run's `cost_usd`.
- A run is limited to the same length as a real turn.
- An account has up to 10 runs queued or running at once; another is refused with `noclick.QuotaExceeded` until one finishes.

## Errors

| Error | When |
| --- | --- |
| `noclick.QuotaExceeded` | The account already has 10 runs queued or running. |
| `noclick.NeedsConnection`, `noclick.CredentialDisconnected` | The agent's model connection isn't connected, or lost its account. |
| `noclick.SpendCapExceeded` | The run would pass the agent's budget or a spend cap. |
| `noclick.PaymentRequired` | Your account can't pay for the run. |
| `noclick.OperationFailed` | The run failed, like a turn that fails. |

In the reference: [`Agent.rehearse`](https://www.noclick.com/docs/reference/python/Agent.md#Agent.rehearse), [`Rehearsal`](https://www.noclick.com/docs/reference/python/Agent.md#Rehearsal) and the [`Toolkit`](https://www.noclick.com/docs/reference/python/Toolkit.md#Toolkit) constructor.
