Skip to content

Testing with scripted models and recorded streams

An agent app is hard to test because the model is not deterministic. The stack splits the problem: scripted models fix the model's choices so everything else — tools, hooks, orchestration, events, state, UI — is tested for real, and live runs measure what a real model does with it.

Scripted models

kaboo-workflows' ScriptedModel replays a script of steps per agent: call this tool with these arguments, hand off to that agent, answer with this text. The real strands event loop runs, so a scripted test exercises your tools, hooks, prompts, interrupts and history exactly as production does (kaboo-workflows: Testing).

scripts:
  analyst:
    - - tool: sql-workbench___create_queries
        input: { items: [{ title: Churn rate by region, sql: SELECT … }] }
      - tool: render_a2ui
        input: { surfaceId: churn, components: [ … ] }
      - text: 362 customers churned this year.

Scenarios

The lab's scenario is the unit of testing across the stack: a workflow, the person's turns, the browser's answers to its tools, a script per agent, and what each turn must produce.

turns:
  - user: How has churn moved this year?
    tools:
      ask_user: { "Which months count as this year?": Since January }
    expect:
      outcome: success
      tool_calls: { analyze: 1, ask_user: 1 }
      surfaces: { a2ui-surface: 1 }
      max_lookups: 0
      max_bytes: 40000
      text_includes: [since January]

yarn lab run <scenario> drives it through the real stack — runtime, pipeline, mock services — and checks every run: verifyEvents on the recorded stream, the workspace afterwards, tool calls, lookups, response bytes. The same scenario runs live with --live, where the expectations become observations. Playwright journeys drive the browser through the same scenarios, including restarts, reloads and a second tab.

Recorded streams

Every lab run records its event stream. A recording is the best fixture a library can have: it is exactly what the other libraries send.

  • kaboo-react folds a recorded delegate run into cards, and checks that the compacted replay folds to the same cards.
  • kaboo-runtime plays the recording through the runner and checks that the stored, compacted log replays as a stream AG-UI accepts, with the same sub-agents.

Re-record when the wire changes, and the fixtures move with it.

Size and strategy

Two kinds of measurement come from the same runs:

  • Size. size-long-turn and size-long-thread gate the bytes a response carries. A response holds only what its turn adds — a deliberately long turn is about 230 KB, every turn of a long thread the same 28 KB — against AgentCore's 100 MiB cap.
  • Strategy. yarn lab compare runs the two sides of a decision and prints their tool calls, lookups, events, bytes, time and tokens side by side: workspace views against read tools, inline answers against canvases, the orchestrator asking against a sub-agent asking. Run them --live for a real model's numbers.

Where to look next