Replay

Replay takes a run you already recorded and executes it again against your current workflow definition, then diffs the two. You change a prompt, swap a model, or tighten a policy — replay shows you exactly how that change would have moved a real past run: did the output change, did routing take a different path, did latency or token usage shift.

Replay works from the local artifacts that tracing writes: the JSONL trace and the persisted run state under .assemble/runs/. If a run was recorded there (every asm run does this automatically), it can be replayed.

Why replay#

Multi-agent workflows are hard to regression-test with unit tests alone, because the interesting failures come from real inputs. Replay turns every recorded run into a regression fixture:

• Edit an agent's instructions, replay yesterday's run, and confirm the output still matches.
• Swap gpt-4o-mini for another model and compare latency and token usage on the same input.
• Reproduce a failure deterministically — tools are mocked from the recorded trace by default, so the replay does not re-issue refunds or re-send emails.

For pinning expected behavior across a whole set of inputs, use evals instead; replay is the tool for investigating one specific run.

replayRun#

replayRun loads the recorded run, rebuilds the workflow with mocked tools, and executes it.

import { replayRun } from "@assemble-dev/sdk";

const replay = await replayRun({
  definition: supervised,
  sourceRunId: "run_1",
  options: { fromStepIndex: 0, mockTools: true },
});

console.log(replay.status, replay.runId);

The result carries status ("completed" or "awaiting_approval"), the new runId, the sourceRunId it was replayed from, the output (or null), an approvalId when paused, plus the full state and ordered events — the same shapes workflow.run returns (see runtime). The replay's own trace brackets the run with replay.started and replay.completed events that link back to the source run.

Deterministic tool mocking#

With mockTools: true (the default), each tool call is answered from the recorded trace instead of executing the real execute function. That makes replays deterministic and side-effect free: the model calls are re-run against your current definition, but the world is not touched. Set mockTools: false only when you deliberately want live tool execution.

Partial replay with fromStepIndex#

fromStepIndex lets you skip the steps you trust. Artifacts and agent exchanges from steps before the index are seeded from the original run, and execution resumes at that step with your current definition. This is useful when only a late agent changed and re-running the early steps would just add noise.

Loading the source run#

loadReplaySource reads the recorded state and events without executing anything — replayRun uses it internally, and you need it yourself to build a comparison.

import { loadReplaySource } from "@assemble-dev/sdk";

const original = loadReplaySource({ runId: "run_1" });

console.log(original.state.status, original.events.length);

It returns { state, events } and throws an AppError with a NOT_FOUND code when the run state or trace file is missing — which usually means the run was never persisted or the directory is wrong.

Comparing the two runs#

compareRunToReplay diffs the original RunState against the replay's.

import { compareRunToReplay, loadReplaySource, replayRun } from "@assemble-dev/sdk";

const replay = await replayRun({ definition: supervised, sourceRunId: "run_1" });
const original = loadReplaySource({ runId: "run_1" });

const comparison = compareRunToReplay({
  original: original.state,
  replay: replay.state,
});

console.log(comparison.outputMatches, comparison.outputDiff);
console.log(comparison.latency.deltaMs);

routingMatches is often the most telling signal in supervisor workflows: an instruction tweak that flips a routing decision shows up here even when the final output happens to match.

Replaying from the CLI#

asm replay runs the same load–replay–compare flow and prints the diff (output, latency, and token usage). --workflow-file is required; the runId argument defaults to the last run recorded in .assemble/last-run.

asm replay run_1 --workflow-file src/workflow.ts

Warning: --no-mock-tools re-executes real side effects — refunds, emails, writes. Only use it when the tools are safe to run twice.

The dashboard replay diff view#

When runs are uploaded to the platform, the dashboard renders the same comparison as a replay diff view: original and replay side by side, output diffs by path, latency and token deltas, and the routing sequence of each run. It is the fastest way to review a batch of replays after a prompt change without reading raw JSON.