Evals

An eval suite pins the expected behavior of a workflow to a set of named cases. Each case runs the workflow with a fixed input and scores the result with deterministic metrics — did the output match, were the right tools called, did the output satisfy the schema, were approvals decided as expected. An optional LLM judge adds a rubric-based score on top.

Evals complement replay: replay investigates one recorded run, while evals guard a whole set of inputs on every change. Because asm eval exits non-zero on any failing case, a suite drops straight into CI.

Defining a suite#

evalSuite takes a name, a workflow, and at least one case. It validates everything up front: the suite name must be non-empty, case names must be unique, every case input must pass the workflow's inputSchema, and an expected.output must pass the workflow's outputSchema.

import { evalSuite } from "@assemble-dev/evals";
import { supervised } from "./workflow";

export const suite = evalSuite({
  name: "support-suite",
  workflow: supervised,
  cases: [
    {
      name: "small refund is auto-resolved",
      input: { ticket: "refund $42" },
      expected: {
        outputFields: { reply: "Refund issued." },
        toolCalls: [{ toolName: "issueRefund" }],
      },
    },
  ],
});

Case shape#

Each case is { name, input, expected? }. Everything in expected is optional — a metric that has nothing to check is reported as skipped, not failed.

Deterministic metrics#

Every case is scored with the same built-in metrics. Each produces a MetricResult with a name, passed, a score between 0 and 1, a human-readable reason, and a skipped flag.

A case passes when every non-skipped metric passes.

Running a suite#

import { runEvalSuite } from "@assemble-dev/evals";
import { suite } from "./workflow.eval";

const result = await runEvalSuite({ suite });

console.log(result.summary.passedCases, result.summary.failedCases);
console.log(result.resultsFilePath);

Useful options on runEvalSuite({ suite, options }):

The result reports evalRunId, a summary, per-case caseResults (each with its runId, passed, metrics, and latency/token performance), the resultsFilePath, and an uploadStatus of "uploaded", "skipped-no-key", or "failed". The summary aggregates totalCases, passedCases, failedCases, passRate, a per-metric metricBreakdown (evaluated / passed / failed / skipped), and suite-level performance (total and average latency, summed token usage).

Adding an LLM judge#

Deterministic metrics catch structural regressions; a judge scores the qualities you cannot assert exactly, like tone. defineJudgeMetric binds a model from providers to a rubric; the judge returns a score between 0 and 1 with its reasoning, and passes at passThreshold.

import { defineJudgeMetric, runEvalSuite } from "@assemble-dev/evals";
import { openai } from "@assemble-dev/providers";

const judge = defineJudgeMetric({
  model: openai({ model: "gpt-4o-mini" }),
  rubric: "Score 1 if the reply is polite and resolves the request, else 0.",
  passThreshold: 0.7,
});

await runEvalSuite({ suite, options: { judges: [judge] } });

Judge results appear alongside the deterministic metrics and count toward the case's pass/fail. Judges are skipped for runs that did not complete.

Note: Judges add a model call per case, so they cost money and are not deterministic. Keep them for qualities you cannot express as an expectation, and keep the pass/fail gate on the deterministic metrics where you can.

Loading cases from JSONL#

Datasets can live outside your source as JSONL — one case per line. loadEvalCasesFromJsonlFile({ filePath }) parses and validates each line; loadEvalCases(source) accepts either an in-code case array or { jsonlFilePath }, so a suite can switch sources without changing shape.

{"name":"small refund","input":{"ticket":"refund $42"},"expected":{"toolCalls":[{"toolName":"issueRefund"}]}}
{"name":"large refund pauses","input":{"ticket":"refund $500"},"expected":{"approvalDecisions":[{"status":"approved"}]}}

import { evalSuite, loadEvalCases } from "@assemble-dev/evals";

export const suite = evalSuite({
  name: "support-suite",
  workflow: supervised,
  cases: loadEvalCases({ jsonlFilePath: "cases.jsonl" }),
});

Serialized cases support output, outputFields, toolCalls, and approvalDecisions — outputPredicate is a function, so it is only available for cases defined in code. Invalid lines fail fast with the file path and line number.

Running from the CLI#

asm eval loads the suite file, runs it, prints the metric table, and saves the results.

asm eval src/workflow.eval.ts

The command exits 0 when every case passes and 1 when any case fails or the suite cannot be loaded — so a plain asm eval step fails your CI job on regression.

Where results land#

Every run writes a results file under .assemble/evals/ containing the run id, suite and workflow names, timestamps, the summary, per-case results, and the emitted eval.started / eval.completed trace events. When you are logged in (see quickstart), the same payload is uploaded to your workspace and appears in the dashboard's eval history on the platform, so you can track pass rates across commits. For a complete worked suite, see examples.