Model providers

Assemble talks to language models through a small adapter interface, so your agents never depend on a specific vendor SDK. You pick an adapter per agent, pass your own key, and the adapter calls the provider's HTTP API directly from your process. Everything else — tracing, replay, evals — works identically no matter which provider sits behind an agent.

Eight adapters ship in @assemble-dev/providers: openai, anthropic, gemini, azureOpenai, bedrock, openrouter, cloudflareWorkersAi, and mockModel. The first seven call the real APIs with credentials you supply; mockModel needs no key at all and is what asm init scaffolds so you can run a full workflow before configuring anything.

The adapter concept#

A ModelAdapter is a plain object with a name and two methods: generateChatCompletion for free-text responses and generateStructuredOutput for responses parsed against a Zod schema. Every adapter returns the response content plus token usage (inputTokens, outputTokens, totalTokens) and latency in milliseconds, which is how traces and eval metrics stay provider-agnostic.

You attach an adapter to an agent through its model option:

import { anthropic } from "@assemble-dev/providers";
import { agent } from "@assemble-dev/sdk";

const reviewer = agent({
  name: "reviewer",
  instructions: "Review the draft for factual errors.",
  model: anthropic({ model: "claude-sonnet-4-5" }),
});

Because the model is just a value on the agent, different agents in one team can use different providers — see Mixing providers.

The openai, anthropic, and gemini adapters accept the same option shape (azureOpenai and bedrock share the tuning options but identify and authenticate differently — see their sections):

Per-request values passed by the runtime take precedence over the adapter-level temperature and maxOutputTokens defaults. Every provider response is validated with Zod before use; a non-2xx status or a malformed body surfaces as a typed error with the provider name and status attached.

openai#

Calls the OpenAI chat completions API (POST /chat/completions) with bearer authentication. The default base URL is https://api.openai.com/v1, so baseURL also works for OpenAI-compatible endpoints. Structured output requests send your schema as strict response_format json_schema — see Structured output.

import { openai } from "@assemble-dev/providers";

const drafter = openai({
  model: "gpt-4o-mini",
  temperature: 0.2,
  maxOutputTokens: 800,
});

The adapter's trace name is openai:<model>, for example openai:gpt-4o-mini.

anthropic#

Calls the Anthropic messages API (POST /v1/messages) with the x-api-key header and API version 2023-06-01. The default base URL is https://api.anthropic.com. System messages are joined into the request's system field — when an agent supplies both instructions and per-request system messages, instructions come first; see System messages and instructions. Because the messages API requires max_tokens, the adapter applies a default of 1024 output tokens when you set neither the adapter option nor a per-request value.

import { anthropic } from "@assemble-dev/providers";

const reviewer = anthropic({
  model: "claude-sonnet-4-5",
  maxOutputTokens: 2048,
});

The adapter's trace name is anthropic:<model>.

Beyond the shared options, anthropic() accepts two Anthropic-specific options:

Prompt caching#

With promptCaching: "auto", the adapter adds a cache_control: { type: "ephemeral" } breakpoint on the system prompt block and on the last tool definition, so Anthropic caches the prompt prefix — instructions plus tool schemas — and later calls that share it read from the cache instead of reprocessing it. Cached usage shows up on tokenUsage as cacheCreationInputTokens (tokens written to the cache) and cacheReadInputTokens (tokens served from it), and both fields flow through traces and run results, so you can see cache behavior per step.

const reviewer = anthropic({
  model: "claude-sonnet-4-5",
  promptCaching: "auto",
});

Cache reads are billed at a fraction of the normal input rate, so long, stable instructions — style guides, schemas, policy text — are exactly what you want cached. Cost estimation prices the cache fields separately.

Beyond the string shorthands, promptCaching accepts a settings object:

ttl: "1h" is for low-frequency agents: a scheduled job that wakes every half hour misses a 5-minute cache on every run and pays cache-write prices for nothing, while an hour-long entry survives between runs.

Incremental caching for multi-turn chats#

mode: "messages" keeps the "auto" breakpoints and adds one more on the final message's last content block. In a multi-turn chat, each request's transcript is an exact prefix of the next request's transcript — the breakpoint on the last message means each turn caches everything up to and including itself, so the next turn reads the whole history from the cache and pays full input price only for the new messages, instead of re-billing the entire transcript every turn:

const support = anthropic({
  model: "claude-sonnet-4-5",
  promptCaching: { mode: "messages" },
});

This is the right default for chat backends that pass a growing messages array on every request — see multi-turn input. The longer the conversation, the larger the saving.

Manual cache breakpoints#

For placement the modes cannot infer, any message accepts a cacheControl field, which puts a breakpoint on that message's last content block:

const result = await analyst.run([
  {
    role: "user",
    content: [{ type: "document", mediaType: "application/pdf", dataBase64: contractPdf }],
    cacheControl: { ttl: "1h" },
  },
  { role: "user", content: "Summarize the termination clauses." },
]);

cacheControl.ttl falls back to the adapter's ttl when omitted. Manual breakpoints work in every mode, including "off"; adapters other than anthropic ignore the field.

Anthropic allows at most four breakpoints per request. "auto" uses up to two (system prompt, last tool definition) and "messages" a third (the final message), leaving one for manual placement — spend it on the largest stable prefix, like a document every turn shares.

Prompt caching and dynamic instructions#

Caching works on exact prefixes. Fetching instructions from a prompt registry is fine — the text is the same on every call, so the cache hits. What breaks caching is any change to the system prompt text between calls: interpolating a user name, a timestamp, a request id, or any other per-request value into instructions produces a different prefix every time, and the hit rate silently drops to zero while you keep paying cache-write prices.

Version your prompts and pin a version per deployment rather than rendering them per request. Put per-request data — the user's name, the current time, the ticket contents — in messages, not in the system prompt:

const support = agent({
  name: "support",
  role: "Answers support tickets",
  model: anthropic({ model: "claude-sonnet-4-5", promptCaching: "auto" }),
  instructions: promptRegistry.getPrompt("support", { version: "v3" }),
});

await support.run([
  { role: "user", content: `Customer: ${customerName}\nTicket: ${ticketBody}` },
]);

If cacheReadInputTokens stays at zero across calls that should share a prompt, something is varying the system prompt text — diff two requests and pin whatever changed.

Extended thinking#

thinking: { budgetTokens } enables Anthropic's extended thinking with the given token budget. The adapter separates the response's thinking blocks from its text blocks, so agent outputs and outputSchema parsing behave exactly as without thinking — you pay for the thinking tokens, and the model reasons before it answers.

The thinking is surfaced, not dropped: ChatCompletionResult.thinkingText carries the full thinking text, and streams emit { type: "thinking_delta"; text: string } events alongside the usual text_deltas, so a UI can render the reasoning as it happens. Other adapters never set thinkingText or emit thinking_delta.

const analyst = anthropic({
  model: "claude-sonnet-4-5",
  thinking: { budgetTokens: 4096 },
  maxOutputTokens: 8192,
});

gemini#

Calls the Gemini generateContent API (POST /v1beta/models/<model>:generateContent) with the x-goog-api-key header. The default base URL is https://generativelanguage.googleapis.com. System messages become the request's systemInstruction, and structured output requests send your schema as the request's responseSchema — see Structured output.

import { gemini } from "@assemble-dev/providers";

const summarizer = gemini({ model: "gemini-1.5-flash" });

The adapter's trace name is gemini:<model>.

azureOpenai#

Calls a deployment on an Azure OpenAI resource. Azure authenticates with an api-key header and puts the deployment in the URL path ({endpoint}/openai/deployments/{deployment}/chat/completions?api-version=...) instead of a bearer token and a model in the body, so openai({ baseURL }) cannot reach it — this adapter can.

import { azureOpenai } from "@assemble-dev/providers";

const drafter = azureOpenai({
  deployment: "gpt-4o-mini",
  resourceName: "my-resource",
});

Beyond the shared temperature, maxOutputTokens, retry, extraBody, and fetchImplementation options, azureOpenai() identifies the deployment instead of taking a model:

The key resolves from apiKey, then AZURE_OPENAI_API_KEY_<NAME> when apiKeyName is set, then AZURE_OPENAI_API_KEY. The adapter supports full streaming with usage, and structured output via strict json_schema exactly like openai — see Structured output.

reasoningModel: true is required for reasoning-family deployments because the deployment URL hides the model name, so the adapter cannot detect the family itself — the plain openai adapter detects these models automatically by name.

The adapter's trace name is azure-openai:<deployment>.

bedrock#

Runs Anthropic models through the Amazon Bedrock Runtime InvokeModel API (POST https://bedrock-runtime.<region>.amazonaws.com/model/<modelId>/invoke), signing every request with a zero-dependency WebCrypto SigV4 implementation — no AWS SDK in the dependency tree, so it works on edge runtimes.

import { bedrock } from "@assemble-dev/providers";

const reviewer = bedrock({
  region: "us-east-1",
  modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
});

The shared temperature, maxOutputTokens, retry, extraBody, and fetchImplementation options apply too. The adapter supports generateChatCompletion and generateStructuredOutput (the schema is inlined into the system instruction rather than sent natively); it does not stream. It slots into agent-level cross-provider fallback chains like any other adapter.

The adapter's trace name is bedrock:<modelId>.

openrouter#

Calls the OpenRouter chat completions API (POST /chat/completions at https://openrouter.ai/api/v1) with bearer authentication — one adapter and one key for effectively the entire open-weights model catalog (Llama, Qwen, DeepSeek, Mistral, Kimi, GLM, and hundreds more) plus the closed providers OpenRouter fronts.

import { openrouter } from "@assemble-dev/providers";

const drafter = openrouter({
  model: "meta-llama/llama-3.3-70b-instruct",
});

The shared options — apiKey, apiKeyName, baseURL, temperature, maxOutputTokens, retry, fallbackModel, extraBody, fetchImplementation — all apply. Beyond them, openrouter() accepts:

The key resolves from apiKey, then OPENROUTER_API_KEY_<NAME> when apiKeyName is set, then OPENROUTER_API_KEY. Structured output uses strict response_format json_schema exactly like openai, with the same Zod v3 JSON-only fallback — see Structured output. Use extraBody for OpenRouter-specific routing fields such as provider preferences or transforms; fallbackModel names another OpenRouter model id, so the availability hedge can even cross underlying vendors (fallbackModel: "qwen/qwen-2.5-72b-instruct").

The adapter's trace name is openrouter:<model>.

cloudflareWorkersAi#

Runs Cloudflare's serverless inference — the @cf/... open-weights catalog — through the account-scoped OpenAI-compatible endpoint (POST https://api.cloudflare.com/client/v4/accounts/<accountId>/ai/v1/chat/completions) with bearer authentication.

import { cloudflareWorkersAi } from "@assemble-dev/providers";

const summarizer = cloudflareWorkersAi({
  model: "@cf/meta/llama-3.1-8b-instruct",
  accountId: "your-account-id",
});

Beyond the shared temperature, maxOutputTokens, retry, fallbackModel, extraBody, and fetchImplementation options:

The token resolves from apiKey, then CLOUDFLARE_API_TOKEN_<NAME> when apiKeyName is set, then CLOUDFLARE_API_TOKEN. Structured output uses strict response_format json_schema like openai — see Structured output. Some Workers AI models omit token usage from their responses; the adapter reports zeroed tokenUsage in that case instead of failing the call.

The adapter's trace name is cloudflare-workers-ai:<model>.

mockModel#

A fully local adapter for key-free development and deterministic tests. It accepts no credentials and makes no network calls:

import { mockModel } from "@assemble-dev/providers";

const model = mockModel({
  scriptedResponses: [
    '{"category":"refund","confidence":0.95}',
    "Refund approved for order ord_1.",
  ],
});

Each call consumes the next scripted response; when the script is exhausted the respond function takes over. If neither can produce a response, the adapter throws instead of inventing output — a test that makes one more model call than you scripted fails loudly. Structured output responses are parsed against the requested schema exactly like the real adapters, and token usage is estimated from text length so metrics stay populated. This makes mockModel the right default for eval suites that must be reproducible and for CI where no provider keys exist.

Structured output#

generateStructuredOutput — the method behind every agent outputSchema — sends your Zod schema to the provider as real JSON Schema, so the model is constrained at the API level rather than asked nicely:

The Zod parse still runs client-side as defense in depth: numeric and string constraints that provider schemas cannot express are stripped from the wire schema but enforced locally, so a response that satisfies the provider yet violates your schema still fails validation instead of flowing downstream. One shape difference to know: OpenAI strict mode always emits optional Zod fields — expect them present in the output rather than omitted.

Native enforcement requires a Zod v4 schema. When you pass a Zod v3 schema, the adapters detect it at runtime (isZodV4Schema) and fall back to the pre-0.3.0 behavior — JSON-object mode plus a JSON-only system instruction (response_format: { type: "json_object" } on openai, azureOpenai, openrouter, and cloudflareWorkersAi) — and the schema's own parse still validates the result client-side either way.

To derive the same wire schema yourself — for example for batch structured output — use convertZodSchemaToJsonSchema from @assemble-dev/providers.

Retries and fallback models#

Every real adapter accepts a retry option that retries transient provider failures with exponential backoff:

Retries apply only to failures that can plausibly succeed on a second try: HTTP 429, 5xx statuses, and network errors. A 400-class error — bad request, invalid key, model not found — is never retried; it fails immediately, because retrying it would produce the same result three times more slowly. Passing retry: {} enables retries with all defaults; omitting retry disables them.

On openai, anthropic, gemini, openrouter, and cloudflareWorkersAi, fallbackModel names a same-provider model that is tried exactly once after the primary model exhausts its retries — and only for those same retryable errors. Non-retryable errors never trigger the fallback.

const reviewer = anthropic({
  model: "claude-sonnet-4-5",
  retry: { maxAttempts: 3, initialDelayMs: 500, maxDelayMs: 8000 },
  fallbackModel: "claude-haiku-4-5",
});

The fallback stays within the provider, so keys, endpoints, and request shape are unchanged — it is an availability hedge, not a router. To fail over to a different provider entirely, chain whole adapters with agent-level fallbackModels.

Cross-provider fallback#

fallbackModels on the agent definition is an ordered chain of adapters tried after the primary. Each entry is a complete adapter, so the chain can cross providers — its own key, endpoint, and request shape:

import { anthropic, openai } from "@assemble-dev/providers";
import { agent } from "@assemble-dev/sdk";

const reviewer = agent({
  name: "reviewer",
  role: "Reviews drafts for factual errors",
  model: anthropic({ model: "claude-sonnet-4-5", retry: {} }),
  fallbackModels: [openai({ model: "gpt-4o" })],
  instructions: "Review the draft for factual errors.",
});

How the chain behaves:

• It advances per model request, after the failing adapter's own retries (and same-provider fallbackModel, if set) are exhausted — and only on the same retryable errors: 429, 5xx, network failures. A 400-class error, a cancellation, or a budget stop fails immediately, because every adapter in the chain would reject the same request.
• Streams fall back only while the connection is being established. Once the first stream event has been delivered, a failure surfaces instead of silently restarting the response on another provider mid-answer.
• In a tool loop the chain resets on every step, so a primary that recovers mid-run takes back over at the next step.
• AgentRunResult.modelName reports the adapter that actually served the final response, and costUsd prices each step by the adapter that served that step — a run that fell back is priced at the fallback's rates, not the primary's.

Keep the distinction in mind: adapter-level fallbackModel swaps the model id within one provider; agent-level fallbackModels swaps the whole adapter. For deliberate routing rather than failure handling, give different agents different adapters — see Mixing providers.

Extra request body fields#

Every real adapter accepts extraBody, a record of fields merged into the provider request body as-is. This is the escape hatch for provider parameters the adapter does not model — stop_sequences, top_p, or any field the provider ships tomorrow — without waiting for an SDK release:

const drafter = anthropic({
  model: "claude-sonnet-4-5",
  extraBody: { top_p: 0.9, stop_sequences: ["\n\nEND"] },
});

When you call an adapter directly, ChatCompletionRequest takes the same extraBody per request. Merge order is generated body ← adapter extraBody ← request extraBody — the caller wins, so a per-request value overrides the adapter default, and both override what the adapter generated. Values pass through unvalidated; a field the provider rejects fails the call with the provider's own error.

Cost estimation#

estimateModelCallCostUsd(modelName, tokenUsage) prices a model call from its token usage against a built-in pricing table:

import { estimateModelCallCostUsd } from "@assemble-dev/providers";

const costUsd = estimateModelCallCostUsd("anthropic:claude-sonnet-4-5", result.tokenUsage);

The table is cache-aware: cacheCreationInputTokens and cacheReadInputTokens are priced at the provider's cache-write and cache-read rates where those differ from the base input rate. The modelName accepts either a bare model id or an adapter trace name like anthropic:claude-sonnet-4-5.

For unknown models the function returns null — it never guesses. Run results surface the same value as costUsd, also null when pricing is unknown, so treat null as "no estimate" rather than "free".

Anthropic Message Batches#

For latency-tolerant jobs — nightly classification, backfills, bulk evals — Anthropic's Message Batches API processes requests asynchronously at roughly half the cost of the live API. createAnthropicBatchClient wraps it:

import { createAnthropicBatchClient } from "@assemble-dev/providers";

const batchClient = createAnthropicBatchClient({ apiKeyName: "batch" });

const batch = await batchClient.submitMessageBatch([
  {
    customId: "ticket-1",
    model: "claude-haiku-4-5",
    messages: [{ role: "user", content: "Classify: refund request for ord_1" }],
    maxOutputTokens: 256,
  },
  {
    customId: "ticket-2",
    model: "claude-haiku-4-5",
    messages: [{ role: "user", content: "Classify: password reset loop" }],
    temperature: 0,
  },
]);

await batchClient.waitForMessageBatchCompletion(batch.id, {
  pollIntervalMs: 30_000,
  timeoutMs: 3_600_000,
});

const results = await batchClient.listMessageBatchResults(batch.id);

The client takes the same credential options as the adapter — apiKey, apiKeyName, baseURL — and resolves keys the same way (see Keys and environment variables).

customId is how you correlate results back to your own records — batch results are not ordered like the submitted requests. Batches run outside the workflow runtime — use them for offline volume, and the adapters for anything a run needs synchronously — but the client emits its own trace events when you pass onTraceEvent; see Batch observability.

Batch prompt caching and structured output#

Each batch request accepts two optional fields beyond the basics. promptCaching: "auto" places a cache breakpoint on the request's system block, exactly like the live adapter's "auto" mode — worthwhile when many requests in one batch share the same long system prompt. Cached usage comes back on each succeeded result's tokenUsage as the same cache fields the adapter reports.

requireJsonResponse: true prepends the same JSON-only system instruction the adapters use for structured output, and parseAnthropicBatchStructuredContent validates each result against your schema:

import { parseAnthropicBatchStructuredContent } from "@assemble-dev/providers";
import { z } from "zod";

const ClassificationSchema = z.object({
  category: z.string(),
  confidence: z.number(),
});

for (const result of results) {
  if (result.outcome.type === "succeeded") {
    const classification = parseAnthropicBatchStructuredContent(
      result.outcome.content,
      ClassificationSchema,
    );
  }
}

The schema is any Zod v3 or v4 schema — or anything with a compatible safeParse, see Zod versions. Content that fails validation throws an AppError instead of passing malformed data downstream.

Batch observability#

The client options accept onTraceEvent and metadata:

const batchClient = createAnthropicBatchClient({
  apiKeyName: "batch",
  onTraceEvent: (traceEvent) => myCollectorSink.writeTraceEvent(traceEvent),
  metadata: { userId: "user_7", tags: ["nightly-classify"] },
});

Three event types cover the batch lifecycle:

metadata takes the same fields as run metadata and is stamped on every event the client emits, so batch spend aggregates by user, session, or tag exactly like run spend. costUsd on batch.result is priced from the model recorded at submission — list results with the same client instance that submitted the batch; a fresh client still emits the events but prices costUsd as null. For bridging these events into your own collector, see Bring your own observability.

Keys and environment variables#

Each adapter resolves its key from the explicit apiKey option first, then from apiKeyName (a named environment variable), then from the provider's default environment variable:

bedrock authenticates with AWS credentials instead of an API key: pass credentials explicitly, or set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and (for temporary credentials) AWS_SESSION_TOKEN.

If no source is set, the adapter throws a clear error at call time telling you which option or variable to set. Prefer environment variables for local development — the scaffolded .env.example from asm init lists the defaults — and pass apiKey as a value when keys arrive through a secret manager or any other source at runtime:

import { anthropic } from "@assemble-dev/providers";

const reviewer = anthropic({
  model: "claude-sonnet-4-5",
  apiKey: await secretManager.getSecret("anthropic-reviewer-key"),
});

Multiple keys per provider#

When different agents should use different keys from the same provider — separate billing, separate rate limits, separate teams — either pass each apiKey as a value, or name the keys with apiKeyName. A named key resolves from <PROVIDER>_API_KEY_<NAME>, where the name is uppercased and hyphens become underscores (team-a → ANTHROPIC_API_KEY_TEAM_A):

import { anthropic } from "@assemble-dev/providers";

const classifier = anthropic({ model: "claude-sonnet-4-5", apiKeyName: "classifier" });
const drafter = anthropic({ model: "claude-sonnet-4-5", apiKeyName: "drafting" });
const qa = anthropic({ model: "claude-sonnet-4-5", apiKeyName: "qa" });

export ANTHROPIC_API_KEY_CLASSIFIER=sk-ant-...
export ANTHROPIC_API_KEY_DRAFTING=sk-ant-...
export ANTHROPIC_API_KEY_QA=sk-ant-...

If the named variable is missing at call time the adapter throws an error naming it exactly, so a misconfigured key never falls back to the default silently.

The CLI can inject named keys per run instead of relying on your shell environment — asm run and asm dev accept repeatable --anthropic-api-key [name=]<key> and --openai-api-key [name=]<key> flags. See the CLI reference.

Bring your own keys#

Model credentials never leave your control:

• Assemble never proxies your model calls. Requests go straight from your process to the provider's API.
• Provider keys are never serialized into trace events, run state, or uploads; they live only in process memory and the adapter closure. Uploaded traces additionally pass through redaction that masks secret-shaped strings — see Security.
• Assemble never bills you for model usage. Your provider charges your own account directly.

Mixing providers in one workflow#

Because each agent carries its own adapter, a single workflow can route work to whichever model fits each step:

import { openai, anthropic, mockModel } from "@assemble-dev/providers";

const classifier = openai({ model: "gpt-4o-mini" });
const writer = anthropic({ model: "claude-sonnet-4-5" });
const stub = mockModel({ scriptedResponses: ["ok"] });

Traces record each step's adapter name, so the run timeline and replay diffs show exactly which provider produced which output — including token usage and latency per step, per provider.

Note: Swapping an agent from mockModel to a real adapter changes nothing else about the workflow. Scaffold with the mock, write your evals against it, then flip the model value when you are ready to spend tokens.