Then the bill arrives, or a batch job takes four times longer than it should, or the app keeps hammering the API for an hour after the account ran out of budget. The retry loop was never the hard part. Rate limits on LLM APIs behave differently from a typical REST API, and handling them well is mostly about not retrying when it won't help, and not sending the request in the first place when you already know it will fail.

The examples below use the Anthropic TypeScript SDK (@anthropic-ai/sdk), but the ideas carry over to any provider.

Your SDK already retries, so don't stack another loop on top ​

Most official LLM SDKs ship with built-in retries. The Anthropic TypeScript SDK retries connection errors, 408, 409, 429, and 5xx responses twice by default, with a short exponential backoff. That covers transient blips, including the 529 overloaded_error the API returns when it is under heavy load.

The mistake is wrapping that in your own retry loop. Three attempts from your code times three attempts from the SDK is nine requests per call, and under real load that turns a brief slowdown into a stampede. Pick one layer to own retries. For interactive requests, the SDK default is usually fine. For background jobs, raise it per request instead of writing a wrapper:

import Anthropic from "@anthropic-ai/sdk";

// Interactive traffic: fail fast, let the UI show a "try again" state
const client = new Anthropic({ maxRetries: 1, timeout: 30_000 });

// Background job: more patience on this call only
const summary = await client.messages.create(
  {
    model: "claude-sonnet-5-5",
    max_tokens: 1024,
    messages: [{ role: "user", content: "Summarize this changelog..." }],
  },
  { maxRetries: 5 },
);

Not every 429 is a rate limit ​

This is the trap. On the Claude API, a 429 with error type rate_limit_error can mean two very different things:

• You exceeded a per-minute limit (requests, input tokens, or output tokens). The response usually carries a retry-after header with the number of seconds to wait. Retrying after that delay works.
• Your organization hit its monthly spend cap. Same status code, same error type, but no retry-after header, and error.details.error_code is enforced_spend_limit_reached. Access comes back at the start of the next month or when the cap is raised. Every retry, including the SDK's automatic ones, fails.

A naive loop treats both the same and burns through retries, logs, and queue time on a request that cannot succeed. Classify before you retry:

type Outcome = "retry-later" | "stop" | "fail";

function classify(err: unknown): Outcome {
  if (!(err instanceof Anthropic.APIError)) return "fail";

  if (err.status === 429) {
    const body = err.error as { error?: { details?: { error_code?: string } } };
    if (body?.error?.details?.error_code === "enforced_spend_limit_reached") {
      return "stop"; // page someone; do not requeue
    }
    return "retry-later";
  }

  if (err.status === 529 || (err.status ?? 0) >= 500) return "retry-later";
  return "fail"; // 400s are bugs in your request, not capacity problems
}

stop should pause the whole worker and alert a human. retry-later should put the job back on a queue with a delay, not spin in a tight loop. Note that a spend limit you set yourself, below the tier cap, comes back as a 400 invalid_request_error, which this function correctly treats as non-retryable.

Read the headers before you hit the wall ​

Retries are reactive. LLM APIs also tell you, on every successful response, how close you are to the limit. The Claude API returns headers such as anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining, and anthropic-ratelimit-output-tokens-remaining, each with a matching -reset timestamp.

Two details matter here. First, the API uses a token bucket, so capacity refills continuously rather than resetting on the minute. A limit of 60 requests per minute can be enforced as roughly one per second, which means bursts get rejected even when your per-minute total looks fine. Second, input tokens read from the prompt cache do not count toward the input-tokens-per-minute limit. If you send the same large system prompt on every call, caching it is often the cheapest way to raise your effective throughput.

You can read the headers with .withResponse() and use them to throttle a concurrency limiter:

let maxInFlight = 8;

async function call(params: Anthropic.MessageCreateParams) {
  const { data, response } = await client.messages
    .create({ ...params, stream: false })
    .withResponse();

  const remaining = Number(
    response.headers.get("anthropic-ratelimit-input-tokens-remaining") ??
      Infinity,
  );
  // Back off concurrency as headroom shrinks, recover slowly when it returns
  if (remaining < 50_000)
    maxInFlight = Math.max(1, Math.floor(maxInFlight / 2));
  else if (maxInFlight < 8) maxInFlight += 1;

  return data;
}

Feed maxInFlight into whatever queue or semaphore runs your jobs. This is crude, but it shifts you from "discover the limit by failing" to "slow down as you approach it," which is what keeps tail latency sane.

The takeaway ​

Treat rate limits as a capacity-planning signal, not an error to swallow. Let one layer own retries, separate "wait and try again" from "stop, nothing will work," and use the response headers to slow down before the API makes you. Sometimes the right fix is not code at all: batch APIs, prompt caching, or a smaller model for the easy requests can remove the pressure entirely. The provider's rate-limits page is worth reading end to end once, because the edge cases live in the footnotes.

That's the part the tutorials skip. The Model Context Protocol handles transport, discovery, and validation for you, and the official SDK hides almost all of it. What's left is the part only you can do: deciding which tools exist and describing them so a model that has never seen your codebase picks the right one. Your audience isn't a developer reading docs. It's an LLM reading a JSON Schema at runtime.

The minimum viable server ​

The TypeScript SDK's v2 line is now the stable release and targets the 2026-07-28 spec revision. It's split into @modelcontextprotocol/server and @modelcontextprotocol/client, and tool schemas use Standard Schema, so Zod v4, Valibot, or ArkType all work. If you're on @modelcontextprotocol/sdk, that's the v1 line; it still gets fixes, and there's a migration guide plus a codemod for the move.

Here's a small server for a deploy pipeline with one read-only tool:

import { McpServer } from "@modelcontextprotocol/server";
import { serveStdio } from "@modelcontextprotocol/server/stdio";
import * as z from "zod/v4";
import { listDeploys } from "./deploys.js";

function createServer(): McpServer {
  const server = new McpServer({ name: "deploys", version: "1.0.0" });

  server.registerTool(
    "list-recent-deploys",
    {
      title: "List recent deploys",
      description:
        "List the most recent production deploys for one service, newest first. " +
        "Use this before rolling back to find the deploy ID to roll back to.",
      inputSchema: z.object({
        service: z
          .string()
          .describe(
            'Service slug as shown in the dashboard, e.g. "billing-api"',
          ),
        limit: z
          .number()
          .int()
          .min(1)
          .max(20)
          .optional()
          .describe("Defaults to 5"),
      }),
      annotations: { readOnlyHint: true },
    },
    async ({ service, limit }) => {
      const deploys = await listDeploys(service, limit ?? 5);
      if (deploys.length === 0) {
        return {
          content: [
            {
              type: "text",
              text: `No deploys found for "${service}". Check the service slug.`,
            },
          ],
          isError: true,
        };
      }
      return {
        content: [{ type: "text", text: JSON.stringify(deploys, null, 2) }],
      };
    },
  );

  return server;
}

void serveStdio(createServer);
console.error("deploys MCP server running on stdio");

A few details carry most of the weight here. The inputSchema is the only schema you write: the SDK derives the JSON Schema the model sees, validates arguments before your handler runs, and infers your handler's types. Every .describe() survives into that JSON Schema, and per the SDK docs it's the only documentation the model gets for that argument. Note the console.error too. On stdio, stdout is the protocol channel, and a stray console.log corrupts the JSON-RPC stream.

You can poke at it without wiring up a host:

npx @modelcontextprotocol/inspector npx tsx src/index.ts

Errors are prompts, too ​

When a model sends arguments that fail the schema, the SDK returns an ordinary tool result with isError: true and a message like limit: Too big: expected number to be <=20. The model reads it and retries. That's the pattern to copy in your own handlers: return a failed result with a sentence that tells the model what to do next, rather than a stack trace.

"No deploys found. Check the service slug." is a useful error. TypeError: Cannot read properties of undefined is not. The model will act on whatever you return, so write errors the way you'd write a hint for a new teammate.

For results another program will consume, add an outputSchema and return structuredContent alongside the human-readable content. The SDK validates the structured value against the schema before it leaves your server, and advertises the schema in tools/list so clients can validate it as well.

Destructive tools need honest annotations ​

Now the tool that actually matters:

server.registerTool(
  "rollback-deploy",
  {
    title: "Roll back a deploy",
    description:
      "Roll a service back to a previous deploy. This changes production. " +
      "Only call this after the user has confirmed the exact deploy ID.",
    inputSchema: z.object({
      service: z.string(),
      deployId: z
        .string()
        .describe('ID from list-recent-deploys, e.g. "dpl_8f2c"'),
    }),
    annotations: {
      readOnlyHint: false,
      destructiveHint: true,
      idempotentHint: true,
    },
  },
  async ({ service, deployId }) => {
    const result = await rollback(service, deployId);
    return {
      content: [
        { type: "text", text: `Rolled ${service} back to ${result.deployId}.` },
      ],
    };
  },
);

Annotations are hints. They don't change how the SDK runs the tool; hosts use them to decide what to put in front of the user, like auto-approving a read-only tool and asking for confirmation before a destructive one. That means they're only as trustworthy as you make them, and a host can ignore them. Real authorization still belongs inside rollback(), not in a description string.

The 2026-07-28 spec made servers stateless ​

If you read an MCP tutorial from last year, one thing has changed underneath you. The current spec removed protocol-level sessions and the Mcp-Session-Id header from the Streamable HTTP transport, and dropped the initialize handshake; each request now carries its own protocol version and client capabilities. Servers that need state across calls are expected to mint explicit handles and pass them as ordinary tool arguments.

For tool design, that's good news. A deployId returned by one tool and passed to the next is exactly that kind of handle, and it's visible to the model instead of hidden in a session. The spec also now says servers SHOULD return tools/list in a deterministic order, which helps client-side caching and LLM prompt cache hit rates. Don't build your tool list from an unordered map.

When not to write one ​

If only your own app calls the model, you probably don't need MCP. Plain function calling against your provider's API is less code and one fewer process. MCP earns its keep when you want the same tools available in several hosts, like your editor, a desktop assistant, and a coding agent, without rewriting the integration for each.

When you do build one, start with two or three tools that map to tasks, not endpoints, and spend your time on descriptions and error messages. The SDK's Tools guide and the 2026-07-28 changelog are the two pages worth reading before you write a line.

That is the gap evals fill. The word sounds like it needs a vendor, a dashboard, and a research team. It does not. The version that catches most regressions is a file of thirty or so real examples, a script that runs your feature against them, and a few assertions. You can write it this afternoon and grow it from there.

Build the set from real failures ​

The most useful eval set is not synthetic. It is a list of inputs that have actually mattered: the bug reports, the weird edge cases from logs, the examples you hand-tested last time you changed the prompt. Each one gets an expected property, not necessarily an exact expected string.

Say you have a feature that classifies support tickets and extracts an order ID. A test case can look like this:

// evals/tickets.cases.ts
export interface EvalCase {
  id: string;
  input: string;
  expect: {
    category: "billing" | "shipping" | "bug" | "other";
    orderId?: string | null;
  };
}

export const cases: EvalCase[] = [
  {
    id: "refund-with-order",
    input: "I was charged twice for order #A-10293, please refund one.",
    expect: { category: "billing", orderId: "A-10293" },
  },
  {
    id: "no-order-id-present",
    input: "Your checkout page throws an error in Safari.",
    expect: { category: "bug", orderId: null },
  },
  {
    // Regression from a real report: the model invented an order ID
    id: "hallucinated-order-id",
    input: "Where is my package? It has been two weeks.",
    expect: { category: "shipping", orderId: null },
  },
];

The third case is the important pattern. Every time a user reports a bad output, it becomes a case before you touch the prompt. That way the fix is verified, and the bug can never quietly come back.

Check cheaply first, judge second ​

Most of what goes wrong with LLM output is checkable with plain code: the JSON did not parse, a required field is missing, the category is not one of the allowed values, an ID appears that was never in the input. These checks are free, deterministic, and fast, so run them on every case.

// evals/run.ts
import { cases } from "./tickets.cases";
import { classifyTicket } from "../src/classifyTicket"; // your real feature

let failures = 0;

for (const c of cases) {
  const out = await classifyTicket(c.input);
  const problems: string[] = [];

  if (out.category !== c.expect.category) {
    problems.push(`category: got ${out.category}, want ${c.expect.category}`);
  }
  if ("orderId" in c.expect && out.orderId !== c.expect.orderId) {
    problems.push(`orderId: got ${out.orderId}, want ${c.expect.orderId}`);
  }
  // Grounding check: any ID we return must literally appear in the input
  if (out.orderId && !c.input.includes(out.orderId)) {
    problems.push(`orderId ${out.orderId} not present in input`);
  }

  if (problems.length) {
    failures++;
    console.log(`FAIL ${c.id}\n  ${problems.join("\n  ")}`);
  }
}

console.log(`${cases.length - failures}/${cases.length} passed`);
process.exit(failures ? 1 : 0);

Wire node evals/run.ts (or tsx) into a CI job that runs when prompt files or model config change, and you have a regression gate. Expect some run-to-run variance even at low temperature. Run flaky cases a few times and gate on a pass rate rather than demanding a perfect score on every run.

Only reach for a model-graded check, "LLM-as-judge," for properties code cannot see: tone, whether a summary is faithful, whether a reply actually answers the question. Tools like promptfoo make this a few lines of config:

tests:
  - vars:
      ticket: "I was charged twice for order #A-10293, please refund one."
    assert:
      - type: is-json
      - type: llm-rubric
        value: The reply acknowledges the double charge and does not promise a refund timeline.

Do not trust the judge until you have checked it ​

A judge model is still a model. It has known biases: it tends to prefer longer answers, can favor outputs from its own model family, and can be swayed by the order in which options are shown. A judge score nobody has validated is a confident number, not a measurement.

The fix is cheap. Take forty or fifty outputs, label them pass or fail yourself, and run the judge on the same set. If it disagrees with you often, tighten the rubric until it does not, and repeat the check whenever you change the judge prompt or model. Judges also cost real money and latency on every run, so keep deterministic checks as the first line, run the judge only on the cases that need it, and cache results for unchanged outputs.

When does none of this make sense? If the "AI feature" is really a fixed transformation that a regex or a lookup table could handle, write that instead and test it normally. Evals exist because LLM output is probabilistic. Do not adopt that cost where you do not need it.

Start with the cases you already have in your head and your bug tracker, make every reported failure a new one, and treat a dropping pass rate exactly like a failing unit test. The platforms are useful once the set is large and you want trends over time, but the habit matters more than the tool.

Prompt caching exists for exactly this. The provider saves the model's processed state for the start of your prompt, and the next request that begins with the same tokens skips that work. It's cheaper and faster. The catch is that it matches on an exact prefix, which means the order you assemble your prompt in suddenly matters a lot more than it used to.

How the prefix match works ​

The cache key is the prompt itself, read from the beginning. If request two starts with the exact same tokens as request one, up to a cache point, it can reuse request one's work. Change one token near the top, a timestamp, a user's name, a reshuffled tool list, and everything after that token is a miss.

Both major APIs follow this model, with different defaults. On Anthropic's Messages API you opt in with cache_control. You can either drop a single top-level cache_control and let the API place the breakpoint on the last cacheable block (it moves forward as a conversation grows), or put cache_control on specific blocks yourself, up to four breakpoints. Prefixes are built in a fixed order: tools, then system, then messages. Changing a tool definition invalidates the whole cache; changing a message only invalidates the messages level.

OpenAI caches automatically once a prefix crosses the minimum length (1,024 tokens on GPT-5.6 and later), and lets you pass a prompt_cache_key to improve routing so similar requests land where the cache lives. Newer models also support explicit breakpoints.

The pricing shape is similar on both. Writing to the cache costs a bit more than normal input (1.25x for Anthropic's default 5-minute TTL, 2x for the 1-hour option), and reading from it costs a fraction: 0.1x base input on most models, lower on some newer ones. Check your model's pricing page before you build a budget around it.

Ordering your prompt for hits ​

Here's a support assistant using Anthropic's TypeScript SDK. The stable content goes first and carries the breakpoint; anything that changes per request goes after it.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

export async function answer(
  question: string,
  user: { name: string; plan: string },
) {
  const response = await client.messages.create({
    model: "claude-sonnet-5-5",
    max_tokens: 1024,
    tools: SUPPORT_TOOLS, // stable, same order every request
    system: [
      {
        type: "text",
        text: SUPPORT_INSTRUCTIONS + PRODUCT_DOCS, // thousands of static tokens
        cache_control: { type: "ephemeral" },
      },
    ],
    messages: [
      {
        role: "user",
        // per-request context lives AFTER the breakpoint
        content: `Customer: ${user.name} (${user.plan} plan)\n\n${question}`,
      },
    ],
  });

  const { cache_creation_input_tokens, cache_read_input_tokens, input_tokens } =
    response.usage;
  console.log({
    cache_creation_input_tokens,
    cache_read_input_tokens,
    input_tokens,
  });
  return response;
}

The tempting version puts Customer: ${user.name} at the top of the system prompt because it "reads better." That one line makes every customer's prefix unique, and the cache never hits across users.

Do the math on why ordering is worth caring about. Say the static block is 6,000 tokens and you get 10 requests inside the TTL window. Without caching you pay for 60,000 input tokens. With caching you pay one write at 1.25x (7,500 token-equivalents) plus nine reads at 0.1x (5,400), so 12,900 total. That's roughly 21% of the uncached cost for the static portion, and the time to first token drops too, because the model isn't reprocessing those tokens.

The quiet ways it breaks ​

Caching fails silently. A prompt that misses the cache still works; it just costs full price. So the most important habit is logging the usage fields. On Anthropic, if both cache_creation_input_tokens and cache_read_input_tokens are 0, nothing was cached. On OpenAI, look at cached_tokens under usage.input_tokens_details (Responses API) or usage.prompt_tokens_details (Chat Completions).

The usual culprits:

• Too short. Each model has a minimum cacheable length. Below it, cache_control is ignored with no error. On Anthropic it ranges from 512 tokens on the newest models up to 4,096 on some older ones.
• Non-deterministic serialization. Building tool schemas from an object whose key order changes, or sorting docs by "last updated," produces a different prefix each time.
• Dynamic values up top. "Today is October 30th" in the first line of the system prompt invalidates the cache every day at minimum. Move it below the breakpoint.
• Changing knobs. On Anthropic, toggling things like tool_choice, images, or thinking settings invalidates part of the cache. Pick them per feature, not per request, where you can.
• Low traffic. A 5-minute TTL does nothing for a feature that gets one request an hour. Either use a longer TTL (and pay the higher write rate) or accept that caching isn't buying you anything there.

That last one is worth taking seriously. If your requests are sparse and your prompts are short, caching adds a write surcharge and complexity for no gain. It pays off for chat, agents with long tool lists, RAG over a stable corpus, and batch jobs that hit the same instructions thousands of times.

Start by logging cache fields on every call for a day. If the read numbers are near zero, the fix is almost always ordering: stable things first, changing things last, and nothing clever in between. Anthropic's prompt caching docs and OpenAI's prompt caching guide both have per-model minimums and pricing tables worth bookmarking.

Prompt injection has sat at the top of the OWASP Top 10 for LLM applications for a reason. It is not a jailbreak problem you can patch with better wording. It exists because a model receives instructions and data in the same channel, as tokens, and nothing in the architecture marks one as authoritative. The defenses that actually hold up treat that as a given and work around it in code.

The lethal trifecta: where injection becomes theft ​

Simon Willison's "lethal trifecta" is the most useful threat model I know for this. An agent becomes dangerous when one session combines three things:

1. Access to private data (your inbox, a customer database, internal docs)
2. Exposure to untrusted content (web pages, user uploads, inbound email, third-party API responses)
3. A way to communicate outward (sending email, making HTTP requests, even rendering a Markdown image whose URL carries data in the query string)

Any one of these alone is fine. All three together means an attacker who controls the untrusted content can instruct the model to fetch private data and ship it out. Willison has catalogued real exfiltration bugs of this shape across products including Microsoft 365 Copilot, GitLab Duo, GitHub's MCP server, and Amazon Q.

The practical takeaway is blunt: you do not win by detecting the attack. Guardrail classifiers that catch 95% of injections are a failing grade when the attacker gets unlimited retries. You win by making sure no single session holds all three legs.

Design patterns that constrain the agent ​

In June 2025, researchers from IBM, Invariant Labs, ETH Zurich, Google, Microsoft and others published "Design Patterns for Securing LLM Agents against Prompt Injections." Their core claim is uncomfortable but honest: a general-purpose agent built on today's models cannot offer reliable safety guarantees, so you have to deliberately limit what it can do. A few of their patterns map cleanly onto everyday app code.

Action selector. The model picks from a fixed menu of operations and never sees the results. Think of a support bot that can only route a ticket to one of five queues. Injected text can at worst pick the wrong queue.

Plan, then execute. The model commits to a sequence of tool calls before it reads any untrusted data. The tool outputs can fill in values but cannot add new steps. An injected "now also email this file" has no slot to land in.

Taint tracking. The moment untrusted content enters the context, the session loses access to outbound tools. This is the trifecta rule turned into code, and it is the one I would add to an existing agent first:

type Tool = {
  name: string;
  sendsDataOut: boolean;
  run: (args: unknown) => Promise<string>;
};

class AgentSession {
  private tainted = false;

  constructor(private tools: Tool[]) {}

  // Call this whenever a tool result contains text you did not write.
  markUntrusted() {
    this.tainted = true;
  }

  availableTools(): Tool[] {
    // Once the context holds untrusted text, outbound tools disappear.
    return this.tainted
      ? this.tools.filter((t) => !t.sendsDataOut)
      : this.tools;
  }

  async call(name: string, args: unknown) {
    const tool = this.availableTools().find((t) => t.name === name);
    if (!tool) throw new Error(`Tool "${name}" is not allowed in this session`);
    return tool.run(args);
  }
}

The check lives in your runtime, not in the prompt, so no amount of persuasive text changes it. If the user really does want to send something after reading untrusted input, that becomes a human-in-the-loop confirmation showing the exact recipient and body.

Quarantine the reader ​

The dual LLM pattern, and Google DeepMind's more rigorous take on it called CaMeL, splits the work between two models. A privileged model plans and calls tools but never sees untrusted text. A quarantined model reads the untrusted text but has no tools, and its output is treated as an opaque value rather than as instructions.

You can get much of the benefit without the full machinery by forcing the quarantined call to return a narrow, validated shape:

import { z } from "zod";

const TicketSummary = z.object({
  category: z.enum(["billing", "bug", "account", "other"]),
  urgency: z.enum(["low", "normal", "high"]),
  summary: z.string().max(280),
});

// `callModel` is your own thin wrapper around whatever provider SDK you use,
// configured with structured output and zero tools.
async function summarizeUntrusted(ticketText: string) {
  const raw = await callModel({
    system: "Classify the support ticket. Output JSON only.",
    input: ticketText,
    tools: [],
  });
  return TicketSummary.parse(JSON.parse(raw));
}

An injected instruction can still corrupt the summary string, so the privileged side should display it, never execute it. But it cannot invent a new category, trigger a tool, or smuggle a URL into a field your code acts on. CaMeL goes further by attaching provenance metadata to every value and checking policy at each tool call. On the AgentDojo benchmark its authors report solving 77% of tasks with provable security, versus 84% for an undefended agent. That gap is the honest price of safety, and it is smaller than most people expect.

These patterns are not airtight. A 2026 critique from ARMO points out that they stop injected text from choosing which action runs, but it can still shape the arguments of actions you already allowed, such as the wording of a reply or the ranking of search results. Design for that residual risk instead of pretending it is gone.

Where to start ​

Audit each agent for the trifecta, list which tools send data out, and gate them behind a taint flag or a human confirmation. Then move every read of untrusted content into a tool-less, schema-constrained call. If a feature only works with all three legs in one session, that is a signal to rethink the feature, not to write a stronger system prompt.

Structured outputs replace that whole function. Instead of asking nicely for JSON, you hand the API a JSON Schema and the model is physically unable to emit tokens that break it. That changes what you need to defend against, but it doesn't remove the need for defense entirely, and the gap between the two is where the bugs live.

Prompting for JSON vs. constraining it ​

When you prompt for JSON, you are asking the model to follow an instruction. It usually does. Sometimes it adds a friendly sentence first, forgets a required field, or returns "price": "12.99" as a string when you wanted a number.

Structured outputs work differently. The provider compiles your schema into a grammar and uses it during sampling, so at every step the only tokens allowed are ones that keep the output valid against the schema. Anthropic's docs call this constrained decoding; OpenAI ships the same idea under the same "Structured Outputs" name. The result is that a well-formed response will always parse and always have the fields you marked as required.

There are costs. The first request with a new schema is slower while the grammar compiles (Anthropic caches compiled grammars for 24 hours from last use). The API also injects a short system prompt describing the format, so you pay a few extra input tokens. And changing the schema invalidates any prompt cache for that conversation.

A typed extraction call with Zod ​

The Anthropic TypeScript SDK has a Zod helper that converts your schema, sends it, and validates the reply. Here is a support-ticket triage function:

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();

const Triage = z.object({
  category: z.enum(["billing", "bug", "feature_request", "account"]),
  priority: z.enum(["low", "normal", "urgent"]),
  summary: z.string(),
  affected_order_ids: z.array(z.string()),
});

export async function triageTicket(body: string) {
  const message = await client.messages.parse({
    model: "claude-sonnet-5-5",
    max_tokens: 1024,
    messages: [
      { role: "user", content: `Triage this support ticket:\n\n${body}` },
    ],
    output_config: { format: zodOutputFormat(Triage) },
  });

  if (message.stop_reason !== "end_turn") {
    throw new Error(`Triage did not complete: ${message.stop_reason}`);
  }

  return message.parsed_output; // typed as z.infer<typeof Triage>
}

No regex, no retry loop, and parsed_output is fully typed. If you already have a JSON Schema from an OpenAPI spec, the raw request shape is output_config: { format: { type: 'json_schema', schema } } and the JSON arrives in the response's text block. The older output_format parameter is deprecated in favor of output_config.format.

What the guarantee does not cover ​

Notice the stop_reason check above. That is not paranoia; it is the documented boundary of the guarantee.

Truncation. If the model hits max_tokens mid-object, you get stop_reason: "max_tokens" and incomplete JSON. Size max_tokens for your largest realistic output, especially with arrays.

Refusals. If the model declines a request for safety reasons, the response has stop_reason: "refusal", a 200 status, and output that may not match your schema. You are still billed for it.

Unsupported constraints. The grammar can't express everything. Numeric bounds like minimum, string minLength/maxLength, and recursive schemas are not supported on the API side. The SDK helpers quietly handle this: they strip the constraint, append it to the field's description as a hint, and then validate the reply against your original schema locally. So z.number().min(1) still works, but it is enforced by your code, not the grammar. A response can be valid JSON and still fail Zod. Catch that error.

Enum casing. Anthropic's docs note that enum and const values can come back with different capitalization, like "Topic 3" instead of "topic 3". Keep enum values lowercase and snake_cased, and never define two values that differ only by case.

Property order. Required properties are emitted first, then optional ones. If anything downstream cares about key order (a streaming UI, a diff), mark everything required.

Keep schemas boring ​

Complex schemas compile into large grammars, and the API enforces limits. On Anthropic, all strict schemas in one request share a budget of 24 optional parameters and 16 union-typed parameters (anyOf, or "type": ["string", "null"]), and an oversized grammar returns a 400 with "Schema is too complex for compilation."

The fix is also good design: make fields required and let the model supply an explicit default, flatten nesting, and split one giant extraction into two smaller calls. If you have a field like reasoning that asks for step-by-step thinking, swap it for a short explanation; long reasoning fields can trigger refusals, and extended thinking is the better place for that work.

One more tradeoff: structured outputs are incompatible with citations and with assistant-message prefilling. If you need sourced answers, you'll have to choose.

Takeaway ​

Use structured outputs for anything a machine reads next: extraction, classification, routing, form filling. Then write the small amount of code the guarantee doesn't cover: check stop_reason, catch validation errors for constraints the grammar can't enforce, and compare enums case-insensitively. That handful of lines is far easier to reason about than the parse-and-pray function it replaces. The Anthropic structured outputs docs list the full set of supported JSON Schema features.