Thinking in failure modes.

01 / The prompt

“Build the checkout for our gift cards.”

You ask an agent for the online checkout of a café chain’s gift cards. Ask a third-party fraud check whether to go ahead, charge the card through the card processor, create the gift card, show its code. What comes back works. You buy a $50 card, the fraud check answers in a fifth of a second, and the code appears. Every test passes.

Nothing in those first ten minutes tells you what the checkout does on the day the fraud check takes 30 seconds to answer. The default is not reassuring. Go’s HTTP client says it in one line: “A Timeout of zero means no timeout” (net/http, Client), and zero is what you get unless you set it. Node’s built-in fetch comes from undici, which waits up to 300 seconds for a reply’s headers before it gives up (headersTimeout, default 300e3). Both defaults were checked on 23 September 2026.

So the checkout waits as long as the fraud check does. Meanwhile the buyer is looking at a spinner, and the button next to it still says Pay.

The prompt never said how long to wait, what to do without a score, or what a second tap means. The agent had to pick something for each, and a demo cannot show you what it picked.

02 / Name the move

Five questions for every call you do not control.

A failure mode is one specific way a call can go wrong, together with what your system does when it happens. Thinking in failure modes is the habit of listing every call that crosses a boundary and asking the same five questions of each one before the code ships.

What if it is slow? Down? Wrong? Sent twice? Half done? Give every answer a policy, and a test that makes it happen.

The gift-card checkout makes three calls. Here is who runs the other side of each, and what the checkout cannot know about it.

03 / Follow one checkout

Watch the same bad day hit two checkouts.

First the hopeful build, on the day the fraud check takes 30 seconds. Then the inventoried build on the same day. Then the inventoried build again, when the processor charges the card and the reply never arrives. Step through at your own pace, or open Try it and break the calls yourself.

04 / Read it in code

A deadline, a shape check, a key, and a ledger.

Basic form answers three questions for one call: slow, down, wrong. In the wild answers the other two for the whole checkout: twice and half done. At the call site the deadline stops being a number and becomes something the network layer enforces.

Notice what the checkout does not do: it never guesses. No score is its own case with its own policy. No reply from the processor is “pending”, not “failed”, because a failure you report wrongly makes the buyer pay again.

export const FRAUD_DEADLINE_MS = 2000;

export type Verdict =
	{ kind: 'scored'; score: number } | { kind: 'timeout' | 'unavailable' | 'invalid' };

/** Ask the fraud check, but only for so long, and believe only a reply that makes sense. */
export async function scoreWithin(deps: Deps, amountCents: number): Promise<Verdict> {
	const reply = await deps.score(amountCents, FRAUD_DEADLINE_MS);
	if (reply.timedOut) return { kind: 'timeout' };
	if (reply.status !== 200) return { kind: 'unavailable' };
	const score = field(reply.body, 'score');
	if (typeof score !== 'number' || !Number.isInteger(score) || score < 0 || score > 100)
		return { kind: 'invalid' };
	return { kind: 'scored', score };
}

const FraudDeadline = 2 * time.Second

// Verdict is "scored", "timeout", "unavailable", or "invalid".
type Verdict struct {
	Kind  string
	Score int
}

// ScoreWithin asks the fraud check, but only for so long, and believes only a reply that makes sense.
func ScoreWithin(deps Deps, amountCents int) Verdict {
	reply := deps.Score(amountCents, FraudDeadline)
	if reply.TimedOut {
		return Verdict{Kind: "timeout"}
	}
	if reply.Status != http.StatusOK {
		return Verdict{Kind: "unavailable"}
	}
	var body struct {
		Score *float64 `json:"score"`
	}
	if json.Unmarshal([]byte(reply.Body), &body) != nil || body.Score == nil {
		return Verdict{Kind: "invalid"}
	}
	score := *body.Score
	if score != float64(int(score)) || score < 0 || score > 100 {
		return Verdict{Kind: "invalid"}
	}
	return Verdict{Kind: "scored", Score: int(score)}
}

module heyrian.dev/lessons/failure-modes

go 1.23

gc-1 $50, fraud answers: 200 issued GC-0001 after 500 ms | charges 1, cards 1
gc-2 $50, fraud slow: 202 review after 2000 ms, then 202 review (stored) after 0 ms | charges 0, cards 0
gc-3 $20, fraud slow: 200 issued GC-0001 after 2320 ms | charges 1, cards 1
gc-4 $50, charge reply lost: 202 pending after 480 ms, then 200 issued GC-0001 after 320 ms | charges 1, cards 1
gc-5 $50, stopped before the card: 502 stopped after 480 ms, then 200 issued GC-0001 after 20 ms | charges 1, cards 1

05 / Review the agent’s diff

“Checkout no longer hangs when a dependency is slow.”

The complaint was real, and adding timeouts is the right instinct. Read what else changed before you decide. The question to ask of each call is not “can this fail?” but “what happens if it is repeated?”

 // server/checkout.ts
			-(removed)const scored = await fraud.score(amountCents);
			+(added)const scored = await withRetry(() => fraud.score(amountCents, { timeoutMs: 2000 }), 3);
			 if (scored.score >= 70) return declined();
			-(removed)const charge = await processor.charge(amountCents);
			+(added)const charge = await withRetry(() => processor.charge(amountCents, { timeoutMs: 5000 }), 3);
			 return issueCard(checkoutId, charge.chargeId);
			+(added)
			+(added)async function withRetry<T>(call: () => Promise<T>, attempts: number) {
			+(added)  for (let i = 1; ; i++) {
			+(added)    try { return await call(); }
			+(added)    catch (error) { if (i === attempts) throw error; }
			+(added)  }
			+(added)}
			

06 / How it fails

The inventory: every call, every question, every row tested.

This is the artifact the procedure produces. Each row is a call and a condition, what the buyer sees, and what the inventoried checkout does. The last column comes from the shared cases, and the two-tabs row from a native test in each language, so each row is a test that fails if the policy changes.

Five rows share one idea: a side effect may be repeated only if the other side can recognize the repeat. The key does that at the processor, and the ledger does it in the checkout. The rest share another: no answer is its own answer. Decide in advance what it means, instead of letting a default decide.

When a row changes nothing, as the high-score row does here, keep it. It records that you asked. When a row’s test passes on the hopeful build too, check the fake before you celebrate: a fake that never loses a reply cannot tell you what a lost reply does.

Each of these has a lesson of its own: Timeouts, deadlines, and races, Idempotency and at-least-once, and Retry, backoff, and idempotency.

07 / Is it worth it?

You pay for a ledger and a review queue. Here is what they buy.

The hopeful build is shorter and just as fast on a good day. Hold both up against the changes every checkout eventually gets.

Before you change a checkout like this, write down what you will measure and what result you would accept, so the change is judged by more than a demo:

• Checkout time at the 99th percentile, and the slowest reply, from the server’s logs. The deadline should cap the worst case; accept nothing over 3 seconds.
• Charges per checkoutId, from the processor’s export. It should be exactly one. Any checkoutId with two is a duplicate charge, and the number before the change is your baseline.
• Orders held for review, and how many were later approved. This is the price of failing closed. If a person cannot clear them in a day, the limit is wrong.
• Pending charges older than five minutes. An outcome that stays unknown is a bug, not a state.

This page did not run the checkout for real buyers, so it has no numbers to give you. How to choose the right measure is Defining success; how to take the before picture is Baseline before you change.

08 / Ask for it

Two prompts, two builds, seven broken dependencies.

We sent two agents the same request for this checkout at the same time, both running Claude Sonnet. One prompt described the product and the two APIs. The other added one paragraph: list every call to another system; for each, decide what happens when it is slow, unreachable, unexpected, repeated, or half done; implement each decision; write the list to FAILURE-MODES.md. It named no deadline, no key, and no policy. Then a script started each build against a fake fraud check and a fake processor and broke them, one question at a time.

Both builds handled almost everything. Both refused to charge without a valid score, both answered a second tap with the first reply, and both sent the checkoutId as the processor’s idempotency key, so no question produced a second charge.

They could, because the plain prompt already carried those decisions in its product sentences. “The buyer’s app creates the checkoutId when the payment screen opens” tells a careful reader that a retry reuses the id. “It accepts an optional Idempotency-Key header” is the processor’s documentation, and both agents used it.

The one thing no sentence said was how long to wait. The plain build asks the fraud check with a bare fetch, and when the fraud check stopped answering, the checker gave up at 40 seconds with the build still waiting.

const res = await fetch(new URL(path, baseUrl), {
  method: "POST",
  headers: { "content-type": "application/json", ...headers },
  body: JSON.stringify(payload),
});

const maxAttempts = 2; // fraud check is stateless/side-effect-free, safe to retry once
const timeoutMs = 5000;

for (let attempt = 1; attempt <= maxAttempts; attempt++) {
  try {
    const res = await fetchWithTimeout(
      `${FRAUD_URL}/score`,
      {
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({ amountCents, email }),
      },
      timeoutMs,
    );

The architecture build did set a deadline: 5 seconds, then one retry. The buyer waited 10.3 seconds. That is the second finding: a deadline per attempt is not a deadline per checkout. Read from its source, a bad day for both vendors could hold a buyer for about 35 seconds: two 5-second fraud attempts, then three 8-second charge attempts, with pauses between.

So the prompt line the runs showed was missing is not about the five questions. The agent asked those well. It is the number the product owns: the buyer hears back within n seconds, however many attempts that allows.

09 / Hold it there

Make the missing deadline a failing check.

An inventory written once drifts the moment someone adds a fourth call. Three kinds of check keep the answers in place.

import { useState } from 'react';

// fetch has three ways to end, and only one of them throws:
// a reply that is ok, a reply that is not ok (a 500 still resolves), and no reply at all.
type Sent =
	{ kind: 'reply'; status: number; body: unknown } | { kind: 'no-reply'; timedOut: boolean };

export async function send(url: string, body: unknown, timeoutMs: number): Promise<Sent> {
	try {
		const response = await fetch(url, {
			method: 'POST',
			headers: { 'content-type': 'application/json' },
			body: JSON.stringify(body),
			// Without a signal, fetch waits as long as the connection stays open.
			signal: AbortSignal.timeout(timeoutMs)
		});
		return {
			kind: 'reply',
			status: response.status,
			body: await response.json().catch(() => null)
		};
	} catch (error) {
		const timedOut = error instanceof DOMException && error.name === 'TimeoutError';
		return { kind: 'no-reply', timedOut };
	}
}

export default function SaveNote({ text }: { text: string }) {
	const [message, setMessage] = useState('');

	async function save() {
		const sent = await send('/api/notes', { text }, 5000);
		if (sent.kind === 'no-reply')
			setMessage(sent.timedOut ? 'Still waiting after 5 s. Try again.' : 'Offline. Try again.');
		else if (sent.status >= 400) setMessage(`Not saved (${sent.status}).`);
		else setMessage('Saved.');
	}

	return (
		<p>
			<button onClick={save}>Save</button> <span role="status">{message}</span>
		</p>
	);
}

10 / Make the call

Ask all five questions everywhere. Pay for policies where a row can hurt.

The inventory is cheap: a table and a fake per call. The policies are what cost something: a ledger, a key, a review queue, a pending state the screen has to show. Build them where a row can lose money, data, or trust. Keep the hopeful version where a failure costs a person one more click and a repeat does no harm: a read-only admin page, a report you run by hand, a preview.

Reopen the inventory when a call starts charging, sending, or writing somewhere; when a new caller arrives with retries of its own, such as a kiosk or a mobile app; and when a vendor changes its API. Those are the moments a row changes.

Take it with you

Explain it without saying “failure mode”: “For each thing my code calls, I asked what happens if it is slow, down, wrong, sent twice, or half done. I decided each answer, and I wrote a test that makes each one happen.” Then open the last feature you built with an AI, list the calls it makes to other systems, and find the deadline on each one, or the place where there is none.

Before writing code, list every call this feature makes to another system.
For each call, decide what happens when it is:
- slow: stop waiting after <deadline>; then <policy>. The whole request
  answers within <budget>, however many attempts that allows.
- unreachable: <policy>.
- wrong: accept only <valid reply>; treat anything else as <policy>.
- sent twice: <side effect> happens once per <key>; a repeat returns the stored result.
- half done: record each step before the call it guards, so a retry resumes
  from <step> instead of starting over.
Write the list to FAILURE-MODES.md, and add a test for each row that makes
that failure happen with a fake.