← Concepts & practices
Concept Concurrency, scheduling, and delivery

Idempotency & at-least-once delivery

Twice should count once.

You already press Send twice when a page hangs, and wonder whether the first one went through. Every system that retries has that question on the other side. Let’s follow a support-ticket webhook from a handler that creates a ticket for every delivery to one that recognizes a delivery it has already handled, even after a crash.

TypeScriptGo One ticket handler, two implementations.

01 / The idea

Creating a ticket for every delivery is a fair start.

Your help desk receives webhooks. Each one describes a customer’s problem, and the handler turns it into a support ticket. While every delivery is a new request, one delivery, one ticket is exactly right.

Read the first handlerTypeScript · the version this lesson starts from
handler.ts
// The first handler: every delivery creates a ticket. Right while each delivery is a new request.
export class TicketHandler {
	private db: Database = { tickets: [], receipts: [] };
	handle(event: Webhook, fault: Fault = 'none'): Outcome {
		if (fault === 'before-commit') return { kind: 'crashed', ticketId: null };
		const ticketId = this.db.tickets.length + 1;
		this.db.tickets.push({ id: ticketId, tenant: event.tenant, subject: event.subject });
		// The ticket is saved. If the worker dies now, the sender never hears that it worked.
		if (fault === 'after-commit') return { kind: 'crashed', ticketId: null };
		return { kind: 'created', ticketId };
	}
	snapshot(): Database {
		return copyDatabase(this.db);
	}
}

Go’s version is the same handler. Both languages meet again at sameEvent in section 02.

Then a worker saves ticket 1 and crashes before it acknowledges the webhook. The sender never hears back, so it delivers evt-42 again. The next worker does exactly what the handler says and creates ticket 2. The customer now has two tickets for one problem.

At-least-once delivery means a message can arrive more than once. An operation is idempotent when doing it again has no further effect. Put them together and the receiver has to recognize a repeat: give each operation an identity, and record that it was handled in the same commit as the work. Stripe’s webhook documentation says it plainly: “Webhook endpoints might occasionally receive the same event more than once,” and suggests logging the event IDs you’ve processed.

The same question sits behind every Send button on a slow connection. Section 05 builds a form that can be sent twice, or reloaded mid-request, and still creates one ticket.

02 / See the shape

Name the delivery, and keep the receipt with the ticket.

The basic form is the identity: which deliveries count as the same event, and what a repeat gets back. In the wild puts the receipt check, the ticket, and the receipt in one commit. At the call site loses an acknowledgment and delivers again.

Both languages run the same handlers against the same 9 shared schedules, with no database or network.

The identity. A delivery is the same event when tenant, source, and event ID match; a known key returns its ticket, or a conflict if the subject changed.

TypeScriptReading
handler.ts
// A delivery's identity: the same event, from the same source, for the same tenant.
export function sameEvent(a: Webhook, b: Webhook): boolean {
	return a.tenant === b.tenant && a.source === b.source && a.eventId === b.eventId;
}

// Seen before: hand back the original ticket, unless the same key now carries a different request.
export function replay(receipt: Receipt, event: Webhook): Outcome {
	if (receipt.subject !== event.subject) return { kind: 'conflict', ticketId: null };
	return { kind: 'replayed', ticketId: receipt.ticketId };
}
GoAlongside
handler.go
// SameEvent is a delivery's identity: the same event, from the same source, for the same tenant.
func SameEvent(a, b Webhook) bool {
	return a.Tenant == b.Tenant && a.Source == b.Source && a.EventID == b.EventID
}

// Replay hands back the original ticket, unless the same key now carries a different request.
func Replay(receipt Receipt, event Webhook) Outcome {
	if receipt.Event.Subject != event.Subject {
		return Outcome{"conflict", 0}
	}
	return Outcome{"replayed", receipt.TicketID}
}
Reading the TypeScriptA staged copy stands in for a transaction

handle copies the database, writes the ticket and the receipt to the copy, and swaps it in with one assignment. That’s a model of a transaction. In a real database, the receipt check, both inserts, and the commit happen inside one transaction, with a unique constraint on the key.

sameEvent compares three fields instead of joining them into one string, so tenant a:b with source c can’t match tenant a with source b:c.

Reading the GoValues in, copies out

copyDatabase copies both slices, so a Snapshot can’t be used to change the store. Receipt embeds the whole Webhook, which is what lets Replay compare the subject.

The handlers expect one caller at a time. Real concurrent workers rely on the database’s transaction, not on Go.

03 / Follow the deliveries

Watch two handlers receive the same deliveries.

Five steps, every row from sending the same deliveries to both handlers you just read. A crash after saving means the ticket is in the database but the sender never heard back. Before each step, guess how many tickets each side ends with.

In Try it, choose the events and crash the workers yourself.

Idempotency

Two handlers, the same deliveries.

One delivery, one ticket

Step 1 of 1

Create every time

Tickets

  • #1Cannot sign in

Receipts

  • Keeps no receipts

Receipt in the same commit

Tickets

  • #1Cannot sign in

Receipts

  • acme/helpdesk/evt-42→ #1

One delivery, one ticket. delivery 1: creating every time created #1, with receipts created #1. Creating every time: 1 ticket, 0 receipts. With receipts: 1 ticket, 1 receipt. While every delivery is a new request, creating a ticket each time is right. Both handlers agree.

01/ 05
Deliver one event

One delivery, one ticket.

evt-42 arrives once, and both handlers save ticket 1.

Reduced motion: choose a scene to see its completed state.

Read this scene

evt-42 arrives once, and both handlers save ticket 1.

One delivery, one ticket. delivery 1: creating every time created #1, with receipts created #1. Creating every time: 1 ticket, 0 receipts. With receipts: 1 ticket, 1 receipt. While every delivery is a new request, creating a ticket each time is right. Both handlers agree.

Watch restarts when you return. Step through keeps your selected step. Try it starts with empty databases each time you open it.

What a receipt buys you

Now put names on what you just watched. These are the words you’ll hear in a design review, and each one points at something on this page.

Safe to deliver again
After a lost acknowledgment, evt-42 comes back and gets ticket 1, not ticket 2.
Retrying stays simple
The sender can deliver as often as it needs to without guessing whether the first try worked.
The same answer again
A repeat returns the original ticket number, so the caller learns what it created the first time.
Mismatches get caught
The same key with a different subject is refused, not answered with an unrelated ticket.
All or nothing
A crash before the commit leaves neither the ticket nor its receipt, so the next delivery isn’t wrongly skipped.

The review words are at-least-once delivery, idempotent, idempotency key for the identity, and exactly-once effect, which is what receipts give you, as opposed to exactly-once delivery. Section 08 covers what they cost.

04 / Try a decision

A check that runs before the commit.

Someone writes the receipt version as three steps: look for a receipt, save the ticket, record the receipt. Each step commits on its own, and the receipt key is unique. It passes every test that delivers one event at a time. The code is in separate.ts, and the lesson’s tests pin what two overlapping workers leave behind.

Two workers pick up evt-42 at the same moment. What’s in the database afterwards?

Each worker checks for a receipt, finds none, saves a ticket, then records the receipt. Each write commits on its own, and receipt keys are unique.

05 / Give it a real job

The receiver decides what counts as the same request.

In the real help desk, the webhook handler runs inside a database transaction. The receipts table has a unique key on tenant, source, and event ID, and the ticket insert shares the transaction. The same rule applies wherever a request can arrive twice, including from your own frontend.

Sender

Keeps the key stable

Every retry of the same event carries the same ID.

Receiver

Commits the receipt with the work

One transaction, and a unique key on the identity.

The caller’s screen

Doesn’t guess

No answer means unknown, not failed.

Malcolm Featonby’s Amazon Builders’ Library article makes the commit point explicit: recording the token together with the changes it protects “must meet the properties for an atomic, consistent, isolated, and durable (ACID) operation.”

The example leaves out receipt expiry, authentication, and effects outside the database. An email sent from inside the handler isn’t rolled back with the ticket, so it needs its own identity.

Build UIs?Every form that posts on a slow connection can be sent twice, and one day a lost response makes you decide what Try again means.

Where it already is in your components

Disabling the Send button stops a double click, not a retry after a timeout or a reload. The textbook panes add a key instead: one crypto.randomUUID() per ticket, sent as an Idempotency-Key header and reused if the user presses Send again. randomUUID has been available across browsers since March 2022, in secure contexts.

The header is described in an IETF draft, which has since expired. It recommends a UUID, and says a server should answer 409 while a request with that key is still being processed, and 422 if the key comes back with a different body.

When you have to own it

Now it’s a support form on a train. The request goes out and nothing comes back. The ticket may exist, so the form shouldn’t say it failed, and it shouldn’t let the user edit and send a different request under the same key.

It keeps the key and the body in sessionStorage, which MDN says “survives over page reloads and restores” but is cleared when the tab closes, and locks the fields until it gets an answer. Try again sends the same key and body. Writing a new request is the user’s deliberate choice, and gets a new key.

submission.ts
// One key per request the user means to send. An attempt whose outcome we never learned keeps its
// key and its body, even across a reload of this tab, so trying again can't create a second one.
export type Pending<T> = { key: string; body: T };
export type KeyStore = Pick<Storage, 'getItem' | 'setItem' | 'removeItem'>;

export interface Submissions<T> {
	pending(): Pending<T> | null;
	begin(body: T): Pending<T>;
	finish(): void;
}

export function submissions<T>(
	storage: KeyStore,
	form: string,
	newKey: () => string = () => crypto.randomUUID()
): Submissions<T> {
	const name = `pending-submission:${form}`;

	function pending(): Pending<T> | null {
		const saved = storage.getItem(name);
		return saved ? (JSON.parse(saved) as Pending<T>) : null;
	}

	return {
		pending,
		// Starts a new submission, or continues the unfinished one with its original key and body.
		begin(body) {
			const unfinished = pending();
			if (unfinished) return unfinished;
			const started = { key: newKey(), body };
			storage.setItem(name, JSON.stringify(started));
			return started;
		},
		// The server gave a definite answer: the next submission is a new request.
		finish() {
			storage.removeItem(name);
		}
	};
}

A ticket form that makes one idempotency key per ticket, sends it as a header, reuses it when Send is pressed again, and makes a new one after success or a definite refusal.

ReactAlready in your code
NewTicketForm.tsx
import { useState, type FormEvent } from 'react';

export function NewTicketForm() {
	const [subject, setSubject] = useState('');
	// One key for this ticket, created once and reused if the user presses Send again.
	const [key, setKey] = useState(() => crypto.randomUUID());
	const [status, setStatus] = useState('');

	async function send(event: FormEvent) {
		event.preventDefault();
		setStatus('Sending…');
		try {
			const response = await fetch('/api/tickets', {
				method: 'POST',
				headers: { 'Content-Type': 'application/json', 'Idempotency-Key': key },
				body: JSON.stringify({ subject })
			});
			// A definite no, like a missing subject: pressing Send again would get the same answer.
			if (response.status >= 400 && response.status < 500 && response.status !== 409) {
				setStatus(`Not created: the server refused it (HTTP ${response.status}).`);
				setKey(crypto.randomUUID()); // A corrected ticket is a new request.
				return;
			}
			// 409 (still in progress) and 5xx leave the outcome open, like no answer at all.
			if (!response.ok) throw new Error(`HTTP ${response.status}`);
			const { id } = (await response.json()) as { id: number };
			setStatus(`Ticket #${id} created.`);
			setSubject('');
			setKey(crypto.randomUUID()); // The next ticket is a new request.
		} catch {
			setStatus('Couldn’t confirm it was sent. Press Send to try again.');
		}
	}

	return (
		<form onSubmit={send}>
			<label>
				Subject
				<input value={subject} onChange={(event) => setSubject(event.target.value)} required />
			</label>
			<button type="submit">Send</button>
			<p role="status">{status}</p>
		</form>
	);
}

06 / Recognize it elsewhere

Anywhere the same request can arrive twice.

You’ve met all of these. For each one, find the identity and what a repeat does.

Familiar operations, their identity, and what a repeat does
Where you’ve seen itThe identityWhat a repeat does
Stripe webhooksThe event IDYour endpoint logs processed IDs and skips repeats.
Stripe’s APIThe Idempotency-Key headerReturns the first request’s saved result.
HTTP PUT and DELETEThe URLIdempotent by definition: several identical requests have the same intended effect as one.
A queue with redeliveryThe message IDDelivered again after a missed acknowledgment; the consumer has to recognize it.
A form’s Send buttonA key made for that submissionThe server returns the ticket it already created.

An operation that’s naturally idempotent, like setting a status to closed, needs no receipt. Creating something is where you need one.

07 / Already in your toolbox

Your platforms already publish this contract.

Three places to look. For each one, find what’s remembered and for how long.

Stripe · Idempotent requests

Keys up to 255 characters, the first result saved for a key whether it succeeded or failed, parameters compared on reuse, and keys removed after at least 24 hours. A complete contract to compare yours against.

Read the reference ↗

Amazon Builders’ Library · Making retries safe with idempotent APIs

Malcolm Featonby on caller-provided request identifiers, and why recording one has to be atomic with the change it protects.

Read the article ↗

IETF · The Idempotency-Key HTTP header field

The draft behind the header: UUID keys, 409 while a request is in flight, and 422 for a key reused with a different body. It has expired, so treat it as a clear description rather than a standard.

Read the draft ↗
A useful counterexample: closing a ticketWhen the operation is already idempotent

Setting a ticket’s status to closed has the same effect however many times it arrives, so it needs no receipt. Adding a comment, sending an email, or creating a ticket does.

Watch the order, though. A late “close” arriving after a “reopen” is a different problem, and versions solve it, not receipts.

08 / The parts to watch

A receipt only helps if it’s checked, kept, and trusted.

These are the places it still goes wrong.

The check has to be inside the commit

Checking for a receipt, then saving, then recording it works one delivery at a time and fails for two at once. That’s section 04’s bug.

A new key per retry defeats it

Make the key once per request the user means, not once per attempt. A fresh key on every retry makes every retry look new.

Receipts don’t last forever

Stripe removes keys after at least 24 hours. A delivery that arrives after its receipt is gone looks new again, so keep receipts longer than the sender keeps retrying.

Know what the server remembers

Stripe saves failed results too, “including 500 errors”, so retrying the same key returns the same failure. A server that only remembers successes behaves differently. What Try again should do depends on which one you’re talking to.

Same text isn’t the same request

Finding repeats by hashing the body would merge two customers reporting the same problem. Identity comes from the event, not its content.

Effects outside the transaction

An email sent by the handler isn’t rolled back with the ticket. Give it its own key, or send it after the commit from a saved record of what to send.

09 / Make the call

What would you have to change tomorrow?

Give both handlers a plausible change and follow the work it creates.

How a change affects a handler that creates every time and one that keeps receipts
The changeCreate every timeReceipt in the same commit
Every delivery really is a new requestCorrect, and simpler.Also correct, with a table that never stops a repeat.
A worker crashes after savingTicket 2 for the same problem.Returns ticket 1.
A worker crashes before savingTicket 1 on the next delivery.Ticket 1 on the next delivery.
Two workers get the same event at onceTwo tickets.One ticket. In a database, the second worker’s insert hits the unique key and rolls back; it has to read the receipt to return ticket 1.
The sender reuses an event ID with a different subjectA second ticket.Refused as a conflict.

Reach for an identity and a receipt whenever the same request can arrive twice, and doing it twice would matter. Webhooks, queues, and a Send button on a slow connection.

Keep the plain handler when repeats can’t happen, or when the operation is already idempotent, like setting a status.

The question I’d leave beside the code is: if this exact request arrived again right now, what should happen?

10 / Take the idea with you

Explain the handler without saying “idempotent.”

“Every event has an ID. In the same transaction that saves a ticket, we save a receipt for that ID, and we check for one first. If it’s there, we send back the ticket we already made.” In a review, the words are at-least-once delivery, idempotency key, exactly-once effect, and conflict.

Before moving on, jot down why the lost acknowledgment made ticket 2, why checking first still let two workers make two tickets, and one request in your own code that could arrive twice.

Connections to follow nextRelated lessons
  • Retry, backoff & idempotency is the sender’s side: when to deliver again, how often, and when to stop.
  • Race conditions in UI lets reads ignore a late answer. Writes like these can’t be ignored, so they need a key instead.
  • Backpressure & queues refuses work when it’s full. A refused request is only safe to send again if a repeat is harmless.
  • Discriminated unions model the form’s editing, sending, unsure, created, and refused states as one value.

Take the handler into your editor. Give receipts an expiry time, deliver evt-42 again after it passes, and decide what should happen.

Back to Concepts & practices →