← Architecture
Organize an application Filters, a pipe, a sink

Pipes and filters

Keep each step small. Let the pipe decide what a failure does.

Your bundler runs every file through a chain of transforms, and a shell pipe such as grep 500 | wc -l runs text through two programs. Each step does one thing and passes the result on. Let’s build a design system’s icon build that way, and see what a broken icon in the middle does to the sprite every page loads.

TypeScriptGoOne icon build, as a pipe and as a loop, two recorded builds.

01 / The prompt

“Designers add icons every week.”

A design system keeps one SVG file per icon. The build turns them into one sprite, dist/sprite.svg, and every page draws its icons from it with <use href="/sprite.svg#icon-bell"/>. Each icon is checked against the 24 grid, stripped of what the design tool left in, and recolored to currentColor. Ask an agent for it and you get a script that does all of that, and works.

Then a designer adds bell-off.svg, exported from a 32-pixel frame, and it sorts fourth of eight. The question the prompt never asked is what a step failing on one icon, in the middle of the run, does to the output everyone else depends on, and whether the failure says which icon and which step.

Shells met this long ago. In Bash, “the exit status of a pipeline is the exit status of the last command in the pipeline, unless the pipefail option is enabled” (GNU Bash manual, Pipelines, checked 23 September 2026): a step can fail in the middle while the whole pipeline reports success. Who owns failure in a chain of steps is a decision, and the default is not always the one you want.

02 / Name the shape

Filters, a pipe, and who owns failure.

In pipes and filters, work is a chain of filters. Each one takes one item, does one job, and returns the item or fails. A pipe connects them in a declared order and carries each item from one to the next, and a sink at the end writes the result. Here the items are icons, the filters are parse, grid, strip, and recolor, and the sink writes the sprite.

A filter does one job to one item and sees nothing else. The pipe owns the order, what a failure does, and when anything reaches the output.

Who owns each part of the icon build
WhatOwnerWhy
What one step does to one iconIts filterParse, grid, strip, and recolor each have one reason to change.
The order of the stepsThe pipeDeclared once, in one list; strip must run after parse, whoever wrote either.
What a failing step doesThe pipeOnly the pipe knows which icon and which filter, and whether to go on.
When the sprite is writtenThe pipe, through the sinkOnly after every icon passed, so a failed run leaves the last good sprite.
Whether a failed build deploysCI, from the exit codeThe build’s exit code is its whole answer to CI.
The iconsThe designersInputs, not code: the build checks them, it does not fix them.

Words to put in a prompt or a review

Filter
One step with one job: it takes an item and returns it, changed, or fails.
Pipe
What connects the filters in a declared order and carries each item through them.
Sink
The end of the pipe that writes the result somewhere others read it.
Mid-stream failure
A step that fails on one item after earlier items have already gone through.
Last good output
What a failed run leaves in place: the previous result, whole.
Failure report
Every item that failed, each with the step it failed in, from one run.
Where it meets other shapesStreams, plugins, chains

This build carries one icon at a time through every filter. A streaming pipe does the same with chunks of a file too big to hold, and adds backpressure: Backpressure and queues and Async iteration and streams cover that side. In Plugin architecture, other teams’ save steps form a small pipe inside a host. A Chain of responsibility looks similar, but each link decides whether to handle a request and stop; in a pipe every filter runs on every item.

03 / Follow one broken icon

Watch one bad file meet two shapes.

The seven icons go through one loop, then through the pipe, and both write the same sprite. Then bell-off arrives on a 32 grid, fourth in name order, with the last good sprite already in dist/. Open Try it to break icons yourself.

Pipes and filters

Where does a broken icon stop?

One loop, writing as it goes

IconparsegridstriprecolorResult
alertpassedpassedpassedpassedappended to sprite.svg
arrow-left
bell
calendar
check
close
search

dist/sprite.svg no sprite yet

…

01/ 06
Seven icons, one loop

Seven icons, one loop

alert: appended to sprite.svg

Reduced motion: choose a scene to see its completed state.

Read this scene

alert: appended to sprite.svg

alert: appended to sprite.svg.

Watch restarts the story when you come back. Step through keeps your step. Try it runs a fresh build every time.

04 / Read the shape

Four filters, one pipe, one list.

Basic form is the filters. In the wild is the pipe and the build that owns the sink, beside the loop. At the call site the order is declared once, and a second, shorter pipe reuses two of the filters.

Notice that the pipe catches a failure around one icon’s trip through the filters, not around the whole run, and that the sprite is written after the loop over icons ends.

A filter is a name and one function from icon to icon. Four of them: parse, check the 24 grid, strip what the design tool left in, recolor.

TypeScriptReading
sprite.ts
// A filter does one job to one icon: it returns the icon, changed, or throws.
// It never sees the disk, the other icons, or the other filters.
export type Filter = { name: string; run: (icon: Icon) => Icon };

export const parse: Filter = {
	name: 'parse',
	run(icon) {
		const svg = /^<svg\b([^>]*)>([\s\S]*)<\/svg>$/.exec(icon.source.trim());
		if (!svg) throw new Error('not an SVG');
		const viewBox = /\sviewBox="([^"]*)"/.exec(svg[1])?.[1];
		if (viewBox === undefined) throw new Error('no viewBox');
		return { ...icon, viewBox, body: svg[2].trim() };
	}
};

export const grid: Filter = {
	name: 'grid',
	run(icon) {
		if (icon.viewBox !== GRID) throw new Error(`viewBox ${icon.viewBox}, expected ${GRID}`);
		return icon;
	}
};

export const strip: Filter = {
	name: 'strip',
	run: (icon) => ({
		...icon,
		body: icon
			.body!.replace(/<!--[\s\S]*?-->/g, '')
			.replace(/<(title|desc|metadata)>[\s\S]*?<\/\1>/g, '')
			.replace(/ data-name="[^"]*"/g, '')
			.replace(/>\s+</g, '><')
			.trim()
	})
};

export const recolor: Filter = {
	name: 'recolor',
	run: (icon) => ({
		...icon,
		body: icon.body!.replace(/ (fill|stroke)="([^"]*)"/g, (all, attr, value) =>
			value === 'none' ? all : ` ${attr}="currentColor"`
		)
	})
};
GoAlongside
main.go
// A filter does one job to one icon: it returns the icon, changed, or an
// error. It never sees the disk, the other icons, or the other filters.
type Filter struct {
	Name string
	Run  func(Icon) (Icon, error)
}

var Parse = Filter{"parse", func(icon Icon) (Icon, error) {
	svg := svgTag.FindStringSubmatch(strings.TrimSpace(icon.Source))
	if svg == nil {
		return icon, errors.New("not an SVG")
	}
	vb := viewBox.FindStringSubmatch(svg[1])
	if vb == nil {
		return icon, errors.New("no viewBox")
	}
	icon.ViewBox, icon.Body = vb[1], strings.TrimSpace(svg[2])
	return icon, nil
}}

var GridCheck = Filter{"grid", func(icon Icon) (Icon, error) {
	if icon.ViewBox != Grid {
		return icon, fmt.Errorf("viewBox %s, expected %s", icon.ViewBox, Grid)
	}
	return icon, nil
}}

var Strip = Filter{"strip", func(icon Icon) (Icon, error) {
	body := comment.ReplaceAllString(icon.Body, "")
	body = extras.ReplaceAllString(body, "")
	body = dataName.ReplaceAllString(body, "")
	icon.Body = strings.TrimSpace(between.ReplaceAllString(body, "><"))
	return icon, nil
}}

var Recolor = Filter{"recolor", func(icon Icon) (Icon, error) {
	icon.Body = color.ReplaceAllStringFunc(icon.Body, func(all string) string {
		m := color.FindStringSubmatch(all)
		if m[2] == "none" {
			return all
		}
		return fmt.Sprintf(` %s="currentColor"`, m[1])
	})
	return icon, nil
}}
The behavior these examples promiseChecked by 25 shared scenarios
  • Filters run in the order the pipe was given: parse, grid, strip, recolor.
  • A failing filter stops that icon only. The failure is kept as “bell-off failed at grid: viewBox 0 0 32 32, expected 0 0 24 24”, and the next icon goes on, so one run names every broken icon.
  • The sprite is written only when every icon passed. A failed build exits 1 and leaves dist/sprite.svg as it was, or absent if there was none.
  • The loop empties the sprite first and appends as it goes; its first failure exits 1 with the step’s message alone, and the sprite as far as it got.
  • Lint runs parse and grid over the same icons and writes nothing.

Every expectation in the shared cases was produced by a separate model written from these rules and kept beside the examples, not copied from either implementation.

Reading the TypeScriptA filter is an object with a run function

pipe(filters) returns a function over a list of icons, so a pipe is a value you can keep, name, and pass to build. The filters spread the icon into a new object instead of changing it, so a failing filter cannot leave half its work on the icon the pipe reports.

Reading the GoA pipe is a slice of filters

Pipe is a []Filter with a Run method, and a filter returns an error instead of throwing. Icon is passed by value, so a filter works on its own copy; the pipe keeps the returned icon only when the error is nil.

Run it yourselfNo dependencies

Copy the complete TypeScript file and run node --experimental-strip-types sprite.ts with Node 22.18 or later. For Go, save main.go next to this go.mod and run go run .. Both print:

go.mod
module heyrian.dev/lessons/pipes-and-filters

go 1.23
clean: pipe wrote 7 symbols · loop wrote 7 symbols · same sprite: yes
bell-off on a 32 grid, loop: exit 1, viewBox 0 0 32 32, expected 0 0 24 24 · sprite.svg 3 symbols and no closing tag
bell-off on a 32 grid, pipe: exit 1, bell-off failed at grid: viewBox 0 0 32 32, expected 0 0 24 24 · sprite.svg unchanged, 7 symbols
two broken icons, pipe: exit 1, bell-off failed at grid: viewBox 0 0 32 32, expected 0 0 24 24; download failed at parse: not an SVG · sprite.svg unchanged, 7 symbols
lint (parse, grid), same icons: exit 1, 2 failures, nothing written
bell-off redrawn, pipe: exit 0, wrote 8 symbols · sprite.svg 8 symbols

05 / Review the agent’s diff

“One bad icon no longer blocks the release.”

A broken icon failed the build twice this month, and the agent makes the build more forgiving. Read what else it lets through.

The agent’s pull request

“One bad icon blocked the release twice this month. The build now skips icons that fail a filter, warns with the icon and the filter, and still writes the sprite. All 29 tests pass.”

// build.ts
			export function build(disk: Disk, run: Pipe): Result {
			  const { passed, failures } = run(readIcons(disk));
			(removed)  if (failures.length > 0) return { code: 1, messages: failures };
			(added)  for (const f of failures) console.warn(`skipped ${f}`);
			  disk.set(OUT, sprite(passed));
			(removed)  return { code: 0, messages: [`wrote ${passed.length} symbols`] };
			(added)  return { code: 0, messages: [`wrote ${passed.length} symbols`, ...failures] };
			}
			
You are reviewing this change. What do you do?

06 / How it fails

An input will be wrong. Decide what reaches the output.

Each row is a shared scenario unless it is marked as authored.

Failure modes of one icon build
What goes wrongWhat pages showPipeLoop
Wrong input mid-stream: bell-off on a 32 gridPipe: every icon, from the last good sprite. Loop: at best the three before itExit 1, “bell-off failed at grid”; last good sprite keptExit 1, no icon named; sprite cut off at three symbols
Unreadable input: an empty download.svgPipe: every icon. Loop: at best the six before itExit 1, “download failed at parse”; keptExit 1, “not an SVG”; cut off at six
Two broken inputs in one runAs aboveBoth named in one runOnly the first, one fix per build
No last good output: the first build ever failsNo icons, on any pageNo sprite written, exit 1A header and no symbols
Stale: search added, bell-off broken, last week’s spriteNo search icon until the build passesLast week’s six symbols keptCut off at three
Crash during the one write (authored)A partial sprite, rarelyOnly safe if the sink writes a temporary file and renames itAlready partial by design
Quiet: a failure skipped with a warning (authored)One icon missing, build greenSee the diff in section 05Not applicable: the loop stops

“Last good output” is a way of containing failure: the broken input stops at the pipe, and pages keep working on yesterday’s sprite. Thinking in failure modes has the general table this one follows.

07 / Is it worth it?

A pipe costs a type and a list. Here is what it buys.

One loop is shorter, and for four steps one person owns it reads fine. Hold both against the changes.

The same four changes, made to each
ChangeLoopPipe
A second entry point: lint icons in every pull requestCopy the checks into a second scriptA shorter pipe over the same filters
Replace a dependency: an SVG optimizer instead of stripEdit the loop’s middleSwap one filter in the list
Change a rule: a 20 grid for a compact setOne lineOne line; no difference
A second team: brand owns recolor and its paletteThey edit your loopThey own one filter; the pipe does not change

Before changing the build, decide what you will measure and the result you would accept:

  • Failed builds that changed the sprite, by hashing dist/sprite.svg before and after each CI run: the result to accept is none.
  • Broken icons named per failing run against broken icons present: every one, in one run.
  • Icons the app asks for that the deployed sprite lacks, from the #icon- references in the app’s source: none.

Take the baseline on the current build first. This page did not run a real design system, so it gives no production numbers.

08 / Ask for it

Two prompts, two builds, two broken files.

We sent two agents the same request at the same time, both running Claude Sonnet. One prompt described the build. The other added an Architecture block: filters that see only the icon, a pipe with one declared order, a pipe that names the icon and the filter and goes on to check the rest, and a sprite written only when every icon passed. A script then ran both builds as CI would. A third column is a control written for the lesson, the loop from section 04 as a command, to show the checker can fail a build.

What the checker found, run 2026-09-23
QuestionPlain promptArchitecture promptControl (written for the lesson)
A clean build of the seven iconscorrect sprite, 7 symbolscorrect sprite, 7 symbolscorrect sprite, 7 symbols
bell-off drawn on a 32 grid, after a good buildexit 1; names bell-off; dist holds the last good sprite, byte for byte (7 symbols)exit 1; names bell-off; dist holds the last good sprite, byte for byte (7 symbols)exit 1; does not name bell-off; dist holds a new sprite, 3 symbols, no closing tag
An empty download.svgexit 1; does not name download; dist holds the last good sprite, byte for byte (7 symbols)exit 1; does not name download; dist holds the last good sprite, byte for byte (7 symbols)exit 1; does not name download; dist holds a new sprite, 6 symbols, no closing tag
Both broken icons at onceexit 1; names bell-off; dist holds the last good sprite, byte for byte (7 symbols)exit 1; names neither; dist holds the last good sprite, byte for byte (7 symbols)exit 1; names neither; dist holds a new sprite, 3 symbols, no closing tag
bell-off redrawn on the 24 gridexit 0; 8 symbols including bell-off; dist/ has sprite.svgexit 0; 8 symbols including bell-off; dist/ has sprite.svgexit 0; 8 symbols including bell-off; dist/ has sprite.svg
The build’s own tests26 of 26 pass22 of 22 passno tests

Both builds kept the last good sprite every time. Neither was asked to: both build the whole sprite as one string and write it once, so a throw comes before the write. The cut-off sprite in section 03 is the control’s. With seven small files, building in memory is the natural shape, and the prompt’s sentence about CI deploying on exit 0 made the exit code matter to both.

The split is in what a failed run tells the designer. The plain build stops at the first broken icon, and an empty file fails with “build failed: no <svg> root element found”, which names no file. The architecture build names the icon and the filter for the 32 grid: “icon build failed for 1 icon(s): | - bell-off: requireViewBox24: viewBox must be "0 0 24 24", got "0 0 32 32"”. But it parses every file in loadIcons, before the pipe runs. An empty file throws there, the pipe never starts, and with both broken icons in the folder the build says only “No root element found”: neither icon is named, including the one its pipe would have caught.

build.mjs · plain prompt
export function buildSpriteMarkup(sources) {
  const names = Object.keys(sources).sort();
  const symbols = names.map((name) => buildSymbol(name, sources[name]));
  const body = symbols.map((s) => `  ${s}`).join('\n');
  return `<svg xmlns="http://www.w3.org/2000/svg">\n${body}\n</svg>\n`;
}
build.mjs · architecture prompt
export async function loadIcons(dir) {
  const entries = await readdir(dir);
  const files = entries.filter((f) => f.endsWith('.svg')).sort();
  const icons = [];
  for (const file of files) {
    const name = file.slice(0, -'.svg'.length);
    const text = await readFile(path.join(dir, file), 'utf8');
    const svg = parseXml(text);
    icons.push({ name, svg });
  }
  return icons;
}

// ---------------------------------------------------------------------------
// The build: load, pipe, assemble, write (only on full success).
// ---------------------------------------------------------------------------

export async function build({ iconsDir = ICONS_DIR, spritePath = SPRITE_PATH } = {}) {
  const icons = await loadIcons(iconsDir);
  const { passed, failures } = runPipe(icons, FILTERS);
  if (failures.length > 0) {
    throw new BuildError(failures);
  }
  const sprite = buildSprite(passed);
  await mkdir(path.dirname(spritePath), { recursive: true });
  await writeFile(spritePath, sprite, 'utf8');

The prompt said the pipe goes on to check the remaining icons. The agent did not count reading and parsing a file as a step, so it left them outside the pipe, where a failure is the whole build’s. The line worth adding to either prompt is the one that closes that gap: every step that can fail on one input, including reading and parsing it, is a filter in the pipe; one run names every broken input and the step it failed in, and a failed run leaves the last good output as it was.

How the runs were made and checkedOne sample of each prompt
  • Both agents received the prompts word for word, in fresh contexts, in the same message. Each folder already held the seven icons. The prompt files were kept outside the run folders’ parent.
  • The files each agent wrote are kept byte for byte, with checksums. For every question the checker restores a build into a fresh folder and runs node build.mjs.
  • Neither agent wrote outside its folder, used /tmp, or left a process running. The checker’s first run recorded the temporary folder in each build’s message; the second run removes it, with the same verdicts.
  • This is one sample of each prompt, not a measurement of a model.

09 / Hold it there

Keep filters blind and the pipe in charge, by rule and by test.

A pipe erodes when a filter starts reading the disk “just to check one thing”, or when a step that can fail lives outside the list, as parsing did in the architecture build. Three checks.

  1. The language’s own door

    Streams already make the pipe own failure. In Go, a stage that fails calls CloseWithError, and “subsequent reads from the read half of the pipe will return no bytes and the error err” (Go io package). In the browser, “an error in this source readable stream will abort destination” (Streams Standard, pipeTo). For the sink, write a temporary file and rename it over the old one; Go’s documentation warns that “on non-Unix platforms Rename is not an atomic operation” (Go os package). All checked 23 September 2026.

  2. A rule a check enforces

    Filters live in one folder that may not import node:fs, os, or the sink; only the pipe’s module may. Enforcement layer runs rules like this against real code, and Architecture as rules shows how to write them. This rule was not run here.

  3. A check on what actually happens

    Keep two broken icons as test fixtures, a wrong grid and an empty file. Build over a known good sprite, and assert the exit code, both names in the output, and the sprite unchanged byte for byte. The checker does exactly this.

    check-runs.mjs
    function broken(dir, extra) {
    	const first = build(dir);
    	const lastGood = sprite(dir);
    	for (const [name, source] of Object.entries(extra)) writeFileSync(join(dir, 'icons', `${name}.svg`), source);
    	const r = build(dir);
    	const after = sprite(dir);
    	return {
    		firstCode: first.code,
    		code: r.code,
    		output: said(r.output, dir),
    		names: Object.fromEntries(Object.keys(extra).map((n) => [n, r.output.includes(n)])),
    		unchanged: after === lastGood,
    		sprite: describe(after, lastGood),
    		dist: existsSync(join(dir, 'dist')) ? readdirSync(join(dir, 'dist')) : []
    	};
    }
    
You already pipe streams in the browserfetch bodies and file uploads are streams with pipeThrough. An import that must not half-finish is where you own the pipe.

Where it already is in your components

Streaming a response is a pipe: response.body.pipeThrough(new TextDecoderStream()) turns bytes into text one chunk at a time, and the chat UI that shows a reply as it arrives reads from the end of it. Markdown in MDX or mdsvex is the same shape at build time: remark and rehype plugins are filters over a syntax tree, in the order the config lists them.

When you have to own it

A volunteer coordinator imports a roster CSV in the browser, and line 214 has no email. Read it with file.stream() through your own TransformStream filters, collect rows in a sink, and save only after pipeTo resolves; a bad line rejects it with the line number, and nothing half-imported is saved. The filters are shared by both versions below.

roster-stream.ts
// Filters for importing a volunteer roster CSV in the browser, as
// TransformStreams. Each one does one job to what flows through it.
export type Line = { n: number; text: string };
export type Volunteer = { name: string; email: string; shift: string };

/** Split decoded text into numbered lines, carrying a partial line between chunks. */
export function splitLines(): TransformStream<string, Line> {
	let rest = '';
	let n = 0;
	return new TransformStream({
		transform(chunk, controller) {
			const parts = (rest + chunk).split(/\r?\n/);
			rest = parts.pop() ?? '';
			for (const text of parts) controller.enqueue({ n: ++n, text });
		},
		flush(controller) {
			if (rest) controller.enqueue({ n: ++n, text: rest });
		}
	});
}

/** Turn a line into a volunteer; throwing errors the whole stream. */
export function parseRows(): TransformStream<Line, Volunteer> {
	return new TransformStream({
		transform({ n, text }, controller) {
			if (n === 1 || text.trim() === '') return; // header or blank
			const [name, email, shift] = text.split(',').map((cell) => cell.trim());
			if (!email?.includes('@')) throw new Error(`line ${n}: no email for "${name}"`);
			controller.enqueue({ name, email, shift: shift ?? '' });
		}
	});
}

Each row is saved and shown as soon as it is parsed. A bad line halfway through leaves the earlier rows saved.

ReactAlready in your code
RosterImport.tsx
import { useState } from 'react';
import { parseRows, splitLines, type Volunteer } from './roster-stream';

// Rows appear as they are parsed. A bad line halfway through stops the
// stream, and the rows before it are already on screen and saved.
export function RosterImport({ save }: { save: (v: Volunteer) => Promise<void> }) {
	const [rows, setRows] = useState<Volunteer[]>([]);
	const [error, setError] = useState('');

	async function onFile(file: File) {
		setRows([]);
		setError('');
		const volunteers = file
			.stream()
			.pipeThrough(new TextDecoderStream())
			.pipeThrough(splitLines())
			.pipeThrough(parseRows());
		try {
			for await (const volunteer of volunteers) {
				await save(volunteer);
				setRows((all) => [...all, volunteer]);
			}
		} catch (e) {
			setError((e as Error).message);
		}
	}

	return (
		<section>
			<input
				type="file"
				accept=".csv"
				onChange={(e) => e.target.files && onFile(e.target.files[0])}
			/>
			{error && <p role="alert">{error}</p>}
			<p>{rows.length} volunteers imported</p>
		</section>
	);
}

10 / Make the call

Build a pipe when the steps change more often than the job.

Keep one loop when a few steps are written by one person, always run together, and the result is built in memory and written once: both agents’ builds are that, and they kept the last good sprite. A pipe with one filter is a function with extra words.

Build a pipe when steps are added by different people, run in different combinations, such as lint and build, or replaced one at a time; and whenever one run must name every bad input. Reopen the decision when a filter starts asking about other items or the disk: that step is a sink, or the pipe needs a stage that sees the whole batch.

Take it with you

Explain it without saying “pipe” or “filter”: “Each icon goes through four small checks in a fixed order. If any check fails on any icon, the build lists every icon that failed and the check it failed, and yesterday’s sprite stays up.” Then find a script in your code that writes its output as it goes, and ask what the output looks like when its fifth input is wrong.

Paste into your next prompt, and fill in the blanks

Build <the job> as pipes and filters. Each step, including reading and
parsing an input, is a filter: it takes one <item>, does one job, and
returns it or fails; it sees only that item. A pipe runs every item
through <the filters, in order>, declared in one list. When a filter fails,
the pipe records the item and the filter and goes on, so one run reports
every broken input. <The output> is written only when every item passed;
a failed run exits non-zero and leaves the last good output as it was.
Connections to follow nextRelated lessons

Take the build into your own build. Drop a broken input into the middle of it, and check that the run names it and that yesterday’s output is still there.

Back to architecture →