Failure is data
“Fail fast” is repeated by people who have not made failing cheap, which is the part that requires actual work.
A failure is only data if you can see what happened, absorb it without losing something that matters, and change something as a result. Without those three, failing fast is just failing at speed.
Cheap requires machinery
Visibility first. A failure nobody noticed teaches nothing, and most systems fail silently for a while before they fail loudly. Instrumentation is what converts an incident into information.
Then containment. The failure has to be survivable — the feature flag, the canary, the backup you have actually restored from rather than merely configured. Every one of these exists to lower the cost of being wrong, which is what makes being wrong affordable often.
Then the loop. Post-mortems that assume systemic causes, because “someone was careless” ends the investigation exactly where it should start. The useful question isn’t who merged it. It’s what made it possible to merge, and what would have caught it.
Backups are the honest test of all this. Nearly every team has them; a much smaller number have ever restored from one. Until you have, you don’t have backups, you have files — and the failure that teaches you this costs whatever the gap between those two things is worth. The mechanism that follows is unglamorous and specific: a scheduled restore into a scratch environment, on a calendar, run by someone who didn’t set it up.
Where it breaks
Not all failure is instructive, and treating it as though it were is how teams normalise being bad at things.
Some failures are just cost. The outage caused by skipping a review you knew you should do teaches nothing you didn’t already know; it converts a known obligation into an expensive reminder. Filing it as “learning” launders it. There’s a real difference between a failed experiment — designed, bounded, informative whichever way it goes — and a preventable mistake dressed up in the same language afterwards.
The blameless framing has a matching failure. Taken absolutely, it removes the possibility of anyone being accountable for anything, and people notice. The version that works separates the two: no blame for the failure, ownership of the fix. The person closest to it drives the change. That’s not punishment, and teams read the difference correctly.
And speed has a floor. In systems where failure reaches someone real — payments, health, a voice agent talking to a customer at 2am — the correct rate of failure is low, and getting there means slower, more boring engineering than the slogan suggests. “Fail fast” is advice for the cheap tier. Knowing which tier you’re in is the whole judgement.
Takeaway: fail fast, learn faster, ship forward.