Notes on buildingXVIII
Metrics that change behaviour
The test for a metric is not whether it’s accurate. It’s whether any plausible value of it would cause you to do something different.
Run that test across a typical dashboard and most of it fails. Total users, cumulative revenue, page views — these can only go up, they resolve nothing, and their function is to be shown to someone rather than to inform a choice.
Leading, and therefore uncomfortable
Lagging indicators tell you what happened. Revenue, churn, retention at 90 days. They’re trustworthy and they’re late — by the time churn moves, the cause is a quarter behind you.
Leading indicators are noisier and earlier. Time to first successful action. Share of sessions that hit an error. Whether a new account did the core thing twice in week one. You can act on these while acting is still cheap, and you’ll sometimes act on noise.
That trade is worth taking, and most teams don’t, because a leading indicator can make you wrong in public and a lagging one can only make you late.
Instrument before you need it
The measurement you want always concerns a period that has already passed. There is no retrofit. Adding events costs an hour before launch and is impossible after, so the discipline is to instrument the boring paths early — not the feature you’re excited about, the one you assume works.
The asymmetry is specific. Anything held as state can be reconstructed later — how many accounts existed, what they were worth, when they cancelled. Anything that happened between states cannot. Where someone hesitated, which step they repeated, what they tried before the thing that worked, how long the first success took: none of it exists unless something was emitting at the time. Which is why the instrumentation that matters is always the boring path nobody expected to be interesting.
Where it breaks
Every metric becomes a target, and every target gets gamed — including by sincere people with no intention of gaming anything.
Measure response time and answers get shorter. Measure engagement and the product gets stickier in ways nobody would defend out loud. The team isn’t cheating; they’re optimising what you told them mattered, and the parts of the job that weren’t measured quietly stop being done. Choosing a metric is choosing what to neglect, and that half is rarely stated.
The engagement case deserves singling out. “Design for sustained attention, not maximal engagement” and “instrument everything” pull in opposite directions, because engagement is the easiest thing to measure and the easiest to grow. A number that goes up while the product gets worse is not a vanity metric — vanity metrics are merely useless. This kind is actively hostile, and it looks like success for several quarters.
Two guards, both cheap. Pair every metric with the one it would degrade — speed with quality, engagement with weekly-active-and-satisfied, conversion with refund rate. And review whether a metric still deserves the dashboard, because the ones that stay longest are the ones nobody argues about, which usually means nobody uses them.
Takeaway: metrics should inform action, not justify past choices.