Found a safeguard that had been written, reviewed, and never actually switched on
A safeguard that prevents duplicate entries in the production log had been built and shipped months ago, and had never once run. The code for it existed, everything referencing it worked, and nothing anywhere reported a problem. It had simply never been applied to the live database, and its absence looked identical to its presence from the outside. It surfaced during hardware testing when a single, uncontested action produced a duplicate that nothing in the app could account for. That was the tell: contention can explain two things colliding, but not one thing misbehaving on its own. The safeguard was deployed, confirmed live by reading it back rather than assuming the update took, and the failing scenario was re-run and passed. The wider lesson has been written into the project's standing practice. An update that replaces existing behaviour, rather than adding something new, fails silently forever when it is skipped — nothing is missing, so nothing complains. Additions announce themselves when they are absent; replacements do not. Verification now means reading back what is actually running, never checking that it exists.
Share This Ship
🔧 Shipped: Found a safeguard that had been written, reviewed, and never actually switched on 🧵
A safeguard that prevents duplicate entries in the production log had been built and shipped months ago, and had never once run. The code for it existed, everything referencing it worked, and nothing anywhere reported a problem.
It had simply never been applied to the live database, and its absence looked identical to its presence from the outside. It surfaced during hardware testing when a single, uncontested action produced a duplicate that nothing in the app could account for.
That was the tell: contention can explain two things colliding, but not one thing misbehaving on its own. The safeguard was deployed, confirmed live by reading it back rather than assuming the update took, and the failing scenario was re-run and passed.
The wider lesson has been written into the project's standing practice. An update that replaces existing behaviour, rather than adding something new, fails silently forever when it is skipped — nothing is missing, so nothing complains.
Additions announce themselves when they are absent; replacements do not. Verification now means reading back what is actually running, never checking that it exists.
Follow the build → build-log-five.vercel.app #reliability #data-integrity #infrastructure #buildinpublic