Production Readiness Checklist

Every item below came out of something that broke, on a system serving real traffic. None of it is generic advice, and each one links to the incident that produced it.

Use it as a review sheet before a launch, a migration or an on-call handover. Skip the sections that do not describe your architecture: a checklist that applies to everything applies to nothing.

Message handling and acknowledgment

If a queue, broker or event bus sits anywhere in the path, these decide whether work can disappear without anyone being told.

• The acknowledgment happens after the work, not after the receive. Anything that acks on delivery has converted a crash into silent data loss.
• The consumer confirms execution rather than assuming it. A downstream call that returned 200 is not the same as a downstream that committed.
• Redelivery is bounded. Unbounded retry turns a loss bug into a hot loop, which is worse and louder.
• Transient and terminal failures are handled differently. Retrying a malformed payload forever is not resilience.
• Dropped work is observable. If the only symptom is a number that should have gone up and did not, you will find out from a customer.

From The Acknowledgment Gap, where every component reported success while messages were being lost.

Request capacity and dependencies

For anything with a bounded pool of workers, threads or connections serving requests.

• Every outbound call has a deadline, and the deadline is smaller than the caller’s. A call with no timeout is a worker you have lent out indefinitely.
• Retries have a budget, not just a count. Three retries per request across three layers is twenty-seven calls.
• Worker occupancy is a graph you can look at. Latency and error rate will not tell you that the pool is full.
• Readiness does not depend on every downstream. A probe that calls everything fails the instance when anything is merely slow, which converts a partial outage into a total one.
• You know what happens at saturation. Shedding load is a decision; queueing until the deadline is also a decision, made by default.

From When “no healthy upstream” isn’t about the upstream you think, where retries turned dependency latency into worker exhaustion.

Replacing a live data path

For migrations where an old and a new system must both be correct for a while.

• The migration transfers authority, not just data. Name the moment the new system starts being the one that decides.
• The shadow path compares decisions, not payloads. Two systems can disagree on every field and agree on every outcome, or the reverse.
• The target is zero unexplained differences, not zero differences. Explained differences are progress; unexplained ones are the whole risk.
• Coverage is measured across scenarios, not request volume. A million requests through one code path proves one code path.
• Rollback does not require a deploy. If reverting means shipping, you will not revert at 3am.

From Replacing a Critical Data Path Without a Flag Day.

Rollout and schema coordination

For anything running more than one replica, which is most things.

• One-shot work runs once. A Deployment starts N copies of a long-running process; a migration is not that shape.
• Coordination is not compatibility. A lock stops two migrations colliding. It does nothing about old pods serving traffic against a new schema.
• Schema changes are expand-and-contract: add compatible, deploy both-aware code, backfill, switch, remove.
• Waiting is logged as waiting. Expected contention that surfaces as a crash loop will be debugged as an outage.
• A failed migration stops the rollout, rather than letting the application start against a schema it does not understand.

From When Three Kubernetes Pods Try to Run One Database Migration.

How to use this

Work down it with someone who disagrees with you. The value is not in the ticks; it is in the three or four items where the honest answer is “I do not actually know”, because those are the ones that will page you.

If a section prompts a question I have not answered here, reply to any issue of the newsletter and I will take a look. Most of these posts started as someone asking exactly that.