If you have built event-driven systems for any length of time, you have probably internalized the mantra of at-least-once delivery: keep retrying until the work is done, and design everything downstream to be idempotent. It’s good advice. But there’s a subtle failure mode that hides right underneath it - one where your system faithfully reports success, commits its progress, and quietly drops work on the floor. No exception, no alert, no dead letter. Just a message that was supposed to do something, and didn’t.

I ran into this recently while debugging why a small percentage of events were mysteriously never being processed. Everything looked healthy. The producer got a 2xx. The consumer committed its offset. The dashboards were green. And yet the work never happened. This post is about that gap - the space between “the system accepted my request” and “the system actually did the work” - and why it’s one of the more dangerous places for a distributed system to lose data.

Motivation

Most write-ups about message-processing reliability focus on the well-known hazards: duplicate delivery, out-of-order messages, poison pills, consumer lag. Those are real, and there’s plenty written about them. What I found much less discussed is the case where the acknowledgment itself is a lie - where a component reports success for an operation that has only been accepted, not completed, and a second component treats that acknowledgment as permission to throw the original message away.

This is a design smell that shows up across all sorts of stacks: a queue consumer that calls an async API, a workflow engine that enqueues a job, a service that hands off to a background worker. Any time you have a handoff across an asynchronous boundary, you have the potential for this gap. So I want to walk through the anatomy of the bug from first principles, then talk about how to close it properly.

Background: two kinds of “yes”

Before we get to the bug, let’s be precise about acknowledgments, because the whole problem lives in some sloppy vocabulary.

When a system replies to your request, it can mean one of two very different things:

• “I have accepted your request.” - I’ve durably recorded your intent, and I promise to try to do the work. Think HTTP 202 Accepted. The work hasn’t happened yet.
• “I have completed your request.” - The work is done, and its effects are durable. Think 200 OK with a result body.

These are worlds apart, and conflating them is the root of a lot of pain. The trouble is that many APIs return the same status code for “accepted” whether or not the work will eventually succeed. A 202 (or a 204 No Content, which is even more ambiguous) tells you the request was received. It tells you nothing about whether the work will run.

Now layer on the consumer side. A huge number of event-driven systems are built on brokers that use offset-based consumer groups - Apache Kafka being the canonical example. If you want a primer, I wrote an introduction to Apache Kafka a while back. The mental model is simple:

• Messages in a partition have monotonically increasing offsets.
• Your consumer reads a message, does some work, and then commits (or “marks”) the offset to say “I’m done with everything up to here.”
• If the consumer crashes before committing, the broker redelivers from the last committed offset. That’s what gives you at-least-once semantics.

💡 The offset commit is a promise about the past. When you commit offset N, you are telling the broker “every message up to and including N has been fully handled, and you never need to give them to me again.” If that statement isn’t actually true, you have manufactured data loss with your own hands.

Hold onto those two ideas - the ambiguous “yes” and the offset-as-promise - because the bug is what happens when they collide.

Anatomy of the pipeline

Let me describe a deliberately generic pipeline. Strip away the specific technologies and almost every async system looks like this:

  ┌──────────────┐      ┌───────────────────┐      ┌──────────────┐
  │ Event source │ ───► │  Message broker   │ ───► │   Consumer   │
  └──────────────┘      │  (offset-based)   │      └──────┬───────┘
                        └───────────────────┘             │ dispatch
                                  ▲                       ▼
                                  │             ┌────────────────────┐
                          commit  │             │  Async job API     │
                          offset  └─────────────│  (control plane)   │
                                                └─────────┬──────────┘
                                                          ▼
                                                ┌────────────────────┐
                                                │ Worker runs the job│
                                                └────────────────────┘

The consumer’s job is to read an event and translate it into an action by calling some downstream control plane - an async job API, a workflow trigger, a task queue. The control plane accepts the request and, at some later point, a worker actually executes it.

The consumer’s loop, in pseudocode, looked roughly like this:

for {
    msg := broker.Read()

    err := dispatchJob(msg)   // calls the async control plane
    if err != nil {
        retryWithBackoff(dispatchJob, msg)
    }

    // Always commit so we never get stuck reprocessing a bad message.
    broker.Commit(msg)
}

At a glance this looks reasonable, even defensive. There’s a retry with backoff. There’s a comment explaining that we always commit to avoid getting wedged on a poison message. Someone clearly thought about failure here.

And that is exactly what makes the bug so insidious.

The bug: the acknowledgment gap

Here’s the sequence that loses data.

1. The consumer reads a message and calls dispatchJob.
2. The control plane returns 204 No Content - “request accepted, a job has been created.” From the consumer’s point of view, this is success. err is nil.
3. Milliseconds later, and entirely outside the consumer’s view, the control plane rejects the job before it executes. Maybe a concurrency quota was exceeded. Maybe an admission controller said no. Maybe the queue was full. The job transitions straight to a terminal “rejected” state without a single line of business logic ever running.
4. Back in the consumer, dispatchJob returned nil, so the retry loop never fires - there was nothing to retry, as far as it knows.
5. The consumer commits the offset.

The offset moves forward. The broker will never redeliver that message. The job never ran. And nobody was told.

Consumer            Control plane            Worker
   │                     │                      │
   │  dispatchJob(msg)   │                      │
   │ ──────────────────► │                      │
   │  204 Accepted       │                      │
   │ ◄────────────────── │                      │
   │ (err == nil, looks  │                      │
   │  like success)      │  admission check     │
   │                     │  fails: quota/full   │
   │                     │  ✗ job rejected      │
   │                     │  (worker never runs) │
   │  commit offset      │                      │
   │ ───────────┐        │                      │
   │            ▼        │                      │
   │   message gone 💀   │                      │

This is the acknowledgment gap. The failure happened in the window between “accepted” and “executed”, and our success signal was wired to the wrong end of that window. We treated “a job was created” as if it meant “a job will run,” and those are not the same statement.

💡 The most dangerous bugs aren’t the ones that throw. They’re the ones that return nil. An exception is a gift - it’s the system telling you something is wrong. Silent, structurally-invisible loss gives you nothing to catch.

What made it worse is that the “always commit” decision - added defensively to avoid an infinite reprocessing loop - turned a recoverable failure into an unrecoverable one. The one safety mechanism that could have saved us (letting the broker redeliver) was disabled precisely when we needed it.

Why the usual instincts don’t save you

When engineers first see this, they reach for familiar fixes. Most of them don’t actually close the gap:

• “Just check the status code.” We did. It was 204. The status code describes the acceptance, not the outcome. The information we needed didn’t exist yet at the moment we got the response.
• “Add a retry.” There was one. It only triggers on a failed dispatch, not a failed execution. You can’t retry something you don’t know failed.
• “Make it idempotent.” Idempotency is necessary but not sufficient here. Idempotency protects you from doing the work twice; it does nothing to protect you from doing it zero times.
• “Use exactly-once semantics.” Setting aside the long debate about whether exactly-once is even a coherent goal across independent systems - the transactional guarantees of your broker do not extend into a third-party control plane you’re calling over HTTP. The moment you cross that boundary, you’re back to coordinating two independent systems with no shared transaction.

The real issue is architectural: we committed our durable progress based on a signal that didn’t actually confirm the work was durable. No amount of tuning the individual pieces fixes that. You have to move the acknowledgment.

Fixing it: verify before you ack

The core principle is a single sentence:

💡 Never acknowledge a message until you have confirmed the work it represents has actually started (or completed) - not merely been accepted.

Everything else is mechanics. Let’s walk through them, because the mechanics are where the interesting distributed-systems problems hide.

1. Close the loop: confirm execution, don’t assume it

Instead of trusting the 204, the consumer now verifies that the dispatched job reached a real running (or terminal-success) state before committing. In practice that means polling the control plane’s read API after dispatch:

started, err := dispatchAndVerify(msg)
if started {
    broker.Commit(msg)   // safe: the work is genuinely underway
} else {
    // do NOT commit - let redelivery give us another shot
    alert(msg, err)
}

dispatchAndVerify dispatches, then polls: did a job actually enter a non-rejected state? If it sees the tell-tale “rejected before executing” terminal state, or it can’t find the job at all within a bounded window, it treats that as a failure - which is the thing our original code could never see.

This is really just applying read-after-write thinking to a control plane. Don’t trust the write acknowledgment; go read the state back and confirm reality matches your intent.

2. The correlation problem

Here’s a genuinely tricky sub-problem that this exposes, and it’s a great example of why distributed systems are hard: the dispatch API often doesn’t tell you the ID of the thing it just created. You fire a request, you get back 204 No Content - literally no content - and now you need to find “the job I just created” among all the jobs.

You’re left correlating on secondary signals:

• A creation timestamp window (“a job created after time T”), which is racy under concurrency - two near-simultaneous dispatches can be ambiguous.
• A business key embedded into the job’s metadata at creation time, if the API lets you set something like a name or a label.

The robust fix is to make the work self-identifying: stamp a correlation key you already own (the entity ID, a request UUID) into the job at dispatch time, so that when you read the state back you can match on it exactly rather than guessing by time. If your control plane supports naming or tagging the work, use it. This is the async equivalent of propagating a trace ID, and it pays for itself the first time you have to debug a race.

💡 Any time you hand work across an async boundary, ask: “When this comes back, how will I know it’s mine?” If the answer is “by timestamp,” you have a race waiting to happen.

3. Bounded redelivery, or: don’t trade loss for a hot loop

The moment you say “don’t commit on failure so the broker redelivers,” someone will rightly point out the opposite failure mode: what if the work keeps failing? Now you’ve built an infinite reprocessing loop, and you’re hammering a control plane that’s already unhappy. This is the eternal tension:

• Commit too eagerly → you lose messages (the original bug).
• Never commit on failure → you can wedge the consumer forever.

The answer is bounded retries with escalation. Track how many times a given message has been through the wringer - keyed by its stable identity (partition + offset, or a business key) - and:

• Retry with exponential backoff while attempts remain, so a transient quota exhaustion gets a chance to clear.
• Once you’ve exhausted the budget, stop, escalate loudly, and then commit so a single doomed message can’t block the whole partition behind it.

That final commit is not “giving up silently” - it’s a deliberate, observable decision to route the message to a human (or a dead-letter queue) instead of blocking the stream. The difference between this and the original bug is everything: the original dropped work with zero signal; this drops it only after N visible, alarmed attempts.

4. Separate transient failures from terminal ones

A subtle but important refinement: when you poll to verify, a failure to read the state is not the same as the work being rejected. If your verification call itself hits a network blip or a 503, and you treat that as “the job failed,” you’ll re-dispatch and potentially create duplicate work - trading a lost-message bug for a double-processing bug.

So the verification loop needs to distinguish:

• “I confirmed the job was rejected” → terminal, re-dispatch is warranted.
• “I couldn’t reach the control plane to check” → transient, just retry the read, don’t re-dispatch.

Only definitive answers should drive irreversible decisions. Everything else is a retryable read. This is the same discipline as not making state transitions on ambiguous signals - you wait until you actually know.

5. Make the invisible visible

The reason this bug survived in production is that it was structurally unobservable. So the last piece is observability, and it’s not optional:

• Emit a metric/event whenever a dispatched unit of work fails to start.
• Alert when the retry budget is exhausted and a message is dropped.
• Log the correlation key, the attempt count, and a link to the rejected work.

If your system is going to make a hard decision like “I’m dropping this after five tries,” that decision must be the loudest thing in the room, not a silent commit. A good rule of thumb: every place your code can decide to discard work should be capable of paging a human. You may choose not to page - but the capability being there forces you to consciously design the failure path instead of falling into one.

Stepping back: the general principle

If you zoom out from the specific mechanics, this whole class of bug reduces to a few reusable principles that are worth carrying into any distributed system you build:

1. Distinguish “accepted” from “completed” - always. Treat them as different events with different names, different metrics, and different handling. Never let one masquerade as the other.
2. Anchor your durable acknowledgment to the durable outcome. Commit your offset (or delete your message, or mark your row done) based on confirmation of the effect you care about, not on a transport-level receipt.
3. A nil error is a claim, not a fact. Verify claims that cross trust boundaries, especially async ones. Read-after-write is cheap insurance.
4. Bound every retry, and escalate at the boundary. Unbounded retries and silent drops are two sides of the same coin; the cure for both is a visible, finite budget with a loud exit.
5. Only act irreversibly on unambiguous signals. Transient “I don’t know” should never trigger a decision that assumes “no.”

None of these are novel on their own. What’s interesting is how a single missing distinction - accepted vs. executed - cascades into silent data loss when it meets an offset commit. It’s a good reminder that in distributed systems, the bugs rarely live inside a component. They live in the seams between components, where two reasonable local decisions add up to one unreasonable global one.

Tradeoffs

Nothing here is free, and I’d be doing you a disservice to pretend otherwise.

What you gain

• No more silent loss - the failure mode that’s hardest to detect and most damaging to trust.
• A verifiable, observable processing pipeline where “done” actually means done.

What it costs

• Extra latency and load. Verifying execution means additional reads against the control plane per message. Poll intervals and budgets need tuning.
• More moving parts. Correlation keys, attempt tracking, and escalation paths are code you now own and test.
• You must embrace at-least-once for real. Verify-and-redeliver will occasionally produce duplicates (e.g., if a job actually started but your confirmation read failed). Idempotency downstream stops being optional - but that was always true; this just makes it honest.

For low-value, high-volume telemetry you might happily accept silent loss and skip all of this. For work where every single message must result in an action, the cost is obviously worth it. As always, the right answer depends on what the data is worth.

Conclusion

The most memorable bugs are the ones that teach you to distrust a word you’d been using carelessly. For me, that word was “success.” A 2xx is not success. A committed offset is not success. Success is the effect you actually wanted, confirmed to be durable. Everything else is just a system being polite.

If you take one thing away: go look at your event-driven pipelines and ask where you commit progress based on an acknowledgment rather than a confirmation. If those two things are wired together, you probably have an acknowledgment gap hiding in there too - quietly green on every dashboard, right up until someone asks where their data went.

Thanks for reading ✌️

References

1. HTTP 202 Accepted - MDN
2. Apache Kafka - Wikipedia
3. Exponential backoff - Wikipedia
4. Dead letter queue - Wikipedia
5. The Two Generals’ Problem - Wikipedia