Exactly-Once vs. At-Least-Once vs. At-Most-Once Delivery

Delivery guarantees describe how many times a message or event is promised to reach and be processed by a consumer in a distributed system. There are three:

• At-most-once : may lose messages
• At-least-once : may duplicate messages
• Exactly-once : neither loses nor duplicates

and understanding the tradeoffs between them is essential for anyone building streaming or event-driven pipelines.

Why It Matters

Every message sent between systems like

• an order event
• a payment notification
• a sensor reading

Travels across a network that can fail in the middle of delivery. When that happens, the sender genuinely doesn’t know whether the message got through. Does it resend and risk a duplicate, or does it not resend and risk losing the message entirely?

Consider a simple sequence:

Now imagine the consumer processes the message successfully, but the network fails before the consumer can acknowledge the message to the broker. From the broker’s perspective: “I don’t know whether the consumer processed it.“ The broker therefore has two choices:

1. Don’t resend: It’s possible message loss
2. Resend: Possible duplicate processing

This is the fundamental acknowledgement ambiguity in distributed systems.

This isn’t a theoretical edge case rather it’s a fundamental, unavoidable property of distributed systems (closely related to why the CAP Theorem exists at all).

Every messaging and streaming system has to pick a stance on this tradeoff, and that choice ripples into how you design every consumer downstream.

If you’re a team lead / architect and building anything event-driven, knowing which guarantee your message queue actually provides and more importantly, whether your consumers are built to handle it.

The Three Delivery Guarantees

At-Most-Once

The sender transmits a message a single time and moves on with no retry, no confirmation required. If the message is lost in transit, it’s simply gone.

Tradeoff: Fast and simple, with the lowest overhead. But data loss is a real, accepted risk and this is only appropriate when losing an occasional message genuinely doesn’t matter.

Example use case: Non-critical metrics or logging where losing an occasional data point has no real consequence, and the overhead of guaranteeing delivery isn’t worth it.

Code Sample:

def send_message(message):
    try:
        broker.send(message)
        print("Message sent")
    except Exception:
        # Do not retry
        print("Message lost")


message = {
    "event": "OrderCreated",
    "order_id": "ORD-1001"
}

send_message(message)

At-Least-Once

The sender keeps retrying until it receives confirmation the message was received. This guarantees the message eventually gets through but if the confirmation itself gets lost (even though the message was actually received), the sender retries anyway, and the consumer ends up seeing the same message more than once.

Tradeoff: No data loss, but duplicates are a real and expected possibility. This is the most common default in modern messaging systems (including Apache Kafka’s default configuration) because losing data is usually worse than occasionally processing a duplicate and provided the consumer knows how to handle duplicates.

This is exactly where Idempotency in Data Pipelines becomes essential – an idempotent consumer can safely receive the same message twice without it causing any harm, which is what makes at-least-once delivery practical to build on.

Code Sample:

Producer Side Code: The producer/broker retries when delivery is not confirmed

def send_message(message):
    max_retries = 3

    for attempt in range(max_retries):
        try:
            broker.send(message)

            # Assume broker confirms successful delivery
            print("Message delivered")
            return

        except Exception:
            print(f"Delivery failed. Retrying... {attempt + 1}")

    print("Message could not be delivered")

Consumer Side Code:

def process_message(message):
    print(f"Processing {message['order_id']}")

    database.save_order(
        order_id=message["order_id"]
    )

This consumer is not idempotent. If the same message arrives twice, it may create problems.

Now we improve the consumer. The consumer keeps track of processed event IDs.

def process_message(message):

    event_id = message["event_id"]

    # Check whether this event was already processed
    if database.exists(event_id):
        print("Duplicate message - skipping")
        return

    # Process the message
    database.save_order(
        order_id=message["order_id"]
    )

    # Record that this event was processed
    database.mark_processed(event_id)

    print("Message processed successfully")

The message might be delivered twice:

message = {
    "event_id": "EVT-10001",
    "event": "OrderCreated",
    "order_id": "ORD-1001"
}

send_message(message)

Exactly-Once

Each message is delivered and processed exactly one time with no loss, no duplication. This is the guarantee everyone wants, and it’s the hardest and most expensive to actually provide.

The honest truth is exactly-once delivery at the network level is essentially impossible to guarantee in a distributed system with unreliable networks and there’s a well-known result in distributed systems theory that no protocol can guarantee this over an unreliable channel. What real systems provide instead is effectively-once processing: at-least-once delivery, combined with idempotent handling on the consumer side, so that even if a message arrives twice, the outcome is as if it arrived exactly once.

Some systems like Kafka’s transactional producer/consumer APIs, or Flink’s exactly-once state checkpointing provide strong exactly-once semantics within their own ecosystem, but the moment data crosses into an external system that isn’t part of that transactional boundary, the guarantee reverts to “effectively-once via idempotency” again.

Why Do We Need All Three? Why Not Just Always Use Exactly-Once?

This is the natural question and if exactly-once is the “best” guarantee, why would anyone deliberately choose something weaker? Two honest reasons:

1. Stronger guarantees cost real performance and money. Guaranteeing exactly-once (or even at-least-once) requires the sender to track acknowledgments, retry on failure, and often persist state about what’s already been sent and all of which adds latency, infrastructure cost, and complexity compared to just firing a message and moving on. For a system logging millions of low-stakes metrics per second, paying that cost for every single message would be wasteful for no real benefit.

2. The tradeoff is fundamental, not a tooling limitation. This connects directly to the CAP Theorem and a classic result in distributed systems theory called the Two Generals’ Problem:

• Over a network where messages can be lost, it’s mathematically impossible for two parties to both be certain a message was received, no matter how many acknowledgments they exchange. You cannot engineer your way past this.
• You can only choose which failure mode you’re willing to accept: silence (at-most-once), duplication (at-least-once), or the cost of building idempotency on top of duplication to approximate exactly-once.

Most experienced teams don’t chase exactly-once everywhere by default rather they default to at-least-once plus idempotent consumers, and only reach for true transactional exactly-once semantics (which usually means staying inside one vendor’s ecosystem, like Kafka-to-Kafka) when the specific use case genuinely demands it.

Related Tools

• Kafka – defaults to at-least-once; supports exactly-once semantics (EOS) within Kafka-to-Kafka pipelines via transactional producers/consumers
• Amazon SQS Standard queues – at-least-once delivery; SQS FIFO queues provide exactly-once processing within the queue
• Apache Flink – supports exactly-once state checkpointing for stream processing
• Payment processors (Stripe, etc.) – expose idempotency keys specifically so API callers can safely retry without double-charging

Conceptual Comparison

Real-World Scenarios