atelier/domains/messaging/delivery-semantics.md

Delivery Semantics — Derived Rules

Derives from domains/messaging/first-principles.md. Applies P4 (Delivery Semantics are Explicit), P3 (Consumers are Idempotent), P2 (Ordering is a Property, Not an Assumption), P5 (Dead-Letter Handling is Defined), and P6 (Backpressure is Bounded) primarily, with P10 (DLQ depth as an alert). For the dead-letter strategy decision, see the comparison table below. Includes a fenced idempotency-key dedup-store example (IDEATE-39). Cross-links domains/concurrency/patterns for the in-process retry/backoff analog, domains/errors/patterns for errors as data for message failures, and domains/observability/metrics for DLQ depth as an alert.

The Three Delivery Semantics (P4 Delivery Semantics are Explicit)

• At-most-once: a message is delivered 0 or 1 times; loss is possible, duplication is not. The producer fires-and-forgets; the broker does not ack; the consumer does not dedup. Lowest latency, lowest implementation cost, lossy. Fits telemetry where a dropped sample is acceptable (MQTT QoS 0, fire-and-forget metrics).
• At-least-once: a message is delivered 1 or more times; duplication is possible, loss is not. The producer sends and waits for a broker ack; the consumer processes and acks; a crash before the consumer's ack triggers redelivery. The consumer must be idempotent (P3). The default for side-effecting operations (orders, payments, commands). The vast majority of broker-backed queues (SQS standard, RabbitMQ ack, MQTT QoS 1).
• Exactly-once: a message is delivered exactly 1 time; no loss, no duplication. In practice, this is at-least-once plus idempotency (P3), or a transactional consume-process-produce loop (P4 — see domains/messaging/streams.md). Jepsen analyses verify broker claims: "exactly-once" requires independent verification; the durable engineering practice is at-least-once with idempotent consumers (P3), which collapses to exactly-once under correct dedup.
• The semantic is declared per channel (P4), not emergent. An unstated semantic is a defect: the consumer guesses, and the guess is wrong under the first failure. See domains/messaging/queues.md for the three-semantics comparison table (semantics, latency cost, implementation cost, when each fits).

Idempotency (P3 Consumers are Idempotent)

• Idempotency is the correctness property that makes at-least-once safe. A consumer that processes the same message twice has the same effect as processing it once. The mechanism is the idempotency key: a per-message unique identifier the consumer uses to dedup redeliveries.
• The idempotency key is per-message, not per-producer or per-session. A consumer that dedups by producer alone drops distinct messages issued in the same window. Use a UUID per message, or a deterministic key derived from the message content (e.g., (entity, operation, version)).
• The dedup store is bounded (P6 — Backpressure is Bounded): a dedup store that grows without bound is a memory leak. Use a TTL window longer than the broker's max-redelivery window, or a bounded LRU. The TTL is the P6 bound: a key seen within the TTL is a redelivery; a key older than the TTL is expired (the broker has given up redelivering it).

Idempotency-Key Dedup Store (IDEATE-39, P3 — fenced example)

The dedup store is the concrete mechanism that makes a consumer idempotent under at-least-once delivery. This is the fenced example required by IDEATE-39 (parallel to the v0.3 IDEATE-29 signed-attestation fenced example): it is NOT prose-only — the consumer-with-dedup-store demonstrates P3 concretely.

# Idempotent consumer with a dedup store (P3 Consumers are
# Idempotent, P6 Backpressure is Bounded — the dedup store is
# TTL-bounded). The consumer dedups by idempotency key before
# processing; a redelivered message is a no-op, not a double-apply.

import time

# P6: the dedup store is bounded by a TTL window. The TTL must
# exceed the broker's max-redelivery window; beyond the TTL, the
# key is expired (the broker has given up). A dedup store with no
# TTL is a memory leak (P6 violation).
DEDUP_TTL_SECONDS = 24 * 3600   # longer than max-redelivery window

class DedupStore:
    """A TTL-bounded idempotency-key dedup store (P3, P6).

    seen(key): True if the key was processed within the TTL.
    mark(key): Record the key as processed (with a timestamp).
    """
    def __init__(self, backend):
        # backend is a Redis, a DB, or an in-process LRU. The
        # backend must be shared across consumer instances if the
        # subscription is shared (P2/P3 — see domains/messaging/
        # pubsub.md on shared vs independent subscriptions).
        self.backend = backend

    def seen(self, key: str) -> bool:
        ts = self.backend.get(key)
        if ts is None:
            return False
        if time.time() - ts > DEDUP_TTL_SECONDS:
            # P6: expired. The broker has given up redelivering;
            # this key is no longer a redelivery signal.
            self.backend.delete(key)
            return False
        return True

    def mark(self, key: str):
        self.backend.set(key, time.time(), ttl=DEDUP_TTL_SECONDS)


# The idempotent consumer: dedup before process, mark after
# process, ack after mark. A crash before mark re-processes (the
# dedup store does not have the key); a crash before ack
# redelivers (the broker did not see the ack) and the dedup store
# makes the redelivery a no-op (P3).
def consume_idempotent(broker, dedup: DedupStore, process):
    for message in broker.receive():
        # P3: dedup BEFORE process. A redelivered message is a
        # no-op, not a double-apply.
        if dedup.seen(message["idempotencyKey"]):
            broker.ack(message)        # already processed; skip
            continue
        try:
            process(message["payload"])
            # P3: mark AFTER process succeeds. A crash between
            # process and mark re-processes (acceptable: the
            # process must be idempotent OR the mark must be
            # transactional with the process — see below).
            dedup.mark(message["idempotencyKey"])
            broker.ack(message)
        except TransientError as exc:
            # P6: bounded retry with backoff. Nack for redelivery;
            # the broker redelivers after exponential backoff.
            broker.nack(message, delay=backoff(message["attempt"]))
        except PoisonError as exc:
            # P5: poison message. Route to DLQ, do NOT retry
            # forever (see the DLQ routing rule below).
            route_to_dlq(broker, message, exc)
            broker.ack(message)       # remove from the origin queue

• The order process → mark → ack gives at-least-once with idempotent dedup: a crash before mark re-processes (the dedup store does not have the key), and a crash before ack redelivers (the dedup store makes the redelivery a no-op). If process is not itself idempotent, the process → mark window must be transactional (e.g., process and mark in one DB transaction) — otherwise a crash in the window double-applies.
• For a non-idempotent process (e.g., a payment that must not double-charge), use a transactional dedup: process and mark in one DB transaction, so the mark commits iff the process commits. This is the "exactly-once via idempotency" pattern (P4): at-least-once delivery plus a transactional process-and-mark collapses to exactly-once under correct transactional semantics.

Ordering (P2 Ordering is a Property, Not an Assumption)

• The delivery semantic interacts with ordering (P2). At-least-once with per-partition ordering: a redelivery within a partition preserves order (the redelivered message re-appears in its original position relative to other messages the consumer has not yet seen). At-least-once with no ordering: a redelivery may appear out of order relative to messages delivered after it.
• The consumer must not assume an ordering property the broker does not provide (P2). A standard queue delivers per-receive-node arrival order but no global order and no order across redeliveries; a FIFO queue delivers strict per-group order including across redeliveries; a partitioned stream delivers strict per-partition order across redeliveries. Document the property; do not assume it. See domains/messaging/queues.md (FIFO vs standard) and domains/messaging/streams.md (per-partition order).

Dead-Letter Strategies (P5 Dead-Letter Handling is Defined)

• A poison message (unparseable, repeatedly failing, or exhausting the retry budget) must be routed to a dead-letter queue, not retried forever or silently dropped (P5). The DLQ is observable (P10 — depth is an alert) and drainable (an operator can inspect, replay, or discard with audit).
• The dead-letter strategy determines when a message is dead-lettered and what the operator sees. The choice is the decision matrix below (D-069).

Dead-Letter Strategy Comparison (D-069)

Strategy	When It Applies	Failure Visibility	Operational Cost
Retry-count-limit	A fixed max-redeliveries count (e.g., 5). After N redeliveries, route to DLQ. Simple, predictable.	The redelivery count is visible in the DLQ entry; the operator sees how many times it was retried.	Low — a counter per message; no backoff tuning. Risk: retries fire as fast as the broker redelivers, hammering a downstream that is already failing (P6 — no backoff = no backpressure escape).
TTL-with-backoff	A max time-to-live for redelivery (e.g., 30 minutes) with exponential backoff between retries. After the TTL, route to DLQ.	The TTL and the backoff schedule are visible; the operator sees the retry timeline.	Medium — backoff tuning per message type. Benefit: backoff gives the downstream time to recover (P6 — the retry rate is bounded); fits transient failures (a downstream that is briefly unavailable).
Poison-queue	A separate queue for messages that fail a specific check (unparseable, schema-invalid, unknown type) before any processing retry. Routed immediately, not retried.	The poison queue is a separate signal from the DLQ; the operator sees parse-vs-process failures distinctly.	Low — a routing rule per check. Benefit: distinguishes "never going to succeed" (poison) from "might succeed on retry" (DLQ). Use for unparseable messages that no retry will fix.
DLQ + alert	Any of the above strategies, plus an alert on DLQ depth. The DLQ is observable (P10) — depth, age, and rate are alerted.	Highest — the operator is paged on DLQ growth; the DLQ is a first-class signal, not a graveyard.	Medium — alerting setup per DLQ. This is the P5/P10 floor: a DLQ without an alert is a silent correctness defect (poison messages accumulate invisibly).

• The default for transient failures is **TTL-with-backoff + DLQ
  • alert**: backoff gives the downstream time to recover (P6), the TTL bounds the retry budget (P5), the DLQ captures the unprocessable, and the alert makes it visible (P10). The default for unparseable messages is poison-queue + alert: route immediately, do not retry a message no retry will fix.
• Retry-count-limit alone (no backoff, no alert) is the messaging-unbounded-retry chaos anti-pattern's cousin: it caps the count but hammers the downstream at full retry rate, and the DLQ grows silently if no alert is wired. Always pair a retry budget with backoff (P6) and an alert (P10).
• The failure-visibility column is the P10 check: every strategy must surface the failure to the operator. A strategy with no visibility is a P10 violation regardless of its retry semantics.
• The operational-cost column is the C8 tradeoff: more visibility and more backoff cost more to set up but pay back in operational stability. The P5/P10 floor is "DLQ + alert"; below that, the strategy is a silent defect waiting to grow.

DLQ Routing Rule (P5, P10)

# DLQ routing rule (P5 dead-letter handling, P6 bounded retry with
# backoff, P10 DLQ depth alert). Combines TTL-with-backoff for
# transient failures and poison-queue for unparseable messages,
# with an alert on DLQ depth.

import json, time

MAX_RETRY_TTL_SECONDS = 30 * 60    # 30 min total retry window
DLQ = "orders-dlq"
POISON = "orders-poison"           # unparseable; never retried

def route_to_dlq(broker, message, reason):
    """Route a message to the DLQ with audit metadata (P5)."""
    broker.send(DLQ, body=json.dumps({
        "original": message,
        "reason": str(reason),
        "deadLetteredAt": now_iso(),
        "redeliveryCount": message.get("attempt", 0),
    }))
    # P10: emit a metric so DLQ depth alerts fire. A DLQ that
    # grows with no alert is a silent correctness defect (P5/P10).
    metrics.increment("dlq.depth", tags={"queue": "orders"})
    # The ack removes the message from the origin queue; the DLQ
    # is the durable record (P5 — observable and drainable).

def consume_with_dlq(broker, dedup, process):
    for message in broker.receive():
        # P1: parse first. An unparseable message is poison —
        # route immediately, do NOT retry (no retry will fix it).
        try:
            payload = json.loads(message["body"])
        except (ValueError, SchemaError) as exc:
            broker.send(POISON, body=json.dumps({
                "original": message["body"],
                "reason": f"parse-failed: {exc}",
                "poisonedAt": now_iso(),
            }))
            metrics.increment("poison.depth", tags={"queue": "orders"})
            broker.ack(message)       # remove from origin; poison queue holds it
            continue

        # P3: dedup before process.
        if dedup.seen(payload["idempotencyKey"]):
            broker.ack(message); continue

        attempt = payload.get("attempt", 0)
        first_attempt_ts = payload.get("firstAttemptTs", time.time())

        try:
            process(payload)
            dedup.mark(payload["idempotencyKey"])
            broker.ack(message)
        except TransientError as exc:
            # P6: TTL-with-backoff. If the retry window is
            # exhausted, route to DLQ; otherwise redeliver with
            # exponential backoff.
            if time.time() - first_attempt_ts > MAX_RETRY_TTL_SECONDS:
                route_to_dlq(broker, message, exc)   # P5
                broker.ack(message)
            else:
                broker.nack(message, delay=backoff(attempt))
        except PermanentError as exc:
            # A permanent error (e.g., a not-found dependency)
            # does not benefit from retry — route to DLQ now.
            route_to_dlq(broker, message, exc)
            broker.ack(message)

• The routing rule distinguishes three failure modes: poison (unparseable — route immediately, no retry), transient (retry with backoff until the TTL, then DLQ), and permanent (a retry will not fix it — DLQ now). This distinction is the P5 discipline: not every failure is a retry; some are immediate DLQs.
• The DLQ entry carries reason, deadLetteredAt, and redeliveryCount — it is auditable (the operator knows why each message was dead-lettered and how many times it was retried). This is the errors-as-data discipline — see domains/errors/patterns for the general principle a DLQ entry instantiates.

Retry Budgets and Backoff (P6 Backpressure is Bounded)

• The retry budget is the cap on redelivery: a count, a TTL, or both. After the budget, the message routes to the DLQ (P5). An unbounded retry budget is the messaging-unbounded-retry chaos anti-pattern (P5 breach): the consumer never makes progress past the poison message.
• Exponential backoff spaces retries: 1s, 2s, 4s, 8s, ... with a jitter to avoid thundering-herd synchrony. Backoff gives the downstream time to recover (P6 — the retry rate is bounded, giving the downstream a chance to catch up). A retry with no backoff hammers the downstream at full rate, making the failure worse.
• The retry budget × the backoff schedule is the P6 bound: the consumer's retry load is bounded by design, not by luck. See domains/concurrency/patterns Pattern 6 (Timeout on Every Block) for the in-process retry/backoff analog; messaging owns the broker-backed instance where the redelivery comes from the broker across a network, not an in-process loop (D-062).

Poison Messages (P5, P1)

• A poison message is one no retry will fix: unparseable (the schema is wrong, P1), unknown type (the consumer does not handle this version, P9), or a permanent failure (a not-found dependency). Retrying a poison message wastes resources and blocks the queue (P6 — the consumer never makes progress).
• Poison messages route to the poison queue immediately (no retry), distinct from the DLQ (which holds messages that exhausted their retry budget on transient failures). The distinction is the P5 discipline: a poison queue is for "never going to succeed"; a DLQ is for "might have succeeded but didn't within the budget."
• A poison queue without an alert is the same defect as a DLQ without an alert (P10): the operator cannot see the poison accumulating. Wire both to domains/observability/metrics.

Cross-Link to Concurrency (P6, cross-link concurrency/patterns)

• The retry/backoff discipline here is the cross-process analog of domains/concurrency/patterns Pattern 6 (Timeout on Every Block) for in-process retry. Concurrency owns the in-process analog (a retry loop within one program, with a timeout per attempt); messaging owns the broker-backed instance (the broker redelivers across a network, the consumer applies backoff via nack-with-delay). The failure model differs: in-process retry fails by a thread crash or a timeout; broker-backed retry fails by network partition, broker restart, or consumer crash-and-retry (D-062).
• The cross-link is one-directional outward (messaging → concurrency) per D-026 extended: messaging references concurrency as the in-process foundation; concurrency does not back-link to messaging.

Cross-Link to Errors (P5, cross-link errors/patterns)

• A poison message is an errors-as-data instance: the failure is captured as a DLQ entry (with reason, redeliveryCount, deadLetteredAt), not swallowed. See domains/errors/patterns for the general errors-as-data discipline a DLQ entry instantiates. The DLQ is the async-messaging instance of an error log — observable, auditable, drainable.
• The cross-link is one-directional outward (messaging → errors): messaging references errors for the errors-as-data pattern; errors does not back-link to messaging.

What Violates Delivery-Semantics Discipline

Violation	Principle
Unstated delivery semantic (at-least-once vs exactly-once guessed)	P4 Delivery Semantics are Explicit
Non-idempotent consumer under at-least-once delivery	P3 Consumers are Idempotent
Dedup store with no TTL (memory leak; grows without bound)	P6 Backpressure is Bounded
Dedup key per-producer (distinct messages in the same window deduped)	P3 Consumers are Idempotent
Retry with no backoff (hammers the downstream at full rate)	P6 Backpressure is Bounded
Unbounded retry budget (consumer never progresses past the poison)	P5 Dead-Letter Handling is Defined
DLQ with no depth alert (poison messages accumulate invisibly)	P10, domains/observability/metrics
Poison message retried forever (no poison queue, no immediate DLQ)	P5 Dead-Letter Handling is Defined
Process-not-idempotent with non-transactional mark (crash in window double-applies)	P3, P4
DLQ entry with no reason/audit metadata (uninspectable failure)	P5, domains/errors/patterns
Ordering assumption the broker does not provide (FIFO assumed on standard queue)	P2 Ordering is a Property, Not an Assumption