atelier/domains/messaging/queues.md

Queues — Derived Rules

Derives from domains/messaging/first-principles.md. Applies P1 (Messages are Contracts), P3 (Consumers are Idempotent), P4 (Delivery Semantics are Explicit), P5 (Dead-Letter Handling is Defined), and P6 (Backpressure is Bounded) primarily, with P2 (ordering), P7 (partitioning), and P10 (observable lag). For the at-least-once / at-most-once / exactly-once decision, see the comparison table below. Cross-links domains/concurrency/patterns for the in-process bounded-queue analog and domains/observability/metrics for consumer lag.

What a Queue Is (P1 Messages are Contracts)

• A queue is a point-to-point async delivery primitive. A producer enqueues a message; exactly one consumer dequeues and processes it. The message has an explicit, versioned schema (P1): the producer and consumer agree on shape before exchange, and the schema is the boundary — a schemaless message is a defect (the consumer breaks silently on the next shape change).
• The boundary is per D-062: messaging owns the cross-process / network-failure-model angle; concurrency owns the in-process analog. A queue is a messaging concern because producer and consumer are separate systems, the broker is the intermediary, and the failure model is network (the message can be lost, duplicated, reordered, or delayed by the broker or the network, not by a thread crash). The in-process bounded buffer (domains/concurrency/patterns Pattern 5) is the analog below this boundary — it fails by OOM; a broker-backed queue fails by partition, broker restart, or consumer crash-and-retry.
• See domains/messaging/pubsub.md for the fan-out (one-to-many) variant and domains/messaging/streams.md for the durable-log (replay-from-offset) variant. A queue deletes on ack; a stream retains for replay — the durability boundary is the distinguishing trait.

Producer / Consumer Model (P1, P3)

• The producer enqueues a message with an idempotency key (P3). The consumer dequeues, processes, and acks. If the consumer crashes before acking, the broker redelivers; the idempotency key makes the redelivery safe (the consumer dedups, not the broker).
• The idempotency key is per-message, not per-producer or per-session. A consumer that dedups by producer alone will drop distinct messages issued in the same window. Use a UUID per message, or a deterministic key derived from the message content (e.g., (entity, operation, version)).

# Producer/consumer pair with idempotency key (P1 contract, P3
# idempotency). The producer tags each message with a versioned
# schema and a unique idempotency key; the consumer dedups by the
# key so a redelivered message is processed once (P3).

# --- Producer ---
import json, uuid

def enqueue(order, broker):
    # P1: versioned schema. The message carries its schema version
    # so the consumer can route by shape (P9 evolution discipline).
    message = {
        "schema": "order.created.v1",
        "id": str(uuid.uuid4()),
        "idempotencyKey": f"order:{order['id']}:{order['version']}",
        "payload": order,
    }
    broker.send(queue="orders", body=json.dumps(message))
    # At-least-once by default (P4): the broker acks the send; the
    # consumer may see this message more than once under retry.

# --- Consumer ---
def consume(broker, dedup_store, process_order):
    for message in broker.receive(queue="orders"):
        # P3: idempotent consumer. Dedup by idempotency key before
        # processing; a redelivered message is a no-op, not a
        # double-apply.
        if dedup_store.seen(message["idempotencyKey"]):
            broker.ack(message)        # already processed; skip
            continue
        try:
            process_order(message["payload"])
            dedup_store.mark(message["idempotencyKey"])
            broker.ack(message)        # success; broker drops it
        except Exception:
            broker.nack(message)       # redeliver (at-least-once, P4)

• The dedup store is bounded (P6 — Backpressure is Bounded): a dedup store that grows without bound is a memory leak. Use a TTL window longer than the broker's max-redelivery window, or a bounded LRU. See domains/messaging/delivery-semantics.md for the full idempotency-key dedup-store pattern.

Ack / Nack (P4 Delivery Semantics are Explicit)

• Ack tells the broker the message was processed; the broker drops it. Nack (negative ack) tells the broker the processing failed; the broker redelivers (at-least-once) or routes to a DLQ (after the retry budget — P5).
• A consumer that neither acks nor nacks within the visibility timeout causes the broker to redeliver (the broker assumes the consumer died). This is the at-least-once default: the broker prefers duplication to loss.
• The semantic is explicit (P4): at-least-once is the default; the consumer must be idempotent (P3). At-most-once is fire-and-forget (no ack; the broker drops on send) — lossy but lowest latency. Exactly-once is at-least-once plus idempotency, or a transactional two-phase commit — see the comparison table below.

Visibility Timeouts and Redelivery (P4, P5)

• The visibility timeout is the window the broker hides a message after delivery, waiting for the ack. If the consumer does not ack within the window, the broker makes the message visible again and redelivers it (to the same consumer or another). This is the at-least-once mechanism: the broker assumes a no-ack-in-time consumer is dead.
• The timeout must be longer than the processing time, or the broker redelivers a message the consumer is still processing — causing duplicate processing (which P3 idempotency makes safe, but which wastes resources). A timeout shorter than processing time is a P6 (backpressure) smell: the consumer is too slow for the configured timeout.
• Redelivery has a budget (P5): after N redeliveries or a TTL, the message routes to the DLQ. An unbounded retry budget is the messaging-unbounded-retry chaos anti-pattern: the consumer never makes progress past the poison message.

FIFO vs Standard Queues (P2 Ordering is a Property, Not an Assumption)

• A standard queue delivers in arrival order per receive-node but offers no global ordering across shards, no per-message- group ordering, and may redeliver out of order under retry. It is the high-throughput default; ordering is not guaranteed (P2: the ordering property is "none" — explicitly documented).
• A FIFO queue delivers strict per-message-group order: all messages with the same group ID are delivered to one consumer in send order. The cost is throughput (FIFO queues cap at lower TPS) and latency (the broker must sequence per group). The ordering property is "per-group strict" — explicitly documented (P2).
• The choice is a P2 decision (which ordering guarantee) and a C8 decision (throughput cost). A consumer that assumes FIFO on a standard queue is a P2 violation: the broker does not provide the guarantee the consumer assumes. Document the property; do not assume it.

Prefetch and Concurrency (P6 Backpressure is Bounded)

• Prefetch (or max-unacked) bounds how many messages the broker delivers to one consumer without an ack. A prefetch of 1 is strict stop-and-wait (lowest throughput, tightest backpressure); a prefetch of N allows the consumer to process N in flight (higher throughput, more memory). An unbounded prefetch is a P6 violation: the broker floods the consumer's memory.
• Consumer concurrency is the number of parallel workers processing from the queue. More workers increase throughput up to the downstream's limit; beyond that, the workers stall the downstream (P6 — the backpressure propagates to the downstream, not the broker).
• The prefetch × concurrency product is the in-flight cap. Declare it (P6): an undeclared cap is a defect — the consumer either underutilizes the broker (prefetch too low) or OOMs under load (prefetch too high). This is the cross-process analog of domains/concurrency/patterns Pattern 5 (bounded queue with backpressure): concurrency owns the in-process analog; messaging owns the broker-backed instance.

Long Polling (P6, C8 Economy)

• Long polling (or ReceiveMessage with a wait-time-seconds) holds the receive request open until a message arrives or the wait expires. This reduces empty-receive round trips (C8 economy of the constrained link) and reduces latency-to-first- message (the message is delivered when it arrives, not on the next poll cycle).
• Long polling is the default for low-throughput queues: short polling burns CPU on empty receives; long polling waits for work. For high-throughput queues, the broker is usually full enough that long polling adds no latency; for low-throughput queues, long polling is the difference between 20ms and 20s latency-to-first-message.

Redelivery + DLQ Flow (P5 Dead-Letter Handling is Defined)

• A poison message (unparseable, repeatedly failing, or exhausting the retry budget) routes to the dead-letter queue. The DLQ is observable (P10 — DLQ depth is an alert) and drainable (an operator can inspect, replay, or discard with audit).
• The retry budget is bounded (P6): N redeliveries, or a TTL with exponential backoff. After the budget is exhausted, the message is moved to the DLQ, not retried forever. An unbounded retry is the messaging-unbounded-retry chaos anti-pattern (P5 breach).
• The DLQ routing rule is a redelivery-count or TTL threshold plus a target queue. See domains/messaging/delivery-semantics.md for the dead-letter strategy comparison table.

# Redelivery + DLQ flow (P5 dead-letter handling, P6 bounded
# retry budget). The consumer tracks redelivery count; after the
# budget, the message routes to the DLQ. The DLQ is observable
# (P10 — depth is an alert) and drainable.

MAX_REDELIVERIES = 5
DLQ = "orders-dlq"

def consume_with_dlq(broker, dedup_store, process_order):
    for message in broker.receive(queue="orders"):
        # P3 idempotency: a redelivered, already-processed message
        # is acked and skipped (not re-processed, not DLQ'd).
        if dedup_store.seen(message["idempotencyKey"]):
            broker.ack(message)
            continue
        try:
            process_order(message["payload"])
            dedup_store.mark(message["idempotencyKey"])
            broker.ack(message)
        except Exception as exc:
            # P5: bounded retry budget. After MAX_REDELIVERIES,
            # route to DLQ — do NOT retry forever.
            count = message.get("redeliveryCount", 0) + 1
            if count >= MAX_REDELIVERIES:
                broker.send(DLQ, body=json.dumps({
                    "original": message,
                    "reason": str(exc),
                    "deadLetteredAt": now_iso(),
                    "redeliveryCount": count,
                }))
                broker.ack(message)  # remove from the origin queue
                # P10: the DLQ depth must alert. A DLQ that grows
                # with no alert is a silent correctness defect.
            else:
                # Nack with backoff: the broker redelivers after a
                # delay. The backoff caps the retry rate (P6).
                broker.nack(message, delay=exponential_backoff(count))

• The deadLetteredAt and reason fields make the DLQ entry observable and auditable: an operator inspecting the DLQ sees why each message was dead-lettered and when. See domains/errors/patterns for the errors-as-data discipline the DLQ entry follows.

Delivery Semantics Comparison (D-069)

Semantic	Guarantee	Latency Cost	Implementation Cost	When It Fits
At-most-once	A message is delivered 0 or 1 times; loss is possible, duplication is not	Lowest (no ack; fire-and-forget)	Lowest (no ack, no dedup)	Telemetry where a dropped sample is acceptable; high-throughput metrics; MQTT QoS 0; logs where a lost line is tolerable. Never for billing, orders, or any side-effecting operation.
At-least-once	A message is delivered 1 or more times; duplication is possible, loss is not	Low (one ack round-trip)	Medium (consumer must be idempotent — P3; dedup store required)	The default for side-effecting operations: orders, payments, commands. The consumer dedups via idempotency keys (P3); the broker guarantees delivery. Fits the vast majority of broker-backed queues (SQS standard, RabbitMQ ack, MQTT QoS 1).
Exactly-once	A message is delivered exactly 1 time; no loss, no duplication	Highest (two-phase commit or transactional producer+consumer)	Highest (requires transactions, a transactional producer, and a transactional consumer — or at-least-once plus idempotency, which collapses to at-least-once with dedup)	Rare. Kafka transactions (consume-process-produce in one transaction); MQTT QoS 2 (four-step handshake). In practice, "exactly-once" is usually at-least-once plus idempotency (P3) — the broker does not guarantee it; the consumer enforces it. Jepsen analyses verify broker claims.

• The default for side-effecting operations is at-least-once with idempotent consumers (P3). At-most-once is for loss-tolerant telemetry. Exactly-once is reserved for the narrow case where the consume-process-produce loop must be transactional (Kafka transactions) — and even then, the consumer should be idempotent as defense-in-depth.
• The latency cost column is the C8 tradeoff: at-most-once is cheapest, exactly-once is most expensive. The implementation cost column is the C1/C3 tradeoff: at-most-once is simplest, exactly-once is most complex (and most fragile — a transactional consumer that partially fails is a bug source). The "when it fits" column is the P4 decision: declare the semantic per channel, do not let it emerge.
• This is the decision matrix required by D-069 for queues; see domains/messaging/delivery-semantics.md for the correctness properties of each semantic and the dead-letter strategy comparison.

Observability — Consumer Lag (P10 Messaging is Observable)

• Consumer lag (messages enqueued minus messages acked, or the age of the oldest unacked message) is the primary queue health signal. A lag that grows without bound is a P6 violation (the consumer is slower than the producer) and a P10 violation if it is not alerted.
• DLQ depth is the secondary signal: a DLQ that grows is a P5 signal (poison messages are accumulating) and a P10 signal if not alerted. Wire both to domains/observability/metrics as SLIs with SLOs (e.g., lag < 1000 messages, DLQ depth < 10).
• A queue with no lag metric is operating blind (P10 violation): the operator cannot see the consumer falling behind until the downstream effect surfaces.

Cross-Link to Concurrency (P6, cross-link concurrency/patterns)

• The broker-backed queue is the cross-process analog of the in-process bounded buffer. domains/concurrency/patterns Pattern 5 (Bounded Queue with Backpressure) owns the in-process instance ("producer is blocked or signaled" within one program); messaging owns the broker-backed instance above it (the producer is the broker's enqueue, the consumer is the broker's dequeue, the backpressure is the prefetch cap and the lag signal). The failure model differs: in-process fails by OOM; broker-backed fails by network partition, broker restart, or consumer crash-and-retry (D-062).
• The cross-link is one-directional outward (messaging → concurrency) per D-026 extended: messaging references concurrency as the in-process foundation; concurrency does not back-link to messaging.

What Violates Queue Discipline

Violation	Principle
Schemaless message (no versioned contract; consumer parses by guess)	P1 Messages are Contracts
Non-idempotent consumer under at-least-once delivery (redelivery doubles the effect)	P3 Consumers are Idempotent
Unstated delivery semantic (at-least-once vs exactly-once guessed)	P4 Delivery Semantics are Explicit
No DLQ (poison message retried forever or silently dropped)	P5 Dead-Letter Handling is Defined
Unbounded prefetch (broker floods consumer memory)	P6 Backpressure is Bounded
Unbounded retry budget (no cap; consumer never progresses past the poison)	P5, P6
Consumer that assumes FIFO on a standard queue (ordering not guaranteed)	P2 Ordering is a Property, Not an Assumption
Visibility timeout shorter than processing time (redeliver while still processing)	P4, P6
DLQ with no depth alert (poison messages accumulate invisibly)	P10, domains/observability/metrics
No consumer-lag metric (consumer falls behind invisibly)	P10, domains/observability/metrics
Dedup store that grows without bound (memory leak)	P6 Backpressure is Bounded
Default prefetch with no rationale (underutilizes or OOMs)	P6, domains/concurrency/patterns