atelier/domains/messaging/queues.md

# Queues — Derived Rules

> Derives from `domains/messaging/first-principles.md`. Applies P1
> (Messages are Contracts), P3 (Consumers are Idempotent), P4
> (Delivery Semantics are Explicit), P5 (Dead-Letter Handling is
> Defined), and P6 (Backpressure is Bounded) primarily, with P2
> (ordering), P7 (partitioning), and P10 (observable lag). For the
> at-least-once / at-most-once / exactly-once decision, see the
> comparison table below. Cross-links `domains/concurrency/patterns`
> for the in-process bounded-queue analog and
> `domains/observability/metrics` for consumer lag.

## What a Queue Is (P1 Messages are Contracts)

- A queue is a point-to-point async delivery primitive. A producer
  enqueues a message; exactly one consumer dequeues and processes
  it. The message has an explicit, versioned schema (P1): the
  producer and consumer agree on shape before exchange, and the
  schema is the boundary — a schemaless message is a defect (the
  consumer breaks silently on the next shape change).
- The boundary is per D-062: messaging owns the cross-process /
  network-failure-model angle; concurrency owns the in-process
  analog. A queue is a messaging concern because producer and
  consumer are separate systems, the broker is the intermediary,
  and the failure model is network (the message can be lost,
  duplicated, reordered, or delayed by the broker or the network,
  not by a thread crash). The in-process bounded buffer
  (`domains/concurrency/patterns` Pattern 5) is the analog below
  this boundary — it fails by OOM; a broker-backed queue fails by
  partition, broker restart, or consumer crash-and-retry.
- See `domains/messaging/pubsub.md` for the fan-out (one-to-many)
  variant and `domains/messaging/streams.md` for the durable-log
  (replay-from-offset) variant. A queue deletes on ack; a stream
  retains for replay — the durability boundary is the
  distinguishing trait.

## Producer / Consumer Model (P1, P3)

- The producer enqueues a message with an idempotency key (P3).
  The consumer dequeues, processes, and acks. If the consumer
  crashes before acking, the broker redelivers; the idempotency
  key makes the redelivery safe (the consumer dedups, not the
  broker).
- The idempotency key is per-message, not per-producer or
  per-session. A consumer that dedups by producer alone will drop
  distinct messages issued in the same window. Use a UUID per
  message, or a deterministic key derived from the message content
  (e.g., `(entity, operation, version)`).

```python
# Producer/consumer pair with idempotency key (P1 contract, P3
# idempotency). The producer tags each message with a versioned
# schema and a unique idempotency key; the consumer dedups by the
# key so a redelivered message is processed once (P3).

# --- Producer ---
import json, uuid

def enqueue(order, broker):
    # P1: versioned schema. The message carries its schema version
    # so the consumer can route by shape (P9 evolution discipline).
    message = {
        "schema": "order.created.v1",
        "id": str(uuid.uuid4()),
        "idempotencyKey": f"order:{order['id']}:{order['version']}",
        "payload": order,
    }
    broker.send(queue="orders", body=json.dumps(message))
    # At-least-once by default (P4): the broker acks the send; the
    # consumer may see this message more than once under retry.

# --- Consumer ---
def consume(broker, dedup_store, process_order):
    for message in broker.receive(queue="orders"):
        # P3: idempotent consumer. Dedup by idempotency key before
        # processing; a redelivered message is a no-op, not a
        # double-apply.
        if dedup_store.seen(message["idempotencyKey"]):
            broker.ack(message)        # already processed; skip
            continue
        try:
            process_order(message["payload"])
            dedup_store.mark(message["idempotencyKey"])
            broker.ack(message)        # success; broker drops it
        except Exception:
            broker.nack(message)       # redeliver (at-least-once, P4)
```

- The dedup store is bounded (P6 — Backpressure is Bounded): a
  dedup store that grows without bound is a memory leak. Use a TTL
  window longer than the broker's max-redelivery window, or a
  bounded LRU. See `domains/messaging/delivery-semantics.md` for
  the full idempotency-key dedup-store pattern.

## Ack / Nack (P4 Delivery Semantics are Explicit)

- **Ack** tells the broker the message was processed; the broker
  drops it. **Nack** (negative ack) tells the broker the
  processing failed; the broker redelivers (at-least-once) or
  routes to a DLQ (after the retry budget — P5).
- A consumer that neither acks nor nacks within the visibility
  timeout causes the broker to redeliver (the broker assumes the
  consumer died). This is the at-least-once default: the broker
  prefers duplication to loss.
- The semantic is explicit (P4): at-least-once is the default; the
  consumer must be idempotent (P3). At-most-once is fire-and-forget
  (no ack; the broker drops on send) — lossy but lowest latency.
  Exactly-once is at-least-once plus idempotency, or a
  transactional two-phase commit — see the comparison table below.

## Visibility Timeouts and Redelivery (P4, P5)

- The visibility timeout is the window the broker hides a message
  after delivery, waiting for the ack. If the consumer does not
  ack within the window, the broker makes the message visible
  again and redelivers it (to the same consumer or another). This
  is the at-least-once mechanism: the broker assumes a
  no-ack-in-time consumer is dead.
- The timeout must be longer than the processing time, or the
  broker redelivers a message the consumer is still processing —
  causing duplicate processing (which P3 idempotency makes safe,
  but which wastes resources). A timeout shorter than processing
  time is a P6 (backpressure) smell: the consumer is too slow for
  the configured timeout.
- Redelivery has a budget (P5): after N redeliveries or a TTL, the
  message routes to the DLQ. An unbounded retry budget is the
  `messaging-unbounded-retry` chaos anti-pattern: the consumer
  never makes progress past the poison message.

## FIFO vs Standard Queues (P2 Ordering is a Property, Not an Assumption)

- A **standard queue** delivers in arrival order per receive-node
  but offers no global ordering across shards, no per-message-
  group ordering, and may redeliver out of order under retry. It
  is the high-throughput default; ordering is *not* guaranteed
  (P2: the ordering property is "none" — explicitly documented).
- A **FIFO queue** delivers strict per-message-group order: all
  messages with the same group ID are delivered to one consumer
  in send order. The cost is throughput (FIFO queues cap at lower
  TPS) and latency (the broker must sequence per group). The
  ordering property is "per-group strict" — explicitly documented
  (P2).
- The choice is a P2 decision (which ordering guarantee) and a C8
  decision (throughput cost). A consumer that assumes FIFO on a
  standard queue is a P2 violation: the broker does not provide
  the guarantee the consumer assumes. Document the property; do
  not assume it.

## Prefetch and Concurrency (P6 Backpressure is Bounded)

- **Prefetch** (or max-unacked) bounds how many messages the
  broker delivers to one consumer without an ack. A prefetch of 1
  is strict stop-and-wait (lowest throughput, tightest backpressure);
  a prefetch of N allows the consumer to process N in flight
  (higher throughput, more memory). An unbounded prefetch is a P6
  violation: the broker floods the consumer's memory.
- **Consumer concurrency** is the number of parallel workers
  processing from the queue. More workers increase throughput up to
  the downstream's limit; beyond that, the workers stall the
  downstream (P6 — the backpressure propagates to the
  downstream, not the broker).
- The prefetch × concurrency product is the in-flight cap. Declare
  it (P6): an undeclared cap is a defect — the consumer either
  underutilizes the broker (prefetch too low) or OOMs under load
  (prefetch too high). This is the cross-process analog of
  `domains/concurrency/patterns` Pattern 5 (bounded queue with
  backpressure): concurrency owns the in-process analog;
  messaging owns the broker-backed instance.

## Long Polling (P6, C8 Economy)

- Long polling (or `ReceiveMessage` with a wait-time-seconds)
  holds the receive request open until a message arrives or the
  wait expires. This reduces empty-receive round trips (C8
  economy of the constrained link) and reduces latency-to-first-
  message (the message is delivered when it arrives, not on the
  next poll cycle).
- Long polling is the default for low-throughput queues: short
  polling burns CPU on empty receives; long polling waits for
  work. For high-throughput queues, the broker is usually full
  enough that long polling adds no latency; for low-throughput
  queues, long polling is the difference between 20ms and 20s
  latency-to-first-message.

## Redelivery + DLQ Flow (P5 Dead-Letter Handling is Defined)

- A poison message (unparseable, repeatedly failing, or exhausting
  the retry budget) routes to the dead-letter queue. The DLQ is
  observable (P10 — DLQ depth is an alert) and drainable (an
  operator can inspect, replay, or discard with audit).
- The retry budget is bounded (P6): N redeliveries, or a TTL with
  exponential backoff. After the budget is exhausted, the message
  is moved to the DLQ, not retried forever. An unbounded retry is
  the `messaging-unbounded-retry` chaos anti-pattern (P5 breach).
- The DLQ routing rule is a redelivery-count or TTL threshold
  plus a target queue. See `domains/messaging/delivery-semantics.md`
  for the dead-letter strategy comparison table.

```python
# Redelivery + DLQ flow (P5 dead-letter handling, P6 bounded
# retry budget). The consumer tracks redelivery count; after the
# budget, the message routes to the DLQ. The DLQ is observable
# (P10 — depth is an alert) and drainable.

MAX_REDELIVERIES = 5
DLQ = "orders-dlq"

def consume_with_dlq(broker, dedup_store, process_order):
    for message in broker.receive(queue="orders"):
        # P3 idempotency: a redelivered, already-processed message
        # is acked and skipped (not re-processed, not DLQ'd).
        if dedup_store.seen(message["idempotencyKey"]):
            broker.ack(message)
            continue
        try:
            process_order(message["payload"])
            dedup_store.mark(message["idempotencyKey"])
            broker.ack(message)
        except Exception as exc:
            # P5: bounded retry budget. After MAX_REDELIVERIES,
            # route to DLQ — do NOT retry forever.
            count = message.get("redeliveryCount", 0) + 1
            if count >= MAX_REDELIVERIES:
                broker.send(DLQ, body=json.dumps({
                    "original": message,
                    "reason": str(exc),
                    "deadLetteredAt": now_iso(),
                    "redeliveryCount": count,
                }))
                broker.ack(message)  # remove from the origin queue
                # P10: the DLQ depth must alert. A DLQ that grows
                # with no alert is a silent correctness defect.
            else:
                # Nack with backoff: the broker redelivers after a
                # delay. The backoff caps the retry rate (P6).
                broker.nack(message, delay=exponential_backoff(count))
```

- The `deadLetteredAt` and `reason` fields make the DLQ entry
  observable and auditable: an operator inspecting the DLQ sees
  why each message was dead-lettered and when. See
  `domains/errors/patterns` for the errors-as-data discipline the
  DLQ entry follows.

## Delivery Semantics Comparison (D-069)

| Semantic | Guarantee | Latency Cost | Implementation Cost | When It Fits |
|----------|-----------|-------------|---------------------|--------------|
| **At-most-once** | A message is delivered 0 or 1 times; loss is possible, duplication is not | Lowest (no ack; fire-and-forget) | Lowest (no ack, no dedup) | Telemetry where a dropped sample is acceptable; high-throughput metrics; MQTT QoS 0; logs where a lost line is tolerable. Never for billing, orders, or any side-effecting operation. |
| **At-least-once** | A message is delivered 1 or more times; duplication is possible, loss is not | Low (one ack round-trip) | Medium (consumer must be idempotent — P3; dedup store required) | The default for side-effecting operations: orders, payments, commands. The consumer dedups via idempotency keys (P3); the broker guarantees delivery. Fits the vast majority of broker-backed queues (SQS standard, RabbitMQ ack, MQTT QoS 1). |
| **Exactly-once** | A message is delivered exactly 1 time; no loss, no duplication | Highest (two-phase commit or transactional producer+consumer) | Highest (requires transactions, a transactional producer, and a transactional consumer — or at-least-once plus idempotency, which collapses to at-least-once with dedup) | Rare. Kafka transactions (consume-process-produce in one transaction); MQTT QoS 2 (four-step handshake). In practice, "exactly-once" is usually at-least-once plus idempotency (P3) — the broker does not guarantee it; the consumer enforces it. Jepsen analyses verify broker claims. |

- The default for side-effecting operations is **at-least-once with
  idempotent consumers** (P3). At-most-once is for loss-tolerant
  telemetry. Exactly-once is reserved for the narrow case where
  the consume-process-produce loop must be transactional (Kafka
  transactions) — and even then, the consumer should be idempotent
  as defense-in-depth.
- The latency cost column is the C8 tradeoff: at-most-once is
  cheapest, exactly-once is most expensive. The implementation
  cost column is the C1/C3 tradeoff: at-most-once is simplest,
  exactly-once is most complex (and most fragile — a transactional
  consumer that partially fails is a bug source). The "when it
  fits" column is the P4 decision: declare the semantic per
  channel, do not let it emerge.
- This is the decision matrix required by D-069 for queues; see
  `domains/messaging/delivery-semantics.md` for the correctness
  properties of each semantic and the dead-letter strategy
  comparison.

## Observability — Consumer Lag (P10 Messaging is Observable)

- Consumer lag (messages enqueued minus messages acked, or the
  age of the oldest unacked message) is the primary queue health
  signal. A lag that grows without bound is a P6 violation (the
  consumer is slower than the producer) and a P10 violation if it
  is not alerted.
- DLQ depth is the secondary signal: a DLQ that grows is a P5
  signal (poison messages are accumulating) and a P10 signal if
  not alerted. Wire both to `domains/observability/metrics` as
  SLIs with SLOs (e.g., lag < 1000 messages, DLQ depth < 10).
- A queue with no lag metric is operating blind (P10 violation):
  the operator cannot see the consumer falling behind until the
  downstream effect surfaces.

## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)

- The broker-backed queue is the cross-process analog of the
  in-process bounded buffer. `domains/concurrency/patterns`
  Pattern 5 (Bounded Queue with Backpressure) owns the in-process
  instance ("producer is blocked or signaled" within one program);
  messaging owns the broker-backed instance above it (the producer
  is the broker's enqueue, the consumer is the broker's dequeue, the
  backpressure is the prefetch cap and the lag signal). The
  failure model differs: in-process fails by OOM; broker-backed
  fails by network partition, broker restart, or consumer
  crash-and-retry (D-062).
- The cross-link is one-directional outward (messaging →
  concurrency) per D-026 extended: messaging references concurrency
  as the in-process foundation; concurrency does not back-link to
  messaging.

## What Violates Queue Discipline

| Violation | Principle |
|-----------|-----------|
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
| Non-idempotent consumer under at-least-once delivery (redelivery doubles the effect) | P3 Consumers are Idempotent |
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
| No DLQ (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
| Unbounded prefetch (broker floods consumer memory) | P6 Backpressure is Bounded |
| Unbounded retry budget (no cap; consumer never progresses past the poison) | P5, P6 |
| Consumer that assumes FIFO on a standard queue (ordering not guaranteed) | P2 Ordering is a Property, Not an Assumption |
| Visibility timeout shorter than processing time (redeliver while still processing) | P4, P6 |
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
| No consumer-lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
| Dedup store that grows without bound (memory leak) | P6 Backpressure is Bounded |
| Default prefetch with no rationale (underutilizes or OOMs) | P6, `domains/concurrency/patterns` |