Jon Chery
f61dffbb5a
---ci--- project: atelier phase: 2 milestone: v0.4 status: complete phase_role: execution phase_tag: v0.3.2 requirements: covered: [ATELIER-97, ATELIER-98, ATELIER-99, ATELIER-100, ATELIER-101] partial: [] ---/ci---
379 lines
20 KiB
Markdown
379 lines
20 KiB
Markdown
# Delivery Semantics — Derived Rules
|
||
|
||
> Derives from `domains/messaging/first-principles.md`. Applies P4
|
||
> (Delivery Semantics are Explicit), P3 (Consumers are Idempotent),
|
||
> P2 (Ordering is a Property, Not an Assumption), P5 (Dead-Letter
|
||
> Handling is Defined), and P6 (Backpressure is Bounded) primarily,
|
||
> with P10 (DLQ depth as an alert). For the dead-letter strategy
|
||
> decision, see the comparison table below. Includes a fenced
|
||
> idempotency-key dedup-store example (IDEATE-39). Cross-links
|
||
> `domains/concurrency/patterns` for the in-process retry/backoff
|
||
> analog, `domains/errors/patterns` for errors as data for message
|
||
> failures, and `domains/observability/metrics` for DLQ depth as an
|
||
> alert.
|
||
|
||
## The Three Delivery Semantics (P4 Delivery Semantics are Explicit)
|
||
|
||
- **At-most-once**: a message is delivered 0 or 1 times; loss is
|
||
possible, duplication is not. The producer fires-and-forgets; the
|
||
broker does not ack; the consumer does not dedup. Lowest latency,
|
||
lowest implementation cost, lossy. Fits telemetry where a dropped
|
||
sample is acceptable (MQTT QoS 0, fire-and-forget metrics).
|
||
- **At-least-once**: a message is delivered 1 or more times;
|
||
duplication is possible, loss is not. The producer sends and
|
||
waits for a broker ack; the consumer processes and acks; a crash
|
||
before the consumer's ack triggers redelivery. The consumer must
|
||
be idempotent (P3). The default for side-effecting operations
|
||
(orders, payments, commands). The vast majority of broker-backed
|
||
queues (SQS standard, RabbitMQ ack, MQTT QoS 1).
|
||
- **Exactly-once**: a message is delivered exactly 1 time; no
|
||
loss, no duplication. In practice, this is at-least-once plus
|
||
idempotency (P3), or a transactional consume-process-produce loop
|
||
(P4 — see `domains/messaging/streams.md`). Jepsen analyses verify
|
||
broker claims: "exactly-once" requires independent verification;
|
||
the durable engineering practice is at-least-once with idempotent
|
||
consumers (P3), which collapses to exactly-once under correct
|
||
dedup.
|
||
- The semantic is declared per channel (P4), not emergent. An
|
||
unstated semantic is a defect: the consumer guesses, and the
|
||
guess is wrong under the first failure. See
|
||
`domains/messaging/queues.md` for the three-semantics comparison
|
||
table (semantics, latency cost, implementation cost, when each
|
||
fits).
|
||
|
||
## Idempotency (P3 Consumers are Idempotent)
|
||
|
||
- Idempotency is the correctness property that makes at-least-once
|
||
safe. A consumer that processes the same message twice has the
|
||
same effect as processing it once. The mechanism is the
|
||
idempotency key: a per-message unique identifier the consumer
|
||
uses to dedup redeliveries.
|
||
- The idempotency key is per-message, not per-producer or
|
||
per-session. A consumer that dedups by producer alone drops
|
||
distinct messages issued in the same window. Use a UUID per
|
||
message, or a deterministic key derived from the message content
|
||
(e.g., `(entity, operation, version)`).
|
||
- The dedup store is bounded (P6 — Backpressure is Bounded): a
|
||
dedup store that grows without bound is a memory leak. Use a TTL
|
||
window longer than the broker's max-redelivery window, or a
|
||
bounded LRU. The TTL is the P6 bound: a key seen within the TTL
|
||
is a redelivery; a key older than the TTL is expired (the broker
|
||
has given up redelivering it).
|
||
|
||
## Idempotency-Key Dedup Store (IDEATE-39, P3 — fenced example)
|
||
|
||
The dedup store is the concrete mechanism that makes a consumer
|
||
idempotent under at-least-once delivery. This is the fenced
|
||
example required by IDEATE-39 (parallel to the v0.3 IDEATE-29
|
||
signed-attestation fenced example): it is NOT prose-only — the
|
||
consumer-with-dedup-store demonstrates P3 concretely.
|
||
|
||
```python
|
||
# Idempotent consumer with a dedup store (P3 Consumers are
|
||
# Idempotent, P6 Backpressure is Bounded — the dedup store is
|
||
# TTL-bounded). The consumer dedups by idempotency key before
|
||
# processing; a redelivered message is a no-op, not a double-apply.
|
||
|
||
import time
|
||
|
||
# P6: the dedup store is bounded by a TTL window. The TTL must
|
||
# exceed the broker's max-redelivery window; beyond the TTL, the
|
||
# key is expired (the broker has given up). A dedup store with no
|
||
# TTL is a memory leak (P6 violation).
|
||
DEDUP_TTL_SECONDS = 24 * 3600 # longer than max-redelivery window
|
||
|
||
class DedupStore:
|
||
"""A TTL-bounded idempotency-key dedup store (P3, P6).
|
||
|
||
seen(key): True if the key was processed within the TTL.
|
||
mark(key): Record the key as processed (with a timestamp).
|
||
"""
|
||
def __init__(self, backend):
|
||
# backend is a Redis, a DB, or an in-process LRU. The
|
||
# backend must be shared across consumer instances if the
|
||
# subscription is shared (P2/P3 — see domains/messaging/
|
||
# pubsub.md on shared vs independent subscriptions).
|
||
self.backend = backend
|
||
|
||
def seen(self, key: str) -> bool:
|
||
ts = self.backend.get(key)
|
||
if ts is None:
|
||
return False
|
||
if time.time() - ts > DEDUP_TTL_SECONDS:
|
||
# P6: expired. The broker has given up redelivering;
|
||
# this key is no longer a redelivery signal.
|
||
self.backend.delete(key)
|
||
return False
|
||
return True
|
||
|
||
def mark(self, key: str):
|
||
self.backend.set(key, time.time(), ttl=DEDUP_TTL_SECONDS)
|
||
|
||
|
||
# The idempotent consumer: dedup before process, mark after
|
||
# process, ack after mark. A crash before mark re-processes (the
|
||
# dedup store does not have the key); a crash before ack
|
||
# redelivers (the broker did not see the ack) and the dedup store
|
||
# makes the redelivery a no-op (P3).
|
||
def consume_idempotent(broker, dedup: DedupStore, process):
|
||
for message in broker.receive():
|
||
# P3: dedup BEFORE process. A redelivered message is a
|
||
# no-op, not a double-apply.
|
||
if dedup.seen(message["idempotencyKey"]):
|
||
broker.ack(message) # already processed; skip
|
||
continue
|
||
try:
|
||
process(message["payload"])
|
||
# P3: mark AFTER process succeeds. A crash between
|
||
# process and mark re-processes (acceptable: the
|
||
# process must be idempotent OR the mark must be
|
||
# transactional with the process — see below).
|
||
dedup.mark(message["idempotencyKey"])
|
||
broker.ack(message)
|
||
except TransientError as exc:
|
||
# P6: bounded retry with backoff. Nack for redelivery;
|
||
# the broker redelivers after exponential backoff.
|
||
broker.nack(message, delay=backoff(message["attempt"]))
|
||
except PoisonError as exc:
|
||
# P5: poison message. Route to DLQ, do NOT retry
|
||
# forever (see the DLQ routing rule below).
|
||
route_to_dlq(broker, message, exc)
|
||
broker.ack(message) # remove from the origin queue
|
||
```
|
||
|
||
- The order `process → mark → ack` gives at-least-once with
|
||
idempotent dedup: a crash before `mark` re-processes (the dedup
|
||
store does not have the key), and a crash before `ack`
|
||
redelivers (the dedup store makes the redelivery a no-op). If
|
||
`process` is not itself idempotent, the `process → mark` window
|
||
must be transactional (e.g., process and mark in one DB
|
||
transaction) — otherwise a crash in the window double-applies.
|
||
- For a non-idempotent `process` (e.g., a payment that must not
|
||
double-charge), use a transactional dedup: process and mark in
|
||
one DB transaction, so the mark commits iff the process
|
||
commits. This is the "exactly-once via idempotency" pattern
|
||
(P4): at-least-once delivery plus a transactional
|
||
process-and-mark collapses to exactly-once under correct
|
||
transactional semantics.
|
||
|
||
## Ordering (P2 Ordering is a Property, Not an Assumption)
|
||
|
||
- The delivery semantic interacts with ordering (P2). At-least-once
|
||
with per-partition ordering: a redelivery within a partition
|
||
preserves order (the redelivered message re-appears in its
|
||
original position relative to other messages the consumer has
|
||
not yet seen). At-least-once with no ordering: a redelivery may
|
||
appear out of order relative to messages delivered after it.
|
||
- The consumer must not assume an ordering property the broker
|
||
does not provide (P2). A standard queue delivers per-receive-node
|
||
arrival order but no global order and no order across redeliveries;
|
||
a FIFO queue delivers strict per-group order including across
|
||
redeliveries; a partitioned stream delivers strict per-partition
|
||
order across redeliveries. Document the property; do not assume
|
||
it. See `domains/messaging/queues.md` (FIFO vs standard) and
|
||
`domains/messaging/streams.md` (per-partition order).
|
||
|
||
## Dead-Letter Strategies (P5 Dead-Letter Handling is Defined)
|
||
|
||
- A poison message (unparseable, repeatedly failing, or exhausting
|
||
the retry budget) must be routed to a dead-letter queue, not
|
||
retried forever or silently dropped (P5). The DLQ is observable
|
||
(P10 — depth is an alert) and drainable (an operator can
|
||
inspect, replay, or discard with audit).
|
||
- The dead-letter strategy determines when a message is
|
||
dead-lettered and what the operator sees. The choice is the
|
||
decision matrix below (D-069).
|
||
|
||
## Dead-Letter Strategy Comparison (D-069)
|
||
|
||
| Strategy | When It Applies | Failure Visibility | Operational Cost |
|
||
|----------|-----------------|-------------------|------------------|
|
||
| **Retry-count-limit** | A fixed max-redeliveries count (e.g., 5). After N redeliveries, route to DLQ. Simple, predictable. | The redelivery count is visible in the DLQ entry; the operator sees how many times it was retried. | Low — a counter per message; no backoff tuning. Risk: retries fire as fast as the broker redelivers, hammering a downstream that is already failing (P6 — no backoff = no backpressure escape). |
|
||
| **TTL-with-backoff** | A max time-to-live for redelivery (e.g., 30 minutes) with exponential backoff between retries. After the TTL, route to DLQ. | The TTL and the backoff schedule are visible; the operator sees the retry timeline. | Medium — backoff tuning per message type. Benefit: backoff gives the downstream time to recover (P6 — the retry rate is bounded); fits transient failures (a downstream that is briefly unavailable). |
|
||
| **Poison-queue** | A separate queue for messages that fail a specific check (unparseable, schema-invalid, unknown type) before any processing retry. Routed immediately, not retried. | The poison queue is a separate signal from the DLQ; the operator sees parse-vs-process failures distinctly. | Low — a routing rule per check. Benefit: distinguishes "never going to succeed" (poison) from "might succeed on retry" (DLQ). Use for unparseable messages that no retry will fix. |
|
||
| **DLQ + alert** | Any of the above strategies, plus an alert on DLQ depth. The DLQ is observable (P10) — depth, age, and rate are alerted. | Highest — the operator is paged on DLQ growth; the DLQ is a first-class signal, not a graveyard. | Medium — alerting setup per DLQ. This is the P5/P10 floor: a DLQ without an alert is a silent correctness defect (poison messages accumulate invisibly). |
|
||
|
||
- The default for transient failures is **TTL-with-backoff + DLQ
|
||
+ alert**: backoff gives the downstream time to recover (P6),
|
||
the TTL bounds the retry budget (P5), the DLQ captures the
|
||
unprocessable, and the alert makes it visible (P10). The default
|
||
for unparseable messages is **poison-queue + alert**: route
|
||
immediately, do not retry a message no retry will fix.
|
||
- **Retry-count-limit alone** (no backoff, no alert) is the
|
||
`messaging-unbounded-retry` chaos anti-pattern's cousin: it
|
||
caps the count but hammers the downstream at full retry rate,
|
||
and the DLQ grows silently if no alert is wired. Always pair
|
||
a retry budget with backoff (P6) and an alert (P10).
|
||
- The failure-visibility column is the P10 check: every strategy
|
||
must surface the failure to the operator. A strategy with no
|
||
visibility is a P10 violation regardless of its retry semantics.
|
||
- The operational-cost column is the C8 tradeoff: more visibility
|
||
and more backoff cost more to set up but pay back in operational
|
||
stability. The P5/P10 floor is "DLQ + alert"; below that, the
|
||
strategy is a silent defect waiting to grow.
|
||
|
||
## DLQ Routing Rule (P5, P10)
|
||
|
||
```python
|
||
# DLQ routing rule (P5 dead-letter handling, P6 bounded retry with
|
||
# backoff, P10 DLQ depth alert). Combines TTL-with-backoff for
|
||
# transient failures and poison-queue for unparseable messages,
|
||
# with an alert on DLQ depth.
|
||
|
||
import json, time
|
||
|
||
MAX_RETRY_TTL_SECONDS = 30 * 60 # 30 min total retry window
|
||
DLQ = "orders-dlq"
|
||
POISON = "orders-poison" # unparseable; never retried
|
||
|
||
def route_to_dlq(broker, message, reason):
|
||
"""Route a message to the DLQ with audit metadata (P5)."""
|
||
broker.send(DLQ, body=json.dumps({
|
||
"original": message,
|
||
"reason": str(reason),
|
||
"deadLetteredAt": now_iso(),
|
||
"redeliveryCount": message.get("attempt", 0),
|
||
}))
|
||
# P10: emit a metric so DLQ depth alerts fire. A DLQ that
|
||
# grows with no alert is a silent correctness defect (P5/P10).
|
||
metrics.increment("dlq.depth", tags={"queue": "orders"})
|
||
# The ack removes the message from the origin queue; the DLQ
|
||
# is the durable record (P5 — observable and drainable).
|
||
|
||
def consume_with_dlq(broker, dedup, process):
|
||
for message in broker.receive():
|
||
# P1: parse first. An unparseable message is poison —
|
||
# route immediately, do NOT retry (no retry will fix it).
|
||
try:
|
||
payload = json.loads(message["body"])
|
||
except (ValueError, SchemaError) as exc:
|
||
broker.send(POISON, body=json.dumps({
|
||
"original": message["body"],
|
||
"reason": f"parse-failed: {exc}",
|
||
"poisonedAt": now_iso(),
|
||
}))
|
||
metrics.increment("poison.depth", tags={"queue": "orders"})
|
||
broker.ack(message) # remove from origin; poison queue holds it
|
||
continue
|
||
|
||
# P3: dedup before process.
|
||
if dedup.seen(payload["idempotencyKey"]):
|
||
broker.ack(message); continue
|
||
|
||
attempt = payload.get("attempt", 0)
|
||
first_attempt_ts = payload.get("firstAttemptTs", time.time())
|
||
|
||
try:
|
||
process(payload)
|
||
dedup.mark(payload["idempotencyKey"])
|
||
broker.ack(message)
|
||
except TransientError as exc:
|
||
# P6: TTL-with-backoff. If the retry window is
|
||
# exhausted, route to DLQ; otherwise redeliver with
|
||
# exponential backoff.
|
||
if time.time() - first_attempt_ts > MAX_RETRY_TTL_SECONDS:
|
||
route_to_dlq(broker, message, exc) # P5
|
||
broker.ack(message)
|
||
else:
|
||
broker.nack(message, delay=backoff(attempt))
|
||
except PermanentError as exc:
|
||
# A permanent error (e.g., a not-found dependency)
|
||
# does not benefit from retry — route to DLQ now.
|
||
route_to_dlq(broker, message, exc)
|
||
broker.ack(message)
|
||
```
|
||
|
||
- The routing rule distinguishes three failure modes: **poison**
|
||
(unparseable — route immediately, no retry), **transient**
|
||
(retry with backoff until the TTL, then DLQ), and **permanent**
|
||
(a retry will not fix it — DLQ now). This distinction is the P5
|
||
discipline: not every failure is a retry; some are immediate
|
||
DLQs.
|
||
- The DLQ entry carries `reason`, `deadLetteredAt`, and
|
||
`redeliveryCount` — it is auditable (the operator knows why each
|
||
message was dead-lettered and how many times it was retried).
|
||
This is the errors-as-data discipline — see
|
||
`domains/errors/patterns` for the general principle a DLQ entry
|
||
instantiates.
|
||
|
||
## Retry Budgets and Backoff (P6 Backpressure is Bounded)
|
||
|
||
- The retry budget is the cap on redelivery: a count, a TTL, or
|
||
both. After the budget, the message routes to the DLQ (P5). An
|
||
unbounded retry budget is the `messaging-unbounded-retry` chaos
|
||
anti-pattern (P5 breach): the consumer never makes progress past
|
||
the poison message.
|
||
- **Exponential backoff** spaces retries: 1s, 2s, 4s, 8s, ... with
|
||
a jitter to avoid thundering-herd synchrony. Backoff gives the
|
||
downstream time to recover (P6 — the retry rate is bounded,
|
||
giving the downstream a chance to catch up). A retry with no
|
||
backoff hammers the downstream at full rate, making the failure
|
||
worse.
|
||
- The retry budget × the backoff schedule is the P6 bound: the
|
||
consumer's retry load is bounded by design, not by luck. See
|
||
`domains/concurrency/patterns` Pattern 6 (Timeout on Every
|
||
Block) for the in-process retry/backoff analog; messaging owns
|
||
the broker-backed instance where the redelivery comes from the
|
||
broker across a network, not an in-process loop (D-062).
|
||
|
||
## Poison Messages (P5, P1)
|
||
|
||
- A poison message is one no retry will fix: unparseable (the
|
||
schema is wrong, P1), unknown type (the consumer does not handle
|
||
this version, P9), or a permanent failure (a not-found
|
||
dependency). Retrying a poison message wastes resources and
|
||
blocks the queue (P6 — the consumer never makes progress).
|
||
- Poison messages route to the poison queue immediately (no
|
||
retry), distinct from the DLQ (which holds messages that
|
||
exhausted their retry budget on transient failures). The
|
||
distinction is the P5 discipline: a poison queue is for
|
||
"never going to succeed"; a DLQ is for "might have succeeded
|
||
but didn't within the budget."
|
||
- A poison queue without an alert is the same defect as a DLQ
|
||
without an alert (P10): the operator cannot see the poison
|
||
accumulating. Wire both to `domains/observability/metrics`.
|
||
|
||
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
|
||
|
||
- The retry/backoff discipline here is the cross-process analog
|
||
of `domains/concurrency/patterns` Pattern 6 (Timeout on Every
|
||
Block) for in-process retry. Concurrency owns the in-process
|
||
analog (a retry loop within one program, with a timeout per
|
||
attempt); messaging owns the broker-backed instance (the broker
|
||
redelivers across a network, the consumer applies backoff via
|
||
nack-with-delay). The failure model differs: in-process retry
|
||
fails by a thread crash or a timeout; broker-backed retry fails
|
||
by network partition, broker restart, or consumer crash-and-retry
|
||
(D-062).
|
||
- The cross-link is one-directional outward (messaging →
|
||
concurrency) per D-026 extended: messaging references concurrency
|
||
as the in-process foundation; concurrency does not back-link to
|
||
messaging.
|
||
|
||
## Cross-Link to Errors (P5, cross-link errors/patterns)
|
||
|
||
- A poison message is an errors-as-data instance: the failure is
|
||
captured as a DLQ entry (with `reason`, `redeliveryCount`,
|
||
`deadLetteredAt`), not swallowed. See `domains/errors/patterns`
|
||
for the general errors-as-data discipline a DLQ entry
|
||
instantiates. The DLQ is the async-messaging instance of an
|
||
error log — observable, auditable, drainable.
|
||
- The cross-link is one-directional outward (messaging → errors):
|
||
messaging references errors for the errors-as-data pattern;
|
||
errors does not back-link to messaging.
|
||
|
||
## What Violates Delivery-Semantics Discipline
|
||
|
||
| Violation | Principle |
|
||
|-----------|-----------|
|
||
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
|
||
| Dedup store with no TTL (memory leak; grows without bound) | P6 Backpressure is Bounded |
|
||
| Dedup key per-producer (distinct messages in the same window deduped) | P3 Consumers are Idempotent |
|
||
| Retry with no backoff (hammers the downstream at full rate) | P6 Backpressure is Bounded |
|
||
| Unbounded retry budget (consumer never progresses past the poison) | P5 Dead-Letter Handling is Defined |
|
||
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
|
||
| Poison message retried forever (no poison queue, no immediate DLQ) | P5 Dead-Letter Handling is Defined |
|
||
| Process-not-idempotent with non-transactional mark (crash in window double-applies) | P3, P4 |
|
||
| DLQ entry with no reason/audit metadata (uninspectable failure) | P5, `domains/errors/patterns` |
|
||
| Ordering assumption the broker does not provide (FIFO assumed on standard queue) | P2 Ordering is a Property, Not an Assumption | |