Jon Chery
f61dffbb5a
---ci--- project: atelier phase: 2 milestone: v0.4 status: complete phase_role: execution phase_tag: v0.3.2 requirements: covered: [ATELIER-97, ATELIER-98, ATELIER-99, ATELIER-100, ATELIER-101] partial: [] ---/ci---
307 lines
18 KiB
Markdown
307 lines
18 KiB
Markdown
# Messaging — First Principles
|
||
|
||
## 1. The Principles
|
||
|
||
### P1. Messages are Contracts
|
||
A message has an explicit, versioned schema. Producer and consumer
|
||
agree on shape before exchange; the schema is the boundary, not a
|
||
guess. A schemaless message — a free-form JSON blob the consumer
|
||
parses by hope — is a defect: the consumer breaks silently on the
|
||
next shape change, and the producer has no contract to evolve
|
||
against. This derives from `C1 Correctness` (the exchange must
|
||
carry what the parties agreed to) and `C2 Clarity` (the schema
|
||
makes the boundary obvious to both sides). This is the
|
||
cross-process expression of the contract discipline that
|
||
`domains/api/rest` owns for synchronous request/response: the
|
||
message schema is to async exchange what the API contract is to
|
||
sync exchange. It is distinct from
|
||
`domains/concurrency/patterns` Pattern 1 (Message Passing), which
|
||
owns the *in-process* channel primitive — here the contract spans
|
||
separate systems and survives network failure (D-062). See
|
||
`domains/messaging/queues.md` for the queue-flavored application
|
||
and `domains/messaging/streams.md` for the durable-log-flavored
|
||
application.
|
||
|
||
### P2. Ordering is a Property, Not an Assumption
|
||
Ordering guarantees — per-partition strict, global, or none — are
|
||
explicit and documented. "It's FIFO" is a claim that must be backed
|
||
by the broker's partitioning contract, not an assumption the
|
||
consumer makes and the broker may not honor. A standard queue
|
||
delivers in arrival order per receive-node but offers no global
|
||
ordering across shards; a FIFO queue delivers strict per-message-
|
||
group order but at a latency cost; a partitioned stream delivers
|
||
strict per-partition order but only within a partition. Each is a
|
||
distinct, declared property. This derives from `C1 Correctness`
|
||
(order is a correctness property — a consumer that assumes order
|
||
the broker does not provide is wrong) and `C2 Clarity` (the
|
||
ordering guarantee is documented, not discovered in production).
|
||
This is distinct from in-process ordering, which
|
||
`domains/concurrency/patterns` Pattern 1 owns for channels within
|
||
one program: messaging ordering survives network failure, broker
|
||
restart, and consumer crash-and-retry — a stronger failure model
|
||
than thread-local channels (D-062). The `messaging-shared-
|
||
subscription` chaos anti-pattern breaches this rule: two consumers
|
||
sharing one subscription break per-consumer ordering because the
|
||
broker dispatches each message to an arbitrary consumer. See
|
||
`domains/messaging/streams.md` for the partition-order contract
|
||
and `domains/messaging/delivery-semantics.md` for the interaction
|
||
of ordering with the three delivery semantics.
|
||
|
||
### P3. Consumers are Idempotent
|
||
Delivery is at-least-once by default across the network; a
|
||
consumer deduplicates via idempotency keys or deterministic
|
||
processing. "Exactly-once" is idempotency plus at-least-once, not a
|
||
broker guarantee — Jepsen analyses of Kafka, RabbitMQ, and NATS
|
||
establish that exactly-once claims require independent
|
||
verification, and the durable engineering practice is to make
|
||
consumers idempotent under redelivery. A non-idempotent consumer
|
||
under at-least-once delivery doubles the effect on every retry; a
|
||
non-idempotent consumer under a claimed exactly-once broker is a
|
||
bug waiting for the broker's exactly-once invariant to break. This
|
||
derives from `C1 Correctness`: correctness under redelivery is the
|
||
contract, not a nice-to-have. This parallels
|
||
`domains/edge/P5 Edge Operations are Idempotent` (the
|
||
cross-partition device-and-cache-flavored analog) and is the
|
||
cross-process instance of the retry-safety discipline that
|
||
`domains/concurrency/patterns` Pattern 6 (Timeout on Every Block)
|
||
implies for in-process retry. It is distinct from in-process
|
||
retry because the redelivery comes from the broker across a
|
||
network, not from an in-process loop (D-062). See
|
||
`domains/messaging/delivery-semantics.md` for the idempotency-key
|
||
dedup-store pattern.
|
||
|
||
### P4. Delivery Semantics are Explicit
|
||
At-least-once / at-most-once / exactly-once is a declared choice
|
||
per channel, not an emergent behavior. The tradeoff — latency cost,
|
||
implementation complexity, operational cost — is made consciously
|
||
and documented. At-most-once is fire-and-forget (low latency, lossy);
|
||
at-least-once is acked with possible duplication (the default,
|
||
requires idempotent consumers per P3); exactly-once is at-least-once
|
||
plus idempotency or a transactional two-phase commit (highest cost,
|
||
narrowest fit). An unstated semantic is a defect: the consumer
|
||
guesses, and the guess is wrong under the first failure. This
|
||
derives from `C1 Correctness` (the chosen semantic must hold) and
|
||
`C2 Clarity` (the tradeoff is visible to the reader and the
|
||
operator). This is the cross-process analog of the explicit-failure-
|
||
mode discipline that `domains/errors/patterns` owns for synchronous
|
||
code — messaging makes the delivery-mode choice as explicit as an
|
||
error-handling choice. See `domains/messaging/queues.md` for the
|
||
three-semantics comparison table and `domains/messaging/delivery-
|
||
semantics.md` for the correctness properties of each.
|
||
|
||
### P5. Dead-Letter Handling is Defined
|
||
Poison messages — unparseable, repeatedly failing, or exhausting
|
||
the retry budget — are routed to a dead-letter queue, not retried
|
||
forever or silently dropped. The DLQ is observable and drainable: an
|
||
operator can inspect it, replay from it, or discard with audit. An
|
||
unbounded retry loop is a livelock: the consumer never makes
|
||
progress past the poison message. A silent drop is a correctness
|
||
defect: the message vanished with no record. This derives from `C1
|
||
Correctness` (poison messages must not livelock the consumer or
|
||
silently disappear) and `C5 Reversibility` (the DLQ is the
|
||
reversibility mechanism — a dead-lettered message can be reprocessed
|
||
after the bug is fixed). This is the cross-process analog of the
|
||
bounded-error discipline that `domains/errors/patterns` owns for
|
||
synchronous code: a poison message is an error-as-data instance
|
||
that must be observable and recoverable, not swallowed. It is
|
||
distinct from in-process error handling because the failure spans a
|
||
network and a consumer restart (D-062). See
|
||
`domains/messaging/delivery-semantics.md` for the dead-letter
|
||
strategy comparison table and the DLQ routing rule pattern.
|
||
|
||
### P6. Backpressure is Bounded
|
||
A slow consumer cannot unbounded-buffer the broker or the
|
||
producer. Backpressure is explicit: consumer lag is visible,
|
||
max-unacked is bounded, the retry budget is capped. A consumer
|
||
that falls behind without a visible signal is a silent backlog —
|
||
the operator cannot fix what they cannot see, and the broker's
|
||
memory grows without bound until it fails. This derives from `C1
|
||
Correctness` (a backlog that grows until OOM is a correctness
|
||
failure) and `C8 Economy` (the broker's memory is bounded by
|
||
design, not by luck). This is distinct from
|
||
`domains/concurrency/P9 Bounded Queues` and
|
||
`domains/concurrency/patterns` Pattern 5 (Bounded Queue with
|
||
Backpressure), which own the *in-process* analog: concurrency's
|
||
bounded queue fails by OOM or thread crash; messaging's bounded
|
||
backpressure fails by network partition, broker restart, or
|
||
consumer crash-and-retry (D-062). The Reactive Streams
|
||
specification (`request(n)`, `onNext` bounded) is the in-process
|
||
instance; messaging's broker-backed backpressure is the
|
||
cross-process instance above it. See `domains/messaging/queues.md`
|
||
for prefetch and max-unacked and `domains/observability/metrics`
|
||
for consumer-lag as an alert.
|
||
|
||
### P7. Partitioning is Intentional
|
||
The partition key determines ordering, parallelism, and hotspots.
|
||
Key choice is a design decision with documented rationale, not a
|
||
default. A key that hashes unevenly creates a hot partition that
|
||
limits throughput; a key that does not match the ordering need
|
||
breaks per-key semantics; a key that is too coarse (one partition
|
||
for the whole topic) serializes all the traffic. The partition
|
||
count is a capacity bound: too few partitions cap parallelism, too
|
||
many partition overhead the broker. This derives from `C4 Locality`
|
||
(ordering and parallelism are co-located with the partition) and
|
||
`C6 Composability` (the partition is the unit of parallelism and
|
||
scaling — consumer groups compose from per-partition workers).
|
||
This is the cross-process analog of the locality discipline that
|
||
`domains/performance/` owns for generic data-near-compute
|
||
optimization: performance's locality is algorithmic (data near
|
||
compute); messaging's locality is partitional (order and
|
||
parallelism near the partition). See `domains/messaging/streams.md`
|
||
for the partitioned-log model and consumer-group rebalance
|
||
strategies.
|
||
|
||
### P8. Replay and Retention are Configured
|
||
Retention windows and replay-from-offset are explicit. A message
|
||
is not ephemeral by default; the broker is a durable log, not a
|
||
pipe. A topic with no retention is a fire-and-forget stream — a
|
||
consumer that falls behind loses data permanently; a topic with
|
||
infinite retention is an unbounded log — the broker grows until
|
||
disk exhaustion. Both are defects: the retention window is a
|
||
declared bound, and replay-from-offset is the mechanism that makes
|
||
the log durable (re-consumable) rather than ephemeral. This derives
|
||
from `C5 Reversibility` (a retained message is reversible — it can
|
||
be re-consumed; an ephemeral message is not) and `C7 Observability`
|
||
(the durable log is itself an observable record of what happened —
|
||
the offset is the position from which to replay). This is the
|
||
foundation for `domains/messaging/streams.md` and the rule that
|
||
distinguishes a stream from a queue (a queue deletes on ack; a
|
||
stream retains for replay). See `domains/messaging/pubsub.md` for
|
||
the pub/sub-vs-stream durability boundary.
|
||
|
||
### P9. Schemas Evolve Compatibly
|
||
Schema changes are backward- and forward-compatible by
|
||
construction. Breaking changes are versioned migrations, not
|
||
silent shape edits. A producer that ships a new field the old
|
||
consumer ignores is backward-compatible; a consumer that handles a
|
||
missing field the new producer omits is forward-compatible. A
|
||
silent schema change — the producer renames a field and the
|
||
consumer parses `undefined` — is a P1 violation (the contract was
|
||
broken) compounded here as an evolution defect. This derives from
|
||
`C5 Reversibility` (a schema change is reversible by versioning —
|
||
the old shape is still readable) and `C6 Composability` (producers
|
||
and consumers of different versions compose because the schema
|
||
evolves compatibly). This parallels `domains/data/migrations`
|
||
(schema migration for databases) and `domains/api/versioning`
|
||
(API contract evolution): messaging's schema evolution is the
|
||
async instance of the same compatibility discipline. See
|
||
`domains/messaging/streams.md` for the stream-schema-evolution
|
||
angle.
|
||
|
||
### P10. Messaging is Observable
|
||
Consumer lag, DLQ depth, throughput, and consumer-group health are
|
||
first-class signals. Silent backlog is a bug, not a feature: a
|
||
consumer that falls behind with no lag metric is invisible until
|
||
the downstream effect surfaces — by which time the backlog may be
|
||
hours or days. A DLQ that grows without an alert is a silent
|
||
correctness defect: poison messages are accumulating and no one
|
||
knows. This derives from `C7 Observability` (the broker's behavior
|
||
is visible to the operator) and `C1 Correctness` (backlog
|
||
detection is a correctness bound — unbounded lag is a failure).
|
||
This is distinct from `domains/observability/metrics`, which owns
|
||
*generic* structured metrics; messaging owns the *broker-specific*
|
||
signals — lag, DLQ depth, partition imbalance, consumer-group
|
||
rebalance events. See `domains/observability/metrics` for the
|
||
generic SLI/SLO discipline and `domains/observability/tracing` for
|
||
cross-partition traces.
|
||
|
||
## 2. Core Principle Trace
|
||
|
||
Each messaging P-rule derives from one or more core C-rules
|
||
(C1–C8). The matrix extension lands in P4 of the v0.4 plan; the
|
||
traces below are authoritative. Messaging is a broad-derivation
|
||
domain touching 7 of 8 core principles (C1, C2, C4, C5, C6, C7,
|
||
C8); C3 (Simplicity) is not a primary derivation — messaging is
|
||
inherently a tradeoff domain where simplicity yields to the
|
||
correctness of delivery guarantees (a simpler-than-necessary
|
||
delivery model does not handle the failure cases, per C3's
|
||
"simpler than necessary is also a violation").
|
||
|
||
| P-rule | Core | Why |
|
||
|--------|------|-----|
|
||
| P1 Messages are Contracts | C1, C2 | Correctness of the exchange; clarity of the schema boundary |
|
||
| P2 Ordering is a Property, Not an Assumption | C1, C2 | Correctness of order; clarity of the guarantee |
|
||
| P3 Consumers are Idempotent | C1 | Correctness under redelivery |
|
||
| P4 Delivery Semantics are Explicit | C1, C2 | Correctness of the chosen semantic; clarity of the tradeoff |
|
||
| P5 Dead-Letter Handling is Defined | C1, C5 | Correctness of poison-message routing; reversibility of reprocessing |
|
||
| P6 Backpressure is Bounded | C1, C8 | Correctness of bounded backlog; economy of broker memory |
|
||
| P7 Partitioning is Intentional | C4, C6 | Locality of order; composability of parallelism |
|
||
| P8 Replay and Retention are Configured | C5, C7 | Reversibility of replay; observability of the durable log |
|
||
| P9 Schemas Evolve Compatibly | C5, C6 | Reversibility of schema changes; composability of versions |
|
||
| P10 Messaging is Observable | C7, C1 | Observability of lag/DLQ; correctness of backlog detection |
|
||
|
||
## 3. What Violates These Principles
|
||
|
||
| Violation | Principle Breached |
|
||
|-----------|-------------------|
|
||
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
|
||
| "It's FIFO" with no documented partition contract | P2 Ordering is a Property, Not an Assumption |
|
||
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
|
||
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||
| No dead-letter queue (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
|
||
| Unbounded retry budget (no cap; slow consumer stalls the partition) | P6 Backpressure is Bounded |
|
||
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
|
||
| Ephemeral broker (no retention; no replay) | P8 Replay and Retention are Configured |
|
||
| Silent schema change (producer breaks consumers with no version bump) | P9 Schemas Evolve Compatibly |
|
||
| Silent backlog (no lag metric; consumer falls behind invisibly) | P10 Messaging is Observable |
|
||
| Shared subscription (two consumers share one subscription; per-consumer ordering breaks) | P2 Ordering is a Property, Not an Assumption (P3 compounding) |
|
||
| Blocking consumer (slow downstream call with no timeout; broker redelivers to the stuck consumer) | P6 Backpressure is Bounded |
|
||
|
||
## 4. Relationship to Other Domains
|
||
|
||
Messaging systems are the engineering discipline of
|
||
**cross-process, cross-system asynchronous communication via
|
||
brokers**. Producer and consumer are separate systems; the broker
|
||
is the intermediary that brokers delivery, ordering, retention,
|
||
and failure semantics. The distinguishing constraints are a
|
||
cross-process failure model (network, not crash), explicit
|
||
delivery semantics, decoupled producer/consumer lifecycle, and
|
||
replay-and-retention as a durable-log property. Messaging overlaps
|
||
`domains/concurrency/` by *subject* (messages, queues,
|
||
backpressure) but not by *failure model*: per D-062, messaging
|
||
owns the cross-process/network-failure-model angle; concurrency
|
||
owns the in-process/crash-failure-model angle. The discriminator
|
||
is the failure model: concurrency's queue fails by OOM or thread
|
||
crash; messaging's queue fails by network partition, broker
|
||
restart, or consumer crash-and-retry. Messaging extends
|
||
concurrency's bounded-queue/backpressure model to the network-
|
||
partition regime. Cross-links are one-directional outward (per
|
||
D-026 extended); no back-link edits to v0.1/v0.2/v0.3 content.
|
||
|
||
- `domains/concurrency/patterns` ← P6 (the broker-backed bounded
|
||
queue is the cross-process analog of the in-process bounded
|
||
buffer — concurrency Pattern 5 owns in-process; messaging owns
|
||
the network-failure-model instance above it, per D-062)
|
||
- `domains/concurrency/patterns` ← P3 (idempotent retry is the
|
||
cross-process analog of in-process retry-safety — the failure
|
||
model differs: broker redelivery across a network vs in-process
|
||
loop)
|
||
- `domains/observability/metrics` ← P10 (consumer lag and DLQ
|
||
depth as alerts; observability owns the generic SLI/SLO
|
||
discipline, messaging owns the broker-specific signals)
|
||
- `domains/observability/tracing` ← P10 (cross-partition traces
|
||
for stream processing; observability owns the generic tracing
|
||
discipline, messaging owns the cross-partition propagation)
|
||
- `domains/data/schema-design` ← P1, P9 (message schema design
|
||
and evolution; data owns the generic schema discipline,
|
||
messaging owns the cross-process message-shape instance)
|
||
- `domains/errors/patterns` ← P5 (errors as data for message
|
||
failures; a poison message is an error-as-data instance that must
|
||
be observable and recoverable, not swallowed)
|
||
- `domains/edge/iot` ← P4 (the edge↔messaging cross-link
|
||
resolves bidirectionally here: edge/iot.md links outward to
|
||
messaging/queues for MQTT QoS parallels to delivery semantics;
|
||
this first-principles doc acknowledges the back-link — the
|
||
edge/iot.md → messaging/queues link from P1 now resolves because
|
||
messaging/queues.md exists, completing the bidirectionality per
|
||
IDEATE-40)
|
||
|
||
> Note: the edge/iot.md → messaging/queues cross-link (MQTT QoS
|
||
> parallels for delivery semantics) was authored in P1 with a
|
||
> dangling reference; this P2 authorship of messaging/queues.md
|
||
> resolves it. The bidirectionality is verified in P5
|
||
> (ATELIER-114 per IDEATE-40). The cross-link is one-directional
|
||
> outward from edge/iot.md; this first-principles doc
|
||
> acknowledges the resolution without editing edge/iot.md (per
|
||
> D-026 extended — no back-link edits to v0.1/v0.2/v0.3 or to
|
||
> P1-authored edge content). |