atelier/domains/messaging/pubsub.md

# Pub/Sub — Derived Rules

> Derives from `domains/messaging/first-principles.md`. Applies P1
> (Messages are Contracts), P2 (Ordering is a Property, Not an
> Assumption), P3 (Consumers are Idempotent), and P4 (Delivery
> Semantics are Explicit) primarily, with P7 (partitioning), P10
> (per-subscription lag). The `messaging-shared-subscription` chaos
> anti-pattern lives here (pre-specified in P4 ATELIER-110).
> Cross-links `domains/messaging/streams` for the pub/sub-vs-stream
> durability boundary and `domains/observability/metrics` for
> per-subscription lag.

## What Pub/Sub Is (P1 Messages are Contracts)

- Pub/sub is the fan-out primitive: a producer publishes a message
  to a topic; N independent subscriptions each receive a copy. The
  message has an explicit, versioned schema (P1): the topic's
  schema is the contract every subscription agrees to before
  subscribing. A schemaless topic is a defect — every subscriber
  breaks silently on the next shape change.
- The boundary with queues is the fan-out ratio. A queue is
  point-to-point (one producer, one consumer); pub/sub is
  one-to-many (one producer, N consumers, each with its own
  subscription). The boundary with streams is the durability model
  — see the cross-link below. Pub/sub is an async concern because
  producer and consumers are separate systems and the failure model
  is network, not crash (D-062).
- See `domains/messaging/queues.md` for the point-to-point variant
  and `domains/messaging/streams.md` for the durable-log variant.

## Topic / Subscription Model (P1, P3, P4)

- A **topic** is the named stream of messages. A **subscription**
  is a durable cursor over the topic: each subscription receives
  every message published after it was created (subject to
  retention and filtering). The subscription is independent — its
  ack, redelivery, and DLQ are per-subscription, not shared.
- Each subscription is a consumer under at-least-once by default
  (P4): the broker redelivers until the subscription acks, and the
  subscriber must be idempotent (P3). A subscription with no
  idempotency dedup duplicates every redelivered message.
- The topic's schema evolves compatibly (P9 — Schemas Evolve
  Compatibly): a new field the old subscriber ignores is
  backward-compatible; a renamed field the old subscriber parses
  as `undefined` is a P1 violation.

```python
# Publish + two independent subscriptions (P1 contract, P3
# idempotency, P4 at-least-once per subscription). Each
# subscription is an independent durable cursor; acking one does
# not affect the other.

import json, uuid

# --- Publisher ---
def publish(topic, event, broker):
    # P1: versioned schema on the topic. All subscribers must
    # understand this schema (or a compatible superset — P9).
    message = {
        "schema": "user.signed-up.v1",
        "id": str(uuid.uuid4()),
        "idempotencyKey": f"user:{event['userId']}:signup",
        "payload": event,
    }
    broker.publish(topic=topic, body=json.dumps(message))

# --- Subscription A: welcome-email service ---
def subscribe_welcome(broker, dedup_store, send_email):
    sub = broker.subscribe(topic="users", subscription="welcome-email")
    for message in sub.receive():
        # P3: idempotent per subscription. A redelivered message is
        # a no-op for THIS subscription, not for the others.
        if dedup_store.seen(("welcome", message["idempotencyKey"])):
            sub.ack(message)
            continue
        send_email(message["payload"]["email"], "Welcome!")
        dedup_store.mark(("welcome", message["idempotencyKey"]))
        sub.ack(message)

# --- Subscription B: analytics-ingest service ---
def subscribe_analytics(broker, dedup_store, ingest):
    # Independent subscription: its own cursor, its own dedup,
    # its own ack. Welcome-email acking does NOT advance this.
    sub = broker.subscribe(topic="users", subscription="analytics")
    for message in sub.receive():
        if dedup_store.seen(("analytics", message["idempotencyKey"])):
            sub.ack(message)
            continue
        ingest(message["payload"])
        dedup_store.mark(("analytics", message["idempotencyKey"]))
        sub.ack(message)
```

- The dedup key is scoped per subscription: `(subscription,
  idempotencyKey)`. A redelivery to subscription A that was already
  processed by A is a no-op for A; the same message delivered to
  subscription B is processed by B independently. Scoping the dedup
  key by subscription prevents one subscription's dedup from
  masking another's redelivery.

## Fan-Out Semantics (P4, P7)

- Fan-out means every subscription receives every published message
  (subject to filtering — see below). The broker duplicates the
  message per subscription; each subscription's delivery is
  independent. The fan-out ratio is the number of subscriptions; the
  broker's cost scales with fan-out × message size.
- Partitioning (P7) applies to topics that are partitioned for
  throughput: a partitioned topic delivers per-partition order, and
  each subscription receives from every partition. A subscription
  that consumes partitions in parallel must handle per-partition
  ordering and cross-partition non-ordering (P2 — document the
  property, do not assume global order).
- The delivery semantic is per-subscription (P4): subscription A
  may be at-least-once, subscription B may be at-most-once (for a
  loss-tolerant analytics feed). The choice is per subscription,
  declared, not emergent.

## Shared vs Independent Subscriptions (P2, P3 — the chaos anti-pattern)

- An **independent subscription** is one durable cursor per
  consumer group: each subscription receives every message in
  topic order (per partition, P2) and acks independently. This is
  the correct default: per-consumer ordering and per-consumer
  idempotency hold.
- A **shared subscription** is one subscription shared by multiple
  consumers: the broker dispatches each message to an arbitrary
  consumer in the shared group. This breaks per-consumer ordering
  (P2 — consumer A sees message 3 before consumer B sees message
  1) and complicates idempotency (P3 — the dedup state must be
  shared across consumers, not per-consumer). This is the
  `messaging-shared-subscription` chaos anti-pattern
  (pre-specified in P4 ATELIER-110): the primary breach is P2
  (ordering); P3 (idempotency) is the compounding consequence.
- A shared subscription is correct ONLY when the consumers are
  stateless, the per-message processing is order-independent, and
  the dedup store is shared (a shared Redis, a shared DB). A shared
  subscription for order-dependent or per-consumer-stateful
  processing is the chaos anti-pattern: the broker's arbitrary
  dispatch breaks the order the consumer assumes.

```python
# The messaging-shared-subscription chaos anti-pattern (P2
# ordering breach, P3 idempotency compounding). Two consumers
# share one subscription; the broker dispatches each message to
# an arbitrary consumer. Per-consumer ordering breaks; dedup
# must be shared (and often is not).

# BAD — shared subscription, per-consumer dedup (chaos):
def shared_subscription_bad(broker, send_email):
    # Both consumers call subscribe with the SAME subscription
    # name. The broker round-robins; consumer A gets msg 1, msg 3;
    # consumer B gets msg 2, msg 4. Per-consumer order is broken.
    # If each consumer has its OWN dedup store, a redelivery to
    # the OTHER consumer re-processes (P3 breach).
    sub = broker.subscribe(topic="users", subscription="shared")
    for message in sub.receive():
        # Per-consumer dedup — WRONG. A redelivered message may
        # land on the other consumer, which has not seen it.
        if local_dedup.seen(message["idempotencyKey"]):  # per-consumer
            sub.ack(message); continue
        send_email(message["payload"]["email"], "Welcome!")
        local_dedup.mark(message["idempotencyKey"])
        sub.ack(message)

# CORRECT — independent subscriptions (per-consumer ordering,
# per-subscription dedup):
def independent_subscriptions_good(broker, send_email):
    sub = broker.subscribe(topic="users", subscription="welcome-email")
    for message in sub.receive():
        if dedup_store.seen(("welcome", message["idempotencyKey"])):
            sub.ack(message); continue
        send_email(message["payload"]["email"], "Welcome!")
        dedup_store.mark(("welcome", message["idempotencyKey"]))
        sub.ack(message)
```

- If a shared subscription is genuinely required (stateless,
  order-independent, shared dedup), document the choice and the
  shared-dedup requirement (P2 — the ordering property is "none
  across consumers"; P3 — the dedup is shared). The default is
  independent subscriptions; shared is an opt-in for the narrow case.

## Filtering (P4, C8 Economy)

- **Subscription filtering** lets a subscription receive only
  messages matching a filter (e.g., `event.type == "order"`).
  Filtering at the broker saves bandwidth (C8 — the subscriber
  does not receive and discard) and reduces subscriber load.
- **Server-side filtering** (broker evaluates the filter before
  delivery) is more efficient than **client-side filtering**
  (subscriber receives and discards). Server-side filtering is the
  default where the broker supports it (GCP Pub/Sub, SNS filtering,
  NATS subject filtering); client-side is the fallback.
- A filter that is too broad wastes bandwidth; a filter that is
  too narrow drops messages the subscriber needed. The filter is
  a P1 (contract) and P4 (semantic) decision: the subscription's
  filter is part of its declared contract.

## Ordering Across Subscriptions (P2)

- A topic with per-partition ordering delivers per-partition order
  to each subscription. Across subscriptions, there is no ordering
  guarantee: subscription A may ack message 3 while subscription B
  is still on message 1. This is correct and expected — each
  subscription is independent.
- Within a subscription, ordering holds per partition (P2 — the
  documented property). A subscription that processes partitions
  in parallel must not assume cross-partition order. A subscription
  that needs global order must use a single partition (sacrificing
  parallelism, P7) or an external sequencing mechanism.
- The `messaging-shared-subscription` anti-pattern breaks even
  per-partition order within a subscription: the broker's arbitrary
  dispatch to consumers in the shared group breaks the per-
  partition sequence each consumer sees.

## Pub/Sub vs Stream — The Durability Boundary (cross-link messaging/streams)

- Pub/sub and streams are both fan-out or one-to-many primitives,
  but their durability model differs. Pub/sub is a
  **push-to-subscription** model: each subscription is a cursor,
  retention is short (the subscription's unacked window), and
  replay is limited to the unacked messages. A subscription that
  falls behind beyond the retention window loses messages
  permanently.
- A stream is a **durable-log** model: messages are retained by
  the log for a configured window (P8 — Replay and Retention are
  Configured), and any consumer group can replay from any offset
  within the window. A stream consumer that falls behind can
  catch up by replaying; a pub/sub subscription that falls behind
  beyond retention cannot.
- The choice is the durability requirement: if the consumer must
  be able to replay (reprocessing, backfill, new consumer starting
  from the beginning), use a stream. If the consumer only needs
  the live feed (and can tolerate loss on a long fall-behind),
  pub/sub is lighter. See `domains/messaging/streams.md` for the
  durable-log model, offsets, and consumer groups.

## Observability — Per-Subscription Lag (P10)

- Per-subscription lag (messages published minus messages acked
  for each subscription, or the age of the oldest unacked message
  per subscription) is the primary pub/sub health signal. Each
  subscription has its own lag — a fast subscription and a slow
  subscription on the same topic are independent signals.
- A subscription whose lag grows beyond the retention window is a
  silent data-loss risk: the broker will drop the oldest messages,
  and the subscription will never see them. Alert on lag relative
  to retention — lag approaching retention is the loss threshold.
- Wire per-subscription lag to `domains/observability/metrics` as
  an SLI per subscription. A topic with N subscriptions has N lag
  metrics; a single aggregate hides the slow one. See
  `domains/observability/metrics` for the generic SLI/SLO
  discipline.

## What Violates Pub/Sub Discipline

| Violation | Principle |
|-----------|-----------|
| Shared subscription for order-dependent processing (broker dispatch breaks per-consumer order) | P2 Ordering is a Property, Not an Assumption |
| Shared subscription with per-consumer dedup (redelivery to the other consumer re-processes) | P3 Consumers are Idempotent |
| Schemaless topic (no versioned contract; subscribers parse by guess) | P1 Messages are Contracts |
| Subscription with no idempotency dedup (redelivered message duplicates the effect) | P3 Consumers are Idempotent |
| Subscription whose lag approaches retention (silent data loss) | P10, `domains/observability/metrics` |
| Unstated delivery semantic per subscription (at-least-once vs at-most-once guessed) | P4 Delivery Semantics are Explicit |
| Filter that is too narrow (drops messages the subscriber needed) | P1, P4 |
| Partitioned topic with no documented per-partition ordering contract | P2 Ordering is a Property, Not an Assumption |
| No per-subscription lag metric (slow subscription invisible) | P10 Messaging is Observable |
| Cross-partition order assumption within a subscription (no global order guarantee) | P2, P7 |