atelier/domains/messaging/pubsub.md

Pub/Sub — Derived Rules

Derives from domains/messaging/first-principles.md. Applies P1 (Messages are Contracts), P2 (Ordering is a Property, Not an Assumption), P3 (Consumers are Idempotent), and P4 (Delivery Semantics are Explicit) primarily, with P7 (partitioning), P10 (per-subscription lag). The messaging-shared-subscription chaos anti-pattern lives here (pre-specified in P4 ATELIER-110). Cross-links domains/messaging/streams for the pub/sub-vs-stream durability boundary and domains/observability/metrics for per-subscription lag.

What Pub/Sub Is (P1 Messages are Contracts)

• Pub/sub is the fan-out primitive: a producer publishes a message to a topic; N independent subscriptions each receive a copy. The message has an explicit, versioned schema (P1): the topic's schema is the contract every subscription agrees to before subscribing. A schemaless topic is a defect — every subscriber breaks silently on the next shape change.
• The boundary with queues is the fan-out ratio. A queue is point-to-point (one producer, one consumer); pub/sub is one-to-many (one producer, N consumers, each with its own subscription). The boundary with streams is the durability model — see the cross-link below. Pub/sub is an async concern because producer and consumers are separate systems and the failure model is network, not crash (D-062).
• See domains/messaging/queues.md for the point-to-point variant and domains/messaging/streams.md for the durable-log variant.

Topic / Subscription Model (P1, P3, P4)

• A topic is the named stream of messages. A subscription is a durable cursor over the topic: each subscription receives every message published after it was created (subject to retention and filtering). The subscription is independent — its ack, redelivery, and DLQ are per-subscription, not shared.
• Each subscription is a consumer under at-least-once by default (P4): the broker redelivers until the subscription acks, and the subscriber must be idempotent (P3). A subscription with no idempotency dedup duplicates every redelivered message.
• The topic's schema evolves compatibly (P9 — Schemas Evolve Compatibly): a new field the old subscriber ignores is backward-compatible; a renamed field the old subscriber parses as undefined is a P1 violation.

# Publish + two independent subscriptions (P1 contract, P3
# idempotency, P4 at-least-once per subscription). Each
# subscription is an independent durable cursor; acking one does
# not affect the other.

import json, uuid

# --- Publisher ---
def publish(topic, event, broker):
    # P1: versioned schema on the topic. All subscribers must
    # understand this schema (or a compatible superset — P9).
    message = {
        "schema": "user.signed-up.v1",
        "id": str(uuid.uuid4()),
        "idempotencyKey": f"user:{event['userId']}:signup",
        "payload": event,
    }
    broker.publish(topic=topic, body=json.dumps(message))

# --- Subscription A: welcome-email service ---
def subscribe_welcome(broker, dedup_store, send_email):
    sub = broker.subscribe(topic="users", subscription="welcome-email")
    for message in sub.receive():
        # P3: idempotent per subscription. A redelivered message is
        # a no-op for THIS subscription, not for the others.
        if dedup_store.seen(("welcome", message["idempotencyKey"])):
            sub.ack(message)
            continue
        send_email(message["payload"]["email"], "Welcome!")
        dedup_store.mark(("welcome", message["idempotencyKey"]))
        sub.ack(message)

# --- Subscription B: analytics-ingest service ---
def subscribe_analytics(broker, dedup_store, ingest):
    # Independent subscription: its own cursor, its own dedup,
    # its own ack. Welcome-email acking does NOT advance this.
    sub = broker.subscribe(topic="users", subscription="analytics")
    for message in sub.receive():
        if dedup_store.seen(("analytics", message["idempotencyKey"])):
            sub.ack(message)
            continue
        ingest(message["payload"])
        dedup_store.mark(("analytics", message["idempotencyKey"]))
        sub.ack(message)

• The dedup key is scoped per subscription: (subscription, idempotencyKey). A redelivery to subscription A that was already processed by A is a no-op for A; the same message delivered to subscription B is processed by B independently. Scoping the dedup key by subscription prevents one subscription's dedup from masking another's redelivery.

Fan-Out Semantics (P4, P7)

• Fan-out means every subscription receives every published message (subject to filtering — see below). The broker duplicates the message per subscription; each subscription's delivery is independent. The fan-out ratio is the number of subscriptions; the broker's cost scales with fan-out × message size.
• Partitioning (P7) applies to topics that are partitioned for throughput: a partitioned topic delivers per-partition order, and each subscription receives from every partition. A subscription that consumes partitions in parallel must handle per-partition ordering and cross-partition non-ordering (P2 — document the property, do not assume global order).
• The delivery semantic is per-subscription (P4): subscription A may be at-least-once, subscription B may be at-most-once (for a loss-tolerant analytics feed). The choice is per subscription, declared, not emergent.

Shared vs Independent Subscriptions (P2, P3 — the chaos anti-pattern)

• An independent subscription is one durable cursor per consumer group: each subscription receives every message in topic order (per partition, P2) and acks independently. This is the correct default: per-consumer ordering and per-consumer idempotency hold.
• A shared subscription is one subscription shared by multiple consumers: the broker dispatches each message to an arbitrary consumer in the shared group. This breaks per-consumer ordering (P2 — consumer A sees message 3 before consumer B sees message
  1. and complicates idempotency (P3 — the dedup state must be shared across consumers, not per-consumer). This is the messaging-shared-subscription chaos anti-pattern (pre-specified in P4 ATELIER-110): the primary breach is P2 (ordering); P3 (idempotency) is the compounding consequence.
• A shared subscription is correct ONLY when the consumers are stateless, the per-message processing is order-independent, and the dedup store is shared (a shared Redis, a shared DB). A shared subscription for order-dependent or per-consumer-stateful processing is the chaos anti-pattern: the broker's arbitrary dispatch breaks the order the consumer assumes.

# The messaging-shared-subscription chaos anti-pattern (P2
# ordering breach, P3 idempotency compounding). Two consumers
# share one subscription; the broker dispatches each message to
# an arbitrary consumer. Per-consumer ordering breaks; dedup
# must be shared (and often is not).

# BAD — shared subscription, per-consumer dedup (chaos):
def shared_subscription_bad(broker, send_email):
    # Both consumers call subscribe with the SAME subscription
    # name. The broker round-robins; consumer A gets msg 1, msg 3;
    # consumer B gets msg 2, msg 4. Per-consumer order is broken.
    # If each consumer has its OWN dedup store, a redelivery to
    # the OTHER consumer re-processes (P3 breach).
    sub = broker.subscribe(topic="users", subscription="shared")
    for message in sub.receive():
        # Per-consumer dedup — WRONG. A redelivered message may
        # land on the other consumer, which has not seen it.
        if local_dedup.seen(message["idempotencyKey"]):  # per-consumer
            sub.ack(message); continue
        send_email(message["payload"]["email"], "Welcome!")
        local_dedup.mark(message["idempotencyKey"])
        sub.ack(message)

# CORRECT — independent subscriptions (per-consumer ordering,
# per-subscription dedup):
def independent_subscriptions_good(broker, send_email):
    sub = broker.subscribe(topic="users", subscription="welcome-email")
    for message in sub.receive():
        if dedup_store.seen(("welcome", message["idempotencyKey"])):
            sub.ack(message); continue
        send_email(message["payload"]["email"], "Welcome!")
        dedup_store.mark(("welcome", message["idempotencyKey"]))
        sub.ack(message)

• If a shared subscription is genuinely required (stateless, order-independent, shared dedup), document the choice and the shared-dedup requirement (P2 — the ordering property is "none across consumers"; P3 — the dedup is shared). The default is independent subscriptions; shared is an opt-in for the narrow case.

Filtering (P4, C8 Economy)

• Subscription filtering lets a subscription receive only messages matching a filter (e.g., event.type == "order"). Filtering at the broker saves bandwidth (C8 — the subscriber does not receive and discard) and reduces subscriber load.
• Server-side filtering (broker evaluates the filter before delivery) is more efficient than client-side filtering (subscriber receives and discards). Server-side filtering is the default where the broker supports it (GCP Pub/Sub, SNS filtering, NATS subject filtering); client-side is the fallback.
• A filter that is too broad wastes bandwidth; a filter that is too narrow drops messages the subscriber needed. The filter is a P1 (contract) and P4 (semantic) decision: the subscription's filter is part of its declared contract.

Ordering Across Subscriptions (P2)

• A topic with per-partition ordering delivers per-partition order to each subscription. Across subscriptions, there is no ordering guarantee: subscription A may ack message 3 while subscription B is still on message 1. This is correct and expected — each subscription is independent.
• Within a subscription, ordering holds per partition (P2 — the documented property). A subscription that processes partitions in parallel must not assume cross-partition order. A subscription that needs global order must use a single partition (sacrificing parallelism, P7) or an external sequencing mechanism.
• The messaging-shared-subscription anti-pattern breaks even per-partition order within a subscription: the broker's arbitrary dispatch to consumers in the shared group breaks the per- partition sequence each consumer sees.

Pub/Sub vs Stream — The Durability Boundary (cross-link messaging/streams)

• Pub/sub and streams are both fan-out or one-to-many primitives, but their durability model differs. Pub/sub is a push-to-subscription model: each subscription is a cursor, retention is short (the subscription's unacked window), and replay is limited to the unacked messages. A subscription that falls behind beyond the retention window loses messages permanently.
• A stream is a durable-log model: messages are retained by the log for a configured window (P8 — Replay and Retention are Configured), and any consumer group can replay from any offset within the window. A stream consumer that falls behind can catch up by replaying; a pub/sub subscription that falls behind beyond retention cannot.
• The choice is the durability requirement: if the consumer must be able to replay (reprocessing, backfill, new consumer starting from the beginning), use a stream. If the consumer only needs the live feed (and can tolerate loss on a long fall-behind), pub/sub is lighter. See domains/messaging/streams.md for the durable-log model, offsets, and consumer groups.

Observability — Per-Subscription Lag (P10)

• Per-subscription lag (messages published minus messages acked for each subscription, or the age of the oldest unacked message per subscription) is the primary pub/sub health signal. Each subscription has its own lag — a fast subscription and a slow subscription on the same topic are independent signals.
• A subscription whose lag grows beyond the retention window is a silent data-loss risk: the broker will drop the oldest messages, and the subscription will never see them. Alert on lag relative to retention — lag approaching retention is the loss threshold.
• Wire per-subscription lag to domains/observability/metrics as an SLI per subscription. A topic with N subscriptions has N lag metrics; a single aggregate hides the slow one. See domains/observability/metrics for the generic SLI/SLO discipline.

What Violates Pub/Sub Discipline

Violation	Principle
Shared subscription for order-dependent processing (broker dispatch breaks per-consumer order)	P2 Ordering is a Property, Not an Assumption
Shared subscription with per-consumer dedup (redelivery to the other consumer re-processes)	P3 Consumers are Idempotent
Schemaless topic (no versioned contract; subscribers parse by guess)	P1 Messages are Contracts
Subscription with no idempotency dedup (redelivered message duplicates the effect)	P3 Consumers are Idempotent
Subscription whose lag approaches retention (silent data loss)	P10, domains/observability/metrics
Unstated delivery semantic per subscription (at-least-once vs at-most-once guessed)	P4 Delivery Semantics are Explicit
Filter that is too narrow (drops messages the subscriber needed)	P1, P4
Partitioned topic with no documented per-partition ordering contract	P2 Ordering is a Property, Not an Assumption
No per-subscription lag metric (slow subscription invisible)	P10 Messaging is Observable
Cross-partition order assumption within a subscription (no global order guarantee)	P2, P7