atelier/domains/messaging/streams.md

Streams — Derived Rules

Derives from domains/messaging/first-principles.md. Applies P8 (Replay and Retention are Configured) primarily, with P2 (per-partition ordering), P3 (idempotent consumers), P4 (exactly- once via transactions), P7 (partitioning), and P10 (observable consumer-group health). For the Kafka / Kinesis / Pulsar / NATS JetStream decision, see the stream-platform comparison table below. For the consumer-group rebalance strategy choice, see the rebalance enumeration below (IDEATE-41). Cross-links domains/messaging/delivery-semantics for exactly-once via transactions, domains/data/schema-design for stream schema, and domains/observability/tracing for cross-partition traces.

What a Stream Is (P8 Replay and Retention are Configured)

• A stream is a durable-log messaging primitive. Messages are appended to a partitioned, replicated log; consumers read from an offset and advance at their own pace. The log is retained for a configured window (P8) — a stream is a durable log, not a pipe. A consumer that falls behind can catch up by replaying from an earlier offset; a consumer that starts fresh can replay from the beginning (within retention).
• The boundary with queues and pub/sub is the durability model. A queue deletes on ack; a pub/sub subscription retains only its unacked window; a stream retains the whole log for the configured retention. This makes a stream replayable (P8 — the reversibility mechanism) and observable as a record (P10 — the log is itself an audit of what happened). See domains/messaging/pubsub.md for the pub/sub-vs-stream durability boundary discussion.
• The boundary with concurrency is per D-062: messaging owns the cross-process/network-failure-model angle. A stream is a messaging concern because the log spans brokers and consumers across a network, and the failure model is partition, broker restart, or consumer crash-and-retry — not in-process OOM or thread crash.

Partitioned Log Model (P2, P7)

• A stream is partitioned for throughput and parallelism. Each partition is an ordered, append-only log; messages within a partition are strictly ordered (P2 — per-partition strict order is the documented property). Across partitions, there is no ordering guarantee: partition 0 and partition 1 are independent logs.
• The partition key (P7 — Partitioning is Intentional) determines which partition a message lands on. A key that hashes evenly spreads load; a key that matches the per-entity ordering need (e.g., userId for user events) keeps a user's events on one partition in order; a key that is too coarse (one partition for the whole topic) serializes all traffic.
• The partition count is a capacity bound: it caps the parallelism (one consumer per partition per consumer group) and the throughput (each partition has a write-throughput limit). Too few partitions cap parallelism; too many partition overhead the broker (file handles, replication, rebalance cost). The choice is documented (P7), not defaulted.

Offsets (P2, P8)

• An offset is a consumer's position in a partition. The consumer reads from its last committed offset; acking (committing the offset) advances it. A consumer that crashes before committing re-reads from the last committed offset (at-least-once by default, P4) — the consumer must be idempotent (P3).
• The offset is per-partition (P2): each partition has its own position, and the consumer commits them independently (or atomically across partitions in a transaction — see below).
• Replay (P8) is resetting the offset backward: a consumer can replay from the beginning of retention, from a timestamp, or from a specific offset. This is the durable-log property that distinguishes a stream from a queue.

Consumer Groups (P3, P7, P10)

• A consumer group is a set of consumers sharing the stream's partitions: each partition is assigned to exactly one consumer in the group. The group is the unit of parallelism and the unit of offset tracking. Within a group, each consumer handles its assigned partitions; across groups, each group independently reads the whole stream (the pub/sub fan-out property, per subscription/group).
• A consumer in the group is idempotent (P3): under at-least-once (the default), a redelivery after a crash-and-retry re-processes messages. The consumer dedups by idempotency key, or processes deterministically (e.g., a stateful aggregation that overwrites with the latest value).
• The consumer group's health is observable (P10): per-partition lag (offset of the consumer vs the log's head), the group's consumption rate, and rebalance events are first-class signals. A group whose lag grows without bound is a P6 (backpressure) smell and a P10 (observability) violation if not alerted.

# Consumer-group reading from offsets (P2 per-partition order,
# P3 idempotent under at-least-once, P7 partition assignment,
# P8 replay from offset). Each consumer in the group handles its
# assigned partitions; the group commits offsets atomically or
# per-partition.

def consume_stream(stream, group, dedup_store, process_event):
    # Assign partitions to this consumer by the group's
    # rebalance strategy (see the enumeration below).
    for partition in stream.assigned_partitions(group, consumer=ME):
        # Read from the last committed offset (P8 — replay by
        # resetting this offset).
        offset = stream.committed_offset(group, partition)
        for message in stream.read(partition, from_offset=offset):
            # P3: idempotent under at-least-once. A redelivery
            # after a crash-and-retry re-processes; dedup by key.
            if dedup_store.seen(message["idempotencyKey"]):
                stream.commit(group, partition, message["offset"])
                continue
            process_event(message["payload"])
            dedup_store.mark(message["idempotencyKey"])
            # Commit the offset to advance (P8 — the position is
            # the replay pointer).
            stream.commit(group, partition, message["offset"])

• The commit-after-process order gives at-least-once (a crash before commit re-reads); the commit-before-process order gives at-most-once (a crash after commit loses the unprocessed message). The default is at-least-once with idempotent consumers (P3, P4).

Consumer-Group Rebalance Strategies (IDEATE-41, ATELIER-100 refinement)

When a consumer joins or leaves the group, the broker must reassign partitions. The rebalance strategy determines the cost and the use-case fit. This enumeration parallels the v0.3 IDEATE-30 drift-type enumeration (each strategy with its stop-the-world cost and use-case fit).

Strategy	Mechanism	Partition Stop-the-World Cost	Use-Case Fit
Eager rebalance (stop-the-world)	Every consumer in the group revokes ALL its partitions, the broker reassigns the full partition set, then consumers resume. Every rebalance pauses the whole group.	High — every partition pauses for every rebalance; the whole group stops processing during the revocation+reassignment window. Throughput drops to zero during rebalance.	Simple brokers, small groups, or rarely-rebalancing groups where the simplicity of full revocation outweighs the pause cost. Kafka's legacy protocol (pre-2.4). Avoid for large groups or frequent scale events.
Sticky (incremental cooperative) rebalance	The broker reassigns only the partitions that must move (the joining/leaving consumer's share); existing partitions stay assigned. The rebalance is incremental and cooperative — no full revocation.	Low — only the moving partitions pause; the rest of the group continues processing. The pause is proportional to the changed partition count, not the total.	The default for large groups, frequent scale events, and rolling deploys. Kafka's CooperativeStickyAssignor (2.4+), Pulsar, NATS JetStream. Prefer for any group where a full stop-the-world on every deploy is unacceptable.
Cooperative (no-revoke) rebalance	A subset of sticky where no partition is revoked unless the consumer leaves; only additions are incremental. The strictest minimization of stop-the-world.	Lowest — only added partitions pause; existing assignments are untouched.	Groups where partition assignment is append-only (consumers join but rarely leave). Useful for long-lived consumers with incremental scaling.

• The default for any non-trivial group is sticky/cooperative: a rolling deploy that triggers an eager rebalance pauses the whole group on every pod restart, which is unacceptable at scale. The eager strategy is a legacy default that survives because it is simple; prefer sticky where the broker supports it.
• The "partition stop-the-world cost" column is the P6 (backpressure) angle: a full stop-the-world during rebalance causes lag to spike (the consumer is paused, the producer is not). Sticky rebalance bounds the spike to the moving partitions.
• A rebalance that pauses without a lag alert is a P10 violation: the operator cannot see the rebalance-induced lag. Wire rebalance events to domains/observability/metrics as an event signal.

Replay and Retention Windows (P8, C5 Reversibility)

• Retention is the configured window the log keeps messages: time- based (e.g., 7 days), size-based (e.g., 10 GB per partition), or compacted (keep the latest value per key — a changelog). A stream with no retention is a pipe, not a log (P8 violation); a stream with infinite retention grows until disk exhaustion (P6 violation — backpressure on the broker).
• Replay (P8) is resetting a consumer's offset to re-read from within the retention window. Use cases: reprocessing after a consumer bug (replay from the timestamp of the buggy deploy), backfilling a new consumer (replay from the beginning), or reindexing (replay to rebuild a derived store).
• Compacted topics (Kafka log-compaction, Pulsar compaction) keep the latest value per key and discard older values for the same key. This turns the log into a changelog — a durable materialized view that replays to the current state. Compaction is a P8 (retention) and C5 (reversibility) mechanism: the log retains the current state per key and is replayable to it.

Stream Processing (P2, P3, P4)

• Stream processing is computing over the stream as it arrives: windowing (tumbling, sliding, session windows), joins (stream- stream, stream-table), aggregations (count, sum, per-key windows), and stateful transformations. The processing is per-partition ordered (P2 — a window over a partition is deterministic; a window across partitions is not unless the window is global).
• Stream processing consumers are idempotent (P3): a redelivery after a crash re-processes a window; the aggregation must tolerate re-application (e.g., a sum is idempotent under replay if the window is keyed by offset range, not by wall time).
• Exactly-once stream processing (P4) requires transactions: the consume-process-produce loop is one transaction — the input offset commit and the output produce are atomic. See the transactional exactly-once producer below.

Exactly-Once via Transactions (P4, P3)

• Exactly-once stream processing is at-least-once plus a transaction: the consumer commits the input offset and produces the output in one transactional operation. If the consumer crashes mid-transaction, neither the offset commit nor the output produce happens — the consumer re-reads from the last committed offset and re-processes (at-least-once), but the transaction ensures the output is produced exactly once.
• This is NOT a broker guarantee of exactly-once delivery; it is at-least-once delivery plus idempotent/transactional processing (P3, P4). Jepsen analyses of Kafka transactions confirm the boundaries: the transaction is atomic within the broker, but the downstream sink must be transactional or idempotent too.
• See domains/messaging/delivery-semantics.md for the full exactly-once-via-idempotency discussion.

# Transactional exactly-once producer (P4 exactly-once via
# transactions, P3 idempotent produce). The consume-process-
# produce loop is one transaction: the input offset commit and
# the output produce are atomic. A crash mid-transaction rolls
# both back; the consumer re-reads and re-processes.

def consume_transform_produce(stream, group, txn_producer):
    # Begin a transaction. All produces and the offset commit in
    # this block are atomic (P4).
    with txn_producer.transaction() as txn:
        for partition in stream.assigned_partitions(group, ME):
            offset = stream.committed_offset(group, partition)
            for message in stream.read(partition, from_offset=offset):
                output = transform(message["payload"])
                # P3: idempotent produce. The txn producer
                # dedups by an epoch+sequence so a retried
                # transaction does not double-produce.
                txn.produce(
                    topic="enriched-events",
                    key=message["key"],
                    value=output,
                    idempotencyKey=message["idempotencyKey"],
                )
                # Commit the input offset within the same
                # transaction (P4 atomicity). A crash before
                # txn.commit() rolls this back; the consumer
                # re-reads from the prior offset.
                txn.commit_offset(group, partition, message["offset"])
        # txn.commit() makes the produces and the offset commit
        # visible atomically. A crash before this point aborts
        # both; a crash after is safe (idempotent produce — P3).

• The transactional producer's idempotency (P3) is the defense against a retried transaction: the broker dedups the output by the producer's epoch and sequence so a re-commit does not double-produce. The transaction (P4) is the defense against a partial failure: the offset and the output commit together.

Stream Schema (P1, P9, cross-link data/schema-design)

• A stream's messages carry a versioned schema (P1). The schema evolves compatibly (P9): a new field the old consumer ignores is backward-compatible; a renamed field the old consumer parses as undefined is a P1 violation compounded as an evolution defect.
• Stream schemas are often registered in a schema registry (Confluent, Apicurio) that enforces compatibility on produce. A producer that tries to publish an incompatible schema is rejected; the registry is the P1/P9 enforcement point.
• See domains/data/schema-design for the generic schema-design discipline (Avro, Protobuf, JSON Schema); messaging owns the stream-specific instance — the registry, the per-topic compatibility mode, the consumer-side routing by schema version.

Stream-Platform Comparison (D-069)

Axis	Apache Kafka	AWS Kinesis	Apache Pulsar	NATS JetStream
Ordering	Per-partition strict (P2); global only via single-partition topic	Per-shard strict; global only via single shard	Per-partition strict; global via single partition; also supports shared (out-of-order) subscriptions	Per-stream strict; per-subject ordering; global via single stream
Partitioning model	Partitions (immutable count post-creation; increase requires recreate); key→partition by hash	Shards (reshardable: split/merge at runtime); key→shard by hash	Partitions (resizable; Pulsar's layered architecture separates compute from storage); key→partition by hash	Streams (subject-based; republish to resize); key→stream by subject
Replay / retention	Time- or size-based retention; compaction (latest-per-key); replay from offset or timestamp	Time-based retention (24h–365d); replay from sequence number or timestamp; no compaction	Time- or size-based; compaction; replay from offset or timestamp; tiered storage (hot/warm/cold)	Time- or size-based; per-stream max-age; replay from sequence; no native compaction
Consumer groups	Group-coordinated; offsets stored in an internal topic; eager (legacy) or sticky/cooperative (2.4+) rebalance	Enhanced fan-out consumers (per-shard HTTP/2 push); KCL for group coordination; no native group rebalance (shard is the unit)	Group-coordinated; shared or failover subscription modes; cooperative rebalance	Per-stream consumers; durable cursors; no native group rebalance (stream is the unit)
Exactly-once	Transactions (KIP-98): consume-process-produce atomic; idempotent producer (KIP-516)	No native exactly-once; at-least-once with consumer-side dedup (P3)	Transactions: produce-ack atomic; idempotent producer	At-least-once by default; dedup window per stream (P3 idempotency)
Use-case fit	High-throughput durable logs, stream processing (Kafka Streams, Flink), event sourcing, multi-consumer replay	AWS-native streaming, log ingestion, simple ETL within AWS; low operational burden	Cloud-native, geo-replication, tiered storage, mixed pub/sub + streaming; multi-tenant	Lightweight, low-latency, edge-friendly; NATS ecosystem; simpler ops than Kafka
Watch out for	Partition count is fixed at creation (resize requires recreate + republish); rebalance cost on large groups; operational complexity	Shard limits per account; no compaction; retention cap at 365 days; AWS lock-in	Two-arch (BookKeeper + Brokers) operational complexity; smaller ecosystem	Smaller ecosystem; no native compaction; fewer stream-processing libraries

• The default for high-throughput durable logs with multi-consumer replay is Kafka; for AWS-native streaming, Kinesis; for geo-replicated multi-tenant or mixed pub/sub + streaming, Pulsar; for lightweight low-latency edge-friendly streaming, NATS JetStream (which also cross-links domains/edge/iot via MQTT parallels — see the edge↔messaging bidirectionality in domains/messaging/first-principles.md §4).
• The ordering column is the P2 check: every platform provides per-partition/per-shard strict order; none provides global order across partitions except by single-partition. The replay/ retention column is the P8 check: every platform retains for a configured window; replay is from offset or timestamp. The exactly-once column is the P4 check: Kafka and Pulsar provide transactions; Kinesis and JetStream rely on at-least-once plus consumer-side idempotency (P3).

Cross-Partition Traces (P10, cross-link observability/tracing)

• A stream-processing pipeline that fans out across partitions must propagate a trace context per event: the trace ID follows the event from source to processed output, even as the event crosses partition boundaries. Without cross-partion traces, a downstream error cannot be traced back to its source event.
• See domains/observability/tracing for the generic distributed- tracing discipline (trace context propagation, span correlation). Messaging owns the stream-specific instance: the trace context is a message header, the span boundary is the consume-process-produce edge, and the cross-partition correlation is by trace ID (not by partition — partitions are independent logs, P2).
• A stream processor with no trace propagation is a P10 violation: the operator cannot trace a processed event back to its source. Wire the trace context into every produce and every consume.

What Violates Stream Discipline

Violation	Principle
Stream with no retention (pipe, not log; no replay)	P8 Replay and Retention are Configured
Infinite retention (grows until disk exhaustion)	P8, P6
Consumer group with eager rebalance at scale (full stop-the-world per deploy)	P6, IDEATE-41 (use sticky/cooperative)
Non-idempotent stream consumer under at-least-once (redelivery re-processes the window)	P3 Consumers are Idempotent
Default partition key (no rationale; hotspot or wrong-order)	P7 Partitioning is Intentional
Partition count too low (caps parallelism) or too high (overhead)	P7
Cross-partition order assumption (no global order guarantee)	P2 Ordering is a Property, Not an Assumption
Exactly-once claimed without transactional consume-process-produce (P4 violation)	P4 Delivery Semantics are Explicit
Stream schema with no registry / no compatibility enforcement (silent shape break)	P1, P9, domains/data/schema-design
No per-partition lag metric (consumer falls behind invisibly)	P10, domains/observability/metrics
No cross-partition trace propagation (downstream error untraceable)	P10, domains/observability/tracing
Compacted topic treated as a full log (old values already discarded)	P8, C1 (compaction is a retention mode, not a full log)