# Streams — Derived Rules

> Derives from `domains/messaging/first-principles.md`. Applies P8
> (Replay and Retention are Configured) primarily, with P2
> (per-partition ordering), P3 (idempotent consumers), P4 (exactly-
> once via transactions), P7 (partitioning), and P10 (observable
> consumer-group health). For the Kafka / Kinesis / Pulsar /
> NATS JetStream decision, see the stream-platform comparison
> table below. For the consumer-group rebalance strategy choice,
> see the rebalance enumeration below (IDEATE-41). Cross-links
> `domains/messaging/delivery-semantics` for exactly-once via
> transactions, `domains/data/schema-design` for stream schema,
> and `domains/observability/tracing` for cross-partition traces.

## What a Stream Is (P8 Replay and Retention are Configured)

- A stream is a durable-log messaging primitive. Messages are
  appended to a partitioned, replicated log; consumers read from an
  offset and advance at their own pace. The log is retained for a
  configured window (P8) — a stream is a durable log, not a pipe.
  A consumer that falls behind can catch up by replaying from an
  earlier offset; a consumer that starts fresh can replay from the
  beginning (within retention).
- The boundary with queues and pub/sub is the durability model. A
  queue deletes on ack; a pub/sub subscription retains only its
  unacked window; a stream retains the whole log for the configured
  retention. This makes a stream replayable (P8 — the
  reversibility mechanism) and observable as a record (P10 — the
  log is itself an audit of what happened). See
  `domains/messaging/pubsub.md` for the pub/sub-vs-stream
  durability boundary discussion.
- The boundary with concurrency is per D-062: messaging owns the
  cross-process/network-failure-model angle. A stream is a
  messaging concern because the log spans brokers and consumers
  across a network, and the failure model is partition, broker
  restart, or consumer crash-and-retry — not in-process OOM or
  thread crash.

## Partitioned Log Model (P2, P7)

- A stream is partitioned for throughput and parallelism. Each
  partition is an ordered, append-only log; messages within a
  partition are strictly ordered (P2 — per-partition strict order
  is the documented property). Across partitions, there is no
  ordering guarantee: partition 0 and partition 1 are independent
  logs.
- The partition key (P7 — Partitioning is Intentional) determines
  which partition a message lands on. A key that hashes evenly
  spreads load; a key that matches the per-entity ordering need
  (e.g., `userId` for user events) keeps a user's events on one
  partition in order; a key that is too coarse (one partition for
  the whole topic) serializes all traffic.
- The partition count is a capacity bound: it caps the parallelism
  (one consumer per partition per consumer group) and the
  throughput (each partition has a write-throughput limit). Too
  few partitions cap parallelism; too many partition overhead the
  broker (file handles, replication, rebalance cost). The choice
  is documented (P7), not defaulted.

## Offsets (P2, P8)

- An offset is a consumer's position in a partition. The consumer
  reads from its last committed offset; acking (committing the
  offset) advances it. A consumer that crashes before committing
  re-reads from the last committed offset (at-least-once by
  default, P4) — the consumer must be idempotent (P3).
- The offset is per-partition (P2): each partition has its own
  position, and the consumer commits them independently (or
  atomically across partitions in a transaction — see below).
- Replay (P8) is resetting the offset backward: a consumer can
  replay from the beginning of retention, from a timestamp, or
  from a specific offset. This is the durable-log property that
  distinguishes a stream from a queue.

## Consumer Groups (P3, P7, P10)

- A consumer group is a set of consumers sharing the stream's
  partitions: each partition is assigned to exactly one consumer
  in the group. The group is the unit of parallelism and the unit
  of offset tracking. Within a group, each consumer handles its
  assigned partitions; across groups, each group independently
  reads the whole stream (the pub/sub fan-out property, per
  subscription/group).
- A consumer in the group is idempotent (P3): under at-least-once
  (the default), a redelivery after a crash-and-retry re-processes
  messages. The consumer dedups by idempotency key, or processes
  deterministically (e.g., a stateful aggregation that overwrites
  with the latest value).
- The consumer group's health is observable (P10): per-partition
  lag (offset of the consumer vs the log's head), the group's
  consumption rate, and rebalance events are first-class signals.
  A group whose lag grows without bound is a P6 (backpressure)
  smell and a P10 (observability) violation if not alerted.

```python
# Consumer-group reading from offsets (P2 per-partition order,
# P3 idempotent under at-least-once, P7 partition assignment,
# P8 replay from offset). Each consumer in the group handles its
# assigned partitions; the group commits offsets atomically or
# per-partition.

def consume_stream(stream, group, dedup_store, process_event):
    # Assign partitions to this consumer by the group's
    # rebalance strategy (see the enumeration below).
    for partition in stream.assigned_partitions(group, consumer=ME):
        # Read from the last committed offset (P8 — replay by
        # resetting this offset).
        offset = stream.committed_offset(group, partition)
        for message in stream.read(partition, from_offset=offset):
            # P3: idempotent under at-least-once. A redelivery
            # after a crash-and-retry re-processes; dedup by key.
            if dedup_store.seen(message["idempotencyKey"]):
                stream.commit(group, partition, message["offset"])
                continue
            process_event(message["payload"])
            dedup_store.mark(message["idempotencyKey"])
            # Commit the offset to advance (P8 — the position is
            # the replay pointer).
            stream.commit(group, partition, message["offset"])
```

- The commit-after-process order gives at-least-once (a crash
  before commit re-reads); the commit-before-process order gives
  at-most-once (a crash after commit loses the unprocessed
  message). The default is at-least-once with idempotent consumers
  (P3, P4).

## Consumer-Group Rebalance Strategies (IDEATE-41, ATELIER-100 refinement)

When a consumer joins or leaves the group, the broker must
reassign partitions. The rebalance strategy determines the cost
and the use-case fit. This enumeration parallels the v0.3
IDEATE-30 drift-type enumeration (each strategy with its
stop-the-world cost and use-case fit).

| Strategy | Mechanism | Partition Stop-the-World Cost | Use-Case Fit |
|----------|-----------|-------------------------------|--------------|
| **Eager rebalance** (stop-the-world) | Every consumer in the group revokes ALL its partitions, the broker reassigns the full partition set, then consumers resume. Every rebalance pauses the whole group. | High — every partition pauses for every rebalance; the whole group stops processing during the revocation+reassignment window. Throughput drops to zero during rebalance. | Simple brokers, small groups, or rarely-rebalancing groups where the simplicity of full revocation outweighs the pause cost. Kafka's legacy protocol (pre-2.4). Avoid for large groups or frequent scale events. |
| **Sticky (incremental cooperative) rebalance** | The broker reassigns only the partitions that must move (the joining/leaving consumer's share); existing partitions stay assigned. The rebalance is incremental and cooperative — no full revocation. | Low — only the moving partitions pause; the rest of the group continues processing. The pause is proportional to the changed partition count, not the total. | The default for large groups, frequent scale events, and rolling deploys. Kafka's CooperativeStickyAssignor (2.4+), Pulsar, NATS JetStream. Prefer for any group where a full stop-the-world on every deploy is unacceptable. |
| **Cooperative (no-revoke) rebalance** | A subset of sticky where no partition is revoked unless the consumer leaves; only additions are incremental. The strictest minimization of stop-the-world. | Lowest — only added partitions pause; existing assignments are untouched. | Groups where partition assignment is append-only (consumers join but rarely leave). Useful for long-lived consumers with incremental scaling. |

- The default for any non-trivial group is **sticky/cooperative**:
  a rolling deploy that triggers an eager rebalance pauses the
  whole group on every pod restart, which is unacceptable at
  scale. The eager strategy is a legacy default that survives
  because it is simple; prefer sticky where the broker supports
  it.
- The "partition stop-the-world cost" column is the P6
  (backpressure) angle: a full stop-the-world during rebalance
  causes lag to spike (the consumer is paused, the producer is
  not). Sticky rebalance bounds the spike to the moving partitions.
- A rebalance that pauses without a lag alert is a P10 violation:
  the operator cannot see the rebalance-induced lag. Wire rebalance
  events to `domains/observability/metrics` as an event signal.

## Replay and Retention Windows (P8, C5 Reversibility)

- Retention is the configured window the log keeps messages: time-
  based (e.g., 7 days), size-based (e.g., 10 GB per partition), or
  compacted (keep the latest value per key — a changelog). A
  stream with no retention is a pipe, not a log (P8 violation); a
  stream with infinite retention grows until disk exhaustion (P6
  violation — backpressure on the broker).
- Replay (P8) is resetting a consumer's offset to re-read from
  within the retention window. Use cases: reprocessing after a
  consumer bug (replay from the timestamp of the buggy deploy),
  backfilling a new consumer (replay from the beginning), or
  reindexing (replay to rebuild a derived store).
- Compacted topics (Kafka log-compaction, Pulsar compaction) keep
  the latest value per key and discard older values for the same
  key. This turns the log into a changelog — a durable
  materialized view that replays to the current state. Compaction
  is a P8 (retention) and C5 (reversibility) mechanism: the log
  retains the current state per key and is replayable to it.

## Stream Processing (P2, P3, P4)

- Stream processing is computing over the stream as it arrives:
  windowing (tumbling, sliding, session windows), joins (stream-
  stream, stream-table), aggregations (count, sum, per-key
  windows), and stateful transformations. The processing is
  per-partition ordered (P2 — a window over a partition is
  deterministic; a window across partitions is not unless the
  window is global).
- Stream processing consumers are idempotent (P3): a redelivery
  after a crash re-processes a window; the aggregation must
  tolerate re-application (e.g., a sum is idempotent under replay
  if the window is keyed by offset range, not by wall time).
- Exactly-once stream processing (P4) requires transactions: the
  consume-process-produce loop is one transaction — the input
  offset commit and the output produce are atomic. See the
  transactional exactly-once producer below.

## Exactly-Once via Transactions (P4, P3)

- Exactly-once stream processing is at-least-once plus a
  transaction: the consumer commits the input offset and produces
  the output in one transactional operation. If the consumer
  crashes mid-transaction, neither the offset commit nor the
  output produce happens — the consumer re-reads from the last
  committed offset and re-processes (at-least-once), but the
  transaction ensures the output is produced exactly once.
- This is NOT a broker guarantee of exactly-once delivery; it is
  at-least-once delivery plus idempotent/transactional processing
  (P3, P4). Jepsen analyses of Kafka transactions confirm the
  boundaries: the transaction is atomic within the broker, but
  the downstream sink must be transactional or idempotent too.
- See `domains/messaging/delivery-semantics.md` for the full
  exactly-once-via-idempotency discussion.

```python
# Transactional exactly-once producer (P4 exactly-once via
# transactions, P3 idempotent produce). The consume-process-
# produce loop is one transaction: the input offset commit and
# the output produce are atomic. A crash mid-transaction rolls
# both back; the consumer re-reads and re-processes.

def consume_transform_produce(stream, group, txn_producer):
    # Begin a transaction. All produces and the offset commit in
    # this block are atomic (P4).
    with txn_producer.transaction() as txn:
        for partition in stream.assigned_partitions(group, ME):
            offset = stream.committed_offset(group, partition)
            for message in stream.read(partition, from_offset=offset):
                output = transform(message["payload"])
                # P3: idempotent produce. The txn producer
                # dedups by an epoch+sequence so a retried
                # transaction does not double-produce.
                txn.produce(
                    topic="enriched-events",
                    key=message["key"],
                    value=output,
                    idempotencyKey=message["idempotencyKey"],
                )
                # Commit the input offset within the same
                # transaction (P4 atomicity). A crash before
                # txn.commit() rolls this back; the consumer
                # re-reads from the prior offset.
                txn.commit_offset(group, partition, message["offset"])
        # txn.commit() makes the produces and the offset commit
        # visible atomically. A crash before this point aborts
        # both; a crash after is safe (idempotent produce — P3).
```

- The transactional producer's idempotency (P3) is the defense
  against a retried transaction: the broker dedups the output by
  the producer's epoch and sequence so a re-commit does not
  double-produce. The transaction (P4) is the defense against a
  partial failure: the offset and the output commit together.

## Stream Schema (P1, P9, cross-link data/schema-design)

- A stream's messages carry a versioned schema (P1). The schema
  evolves compatibly (P9): a new field the old consumer ignores is
  backward-compatible; a renamed field the old consumer parses as
  `undefined` is a P1 violation compounded as an evolution defect.
- Stream schemas are often registered in a schema registry
  (Confluent, Apicurio) that enforces compatibility on produce.
  A producer that tries to publish an incompatible schema is
  rejected; the registry is the P1/P9 enforcement point.
- See `domains/data/schema-design` for the generic schema-design
  discipline (Avro, Protobuf, JSON Schema); messaging owns the
  stream-specific instance — the registry, the per-topic
  compatibility mode, the consumer-side routing by schema
  version.

## Stream-Platform Comparison (D-069)

| Axis | Apache Kafka | AWS Kinesis | Apache Pulsar | NATS JetStream |
|------|--------------|-------------|---------------|----------------|
| **Ordering** | Per-partition strict (P2); global only via single-partition topic | Per-shard strict; global only via single shard | Per-partition strict; global via single partition; also supports shared (out-of-order) subscriptions | Per-stream strict; per-subject ordering; global via single stream |
| **Partitioning model** | Partitions (immutable count post-creation; increase requires recreate); key→partition by hash | Shards (reshardable: split/merge at runtime); key→shard by hash | Partitions (resizable; Pulsar's layered architecture separates compute from storage); key→partition by hash | Streams (subject-based; republish to resize); key→stream by subject |
| **Replay / retention** | Time- or size-based retention; compaction (latest-per-key); replay from offset or timestamp | Time-based retention (24h–365d); replay from sequence number or timestamp; no compaction | Time- or size-based; compaction; replay from offset or timestamp; tiered storage (hot/warm/cold) | Time- or size-based; per-stream max-age; replay from sequence; no native compaction |
| **Consumer groups** | Group-coordinated; offsets stored in an internal topic; eager (legacy) or sticky/cooperative (2.4+) rebalance | Enhanced fan-out consumers (per-shard HTTP/2 push); KCL for group coordination; no native group rebalance (shard is the unit) | Group-coordinated; shared or failover subscription modes; cooperative rebalance | Per-stream consumers; durable cursors; no native group rebalance (stream is the unit) |
| **Exactly-once** | Transactions (KIP-98): consume-process-produce atomic; idempotent producer (KIP-516) | No native exactly-once; at-least-once with consumer-side dedup (P3) | Transactions: produce-ack atomic; idempotent producer | At-least-once by default; dedup window per stream (P3 idempotency) |
| **Use-case fit** | High-throughput durable logs, stream processing (Kafka Streams, Flink), event sourcing, multi-consumer replay | AWS-native streaming, log ingestion, simple ETL within AWS; low operational burden | Cloud-native, geo-replication, tiered storage, mixed pub/sub + streaming; multi-tenant | Lightweight, low-latency, edge-friendly; NATS ecosystem; simpler ops than Kafka |
| **Watch out for** | Partition count is fixed at creation (resize requires recreate + republish); rebalance cost on large groups; operational complexity | Shard limits per account; no compaction; retention cap at 365 days; AWS lock-in | Two-arch (BookKeeper + Brokers) operational complexity; smaller ecosystem | Smaller ecosystem; no native compaction; fewer stream-processing libraries |

- The default for high-throughput durable logs with multi-consumer
  replay is **Kafka**; for AWS-native streaming, **Kinesis**; for
  geo-replicated multi-tenant or mixed pub/sub + streaming,
  **Pulsar**; for lightweight low-latency edge-friendly streaming,
  **NATS JetStream** (which also cross-links `domains/edge/iot`
  via MQTT parallels — see the edge↔messaging bidirectionality in
  `domains/messaging/first-principles.md` §4).
- The ordering column is the P2 check: every platform provides
  per-partition/per-shard strict order; none provides global order
  across partitions except by single-partition. The replay/
  retention column is the P8 check: every platform retains for a
  configured window; replay is from offset or timestamp. The
  exactly-once column is the P4 check: Kafka and Pulsar provide
  transactions; Kinesis and JetStream rely on at-least-once plus
  consumer-side idempotency (P3).

## Cross-Partition Traces (P10, cross-link observability/tracing)

- A stream-processing pipeline that fans out across partitions
  must propagate a trace context per event: the trace ID follows
  the event from source to processed output, even as the event
  crosses partition boundaries. Without cross-partion traces, a
  downstream error cannot be traced back to its source event.
- See `domains/observability/tracing` for the generic distributed-
  tracing discipline (trace context propagation, span
  correlation). Messaging owns the stream-specific instance: the
  trace context is a message header, the span boundary is the
  consume-process-produce edge, and the cross-partition
  correlation is by trace ID (not by partition — partitions are
  independent logs, P2).
- A stream processor with no trace propagation is a P10
  violation: the operator cannot trace a processed event back to
  its source. Wire the trace context into every produce and every
  consume.

## What Violates Stream Discipline

| Violation | Principle |
|-----------|-----------|
| Stream with no retention (pipe, not log; no replay) | P8 Replay and Retention are Configured |
| Infinite retention (grows until disk exhaustion) | P8, P6 |
| Consumer group with eager rebalance at scale (full stop-the-world per deploy) | P6, IDEATE-41 (use sticky/cooperative) |
| Non-idempotent stream consumer under at-least-once (redelivery re-processes the window) | P3 Consumers are Idempotent |
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
| Partition count too low (caps parallelism) or too high (overhead) | P7 |
| Cross-partition order assumption (no global order guarantee) | P2 Ordering is a Property, Not an Assumption |
| Exactly-once claimed without transactional consume-process-produce (P4 violation) | P4 Delivery Semantics are Explicit |
| Stream schema with no registry / no compatibility enforcement (silent shape break) | P1, P9, `domains/data/schema-design` |
| No per-partition lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
| No cross-partition trace propagation (downstream error untraceable) | P10, `domains/observability/tracing` |
| Compacted topic treated as a full log (old values already discarded) | P8, C1 (compaction is a retention mode, not a full log) |