Jon Chery
f61dffbb5a
---ci--- project: atelier phase: 2 milestone: v0.4 status: complete phase_role: execution phase_tag: v0.3.2 requirements: covered: [ATELIER-97, ATELIER-98, ATELIER-99, ATELIER-100, ATELIER-101] partial: [] ---/ci---
329 lines
21 KiB
Markdown
329 lines
21 KiB
Markdown
# Streams — Derived Rules
|
||
|
||
> Derives from `domains/messaging/first-principles.md`. Applies P8
|
||
> (Replay and Retention are Configured) primarily, with P2
|
||
> (per-partition ordering), P3 (idempotent consumers), P4 (exactly-
|
||
> once via transactions), P7 (partitioning), and P10 (observable
|
||
> consumer-group health). For the Kafka / Kinesis / Pulsar /
|
||
> NATS JetStream decision, see the stream-platform comparison
|
||
> table below. For the consumer-group rebalance strategy choice,
|
||
> see the rebalance enumeration below (IDEATE-41). Cross-links
|
||
> `domains/messaging/delivery-semantics` for exactly-once via
|
||
> transactions, `domains/data/schema-design` for stream schema,
|
||
> and `domains/observability/tracing` for cross-partition traces.
|
||
|
||
## What a Stream Is (P8 Replay and Retention are Configured)
|
||
|
||
- A stream is a durable-log messaging primitive. Messages are
|
||
appended to a partitioned, replicated log; consumers read from an
|
||
offset and advance at their own pace. The log is retained for a
|
||
configured window (P8) — a stream is a durable log, not a pipe.
|
||
A consumer that falls behind can catch up by replaying from an
|
||
earlier offset; a consumer that starts fresh can replay from the
|
||
beginning (within retention).
|
||
- The boundary with queues and pub/sub is the durability model. A
|
||
queue deletes on ack; a pub/sub subscription retains only its
|
||
unacked window; a stream retains the whole log for the configured
|
||
retention. This makes a stream replayable (P8 — the
|
||
reversibility mechanism) and observable as a record (P10 — the
|
||
log is itself an audit of what happened). See
|
||
`domains/messaging/pubsub.md` for the pub/sub-vs-stream
|
||
durability boundary discussion.
|
||
- The boundary with concurrency is per D-062: messaging owns the
|
||
cross-process/network-failure-model angle. A stream is a
|
||
messaging concern because the log spans brokers and consumers
|
||
across a network, and the failure model is partition, broker
|
||
restart, or consumer crash-and-retry — not in-process OOM or
|
||
thread crash.
|
||
|
||
## Partitioned Log Model (P2, P7)
|
||
|
||
- A stream is partitioned for throughput and parallelism. Each
|
||
partition is an ordered, append-only log; messages within a
|
||
partition are strictly ordered (P2 — per-partition strict order
|
||
is the documented property). Across partitions, there is no
|
||
ordering guarantee: partition 0 and partition 1 are independent
|
||
logs.
|
||
- The partition key (P7 — Partitioning is Intentional) determines
|
||
which partition a message lands on. A key that hashes evenly
|
||
spreads load; a key that matches the per-entity ordering need
|
||
(e.g., `userId` for user events) keeps a user's events on one
|
||
partition in order; a key that is too coarse (one partition for
|
||
the whole topic) serializes all traffic.
|
||
- The partition count is a capacity bound: it caps the parallelism
|
||
(one consumer per partition per consumer group) and the
|
||
throughput (each partition has a write-throughput limit). Too
|
||
few partitions cap parallelism; too many partition overhead the
|
||
broker (file handles, replication, rebalance cost). The choice
|
||
is documented (P7), not defaulted.
|
||
|
||
## Offsets (P2, P8)
|
||
|
||
- An offset is a consumer's position in a partition. The consumer
|
||
reads from its last committed offset; acking (committing the
|
||
offset) advances it. A consumer that crashes before committing
|
||
re-reads from the last committed offset (at-least-once by
|
||
default, P4) — the consumer must be idempotent (P3).
|
||
- The offset is per-partition (P2): each partition has its own
|
||
position, and the consumer commits them independently (or
|
||
atomically across partitions in a transaction — see below).
|
||
- Replay (P8) is resetting the offset backward: a consumer can
|
||
replay from the beginning of retention, from a timestamp, or
|
||
from a specific offset. This is the durable-log property that
|
||
distinguishes a stream from a queue.
|
||
|
||
## Consumer Groups (P3, P7, P10)
|
||
|
||
- A consumer group is a set of consumers sharing the stream's
|
||
partitions: each partition is assigned to exactly one consumer
|
||
in the group. The group is the unit of parallelism and the unit
|
||
of offset tracking. Within a group, each consumer handles its
|
||
assigned partitions; across groups, each group independently
|
||
reads the whole stream (the pub/sub fan-out property, per
|
||
subscription/group).
|
||
- A consumer in the group is idempotent (P3): under at-least-once
|
||
(the default), a redelivery after a crash-and-retry re-processes
|
||
messages. The consumer dedups by idempotency key, or processes
|
||
deterministically (e.g., a stateful aggregation that overwrites
|
||
with the latest value).
|
||
- The consumer group's health is observable (P10): per-partition
|
||
lag (offset of the consumer vs the log's head), the group's
|
||
consumption rate, and rebalance events are first-class signals.
|
||
A group whose lag grows without bound is a P6 (backpressure)
|
||
smell and a P10 (observability) violation if not alerted.
|
||
|
||
```python
|
||
# Consumer-group reading from offsets (P2 per-partition order,
|
||
# P3 idempotent under at-least-once, P7 partition assignment,
|
||
# P8 replay from offset). Each consumer in the group handles its
|
||
# assigned partitions; the group commits offsets atomically or
|
||
# per-partition.
|
||
|
||
def consume_stream(stream, group, dedup_store, process_event):
|
||
# Assign partitions to this consumer by the group's
|
||
# rebalance strategy (see the enumeration below).
|
||
for partition in stream.assigned_partitions(group, consumer=ME):
|
||
# Read from the last committed offset (P8 — replay by
|
||
# resetting this offset).
|
||
offset = stream.committed_offset(group, partition)
|
||
for message in stream.read(partition, from_offset=offset):
|
||
# P3: idempotent under at-least-once. A redelivery
|
||
# after a crash-and-retry re-processes; dedup by key.
|
||
if dedup_store.seen(message["idempotencyKey"]):
|
||
stream.commit(group, partition, message["offset"])
|
||
continue
|
||
process_event(message["payload"])
|
||
dedup_store.mark(message["idempotencyKey"])
|
||
# Commit the offset to advance (P8 — the position is
|
||
# the replay pointer).
|
||
stream.commit(group, partition, message["offset"])
|
||
```
|
||
|
||
- The commit-after-process order gives at-least-once (a crash
|
||
before commit re-reads); the commit-before-process order gives
|
||
at-most-once (a crash after commit loses the unprocessed
|
||
message). The default is at-least-once with idempotent consumers
|
||
(P3, P4).
|
||
|
||
## Consumer-Group Rebalance Strategies (IDEATE-41, ATELIER-100 refinement)
|
||
|
||
When a consumer joins or leaves the group, the broker must
|
||
reassign partitions. The rebalance strategy determines the cost
|
||
and the use-case fit. This enumeration parallels the v0.3
|
||
IDEATE-30 drift-type enumeration (each strategy with its
|
||
stop-the-world cost and use-case fit).
|
||
|
||
| Strategy | Mechanism | Partition Stop-the-World Cost | Use-Case Fit |
|
||
|----------|-----------|-------------------------------|--------------|
|
||
| **Eager rebalance** (stop-the-world) | Every consumer in the group revokes ALL its partitions, the broker reassigns the full partition set, then consumers resume. Every rebalance pauses the whole group. | High — every partition pauses for every rebalance; the whole group stops processing during the revocation+reassignment window. Throughput drops to zero during rebalance. | Simple brokers, small groups, or rarely-rebalancing groups where the simplicity of full revocation outweighs the pause cost. Kafka's legacy protocol (pre-2.4). Avoid for large groups or frequent scale events. |
|
||
| **Sticky (incremental cooperative) rebalance** | The broker reassigns only the partitions that must move (the joining/leaving consumer's share); existing partitions stay assigned. The rebalance is incremental and cooperative — no full revocation. | Low — only the moving partitions pause; the rest of the group continues processing. The pause is proportional to the changed partition count, not the total. | The default for large groups, frequent scale events, and rolling deploys. Kafka's CooperativeStickyAssignor (2.4+), Pulsar, NATS JetStream. Prefer for any group where a full stop-the-world on every deploy is unacceptable. |
|
||
| **Cooperative (no-revoke) rebalance** | A subset of sticky where no partition is revoked unless the consumer leaves; only additions are incremental. The strictest minimization of stop-the-world. | Lowest — only added partitions pause; existing assignments are untouched. | Groups where partition assignment is append-only (consumers join but rarely leave). Useful for long-lived consumers with incremental scaling. |
|
||
|
||
- The default for any non-trivial group is **sticky/cooperative**:
|
||
a rolling deploy that triggers an eager rebalance pauses the
|
||
whole group on every pod restart, which is unacceptable at
|
||
scale. The eager strategy is a legacy default that survives
|
||
because it is simple; prefer sticky where the broker supports
|
||
it.
|
||
- The "partition stop-the-world cost" column is the P6
|
||
(backpressure) angle: a full stop-the-world during rebalance
|
||
causes lag to spike (the consumer is paused, the producer is
|
||
not). Sticky rebalance bounds the spike to the moving partitions.
|
||
- A rebalance that pauses without a lag alert is a P10 violation:
|
||
the operator cannot see the rebalance-induced lag. Wire rebalance
|
||
events to `domains/observability/metrics` as an event signal.
|
||
|
||
## Replay and Retention Windows (P8, C5 Reversibility)
|
||
|
||
- Retention is the configured window the log keeps messages: time-
|
||
based (e.g., 7 days), size-based (e.g., 10 GB per partition), or
|
||
compacted (keep the latest value per key — a changelog). A
|
||
stream with no retention is a pipe, not a log (P8 violation); a
|
||
stream with infinite retention grows until disk exhaustion (P6
|
||
violation — backpressure on the broker).
|
||
- Replay (P8) is resetting a consumer's offset to re-read from
|
||
within the retention window. Use cases: reprocessing after a
|
||
consumer bug (replay from the timestamp of the buggy deploy),
|
||
backfilling a new consumer (replay from the beginning), or
|
||
reindexing (replay to rebuild a derived store).
|
||
- Compacted topics (Kafka log-compaction, Pulsar compaction) keep
|
||
the latest value per key and discard older values for the same
|
||
key. This turns the log into a changelog — a durable
|
||
materialized view that replays to the current state. Compaction
|
||
is a P8 (retention) and C5 (reversibility) mechanism: the log
|
||
retains the current state per key and is replayable to it.
|
||
|
||
## Stream Processing (P2, P3, P4)
|
||
|
||
- Stream processing is computing over the stream as it arrives:
|
||
windowing (tumbling, sliding, session windows), joins (stream-
|
||
stream, stream-table), aggregations (count, sum, per-key
|
||
windows), and stateful transformations. The processing is
|
||
per-partition ordered (P2 — a window over a partition is
|
||
deterministic; a window across partitions is not unless the
|
||
window is global).
|
||
- Stream processing consumers are idempotent (P3): a redelivery
|
||
after a crash re-processes a window; the aggregation must
|
||
tolerate re-application (e.g., a sum is idempotent under replay
|
||
if the window is keyed by offset range, not by wall time).
|
||
- Exactly-once stream processing (P4) requires transactions: the
|
||
consume-process-produce loop is one transaction — the input
|
||
offset commit and the output produce are atomic. See the
|
||
transactional exactly-once producer below.
|
||
|
||
## Exactly-Once via Transactions (P4, P3)
|
||
|
||
- Exactly-once stream processing is at-least-once plus a
|
||
transaction: the consumer commits the input offset and produces
|
||
the output in one transactional operation. If the consumer
|
||
crashes mid-transaction, neither the offset commit nor the
|
||
output produce happens — the consumer re-reads from the last
|
||
committed offset and re-processes (at-least-once), but the
|
||
transaction ensures the output is produced exactly once.
|
||
- This is NOT a broker guarantee of exactly-once delivery; it is
|
||
at-least-once delivery plus idempotent/transactional processing
|
||
(P3, P4). Jepsen analyses of Kafka transactions confirm the
|
||
boundaries: the transaction is atomic within the broker, but
|
||
the downstream sink must be transactional or idempotent too.
|
||
- See `domains/messaging/delivery-semantics.md` for the full
|
||
exactly-once-via-idempotency discussion.
|
||
|
||
```python
|
||
# Transactional exactly-once producer (P4 exactly-once via
|
||
# transactions, P3 idempotent produce). The consume-process-
|
||
# produce loop is one transaction: the input offset commit and
|
||
# the output produce are atomic. A crash mid-transaction rolls
|
||
# both back; the consumer re-reads and re-processes.
|
||
|
||
def consume_transform_produce(stream, group, txn_producer):
|
||
# Begin a transaction. All produces and the offset commit in
|
||
# this block are atomic (P4).
|
||
with txn_producer.transaction() as txn:
|
||
for partition in stream.assigned_partitions(group, ME):
|
||
offset = stream.committed_offset(group, partition)
|
||
for message in stream.read(partition, from_offset=offset):
|
||
output = transform(message["payload"])
|
||
# P3: idempotent produce. The txn producer
|
||
# dedups by an epoch+sequence so a retried
|
||
# transaction does not double-produce.
|
||
txn.produce(
|
||
topic="enriched-events",
|
||
key=message["key"],
|
||
value=output,
|
||
idempotencyKey=message["idempotencyKey"],
|
||
)
|
||
# Commit the input offset within the same
|
||
# transaction (P4 atomicity). A crash before
|
||
# txn.commit() rolls this back; the consumer
|
||
# re-reads from the prior offset.
|
||
txn.commit_offset(group, partition, message["offset"])
|
||
# txn.commit() makes the produces and the offset commit
|
||
# visible atomically. A crash before this point aborts
|
||
# both; a crash after is safe (idempotent produce — P3).
|
||
```
|
||
|
||
- The transactional producer's idempotency (P3) is the defense
|
||
against a retried transaction: the broker dedups the output by
|
||
the producer's epoch and sequence so a re-commit does not
|
||
double-produce. The transaction (P4) is the defense against a
|
||
partial failure: the offset and the output commit together.
|
||
|
||
## Stream Schema (P1, P9, cross-link data/schema-design)
|
||
|
||
- A stream's messages carry a versioned schema (P1). The schema
|
||
evolves compatibly (P9): a new field the old consumer ignores is
|
||
backward-compatible; a renamed field the old consumer parses as
|
||
`undefined` is a P1 violation compounded as an evolution defect.
|
||
- Stream schemas are often registered in a schema registry
|
||
(Confluent, Apicurio) that enforces compatibility on produce.
|
||
A producer that tries to publish an incompatible schema is
|
||
rejected; the registry is the P1/P9 enforcement point.
|
||
- See `domains/data/schema-design` for the generic schema-design
|
||
discipline (Avro, Protobuf, JSON Schema); messaging owns the
|
||
stream-specific instance — the registry, the per-topic
|
||
compatibility mode, the consumer-side routing by schema
|
||
version.
|
||
|
||
## Stream-Platform Comparison (D-069)
|
||
|
||
| Axis | Apache Kafka | AWS Kinesis | Apache Pulsar | NATS JetStream |
|
||
|------|--------------|-------------|---------------|----------------|
|
||
| **Ordering** | Per-partition strict (P2); global only via single-partition topic | Per-shard strict; global only via single shard | Per-partition strict; global via single partition; also supports shared (out-of-order) subscriptions | Per-stream strict; per-subject ordering; global via single stream |
|
||
| **Partitioning model** | Partitions (immutable count post-creation; increase requires recreate); key→partition by hash | Shards (reshardable: split/merge at runtime); key→shard by hash | Partitions (resizable; Pulsar's layered architecture separates compute from storage); key→partition by hash | Streams (subject-based; republish to resize); key→stream by subject |
|
||
| **Replay / retention** | Time- or size-based retention; compaction (latest-per-key); replay from offset or timestamp | Time-based retention (24h–365d); replay from sequence number or timestamp; no compaction | Time- or size-based; compaction; replay from offset or timestamp; tiered storage (hot/warm/cold) | Time- or size-based; per-stream max-age; replay from sequence; no native compaction |
|
||
| **Consumer groups** | Group-coordinated; offsets stored in an internal topic; eager (legacy) or sticky/cooperative (2.4+) rebalance | Enhanced fan-out consumers (per-shard HTTP/2 push); KCL for group coordination; no native group rebalance (shard is the unit) | Group-coordinated; shared or failover subscription modes; cooperative rebalance | Per-stream consumers; durable cursors; no native group rebalance (stream is the unit) |
|
||
| **Exactly-once** | Transactions (KIP-98): consume-process-produce atomic; idempotent producer (KIP-516) | No native exactly-once; at-least-once with consumer-side dedup (P3) | Transactions: produce-ack atomic; idempotent producer | At-least-once by default; dedup window per stream (P3 idempotency) |
|
||
| **Use-case fit** | High-throughput durable logs, stream processing (Kafka Streams, Flink), event sourcing, multi-consumer replay | AWS-native streaming, log ingestion, simple ETL within AWS; low operational burden | Cloud-native, geo-replication, tiered storage, mixed pub/sub + streaming; multi-tenant | Lightweight, low-latency, edge-friendly; NATS ecosystem; simpler ops than Kafka |
|
||
| **Watch out for** | Partition count is fixed at creation (resize requires recreate + republish); rebalance cost on large groups; operational complexity | Shard limits per account; no compaction; retention cap at 365 days; AWS lock-in | Two-arch (BookKeeper + Brokers) operational complexity; smaller ecosystem | Smaller ecosystem; no native compaction; fewer stream-processing libraries |
|
||
|
||
- The default for high-throughput durable logs with multi-consumer
|
||
replay is **Kafka**; for AWS-native streaming, **Kinesis**; for
|
||
geo-replicated multi-tenant or mixed pub/sub + streaming,
|
||
**Pulsar**; for lightweight low-latency edge-friendly streaming,
|
||
**NATS JetStream** (which also cross-links `domains/edge/iot`
|
||
via MQTT parallels — see the edge↔messaging bidirectionality in
|
||
`domains/messaging/first-principles.md` §4).
|
||
- The ordering column is the P2 check: every platform provides
|
||
per-partition/per-shard strict order; none provides global order
|
||
across partitions except by single-partition. The replay/
|
||
retention column is the P8 check: every platform retains for a
|
||
configured window; replay is from offset or timestamp. The
|
||
exactly-once column is the P4 check: Kafka and Pulsar provide
|
||
transactions; Kinesis and JetStream rely on at-least-once plus
|
||
consumer-side idempotency (P3).
|
||
|
||
## Cross-Partition Traces (P10, cross-link observability/tracing)
|
||
|
||
- A stream-processing pipeline that fans out across partitions
|
||
must propagate a trace context per event: the trace ID follows
|
||
the event from source to processed output, even as the event
|
||
crosses partition boundaries. Without cross-partion traces, a
|
||
downstream error cannot be traced back to its source event.
|
||
- See `domains/observability/tracing` for the generic distributed-
|
||
tracing discipline (trace context propagation, span
|
||
correlation). Messaging owns the stream-specific instance: the
|
||
trace context is a message header, the span boundary is the
|
||
consume-process-produce edge, and the cross-partition
|
||
correlation is by trace ID (not by partition — partitions are
|
||
independent logs, P2).
|
||
- A stream processor with no trace propagation is a P10
|
||
violation: the operator cannot trace a processed event back to
|
||
its source. Wire the trace context into every produce and every
|
||
consume.
|
||
|
||
## What Violates Stream Discipline
|
||
|
||
| Violation | Principle |
|
||
|-----------|-----------|
|
||
| Stream with no retention (pipe, not log; no replay) | P8 Replay and Retention are Configured |
|
||
| Infinite retention (grows until disk exhaustion) | P8, P6 |
|
||
| Consumer group with eager rebalance at scale (full stop-the-world per deploy) | P6, IDEATE-41 (use sticky/cooperative) |
|
||
| Non-idempotent stream consumer under at-least-once (redelivery re-processes the window) | P3 Consumers are Idempotent |
|
||
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
|
||
| Partition count too low (caps parallelism) or too high (overhead) | P7 |
|
||
| Cross-partition order assumption (no global order guarantee) | P2 Ordering is a Property, Not an Assumption |
|
||
| Exactly-once claimed without transactional consume-process-produce (P4 violation) | P4 Delivery Semantics are Explicit |
|
||
| Stream schema with no registry / no compatibility enforcement (silent shape break) | P1, P9, `domains/data/schema-design` |
|
||
| No per-partition lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
|
||
| No cross-partition trace propagation (downstream error untraceable) | P10, `domains/observability/tracing` |
|
||
| Compacted topic treated as a full log (old values already discarded) | P8, C1 (compaction is a retention mode, not a full log) | |