Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 666b137dbb | |||
| 29ffb42898 |
@@ -1,14 +1,15 @@
|
|||||||
{
|
{
|
||||||
"phase": 1,
|
"phase": 6,
|
||||||
"stage": "execute",
|
"stage": "complete",
|
||||||
"milestone": "v0.4",
|
"milestone": "v0.4",
|
||||||
"phase_role": "execution",
|
"phase_role": "final",
|
||||||
"project": "atelier",
|
"project": "atelier",
|
||||||
"attempts": 0,
|
"attempts": 0,
|
||||||
"updated_at": "2026-08-05T06:00:00Z",
|
"updated_at": "2026-08-05T07:30:00Z",
|
||||||
"milestone_complete": false,
|
"milestone_complete": true,
|
||||||
"milestone_branch": "milestone/v0.4-edge-quantum-langs",
|
"milestone_branch": "milestone/v0.4-edge-quantum-langs",
|
||||||
"phase_branch": "phase/01-edge",
|
"phase_branch": "phase/06-final-review-ship",
|
||||||
"tag_base": "v0.3",
|
"tag_base": "v0.3",
|
||||||
"phase_tag": "v0.3.1"
|
"phase_tag": "v0.3.6",
|
||||||
|
"release_id": 488
|
||||||
}
|
}
|
||||||
@@ -16,8 +16,18 @@ atelier/
|
|||||||
│ ├── gitops-operators/ # v0.3: GitOps + Operators (argocd, flux, operators, progressive-delivery)
|
│ ├── gitops-operators/ # v0.3: GitOps + Operators (argocd, flux, operators, progressive-delivery)
|
||||||
│ ├── ai-ml/ # v0.3: ML engineering (data-versioning, model-evaluation, serving, monitoring-drift)
|
│ ├── ai-ml/ # v0.3: ML engineering (data-versioning, model-evaluation, serving, monitoring-drift)
|
||||||
│ ├── i18n/ # v0.3: internationalization (locale-resources, formatting, rtl-bidi, testing-i18n)
|
│ ├── i18n/ # v0.3: internationalization (locale-resources, formatting, rtl-bidi, testing-i18n)
|
||||||
│ └── compliance/ # v0.3: compliance/audit (audit-logs, data-retention, policy-as-code, evidence)
|
│ ├── compliance/ # v0.3: compliance/audit (audit-logs, data-retention, policy-as-code, evidence)
|
||||||
|
│ ├── edge/ # v0.4: edge computing (cdn, offline-first, iot, sync)
|
||||||
|
│ └── messaging/ # v0.4: cross-process messaging (queues, pubsub, streams, delivery-semantics)
|
||||||
├── languages/ # Language-specific application of domains
|
├── languages/ # Language-specific application of domains
|
||||||
|
│ ├── typescript.md # v0.1 first-principles stub
|
||||||
|
│ ├── python.md # v0.1 first-principles stub
|
||||||
|
│ ├── go.md # v0.1 first-principles stub
|
||||||
|
│ ├── rust.md # v0.1 first-principles stub
|
||||||
|
│ ├── ts-*.md # v0.4: 4 derived docs (ts-types, ts-tooling, ts-async, ts-testing)
|
||||||
|
│ ├── py-*.md # v0.4: 4 derived docs (py-types, py-tooling, py-async, py-testing)
|
||||||
|
│ ├── go-*.md # v0.4: 4 derived docs (go-types, go-tooling, go-concurrency, go-testing)
|
||||||
|
│ └── rs-*.md # v0.4: 4 derived docs (rs-ownership, rs-tooling, rs-async, rs-testing)
|
||||||
├── review/ # Evaluation checklists and anti-patterns
|
├── review/ # Evaluation checklists and anti-patterns
|
||||||
├── matrix/ # Cross-reference: domain ↔ core
|
├── matrix/ # Cross-reference: domain ↔ core
|
||||||
└── examples/ # Worked examples (good + bad)
|
└── examples/ # Worked examples (good + bad)
|
||||||
@@ -106,4 +116,28 @@ From the v0.3 ideation stage (IDEATE-17..30), the following architectural refine
|
|||||||
- **Core Principle Coverage table (IDEATE-19 → ATELIER-81):** `matrix/domain-coverage.md` contains two tables — the per-domain row schema table (covered by v0.2 IDEATE-03) AND the "Core Principle Coverage" table mapping C1–C8 → domains. Both must be extended for the 4 new domains; the C-rule counts shift (e.g., C4 Locality adds i18n + gitops; C5 Reversibility adds ai-ml + compliance + gitops + i18n).
|
- **Core Principle Coverage table (IDEATE-19 → ATELIER-81):** `matrix/domain-coverage.md` contains two tables — the per-domain row schema table (covered by v0.2 IDEATE-03) AND the "Core Principle Coverage" table mapping C1–C8 → domains. Both must be extended for the 4 new domains; the C-rule counts shift (e.g., C4 Locality adds i18n + gitops; C5 Reversibility adds ai-ml + compliance + gitops + i18n).
|
||||||
- **Cross-link type unchanged:** v0.3 introduces no new cross-link type. All cross-links remain one-directional outward from new domains to existing (D-033). No back-link edits to v0.1/v0.2 content.
|
- **Cross-link type unchanged:** v0.3 introduces no new cross-link type. All cross-links remain one-directional outward from new domains to existing (D-033). No back-link edits to v0.1/v0.2 content.
|
||||||
|
|
||||||
See `.ciagent/atelier/RESEARCH.md` "v0.3 Research" for the full prior-art survey and principle inventory rationale, and `.ciagent/atelier/PERSONAS.md` for the v0.3 persona roster (5 active: lead-developer, tech-writer, domain-expert + 2 phase-specific platform-engineer, ml-engineer; 3 default personas deactivated).
|
See `.ciagent/atelier/RESEARCH.md` "v0.3 Research" for the full prior-art survey and principle inventory rationale, and `.ciagent/atelier/PERSONAS.md` for the v0.3 persona roster (5 active: lead-developer, tech-writer, domain-expert + 2 phase-specific platform-engineer, ml-engineer; 3 default personas deactivated).
|
||||||
|
|
||||||
|
## v0.4 Domain Additions
|
||||||
|
|
||||||
|
Two new top-level domains and a language-derived extension expand the tree under the same hierarchy rules. All follow the v0.1/v0.2/v0.3 contract: docs-only markdown with illustrative code fences (no runtime/deployable artifacts per D-020). Total matrix grows from 170 → 190 domain principles across 17 → 19 domains.
|
||||||
|
|
||||||
|
- **`edge/`** — edge computing domain. Owns the proximity/location/constraint angle (D-061): compute near the user or data source, offline as a first-class state, constrained resources declared, sync conflicts bounded, partition-survivable telemetry. Boundary vs `performance/` (generic measurement/optimization), `observability/` (generic structured telemetry), and `concurrency/` (in-process primitives): edge owns only the proximity/location/disconnection concerns that arise at the network edge — performance's locality is algorithmic, edge's locality is geographic (C4). 10 P-rules, broad derivation (7 C-rules: C1, C3, C4, C5, C6, C7, C8). Depends on `core/`. Cross-links are one-directional outward (D-026 extended): `performance/frontend` (P6 geographic cache invalidation vs generic caching), `performance/P4` (P3 constrained-device vs generic resource bounds), `observability/metrics` + `observability/logging` (P10 local-first telemetry), `concurrency/patterns` (P5 offline write-queue analog), `security/secrets` (P9 device identity), `security/input-validation` (P6 cache poisoning), `data/migrations` (P4 schema migration under sync). Edge↔messaging intra-v0.4 cross-links resolve bidirectionally in P5 (ATELIER-114, IDEATE-40): `edge/iot.md` → `messaging/queues` (MQTT QoS parallels to delivery semantics). Derived docs: `cdn.md`, `offline-first.md`, `iot.md`, `sync.md`.
|
||||||
|
|
||||||
|
- **`messaging/`** — cross-process messaging domain. Owns the cross-process/network-failure-model angle (D-062): messages as contracts, explicit delivery semantics, idempotent consumers, bounded backpressure, observable lag. Boundary vs `concurrency/` (in-process/crash-failure-model): messaging's queue fails by network partition/broker restart/consumer crash-and-retry; concurrency's queue fails by OOM or thread crash. The discriminator is the failure model (D-062). 10 P-rules, broad derivation (7 C-rules: C1, C2, C4, C5, C6, C7, C8). Depends on `core/`. Cross-links are one-directional outward (D-026 extended): `concurrency/patterns` (P6 broker-backed bounded queue vs in-process Pattern 5; P3 cross-process idempotent retry vs in-process retry), `observability/metrics` + `observability/tracing` (P10 broker signals), `data/schema-design` (P1/P9 message schema evolution), `errors/patterns` (P5 errors as data for poison messages), `edge/iot` (P4 — bidirectional resolution of the MQTT QoS cross-link per IDEATE-40). Derived docs: `queues.md`, `pubsub.md`, `streams.md`, `delivery-semantics.md`.
|
||||||
|
|
||||||
|
### v0.4 Language-Derived Extension
|
||||||
|
|
||||||
|
The `languages/` tree extends from a first-principles-only stub (v0.1) to a full derived-doc tree (v0.4). Each of the 4 v0.1 stubs (`typescript.md`, `python.md`, `go.md`, `rust.md`) is extended with a pointer to its 4 derived docs (16 total). Language-derived docs **apply, not derive** domain principles (D-063): they introduce **no new P-rules** and **no new matrix rows** (D-066). Every section header names the traced core C-rule(s) AND the traced domain P-rule(s) inline (e.g., `## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)`). The first-principles stubs retain their existing section structure — **no P1–P10 numbering** is added to languages (D-063); languages are an application lens, not a principle source, preserving the `core > domains > languages` ordering ("conflicts go UP, never sideways").
|
||||||
|
|
||||||
|
The `matrix/principles-matrix.md` gains **no language rows** — it stays at 190 P-rules / 19 domains (D-066). Language-derived-doc traceability is captured in a new `## Languages Coverage` sub-table in `matrix/domain-coverage.md` (exactly 16 rows, one per derived doc) — the verifiable aggregate; inline section-headers are the per-doc detail. Every derived doc traces to ≥1 domain P-rule (D-066, no orphans).
|
||||||
|
|
||||||
|
Derived docs by language:
|
||||||
|
- **TypeScript** — `ts-types.md`, `ts-tooling.md`, `ts-async.md`, `ts-testing.md`
|
||||||
|
- **Python** — `py-types.md`, `py-tooling.md`, `py-async.md`, `py-testing.md`
|
||||||
|
- **Go** — `go-types.md`, `go-tooling.md`, `go-concurrency.md`, `go-testing.md`
|
||||||
|
- **Rust** — `rs-ownership.md`, `rs-tooling.md`, `rs-async.md`, `rs-testing.md`
|
||||||
|
|
||||||
|
The manifest (`MANIFEST.md`) is extended with a new `## Languages` section (D-065) listing the 4 v0.1 stubs (closing pre-existing drift parallel to the v0.2 ESC-002 / ATELIER-91 precedent) and the 16 new v0.4 derived docs. The `review/` docs (agent-checklist, peer-review-checklist, anti-patterns) gain edge, messaging, and language-derived trigger sections. Examples (P5) are illustrative markdown with fenced code only — no standalone `.ts`/`.py`/`.go`/`.rs` runtime artifacts (D-020 docs-only contract, extended in v0.4 to language-derived-doc artifact types).
|
||||||
|
|
||||||
|
See `.ciagent/atelier/RESEARCH-v0.4.md` for the full prior-art survey (Akamai/Cloudflare, Kafka/RabbitMQ/NATS, Jepsen, Reactive Streams, CloudEvents, CRDTs, PouchDB, MQTT/CoAP) and `.ciagent/atelier/PERSONAS.md` for the v0.4 persona roster (5 active: lead-developer, tech-writer, domain-expert + 2 phase-specific edge-engineer, languages-engineer).
|
||||||
@@ -3,9 +3,10 @@
|
|||||||
> Generated during P0 RESEARCH. Atelier is a documentation-only project;
|
> Generated during P0 RESEARCH. Atelier is a documentation-only project;
|
||||||
> the default runtime persona roster is deactivated. Custom personas
|
> the default runtime persona roster is deactivated. Custom personas
|
||||||
> reflect the docs-as-code nature of the work. Updated per milestone:
|
> reflect the docs-as-code nature of the work. Updated per milestone:
|
||||||
> v0.4 adds `edge-engineer` (P1) and `languages-engineer` (P3) phase-
|
> v0.4 added `edge-engineer` (P1) and `languages-engineer` (P3) phase-
|
||||||
> specific personas (D-067); messaging (P2) is authored by tech-writer
|
> specific personas (D-067); both REMOVED after v0.4 completion — roster
|
||||||
> + domain-expert with edge-engineer consultation.
|
> returned to 3 active personas. Messaging (P2) was authored by
|
||||||
|
> tech-writer + domain-expert with edge-engineer consultation.
|
||||||
|
|
||||||
## Active Personas
|
## Active Personas
|
||||||
|
|
||||||
@@ -23,7 +24,7 @@
|
|||||||
- **frameworks:** []
|
- **frameworks:** []
|
||||||
- **constraints:** ["markdown-first", "examples are mandatory", "why over what", "audience awareness"]
|
- **constraints:** ["markdown-first", "examples are mandatory", "why over what", "audience awareness"]
|
||||||
- **territory:** ["**/*.md", "LICENSE", "CHANGELOG.md", "CONTRIBUTING.md"]
|
- **territory:** ["**/*.md", "LICENSE", "CHANGELOG.md", "CONTRIBUTING.md"]
|
||||||
- **reason:** Primary author of all framework markdown content. Owns docs-as-code discipline (Documentation P1). Excludes `.ciagent/**` governance files (owned by lead-developer). For v0.4: co-authors the messaging domain (P2) with domain-expert, per D-067.
|
- **reason:** Primary author of all framework markdown content. Owns docs-as-code discipline (Documentation P1). Excludes `.ciagent/**` governance files (owned by lead-developer). For v0.4: co-authored the messaging domain (P2) with domain-expert, per D-067. v0.4 complete.
|
||||||
|
|
||||||
### domain-expert
|
### domain-expert
|
||||||
- **active:** true
|
- **active:** true
|
||||||
@@ -31,7 +32,7 @@
|
|||||||
- **frameworks:** []
|
- **frameworks:** []
|
||||||
- **constraints:** ["every principle traces to core", "10 principles per domain", "no orphans in matrix"]
|
- **constraints:** ["every principle traces to core", "10 principles per domain", "no orphans in matrix"]
|
||||||
- **territory:** ["core/**", "domains/**", "matrix/**", "review/**", "examples/**"]
|
- **territory:** ["core/**", "domains/**", "matrix/**", "review/**", "examples/**"]
|
||||||
- **reason:** Validates domain principle correctness and traceability to core principles. Ensures the matrix is complete and no domain rule is orphaned. For v0.4: co-authors the messaging domain (P2) with tech-writer, consulting edge-engineer for the D-062 boundary, per D-067.
|
- **reason:** Validates domain principle correctness and traceability to core principles. Ensures the matrix is complete and no domain rule is orphaned. For v0.4: co-authored the messaging domain (P2) with tech-writer, consulting edge-engineer for the D-062 boundary, per D-067. v0.4 complete.
|
||||||
|
|
||||||
## Inactive Personas (from default roster)
|
## Inactive Personas (from default roster)
|
||||||
|
|
||||||
@@ -49,25 +50,25 @@
|
|||||||
|
|
||||||
## Phase-Specific Personas
|
## Phase-Specific Personas
|
||||||
|
|
||||||
### edge-engineer (v0.4 — REMOVE after milestone completion)
|
### edge-engineer (v0.4 — REMOVED after milestone completion)
|
||||||
|
|
||||||
- **active:** true
|
- **active:** false
|
||||||
- **phase_specific:** true
|
- **phase_specific:** true
|
||||||
- **domain:** edge computing
|
- **domain:** edge computing
|
||||||
- **frameworks:** []
|
- **frameworks:** []
|
||||||
- **constraints:** ["latency-bound first", "resource-constrained examples", "trace to core", "10 P-rules per domain", "no runtime code", "proximity is the distinguishing constraint (D-061)", "offline-capability is non-negotiable"]
|
- **constraints:** ["latency-bound first", "resource-constrained examples", "trace to core", "10 P-rules per domain", "no runtime code", "proximity is the distinguishing constraint (D-061)", "offline-capability is non-negotiable"]
|
||||||
- **territory:** ["domains/edge/**", "examples/good/edge-offline-sync.md", "examples/bad/edge-sync-loop.md"]
|
- **territory:** ["domains/edge/**", "examples/good/edge-offline-sync.md", "examples/bad/edge-sync-loop.md"]
|
||||||
- **reason:** Per D-067: edge domain authoring (cdn, offline-first, iot, sync) benefits from a specialist persona with the proximity-is-correctness and offline-is-non-negotiable constraints. The distinguishing angle is proximity/location (D-061) — distinct from performance (generic), concurrency (in-process), and observability (generic telemetry). Active for v0.4 P1 (Edge Domain) only; consulted during P2 (Messaging) for the D-062 in-process-vs-cross-system boundary; removed after v0.4 completes; roster returns to 3 active personas.
|
- **reason:** Per D-067: edge domain authoring (cdn, offline-first, iot, sync) benefited from a specialist persona with the proximity-is-correctness and offline-is-non-negotiable constraints. The distinguishing angle is proximity/location (D-061) — distinct from performance (generic), concurrency (in-process), and observability (generic telemetry). Was active for v0.4 P1 (Edge Domain) only; consulted during P2 (Messaging) for the D-062 in-process-vs-cross-system boundary; reviewed edge content in P6 Wave 1; removed after v0.4 completed; roster returned to 3 active personas.
|
||||||
|
|
||||||
### languages-engineer (v0.4 — REMOVE after milestone completion)
|
### languages-engineer (v0.4 — REMOVED after milestone completion)
|
||||||
|
|
||||||
- **active:** true
|
- **active:** false
|
||||||
- **phase_specific:** true
|
- **phase_specific:** true
|
||||||
- **domain:** language-specific application
|
- **domain:** language-specific application
|
||||||
- **frameworks:** []
|
- **frameworks:** []
|
||||||
- **constraints:** ["trace to existing domain P-rules (D-063, D-066) — no new matrix rows", "per-language idioms", "fenced code examples in-language", "10 P-rules per domain does NOT apply to language docs", "manifest gains Languages section (D-065)"]
|
- **constraints:** ["trace to existing domain P-rules (D-063, D-066) — no new matrix rows", "per-language idioms", "fenced code examples in-language", "10 P-rules per domain does NOT apply to language docs", "manifest gains Languages section (D-065)"]
|
||||||
- **territory:** ["languages/**"]
|
- **territory:** ["languages/**"]
|
||||||
- **reason:** Per D-067: language-derived extension authoring (16 derived docs across TypeScript, Python, Go, Rust) benefits from a specialist persona enforcing the apply-don't-derive discipline (D-063) and the no-new-matrix-rows invariant (D-066). Languages are an application lens, not a principle source — the existing stubs' section-header-trace pattern is preserved, no P1–P10 numbering is introduced. Active for v0.4 P3 (Language-Derived Extension) only; removed after v0.4 completes; roster returns to 3 active personas.
|
- **reason:** Per D-067: language-derived extension authoring (16 derived docs across TypeScript, Python, Go, Rust) benefited from a specialist persona enforcing the apply-don't-derive discipline (D-063) and the no-new-matrix-rows invariant (D-066). Languages are an application lens, not a principle source — the existing stubs' section-header-trace pattern was preserved, no P1–P10 numbering was introduced. Was active for v0.4 P3 (Language-Derived Extension) only; reviewed language-derived content in P6 Wave 1; removed after v0.4 completed; roster returned to 3 active personas.
|
||||||
|
|
||||||
### platform-engineer (v0.3 — REMOVED after milestone completion)
|
### platform-engineer (v0.3 — REMOVED after milestone completion)
|
||||||
|
|
||||||
@@ -105,11 +106,11 @@ Mode: `warn` (per config.json `personas.territory_enforcement`).
|
|||||||
|
|
||||||
At `warn`, territory violations are logged but not blocked. This is appropriate for a docs project where tech-writer may touch `.ciagent/` files incidentally (e.g., updating ROADMAP status). Strict mode would be appropriate once territories stabilize.
|
At `warn`, territory violations are logged but not blocked. This is appropriate for a docs project where tech-writer may touch `.ciagent/` files incidentally (e.g., updating ROADMAP status). Strict mode would be appropriate once territories stabilize.
|
||||||
|
|
||||||
## v0.4 Persona Roster Summary (active during milestone)
|
## v0.4 Persona Roster Summary (post-milestone)
|
||||||
|
|
||||||
Active personas (3, span full milestone): lead-developer, tech-writer, domain-expert.
|
Active personas (3, span full milestone + ongoing): lead-developer, tech-writer, domain-expert.
|
||||||
Phase-specific personas (2, active during milestone): edge-engineer (P1 Edge Domain), languages-engineer (P3 Language-Derived Extension) — both removed after v0.4 completion per D-067.
|
Phase-specific personas (2, now removed): edge-engineer (P1 Edge Domain), languages-engineer (P3 Language-Derived Extension) — both deactivated after v0.4 completion per D-067; reviewed their authored content in P6 Wave 1 before removal.
|
||||||
Messaging (P2) is authored by tech-writer + domain-expert (the permanent roster) with edge-engineer consultation for the distributed-systems/async overlap (D-062 boundary), per D-067. The edge-engineer's distributed-systems lens survives the P1→P2 persona transition via a task must-have baked into the plan (the D-051 constraints-baked-into-task pattern from v0.3): "P1/P2 demonstrated edge + messaging boundary (D-062); delivery-semantics comparison table required (D-069)."
|
Messaging (P2) was authored by tech-writer + domain-expert (the permanent roster) with edge-engineer consultation for the distributed-systems/async overlap (D-062 boundary), per D-067. The edge-engineer's distributed-systems lens survived the P1→P2 persona transition via a task must-have baked into the plan (the D-051 constraints-baked-into-task pattern from v0.3).
|
||||||
Inactive default personas (3, unchanged): data-engineer, backend-engineer, frontend-engineer.
|
Inactive default personas (3, unchanged): data-engineer, backend-engineer, frontend-engineer.
|
||||||
Historical phase-specific personas (3, inactive): platform-engineer (v0.2), platform-engineer (v0.3 extended), ml-engineer (v0.3) — retained as historical record; "v0.4: still inactive" appended to their reasons.
|
Historical phase-specific personas (5, inactive): edge-engineer (v0.4), languages-engineer (v0.4), platform-engineer (v0.2), platform-engineer (v0.3 extended), ml-engineer (v0.3) — retained as historical record.
|
||||||
Peak active roster during v0.4: 5 (3 permanent + 2 phase-specific), matching v0.3's peak and avoiding persona-explosion risk. Roster returns to 3 active personas after v0.4 completion.
|
Peak active roster during v0.4: 5 (3 permanent + 2 phase-specific), matching v0.3's peak and avoiding persona-explosion risk. Roster has returned to 3 active personas for post-v0.4 maintenance.
|
||||||
@@ -239,32 +239,32 @@ All 35 requirements covered. 8 core principles, 11 domains, 110 domain principle
|
|||||||
|
|
||||||
| REQ-ID | Requirement | Priority | Phase | Status |
|
| REQ-ID | Requirement | Priority | Phase | Status |
|
||||||
|--------|-------------|----------|-------|--------|
|
|--------|-------------|----------|-------|--------|
|
||||||
| ATELIER-92 | `domains/edge/first-principles.md` — 10 edge principles (P1–P10) | P0 | 1 | pending |
|
| ATELIER-92 | `domains/edge/first-principles.md` — 10 edge principles (P1–P10) | P0 | 1 | covered |
|
||||||
| ATELIER-93 | `domains/edge/cdn.md` — CDN/edge-cache derived doc (with edge-cache vs origin decision matrix) | P1 | 1 | pending |
|
| ATELIER-93 | `domains/edge/cdn.md` — CDN/edge-cache derived doc (with edge-cache vs origin decision matrix) | P1 | 1 | covered |
|
||||||
| ATELIER-94 | `domains/edge/offline-first.md` — offline-first derived doc | P1 | 1 | pending |
|
| ATELIER-94 | `domains/edge/offline-first.md` — offline-first derived doc | P1 | 1 | covered |
|
||||||
| ATELIER-95 | `domains/edge/iot.md` — constrained-device/IoT derived doc | P1 | 1 | pending |
|
| ATELIER-95 | `domains/edge/iot.md` — constrained-device/IoT derived doc | P1 | 1 | covered |
|
||||||
| ATELIER-96 | `domains/edge/sync.md` — conflict/sync derived doc (CRDT/last-write-win decision matrix) | P1 | 1 | pending |
|
| ATELIER-96 | `domains/edge/sync.md` — conflict/sync derived doc (CRDT/last-write-win decision matrix) | P1 | 1 | covered |
|
||||||
| ATELIER-97 | `domains/messaging/first-principles.md` — 10 messaging principles (P1–P10) | P0 | 2 | pending |
|
| ATELIER-97 | `domains/messaging/first-principles.md` — 10 messaging principles (P1–P10) | P0 | 2 | covered |
|
||||||
| ATELIER-98 | `domains/messaging/queues.md` — queue derived doc (with at-least-once/at-most-once/exactly-once comparison) | P1 | 2 | pending |
|
| ATELIER-98 | `domains/messaging/queues.md` — queue derived doc (with at-least-once/at-most-once/exactly-once comparison) | P1 | 2 | covered |
|
||||||
| ATELIER-99 | `domains/messaging/pubsub.md` — pub/sub derived doc | P1 | 2 | pending |
|
| ATELIER-99 | `domains/messaging/pubsub.md` — pub/sub derived doc | P1 | 2 | covered |
|
||||||
| ATELIER-100 | `domains/messaging/streams.md` — stream processing derived doc | P1 | 2 | pending |
|
| ATELIER-100 | `domains/messaging/streams.md` — stream processing derived doc | P1 | 2 | covered |
|
||||||
| ATELIER-101 | `domains/messaging/delivery-semantics.md` — delivery semantics derived doc (idempotency, ordering, dead-letter) | P1 | 2 | pending |
|
| ATELIER-101 | `domains/messaging/delivery-semantics.md` — delivery semantics derived doc (idempotency, ordering, dead-letter) | P1 | 2 | covered |
|
||||||
| ATELIER-102 | `languages/typescript.md` extended + `languages/ts-types.md`, `ts-tooling.md`, `ts-async.md`, `ts-testing.md` — 4 TS derived docs | P1 | 3 | pending |
|
| ATELIER-102 | `languages/typescript.md` extended + `languages/ts-types.md`, `ts-tooling.md`, `ts-async.md`, `ts-testing.md` — 4 TS derived docs | P1 | 3 | covered |
|
||||||
| ATELIER-103 | `languages/python.md` extended + `languages/py-types.md`, `py-tooling.md`, `py-async.md`, `py-testing.md` — 4 Python derived docs | P1 | 3 | pending |
|
| ATELIER-103 | `languages/python.md` extended + `languages/py-types.md`, `py-tooling.md`, `py-async.md`, `py-testing.md` — 4 Python derived docs | P1 | 3 | covered |
|
||||||
| ATELIER-104 | `languages/go.md` extended + `languages/go-types.md`, `go-tooling.md`, `go-concurrency.md`, `go-testing.md` — 4 Go derived docs | P1 | 3 | pending |
|
| ATELIER-104 | `languages/go.md` extended + `languages/go-types.md`, `go-tooling.md`, `go-concurrency.md`, `go-testing.md` — 4 Go derived docs | P1 | 3 | covered |
|
||||||
| ATELIER-105 | `languages/rust.md` extended + `languages/rs-ownership.md`, `rs-tooling.md`, `rs-async.md`, `rs-testing.md` — 4 Rust derived docs | P1 | 3 | pending |
|
| ATELIER-105 | `languages/rust.md` extended + `languages/rs-ownership.md`, `rs-tooling.md`, `rs-async.md`, `rs-testing.md` — 4 Rust derived docs | P1 | 3 | covered |
|
||||||
| ATELIER-106 | Extend `matrix/principles-matrix.md` with 20 new P-rules → core C-rule mappings (10 per new domain; review check: row count per domain = 10, each row ≥1 C-rule) — post-v0.4: 19 domains, 190 P-rules | P0 | 4 | pending |
|
| ATELIER-106 | Extend `matrix/principles-matrix.md` with 20 new P-rules → core C-rule mappings (10 per new domain; review check: row count per domain = 10, each row ≥1 C-rule) — post-v0.4: 19 domains, 190 P-rules | P0 | 4 | covered |
|
||||||
| ATELIER-107 | Extend `matrix/domain-coverage.md` with edge + messaging (per-domain row schema) + add `languages/` coverage sub-table (language → derived docs → traced domains) | P1 | 4 | pending |
|
| ATELIER-107 | Extend `matrix/domain-coverage.md` with edge + messaging (per-domain row schema) + add `languages/` coverage sub-table (language → derived docs → traced domains) | P1 | 4 | covered |
|
||||||
| ATELIER-108 | Extend `review/agent-checklist.md` with edge + messaging trigger sections + a language-derived-docs section | P1 | 4 | pending |
|
| ATELIER-108 | Extend `review/agent-checklist.md` with edge + messaging trigger sections + a language-derived-docs section | P1 | 4 | covered |
|
||||||
| ATELIER-109 | Extend `review/peer-review-checklist.md` with edge + messaging + language-derived sections (parity with agent-checklist) | P1 | 4 | pending |
|
| ATELIER-109 | Extend `review/peer-review-checklist.md` with edge + messaging + language-derived sections (parity with agent-checklist) | P1 | 4 | covered |
|
||||||
| ATELIER-110 | Extend `review/anti-patterns.md` with edge + messaging violations incl. orphaned P-rule + deployable example artifact; pre-specify domain chaos anti-patterns (edge: sync conflict loop, TTL-less cache; messaging: shared subscription, blocking consumer) | P1 | 4 | pending |
|
| ATELIER-110 | Extend `review/anti-patterns.md` with edge + messaging violations incl. orphaned P-rule + deployable example artifact; pre-specify domain chaos anti-patterns (edge: sync conflict loop, TTL-less cache; messaging: shared subscription, blocking consumer) | P1 | 4 | covered |
|
||||||
| ATELIER-111 | Update `MANIFEST.md` to list new v0.4 domains + new `languages/` section (first-principles + derived docs per language) + post-v0.4 totals | P0 | 4 | pending |
|
| ATELIER-111 | Update `MANIFEST.md` to list new v0.4 domains + new `languages/` section (first-principles + derived docs per language) + post-v0.4 totals | P0 | 4 | covered |
|
||||||
| ATELIER-112 | `examples/good/edge-offline-sync.md` + `examples/good/messaging-idempotent-consumer.md` — 2 good examples (markdown with fenced code only) | P2 | 5 | pending |
|
| ATELIER-112 | `examples/good/edge-offline-sync.md` + `examples/good/messaging-idempotent-consumer.md` — 2 good examples (markdown with fenced code only) | P2 | 5 | covered |
|
||||||
| ATELIER-113 | `examples/bad/edge-sync-loop.md` + `examples/bad/messaging-shared-subscription.md` — 2 named bad examples (each cites the P-rule breached) | P2 | 5 | pending |
|
| ATELIER-113 | `examples/bad/edge-sync-loop.md` + `examples/bad/messaging-shared-subscription.md` — 2 named bad examples (each cites the P-rule breached) | P2 | 5 | covered |
|
||||||
| ATELIER-114 | Cross-links from new domains to existing devops/observability/data/concurrency/kubernetes/infrastructure-as-code domains; cross-links from language-derived docs to their traced domain docs (review check: every new derived doc ≥1 outbound cross-link to a MANIFEST-listed doc) | P1 | 5 | pending |
|
| ATELIER-114 | Cross-links from new domains to existing devops/observability/data/concurrency/kubernetes/infrastructure-as-code domains; cross-links from language-derived docs to their traced domain docs (review check: every new derived doc ≥1 outbound cross-link to a MANIFEST-listed doc) | P1 | 5 | covered |
|
||||||
| ATELIER-115 | Final review passes (all v0.4 phases reviewed, audit clean) | P0 | 6 | pending |
|
| ATELIER-115 | Final review passes (all v0.4 phases reviewed, audit clean) | P0 | 6 | covered |
|
||||||
| ATELIER-116 | Milestone v0.4 released (tag v0.3.6, merged to main) | P0 | 6 | pending |
|
| ATELIER-116 | Milestone v0.4 released (tag v0.3.6, merged to main) | P0 | 6 | covered |
|
||||||
| ATELIER-117 | Update `ARCHITECTURE.md` for v0.4 — extend the component tree with `edge/` + `messaging/` domain entries, add a `## v0.4 Domain Additions` section documenting the two new domains (boundary per D-061/D-062, derived docs, cross-link targets), and note the `languages/` derived-doc extension (D-063 apply-don't-derive). Closes architecture-tree drift: the file currently stops at v0.3 with no edge/messaging/languages-derived coverage (IDEATE-32). | P1 | 4 | pending |
|
| ATELIER-117 | Update `ARCHITECTURE.md` for v0.4 — extend the component tree with `edge/` + `messaging/` domain entries, add a `## v0.4 Domain Additions` section documenting the two new domains (boundary per D-061/D-062, derived docs, cross-link targets), and note the `languages/` derived-doc extension (D-063 apply-don't-derive). Closes architecture-tree drift: the file currently stops at v0.3 with no edge/messaging/languages-derived coverage (IDEATE-32). | P1 | 4 | covered |
|
||||||
|
|
||||||
## v0.4 Traceability Matrix
|
## v0.4 Traceability Matrix
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,285 @@
|
|||||||
|
# Atelier v0.4 — Final Review (P6)
|
||||||
|
|
||||||
|
## Reviewers: lead-developer, domain-expert, tech-writer, edge-engineer, languages-engineer
|
||||||
|
|
||||||
|
> Phase 6 Wave 1 — multi-persona review of all v0.4 content authored in P1–P5,
|
||||||
|
> plus the autonomous audit. The edge-engineer and languages-engineer personas
|
||||||
|
> review their territory then are removed (phase-specific). The lead-developer
|
||||||
|
> coordinates; domain-expert and tech-writer review messaging and verify
|
||||||
|
> traceability. Autonomy = full; P0 fixes auto-applied; P1+ flagged post-hoc.
|
||||||
|
|
||||||
|
## Findings by Persona
|
||||||
|
|
||||||
|
### edge-engineer (edge domain — D-061)
|
||||||
|
- Files reviewed: 5
|
||||||
|
(`domains/edge/first-principles.md`, `cdn.md`, `offline-first.md`, `iot.md`, `sync.md`)
|
||||||
|
- Boundary check: **PASS**
|
||||||
|
- Edge owns proximity/location/constraint/disconnection (D-061). Each doc
|
||||||
|
explicitly demarcates vs `domains/performance/` (generic measurement /
|
||||||
|
`performance/P4 Resource Bounds`, `performance/P5 Caching with Intent`),
|
||||||
|
`domains/observability/` (generic structured telemetry), and
|
||||||
|
`domains/concurrency/` (in-process primitives). The C4-Locality
|
||||||
|
discriminator is stated in first-principles §4: "performance's locality is
|
||||||
|
algorithmic; edge's locality is geographic." No generic-perf or
|
||||||
|
in-process-concurrency content leaked into edge.
|
||||||
|
- P-rule traces: **PASS — 10/10 trace to ≥1 C-rule**
|
||||||
|
- first-principles §2 Core Principle Trace table has all 10 P-rules with
|
||||||
|
explicit C-rule(s): P1→C4,C1 · P2→C1,C5 · P3→C8,C1 · P4→C1,C5 · P5→C1 ·
|
||||||
|
P6→C1,C3 · P7→C1,C5 · P8→C4,C6 · P9→C1,C8 · P10→C7,C5. Broad derivation
|
||||||
|
(7 of 8 C-rules); C2 is correctly noted as indirect.
|
||||||
|
- Decision matrices: **PASS — cdn, sync**
|
||||||
|
- `cdn.md` has "Edge-Cache vs Origin — Decision Matrix (D-069)" (6 rows:
|
||||||
|
serve-from-PoP, serve-stale-while-revalidate, fetch-fresh-miss,
|
||||||
|
origin-shield-fetch, purge-and-serve-fresh, serve-from-origin-bypass).
|
||||||
|
- `sync.md` has "CRDT vs Last-Write-Win — Decision Matrix (D-069)" (5 rows:
|
||||||
|
CvRDT, CmRDT, LWW-with-vector-clocks, LWW-with-wall-clock-only,
|
||||||
|
three-way-merge). Both carry the correctness-guarantee, cost, and
|
||||||
|
failure-mode columns.
|
||||||
|
- Cross-links: **PASS** (≥1 per derived doc)
|
||||||
|
- cdn.md → 1 (performance/frontend, security/input-validation, observability/metrics)
|
||||||
|
- offline-first.md → 5 (concurrency/patterns, observability/logging,
|
||||||
|
edge/sync, edge/cdn, testing/pyramid, testing/fixtures, observability/metrics)
|
||||||
|
- iot.md → 3 (security/secrets, messaging/queues, observability/metrics)
|
||||||
|
- sync.md → 2 (data/migrations, concurrency/patterns, observability/metrics)
|
||||||
|
- All one-directional outward per D-026 extended; edge↔messaging intra-v0.4
|
||||||
|
link (edge/iot → messaging/queues, MQTT QoS) resolved bidirectionally in
|
||||||
|
P5 per IDEATE-40 — verified: 4 refs in iot.md, 6 back-refs in
|
||||||
|
messaging/first-principles.md.
|
||||||
|
- Testing-pyramid mapping (IDEATE-38): **PASS**
|
||||||
|
- offline-first.md §"Offline Write-Queue and Conflict Detection Mapped to
|
||||||
|
the Testing Pyramid (IDEATE-38, ATELIER-94)" maps to all three tiers
|
||||||
|
(Unit: merge function; Integration: reconnect reconcile; E2e: partition
|
||||||
|
simulation) with the table + per-tier fenced sketches + cross-link to
|
||||||
|
`domains/testing/pyramid` and `domains/testing/fixtures`.
|
||||||
|
- P0 issues: **none**
|
||||||
|
- P1+ issues: **none**
|
||||||
|
|
||||||
|
### languages-engineer (language-derived — D-063)
|
||||||
|
- Files reviewed: 20 (4 stubs + 16 derived)
|
||||||
|
- Stubs: `typescript.md`, `python.md`, `go.md`, `rust.md`
|
||||||
|
- TS derived: `ts-types.md`, `ts-tooling.md`, `ts-async.md`, `ts-testing.md`
|
||||||
|
- Py derived: `py-types.md`, `py-tooling.md`, `py-async.md`, `py-testing.md`
|
||||||
|
- Go derived: `go-types.md`, `go-tooling.md`, `go-concurrency.md`, `go-testing.md`
|
||||||
|
- Rs derived: `rs-ownership.md`, `rs-tooling.md`, `rs-async.md`, `rs-testing.md`
|
||||||
|
- Apply-don't-derive check: **PASS — no new P-rules, no P1–P10 numbering**
|
||||||
|
- All 4 stubs retain their v0.1 section structure (0 `### P[0-9]` headers —
|
||||||
|
verified). All 16 derived docs introduce no new P-rules (0 new-P-rule
|
||||||
|
refs — verified). Each derived doc's blockquote header states
|
||||||
|
"introduces no new P-rules (D-063)". Every section header names the
|
||||||
|
traced domain P-rule(s) inline (e.g.,
|
||||||
|
`## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)`).
|
||||||
|
- Trace check: **PASS — every section traces to a domain P-rule**
|
||||||
|
- Spot-check confirms section headers name ≥1 domain P-rule AND the core
|
||||||
|
C-rule(s). Cross-References sections at the end of each derived doc list
|
||||||
|
the traced domain first-principles + sibling language docs.
|
||||||
|
- D-066 matrix invariant: **PASS** — matrix stays at 190 P-rows / 19 domains.
|
||||||
|
The `## Languages Coverage` sub-table in `matrix/domain-coverage.md` is
|
||||||
|
exactly 16 rows (one per derived doc); no new matrix rows were added.
|
||||||
|
- Runtime artifact check: **PASS — no standalone .ts/.py/.go/.rs**
|
||||||
|
- `find` for standalone runtime artifact files returns empty. All code is
|
||||||
|
fenced inside `.md` files (illustrative only, D-020). The
|
||||||
|
`review/anti-patterns.md` v0.4 Deployable Artifact Types table lists the
|
||||||
|
forbidden extensions explicitly.
|
||||||
|
- Cross-links to new + existing: **PASS**
|
||||||
|
- Every derived doc has a Cross-References section linking to BOTH new
|
||||||
|
(edge/messaging where relevant — e.g., ts-async → messaging/delivery-semantics,
|
||||||
|
py-async → messaging/queues, go-concurrency → messaging/queues,
|
||||||
|
rs-async → messaging/delivery-semantics) AND existing domains
|
||||||
|
(concurrency, errors, data, testing, devops, documentation, api).
|
||||||
|
- P0 issues: **none**
|
||||||
|
- P1+ issues: **none**
|
||||||
|
|
||||||
|
### tech-writer + domain-expert (messaging — D-062)
|
||||||
|
- Files reviewed: 5
|
||||||
|
(`domains/messaging/first-principles.md`, `queues.md`, `pubsub.md`,
|
||||||
|
`streams.md`, `delivery-semantics.md`)
|
||||||
|
- Boundary check: **PASS**
|
||||||
|
- Messaging owns cross-process/network-failure-model (D-062). Each doc
|
||||||
|
explicitly demarcates vs `domains/concurrency/` (in-process/crash-failure-
|
||||||
|
model): concurrency's bounded queue fails by OOM or thread crash;
|
||||||
|
messaging's broker-backed queue fails by network partition, broker
|
||||||
|
restart, or consumer crash-and-retry. The discriminator (failure model)
|
||||||
|
is stated in first-principles §4 and repeated per derived doc.
|
||||||
|
- P-rule traces: **PASS — 10/10**
|
||||||
|
- first-principles §2 Core Principle Trace table: P1→C1,C2 · P2→C1,C2 ·
|
||||||
|
P3→C1 · P4→C1,C2 · P5→C1,C5 · P6→C1,C8 · P7→C4,C6 · P8→C5,C7 · P9→C5,C6 ·
|
||||||
|
P10→C7,C1. Broad derivation (7 C-rules); C3 correctly noted as not
|
||||||
|
primary (messaging is a tradeoff domain).
|
||||||
|
- Comparison tables: **PASS — queues, streams ×2, delivery-semantics ×2**
|
||||||
|
- `queues.md` — "Delivery Semantics Comparison (D-069)" (at-most-once /
|
||||||
|
at-least-once / exactly-once with guarantee, latency cost, implementation
|
||||||
|
cost, when-it-fits).
|
||||||
|
- `streams.md` — "Stream-Platform Comparison (D-069)" (Kafka / Kinesis /
|
||||||
|
Pulsar / NATS JetStream across ordering, partitioning, replay/retention,
|
||||||
|
consumer groups, exactly-once, use-case fit, watch-outs).
|
||||||
|
- `streams.md` — "Consumer-Group Rebalance Strategies (IDEATE-41)" (eager /
|
||||||
|
sticky-incremental-cooperative / cooperative-no-revoke with mechanism,
|
||||||
|
partition stop-the-world cost, use-case fit).
|
||||||
|
- `delivery-semantics.md` — "Dead-Letter Strategy Comparison (D-069)"
|
||||||
|
(retry-count-limit / TTL-with-backoff / poison-queue / DLQ+alert with
|
||||||
|
when-it-applies, failure-visibility, operational-cost).
|
||||||
|
- `delivery-semantics.md` — fenced idempotency-key dedup-store example
|
||||||
|
(IDEATE-39 — not prose-only; the `DedupStore` class with TTL-bounded
|
||||||
|
`seen`/`mark`).
|
||||||
|
- Cross-links: **PASS** (≥1 per derived doc)
|
||||||
|
- queues.md → 6 (concurrency/patterns, observability/metrics,
|
||||||
|
messaging/delivery-semantics, messaging/pubsub, messaging/streams,
|
||||||
|
errors/patterns)
|
||||||
|
- pubsub.md → 4 (messaging/streams, messaging/queues, observability/metrics)
|
||||||
|
- streams.md → 4 (messaging/delivery-semantics, messaging/pubsub,
|
||||||
|
data/schema-design, observability/tracing, observability/metrics)
|
||||||
|
- delivery-semantics.md → 5 (concurrency/patterns, errors/patterns,
|
||||||
|
observability/metrics, messaging/queues, messaging/streams)
|
||||||
|
- All one-directional outward per D-026 extended; edge↔messaging
|
||||||
|
bidirectional link resolved in P5 (messaging/first-principles §4
|
||||||
|
acknowledges the edge/iot back-link without editing edge content).
|
||||||
|
- P0 issues: **none**
|
||||||
|
- P1+ issues: **none**
|
||||||
|
|
||||||
|
### domain-expert (traceability)
|
||||||
|
- 20 new P-rules trace to ≥1 C-rule: **PASS**
|
||||||
|
- Edge (10): all traced in `domains/edge/first-principles.md` §2 AND
|
||||||
|
`matrix/principles-matrix.md` `## Edge` section (10 rows). No orphans.
|
||||||
|
- Messaging (10): all traced in `domains/messaging/first-principles.md` §2
|
||||||
|
AND `matrix/principles-matrix.md` `## Messaging` section (10 rows). No orphans.
|
||||||
|
- The two trace tables (first-principles §2 and matrix) are consistent.
|
||||||
|
- 16 Languages Coverage rows match derived docs: **PASS**
|
||||||
|
- `matrix/domain-coverage.md` `## Languages Coverage` sub-table has exactly
|
||||||
|
16 rows (4 languages × 4 derived docs). Spot-check confirms the traced
|
||||||
|
domain P-rules column matches the derived docs' actual section-header
|
||||||
|
traces (e.g., ts-types.md row: `Data P7, API P1, Errors P1` matches the
|
||||||
|
doc's headers naming Data P7, API P1, Errors P1).
|
||||||
|
- Matrix row count: **190 — PASS**
|
||||||
|
- `grep -c "^| P[0-9]" matrix/principles-matrix.md` = 190 exactly.
|
||||||
|
- Per G-014: 170 pre-existing (v0.1 + v0.2 + v0.3) + 20 new (10 edge + 10
|
||||||
|
messaging) = 190. NOT 171 + 20 (the v0.3 baseline was correctly 170, not 171).
|
||||||
|
- Edge section = 10 P-rows; Messaging section = 10 P-rows; no language rows
|
||||||
|
(D-066 invariant holds).
|
||||||
|
|
||||||
|
## P0 Fixes Applied (auto-applied in this phase)
|
||||||
|
|
||||||
|
1. **MANIFEST.md — 4 v0.4 examples unlisted (MANIFEST reconstruction drift).**
|
||||||
|
The 4 v0.4 example files existed on disk but were NOT listed in
|
||||||
|
`MANIFEST.md`'s Examples section:
|
||||||
|
- `examples/good/edge-offline-sync.md`
|
||||||
|
- `examples/good/messaging-idempotent-consumer.md`
|
||||||
|
- `examples/bad/edge-sync-loop.md`
|
||||||
|
- `examples/bad/messaging-shared-subscription.md`
|
||||||
|
Per the MANIFEST contract ("unlisted = not part of framework") and the
|
||||||
|
ATELIER-115 audit MANIFEST reconstruction test, this was a P0. Fix: added
|
||||||
|
the 4 rows to the Examples table with `✓` status + v0.4 notes, updated the
|
||||||
|
directory count notes (good: 8→10, bad: 7→9), and extended the closing
|
||||||
|
note to mention the 4 v0.4 examples. The 4 example files were verified
|
||||||
|
present on disk; the anti-patterns they instantiate are catalogued in
|
||||||
|
`review/anti-patterns.md` (v0.4 Chaos Anti-Patterns table + the
|
||||||
|
`edge-sync-loop` and `messaging-shared-subscription` named examples).
|
||||||
|
|
||||||
|
## P1+ Issues Flagged for Post-Hoc
|
||||||
|
|
||||||
|
none — review clean. The v0.4 content (edge, messaging, languages-derived)
|
||||||
|
is consistent with the D-061/D-062/D-063 boundaries, the traceability
|
||||||
|
invariants (G-014, D-066), the cross-link discipline (D-026 extended,
|
||||||
|
IDEATE-40), and the manifest contract. The only finding was the P0
|
||||||
|
MANIFEST drift, which has been fixed.
|
||||||
|
|
||||||
|
## Overall Review Verdict: **PASS**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Audit (ATELIER-115)
|
||||||
|
|
||||||
|
### Reconstruction test: **PASS**
|
||||||
|
- git log v0.3.0..HEAD: 11 commits, all with `---ci---` blocks containing
|
||||||
|
`project: atelier` (P0 pre-exec, P1 edge, P2 messaging, P3 languages,
|
||||||
|
P4 matrix+review+manifest+architecture, P5 examples+crosslinks + 5 ship
|
||||||
|
commits). Verified: 0 v0.4 commits missing the `project: atelier` field.
|
||||||
|
- CHECKPOINT.json: phase 5, stage complete — matches current state (P5
|
||||||
|
shipped at v0.3.5, release 487; P6 is the current pending phase per
|
||||||
|
ROADMAP). Note: CHECKPOINT.json will be advanced to phase 6 by the
|
||||||
|
orchestrator at ship (Wave 2); this is expected — the audit verifies
|
||||||
|
reconstruction from commits + ROADMAP + CHECKPOINT, all of which agree.
|
||||||
|
|
||||||
|
### Branch hygiene: **PASS**
|
||||||
|
- `milestone/v0.4-edge-quantum-langs`: exists (local + remote
|
||||||
|
`remotes/origin/milestone/v0.4-edge-quantum-langs`).
|
||||||
|
- `phase/01-05`: deleted post-merge — `git branch -a | grep -E "phase/0[1-5]"`
|
||||||
|
returns empty locally and on remote (phase/01..05 branches were merged and
|
||||||
|
removed per the phase-branch strategy).
|
||||||
|
- `phase/06-final-review-ship`: active (checked out, working tree clean
|
||||||
|
except the P0 MANIFEST fix which is staged for the orchestrator's commit).
|
||||||
|
|
||||||
|
### Commit discipline: **PASS**
|
||||||
|
- Total v0.4 commits (v0.3.0..HEAD): 11
|
||||||
|
- Commits with `---ci---` block: 11 (100%)
|
||||||
|
- Coverage: 11/11 = 100%
|
||||||
|
- Each `---ci---` block includes `project: atelier` (verified by scanning
|
||||||
|
every v0.4 commit body). The v0.4 milestone is fully `---ci---`-compliant.
|
||||||
|
- (Repo-wide: 38 of 46 all-branches commits carry `---ci---` blocks; the
|
||||||
|
non-ci commits are the pre-framework chore commits and the v0.1 legacy
|
||||||
|
commits before the ci-block convention was established — out of v0.4
|
||||||
|
scope.)
|
||||||
|
|
||||||
|
### MANIFEST reconstruction test: **PASS** (after P0 fix)
|
||||||
|
- Listed docs existing on disk: 33/33 paths checked (the 4
|
||||||
|
"missing-on-disk" detections are intentional placeholders —
|
||||||
|
`domains/<relevant>/first-principles.md` and `domains/<relevant>/<topic>.md`
|
||||||
|
in the Reading-Order section, and the `examples/good/` +
|
||||||
|
`examples/bad/` directory rows — not real files).
|
||||||
|
- Existing docs listed in MANIFEST: all 129 on-disk `.md` files under
|
||||||
|
`core/`, `domains/`, `languages/`, `matrix/`, `review/`, `examples/` are
|
||||||
|
listed. The MANIFEST uses a summarized format (Domains table by domain
|
||||||
|
name + derived-doc basenames; Languages table by language name + derived-
|
||||||
|
doc basenames; Examples table by full path; Cross-Cutting table by full
|
||||||
|
path). Verified by basename match for domains/languages + full-path match
|
||||||
|
for examples/core/matrix/review. The 4 v0.4 examples were the only drift;
|
||||||
|
the P0 fix added them.
|
||||||
|
- Drift: **none after P0 fix** (pre-fix: 4 v0.4 examples unlisted → fixed).
|
||||||
|
|
||||||
|
### Matrix row-count test: **PASS**
|
||||||
|
- `grep -c "^| P[0-9]" matrix/principles-matrix.md` = **190** (expected 190).
|
||||||
|
- Per G-014: 170 pre-existing (110 v0.1 + 20 v0.2 + 40 v0.3) + 20 new
|
||||||
|
(10 edge + 10 messaging) = 190. The v0.3 baseline was correctly 170 (NOT
|
||||||
|
171 — the v0.3 audit confirmed this; v0.4 builds on the correct baseline).
|
||||||
|
- Edge section: 10 P-rows. Messaging section: 10 P-rows. No language rows
|
||||||
|
(D-066 invariant — languages apply, not derive; matrix stays at 19
|
||||||
|
domains × 10 P-rules = 190).
|
||||||
|
|
||||||
|
### ARCHITECTURE.md component-tree test: **PASS**
|
||||||
|
- `domains/` on disk (19): ai-ml, api, compliance, concurrency, data,
|
||||||
|
devops, documentation, edge, errors, gitops-operators, i18n,
|
||||||
|
infrastructure-as-code, kubernetes, messaging, observability, performance,
|
||||||
|
security, testing, uiux.
|
||||||
|
- `domains/` in ARCHITECTURE.md: all 19 present (v0.1: 11 summarized as
|
||||||
|
`... (v0.1: 11 domains)`; v0.2: infrastructure-as-code, kubernetes
|
||||||
|
explicit; v0.3: gitops-operators, ai-ml, i18n, compliance explicit; v0.4:
|
||||||
|
edge, messaging explicit). The summary form for v0.1 is intentional and
|
||||||
|
consistent with the v0.2/v0.3 audit precedents.
|
||||||
|
- `languages/` on disk (20): 4 stubs (typescript.md, python.md, go.md,
|
||||||
|
rust.md) + 16 derived (ts-*, py-*, go-*, rs-*).
|
||||||
|
- `languages/` in ARCHITECTURE.md: all 20 present (4 stubs explicit + 4
|
||||||
|
derived-doc globs `ts-*.md`, `py-*.md`, `go-*.md`, `rs-*.md` each
|
||||||
|
enumerating the 4 derived docs, plus the "Derived docs by language"
|
||||||
|
listing in the v0.4 Language-Derived Extension section).
|
||||||
|
- Drift: **none**.
|
||||||
|
|
||||||
|
### Audit Verdict: **PASS**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
- Review verdict: **PASS** (clean across all 5 personas; 1 P0 fixed).
|
||||||
|
- Audit verdict: **PASS** (6 audit checks all pass after P0 fix).
|
||||||
|
- P0 fixes applied: **1** (MANIFEST.md — 4 v0.4 examples unlisted → listed).
|
||||||
|
- P1+ flagged: **0** (review clean).
|
||||||
|
- Matrix row count: **190** (confirmed — 170 pre-existing + 20 new, NOT
|
||||||
|
171 + 20).
|
||||||
|
- MANIFEST drift: **no** (after P0 fix; was: 4 v0.4 examples unlisted).
|
||||||
|
- Ready-to-ship: **yes** — review and audit both PASS; the milestone is
|
||||||
|
ready for Wave 2 (ship) by the orchestrator.
|
||||||
|
|
||||||
|
The v0.4 milestone (edge + messaging + language-derived extension) is
|
||||||
|
coherent, traceable, boundary-respecting, manifest-complete, and
|
||||||
|
audit-clean. The single P0 (MANIFEST examples drift) has been auto-applied;
|
||||||
|
the working tree carries only that one-line fix, staged for the
|
||||||
|
orchestrator's ship commit.
|
||||||
@@ -119,10 +119,10 @@ NFR milestone: no separate minor tag. The final patch (v0.2.6) IS the v0.3 deliv
|
|||||||
|
|
||||||
## Future Milestones
|
## Future Milestones
|
||||||
|
|
||||||
- **v0.4** (ACTIVE): `domains/edge/`, `domains/messaging/`, `languages/`-derived docs (typescript/python/go/rust → 4 derived docs each).
|
- **v0.4** (COMPLETE): `domains/edge/`, `domains/messaging/`, `languages/`-derived docs (typescript/python/go/rust → 4 derived docs each). Shipped v0.3.6.
|
||||||
- **v0.5** (candidates): `domains/quantum/`, tooling adapters (linters), translation/localization of framework docs.
|
- **v0.5** (candidates): `domains/quantum/`, tooling adapters (linters), translation/localization of framework docs.
|
||||||
|
|
||||||
## Milestone: v0.4 — Edge + Messaging + Language-Derived Docs (ACTIVE)
|
## Milestone: v0.4 — Edge + Messaging + Language-Derived Docs (COMPLETE)
|
||||||
|
|
||||||
**Milestone type:** NFR (all phases produce docs — no `feat` code)
|
**Milestone type:** NFR (all phases produce docs — no `feat` code)
|
||||||
**Tag line:** v0.3.x (previous minor from v0.4)
|
**Tag line:** v0.3.x (previous minor from v0.4)
|
||||||
@@ -131,12 +131,12 @@ NFR milestone: no separate minor tag. The final patch (v0.2.6) IS the v0.3 deliv
|
|||||||
| Phase | Name | Type | Status | Key Deliverables |
|
| Phase | Name | Type | Status | Key Deliverables |
|
||||||
|-------|------|------|--------|------------------|
|
|-------|------|------|--------|------------------|
|
||||||
| 0 | Pre-Execution | docs | complete | Spec, clarify, research, ideate, plan, PERSONAS.md (adds edge-engineer + languages-engineer phase-specific personas) — shipped v0.3.0 |
|
| 0 | Pre-Execution | docs | complete | Spec, clarify, research, ideate, plan, PERSONAS.md (adds edge-engineer + languages-engineer phase-specific personas) — shipped v0.3.0 |
|
||||||
| 1 | Edge Domain | docs | pending | domains/edge/{first-principles, cdn, offline-first, iot, sync}.md |
|
| 1 | Edge Domain | docs | complete | domains/edge/{first-principles, cdn, offline-first, iot, sync}.md — shipped v0.3.1 |
|
||||||
| 2 | Messaging Domain | docs | pending | domains/messaging/{first-principles, queues, pubsub, streams, delivery-semantics}.md |
|
| 2 | Messaging Domain | docs | complete | domains/messaging/{first-principles, queues, pubsub, streams, delivery-semantics}.md — shipped v0.3.2 |
|
||||||
| 3 | Language-Derived Extension | docs | pending | languages/ × 4 → first-principles + 4 derived docs each (16 derived docs) |
|
| 3 | Language-Derived Extension | docs | complete | languages/ × 4 → first-principles + 4 derived docs each (16 derived docs) — shipped v0.3.3 |
|
||||||
| 4 | Matrix + Review Integration | docs | pending | matrix/principles-matrix.md (+20 mappings, 170→190), matrix/domain-coverage.md (+ languages/ sub-table), review/{agent-checklist, peer-review-checklist, anti-patterns}.md, MANIFEST.md (languages/ section) |
|
| 4 | Matrix + Review Integration | docs | complete | matrix/principles-matrix.md (+20 mappings, 170→190), matrix/domain-coverage.md (+ languages/ sub-table), review/{agent-checklist, peer-review-checklist, anti-patterns}.md, MANIFEST.md (languages/ section), ARCHITECTURE.md — shipped v0.3.4 |
|
||||||
| 5 | Examples + Cross-Links | docs | pending | examples/good + examples/bad for 2 domains, cross-links to devops/observability/data/concurrency/kubernetes/infrastructure-as-code + language→domain links |
|
| 5 | Examples + Cross-Links | docs | complete | examples/good + examples/bad for 2 domains, cross-links to devops/observability/data/concurrency/kubernetes/infrastructure-as-code + language→domain links — shipped v0.3.5 |
|
||||||
| 6 | Final Review + Ship | docs | pending | Review passed, audit clean, milestone merged to main, tag v0.3.6 |
|
| 6 | Final Review + Ship | docs | complete | Review passed, audit clean, milestone merged to main, tag v0.3.6 — shipped v0.3.6 (IS the v0.4 release) |
|
||||||
|
|
||||||
## v0.4 Phase Tag Mapping
|
## v0.4 Phase Tag Mapping
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
"slug": "atelier",
|
"slug": "atelier",
|
||||||
"name": "Atelier",
|
"name": "Atelier",
|
||||||
"milestone": "v0.4",
|
"milestone": "v0.4",
|
||||||
"status": "active"
|
"status": "complete"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"active_project": "atelier",
|
"active_project": "atelier",
|
||||||
|
|||||||
+28
-9
@@ -40,15 +40,30 @@
|
|||||||
| AI / ML | ✓ | data-versioning, model-evaluation, serving, monitoring-drift |
|
| AI / ML | ✓ | data-versioning, model-evaluation, serving, monitoring-drift |
|
||||||
| i18n | ✓ | locale-resources, formatting, rtl-bidi, testing-i18n |
|
| i18n | ✓ | locale-resources, formatting, rtl-bidi, testing-i18n |
|
||||||
| Compliance | ✓ | audit-logs, data-retention, policy-as-code, evidence |
|
| Compliance | ✓ | audit-logs, data-retention, policy-as-code, evidence |
|
||||||
|
| Edge | ✓ | cdn, offline-first, iot, sync |
|
||||||
|
| Messaging | ✓ | queues, pubsub, streams, delivery-semantics |
|
||||||
|
|
||||||
|
## Languages
|
||||||
|
|
||||||
|
> Language docs apply (not derive) domain principles. They introduce
|
||||||
|
> no new P-rules and no new matrix rows (D-063, D-066). First-
|
||||||
|
> principles stubs from v0.1; derived docs added in v0.4.
|
||||||
|
|
||||||
|
| Language | First Principles | Derived Documents |
|
||||||
|
|----------|------------------|-------------------|
|
||||||
|
| TypeScript | ✓ (v0.1 stub) | ts-types, ts-tooling, ts-async, ts-testing |
|
||||||
|
| Python | ✓ (v0.1 stub) | py-types, py-tooling, py-async, py-testing |
|
||||||
|
| Go | ✓ (v0.1 stub) | go-types, go-tooling, go-concurrency, go-testing |
|
||||||
|
| Rust | ✓ (v0.1 stub) | rs-ownership, rs-tooling, rs-async, rs-testing |
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|
||||||
> Examples are illustrative markdown with fenced code only (no standalone runtime artifacts per D-020 / D-025). The `examples/` directory listing closes the v0.2 ESC-002 drift (IDEATE-17, ATELIER-91). P5 authored the v0.3 examples and promoted all entries from `pending` to `✓` (verified — every listed file exists).
|
> Examples are illustrative markdown with fenced code only (no standalone runtime artifacts per D-020 / D-025). The `examples/` directory listing closes the v0.2 ESC-002 drift (IDEATE-17, ATELIER-91). P5 authored the v0.3 examples and promoted all entries from `pending` to `✓` (verified — every listed file exists). P5 also authored the 4 v0.4 examples (edge + messaging chaos anti-patterns + good examples) and listed them here.
|
||||||
|
|
||||||
| Path | Status | Notes |
|
| Path | Status | Notes |
|
||||||
|------|--------|-------|
|
|------|--------|-------|
|
||||||
| `examples/good/` | ✓ | Good-example directory — 8 examples (v0.1 + v0.2 + v0.3) |
|
| `examples/good/` | ✓ | Good-example directory — 10 examples (v0.1 + v0.2 + v0.3 + v0.4) |
|
||||||
| `examples/bad/` | ✓ | Bad-example directory — 7 examples (v0.1 + v0.2 + v0.3) |
|
| `examples/bad/` | ✓ | Bad-example directory — 9 examples (v0.1 + v0.2 + v0.3 + v0.4) |
|
||||||
| `examples/good/api-endpoint.md` | ✓ | v0.1 example — good REST endpoint |
|
| `examples/good/api-endpoint.md` | ✓ | v0.1 example — good REST endpoint |
|
||||||
| `examples/good/react-component.md` | ✓ | v0.1 example — good React component |
|
| `examples/good/react-component.md` | ✓ | v0.1 example — good React component |
|
||||||
| `examples/good/db-schema.md` | ✓ | v0.1 example — good DB schema |
|
| `examples/good/db-schema.md` | ✓ | v0.1 example — good DB schema |
|
||||||
@@ -64,15 +79,19 @@
|
|||||||
| `examples/good/ai-ml-reproducibility.md` | ✓ | v0.3 example — good reproducible training run |
|
| `examples/good/ai-ml-reproducibility.md` | ✓ | v0.3 example — good reproducible training run |
|
||||||
| `examples/bad/i18n-string-concat.md` | ✓ | v0.3 example — bad i18n string concat |
|
| `examples/bad/i18n-string-concat.md` | ✓ | v0.3 example — bad i18n string concat |
|
||||||
| `examples/bad/compliance-audit-log.md` | ✓ | v0.3 example — bad audit log (P1 + P9 breaches) |
|
| `examples/bad/compliance-audit-log.md` | ✓ | v0.3 example — bad audit log (P1 + P9 breaches) |
|
||||||
|
| `examples/good/edge-offline-sync.md` | ✓ | v0.4 example — good edge offline-first + sync reconcile (CRDT-vs-LWW decision matrix) |
|
||||||
|
| `examples/good/messaging-idempotent-consumer.md` | ✓ | v0.4 example — good idempotent consumer with dedup store + DLQ |
|
||||||
|
| `examples/bad/edge-sync-loop.md` | ✓ | v0.4 example — bad edge sync livelock (`edge-sync-loop` chaos anti-pattern, ATELIER-110) |
|
||||||
|
| `examples/bad/messaging-shared-subscription.md` | ✓ | v0.4 example — bad messaging shared subscription (`messaging-shared-subscription` chaos anti-pattern, ATELIER-110) |
|
||||||
|
|
||||||
> **Note:** The `examples/` section was established in P4 with entries pre-listed as `pending P5`. P5 authored the 4 v0.3 examples and promoted all entries to `✓` after verifying every listed file exists on disk. The manifest remains authoritative — unlisted = not part of framework.
|
> **Note:** The `examples/` section was established in P4 with entries pre-listed as `pending P5`. P5 authored the 4 v0.3 examples and the 4 v0.4 examples, and promoted all entries to `✓` after verifying every listed file exists on disk. The manifest remains authoritative — unlisted = not part of framework.
|
||||||
|
|
||||||
## Cross-Cutting
|
## Cross-Cutting
|
||||||
|
|
||||||
| Document | Purpose |
|
| Document | Purpose |
|
||||||
|-----------------------------------|----------------------------------|
|
|-----------------------------------|----------------------------------|
|
||||||
| `matrix/principles-matrix.md` | Maps domain → core principles (17 domains, 170 P-rules post-v0.3) |
|
| `matrix/principles-matrix.md` | Maps domain → core principles (19 domains, 190 P-rules post-v0.4) |
|
||||||
| `matrix/domain-coverage.md` | Maps core → domains; per-domain coverage (incl. v0.3 Core Principle Coverage) |
|
| `matrix/domain-coverage.md` | Maps core → domains; per-domain coverage (incl. v0.4 Core Principle Coverage + Languages Coverage sub-table) |
|
||||||
| `review/agent-checklist.md` | Pre-completion agent checklist (incl. IaC + k8s + gitops + ai-ml + i18n + compliance triggers) |
|
| `review/agent-checklist.md` | Pre-completion agent checklist (incl. edge + messaging + language-derived triggers) |
|
||||||
| `review/peer-review-checklist.md` | Human peer-review checklist (incl. IaC + k8s + gitops + ai-ml + i18n + compliance sections) |
|
| `review/peer-review-checklist.md` | Human peer-review checklist (incl. edge + messaging + language-derived sections) |
|
||||||
| `review/anti-patterns.md` | Catalog of violations (incl. IaC + k8s + gitops + ai-ml + i18n + compliance + v0.3 chaos anti-patterns) |
|
| `review/anti-patterns.md` | Catalog of violations (incl. edge + messaging + language-derived + v0.4 chaos anti-patterns + .ts/.py/.go/.rs artifact types) |
|
||||||
@@ -0,0 +1,379 @@
|
|||||||
|
# Delivery Semantics — Derived Rules
|
||||||
|
|
||||||
|
> Derives from `domains/messaging/first-principles.md`. Applies P4
|
||||||
|
> (Delivery Semantics are Explicit), P3 (Consumers are Idempotent),
|
||||||
|
> P2 (Ordering is a Property, Not an Assumption), P5 (Dead-Letter
|
||||||
|
> Handling is Defined), and P6 (Backpressure is Bounded) primarily,
|
||||||
|
> with P10 (DLQ depth as an alert). For the dead-letter strategy
|
||||||
|
> decision, see the comparison table below. Includes a fenced
|
||||||
|
> idempotency-key dedup-store example (IDEATE-39). Cross-links
|
||||||
|
> `domains/concurrency/patterns` for the in-process retry/backoff
|
||||||
|
> analog, `domains/errors/patterns` for errors as data for message
|
||||||
|
> failures, and `domains/observability/metrics` for DLQ depth as an
|
||||||
|
> alert.
|
||||||
|
|
||||||
|
## The Three Delivery Semantics (P4 Delivery Semantics are Explicit)
|
||||||
|
|
||||||
|
- **At-most-once**: a message is delivered 0 or 1 times; loss is
|
||||||
|
possible, duplication is not. The producer fires-and-forgets; the
|
||||||
|
broker does not ack; the consumer does not dedup. Lowest latency,
|
||||||
|
lowest implementation cost, lossy. Fits telemetry where a dropped
|
||||||
|
sample is acceptable (MQTT QoS 0, fire-and-forget metrics).
|
||||||
|
- **At-least-once**: a message is delivered 1 or more times;
|
||||||
|
duplication is possible, loss is not. The producer sends and
|
||||||
|
waits for a broker ack; the consumer processes and acks; a crash
|
||||||
|
before the consumer's ack triggers redelivery. The consumer must
|
||||||
|
be idempotent (P3). The default for side-effecting operations
|
||||||
|
(orders, payments, commands). The vast majority of broker-backed
|
||||||
|
queues (SQS standard, RabbitMQ ack, MQTT QoS 1).
|
||||||
|
- **Exactly-once**: a message is delivered exactly 1 time; no
|
||||||
|
loss, no duplication. In practice, this is at-least-once plus
|
||||||
|
idempotency (P3), or a transactional consume-process-produce loop
|
||||||
|
(P4 — see `domains/messaging/streams.md`). Jepsen analyses verify
|
||||||
|
broker claims: "exactly-once" requires independent verification;
|
||||||
|
the durable engineering practice is at-least-once with idempotent
|
||||||
|
consumers (P3), which collapses to exactly-once under correct
|
||||||
|
dedup.
|
||||||
|
- The semantic is declared per channel (P4), not emergent. An
|
||||||
|
unstated semantic is a defect: the consumer guesses, and the
|
||||||
|
guess is wrong under the first failure. See
|
||||||
|
`domains/messaging/queues.md` for the three-semantics comparison
|
||||||
|
table (semantics, latency cost, implementation cost, when each
|
||||||
|
fits).
|
||||||
|
|
||||||
|
## Idempotency (P3 Consumers are Idempotent)
|
||||||
|
|
||||||
|
- Idempotency is the correctness property that makes at-least-once
|
||||||
|
safe. A consumer that processes the same message twice has the
|
||||||
|
same effect as processing it once. The mechanism is the
|
||||||
|
idempotency key: a per-message unique identifier the consumer
|
||||||
|
uses to dedup redeliveries.
|
||||||
|
- The idempotency key is per-message, not per-producer or
|
||||||
|
per-session. A consumer that dedups by producer alone drops
|
||||||
|
distinct messages issued in the same window. Use a UUID per
|
||||||
|
message, or a deterministic key derived from the message content
|
||||||
|
(e.g., `(entity, operation, version)`).
|
||||||
|
- The dedup store is bounded (P6 — Backpressure is Bounded): a
|
||||||
|
dedup store that grows without bound is a memory leak. Use a TTL
|
||||||
|
window longer than the broker's max-redelivery window, or a
|
||||||
|
bounded LRU. The TTL is the P6 bound: a key seen within the TTL
|
||||||
|
is a redelivery; a key older than the TTL is expired (the broker
|
||||||
|
has given up redelivering it).
|
||||||
|
|
||||||
|
## Idempotency-Key Dedup Store (IDEATE-39, P3 — fenced example)
|
||||||
|
|
||||||
|
The dedup store is the concrete mechanism that makes a consumer
|
||||||
|
idempotent under at-least-once delivery. This is the fenced
|
||||||
|
example required by IDEATE-39 (parallel to the v0.3 IDEATE-29
|
||||||
|
signed-attestation fenced example): it is NOT prose-only — the
|
||||||
|
consumer-with-dedup-store demonstrates P3 concretely.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Idempotent consumer with a dedup store (P3 Consumers are
|
||||||
|
# Idempotent, P6 Backpressure is Bounded — the dedup store is
|
||||||
|
# TTL-bounded). The consumer dedups by idempotency key before
|
||||||
|
# processing; a redelivered message is a no-op, not a double-apply.
|
||||||
|
|
||||||
|
import time
|
||||||
|
|
||||||
|
# P6: the dedup store is bounded by a TTL window. The TTL must
|
||||||
|
# exceed the broker's max-redelivery window; beyond the TTL, the
|
||||||
|
# key is expired (the broker has given up). A dedup store with no
|
||||||
|
# TTL is a memory leak (P6 violation).
|
||||||
|
DEDUP_TTL_SECONDS = 24 * 3600 # longer than max-redelivery window
|
||||||
|
|
||||||
|
class DedupStore:
|
||||||
|
"""A TTL-bounded idempotency-key dedup store (P3, P6).
|
||||||
|
|
||||||
|
seen(key): True if the key was processed within the TTL.
|
||||||
|
mark(key): Record the key as processed (with a timestamp).
|
||||||
|
"""
|
||||||
|
def __init__(self, backend):
|
||||||
|
# backend is a Redis, a DB, or an in-process LRU. The
|
||||||
|
# backend must be shared across consumer instances if the
|
||||||
|
# subscription is shared (P2/P3 — see domains/messaging/
|
||||||
|
# pubsub.md on shared vs independent subscriptions).
|
||||||
|
self.backend = backend
|
||||||
|
|
||||||
|
def seen(self, key: str) -> bool:
|
||||||
|
ts = self.backend.get(key)
|
||||||
|
if ts is None:
|
||||||
|
return False
|
||||||
|
if time.time() - ts > DEDUP_TTL_SECONDS:
|
||||||
|
# P6: expired. The broker has given up redelivering;
|
||||||
|
# this key is no longer a redelivery signal.
|
||||||
|
self.backend.delete(key)
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
def mark(self, key: str):
|
||||||
|
self.backend.set(key, time.time(), ttl=DEDUP_TTL_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
# The idempotent consumer: dedup before process, mark after
|
||||||
|
# process, ack after mark. A crash before mark re-processes (the
|
||||||
|
# dedup store does not have the key); a crash before ack
|
||||||
|
# redelivers (the broker did not see the ack) and the dedup store
|
||||||
|
# makes the redelivery a no-op (P3).
|
||||||
|
def consume_idempotent(broker, dedup: DedupStore, process):
|
||||||
|
for message in broker.receive():
|
||||||
|
# P3: dedup BEFORE process. A redelivered message is a
|
||||||
|
# no-op, not a double-apply.
|
||||||
|
if dedup.seen(message["idempotencyKey"]):
|
||||||
|
broker.ack(message) # already processed; skip
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
process(message["payload"])
|
||||||
|
# P3: mark AFTER process succeeds. A crash between
|
||||||
|
# process and mark re-processes (acceptable: the
|
||||||
|
# process must be idempotent OR the mark must be
|
||||||
|
# transactional with the process — see below).
|
||||||
|
dedup.mark(message["idempotencyKey"])
|
||||||
|
broker.ack(message)
|
||||||
|
except TransientError as exc:
|
||||||
|
# P6: bounded retry with backoff. Nack for redelivery;
|
||||||
|
# the broker redelivers after exponential backoff.
|
||||||
|
broker.nack(message, delay=backoff(message["attempt"]))
|
||||||
|
except PoisonError as exc:
|
||||||
|
# P5: poison message. Route to DLQ, do NOT retry
|
||||||
|
# forever (see the DLQ routing rule below).
|
||||||
|
route_to_dlq(broker, message, exc)
|
||||||
|
broker.ack(message) # remove from the origin queue
|
||||||
|
```
|
||||||
|
|
||||||
|
- The order `process → mark → ack` gives at-least-once with
|
||||||
|
idempotent dedup: a crash before `mark` re-processes (the dedup
|
||||||
|
store does not have the key), and a crash before `ack`
|
||||||
|
redelivers (the dedup store makes the redelivery a no-op). If
|
||||||
|
`process` is not itself idempotent, the `process → mark` window
|
||||||
|
must be transactional (e.g., process and mark in one DB
|
||||||
|
transaction) — otherwise a crash in the window double-applies.
|
||||||
|
- For a non-idempotent `process` (e.g., a payment that must not
|
||||||
|
double-charge), use a transactional dedup: process and mark in
|
||||||
|
one DB transaction, so the mark commits iff the process
|
||||||
|
commits. This is the "exactly-once via idempotency" pattern
|
||||||
|
(P4): at-least-once delivery plus a transactional
|
||||||
|
process-and-mark collapses to exactly-once under correct
|
||||||
|
transactional semantics.
|
||||||
|
|
||||||
|
## Ordering (P2 Ordering is a Property, Not an Assumption)
|
||||||
|
|
||||||
|
- The delivery semantic interacts with ordering (P2). At-least-once
|
||||||
|
with per-partition ordering: a redelivery within a partition
|
||||||
|
preserves order (the redelivered message re-appears in its
|
||||||
|
original position relative to other messages the consumer has
|
||||||
|
not yet seen). At-least-once with no ordering: a redelivery may
|
||||||
|
appear out of order relative to messages delivered after it.
|
||||||
|
- The consumer must not assume an ordering property the broker
|
||||||
|
does not provide (P2). A standard queue delivers per-receive-node
|
||||||
|
arrival order but no global order and no order across redeliveries;
|
||||||
|
a FIFO queue delivers strict per-group order including across
|
||||||
|
redeliveries; a partitioned stream delivers strict per-partition
|
||||||
|
order across redeliveries. Document the property; do not assume
|
||||||
|
it. See `domains/messaging/queues.md` (FIFO vs standard) and
|
||||||
|
`domains/messaging/streams.md` (per-partition order).
|
||||||
|
|
||||||
|
## Dead-Letter Strategies (P5 Dead-Letter Handling is Defined)
|
||||||
|
|
||||||
|
- A poison message (unparseable, repeatedly failing, or exhausting
|
||||||
|
the retry budget) must be routed to a dead-letter queue, not
|
||||||
|
retried forever or silently dropped (P5). The DLQ is observable
|
||||||
|
(P10 — depth is an alert) and drainable (an operator can
|
||||||
|
inspect, replay, or discard with audit).
|
||||||
|
- The dead-letter strategy determines when a message is
|
||||||
|
dead-lettered and what the operator sees. The choice is the
|
||||||
|
decision matrix below (D-069).
|
||||||
|
|
||||||
|
## Dead-Letter Strategy Comparison (D-069)
|
||||||
|
|
||||||
|
| Strategy | When It Applies | Failure Visibility | Operational Cost |
|
||||||
|
|----------|-----------------|-------------------|------------------|
|
||||||
|
| **Retry-count-limit** | A fixed max-redeliveries count (e.g., 5). After N redeliveries, route to DLQ. Simple, predictable. | The redelivery count is visible in the DLQ entry; the operator sees how many times it was retried. | Low — a counter per message; no backoff tuning. Risk: retries fire as fast as the broker redelivers, hammering a downstream that is already failing (P6 — no backoff = no backpressure escape). |
|
||||||
|
| **TTL-with-backoff** | A max time-to-live for redelivery (e.g., 30 minutes) with exponential backoff between retries. After the TTL, route to DLQ. | The TTL and the backoff schedule are visible; the operator sees the retry timeline. | Medium — backoff tuning per message type. Benefit: backoff gives the downstream time to recover (P6 — the retry rate is bounded); fits transient failures (a downstream that is briefly unavailable). |
|
||||||
|
| **Poison-queue** | A separate queue for messages that fail a specific check (unparseable, schema-invalid, unknown type) before any processing retry. Routed immediately, not retried. | The poison queue is a separate signal from the DLQ; the operator sees parse-vs-process failures distinctly. | Low — a routing rule per check. Benefit: distinguishes "never going to succeed" (poison) from "might succeed on retry" (DLQ). Use for unparseable messages that no retry will fix. |
|
||||||
|
| **DLQ + alert** | Any of the above strategies, plus an alert on DLQ depth. The DLQ is observable (P10) — depth, age, and rate are alerted. | Highest — the operator is paged on DLQ growth; the DLQ is a first-class signal, not a graveyard. | Medium — alerting setup per DLQ. This is the P5/P10 floor: a DLQ without an alert is a silent correctness defect (poison messages accumulate invisibly). |
|
||||||
|
|
||||||
|
- The default for transient failures is **TTL-with-backoff + DLQ
|
||||||
|
+ alert**: backoff gives the downstream time to recover (P6),
|
||||||
|
the TTL bounds the retry budget (P5), the DLQ captures the
|
||||||
|
unprocessable, and the alert makes it visible (P10). The default
|
||||||
|
for unparseable messages is **poison-queue + alert**: route
|
||||||
|
immediately, do not retry a message no retry will fix.
|
||||||
|
- **Retry-count-limit alone** (no backoff, no alert) is the
|
||||||
|
`messaging-unbounded-retry` chaos anti-pattern's cousin: it
|
||||||
|
caps the count but hammers the downstream at full retry rate,
|
||||||
|
and the DLQ grows silently if no alert is wired. Always pair
|
||||||
|
a retry budget with backoff (P6) and an alert (P10).
|
||||||
|
- The failure-visibility column is the P10 check: every strategy
|
||||||
|
must surface the failure to the operator. A strategy with no
|
||||||
|
visibility is a P10 violation regardless of its retry semantics.
|
||||||
|
- The operational-cost column is the C8 tradeoff: more visibility
|
||||||
|
and more backoff cost more to set up but pay back in operational
|
||||||
|
stability. The P5/P10 floor is "DLQ + alert"; below that, the
|
||||||
|
strategy is a silent defect waiting to grow.
|
||||||
|
|
||||||
|
## DLQ Routing Rule (P5, P10)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# DLQ routing rule (P5 dead-letter handling, P6 bounded retry with
|
||||||
|
# backoff, P10 DLQ depth alert). Combines TTL-with-backoff for
|
||||||
|
# transient failures and poison-queue for unparseable messages,
|
||||||
|
# with an alert on DLQ depth.
|
||||||
|
|
||||||
|
import json, time
|
||||||
|
|
||||||
|
MAX_RETRY_TTL_SECONDS = 30 * 60 # 30 min total retry window
|
||||||
|
DLQ = "orders-dlq"
|
||||||
|
POISON = "orders-poison" # unparseable; never retried
|
||||||
|
|
||||||
|
def route_to_dlq(broker, message, reason):
|
||||||
|
"""Route a message to the DLQ with audit metadata (P5)."""
|
||||||
|
broker.send(DLQ, body=json.dumps({
|
||||||
|
"original": message,
|
||||||
|
"reason": str(reason),
|
||||||
|
"deadLetteredAt": now_iso(),
|
||||||
|
"redeliveryCount": message.get("attempt", 0),
|
||||||
|
}))
|
||||||
|
# P10: emit a metric so DLQ depth alerts fire. A DLQ that
|
||||||
|
# grows with no alert is a silent correctness defect (P5/P10).
|
||||||
|
metrics.increment("dlq.depth", tags={"queue": "orders"})
|
||||||
|
# The ack removes the message from the origin queue; the DLQ
|
||||||
|
# is the durable record (P5 — observable and drainable).
|
||||||
|
|
||||||
|
def consume_with_dlq(broker, dedup, process):
|
||||||
|
for message in broker.receive():
|
||||||
|
# P1: parse first. An unparseable message is poison —
|
||||||
|
# route immediately, do NOT retry (no retry will fix it).
|
||||||
|
try:
|
||||||
|
payload = json.loads(message["body"])
|
||||||
|
except (ValueError, SchemaError) as exc:
|
||||||
|
broker.send(POISON, body=json.dumps({
|
||||||
|
"original": message["body"],
|
||||||
|
"reason": f"parse-failed: {exc}",
|
||||||
|
"poisonedAt": now_iso(),
|
||||||
|
}))
|
||||||
|
metrics.increment("poison.depth", tags={"queue": "orders"})
|
||||||
|
broker.ack(message) # remove from origin; poison queue holds it
|
||||||
|
continue
|
||||||
|
|
||||||
|
# P3: dedup before process.
|
||||||
|
if dedup.seen(payload["idempotencyKey"]):
|
||||||
|
broker.ack(message); continue
|
||||||
|
|
||||||
|
attempt = payload.get("attempt", 0)
|
||||||
|
first_attempt_ts = payload.get("firstAttemptTs", time.time())
|
||||||
|
|
||||||
|
try:
|
||||||
|
process(payload)
|
||||||
|
dedup.mark(payload["idempotencyKey"])
|
||||||
|
broker.ack(message)
|
||||||
|
except TransientError as exc:
|
||||||
|
# P6: TTL-with-backoff. If the retry window is
|
||||||
|
# exhausted, route to DLQ; otherwise redeliver with
|
||||||
|
# exponential backoff.
|
||||||
|
if time.time() - first_attempt_ts > MAX_RETRY_TTL_SECONDS:
|
||||||
|
route_to_dlq(broker, message, exc) # P5
|
||||||
|
broker.ack(message)
|
||||||
|
else:
|
||||||
|
broker.nack(message, delay=backoff(attempt))
|
||||||
|
except PermanentError as exc:
|
||||||
|
# A permanent error (e.g., a not-found dependency)
|
||||||
|
# does not benefit from retry — route to DLQ now.
|
||||||
|
route_to_dlq(broker, message, exc)
|
||||||
|
broker.ack(message)
|
||||||
|
```
|
||||||
|
|
||||||
|
- The routing rule distinguishes three failure modes: **poison**
|
||||||
|
(unparseable — route immediately, no retry), **transient**
|
||||||
|
(retry with backoff until the TTL, then DLQ), and **permanent**
|
||||||
|
(a retry will not fix it — DLQ now). This distinction is the P5
|
||||||
|
discipline: not every failure is a retry; some are immediate
|
||||||
|
DLQs.
|
||||||
|
- The DLQ entry carries `reason`, `deadLetteredAt`, and
|
||||||
|
`redeliveryCount` — it is auditable (the operator knows why each
|
||||||
|
message was dead-lettered and how many times it was retried).
|
||||||
|
This is the errors-as-data discipline — see
|
||||||
|
`domains/errors/patterns` for the general principle a DLQ entry
|
||||||
|
instantiates.
|
||||||
|
|
||||||
|
## Retry Budgets and Backoff (P6 Backpressure is Bounded)
|
||||||
|
|
||||||
|
- The retry budget is the cap on redelivery: a count, a TTL, or
|
||||||
|
both. After the budget, the message routes to the DLQ (P5). An
|
||||||
|
unbounded retry budget is the `messaging-unbounded-retry` chaos
|
||||||
|
anti-pattern (P5 breach): the consumer never makes progress past
|
||||||
|
the poison message.
|
||||||
|
- **Exponential backoff** spaces retries: 1s, 2s, 4s, 8s, ... with
|
||||||
|
a jitter to avoid thundering-herd synchrony. Backoff gives the
|
||||||
|
downstream time to recover (P6 — the retry rate is bounded,
|
||||||
|
giving the downstream a chance to catch up). A retry with no
|
||||||
|
backoff hammers the downstream at full rate, making the failure
|
||||||
|
worse.
|
||||||
|
- The retry budget × the backoff schedule is the P6 bound: the
|
||||||
|
consumer's retry load is bounded by design, not by luck. See
|
||||||
|
`domains/concurrency/patterns` Pattern 6 (Timeout on Every
|
||||||
|
Block) for the in-process retry/backoff analog; messaging owns
|
||||||
|
the broker-backed instance where the redelivery comes from the
|
||||||
|
broker across a network, not an in-process loop (D-062).
|
||||||
|
|
||||||
|
## Poison Messages (P5, P1)
|
||||||
|
|
||||||
|
- A poison message is one no retry will fix: unparseable (the
|
||||||
|
schema is wrong, P1), unknown type (the consumer does not handle
|
||||||
|
this version, P9), or a permanent failure (a not-found
|
||||||
|
dependency). Retrying a poison message wastes resources and
|
||||||
|
blocks the queue (P6 — the consumer never makes progress).
|
||||||
|
- Poison messages route to the poison queue immediately (no
|
||||||
|
retry), distinct from the DLQ (which holds messages that
|
||||||
|
exhausted their retry budget on transient failures). The
|
||||||
|
distinction is the P5 discipline: a poison queue is for
|
||||||
|
"never going to succeed"; a DLQ is for "might have succeeded
|
||||||
|
but didn't within the budget."
|
||||||
|
- A poison queue without an alert is the same defect as a DLQ
|
||||||
|
without an alert (P10): the operator cannot see the poison
|
||||||
|
accumulating. Wire both to `domains/observability/metrics`.
|
||||||
|
|
||||||
|
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
|
||||||
|
|
||||||
|
- The retry/backoff discipline here is the cross-process analog
|
||||||
|
of `domains/concurrency/patterns` Pattern 6 (Timeout on Every
|
||||||
|
Block) for in-process retry. Concurrency owns the in-process
|
||||||
|
analog (a retry loop within one program, with a timeout per
|
||||||
|
attempt); messaging owns the broker-backed instance (the broker
|
||||||
|
redelivers across a network, the consumer applies backoff via
|
||||||
|
nack-with-delay). The failure model differs: in-process retry
|
||||||
|
fails by a thread crash or a timeout; broker-backed retry fails
|
||||||
|
by network partition, broker restart, or consumer crash-and-retry
|
||||||
|
(D-062).
|
||||||
|
- The cross-link is one-directional outward (messaging →
|
||||||
|
concurrency) per D-026 extended: messaging references concurrency
|
||||||
|
as the in-process foundation; concurrency does not back-link to
|
||||||
|
messaging.
|
||||||
|
|
||||||
|
## Cross-Link to Errors (P5, cross-link errors/patterns)
|
||||||
|
|
||||||
|
- A poison message is an errors-as-data instance: the failure is
|
||||||
|
captured as a DLQ entry (with `reason`, `redeliveryCount`,
|
||||||
|
`deadLetteredAt`), not swallowed. See `domains/errors/patterns`
|
||||||
|
for the general errors-as-data discipline a DLQ entry
|
||||||
|
instantiates. The DLQ is the async-messaging instance of an
|
||||||
|
error log — observable, auditable, drainable.
|
||||||
|
- The cross-link is one-directional outward (messaging → errors):
|
||||||
|
messaging references errors for the errors-as-data pattern;
|
||||||
|
errors does not back-link to messaging.
|
||||||
|
|
||||||
|
## What Violates Delivery-Semantics Discipline
|
||||||
|
|
||||||
|
| Violation | Principle |
|
||||||
|
|-----------|-----------|
|
||||||
|
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||||||
|
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
|
||||||
|
| Dedup store with no TTL (memory leak; grows without bound) | P6 Backpressure is Bounded |
|
||||||
|
| Dedup key per-producer (distinct messages in the same window deduped) | P3 Consumers are Idempotent |
|
||||||
|
| Retry with no backoff (hammers the downstream at full rate) | P6 Backpressure is Bounded |
|
||||||
|
| Unbounded retry budget (consumer never progresses past the poison) | P5 Dead-Letter Handling is Defined |
|
||||||
|
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
|
||||||
|
| Poison message retried forever (no poison queue, no immediate DLQ) | P5 Dead-Letter Handling is Defined |
|
||||||
|
| Process-not-idempotent with non-transactional mark (crash in window double-applies) | P3, P4 |
|
||||||
|
| DLQ entry with no reason/audit metadata (uninspectable failure) | P5, `domains/errors/patterns` |
|
||||||
|
| Ordering assumption the broker does not provide (FIFO assumed on standard queue) | P2 Ordering is a Property, Not an Assumption |
|
||||||
@@ -0,0 +1,307 @@
|
|||||||
|
# Messaging — First Principles
|
||||||
|
|
||||||
|
## 1. The Principles
|
||||||
|
|
||||||
|
### P1. Messages are Contracts
|
||||||
|
A message has an explicit, versioned schema. Producer and consumer
|
||||||
|
agree on shape before exchange; the schema is the boundary, not a
|
||||||
|
guess. A schemaless message — a free-form JSON blob the consumer
|
||||||
|
parses by hope — is a defect: the consumer breaks silently on the
|
||||||
|
next shape change, and the producer has no contract to evolve
|
||||||
|
against. This derives from `C1 Correctness` (the exchange must
|
||||||
|
carry what the parties agreed to) and `C2 Clarity` (the schema
|
||||||
|
makes the boundary obvious to both sides). This is the
|
||||||
|
cross-process expression of the contract discipline that
|
||||||
|
`domains/api/rest` owns for synchronous request/response: the
|
||||||
|
message schema is to async exchange what the API contract is to
|
||||||
|
sync exchange. It is distinct from
|
||||||
|
`domains/concurrency/patterns` Pattern 1 (Message Passing), which
|
||||||
|
owns the *in-process* channel primitive — here the contract spans
|
||||||
|
separate systems and survives network failure (D-062). See
|
||||||
|
`domains/messaging/queues.md` for the queue-flavored application
|
||||||
|
and `domains/messaging/streams.md` for the durable-log-flavored
|
||||||
|
application.
|
||||||
|
|
||||||
|
### P2. Ordering is a Property, Not an Assumption
|
||||||
|
Ordering guarantees — per-partition strict, global, or none — are
|
||||||
|
explicit and documented. "It's FIFO" is a claim that must be backed
|
||||||
|
by the broker's partitioning contract, not an assumption the
|
||||||
|
consumer makes and the broker may not honor. A standard queue
|
||||||
|
delivers in arrival order per receive-node but offers no global
|
||||||
|
ordering across shards; a FIFO queue delivers strict per-message-
|
||||||
|
group order but at a latency cost; a partitioned stream delivers
|
||||||
|
strict per-partition order but only within a partition. Each is a
|
||||||
|
distinct, declared property. This derives from `C1 Correctness`
|
||||||
|
(order is a correctness property — a consumer that assumes order
|
||||||
|
the broker does not provide is wrong) and `C2 Clarity` (the
|
||||||
|
ordering guarantee is documented, not discovered in production).
|
||||||
|
This is distinct from in-process ordering, which
|
||||||
|
`domains/concurrency/patterns` Pattern 1 owns for channels within
|
||||||
|
one program: messaging ordering survives network failure, broker
|
||||||
|
restart, and consumer crash-and-retry — a stronger failure model
|
||||||
|
than thread-local channels (D-062). The `messaging-shared-
|
||||||
|
subscription` chaos anti-pattern breaches this rule: two consumers
|
||||||
|
sharing one subscription break per-consumer ordering because the
|
||||||
|
broker dispatches each message to an arbitrary consumer. See
|
||||||
|
`domains/messaging/streams.md` for the partition-order contract
|
||||||
|
and `domains/messaging/delivery-semantics.md` for the interaction
|
||||||
|
of ordering with the three delivery semantics.
|
||||||
|
|
||||||
|
### P3. Consumers are Idempotent
|
||||||
|
Delivery is at-least-once by default across the network; a
|
||||||
|
consumer deduplicates via idempotency keys or deterministic
|
||||||
|
processing. "Exactly-once" is idempotency plus at-least-once, not a
|
||||||
|
broker guarantee — Jepsen analyses of Kafka, RabbitMQ, and NATS
|
||||||
|
establish that exactly-once claims require independent
|
||||||
|
verification, and the durable engineering practice is to make
|
||||||
|
consumers idempotent under redelivery. A non-idempotent consumer
|
||||||
|
under at-least-once delivery doubles the effect on every retry; a
|
||||||
|
non-idempotent consumer under a claimed exactly-once broker is a
|
||||||
|
bug waiting for the broker's exactly-once invariant to break. This
|
||||||
|
derives from `C1 Correctness`: correctness under redelivery is the
|
||||||
|
contract, not a nice-to-have. This parallels
|
||||||
|
`domains/edge/P5 Edge Operations are Idempotent` (the
|
||||||
|
cross-partition device-and-cache-flavored analog) and is the
|
||||||
|
cross-process instance of the retry-safety discipline that
|
||||||
|
`domains/concurrency/patterns` Pattern 6 (Timeout on Every Block)
|
||||||
|
implies for in-process retry. It is distinct from in-process
|
||||||
|
retry because the redelivery comes from the broker across a
|
||||||
|
network, not from an in-process loop (D-062). See
|
||||||
|
`domains/messaging/delivery-semantics.md` for the idempotency-key
|
||||||
|
dedup-store pattern.
|
||||||
|
|
||||||
|
### P4. Delivery Semantics are Explicit
|
||||||
|
At-least-once / at-most-once / exactly-once is a declared choice
|
||||||
|
per channel, not an emergent behavior. The tradeoff — latency cost,
|
||||||
|
implementation complexity, operational cost — is made consciously
|
||||||
|
and documented. At-most-once is fire-and-forget (low latency, lossy);
|
||||||
|
at-least-once is acked with possible duplication (the default,
|
||||||
|
requires idempotent consumers per P3); exactly-once is at-least-once
|
||||||
|
plus idempotency or a transactional two-phase commit (highest cost,
|
||||||
|
narrowest fit). An unstated semantic is a defect: the consumer
|
||||||
|
guesses, and the guess is wrong under the first failure. This
|
||||||
|
derives from `C1 Correctness` (the chosen semantic must hold) and
|
||||||
|
`C2 Clarity` (the tradeoff is visible to the reader and the
|
||||||
|
operator). This is the cross-process analog of the explicit-failure-
|
||||||
|
mode discipline that `domains/errors/patterns` owns for synchronous
|
||||||
|
code — messaging makes the delivery-mode choice as explicit as an
|
||||||
|
error-handling choice. See `domains/messaging/queues.md` for the
|
||||||
|
three-semantics comparison table and `domains/messaging/delivery-
|
||||||
|
semantics.md` for the correctness properties of each.
|
||||||
|
|
||||||
|
### P5. Dead-Letter Handling is Defined
|
||||||
|
Poison messages — unparseable, repeatedly failing, or exhausting
|
||||||
|
the retry budget — are routed to a dead-letter queue, not retried
|
||||||
|
forever or silently dropped. The DLQ is observable and drainable: an
|
||||||
|
operator can inspect it, replay from it, or discard with audit. An
|
||||||
|
unbounded retry loop is a livelock: the consumer never makes
|
||||||
|
progress past the poison message. A silent drop is a correctness
|
||||||
|
defect: the message vanished with no record. This derives from `C1
|
||||||
|
Correctness` (poison messages must not livelock the consumer or
|
||||||
|
silently disappear) and `C5 Reversibility` (the DLQ is the
|
||||||
|
reversibility mechanism — a dead-lettered message can be reprocessed
|
||||||
|
after the bug is fixed). This is the cross-process analog of the
|
||||||
|
bounded-error discipline that `domains/errors/patterns` owns for
|
||||||
|
synchronous code: a poison message is an error-as-data instance
|
||||||
|
that must be observable and recoverable, not swallowed. It is
|
||||||
|
distinct from in-process error handling because the failure spans a
|
||||||
|
network and a consumer restart (D-062). See
|
||||||
|
`domains/messaging/delivery-semantics.md` for the dead-letter
|
||||||
|
strategy comparison table and the DLQ routing rule pattern.
|
||||||
|
|
||||||
|
### P6. Backpressure is Bounded
|
||||||
|
A slow consumer cannot unbounded-buffer the broker or the
|
||||||
|
producer. Backpressure is explicit: consumer lag is visible,
|
||||||
|
max-unacked is bounded, the retry budget is capped. A consumer
|
||||||
|
that falls behind without a visible signal is a silent backlog —
|
||||||
|
the operator cannot fix what they cannot see, and the broker's
|
||||||
|
memory grows without bound until it fails. This derives from `C1
|
||||||
|
Correctness` (a backlog that grows until OOM is a correctness
|
||||||
|
failure) and `C8 Economy` (the broker's memory is bounded by
|
||||||
|
design, not by luck). This is distinct from
|
||||||
|
`domains/concurrency/P9 Bounded Queues` and
|
||||||
|
`domains/concurrency/patterns` Pattern 5 (Bounded Queue with
|
||||||
|
Backpressure), which own the *in-process* analog: concurrency's
|
||||||
|
bounded queue fails by OOM or thread crash; messaging's bounded
|
||||||
|
backpressure fails by network partition, broker restart, or
|
||||||
|
consumer crash-and-retry (D-062). The Reactive Streams
|
||||||
|
specification (`request(n)`, `onNext` bounded) is the in-process
|
||||||
|
instance; messaging's broker-backed backpressure is the
|
||||||
|
cross-process instance above it. See `domains/messaging/queues.md`
|
||||||
|
for prefetch and max-unacked and `domains/observability/metrics`
|
||||||
|
for consumer-lag as an alert.
|
||||||
|
|
||||||
|
### P7. Partitioning is Intentional
|
||||||
|
The partition key determines ordering, parallelism, and hotspots.
|
||||||
|
Key choice is a design decision with documented rationale, not a
|
||||||
|
default. A key that hashes unevenly creates a hot partition that
|
||||||
|
limits throughput; a key that does not match the ordering need
|
||||||
|
breaks per-key semantics; a key that is too coarse (one partition
|
||||||
|
for the whole topic) serializes all the traffic. The partition
|
||||||
|
count is a capacity bound: too few partitions cap parallelism, too
|
||||||
|
many partition overhead the broker. This derives from `C4 Locality`
|
||||||
|
(ordering and parallelism are co-located with the partition) and
|
||||||
|
`C6 Composability` (the partition is the unit of parallelism and
|
||||||
|
scaling — consumer groups compose from per-partition workers).
|
||||||
|
This is the cross-process analog of the locality discipline that
|
||||||
|
`domains/performance/` owns for generic data-near-compute
|
||||||
|
optimization: performance's locality is algorithmic (data near
|
||||||
|
compute); messaging's locality is partitional (order and
|
||||||
|
parallelism near the partition). See `domains/messaging/streams.md`
|
||||||
|
for the partitioned-log model and consumer-group rebalance
|
||||||
|
strategies.
|
||||||
|
|
||||||
|
### P8. Replay and Retention are Configured
|
||||||
|
Retention windows and replay-from-offset are explicit. A message
|
||||||
|
is not ephemeral by default; the broker is a durable log, not a
|
||||||
|
pipe. A topic with no retention is a fire-and-forget stream — a
|
||||||
|
consumer that falls behind loses data permanently; a topic with
|
||||||
|
infinite retention is an unbounded log — the broker grows until
|
||||||
|
disk exhaustion. Both are defects: the retention window is a
|
||||||
|
declared bound, and replay-from-offset is the mechanism that makes
|
||||||
|
the log durable (re-consumable) rather than ephemeral. This derives
|
||||||
|
from `C5 Reversibility` (a retained message is reversible — it can
|
||||||
|
be re-consumed; an ephemeral message is not) and `C7 Observability`
|
||||||
|
(the durable log is itself an observable record of what happened —
|
||||||
|
the offset is the position from which to replay). This is the
|
||||||
|
foundation for `domains/messaging/streams.md` and the rule that
|
||||||
|
distinguishes a stream from a queue (a queue deletes on ack; a
|
||||||
|
stream retains for replay). See `domains/messaging/pubsub.md` for
|
||||||
|
the pub/sub-vs-stream durability boundary.
|
||||||
|
|
||||||
|
### P9. Schemas Evolve Compatibly
|
||||||
|
Schema changes are backward- and forward-compatible by
|
||||||
|
construction. Breaking changes are versioned migrations, not
|
||||||
|
silent shape edits. A producer that ships a new field the old
|
||||||
|
consumer ignores is backward-compatible; a consumer that handles a
|
||||||
|
missing field the new producer omits is forward-compatible. A
|
||||||
|
silent schema change — the producer renames a field and the
|
||||||
|
consumer parses `undefined` — is a P1 violation (the contract was
|
||||||
|
broken) compounded here as an evolution defect. This derives from
|
||||||
|
`C5 Reversibility` (a schema change is reversible by versioning —
|
||||||
|
the old shape is still readable) and `C6 Composability` (producers
|
||||||
|
and consumers of different versions compose because the schema
|
||||||
|
evolves compatibly). This parallels `domains/data/migrations`
|
||||||
|
(schema migration for databases) and `domains/api/versioning`
|
||||||
|
(API contract evolution): messaging's schema evolution is the
|
||||||
|
async instance of the same compatibility discipline. See
|
||||||
|
`domains/messaging/streams.md` for the stream-schema-evolution
|
||||||
|
angle.
|
||||||
|
|
||||||
|
### P10. Messaging is Observable
|
||||||
|
Consumer lag, DLQ depth, throughput, and consumer-group health are
|
||||||
|
first-class signals. Silent backlog is a bug, not a feature: a
|
||||||
|
consumer that falls behind with no lag metric is invisible until
|
||||||
|
the downstream effect surfaces — by which time the backlog may be
|
||||||
|
hours or days. A DLQ that grows without an alert is a silent
|
||||||
|
correctness defect: poison messages are accumulating and no one
|
||||||
|
knows. This derives from `C7 Observability` (the broker's behavior
|
||||||
|
is visible to the operator) and `C1 Correctness` (backlog
|
||||||
|
detection is a correctness bound — unbounded lag is a failure).
|
||||||
|
This is distinct from `domains/observability/metrics`, which owns
|
||||||
|
*generic* structured metrics; messaging owns the *broker-specific*
|
||||||
|
signals — lag, DLQ depth, partition imbalance, consumer-group
|
||||||
|
rebalance events. See `domains/observability/metrics` for the
|
||||||
|
generic SLI/SLO discipline and `domains/observability/tracing` for
|
||||||
|
cross-partition traces.
|
||||||
|
|
||||||
|
## 2. Core Principle Trace
|
||||||
|
|
||||||
|
Each messaging P-rule derives from one or more core C-rules
|
||||||
|
(C1–C8). The matrix extension lands in P4 of the v0.4 plan; the
|
||||||
|
traces below are authoritative. Messaging is a broad-derivation
|
||||||
|
domain touching 7 of 8 core principles (C1, C2, C4, C5, C6, C7,
|
||||||
|
C8); C3 (Simplicity) is not a primary derivation — messaging is
|
||||||
|
inherently a tradeoff domain where simplicity yields to the
|
||||||
|
correctness of delivery guarantees (a simpler-than-necessary
|
||||||
|
delivery model does not handle the failure cases, per C3's
|
||||||
|
"simpler than necessary is also a violation").
|
||||||
|
|
||||||
|
| P-rule | Core | Why |
|
||||||
|
|--------|------|-----|
|
||||||
|
| P1 Messages are Contracts | C1, C2 | Correctness of the exchange; clarity of the schema boundary |
|
||||||
|
| P2 Ordering is a Property, Not an Assumption | C1, C2 | Correctness of order; clarity of the guarantee |
|
||||||
|
| P3 Consumers are Idempotent | C1 | Correctness under redelivery |
|
||||||
|
| P4 Delivery Semantics are Explicit | C1, C2 | Correctness of the chosen semantic; clarity of the tradeoff |
|
||||||
|
| P5 Dead-Letter Handling is Defined | C1, C5 | Correctness of poison-message routing; reversibility of reprocessing |
|
||||||
|
| P6 Backpressure is Bounded | C1, C8 | Correctness of bounded backlog; economy of broker memory |
|
||||||
|
| P7 Partitioning is Intentional | C4, C6 | Locality of order; composability of parallelism |
|
||||||
|
| P8 Replay and Retention are Configured | C5, C7 | Reversibility of replay; observability of the durable log |
|
||||||
|
| P9 Schemas Evolve Compatibly | C5, C6 | Reversibility of schema changes; composability of versions |
|
||||||
|
| P10 Messaging is Observable | C7, C1 | Observability of lag/DLQ; correctness of backlog detection |
|
||||||
|
|
||||||
|
## 3. What Violates These Principles
|
||||||
|
|
||||||
|
| Violation | Principle Breached |
|
||||||
|
|-----------|-------------------|
|
||||||
|
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
|
||||||
|
| "It's FIFO" with no documented partition contract | P2 Ordering is a Property, Not an Assumption |
|
||||||
|
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
|
||||||
|
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||||||
|
| No dead-letter queue (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
|
||||||
|
| Unbounded retry budget (no cap; slow consumer stalls the partition) | P6 Backpressure is Bounded |
|
||||||
|
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
|
||||||
|
| Ephemeral broker (no retention; no replay) | P8 Replay and Retention are Configured |
|
||||||
|
| Silent schema change (producer breaks consumers with no version bump) | P9 Schemas Evolve Compatibly |
|
||||||
|
| Silent backlog (no lag metric; consumer falls behind invisibly) | P10 Messaging is Observable |
|
||||||
|
| Shared subscription (two consumers share one subscription; per-consumer ordering breaks) | P2 Ordering is a Property, Not an Assumption (P3 compounding) |
|
||||||
|
| Blocking consumer (slow downstream call with no timeout; broker redelivers to the stuck consumer) | P6 Backpressure is Bounded |
|
||||||
|
|
||||||
|
## 4. Relationship to Other Domains
|
||||||
|
|
||||||
|
Messaging systems are the engineering discipline of
|
||||||
|
**cross-process, cross-system asynchronous communication via
|
||||||
|
brokers**. Producer and consumer are separate systems; the broker
|
||||||
|
is the intermediary that brokers delivery, ordering, retention,
|
||||||
|
and failure semantics. The distinguishing constraints are a
|
||||||
|
cross-process failure model (network, not crash), explicit
|
||||||
|
delivery semantics, decoupled producer/consumer lifecycle, and
|
||||||
|
replay-and-retention as a durable-log property. Messaging overlaps
|
||||||
|
`domains/concurrency/` by *subject* (messages, queues,
|
||||||
|
backpressure) but not by *failure model*: per D-062, messaging
|
||||||
|
owns the cross-process/network-failure-model angle; concurrency
|
||||||
|
owns the in-process/crash-failure-model angle. The discriminator
|
||||||
|
is the failure model: concurrency's queue fails by OOM or thread
|
||||||
|
crash; messaging's queue fails by network partition, broker
|
||||||
|
restart, or consumer crash-and-retry. Messaging extends
|
||||||
|
concurrency's bounded-queue/backpressure model to the network-
|
||||||
|
partition regime. Cross-links are one-directional outward (per
|
||||||
|
D-026 extended); no back-link edits to v0.1/v0.2/v0.3 content.
|
||||||
|
|
||||||
|
- `domains/concurrency/patterns` ← P6 (the broker-backed bounded
|
||||||
|
queue is the cross-process analog of the in-process bounded
|
||||||
|
buffer — concurrency Pattern 5 owns in-process; messaging owns
|
||||||
|
the network-failure-model instance above it, per D-062)
|
||||||
|
- `domains/concurrency/patterns` ← P3 (idempotent retry is the
|
||||||
|
cross-process analog of in-process retry-safety — the failure
|
||||||
|
model differs: broker redelivery across a network vs in-process
|
||||||
|
loop)
|
||||||
|
- `domains/observability/metrics` ← P10 (consumer lag and DLQ
|
||||||
|
depth as alerts; observability owns the generic SLI/SLO
|
||||||
|
discipline, messaging owns the broker-specific signals)
|
||||||
|
- `domains/observability/tracing` ← P10 (cross-partition traces
|
||||||
|
for stream processing; observability owns the generic tracing
|
||||||
|
discipline, messaging owns the cross-partition propagation)
|
||||||
|
- `domains/data/schema-design` ← P1, P9 (message schema design
|
||||||
|
and evolution; data owns the generic schema discipline,
|
||||||
|
messaging owns the cross-process message-shape instance)
|
||||||
|
- `domains/errors/patterns` ← P5 (errors as data for message
|
||||||
|
failures; a poison message is an error-as-data instance that must
|
||||||
|
be observable and recoverable, not swallowed)
|
||||||
|
- `domains/edge/iot` ← P4 (the edge↔messaging cross-link
|
||||||
|
resolves bidirectionally here: edge/iot.md links outward to
|
||||||
|
messaging/queues for MQTT QoS parallels to delivery semantics;
|
||||||
|
this first-principles doc acknowledges the back-link — the
|
||||||
|
edge/iot.md → messaging/queues link from P1 now resolves because
|
||||||
|
messaging/queues.md exists, completing the bidirectionality per
|
||||||
|
IDEATE-40)
|
||||||
|
|
||||||
|
> Note: the edge/iot.md → messaging/queues cross-link (MQTT QoS
|
||||||
|
> parallels for delivery semantics) was authored in P1 with a
|
||||||
|
> dangling reference; this P2 authorship of messaging/queues.md
|
||||||
|
> resolves it. The bidirectionality is verified in P5
|
||||||
|
> (ATELIER-114 per IDEATE-40). The cross-link is one-directional
|
||||||
|
> outward from edge/iot.md; this first-principles doc
|
||||||
|
> acknowledges the resolution without editing edge/iot.md (per
|
||||||
|
> D-026 extended — no back-link edits to v0.1/v0.2/v0.3 or to
|
||||||
|
> P1-authored edge content).
|
||||||
@@ -0,0 +1,269 @@
|
|||||||
|
# Pub/Sub — Derived Rules
|
||||||
|
|
||||||
|
> Derives from `domains/messaging/first-principles.md`. Applies P1
|
||||||
|
> (Messages are Contracts), P2 (Ordering is a Property, Not an
|
||||||
|
> Assumption), P3 (Consumers are Idempotent), and P4 (Delivery
|
||||||
|
> Semantics are Explicit) primarily, with P7 (partitioning), P10
|
||||||
|
> (per-subscription lag). The `messaging-shared-subscription` chaos
|
||||||
|
> anti-pattern lives here (pre-specified in P4 ATELIER-110).
|
||||||
|
> Cross-links `domains/messaging/streams` for the pub/sub-vs-stream
|
||||||
|
> durability boundary and `domains/observability/metrics` for
|
||||||
|
> per-subscription lag.
|
||||||
|
|
||||||
|
## What Pub/Sub Is (P1 Messages are Contracts)
|
||||||
|
|
||||||
|
- Pub/sub is the fan-out primitive: a producer publishes a message
|
||||||
|
to a topic; N independent subscriptions each receive a copy. The
|
||||||
|
message has an explicit, versioned schema (P1): the topic's
|
||||||
|
schema is the contract every subscription agrees to before
|
||||||
|
subscribing. A schemaless topic is a defect — every subscriber
|
||||||
|
breaks silently on the next shape change.
|
||||||
|
- The boundary with queues is the fan-out ratio. A queue is
|
||||||
|
point-to-point (one producer, one consumer); pub/sub is
|
||||||
|
one-to-many (one producer, N consumers, each with its own
|
||||||
|
subscription). The boundary with streams is the durability model
|
||||||
|
— see the cross-link below. Pub/sub is an async concern because
|
||||||
|
producer and consumers are separate systems and the failure model
|
||||||
|
is network, not crash (D-062).
|
||||||
|
- See `domains/messaging/queues.md` for the point-to-point variant
|
||||||
|
and `domains/messaging/streams.md` for the durable-log variant.
|
||||||
|
|
||||||
|
## Topic / Subscription Model (P1, P3, P4)
|
||||||
|
|
||||||
|
- A **topic** is the named stream of messages. A **subscription**
|
||||||
|
is a durable cursor over the topic: each subscription receives
|
||||||
|
every message published after it was created (subject to
|
||||||
|
retention and filtering). The subscription is independent — its
|
||||||
|
ack, redelivery, and DLQ are per-subscription, not shared.
|
||||||
|
- Each subscription is a consumer under at-least-once by default
|
||||||
|
(P4): the broker redelivers until the subscription acks, and the
|
||||||
|
subscriber must be idempotent (P3). A subscription with no
|
||||||
|
idempotency dedup duplicates every redelivered message.
|
||||||
|
- The topic's schema evolves compatibly (P9 — Schemas Evolve
|
||||||
|
Compatibly): a new field the old subscriber ignores is
|
||||||
|
backward-compatible; a renamed field the old subscriber parses
|
||||||
|
as `undefined` is a P1 violation.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Publish + two independent subscriptions (P1 contract, P3
|
||||||
|
# idempotency, P4 at-least-once per subscription). Each
|
||||||
|
# subscription is an independent durable cursor; acking one does
|
||||||
|
# not affect the other.
|
||||||
|
|
||||||
|
import json, uuid
|
||||||
|
|
||||||
|
# --- Publisher ---
|
||||||
|
def publish(topic, event, broker):
|
||||||
|
# P1: versioned schema on the topic. All subscribers must
|
||||||
|
# understand this schema (or a compatible superset — P9).
|
||||||
|
message = {
|
||||||
|
"schema": "user.signed-up.v1",
|
||||||
|
"id": str(uuid.uuid4()),
|
||||||
|
"idempotencyKey": f"user:{event['userId']}:signup",
|
||||||
|
"payload": event,
|
||||||
|
}
|
||||||
|
broker.publish(topic=topic, body=json.dumps(message))
|
||||||
|
|
||||||
|
# --- Subscription A: welcome-email service ---
|
||||||
|
def subscribe_welcome(broker, dedup_store, send_email):
|
||||||
|
sub = broker.subscribe(topic="users", subscription="welcome-email")
|
||||||
|
for message in sub.receive():
|
||||||
|
# P3: idempotent per subscription. A redelivered message is
|
||||||
|
# a no-op for THIS subscription, not for the others.
|
||||||
|
if dedup_store.seen(("welcome", message["idempotencyKey"])):
|
||||||
|
sub.ack(message)
|
||||||
|
continue
|
||||||
|
send_email(message["payload"]["email"], "Welcome!")
|
||||||
|
dedup_store.mark(("welcome", message["idempotencyKey"]))
|
||||||
|
sub.ack(message)
|
||||||
|
|
||||||
|
# --- Subscription B: analytics-ingest service ---
|
||||||
|
def subscribe_analytics(broker, dedup_store, ingest):
|
||||||
|
# Independent subscription: its own cursor, its own dedup,
|
||||||
|
# its own ack. Welcome-email acking does NOT advance this.
|
||||||
|
sub = broker.subscribe(topic="users", subscription="analytics")
|
||||||
|
for message in sub.receive():
|
||||||
|
if dedup_store.seen(("analytics", message["idempotencyKey"])):
|
||||||
|
sub.ack(message)
|
||||||
|
continue
|
||||||
|
ingest(message["payload"])
|
||||||
|
dedup_store.mark(("analytics", message["idempotencyKey"]))
|
||||||
|
sub.ack(message)
|
||||||
|
```
|
||||||
|
|
||||||
|
- The dedup key is scoped per subscription: `(subscription,
|
||||||
|
idempotencyKey)`. A redelivery to subscription A that was already
|
||||||
|
processed by A is a no-op for A; the same message delivered to
|
||||||
|
subscription B is processed by B independently. Scoping the dedup
|
||||||
|
key by subscription prevents one subscription's dedup from
|
||||||
|
masking another's redelivery.
|
||||||
|
|
||||||
|
## Fan-Out Semantics (P4, P7)
|
||||||
|
|
||||||
|
- Fan-out means every subscription receives every published message
|
||||||
|
(subject to filtering — see below). The broker duplicates the
|
||||||
|
message per subscription; each subscription's delivery is
|
||||||
|
independent. The fan-out ratio is the number of subscriptions; the
|
||||||
|
broker's cost scales with fan-out × message size.
|
||||||
|
- Partitioning (P7) applies to topics that are partitioned for
|
||||||
|
throughput: a partitioned topic delivers per-partition order, and
|
||||||
|
each subscription receives from every partition. A subscription
|
||||||
|
that consumes partitions in parallel must handle per-partition
|
||||||
|
ordering and cross-partition non-ordering (P2 — document the
|
||||||
|
property, do not assume global order).
|
||||||
|
- The delivery semantic is per-subscription (P4): subscription A
|
||||||
|
may be at-least-once, subscription B may be at-most-once (for a
|
||||||
|
loss-tolerant analytics feed). The choice is per subscription,
|
||||||
|
declared, not emergent.
|
||||||
|
|
||||||
|
## Shared vs Independent Subscriptions (P2, P3 — the chaos anti-pattern)
|
||||||
|
|
||||||
|
- An **independent subscription** is one durable cursor per
|
||||||
|
consumer group: each subscription receives every message in
|
||||||
|
topic order (per partition, P2) and acks independently. This is
|
||||||
|
the correct default: per-consumer ordering and per-consumer
|
||||||
|
idempotency hold.
|
||||||
|
- A **shared subscription** is one subscription shared by multiple
|
||||||
|
consumers: the broker dispatches each message to an arbitrary
|
||||||
|
consumer in the shared group. This breaks per-consumer ordering
|
||||||
|
(P2 — consumer A sees message 3 before consumer B sees message
|
||||||
|
1) and complicates idempotency (P3 — the dedup state must be
|
||||||
|
shared across consumers, not per-consumer). This is the
|
||||||
|
`messaging-shared-subscription` chaos anti-pattern
|
||||||
|
(pre-specified in P4 ATELIER-110): the primary breach is P2
|
||||||
|
(ordering); P3 (idempotency) is the compounding consequence.
|
||||||
|
- A shared subscription is correct ONLY when the consumers are
|
||||||
|
stateless, the per-message processing is order-independent, and
|
||||||
|
the dedup store is shared (a shared Redis, a shared DB). A shared
|
||||||
|
subscription for order-dependent or per-consumer-stateful
|
||||||
|
processing is the chaos anti-pattern: the broker's arbitrary
|
||||||
|
dispatch breaks the order the consumer assumes.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# The messaging-shared-subscription chaos anti-pattern (P2
|
||||||
|
# ordering breach, P3 idempotency compounding). Two consumers
|
||||||
|
# share one subscription; the broker dispatches each message to
|
||||||
|
# an arbitrary consumer. Per-consumer ordering breaks; dedup
|
||||||
|
# must be shared (and often is not).
|
||||||
|
|
||||||
|
# BAD — shared subscription, per-consumer dedup (chaos):
|
||||||
|
def shared_subscription_bad(broker, send_email):
|
||||||
|
# Both consumers call subscribe with the SAME subscription
|
||||||
|
# name. The broker round-robins; consumer A gets msg 1, msg 3;
|
||||||
|
# consumer B gets msg 2, msg 4. Per-consumer order is broken.
|
||||||
|
# If each consumer has its OWN dedup store, a redelivery to
|
||||||
|
# the OTHER consumer re-processes (P3 breach).
|
||||||
|
sub = broker.subscribe(topic="users", subscription="shared")
|
||||||
|
for message in sub.receive():
|
||||||
|
# Per-consumer dedup — WRONG. A redelivered message may
|
||||||
|
# land on the other consumer, which has not seen it.
|
||||||
|
if local_dedup.seen(message["idempotencyKey"]): # per-consumer
|
||||||
|
sub.ack(message); continue
|
||||||
|
send_email(message["payload"]["email"], "Welcome!")
|
||||||
|
local_dedup.mark(message["idempotencyKey"])
|
||||||
|
sub.ack(message)
|
||||||
|
|
||||||
|
# CORRECT — independent subscriptions (per-consumer ordering,
|
||||||
|
# per-subscription dedup):
|
||||||
|
def independent_subscriptions_good(broker, send_email):
|
||||||
|
sub = broker.subscribe(topic="users", subscription="welcome-email")
|
||||||
|
for message in sub.receive():
|
||||||
|
if dedup_store.seen(("welcome", message["idempotencyKey"])):
|
||||||
|
sub.ack(message); continue
|
||||||
|
send_email(message["payload"]["email"], "Welcome!")
|
||||||
|
dedup_store.mark(("welcome", message["idempotencyKey"]))
|
||||||
|
sub.ack(message)
|
||||||
|
```
|
||||||
|
|
||||||
|
- If a shared subscription is genuinely required (stateless,
|
||||||
|
order-independent, shared dedup), document the choice and the
|
||||||
|
shared-dedup requirement (P2 — the ordering property is "none
|
||||||
|
across consumers"; P3 — the dedup is shared). The default is
|
||||||
|
independent subscriptions; shared is an opt-in for the narrow case.
|
||||||
|
|
||||||
|
## Filtering (P4, C8 Economy)
|
||||||
|
|
||||||
|
- **Subscription filtering** lets a subscription receive only
|
||||||
|
messages matching a filter (e.g., `event.type == "order"`).
|
||||||
|
Filtering at the broker saves bandwidth (C8 — the subscriber
|
||||||
|
does not receive and discard) and reduces subscriber load.
|
||||||
|
- **Server-side filtering** (broker evaluates the filter before
|
||||||
|
delivery) is more efficient than **client-side filtering**
|
||||||
|
(subscriber receives and discards). Server-side filtering is the
|
||||||
|
default where the broker supports it (GCP Pub/Sub, SNS filtering,
|
||||||
|
NATS subject filtering); client-side is the fallback.
|
||||||
|
- A filter that is too broad wastes bandwidth; a filter that is
|
||||||
|
too narrow drops messages the subscriber needed. The filter is
|
||||||
|
a P1 (contract) and P4 (semantic) decision: the subscription's
|
||||||
|
filter is part of its declared contract.
|
||||||
|
|
||||||
|
## Ordering Across Subscriptions (P2)
|
||||||
|
|
||||||
|
- A topic with per-partition ordering delivers per-partition order
|
||||||
|
to each subscription. Across subscriptions, there is no ordering
|
||||||
|
guarantee: subscription A may ack message 3 while subscription B
|
||||||
|
is still on message 1. This is correct and expected — each
|
||||||
|
subscription is independent.
|
||||||
|
- Within a subscription, ordering holds per partition (P2 — the
|
||||||
|
documented property). A subscription that processes partitions
|
||||||
|
in parallel must not assume cross-partition order. A subscription
|
||||||
|
that needs global order must use a single partition (sacrificing
|
||||||
|
parallelism, P7) or an external sequencing mechanism.
|
||||||
|
- The `messaging-shared-subscription` anti-pattern breaks even
|
||||||
|
per-partition order within a subscription: the broker's arbitrary
|
||||||
|
dispatch to consumers in the shared group breaks the per-
|
||||||
|
partition sequence each consumer sees.
|
||||||
|
|
||||||
|
## Pub/Sub vs Stream — The Durability Boundary (cross-link messaging/streams)
|
||||||
|
|
||||||
|
- Pub/sub and streams are both fan-out or one-to-many primitives,
|
||||||
|
but their durability model differs. Pub/sub is a
|
||||||
|
**push-to-subscription** model: each subscription is a cursor,
|
||||||
|
retention is short (the subscription's unacked window), and
|
||||||
|
replay is limited to the unacked messages. A subscription that
|
||||||
|
falls behind beyond the retention window loses messages
|
||||||
|
permanently.
|
||||||
|
- A stream is a **durable-log** model: messages are retained by
|
||||||
|
the log for a configured window (P8 — Replay and Retention are
|
||||||
|
Configured), and any consumer group can replay from any offset
|
||||||
|
within the window. A stream consumer that falls behind can
|
||||||
|
catch up by replaying; a pub/sub subscription that falls behind
|
||||||
|
beyond retention cannot.
|
||||||
|
- The choice is the durability requirement: if the consumer must
|
||||||
|
be able to replay (reprocessing, backfill, new consumer starting
|
||||||
|
from the beginning), use a stream. If the consumer only needs
|
||||||
|
the live feed (and can tolerate loss on a long fall-behind),
|
||||||
|
pub/sub is lighter. See `domains/messaging/streams.md` for the
|
||||||
|
durable-log model, offsets, and consumer groups.
|
||||||
|
|
||||||
|
## Observability — Per-Subscription Lag (P10)
|
||||||
|
|
||||||
|
- Per-subscription lag (messages published minus messages acked
|
||||||
|
for each subscription, or the age of the oldest unacked message
|
||||||
|
per subscription) is the primary pub/sub health signal. Each
|
||||||
|
subscription has its own lag — a fast subscription and a slow
|
||||||
|
subscription on the same topic are independent signals.
|
||||||
|
- A subscription whose lag grows beyond the retention window is a
|
||||||
|
silent data-loss risk: the broker will drop the oldest messages,
|
||||||
|
and the subscription will never see them. Alert on lag relative
|
||||||
|
to retention — lag approaching retention is the loss threshold.
|
||||||
|
- Wire per-subscription lag to `domains/observability/metrics` as
|
||||||
|
an SLI per subscription. A topic with N subscriptions has N lag
|
||||||
|
metrics; a single aggregate hides the slow one. See
|
||||||
|
`domains/observability/metrics` for the generic SLI/SLO
|
||||||
|
discipline.
|
||||||
|
|
||||||
|
## What Violates Pub/Sub Discipline
|
||||||
|
|
||||||
|
| Violation | Principle |
|
||||||
|
|-----------|-----------|
|
||||||
|
| Shared subscription for order-dependent processing (broker dispatch breaks per-consumer order) | P2 Ordering is a Property, Not an Assumption |
|
||||||
|
| Shared subscription with per-consumer dedup (redelivery to the other consumer re-processes) | P3 Consumers are Idempotent |
|
||||||
|
| Schemaless topic (no versioned contract; subscribers parse by guess) | P1 Messages are Contracts |
|
||||||
|
| Subscription with no idempotency dedup (redelivered message duplicates the effect) | P3 Consumers are Idempotent |
|
||||||
|
| Subscription whose lag approaches retention (silent data loss) | P10, `domains/observability/metrics` |
|
||||||
|
| Unstated delivery semantic per subscription (at-least-once vs at-most-once guessed) | P4 Delivery Semantics are Explicit |
|
||||||
|
| Filter that is too narrow (drops messages the subscriber needed) | P1, P4 |
|
||||||
|
| Partitioned topic with no documented per-partition ordering contract | P2 Ordering is a Property, Not an Assumption |
|
||||||
|
| No per-subscription lag metric (slow subscription invisible) | P10 Messaging is Observable |
|
||||||
|
| Cross-partition order assumption within a subscription (no global order guarantee) | P2, P7 |
|
||||||
@@ -0,0 +1,318 @@
|
|||||||
|
# Queues — Derived Rules
|
||||||
|
|
||||||
|
> Derives from `domains/messaging/first-principles.md`. Applies P1
|
||||||
|
> (Messages are Contracts), P3 (Consumers are Idempotent), P4
|
||||||
|
> (Delivery Semantics are Explicit), P5 (Dead-Letter Handling is
|
||||||
|
> Defined), and P6 (Backpressure is Bounded) primarily, with P2
|
||||||
|
> (ordering), P7 (partitioning), and P10 (observable lag). For the
|
||||||
|
> at-least-once / at-most-once / exactly-once decision, see the
|
||||||
|
> comparison table below. Cross-links `domains/concurrency/patterns`
|
||||||
|
> for the in-process bounded-queue analog and
|
||||||
|
> `domains/observability/metrics` for consumer lag.
|
||||||
|
|
||||||
|
## What a Queue Is (P1 Messages are Contracts)
|
||||||
|
|
||||||
|
- A queue is a point-to-point async delivery primitive. A producer
|
||||||
|
enqueues a message; exactly one consumer dequeues and processes
|
||||||
|
it. The message has an explicit, versioned schema (P1): the
|
||||||
|
producer and consumer agree on shape before exchange, and the
|
||||||
|
schema is the boundary — a schemaless message is a defect (the
|
||||||
|
consumer breaks silently on the next shape change).
|
||||||
|
- The boundary is per D-062: messaging owns the cross-process /
|
||||||
|
network-failure-model angle; concurrency owns the in-process
|
||||||
|
analog. A queue is a messaging concern because producer and
|
||||||
|
consumer are separate systems, the broker is the intermediary,
|
||||||
|
and the failure model is network (the message can be lost,
|
||||||
|
duplicated, reordered, or delayed by the broker or the network,
|
||||||
|
not by a thread crash). The in-process bounded buffer
|
||||||
|
(`domains/concurrency/patterns` Pattern 5) is the analog below
|
||||||
|
this boundary — it fails by OOM; a broker-backed queue fails by
|
||||||
|
partition, broker restart, or consumer crash-and-retry.
|
||||||
|
- See `domains/messaging/pubsub.md` for the fan-out (one-to-many)
|
||||||
|
variant and `domains/messaging/streams.md` for the durable-log
|
||||||
|
(replay-from-offset) variant. A queue deletes on ack; a stream
|
||||||
|
retains for replay — the durability boundary is the
|
||||||
|
distinguishing trait.
|
||||||
|
|
||||||
|
## Producer / Consumer Model (P1, P3)
|
||||||
|
|
||||||
|
- The producer enqueues a message with an idempotency key (P3).
|
||||||
|
The consumer dequeues, processes, and acks. If the consumer
|
||||||
|
crashes before acking, the broker redelivers; the idempotency
|
||||||
|
key makes the redelivery safe (the consumer dedups, not the
|
||||||
|
broker).
|
||||||
|
- The idempotency key is per-message, not per-producer or
|
||||||
|
per-session. A consumer that dedups by producer alone will drop
|
||||||
|
distinct messages issued in the same window. Use a UUID per
|
||||||
|
message, or a deterministic key derived from the message content
|
||||||
|
(e.g., `(entity, operation, version)`).
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Producer/consumer pair with idempotency key (P1 contract, P3
|
||||||
|
# idempotency). The producer tags each message with a versioned
|
||||||
|
# schema and a unique idempotency key; the consumer dedups by the
|
||||||
|
# key so a redelivered message is processed once (P3).
|
||||||
|
|
||||||
|
# --- Producer ---
|
||||||
|
import json, uuid
|
||||||
|
|
||||||
|
def enqueue(order, broker):
|
||||||
|
# P1: versioned schema. The message carries its schema version
|
||||||
|
# so the consumer can route by shape (P9 evolution discipline).
|
||||||
|
message = {
|
||||||
|
"schema": "order.created.v1",
|
||||||
|
"id": str(uuid.uuid4()),
|
||||||
|
"idempotencyKey": f"order:{order['id']}:{order['version']}",
|
||||||
|
"payload": order,
|
||||||
|
}
|
||||||
|
broker.send(queue="orders", body=json.dumps(message))
|
||||||
|
# At-least-once by default (P4): the broker acks the send; the
|
||||||
|
# consumer may see this message more than once under retry.
|
||||||
|
|
||||||
|
# --- Consumer ---
|
||||||
|
def consume(broker, dedup_store, process_order):
|
||||||
|
for message in broker.receive(queue="orders"):
|
||||||
|
# P3: idempotent consumer. Dedup by idempotency key before
|
||||||
|
# processing; a redelivered message is a no-op, not a
|
||||||
|
# double-apply.
|
||||||
|
if dedup_store.seen(message["idempotencyKey"]):
|
||||||
|
broker.ack(message) # already processed; skip
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
process_order(message["payload"])
|
||||||
|
dedup_store.mark(message["idempotencyKey"])
|
||||||
|
broker.ack(message) # success; broker drops it
|
||||||
|
except Exception:
|
||||||
|
broker.nack(message) # redeliver (at-least-once, P4)
|
||||||
|
```
|
||||||
|
|
||||||
|
- The dedup store is bounded (P6 — Backpressure is Bounded): a
|
||||||
|
dedup store that grows without bound is a memory leak. Use a TTL
|
||||||
|
window longer than the broker's max-redelivery window, or a
|
||||||
|
bounded LRU. See `domains/messaging/delivery-semantics.md` for
|
||||||
|
the full idempotency-key dedup-store pattern.
|
||||||
|
|
||||||
|
## Ack / Nack (P4 Delivery Semantics are Explicit)
|
||||||
|
|
||||||
|
- **Ack** tells the broker the message was processed; the broker
|
||||||
|
drops it. **Nack** (negative ack) tells the broker the
|
||||||
|
processing failed; the broker redelivers (at-least-once) or
|
||||||
|
routes to a DLQ (after the retry budget — P5).
|
||||||
|
- A consumer that neither acks nor nacks within the visibility
|
||||||
|
timeout causes the broker to redeliver (the broker assumes the
|
||||||
|
consumer died). This is the at-least-once default: the broker
|
||||||
|
prefers duplication to loss.
|
||||||
|
- The semantic is explicit (P4): at-least-once is the default; the
|
||||||
|
consumer must be idempotent (P3). At-most-once is fire-and-forget
|
||||||
|
(no ack; the broker drops on send) — lossy but lowest latency.
|
||||||
|
Exactly-once is at-least-once plus idempotency, or a
|
||||||
|
transactional two-phase commit — see the comparison table below.
|
||||||
|
|
||||||
|
## Visibility Timeouts and Redelivery (P4, P5)
|
||||||
|
|
||||||
|
- The visibility timeout is the window the broker hides a message
|
||||||
|
after delivery, waiting for the ack. If the consumer does not
|
||||||
|
ack within the window, the broker makes the message visible
|
||||||
|
again and redelivers it (to the same consumer or another). This
|
||||||
|
is the at-least-once mechanism: the broker assumes a
|
||||||
|
no-ack-in-time consumer is dead.
|
||||||
|
- The timeout must be longer than the processing time, or the
|
||||||
|
broker redelivers a message the consumer is still processing —
|
||||||
|
causing duplicate processing (which P3 idempotency makes safe,
|
||||||
|
but which wastes resources). A timeout shorter than processing
|
||||||
|
time is a P6 (backpressure) smell: the consumer is too slow for
|
||||||
|
the configured timeout.
|
||||||
|
- Redelivery has a budget (P5): after N redeliveries or a TTL, the
|
||||||
|
message routes to the DLQ. An unbounded retry budget is the
|
||||||
|
`messaging-unbounded-retry` chaos anti-pattern: the consumer
|
||||||
|
never makes progress past the poison message.
|
||||||
|
|
||||||
|
## FIFO vs Standard Queues (P2 Ordering is a Property, Not an Assumption)
|
||||||
|
|
||||||
|
- A **standard queue** delivers in arrival order per receive-node
|
||||||
|
but offers no global ordering across shards, no per-message-
|
||||||
|
group ordering, and may redeliver out of order under retry. It
|
||||||
|
is the high-throughput default; ordering is *not* guaranteed
|
||||||
|
(P2: the ordering property is "none" — explicitly documented).
|
||||||
|
- A **FIFO queue** delivers strict per-message-group order: all
|
||||||
|
messages with the same group ID are delivered to one consumer
|
||||||
|
in send order. The cost is throughput (FIFO queues cap at lower
|
||||||
|
TPS) and latency (the broker must sequence per group). The
|
||||||
|
ordering property is "per-group strict" — explicitly documented
|
||||||
|
(P2).
|
||||||
|
- The choice is a P2 decision (which ordering guarantee) and a C8
|
||||||
|
decision (throughput cost). A consumer that assumes FIFO on a
|
||||||
|
standard queue is a P2 violation: the broker does not provide
|
||||||
|
the guarantee the consumer assumes. Document the property; do
|
||||||
|
not assume it.
|
||||||
|
|
||||||
|
## Prefetch and Concurrency (P6 Backpressure is Bounded)
|
||||||
|
|
||||||
|
- **Prefetch** (or max-unacked) bounds how many messages the
|
||||||
|
broker delivers to one consumer without an ack. A prefetch of 1
|
||||||
|
is strict stop-and-wait (lowest throughput, tightest backpressure);
|
||||||
|
a prefetch of N allows the consumer to process N in flight
|
||||||
|
(higher throughput, more memory). An unbounded prefetch is a P6
|
||||||
|
violation: the broker floods the consumer's memory.
|
||||||
|
- **Consumer concurrency** is the number of parallel workers
|
||||||
|
processing from the queue. More workers increase throughput up to
|
||||||
|
the downstream's limit; beyond that, the workers stall the
|
||||||
|
downstream (P6 — the backpressure propagates to the
|
||||||
|
downstream, not the broker).
|
||||||
|
- The prefetch × concurrency product is the in-flight cap. Declare
|
||||||
|
it (P6): an undeclared cap is a defect — the consumer either
|
||||||
|
underutilizes the broker (prefetch too low) or OOMs under load
|
||||||
|
(prefetch too high). This is the cross-process analog of
|
||||||
|
`domains/concurrency/patterns` Pattern 5 (bounded queue with
|
||||||
|
backpressure): concurrency owns the in-process analog;
|
||||||
|
messaging owns the broker-backed instance.
|
||||||
|
|
||||||
|
## Long Polling (P6, C8 Economy)
|
||||||
|
|
||||||
|
- Long polling (or `ReceiveMessage` with a wait-time-seconds)
|
||||||
|
holds the receive request open until a message arrives or the
|
||||||
|
wait expires. This reduces empty-receive round trips (C8
|
||||||
|
economy of the constrained link) and reduces latency-to-first-
|
||||||
|
message (the message is delivered when it arrives, not on the
|
||||||
|
next poll cycle).
|
||||||
|
- Long polling is the default for low-throughput queues: short
|
||||||
|
polling burns CPU on empty receives; long polling waits for
|
||||||
|
work. For high-throughput queues, the broker is usually full
|
||||||
|
enough that long polling adds no latency; for low-throughput
|
||||||
|
queues, long polling is the difference between 20ms and 20s
|
||||||
|
latency-to-first-message.
|
||||||
|
|
||||||
|
## Redelivery + DLQ Flow (P5 Dead-Letter Handling is Defined)
|
||||||
|
|
||||||
|
- A poison message (unparseable, repeatedly failing, or exhausting
|
||||||
|
the retry budget) routes to the dead-letter queue. The DLQ is
|
||||||
|
observable (P10 — DLQ depth is an alert) and drainable (an
|
||||||
|
operator can inspect, replay, or discard with audit).
|
||||||
|
- The retry budget is bounded (P6): N redeliveries, or a TTL with
|
||||||
|
exponential backoff. After the budget is exhausted, the message
|
||||||
|
is moved to the DLQ, not retried forever. An unbounded retry is
|
||||||
|
the `messaging-unbounded-retry` chaos anti-pattern (P5 breach).
|
||||||
|
- The DLQ routing rule is a redelivery-count or TTL threshold
|
||||||
|
plus a target queue. See `domains/messaging/delivery-semantics.md`
|
||||||
|
for the dead-letter strategy comparison table.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Redelivery + DLQ flow (P5 dead-letter handling, P6 bounded
|
||||||
|
# retry budget). The consumer tracks redelivery count; after the
|
||||||
|
# budget, the message routes to the DLQ. The DLQ is observable
|
||||||
|
# (P10 — depth is an alert) and drainable.
|
||||||
|
|
||||||
|
MAX_REDELIVERIES = 5
|
||||||
|
DLQ = "orders-dlq"
|
||||||
|
|
||||||
|
def consume_with_dlq(broker, dedup_store, process_order):
|
||||||
|
for message in broker.receive(queue="orders"):
|
||||||
|
# P3 idempotency: a redelivered, already-processed message
|
||||||
|
# is acked and skipped (not re-processed, not DLQ'd).
|
||||||
|
if dedup_store.seen(message["idempotencyKey"]):
|
||||||
|
broker.ack(message)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
process_order(message["payload"])
|
||||||
|
dedup_store.mark(message["idempotencyKey"])
|
||||||
|
broker.ack(message)
|
||||||
|
except Exception as exc:
|
||||||
|
# P5: bounded retry budget. After MAX_REDELIVERIES,
|
||||||
|
# route to DLQ — do NOT retry forever.
|
||||||
|
count = message.get("redeliveryCount", 0) + 1
|
||||||
|
if count >= MAX_REDELIVERIES:
|
||||||
|
broker.send(DLQ, body=json.dumps({
|
||||||
|
"original": message,
|
||||||
|
"reason": str(exc),
|
||||||
|
"deadLetteredAt": now_iso(),
|
||||||
|
"redeliveryCount": count,
|
||||||
|
}))
|
||||||
|
broker.ack(message) # remove from the origin queue
|
||||||
|
# P10: the DLQ depth must alert. A DLQ that grows
|
||||||
|
# with no alert is a silent correctness defect.
|
||||||
|
else:
|
||||||
|
# Nack with backoff: the broker redelivers after a
|
||||||
|
# delay. The backoff caps the retry rate (P6).
|
||||||
|
broker.nack(message, delay=exponential_backoff(count))
|
||||||
|
```
|
||||||
|
|
||||||
|
- The `deadLetteredAt` and `reason` fields make the DLQ entry
|
||||||
|
observable and auditable: an operator inspecting the DLQ sees
|
||||||
|
why each message was dead-lettered and when. See
|
||||||
|
`domains/errors/patterns` for the errors-as-data discipline the
|
||||||
|
DLQ entry follows.
|
||||||
|
|
||||||
|
## Delivery Semantics Comparison (D-069)
|
||||||
|
|
||||||
|
| Semantic | Guarantee | Latency Cost | Implementation Cost | When It Fits |
|
||||||
|
|----------|-----------|-------------|---------------------|--------------|
|
||||||
|
| **At-most-once** | A message is delivered 0 or 1 times; loss is possible, duplication is not | Lowest (no ack; fire-and-forget) | Lowest (no ack, no dedup) | Telemetry where a dropped sample is acceptable; high-throughput metrics; MQTT QoS 0; logs where a lost line is tolerable. Never for billing, orders, or any side-effecting operation. |
|
||||||
|
| **At-least-once** | A message is delivered 1 or more times; duplication is possible, loss is not | Low (one ack round-trip) | Medium (consumer must be idempotent — P3; dedup store required) | The default for side-effecting operations: orders, payments, commands. The consumer dedups via idempotency keys (P3); the broker guarantees delivery. Fits the vast majority of broker-backed queues (SQS standard, RabbitMQ ack, MQTT QoS 1). |
|
||||||
|
| **Exactly-once** | A message is delivered exactly 1 time; no loss, no duplication | Highest (two-phase commit or transactional producer+consumer) | Highest (requires transactions, a transactional producer, and a transactional consumer — or at-least-once plus idempotency, which collapses to at-least-once with dedup) | Rare. Kafka transactions (consume-process-produce in one transaction); MQTT QoS 2 (four-step handshake). In practice, "exactly-once" is usually at-least-once plus idempotency (P3) — the broker does not guarantee it; the consumer enforces it. Jepsen analyses verify broker claims. |
|
||||||
|
|
||||||
|
- The default for side-effecting operations is **at-least-once with
|
||||||
|
idempotent consumers** (P3). At-most-once is for loss-tolerant
|
||||||
|
telemetry. Exactly-once is reserved for the narrow case where
|
||||||
|
the consume-process-produce loop must be transactional (Kafka
|
||||||
|
transactions) — and even then, the consumer should be idempotent
|
||||||
|
as defense-in-depth.
|
||||||
|
- The latency cost column is the C8 tradeoff: at-most-once is
|
||||||
|
cheapest, exactly-once is most expensive. The implementation
|
||||||
|
cost column is the C1/C3 tradeoff: at-most-once is simplest,
|
||||||
|
exactly-once is most complex (and most fragile — a transactional
|
||||||
|
consumer that partially fails is a bug source). The "when it
|
||||||
|
fits" column is the P4 decision: declare the semantic per
|
||||||
|
channel, do not let it emerge.
|
||||||
|
- This is the decision matrix required by D-069 for queues; see
|
||||||
|
`domains/messaging/delivery-semantics.md` for the correctness
|
||||||
|
properties of each semantic and the dead-letter strategy
|
||||||
|
comparison.
|
||||||
|
|
||||||
|
## Observability — Consumer Lag (P10 Messaging is Observable)
|
||||||
|
|
||||||
|
- Consumer lag (messages enqueued minus messages acked, or the
|
||||||
|
age of the oldest unacked message) is the primary queue health
|
||||||
|
signal. A lag that grows without bound is a P6 violation (the
|
||||||
|
consumer is slower than the producer) and a P10 violation if it
|
||||||
|
is not alerted.
|
||||||
|
- DLQ depth is the secondary signal: a DLQ that grows is a P5
|
||||||
|
signal (poison messages are accumulating) and a P10 signal if
|
||||||
|
not alerted. Wire both to `domains/observability/metrics` as
|
||||||
|
SLIs with SLOs (e.g., lag < 1000 messages, DLQ depth < 10).
|
||||||
|
- A queue with no lag metric is operating blind (P10 violation):
|
||||||
|
the operator cannot see the consumer falling behind until the
|
||||||
|
downstream effect surfaces.
|
||||||
|
|
||||||
|
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
|
||||||
|
|
||||||
|
- The broker-backed queue is the cross-process analog of the
|
||||||
|
in-process bounded buffer. `domains/concurrency/patterns`
|
||||||
|
Pattern 5 (Bounded Queue with Backpressure) owns the in-process
|
||||||
|
instance ("producer is blocked or signaled" within one program);
|
||||||
|
messaging owns the broker-backed instance above it (the producer
|
||||||
|
is the broker's enqueue, the consumer is the broker's dequeue, the
|
||||||
|
backpressure is the prefetch cap and the lag signal). The
|
||||||
|
failure model differs: in-process fails by OOM; broker-backed
|
||||||
|
fails by network partition, broker restart, or consumer
|
||||||
|
crash-and-retry (D-062).
|
||||||
|
- The cross-link is one-directional outward (messaging →
|
||||||
|
concurrency) per D-026 extended: messaging references concurrency
|
||||||
|
as the in-process foundation; concurrency does not back-link to
|
||||||
|
messaging.
|
||||||
|
|
||||||
|
## What Violates Queue Discipline
|
||||||
|
|
||||||
|
| Violation | Principle |
|
||||||
|
|-----------|-----------|
|
||||||
|
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
|
||||||
|
| Non-idempotent consumer under at-least-once delivery (redelivery doubles the effect) | P3 Consumers are Idempotent |
|
||||||
|
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||||||
|
| No DLQ (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
|
||||||
|
| Unbounded prefetch (broker floods consumer memory) | P6 Backpressure is Bounded |
|
||||||
|
| Unbounded retry budget (no cap; consumer never progresses past the poison) | P5, P6 |
|
||||||
|
| Consumer that assumes FIFO on a standard queue (ordering not guaranteed) | P2 Ordering is a Property, Not an Assumption |
|
||||||
|
| Visibility timeout shorter than processing time (redeliver while still processing) | P4, P6 |
|
||||||
|
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
|
||||||
|
| No consumer-lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
|
||||||
|
| Dedup store that grows without bound (memory leak) | P6 Backpressure is Bounded |
|
||||||
|
| Default prefetch with no rationale (underutilizes or OOMs) | P6, `domains/concurrency/patterns` |
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
# Streams — Derived Rules
|
||||||
|
|
||||||
|
> Derives from `domains/messaging/first-principles.md`. Applies P8
|
||||||
|
> (Replay and Retention are Configured) primarily, with P2
|
||||||
|
> (per-partition ordering), P3 (idempotent consumers), P4 (exactly-
|
||||||
|
> once via transactions), P7 (partitioning), and P10 (observable
|
||||||
|
> consumer-group health). For the Kafka / Kinesis / Pulsar /
|
||||||
|
> NATS JetStream decision, see the stream-platform comparison
|
||||||
|
> table below. For the consumer-group rebalance strategy choice,
|
||||||
|
> see the rebalance enumeration below (IDEATE-41). Cross-links
|
||||||
|
> `domains/messaging/delivery-semantics` for exactly-once via
|
||||||
|
> transactions, `domains/data/schema-design` for stream schema,
|
||||||
|
> and `domains/observability/tracing` for cross-partition traces.
|
||||||
|
|
||||||
|
## What a Stream Is (P8 Replay and Retention are Configured)
|
||||||
|
|
||||||
|
- A stream is a durable-log messaging primitive. Messages are
|
||||||
|
appended to a partitioned, replicated log; consumers read from an
|
||||||
|
offset and advance at their own pace. The log is retained for a
|
||||||
|
configured window (P8) — a stream is a durable log, not a pipe.
|
||||||
|
A consumer that falls behind can catch up by replaying from an
|
||||||
|
earlier offset; a consumer that starts fresh can replay from the
|
||||||
|
beginning (within retention).
|
||||||
|
- The boundary with queues and pub/sub is the durability model. A
|
||||||
|
queue deletes on ack; a pub/sub subscription retains only its
|
||||||
|
unacked window; a stream retains the whole log for the configured
|
||||||
|
retention. This makes a stream replayable (P8 — the
|
||||||
|
reversibility mechanism) and observable as a record (P10 — the
|
||||||
|
log is itself an audit of what happened). See
|
||||||
|
`domains/messaging/pubsub.md` for the pub/sub-vs-stream
|
||||||
|
durability boundary discussion.
|
||||||
|
- The boundary with concurrency is per D-062: messaging owns the
|
||||||
|
cross-process/network-failure-model angle. A stream is a
|
||||||
|
messaging concern because the log spans brokers and consumers
|
||||||
|
across a network, and the failure model is partition, broker
|
||||||
|
restart, or consumer crash-and-retry — not in-process OOM or
|
||||||
|
thread crash.
|
||||||
|
|
||||||
|
## Partitioned Log Model (P2, P7)
|
||||||
|
|
||||||
|
- A stream is partitioned for throughput and parallelism. Each
|
||||||
|
partition is an ordered, append-only log; messages within a
|
||||||
|
partition are strictly ordered (P2 — per-partition strict order
|
||||||
|
is the documented property). Across partitions, there is no
|
||||||
|
ordering guarantee: partition 0 and partition 1 are independent
|
||||||
|
logs.
|
||||||
|
- The partition key (P7 — Partitioning is Intentional) determines
|
||||||
|
which partition a message lands on. A key that hashes evenly
|
||||||
|
spreads load; a key that matches the per-entity ordering need
|
||||||
|
(e.g., `userId` for user events) keeps a user's events on one
|
||||||
|
partition in order; a key that is too coarse (one partition for
|
||||||
|
the whole topic) serializes all traffic.
|
||||||
|
- The partition count is a capacity bound: it caps the parallelism
|
||||||
|
(one consumer per partition per consumer group) and the
|
||||||
|
throughput (each partition has a write-throughput limit). Too
|
||||||
|
few partitions cap parallelism; too many partition overhead the
|
||||||
|
broker (file handles, replication, rebalance cost). The choice
|
||||||
|
is documented (P7), not defaulted.
|
||||||
|
|
||||||
|
## Offsets (P2, P8)
|
||||||
|
|
||||||
|
- An offset is a consumer's position in a partition. The consumer
|
||||||
|
reads from its last committed offset; acking (committing the
|
||||||
|
offset) advances it. A consumer that crashes before committing
|
||||||
|
re-reads from the last committed offset (at-least-once by
|
||||||
|
default, P4) — the consumer must be idempotent (P3).
|
||||||
|
- The offset is per-partition (P2): each partition has its own
|
||||||
|
position, and the consumer commits them independently (or
|
||||||
|
atomically across partitions in a transaction — see below).
|
||||||
|
- Replay (P8) is resetting the offset backward: a consumer can
|
||||||
|
replay from the beginning of retention, from a timestamp, or
|
||||||
|
from a specific offset. This is the durable-log property that
|
||||||
|
distinguishes a stream from a queue.
|
||||||
|
|
||||||
|
## Consumer Groups (P3, P7, P10)
|
||||||
|
|
||||||
|
- A consumer group is a set of consumers sharing the stream's
|
||||||
|
partitions: each partition is assigned to exactly one consumer
|
||||||
|
in the group. The group is the unit of parallelism and the unit
|
||||||
|
of offset tracking. Within a group, each consumer handles its
|
||||||
|
assigned partitions; across groups, each group independently
|
||||||
|
reads the whole stream (the pub/sub fan-out property, per
|
||||||
|
subscription/group).
|
||||||
|
- A consumer in the group is idempotent (P3): under at-least-once
|
||||||
|
(the default), a redelivery after a crash-and-retry re-processes
|
||||||
|
messages. The consumer dedups by idempotency key, or processes
|
||||||
|
deterministically (e.g., a stateful aggregation that overwrites
|
||||||
|
with the latest value).
|
||||||
|
- The consumer group's health is observable (P10): per-partition
|
||||||
|
lag (offset of the consumer vs the log's head), the group's
|
||||||
|
consumption rate, and rebalance events are first-class signals.
|
||||||
|
A group whose lag grows without bound is a P6 (backpressure)
|
||||||
|
smell and a P10 (observability) violation if not alerted.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Consumer-group reading from offsets (P2 per-partition order,
|
||||||
|
# P3 idempotent under at-least-once, P7 partition assignment,
|
||||||
|
# P8 replay from offset). Each consumer in the group handles its
|
||||||
|
# assigned partitions; the group commits offsets atomically or
|
||||||
|
# per-partition.
|
||||||
|
|
||||||
|
def consume_stream(stream, group, dedup_store, process_event):
|
||||||
|
# Assign partitions to this consumer by the group's
|
||||||
|
# rebalance strategy (see the enumeration below).
|
||||||
|
for partition in stream.assigned_partitions(group, consumer=ME):
|
||||||
|
# Read from the last committed offset (P8 — replay by
|
||||||
|
# resetting this offset).
|
||||||
|
offset = stream.committed_offset(group, partition)
|
||||||
|
for message in stream.read(partition, from_offset=offset):
|
||||||
|
# P3: idempotent under at-least-once. A redelivery
|
||||||
|
# after a crash-and-retry re-processes; dedup by key.
|
||||||
|
if dedup_store.seen(message["idempotencyKey"]):
|
||||||
|
stream.commit(group, partition, message["offset"])
|
||||||
|
continue
|
||||||
|
process_event(message["payload"])
|
||||||
|
dedup_store.mark(message["idempotencyKey"])
|
||||||
|
# Commit the offset to advance (P8 — the position is
|
||||||
|
# the replay pointer).
|
||||||
|
stream.commit(group, partition, message["offset"])
|
||||||
|
```
|
||||||
|
|
||||||
|
- The commit-after-process order gives at-least-once (a crash
|
||||||
|
before commit re-reads); the commit-before-process order gives
|
||||||
|
at-most-once (a crash after commit loses the unprocessed
|
||||||
|
message). The default is at-least-once with idempotent consumers
|
||||||
|
(P3, P4).
|
||||||
|
|
||||||
|
## Consumer-Group Rebalance Strategies (IDEATE-41, ATELIER-100 refinement)
|
||||||
|
|
||||||
|
When a consumer joins or leaves the group, the broker must
|
||||||
|
reassign partitions. The rebalance strategy determines the cost
|
||||||
|
and the use-case fit. This enumeration parallels the v0.3
|
||||||
|
IDEATE-30 drift-type enumeration (each strategy with its
|
||||||
|
stop-the-world cost and use-case fit).
|
||||||
|
|
||||||
|
| Strategy | Mechanism | Partition Stop-the-World Cost | Use-Case Fit |
|
||||||
|
|----------|-----------|-------------------------------|--------------|
|
||||||
|
| **Eager rebalance** (stop-the-world) | Every consumer in the group revokes ALL its partitions, the broker reassigns the full partition set, then consumers resume. Every rebalance pauses the whole group. | High — every partition pauses for every rebalance; the whole group stops processing during the revocation+reassignment window. Throughput drops to zero during rebalance. | Simple brokers, small groups, or rarely-rebalancing groups where the simplicity of full revocation outweighs the pause cost. Kafka's legacy protocol (pre-2.4). Avoid for large groups or frequent scale events. |
|
||||||
|
| **Sticky (incremental cooperative) rebalance** | The broker reassigns only the partitions that must move (the joining/leaving consumer's share); existing partitions stay assigned. The rebalance is incremental and cooperative — no full revocation. | Low — only the moving partitions pause; the rest of the group continues processing. The pause is proportional to the changed partition count, not the total. | The default for large groups, frequent scale events, and rolling deploys. Kafka's CooperativeStickyAssignor (2.4+), Pulsar, NATS JetStream. Prefer for any group where a full stop-the-world on every deploy is unacceptable. |
|
||||||
|
| **Cooperative (no-revoke) rebalance** | A subset of sticky where no partition is revoked unless the consumer leaves; only additions are incremental. The strictest minimization of stop-the-world. | Lowest — only added partitions pause; existing assignments are untouched. | Groups where partition assignment is append-only (consumers join but rarely leave). Useful for long-lived consumers with incremental scaling. |
|
||||||
|
|
||||||
|
- The default for any non-trivial group is **sticky/cooperative**:
|
||||||
|
a rolling deploy that triggers an eager rebalance pauses the
|
||||||
|
whole group on every pod restart, which is unacceptable at
|
||||||
|
scale. The eager strategy is a legacy default that survives
|
||||||
|
because it is simple; prefer sticky where the broker supports
|
||||||
|
it.
|
||||||
|
- The "partition stop-the-world cost" column is the P6
|
||||||
|
(backpressure) angle: a full stop-the-world during rebalance
|
||||||
|
causes lag to spike (the consumer is paused, the producer is
|
||||||
|
not). Sticky rebalance bounds the spike to the moving partitions.
|
||||||
|
- A rebalance that pauses without a lag alert is a P10 violation:
|
||||||
|
the operator cannot see the rebalance-induced lag. Wire rebalance
|
||||||
|
events to `domains/observability/metrics` as an event signal.
|
||||||
|
|
||||||
|
## Replay and Retention Windows (P8, C5 Reversibility)
|
||||||
|
|
||||||
|
- Retention is the configured window the log keeps messages: time-
|
||||||
|
based (e.g., 7 days), size-based (e.g., 10 GB per partition), or
|
||||||
|
compacted (keep the latest value per key — a changelog). A
|
||||||
|
stream with no retention is a pipe, not a log (P8 violation); a
|
||||||
|
stream with infinite retention grows until disk exhaustion (P6
|
||||||
|
violation — backpressure on the broker).
|
||||||
|
- Replay (P8) is resetting a consumer's offset to re-read from
|
||||||
|
within the retention window. Use cases: reprocessing after a
|
||||||
|
consumer bug (replay from the timestamp of the buggy deploy),
|
||||||
|
backfilling a new consumer (replay from the beginning), or
|
||||||
|
reindexing (replay to rebuild a derived store).
|
||||||
|
- Compacted topics (Kafka log-compaction, Pulsar compaction) keep
|
||||||
|
the latest value per key and discard older values for the same
|
||||||
|
key. This turns the log into a changelog — a durable
|
||||||
|
materialized view that replays to the current state. Compaction
|
||||||
|
is a P8 (retention) and C5 (reversibility) mechanism: the log
|
||||||
|
retains the current state per key and is replayable to it.
|
||||||
|
|
||||||
|
## Stream Processing (P2, P3, P4)
|
||||||
|
|
||||||
|
- Stream processing is computing over the stream as it arrives:
|
||||||
|
windowing (tumbling, sliding, session windows), joins (stream-
|
||||||
|
stream, stream-table), aggregations (count, sum, per-key
|
||||||
|
windows), and stateful transformations. The processing is
|
||||||
|
per-partition ordered (P2 — a window over a partition is
|
||||||
|
deterministic; a window across partitions is not unless the
|
||||||
|
window is global).
|
||||||
|
- Stream processing consumers are idempotent (P3): a redelivery
|
||||||
|
after a crash re-processes a window; the aggregation must
|
||||||
|
tolerate re-application (e.g., a sum is idempotent under replay
|
||||||
|
if the window is keyed by offset range, not by wall time).
|
||||||
|
- Exactly-once stream processing (P4) requires transactions: the
|
||||||
|
consume-process-produce loop is one transaction — the input
|
||||||
|
offset commit and the output produce are atomic. See the
|
||||||
|
transactional exactly-once producer below.
|
||||||
|
|
||||||
|
## Exactly-Once via Transactions (P4, P3)
|
||||||
|
|
||||||
|
- Exactly-once stream processing is at-least-once plus a
|
||||||
|
transaction: the consumer commits the input offset and produces
|
||||||
|
the output in one transactional operation. If the consumer
|
||||||
|
crashes mid-transaction, neither the offset commit nor the
|
||||||
|
output produce happens — the consumer re-reads from the last
|
||||||
|
committed offset and re-processes (at-least-once), but the
|
||||||
|
transaction ensures the output is produced exactly once.
|
||||||
|
- This is NOT a broker guarantee of exactly-once delivery; it is
|
||||||
|
at-least-once delivery plus idempotent/transactional processing
|
||||||
|
(P3, P4). Jepsen analyses of Kafka transactions confirm the
|
||||||
|
boundaries: the transaction is atomic within the broker, but
|
||||||
|
the downstream sink must be transactional or idempotent too.
|
||||||
|
- See `domains/messaging/delivery-semantics.md` for the full
|
||||||
|
exactly-once-via-idempotency discussion.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Transactional exactly-once producer (P4 exactly-once via
|
||||||
|
# transactions, P3 idempotent produce). The consume-process-
|
||||||
|
# produce loop is one transaction: the input offset commit and
|
||||||
|
# the output produce are atomic. A crash mid-transaction rolls
|
||||||
|
# both back; the consumer re-reads and re-processes.
|
||||||
|
|
||||||
|
def consume_transform_produce(stream, group, txn_producer):
|
||||||
|
# Begin a transaction. All produces and the offset commit in
|
||||||
|
# this block are atomic (P4).
|
||||||
|
with txn_producer.transaction() as txn:
|
||||||
|
for partition in stream.assigned_partitions(group, ME):
|
||||||
|
offset = stream.committed_offset(group, partition)
|
||||||
|
for message in stream.read(partition, from_offset=offset):
|
||||||
|
output = transform(message["payload"])
|
||||||
|
# P3: idempotent produce. The txn producer
|
||||||
|
# dedups by an epoch+sequence so a retried
|
||||||
|
# transaction does not double-produce.
|
||||||
|
txn.produce(
|
||||||
|
topic="enriched-events",
|
||||||
|
key=message["key"],
|
||||||
|
value=output,
|
||||||
|
idempotencyKey=message["idempotencyKey"],
|
||||||
|
)
|
||||||
|
# Commit the input offset within the same
|
||||||
|
# transaction (P4 atomicity). A crash before
|
||||||
|
# txn.commit() rolls this back; the consumer
|
||||||
|
# re-reads from the prior offset.
|
||||||
|
txn.commit_offset(group, partition, message["offset"])
|
||||||
|
# txn.commit() makes the produces and the offset commit
|
||||||
|
# visible atomically. A crash before this point aborts
|
||||||
|
# both; a crash after is safe (idempotent produce — P3).
|
||||||
|
```
|
||||||
|
|
||||||
|
- The transactional producer's idempotency (P3) is the defense
|
||||||
|
against a retried transaction: the broker dedups the output by
|
||||||
|
the producer's epoch and sequence so a re-commit does not
|
||||||
|
double-produce. The transaction (P4) is the defense against a
|
||||||
|
partial failure: the offset and the output commit together.
|
||||||
|
|
||||||
|
## Stream Schema (P1, P9, cross-link data/schema-design)
|
||||||
|
|
||||||
|
- A stream's messages carry a versioned schema (P1). The schema
|
||||||
|
evolves compatibly (P9): a new field the old consumer ignores is
|
||||||
|
backward-compatible; a renamed field the old consumer parses as
|
||||||
|
`undefined` is a P1 violation compounded as an evolution defect.
|
||||||
|
- Stream schemas are often registered in a schema registry
|
||||||
|
(Confluent, Apicurio) that enforces compatibility on produce.
|
||||||
|
A producer that tries to publish an incompatible schema is
|
||||||
|
rejected; the registry is the P1/P9 enforcement point.
|
||||||
|
- See `domains/data/schema-design` for the generic schema-design
|
||||||
|
discipline (Avro, Protobuf, JSON Schema); messaging owns the
|
||||||
|
stream-specific instance — the registry, the per-topic
|
||||||
|
compatibility mode, the consumer-side routing by schema
|
||||||
|
version.
|
||||||
|
|
||||||
|
## Stream-Platform Comparison (D-069)
|
||||||
|
|
||||||
|
| Axis | Apache Kafka | AWS Kinesis | Apache Pulsar | NATS JetStream |
|
||||||
|
|------|--------------|-------------|---------------|----------------|
|
||||||
|
| **Ordering** | Per-partition strict (P2); global only via single-partition topic | Per-shard strict; global only via single shard | Per-partition strict; global via single partition; also supports shared (out-of-order) subscriptions | Per-stream strict; per-subject ordering; global via single stream |
|
||||||
|
| **Partitioning model** | Partitions (immutable count post-creation; increase requires recreate); key→partition by hash | Shards (reshardable: split/merge at runtime); key→shard by hash | Partitions (resizable; Pulsar's layered architecture separates compute from storage); key→partition by hash | Streams (subject-based; republish to resize); key→stream by subject |
|
||||||
|
| **Replay / retention** | Time- or size-based retention; compaction (latest-per-key); replay from offset or timestamp | Time-based retention (24h–365d); replay from sequence number or timestamp; no compaction | Time- or size-based; compaction; replay from offset or timestamp; tiered storage (hot/warm/cold) | Time- or size-based; per-stream max-age; replay from sequence; no native compaction |
|
||||||
|
| **Consumer groups** | Group-coordinated; offsets stored in an internal topic; eager (legacy) or sticky/cooperative (2.4+) rebalance | Enhanced fan-out consumers (per-shard HTTP/2 push); KCL for group coordination; no native group rebalance (shard is the unit) | Group-coordinated; shared or failover subscription modes; cooperative rebalance | Per-stream consumers; durable cursors; no native group rebalance (stream is the unit) |
|
||||||
|
| **Exactly-once** | Transactions (KIP-98): consume-process-produce atomic; idempotent producer (KIP-516) | No native exactly-once; at-least-once with consumer-side dedup (P3) | Transactions: produce-ack atomic; idempotent producer | At-least-once by default; dedup window per stream (P3 idempotency) |
|
||||||
|
| **Use-case fit** | High-throughput durable logs, stream processing (Kafka Streams, Flink), event sourcing, multi-consumer replay | AWS-native streaming, log ingestion, simple ETL within AWS; low operational burden | Cloud-native, geo-replication, tiered storage, mixed pub/sub + streaming; multi-tenant | Lightweight, low-latency, edge-friendly; NATS ecosystem; simpler ops than Kafka |
|
||||||
|
| **Watch out for** | Partition count is fixed at creation (resize requires recreate + republish); rebalance cost on large groups; operational complexity | Shard limits per account; no compaction; retention cap at 365 days; AWS lock-in | Two-arch (BookKeeper + Brokers) operational complexity; smaller ecosystem | Smaller ecosystem; no native compaction; fewer stream-processing libraries |
|
||||||
|
|
||||||
|
- The default for high-throughput durable logs with multi-consumer
|
||||||
|
replay is **Kafka**; for AWS-native streaming, **Kinesis**; for
|
||||||
|
geo-replicated multi-tenant or mixed pub/sub + streaming,
|
||||||
|
**Pulsar**; for lightweight low-latency edge-friendly streaming,
|
||||||
|
**NATS JetStream** (which also cross-links `domains/edge/iot`
|
||||||
|
via MQTT parallels — see the edge↔messaging bidirectionality in
|
||||||
|
`domains/messaging/first-principles.md` §4).
|
||||||
|
- The ordering column is the P2 check: every platform provides
|
||||||
|
per-partition/per-shard strict order; none provides global order
|
||||||
|
across partitions except by single-partition. The replay/
|
||||||
|
retention column is the P8 check: every platform retains for a
|
||||||
|
configured window; replay is from offset or timestamp. The
|
||||||
|
exactly-once column is the P4 check: Kafka and Pulsar provide
|
||||||
|
transactions; Kinesis and JetStream rely on at-least-once plus
|
||||||
|
consumer-side idempotency (P3).
|
||||||
|
|
||||||
|
## Cross-Partition Traces (P10, cross-link observability/tracing)
|
||||||
|
|
||||||
|
- A stream-processing pipeline that fans out across partitions
|
||||||
|
must propagate a trace context per event: the trace ID follows
|
||||||
|
the event from source to processed output, even as the event
|
||||||
|
crosses partition boundaries. Without cross-partion traces, a
|
||||||
|
downstream error cannot be traced back to its source event.
|
||||||
|
- See `domains/observability/tracing` for the generic distributed-
|
||||||
|
tracing discipline (trace context propagation, span
|
||||||
|
correlation). Messaging owns the stream-specific instance: the
|
||||||
|
trace context is a message header, the span boundary is the
|
||||||
|
consume-process-produce edge, and the cross-partition
|
||||||
|
correlation is by trace ID (not by partition — partitions are
|
||||||
|
independent logs, P2).
|
||||||
|
- A stream processor with no trace propagation is a P10
|
||||||
|
violation: the operator cannot trace a processed event back to
|
||||||
|
its source. Wire the trace context into every produce and every
|
||||||
|
consume.
|
||||||
|
|
||||||
|
## What Violates Stream Discipline
|
||||||
|
|
||||||
|
| Violation | Principle |
|
||||||
|
|-----------|-----------|
|
||||||
|
| Stream with no retention (pipe, not log; no replay) | P8 Replay and Retention are Configured |
|
||||||
|
| Infinite retention (grows until disk exhaustion) | P8, P6 |
|
||||||
|
| Consumer group with eager rebalance at scale (full stop-the-world per deploy) | P6, IDEATE-41 (use sticky/cooperative) |
|
||||||
|
| Non-idempotent stream consumer under at-least-once (redelivery re-processes the window) | P3 Consumers are Idempotent |
|
||||||
|
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
|
||||||
|
| Partition count too low (caps parallelism) or too high (overhead) | P7 |
|
||||||
|
| Cross-partition order assumption (no global order guarantee) | P2 Ordering is a Property, Not an Assumption |
|
||||||
|
| Exactly-once claimed without transactional consume-process-produce (P4 violation) | P4 Delivery Semantics are Explicit |
|
||||||
|
| Stream schema with no registry / no compatibility enforcement (silent shape break) | P1, P9, `domains/data/schema-design` |
|
||||||
|
| No per-partition lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
|
||||||
|
| No cross-partition trace propagation (downstream error untraceable) | P10, `domains/observability/tracing` |
|
||||||
|
| Compacted topic treated as a full log (old values already discarded) | P8, C1 (compaction is a retention mode, not a full log) |
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
# Bad Example: Edge Sync Loop (Livelock)
|
||||||
|
|
||||||
|
> A two-node sync that oscillates divergent state forever. This is
|
||||||
|
> the pre-specified `edge-sync-loop` chaos anti-pattern (ATELIER-110).
|
||||||
|
> **Single-breach per D-068:** the principle breached is **Edge P4
|
||||||
|
> (Sync Conflicts are Bounded, Not Infinite)**. The prose explains
|
||||||
|
> why this is a livelock, not eventual consistency.
|
||||||
|
|
||||||
|
## The Code
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Two edge nodes (a tablet and a dispatcher workstation) sync a
|
||||||
|
// work-order status. The merge uses wall-clock timestamps with no
|
||||||
|
// vector clock and no deterministic tiebreak. A clock skew flips
|
||||||
|
// the winner on every merge pass; the state oscillates forever.
|
||||||
|
|
||||||
|
interface WorkOrderState {
|
||||||
|
orderId: string;
|
||||||
|
status: string;
|
||||||
|
wallClock: number; // P4 VIOLATION: wall time, not monotonic
|
||||||
|
nodeId: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// P4 VIOLATION: the merge picks the later wall-clock write as the
|
||||||
|
// winner. Wall time skews across nodes; a skew of even a few
|
||||||
|
// milliseconds flips the winner. With no vector clock, concurrent
|
||||||
|
// writes are not detected; with no deterministic tiebreak, equal
|
||||||
|
// timestamps are resolved by whichever node's clock is ahead.
|
||||||
|
function mergeBad(local: WorkOrderState, remote: WorkOrderState): WorkOrderState {
|
||||||
|
// No vector clock. No monotonic logical clock. No deterministic
|
||||||
|
// tiebreak by nodeId. This is wall-clock-only LWW on durable
|
||||||
|
// state — the row in the CRDT-vs-LWW decision matrix that is a
|
||||||
|
// P4 violation for anything but ephemeral state.
|
||||||
|
if (local.wallClock >= remote.wallClock) return local;
|
||||||
|
return remote;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The sync loop: each node merges the other's state, writes the
|
||||||
|
// result, and the next sync pass flips it back. The loop never
|
||||||
|
// terminates.
|
||||||
|
async function syncLoop(node: "tablet" | "dispatcher", peer: WorkOrderState) {
|
||||||
|
let local = store.get(peer.orderId);
|
||||||
|
while (true) {
|
||||||
|
const merged = mergeBad(local, peer);
|
||||||
|
store.set(merged.orderId, merged);
|
||||||
|
await pushToPeer(merged); // peer receives, merges, pushes back
|
||||||
|
peer = await pullFromPeer(); // peer's clock is now ahead — flips winner
|
||||||
|
local = store.get(peer.orderId); // local re-merges; flips again
|
||||||
|
// The loop runs forever. Convergence is never reached. This is
|
||||||
|
// a livelock: the system is making progress (each pass writes)
|
||||||
|
// but the state never converges.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The tablet sets `status: "completed"` at wall time `10:00:00.500`.
|
||||||
|
The dispatcher's clock is 50ms ahead; it sets `status: "reassigned"`
|
||||||
|
at wall time `10:00:00.550`. The first merge: dispatcher wins
|
||||||
|
(`10:00:00.550 > 10:00:00.500`). The tablet receives `reassigned`,
|
||||||
|
but its clock drifts ahead by 100ms during the next sync pass; it
|
||||||
|
writes `completed` at `10:00:00.650`. The second merge: tablet
|
||||||
|
wins. The dispatcher's clock drifts ahead again; it writes
|
||||||
|
`reassigned` at `10:00:00.750`. The third merge: dispatcher wins.
|
||||||
|
The state flips between `completed` and `reassigned` on every sync
|
||||||
|
pass. The loop runs forever.
|
||||||
|
|
||||||
|
## Why It Violates
|
||||||
|
|
||||||
|
### Sync Conflicts are Bounded, Not Infinite (Edge P4, C1, C5)
|
||||||
|
|
||||||
|
- **The breach:** the merge has no convergence bound. The CRDT-vs-
|
||||||
|
LWW decision matrix in `domains/edge/sync.md` is explicit:
|
||||||
|
wall-clock-only LWW on durable state carries an **unbounded
|
||||||
|
failure mode** — clock skew = oscillation — and is a P4
|
||||||
|
violation for anything but ephemeral state. This code is the
|
||||||
|
matrix's failure column made real.
|
||||||
|
- **Why it is a livelock, not eventual consistency:** eventual
|
||||||
|
consistency guarantees that, in the absence of new writes, all
|
||||||
|
replicas eventually converge. This system never converges even
|
||||||
|
with no new writes: the clock skew alone drives the oscillation.
|
||||||
|
Each sync pass writes (so the system is "busy"), but the state
|
||||||
|
never settles — the definition of a livelock. The P4 contract is
|
||||||
|
that the merge terminates and converges; this merge does neither.
|
||||||
|
- The missing pieces, per the decision matrix:
|
||||||
|
- **No monotonic logical clock** (a hybrid logical clock or
|
||||||
|
vector clock) — wall time skews, and the skew is unbounded.
|
||||||
|
- **No vector clock** — concurrent writes are not detected, so
|
||||||
|
the conflict is invisible; the merge silently flips instead of
|
||||||
|
surfacing.
|
||||||
|
- **No deterministic tiebreak** (e.g., `nodeId`) on equal
|
||||||
|
timestamps — equal wall times are resolved by whichever node's
|
||||||
|
clock is ahead, which is not a stable property.
|
||||||
|
- The P4 bound requires that the merge function be associative,
|
||||||
|
commutative, and idempotent, terminating in one pass. A
|
||||||
|
wall-clock merge with clock skew satisfies none of these: it is
|
||||||
|
not associative (order of merges flips the winner), not
|
||||||
|
idempotent (a re-merge after a clock drift flips the result),
|
||||||
|
and not terminating (the loop runs forever).
|
||||||
|
|
||||||
|
## The Fix
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Fix: replace wall-clock LWW with a CRDT register (state-based
|
||||||
|
// CvRDT) using a monotonic logical clock and a deterministic
|
||||||
|
// tiebreak by nodeId. The merge is now a total order — no
|
||||||
|
// oscillation (P4 bounded). See the CRDT-vs-LWW decision matrix:
|
||||||
|
// for a register on an unreliable network, a state-based CRDT is
|
||||||
|
// the correct row.
|
||||||
|
|
||||||
|
interface LWWRegister<T> {
|
||||||
|
value: T;
|
||||||
|
timestamp: number; // monotonic logical clock (HLC), not wall time
|
||||||
|
nodeId: string; // deterministic tiebreak
|
||||||
|
}
|
||||||
|
|
||||||
|
function mergeLWWRegister<T>(
|
||||||
|
local: LWWRegister<T>,
|
||||||
|
remote: LWWRegister<T>,
|
||||||
|
): LWWRegister<T> {
|
||||||
|
// Associative, commutative, idempotent (P4 bound, P5 idempotent).
|
||||||
|
// (timestamp, nodeId) is a total order — convergence in one pass.
|
||||||
|
if (local.timestamp > remote.timestamp) return local;
|
||||||
|
if (local.timestamp < remote.timestamp) return remote;
|
||||||
|
return local.nodeId > remote.nodeId ? local : remote;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The sync is now one pass: merge, write, done. No loop, no
|
||||||
|
// oscillation. A re-merge of the same two replicas yields the same
|
||||||
|
// result (P5 idempotent), so a retried sync is safe.
|
||||||
|
async function syncOnce(local: LWWRegister<unknown>, remote: LWWRegister<unknown>) {
|
||||||
|
const converged = mergeLWWRegister(local, remote);
|
||||||
|
store.set(converged);
|
||||||
|
await pushToPeer(converged);
|
||||||
|
// Done. No while(true). The merge terminates.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The fix selects the CRDT row of the decision matrix (a register on
|
||||||
|
an unreliable network): a monotonic logical clock (HLC) eliminates
|
||||||
|
clock skew; a deterministic tiebreak by `nodeId` eliminates the
|
||||||
|
equal-timestamp flip. The merge is now a total order that converges
|
||||||
|
in one pass — the P4 bound. See `examples/good/edge-offline-sync.md`
|
||||||
|
for the full good-example version of this pattern.
|
||||||
|
|
||||||
|
## Cross-Domain Links
|
||||||
|
|
||||||
|
- `domains/edge/sync.md` — the CRDT-vs-LWW decision matrix; the
|
||||||
|
wall-clock-only LWW row is the failure mode this example
|
||||||
|
instantiates; the CRDT row is the fix.
|
||||||
|
- `domains/edge/first-principles.md` — P4 (Sync Conflicts are
|
||||||
|
Bounded, Not Infinite) is the principle breached.
|
||||||
|
- `review/anti-patterns.md` — the `edge-sync-loop` chaos
|
||||||
|
anti-pattern (edge P4, C1, C5 — infinite oscillation is a
|
||||||
|
correctness failure, not eventual consistency; the sync is a
|
||||||
|
livelock).
|
||||||
|
- `examples/good/edge-offline-sync.md` — the good-example version
|
||||||
|
of this pattern: a CRDT register merge that converges in one
|
||||||
|
pass.
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
# Bad Example: Messaging Shared Subscription
|
||||||
|
|
||||||
|
> Two consumers share one subscription; the broker dispatches each
|
||||||
|
> message to an arbitrary consumer. Per-consumer ordering breaks;
|
||||||
|
> per-consumer dedup is wrong. This is the pre-specified
|
||||||
|
> `messaging-shared-subscription` chaos anti-pattern (ATELIER-110).
|
||||||
|
> **Single-breach per D-068:** the primary principle breached is
|
||||||
|
> **Messaging P2 (Ordering is a Property, Not an Assumption)**. P3
|
||||||
|
> (Consumers are Idempotent) is noted as the compounding
|
||||||
|
> consequence — the example remains single-breach in its named
|
||||||
|
> violation.
|
||||||
|
|
||||||
|
## The Code
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Two email workers share ONE subscription on the "users" topic.
|
||||||
|
# The broker round-robins: worker A gets msg 1, msg 3; worker B
|
||||||
|
# gets msg 2, msg 4. Per-consumer order is broken (P2 breach).
|
||||||
|
# Each worker has its OWN dedup store — a redelivery to the other
|
||||||
|
# worker re-processes (P3 compounding).
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
def shared_subscription_bad(broker, send_email, worker_id: str):
|
||||||
|
# P2 VIOLATION: both workers call subscribe with the SAME
|
||||||
|
# subscription name. The broker dispatches each message to an
|
||||||
|
# arbitrary worker in the shared group. Worker A sees msg 3
|
||||||
|
# before worker B sees msg 1; per-worker order is broken.
|
||||||
|
sub = broker.subscribe(topic="users", subscription="welcome-shared")
|
||||||
|
|
||||||
|
# P3 compounding: per-worker dedup. A redelivered message may
|
||||||
|
# land on the OTHER worker, which has not seen it, so the
|
||||||
|
# per-worker dedup store does not catch it — the message is
|
||||||
|
# processed twice across the two workers.
|
||||||
|
local_dedup = DedupStore(backend=redis_for(worker_id))
|
||||||
|
|
||||||
|
for message in sub.receive():
|
||||||
|
payload = json.loads(message["body"])
|
||||||
|
if local_dedup.seen(payload["idempotencyKey"]):
|
||||||
|
sub.ack(message); continue
|
||||||
|
send_email(payload["email"], "Welcome!")
|
||||||
|
local_dedup.mark(payload["idempotencyKey"])
|
||||||
|
sub.ack(message)
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
# The two workers are launched with the same subscription name.
|
||||||
|
# Worker A and Worker B both call subscribe("users", "welcome-shared").
|
||||||
|
# The broker sees one shared subscription; it round-robins.
|
||||||
|
|
||||||
|
def start_workers(broker):
|
||||||
|
# Worker A
|
||||||
|
spawn(shared_subscription_bad, broker, send_email, worker_id="A")
|
||||||
|
# Worker B — same subscription name, same topic
|
||||||
|
spawn(shared_subscription_bad, broker, send_email, worker_id="B")
|
||||||
|
# The broker dispatches: A gets msg 1, B gets msg 2, A gets msg 3,
|
||||||
|
# B gets msg 4. If worker A is slow, B may get msg 2 and msg 4
|
||||||
|
# before A acks msg 1. Per-worker order is broken (P2).
|
||||||
|
```
|
||||||
|
|
||||||
|
The scenario: a user signs up, then immediately updates their
|
||||||
|
email preference (two messages in topic order: `user.signed-up.v1`,
|
||||||
|
`user.preference-updated.v1`). The broker dispatches `signed-up` to
|
||||||
|
worker A and `preference-updated` to worker B. Worker B sends the
|
||||||
|
preference email before worker A sends the welcome email — the
|
||||||
|
user sees the preference confirmation before the welcome. Then the
|
||||||
|
broker redelivers `signed-up` (worker A's ack was slow); this time
|
||||||
|
it dispatches to worker B. Worker B's local dedup store has never
|
||||||
|
seen `signed-up` (it was processed by A), so B sends the welcome
|
||||||
|
email again. The user receives two welcome emails. Per-consumer
|
||||||
|
ordering broke (P2); per-consumer dedup did not catch the
|
||||||
|
cross-worker redelivery (P3 compounding).
|
||||||
|
|
||||||
|
## Why It Violates
|
||||||
|
|
||||||
|
### Ordering is a Property, Not an Assumption (Messaging P2, C1, C2)
|
||||||
|
|
||||||
|
- **The breach (primary):** the two workers share one subscription,
|
||||||
|
and the broker dispatches each message to an arbitrary worker in
|
||||||
|
the shared group. Per-consumer ordering is broken: worker A sees
|
||||||
|
`preference-updated` (msg 2) before it sees `signed-up` (msg 1)
|
||||||
|
if the broker round-robins them to different workers. The
|
||||||
|
workers' code assumes topic order, but the shared subscription
|
||||||
|
provides no per-consumer order guarantee — the broker's dispatch
|
||||||
|
is arbitrary. P2 requires that the ordering property be explicit
|
||||||
|
and documented; here it is assumed (topic order) but not provided
|
||||||
|
(arbitrary dispatch). The assumption is wrong.
|
||||||
|
- The P2 contract is that "it's FIFO" is a claim backed by the
|
||||||
|
broker's partitioning contract, not an assumption the consumer
|
||||||
|
makes. A shared subscription's contract is "no per-consumer
|
||||||
|
order"; the workers' code assumes the opposite. See
|
||||||
|
`domains/messaging/pubsub.md` (Shared vs Independent
|
||||||
|
Subscriptions) and `domains/messaging/first-principles.md` P2.
|
||||||
|
|
||||||
|
### Consumers are Idempotent — compounding consequence (Messaging P3, C1)
|
||||||
|
|
||||||
|
- **The compounding consequence (not the named breach):** each
|
||||||
|
worker has its OWN dedup store. A redelivered message that lands
|
||||||
|
on the *other* worker is not in that worker's dedup store, so it
|
||||||
|
is processed again. The dedup is per-worker, but the subscription
|
||||||
|
is shared — the dedup must be shared across workers to be
|
||||||
|
correct under a shared subscription. The per-worker dedup store
|
||||||
|
is wrong for a shared subscription; a shared dedup store (a
|
||||||
|
shared Redis, a shared DB) is required.
|
||||||
|
- Per D-068, the example remains single-breach in its named
|
||||||
|
violation: P2 is the primary breach (the shared subscription
|
||||||
|
breaks ordering); P3 is the compounding consequence (the
|
||||||
|
per-worker dedup is wrong *because* the subscription is shared).
|
||||||
|
If the subscription were independent, per-worker dedup would be
|
||||||
|
correct. The shared subscription is the root cause; P3 is the
|
||||||
|
downstream effect.
|
||||||
|
|
||||||
|
## The Fix
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Fix 1 (default): independent subscriptions. Each consumer gets
|
||||||
|
# its own durable cursor; per-consumer order holds (P2); per-
|
||||||
|
# consumer dedup is correct (P3). This is the default per
|
||||||
|
# domains/messaging/pubsub.md.
|
||||||
|
|
||||||
|
def independent_subscriptions_good(broker, send_email, worker_id: str):
|
||||||
|
# Each worker has its OWN subscription name. The broker
|
||||||
|
# delivers every message to every subscription in topic order.
|
||||||
|
sub = broker.subscribe(
|
||||||
|
topic="users",
|
||||||
|
subscription=f"welcome-{worker_id}", # per-consumer
|
||||||
|
)
|
||||||
|
# Per-consumer dedup is now correct: a redelivery to THIS
|
||||||
|
# worker is caught by THIS worker's dedup store.
|
||||||
|
dedup = DedupStore(backend=redis_for(worker_id))
|
||||||
|
for message in sub.receive():
|
||||||
|
payload = json.loads(message["body"])
|
||||||
|
if dedup.seen(payload["idempotencyKey"]):
|
||||||
|
sub.ack(message); continue
|
||||||
|
send_email(payload["email"], "Welcome!")
|
||||||
|
dedup.mark(payload["idempotencyKey"])
|
||||||
|
sub.ack(message)
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Fix 2 (if a shared subscription is genuinely required): the
|
||||||
|
# consumers must be stateless, the processing order-independent,
|
||||||
|
# AND the dedup store must be SHARED across workers. Document the
|
||||||
|
# ordering property as "none across consumers" (P2 — the property
|
||||||
|
# is explicit, not assumed) and use a shared dedup backend (P3).
|
||||||
|
|
||||||
|
def shared_subscription_stateless(broker, send_email):
|
||||||
|
# P2: document the ordering property. A shared subscription
|
||||||
|
# provides NO per-consumer order; processing must be order-
|
||||||
|
# independent. Do not assume topic order.
|
||||||
|
sub = broker.subscribe(topic="users", subscription="welcome-shared")
|
||||||
|
# P3: SHARED dedup. A redelivery to any worker is caught by the
|
||||||
|
# shared store.
|
||||||
|
shared_dedup = DedupStore(backend=shared_redis)
|
||||||
|
for message in sub.receive():
|
||||||
|
payload = json.loads(message["body"])
|
||||||
|
if shared_dedup.seen(payload["idempotencyKey"]):
|
||||||
|
sub.ack(message); continue
|
||||||
|
send_email(payload["email"], "Welcome!") # order-independent
|
||||||
|
shared_dedup.mark(payload["idempotencyKey"])
|
||||||
|
sub.ack(message)
|
||||||
|
```
|
||||||
|
|
||||||
|
The default is Fix 1 (independent subscriptions): per-consumer
|
||||||
|
ordering holds, per-consumer dedup is correct, and the code is
|
||||||
|
simpler. Fix 2 is the narrow opt-in for genuinely stateless,
|
||||||
|
order-independent processing — and even then, the dedup must be
|
||||||
|
shared. See `domains/messaging/pubsub.md` for the full
|
||||||
|
shared-vs-independent discussion.
|
||||||
|
|
||||||
|
## Cross-Domain Links
|
||||||
|
|
||||||
|
- `domains/messaging/pubsub.md` — the shared-vs-independent
|
||||||
|
subscriptions section; the `messaging-shared-subscription`
|
||||||
|
anti-pattern lives here (pre-specified in P4 ATELIER-110).
|
||||||
|
- `domains/messaging/first-principles.md` — P2 (Ordering is a
|
||||||
|
Property, Not an Assumption) is the primary breach; P3
|
||||||
|
(Consumers are Idempotent) is the compounding consequence.
|
||||||
|
- `domains/messaging/delivery-semantics.md` — the idempotency-key
|
||||||
|
dedup store; the per-subscription dedup key scoping
|
||||||
|
(`(subscription, idempotencyKey)`) that prevents one
|
||||||
|
subscription's dedup from masking another's redelivery.
|
||||||
|
- `review/anti-patterns.md` — the `messaging-shared-subscription`
|
||||||
|
chaos anti-pattern (messaging P2, P3, C1 — shared subscription
|
||||||
|
breaks ordering and dedup).
|
||||||
|
- `examples/good/messaging-idempotent-consumer.md` — the good-
|
||||||
|
example version of the consumer pattern: an independent consumer
|
||||||
|
with a dedup store and a DLQ routing rule.
|
||||||
@@ -0,0 +1,223 @@
|
|||||||
|
# Good Example: Edge Offline-First + Sync Reconcile
|
||||||
|
|
||||||
|
> A field-service tablet app that operates through a 90-minute network
|
||||||
|
> partition, queues writes locally, and reconciles on reconnect using
|
||||||
|
> the CRDT-vs-LWW decision matrix from `domains/edge/sync.md`. Each
|
||||||
|
> aspect cites the edge principle it satisfies.
|
||||||
|
|
||||||
|
## The Code
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// The local store: an offline-first write queue + a CRDT register
|
||||||
|
// per record. The tablet keeps working through the partition; the
|
||||||
|
// queue drains on reconnect; the merge converges (P4 bounded).
|
||||||
|
|
||||||
|
interface LWWRegister<T> {
|
||||||
|
value: T;
|
||||||
|
timestamp: number; // monotonic logical clock (HLC), NOT wall time
|
||||||
|
nodeId: string; // tablet id — deterministic tiebreak (P4)
|
||||||
|
}
|
||||||
|
|
||||||
|
interface QueuedWrite {
|
||||||
|
idempotencyKey: string; // P5: retried syncs are safe
|
||||||
|
collection: string;
|
||||||
|
recordId: string;
|
||||||
|
register: LWWRegister<unknown>;
|
||||||
|
queuedAt: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
class OfflineStore {
|
||||||
|
private queue: QueuedWrite[] = [];
|
||||||
|
private state: Map<string, LWWRegister<unknown>> = new Map();
|
||||||
|
private telemetryBuffer: TelemetryRecord[] = []; // P10 local-first
|
||||||
|
|
||||||
|
// P2: write succeeds offline. The queue is the durable record of
|
||||||
|
// intent; sync is deferred, not blocked.
|
||||||
|
write(collection: string, recordId: string, value: unknown, key: string): void {
|
||||||
|
const register: LWWRegister<unknown> = {
|
||||||
|
value,
|
||||||
|
timestamp: this.hlc.now(), // monotonic; no wall-clock skew
|
||||||
|
nodeId: this.nodeId,
|
||||||
|
};
|
||||||
|
this.state.set(`${collection}:${recordId}`, register);
|
||||||
|
this.queue.push({
|
||||||
|
idempotencyKey: key, // P5: idempotent sync
|
||||||
|
collection, recordId, register,
|
||||||
|
queuedAt: this.hlc.now(),
|
||||||
|
});
|
||||||
|
// P10: buffer the write event locally; forwarded on reconnect.
|
||||||
|
this.telemetryBuffer.push({ event: "local.write", at: Date.now(), key });
|
||||||
|
}
|
||||||
|
|
||||||
|
// P2: read from local state offline. The UI never blocks on the
|
||||||
|
// network.
|
||||||
|
read(collection: string, recordId: string): unknown {
|
||||||
|
return this.state.get(`${collection}:${recordId}`)?.value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// The CRDT register merge — a state-based CvRDT (convergent). Per the
|
||||||
|
// CRDT-vs-LWW decision matrix in domains/edge/sync.md, a state-based
|
||||||
|
// CRDT is the choice when the data model fits a register and the
|
||||||
|
// network is unreliable (full-state merge tolerates dropped ops).
|
||||||
|
|
||||||
|
function mergeLWWRegister<T>(
|
||||||
|
local: LWWRegister<T>,
|
||||||
|
remote: LWWRegister<T>,
|
||||||
|
): LWWRegister<T> {
|
||||||
|
// Associative, commutative, idempotent (P4 bound, P5 idempotent).
|
||||||
|
// (timestamp, nodeId) is a total order — no oscillation.
|
||||||
|
if (local.timestamp > remote.timestamp) return local;
|
||||||
|
if (local.timestamp < remote.timestamp) return remote;
|
||||||
|
return local.nodeId > remote.nodeId ? local : remote; // deterministic tie
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// The reconnect reconcile: drain the offline queue, pull remote
|
||||||
|
// state, three-way merge (CRDT registers), push converged state.
|
||||||
|
// Idempotent keys make a retried reconcile safe (P5).
|
||||||
|
|
||||||
|
async function reconcile(store: OfflineStore, server: SyncServer): Promise<ReconcileReport> {
|
||||||
|
// P5: the reconcile is idempotent. The idempotency key on each
|
||||||
|
// queued write means a retry (network flapped mid-reconcile) does
|
||||||
|
// not double-apply.
|
||||||
|
const queued = store.drainQueue();
|
||||||
|
let pushed = 0, merged = 0, conflicts = 0;
|
||||||
|
|
||||||
|
// 1. Push local writes. The server dedups by idempotencyKey (P5).
|
||||||
|
for (const w of queued) {
|
||||||
|
await server.applyWrite(w.idempotencyKey, w.collection, w.recordId, w.register);
|
||||||
|
pushed++;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Pull remote state for every record we touched + every record
|
||||||
|
// the server changed since our last sync cursor. Merge via CRDT.
|
||||||
|
const remoteRecords = await server.fetchChanged(store.syncCursor());
|
||||||
|
for (const [key, remoteReg] of remoteRecords) {
|
||||||
|
const localReg = store.localRegister(key);
|
||||||
|
if (localReg) {
|
||||||
|
// P4: CRDT merge converges. The merge is a pure function of
|
||||||
|
// two inputs; the total order (timestamp, nodeId) guarantees
|
||||||
|
// no oscillation.
|
||||||
|
const converged = mergeLWWRegister(localReg, remoteReg);
|
||||||
|
if (converged !== localReg) { conflicts++; }
|
||||||
|
store.setLocal(key, converged);
|
||||||
|
merged++;
|
||||||
|
} else {
|
||||||
|
store.setLocal(key, remoteReg); // remote-only record
|
||||||
|
merged++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. P10: flush the buffered telemetry. The partition did not blind
|
||||||
|
// the operator — the events survived on-node.
|
||||||
|
store.flushTelemetry();
|
||||||
|
|
||||||
|
return { pushed, merged, conflicts, converged: true };
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// P10: local-first telemetry. Events are buffered on-node and
|
||||||
|
// forwarded on reconnect. A fire-and-forget pipeline loses data when
|
||||||
|
// the link drops; a local-first buffer survives.
|
||||||
|
|
||||||
|
interface TelemetryRecord { event: string; at: number; key: string; }
|
||||||
|
|
||||||
|
class TelemetryBuffer {
|
||||||
|
private buffer: TelemetryRecord[] = [];
|
||||||
|
|
||||||
|
push(rec: TelemetryRecord): void { this.buffer.push(rec); }
|
||||||
|
|
||||||
|
// Called from reconcile() on reconnect. The buffer is the P10
|
||||||
|
// guarantee: the operator sees the partition-window activity,
|
||||||
|
// not a gap.
|
||||||
|
async flush(sink: TelemetrySink): Promise<void> {
|
||||||
|
for (const rec of this.buffer) { await sink.emit(rec); }
|
||||||
|
this.buffer = [];
|
||||||
|
}
|
||||||
|
|
||||||
|
depth(): number { return this.buffer.length; }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## The Scenario
|
||||||
|
|
||||||
|
A field-service tablet is dispatched to a basement site with no
|
||||||
|
cellular coverage. The technician updates the work-order status
|
||||||
|
(`started`, `parts-ordered`, `completed`) five times over 90
|
||||||
|
minutes. Each write lands in the local store immediately — the UI
|
||||||
|
never blocks on the network. The writes are queued with an
|
||||||
|
idempotency key (the work-order id + a monotonic sequence).
|
||||||
|
|
||||||
|
When the tablet reconnects, the reconcile drains the queue: the
|
||||||
|
server dedups by idempotency key (a retry mid-reconcile does not
|
||||||
|
double-apply). The server also returns a remote update — the
|
||||||
|
dispatcher re-assigned the work order to a different technician at
|
||||||
|
minute 45, then reverted at minute 60. The CRDT merge converges:
|
||||||
|
the register with the later logical timestamp wins; the tiebreak by
|
||||||
|
node id is deterministic. The merge terminates in one pass (P4
|
||||||
|
bounded); it does not oscillate between the dispatcher's revert and
|
||||||
|
the technician's status updates. The buffered telemetry flushes,
|
||||||
|
and the operator sees the full partition-window activity — no gap.
|
||||||
|
|
||||||
|
## Principles Demonstrated
|
||||||
|
|
||||||
|
### Offline is a First-Class State (Edge P2, C1, C5)
|
||||||
|
- The tablet writes and reads through the partition. The UI never
|
||||||
|
blocks on the network; the offline write queue is the durable
|
||||||
|
record of intent. Partition is the norm, not the exception;
|
||||||
|
reconciliation happens on reconnect. An app that crashes on
|
||||||
|
disconnect has no offline state; this app engineers it.
|
||||||
|
- See `domains/edge/offline-first.md` (offline write-queue) and
|
||||||
|
`domains/edge/first-principles.md` P2.
|
||||||
|
|
||||||
|
### Sync Conflicts are Bounded, Not Infinite (Edge P4, C1, C5)
|
||||||
|
- The CRDT register merge is associative, commutative, and
|
||||||
|
idempotent. The total order `(timestamp, nodeId)` guarantees
|
||||||
|
convergence in one pass — no oscillation, no infinite loop. This
|
||||||
|
is the P4 bound: the merge terminates. The CRDT-vs-LWW decision
|
||||||
|
matrix in `domains/edge/sync.md` selected a state-based CRDT
|
||||||
|
because the data model fits a register and the network is
|
||||||
|
unreliable (full-state merge tolerates dropped ops).
|
||||||
|
- See `domains/edge/sync.md` (CRDT-vs-LWW decision matrix, merge
|
||||||
|
semantics) and `domains/edge/first-principles.md` P4.
|
||||||
|
|
||||||
|
### Edge Operations are Idempotent (Edge P5, C1)
|
||||||
|
- Every queued write carries an idempotency key; the server dedups
|
||||||
|
by key. A reconcile retried mid-flap does not double-apply. The
|
||||||
|
merge function is idempotent (`merge(a, a) === a`) — a retried
|
||||||
|
merge of the same two replicas yields the same result. Sync, the
|
||||||
|
retried-by-nature operation, is safe.
|
||||||
|
- See `domains/edge/sync.md` (Merge Semantics, P5) and
|
||||||
|
`domains/edge/first-principles.md` P5.
|
||||||
|
|
||||||
|
### Edge Observability Survives Partition (Edge P10, C7, C5)
|
||||||
|
- Telemetry is buffered on-node (`telemetryBuffer`) and flushed on
|
||||||
|
reconnect. The partition did not blind the operator — the
|
||||||
|
partition-window activity is forwarded, not lost. A
|
||||||
|
fire-and-forget pipeline would have a 90-minute gap; the
|
||||||
|
local-first buffer survives.
|
||||||
|
- See `domains/edge/offline-first.md` (local-first logging) and
|
||||||
|
`domains/edge/first-principles.md` P10.
|
||||||
|
|
||||||
|
## Cross-Domain Links
|
||||||
|
|
||||||
|
- `domains/edge/offline-first.md` — the offline write-queue pattern
|
||||||
|
this app instantiates; the partition-as-norm discipline.
|
||||||
|
- `domains/edge/sync.md` — the CRDT-vs-LWW decision matrix exercised
|
||||||
|
here (state-based CvRDT chosen for register + unreliable network);
|
||||||
|
the merge-semantics properties (associative, commutative,
|
||||||
|
idempotent).
|
||||||
|
- `domains/edge/first-principles.md` — P2, P4, P5, P10 are the
|
||||||
|
principles demonstrated.
|
||||||
|
- `domains/concurrency/patterns` — the in-process immutability
|
||||||
|
analog (`concurrency/P1 Immutability by Default`) that makes the
|
||||||
|
merge a pure function of two inputs.
|
||||||
|
- `domains/observability/metrics` — the generic SLI/SLO discipline
|
||||||
|
the local-first telemetry buffer builds on.
|
||||||
|
- `review/anti-patterns.md` — the `edge-sync-loop` chaos
|
||||||
|
anti-pattern is the inverse of this example's bounded CRDT merge.
|
||||||
@@ -0,0 +1,226 @@
|
|||||||
|
# Good Example: Idempotent Consumer with Dedup + DLQ
|
||||||
|
|
||||||
|
> An orders-consumer that achieves exactly-once-via-idempotency:
|
||||||
|
> at-least-once delivery plus a TTL-bounded dedup store and a DLQ
|
||||||
|
> routing rule. Each aspect cites the messaging principle it
|
||||||
|
> satisfies. Exercises the idempotency + DLQ guidance in
|
||||||
|
> `domains/messaging/delivery-semantics.md`.
|
||||||
|
|
||||||
|
## The Code
|
||||||
|
|
||||||
|
```python
|
||||||
|
# The idempotency-key dedup store (P3). TTL-bounded (P6): a dedup
|
||||||
|
# store with no TTL is a memory leak. The TTL exceeds the broker's
|
||||||
|
# max-redelivery window; beyond it, the key is expired (the broker
|
||||||
|
# has given up).
|
||||||
|
|
||||||
|
import time, json
|
||||||
|
|
||||||
|
DEDUP_TTL_SECONDS = 24 * 3600 # > broker max-redelivery window
|
||||||
|
|
||||||
|
class DedupStore:
|
||||||
|
"""P3 (idempotent), P6 (TTL-bounded). seen() before process;
|
||||||
|
mark() after process; the order gives at-least-once + dedup."""
|
||||||
|
|
||||||
|
def __init__(self, backend):
|
||||||
|
# backend is Redis or a shared DB. MUST be shared across
|
||||||
|
# consumer instances (see messaging/pubsub.md on shared vs
|
||||||
|
# independent subscriptions).
|
||||||
|
self.backend = backend
|
||||||
|
|
||||||
|
def seen(self, key: str) -> bool:
|
||||||
|
ts = self.backend.get(key)
|
||||||
|
if ts is None:
|
||||||
|
return False
|
||||||
|
if time.time() - ts > DEDUP_TTL_SECONDS:
|
||||||
|
self.backend.delete(key) # P6: expired; not a redelivery
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
def mark(self, key: str):
|
||||||
|
self.backend.set(key, time.time(), ttl=DEDUP_TTL_SECONDS)
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
# The idempotent consumer. Order: dedup BEFORE process, mark AFTER
|
||||||
|
# process, ack AFTER mark. A crash before mark re-processes (the
|
||||||
|
# dedup store lacks the key); a crash before ack redelivers and the
|
||||||
|
# dedup store makes the redelivery a no-op (P3). For a
|
||||||
|
# non-idempotent process (a payment that must not double-charge),
|
||||||
|
# process+mark are one DB transaction — exactly-once via
|
||||||
|
# idempotency (P4).
|
||||||
|
|
||||||
|
def consume_orders(broker, dedup: DedupStore, process_order):
|
||||||
|
for message in broker.receive():
|
||||||
|
payload = json.loads(message["body"])
|
||||||
|
|
||||||
|
# P3: dedup BEFORE process. A redelivered message is a
|
||||||
|
# no-op, not a double-apply.
|
||||||
|
if dedup.seen(payload["idempotencyKey"]):
|
||||||
|
broker.ack(message) # already processed; skip
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
# P4: the declared semantic is at-least-once + idempotent
|
||||||
|
# dedup = exactly-once-via-idempotency. For a payment,
|
||||||
|
# process_order + dedup.mark run in one DB transaction
|
||||||
|
# so the mark commits iff the process commits.
|
||||||
|
process_order(payload)
|
||||||
|
dedup.mark(payload["idempotencyKey"])
|
||||||
|
broker.ack(message)
|
||||||
|
|
||||||
|
except TransientError as exc:
|
||||||
|
# P6: bounded retry with backoff. Nack for redelivery;
|
||||||
|
# the broker redelivers after exponential backoff.
|
||||||
|
broker.nack(message, delay=backoff(payload.get("attempt", 0)))
|
||||||
|
|
||||||
|
except (ValueError, SchemaError) as exc:
|
||||||
|
# P5: poison message — unparseable. Route immediately,
|
||||||
|
# do NOT retry (no retry will fix a bad schema).
|
||||||
|
route_to_dlq(broker, message, exc, kind="poison")
|
||||||
|
broker.ack(message)
|
||||||
|
|
||||||
|
except PermanentError as exc:
|
||||||
|
# P5: permanent failure (e.g., not-found dependency).
|
||||||
|
# Retry will not fix it — DLQ now.
|
||||||
|
route_to_dlq(broker, message, exc, kind="dlq")
|
||||||
|
broker.ack(message)
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
# The DLQ routing rule (P5 dead-letter handling, P10 DLQ depth
|
||||||
|
# alert). Distinguishes poison (unparseable; never retried) from
|
||||||
|
# DLQ (exhausted retry budget on a transient). Both carry audit
|
||||||
|
# metadata; both emit a depth metric.
|
||||||
|
|
||||||
|
DLQ = "orders-dlq"
|
||||||
|
POISON = "orders-poison"
|
||||||
|
MAX_RETRY_TTL_SECONDS = 30 * 60 # 30 min retry window
|
||||||
|
|
||||||
|
def route_to_dlq(broker, message, reason, kind: str):
|
||||||
|
target = POISON if kind == "poison" else DLQ
|
||||||
|
broker.send(target, body=json.dumps({
|
||||||
|
"original": message["body"],
|
||||||
|
"reason": str(reason),
|
||||||
|
"kind": kind, # poison vs dlq
|
||||||
|
"deadLetteredAt": now_iso(),
|
||||||
|
"redeliveryCount": message.get("attempt", 0),
|
||||||
|
}))
|
||||||
|
# P10: emit a metric so DLQ depth alerts fire. A DLQ that grows
|
||||||
|
# with no alert is a silent correctness defect (P5/P10).
|
||||||
|
metrics.increment(f"{kind}.depth", tags={"queue": "orders"})
|
||||||
|
|
||||||
|
|
||||||
|
def consume_with_retry_budget(broker, dedup, process_order):
|
||||||
|
# Combines TTL-with-backoff for transient failures (P6) with
|
||||||
|
# poison-queue + DLQ + alert (P5/P10).
|
||||||
|
for message in broker.receive():
|
||||||
|
payload = json.loads(message["body"])
|
||||||
|
if dedup.seen(payload["idempotencyKey"]):
|
||||||
|
broker.ack(message); continue
|
||||||
|
|
||||||
|
first_attempt_ts = payload.get("firstAttemptTs", time.time())
|
||||||
|
attempt = payload.get("attempt", 0)
|
||||||
|
|
||||||
|
try:
|
||||||
|
process_order(payload)
|
||||||
|
dedup.mark(payload["idempotencyKey"])
|
||||||
|
broker.ack(message)
|
||||||
|
|
||||||
|
except TransientError as exc:
|
||||||
|
# P6: if the retry window is exhausted, route to DLQ;
|
||||||
|
# otherwise redeliver with exponential backoff.
|
||||||
|
if time.time() - first_attempt_ts > MAX_RETRY_TTL_SECONDS:
|
||||||
|
route_to_dlq(broker, message, exc, kind="dlq") # P5
|
||||||
|
broker.ack(message)
|
||||||
|
else:
|
||||||
|
broker.nack(message, delay=backoff(attempt))
|
||||||
|
```
|
||||||
|
|
||||||
|
## The Scenario
|
||||||
|
|
||||||
|
An orders queue delivers `order.created` events to the consumer at
|
||||||
|
**at-least-once** (the declared semantic, P4). The broker redelivers
|
||||||
|
on consumer crash or ack-timeout. Three things happen:
|
||||||
|
|
||||||
|
1. **Normal delivery** — the consumer dedups by `idempotencyKey`,
|
||||||
|
processes, marks, acks.
|
||||||
|
2. **Redelivery after a crash before ack** — the consumer crashed
|
||||||
|
after `mark` but before `ack`. The broker redelivers; `seen()`
|
||||||
|
returns true; the consumer acks without re-processing (P3).
|
||||||
|
3. **Poison message** — a malformed JSON body. The consumer routes
|
||||||
|
it to the poison queue immediately (no retry will fix a parse
|
||||||
|
error), acks the origin, and emits a `poison.depth` metric. The
|
||||||
|
operator is paged on poison-queue growth (P10).
|
||||||
|
|
||||||
|
A transient downstream failure (the payments API is briefly 503)
|
||||||
|
retries with exponential backoff for 30 minutes (P6); if it exceeds
|
||||||
|
the budget, the message routes to the DLQ with `reason`,
|
||||||
|
`redeliveryCount`, and `deadLetteredAt` — auditable, drainable,
|
||||||
|
observable (P5). The DLQ depth metric alerts the operator; the DLQ
|
||||||
|
entry's audit metadata lets the operator replay after the bug is
|
||||||
|
fixed (P5 reversibility).
|
||||||
|
|
||||||
|
## Principles Demonstrated
|
||||||
|
|
||||||
|
### Consumers are Idempotent (Messaging P3, C1)
|
||||||
|
- The consumer dedups by idempotency key before processing. A
|
||||||
|
redelivered message is a no-op, not a double-apply. The
|
||||||
|
`process → mark → ack` order gives at-least-once + idempotent
|
||||||
|
dedup; for a non-idempotent process, `process + mark` are one DB
|
||||||
|
transaction (exactly-once via idempotency, P4).
|
||||||
|
- See `domains/messaging/delivery-semantics.md` (idempotency-key
|
||||||
|
dedup store) and `domains/messaging/first-principles.md` P3.
|
||||||
|
|
||||||
|
### Delivery Semantics are Explicit (Messaging P4, C1, C2)
|
||||||
|
- The channel is declared **at-least-once + idempotent consumer** —
|
||||||
|
the engineering practice that collapses to exactly-once under
|
||||||
|
correct dedup (P3). The semantic is not emergent; it is the
|
||||||
|
declared choice per channel. The tradeoff (dedup-store cost,
|
||||||
|
transactional-process complexity) is conscious and documented.
|
||||||
|
- See `domains/messaging/queues.md` (the three-semantics comparison
|
||||||
|
table) and `domains/messaging/delivery-semantics.md` (exactly-
|
||||||
|
once via idempotency).
|
||||||
|
|
||||||
|
### Dead-Letter Handling is Defined (Messaging P5, C1, C5)
|
||||||
|
- Poison messages (unparseable) route immediately to the poison
|
||||||
|
queue — no retry will fix them. Transient failures retry with
|
||||||
|
backoff until the TTL, then route to the DLQ. Both carry audit
|
||||||
|
metadata (`reason`, `redeliveryCount`, `deadLetteredAt`); both are
|
||||||
|
drainable and observable. The DLQ is the reversibility mechanism
|
||||||
|
— a dead-lettered message can be reprocessed after the bug is
|
||||||
|
fixed.
|
||||||
|
- See `domains/messaging/delivery-semantics.md` (dead-letter
|
||||||
|
strategy comparison table, DLQ routing rule) and
|
||||||
|
`domains/messaging/first-principles.md` P5.
|
||||||
|
|
||||||
|
### Messaging is Observable (Messaging P10, C7, C1)
|
||||||
|
- DLQ depth and poison-queue depth are emitted as metrics and wired
|
||||||
|
to alerts. A DLQ that grows silently is a correctness defect;
|
||||||
|
the alert makes it visible. Silent backlog is a bug, not a
|
||||||
|
feature — the operator is paged on growth, not on a customer
|
||||||
|
report.
|
||||||
|
- See `domains/messaging/delivery-semantics.md` (DLQ depth as an
|
||||||
|
alert) and `domains/messaging/first-principles.md` P10.
|
||||||
|
|
||||||
|
## Cross-Domain Links
|
||||||
|
|
||||||
|
- `domains/messaging/delivery-semantics.md` — the idempotency-key
|
||||||
|
dedup-store pattern and the dead-letter strategy comparison table
|
||||||
|
exercised here (TTL-with-backoff + poison-queue + DLQ + alert).
|
||||||
|
- `domains/messaging/queues.md` — the three-semantics comparison
|
||||||
|
table; the ack/nack/redelivery model this consumer uses.
|
||||||
|
- `domains/messaging/first-principles.md` — P3, P4, P5, P10 are the
|
||||||
|
principles demonstrated.
|
||||||
|
- `domains/concurrency/patterns` — the in-process retry/backoff
|
||||||
|
analog (Pattern 6, Timeout on Every Block); messaging owns the
|
||||||
|
broker-backed instance where redelivery comes across a network.
|
||||||
|
- `domains/errors/patterns` — errors as data: a DLQ entry is the
|
||||||
|
async-messaging instance of an error log (observable, auditable,
|
||||||
|
drainable).
|
||||||
|
- `domains/observability/metrics` — the generic SLI/SLO discipline
|
||||||
|
the DLQ-depth alert builds on.
|
||||||
|
- `review/anti-patterns.md` — the `messaging-unbounded-retry` and
|
||||||
|
`messaging-shared-subscription` chaos anti-patterns are the
|
||||||
|
inverse of this example's bounded retry + independent-consumer
|
||||||
|
discipline.
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
# Go Concurrency — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Go's concurrency specifically. Go's distinctive strength (goroutines, channels, context) earns a dedicated concurrency doc rather than a `go-async.md`.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/go.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Goroutines and Structured Concurrency (Concurrency P1 Immutability by Default, C6 Composability)
|
||||||
|
|
||||||
|
- **`go f()` spawns a goroutine; ensure it does not outlive its parent:** an unstructured `go f()` leaks when the parent returns. Use `sync.WaitGroup`, `errgroup.Group`, or a `context`-scoped pattern to bound lifetime.
|
||||||
|
- **`errgroup.WithContext` for structured concurrency:** a `Group` cancels its context on first error; siblings see the cancellation and exit. Mirrors `TaskGroup` semantics cross-language.
|
||||||
|
- **Goroutines share only immutable inputs:** `go process(snap)` where `snap` is a copy. A goroutine sharing a mutable slice with the parent is a race (Concurrency P1 Immutability, P6 No Silent Races).
|
||||||
|
- **No `go` in a library function without a documented lifetime:** a library that spawns unbounded goroutines leaks them into the caller. Either accept a `context.Context` or return a `Stop()` method.
|
||||||
|
|
||||||
|
```go
|
||||||
|
import "golang.org/x/sync/errgroup"
|
||||||
|
|
||||||
|
func fetchAll(ctx context.Context, ids []string) ([]*User, error) {
|
||||||
|
g, ctx := errgroup.WithContext(ctx)
|
||||||
|
results := make([]*User, len(ids))
|
||||||
|
for i, id := range ids {
|
||||||
|
i, id := i, id // capture loop vars
|
||||||
|
g.Go(func() error {
|
||||||
|
u, err := fetchUser(ctx, id)
|
||||||
|
if err != nil { return err }
|
||||||
|
results[i] = u
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if err := g.Wait(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return results, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Channels: Bounded Queues and Backpressure (Concurrency P9 Bounded Queues, C6 Composability)
|
||||||
|
|
||||||
|
- **Bounded channels apply backpressure:** `make(chan T, N)` blocks the sender when full (Concurrency P9 — bounded queues). Unbounded `make(chan T)` lets the producer run ahead and OOM.
|
||||||
|
- **`select` with `default` for non-blocking send/receive:** a `default` case makes the channel a queue with try semantics; without it, the operation blocks.
|
||||||
|
- **Close channel from the sender, never the receiver:** closing a channel signals "no more sends." A receiver closing it is a race; the sender may still be writing.
|
||||||
|
- **One channel, one responsibility:** do not multiplex control and data on the same channel. Use a `select` over multiple channels instead.
|
||||||
|
- **Applies `messaging/queues`:** a bounded Go channel is an in-process broker — bounded buffer, backpressure, at-most-once handoff. The same semantics apply; the broker is local.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func pipeline(ctx context.Context, in <-chan Job, out chan<- Result) {
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case j, ok := <-in:
|
||||||
|
if !ok { return }
|
||||||
|
r := process(j)
|
||||||
|
select {
|
||||||
|
case out <- r:
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
}
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// bounded: backpressure when out is full
|
||||||
|
out := make(chan Result, 16)
|
||||||
|
```
|
||||||
|
|
||||||
|
## context.Context for Cancellation (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
|
||||||
|
|
||||||
|
- **`context.Context` is the first parameter of every I/O function:** `func fetchUser(ctx context.Context, id string) (*User, error)`. A function that does I/O without a `ctx` cannot be cancelled (Concurrency P7).
|
||||||
|
- **`context.WithTimeout` for a deadline:** `ctx, cancel := context.WithTimeout(ctx, 5*time.Second); defer cancel()`. Every external call races against a deadline (Concurrency P8).
|
||||||
|
- **`cancel()` always called, even on success:** `defer cancel()` immediately after creating the context. A leaked context leaks its timer.
|
||||||
|
- **Never store a `context.Context` in a struct:** pass it as a parameter. A struct holding a `ctx` captures a request-scoped value into a long-lived object.
|
||||||
|
- **Applies `concurrency/P7`:** cancellation propagates via `ctx.Done()`. A `select` on `<-ctx.Done()` is the cancel-aware wait.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func fetchWithTimeout(ctx context.Context, url string) (*Response, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||||
|
resp, err := http.DefaultClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
return nil, ErrTimeout
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return resp, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## select and Multiplexed Channels (Concurrency P7 Cancellation Support, C6 Composability)
|
||||||
|
|
||||||
|
- **`select` multiplexes channel operations:** it picks a ready case at random (fair). A `select` with `<-ctx.Done()` plus a data case is the cancel-aware wait.
|
||||||
|
- **`default` makes `select` non-blocking:** use for "send if ready, else drop" (a bounded queue with drop-oldest policy).
|
||||||
|
- **`select {}` blocks forever:** a `select{}` with no cases is a permanent block. Use only in a goroutine that should run until the process exits.
|
||||||
|
- **Applies `concurrency/P7`:** the `select` over `ctx.Done()` and a result channel is the canonical cancel pattern.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func processUntilCancel(ctx context.Context, jobs <-chan Job) {
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case j, ok := <-jobs:
|
||||||
|
if !ok { return }
|
||||||
|
// ...
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## sync Primitives and Lock Scope (Concurrency P3 Boundaries are Locks, Concurrency P5 Lock Minimization)
|
||||||
|
|
||||||
|
- **`sync.Mutex` scoped minimally:** not held across I/O (a `Send` on a channel, an HTTP call). Hold the lock, mutate, release — then do I/O (Concurrency P3 Lock Scope).
|
||||||
|
- **`sync.RWMutex` for read-heavy, `Mutex` for write-heavy:** RWMutex adds overhead; only prefer it when reads dominate by 10x+.
|
||||||
|
- **`sync.Map` for specific cases (append-only, disjoint keys):** not a general `map[K]V` replacement. For most maps, `Mutex` + `map` is clearer and often faster.
|
||||||
|
- **`sync.Once` for one-time init:** `var once sync.Once; once.Do(func(){ init() })`. Idempotent and race-free.
|
||||||
|
- **Applies `concurrency/P5` (lock minimization):** prefer channels over locks; when a lock is needed, hold it for the smallest possible scope.
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Cache struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
items map[string]*User
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Cache) Get(id string) (*User, bool) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
u, ok := c.items[id]
|
||||||
|
return u, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Cache) Set(id string, u *User) {
|
||||||
|
c.mu.Lock()
|
||||||
|
c.items[id] = u
|
||||||
|
c.mu.Unlock() // explicit unlock before any I/O
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Race Detection (Concurrency P6 No Silent Races)
|
||||||
|
|
||||||
|
- **`go test -race` enforces `P6`:** the race detector instruments memory accesses and fails on data races. See `go-tooling.md` for the CI gate.
|
||||||
|
- **Tests must exercise the concurrent path:** a serial test of a `Mutex`-protected map finds no race. Write tests with N goroutines hitting the map under `-race`.
|
||||||
|
- **Applies `concurrency/P6`:** a race detected at test time is a bug fixed; a race undetected is a production heisenbug.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestCacheConcurrent(t *testing.T) {
|
||||||
|
c := &Cache{items: map[string]*User{}}
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
i := i
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
c.Set(strconv.Itoa(i), &User{})
|
||||||
|
c.Get(strconv.Itoa(i))
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P1, P3, P5, P6, P7, P8, P9 traced throughout.
|
||||||
|
- `domains/messaging/queues.md` — bounded Go channels as in-process brokers; backpressure parallels (IDEATE-40).
|
||||||
|
- `domains/errors/patterns.md` — `errgroup` and error propagation in concurrent code.
|
||||||
|
- `languages/go-types.md` — typed channels carry the named types defined there.
|
||||||
|
- `languages/go-tooling.md` — the `-race` CI gate that enforces Concurrency P6.
|
||||||
|
- `languages/go-testing.md` — concurrent tests that exercise the race detector.
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
# Go Testing — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Go testing specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/go.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Table-Driven Tests (Testing P1 Tests as Specification, C2 Clarity)
|
||||||
|
|
||||||
|
- **Table-driven is the Go idiom:** `cases := []struct{ name string; in X; want Y }{...}`; loop with `t.Run(c.name, ...)`. Each case is a subtest with its own name and failure output.
|
||||||
|
- **Test names read as a spec:** `{"rejects empty email", ...}`, `{"returns persisted id", ...}`. A reader understands the unit from the subtest names (Testing P1).
|
||||||
|
- **No `if got != want { t.Fatal() }` shared across cases:** each case asserts independently; a failure in case 3 does not skip cases 4 and 5.
|
||||||
|
- **`t.Run` enables `-run` filtering:** `go test -run TestCreateUser/rejects_empty_email` runs one case. Essential for debugging a single failure.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestCreateUser(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
email string
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{"rejects empty email", "", true},
|
||||||
|
{"rejects missing @", "no-at-sign", true},
|
||||||
|
{"accepts valid email", "a@b.co", false},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
_, err := CreateUser(c.email)
|
||||||
|
if (err != nil) != c.wantErr {
|
||||||
|
t.Fatalf("err=%v, wantErr=%v", err, c.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## t.Parallel for Independence (Testing P2 Independence, Concurrency P10 Test for Race Conditions)
|
||||||
|
|
||||||
|
- **`t.Parallel()` for independent subtests:** each subtest opts in; the runner executes them concurrently. A test that fails under `Parallel` has hidden state (Testing P2 Independence).
|
||||||
|
- **Capture loop variables:** `c := c` inside the loop, or rely on Go 1.22+ per-iteration scoping. A parallel subtest sharing `c` races on the last value.
|
||||||
|
- **Applies `concurrency/P10` (test for races):** parallel tests are the first line of race detection; combine with `-race` for the full safety net.
|
||||||
|
|
||||||
|
```go
|
||||||
|
for _, c := range cases {
|
||||||
|
c := c // capture for parallel
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
_, err := CreateUser(c.email)
|
||||||
|
if (err != nil) != c.wantErr {
|
||||||
|
t.Fatalf("err=%v, wantErr=%v", err, c.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## t.Cleanup for Teardown (Testing P3 Determinism, Testing P2 Independence)
|
||||||
|
|
||||||
|
- **`t.Cleanup(func() { ... })` for teardown:** runs in LIFO order after the test (and its subtests) complete. Replaces `defer` in a helper that does not know when the test ends.
|
||||||
|
- **Per-test state, not shared:** a `setup(t)` helper creates resources and registers cleanup; each test gets its own. A package-level `var` shared across tests is order coupling.
|
||||||
|
- **`t.TempDir()` for filesystem tests:** creates a unique temp dir and cleans up automatically. No manual `os.RemoveAll` and no cross-test contamination.
|
||||||
|
- **Applies `Testing P3` (determinism):** cleanup is tied to the test lifecycle, not a global teardown that may run before or after depending on order.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func setupStore(t *testing.T) *Store {
|
||||||
|
t.Parallel()
|
||||||
|
dir := t.TempDir() // auto-cleaned
|
||||||
|
s, err := OpenStore(filepath.Join(dir, "db"))
|
||||||
|
if err != nil { t.Fatal(err) }
|
||||||
|
t.Cleanup(func() { s.Close() })
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Race Detector (Testing P9 Edge Case Coverage, Concurrency P6 No Silent Races)
|
||||||
|
|
||||||
|
- **`go test -race` in CI, always:** see `go-tooling.md`. The detector is the enforcement of `concurrency/P6`.
|
||||||
|
- **Tests must exercise the concurrent path:** a serial test of a `Mutex`-protected map finds no race. Write tests with N goroutines.
|
||||||
|
- **`-count=1` to disable result caching:** by default, Go caches passing tests. `-count=1` forces re-run; combine with `-race` and parallelism to surface heisenbugs.
|
||||||
|
- **Applies `Testing P9` (edge case coverage):** the race detector is the edge-case tool for concurrency — it finds the inputs the test author forgot to write.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI gate
|
||||||
|
go test -race -count=1 ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Time and Determinism (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||||
|
|
||||||
|
- **No `time.Now()` in code under test:** inject a `Clock` interface. In tests, a fake clock advances deterministically.
|
||||||
|
- **`time.Sleep` in tests is a smell:** a sleep waits for a real timer, flaky under load. Use a channel or `Eventually`-style polling with a timeout.
|
||||||
|
- **`t.Deadline()` aware helpers:** a helper that may take long checks `t.Deadline()` and bails early. Prevents a slow test from timing out the suite.
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Clock interface { Now() time.Time }
|
||||||
|
|
||||||
|
type fakeClock struct{ t time.Time }
|
||||||
|
func (f *fakeClock) Now() time.Time { return f.t }
|
||||||
|
|
||||||
|
func TestUserHasCreatedAt(t *testing.T) {
|
||||||
|
clk := &fakeClock{time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)}
|
||||||
|
u, _ := CreateUserWithClock("a@b.co", clk)
|
||||||
|
if u.CreatedAt.Year() != 2024 {
|
||||||
|
t.Fatalf("year=%d, want 2024", u.CreatedAt.Year())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Mocks and Interfaces (Testing P7 Realism, API P1 Contract Fidelity)
|
||||||
|
|
||||||
|
- **Mock at the interface, not the struct:** `type Store interface { Get(id string) (*User, error) }` in production; `type mockStore struct{ ... }` in test. The interface is the contract (applies `api/P1`).
|
||||||
|
- **`httptest` for HTTP servers:** `httptest.NewServer` gives a real server on a loopback port; no manual socket plumbing.
|
||||||
|
- **`testify/mock` or hand-written mocks:** hand-written for one-off, `testify` for complex sequencing. Avoid mocking frameworks that generate code at runtime (reflection-heavy) — they hide failures behind stack traces.
|
||||||
|
- **Applies `Testing P7` (realism):** mock the boundary (HTTP, DB), not the unit. Mocking the unit under test tests the mock.
|
||||||
|
|
||||||
|
```go
|
||||||
|
type mockStore struct {
|
||||||
|
users map[string]*User
|
||||||
|
got []string
|
||||||
|
}
|
||||||
|
func (m *mockStore) Get(id string) (*User, error) {
|
||||||
|
m.got = append(m.got, id)
|
||||||
|
return m.users[id], nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGetUserLogs(t *testing.T) {
|
||||||
|
s := &mockStore{users: map[string]*User{"abc": {}}}
|
||||||
|
svc := NewService(s)
|
||||||
|
svc.GetUser("abc")
|
||||||
|
if len(s.got) != 1 || s.got[0] != "abc" {
|
||||||
|
t.Fatalf("got=%v", s.got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/testing/pyramid.md` — where unit/integration/race tests sit; the race job is its own layer.
|
||||||
|
- `domains/testing/fixtures.md` — `t.TempDir` and `t.Cleanup` as the fixture discipline.
|
||||||
|
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P6 (race detector), P10 (test for races).
|
||||||
|
- `languages/go-types.md` — the named types tests assert.
|
||||||
|
- `languages/go-concurrency.md` — concurrent tests exercise the patterns from that doc.
|
||||||
|
- `languages/go-tooling.md` — the `go test` flags (`-race`, `-count`, `-run`) detailed here.
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
# Go Tooling — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Go tooling specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/go.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## go vet and golangci-lint (DevOps P2 Automation, C2 Clarity)
|
||||||
|
|
||||||
|
- **`go vet` is the stdlib baseline:** it catches `printf` format mismatches, lock-copy-by-value, and unreachable code. Run on every build.
|
||||||
|
- **`golangci-lint` aggregates vet + dozens of linters:** enable `errcheck` (no `_ = err`), `govet`, `staticcheck`, `ineffassign`, `unused`, `gofmt`, `goimports`. Each enabled linter has a one-line `# reason:` in `.golangci.yml`.
|
||||||
|
- **`errcheck` enforces `errors/P2` (fail loudly):** a discarded error is a silent failure. `errcheck` fails the build on `_ = doX()`.
|
||||||
|
- **`goimports` over `gofmt`:** `goimports` adds missing imports and removes unused ones, in addition to formatting. The format is not debated in review (Clarity C2).
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# .golangci.yml
|
||||||
|
linters:
|
||||||
|
enable:
|
||||||
|
- errcheck # reason: Errors P2 — no swallowed errors
|
||||||
|
- govet
|
||||||
|
- staticcheck
|
||||||
|
- ineffassign
|
||||||
|
- unused
|
||||||
|
- gofmt
|
||||||
|
- goimports
|
||||||
|
linters-settings:
|
||||||
|
errcheck:
|
||||||
|
check-blank: true # fail on _ = fn()
|
||||||
|
```
|
||||||
|
|
||||||
|
## go test -race (Concurrency P6 No Silent Races)
|
||||||
|
|
||||||
|
- **`go test -race` in CI, always:** the race detector instruments memory accesses and fails on data races. It is the primary enforcement of `concurrency/P6` (no silent races).
|
||||||
|
- **`-race` adds overhead; run it in a separate CI job:** the race build is ~2x slower; keep the fast unit-test job and add a race job.
|
||||||
|
- **`-race` requires tests that actually exercise the concurrent path:** a test that calls `Get`/`Set` serially finds no race. Write tests that spawn goroutines hitting the same map.
|
||||||
|
- **Applies `concurrency/P6`:** a race detected is a bug fixed; a race undetected is a heisenbug in production. The detector is the safety net.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI race job
|
||||||
|
go test -race -count=1 ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Module Discipline (DevOps P1 Reproducibility)
|
||||||
|
|
||||||
|
- **`go mod tidy` on every change that touches imports:** removes unused deps and adds missing ones. A `go.mod` with stale entries breaks reproducibility.
|
||||||
|
- **`go.sum` committed and verified:** `go mod verify` checks the checksums of the module cache against `go.sum`. A drifted `go.sum` is a supply-chain signal.
|
||||||
|
- **Pinned major versions in `go.mod`:** `require github.com/x/y v1.2.3` pins the minor; a `v1.2.4` patch may auto-update. For applications, consider a `go.mod` proxy that pins to exact commits.
|
||||||
|
- **`go mod vendor` for hermetic CI:** vendoring `vendor/` into the repo means CI builds without network. The trade-off is repo size; the win is reproducibility (DevOps P1).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI build gate
|
||||||
|
go mod tidy
|
||||||
|
go mod verify
|
||||||
|
go build ./...
|
||||||
|
go test -race ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reproducible Builds (DevOps P1 Reproducibility, C3 Simplicity)
|
||||||
|
|
||||||
|
- **One Go toolchain version, pinned:** `goenv` or `asdf` pins the Go version per repo; a `.go-version` file declares it. A CI job that uses "latest" Go drifts.
|
||||||
|
- **`CGO_ENABLED=0` for static binaries:** a static binary runs in a scratch container with no libc dependency. Set in CI for all release builds.
|
||||||
|
- **`-trimpath` and `-ldflags='-s -w'` for reproducible output:** strips the build path from the binary and removes debug info. Two builds of the same commit produce byte-identical binaries.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Reproducible release build
|
||||||
|
CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o app ./cmd/app
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||||
|
|
||||||
|
- **`go doc` from comments:** package comments and exported-symbol comments are the API docs; `go doc` and `pkg.go.dev` render them. Missing comments on exported symbols fail `revive`/`golint` (Documentation P1).
|
||||||
|
- **`// Example` functions are run by `go test`:** an `ExampleUser` function with `// Output:` is a tested artifact; a stale output fails the build.
|
||||||
|
- **`README.md` and `docs/` are built by `mkdocs` or similar:** the pipeline validates links and renders; a broken link fails CI (Documentation P1).
|
||||||
|
|
||||||
|
```go
|
||||||
|
// GetUser fetches a user by id.
|
||||||
|
//
|
||||||
|
// Example:
|
||||||
|
//
|
||||||
|
// u, err := GetUser(id)
|
||||||
|
// if err != nil { ... }
|
||||||
|
func GetUser(id UserId) (*User, error) { /* ... */ }
|
||||||
|
|
||||||
|
func ExampleGetUser() {
|
||||||
|
u, err := GetUser("abc")
|
||||||
|
fmt.Println(u, err)
|
||||||
|
// Output: <nil> not found
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/devops/ci-cd.md` — the pipeline gates that host vet/lint/test.
|
||||||
|
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P6 No Silent Races (`-race`).
|
||||||
|
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||||
|
- `languages/go-types.md` — the type rules staticcheck enforces reference this doc.
|
||||||
|
- `languages/go-testing.md` — the `go test` flags (`-race`, `-count`) detailed here.
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
# Go Type System — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Go's type system specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/go.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Named Types for Domain Concepts (C1 Correctness, Data P7 Type Fidelity)
|
||||||
|
|
||||||
|
- **Named types for domain IDs and values:** `type UserId string`, `type OrderId string`. Two named types are distinct even with identical underlying types; the compiler rejects the swap.
|
||||||
|
- **Constructors validate at the boundary:** `func NewUserId(s string) (UserId, error)` returns an error on bad input. A bare `UserId(s)` cast bypasses validation — only the constructor is exported.
|
||||||
|
- **Applies `data/P7` (type fidelity):** a named type carries the domain meaning through the call graph; a `string` parameter does not.
|
||||||
|
- **`any` is the wide type; narrow before use:** Go 1.18+ `any` is an alias for `interface{}`. Use it only at true boundaries (e.g., `json.Unmarshal`); narrow with a type assertion immediately.
|
||||||
|
|
||||||
|
```go
|
||||||
|
type UserId string
|
||||||
|
type OrderId string
|
||||||
|
|
||||||
|
func NewUserId(s string) (UserId, error) {
|
||||||
|
if !regexp.MustCompile(`^[a-z0-9]+$`).MatchString(s) {
|
||||||
|
return "", fmt.Errorf("invalid user id: %q", s)
|
||||||
|
}
|
||||||
|
return UserId(s), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func GetUser(id UserId) (*User, error) { /* ... */ }
|
||||||
|
|
||||||
|
// GetUser("abc") // compile error: string is not UserId
|
||||||
|
// GetUser(OrderId("abc")) // compile error: distinct named types
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generics (C6 Composability, Data P7 Type Fidelity)
|
||||||
|
|
||||||
|
- **Generics (1.18+) preserve element types across containers:** `type Repository[T any] struct { ... }` keeps `T` through `Get`/`Save`, rather than widening to `any`.
|
||||||
|
- **Constrain with `comparable` for map keys, custom interfaces for behavior:** `func dedupe[T comparable](s []T) []T` uses `comparable`; a `Sortable[T]` constraint expresses the `Less` requirement.
|
||||||
|
- **Avoid generics where an interface suffices:** `io.Reader` is not improved by generics. Generics are for type-preserving containers; interfaces are for behavior.
|
||||||
|
- **No generic methods on generic types (not supported):** `func (r Repository[T]) Map[U any](f func(T) U) Repository[U]` is a compile error. Use a free function.
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Entity interface { ID() string }
|
||||||
|
|
||||||
|
type Repository[T Entity] struct {
|
||||||
|
db map[string]T
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repository[T]) Get(id string) (T, bool) {
|
||||||
|
var zero T
|
||||||
|
t, ok := r.db[id]
|
||||||
|
if !ok { return zero, false }
|
||||||
|
return t, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repository[T]) Save(t T) { r.db[t.ID()] = t }
|
||||||
|
```
|
||||||
|
|
||||||
|
## Interfaces (C6 Composability, API P1 Contract Fidelity)
|
||||||
|
|
||||||
|
- **Interfaces defined by the consumer, not the producer:** a package defines its dependencies as interfaces (`type Store interface { Get(id string) (*User, error) }`), and accepts implementations. The producer does not pre-declare "the interface I implement."
|
||||||
|
- **Small interfaces (Go proverb):** `io.Reader` is one method. An interface with 5+ methods is a god-object; split it.
|
||||||
|
- **Accept interfaces, return concrete types:** return a `*UserRepo`, accept a `Store`. The caller gets the implementation; the callee depends on the abstraction.
|
||||||
|
- **Applies `api/P1` (contract fidelity):** the interface is the contract; the concrete type is the implementation. Tests mock the interface, not the struct.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// consumer defines the interface
|
||||||
|
type UserStore interface {
|
||||||
|
Get(id string) (*User, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type Service struct { store UserStore }
|
||||||
|
|
||||||
|
func NewService(s UserStore) *Service { return &Service{store: s} }
|
||||||
|
|
||||||
|
// producer returns concrete; satisfies UserStore implicitly
|
||||||
|
type UserRepo struct { db map[string]*User }
|
||||||
|
func (r *UserRepo) Get(id string) (*User, error) { return r.db[id], nil }
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type Assertion Discipline (C1 Correctness, Errors P1 Errors are Data)
|
||||||
|
|
||||||
|
- **Type assertions return `(T, bool)` — use the bool:** `v, ok := x.(UserId)` distinguishes "wrong type" from "zero value." A bare `x.(UserId)` panics on mismatch.
|
||||||
|
- **`switch x := x.(type)` for multi-variant narrowing:** each case narrows `x` to the case type. The default case is exhaustive (no `never`-style check; Go relies on review).
|
||||||
|
- **Applies `errors/P1` (errors are data):** a failed type assertion is a value (`ok == false`), not an exception. Handle it as a branch, not a panic.
|
||||||
|
- **Never assert across module boundaries silently:** an assertion on a type from another package couples to its internals. Prefer an interface method.
|
||||||
|
|
||||||
|
```go
|
||||||
|
func describe(x any) string {
|
||||||
|
switch v := x.(type) {
|
||||||
|
case UserId:
|
||||||
|
return "user " + string(v)
|
||||||
|
case OrderId:
|
||||||
|
return "order " + string(v)
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("unknown: %T", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// safe form, never panic
|
||||||
|
id, ok := raw.(UserId)
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("expected UserId, got %T", raw)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Types and errors.Is/As (Errors P1 Errors are Data, Errors P3 Fail Specifically)
|
||||||
|
|
||||||
|
- **Sentinel errors for known cases:** `var ErrNotFound = errors.New("not found")`; check with `errors.Is(err, ErrNotFound)`. The sentinel is a value, not an exception class.
|
||||||
|
- **Custom error types for context:** `type ValidationError struct { Field, Msg string }`; check with `var ve *ValidationError; errors.As(err, &ve)`. The type carries structured data (Errors P4 Preserve Context).
|
||||||
|
- **Wrap with `%w`:** `fmt.Errorf("get user %s: %w", id, err)` preserves the chain. `errors.Is`/`As` unwrap it. Bare `%v` breaks the chain.
|
||||||
|
- **Applies `errors/P3` (fail specifically):** `ErrNotFound` is specific; `ErrFailed` is not. The error type names the failure mode.
|
||||||
|
|
||||||
|
```go
|
||||||
|
var ErrNotFound = errors.New("not found")
|
||||||
|
|
||||||
|
type ValidationError struct {
|
||||||
|
Field string
|
||||||
|
Msg string
|
||||||
|
}
|
||||||
|
func (e *ValidationError) Error() string { return e.Field + ": " + e.Msg }
|
||||||
|
|
||||||
|
func GetUser(id UserId) (*User, error) {
|
||||||
|
u, ok := db[string(id)]
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("user %s: %w", id, ErrNotFound)
|
||||||
|
}
|
||||||
|
return u, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// caller
|
||||||
|
if errors.Is(err, ErrNotFound) { /* 404 */ }
|
||||||
|
var ve *ValidationError
|
||||||
|
if errors.As(err, &ve) { /* 422 with ve.Field */ }
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/data/schema-design.md` — named types parallel schema design at the Go boundary.
|
||||||
|
- `domains/data/first-principles.md` — Data P7 Type Fidelity is the primary trace.
|
||||||
|
- `domains/api/rest.md` — contract fidelity for HTTP handlers using interfaces.
|
||||||
|
- `domains/errors/patterns.md` — `errors.Is`/`As` and the wrap-with-`%w` pattern.
|
||||||
|
- `languages/go-concurrency.md` — typed channels carry these named types.
|
||||||
|
- `languages/go-testing.md` — table-driven tests assert type-swap safety.
|
||||||
@@ -2,6 +2,13 @@
|
|||||||
|
|
||||||
> How Atelier's domain principles apply in Go specifically. Derives from `domains/` docs.
|
> How Atelier's domain principles apply in Go specifically. Derives from `domains/` docs.
|
||||||
|
|
||||||
|
## Derived Docs
|
||||||
|
|
||||||
|
- [go-types.md](go-types.md) — named types, generics, interfaces, type assertion discipline.
|
||||||
|
- [go-tooling.md](go-tooling.md) — go vet, golangci-lint, go test -race, module discipline.
|
||||||
|
- [go-concurrency.md](go-concurrency.md) — goroutines, channels, context, select, sync primitives.
|
||||||
|
- [go-testing.md](go-testing.md) — table-driven tests, t.Parallel, t.Cleanup, race detector.
|
||||||
|
|
||||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||||
|
|
||||||
- **Named types for domain concepts:** `type UserId string`, not bare `string`.
|
- **Named types for domain concepts:** `type UserId string`, not bare `string`.
|
||||||
|
|||||||
@@ -0,0 +1,117 @@
|
|||||||
|
# Python Async — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Python async specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/python.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## asyncio and anyio (Concurrency P7 Cancellation Support, C2 Clarity)
|
||||||
|
|
||||||
|
- **`asyncio` for I/O-bound work; threads only for blocking libraries:** `async def` + `await` for network/disk; `run_in_executor` to wrap a blocking call. Mixing threads for I/O is the wrong default.
|
||||||
|
- **`anyio` for runtime portability:** `anyio` abstracts asyncio/trio; a library written against `anyio` runs on either. Use it for libraries; for applications, asyncio directly is fine.
|
||||||
|
- **One event loop, one thread:** `asyncio.run(main())` creates and runs the loop. Do not call `asyncio.run` inside an existing loop (raises `RuntimeError`); do not share a loop across threads.
|
||||||
|
- **Applies `concurrency/P7`:** every `async def` accepts cancellation as a first-class signal; `CancelledError` propagates unless explicitly suppressed (and suppressing it is almost always a bug).
|
||||||
|
|
||||||
|
```python
|
||||||
|
import asyncio
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
async def fetch_user(id: str) -> User:
|
||||||
|
return await api.get(f'/users/{id}')
|
||||||
|
|
||||||
|
# asyncio application
|
||||||
|
async def main():
|
||||||
|
user = await fetch_user('abc')
|
||||||
|
|
||||||
|
asyncio.run(main())
|
||||||
|
|
||||||
|
# anyio library — portable across asyncio/trio
|
||||||
|
async def fetch_all(ids: list[str]) -> list[User]:
|
||||||
|
return await anyio.gather(*(fetch_user(i) for i in ids))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Structured Concurrency (Concurrency P1 Immutability by Default, C6 Composability)
|
||||||
|
|
||||||
|
- **`asyncio.TaskGroup` (3.11+) for structured concurrency:** tasks created in a `TaskGroup` are awaited or cancelled together on exit. No orphan tasks outlive the block.
|
||||||
|
- **No `asyncio.gather(..., return_exceptions=False)` for fallible tasks:** `gather` returns partial results on first exception; `TaskGroup` cancels siblings and propagates the error atomically. Use `TaskGroup` for new code.
|
||||||
|
- **Applies `concurrency/P1` (immutability):** tasks share only immutable inputs; results are collected, not mutated in place. A task that writes to a shared list is a race waiting to happen.
|
||||||
|
- **`anyio.create_task_group()` mirrors `TaskGroup` cross-runtime:** same structured-concurrency guarantee, portable.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
async def fetch_all(ids: list[str]) -> list[User]:
|
||||||
|
results: list[User] = []
|
||||||
|
async with asyncio.TaskGroup() as tg:
|
||||||
|
tasks = [tg.create_task(fetch_user(i)) for i in ids]
|
||||||
|
# all tasks done (or cancelled) by here
|
||||||
|
return [t.result() for t in tasks]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cancellation and Timeout (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
|
||||||
|
|
||||||
|
- **`asyncio.wait_for(coro, timeout)` for a deadline:** every external `await` races against a timeout. A bare `await` is an unbounded wait (Concurrency P8).
|
||||||
|
- **`asyncio.timeout()` (3.11+) as a context manager:** `async with asyncio.timeout(5): await op` — cleaner than `wait_for` for multi-await blocks.
|
||||||
|
- **`CancelledError` propagates; do not catch broadly:** `except Exception` swallows `CancelledError` in 3.7 (it was `BaseException`); in 3.8+ it's `BaseException` and `except Exception` skips it. Catch specifically, never bare `except:`.
|
||||||
|
- **Applies `concurrency/P7`:** cancellation is cooperative — a long synchronous block inside `async def` ignores cancellation. Yield with `await asyncio.sleep(0)` periodically in CPU-bound loops.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
async def fetch_with_timeout(id: str, timeout: float = 5.0) -> User:
|
||||||
|
async with asyncio.timeout(timeout):
|
||||||
|
return await fetch_user(id)
|
||||||
|
|
||||||
|
async def shutdown(token: asyncio.Event) -> None:
|
||||||
|
# cooperative cancel — long-running loop checks the token
|
||||||
|
while not token.is_set():
|
||||||
|
await do_chunk()
|
||||||
|
await asyncio.sleep(0) # yield so cancel can land
|
||||||
|
```
|
||||||
|
|
||||||
|
## Bounded Concurrency and Queues (Concurrency P9 Bounded Queues)
|
||||||
|
|
||||||
|
- **`asyncio.Semaphore(N)` to bound in-flight tasks:** a `Semaphore(8)` wrapping `gather` caps concurrency. Unbounded `gather` on a 10k-item list exhausts file descriptors (Concurrency P9 — bounded queues).
|
||||||
|
- **`asyncio.Queue(maxsize=N)` for producer/consumer:** a bounded queue applies backpressure to the producer. An unbounded queue lets the producer run ahead and OOM.
|
||||||
|
- **Applies `messaging/queues`:** an `asyncio.Queue` is an in-process broker — the same bounded-queue / backpressure semantics apply; the broker is just local.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
async def map_bounded(items: list[str], limit: int = 8) -> list[User]:
|
||||||
|
sem = asyncio.Semaphore(limit)
|
||||||
|
async def guarded(i: str) -> User:
|
||||||
|
async with sem:
|
||||||
|
return await fetch_user(i)
|
||||||
|
return await asyncio.gather(*(guarded(i) for i in items))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling in Async (Errors P5 Recoverable When Possible, Errors P1 Errors are Data)
|
||||||
|
|
||||||
|
- **Retry with backoff for transient failures:** network blips are recoverable (Errors P5). Exponential backoff with jitter, capped retries, and an `anyio`-cancellation-aware `sleep`.
|
||||||
|
- **No retry for non-idempotent operations:** a `POST` creating a resource is not safely retryable without an idempotency key (applies `api/P6` Idempotency).
|
||||||
|
- **`except asyncio.CancelledError: raise`** is the only valid handling — re-raise so the cancellation propagates. Catching and continuing breaks structured concurrency.
|
||||||
|
- **Applies `messaging/delivery-semantics`:** a cancelable async operation is at-most-once; retry-on-cancel is at-least-once. The caller must declare which.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
import random
|
||||||
|
|
||||||
|
async def fetch_retry(id: str, attempts: int = 3) -> User:
|
||||||
|
for i in range(attempts):
|
||||||
|
try:
|
||||||
|
return await fetch_user(id)
|
||||||
|
except (TimeoutError, ConnectionError):
|
||||||
|
if i == attempts - 1:
|
||||||
|
raise
|
||||||
|
await anyio.sleep((2 ** i) * 0.1 + random.random() * 0.1)
|
||||||
|
raise RuntimeError('unreachable')
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P1 Immutability, P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
|
||||||
|
- `domains/messaging/queues.md` — `asyncio.Queue` as an in-process broker; backpressure parallels (IDEATE-40).
|
||||||
|
- `domains/errors/patterns.md` — typed async errors and retry-with-backoff.
|
||||||
|
- `languages/py-types.md` — `Result` and exception hierarchy used in async error handling.
|
||||||
|
- `languages/py-tooling.md` — `pytest-asyncio` config that runs these tests.
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# Python Testing — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Python testing specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/python.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## pytest and Spec-Driven Tests (Testing P1 Tests as Specification, C2 Clarity)
|
||||||
|
|
||||||
|
- **`pytest` is the default; `unittest` only for stdlib-only libraries:** `pytest` fixtures, parametrize, and assertion rewriting beat `unittest`'s boilerplate (Clarity C2).
|
||||||
|
- **Tests co-located with source:** `user.py` → `test_user.py`. A test far from its subject rots (Documentation P5 Discoverability).
|
||||||
|
- **Test names read as a spec:** `def test_create_user_rejects_invalid_email():` — a reader understands the unit from the name. Avoid `def test_user1():`.
|
||||||
|
- **`assert` over `self.assertEqual`:** pytest rewrites `assert` to show the failing values; `assertEqual` is unittest's escape hatch and loses readability.
|
||||||
|
- **Applies `Testing P1`:** the test is a specification; the failure message is the spec violation.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# test_user.py
|
||||||
|
import pytest
|
||||||
|
from user import create_user, ValidationError
|
||||||
|
|
||||||
|
def test_create_user_rejects_invalid_email():
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
create_user(email='not-an-email')
|
||||||
|
|
||||||
|
def test_create_user_returns_persisted_id():
|
||||||
|
u = create_user(email='a@b.co')
|
||||||
|
assert u.id # truthy persisted id
|
||||||
|
```
|
||||||
|
|
||||||
|
## Factories and Fixture Discipline (Testing P2 Independence, Testing P7 Realism)
|
||||||
|
|
||||||
|
- **`factory_boy` or `pytest-factoryboy` over shared fixtures for mutable state:** `UserFactory.build()` returns a fresh object per call; a session-scoped fixture mutated across tests couples them (Testing P2 Independence).
|
||||||
|
- **Fixtures for setup/teardown, factories for data:** a `db` fixture sets up the DB once per test; a `make_user` factory produces fresh data per assertion. Conflating them produces order-dependent tests.
|
||||||
|
- **`scope='function'` is the default and the safe default:** `scope='session'` for read-only resources (a schema migration), never for mutable state.
|
||||||
|
- **Mock at the boundary, not the unit:** `mocker.patch('requests.get')` for HTTP; do not patch `user.User.save` (that mocks the unit under test — Testing P7 realism).
|
||||||
|
|
||||||
|
```python
|
||||||
|
import factory
|
||||||
|
from user import User
|
||||||
|
|
||||||
|
class UserFactory(factory.Factory):
|
||||||
|
class Meta:
|
||||||
|
model = User
|
||||||
|
email = factory.Sequence(lambda n: f'u{n}@b.co')
|
||||||
|
name = 'Test User'
|
||||||
|
|
||||||
|
def test_user_factory_is_fresh():
|
||||||
|
u1 = UserFactory.build()
|
||||||
|
u2 = UserFactory.build()
|
||||||
|
assert u1.email != u2.email # independent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Parametrize and Edge Cases (Testing P9 Edge Case Coverage, Testing P3 Determinism)
|
||||||
|
|
||||||
|
- **`@pytest.mark.parametrize` for input tables:** one parametrized test runs N cases; each is an independent test with its own name and failure output (Testing P9).
|
||||||
|
- **Edge cases as rows, not special tests:** empty list, `None`, max int, unicode — each a row. An ad-hoc `test_handles_edge` with multiple asserts hides which case failed (Testing P6 Failure Specificity).
|
||||||
|
- **`pytest --randomly` catches order coupling:** a test passing alone but failing in a suite has hidden shared state. The random plugin makes it visible (Testing P2 Independence).
|
||||||
|
- **Property tests via `hypothesis`:** for invariants (e.g., "parse(serialize(x)) == x"), `hypothesis` generates hundreds of inputs and shrinks failures to a minimal counterexample.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('email, reason', [
|
||||||
|
('', 'empty'),
|
||||||
|
('a' * 1000 + '@b.co', 'too long'),
|
||||||
|
('no-at-sign', 'missing @'),
|
||||||
|
('a@b', 'missing TLD'),
|
||||||
|
])
|
||||||
|
def test_create_user_rejects(email, reason):
|
||||||
|
with pytest.raises(ValidationError):
|
||||||
|
create_user(email=email)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||||
|
|
||||||
|
- **No `datetime.now()`, `time.time()`, `uuid.uuid4()`, `random.random()` in code under test:** inject a `Clock`, `UUIDGen`, `Random` port. In tests, provide deterministic fakes.
|
||||||
|
- **`freezegun` for time:** `@freeze_time('2024-01-01')` makes `datetime.now()` deterministic. Do not call `datetime.now()` directly in code — wrap it in a `Clock` port so production and tests both inject.
|
||||||
|
- **`pytest --randomly-seed=last` to reproduce a failing order:** when `--randomly` finds an order bug, the seed is logged; re-run with it to debug deterministically.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from freezegun import freeze_time
|
||||||
|
|
||||||
|
@freeze_time('2024-01-01')
|
||||||
|
def test_user_has_created_at():
|
||||||
|
u = create_user(email='a@b.co')
|
||||||
|
assert u.created_at.year == 2024
|
||||||
|
```
|
||||||
|
|
||||||
|
## Async Tests (Concurrency P10 Test for Race Conditions, Testing P1 Tests as Specification)
|
||||||
|
|
||||||
|
- **`pytest-asyncio` (or `anyio`'s pytest plugin) for `async def` tests:** `@pytest.mark.asyncio` runs the coroutine on a loop. Without it, an `async def` test is silently skipped (returns a coroutine, never awaited).
|
||||||
|
- **`anyio`'s plugin runs the same test on asyncio and trio:** one parametrized run across both runtimes catches runtime-specific bugs.
|
||||||
|
- **Race-sensitive tests use `--randomly` and bounded concurrency:** a `Semaphore(1)` test under random order surfaces hidden state.
|
||||||
|
- **Applies `concurrency/P10`:** async tests are the race detector's first line — if a test passes alone but fails under `gather` of N, there's a race.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_async_fetch_returns_user():
|
||||||
|
u = await fetch_user('abc')
|
||||||
|
assert u.email
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/testing/pyramid.md` — where unit/integration/property tests sit; hypothesis is the property layer.
|
||||||
|
- `domains/testing/fixtures.md` — factory-vs-fixture discipline applied via `factory_boy`.
|
||||||
|
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||||
|
- `languages/py-types.md` — the `Result` and Pydantic models that tests assert.
|
||||||
|
- `languages/py-async.md` — async tests use the cancellation/timeout patterns from that doc.
|
||||||
|
- `languages/py-tooling.md` — the `pyproject.toml [tool.pytest]` config that runs these tests.
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
# Python Tooling — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Python tooling specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/python.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## ruff for Lint and Format (DevOps P2 Automation, C2 Clarity)
|
||||||
|
|
||||||
|
- **`ruff` replaces flake8 + black + isort + pyupgrade:** one tool, one config, one order of magnitude faster. Format is not debated in review (Clarity C2).
|
||||||
|
- **Rule selection is principled, not "everything":** `select = ["E", "F", "I", "UP", "B", "SIM"]` — each rule group has a one-line `# reason:` in `pyproject.toml`. Rules without a rationale are noise (Documentation P1 — docs are code).
|
||||||
|
- **`ruff format` is the formatter, `ruff check` is the linter:** run both in CI; the formatter is deterministic, the linter surfaces smells.
|
||||||
|
- **Applies `devops/P2`:** the format/lint gate runs on every push; a developer never waits for a reviewer to comment on style.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# pyproject.toml
|
||||||
|
[tool.ruff]
|
||||||
|
target-version = "py311"
|
||||||
|
line-length = 100
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["E", "F", "I", "UP", "B", "SIM", "RUF"]
|
||||||
|
# reason: E/F = pyflakes+pycodestyle; I = isort; UP = pyupgrade; B = bugbear; SIM = simplification
|
||||||
|
|
||||||
|
[tool.ruff.format]
|
||||||
|
quote-style = "double"
|
||||||
|
```
|
||||||
|
|
||||||
|
## mypy and Type-Check Gate (DevOps P2 Automation, Data P7 Type Fidelity)
|
||||||
|
|
||||||
|
- **`mypy --strict` in CI, not in the editor:** strict flags (`disallow_untyped_defs`, `no_implicit_optional`, `warn_return_any`) are the floor. The editor runs a relaxed mypy for speed; CI runs strict as the gate.
|
||||||
|
- **`pyright` for stricter/async-aware checking:** pyright understands `async` better and reports faster; mypy is the standard. Pick one as the gate, run the other as informational.
|
||||||
|
- **Per-module overrides only with a tracked reason:** `[[tool.mypy.overrides]] module = "legacy.*" ignore_errors = true` — each override block links to a ticket. Untracked overrides accumulate into a permanently untyped core.
|
||||||
|
- **`py.typed` marker for libraries:** ships the type info to consumers. Without it, downstream mypy treats the library as `Any`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI gate
|
||||||
|
mypy --strict src/
|
||||||
|
pyright src/ || true # informational
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dependency Management: poetry and uv (DevOps P1 Reproducibility)
|
||||||
|
|
||||||
|
- **`poetry` or `uv` for lockfile discipline:** both produce a deterministic lock (`poetry.lock` / `uv.lock`). `pip install` alone does not — it resolves at install time, producing different trees across machines.
|
||||||
|
- **`uv` for speed (Rust-based, 10–100x faster):** newer tool, same lockfile semantics. Either is acceptable; do not mix within a repo.
|
||||||
|
- **Lockfile committed for applications:** for libraries, commit the lock for CI reproducibility even though consumers resolve their own tree.
|
||||||
|
- **`--frozen` install in CI:** `poetry install --no-dev --frozen` fails if the lock is out of sync. Prevents a "works on my machine" drift.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI install — deterministic
|
||||||
|
uv sync --frozen --no-dev
|
||||||
|
# or
|
||||||
|
poetry install --no-dev --frozen
|
||||||
|
```
|
||||||
|
|
||||||
|
## Virtualenv Discipline (DevOps P1 Reproducibility, C3 Simplicity)
|
||||||
|
|
||||||
|
- **One virtualenv per project, never the system Python:** `uv venv` or `python -m venv .venv`. System Python drift breaks reproducibility.
|
||||||
|
- **`uv` creates and pins the Python version:** `uv venv --python 3.12` ensures the same interpreter across machines. A pinned Python is part of the reproducibility contract, not just the lockfile.
|
||||||
|
- **No `pip install` into the system Python in CI:** use `uv`/`poetry`'s venv. A CI step that mutates system Python makes the next job non-hermetic.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv venv --python 3.12
|
||||||
|
source .venv/bin/activate
|
||||||
|
uv pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||||
|
|
||||||
|
- **`mkdocs` + `mkdocstrings` from docstrings:** API docs are generated from `google`- or `numpy`-style docstrings; the build fails on missing docstrings for public symbols (Documentation P1).
|
||||||
|
- **`doctest` blocks in docstrings are run by pytest:** a `>>>` example is a tested artifact; a stale example fails the build (Documentation P1, Testing P1).
|
||||||
|
- **`pyproject.toml` is the single source of tool config:** ruff, mypy, pytest, poetry all read from it. Do not scatter `.flake8`, `setup.cfg`, `mypy.ini`. One config file is one place to look (Clarity C2).
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_user(id: UUID) -> User:
|
||||||
|
"""Fetch a user by id.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
id: the user's UUID.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The User.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
NotFoundError: if the user does not exist.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> get_user(UUID('intentional-example-uuid'))
|
||||||
|
User(...)
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/devops/ci-cd.md` — the pipeline gates that host ruff/mypy/poetry.
|
||||||
|
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||||
|
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||||
|
- `languages/py-types.md` — the type rules mypy enforces reference this doc.
|
||||||
|
- `languages/py-testing.md` — the pytest config (`pyproject.toml [tool.pytest]`) detailed here.
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# Python Type System — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Python's type system specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/python.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Type Hints and Gradual Typing (C1 Correctness, Data P7 Type Fidelity)
|
||||||
|
|
||||||
|
- **Type hints on every function signature:** `def get_user(id: UUID) -> User | None:`. Hints are annotations, not enforcement, but `mypy`/`pyright` make them a build gate.
|
||||||
|
- **Gradual typing is opt-in, not opt-out:** start with `--strict` on a package, fix the errors, then expand. A repo-wide `# type: ignore` is a gradual-typing failure.
|
||||||
|
- **`from __future__ import annotations` for forward refs:** all annotations are strings until resolved, so `class User: ...` referencing `User` works without quotes in 3.10+.
|
||||||
|
- **`Any` disables the checker; `object` is the wide type:** `Any` allows any operation; `object` requires narrowing. Use `object` for opaque inputs (e.g., `json.loads` return).
|
||||||
|
- **Applies `data/P7` (type fidelity):** a hint is the contract; the checker verifies it. A missing hint is a missing contract.
|
||||||
|
|
||||||
|
- **Pydantic for runtime validation at the boundary:** hints on `BaseModel` fields are validated at construction, catching bad input from network/config before it deepens into the system.
|
||||||
|
- **`extra='forbid'` by default:** Pydantic allows extra fields silently; forbid them to surface schema drift (e.g., a client sending a typo'd field name).
|
||||||
|
- **Custom types via `Annotated` with validators:** `Email = Annotated[str, validate_email]` keeps the type readable and the validator attached to the type, not the model.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic import BaseModel, ConfigDict
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
|
class UserCreate(BaseModel):
|
||||||
|
model_config = ConfigDict(extra='forbid')
|
||||||
|
email: str
|
||||||
|
name: str
|
||||||
|
|
||||||
|
class User(UserCreate):
|
||||||
|
id: UUID
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pydantic and Schema Fidelity (C1 Correctness, API P1 Contract Fidelity, Data P7 Type Fidelity)
|
||||||
|
|
||||||
|
- **Pydantic models are the API contract:** a FastAPI handler taking `UserCreate` rejects malformed JSON with a 422 before the body runs. This is `api/P1` (contract fidelity) at the type boundary.
|
||||||
|
- **`ConfigDict(extra='forbid')` rejects unknown fields:** silently accepting extras is a contract leak — the server appears to handle fields it ignores.
|
||||||
|
- **Validators raise `ValueError`, not `Exception`:** Pydantic converts `ValueError` to a validation error response; a generic `Exception` becomes a 500 and hides the input bug.
|
||||||
|
- **Applies `api/P1`:** the model is the source of truth; the OpenAPI schema is generated from it, not hand-written. Drift between schema and code is impossible.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import Annotated
|
||||||
|
from pydantic import BaseModel, Field, StringConstraints
|
||||||
|
|
||||||
|
EmailStr = Annotated[str, StringConstraints(pattern=r'^[^@\s]+@[^@\s]+$')]
|
||||||
|
|
||||||
|
class Login(BaseModel):
|
||||||
|
email: EmailStr
|
||||||
|
password: Annotated[str, Field(min_length=8)]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Errors as Data (Errors P1 Errors are Data, C1 Correctness)
|
||||||
|
|
||||||
|
- **`Union[T, Error]` over `Optional[T]` for expected failures:** `Optional[User]` cannot distinguish "not found" from "permission denied". A discriminated `Result` carries the cause.
|
||||||
|
- **Custom exception hierarchy rooted at `AppError`:** `class NotFoundError(AppError)` etc. — callers can `except AppError` for the broad case, or a specific subclass for handling.
|
||||||
|
- **`raise` for exceptional paths, `return Result` for expected:** "user not found" is expected (a `Result`); "DB connection lost" is exceptional (a `raise`). Conflating them makes error handling a guess.
|
||||||
|
- **Applies `errors/P1`:** errors are values, not control-flow magic. A `Result` type encodes this at the type level even where exceptions are the runtime mechanism.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Generic, TypeVar, Union
|
||||||
|
|
||||||
|
T = TypeVar('T')
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Ok(Generic[T]):
|
||||||
|
value: T
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Err:
|
||||||
|
error: Exception
|
||||||
|
|
||||||
|
Result = Union[Ok[T], Err]
|
||||||
|
|
||||||
|
def find_user(id: UUID) -> Result[User]:
|
||||||
|
row = db.get(id)
|
||||||
|
if row is None:
|
||||||
|
return Err(NotFoundError(f'user {id}'))
|
||||||
|
return Ok(User.from_row(row))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generics and Protocols (C6 Composability, C1 Correctness)
|
||||||
|
|
||||||
|
- **`Protocol` for structural typing (PEP 544):** a `Repository` protocol defines `get`/`save` without requiring an inheritance hierarchy; any class matching the shape satisfies it.
|
||||||
|
- **`TypeVar` with bounds for generic functions:** `T = TypeVar('T', bound=Entity)` lets `serialize(t: T) -> dict` access `t.id`.
|
||||||
|
- **`Generic[T]` for container types:** a typed `Repository[T]` preserves the element type across `get`/`save`, rather than widening to `Any`.
|
||||||
|
- **`@overload` for callable overloads:** `def parse(s: str) -> int: ...` vs `def parse(s: bytes) -> int: ...` — the runtime body is one function; the overloads are the type contract.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import Protocol, TypeVar
|
||||||
|
|
||||||
|
T = TypeVar('T')
|
||||||
|
|
||||||
|
class Repository(Protocol[T]):
|
||||||
|
def get(self, id: str) -> T | None: ...
|
||||||
|
def save(self, t: T) -> None: ...
|
||||||
|
|
||||||
|
class UserRepo:
|
||||||
|
def get(self, id: str) -> User | None: ...
|
||||||
|
def save(self, u: User) -> None: ...
|
||||||
|
|
||||||
|
def use_repo(r: Repository[User]) -> None:
|
||||||
|
u = r.get('abc') # type: User | None
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/data/schema-design.md` — Pydantic models parallel schema design at the TS/JSON boundary.
|
||||||
|
- `domains/data/first-principles.md` — Data P7 Type Fidelity is the primary trace for this doc.
|
||||||
|
- `domains/api/rest.md` — contract fidelity for FastAPI handlers consuming Pydantic models.
|
||||||
|
- `domains/errors/patterns.md` — the `Result` discriminated union as error-as-data encoding.
|
||||||
|
- `languages/py-async.md` — typed async results built on the `Result` union here.
|
||||||
|
- `languages/py-tooling.md` — the `mypy`/`pyright` config that enforces these hints.
|
||||||
@@ -2,6 +2,13 @@
|
|||||||
|
|
||||||
> How Atelier's domain principles apply in Python specifically. Derives from `domains/` docs.
|
> How Atelier's domain principles apply in Python specifically. Derives from `domains/` docs.
|
||||||
|
|
||||||
|
## Derived Docs
|
||||||
|
|
||||||
|
- [py-types.md](py-types.md) — type hints + Pydantic, mypy/pyright, gradual typing.
|
||||||
|
- [py-tooling.md](py-tooling.md) — ruff, mypy, poetry, uv, virtualenv discipline.
|
||||||
|
- [py-async.md](py-async.md) — asyncio, anyio, cancellation, structured concurrency.
|
||||||
|
- [py-testing.md](py-testing.md) — pytest, factory_boy, fixture discipline, parametrize.
|
||||||
|
|
||||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||||
|
|
||||||
- **Type hints on every function:** `def get_user(id: UUID) -> User | None:`.
|
- **Type hints on every function:** `def get_user(id: UUID) -> User | None:`.
|
||||||
|
|||||||
@@ -0,0 +1,129 @@
|
|||||||
|
# Rust Async — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Rust async specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/rust.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## tokio and the Async Runtime (Concurrency P5 Lock Minimization, C6 Composability)
|
||||||
|
|
||||||
|
- **`tokio` is the default async runtime:** `#[tokio::main]` for the entry; `tokio::spawn` for a task. The runtime owns the reactor, the I/O driver, and the timer.
|
||||||
|
- **`tokio::spawn` returns a `JoinHandle` like `std::thread::spawn`:** a dropped `JoinHandle` detaches (the task keeps running); `await` the handle to join. Prefer await to detach.
|
||||||
|
- **`tokio::task::JoinSet` for structured concurrency:** a set of tasks awaited together; on drop, all remaining tasks are cancelled. Mirrors `errgroup`/`TaskGroup` semantics.
|
||||||
|
- **`runtime` features are explicit:** `tokio = { version = "1", features = ["full"] }` for a binary; `["rt", "rt-multi-thread", "macros"]` for a library. Pulling `full` into a library bloats downstream.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() {
|
||||||
|
let mut set = tokio::task::JoinSet::new();
|
||||||
|
for id in ["a", "b", "c"] {
|
||||||
|
set.spawn(fetch_user(id.to_string()));
|
||||||
|
}
|
||||||
|
while let Some(res) = set.join_next().await {
|
||||||
|
match res {
|
||||||
|
Ok(Ok(u)) => println!("{}", u.name),
|
||||||
|
Ok(Err(e)) => eprintln!("err: {e}"),
|
||||||
|
Err(join_err) => eprintln!("panic: {join_err}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Async Traits (Concurrency P7 Cancellation Support, C6 Composability)
|
||||||
|
|
||||||
|
- **`async fn` in traits stabilized in Rust 1.75:** `trait Repo { async fn get(&self, id: &str) -> Result<User, Error>; }`. No `async-trait` crate needed for new code on recent toolchains.
|
||||||
|
- **`Box<dyn Trait>` with async methods needs `dyn`-compatibility:** the returned future is `Pin<Box<dyn Future>>`; the compiler boxes it. For hot paths, use generics (`impl Trait`) over `dyn`.
|
||||||
|
- **`async-trait` crate for older toolchains:** macro that desugars to a `Pin<Box<dyn Future>>`. Migrate to native `async fn in trait` when the toolchain allows.
|
||||||
|
- **`Send` bounds on async traits for cross-thread spawn:** `trait Repo: Send { async fn get(&self, id: &str) -> Result<User, Error>; }` — the returned future must be `Send` to spawn on a multi-thread runtime.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
trait UserRepo: Send + Sync {
|
||||||
|
async fn get(&self, id: &str) -> Result<User, Error>;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct PgRepo { pool: PgPool }
|
||||||
|
impl UserRepo for PgRepo {
|
||||||
|
async fn get(&self, id: &str) -> Result<User, Error> {
|
||||||
|
sqlx::query_as::<_, User>("SELECT * FROM users WHERE id = $1")
|
||||||
|
.bind(id).fetch_one(&self.pool).await.map_err(Error::from)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cancellation (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
|
||||||
|
|
||||||
|
- **Cancellation is cooperative via dropping the future:** `tokio::select!` drops the unselected branch, cancelling it. A dropped future stops at its next `.await` point.
|
||||||
|
- **`tokio::time::timeout` for a deadline:** `timeout(Duration::from_secs(5), op).await` returns `Ok(Ok(v))` on success, `Ok(Err(e))` on inner error, `Err(Elapsed)` on timeout. Every external `await` races against a deadline (Concurrency P8).
|
||||||
|
- **`tokio::select!` for cancel-aware waits:** `select! { res = op => res, _ = cancel => return Err(Cancelled), }`. The unselected branch is dropped, cancelling it.
|
||||||
|
- **Cancellation is not atomic:** a future dropped mid-`await` may have partial state. `Drop` runs on cancellation; clean up there (e.g., rollback a transaction).
|
||||||
|
- **Applies `concurrency/P7`:** cancellation is a first-class signal; the runtime propagates it via drop. No `CancelledError` to catch — the future is gone.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use tokio::time::timeout;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
async fn fetch_with_timeout(url: &str) -> Result<Response, Error> {
|
||||||
|
match timeout(Duration::from_secs(5), fetch(url)).await {
|
||||||
|
Ok(Ok(r)) => Ok(r),
|
||||||
|
Ok(Err(e)) => Err(e.into()),
|
||||||
|
Err(_elapsed) => Err(Error::Timeout),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn cancellable(op: impl Future<Output=()>, mut cancel: tokio::sync::oneshot::Receiver<()>) {
|
||||||
|
tokio::select! {
|
||||||
|
_ = op => {},
|
||||||
|
_ = &mut cancel => println!("cancelled"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Pin and Self-Referential Futures (Concurrency P5 Lock Minimization, C1 Correctness)
|
||||||
|
|
||||||
|
- **`async fn` returns a `Future` that is often self-referential:** the generated state machine may hold a borrow into its own stack. Such a future must be `Pin`ned to move safely.
|
||||||
|
- **`Pin<Box<T>>` to box and pin:** `Box::pin(async { ... })` returns a `Pin<Box<dyn Future>>`. The cost is a heap alloc; the win is `Send`/`dyn`-compatibility.
|
||||||
|
- **`Pin<&mut T>` for in-place polling:** `Pin::new(&mut fut)` pins a stack future; the borrow checker prevents moving it. Use for stack-allocated futures in `select!`.
|
||||||
|
- **Do not `unsafe` unpin:** `Pin::get_unchecked_mut` opts out of the pin guarantees. Application code never needs it; library code uses it for `poll` implementations.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use std::pin::Pin;
|
||||||
|
|
||||||
|
async fn boxed() -> Pin<Box<dyn std::future::Future<Output = ()> + Send>> {
|
||||||
|
Box::pin(async {
|
||||||
|
// self-referential state machine is safe to move once pinned
|
||||||
|
})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Bounded Channels and Backpressure (Concurrency P9 Bounded Queues)
|
||||||
|
|
||||||
|
- **`tokio::sync::mpsc::channel(N)` is bounded:** `send().await` blocks when full (backpressure, Concurrency P9). Unbounded `unbounded_channel()` lets the producer run ahead and OOM.
|
||||||
|
- **`tokio::sync::mpsc::Sender::try_send` for non-blocking send:** returns `Err(TrySendError::Full(v))` when full; the caller decides to drop, log, or back off. A bounded queue + `try_send` is the backpressure-aware pattern.
|
||||||
|
- **`tokio::sync::broadcast` for fan-out:** multiple receivers each get a copy; a slow receiver misses (lag). Use for telemetry, not for commands.
|
||||||
|
- **Applies `messaging/queues`:** a bounded tokio channel is an in-process broker — bounded buffer, backpressure, at-most-once handoff. The same semantics apply; the broker is local.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
|
||||||
|
async fn producer(tx: mpsc::Sender<Job>) {
|
||||||
|
for j in jobs() {
|
||||||
|
if tx.send(j).await.is_err() { return; } // receiver dropped
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn consumer(rx: mpsc::Receiver<Job>) {
|
||||||
|
while let Some(j) = rx.recv().await {
|
||||||
|
process(j).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let (tx, rx) = mpsc::channel::<Job>(16); // bounded: backpressure
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P5 Lock Minimization, P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
|
||||||
|
- `domains/messaging/delivery-semantics.md` — at-most-once vs at-least-once framing for async retry/cancel (IDEATE-40).
|
||||||
|
- `languages/rs-ownership.md` — `Send`/`Sync` bounds on futures build on the ownership model here.
|
||||||
|
- `languages/rs-tooling.md` — `tokio` feature flags and the `cargo` build profiles detailed there.
|
||||||
|
- `languages/rs-testing.md` — `#[tokio::test]` and async test patterns.
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
# Rust Ownership — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Rust's ownership model specifically. Rust's distinctive strength (Send/Sync, lifetimes, borrowing) earns a dedicated ownership doc rather than an `rs-types.md`.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/rust.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Ownership and Move Semantics (Concurrency P1 Immutability by Default, C1 Correctness)
|
||||||
|
|
||||||
|
- **Ownership is unique:** at any time, exactly one owner holds a value. Assignment passes ownership (`let y = x;` — `x` is moved, not copied). The compiler rejects use-after-move.
|
||||||
|
- **`Copy` types (integers, `bool`, `&T`) duplicate on assignment; everything else moves.** A `struct` is `Copy` only if all fields are; opt in via `#[derive(Copy, Clone)]` only for small, cheap-to-copy types.
|
||||||
|
- **Pass by `&T` for read-only, `&mut T` for mutation:** a borrow does not transfer ownership; the caller retains the value after the callee returns.
|
||||||
|
- **Applies `concurrency/P1` (immutability by default):** `&T` is shared and immutable; `&mut T` is exclusive and mutable. The compiler enforces "one or many, never both" — aliasing XOR mutation, statically.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let s = String::from("hello");
|
||||||
|
let t = s; // s moved into t
|
||||||
|
// println!("{}", s); // error: use of moved value
|
||||||
|
|
||||||
|
let n = 5;
|
||||||
|
let m = n; // i32 is Copy: n still usable
|
||||||
|
println!("{} {}", n, m);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Borrowing and Lifetimes (C1 Correctness, Data P7 Type Fidelity, Concurrency P3 Boundaries are Locks)
|
||||||
|
|
||||||
|
- **`&'a T` ties a borrow to a lifetime `'a`:** the borrow cannot outlive the owner. Lifetimes are static — the compiler rejects dangling references.
|
||||||
|
- **Lifetime elision when unambiguous:** `fn first<'a>(s: &'a str) -> &'a str` is elided to `fn first(s: &str) -> &str` (one input → output lifetime). When ambiguous, name the lifetime.
|
||||||
|
- **`'static` is the longest lifetime (the whole program):** not "until I drop it." Use `'static` only for values that genuinely live forever (string literals, `const`s); leaking to `'static` to satisfy the checker is a bug.
|
||||||
|
- **`Ref<'a, T>` and `RefMut<'a, T>` from `RefCell` are runtime-checked borrows:** the borrow rules still apply, checked at runtime instead of compile time. A second `RefMut` panics.
|
||||||
|
- **Applies `concurrency/P3` (boundaries are locks):** `&mut T` is the compile-time lock — exclusive access is the boundary; no runtime mutex needed for single-threaded aliasing discipline.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn longest<'a>(a: &'a str, b: &'a str) -> &'a str {
|
||||||
|
if a.len() > b.len() { a } else { b } // borrow tied to both inputs
|
||||||
|
}
|
||||||
|
|
||||||
|
fn dangling() -> &str { // compile error: missing lifetime
|
||||||
|
let s = String::from("local");
|
||||||
|
&s // error: s drops at end of fn
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Send and Sync (Concurrency P1 Immutability by Default, Concurrency P3 Boundaries are Locks, C1 Correctness)
|
||||||
|
|
||||||
|
- **`Send`:** a type `T: Send` may be moved across thread boundaries. Most types are `Send`; `Rc<T>` is not (shared non-atomically refcounted).
|
||||||
|
- **`Sync`:** a type `T: Sync` may be shared (`&T`) across threads. `RefCell<T>` is `!Sync` (interior mutability without atomics); `Mutex<T>` is `Sync` (it synchronizes).
|
||||||
|
- **The compiler enforces `Send`/`Sync` at the thread-spawn boundary:** `std::thread::spawn(move || { ... })` requires the closure's captures to be `Send`.
|
||||||
|
- **Applies `concurrency/P1` and `concurrency/P3`:** `Send` is the move-across-boundary contract; `Sync` is the share-across-boundary contract. Data races are a compile error, not a runtime detector. This is Rust's distinctive strength over Go's race detector.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use std::rc::Rc;
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
let rc = Rc::new(5);
|
||||||
|
// std::thread::spawn(move || { println!("{}", rc) }); // error: Rc is !Send
|
||||||
|
|
||||||
|
let arc = Arc::new(5);
|
||||||
|
std::thread::spawn(move || { println!("{}", arc) }); // ok: Arc<T> is Send+Sync
|
||||||
|
```
|
||||||
|
|
||||||
|
## Shared Mutation: Arc, Mutex, RwLock (Concurrency P3 Boundaries are Locks, Concurrency P5 Lock Minimization)
|
||||||
|
|
||||||
|
- **`Arc<T>` for shared ownership across threads:** atomic refcounted. Clone increases the count; the last drop frees `T`.
|
||||||
|
- **`Mutex<T>` for exclusive mutation across threads:** `lock()` blocks until exclusive; the guard `MutexGuard<T>` derefs to `&mut T` and releases on drop.
|
||||||
|
- **`RwLock<T>` for read-heavy, `Mutex<T>` for write-heavy:** RwLock allows multiple readers or one writer. For most cases, `Mutex` is simpler and faster; prefer it unless reads dominate by 10x+.
|
||||||
|
- **Hold the lock for the smallest scope:** `let g = m.lock().unwrap();` then drop `g` before I/O. RAII releases on scope exit; explicit `drop(g)` clarifies intent.
|
||||||
|
- **Applies `concurrency/P5` (lock minimization):** prefer message passing (`mpsc` channels) over locks. When a lock is needed, scope it minimally.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use std::sync::{Arc, Mutex};
|
||||||
|
use std::thread;
|
||||||
|
|
||||||
|
let counter = Arc::new(Mutex::new(0));
|
||||||
|
let mut handles = vec![];
|
||||||
|
for _ in 0..10 {
|
||||||
|
let c = Arc::clone(&counter);
|
||||||
|
handles.push(thread::spawn(move || {
|
||||||
|
let mut g = c.lock().unwrap();
|
||||||
|
*g += 1;
|
||||||
|
// g drops here, lock released
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
for h in handles { h.join().unwrap(); }
|
||||||
|
println!("{}", *counter.lock().unwrap());
|
||||||
|
```
|
||||||
|
|
||||||
|
## Interior Mutability (Concurrency P1 Immutability by Default, C1 Correctness)
|
||||||
|
|
||||||
|
- **`Cell<T>` for `Copy` types, `RefCell<T>` for non-`Copy`:** interior mutability moves the borrow check from compile time to runtime. `RefCell::borrow_mut()` panics on a second mutable borrow.
|
||||||
|
- **`Mutex<T>`/`RwLock<T>` for thread-safe interior mutability:** the runtime check is the lock, not a panic. Use these across threads; `RefCell` only single-threaded.
|
||||||
|
- **`UnsafeCell<T>` is the primitive; never use directly:** `Cell`, `RefCell`, `Mutex` are safe wrappers. Direct `UnsafeCell` is `unsafe` and opts out of the aliasing guarantee.
|
||||||
|
- **Applies `concurrency/P1`:** interior mutability is the exception, not the default. Reach for it when an API must present `&self` while mutating internally (e.g., a cache); document why.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use std::cell::RefCell;
|
||||||
|
|
||||||
|
struct Cache {
|
||||||
|
inner: RefCell<HashMap<String, User>>,
|
||||||
|
}
|
||||||
|
impl Cache {
|
||||||
|
fn get(&self, id: &str) -> Option<User> {
|
||||||
|
// &self (immutable) but mutates internally
|
||||||
|
self.inner.borrow_mut().entry(id.to_string()).or_insert_with(|| fetch()).clone()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Drop and RAII (C1 Correctness, Concurrency P3 Boundaries are Locks)
|
||||||
|
|
||||||
|
- **`Drop` runs when the owner goes out of scope:** no `defer`, no `finally`. A `MutexGuard` releases, a `File` closes, a `JoinHandle`... does not join (a dropped `JoinHandle` detaches).
|
||||||
|
- **`Drop` is deterministic:** it runs at scope exit, not GC time. This is why `Arc`'s refcount is precise and `Mutex` release is timely.
|
||||||
|
- **`ManuallyDrop<T>` to opt out:** for FFI types whose destructor you must call manually. Rare in application code; common in `unsafe` bindings.
|
||||||
|
- **`Drop` order: fields in declaration order, then the struct itself.** A field that another field's `Drop` depends on must be declared last.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
struct Resource { name: String }
|
||||||
|
impl Drop for Resource {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
println!("dropping {}", self.name); // runs at scope end
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() {
|
||||||
|
let _r = Resource { name: "x".into() };
|
||||||
|
// _r drops here, prints "dropping x"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P1 Immutability, P3 Boundaries are Locks, P5 Lock Minimization.
|
||||||
|
- `domains/data/first-principles.md` — Data P7 Type Fidelity (lifetimes are the type-level fidelity for references).
|
||||||
|
- `domains/concurrency/patterns.md` — message-passing vs lock patterns applied via `Arc`/`Mutex`/`mpsc`.
|
||||||
|
- `domains/errors/patterns.md` — `?` propagation relies on ownership transfer of the error.
|
||||||
|
- `languages/rs-async.md` — async borrows (`Pin`/`&mut`) build on the lifetime model here.
|
||||||
|
- `languages/rs-testing.md` — `Send`/`Sync` tests and ownership-based property tests.
|
||||||
@@ -0,0 +1,159 @@
|
|||||||
|
# Rust Testing — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Rust testing specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/rust.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## #[test] and Co-located Tests (Testing P1 Tests as Specification, C2 Clarity)
|
||||||
|
|
||||||
|
- **`#[test]` on functions in a `#[cfg(test)] mod tests` block:** tests co-located with source, compiled only in `cargo test`. A test file far from its subject rots (Documentation P5 Discoverability).
|
||||||
|
- **Test names read as a spec:** `fn create_user_rejects_invalid_email()` — a reader understands the unit from the name. Avoid `fn test_user_1()`.
|
||||||
|
- **`assert!` / `assert_eq!` / `assert_ne!` over raw `panic!`:** the macros produce readable failure output (`assertion failed: left == right, left: 5, right: 3`). Raw `panic!` gives a message only (Testing P6 Failure Specificity).
|
||||||
|
- **Applies `Testing P1`:** the test is a specification; the failure message is the spec violation.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn create_user_rejects_invalid_email() {
|
||||||
|
let r = create_user("not-an-email");
|
||||||
|
assert!(matches!(r, Err(Error::Validation(_))));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn create_user_returns_persisted_id() {
|
||||||
|
let u = create_user("a@b.co").unwrap();
|
||||||
|
assert!(!u.id.is_empty());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## proptest and Property Tests (Testing P9 Edge Case Coverage, Testing P1 Tests as Specification)
|
||||||
|
|
||||||
|
- **`proptest` (or `quickcheck`) for invariant tests:** declare a property (`parse(serialize(x)) == x`), the framework generates hundreds of inputs and shrinks failures to a minimal counterexample (Testing P9).
|
||||||
|
- **Strategy over hand-written generators:** `proptest::collection::vec(any::<u32>(), 0..100)` generates arbitrary `Vec<u32>`; do not hand-roll a generator for each property.
|
||||||
|
- **`proptest!` macro or `proptest! { ... }` block:** each `case (name) => { ... }` is a property. The block is the spec (Testing P1).
|
||||||
|
- **Property tests complement, not replace, example tests:** examples document the happy path; properties cover the edge space. Both are required.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use proptest::prelude::*;
|
||||||
|
|
||||||
|
proptest! {
|
||||||
|
#[test]
|
||||||
|
fn roundtrips_id(s in "[a-z0-9]{1,32}") {
|
||||||
|
let id = UserId::new(&s).unwrap();
|
||||||
|
assert_eq!(id.as_str(), s);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_invalid_id(s in "[^a-z0-9]+") {
|
||||||
|
assert!(UserId::new(&s).is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Mock Discipline (Testing P2 Independence, Testing P7 Realism)
|
||||||
|
|
||||||
|
- **Mock at the trait, not the struct:** `trait Store { fn get(&self, id: &str) -> Result<User, Error>; }` in production; `#[automock] trait Store` (via `mockall`) in test. The trait is the contract.
|
||||||
|
- **`mockall` for generated mocks:** `#[automock] trait Repo {}` generates `MockRepo` with `expect_*` methods. Each expectation is per-test; no shared mock state (Testing P2 Independence).
|
||||||
|
- **Mock the boundary, not the unit:** mock `Repo`, not `UserService` (the unit). Mocking the unit under test tests the mock (Testing P7 realism).
|
||||||
|
- **No `#[cfg(test)]` on production code paths to inject mocks:** instead, accept the trait as a generic or `dyn` parameter. Test-only branches in production code are dead code in prod.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use mockall::*;
|
||||||
|
|
||||||
|
#[automock]
|
||||||
|
trait UserRepo {
|
||||||
|
fn get(&self, id: &str) -> Result<User, Error>;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn get_user_returns_not_found() {
|
||||||
|
let mut repo = MockUserRepo::new();
|
||||||
|
repo.expect_get()
|
||||||
|
.with(eq("abc"))
|
||||||
|
.returning(|_| Err(Error::NotFound));
|
||||||
|
let svc = UserService::new(Box::new(repo));
|
||||||
|
assert!(matches!(svc.get_user("abc"), Err(Error::NotFound)));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Async Tests (Concurrency P10 Test for Race Conditions, Testing P1 Tests as Specification)
|
||||||
|
|
||||||
|
- **`#[tokio::test]` for `async fn` tests:** runs the coroutine on a tokio runtime. Without it, an `async fn` test returns a future, never awaited (silently passes).
|
||||||
|
- **`#[tokio::test(flavor = "multi_thread")]` for concurrency-sensitive tests:** multi-thread runtime surfaces races that single-thread misses (Concurrency P10).
|
||||||
|
- **`tokio::time::pause()` and `advance()` for time:** freeze and advance the runtime clock deterministically. No `tokio::time::sleep(real)` in tests.
|
||||||
|
- **Race-sensitive tests use `loom` for model-checking:** `loom` simulates all thread interleavings; it catches races `-race`-style detectors miss. Use for lock-free data structures.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[tokio::test]
|
||||||
|
async fn async_fetch_returns_user() {
|
||||||
|
let u = fetch_user("abc").await.unwrap();
|
||||||
|
assert!(!u.name.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||||
|
async fn concurrent_cache_is_safe() {
|
||||||
|
let c = Arc::new(Cache::new());
|
||||||
|
let mut h = vec![];
|
||||||
|
for i in 0..10 {
|
||||||
|
let c = c.clone();
|
||||||
|
h.push(tokio::spawn(async move { c.get(&i.to_string()).await; }));
|
||||||
|
}
|
||||||
|
for x in h { x.await.unwrap(); }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||||
|
|
||||||
|
- **No `SystemTime::now()` or `Instant::now()` in code under test:** inject a `Clock` trait. In tests, a fake clock advances deterministically.
|
||||||
|
- **`tokio::time::pause()` for async time:** freezes the runtime clock; `tokio::time::advance(dur)` moves it. A `sleep(5s)` in test resolves instantly.
|
||||||
|
- **`--test-threads=1` to reproduce order coupling:** by default, `cargo test` runs tests in parallel; a test that passes alone but fails in a suite has hidden shared state. `-1` reproduces.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
trait Clock { fn now(&self) -> std::time::Instant; }
|
||||||
|
|
||||||
|
struct FakeClock(std::time::Instant);
|
||||||
|
impl Clock for FakeClock {
|
||||||
|
fn now(&self) -> std::time::Instant { self.0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn user_has_created_at() {
|
||||||
|
let clk = FakeClock(std::time::Instant::now());
|
||||||
|
let u = create_user_with_clock("a@b.co", &clk).unwrap();
|
||||||
|
assert_eq!(u.created_at, clk.now());
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Doc Tests (Documentation P1 Documentation is Code, Testing P1 Tests as Specification)
|
||||||
|
|
||||||
|
- **`cargo test --doc` runs `///` fenced blocks:** a `///` example with `#`-hidden setup is a tested artifact; a stale output fails the build (Documentation P1).
|
||||||
|
- **`no_run` for examples that should compile but not run:** ```` ```rust,no_run ```` — type-checks the example without executing. Use for examples that need a DB.
|
||||||
|
- **`ignore` for examples that should not compile-check:** ```` ```rust,ignore ```` — skips entirely. Rare; prefer `no_run`.
|
||||||
|
- **Applies `Testing P1`:** the doc example is the spec; the doc test is the spec's regression test.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
/// Fetch a user by id.
|
||||||
|
///
|
||||||
|
/// # Example
|
||||||
|
///
|
||||||
|
/// ```
|
||||||
|
/// # use mycrate::{get_user, Error};
|
||||||
|
/// let u = get_user("abc").unwrap();
|
||||||
|
/// assert!(!u.name.is_empty());
|
||||||
|
/// ```
|
||||||
|
pub fn get_user(id: &str) -> Result<User, Error> { /* ... */ }
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/testing/pyramid.md` — where unit/property/doc tests sit; proptest is the property layer.
|
||||||
|
- `domains/testing/fixtures.md` — `t.Cleanup`-equivalent (`Drop` in tests) as fixture discipline.
|
||||||
|
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P10 (test for races), `loom` model-checking.
|
||||||
|
- `languages/rs-ownership.md` — `Send`/`Sync` tests and ownership-based property tests.
|
||||||
|
- `languages/rs-async.md` — `#[tokio::test]` patterns from that doc.
|
||||||
|
- `languages/rs-tooling.md` — `cargo test` flags (`--doc`, `--test-threads`) detailed here.
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# Rust Tooling — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to Rust tooling specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/rust.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## cargo and Build Discipline (DevOps P2 Automation, DevOps P1 Reproducibility)
|
||||||
|
|
||||||
|
- **`cargo build` for dev, `cargo build --release` for release:** release enables optimizations (LTO, codegen-units=1). The default profile is for fast iteration, not perf.
|
||||||
|
- **`Cargo.lock` committed for applications and CI:** for libraries, commit the lock for CI reproducibility even though consumers resolve their own tree. A drifted lock breaks reproducibility (DevOps P1).
|
||||||
|
- **`cargo update` periodically, with a CI check:** `cargo update` bumps patch versions in the lock; a CI job that fails on lock drift catches a forgotten `cargo update`.
|
||||||
|
- **`cargo vendor` for hermetic CI:** vendors `vendor/` into the repo; CI builds without network. The trade-off is repo size; the win is reproducibility.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# Cargo.toml — profile discipline
|
||||||
|
[profile.release]
|
||||||
|
lto = true
|
||||||
|
codegen-units = 1
|
||||||
|
panic = "abort" # smaller binary, no unwinding
|
||||||
|
```
|
||||||
|
|
||||||
|
## clippy (DevOps P2 Automation, C2 Clarity)
|
||||||
|
|
||||||
|
- **`cargo clippy` is the lint layer over `rustc`:** it catches `clone()` where a borrow would do, `unwrap()` in library code, and needless `Box`. Run on every build.
|
||||||
|
- **`cargo clippy -- -D warnings` in CI:** warnings are errors. A clippy warning is a smell; accumulating them erodes the signal (Clarity C2).
|
||||||
|
- **Per-lint allow only with a tracked reason:** `#[allow(clippy::needless_collect)] // reason: GH-123 — collect needed for len` — each allow links to a ticket. Untracked allows accumulate into a permanently lint-bypassed core.
|
||||||
|
- **`cargo clippy --fix` for safe auto-fixes:** applies the linter's suggested change. Review the diff; do not run blindly on a large commit.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI gate
|
||||||
|
cargo clippy --all-targets --all-features -- -D warnings
|
||||||
|
```
|
||||||
|
|
||||||
|
## cargo fmt (DevOps P2 Automation, C2 Clarity)
|
||||||
|
|
||||||
|
- **`cargo fmt` is the formatter; format is not debated in review:** run in CI as a check (`cargo fmt --check`), not a fix. A failing check blocks the PR.
|
||||||
|
- **`rustfmt.toml` for repo-wide settings:** if the defaults are wrong for the repo, override once and stop. Do not relitigate per-PR.
|
||||||
|
- **Applies `devops/P2`:** the format gate is automated; a reviewer never comments on style.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI gate — fail if unformatted
|
||||||
|
cargo fmt --check
|
||||||
|
```
|
||||||
|
|
||||||
|
## Edition Discipline (DevOps P1 Reproducibility, C5 Reversibility)
|
||||||
|
|
||||||
|
- **`edition` in `Cargo.toml` pins the language edition:** 2015, 2018, 2021, 2024. An edition is a coherent set of language changes; bumping it is a deliberate migration.
|
||||||
|
- **Edition is not the compiler version:** `rustc 1.75` supports edition 2021; edition 2024 needs a newer `rustc`. Pin the toolchain with `rust-toolchain.toml`.
|
||||||
|
- **Bump editions deliberately, not opportunistically:** `cargo fix --edition` applies the migration lint; review the diff. A bump mid-feature conflates two changes.
|
||||||
|
- **Applies `devops/P1` and `C5` (reversibility):** pinning the edition and toolchain makes the build reproducible; bumping is a controlled, reversible change.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# Cargo.toml
|
||||||
|
[package]
|
||||||
|
edition = "2021"
|
||||||
|
rust-version = "1.75"
|
||||||
|
```
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# rust-toolchain.toml
|
||||||
|
[toolchain]
|
||||||
|
channel = "1.75"
|
||||||
|
components = ["clippy", "rustfmt"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||||
|
|
||||||
|
- **`cargo doc` from doc comments:** `///` on items generates API docs; `cargo doc --open` previews. The build fails on broken intra-doc links (`#![warn(rustdoc::broken_intra_doc_links)]`).
|
||||||
|
- **Doc tests are run by `cargo test`:** a `///` fenced block with `#`-hidden setup is a tested artifact; a stale example fails `cargo test --doc` (Documentation P1).
|
||||||
|
- **`#![warn(missing_docs)]` for libraries:** public items without doc comments fail the build. Documentation is a build gate, not an afterthought.
|
||||||
|
- **`cargo readme` or `cargo docs-rs` for landing pages:** the crate's `README.md` is rendered on docs.rs; keep it in sync with `lib.rs`'s top-level doc.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#![warn(missing_docs, rustdoc::broken_intra_doc_links)]
|
||||||
|
|
||||||
|
/// Fetch a user by id.
|
||||||
|
///
|
||||||
|
/// # Example
|
||||||
|
///
|
||||||
|
/// ```
|
||||||
|
/// # use mycrate::get_user;
|
||||||
|
/// let u = get_user("abc").unwrap();
|
||||||
|
/// println!("{}", u.name);
|
||||||
|
/// ```
|
||||||
|
pub fn get_user(id: &str) -> Result<User, Error> { /* ... */ }
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/devops/ci-cd.md` — the pipeline gates that host clippy/fmt/test.
|
||||||
|
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||||
|
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||||
|
- `languages/rs-ownership.md` — `Send`/`Sync` clippy lints reference this doc.
|
||||||
|
- `languages/rs-async.md` — async-runtime tooling (`tokio` features) detailed here.
|
||||||
|
- `languages/rs-testing.md` — `cargo test` flags (`--doc`, `--no-run`) detailed here.
|
||||||
@@ -2,6 +2,13 @@
|
|||||||
|
|
||||||
> How Atelier's domain principles apply in Rust specifically. Derives from `domains/` docs.
|
> How Atelier's domain principles apply in Rust specifically. Derives from `domains/` docs.
|
||||||
|
|
||||||
|
## Derived Docs
|
||||||
|
|
||||||
|
- [rs-ownership.md](rs-ownership.md) — Send/Sync, lifetimes, borrowing, ownership transfer.
|
||||||
|
- [rs-tooling.md](rs-tooling.md) — cargo, clippy, fmt, edition discipline.
|
||||||
|
- [rs-async.md](rs-async.md) — tokio, async traits, cancellation, pin.
|
||||||
|
- [rs-testing.md](rs-testing.md) — #[test], proptest, property testing, mock discipline.
|
||||||
|
|
||||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||||
|
|
||||||
- **Newtypes for domain concepts:** `struct UserId(String);` — zero-cost, type-safe.
|
- **Newtypes for domain concepts:** `struct UserId(String);` — zero-cost, type-safe.
|
||||||
|
|||||||
@@ -0,0 +1,115 @@
|
|||||||
|
# TypeScript Async — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to TypeScript async specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/typescript.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Promises and AbortSignal (Concurrency P7 Cancellation Support, C1 Correctness)
|
||||||
|
|
||||||
|
- **Every async function accepts an optional `AbortSignal`:** cancellation is a first-class parameter, not a side channel. The signal propagates to `fetch`, `setTimeout`, and downstream awaits.
|
||||||
|
- **`AbortController` is the producer side; `AbortSignal` is the consumer side:** a function takes a `signal` (read-only), the caller owns the `controller` and decides when to abort.
|
||||||
|
- **Abort propagates as a rejected `Promise`:** `fetch` rejects with `AbortError`; downstream code sees the rejection, not a silent no-op. This preserves `errors/P5` (recoverable when possible) — the caller can distinguish cancellation from a real failure.
|
||||||
|
- **Applies `concurrency/P7`:** no async operation runs without a path to cancel it. A long-running `await` with no signal is a hung request.
|
||||||
|
- **Never swallow `AbortError`:** re-throw or handle distinctly; cancellation is the caller's intent, not an error to log.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
async function fetchUser(id: UserId, signal?: AbortSignal): Promise<User> {
|
||||||
|
const ctrl = new AbortController();
|
||||||
|
signal?.addEventListener('abort', () => ctrl.abort());
|
||||||
|
const res = await fetch(`/users/${id}`, { signal: ctrl.signal });
|
||||||
|
if (!res.ok) throw new HttpError(res.status);
|
||||||
|
return res.json() as Promise<User>;
|
||||||
|
}
|
||||||
|
|
||||||
|
// caller controls cancellation
|
||||||
|
const ctrl = new AbortController();
|
||||||
|
const timer = setTimeout(() => ctrl.abort(), 5000);
|
||||||
|
try {
|
||||||
|
const u = await fetchUser(id, ctrl.signal);
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timer);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## async/await Discipline (Concurrency P8 Timeout Discipline, C2 Clarity)
|
||||||
|
|
||||||
|
- **`await` is the only async primitive in application code:** no `.then` chains, no callback pyramids. `async`/`await` reads top-to-bottom (Clarity C2).
|
||||||
|
- **Never `await` in a hot loop without batching:** sequential `await` in a `for` loop is O(n) latency. Use `Promise.all` for parallelism; `for await...of` only for genuine streams.
|
||||||
|
- **`Promise.race` for a timeout:** every external `await` has a deadline. `Promise.race([op, timeout])` rejects when the deadline passes.
|
||||||
|
- **`return` vs `return await`:** inside `try`/`finally`, `return await` runs the `finally`; bare `return` of a Promise defers the `finally` to the microtask. Prefer `return await` when cleanup must run.
|
||||||
|
- **Applies `concurrency/P8`:** a bare `await` with no timeout is an unbounded wait. External calls (network, disk) always race against a deadline.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
async function fetchWithTimeout(url: string, ms = 5000, signal?: AbortSignal): Promise<Response> {
|
||||||
|
const ctrl = new AbortController();
|
||||||
|
signal?.addEventListener('abort', () => ctrl.abort());
|
||||||
|
const timer = new Promise<never>((_, reject) =>
|
||||||
|
setTimeout(() => reject(new TimeoutError(ms)), ms)
|
||||||
|
);
|
||||||
|
try {
|
||||||
|
return await Promise.race([fetch(url, { signal: ctrl.signal }), timer]);
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timer); // cleanup runs on success and on race-loss
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling in Async (Errors P5 Recoverable When Possible, Errors P1 Errors are Data)
|
||||||
|
|
||||||
|
- **Catch `unknown`, narrow with a type guard:** `catch (e: unknown)` — TS does not infer the error type. `instanceof` or a discriminator narrows it.
|
||||||
|
- **Retry with backoff for transient failures:** network blips are recoverable (Errors P5). Exponential backoff with jitter, capped retry count, and an `AbortSignal`-aware `setTimeout`.
|
||||||
|
- **No retry for non-idempotent operations:** a `POST` that creates a resource is not safely retryable without an idempotency key (applies `api/P6` Idempotency).
|
||||||
|
- **Typed errors over `Error` subclasses:** a discriminated union `AppError = Network | Timeout | Cancelled` carries context (Errors P4 Preserve Context) without `instanceof` chains.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
async function fetchRetry(url: string, attempts = 3, signal?: AbortSignal): Promise<Response> {
|
||||||
|
for (let i = 0; i < attempts; i++) {
|
||||||
|
try {
|
||||||
|
return await fetchWithTimeout(url, 5000, signal);
|
||||||
|
} catch (e: unknown) {
|
||||||
|
if (e instanceof AbortError) throw e; // do not retry cancellation
|
||||||
|
if (e instanceof TimeoutError && i < attempts - 1) {
|
||||||
|
await sleep(jitter(i), signal); // backoff before retry
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
throw e;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw new Error('unreachable');
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cancellation Propagation (Concurrency P7 Cancellation Support, Concurrency P9 Bounded Queues)
|
||||||
|
|
||||||
|
- **One signal, many consumers:** pass the same `AbortSignal` to every async call in a request. Aborting once cancels the whole tree.
|
||||||
|
- **Bounded concurrency with a semaphore:** a `Semaphore(N)` wrapping `Promise.all` caps in-flight requests (Concurrency P9 — bounded queues). Unbounded `Promise.all` on a 10k-item array exhausts file descriptors.
|
||||||
|
- **Cancellation is cooperative, not preemptive:** a long synchronous block inside an `async` function ignores the signal. Yield with `await Promise.resolve()` periodically in CPU-bound loops, or move to a worker.
|
||||||
|
- **Applies `messaging/delivery-semantics`:** a cancelable async operation is an at-most-once delivery — the caller may stop listening, the result may or may not arrive. Retry-on-cancel is at-least-once; the caller must declare which.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
async function mapBounded<T, U>(items: readonly T[], fn: (t: T, s: AbortSignal) => Promise<U>, limit = 8, signal?: AbortSignal): Promise<U[]> {
|
||||||
|
const ctrl = new AbortController();
|
||||||
|
signal?.addEventListener('abort', () => ctrl.abort());
|
||||||
|
const results: U[] = new Array(items.length);
|
||||||
|
let next = 0;
|
||||||
|
const workers = Array.from({ length: limit }, async () => {
|
||||||
|
while (true) {
|
||||||
|
const i = next++;
|
||||||
|
if (i >= items.length) break;
|
||||||
|
if (ctrl.signal.aborted) throw new AbortError();
|
||||||
|
results[i] = await fn(items[i], ctrl.signal);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
await Promise.all(workers);
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||||
|
- `domains/concurrency/first-principles.md` — Concurrency P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
|
||||||
|
- `domains/messaging/delivery-semantics.md` — at-most-once vs at-least-once framing for async retry/cancel (IDEATE-40).
|
||||||
|
- `domains/errors/patterns.md` — typed async errors and retry-with-backoff.
|
||||||
|
- `languages/ts-types.md` — `Result<T, E>` and discriminated `AppError` used in async error handling.
|
||||||
|
- `languages/ts-tooling.md` — `no-floating-promises` lint rule that enforces these awaits.
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
# TypeScript Testing — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to TypeScript testing specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/typescript.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Vitest and Jest (Testing P1 Tests as Specification, C2 Clarity)
|
||||||
|
|
||||||
|
- **Vitest for new TS projects; Jest for legacy:** Vitest shares `vite`'s transform pipeline (no separate `ts-jest` config); Jest's ecosystem is broader. Either is acceptable — pick one per repo, do not mix.
|
||||||
|
- **Tests co-located with source:** `user.ts` → `user.test.ts`. A test file far from its subject rots (Documentation P5 Discoverability).
|
||||||
|
- **`describe`/`it` mirror the public API:** the test block names read as a specification ("User", "rejects an invalid email", "returns the persisted id"). A reader should understand the unit from test names alone (Testing P1).
|
||||||
|
- **`expect` over `assert`:** Vitest/Jest matchers produce readable failure output (`expect(x).toBe(y)` → "expected 5, received 3"). Raw `assert` gives a stack trace and nothing else (Testing P6 Failure Specificity).
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// user.test.ts
|
||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { createUser } from './user';
|
||||||
|
|
||||||
|
describe('createUser', () => {
|
||||||
|
it('rejects an invalid email', async () => {
|
||||||
|
await expect(createUser({ email: 'not-an-email' })).rejects.toThrow(ValidationError);
|
||||||
|
});
|
||||||
|
it('returns the persisted id', async () => {
|
||||||
|
const u = await createUser({ email: 'a@b.co' });
|
||||||
|
expect(u.id).toMatch(/^[a-z0-9]+$/);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Mock Discipline (Testing P2 Independence, Testing P7 Realism)
|
||||||
|
|
||||||
|
- **Mock at the boundary, not the unit:** replace `fetch` or the DB client, not the function under test. Mocking the unit under test tests the mock, not the code (Testing P7 — realism).
|
||||||
|
- **No partial mocks of the system under test:** if a method must be stubbed, the unit is too large. Extract a collaborator and mock that.
|
||||||
|
- **Each test sets up and tears down its own state:** no shared mutable fixtures. A `beforeEach`/`afterEach` resets; a top-level `let` shared across tests is order-coupling (Testing P2 Independence).
|
||||||
|
- **`vi.useFakeTimers()` for time-dependent code:** never call `Date.now()` directly in code under test; inject a `Clock` port. In tests, fake timers make `setTimeout` synchronous.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { vi, beforeEach, afterEach } from 'vitest';
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
vi.useFakeTimers();
|
||||||
|
global.fetch = vi.fn(); // boundary mock
|
||||||
|
});
|
||||||
|
afterEach(() => {
|
||||||
|
vi.useRealTimers();
|
||||||
|
vi.restoreAllMocks();
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type-Level Tests (Testing P1 Tests as Specification, Data P7 Type Fidelity)
|
||||||
|
|
||||||
|
- **Type-level tests assert the type system, not runtime behavior:** `expectTypeOf<T>().toMatchTypeOf<U>` and `tsd`/`expect-type` fail the build when a type assertion is wrong.
|
||||||
|
- **Negative type tests are required:** `// @ts-expect-error` proves the compiler rejects what it should. A `@ts-expect-error` that no longer errors is itself an error (the comment must be consumed).
|
||||||
|
- **Branded types and utility types get type tests:** a `UserId` should not be assignable to `string`; a `Readonly<T>` should not allow assignment. These invariants are part of the spec (Testing P1).
|
||||||
|
- **Applies `data/P7` (type fidelity):** a type-level test is a regression test for the type checker — if a refactor silently widens a type, the test fails.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { expectTypeOf } from 'expect-type';
|
||||||
|
import type { User, UserPatch, UserId } from './user';
|
||||||
|
|
||||||
|
test('UserPatch omits id and makes fields optional', () => {
|
||||||
|
expectTypeOf<UserPatch>().toMatchTypeOf<{ name?: string; email?: string }>();
|
||||||
|
expectTypeOf<UserPatch>().not.toHaveProperty('id');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('UserId is not assignable to bare string', () => {
|
||||||
|
// @ts-expect-error — brand prevents widening
|
||||||
|
const s: string = {} as UserId;
|
||||||
|
expect(s).toBeDefined();
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Parametrize and Factories (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||||
|
|
||||||
|
- **`it.each` / `test.each` for parametrized cases:** one table drives many runs; each row is an independent test with its own name and failure output.
|
||||||
|
- **Factories over fixtures:** `makeUser(overrides)` returns a fresh object per call. A shared `const user = {...}` across tests couples them and breaks determinism when one test mutates it (Testing P3).
|
||||||
|
- **Edge cases as rows, not special tests:** empty array, single element, max int, null, undefined — each a row in a `test.each` table. An ad-hoc `it('handles edge')` with multiple asserts hides which case failed (Testing P9 — edge case coverage, P6 failure specificity).
|
||||||
|
- **Property-style tests via `fast-check`:** for invariants (e.g., "parse(serialize(x)) === x"), `fast-check` generates hundreds of inputs and shrinks failures to a minimal counterexample.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { test, expect } from 'vitest';
|
||||||
|
import { makeUser } from './user.factory';
|
||||||
|
|
||||||
|
test.each([
|
||||||
|
{ input: '', reason: 'empty' },
|
||||||
|
{ input: 'a'.repeat(1000), reason: 'too long' },
|
||||||
|
{ input: 'not-an-email', reason: 'no @' },
|
||||||
|
])('rejects email: $reason', async ({ input }) => {
|
||||||
|
await expect(makeUser({ email: input })).rejects.toThrow(ValidationError);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||||
|
|
||||||
|
- **No `Date.now()`, `Math.random()`, or `crypto.randomUUID()` in code under test:** inject a `Clock`, `Random`, and `IdGen` port. In tests, provide deterministic fakes.
|
||||||
|
- **`--random` test order (Vitest `sequence.shuffle: true` default) catches order coupling:** a test that passes alone but fails in a suite has hidden state. The shuffle makes that state visible (Testing P2).
|
||||||
|
- **Race-detector parallelism for async tests:** run async tests concurrently by default; a test that assumes serial execution breaks under parallelism. Vitest's `concurrent` flag surfaces the bug.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { vi, test, expect } from 'vitest';
|
||||||
|
|
||||||
|
test.concurrent('parallel fetch does not interleave state', async () => {
|
||||||
|
const store = new Store();
|
||||||
|
await Promise.all([store.put('a', 1), store.put('b', 2)]);
|
||||||
|
expect(store.get('a')).toBe(1);
|
||||||
|
expect(store.get('b')).toBe(2);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/testing/pyramid.md` — where unit/type/integration tests sit; the type-level tests here are the base layer.
|
||||||
|
- `domains/testing/fixtures.md` — factory-vs-fixture discipline applied via `makeUser`.
|
||||||
|
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||||
|
- `languages/ts-types.md` — the branded types and utility types that type-level tests assert.
|
||||||
|
- `languages/ts-async.md` — async tests use the cancellation/timeout patterns from that doc.
|
||||||
|
- `languages/ts-tooling.md` — `ts-jest`/`vitest` config and the `expect-type`/`tsd` toolchain.
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
# TypeScript Tooling — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to TypeScript tooling specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/typescript.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## tsc and tsconfig Discipline (DevOps P2 Automation, DevOps P1 Reproducibility)
|
||||||
|
|
||||||
|
- **`strict: true` is the floor, not the ceiling:** it enables `strictNullChecks`, `noImplicitAny`, `strictFunctionTypes`, and more. Disable sub-flags only with a justification comment.
|
||||||
|
- **`tsc --noEmit` in CI:** type-checking is a build gate; emission is the bundler's job. Separate the two so a type error fails CI even when the bundler would have succeeded.
|
||||||
|
- **`tsconfig` is per-project, not inherited verbatim:** a shared base (`extends`) encodes org defaults; each project overrides the deltas it needs. Avoids the "one monoreto-config-fits-all" trap.
|
||||||
|
- **`noUncheckedIndexedAccess` for safety:** `arr[i]` becomes `T | undefined`, forcing narrowing. Costs little, prevents a class of out-of-bounds deref bugs.
|
||||||
|
- **Applies `devops/P1` (reproducibility):** pinned `typescript` version in `package.json` and `lockfile` ensure every CI run type-checks against the same compiler.
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// tsconfig.json — base
|
||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"strict": true,
|
||||||
|
"noUncheckedIndexedAccess": true,
|
||||||
|
"exactOptionalPropertyTypes": true,
|
||||||
|
"noEmit": true,
|
||||||
|
"moduleResolution": "bundler",
|
||||||
|
"isolatedModules": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## ESLint and @typescript-eslint (DevOps P2 Automation, Documentation P9 Living Documents)
|
||||||
|
|
||||||
|
- **ESLint with `@typescript-eslint` strict ruleset:** `recommended-type-checked` enables rules that require the type checker (`no-floating-promises`, `no-misused-promises`).
|
||||||
|
- **Rules encode decisions, not taste:** every custom rule in the config has a one-line `// reason:` comment linking to the principle it enforces. This makes the config a living document (Documentation P9).
|
||||||
|
- **Format is Prettier's job; ESLint lints:** `eslint-config-prettier` disables conflicting format rules. Do not relitigate formatting in code review.
|
||||||
|
- **`no-floating-promises` enforces `concurrency/P8` (timeout discipline):** an un-awaited `Promise` is a fire-and-forget that swallows errors and timeouts. The rule forces `.catch()` or `await`.
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// .eslintrc.json
|
||||||
|
{
|
||||||
|
"extends": [
|
||||||
|
"eslint:recommended",
|
||||||
|
"plugin:@typescript-eslint/recommended-type-checked",
|
||||||
|
"prettier"
|
||||||
|
],
|
||||||
|
"parserOptions": { "project": "./tsconfig.json" },
|
||||||
|
"rules": {
|
||||||
|
// reason: enforce Concurrency P8 — no un-awaited promises
|
||||||
|
"@typescript-eslint/no-floating-promises": "error",
|
||||||
|
// reason: enforce Data P7 — no `any` escaping the type checker
|
||||||
|
"@typescript-eslint/no-explicit-any": "error"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Project References and ts-jest (DevOps P2 Automation, C6 Composability)
|
||||||
|
|
||||||
|
- **Project references for monorepos:** `composite: true` + `references` let `tsc --build` incrementally type-check only changed projects, and enforce the dependency graph at the type level.
|
||||||
|
- **`paths` aliases mirror the import structure:** `@app/*` → `src/*`. Configure once in `tsconfig`, mirror in the bundler and the test runner so all three agree.
|
||||||
|
- **`ts-jest` (or `vitest`) with `isolatedModules: true`:** each test file is type-checked in isolation, matching how the bundler transpiles. Catches the "passes in `tsc` but fails in the bundler" gap.
|
||||||
|
- **Applies `devops/P2`:** the build pipeline (tsc → lint → test → bundle) is automated; a developer never runs a manual sequence.
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// tsconfig.references.json
|
||||||
|
{
|
||||||
|
"files": [],
|
||||||
|
"references": [
|
||||||
|
{ "path": "./packages/core" },
|
||||||
|
{ "path": "./packages/api" },
|
||||||
|
{ "path": "./packages/web" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Lockfile and Reproducible Install (DevOps P1 Reproducibility)
|
||||||
|
|
||||||
|
- **`npm ci` in CI, not `npm install`:** `ci` reads the lockfile exactly and fails on drift. `install` mutates the lockfile.
|
||||||
|
- **Lockfile committed for applications:** for libraries, commit `package-lock.json` for CI reproducibility even though consumers resolve their own tree.
|
||||||
|
- **No floating ranges in `package.json`:** `^` and `~` are CI's job to resolve; pin the resolved version in the lockfile. An unpinned `*` is a supply-chain attack surface.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CI install step — deterministic
|
||||||
|
npm ci
|
||||||
|
# Type-check gate
|
||||||
|
npx tsc --noEmit
|
||||||
|
# Lint gate
|
||||||
|
npx eslint .
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||||
|
|
||||||
|
- **Type-checked JSDoc:** `typedoc` (or `TypeDoc`) generates API docs from `tsdoc` comments. The compiler enforces that `@param` names match real parameters.
|
||||||
|
- **`@example` blocks are compiled:** a `tsdoc` `@example` fenced block is type-checked as part of the doc build. Stale examples fail the pipeline (Documentation P1 — docs are code).
|
||||||
|
- **README badges reflect CI status:** the build/lint/test/type-check gates are the source of truth; badges surface them. Do not hand-edit status tables.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
/**
|
||||||
|
* Fetch a user by ID.
|
||||||
|
*
|
||||||
|
* @param id - a branded UserId (see ts-types.md).
|
||||||
|
* @throws {NotFoundError} if the user does not exist.
|
||||||
|
* @example
|
||||||
|
* ```ts
|
||||||
|
* const u = await getUser(userId('abc'));
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
async function getUser(id: UserId): Promise<User> { /* ... */ }
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/devops/ci-cd.md` — the pipeline gates that host tsc/ESLint/ts-jest.
|
||||||
|
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||||
|
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||||
|
- `languages/ts-types.md` — the type rules ESLint enforces reference this doc.
|
||||||
|
- `languages/ts-testing.md` — the test-runner config (`ts-jest`/`vitest`) detailed here.
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# TypeScript Type System — Derived Application
|
||||||
|
|
||||||
|
> Applies Atelier's domain principles to TypeScript's type system specifically.
|
||||||
|
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||||
|
> See `languages/typescript.md` for the language first-principles stub.
|
||||||
|
|
||||||
|
## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)
|
||||||
|
|
||||||
|
- **TypeScript is structurally typed:** two types with the same shape are assignable. This is convenient but erases domain boundaries — a `UserId` and `PostId` both `string` are interchangeable.
|
||||||
|
- **Branded (nominal) types for domain IDs:** intersect with a phantom brand to simulate nominal typing. The brand is never constructed at runtime; it exists only to the type checker.
|
||||||
|
- **Applies `data/P7` (type fidelity)** at the value boundary: a branded `UserId` cannot be passed where a `PostId` is expected, preventing an entire class of swap bugs.
|
||||||
|
- **Applies `api/P1` (contract fidelity):** branded types make API contracts explicit — handlers cannot accept "any string" for an ID.
|
||||||
|
- **Brand is opaque to consumers:** do not export the brand symbol; construction goes through a validated factory.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
type UserId = string & { readonly __brand: 'UserId' };
|
||||||
|
type PostId = string & { readonly __brand: 'PostId' };
|
||||||
|
|
||||||
|
function userId(s: string): UserId {
|
||||||
|
if (!/^[a-zA-Z0-9]+$/.test(s)) throw new Error('invalid id');
|
||||||
|
return s as UserId;
|
||||||
|
}
|
||||||
|
|
||||||
|
function getUser(id: UserId): User { /* ... */ }
|
||||||
|
getUser('abc'); // type error
|
||||||
|
getUser(userId('abc')); // ok
|
||||||
|
getUser(postId('xyz')); // type error — distinct brands
|
||||||
|
```
|
||||||
|
|
||||||
|
## Generics (C6 Composability, Data P7 Type Fidelity)
|
||||||
|
|
||||||
|
- **Generics preserve type information across boundaries:** a `Repository<T>` keeps the element type through `find`/`save` rather than widening to `any`.
|
||||||
|
- **Constrain with `extends`:** `<T extends Entity>` documents the contract and gives the body access to `T.id`.
|
||||||
|
- **Avoid unnecessary generics:** if a function accepts "any value and returns it unchanged," `T` is noise. Prefer `unknown` for truly opaque inputs.
|
||||||
|
- **Variance is structural:** TS does not enforce sound variance; mark mutation points with `readonly` to keep `T[]` assignable to `readonly T[]`.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface Entity { id: string }
|
||||||
|
class Repository<T extends Entity> {
|
||||||
|
constructor(private db: Map<string, T>) {}
|
||||||
|
find(id: string): T | undefined { return this.db.get(id); }
|
||||||
|
save(t: T): void { this.db.set(t.id, t); }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Narrowing and Type Guards (C1 Correctness, Errors P1 Errors are Data)
|
||||||
|
|
||||||
|
- **Narrowing is how TS handles `unknown` and union types safely:** `typeof`, `in`, `instanceof`, and discriminators collapse a wide type to a precise one before use.
|
||||||
|
- **User-defined type guards (`x is T`) encode domain predicates:** `isUser(x): x is User` lets the checker track the narrow across call sites.
|
||||||
|
- **Applies `errors/P1` (errors are data):** a `Result<T, E>` discriminated union is narrowed with `if (r.ok)` — no `try`/`catch` needed for expected failures.
|
||||||
|
- **Never use `as` to widen past a check:** `as` lies to the compiler. If narrowing does not reach the type you need, the predicate is wrong, not the cast.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
type Result<T, E> = { ok: true; value: T } | { ok: false; error: E };
|
||||||
|
|
||||||
|
function unwrap<T, E>(r: Result<T, E>): T {
|
||||||
|
if (r.ok) return r.value; // narrowed to { ok: true; value: T }
|
||||||
|
throw new Error(JSON.stringify(r.error));
|
||||||
|
}
|
||||||
|
|
||||||
|
function isUser(x: unknown): x is User {
|
||||||
|
return typeof x === 'object' && x !== null && 'id' in x && 'name' in x;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Utility Types (C5 Reversibility, C6 Composability)
|
||||||
|
|
||||||
|
- **`Partial<T>`, `Pick<T,K>`, `Omit<T,K>`, `Readonly<T>` are derived views:** they derive from a source-of-truth `T` rather than redeclaring fields, so the source change propagates (reversibility).
|
||||||
|
- **`Readonly<T>` enforces immutability at the type level** — applies `concurrency/P1` (immutability by default) without runtime cost.
|
||||||
|
- **`Record<K, V>` over `{ [k: string]: V }`:** the index signature form allows any string key including prototype pollution vectors; `Record` is exact.
|
||||||
|
- **Compose, don't accumulate:** `type Patch<T> = Partial<Omit<T, 'id'>>` reads as a transformation; restate it if `T` changes shape, rather than maintaining a parallel `Patch` type.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface User { id: string; name: string; email: string; }
|
||||||
|
type UserPatch = Partial<Omit<User, 'id'>>;
|
||||||
|
type ReadonlyUser = Readonly<User>;
|
||||||
|
type UsersById = Record<string, User>;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Discriminated Unions (C1 Correctness, Data P7 Type Fidelity, Errors P1 Errors are Data)
|
||||||
|
|
||||||
|
- **Discriminated unions over enums:** `type Status = { type: 'pending' } | { type: 'paid'; amount: number }` is exhaustive and carries payload per variant; an `enum` carries neither.
|
||||||
|
- **The discriminant is a literal `type` (or `kind`) field:** the checker narrows on it in `switch` and `if` without a custom guard.
|
||||||
|
- **Exhaustiveness via `never`:** assign the narrowed value to `never` in the default branch; if a variant is added, the default fails to compile.
|
||||||
|
- **Applies `errors/P1`:** model domain errors as a discriminated union `AppError = NotFound | Validation | Conflict`, not as exception classes — the type system carries the error set.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
type Status =
|
||||||
|
| { type: 'pending' }
|
||||||
|
| { type: 'paid'; amount: number }
|
||||||
|
| { type: 'refunded'; reason: string };
|
||||||
|
|
||||||
|
function describe(s: Status): string {
|
||||||
|
switch (s.type) {
|
||||||
|
case 'pending': return 'awaiting payment';
|
||||||
|
case 'paid': return `paid ${s.amount}`;
|
||||||
|
case 'refunded': return `refunded: ${s.reason}`;
|
||||||
|
default:
|
||||||
|
const _exhaustive: never = s; // compile error if a variant is added
|
||||||
|
throw new Error('unhandled');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cross-References
|
||||||
|
|
||||||
|
- `domains/data/schema-design.md` — schema-level fidelity parallels branded types at the TS boundary.
|
||||||
|
- `domains/data/first-principles.md` — Data P7 Type Fidelity, the primary trace for this doc.
|
||||||
|
- `domains/api/rest.md` — contract fidelity for API handlers consuming branded IDs.
|
||||||
|
- `domains/errors/patterns.md` — discriminated unions as the error-as-data encoding.
|
||||||
|
- `languages/ts-async.md` — typed async results built on the `Result` union here.
|
||||||
@@ -2,6 +2,13 @@
|
|||||||
|
|
||||||
> How Atelier's domain principles apply in TypeScript specifically. Derives from `domains/` docs; this file is the language-specific lens.
|
> How Atelier's domain principles apply in TypeScript specifically. Derives from `domains/` docs; this file is the language-specific lens.
|
||||||
|
|
||||||
|
## Derived Docs
|
||||||
|
|
||||||
|
- [ts-types.md](ts-types.md) — TS type system: nominal-via-branding, generics, narrowing, utility types, discriminated unions.
|
||||||
|
- [ts-tooling.md](ts-tooling.md) — tsc, ESLint, ts-jest, project references, tsconfig discipline.
|
||||||
|
- [ts-async.md](ts-async.md) — Promises + AbortSignal, async/await, error handling, cancellation.
|
||||||
|
- [ts-testing.md](ts-testing.md) — Vitest/Jest, mock discipline, type-level tests.
|
||||||
|
|
||||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||||
|
|
||||||
- **Strict mode on:** `strict: true` in `tsconfig.json`. No `any` without justification.
|
- **Strict mode on:** `strict: true` in `tsconfig.json`. No `any` without justification.
|
||||||
|
|||||||
@@ -6,14 +6,14 @@
|
|||||||
|
|
||||||
| Core Principle | Domains that derive from it | Count |
|
| Core Principle | Domains that derive from it | Count |
|
||||||
|----------------|---------------------------|-------|
|
|----------------|---------------------------|-------|
|
||||||
| C1 Correctness | All 17 (v0.1: 11; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance) | Universal |
|
| C1 Correctness | All 19 (v0.1: 11; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging) | Universal |
|
||||||
| C2 Clarity | v0.1: uiux, api, data, testing, observability, errors, documentation, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 14 |
|
| C2 Clarity | v0.1: uiux, api, data, testing, observability, errors, documentation, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: messaging | 16 |
|
||||||
| C3 Simplicity | v0.1: security, data, testing, performance, documentation, concurrency, devops; v0.2: infrastructure-as-code; v0.3: gitops-operators, i18n, compliance | 11 |
|
| C3 Simplicity | v0.1: security, data, testing, performance, documentation, concurrency, devops; v0.2: infrastructure-as-code; v0.3: gitops-operators, i18n, compliance; v0.4: edge | 12 |
|
||||||
| C4 Locality | v0.1: testing, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, i18n | 6 |
|
| C4 Locality | v0.1: testing, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, i18n; v0.4: edge, messaging | 8 |
|
||||||
| C5 Reversibility | v0.1: api, data, uiux, concurrency, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 11 |
|
| C5 Reversibility | v0.1: api, data, uiux, concurrency, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging | 13 |
|
||||||
| C6 Composability | v0.1: api, security, observability, errors, documentation, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 12 |
|
| C6 Composability | v0.1: api, security, observability, errors, documentation, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging | 14 |
|
||||||
| C7 Observability | v0.1: api, data, testing, performance, observability, errors, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 13 |
|
| C7 Observability | v0.1: api, data, testing, performance, observability, errors, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging | 15 |
|
||||||
| C8 Economy | v0.1: security, testing, performance, observability, concurrency; v0.2: kubernetes; v0.3: gitops-operators, i18n, compliance | 9 |
|
| C8 Economy | v0.1: security, testing, performance, observability, concurrency; v0.2: kubernetes; v0.3: gitops-operators, i18n, compliance; v0.4: edge, messaging | 11 |
|
||||||
|
|
||||||
## Interpretation
|
## Interpretation
|
||||||
|
|
||||||
@@ -43,6 +43,8 @@
|
|||||||
| AI / ML | C1, C2, C5, C6, C7 | Reproducibility + lineage + serving observability |
|
| AI / ML | C1, C2, C5, C6, C7 | Reproducibility + lineage + serving observability |
|
||||||
| i18n | C1, C2, C3, C4, C5, C6, C7, C8 | Locale + formatting + direction + reversibility; broadest derivation (8 C-rules) |
|
| i18n | C1, C2, C3, C4, C5, C6, C7, C8 | Locale + formatting + direction + reversibility; broadest derivation (8 C-rules) |
|
||||||
| Compliance | C1, C2, C3, C5, C6, C7, C8 | Audit + policy-as-code + retention + posture |
|
| Compliance | C1, C2, C3, C5, C6, C7, C8 | Audit + policy-as-code + retention + posture |
|
||||||
|
| Edge | C1, C3, C4, C5, C6, C7, C8 | Proximity + offline + constraint; broad derivation (7 C-rules) |
|
||||||
|
| Messaging | C1, C2, C4, C5, C6, C7, C8 | Cross-process delivery + ordering + idempotency; broad derivation (7 C-rules) |
|
||||||
|
|
||||||
## v0.2 Domain Coverage (per IDEATE-03 schema)
|
## v0.2 Domain Coverage (per IDEATE-03 schema)
|
||||||
|
|
||||||
@@ -60,6 +62,39 @@
|
|||||||
| i18n | 10 | 4 (locale-resources, formatting, rtl-bidi, testing-i18n) | ✓ | complete |
|
| i18n | 10 | 4 (locale-resources, formatting, rtl-bidi, testing-i18n) | ✓ | complete |
|
||||||
| Compliance | 10 | 4 (audit-logs, data-retention, policy-as-code, evidence) | ✓ | complete |
|
| Compliance | 10 | 4 (audit-logs, data-retention, policy-as-code, evidence) | ✓ | complete |
|
||||||
|
|
||||||
|
## v0.4 Domain Coverage (per IDEATE-03 schema)
|
||||||
|
|
||||||
|
| Domain | P-count | Derived-doc-count | Manifest-listed | Status |
|
||||||
|
|--------|---------|-------------------|-----------------|--------|
|
||||||
|
| Edge | 10 | 4 (cdn, offline-first, iot, sync) | ✓ | complete |
|
||||||
|
| Messaging | 10 | 4 (queues, pubsub, streams, delivery-semantics) | ✓ | complete |
|
||||||
|
|
||||||
|
## Languages Coverage (v0.4, per D-066)
|
||||||
|
|
||||||
|
> Language-derived docs apply existing domain P-rules. They introduce
|
||||||
|
> no new matrix rows (D-063, D-066). The sub-table is the verifiable
|
||||||
|
> aggregate; inline section-headers are the per-doc detail. Every
|
||||||
|
> derived doc traces to ≥1 domain P-rule (D-066, no orphans).
|
||||||
|
|
||||||
|
| Language | Derived Doc | Traced Domain P-rules | Traced Core C-rules |
|
||||||
|
|----------|-------------|------------------------|---------------------|
|
||||||
|
| TypeScript | ts-types.md | Data P7, API P1, Errors P1 | C1, C5, C6 |
|
||||||
|
| TypeScript | ts-tooling.md | DevOps P2, DevOps P1, Documentation P1, DevOps P9 | C1, C2, C6, C8 |
|
||||||
|
| TypeScript | ts-async.md | Concurrency P7, Concurrency P8, Concurrency P9, Errors P5, Errors P1 | C1, C2, C5 |
|
||||||
|
| TypeScript | ts-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Data P7 | C1, C2 |
|
||||||
|
| Python | py-types.md | Data P7, API P1, Errors P1 | C1, C2, C6 |
|
||||||
|
| Python | py-tooling.md | DevOps P2, DevOps P1, Documentation P1, DevOps P9 | C1, C2, C3, C8 |
|
||||||
|
| Python | py-async.md | Concurrency P1, Concurrency P7, Concurrency P8, Concurrency P9, Errors P5, Errors P1 | C1, C2, C5, C6 |
|
||||||
|
| Python | py-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Concurrency P10 | C1, C2 |
|
||||||
|
| Go | go-types.md | Data P7, API P1, Errors P1, Errors P3 | C1, C2, C6 |
|
||||||
|
| Go | go-tooling.md | DevOps P2, DevOps P1, Concurrency P6, Documentation P1, DevOps P9 | C1, C2, C3, C7, C8 |
|
||||||
|
| Go | go-concurrency.md | Concurrency P1, Concurrency P3, Concurrency P5, Concurrency P6, Concurrency P7, Concurrency P8, Concurrency P9 | C1, C3, C4, C5, C6, C8 |
|
||||||
|
| Go | go-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Concurrency P6, Concurrency P10, API P1 | C1, C2 |
|
||||||
|
| Rust | rs-ownership.md | Concurrency P1, Concurrency P3, Concurrency P5, Data P7 | C1, C3, C4 |
|
||||||
|
| Rust | rs-tooling.md | DevOps P2, DevOps P1, Documentation P1, DevOps P9 | C1, C2, C3, C5, C8 |
|
||||||
|
| Rust | rs-async.md | Concurrency P5, Concurrency P7, Concurrency P8, Concurrency P9 | C1, C3, C5, C6, C8 |
|
||||||
|
| Rust | rs-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Concurrency P10, Documentation P1 | C1, C2 |
|
||||||
|
|
||||||
## Gaps and Notes
|
## Gaps and Notes
|
||||||
|
|
||||||
- No domain derives from only one C-rule. The minimum is 4 (UI/UX: C1, C2, C3, C5, C7 — actually 5). Every domain is multi-rooted.
|
- No domain derives from only one C-rule. The minimum is 4 (UI/UX: C1, C2, C3, C5, C7 — actually 5). Every domain is multi-rooted.
|
||||||
@@ -68,4 +103,6 @@
|
|||||||
- **UI/UX** and **API** are the most user-facing; they emphasize C2 (Clarity) heavily.
|
- **UI/UX** and **API** are the most user-facing; they emphasize C2 (Clarity) heavily.
|
||||||
- **Security** is the only domain with explicit non-tradeable declarations; this promotes 8 of its rules to C1-equivalent per `core/conflict-resolution.md` §6.
|
- **Security** is the only domain with explicit non-tradeable declarations; this promotes 8 of its rules to C1-equivalent per `core/conflict-resolution.md` §6.
|
||||||
- **v0.2 expansion:** C4 (Locality) grew from 2 to 4 domains (added infrastructure-as-code state locality, kubernetes namespace blast-radius). C6 (Composability) grew from 6 to 8. The two new domains are broad-derivation domains (7 C-rules each), consistent with Concurrency's breadth.
|
- **v0.2 expansion:** C4 (Locality) grew from 2 to 4 domains (added infrastructure-as-code state locality, kubernetes namespace blast-radius). C6 (Composability) grew from 6 to 8. The two new domains are broad-derivation domains (7 C-rules each), consistent with Concurrency's breadth.
|
||||||
- **v0.3 expansion:** C3 (Simplicity) grew from 7 to 11 (added gitops-operators declarative simplicity, i18n flexible layout, compliance structural redaction). C4 (Locality) grew from 4 to 6 (added gitops-operators pull-credential locality, i18n resource/text-direction locality). C5 (Reversibility) grew from 7 to 11 (added all four v0.3 domains — gitops history, ai-ml reproducibility, i18n translation versioning, compliance append-only/retention). C6 (Composability) grew from 8 to 12. C7 (Observability) grew from 9 to 13 (added all four v0.3 domains — reconciliation, drift detection, format correctness, posture). C2 (Clarity) grew from 10 to 14. The v0.3 expansion broadens every non-universal C-rule's coverage, confirming the four new domains are cross-cutting and well-rooted.
|
- **v0.3 expansion:** C3 (Simplicity) grew from 7 to 11 (added gitops-operators declarative simplicity, i18n flexible layout, compliance structural redaction). C4 (Locality) grew from 4 to 6 (added gitops-operators pull-credential locality, i18n resource/text-direction locality). C5 (Reversibility) grew from 7 to 11 (added all four v0.3 domains — gitops history, ai-ml reproducibility, i18n translation versioning, compliance append-only/retention). C6 (Composability) grew from 8 to 12. C7 (Observability) grew from 9 to 13 (added all four v0.3 domains — reconciliation, drift detection, format correctness, posture). C2 (Clarity) grew from 10 to 14. The v0.3 expansion broadens every non-universal C-rule's coverage, confirming the four new domains are cross-cutting and well-rooted.
|
||||||
|
- **v0.4 expansion:** C2 (Clarity) grew from 14 to 16 (added messaging P1/P2/P4 — contract clarity). C4 (Locality) grew from 6 to 8 (added edge P1/P8 geographic locality, messaging P7 partition locality). C5 (Reversibility) grew from 11 to 13 (added edge P2/P4/P7/P10 reconciliation + degradation, messaging P5/P8/P9 replay + evolution). C6 (Composability) grew from 12 to 14 (added edge P8 fleet composition, messaging P7/P9 partition + version composition). C7 (Observability) grew from 13 to 15 (added edge P10 partition-survivable telemetry, messaging P8/P10 durable-log + lag). C8 (Economy) grew from 9 to 11 (added edge P3/P9 constrained resources + trust, messaging P6 bounded backpressure). C3 (Simplicity) grew from 11 to 12 (added edge P6 explicit invalidation simplicity). Both v0.4 domains are broad-derivation (7 C-rules each), consistent with the v0.2/v0.3 precedent.
|
||||||
|
- **Languages Coverage (v0.4, D-066):** The 16 language-derived docs (4 languages × 4 docs) apply existing domain P-rules; they introduce no new matrix rows (D-063, D-066). The sub-table above is the verifiable aggregate — every derived doc traces to ≥1 domain P-rule (no orphans). Languages are an application lens, not a principle source.
|
||||||
@@ -265,8 +265,39 @@ C5=Reversibility · C6=Composability · C7=Observability · C8=Economy
|
|||||||
| P9 Secrets and Sensitive Data are Redacted in Audit | C1, C3 | Correctness via security; simplicity of structural redaction |
|
| P9 Secrets and Sensitive Data are Redacted in Audit | C1, C3 | Correctness via security; simplicity of structural redaction |
|
||||||
| P10 Compliance Posture is Observable | C7, C1 | Observability of compliance; correctness of posture |
|
| P10 Compliance Posture is Observable | C7, C1 | Observability of compliance; correctness of posture |
|
||||||
|
|
||||||
## Coverage Summary (post-v0.3)
|
## Edge
|
||||||
|
|
||||||
- 17 domains (11 v0.1 + 2 v0.2: infrastructure-as-code, kubernetes; 4 v0.3: gitops-operators, ai-ml, i18n, compliance)
|
| Edge Principle | Core | Why |
|
||||||
- 170 domain principles total (110 v0.1 + 20 v0.2 + 40 v0.3)
|
|-------------------------------------------------------|-------|--------------------------------------------------|
|
||||||
- Every domain P-rule traces to ≥1 core C-rule (C1–C8). No orphans.
|
| P1 Proximity is the Design Driver | C4, C1 | Locality of compute near user/data; correctness via latency |
|
||||||
|
| P2 Offline is a First-Class State | C1, C5 | Correctness under partition; reversibility of reconciliation |
|
||||||
|
| P3 Resources are Constrained and Declared | C8, C1 | Economy of constrained nodes; correctness of declared bounds |
|
||||||
|
| P4 Sync Conflicts are Bounded, Not Infinite | C1, C5 | Correctness of convergence; reversibility of divergent state |
|
||||||
|
| P5 Edge Operations are Idempotent | C1 | Correctness under retry |
|
||||||
|
| P6 Cache Invalidation is Explicit | C1, C3 | Correctness of cached state; simplicity of defined invalidation |
|
||||||
|
| P7 Partial Degradation is Engineered | C1, C5 | Correctness of degraded modes; reversibility of recovery |
|
||||||
|
| P8 Geographic Distribution is a First-Class Constraint | C4, C6 | Locality of placement; composability of the fleet |
|
||||||
|
| P9 Identity is Constrained at the Edge | C1, C8 | Correctness via security; economy of trust |
|
||||||
|
| P10 Edge Observability Survives Partition | C7, C5 | Observability of the fleet; reversibility of buffered telemetry |
|
||||||
|
|
||||||
|
## Messaging
|
||||||
|
|
||||||
|
| Messaging Principle | Core | Why |
|
||||||
|
|--------------------------------------------------|-------|--------------------------------------------------|
|
||||||
|
| P1 Messages are Contracts | C1, C2 | Correctness of the exchange; clarity of the schema boundary |
|
||||||
|
| P2 Ordering is a Property, Not an Assumption | C1, C2 | Correctness of order; clarity of the guarantee |
|
||||||
|
| P3 Consumers are Idempotent | C1 | Correctness under redelivery |
|
||||||
|
| P4 Delivery Semantics are Explicit | C1, C2 | Correctness of the chosen semantic; clarity of the tradeoff |
|
||||||
|
| P5 Dead-Letter Handling is Defined | C1, C5 | Correctness of poison-message routing; reversibility of reprocessing |
|
||||||
|
| P6 Backpressure is Bounded | C1, C8 | Correctness of bounded backlog; economy of broker memory |
|
||||||
|
| P7 Partitioning is Intentional | C4, C6 | Locality of order; composability of parallelism |
|
||||||
|
| P8 Replay and Retention are Configured | C5, C7 | Reversibility of replay; observability of the durable log |
|
||||||
|
| P9 Schemas Evolve Compatibly | C5, C6 | Reversibility of schema changes; composability of versions |
|
||||||
|
| P10 Messaging is Observable | C7, C1 | Observability of lag/DLQ; correctness of backlog detection |
|
||||||
|
|
||||||
|
## Coverage Summary (post-v0.4)
|
||||||
|
|
||||||
|
- 19 domains (11 v0.1 + 2 v0.2: infrastructure-as-code, kubernetes; 4 v0.3: gitops-operators, ai-ml, i18n, compliance; 2 v0.4: edge, messaging)
|
||||||
|
- 190 domain principles total (110 v0.1 + 20 v0.2 + 40 v0.3 + 20 v0.4)
|
||||||
|
- Every domain P-rule traces to ≥1 core C-rule (C1–C8). No orphans.
|
||||||
|
- Language-derived docs (16) apply existing domain P-rules; they introduce no new matrix rows (D-063, D-066). See `matrix/domain-coverage.md` `## Languages Coverage` sub-table.
|
||||||
@@ -197,6 +197,37 @@ If the task touches a domain, run that domain's checklist:
|
|||||||
- [ ] Audit logs do not leak secrets; redaction is structural, not opportunistic (P9)
|
- [ ] Audit logs do not leak secrets; redaction is structural, not opportunistic (P9)
|
||||||
- [ ] System reports its own compliance state (drift from policy, open violations, retention status) (P10)
|
- [ ] System reports its own compliance state (drift from policy, open violations, retention status) (P10)
|
||||||
|
|
||||||
|
### If Edge (see `domains/edge/`)
|
||||||
|
- [ ] Compute is placed near the user/data source; latency is treated as a correctness constraint, not a perf preference (P1)
|
||||||
|
- [ ] The system continues to operate offline; partition is the norm, not the exception; reconciliation happens on reconnect (P2)
|
||||||
|
- [ ] Edge-node resource constraints (CPU/memory/power/bandwidth) are declared per node class, not assumed infinite (P3)
|
||||||
|
- [ ] Sync conflicts converge; no oscillation or infinite sync loops (P4)
|
||||||
|
- [ ] Sync, cache-fill, and device commands are idempotent — retries are safe (P5)
|
||||||
|
- [ ] Edge caches have an explicit TTL or invalidation strategy; no TTL-less caches under partition (P6)
|
||||||
|
- [ ] Partial degradation is a designed state with a defined contract, not a crash (P7)
|
||||||
|
- [ ] Routing, fan-out, and data placement are location-aware decisions, not accidents of deployment (P8)
|
||||||
|
- [ ] Edge-device credentials are scoped and per-device; no edge node is a cluster-admin-equivalent (P9)
|
||||||
|
- [ ] Telemetry is local-first: buffered on-node and forwarded on reconnect; partition does not blind the operator (P10)
|
||||||
|
|
||||||
|
### If Messaging (see `domains/messaging/`)
|
||||||
|
- [ ] Messages have an explicit, versioned schema; producer and consumer agree on shape before exchange (P1)
|
||||||
|
- [ ] Ordering guarantees (per-partition, global, none) are explicit and documented; "FIFO" is backed by the broker contract (P2)
|
||||||
|
- [ ] Consumers are idempotent — redelivery is deduped via idempotency keys or deterministic processing (P3)
|
||||||
|
- [ ] Delivery semantics (at-least-once/at-most-once/exactly-once) are a declared choice per channel (P4)
|
||||||
|
- [ ] Poison messages route to a dead-letter queue; the DLQ is observable and drainable (P5)
|
||||||
|
- [ ] Backpressure is bounded — consumer lag visible, max-unacked bounded, retry budget capped (P6)
|
||||||
|
- [ ] Partition key choice is a documented design decision (ordering vs parallelism vs hotspots) (P7)
|
||||||
|
- [ ] Retention windows and replay-from-offset are explicit; the broker is a durable log, not a pipe (P8)
|
||||||
|
- [ ] Schema changes are backward/forward-compatible; breaking changes are versioned migrations, not silent shape edits (P9)
|
||||||
|
- [ ] Consumer lag, DLQ depth, throughput, and consumer-group health are observed; silent backlog is a bug (P10)
|
||||||
|
|
||||||
|
### If Language-Derived Docs (see `languages/`)
|
||||||
|
- [ ] Scope check: no new P-rules introduced — every section traces to an existing domain P-rule (D-063, D-066)
|
||||||
|
- [ ] Every section header names ≥1 traced domain P-rule AND the core C-rule(s) inline (e.g., `## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)`)
|
||||||
|
- [ ] Fenced code examples are in-language and illustrative only — no standalone `.ts`/`.py`/`.go`/`.rs` runtime artifacts (D-020)
|
||||||
|
- [ ] The first-principles stub retains its section structure — no P1–P10 numbering added to languages (D-063)
|
||||||
|
- [ ] Cross-links to traced domain docs are present (≥1 outbound per derived doc, ATELIER-114)
|
||||||
|
|
||||||
## Final Gate
|
## Final Gate
|
||||||
|
|
||||||
- [ ] Have I read the relevant domain's first-principles?
|
- [ ] Have I read the relevant domain's first-principles?
|
||||||
|
|||||||
@@ -188,6 +188,72 @@ The following standalone file types are forbidden under `examples/` and elsewher
|
|||||||
| Signed manifests as standalone files (`.sig`, `.att`, `.intoto.jsonl`) | fenced code in `examples/*.md` | Runtime attestation artifact; violates docs-only contract |
|
| Signed manifests as standalone files (`.sig`, `.att`, `.intoto.jsonl`) | fenced code in `examples/*.md` | Runtime attestation artifact; violates docs-only contract |
|
||||||
| Standalone `.yaml` / `.tf` / `.sh` | fenced code in `examples/*.md` | (Carried forward from v0.2) Runtime deployable artifact |
|
| Standalone `.yaml` / `.tf` / `.sh` | fenced code in `examples/*.md` | (Carried forward from v0.2) Runtime deployable artifact |
|
||||||
|
|
||||||
|
## v0.4 Chaos Anti-Patterns (from IDEATE-36, IDEATE-37, D-068, ATELIER-110)
|
||||||
|
|
||||||
|
These are named, cross-cutting violations specific to the v0.4 domains. Single-breach per D-068. Reject on sight.
|
||||||
|
|
||||||
|
| Anti-Pattern | Breaches | Why |
|
||||||
|
|--------------|----------|-----|
|
||||||
|
| Edge sync conflict loop (two nodes oscillate divergent state forever; no convergence) | edge P4 Sync Conflicts are Bounded, Not Infinite; C1, C5 | Infinite oscillation is a correctness failure, not eventual consistency; the sync is a livelock |
|
||||||
|
| Edge TTL-less cache (edge cache with no TTL and no explicit invalidation; stale forever under partition) | edge P6 Cache Invalidation is Explicit; C1, C3 | Stale-forever under partition is a silent correctness defect |
|
||||||
|
| Edge blocking call on constrained device (a synchronous blocking I/O call on a low-power IoT device with no timeout) | edge P3 Resources are Constrained, P5 (retry safety); C8, C1 | Blocks the constrained node; no timeout = hang; retry is unsafe without idempotency |
|
||||||
|
| Messaging shared subscription (two consumers share one subscription; each message processed by an arbitrary consumer, breaking per-consumer ordering and idempotency) | messaging P2 Ordering, P3 Idempotency; C1 | Shared subscription breaks ordering and dedup; the `messaging-shared-subscription` bad example lives here |
|
||||||
|
| Messaging blocking consumer (a consumer blocks on a slow downstream call with no timeout; the broker re-delivers to the same stuck consumer) | messaging P6 Backpressure is Bounded; C1, C8 | Blocks the partition; lag grows; no backpressure escape |
|
||||||
|
| Messaging unbounded retry / no DLQ (a poison message is retried forever with no dead-letter queue; the consumer never makes progress) | messaging P5 Dead-Letter Handling is Defined; C1, C5 | Infinite retry is a livelock; poison messages must DLQ |
|
||||||
|
|
||||||
|
### v0.4 Deployable Artifact Types (IDEATE-36, D-020)
|
||||||
|
|
||||||
|
The following standalone file types are forbidden under `languages/` and elsewhere in the framework (carried forward from v0.2/v0.3 and extended for language-derived docs). All examples and derived-doc code are `.md` files with fenced code only.
|
||||||
|
|
||||||
|
| Forbidden standalone artifact | Belongs in | Why |
|
||||||
|
|-------------------------------|-----------|-----|
|
||||||
|
| Standalone `.ts` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime TypeScript artifact; violates docs-only contract (v0.4 extension) |
|
||||||
|
| Standalone `.py` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime Python artifact; violates docs-only contract (v0.4 extension) |
|
||||||
|
| Standalone `.go` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime Go artifact; violates docs-only contract (v0.4 extension) |
|
||||||
|
| Standalone `.rs` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime Rust artifact; violates docs-only contract (v0.4 extension) |
|
||||||
|
| Standalone `.yaml` / `.tf` / `.sh` | fenced code in `examples/*.md` | (Carried forward from v0.2/v0.3) Runtime deployable artifact |
|
||||||
|
|
||||||
|
### v0.4 Language-Derived-Doc Anti-Patterns
|
||||||
|
|
||||||
|
| Anti-Pattern | Breaches | Why |
|
||||||
|
|--------------|----------|-----|
|
||||||
|
| Language-derived doc introducing a new P-rule (e.g., "TS P1", "Python P2") | D-063, ARCHITECTURE.md hierarchy, matrix row-count invariant | Languages apply, not derive; breaks the hierarchy (`core > domains > languages`) and the 190-row invariant |
|
||||||
|
| Language-derived doc with no domain P-rule trace (orphan section — header names no domain P-rule) | D-066, matrix completeness | No trace = no traceability; the doc is not connected to the framework |
|
||||||
|
| Language-derived doc with runtime artifact (standalone `.ts`/`.py`/`.go`/`.rs` file) | D-020, PROJECT.md no-runtime-code | Violates the docs-only contract; must be fenced code in `.md` |
|
||||||
|
| Language first-principles stub gaining P1–P10 numbering | D-063 | Languages are not domains; no P-rules to number; breaks the apply-don't-derive scope |
|
||||||
|
|
||||||
|
## v0.4 Domain-Specific Anti-Patterns
|
||||||
|
|
||||||
|
### Edge
|
||||||
|
|
||||||
|
| Anti-Pattern | Breaches | Why |
|
||||||
|
|--------------|----------|-----|
|
||||||
|
| Central-region-only deployment for a latency-bound workload | P1 Proximity is the Design Driver | Violates the proximity constraint; latency is a correctness defect |
|
||||||
|
| App that crashes on disconnect (no offline state) | P2 Offline is a First-Class State | Partition is the norm; crash-on-disconnect is unengineered |
|
||||||
|
| Undeclared edge-node resource budget (assumes infinite CPU/memory) | P3 Resources are Constrained and Declared | Unbounded = OOM on constrained nodes |
|
||||||
|
| Sync loop that oscillates forever (CRDT without merge-semantics, LWW without monotonic clock) | P4 Sync Conflicts are Bounded | Livelock; convergence is never reached |
|
||||||
|
| Non-idempotent edge write (cache-fill or device command retried with side effects) | P5 Edge Operations are Idempotent | Retry doubles the side effect |
|
||||||
|
| TTL-less edge cache under partition | P6 Cache Invalidation is Explicit | Stale-forever; silent staleness |
|
||||||
|
| Crash-on-node-failure (no partial-degradation contract) | P7 Partial Degradation is Engineered | Whole-system crash on one node's failure |
|
||||||
|
| Random geographic placement (no location-aware routing) | P8 Geographic Distribution is a First-Class Constraint | Latency and data-residency uncontrolled |
|
||||||
|
| Shared edge-device credential (one key for the whole fleet) | P9 Identity is Constrained at the Edge | One compromise = fleet compromise |
|
||||||
|
| Fire-and-forget telemetry (no on-node buffer; data lost on partition) | P10 Edge Observability Survives Partition | Partition blinds the operator |
|
||||||
|
|
||||||
|
### Messaging
|
||||||
|
|
||||||
|
| Anti-Pattern | Breaches | Why |
|
||||||
|
|--------------|----------|-----|
|
||||||
|
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts | No boundary; consumer breaks silently on shape change |
|
||||||
|
| "It's FIFO" with no documented partition contract | P2 Ordering is a Property, Not an Assumption | Undocumented ordering = assumed ordering = wrong ordering |
|
||||||
|
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent | Redelivery doubles the effect |
|
||||||
|
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit | The tradeoff is made blindly |
|
||||||
|
| No dead-letter queue (poison message retried forever or dropped) | P5 Dead-Letter Handling is Defined | Livelock or silent loss |
|
||||||
|
| Unbounded retry budget (no cap; slow consumer stalls the partition) | P6 Backpressure is Bounded | Lag grows without bound |
|
||||||
|
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional | Hotspot or ordering violation |
|
||||||
|
| Ephemeral broker (no retention; no replay) | P8 Replay and Retention are Configured | Not a durable log; replay impossible |
|
||||||
|
| Silent schema change (producer breaks consumers with no version bump) | P9 Schemas Evolve Compatibly | Breaking change without migration |
|
||||||
|
| Silent backlog (no lag metric; consumer falls behind invisibly) | P10 Messaging is Observable | Cannot fix what you cannot see |
|
||||||
|
|
||||||
## v0.3 Domain-Specific Anti-Patterns
|
## v0.3 Domain-Specific Anti-Patterns
|
||||||
|
|
||||||
### GitOps + Operators
|
### GitOps + Operators
|
||||||
|
|||||||
@@ -129,6 +129,39 @@ Run the relevant domain section from `agent-checklist.md` (UI/UX, API, Security,
|
|||||||
- [ ] Do audit logs avoid leaking secrets (redaction is structural, not opportunistic)?
|
- [ ] Do audit logs avoid leaking secrets (redaction is structural, not opportunistic)?
|
||||||
- [ ] Does the system report its own compliance state (drift from policy, open violations, retention status)?
|
- [ ] Does the system report its own compliance state (drift from policy, open violations, retention status)?
|
||||||
|
|
||||||
|
### If Edge
|
||||||
|
- [ ] Scope check: is this the proximity/location/constraint angle, NOT generic perf or in-process concurrency? (D-061 — reject generic-performance or in-process-concurrency content as out-of-scope)
|
||||||
|
- [ ] Is compute placed near the user/data source, with latency treated as a correctness constraint (not a perf preference)? (P1)
|
||||||
|
- [ ] Does the system continue to operate offline; is partition the norm, not the exception? (P2)
|
||||||
|
- [ ] Are edge-node resource constraints (CPU/memory/power/bandwidth) declared per node class, not assumed infinite? (P3)
|
||||||
|
- [ ] Do sync conflicts converge; is there no oscillation or infinite sync loop? (P4)
|
||||||
|
- [ ] Are sync, cache-fill, and device commands idempotent (retries safe)? (P5)
|
||||||
|
- [ ] Do edge caches have an explicit TTL or invalidation strategy (no TTL-less caches under partition)? (P6)
|
||||||
|
- [ ] Is partial degradation a designed state with a defined contract (not a crash)? (P7)
|
||||||
|
- [ ] Are routing, fan-out, and data placement location-aware decisions? (P8)
|
||||||
|
- [ ] Are edge-device credentials scoped and per-device (no edge cluster-admin)? (P9)
|
||||||
|
- [ ] Is telemetry local-first (buffered on-node, forwarded on reconnect; partition does not blind the operator)? (P10)
|
||||||
|
|
||||||
|
### If Messaging
|
||||||
|
- [ ] Scope check: is this cross-process/network-failure-model, NOT in-process concurrency? (D-062 — reject in-process-concurrency content as out-of-scope)
|
||||||
|
- [ ] Do messages have an explicit, versioned schema (producer and consumer agree on shape)? (P1)
|
||||||
|
- [ ] Are ordering guarantees (per-partition, global, none) explicit and documented? (P2)
|
||||||
|
- [ ] Are consumers idempotent (redelivery deduped via idempotency keys or deterministic processing)? (P3)
|
||||||
|
- [ ] Are delivery semantics (at-least-once/at-most-once/exactly-once) a declared choice per channel? (P4)
|
||||||
|
- [ ] Do poison messages route to a dead-letter queue (DLQ observable and drainable)? (P5)
|
||||||
|
- [ ] Is backpressure bounded (consumer lag visible, max-unacked bounded, retry budget capped)? (P6)
|
||||||
|
- [ ] Is partition key choice a documented design decision (ordering vs parallelism vs hotspots)? (P7)
|
||||||
|
- [ ] Are retention windows and replay-from-offset explicit (durable log, not pipe)? (P8)
|
||||||
|
- [ ] Are schema changes backward/forward-compatible (breaking changes are versioned migrations)? (P9)
|
||||||
|
- [ ] Are consumer lag, DLQ depth, throughput, and consumer-group health observed (silent backlog is a bug)? (P10)
|
||||||
|
|
||||||
|
### If Language-Derived Docs
|
||||||
|
- [ ] Scope check: apply, don't derive — no new P-rules, no new matrix rows (D-063, D-066 — reject any "TS P1"/"Python P2" etc. new-P-rule content)
|
||||||
|
- [ ] Does every section header name ≥1 traced domain P-rule AND the core C-rule(s) inline?
|
||||||
|
- [ ] Are fenced code examples in-language and illustrative only (no standalone `.ts`/`.py`/`.go`/`.rs` runtime artifacts)? (D-020)
|
||||||
|
- [ ] Does the first-principles stub retain its section structure (no P1–P10 numbering added to languages)? (D-063)
|
||||||
|
- [ ] Are cross-links to traced domain docs present (≥1 outbound per derived doc)? (ATELIER-114)
|
||||||
|
|
||||||
## Review Etiquette
|
## Review Etiquette
|
||||||
|
|
||||||
- **Comment, don't command.** "This could be X" not "Change this to X."
|
- **Comment, don't command.** "This could be X" not "Change this to X."
|
||||||
|
|||||||
Reference in New Issue
Block a user