Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Jon Chery
|
7fdd143d03 | ||
|
Jon Chery
|
9f5ed4b5cd | ||
|
Jon Chery
|
4e433158cd | ||
|
Jon Chery
|
0fdf892d81 | ||
|
Jon Chery
|
f61dffbb5a | ||
|
Jon Chery
|
f7f007dce8 | ||
|
Jon Chery
|
20992883ff | ||
|
Jon Chery
|
a622419e7d |
@@ -1,13 +1,15 @@
|
||||
{
|
||||
"phase": 0,
|
||||
"stage": "plan",
|
||||
"phase": 3,
|
||||
"stage": "complete",
|
||||
"milestone": "v0.4",
|
||||
"phase_role": "pre_execution",
|
||||
"phase_role": "execution",
|
||||
"project": "atelier",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-05T05:20:00Z",
|
||||
"updated_at": "2026-08-05T06:45:00Z",
|
||||
"milestone_complete": false,
|
||||
"milestone_branch": "milestone/v0.4-edge-quantum-langs",
|
||||
"phase_branch": "phase/00-pre-execution",
|
||||
"tag_base": "v0.3"
|
||||
"phase_branch": "phase/03-languages-derived",
|
||||
"tag_base": "v0.3",
|
||||
"phase_tag": "v0.3.3",
|
||||
"release_id": 485
|
||||
}
|
||||
@@ -16,8 +16,18 @@ atelier/
|
||||
│ ├── gitops-operators/ # v0.3: GitOps + Operators (argocd, flux, operators, progressive-delivery)
|
||||
│ ├── ai-ml/ # v0.3: ML engineering (data-versioning, model-evaluation, serving, monitoring-drift)
|
||||
│ ├── i18n/ # v0.3: internationalization (locale-resources, formatting, rtl-bidi, testing-i18n)
|
||||
│ └── compliance/ # v0.3: compliance/audit (audit-logs, data-retention, policy-as-code, evidence)
|
||||
│ ├── compliance/ # v0.3: compliance/audit (audit-logs, data-retention, policy-as-code, evidence)
|
||||
│ ├── edge/ # v0.4: edge computing (cdn, offline-first, iot, sync)
|
||||
│ └── messaging/ # v0.4: cross-process messaging (queues, pubsub, streams, delivery-semantics)
|
||||
├── languages/ # Language-specific application of domains
|
||||
│ ├── typescript.md # v0.1 first-principles stub
|
||||
│ ├── python.md # v0.1 first-principles stub
|
||||
│ ├── go.md # v0.1 first-principles stub
|
||||
│ ├── rust.md # v0.1 first-principles stub
|
||||
│ ├── ts-*.md # v0.4: 4 derived docs (ts-types, ts-tooling, ts-async, ts-testing)
|
||||
│ ├── py-*.md # v0.4: 4 derived docs (py-types, py-tooling, py-async, py-testing)
|
||||
│ ├── go-*.md # v0.4: 4 derived docs (go-types, go-tooling, go-concurrency, go-testing)
|
||||
│ └── rs-*.md # v0.4: 4 derived docs (rs-ownership, rs-tooling, rs-async, rs-testing)
|
||||
├── review/ # Evaluation checklists and anti-patterns
|
||||
├── matrix/ # Cross-reference: domain ↔ core
|
||||
└── examples/ # Worked examples (good + bad)
|
||||
@@ -106,4 +116,28 @@ From the v0.3 ideation stage (IDEATE-17..30), the following architectural refine
|
||||
- **Core Principle Coverage table (IDEATE-19 → ATELIER-81):** `matrix/domain-coverage.md` contains two tables — the per-domain row schema table (covered by v0.2 IDEATE-03) AND the "Core Principle Coverage" table mapping C1–C8 → domains. Both must be extended for the 4 new domains; the C-rule counts shift (e.g., C4 Locality adds i18n + gitops; C5 Reversibility adds ai-ml + compliance + gitops + i18n).
|
||||
- **Cross-link type unchanged:** v0.3 introduces no new cross-link type. All cross-links remain one-directional outward from new domains to existing (D-033). No back-link edits to v0.1/v0.2 content.
|
||||
|
||||
See `.ciagent/atelier/RESEARCH.md` "v0.3 Research" for the full prior-art survey and principle inventory rationale, and `.ciagent/atelier/PERSONAS.md` for the v0.3 persona roster (5 active: lead-developer, tech-writer, domain-expert + 2 phase-specific platform-engineer, ml-engineer; 3 default personas deactivated).
|
||||
See `.ciagent/atelier/RESEARCH.md` "v0.3 Research" for the full prior-art survey and principle inventory rationale, and `.ciagent/atelier/PERSONAS.md` for the v0.3 persona roster (5 active: lead-developer, tech-writer, domain-expert + 2 phase-specific platform-engineer, ml-engineer; 3 default personas deactivated).
|
||||
|
||||
## v0.4 Domain Additions
|
||||
|
||||
Two new top-level domains and a language-derived extension expand the tree under the same hierarchy rules. All follow the v0.1/v0.2/v0.3 contract: docs-only markdown with illustrative code fences (no runtime/deployable artifacts per D-020). Total matrix grows from 170 → 190 domain principles across 17 → 19 domains.
|
||||
|
||||
- **`edge/`** — edge computing domain. Owns the proximity/location/constraint angle (D-061): compute near the user or data source, offline as a first-class state, constrained resources declared, sync conflicts bounded, partition-survivable telemetry. Boundary vs `performance/` (generic measurement/optimization), `observability/` (generic structured telemetry), and `concurrency/` (in-process primitives): edge owns only the proximity/location/disconnection concerns that arise at the network edge — performance's locality is algorithmic, edge's locality is geographic (C4). 10 P-rules, broad derivation (7 C-rules: C1, C3, C4, C5, C6, C7, C8). Depends on `core/`. Cross-links are one-directional outward (D-026 extended): `performance/frontend` (P6 geographic cache invalidation vs generic caching), `performance/P4` (P3 constrained-device vs generic resource bounds), `observability/metrics` + `observability/logging` (P10 local-first telemetry), `concurrency/patterns` (P5 offline write-queue analog), `security/secrets` (P9 device identity), `security/input-validation` (P6 cache poisoning), `data/migrations` (P4 schema migration under sync). Edge↔messaging intra-v0.4 cross-links resolve bidirectionally in P5 (ATELIER-114, IDEATE-40): `edge/iot.md` → `messaging/queues` (MQTT QoS parallels to delivery semantics). Derived docs: `cdn.md`, `offline-first.md`, `iot.md`, `sync.md`.
|
||||
|
||||
- **`messaging/`** — cross-process messaging domain. Owns the cross-process/network-failure-model angle (D-062): messages as contracts, explicit delivery semantics, idempotent consumers, bounded backpressure, observable lag. Boundary vs `concurrency/` (in-process/crash-failure-model): messaging's queue fails by network partition/broker restart/consumer crash-and-retry; concurrency's queue fails by OOM or thread crash. The discriminator is the failure model (D-062). 10 P-rules, broad derivation (7 C-rules: C1, C2, C4, C5, C6, C7, C8). Depends on `core/`. Cross-links are one-directional outward (D-026 extended): `concurrency/patterns` (P6 broker-backed bounded queue vs in-process Pattern 5; P3 cross-process idempotent retry vs in-process retry), `observability/metrics` + `observability/tracing` (P10 broker signals), `data/schema-design` (P1/P9 message schema evolution), `errors/patterns` (P5 errors as data for poison messages), `edge/iot` (P4 — bidirectional resolution of the MQTT QoS cross-link per IDEATE-40). Derived docs: `queues.md`, `pubsub.md`, `streams.md`, `delivery-semantics.md`.
|
||||
|
||||
### v0.4 Language-Derived Extension
|
||||
|
||||
The `languages/` tree extends from a first-principles-only stub (v0.1) to a full derived-doc tree (v0.4). Each of the 4 v0.1 stubs (`typescript.md`, `python.md`, `go.md`, `rust.md`) is extended with a pointer to its 4 derived docs (16 total). Language-derived docs **apply, not derive** domain principles (D-063): they introduce **no new P-rules** and **no new matrix rows** (D-066). Every section header names the traced core C-rule(s) AND the traced domain P-rule(s) inline (e.g., `## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)`). The first-principles stubs retain their existing section structure — **no P1–P10 numbering** is added to languages (D-063); languages are an application lens, not a principle source, preserving the `core > domains > languages` ordering ("conflicts go UP, never sideways").
|
||||
|
||||
The `matrix/principles-matrix.md` gains **no language rows** — it stays at 190 P-rules / 19 domains (D-066). Language-derived-doc traceability is captured in a new `## Languages Coverage` sub-table in `matrix/domain-coverage.md` (exactly 16 rows, one per derived doc) — the verifiable aggregate; inline section-headers are the per-doc detail. Every derived doc traces to ≥1 domain P-rule (D-066, no orphans).
|
||||
|
||||
Derived docs by language:
|
||||
- **TypeScript** — `ts-types.md`, `ts-tooling.md`, `ts-async.md`, `ts-testing.md`
|
||||
- **Python** — `py-types.md`, `py-tooling.md`, `py-async.md`, `py-testing.md`
|
||||
- **Go** — `go-types.md`, `go-tooling.md`, `go-concurrency.md`, `go-testing.md`
|
||||
- **Rust** — `rs-ownership.md`, `rs-tooling.md`, `rs-async.md`, `rs-testing.md`
|
||||
|
||||
The manifest (`MANIFEST.md`) is extended with a new `## Languages` section (D-065) listing the 4 v0.1 stubs (closing pre-existing drift parallel to the v0.2 ESC-002 / ATELIER-91 precedent) and the 16 new v0.4 derived docs. The `review/` docs (agent-checklist, peer-review-checklist, anti-patterns) gain edge, messaging, and language-derived trigger sections. Examples (P5) are illustrative markdown with fenced code only — no standalone `.ts`/`.py`/`.go`/`.rs` runtime artifacts (D-020 docs-only contract, extended in v0.4 to language-derived-doc artifact types).
|
||||
|
||||
See `.ciagent/atelier/RESEARCH-v0.4.md` for the full prior-art survey (Akamai/Cloudflare, Kafka/RabbitMQ/NATS, Jepsen, Reactive Streams, CloudEvents, CRDTs, PouchDB, MQTT/CoAP) and `.ciagent/atelier/PERSONAS.md` for the v0.4 persona roster (5 active: lead-developer, tech-writer, domain-expert + 2 phase-specific edge-engineer, languages-engineer).
|
||||
@@ -130,10 +130,10 @@ NFR milestone: no separate minor tag. The final patch (v0.2.6) IS the v0.3 deliv
|
||||
|
||||
| Phase | Name | Type | Status | Key Deliverables |
|
||||
|-------|------|------|--------|------------------|
|
||||
| 0 | Pre-Execution | docs | active | Spec, clarify, research, ideate, plan, PERSONAS.md (adds edge-engineer + languages-engineer phase-specific personas) |
|
||||
| 1 | Edge Domain | docs | pending | domains/edge/{first-principles, cdn, offline-first, iot, sync}.md |
|
||||
| 2 | Messaging Domain | docs | pending | domains/messaging/{first-principles, queues, pubsub, streams, delivery-semantics}.md |
|
||||
| 3 | Language-Derived Extension | docs | pending | languages/ × 4 → first-principles + 4 derived docs each (16 derived docs) |
|
||||
| 0 | Pre-Execution | docs | complete | Spec, clarify, research, ideate, plan, PERSONAS.md (adds edge-engineer + languages-engineer phase-specific personas) — shipped v0.3.0 |
|
||||
| 1 | Edge Domain | docs | complete | domains/edge/{first-principles, cdn, offline-first, iot, sync}.md — shipped v0.3.1 |
|
||||
| 2 | Messaging Domain | docs | complete | domains/messaging/{first-principles, queues, pubsub, streams, delivery-semantics}.md — shipped v0.3.2 |
|
||||
| 3 | Language-Derived Extension | docs | complete | languages/ × 4 → first-principles + 4 derived docs each (16 derived docs) — shipped v0.3.3 |
|
||||
| 4 | Matrix + Review Integration | docs | pending | matrix/principles-matrix.md (+20 mappings, 170→190), matrix/domain-coverage.md (+ languages/ sub-table), review/{agent-checklist, peer-review-checklist, anti-patterns}.md, MANIFEST.md (languages/ section) |
|
||||
| 5 | Examples + Cross-Links | docs | pending | examples/good + examples/bad for 2 domains, cross-links to devops/observability/data/concurrency/kubernetes/infrastructure-as-code + language→domain links |
|
||||
| 6 | Final Review + Ship | docs | pending | Review passed, audit clean, milestone merged to main, tag v0.3.6 |
|
||||
|
||||
+20
-5
@@ -40,6 +40,21 @@
|
||||
| AI / ML | ✓ | data-versioning, model-evaluation, serving, monitoring-drift |
|
||||
| i18n | ✓ | locale-resources, formatting, rtl-bidi, testing-i18n |
|
||||
| Compliance | ✓ | audit-logs, data-retention, policy-as-code, evidence |
|
||||
| Edge | ✓ | cdn, offline-first, iot, sync |
|
||||
| Messaging | ✓ | queues, pubsub, streams, delivery-semantics |
|
||||
|
||||
## Languages
|
||||
|
||||
> Language docs apply (not derive) domain principles. They introduce
|
||||
> no new P-rules and no new matrix rows (D-063, D-066). First-
|
||||
> principles stubs from v0.1; derived docs added in v0.4.
|
||||
|
||||
| Language | First Principles | Derived Documents |
|
||||
|----------|------------------|-------------------|
|
||||
| TypeScript | ✓ (v0.1 stub) | ts-types, ts-tooling, ts-async, ts-testing |
|
||||
| Python | ✓ (v0.1 stub) | py-types, py-tooling, py-async, py-testing |
|
||||
| Go | ✓ (v0.1 stub) | go-types, go-tooling, go-concurrency, go-testing |
|
||||
| Rust | ✓ (v0.1 stub) | rs-ownership, rs-tooling, rs-async, rs-testing |
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -71,8 +86,8 @@
|
||||
|
||||
| Document | Purpose |
|
||||
|-----------------------------------|----------------------------------|
|
||||
| `matrix/principles-matrix.md` | Maps domain → core principles (17 domains, 170 P-rules post-v0.3) |
|
||||
| `matrix/domain-coverage.md` | Maps core → domains; per-domain coverage (incl. v0.3 Core Principle Coverage) |
|
||||
| `review/agent-checklist.md` | Pre-completion agent checklist (incl. IaC + k8s + gitops + ai-ml + i18n + compliance triggers) |
|
||||
| `review/peer-review-checklist.md` | Human peer-review checklist (incl. IaC + k8s + gitops + ai-ml + i18n + compliance sections) |
|
||||
| `review/anti-patterns.md` | Catalog of violations (incl. IaC + k8s + gitops + ai-ml + i18n + compliance + v0.3 chaos anti-patterns) |
|
||||
| `matrix/principles-matrix.md` | Maps domain → core principles (19 domains, 190 P-rules post-v0.4) |
|
||||
| `matrix/domain-coverage.md` | Maps core → domains; per-domain coverage (incl. v0.4 Core Principle Coverage + Languages Coverage sub-table) |
|
||||
| `review/agent-checklist.md` | Pre-completion agent checklist (incl. edge + messaging + language-derived triggers) |
|
||||
| `review/peer-review-checklist.md` | Human peer-review checklist (incl. edge + messaging + language-derived sections) |
|
||||
| `review/anti-patterns.md` | Catalog of violations (incl. edge + messaging + language-derived + v0.4 chaos anti-patterns + .ts/.py/.go/.rs artifact types) |
|
||||
@@ -0,0 +1,227 @@
|
||||
# CDN — Derived Rules
|
||||
|
||||
> Derives from `domains/edge/first-principles.md`. Applies P1
|
||||
> (Proximity is the Design Driver) and P6 (Cache Invalidation is
|
||||
> Explicit) primarily, with P5 (idempotent cache fill), P8
|
||||
> (geographic distribution), and P9 (identity at the edge). For the
|
||||
> edge-cache-vs-origin decision, see the decision matrix below.
|
||||
> Cross-links `domains/performance/frontend` for generic caching,
|
||||
> `domains/security/input-validation` for cache poisoning, and
|
||||
> `domains/observability/metrics` for cache-hit ratio.
|
||||
|
||||
## What a CDN Is (P1 Proximity is the Design Driver)
|
||||
|
||||
- A content delivery network is a fleet of PoPs (points of presence)
|
||||
placed near users. The PoP serves cached content; the origin is
|
||||
the authoritative source. The CDN's whole purpose is P1: compute
|
||||
(the cache) is placed near the user so the round trip to the origin
|
||||
does not bound latency. Latency is a correctness constraint at the
|
||||
edge (C1), not a performance preference.
|
||||
- The CDN is the canonical edge-cache architecture (Akamai,
|
||||
Cloudflare, Fastly): PoPs near users, origin shielding, cache-key
|
||||
normalization, purge APIs. Atelier derives the
|
||||
placement/invalidation principles, not the vendor config.
|
||||
- The boundary with `domains/performance/frontend` is per D-061:
|
||||
performance owns *generic* caching and optimization (cache what is
|
||||
expensive, stable, read often — `performance/P5 Caching with
|
||||
Intent`); edge owns the *geographic, partition-aware* placement and
|
||||
invalidation angle. A CDN is an edge concern because its defining
|
||||
trait is geographic distribution (P8) and partition-aware
|
||||
invalidation (P6), not measurement.
|
||||
|
||||
## Cache Key Design (P6 Cache Invalidation is Explicit)
|
||||
|
||||
- The cache key is the contract between the URL and the cached
|
||||
representation. A key that varies on the wrong dimensions serves
|
||||
the wrong content; a key that varies on too many dimensions
|
||||
collapses the hit ratio. Key design *is* the invalidation
|
||||
surface: a key that includes a content hash or version segment
|
||||
makes invalidation explicit; a key that ignores `Vary` headers
|
||||
serves stale variants.
|
||||
- Normalize the key: lower-case the host, strip default ports,
|
||||
sort query parameters, ignore tracking parameters. A
|
||||
non-normalized key is a cache-poisoning vector (see
|
||||
`domains/security/input-validation`) and a hit-ratio destroyer
|
||||
(see `domains/observability/metrics`).
|
||||
- A cache with no explicit key strategy is a TTL-less cache under
|
||||
partition (P6 violation): staleness is silent and unbounded.
|
||||
|
||||
```http
|
||||
# Cache key derivation: vary on what changes content, ignore what
|
||||
# does not. The key is the tuple (host, normalized-path, sorted-
|
||||
# query, Vary-headers); the cache entry is the representation + TTL.
|
||||
Cache-Key: example.com /api/v1/products?sort=price®ion=us Vary:Accept-Encoding
|
||||
Cache-Control: public, max-age=60, s-maxage=600, stale-while-revalidate=300
|
||||
Vary: Accept-Encoding
|
||||
```
|
||||
|
||||
- `max-age` bounds the browser cache; `s-maxage` bounds the CDN
|
||||
PoP; `stale-while-revalidate` allows serving stale while
|
||||
refetching. Each is an explicit invalidation strategy (P6).
|
||||
|
||||
## TTL vs Explicit Invalidation (P6, C3 Simplicity)
|
||||
|
||||
- **TTL-based invalidation** (`max-age`, `s-maxage`): the cache entry
|
||||
expires after a duration. Simple, no origin contact required to
|
||||
invalidate, but bounded staleness is the contract — the entry may
|
||||
be stale up to TTL. Fits content where eventual consistency is
|
||||
acceptable (asset fingerprints, lists, derived images).
|
||||
- **Explicit invalidation** (purge, surrogate keys): the operator
|
||||
signals the cache to drop entries. Tighter staleness bounds, but
|
||||
requires the origin or operator to know which entries to purge.
|
||||
Fits content where staleness is a correctness defect (price
|
||||
updates, availability, breaking news).
|
||||
- A TTL-less cache with no explicit invalidation is a P6 violation:
|
||||
stale-forever under partition. Every cache must have one or the
|
||||
other (or both), and the choice is documented per content type.
|
||||
|
||||
## Cache-Hit / Miss / Origin-Fetch (P5, P6)
|
||||
|
||||
- **Hit**: the PoP serves from cache. Latency is PoP-local (P1).
|
||||
- **Miss**: the PoP has no entry; it fetches from the origin (or an
|
||||
origin-shield PoP). The fetch must be idempotent (P5) — a retried
|
||||
miss must not corrupt the cache or double-write side effects.
|
||||
- **Revalidate**: the PoP holds a stale entry and asks the origin
|
||||
(`If-None-Match`, `If-Modified-Since`); a 304 refreshes the TTL
|
||||
without re-fetching the body. Revalidation is the bandwidth-economical
|
||||
middle ground (C8).
|
||||
|
||||
```http
|
||||
# Conditional revalidation — the PoP asks the origin "is this still
|
||||
# current?" The 304 response refreshes the TTL without a body.
|
||||
GET /api/v1/products HTTP/1.1
|
||||
Host: example.com
|
||||
If-None-Match: "etag-7a3f"
|
||||
|
||||
HTTP/1.1 304 Not Modified
|
||||
ETag: "etag-7a3f"
|
||||
Cache-Control: s-maxage=600
|
||||
```
|
||||
|
||||
- A cache-hit ratio that is not measured is a gate on noise — see
|
||||
`domains/observability/metrics` for the SLI/SLO discipline that
|
||||
makes the hit ratio a meaningful signal. A CDN with no hit-ratio
|
||||
metric is operating blind (P10 analog).
|
||||
|
||||
## Origin Shielding (P1, P8, C8 Economy)
|
||||
|
||||
- Origin shielding routes all origin fetches through a single
|
||||
shield PoP (or shield region). The shield absorbs the
|
||||
thundering-herd: 10 000 PoPs missing the same URL fetch the origin
|
||||
once, not 10 000 times. This is C8 Economy (origin bandwidth is
|
||||
bounded) and P1 (the shield is itself a proximity layer for the
|
||||
origin).
|
||||
- Shielding is a geographic decision (P8): the shield sits in a
|
||||
region close to the origin, not close to the user. The shield is
|
||||
the inner ring of the CDN; the user-facing PoPs are the outer ring.
|
||||
- A CDN without origin shielding under a stampede will overload the
|
||||
origin; a shield that is itself partitioned from the origin must
|
||||
degrade gracefully (P7) — serve stale per `stale-while-revalidate`
|
||||
rather than 500.
|
||||
|
||||
## Purge Strategies (P6, C3 Simplicity)
|
||||
|
||||
| Strategy | Granularity | Latency to Invalidate | Cost | Best for |
|
||||
|----------|-------------|-----------------------|------|----------|
|
||||
| URL purge | One URL | Seconds | Low (one entry) | Surgical fixes, single-page corrections |
|
||||
| Soft purge | One URL (mark stale, serve while refetch) | Seconds | Low | High-traffic URLs where a hard purge causes a stampede |
|
||||
| Surrogate-key purge | A tag set (e.g., `product:123`, `category:shoes`) | Seconds | Medium (key indexing) | Related-content invalidation (a product update purges all its category pages) |
|
||||
| Wildcard purge | A path prefix or pattern | Seconds to minutes | High (scan) | Site-wide template changes |
|
||||
| All-cache purge | Everything | Seconds | Very high (origin stampede) | Disaster recovery only; never the steady-state invalidation path |
|
||||
|
||||
- Surrogate-key purge (Fastly, Akamai) is the highest-value
|
||||
strategy: tag cache entries with content keys, then purge by tag.
|
||||
This is explicit invalidation at scale (P6) without the origin
|
||||
stampede of an all-cache purge.
|
||||
- An all-cache purge as the steady-state invalidation path is a P6
|
||||
violation dressed as a feature — it pushes the origin load back to
|
||||
100% miss, defeating the CDN's purpose (P1).
|
||||
|
||||
## Cache Poisoning Prevention (P9, cross-link security/input-validation)
|
||||
|
||||
- A cache poisoned by a crafted request (a URL with a malicious
|
||||
header that gets cached and served to others) is a correctness
|
||||
defect (C1) and a security breach (P9 — the edge node is
|
||||
exploited). Prevent poisoning by:
|
||||
- Normalizing the cache key (strip untrusted query parameters,
|
||||
ignore unknown headers, lower-case the host).
|
||||
- Validating `Vary` against an allow-list; never `Vary: *` on a
|
||||
shared cache (poisonable via header injection).
|
||||
- Treating uncacheable responses (`Set-Cookie`,
|
||||
`Cache-Control: private`) as never-stored.
|
||||
- See `domains/security/input-validation` for the general
|
||||
input-validation discipline the cache key must follow. The cache
|
||||
key is a validation surface; a non-validated key is an attack
|
||||
surface.
|
||||
|
||||
## Multi-CDN Routing (P8 Geographic Distribution)
|
||||
|
||||
- A multi-CDN strategy routes each request to the best PoP across
|
||||
providers (Akamai + Cloudflare + Fastly). Routing is
|
||||
location-aware (P8): latency, cost, and availability vary by
|
||||
region and provider. The DNS layer (or a client-side router)
|
||||
selects the CDN per request.
|
||||
- Multi-CDN is a P8 decision, not a vendor-management decision:
|
||||
geographic distribution is the first-class constraint. A
|
||||
single-CDN deployment routes everything to one provider's PoPs;
|
||||
a multi-CDN deployment routes by region, latency, and cost.
|
||||
- Invalidation across multiple CDNs is harder (P6): each provider
|
||||
has its own purge API and surrogate-key scheme. A multi-CDN purge
|
||||
must fan out to all providers; a purge that reaches only one CDN
|
||||
leaves the others stale. Track purge completion per provider —
|
||||
see `domains/observability/metrics` for the per-CDN hit-ratio and
|
||||
purge-latency signals.
|
||||
|
||||
```http
|
||||
# A CDN config example: cache-control headers + a purge rule.
|
||||
# Origin response: declare the cache contract (P6).
|
||||
HTTP/1.1 200 OK
|
||||
Cache-Control: public, max-age=60, s-maxage=600, stale-while-revalidate=300
|
||||
Surrogate-Key: product:123 category:shoes
|
||||
ETag: "etag-7a3f"
|
||||
Vary: Accept-Encoding
|
||||
|
||||
# Purge rule (Fastly-style surrogate-key): when product 123
|
||||
# updates, purge every cache entry tagged product:123 OR
|
||||
# category:shoes. Explicit, bounded, no origin stampede (P6).
|
||||
POST /service/svc1/purge
|
||||
Surrogate-Key: product:123 category:shoes
|
||||
# Returns: {"status": "ok", "id": "purge-abc"} — poll the purge
|
||||
# status to confirm completion across all PoPs (P8, P10).
|
||||
```
|
||||
|
||||
## Edge-Cache vs Origin — Decision Matrix (D-069)
|
||||
|
||||
| Strategy | When | Latency | Origin Load | Correctness Risk |
|
||||
|----------|------|---------|-------------|------------------|
|
||||
| Serve from PoP (cache hit) | The PoP holds a fresh entry (within TTL or revalidated) | Lowest (PoP-local, P1) | None | Low — bounded by TTL staleness (P6) |
|
||||
| Serve stale while revalidate | The PoP holds a stale entry and `stale-while-revalidate` is set | Low (stale served immediately, refetch in background) | Background refetch (1 per entry) | Medium — stale served up to the revalidate window; acceptable for eventually-consistent content |
|
||||
| Fetch fresh from origin (miss) | The PoP has no entry, or the content is non-cacheable | High (origin round trip) | Full fetch per miss | Low — fresh by construction; the miss is the correctness floor |
|
||||
| Origin-shield fetch | Multiple PoPs miss the same URL; the shield collapses the herd | Medium (PoP → shield → origin) | Bounded to one origin fetch per shield (C8) | Low — shield is the inner ring; staleness bounded by shield TTL |
|
||||
| Purge and serve fresh | Explicit invalidation received (surrogate-key or URL purge) | Medium (purge propagates, then fresh fetch) | Full fetch post-purge | Lowest — explicit invalidation is the tightest staleness bound (P6) |
|
||||
| Serve from origin directly (bypass cache) | Content is non-cacheable (personalized, real-time) | Highest (every request hits origin) | Full fetch per request | Lowest for correctness, highest for origin load — use sparingly |
|
||||
|
||||
- The default is **serve from PoP** when fresh, **fetch fresh from
|
||||
origin** on miss with **origin-shield** to bound origin load, and
|
||||
**purge and serve fresh** when explicit invalidation is required.
|
||||
Bypass-the-cache is for non-cacheable content only — bypassing for
|
||||
cacheable content is a P1 violation (you have defeated the CDN).
|
||||
- The correctness risk column is bounded by the invalidation
|
||||
strategy (P6): every row except "bypass" carries staleness risk
|
||||
that is bounded by TTL or explicit purge. A row with no
|
||||
invalidation strategy is a P6 violation.
|
||||
|
||||
## What Violates CDN Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| TTL-less edge cache under partition (stale-forever, no explicit invalidation) | P6 Cache Invalidation is Explicit |
|
||||
| Cache key that varies on untrusted query parameters (poisonable) | P6, P9 (`domains/security/input-validation`) |
|
||||
| All-cache purge as the steady-state invalidation path (origin stampede) | P6, C8 Economy |
|
||||
| Non-normalized cache key (case-sensitive host, unsorted query) | P6, `domains/security/input-validation` |
|
||||
| Bypass-the-cache for cacheable content | P1 Proximity is the Design Driver (defeats the CDN) |
|
||||
| Multi-CDN with no per-CDN purge completion tracking | P8, P10 (stale cache invisible to the operator) |
|
||||
| Origin fetch that is not idempotent under retry | P5 Edge Operations are Idempotent |
|
||||
| Cache-hit ratio not measured | P10, `domains/observability/metrics` |
|
||||
| Single-CDN deployed where geographic distribution requires multi-CDN | P8 Geographic Distribution |
|
||||
| Shield PoP that 500s instead of serving stale under partition | P7 Partial Degradation is Engineered |
|
||||
@@ -0,0 +1,218 @@
|
||||
# Edge — First Principles
|
||||
|
||||
## 1. The Principles
|
||||
|
||||
### P1. Proximity is the Design Driver
|
||||
Compute, storage, and data are placed near the user or the data
|
||||
source. At the edge, latency is a correctness constraint (C1), not a
|
||||
performance preference — a late answer is a wrong answer when the
|
||||
round trip to a central region exceeds the user's or device's
|
||||
tolerance. This is the geographic expression of `C4 Locality`:
|
||||
performance's locality is algorithmic (data near compute); edge's
|
||||
locality is geographic (compute near user/data source). Placement is
|
||||
a design decision, not an accident of deployment, and it is
|
||||
constrained by `P8 Geographic Distribution`. The proximity angle is
|
||||
the distinguishing trait of the edge domain per D-061: this is what
|
||||
separates edge from `domains/performance/` (which owns *generic*
|
||||
measurement and optimization, not placement).
|
||||
|
||||
### P2. Offline is a First-Class State
|
||||
The system continues to operate when disconnected from the center.
|
||||
Partition is the norm, not the exception; reconciliation happens on
|
||||
reconnect, never assumed to be instant. An app that crashes on
|
||||
disconnect has no offline state and is unengineered. Offline
|
||||
operation derives from `C5 Reversibility` — the disconnected state
|
||||
is reversible back to consistency via reconciliation — and `C1
|
||||
Correctness`, because correctness under partition is the contract,
|
||||
not eventual correctness as a hedge. This is the foundation for
|
||||
`domains/edge/offline-first.md` and the precondition for the
|
||||
bounded-conflict discipline of `P4`.
|
||||
|
||||
### P3. Resources are Constrained and Declared
|
||||
Edge nodes — IoT sensors, gateways, point-of-sale devices, CDN PoP
|
||||
caches, 5G MEC nodes — have bounded CPU, memory, power, and
|
||||
bandwidth. Constraints are declared per node class, never assumed
|
||||
infinite. An undeclared budget is a defect: unbounded growth is a
|
||||
bug, and a constrained device with no budget will OOM or exhaust
|
||||
power. This derives from `C8 Economy` (use no more than the task
|
||||
requires) and `C1 Correctness` (a node that exceeds its bounds has
|
||||
failed). This is the edge-specific angle on `domains/performance/P4
|
||||
Resource Bounds` — performance owns the generic principle; edge owns
|
||||
the constrained-device reality. See `domains/edge/iot.md` for the
|
||||
per-device-class application.
|
||||
|
||||
### P4. Sync Conflicts are Bounded, Not Infinite
|
||||
Divergent state across partitioned nodes converges. Oscillation and
|
||||
infinite sync loops are correctness failures, not eventual
|
||||
consistency. A merge that never terminates is a livelock; a CRDT
|
||||
without merge semantics or an LWW without a monotonic clock can
|
||||
oscillate forever. This derives from `C1 Correctness` (convergence is
|
||||
a correctness contract) and `C5 Reversibility` (divergent state is
|
||||
reversible back to convergence). The bound may be eventual (CRDTs) or
|
||||
arbitrated (LWW with vector clocks), but it must exist. This is the
|
||||
foundation for `domains/edge/sync.md` and the rule the
|
||||
`edge-sync-loop` chaos anti-pattern breaches.
|
||||
|
||||
### P5. Edge Operations are Idempotent
|
||||
Sync, cache fill, and device commands are retried by nature — the
|
||||
network is partition-prone and the operation will be re-attempted.
|
||||
Idempotency keys (or deterministic operations) make retries safe. A
|
||||
non-idempotent edge write retried with side effects doubles the
|
||||
effect; a non-idempotent cache fill under retry corrupts the cache.
|
||||
This derives from `C1 Correctness`: correctness under retry is the
|
||||
contract, not a nice-to-have. This parallels `domains/messaging/P3
|
||||
Consumers are Idempotent` (cross-process delivery) and is the edge's
|
||||
device-and-cache-flavored analog — see `domains/edge/iot.md` for
|
||||
device command idempotency and `domains/edge/cdn.md` for cache-fill
|
||||
idempotency.
|
||||
|
||||
### P6. Cache Invalidation is Explicit
|
||||
Edge caches carry a defined invalidation or TTL strategy. A
|
||||
stale-forever cache under partition is a silent correctness defect;
|
||||
a TTL-less cache with no explicit invalidation is a bug, not a
|
||||
feature. This derives from `C1 Correctness` (cached state must be
|
||||
correct) and `C3 Simplicity` (a defined invalidation strategy is
|
||||
simpler and clearer than ad-hoc staleness). This is distinct from
|
||||
`domains/performance/P5 Caching with Intent`, which owns *generic*
|
||||
caching and optimization; edge owns the *geographic,
|
||||
partition-aware* invalidation angle — when a PoP is partitioned from
|
||||
the origin, the invalidation strategy is the correctness mechanism.
|
||||
See `domains/edge/cdn.md` for purge strategies (URL vs soft vs
|
||||
surrogate-key) and the edge-cache-vs-origin decision matrix.
|
||||
|
||||
### P7. Partial Degradation is Engineered
|
||||
The system degrades gracefully when an edge node or link fails. A
|
||||
partial service is a designed state with a defined contract, not a
|
||||
crash. One node's failure must not collapse the whole fleet; the
|
||||
degraded mode is documented, observable, and recoverable. This
|
||||
derives from `C1 Correctness` (the degraded contract is a
|
||||
correctness bound) and `C5 Reversibility` (recovery from degradation
|
||||
is reversible by construction). A crash-on-node-failure system has
|
||||
no degradation contract — it has an all-or-nothing failure mode that
|
||||
violates the fleet assumption. See `domains/edge/iot.md` for
|
||||
device-drop degradation and `domains/edge/offline-first.md` for
|
||||
partition degradation.
|
||||
|
||||
### P8. Geographic Distribution is a First-Class Constraint
|
||||
The fleet is geo-distributed; routing, fan-out, and data placement
|
||||
are location-aware decisions, not accidents of deployment. The
|
||||
system is many nodes across many locations, not a single deployment.
|
||||
Data residency, regional latency, and PoP selection are engineered,
|
||||
not discovered in production. This derives from `C4 Locality` (the
|
||||
placement of data and compute is a locality decision) and `C6
|
||||
Composability` (the fleet composes from location-aware parts, each
|
||||
with its own contract). This is the structural companion to `P1
|
||||
Proximity`: P1 says *where* compute should be (near the user); P8
|
||||
says the *distribution* of compute across geographies is a
|
||||
first-class constraint. See `domains/edge/cdn.md` for multi-CDN
|
||||
routing.
|
||||
|
||||
### P9. Identity is Constrained at the Edge
|
||||
Edge devices and nodes hold scoped, minimal credentials. No edge
|
||||
node is a cluster-admin-equivalent; device identity is per-device,
|
||||
not shared. One compromise must not equal a fleet compromise. This
|
||||
derives from `C1 Correctness` (security is a subset of correctness —
|
||||
an exploitable edge node does not do what it was supposed to do) and
|
||||
`C8 Economy` of trust (the credential scope is minimal for the task).
|
||||
A shared edge-device credential is the edge analog of a
|
||||
cluster-admin GitOps robot — blast radius is unbounded. See
|
||||
`domains/security/secrets.md` for the general secret-hygiene
|
||||
principles and `domains/edge/iot.md` for device provisioning.
|
||||
|
||||
### P10. Edge Observability Survives Partition
|
||||
Telemetry is local-first: buffered on the node and forwarded on
|
||||
reconnect. Partition does not blind the operator. A fire-and-forget
|
||||
telemetry pipeline loses data when the link drops; a local-first
|
||||
buffer survives. This derives from `C7 Observability` (the fleet's
|
||||
behavior is visible to the operator) and `C5 Reversibility` (the
|
||||
buffered telemetry is reversible back to visibility on reconnect).
|
||||
This is distinct from `domains/observability/P1 Structured by
|
||||
Default`, which owns *generic* structured telemetry; edge owns the
|
||||
*partition-survivable, local-first* angle. See
|
||||
`domains/observability/metrics.md` and
|
||||
`domains/observability/logging.md` for the generic structured-
|
||||
telemetry foundations edge builds on.
|
||||
|
||||
## 2. Core Principle Trace
|
||||
|
||||
Each edge P-rule derives from one or more core C-rules (C1–C8). The
|
||||
matrix extension lands in P4 of the v0.4 plan; the traces below are
|
||||
authoritative. Edge is a broad-derivation domain touching 7 of 8
|
||||
core principles (C1, C3, C4, C5, C6, C7, C8); C2 (Clarity) is not a
|
||||
primary derivation — edge clarity is indirect (a cache with explicit
|
||||
invalidation is clearer than one without, but the primary trace is
|
||||
C1/C3).
|
||||
|
||||
| P-rule | Core | Why |
|
||||
|--------|------|-----|
|
||||
| P1 Proximity is the Design Driver | C4, C1 | Locality of compute near user/data; correctness via latency |
|
||||
| P2 Offline is a First-Class State | C1, C5 | Correctness under partition; reversibility of reconciliation |
|
||||
| P3 Resources are Constrained and Declared | C8, C1 | Economy of constrained nodes; correctness of declared bounds |
|
||||
| P4 Sync Conflicts are Bounded, Not Infinite | C1, C5 | Correctness of convergence; reversibility of divergent state |
|
||||
| P5 Edge Operations are Idempotent | C1 | Correctness under retry |
|
||||
| P6 Cache Invalidation is Explicit | C1, C3 | Correctness of cached state; simplicity of defined invalidation |
|
||||
| P7 Partial Degradation is Engineered | C1, C5 | Correctness of degraded modes; reversibility of recovery |
|
||||
| P8 Geographic Distribution is a First-Class Constraint | C4, C6 | Locality of placement; composability of the fleet |
|
||||
| P9 Identity is Constrained at the Edge | C1, C8 | Correctness via security; economy of trust |
|
||||
| P10 Edge Observability Survives Partition | C7, C5 | Observability of the fleet; reversibility of buffered telemetry |
|
||||
|
||||
## 3. What Violates These Principles
|
||||
|
||||
| Violation | Principle Breached |
|
||||
|-----------|-------------------|
|
||||
| Central-region-only deployment for a latency-bound workload | P1 Proximity is the Design Driver |
|
||||
| App that crashes on disconnect (no offline state) | P2 Offline is a First-Class State |
|
||||
| Undeclared edge-node resource budget (assumes infinite CPU/memory) | P3 Resources are Constrained and Declared |
|
||||
| Sync loop that oscillates forever (CRDT without merge-semantics, LWW without monotonic clock) | P4 Sync Conflicts are Bounded, Not Infinite |
|
||||
| Non-idempotent edge write (cache-fill or device command retried with side effects) | P5 Edge Operations are Idempotent |
|
||||
| TTL-less edge cache under partition (stale-forever, no explicit invalidation) | P6 Cache Invalidation is Explicit |
|
||||
| Crash-on-node-failure (no partial-degradation contract) | P7 Partial Degradation is Engineered |
|
||||
| Random geographic placement (no location-aware routing) | P8 Geographic Distribution is a First-Class Constraint |
|
||||
| Shared edge-device credential (one key for the whole fleet) | P9 Identity is Constrained at the Edge |
|
||||
| Fire-and-forget telemetry (no on-node buffer; data lost on partition) | P10 Edge Observability Survives Partition |
|
||||
| Blocking call on a constrained IoT device with no timeout | P3, P5 (blocks the node; retry unsafe without idempotency) |
|
||||
| Multi-CDN routing with no PoP-selection logic (latency uncontrolled) | P8, P1 (placement not a design decision) |
|
||||
|
||||
## 4. Relationship to Other Domains
|
||||
|
||||
Edge computing is the engineering discipline of placing compute,
|
||||
storage, and data **near the source of generation or consumption**
|
||||
rather than in a centralized cloud. The distinguishing constraints are
|
||||
latency-bound operation, resource-constrained nodes,
|
||||
geo-distribution as a fleet, and partition-prone operation. Edge
|
||||
overlaps three existing domains by *subject* but not by *angle*: per
|
||||
D-061, edge owns the proximity/location/constraint/disconnection
|
||||
concerns that only arise at the network edge. The C4 Locality
|
||||
emphasis is the discriminator: performance's locality is algorithmic
|
||||
(data near compute); edge's locality is geographic (compute near
|
||||
user/data source). Cross-links are one-directional outward (per
|
||||
D-026 extended); no back-link edits to v0.1/v0.2/v0.3 content.
|
||||
|
||||
- `domains/performance/frontend` ← P6 (edge owns geographic,
|
||||
partition-aware cache invalidation; performance owns *generic*
|
||||
caching and measurement — D-061 boundary)
|
||||
- `domains/performance/P4 Resource Bounds` ← P3 (edge owns
|
||||
constrained-device reality; performance owns the generic
|
||||
unbounded-growth-is-a-bug principle)
|
||||
- `domains/observability/metrics` ← P10 (cache-hit ratio, edge
|
||||
telemetry aggregation; edge owns the local-first angle, observability
|
||||
owns generic structured metrics)
|
||||
- `domains/observability/logging` ← P10 (local-first logging buffered
|
||||
on-node and forwarded on reconnect)
|
||||
- `domains/concurrency/patterns` ← P5 (the offline write-queue is
|
||||
the cross-partition analog of the in-process bounded buffer —
|
||||
concurrency owns in-process; edge owns partition-survivable)
|
||||
- `domains/security/secrets` ← P9 (device credentials are scoped,
|
||||
per-device, never shared — edge owns the constrained-identity
|
||||
angle; security owns the general secret hygiene)
|
||||
- `domains/security/input-validation` ← P6 (cache poisoning
|
||||
prevention — edge cache keys are a validation surface)
|
||||
- `domains/data/migrations` ← P4 (schema migration under sync must
|
||||
reconcile across partitioned nodes; data owns the generic migration
|
||||
discipline, edge owns the partitioned-reconcile angle)
|
||||
|
||||
> Note: cross-links to `domains/messaging/` (e.g., MQTT QoS parallels
|
||||
> for delivery semantics) are intentionally omitted here — the
|
||||
> messaging domain is authored in P2. The intra-v0.4 edge↔messaging
|
||||
> links are added in P5 (ATELIER-114 per IDEATE-40) once both
|
||||
> domains exist; the dangling link is acceptable per D-053.
|
||||
@@ -0,0 +1,258 @@
|
||||
# IoT — Derived Rules
|
||||
|
||||
> Derives from `domains/edge/first-principles.md`. Applies P3
|
||||
> (Resources are Constrained and Declared) primarily, with P5
|
||||
> (command idempotency), P7 (partial degradation when devices drop),
|
||||
> P9 (device identity and provisioning), and P10 (telemetry from
|
||||
> devices). Cross-links `domains/security/secrets` for device
|
||||
> credentials and `domains/messaging/queues` for the MQTT QoS
|
||||
> parallels to delivery semantics.
|
||||
|
||||
## What IoT at the Edge Is (P3 Resources are Constrained and Declared)
|
||||
|
||||
- IoT at the edge is the engineering discipline of operating
|
||||
constrained devices — sensors, actuators, gateways, microcontrollers
|
||||
— as first-class participants in a distributed system. The
|
||||
distinguishing constraint is per-device resource bounds (P3): a
|
||||
battery-powered sensor has kilobytes of RAM, a constrained
|
||||
protocol, and a multi-year sleep budget. These constraints are
|
||||
declared per device class, never assumed infinite.
|
||||
- The boundary is per D-061: edge owns the constrained-device
|
||||
reality; performance owns the generic unbounded-growth-is-a-bug
|
||||
principle (`performance/P4 Resource Bounds`); concurrency owns
|
||||
in-process primitives. IoT is an edge concern because its defining
|
||||
traits are constrained resources (P3), geographic distribution as
|
||||
a fleet (P8), partition-prone operation (P2), and device-scoped
|
||||
identity (P9) — concerns that only arise at the network edge.
|
||||
- See `domains/edge/offline-first.md` for the partition-survival
|
||||
discipline that constrained devices depend on, and
|
||||
`domains/edge/sync.md` for the reconciliation of device state
|
||||
across partitions.
|
||||
|
||||
## Device Resource Classes (P3, C8 Economy)
|
||||
|
||||
- A device resource class declares the bounds for a class of
|
||||
devices: CPU (MHz, cores), memory (KB/MB), power (battery mAh,
|
||||
duty-cycle budget), bandwidth (bytes/sec, latency budget), and
|
||||
storage (KB/MB). Every device in the fleet is assigned to a class;
|
||||
every operation is budgeted against its class.
|
||||
- An undeclared budget is a defect (P3 violation): a sensor that
|
||||
sends telemetry every second without a duty-cycle budget exhausts
|
||||
its battery in days, not years. The budget is the correctness
|
||||
bound (C1) and the economy bound (C8).
|
||||
- A device class implies a protocol choice: a class-0 device
|
||||
(constrained sensor, KB RAM) speaks CoAP; a class-1 device
|
||||
(gateway, MB RAM) speaks MQTT; a class-2 device (edge compute
|
||||
node, GB RAM) speaks HTTP. The protocol follows the constraint,
|
||||
not the reverse.
|
||||
|
||||
| Class | RAM | Power | Protocol | Typical role |
|
||||
|-------|-----|-------|----------|--------------|
|
||||
| 0 (constrained sensor) | < 10 KB | Battery, multi-year | CoAP, LoRaWAN | Telemetry only, no inbound commands |
|
||||
| 1 (actuator, gateway) | 10 KB – 1 MB | Battery or wired, weeks-months | MQTT, CoAP | Telemetry + commands, queue-and-forward |
|
||||
| 2 (edge compute) | > 1 MB | Wired, continuous | HTTP, MQTT | Local aggregation, gateway, edge inference |
|
||||
|
||||
## Constrained Protocols — MQTT and CoAP (P3, P5)
|
||||
|
||||
- **MQTT** is the canonical pub/sub protocol for constrained devices.
|
||||
It is lightweight (2-byte header), broker-backed, and provides QoS
|
||||
levels (0, 1, 2) that map to delivery semantics. MQTT is the
|
||||
cross-process analog of message-queue delivery — see
|
||||
`domains/messaging/queues` for the general queue/delivery-semantics
|
||||
discipline; the cross-link is one-directional outward (edge →
|
||||
messaging) per D-062 and D-026 extended.
|
||||
- **CoAP** is the REST analog for constrained devices: UDP-based,
|
||||
low-overhead, with confirmable (CON) and non-confirmable (NON)
|
||||
message types. CoAP fits class-0 devices where TCP is too heavy.
|
||||
- A blocking synchronous call on a constrained device with no
|
||||
timeout is the `blocking-call-on-constrained-device` chaos
|
||||
anti-pattern: it blocks the node, has no timeout (= hang), and
|
||||
retries are unsafe without idempotency (P3 + P5 breach). Every
|
||||
device operation must be async with a timeout, and every retried
|
||||
operation must be idempotent.
|
||||
|
||||
```json
|
||||
// MQTT publish/subscribe payload with QoS levels (P5 idempotency,
|
||||
// P3 constrained protocol).
|
||||
// QoS 0 — at-most-once: fire-and-forget, no ack. For telemetry
|
||||
// where a dropped sample is acceptable (P3 economy of the
|
||||
// constrained link).
|
||||
{
|
||||
"topic": "devices/sensor-7/temperature",
|
||||
"qos": 0,
|
||||
"payload": {
|
||||
"device": "sensor-7",
|
||||
"ts": 1700000000,
|
||||
"value": 21.4,
|
||||
"unit": "C"
|
||||
}
|
||||
}
|
||||
|
||||
// QoS 1 — at-least-once: acked, may duplicate. The consumer must
|
||||
// be idempotent (P5) — dedup by (device, ts) or an idempotency key.
|
||||
{
|
||||
"topic": "devices/actuator-3/command",
|
||||
"qos": 1,
|
||||
"payload": {
|
||||
"device": "actuator-3",
|
||||
"idempotencyKey": "cmd-1700000000-1",
|
||||
"command": "set-point",
|
||||
"value": 22.0
|
||||
}
|
||||
}
|
||||
|
||||
// QoS 2 — exactly-once: four-step handshake, no duplication. The
|
||||
// heaviest QoS; use only where duplicates are intolerable AND the
|
||||
// device has the budget for the handshake (class-1+ only, P3).
|
||||
{
|
||||
"topic": "devices/actuator-3/irreversible-command",
|
||||
"qos": 2,
|
||||
"payload": {
|
||||
"device": "actuator-3",
|
||||
"idempotencyKey": "cmd-1700000000-2",
|
||||
"command": "calibrate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- The QoS choice is a P5 (idempotency) and P3 (resource) decision:
|
||||
QoS 0 is cheapest (no ack) but lossy; QoS 1 requires consumer
|
||||
idempotency (P5); QoS 2 is exactly-once but costs a four-step
|
||||
handshake on a constrained link. The default for telemetry is QoS
|
||||
0 or 1; the default for commands is QoS 1 with an idempotency key
|
||||
(P5); QoS 2 is reserved for irreversible commands where the
|
||||
device budget permits.
|
||||
|
||||
## Device Identity and Provisioning (P9 Identity is Constrained at the Edge)
|
||||
|
||||
- Every device holds a per-device identity: a unique device ID and a
|
||||
scoped credential (X.509 certificate, API token, or rotating
|
||||
key). No shared fleet credential — one compromise must not equal
|
||||
a fleet compromise (P9). The credential scope is minimal: a
|
||||
device can publish to `devices/<its-id>/+` and subscribe to
|
||||
`devices/<its-id>/commands`, nothing else.
|
||||
- Provisioning is the act of assigning a device identity at
|
||||
enrollment time. The provisioning manifest declares the device,
|
||||
its class, its allowed topics, and its credential. The manifest is
|
||||
the P9 contract — a device operating outside its manifest scope is
|
||||
a violation.
|
||||
- A device that is provisioned with a shared fleet key (the
|
||||
`shared-edge-device-credential` anti-pattern) is a P9 violation:
|
||||
blast radius is unbounded. See `domains/security/secrets` for the
|
||||
general secret-hygiene discipline (per-identity credentials,
|
||||
rotation, minimal scope) that device provisioning builds on.
|
||||
|
||||
```yaml
|
||||
# Device provisioning manifest (P9 per-device identity + scoped
|
||||
# credentials). The manifest is the contract; the device operates
|
||||
# only within its declared scope.
|
||||
device:
|
||||
id: sensor-7
|
||||
class: 0 # P3 resource class
|
||||
model: temp-sensor-v2
|
||||
firmware: 1.4.2
|
||||
identity:
|
||||
cert: "sha256-of-device-cert"
|
||||
credentialScope:
|
||||
publish:
|
||||
- "devices/sensor-7/temperature"
|
||||
- "devices/sensor-7/status"
|
||||
subscribe:
|
||||
- "devices/sensor-7/commands"
|
||||
# No wildcard, no fleet-wide topics (P9).
|
||||
provisioning:
|
||||
enrolledAt: 2024-01-15T00:00:00Z
|
||||
rotatesEvery: 90d
|
||||
# Per-device credential; never shared (P9, domains/security/secrets).
|
||||
```
|
||||
|
||||
## Telemetry from Devices (P10 Edge Observability Survives Partition)
|
||||
|
||||
- Device telemetry is local-first (P10): the device buffers telemetry
|
||||
on-node and forwards on reconnect. A fire-and-forget telemetry
|
||||
pipeline loses data when the link drops; a buffered pipeline
|
||||
survives. The buffer is bounded by the device class (P3): a
|
||||
class-0 sensor buffers minutes of telemetry, not hours.
|
||||
- Telemetry is observable in aggregate: the operator sees the fleet's
|
||||
behavior, not just per-device. A device that has not reported in
|
||||
its expected interval is itself a signal (a dead device, a
|
||||
partitioned device, a drained battery). See
|
||||
`domains/observability/metrics` for the generic structured-metrics
|
||||
discipline; edge owns the partition-survivable, local-first angle.
|
||||
- Telemetry must not be a secrets channel (P9 analog, see
|
||||
`domains/observability/P6 No Secrets in Observability`): device
|
||||
credentials, PII, and personally-identifying location must not
|
||||
enter telemetry payloads.
|
||||
|
||||
## Command Idempotency (P5 Edge Operations are Idempotent)
|
||||
|
||||
- Device commands are retried by nature (the network is
|
||||
partition-prone). Every command carries an idempotency key so a
|
||||
retried command does not double-apply (P5). A `set-point` command
|
||||
retried with the same idempotency key sets the point once, not
|
||||
twice; an `open-valve` command retried is safe because the valve
|
||||
is already open.
|
||||
- Irreversible commands (a calibration burn-in, a firmware flash)
|
||||
require stronger idempotency: the device tracks applied
|
||||
idempotency keys and refuses re-application. A retried irreversible
|
||||
command without idempotency tracking double-applies the effect
|
||||
(P5 violation, possibly a physical-side-effect bug).
|
||||
- The idempotency key is per-command, not per-device. A device that
|
||||
dedups by device ID alone will drop distinct commands issued in
|
||||
the same window. Use `(device, command-id, ts-window)` or a
|
||||
UUID per command.
|
||||
|
||||
## Partial Degradation When Devices Drop (P7 Partial Degradation is Engineered)
|
||||
|
||||
- A fleet degrades when devices drop (battery exhaustion, partition,
|
||||
hardware failure). The system must continue to operate with the
|
||||
remaining devices; a whole-system crash on one device's failure is
|
||||
a P7 violation. The degraded mode is documented: which
|
||||
aggregations are valid with N-1 devices, which alerts fire, which
|
||||
fallbacks engage.
|
||||
- A device that drops is not an incident by itself — fleets expect
|
||||
churn. The operator-facing signal is the *aggregate* health (X%
|
||||
of devices reporting, Y% partitioned for >Z minutes), not the
|
||||
per-device drop. Per-device drop alerts are noise; aggregate
|
||||
degradation alerts are signal (see `domains/observability/metrics`).
|
||||
- A command to a dropped device must time out (P5 — idempotent
|
||||
retry) and degrade (P7 — the fleet continues without that
|
||||
device). A command that blocks forever waiting for a dropped
|
||||
device is the `blocking-call-on-constrained-device` chaos
|
||||
anti-pattern (P3 + P5 breach).
|
||||
|
||||
## Cross-Link to Messaging (P5, cross-link messaging/queues)
|
||||
|
||||
- MQTT QoS 0/1/2 maps to at-most-once / at-least-once / exactly-once
|
||||
delivery semantics — the same three-way tradeoff documented in
|
||||
`domains/messaging/queues`. The cross-link is one-directional
|
||||
outward (edge → messaging) per D-026 extended: edge owns the
|
||||
constrained-device protocol angle; messaging owns the generic
|
||||
cross-process delivery-semantics angle.
|
||||
- This link dangles until P2 (the messaging domain is authored in
|
||||
P2); it is verified bidirectional in P5 (ATELIER-114 per
|
||||
IDEATE-40). Acceptable per D-053 (vertical-slice integrity — P1
|
||||
ships the edge domain self-consistent; the messaging cross-link
|
||||
resolves by the P6 ship).
|
||||
- The parallel: a constrained device's QoS 1 publish is the
|
||||
device-flavored instance of an at-least-once queue delivery — the
|
||||
consumer (the broker or the downstream service) must be
|
||||
idempotent (P5 here, `messaging/P3 Consumers are Idempotent`
|
||||
there). The idempotency discipline is the same; the protocol and
|
||||
failure model differ (constrained-device link vs broker-backed
|
||||
network).
|
||||
|
||||
## What Violates IoT-at-the-Edge Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Undeclared device resource budget (assumes infinite battery/RAM) | P3 Resources are Constrained and Declared |
|
||||
| Shared fleet credential (one key for all devices) | P9 Identity is Constrained at the Edge |
|
||||
| Non-idempotent device command (retried command doubles the effect) | P5 Edge Operations are Idempotent |
|
||||
| Blocking synchronous call on a constrained device with no timeout | P3, P5 (blocks the node; retry unsafe) |
|
||||
| Fire-and-forget telemetry with no on-device buffer (lost on partition) | P10 Edge Observability Survives Partition |
|
||||
| Whole-system crash on one device's failure (no degradation contract) | P7 Partial Degradation is Engineered |
|
||||
| Device credential scope that includes fleet-wide topics (over-scoped) | P9, `domains/security/secrets` |
|
||||
| QoS 2 used on a class-0 device (no budget for the handshake) | P3 Resources are Constrained and Declared |
|
||||
| Per-device-drop alert (noise; aggregate degradation is the signal) | P7, `domains/observability/metrics` |
|
||||
| Telemetry payload that includes device credentials or PII | P9, `domains/observability/P6 No Secrets in Observability` |
|
||||
@@ -0,0 +1,354 @@
|
||||
# Offline-First — Derived Rules
|
||||
|
||||
> Derives from `domains/edge/first-principles.md`. Applies P2
|
||||
> (Offline is a First-Class State) primarily, with P5 (idempotent
|
||||
> queue-and-forward), P4 (bounded sync conflicts on reconnect), P7
|
||||
> (partial degradation), and P10 (local-first telemetry). Cross-links
|
||||
> `domains/concurrency/patterns` for the in-process bounded-buffer
|
||||
> analog and `domains/observability/logging` for local-first logging.
|
||||
|
||||
## What Offline-First Is (P2 Offline is a First-Class State)
|
||||
|
||||
- Offline-first is the design discipline in which the system
|
||||
continues to operate when disconnected from the center. Partition
|
||||
is the norm, not the exception; reconciliation happens on
|
||||
reconnect. The offline state is engineered, not a degenerate mode
|
||||
the app falls into by accident.
|
||||
- The boundary is per D-061: edge owns the
|
||||
proximity/location/disconnection angle. An offline-first web app
|
||||
is an edge concern because its defining trait is partition-survival
|
||||
(P2), not generic performance. The local-first storage is the
|
||||
edge device's constrained-resource reality (P3).
|
||||
- Offline-first is the precondition for the bounded-conflict
|
||||
discipline of `P4 Sync Conflicts are Bounded, Not Infinite`:
|
||||
without offline operation there is nothing to reconcile; with it,
|
||||
the reconnect reconciliation is the correctness mechanism. See
|
||||
`domains/edge/sync.md` for the conflict-resolution strategies.
|
||||
|
||||
## Local-First Storage (P2, P3)
|
||||
|
||||
- Local-first storage holds the working copy on the device:
|
||||
IndexedDB (browser), SQLite (mobile, embedded), or on-device file
|
||||
storage (desktop, IoT gateway). The local store is the authority
|
||||
while offline; the server is reconciled later, not consulted per
|
||||
read.
|
||||
- The local store is bounded by the device (P3 — Resources are
|
||||
Constrained and Declared). A local store that grows without bound
|
||||
is a defect: declare a budget (e.g., a 50 MB IndexedDB quota, a
|
||||
30-day rolling window), and evict outside the budget deterministically.
|
||||
- The local store is the offline state; without it the app is
|
||||
online-only and crashes on disconnect (P2 violation). The store is
|
||||
the reversibility mechanism (C5): every local write is reversible
|
||||
on reconcile.
|
||||
|
||||
```typescript
|
||||
// Local-first store sketch (IndexedDB). The app reads from the
|
||||
// local store, never the network, while offline. Writes queue
|
||||
// locally and forward on reconnect (P2, P5).
|
||||
const db = await openDB("atelier-offline", 1, {
|
||||
upgrade(db) {
|
||||
const store = db.createObjectStore("pending-writes", {
|
||||
keyPath: "id",
|
||||
});
|
||||
store.createIndex("by-createdAt", "createdAt");
|
||||
},
|
||||
});
|
||||
|
||||
async function readRecord(id: string) {
|
||||
// Read from local store first; the network is a reconcile path,
|
||||
// not the read path.
|
||||
return db.get("pending-writes", id);
|
||||
}
|
||||
```
|
||||
|
||||
## Queue-and-Forward for Writes (P5 Edge Operations are Idempotent)
|
||||
|
||||
- Every write while offline is queued locally and forwarded to the
|
||||
server on reconnect. The queue is the offline write-queue; the
|
||||
forward is the reconcile. Each queued write carries an idempotency
|
||||
key so a retried forward (the network is partition-prone) does not
|
||||
double-apply (P5).
|
||||
- The queue is bounded (P3): a queue that grows without limit on a
|
||||
constrained device will exhaust it. Declare a max-queue-depth and
|
||||
a max-queue-bytes; reject or evict beyond the bound with a defined
|
||||
policy (oldest-first, lowest-priority-first).
|
||||
- The queue is the cross-partition analog of the in-process bounded
|
||||
buffer — see `domains/concurrency/patterns` (bounded buffer,
|
||||
backpressure). Concurrency owns the in-process analog; edge owns
|
||||
the partition-survivable analog. The failure model differs: the
|
||||
in-process buffer fails by OOM; the offline write-queue fails by
|
||||
partition or device loss.
|
||||
|
||||
```typescript
|
||||
// Offline write-queue sketch. Each entry carries an idempotency
|
||||
// key (P5) so a retried forward is safe. The queue is bounded by
|
||||
// maxDepth (P3).
|
||||
interface PendingWrite {
|
||||
id: string; // local id
|
||||
idempotencyKey: string; // server-side dedup key (P5)
|
||||
collection: string;
|
||||
payload: unknown;
|
||||
createdAt: number;
|
||||
}
|
||||
|
||||
const MAX_DEPTH = 1000;
|
||||
|
||||
async function queueWrite(write: Omit<PendingWrite, "id" | "idempotencyKey" | "createdAt">) {
|
||||
const depth = await db.count("pending-writes");
|
||||
if (depth >= MAX_DEPTH) {
|
||||
// P3: bounded queue. Evict the oldest pending write or reject.
|
||||
// Rejecting is correct when the write is higher-priority than
|
||||
// the oldest; evicting is correct when the newest is lowest.
|
||||
throw new Error("offline-queue-full");
|
||||
}
|
||||
const entry: PendingWrite = {
|
||||
...write,
|
||||
id: crypto.randomUUID(),
|
||||
idempotencyKey: `${write.collection}:${crypto.randomUUID()}`,
|
||||
createdAt: Date.now(),
|
||||
};
|
||||
await db.put("pending-writes", entry);
|
||||
// The forward loop picks this up when connectivity returns.
|
||||
}
|
||||
|
||||
async function forwardPendingWrites(server: Server) {
|
||||
const pending = await db.getAllFromIndex("pending-writes", "by-createdAt");
|
||||
for (const write of pending) {
|
||||
// P5: idempotent — the server dedups by idempotencyKey.
|
||||
await server.apply(write, write.idempotencyKey);
|
||||
await db.delete("pending-writes", write.id);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Conflict Detection on Reconnect (P4 Sync Conflicts are Bounded)
|
||||
|
||||
- On reconnect, the queued writes are forwarded; the server may
|
||||
have advanced while the device was offline. A conflict is when the
|
||||
local write and the server state diverge. Conflict detection is
|
||||
the precondition for bounded reconciliation (P4): a write forwarded
|
||||
blindly (last-write-wins with no clock) is a P4 violation waiting
|
||||
to happen.
|
||||
- Conflict resolution strategies (CRDT, LWW with vector clocks,
|
||||
application-specific merge) are the subject of
|
||||
`domains/edge/sync.md` — the CRDT-vs-LWW decision matrix there
|
||||
determines which applies. Offline-first owns the *detection*; sync
|
||||
owns the *resolution*.
|
||||
- A reconnect that detects no conflicts when conflicts exist is a
|
||||
silent correctness defect (C1, P4). Detection must be conservative:
|
||||
when in doubt, flag a conflict and surface it to the merge
|
||||
function or the user.
|
||||
|
||||
## UI for Offline State (P7 Partial Degradation is Engineered)
|
||||
|
||||
- The UI must reflect the offline state visibly: a "you are offline,
|
||||
changes will sync when connected" banner, a pending-writes counter,
|
||||
a last-synced timestamp. A UI that hides the offline state
|
||||
violates P7 — the degraded mode is a designed state with a defined
|
||||
contract, not a silent fall-through.
|
||||
- The UI must function while offline: reads from local-first
|
||||
storage, writes to the queue, navigation that does not require the
|
||||
network. An app that shows a blank screen or a spinner-forever when
|
||||
offline has no offline state (P2 violation) and no degradation
|
||||
contract (P7 violation).
|
||||
- The pending-writes counter is the local-first analog of the
|
||||
messaging consumer-lag metric — see
|
||||
`domains/observability/metrics` for the lag-discipline parallel.
|
||||
|
||||
## Service Workers (P2, P6)
|
||||
|
||||
- A service worker is a client-side proxy that intercepts network
|
||||
requests and serves from a local cache. It is the browser's
|
||||
offline-first primitive: the service worker cache is the
|
||||
offline-capable store for assets; the IndexedDB store is the
|
||||
offline-capable store for data.
|
||||
- The service worker cache is an edge cache (P6 — Cache Invalidation
|
||||
is Explicit): it must carry a TTL or explicit invalidation
|
||||
strategy. A service worker that caches forever and never
|
||||
invalidates is a TTL-less edge cache under partition — a P6
|
||||
violation (stale-forever).
|
||||
- See `domains/edge/cdn.md` for the generic edge-cache invalidation
|
||||
discipline; the service worker is the on-device instance of it.
|
||||
|
||||
```javascript
|
||||
// Service worker cache strategy: stale-while-revalidate for
|
||||
// assets, network-first for data, explicit version-bump for
|
||||
// breaking changes (P6).
|
||||
const CACHE = "atelier-v3"; // bump on deploy to invalidate (P6)
|
||||
const ASSETS = ["/", "/app.js", "/styles.css"];
|
||||
|
||||
self.addEventListener("install", (event) => {
|
||||
event.waitUntil(
|
||||
caches.open(CACHE).then((cache) => cache.addAll(ASSETS))
|
||||
);
|
||||
});
|
||||
|
||||
self.addEventListener("fetch", (event) => {
|
||||
const url = new URL(event.request.url);
|
||||
if (url.pathname.startsWith("/api/")) {
|
||||
// Network-first for data; fall back to cache on partition (P2).
|
||||
event.respondWith(
|
||||
fetch(event.request).catch(() => caches.match(event.request))
|
||||
);
|
||||
} else {
|
||||
// Stale-while-revalidate for assets (P6 explicit invalidation).
|
||||
event.respondWith(
|
||||
caches.open(CACHE).then(async (cache) => {
|
||||
const cached = await cache.match(event.request);
|
||||
const network = fetch(event.request).then((resp) => {
|
||||
cache.put(event.request, resp.clone());
|
||||
return resp;
|
||||
}).catch(() => cached);
|
||||
return cached || network;
|
||||
})
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
self.addEventListener("activate", (event) => {
|
||||
// P6: explicit invalidation. Drop old caches on activate.
|
||||
event.waitUntil(
|
||||
caches.keys().then((keys) =>
|
||||
Promise.all(keys.filter((k) => k !== CACHE).map((k) => caches.delete(k)))
|
||||
)
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
## Offline Write-Queue and Conflict Detection Mapped to the Testing Pyramid (IDEATE-38, ATELIER-94)
|
||||
|
||||
The offline write-queue and conflict-detection patterns must be
|
||||
tested at every tier of the testing pyramid. Each tier exercises a
|
||||
different failure mode; skipping a tier leaves a correctness gap
|
||||
(P2, P4 violations that surface only in production partitions).
|
||||
|
||||
| Pyramid Tier | What it exercises | What it proves |
|
||||
|-------------|-------------------|----------------|
|
||||
| **Unit** | Conflict detection on a merge function (pure inputs → expected merge result) | The merge logic is correct in isolation (P4) — given two divergent states, the merge returns the converged state |
|
||||
| **Integration** | Reconnect reconcile against a local store (fake server, real IndexedDB/SQLite) | The queue-and-forward loop drains correctly; the local store and server converge after reconnect (P2, P5) |
|
||||
| **E2e** | Partition simulation with a fake network (the app runs in a browser, the network is cut and restored) | The offline state, UI, and reconcile work end-to-end under partition (P2, P7) |
|
||||
|
||||
- **Unit — conflict detection on a merge function.** The merge
|
||||
function is pure: given two divergent states and a clock, it
|
||||
returns the converged state. Test every merge case (LWW, CRDT
|
||||
register, set union, application-specific three-way merge) as a
|
||||
pure function. This is the cheapest tier and the highest coverage
|
||||
per test — see `domains/testing/pyramid`.
|
||||
|
||||
```typescript
|
||||
// Unit test sketch: conflict detection on a merge function (P4).
|
||||
// The merge function is pure; no network, no store. Test that
|
||||
// divergent states converge and that the merge is bounded (no
|
||||
// oscillation).
|
||||
|
||||
function mergeLWW(local: State, remote: State, clock: Clock): State {
|
||||
// Last-write-wins: the state with the later vector-clock wins.
|
||||
// Returns the converged state (P4).
|
||||
return clock.compare(local.clock, remote.clock) >= 0 ? local : remote;
|
||||
}
|
||||
|
||||
// Unit cases:
|
||||
// - local ahead → local wins
|
||||
// - remote ahead → remote wins
|
||||
// - concurrent (clocks incomparable) → conflict flagged or LWW tiebreak
|
||||
// - identical → no-op convergence (bounded, no oscillation)
|
||||
test("mergeLWW converges when local is ahead", () => {
|
||||
const local = { v: 2, clock: { a: 2 } };
|
||||
const remote = { v: 1, clock: { a: 1 } };
|
||||
expect(mergeLWW(local, remote, { compare: (a, b) => a.a - b.a })).toEqual(local);
|
||||
});
|
||||
```
|
||||
|
||||
- **Integration — reconnect reconcile against a local store.** A
|
||||
fake server stands in for the network; the real IndexedDB (or
|
||||
SQLite) holds the queue. The test fills the queue while offline,
|
||||
reconnects, and asserts the queue drains and the server and local
|
||||
store converge. This exercises the queue-and-forward loop (P5)
|
||||
and the reconcile against real storage.
|
||||
|
||||
```typescript
|
||||
// Integration test sketch: reconnect reconcile against a local
|
||||
// store. A fake server; real IndexedDB. The queue drains; the
|
||||
// server and local store converge after reconnect (P2, P5).
|
||||
|
||||
test("reconnect reconciles pending writes against the server", async () => {
|
||||
const db = await openDB("test-offline", 1, { /* schema */ });
|
||||
const server = new FakeServer();
|
||||
await queueWrite(db, { collection: "docs", payload: { v: 1 } });
|
||||
// Simulate offline: server is unreachable.
|
||||
server.offline();
|
||||
await queueWrite(db, { collection: "docs", payload: { v: 2 } });
|
||||
expect(await db.count("pending-writes")).toBe(2);
|
||||
// Simulate reconnect: server is reachable.
|
||||
server.online();
|
||||
await forwardPendingWrites(server, db);
|
||||
expect(await db.count("pending-writes")).toBe(0);
|
||||
expect(await server.latest("docs")).toEqual({ v: 2 });
|
||||
});
|
||||
```
|
||||
|
||||
- **E2e — partition simulation with a fake network.** The app runs
|
||||
in a real browser; a fake network layer cuts and restores the
|
||||
connection. The test asserts the UI shows the offline state, the
|
||||
writes queue, the reconnect reconciles, and the UI returns to
|
||||
online. This is the highest-fidelity tier and the lowest coverage
|
||||
per test — run a small number of representative scenarios, not a
|
||||
combinatorial matrix.
|
||||
|
||||
```typescript
|
||||
// E2e test sketch: partition simulation with a fake network. The
|
||||
// app runs in a browser; the network is cut and restored. Asserts
|
||||
// the offline UI state, the queue, the reconcile, and the online
|
||||
// recovery (P2, P7).
|
||||
|
||||
test("app survives a network partition and reconciles on reconnect", async () => {
|
||||
await page.goto("https://app.example.com");
|
||||
await page.click("text=Edit document");
|
||||
await page.fill("textarea", "offline edit");
|
||||
// Cut the network.
|
||||
await page.setOffline(true);
|
||||
await page.click("text=Save");
|
||||
await expect(page.locator("text=You are offline")).toBeVisible();
|
||||
await expect(page.locator("text=1 pending change")).toBeVisible();
|
||||
// Restore the network.
|
||||
await page.setOffline(false);
|
||||
await expect(page.locator("text=All changes synced")).toBeVisible();
|
||||
await expect(page.locator("text=0 pending changes")).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
- The three tiers are complementary: unit proves the merge logic,
|
||||
integration proves the reconcile loop, e2e proves the partition
|
||||
behavior. Skipping any tier leaves a correctness gap. See
|
||||
`domains/testing/pyramid` for the pyramid discipline and
|
||||
`domains/testing/fixtures` for the fake-server and fake-network
|
||||
fixture patterns.
|
||||
|
||||
## Observability (P10 Edge Observability Survives Partition)
|
||||
|
||||
- The offline state is itself an observable signal: the
|
||||
pending-writes count, the last-synced timestamp, the
|
||||
reconcile-failure count. A device stuck offline for days with a
|
||||
full queue is an incident; without local-first telemetry it is
|
||||
invisible (P10 violation).
|
||||
- Local-first logging (buffered on-device, forwarded on reconnect)
|
||||
is the offline-first instance of `P10 Edge Observability Survives
|
||||
Partition`. See `domains/observability/logging` for the generic
|
||||
structured-logging discipline the local-first buffer builds on.
|
||||
- A reconcile failure that is not logged locally is a silent defect
|
||||
— the operator cannot debug what they cannot see (C7, P10).
|
||||
|
||||
## What Violates Offline-First Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| App that crashes on disconnect (no offline state) | P2 Offline is a First-Class State |
|
||||
| Unbounded offline write-queue (grows until device OOM) | P3 Resources are Constrained and Declared |
|
||||
| Queued write forwarded without an idempotency key (retry doubles the effect) | P5 Edge Operations are Idempotent |
|
||||
| Reconnect that detects no conflicts when conflicts exist | P4 Sync Conflicts are Bounded, Not Infinite |
|
||||
| UI that hides the offline state (no banner, no pending counter) | P7 Partial Degradation is Engineered |
|
||||
| Service worker cache with no TTL and no explicit invalidation | P6 Cache Invalidation is Explicit |
|
||||
| Reconcile failure with no local log (silent under partition) | P10 Edge Observability Survives Partition |
|
||||
| Merge function that oscillates (no convergence guarantee) | P4, `domains/edge/sync.md` |
|
||||
| Local-first store with no declared budget (grows without bound) | P3, `domains/concurrency/patterns` (bounded buffer analog) |
|
||||
| E2e tests that never simulate a partition (offline path untested) | P2, `domains/testing/pyramid` |
|
||||
@@ -0,0 +1,315 @@
|
||||
# Sync — Derived Rules
|
||||
|
||||
> Derives from `domains/edge/first-principles.md`. Applies P4 (Sync
|
||||
> Conflicts are Bounded, Not Infinite) primarily, with P5
|
||||
> (idempotent merge operations), P2 (offline as the precondition),
|
||||
> and P10 (sync is observable). Cross-links `domains/data/migrations`
|
||||
> for schema migration under sync and `domains/concurrency/patterns`
|
||||
> for the immutability-aid-merge principle.
|
||||
|
||||
## What the Sync Problem Is (P4 Sync Conflicts are Bounded, Not Infinite)
|
||||
|
||||
- Sync is the discipline of reconciling divergent state across
|
||||
partitioned nodes. While partitioned, each node accepts writes
|
||||
independently; on reconnect, the divergent state must converge.
|
||||
The correctness contract is that the merge terminates and
|
||||
converges — oscillation and infinite sync loops are correctness
|
||||
failures, not eventual consistency (P4).
|
||||
- Sync is the edge domain's deepest problem: it is the
|
||||
reconciliation layer above `P2 Offline is a First-Class State`.
|
||||
Without offline operation there is nothing to sync; with it, the
|
||||
reconnect reconciliation is the correctness mechanism. See
|
||||
`domains/edge/offline-first.md` for the offline write-queue that
|
||||
produces the divergent state to be reconciled.
|
||||
- The boundary is per D-061: edge owns the partitioned-reconcile
|
||||
angle; concurrency owns the in-process analog
|
||||
(`concurrency/P1 Immutability by Default` — immutability aids
|
||||
merge); data owns the generic migration discipline
|
||||
(`data/migrations`). Sync is an edge concern because its defining
|
||||
trait is partitioned divergence, a concern that only arises at the
|
||||
network edge.
|
||||
|
||||
## Conflict Resolution Strategies (P4, C1 Correctness, C5 Reversibility)
|
||||
|
||||
- A conflict is when two nodes have divergent state for the same
|
||||
logical entity and no total order determines which is correct.
|
||||
Resolution strategies fall into two families:
|
||||
- **Conflict-free**: the data type guarantees convergence by
|
||||
construction (CRDTs). The merge is deterministic; no conflict
|
||||
surfaces to the user or the application.
|
||||
- **Conflict-tolerant**: the data type can conflict; the
|
||||
resolution policy (last-write-win, three-way merge,
|
||||
application-specific) arbitrates. Conflicts may surface to the
|
||||
user or be silently resolved per a documented policy.
|
||||
- The choice is a P4 decision: conflict-free types guarantee the
|
||||
bound (convergence) but constrain the data model; conflict-tolerant
|
||||
types are flexible but require the resolution policy to be correct
|
||||
and bounded (no oscillation). See the decision matrix below.
|
||||
|
||||
## CRDTs — Conflict-Free Replicated Data Types (P4, C5, C6)
|
||||
|
||||
- A CRDT is a data type whose merge operation is associative,
|
||||
commutative, and idempotent. Given any set of divergent replicas,
|
||||
merging them in any order converges to the same state — the merge
|
||||
is deterministic and terminating (P4 bound). CRDTs derive from C5
|
||||
Reversibility (divergent state reverses to convergence) and C6
|
||||
Composability (CRDTs compose: a CRDT map of CRDT registers is
|
||||
itself a CRDT).
|
||||
- **State-based (CvRDT — convergent):** each replica carries its
|
||||
full state; merge is a least-upper-bound on a semi-lattice. The
|
||||
payload is larger (full state per merge); the merge is simple
|
||||
(one function). Fits small state and unreliable networks.
|
||||
- **Operation-based (CmRDT — commutative):** each replica carries
|
||||
operations; merge is applying the operations in causal order. The
|
||||
payload is smaller (ops, not state); the delivery must be
|
||||
reliable and causally ordered. Fits large state and reliable
|
||||
transport.
|
||||
- The tradeoff: state-based is simpler but heavier; operation-based
|
||||
is lighter but requires causal delivery. Both guarantee
|
||||
convergence (P4); the choice is a C8 Economy decision (bandwidth
|
||||
vs delivery complexity).
|
||||
|
||||
```typescript
|
||||
// CRDT register: LWW-element-set (state-based, CvRDT). The merge
|
||||
// is deterministic — the register with the later timestamp wins.
|
||||
// Convergence is guaranteed (P4); the merge is idempotent (P5).
|
||||
|
||||
interface LWWRegister<T> {
|
||||
value: T;
|
||||
timestamp: number; // monotonic clock; ties broken by node id
|
||||
nodeId: string;
|
||||
}
|
||||
|
||||
function mergeLWWRegister<T>(
|
||||
local: LWWRegister<T>,
|
||||
remote: LWWRegister<T>,
|
||||
): LWWRegister<T> {
|
||||
// The merge is associative, commutative, idempotent (P4, P5).
|
||||
// (local.timestamp, local.nodeId) > (remote.timestamp, remote.nodeId)
|
||||
// is a total order — no oscillation, no infinite loop.
|
||||
if (local.timestamp > remote.timestamp) return local;
|
||||
if (local.timestamp < remote.timestamp) return remote;
|
||||
// Tie: break by node id for a deterministic total order.
|
||||
return local.nodeId > remote.nodeId ? local : remote;
|
||||
}
|
||||
|
||||
// The register is a CRDT: merge(merge(a, b), c) === merge(a, merge(b, c))
|
||||
// for any replicas a, b, c. Convergence is guaranteed (P4).
|
||||
```
|
||||
|
||||
```typescript
|
||||
// CRDT set: add-wins last-write-wins element set (state-based).
|
||||
// Each element carries a timestamp; remove only wins if the
|
||||
// remove-timestamp is later than the add-timestamp. This avoids
|
||||
// the remove-wins-vs-add race (P4) without surfacing a conflict.
|
||||
|
||||
interface AWLWWSet<T> {
|
||||
adds: Map<T, number>; // element -> add-timestamp
|
||||
removes: Map<T, number>; // element -> remove-timestamp
|
||||
}
|
||||
|
||||
function mergeAWLWWSet<T>(a: AWLWWSet<T>, b: AWLWWSet<T>): AWLWWSet<T> {
|
||||
const adds = new Map<T, number>(a.adds);
|
||||
const removes = new Map<T, number>(a.removes);
|
||||
for (const [el, ts] of b.adds) {
|
||||
adds.set(el, Math.max(adds.get(el) ?? 0, ts)); // add-wins union
|
||||
}
|
||||
for (const [el, ts] of b.removes) {
|
||||
removes.set(el, Math.max(removes.get(el) ?? 0, ts));
|
||||
}
|
||||
return { adds, removes };
|
||||
}
|
||||
|
||||
function contains<T>(set: AWLWWSet<T>, el: T): boolean {
|
||||
const addTs = set.adds.get(el) ?? 0;
|
||||
const rmTs = set.removes.get(el) ?? 0;
|
||||
return addTs > rmTs; // add wins on equal timestamp (P4 bounded)
|
||||
}
|
||||
```
|
||||
|
||||
## Last-Write-Win (LWW) with Vector Clocks (P4, C1, C5)
|
||||
|
||||
- LWW is the simplest conflict-tolerant strategy: the write with the
|
||||
latest timestamp wins. It is cheap, but it silently discards
|
||||
concurrent writes — the "lost update" is the correctness cost. LWW
|
||||
is correct only when the timestamp is a total order (a monotonic
|
||||
clock, not wall time), and when lost concurrent writes are
|
||||
acceptable (e.g., caching, presence, ephemeral state).
|
||||
- **Vector clocks** are the timestamp that knows about concurrency.
|
||||
A vector clock records the logical time of each node; two writes
|
||||
are concurrent iff neither vector dominates the other. LWW with
|
||||
vector clocks: a write that is causally later wins; a write that
|
||||
is concurrent conflicts and is resolved by a tiebreak (node id,
|
||||
wall time, or application policy).
|
||||
- The tiebreak is the P4 bound: the conflict must be resolved
|
||||
deterministically (no oscillation) and the resolution must be
|
||||
documented. A tiebreak by wall time alone (no vector clock) is a
|
||||
P4 violation waiting to happen — wall time skews across nodes,
|
||||
and a clock skew can flip the tiebreak, oscillating the merge.
|
||||
|
||||
```typescript
|
||||
// LWW with vector clocks (conflict-tolerant, P4 bounded). The
|
||||
// vector clock records causal order; concurrent writes conflict;
|
||||
// the conflict is tiebroken deterministically (no oscillation).
|
||||
|
||||
type VectorClock = Record<string, number>; // nodeId -> counter
|
||||
|
||||
function compareClock(a: VectorClock, b: VectorClock): "before" | "after" | "equal" | "concurrent" {
|
||||
let aBefore = false, bBefore = false;
|
||||
const keys = new Set([...Object.keys(a), ...Object.keys(b)]);
|
||||
for (const k of keys) {
|
||||
const av = a[k] ?? 0;
|
||||
const bv = b[k] ?? 0;
|
||||
if (av < bv) aBefore = true;
|
||||
if (av > bv) bBefore = true;
|
||||
}
|
||||
if (aBefore && bBefore) return "concurrent";
|
||||
if (aBefore) return "before";
|
||||
if (bBefore) return "after";
|
||||
return "equal";
|
||||
}
|
||||
|
||||
interface LWWVectorState<T> {
|
||||
value: T;
|
||||
clock: VectorClock;
|
||||
writerId: string; // tiebreak: deterministic, no oscillation (P4)
|
||||
}
|
||||
|
||||
function mergeLWWVector<T>(
|
||||
local: LWWVectorState<T>,
|
||||
remote: LWWVectorState<T>,
|
||||
): LWWVectorState<T> {
|
||||
const order = compareClock(local.clock, remote.clock);
|
||||
if (order === "before") return remote; // remote causally later
|
||||
if (order === "after" || order === "equal") return local;
|
||||
// Concurrent: tiebreak by writer id (deterministic, P4 bounded).
|
||||
return local.writerId > remote.writerId ? local : remote;
|
||||
}
|
||||
```
|
||||
|
||||
- The merge is idempotent (P5): merging the same two replicas twice
|
||||
yields the same result. The tiebreak by `writerId` is a total
|
||||
order, so the merge cannot oscillate (P4 bound).
|
||||
- A vector-clock merge that surfaces the concurrent conflict to the
|
||||
application (instead of tiebreaking) is also valid — the
|
||||
application resolves per its own policy. The P4 bound is that the
|
||||
resolution terminates; the policy determines whether the user sees
|
||||
the conflict or the system silences it.
|
||||
|
||||
## Merge Semantics (P4, P5, cross-link concurrency/patterns)
|
||||
|
||||
- The merge function is the heart of sync. Its properties (P4):
|
||||
- **Associative**: `merge(merge(a, b), c) === merge(a, merge(b, c))`.
|
||||
- **Commutative**: `merge(a, b) === merge(b, a)`.
|
||||
- **Idempotent**: `merge(a, a) === a` (P5 — retried merges are safe).
|
||||
- Immutability aids merge: an immutable state representation (the
|
||||
CRDT payload, the LWW register with a clock) makes the merge a
|
||||
pure function of two inputs, with no in-place mutation race. See
|
||||
`domains/concurrency/patterns` (`concurrency/P1 Immutability by
|
||||
Default`) for the in-process immutability principle; sync is the
|
||||
cross-partition instance of it.
|
||||
- A merge that mutates in place is a P5 violation waiting to
|
||||
happen: a retried merge mutates the same state twice, and the
|
||||
result is not idempotent. Always merge into a new state; never
|
||||
mutate the inputs.
|
||||
|
||||
## Conflict-Free vs Conflict-Tolerant Data Types (P4, C3 Simplicity)
|
||||
|
||||
- **Conflict-free (CRDTs):** the data type guarantees convergence.
|
||||
The application never sees a conflict; the merge is deterministic.
|
||||
The cost: the data model is constrained (counters, sets, registers,
|
||||
maps of these). A conflict-free type for arbitrary JSON is hard;
|
||||
a conflict-free type for a counter is a PN-counter.
|
||||
- **Conflict-tolerant (LWW, three-way merge, application policy):**
|
||||
the data type can conflict; the resolution policy arbitrates. The
|
||||
cost: the policy must be correct and bounded (no oscillation), and
|
||||
the conflict may surface to the user. The benefit: any data model
|
||||
can be made conflict-tolerant (just pick a tiebreak).
|
||||
- The choice is the decision matrix below. It is a P4 decision
|
||||
(which bound), a C1 decision (which correctness cost is
|
||||
acceptable), and a C3 decision (which simplicity is affordable).
|
||||
See also `domains/data/migrations` for the schema-evolution angle
|
||||
— a schema change under sync must be compatible with both
|
||||
replicas, or the merge fails on the new shape.
|
||||
|
||||
## Schema Migration Under Sync (P4, cross-link data/migrations)
|
||||
|
||||
- A schema migration under sync is harder than a single-node
|
||||
migration: both replicas must understand the new shape, or the
|
||||
merge fails. The migration must be forward-and-backward compatible
|
||||
across all replicas that may still hold the old shape — see
|
||||
`domains/data/migrations` for the generic compatibility discipline.
|
||||
- A breaking schema change under sync requires a staged migration:
|
||||
deploy the new-shape-aware merge first (it accepts both shapes),
|
||||
then deploy the new shape, then deploy the old-shape-removing
|
||||
merge. A big-bang schema change under sync is a P4 violation: the
|
||||
replicas that have not yet upgraded will fail the merge, and the
|
||||
sync will not converge.
|
||||
- The merge function's version awareness is the P4 bound: the merge
|
||||
must handle every shape version that may exist in the fleet, or
|
||||
reject (and surface) the merge rather than silently corrupting.
|
||||
|
||||
## CRDT vs Last-Write-Win — Decision Matrix (D-069)
|
||||
|
||||
| Strategy | When | Correctness Guarantee | Operational Cost | Failure Mode |
|
||||
|----------|------|------------------------|-------------------|--------------|
|
||||
| CRDT (state-based, CvRDT) | The data model fits a CRDT (counter, set, register, map of these); convergence must be guaranteed without surfacing conflicts; the network is unreliable (full-state merge tolerates dropped ops) | Strong eventual convergence — `merge(a, b) === merge(b, a)` for any replicas (P4 bound by construction) | Medium — full state per merge (bandwidth); semi-lattice merge function per type; CRDT library or hand-rolled | A bug in the merge function = silent divergence (C1); large state = bandwidth cost on constrained links (P3) |
|
||||
| CRDT (operation-based, CmRDT) | The data model fits a CRDT; bandwidth is constrained (ops are smaller than state); the transport is reliable and causally ordered | Strong eventual convergence — same guarantee, smaller payload | High — requires causal delivery (vector clock or broker with ordering); op transform must be idempotent (P5) | Causal-delivery violation = lost ops = divergence; op-transform bug = silent divergence |
|
||||
| Last-Write-Win (LWW) with vector clocks | The data model is arbitrary (any JSON, any record); concurrent writes are acceptable to discard or tiebreak; a total order tiebreak (node id) is acceptable | Bounded convergence — causally-later writes win; concurrent writes are tiebroken deterministically (P4 bound via tiebreak) | Low — simple merge (compare clocks, pick winner); no CRDT library; small payload | Concurrent writes are silently discarded (lost update); tiebreak by wall time = clock-skew oscillation (P4 violation); no vector clock = no concurrent-write detection = silent loss |
|
||||
| LWW with wall-clock timestamp only | The data model is ephemeral (cache, presence); lost updates are acceptable; the clock is roughly synchronized (NTP) | Weak — convergence eventually, but concurrent writes may oscillate with clock skew; no concurrent-write detection | Lowest — one timestamp per write; no clock vector | Clock skew = oscillation (P4 violation); concurrent writes silently lost; not a correctness-safe strategy for durable state |
|
||||
| Three-way merge (application-specific) | The data model is structured (documents, forms); conflicts should surface to the user or a domain-specific resolver; the merge is field-level | Bounded if the merge function is correct (associative, commutative, idempotent — P4, P5); conflicts surface per field | High — application-specific merge function per type; UI for conflict resolution; user-facing conflict surface | Merge-function bug = silent divergence or oscillation; unbounded conflict UI = user fatigue |
|
||||
|
||||
- The default for structured state that must converge silently is a
|
||||
**CRDT** (state-based for unreliable networks, operation-based for
|
||||
bandwidth-constrained reliable transport). The default for
|
||||
arbitrary JSON where lost concurrent updates are acceptable is
|
||||
**LWW with vector clocks** (never wall-clock-only for durable
|
||||
state). The default for user-facing documents where conflicts
|
||||
should surface is **three-way merge** with a documented resolution
|
||||
policy.
|
||||
- The failure-mode column is the P4 check: every row except
|
||||
wall-clock-only LWW carries a bounded failure mode (the bug is in
|
||||
the implementation, not the strategy). Wall-clock-only LWW carries
|
||||
an unbounded failure mode (clock skew = oscillation) and is a P4
|
||||
violation for durable state. Use it only for ephemeral state
|
||||
where lost updates are acceptable.
|
||||
- The choice is a P4 decision (which bound) and a C1 decision
|
||||
(which correctness cost). A CRDT guarantees convergence but
|
||||
constrains the data model; LWW is flexible but discards concurrent
|
||||
writes. Neither is universally correct; the matrix is the
|
||||
decision tool.
|
||||
|
||||
## Observability of Sync (P10 Edge Observability Survives Partition)
|
||||
|
||||
- Sync is itself an observable operation: the merge count, the
|
||||
conflict count, the convergence lag (time from reconnect to
|
||||
convergence), and the divergent-replica count are first-class
|
||||
signals. A sync that runs forever without converging is the
|
||||
`edge-sync-loop` chaos anti-pattern (P4 breach); without
|
||||
observability it is invisible until the user notices the stale
|
||||
state.
|
||||
- A conflict that is silently resolved should be logged (the
|
||||
resolution policy applied, the discarded write's idempotency key,
|
||||
the winning write's clock). A conflict that surfaces to the user
|
||||
should be metricated (the conflict rate, the resolution time).
|
||||
See `domains/observability/metrics` for the generic discipline.
|
||||
- A divergent replica that has not converged after the expected
|
||||
window is an incident; without a metric it is invisible (P10
|
||||
breach). Wire sync convergence to an alert — the
|
||||
`divergent-replica-count` is the sync analog of the messaging
|
||||
`consumer-lag` metric.
|
||||
|
||||
## What Violates Sync Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Sync loop that oscillates forever (CRDT without merge-semantics, LWW without monotonic clock) | P4 Sync Conflicts are Bounded, Not Infinite |
|
||||
| LWW with wall-clock timestamp only on durable state (clock skew = oscillation) | P4, C1 (no concurrent-write detection) |
|
||||
| Merge function that mutates inputs in place (retried merge is not idempotent) | P5 Edge Operations are Idempotent, `domains/concurrency/patterns` |
|
||||
| Big-bang schema change under sync (replicas fail the merge) | P4, `domains/data/migrations` |
|
||||
| Conflict silently resolved with no log (the policy is invisible) | P10 Edge Observability Survives Partition |
|
||||
| Divergent replica with no convergence-lag metric (invisible stale state) | P10, `domains/observability/metrics` |
|
||||
| Three-way merge with an unbounded conflict UI (user fatigue, no termination) | P4 (the merge must terminate) |
|
||||
| Operation-based CRDT without causal delivery (lost ops = divergence) | P4, C1 (the delivery contract is the bound) |
|
||||
| Merge that surfaces every concurrent conflict to the user (no default policy) | P4, C3 (the default policy is the simplicity bound) |
|
||||
| Sync with no convergence test (the merge is untested under partition) | P4, `domains/testing/pyramid` |
|
||||
@@ -0,0 +1,379 @@
|
||||
# Delivery Semantics — Derived Rules
|
||||
|
||||
> Derives from `domains/messaging/first-principles.md`. Applies P4
|
||||
> (Delivery Semantics are Explicit), P3 (Consumers are Idempotent),
|
||||
> P2 (Ordering is a Property, Not an Assumption), P5 (Dead-Letter
|
||||
> Handling is Defined), and P6 (Backpressure is Bounded) primarily,
|
||||
> with P10 (DLQ depth as an alert). For the dead-letter strategy
|
||||
> decision, see the comparison table below. Includes a fenced
|
||||
> idempotency-key dedup-store example (IDEATE-39). Cross-links
|
||||
> `domains/concurrency/patterns` for the in-process retry/backoff
|
||||
> analog, `domains/errors/patterns` for errors as data for message
|
||||
> failures, and `domains/observability/metrics` for DLQ depth as an
|
||||
> alert.
|
||||
|
||||
## The Three Delivery Semantics (P4 Delivery Semantics are Explicit)
|
||||
|
||||
- **At-most-once**: a message is delivered 0 or 1 times; loss is
|
||||
possible, duplication is not. The producer fires-and-forgets; the
|
||||
broker does not ack; the consumer does not dedup. Lowest latency,
|
||||
lowest implementation cost, lossy. Fits telemetry where a dropped
|
||||
sample is acceptable (MQTT QoS 0, fire-and-forget metrics).
|
||||
- **At-least-once**: a message is delivered 1 or more times;
|
||||
duplication is possible, loss is not. The producer sends and
|
||||
waits for a broker ack; the consumer processes and acks; a crash
|
||||
before the consumer's ack triggers redelivery. The consumer must
|
||||
be idempotent (P3). The default for side-effecting operations
|
||||
(orders, payments, commands). The vast majority of broker-backed
|
||||
queues (SQS standard, RabbitMQ ack, MQTT QoS 1).
|
||||
- **Exactly-once**: a message is delivered exactly 1 time; no
|
||||
loss, no duplication. In practice, this is at-least-once plus
|
||||
idempotency (P3), or a transactional consume-process-produce loop
|
||||
(P4 — see `domains/messaging/streams.md`). Jepsen analyses verify
|
||||
broker claims: "exactly-once" requires independent verification;
|
||||
the durable engineering practice is at-least-once with idempotent
|
||||
consumers (P3), which collapses to exactly-once under correct
|
||||
dedup.
|
||||
- The semantic is declared per channel (P4), not emergent. An
|
||||
unstated semantic is a defect: the consumer guesses, and the
|
||||
guess is wrong under the first failure. See
|
||||
`domains/messaging/queues.md` for the three-semantics comparison
|
||||
table (semantics, latency cost, implementation cost, when each
|
||||
fits).
|
||||
|
||||
## Idempotency (P3 Consumers are Idempotent)
|
||||
|
||||
- Idempotency is the correctness property that makes at-least-once
|
||||
safe. A consumer that processes the same message twice has the
|
||||
same effect as processing it once. The mechanism is the
|
||||
idempotency key: a per-message unique identifier the consumer
|
||||
uses to dedup redeliveries.
|
||||
- The idempotency key is per-message, not per-producer or
|
||||
per-session. A consumer that dedups by producer alone drops
|
||||
distinct messages issued in the same window. Use a UUID per
|
||||
message, or a deterministic key derived from the message content
|
||||
(e.g., `(entity, operation, version)`).
|
||||
- The dedup store is bounded (P6 — Backpressure is Bounded): a
|
||||
dedup store that grows without bound is a memory leak. Use a TTL
|
||||
window longer than the broker's max-redelivery window, or a
|
||||
bounded LRU. The TTL is the P6 bound: a key seen within the TTL
|
||||
is a redelivery; a key older than the TTL is expired (the broker
|
||||
has given up redelivering it).
|
||||
|
||||
## Idempotency-Key Dedup Store (IDEATE-39, P3 — fenced example)
|
||||
|
||||
The dedup store is the concrete mechanism that makes a consumer
|
||||
idempotent under at-least-once delivery. This is the fenced
|
||||
example required by IDEATE-39 (parallel to the v0.3 IDEATE-29
|
||||
signed-attestation fenced example): it is NOT prose-only — the
|
||||
consumer-with-dedup-store demonstrates P3 concretely.
|
||||
|
||||
```python
|
||||
# Idempotent consumer with a dedup store (P3 Consumers are
|
||||
# Idempotent, P6 Backpressure is Bounded — the dedup store is
|
||||
# TTL-bounded). The consumer dedups by idempotency key before
|
||||
# processing; a redelivered message is a no-op, not a double-apply.
|
||||
|
||||
import time
|
||||
|
||||
# P6: the dedup store is bounded by a TTL window. The TTL must
|
||||
# exceed the broker's max-redelivery window; beyond the TTL, the
|
||||
# key is expired (the broker has given up). A dedup store with no
|
||||
# TTL is a memory leak (P6 violation).
|
||||
DEDUP_TTL_SECONDS = 24 * 3600 # longer than max-redelivery window
|
||||
|
||||
class DedupStore:
|
||||
"""A TTL-bounded idempotency-key dedup store (P3, P6).
|
||||
|
||||
seen(key): True if the key was processed within the TTL.
|
||||
mark(key): Record the key as processed (with a timestamp).
|
||||
"""
|
||||
def __init__(self, backend):
|
||||
# backend is a Redis, a DB, or an in-process LRU. The
|
||||
# backend must be shared across consumer instances if the
|
||||
# subscription is shared (P2/P3 — see domains/messaging/
|
||||
# pubsub.md on shared vs independent subscriptions).
|
||||
self.backend = backend
|
||||
|
||||
def seen(self, key: str) -> bool:
|
||||
ts = self.backend.get(key)
|
||||
if ts is None:
|
||||
return False
|
||||
if time.time() - ts > DEDUP_TTL_SECONDS:
|
||||
# P6: expired. The broker has given up redelivering;
|
||||
# this key is no longer a redelivery signal.
|
||||
self.backend.delete(key)
|
||||
return False
|
||||
return True
|
||||
|
||||
def mark(self, key: str):
|
||||
self.backend.set(key, time.time(), ttl=DEDUP_TTL_SECONDS)
|
||||
|
||||
|
||||
# The idempotent consumer: dedup before process, mark after
|
||||
# process, ack after mark. A crash before mark re-processes (the
|
||||
# dedup store does not have the key); a crash before ack
|
||||
# redelivers (the broker did not see the ack) and the dedup store
|
||||
# makes the redelivery a no-op (P3).
|
||||
def consume_idempotent(broker, dedup: DedupStore, process):
|
||||
for message in broker.receive():
|
||||
# P3: dedup BEFORE process. A redelivered message is a
|
||||
# no-op, not a double-apply.
|
||||
if dedup.seen(message["idempotencyKey"]):
|
||||
broker.ack(message) # already processed; skip
|
||||
continue
|
||||
try:
|
||||
process(message["payload"])
|
||||
# P3: mark AFTER process succeeds. A crash between
|
||||
# process and mark re-processes (acceptable: the
|
||||
# process must be idempotent OR the mark must be
|
||||
# transactional with the process — see below).
|
||||
dedup.mark(message["idempotencyKey"])
|
||||
broker.ack(message)
|
||||
except TransientError as exc:
|
||||
# P6: bounded retry with backoff. Nack for redelivery;
|
||||
# the broker redelivers after exponential backoff.
|
||||
broker.nack(message, delay=backoff(message["attempt"]))
|
||||
except PoisonError as exc:
|
||||
# P5: poison message. Route to DLQ, do NOT retry
|
||||
# forever (see the DLQ routing rule below).
|
||||
route_to_dlq(broker, message, exc)
|
||||
broker.ack(message) # remove from the origin queue
|
||||
```
|
||||
|
||||
- The order `process → mark → ack` gives at-least-once with
|
||||
idempotent dedup: a crash before `mark` re-processes (the dedup
|
||||
store does not have the key), and a crash before `ack`
|
||||
redelivers (the dedup store makes the redelivery a no-op). If
|
||||
`process` is not itself idempotent, the `process → mark` window
|
||||
must be transactional (e.g., process and mark in one DB
|
||||
transaction) — otherwise a crash in the window double-applies.
|
||||
- For a non-idempotent `process` (e.g., a payment that must not
|
||||
double-charge), use a transactional dedup: process and mark in
|
||||
one DB transaction, so the mark commits iff the process
|
||||
commits. This is the "exactly-once via idempotency" pattern
|
||||
(P4): at-least-once delivery plus a transactional
|
||||
process-and-mark collapses to exactly-once under correct
|
||||
transactional semantics.
|
||||
|
||||
## Ordering (P2 Ordering is a Property, Not an Assumption)
|
||||
|
||||
- The delivery semantic interacts with ordering (P2). At-least-once
|
||||
with per-partition ordering: a redelivery within a partition
|
||||
preserves order (the redelivered message re-appears in its
|
||||
original position relative to other messages the consumer has
|
||||
not yet seen). At-least-once with no ordering: a redelivery may
|
||||
appear out of order relative to messages delivered after it.
|
||||
- The consumer must not assume an ordering property the broker
|
||||
does not provide (P2). A standard queue delivers per-receive-node
|
||||
arrival order but no global order and no order across redeliveries;
|
||||
a FIFO queue delivers strict per-group order including across
|
||||
redeliveries; a partitioned stream delivers strict per-partition
|
||||
order across redeliveries. Document the property; do not assume
|
||||
it. See `domains/messaging/queues.md` (FIFO vs standard) and
|
||||
`domains/messaging/streams.md` (per-partition order).
|
||||
|
||||
## Dead-Letter Strategies (P5 Dead-Letter Handling is Defined)
|
||||
|
||||
- A poison message (unparseable, repeatedly failing, or exhausting
|
||||
the retry budget) must be routed to a dead-letter queue, not
|
||||
retried forever or silently dropped (P5). The DLQ is observable
|
||||
(P10 — depth is an alert) and drainable (an operator can
|
||||
inspect, replay, or discard with audit).
|
||||
- The dead-letter strategy determines when a message is
|
||||
dead-lettered and what the operator sees. The choice is the
|
||||
decision matrix below (D-069).
|
||||
|
||||
## Dead-Letter Strategy Comparison (D-069)
|
||||
|
||||
| Strategy | When It Applies | Failure Visibility | Operational Cost |
|
||||
|----------|-----------------|-------------------|------------------|
|
||||
| **Retry-count-limit** | A fixed max-redeliveries count (e.g., 5). After N redeliveries, route to DLQ. Simple, predictable. | The redelivery count is visible in the DLQ entry; the operator sees how many times it was retried. | Low — a counter per message; no backoff tuning. Risk: retries fire as fast as the broker redelivers, hammering a downstream that is already failing (P6 — no backoff = no backpressure escape). |
|
||||
| **TTL-with-backoff** | A max time-to-live for redelivery (e.g., 30 minutes) with exponential backoff between retries. After the TTL, route to DLQ. | The TTL and the backoff schedule are visible; the operator sees the retry timeline. | Medium — backoff tuning per message type. Benefit: backoff gives the downstream time to recover (P6 — the retry rate is bounded); fits transient failures (a downstream that is briefly unavailable). |
|
||||
| **Poison-queue** | A separate queue for messages that fail a specific check (unparseable, schema-invalid, unknown type) before any processing retry. Routed immediately, not retried. | The poison queue is a separate signal from the DLQ; the operator sees parse-vs-process failures distinctly. | Low — a routing rule per check. Benefit: distinguishes "never going to succeed" (poison) from "might succeed on retry" (DLQ). Use for unparseable messages that no retry will fix. |
|
||||
| **DLQ + alert** | Any of the above strategies, plus an alert on DLQ depth. The DLQ is observable (P10) — depth, age, and rate are alerted. | Highest — the operator is paged on DLQ growth; the DLQ is a first-class signal, not a graveyard. | Medium — alerting setup per DLQ. This is the P5/P10 floor: a DLQ without an alert is a silent correctness defect (poison messages accumulate invisibly). |
|
||||
|
||||
- The default for transient failures is **TTL-with-backoff + DLQ
|
||||
+ alert**: backoff gives the downstream time to recover (P6),
|
||||
the TTL bounds the retry budget (P5), the DLQ captures the
|
||||
unprocessable, and the alert makes it visible (P10). The default
|
||||
for unparseable messages is **poison-queue + alert**: route
|
||||
immediately, do not retry a message no retry will fix.
|
||||
- **Retry-count-limit alone** (no backoff, no alert) is the
|
||||
`messaging-unbounded-retry` chaos anti-pattern's cousin: it
|
||||
caps the count but hammers the downstream at full retry rate,
|
||||
and the DLQ grows silently if no alert is wired. Always pair
|
||||
a retry budget with backoff (P6) and an alert (P10).
|
||||
- The failure-visibility column is the P10 check: every strategy
|
||||
must surface the failure to the operator. A strategy with no
|
||||
visibility is a P10 violation regardless of its retry semantics.
|
||||
- The operational-cost column is the C8 tradeoff: more visibility
|
||||
and more backoff cost more to set up but pay back in operational
|
||||
stability. The P5/P10 floor is "DLQ + alert"; below that, the
|
||||
strategy is a silent defect waiting to grow.
|
||||
|
||||
## DLQ Routing Rule (P5, P10)
|
||||
|
||||
```python
|
||||
# DLQ routing rule (P5 dead-letter handling, P6 bounded retry with
|
||||
# backoff, P10 DLQ depth alert). Combines TTL-with-backoff for
|
||||
# transient failures and poison-queue for unparseable messages,
|
||||
# with an alert on DLQ depth.
|
||||
|
||||
import json, time
|
||||
|
||||
MAX_RETRY_TTL_SECONDS = 30 * 60 # 30 min total retry window
|
||||
DLQ = "orders-dlq"
|
||||
POISON = "orders-poison" # unparseable; never retried
|
||||
|
||||
def route_to_dlq(broker, message, reason):
|
||||
"""Route a message to the DLQ with audit metadata (P5)."""
|
||||
broker.send(DLQ, body=json.dumps({
|
||||
"original": message,
|
||||
"reason": str(reason),
|
||||
"deadLetteredAt": now_iso(),
|
||||
"redeliveryCount": message.get("attempt", 0),
|
||||
}))
|
||||
# P10: emit a metric so DLQ depth alerts fire. A DLQ that
|
||||
# grows with no alert is a silent correctness defect (P5/P10).
|
||||
metrics.increment("dlq.depth", tags={"queue": "orders"})
|
||||
# The ack removes the message from the origin queue; the DLQ
|
||||
# is the durable record (P5 — observable and drainable).
|
||||
|
||||
def consume_with_dlq(broker, dedup, process):
|
||||
for message in broker.receive():
|
||||
# P1: parse first. An unparseable message is poison —
|
||||
# route immediately, do NOT retry (no retry will fix it).
|
||||
try:
|
||||
payload = json.loads(message["body"])
|
||||
except (ValueError, SchemaError) as exc:
|
||||
broker.send(POISON, body=json.dumps({
|
||||
"original": message["body"],
|
||||
"reason": f"parse-failed: {exc}",
|
||||
"poisonedAt": now_iso(),
|
||||
}))
|
||||
metrics.increment("poison.depth", tags={"queue": "orders"})
|
||||
broker.ack(message) # remove from origin; poison queue holds it
|
||||
continue
|
||||
|
||||
# P3: dedup before process.
|
||||
if dedup.seen(payload["idempotencyKey"]):
|
||||
broker.ack(message); continue
|
||||
|
||||
attempt = payload.get("attempt", 0)
|
||||
first_attempt_ts = payload.get("firstAttemptTs", time.time())
|
||||
|
||||
try:
|
||||
process(payload)
|
||||
dedup.mark(payload["idempotencyKey"])
|
||||
broker.ack(message)
|
||||
except TransientError as exc:
|
||||
# P6: TTL-with-backoff. If the retry window is
|
||||
# exhausted, route to DLQ; otherwise redeliver with
|
||||
# exponential backoff.
|
||||
if time.time() - first_attempt_ts > MAX_RETRY_TTL_SECONDS:
|
||||
route_to_dlq(broker, message, exc) # P5
|
||||
broker.ack(message)
|
||||
else:
|
||||
broker.nack(message, delay=backoff(attempt))
|
||||
except PermanentError as exc:
|
||||
# A permanent error (e.g., a not-found dependency)
|
||||
# does not benefit from retry — route to DLQ now.
|
||||
route_to_dlq(broker, message, exc)
|
||||
broker.ack(message)
|
||||
```
|
||||
|
||||
- The routing rule distinguishes three failure modes: **poison**
|
||||
(unparseable — route immediately, no retry), **transient**
|
||||
(retry with backoff until the TTL, then DLQ), and **permanent**
|
||||
(a retry will not fix it — DLQ now). This distinction is the P5
|
||||
discipline: not every failure is a retry; some are immediate
|
||||
DLQs.
|
||||
- The DLQ entry carries `reason`, `deadLetteredAt`, and
|
||||
`redeliveryCount` — it is auditable (the operator knows why each
|
||||
message was dead-lettered and how many times it was retried).
|
||||
This is the errors-as-data discipline — see
|
||||
`domains/errors/patterns` for the general principle a DLQ entry
|
||||
instantiates.
|
||||
|
||||
## Retry Budgets and Backoff (P6 Backpressure is Bounded)
|
||||
|
||||
- The retry budget is the cap on redelivery: a count, a TTL, or
|
||||
both. After the budget, the message routes to the DLQ (P5). An
|
||||
unbounded retry budget is the `messaging-unbounded-retry` chaos
|
||||
anti-pattern (P5 breach): the consumer never makes progress past
|
||||
the poison message.
|
||||
- **Exponential backoff** spaces retries: 1s, 2s, 4s, 8s, ... with
|
||||
a jitter to avoid thundering-herd synchrony. Backoff gives the
|
||||
downstream time to recover (P6 — the retry rate is bounded,
|
||||
giving the downstream a chance to catch up). A retry with no
|
||||
backoff hammers the downstream at full rate, making the failure
|
||||
worse.
|
||||
- The retry budget × the backoff schedule is the P6 bound: the
|
||||
consumer's retry load is bounded by design, not by luck. See
|
||||
`domains/concurrency/patterns` Pattern 6 (Timeout on Every
|
||||
Block) for the in-process retry/backoff analog; messaging owns
|
||||
the broker-backed instance where the redelivery comes from the
|
||||
broker across a network, not an in-process loop (D-062).
|
||||
|
||||
## Poison Messages (P5, P1)
|
||||
|
||||
- A poison message is one no retry will fix: unparseable (the
|
||||
schema is wrong, P1), unknown type (the consumer does not handle
|
||||
this version, P9), or a permanent failure (a not-found
|
||||
dependency). Retrying a poison message wastes resources and
|
||||
blocks the queue (P6 — the consumer never makes progress).
|
||||
- Poison messages route to the poison queue immediately (no
|
||||
retry), distinct from the DLQ (which holds messages that
|
||||
exhausted their retry budget on transient failures). The
|
||||
distinction is the P5 discipline: a poison queue is for
|
||||
"never going to succeed"; a DLQ is for "might have succeeded
|
||||
but didn't within the budget."
|
||||
- A poison queue without an alert is the same defect as a DLQ
|
||||
without an alert (P10): the operator cannot see the poison
|
||||
accumulating. Wire both to `domains/observability/metrics`.
|
||||
|
||||
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
|
||||
|
||||
- The retry/backoff discipline here is the cross-process analog
|
||||
of `domains/concurrency/patterns` Pattern 6 (Timeout on Every
|
||||
Block) for in-process retry. Concurrency owns the in-process
|
||||
analog (a retry loop within one program, with a timeout per
|
||||
attempt); messaging owns the broker-backed instance (the broker
|
||||
redelivers across a network, the consumer applies backoff via
|
||||
nack-with-delay). The failure model differs: in-process retry
|
||||
fails by a thread crash or a timeout; broker-backed retry fails
|
||||
by network partition, broker restart, or consumer crash-and-retry
|
||||
(D-062).
|
||||
- The cross-link is one-directional outward (messaging →
|
||||
concurrency) per D-026 extended: messaging references concurrency
|
||||
as the in-process foundation; concurrency does not back-link to
|
||||
messaging.
|
||||
|
||||
## Cross-Link to Errors (P5, cross-link errors/patterns)
|
||||
|
||||
- A poison message is an errors-as-data instance: the failure is
|
||||
captured as a DLQ entry (with `reason`, `redeliveryCount`,
|
||||
`deadLetteredAt`), not swallowed. See `domains/errors/patterns`
|
||||
for the general errors-as-data discipline a DLQ entry
|
||||
instantiates. The DLQ is the async-messaging instance of an
|
||||
error log — observable, auditable, drainable.
|
||||
- The cross-link is one-directional outward (messaging → errors):
|
||||
messaging references errors for the errors-as-data pattern;
|
||||
errors does not back-link to messaging.
|
||||
|
||||
## What Violates Delivery-Semantics Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||||
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
|
||||
| Dedup store with no TTL (memory leak; grows without bound) | P6 Backpressure is Bounded |
|
||||
| Dedup key per-producer (distinct messages in the same window deduped) | P3 Consumers are Idempotent |
|
||||
| Retry with no backoff (hammers the downstream at full rate) | P6 Backpressure is Bounded |
|
||||
| Unbounded retry budget (consumer never progresses past the poison) | P5 Dead-Letter Handling is Defined |
|
||||
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
|
||||
| Poison message retried forever (no poison queue, no immediate DLQ) | P5 Dead-Letter Handling is Defined |
|
||||
| Process-not-idempotent with non-transactional mark (crash in window double-applies) | P3, P4 |
|
||||
| DLQ entry with no reason/audit metadata (uninspectable failure) | P5, `domains/errors/patterns` |
|
||||
| Ordering assumption the broker does not provide (FIFO assumed on standard queue) | P2 Ordering is a Property, Not an Assumption |
|
||||
@@ -0,0 +1,307 @@
|
||||
# Messaging — First Principles
|
||||
|
||||
## 1. The Principles
|
||||
|
||||
### P1. Messages are Contracts
|
||||
A message has an explicit, versioned schema. Producer and consumer
|
||||
agree on shape before exchange; the schema is the boundary, not a
|
||||
guess. A schemaless message — a free-form JSON blob the consumer
|
||||
parses by hope — is a defect: the consumer breaks silently on the
|
||||
next shape change, and the producer has no contract to evolve
|
||||
against. This derives from `C1 Correctness` (the exchange must
|
||||
carry what the parties agreed to) and `C2 Clarity` (the schema
|
||||
makes the boundary obvious to both sides). This is the
|
||||
cross-process expression of the contract discipline that
|
||||
`domains/api/rest` owns for synchronous request/response: the
|
||||
message schema is to async exchange what the API contract is to
|
||||
sync exchange. It is distinct from
|
||||
`domains/concurrency/patterns` Pattern 1 (Message Passing), which
|
||||
owns the *in-process* channel primitive — here the contract spans
|
||||
separate systems and survives network failure (D-062). See
|
||||
`domains/messaging/queues.md` for the queue-flavored application
|
||||
and `domains/messaging/streams.md` for the durable-log-flavored
|
||||
application.
|
||||
|
||||
### P2. Ordering is a Property, Not an Assumption
|
||||
Ordering guarantees — per-partition strict, global, or none — are
|
||||
explicit and documented. "It's FIFO" is a claim that must be backed
|
||||
by the broker's partitioning contract, not an assumption the
|
||||
consumer makes and the broker may not honor. A standard queue
|
||||
delivers in arrival order per receive-node but offers no global
|
||||
ordering across shards; a FIFO queue delivers strict per-message-
|
||||
group order but at a latency cost; a partitioned stream delivers
|
||||
strict per-partition order but only within a partition. Each is a
|
||||
distinct, declared property. This derives from `C1 Correctness`
|
||||
(order is a correctness property — a consumer that assumes order
|
||||
the broker does not provide is wrong) and `C2 Clarity` (the
|
||||
ordering guarantee is documented, not discovered in production).
|
||||
This is distinct from in-process ordering, which
|
||||
`domains/concurrency/patterns` Pattern 1 owns for channels within
|
||||
one program: messaging ordering survives network failure, broker
|
||||
restart, and consumer crash-and-retry — a stronger failure model
|
||||
than thread-local channels (D-062). The `messaging-shared-
|
||||
subscription` chaos anti-pattern breaches this rule: two consumers
|
||||
sharing one subscription break per-consumer ordering because the
|
||||
broker dispatches each message to an arbitrary consumer. See
|
||||
`domains/messaging/streams.md` for the partition-order contract
|
||||
and `domains/messaging/delivery-semantics.md` for the interaction
|
||||
of ordering with the three delivery semantics.
|
||||
|
||||
### P3. Consumers are Idempotent
|
||||
Delivery is at-least-once by default across the network; a
|
||||
consumer deduplicates via idempotency keys or deterministic
|
||||
processing. "Exactly-once" is idempotency plus at-least-once, not a
|
||||
broker guarantee — Jepsen analyses of Kafka, RabbitMQ, and NATS
|
||||
establish that exactly-once claims require independent
|
||||
verification, and the durable engineering practice is to make
|
||||
consumers idempotent under redelivery. A non-idempotent consumer
|
||||
under at-least-once delivery doubles the effect on every retry; a
|
||||
non-idempotent consumer under a claimed exactly-once broker is a
|
||||
bug waiting for the broker's exactly-once invariant to break. This
|
||||
derives from `C1 Correctness`: correctness under redelivery is the
|
||||
contract, not a nice-to-have. This parallels
|
||||
`domains/edge/P5 Edge Operations are Idempotent` (the
|
||||
cross-partition device-and-cache-flavored analog) and is the
|
||||
cross-process instance of the retry-safety discipline that
|
||||
`domains/concurrency/patterns` Pattern 6 (Timeout on Every Block)
|
||||
implies for in-process retry. It is distinct from in-process
|
||||
retry because the redelivery comes from the broker across a
|
||||
network, not from an in-process loop (D-062). See
|
||||
`domains/messaging/delivery-semantics.md` for the idempotency-key
|
||||
dedup-store pattern.
|
||||
|
||||
### P4. Delivery Semantics are Explicit
|
||||
At-least-once / at-most-once / exactly-once is a declared choice
|
||||
per channel, not an emergent behavior. The tradeoff — latency cost,
|
||||
implementation complexity, operational cost — is made consciously
|
||||
and documented. At-most-once is fire-and-forget (low latency, lossy);
|
||||
at-least-once is acked with possible duplication (the default,
|
||||
requires idempotent consumers per P3); exactly-once is at-least-once
|
||||
plus idempotency or a transactional two-phase commit (highest cost,
|
||||
narrowest fit). An unstated semantic is a defect: the consumer
|
||||
guesses, and the guess is wrong under the first failure. This
|
||||
derives from `C1 Correctness` (the chosen semantic must hold) and
|
||||
`C2 Clarity` (the tradeoff is visible to the reader and the
|
||||
operator). This is the cross-process analog of the explicit-failure-
|
||||
mode discipline that `domains/errors/patterns` owns for synchronous
|
||||
code — messaging makes the delivery-mode choice as explicit as an
|
||||
error-handling choice. See `domains/messaging/queues.md` for the
|
||||
three-semantics comparison table and `domains/messaging/delivery-
|
||||
semantics.md` for the correctness properties of each.
|
||||
|
||||
### P5. Dead-Letter Handling is Defined
|
||||
Poison messages — unparseable, repeatedly failing, or exhausting
|
||||
the retry budget — are routed to a dead-letter queue, not retried
|
||||
forever or silently dropped. The DLQ is observable and drainable: an
|
||||
operator can inspect it, replay from it, or discard with audit. An
|
||||
unbounded retry loop is a livelock: the consumer never makes
|
||||
progress past the poison message. A silent drop is a correctness
|
||||
defect: the message vanished with no record. This derives from `C1
|
||||
Correctness` (poison messages must not livelock the consumer or
|
||||
silently disappear) and `C5 Reversibility` (the DLQ is the
|
||||
reversibility mechanism — a dead-lettered message can be reprocessed
|
||||
after the bug is fixed). This is the cross-process analog of the
|
||||
bounded-error discipline that `domains/errors/patterns` owns for
|
||||
synchronous code: a poison message is an error-as-data instance
|
||||
that must be observable and recoverable, not swallowed. It is
|
||||
distinct from in-process error handling because the failure spans a
|
||||
network and a consumer restart (D-062). See
|
||||
`domains/messaging/delivery-semantics.md` for the dead-letter
|
||||
strategy comparison table and the DLQ routing rule pattern.
|
||||
|
||||
### P6. Backpressure is Bounded
|
||||
A slow consumer cannot unbounded-buffer the broker or the
|
||||
producer. Backpressure is explicit: consumer lag is visible,
|
||||
max-unacked is bounded, the retry budget is capped. A consumer
|
||||
that falls behind without a visible signal is a silent backlog —
|
||||
the operator cannot fix what they cannot see, and the broker's
|
||||
memory grows without bound until it fails. This derives from `C1
|
||||
Correctness` (a backlog that grows until OOM is a correctness
|
||||
failure) and `C8 Economy` (the broker's memory is bounded by
|
||||
design, not by luck). This is distinct from
|
||||
`domains/concurrency/P9 Bounded Queues` and
|
||||
`domains/concurrency/patterns` Pattern 5 (Bounded Queue with
|
||||
Backpressure), which own the *in-process* analog: concurrency's
|
||||
bounded queue fails by OOM or thread crash; messaging's bounded
|
||||
backpressure fails by network partition, broker restart, or
|
||||
consumer crash-and-retry (D-062). The Reactive Streams
|
||||
specification (`request(n)`, `onNext` bounded) is the in-process
|
||||
instance; messaging's broker-backed backpressure is the
|
||||
cross-process instance above it. See `domains/messaging/queues.md`
|
||||
for prefetch and max-unacked and `domains/observability/metrics`
|
||||
for consumer-lag as an alert.
|
||||
|
||||
### P7. Partitioning is Intentional
|
||||
The partition key determines ordering, parallelism, and hotspots.
|
||||
Key choice is a design decision with documented rationale, not a
|
||||
default. A key that hashes unevenly creates a hot partition that
|
||||
limits throughput; a key that does not match the ordering need
|
||||
breaks per-key semantics; a key that is too coarse (one partition
|
||||
for the whole topic) serializes all the traffic. The partition
|
||||
count is a capacity bound: too few partitions cap parallelism, too
|
||||
many partition overhead the broker. This derives from `C4 Locality`
|
||||
(ordering and parallelism are co-located with the partition) and
|
||||
`C6 Composability` (the partition is the unit of parallelism and
|
||||
scaling — consumer groups compose from per-partition workers).
|
||||
This is the cross-process analog of the locality discipline that
|
||||
`domains/performance/` owns for generic data-near-compute
|
||||
optimization: performance's locality is algorithmic (data near
|
||||
compute); messaging's locality is partitional (order and
|
||||
parallelism near the partition). See `domains/messaging/streams.md`
|
||||
for the partitioned-log model and consumer-group rebalance
|
||||
strategies.
|
||||
|
||||
### P8. Replay and Retention are Configured
|
||||
Retention windows and replay-from-offset are explicit. A message
|
||||
is not ephemeral by default; the broker is a durable log, not a
|
||||
pipe. A topic with no retention is a fire-and-forget stream — a
|
||||
consumer that falls behind loses data permanently; a topic with
|
||||
infinite retention is an unbounded log — the broker grows until
|
||||
disk exhaustion. Both are defects: the retention window is a
|
||||
declared bound, and replay-from-offset is the mechanism that makes
|
||||
the log durable (re-consumable) rather than ephemeral. This derives
|
||||
from `C5 Reversibility` (a retained message is reversible — it can
|
||||
be re-consumed; an ephemeral message is not) and `C7 Observability`
|
||||
(the durable log is itself an observable record of what happened —
|
||||
the offset is the position from which to replay). This is the
|
||||
foundation for `domains/messaging/streams.md` and the rule that
|
||||
distinguishes a stream from a queue (a queue deletes on ack; a
|
||||
stream retains for replay). See `domains/messaging/pubsub.md` for
|
||||
the pub/sub-vs-stream durability boundary.
|
||||
|
||||
### P9. Schemas Evolve Compatibly
|
||||
Schema changes are backward- and forward-compatible by
|
||||
construction. Breaking changes are versioned migrations, not
|
||||
silent shape edits. A producer that ships a new field the old
|
||||
consumer ignores is backward-compatible; a consumer that handles a
|
||||
missing field the new producer omits is forward-compatible. A
|
||||
silent schema change — the producer renames a field and the
|
||||
consumer parses `undefined` — is a P1 violation (the contract was
|
||||
broken) compounded here as an evolution defect. This derives from
|
||||
`C5 Reversibility` (a schema change is reversible by versioning —
|
||||
the old shape is still readable) and `C6 Composability` (producers
|
||||
and consumers of different versions compose because the schema
|
||||
evolves compatibly). This parallels `domains/data/migrations`
|
||||
(schema migration for databases) and `domains/api/versioning`
|
||||
(API contract evolution): messaging's schema evolution is the
|
||||
async instance of the same compatibility discipline. See
|
||||
`domains/messaging/streams.md` for the stream-schema-evolution
|
||||
angle.
|
||||
|
||||
### P10. Messaging is Observable
|
||||
Consumer lag, DLQ depth, throughput, and consumer-group health are
|
||||
first-class signals. Silent backlog is a bug, not a feature: a
|
||||
consumer that falls behind with no lag metric is invisible until
|
||||
the downstream effect surfaces — by which time the backlog may be
|
||||
hours or days. A DLQ that grows without an alert is a silent
|
||||
correctness defect: poison messages are accumulating and no one
|
||||
knows. This derives from `C7 Observability` (the broker's behavior
|
||||
is visible to the operator) and `C1 Correctness` (backlog
|
||||
detection is a correctness bound — unbounded lag is a failure).
|
||||
This is distinct from `domains/observability/metrics`, which owns
|
||||
*generic* structured metrics; messaging owns the *broker-specific*
|
||||
signals — lag, DLQ depth, partition imbalance, consumer-group
|
||||
rebalance events. See `domains/observability/metrics` for the
|
||||
generic SLI/SLO discipline and `domains/observability/tracing` for
|
||||
cross-partition traces.
|
||||
|
||||
## 2. Core Principle Trace
|
||||
|
||||
Each messaging P-rule derives from one or more core C-rules
|
||||
(C1–C8). The matrix extension lands in P4 of the v0.4 plan; the
|
||||
traces below are authoritative. Messaging is a broad-derivation
|
||||
domain touching 7 of 8 core principles (C1, C2, C4, C5, C6, C7,
|
||||
C8); C3 (Simplicity) is not a primary derivation — messaging is
|
||||
inherently a tradeoff domain where simplicity yields to the
|
||||
correctness of delivery guarantees (a simpler-than-necessary
|
||||
delivery model does not handle the failure cases, per C3's
|
||||
"simpler than necessary is also a violation").
|
||||
|
||||
| P-rule | Core | Why |
|
||||
|--------|------|-----|
|
||||
| P1 Messages are Contracts | C1, C2 | Correctness of the exchange; clarity of the schema boundary |
|
||||
| P2 Ordering is a Property, Not an Assumption | C1, C2 | Correctness of order; clarity of the guarantee |
|
||||
| P3 Consumers are Idempotent | C1 | Correctness under redelivery |
|
||||
| P4 Delivery Semantics are Explicit | C1, C2 | Correctness of the chosen semantic; clarity of the tradeoff |
|
||||
| P5 Dead-Letter Handling is Defined | C1, C5 | Correctness of poison-message routing; reversibility of reprocessing |
|
||||
| P6 Backpressure is Bounded | C1, C8 | Correctness of bounded backlog; economy of broker memory |
|
||||
| P7 Partitioning is Intentional | C4, C6 | Locality of order; composability of parallelism |
|
||||
| P8 Replay and Retention are Configured | C5, C7 | Reversibility of replay; observability of the durable log |
|
||||
| P9 Schemas Evolve Compatibly | C5, C6 | Reversibility of schema changes; composability of versions |
|
||||
| P10 Messaging is Observable | C7, C1 | Observability of lag/DLQ; correctness of backlog detection |
|
||||
|
||||
## 3. What Violates These Principles
|
||||
|
||||
| Violation | Principle Breached |
|
||||
|-----------|-------------------|
|
||||
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
|
||||
| "It's FIFO" with no documented partition contract | P2 Ordering is a Property, Not an Assumption |
|
||||
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
|
||||
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||||
| No dead-letter queue (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
|
||||
| Unbounded retry budget (no cap; slow consumer stalls the partition) | P6 Backpressure is Bounded |
|
||||
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
|
||||
| Ephemeral broker (no retention; no replay) | P8 Replay and Retention are Configured |
|
||||
| Silent schema change (producer breaks consumers with no version bump) | P9 Schemas Evolve Compatibly |
|
||||
| Silent backlog (no lag metric; consumer falls behind invisibly) | P10 Messaging is Observable |
|
||||
| Shared subscription (two consumers share one subscription; per-consumer ordering breaks) | P2 Ordering is a Property, Not an Assumption (P3 compounding) |
|
||||
| Blocking consumer (slow downstream call with no timeout; broker redelivers to the stuck consumer) | P6 Backpressure is Bounded |
|
||||
|
||||
## 4. Relationship to Other Domains
|
||||
|
||||
Messaging systems are the engineering discipline of
|
||||
**cross-process, cross-system asynchronous communication via
|
||||
brokers**. Producer and consumer are separate systems; the broker
|
||||
is the intermediary that brokers delivery, ordering, retention,
|
||||
and failure semantics. The distinguishing constraints are a
|
||||
cross-process failure model (network, not crash), explicit
|
||||
delivery semantics, decoupled producer/consumer lifecycle, and
|
||||
replay-and-retention as a durable-log property. Messaging overlaps
|
||||
`domains/concurrency/` by *subject* (messages, queues,
|
||||
backpressure) but not by *failure model*: per D-062, messaging
|
||||
owns the cross-process/network-failure-model angle; concurrency
|
||||
owns the in-process/crash-failure-model angle. The discriminator
|
||||
is the failure model: concurrency's queue fails by OOM or thread
|
||||
crash; messaging's queue fails by network partition, broker
|
||||
restart, or consumer crash-and-retry. Messaging extends
|
||||
concurrency's bounded-queue/backpressure model to the network-
|
||||
partition regime. Cross-links are one-directional outward (per
|
||||
D-026 extended); no back-link edits to v0.1/v0.2/v0.3 content.
|
||||
|
||||
- `domains/concurrency/patterns` ← P6 (the broker-backed bounded
|
||||
queue is the cross-process analog of the in-process bounded
|
||||
buffer — concurrency Pattern 5 owns in-process; messaging owns
|
||||
the network-failure-model instance above it, per D-062)
|
||||
- `domains/concurrency/patterns` ← P3 (idempotent retry is the
|
||||
cross-process analog of in-process retry-safety — the failure
|
||||
model differs: broker redelivery across a network vs in-process
|
||||
loop)
|
||||
- `domains/observability/metrics` ← P10 (consumer lag and DLQ
|
||||
depth as alerts; observability owns the generic SLI/SLO
|
||||
discipline, messaging owns the broker-specific signals)
|
||||
- `domains/observability/tracing` ← P10 (cross-partition traces
|
||||
for stream processing; observability owns the generic tracing
|
||||
discipline, messaging owns the cross-partition propagation)
|
||||
- `domains/data/schema-design` ← P1, P9 (message schema design
|
||||
and evolution; data owns the generic schema discipline,
|
||||
messaging owns the cross-process message-shape instance)
|
||||
- `domains/errors/patterns` ← P5 (errors as data for message
|
||||
failures; a poison message is an error-as-data instance that must
|
||||
be observable and recoverable, not swallowed)
|
||||
- `domains/edge/iot` ← P4 (the edge↔messaging cross-link
|
||||
resolves bidirectionally here: edge/iot.md links outward to
|
||||
messaging/queues for MQTT QoS parallels to delivery semantics;
|
||||
this first-principles doc acknowledges the back-link — the
|
||||
edge/iot.md → messaging/queues link from P1 now resolves because
|
||||
messaging/queues.md exists, completing the bidirectionality per
|
||||
IDEATE-40)
|
||||
|
||||
> Note: the edge/iot.md → messaging/queues cross-link (MQTT QoS
|
||||
> parallels for delivery semantics) was authored in P1 with a
|
||||
> dangling reference; this P2 authorship of messaging/queues.md
|
||||
> resolves it. The bidirectionality is verified in P5
|
||||
> (ATELIER-114 per IDEATE-40). The cross-link is one-directional
|
||||
> outward from edge/iot.md; this first-principles doc
|
||||
> acknowledges the resolution without editing edge/iot.md (per
|
||||
> D-026 extended — no back-link edits to v0.1/v0.2/v0.3 or to
|
||||
> P1-authored edge content).
|
||||
@@ -0,0 +1,269 @@
|
||||
# Pub/Sub — Derived Rules
|
||||
|
||||
> Derives from `domains/messaging/first-principles.md`. Applies P1
|
||||
> (Messages are Contracts), P2 (Ordering is a Property, Not an
|
||||
> Assumption), P3 (Consumers are Idempotent), and P4 (Delivery
|
||||
> Semantics are Explicit) primarily, with P7 (partitioning), P10
|
||||
> (per-subscription lag). The `messaging-shared-subscription` chaos
|
||||
> anti-pattern lives here (pre-specified in P4 ATELIER-110).
|
||||
> Cross-links `domains/messaging/streams` for the pub/sub-vs-stream
|
||||
> durability boundary and `domains/observability/metrics` for
|
||||
> per-subscription lag.
|
||||
|
||||
## What Pub/Sub Is (P1 Messages are Contracts)
|
||||
|
||||
- Pub/sub is the fan-out primitive: a producer publishes a message
|
||||
to a topic; N independent subscriptions each receive a copy. The
|
||||
message has an explicit, versioned schema (P1): the topic's
|
||||
schema is the contract every subscription agrees to before
|
||||
subscribing. A schemaless topic is a defect — every subscriber
|
||||
breaks silently on the next shape change.
|
||||
- The boundary with queues is the fan-out ratio. A queue is
|
||||
point-to-point (one producer, one consumer); pub/sub is
|
||||
one-to-many (one producer, N consumers, each with its own
|
||||
subscription). The boundary with streams is the durability model
|
||||
— see the cross-link below. Pub/sub is an async concern because
|
||||
producer and consumers are separate systems and the failure model
|
||||
is network, not crash (D-062).
|
||||
- See `domains/messaging/queues.md` for the point-to-point variant
|
||||
and `domains/messaging/streams.md` for the durable-log variant.
|
||||
|
||||
## Topic / Subscription Model (P1, P3, P4)
|
||||
|
||||
- A **topic** is the named stream of messages. A **subscription**
|
||||
is a durable cursor over the topic: each subscription receives
|
||||
every message published after it was created (subject to
|
||||
retention and filtering). The subscription is independent — its
|
||||
ack, redelivery, and DLQ are per-subscription, not shared.
|
||||
- Each subscription is a consumer under at-least-once by default
|
||||
(P4): the broker redelivers until the subscription acks, and the
|
||||
subscriber must be idempotent (P3). A subscription with no
|
||||
idempotency dedup duplicates every redelivered message.
|
||||
- The topic's schema evolves compatibly (P9 — Schemas Evolve
|
||||
Compatibly): a new field the old subscriber ignores is
|
||||
backward-compatible; a renamed field the old subscriber parses
|
||||
as `undefined` is a P1 violation.
|
||||
|
||||
```python
|
||||
# Publish + two independent subscriptions (P1 contract, P3
|
||||
# idempotency, P4 at-least-once per subscription). Each
|
||||
# subscription is an independent durable cursor; acking one does
|
||||
# not affect the other.
|
||||
|
||||
import json, uuid
|
||||
|
||||
# --- Publisher ---
|
||||
def publish(topic, event, broker):
|
||||
# P1: versioned schema on the topic. All subscribers must
|
||||
# understand this schema (or a compatible superset — P9).
|
||||
message = {
|
||||
"schema": "user.signed-up.v1",
|
||||
"id": str(uuid.uuid4()),
|
||||
"idempotencyKey": f"user:{event['userId']}:signup",
|
||||
"payload": event,
|
||||
}
|
||||
broker.publish(topic=topic, body=json.dumps(message))
|
||||
|
||||
# --- Subscription A: welcome-email service ---
|
||||
def subscribe_welcome(broker, dedup_store, send_email):
|
||||
sub = broker.subscribe(topic="users", subscription="welcome-email")
|
||||
for message in sub.receive():
|
||||
# P3: idempotent per subscription. A redelivered message is
|
||||
# a no-op for THIS subscription, not for the others.
|
||||
if dedup_store.seen(("welcome", message["idempotencyKey"])):
|
||||
sub.ack(message)
|
||||
continue
|
||||
send_email(message["payload"]["email"], "Welcome!")
|
||||
dedup_store.mark(("welcome", message["idempotencyKey"]))
|
||||
sub.ack(message)
|
||||
|
||||
# --- Subscription B: analytics-ingest service ---
|
||||
def subscribe_analytics(broker, dedup_store, ingest):
|
||||
# Independent subscription: its own cursor, its own dedup,
|
||||
# its own ack. Welcome-email acking does NOT advance this.
|
||||
sub = broker.subscribe(topic="users", subscription="analytics")
|
||||
for message in sub.receive():
|
||||
if dedup_store.seen(("analytics", message["idempotencyKey"])):
|
||||
sub.ack(message)
|
||||
continue
|
||||
ingest(message["payload"])
|
||||
dedup_store.mark(("analytics", message["idempotencyKey"]))
|
||||
sub.ack(message)
|
||||
```
|
||||
|
||||
- The dedup key is scoped per subscription: `(subscription,
|
||||
idempotencyKey)`. A redelivery to subscription A that was already
|
||||
processed by A is a no-op for A; the same message delivered to
|
||||
subscription B is processed by B independently. Scoping the dedup
|
||||
key by subscription prevents one subscription's dedup from
|
||||
masking another's redelivery.
|
||||
|
||||
## Fan-Out Semantics (P4, P7)
|
||||
|
||||
- Fan-out means every subscription receives every published message
|
||||
(subject to filtering — see below). The broker duplicates the
|
||||
message per subscription; each subscription's delivery is
|
||||
independent. The fan-out ratio is the number of subscriptions; the
|
||||
broker's cost scales with fan-out × message size.
|
||||
- Partitioning (P7) applies to topics that are partitioned for
|
||||
throughput: a partitioned topic delivers per-partition order, and
|
||||
each subscription receives from every partition. A subscription
|
||||
that consumes partitions in parallel must handle per-partition
|
||||
ordering and cross-partition non-ordering (P2 — document the
|
||||
property, do not assume global order).
|
||||
- The delivery semantic is per-subscription (P4): subscription A
|
||||
may be at-least-once, subscription B may be at-most-once (for a
|
||||
loss-tolerant analytics feed). The choice is per subscription,
|
||||
declared, not emergent.
|
||||
|
||||
## Shared vs Independent Subscriptions (P2, P3 — the chaos anti-pattern)
|
||||
|
||||
- An **independent subscription** is one durable cursor per
|
||||
consumer group: each subscription receives every message in
|
||||
topic order (per partition, P2) and acks independently. This is
|
||||
the correct default: per-consumer ordering and per-consumer
|
||||
idempotency hold.
|
||||
- A **shared subscription** is one subscription shared by multiple
|
||||
consumers: the broker dispatches each message to an arbitrary
|
||||
consumer in the shared group. This breaks per-consumer ordering
|
||||
(P2 — consumer A sees message 3 before consumer B sees message
|
||||
1) and complicates idempotency (P3 — the dedup state must be
|
||||
shared across consumers, not per-consumer). This is the
|
||||
`messaging-shared-subscription` chaos anti-pattern
|
||||
(pre-specified in P4 ATELIER-110): the primary breach is P2
|
||||
(ordering); P3 (idempotency) is the compounding consequence.
|
||||
- A shared subscription is correct ONLY when the consumers are
|
||||
stateless, the per-message processing is order-independent, and
|
||||
the dedup store is shared (a shared Redis, a shared DB). A shared
|
||||
subscription for order-dependent or per-consumer-stateful
|
||||
processing is the chaos anti-pattern: the broker's arbitrary
|
||||
dispatch breaks the order the consumer assumes.
|
||||
|
||||
```python
|
||||
# The messaging-shared-subscription chaos anti-pattern (P2
|
||||
# ordering breach, P3 idempotency compounding). Two consumers
|
||||
# share one subscription; the broker dispatches each message to
|
||||
# an arbitrary consumer. Per-consumer ordering breaks; dedup
|
||||
# must be shared (and often is not).
|
||||
|
||||
# BAD — shared subscription, per-consumer dedup (chaos):
|
||||
def shared_subscription_bad(broker, send_email):
|
||||
# Both consumers call subscribe with the SAME subscription
|
||||
# name. The broker round-robins; consumer A gets msg 1, msg 3;
|
||||
# consumer B gets msg 2, msg 4. Per-consumer order is broken.
|
||||
# If each consumer has its OWN dedup store, a redelivery to
|
||||
# the OTHER consumer re-processes (P3 breach).
|
||||
sub = broker.subscribe(topic="users", subscription="shared")
|
||||
for message in sub.receive():
|
||||
# Per-consumer dedup — WRONG. A redelivered message may
|
||||
# land on the other consumer, which has not seen it.
|
||||
if local_dedup.seen(message["idempotencyKey"]): # per-consumer
|
||||
sub.ack(message); continue
|
||||
send_email(message["payload"]["email"], "Welcome!")
|
||||
local_dedup.mark(message["idempotencyKey"])
|
||||
sub.ack(message)
|
||||
|
||||
# CORRECT — independent subscriptions (per-consumer ordering,
|
||||
# per-subscription dedup):
|
||||
def independent_subscriptions_good(broker, send_email):
|
||||
sub = broker.subscribe(topic="users", subscription="welcome-email")
|
||||
for message in sub.receive():
|
||||
if dedup_store.seen(("welcome", message["idempotencyKey"])):
|
||||
sub.ack(message); continue
|
||||
send_email(message["payload"]["email"], "Welcome!")
|
||||
dedup_store.mark(("welcome", message["idempotencyKey"]))
|
||||
sub.ack(message)
|
||||
```
|
||||
|
||||
- If a shared subscription is genuinely required (stateless,
|
||||
order-independent, shared dedup), document the choice and the
|
||||
shared-dedup requirement (P2 — the ordering property is "none
|
||||
across consumers"; P3 — the dedup is shared). The default is
|
||||
independent subscriptions; shared is an opt-in for the narrow case.
|
||||
|
||||
## Filtering (P4, C8 Economy)
|
||||
|
||||
- **Subscription filtering** lets a subscription receive only
|
||||
messages matching a filter (e.g., `event.type == "order"`).
|
||||
Filtering at the broker saves bandwidth (C8 — the subscriber
|
||||
does not receive and discard) and reduces subscriber load.
|
||||
- **Server-side filtering** (broker evaluates the filter before
|
||||
delivery) is more efficient than **client-side filtering**
|
||||
(subscriber receives and discards). Server-side filtering is the
|
||||
default where the broker supports it (GCP Pub/Sub, SNS filtering,
|
||||
NATS subject filtering); client-side is the fallback.
|
||||
- A filter that is too broad wastes bandwidth; a filter that is
|
||||
too narrow drops messages the subscriber needed. The filter is
|
||||
a P1 (contract) and P4 (semantic) decision: the subscription's
|
||||
filter is part of its declared contract.
|
||||
|
||||
## Ordering Across Subscriptions (P2)
|
||||
|
||||
- A topic with per-partition ordering delivers per-partition order
|
||||
to each subscription. Across subscriptions, there is no ordering
|
||||
guarantee: subscription A may ack message 3 while subscription B
|
||||
is still on message 1. This is correct and expected — each
|
||||
subscription is independent.
|
||||
- Within a subscription, ordering holds per partition (P2 — the
|
||||
documented property). A subscription that processes partitions
|
||||
in parallel must not assume cross-partition order. A subscription
|
||||
that needs global order must use a single partition (sacrificing
|
||||
parallelism, P7) or an external sequencing mechanism.
|
||||
- The `messaging-shared-subscription` anti-pattern breaks even
|
||||
per-partition order within a subscription: the broker's arbitrary
|
||||
dispatch to consumers in the shared group breaks the per-
|
||||
partition sequence each consumer sees.
|
||||
|
||||
## Pub/Sub vs Stream — The Durability Boundary (cross-link messaging/streams)
|
||||
|
||||
- Pub/sub and streams are both fan-out or one-to-many primitives,
|
||||
but their durability model differs. Pub/sub is a
|
||||
**push-to-subscription** model: each subscription is a cursor,
|
||||
retention is short (the subscription's unacked window), and
|
||||
replay is limited to the unacked messages. A subscription that
|
||||
falls behind beyond the retention window loses messages
|
||||
permanently.
|
||||
- A stream is a **durable-log** model: messages are retained by
|
||||
the log for a configured window (P8 — Replay and Retention are
|
||||
Configured), and any consumer group can replay from any offset
|
||||
within the window. A stream consumer that falls behind can
|
||||
catch up by replaying; a pub/sub subscription that falls behind
|
||||
beyond retention cannot.
|
||||
- The choice is the durability requirement: if the consumer must
|
||||
be able to replay (reprocessing, backfill, new consumer starting
|
||||
from the beginning), use a stream. If the consumer only needs
|
||||
the live feed (and can tolerate loss on a long fall-behind),
|
||||
pub/sub is lighter. See `domains/messaging/streams.md` for the
|
||||
durable-log model, offsets, and consumer groups.
|
||||
|
||||
## Observability — Per-Subscription Lag (P10)
|
||||
|
||||
- Per-subscription lag (messages published minus messages acked
|
||||
for each subscription, or the age of the oldest unacked message
|
||||
per subscription) is the primary pub/sub health signal. Each
|
||||
subscription has its own lag — a fast subscription and a slow
|
||||
subscription on the same topic are independent signals.
|
||||
- A subscription whose lag grows beyond the retention window is a
|
||||
silent data-loss risk: the broker will drop the oldest messages,
|
||||
and the subscription will never see them. Alert on lag relative
|
||||
to retention — lag approaching retention is the loss threshold.
|
||||
- Wire per-subscription lag to `domains/observability/metrics` as
|
||||
an SLI per subscription. A topic with N subscriptions has N lag
|
||||
metrics; a single aggregate hides the slow one. See
|
||||
`domains/observability/metrics` for the generic SLI/SLO
|
||||
discipline.
|
||||
|
||||
## What Violates Pub/Sub Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Shared subscription for order-dependent processing (broker dispatch breaks per-consumer order) | P2 Ordering is a Property, Not an Assumption |
|
||||
| Shared subscription with per-consumer dedup (redelivery to the other consumer re-processes) | P3 Consumers are Idempotent |
|
||||
| Schemaless topic (no versioned contract; subscribers parse by guess) | P1 Messages are Contracts |
|
||||
| Subscription with no idempotency dedup (redelivered message duplicates the effect) | P3 Consumers are Idempotent |
|
||||
| Subscription whose lag approaches retention (silent data loss) | P10, `domains/observability/metrics` |
|
||||
| Unstated delivery semantic per subscription (at-least-once vs at-most-once guessed) | P4 Delivery Semantics are Explicit |
|
||||
| Filter that is too narrow (drops messages the subscriber needed) | P1, P4 |
|
||||
| Partitioned topic with no documented per-partition ordering contract | P2 Ordering is a Property, Not an Assumption |
|
||||
| No per-subscription lag metric (slow subscription invisible) | P10 Messaging is Observable |
|
||||
| Cross-partition order assumption within a subscription (no global order guarantee) | P2, P7 |
|
||||
@@ -0,0 +1,318 @@
|
||||
# Queues — Derived Rules
|
||||
|
||||
> Derives from `domains/messaging/first-principles.md`. Applies P1
|
||||
> (Messages are Contracts), P3 (Consumers are Idempotent), P4
|
||||
> (Delivery Semantics are Explicit), P5 (Dead-Letter Handling is
|
||||
> Defined), and P6 (Backpressure is Bounded) primarily, with P2
|
||||
> (ordering), P7 (partitioning), and P10 (observable lag). For the
|
||||
> at-least-once / at-most-once / exactly-once decision, see the
|
||||
> comparison table below. Cross-links `domains/concurrency/patterns`
|
||||
> for the in-process bounded-queue analog and
|
||||
> `domains/observability/metrics` for consumer lag.
|
||||
|
||||
## What a Queue Is (P1 Messages are Contracts)
|
||||
|
||||
- A queue is a point-to-point async delivery primitive. A producer
|
||||
enqueues a message; exactly one consumer dequeues and processes
|
||||
it. The message has an explicit, versioned schema (P1): the
|
||||
producer and consumer agree on shape before exchange, and the
|
||||
schema is the boundary — a schemaless message is a defect (the
|
||||
consumer breaks silently on the next shape change).
|
||||
- The boundary is per D-062: messaging owns the cross-process /
|
||||
network-failure-model angle; concurrency owns the in-process
|
||||
analog. A queue is a messaging concern because producer and
|
||||
consumer are separate systems, the broker is the intermediary,
|
||||
and the failure model is network (the message can be lost,
|
||||
duplicated, reordered, or delayed by the broker or the network,
|
||||
not by a thread crash). The in-process bounded buffer
|
||||
(`domains/concurrency/patterns` Pattern 5) is the analog below
|
||||
this boundary — it fails by OOM; a broker-backed queue fails by
|
||||
partition, broker restart, or consumer crash-and-retry.
|
||||
- See `domains/messaging/pubsub.md` for the fan-out (one-to-many)
|
||||
variant and `domains/messaging/streams.md` for the durable-log
|
||||
(replay-from-offset) variant. A queue deletes on ack; a stream
|
||||
retains for replay — the durability boundary is the
|
||||
distinguishing trait.
|
||||
|
||||
## Producer / Consumer Model (P1, P3)
|
||||
|
||||
- The producer enqueues a message with an idempotency key (P3).
|
||||
The consumer dequeues, processes, and acks. If the consumer
|
||||
crashes before acking, the broker redelivers; the idempotency
|
||||
key makes the redelivery safe (the consumer dedups, not the
|
||||
broker).
|
||||
- The idempotency key is per-message, not per-producer or
|
||||
per-session. A consumer that dedups by producer alone will drop
|
||||
distinct messages issued in the same window. Use a UUID per
|
||||
message, or a deterministic key derived from the message content
|
||||
(e.g., `(entity, operation, version)`).
|
||||
|
||||
```python
|
||||
# Producer/consumer pair with idempotency key (P1 contract, P3
|
||||
# idempotency). The producer tags each message with a versioned
|
||||
# schema and a unique idempotency key; the consumer dedups by the
|
||||
# key so a redelivered message is processed once (P3).
|
||||
|
||||
# --- Producer ---
|
||||
import json, uuid
|
||||
|
||||
def enqueue(order, broker):
|
||||
# P1: versioned schema. The message carries its schema version
|
||||
# so the consumer can route by shape (P9 evolution discipline).
|
||||
message = {
|
||||
"schema": "order.created.v1",
|
||||
"id": str(uuid.uuid4()),
|
||||
"idempotencyKey": f"order:{order['id']}:{order['version']}",
|
||||
"payload": order,
|
||||
}
|
||||
broker.send(queue="orders", body=json.dumps(message))
|
||||
# At-least-once by default (P4): the broker acks the send; the
|
||||
# consumer may see this message more than once under retry.
|
||||
|
||||
# --- Consumer ---
|
||||
def consume(broker, dedup_store, process_order):
|
||||
for message in broker.receive(queue="orders"):
|
||||
# P3: idempotent consumer. Dedup by idempotency key before
|
||||
# processing; a redelivered message is a no-op, not a
|
||||
# double-apply.
|
||||
if dedup_store.seen(message["idempotencyKey"]):
|
||||
broker.ack(message) # already processed; skip
|
||||
continue
|
||||
try:
|
||||
process_order(message["payload"])
|
||||
dedup_store.mark(message["idempotencyKey"])
|
||||
broker.ack(message) # success; broker drops it
|
||||
except Exception:
|
||||
broker.nack(message) # redeliver (at-least-once, P4)
|
||||
```
|
||||
|
||||
- The dedup store is bounded (P6 — Backpressure is Bounded): a
|
||||
dedup store that grows without bound is a memory leak. Use a TTL
|
||||
window longer than the broker's max-redelivery window, or a
|
||||
bounded LRU. See `domains/messaging/delivery-semantics.md` for
|
||||
the full idempotency-key dedup-store pattern.
|
||||
|
||||
## Ack / Nack (P4 Delivery Semantics are Explicit)
|
||||
|
||||
- **Ack** tells the broker the message was processed; the broker
|
||||
drops it. **Nack** (negative ack) tells the broker the
|
||||
processing failed; the broker redelivers (at-least-once) or
|
||||
routes to a DLQ (after the retry budget — P5).
|
||||
- A consumer that neither acks nor nacks within the visibility
|
||||
timeout causes the broker to redeliver (the broker assumes the
|
||||
consumer died). This is the at-least-once default: the broker
|
||||
prefers duplication to loss.
|
||||
- The semantic is explicit (P4): at-least-once is the default; the
|
||||
consumer must be idempotent (P3). At-most-once is fire-and-forget
|
||||
(no ack; the broker drops on send) — lossy but lowest latency.
|
||||
Exactly-once is at-least-once plus idempotency, or a
|
||||
transactional two-phase commit — see the comparison table below.
|
||||
|
||||
## Visibility Timeouts and Redelivery (P4, P5)
|
||||
|
||||
- The visibility timeout is the window the broker hides a message
|
||||
after delivery, waiting for the ack. If the consumer does not
|
||||
ack within the window, the broker makes the message visible
|
||||
again and redelivers it (to the same consumer or another). This
|
||||
is the at-least-once mechanism: the broker assumes a
|
||||
no-ack-in-time consumer is dead.
|
||||
- The timeout must be longer than the processing time, or the
|
||||
broker redelivers a message the consumer is still processing —
|
||||
causing duplicate processing (which P3 idempotency makes safe,
|
||||
but which wastes resources). A timeout shorter than processing
|
||||
time is a P6 (backpressure) smell: the consumer is too slow for
|
||||
the configured timeout.
|
||||
- Redelivery has a budget (P5): after N redeliveries or a TTL, the
|
||||
message routes to the DLQ. An unbounded retry budget is the
|
||||
`messaging-unbounded-retry` chaos anti-pattern: the consumer
|
||||
never makes progress past the poison message.
|
||||
|
||||
## FIFO vs Standard Queues (P2 Ordering is a Property, Not an Assumption)
|
||||
|
||||
- A **standard queue** delivers in arrival order per receive-node
|
||||
but offers no global ordering across shards, no per-message-
|
||||
group ordering, and may redeliver out of order under retry. It
|
||||
is the high-throughput default; ordering is *not* guaranteed
|
||||
(P2: the ordering property is "none" — explicitly documented).
|
||||
- A **FIFO queue** delivers strict per-message-group order: all
|
||||
messages with the same group ID are delivered to one consumer
|
||||
in send order. The cost is throughput (FIFO queues cap at lower
|
||||
TPS) and latency (the broker must sequence per group). The
|
||||
ordering property is "per-group strict" — explicitly documented
|
||||
(P2).
|
||||
- The choice is a P2 decision (which ordering guarantee) and a C8
|
||||
decision (throughput cost). A consumer that assumes FIFO on a
|
||||
standard queue is a P2 violation: the broker does not provide
|
||||
the guarantee the consumer assumes. Document the property; do
|
||||
not assume it.
|
||||
|
||||
## Prefetch and Concurrency (P6 Backpressure is Bounded)
|
||||
|
||||
- **Prefetch** (or max-unacked) bounds how many messages the
|
||||
broker delivers to one consumer without an ack. A prefetch of 1
|
||||
is strict stop-and-wait (lowest throughput, tightest backpressure);
|
||||
a prefetch of N allows the consumer to process N in flight
|
||||
(higher throughput, more memory). An unbounded prefetch is a P6
|
||||
violation: the broker floods the consumer's memory.
|
||||
- **Consumer concurrency** is the number of parallel workers
|
||||
processing from the queue. More workers increase throughput up to
|
||||
the downstream's limit; beyond that, the workers stall the
|
||||
downstream (P6 — the backpressure propagates to the
|
||||
downstream, not the broker).
|
||||
- The prefetch × concurrency product is the in-flight cap. Declare
|
||||
it (P6): an undeclared cap is a defect — the consumer either
|
||||
underutilizes the broker (prefetch too low) or OOMs under load
|
||||
(prefetch too high). This is the cross-process analog of
|
||||
`domains/concurrency/patterns` Pattern 5 (bounded queue with
|
||||
backpressure): concurrency owns the in-process analog;
|
||||
messaging owns the broker-backed instance.
|
||||
|
||||
## Long Polling (P6, C8 Economy)
|
||||
|
||||
- Long polling (or `ReceiveMessage` with a wait-time-seconds)
|
||||
holds the receive request open until a message arrives or the
|
||||
wait expires. This reduces empty-receive round trips (C8
|
||||
economy of the constrained link) and reduces latency-to-first-
|
||||
message (the message is delivered when it arrives, not on the
|
||||
next poll cycle).
|
||||
- Long polling is the default for low-throughput queues: short
|
||||
polling burns CPU on empty receives; long polling waits for
|
||||
work. For high-throughput queues, the broker is usually full
|
||||
enough that long polling adds no latency; for low-throughput
|
||||
queues, long polling is the difference between 20ms and 20s
|
||||
latency-to-first-message.
|
||||
|
||||
## Redelivery + DLQ Flow (P5 Dead-Letter Handling is Defined)
|
||||
|
||||
- A poison message (unparseable, repeatedly failing, or exhausting
|
||||
the retry budget) routes to the dead-letter queue. The DLQ is
|
||||
observable (P10 — DLQ depth is an alert) and drainable (an
|
||||
operator can inspect, replay, or discard with audit).
|
||||
- The retry budget is bounded (P6): N redeliveries, or a TTL with
|
||||
exponential backoff. After the budget is exhausted, the message
|
||||
is moved to the DLQ, not retried forever. An unbounded retry is
|
||||
the `messaging-unbounded-retry` chaos anti-pattern (P5 breach).
|
||||
- The DLQ routing rule is a redelivery-count or TTL threshold
|
||||
plus a target queue. See `domains/messaging/delivery-semantics.md`
|
||||
for the dead-letter strategy comparison table.
|
||||
|
||||
```python
|
||||
# Redelivery + DLQ flow (P5 dead-letter handling, P6 bounded
|
||||
# retry budget). The consumer tracks redelivery count; after the
|
||||
# budget, the message routes to the DLQ. The DLQ is observable
|
||||
# (P10 — depth is an alert) and drainable.
|
||||
|
||||
MAX_REDELIVERIES = 5
|
||||
DLQ = "orders-dlq"
|
||||
|
||||
def consume_with_dlq(broker, dedup_store, process_order):
|
||||
for message in broker.receive(queue="orders"):
|
||||
# P3 idempotency: a redelivered, already-processed message
|
||||
# is acked and skipped (not re-processed, not DLQ'd).
|
||||
if dedup_store.seen(message["idempotencyKey"]):
|
||||
broker.ack(message)
|
||||
continue
|
||||
try:
|
||||
process_order(message["payload"])
|
||||
dedup_store.mark(message["idempotencyKey"])
|
||||
broker.ack(message)
|
||||
except Exception as exc:
|
||||
# P5: bounded retry budget. After MAX_REDELIVERIES,
|
||||
# route to DLQ — do NOT retry forever.
|
||||
count = message.get("redeliveryCount", 0) + 1
|
||||
if count >= MAX_REDELIVERIES:
|
||||
broker.send(DLQ, body=json.dumps({
|
||||
"original": message,
|
||||
"reason": str(exc),
|
||||
"deadLetteredAt": now_iso(),
|
||||
"redeliveryCount": count,
|
||||
}))
|
||||
broker.ack(message) # remove from the origin queue
|
||||
# P10: the DLQ depth must alert. A DLQ that grows
|
||||
# with no alert is a silent correctness defect.
|
||||
else:
|
||||
# Nack with backoff: the broker redelivers after a
|
||||
# delay. The backoff caps the retry rate (P6).
|
||||
broker.nack(message, delay=exponential_backoff(count))
|
||||
```
|
||||
|
||||
- The `deadLetteredAt` and `reason` fields make the DLQ entry
|
||||
observable and auditable: an operator inspecting the DLQ sees
|
||||
why each message was dead-lettered and when. See
|
||||
`domains/errors/patterns` for the errors-as-data discipline the
|
||||
DLQ entry follows.
|
||||
|
||||
## Delivery Semantics Comparison (D-069)
|
||||
|
||||
| Semantic | Guarantee | Latency Cost | Implementation Cost | When It Fits |
|
||||
|----------|-----------|-------------|---------------------|--------------|
|
||||
| **At-most-once** | A message is delivered 0 or 1 times; loss is possible, duplication is not | Lowest (no ack; fire-and-forget) | Lowest (no ack, no dedup) | Telemetry where a dropped sample is acceptable; high-throughput metrics; MQTT QoS 0; logs where a lost line is tolerable. Never for billing, orders, or any side-effecting operation. |
|
||||
| **At-least-once** | A message is delivered 1 or more times; duplication is possible, loss is not | Low (one ack round-trip) | Medium (consumer must be idempotent — P3; dedup store required) | The default for side-effecting operations: orders, payments, commands. The consumer dedups via idempotency keys (P3); the broker guarantees delivery. Fits the vast majority of broker-backed queues (SQS standard, RabbitMQ ack, MQTT QoS 1). |
|
||||
| **Exactly-once** | A message is delivered exactly 1 time; no loss, no duplication | Highest (two-phase commit or transactional producer+consumer) | Highest (requires transactions, a transactional producer, and a transactional consumer — or at-least-once plus idempotency, which collapses to at-least-once with dedup) | Rare. Kafka transactions (consume-process-produce in one transaction); MQTT QoS 2 (four-step handshake). In practice, "exactly-once" is usually at-least-once plus idempotency (P3) — the broker does not guarantee it; the consumer enforces it. Jepsen analyses verify broker claims. |
|
||||
|
||||
- The default for side-effecting operations is **at-least-once with
|
||||
idempotent consumers** (P3). At-most-once is for loss-tolerant
|
||||
telemetry. Exactly-once is reserved for the narrow case where
|
||||
the consume-process-produce loop must be transactional (Kafka
|
||||
transactions) — and even then, the consumer should be idempotent
|
||||
as defense-in-depth.
|
||||
- The latency cost column is the C8 tradeoff: at-most-once is
|
||||
cheapest, exactly-once is most expensive. The implementation
|
||||
cost column is the C1/C3 tradeoff: at-most-once is simplest,
|
||||
exactly-once is most complex (and most fragile — a transactional
|
||||
consumer that partially fails is a bug source). The "when it
|
||||
fits" column is the P4 decision: declare the semantic per
|
||||
channel, do not let it emerge.
|
||||
- This is the decision matrix required by D-069 for queues; see
|
||||
`domains/messaging/delivery-semantics.md` for the correctness
|
||||
properties of each semantic and the dead-letter strategy
|
||||
comparison.
|
||||
|
||||
## Observability — Consumer Lag (P10 Messaging is Observable)
|
||||
|
||||
- Consumer lag (messages enqueued minus messages acked, or the
|
||||
age of the oldest unacked message) is the primary queue health
|
||||
signal. A lag that grows without bound is a P6 violation (the
|
||||
consumer is slower than the producer) and a P10 violation if it
|
||||
is not alerted.
|
||||
- DLQ depth is the secondary signal: a DLQ that grows is a P5
|
||||
signal (poison messages are accumulating) and a P10 signal if
|
||||
not alerted. Wire both to `domains/observability/metrics` as
|
||||
SLIs with SLOs (e.g., lag < 1000 messages, DLQ depth < 10).
|
||||
- A queue with no lag metric is operating blind (P10 violation):
|
||||
the operator cannot see the consumer falling behind until the
|
||||
downstream effect surfaces.
|
||||
|
||||
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
|
||||
|
||||
- The broker-backed queue is the cross-process analog of the
|
||||
in-process bounded buffer. `domains/concurrency/patterns`
|
||||
Pattern 5 (Bounded Queue with Backpressure) owns the in-process
|
||||
instance ("producer is blocked or signaled" within one program);
|
||||
messaging owns the broker-backed instance above it (the producer
|
||||
is the broker's enqueue, the consumer is the broker's dequeue, the
|
||||
backpressure is the prefetch cap and the lag signal). The
|
||||
failure model differs: in-process fails by OOM; broker-backed
|
||||
fails by network partition, broker restart, or consumer
|
||||
crash-and-retry (D-062).
|
||||
- The cross-link is one-directional outward (messaging →
|
||||
concurrency) per D-026 extended: messaging references concurrency
|
||||
as the in-process foundation; concurrency does not back-link to
|
||||
messaging.
|
||||
|
||||
## What Violates Queue Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
|
||||
| Non-idempotent consumer under at-least-once delivery (redelivery doubles the effect) | P3 Consumers are Idempotent |
|
||||
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
|
||||
| No DLQ (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
|
||||
| Unbounded prefetch (broker floods consumer memory) | P6 Backpressure is Bounded |
|
||||
| Unbounded retry budget (no cap; consumer never progresses past the poison) | P5, P6 |
|
||||
| Consumer that assumes FIFO on a standard queue (ordering not guaranteed) | P2 Ordering is a Property, Not an Assumption |
|
||||
| Visibility timeout shorter than processing time (redeliver while still processing) | P4, P6 |
|
||||
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
|
||||
| No consumer-lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
|
||||
| Dedup store that grows without bound (memory leak) | P6 Backpressure is Bounded |
|
||||
| Default prefetch with no rationale (underutilizes or OOMs) | P6, `domains/concurrency/patterns` |
|
||||
@@ -0,0 +1,329 @@
|
||||
# Streams — Derived Rules
|
||||
|
||||
> Derives from `domains/messaging/first-principles.md`. Applies P8
|
||||
> (Replay and Retention are Configured) primarily, with P2
|
||||
> (per-partition ordering), P3 (idempotent consumers), P4 (exactly-
|
||||
> once via transactions), P7 (partitioning), and P10 (observable
|
||||
> consumer-group health). For the Kafka / Kinesis / Pulsar /
|
||||
> NATS JetStream decision, see the stream-platform comparison
|
||||
> table below. For the consumer-group rebalance strategy choice,
|
||||
> see the rebalance enumeration below (IDEATE-41). Cross-links
|
||||
> `domains/messaging/delivery-semantics` for exactly-once via
|
||||
> transactions, `domains/data/schema-design` for stream schema,
|
||||
> and `domains/observability/tracing` for cross-partition traces.
|
||||
|
||||
## What a Stream Is (P8 Replay and Retention are Configured)
|
||||
|
||||
- A stream is a durable-log messaging primitive. Messages are
|
||||
appended to a partitioned, replicated log; consumers read from an
|
||||
offset and advance at their own pace. The log is retained for a
|
||||
configured window (P8) — a stream is a durable log, not a pipe.
|
||||
A consumer that falls behind can catch up by replaying from an
|
||||
earlier offset; a consumer that starts fresh can replay from the
|
||||
beginning (within retention).
|
||||
- The boundary with queues and pub/sub is the durability model. A
|
||||
queue deletes on ack; a pub/sub subscription retains only its
|
||||
unacked window; a stream retains the whole log for the configured
|
||||
retention. This makes a stream replayable (P8 — the
|
||||
reversibility mechanism) and observable as a record (P10 — the
|
||||
log is itself an audit of what happened). See
|
||||
`domains/messaging/pubsub.md` for the pub/sub-vs-stream
|
||||
durability boundary discussion.
|
||||
- The boundary with concurrency is per D-062: messaging owns the
|
||||
cross-process/network-failure-model angle. A stream is a
|
||||
messaging concern because the log spans brokers and consumers
|
||||
across a network, and the failure model is partition, broker
|
||||
restart, or consumer crash-and-retry — not in-process OOM or
|
||||
thread crash.
|
||||
|
||||
## Partitioned Log Model (P2, P7)
|
||||
|
||||
- A stream is partitioned for throughput and parallelism. Each
|
||||
partition is an ordered, append-only log; messages within a
|
||||
partition are strictly ordered (P2 — per-partition strict order
|
||||
is the documented property). Across partitions, there is no
|
||||
ordering guarantee: partition 0 and partition 1 are independent
|
||||
logs.
|
||||
- The partition key (P7 — Partitioning is Intentional) determines
|
||||
which partition a message lands on. A key that hashes evenly
|
||||
spreads load; a key that matches the per-entity ordering need
|
||||
(e.g., `userId` for user events) keeps a user's events on one
|
||||
partition in order; a key that is too coarse (one partition for
|
||||
the whole topic) serializes all traffic.
|
||||
- The partition count is a capacity bound: it caps the parallelism
|
||||
(one consumer per partition per consumer group) and the
|
||||
throughput (each partition has a write-throughput limit). Too
|
||||
few partitions cap parallelism; too many partition overhead the
|
||||
broker (file handles, replication, rebalance cost). The choice
|
||||
is documented (P7), not defaulted.
|
||||
|
||||
## Offsets (P2, P8)
|
||||
|
||||
- An offset is a consumer's position in a partition. The consumer
|
||||
reads from its last committed offset; acking (committing the
|
||||
offset) advances it. A consumer that crashes before committing
|
||||
re-reads from the last committed offset (at-least-once by
|
||||
default, P4) — the consumer must be idempotent (P3).
|
||||
- The offset is per-partition (P2): each partition has its own
|
||||
position, and the consumer commits them independently (or
|
||||
atomically across partitions in a transaction — see below).
|
||||
- Replay (P8) is resetting the offset backward: a consumer can
|
||||
replay from the beginning of retention, from a timestamp, or
|
||||
from a specific offset. This is the durable-log property that
|
||||
distinguishes a stream from a queue.
|
||||
|
||||
## Consumer Groups (P3, P7, P10)
|
||||
|
||||
- A consumer group is a set of consumers sharing the stream's
|
||||
partitions: each partition is assigned to exactly one consumer
|
||||
in the group. The group is the unit of parallelism and the unit
|
||||
of offset tracking. Within a group, each consumer handles its
|
||||
assigned partitions; across groups, each group independently
|
||||
reads the whole stream (the pub/sub fan-out property, per
|
||||
subscription/group).
|
||||
- A consumer in the group is idempotent (P3): under at-least-once
|
||||
(the default), a redelivery after a crash-and-retry re-processes
|
||||
messages. The consumer dedups by idempotency key, or processes
|
||||
deterministically (e.g., a stateful aggregation that overwrites
|
||||
with the latest value).
|
||||
- The consumer group's health is observable (P10): per-partition
|
||||
lag (offset of the consumer vs the log's head), the group's
|
||||
consumption rate, and rebalance events are first-class signals.
|
||||
A group whose lag grows without bound is a P6 (backpressure)
|
||||
smell and a P10 (observability) violation if not alerted.
|
||||
|
||||
```python
|
||||
# Consumer-group reading from offsets (P2 per-partition order,
|
||||
# P3 idempotent under at-least-once, P7 partition assignment,
|
||||
# P8 replay from offset). Each consumer in the group handles its
|
||||
# assigned partitions; the group commits offsets atomically or
|
||||
# per-partition.
|
||||
|
||||
def consume_stream(stream, group, dedup_store, process_event):
|
||||
# Assign partitions to this consumer by the group's
|
||||
# rebalance strategy (see the enumeration below).
|
||||
for partition in stream.assigned_partitions(group, consumer=ME):
|
||||
# Read from the last committed offset (P8 — replay by
|
||||
# resetting this offset).
|
||||
offset = stream.committed_offset(group, partition)
|
||||
for message in stream.read(partition, from_offset=offset):
|
||||
# P3: idempotent under at-least-once. A redelivery
|
||||
# after a crash-and-retry re-processes; dedup by key.
|
||||
if dedup_store.seen(message["idempotencyKey"]):
|
||||
stream.commit(group, partition, message["offset"])
|
||||
continue
|
||||
process_event(message["payload"])
|
||||
dedup_store.mark(message["idempotencyKey"])
|
||||
# Commit the offset to advance (P8 — the position is
|
||||
# the replay pointer).
|
||||
stream.commit(group, partition, message["offset"])
|
||||
```
|
||||
|
||||
- The commit-after-process order gives at-least-once (a crash
|
||||
before commit re-reads); the commit-before-process order gives
|
||||
at-most-once (a crash after commit loses the unprocessed
|
||||
message). The default is at-least-once with idempotent consumers
|
||||
(P3, P4).
|
||||
|
||||
## Consumer-Group Rebalance Strategies (IDEATE-41, ATELIER-100 refinement)
|
||||
|
||||
When a consumer joins or leaves the group, the broker must
|
||||
reassign partitions. The rebalance strategy determines the cost
|
||||
and the use-case fit. This enumeration parallels the v0.3
|
||||
IDEATE-30 drift-type enumeration (each strategy with its
|
||||
stop-the-world cost and use-case fit).
|
||||
|
||||
| Strategy | Mechanism | Partition Stop-the-World Cost | Use-Case Fit |
|
||||
|----------|-----------|-------------------------------|--------------|
|
||||
| **Eager rebalance** (stop-the-world) | Every consumer in the group revokes ALL its partitions, the broker reassigns the full partition set, then consumers resume. Every rebalance pauses the whole group. | High — every partition pauses for every rebalance; the whole group stops processing during the revocation+reassignment window. Throughput drops to zero during rebalance. | Simple brokers, small groups, or rarely-rebalancing groups where the simplicity of full revocation outweighs the pause cost. Kafka's legacy protocol (pre-2.4). Avoid for large groups or frequent scale events. |
|
||||
| **Sticky (incremental cooperative) rebalance** | The broker reassigns only the partitions that must move (the joining/leaving consumer's share); existing partitions stay assigned. The rebalance is incremental and cooperative — no full revocation. | Low — only the moving partitions pause; the rest of the group continues processing. The pause is proportional to the changed partition count, not the total. | The default for large groups, frequent scale events, and rolling deploys. Kafka's CooperativeStickyAssignor (2.4+), Pulsar, NATS JetStream. Prefer for any group where a full stop-the-world on every deploy is unacceptable. |
|
||||
| **Cooperative (no-revoke) rebalance** | A subset of sticky where no partition is revoked unless the consumer leaves; only additions are incremental. The strictest minimization of stop-the-world. | Lowest — only added partitions pause; existing assignments are untouched. | Groups where partition assignment is append-only (consumers join but rarely leave). Useful for long-lived consumers with incremental scaling. |
|
||||
|
||||
- The default for any non-trivial group is **sticky/cooperative**:
|
||||
a rolling deploy that triggers an eager rebalance pauses the
|
||||
whole group on every pod restart, which is unacceptable at
|
||||
scale. The eager strategy is a legacy default that survives
|
||||
because it is simple; prefer sticky where the broker supports
|
||||
it.
|
||||
- The "partition stop-the-world cost" column is the P6
|
||||
(backpressure) angle: a full stop-the-world during rebalance
|
||||
causes lag to spike (the consumer is paused, the producer is
|
||||
not). Sticky rebalance bounds the spike to the moving partitions.
|
||||
- A rebalance that pauses without a lag alert is a P10 violation:
|
||||
the operator cannot see the rebalance-induced lag. Wire rebalance
|
||||
events to `domains/observability/metrics` as an event signal.
|
||||
|
||||
## Replay and Retention Windows (P8, C5 Reversibility)
|
||||
|
||||
- Retention is the configured window the log keeps messages: time-
|
||||
based (e.g., 7 days), size-based (e.g., 10 GB per partition), or
|
||||
compacted (keep the latest value per key — a changelog). A
|
||||
stream with no retention is a pipe, not a log (P8 violation); a
|
||||
stream with infinite retention grows until disk exhaustion (P6
|
||||
violation — backpressure on the broker).
|
||||
- Replay (P8) is resetting a consumer's offset to re-read from
|
||||
within the retention window. Use cases: reprocessing after a
|
||||
consumer bug (replay from the timestamp of the buggy deploy),
|
||||
backfilling a new consumer (replay from the beginning), or
|
||||
reindexing (replay to rebuild a derived store).
|
||||
- Compacted topics (Kafka log-compaction, Pulsar compaction) keep
|
||||
the latest value per key and discard older values for the same
|
||||
key. This turns the log into a changelog — a durable
|
||||
materialized view that replays to the current state. Compaction
|
||||
is a P8 (retention) and C5 (reversibility) mechanism: the log
|
||||
retains the current state per key and is replayable to it.
|
||||
|
||||
## Stream Processing (P2, P3, P4)
|
||||
|
||||
- Stream processing is computing over the stream as it arrives:
|
||||
windowing (tumbling, sliding, session windows), joins (stream-
|
||||
stream, stream-table), aggregations (count, sum, per-key
|
||||
windows), and stateful transformations. The processing is
|
||||
per-partition ordered (P2 — a window over a partition is
|
||||
deterministic; a window across partitions is not unless the
|
||||
window is global).
|
||||
- Stream processing consumers are idempotent (P3): a redelivery
|
||||
after a crash re-processes a window; the aggregation must
|
||||
tolerate re-application (e.g., a sum is idempotent under replay
|
||||
if the window is keyed by offset range, not by wall time).
|
||||
- Exactly-once stream processing (P4) requires transactions: the
|
||||
consume-process-produce loop is one transaction — the input
|
||||
offset commit and the output produce are atomic. See the
|
||||
transactional exactly-once producer below.
|
||||
|
||||
## Exactly-Once via Transactions (P4, P3)
|
||||
|
||||
- Exactly-once stream processing is at-least-once plus a
|
||||
transaction: the consumer commits the input offset and produces
|
||||
the output in one transactional operation. If the consumer
|
||||
crashes mid-transaction, neither the offset commit nor the
|
||||
output produce happens — the consumer re-reads from the last
|
||||
committed offset and re-processes (at-least-once), but the
|
||||
transaction ensures the output is produced exactly once.
|
||||
- This is NOT a broker guarantee of exactly-once delivery; it is
|
||||
at-least-once delivery plus idempotent/transactional processing
|
||||
(P3, P4). Jepsen analyses of Kafka transactions confirm the
|
||||
boundaries: the transaction is atomic within the broker, but
|
||||
the downstream sink must be transactional or idempotent too.
|
||||
- See `domains/messaging/delivery-semantics.md` for the full
|
||||
exactly-once-via-idempotency discussion.
|
||||
|
||||
```python
|
||||
# Transactional exactly-once producer (P4 exactly-once via
|
||||
# transactions, P3 idempotent produce). The consume-process-
|
||||
# produce loop is one transaction: the input offset commit and
|
||||
# the output produce are atomic. A crash mid-transaction rolls
|
||||
# both back; the consumer re-reads and re-processes.
|
||||
|
||||
def consume_transform_produce(stream, group, txn_producer):
|
||||
# Begin a transaction. All produces and the offset commit in
|
||||
# this block are atomic (P4).
|
||||
with txn_producer.transaction() as txn:
|
||||
for partition in stream.assigned_partitions(group, ME):
|
||||
offset = stream.committed_offset(group, partition)
|
||||
for message in stream.read(partition, from_offset=offset):
|
||||
output = transform(message["payload"])
|
||||
# P3: idempotent produce. The txn producer
|
||||
# dedups by an epoch+sequence so a retried
|
||||
# transaction does not double-produce.
|
||||
txn.produce(
|
||||
topic="enriched-events",
|
||||
key=message["key"],
|
||||
value=output,
|
||||
idempotencyKey=message["idempotencyKey"],
|
||||
)
|
||||
# Commit the input offset within the same
|
||||
# transaction (P4 atomicity). A crash before
|
||||
# txn.commit() rolls this back; the consumer
|
||||
# re-reads from the prior offset.
|
||||
txn.commit_offset(group, partition, message["offset"])
|
||||
# txn.commit() makes the produces and the offset commit
|
||||
# visible atomically. A crash before this point aborts
|
||||
# both; a crash after is safe (idempotent produce — P3).
|
||||
```
|
||||
|
||||
- The transactional producer's idempotency (P3) is the defense
|
||||
against a retried transaction: the broker dedups the output by
|
||||
the producer's epoch and sequence so a re-commit does not
|
||||
double-produce. The transaction (P4) is the defense against a
|
||||
partial failure: the offset and the output commit together.
|
||||
|
||||
## Stream Schema (P1, P9, cross-link data/schema-design)
|
||||
|
||||
- A stream's messages carry a versioned schema (P1). The schema
|
||||
evolves compatibly (P9): a new field the old consumer ignores is
|
||||
backward-compatible; a renamed field the old consumer parses as
|
||||
`undefined` is a P1 violation compounded as an evolution defect.
|
||||
- Stream schemas are often registered in a schema registry
|
||||
(Confluent, Apicurio) that enforces compatibility on produce.
|
||||
A producer that tries to publish an incompatible schema is
|
||||
rejected; the registry is the P1/P9 enforcement point.
|
||||
- See `domains/data/schema-design` for the generic schema-design
|
||||
discipline (Avro, Protobuf, JSON Schema); messaging owns the
|
||||
stream-specific instance — the registry, the per-topic
|
||||
compatibility mode, the consumer-side routing by schema
|
||||
version.
|
||||
|
||||
## Stream-Platform Comparison (D-069)
|
||||
|
||||
| Axis | Apache Kafka | AWS Kinesis | Apache Pulsar | NATS JetStream |
|
||||
|------|--------------|-------------|---------------|----------------|
|
||||
| **Ordering** | Per-partition strict (P2); global only via single-partition topic | Per-shard strict; global only via single shard | Per-partition strict; global via single partition; also supports shared (out-of-order) subscriptions | Per-stream strict; per-subject ordering; global via single stream |
|
||||
| **Partitioning model** | Partitions (immutable count post-creation; increase requires recreate); key→partition by hash | Shards (reshardable: split/merge at runtime); key→shard by hash | Partitions (resizable; Pulsar's layered architecture separates compute from storage); key→partition by hash | Streams (subject-based; republish to resize); key→stream by subject |
|
||||
| **Replay / retention** | Time- or size-based retention; compaction (latest-per-key); replay from offset or timestamp | Time-based retention (24h–365d); replay from sequence number or timestamp; no compaction | Time- or size-based; compaction; replay from offset or timestamp; tiered storage (hot/warm/cold) | Time- or size-based; per-stream max-age; replay from sequence; no native compaction |
|
||||
| **Consumer groups** | Group-coordinated; offsets stored in an internal topic; eager (legacy) or sticky/cooperative (2.4+) rebalance | Enhanced fan-out consumers (per-shard HTTP/2 push); KCL for group coordination; no native group rebalance (shard is the unit) | Group-coordinated; shared or failover subscription modes; cooperative rebalance | Per-stream consumers; durable cursors; no native group rebalance (stream is the unit) |
|
||||
| **Exactly-once** | Transactions (KIP-98): consume-process-produce atomic; idempotent producer (KIP-516) | No native exactly-once; at-least-once with consumer-side dedup (P3) | Transactions: produce-ack atomic; idempotent producer | At-least-once by default; dedup window per stream (P3 idempotency) |
|
||||
| **Use-case fit** | High-throughput durable logs, stream processing (Kafka Streams, Flink), event sourcing, multi-consumer replay | AWS-native streaming, log ingestion, simple ETL within AWS; low operational burden | Cloud-native, geo-replication, tiered storage, mixed pub/sub + streaming; multi-tenant | Lightweight, low-latency, edge-friendly; NATS ecosystem; simpler ops than Kafka |
|
||||
| **Watch out for** | Partition count is fixed at creation (resize requires recreate + republish); rebalance cost on large groups; operational complexity | Shard limits per account; no compaction; retention cap at 365 days; AWS lock-in | Two-arch (BookKeeper + Brokers) operational complexity; smaller ecosystem | Smaller ecosystem; no native compaction; fewer stream-processing libraries |
|
||||
|
||||
- The default for high-throughput durable logs with multi-consumer
|
||||
replay is **Kafka**; for AWS-native streaming, **Kinesis**; for
|
||||
geo-replicated multi-tenant or mixed pub/sub + streaming,
|
||||
**Pulsar**; for lightweight low-latency edge-friendly streaming,
|
||||
**NATS JetStream** (which also cross-links `domains/edge/iot`
|
||||
via MQTT parallels — see the edge↔messaging bidirectionality in
|
||||
`domains/messaging/first-principles.md` §4).
|
||||
- The ordering column is the P2 check: every platform provides
|
||||
per-partition/per-shard strict order; none provides global order
|
||||
across partitions except by single-partition. The replay/
|
||||
retention column is the P8 check: every platform retains for a
|
||||
configured window; replay is from offset or timestamp. The
|
||||
exactly-once column is the P4 check: Kafka and Pulsar provide
|
||||
transactions; Kinesis and JetStream rely on at-least-once plus
|
||||
consumer-side idempotency (P3).
|
||||
|
||||
## Cross-Partition Traces (P10, cross-link observability/tracing)
|
||||
|
||||
- A stream-processing pipeline that fans out across partitions
|
||||
must propagate a trace context per event: the trace ID follows
|
||||
the event from source to processed output, even as the event
|
||||
crosses partition boundaries. Without cross-partion traces, a
|
||||
downstream error cannot be traced back to its source event.
|
||||
- See `domains/observability/tracing` for the generic distributed-
|
||||
tracing discipline (trace context propagation, span
|
||||
correlation). Messaging owns the stream-specific instance: the
|
||||
trace context is a message header, the span boundary is the
|
||||
consume-process-produce edge, and the cross-partition
|
||||
correlation is by trace ID (not by partition — partitions are
|
||||
independent logs, P2).
|
||||
- A stream processor with no trace propagation is a P10
|
||||
violation: the operator cannot trace a processed event back to
|
||||
its source. Wire the trace context into every produce and every
|
||||
consume.
|
||||
|
||||
## What Violates Stream Discipline
|
||||
|
||||
| Violation | Principle |
|
||||
|-----------|-----------|
|
||||
| Stream with no retention (pipe, not log; no replay) | P8 Replay and Retention are Configured |
|
||||
| Infinite retention (grows until disk exhaustion) | P8, P6 |
|
||||
| Consumer group with eager rebalance at scale (full stop-the-world per deploy) | P6, IDEATE-41 (use sticky/cooperative) |
|
||||
| Non-idempotent stream consumer under at-least-once (redelivery re-processes the window) | P3 Consumers are Idempotent |
|
||||
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
|
||||
| Partition count too low (caps parallelism) or too high (overhead) | P7 |
|
||||
| Cross-partition order assumption (no global order guarantee) | P2 Ordering is a Property, Not an Assumption |
|
||||
| Exactly-once claimed without transactional consume-process-produce (P4 violation) | P4 Delivery Semantics are Explicit |
|
||||
| Stream schema with no registry / no compatibility enforcement (silent shape break) | P1, P9, `domains/data/schema-design` |
|
||||
| No per-partition lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
|
||||
| No cross-partition trace propagation (downstream error untraceable) | P10, `domains/observability/tracing` |
|
||||
| Compacted topic treated as a full log (old values already discarded) | P8, C1 (compaction is a retention mode, not a full log) |
|
||||
@@ -0,0 +1,171 @@
|
||||
# Go Concurrency — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Go's concurrency specifically. Go's distinctive strength (goroutines, channels, context) earns a dedicated concurrency doc rather than a `go-async.md`.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/go.md` for the language first-principles stub.
|
||||
|
||||
## Goroutines and Structured Concurrency (Concurrency P1 Immutability by Default, C6 Composability)
|
||||
|
||||
- **`go f()` spawns a goroutine; ensure it does not outlive its parent:** an unstructured `go f()` leaks when the parent returns. Use `sync.WaitGroup`, `errgroup.Group`, or a `context`-scoped pattern to bound lifetime.
|
||||
- **`errgroup.WithContext` for structured concurrency:** a `Group` cancels its context on first error; siblings see the cancellation and exit. Mirrors `TaskGroup` semantics cross-language.
|
||||
- **Goroutines share only immutable inputs:** `go process(snap)` where `snap` is a copy. A goroutine sharing a mutable slice with the parent is a race (Concurrency P1 Immutability, P6 No Silent Races).
|
||||
- **No `go` in a library function without a documented lifetime:** a library that spawns unbounded goroutines leaks them into the caller. Either accept a `context.Context` or return a `Stop()` method.
|
||||
|
||||
```go
|
||||
import "golang.org/x/sync/errgroup"
|
||||
|
||||
func fetchAll(ctx context.Context, ids []string) ([]*User, error) {
|
||||
g, ctx := errgroup.WithContext(ctx)
|
||||
results := make([]*User, len(ids))
|
||||
for i, id := range ids {
|
||||
i, id := i, id // capture loop vars
|
||||
g.Go(func() error {
|
||||
u, err := fetchUser(ctx, id)
|
||||
if err != nil { return err }
|
||||
results[i] = u
|
||||
return nil
|
||||
})
|
||||
}
|
||||
if err := g.Wait(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return results, nil
|
||||
}
|
||||
```
|
||||
|
||||
## Channels: Bounded Queues and Backpressure (Concurrency P9 Bounded Queues, C6 Composability)
|
||||
|
||||
- **Bounded channels apply backpressure:** `make(chan T, N)` blocks the sender when full (Concurrency P9 — bounded queues). Unbounded `make(chan T)` lets the producer run ahead and OOM.
|
||||
- **`select` with `default` for non-blocking send/receive:** a `default` case makes the channel a queue with try semantics; without it, the operation blocks.
|
||||
- **Close channel from the sender, never the receiver:** closing a channel signals "no more sends." A receiver closing it is a race; the sender may still be writing.
|
||||
- **One channel, one responsibility:** do not multiplex control and data on the same channel. Use a `select` over multiple channels instead.
|
||||
- **Applies `messaging/queues`:** a bounded Go channel is an in-process broker — bounded buffer, backpressure, at-most-once handoff. The same semantics apply; the broker is local.
|
||||
|
||||
```go
|
||||
func pipeline(ctx context.Context, in <-chan Job, out chan<- Result) {
|
||||
for {
|
||||
select {
|
||||
case j, ok := <-in:
|
||||
if !ok { return }
|
||||
r := process(j)
|
||||
select {
|
||||
case out <- r:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// bounded: backpressure when out is full
|
||||
out := make(chan Result, 16)
|
||||
```
|
||||
|
||||
## context.Context for Cancellation (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
|
||||
|
||||
- **`context.Context` is the first parameter of every I/O function:** `func fetchUser(ctx context.Context, id string) (*User, error)`. A function that does I/O without a `ctx` cannot be cancelled (Concurrency P7).
|
||||
- **`context.WithTimeout` for a deadline:** `ctx, cancel := context.WithTimeout(ctx, 5*time.Second); defer cancel()`. Every external call races against a deadline (Concurrency P8).
|
||||
- **`cancel()` always called, even on success:** `defer cancel()` immediately after creating the context. A leaked context leaks its timer.
|
||||
- **Never store a `context.Context` in a struct:** pass it as a parameter. A struct holding a `ctx` captures a request-scoped value into a long-lived object.
|
||||
- **Applies `concurrency/P7`:** cancellation propagates via `ctx.Done()`. A `select` on `<-ctx.Done()` is the cancel-aware wait.
|
||||
|
||||
```go
|
||||
func fetchWithTimeout(ctx context.Context, url string) (*Response, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
return nil, ErrTimeout
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
```
|
||||
|
||||
## select and Multiplexed Channels (Concurrency P7 Cancellation Support, C6 Composability)
|
||||
|
||||
- **`select` multiplexes channel operations:** it picks a ready case at random (fair). A `select` with `<-ctx.Done()` plus a data case is the cancel-aware wait.
|
||||
- **`default` makes `select` non-blocking:** use for "send if ready, else drop" (a bounded queue with drop-oldest policy).
|
||||
- **`select {}` blocks forever:** a `select{}` with no cases is a permanent block. Use only in a goroutine that should run until the process exits.
|
||||
- **Applies `concurrency/P7`:** the `select` over `ctx.Done()` and a result channel is the canonical cancel pattern.
|
||||
|
||||
```go
|
||||
func processUntilCancel(ctx context.Context, jobs <-chan Job) {
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case j, ok := <-jobs:
|
||||
if !ok { return }
|
||||
// ...
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## sync Primitives and Lock Scope (Concurrency P3 Boundaries are Locks, Concurrency P5 Lock Minimization)
|
||||
|
||||
- **`sync.Mutex` scoped minimally:** not held across I/O (a `Send` on a channel, an HTTP call). Hold the lock, mutate, release — then do I/O (Concurrency P3 Lock Scope).
|
||||
- **`sync.RWMutex` for read-heavy, `Mutex` for write-heavy:** RWMutex adds overhead; only prefer it when reads dominate by 10x+.
|
||||
- **`sync.Map` for specific cases (append-only, disjoint keys):** not a general `map[K]V` replacement. For most maps, `Mutex` + `map` is clearer and often faster.
|
||||
- **`sync.Once` for one-time init:** `var once sync.Once; once.Do(func(){ init() })`. Idempotent and race-free.
|
||||
- **Applies `concurrency/P5` (lock minimization):** prefer channels over locks; when a lock is needed, hold it for the smallest possible scope.
|
||||
|
||||
```go
|
||||
type Cache struct {
|
||||
mu sync.Mutex
|
||||
items map[string]*User
|
||||
}
|
||||
|
||||
func (c *Cache) Get(id string) (*User, bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
u, ok := c.items[id]
|
||||
return u, ok
|
||||
}
|
||||
|
||||
func (c *Cache) Set(id string, u *User) {
|
||||
c.mu.Lock()
|
||||
c.items[id] = u
|
||||
c.mu.Unlock() // explicit unlock before any I/O
|
||||
}
|
||||
```
|
||||
|
||||
## Race Detection (Concurrency P6 No Silent Races)
|
||||
|
||||
- **`go test -race` enforces `P6`:** the race detector instruments memory accesses and fails on data races. See `go-tooling.md` for the CI gate.
|
||||
- **Tests must exercise the concurrent path:** a serial test of a `Mutex`-protected map finds no race. Write tests with N goroutines hitting the map under `-race`.
|
||||
- **Applies `concurrency/P6`:** a race detected at test time is a bug fixed; a race undetected is a production heisenbug.
|
||||
|
||||
```go
|
||||
func TestCacheConcurrent(t *testing.T) {
|
||||
c := &Cache{items: map[string]*User{}}
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 100; i++ {
|
||||
i := i
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
c.Set(strconv.Itoa(i), &User{})
|
||||
c.Get(strconv.Itoa(i))
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P1, P3, P5, P6, P7, P8, P9 traced throughout.
|
||||
- `domains/messaging/queues.md` — bounded Go channels as in-process brokers; backpressure parallels (IDEATE-40).
|
||||
- `domains/errors/patterns.md` — `errgroup` and error propagation in concurrent code.
|
||||
- `languages/go-types.md` — typed channels carry the named types defined there.
|
||||
- `languages/go-tooling.md` — the `-race` CI gate that enforces Concurrency P6.
|
||||
- `languages/go-testing.md` — concurrent tests that exercise the race detector.
|
||||
@@ -0,0 +1,141 @@
|
||||
# Go Testing — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Go testing specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/go.md` for the language first-principles stub.
|
||||
|
||||
## Table-Driven Tests (Testing P1 Tests as Specification, C2 Clarity)
|
||||
|
||||
- **Table-driven is the Go idiom:** `cases := []struct{ name string; in X; want Y }{...}`; loop with `t.Run(c.name, ...)`. Each case is a subtest with its own name and failure output.
|
||||
- **Test names read as a spec:** `{"rejects empty email", ...}`, `{"returns persisted id", ...}`. A reader understands the unit from the subtest names (Testing P1).
|
||||
- **No `if got != want { t.Fatal() }` shared across cases:** each case asserts independently; a failure in case 3 does not skip cases 4 and 5.
|
||||
- **`t.Run` enables `-run` filtering:** `go test -run TestCreateUser/rejects_empty_email` runs one case. Essential for debugging a single failure.
|
||||
|
||||
```go
|
||||
func TestCreateUser(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
email string
|
||||
wantErr bool
|
||||
}{
|
||||
{"rejects empty email", "", true},
|
||||
{"rejects missing @", "no-at-sign", true},
|
||||
{"accepts valid email", "a@b.co", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
_, err := CreateUser(c.email)
|
||||
if (err != nil) != c.wantErr {
|
||||
t.Fatalf("err=%v, wantErr=%v", err, c.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## t.Parallel for Independence (Testing P2 Independence, Concurrency P10 Test for Race Conditions)
|
||||
|
||||
- **`t.Parallel()` for independent subtests:** each subtest opts in; the runner executes them concurrently. A test that fails under `Parallel` has hidden state (Testing P2 Independence).
|
||||
- **Capture loop variables:** `c := c` inside the loop, or rely on Go 1.22+ per-iteration scoping. A parallel subtest sharing `c` races on the last value.
|
||||
- **Applies `concurrency/P10` (test for races):** parallel tests are the first line of race detection; combine with `-race` for the full safety net.
|
||||
|
||||
```go
|
||||
for _, c := range cases {
|
||||
c := c // capture for parallel
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
_, err := CreateUser(c.email)
|
||||
if (err != nil) != c.wantErr {
|
||||
t.Fatalf("err=%v, wantErr=%v", err, c.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## t.Cleanup for Teardown (Testing P3 Determinism, Testing P2 Independence)
|
||||
|
||||
- **`t.Cleanup(func() { ... })` for teardown:** runs in LIFO order after the test (and its subtests) complete. Replaces `defer` in a helper that does not know when the test ends.
|
||||
- **Per-test state, not shared:** a `setup(t)` helper creates resources and registers cleanup; each test gets its own. A package-level `var` shared across tests is order coupling.
|
||||
- **`t.TempDir()` for filesystem tests:** creates a unique temp dir and cleans up automatically. No manual `os.RemoveAll` and no cross-test contamination.
|
||||
- **Applies `Testing P3` (determinism):** cleanup is tied to the test lifecycle, not a global teardown that may run before or after depending on order.
|
||||
|
||||
```go
|
||||
func setupStore(t *testing.T) *Store {
|
||||
t.Parallel()
|
||||
dir := t.TempDir() // auto-cleaned
|
||||
s, err := OpenStore(filepath.Join(dir, "db"))
|
||||
if err != nil { t.Fatal(err) }
|
||||
t.Cleanup(func() { s.Close() })
|
||||
return s
|
||||
}
|
||||
```
|
||||
|
||||
## Race Detector (Testing P9 Edge Case Coverage, Concurrency P6 No Silent Races)
|
||||
|
||||
- **`go test -race` in CI, always:** see `go-tooling.md`. The detector is the enforcement of `concurrency/P6`.
|
||||
- **Tests must exercise the concurrent path:** a serial test of a `Mutex`-protected map finds no race. Write tests with N goroutines.
|
||||
- **`-count=1` to disable result caching:** by default, Go caches passing tests. `-count=1` forces re-run; combine with `-race` and parallelism to surface heisenbugs.
|
||||
- **Applies `Testing P9` (edge case coverage):** the race detector is the edge-case tool for concurrency — it finds the inputs the test author forgot to write.
|
||||
|
||||
```bash
|
||||
# CI gate
|
||||
go test -race -count=1 ./...
|
||||
```
|
||||
|
||||
## Time and Determinism (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||
|
||||
- **No `time.Now()` in code under test:** inject a `Clock` interface. In tests, a fake clock advances deterministically.
|
||||
- **`time.Sleep` in tests is a smell:** a sleep waits for a real timer, flaky under load. Use a channel or `Eventually`-style polling with a timeout.
|
||||
- **`t.Deadline()` aware helpers:** a helper that may take long checks `t.Deadline()` and bails early. Prevents a slow test from timing out the suite.
|
||||
|
||||
```go
|
||||
type Clock interface { Now() time.Time }
|
||||
|
||||
type fakeClock struct{ t time.Time }
|
||||
func (f *fakeClock) Now() time.Time { return f.t }
|
||||
|
||||
func TestUserHasCreatedAt(t *testing.T) {
|
||||
clk := &fakeClock{time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)}
|
||||
u, _ := CreateUserWithClock("a@b.co", clk)
|
||||
if u.CreatedAt.Year() != 2024 {
|
||||
t.Fatalf("year=%d, want 2024", u.CreatedAt.Year())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Mocks and Interfaces (Testing P7 Realism, API P1 Contract Fidelity)
|
||||
|
||||
- **Mock at the interface, not the struct:** `type Store interface { Get(id string) (*User, error) }` in production; `type mockStore struct{ ... }` in test. The interface is the contract (applies `api/P1`).
|
||||
- **`httptest` for HTTP servers:** `httptest.NewServer` gives a real server on a loopback port; no manual socket plumbing.
|
||||
- **`testify/mock` or hand-written mocks:** hand-written for one-off, `testify` for complex sequencing. Avoid mocking frameworks that generate code at runtime (reflection-heavy) — they hide failures behind stack traces.
|
||||
- **Applies `Testing P7` (realism):** mock the boundary (HTTP, DB), not the unit. Mocking the unit under test tests the mock.
|
||||
|
||||
```go
|
||||
type mockStore struct {
|
||||
users map[string]*User
|
||||
got []string
|
||||
}
|
||||
func (m *mockStore) Get(id string) (*User, error) {
|
||||
m.got = append(m.got, id)
|
||||
return m.users[id], nil
|
||||
}
|
||||
|
||||
func TestGetUserLogs(t *testing.T) {
|
||||
s := &mockStore{users: map[string]*User{"abc": {}}}
|
||||
svc := NewService(s)
|
||||
svc.GetUser("abc")
|
||||
if len(s.got) != 1 || s.got[0] != "abc" {
|
||||
t.Fatalf("got=%v", s.got)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/testing/pyramid.md` — where unit/integration/race tests sit; the race job is its own layer.
|
||||
- `domains/testing/fixtures.md` — `t.TempDir` and `t.Cleanup` as the fixture discipline.
|
||||
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P6 (race detector), P10 (test for races).
|
||||
- `languages/go-types.md` — the named types tests assert.
|
||||
- `languages/go-concurrency.md` — concurrent tests exercise the patterns from that doc.
|
||||
- `languages/go-tooling.md` — the `go test` flags (`-race`, `-count`, `-run`) detailed here.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Go Tooling — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Go tooling specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/go.md` for the language first-principles stub.
|
||||
|
||||
## go vet and golangci-lint (DevOps P2 Automation, C2 Clarity)
|
||||
|
||||
- **`go vet` is the stdlib baseline:** it catches `printf` format mismatches, lock-copy-by-value, and unreachable code. Run on every build.
|
||||
- **`golangci-lint` aggregates vet + dozens of linters:** enable `errcheck` (no `_ = err`), `govet`, `staticcheck`, `ineffassign`, `unused`, `gofmt`, `goimports`. Each enabled linter has a one-line `# reason:` in `.golangci.yml`.
|
||||
- **`errcheck` enforces `errors/P2` (fail loudly):** a discarded error is a silent failure. `errcheck` fails the build on `_ = doX()`.
|
||||
- **`goimports` over `gofmt`:** `goimports` adds missing imports and removes unused ones, in addition to formatting. The format is not debated in review (Clarity C2).
|
||||
|
||||
```yaml
|
||||
# .golangci.yml
|
||||
linters:
|
||||
enable:
|
||||
- errcheck # reason: Errors P2 — no swallowed errors
|
||||
- govet
|
||||
- staticcheck
|
||||
- ineffassign
|
||||
- unused
|
||||
- gofmt
|
||||
- goimports
|
||||
linters-settings:
|
||||
errcheck:
|
||||
check-blank: true # fail on _ = fn()
|
||||
```
|
||||
|
||||
## go test -race (Concurrency P6 No Silent Races)
|
||||
|
||||
- **`go test -race` in CI, always:** the race detector instruments memory accesses and fails on data races. It is the primary enforcement of `concurrency/P6` (no silent races).
|
||||
- **`-race` adds overhead; run it in a separate CI job:** the race build is ~2x slower; keep the fast unit-test job and add a race job.
|
||||
- **`-race` requires tests that actually exercise the concurrent path:** a test that calls `Get`/`Set` serially finds no race. Write tests that spawn goroutines hitting the same map.
|
||||
- **Applies `concurrency/P6`:** a race detected is a bug fixed; a race undetected is a heisenbug in production. The detector is the safety net.
|
||||
|
||||
```bash
|
||||
# CI race job
|
||||
go test -race -count=1 ./...
|
||||
```
|
||||
|
||||
## Module Discipline (DevOps P1 Reproducibility)
|
||||
|
||||
- **`go mod tidy` on every change that touches imports:** removes unused deps and adds missing ones. A `go.mod` with stale entries breaks reproducibility.
|
||||
- **`go.sum` committed and verified:** `go mod verify` checks the checksums of the module cache against `go.sum`. A drifted `go.sum` is a supply-chain signal.
|
||||
- **Pinned major versions in `go.mod`:** `require github.com/x/y v1.2.3` pins the minor; a `v1.2.4` patch may auto-update. For applications, consider a `go.mod` proxy that pins to exact commits.
|
||||
- **`go mod vendor` for hermetic CI:** vendoring `vendor/` into the repo means CI builds without network. The trade-off is repo size; the win is reproducibility (DevOps P1).
|
||||
|
||||
```bash
|
||||
# CI build gate
|
||||
go mod tidy
|
||||
go mod verify
|
||||
go build ./...
|
||||
go test -race ./...
|
||||
```
|
||||
|
||||
## Reproducible Builds (DevOps P1 Reproducibility, C3 Simplicity)
|
||||
|
||||
- **One Go toolchain version, pinned:** `goenv` or `asdf` pins the Go version per repo; a `.go-version` file declares it. A CI job that uses "latest" Go drifts.
|
||||
- **`CGO_ENABLED=0` for static binaries:** a static binary runs in a scratch container with no libc dependency. Set in CI for all release builds.
|
||||
- **`-trimpath` and `-ldflags='-s -w'` for reproducible output:** strips the build path from the binary and removes debug info. Two builds of the same commit produce byte-identical binaries.
|
||||
|
||||
```bash
|
||||
# Reproducible release build
|
||||
CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o app ./cmd/app
|
||||
```
|
||||
|
||||
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||
|
||||
- **`go doc` from comments:** package comments and exported-symbol comments are the API docs; `go doc` and `pkg.go.dev` render them. Missing comments on exported symbols fail `revive`/`golint` (Documentation P1).
|
||||
- **`// Example` functions are run by `go test`:** an `ExampleUser` function with `// Output:` is a tested artifact; a stale output fails the build.
|
||||
- **`README.md` and `docs/` are built by `mkdocs` or similar:** the pipeline validates links and renders; a broken link fails CI (Documentation P1).
|
||||
|
||||
```go
|
||||
// GetUser fetches a user by id.
|
||||
//
|
||||
// Example:
|
||||
//
|
||||
// u, err := GetUser(id)
|
||||
// if err != nil { ... }
|
||||
func GetUser(id UserId) (*User, error) { /* ... */ }
|
||||
|
||||
func ExampleGetUser() {
|
||||
u, err := GetUser("abc")
|
||||
fmt.Println(u, err)
|
||||
// Output: <nil> not found
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/devops/ci-cd.md` — the pipeline gates that host vet/lint/test.
|
||||
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P6 No Silent Races (`-race`).
|
||||
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||
- `languages/go-types.md` — the type rules staticcheck enforces reference this doc.
|
||||
- `languages/go-testing.md` — the `go test` flags (`-race`, `-count`) detailed here.
|
||||
@@ -0,0 +1,140 @@
|
||||
# Go Type System — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Go's type system specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/go.md` for the language first-principles stub.
|
||||
|
||||
## Named Types for Domain Concepts (C1 Correctness, Data P7 Type Fidelity)
|
||||
|
||||
- **Named types for domain IDs and values:** `type UserId string`, `type OrderId string`. Two named types are distinct even with identical underlying types; the compiler rejects the swap.
|
||||
- **Constructors validate at the boundary:** `func NewUserId(s string) (UserId, error)` returns an error on bad input. A bare `UserId(s)` cast bypasses validation — only the constructor is exported.
|
||||
- **Applies `data/P7` (type fidelity):** a named type carries the domain meaning through the call graph; a `string` parameter does not.
|
||||
- **`any` is the wide type; narrow before use:** Go 1.18+ `any` is an alias for `interface{}`. Use it only at true boundaries (e.g., `json.Unmarshal`); narrow with a type assertion immediately.
|
||||
|
||||
```go
|
||||
type UserId string
|
||||
type OrderId string
|
||||
|
||||
func NewUserId(s string) (UserId, error) {
|
||||
if !regexp.MustCompile(`^[a-z0-9]+$`).MatchString(s) {
|
||||
return "", fmt.Errorf("invalid user id: %q", s)
|
||||
}
|
||||
return UserId(s), nil
|
||||
}
|
||||
|
||||
func GetUser(id UserId) (*User, error) { /* ... */ }
|
||||
|
||||
// GetUser("abc") // compile error: string is not UserId
|
||||
// GetUser(OrderId("abc")) // compile error: distinct named types
|
||||
```
|
||||
|
||||
## Generics (C6 Composability, Data P7 Type Fidelity)
|
||||
|
||||
- **Generics (1.18+) preserve element types across containers:** `type Repository[T any] struct { ... }` keeps `T` through `Get`/`Save`, rather than widening to `any`.
|
||||
- **Constrain with `comparable` for map keys, custom interfaces for behavior:** `func dedupe[T comparable](s []T) []T` uses `comparable`; a `Sortable[T]` constraint expresses the `Less` requirement.
|
||||
- **Avoid generics where an interface suffices:** `io.Reader` is not improved by generics. Generics are for type-preserving containers; interfaces are for behavior.
|
||||
- **No generic methods on generic types (not supported):** `func (r Repository[T]) Map[U any](f func(T) U) Repository[U]` is a compile error. Use a free function.
|
||||
|
||||
```go
|
||||
type Entity interface { ID() string }
|
||||
|
||||
type Repository[T Entity] struct {
|
||||
db map[string]T
|
||||
}
|
||||
|
||||
func (r *Repository[T]) Get(id string) (T, bool) {
|
||||
var zero T
|
||||
t, ok := r.db[id]
|
||||
if !ok { return zero, false }
|
||||
return t, true
|
||||
}
|
||||
|
||||
func (r *Repository[T]) Save(t T) { r.db[t.ID()] = t }
|
||||
```
|
||||
|
||||
## Interfaces (C6 Composability, API P1 Contract Fidelity)
|
||||
|
||||
- **Interfaces defined by the consumer, not the producer:** a package defines its dependencies as interfaces (`type Store interface { Get(id string) (*User, error) }`), and accepts implementations. The producer does not pre-declare "the interface I implement."
|
||||
- **Small interfaces (Go proverb):** `io.Reader` is one method. An interface with 5+ methods is a god-object; split it.
|
||||
- **Accept interfaces, return concrete types:** return a `*UserRepo`, accept a `Store`. The caller gets the implementation; the callee depends on the abstraction.
|
||||
- **Applies `api/P1` (contract fidelity):** the interface is the contract; the concrete type is the implementation. Tests mock the interface, not the struct.
|
||||
|
||||
```go
|
||||
// consumer defines the interface
|
||||
type UserStore interface {
|
||||
Get(id string) (*User, error)
|
||||
}
|
||||
|
||||
type Service struct { store UserStore }
|
||||
|
||||
func NewService(s UserStore) *Service { return &Service{store: s} }
|
||||
|
||||
// producer returns concrete; satisfies UserStore implicitly
|
||||
type UserRepo struct { db map[string]*User }
|
||||
func (r *UserRepo) Get(id string) (*User, error) { return r.db[id], nil }
|
||||
```
|
||||
|
||||
## Type Assertion Discipline (C1 Correctness, Errors P1 Errors are Data)
|
||||
|
||||
- **Type assertions return `(T, bool)` — use the bool:** `v, ok := x.(UserId)` distinguishes "wrong type" from "zero value." A bare `x.(UserId)` panics on mismatch.
|
||||
- **`switch x := x.(type)` for multi-variant narrowing:** each case narrows `x` to the case type. The default case is exhaustive (no `never`-style check; Go relies on review).
|
||||
- **Applies `errors/P1` (errors are data):** a failed type assertion is a value (`ok == false`), not an exception. Handle it as a branch, not a panic.
|
||||
- **Never assert across module boundaries silently:** an assertion on a type from another package couples to its internals. Prefer an interface method.
|
||||
|
||||
```go
|
||||
func describe(x any) string {
|
||||
switch v := x.(type) {
|
||||
case UserId:
|
||||
return "user " + string(v)
|
||||
case OrderId:
|
||||
return "order " + string(v)
|
||||
default:
|
||||
return fmt.Sprintf("unknown: %T", v)
|
||||
}
|
||||
}
|
||||
|
||||
// safe form, never panic
|
||||
id, ok := raw.(UserId)
|
||||
if !ok {
|
||||
return fmt.Errorf("expected UserId, got %T", raw)
|
||||
}
|
||||
```
|
||||
|
||||
## Error Types and errors.Is/As (Errors P1 Errors are Data, Errors P3 Fail Specifically)
|
||||
|
||||
- **Sentinel errors for known cases:** `var ErrNotFound = errors.New("not found")`; check with `errors.Is(err, ErrNotFound)`. The sentinel is a value, not an exception class.
|
||||
- **Custom error types for context:** `type ValidationError struct { Field, Msg string }`; check with `var ve *ValidationError; errors.As(err, &ve)`. The type carries structured data (Errors P4 Preserve Context).
|
||||
- **Wrap with `%w`:** `fmt.Errorf("get user %s: %w", id, err)` preserves the chain. `errors.Is`/`As` unwrap it. Bare `%v` breaks the chain.
|
||||
- **Applies `errors/P3` (fail specifically):** `ErrNotFound` is specific; `ErrFailed` is not. The error type names the failure mode.
|
||||
|
||||
```go
|
||||
var ErrNotFound = errors.New("not found")
|
||||
|
||||
type ValidationError struct {
|
||||
Field string
|
||||
Msg string
|
||||
}
|
||||
func (e *ValidationError) Error() string { return e.Field + ": " + e.Msg }
|
||||
|
||||
func GetUser(id UserId) (*User, error) {
|
||||
u, ok := db[string(id)]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("user %s: %w", id, ErrNotFound)
|
||||
}
|
||||
return u, nil
|
||||
}
|
||||
|
||||
// caller
|
||||
if errors.Is(err, ErrNotFound) { /* 404 */ }
|
||||
var ve *ValidationError
|
||||
if errors.As(err, &ve) { /* 422 with ve.Field */ }
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/data/schema-design.md` — named types parallel schema design at the Go boundary.
|
||||
- `domains/data/first-principles.md` — Data P7 Type Fidelity is the primary trace.
|
||||
- `domains/api/rest.md` — contract fidelity for HTTP handlers using interfaces.
|
||||
- `domains/errors/patterns.md` — `errors.Is`/`As` and the wrap-with-`%w` pattern.
|
||||
- `languages/go-concurrency.md` — typed channels carry these named types.
|
||||
- `languages/go-testing.md` — table-driven tests assert type-swap safety.
|
||||
@@ -2,6 +2,13 @@
|
||||
|
||||
> How Atelier's domain principles apply in Go specifically. Derives from `domains/` docs.
|
||||
|
||||
## Derived Docs
|
||||
|
||||
- [go-types.md](go-types.md) — named types, generics, interfaces, type assertion discipline.
|
||||
- [go-tooling.md](go-tooling.md) — go vet, golangci-lint, go test -race, module discipline.
|
||||
- [go-concurrency.md](go-concurrency.md) — goroutines, channels, context, select, sync primitives.
|
||||
- [go-testing.md](go-testing.md) — table-driven tests, t.Parallel, t.Cleanup, race detector.
|
||||
|
||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||
|
||||
- **Named types for domain concepts:** `type UserId string`, not bare `string`.
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# Python Async — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Python async specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/python.md` for the language first-principles stub.
|
||||
|
||||
## asyncio and anyio (Concurrency P7 Cancellation Support, C2 Clarity)
|
||||
|
||||
- **`asyncio` for I/O-bound work; threads only for blocking libraries:** `async def` + `await` for network/disk; `run_in_executor` to wrap a blocking call. Mixing threads for I/O is the wrong default.
|
||||
- **`anyio` for runtime portability:** `anyio` abstracts asyncio/trio; a library written against `anyio` runs on either. Use it for libraries; for applications, asyncio directly is fine.
|
||||
- **One event loop, one thread:** `asyncio.run(main())` creates and runs the loop. Do not call `asyncio.run` inside an existing loop (raises `RuntimeError`); do not share a loop across threads.
|
||||
- **Applies `concurrency/P7`:** every `async def` accepts cancellation as a first-class signal; `CancelledError` propagates unless explicitly suppressed (and suppressing it is almost always a bug).
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import anyio
|
||||
|
||||
async def fetch_user(id: str) -> User:
|
||||
return await api.get(f'/users/{id}')
|
||||
|
||||
# asyncio application
|
||||
async def main():
|
||||
user = await fetch_user('abc')
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
# anyio library — portable across asyncio/trio
|
||||
async def fetch_all(ids: list[str]) -> list[User]:
|
||||
return await anyio.gather(*(fetch_user(i) for i in ids))
|
||||
```
|
||||
|
||||
## Structured Concurrency (Concurrency P1 Immutability by Default, C6 Composability)
|
||||
|
||||
- **`asyncio.TaskGroup` (3.11+) for structured concurrency:** tasks created in a `TaskGroup` are awaited or cancelled together on exit. No orphan tasks outlive the block.
|
||||
- **No `asyncio.gather(..., return_exceptions=False)` for fallible tasks:** `gather` returns partial results on first exception; `TaskGroup` cancels siblings and propagates the error atomically. Use `TaskGroup` for new code.
|
||||
- **Applies `concurrency/P1` (immutability):** tasks share only immutable inputs; results are collected, not mutated in place. A task that writes to a shared list is a race waiting to happen.
|
||||
- **`anyio.create_task_group()` mirrors `TaskGroup` cross-runtime:** same structured-concurrency guarantee, portable.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def fetch_all(ids: list[str]) -> list[User]:
|
||||
results: list[User] = []
|
||||
async with asyncio.TaskGroup() as tg:
|
||||
tasks = [tg.create_task(fetch_user(i)) for i in ids]
|
||||
# all tasks done (or cancelled) by here
|
||||
return [t.result() for t in tasks]
|
||||
```
|
||||
|
||||
## Cancellation and Timeout (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
|
||||
|
||||
- **`asyncio.wait_for(coro, timeout)` for a deadline:** every external `await` races against a timeout. A bare `await` is an unbounded wait (Concurrency P8).
|
||||
- **`asyncio.timeout()` (3.11+) as a context manager:** `async with asyncio.timeout(5): await op` — cleaner than `wait_for` for multi-await blocks.
|
||||
- **`CancelledError` propagates; do not catch broadly:** `except Exception` swallows `CancelledError` in 3.7 (it was `BaseException`); in 3.8+ it's `BaseException` and `except Exception` skips it. Catch specifically, never bare `except:`.
|
||||
- **Applies `concurrency/P7`:** cancellation is cooperative — a long synchronous block inside `async def` ignores cancellation. Yield with `await asyncio.sleep(0)` periodically in CPU-bound loops.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def fetch_with_timeout(id: str, timeout: float = 5.0) -> User:
|
||||
async with asyncio.timeout(timeout):
|
||||
return await fetch_user(id)
|
||||
|
||||
async def shutdown(token: asyncio.Event) -> None:
|
||||
# cooperative cancel — long-running loop checks the token
|
||||
while not token.is_set():
|
||||
await do_chunk()
|
||||
await asyncio.sleep(0) # yield so cancel can land
|
||||
```
|
||||
|
||||
## Bounded Concurrency and Queues (Concurrency P9 Bounded Queues)
|
||||
|
||||
- **`asyncio.Semaphore(N)` to bound in-flight tasks:** a `Semaphore(8)` wrapping `gather` caps concurrency. Unbounded `gather` on a 10k-item list exhausts file descriptors (Concurrency P9 — bounded queues).
|
||||
- **`asyncio.Queue(maxsize=N)` for producer/consumer:** a bounded queue applies backpressure to the producer. An unbounded queue lets the producer run ahead and OOM.
|
||||
- **Applies `messaging/queues`:** an `asyncio.Queue` is an in-process broker — the same bounded-queue / backpressure semantics apply; the broker is just local.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
async def map_bounded(items: list[str], limit: int = 8) -> list[User]:
|
||||
sem = asyncio.Semaphore(limit)
|
||||
async def guarded(i: str) -> User:
|
||||
async with sem:
|
||||
return await fetch_user(i)
|
||||
return await asyncio.gather(*(guarded(i) for i in items))
|
||||
```
|
||||
|
||||
## Error Handling in Async (Errors P5 Recoverable When Possible, Errors P1 Errors are Data)
|
||||
|
||||
- **Retry with backoff for transient failures:** network blips are recoverable (Errors P5). Exponential backoff with jitter, capped retries, and an `anyio`-cancellation-aware `sleep`.
|
||||
- **No retry for non-idempotent operations:** a `POST` creating a resource is not safely retryable without an idempotency key (applies `api/P6` Idempotency).
|
||||
- **`except asyncio.CancelledError: raise`** is the only valid handling — re-raise so the cancellation propagates. Catching and continuing breaks structured concurrency.
|
||||
- **Applies `messaging/delivery-semantics`:** a cancelable async operation is at-most-once; retry-on-cancel is at-least-once. The caller must declare which.
|
||||
|
||||
```python
|
||||
import anyio
|
||||
import random
|
||||
|
||||
async def fetch_retry(id: str, attempts: int = 3) -> User:
|
||||
for i in range(attempts):
|
||||
try:
|
||||
return await fetch_user(id)
|
||||
except (TimeoutError, ConnectionError):
|
||||
if i == attempts - 1:
|
||||
raise
|
||||
await anyio.sleep((2 ** i) * 0.1 + random.random() * 0.1)
|
||||
raise RuntimeError('unreachable')
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P1 Immutability, P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
|
||||
- `domains/messaging/queues.md` — `asyncio.Queue` as an in-process broker; backpressure parallels (IDEATE-40).
|
||||
- `domains/errors/patterns.md` — typed async errors and retry-with-backoff.
|
||||
- `languages/py-types.md` — `Result` and exception hierarchy used in async error handling.
|
||||
- `languages/py-tooling.md` — `pytest-asyncio` config that runs these tests.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Python Testing — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Python testing specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/python.md` for the language first-principles stub.
|
||||
|
||||
## pytest and Spec-Driven Tests (Testing P1 Tests as Specification, C2 Clarity)
|
||||
|
||||
- **`pytest` is the default; `unittest` only for stdlib-only libraries:** `pytest` fixtures, parametrize, and assertion rewriting beat `unittest`'s boilerplate (Clarity C2).
|
||||
- **Tests co-located with source:** `user.py` → `test_user.py`. A test far from its subject rots (Documentation P5 Discoverability).
|
||||
- **Test names read as a spec:** `def test_create_user_rejects_invalid_email():` — a reader understands the unit from the name. Avoid `def test_user1():`.
|
||||
- **`assert` over `self.assertEqual`:** pytest rewrites `assert` to show the failing values; `assertEqual` is unittest's escape hatch and loses readability.
|
||||
- **Applies `Testing P1`:** the test is a specification; the failure message is the spec violation.
|
||||
|
||||
```python
|
||||
# test_user.py
|
||||
import pytest
|
||||
from user import create_user, ValidationError
|
||||
|
||||
def test_create_user_rejects_invalid_email():
|
||||
with pytest.raises(ValidationError):
|
||||
create_user(email='not-an-email')
|
||||
|
||||
def test_create_user_returns_persisted_id():
|
||||
u = create_user(email='a@b.co')
|
||||
assert u.id # truthy persisted id
|
||||
```
|
||||
|
||||
## Factories and Fixture Discipline (Testing P2 Independence, Testing P7 Realism)
|
||||
|
||||
- **`factory_boy` or `pytest-factoryboy` over shared fixtures for mutable state:** `UserFactory.build()` returns a fresh object per call; a session-scoped fixture mutated across tests couples them (Testing P2 Independence).
|
||||
- **Fixtures for setup/teardown, factories for data:** a `db` fixture sets up the DB once per test; a `make_user` factory produces fresh data per assertion. Conflating them produces order-dependent tests.
|
||||
- **`scope='function'` is the default and the safe default:** `scope='session'` for read-only resources (a schema migration), never for mutable state.
|
||||
- **Mock at the boundary, not the unit:** `mocker.patch('requests.get')` for HTTP; do not patch `user.User.save` (that mocks the unit under test — Testing P7 realism).
|
||||
|
||||
```python
|
||||
import factory
|
||||
from user import User
|
||||
|
||||
class UserFactory(factory.Factory):
|
||||
class Meta:
|
||||
model = User
|
||||
email = factory.Sequence(lambda n: f'u{n}@b.co')
|
||||
name = 'Test User'
|
||||
|
||||
def test_user_factory_is_fresh():
|
||||
u1 = UserFactory.build()
|
||||
u2 = UserFactory.build()
|
||||
assert u1.email != u2.email # independent
|
||||
```
|
||||
|
||||
## Parametrize and Edge Cases (Testing P9 Edge Case Coverage, Testing P3 Determinism)
|
||||
|
||||
- **`@pytest.mark.parametrize` for input tables:** one parametrized test runs N cases; each is an independent test with its own name and failure output (Testing P9).
|
||||
- **Edge cases as rows, not special tests:** empty list, `None`, max int, unicode — each a row. An ad-hoc `test_handles_edge` with multiple asserts hides which case failed (Testing P6 Failure Specificity).
|
||||
- **`pytest --randomly` catches order coupling:** a test passing alone but failing in a suite has hidden shared state. The random plugin makes it visible (Testing P2 Independence).
|
||||
- **Property tests via `hypothesis`:** for invariants (e.g., "parse(serialize(x)) == x"), `hypothesis` generates hundreds of inputs and shrinks failures to a minimal counterexample.
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.mark.parametrize('email, reason', [
|
||||
('', 'empty'),
|
||||
('a' * 1000 + '@b.co', 'too long'),
|
||||
('no-at-sign', 'missing @'),
|
||||
('a@b', 'missing TLD'),
|
||||
])
|
||||
def test_create_user_rejects(email, reason):
|
||||
with pytest.raises(ValidationError):
|
||||
create_user(email=email)
|
||||
```
|
||||
|
||||
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||
|
||||
- **No `datetime.now()`, `time.time()`, `uuid.uuid4()`, `random.random()` in code under test:** inject a `Clock`, `UUIDGen`, `Random` port. In tests, provide deterministic fakes.
|
||||
- **`freezegun` for time:** `@freeze_time('2024-01-01')` makes `datetime.now()` deterministic. Do not call `datetime.now()` directly in code — wrap it in a `Clock` port so production and tests both inject.
|
||||
- **`pytest --randomly-seed=last` to reproduce a failing order:** when `--randomly` finds an order bug, the seed is logged; re-run with it to debug deterministically.
|
||||
|
||||
```python
|
||||
from freezegun import freeze_time
|
||||
|
||||
@freeze_time('2024-01-01')
|
||||
def test_user_has_created_at():
|
||||
u = create_user(email='a@b.co')
|
||||
assert u.created_at.year == 2024
|
||||
```
|
||||
|
||||
## Async Tests (Concurrency P10 Test for Race Conditions, Testing P1 Tests as Specification)
|
||||
|
||||
- **`pytest-asyncio` (or `anyio`'s pytest plugin) for `async def` tests:** `@pytest.mark.asyncio` runs the coroutine on a loop. Without it, an `async def` test is silently skipped (returns a coroutine, never awaited).
|
||||
- **`anyio`'s plugin runs the same test on asyncio and trio:** one parametrized run across both runtimes catches runtime-specific bugs.
|
||||
- **Race-sensitive tests use `--randomly` and bounded concurrency:** a `Semaphore(1)` test under random order surfaces hidden state.
|
||||
- **Applies `concurrency/P10`:** async tests are the race detector's first line — if a test passes alone but fails under `gather` of N, there's a race.
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_fetch_returns_user():
|
||||
u = await fetch_user('abc')
|
||||
assert u.email
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/testing/pyramid.md` — where unit/integration/property tests sit; hypothesis is the property layer.
|
||||
- `domains/testing/fixtures.md` — factory-vs-fixture discipline applied via `factory_boy`.
|
||||
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||
- `languages/py-types.md` — the `Result` and Pydantic models that tests assert.
|
||||
- `languages/py-async.md` — async tests use the cancellation/timeout patterns from that doc.
|
||||
- `languages/py-tooling.md` — the `pyproject.toml [tool.pytest]` config that runs these tests.
|
||||
@@ -0,0 +1,99 @@
|
||||
# Python Tooling — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Python tooling specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/python.md` for the language first-principles stub.
|
||||
|
||||
## ruff for Lint and Format (DevOps P2 Automation, C2 Clarity)
|
||||
|
||||
- **`ruff` replaces flake8 + black + isort + pyupgrade:** one tool, one config, one order of magnitude faster. Format is not debated in review (Clarity C2).
|
||||
- **Rule selection is principled, not "everything":** `select = ["E", "F", "I", "UP", "B", "SIM"]` — each rule group has a one-line `# reason:` in `pyproject.toml`. Rules without a rationale are noise (Documentation P1 — docs are code).
|
||||
- **`ruff format` is the formatter, `ruff check` is the linter:** run both in CI; the formatter is deterministic, the linter surfaces smells.
|
||||
- **Applies `devops/P2`:** the format/lint gate runs on every push; a developer never waits for a reviewer to comment on style.
|
||||
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[tool.ruff]
|
||||
target-version = "py311"
|
||||
line-length = 100
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "UP", "B", "SIM", "RUF"]
|
||||
# reason: E/F = pyflakes+pycodestyle; I = isort; UP = pyupgrade; B = bugbear; SIM = simplification
|
||||
|
||||
[tool.ruff.format]
|
||||
quote-style = "double"
|
||||
```
|
||||
|
||||
## mypy and Type-Check Gate (DevOps P2 Automation, Data P7 Type Fidelity)
|
||||
|
||||
- **`mypy --strict` in CI, not in the editor:** strict flags (`disallow_untyped_defs`, `no_implicit_optional`, `warn_return_any`) are the floor. The editor runs a relaxed mypy for speed; CI runs strict as the gate.
|
||||
- **`pyright` for stricter/async-aware checking:** pyright understands `async` better and reports faster; mypy is the standard. Pick one as the gate, run the other as informational.
|
||||
- **Per-module overrides only with a tracked reason:** `[[tool.mypy.overrides]] module = "legacy.*" ignore_errors = true` — each override block links to a ticket. Untracked overrides accumulate into a permanently untyped core.
|
||||
- **`py.typed` marker for libraries:** ships the type info to consumers. Without it, downstream mypy treats the library as `Any`.
|
||||
|
||||
```bash
|
||||
# CI gate
|
||||
mypy --strict src/
|
||||
pyright src/ || true # informational
|
||||
```
|
||||
|
||||
## Dependency Management: poetry and uv (DevOps P1 Reproducibility)
|
||||
|
||||
- **`poetry` or `uv` for lockfile discipline:** both produce a deterministic lock (`poetry.lock` / `uv.lock`). `pip install` alone does not — it resolves at install time, producing different trees across machines.
|
||||
- **`uv` for speed (Rust-based, 10–100x faster):** newer tool, same lockfile semantics. Either is acceptable; do not mix within a repo.
|
||||
- **Lockfile committed for applications:** for libraries, commit the lock for CI reproducibility even though consumers resolve their own tree.
|
||||
- **`--frozen` install in CI:** `poetry install --no-dev --frozen` fails if the lock is out of sync. Prevents a "works on my machine" drift.
|
||||
|
||||
```bash
|
||||
# CI install — deterministic
|
||||
uv sync --frozen --no-dev
|
||||
# or
|
||||
poetry install --no-dev --frozen
|
||||
```
|
||||
|
||||
## Virtualenv Discipline (DevOps P1 Reproducibility, C3 Simplicity)
|
||||
|
||||
- **One virtualenv per project, never the system Python:** `uv venv` or `python -m venv .venv`. System Python drift breaks reproducibility.
|
||||
- **`uv` creates and pins the Python version:** `uv venv --python 3.12` ensures the same interpreter across machines. A pinned Python is part of the reproducibility contract, not just the lockfile.
|
||||
- **No `pip install` into the system Python in CI:** use `uv`/`poetry`'s venv. A CI step that mutates system Python makes the next job non-hermetic.
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||
|
||||
- **`mkdocs` + `mkdocstrings` from docstrings:** API docs are generated from `google`- or `numpy`-style docstrings; the build fails on missing docstrings for public symbols (Documentation P1).
|
||||
- **`doctest` blocks in docstrings are run by pytest:** a `>>>` example is a tested artifact; a stale example fails the build (Documentation P1, Testing P1).
|
||||
- **`pyproject.toml` is the single source of tool config:** ruff, mypy, pytest, poetry all read from it. Do not scatter `.flake8`, `setup.cfg`, `mypy.ini`. One config file is one place to look (Clarity C2).
|
||||
|
||||
```python
|
||||
def get_user(id: UUID) -> User:
|
||||
"""Fetch a user by id.
|
||||
|
||||
Args:
|
||||
id: the user's UUID.
|
||||
|
||||
Returns:
|
||||
The User.
|
||||
|
||||
Raises:
|
||||
NotFoundError: if the user does not exist.
|
||||
|
||||
Example:
|
||||
>>> get_user(UUID('intentional-example-uuid'))
|
||||
User(...)
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/devops/ci-cd.md` — the pipeline gates that host ruff/mypy/poetry.
|
||||
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||
- `languages/py-types.md` — the type rules mypy enforces reference this doc.
|
||||
- `languages/py-testing.md` — the pytest config (`pyproject.toml [tool.pytest]`) detailed here.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Python Type System — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Python's type system specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/python.md` for the language first-principles stub.
|
||||
|
||||
## Type Hints and Gradual Typing (C1 Correctness, Data P7 Type Fidelity)
|
||||
|
||||
- **Type hints on every function signature:** `def get_user(id: UUID) -> User | None:`. Hints are annotations, not enforcement, but `mypy`/`pyright` make them a build gate.
|
||||
- **Gradual typing is opt-in, not opt-out:** start with `--strict` on a package, fix the errors, then expand. A repo-wide `# type: ignore` is a gradual-typing failure.
|
||||
- **`from __future__ import annotations` for forward refs:** all annotations are strings until resolved, so `class User: ...` referencing `User` works without quotes in 3.10+.
|
||||
- **`Any` disables the checker; `object` is the wide type:** `Any` allows any operation; `object` requires narrowing. Use `object` for opaque inputs (e.g., `json.loads` return).
|
||||
- **Applies `data/P7` (type fidelity):** a hint is the contract; the checker verifies it. A missing hint is a missing contract.
|
||||
|
||||
- **Pydantic for runtime validation at the boundary:** hints on `BaseModel` fields are validated at construction, catching bad input from network/config before it deepens into the system.
|
||||
- **`extra='forbid'` by default:** Pydantic allows extra fields silently; forbid them to surface schema drift (e.g., a client sending a typo'd field name).
|
||||
- **Custom types via `Annotated` with validators:** `Email = Annotated[str, validate_email]` keeps the type readable and the validator attached to the type, not the model.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
from uuid import UUID
|
||||
|
||||
class UserCreate(BaseModel):
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
email: str
|
||||
name: str
|
||||
|
||||
class User(UserCreate):
|
||||
id: UUID
|
||||
```
|
||||
|
||||
## Pydantic and Schema Fidelity (C1 Correctness, API P1 Contract Fidelity, Data P7 Type Fidelity)
|
||||
|
||||
- **Pydantic models are the API contract:** a FastAPI handler taking `UserCreate` rejects malformed JSON with a 422 before the body runs. This is `api/P1` (contract fidelity) at the type boundary.
|
||||
- **`ConfigDict(extra='forbid')` rejects unknown fields:** silently accepting extras is a contract leak — the server appears to handle fields it ignores.
|
||||
- **Validators raise `ValueError`, not `Exception`:** Pydantic converts `ValueError` to a validation error response; a generic `Exception` becomes a 500 and hides the input bug.
|
||||
- **Applies `api/P1`:** the model is the source of truth; the OpenAPI schema is generated from it, not hand-written. Drift between schema and code is impossible.
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from pydantic import BaseModel, Field, StringConstraints
|
||||
|
||||
EmailStr = Annotated[str, StringConstraints(pattern=r'^[^@\s]+@[^@\s]+$')]
|
||||
|
||||
class Login(BaseModel):
|
||||
email: EmailStr
|
||||
password: Annotated[str, Field(min_length=8)]
|
||||
```
|
||||
|
||||
## Errors as Data (Errors P1 Errors are Data, C1 Correctness)
|
||||
|
||||
- **`Union[T, Error]` over `Optional[T]` for expected failures:** `Optional[User]` cannot distinguish "not found" from "permission denied". A discriminated `Result` carries the cause.
|
||||
- **Custom exception hierarchy rooted at `AppError`:** `class NotFoundError(AppError)` etc. — callers can `except AppError` for the broad case, or a specific subclass for handling.
|
||||
- **`raise` for exceptional paths, `return Result` for expected:** "user not found" is expected (a `Result`); "DB connection lost" is exceptional (a `raise`). Conflating them makes error handling a guess.
|
||||
- **Applies `errors/P1`:** errors are values, not control-flow magic. A `Result` type encodes this at the type level even where exceptions are the runtime mechanism.
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from typing import Generic, TypeVar, Union
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Ok(Generic[T]):
|
||||
value: T
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Err:
|
||||
error: Exception
|
||||
|
||||
Result = Union[Ok[T], Err]
|
||||
|
||||
def find_user(id: UUID) -> Result[User]:
|
||||
row = db.get(id)
|
||||
if row is None:
|
||||
return Err(NotFoundError(f'user {id}'))
|
||||
return Ok(User.from_row(row))
|
||||
```
|
||||
|
||||
## Generics and Protocols (C6 Composability, C1 Correctness)
|
||||
|
||||
- **`Protocol` for structural typing (PEP 544):** a `Repository` protocol defines `get`/`save` without requiring an inheritance hierarchy; any class matching the shape satisfies it.
|
||||
- **`TypeVar` with bounds for generic functions:** `T = TypeVar('T', bound=Entity)` lets `serialize(t: T) -> dict` access `t.id`.
|
||||
- **`Generic[T]` for container types:** a typed `Repository[T]` preserves the element type across `get`/`save`, rather than widening to `Any`.
|
||||
- **`@overload` for callable overloads:** `def parse(s: str) -> int: ...` vs `def parse(s: bytes) -> int: ...` — the runtime body is one function; the overloads are the type contract.
|
||||
|
||||
```python
|
||||
from typing import Protocol, TypeVar
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
class Repository(Protocol[T]):
|
||||
def get(self, id: str) -> T | None: ...
|
||||
def save(self, t: T) -> None: ...
|
||||
|
||||
class UserRepo:
|
||||
def get(self, id: str) -> User | None: ...
|
||||
def save(self, u: User) -> None: ...
|
||||
|
||||
def use_repo(r: Repository[User]) -> None:
|
||||
u = r.get('abc') # type: User | None
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/data/schema-design.md` — Pydantic models parallel schema design at the TS/JSON boundary.
|
||||
- `domains/data/first-principles.md` — Data P7 Type Fidelity is the primary trace for this doc.
|
||||
- `domains/api/rest.md` — contract fidelity for FastAPI handlers consuming Pydantic models.
|
||||
- `domains/errors/patterns.md` — the `Result` discriminated union as error-as-data encoding.
|
||||
- `languages/py-async.md` — typed async results built on the `Result` union here.
|
||||
- `languages/py-tooling.md` — the `mypy`/`pyright` config that enforces these hints.
|
||||
@@ -2,6 +2,13 @@
|
||||
|
||||
> How Atelier's domain principles apply in Python specifically. Derives from `domains/` docs.
|
||||
|
||||
## Derived Docs
|
||||
|
||||
- [py-types.md](py-types.md) — type hints + Pydantic, mypy/pyright, gradual typing.
|
||||
- [py-tooling.md](py-tooling.md) — ruff, mypy, poetry, uv, virtualenv discipline.
|
||||
- [py-async.md](py-async.md) — asyncio, anyio, cancellation, structured concurrency.
|
||||
- [py-testing.md](py-testing.md) — pytest, factory_boy, fixture discipline, parametrize.
|
||||
|
||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||
|
||||
- **Type hints on every function:** `def get_user(id: UUID) -> User | None:`.
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
# Rust Async — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Rust async specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/rust.md` for the language first-principles stub.
|
||||
|
||||
## tokio and the Async Runtime (Concurrency P5 Lock Minimization, C6 Composability)
|
||||
|
||||
- **`tokio` is the default async runtime:** `#[tokio::main]` for the entry; `tokio::spawn` for a task. The runtime owns the reactor, the I/O driver, and the timer.
|
||||
- **`tokio::spawn` returns a `JoinHandle` like `std::thread::spawn`:** a dropped `JoinHandle` detaches (the task keeps running); `await` the handle to join. Prefer await to detach.
|
||||
- **`tokio::task::JoinSet` for structured concurrency:** a set of tasks awaited together; on drop, all remaining tasks are cancelled. Mirrors `errgroup`/`TaskGroup` semantics.
|
||||
- **`runtime` features are explicit:** `tokio = { version = "1", features = ["full"] }` for a binary; `["rt", "rt-multi-thread", "macros"]` for a library. Pulling `full` into a library bloats downstream.
|
||||
|
||||
```rust
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for id in ["a", "b", "c"] {
|
||||
set.spawn(fetch_user(id.to_string()));
|
||||
}
|
||||
while let Some(res) = set.join_next().await {
|
||||
match res {
|
||||
Ok(Ok(u)) => println!("{}", u.name),
|
||||
Ok(Err(e)) => eprintln!("err: {e}"),
|
||||
Err(join_err) => eprintln!("panic: {join_err}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Async Traits (Concurrency P7 Cancellation Support, C6 Composability)
|
||||
|
||||
- **`async fn` in traits stabilized in Rust 1.75:** `trait Repo { async fn get(&self, id: &str) -> Result<User, Error>; }`. No `async-trait` crate needed for new code on recent toolchains.
|
||||
- **`Box<dyn Trait>` with async methods needs `dyn`-compatibility:** the returned future is `Pin<Box<dyn Future>>`; the compiler boxes it. For hot paths, use generics (`impl Trait`) over `dyn`.
|
||||
- **`async-trait` crate for older toolchains:** macro that desugars to a `Pin<Box<dyn Future>>`. Migrate to native `async fn in trait` when the toolchain allows.
|
||||
- **`Send` bounds on async traits for cross-thread spawn:** `trait Repo: Send { async fn get(&self, id: &str) -> Result<User, Error>; }` — the returned future must be `Send` to spawn on a multi-thread runtime.
|
||||
|
||||
```rust
|
||||
trait UserRepo: Send + Sync {
|
||||
async fn get(&self, id: &str) -> Result<User, Error>;
|
||||
}
|
||||
|
||||
struct PgRepo { pool: PgPool }
|
||||
impl UserRepo for PgRepo {
|
||||
async fn get(&self, id: &str) -> Result<User, Error> {
|
||||
sqlx::query_as::<_, User>("SELECT * FROM users WHERE id = $1")
|
||||
.bind(id).fetch_one(&self.pool).await.map_err(Error::from)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Cancellation (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
|
||||
|
||||
- **Cancellation is cooperative via dropping the future:** `tokio::select!` drops the unselected branch, cancelling it. A dropped future stops at its next `.await` point.
|
||||
- **`tokio::time::timeout` for a deadline:** `timeout(Duration::from_secs(5), op).await` returns `Ok(Ok(v))` on success, `Ok(Err(e))` on inner error, `Err(Elapsed)` on timeout. Every external `await` races against a deadline (Concurrency P8).
|
||||
- **`tokio::select!` for cancel-aware waits:** `select! { res = op => res, _ = cancel => return Err(Cancelled), }`. The unselected branch is dropped, cancelling it.
|
||||
- **Cancellation is not atomic:** a future dropped mid-`await` may have partial state. `Drop` runs on cancellation; clean up there (e.g., rollback a transaction).
|
||||
- **Applies `concurrency/P7`:** cancellation is a first-class signal; the runtime propagates it via drop. No `CancelledError` to catch — the future is gone.
|
||||
|
||||
```rust
|
||||
use tokio::time::timeout;
|
||||
use std::time::Duration;
|
||||
|
||||
async fn fetch_with_timeout(url: &str) -> Result<Response, Error> {
|
||||
match timeout(Duration::from_secs(5), fetch(url)).await {
|
||||
Ok(Ok(r)) => Ok(r),
|
||||
Ok(Err(e)) => Err(e.into()),
|
||||
Err(_elapsed) => Err(Error::Timeout),
|
||||
}
|
||||
}
|
||||
|
||||
async fn cancellable(op: impl Future<Output=()>, mut cancel: tokio::sync::oneshot::Receiver<()>) {
|
||||
tokio::select! {
|
||||
_ = op => {},
|
||||
_ = &mut cancel => println!("cancelled"),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Pin and Self-Referential Futures (Concurrency P5 Lock Minimization, C1 Correctness)
|
||||
|
||||
- **`async fn` returns a `Future` that is often self-referential:** the generated state machine may hold a borrow into its own stack. Such a future must be `Pin`ned to move safely.
|
||||
- **`Pin<Box<T>>` to box and pin:** `Box::pin(async { ... })` returns a `Pin<Box<dyn Future>>`. The cost is a heap alloc; the win is `Send`/`dyn`-compatibility.
|
||||
- **`Pin<&mut T>` for in-place polling:** `Pin::new(&mut fut)` pins a stack future; the borrow checker prevents moving it. Use for stack-allocated futures in `select!`.
|
||||
- **Do not `unsafe` unpin:** `Pin::get_unchecked_mut` opts out of the pin guarantees. Application code never needs it; library code uses it for `poll` implementations.
|
||||
|
||||
```rust
|
||||
use std::pin::Pin;
|
||||
|
||||
async fn boxed() -> Pin<Box<dyn std::future::Future<Output = ()> + Send>> {
|
||||
Box::pin(async {
|
||||
// self-referential state machine is safe to move once pinned
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Bounded Channels and Backpressure (Concurrency P9 Bounded Queues)
|
||||
|
||||
- **`tokio::sync::mpsc::channel(N)` is bounded:** `send().await` blocks when full (backpressure, Concurrency P9). Unbounded `unbounded_channel()` lets the producer run ahead and OOM.
|
||||
- **`tokio::sync::mpsc::Sender::try_send` for non-blocking send:** returns `Err(TrySendError::Full(v))` when full; the caller decides to drop, log, or back off. A bounded queue + `try_send` is the backpressure-aware pattern.
|
||||
- **`tokio::sync::broadcast` for fan-out:** multiple receivers each get a copy; a slow receiver misses (lag). Use for telemetry, not for commands.
|
||||
- **Applies `messaging/queues`:** a bounded tokio channel is an in-process broker — bounded buffer, backpressure, at-most-once handoff. The same semantics apply; the broker is local.
|
||||
|
||||
```rust
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
async fn producer(tx: mpsc::Sender<Job>) {
|
||||
for j in jobs() {
|
||||
if tx.send(j).await.is_err() { return; } // receiver dropped
|
||||
}
|
||||
}
|
||||
|
||||
async fn consumer(rx: mpsc::Receiver<Job>) {
|
||||
while let Some(j) = rx.recv().await {
|
||||
process(j).await;
|
||||
}
|
||||
}
|
||||
|
||||
let (tx, rx) = mpsc::channel::<Job>(16); // bounded: backpressure
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P5 Lock Minimization, P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
|
||||
- `domains/messaging/delivery-semantics.md` — at-most-once vs at-least-once framing for async retry/cancel (IDEATE-40).
|
||||
- `languages/rs-ownership.md` — `Send`/`Sync` bounds on futures build on the ownership model here.
|
||||
- `languages/rs-tooling.md` — `tokio` feature flags and the `cargo` build profiles detailed there.
|
||||
- `languages/rs-testing.md` — `#[tokio::test]` and async test patterns.
|
||||
@@ -0,0 +1,136 @@
|
||||
# Rust Ownership — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Rust's ownership model specifically. Rust's distinctive strength (Send/Sync, lifetimes, borrowing) earns a dedicated ownership doc rather than an `rs-types.md`.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/rust.md` for the language first-principles stub.
|
||||
|
||||
## Ownership and Move Semantics (Concurrency P1 Immutability by Default, C1 Correctness)
|
||||
|
||||
- **Ownership is unique:** at any time, exactly one owner holds a value. Assignment passes ownership (`let y = x;` — `x` is moved, not copied). The compiler rejects use-after-move.
|
||||
- **`Copy` types (integers, `bool`, `&T`) duplicate on assignment; everything else moves.** A `struct` is `Copy` only if all fields are; opt in via `#[derive(Copy, Clone)]` only for small, cheap-to-copy types.
|
||||
- **Pass by `&T` for read-only, `&mut T` for mutation:** a borrow does not transfer ownership; the caller retains the value after the callee returns.
|
||||
- **Applies `concurrency/P1` (immutability by default):** `&T` is shared and immutable; `&mut T` is exclusive and mutable. The compiler enforces "one or many, never both" — aliasing XOR mutation, statically.
|
||||
|
||||
```rust
|
||||
let s = String::from("hello");
|
||||
let t = s; // s moved into t
|
||||
// println!("{}", s); // error: use of moved value
|
||||
|
||||
let n = 5;
|
||||
let m = n; // i32 is Copy: n still usable
|
||||
println!("{} {}", n, m);
|
||||
```
|
||||
|
||||
## Borrowing and Lifetimes (C1 Correctness, Data P7 Type Fidelity, Concurrency P3 Boundaries are Locks)
|
||||
|
||||
- **`&'a T` ties a borrow to a lifetime `'a`:** the borrow cannot outlive the owner. Lifetimes are static — the compiler rejects dangling references.
|
||||
- **Lifetime elision when unambiguous:** `fn first<'a>(s: &'a str) -> &'a str` is elided to `fn first(s: &str) -> &str` (one input → output lifetime). When ambiguous, name the lifetime.
|
||||
- **`'static` is the longest lifetime (the whole program):** not "until I drop it." Use `'static` only for values that genuinely live forever (string literals, `const`s); leaking to `'static` to satisfy the checker is a bug.
|
||||
- **`Ref<'a, T>` and `RefMut<'a, T>` from `RefCell` are runtime-checked borrows:** the borrow rules still apply, checked at runtime instead of compile time. A second `RefMut` panics.
|
||||
- **Applies `concurrency/P3` (boundaries are locks):** `&mut T` is the compile-time lock — exclusive access is the boundary; no runtime mutex needed for single-threaded aliasing discipline.
|
||||
|
||||
```rust
|
||||
fn longest<'a>(a: &'a str, b: &'a str) -> &'a str {
|
||||
if a.len() > b.len() { a } else { b } // borrow tied to both inputs
|
||||
}
|
||||
|
||||
fn dangling() -> &str { // compile error: missing lifetime
|
||||
let s = String::from("local");
|
||||
&s // error: s drops at end of fn
|
||||
}
|
||||
```
|
||||
|
||||
## Send and Sync (Concurrency P1 Immutability by Default, Concurrency P3 Boundaries are Locks, C1 Correctness)
|
||||
|
||||
- **`Send`:** a type `T: Send` may be moved across thread boundaries. Most types are `Send`; `Rc<T>` is not (shared non-atomically refcounted).
|
||||
- **`Sync`:** a type `T: Sync` may be shared (`&T`) across threads. `RefCell<T>` is `!Sync` (interior mutability without atomics); `Mutex<T>` is `Sync` (it synchronizes).
|
||||
- **The compiler enforces `Send`/`Sync` at the thread-spawn boundary:** `std::thread::spawn(move || { ... })` requires the closure's captures to be `Send`.
|
||||
- **Applies `concurrency/P1` and `concurrency/P3`:** `Send` is the move-across-boundary contract; `Sync` is the share-across-boundary contract. Data races are a compile error, not a runtime detector. This is Rust's distinctive strength over Go's race detector.
|
||||
|
||||
```rust
|
||||
use std::rc::Rc;
|
||||
use std::sync::Arc;
|
||||
|
||||
let rc = Rc::new(5);
|
||||
// std::thread::spawn(move || { println!("{}", rc) }); // error: Rc is !Send
|
||||
|
||||
let arc = Arc::new(5);
|
||||
std::thread::spawn(move || { println!("{}", arc) }); // ok: Arc<T> is Send+Sync
|
||||
```
|
||||
|
||||
## Shared Mutation: Arc, Mutex, RwLock (Concurrency P3 Boundaries are Locks, Concurrency P5 Lock Minimization)
|
||||
|
||||
- **`Arc<T>` for shared ownership across threads:** atomic refcounted. Clone increases the count; the last drop frees `T`.
|
||||
- **`Mutex<T>` for exclusive mutation across threads:** `lock()` blocks until exclusive; the guard `MutexGuard<T>` derefs to `&mut T` and releases on drop.
|
||||
- **`RwLock<T>` for read-heavy, `Mutex<T>` for write-heavy:** RwLock allows multiple readers or one writer. For most cases, `Mutex` is simpler and faster; prefer it unless reads dominate by 10x+.
|
||||
- **Hold the lock for the smallest scope:** `let g = m.lock().unwrap();` then drop `g` before I/O. RAII releases on scope exit; explicit `drop(g)` clarifies intent.
|
||||
- **Applies `concurrency/P5` (lock minimization):** prefer message passing (`mpsc` channels) over locks. When a lock is needed, scope it minimally.
|
||||
|
||||
```rust
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::thread;
|
||||
|
||||
let counter = Arc::new(Mutex::new(0));
|
||||
let mut handles = vec![];
|
||||
for _ in 0..10 {
|
||||
let c = Arc::clone(&counter);
|
||||
handles.push(thread::spawn(move || {
|
||||
let mut g = c.lock().unwrap();
|
||||
*g += 1;
|
||||
// g drops here, lock released
|
||||
}));
|
||||
}
|
||||
for h in handles { h.join().unwrap(); }
|
||||
println!("{}", *counter.lock().unwrap());
|
||||
```
|
||||
|
||||
## Interior Mutability (Concurrency P1 Immutability by Default, C1 Correctness)
|
||||
|
||||
- **`Cell<T>` for `Copy` types, `RefCell<T>` for non-`Copy`:** interior mutability moves the borrow check from compile time to runtime. `RefCell::borrow_mut()` panics on a second mutable borrow.
|
||||
- **`Mutex<T>`/`RwLock<T>` for thread-safe interior mutability:** the runtime check is the lock, not a panic. Use these across threads; `RefCell` only single-threaded.
|
||||
- **`UnsafeCell<T>` is the primitive; never use directly:** `Cell`, `RefCell`, `Mutex` are safe wrappers. Direct `UnsafeCell` is `unsafe` and opts out of the aliasing guarantee.
|
||||
- **Applies `concurrency/P1`:** interior mutability is the exception, not the default. Reach for it when an API must present `&self` while mutating internally (e.g., a cache); document why.
|
||||
|
||||
```rust
|
||||
use std::cell::RefCell;
|
||||
|
||||
struct Cache {
|
||||
inner: RefCell<HashMap<String, User>>,
|
||||
}
|
||||
impl Cache {
|
||||
fn get(&self, id: &str) -> Option<User> {
|
||||
// &self (immutable) but mutates internally
|
||||
self.inner.borrow_mut().entry(id.to_string()).or_insert_with(|| fetch()).clone()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Drop and RAII (C1 Correctness, Concurrency P3 Boundaries are Locks)
|
||||
|
||||
- **`Drop` runs when the owner goes out of scope:** no `defer`, no `finally`. A `MutexGuard` releases, a `File` closes, a `JoinHandle`... does not join (a dropped `JoinHandle` detaches).
|
||||
- **`Drop` is deterministic:** it runs at scope exit, not GC time. This is why `Arc`'s refcount is precise and `Mutex` release is timely.
|
||||
- **`ManuallyDrop<T>` to opt out:** for FFI types whose destructor you must call manually. Rare in application code; common in `unsafe` bindings.
|
||||
- **`Drop` order: fields in declaration order, then the struct itself.** A field that another field's `Drop` depends on must be declared last.
|
||||
|
||||
```rust
|
||||
struct Resource { name: String }
|
||||
impl Drop for Resource {
|
||||
fn drop(&mut self) {
|
||||
println!("dropping {}", self.name); // runs at scope end
|
||||
}
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let _r = Resource { name: "x".into() };
|
||||
// _r drops here, prints "dropping x"
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P1 Immutability, P3 Boundaries are Locks, P5 Lock Minimization.
|
||||
- `domains/data/first-principles.md` — Data P7 Type Fidelity (lifetimes are the type-level fidelity for references).
|
||||
- `domains/concurrency/patterns.md` — message-passing vs lock patterns applied via `Arc`/`Mutex`/`mpsc`.
|
||||
- `domains/errors/patterns.md` — `?` propagation relies on ownership transfer of the error.
|
||||
- `languages/rs-async.md` — async borrows (`Pin`/`&mut`) build on the lifetime model here.
|
||||
- `languages/rs-testing.md` — `Send`/`Sync` tests and ownership-based property tests.
|
||||
@@ -0,0 +1,159 @@
|
||||
# Rust Testing — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Rust testing specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/rust.md` for the language first-principles stub.
|
||||
|
||||
## #[test] and Co-located Tests (Testing P1 Tests as Specification, C2 Clarity)
|
||||
|
||||
- **`#[test]` on functions in a `#[cfg(test)] mod tests` block:** tests co-located with source, compiled only in `cargo test`. A test file far from its subject rots (Documentation P5 Discoverability).
|
||||
- **Test names read as a spec:** `fn create_user_rejects_invalid_email()` — a reader understands the unit from the name. Avoid `fn test_user_1()`.
|
||||
- **`assert!` / `assert_eq!` / `assert_ne!` over raw `panic!`:** the macros produce readable failure output (`assertion failed: left == right, left: 5, right: 3`). Raw `panic!` gives a message only (Testing P6 Failure Specificity).
|
||||
- **Applies `Testing P1`:** the test is a specification; the failure message is the spec violation.
|
||||
|
||||
```rust
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn create_user_rejects_invalid_email() {
|
||||
let r = create_user("not-an-email");
|
||||
assert!(matches!(r, Err(Error::Validation(_))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn create_user_returns_persisted_id() {
|
||||
let u = create_user("a@b.co").unwrap();
|
||||
assert!(!u.id.is_empty());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## proptest and Property Tests (Testing P9 Edge Case Coverage, Testing P1 Tests as Specification)
|
||||
|
||||
- **`proptest` (or `quickcheck`) for invariant tests:** declare a property (`parse(serialize(x)) == x`), the framework generates hundreds of inputs and shrinks failures to a minimal counterexample (Testing P9).
|
||||
- **Strategy over hand-written generators:** `proptest::collection::vec(any::<u32>(), 0..100)` generates arbitrary `Vec<u32>`; do not hand-roll a generator for each property.
|
||||
- **`proptest!` macro or `proptest! { ... }` block:** each `case (name) => { ... }` is a property. The block is the spec (Testing P1).
|
||||
- **Property tests complement, not replace, example tests:** examples document the happy path; properties cover the edge space. Both are required.
|
||||
|
||||
```rust
|
||||
use proptest::prelude::*;
|
||||
|
||||
proptest! {
|
||||
#[test]
|
||||
fn roundtrips_id(s in "[a-z0-9]{1,32}") {
|
||||
let id = UserId::new(&s).unwrap();
|
||||
assert_eq!(id.as_str(), s);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_invalid_id(s in "[^a-z0-9]+") {
|
||||
assert!(UserId::new(&s).is_err());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Mock Discipline (Testing P2 Independence, Testing P7 Realism)
|
||||
|
||||
- **Mock at the trait, not the struct:** `trait Store { fn get(&self, id: &str) -> Result<User, Error>; }` in production; `#[automock] trait Store` (via `mockall`) in test. The trait is the contract.
|
||||
- **`mockall` for generated mocks:** `#[automock] trait Repo {}` generates `MockRepo` with `expect_*` methods. Each expectation is per-test; no shared mock state (Testing P2 Independence).
|
||||
- **Mock the boundary, not the unit:** mock `Repo`, not `UserService` (the unit). Mocking the unit under test tests the mock (Testing P7 realism).
|
||||
- **No `#[cfg(test)]` on production code paths to inject mocks:** instead, accept the trait as a generic or `dyn` parameter. Test-only branches in production code are dead code in prod.
|
||||
|
||||
```rust
|
||||
use mockall::*;
|
||||
|
||||
#[automock]
|
||||
trait UserRepo {
|
||||
fn get(&self, id: &str) -> Result<User, Error>;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn get_user_returns_not_found() {
|
||||
let mut repo = MockUserRepo::new();
|
||||
repo.expect_get()
|
||||
.with(eq("abc"))
|
||||
.returning(|_| Err(Error::NotFound));
|
||||
let svc = UserService::new(Box::new(repo));
|
||||
assert!(matches!(svc.get_user("abc"), Err(Error::NotFound)));
|
||||
}
|
||||
```
|
||||
|
||||
## Async Tests (Concurrency P10 Test for Race Conditions, Testing P1 Tests as Specification)
|
||||
|
||||
- **`#[tokio::test]` for `async fn` tests:** runs the coroutine on a tokio runtime. Without it, an `async fn` test returns a future, never awaited (silently passes).
|
||||
- **`#[tokio::test(flavor = "multi_thread")]` for concurrency-sensitive tests:** multi-thread runtime surfaces races that single-thread misses (Concurrency P10).
|
||||
- **`tokio::time::pause()` and `advance()` for time:** freeze and advance the runtime clock deterministically. No `tokio::time::sleep(real)` in tests.
|
||||
- **Race-sensitive tests use `loom` for model-checking:** `loom` simulates all thread interleavings; it catches races `-race`-style detectors miss. Use for lock-free data structures.
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn async_fetch_returns_user() {
|
||||
let u = fetch_user("abc").await.unwrap();
|
||||
assert!(!u.name.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn concurrent_cache_is_safe() {
|
||||
let c = Arc::new(Cache::new());
|
||||
let mut h = vec![];
|
||||
for i in 0..10 {
|
||||
let c = c.clone();
|
||||
h.push(tokio::spawn(async move { c.get(&i.to_string()).await; }));
|
||||
}
|
||||
for x in h { x.await.unwrap(); }
|
||||
}
|
||||
```
|
||||
|
||||
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||
|
||||
- **No `SystemTime::now()` or `Instant::now()` in code under test:** inject a `Clock` trait. In tests, a fake clock advances deterministically.
|
||||
- **`tokio::time::pause()` for async time:** freezes the runtime clock; `tokio::time::advance(dur)` moves it. A `sleep(5s)` in test resolves instantly.
|
||||
- **`--test-threads=1` to reproduce order coupling:** by default, `cargo test` runs tests in parallel; a test that passes alone but fails in a suite has hidden shared state. `-1` reproduces.
|
||||
|
||||
```rust
|
||||
trait Clock { fn now(&self) -> std::time::Instant; }
|
||||
|
||||
struct FakeClock(std::time::Instant);
|
||||
impl Clock for FakeClock {
|
||||
fn now(&self) -> std::time::Instant { self.0 }
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn user_has_created_at() {
|
||||
let clk = FakeClock(std::time::Instant::now());
|
||||
let u = create_user_with_clock("a@b.co", &clk).unwrap();
|
||||
assert_eq!(u.created_at, clk.now());
|
||||
}
|
||||
```
|
||||
|
||||
## Doc Tests (Documentation P1 Documentation is Code, Testing P1 Tests as Specification)
|
||||
|
||||
- **`cargo test --doc` runs `///` fenced blocks:** a `///` example with `#`-hidden setup is a tested artifact; a stale output fails the build (Documentation P1).
|
||||
- **`no_run` for examples that should compile but not run:** ```` ```rust,no_run ```` — type-checks the example without executing. Use for examples that need a DB.
|
||||
- **`ignore` for examples that should not compile-check:** ```` ```rust,ignore ```` — skips entirely. Rare; prefer `no_run`.
|
||||
- **Applies `Testing P1`:** the doc example is the spec; the doc test is the spec's regression test.
|
||||
|
||||
```rust
|
||||
/// Fetch a user by id.
|
||||
///
|
||||
/// # Example
|
||||
///
|
||||
/// ```
|
||||
/// # use mycrate::{get_user, Error};
|
||||
/// let u = get_user("abc").unwrap();
|
||||
/// assert!(!u.name.is_empty());
|
||||
/// ```
|
||||
pub fn get_user(id: &str) -> Result<User, Error> { /* ... */ }
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/testing/pyramid.md` — where unit/property/doc tests sit; proptest is the property layer.
|
||||
- `domains/testing/fixtures.md` — `t.Cleanup`-equivalent (`Drop` in tests) as fixture discipline.
|
||||
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P10 (test for races), `loom` model-checking.
|
||||
- `languages/rs-ownership.md` — `Send`/`Sync` tests and ownership-based property tests.
|
||||
- `languages/rs-async.md` — `#[tokio::test]` patterns from that doc.
|
||||
- `languages/rs-tooling.md` — `cargo test` flags (`--doc`, `--test-threads`) detailed here.
|
||||
@@ -0,0 +1,95 @@
|
||||
# Rust Tooling — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to Rust tooling specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/rust.md` for the language first-principles stub.
|
||||
|
||||
## cargo and Build Discipline (DevOps P2 Automation, DevOps P1 Reproducibility)
|
||||
|
||||
- **`cargo build` for dev, `cargo build --release` for release:** release enables optimizations (LTO, codegen-units=1). The default profile is for fast iteration, not perf.
|
||||
- **`Cargo.lock` committed for applications and CI:** for libraries, commit the lock for CI reproducibility even though consumers resolve their own tree. A drifted lock breaks reproducibility (DevOps P1).
|
||||
- **`cargo update` periodically, with a CI check:** `cargo update` bumps patch versions in the lock; a CI job that fails on lock drift catches a forgotten `cargo update`.
|
||||
- **`cargo vendor` for hermetic CI:** vendors `vendor/` into the repo; CI builds without network. The trade-off is repo size; the win is reproducibility.
|
||||
|
||||
```toml
|
||||
# Cargo.toml — profile discipline
|
||||
[profile.release]
|
||||
lto = true
|
||||
codegen-units = 1
|
||||
panic = "abort" # smaller binary, no unwinding
|
||||
```
|
||||
|
||||
## clippy (DevOps P2 Automation, C2 Clarity)
|
||||
|
||||
- **`cargo clippy` is the lint layer over `rustc`:** it catches `clone()` where a borrow would do, `unwrap()` in library code, and needless `Box`. Run on every build.
|
||||
- **`cargo clippy -- -D warnings` in CI:** warnings are errors. A clippy warning is a smell; accumulating them erodes the signal (Clarity C2).
|
||||
- **Per-lint allow only with a tracked reason:** `#[allow(clippy::needless_collect)] // reason: GH-123 — collect needed for len` — each allow links to a ticket. Untracked allows accumulate into a permanently lint-bypassed core.
|
||||
- **`cargo clippy --fix` for safe auto-fixes:** applies the linter's suggested change. Review the diff; do not run blindly on a large commit.
|
||||
|
||||
```bash
|
||||
# CI gate
|
||||
cargo clippy --all-targets --all-features -- -D warnings
|
||||
```
|
||||
|
||||
## cargo fmt (DevOps P2 Automation, C2 Clarity)
|
||||
|
||||
- **`cargo fmt` is the formatter; format is not debated in review:** run in CI as a check (`cargo fmt --check`), not a fix. A failing check blocks the PR.
|
||||
- **`rustfmt.toml` for repo-wide settings:** if the defaults are wrong for the repo, override once and stop. Do not relitigate per-PR.
|
||||
- **Applies `devops/P2`:** the format gate is automated; a reviewer never comments on style.
|
||||
|
||||
```bash
|
||||
# CI gate — fail if unformatted
|
||||
cargo fmt --check
|
||||
```
|
||||
|
||||
## Edition Discipline (DevOps P1 Reproducibility, C5 Reversibility)
|
||||
|
||||
- **`edition` in `Cargo.toml` pins the language edition:** 2015, 2018, 2021, 2024. An edition is a coherent set of language changes; bumping it is a deliberate migration.
|
||||
- **Edition is not the compiler version:** `rustc 1.75` supports edition 2021; edition 2024 needs a newer `rustc`. Pin the toolchain with `rust-toolchain.toml`.
|
||||
- **Bump editions deliberately, not opportunistically:** `cargo fix --edition` applies the migration lint; review the diff. A bump mid-feature conflates two changes.
|
||||
- **Applies `devops/P1` and `C5` (reversibility):** pinning the edition and toolchain makes the build reproducible; bumping is a controlled, reversible change.
|
||||
|
||||
```toml
|
||||
# Cargo.toml
|
||||
[package]
|
||||
edition = "2021"
|
||||
rust-version = "1.75"
|
||||
```
|
||||
|
||||
```toml
|
||||
# rust-toolchain.toml
|
||||
[toolchain]
|
||||
channel = "1.75"
|
||||
components = ["clippy", "rustfmt"]
|
||||
```
|
||||
|
||||
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||
|
||||
- **`cargo doc` from doc comments:** `///` on items generates API docs; `cargo doc --open` previews. The build fails on broken intra-doc links (`#![warn(rustdoc::broken_intra_doc_links)]`).
|
||||
- **Doc tests are run by `cargo test`:** a `///` fenced block with `#`-hidden setup is a tested artifact; a stale example fails `cargo test --doc` (Documentation P1).
|
||||
- **`#![warn(missing_docs)]` for libraries:** public items without doc comments fail the build. Documentation is a build gate, not an afterthought.
|
||||
- **`cargo readme` or `cargo docs-rs` for landing pages:** the crate's `README.md` is rendered on docs.rs; keep it in sync with `lib.rs`'s top-level doc.
|
||||
|
||||
```rust
|
||||
#![warn(missing_docs, rustdoc::broken_intra_doc_links)]
|
||||
|
||||
/// Fetch a user by id.
|
||||
///
|
||||
/// # Example
|
||||
///
|
||||
/// ```
|
||||
/// # use mycrate::get_user;
|
||||
/// let u = get_user("abc").unwrap();
|
||||
/// println!("{}", u.name);
|
||||
/// ```
|
||||
pub fn get_user(id: &str) -> Result<User, Error> { /* ... */ }
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/devops/ci-cd.md` — the pipeline gates that host clippy/fmt/test.
|
||||
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||
- `languages/rs-ownership.md` — `Send`/`Sync` clippy lints reference this doc.
|
||||
- `languages/rs-async.md` — async-runtime tooling (`tokio` features) detailed here.
|
||||
- `languages/rs-testing.md` — `cargo test` flags (`--doc`, `--no-run`) detailed here.
|
||||
@@ -2,6 +2,13 @@
|
||||
|
||||
> How Atelier's domain principles apply in Rust specifically. Derives from `domains/` docs.
|
||||
|
||||
## Derived Docs
|
||||
|
||||
- [rs-ownership.md](rs-ownership.md) — Send/Sync, lifetimes, borrowing, ownership transfer.
|
||||
- [rs-tooling.md](rs-tooling.md) — cargo, clippy, fmt, edition discipline.
|
||||
- [rs-async.md](rs-async.md) — tokio, async traits, cancellation, pin.
|
||||
- [rs-testing.md](rs-testing.md) — #[test], proptest, property testing, mock discipline.
|
||||
|
||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||
|
||||
- **Newtypes for domain concepts:** `struct UserId(String);` — zero-cost, type-safe.
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
# TypeScript Async — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to TypeScript async specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/typescript.md` for the language first-principles stub.
|
||||
|
||||
## Promises and AbortSignal (Concurrency P7 Cancellation Support, C1 Correctness)
|
||||
|
||||
- **Every async function accepts an optional `AbortSignal`:** cancellation is a first-class parameter, not a side channel. The signal propagates to `fetch`, `setTimeout`, and downstream awaits.
|
||||
- **`AbortController` is the producer side; `AbortSignal` is the consumer side:** a function takes a `signal` (read-only), the caller owns the `controller` and decides when to abort.
|
||||
- **Abort propagates as a rejected `Promise`:** `fetch` rejects with `AbortError`; downstream code sees the rejection, not a silent no-op. This preserves `errors/P5` (recoverable when possible) — the caller can distinguish cancellation from a real failure.
|
||||
- **Applies `concurrency/P7`:** no async operation runs without a path to cancel it. A long-running `await` with no signal is a hung request.
|
||||
- **Never swallow `AbortError`:** re-throw or handle distinctly; cancellation is the caller's intent, not an error to log.
|
||||
|
||||
```typescript
|
||||
async function fetchUser(id: UserId, signal?: AbortSignal): Promise<User> {
|
||||
const ctrl = new AbortController();
|
||||
signal?.addEventListener('abort', () => ctrl.abort());
|
||||
const res = await fetch(`/users/${id}`, { signal: ctrl.signal });
|
||||
if (!res.ok) throw new HttpError(res.status);
|
||||
return res.json() as Promise<User>;
|
||||
}
|
||||
|
||||
// caller controls cancellation
|
||||
const ctrl = new AbortController();
|
||||
const timer = setTimeout(() => ctrl.abort(), 5000);
|
||||
try {
|
||||
const u = await fetchUser(id, ctrl.signal);
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
```
|
||||
|
||||
## async/await Discipline (Concurrency P8 Timeout Discipline, C2 Clarity)
|
||||
|
||||
- **`await` is the only async primitive in application code:** no `.then` chains, no callback pyramids. `async`/`await` reads top-to-bottom (Clarity C2).
|
||||
- **Never `await` in a hot loop without batching:** sequential `await` in a `for` loop is O(n) latency. Use `Promise.all` for parallelism; `for await...of` only for genuine streams.
|
||||
- **`Promise.race` for a timeout:** every external `await` has a deadline. `Promise.race([op, timeout])` rejects when the deadline passes.
|
||||
- **`return` vs `return await`:** inside `try`/`finally`, `return await` runs the `finally`; bare `return` of a Promise defers the `finally` to the microtask. Prefer `return await` when cleanup must run.
|
||||
- **Applies `concurrency/P8`:** a bare `await` with no timeout is an unbounded wait. External calls (network, disk) always race against a deadline.
|
||||
|
||||
```typescript
|
||||
async function fetchWithTimeout(url: string, ms = 5000, signal?: AbortSignal): Promise<Response> {
|
||||
const ctrl = new AbortController();
|
||||
signal?.addEventListener('abort', () => ctrl.abort());
|
||||
const timer = new Promise<never>((_, reject) =>
|
||||
setTimeout(() => reject(new TimeoutError(ms)), ms)
|
||||
);
|
||||
try {
|
||||
return await Promise.race([fetch(url, { signal: ctrl.signal }), timer]);
|
||||
} finally {
|
||||
clearTimeout(timer); // cleanup runs on success and on race-loss
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling in Async (Errors P5 Recoverable When Possible, Errors P1 Errors are Data)
|
||||
|
||||
- **Catch `unknown`, narrow with a type guard:** `catch (e: unknown)` — TS does not infer the error type. `instanceof` or a discriminator narrows it.
|
||||
- **Retry with backoff for transient failures:** network blips are recoverable (Errors P5). Exponential backoff with jitter, capped retry count, and an `AbortSignal`-aware `setTimeout`.
|
||||
- **No retry for non-idempotent operations:** a `POST` that creates a resource is not safely retryable without an idempotency key (applies `api/P6` Idempotency).
|
||||
- **Typed errors over `Error` subclasses:** a discriminated union `AppError = Network | Timeout | Cancelled` carries context (Errors P4 Preserve Context) without `instanceof` chains.
|
||||
|
||||
```typescript
|
||||
async function fetchRetry(url: string, attempts = 3, signal?: AbortSignal): Promise<Response> {
|
||||
for (let i = 0; i < attempts; i++) {
|
||||
try {
|
||||
return await fetchWithTimeout(url, 5000, signal);
|
||||
} catch (e: unknown) {
|
||||
if (e instanceof AbortError) throw e; // do not retry cancellation
|
||||
if (e instanceof TimeoutError && i < attempts - 1) {
|
||||
await sleep(jitter(i), signal); // backoff before retry
|
||||
continue;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
throw new Error('unreachable');
|
||||
}
|
||||
```
|
||||
|
||||
## Cancellation Propagation (Concurrency P7 Cancellation Support, Concurrency P9 Bounded Queues)
|
||||
|
||||
- **One signal, many consumers:** pass the same `AbortSignal` to every async call in a request. Aborting once cancels the whole tree.
|
||||
- **Bounded concurrency with a semaphore:** a `Semaphore(N)` wrapping `Promise.all` caps in-flight requests (Concurrency P9 — bounded queues). Unbounded `Promise.all` on a 10k-item array exhausts file descriptors.
|
||||
- **Cancellation is cooperative, not preemptive:** a long synchronous block inside an `async` function ignores the signal. Yield with `await Promise.resolve()` periodically in CPU-bound loops, or move to a worker.
|
||||
- **Applies `messaging/delivery-semantics`:** a cancelable async operation is an at-most-once delivery — the caller may stop listening, the result may or may not arrive. Retry-on-cancel is at-least-once; the caller must declare which.
|
||||
|
||||
```typescript
|
||||
async function mapBounded<T, U>(items: readonly T[], fn: (t: T, s: AbortSignal) => Promise<U>, limit = 8, signal?: AbortSignal): Promise<U[]> {
|
||||
const ctrl = new AbortController();
|
||||
signal?.addEventListener('abort', () => ctrl.abort());
|
||||
const results: U[] = new Array(items.length);
|
||||
let next = 0;
|
||||
const workers = Array.from({ length: limit }, async () => {
|
||||
while (true) {
|
||||
const i = next++;
|
||||
if (i >= items.length) break;
|
||||
if (ctrl.signal.aborted) throw new AbortError();
|
||||
results[i] = await fn(items[i], ctrl.signal);
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
return results;
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
|
||||
- `domains/concurrency/first-principles.md` — Concurrency P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
|
||||
- `domains/messaging/delivery-semantics.md` — at-most-once vs at-least-once framing for async retry/cancel (IDEATE-40).
|
||||
- `domains/errors/patterns.md` — typed async errors and retry-with-backoff.
|
||||
- `languages/ts-types.md` — `Result<T, E>` and discriminated `AppError` used in async error handling.
|
||||
- `languages/ts-tooling.md` — `no-floating-promises` lint rule that enforces these awaits.
|
||||
@@ -0,0 +1,117 @@
|
||||
# TypeScript Testing — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to TypeScript testing specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/typescript.md` for the language first-principles stub.
|
||||
|
||||
## Vitest and Jest (Testing P1 Tests as Specification, C2 Clarity)
|
||||
|
||||
- **Vitest for new TS projects; Jest for legacy:** Vitest shares `vite`'s transform pipeline (no separate `ts-jest` config); Jest's ecosystem is broader. Either is acceptable — pick one per repo, do not mix.
|
||||
- **Tests co-located with source:** `user.ts` → `user.test.ts`. A test file far from its subject rots (Documentation P5 Discoverability).
|
||||
- **`describe`/`it` mirror the public API:** the test block names read as a specification ("User", "rejects an invalid email", "returns the persisted id"). A reader should understand the unit from test names alone (Testing P1).
|
||||
- **`expect` over `assert`:** Vitest/Jest matchers produce readable failure output (`expect(x).toBe(y)` → "expected 5, received 3"). Raw `assert` gives a stack trace and nothing else (Testing P6 Failure Specificity).
|
||||
|
||||
```typescript
|
||||
// user.test.ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { createUser } from './user';
|
||||
|
||||
describe('createUser', () => {
|
||||
it('rejects an invalid email', async () => {
|
||||
await expect(createUser({ email: 'not-an-email' })).rejects.toThrow(ValidationError);
|
||||
});
|
||||
it('returns the persisted id', async () => {
|
||||
const u = await createUser({ email: 'a@b.co' });
|
||||
expect(u.id).toMatch(/^[a-z0-9]+$/);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## Mock Discipline (Testing P2 Independence, Testing P7 Realism)
|
||||
|
||||
- **Mock at the boundary, not the unit:** replace `fetch` or the DB client, not the function under test. Mocking the unit under test tests the mock, not the code (Testing P7 — realism).
|
||||
- **No partial mocks of the system under test:** if a method must be stubbed, the unit is too large. Extract a collaborator and mock that.
|
||||
- **Each test sets up and tears down its own state:** no shared mutable fixtures. A `beforeEach`/`afterEach` resets; a top-level `let` shared across tests is order-coupling (Testing P2 Independence).
|
||||
- **`vi.useFakeTimers()` for time-dependent code:** never call `Date.now()` directly in code under test; inject a `Clock` port. In tests, fake timers make `setTimeout` synchronous.
|
||||
|
||||
```typescript
|
||||
import { vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
global.fetch = vi.fn(); // boundary mock
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
```
|
||||
|
||||
## Type-Level Tests (Testing P1 Tests as Specification, Data P7 Type Fidelity)
|
||||
|
||||
- **Type-level tests assert the type system, not runtime behavior:** `expectTypeOf<T>().toMatchTypeOf<U>` and `tsd`/`expect-type` fail the build when a type assertion is wrong.
|
||||
- **Negative type tests are required:** `// @ts-expect-error` proves the compiler rejects what it should. A `@ts-expect-error` that no longer errors is itself an error (the comment must be consumed).
|
||||
- **Branded types and utility types get type tests:** a `UserId` should not be assignable to `string`; a `Readonly<T>` should not allow assignment. These invariants are part of the spec (Testing P1).
|
||||
- **Applies `data/P7` (type fidelity):** a type-level test is a regression test for the type checker — if a refactor silently widens a type, the test fails.
|
||||
|
||||
```typescript
|
||||
import { expectTypeOf } from 'expect-type';
|
||||
import type { User, UserPatch, UserId } from './user';
|
||||
|
||||
test('UserPatch omits id and makes fields optional', () => {
|
||||
expectTypeOf<UserPatch>().toMatchTypeOf<{ name?: string; email?: string }>();
|
||||
expectTypeOf<UserPatch>().not.toHaveProperty('id');
|
||||
});
|
||||
|
||||
test('UserId is not assignable to bare string', () => {
|
||||
// @ts-expect-error — brand prevents widening
|
||||
const s: string = {} as UserId;
|
||||
expect(s).toBeDefined();
|
||||
});
|
||||
```
|
||||
|
||||
## Parametrize and Factories (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||
|
||||
- **`it.each` / `test.each` for parametrized cases:** one table drives many runs; each row is an independent test with its own name and failure output.
|
||||
- **Factories over fixtures:** `makeUser(overrides)` returns a fresh object per call. A shared `const user = {...}` across tests couples them and breaks determinism when one test mutates it (Testing P3).
|
||||
- **Edge cases as rows, not special tests:** empty array, single element, max int, null, undefined — each a row in a `test.each` table. An ad-hoc `it('handles edge')` with multiple asserts hides which case failed (Testing P9 — edge case coverage, P6 failure specificity).
|
||||
- **Property-style tests via `fast-check`:** for invariants (e.g., "parse(serialize(x)) === x"), `fast-check` generates hundreds of inputs and shrinks failures to a minimal counterexample.
|
||||
|
||||
```typescript
|
||||
import { test, expect } from 'vitest';
|
||||
import { makeUser } from './user.factory';
|
||||
|
||||
test.each([
|
||||
{ input: '', reason: 'empty' },
|
||||
{ input: 'a'.repeat(1000), reason: 'too long' },
|
||||
{ input: 'not-an-email', reason: 'no @' },
|
||||
])('rejects email: $reason', async ({ input }) => {
|
||||
await expect(makeUser({ email: input })).rejects.toThrow(ValidationError);
|
||||
});
|
||||
```
|
||||
|
||||
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
|
||||
|
||||
- **No `Date.now()`, `Math.random()`, or `crypto.randomUUID()` in code under test:** inject a `Clock`, `Random`, and `IdGen` port. In tests, provide deterministic fakes.
|
||||
- **`--random` test order (Vitest `sequence.shuffle: true` default) catches order coupling:** a test that passes alone but fails in a suite has hidden state. The shuffle makes that state visible (Testing P2).
|
||||
- **Race-detector parallelism for async tests:** run async tests concurrently by default; a test that assumes serial execution breaks under parallelism. Vitest's `concurrent` flag surfaces the bug.
|
||||
|
||||
```typescript
|
||||
import { vi, test, expect } from 'vitest';
|
||||
|
||||
test.concurrent('parallel fetch does not interleave state', async () => {
|
||||
const store = new Store();
|
||||
await Promise.all([store.put('a', 1), store.put('b', 2)]);
|
||||
expect(store.get('a')).toBe(1);
|
||||
expect(store.get('b')).toBe(2);
|
||||
});
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/testing/pyramid.md` — where unit/type/integration tests sit; the type-level tests here are the base layer.
|
||||
- `domains/testing/fixtures.md` — factory-vs-fixture discipline applied via `makeUser`.
|
||||
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
|
||||
- `languages/ts-types.md` — the branded types and utility types that type-level tests assert.
|
||||
- `languages/ts-async.md` — async tests use the cancellation/timeout patterns from that doc.
|
||||
- `languages/ts-tooling.md` — `ts-jest`/`vitest` config and the `expect-type`/`tsd` toolchain.
|
||||
@@ -0,0 +1,114 @@
|
||||
# TypeScript Tooling — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to TypeScript tooling specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/typescript.md` for the language first-principles stub.
|
||||
|
||||
## tsc and tsconfig Discipline (DevOps P2 Automation, DevOps P1 Reproducibility)
|
||||
|
||||
- **`strict: true` is the floor, not the ceiling:** it enables `strictNullChecks`, `noImplicitAny`, `strictFunctionTypes`, and more. Disable sub-flags only with a justification comment.
|
||||
- **`tsc --noEmit` in CI:** type-checking is a build gate; emission is the bundler's job. Separate the two so a type error fails CI even when the bundler would have succeeded.
|
||||
- **`tsconfig` is per-project, not inherited verbatim:** a shared base (`extends`) encodes org defaults; each project overrides the deltas it needs. Avoids the "one monoreto-config-fits-all" trap.
|
||||
- **`noUncheckedIndexedAccess` for safety:** `arr[i]` becomes `T | undefined`, forcing narrowing. Costs little, prevents a class of out-of-bounds deref bugs.
|
||||
- **Applies `devops/P1` (reproducibility):** pinned `typescript` version in `package.json` and `lockfile` ensure every CI run type-checks against the same compiler.
|
||||
|
||||
```jsonc
|
||||
// tsconfig.json — base
|
||||
{
|
||||
"compilerOptions": {
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"exactOptionalPropertyTypes": true,
|
||||
"noEmit": true,
|
||||
"moduleResolution": "bundler",
|
||||
"isolatedModules": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## ESLint and @typescript-eslint (DevOps P2 Automation, Documentation P9 Living Documents)
|
||||
|
||||
- **ESLint with `@typescript-eslint` strict ruleset:** `recommended-type-checked` enables rules that require the type checker (`no-floating-promises`, `no-misused-promises`).
|
||||
- **Rules encode decisions, not taste:** every custom rule in the config has a one-line `// reason:` comment linking to the principle it enforces. This makes the config a living document (Documentation P9).
|
||||
- **Format is Prettier's job; ESLint lints:** `eslint-config-prettier` disables conflicting format rules. Do not relitigate formatting in code review.
|
||||
- **`no-floating-promises` enforces `concurrency/P8` (timeout discipline):** an un-awaited `Promise` is a fire-and-forget that swallows errors and timeouts. The rule forces `.catch()` or `await`.
|
||||
|
||||
```jsonc
|
||||
// .eslintrc.json
|
||||
{
|
||||
"extends": [
|
||||
"eslint:recommended",
|
||||
"plugin:@typescript-eslint/recommended-type-checked",
|
||||
"prettier"
|
||||
],
|
||||
"parserOptions": { "project": "./tsconfig.json" },
|
||||
"rules": {
|
||||
// reason: enforce Concurrency P8 — no un-awaited promises
|
||||
"@typescript-eslint/no-floating-promises": "error",
|
||||
// reason: enforce Data P7 — no `any` escaping the type checker
|
||||
"@typescript-eslint/no-explicit-any": "error"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Project References and ts-jest (DevOps P2 Automation, C6 Composability)
|
||||
|
||||
- **Project references for monorepos:** `composite: true` + `references` let `tsc --build` incrementally type-check only changed projects, and enforce the dependency graph at the type level.
|
||||
- **`paths` aliases mirror the import structure:** `@app/*` → `src/*`. Configure once in `tsconfig`, mirror in the bundler and the test runner so all three agree.
|
||||
- **`ts-jest` (or `vitest`) with `isolatedModules: true`:** each test file is type-checked in isolation, matching how the bundler transpiles. Catches the "passes in `tsc` but fails in the bundler" gap.
|
||||
- **Applies `devops/P2`:** the build pipeline (tsc → lint → test → bundle) is automated; a developer never runs a manual sequence.
|
||||
|
||||
```jsonc
|
||||
// tsconfig.references.json
|
||||
{
|
||||
"files": [],
|
||||
"references": [
|
||||
{ "path": "./packages/core" },
|
||||
{ "path": "./packages/api" },
|
||||
{ "path": "./packages/web" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Lockfile and Reproducible Install (DevOps P1 Reproducibility)
|
||||
|
||||
- **`npm ci` in CI, not `npm install`:** `ci` reads the lockfile exactly and fails on drift. `install` mutates the lockfile.
|
||||
- **Lockfile committed for applications:** for libraries, commit `package-lock.json` for CI reproducibility even though consumers resolve their own tree.
|
||||
- **No floating ranges in `package.json`:** `^` and `~` are CI's job to resolve; pin the resolved version in the lockfile. An unpinned `*` is a supply-chain attack surface.
|
||||
|
||||
```bash
|
||||
# CI install step — deterministic
|
||||
npm ci
|
||||
# Type-check gate
|
||||
npx tsc --noEmit
|
||||
# Lint gate
|
||||
npx eslint .
|
||||
```
|
||||
|
||||
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
|
||||
|
||||
- **Type-checked JSDoc:** `typedoc` (or `TypeDoc`) generates API docs from `tsdoc` comments. The compiler enforces that `@param` names match real parameters.
|
||||
- **`@example` blocks are compiled:** a `tsdoc` `@example` fenced block is type-checked as part of the doc build. Stale examples fail the pipeline (Documentation P1 — docs are code).
|
||||
- **README badges reflect CI status:** the build/lint/test/type-check gates are the source of truth; badges surface them. Do not hand-edit status tables.
|
||||
|
||||
```typescript
|
||||
/**
|
||||
* Fetch a user by ID.
|
||||
*
|
||||
* @param id - a branded UserId (see ts-types.md).
|
||||
* @throws {NotFoundError} if the user does not exist.
|
||||
* @example
|
||||
* ```ts
|
||||
* const u = await getUser(userId('abc'));
|
||||
* ```
|
||||
*/
|
||||
async function getUser(id: UserId): Promise<User> { /* ... */ }
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/devops/ci-cd.md` — the pipeline gates that host tsc/ESLint/ts-jest.
|
||||
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
|
||||
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
|
||||
- `languages/ts-types.md` — the type rules ESLint enforces reference this doc.
|
||||
- `languages/ts-testing.md` — the test-runner config (`ts-jest`/`vitest`) detailed here.
|
||||
@@ -0,0 +1,111 @@
|
||||
# TypeScript Type System — Derived Application
|
||||
|
||||
> Applies Atelier's domain principles to TypeScript's type system specifically.
|
||||
> Derives from `domains/` docs; introduces no new P-rules (D-063).
|
||||
> See `languages/typescript.md` for the language first-principles stub.
|
||||
|
||||
## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)
|
||||
|
||||
- **TypeScript is structurally typed:** two types with the same shape are assignable. This is convenient but erases domain boundaries — a `UserId` and `PostId` both `string` are interchangeable.
|
||||
- **Branded (nominal) types for domain IDs:** intersect with a phantom brand to simulate nominal typing. The brand is never constructed at runtime; it exists only to the type checker.
|
||||
- **Applies `data/P7` (type fidelity)** at the value boundary: a branded `UserId` cannot be passed where a `PostId` is expected, preventing an entire class of swap bugs.
|
||||
- **Applies `api/P1` (contract fidelity):** branded types make API contracts explicit — handlers cannot accept "any string" for an ID.
|
||||
- **Brand is opaque to consumers:** do not export the brand symbol; construction goes through a validated factory.
|
||||
|
||||
```typescript
|
||||
type UserId = string & { readonly __brand: 'UserId' };
|
||||
type PostId = string & { readonly __brand: 'PostId' };
|
||||
|
||||
function userId(s: string): UserId {
|
||||
if (!/^[a-zA-Z0-9]+$/.test(s)) throw new Error('invalid id');
|
||||
return s as UserId;
|
||||
}
|
||||
|
||||
function getUser(id: UserId): User { /* ... */ }
|
||||
getUser('abc'); // type error
|
||||
getUser(userId('abc')); // ok
|
||||
getUser(postId('xyz')); // type error — distinct brands
|
||||
```
|
||||
|
||||
## Generics (C6 Composability, Data P7 Type Fidelity)
|
||||
|
||||
- **Generics preserve type information across boundaries:** a `Repository<T>` keeps the element type through `find`/`save` rather than widening to `any`.
|
||||
- **Constrain with `extends`:** `<T extends Entity>` documents the contract and gives the body access to `T.id`.
|
||||
- **Avoid unnecessary generics:** if a function accepts "any value and returns it unchanged," `T` is noise. Prefer `unknown` for truly opaque inputs.
|
||||
- **Variance is structural:** TS does not enforce sound variance; mark mutation points with `readonly` to keep `T[]` assignable to `readonly T[]`.
|
||||
|
||||
```typescript
|
||||
interface Entity { id: string }
|
||||
class Repository<T extends Entity> {
|
||||
constructor(private db: Map<string, T>) {}
|
||||
find(id: string): T | undefined { return this.db.get(id); }
|
||||
save(t: T): void { this.db.set(t.id, t); }
|
||||
}
|
||||
```
|
||||
|
||||
## Narrowing and Type Guards (C1 Correctness, Errors P1 Errors are Data)
|
||||
|
||||
- **Narrowing is how TS handles `unknown` and union types safely:** `typeof`, `in`, `instanceof`, and discriminators collapse a wide type to a precise one before use.
|
||||
- **User-defined type guards (`x is T`) encode domain predicates:** `isUser(x): x is User` lets the checker track the narrow across call sites.
|
||||
- **Applies `errors/P1` (errors are data):** a `Result<T, E>` discriminated union is narrowed with `if (r.ok)` — no `try`/`catch` needed for expected failures.
|
||||
- **Never use `as` to widen past a check:** `as` lies to the compiler. If narrowing does not reach the type you need, the predicate is wrong, not the cast.
|
||||
|
||||
```typescript
|
||||
type Result<T, E> = { ok: true; value: T } | { ok: false; error: E };
|
||||
|
||||
function unwrap<T, E>(r: Result<T, E>): T {
|
||||
if (r.ok) return r.value; // narrowed to { ok: true; value: T }
|
||||
throw new Error(JSON.stringify(r.error));
|
||||
}
|
||||
|
||||
function isUser(x: unknown): x is User {
|
||||
return typeof x === 'object' && x !== null && 'id' in x && 'name' in x;
|
||||
}
|
||||
```
|
||||
|
||||
## Utility Types (C5 Reversibility, C6 Composability)
|
||||
|
||||
- **`Partial<T>`, `Pick<T,K>`, `Omit<T,K>`, `Readonly<T>` are derived views:** they derive from a source-of-truth `T` rather than redeclaring fields, so the source change propagates (reversibility).
|
||||
- **`Readonly<T>` enforces immutability at the type level** — applies `concurrency/P1` (immutability by default) without runtime cost.
|
||||
- **`Record<K, V>` over `{ [k: string]: V }`:** the index signature form allows any string key including prototype pollution vectors; `Record` is exact.
|
||||
- **Compose, don't accumulate:** `type Patch<T> = Partial<Omit<T, 'id'>>` reads as a transformation; restate it if `T` changes shape, rather than maintaining a parallel `Patch` type.
|
||||
|
||||
```typescript
|
||||
interface User { id: string; name: string; email: string; }
|
||||
type UserPatch = Partial<Omit<User, 'id'>>;
|
||||
type ReadonlyUser = Readonly<User>;
|
||||
type UsersById = Record<string, User>;
|
||||
```
|
||||
|
||||
## Discriminated Unions (C1 Correctness, Data P7 Type Fidelity, Errors P1 Errors are Data)
|
||||
|
||||
- **Discriminated unions over enums:** `type Status = { type: 'pending' } | { type: 'paid'; amount: number }` is exhaustive and carries payload per variant; an `enum` carries neither.
|
||||
- **The discriminant is a literal `type` (or `kind`) field:** the checker narrows on it in `switch` and `if` without a custom guard.
|
||||
- **Exhaustiveness via `never`:** assign the narrowed value to `never` in the default branch; if a variant is added, the default fails to compile.
|
||||
- **Applies `errors/P1`:** model domain errors as a discriminated union `AppError = NotFound | Validation | Conflict`, not as exception classes — the type system carries the error set.
|
||||
|
||||
```typescript
|
||||
type Status =
|
||||
| { type: 'pending' }
|
||||
| { type: 'paid'; amount: number }
|
||||
| { type: 'refunded'; reason: string };
|
||||
|
||||
function describe(s: Status): string {
|
||||
switch (s.type) {
|
||||
case 'pending': return 'awaiting payment';
|
||||
case 'paid': return `paid ${s.amount}`;
|
||||
case 'refunded': return `refunded: ${s.reason}`;
|
||||
default:
|
||||
const _exhaustive: never = s; // compile error if a variant is added
|
||||
throw new Error('unhandled');
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Cross-References
|
||||
|
||||
- `domains/data/schema-design.md` — schema-level fidelity parallels branded types at the TS boundary.
|
||||
- `domains/data/first-principles.md` — Data P7 Type Fidelity, the primary trace for this doc.
|
||||
- `domains/api/rest.md` — contract fidelity for API handlers consuming branded IDs.
|
||||
- `domains/errors/patterns.md` — discriminated unions as the error-as-data encoding.
|
||||
- `languages/ts-async.md` — typed async results built on the `Result` union here.
|
||||
@@ -2,6 +2,13 @@
|
||||
|
||||
> How Atelier's domain principles apply in TypeScript specifically. Derives from `domains/` docs; this file is the language-specific lens.
|
||||
|
||||
## Derived Docs
|
||||
|
||||
- [ts-types.md](ts-types.md) — TS type system: nominal-via-branding, generics, narrowing, utility types, discriminated unions.
|
||||
- [ts-tooling.md](ts-tooling.md) — tsc, ESLint, ts-jest, project references, tsconfig discipline.
|
||||
- [ts-async.md](ts-async.md) — Promises + AbortSignal, async/await, error handling, cancellation.
|
||||
- [ts-testing.md](ts-testing.md) — Vitest/Jest, mock discipline, type-level tests.
|
||||
|
||||
## Type System (C1 Correctness, Data P7 Type Fidelity)
|
||||
|
||||
- **Strict mode on:** `strict: true` in `tsconfig.json`. No `any` without justification.
|
||||
|
||||
@@ -6,14 +6,14 @@
|
||||
|
||||
| Core Principle | Domains that derive from it | Count |
|
||||
|----------------|---------------------------|-------|
|
||||
| C1 Correctness | All 17 (v0.1: 11; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance) | Universal |
|
||||
| C2 Clarity | v0.1: uiux, api, data, testing, observability, errors, documentation, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 14 |
|
||||
| C3 Simplicity | v0.1: security, data, testing, performance, documentation, concurrency, devops; v0.2: infrastructure-as-code; v0.3: gitops-operators, i18n, compliance | 11 |
|
||||
| C4 Locality | v0.1: testing, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, i18n | 6 |
|
||||
| C5 Reversibility | v0.1: api, data, uiux, concurrency, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 11 |
|
||||
| C6 Composability | v0.1: api, security, observability, errors, documentation, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 12 |
|
||||
| C7 Observability | v0.1: api, data, testing, performance, observability, errors, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance | 13 |
|
||||
| C8 Economy | v0.1: security, testing, performance, observability, concurrency; v0.2: kubernetes; v0.3: gitops-operators, i18n, compliance | 9 |
|
||||
| C1 Correctness | All 19 (v0.1: 11; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging) | Universal |
|
||||
| C2 Clarity | v0.1: uiux, api, data, testing, observability, errors, documentation, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: messaging | 16 |
|
||||
| C3 Simplicity | v0.1: security, data, testing, performance, documentation, concurrency, devops; v0.2: infrastructure-as-code; v0.3: gitops-operators, i18n, compliance; v0.4: edge | 12 |
|
||||
| C4 Locality | v0.1: testing, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, i18n; v0.4: edge, messaging | 8 |
|
||||
| C5 Reversibility | v0.1: api, data, uiux, concurrency, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging | 13 |
|
||||
| C6 Composability | v0.1: api, security, observability, errors, documentation, concurrency; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging | 14 |
|
||||
| C7 Observability | v0.1: api, data, testing, performance, observability, errors, devops; v0.2: infrastructure-as-code, kubernetes; v0.3: gitops-operators, ai-ml, i18n, compliance; v0.4: edge, messaging | 15 |
|
||||
| C8 Economy | v0.1: security, testing, performance, observability, concurrency; v0.2: kubernetes; v0.3: gitops-operators, i18n, compliance; v0.4: edge, messaging | 11 |
|
||||
|
||||
## Interpretation
|
||||
|
||||
@@ -43,6 +43,8 @@
|
||||
| AI / ML | C1, C2, C5, C6, C7 | Reproducibility + lineage + serving observability |
|
||||
| i18n | C1, C2, C3, C4, C5, C6, C7, C8 | Locale + formatting + direction + reversibility; broadest derivation (8 C-rules) |
|
||||
| Compliance | C1, C2, C3, C5, C6, C7, C8 | Audit + policy-as-code + retention + posture |
|
||||
| Edge | C1, C3, C4, C5, C6, C7, C8 | Proximity + offline + constraint; broad derivation (7 C-rules) |
|
||||
| Messaging | C1, C2, C4, C5, C6, C7, C8 | Cross-process delivery + ordering + idempotency; broad derivation (7 C-rules) |
|
||||
|
||||
## v0.2 Domain Coverage (per IDEATE-03 schema)
|
||||
|
||||
@@ -60,6 +62,39 @@
|
||||
| i18n | 10 | 4 (locale-resources, formatting, rtl-bidi, testing-i18n) | ✓ | complete |
|
||||
| Compliance | 10 | 4 (audit-logs, data-retention, policy-as-code, evidence) | ✓ | complete |
|
||||
|
||||
## v0.4 Domain Coverage (per IDEATE-03 schema)
|
||||
|
||||
| Domain | P-count | Derived-doc-count | Manifest-listed | Status |
|
||||
|--------|---------|-------------------|-----------------|--------|
|
||||
| Edge | 10 | 4 (cdn, offline-first, iot, sync) | ✓ | complete |
|
||||
| Messaging | 10 | 4 (queues, pubsub, streams, delivery-semantics) | ✓ | complete |
|
||||
|
||||
## Languages Coverage (v0.4, per D-066)
|
||||
|
||||
> Language-derived docs apply existing domain P-rules. They introduce
|
||||
> no new matrix rows (D-063, D-066). The sub-table is the verifiable
|
||||
> aggregate; inline section-headers are the per-doc detail. Every
|
||||
> derived doc traces to ≥1 domain P-rule (D-066, no orphans).
|
||||
|
||||
| Language | Derived Doc | Traced Domain P-rules | Traced Core C-rules |
|
||||
|----------|-------------|------------------------|---------------------|
|
||||
| TypeScript | ts-types.md | Data P7, API P1, Errors P1 | C1, C5, C6 |
|
||||
| TypeScript | ts-tooling.md | DevOps P2, DevOps P1, Documentation P1, DevOps P9 | C1, C2, C6, C8 |
|
||||
| TypeScript | ts-async.md | Concurrency P7, Concurrency P8, Concurrency P9, Errors P5, Errors P1 | C1, C2, C5 |
|
||||
| TypeScript | ts-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Data P7 | C1, C2 |
|
||||
| Python | py-types.md | Data P7, API P1, Errors P1 | C1, C2, C6 |
|
||||
| Python | py-tooling.md | DevOps P2, DevOps P1, Documentation P1, DevOps P9 | C1, C2, C3, C8 |
|
||||
| Python | py-async.md | Concurrency P1, Concurrency P7, Concurrency P8, Concurrency P9, Errors P5, Errors P1 | C1, C2, C5, C6 |
|
||||
| Python | py-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Concurrency P10 | C1, C2 |
|
||||
| Go | go-types.md | Data P7, API P1, Errors P1, Errors P3 | C1, C2, C6 |
|
||||
| Go | go-tooling.md | DevOps P2, DevOps P1, Concurrency P6, Documentation P1, DevOps P9 | C1, C2, C3, C7, C8 |
|
||||
| Go | go-concurrency.md | Concurrency P1, Concurrency P3, Concurrency P5, Concurrency P6, Concurrency P7, Concurrency P8, Concurrency P9 | C1, C3, C4, C5, C6, C8 |
|
||||
| Go | go-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Concurrency P6, Concurrency P10, API P1 | C1, C2 |
|
||||
| Rust | rs-ownership.md | Concurrency P1, Concurrency P3, Concurrency P5, Data P7 | C1, C3, C4 |
|
||||
| Rust | rs-tooling.md | DevOps P2, DevOps P1, Documentation P1, DevOps P9 | C1, C2, C3, C5, C8 |
|
||||
| Rust | rs-async.md | Concurrency P5, Concurrency P7, Concurrency P8, Concurrency P9 | C1, C3, C5, C6, C8 |
|
||||
| Rust | rs-testing.md | Testing P1, Testing P2, Testing P3, Testing P7, Testing P9, Concurrency P10, Documentation P1 | C1, C2 |
|
||||
|
||||
## Gaps and Notes
|
||||
|
||||
- No domain derives from only one C-rule. The minimum is 4 (UI/UX: C1, C2, C3, C5, C7 — actually 5). Every domain is multi-rooted.
|
||||
@@ -68,4 +103,6 @@
|
||||
- **UI/UX** and **API** are the most user-facing; they emphasize C2 (Clarity) heavily.
|
||||
- **Security** is the only domain with explicit non-tradeable declarations; this promotes 8 of its rules to C1-equivalent per `core/conflict-resolution.md` §6.
|
||||
- **v0.2 expansion:** C4 (Locality) grew from 2 to 4 domains (added infrastructure-as-code state locality, kubernetes namespace blast-radius). C6 (Composability) grew from 6 to 8. The two new domains are broad-derivation domains (7 C-rules each), consistent with Concurrency's breadth.
|
||||
- **v0.3 expansion:** C3 (Simplicity) grew from 7 to 11 (added gitops-operators declarative simplicity, i18n flexible layout, compliance structural redaction). C4 (Locality) grew from 4 to 6 (added gitops-operators pull-credential locality, i18n resource/text-direction locality). C5 (Reversibility) grew from 7 to 11 (added all four v0.3 domains — gitops history, ai-ml reproducibility, i18n translation versioning, compliance append-only/retention). C6 (Composability) grew from 8 to 12. C7 (Observability) grew from 9 to 13 (added all four v0.3 domains — reconciliation, drift detection, format correctness, posture). C2 (Clarity) grew from 10 to 14. The v0.3 expansion broadens every non-universal C-rule's coverage, confirming the four new domains are cross-cutting and well-rooted.
|
||||
- **v0.3 expansion:** C3 (Simplicity) grew from 7 to 11 (added gitops-operators declarative simplicity, i18n flexible layout, compliance structural redaction). C4 (Locality) grew from 4 to 6 (added gitops-operators pull-credential locality, i18n resource/text-direction locality). C5 (Reversibility) grew from 7 to 11 (added all four v0.3 domains — gitops history, ai-ml reproducibility, i18n translation versioning, compliance append-only/retention). C6 (Composability) grew from 8 to 12. C7 (Observability) grew from 9 to 13 (added all four v0.3 domains — reconciliation, drift detection, format correctness, posture). C2 (Clarity) grew from 10 to 14. The v0.3 expansion broadens every non-universal C-rule's coverage, confirming the four new domains are cross-cutting and well-rooted.
|
||||
- **v0.4 expansion:** C2 (Clarity) grew from 14 to 16 (added messaging P1/P2/P4 — contract clarity). C4 (Locality) grew from 6 to 8 (added edge P1/P8 geographic locality, messaging P7 partition locality). C5 (Reversibility) grew from 11 to 13 (added edge P2/P4/P7/P10 reconciliation + degradation, messaging P5/P8/P9 replay + evolution). C6 (Composability) grew from 12 to 14 (added edge P8 fleet composition, messaging P7/P9 partition + version composition). C7 (Observability) grew from 13 to 15 (added edge P10 partition-survivable telemetry, messaging P8/P10 durable-log + lag). C8 (Economy) grew from 9 to 11 (added edge P3/P9 constrained resources + trust, messaging P6 bounded backpressure). C3 (Simplicity) grew from 11 to 12 (added edge P6 explicit invalidation simplicity). Both v0.4 domains are broad-derivation (7 C-rules each), consistent with the v0.2/v0.3 precedent.
|
||||
- **Languages Coverage (v0.4, D-066):** The 16 language-derived docs (4 languages × 4 docs) apply existing domain P-rules; they introduce no new matrix rows (D-063, D-066). The sub-table above is the verifiable aggregate — every derived doc traces to ≥1 domain P-rule (no orphans). Languages are an application lens, not a principle source.
|
||||
@@ -265,8 +265,39 @@ C5=Reversibility · C6=Composability · C7=Observability · C8=Economy
|
||||
| P9 Secrets and Sensitive Data are Redacted in Audit | C1, C3 | Correctness via security; simplicity of structural redaction |
|
||||
| P10 Compliance Posture is Observable | C7, C1 | Observability of compliance; correctness of posture |
|
||||
|
||||
## Coverage Summary (post-v0.3)
|
||||
## Edge
|
||||
|
||||
- 17 domains (11 v0.1 + 2 v0.2: infrastructure-as-code, kubernetes; 4 v0.3: gitops-operators, ai-ml, i18n, compliance)
|
||||
- 170 domain principles total (110 v0.1 + 20 v0.2 + 40 v0.3)
|
||||
- Every domain P-rule traces to ≥1 core C-rule (C1–C8). No orphans.
|
||||
| Edge Principle | Core | Why |
|
||||
|-------------------------------------------------------|-------|--------------------------------------------------|
|
||||
| P1 Proximity is the Design Driver | C4, C1 | Locality of compute near user/data; correctness via latency |
|
||||
| P2 Offline is a First-Class State | C1, C5 | Correctness under partition; reversibility of reconciliation |
|
||||
| P3 Resources are Constrained and Declared | C8, C1 | Economy of constrained nodes; correctness of declared bounds |
|
||||
| P4 Sync Conflicts are Bounded, Not Infinite | C1, C5 | Correctness of convergence; reversibility of divergent state |
|
||||
| P5 Edge Operations are Idempotent | C1 | Correctness under retry |
|
||||
| P6 Cache Invalidation is Explicit | C1, C3 | Correctness of cached state; simplicity of defined invalidation |
|
||||
| P7 Partial Degradation is Engineered | C1, C5 | Correctness of degraded modes; reversibility of recovery |
|
||||
| P8 Geographic Distribution is a First-Class Constraint | C4, C6 | Locality of placement; composability of the fleet |
|
||||
| P9 Identity is Constrained at the Edge | C1, C8 | Correctness via security; economy of trust |
|
||||
| P10 Edge Observability Survives Partition | C7, C5 | Observability of the fleet; reversibility of buffered telemetry |
|
||||
|
||||
## Messaging
|
||||
|
||||
| Messaging Principle | Core | Why |
|
||||
|--------------------------------------------------|-------|--------------------------------------------------|
|
||||
| P1 Messages are Contracts | C1, C2 | Correctness of the exchange; clarity of the schema boundary |
|
||||
| P2 Ordering is a Property, Not an Assumption | C1, C2 | Correctness of order; clarity of the guarantee |
|
||||
| P3 Consumers are Idempotent | C1 | Correctness under redelivery |
|
||||
| P4 Delivery Semantics are Explicit | C1, C2 | Correctness of the chosen semantic; clarity of the tradeoff |
|
||||
| P5 Dead-Letter Handling is Defined | C1, C5 | Correctness of poison-message routing; reversibility of reprocessing |
|
||||
| P6 Backpressure is Bounded | C1, C8 | Correctness of bounded backlog; economy of broker memory |
|
||||
| P7 Partitioning is Intentional | C4, C6 | Locality of order; composability of parallelism |
|
||||
| P8 Replay and Retention are Configured | C5, C7 | Reversibility of replay; observability of the durable log |
|
||||
| P9 Schemas Evolve Compatibly | C5, C6 | Reversibility of schema changes; composability of versions |
|
||||
| P10 Messaging is Observable | C7, C1 | Observability of lag/DLQ; correctness of backlog detection |
|
||||
|
||||
## Coverage Summary (post-v0.4)
|
||||
|
||||
- 19 domains (11 v0.1 + 2 v0.2: infrastructure-as-code, kubernetes; 4 v0.3: gitops-operators, ai-ml, i18n, compliance; 2 v0.4: edge, messaging)
|
||||
- 190 domain principles total (110 v0.1 + 20 v0.2 + 40 v0.3 + 20 v0.4)
|
||||
- Every domain P-rule traces to ≥1 core C-rule (C1–C8). No orphans.
|
||||
- Language-derived docs (16) apply existing domain P-rules; they introduce no new matrix rows (D-063, D-066). See `matrix/domain-coverage.md` `## Languages Coverage` sub-table.
|
||||
@@ -197,6 +197,37 @@ If the task touches a domain, run that domain's checklist:
|
||||
- [ ] Audit logs do not leak secrets; redaction is structural, not opportunistic (P9)
|
||||
- [ ] System reports its own compliance state (drift from policy, open violations, retention status) (P10)
|
||||
|
||||
### If Edge (see `domains/edge/`)
|
||||
- [ ] Compute is placed near the user/data source; latency is treated as a correctness constraint, not a perf preference (P1)
|
||||
- [ ] The system continues to operate offline; partition is the norm, not the exception; reconciliation happens on reconnect (P2)
|
||||
- [ ] Edge-node resource constraints (CPU/memory/power/bandwidth) are declared per node class, not assumed infinite (P3)
|
||||
- [ ] Sync conflicts converge; no oscillation or infinite sync loops (P4)
|
||||
- [ ] Sync, cache-fill, and device commands are idempotent — retries are safe (P5)
|
||||
- [ ] Edge caches have an explicit TTL or invalidation strategy; no TTL-less caches under partition (P6)
|
||||
- [ ] Partial degradation is a designed state with a defined contract, not a crash (P7)
|
||||
- [ ] Routing, fan-out, and data placement are location-aware decisions, not accidents of deployment (P8)
|
||||
- [ ] Edge-device credentials are scoped and per-device; no edge node is a cluster-admin-equivalent (P9)
|
||||
- [ ] Telemetry is local-first: buffered on-node and forwarded on reconnect; partition does not blind the operator (P10)
|
||||
|
||||
### If Messaging (see `domains/messaging/`)
|
||||
- [ ] Messages have an explicit, versioned schema; producer and consumer agree on shape before exchange (P1)
|
||||
- [ ] Ordering guarantees (per-partition, global, none) are explicit and documented; "FIFO" is backed by the broker contract (P2)
|
||||
- [ ] Consumers are idempotent — redelivery is deduped via idempotency keys or deterministic processing (P3)
|
||||
- [ ] Delivery semantics (at-least-once/at-most-once/exactly-once) are a declared choice per channel (P4)
|
||||
- [ ] Poison messages route to a dead-letter queue; the DLQ is observable and drainable (P5)
|
||||
- [ ] Backpressure is bounded — consumer lag visible, max-unacked bounded, retry budget capped (P6)
|
||||
- [ ] Partition key choice is a documented design decision (ordering vs parallelism vs hotspots) (P7)
|
||||
- [ ] Retention windows and replay-from-offset are explicit; the broker is a durable log, not a pipe (P8)
|
||||
- [ ] Schema changes are backward/forward-compatible; breaking changes are versioned migrations, not silent shape edits (P9)
|
||||
- [ ] Consumer lag, DLQ depth, throughput, and consumer-group health are observed; silent backlog is a bug (P10)
|
||||
|
||||
### If Language-Derived Docs (see `languages/`)
|
||||
- [ ] Scope check: no new P-rules introduced — every section traces to an existing domain P-rule (D-063, D-066)
|
||||
- [ ] Every section header names ≥1 traced domain P-rule AND the core C-rule(s) inline (e.g., `## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)`)
|
||||
- [ ] Fenced code examples are in-language and illustrative only — no standalone `.ts`/`.py`/`.go`/`.rs` runtime artifacts (D-020)
|
||||
- [ ] The first-principles stub retains its section structure — no P1–P10 numbering added to languages (D-063)
|
||||
- [ ] Cross-links to traced domain docs are present (≥1 outbound per derived doc, ATELIER-114)
|
||||
|
||||
## Final Gate
|
||||
|
||||
- [ ] Have I read the relevant domain's first-principles?
|
||||
|
||||
@@ -188,6 +188,72 @@ The following standalone file types are forbidden under `examples/` and elsewher
|
||||
| Signed manifests as standalone files (`.sig`, `.att`, `.intoto.jsonl`) | fenced code in `examples/*.md` | Runtime attestation artifact; violates docs-only contract |
|
||||
| Standalone `.yaml` / `.tf` / `.sh` | fenced code in `examples/*.md` | (Carried forward from v0.2) Runtime deployable artifact |
|
||||
|
||||
## v0.4 Chaos Anti-Patterns (from IDEATE-36, IDEATE-37, D-068, ATELIER-110)
|
||||
|
||||
These are named, cross-cutting violations specific to the v0.4 domains. Single-breach per D-068. Reject on sight.
|
||||
|
||||
| Anti-Pattern | Breaches | Why |
|
||||
|--------------|----------|-----|
|
||||
| Edge sync conflict loop (two nodes oscillate divergent state forever; no convergence) | edge P4 Sync Conflicts are Bounded, Not Infinite; C1, C5 | Infinite oscillation is a correctness failure, not eventual consistency; the sync is a livelock |
|
||||
| Edge TTL-less cache (edge cache with no TTL and no explicit invalidation; stale forever under partition) | edge P6 Cache Invalidation is Explicit; C1, C3 | Stale-forever under partition is a silent correctness defect |
|
||||
| Edge blocking call on constrained device (a synchronous blocking I/O call on a low-power IoT device with no timeout) | edge P3 Resources are Constrained, P5 (retry safety); C8, C1 | Blocks the constrained node; no timeout = hang; retry is unsafe without idempotency |
|
||||
| Messaging shared subscription (two consumers share one subscription; each message processed by an arbitrary consumer, breaking per-consumer ordering and idempotency) | messaging P2 Ordering, P3 Idempotency; C1 | Shared subscription breaks ordering and dedup; the `messaging-shared-subscription` bad example lives here |
|
||||
| Messaging blocking consumer (a consumer blocks on a slow downstream call with no timeout; the broker re-delivers to the same stuck consumer) | messaging P6 Backpressure is Bounded; C1, C8 | Blocks the partition; lag grows; no backpressure escape |
|
||||
| Messaging unbounded retry / no DLQ (a poison message is retried forever with no dead-letter queue; the consumer never makes progress) | messaging P5 Dead-Letter Handling is Defined; C1, C5 | Infinite retry is a livelock; poison messages must DLQ |
|
||||
|
||||
### v0.4 Deployable Artifact Types (IDEATE-36, D-020)
|
||||
|
||||
The following standalone file types are forbidden under `languages/` and elsewhere in the framework (carried forward from v0.2/v0.3 and extended for language-derived docs). All examples and derived-doc code are `.md` files with fenced code only.
|
||||
|
||||
| Forbidden standalone artifact | Belongs in | Why |
|
||||
|-------------------------------|-----------|-----|
|
||||
| Standalone `.ts` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime TypeScript artifact; violates docs-only contract (v0.4 extension) |
|
||||
| Standalone `.py` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime Python artifact; violates docs-only contract (v0.4 extension) |
|
||||
| Standalone `.go` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime Go artifact; violates docs-only contract (v0.4 extension) |
|
||||
| Standalone `.rs` file | fenced code in `languages/*.md` (or `examples/*.md`) | Runtime Rust artifact; violates docs-only contract (v0.4 extension) |
|
||||
| Standalone `.yaml` / `.tf` / `.sh` | fenced code in `examples/*.md` | (Carried forward from v0.2/v0.3) Runtime deployable artifact |
|
||||
|
||||
### v0.4 Language-Derived-Doc Anti-Patterns
|
||||
|
||||
| Anti-Pattern | Breaches | Why |
|
||||
|--------------|----------|-----|
|
||||
| Language-derived doc introducing a new P-rule (e.g., "TS P1", "Python P2") | D-063, ARCHITECTURE.md hierarchy, matrix row-count invariant | Languages apply, not derive; breaks the hierarchy (`core > domains > languages`) and the 190-row invariant |
|
||||
| Language-derived doc with no domain P-rule trace (orphan section — header names no domain P-rule) | D-066, matrix completeness | No trace = no traceability; the doc is not connected to the framework |
|
||||
| Language-derived doc with runtime artifact (standalone `.ts`/`.py`/`.go`/`.rs` file) | D-020, PROJECT.md no-runtime-code | Violates the docs-only contract; must be fenced code in `.md` |
|
||||
| Language first-principles stub gaining P1–P10 numbering | D-063 | Languages are not domains; no P-rules to number; breaks the apply-don't-derive scope |
|
||||
|
||||
## v0.4 Domain-Specific Anti-Patterns
|
||||
|
||||
### Edge
|
||||
|
||||
| Anti-Pattern | Breaches | Why |
|
||||
|--------------|----------|-----|
|
||||
| Central-region-only deployment for a latency-bound workload | P1 Proximity is the Design Driver | Violates the proximity constraint; latency is a correctness defect |
|
||||
| App that crashes on disconnect (no offline state) | P2 Offline is a First-Class State | Partition is the norm; crash-on-disconnect is unengineered |
|
||||
| Undeclared edge-node resource budget (assumes infinite CPU/memory) | P3 Resources are Constrained and Declared | Unbounded = OOM on constrained nodes |
|
||||
| Sync loop that oscillates forever (CRDT without merge-semantics, LWW without monotonic clock) | P4 Sync Conflicts are Bounded | Livelock; convergence is never reached |
|
||||
| Non-idempotent edge write (cache-fill or device command retried with side effects) | P5 Edge Operations are Idempotent | Retry doubles the side effect |
|
||||
| TTL-less edge cache under partition | P6 Cache Invalidation is Explicit | Stale-forever; silent staleness |
|
||||
| Crash-on-node-failure (no partial-degradation contract) | P7 Partial Degradation is Engineered | Whole-system crash on one node's failure |
|
||||
| Random geographic placement (no location-aware routing) | P8 Geographic Distribution is a First-Class Constraint | Latency and data-residency uncontrolled |
|
||||
| Shared edge-device credential (one key for the whole fleet) | P9 Identity is Constrained at the Edge | One compromise = fleet compromise |
|
||||
| Fire-and-forget telemetry (no on-node buffer; data lost on partition) | P10 Edge Observability Survives Partition | Partition blinds the operator |
|
||||
|
||||
### Messaging
|
||||
|
||||
| Anti-Pattern | Breaches | Why |
|
||||
|--------------|----------|-----|
|
||||
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts | No boundary; consumer breaks silently on shape change |
|
||||
| "It's FIFO" with no documented partition contract | P2 Ordering is a Property, Not an Assumption | Undocumented ordering = assumed ordering = wrong ordering |
|
||||
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent | Redelivery doubles the effect |
|
||||
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit | The tradeoff is made blindly |
|
||||
| No dead-letter queue (poison message retried forever or dropped) | P5 Dead-Letter Handling is Defined | Livelock or silent loss |
|
||||
| Unbounded retry budget (no cap; slow consumer stalls the partition) | P6 Backpressure is Bounded | Lag grows without bound |
|
||||
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional | Hotspot or ordering violation |
|
||||
| Ephemeral broker (no retention; no replay) | P8 Replay and Retention are Configured | Not a durable log; replay impossible |
|
||||
| Silent schema change (producer breaks consumers with no version bump) | P9 Schemas Evolve Compatibly | Breaking change without migration |
|
||||
| Silent backlog (no lag metric; consumer falls behind invisibly) | P10 Messaging is Observable | Cannot fix what you cannot see |
|
||||
|
||||
## v0.3 Domain-Specific Anti-Patterns
|
||||
|
||||
### GitOps + Operators
|
||||
|
||||
@@ -129,6 +129,39 @@ Run the relevant domain section from `agent-checklist.md` (UI/UX, API, Security,
|
||||
- [ ] Do audit logs avoid leaking secrets (redaction is structural, not opportunistic)?
|
||||
- [ ] Does the system report its own compliance state (drift from policy, open violations, retention status)?
|
||||
|
||||
### If Edge
|
||||
- [ ] Scope check: is this the proximity/location/constraint angle, NOT generic perf or in-process concurrency? (D-061 — reject generic-performance or in-process-concurrency content as out-of-scope)
|
||||
- [ ] Is compute placed near the user/data source, with latency treated as a correctness constraint (not a perf preference)? (P1)
|
||||
- [ ] Does the system continue to operate offline; is partition the norm, not the exception? (P2)
|
||||
- [ ] Are edge-node resource constraints (CPU/memory/power/bandwidth) declared per node class, not assumed infinite? (P3)
|
||||
- [ ] Do sync conflicts converge; is there no oscillation or infinite sync loop? (P4)
|
||||
- [ ] Are sync, cache-fill, and device commands idempotent (retries safe)? (P5)
|
||||
- [ ] Do edge caches have an explicit TTL or invalidation strategy (no TTL-less caches under partition)? (P6)
|
||||
- [ ] Is partial degradation a designed state with a defined contract (not a crash)? (P7)
|
||||
- [ ] Are routing, fan-out, and data placement location-aware decisions? (P8)
|
||||
- [ ] Are edge-device credentials scoped and per-device (no edge cluster-admin)? (P9)
|
||||
- [ ] Is telemetry local-first (buffered on-node, forwarded on reconnect; partition does not blind the operator)? (P10)
|
||||
|
||||
### If Messaging
|
||||
- [ ] Scope check: is this cross-process/network-failure-model, NOT in-process concurrency? (D-062 — reject in-process-concurrency content as out-of-scope)
|
||||
- [ ] Do messages have an explicit, versioned schema (producer and consumer agree on shape)? (P1)
|
||||
- [ ] Are ordering guarantees (per-partition, global, none) explicit and documented? (P2)
|
||||
- [ ] Are consumers idempotent (redelivery deduped via idempotency keys or deterministic processing)? (P3)
|
||||
- [ ] Are delivery semantics (at-least-once/at-most-once/exactly-once) a declared choice per channel? (P4)
|
||||
- [ ] Do poison messages route to a dead-letter queue (DLQ observable and drainable)? (P5)
|
||||
- [ ] Is backpressure bounded (consumer lag visible, max-unacked bounded, retry budget capped)? (P6)
|
||||
- [ ] Is partition key choice a documented design decision (ordering vs parallelism vs hotspots)? (P7)
|
||||
- [ ] Are retention windows and replay-from-offset explicit (durable log, not pipe)? (P8)
|
||||
- [ ] Are schema changes backward/forward-compatible (breaking changes are versioned migrations)? (P9)
|
||||
- [ ] Are consumer lag, DLQ depth, throughput, and consumer-group health observed (silent backlog is a bug)? (P10)
|
||||
|
||||
### If Language-Derived Docs
|
||||
- [ ] Scope check: apply, don't derive — no new P-rules, no new matrix rows (D-063, D-066 — reject any "TS P1"/"Python P2" etc. new-P-rule content)
|
||||
- [ ] Does every section header name ≥1 traced domain P-rule AND the core C-rule(s) inline?
|
||||
- [ ] Are fenced code examples in-language and illustrative only (no standalone `.ts`/`.py`/`.go`/`.rs` runtime artifacts)? (D-020)
|
||||
- [ ] Does the first-principles stub retain its section structure (no P1–P10 numbering added to languages)? (D-063)
|
||||
- [ ] Are cross-links to traced domain docs present (≥1 outbound per derived doc)? (ATELIER-114)
|
||||
|
||||
## Review Etiquette
|
||||
|
||||
- **Comment, don't command.** "This could be X" not "Change this to X."
|
||||
|
||||
Reference in New Issue
Block a user