6 Commits

Author SHA1 Message Date
Jon Chery 4e433158cd docs(P03): complete language-derived extension — v0.4
---ci---
project: atelier
phase: 3
milestone: v0.4
status: complete
phase_role: execution
phase_tag: v0.3.3
requirements:
  covered: [ATELIER-102, ATELIER-103, ATELIER-104, ATELIER-105]
  partial: []
---/ci---
2026-08-05 16:07:21 +00:00
Jon Chery 0fdf892d81 docs(ship): P2 complete — v0.3.2 tagged, release 483
---ci---
project: atelier
phase: 2
milestone: v0.4
status: complete
phase_role: execution
phase_tag: v0.3.2
release_id: 483
---/ci---
2026-08-05 16:01:08 +00:00
Jon Chery f61dffbb5a docs(P02): complete messaging domain phase — v0.4
---ci---
project: atelier
phase: 2
milestone: v0.4
status: complete
phase_role: execution
phase_tag: v0.3.2
requirements:
  covered: [ATELIER-97, ATELIER-98, ATELIER-99, ATELIER-100, ATELIER-101]
  partial: []
---/ci---
2026-08-05 16:00:31 +00:00
Jon Chery f7f007dce8 docs(ship): P1 complete — v0.3.1 tagged, release 479
---ci---
project: atelier
phase: 1
milestone: v0.4
status: complete
phase_role: execution
phase_tag: v0.3.1
release_id: 479
---/ci---
2026-08-05 15:53:16 +00:00
Jon Chery 20992883ff docs(P01): complete edge domain phase — v0.4
---ci---
project: atelier
phase: 1
milestone: v0.4
status: complete
phase_role: execution
phase_tag: v0.3.1
requirements:
  covered: [ATELIER-92, ATELIER-93, ATELIER-94, ATELIER-95, ATELIER-96]
  partial: []
---/ci---
2026-08-05 15:51:03 +00:00
Jon Chery a622419e7d docs(ship): P0 complete — v0.3.0 tagged, release 475
---ci---
project: atelier
phase: 0
milestone: v0.4
status: complete
phase_role: pre_execution
phase_tag: v0.3.0
release_id: 475
---/ci---
2026-08-05 14:03:35 +00:00
32 changed files with 4976 additions and 9 deletions
+8 -6
View File
@@ -1,13 +1,15 @@
{
"phase": 0,
"stage": "plan",
"phase": 2,
"stage": "complete",
"milestone": "v0.4",
"phase_role": "pre_execution",
"phase_role": "execution",
"project": "atelier",
"attempts": 0,
"updated_at": "2026-08-05T05:20:00Z",
"updated_at": "2026-08-05T06:30:00Z",
"milestone_complete": false,
"milestone_branch": "milestone/v0.4-edge-quantum-langs",
"phase_branch": "phase/00-pre-execution",
"tag_base": "v0.3"
"phase_branch": "phase/02-messaging",
"tag_base": "v0.3",
"phase_tag": "v0.3.2",
"release_id": 483
}
+3 -3
View File
@@ -130,9 +130,9 @@ NFR milestone: no separate minor tag. The final patch (v0.2.6) IS the v0.3 deliv
| Phase | Name | Type | Status | Key Deliverables |
|-------|------|------|--------|------------------|
| 0 | Pre-Execution | docs | active | Spec, clarify, research, ideate, plan, PERSONAS.md (adds edge-engineer + languages-engineer phase-specific personas) |
| 1 | Edge Domain | docs | pending | domains/edge/{first-principles, cdn, offline-first, iot, sync}.md |
| 2 | Messaging Domain | docs | pending | domains/messaging/{first-principles, queues, pubsub, streams, delivery-semantics}.md |
| 0 | Pre-Execution | docs | complete | Spec, clarify, research, ideate, plan, PERSONAS.md (adds edge-engineer + languages-engineer phase-specific personas) — shipped v0.3.0 |
| 1 | Edge Domain | docs | complete | domains/edge/{first-principles, cdn, offline-first, iot, sync}.md — shipped v0.3.1 |
| 2 | Messaging Domain | docs | complete | domains/messaging/{first-principles, queues, pubsub, streams, delivery-semantics}.md — shipped v0.3.2 |
| 3 | Language-Derived Extension | docs | pending | languages/ × 4 → first-principles + 4 derived docs each (16 derived docs) |
| 4 | Matrix + Review Integration | docs | pending | matrix/principles-matrix.md (+20 mappings, 170→190), matrix/domain-coverage.md (+ languages/ sub-table), review/{agent-checklist, peer-review-checklist, anti-patterns}.md, MANIFEST.md (languages/ section) |
| 5 | Examples + Cross-Links | docs | pending | examples/good + examples/bad for 2 domains, cross-links to devops/observability/data/concurrency/kubernetes/infrastructure-as-code + language→domain links |
+227
View File
@@ -0,0 +1,227 @@
# CDN — Derived Rules
> Derives from `domains/edge/first-principles.md`. Applies P1
> (Proximity is the Design Driver) and P6 (Cache Invalidation is
> Explicit) primarily, with P5 (idempotent cache fill), P8
> (geographic distribution), and P9 (identity at the edge). For the
> edge-cache-vs-origin decision, see the decision matrix below.
> Cross-links `domains/performance/frontend` for generic caching,
> `domains/security/input-validation` for cache poisoning, and
> `domains/observability/metrics` for cache-hit ratio.
## What a CDN Is (P1 Proximity is the Design Driver)
- A content delivery network is a fleet of PoPs (points of presence)
placed near users. The PoP serves cached content; the origin is
the authoritative source. The CDN's whole purpose is P1: compute
(the cache) is placed near the user so the round trip to the origin
does not bound latency. Latency is a correctness constraint at the
edge (C1), not a performance preference.
- The CDN is the canonical edge-cache architecture (Akamai,
Cloudflare, Fastly): PoPs near users, origin shielding, cache-key
normalization, purge APIs. Atelier derives the
placement/invalidation principles, not the vendor config.
- The boundary with `domains/performance/frontend` is per D-061:
performance owns *generic* caching and optimization (cache what is
expensive, stable, read often — `performance/P5 Caching with
Intent`); edge owns the *geographic, partition-aware* placement and
invalidation angle. A CDN is an edge concern because its defining
trait is geographic distribution (P8) and partition-aware
invalidation (P6), not measurement.
## Cache Key Design (P6 Cache Invalidation is Explicit)
- The cache key is the contract between the URL and the cached
representation. A key that varies on the wrong dimensions serves
the wrong content; a key that varies on too many dimensions
collapses the hit ratio. Key design *is* the invalidation
surface: a key that includes a content hash or version segment
makes invalidation explicit; a key that ignores `Vary` headers
serves stale variants.
- Normalize the key: lower-case the host, strip default ports,
sort query parameters, ignore tracking parameters. A
non-normalized key is a cache-poisoning vector (see
`domains/security/input-validation`) and a hit-ratio destroyer
(see `domains/observability/metrics`).
- A cache with no explicit key strategy is a TTL-less cache under
partition (P6 violation): staleness is silent and unbounded.
```http
# Cache key derivation: vary on what changes content, ignore what
# does not. The key is the tuple (host, normalized-path, sorted-
# query, Vary-headers); the cache entry is the representation + TTL.
Cache-Key: example.com /api/v1/products?sort=price&region=us Vary:Accept-Encoding
Cache-Control: public, max-age=60, s-maxage=600, stale-while-revalidate=300
Vary: Accept-Encoding
```
- `max-age` bounds the browser cache; `s-maxage` bounds the CDN
PoP; `stale-while-revalidate` allows serving stale while
refetching. Each is an explicit invalidation strategy (P6).
## TTL vs Explicit Invalidation (P6, C3 Simplicity)
- **TTL-based invalidation** (`max-age`, `s-maxage`): the cache entry
expires after a duration. Simple, no origin contact required to
invalidate, but bounded staleness is the contract — the entry may
be stale up to TTL. Fits content where eventual consistency is
acceptable (asset fingerprints, lists, derived images).
- **Explicit invalidation** (purge, surrogate keys): the operator
signals the cache to drop entries. Tighter staleness bounds, but
requires the origin or operator to know which entries to purge.
Fits content where staleness is a correctness defect (price
updates, availability, breaking news).
- A TTL-less cache with no explicit invalidation is a P6 violation:
stale-forever under partition. Every cache must have one or the
other (or both), and the choice is documented per content type.
## Cache-Hit / Miss / Origin-Fetch (P5, P6)
- **Hit**: the PoP serves from cache. Latency is PoP-local (P1).
- **Miss**: the PoP has no entry; it fetches from the origin (or an
origin-shield PoP). The fetch must be idempotent (P5) — a retried
miss must not corrupt the cache or double-write side effects.
- **Revalidate**: the PoP holds a stale entry and asks the origin
(`If-None-Match`, `If-Modified-Since`); a 304 refreshes the TTL
without re-fetching the body. Revalidation is the bandwidth-economical
middle ground (C8).
```http
# Conditional revalidation the PoP asks the origin "is this still
# current?" The 304 response refreshes the TTL without a body.
GET /api/v1/products HTTP/1.1
Host: example.com
If-None-Match: "etag-7a3f"
HTTP/1.1 304 Not Modified
ETag: "etag-7a3f"
Cache-Control: s-maxage=600
```
- A cache-hit ratio that is not measured is a gate on noise — see
`domains/observability/metrics` for the SLI/SLO discipline that
makes the hit ratio a meaningful signal. A CDN with no hit-ratio
metric is operating blind (P10 analog).
## Origin Shielding (P1, P8, C8 Economy)
- Origin shielding routes all origin fetches through a single
shield PoP (or shield region). The shield absorbs the
thundering-herd: 10 000 PoPs missing the same URL fetch the origin
once, not 10 000 times. This is C8 Economy (origin bandwidth is
bounded) and P1 (the shield is itself a proximity layer for the
origin).
- Shielding is a geographic decision (P8): the shield sits in a
region close to the origin, not close to the user. The shield is
the inner ring of the CDN; the user-facing PoPs are the outer ring.
- A CDN without origin shielding under a stampede will overload the
origin; a shield that is itself partitioned from the origin must
degrade gracefully (P7) — serve stale per `stale-while-revalidate`
rather than 500.
## Purge Strategies (P6, C3 Simplicity)
| Strategy | Granularity | Latency to Invalidate | Cost | Best for |
|----------|-------------|-----------------------|------|----------|
| URL purge | One URL | Seconds | Low (one entry) | Surgical fixes, single-page corrections |
| Soft purge | One URL (mark stale, serve while refetch) | Seconds | Low | High-traffic URLs where a hard purge causes a stampede |
| Surrogate-key purge | A tag set (e.g., `product:123`, `category:shoes`) | Seconds | Medium (key indexing) | Related-content invalidation (a product update purges all its category pages) |
| Wildcard purge | A path prefix or pattern | Seconds to minutes | High (scan) | Site-wide template changes |
| All-cache purge | Everything | Seconds | Very high (origin stampede) | Disaster recovery only; never the steady-state invalidation path |
- Surrogate-key purge (Fastly, Akamai) is the highest-value
strategy: tag cache entries with content keys, then purge by tag.
This is explicit invalidation at scale (P6) without the origin
stampede of an all-cache purge.
- An all-cache purge as the steady-state invalidation path is a P6
violation dressed as a feature — it pushes the origin load back to
100% miss, defeating the CDN's purpose (P1).
## Cache Poisoning Prevention (P9, cross-link security/input-validation)
- A cache poisoned by a crafted request (a URL with a malicious
header that gets cached and served to others) is a correctness
defect (C1) and a security breach (P9 — the edge node is
exploited). Prevent poisoning by:
- Normalizing the cache key (strip untrusted query parameters,
ignore unknown headers, lower-case the host).
- Validating `Vary` against an allow-list; never `Vary: *` on a
shared cache (poisonable via header injection).
- Treating uncacheable responses (`Set-Cookie`,
`Cache-Control: private`) as never-stored.
- See `domains/security/input-validation` for the general
input-validation discipline the cache key must follow. The cache
key is a validation surface; a non-validated key is an attack
surface.
## Multi-CDN Routing (P8 Geographic Distribution)
- A multi-CDN strategy routes each request to the best PoP across
providers (Akamai + Cloudflare + Fastly). Routing is
location-aware (P8): latency, cost, and availability vary by
region and provider. The DNS layer (or a client-side router)
selects the CDN per request.
- Multi-CDN is a P8 decision, not a vendor-management decision:
geographic distribution is the first-class constraint. A
single-CDN deployment routes everything to one provider's PoPs;
a multi-CDN deployment routes by region, latency, and cost.
- Invalidation across multiple CDNs is harder (P6): each provider
has its own purge API and surrogate-key scheme. A multi-CDN purge
must fan out to all providers; a purge that reaches only one CDN
leaves the others stale. Track purge completion per provider —
see `domains/observability/metrics` for the per-CDN hit-ratio and
purge-latency signals.
```http
# A CDN config example: cache-control headers + a purge rule.
# Origin response: declare the cache contract (P6).
HTTP/1.1 200 OK
Cache-Control: public, max-age=60, s-maxage=600, stale-while-revalidate=300
Surrogate-Key: product:123 category:shoes
ETag: "etag-7a3f"
Vary: Accept-Encoding
# Purge rule (Fastly-style surrogate-key): when product 123
# updates, purge every cache entry tagged product:123 OR
# category:shoes. Explicit, bounded, no origin stampede (P6).
POST /service/svc1/purge
Surrogate-Key: product:123 category:shoes
# Returns: {"status": "ok", "id": "purge-abc"} — poll the purge
# status to confirm completion across all PoPs (P8, P10).
```
## Edge-Cache vs Origin — Decision Matrix (D-069)
| Strategy | When | Latency | Origin Load | Correctness Risk |
|----------|------|---------|-------------|------------------|
| Serve from PoP (cache hit) | The PoP holds a fresh entry (within TTL or revalidated) | Lowest (PoP-local, P1) | None | Low — bounded by TTL staleness (P6) |
| Serve stale while revalidate | The PoP holds a stale entry and `stale-while-revalidate` is set | Low (stale served immediately, refetch in background) | Background refetch (1 per entry) | Medium — stale served up to the revalidate window; acceptable for eventually-consistent content |
| Fetch fresh from origin (miss) | The PoP has no entry, or the content is non-cacheable | High (origin round trip) | Full fetch per miss | Low — fresh by construction; the miss is the correctness floor |
| Origin-shield fetch | Multiple PoPs miss the same URL; the shield collapses the herd | Medium (PoP → shield → origin) | Bounded to one origin fetch per shield (C8) | Low — shield is the inner ring; staleness bounded by shield TTL |
| Purge and serve fresh | Explicit invalidation received (surrogate-key or URL purge) | Medium (purge propagates, then fresh fetch) | Full fetch post-purge | Lowest — explicit invalidation is the tightest staleness bound (P6) |
| Serve from origin directly (bypass cache) | Content is non-cacheable (personalized, real-time) | Highest (every request hits origin) | Full fetch per request | Lowest for correctness, highest for origin load — use sparingly |
- The default is **serve from PoP** when fresh, **fetch fresh from
origin** on miss with **origin-shield** to bound origin load, and
**purge and serve fresh** when explicit invalidation is required.
Bypass-the-cache is for non-cacheable content only — bypassing for
cacheable content is a P1 violation (you have defeated the CDN).
- The correctness risk column is bounded by the invalidation
strategy (P6): every row except "bypass" carries staleness risk
that is bounded by TTL or explicit purge. A row with no
invalidation strategy is a P6 violation.
## What Violates CDN Discipline
| Violation | Principle |
|-----------|-----------|
| TTL-less edge cache under partition (stale-forever, no explicit invalidation) | P6 Cache Invalidation is Explicit |
| Cache key that varies on untrusted query parameters (poisonable) | P6, P9 (`domains/security/input-validation`) |
| All-cache purge as the steady-state invalidation path (origin stampede) | P6, C8 Economy |
| Non-normalized cache key (case-sensitive host, unsorted query) | P6, `domains/security/input-validation` |
| Bypass-the-cache for cacheable content | P1 Proximity is the Design Driver (defeats the CDN) |
| Multi-CDN with no per-CDN purge completion tracking | P8, P10 (stale cache invisible to the operator) |
| Origin fetch that is not idempotent under retry | P5 Edge Operations are Idempotent |
| Cache-hit ratio not measured | P10, `domains/observability/metrics` |
| Single-CDN deployed where geographic distribution requires multi-CDN | P8 Geographic Distribution |
| Shield PoP that 500s instead of serving stale under partition | P7 Partial Degradation is Engineered |
+218
View File
@@ -0,0 +1,218 @@
# Edge — First Principles
## 1. The Principles
### P1. Proximity is the Design Driver
Compute, storage, and data are placed near the user or the data
source. At the edge, latency is a correctness constraint (C1), not a
performance preference — a late answer is a wrong answer when the
round trip to a central region exceeds the user's or device's
tolerance. This is the geographic expression of `C4 Locality`:
performance's locality is algorithmic (data near compute); edge's
locality is geographic (compute near user/data source). Placement is
a design decision, not an accident of deployment, and it is
constrained by `P8 Geographic Distribution`. The proximity angle is
the distinguishing trait of the edge domain per D-061: this is what
separates edge from `domains/performance/` (which owns *generic*
measurement and optimization, not placement).
### P2. Offline is a First-Class State
The system continues to operate when disconnected from the center.
Partition is the norm, not the exception; reconciliation happens on
reconnect, never assumed to be instant. An app that crashes on
disconnect has no offline state and is unengineered. Offline
operation derives from `C5 Reversibility` — the disconnected state
is reversible back to consistency via reconciliation — and `C1
Correctness`, because correctness under partition is the contract,
not eventual correctness as a hedge. This is the foundation for
`domains/edge/offline-first.md` and the precondition for the
bounded-conflict discipline of `P4`.
### P3. Resources are Constrained and Declared
Edge nodes — IoT sensors, gateways, point-of-sale devices, CDN PoP
caches, 5G MEC nodes — have bounded CPU, memory, power, and
bandwidth. Constraints are declared per node class, never assumed
infinite. An undeclared budget is a defect: unbounded growth is a
bug, and a constrained device with no budget will OOM or exhaust
power. This derives from `C8 Economy` (use no more than the task
requires) and `C1 Correctness` (a node that exceeds its bounds has
failed). This is the edge-specific angle on `domains/performance/P4
Resource Bounds` — performance owns the generic principle; edge owns
the constrained-device reality. See `domains/edge/iot.md` for the
per-device-class application.
### P4. Sync Conflicts are Bounded, Not Infinite
Divergent state across partitioned nodes converges. Oscillation and
infinite sync loops are correctness failures, not eventual
consistency. A merge that never terminates is a livelock; a CRDT
without merge semantics or an LWW without a monotonic clock can
oscillate forever. This derives from `C1 Correctness` (convergence is
a correctness contract) and `C5 Reversibility` (divergent state is
reversible back to convergence). The bound may be eventual (CRDTs) or
arbitrated (LWW with vector clocks), but it must exist. This is the
foundation for `domains/edge/sync.md` and the rule the
`edge-sync-loop` chaos anti-pattern breaches.
### P5. Edge Operations are Idempotent
Sync, cache fill, and device commands are retried by nature — the
network is partition-prone and the operation will be re-attempted.
Idempotency keys (or deterministic operations) make retries safe. A
non-idempotent edge write retried with side effects doubles the
effect; a non-idempotent cache fill under retry corrupts the cache.
This derives from `C1 Correctness`: correctness under retry is the
contract, not a nice-to-have. This parallels `domains/messaging/P3
Consumers are Idempotent` (cross-process delivery) and is the edge's
device-and-cache-flavored analog — see `domains/edge/iot.md` for
device command idempotency and `domains/edge/cdn.md` for cache-fill
idempotency.
### P6. Cache Invalidation is Explicit
Edge caches carry a defined invalidation or TTL strategy. A
stale-forever cache under partition is a silent correctness defect;
a TTL-less cache with no explicit invalidation is a bug, not a
feature. This derives from `C1 Correctness` (cached state must be
correct) and `C3 Simplicity` (a defined invalidation strategy is
simpler and clearer than ad-hoc staleness). This is distinct from
`domains/performance/P5 Caching with Intent`, which owns *generic*
caching and optimization; edge owns the *geographic,
partition-aware* invalidation angle — when a PoP is partitioned from
the origin, the invalidation strategy is the correctness mechanism.
See `domains/edge/cdn.md` for purge strategies (URL vs soft vs
surrogate-key) and the edge-cache-vs-origin decision matrix.
### P7. Partial Degradation is Engineered
The system degrades gracefully when an edge node or link fails. A
partial service is a designed state with a defined contract, not a
crash. One node's failure must not collapse the whole fleet; the
degraded mode is documented, observable, and recoverable. This
derives from `C1 Correctness` (the degraded contract is a
correctness bound) and `C5 Reversibility` (recovery from degradation
is reversible by construction). A crash-on-node-failure system has
no degradation contract — it has an all-or-nothing failure mode that
violates the fleet assumption. See `domains/edge/iot.md` for
device-drop degradation and `domains/edge/offline-first.md` for
partition degradation.
### P8. Geographic Distribution is a First-Class Constraint
The fleet is geo-distributed; routing, fan-out, and data placement
are location-aware decisions, not accidents of deployment. The
system is many nodes across many locations, not a single deployment.
Data residency, regional latency, and PoP selection are engineered,
not discovered in production. This derives from `C4 Locality` (the
placement of data and compute is a locality decision) and `C6
Composability` (the fleet composes from location-aware parts, each
with its own contract). This is the structural companion to `P1
Proximity`: P1 says *where* compute should be (near the user); P8
says the *distribution* of compute across geographies is a
first-class constraint. See `domains/edge/cdn.md` for multi-CDN
routing.
### P9. Identity is Constrained at the Edge
Edge devices and nodes hold scoped, minimal credentials. No edge
node is a cluster-admin-equivalent; device identity is per-device,
not shared. One compromise must not equal a fleet compromise. This
derives from `C1 Correctness` (security is a subset of correctness —
an exploitable edge node does not do what it was supposed to do) and
`C8 Economy` of trust (the credential scope is minimal for the task).
A shared edge-device credential is the edge analog of a
cluster-admin GitOps robot — blast radius is unbounded. See
`domains/security/secrets.md` for the general secret-hygiene
principles and `domains/edge/iot.md` for device provisioning.
### P10. Edge Observability Survives Partition
Telemetry is local-first: buffered on the node and forwarded on
reconnect. Partition does not blind the operator. A fire-and-forget
telemetry pipeline loses data when the link drops; a local-first
buffer survives. This derives from `C7 Observability` (the fleet's
behavior is visible to the operator) and `C5 Reversibility` (the
buffered telemetry is reversible back to visibility on reconnect).
This is distinct from `domains/observability/P1 Structured by
Default`, which owns *generic* structured telemetry; edge owns the
*partition-survivable, local-first* angle. See
`domains/observability/metrics.md` and
`domains/observability/logging.md` for the generic structured-
telemetry foundations edge builds on.
## 2. Core Principle Trace
Each edge P-rule derives from one or more core C-rules (C1C8). The
matrix extension lands in P4 of the v0.4 plan; the traces below are
authoritative. Edge is a broad-derivation domain touching 7 of 8
core principles (C1, C3, C4, C5, C6, C7, C8); C2 (Clarity) is not a
primary derivation — edge clarity is indirect (a cache with explicit
invalidation is clearer than one without, but the primary trace is
C1/C3).
| P-rule | Core | Why |
|--------|------|-----|
| P1 Proximity is the Design Driver | C4, C1 | Locality of compute near user/data; correctness via latency |
| P2 Offline is a First-Class State | C1, C5 | Correctness under partition; reversibility of reconciliation |
| P3 Resources are Constrained and Declared | C8, C1 | Economy of constrained nodes; correctness of declared bounds |
| P4 Sync Conflicts are Bounded, Not Infinite | C1, C5 | Correctness of convergence; reversibility of divergent state |
| P5 Edge Operations are Idempotent | C1 | Correctness under retry |
| P6 Cache Invalidation is Explicit | C1, C3 | Correctness of cached state; simplicity of defined invalidation |
| P7 Partial Degradation is Engineered | C1, C5 | Correctness of degraded modes; reversibility of recovery |
| P8 Geographic Distribution is a First-Class Constraint | C4, C6 | Locality of placement; composability of the fleet |
| P9 Identity is Constrained at the Edge | C1, C8 | Correctness via security; economy of trust |
| P10 Edge Observability Survives Partition | C7, C5 | Observability of the fleet; reversibility of buffered telemetry |
## 3. What Violates These Principles
| Violation | Principle Breached |
|-----------|-------------------|
| Central-region-only deployment for a latency-bound workload | P1 Proximity is the Design Driver |
| App that crashes on disconnect (no offline state) | P2 Offline is a First-Class State |
| Undeclared edge-node resource budget (assumes infinite CPU/memory) | P3 Resources are Constrained and Declared |
| Sync loop that oscillates forever (CRDT without merge-semantics, LWW without monotonic clock) | P4 Sync Conflicts are Bounded, Not Infinite |
| Non-idempotent edge write (cache-fill or device command retried with side effects) | P5 Edge Operations are Idempotent |
| TTL-less edge cache under partition (stale-forever, no explicit invalidation) | P6 Cache Invalidation is Explicit |
| Crash-on-node-failure (no partial-degradation contract) | P7 Partial Degradation is Engineered |
| Random geographic placement (no location-aware routing) | P8 Geographic Distribution is a First-Class Constraint |
| Shared edge-device credential (one key for the whole fleet) | P9 Identity is Constrained at the Edge |
| Fire-and-forget telemetry (no on-node buffer; data lost on partition) | P10 Edge Observability Survives Partition |
| Blocking call on a constrained IoT device with no timeout | P3, P5 (blocks the node; retry unsafe without idempotency) |
| Multi-CDN routing with no PoP-selection logic (latency uncontrolled) | P8, P1 (placement not a design decision) |
## 4. Relationship to Other Domains
Edge computing is the engineering discipline of placing compute,
storage, and data **near the source of generation or consumption**
rather than in a centralized cloud. The distinguishing constraints are
latency-bound operation, resource-constrained nodes,
geo-distribution as a fleet, and partition-prone operation. Edge
overlaps three existing domains by *subject* but not by *angle*: per
D-061, edge owns the proximity/location/constraint/disconnection
concerns that only arise at the network edge. The C4 Locality
emphasis is the discriminator: performance's locality is algorithmic
(data near compute); edge's locality is geographic (compute near
user/data source). Cross-links are one-directional outward (per
D-026 extended); no back-link edits to v0.1/v0.2/v0.3 content.
- `domains/performance/frontend` ← P6 (edge owns geographic,
partition-aware cache invalidation; performance owns *generic*
caching and measurement — D-061 boundary)
- `domains/performance/P4 Resource Bounds` ← P3 (edge owns
constrained-device reality; performance owns the generic
unbounded-growth-is-a-bug principle)
- `domains/observability/metrics` ← P10 (cache-hit ratio, edge
telemetry aggregation; edge owns the local-first angle, observability
owns generic structured metrics)
- `domains/observability/logging` ← P10 (local-first logging buffered
on-node and forwarded on reconnect)
- `domains/concurrency/patterns` ← P5 (the offline write-queue is
the cross-partition analog of the in-process bounded buffer —
concurrency owns in-process; edge owns partition-survivable)
- `domains/security/secrets` ← P9 (device credentials are scoped,
per-device, never shared — edge owns the constrained-identity
angle; security owns the general secret hygiene)
- `domains/security/input-validation` ← P6 (cache poisoning
prevention — edge cache keys are a validation surface)
- `domains/data/migrations` ← P4 (schema migration under sync must
reconcile across partitioned nodes; data owns the generic migration
discipline, edge owns the partitioned-reconcile angle)
> Note: cross-links to `domains/messaging/` (e.g., MQTT QoS parallels
> for delivery semantics) are intentionally omitted here — the
> messaging domain is authored in P2. The intra-v0.4 edge↔messaging
> links are added in P5 (ATELIER-114 per IDEATE-40) once both
> domains exist; the dangling link is acceptable per D-053.
+258
View File
@@ -0,0 +1,258 @@
# IoT — Derived Rules
> Derives from `domains/edge/first-principles.md`. Applies P3
> (Resources are Constrained and Declared) primarily, with P5
> (command idempotency), P7 (partial degradation when devices drop),
> P9 (device identity and provisioning), and P10 (telemetry from
> devices). Cross-links `domains/security/secrets` for device
> credentials and `domains/messaging/queues` for the MQTT QoS
> parallels to delivery semantics.
## What IoT at the Edge Is (P3 Resources are Constrained and Declared)
- IoT at the edge is the engineering discipline of operating
constrained devices — sensors, actuators, gateways, microcontrollers
— as first-class participants in a distributed system. The
distinguishing constraint is per-device resource bounds (P3): a
battery-powered sensor has kilobytes of RAM, a constrained
protocol, and a multi-year sleep budget. These constraints are
declared per device class, never assumed infinite.
- The boundary is per D-061: edge owns the constrained-device
reality; performance owns the generic unbounded-growth-is-a-bug
principle (`performance/P4 Resource Bounds`); concurrency owns
in-process primitives. IoT is an edge concern because its defining
traits are constrained resources (P3), geographic distribution as
a fleet (P8), partition-prone operation (P2), and device-scoped
identity (P9) — concerns that only arise at the network edge.
- See `domains/edge/offline-first.md` for the partition-survival
discipline that constrained devices depend on, and
`domains/edge/sync.md` for the reconciliation of device state
across partitions.
## Device Resource Classes (P3, C8 Economy)
- A device resource class declares the bounds for a class of
devices: CPU (MHz, cores), memory (KB/MB), power (battery mAh,
duty-cycle budget), bandwidth (bytes/sec, latency budget), and
storage (KB/MB). Every device in the fleet is assigned to a class;
every operation is budgeted against its class.
- An undeclared budget is a defect (P3 violation): a sensor that
sends telemetry every second without a duty-cycle budget exhausts
its battery in days, not years. The budget is the correctness
bound (C1) and the economy bound (C8).
- A device class implies a protocol choice: a class-0 device
(constrained sensor, KB RAM) speaks CoAP; a class-1 device
(gateway, MB RAM) speaks MQTT; a class-2 device (edge compute
node, GB RAM) speaks HTTP. The protocol follows the constraint,
not the reverse.
| Class | RAM | Power | Protocol | Typical role |
|-------|-----|-------|----------|--------------|
| 0 (constrained sensor) | < 10 KB | Battery, multi-year | CoAP, LoRaWAN | Telemetry only, no inbound commands |
| 1 (actuator, gateway) | 10 KB 1 MB | Battery or wired, weeks-months | MQTT, CoAP | Telemetry + commands, queue-and-forward |
| 2 (edge compute) | > 1 MB | Wired, continuous | HTTP, MQTT | Local aggregation, gateway, edge inference |
## Constrained Protocols — MQTT and CoAP (P3, P5)
- **MQTT** is the canonical pub/sub protocol for constrained devices.
It is lightweight (2-byte header), broker-backed, and provides QoS
levels (0, 1, 2) that map to delivery semantics. MQTT is the
cross-process analog of message-queue delivery — see
`domains/messaging/queues` for the general queue/delivery-semantics
discipline; the cross-link is one-directional outward (edge →
messaging) per D-062 and D-026 extended.
- **CoAP** is the REST analog for constrained devices: UDP-based,
low-overhead, with confirmable (CON) and non-confirmable (NON)
message types. CoAP fits class-0 devices where TCP is too heavy.
- A blocking synchronous call on a constrained device with no
timeout is the `blocking-call-on-constrained-device` chaos
anti-pattern: it blocks the node, has no timeout (= hang), and
retries are unsafe without idempotency (P3 + P5 breach). Every
device operation must be async with a timeout, and every retried
operation must be idempotent.
```json
// MQTT publish/subscribe payload with QoS levels (P5 idempotency,
// P3 constrained protocol).
// QoS 0 — at-most-once: fire-and-forget, no ack. For telemetry
// where a dropped sample is acceptable (P3 economy of the
// constrained link).
{
"topic": "devices/sensor-7/temperature",
"qos": 0,
"payload": {
"device": "sensor-7",
"ts": 1700000000,
"value": 21.4,
"unit": "C"
}
}
// QoS 1 — at-least-once: acked, may duplicate. The consumer must
// be idempotent (P5) — dedup by (device, ts) or an idempotency key.
{
"topic": "devices/actuator-3/command",
"qos": 1,
"payload": {
"device": "actuator-3",
"idempotencyKey": "cmd-1700000000-1",
"command": "set-point",
"value": 22.0
}
}
// QoS 2 — exactly-once: four-step handshake, no duplication. The
// heaviest QoS; use only where duplicates are intolerable AND the
// device has the budget for the handshake (class-1+ only, P3).
{
"topic": "devices/actuator-3/irreversible-command",
"qos": 2,
"payload": {
"device": "actuator-3",
"idempotencyKey": "cmd-1700000000-2",
"command": "calibrate"
}
}
```
- The QoS choice is a P5 (idempotency) and P3 (resource) decision:
QoS 0 is cheapest (no ack) but lossy; QoS 1 requires consumer
idempotency (P5); QoS 2 is exactly-once but costs a four-step
handshake on a constrained link. The default for telemetry is QoS
0 or 1; the default for commands is QoS 1 with an idempotency key
(P5); QoS 2 is reserved for irreversible commands where the
device budget permits.
## Device Identity and Provisioning (P9 Identity is Constrained at the Edge)
- Every device holds a per-device identity: a unique device ID and a
scoped credential (X.509 certificate, API token, or rotating
key). No shared fleet credential — one compromise must not equal
a fleet compromise (P9). The credential scope is minimal: a
device can publish to `devices/<its-id>/+` and subscribe to
`devices/<its-id>/commands`, nothing else.
- Provisioning is the act of assigning a device identity at
enrollment time. The provisioning manifest declares the device,
its class, its allowed topics, and its credential. The manifest is
the P9 contract — a device operating outside its manifest scope is
a violation.
- A device that is provisioned with a shared fleet key (the
`shared-edge-device-credential` anti-pattern) is a P9 violation:
blast radius is unbounded. See `domains/security/secrets` for the
general secret-hygiene discipline (per-identity credentials,
rotation, minimal scope) that device provisioning builds on.
```yaml
# Device provisioning manifest (P9 per-device identity + scoped
# credentials). The manifest is the contract; the device operates
# only within its declared scope.
device:
id: sensor-7
class: 0 # P3 resource class
model: temp-sensor-v2
firmware: 1.4.2
identity:
cert: "sha256-of-device-cert"
credentialScope:
publish:
- "devices/sensor-7/temperature"
- "devices/sensor-7/status"
subscribe:
- "devices/sensor-7/commands"
# No wildcard, no fleet-wide topics (P9).
provisioning:
enrolledAt: 2024-01-15T00:00:00Z
rotatesEvery: 90d
# Per-device credential; never shared (P9, domains/security/secrets).
```
## Telemetry from Devices (P10 Edge Observability Survives Partition)
- Device telemetry is local-first (P10): the device buffers telemetry
on-node and forwards on reconnect. A fire-and-forget telemetry
pipeline loses data when the link drops; a buffered pipeline
survives. The buffer is bounded by the device class (P3): a
class-0 sensor buffers minutes of telemetry, not hours.
- Telemetry is observable in aggregate: the operator sees the fleet's
behavior, not just per-device. A device that has not reported in
its expected interval is itself a signal (a dead device, a
partitioned device, a drained battery). See
`domains/observability/metrics` for the generic structured-metrics
discipline; edge owns the partition-survivable, local-first angle.
- Telemetry must not be a secrets channel (P9 analog, see
`domains/observability/P6 No Secrets in Observability`): device
credentials, PII, and personally-identifying location must not
enter telemetry payloads.
## Command Idempotency (P5 Edge Operations are Idempotent)
- Device commands are retried by nature (the network is
partition-prone). Every command carries an idempotency key so a
retried command does not double-apply (P5). A `set-point` command
retried with the same idempotency key sets the point once, not
twice; an `open-valve` command retried is safe because the valve
is already open.
- Irreversible commands (a calibration burn-in, a firmware flash)
require stronger idempotency: the device tracks applied
idempotency keys and refuses re-application. A retried irreversible
command without idempotency tracking double-applies the effect
(P5 violation, possibly a physical-side-effect bug).
- The idempotency key is per-command, not per-device. A device that
dedups by device ID alone will drop distinct commands issued in
the same window. Use `(device, command-id, ts-window)` or a
UUID per command.
## Partial Degradation When Devices Drop (P7 Partial Degradation is Engineered)
- A fleet degrades when devices drop (battery exhaustion, partition,
hardware failure). The system must continue to operate with the
remaining devices; a whole-system crash on one device's failure is
a P7 violation. The degraded mode is documented: which
aggregations are valid with N-1 devices, which alerts fire, which
fallbacks engage.
- A device that drops is not an incident by itself — fleets expect
churn. The operator-facing signal is the *aggregate* health (X%
of devices reporting, Y% partitioned for >Z minutes), not the
per-device drop. Per-device drop alerts are noise; aggregate
degradation alerts are signal (see `domains/observability/metrics`).
- A command to a dropped device must time out (P5 — idempotent
retry) and degrade (P7 — the fleet continues without that
device). A command that blocks forever waiting for a dropped
device is the `blocking-call-on-constrained-device` chaos
anti-pattern (P3 + P5 breach).
## Cross-Link to Messaging (P5, cross-link messaging/queues)
- MQTT QoS 0/1/2 maps to at-most-once / at-least-once / exactly-once
delivery semantics — the same three-way tradeoff documented in
`domains/messaging/queues`. The cross-link is one-directional
outward (edge → messaging) per D-026 extended: edge owns the
constrained-device protocol angle; messaging owns the generic
cross-process delivery-semantics angle.
- This link dangles until P2 (the messaging domain is authored in
P2); it is verified bidirectional in P5 (ATELIER-114 per
IDEATE-40). Acceptable per D-053 (vertical-slice integrity — P1
ships the edge domain self-consistent; the messaging cross-link
resolves by the P6 ship).
- The parallel: a constrained device's QoS 1 publish is the
device-flavored instance of an at-least-once queue delivery — the
consumer (the broker or the downstream service) must be
idempotent (P5 here, `messaging/P3 Consumers are Idempotent`
there). The idempotency discipline is the same; the protocol and
failure model differ (constrained-device link vs broker-backed
network).
## What Violates IoT-at-the-Edge Discipline
| Violation | Principle |
|-----------|-----------|
| Undeclared device resource budget (assumes infinite battery/RAM) | P3 Resources are Constrained and Declared |
| Shared fleet credential (one key for all devices) | P9 Identity is Constrained at the Edge |
| Non-idempotent device command (retried command doubles the effect) | P5 Edge Operations are Idempotent |
| Blocking synchronous call on a constrained device with no timeout | P3, P5 (blocks the node; retry unsafe) |
| Fire-and-forget telemetry with no on-device buffer (lost on partition) | P10 Edge Observability Survives Partition |
| Whole-system crash on one device's failure (no degradation contract) | P7 Partial Degradation is Engineered |
| Device credential scope that includes fleet-wide topics (over-scoped) | P9, `domains/security/secrets` |
| QoS 2 used on a class-0 device (no budget for the handshake) | P3 Resources are Constrained and Declared |
| Per-device-drop alert (noise; aggregate degradation is the signal) | P7, `domains/observability/metrics` |
| Telemetry payload that includes device credentials or PII | P9, `domains/observability/P6 No Secrets in Observability` |
+354
View File
@@ -0,0 +1,354 @@
# Offline-First — Derived Rules
> Derives from `domains/edge/first-principles.md`. Applies P2
> (Offline is a First-Class State) primarily, with P5 (idempotent
> queue-and-forward), P4 (bounded sync conflicts on reconnect), P7
> (partial degradation), and P10 (local-first telemetry). Cross-links
> `domains/concurrency/patterns` for the in-process bounded-buffer
> analog and `domains/observability/logging` for local-first logging.
## What Offline-First Is (P2 Offline is a First-Class State)
- Offline-first is the design discipline in which the system
continues to operate when disconnected from the center. Partition
is the norm, not the exception; reconciliation happens on
reconnect. The offline state is engineered, not a degenerate mode
the app falls into by accident.
- The boundary is per D-061: edge owns the
proximity/location/disconnection angle. An offline-first web app
is an edge concern because its defining trait is partition-survival
(P2), not generic performance. The local-first storage is the
edge device's constrained-resource reality (P3).
- Offline-first is the precondition for the bounded-conflict
discipline of `P4 Sync Conflicts are Bounded, Not Infinite`:
without offline operation there is nothing to reconcile; with it,
the reconnect reconciliation is the correctness mechanism. See
`domains/edge/sync.md` for the conflict-resolution strategies.
## Local-First Storage (P2, P3)
- Local-first storage holds the working copy on the device:
IndexedDB (browser), SQLite (mobile, embedded), or on-device file
storage (desktop, IoT gateway). The local store is the authority
while offline; the server is reconciled later, not consulted per
read.
- The local store is bounded by the device (P3 — Resources are
Constrained and Declared). A local store that grows without bound
is a defect: declare a budget (e.g., a 50 MB IndexedDB quota, a
30-day rolling window), and evict outside the budget deterministically.
- The local store is the offline state; without it the app is
online-only and crashes on disconnect (P2 violation). The store is
the reversibility mechanism (C5): every local write is reversible
on reconcile.
```typescript
// Local-first store sketch (IndexedDB). The app reads from the
// local store, never the network, while offline. Writes queue
// locally and forward on reconnect (P2, P5).
const db = await openDB("atelier-offline", 1, {
upgrade(db) {
const store = db.createObjectStore("pending-writes", {
keyPath: "id",
});
store.createIndex("by-createdAt", "createdAt");
},
});
async function readRecord(id: string) {
// Read from local store first; the network is a reconcile path,
// not the read path.
return db.get("pending-writes", id);
}
```
## Queue-and-Forward for Writes (P5 Edge Operations are Idempotent)
- Every write while offline is queued locally and forwarded to the
server on reconnect. The queue is the offline write-queue; the
forward is the reconcile. Each queued write carries an idempotency
key so a retried forward (the network is partition-prone) does not
double-apply (P5).
- The queue is bounded (P3): a queue that grows without limit on a
constrained device will exhaust it. Declare a max-queue-depth and
a max-queue-bytes; reject or evict beyond the bound with a defined
policy (oldest-first, lowest-priority-first).
- The queue is the cross-partition analog of the in-process bounded
buffer — see `domains/concurrency/patterns` (bounded buffer,
backpressure). Concurrency owns the in-process analog; edge owns
the partition-survivable analog. The failure model differs: the
in-process buffer fails by OOM; the offline write-queue fails by
partition or device loss.
```typescript
// Offline write-queue sketch. Each entry carries an idempotency
// key (P5) so a retried forward is safe. The queue is bounded by
// maxDepth (P3).
interface PendingWrite {
id: string; // local id
idempotencyKey: string; // server-side dedup key (P5)
collection: string;
payload: unknown;
createdAt: number;
}
const MAX_DEPTH = 1000;
async function queueWrite(write: Omit<PendingWrite, "id" | "idempotencyKey" | "createdAt">) {
const depth = await db.count("pending-writes");
if (depth >= MAX_DEPTH) {
// P3: bounded queue. Evict the oldest pending write or reject.
// Rejecting is correct when the write is higher-priority than
// the oldest; evicting is correct when the newest is lowest.
throw new Error("offline-queue-full");
}
const entry: PendingWrite = {
...write,
id: crypto.randomUUID(),
idempotencyKey: `${write.collection}:${crypto.randomUUID()}`,
createdAt: Date.now(),
};
await db.put("pending-writes", entry);
// The forward loop picks this up when connectivity returns.
}
async function forwardPendingWrites(server: Server) {
const pending = await db.getAllFromIndex("pending-writes", "by-createdAt");
for (const write of pending) {
// P5: idempotent — the server dedups by idempotencyKey.
await server.apply(write, write.idempotencyKey);
await db.delete("pending-writes", write.id);
}
}
```
## Conflict Detection on Reconnect (P4 Sync Conflicts are Bounded)
- On reconnect, the queued writes are forwarded; the server may
have advanced while the device was offline. A conflict is when the
local write and the server state diverge. Conflict detection is
the precondition for bounded reconciliation (P4): a write forwarded
blindly (last-write-wins with no clock) is a P4 violation waiting
to happen.
- Conflict resolution strategies (CRDT, LWW with vector clocks,
application-specific merge) are the subject of
`domains/edge/sync.md` — the CRDT-vs-LWW decision matrix there
determines which applies. Offline-first owns the *detection*; sync
owns the *resolution*.
- A reconnect that detects no conflicts when conflicts exist is a
silent correctness defect (C1, P4). Detection must be conservative:
when in doubt, flag a conflict and surface it to the merge
function or the user.
## UI for Offline State (P7 Partial Degradation is Engineered)
- The UI must reflect the offline state visibly: a "you are offline,
changes will sync when connected" banner, a pending-writes counter,
a last-synced timestamp. A UI that hides the offline state
violates P7 — the degraded mode is a designed state with a defined
contract, not a silent fall-through.
- The UI must function while offline: reads from local-first
storage, writes to the queue, navigation that does not require the
network. An app that shows a blank screen or a spinner-forever when
offline has no offline state (P2 violation) and no degradation
contract (P7 violation).
- The pending-writes counter is the local-first analog of the
messaging consumer-lag metric — see
`domains/observability/metrics` for the lag-discipline parallel.
## Service Workers (P2, P6)
- A service worker is a client-side proxy that intercepts network
requests and serves from a local cache. It is the browser's
offline-first primitive: the service worker cache is the
offline-capable store for assets; the IndexedDB store is the
offline-capable store for data.
- The service worker cache is an edge cache (P6 — Cache Invalidation
is Explicit): it must carry a TTL or explicit invalidation
strategy. A service worker that caches forever and never
invalidates is a TTL-less edge cache under partition — a P6
violation (stale-forever).
- See `domains/edge/cdn.md` for the generic edge-cache invalidation
discipline; the service worker is the on-device instance of it.
```javascript
// Service worker cache strategy: stale-while-revalidate for
// assets, network-first for data, explicit version-bump for
// breaking changes (P6).
const CACHE = "atelier-v3"; // bump on deploy to invalidate (P6)
const ASSETS = ["/", "/app.js", "/styles.css"];
self.addEventListener("install", (event) => {
event.waitUntil(
caches.open(CACHE).then((cache) => cache.addAll(ASSETS))
);
});
self.addEventListener("fetch", (event) => {
const url = new URL(event.request.url);
if (url.pathname.startsWith("/api/")) {
// Network-first for data; fall back to cache on partition (P2).
event.respondWith(
fetch(event.request).catch(() => caches.match(event.request))
);
} else {
// Stale-while-revalidate for assets (P6 explicit invalidation).
event.respondWith(
caches.open(CACHE).then(async (cache) => {
const cached = await cache.match(event.request);
const network = fetch(event.request).then((resp) => {
cache.put(event.request, resp.clone());
return resp;
}).catch(() => cached);
return cached || network;
})
);
}
});
self.addEventListener("activate", (event) => {
// P6: explicit invalidation. Drop old caches on activate.
event.waitUntil(
caches.keys().then((keys) =>
Promise.all(keys.filter((k) => k !== CACHE).map((k) => caches.delete(k)))
)
);
});
```
## Offline Write-Queue and Conflict Detection Mapped to the Testing Pyramid (IDEATE-38, ATELIER-94)
The offline write-queue and conflict-detection patterns must be
tested at every tier of the testing pyramid. Each tier exercises a
different failure mode; skipping a tier leaves a correctness gap
(P2, P4 violations that surface only in production partitions).
| Pyramid Tier | What it exercises | What it proves |
|-------------|-------------------|----------------|
| **Unit** | Conflict detection on a merge function (pure inputs → expected merge result) | The merge logic is correct in isolation (P4) — given two divergent states, the merge returns the converged state |
| **Integration** | Reconnect reconcile against a local store (fake server, real IndexedDB/SQLite) | The queue-and-forward loop drains correctly; the local store and server converge after reconnect (P2, P5) |
| **E2e** | Partition simulation with a fake network (the app runs in a browser, the network is cut and restored) | The offline state, UI, and reconcile work end-to-end under partition (P2, P7) |
- **Unit — conflict detection on a merge function.** The merge
function is pure: given two divergent states and a clock, it
returns the converged state. Test every merge case (LWW, CRDT
register, set union, application-specific three-way merge) as a
pure function. This is the cheapest tier and the highest coverage
per test — see `domains/testing/pyramid`.
```typescript
// Unit test sketch: conflict detection on a merge function (P4).
// The merge function is pure; no network, no store. Test that
// divergent states converge and that the merge is bounded (no
// oscillation).
function mergeLWW(local: State, remote: State, clock: Clock): State {
// Last-write-wins: the state with the later vector-clock wins.
// Returns the converged state (P4).
return clock.compare(local.clock, remote.clock) >= 0 ? local : remote;
}
// Unit cases:
// - local ahead → local wins
// - remote ahead → remote wins
// - concurrent (clocks incomparable) → conflict flagged or LWW tiebreak
// - identical → no-op convergence (bounded, no oscillation)
test("mergeLWW converges when local is ahead", () => {
const local = { v: 2, clock: { a: 2 } };
const remote = { v: 1, clock: { a: 1 } };
expect(mergeLWW(local, remote, { compare: (a, b) => a.a - b.a })).toEqual(local);
});
```
- **Integration — reconnect reconcile against a local store.** A
fake server stands in for the network; the real IndexedDB (or
SQLite) holds the queue. The test fills the queue while offline,
reconnects, and asserts the queue drains and the server and local
store converge. This exercises the queue-and-forward loop (P5)
and the reconcile against real storage.
```typescript
// Integration test sketch: reconnect reconcile against a local
// store. A fake server; real IndexedDB. The queue drains; the
// server and local store converge after reconnect (P2, P5).
test("reconnect reconciles pending writes against the server", async () => {
const db = await openDB("test-offline", 1, { /* schema */ });
const server = new FakeServer();
await queueWrite(db, { collection: "docs", payload: { v: 1 } });
// Simulate offline: server is unreachable.
server.offline();
await queueWrite(db, { collection: "docs", payload: { v: 2 } });
expect(await db.count("pending-writes")).toBe(2);
// Simulate reconnect: server is reachable.
server.online();
await forwardPendingWrites(server, db);
expect(await db.count("pending-writes")).toBe(0);
expect(await server.latest("docs")).toEqual({ v: 2 });
});
```
- **E2e — partition simulation with a fake network.** The app runs
in a real browser; a fake network layer cuts and restores the
connection. The test asserts the UI shows the offline state, the
writes queue, the reconnect reconciles, and the UI returns to
online. This is the highest-fidelity tier and the lowest coverage
per test — run a small number of representative scenarios, not a
combinatorial matrix.
```typescript
// E2e test sketch: partition simulation with a fake network. The
// app runs in a browser; the network is cut and restored. Asserts
// the offline UI state, the queue, the reconcile, and the online
// recovery (P2, P7).
test("app survives a network partition and reconciles on reconnect", async () => {
await page.goto("https://app.example.com");
await page.click("text=Edit document");
await page.fill("textarea", "offline edit");
// Cut the network.
await page.setOffline(true);
await page.click("text=Save");
await expect(page.locator("text=You are offline")).toBeVisible();
await expect(page.locator("text=1 pending change")).toBeVisible();
// Restore the network.
await page.setOffline(false);
await expect(page.locator("text=All changes synced")).toBeVisible();
await expect(page.locator("text=0 pending changes")).toBeVisible();
});
```
- The three tiers are complementary: unit proves the merge logic,
integration proves the reconcile loop, e2e proves the partition
behavior. Skipping any tier leaves a correctness gap. See
`domains/testing/pyramid` for the pyramid discipline and
`domains/testing/fixtures` for the fake-server and fake-network
fixture patterns.
## Observability (P10 Edge Observability Survives Partition)
- The offline state is itself an observable signal: the
pending-writes count, the last-synced timestamp, the
reconcile-failure count. A device stuck offline for days with a
full queue is an incident; without local-first telemetry it is
invisible (P10 violation).
- Local-first logging (buffered on-device, forwarded on reconnect)
is the offline-first instance of `P10 Edge Observability Survives
Partition`. See `domains/observability/logging` for the generic
structured-logging discipline the local-first buffer builds on.
- A reconcile failure that is not logged locally is a silent defect
— the operator cannot debug what they cannot see (C7, P10).
## What Violates Offline-First Discipline
| Violation | Principle |
|-----------|-----------|
| App that crashes on disconnect (no offline state) | P2 Offline is a First-Class State |
| Unbounded offline write-queue (grows until device OOM) | P3 Resources are Constrained and Declared |
| Queued write forwarded without an idempotency key (retry doubles the effect) | P5 Edge Operations are Idempotent |
| Reconnect that detects no conflicts when conflicts exist | P4 Sync Conflicts are Bounded, Not Infinite |
| UI that hides the offline state (no banner, no pending counter) | P7 Partial Degradation is Engineered |
| Service worker cache with no TTL and no explicit invalidation | P6 Cache Invalidation is Explicit |
| Reconcile failure with no local log (silent under partition) | P10 Edge Observability Survives Partition |
| Merge function that oscillates (no convergence guarantee) | P4, `domains/edge/sync.md` |
| Local-first store with no declared budget (grows without bound) | P3, `domains/concurrency/patterns` (bounded buffer analog) |
| E2e tests that never simulate a partition (offline path untested) | P2, `domains/testing/pyramid` |
+315
View File
@@ -0,0 +1,315 @@
# Sync — Derived Rules
> Derives from `domains/edge/first-principles.md`. Applies P4 (Sync
> Conflicts are Bounded, Not Infinite) primarily, with P5
> (idempotent merge operations), P2 (offline as the precondition),
> and P10 (sync is observable). Cross-links `domains/data/migrations`
> for schema migration under sync and `domains/concurrency/patterns`
> for the immutability-aid-merge principle.
## What the Sync Problem Is (P4 Sync Conflicts are Bounded, Not Infinite)
- Sync is the discipline of reconciling divergent state across
partitioned nodes. While partitioned, each node accepts writes
independently; on reconnect, the divergent state must converge.
The correctness contract is that the merge terminates and
converges — oscillation and infinite sync loops are correctness
failures, not eventual consistency (P4).
- Sync is the edge domain's deepest problem: it is the
reconciliation layer above `P2 Offline is a First-Class State`.
Without offline operation there is nothing to sync; with it, the
reconnect reconciliation is the correctness mechanism. See
`domains/edge/offline-first.md` for the offline write-queue that
produces the divergent state to be reconciled.
- The boundary is per D-061: edge owns the partitioned-reconcile
angle; concurrency owns the in-process analog
(`concurrency/P1 Immutability by Default` — immutability aids
merge); data owns the generic migration discipline
(`data/migrations`). Sync is an edge concern because its defining
trait is partitioned divergence, a concern that only arises at the
network edge.
## Conflict Resolution Strategies (P4, C1 Correctness, C5 Reversibility)
- A conflict is when two nodes have divergent state for the same
logical entity and no total order determines which is correct.
Resolution strategies fall into two families:
- **Conflict-free**: the data type guarantees convergence by
construction (CRDTs). The merge is deterministic; no conflict
surfaces to the user or the application.
- **Conflict-tolerant**: the data type can conflict; the
resolution policy (last-write-win, three-way merge,
application-specific) arbitrates. Conflicts may surface to the
user or be silently resolved per a documented policy.
- The choice is a P4 decision: conflict-free types guarantee the
bound (convergence) but constrain the data model; conflict-tolerant
types are flexible but require the resolution policy to be correct
and bounded (no oscillation). See the decision matrix below.
## CRDTs — Conflict-Free Replicated Data Types (P4, C5, C6)
- A CRDT is a data type whose merge operation is associative,
commutative, and idempotent. Given any set of divergent replicas,
merging them in any order converges to the same state — the merge
is deterministic and terminating (P4 bound). CRDTs derive from C5
Reversibility (divergent state reverses to convergence) and C6
Composability (CRDTs compose: a CRDT map of CRDT registers is
itself a CRDT).
- **State-based (CvRDT — convergent):** each replica carries its
full state; merge is a least-upper-bound on a semi-lattice. The
payload is larger (full state per merge); the merge is simple
(one function). Fits small state and unreliable networks.
- **Operation-based (CmRDT — commutative):** each replica carries
operations; merge is applying the operations in causal order. The
payload is smaller (ops, not state); the delivery must be
reliable and causally ordered. Fits large state and reliable
transport.
- The tradeoff: state-based is simpler but heavier; operation-based
is lighter but requires causal delivery. Both guarantee
convergence (P4); the choice is a C8 Economy decision (bandwidth
vs delivery complexity).
```typescript
// CRDT register: LWW-element-set (state-based, CvRDT). The merge
// is deterministic — the register with the later timestamp wins.
// Convergence is guaranteed (P4); the merge is idempotent (P5).
interface LWWRegister<T> {
value: T;
timestamp: number; // monotonic clock; ties broken by node id
nodeId: string;
}
function mergeLWWRegister<T>(
local: LWWRegister<T>,
remote: LWWRegister<T>,
): LWWRegister<T> {
// The merge is associative, commutative, idempotent (P4, P5).
// (local.timestamp, local.nodeId) > (remote.timestamp, remote.nodeId)
// is a total order — no oscillation, no infinite loop.
if (local.timestamp > remote.timestamp) return local;
if (local.timestamp < remote.timestamp) return remote;
// Tie: break by node id for a deterministic total order.
return local.nodeId > remote.nodeId ? local : remote;
}
// The register is a CRDT: merge(merge(a, b), c) === merge(a, merge(b, c))
// for any replicas a, b, c. Convergence is guaranteed (P4).
```
```typescript
// CRDT set: add-wins last-write-wins element set (state-based).
// Each element carries a timestamp; remove only wins if the
// remove-timestamp is later than the add-timestamp. This avoids
// the remove-wins-vs-add race (P4) without surfacing a conflict.
interface AWLWWSet<T> {
adds: Map<T, number>; // element -> add-timestamp
removes: Map<T, number>; // element -> remove-timestamp
}
function mergeAWLWWSet<T>(a: AWLWWSet<T>, b: AWLWWSet<T>): AWLWWSet<T> {
const adds = new Map<T, number>(a.adds);
const removes = new Map<T, number>(a.removes);
for (const [el, ts] of b.adds) {
adds.set(el, Math.max(adds.get(el) ?? 0, ts)); // add-wins union
}
for (const [el, ts] of b.removes) {
removes.set(el, Math.max(removes.get(el) ?? 0, ts));
}
return { adds, removes };
}
function contains<T>(set: AWLWWSet<T>, el: T): boolean {
const addTs = set.adds.get(el) ?? 0;
const rmTs = set.removes.get(el) ?? 0;
return addTs > rmTs; // add wins on equal timestamp (P4 bounded)
}
```
## Last-Write-Win (LWW) with Vector Clocks (P4, C1, C5)
- LWW is the simplest conflict-tolerant strategy: the write with the
latest timestamp wins. It is cheap, but it silently discards
concurrent writes — the "lost update" is the correctness cost. LWW
is correct only when the timestamp is a total order (a monotonic
clock, not wall time), and when lost concurrent writes are
acceptable (e.g., caching, presence, ephemeral state).
- **Vector clocks** are the timestamp that knows about concurrency.
A vector clock records the logical time of each node; two writes
are concurrent iff neither vector dominates the other. LWW with
vector clocks: a write that is causally later wins; a write that
is concurrent conflicts and is resolved by a tiebreak (node id,
wall time, or application policy).
- The tiebreak is the P4 bound: the conflict must be resolved
deterministically (no oscillation) and the resolution must be
documented. A tiebreak by wall time alone (no vector clock) is a
P4 violation waiting to happen — wall time skews across nodes,
and a clock skew can flip the tiebreak, oscillating the merge.
```typescript
// LWW with vector clocks (conflict-tolerant, P4 bounded). The
// vector clock records causal order; concurrent writes conflict;
// the conflict is tiebroken deterministically (no oscillation).
type VectorClock = Record<string, number>; // nodeId -> counter
function compareClock(a: VectorClock, b: VectorClock): "before" | "after" | "equal" | "concurrent" {
let aBefore = false, bBefore = false;
const keys = new Set([...Object.keys(a), ...Object.keys(b)]);
for (const k of keys) {
const av = a[k] ?? 0;
const bv = b[k] ?? 0;
if (av < bv) aBefore = true;
if (av > bv) bBefore = true;
}
if (aBefore && bBefore) return "concurrent";
if (aBefore) return "before";
if (bBefore) return "after";
return "equal";
}
interface LWWVectorState<T> {
value: T;
clock: VectorClock;
writerId: string; // tiebreak: deterministic, no oscillation (P4)
}
function mergeLWWVector<T>(
local: LWWVectorState<T>,
remote: LWWVectorState<T>,
): LWWVectorState<T> {
const order = compareClock(local.clock, remote.clock);
if (order === "before") return remote; // remote causally later
if (order === "after" || order === "equal") return local;
// Concurrent: tiebreak by writer id (deterministic, P4 bounded).
return local.writerId > remote.writerId ? local : remote;
}
```
- The merge is idempotent (P5): merging the same two replicas twice
yields the same result. The tiebreak by `writerId` is a total
order, so the merge cannot oscillate (P4 bound).
- A vector-clock merge that surfaces the concurrent conflict to the
application (instead of tiebreaking) is also valid — the
application resolves per its own policy. The P4 bound is that the
resolution terminates; the policy determines whether the user sees
the conflict or the system silences it.
## Merge Semantics (P4, P5, cross-link concurrency/patterns)
- The merge function is the heart of sync. Its properties (P4):
- **Associative**: `merge(merge(a, b), c) === merge(a, merge(b, c))`.
- **Commutative**: `merge(a, b) === merge(b, a)`.
- **Idempotent**: `merge(a, a) === a` (P5 — retried merges are safe).
- Immutability aids merge: an immutable state representation (the
CRDT payload, the LWW register with a clock) makes the merge a
pure function of two inputs, with no in-place mutation race. See
`domains/concurrency/patterns` (`concurrency/P1 Immutability by
Default`) for the in-process immutability principle; sync is the
cross-partition instance of it.
- A merge that mutates in place is a P5 violation waiting to
happen: a retried merge mutates the same state twice, and the
result is not idempotent. Always merge into a new state; never
mutate the inputs.
## Conflict-Free vs Conflict-Tolerant Data Types (P4, C3 Simplicity)
- **Conflict-free (CRDTs):** the data type guarantees convergence.
The application never sees a conflict; the merge is deterministic.
The cost: the data model is constrained (counters, sets, registers,
maps of these). A conflict-free type for arbitrary JSON is hard;
a conflict-free type for a counter is a PN-counter.
- **Conflict-tolerant (LWW, three-way merge, application policy):**
the data type can conflict; the resolution policy arbitrates. The
cost: the policy must be correct and bounded (no oscillation), and
the conflict may surface to the user. The benefit: any data model
can be made conflict-tolerant (just pick a tiebreak).
- The choice is the decision matrix below. It is a P4 decision
(which bound), a C1 decision (which correctness cost is
acceptable), and a C3 decision (which simplicity is affordable).
See also `domains/data/migrations` for the schema-evolution angle
— a schema change under sync must be compatible with both
replicas, or the merge fails on the new shape.
## Schema Migration Under Sync (P4, cross-link data/migrations)
- A schema migration under sync is harder than a single-node
migration: both replicas must understand the new shape, or the
merge fails. The migration must be forward-and-backward compatible
across all replicas that may still hold the old shape — see
`domains/data/migrations` for the generic compatibility discipline.
- A breaking schema change under sync requires a staged migration:
deploy the new-shape-aware merge first (it accepts both shapes),
then deploy the new shape, then deploy the old-shape-removing
merge. A big-bang schema change under sync is a P4 violation: the
replicas that have not yet upgraded will fail the merge, and the
sync will not converge.
- The merge function's version awareness is the P4 bound: the merge
must handle every shape version that may exist in the fleet, or
reject (and surface) the merge rather than silently corrupting.
## CRDT vs Last-Write-Win — Decision Matrix (D-069)
| Strategy | When | Correctness Guarantee | Operational Cost | Failure Mode |
|----------|------|------------------------|-------------------|--------------|
| CRDT (state-based, CvRDT) | The data model fits a CRDT (counter, set, register, map of these); convergence must be guaranteed without surfacing conflicts; the network is unreliable (full-state merge tolerates dropped ops) | Strong eventual convergence — `merge(a, b) === merge(b, a)` for any replicas (P4 bound by construction) | Medium — full state per merge (bandwidth); semi-lattice merge function per type; CRDT library or hand-rolled | A bug in the merge function = silent divergence (C1); large state = bandwidth cost on constrained links (P3) |
| CRDT (operation-based, CmRDT) | The data model fits a CRDT; bandwidth is constrained (ops are smaller than state); the transport is reliable and causally ordered | Strong eventual convergence — same guarantee, smaller payload | High — requires causal delivery (vector clock or broker with ordering); op transform must be idempotent (P5) | Causal-delivery violation = lost ops = divergence; op-transform bug = silent divergence |
| Last-Write-Win (LWW) with vector clocks | The data model is arbitrary (any JSON, any record); concurrent writes are acceptable to discard or tiebreak; a total order tiebreak (node id) is acceptable | Bounded convergence — causally-later writes win; concurrent writes are tiebroken deterministically (P4 bound via tiebreak) | Low — simple merge (compare clocks, pick winner); no CRDT library; small payload | Concurrent writes are silently discarded (lost update); tiebreak by wall time = clock-skew oscillation (P4 violation); no vector clock = no concurrent-write detection = silent loss |
| LWW with wall-clock timestamp only | The data model is ephemeral (cache, presence); lost updates are acceptable; the clock is roughly synchronized (NTP) | Weak — convergence eventually, but concurrent writes may oscillate with clock skew; no concurrent-write detection | Lowest — one timestamp per write; no clock vector | Clock skew = oscillation (P4 violation); concurrent writes silently lost; not a correctness-safe strategy for durable state |
| Three-way merge (application-specific) | The data model is structured (documents, forms); conflicts should surface to the user or a domain-specific resolver; the merge is field-level | Bounded if the merge function is correct (associative, commutative, idempotent — P4, P5); conflicts surface per field | High — application-specific merge function per type; UI for conflict resolution; user-facing conflict surface | Merge-function bug = silent divergence or oscillation; unbounded conflict UI = user fatigue |
- The default for structured state that must converge silently is a
**CRDT** (state-based for unreliable networks, operation-based for
bandwidth-constrained reliable transport). The default for
arbitrary JSON where lost concurrent updates are acceptable is
**LWW with vector clocks** (never wall-clock-only for durable
state). The default for user-facing documents where conflicts
should surface is **three-way merge** with a documented resolution
policy.
- The failure-mode column is the P4 check: every row except
wall-clock-only LWW carries a bounded failure mode (the bug is in
the implementation, not the strategy). Wall-clock-only LWW carries
an unbounded failure mode (clock skew = oscillation) and is a P4
violation for durable state. Use it only for ephemeral state
where lost updates are acceptable.
- The choice is a P4 decision (which bound) and a C1 decision
(which correctness cost). A CRDT guarantees convergence but
constrains the data model; LWW is flexible but discards concurrent
writes. Neither is universally correct; the matrix is the
decision tool.
## Observability of Sync (P10 Edge Observability Survives Partition)
- Sync is itself an observable operation: the merge count, the
conflict count, the convergence lag (time from reconnect to
convergence), and the divergent-replica count are first-class
signals. A sync that runs forever without converging is the
`edge-sync-loop` chaos anti-pattern (P4 breach); without
observability it is invisible until the user notices the stale
state.
- A conflict that is silently resolved should be logged (the
resolution policy applied, the discarded write's idempotency key,
the winning write's clock). A conflict that surfaces to the user
should be metricated (the conflict rate, the resolution time).
See `domains/observability/metrics` for the generic discipline.
- A divergent replica that has not converged after the expected
window is an incident; without a metric it is invisible (P10
breach). Wire sync convergence to an alert — the
`divergent-replica-count` is the sync analog of the messaging
`consumer-lag` metric.
## What Violates Sync Discipline
| Violation | Principle |
|-----------|-----------|
| Sync loop that oscillates forever (CRDT without merge-semantics, LWW without monotonic clock) | P4 Sync Conflicts are Bounded, Not Infinite |
| LWW with wall-clock timestamp only on durable state (clock skew = oscillation) | P4, C1 (no concurrent-write detection) |
| Merge function that mutates inputs in place (retried merge is not idempotent) | P5 Edge Operations are Idempotent, `domains/concurrency/patterns` |
| Big-bang schema change under sync (replicas fail the merge) | P4, `domains/data/migrations` |
| Conflict silently resolved with no log (the policy is invisible) | P10 Edge Observability Survives Partition |
| Divergent replica with no convergence-lag metric (invisible stale state) | P10, `domains/observability/metrics` |
| Three-way merge with an unbounded conflict UI (user fatigue, no termination) | P4 (the merge must terminate) |
| Operation-based CRDT without causal delivery (lost ops = divergence) | P4, C1 (the delivery contract is the bound) |
| Merge that surfaces every concurrent conflict to the user (no default policy) | P4, C3 (the default policy is the simplicity bound) |
| Sync with no convergence test (the merge is untested under partition) | P4, `domains/testing/pyramid` |
+379
View File
@@ -0,0 +1,379 @@
# Delivery Semantics — Derived Rules
> Derives from `domains/messaging/first-principles.md`. Applies P4
> (Delivery Semantics are Explicit), P3 (Consumers are Idempotent),
> P2 (Ordering is a Property, Not an Assumption), P5 (Dead-Letter
> Handling is Defined), and P6 (Backpressure is Bounded) primarily,
> with P10 (DLQ depth as an alert). For the dead-letter strategy
> decision, see the comparison table below. Includes a fenced
> idempotency-key dedup-store example (IDEATE-39). Cross-links
> `domains/concurrency/patterns` for the in-process retry/backoff
> analog, `domains/errors/patterns` for errors as data for message
> failures, and `domains/observability/metrics` for DLQ depth as an
> alert.
## The Three Delivery Semantics (P4 Delivery Semantics are Explicit)
- **At-most-once**: a message is delivered 0 or 1 times; loss is
possible, duplication is not. The producer fires-and-forgets; the
broker does not ack; the consumer does not dedup. Lowest latency,
lowest implementation cost, lossy. Fits telemetry where a dropped
sample is acceptable (MQTT QoS 0, fire-and-forget metrics).
- **At-least-once**: a message is delivered 1 or more times;
duplication is possible, loss is not. The producer sends and
waits for a broker ack; the consumer processes and acks; a crash
before the consumer's ack triggers redelivery. The consumer must
be idempotent (P3). The default for side-effecting operations
(orders, payments, commands). The vast majority of broker-backed
queues (SQS standard, RabbitMQ ack, MQTT QoS 1).
- **Exactly-once**: a message is delivered exactly 1 time; no
loss, no duplication. In practice, this is at-least-once plus
idempotency (P3), or a transactional consume-process-produce loop
(P4 — see `domains/messaging/streams.md`). Jepsen analyses verify
broker claims: "exactly-once" requires independent verification;
the durable engineering practice is at-least-once with idempotent
consumers (P3), which collapses to exactly-once under correct
dedup.
- The semantic is declared per channel (P4), not emergent. An
unstated semantic is a defect: the consumer guesses, and the
guess is wrong under the first failure. See
`domains/messaging/queues.md` for the three-semantics comparison
table (semantics, latency cost, implementation cost, when each
fits).
## Idempotency (P3 Consumers are Idempotent)
- Idempotency is the correctness property that makes at-least-once
safe. A consumer that processes the same message twice has the
same effect as processing it once. The mechanism is the
idempotency key: a per-message unique identifier the consumer
uses to dedup redeliveries.
- The idempotency key is per-message, not per-producer or
per-session. A consumer that dedups by producer alone drops
distinct messages issued in the same window. Use a UUID per
message, or a deterministic key derived from the message content
(e.g., `(entity, operation, version)`).
- The dedup store is bounded (P6 — Backpressure is Bounded): a
dedup store that grows without bound is a memory leak. Use a TTL
window longer than the broker's max-redelivery window, or a
bounded LRU. The TTL is the P6 bound: a key seen within the TTL
is a redelivery; a key older than the TTL is expired (the broker
has given up redelivering it).
## Idempotency-Key Dedup Store (IDEATE-39, P3 — fenced example)
The dedup store is the concrete mechanism that makes a consumer
idempotent under at-least-once delivery. This is the fenced
example required by IDEATE-39 (parallel to the v0.3 IDEATE-29
signed-attestation fenced example): it is NOT prose-only — the
consumer-with-dedup-store demonstrates P3 concretely.
```python
# Idempotent consumer with a dedup store (P3 Consumers are
# Idempotent, P6 Backpressure is Bounded — the dedup store is
# TTL-bounded). The consumer dedups by idempotency key before
# processing; a redelivered message is a no-op, not a double-apply.
import time
# P6: the dedup store is bounded by a TTL window. The TTL must
# exceed the broker's max-redelivery window; beyond the TTL, the
# key is expired (the broker has given up). A dedup store with no
# TTL is a memory leak (P6 violation).
DEDUP_TTL_SECONDS = 24 * 3600 # longer than max-redelivery window
class DedupStore:
"""A TTL-bounded idempotency-key dedup store (P3, P6).
seen(key): True if the key was processed within the TTL.
mark(key): Record the key as processed (with a timestamp).
"""
def __init__(self, backend):
# backend is a Redis, a DB, or an in-process LRU. The
# backend must be shared across consumer instances if the
# subscription is shared (P2/P3 — see domains/messaging/
# pubsub.md on shared vs independent subscriptions).
self.backend = backend
def seen(self, key: str) -> bool:
ts = self.backend.get(key)
if ts is None:
return False
if time.time() - ts > DEDUP_TTL_SECONDS:
# P6: expired. The broker has given up redelivering;
# this key is no longer a redelivery signal.
self.backend.delete(key)
return False
return True
def mark(self, key: str):
self.backend.set(key, time.time(), ttl=DEDUP_TTL_SECONDS)
# The idempotent consumer: dedup before process, mark after
# process, ack after mark. A crash before mark re-processes (the
# dedup store does not have the key); a crash before ack
# redelivers (the broker did not see the ack) and the dedup store
# makes the redelivery a no-op (P3).
def consume_idempotent(broker, dedup: DedupStore, process):
for message in broker.receive():
# P3: dedup BEFORE process. A redelivered message is a
# no-op, not a double-apply.
if dedup.seen(message["idempotencyKey"]):
broker.ack(message) # already processed; skip
continue
try:
process(message["payload"])
# P3: mark AFTER process succeeds. A crash between
# process and mark re-processes (acceptable: the
# process must be idempotent OR the mark must be
# transactional with the process — see below).
dedup.mark(message["idempotencyKey"])
broker.ack(message)
except TransientError as exc:
# P6: bounded retry with backoff. Nack for redelivery;
# the broker redelivers after exponential backoff.
broker.nack(message, delay=backoff(message["attempt"]))
except PoisonError as exc:
# P5: poison message. Route to DLQ, do NOT retry
# forever (see the DLQ routing rule below).
route_to_dlq(broker, message, exc)
broker.ack(message) # remove from the origin queue
```
- The order `process → mark → ack` gives at-least-once with
idempotent dedup: a crash before `mark` re-processes (the dedup
store does not have the key), and a crash before `ack`
redelivers (the dedup store makes the redelivery a no-op). If
`process` is not itself idempotent, the `process → mark` window
must be transactional (e.g., process and mark in one DB
transaction) — otherwise a crash in the window double-applies.
- For a non-idempotent `process` (e.g., a payment that must not
double-charge), use a transactional dedup: process and mark in
one DB transaction, so the mark commits iff the process
commits. This is the "exactly-once via idempotency" pattern
(P4): at-least-once delivery plus a transactional
process-and-mark collapses to exactly-once under correct
transactional semantics.
## Ordering (P2 Ordering is a Property, Not an Assumption)
- The delivery semantic interacts with ordering (P2). At-least-once
with per-partition ordering: a redelivery within a partition
preserves order (the redelivered message re-appears in its
original position relative to other messages the consumer has
not yet seen). At-least-once with no ordering: a redelivery may
appear out of order relative to messages delivered after it.
- The consumer must not assume an ordering property the broker
does not provide (P2). A standard queue delivers per-receive-node
arrival order but no global order and no order across redeliveries;
a FIFO queue delivers strict per-group order including across
redeliveries; a partitioned stream delivers strict per-partition
order across redeliveries. Document the property; do not assume
it. See `domains/messaging/queues.md` (FIFO vs standard) and
`domains/messaging/streams.md` (per-partition order).
## Dead-Letter Strategies (P5 Dead-Letter Handling is Defined)
- A poison message (unparseable, repeatedly failing, or exhausting
the retry budget) must be routed to a dead-letter queue, not
retried forever or silently dropped (P5). The DLQ is observable
(P10 — depth is an alert) and drainable (an operator can
inspect, replay, or discard with audit).
- The dead-letter strategy determines when a message is
dead-lettered and what the operator sees. The choice is the
decision matrix below (D-069).
## Dead-Letter Strategy Comparison (D-069)
| Strategy | When It Applies | Failure Visibility | Operational Cost |
|----------|-----------------|-------------------|------------------|
| **Retry-count-limit** | A fixed max-redeliveries count (e.g., 5). After N redeliveries, route to DLQ. Simple, predictable. | The redelivery count is visible in the DLQ entry; the operator sees how many times it was retried. | Low — a counter per message; no backoff tuning. Risk: retries fire as fast as the broker redelivers, hammering a downstream that is already failing (P6 — no backoff = no backpressure escape). |
| **TTL-with-backoff** | A max time-to-live for redelivery (e.g., 30 minutes) with exponential backoff between retries. After the TTL, route to DLQ. | The TTL and the backoff schedule are visible; the operator sees the retry timeline. | Medium — backoff tuning per message type. Benefit: backoff gives the downstream time to recover (P6 — the retry rate is bounded); fits transient failures (a downstream that is briefly unavailable). |
| **Poison-queue** | A separate queue for messages that fail a specific check (unparseable, schema-invalid, unknown type) before any processing retry. Routed immediately, not retried. | The poison queue is a separate signal from the DLQ; the operator sees parse-vs-process failures distinctly. | Low — a routing rule per check. Benefit: distinguishes "never going to succeed" (poison) from "might succeed on retry" (DLQ). Use for unparseable messages that no retry will fix. |
| **DLQ + alert** | Any of the above strategies, plus an alert on DLQ depth. The DLQ is observable (P10) — depth, age, and rate are alerted. | Highest — the operator is paged on DLQ growth; the DLQ is a first-class signal, not a graveyard. | Medium — alerting setup per DLQ. This is the P5/P10 floor: a DLQ without an alert is a silent correctness defect (poison messages accumulate invisibly). |
- The default for transient failures is **TTL-with-backoff + DLQ
+ alert**: backoff gives the downstream time to recover (P6),
the TTL bounds the retry budget (P5), the DLQ captures the
unprocessable, and the alert makes it visible (P10). The default
for unparseable messages is **poison-queue + alert**: route
immediately, do not retry a message no retry will fix.
- **Retry-count-limit alone** (no backoff, no alert) is the
`messaging-unbounded-retry` chaos anti-pattern's cousin: it
caps the count but hammers the downstream at full retry rate,
and the DLQ grows silently if no alert is wired. Always pair
a retry budget with backoff (P6) and an alert (P10).
- The failure-visibility column is the P10 check: every strategy
must surface the failure to the operator. A strategy with no
visibility is a P10 violation regardless of its retry semantics.
- The operational-cost column is the C8 tradeoff: more visibility
and more backoff cost more to set up but pay back in operational
stability. The P5/P10 floor is "DLQ + alert"; below that, the
strategy is a silent defect waiting to grow.
## DLQ Routing Rule (P5, P10)
```python
# DLQ routing rule (P5 dead-letter handling, P6 bounded retry with
# backoff, P10 DLQ depth alert). Combines TTL-with-backoff for
# transient failures and poison-queue for unparseable messages,
# with an alert on DLQ depth.
import json, time
MAX_RETRY_TTL_SECONDS = 30 * 60 # 30 min total retry window
DLQ = "orders-dlq"
POISON = "orders-poison" # unparseable; never retried
def route_to_dlq(broker, message, reason):
"""Route a message to the DLQ with audit metadata (P5)."""
broker.send(DLQ, body=json.dumps({
"original": message,
"reason": str(reason),
"deadLetteredAt": now_iso(),
"redeliveryCount": message.get("attempt", 0),
}))
# P10: emit a metric so DLQ depth alerts fire. A DLQ that
# grows with no alert is a silent correctness defect (P5/P10).
metrics.increment("dlq.depth", tags={"queue": "orders"})
# The ack removes the message from the origin queue; the DLQ
# is the durable record (P5 — observable and drainable).
def consume_with_dlq(broker, dedup, process):
for message in broker.receive():
# P1: parse first. An unparseable message is poison —
# route immediately, do NOT retry (no retry will fix it).
try:
payload = json.loads(message["body"])
except (ValueError, SchemaError) as exc:
broker.send(POISON, body=json.dumps({
"original": message["body"],
"reason": f"parse-failed: {exc}",
"poisonedAt": now_iso(),
}))
metrics.increment("poison.depth", tags={"queue": "orders"})
broker.ack(message) # remove from origin; poison queue holds it
continue
# P3: dedup before process.
if dedup.seen(payload["idempotencyKey"]):
broker.ack(message); continue
attempt = payload.get("attempt", 0)
first_attempt_ts = payload.get("firstAttemptTs", time.time())
try:
process(payload)
dedup.mark(payload["idempotencyKey"])
broker.ack(message)
except TransientError as exc:
# P6: TTL-with-backoff. If the retry window is
# exhausted, route to DLQ; otherwise redeliver with
# exponential backoff.
if time.time() - first_attempt_ts > MAX_RETRY_TTL_SECONDS:
route_to_dlq(broker, message, exc) # P5
broker.ack(message)
else:
broker.nack(message, delay=backoff(attempt))
except PermanentError as exc:
# A permanent error (e.g., a not-found dependency)
# does not benefit from retry — route to DLQ now.
route_to_dlq(broker, message, exc)
broker.ack(message)
```
- The routing rule distinguishes three failure modes: **poison**
(unparseable — route immediately, no retry), **transient**
(retry with backoff until the TTL, then DLQ), and **permanent**
(a retry will not fix it — DLQ now). This distinction is the P5
discipline: not every failure is a retry; some are immediate
DLQs.
- The DLQ entry carries `reason`, `deadLetteredAt`, and
`redeliveryCount` — it is auditable (the operator knows why each
message was dead-lettered and how many times it was retried).
This is the errors-as-data discipline — see
`domains/errors/patterns` for the general principle a DLQ entry
instantiates.
## Retry Budgets and Backoff (P6 Backpressure is Bounded)
- The retry budget is the cap on redelivery: a count, a TTL, or
both. After the budget, the message routes to the DLQ (P5). An
unbounded retry budget is the `messaging-unbounded-retry` chaos
anti-pattern (P5 breach): the consumer never makes progress past
the poison message.
- **Exponential backoff** spaces retries: 1s, 2s, 4s, 8s, ... with
a jitter to avoid thundering-herd synchrony. Backoff gives the
downstream time to recover (P6 — the retry rate is bounded,
giving the downstream a chance to catch up). A retry with no
backoff hammers the downstream at full rate, making the failure
worse.
- The retry budget × the backoff schedule is the P6 bound: the
consumer's retry load is bounded by design, not by luck. See
`domains/concurrency/patterns` Pattern 6 (Timeout on Every
Block) for the in-process retry/backoff analog; messaging owns
the broker-backed instance where the redelivery comes from the
broker across a network, not an in-process loop (D-062).
## Poison Messages (P5, P1)
- A poison message is one no retry will fix: unparseable (the
schema is wrong, P1), unknown type (the consumer does not handle
this version, P9), or a permanent failure (a not-found
dependency). Retrying a poison message wastes resources and
blocks the queue (P6 — the consumer never makes progress).
- Poison messages route to the poison queue immediately (no
retry), distinct from the DLQ (which holds messages that
exhausted their retry budget on transient failures). The
distinction is the P5 discipline: a poison queue is for
"never going to succeed"; a DLQ is for "might have succeeded
but didn't within the budget."
- A poison queue without an alert is the same defect as a DLQ
without an alert (P10): the operator cannot see the poison
accumulating. Wire both to `domains/observability/metrics`.
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
- The retry/backoff discipline here is the cross-process analog
of `domains/concurrency/patterns` Pattern 6 (Timeout on Every
Block) for in-process retry. Concurrency owns the in-process
analog (a retry loop within one program, with a timeout per
attempt); messaging owns the broker-backed instance (the broker
redelivers across a network, the consumer applies backoff via
nack-with-delay). The failure model differs: in-process retry
fails by a thread crash or a timeout; broker-backed retry fails
by network partition, broker restart, or consumer crash-and-retry
(D-062).
- The cross-link is one-directional outward (messaging →
concurrency) per D-026 extended: messaging references concurrency
as the in-process foundation; concurrency does not back-link to
messaging.
## Cross-Link to Errors (P5, cross-link errors/patterns)
- A poison message is an errors-as-data instance: the failure is
captured as a DLQ entry (with `reason`, `redeliveryCount`,
`deadLetteredAt`), not swallowed. See `domains/errors/patterns`
for the general errors-as-data discipline a DLQ entry
instantiates. The DLQ is the async-messaging instance of an
error log — observable, auditable, drainable.
- The cross-link is one-directional outward (messaging → errors):
messaging references errors for the errors-as-data pattern;
errors does not back-link to messaging.
## What Violates Delivery-Semantics Discipline
| Violation | Principle |
|-----------|-----------|
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
| Dedup store with no TTL (memory leak; grows without bound) | P6 Backpressure is Bounded |
| Dedup key per-producer (distinct messages in the same window deduped) | P3 Consumers are Idempotent |
| Retry with no backoff (hammers the downstream at full rate) | P6 Backpressure is Bounded |
| Unbounded retry budget (consumer never progresses past the poison) | P5 Dead-Letter Handling is Defined |
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
| Poison message retried forever (no poison queue, no immediate DLQ) | P5 Dead-Letter Handling is Defined |
| Process-not-idempotent with non-transactional mark (crash in window double-applies) | P3, P4 |
| DLQ entry with no reason/audit metadata (uninspectable failure) | P5, `domains/errors/patterns` |
| Ordering assumption the broker does not provide (FIFO assumed on standard queue) | P2 Ordering is a Property, Not an Assumption |
+307
View File
@@ -0,0 +1,307 @@
# Messaging — First Principles
## 1. The Principles
### P1. Messages are Contracts
A message has an explicit, versioned schema. Producer and consumer
agree on shape before exchange; the schema is the boundary, not a
guess. A schemaless message — a free-form JSON blob the consumer
parses by hope — is a defect: the consumer breaks silently on the
next shape change, and the producer has no contract to evolve
against. This derives from `C1 Correctness` (the exchange must
carry what the parties agreed to) and `C2 Clarity` (the schema
makes the boundary obvious to both sides). This is the
cross-process expression of the contract discipline that
`domains/api/rest` owns for synchronous request/response: the
message schema is to async exchange what the API contract is to
sync exchange. It is distinct from
`domains/concurrency/patterns` Pattern 1 (Message Passing), which
owns the *in-process* channel primitive — here the contract spans
separate systems and survives network failure (D-062). See
`domains/messaging/queues.md` for the queue-flavored application
and `domains/messaging/streams.md` for the durable-log-flavored
application.
### P2. Ordering is a Property, Not an Assumption
Ordering guarantees — per-partition strict, global, or none — are
explicit and documented. "It's FIFO" is a claim that must be backed
by the broker's partitioning contract, not an assumption the
consumer makes and the broker may not honor. A standard queue
delivers in arrival order per receive-node but offers no global
ordering across shards; a FIFO queue delivers strict per-message-
group order but at a latency cost; a partitioned stream delivers
strict per-partition order but only within a partition. Each is a
distinct, declared property. This derives from `C1 Correctness`
(order is a correctness property — a consumer that assumes order
the broker does not provide is wrong) and `C2 Clarity` (the
ordering guarantee is documented, not discovered in production).
This is distinct from in-process ordering, which
`domains/concurrency/patterns` Pattern 1 owns for channels within
one program: messaging ordering survives network failure, broker
restart, and consumer crash-and-retry — a stronger failure model
than thread-local channels (D-062). The `messaging-shared-
subscription` chaos anti-pattern breaches this rule: two consumers
sharing one subscription break per-consumer ordering because the
broker dispatches each message to an arbitrary consumer. See
`domains/messaging/streams.md` for the partition-order contract
and `domains/messaging/delivery-semantics.md` for the interaction
of ordering with the three delivery semantics.
### P3. Consumers are Idempotent
Delivery is at-least-once by default across the network; a
consumer deduplicates via idempotency keys or deterministic
processing. "Exactly-once" is idempotency plus at-least-once, not a
broker guarantee — Jepsen analyses of Kafka, RabbitMQ, and NATS
establish that exactly-once claims require independent
verification, and the durable engineering practice is to make
consumers idempotent under redelivery. A non-idempotent consumer
under at-least-once delivery doubles the effect on every retry; a
non-idempotent consumer under a claimed exactly-once broker is a
bug waiting for the broker's exactly-once invariant to break. This
derives from `C1 Correctness`: correctness under redelivery is the
contract, not a nice-to-have. This parallels
`domains/edge/P5 Edge Operations are Idempotent` (the
cross-partition device-and-cache-flavored analog) and is the
cross-process instance of the retry-safety discipline that
`domains/concurrency/patterns` Pattern 6 (Timeout on Every Block)
implies for in-process retry. It is distinct from in-process
retry because the redelivery comes from the broker across a
network, not from an in-process loop (D-062). See
`domains/messaging/delivery-semantics.md` for the idempotency-key
dedup-store pattern.
### P4. Delivery Semantics are Explicit
At-least-once / at-most-once / exactly-once is a declared choice
per channel, not an emergent behavior. The tradeoff — latency cost,
implementation complexity, operational cost — is made consciously
and documented. At-most-once is fire-and-forget (low latency, lossy);
at-least-once is acked with possible duplication (the default,
requires idempotent consumers per P3); exactly-once is at-least-once
plus idempotency or a transactional two-phase commit (highest cost,
narrowest fit). An unstated semantic is a defect: the consumer
guesses, and the guess is wrong under the first failure. This
derives from `C1 Correctness` (the chosen semantic must hold) and
`C2 Clarity` (the tradeoff is visible to the reader and the
operator). This is the cross-process analog of the explicit-failure-
mode discipline that `domains/errors/patterns` owns for synchronous
code — messaging makes the delivery-mode choice as explicit as an
error-handling choice. See `domains/messaging/queues.md` for the
three-semantics comparison table and `domains/messaging/delivery-
semantics.md` for the correctness properties of each.
### P5. Dead-Letter Handling is Defined
Poison messages — unparseable, repeatedly failing, or exhausting
the retry budget — are routed to a dead-letter queue, not retried
forever or silently dropped. The DLQ is observable and drainable: an
operator can inspect it, replay from it, or discard with audit. An
unbounded retry loop is a livelock: the consumer never makes
progress past the poison message. A silent drop is a correctness
defect: the message vanished with no record. This derives from `C1
Correctness` (poison messages must not livelock the consumer or
silently disappear) and `C5 Reversibility` (the DLQ is the
reversibility mechanism — a dead-lettered message can be reprocessed
after the bug is fixed). This is the cross-process analog of the
bounded-error discipline that `domains/errors/patterns` owns for
synchronous code: a poison message is an error-as-data instance
that must be observable and recoverable, not swallowed. It is
distinct from in-process error handling because the failure spans a
network and a consumer restart (D-062). See
`domains/messaging/delivery-semantics.md` for the dead-letter
strategy comparison table and the DLQ routing rule pattern.
### P6. Backpressure is Bounded
A slow consumer cannot unbounded-buffer the broker or the
producer. Backpressure is explicit: consumer lag is visible,
max-unacked is bounded, the retry budget is capped. A consumer
that falls behind without a visible signal is a silent backlog —
the operator cannot fix what they cannot see, and the broker's
memory grows without bound until it fails. This derives from `C1
Correctness` (a backlog that grows until OOM is a correctness
failure) and `C8 Economy` (the broker's memory is bounded by
design, not by luck). This is distinct from
`domains/concurrency/P9 Bounded Queues` and
`domains/concurrency/patterns` Pattern 5 (Bounded Queue with
Backpressure), which own the *in-process* analog: concurrency's
bounded queue fails by OOM or thread crash; messaging's bounded
backpressure fails by network partition, broker restart, or
consumer crash-and-retry (D-062). The Reactive Streams
specification (`request(n)`, `onNext` bounded) is the in-process
instance; messaging's broker-backed backpressure is the
cross-process instance above it. See `domains/messaging/queues.md`
for prefetch and max-unacked and `domains/observability/metrics`
for consumer-lag as an alert.
### P7. Partitioning is Intentional
The partition key determines ordering, parallelism, and hotspots.
Key choice is a design decision with documented rationale, not a
default. A key that hashes unevenly creates a hot partition that
limits throughput; a key that does not match the ordering need
breaks per-key semantics; a key that is too coarse (one partition
for the whole topic) serializes all the traffic. The partition
count is a capacity bound: too few partitions cap parallelism, too
many partition overhead the broker. This derives from `C4 Locality`
(ordering and parallelism are co-located with the partition) and
`C6 Composability` (the partition is the unit of parallelism and
scaling — consumer groups compose from per-partition workers).
This is the cross-process analog of the locality discipline that
`domains/performance/` owns for generic data-near-compute
optimization: performance's locality is algorithmic (data near
compute); messaging's locality is partitional (order and
parallelism near the partition). See `domains/messaging/streams.md`
for the partitioned-log model and consumer-group rebalance
strategies.
### P8. Replay and Retention are Configured
Retention windows and replay-from-offset are explicit. A message
is not ephemeral by default; the broker is a durable log, not a
pipe. A topic with no retention is a fire-and-forget stream — a
consumer that falls behind loses data permanently; a topic with
infinite retention is an unbounded log — the broker grows until
disk exhaustion. Both are defects: the retention window is a
declared bound, and replay-from-offset is the mechanism that makes
the log durable (re-consumable) rather than ephemeral. This derives
from `C5 Reversibility` (a retained message is reversible — it can
be re-consumed; an ephemeral message is not) and `C7 Observability`
(the durable log is itself an observable record of what happened —
the offset is the position from which to replay). This is the
foundation for `domains/messaging/streams.md` and the rule that
distinguishes a stream from a queue (a queue deletes on ack; a
stream retains for replay). See `domains/messaging/pubsub.md` for
the pub/sub-vs-stream durability boundary.
### P9. Schemas Evolve Compatibly
Schema changes are backward- and forward-compatible by
construction. Breaking changes are versioned migrations, not
silent shape edits. A producer that ships a new field the old
consumer ignores is backward-compatible; a consumer that handles a
missing field the new producer omits is forward-compatible. A
silent schema change — the producer renames a field and the
consumer parses `undefined` — is a P1 violation (the contract was
broken) compounded here as an evolution defect. This derives from
`C5 Reversibility` (a schema change is reversible by versioning —
the old shape is still readable) and `C6 Composability` (producers
and consumers of different versions compose because the schema
evolves compatibly). This parallels `domains/data/migrations`
(schema migration for databases) and `domains/api/versioning`
(API contract evolution): messaging's schema evolution is the
async instance of the same compatibility discipline. See
`domains/messaging/streams.md` for the stream-schema-evolution
angle.
### P10. Messaging is Observable
Consumer lag, DLQ depth, throughput, and consumer-group health are
first-class signals. Silent backlog is a bug, not a feature: a
consumer that falls behind with no lag metric is invisible until
the downstream effect surfaces — by which time the backlog may be
hours or days. A DLQ that grows without an alert is a silent
correctness defect: poison messages are accumulating and no one
knows. This derives from `C7 Observability` (the broker's behavior
is visible to the operator) and `C1 Correctness` (backlog
detection is a correctness bound — unbounded lag is a failure).
This is distinct from `domains/observability/metrics`, which owns
*generic* structured metrics; messaging owns the *broker-specific*
signals — lag, DLQ depth, partition imbalance, consumer-group
rebalance events. See `domains/observability/metrics` for the
generic SLI/SLO discipline and `domains/observability/tracing` for
cross-partition traces.
## 2. Core Principle Trace
Each messaging P-rule derives from one or more core C-rules
(C1C8). The matrix extension lands in P4 of the v0.4 plan; the
traces below are authoritative. Messaging is a broad-derivation
domain touching 7 of 8 core principles (C1, C2, C4, C5, C6, C7,
C8); C3 (Simplicity) is not a primary derivation — messaging is
inherently a tradeoff domain where simplicity yields to the
correctness of delivery guarantees (a simpler-than-necessary
delivery model does not handle the failure cases, per C3's
"simpler than necessary is also a violation").
| P-rule | Core | Why |
|--------|------|-----|
| P1 Messages are Contracts | C1, C2 | Correctness of the exchange; clarity of the schema boundary |
| P2 Ordering is a Property, Not an Assumption | C1, C2 | Correctness of order; clarity of the guarantee |
| P3 Consumers are Idempotent | C1 | Correctness under redelivery |
| P4 Delivery Semantics are Explicit | C1, C2 | Correctness of the chosen semantic; clarity of the tradeoff |
| P5 Dead-Letter Handling is Defined | C1, C5 | Correctness of poison-message routing; reversibility of reprocessing |
| P6 Backpressure is Bounded | C1, C8 | Correctness of bounded backlog; economy of broker memory |
| P7 Partitioning is Intentional | C4, C6 | Locality of order; composability of parallelism |
| P8 Replay and Retention are Configured | C5, C7 | Reversibility of replay; observability of the durable log |
| P9 Schemas Evolve Compatibly | C5, C6 | Reversibility of schema changes; composability of versions |
| P10 Messaging is Observable | C7, C1 | Observability of lag/DLQ; correctness of backlog detection |
## 3. What Violates These Principles
| Violation | Principle Breached |
|-----------|-------------------|
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
| "It's FIFO" with no documented partition contract | P2 Ordering is a Property, Not an Assumption |
| Non-idempotent consumer under at-least-once delivery | P3 Consumers are Idempotent |
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
| No dead-letter queue (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
| Unbounded retry budget (no cap; slow consumer stalls the partition) | P6 Backpressure is Bounded |
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
| Ephemeral broker (no retention; no replay) | P8 Replay and Retention are Configured |
| Silent schema change (producer breaks consumers with no version bump) | P9 Schemas Evolve Compatibly |
| Silent backlog (no lag metric; consumer falls behind invisibly) | P10 Messaging is Observable |
| Shared subscription (two consumers share one subscription; per-consumer ordering breaks) | P2 Ordering is a Property, Not an Assumption (P3 compounding) |
| Blocking consumer (slow downstream call with no timeout; broker redelivers to the stuck consumer) | P6 Backpressure is Bounded |
## 4. Relationship to Other Domains
Messaging systems are the engineering discipline of
**cross-process, cross-system asynchronous communication via
brokers**. Producer and consumer are separate systems; the broker
is the intermediary that brokers delivery, ordering, retention,
and failure semantics. The distinguishing constraints are a
cross-process failure model (network, not crash), explicit
delivery semantics, decoupled producer/consumer lifecycle, and
replay-and-retention as a durable-log property. Messaging overlaps
`domains/concurrency/` by *subject* (messages, queues,
backpressure) but not by *failure model*: per D-062, messaging
owns the cross-process/network-failure-model angle; concurrency
owns the in-process/crash-failure-model angle. The discriminator
is the failure model: concurrency's queue fails by OOM or thread
crash; messaging's queue fails by network partition, broker
restart, or consumer crash-and-retry. Messaging extends
concurrency's bounded-queue/backpressure model to the network-
partition regime. Cross-links are one-directional outward (per
D-026 extended); no back-link edits to v0.1/v0.2/v0.3 content.
- `domains/concurrency/patterns` ← P6 (the broker-backed bounded
queue is the cross-process analog of the in-process bounded
buffer — concurrency Pattern 5 owns in-process; messaging owns
the network-failure-model instance above it, per D-062)
- `domains/concurrency/patterns` ← P3 (idempotent retry is the
cross-process analog of in-process retry-safety — the failure
model differs: broker redelivery across a network vs in-process
loop)
- `domains/observability/metrics` ← P10 (consumer lag and DLQ
depth as alerts; observability owns the generic SLI/SLO
discipline, messaging owns the broker-specific signals)
- `domains/observability/tracing` ← P10 (cross-partition traces
for stream processing; observability owns the generic tracing
discipline, messaging owns the cross-partition propagation)
- `domains/data/schema-design` ← P1, P9 (message schema design
and evolution; data owns the generic schema discipline,
messaging owns the cross-process message-shape instance)
- `domains/errors/patterns` ← P5 (errors as data for message
failures; a poison message is an error-as-data instance that must
be observable and recoverable, not swallowed)
- `domains/edge/iot` ← P4 (the edge↔messaging cross-link
resolves bidirectionally here: edge/iot.md links outward to
messaging/queues for MQTT QoS parallels to delivery semantics;
this first-principles doc acknowledges the back-link — the
edge/iot.md → messaging/queues link from P1 now resolves because
messaging/queues.md exists, completing the bidirectionality per
IDEATE-40)
> Note: the edge/iot.md → messaging/queues cross-link (MQTT QoS
> parallels for delivery semantics) was authored in P1 with a
> dangling reference; this P2 authorship of messaging/queues.md
> resolves it. The bidirectionality is verified in P5
> (ATELIER-114 per IDEATE-40). The cross-link is one-directional
> outward from edge/iot.md; this first-principles doc
> acknowledges the resolution without editing edge/iot.md (per
> D-026 extended — no back-link edits to v0.1/v0.2/v0.3 or to
> P1-authored edge content).
+269
View File
@@ -0,0 +1,269 @@
# Pub/Sub — Derived Rules
> Derives from `domains/messaging/first-principles.md`. Applies P1
> (Messages are Contracts), P2 (Ordering is a Property, Not an
> Assumption), P3 (Consumers are Idempotent), and P4 (Delivery
> Semantics are Explicit) primarily, with P7 (partitioning), P10
> (per-subscription lag). The `messaging-shared-subscription` chaos
> anti-pattern lives here (pre-specified in P4 ATELIER-110).
> Cross-links `domains/messaging/streams` for the pub/sub-vs-stream
> durability boundary and `domains/observability/metrics` for
> per-subscription lag.
## What Pub/Sub Is (P1 Messages are Contracts)
- Pub/sub is the fan-out primitive: a producer publishes a message
to a topic; N independent subscriptions each receive a copy. The
message has an explicit, versioned schema (P1): the topic's
schema is the contract every subscription agrees to before
subscribing. A schemaless topic is a defect — every subscriber
breaks silently on the next shape change.
- The boundary with queues is the fan-out ratio. A queue is
point-to-point (one producer, one consumer); pub/sub is
one-to-many (one producer, N consumers, each with its own
subscription). The boundary with streams is the durability model
— see the cross-link below. Pub/sub is an async concern because
producer and consumers are separate systems and the failure model
is network, not crash (D-062).
- See `domains/messaging/queues.md` for the point-to-point variant
and `domains/messaging/streams.md` for the durable-log variant.
## Topic / Subscription Model (P1, P3, P4)
- A **topic** is the named stream of messages. A **subscription**
is a durable cursor over the topic: each subscription receives
every message published after it was created (subject to
retention and filtering). The subscription is independent — its
ack, redelivery, and DLQ are per-subscription, not shared.
- Each subscription is a consumer under at-least-once by default
(P4): the broker redelivers until the subscription acks, and the
subscriber must be idempotent (P3). A subscription with no
idempotency dedup duplicates every redelivered message.
- The topic's schema evolves compatibly (P9 — Schemas Evolve
Compatibly): a new field the old subscriber ignores is
backward-compatible; a renamed field the old subscriber parses
as `undefined` is a P1 violation.
```python
# Publish + two independent subscriptions (P1 contract, P3
# idempotency, P4 at-least-once per subscription). Each
# subscription is an independent durable cursor; acking one does
# not affect the other.
import json, uuid
# --- Publisher ---
def publish(topic, event, broker):
# P1: versioned schema on the topic. All subscribers must
# understand this schema (or a compatible superset — P9).
message = {
"schema": "user.signed-up.v1",
"id": str(uuid.uuid4()),
"idempotencyKey": f"user:{event['userId']}:signup",
"payload": event,
}
broker.publish(topic=topic, body=json.dumps(message))
# --- Subscription A: welcome-email service ---
def subscribe_welcome(broker, dedup_store, send_email):
sub = broker.subscribe(topic="users", subscription="welcome-email")
for message in sub.receive():
# P3: idempotent per subscription. A redelivered message is
# a no-op for THIS subscription, not for the others.
if dedup_store.seen(("welcome", message["idempotencyKey"])):
sub.ack(message)
continue
send_email(message["payload"]["email"], "Welcome!")
dedup_store.mark(("welcome", message["idempotencyKey"]))
sub.ack(message)
# --- Subscription B: analytics-ingest service ---
def subscribe_analytics(broker, dedup_store, ingest):
# Independent subscription: its own cursor, its own dedup,
# its own ack. Welcome-email acking does NOT advance this.
sub = broker.subscribe(topic="users", subscription="analytics")
for message in sub.receive():
if dedup_store.seen(("analytics", message["idempotencyKey"])):
sub.ack(message)
continue
ingest(message["payload"])
dedup_store.mark(("analytics", message["idempotencyKey"]))
sub.ack(message)
```
- The dedup key is scoped per subscription: `(subscription,
idempotencyKey)`. A redelivery to subscription A that was already
processed by A is a no-op for A; the same message delivered to
subscription B is processed by B independently. Scoping the dedup
key by subscription prevents one subscription's dedup from
masking another's redelivery.
## Fan-Out Semantics (P4, P7)
- Fan-out means every subscription receives every published message
(subject to filtering — see below). The broker duplicates the
message per subscription; each subscription's delivery is
independent. The fan-out ratio is the number of subscriptions; the
broker's cost scales with fan-out × message size.
- Partitioning (P7) applies to topics that are partitioned for
throughput: a partitioned topic delivers per-partition order, and
each subscription receives from every partition. A subscription
that consumes partitions in parallel must handle per-partition
ordering and cross-partition non-ordering (P2 — document the
property, do not assume global order).
- The delivery semantic is per-subscription (P4): subscription A
may be at-least-once, subscription B may be at-most-once (for a
loss-tolerant analytics feed). The choice is per subscription,
declared, not emergent.
## Shared vs Independent Subscriptions (P2, P3 — the chaos anti-pattern)
- An **independent subscription** is one durable cursor per
consumer group: each subscription receives every message in
topic order (per partition, P2) and acks independently. This is
the correct default: per-consumer ordering and per-consumer
idempotency hold.
- A **shared subscription** is one subscription shared by multiple
consumers: the broker dispatches each message to an arbitrary
consumer in the shared group. This breaks per-consumer ordering
(P2 — consumer A sees message 3 before consumer B sees message
1) and complicates idempotency (P3 — the dedup state must be
shared across consumers, not per-consumer). This is the
`messaging-shared-subscription` chaos anti-pattern
(pre-specified in P4 ATELIER-110): the primary breach is P2
(ordering); P3 (idempotency) is the compounding consequence.
- A shared subscription is correct ONLY when the consumers are
stateless, the per-message processing is order-independent, and
the dedup store is shared (a shared Redis, a shared DB). A shared
subscription for order-dependent or per-consumer-stateful
processing is the chaos anti-pattern: the broker's arbitrary
dispatch breaks the order the consumer assumes.
```python
# The messaging-shared-subscription chaos anti-pattern (P2
# ordering breach, P3 idempotency compounding). Two consumers
# share one subscription; the broker dispatches each message to
# an arbitrary consumer. Per-consumer ordering breaks; dedup
# must be shared (and often is not).
# BAD — shared subscription, per-consumer dedup (chaos):
def shared_subscription_bad(broker, send_email):
# Both consumers call subscribe with the SAME subscription
# name. The broker round-robins; consumer A gets msg 1, msg 3;
# consumer B gets msg 2, msg 4. Per-consumer order is broken.
# If each consumer has its OWN dedup store, a redelivery to
# the OTHER consumer re-processes (P3 breach).
sub = broker.subscribe(topic="users", subscription="shared")
for message in sub.receive():
# Per-consumer dedup — WRONG. A redelivered message may
# land on the other consumer, which has not seen it.
if local_dedup.seen(message["idempotencyKey"]): # per-consumer
sub.ack(message); continue
send_email(message["payload"]["email"], "Welcome!")
local_dedup.mark(message["idempotencyKey"])
sub.ack(message)
# CORRECT — independent subscriptions (per-consumer ordering,
# per-subscription dedup):
def independent_subscriptions_good(broker, send_email):
sub = broker.subscribe(topic="users", subscription="welcome-email")
for message in sub.receive():
if dedup_store.seen(("welcome", message["idempotencyKey"])):
sub.ack(message); continue
send_email(message["payload"]["email"], "Welcome!")
dedup_store.mark(("welcome", message["idempotencyKey"]))
sub.ack(message)
```
- If a shared subscription is genuinely required (stateless,
order-independent, shared dedup), document the choice and the
shared-dedup requirement (P2 — the ordering property is "none
across consumers"; P3 — the dedup is shared). The default is
independent subscriptions; shared is an opt-in for the narrow case.
## Filtering (P4, C8 Economy)
- **Subscription filtering** lets a subscription receive only
messages matching a filter (e.g., `event.type == "order"`).
Filtering at the broker saves bandwidth (C8 — the subscriber
does not receive and discard) and reduces subscriber load.
- **Server-side filtering** (broker evaluates the filter before
delivery) is more efficient than **client-side filtering**
(subscriber receives and discards). Server-side filtering is the
default where the broker supports it (GCP Pub/Sub, SNS filtering,
NATS subject filtering); client-side is the fallback.
- A filter that is too broad wastes bandwidth; a filter that is
too narrow drops messages the subscriber needed. The filter is
a P1 (contract) and P4 (semantic) decision: the subscription's
filter is part of its declared contract.
## Ordering Across Subscriptions (P2)
- A topic with per-partition ordering delivers per-partition order
to each subscription. Across subscriptions, there is no ordering
guarantee: subscription A may ack message 3 while subscription B
is still on message 1. This is correct and expected — each
subscription is independent.
- Within a subscription, ordering holds per partition (P2 — the
documented property). A subscription that processes partitions
in parallel must not assume cross-partition order. A subscription
that needs global order must use a single partition (sacrificing
parallelism, P7) or an external sequencing mechanism.
- The `messaging-shared-subscription` anti-pattern breaks even
per-partition order within a subscription: the broker's arbitrary
dispatch to consumers in the shared group breaks the per-
partition sequence each consumer sees.
## Pub/Sub vs Stream — The Durability Boundary (cross-link messaging/streams)
- Pub/sub and streams are both fan-out or one-to-many primitives,
but their durability model differs. Pub/sub is a
**push-to-subscription** model: each subscription is a cursor,
retention is short (the subscription's unacked window), and
replay is limited to the unacked messages. A subscription that
falls behind beyond the retention window loses messages
permanently.
- A stream is a **durable-log** model: messages are retained by
the log for a configured window (P8 — Replay and Retention are
Configured), and any consumer group can replay from any offset
within the window. A stream consumer that falls behind can
catch up by replaying; a pub/sub subscription that falls behind
beyond retention cannot.
- The choice is the durability requirement: if the consumer must
be able to replay (reprocessing, backfill, new consumer starting
from the beginning), use a stream. If the consumer only needs
the live feed (and can tolerate loss on a long fall-behind),
pub/sub is lighter. See `domains/messaging/streams.md` for the
durable-log model, offsets, and consumer groups.
## Observability — Per-Subscription Lag (P10)
- Per-subscription lag (messages published minus messages acked
for each subscription, or the age of the oldest unacked message
per subscription) is the primary pub/sub health signal. Each
subscription has its own lag — a fast subscription and a slow
subscription on the same topic are independent signals.
- A subscription whose lag grows beyond the retention window is a
silent data-loss risk: the broker will drop the oldest messages,
and the subscription will never see them. Alert on lag relative
to retention — lag approaching retention is the loss threshold.
- Wire per-subscription lag to `domains/observability/metrics` as
an SLI per subscription. A topic with N subscriptions has N lag
metrics; a single aggregate hides the slow one. See
`domains/observability/metrics` for the generic SLI/SLO
discipline.
## What Violates Pub/Sub Discipline
| Violation | Principle |
|-----------|-----------|
| Shared subscription for order-dependent processing (broker dispatch breaks per-consumer order) | P2 Ordering is a Property, Not an Assumption |
| Shared subscription with per-consumer dedup (redelivery to the other consumer re-processes) | P3 Consumers are Idempotent |
| Schemaless topic (no versioned contract; subscribers parse by guess) | P1 Messages are Contracts |
| Subscription with no idempotency dedup (redelivered message duplicates the effect) | P3 Consumers are Idempotent |
| Subscription whose lag approaches retention (silent data loss) | P10, `domains/observability/metrics` |
| Unstated delivery semantic per subscription (at-least-once vs at-most-once guessed) | P4 Delivery Semantics are Explicit |
| Filter that is too narrow (drops messages the subscriber needed) | P1, P4 |
| Partitioned topic with no documented per-partition ordering contract | P2 Ordering is a Property, Not an Assumption |
| No per-subscription lag metric (slow subscription invisible) | P10 Messaging is Observable |
| Cross-partition order assumption within a subscription (no global order guarantee) | P2, P7 |
+318
View File
@@ -0,0 +1,318 @@
# Queues — Derived Rules
> Derives from `domains/messaging/first-principles.md`. Applies P1
> (Messages are Contracts), P3 (Consumers are Idempotent), P4
> (Delivery Semantics are Explicit), P5 (Dead-Letter Handling is
> Defined), and P6 (Backpressure is Bounded) primarily, with P2
> (ordering), P7 (partitioning), and P10 (observable lag). For the
> at-least-once / at-most-once / exactly-once decision, see the
> comparison table below. Cross-links `domains/concurrency/patterns`
> for the in-process bounded-queue analog and
> `domains/observability/metrics` for consumer lag.
## What a Queue Is (P1 Messages are Contracts)
- A queue is a point-to-point async delivery primitive. A producer
enqueues a message; exactly one consumer dequeues and processes
it. The message has an explicit, versioned schema (P1): the
producer and consumer agree on shape before exchange, and the
schema is the boundary — a schemaless message is a defect (the
consumer breaks silently on the next shape change).
- The boundary is per D-062: messaging owns the cross-process /
network-failure-model angle; concurrency owns the in-process
analog. A queue is a messaging concern because producer and
consumer are separate systems, the broker is the intermediary,
and the failure model is network (the message can be lost,
duplicated, reordered, or delayed by the broker or the network,
not by a thread crash). The in-process bounded buffer
(`domains/concurrency/patterns` Pattern 5) is the analog below
this boundary — it fails by OOM; a broker-backed queue fails by
partition, broker restart, or consumer crash-and-retry.
- See `domains/messaging/pubsub.md` for the fan-out (one-to-many)
variant and `domains/messaging/streams.md` for the durable-log
(replay-from-offset) variant. A queue deletes on ack; a stream
retains for replay — the durability boundary is the
distinguishing trait.
## Producer / Consumer Model (P1, P3)
- The producer enqueues a message with an idempotency key (P3).
The consumer dequeues, processes, and acks. If the consumer
crashes before acking, the broker redelivers; the idempotency
key makes the redelivery safe (the consumer dedups, not the
broker).
- The idempotency key is per-message, not per-producer or
per-session. A consumer that dedups by producer alone will drop
distinct messages issued in the same window. Use a UUID per
message, or a deterministic key derived from the message content
(e.g., `(entity, operation, version)`).
```python
# Producer/consumer pair with idempotency key (P1 contract, P3
# idempotency). The producer tags each message with a versioned
# schema and a unique idempotency key; the consumer dedups by the
# key so a redelivered message is processed once (P3).
# --- Producer ---
import json, uuid
def enqueue(order, broker):
# P1: versioned schema. The message carries its schema version
# so the consumer can route by shape (P9 evolution discipline).
message = {
"schema": "order.created.v1",
"id": str(uuid.uuid4()),
"idempotencyKey": f"order:{order['id']}:{order['version']}",
"payload": order,
}
broker.send(queue="orders", body=json.dumps(message))
# At-least-once by default (P4): the broker acks the send; the
# consumer may see this message more than once under retry.
# --- Consumer ---
def consume(broker, dedup_store, process_order):
for message in broker.receive(queue="orders"):
# P3: idempotent consumer. Dedup by idempotency key before
# processing; a redelivered message is a no-op, not a
# double-apply.
if dedup_store.seen(message["idempotencyKey"]):
broker.ack(message) # already processed; skip
continue
try:
process_order(message["payload"])
dedup_store.mark(message["idempotencyKey"])
broker.ack(message) # success; broker drops it
except Exception:
broker.nack(message) # redeliver (at-least-once, P4)
```
- The dedup store is bounded (P6 — Backpressure is Bounded): a
dedup store that grows without bound is a memory leak. Use a TTL
window longer than the broker's max-redelivery window, or a
bounded LRU. See `domains/messaging/delivery-semantics.md` for
the full idempotency-key dedup-store pattern.
## Ack / Nack (P4 Delivery Semantics are Explicit)
- **Ack** tells the broker the message was processed; the broker
drops it. **Nack** (negative ack) tells the broker the
processing failed; the broker redelivers (at-least-once) or
routes to a DLQ (after the retry budget — P5).
- A consumer that neither acks nor nacks within the visibility
timeout causes the broker to redeliver (the broker assumes the
consumer died). This is the at-least-once default: the broker
prefers duplication to loss.
- The semantic is explicit (P4): at-least-once is the default; the
consumer must be idempotent (P3). At-most-once is fire-and-forget
(no ack; the broker drops on send) — lossy but lowest latency.
Exactly-once is at-least-once plus idempotency, or a
transactional two-phase commit — see the comparison table below.
## Visibility Timeouts and Redelivery (P4, P5)
- The visibility timeout is the window the broker hides a message
after delivery, waiting for the ack. If the consumer does not
ack within the window, the broker makes the message visible
again and redelivers it (to the same consumer or another). This
is the at-least-once mechanism: the broker assumes a
no-ack-in-time consumer is dead.
- The timeout must be longer than the processing time, or the
broker redelivers a message the consumer is still processing —
causing duplicate processing (which P3 idempotency makes safe,
but which wastes resources). A timeout shorter than processing
time is a P6 (backpressure) smell: the consumer is too slow for
the configured timeout.
- Redelivery has a budget (P5): after N redeliveries or a TTL, the
message routes to the DLQ. An unbounded retry budget is the
`messaging-unbounded-retry` chaos anti-pattern: the consumer
never makes progress past the poison message.
## FIFO vs Standard Queues (P2 Ordering is a Property, Not an Assumption)
- A **standard queue** delivers in arrival order per receive-node
but offers no global ordering across shards, no per-message-
group ordering, and may redeliver out of order under retry. It
is the high-throughput default; ordering is *not* guaranteed
(P2: the ordering property is "none" — explicitly documented).
- A **FIFO queue** delivers strict per-message-group order: all
messages with the same group ID are delivered to one consumer
in send order. The cost is throughput (FIFO queues cap at lower
TPS) and latency (the broker must sequence per group). The
ordering property is "per-group strict" — explicitly documented
(P2).
- The choice is a P2 decision (which ordering guarantee) and a C8
decision (throughput cost). A consumer that assumes FIFO on a
standard queue is a P2 violation: the broker does not provide
the guarantee the consumer assumes. Document the property; do
not assume it.
## Prefetch and Concurrency (P6 Backpressure is Bounded)
- **Prefetch** (or max-unacked) bounds how many messages the
broker delivers to one consumer without an ack. A prefetch of 1
is strict stop-and-wait (lowest throughput, tightest backpressure);
a prefetch of N allows the consumer to process N in flight
(higher throughput, more memory). An unbounded prefetch is a P6
violation: the broker floods the consumer's memory.
- **Consumer concurrency** is the number of parallel workers
processing from the queue. More workers increase throughput up to
the downstream's limit; beyond that, the workers stall the
downstream (P6 — the backpressure propagates to the
downstream, not the broker).
- The prefetch × concurrency product is the in-flight cap. Declare
it (P6): an undeclared cap is a defect — the consumer either
underutilizes the broker (prefetch too low) or OOMs under load
(prefetch too high). This is the cross-process analog of
`domains/concurrency/patterns` Pattern 5 (bounded queue with
backpressure): concurrency owns the in-process analog;
messaging owns the broker-backed instance.
## Long Polling (P6, C8 Economy)
- Long polling (or `ReceiveMessage` with a wait-time-seconds)
holds the receive request open until a message arrives or the
wait expires. This reduces empty-receive round trips (C8
economy of the constrained link) and reduces latency-to-first-
message (the message is delivered when it arrives, not on the
next poll cycle).
- Long polling is the default for low-throughput queues: short
polling burns CPU on empty receives; long polling waits for
work. For high-throughput queues, the broker is usually full
enough that long polling adds no latency; for low-throughput
queues, long polling is the difference between 20ms and 20s
latency-to-first-message.
## Redelivery + DLQ Flow (P5 Dead-Letter Handling is Defined)
- A poison message (unparseable, repeatedly failing, or exhausting
the retry budget) routes to the dead-letter queue. The DLQ is
observable (P10 — DLQ depth is an alert) and drainable (an
operator can inspect, replay, or discard with audit).
- The retry budget is bounded (P6): N redeliveries, or a TTL with
exponential backoff. After the budget is exhausted, the message
is moved to the DLQ, not retried forever. An unbounded retry is
the `messaging-unbounded-retry` chaos anti-pattern (P5 breach).
- The DLQ routing rule is a redelivery-count or TTL threshold
plus a target queue. See `domains/messaging/delivery-semantics.md`
for the dead-letter strategy comparison table.
```python
# Redelivery + DLQ flow (P5 dead-letter handling, P6 bounded
# retry budget). The consumer tracks redelivery count; after the
# budget, the message routes to the DLQ. The DLQ is observable
# (P10 — depth is an alert) and drainable.
MAX_REDELIVERIES = 5
DLQ = "orders-dlq"
def consume_with_dlq(broker, dedup_store, process_order):
for message in broker.receive(queue="orders"):
# P3 idempotency: a redelivered, already-processed message
# is acked and skipped (not re-processed, not DLQ'd).
if dedup_store.seen(message["idempotencyKey"]):
broker.ack(message)
continue
try:
process_order(message["payload"])
dedup_store.mark(message["idempotencyKey"])
broker.ack(message)
except Exception as exc:
# P5: bounded retry budget. After MAX_REDELIVERIES,
# route to DLQ — do NOT retry forever.
count = message.get("redeliveryCount", 0) + 1
if count >= MAX_REDELIVERIES:
broker.send(DLQ, body=json.dumps({
"original": message,
"reason": str(exc),
"deadLetteredAt": now_iso(),
"redeliveryCount": count,
}))
broker.ack(message) # remove from the origin queue
# P10: the DLQ depth must alert. A DLQ that grows
# with no alert is a silent correctness defect.
else:
# Nack with backoff: the broker redelivers after a
# delay. The backoff caps the retry rate (P6).
broker.nack(message, delay=exponential_backoff(count))
```
- The `deadLetteredAt` and `reason` fields make the DLQ entry
observable and auditable: an operator inspecting the DLQ sees
why each message was dead-lettered and when. See
`domains/errors/patterns` for the errors-as-data discipline the
DLQ entry follows.
## Delivery Semantics Comparison (D-069)
| Semantic | Guarantee | Latency Cost | Implementation Cost | When It Fits |
|----------|-----------|-------------|---------------------|--------------|
| **At-most-once** | A message is delivered 0 or 1 times; loss is possible, duplication is not | Lowest (no ack; fire-and-forget) | Lowest (no ack, no dedup) | Telemetry where a dropped sample is acceptable; high-throughput metrics; MQTT QoS 0; logs where a lost line is tolerable. Never for billing, orders, or any side-effecting operation. |
| **At-least-once** | A message is delivered 1 or more times; duplication is possible, loss is not | Low (one ack round-trip) | Medium (consumer must be idempotent — P3; dedup store required) | The default for side-effecting operations: orders, payments, commands. The consumer dedups via idempotency keys (P3); the broker guarantees delivery. Fits the vast majority of broker-backed queues (SQS standard, RabbitMQ ack, MQTT QoS 1). |
| **Exactly-once** | A message is delivered exactly 1 time; no loss, no duplication | Highest (two-phase commit or transactional producer+consumer) | Highest (requires transactions, a transactional producer, and a transactional consumer — or at-least-once plus idempotency, which collapses to at-least-once with dedup) | Rare. Kafka transactions (consume-process-produce in one transaction); MQTT QoS 2 (four-step handshake). In practice, "exactly-once" is usually at-least-once plus idempotency (P3) — the broker does not guarantee it; the consumer enforces it. Jepsen analyses verify broker claims. |
- The default for side-effecting operations is **at-least-once with
idempotent consumers** (P3). At-most-once is for loss-tolerant
telemetry. Exactly-once is reserved for the narrow case where
the consume-process-produce loop must be transactional (Kafka
transactions) — and even then, the consumer should be idempotent
as defense-in-depth.
- The latency cost column is the C8 tradeoff: at-most-once is
cheapest, exactly-once is most expensive. The implementation
cost column is the C1/C3 tradeoff: at-most-once is simplest,
exactly-once is most complex (and most fragile — a transactional
consumer that partially fails is a bug source). The "when it
fits" column is the P4 decision: declare the semantic per
channel, do not let it emerge.
- This is the decision matrix required by D-069 for queues; see
`domains/messaging/delivery-semantics.md` for the correctness
properties of each semantic and the dead-letter strategy
comparison.
## Observability — Consumer Lag (P10 Messaging is Observable)
- Consumer lag (messages enqueued minus messages acked, or the
age of the oldest unacked message) is the primary queue health
signal. A lag that grows without bound is a P6 violation (the
consumer is slower than the producer) and a P10 violation if it
is not alerted.
- DLQ depth is the secondary signal: a DLQ that grows is a P5
signal (poison messages are accumulating) and a P10 signal if
not alerted. Wire both to `domains/observability/metrics` as
SLIs with SLOs (e.g., lag < 1000 messages, DLQ depth < 10).
- A queue with no lag metric is operating blind (P10 violation):
the operator cannot see the consumer falling behind until the
downstream effect surfaces.
## Cross-Link to Concurrency (P6, cross-link concurrency/patterns)
- The broker-backed queue is the cross-process analog of the
in-process bounded buffer. `domains/concurrency/patterns`
Pattern 5 (Bounded Queue with Backpressure) owns the in-process
instance ("producer is blocked or signaled" within one program);
messaging owns the broker-backed instance above it (the producer
is the broker's enqueue, the consumer is the broker's dequeue, the
backpressure is the prefetch cap and the lag signal). The
failure model differs: in-process fails by OOM; broker-backed
fails by network partition, broker restart, or consumer
crash-and-retry (D-062).
- The cross-link is one-directional outward (messaging →
concurrency) per D-026 extended: messaging references concurrency
as the in-process foundation; concurrency does not back-link to
messaging.
## What Violates Queue Discipline
| Violation | Principle |
|-----------|-----------|
| Schemaless message (no versioned contract; consumer parses by guess) | P1 Messages are Contracts |
| Non-idempotent consumer under at-least-once delivery (redelivery doubles the effect) | P3 Consumers are Idempotent |
| Unstated delivery semantic (at-least-once vs exactly-once guessed) | P4 Delivery Semantics are Explicit |
| No DLQ (poison message retried forever or silently dropped) | P5 Dead-Letter Handling is Defined |
| Unbounded prefetch (broker floods consumer memory) | P6 Backpressure is Bounded |
| Unbounded retry budget (no cap; consumer never progresses past the poison) | P5, P6 |
| Consumer that assumes FIFO on a standard queue (ordering not guaranteed) | P2 Ordering is a Property, Not an Assumption |
| Visibility timeout shorter than processing time (redeliver while still processing) | P4, P6 |
| DLQ with no depth alert (poison messages accumulate invisibly) | P10, `domains/observability/metrics` |
| No consumer-lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
| Dedup store that grows without bound (memory leak) | P6 Backpressure is Bounded |
| Default prefetch with no rationale (underutilizes or OOMs) | P6, `domains/concurrency/patterns` |
+329
View File
@@ -0,0 +1,329 @@
# Streams — Derived Rules
> Derives from `domains/messaging/first-principles.md`. Applies P8
> (Replay and Retention are Configured) primarily, with P2
> (per-partition ordering), P3 (idempotent consumers), P4 (exactly-
> once via transactions), P7 (partitioning), and P10 (observable
> consumer-group health). For the Kafka / Kinesis / Pulsar /
> NATS JetStream decision, see the stream-platform comparison
> table below. For the consumer-group rebalance strategy choice,
> see the rebalance enumeration below (IDEATE-41). Cross-links
> `domains/messaging/delivery-semantics` for exactly-once via
> transactions, `domains/data/schema-design` for stream schema,
> and `domains/observability/tracing` for cross-partition traces.
## What a Stream Is (P8 Replay and Retention are Configured)
- A stream is a durable-log messaging primitive. Messages are
appended to a partitioned, replicated log; consumers read from an
offset and advance at their own pace. The log is retained for a
configured window (P8) — a stream is a durable log, not a pipe.
A consumer that falls behind can catch up by replaying from an
earlier offset; a consumer that starts fresh can replay from the
beginning (within retention).
- The boundary with queues and pub/sub is the durability model. A
queue deletes on ack; a pub/sub subscription retains only its
unacked window; a stream retains the whole log for the configured
retention. This makes a stream replayable (P8 — the
reversibility mechanism) and observable as a record (P10 — the
log is itself an audit of what happened). See
`domains/messaging/pubsub.md` for the pub/sub-vs-stream
durability boundary discussion.
- The boundary with concurrency is per D-062: messaging owns the
cross-process/network-failure-model angle. A stream is a
messaging concern because the log spans brokers and consumers
across a network, and the failure model is partition, broker
restart, or consumer crash-and-retry — not in-process OOM or
thread crash.
## Partitioned Log Model (P2, P7)
- A stream is partitioned for throughput and parallelism. Each
partition is an ordered, append-only log; messages within a
partition are strictly ordered (P2 — per-partition strict order
is the documented property). Across partitions, there is no
ordering guarantee: partition 0 and partition 1 are independent
logs.
- The partition key (P7 — Partitioning is Intentional) determines
which partition a message lands on. A key that hashes evenly
spreads load; a key that matches the per-entity ordering need
(e.g., `userId` for user events) keeps a user's events on one
partition in order; a key that is too coarse (one partition for
the whole topic) serializes all traffic.
- The partition count is a capacity bound: it caps the parallelism
(one consumer per partition per consumer group) and the
throughput (each partition has a write-throughput limit). Too
few partitions cap parallelism; too many partition overhead the
broker (file handles, replication, rebalance cost). The choice
is documented (P7), not defaulted.
## Offsets (P2, P8)
- An offset is a consumer's position in a partition. The consumer
reads from its last committed offset; acking (committing the
offset) advances it. A consumer that crashes before committing
re-reads from the last committed offset (at-least-once by
default, P4) — the consumer must be idempotent (P3).
- The offset is per-partition (P2): each partition has its own
position, and the consumer commits them independently (or
atomically across partitions in a transaction — see below).
- Replay (P8) is resetting the offset backward: a consumer can
replay from the beginning of retention, from a timestamp, or
from a specific offset. This is the durable-log property that
distinguishes a stream from a queue.
## Consumer Groups (P3, P7, P10)
- A consumer group is a set of consumers sharing the stream's
partitions: each partition is assigned to exactly one consumer
in the group. The group is the unit of parallelism and the unit
of offset tracking. Within a group, each consumer handles its
assigned partitions; across groups, each group independently
reads the whole stream (the pub/sub fan-out property, per
subscription/group).
- A consumer in the group is idempotent (P3): under at-least-once
(the default), a redelivery after a crash-and-retry re-processes
messages. The consumer dedups by idempotency key, or processes
deterministically (e.g., a stateful aggregation that overwrites
with the latest value).
- The consumer group's health is observable (P10): per-partition
lag (offset of the consumer vs the log's head), the group's
consumption rate, and rebalance events are first-class signals.
A group whose lag grows without bound is a P6 (backpressure)
smell and a P10 (observability) violation if not alerted.
```python
# Consumer-group reading from offsets (P2 per-partition order,
# P3 idempotent under at-least-once, P7 partition assignment,
# P8 replay from offset). Each consumer in the group handles its
# assigned partitions; the group commits offsets atomically or
# per-partition.
def consume_stream(stream, group, dedup_store, process_event):
# Assign partitions to this consumer by the group's
# rebalance strategy (see the enumeration below).
for partition in stream.assigned_partitions(group, consumer=ME):
# Read from the last committed offset (P8 — replay by
# resetting this offset).
offset = stream.committed_offset(group, partition)
for message in stream.read(partition, from_offset=offset):
# P3: idempotent under at-least-once. A redelivery
# after a crash-and-retry re-processes; dedup by key.
if dedup_store.seen(message["idempotencyKey"]):
stream.commit(group, partition, message["offset"])
continue
process_event(message["payload"])
dedup_store.mark(message["idempotencyKey"])
# Commit the offset to advance (P8 — the position is
# the replay pointer).
stream.commit(group, partition, message["offset"])
```
- The commit-after-process order gives at-least-once (a crash
before commit re-reads); the commit-before-process order gives
at-most-once (a crash after commit loses the unprocessed
message). The default is at-least-once with idempotent consumers
(P3, P4).
## Consumer-Group Rebalance Strategies (IDEATE-41, ATELIER-100 refinement)
When a consumer joins or leaves the group, the broker must
reassign partitions. The rebalance strategy determines the cost
and the use-case fit. This enumeration parallels the v0.3
IDEATE-30 drift-type enumeration (each strategy with its
stop-the-world cost and use-case fit).
| Strategy | Mechanism | Partition Stop-the-World Cost | Use-Case Fit |
|----------|-----------|-------------------------------|--------------|
| **Eager rebalance** (stop-the-world) | Every consumer in the group revokes ALL its partitions, the broker reassigns the full partition set, then consumers resume. Every rebalance pauses the whole group. | High — every partition pauses for every rebalance; the whole group stops processing during the revocation+reassignment window. Throughput drops to zero during rebalance. | Simple brokers, small groups, or rarely-rebalancing groups where the simplicity of full revocation outweighs the pause cost. Kafka's legacy protocol (pre-2.4). Avoid for large groups or frequent scale events. |
| **Sticky (incremental cooperative) rebalance** | The broker reassigns only the partitions that must move (the joining/leaving consumer's share); existing partitions stay assigned. The rebalance is incremental and cooperative — no full revocation. | Low — only the moving partitions pause; the rest of the group continues processing. The pause is proportional to the changed partition count, not the total. | The default for large groups, frequent scale events, and rolling deploys. Kafka's CooperativeStickyAssignor (2.4+), Pulsar, NATS JetStream. Prefer for any group where a full stop-the-world on every deploy is unacceptable. |
| **Cooperative (no-revoke) rebalance** | A subset of sticky where no partition is revoked unless the consumer leaves; only additions are incremental. The strictest minimization of stop-the-world. | Lowest — only added partitions pause; existing assignments are untouched. | Groups where partition assignment is append-only (consumers join but rarely leave). Useful for long-lived consumers with incremental scaling. |
- The default for any non-trivial group is **sticky/cooperative**:
a rolling deploy that triggers an eager rebalance pauses the
whole group on every pod restart, which is unacceptable at
scale. The eager strategy is a legacy default that survives
because it is simple; prefer sticky where the broker supports
it.
- The "partition stop-the-world cost" column is the P6
(backpressure) angle: a full stop-the-world during rebalance
causes lag to spike (the consumer is paused, the producer is
not). Sticky rebalance bounds the spike to the moving partitions.
- A rebalance that pauses without a lag alert is a P10 violation:
the operator cannot see the rebalance-induced lag. Wire rebalance
events to `domains/observability/metrics` as an event signal.
## Replay and Retention Windows (P8, C5 Reversibility)
- Retention is the configured window the log keeps messages: time-
based (e.g., 7 days), size-based (e.g., 10 GB per partition), or
compacted (keep the latest value per key — a changelog). A
stream with no retention is a pipe, not a log (P8 violation); a
stream with infinite retention grows until disk exhaustion (P6
violation — backpressure on the broker).
- Replay (P8) is resetting a consumer's offset to re-read from
within the retention window. Use cases: reprocessing after a
consumer bug (replay from the timestamp of the buggy deploy),
backfilling a new consumer (replay from the beginning), or
reindexing (replay to rebuild a derived store).
- Compacted topics (Kafka log-compaction, Pulsar compaction) keep
the latest value per key and discard older values for the same
key. This turns the log into a changelog — a durable
materialized view that replays to the current state. Compaction
is a P8 (retention) and C5 (reversibility) mechanism: the log
retains the current state per key and is replayable to it.
## Stream Processing (P2, P3, P4)
- Stream processing is computing over the stream as it arrives:
windowing (tumbling, sliding, session windows), joins (stream-
stream, stream-table), aggregations (count, sum, per-key
windows), and stateful transformations. The processing is
per-partition ordered (P2 — a window over a partition is
deterministic; a window across partitions is not unless the
window is global).
- Stream processing consumers are idempotent (P3): a redelivery
after a crash re-processes a window; the aggregation must
tolerate re-application (e.g., a sum is idempotent under replay
if the window is keyed by offset range, not by wall time).
- Exactly-once stream processing (P4) requires transactions: the
consume-process-produce loop is one transaction — the input
offset commit and the output produce are atomic. See the
transactional exactly-once producer below.
## Exactly-Once via Transactions (P4, P3)
- Exactly-once stream processing is at-least-once plus a
transaction: the consumer commits the input offset and produces
the output in one transactional operation. If the consumer
crashes mid-transaction, neither the offset commit nor the
output produce happens — the consumer re-reads from the last
committed offset and re-processes (at-least-once), but the
transaction ensures the output is produced exactly once.
- This is NOT a broker guarantee of exactly-once delivery; it is
at-least-once delivery plus idempotent/transactional processing
(P3, P4). Jepsen analyses of Kafka transactions confirm the
boundaries: the transaction is atomic within the broker, but
the downstream sink must be transactional or idempotent too.
- See `domains/messaging/delivery-semantics.md` for the full
exactly-once-via-idempotency discussion.
```python
# Transactional exactly-once producer (P4 exactly-once via
# transactions, P3 idempotent produce). The consume-process-
# produce loop is one transaction: the input offset commit and
# the output produce are atomic. A crash mid-transaction rolls
# both back; the consumer re-reads and re-processes.
def consume_transform_produce(stream, group, txn_producer):
# Begin a transaction. All produces and the offset commit in
# this block are atomic (P4).
with txn_producer.transaction() as txn:
for partition in stream.assigned_partitions(group, ME):
offset = stream.committed_offset(group, partition)
for message in stream.read(partition, from_offset=offset):
output = transform(message["payload"])
# P3: idempotent produce. The txn producer
# dedups by an epoch+sequence so a retried
# transaction does not double-produce.
txn.produce(
topic="enriched-events",
key=message["key"],
value=output,
idempotencyKey=message["idempotencyKey"],
)
# Commit the input offset within the same
# transaction (P4 atomicity). A crash before
# txn.commit() rolls this back; the consumer
# re-reads from the prior offset.
txn.commit_offset(group, partition, message["offset"])
# txn.commit() makes the produces and the offset commit
# visible atomically. A crash before this point aborts
# both; a crash after is safe (idempotent produce — P3).
```
- The transactional producer's idempotency (P3) is the defense
against a retried transaction: the broker dedups the output by
the producer's epoch and sequence so a re-commit does not
double-produce. The transaction (P4) is the defense against a
partial failure: the offset and the output commit together.
## Stream Schema (P1, P9, cross-link data/schema-design)
- A stream's messages carry a versioned schema (P1). The schema
evolves compatibly (P9): a new field the old consumer ignores is
backward-compatible; a renamed field the old consumer parses as
`undefined` is a P1 violation compounded as an evolution defect.
- Stream schemas are often registered in a schema registry
(Confluent, Apicurio) that enforces compatibility on produce.
A producer that tries to publish an incompatible schema is
rejected; the registry is the P1/P9 enforcement point.
- See `domains/data/schema-design` for the generic schema-design
discipline (Avro, Protobuf, JSON Schema); messaging owns the
stream-specific instance — the registry, the per-topic
compatibility mode, the consumer-side routing by schema
version.
## Stream-Platform Comparison (D-069)
| Axis | Apache Kafka | AWS Kinesis | Apache Pulsar | NATS JetStream |
|------|--------------|-------------|---------------|----------------|
| **Ordering** | Per-partition strict (P2); global only via single-partition topic | Per-shard strict; global only via single shard | Per-partition strict; global via single partition; also supports shared (out-of-order) subscriptions | Per-stream strict; per-subject ordering; global via single stream |
| **Partitioning model** | Partitions (immutable count post-creation; increase requires recreate); key→partition by hash | Shards (reshardable: split/merge at runtime); key→shard by hash | Partitions (resizable; Pulsar's layered architecture separates compute from storage); key→partition by hash | Streams (subject-based; republish to resize); key→stream by subject |
| **Replay / retention** | Time- or size-based retention; compaction (latest-per-key); replay from offset or timestamp | Time-based retention (24h365d); replay from sequence number or timestamp; no compaction | Time- or size-based; compaction; replay from offset or timestamp; tiered storage (hot/warm/cold) | Time- or size-based; per-stream max-age; replay from sequence; no native compaction |
| **Consumer groups** | Group-coordinated; offsets stored in an internal topic; eager (legacy) or sticky/cooperative (2.4+) rebalance | Enhanced fan-out consumers (per-shard HTTP/2 push); KCL for group coordination; no native group rebalance (shard is the unit) | Group-coordinated; shared or failover subscription modes; cooperative rebalance | Per-stream consumers; durable cursors; no native group rebalance (stream is the unit) |
| **Exactly-once** | Transactions (KIP-98): consume-process-produce atomic; idempotent producer (KIP-516) | No native exactly-once; at-least-once with consumer-side dedup (P3) | Transactions: produce-ack atomic; idempotent producer | At-least-once by default; dedup window per stream (P3 idempotency) |
| **Use-case fit** | High-throughput durable logs, stream processing (Kafka Streams, Flink), event sourcing, multi-consumer replay | AWS-native streaming, log ingestion, simple ETL within AWS; low operational burden | Cloud-native, geo-replication, tiered storage, mixed pub/sub + streaming; multi-tenant | Lightweight, low-latency, edge-friendly; NATS ecosystem; simpler ops than Kafka |
| **Watch out for** | Partition count is fixed at creation (resize requires recreate + republish); rebalance cost on large groups; operational complexity | Shard limits per account; no compaction; retention cap at 365 days; AWS lock-in | Two-arch (BookKeeper + Brokers) operational complexity; smaller ecosystem | Smaller ecosystem; no native compaction; fewer stream-processing libraries |
- The default for high-throughput durable logs with multi-consumer
replay is **Kafka**; for AWS-native streaming, **Kinesis**; for
geo-replicated multi-tenant or mixed pub/sub + streaming,
**Pulsar**; for lightweight low-latency edge-friendly streaming,
**NATS JetStream** (which also cross-links `domains/edge/iot`
via MQTT parallels — see the edge↔messaging bidirectionality in
`domains/messaging/first-principles.md` §4).
- The ordering column is the P2 check: every platform provides
per-partition/per-shard strict order; none provides global order
across partitions except by single-partition. The replay/
retention column is the P8 check: every platform retains for a
configured window; replay is from offset or timestamp. The
exactly-once column is the P4 check: Kafka and Pulsar provide
transactions; Kinesis and JetStream rely on at-least-once plus
consumer-side idempotency (P3).
## Cross-Partition Traces (P10, cross-link observability/tracing)
- A stream-processing pipeline that fans out across partitions
must propagate a trace context per event: the trace ID follows
the event from source to processed output, even as the event
crosses partition boundaries. Without cross-partion traces, a
downstream error cannot be traced back to its source event.
- See `domains/observability/tracing` for the generic distributed-
tracing discipline (trace context propagation, span
correlation). Messaging owns the stream-specific instance: the
trace context is a message header, the span boundary is the
consume-process-produce edge, and the cross-partition
correlation is by trace ID (not by partition — partitions are
independent logs, P2).
- A stream processor with no trace propagation is a P10
violation: the operator cannot trace a processed event back to
its source. Wire the trace context into every produce and every
consume.
## What Violates Stream Discipline
| Violation | Principle |
|-----------|-----------|
| Stream with no retention (pipe, not log; no replay) | P8 Replay and Retention are Configured |
| Infinite retention (grows until disk exhaustion) | P8, P6 |
| Consumer group with eager rebalance at scale (full stop-the-world per deploy) | P6, IDEATE-41 (use sticky/cooperative) |
| Non-idempotent stream consumer under at-least-once (redelivery re-processes the window) | P3 Consumers are Idempotent |
| Default partition key (no rationale; hotspot or wrong-order) | P7 Partitioning is Intentional |
| Partition count too low (caps parallelism) or too high (overhead) | P7 |
| Cross-partition order assumption (no global order guarantee) | P2 Ordering is a Property, Not an Assumption |
| Exactly-once claimed without transactional consume-process-produce (P4 violation) | P4 Delivery Semantics are Explicit |
| Stream schema with no registry / no compatibility enforcement (silent shape break) | P1, P9, `domains/data/schema-design` |
| No per-partition lag metric (consumer falls behind invisibly) | P10, `domains/observability/metrics` |
| No cross-partition trace propagation (downstream error untraceable) | P10, `domains/observability/tracing` |
| Compacted topic treated as a full log (old values already discarded) | P8, C1 (compaction is a retention mode, not a full log) |
+171
View File
@@ -0,0 +1,171 @@
# Go Concurrency — Derived Application
> Applies Atelier's domain principles to Go's concurrency specifically. Go's distinctive strength (goroutines, channels, context) earns a dedicated concurrency doc rather than a `go-async.md`.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/go.md` for the language first-principles stub.
## Goroutines and Structured Concurrency (Concurrency P1 Immutability by Default, C6 Composability)
- **`go f()` spawns a goroutine; ensure it does not outlive its parent:** an unstructured `go f()` leaks when the parent returns. Use `sync.WaitGroup`, `errgroup.Group`, or a `context`-scoped pattern to bound lifetime.
- **`errgroup.WithContext` for structured concurrency:** a `Group` cancels its context on first error; siblings see the cancellation and exit. Mirrors `TaskGroup` semantics cross-language.
- **Goroutines share only immutable inputs:** `go process(snap)` where `snap` is a copy. A goroutine sharing a mutable slice with the parent is a race (Concurrency P1 Immutability, P6 No Silent Races).
- **No `go` in a library function without a documented lifetime:** a library that spawns unbounded goroutines leaks them into the caller. Either accept a `context.Context` or return a `Stop()` method.
```go
import "golang.org/x/sync/errgroup"
func fetchAll(ctx context.Context, ids []string) ([]*User, error) {
g, ctx := errgroup.WithContext(ctx)
results := make([]*User, len(ids))
for i, id := range ids {
i, id := i, id // capture loop vars
g.Go(func() error {
u, err := fetchUser(ctx, id)
if err != nil { return err }
results[i] = u
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
```
## Channels: Bounded Queues and Backpressure (Concurrency P9 Bounded Queues, C6 Composability)
- **Bounded channels apply backpressure:** `make(chan T, N)` blocks the sender when full (Concurrency P9 — bounded queues). Unbounded `make(chan T)` lets the producer run ahead and OOM.
- **`select` with `default` for non-blocking send/receive:** a `default` case makes the channel a queue with try semantics; without it, the operation blocks.
- **Close channel from the sender, never the receiver:** closing a channel signals "no more sends." A receiver closing it is a race; the sender may still be writing.
- **One channel, one responsibility:** do not multiplex control and data on the same channel. Use a `select` over multiple channels instead.
- **Applies `messaging/queues`:** a bounded Go channel is an in-process broker — bounded buffer, backpressure, at-most-once handoff. The same semantics apply; the broker is local.
```go
func pipeline(ctx context.Context, in <-chan Job, out chan<- Result) {
for {
select {
case j, ok := <-in:
if !ok { return }
r := process(j)
select {
case out <- r:
case <-ctx.Done():
return
}
case <-ctx.Done():
return
}
}
}
// bounded: backpressure when out is full
out := make(chan Result, 16)
```
## context.Context for Cancellation (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
- **`context.Context` is the first parameter of every I/O function:** `func fetchUser(ctx context.Context, id string) (*User, error)`. A function that does I/O without a `ctx` cannot be cancelled (Concurrency P7).
- **`context.WithTimeout` for a deadline:** `ctx, cancel := context.WithTimeout(ctx, 5*time.Second); defer cancel()`. Every external call races against a deadline (Concurrency P8).
- **`cancel()` always called, even on success:** `defer cancel()` immediately after creating the context. A leaked context leaks its timer.
- **Never store a `context.Context` in a struct:** pass it as a parameter. A struct holding a `ctx` captures a request-scoped value into a long-lived object.
- **Applies `concurrency/P7`:** cancellation propagates via `ctx.Done()`. A `select` on `<-ctx.Done()` is the cancel-aware wait.
```go
func fetchWithTimeout(ctx context.Context, url string) (*Response, error) {
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
if errors.Is(err, context.DeadlineExceeded) {
return nil, ErrTimeout
}
return nil, err
}
return resp, nil
}
```
## select and Multiplexed Channels (Concurrency P7 Cancellation Support, C6 Composability)
- **`select` multiplexes channel operations:** it picks a ready case at random (fair). A `select` with `<-ctx.Done()` plus a data case is the cancel-aware wait.
- **`default` makes `select` non-blocking:** use for "send if ready, else drop" (a bounded queue with drop-oldest policy).
- **`select {}` blocks forever:** a `select{}` with no cases is a permanent block. Use only in a goroutine that should run until the process exits.
- **Applies `concurrency/P7`:** the `select` over `ctx.Done()` and a result channel is the canonical cancel pattern.
```go
func processUntilCancel(ctx context.Context, jobs <-chan Job) {
for {
select {
case <-ctx.Done():
return
case j, ok := <-jobs:
if !ok { return }
// ...
}
}
}
```
## sync Primitives and Lock Scope (Concurrency P3 Boundaries are Locks, Concurrency P5 Lock Minimization)
- **`sync.Mutex` scoped minimally:** not held across I/O (a `Send` on a channel, an HTTP call). Hold the lock, mutate, release — then do I/O (Concurrency P3 Lock Scope).
- **`sync.RWMutex` for read-heavy, `Mutex` for write-heavy:** RWMutex adds overhead; only prefer it when reads dominate by 10x+.
- **`sync.Map` for specific cases (append-only, disjoint keys):** not a general `map[K]V` replacement. For most maps, `Mutex` + `map` is clearer and often faster.
- **`sync.Once` for one-time init:** `var once sync.Once; once.Do(func(){ init() })`. Idempotent and race-free.
- **Applies `concurrency/P5` (lock minimization):** prefer channels over locks; when a lock is needed, hold it for the smallest possible scope.
```go
type Cache struct {
mu sync.Mutex
items map[string]*User
}
func (c *Cache) Get(id string) (*User, bool) {
c.mu.Lock()
defer c.mu.Unlock()
u, ok := c.items[id]
return u, ok
}
func (c *Cache) Set(id string, u *User) {
c.mu.Lock()
c.items[id] = u
c.mu.Unlock() // explicit unlock before any I/O
}
```
## Race Detection (Concurrency P6 No Silent Races)
- **`go test -race` enforces `P6`:** the race detector instruments memory accesses and fails on data races. See `go-tooling.md` for the CI gate.
- **Tests must exercise the concurrent path:** a serial test of a `Mutex`-protected map finds no race. Write tests with N goroutines hitting the map under `-race`.
- **Applies `concurrency/P6`:** a race detected at test time is a bug fixed; a race undetected is a production heisenbug.
```go
func TestCacheConcurrent(t *testing.T) {
c := &Cache{items: map[string]*User{}}
var wg sync.WaitGroup
for i := 0; i < 100; i++ {
i := i
wg.Add(1)
go func() {
defer wg.Done()
c.Set(strconv.Itoa(i), &User{})
c.Get(strconv.Itoa(i))
}()
}
wg.Wait()
}
```
## Cross-References
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
- `domains/concurrency/first-principles.md` — Concurrency P1, P3, P5, P6, P7, P8, P9 traced throughout.
- `domains/messaging/queues.md` — bounded Go channels as in-process brokers; backpressure parallels (IDEATE-40).
- `domains/errors/patterns.md``errgroup` and error propagation in concurrent code.
- `languages/go-types.md` — typed channels carry the named types defined there.
- `languages/go-tooling.md` — the `-race` CI gate that enforces Concurrency P6.
- `languages/go-testing.md` — concurrent tests that exercise the race detector.
+141
View File
@@ -0,0 +1,141 @@
# Go Testing — Derived Application
> Applies Atelier's domain principles to Go testing specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/go.md` for the language first-principles stub.
## Table-Driven Tests (Testing P1 Tests as Specification, C2 Clarity)
- **Table-driven is the Go idiom:** `cases := []struct{ name string; in X; want Y }{...}`; loop with `t.Run(c.name, ...)`. Each case is a subtest with its own name and failure output.
- **Test names read as a spec:** `{"rejects empty email", ...}`, `{"returns persisted id", ...}`. A reader understands the unit from the subtest names (Testing P1).
- **No `if got != want { t.Fatal() }` shared across cases:** each case asserts independently; a failure in case 3 does not skip cases 4 and 5.
- **`t.Run` enables `-run` filtering:** `go test -run TestCreateUser/rejects_empty_email` runs one case. Essential for debugging a single failure.
```go
func TestCreateUser(t *testing.T) {
cases := []struct {
name string
email string
wantErr bool
}{
{"rejects empty email", "", true},
{"rejects missing @", "no-at-sign", true},
{"accepts valid email", "a@b.co", false},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
_, err := CreateUser(c.email)
if (err != nil) != c.wantErr {
t.Fatalf("err=%v, wantErr=%v", err, c.wantErr)
}
})
}
}
```
## t.Parallel for Independence (Testing P2 Independence, Concurrency P10 Test for Race Conditions)
- **`t.Parallel()` for independent subtests:** each subtest opts in; the runner executes them concurrently. A test that fails under `Parallel` has hidden state (Testing P2 Independence).
- **Capture loop variables:** `c := c` inside the loop, or rely on Go 1.22+ per-iteration scoping. A parallel subtest sharing `c` races on the last value.
- **Applies `concurrency/P10` (test for races):** parallel tests are the first line of race detection; combine with `-race` for the full safety net.
```go
for _, c := range cases {
c := c // capture for parallel
t.Run(c.name, func(t *testing.T) {
t.Parallel()
_, err := CreateUser(c.email)
if (err != nil) != c.wantErr {
t.Fatalf("err=%v, wantErr=%v", err, c.wantErr)
}
})
}
```
## t.Cleanup for Teardown (Testing P3 Determinism, Testing P2 Independence)
- **`t.Cleanup(func() { ... })` for teardown:** runs in LIFO order after the test (and its subtests) complete. Replaces `defer` in a helper that does not know when the test ends.
- **Per-test state, not shared:** a `setup(t)` helper creates resources and registers cleanup; each test gets its own. A package-level `var` shared across tests is order coupling.
- **`t.TempDir()` for filesystem tests:** creates a unique temp dir and cleans up automatically. No manual `os.RemoveAll` and no cross-test contamination.
- **Applies `Testing P3` (determinism):** cleanup is tied to the test lifecycle, not a global teardown that may run before or after depending on order.
```go
func setupStore(t *testing.T) *Store {
t.Parallel()
dir := t.TempDir() // auto-cleaned
s, err := OpenStore(filepath.Join(dir, "db"))
if err != nil { t.Fatal(err) }
t.Cleanup(func() { s.Close() })
return s
}
```
## Race Detector (Testing P9 Edge Case Coverage, Concurrency P6 No Silent Races)
- **`go test -race` in CI, always:** see `go-tooling.md`. The detector is the enforcement of `concurrency/P6`.
- **Tests must exercise the concurrent path:** a serial test of a `Mutex`-protected map finds no race. Write tests with N goroutines.
- **`-count=1` to disable result caching:** by default, Go caches passing tests. `-count=1` forces re-run; combine with `-race` and parallelism to surface heisenbugs.
- **Applies `Testing P9` (edge case coverage):** the race detector is the edge-case tool for concurrency — it finds the inputs the test author forgot to write.
```bash
# CI gate
go test -race -count=1 ./...
```
## Time and Determinism (Testing P3 Determinism, Testing P9 Edge Case Coverage)
- **No `time.Now()` in code under test:** inject a `Clock` interface. In tests, a fake clock advances deterministically.
- **`time.Sleep` in tests is a smell:** a sleep waits for a real timer, flaky under load. Use a channel or `Eventually`-style polling with a timeout.
- **`t.Deadline()` aware helpers:** a helper that may take long checks `t.Deadline()` and bails early. Prevents a slow test from timing out the suite.
```go
type Clock interface { Now() time.Time }
type fakeClock struct{ t time.Time }
func (f *fakeClock) Now() time.Time { return f.t }
func TestUserHasCreatedAt(t *testing.T) {
clk := &fakeClock{time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)}
u, _ := CreateUserWithClock("a@b.co", clk)
if u.CreatedAt.Year() != 2024 {
t.Fatalf("year=%d, want 2024", u.CreatedAt.Year())
}
}
```
## Mocks and Interfaces (Testing P7 Realism, API P1 Contract Fidelity)
- **Mock at the interface, not the struct:** `type Store interface { Get(id string) (*User, error) }` in production; `type mockStore struct{ ... }` in test. The interface is the contract (applies `api/P1`).
- **`httptest` for HTTP servers:** `httptest.NewServer` gives a real server on a loopback port; no manual socket plumbing.
- **`testify/mock` or hand-written mocks:** hand-written for one-off, `testify` for complex sequencing. Avoid mocking frameworks that generate code at runtime (reflection-heavy) — they hide failures behind stack traces.
- **Applies `Testing P7` (realism):** mock the boundary (HTTP, DB), not the unit. Mocking the unit under test tests the mock.
```go
type mockStore struct {
users map[string]*User
got []string
}
func (m *mockStore) Get(id string) (*User, error) {
m.got = append(m.got, id)
return m.users[id], nil
}
func TestGetUserLogs(t *testing.T) {
s := &mockStore{users: map[string]*User{"abc": {}}}
svc := NewService(s)
svc.GetUser("abc")
if len(s.got) != 1 || s.got[0] != "abc" {
t.Fatalf("got=%v", s.got)
}
}
```
## Cross-References
- `domains/testing/pyramid.md` — where unit/integration/race tests sit; the race job is its own layer.
- `domains/testing/fixtures.md``t.TempDir` and `t.Cleanup` as the fixture discipline.
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
- `domains/concurrency/first-principles.md` — Concurrency P6 (race detector), P10 (test for races).
- `languages/go-types.md` — the named types tests assert.
- `languages/go-concurrency.md` — concurrent tests exercise the patterns from that doc.
- `languages/go-tooling.md` — the `go test` flags (`-race`, `-count`, `-run`) detailed here.
+97
View File
@@ -0,0 +1,97 @@
# Go Tooling — Derived Application
> Applies Atelier's domain principles to Go tooling specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/go.md` for the language first-principles stub.
## go vet and golangci-lint (DevOps P2 Automation, C2 Clarity)
- **`go vet` is the stdlib baseline:** it catches `printf` format mismatches, lock-copy-by-value, and unreachable code. Run on every build.
- **`golangci-lint` aggregates vet + dozens of linters:** enable `errcheck` (no `_ = err`), `govet`, `staticcheck`, `ineffassign`, `unused`, `gofmt`, `goimports`. Each enabled linter has a one-line `# reason:` in `.golangci.yml`.
- **`errcheck` enforces `errors/P2` (fail loudly):** a discarded error is a silent failure. `errcheck` fails the build on `_ = doX()`.
- **`goimports` over `gofmt`:** `goimports` adds missing imports and removes unused ones, in addition to formatting. The format is not debated in review (Clarity C2).
```yaml
# .golangci.yml
linters:
enable:
- errcheck # reason: Errors P2 — no swallowed errors
- govet
- staticcheck
- ineffassign
- unused
- gofmt
- goimports
linters-settings:
errcheck:
check-blank: true # fail on _ = fn()
```
## go test -race (Concurrency P6 No Silent Races)
- **`go test -race` in CI, always:** the race detector instruments memory accesses and fails on data races. It is the primary enforcement of `concurrency/P6` (no silent races).
- **`-race` adds overhead; run it in a separate CI job:** the race build is ~2x slower; keep the fast unit-test job and add a race job.
- **`-race` requires tests that actually exercise the concurrent path:** a test that calls `Get`/`Set` serially finds no race. Write tests that spawn goroutines hitting the same map.
- **Applies `concurrency/P6`:** a race detected is a bug fixed; a race undetected is a heisenbug in production. The detector is the safety net.
```bash
# CI race job
go test -race -count=1 ./...
```
## Module Discipline (DevOps P1 Reproducibility)
- **`go mod tidy` on every change that touches imports:** removes unused deps and adds missing ones. A `go.mod` with stale entries breaks reproducibility.
- **`go.sum` committed and verified:** `go mod verify` checks the checksums of the module cache against `go.sum`. A drifted `go.sum` is a supply-chain signal.
- **Pinned major versions in `go.mod`:** `require github.com/x/y v1.2.3` pins the minor; a `v1.2.4` patch may auto-update. For applications, consider a `go.mod` proxy that pins to exact commits.
- **`go mod vendor` for hermetic CI:** vendoring `vendor/` into the repo means CI builds without network. The trade-off is repo size; the win is reproducibility (DevOps P1).
```bash
# CI build gate
go mod tidy
go mod verify
go build ./...
go test -race ./...
```
## Reproducible Builds (DevOps P1 Reproducibility, C3 Simplicity)
- **One Go toolchain version, pinned:** `goenv` or `asdf` pins the Go version per repo; a `.go-version` file declares it. A CI job that uses "latest" Go drifts.
- **`CGO_ENABLED=0` for static binaries:** a static binary runs in a scratch container with no libc dependency. Set in CI for all release builds.
- **`-trimpath` and `-ldflags='-s -w'` for reproducible output:** strips the build path from the binary and removes debug info. Two builds of the same commit produce byte-identical binaries.
```bash
# Reproducible release build
CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o app ./cmd/app
```
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
- **`go doc` from comments:** package comments and exported-symbol comments are the API docs; `go doc` and `pkg.go.dev` render them. Missing comments on exported symbols fail `revive`/`golint` (Documentation P1).
- **`// Example` functions are run by `go test`:** an `ExampleUser` function with `// Output:` is a tested artifact; a stale output fails the build.
- **`README.md` and `docs/` are built by `mkdocs` or similar:** the pipeline validates links and renders; a broken link fails CI (Documentation P1).
```go
// GetUser fetches a user by id.
//
// Example:
//
// u, err := GetUser(id)
// if err != nil { ... }
func GetUser(id UserId) (*User, error) { /* ... */ }
func ExampleGetUser() {
u, err := GetUser("abc")
fmt.Println(u, err)
// Output: <nil> not found
}
```
## Cross-References
- `domains/devops/ci-cd.md` — the pipeline gates that host vet/lint/test.
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
- `domains/concurrency/first-principles.md` — Concurrency P6 No Silent Races (`-race`).
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
- `languages/go-types.md` — the type rules staticcheck enforces reference this doc.
- `languages/go-testing.md` — the `go test` flags (`-race`, `-count`) detailed here.
+140
View File
@@ -0,0 +1,140 @@
# Go Type System — Derived Application
> Applies Atelier's domain principles to Go's type system specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/go.md` for the language first-principles stub.
## Named Types for Domain Concepts (C1 Correctness, Data P7 Type Fidelity)
- **Named types for domain IDs and values:** `type UserId string`, `type OrderId string`. Two named types are distinct even with identical underlying types; the compiler rejects the swap.
- **Constructors validate at the boundary:** `func NewUserId(s string) (UserId, error)` returns an error on bad input. A bare `UserId(s)` cast bypasses validation — only the constructor is exported.
- **Applies `data/P7` (type fidelity):** a named type carries the domain meaning through the call graph; a `string` parameter does not.
- **`any` is the wide type; narrow before use:** Go 1.18+ `any` is an alias for `interface{}`. Use it only at true boundaries (e.g., `json.Unmarshal`); narrow with a type assertion immediately.
```go
type UserId string
type OrderId string
func NewUserId(s string) (UserId, error) {
if !regexp.MustCompile(`^[a-z0-9]+$`).MatchString(s) {
return "", fmt.Errorf("invalid user id: %q", s)
}
return UserId(s), nil
}
func GetUser(id UserId) (*User, error) { /* ... */ }
// GetUser("abc") // compile error: string is not UserId
// GetUser(OrderId("abc")) // compile error: distinct named types
```
## Generics (C6 Composability, Data P7 Type Fidelity)
- **Generics (1.18+) preserve element types across containers:** `type Repository[T any] struct { ... }` keeps `T` through `Get`/`Save`, rather than widening to `any`.
- **Constrain with `comparable` for map keys, custom interfaces for behavior:** `func dedupe[T comparable](s []T) []T` uses `comparable`; a `Sortable[T]` constraint expresses the `Less` requirement.
- **Avoid generics where an interface suffices:** `io.Reader` is not improved by generics. Generics are for type-preserving containers; interfaces are for behavior.
- **No generic methods on generic types (not supported):** `func (r Repository[T]) Map[U any](f func(T) U) Repository[U]` is a compile error. Use a free function.
```go
type Entity interface { ID() string }
type Repository[T Entity] struct {
db map[string]T
}
func (r *Repository[T]) Get(id string) (T, bool) {
var zero T
t, ok := r.db[id]
if !ok { return zero, false }
return t, true
}
func (r *Repository[T]) Save(t T) { r.db[t.ID()] = t }
```
## Interfaces (C6 Composability, API P1 Contract Fidelity)
- **Interfaces defined by the consumer, not the producer:** a package defines its dependencies as interfaces (`type Store interface { Get(id string) (*User, error) }`), and accepts implementations. The producer does not pre-declare "the interface I implement."
- **Small interfaces (Go proverb):** `io.Reader` is one method. An interface with 5+ methods is a god-object; split it.
- **Accept interfaces, return concrete types:** return a `*UserRepo`, accept a `Store`. The caller gets the implementation; the callee depends on the abstraction.
- **Applies `api/P1` (contract fidelity):** the interface is the contract; the concrete type is the implementation. Tests mock the interface, not the struct.
```go
// consumer defines the interface
type UserStore interface {
Get(id string) (*User, error)
}
type Service struct { store UserStore }
func NewService(s UserStore) *Service { return &Service{store: s} }
// producer returns concrete; satisfies UserStore implicitly
type UserRepo struct { db map[string]*User }
func (r *UserRepo) Get(id string) (*User, error) { return r.db[id], nil }
```
## Type Assertion Discipline (C1 Correctness, Errors P1 Errors are Data)
- **Type assertions return `(T, bool)` — use the bool:** `v, ok := x.(UserId)` distinguishes "wrong type" from "zero value." A bare `x.(UserId)` panics on mismatch.
- **`switch x := x.(type)` for multi-variant narrowing:** each case narrows `x` to the case type. The default case is exhaustive (no `never`-style check; Go relies on review).
- **Applies `errors/P1` (errors are data):** a failed type assertion is a value (`ok == false`), not an exception. Handle it as a branch, not a panic.
- **Never assert across module boundaries silently:** an assertion on a type from another package couples to its internals. Prefer an interface method.
```go
func describe(x any) string {
switch v := x.(type) {
case UserId:
return "user " + string(v)
case OrderId:
return "order " + string(v)
default:
return fmt.Sprintf("unknown: %T", v)
}
}
// safe form, never panic
id, ok := raw.(UserId)
if !ok {
return fmt.Errorf("expected UserId, got %T", raw)
}
```
## Error Types and errors.Is/As (Errors P1 Errors are Data, Errors P3 Fail Specifically)
- **Sentinel errors for known cases:** `var ErrNotFound = errors.New("not found")`; check with `errors.Is(err, ErrNotFound)`. The sentinel is a value, not an exception class.
- **Custom error types for context:** `type ValidationError struct { Field, Msg string }`; check with `var ve *ValidationError; errors.As(err, &ve)`. The type carries structured data (Errors P4 Preserve Context).
- **Wrap with `%w`:** `fmt.Errorf("get user %s: %w", id, err)` preserves the chain. `errors.Is`/`As` unwrap it. Bare `%v` breaks the chain.
- **Applies `errors/P3` (fail specifically):** `ErrNotFound` is specific; `ErrFailed` is not. The error type names the failure mode.
```go
var ErrNotFound = errors.New("not found")
type ValidationError struct {
Field string
Msg string
}
func (e *ValidationError) Error() string { return e.Field + ": " + e.Msg }
func GetUser(id UserId) (*User, error) {
u, ok := db[string(id)]
if !ok {
return nil, fmt.Errorf("user %s: %w", id, ErrNotFound)
}
return u, nil
}
// caller
if errors.Is(err, ErrNotFound) { /* 404 */ }
var ve *ValidationError
if errors.As(err, &ve) { /* 422 with ve.Field */ }
```
## Cross-References
- `domains/data/schema-design.md` — named types parallel schema design at the Go boundary.
- `domains/data/first-principles.md` — Data P7 Type Fidelity is the primary trace.
- `domains/api/rest.md` — contract fidelity for HTTP handlers using interfaces.
- `domains/errors/patterns.md``errors.Is`/`As` and the wrap-with-`%w` pattern.
- `languages/go-concurrency.md` — typed channels carry these named types.
- `languages/go-testing.md` — table-driven tests assert type-swap safety.
+7
View File
@@ -2,6 +2,13 @@
> How Atelier's domain principles apply in Go specifically. Derives from `domains/` docs.
## Derived Docs
- [go-types.md](go-types.md) — named types, generics, interfaces, type assertion discipline.
- [go-tooling.md](go-tooling.md) — go vet, golangci-lint, go test -race, module discipline.
- [go-concurrency.md](go-concurrency.md) — goroutines, channels, context, select, sync primitives.
- [go-testing.md](go-testing.md) — table-driven tests, t.Parallel, t.Cleanup, race detector.
## Type System (C1 Correctness, Data P7 Type Fidelity)
- **Named types for domain concepts:** `type UserId string`, not bare `string`.
+117
View File
@@ -0,0 +1,117 @@
# Python Async — Derived Application
> Applies Atelier's domain principles to Python async specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/python.md` for the language first-principles stub.
## asyncio and anyio (Concurrency P7 Cancellation Support, C2 Clarity)
- **`asyncio` for I/O-bound work; threads only for blocking libraries:** `async def` + `await` for network/disk; `run_in_executor` to wrap a blocking call. Mixing threads for I/O is the wrong default.
- **`anyio` for runtime portability:** `anyio` abstracts asyncio/trio; a library written against `anyio` runs on either. Use it for libraries; for applications, asyncio directly is fine.
- **One event loop, one thread:** `asyncio.run(main())` creates and runs the loop. Do not call `asyncio.run` inside an existing loop (raises `RuntimeError`); do not share a loop across threads.
- **Applies `concurrency/P7`:** every `async def` accepts cancellation as a first-class signal; `CancelledError` propagates unless explicitly suppressed (and suppressing it is almost always a bug).
```python
import asyncio
import anyio
async def fetch_user(id: str) -> User:
return await api.get(f'/users/{id}')
# asyncio application
async def main():
user = await fetch_user('abc')
asyncio.run(main())
# anyio library — portable across asyncio/trio
async def fetch_all(ids: list[str]) -> list[User]:
return await anyio.gather(*(fetch_user(i) for i in ids))
```
## Structured Concurrency (Concurrency P1 Immutability by Default, C6 Composability)
- **`asyncio.TaskGroup` (3.11+) for structured concurrency:** tasks created in a `TaskGroup` are awaited or cancelled together on exit. No orphan tasks outlive the block.
- **No `asyncio.gather(..., return_exceptions=False)` for fallible tasks:** `gather` returns partial results on first exception; `TaskGroup` cancels siblings and propagates the error atomically. Use `TaskGroup` for new code.
- **Applies `concurrency/P1` (immutability):** tasks share only immutable inputs; results are collected, not mutated in place. A task that writes to a shared list is a race waiting to happen.
- **`anyio.create_task_group()` mirrors `TaskGroup` cross-runtime:** same structured-concurrency guarantee, portable.
```python
import asyncio
async def fetch_all(ids: list[str]) -> list[User]:
results: list[User] = []
async with asyncio.TaskGroup() as tg:
tasks = [tg.create_task(fetch_user(i)) for i in ids]
# all tasks done (or cancelled) by here
return [t.result() for t in tasks]
```
## Cancellation and Timeout (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
- **`asyncio.wait_for(coro, timeout)` for a deadline:** every external `await` races against a timeout. A bare `await` is an unbounded wait (Concurrency P8).
- **`asyncio.timeout()` (3.11+) as a context manager:** `async with asyncio.timeout(5): await op` — cleaner than `wait_for` for multi-await blocks.
- **`CancelledError` propagates; do not catch broadly:** `except Exception` swallows `CancelledError` in 3.7 (it was `BaseException`); in 3.8+ it's `BaseException` and `except Exception` skips it. Catch specifically, never bare `except:`.
- **Applies `concurrency/P7`:** cancellation is cooperative — a long synchronous block inside `async def` ignores cancellation. Yield with `await asyncio.sleep(0)` periodically in CPU-bound loops.
```python
import asyncio
async def fetch_with_timeout(id: str, timeout: float = 5.0) -> User:
async with asyncio.timeout(timeout):
return await fetch_user(id)
async def shutdown(token: asyncio.Event) -> None:
# cooperative cancel — long-running loop checks the token
while not token.is_set():
await do_chunk()
await asyncio.sleep(0) # yield so cancel can land
```
## Bounded Concurrency and Queues (Concurrency P9 Bounded Queues)
- **`asyncio.Semaphore(N)` to bound in-flight tasks:** a `Semaphore(8)` wrapping `gather` caps concurrency. Unbounded `gather` on a 10k-item list exhausts file descriptors (Concurrency P9 — bounded queues).
- **`asyncio.Queue(maxsize=N)` for producer/consumer:** a bounded queue applies backpressure to the producer. An unbounded queue lets the producer run ahead and OOM.
- **Applies `messaging/queues`:** an `asyncio.Queue` is an in-process broker — the same bounded-queue / backpressure semantics apply; the broker is just local.
```python
import asyncio
async def map_bounded(items: list[str], limit: int = 8) -> list[User]:
sem = asyncio.Semaphore(limit)
async def guarded(i: str) -> User:
async with sem:
return await fetch_user(i)
return await asyncio.gather(*(guarded(i) for i in items))
```
## Error Handling in Async (Errors P5 Recoverable When Possible, Errors P1 Errors are Data)
- **Retry with backoff for transient failures:** network blips are recoverable (Errors P5). Exponential backoff with jitter, capped retries, and an `anyio`-cancellation-aware `sleep`.
- **No retry for non-idempotent operations:** a `POST` creating a resource is not safely retryable without an idempotency key (applies `api/P6` Idempotency).
- **`except asyncio.CancelledError: raise`** is the only valid handling — re-raise so the cancellation propagates. Catching and continuing breaks structured concurrency.
- **Applies `messaging/delivery-semantics`:** a cancelable async operation is at-most-once; retry-on-cancel is at-least-once. The caller must declare which.
```python
import anyio
import random
async def fetch_retry(id: str, attempts: int = 3) -> User:
for i in range(attempts):
try:
return await fetch_user(id)
except (TimeoutError, ConnectionError):
if i == attempts - 1:
raise
await anyio.sleep((2 ** i) * 0.1 + random.random() * 0.1)
raise RuntimeError('unreachable')
```
## Cross-References
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
- `domains/concurrency/first-principles.md` — Concurrency P1 Immutability, P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
- `domains/messaging/queues.md``asyncio.Queue` as an in-process broker; backpressure parallels (IDEATE-40).
- `domains/errors/patterns.md` — typed async errors and retry-with-backoff.
- `languages/py-types.md``Result` and exception hierarchy used in async error handling.
- `languages/py-tooling.md``pytest-asyncio` config that runs these tests.
+111
View File
@@ -0,0 +1,111 @@
# Python Testing — Derived Application
> Applies Atelier's domain principles to Python testing specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/python.md` for the language first-principles stub.
## pytest and Spec-Driven Tests (Testing P1 Tests as Specification, C2 Clarity)
- **`pytest` is the default; `unittest` only for stdlib-only libraries:** `pytest` fixtures, parametrize, and assertion rewriting beat `unittest`'s boilerplate (Clarity C2).
- **Tests co-located with source:** `user.py``test_user.py`. A test far from its subject rots (Documentation P5 Discoverability).
- **Test names read as a spec:** `def test_create_user_rejects_invalid_email():` — a reader understands the unit from the name. Avoid `def test_user1():`.
- **`assert` over `self.assertEqual`:** pytest rewrites `assert` to show the failing values; `assertEqual` is unittest's escape hatch and loses readability.
- **Applies `Testing P1`:** the test is a specification; the failure message is the spec violation.
```python
# test_user.py
import pytest
from user import create_user, ValidationError
def test_create_user_rejects_invalid_email():
with pytest.raises(ValidationError):
create_user(email='not-an-email')
def test_create_user_returns_persisted_id():
u = create_user(email='a@b.co')
assert u.id # truthy persisted id
```
## Factories and Fixture Discipline (Testing P2 Independence, Testing P7 Realism)
- **`factory_boy` or `pytest-factoryboy` over shared fixtures for mutable state:** `UserFactory.build()` returns a fresh object per call; a session-scoped fixture mutated across tests couples them (Testing P2 Independence).
- **Fixtures for setup/teardown, factories for data:** a `db` fixture sets up the DB once per test; a `make_user` factory produces fresh data per assertion. Conflating them produces order-dependent tests.
- **`scope='function'` is the default and the safe default:** `scope='session'` for read-only resources (a schema migration), never for mutable state.
- **Mock at the boundary, not the unit:** `mocker.patch('requests.get')` for HTTP; do not patch `user.User.save` (that mocks the unit under test — Testing P7 realism).
```python
import factory
from user import User
class UserFactory(factory.Factory):
class Meta:
model = User
email = factory.Sequence(lambda n: f'u{n}@b.co')
name = 'Test User'
def test_user_factory_is_fresh():
u1 = UserFactory.build()
u2 = UserFactory.build()
assert u1.email != u2.email # independent
```
## Parametrize and Edge Cases (Testing P9 Edge Case Coverage, Testing P3 Determinism)
- **`@pytest.mark.parametrize` for input tables:** one parametrized test runs N cases; each is an independent test with its own name and failure output (Testing P9).
- **Edge cases as rows, not special tests:** empty list, `None`, max int, unicode — each a row. An ad-hoc `test_handles_edge` with multiple asserts hides which case failed (Testing P6 Failure Specificity).
- **`pytest --randomly` catches order coupling:** a test passing alone but failing in a suite has hidden shared state. The random plugin makes it visible (Testing P2 Independence).
- **Property tests via `hypothesis`:** for invariants (e.g., "parse(serialize(x)) == x"), `hypothesis` generates hundreds of inputs and shrinks failures to a minimal counterexample.
```python
import pytest
@pytest.mark.parametrize('email, reason', [
('', 'empty'),
('a' * 1000 + '@b.co', 'too long'),
('no-at-sign', 'missing @'),
('a@b', 'missing TLD'),
])
def test_create_user_rejects(email, reason):
with pytest.raises(ValidationError):
create_user(email=email)
```
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
- **No `datetime.now()`, `time.time()`, `uuid.uuid4()`, `random.random()` in code under test:** inject a `Clock`, `UUIDGen`, `Random` port. In tests, provide deterministic fakes.
- **`freezegun` for time:** `@freeze_time('2024-01-01')` makes `datetime.now()` deterministic. Do not call `datetime.now()` directly in code — wrap it in a `Clock` port so production and tests both inject.
- **`pytest --randomly-seed=last` to reproduce a failing order:** when `--randomly` finds an order bug, the seed is logged; re-run with it to debug deterministically.
```python
from freezegun import freeze_time
@freeze_time('2024-01-01')
def test_user_has_created_at():
u = create_user(email='a@b.co')
assert u.created_at.year == 2024
```
## Async Tests (Concurrency P10 Test for Race Conditions, Testing P1 Tests as Specification)
- **`pytest-asyncio` (or `anyio`'s pytest plugin) for `async def` tests:** `@pytest.mark.asyncio` runs the coroutine on a loop. Without it, an `async def` test is silently skipped (returns a coroutine, never awaited).
- **`anyio`'s plugin runs the same test on asyncio and trio:** one parametrized run across both runtimes catches runtime-specific bugs.
- **Race-sensitive tests use `--randomly` and bounded concurrency:** a `Semaphore(1)` test under random order surfaces hidden state.
- **Applies `concurrency/P10`:** async tests are the race detector's first line — if a test passes alone but fails under `gather` of N, there's a race.
```python
import pytest
@pytest.mark.asyncio
async def test_async_fetch_returns_user():
u = await fetch_user('abc')
assert u.email
```
## Cross-References
- `domains/testing/pyramid.md` — where unit/integration/property tests sit; hypothesis is the property layer.
- `domains/testing/fixtures.md` — factory-vs-fixture discipline applied via `factory_boy`.
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
- `languages/py-types.md` — the `Result` and Pydantic models that tests assert.
- `languages/py-async.md` — async tests use the cancellation/timeout patterns from that doc.
- `languages/py-tooling.md` — the `pyproject.toml [tool.pytest]` config that runs these tests.
+99
View File
@@ -0,0 +1,99 @@
# Python Tooling — Derived Application
> Applies Atelier's domain principles to Python tooling specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/python.md` for the language first-principles stub.
## ruff for Lint and Format (DevOps P2 Automation, C2 Clarity)
- **`ruff` replaces flake8 + black + isort + pyupgrade:** one tool, one config, one order of magnitude faster. Format is not debated in review (Clarity C2).
- **Rule selection is principled, not "everything":** `select = ["E", "F", "I", "UP", "B", "SIM"]` — each rule group has a one-line `# reason:` in `pyproject.toml`. Rules without a rationale are noise (Documentation P1 — docs are code).
- **`ruff format` is the formatter, `ruff check` is the linter:** run both in CI; the formatter is deterministic, the linter surfaces smells.
- **Applies `devops/P2`:** the format/lint gate runs on every push; a developer never waits for a reviewer to comment on style.
```toml
# pyproject.toml
[tool.ruff]
target-version = "py311"
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM", "RUF"]
# reason: E/F = pyflakes+pycodestyle; I = isort; UP = pyupgrade; B = bugbear; SIM = simplification
[tool.ruff.format]
quote-style = "double"
```
## mypy and Type-Check Gate (DevOps P2 Automation, Data P7 Type Fidelity)
- **`mypy --strict` in CI, not in the editor:** strict flags (`disallow_untyped_defs`, `no_implicit_optional`, `warn_return_any`) are the floor. The editor runs a relaxed mypy for speed; CI runs strict as the gate.
- **`pyright` for stricter/async-aware checking:** pyright understands `async` better and reports faster; mypy is the standard. Pick one as the gate, run the other as informational.
- **Per-module overrides only with a tracked reason:** `[[tool.mypy.overrides]] module = "legacy.*" ignore_errors = true` — each override block links to a ticket. Untracked overrides accumulate into a permanently untyped core.
- **`py.typed` marker for libraries:** ships the type info to consumers. Without it, downstream mypy treats the library as `Any`.
```bash
# CI gate
mypy --strict src/
pyright src/ || true # informational
```
## Dependency Management: poetry and uv (DevOps P1 Reproducibility)
- **`poetry` or `uv` for lockfile discipline:** both produce a deterministic lock (`poetry.lock` / `uv.lock`). `pip install` alone does not — it resolves at install time, producing different trees across machines.
- **`uv` for speed (Rust-based, 10100x faster):** newer tool, same lockfile semantics. Either is acceptable; do not mix within a repo.
- **Lockfile committed for applications:** for libraries, commit the lock for CI reproducibility even though consumers resolve their own tree.
- **`--frozen` install in CI:** `poetry install --no-dev --frozen` fails if the lock is out of sync. Prevents a "works on my machine" drift.
```bash
# CI install — deterministic
uv sync --frozen --no-dev
# or
poetry install --no-dev --frozen
```
## Virtualenv Discipline (DevOps P1 Reproducibility, C3 Simplicity)
- **One virtualenv per project, never the system Python:** `uv venv` or `python -m venv .venv`. System Python drift breaks reproducibility.
- **`uv` creates and pins the Python version:** `uv venv --python 3.12` ensures the same interpreter across machines. A pinned Python is part of the reproducibility contract, not just the lockfile.
- **No `pip install` into the system Python in CI:** use `uv`/`poetry`'s venv. A CI step that mutates system Python makes the next job non-hermetic.
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install -r requirements.txt
```
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
- **`mkdocs` + `mkdocstrings` from docstrings:** API docs are generated from `google`- or `numpy`-style docstrings; the build fails on missing docstrings for public symbols (Documentation P1).
- **`doctest` blocks in docstrings are run by pytest:** a `>>>` example is a tested artifact; a stale example fails the build (Documentation P1, Testing P1).
- **`pyproject.toml` is the single source of tool config:** ruff, mypy, pytest, poetry all read from it. Do not scatter `.flake8`, `setup.cfg`, `mypy.ini`. One config file is one place to look (Clarity C2).
```python
def get_user(id: UUID) -> User:
"""Fetch a user by id.
Args:
id: the user's UUID.
Returns:
The User.
Raises:
NotFoundError: if the user does not exist.
Example:
>>> get_user(UUID('intentional-example-uuid'))
User(...)
"""
...
```
## Cross-References
- `domains/devops/ci-cd.md` — the pipeline gates that host ruff/mypy/poetry.
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
- `languages/py-types.md` — the type rules mypy enforces reference this doc.
- `languages/py-testing.md` — the pytest config (`pyproject.toml [tool.pytest]`) detailed here.
+111
View File
@@ -0,0 +1,111 @@
# Python Type System — Derived Application
> Applies Atelier's domain principles to Python's type system specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/python.md` for the language first-principles stub.
## Type Hints and Gradual Typing (C1 Correctness, Data P7 Type Fidelity)
- **Type hints on every function signature:** `def get_user(id: UUID) -> User | None:`. Hints are annotations, not enforcement, but `mypy`/`pyright` make them a build gate.
- **Gradual typing is opt-in, not opt-out:** start with `--strict` on a package, fix the errors, then expand. A repo-wide `# type: ignore` is a gradual-typing failure.
- **`from __future__ import annotations` for forward refs:** all annotations are strings until resolved, so `class User: ...` referencing `User` works without quotes in 3.10+.
- **`Any` disables the checker; `object` is the wide type:** `Any` allows any operation; `object` requires narrowing. Use `object` for opaque inputs (e.g., `json.loads` return).
- **Applies `data/P7` (type fidelity):** a hint is the contract; the checker verifies it. A missing hint is a missing contract.
- **Pydantic for runtime validation at the boundary:** hints on `BaseModel` fields are validated at construction, catching bad input from network/config before it deepens into the system.
- **`extra='forbid'` by default:** Pydantic allows extra fields silently; forbid them to surface schema drift (e.g., a client sending a typo'd field name).
- **Custom types via `Annotated` with validators:** `Email = Annotated[str, validate_email]` keeps the type readable and the validator attached to the type, not the model.
```python
from pydantic import BaseModel, ConfigDict
from uuid import UUID
class UserCreate(BaseModel):
model_config = ConfigDict(extra='forbid')
email: str
name: str
class User(UserCreate):
id: UUID
```
## Pydantic and Schema Fidelity (C1 Correctness, API P1 Contract Fidelity, Data P7 Type Fidelity)
- **Pydantic models are the API contract:** a FastAPI handler taking `UserCreate` rejects malformed JSON with a 422 before the body runs. This is `api/P1` (contract fidelity) at the type boundary.
- **`ConfigDict(extra='forbid')` rejects unknown fields:** silently accepting extras is a contract leak — the server appears to handle fields it ignores.
- **Validators raise `ValueError`, not `Exception`:** Pydantic converts `ValueError` to a validation error response; a generic `Exception` becomes a 500 and hides the input bug.
- **Applies `api/P1`:** the model is the source of truth; the OpenAPI schema is generated from it, not hand-written. Drift between schema and code is impossible.
```python
from typing import Annotated
from pydantic import BaseModel, Field, StringConstraints
EmailStr = Annotated[str, StringConstraints(pattern=r'^[^@\s]+@[^@\s]+$')]
class Login(BaseModel):
email: EmailStr
password: Annotated[str, Field(min_length=8)]
```
## Errors as Data (Errors P1 Errors are Data, C1 Correctness)
- **`Union[T, Error]` over `Optional[T]` for expected failures:** `Optional[User]` cannot distinguish "not found" from "permission denied". A discriminated `Result` carries the cause.
- **Custom exception hierarchy rooted at `AppError`:** `class NotFoundError(AppError)` etc. — callers can `except AppError` for the broad case, or a specific subclass for handling.
- **`raise` for exceptional paths, `return Result` for expected:** "user not found" is expected (a `Result`); "DB connection lost" is exceptional (a `raise`). Conflating them makes error handling a guess.
- **Applies `errors/P1`:** errors are values, not control-flow magic. A `Result` type encodes this at the type level even where exceptions are the runtime mechanism.
```python
from dataclasses import dataclass
from typing import Generic, TypeVar, Union
T = TypeVar('T')
@dataclass(frozen=True)
class Ok(Generic[T]):
value: T
@dataclass(frozen=True)
class Err:
error: Exception
Result = Union[Ok[T], Err]
def find_user(id: UUID) -> Result[User]:
row = db.get(id)
if row is None:
return Err(NotFoundError(f'user {id}'))
return Ok(User.from_row(row))
```
## Generics and Protocols (C6 Composability, C1 Correctness)
- **`Protocol` for structural typing (PEP 544):** a `Repository` protocol defines `get`/`save` without requiring an inheritance hierarchy; any class matching the shape satisfies it.
- **`TypeVar` with bounds for generic functions:** `T = TypeVar('T', bound=Entity)` lets `serialize(t: T) -> dict` access `t.id`.
- **`Generic[T]` for container types:** a typed `Repository[T]` preserves the element type across `get`/`save`, rather than widening to `Any`.
- **`@overload` for callable overloads:** `def parse(s: str) -> int: ...` vs `def parse(s: bytes) -> int: ...` — the runtime body is one function; the overloads are the type contract.
```python
from typing import Protocol, TypeVar
T = TypeVar('T')
class Repository(Protocol[T]):
def get(self, id: str) -> T | None: ...
def save(self, t: T) -> None: ...
class UserRepo:
def get(self, id: str) -> User | None: ...
def save(self, u: User) -> None: ...
def use_repo(r: Repository[User]) -> None:
u = r.get('abc') # type: User | None
```
## Cross-References
- `domains/data/schema-design.md` — Pydantic models parallel schema design at the TS/JSON boundary.
- `domains/data/first-principles.md` — Data P7 Type Fidelity is the primary trace for this doc.
- `domains/api/rest.md` — contract fidelity for FastAPI handlers consuming Pydantic models.
- `domains/errors/patterns.md` — the `Result` discriminated union as error-as-data encoding.
- `languages/py-async.md` — typed async results built on the `Result` union here.
- `languages/py-tooling.md` — the `mypy`/`pyright` config that enforces these hints.
+7
View File
@@ -2,6 +2,13 @@
> How Atelier's domain principles apply in Python specifically. Derives from `domains/` docs.
## Derived Docs
- [py-types.md](py-types.md) — type hints + Pydantic, mypy/pyright, gradual typing.
- [py-tooling.md](py-tooling.md) — ruff, mypy, poetry, uv, virtualenv discipline.
- [py-async.md](py-async.md) — asyncio, anyio, cancellation, structured concurrency.
- [py-testing.md](py-testing.md) — pytest, factory_boy, fixture discipline, parametrize.
## Type System (C1 Correctness, Data P7 Type Fidelity)
- **Type hints on every function:** `def get_user(id: UUID) -> User | None:`.
+129
View File
@@ -0,0 +1,129 @@
# Rust Async — Derived Application
> Applies Atelier's domain principles to Rust async specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/rust.md` for the language first-principles stub.
## tokio and the Async Runtime (Concurrency P5 Lock Minimization, C6 Composability)
- **`tokio` is the default async runtime:** `#[tokio::main]` for the entry; `tokio::spawn` for a task. The runtime owns the reactor, the I/O driver, and the timer.
- **`tokio::spawn` returns a `JoinHandle` like `std::thread::spawn`:** a dropped `JoinHandle` detaches (the task keeps running); `await` the handle to join. Prefer await to detach.
- **`tokio::task::JoinSet` for structured concurrency:** a set of tasks awaited together; on drop, all remaining tasks are cancelled. Mirrors `errgroup`/`TaskGroup` semantics.
- **`runtime` features are explicit:** `tokio = { version = "1", features = ["full"] }` for a binary; `["rt", "rt-multi-thread", "macros"]` for a library. Pulling `full` into a library bloats downstream.
```rust
#[tokio::main]
async fn main() {
let mut set = tokio::task::JoinSet::new();
for id in ["a", "b", "c"] {
set.spawn(fetch_user(id.to_string()));
}
while let Some(res) = set.join_next().await {
match res {
Ok(Ok(u)) => println!("{}", u.name),
Ok(Err(e)) => eprintln!("err: {e}"),
Err(join_err) => eprintln!("panic: {join_err}"),
}
}
}
```
## Async Traits (Concurrency P7 Cancellation Support, C6 Composability)
- **`async fn` in traits stabilized in Rust 1.75:** `trait Repo { async fn get(&self, id: &str) -> Result<User, Error>; }`. No `async-trait` crate needed for new code on recent toolchains.
- **`Box<dyn Trait>` with async methods needs `dyn`-compatibility:** the returned future is `Pin<Box<dyn Future>>`; the compiler boxes it. For hot paths, use generics (`impl Trait`) over `dyn`.
- **`async-trait` crate for older toolchains:** macro that desugars to a `Pin<Box<dyn Future>>`. Migrate to native `async fn in trait` when the toolchain allows.
- **`Send` bounds on async traits for cross-thread spawn:** `trait Repo: Send { async fn get(&self, id: &str) -> Result<User, Error>; }` — the returned future must be `Send` to spawn on a multi-thread runtime.
```rust
trait UserRepo: Send + Sync {
async fn get(&self, id: &str) -> Result<User, Error>;
}
struct PgRepo { pool: PgPool }
impl UserRepo for PgRepo {
async fn get(&self, id: &str) -> Result<User, Error> {
sqlx::query_as::<_, User>("SELECT * FROM users WHERE id = $1")
.bind(id).fetch_one(&self.pool).await.map_err(Error::from)
}
}
```
## Cancellation (Concurrency P7 Cancellation Support, Concurrency P8 Timeout Discipline)
- **Cancellation is cooperative via dropping the future:** `tokio::select!` drops the unselected branch, cancelling it. A dropped future stops at its next `.await` point.
- **`tokio::time::timeout` for a deadline:** `timeout(Duration::from_secs(5), op).await` returns `Ok(Ok(v))` on success, `Ok(Err(e))` on inner error, `Err(Elapsed)` on timeout. Every external `await` races against a deadline (Concurrency P8).
- **`tokio::select!` for cancel-aware waits:** `select! { res = op => res, _ = cancel => return Err(Cancelled), }`. The unselected branch is dropped, cancelling it.
- **Cancellation is not atomic:** a future dropped mid-`await` may have partial state. `Drop` runs on cancellation; clean up there (e.g., rollback a transaction).
- **Applies `concurrency/P7`:** cancellation is a first-class signal; the runtime propagates it via drop. No `CancelledError` to catch — the future is gone.
```rust
use tokio::time::timeout;
use std::time::Duration;
async fn fetch_with_timeout(url: &str) -> Result<Response, Error> {
match timeout(Duration::from_secs(5), fetch(url)).await {
Ok(Ok(r)) => Ok(r),
Ok(Err(e)) => Err(e.into()),
Err(_elapsed) => Err(Error::Timeout),
}
}
async fn cancellable(op: impl Future<Output=()>, mut cancel: tokio::sync::oneshot::Receiver<()>) {
tokio::select! {
_ = op => {},
_ = &mut cancel => println!("cancelled"),
}
}
```
## Pin and Self-Referential Futures (Concurrency P5 Lock Minimization, C1 Correctness)
- **`async fn` returns a `Future` that is often self-referential:** the generated state machine may hold a borrow into its own stack. Such a future must be `Pin`ned to move safely.
- **`Pin<Box<T>>` to box and pin:** `Box::pin(async { ... })` returns a `Pin<Box<dyn Future>>`. The cost is a heap alloc; the win is `Send`/`dyn`-compatibility.
- **`Pin<&mut T>` for in-place polling:** `Pin::new(&mut fut)` pins a stack future; the borrow checker prevents moving it. Use for stack-allocated futures in `select!`.
- **Do not `unsafe` unpin:** `Pin::get_unchecked_mut` opts out of the pin guarantees. Application code never needs it; library code uses it for `poll` implementations.
```rust
use std::pin::Pin;
async fn boxed() -> Pin<Box<dyn std::future::Future<Output = ()> + Send>> {
Box::pin(async {
// self-referential state machine is safe to move once pinned
})
}
```
## Bounded Channels and Backpressure (Concurrency P9 Bounded Queues)
- **`tokio::sync::mpsc::channel(N)` is bounded:** `send().await` blocks when full (backpressure, Concurrency P9). Unbounded `unbounded_channel()` lets the producer run ahead and OOM.
- **`tokio::sync::mpsc::Sender::try_send` for non-blocking send:** returns `Err(TrySendError::Full(v))` when full; the caller decides to drop, log, or back off. A bounded queue + `try_send` is the backpressure-aware pattern.
- **`tokio::sync::broadcast` for fan-out:** multiple receivers each get a copy; a slow receiver misses (lag). Use for telemetry, not for commands.
- **Applies `messaging/queues`:** a bounded tokio channel is an in-process broker — bounded buffer, backpressure, at-most-once handoff. The same semantics apply; the broker is local.
```rust
use tokio::sync::mpsc;
async fn producer(tx: mpsc::Sender<Job>) {
for j in jobs() {
if tx.send(j).await.is_err() { return; } // receiver dropped
}
}
async fn consumer(rx: mpsc::Receiver<Job>) {
while let Some(j) = rx.recv().await {
process(j).await;
}
}
let (tx, rx) = mpsc::channel::<Job>(16); // bounded: backpressure
```
## Cross-References
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
- `domains/concurrency/first-principles.md` — Concurrency P5 Lock Minimization, P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
- `domains/messaging/delivery-semantics.md` — at-most-once vs at-least-once framing for async retry/cancel (IDEATE-40).
- `languages/rs-ownership.md``Send`/`Sync` bounds on futures build on the ownership model here.
- `languages/rs-tooling.md``tokio` feature flags and the `cargo` build profiles detailed there.
- `languages/rs-testing.md``#[tokio::test]` and async test patterns.
+136
View File
@@ -0,0 +1,136 @@
# Rust Ownership — Derived Application
> Applies Atelier's domain principles to Rust's ownership model specifically. Rust's distinctive strength (Send/Sync, lifetimes, borrowing) earns a dedicated ownership doc rather than an `rs-types.md`.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/rust.md` for the language first-principles stub.
## Ownership and Move Semantics (Concurrency P1 Immutability by Default, C1 Correctness)
- **Ownership is unique:** at any time, exactly one owner holds a value. Assignment passes ownership (`let y = x;``x` is moved, not copied). The compiler rejects use-after-move.
- **`Copy` types (integers, `bool`, `&T`) duplicate on assignment; everything else moves.** A `struct` is `Copy` only if all fields are; opt in via `#[derive(Copy, Clone)]` only for small, cheap-to-copy types.
- **Pass by `&T` for read-only, `&mut T` for mutation:** a borrow does not transfer ownership; the caller retains the value after the callee returns.
- **Applies `concurrency/P1` (immutability by default):** `&T` is shared and immutable; `&mut T` is exclusive and mutable. The compiler enforces "one or many, never both" — aliasing XOR mutation, statically.
```rust
let s = String::from("hello");
let t = s; // s moved into t
// println!("{}", s); // error: use of moved value
let n = 5;
let m = n; // i32 is Copy: n still usable
println!("{} {}", n, m);
```
## Borrowing and Lifetimes (C1 Correctness, Data P7 Type Fidelity, Concurrency P3 Boundaries are Locks)
- **`&'a T` ties a borrow to a lifetime `'a`:** the borrow cannot outlive the owner. Lifetimes are static — the compiler rejects dangling references.
- **Lifetime elision when unambiguous:** `fn first<'a>(s: &'a str) -> &'a str` is elided to `fn first(s: &str) -> &str` (one input → output lifetime). When ambiguous, name the lifetime.
- **`'static` is the longest lifetime (the whole program):** not "until I drop it." Use `'static` only for values that genuinely live forever (string literals, `const`s); leaking to `'static` to satisfy the checker is a bug.
- **`Ref<'a, T>` and `RefMut<'a, T>` from `RefCell` are runtime-checked borrows:** the borrow rules still apply, checked at runtime instead of compile time. A second `RefMut` panics.
- **Applies `concurrency/P3` (boundaries are locks):** `&mut T` is the compile-time lock — exclusive access is the boundary; no runtime mutex needed for single-threaded aliasing discipline.
```rust
fn longest<'a>(a: &'a str, b: &'a str) -> &'a str {
if a.len() > b.len() { a } else { b } // borrow tied to both inputs
}
fn dangling() -> &str { // compile error: missing lifetime
let s = String::from("local");
&s // error: s drops at end of fn
}
```
## Send and Sync (Concurrency P1 Immutability by Default, Concurrency P3 Boundaries are Locks, C1 Correctness)
- **`Send`:** a type `T: Send` may be moved across thread boundaries. Most types are `Send`; `Rc<T>` is not (shared non-atomically refcounted).
- **`Sync`:** a type `T: Sync` may be shared (`&T`) across threads. `RefCell<T>` is `!Sync` (interior mutability without atomics); `Mutex<T>` is `Sync` (it synchronizes).
- **The compiler enforces `Send`/`Sync` at the thread-spawn boundary:** `std::thread::spawn(move || { ... })` requires the closure's captures to be `Send`.
- **Applies `concurrency/P1` and `concurrency/P3`:** `Send` is the move-across-boundary contract; `Sync` is the share-across-boundary contract. Data races are a compile error, not a runtime detector. This is Rust's distinctive strength over Go's race detector.
```rust
use std::rc::Rc;
use std::sync::Arc;
let rc = Rc::new(5);
// std::thread::spawn(move || { println!("{}", rc) }); // error: Rc is !Send
let arc = Arc::new(5);
std::thread::spawn(move || { println!("{}", arc) }); // ok: Arc<T> is Send+Sync
```
## Shared Mutation: Arc, Mutex, RwLock (Concurrency P3 Boundaries are Locks, Concurrency P5 Lock Minimization)
- **`Arc<T>` for shared ownership across threads:** atomic refcounted. Clone increases the count; the last drop frees `T`.
- **`Mutex<T>` for exclusive mutation across threads:** `lock()` blocks until exclusive; the guard `MutexGuard<T>` derefs to `&mut T` and releases on drop.
- **`RwLock<T>` for read-heavy, `Mutex<T>` for write-heavy:** RwLock allows multiple readers or one writer. For most cases, `Mutex` is simpler and faster; prefer it unless reads dominate by 10x+.
- **Hold the lock for the smallest scope:** `let g = m.lock().unwrap();` then drop `g` before I/O. RAII releases on scope exit; explicit `drop(g)` clarifies intent.
- **Applies `concurrency/P5` (lock minimization):** prefer message passing (`mpsc` channels) over locks. When a lock is needed, scope it minimally.
```rust
use std::sync::{Arc, Mutex};
use std::thread;
let counter = Arc::new(Mutex::new(0));
let mut handles = vec![];
for _ in 0..10 {
let c = Arc::clone(&counter);
handles.push(thread::spawn(move || {
let mut g = c.lock().unwrap();
*g += 1;
// g drops here, lock released
}));
}
for h in handles { h.join().unwrap(); }
println!("{}", *counter.lock().unwrap());
```
## Interior Mutability (Concurrency P1 Immutability by Default, C1 Correctness)
- **`Cell<T>` for `Copy` types, `RefCell<T>` for non-`Copy`:** interior mutability moves the borrow check from compile time to runtime. `RefCell::borrow_mut()` panics on a second mutable borrow.
- **`Mutex<T>`/`RwLock<T>` for thread-safe interior mutability:** the runtime check is the lock, not a panic. Use these across threads; `RefCell` only single-threaded.
- **`UnsafeCell<T>` is the primitive; never use directly:** `Cell`, `RefCell`, `Mutex` are safe wrappers. Direct `UnsafeCell` is `unsafe` and opts out of the aliasing guarantee.
- **Applies `concurrency/P1`:** interior mutability is the exception, not the default. Reach for it when an API must present `&self` while mutating internally (e.g., a cache); document why.
```rust
use std::cell::RefCell;
struct Cache {
inner: RefCell<HashMap<String, User>>,
}
impl Cache {
fn get(&self, id: &str) -> Option<User> {
// &self (immutable) but mutates internally
self.inner.borrow_mut().entry(id.to_string()).or_insert_with(|| fetch()).clone()
}
}
```
## Drop and RAII (C1 Correctness, Concurrency P3 Boundaries are Locks)
- **`Drop` runs when the owner goes out of scope:** no `defer`, no `finally`. A `MutexGuard` releases, a `File` closes, a `JoinHandle`... does not join (a dropped `JoinHandle` detaches).
- **`Drop` is deterministic:** it runs at scope exit, not GC time. This is why `Arc`'s refcount is precise and `Mutex` release is timely.
- **`ManuallyDrop<T>` to opt out:** for FFI types whose destructor you must call manually. Rare in application code; common in `unsafe` bindings.
- **`Drop` order: fields in declaration order, then the struct itself.** A field that another field's `Drop` depends on must be declared last.
```rust
struct Resource { name: String }
impl Drop for Resource {
fn drop(&mut self) {
println!("dropping {}", self.name); // runs at scope end
}
}
fn main() {
let _r = Resource { name: "x".into() };
// _r drops here, prints "dropping x"
}
```
## Cross-References
- `domains/concurrency/first-principles.md` — Concurrency P1 Immutability, P3 Boundaries are Locks, P5 Lock Minimization.
- `domains/data/first-principles.md` — Data P7 Type Fidelity (lifetimes are the type-level fidelity for references).
- `domains/concurrency/patterns.md` — message-passing vs lock patterns applied via `Arc`/`Mutex`/`mpsc`.
- `domains/errors/patterns.md``?` propagation relies on ownership transfer of the error.
- `languages/rs-async.md` — async borrows (`Pin`/`&mut`) build on the lifetime model here.
- `languages/rs-testing.md``Send`/`Sync` tests and ownership-based property tests.
+159
View File
@@ -0,0 +1,159 @@
# Rust Testing — Derived Application
> Applies Atelier's domain principles to Rust testing specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/rust.md` for the language first-principles stub.
## #[test] and Co-located Tests (Testing P1 Tests as Specification, C2 Clarity)
- **`#[test]` on functions in a `#[cfg(test)] mod tests` block:** tests co-located with source, compiled only in `cargo test`. A test file far from its subject rots (Documentation P5 Discoverability).
- **Test names read as a spec:** `fn create_user_rejects_invalid_email()` — a reader understands the unit from the name. Avoid `fn test_user_1()`.
- **`assert!` / `assert_eq!` / `assert_ne!` over raw `panic!`:** the macros produce readable failure output (`assertion failed: left == right, left: 5, right: 3`). Raw `panic!` gives a message only (Testing P6 Failure Specificity).
- **Applies `Testing P1`:** the test is a specification; the failure message is the spec violation.
```rust
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn create_user_rejects_invalid_email() {
let r = create_user("not-an-email");
assert!(matches!(r, Err(Error::Validation(_))));
}
#[test]
fn create_user_returns_persisted_id() {
let u = create_user("a@b.co").unwrap();
assert!(!u.id.is_empty());
}
}
```
## proptest and Property Tests (Testing P9 Edge Case Coverage, Testing P1 Tests as Specification)
- **`proptest` (or `quickcheck`) for invariant tests:** declare a property (`parse(serialize(x)) == x`), the framework generates hundreds of inputs and shrinks failures to a minimal counterexample (Testing P9).
- **Strategy over hand-written generators:** `proptest::collection::vec(any::<u32>(), 0..100)` generates arbitrary `Vec<u32>`; do not hand-roll a generator for each property.
- **`proptest!` macro or `proptest! { ... }` block:** each `case (name) => { ... }` is a property. The block is the spec (Testing P1).
- **Property tests complement, not replace, example tests:** examples document the happy path; properties cover the edge space. Both are required.
```rust
use proptest::prelude::*;
proptest! {
#[test]
fn roundtrips_id(s in "[a-z0-9]{1,32}") {
let id = UserId::new(&s).unwrap();
assert_eq!(id.as_str(), s);
}
#[test]
fn rejects_invalid_id(s in "[^a-z0-9]+") {
assert!(UserId::new(&s).is_err());
}
}
```
## Mock Discipline (Testing P2 Independence, Testing P7 Realism)
- **Mock at the trait, not the struct:** `trait Store { fn get(&self, id: &str) -> Result<User, Error>; }` in production; `#[automock] trait Store` (via `mockall`) in test. The trait is the contract.
- **`mockall` for generated mocks:** `#[automock] trait Repo {}` generates `MockRepo` with `expect_*` methods. Each expectation is per-test; no shared mock state (Testing P2 Independence).
- **Mock the boundary, not the unit:** mock `Repo`, not `UserService` (the unit). Mocking the unit under test tests the mock (Testing P7 realism).
- **No `#[cfg(test)]` on production code paths to inject mocks:** instead, accept the trait as a generic or `dyn` parameter. Test-only branches in production code are dead code in prod.
```rust
use mockall::*;
#[automock]
trait UserRepo {
fn get(&self, id: &str) -> Result<User, Error>;
}
#[test]
fn get_user_returns_not_found() {
let mut repo = MockUserRepo::new();
repo.expect_get()
.with(eq("abc"))
.returning(|_| Err(Error::NotFound));
let svc = UserService::new(Box::new(repo));
assert!(matches!(svc.get_user("abc"), Err(Error::NotFound)));
}
```
## Async Tests (Concurrency P10 Test for Race Conditions, Testing P1 Tests as Specification)
- **`#[tokio::test]` for `async fn` tests:** runs the coroutine on a tokio runtime. Without it, an `async fn` test returns a future, never awaited (silently passes).
- **`#[tokio::test(flavor = "multi_thread")]` for concurrency-sensitive tests:** multi-thread runtime surfaces races that single-thread misses (Concurrency P10).
- **`tokio::time::pause()` and `advance()` for time:** freeze and advance the runtime clock deterministically. No `tokio::time::sleep(real)` in tests.
- **Race-sensitive tests use `loom` for model-checking:** `loom` simulates all thread interleavings; it catches races `-race`-style detectors miss. Use for lock-free data structures.
```rust
#[tokio::test]
async fn async_fetch_returns_user() {
let u = fetch_user("abc").await.unwrap();
assert!(!u.name.is_empty());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn concurrent_cache_is_safe() {
let c = Arc::new(Cache::new());
let mut h = vec![];
for i in 0..10 {
let c = c.clone();
h.push(tokio::spawn(async move { c.get(&i.to_string()).await; }));
}
for x in h { x.await.unwrap(); }
}
```
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
- **No `SystemTime::now()` or `Instant::now()` in code under test:** inject a `Clock` trait. In tests, a fake clock advances deterministically.
- **`tokio::time::pause()` for async time:** freezes the runtime clock; `tokio::time::advance(dur)` moves it. A `sleep(5s)` in test resolves instantly.
- **`--test-threads=1` to reproduce order coupling:** by default, `cargo test` runs tests in parallel; a test that passes alone but fails in a suite has hidden shared state. `-1` reproduces.
```rust
trait Clock { fn now(&self) -> std::time::Instant; }
struct FakeClock(std::time::Instant);
impl Clock for FakeClock {
fn now(&self) -> std::time::Instant { self.0 }
}
#[test]
fn user_has_created_at() {
let clk = FakeClock(std::time::Instant::now());
let u = create_user_with_clock("a@b.co", &clk).unwrap();
assert_eq!(u.created_at, clk.now());
}
```
## Doc Tests (Documentation P1 Documentation is Code, Testing P1 Tests as Specification)
- **`cargo test --doc` runs `///` fenced blocks:** a `///` example with `#`-hidden setup is a tested artifact; a stale output fails the build (Documentation P1).
- **`no_run` for examples that should compile but not run:** ```` ```rust,no_run ```` — type-checks the example without executing. Use for examples that need a DB.
- **`ignore` for examples that should not compile-check:** ```` ```rust,ignore ```` — skips entirely. Rare; prefer `no_run`.
- **Applies `Testing P1`:** the doc example is the spec; the doc test is the spec's regression test.
```rust
/// Fetch a user by id.
///
/// # Example
///
/// ```
/// # use mycrate::{get_user, Error};
/// let u = get_user("abc").unwrap();
/// assert!(!u.name.is_empty());
/// ```
pub fn get_user(id: &str) -> Result<User, Error> { /* ... */ }
```
## Cross-References
- `domains/testing/pyramid.md` — where unit/property/doc tests sit; proptest is the property layer.
- `domains/testing/fixtures.md``t.Cleanup`-equivalent (`Drop` in tests) as fixture discipline.
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
- `domains/concurrency/first-principles.md` — Concurrency P10 (test for races), `loom` model-checking.
- `languages/rs-ownership.md``Send`/`Sync` tests and ownership-based property tests.
- `languages/rs-async.md``#[tokio::test]` patterns from that doc.
- `languages/rs-tooling.md``cargo test` flags (`--doc`, `--test-threads`) detailed here.
+95
View File
@@ -0,0 +1,95 @@
# Rust Tooling — Derived Application
> Applies Atelier's domain principles to Rust tooling specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/rust.md` for the language first-principles stub.
## cargo and Build Discipline (DevOps P2 Automation, DevOps P1 Reproducibility)
- **`cargo build` for dev, `cargo build --release` for release:** release enables optimizations (LTO, codegen-units=1). The default profile is for fast iteration, not perf.
- **`Cargo.lock` committed for applications and CI:** for libraries, commit the lock for CI reproducibility even though consumers resolve their own tree. A drifted lock breaks reproducibility (DevOps P1).
- **`cargo update` periodically, with a CI check:** `cargo update` bumps patch versions in the lock; a CI job that fails on lock drift catches a forgotten `cargo update`.
- **`cargo vendor` for hermetic CI:** vendors `vendor/` into the repo; CI builds without network. The trade-off is repo size; the win is reproducibility.
```toml
# Cargo.toml — profile discipline
[profile.release]
lto = true
codegen-units = 1
panic = "abort" # smaller binary, no unwinding
```
## clippy (DevOps P2 Automation, C2 Clarity)
- **`cargo clippy` is the lint layer over `rustc`:** it catches `clone()` where a borrow would do, `unwrap()` in library code, and needless `Box`. Run on every build.
- **`cargo clippy -- -D warnings` in CI:** warnings are errors. A clippy warning is a smell; accumulating them erodes the signal (Clarity C2).
- **Per-lint allow only with a tracked reason:** `#[allow(clippy::needless_collect)] // reason: GH-123 — collect needed for len` — each allow links to a ticket. Untracked allows accumulate into a permanently lint-bypassed core.
- **`cargo clippy --fix` for safe auto-fixes:** applies the linter's suggested change. Review the diff; do not run blindly on a large commit.
```bash
# CI gate
cargo clippy --all-targets --all-features -- -D warnings
```
## cargo fmt (DevOps P2 Automation, C2 Clarity)
- **`cargo fmt` is the formatter; format is not debated in review:** run in CI as a check (`cargo fmt --check`), not a fix. A failing check blocks the PR.
- **`rustfmt.toml` for repo-wide settings:** if the defaults are wrong for the repo, override once and stop. Do not relitigate per-PR.
- **Applies `devops/P2`:** the format gate is automated; a reviewer never comments on style.
```bash
# CI gate — fail if unformatted
cargo fmt --check
```
## Edition Discipline (DevOps P1 Reproducibility, C5 Reversibility)
- **`edition` in `Cargo.toml` pins the language edition:** 2015, 2018, 2021, 2024. An edition is a coherent set of language changes; bumping it is a deliberate migration.
- **Edition is not the compiler version:** `rustc 1.75` supports edition 2021; edition 2024 needs a newer `rustc`. Pin the toolchain with `rust-toolchain.toml`.
- **Bump editions deliberately, not opportunistically:** `cargo fix --edition` applies the migration lint; review the diff. A bump mid-feature conflates two changes.
- **Applies `devops/P1` and `C5` (reversibility):** pinning the edition and toolchain makes the build reproducible; bumping is a controlled, reversible change.
```toml
# Cargo.toml
[package]
edition = "2021"
rust-version = "1.75"
```
```toml
# rust-toolchain.toml
[toolchain]
channel = "1.75"
components = ["clippy", "rustfmt"]
```
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
- **`cargo doc` from doc comments:** `///` on items generates API docs; `cargo doc --open` previews. The build fails on broken intra-doc links (`#![warn(rustdoc::broken_intra_doc_links)]`).
- **Doc tests are run by `cargo test`:** a `///` fenced block with `#`-hidden setup is a tested artifact; a stale example fails `cargo test --doc` (Documentation P1).
- **`#![warn(missing_docs)]` for libraries:** public items without doc comments fail the build. Documentation is a build gate, not an afterthought.
- **`cargo readme` or `cargo docs-rs` for landing pages:** the crate's `README.md` is rendered on docs.rs; keep it in sync with `lib.rs`'s top-level doc.
```rust
#![warn(missing_docs, rustdoc::broken_intra_doc_links)]
/// Fetch a user by id.
///
/// # Example
///
/// ```
/// # use mycrate::get_user;
/// let u = get_user("abc").unwrap();
/// println!("{}", u.name);
/// ```
pub fn get_user(id: &str) -> Result<User, Error> { /* ... */ }
```
## Cross-References
- `domains/devops/ci-cd.md` — the pipeline gates that host clippy/fmt/test.
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
- `languages/rs-ownership.md``Send`/`Sync` clippy lints reference this doc.
- `languages/rs-async.md` — async-runtime tooling (`tokio` features) detailed here.
- `languages/rs-testing.md``cargo test` flags (`--doc`, `--no-run`) detailed here.
+7
View File
@@ -2,6 +2,13 @@
> How Atelier's domain principles apply in Rust specifically. Derives from `domains/` docs.
## Derived Docs
- [rs-ownership.md](rs-ownership.md) — Send/Sync, lifetimes, borrowing, ownership transfer.
- [rs-tooling.md](rs-tooling.md) — cargo, clippy, fmt, edition discipline.
- [rs-async.md](rs-async.md) — tokio, async traits, cancellation, pin.
- [rs-testing.md](rs-testing.md) — #[test], proptest, property testing, mock discipline.
## Type System (C1 Correctness, Data P7 Type Fidelity)
- **Newtypes for domain concepts:** `struct UserId(String);` — zero-cost, type-safe.
+115
View File
@@ -0,0 +1,115 @@
# TypeScript Async — Derived Application
> Applies Atelier's domain principles to TypeScript async specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/typescript.md` for the language first-principles stub.
## Promises and AbortSignal (Concurrency P7 Cancellation Support, C1 Correctness)
- **Every async function accepts an optional `AbortSignal`:** cancellation is a first-class parameter, not a side channel. The signal propagates to `fetch`, `setTimeout`, and downstream awaits.
- **`AbortController` is the producer side; `AbortSignal` is the consumer side:** a function takes a `signal` (read-only), the caller owns the `controller` and decides when to abort.
- **Abort propagates as a rejected `Promise`:** `fetch` rejects with `AbortError`; downstream code sees the rejection, not a silent no-op. This preserves `errors/P5` (recoverable when possible) — the caller can distinguish cancellation from a real failure.
- **Applies `concurrency/P7`:** no async operation runs without a path to cancel it. A long-running `await` with no signal is a hung request.
- **Never swallow `AbortError`:** re-throw or handle distinctly; cancellation is the caller's intent, not an error to log.
```typescript
async function fetchUser(id: UserId, signal?: AbortSignal): Promise<User> {
const ctrl = new AbortController();
signal?.addEventListener('abort', () => ctrl.abort());
const res = await fetch(`/users/${id}`, { signal: ctrl.signal });
if (!res.ok) throw new HttpError(res.status);
return res.json() as Promise<User>;
}
// caller controls cancellation
const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), 5000);
try {
const u = await fetchUser(id, ctrl.signal);
} finally {
clearTimeout(timer);
}
```
## async/await Discipline (Concurrency P8 Timeout Discipline, C2 Clarity)
- **`await` is the only async primitive in application code:** no `.then` chains, no callback pyramids. `async`/`await` reads top-to-bottom (Clarity C2).
- **Never `await` in a hot loop without batching:** sequential `await` in a `for` loop is O(n) latency. Use `Promise.all` for parallelism; `for await...of` only for genuine streams.
- **`Promise.race` for a timeout:** every external `await` has a deadline. `Promise.race([op, timeout])` rejects when the deadline passes.
- **`return` vs `return await`:** inside `try`/`finally`, `return await` runs the `finally`; bare `return` of a Promise defers the `finally` to the microtask. Prefer `return await` when cleanup must run.
- **Applies `concurrency/P8`:** a bare `await` with no timeout is an unbounded wait. External calls (network, disk) always race against a deadline.
```typescript
async function fetchWithTimeout(url: string, ms = 5000, signal?: AbortSignal): Promise<Response> {
const ctrl = new AbortController();
signal?.addEventListener('abort', () => ctrl.abort());
const timer = new Promise<never>((_, reject) =>
setTimeout(() => reject(new TimeoutError(ms)), ms)
);
try {
return await Promise.race([fetch(url, { signal: ctrl.signal }), timer]);
} finally {
clearTimeout(timer); // cleanup runs on success and on race-loss
}
}
```
## Error Handling in Async (Errors P5 Recoverable When Possible, Errors P1 Errors are Data)
- **Catch `unknown`, narrow with a type guard:** `catch (e: unknown)` — TS does not infer the error type. `instanceof` or a discriminator narrows it.
- **Retry with backoff for transient failures:** network blips are recoverable (Errors P5). Exponential backoff with jitter, capped retry count, and an `AbortSignal`-aware `setTimeout`.
- **No retry for non-idempotent operations:** a `POST` that creates a resource is not safely retryable without an idempotency key (applies `api/P6` Idempotency).
- **Typed errors over `Error` subclasses:** a discriminated union `AppError = Network | Timeout | Cancelled` carries context (Errors P4 Preserve Context) without `instanceof` chains.
```typescript
async function fetchRetry(url: string, attempts = 3, signal?: AbortSignal): Promise<Response> {
for (let i = 0; i < attempts; i++) {
try {
return await fetchWithTimeout(url, 5000, signal);
} catch (e: unknown) {
if (e instanceof AbortError) throw e; // do not retry cancellation
if (e instanceof TimeoutError && i < attempts - 1) {
await sleep(jitter(i), signal); // backoff before retry
continue;
}
throw e;
}
}
throw new Error('unreachable');
}
```
## Cancellation Propagation (Concurrency P7 Cancellation Support, Concurrency P9 Bounded Queues)
- **One signal, many consumers:** pass the same `AbortSignal` to every async call in a request. Aborting once cancels the whole tree.
- **Bounded concurrency with a semaphore:** a `Semaphore(N)` wrapping `Promise.all` caps in-flight requests (Concurrency P9 — bounded queues). Unbounded `Promise.all` on a 10k-item array exhausts file descriptors.
- **Cancellation is cooperative, not preemptive:** a long synchronous block inside an `async` function ignores the signal. Yield with `await Promise.resolve()` periodically in CPU-bound loops, or move to a worker.
- **Applies `messaging/delivery-semantics`:** a cancelable async operation is an at-most-once delivery — the caller may stop listening, the result may or may not arrive. Retry-on-cancel is at-least-once; the caller must declare which.
```typescript
async function mapBounded<T, U>(items: readonly T[], fn: (t: T, s: AbortSignal) => Promise<U>, limit = 8, signal?: AbortSignal): Promise<U[]> {
const ctrl = new AbortController();
signal?.addEventListener('abort', () => ctrl.abort());
const results: U[] = new Array(items.length);
let next = 0;
const workers = Array.from({ length: limit }, async () => {
while (true) {
const i = next++;
if (i >= items.length) break;
if (ctrl.signal.aborted) throw new AbortError();
results[i] = await fn(items[i], ctrl.signal);
}
});
await Promise.all(workers);
return results;
}
```
## Cross-References
- `domains/concurrency/patterns.md` — the cancellation/timeout/semaphore patterns applied here.
- `domains/concurrency/first-principles.md` — Concurrency P7 Cancellation Support, P8 Timeout Discipline, P9 Bounded Queues.
- `domains/messaging/delivery-semantics.md` — at-most-once vs at-least-once framing for async retry/cancel (IDEATE-40).
- `domains/errors/patterns.md` — typed async errors and retry-with-backoff.
- `languages/ts-types.md``Result<T, E>` and discriminated `AppError` used in async error handling.
- `languages/ts-tooling.md``no-floating-promises` lint rule that enforces these awaits.
+117
View File
@@ -0,0 +1,117 @@
# TypeScript Testing — Derived Application
> Applies Atelier's domain principles to TypeScript testing specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/typescript.md` for the language first-principles stub.
## Vitest and Jest (Testing P1 Tests as Specification, C2 Clarity)
- **Vitest for new TS projects; Jest for legacy:** Vitest shares `vite`'s transform pipeline (no separate `ts-jest` config); Jest's ecosystem is broader. Either is acceptable — pick one per repo, do not mix.
- **Tests co-located with source:** `user.ts``user.test.ts`. A test file far from its subject rots (Documentation P5 Discoverability).
- **`describe`/`it` mirror the public API:** the test block names read as a specification ("User", "rejects an invalid email", "returns the persisted id"). A reader should understand the unit from test names alone (Testing P1).
- **`expect` over `assert`:** Vitest/Jest matchers produce readable failure output (`expect(x).toBe(y)` → "expected 5, received 3"). Raw `assert` gives a stack trace and nothing else (Testing P6 Failure Specificity).
```typescript
// user.test.ts
import { describe, it, expect } from 'vitest';
import { createUser } from './user';
describe('createUser', () => {
it('rejects an invalid email', async () => {
await expect(createUser({ email: 'not-an-email' })).rejects.toThrow(ValidationError);
});
it('returns the persisted id', async () => {
const u = await createUser({ email: 'a@b.co' });
expect(u.id).toMatch(/^[a-z0-9]+$/);
});
});
```
## Mock Discipline (Testing P2 Independence, Testing P7 Realism)
- **Mock at the boundary, not the unit:** replace `fetch` or the DB client, not the function under test. Mocking the unit under test tests the mock, not the code (Testing P7 — realism).
- **No partial mocks of the system under test:** if a method must be stubbed, the unit is too large. Extract a collaborator and mock that.
- **Each test sets up and tears down its own state:** no shared mutable fixtures. A `beforeEach`/`afterEach` resets; a top-level `let` shared across tests is order-coupling (Testing P2 Independence).
- **`vi.useFakeTimers()` for time-dependent code:** never call `Date.now()` directly in code under test; inject a `Clock` port. In tests, fake timers make `setTimeout` synchronous.
```typescript
import { vi, beforeEach, afterEach } from 'vitest';
beforeEach(() => {
vi.useFakeTimers();
global.fetch = vi.fn(); // boundary mock
});
afterEach(() => {
vi.useRealTimers();
vi.restoreAllMocks();
});
```
## Type-Level Tests (Testing P1 Tests as Specification, Data P7 Type Fidelity)
- **Type-level tests assert the type system, not runtime behavior:** `expectTypeOf<T>().toMatchTypeOf<U>` and `tsd`/`expect-type` fail the build when a type assertion is wrong.
- **Negative type tests are required:** `// @ts-expect-error` proves the compiler rejects what it should. A `@ts-expect-error` that no longer errors is itself an error (the comment must be consumed).
- **Branded types and utility types get type tests:** a `UserId` should not be assignable to `string`; a `Readonly<T>` should not allow assignment. These invariants are part of the spec (Testing P1).
- **Applies `data/P7` (type fidelity):** a type-level test is a regression test for the type checker — if a refactor silently widens a type, the test fails.
```typescript
import { expectTypeOf } from 'expect-type';
import type { User, UserPatch, UserId } from './user';
test('UserPatch omits id and makes fields optional', () => {
expectTypeOf<UserPatch>().toMatchTypeOf<{ name?: string; email?: string }>();
expectTypeOf<UserPatch>().not.toHaveProperty('id');
});
test('UserId is not assignable to bare string', () => {
// @ts-expect-error — brand prevents widening
const s: string = {} as UserId;
expect(s).toBeDefined();
});
```
## Parametrize and Factories (Testing P3 Determinism, Testing P9 Edge Case Coverage)
- **`it.each` / `test.each` for parametrized cases:** one table drives many runs; each row is an independent test with its own name and failure output.
- **Factories over fixtures:** `makeUser(overrides)` returns a fresh object per call. A shared `const user = {...}` across tests couples them and breaks determinism when one test mutates it (Testing P3).
- **Edge cases as rows, not special tests:** empty array, single element, max int, null, undefined — each a row in a `test.each` table. An ad-hoc `it('handles edge')` with multiple asserts hides which case failed (Testing P9 — edge case coverage, P6 failure specificity).
- **Property-style tests via `fast-check`:** for invariants (e.g., "parse(serialize(x)) === x"), `fast-check` generates hundreds of inputs and shrinks failures to a minimal counterexample.
```typescript
import { test, expect } from 'vitest';
import { makeUser } from './user.factory';
test.each([
{ input: '', reason: 'empty' },
{ input: 'a'.repeat(1000), reason: 'too long' },
{ input: 'not-an-email', reason: 'no @' },
])('rejects email: $reason', async ({ input }) => {
await expect(makeUser({ email: input })).rejects.toThrow(ValidationError);
});
```
## Determinism and Time (Testing P3 Determinism, Testing P9 Edge Case Coverage)
- **No `Date.now()`, `Math.random()`, or `crypto.randomUUID()` in code under test:** inject a `Clock`, `Random`, and `IdGen` port. In tests, provide deterministic fakes.
- **`--random` test order (Vitest `sequence.shuffle: true` default) catches order coupling:** a test that passes alone but fails in a suite has hidden state. The shuffle makes that state visible (Testing P2).
- **Race-detector parallelism for async tests:** run async tests concurrently by default; a test that assumes serial execution breaks under parallelism. Vitest's `concurrent` flag surfaces the bug.
```typescript
import { vi, test, expect } from 'vitest';
test.concurrent('parallel fetch does not interleave state', async () => {
const store = new Store();
await Promise.all([store.put('a', 1), store.put('b', 2)]);
expect(store.get('a')).toBe(1);
expect(store.get('b')).toBe(2);
});
```
## Cross-References
- `domains/testing/pyramid.md` — where unit/type/integration tests sit; the type-level tests here are the base layer.
- `domains/testing/fixtures.md` — factory-vs-fixture discipline applied via `makeUser`.
- `domains/testing/first-principles.md` — Testing P1 Specification, P2 Independence, P3 Determinism, P9 Edge Coverage.
- `languages/ts-types.md` — the branded types and utility types that type-level tests assert.
- `languages/ts-async.md` — async tests use the cancellation/timeout patterns from that doc.
- `languages/ts-tooling.md``ts-jest`/`vitest` config and the `expect-type`/`tsd` toolchain.
+114
View File
@@ -0,0 +1,114 @@
# TypeScript Tooling — Derived Application
> Applies Atelier's domain principles to TypeScript tooling specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/typescript.md` for the language first-principles stub.
## tsc and tsconfig Discipline (DevOps P2 Automation, DevOps P1 Reproducibility)
- **`strict: true` is the floor, not the ceiling:** it enables `strictNullChecks`, `noImplicitAny`, `strictFunctionTypes`, and more. Disable sub-flags only with a justification comment.
- **`tsc --noEmit` in CI:** type-checking is a build gate; emission is the bundler's job. Separate the two so a type error fails CI even when the bundler would have succeeded.
- **`tsconfig` is per-project, not inherited verbatim:** a shared base (`extends`) encodes org defaults; each project overrides the deltas it needs. Avoids the "one monoreto-config-fits-all" trap.
- **`noUncheckedIndexedAccess` for safety:** `arr[i]` becomes `T | undefined`, forcing narrowing. Costs little, prevents a class of out-of-bounds deref bugs.
- **Applies `devops/P1` (reproducibility):** pinned `typescript` version in `package.json` and `lockfile` ensure every CI run type-checks against the same compiler.
```jsonc
// tsconfig.json — base
{
"compilerOptions": {
"strict": true,
"noUncheckedIndexedAccess": true,
"exactOptionalPropertyTypes": true,
"noEmit": true,
"moduleResolution": "bundler",
"isolatedModules": true
}
}
```
## ESLint and @typescript-eslint (DevOps P2 Automation, Documentation P9 Living Documents)
- **ESLint with `@typescript-eslint` strict ruleset:** `recommended-type-checked` enables rules that require the type checker (`no-floating-promises`, `no-misused-promises`).
- **Rules encode decisions, not taste:** every custom rule in the config has a one-line `// reason:` comment linking to the principle it enforces. This makes the config a living document (Documentation P9).
- **Format is Prettier's job; ESLint lints:** `eslint-config-prettier` disables conflicting format rules. Do not relitigate formatting in code review.
- **`no-floating-promises` enforces `concurrency/P8` (timeout discipline):** an un-awaited `Promise` is a fire-and-forget that swallows errors and timeouts. The rule forces `.catch()` or `await`.
```jsonc
// .eslintrc.json
{
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/recommended-type-checked",
"prettier"
],
"parserOptions": { "project": "./tsconfig.json" },
"rules": {
// reason: enforce Concurrency P8 — no un-awaited promises
"@typescript-eslint/no-floating-promises": "error",
// reason: enforce Data P7 — no `any` escaping the type checker
"@typescript-eslint/no-explicit-any": "error"
}
}
```
## Project References and ts-jest (DevOps P2 Automation, C6 Composability)
- **Project references for monorepos:** `composite: true` + `references` let `tsc --build` incrementally type-check only changed projects, and enforce the dependency graph at the type level.
- **`paths` aliases mirror the import structure:** `@app/*``src/*`. Configure once in `tsconfig`, mirror in the bundler and the test runner so all three agree.
- **`ts-jest` (or `vitest`) with `isolatedModules: true`:** each test file is type-checked in isolation, matching how the bundler transpiles. Catches the "passes in `tsc` but fails in the bundler" gap.
- **Applies `devops/P2`:** the build pipeline (tsc → lint → test → bundle) is automated; a developer never runs a manual sequence.
```jsonc
// tsconfig.references.json
{
"files": [],
"references": [
{ "path": "./packages/core" },
{ "path": "./packages/api" },
{ "path": "./packages/web" }
]
}
```
## Lockfile and Reproducible Install (DevOps P1 Reproducibility)
- **`npm ci` in CI, not `npm install`:** `ci` reads the lockfile exactly and fails on drift. `install` mutates the lockfile.
- **Lockfile committed for applications:** for libraries, commit `package-lock.json` for CI reproducibility even though consumers resolve their own tree.
- **No floating ranges in `package.json`:** `^` and `~` are CI's job to resolve; pin the resolved version in the lockfile. An unpinned `*` is a supply-chain attack surface.
```bash
# CI install step — deterministic
npm ci
# Type-check gate
npx tsc --noEmit
# Lint gate
npx eslint .
```
## Documentation in the Pipeline (Documentation P1 Documentation is Code, DevOps P9 Documentation in the Pipeline)
- **Type-checked JSDoc:** `typedoc` (or `TypeDoc`) generates API docs from `tsdoc` comments. The compiler enforces that `@param` names match real parameters.
- **`@example` blocks are compiled:** a `tsdoc` `@example` fenced block is type-checked as part of the doc build. Stale examples fail the pipeline (Documentation P1 — docs are code).
- **README badges reflect CI status:** the build/lint/test/type-check gates are the source of truth; badges surface them. Do not hand-edit status tables.
```typescript
/**
* Fetch a user by ID.
*
* @param id - a branded UserId (see ts-types.md).
* @throws {NotFoundError} if the user does not exist.
* @example
* ```ts
* const u = await getUser(userId('abc'));
* ```
*/
async function getUser(id: UserId): Promise<User> { /* ... */ }
```
## Cross-References
- `domains/devops/ci-cd.md` — the pipeline gates that host tsc/ESLint/ts-jest.
- `domains/devops/first-principles.md` — DevOps P1 Reproducibility, P2 Automation.
- `domains/documentation/first-principles.md` — Documentation P1 Documentation is Code.
- `languages/ts-types.md` — the type rules ESLint enforces reference this doc.
- `languages/ts-testing.md` — the test-runner config (`ts-jest`/`vitest`) detailed here.
+111
View File
@@ -0,0 +1,111 @@
# TypeScript Type System — Derived Application
> Applies Atelier's domain principles to TypeScript's type system specifically.
> Derives from `domains/` docs; introduces no new P-rules (D-063).
> See `languages/typescript.md` for the language first-principles stub.
## Nominal vs Structural Typing (C1 Correctness, Data P7 Type Fidelity, API P1 Contract Fidelity)
- **TypeScript is structurally typed:** two types with the same shape are assignable. This is convenient but erases domain boundaries — a `UserId` and `PostId` both `string` are interchangeable.
- **Branded (nominal) types for domain IDs:** intersect with a phantom brand to simulate nominal typing. The brand is never constructed at runtime; it exists only to the type checker.
- **Applies `data/P7` (type fidelity)** at the value boundary: a branded `UserId` cannot be passed where a `PostId` is expected, preventing an entire class of swap bugs.
- **Applies `api/P1` (contract fidelity):** branded types make API contracts explicit — handlers cannot accept "any string" for an ID.
- **Brand is opaque to consumers:** do not export the brand symbol; construction goes through a validated factory.
```typescript
type UserId = string & { readonly __brand: 'UserId' };
type PostId = string & { readonly __brand: 'PostId' };
function userId(s: string): UserId {
if (!/^[a-zA-Z0-9]+$/.test(s)) throw new Error('invalid id');
return s as UserId;
}
function getUser(id: UserId): User { /* ... */ }
getUser('abc'); // type error
getUser(userId('abc')); // ok
getUser(postId('xyz')); // type error — distinct brands
```
## Generics (C6 Composability, Data P7 Type Fidelity)
- **Generics preserve type information across boundaries:** a `Repository<T>` keeps the element type through `find`/`save` rather than widening to `any`.
- **Constrain with `extends`:** `<T extends Entity>` documents the contract and gives the body access to `T.id`.
- **Avoid unnecessary generics:** if a function accepts "any value and returns it unchanged," `T` is noise. Prefer `unknown` for truly opaque inputs.
- **Variance is structural:** TS does not enforce sound variance; mark mutation points with `readonly` to keep `T[]` assignable to `readonly T[]`.
```typescript
interface Entity { id: string }
class Repository<T extends Entity> {
constructor(private db: Map<string, T>) {}
find(id: string): T | undefined { return this.db.get(id); }
save(t: T): void { this.db.set(t.id, t); }
}
```
## Narrowing and Type Guards (C1 Correctness, Errors P1 Errors are Data)
- **Narrowing is how TS handles `unknown` and union types safely:** `typeof`, `in`, `instanceof`, and discriminators collapse a wide type to a precise one before use.
- **User-defined type guards (`x is T`) encode domain predicates:** `isUser(x): x is User` lets the checker track the narrow across call sites.
- **Applies `errors/P1` (errors are data):** a `Result<T, E>` discriminated union is narrowed with `if (r.ok)` — no `try`/`catch` needed for expected failures.
- **Never use `as` to widen past a check:** `as` lies to the compiler. If narrowing does not reach the type you need, the predicate is wrong, not the cast.
```typescript
type Result<T, E> = { ok: true; value: T } | { ok: false; error: E };
function unwrap<T, E>(r: Result<T, E>): T {
if (r.ok) return r.value; // narrowed to { ok: true; value: T }
throw new Error(JSON.stringify(r.error));
}
function isUser(x: unknown): x is User {
return typeof x === 'object' && x !== null && 'id' in x && 'name' in x;
}
```
## Utility Types (C5 Reversibility, C6 Composability)
- **`Partial<T>`, `Pick<T,K>`, `Omit<T,K>`, `Readonly<T>` are derived views:** they derive from a source-of-truth `T` rather than redeclaring fields, so the source change propagates (reversibility).
- **`Readonly<T>` enforces immutability at the type level** — applies `concurrency/P1` (immutability by default) without runtime cost.
- **`Record<K, V>` over `{ [k: string]: V }`:** the index signature form allows any string key including prototype pollution vectors; `Record` is exact.
- **Compose, don't accumulate:** `type Patch<T> = Partial<Omit<T, 'id'>>` reads as a transformation; restate it if `T` changes shape, rather than maintaining a parallel `Patch` type.
```typescript
interface User { id: string; name: string; email: string; }
type UserPatch = Partial<Omit<User, 'id'>>;
type ReadonlyUser = Readonly<User>;
type UsersById = Record<string, User>;
```
## Discriminated Unions (C1 Correctness, Data P7 Type Fidelity, Errors P1 Errors are Data)
- **Discriminated unions over enums:** `type Status = { type: 'pending' } | { type: 'paid'; amount: number }` is exhaustive and carries payload per variant; an `enum` carries neither.
- **The discriminant is a literal `type` (or `kind`) field:** the checker narrows on it in `switch` and `if` without a custom guard.
- **Exhaustiveness via `never`:** assign the narrowed value to `never` in the default branch; if a variant is added, the default fails to compile.
- **Applies `errors/P1`:** model domain errors as a discriminated union `AppError = NotFound | Validation | Conflict`, not as exception classes — the type system carries the error set.
```typescript
type Status =
| { type: 'pending' }
| { type: 'paid'; amount: number }
| { type: 'refunded'; reason: string };
function describe(s: Status): string {
switch (s.type) {
case 'pending': return 'awaiting payment';
case 'paid': return `paid ${s.amount}`;
case 'refunded': return `refunded: ${s.reason}`;
default:
const _exhaustive: never = s; // compile error if a variant is added
throw new Error('unhandled');
}
}
```
## Cross-References
- `domains/data/schema-design.md` — schema-level fidelity parallels branded types at the TS boundary.
- `domains/data/first-principles.md` — Data P7 Type Fidelity, the primary trace for this doc.
- `domains/api/rest.md` — contract fidelity for API handlers consuming branded IDs.
- `domains/errors/patterns.md` — discriminated unions as the error-as-data encoding.
- `languages/ts-async.md` — typed async results built on the `Result` union here.
+7
View File
@@ -2,6 +2,13 @@
> How Atelier's domain principles apply in TypeScript specifically. Derives from `domains/` docs; this file is the language-specific lens.
## Derived Docs
- [ts-types.md](ts-types.md) — TS type system: nominal-via-branding, generics, narrowing, utility types, discriminated unions.
- [ts-tooling.md](ts-tooling.md) — tsc, ESLint, ts-jest, project references, tsconfig discipline.
- [ts-async.md](ts-async.md) — Promises + AbortSignal, async/await, error handling, cancellation.
- [ts-testing.md](ts-testing.md) — Vitest/Jest, mock discipline, type-level tests.
## Type System (C1 Correctness, Data P7 Type Fidelity)
- **Strict mode on:** `strict: true` in `tsconfig.json`. No `any` without justification.