Jon Chery
496303471d
---ci--- project: atelier phase: 7 milestone: v0.1 status: complete phase_role: final milestone_complete: true requirements: covered: [ATELIER-01, ATELIER-02, ATELIER-03, ATELIER-04, ATELIER-05, ATELIER-06, ATELIER-07, ATELIER-08, ATELIER-09, ATELIER-10, ATELIER-11, ATELIER-12, ATELIER-13, ATELIER-14, ATELIER-15, ATELIER-16, ATELIER-17, ATELIER-18, ATELIER-19, ATELIER-20, ATELIER-21, ATELIER-22, ATELIER-23, ATELIER-24, ATELIER-25, ATELIER-26, ATELIER-27, ATELIER-28, ATELIER-29, ATELIER-30, ATELIER-31, ATELIER-32, ATELIER-33, ATELIER-34, ATELIER-35] partial: [] ship: milestone: v0.1 type: NFR tag: v0.0.7 merge: milestone/v0.1-atelier -> main release: https://git.cloudinit.dev/cloudinit-bot/atelier/releases/tag/v0.0.7 ---/ci--- Milestone v0.1 — Initial Framework (NFR, complete). 8 core principles (C1-C8), 11 domains, 110 domain principles, 27 derived docs, 4 good + 3 bad examples, 4 language docs, full matrix, 3 review docs. All 35 requirements covered. 7 patches (v0.0.0 pre-execution through v0.0.7 final). v0.0.7 IS the v0.1.0 milestone release.
2.4 KiB
2.4 KiB
Test Pyramid — Derived Rules
Derives from
domains/testing/first-principles.mdP4 (Fast Feedback), P5 (Coverage of Behavior), P10 (No Test Theater).
The Pyramid
/\
/e2e\ few, slow, integration
/------\
/ integ \ some, medium, contract
/----------\
/ unit \ many, fast, isolated
/--------------\
- Unit (many): test a function/class in isolation. Fast (< 10ms each). The bulk of tests.
- Integration (some): test components together (DB, API client, queue). Medium (< 1s each).
- E2E (few): test the whole system from outside. Slow (> 1s each). The tip of the pyramid.
Why a Pyramid (P4 Fast Feedback)
- A pyramid inverts to a "ice cream cone" (many e2e, few unit) when devs avoid unit tests.
- Inverted pyramids are slow and flaky. The feedback loop breaks.
- The pyramid shape preserves fast feedback: most failures are unit failures, found in < 10ms.
What Goes Where
| Test Type | What it Covers | Speed | Count |
|---|---|---|---|
| Unit | A function, a class, a pure module | < 10ms | Many |
| Integration | DB queries, API contract, queue behavior | < 1s | Some |
| E2E | A user flow, an API request → response end-to-end | > 1s | Few |
- A unit test does not hit the database. A unit test does not make a network call.
- An integration test does not test business logic; it tests the integration.
- An e2e test does not test edge cases; it tests the happy path. Edge cases are unit tests.
Anti-Patterns (P10 No Test Theater)
- Ice cream cone: many e2e, few unit. Slow, flaky, no signal.
- Cupcake: same count at every level. No pyramid shape. Slow.
- Only unit: 100% unit coverage, 0% integration. The system is untested as a whole.
- Only e2e: every edge case is an e2e test. The suite takes an hour.
Coverage (P5)
- Unit coverage of behavior: every branch, every edge case, every error path.
- Integration coverage of contracts: every API endpoint, every DB query, every queue interaction.
- E2E coverage of flows: the top 3–5 critical user flows. Not every permutation.
What Violates the Pyramid
| Violation | Principle |
|---|---|
| E2E test for an edge case | P4 (slow feedback) |
| Unit test that hits the DB | P2 (not isolated) |
| 0 integration tests | P5 (no coverage of contracts) |
| 500 e2e tests, 50 unit tests | P10 (theater) |
| A 30-second test suite | P4 (feedback loop broken) |