From 186cdde792087ba933805ff6fa9df52d56bca93d Mon Sep 17 00:00:00 2001 From: Jon Chery Date: Tue, 28 Jul 2026 20:39:23 +0000 Subject: [PATCH] =?UTF-8?q?feat(P64):=20pre-mortem=20=E2=80=94=20v1.10=20p?= =?UTF-8?q?ost-mortem=20+=20forward=20pre-mortem?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cherry-picked from e868a16 (branch phase/60-pre-mortem-doc) and finalized for P64. The pre-mortem was originally authored at P60 but the roadmap assigns it to P64 (REQ-120). Part 1 — v1.10 decay incident post-mortem: - Root cause: VERIFY was diff-scoped — checked the phase diff only, never re-ran underlying capability. 8 NFR-patch phases (v1.9.1– v1.9.8) passed VERIFY while the platform decayed. - Mitigations landed in v1.10: D-091 regression gate, D-092 local emulators, D-093 capability inventory, D-094 verified-reality rewrite. Part 2 — forward pre-mortem (OSS reference + leadership pitch): - FM-1 IAM drift recurs → IAM_POLICY.md baseline test (REQ-116) - FM-2 cost spike from un-torn-down stacks → D-096 teardown mandatory - FM-3 deck overstates capability → Verified-only claims (REQ-121) - FM-4 pilot consumer hits a contract gap → honest scope (G-010) ---ci--- project: acdl phase: P64 milestone: v1.11 status: execute ---/ci--- --- .ciagent/PRE_MORTEM.md | 229 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 .ciagent/PRE_MORTEM.md diff --git a/.ciagent/PRE_MORTEM.md b/.ciagent/PRE_MORTEM.md new file mode 100644 index 0000000..2182945 --- /dev/null +++ b/.ciagent/PRE_MORTEM.md @@ -0,0 +1,229 @@ +# ACDL — Pre-mortem (v1.11, REQ-120) + +> Authored: 2026-07-28, Phase 64 (previously drafted at P60, finalized here). +> Mandated by: GRILL Axis 7 Q4 (no pre-mortem on file — flagged, no +> binding decision; user accepted autonomous governance in G-009). +> Structure: (1) v1.10 decay incident post-mortem, (2) forward pre-mortem +> for the OSS reference + leadership pitch. + +--- + +## Part 1 — Post-mortem: v1.10 capability decay incident + +### Summary + +Capabilities marked complete in v1.1–v1.8 ran successfully at the time +of tagging. As of 2026-07-27 they were **not reproducible** — the v1.7/ +v1.8 platform simplification introduced 7 adapter defects in +`adapters/terraform/adapter.py` that prevented `terraform init/ +validate/plan` from succeeding against live AWS. The decks (v1.9.1– +v1.9.8) presented the capability as current across 8 NFR-patch phases +**without disclosing the decay**. v1.10 (Phases 52–55) re-verified every +advertised capability, fixed all 7 defects in-sweep (D-090: no cap), and +rewrote PROJECT/ROADMAP/decks to match verified reality. + +### Timeline + +| Date | Event | +|------|-------| +| 2026-07-21 | v1.7 Phases 22–27 ship. The adapter simplification lands (the 7 defects are introduced here). | +| 2026-07-21 | v1.8 Phases 28–38 ship. The defects persist undetected; VERIFY is diff-scoped so the decay is invisible. | +| 2026-07-21 → 2026-07-27 | v1.9.0 + v1.9.1–v1.9.8 (8 NFR-patch phases) ship. Each passes VERIFY (diff-scoped — checks the phase diff only, never re-runs underlying capability). Decks present capability as current. | +| 2026-07-27 | CLARIFY/RESEARCH for v1.10 surfaces the structural defect: VERIFY is diff-scoped; advertised capability is not reproducible; deck work was sequenced backwards. | +| 2026-07-27 | User decisions D-090 (no cap on sweep), D-091 (regression-class VERIFY), D-092 (local emulating adapters), D-093 (re-verify v1.1→v1.8), D-094 (rewrite to verified reality). | +| 2026-07-27 | Phase 52 adds the regression-class VERIFY. Phase 53 builds local emulating adapters. Phase 54 enumerates + re-verifies every capability — finds 7 adapter defects, fixes all in-sweep. Phase 55 rewrites PROJECT/ROADMAP/decks to verified reality. | +| 2026-07-27 | v1.10.0 tagged; all 16 auto-verifiable capabilities Verified. 6 IAM-gated capabilities (CAP-017..022) escalated (G-005). | + +### Root cause + +**VERIFY was diff-scoped.** The standard VERIFY stage checked the phase +diff only — the files changed in that phase — and never re-ran the +underlying platform capability. 8 NFR-patch phases (v1.9.1→v1.9.8) +passed VERIFY while the platform decayed underneath, because each +phase's diff was docs-only (decks) and the decay was in code the diff +didn't touch. The VERIFY gate was structurally incapable of catching +decay in code outside the phase diff. + +### Contributing factors + +1. **Deck work was sequenced backwards.** The honest order is + re-verify → rewrite → polish. v1.9.x did it backwards: polish the + decks first, then discover (in v1.10) that the capability they + advertised had decayed. +2. **No regression-class gate existed.** Each milestone's VERIFY + re-checked the phase diff, not the cumulative capability. There was + no mechanism to ask "does everything we previously claimed still + work?" +3. **Local emulating adapters did not exist.** Without a local tier, + re-verification required live AWS access on every phase — costly and + not run. The decay was therefore never re-probed between v1.7 and + v1.10. +4. **Decks were frozen before re-verification.** The v1.9.x decks + presented capability as current without a re-verification step + gating the claim. + +### Impact + +- **8 phases of inaccurate status reporting.** v1.9.1–v1.9.8 decks + advertised capability as current that was not reproducible. +- **7 adapter defects shipped undetected.** Duplicate output + definitions, duplicate args, missing required args, deprecated AWS + provider v5 arg names — all in `adapters/terraform/adapter.py`. +- **Credibility gap.** The OSS reference's headline E2E did not run + against live AWS between v1.7 and v1.10. The grill (G-005) flagged + this as the project-killing risk. + +### Mitigations (landed in v1.10) + +| Mitigation | Decision | Status | +|-----------|----------|--------| +| Regression-class VERIFY that re-runs capability checks at milestone completion | D-091 (REQ-112) | Landed — `scripts/run_regression.sh` + `core/regression_verify.py`. 16/16 Verified at v1.10.0. | +| Local emulating adapters so the platform is fully locally testable without cloud credentials | D-092 (REQ-113) | Landed — flat-file DynamoDB outbox, local ECS Fargate emulator, local S3 state, local Lambda stub. Headline E2E runs locally. | +| Capability inventory with per-capability Verified/Decayed/Broken tags | D-093 (REQ-114) | Landed — `.ciagent/CAPABILITY_INVENTORY.md`. 16/16 Verified; 6 IAM-gated escalated (G-005). | +| Rewrite docs/decks to verified reality; decks unfrozen only after re-verification | D-094 (REQ-115) | Landed — PROJECT.md §Capability Status (Re-Verified 2026-07-27), ROADMAP v1.9.x noted as superseded-by-reverification, both decks rewritten. | + +### Follow-up (accepted debt) + +- **G-007 (per-phase regression):** the regression gate runs at + milestone completion, not per-phase. Inter-milestone decay between + phase N and milestone COMPLETE is an accepted trade-off (grill Axis 3 + Q4, confidence 0.70). Per-phase regression hardening is a separate + future milestone. +- **G-005 (IAM-gated capabilities):** 6 capabilities (CAP-017..022) + remain deploy-unverified as of v1.10 — the spike-runner cannot fix + its own IAM. v1.11 (this milestone) closes G-005 by re-bootstrapping + IAM and live-deploying the stacks. + +--- + +## Part 2 — Forward pre-mortem: OSS reference + leadership pitch + +### Scenario + +It is 90 days after the v1.11 ship. The leadership pitch has been +delivered. The grill's 90-day conditions (G-001 pitch yields a pilot +platform team; G-005 deploy path verifiable; G-008 cost operating model +documented) were the success criteria. **Assume the project has failed.** +What killed it? + +### Top failure modes + mitigations + +#### FM-1 — IAM drift recurs (the spike-runner loses permissions again) + +**How it kills the project:** the v1.11 IAM re-bootstrap grants are +revoked or drift (admin action, account re-organization, SCP change). +The next regression run (D-091) fails closed on CAP-017..022. The +verified-reality claim in the decks becomes false again — a repeat of +the v1.10 incident in a different shape. Leadership loses trust. + +**Mitigation (user-owned):** +- The IAM policy baseline is now regression-tested + (`tests/test_iam_policy_baseline.py`, REQ-116). Any permission removal + surfaces as a test failure at the next milestone COMPLETE — the gate + fails closed, the false claim never ships. +- `.ciagent/IAM_POLICY.md` documents the required grants. An admin who + re-organizes the account can read the baseline and re-grant. +- The user reviews the baseline test at each milestone COMPLETE. If the + grants have drifted, the user re-bootstraps (D-095 path) before + re-attempting COMPLETE. + +#### FM-2 — Cost spike from un-torn-down stacks + +**How it kills the project:** the v1.11 deploy-verification leaves the +microservice + static-assets + uptime stacks running. Live ECS Fargate + +CloudFront + WAF accrue spend. The COST.md (REQ-119) documents the +v1.0–v1.10 window, not the ongoing burn. A pilot platform team clones +the reference, runs the same apply, and leaves it running — multiply +the spend by the number of clones. AWS budget alerts fire at leadership +level. The reference is perceived as expensive. + +**Mitigation (user-owned):** +- **D-096 (teardown mandatory before milestone COMPLETE).** Phase 61 + tears down the stacks via D-070 decommission mode. The live AWS + account returns to zero-cost steady state. The milestone does not + complete until teardown is verified. +- **COST.md teardown guidance.** REQ-119 documents the teardown path + + cost-ceiling guidance for downstream clones. A clone that follows + the guidance runs the same teardown. +- The user enforces D-096 at Phase 61 — no merge to main until + `terraform show` confirms no resources. The `decommissioned: + { stack, cr_id, completed_at }` record in the `---ci---` block is + the audit trail. + +#### FM-3 — Deck overstates capability (a future v1.9.x-style incident) + +**How it kills the project:** a future NFR-patch milestone adds a deck +slide claiming a capability that hasn't been re-verified. The +regression gate runs at milestone COMPLETE and catches the underlying +decay — but the deck has already been rendered and uploaded to a +release. Leadership sees the deck before the regression gate fails. +Repeat of the v1.9.x sequencing incident. + +**Mitigation (user-owned):** +- **Verified-only claims.** REQ-121 enforces that decks match + `CAPABILITY_INVENTORY.md` exactly; `ci-doc-verifier` confirms no + stale claims. Any deck claim must trace to a Verified capability. +- **Decks unfrozen only after re-verification.** The v1.10 lesson + (D-094) is codified: decks are frozen until the regression gate + passes. A future milestone that adds a deck slide must land the + capability re-verification in the same milestone. +- The user reviews the `ci-doc-verifier` output at each milestone + COMPLETE. If a stale claim is found, the milestone does not complete + until the deck is corrected. + +#### FM-4 — Pilot consumer hits a contract gap + +**How it kills the project:** a pilot platform team (post-pitch) clones +the reference and tries to deploy a stack the L2 catalog doesn't cover +(e.g. a worker queue, a scheduled job, a database-backed service). The +contract schema + L2 compositions support only microservice + static- +assets. The pilot team concludes the reference is a demo, not a +foundation. The pitch's "feature-complete MVP" claim (G-001) is +undermined. + +**Mitigation (user-owned):** +- **CONSUMER_GUIDE.md + L2 catalog coverage.** `docs/CONSUMER_GUIDE.md` + documents the supported L2 compositions; the L2 catalog + (`modules/l2/`) is the supported surface. A pilot team that reads the + guide knows the boundary before cloning. +- **Honest scope.** The grill (G-010) accepted OSS scope as + contributor-bounded. The pitch should not claim "any stack" — it + should claim "microservice + static-assets today; the L2 pattern is + extensible." The v1.9.5 Anti-goals slide (What This Platform Is — + and Isn't) is the honest framing. +- The user adds L2 compositions as pilot demand surfaces. The reference + value is the *shape* (contract → IR → adapter → terraform → + confidence → outbox), not the catalog size. A pilot team that + understands the shape can extend it. + +### What the pre-mortem tells us + +The four failure modes all reduce to the same root pattern: **a claim +outruns the verification that backs it.** v1.10 was the first instance +(decks outran capability). v1.11 closes G-005 + G-008 by making the +verification back the claim. The mitigations are all structural — +regression-testable baselines, mandatory teardown, Verified-only deck +claims, honest scope — not procedural. The user owns enforcement at +each milestone COMPLETE. + +### Confidence + +- FM-1 (IAM drift recurs): confidence 0.75 — the baseline test catches + it; the user enforces re-bootstrap at COMPLETE. +- FM-2 (cost spike): confidence 0.85 — D-096 teardown is mandatory and + audited in the `---ci---` block. +- FM-3 (deck overstates): confidence 0.70 — `ci-doc-verifier` is + automated; the sequencing risk is procedural. +- FM-4 (pilot contract gap): confidence 0.65 — the mitigation is + honest framing, not catalog completeness; a pilot may still hit the + gap. + +### Links to existing controls + +- D-091 regression gate (REQ-112) — `scripts/run_regression.sh`. +- D-094 verified-reality rewrite (REQ-115) — decks match + `CAPABILITY_INVENTORY.md`. +- D-096 teardown mandatory (v1.11) — Phase 61. +- G-005 deploy verification (v1.11) — Phases 56–58. +- G-008 cost documentation (v1.11) — Phase 59. +- G-010 contributor-bounded scope — honest pitch framing. \ No newline at end of file