feat(P64): pre-mortem — v1.10 post-mortem + forward pre-mortem

Cherry-picked from e868a16 (branch phase/60-pre-mortem-doc) and finalized for P64. The pre-mortem was originally authored at P60 but the roadmap assigns it to P64 (REQ-120). Part 1 — v1.10 decay incident post-mortem: - Root cause: VERIFY was diff-scoped — checked the phase diff only, never re-ran underlying capability. 8 NFR-patch phases (v1.9.1– v1.9.8) passed VERIFY while the platform decayed. - Mitigations landed in v1.10: D-091 regression gate, D-092 local emulators, D-093 capability inventory, D-094 verified-reality rewrite. Part 2 — forward pre-mortem (OSS reference + leadership pitch): - FM-1 IAM drift recurs → IAM_POLICY.md baseline test (REQ-116) - FM-2 cost spike from un-torn-down stacks → D-096 teardown mandatory - FM-3 deck overstates capability → Verified-only claims (REQ-121) - FM-4 pilot consumer hits a contract gap → honest scope (G-010) ---ci--- project: acdl phase: P64 milestone: v1.11 status: execute ---/ci---
2026-07-28 20:39:23 +00:00
parent 92bb03e808
commit 186cdde792
1 changed files with 229 additions and 0 deletions
@@ -0,0 +1,229 @@
+# ACDL — Pre-mortem (v1.11, REQ-120)
+
+> Authored: 2026-07-28, Phase 64 (previously drafted at P60, finalized here).
+> Mandated by: GRILL Axis 7 Q4 (no pre-mortem on file — flagged, no
+> binding decision; user accepted autonomous governance in G-009).
+> Structure: (1) v1.10 decay incident post-mortem, (2) forward pre-mortem
+> for the OSS reference + leadership pitch.
+
+---
+
+## Part 1 — Post-mortem: v1.10 capability decay incident
+
+### Summary
+
+Capabilities marked complete in v1.1–v1.8 ran successfully at the time
+of tagging. As of 2026-07-27 they were **not reproducible** — the v1.7/
+v1.8 platform simplification introduced 7 adapter defects in
+`adapters/terraform/adapter.py` that prevented `terraform init/
+validate/plan` from succeeding against live AWS. The decks (v1.9.1–
+v1.9.8) presented the capability as current across 8 NFR-patch phases
+**without disclosing the decay**. v1.10 (Phases 52–55) re-verified every
+advertised capability, fixed all 7 defects in-sweep (D-090: no cap), and
+rewrote PROJECT/ROADMAP/decks to match verified reality.
+
+### Timeline
+
+| Date | Event |
+|------|-------|
+| 2026-07-21 | v1.7 Phases 22–27 ship. The adapter simplification lands (the 7 defects are introduced here). |
+| 2026-07-21 | v1.8 Phases 28–38 ship. The defects persist undetected; VERIFY is diff-scoped so the decay is invisible. |
+| 2026-07-21 → 2026-07-27 | v1.9.0 + v1.9.1–v1.9.8 (8 NFR-patch phases) ship. Each passes VERIFY (diff-scoped — checks the phase diff only, never re-runs underlying capability). Decks present capability as current. |
+| 2026-07-27 | CLARIFY/RESEARCH for v1.10 surfaces the structural defect: VERIFY is diff-scoped; advertised capability is not reproducible; deck work was sequenced backwards. |
+| 2026-07-27 | User decisions D-090 (no cap on sweep), D-091 (regression-class VERIFY), D-092 (local emulating adapters), D-093 (re-verify v1.1→v1.8), D-094 (rewrite to verified reality). |
+| 2026-07-27 | Phase 52 adds the regression-class VERIFY. Phase 53 builds local emulating adapters. Phase 54 enumerates + re-verifies every capability — finds 7 adapter defects, fixes all in-sweep. Phase 55 rewrites PROJECT/ROADMAP/decks to verified reality. |
+| 2026-07-27 | v1.10.0 tagged; all 16 auto-verifiable capabilities Verified. 6 IAM-gated capabilities (CAP-017..022) escalated (G-005). |
+
+### Root cause
+
+**VERIFY was diff-scoped.** The standard VERIFY stage checked the phase
+diff only — the files changed in that phase — and never re-ran the
+underlying platform capability. 8 NFR-patch phases (v1.9.1→v1.9.8)
+passed VERIFY while the platform decayed underneath, because each
+phase's diff was docs-only (decks) and the decay was in code the diff
+didn't touch. The VERIFY gate was structurally incapable of catching
+decay in code outside the phase diff.
+
+### Contributing factors
+
+1. **Deck work was sequenced backwards.** The honest order is
+   re-verify → rewrite → polish. v1.9.x did it backwards: polish the
+   decks first, then discover (in v1.10) that the capability they
+   advertised had decayed.
+2. **No regression-class gate existed.** Each milestone's VERIFY
+   re-checked the phase diff, not the cumulative capability. There was
+   no mechanism to ask "does everything we previously claimed still
+   work?"
+3. **Local emulating adapters did not exist.** Without a local tier,
+   re-verification required live AWS access on every phase — costly and
+   not run. The decay was therefore never re-probed between v1.7 and
+   v1.10.
+4. **Decks were frozen before re-verification.** The v1.9.x decks
+   presented capability as current without a re-verification step
+   gating the claim.
+
+### Impact
+
+- **8 phases of inaccurate status reporting.** v1.9.1–v1.9.8 decks
+  advertised capability as current that was not reproducible.
+- **7 adapter defects shipped undetected.** Duplicate output
+  definitions, duplicate args, missing required args, deprecated AWS
+  provider v5 arg names — all in `adapters/terraform/adapter.py`.
+- **Credibility gap.** The OSS reference's headline E2E did not run
+  against live AWS between v1.7 and v1.10. The grill (G-005) flagged
+  this as the project-killing risk.
+
+### Mitigations (landed in v1.10)
+
+| Mitigation | Decision | Status |
+|-----------|----------|--------|
+| Regression-class VERIFY that re-runs capability checks at milestone completion | D-091 (REQ-112) | Landed — `scripts/run_regression.sh` + `core/regression_verify.py`. 16/16 Verified at v1.10.0. |
+| Local emulating adapters so the platform is fully locally testable without cloud credentials | D-092 (REQ-113) | Landed — flat-file DynamoDB outbox, local ECS Fargate emulator, local S3 state, local Lambda stub. Headline E2E runs locally. |
+| Capability inventory with per-capability Verified/Decayed/Broken tags | D-093 (REQ-114) | Landed — `.ciagent/CAPABILITY_INVENTORY.md`. 16/16 Verified; 6 IAM-gated escalated (G-005). |
+| Rewrite docs/decks to verified reality; decks unfrozen only after re-verification | D-094 (REQ-115) | Landed — PROJECT.md §Capability Status (Re-Verified 2026-07-27), ROADMAP v1.9.x noted as superseded-by-reverification, both decks rewritten. |
+
+### Follow-up (accepted debt)
+
+- **G-007 (per-phase regression):** the regression gate runs at
+  milestone completion, not per-phase. Inter-milestone decay between
+  phase N and milestone COMPLETE is an accepted trade-off (grill Axis 3
+  Q4, confidence 0.70). Per-phase regression hardening is a separate
+  future milestone.
+- **G-005 (IAM-gated capabilities):** 6 capabilities (CAP-017..022)
+  remain deploy-unverified as of v1.10 — the spike-runner cannot fix
+  its own IAM. v1.11 (this milestone) closes G-005 by re-bootstrapping
+  IAM and live-deploying the stacks.
+
+---
+
+## Part 2 — Forward pre-mortem: OSS reference + leadership pitch
+
+### Scenario
+
+It is 90 days after the v1.11 ship. The leadership pitch has been
+delivered. The grill's 90-day conditions (G-001 pitch yields a pilot
+platform team; G-005 deploy path verifiable; G-008 cost operating model
+documented) were the success criteria. **Assume the project has failed.**
+What killed it?
+
+### Top failure modes + mitigations
+
+#### FM-1 — IAM drift recurs (the spike-runner loses permissions again)
+
+**How it kills the project:** the v1.11 IAM re-bootstrap grants are
+revoked or drift (admin action, account re-organization, SCP change).
+The next regression run (D-091) fails closed on CAP-017..022. The
+verified-reality claim in the decks becomes false again — a repeat of
+the v1.10 incident in a different shape. Leadership loses trust.
+
+**Mitigation (user-owned):**
+- The IAM policy baseline is now regression-tested
+  (`tests/test_iam_policy_baseline.py`, REQ-116). Any permission removal
+  surfaces as a test failure at the next milestone COMPLETE — the gate
+  fails closed, the false claim never ships.
+- `.ciagent/IAM_POLICY.md` documents the required grants. An admin who
+  re-organizes the account can read the baseline and re-grant.
+- The user reviews the baseline test at each milestone COMPLETE. If the
+  grants have drifted, the user re-bootstraps (D-095 path) before
+  re-attempting COMPLETE.
+
+#### FM-2 — Cost spike from un-torn-down stacks
+
+**How it kills the project:** the v1.11 deploy-verification leaves the
+microservice + static-assets + uptime stacks running. Live ECS Fargate +
+CloudFront + WAF accrue spend. The COST.md (REQ-119) documents the
+v1.0–v1.10 window, not the ongoing burn. A pilot platform team clones
+the reference, runs the same apply, and leaves it running — multiply
+the spend by the number of clones. AWS budget alerts fire at leadership
+level. The reference is perceived as expensive.
+
+**Mitigation (user-owned):**
+- **D-096 (teardown mandatory before milestone COMPLETE).** Phase 61
+  tears down the stacks via D-070 decommission mode. The live AWS
+  account returns to zero-cost steady state. The milestone does not
+  complete until teardown is verified.
+- **COST.md teardown guidance.** REQ-119 documents the teardown path +
+  cost-ceiling guidance for downstream clones. A clone that follows
+  the guidance runs the same teardown.
+- The user enforces D-096 at Phase 61 — no merge to main until
+  `terraform show` confirms no resources. The `decommissioned:
+  { stack, cr_id, completed_at }` record in the `---ci---` block is
+  the audit trail.
+
+#### FM-3 — Deck overstates capability (a future v1.9.x-style incident)
+
+**How it kills the project:** a future NFR-patch milestone adds a deck
+slide claiming a capability that hasn't been re-verified. The
+regression gate runs at milestone COMPLETE and catches the underlying
+decay — but the deck has already been rendered and uploaded to a
+release. Leadership sees the deck before the regression gate fails.
+Repeat of the v1.9.x sequencing incident.
+
+**Mitigation (user-owned):**
+- **Verified-only claims.** REQ-121 enforces that decks match
+  `CAPABILITY_INVENTORY.md` exactly; `ci-doc-verifier` confirms no
+  stale claims. Any deck claim must trace to a Verified capability.
+- **Decks unfrozen only after re-verification.** The v1.10 lesson
+  (D-094) is codified: decks are frozen until the regression gate
+  passes. A future milestone that adds a deck slide must land the
+  capability re-verification in the same milestone.
+- The user reviews the `ci-doc-verifier` output at each milestone
+  COMPLETE. If a stale claim is found, the milestone does not complete
+  until the deck is corrected.
+
+#### FM-4 — Pilot consumer hits a contract gap
+
+**How it kills the project:** a pilot platform team (post-pitch) clones
+the reference and tries to deploy a stack the L2 catalog doesn't cover
+(e.g. a worker queue, a scheduled job, a database-backed service). The
+contract schema + L2 compositions support only microservice + static-
+assets. The pilot team concludes the reference is a demo, not a
+foundation. The pitch's "feature-complete MVP" claim (G-001) is
+undermined.
+
+**Mitigation (user-owned):**
+- **CONSUMER_GUIDE.md + L2 catalog coverage.** `docs/CONSUMER_GUIDE.md`
+  documents the supported L2 compositions; the L2 catalog
+  (`modules/l2/`) is the supported surface. A pilot team that reads the
+  guide knows the boundary before cloning.
+- **Honest scope.** The grill (G-010) accepted OSS scope as
+  contributor-bounded. The pitch should not claim "any stack" — it
+  should claim "microservice + static-assets today; the L2 pattern is
+  extensible." The v1.9.5 Anti-goals slide (What This Platform Is —
+  and Isn't) is the honest framing.
+- The user adds L2 compositions as pilot demand surfaces. The reference
+  value is the *shape* (contract → IR → adapter → terraform →
+  confidence → outbox), not the catalog size. A pilot team that
+  understands the shape can extend it.
+
+### What the pre-mortem tells us
+
+The four failure modes all reduce to the same root pattern: **a claim
+outruns the verification that backs it.** v1.10 was the first instance
+(decks outran capability). v1.11 closes G-005 + G-008 by making the
+verification back the claim. The mitigations are all structural —
+regression-testable baselines, mandatory teardown, Verified-only deck
+claims, honest scope — not procedural. The user owns enforcement at
+each milestone COMPLETE.
+
+### Confidence
+
+- FM-1 (IAM drift recurs): confidence 0.75 — the baseline test catches
+  it; the user enforces re-bootstrap at COMPLETE.
+- FM-2 (cost spike): confidence 0.85 — D-096 teardown is mandatory and
+  audited in the `---ci---` block.
+- FM-3 (deck overstates): confidence 0.70 — `ci-doc-verifier` is
+  automated; the sequencing risk is procedural.
+- FM-4 (pilot contract gap): confidence 0.65 — the mitigation is
+  honest framing, not catalog completeness; a pilot may still hit the
+  gap.
+
+### Links to existing controls
+
+- D-091 regression gate (REQ-112) — `scripts/run_regression.sh`.
+- D-094 verified-reality rewrite (REQ-115) — decks match
+  `CAPABILITY_INVENTORY.md`.
+- D-096 teardown mandatory (v1.11) — Phase 61.
+- G-005 deploy verification (v1.11) — Phases 56–58.
+- G-008 cost documentation (v1.11) — Phase 59.
+- G-010 contributor-bounded scope — honest pitch framing.