acdl/.ciagent/archive/PRE_MORTEM.md

ACDL — Pre-mortem (v1.11, REQ-120)

Authored: 2026-07-28, Phase 64 (previously drafted at P60, finalized here). Mandated by: GRILL Axis 7 Q4 (no pre-mortem on file — flagged, no binding decision; user accepted autonomous governance in G-009). Structure: (1) v1.10 decay incident post-mortem, (2) forward pre-mortem for the OSS reference + leadership pitch.

---

Part 1 — Post-mortem: v1.10 capability decay incident

Summary

Capabilities marked complete in v1.1–v1.8 ran successfully at the time of tagging. As of 2026-07-27 they were not reproducible — the v1.7/ v1.8 platform simplification introduced 7 adapter defects in adapters/terraform/adapter.py that prevented terraform init/ validate/plan from succeeding against live AWS. The decks (v1.9.1– v1.9.8) presented the capability as current across 8 NFR-patch phases without disclosing the decay. v1.10 (Phases 52–55) re-verified every advertised capability, fixed all 7 defects in-sweep (D-090: no cap), and rewrote PROJECT/ROADMAP/decks to match verified reality.

Timeline

Date	Event
2026-07-21	v1.7 Phases 22–27 ship. The adapter simplification lands (the 7 defects are introduced here).
2026-07-21	v1.8 Phases 28–38 ship. The defects persist undetected; VERIFY is diff-scoped so the decay is invisible.
2026-07-21 → 2026-07-27	v1.9.0 + v1.9.1–v1.9.8 (8 NFR-patch phases) ship. Each passes VERIFY (diff-scoped — checks the phase diff only, never re-runs underlying capability). Decks present capability as current.
2026-07-27	CLARIFY/RESEARCH for v1.10 surfaces the structural defect: VERIFY is diff-scoped; advertised capability is not reproducible; deck work was sequenced backwards.
2026-07-27	User decisions D-090 (no cap on sweep), D-091 (regression-class VERIFY), D-092 (local emulating adapters), D-093 (re-verify v1.1→v1.8), D-094 (rewrite to verified reality).
2026-07-27	Phase 52 adds the regression-class VERIFY. Phase 53 builds local emulating adapters. Phase 54 enumerates + re-verifies every capability — finds 7 adapter defects, fixes all in-sweep. Phase 55 rewrites PROJECT/ROADMAP/decks to verified reality.
2026-07-27	v1.10.0 tagged; all 16 auto-verifiable capabilities Verified. 6 IAM-gated capabilities (CAP-017..022) escalated (G-005).

Root cause

VERIFY was diff-scoped. The standard VERIFY stage checked the phase diff only — the files changed in that phase — and never re-ran the underlying platform capability. 8 NFR-patch phases (v1.9.1→v1.9.8) passed VERIFY while the platform decayed underneath, because each phase's diff was docs-only (decks) and the decay was in code the diff didn't touch. The VERIFY gate was structurally incapable of catching decay in code outside the phase diff.

Contributing factors

1. Deck work was sequenced backwards. The honest order is re-verify → rewrite → polish. v1.9.x did it backwards: polish the decks first, then discover (in v1.10) that the capability they advertised had decayed.
2. No regression-class gate existed. Each milestone's VERIFY re-checked the phase diff, not the cumulative capability. There was no mechanism to ask "does everything we previously claimed still work?"
3. Local emulating adapters did not exist. Without a local tier, re-verification required live AWS access on every phase — costly and not run. The decay was therefore never re-probed between v1.7 and v1.10.
4. Decks were frozen before re-verification. The v1.9.x decks presented capability as current without a re-verification step gating the claim.

Impact

• 8 phases of inaccurate status reporting. v1.9.1–v1.9.8 decks advertised capability as current that was not reproducible.
• 7 adapter defects shipped undetected. Duplicate output definitions, duplicate args, missing required args, deprecated AWS provider v5 arg names — all in adapters/terraform/adapter.py.
• Credibility gap. The OSS reference's headline E2E did not run against live AWS between v1.7 and v1.10. The grill (G-005) flagged this as the project-killing risk.

Mitigations (landed in v1.10)

Mitigation	Decision	Status
Regression-class VERIFY that re-runs capability checks at milestone completion	D-091 (REQ-112)	Landed — scripts/run_regression.sh + core/regression_verify.py. 16/16 Verified at v1.10.0.
Local emulating adapters so the platform is fully locally testable without cloud credentials	D-092 (REQ-113)	Landed — flat-file DynamoDB outbox, local ECS Fargate emulator, local S3 state, local Lambda stub. Headline E2E runs locally.
Capability inventory with per-capability Verified/Decayed/Broken tags	D-093 (REQ-114)	Landed — .ciagent/CAPABILITY_INVENTORY.md. 16/16 Verified; 6 IAM-gated escalated (G-005).
Rewrite docs/decks to verified reality; decks unfrozen only after re-verification	D-094 (REQ-115)	Landed — PROJECT.md §Capability Status (Re-Verified 2026-07-27), ROADMAP v1.9.x noted as superseded-by-reverification, both decks rewritten.

Follow-up (accepted debt)

• G-007 (per-phase regression): the regression gate runs at milestone completion, not per-phase. Inter-milestone decay between phase N and milestone COMPLETE is an accepted trade-off (grill Axis 3 Q4, confidence 0.70). Per-phase regression hardening is a separate future milestone.
• G-005 (IAM-gated capabilities): 6 capabilities (CAP-017..022) remain deploy-unverified as of v1.10 — the spike-runner cannot fix its own IAM. v1.11 (this milestone) closes G-005 by re-bootstrapping IAM and live-deploying the stacks.

---

Part 2 — Forward pre-mortem: OSS reference + leadership pitch

Scenario

It is 90 days after the v1.11 ship. The leadership pitch has been delivered. The grill's 90-day conditions (G-001 pitch yields a pilot platform team; G-005 deploy path verifiable; G-008 cost operating model documented) were the success criteria. Assume the project has failed. What killed it?

Top failure modes + mitigations

FM-1 — IAM drift recurs (the spike-runner loses permissions again)

How it kills the project: the v1.11 IAM re-bootstrap grants are revoked or drift (admin action, account re-organization, SCP change). The next regression run (D-091) fails closed on CAP-017..022. The verified-reality claim in the decks becomes false again — a repeat of the v1.10 incident in a different shape. Leadership loses trust.

Mitigation (user-owned):

• The IAM policy baseline is now regression-tested (tests/test_iam_policy_baseline.py, REQ-116). Any permission removal surfaces as a test failure at the next milestone COMPLETE — the gate fails closed, the false claim never ships.
• .ciagent/IAM_POLICY.md documents the required grants. An admin who re-organizes the account can read the baseline and re-grant.
• The user reviews the baseline test at each milestone COMPLETE. If the grants have drifted, the user re-bootstraps (D-095 path) before re-attempting COMPLETE.

FM-2 — Cost spike from un-torn-down stacks

How it kills the project: the v1.11 deploy-verification leaves the microservice + static-assets + uptime stacks running. Live ECS Fargate + CloudFront + WAF accrue spend. The COST.md (REQ-119) documents the v1.0–v1.10 window, not the ongoing burn. A pilot platform team clones the reference, runs the same apply, and leaves it running — multiply the spend by the number of clones. AWS budget alerts fire at leadership level. The reference is perceived as expensive.

Mitigation (user-owned):

• D-096 (teardown mandatory before milestone COMPLETE). Phase 61 tears down the stacks via D-070 decommission mode. The live AWS account returns to zero-cost steady state. The milestone does not complete until teardown is verified.
• COST.md teardown guidance. REQ-119 documents the teardown path + cost-ceiling guidance for downstream clones. A clone that follows the guidance runs the same teardown.
• The user enforces D-096 at Phase 61 — no merge to main until terraform show confirms no resources. The decommissioned: { stack, cr_id, completed_at } record in the ---ci--- block is the audit trail.

FM-3 — Deck overstates capability (a future v1.9.x-style incident)

How it kills the project: a future NFR-patch milestone adds a deck slide claiming a capability that hasn't been re-verified. The regression gate runs at milestone COMPLETE and catches the underlying decay — but the deck has already been rendered and uploaded to a release. Leadership sees the deck before the regression gate fails. Repeat of the v1.9.x sequencing incident.

Mitigation (user-owned):

• Verified-only claims. REQ-121 enforces that decks match CAPABILITY_INVENTORY.md exactly; ci-doc-verifier confirms no stale claims. Any deck claim must trace to a Verified capability.
• Decks unfrozen only after re-verification. The v1.10 lesson (D-094) is codified: decks are frozen until the regression gate passes. A future milestone that adds a deck slide must land the capability re-verification in the same milestone.
• The user reviews the ci-doc-verifier output at each milestone COMPLETE. If a stale claim is found, the milestone does not complete until the deck is corrected.

FM-4 — Pilot consumer hits a contract gap

How it kills the project: a pilot platform team (post-pitch) clones the reference and tries to deploy a stack the L2 catalog doesn't cover (e.g. a worker queue, a scheduled job, a database-backed service). The contract schema + L2 compositions support only microservice + static- assets. The pilot team concludes the reference is a demo, not a foundation. The pitch's "feature-complete MVP" claim (G-001) is undermined.

Mitigation (user-owned):

• CONSUMER_GUIDE.md + L2 catalog coverage. docs/CONSUMER_GUIDE.md documents the supported L2 compositions; the L2 catalog (modules/l2/) is the supported surface. A pilot team that reads the guide knows the boundary before cloning.
• Honest scope. The grill (G-010) accepted OSS scope as contributor-bounded. The pitch should not claim "any stack" — it should claim "microservice + static-assets today; the L2 pattern is extensible." The v1.9.5 Anti-goals slide (What This Platform Is — and Isn't) is the honest framing.
• The user adds L2 compositions as pilot demand surfaces. The reference value is the shape (contract → IR → adapter → terraform → confidence → outbox), not the catalog size. A pilot team that understands the shape can extend it.

What the pre-mortem tells us

The four failure modes all reduce to the same root pattern: a claim outruns the verification that backs it. v1.10 was the first instance (decks outran capability). v1.11 closes G-005 + G-008 by making the verification back the claim. The mitigations are all structural — regression-testable baselines, mandatory teardown, Verified-only deck claims, honest scope — not procedural. The user owns enforcement at each milestone COMPLETE.

Confidence

• FM-1 (IAM drift recurs): confidence 0.75 — the baseline test catches it; the user enforces re-bootstrap at COMPLETE.
• FM-2 (cost spike): confidence 0.85 — D-096 teardown is mandatory and audited in the ---ci--- block.
• FM-3 (deck overstates): confidence 0.70 — ci-doc-verifier is automated; the sequencing risk is procedural.
• FM-4 (pilot contract gap): confidence 0.65 — the mitigation is honest framing, not catalog completeness; a pilot may still hit the gap.

Links to existing controls

• D-091 regression gate (REQ-112) — scripts/run_regression.sh.
• D-094 verified-reality rewrite (REQ-115) — decks match CAPABILITY_INVENTORY.md.
• D-096 teardown mandatory (v1.11) — Phase 61.
• G-005 deploy verification (v1.11) — Phases 56–58.
• G-008 cost documentation (v1.11) — Phase 59.
• G-010 contributor-bounded scope — honest pitch framing.