Jon Chery
186cdde792
Cherry-picked from e868a16 (branch phase/60-pre-mortem-doc) and finalized for P64. The pre-mortem was originally authored at P60 but the roadmap assigns it to P64 (REQ-120). Part 1 — v1.10 decay incident post-mortem: - Root cause: VERIFY was diff-scoped — checked the phase diff only, never re-ran underlying capability. 8 NFR-patch phases (v1.9.1– v1.9.8) passed VERIFY while the platform decayed. - Mitigations landed in v1.10: D-091 regression gate, D-092 local emulators, D-093 capability inventory, D-094 verified-reality rewrite. Part 2 — forward pre-mortem (OSS reference + leadership pitch): - FM-1 IAM drift recurs → IAM_POLICY.md baseline test (REQ-116) - FM-2 cost spike from un-torn-down stacks → D-096 teardown mandatory - FM-3 deck overstates capability → Verified-only claims (REQ-121) - FM-4 pilot consumer hits a contract gap → honest scope (G-010) ---ci--- project: acdl phase: P64 milestone: v1.11 status: execute ---/ci---
229 lines
12 KiB
Markdown
229 lines
12 KiB
Markdown
# ACDL — Pre-mortem (v1.11, REQ-120)
|
||
|
||
> Authored: 2026-07-28, Phase 64 (previously drafted at P60, finalized here).
|
||
> Mandated by: GRILL Axis 7 Q4 (no pre-mortem on file — flagged, no
|
||
> binding decision; user accepted autonomous governance in G-009).
|
||
> Structure: (1) v1.10 decay incident post-mortem, (2) forward pre-mortem
|
||
> for the OSS reference + leadership pitch.
|
||
|
||
---
|
||
|
||
## Part 1 — Post-mortem: v1.10 capability decay incident
|
||
|
||
### Summary
|
||
|
||
Capabilities marked complete in v1.1–v1.8 ran successfully at the time
|
||
of tagging. As of 2026-07-27 they were **not reproducible** — the v1.7/
|
||
v1.8 platform simplification introduced 7 adapter defects in
|
||
`adapters/terraform/adapter.py` that prevented `terraform init/
|
||
validate/plan` from succeeding against live AWS. The decks (v1.9.1–
|
||
v1.9.8) presented the capability as current across 8 NFR-patch phases
|
||
**without disclosing the decay**. v1.10 (Phases 52–55) re-verified every
|
||
advertised capability, fixed all 7 defects in-sweep (D-090: no cap), and
|
||
rewrote PROJECT/ROADMAP/decks to match verified reality.
|
||
|
||
### Timeline
|
||
|
||
| Date | Event |
|
||
|------|-------|
|
||
| 2026-07-21 | v1.7 Phases 22–27 ship. The adapter simplification lands (the 7 defects are introduced here). |
|
||
| 2026-07-21 | v1.8 Phases 28–38 ship. The defects persist undetected; VERIFY is diff-scoped so the decay is invisible. |
|
||
| 2026-07-21 → 2026-07-27 | v1.9.0 + v1.9.1–v1.9.8 (8 NFR-patch phases) ship. Each passes VERIFY (diff-scoped — checks the phase diff only, never re-runs underlying capability). Decks present capability as current. |
|
||
| 2026-07-27 | CLARIFY/RESEARCH for v1.10 surfaces the structural defect: VERIFY is diff-scoped; advertised capability is not reproducible; deck work was sequenced backwards. |
|
||
| 2026-07-27 | User decisions D-090 (no cap on sweep), D-091 (regression-class VERIFY), D-092 (local emulating adapters), D-093 (re-verify v1.1→v1.8), D-094 (rewrite to verified reality). |
|
||
| 2026-07-27 | Phase 52 adds the regression-class VERIFY. Phase 53 builds local emulating adapters. Phase 54 enumerates + re-verifies every capability — finds 7 adapter defects, fixes all in-sweep. Phase 55 rewrites PROJECT/ROADMAP/decks to verified reality. |
|
||
| 2026-07-27 | v1.10.0 tagged; all 16 auto-verifiable capabilities Verified. 6 IAM-gated capabilities (CAP-017..022) escalated (G-005). |
|
||
|
||
### Root cause
|
||
|
||
**VERIFY was diff-scoped.** The standard VERIFY stage checked the phase
|
||
diff only — the files changed in that phase — and never re-ran the
|
||
underlying platform capability. 8 NFR-patch phases (v1.9.1→v1.9.8)
|
||
passed VERIFY while the platform decayed underneath, because each
|
||
phase's diff was docs-only (decks) and the decay was in code the diff
|
||
didn't touch. The VERIFY gate was structurally incapable of catching
|
||
decay in code outside the phase diff.
|
||
|
||
### Contributing factors
|
||
|
||
1. **Deck work was sequenced backwards.** The honest order is
|
||
re-verify → rewrite → polish. v1.9.x did it backwards: polish the
|
||
decks first, then discover (in v1.10) that the capability they
|
||
advertised had decayed.
|
||
2. **No regression-class gate existed.** Each milestone's VERIFY
|
||
re-checked the phase diff, not the cumulative capability. There was
|
||
no mechanism to ask "does everything we previously claimed still
|
||
work?"
|
||
3. **Local emulating adapters did not exist.** Without a local tier,
|
||
re-verification required live AWS access on every phase — costly and
|
||
not run. The decay was therefore never re-probed between v1.7 and
|
||
v1.10.
|
||
4. **Decks were frozen before re-verification.** The v1.9.x decks
|
||
presented capability as current without a re-verification step
|
||
gating the claim.
|
||
|
||
### Impact
|
||
|
||
- **8 phases of inaccurate status reporting.** v1.9.1–v1.9.8 decks
|
||
advertised capability as current that was not reproducible.
|
||
- **7 adapter defects shipped undetected.** Duplicate output
|
||
definitions, duplicate args, missing required args, deprecated AWS
|
||
provider v5 arg names — all in `adapters/terraform/adapter.py`.
|
||
- **Credibility gap.** The OSS reference's headline E2E did not run
|
||
against live AWS between v1.7 and v1.10. The grill (G-005) flagged
|
||
this as the project-killing risk.
|
||
|
||
### Mitigations (landed in v1.10)
|
||
|
||
| Mitigation | Decision | Status |
|
||
|-----------|----------|--------|
|
||
| Regression-class VERIFY that re-runs capability checks at milestone completion | D-091 (REQ-112) | Landed — `scripts/run_regression.sh` + `core/regression_verify.py`. 16/16 Verified at v1.10.0. |
|
||
| Local emulating adapters so the platform is fully locally testable without cloud credentials | D-092 (REQ-113) | Landed — flat-file DynamoDB outbox, local ECS Fargate emulator, local S3 state, local Lambda stub. Headline E2E runs locally. |
|
||
| Capability inventory with per-capability Verified/Decayed/Broken tags | D-093 (REQ-114) | Landed — `.ciagent/CAPABILITY_INVENTORY.md`. 16/16 Verified; 6 IAM-gated escalated (G-005). |
|
||
| Rewrite docs/decks to verified reality; decks unfrozen only after re-verification | D-094 (REQ-115) | Landed — PROJECT.md §Capability Status (Re-Verified 2026-07-27), ROADMAP v1.9.x noted as superseded-by-reverification, both decks rewritten. |
|
||
|
||
### Follow-up (accepted debt)
|
||
|
||
- **G-007 (per-phase regression):** the regression gate runs at
|
||
milestone completion, not per-phase. Inter-milestone decay between
|
||
phase N and milestone COMPLETE is an accepted trade-off (grill Axis 3
|
||
Q4, confidence 0.70). Per-phase regression hardening is a separate
|
||
future milestone.
|
||
- **G-005 (IAM-gated capabilities):** 6 capabilities (CAP-017..022)
|
||
remain deploy-unverified as of v1.10 — the spike-runner cannot fix
|
||
its own IAM. v1.11 (this milestone) closes G-005 by re-bootstrapping
|
||
IAM and live-deploying the stacks.
|
||
|
||
---
|
||
|
||
## Part 2 — Forward pre-mortem: OSS reference + leadership pitch
|
||
|
||
### Scenario
|
||
|
||
It is 90 days after the v1.11 ship. The leadership pitch has been
|
||
delivered. The grill's 90-day conditions (G-001 pitch yields a pilot
|
||
platform team; G-005 deploy path verifiable; G-008 cost operating model
|
||
documented) were the success criteria. **Assume the project has failed.**
|
||
What killed it?
|
||
|
||
### Top failure modes + mitigations
|
||
|
||
#### FM-1 — IAM drift recurs (the spike-runner loses permissions again)
|
||
|
||
**How it kills the project:** the v1.11 IAM re-bootstrap grants are
|
||
revoked or drift (admin action, account re-organization, SCP change).
|
||
The next regression run (D-091) fails closed on CAP-017..022. The
|
||
verified-reality claim in the decks becomes false again — a repeat of
|
||
the v1.10 incident in a different shape. Leadership loses trust.
|
||
|
||
**Mitigation (user-owned):**
|
||
- The IAM policy baseline is now regression-tested
|
||
(`tests/test_iam_policy_baseline.py`, REQ-116). Any permission removal
|
||
surfaces as a test failure at the next milestone COMPLETE — the gate
|
||
fails closed, the false claim never ships.
|
||
- `.ciagent/IAM_POLICY.md` documents the required grants. An admin who
|
||
re-organizes the account can read the baseline and re-grant.
|
||
- The user reviews the baseline test at each milestone COMPLETE. If the
|
||
grants have drifted, the user re-bootstraps (D-095 path) before
|
||
re-attempting COMPLETE.
|
||
|
||
#### FM-2 — Cost spike from un-torn-down stacks
|
||
|
||
**How it kills the project:** the v1.11 deploy-verification leaves the
|
||
microservice + static-assets + uptime stacks running. Live ECS Fargate +
|
||
CloudFront + WAF accrue spend. The COST.md (REQ-119) documents the
|
||
v1.0–v1.10 window, not the ongoing burn. A pilot platform team clones
|
||
the reference, runs the same apply, and leaves it running — multiply
|
||
the spend by the number of clones. AWS budget alerts fire at leadership
|
||
level. The reference is perceived as expensive.
|
||
|
||
**Mitigation (user-owned):**
|
||
- **D-096 (teardown mandatory before milestone COMPLETE).** Phase 61
|
||
tears down the stacks via D-070 decommission mode. The live AWS
|
||
account returns to zero-cost steady state. The milestone does not
|
||
complete until teardown is verified.
|
||
- **COST.md teardown guidance.** REQ-119 documents the teardown path +
|
||
cost-ceiling guidance for downstream clones. A clone that follows
|
||
the guidance runs the same teardown.
|
||
- The user enforces D-096 at Phase 61 — no merge to main until
|
||
`terraform show` confirms no resources. The `decommissioned:
|
||
{ stack, cr_id, completed_at }` record in the `---ci---` block is
|
||
the audit trail.
|
||
|
||
#### FM-3 — Deck overstates capability (a future v1.9.x-style incident)
|
||
|
||
**How it kills the project:** a future NFR-patch milestone adds a deck
|
||
slide claiming a capability that hasn't been re-verified. The
|
||
regression gate runs at milestone COMPLETE and catches the underlying
|
||
decay — but the deck has already been rendered and uploaded to a
|
||
release. Leadership sees the deck before the regression gate fails.
|
||
Repeat of the v1.9.x sequencing incident.
|
||
|
||
**Mitigation (user-owned):**
|
||
- **Verified-only claims.** REQ-121 enforces that decks match
|
||
`CAPABILITY_INVENTORY.md` exactly; `ci-doc-verifier` confirms no
|
||
stale claims. Any deck claim must trace to a Verified capability.
|
||
- **Decks unfrozen only after re-verification.** The v1.10 lesson
|
||
(D-094) is codified: decks are frozen until the regression gate
|
||
passes. A future milestone that adds a deck slide must land the
|
||
capability re-verification in the same milestone.
|
||
- The user reviews the `ci-doc-verifier` output at each milestone
|
||
COMPLETE. If a stale claim is found, the milestone does not complete
|
||
until the deck is corrected.
|
||
|
||
#### FM-4 — Pilot consumer hits a contract gap
|
||
|
||
**How it kills the project:** a pilot platform team (post-pitch) clones
|
||
the reference and tries to deploy a stack the L2 catalog doesn't cover
|
||
(e.g. a worker queue, a scheduled job, a database-backed service). The
|
||
contract schema + L2 compositions support only microservice + static-
|
||
assets. The pilot team concludes the reference is a demo, not a
|
||
foundation. The pitch's "feature-complete MVP" claim (G-001) is
|
||
undermined.
|
||
|
||
**Mitigation (user-owned):**
|
||
- **CONSUMER_GUIDE.md + L2 catalog coverage.** `docs/CONSUMER_GUIDE.md`
|
||
documents the supported L2 compositions; the L2 catalog
|
||
(`modules/l2/`) is the supported surface. A pilot team that reads the
|
||
guide knows the boundary before cloning.
|
||
- **Honest scope.** The grill (G-010) accepted OSS scope as
|
||
contributor-bounded. The pitch should not claim "any stack" — it
|
||
should claim "microservice + static-assets today; the L2 pattern is
|
||
extensible." The v1.9.5 Anti-goals slide (What This Platform Is —
|
||
and Isn't) is the honest framing.
|
||
- The user adds L2 compositions as pilot demand surfaces. The reference
|
||
value is the *shape* (contract → IR → adapter → terraform →
|
||
confidence → outbox), not the catalog size. A pilot team that
|
||
understands the shape can extend it.
|
||
|
||
### What the pre-mortem tells us
|
||
|
||
The four failure modes all reduce to the same root pattern: **a claim
|
||
outruns the verification that backs it.** v1.10 was the first instance
|
||
(decks outran capability). v1.11 closes G-005 + G-008 by making the
|
||
verification back the claim. The mitigations are all structural —
|
||
regression-testable baselines, mandatory teardown, Verified-only deck
|
||
claims, honest scope — not procedural. The user owns enforcement at
|
||
each milestone COMPLETE.
|
||
|
||
### Confidence
|
||
|
||
- FM-1 (IAM drift recurs): confidence 0.75 — the baseline test catches
|
||
it; the user enforces re-bootstrap at COMPLETE.
|
||
- FM-2 (cost spike): confidence 0.85 — D-096 teardown is mandatory and
|
||
audited in the `---ci---` block.
|
||
- FM-3 (deck overstates): confidence 0.70 — `ci-doc-verifier` is
|
||
automated; the sequencing risk is procedural.
|
||
- FM-4 (pilot contract gap): confidence 0.65 — the mitigation is
|
||
honest framing, not catalog completeness; a pilot may still hit the
|
||
gap.
|
||
|
||
### Links to existing controls
|
||
|
||
- D-091 regression gate (REQ-112) — `scripts/run_regression.sh`.
|
||
- D-094 verified-reality rewrite (REQ-115) — decks match
|
||
`CAPABILITY_INVENTORY.md`.
|
||
- D-096 teardown mandatory (v1.11) — Phase 61.
|
||
- G-005 deploy verification (v1.11) — Phases 56–58.
|
||
- G-008 cost documentation (v1.11) — Phase 59.
|
||
- G-010 contributor-bounded scope — honest pitch framing. |