feat(P64): pre-mortem — v1.10 post-mortem + forward pre-mortem
Cherry-picked from e868a16 (branch phase/60-pre-mortem-doc) and finalized for P64. The pre-mortem was originally authored at P60 but the roadmap assigns it to P64 (REQ-120). Part 1 — v1.10 decay incident post-mortem: - Root cause: VERIFY was diff-scoped — checked the phase diff only, never re-ran underlying capability. 8 NFR-patch phases (v1.9.1– v1.9.8) passed VERIFY while the platform decayed. - Mitigations landed in v1.10: D-091 regression gate, D-092 local emulators, D-093 capability inventory, D-094 verified-reality rewrite. Part 2 — forward pre-mortem (OSS reference + leadership pitch): - FM-1 IAM drift recurs → IAM_POLICY.md baseline test (REQ-116) - FM-2 cost spike from un-torn-down stacks → D-096 teardown mandatory - FM-3 deck overstates capability → Verified-only claims (REQ-121) - FM-4 pilot consumer hits a contract gap → honest scope (G-010) ---ci--- project: acdl phase: P64 milestone: v1.11 status: execute ---/ci---
This commit is contained in:
Jon Chery
@@ -0,0 +1,229 @@
|
|||||||
|
# ACDL — Pre-mortem (v1.11, REQ-120)
|
||||||
|
|
||||||
|
> Authored: 2026-07-28, Phase 64 (previously drafted at P60, finalized here).
|
||||||
|
> Mandated by: GRILL Axis 7 Q4 (no pre-mortem on file — flagged, no
|
||||||
|
> binding decision; user accepted autonomous governance in G-009).
|
||||||
|
> Structure: (1) v1.10 decay incident post-mortem, (2) forward pre-mortem
|
||||||
|
> for the OSS reference + leadership pitch.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 1 — Post-mortem: v1.10 capability decay incident
|
||||||
|
|
||||||
|
### Summary
|
||||||
|
|
||||||
|
Capabilities marked complete in v1.1–v1.8 ran successfully at the time
|
||||||
|
of tagging. As of 2026-07-27 they were **not reproducible** — the v1.7/
|
||||||
|
v1.8 platform simplification introduced 7 adapter defects in
|
||||||
|
`adapters/terraform/adapter.py` that prevented `terraform init/
|
||||||
|
validate/plan` from succeeding against live AWS. The decks (v1.9.1–
|
||||||
|
v1.9.8) presented the capability as current across 8 NFR-patch phases
|
||||||
|
**without disclosing the decay**. v1.10 (Phases 52–55) re-verified every
|
||||||
|
advertised capability, fixed all 7 defects in-sweep (D-090: no cap), and
|
||||||
|
rewrote PROJECT/ROADMAP/decks to match verified reality.
|
||||||
|
|
||||||
|
### Timeline
|
||||||
|
|
||||||
|
| Date | Event |
|
||||||
|
|------|-------|
|
||||||
|
| 2026-07-21 | v1.7 Phases 22–27 ship. The adapter simplification lands (the 7 defects are introduced here). |
|
||||||
|
| 2026-07-21 | v1.8 Phases 28–38 ship. The defects persist undetected; VERIFY is diff-scoped so the decay is invisible. |
|
||||||
|
| 2026-07-21 → 2026-07-27 | v1.9.0 + v1.9.1–v1.9.8 (8 NFR-patch phases) ship. Each passes VERIFY (diff-scoped — checks the phase diff only, never re-runs underlying capability). Decks present capability as current. |
|
||||||
|
| 2026-07-27 | CLARIFY/RESEARCH for v1.10 surfaces the structural defect: VERIFY is diff-scoped; advertised capability is not reproducible; deck work was sequenced backwards. |
|
||||||
|
| 2026-07-27 | User decisions D-090 (no cap on sweep), D-091 (regression-class VERIFY), D-092 (local emulating adapters), D-093 (re-verify v1.1→v1.8), D-094 (rewrite to verified reality). |
|
||||||
|
| 2026-07-27 | Phase 52 adds the regression-class VERIFY. Phase 53 builds local emulating adapters. Phase 54 enumerates + re-verifies every capability — finds 7 adapter defects, fixes all in-sweep. Phase 55 rewrites PROJECT/ROADMAP/decks to verified reality. |
|
||||||
|
| 2026-07-27 | v1.10.0 tagged; all 16 auto-verifiable capabilities Verified. 6 IAM-gated capabilities (CAP-017..022) escalated (G-005). |
|
||||||
|
|
||||||
|
### Root cause
|
||||||
|
|
||||||
|
**VERIFY was diff-scoped.** The standard VERIFY stage checked the phase
|
||||||
|
diff only — the files changed in that phase — and never re-ran the
|
||||||
|
underlying platform capability. 8 NFR-patch phases (v1.9.1→v1.9.8)
|
||||||
|
passed VERIFY while the platform decayed underneath, because each
|
||||||
|
phase's diff was docs-only (decks) and the decay was in code the diff
|
||||||
|
didn't touch. The VERIFY gate was structurally incapable of catching
|
||||||
|
decay in code outside the phase diff.
|
||||||
|
|
||||||
|
### Contributing factors
|
||||||
|
|
||||||
|
1. **Deck work was sequenced backwards.** The honest order is
|
||||||
|
re-verify → rewrite → polish. v1.9.x did it backwards: polish the
|
||||||
|
decks first, then discover (in v1.10) that the capability they
|
||||||
|
advertised had decayed.
|
||||||
|
2. **No regression-class gate existed.** Each milestone's VERIFY
|
||||||
|
re-checked the phase diff, not the cumulative capability. There was
|
||||||
|
no mechanism to ask "does everything we previously claimed still
|
||||||
|
work?"
|
||||||
|
3. **Local emulating adapters did not exist.** Without a local tier,
|
||||||
|
re-verification required live AWS access on every phase — costly and
|
||||||
|
not run. The decay was therefore never re-probed between v1.7 and
|
||||||
|
v1.10.
|
||||||
|
4. **Decks were frozen before re-verification.** The v1.9.x decks
|
||||||
|
presented capability as current without a re-verification step
|
||||||
|
gating the claim.
|
||||||
|
|
||||||
|
### Impact
|
||||||
|
|
||||||
|
- **8 phases of inaccurate status reporting.** v1.9.1–v1.9.8 decks
|
||||||
|
advertised capability as current that was not reproducible.
|
||||||
|
- **7 adapter defects shipped undetected.** Duplicate output
|
||||||
|
definitions, duplicate args, missing required args, deprecated AWS
|
||||||
|
provider v5 arg names — all in `adapters/terraform/adapter.py`.
|
||||||
|
- **Credibility gap.** The OSS reference's headline E2E did not run
|
||||||
|
against live AWS between v1.7 and v1.10. The grill (G-005) flagged
|
||||||
|
this as the project-killing risk.
|
||||||
|
|
||||||
|
### Mitigations (landed in v1.10)
|
||||||
|
|
||||||
|
| Mitigation | Decision | Status |
|
||||||
|
|-----------|----------|--------|
|
||||||
|
| Regression-class VERIFY that re-runs capability checks at milestone completion | D-091 (REQ-112) | Landed — `scripts/run_regression.sh` + `core/regression_verify.py`. 16/16 Verified at v1.10.0. |
|
||||||
|
| Local emulating adapters so the platform is fully locally testable without cloud credentials | D-092 (REQ-113) | Landed — flat-file DynamoDB outbox, local ECS Fargate emulator, local S3 state, local Lambda stub. Headline E2E runs locally. |
|
||||||
|
| Capability inventory with per-capability Verified/Decayed/Broken tags | D-093 (REQ-114) | Landed — `.ciagent/CAPABILITY_INVENTORY.md`. 16/16 Verified; 6 IAM-gated escalated (G-005). |
|
||||||
|
| Rewrite docs/decks to verified reality; decks unfrozen only after re-verification | D-094 (REQ-115) | Landed — PROJECT.md §Capability Status (Re-Verified 2026-07-27), ROADMAP v1.9.x noted as superseded-by-reverification, both decks rewritten. |
|
||||||
|
|
||||||
|
### Follow-up (accepted debt)
|
||||||
|
|
||||||
|
- **G-007 (per-phase regression):** the regression gate runs at
|
||||||
|
milestone completion, not per-phase. Inter-milestone decay between
|
||||||
|
phase N and milestone COMPLETE is an accepted trade-off (grill Axis 3
|
||||||
|
Q4, confidence 0.70). Per-phase regression hardening is a separate
|
||||||
|
future milestone.
|
||||||
|
- **G-005 (IAM-gated capabilities):** 6 capabilities (CAP-017..022)
|
||||||
|
remain deploy-unverified as of v1.10 — the spike-runner cannot fix
|
||||||
|
its own IAM. v1.11 (this milestone) closes G-005 by re-bootstrapping
|
||||||
|
IAM and live-deploying the stacks.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part 2 — Forward pre-mortem: OSS reference + leadership pitch
|
||||||
|
|
||||||
|
### Scenario
|
||||||
|
|
||||||
|
It is 90 days after the v1.11 ship. The leadership pitch has been
|
||||||
|
delivered. The grill's 90-day conditions (G-001 pitch yields a pilot
|
||||||
|
platform team; G-005 deploy path verifiable; G-008 cost operating model
|
||||||
|
documented) were the success criteria. **Assume the project has failed.**
|
||||||
|
What killed it?
|
||||||
|
|
||||||
|
### Top failure modes + mitigations
|
||||||
|
|
||||||
|
#### FM-1 — IAM drift recurs (the spike-runner loses permissions again)
|
||||||
|
|
||||||
|
**How it kills the project:** the v1.11 IAM re-bootstrap grants are
|
||||||
|
revoked or drift (admin action, account re-organization, SCP change).
|
||||||
|
The next regression run (D-091) fails closed on CAP-017..022. The
|
||||||
|
verified-reality claim in the decks becomes false again — a repeat of
|
||||||
|
the v1.10 incident in a different shape. Leadership loses trust.
|
||||||
|
|
||||||
|
**Mitigation (user-owned):**
|
||||||
|
- The IAM policy baseline is now regression-tested
|
||||||
|
(`tests/test_iam_policy_baseline.py`, REQ-116). Any permission removal
|
||||||
|
surfaces as a test failure at the next milestone COMPLETE — the gate
|
||||||
|
fails closed, the false claim never ships.
|
||||||
|
- `.ciagent/IAM_POLICY.md` documents the required grants. An admin who
|
||||||
|
re-organizes the account can read the baseline and re-grant.
|
||||||
|
- The user reviews the baseline test at each milestone COMPLETE. If the
|
||||||
|
grants have drifted, the user re-bootstraps (D-095 path) before
|
||||||
|
re-attempting COMPLETE.
|
||||||
|
|
||||||
|
#### FM-2 — Cost spike from un-torn-down stacks
|
||||||
|
|
||||||
|
**How it kills the project:** the v1.11 deploy-verification leaves the
|
||||||
|
microservice + static-assets + uptime stacks running. Live ECS Fargate +
|
||||||
|
CloudFront + WAF accrue spend. The COST.md (REQ-119) documents the
|
||||||
|
v1.0–v1.10 window, not the ongoing burn. A pilot platform team clones
|
||||||
|
the reference, runs the same apply, and leaves it running — multiply
|
||||||
|
the spend by the number of clones. AWS budget alerts fire at leadership
|
||||||
|
level. The reference is perceived as expensive.
|
||||||
|
|
||||||
|
**Mitigation (user-owned):**
|
||||||
|
- **D-096 (teardown mandatory before milestone COMPLETE).** Phase 61
|
||||||
|
tears down the stacks via D-070 decommission mode. The live AWS
|
||||||
|
account returns to zero-cost steady state. The milestone does not
|
||||||
|
complete until teardown is verified.
|
||||||
|
- **COST.md teardown guidance.** REQ-119 documents the teardown path +
|
||||||
|
cost-ceiling guidance for downstream clones. A clone that follows
|
||||||
|
the guidance runs the same teardown.
|
||||||
|
- The user enforces D-096 at Phase 61 — no merge to main until
|
||||||
|
`terraform show` confirms no resources. The `decommissioned:
|
||||||
|
{ stack, cr_id, completed_at }` record in the `---ci---` block is
|
||||||
|
the audit trail.
|
||||||
|
|
||||||
|
#### FM-3 — Deck overstates capability (a future v1.9.x-style incident)
|
||||||
|
|
||||||
|
**How it kills the project:** a future NFR-patch milestone adds a deck
|
||||||
|
slide claiming a capability that hasn't been re-verified. The
|
||||||
|
regression gate runs at milestone COMPLETE and catches the underlying
|
||||||
|
decay — but the deck has already been rendered and uploaded to a
|
||||||
|
release. Leadership sees the deck before the regression gate fails.
|
||||||
|
Repeat of the v1.9.x sequencing incident.
|
||||||
|
|
||||||
|
**Mitigation (user-owned):**
|
||||||
|
- **Verified-only claims.** REQ-121 enforces that decks match
|
||||||
|
`CAPABILITY_INVENTORY.md` exactly; `ci-doc-verifier` confirms no
|
||||||
|
stale claims. Any deck claim must trace to a Verified capability.
|
||||||
|
- **Decks unfrozen only after re-verification.** The v1.10 lesson
|
||||||
|
(D-094) is codified: decks are frozen until the regression gate
|
||||||
|
passes. A future milestone that adds a deck slide must land the
|
||||||
|
capability re-verification in the same milestone.
|
||||||
|
- The user reviews the `ci-doc-verifier` output at each milestone
|
||||||
|
COMPLETE. If a stale claim is found, the milestone does not complete
|
||||||
|
until the deck is corrected.
|
||||||
|
|
||||||
|
#### FM-4 — Pilot consumer hits a contract gap
|
||||||
|
|
||||||
|
**How it kills the project:** a pilot platform team (post-pitch) clones
|
||||||
|
the reference and tries to deploy a stack the L2 catalog doesn't cover
|
||||||
|
(e.g. a worker queue, a scheduled job, a database-backed service). The
|
||||||
|
contract schema + L2 compositions support only microservice + static-
|
||||||
|
assets. The pilot team concludes the reference is a demo, not a
|
||||||
|
foundation. The pitch's "feature-complete MVP" claim (G-001) is
|
||||||
|
undermined.
|
||||||
|
|
||||||
|
**Mitigation (user-owned):**
|
||||||
|
- **CONSUMER_GUIDE.md + L2 catalog coverage.** `docs/CONSUMER_GUIDE.md`
|
||||||
|
documents the supported L2 compositions; the L2 catalog
|
||||||
|
(`modules/l2/`) is the supported surface. A pilot team that reads the
|
||||||
|
guide knows the boundary before cloning.
|
||||||
|
- **Honest scope.** The grill (G-010) accepted OSS scope as
|
||||||
|
contributor-bounded. The pitch should not claim "any stack" — it
|
||||||
|
should claim "microservice + static-assets today; the L2 pattern is
|
||||||
|
extensible." The v1.9.5 Anti-goals slide (What This Platform Is —
|
||||||
|
and Isn't) is the honest framing.
|
||||||
|
- The user adds L2 compositions as pilot demand surfaces. The reference
|
||||||
|
value is the *shape* (contract → IR → adapter → terraform →
|
||||||
|
confidence → outbox), not the catalog size. A pilot team that
|
||||||
|
understands the shape can extend it.
|
||||||
|
|
||||||
|
### What the pre-mortem tells us
|
||||||
|
|
||||||
|
The four failure modes all reduce to the same root pattern: **a claim
|
||||||
|
outruns the verification that backs it.** v1.10 was the first instance
|
||||||
|
(decks outran capability). v1.11 closes G-005 + G-008 by making the
|
||||||
|
verification back the claim. The mitigations are all structural —
|
||||||
|
regression-testable baselines, mandatory teardown, Verified-only deck
|
||||||
|
claims, honest scope — not procedural. The user owns enforcement at
|
||||||
|
each milestone COMPLETE.
|
||||||
|
|
||||||
|
### Confidence
|
||||||
|
|
||||||
|
- FM-1 (IAM drift recurs): confidence 0.75 — the baseline test catches
|
||||||
|
it; the user enforces re-bootstrap at COMPLETE.
|
||||||
|
- FM-2 (cost spike): confidence 0.85 — D-096 teardown is mandatory and
|
||||||
|
audited in the `---ci---` block.
|
||||||
|
- FM-3 (deck overstates): confidence 0.70 — `ci-doc-verifier` is
|
||||||
|
automated; the sequencing risk is procedural.
|
||||||
|
- FM-4 (pilot contract gap): confidence 0.65 — the mitigation is
|
||||||
|
honest framing, not catalog completeness; a pilot may still hit the
|
||||||
|
gap.
|
||||||
|
|
||||||
|
### Links to existing controls
|
||||||
|
|
||||||
|
- D-091 regression gate (REQ-112) — `scripts/run_regression.sh`.
|
||||||
|
- D-094 verified-reality rewrite (REQ-115) — decks match
|
||||||
|
`CAPABILITY_INVENTORY.md`.
|
||||||
|
- D-096 teardown mandatory (v1.11) — Phase 61.
|
||||||
|
- G-005 deploy verification (v1.11) — Phases 56–58.
|
||||||
|
- G-008 cost documentation (v1.11) — Phase 59.
|
||||||
|
- G-010 contributor-bounded scope — honest pitch framing.
|
||||||
Reference in New Issue
Block a user