docs(P04): complete operator-guide-reference-tracking phase (REQ-OPS-GUIDE, v1.28.4)

---ci---
project: acdl
phase: 4
milestone: v1.29
status: complete
---/ci---

docs/operator-guide-platform-ops.md

@@ -0,0 +1,747 @@
+# Operator Guide — Nova Platform Ops (`nova-platform-ops`)
+
+> **REQ-OPS-GUIDE** — the operator-facing runbook for the
+> `nova-platform-ops` Terraform repo. This is the verification surface
+> for the covered-reference REQs (355-366, 371): their cutover gates
+> (M1/M1.5/M2) are documented in §18 below, and each REQ has a
+> **"Result" column** that the operator fills in after running the gate.
+> P6 audit verifies every covered-reference REQ has a non-empty, green
+> Result (grill CF-2/G-5). **HARD P6 ship gate:** §3 contains the
+> operator-attested "M1.5 Verification Gate Result" row (grill
+> CF-1/G-2.1) — the milestone does not ship until that row is filled.
+>
+> Audience: platform operators / SREs running the live Nova platform in
+> AWS account `581513795199`. For the developer auth flows, see
+> `docs/developer-guide-auth.md`; for the legacy CloudFormation path,
+> see `docs/archive/nova-idp-cfn-v1.28.md`.
+
+## 1. Overview + reposplit rationale
+
+Nova's platform operations live in a dedicated, Gitea-private Terraform
+repository — `nova-platform-ops` — separate from the engineering repo
+`acdl/acdl`. The split is grounded in Vision §4 (Domain Boundaries):
+
+> *The platform begins where the artifact is compiled and ends where it
+> runs in production under operational guardrails.*
+
+That is two distinct disciplines with two distinct ownership surfaces:
+
+| Discipline | Ends | Begins | Repo | Surface |
+|------------|------|--------|------|---------|
+| Engineering | at the compiled artifact | — | `acdl/acdl` (GitHub) | `publish.yml` + GitHub Releases |
+| Operations | — | at the live platform under guardrails | `nova-platform-ops` (Gitea-private) | Terraform modules |
+
+**Scope split (CLARIFY Q-P1, D-232):**
+
+- `acdl/acdl` authors `publish.yml` (the artifact publish pipeline) +
+  the artifacts themselves (Lambda zip, layer wheel, Python wheel, ECR
+  container image with the static `kj` binary). Each tag `v1.29.x`
+  produces a GitHub Release with SHA-256-verified artifacts (REQ-354).
+- `nova-platform-ops` authors the Terraform modules
+  (`networking`/`kms`/`identity`/`contract-ingest`/`bootstrap`/`edge`)
+  that bring those artifacts live in `581513795199`.
+
+The handoff between the two repos is the **tag-pin** (D-235):
+`nova-platform-ops` declares `local.nova_platform_version` +
+`local.kj_source_sha` and resolves substrates through a single
+`data.aws_ecr_image.kj_image`. The engineering repo never knows which
+tag is live; the ops repo never authors artifacts. Vision §6
+immutability + Vision §5 narrow interfaces.
+
+The covered-reference REQs (355-366, 371) are authored in
+`nova-platform-ops` (out-of-band). CIAgent in `acdl` tracks them for
+milestone completeness; their verification surface is the cutover
+gates in §18 of this guide.
+
+## 2. Day-0 cutover procedure (M1)
+
+The M1 cutover is the one-time conversion of the live AWS account
+`581513795199` from CloudFormation-managed (or manually-created)
+resources to Terraform-managed resources in `nova-platform-ops`. It is
+conditional on the M1.5 verification gate passing (§3, Q7 carry-forward,
+D-236).
+
+The 10-step Journey 2 (spec §3.2):
+
+1. **Create `nova-platform-ops` in Gitea** — private (`private: true`,
+   OPER-PRIV, REQ-359), no GitHub mirror. The repo is operator-owned;
+   CIAgent has no presence there.
+
+2. **Commit the initial Terraform structure** — the module tree
+   (`networking`/`kms`/`identity`/`contract-ingest`/`bootstrap`/`edge`)
+   + `importable-resources.tf` (§12) + `versions.tf` + `backend.tf`
+   (S3 state in the imported bucket).
+
+3. **`terraform init`** — initialize the S3 backend against the
+   state bucket (`nova-tfstate-581513795199-us-east-1`, imported in
+   step 5). The bucket is created manually once (operator's secure
+   scratch, spec §7.1, D-235) before Terraform adopts it.
+
+4. **`terraform import` for existing live resources** — adopt the
+   resources that already exist in `581513795199` into Terraform state
+   without recreating them. The import map is in
+   `importable-resources.tf` (§12):
+   - `aws_s3_bucket.nova_tfstate` ← `nova-tfstate-581513795199-us-east-1`
+   - `aws_dynamodb_table.nova_contracts` ← `nova-contracts`
+   - `aws_dynamodb_table.nova_change_requests` ← `nova-change-requests`
+   - `aws_dynamodb_table.nova_outbox` ← `nova-outbox`
+   - `aws_iam_role.acdl_act_runner` ← `acdl-act-runner-role`
+   - per-stack CMKs (KMS keys)
+
+   Re-import exits non-zero with `resource_already_imported`
+   (IMPORT-IDEMPOTENT, REQ-361). CI import treats this as idempotent
+   success — the import workflow greps the error stream + exits 0 on
+   that string.
+
+5. **(state bucket is imported in step 4)** — listed here for sequence
+   clarity; the S3 state bucket is the first import because the
+   backend depends on it.
+
+6. **Add new resources** that do not yet exist in the account:
+   - KMS alias `alias/nova-oidc-signing` (§9, D-234).
+   - Identity DynamoDB tables: `nova-users`, `nova-sessions`,
+     `nova-pats` (§11).
+   - JWKS Function URL with `AuthType: AWS_IAM` (NOT `NONE` — §10,
+     INV-18, RESEARCH §4 critical pitfall).
+   - CloudFront distribution + OAC + WAF WebACL + ACM certificate +
+     Route53 alias (§14, REQ-364/365/366).
+
+7. **`terraform plan`** — expect zero diff on the imported resources
+   (they are already in their desired state) + a pure-add diff on the
+   new resources. If the plan shows a diff on an imported resource,
+   the import map or the Terraform resource block is wrong — fix
+   before apply. **DRIFT-DETECT (REQ-356):** `terraform plan` exit 2
+   (drift) fails the apply workflow; manual reconciliation required.
+
+8. **HITL approval** — `terraform apply` against `main` requires a
+   Gitea Actions approval from a user **distinct from the PR author**
+   (TFM-HITL, REQ-357, INV-3). Self-approval is rejected:
+   `gitea.triggering_actor == pull_request.user.login` → apply fails
+   closed (M1.5 item 11).
+
+9. **`terraform apply`** — on approval, the apply creates the new
+   resources + adopts the imported ones. Smoke test (step 10) before
+   declaring M1 done.
+
+10. **Smoke test + CFN→Terraform conversion** — verify the live
+    account is in the desired state (JWKS reachable via CloudFront,
+    KMS round-trip, ABAC fail-closed). The CFN template in
+    `acdl/acdl/nova/idp/setup.py` is archived to
+    `docs/archive/nova-idp-cfn-v1.28.md` as read-only reference
+    (REQ-369); the active path is now `terraform apply` in
+    `nova-platform-ops`.
+
+## 3. M1.5 verification gate (12-item spike)
+
+The M1.5 gate is the 12-item spike from PLAN.md "Happy Path" (spec
+§3.3 Edge 5 items 1-8 + grill CF-1 items 9-12). **3 consecutive green
+rebuilds are required** in `nova-platform-ops` CI.
+
+The 12 items:
+
+1. `kj` v0.0.3 (pinned SHA in `platform/abac/kj-version.txt`) compiles
+   with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64`.
+2. Resulting binary reports `file kj → ELF 64-bit LSB executable,
+   x86-64, statically linked, no shared library` (KJ-STATIC).
+3. Container image built from
+   `public.ecr.aws/lambda/python:3.12-al2023` with the binary copied
+   to `/opt/kj/kj`, `chmod 0555`, owned by `sbx_user:1051`.
+4. Lambda runtime `python3.12` executes
+   `nova_idp_token_vend.handler`; the handler invokes
+   `subprocess.run(['/opt/kj/kj', 'apply', ...])` and parses stdout
+   JSON.
+5. `tests/test_idp_auth.py` passes against the live image in moto-DDB.
+6. `tests/test_kms_roundtrip.py` passes against the live KMS key
+   `alias/nova-oidc-signing` (REQ-362 path — covered-reference).
+7. End-to-end: known PAT → known ABAC-allowed action → signed OIDC
+   token → `jose` verification → green. Known PAT + ABAC-denied action
+   → 403 with deny reason logged (INV-17 fail-closed).
+8. Image URI is recorded in Terraform state and in this operator
+   guide (§18, REQ-371 Result row).
+9. **(grill CF-1) JWKS-EDGE-ONLY:** direct JWKS Function URL GET
+   (bypassing CloudFront) returns **403**; via-CloudFront GET returns
+   **200** (INV-18). Proves `AuthType: AWS_IAM` + OAC pinning.
+10. **(grill CF-1) IAM-NARROW:** `aws iam get-role-policy` on the
+    Gitea OIDC role asserts no `Action: "*"` and no `Resource: "*"`
+    (REQ-360).
+11. **(grill CF-1) TFM-HITL:** a `terraform apply`
+    `workflow_dispatch` triggered by the PR author is **rejected**
+    (exit non-zero); a dispatch by a distinct user proceeds (REQ-357).
+12. **(grill CF-1) rollback drill:** revert `nova_platform_version`
+    pin → `terraform apply` → assert the prior ECR digest runs
+    (D-236, guards against ECR tag mutability).
+
+If items 1-7 fail three consecutive rebuilds, M2a activates (§5,
+REQ-363b Fargate toggle) with the same ECR image — no warmup hit
+because the standby is always running the same digest (KJ-LOCKSTEP).
+
+### HARD P6 ship gate (grill CF-1/G-2.1)
+
+P6 must not ship `v1.28.6` until the operator attests the M1.5 result
+in the row below. The operator fills this in **after** the gate passes
+3 consecutive green rebuilds in `nova-platform-ops` CI. P6 audit
+verifies the row exists + is non-empty.
+
+#### M1.5 Verification Gate Result
+
+| Rebuild # | Run ID / commit SHA | All 12 items green? | Attestor identity | Attested at (UTC) |
+|-----------|---------------------|---------------------|-------------------|-------------------|
+| 1 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
+| 2 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
+| 3 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
+
+> **P6 audit rule:** all three rows must be present, all 12 items
+> green on each, the three run IDs/SHAs distinct (consecutive
+> rebuilds, not one run copied thrice), and the attestor identity
+> non-empty. Empty or red → P6 blocks → escalate.
+
+## 4. M2 operational handoff loop
+
+M2 is the steady-state operational loop for rolling out an engineering
+change after M1.5 is green. The loop is the tag-pin bump → plan →
+HITL → apply cycle (D-235, D-238).
+
+1. **Tag-pin bump** — the operator opens a PR in `nova-platform-ops`
+   bumping `local.nova_platform_version` (e.g. `v1.29.3` → `v1.29.4`)
+   + `local.kj_source_sha` (the `kj` source SHA from
+   `platform/abac/kj-version.txt` at the new tag). Both pins move
+   together — there is one ECR image identity (KJ-LOCKSTEP, REQ-371).
+
+2. **`terraform plan`** — CI runs `terraform plan` on the PR. The
+   KJ-LOCKSTEP precondition (a `lifecycle.precondition` on both
+   image-bearing resources — the Lambda `image_uri` and the Fargate
+   task `container_definitions[0].image`) checks that both
+   `image_uri` attributes resolve to the **same ECR digest** via
+   `data.aws_ecr_image.kj_image`. If the two diverge, the plan fails
+   closed — no second pipeline, no second SHA pin (D-238).
+
+3. **HITL approval** — a Gitea Actions approver **distinct from the
+   PR author** approves the apply (TFM-HITL, REQ-357). Self-approval
+   is rejected (M1.5 item 11).
+
+4. **`terraform apply`** — on approval, the apply updates both
+   `aws_lambda_function.nova_idp_token_vend.image_uri` and
+   `aws_ecs_task_definition.kj.container_definitions[0].image` to the
+   same ECR digest. The Lambda image + the Fargate task redeploy to
+   the same digest in one apply. Zero diff on KMS, DDB, IAM, edge
+   (the only change is the image reference).
+
+**Verification:** after the apply, `aws lambda get-function
+--function-name nova-idp-token-vend --query Configuration.Code.ImageUri`
+and `aws ecs describe-tasks` on the Fargate task both report the same
+digest. This is the M2 acceptance gate (PLAN §UX Acceptance Criteria
+3) + the REQ-371 Result row in §18.
+
+## 5. M2a Fargate activation (conditional)
+
+M2a activates **only if M1.5 fails 3 consecutive rebuilds** (D-236).
+It is the REQ-363b Fargate toggle — an always-warm minimal Fargate
+standby running the **same ECR image** as the Lambda (KJ-LOCKSTEP).
+
+Because the standby is always running the same digest as the Lambda,
+activating M2a is **not** a warmup hit — the standby is already
+serving `GET /health → 200` every 10s (KJ-WARMUP-HEALTH, §15). The
+toggle repoints token-vend traffic from the Lambda to the Fargate
+task; no cold start, no image pull.
+
+If both the Lambda path and the Fargate path fail (M1.5 items 1-7
+fail on both substrates), the operator escalates — Nova-idp ships in
+read-only partial mode (no token issuance) until `kj` is verified
+(Q7 carry-forward, spec §7.7).
+
+**Fargate sunset discipline (D-237):** the standby (~$15-20/month,
+§7) may not be deleted unless REQ-363 has been green in production
+for **≥30 consecutive days**. Sunset requires an architecture review.
+See §15 for the health-check procedure.
+
+## 6. Rollback procedure (D-236)
+
+Rollback is a tag-pin revert — the same mechanism as the M2 rollout
+(§4), in reverse.
+
+1. **Revert `nova_platform_version`** in `nova-platform-ops` to the
+   prior tag (e.g. `v1.29.4` → `v1.29.3`). Open a PR, get HITL
+   approval (TFM-HITL, same as rollout).
+
+2. **`terraform apply`** — the apply reverts both the Lambda
+   `image_uri` and the Fargate task `image` to the prior ECR digest.
+   The prior tag's artifacts remain downloadable (GitHub Releases are
+   append-only per tag, REQ-354 AC 2) — no artifact is re-built.
+
+3. **Verify** the prior digest is running:
+   ```sh
+   aws lambda get-function --function-name nova-idp-token-vend \
+     --query Configuration.Code.ImageUri --output text
+   # → <account>.dkr.ecr.us-east-1.amazonaws.com/nova-kj@sha256:<prior-digest>
+   ```
+
+This is the M1.5 item 12 rollback drill + the operational rollback
+procedure. It guards against ECR tag mutability (RESEARCH §2) — the
+digest is immutable even if a tag is re-pushed.
+
+## 7. Cost section
+
+Monthly estimate for the `nova-platform-ops` live platform in account
+`581513795199` (pilot volume):
+
+| Resource | Quantity | Est. monthly | Notes |
+|----------|----------|-------------|-------|
+| WAF WebACL (CloudFront-scoped) | 1 | ~$5-10 | + per-request; REQ-365 |
+| Fargate standby (0.25 vCPU, 512 MB) | 1 task | ~$15-20 | REQ-363b AC 4; largest line item |
+| KMS asymmetric key | 1 | ~$1 | `alias/nova-oidc-signing`, ECC_NIST_P256 |
+| DynamoDB (on-demand, 6 tables) | 6 | ~$2 | §11 tables |
+| Lambda invocations (3 Lambdas) | 3 | ~$2 | low pilot volume |
+| ECR image storage | ~100 MB | <$1 | the `kj` image |
+| S3 state bucket + access logs | 1 | <$1 | `nova-tfstate-*` |
+| CloudFront + ACM + Route53 | 1 distribution | ~$1 | ACM free for CloudFront-attached |
+| **Total** | | **~$30-40/month** | |
+
+**Fargate standby is the largest line item** (~$15-20/month, REQ-363b
+AC 4). It is explicitly documented here with the D-237 sunset
+discipline (§5, §15): ≥30 consecutive days green before deletion +
+architecture review. Do not delete the standby to save ~$15/month
+without that review — it is the defensive fallback for the `kj`
+substrate.
+
+## 8. Artifact-mirror fallback (Edge 6)
+
+When the Gitea `act_runner` in `nova-platform-ops` CI cannot reach
+GitHub Releases (network partition, egress restriction, GitHub
+outage), the operator mirrors the artifact bundle locally by SHA-256.
+
+**Procedure:**
+
+1. **Download the GitHub Release bundle** for the target tag
+   (`v1.29.x`) from a machine that can reach GitHub Releases:
+   ```sh
+   gh release download v1.29.0 \
+     --repo continuous-intelligence/acdl \
+     --pattern 'nova-lambda-token-vend-*.zip' \
+     --pattern 'nova-cli-layer-*.zip' \
+     --pattern 'nova-*-py3-none-any.whl' \
+     --dir ./artifact-cache
+   ```
+
+2. **Verify SHA-256** against the release body (each artifact's
+   SHA-256 is listed in the GitHub Release body, REQ-354):
+   ```sh
+   sha256sum ./artifact-cache/nova-lambda-token-vend-v1.29.0.zip
+   # → must match the SHA-256 in the release body
+   ```
+
+3. **Place the bundle in the operator's local artifact cache** — a
+   directory the `act_runner` can read (e.g. a Gitea-lfs-tracked path
+   in `nova-platform-ops`, or an S3 bucket the runner can reach).
+
+4. **Reference by SHA-256 in the terraform variables** — the
+   `nova-platform-ops` Terraform accepts an override for the artifact
+   source: `nova_artifact_mirror_sha256 = "<sha256>"`. When set, the
+   `data` sources resolve from the local cache by SHA-256 instead of
+   from GitHub Releases. Unset → resume GitHub Releases resolution.
+
+This fallback is for CI continuity only; the live `terraform apply`
+still resolves the ECR image by digest (KJ-LOCKSTEP), which is
+independent of GitHub Releases availability.
+
+## 9. KMS rotation (D-234)
+
+The OIDC signing key `alias/nova-oidc-signing` is provisioned with
+`KeySpec: ECC_NIST_P256`, `KeyUsage: SIGN_VERIFY`, on a **90-day
+rotation cadence** (matches per-stack CMK rotation per D-069).
+
+**Verify the key spec + rotation status:**
+```sh
+aws kms describe-key --key-id alias/nova-oidc-signing \
+  --query 'KeyMetadata.[KeySpec,KeyUsage,Description]' --output text
+# → ECC_NIST_P256    SIGN_VERIFY    <description>
+```
+
+**Apply a rotation policy** (key re-point, not key deletion — the
+alias moves to a new key while the old key stays valid during the
+overlap window, §17 JWKS-ROTATION):
+
+1. Create the new key (same spec):
+   ```sh
+   NEW_KEY=$(aws kms create-key \
+     --key-spec ECC_NIST_P256 \
+     --key-usage SIGN_VERIFY \
+     --description "nova-oidc-signing-$(date +%Y%m%d)" \
+     --query KeyId --output text)
+   ```
+
+2. Re-point the alias:
+   ```sh
+   aws kms update-alias --alias-name alias/nova-oidc-signing \
+     --target-key-id "$NEW_KEY"
+   ```
+
+3. Apply the rotation policy (the key policy grants `kms:Sign` to the
+   token-vend Lambda role + `kms:GetPublicKey` to the JWKS Lambda
+   role):
+   ```sh
+   aws kms put-key-policy --key-id "$NEW_KEY" \
+     --policy-name default --policy file://kms-signing-key-policy.json
+   ```
+
+4. After the 24-hour overlap window (§17), disable + schedule deletion
+   of the old key:
+   ```sh
+   aws kms disable-key --key-id "<old-key-id>"
+   aws kms schedule-key-deletion --key-id "<old-key-id>" \
+     --pending-window-in-days 7
+   ```
+
+5. Verify the new key is active: `nova idp setup --verify` (the KMS
+   round-trip test, REQ-362).
+
+**Audit:** emit a `kms.key_rotated` event with `old_key_id`,
+`new_key_id`, `rotated_at`.
+
+## 10. JWKS reachability via CloudFront edge (D-233, INV-18)
+
+The JWKS endpoint is the **only public read surface** of the live
+platform (INV-18, D-233). All other platform endpoints gate with
+`AuthType: AWS_IAM`. CloudFront + OAC pinning replaces direct Lambda
+Function URL exposure.
+
+**Critical pitfall (RESEARCH §4):** the JWKS Function URL
+`AuthType` MUST be `AWS_IAM`, NOT `NONE`. A common mistake is to set
+`AuthType: NONE` on the Function URL (thinking CloudFront is the
+gate) — that exposes the JWKS endpoint directly to the internet,
+bypassing OAC. The correct configuration:
+
+| Setting | Value |
+|---------|-------|
+| Function URL `AuthType` | `AWS_IAM` (NOT `NONE`) |
+| CloudFront OAC `OriginAccessControlOriginType` | `lambda` |
+| CloudFront OAC `SigningBehavior` | `always` |
+| Lambda resource policy | `lambda:InvokeFunctionUrl` scoped to the CloudFront distribution ARN |
+
+With `AuthType: AWS_IAM` + OAC `always` signing, CloudFront signs
+every origin request with SigV4; a direct Function URL request has no
+SigV4 signature → 403. Only CloudFront can reach the origin.
+
+**Verification (M1.5 item 9):**
+```sh
+# Via CloudFront → 200
+curl -sI https://<jwks-domain>/.well-known/jwks.json | head -1
+# → HTTP/2 200
+
+# Direct Function URL → 403
+curl -sI "<raw-function-url>/.well-known/jwks.json" | head -1
+# → HTTP/2 403
+```
+
+If the direct Function URL returns 200, the `AuthType` is `NONE` —
+fix the Terraform + re-apply before declaring M1.5 green.
+
+## 11. PITR restore (data-engineer section)
+
+DynamoDB point-in-time recovery (PITR) is enabled on every identity +
+contract table. PITR lets you restore a table to any second in the
+last **35 days** (the AWS retention window).
+
+**Tables with PITR enabled:**
+
+| Table | Purpose |
+|-------|---------|
+| `nova-contracts` | contract ingestor records |
+| `nova-change-requests` | change request ledger |
+| `nova-outbox` | audit outbox |
+| `nova-users` | Nova-idp users (Argon2id hashes) |
+| `nova-sessions` | Nova-idp sessions (TTL `expires_at`) |
+| `nova-pats` | Nova-idp PATs (revocation strong-read, D-229) |
+
+**Enable PITR (on a new/restored table — PITR does not carry over
+from the source):**
+```sh
+aws dynamodb update-continuous-backups \
+  --table-name <table> \
+  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true
+```
+
+**Restore a table to a point in time** (PITR never overwrites the
+source — restore to a NEW table, then repoint):
+```sh
+RESTORE_TO=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)
+
+aws dynamodb restore-table-to-point-in-time \
+  --source-table-name <table> \
+  --target-table-name <table>-restored \
+  --restore-date-time "$RESTORE_TO" \
+  --billing-mode-restore-as-is
+
+# After the restore completes (status ACTIVE), repoint the app:
+# - update the stack env var to the restored table name, or
+# - rename: delete <table>, then update-table --new-table-name <table>
+# Then re-enable PITR on the restored table (see above).
+```
+
+**Verify PITR is enabled on all tables:**
+```sh
+for t in nova-contracts nova-change-requests nova-outbox \
+         nova-users nova-sessions nova-pats; do
+  aws dynamodb describe-continuous-backups --table-name "$t" \
+    --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus' \
+    --output text
+done
+# → ENABLED (x6)
+```
+
+Restores older than 35 days are impossible — for longer retention,
+export to S3 via the on-demand export or a scheduled AWS Backup plan.
+
+## 12. DynamoDB import addresses (REQ-361, covered-reference)
+
+The `importable-resources.tf` map in `nova-platform-ops` lists the
+existing live resources that `terraform import` adopts at M1 cutover
+(§2 step 4). Re-import exits non-zero with
+`resource_already_imported` (IMPORT-IDEMPOTENT); CI import treats this
+as idempotent success.
+
+| Terraform address | AWS resource | Type |
+|-------------------|--------------|------|
+| `aws_s3_bucket.nova_tfstate` | `nova-tfstate-581513795199-us-east-1` | S3 bucket (state backend) |
+| `aws_dynamodb_table.nova_contracts` | `nova-contracts` | DynamoDB table |
+| `aws_dynamodb_table.nova_change_requests` | `nova-change-requests` | DynamoDB table |
+| `aws_dynamodb_table.nova_outbox` | `nova-outbox` | DynamoDB table |
+| `aws_iam_role.acdl_act_runner` | `acdl-act-runner-role` | IAM role (reused, spec §7.6) |
+| `aws_kms_key.<per_stack_cmk>` | per-stack CMKs | KMS key (one per stack) |
+
+The identity tables (`nova-users`, `nova-sessions`, `nova-pats`) are
+**new** resources added at M1 (§2 step 6), not imported — they do
+not yet exist in the account at M1.
+
+## 13. PAT revocation (D-229)
+
+PAT revocation has a **60s SLO**: the token-vend Lambda does a
+strongly-consistent DynamoDB read (`ConsistentRead=True`) on every
+token-vend request. A revoked PAT is reflected on the next vend,
+within 60s P95.
+
+**Verify a PAT's revocation status (strong read):**
+```sh
+aws dynamodb get-item \
+  --table-name nova-pats \
+  --key '{"jti":{"S":"<pat-id>"}}' \
+  --consistent-read \
+  --query 'Item.status.S' --output text
+# → active  (still valid)
+# → revoked (next token-vend returns 403)
+```
+
+**Revoke a PAT at the DDB level** (emergency — when the CLI is
+unavailable; the `jti` is known but the raw PAT is not):
+```sh
+aws dynamodb update-item \
+  --table-name nova-pats \
+  --key '{"jti":{"S":"<pat-id>"}}' \
+  --update-expression "SET #s = :r" \
+  --expression-attribute-names '{"#s":"status"}' \
+  --expression-attribute-values '{":r":{"S":"revoked"}}'
+```
+
+The item is **retained** (not deleted) so the audit trail is intact —
+only `status` flips from `active` to `revoked`. The next `token-vend`
+call with that `jti` returns `403 pat_revoked` immediately (D-229:
+the strong read is synchronous).
+
+## 14. Edge configuration (REQ-364/365/366, covered-reference)
+
+The edge stack fronts the JWKS Lambda with CloudFront + WAF + ACM +
+Route53. This is the public read surface (§10, INV-18).
+
+### CloudFront + OAC (REQ-364)
+
+- Distribution origin = the JWKS Lambda Function URL.
+- OAC: `OriginAccessControlOriginType: lambda`,
+  `SigningBehavior: always` (§10).
+- Cache behavior: `Cache-Control: max-age=3600` honored (JWKS-SLO).
+
+### WAF WebACL (REQ-365)
+
+- Scope: `CLOUDFRONT` (the WebACL is in `us-east-1`, the only region
+  for CloudFront-scoped WebACLs).
+- Rate-based rule: `RateBasedStatement` with `Limit: 3000`,
+  `AggregateKeyType: IP`, `EvaluationWindowSec: 300` (3000 requests
+  per 5 minutes per IP).
+- Managed rules: `AWSManagedRulesCommonRuleSet` (the AWS managed rule
+  group for common attacks).
+
+### ACM certificate (REQ-366)
+
+- Certificate in `us-east-1` (CloudFront requires the cert in
+  us-east-1).
+- DNS validation (a CNAME record per validation record is written to
+  Route53). The cert status MUST be `ISSUED` (not
+  `PENDING_VALIDATION`) before the CloudFront distribution can serve
+  the domain.
+
+### Route53 (REQ-366)
+
+- An A-alias record pointing to the CloudFront distribution's domain
+  name.
+
+### `route53_record_not_resolvable` debugging
+
+If the JWKS domain does not resolve (`route53_record_not_resolvable`
+or `NXDOMAIN`):
+
+1. **Check ACM cert status:**
+   ```sh
+   aws acm describe-certificate --certificate-arn <arn> \
+     --query 'Certificate.Status' --output text
+   # → must be ISSUED, not PENDING_VALIDATION
+   ```
+   If `PENDING_VALIDATION`, the DNS validation CNAME records are not
+   in Route53 (or not propagated). Re-apply the validation records +
+   wait for AWS to validate (typically minutes).
+
+2. **Check CloudFront status:**
+   ```sh
+   aws cloudfront get-distribution --id <id> \
+     --query 'Distribution.Status' --output text
+   # → must be Deployed
+   ```
+   If `InProgress`, wait for the deployment to finish. CloudFront
+   deployments take ~5-15 minutes.
+
+3. **Check the Route53 alias record** points to the CloudFront
+   distribution domain name (not the Function URL).
+
+## 15. Fargate standby health (KJ-WARMUP-HEALTH, REQ-363b)
+
+The Fargate standby is the always-warm minimal defensive fallback
+(REQ-363b). It runs the **same ECR image** as the Lambda (KJ-LOCKSTEP,
+REQ-371) — so it is always running the current digest, never a stale
+one.
+
+**Health probe:** `GET /health → 200` every **10s**
+(KJ-WARMUP-HEALTH).
+
+**Failure handling:** 3 consecutive probe failures → alert + the
+token-vend path **fails closed** (no signing). The standby does not
+silently degrade — if it is not healthy, token-vend does not fall
+back to it; it fails closed (INV-17 ABAC discipline extended to the
+substrate).
+
+**Verify the standby is `READY` before M1 cutover:**
+```sh
+# The Fargate task health check (target group)
+aws elbv2 describe-target-health \
+  --target-group-arn <tg-arn> \
+  --query 'TargetHealthDescriptions[0].TargetHealth.State' --output text
+# → healthy
+
+# Direct probe
+curl -sI https://<fargate-endpoint>/health | head -1
+# → HTTP/1.1 200
+```
+
+**Fargate sunset discipline (D-237):** the standby may not be deleted
+unless REQ-363 has been green in production for **≥30 consecutive
+days**. Sunset requires an architecture review. Do not delete the
+standby to save ~$15/month (§7) without that review — it is the
+defensive fallback for the `kj` substrate.
+
+## 16. IAM scope (IAM-NARROW, REQ-360, covered-reference)
+
+The Gitea OIDC role for `act_runner` (reused `acdl-act-runner-role`,
+spec §7.6) is bounded per REQ-360. **No `Action: "*"` or `Resource:
+"*"`** (IAM-NARROW).
+
+The scope covers only:
+
+| Action | Scope | Why |
+|--------|-------|-----|
+| `kms:*` | customer-managed keys in `581513795199` | KMS signing + rotation |
+| `dynamodb:*` | tables prefixed `nova-` | identity + contract tables |
+| `lambda:*` | functions prefixed `nova-` | the 3 Nova-idp Lambdas |
+| `s3:*` | buckets prefixed `nova-` | state bucket + artifact cache |
+| `cloudfront:*` | tagged resources | the JWKS distribution |
+| `wafv2:*` | tagged resources | the WebACL |
+| `acm:*` | tagged resources | the JWKS cert |
+| `route53:*` | tagged resources | the JWKS alias |
+| `iam:PassRole` | roles tagged `nova-ops-only` | pass roles to Lambda/ECS only |
+
+**Verify (M1.5 item 10):**
+```sh
+aws iam get-role-policy --role-name acdl-act-runner-role \
+  --policy-name <policy-name> --query 'PolicyDocument' --output json \
+  | jq '.Statement[].Action, .Statement[].Resource'
+# → no "*" in either list
+```
+
+If `Action: "*"` or `Resource: "*"` appears, the IAM policy is too
+broad — fix the Terraform + re-apply before declaring M1.5 green.
+
+## 17. JWKS-ROTATION
+
+On KMS key rotation (§9), **both old + new public keys** are
+published in the JWKS during a **24-hour overlap window**. The old
+key is removed from the JWKS only after consumers pick up the new
+one.
+
+- During the overlap: the JWKS Lambda lists all keys the alias has
+  pointed at that are still enabled. Already-issued OIDC tokens
+  (signed with the old key) keep verifying until they expire (OIDC
+  TTL default 15 min; PAT TTL ≤ 24h dev / ≤ 1h service-account).
+- **Do not disable the old key until at least the max PAT TTL (24h)
+  has elapsed.**
+- After the overlap, the old key is removed from the JWKS + disabled +
+  scheduled for deletion (§9 step 4).
+
+This is JWKS-ROTATION (NFR) — the rotation is non-disruptive because
+consumers cache the JWKS for up to `max-age=3600` (1h, JWKS-SLO) and
+re-fetch within that window, picking up both keys during the overlap.
+
+## 18. Cutover Gates (grill CF-2/G-5)
+
+Each covered-reference REQ has a cutover gate (M1/M1.5/M2) with a
+verification command + a **"Result" column**. The operator fills the
+Result column after running the gate in `nova-platform-ops` CI.
+**P6 audit verifies every covered-reference REQ has a non-empty,
+green Result.** Empty or red → P6 blocks (grill CF-2/G-5).
+
+| REQ | Gate | Verification command | Result |
+|-----|------|----------------------|--------|
+| REQ-355 | M1 | `terraform plan` resolves `data.aws_ecr_image.kj_image` from `local.nova_platform_version` + `local.kj_source_sha`; both image_uri attributes present | _(operator fills: green/red + run ID/SHA + attestor)_ |
+| REQ-356 | M1 | `terraform plan` exit 0 (no drift) on a clean checkout of `main`; exit 2 → `drift_detected` fails the apply workflow | _(operator fills)_ |
+| REQ-357 | M1.5 | `terraform apply` `workflow_dispatch` triggered by PR author → rejected; distinct user → proceeds (M1.5 item 11) | _(operator fills)_ |
+| REQ-358 | M2 | bump `nova_platform_version` → `terraform apply` → `aws lambda get-function ... ImageUri` `CodeSha256` matches the artifact SHA-256 from the GitHub Release body | _(operator fills)_ |
+| REQ-359 | M1 | `git -C nova-platform-ops remote -v` shows only the Gitea private remote (no GitHub mirror); Gitea repo `private: true` | _(operator fills)_ |
+| REQ-360 | M1.5 | `aws iam get-role-policy` on the OIDC role asserts no `Action: "*"` + no `Resource: "*"` (M1.5 item 10, §16) | _(operator fills)_ |
+| REQ-361 | M1 | `terraform import` on each address in `importable-resources.tf` (§12) succeeds; re-import exits `resource_already_imported` → CI treats as idempotent success (IMPORT-IDEMPOTENT) | _(operator fills)_ |
+| REQ-362 | M1.5 | `nova idp setup --verify` (KMS round-trip) against `alias/nova-oidc-signing` (`ECC_NIST_P256`, `SIGN_VERIFY`) → `{"passed":true}` (M1.5 item 6) | _(operator fills)_ |
+| REQ-363 | M1.5 | `nova_idp_token_vend.handler` invokes `subprocess.run(['/opt/kj/kj','apply',...])` on the live image; `file(1)` reports `statically linked` (M1.5 items 2-4, KJ-STATIC) | _(operator fills)_ |
+| REQ-363b | M1.5 | Fargate standby `GET /health → 200` every 10s (KJ-WARMUP-HEALTH); same ECR digest as the Lambda (KJ-LOCKSTEP); activates only if M1.5 items 1-7 fail 3× (§5) | _(operator fills)_ |
+| REQ-364 | M1.5 | direct JWKS Function URL → 403; via-CloudFront → 200 (M1.5 item 9, §10, INV-18) | _(operator fills)_ |
+| REQ-365 | M1 | `aws wafv2 get-web-acl` shows `RateBasedStatement` Limit 3000, AggregateKeyType IP, EvaluationWindowSec 300 + `AWSManagedRulesCommonRuleSet`; Scope CLOUDFRONT in us-east-1 (§14) | _(operator fills)_ |
+| REQ-366 | M1 | `aws acm describe-certificate` Status `ISSUED`; Route53 A-alias resolves to the CloudFront distribution domain (§14) | _(operator fills)_ |
+| REQ-371 | M2 | after `terraform apply`, both `aws_lambda_function.nova_idp_token_vend.image_uri` and `aws_ecs_task_definition.kj.container_definitions[0].image` report the same ECR digest (KJ-LOCKSTEP precondition green at plan) | _(operator fills)_ |
+
+> **P6 audit rule (grill CF-2/G-5):** every row's Result column must
+> be non-empty + green. An empty or red Result blocks the milestone
+> ship. The operator attestation is the acdl-side evidence surface;
+> the live verification runs in `nova-platform-ops` CI.
+
+---
+
+## Appendix — quick reference
+
+| Procedure | Cadence / trigger | Section |
+|-----------|-------------------|---------|
+| Day-0 cutover (M1) | one-time | §2 |
+| M1.5 verification gate | one-time (3 consecutive green rebuilds) | §3 |
+| M2 operational handoff | per engineering change (tag-pin bump) | §4 |
+| M2a Fargate activation | conditional (M1.5 fails 3×) | §5 |
+| Rollback | on regression | §6 |
+| Artifact-mirror fallback | on GitHub Releases unreachable | §8 |
+| KMS rotation | every 90 days | §9 |
+| JWKS-ROTATION overlap | on each KMS rotation (24h window) | §17 |
+| PITR restore | on data loss / corruption (35-day window) | §11 |
+| Emergency PAT revocation | on compromise (DDB-level, immediate) | §13 |
+| Fargate standby health check | continuous (every 10s) | §15 |
+| Fargate sunset | ≥30 consecutive days green + architecture review | §5, §15 |
+| `route53_record_not_resolvable` debug | on JWKS domain not resolving | §14 |
+| Cutover gate attestation | at M1/M1.5/M2 (operator fills Result column) | §18 |