acdl/docs/operator-guide-platform-ops.md

# Operator Guide — Nova Platform Ops (`nova-platform-ops`)

> **REQ-OPS-GUIDE** — the operator-facing runbook for the
> `nova-platform-ops` Terraform repo. This is the verification surface
> for the covered-reference REQs (355-366, 371): their cutover gates
> (M1/M1.5/M2) are documented in §18 below, and each REQ has a
> **"Result" column** that the operator fills in after running the gate.
> P6 audit verifies every covered-reference REQ has a non-empty, green
> Result (grill CF-2/G-5). **HARD P6 ship gate:** §3 contains the
> operator-attested "M1.5 Verification Gate Result" row (grill
> CF-1/G-2.1) — the milestone does not ship until that row is filled.
>
> Audience: platform operators / SREs running the live Nova platform in
> AWS account `581513795199`. For the developer auth flows, see
> `docs/developer-guide-auth.md`; for the legacy CloudFormation path,
> see `docs/archive/nova-idp-cfn-v1.28.md`.

## 1. Overview + reposplit rationale

Nova's platform operations live in a dedicated, Gitea-private Terraform
repository — `nova-platform-ops` — separate from the engineering repo
`acdl/acdl`. The split is grounded in Vision §4 (Domain Boundaries):

> *The platform begins where the artifact is compiled and ends where it
> runs in production under operational guardrails.*

That is two distinct disciplines with two distinct ownership surfaces:

| Discipline | Ends | Begins | Repo | Surface |
|------------|------|--------|------|---------|
| Engineering | at the compiled artifact | — | `acdl/acdl` (GitHub) | `publish.yml` + GitHub Releases |
| Operations | — | at the live platform under guardrails | `nova-platform-ops` (Gitea-private) | Terraform modules |

**Scope split (CLARIFY Q-P1, D-232):**

- `acdl/acdl` authors `publish.yml` (the artifact publish pipeline) +
  the artifacts themselves (Lambda zip, layer wheel, Python wheel, ECR
  container image with the static `kj` binary). Each tag `v1.29.x`
  produces a GitHub Release with SHA-256-verified artifacts (REQ-354).
- `nova-platform-ops` authors the Terraform modules
  (`networking`/`kms`/`identity`/`contract-ingest`/`bootstrap`/`edge`)
  that bring those artifacts live in `581513795199`.

The handoff between the two repos is the **tag-pin** (D-235):
`nova-platform-ops` declares `local.nova_platform_version` +
`local.kj_source_sha` and resolves substrates through a single
`data.aws_ecr_image.kj_image`. The engineering repo never knows which
tag is live; the ops repo never authors artifacts. Vision §6
immutability + Vision §5 narrow interfaces.

The covered-reference REQs (355-366, 371) are authored in
`nova-platform-ops` (out-of-band). CIAgent in `acdl` tracks them for
milestone completeness; their verification surface is the cutover
gates in §18 of this guide.

## 2. Day-0 cutover procedure (M1)

The M1 cutover is the one-time conversion of the live AWS account
`581513795199` from CloudFormation-managed (or manually-created)
resources to Terraform-managed resources in `nova-platform-ops`. It is
conditional on the M1.5 verification gate passing (§3, Q7 carry-forward,
D-236).

The 10-step Journey 2 (spec §3.2):

1. **Create `nova-platform-ops` in Gitea** — private (`private: true`,
   OPER-PRIV, REQ-359), no GitHub mirror. The repo is operator-owned;
   CIAgent has no presence there.

2. **Commit the initial Terraform structure** — the module tree
   (`networking`/`kms`/`identity`/`contract-ingest`/`bootstrap`/`edge`)
   + `importable-resources.tf` (§12) + `versions.tf` + `backend.tf`
   (S3 state in the imported bucket).

3. **`terraform init`** — initialize the S3 backend against the
   state bucket (`nova-tfstate-581513795199-us-east-1`, imported in
   step 5). The bucket is created manually once (operator's secure
   scratch, spec §7.1, D-235) before Terraform adopts it.

4. **`terraform import` for existing live resources** — adopt the
   resources that already exist in `581513795199` into Terraform state
   without recreating them. The import map is in
   `importable-resources.tf` (§12):
   - `aws_s3_bucket.nova_tfstate` ← `nova-tfstate-581513795199-us-east-1`
   - `aws_dynamodb_table.nova_contracts` ← `nova-contracts`
   - `aws_dynamodb_table.nova_change_requests` ← `nova-change-requests`
   - `aws_dynamodb_table.nova_outbox` ← `nova-outbox`
   - `aws_iam_role.acdl_act_runner` ← `acdl-act-runner-role`
   - per-stack CMKs (KMS keys)

   Re-import exits non-zero with `resource_already_imported`
   (IMPORT-IDEMPOTENT, REQ-361). CI import treats this as idempotent
   success — the import workflow greps the error stream + exits 0 on
   that string.

5. **(state bucket is imported in step 4)** — listed here for sequence
   clarity; the S3 state bucket is the first import because the
   backend depends on it.

6. **Add new resources** that do not yet exist in the account:
   - KMS alias `alias/nova-oidc-signing` (§9, D-234).
   - Identity DynamoDB tables: `nova-users`, `nova-sessions`,
     `nova-pats` (§11).
   - JWKS Function URL with `AuthType: AWS_IAM` (NOT `NONE` — §10,
     INV-18, RESEARCH §4 critical pitfall).
   - CloudFront distribution + OAC + WAF WebACL + ACM certificate +
     Route53 alias (§14, REQ-364/365/366).

7. **`terraform plan`** — expect zero diff on the imported resources
   (they are already in their desired state) + a pure-add diff on the
   new resources. If the plan shows a diff on an imported resource,
   the import map or the Terraform resource block is wrong — fix
   before apply. **DRIFT-DETECT (REQ-356):** `terraform plan` exit 2
   (drift) fails the apply workflow; manual reconciliation required.

8. **HITL approval** — `terraform apply` against `main` requires a
   Gitea Actions approval from a user **distinct from the PR author**
   (TFM-HITL, REQ-357, INV-3). Self-approval is rejected:
   `gitea.triggering_actor == pull_request.user.login` → apply fails
   closed (M1.5 item 11).

9. **`terraform apply`** — on approval, the apply creates the new
   resources + adopts the imported ones. Smoke test (step 10) before
   declaring M1 done.

10. **Smoke test + CFN→Terraform conversion** — verify the live
    account is in the desired state (JWKS reachable via CloudFront,
    KMS round-trip, ABAC fail-closed). The CFN template in
    `acdl/acdl/nova/idp/setup.py` is archived to
    `docs/archive/nova-idp-cfn-v1.28.md` as read-only reference
    (REQ-369); the active path is now `terraform apply` in
    `nova-platform-ops`.

## 3. M1.5 verification gate (12-item spike)

The M1.5 gate is the 12-item spike from PLAN.md "Happy Path" (spec
§3.3 Edge 5 items 1-8 + grill CF-1 items 9-12). **3 consecutive green
rebuilds are required** in `nova-platform-ops` CI.

The 12 items:

1. `kj` v0.0.3 (pinned SHA in `platform/abac/kj-version.txt`) compiles
   with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64`.
2. Resulting binary reports `file kj → ELF 64-bit LSB executable,
   x86-64, statically linked, no shared library` (KJ-STATIC).
3. Container image built from
   `public.ecr.aws/lambda/python:3.12-al2023` with the binary copied
   to `/opt/kj/kj`, `chmod 0555`, owned by `sbx_user:1051`.
4. Lambda runtime `python3.12` executes
   `nova_idp_token_vend.handler`; the handler invokes
   `subprocess.run(['/opt/kj/kj', 'apply', ...])` and parses stdout
   JSON.
5. `tests/test_idp_auth.py` passes against the live image in moto-DDB.
6. `tests/test_kms_roundtrip.py` passes against the live KMS key
   `alias/nova-oidc-signing` (REQ-362 path — covered-reference).
7. End-to-end: known PAT → known ABAC-allowed action → signed OIDC
   token → `jose` verification → green. Known PAT + ABAC-denied action
   → 403 with deny reason logged (INV-17 fail-closed).
8. Image URI is recorded in Terraform state and in this operator
   guide (§18, REQ-371 Result row).
9. **(grill CF-1) JWKS-EDGE-ONLY:** direct JWKS Function URL GET
   (bypassing CloudFront) returns **403**; via-CloudFront GET returns
   **200** (INV-18). Proves `AuthType: AWS_IAM` + OAC pinning.
10. **(grill CF-1) IAM-NARROW:** `aws iam get-role-policy` on the
    Gitea OIDC role asserts no `Action: "*"` and no `Resource: "*"`
    (REQ-360).
11. **(grill CF-1) TFM-HITL:** a `terraform apply`
    `workflow_dispatch` triggered by the PR author is **rejected**
    (exit non-zero); a dispatch by a distinct user proceeds (REQ-357).
12. **(grill CF-1) rollback drill:** revert `nova_platform_version`
    pin → `terraform apply` → assert the prior ECR digest runs
    (D-236, guards against ECR tag mutability).

If items 1-7 fail three consecutive rebuilds, M2a activates (§5,
REQ-363b Fargate toggle) with the same ECR image — no warmup hit
because the standby is always running the same digest (KJ-LOCKSTEP).

### HARD P6 ship gate (grill CF-1/G-2.1)

P6 must not ship `v1.28.6` until the operator attests the M1.5 result
in the row below. The operator fills this in **after** the gate passes
3 consecutive green rebuilds in `nova-platform-ops` CI. P6 audit
verifies the row exists + is non-empty.

#### M1.5 Verification Gate Result

| Rebuild # | Run ID / commit SHA | All 12 items green? | Attestor identity | Attested at (UTC) |
|-----------|---------------------|---------------------|-------------------|-------------------|
| 1 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
| 2 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
| 3 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |

> **P6 audit rule:** all three rows must be present, all 12 items
> green on each, the three run IDs/SHAs distinct (consecutive
> rebuilds, not one run copied thrice), and the attestor identity
> non-empty. Empty or red → P6 blocks → escalate.

## 4. M2 operational handoff loop

M2 is the steady-state operational loop for rolling out an engineering
change after M1.5 is green. The loop is the tag-pin bump → plan →
HITL → apply cycle (D-235, D-238).

1. **Tag-pin bump** — the operator opens a PR in `nova-platform-ops`
   bumping `local.nova_platform_version` (e.g. `v1.29.3` → `v1.29.4`)
   + `local.kj_source_sha` (the `kj` source SHA from
   `platform/abac/kj-version.txt` at the new tag). Both pins move
   together — there is one ECR image identity (KJ-LOCKSTEP, REQ-371).

2. **`terraform plan`** — CI runs `terraform plan` on the PR. The
   KJ-LOCKSTEP precondition (a `lifecycle.precondition` on both
   image-bearing resources — the Lambda `image_uri` and the Fargate
   task `container_definitions[0].image`) checks that both
   `image_uri` attributes resolve to the **same ECR digest** via
   `data.aws_ecr_image.kj_image`. If the two diverge, the plan fails
   closed — no second pipeline, no second SHA pin (D-238).

3. **HITL approval** — a Gitea Actions approver **distinct from the
   PR author** approves the apply (TFM-HITL, REQ-357). Self-approval
   is rejected (M1.5 item 11).

4. **`terraform apply`** — on approval, the apply updates both
   `aws_lambda_function.nova_idp_token_vend.image_uri` and
   `aws_ecs_task_definition.kj.container_definitions[0].image` to the
   same ECR digest. The Lambda image + the Fargate task redeploy to
   the same digest in one apply. Zero diff on KMS, DDB, IAM, edge
   (the only change is the image reference).

**Verification:** after the apply, `aws lambda get-function
--function-name nova-idp-token-vend --query Configuration.Code.ImageUri`
and `aws ecs describe-tasks` on the Fargate task both report the same
digest. This is the M2 acceptance gate (PLAN §UX Acceptance Criteria
3) + the REQ-371 Result row in §18.

## 5. M2a Fargate activation (conditional)

M2a activates **only if M1.5 fails 3 consecutive rebuilds** (D-236).
It is the REQ-363b Fargate toggle — an always-warm minimal Fargate
standby running the **same ECR image** as the Lambda (KJ-LOCKSTEP).

Because the standby is always running the same digest as the Lambda,
activating M2a is **not** a warmup hit — the standby is already
serving `GET /health → 200` every 10s (KJ-WARMUP-HEALTH, §15). The
toggle repoints token-vend traffic from the Lambda to the Fargate
task; no cold start, no image pull.

If both the Lambda path and the Fargate path fail (M1.5 items 1-7
fail on both substrates), the operator escalates — Nova-idp ships in
read-only partial mode (no token issuance) until `kj` is verified
(Q7 carry-forward, spec §7.7).

**Fargate sunset discipline (D-237):** the standby (~$15-20/month,
§7) may not be deleted unless REQ-363 has been green in production
for **≥30 consecutive days**. Sunset requires an architecture review.
See §15 for the health-check procedure.

## 6. Rollback procedure (D-236)

Rollback is a tag-pin revert — the same mechanism as the M2 rollout
(§4), in reverse.

1. **Revert `nova_platform_version`** in `nova-platform-ops` to the
   prior tag (e.g. `v1.29.4` → `v1.29.3`). Open a PR, get HITL
   approval (TFM-HITL, same as rollout).

2. **`terraform apply`** — the apply reverts both the Lambda
   `image_uri` and the Fargate task `image` to the prior ECR digest.
   The prior tag's artifacts remain downloadable (GitHub Releases are
   append-only per tag, REQ-354 AC 2) — no artifact is re-built.

3. **Verify** the prior digest is running:
   ```sh
   aws lambda get-function --function-name nova-idp-token-vend \
     --query Configuration.Code.ImageUri --output text
   # → <account>.dkr.ecr.us-east-1.amazonaws.com/nova-kj@sha256:<prior-digest>
   ```

This is the M1.5 item 12 rollback drill + the operational rollback
procedure. It guards against ECR tag mutability (RESEARCH §2) — the
digest is immutable even if a tag is re-pushed.

## 7. Cost section

Monthly estimate for the `nova-platform-ops` live platform in account
`581513795199` (pilot volume):

| Resource | Quantity | Est. monthly | Notes |
|----------|----------|-------------|-------|
| WAF WebACL (CloudFront-scoped) | 1 | ~$5-10 | + per-request; REQ-365 |
| Fargate standby (0.25 vCPU, 512 MB) | 1 task | ~$15-20 | REQ-363b AC 4; largest line item |
| KMS asymmetric key | 1 | ~$1 | `alias/nova-oidc-signing`, ECC_NIST_P256 |
| DynamoDB (on-demand, 6 tables) | 6 | ~$2 | §11 tables |
| Lambda invocations (3 Lambdas) | 3 | ~$2 | low pilot volume |
| ECR image storage | ~100 MB | <$1 | the `kj` image |
| S3 state bucket + access logs | 1 | <$1 | `nova-tfstate-*` |
| CloudFront + ACM + Route53 | 1 distribution | ~$1 | ACM free for CloudFront-attached |
| **Total** | | **~$30-40/month** | |

**Fargate standby is the largest line item** (~$15-20/month, REQ-363b
AC 4). It is explicitly documented here with the D-237 sunset
discipline (§5, §15): ≥30 consecutive days green before deletion +
architecture review. Do not delete the standby to save ~$15/month
without that review — it is the defensive fallback for the `kj`
substrate.

## 8. Artifact-mirror fallback (Edge 6)

When the Gitea `act_runner` in `nova-platform-ops` CI cannot reach
GitHub Releases (network partition, egress restriction, GitHub
outage), the operator mirrors the artifact bundle locally by SHA-256.

**Procedure:**

1. **Download the GitHub Release bundle** for the target tag
   (`v1.29.x`) from a machine that can reach GitHub Releases:
   ```sh
   gh release download v1.29.0 \
     --repo continuous-intelligence/acdl \
     --pattern 'nova-lambda-token-vend-*.zip' \
     --pattern 'nova-cli-layer-*.zip' \
     --pattern 'nova-*-py3-none-any.whl' \
     --dir ./artifact-cache
   ```

2. **Verify SHA-256** against the release body (each artifact's
   SHA-256 is listed in the GitHub Release body, REQ-354):
   ```sh
   sha256sum ./artifact-cache/nova-lambda-token-vend-v1.29.0.zip
   # → must match the SHA-256 in the release body
   ```

3. **Place the bundle in the operator's local artifact cache** — a
   directory the `act_runner` can read (e.g. a Gitea-lfs-tracked path
   in `nova-platform-ops`, or an S3 bucket the runner can reach).

4. **Reference by SHA-256 in the terraform variables** — the
   `nova-platform-ops` Terraform accepts an override for the artifact
   source: `nova_artifact_mirror_sha256 = "<sha256>"`. When set, the
   `data` sources resolve from the local cache by SHA-256 instead of
   from GitHub Releases. Unset → resume GitHub Releases resolution.

This fallback is for CI continuity only; the live `terraform apply`
still resolves the ECR image by digest (KJ-LOCKSTEP), which is
independent of GitHub Releases availability.

## 9. KMS rotation (D-234)

The OIDC signing key `alias/nova-oidc-signing` is provisioned with
`KeySpec: ECC_NIST_P256`, `KeyUsage: SIGN_VERIFY`, on a **90-day
rotation cadence** (matches per-stack CMK rotation per D-069).

**Verify the key spec + rotation status:**
```sh
aws kms describe-key --key-id alias/nova-oidc-signing \
  --query 'KeyMetadata.[KeySpec,KeyUsage,Description]' --output text
# → ECC_NIST_P256    SIGN_VERIFY    <description>
```

**Apply a rotation policy** (key re-point, not key deletion — the
alias moves to a new key while the old key stays valid during the
overlap window, §17 JWKS-ROTATION):

1. Create the new key (same spec):
   ```sh
   NEW_KEY=$(aws kms create-key \
     --key-spec ECC_NIST_P256 \
     --key-usage SIGN_VERIFY \
     --description "nova-oidc-signing-$(date +%Y%m%d)" \
     --query KeyId --output text)
   ```

2. Re-point the alias:
   ```sh
   aws kms update-alias --alias-name alias/nova-oidc-signing \
     --target-key-id "$NEW_KEY"
   ```

3. Apply the rotation policy (the key policy grants `kms:Sign` to the
   token-vend Lambda role + `kms:GetPublicKey` to the JWKS Lambda
   role):
   ```sh
   aws kms put-key-policy --key-id "$NEW_KEY" \
     --policy-name default --policy file://kms-signing-key-policy.json
   ```

4. After the 24-hour overlap window (§17), disable + schedule deletion
   of the old key:
   ```sh
   aws kms disable-key --key-id "<old-key-id>"
   aws kms schedule-key-deletion --key-id "<old-key-id>" \
     --pending-window-in-days 7
   ```

5. Verify the new key is active: `nova idp setup --verify` (the KMS
   round-trip test, REQ-362).

**Audit:** emit a `kms.key_rotated` event with `old_key_id`,
`new_key_id`, `rotated_at`.

## 10. JWKS reachability via CloudFront edge (D-233, INV-18)

The JWKS endpoint is the **only public read surface** of the live
platform (INV-18, D-233). All other platform endpoints gate with
`AuthType: AWS_IAM`. CloudFront + OAC pinning replaces direct Lambda
Function URL exposure.

**Critical pitfall (RESEARCH §4):** the JWKS Function URL
`AuthType` MUST be `AWS_IAM`, NOT `NONE`. A common mistake is to set
`AuthType: NONE` on the Function URL (thinking CloudFront is the
gate) — that exposes the JWKS endpoint directly to the internet,
bypassing OAC. The correct configuration:

| Setting | Value |
|---------|-------|
| Function URL `AuthType` | `AWS_IAM` (NOT `NONE`) |
| CloudFront OAC `OriginAccessControlOriginType` | `lambda` |
| CloudFront OAC `SigningBehavior` | `always` |
| Lambda resource policy | `lambda:InvokeFunctionUrl` scoped to the CloudFront distribution ARN |

With `AuthType: AWS_IAM` + OAC `always` signing, CloudFront signs
every origin request with SigV4; a direct Function URL request has no
SigV4 signature → 403. Only CloudFront can reach the origin.

**Verification (M1.5 item 9):**
```sh
# Via CloudFront → 200
curl -sI https://<jwks-domain>/.well-known/jwks.json | head -1
# → HTTP/2 200

# Direct Function URL → 403
curl -sI "<raw-function-url>/.well-known/jwks.json" | head -1
# → HTTP/2 403
```

If the direct Function URL returns 200, the `AuthType` is `NONE` —
fix the Terraform + re-apply before declaring M1.5 green.

## 11. PITR restore (data-engineer section)

DynamoDB point-in-time recovery (PITR) is enabled on every identity +
contract table. PITR lets you restore a table to any second in the
last **35 days** (the AWS retention window).

**Tables with PITR enabled:**

| Table | Purpose |
|-------|---------|
| `nova-contracts` | contract ingestor records |
| `nova-change-requests` | change request ledger |
| `nova-outbox` | audit outbox |
| `nova-users` | Nova-idp users (Argon2id hashes) |
| `nova-sessions` | Nova-idp sessions (TTL `expires_at`) |
| `nova-pats` | Nova-idp PATs (revocation strong-read, D-229) |

**Enable PITR (on a new/restored table — PITR does not carry over
from the source):**
```sh
aws dynamodb update-continuous-backups \
  --table-name <table> \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true
```

**Restore a table to a point in time** (PITR never overwrites the
source — restore to a NEW table, then repoint):
```sh
RESTORE_TO=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)

aws dynamodb restore-table-to-point-in-time \
  --source-table-name <table> \
  --target-table-name <table>-restored \
  --restore-date-time "$RESTORE_TO" \
  --billing-mode-restore-as-is

# After the restore completes (status ACTIVE), repoint the app:
# - update the stack env var to the restored table name, or
# - rename: delete <table>, then update-table --new-table-name <table>
# Then re-enable PITR on the restored table (see above).
```

**Verify PITR is enabled on all tables:**
```sh
for t in nova-contracts nova-change-requests nova-outbox \
         nova-users nova-sessions nova-pats; do
  aws dynamodb describe-continuous-backups --table-name "$t" \
    --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus' \
    --output text
done
# → ENABLED (x6)
```

Restores older than 35 days are impossible — for longer retention,
export to S3 via the on-demand export or a scheduled AWS Backup plan.

## 12. DynamoDB import addresses (REQ-361, covered-reference)

The `importable-resources.tf` map in `nova-platform-ops` lists the
existing live resources that `terraform import` adopts at M1 cutover
(§2 step 4). Re-import exits non-zero with
`resource_already_imported` (IMPORT-IDEMPOTENT); CI import treats this
as idempotent success.

| Terraform address | AWS resource | Type |
|-------------------|--------------|------|
| `aws_s3_bucket.nova_tfstate` | `nova-tfstate-581513795199-us-east-1` | S3 bucket (state backend) |
| `aws_dynamodb_table.nova_contracts` | `nova-contracts` | DynamoDB table |
| `aws_dynamodb_table.nova_change_requests` | `nova-change-requests` | DynamoDB table |
| `aws_dynamodb_table.nova_outbox` | `nova-outbox` | DynamoDB table |
| `aws_iam_role.acdl_act_runner` | `acdl-act-runner-role` | IAM role (reused, spec §7.6) |
| `aws_kms_key.<per_stack_cmk>` | per-stack CMKs | KMS key (one per stack) |

The identity tables (`nova-users`, `nova-sessions`, `nova-pats`) are
**new** resources added at M1 (§2 step 6), not imported — they do
not yet exist in the account at M1.

## 13. PAT revocation (D-229)

PAT revocation has a **60s SLO**: the token-vend Lambda does a
strongly-consistent DynamoDB read (`ConsistentRead=True`) on every
token-vend request. A revoked PAT is reflected on the next vend,
within 60s P95.

**Verify a PAT's revocation status (strong read):**
```sh
aws dynamodb get-item \
  --table-name nova-pats \
  --key '{"jti":{"S":"<pat-id>"}}' \
  --consistent-read \
  --query 'Item.status.S' --output text
# → active  (still valid)
# → revoked (next token-vend returns 403)
```

**Revoke a PAT at the DDB level** (emergency — when the CLI is
unavailable; the `jti` is known but the raw PAT is not):
```sh
aws dynamodb update-item \
  --table-name nova-pats \
  --key '{"jti":{"S":"<pat-id>"}}' \
  --update-expression "SET #s = :r" \
  --expression-attribute-names '{"#s":"status"}' \
  --expression-attribute-values '{":r":{"S":"revoked"}}'
```

The item is **retained** (not deleted) so the audit trail is intact —
only `status` flips from `active` to `revoked`. The next `token-vend`
call with that `jti` returns `403 pat_revoked` immediately (D-229:
the strong read is synchronous).

## 14. Edge configuration (REQ-364/365/366, covered-reference)

The edge stack fronts the JWKS Lambda with CloudFront + WAF + ACM +
Route53. This is the public read surface (§10, INV-18).

### CloudFront + OAC (REQ-364)

- Distribution origin = the JWKS Lambda Function URL.
- OAC: `OriginAccessControlOriginType: lambda`,
  `SigningBehavior: always` (§10).
- Cache behavior: `Cache-Control: max-age=3600` honored (JWKS-SLO).

### WAF WebACL (REQ-365)

- Scope: `CLOUDFRONT` (the WebACL is in `us-east-1`, the only region
  for CloudFront-scoped WebACLs).
- Rate-based rule: `RateBasedStatement` with `Limit: 3000`,
  `AggregateKeyType: IP`, `EvaluationWindowSec: 300` (3000 requests
  per 5 minutes per IP).
- Managed rules: `AWSManagedRulesCommonRuleSet` (the AWS managed rule
  group for common attacks).

### ACM certificate (REQ-366)

- Certificate in `us-east-1` (CloudFront requires the cert in
  us-east-1).
- DNS validation (a CNAME record per validation record is written to
  Route53). The cert status MUST be `ISSUED` (not
  `PENDING_VALIDATION`) before the CloudFront distribution can serve
  the domain.

### Route53 (REQ-366)

- An A-alias record pointing to the CloudFront distribution's domain
  name.

### `route53_record_not_resolvable` debugging

If the JWKS domain does not resolve (`route53_record_not_resolvable`
or `NXDOMAIN`):

1. **Check ACM cert status:**
   ```sh
   aws acm describe-certificate --certificate-arn <arn> \
     --query 'Certificate.Status' --output text
   # → must be ISSUED, not PENDING_VALIDATION
   ```
   If `PENDING_VALIDATION`, the DNS validation CNAME records are not
   in Route53 (or not propagated). Re-apply the validation records +
   wait for AWS to validate (typically minutes).

2. **Check CloudFront status:**
   ```sh
   aws cloudfront get-distribution --id <id> \
     --query 'Distribution.Status' --output text
   # → must be Deployed
   ```
   If `InProgress`, wait for the deployment to finish. CloudFront
   deployments take ~5-15 minutes.

3. **Check the Route53 alias record** points to the CloudFront
   distribution domain name (not the Function URL).

## 15. Fargate standby health (KJ-WARMUP-HEALTH, REQ-363b)

The Fargate standby is the always-warm minimal defensive fallback
(REQ-363b). It runs the **same ECR image** as the Lambda (KJ-LOCKSTEP,
REQ-371) — so it is always running the current digest, never a stale
one.

**Health probe:** `GET /health → 200` every **10s**
(KJ-WARMUP-HEALTH).

**Failure handling:** 3 consecutive probe failures → alert + the
token-vend path **fails closed** (no signing). The standby does not
silently degrade — if it is not healthy, token-vend does not fall
back to it; it fails closed (INV-17 ABAC discipline extended to the
substrate).

**Verify the standby is `READY` before M1 cutover:**
```sh
# The Fargate task health check (target group)
aws elbv2 describe-target-health \
  --target-group-arn <tg-arn> \
  --query 'TargetHealthDescriptions[0].TargetHealth.State' --output text
# → healthy

# Direct probe
curl -sI https://<fargate-endpoint>/health | head -1
# → HTTP/1.1 200
```

**Fargate sunset discipline (D-237):** the standby may not be deleted
unless REQ-363 has been green in production for **≥30 consecutive
days**. Sunset requires an architecture review. Do not delete the
standby to save ~$15/month (§7) without that review — it is the
defensive fallback for the `kj` substrate.

## 16. IAM scope (IAM-NARROW, REQ-360, covered-reference)

The Gitea OIDC role for `act_runner` (reused `acdl-act-runner-role`,
spec §7.6) is bounded per REQ-360. **No `Action: "*"` or `Resource:
"*"`** (IAM-NARROW).

The scope covers only:

| Action | Scope | Why |
|--------|-------|-----|
| `kms:*` | customer-managed keys in `581513795199` | KMS signing + rotation |
| `dynamodb:*` | tables prefixed `nova-` | identity + contract tables |
| `lambda:*` | functions prefixed `nova-` | the 3 Nova-idp Lambdas |
| `s3:*` | buckets prefixed `nova-` | state bucket + artifact cache |
| `cloudfront:*` | tagged resources | the JWKS distribution |
| `wafv2:*` | tagged resources | the WebACL |
| `acm:*` | tagged resources | the JWKS cert |
| `route53:*` | tagged resources | the JWKS alias |
| `iam:PassRole` | roles tagged `nova-ops-only` | pass roles to Lambda/ECS only |

**Verify (M1.5 item 10):**
```sh
aws iam get-role-policy --role-name acdl-act-runner-role \
  --policy-name <policy-name> --query 'PolicyDocument' --output json \
  | jq '.Statement[].Action, .Statement[].Resource'
# → no "*" in either list
```

If `Action: "*"` or `Resource: "*"` appears, the IAM policy is too
broad — fix the Terraform + re-apply before declaring M1.5 green.

## 17. JWKS-ROTATION

On KMS key rotation (§9), **both old + new public keys** are
published in the JWKS during a **24-hour overlap window**. The old
key is removed from the JWKS only after consumers pick up the new
one.

- During the overlap: the JWKS Lambda lists all keys the alias has
  pointed at that are still enabled. Already-issued OIDC tokens
  (signed with the old key) keep verifying until they expire (OIDC
  TTL default 15 min; PAT TTL ≤ 24h dev / ≤ 1h service-account).
- **Do not disable the old key until at least the max PAT TTL (24h)
  has elapsed.**
- After the overlap, the old key is removed from the JWKS + disabled +
  scheduled for deletion (§9 step 4).

This is JWKS-ROTATION (NFR) — the rotation is non-disruptive because
consumers cache the JWKS for up to `max-age=3600` (1h, JWKS-SLO) and
re-fetch within that window, picking up both keys during the overlap.

## 18. Cutover Gates (grill CF-2/G-5)

Each covered-reference REQ has a cutover gate (M1/M1.5/M2) with a
verification command + a **"Result" column**. The operator fills the
Result column after running the gate in `nova-platform-ops` CI.
**P6 audit verifies every covered-reference REQ has a non-empty,
green Result.** Empty or red → P6 blocks (grill CF-2/G-5).

| REQ | Gate | Verification command | Result |
|-----|------|----------------------|--------|
| REQ-355 | M1 | `terraform plan` resolves `data.aws_ecr_image.kj_image` from `local.nova_platform_version` + `local.kj_source_sha`; both image_uri attributes present | _(operator fills: green/red + run ID/SHA + attestor)_ |
| REQ-356 | M1 | `terraform plan` exit 0 (no drift) on a clean checkout of `main`; exit 2 → `drift_detected` fails the apply workflow | _(operator fills)_ |
| REQ-357 | M1.5 | `terraform apply` `workflow_dispatch` triggered by PR author → rejected; distinct user → proceeds (M1.5 item 11) | _(operator fills)_ |
| REQ-358 | M2 | bump `nova_platform_version` → `terraform apply` → `aws lambda get-function ... ImageUri` `CodeSha256` matches the artifact SHA-256 from the GitHub Release body | _(operator fills)_ |
| REQ-359 | M1 | `git -C nova-platform-ops remote -v` shows only the Gitea private remote (no GitHub mirror); Gitea repo `private: true` | _(operator fills)_ |
| REQ-360 | M1.5 | `aws iam get-role-policy` on the OIDC role asserts no `Action: "*"` + no `Resource: "*"` (M1.5 item 10, §16) | _(operator fills)_ |
| REQ-361 | M1 | `terraform import` on each address in `importable-resources.tf` (§12) succeeds; re-import exits `resource_already_imported` → CI treats as idempotent success (IMPORT-IDEMPOTENT) | _(operator fills)_ |
| REQ-362 | M1.5 | `nova idp setup --verify` (KMS round-trip) against `alias/nova-oidc-signing` (`ECC_NIST_P256`, `SIGN_VERIFY`) → `{"passed":true}` (M1.5 item 6) | _(operator fills)_ |
| REQ-363 | M1.5 | `nova_idp_token_vend.handler` invokes `subprocess.run(['/opt/kj/kj','apply',...])` on the live image; `file(1)` reports `statically linked` (M1.5 items 2-4, KJ-STATIC) | _(operator fills)_ |
| REQ-363b | M1.5 | Fargate standby `GET /health → 200` every 10s (KJ-WARMUP-HEALTH); same ECR digest as the Lambda (KJ-LOCKSTEP); activates only if M1.5 items 1-7 fail 3× (§5) | _(operator fills)_ |
| REQ-364 | M1.5 | direct JWKS Function URL → 403; via-CloudFront → 200 (M1.5 item 9, §10, INV-18) | _(operator fills)_ |
| REQ-365 | M1 | `aws wafv2 get-web-acl` shows `RateBasedStatement` Limit 3000, AggregateKeyType IP, EvaluationWindowSec 300 + `AWSManagedRulesCommonRuleSet`; Scope CLOUDFRONT in us-east-1 (§14) | _(operator fills)_ |
| REQ-366 | M1 | `aws acm describe-certificate` Status `ISSUED`; Route53 A-alias resolves to the CloudFront distribution domain (§14) | _(operator fills)_ |
| REQ-371 | M2 | after `terraform apply`, both `aws_lambda_function.nova_idp_token_vend.image_uri` and `aws_ecs_task_definition.kj.container_definitions[0].image` report the same ECR digest (KJ-LOCKSTEP precondition green at plan) | _(operator fills)_ |

> **P6 audit rule (grill CF-2/G-5):** every row's Result column must
> be non-empty + green. An empty or red Result blocks the milestone
> ship. The operator attestation is the acdl-side evidence surface;
> the live verification runs in `nova-platform-ops` CI.

---

## Appendix — quick reference

| Procedure | Cadence / trigger | Section |
|-----------|-------------------|---------|
| Day-0 cutover (M1) | one-time | §2 |
| M1.5 verification gate | one-time (3 consecutive green rebuilds) | §3 |
| M2 operational handoff | per engineering change (tag-pin bump) | §4 |
| M2a Fargate activation | conditional (M1.5 fails 3×) | §5 |
| Rollback | on regression | §6 |
| Artifact-mirror fallback | on GitHub Releases unreachable | §8 |
| KMS rotation | every 90 days | §9 |
| JWKS-ROTATION overlap | on each KMS rotation (24h window) | §17 |
| PITR restore | on data loss / corruption (35-day window) | §11 |
| Emergency PAT revocation | on compromise (DDB-level, immediate) | §13 |
| Fargate standby health check | continuous (every 10s) | §15 |
| Fargate sunset | ≥30 consecutive days green + architecture review | §5, §15 |
| `route53_record_not_resolvable` debug | on JWKS domain not resolving | §14 |
| Cutover gate attestation | at M1/M1.5/M2 (operator fills Result column) | §18 |