@@ -0,0 +1,747 @@
# Operator Guide — Nova Platform Ops (`nova-platform-ops`)
> **REQ-OPS-GUIDE** — the operator-facing runbook for the
> `nova-platform-ops` Terraform repo. This is the verification surface
> for the covered-reference REQs (355-366, 371): their cutover gates
> (M1/M1.5/M2) are documented in §18 below, and each REQ has a
> **"Result" column** that the operator fills in after running the gate.
> P6 audit verifies every covered-reference REQ has a non-empty, green
> Result (grill CF-2/G-5). **HARD P6 ship gate:** §3 contains the
> operator-attested "M1.5 Verification Gate Result" row (grill
> CF-1/G-2.1) — the milestone does not ship until that row is filled.
>
> Audience: platform operators / SREs running the live Nova platform in
> AWS account `581513795199`. For the developer auth flows, see
> `docs/developer-guide-auth.md`; for the legacy CloudFormation path,
> see `docs/archive/nova-idp-cfn-v1.28.md`.
## 1. Overview + reposplit rationale
Nova's platform operations live in a dedicated, Gitea-private Terraform
repository — `nova-platform-ops` — separate from the engineering repo
`acdl/acdl` . The split is grounded in Vision §4 (Domain Boundaries):
> *The platform begins where the artifact is compiled and ends where it
> runs in production under operational guardrails.*
That is two distinct disciplines with two distinct ownership surfaces:
| Discipline | Ends | Begins | Repo | Surface |
|------------|------|--------|------|---------|
| Engineering | at the compiled artifact | — | `acdl/acdl` (GitHub) | `publish.yml` + GitHub Releases |
| Operations | — | at the live platform under guardrails | `nova-platform-ops` (Gitea-private) | Terraform modules |
**Scope split (CLARIFY Q-P1, D-232): **
- `acdl/acdl` authors `publish.yml` (the artifact publish pipeline) +
the artifacts themselves (Lambda zip, layer wheel, Python wheel, ECR
container image with the static `kj` binary). Each tag `v1.29.x`
produces a GitHub Release with SHA-256-verified artifacts (REQ-354).
- `nova-platform-ops` authors the Terraform modules
(`networking` /`kms` /`identity` /`contract-ingest` /`bootstrap` /`edge` )
that bring those artifacts live in `581513795199` .
The handoff between the two repos is the **tag-pin ** (D-235):
`nova-platform-ops` declares `local.nova_platform_version` +
`local.kj_source_sha` and resolves substrates through a single
`data.aws_ecr_image.kj_image` . The engineering repo never knows which
tag is live; the ops repo never authors artifacts. Vision §6
immutability + Vision §5 narrow interfaces.
The covered-reference REQs (355-366, 371) are authored in
`nova-platform-ops` (out-of-band). CIAgent in `acdl` tracks them for
milestone completeness; their verification surface is the cutover
gates in §18 of this guide.
## 2. Day-0 cutover procedure (M1)
The M1 cutover is the one-time conversion of the live AWS account
`581513795199` from CloudFormation-managed (or manually-created)
resources to Terraform-managed resources in `nova-platform-ops` . It is
conditional on the M1.5 verification gate passing (§3, Q7 carry-forward,
D-236).
The 10-step Journey 2 (spec §3.2):
1. **Create `nova-platform-ops` in Gitea ** — private (`private: true` ,
OPER-PRIV, REQ-359), no GitHub mirror. The repo is operator-owned;
CIAgent has no presence there.
2. **Commit the initial Terraform structure ** — the module tree
(`networking` /`kms` /`identity` /`contract-ingest` /`bootstrap` /`edge` )
+ `importable-resources.tf` (§12) + `versions.tf` + `backend.tf`
(S3 state in the imported bucket).
3. * * `terraform init` ** — initialize the S3 backend against the
state bucket (`nova-tfstate-581513795199-us-east-1` , imported in
step 5). The bucket is created manually once (operator's secure
scratch, spec §7.1, D-235) before Terraform adopts it.
4. * * `terraform import` for existing live resources** — adopt the
resources that already exist in `581513795199` into Terraform state
without recreating them. The import map is in
`importable-resources.tf` (§12):
- `aws_s3_bucket.nova_tfstate` ← `nova-tfstate-581513795199-us-east-1`
- `aws_dynamodb_table.nova_contracts` ← `nova-contracts`
- `aws_dynamodb_table.nova_change_requests` ← `nova-change-requests`
- `aws_dynamodb_table.nova_outbox` ← `nova-outbox`
- `aws_iam_role.acdl_act_runner` ← `acdl-act-runner-role`
- per-stack CMKs (KMS keys)
Re-import exits non-zero with `resource_already_imported`
(IMPORT-IDEMPOTENT, REQ-361). CI import treats this as idempotent
success — the import workflow greps the error stream + exits 0 on
that string.
5. * * (state bucket is imported in step 4)** — listed here for sequence
clarity; the S3 state bucket is the first import because the
backend depends on it.
6. **Add new resources ** that do not yet exist in the account:
- KMS alias `alias/nova-oidc-signing` (§9, D-234).
- Identity DynamoDB tables: `nova-users` , `nova-sessions` ,
`nova-pats` (§11).
- JWKS Function URL with `AuthType: AWS_IAM` (NOT `NONE` — §10,
INV-18, RESEARCH §4 critical pitfall).
- CloudFront distribution + OAC + WAF WebACL + ACM certificate +
Route53 alias (§14, REQ-364/365/366).
7. * * `terraform plan` ** — expect zero diff on the imported resources
(they are already in their desired state) + a pure-add diff on the
new resources. If the plan shows a diff on an imported resource,
the import map or the Terraform resource block is wrong — fix
before apply. **DRIFT-DETECT (REQ-356): ** `terraform plan` exit 2
(drift) fails the apply workflow; manual reconciliation required.
8. **HITL approval ** — `terraform apply` against `main` requires a
Gitea Actions approval from a user **distinct from the PR author **
(TFM-HITL, REQ-357, INV-3). Self-approval is rejected:
`gitea.triggering_actor == pull_request.user.login` → apply fails
closed (M1.5 item 11).
9. * * `terraform apply` ** — on approval, the apply creates the new
resources + adopts the imported ones. Smoke test (step 10) before
declaring M1 done.
10. **Smoke test + CFN→Terraform conversion ** — verify the live
account is in the desired state (JWKS reachable via CloudFront,
KMS round-trip, ABAC fail-closed). The CFN template in
`acdl/acdl/nova/idp/setup.py` is archived to
`docs/archive/nova-idp-cfn-v1.28.md` as read-only reference
(REQ-369); the active path is now `terraform apply` in
`nova-platform-ops` .
## 3. M1.5 verification gate (12-item spike)
The M1.5 gate is the 12-item spike from PLAN.md "Happy Path" (spec
§3.3 Edge 5 items 1-8 + grill CF-1 items 9-12). **3 consecutive green
rebuilds are required** in `nova-platform-ops` CI.
The 12 items:
1. `kj` v0.0.3 (pinned SHA in `platform/abac/kj-version.txt` ) compiles
with `CGO_ENABLED=0 GOOS=linux GOARCH=amd64` .
2. Resulting binary reports `file kj → ELF 64-bit LSB executable,
x86-64, statically linked, no shared library` (KJ-STATIC).
3. Container image built from
` public.ecr.aws/lambda/python:3.12-al2023` with the binary copied
to ` /opt/kj/kj`, ` chmod 0555`, owned by ` sbx_user:1051`.
4. Lambda runtime ` python3.12` executes
` nova_idp_token_vend.handler`; the handler invokes
` subprocess.run(['/opt/kj/kj', 'apply', ...])` and parses stdout
JSON.
5. ` tests/test_idp_auth.py` passes against the live image in moto-DDB.
6. ` tests/test_kms_roundtrip.py` passes against the live KMS key
` alias/nova-oidc-signing` (REQ-362 path — covered-reference).
7. End-to-end: known PAT → known ABAC-allowed action → signed OIDC
token → ` jose` verification → green. Known PAT + ABAC-denied action
→ 403 with deny reason logged (INV-17 fail-closed).
8. Image URI is recorded in Terraform state and in this operator
guide (§18, REQ-371 Result row).
9. **(grill CF-1) JWKS-EDGE-ONLY:** direct JWKS Function URL GET
(bypassing CloudFront) returns **403**; via-CloudFront GET returns
**200** (INV-18). Proves ` AuthType: AWS_IAM` + OAC pinning.
10. **(grill CF-1) IAM-NARROW:** ` aws iam get-role-policy` on the
Gitea OIDC role asserts no ` Action: "*"` and no ` Resource: "*"`
(REQ-360).
11. **(grill CF-1) TFM-HITL:** a ` terraform apply`
` workflow_dispatch` triggered by the PR author is **rejected**
(exit non-zero); a dispatch by a distinct user proceeds (REQ-357).
12. **(grill CF-1) rollback drill:** revert ` nova_platform_version`
pin → ` terraform apply` → assert the prior ECR digest runs
(D-236, guards against ECR tag mutability).
If items 1-7 fail three consecutive rebuilds, M2a activates (§5,
REQ-363b Fargate toggle) with the same ECR image — no warmup hit
because the standby is always running the same digest (KJ-LOCKSTEP).
### HARD P6 ship gate (grill CF-1/G-2.1)
P6 must not ship ` v1.28.6` until the operator attests the M1.5 result
in the row below. The operator fills this in **after** the gate passes
3 consecutive green rebuilds in ` nova-platform-ops` CI. P6 audit
verifies the row exists + is non-empty.
#### M1.5 Verification Gate Result
| Rebuild # | Run ID / commit SHA | All 12 items green? | Attestor identity | Attested at (UTC) |
|-----------|---------------------|---------------------|-------------------|-------------------|
| 1 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
| 2 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
| 3 | _(operator fills)_ | _(yes/no)_ | _(operator fills)_ | _(operator fills)_ |
> **P6 audit rule:** all three rows must be present, all 12 items
> green on each, the three run IDs/SHAs distinct (consecutive
> rebuilds, not one run copied thrice), and the attestor identity
> non-empty. Empty or red → P6 blocks → escalate.
## 4. M2 operational handoff loop
M2 is the steady-state operational loop for rolling out an engineering
change after M1.5 is green. The loop is the tag-pin bump → plan →
HITL → apply cycle (D-235, D-238).
1. **Tag-pin bump** — the operator opens a PR in ` nova-platform-ops`
bumping ` local.nova_platform_version` (e.g. ` v1.29.3` → ` v1.29.4`)
+ ` local.kj_source_sha` (the ` kj` source SHA from
` platform/abac/kj-version.txt` at the new tag). Both pins move
together — there is one ECR image identity (KJ-LOCKSTEP, REQ-371).
2. **` terraform plan`** — CI runs ` terraform plan` on the PR. The
KJ-LOCKSTEP precondition (a ` lifecycle.precondition` on both
image-bearing resources — the Lambda ` image_uri` and the Fargate
task ` container_definitions[0].image`) checks that both
` image_uri` attributes resolve to the **same ECR digest** via
` data.aws_ecr_image.kj_image`. If the two diverge, the plan fails
closed — no second pipeline, no second SHA pin (D-238).
3. **HITL approval** — a Gitea Actions approver **distinct from the
PR author** approves the apply (TFM-HITL, REQ-357). Self-approval
is rejected (M1.5 item 11).
4. **` terraform apply`** — on approval, the apply updates both
` aws_lambda_function.nova_idp_token_vend.image_uri` and
` aws_ecs_task_definition.kj.container_definitions[0].image` to the
same ECR digest. The Lambda image + the Fargate task redeploy to
the same digest in one apply. Zero diff on KMS, DDB, IAM, edge
(the only change is the image reference).
**Verification:** after the apply, ` aws lambda get-function
--function-name nova-idp-token-vend --query Configuration.Code.ImageUri`
and ` aws ecs describe-tasks` on the Fargate task both report the same
digest. This is the M2 acceptance gate (PLAN §UX Acceptance Criteria
3) + the REQ-371 Result row in §18.
## 5. M2a Fargate activation (conditional)
M2a activates **only if M1.5 fails 3 consecutive rebuilds** (D-236).
It is the REQ-363b Fargate toggle — an always-warm minimal Fargate
standby running the **same ECR image** as the Lambda (KJ-LOCKSTEP).
Because the standby is always running the same digest as the Lambda,
activating M2a is **not** a warmup hit — the standby is already
serving ` GET /health → 200` every 10s (KJ-WARMUP-HEALTH, §15). The
toggle repoints token-vend traffic from the Lambda to the Fargate
task; no cold start, no image pull.
If both the Lambda path and the Fargate path fail (M1.5 items 1-7
fail on both substrates), the operator escalates — Nova-idp ships in
read-only partial mode (no token issuance) until ` kj` is verified
(Q7 carry-forward, spec §7.7).
**Fargate sunset discipline (D-237):** the standby (~$15-20/month,
§7) may not be deleted unless REQ-363 has been green in production
for **≥30 consecutive days**. Sunset requires an architecture review.
See §15 for the health-check procedure.
## 6. Rollback procedure (D-236)
Rollback is a tag-pin revert — the same mechanism as the M2 rollout
(§4), in reverse.
1. **Revert ` nova_platform_version`** in ` nova-platform-ops` to the
prior tag (e.g. ` v1.29.4` → ` v1.29.3`). Open a PR, get HITL
approval (TFM-HITL, same as rollout).
2. **` terraform apply`** — the apply reverts both the Lambda
` image_uri` and the Fargate task ` image` to the prior ECR digest.
The prior tag's artifacts remain downloadable (GitHub Releases are
append-only per tag, REQ-354 AC 2) — no artifact is re-built.
3. **Verify** the prior digest is running:
` ``sh
aws lambda get-function --function-name nova-idp-token-vend \
--query Configuration.Code.ImageUri --output text
# → <account>.dkr.ecr.us-east-1.amazonaws.com/nova-kj@sha256:<prior-digest>
` ``
This is the M1.5 item 12 rollback drill + the operational rollback
procedure. It guards against ECR tag mutability (RESEARCH §2) — the
digest is immutable even if a tag is re-pushed.
## 7. Cost section
Monthly estimate for the ` nova-platform-ops` live platform in account
` 581513795199` (pilot volume):
| Resource | Quantity | Est. monthly | Notes |
|----------|----------|-------------|-------|
| WAF WebACL (CloudFront-scoped) | 1 | ~$5-10 | + per-request; REQ-365 |
| Fargate standby (0.25 vCPU, 512 MB) | 1 task | ~$15-20 | REQ-363b AC 4; largest line item |
| KMS asymmetric key | 1 | ~$1 | ` alias/nova-oidc-signing`, ECC_NIST_P256 |
| DynamoDB (on-demand, 6 tables) | 6 | ~$2 | §11 tables |
| Lambda invocations (3 Lambdas) | 3 | ~$2 | low pilot volume |
| ECR image storage | ~100 MB | <$1 | the ` kj` image |
| S3 state bucket + access logs | 1 | <$1 | ` nova-tfstate-*` |
| CloudFront + ACM + Route53 | 1 distribution | ~$1 | ACM free for CloudFront-attached |
| **Total** | | **~$30-40/month** | |
**Fargate standby is the largest line item** (~$15-20/month, REQ-363b
AC 4). It is explicitly documented here with the D-237 sunset
discipline (§5, §15): ≥30 consecutive days green before deletion +
architecture review. Do not delete the standby to save ~$15/month
without that review — it is the defensive fallback for the ` kj`
substrate.
## 8. Artifact-mirror fallback (Edge 6)
When the Gitea ` act_runner` in ` nova-platform-ops` CI cannot reach
GitHub Releases (network partition, egress restriction, GitHub
outage), the operator mirrors the artifact bundle locally by SHA-256.
**Procedure:**
1. **Download the GitHub Release bundle** for the target tag
(` v1.29.x`) from a machine that can reach GitHub Releases:
` ``sh
gh release download v1.29.0 \
--repo continuous-intelligence/acdl \
--pattern 'nova-lambda-token-vend-*.zip' \
--pattern 'nova-cli-layer-*.zip' \
--pattern 'nova-*-py3-none-any.whl' \
--dir ./artifact-cache
` ``
2. **Verify SHA-256** against the release body (each artifact's
SHA-256 is listed in the GitHub Release body, REQ-354):
` ``sh
sha256sum ./artifact-cache/nova-lambda-token-vend-v1.29.0.zip
# → must match the SHA-256 in the release body
` ``
3. **Place the bundle in the operator's local artifact cache** — a
directory the ` act_runner` can read (e.g. a Gitea-lfs-tracked path
in ` nova-platform-ops`, or an S3 bucket the runner can reach).
4. **Reference by SHA-256 in the terraform variables** — the
` nova-platform-ops` Terraform accepts an override for the artifact
source: ` nova_artifact_mirror_sha256 = "<sha256>"`. When set, the
` data` sources resolve from the local cache by SHA-256 instead of
from GitHub Releases. Unset → resume GitHub Releases resolution.
This fallback is for CI continuity only; the live ` terraform apply`
still resolves the ECR image by digest (KJ-LOCKSTEP), which is
independent of GitHub Releases availability.
## 9. KMS rotation (D-234)
The OIDC signing key ` alias/nova-oidc-signing` is provisioned with
` KeySpec: ECC_NIST_P256`, ` KeyUsage: SIGN_VERIFY`, on a **90-day
rotation cadence** (matches per-stack CMK rotation per D-069).
**Verify the key spec + rotation status:**
` ``sh
aws kms describe-key --key-id alias/nova-oidc-signing \
--query 'KeyMetadata.[KeySpec,KeyUsage,Description]' --output text
# → ECC_NIST_P256 SIGN_VERIFY <description>
` ``
**Apply a rotation policy** (key re-point, not key deletion — the
alias moves to a new key while the old key stays valid during the
overlap window, §17 JWKS-ROTATION):
1. Create the new key (same spec):
` ``sh
NEW_KEY=$(aws kms create-key \
--key-spec ECC_NIST_P256 \
--key-usage SIGN_VERIFY \
--description "nova-oidc-signing-$(date +%Y%m%d)" \
--query KeyId --output text)
` ``
2. Re-point the alias:
` ``sh
aws kms update-alias --alias-name alias/nova-oidc-signing \
--target-key-id "$NEW_KEY"
` ``
3. Apply the rotation policy (the key policy grants ` kms:Sign` to the
token-vend Lambda role + ` kms:GetPublicKey` to the JWKS Lambda
role):
` ``sh
aws kms put-key-policy --key-id "$NEW_KEY" \
--policy-name default --policy file://kms-signing-key-policy.json
` ``
4. After the 24-hour overlap window (§17), disable + schedule deletion
of the old key:
` ``sh
aws kms disable-key --key-id "<old-key-id>"
aws kms schedule-key-deletion --key-id "<old-key-id>" \
--pending-window-in-days 7
` ``
5. Verify the new key is active: ` nova idp setup --verify` (the KMS
round-trip test, REQ-362).
**Audit:** emit a ` kms.key_rotated` event with ` old_key_id`,
` new_key_id`, ` rotated_at`.
## 10. JWKS reachability via CloudFront edge (D-233, INV-18)
The JWKS endpoint is the **only public read surface** of the live
platform (INV-18, D-233). All other platform endpoints gate with
` AuthType: AWS_IAM`. CloudFront + OAC pinning replaces direct Lambda
Function URL exposure.
**Critical pitfall (RESEARCH §4):** the JWKS Function URL
` AuthType` MUST be ` AWS_IAM`, NOT ` NONE`. A common mistake is to set
` AuthType: NONE` on the Function URL (thinking CloudFront is the
gate) — that exposes the JWKS endpoint directly to the internet,
bypassing OAC. The correct configuration:
| Setting | Value |
|---------|-------|
| Function URL ` AuthType` | ` AWS_IAM` (NOT ` NONE`) |
| CloudFront OAC ` OriginAccessControlOriginType` | ` lambda` |
| CloudFront OAC ` SigningBehavior` | ` always` |
| Lambda resource policy | ` lambda:InvokeFunctionUrl` scoped to the CloudFront distribution ARN |
With ` AuthType: AWS_IAM` + OAC ` always` signing, CloudFront signs
every origin request with SigV4; a direct Function URL request has no
SigV4 signature → 403. Only CloudFront can reach the origin.
**Verification (M1.5 item 9):**
` ``sh
# Via CloudFront → 200
curl -sI https://<jwks-domain>/.well-known/jwks.json | head -1
# → HTTP/2 200
# Direct Function URL → 403
curl -sI "<raw-function-url>/.well-known/jwks.json" | head -1
# → HTTP/2 403
` ``
If the direct Function URL returns 200, the ` AuthType` is ` NONE` —
fix the Terraform + re-apply before declaring M1.5 green.
## 11. PITR restore (data-engineer section)
DynamoDB point-in-time recovery (PITR) is enabled on every identity +
contract table. PITR lets you restore a table to any second in the
last **35 days** (the AWS retention window).
**Tables with PITR enabled:**
| Table | Purpose |
|-------|---------|
| ` nova-contracts` | contract ingestor records |
| ` nova-change-requests` | change request ledger |
| ` nova-outbox` | audit outbox |
| ` nova-users` | Nova-idp users (Argon2id hashes) |
| ` nova-sessions` | Nova-idp sessions (TTL ` expires_at`) |
| ` nova-pats` | Nova-idp PATs (revocation strong-read, D-229) |
**Enable PITR (on a new/restored table — PITR does not carry over
from the source):**
` ``sh
aws dynamodb update-continuous-backups \
--table-name <table> \
--point-in-time-recovery-specification PointInTimeRecoveryEnabled=true
` ``
**Restore a table to a point in time** (PITR never overwrites the
source — restore to a NEW table, then repoint):
` ``sh
RESTORE_TO=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)
aws dynamodb restore-table-to-point-in-time \
--source-table-name <table> \
--target-table-name <table>-restored \
--restore-date-time "$RESTORE_TO" \
--billing-mode-restore-as-is
# After the restore completes (status ACTIVE), repoint the app:
# - update the stack env var to the restored table name, or
# - rename: delete <table>, then update-table --new-table-name <table>
# Then re-enable PITR on the restored table (see above).
` ``
**Verify PITR is enabled on all tables:**
` ``sh
for t in nova-contracts nova-change-requests nova-outbox \
nova-users nova-sessions nova-pats; do
aws dynamodb describe-continuous-backups --table-name "$t" \
--query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus' \
--output text
done
# → ENABLED (x6)
` ``
Restores older than 35 days are impossible — for longer retention,
export to S3 via the on-demand export or a scheduled AWS Backup plan.
## 12. DynamoDB import addresses (REQ-361, covered-reference)
The ` importable-resources.tf` map in ` nova-platform-ops` lists the
existing live resources that ` terraform import` adopts at M1 cutover
(§2 step 4). Re-import exits non-zero with
` resource_already_imported` (IMPORT-IDEMPOTENT); CI import treats this
as idempotent success.
| Terraform address | AWS resource | Type |
|-------------------|--------------|------|
| ` aws_s3_bucket.nova_tfstate` | ` nova-tfstate-581513795199-us-east-1` | S3 bucket (state backend) |
| ` aws_dynamodb_table.nova_contracts` | ` nova-contracts` | DynamoDB table |
| ` aws_dynamodb_table.nova_change_requests` | ` nova-change-requests` | DynamoDB table |
| ` aws_dynamodb_table.nova_outbox` | ` nova-outbox` | DynamoDB table |
| ` aws_iam_role.acdl_act_runner` | ` acdl-act-runner-role` | IAM role (reused, spec §7.6) |
| ` aws_kms_key.<per_stack_cmk>` | per-stack CMKs | KMS key (one per stack) |
The identity tables (` nova-users`, ` nova-sessions`, ` nova-pats`) are
**new** resources added at M1 (§2 step 6), not imported — they do
not yet exist in the account at M1.
## 13. PAT revocation (D-229)
PAT revocation has a **60s SLO**: the token-vend Lambda does a
strongly-consistent DynamoDB read (` ConsistentRead=True`) on every
token-vend request. A revoked PAT is reflected on the next vend,
within 60s P95.
**Verify a PAT's revocation status (strong read):**
` ``sh
aws dynamodb get-item \
--table-name nova-pats \
--key '{"jti":{"S":"<pat-id>"}}' \
--consistent-read \
--query 'Item.status.S' --output text
# → active (still valid)
# → revoked (next token-vend returns 403)
` ``
**Revoke a PAT at the DDB level** (emergency — when the CLI is
unavailable; the ` jti` is known but the raw PAT is not):
` ``sh
aws dynamodb update-item \
--table-name nova-pats \
--key '{"jti":{"S":"<pat-id>"}}' \
--update-expression "SET #s = :r" \
--expression-attribute-names '{"#s":"status"}' \
--expression-attribute-values '{":r":{"S":"revoked"}}'
` ``
The item is **retained** (not deleted) so the audit trail is intact —
only ` status` flips from ` active` to ` revoked`. The next ` token-vend`
call with that ` jti` returns ` 403 pat_revoked` immediately (D-229:
the strong read is synchronous).
## 14. Edge configuration (REQ-364/365/366, covered-reference)
The edge stack fronts the JWKS Lambda with CloudFront + WAF + ACM +
Route53. This is the public read surface (§10, INV-18).
### CloudFront + OAC (REQ-364)
- Distribution origin = the JWKS Lambda Function URL.
- OAC: ` OriginAccessControlOriginType: lambda`,
` SigningBehavior: always` (§10).
- Cache behavior: ` Cache-Control: max-age=3600` honored (JWKS-SLO).
### WAF WebACL (REQ-365)
- Scope: ` CLOUDFRONT` (the WebACL is in ` us-east-1`, the only region
for CloudFront-scoped WebACLs).
- Rate-based rule: ` RateBasedStatement` with ` Limit: 3000`,
` AggregateKeyType: IP`, ` EvaluationWindowSec: 300` (3000 requests
per 5 minutes per IP).
- Managed rules: ` AWSManagedRulesCommonRuleSet` (the AWS managed rule
group for common attacks).
### ACM certificate (REQ-366)
- Certificate in ` us-east-1` (CloudFront requires the cert in
us-east-1).
- DNS validation (a CNAME record per validation record is written to
Route53). The cert status MUST be ` ISSUED` (not
` PENDING_VALIDATION`) before the CloudFront distribution can serve
the domain.
### Route53 (REQ-366)
- An A-alias record pointing to the CloudFront distribution's domain
name.
### ` route53_record_not_resolvable` debugging
If the JWKS domain does not resolve (` route53_record_not_resolvable`
or ` NXDOMAIN`):
1. **Check ACM cert status:**
` ``sh
aws acm describe-certificate --certificate-arn <arn> \
--query 'Certificate.Status' --output text
# → must be ISSUED, not PENDING_VALIDATION
` ``
If ` PENDING_VALIDATION`, the DNS validation CNAME records are not
in Route53 (or not propagated). Re-apply the validation records +
wait for AWS to validate (typically minutes).
2. **Check CloudFront status:**
` ``sh
aws cloudfront get-distribution --id <id> \
--query 'Distribution.Status' --output text
# → must be Deployed
` ``
If ` InProgress`, wait for the deployment to finish. CloudFront
deployments take ~5-15 minutes.
3. **Check the Route53 alias record** points to the CloudFront
distribution domain name (not the Function URL).
## 15. Fargate standby health (KJ-WARMUP-HEALTH, REQ-363b)
The Fargate standby is the always-warm minimal defensive fallback
(REQ-363b). It runs the **same ECR image** as the Lambda (KJ-LOCKSTEP,
REQ-371) — so it is always running the current digest, never a stale
one.
**Health probe:** ` GET /health → 200` every **10s**
(KJ-WARMUP-HEALTH).
**Failure handling:** 3 consecutive probe failures → alert + the
token-vend path **fails closed** (no signing). The standby does not
silently degrade — if it is not healthy, token-vend does not fall
back to it; it fails closed (INV-17 ABAC discipline extended to the
substrate).
**Verify the standby is ` READY` before M1 cutover:**
` ``sh
# The Fargate task health check (target group)
aws elbv2 describe-target-health \
--target-group-arn <tg-arn> \
--query 'TargetHealthDescriptions[0].TargetHealth.State' --output text
# → healthy
# Direct probe
curl -sI https://<fargate-endpoint>/health | head -1
# → HTTP/1.1 200
` ``
**Fargate sunset discipline (D-237):** the standby may not be deleted
unless REQ-363 has been green in production for **≥30 consecutive
days**. Sunset requires an architecture review. Do not delete the
standby to save ~$15/month (§7) without that review — it is the
defensive fallback for the ` kj` substrate.
## 16. IAM scope (IAM-NARROW, REQ-360, covered-reference)
The Gitea OIDC role for ` act_runner` (reused ` acdl-act-runner-role`,
spec §7.6) is bounded per REQ-360. **No ` Action: "*"` or ` Resource:
"*"`** (IAM-NARROW).
The scope covers only:
| Action | Scope | Why |
|--------|-------|-----|
| ` kms:*` | customer-managed keys in ` 581513795199` | KMS signing + rotation |
| ` dynamodb:*` | tables prefixed ` nova-` | identity + contract tables |
| ` lambda:*` | functions prefixed ` nova-` | the 3 Nova-idp Lambdas |
| ` s3:*` | buckets prefixed ` nova-` | state bucket + artifact cache |
| ` cloudfront:*` | tagged resources | the JWKS distribution |
| ` wafv2:*` | tagged resources | the WebACL |
| ` acm:*` | tagged resources | the JWKS cert |
| ` route53:*` | tagged resources | the JWKS alias |
| ` iam:PassRole` | roles tagged ` nova-ops-only` | pass roles to Lambda/ECS only |
**Verify (M1.5 item 10):**
` ``sh
aws iam get-role-policy --role-name acdl-act-runner-role \
--policy-name <policy-name> --query 'PolicyDocument' --output json \
| jq '.Statement[].Action, .Statement[].Resource'
# → no "*" in either list
` ``
If ` Action: "*"` or ` Resource: "*"` appears, the IAM policy is too
broad — fix the Terraform + re-apply before declaring M1.5 green.
## 17. JWKS-ROTATION
On KMS key rotation (§9), **both old + new public keys** are
published in the JWKS during a **24-hour overlap window**. The old
key is removed from the JWKS only after consumers pick up the new
one.
- During the overlap: the JWKS Lambda lists all keys the alias has
pointed at that are still enabled. Already-issued OIDC tokens
(signed with the old key) keep verifying until they expire (OIDC
TTL default 15 min; PAT TTL ≤ 24h dev / ≤ 1h service-account).
- **Do not disable the old key until at least the max PAT TTL (24h)
has elapsed.**
- After the overlap, the old key is removed from the JWKS + disabled +
scheduled for deletion (§9 step 4).
This is JWKS-ROTATION (NFR) — the rotation is non-disruptive because
consumers cache the JWKS for up to ` max-age=3600` (1h, JWKS-SLO) and
re-fetch within that window, picking up both keys during the overlap.
## 18. Cutover Gates (grill CF-2/G-5)
Each covered-reference REQ has a cutover gate (M1/M1.5/M2) with a
verification command + a **"Result" column**. The operator fills the
Result column after running the gate in ` nova-platform-ops` CI.
**P6 audit verifies every covered-reference REQ has a non-empty,
green Result.** Empty or red → P6 blocks (grill CF-2/G-5).
| REQ | Gate | Verification command | Result |
|-----|------|----------------------|--------|
| REQ-355 | M1 | ` terraform plan` resolves ` data.aws_ecr_image.kj_image` from ` local.nova_platform_version` + ` local.kj_source_sha`; both image_uri attributes present | _(operator fills: green/red + run ID/SHA + attestor)_ |
| REQ-356 | M1 | ` terraform plan` exit 0 (no drift) on a clean checkout of ` main`; exit 2 → ` drift_detected` fails the apply workflow | _(operator fills)_ |
| REQ-357 | M1.5 | ` terraform apply` ` workflow_dispatch` triggered by PR author → rejected; distinct user → proceeds (M1.5 item 11) | _(operator fills)_ |
| REQ-358 | M2 | bump ` nova_platform_version` → ` terraform apply` → ` aws lambda get-function ... ImageUri` ` CodeSha256` matches the artifact SHA-256 from the GitHub Release body | _(operator fills)_ |
| REQ-359 | M1 | ` git -C nova-platform-ops remote -v` shows only the Gitea private remote (no GitHub mirror); Gitea repo ` private: true` | _(operator fills)_ |
| REQ-360 | M1.5 | ` aws iam get-role-policy` on the OIDC role asserts no ` Action: "*"` + no ` Resource: "*"` (M1.5 item 10, §16) | _(operator fills)_ |
| REQ-361 | M1 | ` terraform import` on each address in ` importable-resources.tf` (§12) succeeds; re-import exits ` resource_already_imported` → CI treats as idempotent success (IMPORT-IDEMPOTENT) | _(operator fills)_ |
| REQ-362 | M1.5 | ` nova idp setup --verify` (KMS round-trip) against ` alias/nova-oidc-signing` (` ECC_NIST_P256`, ` SIGN_VERIFY`) → ` {"passed":true}` (M1.5 item 6) | _(operator fills)_ |
| REQ-363 | M1.5 | ` nova_idp_token_vend.handler` invokes ` subprocess.run(['/opt/kj/kj','apply',...])` on the live image; ` file(1)` reports ` statically linked` (M1.5 items 2-4, KJ-STATIC) | _(operator fills)_ |
| REQ-363b | M1.5 | Fargate standby ` GET /health → 200` every 10s (KJ-WARMUP-HEALTH); same ECR digest as the Lambda (KJ-LOCKSTEP); activates only if M1.5 items 1-7 fail 3×
| REQ-364 | M1.5 | direct JWKS Function URL → 403; via-CloudFront → 200 (M1.5 item 9, §10, INV-18) | _(operator fills)_ |
| REQ-365 | M1 | ` aws wafv2 get-web-acl` shows ` RateBasedStatement` Limit 3000, AggregateKeyType IP, EvaluationWindowSec 300 + ` AWSManagedRulesCommonRuleSet`; Scope CLOUDFRONT in us-east-1 (§14) | _(operator fills)_ |
| REQ-366 | M1 | ` aws acm describe-certificate` Status ` ISSUED`; Route53 A-alias resolves to the CloudFront distribution domain (§14) | _(operator fills)_ |
| REQ-371 | M2 | after ` terraform apply`, both ` aws_lambda_function.nova_idp_token_vend.image_uri` and ` aws_ecs_task_definition.kj.container_definitions[0].image` report the same ECR digest (KJ-LOCKSTEP precondition green at plan) | _(operator fills)_ |
> **P6 audit rule (grill CF-2/G-5):** every row's Result column must
> be non-empty + green. An empty or red Result blocks the milestone
> ship. The operator attestation is the acdl-side evidence surface;
> the live verification runs in ` nova-platform-ops` CI.
---
## Appendix — quick reference
| Procedure | Cadence / trigger | Section |
|-----------|-------------------|---------|
| Day-0 cutover (M1) | one-time | §2 |
| M1.5 verification gate | one-time (3 consecutive green rebuilds) | §3 |
| M2 operational handoff | per engineering change (tag-pin bump) | §4 |
| M2a Fargate activation | conditional (M1.5 fails 3×
| Rollback | on regression | §6 |
| Artifact-mirror fallback | on GitHub Releases unreachable | §8 |
| KMS rotation | every 90 days | §9 |
| JWKS-ROTATION overlap | on each KMS rotation (24h window) | §17 |
| PITR restore | on data loss / corruption (35-day window) | §11 |
| Emergency PAT revocation | on compromise (DDB-level, immediate) | §13 |
| Fargate standby health check | continuous (every 10s) | §15 |
| Fargate sunset | ≥30 consecutive days green + architecture review | §5, §15 |
| ` route53_record_not_resolvable` debug | on JWKS domain not resolving | §14 |
| Cutover gate attestation | at M1/M1.5/M2 (operator fills Result column) | §18 |
CIAgent Orchestrator