continuous-intelligence/acdl
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 128 Wiki Activity
Files
2ea9fb4e51a78bffefbbe3d31fa451ed5ac9193a
acdl/docs/operator-guide-platform-ops.md
T
CIAgent Orchestrator 932923ee99
Nova Slides Render / render (push) Failing after 22s
Details
merge(milestone): v1.29 Reposplit + Identity Layer Bring-Live to main (release v1.28.6) 
---ci---
project: acdl
phase: 6
milestone: v1.29
status: complete
---/ci---
2026-08-20 05:29:46 +00:00

33 KiB
Raw Blame History

Operator Guide — Nova Platform Ops (nova-platform-ops)

REQ-OPS-GUIDE — the operator-facing runbook for the nova-platform-ops Terraform repo. This is the verification surface for the covered-reference REQs (355-366, 371): their cutover gates (M1/M1.5/M2) are documented in §18 below, and each REQ has a "Result" column that the operator fills in after running the gate. P6 audit verifies every covered-reference REQ has a non-empty, green Result (grill CF-2/G-5). HARD P6 ship gate: §3 contains the operator-attested "M1.5 Verification Gate Result" row (grill CF-1/G-2.1) — the milestone does not ship until that row is filled.

Audience: platform operators / SREs running the live Nova platform in AWS account 581513795199. For the developer auth flows, see docs/developer-guide-auth.md; for the legacy CloudFormation path, see docs/archive/nova-idp-cfn-v1.28.md.

1. Overview + reposplit rationale

Nova's platform operations live in a dedicated, Gitea-private Terraform repository — nova-platform-ops — separate from the engineering repo acdl/acdl. The split is grounded in Vision §4 (Domain Boundaries):

The platform begins where the artifact is compiled and ends where it runs in production under operational guardrails.

That is two distinct disciplines with two distinct ownership surfaces:

Discipline Ends Begins Repo Surface
Engineering at the compiled artifact — acdl/acdl (GitHub) publish.yml + GitHub Releases
Operations — at the live platform under guardrails nova-platform-ops (Gitea-private) Terraform modules

Scope split (CLARIFY Q-P1, D-232):

  • acdl/acdl authors publish.yml (the artifact publish pipeline) + the artifacts themselves (Lambda zip, layer wheel, Python wheel, ECR container image with the static kj binary). Each tag v1.29.x produces a GitHub Release with SHA-256-verified artifacts (REQ-354).
  • nova-platform-ops authors the Terraform modules (networking/kms/identity/contract-ingest/bootstrap/edge) that bring those artifacts live in 581513795199.

The handoff between the two repos is the tag-pin (D-235): nova-platform-ops declares local.nova_platform_version + local.kj_source_sha and resolves substrates through a single data.aws_ecr_image.kj_image. The engineering repo never knows which tag is live; the ops repo never authors artifacts. Vision §6 immutability + Vision §5 narrow interfaces.

The covered-reference REQs (355-366, 371) are authored in nova-platform-ops (out-of-band). CIAgent in acdl tracks them for milestone completeness; their verification surface is the cutover gates in §18 of this guide.

2. Day-0 cutover procedure (M1)

The M1 cutover is the one-time conversion of the live AWS account 581513795199 from CloudFormation-managed (or manually-created) resources to Terraform-managed resources in nova-platform-ops. It is conditional on the M1.5 verification gate passing (§3, Q7 carry-forward, D-236).

The 10-step Journey 2 (spec §3.2):

  1. Create nova-platform-ops in Gitea — private (private: true, OPER-PRIV, REQ-359), no GitHub mirror. The repo is operator-owned; CIAgent has no presence there.

  2. Commit the initial Terraform structure — the module tree (networking/kms/identity/contract-ingest/bootstrap/edge)

    • importable-resources.tf (§12) + versions.tf + backend.tf (S3 state in the imported bucket).

  3. terraform init — initialize the S3 backend against the state bucket (nova-tfstate-581513795199-us-east-1, imported in step 5). The bucket is created manually once (operator's secure scratch, spec §7.1, D-235) before Terraform adopts it.

  4. terraform import for existing live resources — adopt the resources that already exist in 581513795199 into Terraform state without recreating them. The import map is in importable-resources.tf (§12):

    • aws_s3_bucket.nova_tfstate ← nova-tfstate-581513795199-us-east-1
    • aws_dynamodb_table.nova_contracts ← nova-contracts
    • aws_dynamodb_table.nova_change_requests ← nova-change-requests
    • aws_dynamodb_table.nova_outbox ← nova-outbox
    • aws_iam_role.acdl_act_runner ← acdl-act-runner-role
    • per-stack CMKs (KMS keys)

    Re-import exits non-zero with resource_already_imported (IMPORT-IDEMPOTENT, REQ-361). CI import treats this as idempotent success — the import workflow greps the error stream + exits 0 on that string.

  5. (state bucket is imported in step 4) — listed here for sequence clarity; the S3 state bucket is the first import because the backend depends on it.

  6. Add new resources that do not yet exist in the account:

    • KMS alias alias/nova-oidc-signing (§9, D-234).
    • Identity DynamoDB tables: nova-users, nova-sessions, nova-pats (§11).
    • JWKS Function URL with AuthType: AWS_IAM (NOT NONE — §10, INV-18, RESEARCH §4 critical pitfall).
    • CloudFront distribution + OAC + WAF WebACL + ACM certificate + Route53 alias (§14, REQ-364/365/366).

  7. terraform plan — expect zero diff on the imported resources (they are already in their desired state) + a pure-add diff on the new resources. If the plan shows a diff on an imported resource, the import map or the Terraform resource block is wrong — fix before apply. DRIFT-DETECT (REQ-356): terraform plan exit 2 (drift) fails the apply workflow; manual reconciliation required.

  8. HITL approval — terraform apply against main requires a Gitea Actions approval from a user distinct from the PR author (TFM-HITL, REQ-357, INV-3). Self-approval is rejected: gitea.triggering_actor == pull_request.user.login → apply fails closed (M1.5 item 11).

  9. terraform apply — on approval, the apply creates the new resources + adopts the imported ones. Smoke test (step 10) before declaring M1 done.

  10. Smoke test + CFN→Terraform conversion — verify the live account is in the desired state (JWKS reachable via CloudFront, KMS round-trip, ABAC fail-closed). The CFN template in acdl/acdl/nova/idp/setup.py is archived to docs/archive/nova-idp-cfn-v1.28.md as read-only reference (REQ-369); the active path is now terraform apply in nova-platform-ops.

3. M1.5 verification gate (12-item spike)

The M1.5 gate is the 12-item spike from PLAN.md "Happy Path" (spec §3.3 Edge 5 items 1-8 + grill CF-1 items 9-12). 3 consecutive green rebuilds are required in nova-platform-ops CI.

The 12 items:

  1. kj v0.0.3 (pinned SHA in platform/abac/kj-version.txt) compiles with CGO_ENABLED=0 GOOS=linux GOARCH=amd64.
  2. Resulting binary reports file kj → ELF 64-bit LSB executable, x86-64, statically linked, no shared library (KJ-STATIC).
  3. Container image built from public.ecr.aws/lambda/python:3.12-al2023 with the binary copied to /opt/kj/kj, chmod 0555, owned by sbx_user:1051.
  4. Lambda runtime python3.12 executes nova_idp_token_vend.handler; the handler invokes subprocess.run(['/opt/kj/kj', 'apply', ...]) and parses stdout JSON.
  5. tests/test_idp_auth.py passes against the live image in moto-DDB.
  6. tests/test_kms_roundtrip.py passes against the live KMS key alias/nova-oidc-signing (REQ-362 path — covered-reference).
  7. End-to-end: known PAT → known ABAC-allowed action → signed OIDC token → jose verification → green. Known PAT + ABAC-denied action → 403 with deny reason logged (INV-17 fail-closed).
  8. Image URI is recorded in Terraform state and in this operator guide (§18, REQ-371 Result row).
  9. (grill CF-1) JWKS-EDGE-ONLY: direct JWKS Function URL GET (bypassing CloudFront) returns 403; via-CloudFront GET returns 200 (INV-18). Proves AuthType: AWS_IAM + OAC pinning.
  10. (grill CF-1) IAM-NARROW: aws iam get-role-policy on the Gitea OIDC role asserts no Action: "*" and no Resource: "*" (REQ-360).
  11. (grill CF-1) TFM-HITL: a terraform apply workflow_dispatch triggered by the PR author is rejected (exit non-zero); a dispatch by a distinct user proceeds (REQ-357).
  12. (grill CF-1) rollback drill: revert nova_platform_version pin → terraform apply → assert the prior ECR digest runs (D-236, guards against ECR tag mutability).

If items 1-7 fail three consecutive rebuilds, M2a activates (§5, REQ-363b Fargate toggle) with the same ECR image — no warmup hit because the standby is always running the same digest (KJ-LOCKSTEP).

HARD P6 ship gate (grill CF-1/G-2.1)

P6 must not ship v1.28.6 until the operator attests the M1.5 result in the row below. The operator fills this in after the gate passes 3 consecutive green rebuilds in nova-platform-ops CI. P6 audit verifies the row exists + is non-empty.

M1.5 Verification Gate Result

Rebuild # Run ID / commit SHA All 12 items green? Attestor identity Attested at (UTC)
1 (operator fills) (yes/no) (operator fills) (operator fills)
2 (operator fills) (yes/no) (operator fills) (operator fills)
3 (operator fills) (yes/no) (operator fills) (operator fills)

P6 audit rule: all three rows must be present, all 12 items green on each, the three run IDs/SHAs distinct (consecutive rebuilds, not one run copied thrice), and the attestor identity non-empty. Empty or red → P6 blocks → escalate.

4. M2 operational handoff loop

M2 is the steady-state operational loop for rolling out an engineering change after M1.5 is green. The loop is the tag-pin bump → plan → HITL → apply cycle (D-235, D-238).

  1. Tag-pin bump — the operator opens a PR in nova-platform-ops bumping local.nova_platform_version (e.g. v1.29.3 → v1.29.4)

    • local.kj_source_sha (the kj source SHA from platform/abac/kj-version.txt at the new tag). Both pins move together — there is one ECR image identity (KJ-LOCKSTEP, REQ-371).

  2. terraform plan — CI runs terraform plan on the PR. The KJ-LOCKSTEP precondition (a lifecycle.precondition on both image-bearing resources — the Lambda image_uri and the Fargate task container_definitions[0].image) checks that both image_uri attributes resolve to the same ECR digest via data.aws_ecr_image.kj_image. If the two diverge, the plan fails closed — no second pipeline, no second SHA pin (D-238).

  3. HITL approval — a Gitea Actions approver distinct from the PR author approves the apply (TFM-HITL, REQ-357). Self-approval is rejected (M1.5 item 11).

  4. terraform apply — on approval, the apply updates both aws_lambda_function.nova_idp_token_vend.image_uri and aws_ecs_task_definition.kj.container_definitions[0].image to the same ECR digest. The Lambda image + the Fargate task redeploy to the same digest in one apply. Zero diff on KMS, DDB, IAM, edge (the only change is the image reference).

Verification: after the apply, aws lambda get-function --function-name nova-idp-token-vend --query Configuration.Code.ImageUri and aws ecs describe-tasks on the Fargate task both report the same digest. This is the M2 acceptance gate (PLAN §UX Acceptance Criteria 3) + the REQ-371 Result row in §18.

5. M2a Fargate activation (conditional)

M2a activates only if M1.5 fails 3 consecutive rebuilds (D-236). It is the REQ-363b Fargate toggle — an always-warm minimal Fargate standby running the same ECR image as the Lambda (KJ-LOCKSTEP).

Because the standby is always running the same digest as the Lambda, activating M2a is not a warmup hit — the standby is already serving GET /health → 200 every 10s (KJ-WARMUP-HEALTH, §15). The toggle repoints token-vend traffic from the Lambda to the Fargate task; no cold start, no image pull.

If both the Lambda path and the Fargate path fail (M1.5 items 1-7 fail on both substrates), the operator escalates — Nova-idp ships in read-only partial mode (no token issuance) until kj is verified (Q7 carry-forward, spec §7.7).

Fargate sunset discipline (D-237): the standby (~$15-20/month, §7) may not be deleted unless REQ-363 has been green in production for ≥30 consecutive days. Sunset requires an architecture review. See §15 for the health-check procedure.

6. Rollback procedure (D-236)

Rollback is a tag-pin revert — the same mechanism as the M2 rollout (§4), in reverse.

  1. Revert nova_platform_version in nova-platform-ops to the prior tag (e.g. v1.29.4 → v1.29.3). Open a PR, get HITL approval (TFM-HITL, same as rollout).

  2. terraform apply — the apply reverts both the Lambda image_uri and the Fargate task image to the prior ECR digest. The prior tag's artifacts remain downloadable (GitHub Releases are append-only per tag, REQ-354 AC 2) — no artifact is re-built.

  3. Verify the prior digest is running:

    aws lambda get-function --function-name nova-idp-token-vend \
  --query Configuration.Code.ImageUri --output text
# → <account>.dkr.ecr.us-east-1.amazonaws.com/nova-kj@sha256:<prior-digest>

This is the M1.5 item 12 rollback drill + the operational rollback procedure. It guards against ECR tag mutability (RESEARCH §2) — the digest is immutable even if a tag is re-pushed.

7. Cost section

Monthly estimate for the nova-platform-ops live platform in account 581513795199 (pilot volume):

Resource Quantity Est. monthly Notes
WAF WebACL (CloudFront-scoped) 1 ~$5-10 + per-request; REQ-365
Fargate standby (0.25 vCPU, 512 MB) 1 task ~$15-20 REQ-363b AC 4; largest line item
KMS asymmetric key 1 ~$1 alias/nova-oidc-signing, ECC_NIST_P256
DynamoDB (on-demand, 6 tables) 6 ~$2 §11 tables
Lambda invocations (3 Lambdas) 3 ~$2 low pilot volume
ECR image storage ~100 MB <$1 the kj image
S3 state bucket + access logs 1 <$1 nova-tfstate-*
CloudFront + ACM + Route53 1 distribution ~$1 ACM free for CloudFront-attached
Total ~$30-40/month

Fargate standby is the largest line item (~$15-20/month, REQ-363b AC 4). It is explicitly documented here with the D-237 sunset discipline (§5, §15): ≥30 consecutive days green before deletion + architecture review. Do not delete the standby to save ~$15/month without that review — it is the defensive fallback for the kj substrate.

8. Artifact-mirror fallback (Edge 6)

When the Gitea act_runner in nova-platform-ops CI cannot reach GitHub Releases (network partition, egress restriction, GitHub outage), the operator mirrors the artifact bundle locally by SHA-256.

Procedure:

  1. Download the GitHub Release bundle for the target tag (v1.29.x) from a machine that can reach GitHub Releases:

    gh release download v1.29.0 \
  --repo continuous-intelligence/acdl \
  --pattern 'nova-lambda-token-vend-*.zip' \
  --pattern 'nova-cli-layer-*.zip' \
  --pattern 'nova-*-py3-none-any.whl' \
  --dir ./artifact-cache

  2. Verify SHA-256 against the release body (each artifact's SHA-256 is listed in the GitHub Release body, REQ-354):

    sha256sum ./artifact-cache/nova-lambda-token-vend-v1.29.0.zip
# → must match the SHA-256 in the release body

  3. Place the bundle in the operator's local artifact cache — a directory the act_runner can read (e.g. a Gitea-lfs-tracked path in nova-platform-ops, or an S3 bucket the runner can reach).

  4. Reference by SHA-256 in the terraform variables — the nova-platform-ops Terraform accepts an override for the artifact source: nova_artifact_mirror_sha256 = "<sha256>". When set, the data sources resolve from the local cache by SHA-256 instead of from GitHub Releases. Unset → resume GitHub Releases resolution.

This fallback is for CI continuity only; the live terraform apply still resolves the ECR image by digest (KJ-LOCKSTEP), which is independent of GitHub Releases availability.

9. KMS rotation (D-234)

The OIDC signing key alias/nova-oidc-signing is provisioned with KeySpec: ECC_NIST_P256, KeyUsage: SIGN_VERIFY, on a 90-day rotation cadence (matches per-stack CMK rotation per D-069).

Verify the key spec + rotation status:

aws kms describe-key --key-id alias/nova-oidc-signing \
  --query 'KeyMetadata.[KeySpec,KeyUsage,Description]' --output text
# → ECC_NIST_P256    SIGN_VERIFY    <description>

Apply a rotation policy (key re-point, not key deletion — the alias moves to a new key while the old key stays valid during the overlap window, §17 JWKS-ROTATION):

  1. Create the new key (same spec):

    NEW_KEY=$(aws kms create-key \
  --key-spec ECC_NIST_P256 \
  --key-usage SIGN_VERIFY \
  --description "nova-oidc-signing-$(date +%Y%m%d)" \
  --query KeyId --output text)

  2. Re-point the alias:

    aws kms update-alias --alias-name alias/nova-oidc-signing \
  --target-key-id "$NEW_KEY"

  3. Apply the rotation policy (the key policy grants kms:Sign to the token-vend Lambda role + kms:GetPublicKey to the JWKS Lambda role):

    aws kms put-key-policy --key-id "$NEW_KEY" \
  --policy-name default --policy file://kms-signing-key-policy.json

  4. After the 24-hour overlap window (§17), disable + schedule deletion of the old key:

    aws kms disable-key --key-id "<old-key-id>"
aws kms schedule-key-deletion --key-id "<old-key-id>" \
  --pending-window-in-days 7

  5. Verify the new key is active: nova idp setup --verify (the KMS round-trip test, REQ-362).

Audit: emit a kms.key_rotated event with old_key_id, new_key_id, rotated_at.

10. JWKS reachability via CloudFront edge (D-233, INV-18)

The JWKS endpoint is the only public read surface of the live platform (INV-18, D-233). All other platform endpoints gate with AuthType: AWS_IAM. CloudFront + OAC pinning replaces direct Lambda Function URL exposure.

Critical pitfall (RESEARCH §4): the JWKS Function URL AuthType MUST be AWS_IAM, NOT NONE. A common mistake is to set AuthType: NONE on the Function URL (thinking CloudFront is the gate) — that exposes the JWKS endpoint directly to the internet, bypassing OAC. The correct configuration:

Setting Value
Function URL AuthType AWS_IAM (NOT NONE)
CloudFront OAC OriginAccessControlOriginType lambda
CloudFront OAC SigningBehavior always
Lambda resource policy lambda:InvokeFunctionUrl scoped to the CloudFront distribution ARN

With AuthType: AWS_IAM + OAC always signing, CloudFront signs every origin request with SigV4; a direct Function URL request has no SigV4 signature → 403. Only CloudFront can reach the origin.

Verification (M1.5 item 9):

# Via CloudFront → 200
curl -sI https://<jwks-domain>/.well-known/jwks.json | head -1
# → HTTP/2 200

# Direct Function URL → 403
curl -sI "<raw-function-url>/.well-known/jwks.json" | head -1
# → HTTP/2 403

If the direct Function URL returns 200, the AuthType is NONE — fix the Terraform + re-apply before declaring M1.5 green.

11. PITR restore (data-engineer section)

DynamoDB point-in-time recovery (PITR) is enabled on every identity + contract table. PITR lets you restore a table to any second in the last 35 days (the AWS retention window).

Tables with PITR enabled:

Table Purpose
nova-contracts contract ingestor records
nova-change-requests change request ledger
nova-outbox audit outbox
nova-users Nova-idp users (Argon2id hashes)
nova-sessions Nova-idp sessions (TTL expires_at)
nova-pats Nova-idp PATs (revocation strong-read, D-229)

Enable PITR (on a new/restored table — PITR does not carry over from the source):

aws dynamodb update-continuous-backups \
  --table-name <table> \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

Restore a table to a point in time (PITR never overwrites the source — restore to a NEW table, then repoint):

RESTORE_TO=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)

aws dynamodb restore-table-to-point-in-time \
  --source-table-name <table> \
  --target-table-name <table>-restored \
  --restore-date-time "$RESTORE_TO" \
  --billing-mode-restore-as-is

# After the restore completes (status ACTIVE), repoint the app:
# - update the stack env var to the restored table name, or
# - rename: delete <table>, then update-table --new-table-name <table>
# Then re-enable PITR on the restored table (see above).

Verify PITR is enabled on all tables:

for t in nova-contracts nova-change-requests nova-outbox \
         nova-users nova-sessions nova-pats; do
  aws dynamodb describe-continuous-backups --table-name "$t" \
    --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus' \
    --output text
done
# → ENABLED (x6)

Restores older than 35 days are impossible — for longer retention, export to S3 via the on-demand export or a scheduled AWS Backup plan.

12. DynamoDB import addresses (REQ-361, covered-reference)

The importable-resources.tf map in nova-platform-ops lists the existing live resources that terraform import adopts at M1 cutover (§2 step 4). Re-import exits non-zero with resource_already_imported (IMPORT-IDEMPOTENT); CI import treats this as idempotent success.

Terraform address AWS resource Type
aws_s3_bucket.nova_tfstate nova-tfstate-581513795199-us-east-1 S3 bucket (state backend)
aws_dynamodb_table.nova_contracts nova-contracts DynamoDB table
aws_dynamodb_table.nova_change_requests nova-change-requests DynamoDB table
aws_dynamodb_table.nova_outbox nova-outbox DynamoDB table
aws_iam_role.acdl_act_runner acdl-act-runner-role IAM role (reused, spec §7.6)
aws_kms_key.<per_stack_cmk> per-stack CMKs KMS key (one per stack)

The identity tables (nova-users, nova-sessions, nova-pats) are new resources added at M1 (§2 step 6), not imported — they do not yet exist in the account at M1.

13. PAT revocation (D-229)

PAT revocation has a 60s SLO: the token-vend Lambda does a strongly-consistent DynamoDB read (ConsistentRead=True) on every token-vend request. A revoked PAT is reflected on the next vend, within 60s P95.

Verify a PAT's revocation status (strong read):

aws dynamodb get-item \
  --table-name nova-pats \
  --key '{"jti":{"S":"<pat-id>"}}' \
  --consistent-read \
  --query 'Item.status.S' --output text
# → active  (still valid)
# → revoked (next token-vend returns 403)

Revoke a PAT at the DDB level (emergency — when the CLI is unavailable; the jti is known but the raw PAT is not):

aws dynamodb update-item \
  --table-name nova-pats \
  --key '{"jti":{"S":"<pat-id>"}}' \
  --update-expression "SET #s = :r" \
  --expression-attribute-names '{"#s":"status"}' \
  --expression-attribute-values '{":r":{"S":"revoked"}}'

The item is retained (not deleted) so the audit trail is intact — only status flips from active to revoked. The next token-vend call with that jti returns 403 pat_revoked immediately (D-229: the strong read is synchronous).

14. Edge configuration (REQ-364/365/366, covered-reference)

The edge stack fronts the JWKS Lambda with CloudFront + WAF + ACM + Route53. This is the public read surface (§10, INV-18).

CloudFront + OAC (REQ-364)

  • Distribution origin = the JWKS Lambda Function URL.
  • OAC: OriginAccessControlOriginType: lambda, SigningBehavior: always (§10).
  • Cache behavior: Cache-Control: max-age=3600 honored (JWKS-SLO).

WAF WebACL (REQ-365)

  • Scope: CLOUDFRONT (the WebACL is in us-east-1, the only region for CloudFront-scoped WebACLs).
  • Rate-based rule: RateBasedStatement with Limit: 3000, AggregateKeyType: IP, EvaluationWindowSec: 300 (3000 requests per 5 minutes per IP).
  • Managed rules: AWSManagedRulesCommonRuleSet (the AWS managed rule group for common attacks).

ACM certificate (REQ-366)

  • Certificate in us-east-1 (CloudFront requires the cert in us-east-1).
  • DNS validation (a CNAME record per validation record is written to Route53). The cert status MUST be ISSUED (not PENDING_VALIDATION) before the CloudFront distribution can serve the domain.

Route53 (REQ-366)

  • An A-alias record pointing to the CloudFront distribution's domain name.

route53_record_not_resolvable debugging

If the JWKS domain does not resolve (route53_record_not_resolvable or NXDOMAIN):

  1. Check ACM cert status:

    aws acm describe-certificate --certificate-arn <arn> \
  --query 'Certificate.Status' --output text
# → must be ISSUED, not PENDING_VALIDATION

    If PENDING_VALIDATION, the DNS validation CNAME records are not in Route53 (or not propagated). Re-apply the validation records + wait for AWS to validate (typically minutes).

  2. Check CloudFront status:

    aws cloudfront get-distribution --id <id> \
  --query 'Distribution.Status' --output text
# → must be Deployed

    If InProgress, wait for the deployment to finish. CloudFront deployments take ~5-15 minutes.

  3. Check the Route53 alias record points to the CloudFront distribution domain name (not the Function URL).

15. Fargate standby health (KJ-WARMUP-HEALTH, REQ-363b)

The Fargate standby is the always-warm minimal defensive fallback (REQ-363b). It runs the same ECR image as the Lambda (KJ-LOCKSTEP, REQ-371) — so it is always running the current digest, never a stale one.

Health probe: GET /health → 200 every 10s (KJ-WARMUP-HEALTH).

Failure handling: 3 consecutive probe failures → alert + the token-vend path fails closed (no signing). The standby does not silently degrade — if it is not healthy, token-vend does not fall back to it; it fails closed (INV-17 ABAC discipline extended to the substrate).

Verify the standby is READY before M1 cutover:

# The Fargate task health check (target group)
aws elbv2 describe-target-health \
  --target-group-arn <tg-arn> \
  --query 'TargetHealthDescriptions[0].TargetHealth.State' --output text
# → healthy

# Direct probe
curl -sI https://<fargate-endpoint>/health | head -1
# → HTTP/1.1 200

Fargate sunset discipline (D-237): the standby may not be deleted unless REQ-363 has been green in production for ≥30 consecutive days. Sunset requires an architecture review. Do not delete the standby to save ~$15/month (§7) without that review — it is the defensive fallback for the kj substrate.

16. IAM scope (IAM-NARROW, REQ-360, covered-reference)

The Gitea OIDC role for act_runner (reused acdl-act-runner-role, spec §7.6) is bounded per REQ-360. No Action: "*" or Resource: "*" (IAM-NARROW).

The scope covers only:

Action Scope Why
kms:* customer-managed keys in 581513795199 KMS signing + rotation
dynamodb:* tables prefixed nova- identity + contract tables
lambda:* functions prefixed nova- the 3 Nova-idp Lambdas
s3:* buckets prefixed nova- state bucket + artifact cache
cloudfront:* tagged resources the JWKS distribution
wafv2:* tagged resources the WebACL
acm:* tagged resources the JWKS cert
route53:* tagged resources the JWKS alias
iam:PassRole roles tagged nova-ops-only pass roles to Lambda/ECS only

Verify (M1.5 item 10):

aws iam get-role-policy --role-name acdl-act-runner-role \
  --policy-name <policy-name> --query 'PolicyDocument' --output json \
  | jq '.Statement[].Action, .Statement[].Resource'
# → no "*" in either list

If Action: "*" or Resource: "*" appears, the IAM policy is too broad — fix the Terraform + re-apply before declaring M1.5 green.

17. JWKS-ROTATION

On KMS key rotation (§9), both old + new public keys are published in the JWKS during a 24-hour overlap window. The old key is removed from the JWKS only after consumers pick up the new one.

  • During the overlap: the JWKS Lambda lists all keys the alias has pointed at that are still enabled. Already-issued OIDC tokens (signed with the old key) keep verifying until they expire (OIDC TTL default 15 min; PAT TTL ≤ 24h dev / ≤ 1h service-account).
  • Do not disable the old key until at least the max PAT TTL (24h) has elapsed.
  • After the overlap, the old key is removed from the JWKS + disabled + scheduled for deletion (§9 step 4).

This is JWKS-ROTATION (NFR) — the rotation is non-disruptive because consumers cache the JWKS for up to max-age=3600 (1h, JWKS-SLO) and re-fetch within that window, picking up both keys during the overlap.

18. Cutover Gates (grill CF-2/G-5)

Each covered-reference REQ has a cutover gate (M1/M1.5/M2) with a verification command + a "Result" column. The operator fills the Result column after running the gate in nova-platform-ops CI. P6 audit verifies every covered-reference REQ has a non-empty, green Result. Empty or red → P6 blocks (grill CF-2/G-5).

REQ Gate Verification command Result
REQ-355 M1 terraform plan resolves data.aws_ecr_image.kj_image from local.nova_platform_version + local.kj_source_sha; both image_uri attributes present (operator fills: green/red + run ID/SHA + attestor)
REQ-356 M1 terraform plan exit 0 (no drift) on a clean checkout of main; exit 2 → drift_detected fails the apply workflow (operator fills)
REQ-357 M1.5 terraform apply workflow_dispatch triggered by PR author → rejected; distinct user → proceeds (M1.5 item 11) (operator fills)
REQ-358 M2 bump nova_platform_version → terraform apply → aws lambda get-function ... ImageUri CodeSha256 matches the artifact SHA-256 from the GitHub Release body (operator fills)
REQ-359 M1 git -C nova-platform-ops remote -v shows only the Gitea private remote (no GitHub mirror); Gitea repo private: true (operator fills)
REQ-360 M1.5 aws iam get-role-policy on the OIDC role asserts no Action: "*" + no Resource: "*" (M1.5 item 10, §16) (operator fills)
REQ-361 M1 terraform import on each address in importable-resources.tf (§12) succeeds; re-import exits resource_already_imported → CI treats as idempotent success (IMPORT-IDEMPOTENT) (operator fills)
REQ-362 M1.5 nova idp setup --verify (KMS round-trip) against alias/nova-oidc-signing (ECC_NIST_P256, SIGN_VERIFY) → {"passed":true} (M1.5 item 6) (operator fills)
REQ-363 M1.5 nova_idp_token_vend.handler invokes subprocess.run(['/opt/kj/kj','apply',...]) on the live image; file(1) reports statically linked (M1.5 items 2-4, KJ-STATIC) (operator fills)
REQ-363b M1.5 Fargate standby GET /health → 200 every 10s (KJ-WARMUP-HEALTH); same ECR digest as the Lambda (KJ-LOCKSTEP); activates only if M1.5 items 1-7 fail 3× (§5) (operator fills)
REQ-364 M1.5 direct JWKS Function URL → 403; via-CloudFront → 200 (M1.5 item 9, §10, INV-18) (operator fills)
REQ-365 M1 aws wafv2 get-web-acl shows RateBasedStatement Limit 3000, AggregateKeyType IP, EvaluationWindowSec 300 + AWSManagedRulesCommonRuleSet; Scope CLOUDFRONT in us-east-1 (§14) (operator fills)
REQ-366 M1 aws acm describe-certificate Status ISSUED; Route53 A-alias resolves to the CloudFront distribution domain (§14) (operator fills)
REQ-371 M2 after terraform apply, both aws_lambda_function.nova_idp_token_vend.image_uri and aws_ecs_task_definition.kj.container_definitions[0].image report the same ECR digest (KJ-LOCKSTEP precondition green at plan) (operator fills)

P6 audit rule (grill CF-2/G-5): every row's Result column must be non-empty + green. An empty or red Result blocks the milestone ship. The operator attestation is the acdl-side evidence surface; the live verification runs in nova-platform-ops CI.

Appendix — quick reference

Procedure Cadence / trigger Section
Day-0 cutover (M1) one-time §2
M1.5 verification gate one-time (3 consecutive green rebuilds) §3
M2 operational handoff per engineering change (tag-pin bump) §4
M2a Fargate activation conditional (M1.5 fails 3×) §5
Rollback on regression §6
Artifact-mirror fallback on GitHub Releases unreachable §8
KMS rotation every 90 days §9
JWKS-ROTATION overlap on each KMS rotation (24h window) §17
PITR restore on data loss / corruption (35-day window) §11
Emergency PAT revocation on compromise (DDB-level, immediate) §13
Fargate standby health check continuous (every 10s) §15
Fargate sunset ≥30 consecutive days green + architecture review §5, §15
route53_record_not_resolvable debug on JWKS domain not resolving §14
Cutover gate attestation at M1/M1.5/M2 (operator fills Result column) §18
Reference in New Issue View Git Blame Copy Permalink