acdl/docs/NOVA_AWS_MIGRATION.md

Nova AWS Resource Migration Runbook (REQ-163, P4)

Milestone: v1.15-Nova (Wave 4, P4). Renames every acdl-* AWS resource name → nova-* via Terraform. This is the heaviest Terraform phase of the rebrand and requires a maintenance window.

Plan-validated only. Per A1, NOVA_LIFECYCLE_MODE defaults to plan (no live AWS mutation from CI). terraform validate passes; the live apply steps below are executed by a platform operator during the scheduled maintenance window. Each step has a verification + rollback.

Scope (renamed resources)

AWS resource	Before	After	Strategy
KMS alias	alias/acdl-platform	alias/nova-platform	cheap rename
SNS topic	acdl-sod-halt	nova-sod-halt	recreate
Security group	acdl-ecs-sg	nova-ecs-sg	recreate
Lambda (role/policy/function)	acdl-contract-ingestor	nova-contract-ingestor	recreate
DynamoDB contracts	acdl-contracts	nova-contracts	scan + copy
DynamoDB change-requests	acdl-change-requests	nova-change-requests	scan + copy
Secrets Manager secret	acdl/github-token	nova/github-token	recreate + re-store
ECR repo	acdl-microservice	nova-microservice	re-push
ECS cluster/service/task/role	acdl-microservice	nova-microservice	recreate
IAM user + policy	acdl-spike-runner (+ -policy)	nova-spike-runner (+ -policy)	re-bootstrap
IAM act-runner role	acdl-act-runner-role	nova-act-runner-role	re-bootstrap
IAM deploy role	acdl-deploy-<repo>	nova-deploy-<repo>	re-bootstrap
S3 state bucket	acdl-tfstate-581513795199-us-east-1	nova-tfstate-581513795199-us-east-1	-migrate-state
DynamoDB outbox	acdl-outbox	nova-outbox	scan + copy
Platform VPC/subnet/IGW/RT	acdl-shared*	nova-shared*	recreate (brief downtime)
CI VPC/subnet/SG/cluster	acdl-ci-*	nova-ci-*	recreate (CI-only)
ALB name prefix	acdl-alb	nova-alb	recreate (brief downtime, LAST)

Migration ordering (binding)

Order: KMS alias → SNS/SG → Lambda → DynamoDB → ECR → IAM → state bucket → ALB. Each step is independently rollback-able. The ALB is last because it requires the briefest downtime window.

---

Pre-flight

1. Announce the maintenance window (consumers are notified via the P1 migration guide docs/NOVA_MIGRATION.md).
2. Back up state for every stack (see §State bucket — back up the state JSON before -migrate-state).
3. Confirm NOVA_LIFECYCLE_MODE=plan (default) so CI does not mutate AWS during the window.
4. Confirm the new nova-* destination tables/repos will be created by the same Terraform apply (no manual pre-creation needed).

Step 1 — KMS alias (alias/acdl-platform → alias/nova-platform)

• Command (in terraform/platform/):

  terraform init -upgrade
  terraform apply -replace=aws_kms_alias.nova_platform
  

  (Terraform destroys the old alias + creates the new one — aliases are cheap; the underlying key ID is unchanged.)
• Verify: aws kms list-aliases --query 'Aliases[?AliasName==alias/nova-platform]' returns the new alias; alias/acdl-platform is gone.
• Rollback: terraform apply -replace=aws_kms_alias.nova_platform against the prior revision (re-creates alias/acdl-platform). Resources encrypted by the key are unaffected (key ID unchanged).

Step 2 — SNS topic + Security group (recreate)

• Command: terraform apply in terraform/platform/.
  • SNS acdl-sod-halt → nova-sod-halt (the topic ARN changes; update NOVA_SOD_HALT_TOPIC_ARN wherever it is set).
  • SG acdl-ecs-sg → nova-ecs-sg (the security group is re-attached to running ECS tasks; brief task restart).
• Verify: aws sns list-topics shows nova-sod-halt; aws ec2 describe-security-groups shows nova-ecs-sg.
• Rollback: terraform apply the prior revision re-creates the acdl-* names. The SNS topic has no message backlog (halt artifacts are fire-and-forget); the SG drift resolves on next task deploy.

Step 3 — Lambda (recreate)

• Command: terraform apply in terraform/platform/.
  • Lambda function acdl-contract-ingestor → nova-contract-ingestor.
  • Execution role acdl-contract-ingestor-role → nova-contract-ingestor-role.
  • Inline policy acdl-contract-ingestor-policy → nova-contract-ingestor-policy.
  • The Lambda env vars (CONTRACTS_TABLE, GITHUB_TOKEN_SECRET_ID) now resolve to nova-* defaults.
• Verify: aws lambda list-functions shows nova-contract-ingestor; the Function URL returns 200 on a SigV4-signed invoke. The consumer_invoke_policy.json rendered output (Terraform consumer_invoke_policy_rendered) now references function:nova-contract-ingestor — re-distribute to consumer deploy roles.
• Rollback: terraform apply the prior revision re-creates acdl-contract-ingestor. Consumer deploy roles must point back at the old Function ARN (re-distribute the prior consumer_invoke_policy.json).

Step 4 — DynamoDB (scan + copy)

DynamoDB table names are immutable post-creation, so the migration is a scan + copy (not a rename). The new nova-* tables are created by the same Terraform apply (Step 3). The data-migration script copies every item and verifies row counts.

• Command (from repo root):

  # Dry-run first (no writes):
  python3 scripts/migrate_dynamodb_data.py
  # Execute the copy:
  python3 scripts/migrate_dynamodb_data.py --apply
  # A single table:
  python3 scripts/migrate_dynamodb_data.py --table contracts --apply
  

  The script scans acdl-contracts → copies to nova-contracts, and acdl-change-requests → nova-change-requests, then verifies the destination row count == source row count (re-scan, not DescribeTable.ItemCount which lags ~6h).
• Verify:

  # Row counts must match (printed by the script). Manual cross-check:
  aws dynamodb scan --table-name nova-contracts --select COUNT
  aws dynamodb scan --table-name acdl-contracts --select COUNT
  

  Then point consumers at the new tables (the Lambda already reads nova-* defaults; any direct DynamoDB consumers update their env).
• Keep the old tables (acdl-contracts, acdl-change-requests) until consumers are verified reading from nova-*. Deletion is a manual post-verification step:

  aws dynamodb delete-table --table-name acdl-contracts
  aws dynamodb delete-table --table-name acdl-change-requests
  

  Only delete after a full soak period confirms nova-* reads succeed.
• Rollback: Re-point consumers at acdl-* (the old tables are retained). The copy is additive (no data loss). To roll back a partial copy, re-run --apply (idempotent — PutItem overwrites).

Outbox table (acdl-outbox → nova-outbox)

The evidence outbox table follows the same scan+copy pattern (it is created by terraform/bootstrap/create_state_backend.py).

• Command: python3 scripts/migrate_dynamodb_data.py --source acdl-outbox --dest nova-outbox --apply
• The core/outbox_writer.py default + core/regression_verify.py CAP-015 probe now reference nova-outbox (P4 updated both). The regression gate's live-AWS CAP-015 will return Verified once the nova-outbox table exists live; until then it is Decayed (the gate is re-run at milestone complete after the live migration).

Step 5 — ECR (re-push)

• Command: terraform apply in terraform/microservice/ creates the new nova-microservice ECR repo. Re-push the image:

  python3 scripts/push_consumer_image.py   # creates nova-microservice + prints docker tag/push
  

  (The script's ECR_REPO_NAME is now nova-microservice.)
• Verify: aws ecr describe-repositories shows nova-microservice; docker pull <acct>.dkr.ecr.us-east-1.amazonaws.com/nova-microservice:latest succeeds.
• Rollback: The old acdl-microservice repo is retained until the soak passes. Re-push to it if a rollback is needed. Delete it manually: aws ecr delete-repository --repository-name acdl-microservice --force.

Step 6 — IAM (re-bootstrap)

• Command:

  export NOVA_BOOTSTRAP_AWS_ACCESS_KEY_ID="<root key>"
  export NOVA_BOOTSTRAP_AWS_SECRET_ACCESS_KEY="<root secret>"
  python3 terraform/bootstrap/create_state_backend.py        # creates nova-outbox (idempotent)
  python3 terraform/bootstrap/create_iam_user.py             # creates nova-spike-runner
  python3 terraform/bootstrap/apply_iam_baseline.py          # creates nova-spike-runner-policy + nova-act-runner-role
  bash scripts/rotate_spike_key.sh                          # rotates the nova-spike-runner key
  

  The deploy role acdl-deploy-<repo> → nova-deploy-<repo> is created by the bootstrap (the deploy workflow .gitea/.github/workflows/deploy.yml now references role/nova-deploy-{1}).
• Verify: aws iam get-user --user-name nova-spike-runner; aws iam list-attached-user-policies --user-name nova-spike-runner shows nova-spike-runner-policy; aws iam get-role --role-name nova-act-runner-role.
• Rollback: Re-run the prior bootstrap scripts (they create acdl-spike-runner + acdl-act-runner-role). The deploy workflow's role-to-assume must be reverted to acdl-deploy- (prior revision).

Step 7 — State bucket (acdl-tfstate-* → nova-tfstate-*, -migrate-state)

The S3 state backend is renamed. Terraform's -migrate-state copies the state objects to the new bucket. Back up the state JSON first.

• Back up state (per stack):

  for stack in platform microservice ci-vpc; do
    aws s3 cp s3://acdl-tfstate-581513795199-us-east-1/$stack/terraform.tfstate \
      ./backup-$stack.tfstate
  done
  

• Command (per stack): the backend config in each terraform/*/terraform.tf now points at nova-tfstate-....

  cd terraform/platform
  terraform init -migrate-state        # copies state acdl-tfstate → nova-tfstate
  cd ../microservice
  terraform init -migrate-state
  cd ../ci-vpc
  terraform init -migrate-state
  

• Verify: aws s3 ls s3://nova-tfstate-581513795199-us-east-1/ shows the state keys; terraform state list in each dir lists the expected resources.
• Rollback: Point the backend back at acdl-tfstate-* and re-run terraform init -migrate-state (restores from the backup bucket). The old acdl-tfstate-* bucket is retained until the soak passes. Delete it manually: aws s3 rb s3://acdl-tfstate-581513795199-us-east-1 --force.

Step 8 — ALB (recreate, brief downtime, LAST)

The ALB is last because its recreation requires the briefest downtime window (the ECS service is re-attached to the new target group).

• Command: terraform apply in terraform/microservice/. The ALB acdl-microservice / acdl-alb → nova-microservice / nova-alb.
• Verify: aws elbv2 describe-load-balancers shows the new ALB; curl http://<new-alb-dns>/ returns 200.
• Rollback: terraform apply the prior revision re-creates the acdl-* ALB (brief downtime again). The old ALB DNS is retained until consumers are re-pointed.

---

Post-migration

1. Soak: run consumers against nova-* for a full verification window (deploy a test contract end-to-end).
2. Delete old resources (manual, only after soak):
   • DynamoDB: acdl-contracts, acdl-change-requests, acdl-outbox
   • ECR: acdl-microservice
   • IAM: acdl-spike-runner (+ policy), acdl-act-runner-role, acdl-deploy-<repo>
   • S3: acdl-tfstate-581513795199-us-east-1
   • SNS: acdl-sod-halt
   • SG: acdl-ecs-sg
   • Secrets Manager: acdl/github-token
   • KMS alias: alias/acdl-platform
   • ALB: acdl-alb / acdl-microservice
3. Regression gate: re-run bash scripts/run_regression.sh. The live-AWS CAP-013..016 probes should return Verified (the nova-* tables + state bucket exist). CAP-015 (outbox) flips from Decayed → Verified once nova-outbox is live.

What P5 owns (not P4)

• Remove dual-read fallback: core/env.py get_env() drops the ACDL_* fallback; shell scripts drop :-$ACDL_X. P4 keeps the dual-read (deployments don't break mid-window).
• nova_tagging.py hard-fail on acdl:*: P3 set hard mode (no acdl:*-only tags); P5 tightens to fail on any acdl:* presence. P4 leaves P3's behavior.
• Delete ACDL_* Gitea secrets: the NOVA_* aliases created in P2 are now the only source.
• Finalize docs/NOVA_MIGRATION.md: mark the migration complete (cutoff passed).
• Milestone ship: tag v1.15.4, merge to main, Gitea release.

Files touched in P4

• terraform/platform/main.tf, terraform/microservice/main.tf, terraform/ci-vpc/main.tf — resource renames + backend bucket.
• terraform/{platform,microservice,ci-vpc}/terraform.tf — state bucket.
• terraform/platform/consumer_invoke_policy.json — Lambda ARN.
• terraform/bootstrap/{create_state_backend,create_iam_user,apply_iam_baseline}.py, spike_runner_policy.json, .bootstrap_state.json, README.md — IAM/outbox/state-bucket renames.
• modules/l1/*/terraform/** + modules/l1/alb/instance.json — L1 resource-name defaults.
• modules/l2/microservice/composition.json — nova-app-role default.
• core/lambda/contract_ingestor.py — default table names (D-111).
• core/outbox_writer.py, core/regression_verify.py, core/local_emulators.py — outbox table consistency (cross-territory, minimal).
• .gitea/workflows/deploy.yml + .github/workflows/deploy.yml — nova-deploy- role ARN + artifact names.
• scripts/migrate_dynamodb_data.py (NEW), scripts/rotate_spike_key.sh, scripts/push_consumer_image.py.
• tests/** — fixtures updated to assert nova-*.