acdl/docs/NOVA_AWS_MIGRATION.md

# Nova AWS Resource Migration Runbook (REQ-163, P4)

> **Milestone:** v1.15-Nova (Wave 4, P4). Renames every `acdl-*` AWS
> resource name → `nova-*` via Terraform. This is the heaviest Terraform
> phase of the rebrand and requires a **maintenance window**.
>
> **Plan-validated only.** Per A1, `NOVA_LIFECYCLE_MODE` defaults to
> `plan` (no live AWS mutation from CI). `terraform validate` passes; the
> live apply steps below are executed by a platform operator during the
> scheduled maintenance window. Each step has a verification + rollback.

## Scope (renamed resources)

| AWS resource | Before | After | Strategy |
|---|---|---|---|
| KMS alias | `alias/acdl-platform` | `alias/nova-platform` | cheap rename |
| SNS topic | `acdl-sod-halt` | `nova-sod-halt` | recreate |
| Security group | `acdl-ecs-sg` | `nova-ecs-sg` | recreate |
| Lambda (role/policy/function) | `acdl-contract-ingestor` | `nova-contract-ingestor` | recreate |
| DynamoDB contracts | `acdl-contracts` | `nova-contracts` | scan + copy |
| DynamoDB change-requests | `acdl-change-requests` | `nova-change-requests` | scan + copy |
| Secrets Manager secret | `acdl/github-token` | `nova/github-token` | recreate + re-store |
| ECR repo | `acdl-microservice` | `nova-microservice` | re-push |
| ECS cluster/service/task/role | `acdl-microservice` | `nova-microservice` | recreate |
| IAM user + policy | `acdl-spike-runner` (+ `-policy`) | `nova-spike-runner` (+ `-policy`) | re-bootstrap |
| IAM act-runner role | `acdl-act-runner-role` | `nova-act-runner-role` | re-bootstrap |
| IAM deploy role | `acdl-deploy-<repo>` | `nova-deploy-<repo>` | re-bootstrap |
| S3 state bucket | `acdl-tfstate-581513795199-us-east-1` | `nova-tfstate-581513795199-us-east-1` | `-migrate-state` |
| DynamoDB outbox | `acdl-outbox` | `nova-outbox` | scan + copy |
| Platform VPC/subnet/IGW/RT | `acdl-shared*` | `nova-shared*` | recreate (brief downtime) |
| CI VPC/subnet/SG/cluster | `acdl-ci-*` | `nova-ci-*` | recreate (CI-only) |
| ALB name prefix | `acdl-alb` | `nova-alb` | recreate (brief downtime, LAST) |

## Migration ordering (binding)

Order: **KMS alias → SNS/SG → Lambda → DynamoDB → ECR → IAM → state bucket → ALB**.
Each step is independently rollback-able. The ALB is last because it
requires the briefest downtime window.

---

## Pre-flight

1. **Announce the maintenance window** (consumers are notified via the
   P1 migration guide `docs/NOVA_MIGRATION.md`).
2. **Back up state** for every stack (see §State bucket — back up the
   state JSON *before* `-migrate-state`).
3. Confirm `NOVA_LIFECYCLE_MODE=plan` (default) so CI does not mutate
   AWS during the window.
4. Confirm the new `nova-*` destination tables/repos will be created by
   the same Terraform apply (no manual pre-creation needed).

## Step 1 — KMS alias (`alias/acdl-platform` → `alias/nova-platform`)

- **Command (in `terraform/platform/`):**
  ```bash
  terraform init -upgrade
  terraform apply -replace=aws_kms_alias.nova_platform
  ```
  (Terraform destroys the old alias + creates the new one — aliases are
  cheap; the underlying key ID is unchanged.)
- **Verify:** `aws kms list-aliases --query 'Aliases[?AliasName==`alias/nova-platform`]'` returns the new alias; `alias/acdl-platform` is gone.
- **Rollback:** `terraform apply -replace=aws_kms_alias.nova_platform` against the prior revision (re-creates `alias/acdl-platform`). Resources encrypted by the key are unaffected (key ID unchanged).

## Step 2 — SNS topic + Security group (recreate)

- **Command:** `terraform apply` in `terraform/platform/`.
  - SNS `acdl-sod-halt` → `nova-sod-halt` (the topic ARN changes; update `NOVA_SOD_HALT_TOPIC_ARN` wherever it is set).
  - SG `acdl-ecs-sg` → `nova-ecs-sg` (the security group is re-attached to running ECS tasks; brief task restart).
- **Verify:** `aws sns list-topics` shows `nova-sod-halt`; `aws ec2 describe-security-groups` shows `nova-ecs-sg`.
- **Rollback:** `terraform apply` the prior revision re-creates the `acdl-*` names. The SNS topic has no message backlog (halt artifacts are fire-and-forget); the SG drift resolves on next task deploy.

## Step 3 — Lambda (recreate)

- **Command:** `terraform apply` in `terraform/platform/`.
  - Lambda function `acdl-contract-ingestor` → `nova-contract-ingestor`.
  - Execution role `acdl-contract-ingestor-role` → `nova-contract-ingestor-role`.
  - Inline policy `acdl-contract-ingestor-policy` → `nova-contract-ingestor-policy`.
  - The Lambda env vars (`CONTRACTS_TABLE`, `GITHUB_TOKEN_SECRET_ID`) now resolve to `nova-*` defaults.
- **Verify:** `aws lambda list-functions` shows `nova-contract-ingestor`; the Function URL returns 200 on a SigV4-signed invoke. The `consumer_invoke_policy.json` rendered output (Terraform `consumer_invoke_policy_rendered`) now references `function:nova-contract-ingestor` — re-distribute to consumer deploy roles.
- **Rollback:** `terraform apply` the prior revision re-creates `acdl-contract-ingestor`. Consumer deploy roles must point back at the old Function ARN (re-distribute the prior `consumer_invoke_policy.json`).

## Step 4 — DynamoDB (scan + copy)

DynamoDB table names are immutable post-creation, so the migration is a
**scan + copy** (not a rename). The new `nova-*` tables are created by
the same Terraform apply (Step 3). The data-migration script copies
every item and verifies row counts.

- **Command (from repo root):**
  ```bash
  # Dry-run first (no writes):
  python3 scripts/migrate_dynamodb_data.py
  # Execute the copy:
  python3 scripts/migrate_dynamodb_data.py --apply
  # A single table:
  python3 scripts/migrate_dynamodb_data.py --table contracts --apply
  ```
  The script scans `acdl-contracts` → copies to `nova-contracts`, and
  `acdl-change-requests` → `nova-change-requests`, then verifies the
  destination row count == source row count (re-scan, not
  `DescribeTable.ItemCount` which lags ~6h).
- **Verify:**
  ```bash
  # Row counts must match (printed by the script). Manual cross-check:
  aws dynamodb scan --table-name nova-contracts --select COUNT
  aws dynamodb scan --table-name acdl-contracts --select COUNT
  ```
  Then **point consumers at the new tables** (the Lambda already reads
  `nova-*` defaults; any direct DynamoDB consumers update their env).
- **Keep the old tables** (`acdl-contracts`, `acdl-change-requests`)
  until consumers are verified reading from `nova-*`. **Deletion is a
  manual post-verification step:**
  ```bash
  aws dynamodb delete-table --table-name acdl-contracts
  aws dynamodb delete-table --table-name acdl-change-requests
  ```
  Only delete after a full soak period confirms `nova-*` reads succeed.
- **Rollback:** Re-point consumers at `acdl-*` (the old tables are
  retained). The copy is additive (no data loss). To roll back a partial
  copy, re-run `--apply` (idempotent — `PutItem` overwrites).

### Outbox table (`acdl-outbox` → `nova-outbox`)

The evidence outbox table follows the same scan+copy pattern (it is
created by `terraform/bootstrap/create_state_backend.py`).
- **Command:** `python3 scripts/migrate_dynamodb_data.py --source acdl-outbox --dest nova-outbox --apply`
- The `core/outbox_writer.py` default + `core/regression_verify.py`
  CAP-015 probe now reference `nova-outbox` (P4 updated both). The
  regression gate's live-AWS CAP-015 will return `Verified` once the
  `nova-outbox` table exists live; until then it is `Decayed` (the gate
  is re-run at milestone complete after the live migration).

## Step 5 — ECR (re-push)

- **Command:** `terraform apply` in `terraform/microservice/` creates
  the new `nova-microservice` ECR repo. Re-push the image:
  ```bash
  python3 scripts/push_consumer_image.py   # creates nova-microservice + prints docker tag/push
  ```
  (The script's `ECR_REPO_NAME` is now `nova-microservice`.)
- **Verify:** `aws ecr describe-repositories` shows `nova-microservice`; `docker pull <acct>.dkr.ecr.us-east-1.amazonaws.com/nova-microservice:latest` succeeds.
- **Rollback:** The old `acdl-microservice` repo is retained until the
  soak passes. Re-push to it if a rollback is needed. Delete it manually:
  `aws ecr delete-repository --repository-name acdl-microservice --force`.

## Step 6 — IAM (re-bootstrap)

- **Command:**
  ```bash
  export NOVA_BOOTSTRAP_AWS_ACCESS_KEY_ID="<root key>"
  export NOVA_BOOTSTRAP_AWS_SECRET_ACCESS_KEY="<root secret>"
  python3 terraform/bootstrap/create_state_backend.py        # creates nova-outbox (idempotent)
  python3 terraform/bootstrap/create_iam_user.py             # creates nova-spike-runner
  python3 terraform/bootstrap/apply_iam_baseline.py          # creates nova-spike-runner-policy + nova-act-runner-role
  bash scripts/rotate_spike_key.sh                          # rotates the nova-spike-runner key
  ```
  The deploy role `acdl-deploy-<repo>` → `nova-deploy-<repo>` is
  created by the bootstrap (the deploy workflow
  `.gitea/.github/workflows/deploy.yml` now references
  `role/nova-deploy-{1}`).
- **Verify:** `aws iam get-user --user-name nova-spike-runner`;
  `aws iam list-attached-user-policies --user-name nova-spike-runner`
  shows `nova-spike-runner-policy`;
  `aws iam get-role --role-name nova-act-runner-role`.
- **Rollback:** Re-run the prior bootstrap scripts (they create
  `acdl-spike-runner` + `acdl-act-runner-role`). The deploy workflow's
  `role-to-assume` must be reverted to `acdl-deploy-` (prior revision).

## Step 7 — State bucket (`acdl-tfstate-*` → `nova-tfstate-*`, `-migrate-state`)

The S3 state backend is renamed. Terraform's `-migrate-state` copies the
state objects to the new bucket. **Back up the state JSON first.**

- **Back up state (per stack):**
  ```bash
  for stack in platform microservice ci-vpc; do
    aws s3 cp s3://acdl-tfstate-581513795199-us-east-1/$stack/terraform.tfstate \
      ./backup-$stack.tfstate
  done
  ```
- **Command (per stack):** the backend config in each
  `terraform/*/terraform.tf` now points at `nova-tfstate-...`.
  ```bash
  cd terraform/platform
  terraform init -migrate-state        # copies state acdl-tfstate → nova-tfstate
  cd ../microservice
  terraform init -migrate-state
  cd ../ci-vpc
  terraform init -migrate-state
  ```
- **Verify:** `aws s3 ls s3://nova-tfstate-581513795199-us-east-1/`
  shows the state keys; `terraform state list` in each dir lists the
  expected resources.
- **Rollback:** Point the backend back at `acdl-tfstate-*` and re-run
  `terraform init -migrate-state` (restores from the backup bucket). The
  old `acdl-tfstate-*` bucket is retained until the soak passes. Delete
  it manually:
  `aws s3 rb s3://acdl-tfstate-581513795199-us-east-1 --force`.

## Step 8 — ALB (recreate, brief downtime, LAST)

The ALB is last because its recreation requires the briefest downtime
window (the ECS service is re-attached to the new target group).

- **Command:** `terraform apply` in `terraform/microservice/`. The ALB
  `acdl-microservice` / `acdl-alb` → `nova-microservice` / `nova-alb`.
- **Verify:** `aws elbv2 describe-load-balancers` shows the new ALB;
  `curl http://<new-alb-dns>/` returns 200.
- **Rollback:** `terraform apply` the prior revision re-creates the
  `acdl-*` ALB (brief downtime again). The old ALB DNS is retained until
  consumers are re-pointed.

---

## Post-migration

1. **Soak:** run consumers against `nova-*` for a full verification
   window (deploy a test contract end-to-end).
2. **Delete old resources** (manual, only after soak):
   - DynamoDB: `acdl-contracts`, `acdl-change-requests`, `acdl-outbox`
   - ECR: `acdl-microservice`
   - IAM: `acdl-spike-runner` (+ policy), `acdl-act-runner-role`,
     `acdl-deploy-<repo>`
   - S3: `acdl-tfstate-581513795199-us-east-1`
   - SNS: `acdl-sod-halt`
   - SG: `acdl-ecs-sg`
   - Secrets Manager: `acdl/github-token`
   - KMS alias: `alias/acdl-platform`
   - ALB: `acdl-alb` / `acdl-microservice`
3. **Regression gate:** re-run `bash scripts/run_regression.sh`. The
   live-AWS CAP-013..016 probes should return `Verified` (the `nova-*`
   tables + state bucket exist). CAP-015 (outbox) flips from `Decayed`
   → `Verified` once `nova-outbox` is live.

## What P5 owns (not P4)

- **Remove dual-read fallback:** `core/env.py` `get_env()` drops the
  `ACDL_*` fallback; shell scripts drop `:-$ACDL_X`. P4 keeps the
  dual-read (deployments don't break mid-window).
- **`nova_tagging.py` hard-fail on `acdl:*`:** P3 set hard mode (no
  `acdl:*`-only tags); P5 tightens to fail on any `acdl:*` presence. P4
  leaves P3's behavior.
- **Delete `ACDL_*` Gitea secrets:** the `NOVA_*` aliases created in P2
  are now the only source.
- **Finalize `docs/NOVA_MIGRATION.md`:** mark the migration complete
  (cutoff passed).
- **Milestone ship:** tag `v1.15.4`, merge to `main`, Gitea release.

## Files touched in P4

- `terraform/platform/main.tf`, `terraform/microservice/main.tf`,
  `terraform/ci-vpc/main.tf` — resource renames + backend bucket.
- `terraform/{platform,microservice,ci-vpc}/terraform.tf` — state bucket.
- `terraform/platform/consumer_invoke_policy.json` — Lambda ARN.
- `terraform/bootstrap/{create_state_backend,create_iam_user,apply_iam_baseline}.py`,
  `spike_runner_policy.json`, `.bootstrap_state.json`, `README.md` —
  IAM/outbox/state-bucket renames.
- `modules/l1/*/terraform/**` + `modules/l1/alb/instance.json` — L1
  resource-name defaults.
- `modules/l2/microservice/composition.json` — `nova-app-role` default.
- `core/lambda/contract_ingestor.py` — default table names (D-111).
- `core/outbox_writer.py`, `core/regression_verify.py`,
  `core/local_emulators.py` — outbox table consistency (cross-territory,
  minimal).
- `.gitea/workflows/deploy.yml` + `.github/workflows/deploy.yml` —
  `nova-deploy-` role ARN + artifact names.
- `scripts/migrate_dynamodb_data.py` (NEW), `scripts/rotate_spike_key.sh`,
  `scripts/push_consumer_image.py`.
- `tests/**` — fixtures updated to assert `nova-*`.