acdl/docs/operator-guide-idp.md

# Operator Guide — Nova IdP Setup (`nova idp setup`)

> **REQ-345** — operator guide for `nova idp setup`. Covers `--check`,
> `--apply`, `--verify`, the prerequisite IAM policy, the CloudFormation
> review flow, and the **C-6.3 grill additions**: KMS key rotation
> (90 days), Lambda layer update, DDB PITR restore, emergency PAT
> revocation (DDB-level, not CLI).
>
> Audience: platform operators / SREs deploying the Nova identity stack
> into AWS account `581513795199` (or a fresh account). No developer
> auth flows here — see `docs/developer-guide-auth.md` for those.

## 1. Overview

`nova idp setup` provisions the Nova identity layer (Nova-idp) as a
CloudFormation stack. The stack contains:

| Resource | Count | Notes |
|----------|-------|-------|
| Lambda functions | 3 | `nova-idp-auth`, `nova-idp-token-vend`, `nova-idp-jwks` |
| DynamoDB tables | 4 | `nova-users`, `nova-sessions`, `nova-password-resets`, `nova-pats` (PITR enabled on each, REQ-335) |
| KMS asymmetric key | 1 | `alias/nova-oidc-signing` (`ECC_NIST_P256`, `SIGN_VERIFY`) |
| Lambda function URLs | 3 | auth + token-vend (IAM auth), jwks (`AuthType: NONE`) |
| IAM roles | 3+ | one per Lambda + the CloudFormation service role |
| Optional CloudFront + WAF + ACM | 0/3 | only with `--public-jwks-domain` |

The command has three modes — `--check`, `--apply`, `--verify` — plus
`--dry-run` for a resource-only preview. All modes are safe to re-run.

## 2. `nova idp setup --check`

Run **before** `--apply` to verify the deploying principal has the
permissions and environment the stack needs.

```sh
nova idp setup --check
```

### What it checks

1. **AWS credentials** — `aws sts get-caller-identity` succeeds and
   returns an `Account` id. If this fails, run `aws configure` or export
   `AWS_PROFILE` / `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY`.
2. **AWS region** — `AWS_DEFAULT_REGION` or `AWS_REGION` is set. The
   stack is regional (single-region); pick the region you want all
   resources to live in.
3. **CloudFormation permissions** — the principal can create/describe
   stacks (see §5 for the full IAM delta).
4. **KMS permissions** — `kms:CreateKey` + `kms:CreateAlias` (needed to
   mint `alias/nova-oidc-signing`).
5. **Lambda layer exists** — the `nova-cli` Lambda layer (published by
   the P1 Wave 4 pipeline) is referenced by the stack; `--check` reports
   whether the layer ARN in SSM (`/nova/layer/nova-cli/version`) is
   present. If absent, run the publish workflow or `nova layer update`.

### Reading the IAM policy delta

`--check` prints a report like:

```json
{
  "aws_creds": true,
  "region": "us-east-1",
  "missing": [],
  "iam_delta": [
    "cloudformation:*",
    "iam:CreateRole",
    "iam:PassRole",
    "lambda:CreateFunction",
    "lambda:CreateFunctionUrlConfig",
    "dynamodb:CreateTable",
    "kms:CreateKey",
    "kms:CreateAlias"
  ]
}
```

`iam_delta` is the **delta** between what the deploying principal
currently has (the `nova-spike-runner` grants in this account) and what
`--apply` needs. Each entry is a grant you must add to the principal's
policy before `--apply` will succeed. `--check` never makes changes.

## 3. `nova idp setup --apply`

Generates the CloudFormation template, presents it for review, and
deploys **only after explicit `y/N` approval** (NFR-10).

```sh
nova idp setup --apply
```

### Review flow

1. **Resource summary** printed to stdout (resource type → count):
   ```
   Resource summary:
     AWS::DynamoDB::Table: 4
     AWS::IAM::Role: 3
     AWS::KMS::Key: 1
     AWS::Lambda::Function: 3
     AWS::Lambda::Url: 3
   ```
2. **Full template** opened in `$PAGER` (if set and stdin is a TTY);
   otherwise the path to the temp file is printed. Review every
   resource, especially the KMS key policy and the IAM roles.
3. **`Apply? [y/N]` prompt.** Type `y` + Enter to deploy; anything else
   aborts. No resource is created before this approval.
4. On approval: `aws cloudformation deploy --stack-name nova-idp
   --template-file <tmp> --capabilities CAPABILITY_IAM`.

### `--dry-run` — resource list only

```sh
nova idp setup --dry-run
```

Generates the template and prints the resource summary **without** the
pager, the prompt, or any deploy. Use this to audit the stack shape in
CI or before a manual `--apply`.

### `--public-jwks-domain` — optional custom domain + WAF

```sh
nova idp setup --apply --public-jwks-domain jwks.nova.example.com
```

Adds a CloudFront distribution fronting the JWKS Lambda function URL, an
ACM certificate (DNS-validated) for the domain, and a WAF web ACL with
a rate-based rule (see §C-6.3 and the threat model). Without this flag
the JWKS endpoint is a bare function URL (`AuthType: NONE`) — fine for
piloting but exposed to the internet without rate limiting. **For any
public deployment, set `--public-jwks-domain`.**

## 4. `nova idp setup --verify`

Runs the KMS round-trip test (CAP-037) against the deployed stack.

```sh
nova idp setup --verify
```

It signs a test JWT via `core.kms_signing.sign_jwt()` (using the real
KMS key `alias/nova-oidc-signing`), fetches the JWKS endpoint, and
verifies the JWT signature with `pyjwt` + the JWKS key. This exercises
the full DER → raw ECDSA conversion path (the #1 implementation risk —
see `docs/threat-model.md`).

**Success output:**
```json
{"passed": true, "detail": "KMS round-trip OK"}
```

**Failure output:**
```json
{"passed": false, "detail": "verify error: <exception>"}
```

Common failure causes:
- The KMS key policy doesn't grant `kms:Sign` to the verify caller.
- The JWKS function URL is not deployed or returns a non-200.
- The KMS key spec isn't `ECC_NIST_P256` (the DER→raw conversion
  assumes P-256, 32-byte coordinates).

## 5. Required IAM policy

The delta `--check` reports is the set of grants the deploying
principal needs **in addition** to the existing `nova-spike-runner`
grants. The full required set:

| Action | Why |
|--------|-----|
| `cloudformation:*` | create/deploy/describe the `nova-idp` stack |
| `codeartifact:*` | (already on `nova-spike-runner`) publish the wheel + layer |
| `iam:CreateRole` | create the per-Lambda execution roles |
| `iam:PassRole` | pass those roles to Lambda + CloudFormation |
| `lambda:CreateFunction` | create the 3 Lambda functions |
| `lambda:CreateFunctionUrlConfig` | create the 3 function URLs |
| `dynamodb:CreateTable` | create the 4 DDB tables (with PITR) |
| `kms:CreateKey` | mint the `ECC_NIST_P256` signing key |
| `kms:CreateAlias` | bind `alias/nova-oidc-signing` to the key |
| `ssm:PutParameter` | write the layer-version mapping to SSM |

Attach these to the deploying principal's policy before `--apply`.
`--check` will then report an empty `missing` list.

---

## C-6.3 Grill additions — operational runbooks

The grill (C-6.3) requires four operational procedures beyond the
setup flow. Each is a runbook an on-call SRE can follow without reading
source code.

### 6. KMS key rotation (90-day cadence)

**Cadence:** rotate `alias/nova-oidc-signing` every **90 days**. The
rotation is a *key re-point*, not a key deletion — the alias is moved
to a new key while the old key stays valid during the token-overlap
window so already-issued tokens keep verifying.

**Procedure:**

1. **Create the new key** (same spec):
   ```sh
   NEW_KEY=$(aws kms create-key \
     --key-spec ECC_NIST_P256 \
     --key-usage SIGN_VERIFY \
     --description "nova-oidc-signing-$(date +%Y%m%d)" \
     --query KeyId --output text)
   ```
2. **Re-point the alias** to the new key:
   ```sh
   aws kms update-alias --alias-name alias/nova-oidc-signing \
     --target-key-id "$NEW_KEY"
   ```
3. **JWKS serves both `kid`s during the overlap window.** The JWKS
   Lambda lists **all** keys the alias has pointed at that are still
   enabled. Already-issued OIDC tokens (signed with the old key) keep
   verifying until they expire (OIDC TTL default 15 min; PAT TTL ≤ 24h
   dev / ≤ 1h service-account). **Do not disable the old key until at
   least the max PAT TTL (24h) has elapsed.**
4. **After the overlap window** (≥ 24h), disable + schedule deletion of
   the old key:
   ```sh
   aws kms disable-key --key-id "<old-key-id>"
   aws kms schedule-key-deletion --key-id "<old-key-id>" --pending-window-in-days 7
   ```
5. **Verify** the new key is active:
   ```sh
   nova idp setup --verify
   ```

**Audit:** emit a manual `kms.key_rotated` event to the audit stream
with `old_key_id`, `new_key_id`, `rotated_at`. The rotation is a
 CloudFormation-less operation (KMS aliases are mutable); it does not
 require a stack update.

### 7. Lambda layer update

The `nova-cli` Lambda layer (the shared dependency bundle:
`argon2-cffi`, `cryptography`, `pyjwt`, `kj` binary) is republished
**automatically on every merge to `main`** by the P1 Wave 4 publish
workflow (the byte-identical GitHub + internal-forge workflow files).
On a successful publish, the new layer version ARN is written to SSM
`/nova/layer/nova-cli/version`.

**When to update manually:**
- A dependency CVE requires an out-of-band patch before the next merge.
- The `kj` binary pinned version changes (C-8.2 supply-chain safety).

**Manual procedure:**

```sh
nova layer update
```

This rebuilds the layer (`pip install --target layer/python/` + the
pinned `kj` binary, SHA256 verified against `layer/kj.sha256`),
publishes a new `lambda:PublishLayerVersion`, and updates the SSM
parameter. The 3 Nova-idp Lambdas pick up the new layer on their next
cold start (or force a redeploy with `aws lambda update-function-configuration
--layers <new-arn>` on each).

**Verify:** `nova idp setup --verify` after the Lambdas reload.

### 8. DynamoDB PITR restore

All 4 identity tables have point-in-time recovery (PITR) enabled
(REQ-335): `nova-users`, `nova-sessions`, `nova-password-resets`,
`nova-pats`. PITR lets you restore a table to any second in the last
**35 days** (the AWS retention window).

**Procedure (restore `nova-pats` to 1 hour ago):**

```sh
# 1. Find the restore target time (ISO 8601, UTC, within the last 35d).
RESTORE_TO=$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)

# 2. Restore to a NEW table (PITR never overwrites the source).
aws dynamodb restore-table-to-point-in-time \
  --source-table-name nova-pats \
  --target-table-name nova-pats-restored \
  --restore-date-time "$RESTORE_TO" \
  --billing-mode-restore-as-is

# 3. After the restore completes (status ACTIVE), repoint the app:
#    - update the stack env var NOVA_PATS_TABLE=nova-pats-restored, or
#    - rename: delete nova-pats, then aws dynamodb update-table --table-name
#      nova-pats-restored --new-table-name nova-pats (downtime window).
# 4. Re-enable PITR on the restored table (PITR does not carry over).
aws dynamodb update-continuous-backups \
  --table-name nova-pats-restored \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true
```

**Which tables have PITR:** all 4 (`nova-users`, `nova-sessions`,
`nova-password-resets`, `nova-pats`). Verify with:
```sh
for t in nova-users nova-sessions nova-password-resets nova-pats; do
  aws dynamodb describe-continuous-backups --table-name "$t" \
    --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription' --output text
done
```

**Recovery window:** 35 days (AWS PITR). Restores older than 35 days
are impossible — for longer retention, export to S3 via the on-demand
export or a scheduled AWS Backup plan.

### 9. Emergency PAT revocation (DDB-level, not CLI)

**When to use:** a PAT is known-compromised and the `nova auth revoke`
CLI is unavailable (e.g. the operator machine is offline, or the PAT
`jti` is known but the raw PAT is not — revocation is keyed on `jti`,
not the token string). This is a **DDB-level** operation; it bypasses
the CLI but still satisfies the D-229 strong-read SLO (the token-vend
Lambda does a `ConsistentRead=True` `GetItem` on `jti` on every vend —
the revocation is reflected on the next vend, within 60s P95).

**Procedure:**

```sh
aws dynamodb update-item \
  --table-name nova-pats \
  --key '{"jti":{"S":"<jti>"}}' \
  --update-expression "SET #s = :r" \
  --expression-attribute-names '{"#s":"status"}' \
  --expression-attribute-values '{":r":{"S":"revoked"}}'
```

Replace `<jti>` with the PAT's `jti` claim (a uuid4; find it in the
`pat.issued` audit event or by scanning the `sub-index` GSI for the
compromised subject). The item is **retained** (not deleted) so the
audit trail is intact — only `status` flips from `active` to `revoked`.

**Verify the revocation took effect:**

```sh
aws dynamodb get-item \
  --table-name nova-pats \
  --key '{"jti":{"S":"<jti>"}}' \
  --consistent-read \
  --query 'Item.status.S' --output text
# → revoked
```

The next `token-vend` call with that `jti` returns `403
pat_revoked` immediately (D-229: the strong read is synchronous).

**Bulk revocation** (revoke all of a subject's PATs):

```sh
SUB="<sub>"
JTIS=$(aws dynamodb query \
  --table-name nova-pats \
  --index-name sub-index \
  --key-condition-expression "sub = :s" \
  --expression-attribute-values "{\":s\":{\"S\":\"$SUB\"}}" \
  --query 'Items[?status.S==`active`].jti.S' --output text)
for jti in $JTIS; do
  aws dynamodb update-item --table-name nova-pats \
    --key "{\"jti\":{\"S\":\"$jti\"}}" \
    --update-expression "SET #s = :r" \
    --expression-attribute-names '{"#s":"status"}' \
    --expression-attribute-values '{":r":{"S":"revoked"}}'
done
```

---

## Appendix — quick reference

| Command | What it does |
|---------|--------------|
| `nova idp setup --check` | prerequisites + IAM delta (no changes) |
| `nova idp setup --dry-run` | resource summary only (no deploy) |
| `nova idp setup --apply` | review template → `y/N` → deploy |
| `nova idp setup --apply --public-jwks-domain <fqdn>` | add CloudFront + WAF + ACM |
| `nova idp setup --verify` | KMS round-trip test (CAP-037) |

| Runbook | Cadence / trigger |
|---------|-------------------|
| KMS key rotation | every 90 days |
| Lambda layer update | on merge (auto) or manually via `nova layer update` |
| DDB PITR restore | on data loss / corruption (35-day window) |
| Emergency PAT revocation | on compromise (DDB-level, immediate) |