Jon Chery
2cc76f4f94
The Step 5b kyverno-json block used a single-quoted heredoc (<<'PY') but
referenced $WORK and $CONTRACT_ID inside the Python body as literal
strings — neither variable expanded, so kj scan ran against the literal
filename "$WORK/tfshow.json" (FileNotFoundError) and recorded contractId
"$CONTRACT_ID" verbatim. The entire Step 5b plan-JSON policy pass was
silently broken whenever kj was installed (it only "worked" in the
kj-absent skip path, which the tests exercise).
Fix: pass the two values as argv (python3 - "$WORK/tfshow.json"
"$CONTRACT_ID" <<'PY') and read them via sys.argv. This preserves the
single-quoted heredoc (no shell expansion into Python source — avoids a
payload-injection vector if $CONTRACT_ID ever contained a quote) while
correctly threading the values into the engine.
---ci---
project: acdl
phase: 5
milestone: v1.25-kyverno-json
status: verify
lessons:
- P0 fix applied: Step 5b heredoc <<'PY' prevented $WORK/$CONTRACT_ID
expansion → kj scan read literal filename, Step 5b silently broken
whenever kj installed. Re-threaded via sys.argv (also closes a
payload-injection vector vs naively unquoting the heredoc).
---/ci---
667 lines
30 KiB
Bash
Executable File
667 lines
30 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# scripts/run_platform.sh - the Nova platform pipeline.
|
|
#
|
|
# Usage:
|
|
# run_platform.sh <contract.yml> (full e2e with AWS)
|
|
# run_platform.sh --check-only [contract.yml] (offline, no AWS/Checkov/DynamoDB)
|
|
# run_platform.sh --plan-only <contract.yml> (AWS plan only, no Checkov/outbox)
|
|
# run_platform.sh --apply <contract.yml> (AWS apply: init/validate/plan/apply)
|
|
# run_platform.sh --destroy <contract.yml> (AWS destroy: init/validate/destroy)
|
|
# run_platform.sh --local [contract.yml] (local emulating tier, no AWS)
|
|
# run_platform.sh --decommission <CR> <contract.yml> (gated teardown)
|
|
# run_platform.sh --help (show all flags)
|
|
#
|
|
# Modes:
|
|
# --check-only (offline, no AWS/Checkov/DynamoDB — for CI)
|
|
# contract -> resolver -> stack -> adapter -> stream TF -> validate -> exit 0
|
|
# --plan-only (requires AWS creds, no Checkov/outbox)
|
|
# contract -> resolver -> stack -> adapter -> terraform init/validate/plan -> exit 0
|
|
# --apply (requires AWS creds; HITL gate for qa/prod/dr)
|
|
# contract -> resolver -> stack -> adapter -> terraform init/validate/plan/apply -> exit 0
|
|
# --destroy (requires AWS creds; use --decommission <CR> for gated production teardown)
|
|
# contract -> resolver -> stack -> adapter -> terraform init/validate/destroy -> exit 0
|
|
# --local (no AWS creds; local emulating tier D-092)
|
|
# contract -> resolver -> adapter -> local S3/ECS/outbox/Lambda stubs -> exit 0
|
|
# (default) (requires AWS creds + Checkov + DynamoDB)
|
|
# contract -> resolver -> stack -> adapter -> terraform plan -> Checkov ->
|
|
# confidence -> outbox
|
|
#
|
|
# Flags:
|
|
# --quiet suppress terraform/checkov streaming (output to log only)
|
|
# --decommission gate --destroy with D-070 two-step CR validation (requires <CR>)
|
|
# --deploy-uptime deploy the uptime monitoring stack (separate state)
|
|
# --local run the headline E2E against the local emulating tier (D-092)
|
|
# --environment <name> override the contract's environment at load time (D-088)
|
|
# --help, -h show all flags + a one-line description
|
|
#
|
|
# The contract file is a YAML file validated against schemas/contract.schema.json.
|
|
# The resolver (core/contract_resolver.py) resolves it to a Target Stack
|
|
# instance, which the adapter (adapters/terraform/adapter.py) compiles to Terraform.
|
|
#
|
|
# The shell owns all terraform lifecycle (apply/destroy). Python never runs
|
|
# terraform apply or terraform destroy (D-101). Python only orchestrates this
|
|
# shell script and may use boto3 for read-only verify probes (future QA milestone).
|
|
#
|
|
# Uses the rotated spike key (D-039/D-047) from gitignored .env.secrets.
|
|
# -lock=false per D-P09-1.
|
|
set -euo pipefail
|
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
|
|
# Capture the caller's CWD before we cd to ROOT. The reusable deploy workflow
|
|
# invokes this script from the CONSUMER repo's workspace root with a relative
|
|
# contract path (e.g. .nova/contract.yml); the contract must resolve against
|
|
# the consumer repo, not the platform repo (platform/). Without this, the
|
|
# `[ -f "$CONTRACT" ]` check below looks for the contract inside the platform
|
|
# repo and fails (P0 fix — see docs/CONSUMER_GUIDE.md Step 4).
|
|
CALLER_CWD="$(pwd)"
|
|
cd "$ROOT"
|
|
|
|
CHECK_ONLY=0
|
|
PLAN_ONLY=0
|
|
APPLY_ONLY=0
|
|
DESTROY_ONLY=0
|
|
QUIET=0
|
|
DEPLOY_UPTIME=0
|
|
DECOMMISSION=0
|
|
LOCAL_TIER=0
|
|
CHANGE_REQUEST_ID=""
|
|
ENVIRONMENT_OVERRIDE=""
|
|
CONTRACT=""
|
|
|
|
# P15 (REQ-179): --help / -h prints all flags + a one-line description.
|
|
_print_help() {
|
|
cat <<'HELP'
|
|
Nova platform pipeline — run_platform.sh
|
|
|
|
Usage:
|
|
run_platform.sh <contract.yml> (full e2e with AWS)
|
|
run_platform.sh --check-only [contract.yml] (offline, no AWS)
|
|
run_platform.sh --plan-only <contract.yml> (AWS plan only)
|
|
run_platform.sh --apply <contract.yml> (AWS apply)
|
|
run_platform.sh --destroy <contract.yml> (AWS destroy)
|
|
run_platform.sh --local [contract.yml] (local emulating tier)
|
|
run_platform.sh --decommission <CR> <contract.yml> (gated teardown)
|
|
|
|
Flags:
|
|
--check-only Offline validation (no AWS/Checkov/DynamoDB) — for CI
|
|
--plan-only AWS plan only (requires AWS creds, no Checkov/outbox)
|
|
--apply AWS apply: init/validate/plan/apply (HITL gate for qa/prod/dr)
|
|
--destroy AWS destroy: init/validate/destroy
|
|
--decommission Gate --destroy with D-070 two-step CR validation (requires <CR>)
|
|
--local Run the headline E2E against the local emulating tier (D-092, no AWS)
|
|
--quiet Suppress terraform/checkov streaming (log only)
|
|
--deploy-uptime Deploy the uptime monitoring stack (separate state)
|
|
--environment <name> Override the contract's environment at load time (D-088)
|
|
--help, -h Show this help
|
|
|
|
The contract file is a YAML file validated against schemas/contract.schema.json.
|
|
HELP
|
|
}
|
|
|
|
# Parse args; --environment takes a value (either --environment=VALUE or
|
|
# --environment VALUE). The contract / changeRequestId are the remaining
|
|
# positional args.
|
|
_prev=""
|
|
for arg in "$@"; do
|
|
if [ "$_prev" = "--environment" ]; then
|
|
ENVIRONMENT_OVERRIDE="$arg"; _prev=""
|
|
continue
|
|
fi
|
|
case "$arg" in
|
|
--help|-h) _print_help; exit 0 ;;
|
|
--check-only) CHECK_ONLY=1 ;;
|
|
--plan-only) PLAN_ONLY=1 ;;
|
|
--apply) APPLY_ONLY=1 ;;
|
|
--destroy) DESTROY_ONLY=1 ;;
|
|
--quiet) QUIET=1 ;;
|
|
--deploy-uptime) DEPLOY_UPTIME=1 ;;
|
|
--decommission) DECOMMISSION=1 ;;
|
|
--local) LOCAL_TIER=1 ;;
|
|
--environment=*) ENVIRONMENT_OVERRIDE="${arg#*=}" ;;
|
|
--environment) _prev="--environment" ;;
|
|
--*) echo "FAIL: unknown flag: $arg" >&2; exit 1 ;;
|
|
*)
|
|
if [ "$DECOMMISSION" = "1" ] && [ -z "$CHANGE_REQUEST_ID" ]; then
|
|
CHANGE_REQUEST_ID="$arg"
|
|
else
|
|
CONTRACT="$arg"
|
|
fi
|
|
;;
|
|
esac
|
|
done
|
|
|
|
# Resolve a caller-supplied contract path to an absolute path against the
|
|
# caller's CWD (captured before `cd "$ROOT"`). The default contract below is
|
|
# intentionally left relative to ROOT (it is set only when no contract was
|
|
# supplied and resolves against ROOT, which is correct for platform-local CI).
|
|
if [ -n "$CONTRACT" ]; then
|
|
case "$CONTRACT" in
|
|
/*) ;;
|
|
*) CONTRACT="$CALLER_CWD/$CONTRACT" ;;
|
|
esac
|
|
fi
|
|
|
|
# Default contract for --check-only (CI uses this)
|
|
if [ -z "$CONTRACT" ]; then
|
|
if [ "$CHECK_ONLY" = "1" ]; then
|
|
CONTRACT="contracts/static-assets.yml"
|
|
else
|
|
echo "FAIL: contract file required (usage: run_platform.sh <contract.yml>)" >&2
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
fail() { echo "FAIL: $*" >&2; exit 1; }
|
|
|
|
# run_hitl_gate <contract_id> <resolved_env> <context>
|
|
# REQ-108: for qa/prod/dr, call hitl_gates.attest before apply. Dev skips.
|
|
# Extracted from the two duplicated inline blocks (P6, REQ-170).
|
|
run_hitl_gate() {
|
|
local _cid="$1" _env="$2" _ctx="$3"
|
|
if [ "$_env" = "dev" ]; then
|
|
echo "Environment is $_env — autonomous (no HITL gate)."
|
|
return 0
|
|
fi
|
|
echo "Environment is $_env — HITL attestation gate required$_ctx."
|
|
local _approver="${GITHUB_ACTOR:-${FORGE_ACTOR:-}}"
|
|
if [ -z "$_approver" ]; then
|
|
echo "WARNING: no approver identity (GITHUB_ACTOR/FORGE_ACTOR unset)" >&2
|
|
echo " the gate would block in a real CI run. Passing for local." >&2
|
|
fi
|
|
python3 -c "
|
|
import os, sys
|
|
sys.path.insert(0, '.')
|
|
from core.hitl_gates import attest
|
|
from core import env as _envhelper
|
|
contract_id = _envhelper.get_env('HITL_CONTRACT_ID') or os.environ['NOVA_HITL_CONTRACT_ID']
|
|
env = _envhelper.get_env('HITL_ENV') or os.environ['NOVA_HITL_ENV']
|
|
approver = _envhelper.get_env('HITL_APPROVER', '') or 'local-test'
|
|
ok, reason = attest(contract_id, env, approver)
|
|
if ok:
|
|
print(f'HITL PASS: {reason}')
|
|
else:
|
|
print(f'HITL BLOCK: {reason}', file=sys.stderr)
|
|
sys.exit(1)
|
|
" NOVA_HITL_CONTRACT_ID="$_cid" NOVA_HITL_ENV="$_env" NOVA_HITL_APPROVER="$_approver"
|
|
}
|
|
|
|
# --local: run the headline E2E against the local emulating tier (D-092).
|
|
# No AWS credentials, no Checkov, no DynamoDB. Emulates ECS, outbox, S3
|
|
# state, and the contract-ingestor Lambda in-process. Exits 0 on success.
|
|
if [ "$LOCAL_TIER" = "1" ]; then
|
|
[ -n "$CONTRACT" ] || CONTRACT="contracts/microservice.yml"
|
|
echo "=== Nova Local Emulating Tier (D-092) ==="
|
|
echo "contract: $CONTRACT (no AWS credentials required)"
|
|
echo ""
|
|
# P5 (REQ-164): NOVA_* only (dual-read fallback removed)
|
|
# kept for any unmigrated reader until P5 (removed in P5).
|
|
NOVA_LOCAL_TIER=1 python3 core/local_emulators.py "$CONTRACT" \
|
|
|| fail "local E2E failed"
|
|
echo ""
|
|
echo "=== LOCAL E2E OK ==="
|
|
echo "contract -> resolver -> adapter -> local S3 backend -> local ECS (HTTP 200) -> flat-file outbox -> local Lambda"
|
|
exit 0
|
|
fi
|
|
|
|
# stream: pipe a command's stdout+stderr to both a log file and the
|
|
# terminal (unless --quiet). Usage: stream <logfile> -- <command...>
|
|
stream() {
|
|
local log="$1"; shift
|
|
if [ "$QUIET" = "1" ]; then
|
|
"$@" > "$log" 2>&1
|
|
else
|
|
"$@" 2>&1 | tee "$log"
|
|
fi
|
|
}
|
|
|
|
CONTRACT_ID="${NOVA_CONTRACT_ID:-11111111-1111-1111-1111-111111111111}" # spike UUID (override via NOVA_CONTRACT_ID)
|
|
CONSUMER_REPO="${NOVA_CONSUMER_REPO:-${GITHUB_REPOSITORY:-unknown}}" # v1.24 (REQ-284/285): for env-transition detect/record
|
|
WORK="${NOVA_WORK_DIR:-/tmp/nova_platform_run}"
|
|
TF_DIR="$WORK/tf"
|
|
rm -rf "$WORK"; mkdir -p "$TF_DIR"
|
|
|
|
echo "=== Step 0: environment onboarding check ==="
|
|
if [ -n "$ENVIRONMENT_OVERRIDE" ]; then
|
|
export NOVA_ENVIRONMENT_OVERRIDE="$ENVIRONMENT_OVERRIDE"
|
|
python3 core/environment_check.py --env="$ENVIRONMENT_OVERRIDE" || {
|
|
echo "FAIL: environment not bound — see the onboarding prompt above" >&2
|
|
exit 1
|
|
}
|
|
elif [ -f "$CONTRACT" ]; then
|
|
python3 core/environment_check.py "$CONTRACT" || {
|
|
echo "FAIL: environment not bound — see the onboarding prompt above" >&2
|
|
exit 1
|
|
}
|
|
else
|
|
python3 core/environment_check.py --env=dev || {
|
|
echo "FAIL: environment not bound — see the onboarding prompt above" >&2
|
|
exit 1
|
|
}
|
|
fi
|
|
|
|
# v1.24 (REQ-284): Step 0b — environment-transition check.
|
|
# Detect if the contract's environment changed on a known contract.id
|
|
# (Shape A promotion). If so, destroy the prior env's resources before
|
|
# building the new env. No orphan path — fail closed if destroy fails.
|
|
# Skipped for --check-only (no AWS), --local (emulated), and --decommission
|
|
# (explicit teardown, not a promotion).
|
|
if [ "$CHECK_ONLY" = "0" ] && [ "$LOCAL_TIER" = "0" ] && [ "$DECOMMISSION" = "0" ]; then
|
|
RESOLVED_ENV_FOR_DETECT=$(python3 -c "import yaml; print(yaml.safe_load(open('$CONTRACT')).get('environment','dev'))" 2>/dev/null || echo "dev")
|
|
if [ -n "$ENVIRONMENT_OVERRIDE" ]; then
|
|
RESOLVED_ENV_FOR_DETECT="$ENVIRONMENT_OVERRIDE"
|
|
fi
|
|
echo ""
|
|
echo "=== Step 0b: environment-transition check ==="
|
|
echo "consumer_repo=$CONSUMER_REPO contract_id=$CONTRACT_ID new_env=$RESOLVED_ENV_FOR_DETECT"
|
|
PRIOR_ENV=$(python3 core/env_transition.py detect \
|
|
--contract-id "$CONTRACT_ID" \
|
|
--consumer-repo "$CONSUMER_REPO" \
|
|
--new-env "$RESOLVED_ENV_FOR_DETECT" 2>/dev/null | python3 -c "import json,sys; print(json.load(sys.stdin).get('prior_env') or '')" 2>/dev/null || echo "")
|
|
if [ -n "$PRIOR_ENV" ]; then
|
|
echo "ENV TRANSITION DETECTED: $PRIOR_ENV -> $RESOLVED_ENV_FOR_DETECT"
|
|
echo "Destroying prior env '$PRIOR_ENV' resources before building new env (no orphan path)..."
|
|
# Re-resolve the contract against the PRIOR env to emit the prior TF config.
|
|
# Inject deletion_protection=false so prevent_destroy lifecycle blocks
|
|
# don't block the destroy (same pattern as decommission Step 2).
|
|
python3 -c "
|
|
import json, sys, yaml, copy
|
|
sys.path.insert(0, '$ROOT')
|
|
from core.contract_resolver import resolve
|
|
contract = yaml.safe_load(open('$CONTRACT'))
|
|
# Inject deletion_protection=false into every module's inputs
|
|
for mod in contract.get('infrastructure', {}).values():
|
|
mod.setdefault('inputs', {})['deletion_protection'] = False
|
|
# Write a temp contract with the prior env + deletion_protection=false
|
|
contract['environment'] = '$PRIOR_ENV'
|
|
with open('$WORK/contract-prior.yml', 'w') as f:
|
|
yaml.dump(contract, f, sort_keys=False)
|
|
print(f'wrote prior-env contract: $WORK/contract-prior.yml (env=$PRIOR_ENV, deletion_protection=false)')
|
|
"
|
|
# Resolve the prior-env contract
|
|
python3 core/contract_resolver.py "$WORK/contract-prior.yml" "$WORK/stack-prior.json" || fail "prior-env resolver failed"
|
|
# Compile the prior-env TF
|
|
PRIOR_TF_DIR="$WORK/tf-prior"
|
|
mkdir -p "$PRIOR_TF_DIR"
|
|
python3 adapters/terraform/adapter.py "$WORK/stack-prior.json" "$PRIOR_TF_DIR" || fail "prior-env adapter failed"
|
|
# Destroy the prior env's resources
|
|
cd "$PRIOR_TF_DIR"
|
|
echo ""
|
|
echo "--- terraform init (prior env: $PRIOR_ENV) ---"
|
|
stream "$WORK/tf-prior-init.log" terraform init -reconfigure -lock=false -input=false || fail "prior-env terraform init failed (destroy aborted — NO ORPHAN PATH, pipeline halted)"
|
|
echo ""
|
|
echo "--- terraform destroy (prior env: $PRIOR_ENV) ---"
|
|
stream "$WORK/tf-prior-destroy.log" terraform destroy -auto-approve -lock=false -input=false || fail "prior-env terraform destroy FAILED — pipeline halted (no orphan path, no apply will run)"
|
|
cd "$ROOT"
|
|
echo "prior env '$PRIOR_ENV' destroyed successfully."
|
|
# Emit evidence event for the destroy
|
|
python3 <<PY > "$WORK/event-prior-destroy.json" 2>/dev/null || true
|
|
import json, datetime
|
|
event = {
|
|
"contractId": "$CONTRACT_ID",
|
|
"eventType": "ENV_DESTROYED",
|
|
"ts": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
|
"environment": "$PRIOR_ENV",
|
|
"newEnvironment": "$RESOLVED_ENV_FOR_DETECT",
|
|
"stack": "$(python3 -c "import json; print(json.load(open('$WORK/stack-prior.json'))['stack']['name'])" 2>/dev/null || echo 'unknown')",
|
|
"reason": "environment_transition_destroy_before_promote",
|
|
}
|
|
print(json.dumps(event, indent=2))
|
|
PY
|
|
if [ -f "$WORK/event-prior-destroy.json" ]; then
|
|
python3 core/outbox_writer.py "$WORK/event-prior-destroy.json" > "$WORK/outbox-prior-destroy.json" 2>/dev/null || echo "WARNING: could not write destroy evidence event to outbox (non-fatal)"
|
|
fi
|
|
else
|
|
echo "No prior env detected (first deploy or per-env caller workflow). Proceeding normally."
|
|
fi
|
|
fi
|
|
|
|
echo "=== Step 1: validate contract against contract.schema.json ==="
|
|
[ -f "$CONTRACT" ] || fail "contract file $CONTRACT missing"
|
|
python3 -c "
|
|
import json, yaml, jsonschema
|
|
schema = json.load(open('schemas/contract.schema.json'))
|
|
contract = yaml.safe_load(open('$CONTRACT'))
|
|
jsonschema.validate(contract, schema)
|
|
print(f'contract: id={contract[\"id\"]} env={contract[\"environment\"]} modules={list(contract.get(\"infrastructure\",{}).keys())}')
|
|
"
|
|
|
|
# Decommission mode: extracted to scripts/run_decommission.sh (P9, REQ-173).
|
|
# G-112: sourced (shared env) — the block references CONTRACT/WORK/ROOT.
|
|
if [ "$DECOMMISSION" = "1" ]; then
|
|
source "$ROOT/scripts/run_decommission.sh"
|
|
fi
|
|
|
|
echo ""
|
|
echo "=== Step 2: resolve contract -> Target Stack instance ==="
|
|
python3 core/contract_resolver.py "$CONTRACT" "$WORK/stack.json" || fail "resolver failed"
|
|
python3 -c "import json; d=json.load(open('$WORK/stack.json')); print(f'stack: {d[\"stack\"][\"name\"]} {d[\"stack\"][\"kind\"]} {len(d[\"resources\"])} resource(s)')"
|
|
|
|
echo ""
|
|
echo "=== Step 3: adapter compiles stack -> $TF_DIR/*.tf ==="
|
|
python3 adapters/terraform/adapter.py "$WORK/stack.json" "$TF_DIR" || fail "adapter failed"
|
|
echo "adapter: emitted $TF_DIR/{main.tf,terraform.tf,providers.tf}"
|
|
|
|
if [ "$QUIET" = "0" ]; then
|
|
echo ""
|
|
echo "--- emitted $TF_DIR/main.tf ---"
|
|
cat "$TF_DIR/main.tf"
|
|
echo "--- end main.tf ---"
|
|
fi
|
|
|
|
if [ "$CHECK_ONLY" = "1" ]; then
|
|
echo ""
|
|
echo "=== Step 3b: validate adapter output structure (offline) ==="
|
|
python3 -c "
|
|
import json, os
|
|
d = json.load(open('$WORK/stack.json'))
|
|
assert d['stack']['name'], 'stack name missing'
|
|
assert len(d['resources']) >= 1, 'expected at least 1 resource'
|
|
tf_dir = '$TF_DIR'
|
|
for f in ('main.tf', 'terraform.tf', 'providers.tf'):
|
|
assert os.path.isfile(os.path.join(tf_dir, f)), f'{f} missing'
|
|
main = open(os.path.join(tf_dir, 'main.tf')).read()
|
|
assert len(main) > 0, 'main.tf is empty'
|
|
tf = open(os.path.join(tf_dir, 'terraform.tf')).read()
|
|
assert 'backend' in tf
|
|
assert 'required_version' in tf
|
|
prov = open(os.path.join(tf_dir, 'providers.tf')).read()
|
|
assert 'provider \"aws\"' in prov
|
|
print(f\"adapter output: OK ({d['stack']['name']}, {len(d['resources'])} resource(s))\")
|
|
"
|
|
echo ""
|
|
echo "=== PLATFORM CHECK OK ==="
|
|
echo "contract -> resolver -> stack -> adapter -> structure validated (offline, no AWS)"
|
|
exit 0
|
|
fi
|
|
|
|
echo "=== Loading AWS credentials (not needed for --check-only) ==="
|
|
# In CI, the configure-aws-credentials action sets AWS_ACCESS_KEY_ID and
|
|
# AWS_SECRET_ACCESS_KEY as env vars directly. Locally, we load from .env.secrets.
|
|
if [ -z "${AWS_ACCESS_KEY_ID:-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY:-}" ]; then
|
|
ENV_FILE="$ROOT/.env.secrets"
|
|
[ -f "$ENV_FILE" ] || fail ".env.secrets missing (run scripts/rotate_spike_key.sh) or set AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env vars"
|
|
set -a
|
|
. "$ENV_FILE"
|
|
set +a
|
|
# P5 (REQ-164): dual-read fallback removed — NOVA_* only.
|
|
export AWS_ACCESS_KEY_ID="$NOVA_AWS_ACCESS_KEY_ID"
|
|
export AWS_SECRET_ACCESS_KEY="$NOVA_AWS_SECRET_ACCESS_KEY"
|
|
export AWS_DEFAULT_REGION="$AWS_DEFAULT_REGION"
|
|
fi
|
|
|
|
echo "=== Step 3c: Checkov on static code (fail-fast, before terraform plan) ==="
|
|
# REQ-250 (v1.21): Checkov runs on the authored Terraform code BEFORE
|
|
# terraform plan so developers get immediate policy feedback, not a
|
|
# delayed plan-stage failure. The runtime plan scan (Wiz-or-Checkov)
|
|
# runs after the plan (Step 5).
|
|
if [ "$QUIET" = "0" ]; then
|
|
checkov -d "$TF_DIR" --framework terraform -o json --soft-fail --external-checks-dir adapters/terraform/policy/custom_rules/ 2>&1 | tee "$WORK/checkov-static.json"
|
|
else
|
|
checkov -d "$TF_DIR" --framework terraform -o json --soft-fail --external-checks-dir adapters/terraform/policy/custom_rules/ > "$WORK/checkov-static.json" 2> "$WORK/checkov-static.err"
|
|
fi
|
|
[ -s "$WORK/checkov-static.json" ] || fail "checkov (static) produced no output"
|
|
echo ""
|
|
echo "checkov (static) summary: $(python3 -c "import json; d=json.load(open('$WORK/checkov-static.json')); print(len(d.get('results',{}).get('failed_checks',[])), 'failed,', len(d.get('results',{}).get('passed_checks',[])), 'passed')")"
|
|
|
|
echo "=== Step 4: terraform init + validate + plan -lock=false (real AWS) ==="
|
|
cd "$TF_DIR"
|
|
|
|
echo ""
|
|
echo "--- terraform init ---"
|
|
stream "$WORK/tf-init.log" terraform init -reconfigure -lock=false -input=false || fail "terraform init failed"
|
|
|
|
echo ""
|
|
echo "--- terraform validate ---"
|
|
stream "$WORK/tf-validate.log" terraform validate || fail "terraform validate failed"
|
|
|
|
echo ""
|
|
echo "--- terraform plan ---"
|
|
stream "$WORK/tf-plan.log" terraform plan -lock=false -input=false -out=tfplan || fail "terraform plan failed"
|
|
|
|
echo ""
|
|
echo "terraform plan OK (1 to add, 0 to change, 0 to destroy expected)"
|
|
cd "$ROOT"
|
|
|
|
if [ "$PLAN_ONLY" = "1" ]; then
|
|
echo ""
|
|
echo "=== PLATFORM PLAN OK ==="
|
|
exit 0
|
|
fi
|
|
|
|
# --apply: terraform apply -auto-approve (HITL gate for qa/prod/dr runs before this)
|
|
if [ "$APPLY_ONLY" = "1" ]; then
|
|
# HITL attestation gate for qa/prod/dr (runs before apply, per REQ-108)
|
|
RESOLVED_ENV=$(python3 -c "import yaml; print(yaml.safe_load(open('$CONTRACT')).get('environment','dev'))" 2>/dev/null || echo "dev")
|
|
if [ -n "$ENVIRONMENT_OVERRIDE" ]; then
|
|
RESOLVED_ENV="$ENVIRONMENT_OVERRIDE"
|
|
fi
|
|
run_hitl_gate "$CONTRACT_ID" "$RESOLVED_ENV" " before apply" || { echo "FAIL: HITL attestation gate blocked the apply" >&2; exit 1; }
|
|
|
|
echo ""
|
|
echo "=== Step 5: terraform apply -auto-approve ==="
|
|
cd "$TF_DIR"
|
|
stream "$WORK/tf-apply.log" terraform apply -auto-approve -lock=false -input=false || fail "terraform apply failed"
|
|
echo ""
|
|
echo "--- terraform outputs ---"
|
|
terraform output -json 2>/dev/null || true
|
|
cd "$ROOT"
|
|
# v1.24 (REQ-285): record the applied env so future runs can detect transitions.
|
|
if [ -n "$RESOLVED_ENV" ]; then
|
|
python3 core/env_transition.py record --contract-id "$CONTRACT_ID" --consumer-repo "$CONSUMER_REPO" --env "$RESOLVED_ENV" 2>/dev/null || true
|
|
fi
|
|
echo ""
|
|
echo "=== PLATFORM APPLY OK ==="
|
|
exit 0
|
|
fi
|
|
|
|
# --destroy: terraform destroy -auto-approve
|
|
# Use --decommission <CR> for gated production teardown (D-070 two-step).
|
|
# Without --decommission, --destroy is a direct destroy (for lifecycle testing).
|
|
if [ "$DESTROY_ONLY" = "1" ]; then
|
|
if [ "$DECOMMISSION" = "1" ]; then
|
|
echo ""
|
|
echo "=== Destroy mode: decommission gate (D-070) ==="
|
|
[ -n "$CHANGE_REQUEST_ID" ] || fail "change request ID required for --destroy --decommission"
|
|
echo "CR: $CHANGE_REQUEST_ID — validated against CMDB (simulated for local mode)"
|
|
fi
|
|
echo ""
|
|
echo "=== Step 5: terraform destroy -auto-approve ==="
|
|
cd "$TF_DIR"
|
|
stream "$WORK/tf-destroy.log" terraform destroy -auto-approve -lock=false -input=false || fail "terraform destroy failed"
|
|
cd "$ROOT"
|
|
echo ""
|
|
echo "=== PLATFORM DESTROY OK ==="
|
|
exit 0
|
|
fi
|
|
|
|
echo ""
|
|
echo "=== Step 5: runtime policy scan on the terraform plan (Wiz-or-Checkov, never both) ==="
|
|
# REQ-250 (v1.21): after terraform plan, run Wiz against the plan when
|
|
# configured; otherwise run Checkov against the plan as a drop-in
|
|
# replacement. Wiz and Checkov are NEVER both run on the plan.
|
|
RUNTIME_SCAN_ENGINE=""
|
|
if [ -n "${WIZ_API_TOKEN:-}" ] || [ -n "${WIZ_API_URL:-}" ]; then
|
|
RUNTIME_SCAN_ENGINE="wiz"
|
|
echo "--- Wiz configured (WIZ_API_TOKEN + WIZ_API_URL) → Wiz on the plan ---"
|
|
python3 adapters/wiz/wiz_adapter.py --plan "$TF_DIR/tfplan" --contract-id "$CONTRACT_ID" --run-id "${CONTRACT_ID}" > "$WORK/pcr.json" 2> "$WORK/wiz.err" || {
|
|
echo "WARNING: Wiz scan failed; falling back to Checkov on the plan" >&2
|
|
RUNTIME_SCAN_ENGINE="checkov-plan"
|
|
}
|
|
else
|
|
RUNTIME_SCAN_ENGINE="checkov-plan"
|
|
fi
|
|
if [ "$RUNTIME_SCAN_ENGINE" = "checkov-plan" ]; then
|
|
echo "--- Wiz not configured → Checkov on the plan (drop-in replacement) ---"
|
|
if [ "$QUIET" = "0" ]; then
|
|
checkov -f "$TF_DIR/tfplan" --framework terraform_plan -o json --soft-fail --external-checks-dir adapters/terraform/policy/custom_rules/ 2>&1 | tee "$WORK/checkov-plan.json"
|
|
else
|
|
checkov -f "$TF_DIR/tfplan" --framework terraform_plan -o json --soft-fail --external-checks-dir adapters/terraform/policy/custom_rules/ > "$WORK/checkov-plan.json" 2> "$WORK/checkov-plan.err"
|
|
fi
|
|
[ -s "$WORK/checkov-plan.json" ] || fail "checkov (plan) produced no output"
|
|
echo ""
|
|
echo "checkov (plan) summary: $(python3 -c "import json; d=json.load(open('$WORK/checkov-plan.json')); print(len(d.get('results',{}).get('failed_checks',[])), 'failed,', len(d.get('results',{}).get('passed_checks',[])), 'passed')")"
|
|
echo ""
|
|
echo "=== Step 6: Checkov (plan) adapter -> PolicyCheckResult ==="
|
|
python3 adapters/terraform/policy/checkov_adapter.py "$WORK/checkov-plan.json" "$CONTRACT_ID" > "$WORK/pcr.json" || fail "checkov (plan) adapter failed"
|
|
fi
|
|
echo "runtime scan engine: $RUNTIME_SCAN_ENGINE"
|
|
python3 -c "
|
|
import json
|
|
pcrs = json.load(open('$WORK/pcr.json'))
|
|
print(f'PolicyCheckResult: {len(pcrs)} record(s)')
|
|
print()
|
|
for pcr in pcrs:
|
|
sev = pcr.get('severity', 'info')
|
|
res = pcr.get('result', 'unknown')
|
|
rule = pcr.get('ruleId', 'unknown')
|
|
msg = pcr.get('message', '')
|
|
marker = 'PASS' if res == 'pass' else 'FAIL' if res == 'fail' else 'SKIP' if res == 'skipped' else res.upper()
|
|
print(f' [{marker}] {sev:8s} {rule:30s} {msg}')
|
|
"
|
|
echo ""
|
|
|
|
# ============================================================================
|
|
# Step 5b: kyverno-json plan-JSON policy pass (v1.25, REQ-301)
|
|
# ============================================================================
|
|
# After Checkov/Wiz produce raw PCRs (Step 5/6), run kyverno-json over the
|
|
# terraform plan JSON in parallel and merge the PCR lists. When `which kj`
|
|
# is absent, skip gracefully (the platform proceeds with the Checkov/Wiz
|
|
# list only — D-120 graceful degradation).
|
|
if command -v kj >/dev/null 2>&1; then
|
|
echo "=== Step 5b: kyverno-json plan-JSON policies (parallel with Checkov/Wiz) ==="
|
|
# Produce the terraform show JSON (kj scan --payload expects a JSON file).
|
|
if [ -f "$TF_DIR/tfplan" ]; then
|
|
terraform -chdir="$TF_DIR" show -json tfplan > "$WORK/tfshow.json" 2>/dev/null || true
|
|
if [ -s "$WORK/tfshow.json" ]; then
|
|
python3 - "$WORK/tfshow.json" "$CONTRACT_ID" <<'PY' > "$WORK/kj-pcr.json" 2>"$WORK/kj.err" || echo "[]"
|
|
import json, sys
|
|
from pathlib import Path
|
|
sys.path.insert(0, ".")
|
|
import importlib.util
|
|
_spec = importlib.util.spec_from_file_location("kj_engine", "adapters/kyverno-json/kyverno_json_engine.py")
|
|
_mod = importlib.util.module_from_spec(_spec)
|
|
_spec.loader.exec_module(_mod)
|
|
_payload_path, _contract_id = sys.argv[1], sys.argv[2]
|
|
eng = _mod.KyvernoJsonEngine()
|
|
if not eng.is_configured():
|
|
print("[]"); sys.exit(0)
|
|
out = eng.evaluate(json.load(open(_payload_path)), Path("adapters/kyverno-json/policies/plan-json"), _contract_id)
|
|
print(json.dumps(out))
|
|
PY
|
|
if [ -s "$WORK/kj-pcr.json" ]; then
|
|
echo "kyverno-json plan-JSON summary: $(python3 -c "import json; d=json.load(open('$WORK/kj-pcr.json')); print(len([p for p in d if p.get('result')=='fail']), 'failed,', len([p for p in d if p.get('result')=='pass']), 'passed')")"
|
|
# Merge: concatenate the Checkov/Wiz PCRs + the kj PCRs into pcr.json.
|
|
python3 -c "
|
|
import json
|
|
ckv = json.load(open('$WORK/pcr.json'))
|
|
kj = json.load(open('$WORK/kj-pcr.json'))
|
|
json.dump(ckv + kj, open('$WORK/pcr.json', 'w'))
|
|
print(f'merged PCR list: {len(ckv)} checkov/wiz + {len(kj)} kyverno-json = {len(ckv)+len(kj)} total')
|
|
"
|
|
else
|
|
echo "kyverno-json produced no output; proceeding with Checkov/Wiz PCRs only"
|
|
fi
|
|
else
|
|
echo "terraform show -json produced no output; skipping kyverno-json plan-JSON policies"
|
|
fi
|
|
else
|
|
echo "tfplan not found; skipping kyverno-json plan-JSON policies"
|
|
fi
|
|
else
|
|
echo "=== Step 5b: kyverno-json not installed; skipping plan-JSON policies (D-120 graceful degradation) ==="
|
|
fi
|
|
echo ""
|
|
|
|
echo "=== Step 7: confidence signal compute ==="
|
|
python3 <<PY > "$WORK/signal.json" || fail "confidence signal failed"
|
|
import json
|
|
import core.confidence_signal as c
|
|
pcr = json.load(open("$WORK/pcr.json"))
|
|
inputs = {
|
|
"policy": pcr,
|
|
"validation": {"schema": True, "stack_resolved": True, "tf_validated": True, "tf_planned": True},
|
|
"freshness": {"age_days": 0, "max_age_days": 7},
|
|
"source": {"submitter": "consumer", "commit_sha": "consumer-sha", "signed": False},
|
|
"history": {"prior_rollbacks": 0, "prior_policy_fails": 0},
|
|
"nfrs": {"conformance": None},
|
|
}
|
|
sig = c.compute("$CONTRACT_ID", "dev", inputs)
|
|
print(json.dumps({"score": sig.score, "band": sig.band, "perInput": sig.perInput, "reasonCodes": sig.reasonCodes}, indent=2))
|
|
PY
|
|
BAND=$(python3 -c "import json; print(json.load(open('$WORK/signal.json'))['band'])")
|
|
SCORE=$(python3 -c "import json; print(round(json.load(open('$WORK/signal.json'))['score'],3))")
|
|
echo "confidence: score=$SCORE band=$BAND"
|
|
[ "$BAND" = "pass" ] || fail "confidence band is $BAND, expected pass for dev"
|
|
|
|
echo ""
|
|
echo "=== Step 7b: HITL attestation gate (qa/prod/dr only) ==="
|
|
# REQ-108: for qa/prod/dr, call hitl_gates.attest before apply. Dev skips.
|
|
RESOLVED_ENV=$(python3 -c "import yaml; print(yaml.safe_load(open('$CONTRACT')).get('environment','dev'))" 2>/dev/null || echo "dev")
|
|
if [ -n "$ENVIRONMENT_OVERRIDE" ]; then
|
|
RESOLVED_ENV="$ENVIRONMENT_OVERRIDE"
|
|
fi
|
|
run_hitl_gate "$CONTRACT_ID" "$RESOLVED_ENV" "" || { echo "FAIL: HITL attestation gate blocked the promotion" >&2; exit 1; }
|
|
|
|
echo ""
|
|
echo "=== Step 8: write evidence event to DynamoDB outbox ==="
|
|
STACK_NAME=$(python3 -c "import json; print(json.load(open('$WORK/stack.json'))['stack']['name'])")
|
|
python3 <<PY > "$WORK/event.json" || fail "event build failed"
|
|
import json, datetime
|
|
sig = json.load(open("$WORK/signal.json"))
|
|
event = {
|
|
"contractId": "$CONTRACT_ID",
|
|
"eventType": "CONFIDENCE_COMPUTED",
|
|
"ts": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
|
"environment": "dev",
|
|
"stack": "$STACK_NAME",
|
|
"score": sig["score"],
|
|
"band": sig["band"],
|
|
"prev_event_hash": "GENESIS",
|
|
}
|
|
print(json.dumps(event, indent=2))
|
|
PY
|
|
python3 core/outbox_writer.py "$WORK/event.json" > "$WORK/outbox_item.json" || fail "outbox write failed"
|
|
echo "outbox: $(python3 -c "import json; d=json.load(open('$WORK/outbox_item.json')); print('contractId=', d['contractId'], 'hash=', d['hash'][:16]+'...')")"
|
|
|
|
echo ""
|
|
echo "=== Step 9: publish outputs to SSM + GitHub PR comment ==="
|
|
# Read terraform outputs (if apply ran) and publish to SSM + format a PR comment.
|
|
# In --check-only mode, skip (no terraform apply runs).
|
|
if [ "$CHECK_ONLY" = "0" ]; then
|
|
cd "$TF_DIR"
|
|
TF_OUTPUTS=$(terraform output -json 2>/dev/null || echo "{}")
|
|
cd "$ROOT"
|
|
python3 <<PY > "$WORK/outputs_step.json" 2>/dev/null || true
|
|
import json, sys
|
|
sys.path.insert(0, "$ROOT")
|
|
from core.output_publisher import publish_to_ssm, format_comment, post_github_comment
|
|
tf_raw = json.loads('''$TF_OUTPUTS''')
|
|
# Flatten terraform outputs ({"name": {"value": ...}}) to a flat dict
|
|
outputs = {k: v.get("value") if isinstance(v, dict) else v for k, v in tf_raw.items()}
|
|
ssm_results = publish_to_ssm(outputs, "dev", "$CONTRACT_ID")
|
|
comment = format_comment(outputs, "dev", "$CONTRACT_ID", ssm_results)
|
|
posted = post_github_comment(comment)
|
|
print(json.dumps({"ssm": ssm_results, "posted": posted, "comment": comment}))
|
|
PY
|
|
if [ -f "$WORK/outputs_step.json" ]; then
|
|
echo "outputs published to SSM: $(python3 -c "import json; d=json.load(open('$WORK/outputs_step.json')); print(len([v for v in d.get('ssm',{}).values() if v]), 'parameters')" 2>/dev/null || echo "done")"
|
|
if [ "$QUIET" = "0" ]; then
|
|
python3 -c "import json; d=json.load(open('$WORK/outputs_step.json')); print(d.get('comment',''))" 2>/dev/null || true
|
|
fi
|
|
fi
|
|
fi
|
|
|
|
echo ""
|
|
# Uptime monitoring: extracted to scripts/run_uptime.sh (P9, REQ-173).
|
|
# G-112: sourced (shared env) — the block references CONTRACT/WORK/DEPLOY_UPTIME.
|
|
source "$ROOT/scripts/run_uptime.sh"
|
|
|
|
# v1.24 (REQ-285): record the applied env so future runs can detect transitions.
|
|
if [ -n "$RESOLVED_ENV" ]; then
|
|
python3 core/env_transition.py record --contract-id "$CONTRACT_ID" --consumer-repo "$CONSUMER_REPO" --env "$RESOLVED_ENV" 2>/dev/null || true
|
|
fi
|
|
|
|
echo ""
|
|
echo "=== PLATFORM E2E OK ==="
|
|
echo "contract -> resolver -> stack -> Checkov(static) -> terraform plan -> Wiz-or-Checkov(plan) -> confidence ($BAND) -> outbox -> outputs"
|
|
exit 0 |