docs(P00): research findings — v1.25 kyverno-json engine surface, 4 policy targets, PolicyEngine swap boundary
RESEARCH.md: kyverno-json CLI (kj scan), ValidatingPolicy structure, assertion trees + ~ modifier + JMESPath, output shape, severity-via- annotation convention, 4 policy targets (contract/stack-IR/plan-JSON/ meta), PolicyEngine protocol + OPA-equivalent swap surface, latency <1s (parallel with checkov), deterministic-not-AI tenet, ECS catalog prior art, 5 logged assumptions (A1..A5). PERSONAS.md: 4 active personas (lead-developer, backend-engineer, new policy-engineer, data-engineer); frontend-engineer deactivated. policy-engineer owns kyverno-json policies + engine translation + STANDARDS.md policy-authoring section. ARCHITECTURE.md §12.7: Policy Engine Registry — protocol, registry, NullEngine fallback, engine enum reuse (D-116), defense-in-depth critical-override (D-119), graceful degradation (D-120). ---ci--- project: acdl phase: 0 milestone: v1.25 status: research ---/ci---
This commit is contained in:
Jon Chery
@@ -879,3 +879,67 @@ config entry in `config.json` (`strategic_direction_file:
|
||||
".ciagent/NORTH_STAR.md"`) that the run workflow reads at SPECIFY. This
|
||||
ensures the strategic direction survives across milestones without
|
||||
being overwritten by status updates.
|
||||
|
||||
### §12.7 — Policy Engine Registry (v1.25, REQ-291)
|
||||
|
||||
The policy-engine abstraction is first-class: a swappable `PolicyEngine`
|
||||
protocol so the engine may change without touching the confidence
|
||||
signal, the pipeline, or the `PolicyCheckResult` schema. This is the
|
||||
**swap boundary** that keeps the platform's compliance posture
|
||||
replaceable (Strategic Objective #2 — provable trust via a replaceable
|
||||
substrate, not a vendor lock-in).
|
||||
|
||||
```
|
||||
contract.yml ─┐ ┌─→ list[PolicyCheckResult] ─┐
|
||||
stack IR ─────┼─→ PolicyEngine.evaluate ├─→ list[PolicyCheckResult] ─┼─→ confidence_signal
|
||||
plan JSON ────┤ (protocol) └─→ list[PolicyCheckResult] ─┘ (engine-agnostic,
|
||||
PCR list ─────┘ unchanged)
|
||||
│
|
||||
▼
|
||||
┌─ KyvernoJsonEngine (shells to `kj scan`; engine: "kyverno")
|
||||
└─ OpaEngine (future — same protocol; engine: "opa")
|
||||
|
||||
checkov/wiz ──→ raw findings ──→ (merged PCR list is the meta-policy payload)
|
||||
```
|
||||
|
||||
**The protocol (`core/policy_engine.py`):**
|
||||
```python
|
||||
class PolicyEngine(Protocol):
|
||||
@property
|
||||
def name(self) -> str: ...
|
||||
def is_configured(self) -> bool: ...
|
||||
def evaluate(self, payload, policy_dir: Path, contract_id: str) -> list[dict]: ...
|
||||
```
|
||||
|
||||
**The registry** reads `config.json.policy.engine` (default
|
||||
`"kyverno-json"`) and returns the active engine. A `NullEngine` is the
|
||||
fallback when the `policy` key is absent (emits `SKIPPED` PCRs —
|
||||
backward compatibility for tests that don't set the key). The
|
||||
confidence signal is **untouched** — it already consumes
|
||||
`list[PolicyCheckResult]` engine-agnostically (§12.6). v1.25 only
|
||||
changes *who produces* the PCR list, not *what* the list is.
|
||||
|
||||
**Engine enum reuse (D-116):** kyverno-json PCR records carry
|
||||
`engine: "kyverno"` (no new enum value). The `engine` field records the
|
||||
policy-engine *family*, not the specific binary. The K8s Kyverno adapter
|
||||
and the kyverno-json engine are distinguished by `ruleId` prefix
|
||||
(`KYVERNO_` vs `KJ_`) and `evidence` payload shape (`namespace`/`kind`
|
||||
vs `assertion`/`jmespath`).
|
||||
|
||||
**Defense-in-depth (D-119):** the declarative meta-policy
|
||||
`block-on-any-critical` (asserts no PCR has `severity: critical` +
|
||||
`result: fail`) is the *source of truth* for "critical = block". The
|
||||
`confidence_signal.py` `PENALTY["critical"]: None` hard-override stays
|
||||
as the *imperative* safety net — the meta-policy runs *before* the
|
||||
confidence signal (produces PCRs that flow in), the hard-override runs
|
||||
*inside* it (the last gate). Removing the hard-override would make the
|
||||
"critical = block" guarantee depend on a single policy file — a
|
||||
regression in provable trust.
|
||||
|
||||
**Graceful degradation (D-120):** `KyvernoJsonEngine.is_configured()`
|
||||
returns false when `which kj` is absent → `evaluate()` returns a single
|
||||
`SKIPPED` PCR (`ruleId: "KJ_ENGINE_NOT_CONFIGURED"`). The platform
|
||||
functions without the binary (the "platform functions without AI /
|
||||
deterministic scripts" tenet holds — kyverno-json is deterministic, not
|
||||
AI; the `is_configured()` guard ensures the platform runs even when the
|
||||
binary is not installed).
|
||||
|
||||
@@ -1,16 +1,17 @@
|
||||
{
|
||||
"phase": 0,
|
||||
"stage": "clarify",
|
||||
"stage": "research",
|
||||
"milestone": "v1.25",
|
||||
"phase_role": "pre_execution",
|
||||
"attempts": 0,
|
||||
"updated_at": "2026-08-12T16:25:00Z",
|
||||
"updated_at": "2026-08-12T16:35:00Z",
|
||||
"project": "acdl",
|
||||
"milestone_complete": false,
|
||||
"tag_line": "v1.24.x",
|
||||
"next_tag": "v1.24.0",
|
||||
"decisions": ["D-115", "D-116", "D-117", "D-118", "D-119", "D-120"],
|
||||
"ambiguities_resolved": 6,
|
||||
"escalations": 0,
|
||||
"notes": "v1.25 CLARIFY complete — 6 ambiguities auto-resolved at full autonomy (D-115..D-120). Install path = go install; engine enum reuse kyverno; checkov/wiz signatures unchanged; tagging-rule cross-check; critical-override defense-in-depth; kyverno-json is deterministic not AI."
|
||||
"personas_active": ["lead-developer", "backend-engineer", "policy-engineer", "data-engineer"],
|
||||
"personas_deactivated": ["frontend-engineer"],
|
||||
"new_personas": ["policy-engineer"],
|
||||
"assumptions": ["A1", "A2", "A3", "A4", "A5"],
|
||||
"notes": "v1.25 RESEARCH complete — kyverno-json CLI/policy/assertion-tree surface documented; 4 policy targets analyzed; PolicyEngine protocol + OPA-equivalent swap surface; new policy-engineer persona; ARCHITECTURE.md §12.7 added. Latency impact <1s (parallel with checkov)."
|
||||
}
|
||||
+108
-59
@@ -1,83 +1,132 @@
|
||||
---
|
||||
project: acdl
|
||||
milestone: v1.24
|
||||
milestone: v1.25
|
||||
generated_at: 2026-08-12
|
||||
generator: lead-developer
|
||||
verification_toolchain:
|
||||
typecheck: "python3 -m py_compile core/env_transition.py tests/test_env_transition.py tests/test_run_platform_env_transition.py"
|
||||
test: "pytest tests/test_env_transition.py tests/test_run_platform_env_transition.py tests/test_consumer_guide_per_env_section.py tests/test_adapter.py tests/test_contract_resolver.py tests/test_deploy_workflow_env_input.py tests/test_pipeline.py -v"
|
||||
lint: "ruff check core/env_transition.py tests/test_env_transition.py tests/test_run_platform_env_transition.py 2>/dev/null || python3 -m py_compile core/env_transition.py"
|
||||
typecheck: "python3 -m py_compile core/policy_engine.py adapters/kyverno-json/kyverno_json_engine.py tests/test_policy_engine.py tests/test_kyverno_json_engine.py"
|
||||
test: "pytest tests/test_policy_engine.py tests/test_kyverno_json_engine.py tests/test_adapter.py tests/test_contract_resolver.py tests/test_confidence_signal.py tests/test_checkov_adapter.py tests/test_kyverno_adapter.py tests/test_pipeline.py -v"
|
||||
lint: "ruff check core/policy_engine.py adapters/kyverno-json/ 2>/dev/null || python3 -m py_compile core/policy_engine.py"
|
||||
note: |
|
||||
v1.24 is the Consumer Guide Accuracy & Env-Promotion Lifecycle
|
||||
Enforcement milestone — a mixed docs+feat+test milestone. Two active
|
||||
personas: lead-developer (consumer-guide.md edits + guide test
|
||||
updates), backend-engineer (core/env_transition.py + run_platform.sh
|
||||
Step 0b + deploy.yml + adapter doc comment + env_transition tests +
|
||||
pipeline tests). frontend-engineer stays deactivated (no UI). No
|
||||
data-engineer (no schema changes — the nova-contracts table already
|
||||
exists). No new personas.
|
||||
v1.25 is the kyverno-json Unified Policy Engine milestone — a feat
|
||||
milestone. Four active personas: lead-developer (coordination +
|
||||
docs + ARCHITECTURE.md §12.7), backend-engineer (core/policy_engine.py
|
||||
protocol + registry + contract_resolver.py wiring + run_platform.sh
|
||||
Step 5 + pipeline tests), policy-engineer (adapters/kyverno-json/
|
||||
engine + policies across all 4 target dirs + meta-policies + policy
|
||||
tests + adapter README + STANDARDS.md policy-authoring section),
|
||||
data-engineer (config.json policy object + schemas/README.md note +
|
||||
capability-inventory JSON fixture for regression policies).
|
||||
frontend-engineer stays deactivated (no UI). The policy-engineer is a
|
||||
new custom persona created for this milestone's policy domain (see
|
||||
RESEARCH.md §4 — kyverno-json + JMESPath is a distinct framework from
|
||||
backend-engineer's fastify/hono).
|
||||
---
|
||||
|
||||
# ACDL — Persona Roster (v1.24 Consumer Guide Accuracy & Env-Promotion Lifecycle Enforcement)
|
||||
# ACDL — Persona Roster (v1.25 kyverno-json Unified Policy Engine)
|
||||
|
||||
> v1.24 roster. Two active personas + two deactivated. This is a mixed
|
||||
> docs+feat+test milestone: the work is consumer-guide accuracy fixes
|
||||
> (lead-developer), a new env-transition detect-and-destroy platform
|
||||
> feature (backend-engineer), and test coverage for both (split).
|
||||
> v1.25 roster. Four active personas + one deactivated. This is a feat
|
||||
> milestone: the work is a swappable policy-engine protocol + a new
|
||||
> adapter + policies across 4 Nova artifacts + pipeline wiring + docs.
|
||||
> The policy-engineer is a new custom persona — kyverno-json + JMESPath
|
||||
> is a specialized domain that doesn't fit backend-engineer's
|
||||
> fastify/hono frameworks or data-engineer's drizzle/postgresql.
|
||||
|
||||
## Active personas
|
||||
|
||||
### lead-developer
|
||||
- **Domain:** coordination + docs
|
||||
- **Frameworks:** []
|
||||
- **Constraints:** ["pragmatic", "battle-tested defaults", "docs match code"]
|
||||
- **Constraints:** ["pragmatic", "battle-tested defaults", "docs match code", "swap boundary is the moat"]
|
||||
- **Territory:**
|
||||
- `docs/consumer-guide.md`
|
||||
- `tests/test_consumer_guide_per_env_section.py`
|
||||
- `.ciagent/*.md` (PLAN.md, RESEARCH.md, etc.)
|
||||
- **Reason:** Owns the consumer guide narrative — the 5 accuracy fixes
|
||||
(stale fields table, inconsistent caller, "dev only" phrasing, Step 8
|
||||
rewrite with destroy semantics, reference table wording) and the
|
||||
consumer-guide test updates (rename + new destroy-on-env-change test).
|
||||
No UI work (frontend-engineer deactivated). No Python/bash platform
|
||||
code (backend-engineer territory).
|
||||
- `.ciagent/ARCHITECTURE.md` (§12.7 Policy Engine Registry — NEW)
|
||||
- `.ciagent/PROJECT.md` (v1.25 section)
|
||||
- `.ciagent/REQUIREMENTS.md` (v1.25 section)
|
||||
- `.ciagent/ROADMAP.md` (v1.25 section)
|
||||
- `.ciagent/PLAN.md`, `.ciagent/RESEARCH.md`, `.ciagent/CLARIFY.md`,
|
||||
`.ciagent/GRILL.md`, `.ciagent/PERSONAS.md`
|
||||
- `docs/METRICS.md` (swappable engine narrative — REQ-307)
|
||||
- **Reason:** Owns the milestone coordination + the architecture
|
||||
narrative. The swap boundary (PolicyEngine protocol) is the moat per
|
||||
Strategic Objective #2 — the lead-developer owns the boundary
|
||||
description in ARCHITECTURE.md §12.7 and the docs/METRICS.md note.
|
||||
No Python policy code (backend-engineer + policy-engineer territory).
|
||||
No UI (frontend-engineer deactivated).
|
||||
|
||||
### backend-engineer
|
||||
- **Domain:** backend (Python + bash + YAML)
|
||||
- **Domain:** backend (Python + bash + pipeline wiring)
|
||||
- **Frameworks:** ["boto3", "terraform"]
|
||||
- **Constraints:** ["api-first", "fail-closed", "no orphan paths", "state-key determinism"]
|
||||
- **Constraints:** ["api-first", "strict-typing", "engine-agnostic confidence signal", "fail-soft when kj absent"]
|
||||
- **Territory:**
|
||||
- `core/env_transition.py` (NEW)
|
||||
- `scripts/run_platform.sh` (Step 0b insert + record-applied-env)
|
||||
- `.github/workflows/deploy.yml` (NOVA_CONSUMER_REPO env)
|
||||
- `adapters/terraform/adapter.py` (doc comment only)
|
||||
- `tests/test_env_transition.py` (NEW)
|
||||
- `tests/test_run_platform_env_transition.py` (NEW)
|
||||
- **Reason:** Owns the env-transition detect-and-destroy feature: the new
|
||||
`core/env_transition.py` module (DynamoDB query + record), the
|
||||
`run_platform.sh` Step 0b orchestrator (re-resolve + terraform destroy +
|
||||
evidence event + fail-closed), the deploy.yml env var passthrough, and
|
||||
the two new test files. Uses boto3 (DynamoDB) + terraform (destroy) +
|
||||
bash (pipeline orchestration).
|
||||
- `core/policy_engine.py` (NEW — PolicyEngine Protocol + PolicyEngineRegistry + NullEngine)
|
||||
- `core/contract_resolver.py` (MODIFIED — invoke registry pre/post resolve)
|
||||
- `scripts/run_platform.sh` (MODIFIED — Step 5 kyverno-json parallel pass)
|
||||
- `scripts/install-kyverno-json.sh` (NEW)
|
||||
- `tests/test_policy_engine.py` (NEW — protocol conformance, registry, NullEngine)
|
||||
- `tests/test_run_platform_plan_json_policies.py` (NEW — script-substring assertion)
|
||||
- `.github/workflows/ci.yml` + `.gitea/workflows/ci.yml` (MODIFIED — Go + kj install)
|
||||
- **Reason:** Owns the Python protocol layer + the pipeline wiring. The
|
||||
`PolicyEngine` Protocol + `PolicyEngineRegistry` are Python structural-
|
||||
typing constructs (PEP 544) — backend-engineer's strict-typing
|
||||
constraint. The `contract_resolver.py` wiring + `run_platform.sh`
|
||||
Step 5 are backend territory. Does NOT write kyverno-json policy
|
||||
files (policy-engineer territory) — only the Python that *invokes* the
|
||||
engine. Does NOT modify the confidence signal (it already consumes
|
||||
`list[PolicyCheckResult]` engine-agnostically — PROJECT.md hard-
|
||||
constraint).
|
||||
|
||||
### policy-engineer
|
||||
- **Domain:** policy (declarative compliance rules)
|
||||
- **Frameworks:** ["kyverno-json", "jmespath", "kyverno ValidatingPolicy"]
|
||||
- **Constraints:** ["declarative-policies", "no-imperative-rules", "schema-validated", "severity-via-annotation", "assertion-trees-not-foreach"]
|
||||
- **Territory:**
|
||||
- `adapters/kyverno-json/` (NEW — engine impl + __init__.py + README)
|
||||
- `adapters/kyverno-json/kyverno_json_engine.py` (NEW — KyvernoJsonEngine)
|
||||
- `adapters/kyverno-json/policies/` (NEW — all 4 target dirs: contract/, stack-ir/, plan-json/, meta/, regression/)
|
||||
- `adapters/kyverno-json/policies/_smoke.json` (NEW)
|
||||
- `adapters/README.md` (MODIFIED — new adapter row + PolicyEngine Protocol section)
|
||||
- `tests/test_kyverno_json_engine.py` (NEW — PCR schema validity, defensive parsing)
|
||||
- `tests/test_stack_ir_policies.py` (NEW)
|
||||
- `tests/test_plan_json_policies.py` (NEW)
|
||||
- `tests/test_meta_policies.py` (NEW)
|
||||
- `tests/test_regression_policies.py` (NEW)
|
||||
- `tests/fixtures/stack_ir/`, `tests/fixtures/plan_json/`, `tests/fixtures/capability_inventory.json` (NEW)
|
||||
- `modules/STANDARDS.md` (MODIFIED — Policy authoring standard section — REQ-307)
|
||||
- **Reason:** The policy-engineer owns the declarative policy artifacts.
|
||||
kyverno-json's `ValidatingPolicy` + assertion trees + JMESPath is a
|
||||
distinct framework from backend-engineer's fastify/hono and requires
|
||||
its own constraints: no imperative rules (everything is an assertion
|
||||
tree), severity via the `nova.cloudinit.dev/severity` annotation (not
|
||||
in the engine adapter), no `forEach` (use the `~` modifier). The
|
||||
adapter pattern (engine ↔ protocol ↔ registry) is backend-engineer
|
||||
territory, but the policy *content* and the engine *translation*
|
||||
(`_to_pcr()`) are policy-engineer territory because they require
|
||||
kyverno-json output-shape knowledge. Created per RESEARCH.md §4 — this
|
||||
is a phase-spanning persona (active for P1..P4), not phase-specific.
|
||||
|
||||
### data-engineer
|
||||
- **Domain:** data (config schema + structured fixtures)
|
||||
- **Frameworks:** ["jsonschema", "yaml"]
|
||||
- **Constraints:** ["schema-first", "type-safe config", "backward-compatible additions"]
|
||||
- **Territory:**
|
||||
- `.ciagent/config.json` (MODIFIED — new `policy` object: engine + policy_root)
|
||||
- `schemas/policy_check_result.schema.json` (READ-ONLY — no change per D-116)
|
||||
- `schemas/README.md` (MODIFIED — note engine: "kyverno" shared by K8s adapter + kj)
|
||||
- `tests/fixtures/capability_inventory.json` (NEW — clean + drifted inventory fixtures for regression policies)
|
||||
- **Reason:** The `config.json.policy` object is a schema-first addition
|
||||
(new top-level key with `engine` + `policy_root` fields). The
|
||||
capability-inventory JSON fixtures for the regression-gate policies
|
||||
(REQ-304) are structured data — the data-engineer owns the fixture
|
||||
shape. The `policy_check_result.schema.json` is read-only (D-116 — no
|
||||
enum change); the data-engineer documents the `engine: "kyverno"`
|
||||
sharing in `schemas/README.md`. No migrations (no database). No Python
|
||||
(backend-engineer + policy-engineer territory).
|
||||
|
||||
## Deactivated personas
|
||||
|
||||
### frontend-engineer
|
||||
- **Active:** false
|
||||
- **Reason:** No UI work in v1.24. The consumer guide is markdown docs
|
||||
(lead-developer territory). Deactivated per v1.17/v1.18/v1.22/v1.23
|
||||
precedent.
|
||||
|
||||
### data-engineer
|
||||
- **Active:** false
|
||||
- **Reason:** No schema changes in v1.24. The `nova-contracts` DynamoDB
|
||||
table already exists with the right shape (PK `consumerRepo`, SK
|
||||
`contractId#submittedAt`). The env-transition module only adds a new
|
||||
`#LAST_APPLIED` SK suffix — no schema migration, no ORM, no new tables.
|
||||
The backend-engineer handles the boto3 queries.
|
||||
|
||||
## Phase-specific notes
|
||||
|
||||
- No phase-specific personas. Both active personas span P1-P3.
|
||||
- P4 (final review + audit + ship) is lead-developer territory
|
||||
(orchestration + docs completion).
|
||||
- **active:** false
|
||||
- **Reason:** ACDL has no frontend (no package.json — confirmed in
|
||||
config.json personas.personas[frontend-engineer].reason). v1.25 adds
|
||||
no UI work — the policy engine is backend + policy artifacts only.
|
||||
Deactivated per the v1.15+ convention.
|
||||
+402
-151
@@ -1,187 +1,438 @@
|
||||
# Nova — v1.24 Research Findings
|
||||
# Nova — v1.25 Research Findings
|
||||
|
||||
> Phase: research (pre-execution). Milestone: v1.24 (Consumer Guide Accuracy
|
||||
> & Env-Promotion Lifecycle Enforcement). Status: research.
|
||||
> Researcher: ci-researcher. Autonomy: full.
|
||||
> Phase: research (pre-execution). Milestone: v1.25 (kyverno-json Unified
|
||||
> Policy Engine). Status: research. Researcher: ci-researcher.
|
||||
> Autonomy: full.
|
||||
|
||||
## 1. Problem domain
|
||||
|
||||
Two distinct problem spaces in one milestone:
|
||||
Nova's compliance/policy posture is fragmented across three engines with
|
||||
three rule languages and three adapter shapes (see PROJECT.md v1.25
|
||||
"Why" for the full diagnosis). The `PolicyCheckResult` schema
|
||||
(`schemas/policy_check_result.schema.json`) is already the engine-agnostic
|
||||
contract that `core/confidence_signal.py` consumes — the *contract* is
|
||||
right; the *orchestration* is fragmented. There is no single declarative
|
||||
place where "what Nova considers compliant" lives. The K8s-only Kyverno
|
||||
adapter (`adapters/kyverno/`) can't help because it only speaks to K8s
|
||||
manifests and the platform emits Terraform (D-053).
|
||||
|
||||
### 1a. Consumer guide accuracy (docs)
|
||||
`kyverno-json` is the correction: a Kyverno-ecosystem runtime that applies
|
||||
Kyverno policies to **any** JSON/YAML payload. It becomes the **unified
|
||||
orchestrator** of compliance checks, behind a swappable `PolicyEngine`
|
||||
protocol so OPA can replace it one day. Checkov and Wiz remain as
|
||||
raw-finding adapters feeding *into* kyverno-json meta-policies.
|
||||
|
||||
The consumer guide (`docs/consumer-guide.md`, 477 lines) has 5 accuracy
|
||||
defects identified in review:
|
||||
## 2. kyverno-json — the engine surface
|
||||
|
||||
1. **Step 3 contract fields table (lines 141-147)** lists `uses`, `module`,
|
||||
`environment`, `inputs`. The actual schema
|
||||
(`schemas/contract.schema.json:7`) requires `id`, `name`, `environment`,
|
||||
`infrastructure`. The `uses` field was dropped in v1.10.2 (REQ-50
|
||||
superseded) and `module` was replaced by the `infrastructure` map key.
|
||||
The worked examples (lines 111-137) use the correct fields.
|
||||
### 2.1 What it is
|
||||
|
||||
2. **Step 4 caller (lines 173-183)** omits `environment:` in `with:`, while
|
||||
Step 2 (lines 94-101) shows `environment: dev`. The two canonical caller
|
||||
snippets disagree.
|
||||
[kyverno-json](https://github.com/kyverno/kyverno-json) is a standalone Go
|
||||
binary from the Kyverno project. It is a **separate runtime** from the
|
||||
Kyverno K8s admission controller — same policy lineage, different
|
||||
application target. Where Kyverno (K8s) evaluates `ClusterPolicy`
|
||||
resources against Kubernetes manifests at admission time, kyverno-json
|
||||
evaluates `ValidatingPolicy` resources against **any** JSON or YAML
|
||||
payload file via the CLI (`kj scan`) or a Go library. It is **not** a
|
||||
Python package (no PyPI release); it is installed via
|
||||
`go install github.com/kyverno/kyverno-json/cmd/kj@latest` (D-115) or by
|
||||
downloading a pinned binary from GitHub releases.
|
||||
|
||||
3. **Step 5 stage 8 (lines 232, 253)** says "(dev only)" for the apply
|
||||
stage. Higher environments *do* apply — they apply after HITL
|
||||
attestation per `docs/environments/index.md:44-54`.
|
||||
### 2.2 CLI surface (the v1.25 invocation path)
|
||||
|
||||
4. **Step 8 (lines 290-306)** says "Change `environment` in your contract"
|
||||
to promote, which contradicts the same doc's "Per-environment
|
||||
deployment" section (lines 398-402): "you do not edit the `environment:`
|
||||
field… Promotion = running the matching job." The test
|
||||
`test_consumer_guide_states_no_field_editing` asserts the no-editing
|
||||
model.
|
||||
The v1.25 engine uses the `kj scan` subcommand:
|
||||
|
||||
5. **Reference table (lines 329-330)** says sample contracts "use `@v1.19`"
|
||||
but the sample contracts (`contracts/static-assets.yml`,
|
||||
`contracts/microservice.yml`) don't carry `uses:` — the version pin
|
||||
lives in the caller workflow.
|
||||
```
|
||||
kyverno-json scan [flags]
|
||||
|
||||
### 1b. Environment-promotion lifecycle enforcement (feat)
|
||||
Flags:
|
||||
--labels strings Labels selectors for policies
|
||||
--output string Output format (text or json) (default "text")
|
||||
--payload string Path to payload (json or yaml file)
|
||||
--policy strings Path to kyverno-json policies
|
||||
--pre-process strings JMESPath expression used to pre process payload
|
||||
```
|
||||
|
||||
**Root cause confirmed by code inspection:**
|
||||
The `KyvernoJsonEngine.evaluate()` implementation (REQ-293) invokes:
|
||||
```
|
||||
kj scan --policy <policy_dir> --payload <payload.json> --output json
|
||||
```
|
||||
and parses the JSON `results[]` array. The `--pre-process` flag is
|
||||
available for JMESPath pre-projection (noted for the meta-policy use case
|
||||
where the payload is the merged PCR list and a pre-process expression
|
||||
can index by `ruleId` — recorded as a future optimization, not used in
|
||||
v1.25's initial implementation).
|
||||
|
||||
- `adapters/terraform/adapter.py:129` sets the Terraform state key to:
|
||||
`spike/{stack_name}/{environment}/terraform.tfstate`
|
||||
- `stack_name` = `contract["id"]` (stable across env changes, per
|
||||
`core/contract_resolver.py:584`).
|
||||
- When a consumer edits `environment:` from `dev` → `qa` on the same
|
||||
contract `id`, the state key changes from `spike/assets/dev/` to
|
||||
`spike/assets/qa/`. Terraform initializes a **fresh state file** in the
|
||||
new env's state path. The prior env's resources remain live in AWS with
|
||||
their state file untouched. **No destroy ever runs.** This orphans
|
||||
resources.
|
||||
Other subcommands (`kj jp`, `kj serve`, `kj playground`, `kj docs`) are
|
||||
out of scope for v1.25. `kj serve` is the long-running web-app mode
|
||||
(noted as a future consideration for lower-latency evaluation in the
|
||||
Out of Scope section of REQUIREMENTS.md). `kj jp` is the JMESPath REPL —
|
||||
useful for policy authoring/debugging, not invoked by the engine.
|
||||
|
||||
**The user's binding directive:** Editing `environment:` on a stable
|
||||
`contract.id` is a valid promotion path (Shape A). The platform **must**
|
||||
destroy the prior env's resources before building the new env. There must
|
||||
be **no path that orphans resources** — fail closed if the destroy fails.
|
||||
### 2.3 Policy structure (the `ValidatingPolicy` resource)
|
||||
|
||||
## 2. Existing codebase structure (integration points)
|
||||
kyverno-json policies are Kubernetes-style resources (cluster-scoped)
|
||||
belonging to the `json.kyverno.io` API group, kind `ValidatingPolicy`,
|
||||
version `v1alpha1`:
|
||||
|
||||
### DynamoDB `nova-contracts` table (the prior-env source of truth)
|
||||
```yaml
|
||||
apiVersion: json.kyverno.io/v1alpha1
|
||||
kind: ValidatingPolicy
|
||||
metadata:
|
||||
name: <policy-name> # becomes the KJ_<policy-name> ruleId prefix
|
||||
spec:
|
||||
rules:
|
||||
- name: <rule-name>
|
||||
identifier: <jmespath> # optional — path to the unique entry id
|
||||
match: # assertion tree — which payload entries
|
||||
any: # the rule applies to
|
||||
- <assertion>
|
||||
exclude: # optional — exclude matching entries
|
||||
any:
|
||||
- <assertion>
|
||||
context: # optional — named bindings available to
|
||||
- name: <binding> # the rule's assertions ($<binding>)
|
||||
variable: <value>
|
||||
validate:
|
||||
message: "<human-readable>" # optional per-rule message
|
||||
assert:
|
||||
all: # all assertions must hold
|
||||
- check: <assertion-tree>
|
||||
message: "<per-check>"
|
||||
# OR
|
||||
any: # at least one assertion must hold
|
||||
- check: <assertion-tree>
|
||||
```
|
||||
|
||||
- **Table:** `nova-contracts` (env var `CONTRACTS_TABLE`, default
|
||||
`nova-contracts`). Defined in `core/lambda/contract_ingestor.py:25`.
|
||||
- **Schema:** PK `consumerRepo` (S), SK `contractId#submittedAt` (S).
|
||||
Attributes: `contractId`, `contract`, `environment`, `status`,
|
||||
`submittedAt`.
|
||||
- **Written by:** `_submit_contract()` at
|
||||
`core/lambda/contract_ingestor.py:135-176`. The consumer's deploy
|
||||
workflow submits the contract via the Lambda Function URL.
|
||||
- **Read pattern for env-transition:** Query by PK `consumerRepo` + SK
|
||||
begins_with `contractId#` + FilterExpression `status = "submitted"` →
|
||||
sort by `submittedAt` desc → take the latest → read its `environment`.
|
||||
This is the last-submitted env. For the last-*applied* env, a new
|
||||
`#LAST_APPLIED` SK suffix is added (REQ-283).
|
||||
- **Test pattern:** `tests/test_contract_ingestor.py:74-110`
|
||||
(`moto_contracts_table` fixture) uses moto `mock_aws` to create the
|
||||
table. The env-transition tests will mirror this pattern.
|
||||
Key differences from K8s Kyverno policies:
|
||||
- **Always cluster-scoped** — no `namespace` field.
|
||||
- **No `forEach`, pattern operators, anchors, or wildcards.** Iteration
|
||||
is done via the `~` projection modifier in assertion trees (see §2.4).
|
||||
- **Assertion trees** with JMESPath expressions replace Kyverno's
|
||||
pattern-matching syntax (see §2.4).
|
||||
|
||||
### Outbox writer (evidence events)
|
||||
### 2.4 Assertion trees (the rule language)
|
||||
|
||||
- `core/outbox_writer.py` writes hash-chained evidence events to
|
||||
`nova-outbox` table. PK `contractId`, SK `eventType#eventTs`.
|
||||
- The env-transition destroy step emits a `nova.env.destroyed` event via
|
||||
this writer (REQ-284c). Pattern: build an event dict with `contractId`,
|
||||
`eventType: "ENV_DESTROYED"`, `environment: <prior_env>`, `ts`, then
|
||||
call `write_event()`.
|
||||
An `assert` declaration contains an `all` or `any` list. Each entry has a
|
||||
`check` (the assertion tree — a nested JMESPath projection) and an
|
||||
optional `message`. **All comparisons happen in the leaves of the tree.**
|
||||
|
||||
### Contract resolver (environment override)
|
||||
A simple example (assert a pod doesn't use the default service account):
|
||||
```yaml
|
||||
validate:
|
||||
assert:
|
||||
all:
|
||||
- message: "serviceAccountName 'default' is not allowed"
|
||||
check:
|
||||
spec:
|
||||
(serviceAccountName == 'default'): false
|
||||
```
|
||||
|
||||
- `core/contract_resolver.py:460-483` `resolve()` accepts
|
||||
`environment_override` — when set, it overrides the contract's
|
||||
`environment` field **before** schema validation and interpolation
|
||||
(D-088). This is the mechanism the destroy step uses to re-resolve the
|
||||
contract against the prior env: `resolve(contract, env_override=prior_env)`.
|
||||
- Already tested in `tests/test_deploy_workflow_env_input.py:35-52`.
|
||||
The `(expression)` syntax evaluates a JMESPath expression; the result
|
||||
becomes the current object for descendants; the leaf value is compared
|
||||
to the expected value.
|
||||
|
||||
### run_platform.sh (where Step 0b goes)
|
||||
**Iteration via the `~` modifier.** The `~` prefix on a key applies
|
||||
descendant assertions to **each element** of an array/map individually
|
||||
(rather than comparing the whole array). Given `foo.bar: [1,2,3]`:
|
||||
```yaml
|
||||
check:
|
||||
foo:
|
||||
~.bar: # iterate each element
|
||||
(@ < `5`): true # assert each element < 5
|
||||
```
|
||||
The `~index_name.bar` form binds the index (array) or key (map) to
|
||||
`$index_name` for use in descendants. This is how v1.25 iterates
|
||||
`resources[]` in the Stack IR policies (REQ-297) and
|
||||
`planned_values.root_module.resources[]` in the plan-JSON policies
|
||||
(REQ-300).
|
||||
|
||||
- `scripts/run_platform.sh` is the platform pipeline. Step 0 (lines 222-239)
|
||||
is the environment onboarding check. Step 1 (lines 241-249) is contract
|
||||
validation. **Step 0b goes between them** (after onboarding, before
|
||||
validation).
|
||||
- The destroy step mirrors the existing `--destroy` mode (lines 376-394):
|
||||
`terraform init -reconfigure` + `terraform destroy -auto-approve`. The
|
||||
difference: it runs against the *prior* env's state key, not the current
|
||||
one.
|
||||
- `CONTRACT_ID` is already set at line 217 (`NOVA_CONTRACT_ID` with a
|
||||
default). `CONSUMER_REPO` needs to be derived from `GITHUB_REPOSITORY`
|
||||
or a new `NOVA_CONSUMER_REPO` env var (REQ-286).
|
||||
**Explicit bindings** via `->binding_name` allow descendants to refer
|
||||
to a parent node via `$binding_name`. Built-in bindings: `$payload`
|
||||
(the whole input), `$policy`, `$rule`.
|
||||
|
||||
### deploy.yml (consumer repo → platform)
|
||||
**Escaping** via `\key\` prevents projection when a payload key collides
|
||||
with the projection syntax. Not needed for Nova payloads (no `(key)`
|
||||
fields), noted for completeness.
|
||||
|
||||
- `.github/workflows/deploy.yml:132` runs
|
||||
`bash platform/scripts/run_platform.sh $MODE_FLAG $ENV_FLAG "${{ inputs.contract }}"`.
|
||||
- To pass `NOVA_CONSUMER_REPO`, add `NOVA_CONSUMER_REPO=${{ github.repository }}`
|
||||
as an env var on the "Run the platform pipeline" step (REQ-286).
|
||||
### 2.5 Output shape (what `kj scan --output json` produces)
|
||||
|
||||
### Test patterns
|
||||
The JSON output is a `results[]` array. Each result entry has (at
|
||||
minimum):
|
||||
- `policy`: the policy metadata.name
|
||||
- `rule`: the rule name
|
||||
- `result`: `"pass"` | `"fail"` | `"error"` | `"skip"` (lowercase)
|
||||
- `message`: the assertion message (or engine error message)
|
||||
- `resource`: the matched payload entry (the `identifier` value, or the
|
||||
whole payload when no identifier/match)
|
||||
- `namespace`/`kind`/`name`: K8s-style fields (present but empty for
|
||||
non-K8s payloads — the K8s Kyverno adapter's evidence uses these; the
|
||||
kyverno-json engine's evidence uses `assertion`/`jmespath` instead)
|
||||
- `severity`: not present by default (kyverno-json does not assign
|
||||
severities — the Nova policy author assigns severity via a Nova-
|
||||
specific annotation; see §2.6)
|
||||
|
||||
- **Shell script assertions:** `tests/test_pipeline.py:79-95` reads the
|
||||
script text and asserts substrings. The env-transition tests follow this
|
||||
pattern for `run_platform.sh`.
|
||||
- **DynamoDB mocking:** `tests/test_contract_ingestor.py:74-110` uses moto
|
||||
`mock_aws` + `boto3.client` + `create_table`. The env-transition tests
|
||||
follow this pattern.
|
||||
- **Consumer guide assertions:** `tests/test_consumer_guide_per_env_section.py`
|
||||
reads `docs/consumer-guide.md` text and asserts substrings. The updated
|
||||
tests follow this pattern.
|
||||
The `KyvernoJsonEngine._to_pcr()` translator (REQ-293) maps:
|
||||
- `policy` → `ruleId` (prefixed `KJ_<policy_name>` per D-116)
|
||||
- `result` → `result` (`pass`/`fail`/`error` → pass/fail/error;
|
||||
`skip`/`skipped` → skipped)
|
||||
- `message` → `message`
|
||||
- `resource` → `resourceRef` + `evidence.resource`
|
||||
- severity from the policy's `metadata.annotations` (see §2.6)
|
||||
- `engine: "kyverno"` (per D-116 — no new enum value)
|
||||
|
||||
## 3. Persona assessment (v1.24)
|
||||
### 2.6 Severity assignment (Nova convention)
|
||||
|
||||
This milestone has two distinct work territories:
|
||||
kyverno-json does not natively assign severities to results. Nova's
|
||||
confidence signal requires a `severity` per PCR (critical/high/medium/
|
||||
low/info). The v1.25 convention: each Nova policy file declares its
|
||||
severity via a `metadata.annotations` field:
|
||||
|
||||
1. **Docs (P1):** `docs/consumer-guide.md` edits + test updates. This is
|
||||
lead-developer territory (narrative/docs + test assertions).
|
||||
2. **Platform code (P2):** `core/env_transition.py` (new), `scripts/run_platform.sh`
|
||||
edits, `.github/workflows/deploy.yml` edit, `adapters/terraform/adapter.py`
|
||||
doc comment. This is backend-engineer territory (Python + bash + YAML).
|
||||
3. **Tests (P3):** `tests/test_env_transition.py` (new), `tests/test_run_platform_env_transition.py`
|
||||
(new), `tests/test_consumer_guide_per_env_section.py` updates. Split:
|
||||
backend-engineer for the env_transition + pipeline tests; lead-developer
|
||||
for the consumer guide test updates.
|
||||
```yaml
|
||||
metadata:
|
||||
name: forbid-public-ingress
|
||||
annotations:
|
||||
nova.cloudinit.dev/severity: high
|
||||
```
|
||||
|
||||
**Roster:** lead-developer (docs + guide tests) + backend-engineer (Python +
|
||||
bash + YAML + pipeline tests). frontend-engineer stays deactivated (no UI).
|
||||
data-engineer not needed (no schema changes — the `nova-contracts` table
|
||||
already exists with the right shape; we only add a new SK suffix). No new
|
||||
personas.
|
||||
The `KyvernoJsonEngine._to_pcr()` reads this annotation from the loaded
|
||||
policy YAML (not from the scan result — the result doesn't carry it) and
|
||||
applies it to every result that policy produces. Default when absent:
|
||||
`info`. This keeps severity in the policy (declarative, version-
|
||||
controlled) rather than in the engine adapter (imperative). The
|
||||
annotation key is `nova.cloudinit.dev/severity` (matches the existing
|
||||
`nova.cloudinit.dev` namespace used in `schemas/tagging-standard.json`).
|
||||
|
||||
## 4. Key decisions logged
|
||||
## 3. The four policy targets (v1.25 scope)
|
||||
|
||||
| ID | Decision | Confidence | Source |
|
||||
|----|----------|------------|--------|
|
||||
| D-201 | Both promotion shapes supported (A: edit+destroy, B: per-env callers) | 0.95 | User directive |
|
||||
| D-202 | Prior-env source of truth = `nova-contracts` DynamoDB table | 0.85 | User directive + code inspection |
|
||||
| D-203 | Detect-and-destroy at pipeline start (Step 0b) | 0.85 | User directive |
|
||||
| D-204 | Fail closed on destroy failure (no orphan path) | 0.95 | User directive |
|
||||
| D-205 | Cross-account destroy out of scope (same-account only) | 0.80 | CLARIFY A3 |
|
||||
| D-206 | Env-transition destroy is NOT the HITL decommission pipeline | 0.85 | CLARIFY A5 |
|
||||
| D-207 | Last-applied env recorded via `#LAST_APPLIED` SK in `nova-contracts` | 0.85 | RESEARCH §2 |
|
||||
| D-208 | State key `spike/{id}/{env}/` stays as-is (correct for both shapes) | 0.90 | RESEARCH §1b |
|
||||
### 3.1 Consumer contract JSON (REQ-295)
|
||||
|
||||
## 5. Pitfalls
|
||||
The payload is the parsed contract dict (the raw YAML loaded as JSON).
|
||||
Policies assert the `contract.schema.json` constraints declaratively:
|
||||
`require-id-pattern` (JMESPath regex `^[a-z][a-z0-9-]{2,5}$` over
|
||||
`id`), `require-env-in-enum` (`environment` in `["dev","qa","prod","dr"]`),
|
||||
`require-infrastructure-min-1` (`length(infrastructure) > 0`),
|
||||
`forbid-unknown-fields` (keys subset of the 4 allowed). These are the
|
||||
declarative equivalent of the jsonschema constraints — they let Nova
|
||||
apply its own compliance posture (e.g. forbid a specific env for a
|
||||
specific consumer) on top of schema validity without editing the
|
||||
jsonschema.
|
||||
|
||||
1. **Destroy needs the prior env's Terraform config, not the new env's.**
|
||||
The destroy step must re-resolve the contract with
|
||||
`environment_override=prior_env` so the emitted TF matches the prior
|
||||
env's resources. If we resolve with the new env, the destroy plan won't
|
||||
match the prior state → terraform tries to create, not destroy.
|
||||
2. **`terraform init -reconfigure` is required** when switching state
|
||||
backends between envs (if envs use different state buckets). The
|
||||
`-reconfigure` flag tells Terraform to forget the previous backend config.
|
||||
3. **The `deletion_protection` NFR (REQ-86) blocks destroy.** The destroy
|
||||
step must resolve with `deletion_protection: false` injected (same as
|
||||
decommission Step 2 in `scripts/run_decommission.sh:34-37`). Without
|
||||
this, `terraform destroy` fails on `prevent_destroy` lifecycle blocks.
|
||||
4. **DynamoDB may not be reachable in local/CI mode.** The detect step
|
||||
must handle `ClientError` / `EndpointNotFound` gracefully → log warning
|
||||
+ return `None` (conservative). This is documented in REQ-282.
|
||||
5. **The consumer guide test `test_consumer_guide_states_no_field_editing`
|
||||
will fail after the Step 8 rewrite.** It must be updated in the same
|
||||
phase as the guide edit (P1) or the test suite breaks.
|
||||
**Invocation point:** `core/contract_resolver.py` pre-resolve (REQ-296).
|
||||
Early-fail: if a contract policy fails, the resolver still proceeds
|
||||
(the confidence signal decides the gate, consistent with the existing
|
||||
`--soft-fail` Checkov pattern) — but the failing PCRs are in the
|
||||
`policy` input, which lowers the score.
|
||||
|
||||
### 3.2 Resolved Target Stack IR JSON (REQ-297)
|
||||
|
||||
The payload is the resolved Stack IR dict produced by
|
||||
`core/contract_resolver.py` (the merged module outputs). Policies
|
||||
assert over `resources[]` (the array of resolved resources):
|
||||
`require-tagging-standard` (every resource's `tags` has `nova:owner` +
|
||||
`nova:environment` — ports
|
||||
`adapters/terraform/policy/custom_rules/nova_tagging.py`),
|
||||
`forbid-public-ingress` (no resource has `public_ingress: true` — the
|
||||
v1.0 demo rule, now declarative), `require-encryption-by-default` (every
|
||||
S3/EBS/KMS-aliased resource carries encryption config — ports the v1.8
|
||||
D-encryption-default rule). The `~` modifier iterates `resources[]`.
|
||||
|
||||
**Invocation point:** `core/contract_resolver.py` post-resolve (REQ-298).
|
||||
Additive — the resolver's return values and exceptions are unchanged;
|
||||
the PCRs are appended to the contract-policy PCRs.
|
||||
|
||||
### 3.3 Terraform plan JSON (REQ-300)
|
||||
|
||||
The payload is `terraform show -json <tfplan>` output. Policies assert
|
||||
over `planned_values.root_module.resources[]`:
|
||||
`forbid-plaintext-secrets` (no `aws_db_instance.password` /
|
||||
`aws_iam_user.login_profile.password` in plaintext — ports
|
||||
`CKV_AWS_41/45/46`), `forbid-iam-wildcard` (no `Action: "*"` or
|
||||
`Resource: "*"` in `aws_iam_policy.PolicyDocument` — ports
|
||||
`CKV_AWS_1/40`), `require-kms-reference` (KMS keys referenced by alias,
|
||||
not inline key material — ports `CKV_AWS_7/33`). These are declarative
|
||||
**mirrors** of `checkov_adapter.py:RULE_MAP` — the Checkov rule stays
|
||||
the source of truth for `terraform_plan` scanning; the kyverno-json
|
||||
policy covers the same plan JSON with a different rule language
|
||||
(defense-in-depth against engine drift).
|
||||
|
||||
**Invocation point:** `run_platform.sh` Step 5 (REQ-301). After
|
||||
Checkov/Wiz produce raw PCRs, the script runs `kj scan` over the plan
|
||||
JSON; both PCR lists concatenate into the confidence signal's `policy`
|
||||
input. When `which kj` is false, the script logs and proceeds with the
|
||||
Checkov/Wiz list only.
|
||||
|
||||
### 3.4 PolicyCheckResult records (meta-policies, REQ-303)
|
||||
|
||||
The payload is the **merged** `list[PolicyCheckResult]` produced by
|
||||
checkov + wiz + the plan-JSON policies. This is the most novel target —
|
||||
kyverno-json policies over the policy results themselves.
|
||||
`block-on-any-critical` asserts no PCR has `severity: "critical"` +
|
||||
`result: "fail"`; if any does, the meta-policy emits a `fail` PCR with
|
||||
`ruleId: "KJ_META_BLOCK_CRITICAL"` and severity `critical`. This is the
|
||||
declarative source of truth for "critical = block" (D-119 — the
|
||||
`confidence_signal.py` `PENALTY["critical"]: None` hard-override stays
|
||||
as defense-in-depth). `tagging-rules-agree` cross-checks the Checkov
|
||||
`NOVA_TAG_NAMING` result against the kyverno-json
|
||||
`KJ_REQUIRE_TAGGING_STANDARD` result by `resourceRef`; divergence emits
|
||||
an `error` PCR (D-118).
|
||||
|
||||
**Invocation point:** after the three target policies (contract/stack-
|
||||
IR/plan-JSON) produce their PCR lists, the merged list is the payload
|
||||
for the meta-policies. The meta-policy PCRs are appended to the merged
|
||||
list, which is what the confidence signal consumes.
|
||||
|
||||
## 4. The `PolicyEngine` swap boundary
|
||||
|
||||
### 4.1 Protocol shape (REQ-291)
|
||||
|
||||
A Python `Protocol` (PEP 544 — structural subtyping, no inheritance):
|
||||
```python
|
||||
class PolicyEngine(Protocol):
|
||||
@property
|
||||
def name(self) -> str: ...
|
||||
def is_configured(self) -> bool: ...
|
||||
def evaluate(self, payload: dict | str, policy_dir: Path,
|
||||
contract_id: str) -> list[dict]: ...
|
||||
```
|
||||
`list[dict]` (not `list[PolicyCheckResult]` — there's no dataclass; the
|
||||
schema is enforced via `jsonschema` validation in tests, matching the
|
||||
existing adapter pattern). The registry selects the active engine from
|
||||
`config.json.policy.engine`. A `NullEngine` is the fallback when the
|
||||
`policy` key is absent (emits `SKIPPED` — backward compatibility for
|
||||
tests that don't set the key).
|
||||
|
||||
### 4.2 The OPA-equivalent surface (future swap)
|
||||
|
||||
OPA (Open Policy Agent) is the most likely future replacement. The
|
||||
mapping:
|
||||
| Nova `PolicyEngine` member | kyverno-json impl | OPA equivalent |
|
||||
|---|---|---|
|
||||
| `name` | `"kyverno-json"` | `"opa"` |
|
||||
| `is_configured()` | `which kj` | `which opa` |
|
||||
| `evaluate(payload, policy_dir, contract_id)` | `kj scan --policy <dir> --payload <json> -o json` | `opa eval -d <dir> -i <json> 'data.nova.<...>'` |
|
||||
| Policy file format | `ValidatingPolicy` (YAML) | Rego (`.rego`) |
|
||||
| Result shape | `results[]` (pass/fail/error/skip) | `result` (set of violations) |
|
||||
| Severity | Nova annotation `nova.cloudinit.dev/severity` | Nova convention (Rego `metadata` or a wrapper) |
|
||||
|
||||
The protocol is minimal (3 members) specifically so the OPA
|
||||
implementation is a known quantity: an `OpaEngine` class that shells to
|
||||
`opa eval`, translates the Rego violation set to PCR dicts, and
|
||||
implements `is_configured()` via `which opa`. The policy *files* would
|
||||
need rewriting (Rego, not ValidatingPolicy) — but the protocol, the
|
||||
registry, the confidence signal, and the PCR schema are all untouched.
|
||||
This is the swap boundary the user asked for ("Implemented as an
|
||||
adapter since we might one day decide to replace it with something else
|
||||
like OPA").
|
||||
|
||||
### 4.3 Why not a full plugin registry?
|
||||
|
||||
A `setuptools` entry-point plugin registry (like checkov's
|
||||
`--external-checks-dir`) was considered and rejected: Nova has 1 active
|
||||
engine today (kyverno-json) and at most 2 in the foreseeable future
|
||||
(kyverno-json + OPA). A `Protocol` + `dict` registry in
|
||||
`core/policy_engine.py` is the right weight — discoverable, typed,
|
||||
testable, and ~40 lines. An entry-point registry adds packaging
|
||||
complexity (entry-point metadata, version resolution) for no gain at
|
||||
this scale. The `register(name, factory)` method on the registry is
|
||||
the extension point if a future milestone needs runtime plugin
|
||||
discovery.
|
||||
|
||||
## 5. Latency / MTTR impact (G-Q3 anticipation)
|
||||
|
||||
NORTH_STAR.md MTTR target: < 60s p95. `run_platform.sh` Step 5 today
|
||||
runs Checkov over the terraform plan (typically 5-15s for a small
|
||||
stack). Adding `kj scan` over the same plan JSON adds:
|
||||
- Process spawn: ~50ms (Go binary startup)
|
||||
- Policy load: ~20ms (a handful of YAML files)
|
||||
- Assertion evaluation: ~100-500ms (JMESPath over a small plan)
|
||||
- Total: < 1s for a typical Nova stack
|
||||
|
||||
The kyverno-json pass runs **in parallel** with Checkov (REQ-301 — the
|
||||
script launches both and waits on both), so the wall-clock impact is
|
||||
`max(checkov_time, kj_time)` ≈ checkov_time (kj is faster). The
|
||||
contract + stack-IR policies run during resolve (already a fast step).
|
||||
Meta-policies run over the merged list (in-memory, < 10ms). **No
|
||||
measurable MTTR impact** is expected. This will be verified in P3
|
||||
VERIFY with a timing assertion.
|
||||
|
||||
## 6. "Platform functions without AI" tenet (G-Q1 / D-120)
|
||||
|
||||
kyverno-json is deterministic (same policy + payload → same result,
|
||||
every run). It is not an LLM, not a probabilistic model, not a
|
||||
"judgement" engine. The NORTH_STAR.md tenet ("the platform functions
|
||||
without AI — 'AI decisions' are really automated decisions")
|
||||
distinguishes AI (non-reproducible) from automation (reproducible).
|
||||
kyverno-json is the latter. Adding it is **more** aligned with the
|
||||
tenet than the current imperative Python in `core/env_transition.py`
|
||||
and `core/regression_verify.py`, because the policy is declarative
|
||||
(visible, auditable, version-controlled) rather than imperative (logic
|
||||
hidden in function bodies). The `is_configured()` guard ensures the
|
||||
platform functions without the binary (graceful skip → `SKIPPED` PCR
|
||||
→ confidence signal proceeds).
|
||||
|
||||
## 7. ECS policy catalog overlap (prior art)
|
||||
|
||||
The kyverno-json catalog ships ECS policies that overlap with Nova's
|
||||
L1 modules: `ecs-cluster-enable-logging`, `ecs-cluster-required-
|
||||
container-insights`, `ecs-service-public-ip`, `ecs-service-required-
|
||||
latest-platform-fargate`, `ecs-task-definition-fs-read-only`. These are
|
||||
**reference policies**, not drop-in Nova policies — they target the
|
||||
AWS ECS API shape (`type: aws_ecs_service` etc.), not Nova's Stack IR
|
||||
shape. v1.25 policies target the Nova IR (REQ-297) and the terraform
|
||||
plan JSON (REQ-300), not the raw AWS API. The catalog is useful as
|
||||
prior art for JMESPath patterns over ECS resources — the
|
||||
`ecs-service-public-ip` policy's `contains('$allowed-values',
|
||||
@.assign_public_ip)` pattern informs the Nova `forbid-public-ingress`
|
||||
policy shape. No catalog policies are imported directly in v1.25.
|
||||
|
||||
## 8. Risks & mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|---|---|---|---|
|
||||
| `kj` binary not in CI image | medium | blocks P3+ tests | `is_configured()` guard + `pytest.skip` + `scripts/install-kyverno-json.sh` |
|
||||
| kyverno-json output shape changes across versions | low | breaks `_to_pcr()` | pin `@latest` to a known-good commit in `install-kyverno-json.sh` after P1 smoke; defensive parsing (malformed → `error` PCR, not exception) |
|
||||
| Policy explosion (4 targets × N rules) | medium | maintenance load | wave ordering (PLAN); policies co-located per target dir; meta-policy cross-check keeps the set auditable |
|
||||
| Checkov + kj tagging-rule drift | medium | false `error` PCRs | `tagging-rules-agree` meta-policy emits `error` on divergence (visible, not silent); the Checkov rule stays source of truth for HCL, kj for IR |
|
||||
| OPA swap turns out harder than the protocol implies | low | future milestone rework | RESEARCH §4.2 documents the OPA-equivalent surface; the protocol is the contract, not the implementation |
|
||||
| `--pre-process` needed for meta-policies but undocumented behavior | low | meta-policy bugs | v1.25 meta-policies use plain assertion trees over the PCR list (no pre-process); `--pre-process` noted as a future optimization only |
|
||||
|
||||
## 9. Assumptions (logged, full autonomy)
|
||||
|
||||
- A1: `kj scan --output json` produces a stable `results[]` array shape.
|
||||
Will be verified in P1 smoke test (`_smoke.json` policy + a trivial
|
||||
payload); if the shape differs, `_to_pcr()` is adjusted defensively
|
||||
(malformed → `error` PCR). Confidence: 0.85.
|
||||
- A2: The `nova.cloudinit.dev/severity` annotation convention is
|
||||
read by the engine from the policy YAML (loaded once per evaluate()
|
||||
call). kyverno-json does not validate unknown annotations — they pass
|
||||
through. Confidence: 0.90.
|
||||
- A3: The `~` projection modifier iterates `resources[]` in the Stack
|
||||
IR and `planned_values.root_module.resources[]` in the plan JSON
|
||||
correctly. Verified in P2/P3 tests. Confidence: 0.85.
|
||||
- A4: `go install` works in the CI image (Go toolchain available or
|
||||
installable). If not, the binary-release download path is the
|
||||
documented fallback in `install-kyverno-json.sh`. Confidence: 0.80.
|
||||
- A5: The `NullEngine` fallback (when `policy` key absent in
|
||||
config.json) keeps all existing tests passing — they don't set the
|
||||
key, so they get `NullEngine` → `SKIPPED` PCRs → confidence signal
|
||||
proceeds with `policy` input `[SKIPPED]` → per-input score 1.0
|
||||
(skipped counts as pass in `_per_input_score`). Confidence: 0.95
|
||||
(verified against `confidence_signal.py:84-89`).
|
||||
|
||||
## 10. Decisions referenced
|
||||
|
||||
D-115 (install path), D-116 (engine enum reuse), D-117 (adapter
|
||||
signatures unchanged), D-118 (tagging cross-check), D-119 (critical-
|
||||
override defense-in-depth), D-120 (deterministic not AI). See
|
||||
CLARIFY.md for the full resolution text.
|
||||
|
||||
## 11. Architecture updates (deferred to RESEARCH-stage file edits)
|
||||
|
||||
- `.ciagent/ARCHITECTURE.md` gains §12.7 "Policy Engine Registry" with
|
||||
the registry diagram. Deferred to the RESEARCH commit (this file's
|
||||
commit) — the section is authored as part of this research.
|
||||
- `schemas/README.md` notes `engine: "kyverno"` is shared by the K8s
|
||||
adapter and kyverno-json (distinguished by `ruleId` prefix).
|
||||
- `modules/STANDARDS.md` gains a "Policy authoring standard" section
|
||||
(P4, REQ-307).
|
||||
- `docs/METRICS.md` notes the policy engine is swappable (P4, REQ-307).
|
||||
Reference in New Issue
Block a user