acdl/.ciagent/PLAN.md

PLAN — v1.25 (kyverno-json Unified Policy Engine)

Feature milestone. Tags on the v1.24.x line: v1.24.0 (P0) → v1.24.1 (P1) → v1.24.2 (P2) → v1.24.3 (P3) → v1.24.4 (P4) → v1.24.5 (P5 final = milestone release). 19 requirements (REQ-291..309), 4 execution phases + P0 pre-execution + P5 final review/ship.

Wave model

Each phase is a vertical slice (end-to-end: policy files + Python wiring + tests + docs). Phases are ordered by dependency: the engine protocol (P1) must exist before policies (P2/P3) can be wired; the pipeline wiring (P3) must exist before the meta-policies (P3) can consume the merged PCR list; the regression-gate policies (P4) are independent of the pipeline and can be authored in parallel with P3's tests, but ship after P3 because they reference the engine registry finalized in P1. Within each phase, the waves are the persona task groups (parallelizable across personas when parallelization.enabled: true, max_concurrent_agents: 5).

Phase breakdown

Phase P1 — engine-core (Wave 1, backend-engineer + policy-engineer + data-engineer)

Type: feat (engine protocol + registry + kyverno-json engine adapter + install + tests)

Requirements: REQ-291, REQ-292, REQ-293, REQ-294, REQ-308, REQ-309

Must-haves:

• core/policy_engine.py — PolicyEngine Protocol (PEP 544) + PolicyEngineRegistry (selects from config.json.policy.engine) + NullEngine fallback (emits SKIPPED when policy key absent) (REQ-291)
• .ciagent/config.json gains policy object: {"engine": "kyverno-json", "policy_root": "adapters/kyverno-json/policies"} (REQ-292)
• adapters/kyverno-json/kyverno_json_engine.py — KyvernoJsonEngine implementing the protocol: is_configured() guards on which kj; evaluate() writes payload to temp JSON, invokes kj scan --policy <dir> --payload <json> --output json, translates native output → list[dict] PCR records (engine: "kyverno", ruleId prefixed KJ_<policy_name>, severity from nova.cloudinit.dev/severity annotation); defensive parsing (malformed → error PCR, never exception); is_configured()==false → single SKIPPED PCR (KJ_ENGINE_NOT_CONFIGURED) (REQ-293)
• adapters/kyverno-json/__init__.py exports KyvernoJsonEngine; adapters/kyverno-json/policies/_smoke.json trivial require-contract-id policy for round-trip validation; scripts/install-kyverno-json.sh runs go install github.com/kyverno/kyverno-json/cmd/kj@latest; .github/workflows/ci.yml + .gitea/workflows/ci.yml install Go + kj (cached) (REQ-294)
• tests/test_policy_engine.py — protocol conformance, registry selection, unknown-engine KeyError, NullEngine fallback, is_configured() false when which kj absent (mocked) (REQ-308)
• tests/test_kyverno_json_engine.py — evaluate() returns PCR dicts validating against schemas/policy_check_result.schema.json (via jsonschema); defensive parsing (malformed kyverno-json output → error PCR); is_configured()==false → SKIPPED with KJ_ENGINE_NOT_CONFIGURED; pytest.skip("kj not installed") when which kj absent (REQ-309)

Vertical slice: The PolicyEngineRegistry.get_engine() returns a configured KyvernoJsonEngine that can evaluate() a trivial payload against _smoke.json and produce a valid PCR list. The confidence signal is unchanged — it already consumes list[PolicyCheckResult]. The platform runs with or without the kj binary (is_configured() guard). All existing tests pass (NullEngine fallback when policy key absent in test config — but the v1.25 config.json sets the key, so existing tests that use the real config get KyvernoJsonEngine with is_configured()==false → SKIPPED).

Files touched:

• core/policy_engine.py (NEW)
• .ciagent/config.json (MODIFIED — policy object)
• adapters/kyverno-json/__init__.py (NEW)
• adapters/kyverno-json/kyverno_json_engine.py (NEW)
• adapters/kyverno-json/policies/_smoke.json (NEW)
• scripts/install-kyverno-json.sh (NEW)
• .github/workflows/ci.yml (MODIFIED — Go + kj install step)
• .gitea/workflows/ci.yml (MODIFIED — Go + kj install step)
• tests/test_policy_engine.py (NEW)
• tests/test_kyverno_json_engine.py (NEW)

Verification: pytest tests/test_policy_engine.py tests/test_kyverno_json_engine.py tests/test_confidence_signal.py tests/test_adapter.py tests/test_checkov_adapter.py tests/test_kyverno_adapter.py -v (new tests pass or skip-without-kj; existing adapter/confidence tests unchanged). python3 -m py_compile core/policy_engine.py adapters/kyverno-json/kyverno_json_engine.py.

---

Phase P2 — contract + stack-IR policies (Wave 2, policy-engineer + backend-engineer)

Type: feat (policies + resolver wiring + tests)

Requirements: REQ-295, REQ-296, REQ-297, REQ-298, REQ-299

Must-haves:

• adapters/kyverno-json/policies/contract/ — 4 policies over consumer contract JSON: require-id-pattern.json, require-env-in-enum.json, require-infrastructure-min-1.json, forbid-unknown-fields.json — each a ValidatingPolicy with one validate.assert rule using JMESPath against the payload root; severity via nova.cloudinit.dev/severity annotation (REQ-295)
• core/contract_resolver.py invokes PolicyEngineRegistry.get_engine().evaluate(contract_dict, policies/contract/, contract_id) before resolving; failures feed the policy input as fail PCRs (no resolver exit — confidence signal decides the gate, --soft-fail pattern); emits nova.policy.evaluated metrics event (REQ-296)
• adapters/kyverno-json/policies/stack-ir/ — 3 policies over resolved Stack IR: require-tagging-standard.json (ports nova_tagging.py — nova:owner + nova:environment tags on every resources[] entry), forbid-public-ingress.json (v1.0 demo rule), require-encryption-by-default.json (v1.8 D-encryption-default); ~ modifier iterates resources[] (REQ-297)
• core/contract_resolver.py invokes the engine with the resolved Stack IR and policies/stack-ir/ after resolving; resulting PCRs appended to the contract-policy PCRs; resolver return values and exceptions unchanged (additive) (REQ-298)
• tests/test_stack_ir_policies.py + tests/fixtures/stack_ir/ — passing IR (all tags + encryption) + failing IR (missing tags, public ingress, plaintext bucket); each policy in isolation + full dir as bundle; pytest.skip("kj not installed") when which kj absent (REQ-299)

Vertical slice: A consumer contract passes through the resolver and produces two PCR lists (contract policies pre-resolve, stack-IR policies post-resolve) that feed the confidence signal. A contract with a bad id or missing tags produces fail PCRs that lower the confidence score. The resolver's existing tests pass unchanged (the policy call is additive — it does not change resolver return values or exceptions).

Files touched:

• adapters/kyverno-json/policies/contract/require-id-pattern.json (NEW)
• adapters/kyverno-json/policies/contract/require-env-in-enum.json (NEW)
• adapters/kyverno-json/policies/contract/require-infrastructure-min-1.json (NEW)
• adapters/kyverno-json/policies/contract/forbid-unknown-fields.json (NEW)
• adapters/kyverno-json/policies/stack-ir/require-tagging-standard.json (NEW)
• adapters/kyverno-json/policies/stack-ir/forbid-public-ingress.json (NEW)
• adapters/kyverno-json/policies/stack-ir/require-encryption-by-default.json (NEW)
• core/contract_resolver.py (MODIFIED — pre/post resolve engine calls)
• tests/test_stack_ir_policies.py (NEW)
• tests/fixtures/stack_ir/passing.json (NEW)
• tests/fixtures/stack_ir/failing.json (NEW)

Verification: pytest tests/test_contract_resolver.py tests/test_stack_ir_policies.py tests/test_policy_engine.py -v (existing resolver tests pass; new policy tests pass or skip-without- kj). python3 -m py_compile core/contract_resolver.py.

---

Phase P3 — plan-JSON policies + meta-orchestration + pipeline wiring (Wave 3, policy-engineer + backend-engineer)

Type: feat (plan-JSON policies + meta-policies + run_platform.sh wiring + tests)

Requirements: REQ-300, REQ-301, REQ-302, REQ-303

Must-haves:

• adapters/kyverno-json/policies/plan-json/ — 3 policies over terraform show -json output: forbid-plaintext-secrets.json (ports CKV_AWS_41/45/46), forbid-iam-wildcard.json (ports CKV_AWS_1/40), require-kms-reference.json (ports CKV_AWS_7/33); JMESPath over planned_values.root_module.resources[] (REQ-300)
• run_platform.sh Step 5 gains a parallel kyverno-json pass: after Checkov/Wiz produce raw PCRs, the script runs kj scan --policy adapters/kyverno-json/policies/plan-json/ --payload <tfshow.json> -o json and pipes through adapters/kyverno-json/kyverno_json_engine.py to produce a second PCR list; both lists concatenated and fed to the confidence signal; nova.policy.evaluated event with both engine names; when which kj is false, logs and proceeds with Checkov/Wiz list only (no hard failure) (REQ-301)
• tests/test_plan_json_policies.py + tests/fixtures/plan_json/ — passing plan (no secrets, no wildcard, KMS alias) + failing plan (plaintext password, Action: "*", inline KMS key); policies in isolation + bundle; tests/test_run_platform_plan_json_policies.py asserts run_platform.sh has the kyverno-json Step 5 block + concatenates PCR lists (script-substring assertion, pattern from tests/test_pipeline.py:79-95) (REQ-302)
• adapters/kyverno-json/policies/meta/ — block-on-any-critical.json (asserts no PCR in merged list has severity: critical + result: fail; if any does, emits fail PCR KJ_META_BLOCK_CRITICAL severity critical — declarative source of truth; the confidence_signal.py hard-override stays as defense-in-depth per D-119) + tagging-rules-agree.json (cross-checks Checkov NOVA_TAG_NAMING vs kj KJ_REQUIRE_TAGGING_STANDARD by resourceRef; divergence emits error PCR per D-118); tests/test_meta_policies.py (REQ-303)

Vertical slice: run_platform.sh Step 5 produces a merged PCR list (Checkov/Wiz + kj plan-JSON policies + kj meta-policies over the merged list) that feeds the confidence signal. A plan with a plaintext secret produces two fail PCRs (one Checkov, one kj) for the same resource — visible defense-in-depth. A critical finding anywhere produces a KJ_META_BLOCK_CRITICAL meta-PCR that the confidence signal's hard-override blocks. The pipeline runs with or without kj (graceful skip).

Files touched:

• adapters/kyverno-json/policies/plan-json/forbid-plaintext-secrets.json (NEW)
• adapters/kyverno-json/policies/plan-json/forbid-iam-wildcard.json (NEW)
• adapters/kyverno-json/policies/plan-json/require-kms-reference.json (NEW)
• adapters/kyverno-json/policies/meta/block-on-any-critical.json (NEW)
• adapters/kyverno-json/policies/meta/tagging-rules-agree.json (NEW)
• scripts/run_platform.sh (MODIFIED — Step 5 kj parallel pass)
• tests/test_plan_json_policies.py (NEW)
• tests/test_meta_policies.py (NEW)
• tests/test_run_platform_plan_json_policies.py (NEW)
• tests/fixtures/plan_json/passing.json (NEW)
• tests/fixtures/plan_json/failing.json (NEW)

Verification: pytest tests/test_plan_json_policies.py tests/test_meta_policies.py tests/test_run_platform_plan_json_policies.py tests/test_pipeline.py -v (new tests pass or skip-without-kj; existing pipeline tests pass). python3 -m py_compile on any modified Python. Shellcheck on run_platform.sh if available.

---

Phase P4 — regression-gate policies + docs (Wave 4, policy-engineer + data-engineer + lead-developer)

Type: feat (regression policies) + docs (adapter READMEs + ARCHITECTURE + STANDARDS + METRICS)

Requirements: REQ-304, REQ-305, REQ-306, REQ-307

Must-haves:

• adapters/kyverno-json/policies/regression/ — 3 policies over capability-inventory JSON frontmatter: cap-013-adapter-dedup.json, cap-023-metrics-collector.json, cap-024-deck-structure.json; emit pass/fail PCRs per capability; the existing core/regression_verify.py is kept (drives the CI gate); the policies are the declarative mirror (REQ-304)
• tests/test_regression_policies.py + tests/fixtures/capability_inventory/clean.json + tests/fixtures/capability_inventory/drifted.json — clean (all caps pass) + drifted (duplicate adapter, missing metric status, broken deck arc); regression gate still 287/287 baseline (new tests additive, skip-without-kj) (REQ-305)
• adapters/README.md gains new kyverno-json adapter row + "Policy Engine Protocol" section (Protocol, registry, swap boundary, how-to-add-OpaEngine); adapters/kyverno-json/README.md documents the engine, install path, policy directory layout, 4 policy categories (REQ-306)
• .ciagent/ARCHITECTURE.md §12.7 (added in RESEARCH) is finalized; schemas/README.md notes engine: "kyverno" shared by K8s adapter
  • kj (distinguished by ruleId prefix); modules/STANDARDS.md gains "Policy authoring standard" section for module owners; docs/METRICS.md notes the policy engine is swappable (Strategic Objective #2 — provable trust via a replaceable substrate) (REQ-307)

Vertical slice: The regression gate's capability checks are now declarative policies auditable as artifacts. A new module owner can read modules/STANDARDS.md "Policy authoring standard" and write a per-module kyverno-json policy. A new engineer can read adapters/README.md "Policy Engine Protocol" and implement an OpaEngine. The 287/287 baseline is unchanged.

Files touched:

• adapters/kyverno-json/policies/regression/cap-013-adapter-dedup.json (NEW)
• adapters/kyverno-json/policies/regression/cap-023-metrics-collector.json (NEW)
• adapters/kyverno-json/policies/regression/cap-024-deck-structure.json (NEW)
• tests/test_regression_policies.py (NEW)
• tests/fixtures/capability_inventory/clean.json (NEW)
• tests/fixtures/capability_inventory/drifted.json (NEW)
• adapters/README.md (MODIFIED — new row + PolicyEngine Protocol section)
• adapters/kyverno-json/README.md (NEW)
• schemas/README.md (MODIFIED — engine enum note)
• modules/STANDARDS.md (MODIFIED — Policy authoring standard section)
• docs/METRICS.md (MODIFIED — swappable engine narrative)

Verification: pytest tests/test_regression_policies.py tests/test_kyverno_json_engine.py -v (new tests pass or skip-without- kj). Full regression gate pytest tests/ still at 287/287 baseline + new tests (skip without kj). Manual read of adapters/README.md + adapters/kyverno-json/README.md + modules/STANDARDS.md policy section for clarity.

---

Phase P5 — final review + audit + milestone ship (Final Phase)

Type: docs (review + audit + milestone completion)

Requirements: All REQ-291..309 (mark complete)

Must-haves:

• ciagent-review multi-persona code review across P1..P4 (lead-developer, backend-engineer, data-engineer, policy-engineer). Auto-fix P0; flag P1+ for post-hoc review. If P1+ issues found, fix them in this final phase (not loop back to EXECUTE).
• ciagent-audit — reconstruction test (git log ↔ .ciagent/ files), .ciagent/ file discipline, branch hygiene, commit discipline. Critical issues fixed in this phase.
• ciagent-ship (milestone) — merge phase/05-final-review-ship → milestone/v1.25-kyverno-json → main; tag v1.24.5 (= the v1.25 release per the prev-minor tagging rule); create Gitea release with full milestone summary (all phases, all requirements); delete all milestone branches (local + remote).
• Update REQUIREMENTS.md (mark REQ-291..309 complete), ROADMAP.md (mark v1.25 complete), CHECKPOINT.json (milestone_complete: true), NORTH_STAR.md (note Strategic Objective #2 — provable trust via a replaceable policy-engine substrate).

Vertical slice: The v1.25 milestone is complete: kyverno-json is the primary policy tool, behind a swappable adapter, with policies over all 4 Nova artifacts. Tags v1.24.0..v1.24.5 on the v1.24.x line. The milestone branch merges to main.

Verification: pytest tests/ -v full suite passes (287 baseline + new tests). git log --oneline shows the v1.25 phase commits. git tag shows v1.24.0..v1.24.5. git branch shows no leftover milestone/phase branches (all deleted post-ship).

---

Wave ordering (parallelization)

With parallelization.enabled: true, max_concurrent_agents: 5, min_plans_for_parallel: 2:

• P1 Wave 1: backend-engineer (protocol + registry + install) ‖ data-engineer (config.json policy object) ‖ policy-engineer (engine adapter + smoke policy). 3 concurrent personas. Merge in order: data-engineer → backend-engineer → policy-engineer.
• P2 Wave 2: policy-engineer (contract + stack-IR policies) ‖ backend-engineer (resolver wiring — depends on P1 registry). 2 concurrent. Merge: policy-engineer → backend-engineer (wiring references the policy dirs).
• P3 Wave 3: policy-engineer (plan-JSON + meta policies) ‖ backend-engineer (run_platform.sh wiring — depends on P1 engine + P2 resolver pattern). 2 concurrent. Merge: policy-engineer → backend-engineer.
• P4 Wave 4: policy-engineer (regression policies) ‖ data-engineer (capability-inventory fixtures) ‖ lead-developer (docs: READMEs, STANDARDS, METRICS). 3 concurrent. Merge: data-engineer → policy-engineer → lead-developer.

Territory enforcement: warn mode (per config.json personas.territory_enforcement: "warn"). Cross-territory edits (e.g., backend-engineer touching a policy file) emit a warning, not a block.

Requirement → phase → persona matrix

REQ	Phase	Primary persona	Type
REQ-291	P1	backend-engineer	feat
REQ-292	P1	data-engineer	feat (config)
REQ-293	P1	policy-engineer	feat
REQ-294	P1	backend-engineer	feat (install)
REQ-295	P2	policy-engineer	feat
REQ-296	P2	backend-engineer	feat (wiring)
REQ-297	P2	policy-engineer	feat
REQ-298	P2	backend-engineer	feat (wiring)
REQ-299	P2	policy-engineer	test
REQ-300	P3	policy-engineer	feat
REQ-301	P3	backend-engineer	feat (pipeline)
REQ-302	P3	policy-engineer + backend-engineer	test
REQ-303	P3	policy-engineer	feat (meta)
REQ-304	P4	policy-engineer	feat
REQ-305	P4	policy-engineer + data-engineer	test
REQ-306	P4	policy-engineer + lead-developer	docs
REQ-307	P4	lead-developer	docs
REQ-308	P1	backend-engineer	test
REQ-309	P1	policy-engineer	test