acdl/.ciagent/RESEARCH.md

Nova — v1.25 Research Findings

Phase: research (pre-execution). Milestone: v1.25 (kyverno-json Unified Policy Engine). Status: research. Researcher: ci-researcher. Autonomy: full.

1. Problem domain

Nova's compliance/policy posture is fragmented across three engines with three rule languages and three adapter shapes (see PROJECT.md v1.25 "Why" for the full diagnosis). The PolicyCheckResult schema (schemas/policy_check_result.schema.json) is already the engine-agnostic contract that core/confidence_signal.py consumes — the contract is right; the orchestration is fragmented. There is no single declarative place where "what Nova considers compliant" lives. The K8s-only Kyverno adapter (adapters/kyverno/) can't help because it only speaks to K8s manifests and the platform emits Terraform (D-053).

kyverno-json is the correction: a Kyverno-ecosystem runtime that applies Kyverno policies to any JSON/YAML payload. It becomes the unified orchestrator of compliance checks, behind a swappable PolicyEngine protocol so OPA can replace it one day. Checkov and Wiz remain as raw-finding adapters feeding into kyverno-json meta-policies.

2. kyverno-json — the engine surface

2.1 What it is

kyverno-json is a standalone Go binary from the Kyverno project. It is a separate runtime from the Kyverno K8s admission controller — same policy lineage, different application target. Where Kyverno (K8s) evaluates ClusterPolicy resources against Kubernetes manifests at admission time, kyverno-json evaluates ValidatingPolicy resources against any JSON or YAML payload file via the CLI (kj scan) or a Go library. It is not a Python package (no PyPI release); it is installed via go install github.com/kyverno/kyverno-json/cmd/kj@latest (D-115) or by downloading a pinned binary from GitHub releases.

2.2 CLI surface (the v1.25 invocation path)

The v1.25 engine uses the kj scan subcommand:

kyverno-json scan [flags]

Flags:
  --labels strings     Labels selectors for policies
  --output string      Output format (text or json) (default "text")
  --payload string     Path to payload (json or yaml file)
  --policy strings     Path to kyverno-json policies
  --pre-process strings  JMESPath expression used to pre process payload

The KyvernoJsonEngine.evaluate() implementation (REQ-293) invokes:

kj scan --policy <policy_dir> --payload <payload.json> --output json

and parses the JSON results[] array. The --pre-process flag is available for JMESPath pre-projection (noted for the meta-policy use case where the payload is the merged PCR list and a pre-process expression can index by ruleId — recorded as a future optimization, not used in v1.25's initial implementation).

Other subcommands (kj jp, kj serve, kj playground, kj docs) are out of scope for v1.25. kj serve is the long-running web-app mode (noted as a future consideration for lower-latency evaluation in the Out of Scope section of REQUIREMENTS.md). kj jp is the JMESPath REPL — useful for policy authoring/debugging, not invoked by the engine.

2.3 Policy structure (the ValidatingPolicy resource)

kyverno-json policies are Kubernetes-style resources (cluster-scoped) belonging to the json.kyverno.io API group, kind ValidatingPolicy, version v1alpha1:

apiVersion: json.kyverno.io/v1alpha1
kind: ValidatingPolicy
metadata:
  name: <policy-name>           # becomes the KJ_<policy-name> ruleId prefix
spec:
  rules:
    - name: <rule-name>
      identifier: <jmespath>     # optional — path to the unique entry id
      match:                    # assertion tree — which payload entries
        any:                     #   the rule applies to
        - <assertion>
      exclude:                  # optional — exclude matching entries
        any:
        - <assertion>
      context:                  # optional — named bindings available to
        - name: <binding>       #   the rule's assertions ($<binding>)
          variable: <value>
      validate:
        message: "<human-readable>"   # optional per-rule message
        assert:
          all:                  # all assertions must hold
          - check: <assertion-tree>
            message: "<per-check>"
        # OR
          any:                  # at least one assertion must hold
          - check: <assertion-tree>

Key differences from K8s Kyverno policies:

• Always cluster-scoped — no namespace field.
• No forEach, pattern operators, anchors, or wildcards. Iteration is done via the ~ projection modifier in assertion trees (see §2.4).
• Assertion trees with JMESPath expressions replace Kyverno's pattern-matching syntax (see §2.4).

2.4 Assertion trees (the rule language)

An assert declaration contains an all or any list. Each entry has a check (the assertion tree — a nested JMESPath projection) and an optional message. All comparisons happen in the leaves of the tree.

A simple example (assert a pod doesn't use the default service account):

validate:
  assert:
    all:
    - message: "serviceAccountName 'default' is not allowed"
      check:
        spec:
          (serviceAccountName == 'default'): false

The (expression) syntax evaluates a JMESPath expression; the result becomes the current object for descendants; the leaf value is compared to the expected value.

Iteration via the ~ modifier. The ~ prefix on a key applies descendant assertions to each element of an array/map individually (rather than comparing the whole array). Given foo.bar: [1,2,3]:

check:
  foo:
    ~.bar:               # iterate each element
      (@ < `5`): true    # assert each element < 5

The ~index_name.bar form binds the index (array) or key (map) to $index_name for use in descendants. This is how v1.25 iterates resources[] in the Stack IR policies (REQ-297) and planned_values.root_module.resources[] in the plan-JSON policies (REQ-300).

Explicit bindings via ->binding_name allow descendants to refer to a parent node via $binding_name. Built-in bindings: $payload (the whole input), $policy, $rule.

Escaping via \key\ prevents projection when a payload key collides with the projection syntax. Not needed for Nova payloads (no (key) fields), noted for completeness.

2.5 Output shape (what kj scan --output json produces)

The JSON output is a results[] array. Each result entry has (at minimum):

• policy: the policy metadata.name
• rule: the rule name
• result: "pass" | "fail" | "error" | "skip" (lowercase)
• message: the assertion message (or engine error message)
• resource: the matched payload entry (the identifier value, or the whole payload when no identifier/match)
• namespace/kind/name: K8s-style fields (present but empty for non-K8s payloads — the K8s Kyverno adapter's evidence uses these; the kyverno-json engine's evidence uses assertion/jmespath instead)
• severity: not present by default (kyverno-json does not assign severities — the Nova policy author assigns severity via a Nova- specific annotation; see §2.6)

The KyvernoJsonEngine._to_pcr() translator (REQ-293) maps:

• policy → ruleId (prefixed KJ_<policy_name> per D-116)
• result → result (pass/fail/error → pass/fail/error; skip/skipped → skipped)
• message → message
• resource → resourceRef + evidence.resource
• severity from the policy's metadata.annotations (see §2.6)
• engine: "kyverno" (per D-116 — no new enum value)

2.6 Severity assignment (Nova convention)

kyverno-json does not natively assign severities to results. Nova's confidence signal requires a severity per PCR (critical/high/medium/ low/info). The v1.25 convention: each Nova policy file declares its severity via a metadata.annotations field:

metadata:
  name: forbid-public-ingress
  annotations:
    nova.cloudinit.dev/severity: high

The KyvernoJsonEngine._to_pcr() reads this annotation from the loaded policy YAML (not from the scan result — the result doesn't carry it) and applies it to every result that policy produces. Default when absent: info. This keeps severity in the policy (declarative, version- controlled) rather than in the engine adapter (imperative). The annotation key is nova.cloudinit.dev/severity (matches the existing nova.cloudinit.dev namespace used in schemas/tagging-standard.json).

3. The four policy targets (v1.25 scope)

3.1 Consumer contract JSON (REQ-295)

The payload is the parsed contract dict (the raw YAML loaded as JSON). Policies assert the contract.schema.json constraints declaratively: require-id-pattern (JMESPath regex ^[a-z][a-z0-9-]{2,5}$ over id), require-env-in-enum (environment in ["dev","qa","prod","dr"]), require-infrastructure-min-1 (length(infrastructure) > 0), forbid-unknown-fields (keys subset of the 4 allowed). These are the declarative equivalent of the jsonschema constraints — they let Nova apply its own compliance posture (e.g. forbid a specific env for a specific consumer) on top of schema validity without editing the jsonschema.

Invocation point: core/contract_resolver.py pre-resolve (REQ-296). Early-fail: if a contract policy fails, the resolver still proceeds (the confidence signal decides the gate, consistent with the existing --soft-fail Checkov pattern) — but the failing PCRs are in the policy input, which lowers the score.

3.2 Resolved Target Stack IR JSON (REQ-297)

The payload is the resolved Stack IR dict produced by core/contract_resolver.py (the merged module outputs). Policies assert over resources[] (the array of resolved resources): require-tagging-standard (every resource's tags has nova:owner + nova:environment — ports adapters/terraform/policy/custom_rules/nova_tagging.py), forbid-public-ingress (no resource has public_ingress: true — the v1.0 demo rule, now declarative), require-encryption-by-default (every S3/EBS/KMS-aliased resource carries encryption config — ports the v1.8 D-encryption-default rule). The ~ modifier iterates resources[].

Invocation point: core/contract_resolver.py post-resolve (REQ-298). Additive — the resolver's return values and exceptions are unchanged; the PCRs are appended to the contract-policy PCRs.

3.3 Terraform plan JSON (REQ-300)

The payload is terraform show -json <tfplan> output. Policies assert over planned_values.root_module.resources[]: forbid-plaintext-secrets (no aws_db_instance.password / aws_iam_user.login_profile.password in plaintext — ports CKV_AWS_41/45/46), forbid-iam-wildcard (no Action: "*" or Resource: "*" in aws_iam_policy.PolicyDocument — ports CKV_AWS_1/40), require-kms-reference (KMS keys referenced by alias, not inline key material — ports CKV_AWS_7/33). These are declarative mirrors of checkov_adapter.py:RULE_MAP — the Checkov rule stays the source of truth for terraform_plan scanning; the kyverno-json policy covers the same plan JSON with a different rule language (defense-in-depth against engine drift).

Invocation point: run_platform.sh Step 5 (REQ-301). After Checkov/Wiz produce raw PCRs, the script runs kj scan over the plan JSON; both PCR lists concatenate into the confidence signal's policy input. When which kj is false, the script logs and proceeds with the Checkov/Wiz list only.

3.4 PolicyCheckResult records (meta-policies, REQ-303)

The payload is the merged list[PolicyCheckResult] produced by checkov + wiz + the plan-JSON policies. This is the most novel target — kyverno-json policies over the policy results themselves. block-on-any-critical asserts no PCR has severity: "critical" + result: "fail"; if any does, the meta-policy emits a fail PCR with ruleId: "KJ_META_BLOCK_CRITICAL" and severity critical. This is the declarative source of truth for "critical = block" (D-119 — the confidence_signal.py PENALTY["critical"]: None hard-override stays as defense-in-depth). tagging-rules-agree cross-checks the Checkov NOVA_TAG_NAMING result against the kyverno-json KJ_REQUIRE_TAGGING_STANDARD result by resourceRef; divergence emits an error PCR (D-118).

Invocation point: after the three target policies (contract/stack- IR/plan-JSON) produce their PCR lists, the merged list is the payload for the meta-policies. The meta-policy PCRs are appended to the merged list, which is what the confidence signal consumes.

4. The PolicyEngine swap boundary

4.1 Protocol shape (REQ-291)

A Python Protocol (PEP 544 — structural subtyping, no inheritance):

class PolicyEngine(Protocol):
    @property
    def name(self) -> str: ...
    def is_configured(self) -> bool: ...
    def evaluate(self, payload: dict | str, policy_dir: Path,
                 contract_id: str) -> list[dict]: ...

list[dict] (not list[PolicyCheckResult] — there's no dataclass; the schema is enforced via jsonschema validation in tests, matching the existing adapter pattern). The registry selects the active engine from config.json.policy.engine. A NullEngine is the fallback when the policy key is absent (emits SKIPPED — backward compatibility for tests that don't set the key).

4.2 The OPA-equivalent surface (future swap)

OPA (Open Policy Agent) is the most likely future replacement. The mapping:

Nova PolicyEngine member	kyverno-json impl	OPA equivalent
name	"kyverno-json"	"opa"
is_configured()	which kj	which opa
evaluate(payload, policy_dir, contract_id)	kj scan --policy <dir> --payload <json> -o json	opa eval -d <dir> -i <json> 'data.nova.<...>'
Policy file format	ValidatingPolicy (YAML)	Rego (.rego)
Result shape	results[] (pass/fail/error/skip)	result (set of violations)
Severity	Nova annotation nova.cloudinit.dev/severity	Nova convention (Rego metadata or a wrapper)

The protocol is minimal (3 members) specifically so the OPA implementation is a known quantity: an OpaEngine class that shells to opa eval, translates the Rego violation set to PCR dicts, and implements is_configured() via which opa. The policy files would need rewriting (Rego, not ValidatingPolicy) — but the protocol, the registry, the confidence signal, and the PCR schema are all untouched. This is the swap boundary the user asked for ("Implemented as an adapter since we might one day decide to replace it with something else like OPA").

4.3 Why not a full plugin registry?

A setuptools entry-point plugin registry (like checkov's --external-checks-dir) was considered and rejected: Nova has 1 active engine today (kyverno-json) and at most 2 in the foreseeable future (kyverno-json + OPA). A Protocol + dict registry in core/policy_engine.py is the right weight — discoverable, typed, testable, and ~40 lines. An entry-point registry adds packaging complexity (entry-point metadata, version resolution) for no gain at this scale. The register(name, factory) method on the registry is the extension point if a future milestone needs runtime plugin discovery.

5. Latency / MTTR impact (G-Q3 anticipation)

NORTH_STAR.md MTTR target: < 60s p95. run_platform.sh Step 5 today runs Checkov over the terraform plan (typically 5-15s for a small stack). Adding kj scan over the same plan JSON adds:

• Process spawn: ~50ms (Go binary startup)
• Policy load: ~20ms (a handful of YAML files)
• Assertion evaluation: ~100-500ms (JMESPath over a small plan)
• Total: < 1s for a typical Nova stack

The kyverno-json pass runs in parallel with Checkov (REQ-301 — the script launches both and waits on both), so the wall-clock impact is max(checkov_time, kj_time) ≈ checkov_time (kj is faster). The contract + stack-IR policies run during resolve (already a fast step). Meta-policies run over the merged list (in-memory, < 10ms). No measurable MTTR impact is expected. This will be verified in P3 VERIFY with a timing assertion.

6. "Platform functions without AI" tenet (G-Q1 / D-120)

kyverno-json is deterministic (same policy + payload → same result, every run). It is not an LLM, not a probabilistic model, not a "judgement" engine. The NORTH_STAR.md tenet ("the platform functions without AI — 'AI decisions' are really automated decisions") distinguishes AI (non-reproducible) from automation (reproducible). kyverno-json is the latter. Adding it is more aligned with the tenet than the current imperative Python in core/env_transition.py and core/regression_verify.py, because the policy is declarative (visible, auditable, version-controlled) rather than imperative (logic hidden in function bodies). The is_configured() guard ensures the platform functions without the binary (graceful skip → SKIPPED PCR → confidence signal proceeds).

7. ECS policy catalog overlap (prior art)

The kyverno-json catalog ships ECS policies that overlap with Nova's L1 modules: ecs-cluster-enable-logging, ecs-cluster-required- container-insights, ecs-service-public-ip, ecs-service-required- latest-platform-fargate, ecs-task-definition-fs-read-only. These are reference policies, not drop-in Nova policies — they target the AWS ECS API shape (type: aws_ecs_service etc.), not Nova's Stack IR shape. v1.25 policies target the Nova IR (REQ-297) and the terraform plan JSON (REQ-300), not the raw AWS API. The catalog is useful as prior art for JMESPath patterns over ECS resources — the ecs-service-public-ip policy's contains('$allowed-values', @.assign_public_ip) pattern informs the Nova forbid-public-ingress policy shape. No catalog policies are imported directly in v1.25.

8. Risks & mitigations

Risk	Likelihood	Impact	Mitigation
kj binary not in CI image	medium	blocks P3+ tests	is_configured() guard + pytest.skip + scripts/install-kyverno-json.sh
kyverno-json output shape changes across versions	low	breaks _to_pcr()	pin @latest to a known-good commit in install-kyverno-json.sh after P1 smoke; defensive parsing (malformed → error PCR, not exception)
Policy explosion (4 targets × N rules)	medium	maintenance load	wave ordering (PLAN); policies co-located per target dir; meta-policy cross-check keeps the set auditable
Checkov + kj tagging-rule drift	medium	false error PCRs	tagging-rules-agree meta-policy emits error on divergence (visible, not silent); the Checkov rule stays source of truth for HCL, kj for IR
OPA swap turns out harder than the protocol implies	low	future milestone rework	RESEARCH §4.2 documents the OPA-equivalent surface; the protocol is the contract, not the implementation
--pre-process needed for meta-policies but undocumented behavior	low	meta-policy bugs	v1.25 meta-policies use plain assertion trees over the PCR list (no pre-process); --pre-process noted as a future optimization only

9. Assumptions (logged, full autonomy)

• A1: kj scan --output json produces a stable results[] array shape. Will be verified in P1 smoke test (_smoke.json policy + a trivial payload); if the shape differs, _to_pcr() is adjusted defensively (malformed → error PCR). Confidence: 0.85.
• A2: The nova.cloudinit.dev/severity annotation convention is read by the engine from the policy YAML (loaded once per evaluate() call). kyverno-json does not validate unknown annotations — they pass through. Confidence: 0.90.
• A3: The ~ projection modifier iterates resources[] in the Stack IR and planned_values.root_module.resources[] in the plan JSON correctly. Verified in P2/P3 tests. Confidence: 0.85.
• A4: go install works in the CI image (Go toolchain available or installable). If not, the binary-release download path is the documented fallback in install-kyverno-json.sh. Confidence: 0.80.
• A5: The NullEngine fallback (when policy key absent in config.json) keeps all existing tests passing — they don't set the key, so they get NullEngine → SKIPPED PCRs → confidence signal proceeds with policy input [SKIPPED] → per-input score 1.0 (skipped counts as pass in _per_input_score). Confidence: 0.95 (verified against confidence_signal.py:84-89).

10. Decisions referenced

D-115 (install path), D-116 (engine enum reuse), D-117 (adapter signatures unchanged), D-118 (tagging cross-check), D-119 (critical- override defense-in-depth), D-120 (deterministic not AI). See CLARIFY.md for the full resolution text.

11. Architecture updates (deferred to RESEARCH-stage file edits)

• .ciagent/ARCHITECTURE.md gains §12.7 "Policy Engine Registry" with the registry diagram. Deferred to the RESEARCH commit (this file's commit) — the section is authored as part of this research.
• schemas/README.md notes engine: "kyverno" is shared by the K8s adapter and kyverno-json (distinguished by ruleId prefix).
• modules/STANDARDS.md gains a "Policy authoring standard" section (P4, REQ-307).
• docs/METRICS.md notes the policy engine is swappable (P4, REQ-307).