acdl/.ciagent/RESEARCH.md

# Nova — v1.25 Research Findings

> Phase: research (pre-execution). Milestone: v1.25 (kyverno-json Unified
> Policy Engine). Status: research. Researcher: ci-researcher.
> Autonomy: full.

## 1. Problem domain

Nova's compliance/policy posture is fragmented across three engines with
three rule languages and three adapter shapes (see PROJECT.md v1.25
"Why" for the full diagnosis). The `PolicyCheckResult` schema
(`schemas/policy_check_result.schema.json`) is already the engine-agnostic
contract that `core/confidence_signal.py` consumes — the *contract* is
right; the *orchestration* is fragmented. There is no single declarative
place where "what Nova considers compliant" lives. The K8s-only Kyverno
adapter (`adapters/kyverno/`) can't help because it only speaks to K8s
manifests and the platform emits Terraform (D-053).

`kyverno-json` is the correction: a Kyverno-ecosystem runtime that applies
Kyverno policies to **any** JSON/YAML payload. It becomes the **unified
orchestrator** of compliance checks, behind a swappable `PolicyEngine`
protocol so OPA can replace it one day. Checkov and Wiz remain as
raw-finding adapters feeding *into* kyverno-json meta-policies.

## 2. kyverno-json — the engine surface

### 2.1 What it is

[kyverno-json](https://github.com/kyverno/kyverno-json) is a standalone Go
binary from the Kyverno project. It is a **separate runtime** from the
Kyverno K8s admission controller — same policy lineage, different
application target. Where Kyverno (K8s) evaluates `ClusterPolicy`
resources against Kubernetes manifests at admission time, kyverno-json
evaluates `ValidatingPolicy` resources against **any** JSON or YAML
payload file via the CLI (`kj scan`) or a Go library. It is **not** a
Python package (no PyPI release); it is installed via
`go install github.com/kyverno/kyverno-json/cmd/kj@latest` (D-115) or by
downloading a pinned binary from GitHub releases.

### 2.2 CLI surface (the v1.25 invocation path)

The v1.25 engine uses the `kj scan` subcommand:

```
kyverno-json scan [flags]

Flags:
  --labels strings     Labels selectors for policies
  --output string      Output format (text or json) (default "text")
  --payload string     Path to payload (json or yaml file)
  --policy strings     Path to kyverno-json policies
  --pre-process strings  JMESPath expression used to pre process payload
```

The `KyvernoJsonEngine.evaluate()` implementation (REQ-293) invokes:
```
kj scan --policy <policy_dir> --payload <payload.json> --output json
```
and parses the JSON `results[]` array. The `--pre-process` flag is
available for JMESPath pre-projection (noted for the meta-policy use case
where the payload is the merged PCR list and a pre-process expression
can index by `ruleId` — recorded as a future optimization, not used in
v1.25's initial implementation).

Other subcommands (`kj jp`, `kj serve`, `kj playground`, `kj docs`) are
out of scope for v1.25. `kj serve` is the long-running web-app mode
(noted as a future consideration for lower-latency evaluation in the
Out of Scope section of REQUIREMENTS.md). `kj jp` is the JMESPath REPL —
useful for policy authoring/debugging, not invoked by the engine.

### 2.3 Policy structure (the `ValidatingPolicy` resource)

kyverno-json policies are Kubernetes-style resources (cluster-scoped)
belonging to the `json.kyverno.io` API group, kind `ValidatingPolicy`,
version `v1alpha1`:

```yaml
apiVersion: json.kyverno.io/v1alpha1
kind: ValidatingPolicy
metadata:
  name: <policy-name>           # becomes the KJ_<policy-name> ruleId prefix
spec:
  rules:
    - name: <rule-name>
      identifier: <jmespath>     # optional — path to the unique entry id
      match:                    # assertion tree — which payload entries
        any:                     #   the rule applies to
        - <assertion>
      exclude:                  # optional — exclude matching entries
        any:
        - <assertion>
      context:                  # optional — named bindings available to
        - name: <binding>       #   the rule's assertions ($<binding>)
          variable: <value>
      validate:
        message: "<human-readable>"   # optional per-rule message
        assert:
          all:                  # all assertions must hold
          - check: <assertion-tree>
            message: "<per-check>"
        # OR
          any:                  # at least one assertion must hold
          - check: <assertion-tree>
```

Key differences from K8s Kyverno policies:
- **Always cluster-scoped** — no `namespace` field.
- **No `forEach`, pattern operators, anchors, or wildcards.** Iteration
  is done via the `~` projection modifier in assertion trees (see §2.4).
- **Assertion trees** with JMESPath expressions replace Kyverno's
  pattern-matching syntax (see §2.4).

### 2.4 Assertion trees (the rule language)

An `assert` declaration contains an `all` or `any` list. Each entry has a
`check` (the assertion tree — a nested JMESPath projection) and an
optional `message`. **All comparisons happen in the leaves of the tree.**

A simple example (assert a pod doesn't use the default service account):
```yaml
validate:
  assert:
    all:
    - message: "serviceAccountName 'default' is not allowed"
      check:
        spec:
          (serviceAccountName == 'default'): false
```

The `(expression)` syntax evaluates a JMESPath expression; the result
becomes the current object for descendants; the leaf value is compared
to the expected value.

**Iteration via the `~` modifier.** The `~` prefix on a key applies
descendant assertions to **each element** of an array/map individually
(rather than comparing the whole array). Given `foo.bar: [1,2,3]`:
```yaml
check:
  foo:
    ~.bar:               # iterate each element
      (@ < `5`): true    # assert each element < 5
```
The `~index_name.bar` form binds the index (array) or key (map) to
`$index_name` for use in descendants. This is how v1.25 iterates
`resources[]` in the Stack IR policies (REQ-297) and
`planned_values.root_module.resources[]` in the plan-JSON policies
(REQ-300).

**Explicit bindings** via `->binding_name` allow descendants to refer
to a parent node via `$binding_name`. Built-in bindings: `$payload`
(the whole input), `$policy`, `$rule`.

**Escaping** via `\key\` prevents projection when a payload key collides
with the projection syntax. Not needed for Nova payloads (no `(key)`
fields), noted for completeness.

### 2.5 Output shape (what `kj scan --output json` produces)

The JSON output is a `results[]` array. Each result entry has (at
minimum):
- `policy`: the policy metadata.name
- `rule`: the rule name
- `result`: `"pass"` | `"fail"` | `"error"` | `"skip"` (lowercase)
- `message`: the assertion message (or engine error message)
- `resource`: the matched payload entry (the `identifier` value, or the
  whole payload when no identifier/match)
- `namespace`/`kind`/`name`: K8s-style fields (present but empty for
  non-K8s payloads — the K8s Kyverno adapter's evidence uses these; the
  kyverno-json engine's evidence uses `assertion`/`jmespath` instead)
- `severity`: not present by default (kyverno-json does not assign
  severities — the Nova policy author assigns severity via a Nova-
  specific annotation; see §2.6)

The `KyvernoJsonEngine._to_pcr()` translator (REQ-293) maps:
- `policy` → `ruleId` (prefixed `KJ_<policy_name>` per D-116)
- `result` → `result` (`pass`/`fail`/`error` → pass/fail/error;
  `skip`/`skipped` → skipped)
- `message` → `message`
- `resource` → `resourceRef` + `evidence.resource`
- severity from the policy's `metadata.annotations` (see §2.6)
- `engine: "kyverno"` (per D-116 — no new enum value)

### 2.6 Severity assignment (Nova convention)

kyverno-json does not natively assign severities to results. Nova's
confidence signal requires a `severity` per PCR (critical/high/medium/
low/info). The v1.25 convention: each Nova policy file declares its
severity via a `metadata.annotations` field:

```yaml
metadata:
  name: forbid-public-ingress
  annotations:
    nova.cloudinit.dev/severity: high
```

The `KyvernoJsonEngine._to_pcr()` reads this annotation from the loaded
policy YAML (not from the scan result — the result doesn't carry it) and
applies it to every result that policy produces. Default when absent:
`info`. This keeps severity in the policy (declarative, version-
controlled) rather than in the engine adapter (imperative). The
annotation key is `nova.cloudinit.dev/severity` (matches the existing
`nova.cloudinit.dev` namespace used in `schemas/tagging-standard.json`).

## 3. The four policy targets (v1.25 scope)

### 3.1 Consumer contract JSON (REQ-295)

The payload is the parsed contract dict (the raw YAML loaded as JSON).
Policies assert the `contract.schema.json` constraints declaratively:
`require-id-pattern` (JMESPath regex `^[a-z][a-z0-9-]{2,5}$` over
`id`), `require-env-in-enum` (`environment` in `["dev","qa","prod","dr"]`),
`require-infrastructure-min-1` (`length(infrastructure) > 0`),
`forbid-unknown-fields` (keys subset of the 4 allowed). These are the
declarative equivalent of the jsonschema constraints — they let Nova
apply its own compliance posture (e.g. forbid a specific env for a
specific consumer) on top of schema validity without editing the
jsonschema.

**Invocation point:** `core/contract_resolver.py` pre-resolve (REQ-296).
Early-fail: if a contract policy fails, the resolver still proceeds
(the confidence signal decides the gate, consistent with the existing
`--soft-fail` Checkov pattern) — but the failing PCRs are in the
`policy` input, which lowers the score.

### 3.2 Resolved Target Stack IR JSON (REQ-297)

The payload is the resolved Stack IR dict produced by
`core/contract_resolver.py` (the merged module outputs). Policies
assert over `resources[]` (the array of resolved resources):
`require-tagging-standard` (every resource's `tags` has `nova:owner` +
`nova:environment` — ports
`adapters/terraform/policy/custom_rules/nova_tagging.py`),
`forbid-public-ingress` (no resource has `public_ingress: true` — the
v1.0 demo rule, now declarative), `require-encryption-by-default` (every
S3/EBS/KMS-aliased resource carries encryption config — ports the v1.8
D-encryption-default rule). The `~` modifier iterates `resources[]`.

**Invocation point:** `core/contract_resolver.py` post-resolve (REQ-298).
Additive — the resolver's return values and exceptions are unchanged;
the PCRs are appended to the contract-policy PCRs.

### 3.3 Terraform plan JSON (REQ-300)

The payload is `terraform show -json <tfplan>` output. Policies assert
over `planned_values.root_module.resources[]`:
`forbid-plaintext-secrets` (no `aws_db_instance.password` /
`aws_iam_user.login_profile.password` in plaintext — ports
`CKV_AWS_41/45/46`), `forbid-iam-wildcard` (no `Action: "*"` or
`Resource: "*"` in `aws_iam_policy.PolicyDocument` — ports
`CKV_AWS_1/40`), `require-kms-reference` (KMS keys referenced by alias,
not inline key material — ports `CKV_AWS_7/33`). These are declarative
**mirrors** of `checkov_adapter.py:RULE_MAP` — the Checkov rule stays
the source of truth for `terraform_plan` scanning; the kyverno-json
policy covers the same plan JSON with a different rule language
(defense-in-depth against engine drift).

**Invocation point:** `run_platform.sh` Step 5 (REQ-301). After
Checkov/Wiz produce raw PCRs, the script runs `kj scan` over the plan
JSON; both PCR lists concatenate into the confidence signal's `policy`
input. When `which kj` is false, the script logs and proceeds with the
Checkov/Wiz list only.

### 3.4 PolicyCheckResult records (meta-policies, REQ-303)

The payload is the **merged** `list[PolicyCheckResult]` produced by
checkov + wiz + the plan-JSON policies. This is the most novel target —
kyverno-json policies over the policy results themselves.
`block-on-any-critical` asserts no PCR has `severity: "critical"` +
`result: "fail"`; if any does, the meta-policy emits a `fail` PCR with
`ruleId: "KJ_META_BLOCK_CRITICAL"` and severity `critical`. This is the
declarative source of truth for "critical = block" (D-119 — the
`confidence_signal.py` `PENALTY["critical"]: None` hard-override stays
as defense-in-depth). `tagging-rules-agree` cross-checks the Checkov
`NOVA_TAG_NAMING` result against the kyverno-json
`KJ_REQUIRE_TAGGING_STANDARD` result by `resourceRef`; divergence emits
an `error` PCR (D-118).

**Invocation point:** after the three target policies (contract/stack-
IR/plan-JSON) produce their PCR lists, the merged list is the payload
for the meta-policies. The meta-policy PCRs are appended to the merged
list, which is what the confidence signal consumes.

## 4. The `PolicyEngine` swap boundary

### 4.1 Protocol shape (REQ-291)

A Python `Protocol` (PEP 544 — structural subtyping, no inheritance):
```python
class PolicyEngine(Protocol):
    @property
    def name(self) -> str: ...
    def is_configured(self) -> bool: ...
    def evaluate(self, payload: dict | str, policy_dir: Path,
                 contract_id: str) -> list[dict]: ...
```
`list[dict]` (not `list[PolicyCheckResult]` — there's no dataclass; the
schema is enforced via `jsonschema` validation in tests, matching the
existing adapter pattern). The registry selects the active engine from
`config.json.policy.engine`. A `NullEngine` is the fallback when the
`policy` key is absent (emits `SKIPPED` — backward compatibility for
tests that don't set the key).

### 4.2 The OPA-equivalent surface (future swap)

OPA (Open Policy Agent) is the most likely future replacement. The
mapping:
| Nova `PolicyEngine` member | kyverno-json impl | OPA equivalent |
|---|---|---|
| `name` | `"kyverno-json"` | `"opa"` |
| `is_configured()` | `which kj` | `which opa` |
| `evaluate(payload, policy_dir, contract_id)` | `kj scan --policy <dir> --payload <json> -o json` | `opa eval -d <dir> -i <json> 'data.nova.<...>'` |
| Policy file format | `ValidatingPolicy` (YAML) | Rego (`.rego`) |
| Result shape | `results[]` (pass/fail/error/skip) | `result` (set of violations) |
| Severity | Nova annotation `nova.cloudinit.dev/severity` | Nova convention (Rego `metadata` or a wrapper) |

The protocol is minimal (3 members) specifically so the OPA
implementation is a known quantity: an `OpaEngine` class that shells to
`opa eval`, translates the Rego violation set to PCR dicts, and
implements `is_configured()` via `which opa`. The policy *files* would
need rewriting (Rego, not ValidatingPolicy) — but the protocol, the
registry, the confidence signal, and the PCR schema are all untouched.
This is the swap boundary the user asked for ("Implemented as an
adapter since we might one day decide to replace it with something else
like OPA").

### 4.3 Why not a full plugin registry?

A `setuptools` entry-point plugin registry (like checkov's
`--external-checks-dir`) was considered and rejected: Nova has 1 active
engine today (kyverno-json) and at most 2 in the foreseeable future
(kyverno-json + OPA). A `Protocol` + `dict` registry in
`core/policy_engine.py` is the right weight — discoverable, typed,
testable, and ~40 lines. An entry-point registry adds packaging
complexity (entry-point metadata, version resolution) for no gain at
this scale. The `register(name, factory)` method on the registry is
the extension point if a future milestone needs runtime plugin
discovery.

## 5. Latency / MTTR impact (G-Q3 anticipation)

NORTH_STAR.md MTTR target: < 60s p95. `run_platform.sh` Step 5 today
runs Checkov over the terraform plan (typically 5-15s for a small
stack). Adding `kj scan` over the same plan JSON adds:
- Process spawn: ~50ms (Go binary startup)
- Policy load: ~20ms (a handful of YAML files)
- Assertion evaluation: ~100-500ms (JMESPath over a small plan)
- Total: < 1s for a typical Nova stack

The kyverno-json pass runs **in parallel** with Checkov (REQ-301 — the
script launches both and waits on both), so the wall-clock impact is
`max(checkov_time, kj_time)` ≈ checkov_time (kj is faster). The
contract + stack-IR policies run during resolve (already a fast step).
Meta-policies run over the merged list (in-memory, < 10ms). **No
measurable MTTR impact** is expected. This will be verified in P3
VERIFY with a timing assertion.

## 6. "Platform functions without AI" tenet (G-Q1 / D-120)

kyverno-json is deterministic (same policy + payload → same result,
every run). It is not an LLM, not a probabilistic model, not a
"judgement" engine. The NORTH_STAR.md tenet ("the platform functions
without AI — 'AI decisions' are really automated decisions")
distinguishes AI (non-reproducible) from automation (reproducible).
kyverno-json is the latter. Adding it is **more** aligned with the
tenet than the current imperative Python in `core/env_transition.py`
and `core/regression_verify.py`, because the policy is declarative
(visible, auditable, version-controlled) rather than imperative (logic
hidden in function bodies). The `is_configured()` guard ensures the
platform functions without the binary (graceful skip → `SKIPPED` PCR
→ confidence signal proceeds).

## 7. ECS policy catalog overlap (prior art)

The kyverno-json catalog ships ECS policies that overlap with Nova's
L1 modules: `ecs-cluster-enable-logging`, `ecs-cluster-required-
container-insights`, `ecs-service-public-ip`, `ecs-service-required-
latest-platform-fargate`, `ecs-task-definition-fs-read-only`. These are
**reference policies**, not drop-in Nova policies — they target the
AWS ECS API shape (`type: aws_ecs_service` etc.), not Nova's Stack IR
shape. v1.25 policies target the Nova IR (REQ-297) and the terraform
plan JSON (REQ-300), not the raw AWS API. The catalog is useful as
prior art for JMESPath patterns over ECS resources — the
`ecs-service-public-ip` policy's `contains('$allowed-values',
@.assign_public_ip)` pattern informs the Nova `forbid-public-ingress`
policy shape. No catalog policies are imported directly in v1.25.

## 8. Risks & mitigations

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| `kj` binary not in CI image | medium | blocks P3+ tests | `is_configured()` guard + `pytest.skip` + `scripts/install-kyverno-json.sh` |
| kyverno-json output shape changes across versions | low | breaks `_to_pcr()` | pin `@latest` to a known-good commit in `install-kyverno-json.sh` after P1 smoke; defensive parsing (malformed → `error` PCR, not exception) |
| Policy explosion (4 targets × N rules) | medium | maintenance load | wave ordering (PLAN); policies co-located per target dir; meta-policy cross-check keeps the set auditable |
| Checkov + kj tagging-rule drift | medium | false `error` PCRs | `tagging-rules-agree` meta-policy emits `error` on divergence (visible, not silent); the Checkov rule stays source of truth for HCL, kj for IR |
| OPA swap turns out harder than the protocol implies | low | future milestone rework | RESEARCH §4.2 documents the OPA-equivalent surface; the protocol is the contract, not the implementation |
| `--pre-process` needed for meta-policies but undocumented behavior | low | meta-policy bugs | v1.25 meta-policies use plain assertion trees over the PCR list (no pre-process); `--pre-process` noted as a future optimization only |

## 9. Assumptions (logged, full autonomy)

- A1: `kj scan --output json` produces a stable `results[]` array shape.
  Will be verified in P1 smoke test (`_smoke.json` policy + a trivial
  payload); if the shape differs, `_to_pcr()` is adjusted defensively
  (malformed → `error` PCR). Confidence: 0.85.
- A2: The `nova.cloudinit.dev/severity` annotation convention is
  read by the engine from the policy YAML (loaded once per evaluate()
  call). kyverno-json does not validate unknown annotations — they pass
  through. Confidence: 0.90.
- A3: The `~` projection modifier iterates `resources[]` in the Stack
  IR and `planned_values.root_module.resources[]` in the plan JSON
  correctly. Verified in P2/P3 tests. Confidence: 0.85.
- A4: `go install` works in the CI image (Go toolchain available or
  installable). If not, the binary-release download path is the
  documented fallback in `install-kyverno-json.sh`. Confidence: 0.80.
- A5: The `NullEngine` fallback (when `policy` key absent in
  config.json) keeps all existing tests passing — they don't set the
  key, so they get `NullEngine` → `SKIPPED` PCRs → confidence signal
  proceeds with `policy` input `[SKIPPED]` → per-input score 1.0
  (skipped counts as pass in `_per_input_score`). Confidence: 0.95
  (verified against `confidence_signal.py:84-89`).

## 10. Decisions referenced

D-115 (install path), D-116 (engine enum reuse), D-117 (adapter
signatures unchanged), D-118 (tagging cross-check), D-119 (critical-
override defense-in-depth), D-120 (deterministic not AI). See
CLARIFY.md for the full resolution text.

## 11. Architecture updates (deferred to RESEARCH-stage file edits)

- `.ciagent/ARCHITECTURE.md` gains §12.7 "Policy Engine Registry" with
  the registry diagram. Deferred to the RESEARCH commit (this file's
  commit) — the section is authored as part of this research.
- `schemas/README.md` notes `engine: "kyverno"` is shared by the K8s
  adapter and kyverno-json (distinguished by `ruleId` prefix).
- `modules/STANDARDS.md` gains a "Policy authoring standard" section
  (P4, REQ-307).
- `docs/METRICS.md` notes the policy engine is swappable (P4, REQ-307).