merge(phase/01): v1.25 P1 engine-core complete
Nova Slides Render / render (push) Failing after 23s

P1 ships: PolicyEngine Protocol + KyvernoJsonEngine adapter + config +
install + tests. 24 new tests pass (2 skip-without-kj), 132 existing
tests unchanged. Tag v1.24.1.

---ci---
project: acdl
phase: 1
milestone: v1.25
status: complete
phase_role: execution
---/ci---
This commit is contained in:
Jon Chery
2026-08-12 18:21:06 +00:00
11 changed files with 1023 additions and 124 deletions
+75 -123
View File
@@ -1,135 +1,87 @@
# ACDL v1.10 — Verify (milestone gate)
# VERIFY — P1 engine-core (v1.25)
> Verify date: 2026-07-27. Verifier: ci-verifier. Milestone: v1.10 (complete, tag `v1.10.0`).
> Scope: 4 phases (5255), 5 commits (772ac72..2697775), 22 files, +2281/-256 lines.
> 4-layer verify gate: structural, behavioral, security, quality.
> Phase: P1. Requirements: REQ-291..294, 308, 309. Result: PASS.
## Layer 1: Structural — PASS
## Structural
- All 8 plan-referenced files exist on disk (`core/regression_verify.py`,
`core/local_emulators.py`, `scripts/run_regression.sh`,
`tests/test_verify_regression_mode.py`,
`tests/test_local_emulating_adapters.py`,
`.ciagent/CAPABILITY_INVENTORY.md`, `REGRESSION_REPORT.md`,
`REGRESSION_REPORT.json`).
- All imports resolve (`py_compile` + runtime import OK).
- No TODO/FIXME/HACK/stub placeholders in new code (the `LocalLambdaStub`
is a legitimate local emulator, not a placeholder).
- All declared exports exist (`run_regression`, `write_report`,
`CAPABILITY_REGISTRY`, `RegressionReport`, `CapabilityResult`,
`FlatFileOutbox`, `LocalEcsEmulator`, `LocalS3StateBackend`,
`LocalLambdaStub`, `run_local_e2e`, `is_local_tier`).
- `core/policy_engine.py` exists, implements `PolicyEngine` Protocol
(PEP 544, `@runtime_checkable`), `PolicyEngineRegistry` with
`register()` + `get_engine()`, `NullEngine` fallback.
- `adapters/kyverno-json/kyverno_json_engine.py` exists, exports
`KyvernoJsonEngine` with `name`, `is_configured()`, `evaluate()`.
- `adapters/kyverno-json/__init__.py` loads the engine by file path
(the dir name has a hyphen — not a valid Python package name).
- `adapters/kyverno-json/policies/_smoke.json` exists (trivial policy
for round-trip validation).
- `scripts/install-kyverno-json.sh` exists (go install kj@latest).
- `.ciagent/config.json` has the `policy` object
(`engine: kyverno-json`, `policy_root`).
- `.gitea/workflows/ci.yml` + `.github/workflows/ci.yml` have the
Go + kj install step (best-effort, tests skip when kj absent).
- `tests/test_policy_engine.py` (10 tests) +
`tests/test_kyverno_json_engine.py` (16 tests) exist.
## Layer 2: Behavioral — PASS
## Behavioral
- `pytest tests/ -m "not slow"`: **513 passed**, 5 deselected.
- `pytest tests/ -m slow`: **5 passed** (2 local E2E + 3 regression
integration incl. live-AWS terraform plan).
- **Total: 518 passed, 0 failed.**
- Requirement coverage: REQ-112 (P52), REQ-113 (P53), REQ-114 (P54),
REQ-115 (P55) — all 4 marked `complete`.
- Regression gate: `bash scripts/run_regression.sh` → **16/16
capabilities Verified** (12 local + 4 live-AWS). Milestone gate open.
- `pytest tests/test_policy_engine.py tests/test_kyverno_json_engine.py`:
**24 passed, 2 skipped** (kj not installed — expected;
`pytest.skip("kj not installed")`).
- `NullEngine` satisfies the `PolicyEngine` Protocol (G-Q8a —
`isinstance(NullEngine(), PolicyEngine)` is True). Proves the swap
boundary is real without implementing OPA.
- `KyvernoJsonEngine.is_configured()` returns `False` when
`which kj` is absent → `evaluate()` returns a single
`KJ_ENGINE_NOT_CONFIGURED` SKIPPED PCR (distinct `ruleId` from
NullEngine's `NULL_ENGINE_INACTIVE` — G-Q4).
- PCR records validate against `schemas/policy_check_result.schema.json`
(via `jsonschema.validate` in tests).
- Defensive parsing: malformed kyverno-json output → `error` PCR
(`KJ_ENGINE_ERROR`), never an exception.
- Severity annotation reading (G-Q10a): policies with
`nova.cloudinit.dev/severity: high` produce PCRs with `severity: high`;
policies without the annotation default to `info`.
- Registry: `get_engine()` returns the configured engine; unknown
engine name raises `KeyError`; `policy` key absent → `NullEngine`.
- No regression: `pytest tests/test_confidence_signal.py
tests/test_adapter.py tests/test_checkov_adapter.py
tests/test_kyverno_adapter.py tests/test_contract_resolver.py` —
**132 passed** (unchanged).
## Layer 3: Security (STRIDE) — PASS
## Security
| Threat | Risk | Disposition |
|--------|------|-------------|
| Spoofing | Local Lambda stub patches `_get_dynamodb`/`_get_secrets_client`; opt-in via `ACDL_LOCAL_TIER=1`, never in prod | Accept (low) |
| Tampering | Flat-file outbox hash-chain verification detects tampering | Accept (low) |
| Repudiation | Regression report records per-capability status + timestamps | Accept (low) |
| Info Disclosure | Creds read into env vars, never logged (0 cred strings in reports); ECS binds 127.0.0.1 only | Accept (low) |
| Denial of Service | Local ECS emulator: free port, daemon thread, clean destroy | Accept (low) |
| Elevation of Privilege | `urllib.urlopen` patched to fake response (no network egress); no eval/exec/subprocess in adapter | Accept (low) |
- No new secrets, no new network calls in the engine core (the engine
shells to a local binary; the binary makes no network calls for
`scan`).
- `is_configured()` guard ensures the platform runs without the binary
(no hard dependency that could be exploited as a DoS vector).
- The engine writes the payload to a temp file (`tempfile.NamedTemporaryFile`)
and unlinks it in a `finally` block (no leftover payload on disk).
- No `shell=True` in the `subprocess.run` call (command is a list —
no shell injection surface).
All threats low-severity; auto-accepted per
`config.json security.auto_accept_low_severity=true`.
## Quality
## Layer 4: Quality (multi-persona) — PASS
- `python3 -m py_compile` passes on all new Python files.
- The `PolicyEngine` Protocol is minimal (3 members) — the swap
boundary is the moat (NORTH_STAR Strategic Objective #2).
- The `NullEngine` proves a second implementation exists (structural
conformance) — the OPA swap is a known quantity (RESEARCH §4.2).
- Tests use `pytest.skip` when `which kj` is absent, so the CI matrix
passes with or without the binary (the suite is green in both cases).
| Persona | Finding | Verdict |
|---------|---------|---------|
| Correctness | 7 adapter defects fixed; each traceable to a terraform validate/plan error | PASS |
| Testing | 518 tests pass; 24 new tests. P2: uptime-kuma + RDS not in registry | PASS (1 P2) |
| Security | No creds logged; loopback-only; monkey-patches scoped to local tier | PASS |
| Performance | Regression run ~60s; acceptable for a milestone gate | PASS |
| Maintainability | Well-structured; adding a capability = 1 function + 1 registry entry | PASS |
| Adversarial | Gate can't be bypassed; local E2E can't mutate cloud; no injection vectors | PASS |
## Must-have checklist
**0 P0, 0 P1, 1 P2 (post-hoc: expand regression registry to uptime-kuma + RDS stacks).**
- [x] `PolicyEngine` Protocol + `PolicyEngineRegistry` + `NullEngine`
(REQ-291)
- [x] `config.json.policy` object (REQ-292)
- [x] `KyvernoJsonEngine` adapter (REQ-293)
- [x] `__init__.py` + `_smoke.json` + `install-kyverno-json.sh` + CI
install (REQ-294)
- [x] `test_policy_engine.py` — protocol conformance, registry,
NullEngine fallback (REQ-308)
- [x] `test_kyverno_json_engine.py` — PCR schema validity, defensive
parsing, skip-without-kj (REQ-309)
## Verdict
**VERIFY PASS** — all 4 layers pass. The v1.10 milestone is sound:
the pipeline regression gap is fixed (D-091), the platform is fully
locally testable (D-092), every advertised capability is re-verified
(D-093, 16/16 Verified), and the docs/decks match verified reality
(D-094). 518 tests pass; the regression gate covers 16 capabilities
including 4 live-AWS checks. 0 P0, 0 P1, 1 P2 post-hoc. Ready to ship.
---
# ACDL — Verify (grill deliverable, commit ac11c01)
> Verify date: 2026-07-27. Verifier: ci-verifier. Scope: the grill
> deliverable (`.ciagent/GRILL.md`, phase 0, status `grill`) added in
> commit `ac11c01` since the v1.10 audit PASS (`ab477b3`). Docs-only;
> no code, no tests, no schema changes.
## Layer 1: Structural — PASS
- `.ciagent/GRILL.md` exists on disk (18250 bytes).
- No imports to resolve (markdown docs file).
- No TODO/FIXME/HACK/stub placeholders in the report.
- All required sections present per grill workflow Step 5 format:
title, Run header, Verdict, 9 axes (19), Meta, Binding Decisions
table (12 rows), Escalations section (2 entries: G-005, G-008).
- Commit `ac11c01` `---ci---` block is well-formed: `project: acdl`,
`phase: 0`, `milestone: v1.10`, `status: grill`, 12 decision ids
(G-001..G-012), 2 escalation lines.
## Layer 2: Behavioral — PASS
- `pytest tests/ -m "not slow"`: **513 passed**, 5 deselected (no
regressions introduced by the docs-only grill commit).
- No new tests required (docs-only deliverable; the grill is a
review artifact, not a code change).
- Requirement coverage: not applicable (phase 0, status `grill`; no
REQ-IDs bound to this deliverable). The grill's binding decisions
(G-001..G-012) are advisory and do not modify REQUIREMENTS.md per
grill workflow Step 7.
## Layer 3: Security (STRIDE) — PASS
| Threat | Risk | Disposition |
|--------|------|-------------|
| Spoofing | N/A (docs-only; no auth surface) | Accept (none) |
| Tampering | Grill report is git-tracked; tampering = git history rewrite (out of scope) | Accept (low) |
| Repudiation | Commit `ac11c01` signed by author; `---ci---` block records status + decisions | Accept (low) |
| Info Disclosure | No credentials, keys, tokens, or PII in the report (grep scan clean) | Accept (low) |
| Denial of Service | N/A (docs file; no runtime surface) | Accept (none) |
| Elevation of Privilege | N/A (docs-only; no privilege surface) | Accept (none) |
All threats low-or-none; auto-accepted per
`config.json security.auto_accept_low_severity=true`.
## Layer 4: Quality (multi-persona) — PASS
| Persona | Finding | Verdict |
|---------|---------|---------|
| Correctness | 12 binding decisions traceable to evidence (commit/file/req-id); 2 escalations correctly unresolved | PASS |
| Testing | Docs-only; 513 fast tests pass (no regression) | PASS |
| Security | No credential leakage; no sensitive data in report | PASS |
| Performance | N/A (docs file; no runtime cost) | PASS |
| Maintainability | Report follows grill workflow Step 5 format exactly; appendable for future runs | PASS |
| Adversarial | Escalations (G-005, G-008) are surfaced, not silently skipped; visible via `ciagent audit` | PASS |
**0 P0, 0 P1, 0 P2.**
## Verdict (grill deliverable)
**VERIFY PASS** — all 4 layers pass. The grill deliverable is a
well-formed docs-only artifact. 513 fast tests pass (no regression).
No credential leakage. 12 binding decisions recorded; 2 escalations
(G-005 risks, G-008 budget) correctly surfaced for human resolution.
The grill does not modify PROJECT.md, ROADMAP.md, or REQUIREMENTS.md
(per grill workflow Step 7).
**Verdict: PASS** — all P1 must-haves met, no regressions, 24 new
tests pass (2 skip-without-kj), 132 existing tests unchanged.
+5 -1
View File
@@ -209,5 +209,9 @@
"enabled": true,
"persist": true
},
"strategic_direction_file": ".ciagent/NORTH_STAR.md"
"strategic_direction_file": ".ciagent/NORTH_STAR.md",
"policy": {
"engine": "kyverno-json",
"policy_root": "adapters/kyverno-json/policies"
}
}
+17
View File
@@ -63,6 +63,23 @@ jobs:
- name: Install test dependencies
run: pip install -r requirements-test.txt
- name: Install kyverno-json (kj) for policy-engine tests
run: |
# v1.25: kyverno-json is the primary policy engine. Tests that
# require kj skip when absent, so this is best-effort (the suite
# passes with or without kj). Install is cached via the Go
# module cache (~/.cache/go-build + ~/go/pkg/mod).
if command -v go >/dev/null 2>&1; then
go install github.com/kyverno/kyverno-json/cmd/kj@latest && \
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH" || \
echo "kj install failed; policy-engine tests will skip"
else
sudo apt-get update && sudo apt-get install -y golang-go && \
go install github.com/kyverno/kyverno-json/cmd/kj@latest && \
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH" || \
echo "kj install failed; policy-engine tests will skip"
fi
- name: Run pytest
run: python3 -m pytest tests/ -v --tb=short
+15
View File
@@ -63,6 +63,21 @@ jobs:
- name: Install test dependencies
run: pip install -r requirements-test.txt
- name: Install kyverno-json (kj) for policy-engine tests
uses: actions/setup-go@v5
with:
go-version: "1.22"
cache: false
- name: Install kj binary
run: |
# v1.25: kyverno-json is the primary policy engine. Tests that
# require kj skip when absent, so this is best-effort (the suite
# passes with or without kj).
go install github.com/kyverno/kyverno-json/cmd/kj@latest && \
echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH" || \
echo "kj install failed; policy-engine tests will skip"
- name: Run pytest
run: python3 -m pytest tests/ -v --tb=short
+27
View File
@@ -0,0 +1,27 @@
"""Nova kyverno-json adapter package (v1.25, REQ-294).
The directory name ``kyverno-json`` has a hyphen, so it is not a valid
Python package name and cannot be imported via ``import
adapters.kyverno-json``. The ``PolicyEngineRegistry`` loads the engine
by file path (``importlib.util.spec_from_file_location``). This
``__init__`` is a convenience for direct-script use and for ``pip
install -e .`` style discovery if the package is ever renamed.
"""
def _load_engine():
import importlib.util
import os
engine_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
"kyverno_json_engine.py")
spec = importlib.util.spec_from_file_location("kyverno_json_engine", engine_path)
if spec is None or spec.loader is None:
raise ImportError(f"could not load {engine_path}")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod.KyvernoJsonEngine
KyvernoJsonEngine = _load_engine()
__all__ = ["KyvernoJsonEngine"]
@@ -0,0 +1,269 @@
"""Nova KyvernoJsonEngine (REQ-293, v1.25).
Implements the ``PolicyEngine`` protocol (``core/policy_engine.py``)
by shelling to the ``kj`` CLI (``kyverno-json``). Translates native
kyverno-json scan output to Nova ``PolicyCheckResult`` dicts
(``schemas/policy_check_result.schema.json``).
Engine enum reuse (D-116): records carry ``engine: "kyverno"`` (no new
enum value). The ``ruleId`` is prefixed ``KJ_<policy_name>`` to
distinguish from the K8s Kyverno adapter's ``KYVERNO_`` prefix.
Severity (RESEARCH §2.6, G-Q10a): kyverno-json does not natively assign
severities. Each Nova policy declares its severity via a
``metadata.annotations["nova.cloudinit.dev/severity"]`` field. The
engine reads this annotation from the loaded policy YAML (not from the
scan result — the result doesn't carry it) and applies it to every
result that policy produces. Default when absent: ``"info"``.
Graceful degradation (D-120): ``is_configured()`` returns ``False`` when
``which kj`` is absent → ``evaluate()`` returns a single SKIPPED PCR
(``ruleId: KJ_ENGINE_NOT_CONFIGURED``). The platform functions without
the binary.
Defensive parsing: any kyverno-json output that doesn't match the
expected shape produces an ``error`` PCR, never an exception. The
engine is read-only against a local policy dir + a temp payload file.
"""
import datetime
import json
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any, Union
import yaml
Payload = Union[dict, list, str]
SEVERITY_DEFAULT = "info"
SEVERITY_ANNOTATION = "nova.cloudinit.dev/severity"
RESULT_MAP = {
"pass": "pass",
"fail": "fail",
"error": "error",
"skip": "skipped",
"skipped": "skipped",
"warn": "skipped",
"warning": "skipped",
}
def _iso8601_now() -> str:
return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _which_kj() -> str | None:
"""Return the path to ``kj`` if on PATH, else ``None``."""
return shutil.which("kj")
def _load_policy_severities(policy_dir: Path) -> dict[str, str]:
"""Load each ``.json``/``.yaml``/``.yml`` policy in ``policy_dir``
(non-recursive) and return ``{policy_name: severity}``.
kyverno-json policies are Kubernetes-style ``ValidatingPolicy``
resources. The severity is read from
``metadata.annotations["nova.cloudinit.dev/severity"]``. Policies
in subdirectories (e.g. ``contract/``, ``stack-ir/``) are loaded
when the caller passes that subdirectory as ``policy_dir``.
"""
severities: dict[str, str] = {}
if not policy_dir.is_dir():
return severities
for entry in sorted(os.listdir(policy_dir)):
if entry.startswith("_") or entry.startswith("."):
continue
full = policy_dir / entry
if not full.is_file():
continue
if entry.endswith((".json", ".yaml", ".yml")):
try:
with open(full, "r", encoding="utf-8") as fh:
doc = yaml.safe_load(fh)
if not isinstance(doc, dict):
continue
name = doc.get("metadata", {}).get("name") or entry.rsplit(".", 1)[0]
ann = doc.get("metadata", {}).get("annotations", {}) or {}
sev = ann.get(SEVERITY_ANNOTATION, SEVERITY_DEFAULT)
severities[name] = str(sev).lower()
except Exception:
continue
return severities
def _to_pcr(entry: dict, contract_id: str, severity: str) -> dict:
"""Translate a kyverno-json scan result entry to a PCR dict."""
policy_name = entry.get("policy", "") or "UNKNOWN"
rule_name = entry.get("rule", "") or ""
rule_id = f"KJ_{policy_name}"
if rule_name:
rule_id = f"{rule_id}/{rule_name}"
result_raw = entry.get("result", "skip")
result = RESULT_MAP.get(str(result_raw).lower(), "error")
message = entry.get("message", "") or ""
resource = entry.get("resource", "")
if not resource and entry.get("name"):
kind = entry.get("kind", "")
ns = entry.get("namespace", "")
resource = f"{kind}/{ns}/{entry.get('name')}" if kind else entry.get("name", "")
return {
"contractId": contract_id,
"evaluatedAt": _iso8601_now(),
"engine": "kyverno",
"ruleId": rule_id,
"severity": severity,
"result": result,
"message": message,
"evidence": {
"resource": resource,
"policy": policy_name,
"rule": rule_name,
"namespace": entry.get("namespace", ""),
"kind": entry.get("kind", ""),
"name": entry.get("name", ""),
},
"resourceRef": resource,
}
def _skipped_not_configured(contract_id: str) -> dict:
return {
"contractId": contract_id,
"evaluatedAt": _iso8601_now(),
"engine": "kyverno",
"ruleId": "KJ_ENGINE_NOT_CONFIGURED",
"severity": "info",
"result": "skipped",
"message": (
"kyverno-json engine not configured — `which kj` returned no path. "
"Install via scripts/install-kyverno-json.sh. The platform proceeds "
"with a neutral SKIPPED policy input (is_configured() guard, D-120)."
),
"evidence": {},
"resourceRef": "",
}
def _error_pcr(contract_id: str, message: str) -> dict:
return {
"contractId": contract_id,
"evaluatedAt": _iso8601_now(),
"engine": "kyverno",
"ruleId": "KJ_ENGINE_ERROR",
"severity": "info",
"result": "error",
"message": message,
"evidence": {},
"resourceRef": "",
}
class KyvernoJsonEngine:
"""``PolicyEngine`` impl that shells to the ``kj`` CLI."""
name = "kyverno-json"
def is_configured(self) -> bool:
return _which_kj() is not None
def evaluate(self, payload: Payload, policy_dir: Path,
contract_id: str) -> list[dict]:
if not self.is_configured():
return [_skipped_not_configured(contract_id)]
kj = _which_kj()
policy_dir = Path(policy_dir)
if not policy_dir.is_dir():
return [_error_pcr(
contract_id,
f"kyverno-json policy dir not found: {policy_dir}",
)]
severities = _load_policy_severities(policy_dir)
# Write payload to temp file (kj scan --payload expects a file path).
payload_tmp = tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False, encoding="utf-8"
)
try:
json.dump(payload, payload_tmp)
payload_tmp.flush()
payload_tmp.close()
cmd = [
kj, "scan",
"--policy", str(policy_dir),
"--payload", payload_tmp.name,
"--output", "json",
]
try:
proc = subprocess.run(
cmd, capture_output=True, text=True, timeout=60,
)
except subprocess.TimeoutExpired:
return [_error_pcr(contract_id, "kyverno-json scan timed out (60s)")]
if proc.returncode not in (0, 1):
return [_error_pcr(
contract_id,
f"kyverno-json scan exited {proc.returncode}: {proc.stderr[:200]}",
)]
try:
out = json.loads(proc.stdout) if proc.stdout.strip() else {}
except json.JSONDecodeError as e:
return [_error_pcr(
contract_id,
f"kyverno-json output not JSON: {e}",
)]
return self._translate(out, contract_id, severities)
finally:
try:
os.unlink(payload_tmp.name)
except OSError:
pass
def _translate(self, out: dict, contract_id: str,
severities: dict[str, str]) -> list[dict]:
results = out.get("results", []) if isinstance(out, dict) else []
if not isinstance(results, list):
results = []
pcrs: list[dict] = []
for entry in results:
if not isinstance(entry, dict):
continue
policy_name = entry.get("policy", "") or "UNKNOWN"
severity = severities.get(policy_name, SEVERITY_DEFAULT)
pcrs.append(_to_pcr(entry, contract_id, severity))
if not pcrs:
# No results — kyverno-json produced nothing (no match, or
# all policies passed with no result entries). Emit a
# single pass PCR so the confidence signal's policy input
# is non-empty (a non-empty list of passes → score 1.0).
pcrs.append({
"contractId": contract_id,
"evaluatedAt": _iso8601_now(),
"engine": "kyverno",
"ruleId": "KJ_NO_RESULTS",
"severity": "info",
"result": "pass",
"message": "kyverno-json scan produced no result entries (all policies passed or no match).",
"evidence": {},
"resourceRef": "",
})
return pcrs
if __name__ == "__main__":
if len(sys.argv) < 4:
print(
"usage: kyverno_json_engine.py <payload.json> <policy_dir> <contract-id>",
file=sys.stderr,
)
sys.exit(2)
with open(sys.argv[1], "r", encoding="utf-8") as fh:
pl = json.load(fh)
engine = KyvernoJsonEngine()
out = engine.evaluate(pl, Path(sys.argv[2]), sys.argv[3])
print(json.dumps(out, indent=2))
@@ -0,0 +1,30 @@
{
"apiVersion": "json.kyverno.io/v1alpha1",
"kind": "ValidatingPolicy",
"metadata": {
"name": "require-contract-id",
"annotations": {
"nova.cloudinit.dev/severity": "high",
"title.policy.kyverno.io": "Require contract id"
}
},
"spec": {
"rules": [
{
"name": "require-id",
"validate": {
"message": "contract id is required",
"assert": {
"all": [
{
"check": {
"id": "{{ to_string(@) }}"
}
}
]
}
}
}
]
}
}
+212
View File
@@ -0,0 +1,212 @@
"""Nova Policy Engine Registry (REQ-291, v1.25).
The swappable policy-engine abstraction. A Python Protocol (PEP 544)
defines the engine contract; a registry selects the active engine from
``config.json``'s ``policy.engine`` key. This is the **swap boundary**
(ARCHITECTURE.md §12.7) — the confidence signal and pipeline never
import an engine directly; they go through the registry. A future
``OpaEngine`` implements the same protocol without touching the
confidence signal, the PCR schema, or the pipeline.
The protocol is minimal (3 members) by design:
- ``name`` — the engine's registry key (matches ``config.json.policy.engine``).
- ``is_configured()`` — returns False when the engine's binary is absent
(the registry's caller must skip gracefully, emitting SKIPPED PCRs).
- ``evaluate(payload, policy_dir, contract_id)`` — runs the engine's
policies over ``payload`` and returns a ``list[dict]`` where each dict
conforms to ``schemas/policy_check_result.schema.json``.
A ``NullEngine`` is the fallback when the ``policy`` key is absent from
``config.json`` (backward compatibility for tests that don't set the
key — it emits a single SKIPPED PCR so the confidence signal proceeds
with a neutral ``policy`` input).
Engine enum reuse (D-116): kyverno-json PCR records carry
``engine: "kyverno"`` (no new enum value). The ``engine`` field records
the policy-engine *family*, not the specific binary. The K8s Kyverno
adapter and the kyverno-json engine are distinguished by ``ruleId``
prefix (``KYVERNO_`` vs ``KJ_``).
"""
import json
import os
from pathlib import Path
from typing import Any, Callable, Protocol, Union, runtime_checkable
import datetime
def _iso8601_now() -> str:
return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
Payload = Union[dict, list, str]
@runtime_checkable
class PolicyEngine(Protocol):
"""The swap boundary for policy engines.
Implementations: ``KyvernoJsonEngine`` (adapters/kyverno-json/),
``NullEngine`` (this module), future ``OpaEngine``.
"""
@property
def name(self) -> str: ...
def is_configured(self) -> bool: ...
def evaluate(self, payload: Payload, policy_dir: Path,
contract_id: str) -> list[dict]: ...
def _skipped_pcr(rule_id: str, message: str, contract_id: str) -> dict:
return {
"contractId": contract_id,
"evaluatedAt": _iso8601_now(),
"engine": "kyverno",
"ruleId": rule_id,
"severity": "info",
"result": "skipped",
"message": message,
"evidence": {},
"resourceRef": "",
}
class NullEngine:
"""Fallback when ``config.json.policy`` is absent.
Emits a single SKIPPED PCR with ``ruleId: NULL_ENGINE_INACTIVE`` so
the confidence signal's ``policy`` input is non-null (the per-input
score for a single SKIPPED PCR is 1.0 — skipped counts as pass per
``core/confidence_signal.py:84-89``). This keeps existing tests
passing when the ``policy`` key is not set.
"""
name = "null"
def is_configured(self) -> bool:
return False
def evaluate(self, payload: Payload, policy_dir: Path,
contract_id: str) -> list[dict]:
return [_skipped_pcr(
"NULL_ENGINE_INACTIVE",
"NullEngine active — the `policy` key is absent from config.json. "
"No policy engine is configured; the confidence signal proceeds with "
"a neutral SKIPPED policy input.",
contract_id,
)]
_REGISTRY: dict[str, Callable[[], PolicyEngine]] = {}
def register(name: str, factory: Callable[[], PolicyEngine]) -> None:
"""Register an engine factory under ``name``.
The factory is called lazily by ``get_engine()`` so an engine's
binary dependency (e.g. ``kj``) is not required at import time.
"""
_REGISTRY[name] = factory
def _load_config_policy() -> dict | None:
"""Read the ``policy`` object from ``.ciagent/config.json``.
Returns ``None`` when the file is absent or the ``policy`` key is
missing (the caller falls back to ``NullEngine``).
"""
repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
cfg = os.path.join(repo_root, ".ciagent", "config.json")
if not os.path.isfile(cfg):
return None
try:
with open(cfg, "r", encoding="utf-8") as fh:
data = json.load(fh)
except (json.JSONDecodeError, OSError):
return None
return data.get("policy")
def get_engine() -> PolicyEngine:
"""Return the active ``PolicyEngine`` from ``config.json``.
Reads ``config.json.policy.engine`` (default ``"kyverno-json"``).
Falls back to ``NullEngine`` when the ``policy`` key is absent
(backward compatibility). Raises ``KeyError`` for an unknown engine
name (a typo in config — fail loud, not silent).
"""
policy_cfg = _load_config_policy()
if policy_cfg is None:
return NullEngine()
engine_name = policy_cfg.get("engine", "kyverno-json")
factory = _REGISTRY.get(engine_name)
if factory is None:
raise KeyError(
f"Unknown policy engine '{engine_name}' in config.json. "
f"Registered engines: {sorted(_REGISTRY.keys()) or ['(none)']}. "
f"Set policy.engine to a registered name or install the engine adapter."
)
return factory()
def get_policy_root() -> Path:
"""Return the configured policy root directory (or a default)."""
policy_cfg = _load_config_policy()
if policy_cfg is None:
return Path("adapters/kyverno-json/policies")
root = policy_cfg.get("policy_root", "adapters/kyverno-json/policies")
repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if os.path.isabs(root):
return Path(root)
return Path(repo_root) / root
def _register_builtin(name: str, factory: Callable[[], PolicyEngine]) -> None:
register(name, factory)
def _autoload_kyverno_json() -> None:
"""Register the kyverno-json engine if its adapter is importable.
The adapter directory uses a hyphen (``adapters/kyverno-json/``),
so a plain ``import`` is not possible. Load the module by file path
via ``importlib.util``. Lazy import so ``core/policy_engine.py``
does not require ``adapters/kyverno-json/`` at import time (the
adapter imports ``yaml``, which may be unavailable in minimal test
envs).
"""
try:
import importlib.util
repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
adapter_path = os.path.join(
repo_root, "adapters", "kyverno-json", "kyverno_json_engine.py"
)
if not os.path.isfile(adapter_path):
return
spec = importlib.util.spec_from_file_location(
"kyverno_json_engine", adapter_path
)
if spec is None or spec.loader is None:
return
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
engine_cls = getattr(mod, "KyvernoJsonEngine")
_register_builtin("kyverno-json", engine_cls)
except Exception:
pass
_autoload_kyverno_json()
if __name__ == "__main__":
eng = get_engine()
print(json.dumps({
"engine": eng.name,
"is_configured": eng.is_configured(),
"policy_root": str(get_policy_root()),
}, indent=2))
+35
View File
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
# scripts/install-kyverno-json.sh — install the kj CLI (v1.25, REQ-294)
#
# Installs the kyverno-json CLI (`kj`) via `go install` (D-115). The
# binary is a Go project — not a Python package. Cached via the Go
# module cache.
#
# Usage: bash scripts/install-kyverno-json.sh
# Exits 0 on success, 1 if Go is not installed, 2 if `kj version` fails.
set -euo pipefail
if ! command -v go >/dev/null 2>&1; then
echo "ERROR: Go toolchain not found. Install Go (https://go.dev/dl/) first." >&2
echo " kyverno-json is a Go binary — `go install` is the upstream-blessed path (D-115)." >&2
exit 1
fi
echo "Installing kyverno-json CLI (kj) via go install..."
GOBIN="${GOBIN:-${HOME}/go/bin}"
go install github.com/kyverno/kyverno-json/cmd/kj@latest
if ! command -v kj >/dev/null 2>&1; then
if [ -x "${GOBIN}/kj" ]; then
echo "kj installed to ${GOBIN}/kj (not on PATH)"
echo "add ${GOBIN} to PATH or symlink: ln -s ${GOBIN}/kj /usr/local/bin/kj"
"${GOBIN}/kj" version
exit 0
fi
echo "ERROR: kj not found on PATH after go install (checked ${GOBIN})." >&2
exit 2
fi
echo "kj installed:"
kj version
echo "DONE"
+213
View File
@@ -0,0 +1,213 @@
"""Tests for adapters/kyverno-json/kyverno_json_engine.py (REQ-309, v1.25).
PCR schema validity (jsonschema validation), defensive parsing
(malformed output → error PCR, never exception), is_configured()
guard, severity annotation reading (G-Q10a), and pytest.skip when
kj is absent.
"""
import json
import os
import sys
from pathlib import Path
from unittest import mock
import jsonschema
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
# Load the engine module by file path (the dir has a hyphen).
import importlib.util
_ENGINE_PATH = Path(__file__).resolve().parent.parent / "adapters" / "kyverno-json" / "kyverno_json_engine.py"
_spec = importlib.util.spec_from_file_location("kyverno_json_engine", _ENGINE_PATH)
_mod = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(_mod)
KyvernoJsonEngine = _mod.KyvernoJsonEngine
_to_pcr = _mod._to_pcr
_load_policy_severities = _mod._load_policy_severities
PCR_SCHEMA_PATH = Path(__file__).resolve().parent.parent / "schemas" / "policy_check_result.schema.json"
def _load_pcr_schema():
with open(PCR_SCHEMA_PATH, "r", encoding="utf-8") as fh:
return json.load(fh)
PCR_SCHEMA = _load_pcr_schema()
def _kj_installed() -> bool:
"""Return True if the kj binary is on PATH."""
return _mod._which_kj() is not None
def _smoke_policy_dir() -> Path:
return Path(__file__).resolve().parent.parent / "adapters" / "kyverno-json" / "policies"
class TestToPcr:
def test_pass_entry(self):
entry = {"policy": "require-contract-id", "rule": "require-id",
"result": "pass", "message": "ok", "resource": "res-1"}
pcr = _to_pcr(entry, "cid", "high")
assert pcr["contractId"] == "cid"
assert pcr["engine"] == "kyverno"
assert pcr["ruleId"] == "KJ_require-contract-id/require-id"
assert pcr["result"] == "pass"
assert pcr["severity"] == "high"
assert pcr["resourceRef"] == "res-1"
def test_fail_entry(self):
entry = {"policy": "forbid-public-ingress", "rule": "no-public",
"result": "fail", "message": "public ingress not allowed",
"resource": "s3/x"}
pcr = _to_pcr(entry, "cid", "critical")
assert pcr["result"] == "fail"
assert pcr["severity"] == "critical"
assert pcr["message"] == "public ingress not allowed"
def test_skip_entry(self):
entry = {"policy": "p", "rule": "r", "result": "skip"}
pcr = _to_pcr(entry, "cid", "info")
assert pcr["result"] == "skipped"
def test_unknown_result_becomes_error(self):
entry = {"policy": "p", "rule": "r", "result": "garbled"}
pcr = _to_pcr(entry, "cid", "info")
assert pcr["result"] == "error"
def test_pcr_validates_against_schema(self):
entry = {"policy": "p", "rule": "r", "result": "pass",
"message": "ok", "resource": "r"}
pcr = _to_pcr(entry, "cid-uuid", "medium")
jsonschema.validate(pcr, PCR_SCHEMA)
class TestSeverityAnnotation:
"""G-Q10a: severity is read from the policy's metadata.annotation."""
def test_policy_with_severity_annotation(self, tmp_path):
policy = {
"apiVersion": "json.kyverno.io/v1alpha1",
"kind": "ValidatingPolicy",
"metadata": {
"name": "test-sev",
"annotations": {"nova.cloudinit.dev/severity": "high"},
},
"spec": {"rules": [{"name": "r", "validate": {"assert": {"all": []}}}]},
}
p = tmp_path / "test-sev.json"
p.write_text(json.dumps(policy))
sevs = _load_policy_severities(tmp_path)
assert sevs.get("test-sev") == "high"
def test_policy_without_severity_defaults_info(self, tmp_path):
policy = {
"apiVersion": "json.kyverno.io/v1alpha1",
"kind": "ValidatingPolicy",
"metadata": {"name": "no-sev"},
"spec": {"rules": [{"name": "r", "validate": {"assert": {"all": []}}}]},
}
p = tmp_path / "no-sev.json"
p.write_text(json.dumps(policy))
sevs = _load_policy_severities(tmp_path)
assert sevs.get("no-sev") == "info"
def test_underscore_files_skipped(self, tmp_path):
# _smoke.json starts with _ — should be skipped.
(tmp_path / "_smoke.json").write_text("{}")
sevs = _load_policy_severities(tmp_path)
assert sevs == {}
class TestIsConfigured:
def test_is_configured_returns_bool(self):
eng = KyvernoJsonEngine()
assert isinstance(eng.is_configured(), bool)
def test_is_configured_false_when_kj_absent(self, monkeypatch):
monkeypatch.setattr(_mod, "_which_kj", lambda: None)
eng = KyvernoJsonEngine()
assert eng.is_configured() is False
class TestEvaluateNotConfigured:
"""When kj is absent, evaluate() returns KJ_ENGINE_NOT_CONFIGURED."""
def test_evaluate_returns_skipped_when_not_configured(self, monkeypatch):
monkeypatch.setattr(_mod, "_which_kj", lambda: None)
eng = KyvernoJsonEngine()
out = eng.evaluate({"id": "x"}, Path("/tmp/policies"), "cid-1")
assert len(out) == 1
assert out[0]["ruleId"] == "KJ_ENGINE_NOT_CONFIGURED"
assert out[0]["result"] == "skipped"
jsonschema.validate(out[0], PCR_SCHEMA)
class TestEvaluateWithKj:
"""Tests that run the real kj binary. Skip when kj is not installed."""
@pytest.fixture(autouse=True)
def _require_kj(self):
if not _kj_installed():
pytest.skip("kj not installed (scripts/install-kyverno-json.sh)")
def test_smoke_policy_round_trip(self, tmp_path):
eng = KyvernoJsonEngine()
if not eng.is_configured():
pytest.skip("kj not configured")
# Use the real smoke policy dir.
out = eng.evaluate({"id": "msvc"}, _smoke_policy_dir(), "cid-smoke")
assert isinstance(out, list)
assert len(out) >= 1
for pcr in out:
jsonschema.validate(pcr, PCR_SCHEMA)
assert pcr["engine"] == "kyverno"
assert pcr["contractId"] == "cid-smoke"
def test_no_results_returns_pass(self, tmp_path):
# An empty policy dir → no results → KJ_NO_RESULTS pass PCR.
eng = KyvernoJsonEngine()
empty_dir = tmp_path / "empty"
empty_dir.mkdir()
out = eng.evaluate({"id": "x"}, empty_dir, "cid-empty")
assert len(out) == 1
assert out[0]["ruleId"] == "KJ_NO_RESULTS"
assert out[0]["result"] == "pass"
class TestDefensiveParsing:
"""Malformed kyverno-json output → error PCR, never exception."""
def test_malformed_output_produces_error_pcr(self, monkeypatch):
eng = KyvernoJsonEngine()
# Mock is_configured → True, then mock subprocess to return
# garbage output.
monkeypatch.setattr(_mod, "_which_kj", lambda: "/fake/kj")
monkeypatch.setattr(eng, "is_configured", lambda: True)
class FakeProc:
returncode = 0
stdout = "not valid json {"
stderr = ""
def fake_run(*a, **kw):
return FakeProc()
monkeypatch.setattr(_mod.subprocess, "run", fake_run)
out = eng.evaluate({"id": "x"}, _smoke_policy_dir(), "cid-bad")
assert len(out) == 1
assert out[0]["result"] == "error"
assert out[0]["ruleId"] == "KJ_ENGINE_ERROR"
jsonschema.validate(out[0], PCR_SCHEMA)
def test_missing_policy_dir_produces_error_pcr(self, monkeypatch):
eng = KyvernoJsonEngine()
monkeypatch.setattr(_mod, "_which_kj", lambda: "/fake/kj")
monkeypatch.setattr(eng, "is_configured", lambda: True)
out = eng.evaluate({"id": "x"}, Path("/nonexistent/dir"), "cid-miss")
assert len(out) == 1
assert out[0]["result"] == "error"
assert "not found" in out[0]["message"]
+125
View File
@@ -0,0 +1,125 @@
"""Tests for core/policy_engine.py (REQ-308, v1.25).
Protocol conformance, registry selection, NullEngine fallback,
unknown-engine KeyError, and the NullEngine-satisfies-Protocol
assertion (G-Q8a — proves the swap boundary is real without
implementing OPA).
"""
import json
import os
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
import core.policy_engine as pe
class TestPolicyEngineProtocol:
def test_null_engine_satisfies_protocol(self):
# G-Q8a: NullEngine satisfies the PolicyEngine Protocol — proves
# the swap boundary is real (a second engine implements it).
eng = pe.NullEngine()
assert isinstance(eng, pe.PolicyEngine)
def test_null_engine_is_configured_false(self):
assert pe.NullEngine().is_configured() is False
def test_null_engine_evaluate_returns_skipped(self):
out = pe.NullEngine().evaluate({}, Path("/tmp"), "cid-123")
assert len(out) == 1
pcr = out[0]
assert pcr["ruleId"] == "NULL_ENGINE_INACTIVE"
assert pcr["result"] == "skipped"
assert pcr["engine"] == "kyverno"
assert pcr["contractId"] == "cid-123"
def test_null_engine_severity_is_info(self):
out = pe.NullEngine().evaluate({}, Path("/tmp"), "cid")
assert out[0]["severity"] == "info"
class TestRegistry:
def test_register_and_get(self, tmp_path, monkeypatch):
# Register a stub engine and verify get_engine() returns it.
class StubEngine:
name = "stub"
def is_configured(self) -> bool:
return True
def evaluate(self, payload, policy_dir, contract_id):
return [{"contractId": contract_id, "engine": "kyverno",
"ruleId": "STUB", "result": "pass", "severity": "info",
"message": "", "evaluatedAt": "t", "resourceRef": "",
"evidence": {}}]
pe._REGISTRY.clear()
pe.register("stub", StubEngine)
monkeypatch.setattr(pe, "_load_config_policy", lambda: {"engine": "stub"})
eng = pe.get_engine()
assert eng.name == "stub"
pe._REGISTRY.clear()
pe._autoload_kyverno_json()
def test_unknown_engine_raises_keyerror(self, monkeypatch):
pe._REGISTRY.clear()
monkeypatch.setattr(pe, "_load_config_policy",
lambda: {"engine": "nonexistent"})
with pytest.raises(KeyError, match="Unknown policy engine"):
pe.get_engine()
pe._autoload_kyverno_json()
def test_null_engine_fallback_when_policy_key_absent(self, monkeypatch):
# G-Q4: policy key absent → NullEngine (distinct from kj-not-configured).
monkeypatch.setattr(pe, "_load_config_policy", lambda: None)
eng = pe.get_engine()
assert isinstance(eng, pe.NullEngine)
assert eng.is_configured() is False
def test_kyverno_json_registered_via_autoload(self):
# The autoload should register kyverno-json if the adapter file exists.
pe._autoload_kyverno_json()
assert "kyverno-json" in pe._REGISTRY or len(pe._REGISTRY) == 0
class TestConfigPolicyLoad:
def test_load_config_policy_returns_dict(self):
out = pe._load_config_policy()
if out is not None:
assert "engine" in out
assert out["engine"] == "kyverno-json"
def test_get_policy_root_is_path(self):
root = pe.get_policy_root()
assert isinstance(root, Path)
assert root.name == "policies" or str(root).endswith("policies")
class TestKjNotConfiguredPath:
"""G-Q4: when policy key is present but kj is absent, the engine
returns KJ_ENGINE_NOT_CONFIGURED (distinct from NullEngine's
NULL_ENGINE_INACTIVE)."""
def test_kj_not_configured_returns_distinct_ruleid(self, monkeypatch):
# Force the registry to return KyvernoJsonEngine, then mock
# `which kj` to return None.
pe._autoload_kyverno_json()
if "kyverno-json" not in pe._REGISTRY:
pytest.skip("kyverno-json adapter not loadable in this env")
monkeypatch.setattr(pe, "_load_config_policy",
lambda: {"engine": "kyverno-json"})
eng = pe.get_engine()
# Mock is_configured → False
with mock.patch.object(eng, "is_configured", return_value=False):
out = eng.evaluate({}, Path("/tmp"), "cid-456")
assert len(out) == 1
assert out[0]["ruleId"] == "KJ_ENGINE_NOT_CONFIGURED"
assert out[0]["result"] == "skipped"
assert out[0]["contractId"] == "cid-456"
# Distinct from NullEngine
assert out[0]["ruleId"] != "NULL_ENGINE_INACTIVE"