diff --git a/.ciagent/VERIFY.md b/.ciagent/VERIFY.md index f6910dd..61e9e0a 100644 --- a/.ciagent/VERIFY.md +++ b/.ciagent/VERIFY.md @@ -1,135 +1,87 @@ -# ACDL v1.10 — Verify (milestone gate) +# VERIFY — P1 engine-core (v1.25) -> Verify date: 2026-07-27. Verifier: ci-verifier. Milestone: v1.10 (complete, tag `v1.10.0`). -> Scope: 4 phases (52–55), 5 commits (772ac72..2697775), 22 files, +2281/-256 lines. +> 4-layer verify gate: structural, behavioral, security, quality. +> Phase: P1. Requirements: REQ-291..294, 308, 309. Result: PASS. -## Layer 1: Structural — PASS +## Structural -- All 8 plan-referenced files exist on disk (`core/regression_verify.py`, - `core/local_emulators.py`, `scripts/run_regression.sh`, - `tests/test_verify_regression_mode.py`, - `tests/test_local_emulating_adapters.py`, - `.ciagent/CAPABILITY_INVENTORY.md`, `REGRESSION_REPORT.md`, - `REGRESSION_REPORT.json`). -- All imports resolve (`py_compile` + runtime import OK). -- No TODO/FIXME/HACK/stub placeholders in new code (the `LocalLambdaStub` - is a legitimate local emulator, not a placeholder). -- All declared exports exist (`run_regression`, `write_report`, - `CAPABILITY_REGISTRY`, `RegressionReport`, `CapabilityResult`, - `FlatFileOutbox`, `LocalEcsEmulator`, `LocalS3StateBackend`, - `LocalLambdaStub`, `run_local_e2e`, `is_local_tier`). +- `core/policy_engine.py` exists, implements `PolicyEngine` Protocol + (PEP 544, `@runtime_checkable`), `PolicyEngineRegistry` with + `register()` + `get_engine()`, `NullEngine` fallback. +- `adapters/kyverno-json/kyverno_json_engine.py` exists, exports + `KyvernoJsonEngine` with `name`, `is_configured()`, `evaluate()`. +- `adapters/kyverno-json/__init__.py` loads the engine by file path + (the dir name has a hyphen — not a valid Python package name). +- `adapters/kyverno-json/policies/_smoke.json` exists (trivial policy + for round-trip validation). +- `scripts/install-kyverno-json.sh` exists (go install kj@latest). +- `.ciagent/config.json` has the `policy` object + (`engine: kyverno-json`, `policy_root`). +- `.gitea/workflows/ci.yml` + `.github/workflows/ci.yml` have the + Go + kj install step (best-effort, tests skip when kj absent). +- `tests/test_policy_engine.py` (10 tests) + + `tests/test_kyverno_json_engine.py` (16 tests) exist. -## Layer 2: Behavioral — PASS +## Behavioral -- `pytest tests/ -m "not slow"`: **513 passed**, 5 deselected. -- `pytest tests/ -m slow`: **5 passed** (2 local E2E + 3 regression - integration incl. live-AWS terraform plan). -- **Total: 518 passed, 0 failed.** -- Requirement coverage: REQ-112 (P52), REQ-113 (P53), REQ-114 (P54), - REQ-115 (P55) — all 4 marked `complete`. -- Regression gate: `bash scripts/run_regression.sh` → **16/16 - capabilities Verified** (12 local + 4 live-AWS). Milestone gate open. +- `pytest tests/test_policy_engine.py tests/test_kyverno_json_engine.py`: + **24 passed, 2 skipped** (kj not installed — expected; + `pytest.skip("kj not installed")`). +- `NullEngine` satisfies the `PolicyEngine` Protocol (G-Q8a — + `isinstance(NullEngine(), PolicyEngine)` is True). Proves the swap + boundary is real without implementing OPA. +- `KyvernoJsonEngine.is_configured()` returns `False` when + `which kj` is absent → `evaluate()` returns a single + `KJ_ENGINE_NOT_CONFIGURED` SKIPPED PCR (distinct `ruleId` from + NullEngine's `NULL_ENGINE_INACTIVE` — G-Q4). +- PCR records validate against `schemas/policy_check_result.schema.json` + (via `jsonschema.validate` in tests). +- Defensive parsing: malformed kyverno-json output → `error` PCR + (`KJ_ENGINE_ERROR`), never an exception. +- Severity annotation reading (G-Q10a): policies with + `nova.cloudinit.dev/severity: high` produce PCRs with `severity: high`; + policies without the annotation default to `info`. +- Registry: `get_engine()` returns the configured engine; unknown + engine name raises `KeyError`; `policy` key absent → `NullEngine`. +- No regression: `pytest tests/test_confidence_signal.py + tests/test_adapter.py tests/test_checkov_adapter.py + tests/test_kyverno_adapter.py tests/test_contract_resolver.py` — + **132 passed** (unchanged). -## Layer 3: Security (STRIDE) — PASS +## Security -| Threat | Risk | Disposition | -|--------|------|-------------| -| Spoofing | Local Lambda stub patches `_get_dynamodb`/`_get_secrets_client`; opt-in via `ACDL_LOCAL_TIER=1`, never in prod | Accept (low) | -| Tampering | Flat-file outbox hash-chain verification detects tampering | Accept (low) | -| Repudiation | Regression report records per-capability status + timestamps | Accept (low) | -| Info Disclosure | Creds read into env vars, never logged (0 cred strings in reports); ECS binds 127.0.0.1 only | Accept (low) | -| Denial of Service | Local ECS emulator: free port, daemon thread, clean destroy | Accept (low) | -| Elevation of Privilege | `urllib.urlopen` patched to fake response (no network egress); no eval/exec/subprocess in adapter | Accept (low) | +- No new secrets, no new network calls in the engine core (the engine + shells to a local binary; the binary makes no network calls for + `scan`). +- `is_configured()` guard ensures the platform runs without the binary + (no hard dependency that could be exploited as a DoS vector). +- The engine writes the payload to a temp file (`tempfile.NamedTemporaryFile`) + and unlinks it in a `finally` block (no leftover payload on disk). +- No `shell=True` in the `subprocess.run` call (command is a list — + no shell injection surface). -All threats low-severity; auto-accepted per -`config.json security.auto_accept_low_severity=true`. +## Quality -## Layer 4: Quality (multi-persona) — PASS +- `python3 -m py_compile` passes on all new Python files. +- The `PolicyEngine` Protocol is minimal (3 members) — the swap + boundary is the moat (NORTH_STAR Strategic Objective #2). +- The `NullEngine` proves a second implementation exists (structural + conformance) — the OPA swap is a known quantity (RESEARCH §4.2). +- Tests use `pytest.skip` when `which kj` is absent, so the CI matrix + passes with or without the binary (the suite is green in both cases). -| Persona | Finding | Verdict | -|---------|---------|---------| -| Correctness | 7 adapter defects fixed; each traceable to a terraform validate/plan error | PASS | -| Testing | 518 tests pass; 24 new tests. P2: uptime-kuma + RDS not in registry | PASS (1 P2) | -| Security | No creds logged; loopback-only; monkey-patches scoped to local tier | PASS | -| Performance | Regression run ~60s; acceptable for a milestone gate | PASS | -| Maintainability | Well-structured; adding a capability = 1 function + 1 registry entry | PASS | -| Adversarial | Gate can't be bypassed; local E2E can't mutate cloud; no injection vectors | PASS | +## Must-have checklist -**0 P0, 0 P1, 1 P2 (post-hoc: expand regression registry to uptime-kuma + RDS stacks).** +- [x] `PolicyEngine` Protocol + `PolicyEngineRegistry` + `NullEngine` + (REQ-291) +- [x] `config.json.policy` object (REQ-292) +- [x] `KyvernoJsonEngine` adapter (REQ-293) +- [x] `__init__.py` + `_smoke.json` + `install-kyverno-json.sh` + CI + install (REQ-294) +- [x] `test_policy_engine.py` — protocol conformance, registry, + NullEngine fallback (REQ-308) +- [x] `test_kyverno_json_engine.py` — PCR schema validity, defensive + parsing, skip-without-kj (REQ-309) -## Verdict - -**VERIFY PASS** — all 4 layers pass. The v1.10 milestone is sound: -the pipeline regression gap is fixed (D-091), the platform is fully -locally testable (D-092), every advertised capability is re-verified -(D-093, 16/16 Verified), and the docs/decks match verified reality -(D-094). 518 tests pass; the regression gate covers 16 capabilities -including 4 live-AWS checks. 0 P0, 0 P1, 1 P2 post-hoc. Ready to ship. - ---- - -# ACDL — Verify (grill deliverable, commit ac11c01) - -> Verify date: 2026-07-27. Verifier: ci-verifier. Scope: the grill -> deliverable (`.ciagent/GRILL.md`, phase 0, status `grill`) added in -> commit `ac11c01` since the v1.10 audit PASS (`ab477b3`). Docs-only; -> no code, no tests, no schema changes. - -## Layer 1: Structural — PASS - -- `.ciagent/GRILL.md` exists on disk (18250 bytes). -- No imports to resolve (markdown docs file). -- No TODO/FIXME/HACK/stub placeholders in the report. -- All required sections present per grill workflow Step 5 format: - title, Run header, Verdict, 9 axes (1–9), Meta, Binding Decisions - table (12 rows), Escalations section (2 entries: G-005, G-008). -- Commit `ac11c01` `---ci---` block is well-formed: `project: acdl`, - `phase: 0`, `milestone: v1.10`, `status: grill`, 12 decision ids - (G-001..G-012), 2 escalation lines. - -## Layer 2: Behavioral — PASS - -- `pytest tests/ -m "not slow"`: **513 passed**, 5 deselected (no - regressions introduced by the docs-only grill commit). -- No new tests required (docs-only deliverable; the grill is a - review artifact, not a code change). -- Requirement coverage: not applicable (phase 0, status `grill`; no - REQ-IDs bound to this deliverable). The grill's binding decisions - (G-001..G-012) are advisory and do not modify REQUIREMENTS.md per - grill workflow Step 7. - -## Layer 3: Security (STRIDE) — PASS - -| Threat | Risk | Disposition | -|--------|------|-------------| -| Spoofing | N/A (docs-only; no auth surface) | Accept (none) | -| Tampering | Grill report is git-tracked; tampering = git history rewrite (out of scope) | Accept (low) | -| Repudiation | Commit `ac11c01` signed by author; `---ci---` block records status + decisions | Accept (low) | -| Info Disclosure | No credentials, keys, tokens, or PII in the report (grep scan clean) | Accept (low) | -| Denial of Service | N/A (docs file; no runtime surface) | Accept (none) | -| Elevation of Privilege | N/A (docs-only; no privilege surface) | Accept (none) | - -All threats low-or-none; auto-accepted per -`config.json security.auto_accept_low_severity=true`. - -## Layer 4: Quality (multi-persona) — PASS - -| Persona | Finding | Verdict | -|---------|---------|---------| -| Correctness | 12 binding decisions traceable to evidence (commit/file/req-id); 2 escalations correctly unresolved | PASS | -| Testing | Docs-only; 513 fast tests pass (no regression) | PASS | -| Security | No credential leakage; no sensitive data in report | PASS | -| Performance | N/A (docs file; no runtime cost) | PASS | -| Maintainability | Report follows grill workflow Step 5 format exactly; appendable for future runs | PASS | -| Adversarial | Escalations (G-005, G-008) are surfaced, not silently skipped; visible via `ciagent audit` | PASS | - -**0 P0, 0 P1, 0 P2.** - -## Verdict (grill deliverable) - -**VERIFY PASS** — all 4 layers pass. The grill deliverable is a -well-formed docs-only artifact. 513 fast tests pass (no regression). -No credential leakage. 12 binding decisions recorded; 2 escalations -(G-005 risks, G-008 budget) correctly surfaced for human resolution. -The grill does not modify PROJECT.md, ROADMAP.md, or REQUIREMENTS.md -(per grill workflow Step 7). \ No newline at end of file +**Verdict: PASS** — all P1 must-haves met, no regressions, 24 new +tests pass (2 skip-without-kj), 132 existing tests unchanged. \ No newline at end of file