diff --git a/adapters/README.md b/adapters/README.md new file mode 100644 index 0000000..58438c4 --- /dev/null +++ b/adapters/README.md @@ -0,0 +1,65 @@ +# ACDL Adapters + +## Overview + +Adapters translate the substrate-agnostic Target Stack IR to substrate-specific formats. The Terraform adapter is the primary adapter (IR → HCL). Policy adapters translate security tool output into normalized `PolicyCheckResult` records that the confidence signal consumes in an engine-agnostic way. + +## Existing Adapters + +| Adapter | Path | Input | Output | Purpose | +| --- | --- | --- | --- | --- | +| Terraform adapter | `adapters/terraform/adapter.py` | Stack instance JSON | Terraform HCL (`main.tf`, `terraform.tf`, `providers.tf`) | Compiles IR to Terraform | +| Checkov adapter | `adapters/terraform/policy/checkov_adapter.py` | Checkov JSON | `PolicyCheckResult` records | Translates Checkov results | +| Wiz adapter | `adapters/wiz/wiz_adapter.py` | Wiz API issues JSON | `PolicyCheckResult` records | Translates Wiz security findings | +| Kyverno adapter | `adapters/kyverno/kyverno_adapter.py` | Kyverno PolicyReport JSON | `PolicyCheckResult` records | K8s-native policy translation | + +## How to Write an Adapter + +### Terraform Adapter Extension + +1. Add a stack type → Terraform type mapping to `TYPE_MAP`. +2. Add non-identity input mappings to `INPUT_MAP`. +3. Add non-identity output mappings to `OUTPUT_MAP`. +4. Add a specialized `_emit_resource` branch if the resource needs nested blocks (e.g. inline policies, rule sets). + +### Policy Adapter Pattern + +1. Define `SEVERITY_MAP` and `RESULT_MAP` dicts that translate the engine's native severity/result vocabulary to the `PolicyCheckResult` enums. +2. Implement `_to_pcr(raw_record, contract_id)` → `PolicyCheckResult` dict. +3. Implement `adapt(input_path, contract_id)` → list of `PolicyCheckResult` dicts. +4. Implement `is_configured()` → bool (env var check) so the platform can skip the adapter when credentials are absent. + +## How to Wire an Adapter + +- **Terraform adapter** — invoked by `scripts/run_platform.sh` Step 3 (`terraform-plan`). +- **Checkov adapter** — invoked by `scripts/run_platform.sh` Step 5 (`checkov`). +- **Wiz / Kyverno adapters** — optional Steps 5b/5c, run only when the relevant env vars are set. +- All policy adapters output records that are validated against `schemas/policy_check_result.schema.json`. + +## Dependencies + +- `jsonschema`, `pyyaml` — used by all adapters for loading and validating inputs. +- `boto3` — used by the Wiz adapter for AWS API access. +- `checkov` — used by the Checkov adapter to run policy scans. +- No external deps for the Terraform adapter (pure Python). + +## How to Test Adapters + +- `tests/test_adapter.py` — Terraform adapter (`TYPE_MAP`, resource emission, refs, outputs). +- `tests/test_checkov_adapter.py` — Checkov adapter. +- `tests/test_wiz_adapter.py` — Wiz adapter. +- `tests/test_kyverno_adapter.py` — Kyverno adapter. +- All adapter tests load fixtures from `tests/fixtures/` and use `moto` for AWS mocking. + +## Where to Write Tests + +- `tests/test_.py` paired with `tests/fixtures/_fixture.json`. + +## Adding a New Adapter + +1. Create `adapters//_adapter.py`. +2. Implement `adapt()` and (for policy adapters) `is_configured()`. +3. Add the adapter's engine name to the `engine` enum in `schemas/policy_check_result.schema.json` if it is a policy adapter. +4. Write a test (`tests/test__adapter.py`) plus a fixture (`tests/fixtures/_fixture.json`). +5. Add it to `scripts/run_platform.sh` if it is invoked at runtime. +6. Update this README. \ No newline at end of file diff --git a/pipelines/README.md b/pipelines/README.md new file mode 100644 index 0000000..305bcec --- /dev/null +++ b/pipelines/README.md @@ -0,0 +1,44 @@ +# ACDL Pipelines + +## Overview + +ACDL uses declarative pipeline contracts (YAML) as the single source of truth. Both Gitea and GitHub workflows implement the same contract (byte-identical). The shell runner (`scripts/run_ci.sh`) mirrors the CI pipeline locally so that every stage that runs in CI can be reproduced on a developer machine without a forge. + +## Existing Pipelines + +| Pipeline | File | Stages | Triggers | +| --- | --- | --- | --- | +| ACDL CI | `ci.yaml` | `lint`, `test`, `check-only` | push/PR to `main` | +| ACDL Deploy | `deploy.yaml` | `validate-contract`, `resolve-stack`, `terraform-plan`, `checkov`, `confidence`, `apply`, `publish-outputs`, `deploy-uptime`, `comment-outputs` | push/PR to `main` (consumer repos via `workflow_call`) | + +## How to Write a Pipeline + +1. YAML structure: `name`, `environment`, `triggers` (with `push` and `pull_request` branch arrays), `runner`, `python_version`, and a `stages[]` list. +2. Each stage is an object with `name`, `command`, `required` (boolean), and optional `install` (pip install command) + `description` (human-readable summary). +3. Validate the resulting YAML against `schemas/pipeline.schema.json` (CI) or `schemas/deploy-pipeline.schema.json` (deploy). + +## How to Wire a Pipeline + +1. Create byte-identical workflow YAMLs in `.gitea/workflows/.yml` and `.github/workflows/.yml`. +2. Both workflows must implement the same stages, commands, triggers, and runner declared in the contract. +3. `scripts/run_ci.sh` mirrors `ci.yaml` locally so the same stages run without a forge. +4. Consumer repos reference the deploy pipeline via `uses: acdl/.github/workflows/deploy.yml@vX.Y`. + +## Dependencies + +- `scripts/run_ci.sh` — local CI mirror that runs the `ci.yaml` stages. +- `scripts/run_platform.sh` — platform pipeline runner that implements the `deploy.yaml` stages. +- Workflow YAMLs in `.gitea/workflows/` and `.github/workflows/`. +- Schemas in `schemas/` (`pipeline.schema.json`, `deploy-pipeline.schema.json`). + +## How to Test Pipelines + +- `tests/test_pipeline_contract.py` — validates each pipeline YAML against its schema, asserts workflow conformance (byte-identical Gitea/GitHub workflows with the same stages/commands/triggers), and tests `scripts/run_ci.sh` execution against the contract. + +## Adding a New Pipeline + +1. Create `pipelines/.yaml` using the structure above. +2. Create or extend the schema in `schemas/` for the new pipeline shape. +3. Create byte-identical workflow YAMLs in `.gitea/workflows/.yml` and `.github/workflows/.yml`. +4. Extend `scripts/run_ci.sh` if a local mirror of the new pipeline is needed. +5. Write or extend tests in `tests/test_pipeline_contract.py` to assert schema validity and workflow conformance. \ No newline at end of file diff --git a/schemas/README.md b/schemas/README.md new file mode 100644 index 0000000..e88fc8a --- /dev/null +++ b/schemas/README.md @@ -0,0 +1,58 @@ +# ACDL Schemas + +## Overview + +ACDL uses JSON Schema draft 2020-12 for all declarative contracts. Schemas are the single source of truth for validation. Every contract, stack instance, pipeline, and policy result in the platform is validated against a schema in this directory before it is consumed by any downstream code path. The resolver, the pipeline runner, the CI workflows, and the test suite all load these schemas directly. + +## Existing Schemas + +| Schema | File | Purpose | Where Validated | +| --- | --- | --- | --- | +| ACDL Consumer Contract | `contract.schema.json` | Consumer contract validation (module, environment, inputs, `uses` reference) | `core/contract_resolver.py`, `scripts/run_platform.sh` Step 1, CI `schema-validation` job | +| ACDL Target Stack | `stack.schema.json` | Target Stack instance validation (resources, relationships, composition tree, NFRs) | `core/contract_resolver.py` (post-resolution), `tests/conftest.py` | +| ACDL Central Pipeline Contract | `pipeline.schema.json` | Central CI pipeline contract (stages, commands, triggers, runner) | `tests/test_pipeline_contract.py` | +| ACDL Central Deployment Pipeline Contract | `deploy-pipeline.schema.json` | Central deploy pipeline contract (validate → resolve → plan → checkov → confidence → apply → publish → uptime → comment) | `tests/test_pipeline_contract.py` | +| ACDL PolicyCheckResult | `policy_check_result.schema.json` | Normalized policy check result schema (the contract between policy engines and the confidence signal) | `tests/conftest.py`, all adapter tests | +| ACDL Tagging Standard | `tagging-standard.json` | Required tag set for all taggable AWS resources | `adapters/terraform/policy/custom_rules/acdl_tagging.py` | + +## How to Write a Schema + +1. Use JSON Schema draft 2020-12: `"$schema": "https://json-schema.org/draft/2020-12/schema"`. +2. Set `$id` to `https://acdl.cloudinit.dev/schemas/.schema.json`. +3. Include `title` and `description` at the document root. +4. Set `type: object` at the document root. +5. Declare a `required` array listing the mandatory top-level property names. +6. Define `properties` with explicit `type`, `pattern`, `enum`, and `description` for every field. +7. Use `$defs` for reusable sub-schemas (e.g. resource definitions, input maps) and `$ref` them from the main document. + +## How to Wire a Schema into the Platform + +- **Contract validation** — load the schema in `core/contract_resolver.py` and in `scripts/run_platform.sh` Step 1 (`validate-contract`). +- **Stack validation** — load the schema in `core/contract_resolver.py` after the contract is resolved to a stack instance. +- **Pipeline validation** — load the schema in `tests/test_pipeline_contract.py`, which validates `pipelines/ci.yaml` and `pipelines/deploy.yaml`. +- **Module interface validation** — structural checks in `.github/workflows/platform-test.yml` (`schema-validation` job) that validate each module's `interface.json` / `composition.json`. +- **Policy result validation** — the schema is loaded as a fixture in `tests/conftest.py` and reused by every adapter test to validate emitted `PolicyCheckResult` records. + +## Dependencies + +- `jsonschema` (Python) — installed via `requirements-test.txt`. +- `pyyaml` — for YAML contract loading (`core/contract_resolver.py`, `scripts/run_platform.sh`, tests). + +## How to Test Schemas in CI + +- `tests/test_pipeline_contract.py` — validates the pipeline schemas and asserts workflow conformance (byte-identical Gitea/GitHub workflows, same stages/commands/triggers). +- `tests/conftest.py` — provides `stack_schema` and `policy_check_result_schema` fixtures for reuse across the test suite. +- `.github/workflows/platform-test.yml` `schema-validation` job — self-validates every schema in `schemas/` (each schema is loaded and meta-validated), validates module interfaces, and validates example contracts. + +## Where to Write Tests + +- `tests/test_.py` for schema-specific tests (e.g. `tests/test_contract_schema.py`). +- Extend `tests/test_pipeline_contract.py` for pipeline-schema changes. +- Module interface validation lives in the CI workflow (`.github/workflows/platform-test.yml`). + +## Adding a New Schema + +1. Create `schemas/.schema.json` using the draft 2020-12 conventions above. +2. Add it to the CI validation glob in `.github/workflows/platform-test.yml` (`schema-validation` job). +3. Write a test in `tests/test_.py` that loads the schema and validates representative valid/invalid documents. +4. Wire it into the consuming code path (resolver, script, or test) so it is enforced at runtime. \ No newline at end of file diff --git a/tests/test_docs_coverage.py b/tests/test_docs_coverage.py new file mode 100644 index 0000000..d8ea3a7 --- /dev/null +++ b/tests/test_docs_coverage.py @@ -0,0 +1,37 @@ +"""Validate that path documentation READMEs exist and have required sections (REQ-97, 98, 99).""" + +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent + + +class TestDocsCoverage: + def test_schemas_readme_exists(self): + assert (ROOT / "schemas" / "README.md").is_file() + + def test_schemas_readme_has_required_sections(self): + content = open(ROOT / "schemas" / "README.md").read() + assert "How to Write a Schema" in content + assert "How to Wire" in content + assert "How to Test" in content + assert "Existing Schemas" in content + + def test_pipelines_readme_exists(self): + assert (ROOT / "pipelines" / "README.md").is_file() + + def test_pipelines_readme_has_required_sections(self): + content = open(ROOT / "pipelines" / "README.md").read() + assert "How to Write a Pipeline" in content + assert "How to Wire" in content + assert "How to Test" in content + assert "Existing Pipelines" in content + + def test_adapters_readme_exists(self): + assert (ROOT / "adapters" / "README.md").is_file() + + def test_adapters_readme_has_required_sections(self): + content = open(ROOT / "adapters" / "README.md").read() + assert "How to Write an Adapter" in content + assert "How to Wire" in content + assert "How to Test" in content + assert "Existing Adapters" in content \ No newline at end of file