feat(P19): central pipeline contract + shell reproducibility + output streaming (v1.4.1)

---ci---
project: acdl
phase: 19
milestone: v1.4
status: execute
---

Add declarative pipeline contract (schemas/pipeline.schema.json +
pipelines/ci.yaml) as single source of truth for both Gitea Actions (dev)
and GitHub Actions (production) workflows. Both workflow files are
byte-identical and validated against the contract by 32 new tests.

Add scripts/run_ci.sh for shell reproducibility — mirrors the CI pipeline
locally (lint → test → check-only), exits 0 with 'CI PIPELINE OK'.

Update scripts/run_platform.sh to stream output by default: terraform
init/validate/plan via tee, Checkov compliance results with per-record
severity/rule/pass-fail, and emitted Terraform in --check-only. New
--quiet flag for log-only mode.

Requirements: REQ-43 (central pipeline contract), REQ-44 (shell
reproducibility), REQ-45 (output streaming). 122 tests pass (90 + 32).

.ciagent/PROJECT.md

@@ -84,7 +84,7 @@ id 202 published. D-034 closed (root key deactivated by user).**
 Milestone COMPLETE gate: review → ship `v1.2.0` (feature milestone, next
 minor per ship.md) → audit. **DONE.**
 
-## Objective for Milestone v1.2 (active)
+## Objective for Milestone v1.2 (prior — complete)
 
 Platform hardening + first real consumer deployment. The v1.1 spike proved
 the IR commitments hold on a single dev-only `terraform plan` for one S3
@@ -133,6 +133,40 @@ microservice), not just a plan.
 Milestone COMPLETE gate: review → ship `v1.3.0` (feature milestone, next
 minor per ship.md — v1.1 shipped `v1.2.0`) → audit.
 
+## Objective for Milestone v1.4 (active)
+
+Central pipeline contract + shell reproducibility + output streaming. The
+v1.3 milestone (Phases 17–18) created identical CI/CD pipelines for Gitea
+and GitHub but they were duplicated copies with no single source of truth.
+v1.4 makes the pipeline a declarative contract, enables full shell
+reproducibility, and streams terraform/checkov output so users can see
+what the platform is doing.
+
+Three scope axes:
+
+1. **Central pipeline contract.** A JSON Schema
+   (`schemas/pipeline.schema.json`) + YAML instance (`pipelines/ci.yaml`)
+   declares the pipeline stages, commands, triggers, and runner. Both
+   `.gitea/workflows/ci.yml` (Gitea Actions, dev) and
+   `.github/workflows/ci.yml` (GitHub Actions, production) implement the
+   contract. A test validates conformance.
+2. **Shell reproducibility.** `scripts/run_ci.sh` mirrors the CI pipeline
+   locally — runs the same 3 stages (lint, test, check-only) in sequence.
+   The pipeline is fully reproducible from the shell, not just in CI.
+3. **Output streaming.** `scripts/run_platform.sh` streams terraform
+   init/validate/plan output, Checkov compliance results, and
+   PolicyCheckResult records to stdout by default, so the user sees what
+   is happening. A `--quiet` flag suppresses streaming for log-only mode.
+
+## Milestone v1.4 Phases
+
+| Phase | Name | Goal |
+|-------|------|------|
+| 19 | central-pipeline-contract-and-shell-reproducibility | Create the central pipeline contract (JSON Schema + YAML instance). Create `scripts/run_ci.sh` for shell reproducibility. Update `run_platform.sh` to stream terraform/checkov output. Update both workflow YAMLs with contract references (staying byte-identical). Add tests for contract validation, workflow conformance, and streaming. |
+
+Milestone COMPLETE gate: review → ship `v1.4.1` (feature milestone, next
+minor per ship.md — v1.3 shipped `v1.3.2`) → audit.
+
 ## Requirements
 
 ### v1.0 (Prior milestone — the demo)
@@ -168,7 +202,7 @@ New requirements REQ-16..REQ-28 — see `REQUIREMENTS.md` §v1.1. Summary:
 - **REQ-28:** Spike verification proves the IR-shaped commitments hold (no
   polyglot mess; the adapter is the only substrate-specific code).
 
-### v1.2 (Active milestone — platform hardening + first real consumer deployment)
+### v1.2 (Prior milestone — platform hardening + first real consumer deployment, complete)
 
 New requirements REQ-29..REQ-35 — see `REQUIREMENTS.md` §v1.2. Summary:
 
@@ -189,6 +223,18 @@ New requirements REQ-29..REQ-35 — see `REQUIREMENTS.md` §v1.2. Summary:
 - **REQ-35:** End-to-end verification — consumer commit → live ECS service
   (HTTP 200) → evidence event → timeline.
 
+### v1.4 (Active milestone — central pipeline contract + shell reproducibility + streaming)
+
+New requirements REQ-43..REQ-45 — see `REQUIREMENTS.md` §v1.4. Summary:
+
+- **REQ-43:** Central pipeline contract — `schemas/pipeline.schema.json` +
+  `pipelines/ci.yaml`. Both Gitea and GitHub workflows implement the
+  contract; a test validates conformance.
+- **REQ-44:** `scripts/run_ci.sh` mirrors the CI pipeline locally (lint →
+  test → check-only), exiting 0 with "CI PIPELINE OK".
+- **REQ-45:** `scripts/run_platform.sh` streams terraform/checkov output by
+  default (with `--quiet` for log-only mode). Both workflows byte-identical.
+
 ## Constraints
 
 - **Forge:** Gitea at `https://git.cloudinit.dev`, org `continuous-intelligence`.

.ciagent/REQUIREMENTS.md

@@ -91,7 +91,7 @@
 ### Category: End-to-End Verification
 - **REQ-35:** One end-to-end flow: consumer commit to `acdl-consumer-microservice` → pipeline triggered → contract→IR resolution → `terraform plan` → `terraform apply` (dev) → a live ECS Fargate service serving HTTP 200 on its ALB → evidence event written to the DynamoDB outbox → the event renders on the `acdl-evidence` timeline. `scripts/verify_phase16.sh` proves the full flow green.
 
-## v1.3 (Active — module documentation + thin-composition removal)
+## v1.3 (Prior — module documentation + thin-composition removal, complete)
 
 ### Category: Thin-Composition Removal
 - **REQ-36:** The L2 thin-composition layer is removed completely: `composition.json` files, `acdl_platform/contract_resolver.py`, `schemas/contract.schema.json`, `contracts/spike.yaml`, `contracts/microservice.yaml`, and L2 entries in `modules-ir/registry.json` are deleted. The L2 directories are kept as placeholders with READMEs. The downstream pipeline (adapter → checkov → confidence → outbox) is patched to load a pre-existing IR instance instead of resolving a contract.
@@ -109,6 +109,17 @@
 
 - **REQ-42:** `pyproject.toml` exists at the repo root with pytest configuration (testpaths, markers) and the project metadata. `requirements-test.txt` pins test-only dependencies separate from runtime dependencies.
 
+## v1.4 (Active — central pipeline contract + shell reproducibility + streaming)
+
+### Category: Central Pipeline Contract
+- **REQ-43:** A central pipeline contract exists as `schemas/pipeline.schema.json` (JSON Schema draft 2020-12) + `pipelines/ci.yaml` (YAML instance). The contract declares the pipeline name, triggers (push/PR branches), runner, Python version, and stages (name + command + required + install + description). Both `.gitea/workflows/ci.yml` (Gitea Actions, dev) and `.github/workflows/ci.yml` (GitHub Actions, production) implement the same stages, commands, triggers, and runner as declared in the contract. A test (`tests/test_pipeline_contract.py`) validates the contract against the schema and asserts both workflows conform (same jobs, same commands, same triggers, same runner, byte-identical).
+
+### Category: Shell Reproducibility
+- **REQ-44:** `scripts/run_ci.sh` reproduces the CI pipeline locally — runs the same 3 stages (lint, test, check-only) in sequence with proper exit codes, failing on first error. The script exits 0 with "CI PIPELINE OK" on success. A `--quiet` flag suppresses per-stage banners. The script mirrors the central pipeline contract (`pipelines/ci.yaml`) so the shell and CI environments produce identical outcomes.
+
+### Category: Pipeline Streaming
+- **REQ-45:** `scripts/run_platform.sh` streams output by default: terraform init/validate/plan output is piped to stdout via `tee` (visible to the user and logged), Checkov results are printed in human-readable form, and PolicyCheckResult records are displayed with severity, rule ID, and pass/fail status per record. The `--check-only` mode streams the emitted Terraform file content. A `--quiet` flag suppresses streaming (output to log files only) for backwards compatibility. Both gitea and github workflows are byte-identical (identical outcomes — the only difference is the forge runtime).
+
 ## Out of Scope (v1.2)
 
 | REQ | Original criterion | Clarified criterion (effective) | Decision |
@@ -179,7 +190,7 @@
 | REQ-34 | 15 | complete (v1.2.5) |
 | REQ-35 | 16 | partial (v1.2.6, IAM-blocked) |
 
-### v1.3 (active — module documentation + thin-composition removal)
+### v1.3 (prior — module documentation + thin-composition removal, complete)
 
 | Requirement | Phase | Status |
 |-------------|-------|--------|
@@ -189,4 +200,12 @@
 | REQ-39 | 18 | complete (v1.3.2) |
 | REQ-40 | 18 | complete (v1.3.2) |
 | REQ-41 | 18 | complete (v1.3.2) |
-| REQ-42 | 18 | complete (v1.3.2) |
+| REQ-42 | 18 | complete (v1.3.2) |
+
+### v1.4 (active — central pipeline contract + shell reproducibility + streaming)
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| REQ-43 | 19 | complete (v1.4.1) |
+| REQ-44 | 19 | complete (v1.4.1) |
+| REQ-45 | 19 | complete (v1.4.1) |

.ciagent/ROADMAP.md

@@ -5,7 +5,8 @@
 - **v1.0 (demo):** complete — tag `v1.1.0`, 2026-07-21. All 5 phases shipped + audited PASS.
 - **v1.1 (complete):** architecture finalization + v1 spike. 5 phases (06–10). Tag `v1.2.0`, 2026-07-21. All 5 phases shipped + verified; review READY TO SHIP (0 P0); audit CLEAN. Gitea release id 202.
 - **v1.2 (complete):** platform hardening + first real consumer deployment. 6 phases (11–16). Tag `v1.3.0`, 2026-07-21. All 6 phases shipped + verified; review READY TO SHIP (1 P0 operator action, 1 P1 deferred); audit CLEAN.
-- **v1.3 (active):** module documentation + thin-composition removal. The L2 composition layer is removed; module READMEs are built out.
+- **v1.3 (complete):** module documentation + thin-composition removal. The L2 composition layer is removed; module READMEs are built out. Tag `v1.3.2`.
+- **v1.4 (active):** central pipeline contract + shell reproducibility + output streaming. A declarative pipeline contract (`schemas/pipeline.schema.json` + `pipelines/ci.yaml`) binds the Gitea and GitHub workflows to a single source of truth. `scripts/run_ci.sh` mirrors the CI pipeline locally. `scripts/run_platform.sh` streams terraform/checkov output by default.
 - **v1.0 demo URL:** https://git.cloudinit.dev/continuous-intelligence/acdl-evidence/raw/branch/main/index.html
 
 ---
@@ -221,7 +222,7 @@ After Phase 16: COMPLETE gate — review → ship `v1.3.0` → audit.
 
 ---
 
-## v1.3 (Active — module documentation + thin-composition removal)
+## v1.3 (Complete — module documentation + thin-composition removal)
 
 The v1.3 milestone starts with simplification: removing the unsatisfactory
 thin-composition layer and building out proper module documentation. The
@@ -249,4 +250,29 @@ L2 composition mechanism will be redesigned in a later phase.
   - `pytest` runs and passes offline (no AWS, no Checkov, no DynamoDB).
   - `run_platform.sh --check-only` runs offline and exits 0.
   - `.gitea/workflows/ci.yml` and `.github/workflows/ci.yml` exist with identical job stages (lint, test, check-only).
-  - `pyproject.toml` + `requirements-test.txt` pin test dependencies.
+  - `pyproject.toml` + `requirements-test.txt` pin test dependencies.
+
+After Phase 18: COMPLETE gate — review → ship `v1.3.2` → audit.
+
+---
+
+## v1.4 (Active — central pipeline contract + shell reproducibility + streaming)
+
+The v1.4 milestone makes the CI/CD pipeline a declarative contract rather
+than duplicated workflow copies, enables full shell reproducibility of the
+CI pipeline, and streams terraform/checkov output so users can see what
+the platform is doing.
+
+### Phase 19 — central-pipeline-contract-and-shell-reproducibility
+- **Description:** Create a central pipeline contract (`schemas/pipeline.schema.json` JSON Schema + `pipelines/ci.yaml` YAML instance) that both `.gitea/workflows/ci.yml` (Gitea Actions, dev) and `.github/workflows/ci.yml` (GitHub Actions, production) implement. Create `scripts/run_ci.sh` that mirrors the CI pipeline locally (lint → test → check-only). Update `scripts/run_platform.sh` to stream terraform init/validate/plan output, Checkov compliance results, and PolicyCheckResult records to stdout by default (with `--quiet` for log-only mode). Add `tests/test_pipeline_contract.py` validating the contract schema, workflow conformance, and run_ci.sh. Update both workflow YAMLs with contract reference headers (staying byte-identical).
+- **Status:** complete (v1.4.1)
+- **Depends on:** [18]
+- **Requirements:** REQ-43, REQ-44, REQ-45
+- **Success Criteria:**
+  - `pipelines/ci.yaml` validates against `schemas/pipeline.schema.json`.
+  - Both `.gitea/workflows/ci.yml` and `.github/workflows/ci.yml` are byte-identical.
+  - A test parses both workflows and asserts their stages/commands match the contract.
+  - `scripts/run_ci.sh` exits 0 and outputs "CI PIPELINE OK".
+  - `scripts/run_platform.sh --check-only` streams the emitted Terraform to stdout.
+  - `scripts/run_platform.sh --check-only --quiet` suppresses the Terraform stream.
+  - `pytest` total count increases from 90 to 122 (32 new contract/streaming tests).

.ciagent/config.json

@@ -4,7 +4,7 @@
     {
       "slug": "acdl",
       "name": "Agentic Cloud Delivery Platform",
-      "milestone": "v1.3",
+      "milestone": "v1.4",
       "status": "active"
     }
   ],