From 7f1eff622d50481cbede44f491c33afd4ee33a2a Mon Sep 17 00:00:00 2001 From: Jon Chery Date: Wed, 22 Jul 2026 22:01:13 +0000 Subject: [PATCH] docs(P22-38): research findings + 6 phase plans ---ci--- project: acdl phase: 22-38 milestone: v1.8 status: research ---/ci--- v1.8 research: 6 targets investigated + 5 decisions surfaced (D-073..D-077). TARGET 1: uptime-kuma on ECS Fargate (image louislam/uptime-kuma:1, port 3001, EFS volume, ALB, monitor seeding via post-deploy API script). TARGET 2: Terraform prevent_destroy lifecycle (2-step decommission pattern). TARGET 3: AWS KMS rotation (enable_key_rotation=true, annual AWS-managed; 90-day is roadmap item D-075). TARGET 4: Forge-agnostic API URLs (GITHUB_API_BASE + _forge_type helper). TARGET 5: DynamoDB as CMDB (acdl-change-requests table + validate_change_request). TARGET 6: Module engineering standards scan (codified patterns from 12 modules). PERSONAS.md updated for v1.8: lambda-engineer reactivated (D-068 CMDB + D-065 API portability). 11 phase-specific overrides added. --- .ciagent/PERSONAS.md | 41 ++++--- .ciagent/RESEARCH.md | 259 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 285 insertions(+), 15 deletions(-) diff --git a/.ciagent/PERSONAS.md b/.ciagent/PERSONAS.md index ab5797f..6768169 100644 --- a/.ciagent/PERSONAS.md +++ b/.ciagent/PERSONAS.md @@ -1,6 +1,6 @@ --- project: acdl -milestone: v1.7 +milestone: v1.8 generated_at: 2026-07-22 generator: lead-developer verification_toolchain: @@ -16,7 +16,7 @@ verification_toolchain: ci-* agents read PERSONAS.md before running verification commands. --- -# ACDL — Persona Roster (project-level, v1.7) +# ACDL — Persona Roster (project-level, v1.8) ## Active personas @@ -56,14 +56,14 @@ verification_toolchain: - **Territory:** `core/hitl_matrix_design.md`, `core/audit_ledger_design.md`, `adapters/terraform/policy/**` (Checkov adapter + custom rules), `adapters/wiz/**` (Wiz adapter), `adapters/kyverno/**` (Kyverno adapter + sample policies), `core/separation_of_duties.py`, `schemas/tagging-standard.json`, `schemas/policy_check_result.schema.json` (engine enum) - **Reason:** Owns the HITL matrix design, separation-of-duties, the audit ledger design, the Checkov→PolicyCheckResult adapter + the custom tagging rule (D-054, D-043 closure), the Wiz adapter (D-052), the Kyverno adapter (D-053), and the tagging standard. Enforces the "Safety is Computed, Not Assumed" + "Audit truth lives outside the repository" vision tenets. -### lambda-engineer (custom, v1.7) +### lambda-engineer (custom, v1.8) - **Domain:** serverless - **Active:** true -- **Phase-specific:** true (created for v1.7; removed after milestone COMPLETE) -- **Frameworks:** python, aws-lambda, boto3, dynamodb, aws-secretsmanager, github-api -- **Constraints:** lambda-is-stateless, dynamodb-is-the-state-store, secrets-from-secrets-manager-never-logged, idempotent-actions, cross-account-iam-via-abac -- **Territory:** `core/lambda/**` (contract_ingestor.py + handler), `terraform/platform/main.tf` (Lambda + Function URL + DynamoDB + KMS + Secrets Manager + IAM), `terraform/platform/consumer_invoke_policy.json` -- **Reason:** Owns the platform Lambda for contract ingestion (D-051) + error reporting (D-055). The Lambda is stateless; all state is in DynamoDB. The Lambda holds a GitHub token (Secrets Manager) scoped to the platform repo. Cross-account invocation is scoped via ABAC. Created for v1.7; the role is removed from the roster after milestone COMPLETE (the code persists, but the persona is no longer active). +- **Phase-specific:** true (reactivated for v1.8; removed after milestone COMPLETE) +- **Frameworks:** python, aws-lambda, boto3, dynamodb, aws-secretsmanager, github-api, gitea-api +- **Constraints:** lambda-is-stateless, dynamodb-is-the-state-store, secrets-from-secrets-manager-never-logged, idempotent-actions, cross-account-iam-via-abac, forge-agnostic-api-urls +- **Territory:** `core/lambda/**` (contract_ingestor.py + handler), `terraform/platform/main.tf` (Lambda + Function URL + DynamoDB + KMS + Secrets Manager + IAM + acdl-change-requests table), `terraform/platform/consumer_invoke_policy.json`, `terraform/platform/variables.tf` +- **Reason:** Reactivated for v1.8 Phase 29 (Terraform-rendered invoke policy), Phase 30 (forge-agnostic API URLs in contract_ingestor.py), Phase 34 (validate_change_request Lambda action + acdl-change-requests DynamoDB table). The Lambda is stateless; all state is in DynamoDB. Forge-agnostic API URLs (GitHub + Gitea) via GITHUB_API_BASE env var. Removed from the roster after milestone COMPLETE (the code persists, but the persona is no longer active). ### frontend-engineer - **Domain:** frontend @@ -96,18 +96,29 @@ verification_toolchain: | Phase | Personas active | Notes | |-------|------------------|-------| -| 22 rename-and-production-static-assets-stack | lead-developer, platform-engineer (lead: rename + cloudfront/waf primitives + adapter), backend-engineer (contract schema + pipelines), security-engineer (review new primitives) | lambda/frontend idle | -| 23 tagging-standards-and-security-adapters | security-engineer (lead: tagging rule + Wiz + Kyverno), platform-engineer (custom rule loading), backend-engineer (schema enum) | lambda/frontend idle | -| 24 platform-lambda-and-contract-ingestion | lambda-engineer (lead: Lambda + DynamoDB + Terraform), platform-engineer (Terraform infra review), security-engineer (cross-account IAM review), backend-engineer (onboarding docs) | frontend idle | -| 25 deploy-pipeline-dx-outputs-and-error-reporting | backend-engineer (lead: outputs + stage comments + error-report step), lambda-engineer (report_error action), security-engineer (no-secrets-in-logs review) | platform/frontend idle | -| 26 platform-pipelines-and-release-automation | backend-engineer (lead: 3 pipelines + release job), lead-developer (verification scripts) | platform/security/lambda/frontend idle | -| 27 remove-legacy-consumer-repos-and-module-documentation-examples | lead-developer (lead: examples + docs), platform-engineer (RDS primitive + adapter), backend-engineer (schema-validation) | security/lambda/frontend idle | +| 28 adapter-waf-and-resolver-outputs | platform-engineer (lead: WAF HCL fix + adapter output blocks), backend-engineer (resolver outputs processing) | security/lambda/frontend idle | +| 29 ssm-kms-and-invoke-policy | backend-engineer (lead: SSM fail-loud), lambda-engineer (Terraform-rendered invoke policy), security-engineer (CMK enforcement review) | platform/frontend idle | +| 30 run-platform-isolation-and-api-portability | backend-engineer (lead: run_platform.sh temp dir + deploy.yml static-key), lambda-engineer (forge-agnostic API URLs) | platform/security/frontend idle | +| 31 encryption-by-default-and-per-stack-cmk | platform-engineer (lead: kms-key primitive + adapter expansion + L2 wiring), security-engineer (encryption NFR enforcement review) | backend/lambda/frontend idle | +| 32 deletion-protection-by-default-and-l2-feature-flag | platform-engineer (lead: prevent_destroy emission + L2 feature flag), backend-engineer (contract schema update) | security/lambda/frontend idle | +| 33 uptime-kuma-primitive | platform-engineer (lead: uptime primitive + adapter + separate state), backend-engineer (deploy-uptime pipeline stage + run_platform.sh + PR comment) | security/lambda/frontend idle | +| 34 decommission-alias-and-cmdb-validation | backend-engineer (lead: decommission pipeline mode + run_platform.sh + consumer docs), lambda-engineer (validate_change_request + acdl-change-requests table), security-engineer (HITL SRE gates review) | platform/frontend idle | +| 35 module-engineering-standards | lead-developer (lead: STANDARDS.md + catalog fix + template), platform-engineer (standards content review), backend-engineer (automated standards test) | security/lambda/frontend idle | +| 36 schemas-adapters-pipelines-readmes | lead-developer (lead: 3 READMEs), backend-engineer (pipelines + schemas README content), platform-engineer (adapters README content) | security/lambda/frontend idle | +| 37 verify | lead-developer (lead: 4-layer verification), all personas (review their territory) | — | +| 38 review-audit-complete | lead-developer (lead: review + audit + milestone completion), all personas (review participation) | — | ## Domain priority (used by TaskDecomposer) `coordination → security → platform → backend → lambda → frontend` -Rationale: in v1.7, the security commitments (tagging, Wiz, Kyverno, cross-account IAM) and the platform commitments (CloudFront, WAF, RDS, Lambda, DynamoDB) are the binding constraints; backend wires the pipeline + outputs; lambda owns the ingestion + error reporting; frontend surfaces the evidence (unchanged from v1.0). +Rationale: in v1.8, the security commitments (encryption by default, +KMS rotation, deletion protection, CMDB validation, HITL SRE gates) +and the platform commitments (kms-key primitive, uptime primitive, +adapter expansion, prevent_destroy emission) are the binding +constraints; backend wires the pipeline + decommission mode + API +portability; lambda owns the CMDB validation + forge-agnostic APIs; +frontend is unchanged from v1.0 (evidence timeline). ## Conflict resolutions (lead-developer arbitration) diff --git a/.ciagent/RESEARCH.md b/.ciagent/RESEARCH.md index 930dc27..cf36d9b 100644 --- a/.ciagent/RESEARCH.md +++ b/.ciagent/RESEARCH.md @@ -1462,4 +1462,263 @@ thin-composition references all six (depth ≤ 5). --- +## v1.8 Research Addendum + +> Phase: research (pre-Phase 28). Milestone: v1.8. Status: active. +> Researcher: ci-researcher. Autonomy: full. +> Sources: web (uptime-kuma GitHub, Terraform docs, AWS KMS docs, AWS +> ECS Fargate docs, GitHub Actions docs) + ACDL codebase analysis. + +### RESEARCH TARGET 1 — uptime-kuma deployment on ECS Fargate + +**Verdict: ECS Fargate is the most cost-effective cloud-native option +for deploying uptime-kuma, consistent with the existing platform +primitives (ecs-cluster, ecs-service, alb).** + +Findings (verified 2026-07-22): + +1. **uptime-kuma Docker image:** `louislam/uptime-kuma:1` (v1) or + `louislam/uptime-kuma:2` (v2, latest stable 2.4.0 as of 2026-05-31). + The container listens on port 3001. Data is stored in `/app/data` + (SQLite + uploaded files). NFS is not supported for the data volume; + EFS is the AWS-native equivalent and works with ECS Fargate. + +2. **Monitoring capabilities:** HTTP(s), TCP, HTTP(s) Keyword, HTTP(s) + JSON Query, WebSocket, Ping, DNS Record, Push, Steam Game Server, + Docker Containers. 20-second intervals minimum. Certificate info. + Proxy support. 2FA support. + +3. **Notification services (90+):** Telegram, Discord, Gotify, Slack, + Pushover, Email (SMTP), Microsoft Teams (via webhook), and many + others. For the ACDL primitive, we expose: Teams webhook, email + (SMTP), SMS (via SNS or an external gateway), and GitHub issues + (via the GitHub API). + +4. **ECS Fargate deployment shape:** + - Task definition: 1 container (`louislam/uptime-kuma:1`), port 3001, + CPU 256 (.25 vCPU), Memory 512 (.5 GB) — minimal cost (~$5/mo + at us-east-1 on-demand pricing for .25 vCPU + .5 GB running 24/7). + - EFS volume for `/app/data` (persistent storage across task + restarts; Fargate + EFS is the standard pattern for stateful + containers). + - ALB + listener for a stable public URL (the uptime dashboard). + - CloudWatch log group (encrypted with the per-stack CMK). + +5. **Endpoint seeding:** uptime-kuma has a REST API (socket.io-based). + The platform can seed monitors by either: + - (a) Passing `UPTIMA_KUMA__monitors` env var (JSON array) consumed + by a startup script — but uptime-kuma does not natively read env + for monitor config. + - (b) A post-deploy seeding script that calls the uptime-kuma API + (`POST /api/monitor`) to create monitors from the `monitored_endpoints` + input. This is the cleaner approach — the platform runs a Python + script after the ECS service is up that creates monitors via the + API. + - **Recommendation:** (b) — a `scripts/seed_uptime_monitors.py` that + reads the `monitored_endpoints` from the stack outputs + calls the + uptime-kuma API. This is testable offline (mocked API) and + decouples container startup from monitor configuration. + +6. **Separate terraform state:** The uptime stack uses a separate S3 + key prefix (`uptime/{consumerRepo}/{contractId}/`) so it is + independent of the consumer stack's state. The uptime stack has its + own VPC + ALB + ECS cluster (or shares the consumer's — design + decision: **separate** to avoid state coupling, per the requirement + "separate terraform run, with a separate state"). + +7. **Feature flag:** The `feature_flag_enabled` input (set from the + consumer contract `inputs.uptime_enabled`, default true) controls + whether the `deploy-uptime` pipeline stage runs. When false, the + stage is skipped entirely (no resources emitted, no API calls). + +### RESEARCH TARGET 2 — Terraform prevent_destroy lifecycle + +**Verdict: `lifecycle { prevent_destroy = true }` is the correct +Terraform mechanism for deletion protection. It prevents `terraform +destroy` from destroying the resource without first setting +`prevent_destroy = false`.** + +Findings (verified 2026-07-22): + +1. **`prevent_destroy`** is a meta-argument inside a `lifecycle {}` + block within a resource. When set to `true`, any Terraform plan + that would destroy the resource will fail with an error. To destroy, + the user must first set `prevent_destroy = false` and apply, then + destroy. + +2. **This is exactly the 2-step decommission pattern the user + requested:** Step 1: set `deletion_protection = false` (which the + adapter translates to `prevent_destroy = false`) + apply. Step 2: + set all counts to 0 + apply (which destroys the resources now that + prevent_destroy is false). + +3. **Adapter emission:** The adapter should emit `lifecycle { prevent_destroy = true }` + inside each resource block when the `deletion_protection` NFR is + true. When false, omit the `lifecycle` block (or set + `prevent_destroy = false`). This is a per-resource meta-argument, + not a provider-level setting. + +4. **RDS special case:** RDS already has a `deletion_protection` + argument on `aws_db_instance` (not a lifecycle meta-arg). The + adapter should emit BOTH: the `deletion_protection` argument (for + the RDS API-level protection) AND `lifecycle { prevent_destroy = true }` + (for the Terraform-level protection). This is defense-in-depth. + +### RESEARCH TARGET 3 — AWS KMS key rotation + +**Verdict: `enable_key_rotation = true` on `aws_kms_key` enables +automatic annual rotation (AWS rotates the key material annually). +For 90-day rotation, a custom key rotation policy is needed (AWS +managed rotation is annual only; 90-day requires a manual rotation +schedule or a custom multi-region key + rotation Lambda).** + +Findings (verified 2026-07-22): + +1. **`aws_kms_key`** with `enable_key_rotation = true` enables AWS's + automatic key material rotation. AWS rotates the backing key material + annually (365 days). This is the simplest option and is the AWS + best practice for most use cases. + +2. **90-day rotation:** AWS does not support custom rotation periods + for managed keys. To achieve 90-day rotation: + - (a) Use `aws_kms_key` with `enable_key_rotation = true` (annual + AWS-managed rotation) + a CloudWatch Events rule that triggers a + Lambda every 90 days to create a new key + update the alias. This + is complex and overkill for v1.8. + - (b) Accept annual AWS-managed rotation as the default and document + that 90-day rotation requires a custom rotation pipeline (roadmap + item). The `enable_key_rotation = true` is the v1.8 implementation; + the 90-day requirement is a roadmap enhancement. + + **Recommendation:** (b) — `enable_key_rotation = true` (AWS-managed + annual rotation) as the v1.8 implementation. The 90-day requirement + is documented as a roadmap item (custom rotation Lambda). The NFR + `enable_rotation` (default true) controls the `enable_key_rotation` + argument. This is pragmatic; annual rotation is AWS's best practice + and 90-day is a future enhancement. + +3. **Per-stack CMK pattern:** Each L2 deployment creates its own + `aws_kms_key` + `aws_kms_alias` (alias/acdl--). + The key is tagged with `acdl:owner` + `acdl:environment`. All + primitives in the stack reference this key via `kms_key_arn`. + No shared keys across stacks. + +4. **Managed KMS fallback:** When a primitive is deployed standalone + (L1 without an L2 CMK), the adapter uses `alias/aws/` + (e.g. `alias/aws/s3`, `alias/aws/rds`). This is the AWS-managed + key for that service. The adapter emits a stderr warning when + falling back. The `kms_key_arn` input is optional; the + `encryption_enabled` NFR defaults to true. + +### RESEARCH TARGET 4 — Forge-agnostic API URLs (P1-9) + +**Verdict: GitHub and Gitea have compatible issue APIs but different +search endpoints. A `GITHUB_API_BASE` env var + `_forge_type()` +helper branches the search URL.** + +Findings (verified 2026-07-22): + +1. **GitHub API:** `https://api.github.com/search/issues?q=...` for + search; `https://api.github.com/repos/{owner}/{repo}/issues` for + create; `https://api.github.com/repos/{owner}/{repo}/issues/{n}/comments` + for comments. + +2. **Gitea API:** `https://git.cloudinit.dev/api/v1/repos/{owner}/{repo}/issues?...` + for search (no `/search/issues` endpoint — issues are listed via + the repo issues endpoint with query params); `https://git.cloudinit.dev/api/v1/repos/{owner}/{repo}/issues` + for create; `https://git.cloudinit.dev/api/v1/repos/{owner}/{repo}/issues/{n}/comments` + for comments. + +3. **Detection:** If `GITHUB_API_BASE` contains `/api/v1`, it's Gitea; + otherwise it's GitHub. The `_forge_type()` helper returns `"gitea"` + or `"github"` based on this. The search URL is branched accordingly; + the create + comment URLs are the same pattern (`{base}/repos/{owner}/{repo}/issues`). + +4. **Auth:** Both use `Authorization: token ` header. GitHub + also accepts `Authorization: Bearer `; Gitea uses `token`. + The existing `token` header works for both. + +### RESEARCH TARGET 5 — DynamoDB as CMDB for change requests + +**Verdict: A DynamoDB `acdl-change-requests` table is consistent with +the existing platform Lambda + DynamoDB pattern (D-051). The +`validate_change_request` Lambda action queries the table + asserts +status=approved.** + +Findings (verified 2026-07-22): + +1. **Table schema:** PK `changeRequestId` (string), SK `submittedAt` + (string). Attributes: `consumerRepo`, `contractId`, `status` + (enum: `requested|approved|rejected|executed`), `requestedBy`, + `approvedBy`, `submittedAt`, `executedAt`. + +2. **Validation flow:** The decommission pipeline's + `validate-change-request` stage invokes the Lambda with + `action: validate_change_request`, `changeRequestId: `, + `consumerRepo: `. The Lambda queries the table; if the item + exists + `status == "approved"` + `consumerRepo` matches, returns + 200 with the CR details. Otherwise returns 403. + +3. **Terraform:** Add the table to `terraform/platform/main.tf` with + SSE via the platform CMK + point-in-time recovery (matching the + `acdl-contracts` table pattern from D-051). + +### RESEARCH TARGET 6 — Module engineering standards (scan of current modules) + +**Verdict: The current modules follow a consistent pattern that can +be codified into standards. Key patterns identified:** + +1. **L1 required files:** `interface.json`, `instance.json`, + `README.md`, `examples/simple.yaml`, `examples/complex.yaml`. + Multi-resource L1s add `resources[]` + `intra_refs[]` to + `interface.json`. + +2. **L2 required files:** `composition.json`, `README.md`, + `examples/simple.yaml`, `examples/complex.yaml`. No `instance.json`. + +3. **Interface shape:** `name`, `version`, `kind` ("l1"|"l2"), + `type` (L1 only, `aws::`), `description`, + `inputs` (object keyed by name), `outputs` (object keyed by name), + `nfrs` (object keyed by name). Multi-resource L1s add `resources[]` + (array of `{type, description, inputs[], outputs[]}`) + + `intra_refs[]` (array of `{from, to}`). + +4. **Input shape:** `{type, description, required, [default], [enum]}`. + Output shape: `{type, description}`. NFR shape: + `{type, description, default}`. + +5. **NFR conventions (v1.8 additions):** Every L1 MUST have + `deletion_protection` (boolean, default true) + `encryption_enabled` + (boolean, default true) NFRs. L2 modules MUST expose + `features.deletion_protection` (default true) + + `features.uptime_enabled` (default true). + +6. **Registry:** Every module MUST be registered in + `modules/registry.json` at its semver. Entry: + `{"interface": "", "published_at": "", "deprecated": false}`. + +7. **Adapter extension:** 3-table pattern (TYPE_MAP + INPUT_MAP + + OUTPUT_MAP) + specialized `_emit_resource` branches for complex + resources (nested blocks like `origin {}`, `rules {}`, + `default_cache_behavior {}`). + +8. **README structure:** `# — `, `## Resources`, + `## Inputs`, `## Outputs`, `## NFRs`, `## Usage`, `## Compliance + extension points`, `## Examples`, `## Versioning`. + +9. **Catalog index gap:** `modules/README.md` Primitives table is + missing `rds` (flagged during scan). Must be fixed in Phase 35. + +### Decisions surfaced (v1.8) + +| ID | Decision | Rationale | Confidence | Alternatives | +|----|----------|-----------|------------|--------------| +| **D-073** | uptime-kuma v1 (`louislam/uptime-kuma:1`) as the default container image. | v1 is stable + widely deployed. v2 (2.4.0) is newer but has breaking changes. v1 is the safer default; consumers can override via `container_image` input. | 0.85 | v2 (breaking changes risk); pin to a specific v1 tag (maintenance burden). | +| **D-074** | Monitor seeding via post-deploy API script (`scripts/seed_uptime_monitors.py`), not env vars. | uptime-kuma does not natively read env for monitor config. A post-deploy script calling the API is cleaner + testable offline. | 0.90 | Env var config (not supported by uptime-kuma); manual config (defeats automation). | +| **D-075** | KMS rotation = `enable_key_rotation = true` (AWS-managed annual). 90-day rotation is a roadmap item (custom rotation Lambda). | AWS does not support custom rotation periods for managed keys. Annual is the AWS best practice. 90-day requires a custom Lambda + CloudWatch Events rule — overkill for v1.8. | 0.80 | Custom rotation Lambda (complex, overkill); no rotation (violates requirement). | +| **D-076** | uptime stack = separate VPC + ALB + ECS cluster (not shared with consumer stack). | Requirement says "separate terraform run, with a separate state". Sharing the consumer's VPC/ALB would couple the states. Separate infra is cleaner + isolates the uptime stack's lifecycle. | 0.85 | Share consumer's VPC/ALB (state coupling); use App Runner (new service type). | +| **D-077** | EFS volume for uptime-kuma `/app/data` (persistent storage across task restarts). | Fargate + EFS is the standard pattern for stateful containers. NFS is not supported by uptime-kuma, but EFS is NFS-compatible + works with Fargate. | 0.90 | S3-backed (uptime-kuma doesn't support S3); no persistent storage (data lost on restart). | + +--- + *End of RESEARCH.md. Path: `/root/acdl/.ciagent/RESEARCH.md`.* \ No newline at end of file