orca/.ciagent/RESEARCH_v0.16.md

# RESEARCH v0.16: Release Binary Asset Fix

## Root Cause: Two Compounding Bugs

### Bug 1: Gitea Actions workflow cannot clone private coreci repo

`.gitea/workflows/release.yml` step "Install CoreCI" runs:
```bash
git clone --depth=1 https://git.cloudinit.dev/coreci/coreci.git /tmp/coreci
```

The `coreci` repo is **private** (confirmed: `curl -s -o /dev/null -w "%{http_code}" https://git.cloudinit.dev/coreci/coreci` returns 404 without auth; the Gitea API reports `"private": true`).

The `actions/checkout@v4` step injects auth only for the orca repo:
```
[command]/usr/bin/git config --local http.https://git.cloudinit.dev/.extraheader AUTHORIZATION: basic ***
```

The subsequent bare `git clone` of the coreci repo has no credentials. Runner logs confirm the failure:
```
fatal: could not read Username for 'https://git.cloudinit.dev': No such device or address
❌  Failure - Main Install CoreCI
exitcode '128': failure
```

Both Gitea Actions runs (v0.14.1 run #3540, v0.14.2 run #3544) failed at this step. The `container-orca` and `container-traefik` jobs (with `needs: ci`) were skipped. `coreci run` never executed.

**Fix (REQ-183):** Pass `GITEA_TOKEN: ${{ secrets.PAT_TOKEN }}` as env to the "Install CoreCI" step and embed it in the clone URL:
```bash
git clone --depth=1 https://cloudinit-bot:${GITEA_TOKEN}@git.cloudinit.dev/coreci/coreci.git /tmp/coreci
```

### Bug 2: .coreci.yml uses a format CoreCI does not understand

The orca `.coreci.yml` uses:
```yaml
pipelines:
  validate:
    steps:
      - name: go-version
        image: golang:1.25.12
        commands: [go version, gofmt -l ., go vet ./...]
  release:
    when:
      ref: "refs/tags/v*"
    steps: [...]
```

CoreCI's actual format (validated against `workflows/pipeline/pipeline.go`):
```go
type Pipeline struct {
    Jobs     map[string]Job     `yaml:"jobs"`
    Services map[string]Service `yaml:"services,omitempty"`
    Env      EnvBlock           `yaml:"env,omitempty"`
}
type Job struct {
    Needs  []string `yaml:"needs,omitempty"`
    Plugin string   `yaml:"plugin,omitempty"`
    Invoke string   `yaml:"invoke,omitempty"`
    Vars   map[string]string `yaml:"vars,omitempty"`
    // ...
}
```

`yaml.Unmarshal` into a struct silently drops unknown fields (no `KnownFields(true)`). The `pipelines:` top-level key doesn't map to any struct field → `Jobs` is an empty map. CoreCI's `validate()` does NOT reject empty jobs:
```go
func validate(p *Pipeline) error {
    seen := make(map[string]struct{}, len(p.Jobs))
    for name, job := range p.Jobs { ... }
    return nil  // empty Jobs → no iterations → nil error
}
```

So `coreci run` loads an empty pipeline, runs zero jobs, and exits successfully — no build, no tarball, no asset upload.

Additional format mismatches in the current `.coreci.yml`:
- `when: ref: "refs/tags/v*"` — CoreCI has no `when` at pipeline/job level; it uses `rules: [{if: "CI_COMMIT_TAG"}]` on jobs. And CoreCI's `github.go` doesn't set `CI_COMMIT_TAG` for Gitea Actions. But the Gitea Actions workflow already gates on `on: push: tags: ['v*']`, so no conditional is needed.
- `${CI_COMMIT_TAG:-dev}` in `env:` blocks — CoreCI does not expand `${VAR}` at YAML parse time. Only `${{ secrets.KEY }}` is interpolated. Shell expansion works inside `invoke:` via `sh -c`.
- `GITEA_TOKEN: ${GITEA_TOKEN}` in `env:` blocks — CoreCI's `env` block doesn't do env-var interpolation. Secrets must go in job `vars:` via `${{ secrets.GITEA_TOKEN }}`.

**Fix (REQ-184):** Full rewrite to CoreCI native `jobs:` format with a proper DAG.

## CoreCI Execution Model (validated against source)

### Executor Chain
Default: `podman,docker,shell-isolated` (`NewChainExecutorFromConfig`).
- `ErrExecutorUnavailable` (executor missing) → fall through to next.
- `ErrExecutorFailed` (non-zero exit) → stop immediately, no fallback.

### Job Execution with `plugin` + `invoke`
A job with BOTH `plugin: docker://golang:1.25` AND `invoke: "go build ..."`:
- **Podman/Docker executor**: pulls image, runs `podman/docker run ... <image> sh -c "<invoke>"`. Env vars from the `env` map are injected via `-e K=V`.
- **Shell-isolated executor**: checks `invoke` first (line 52). If non-empty, runs `sh -c "<invoke>"` directly, ignoring the `plugin` image. This is the fallback when no container runtime is available.
- This dual pattern is correct: container if available, shell fallback if not.

### Environment Variables in Jobs
`buildIsolatedEnv` (`isolated_shell.go:34`):
```go
func buildIsolatedEnv(env map[string]string) []string {
    var out []string
    for _, key := range envAllowlist { // PATH, HOME, LANG, TMPDIR, TERM, CI
        if val, ok := os.LookupEnv(key); ok { out = append(out, ...) }
    }
    for _, kv := range os.Environ() {
        if strings.HasPrefix(kv, "CORECI_") || strings.HasPrefix(kv, "CI_") { out = append(out, kv) }
    }
    for k, v := range env { out = append(out, fmt.Sprintf("%s=%s", k, v)) }  // env map
    out = append(out, "CORECI_SANDBOX=isolated-shell")
    return out
}
```

The `env` map (3rd loop) is built in `run.go` from:
1. `cicontext.Detect()` → CI context (`CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`, etc.)
2. `envfwd.PassThroughEnv()` → user-defined host env vars NOT in deny-list
3. Job `vars` (with `${{ secrets.* }}` interpolation)

So in a job's `invoke:` script, these env vars are available:
- `CI_COMMIT_BRANCH` — tag name on tag push (from `GITHUB_REF_NAME`)
- `CI_COMMIT_SHA` — commit SHA
- `GITEA_TOKEN` — forwarded by PassThroughEnv (not in deny-list)
- `CORECI_*` prefixed vars — always forwarded
- `CI_*` prefixed vars — always forwarded
- Any job `vars` values

### Secret Resolution
`run.go:146-153`:
```go
resolver := func(key string) (string, error) {
    if v, ok := secretMap[key]; ok { return v, nil }  // local DB first
    if v := os.Getenv(key); v != "" { return v, nil } // then env
    return "", secrets.ErrMissingKey
}
```
Secret interpolation `${{ secrets.GITEA_TOKEN }}` in job `vars:` is resolved by checking the local CoreCI secret store first, then `os.Getenv`. Since `GITEA_TOKEN` is in the env (from the Gitea Actions step env), `os.Getenv("GITEA_TOKEN")` succeeds.

## Gitea Actions CI Context

On a tag push (`refs/tags/v0.15.1`), Gitea Actions sets:
- `GITHUB_ACTIONS=true`
- `GITHUB_REF=refs/tags/v0.15.1`
- `GITHUB_REF_NAME=v0.15.1`
- `GITHUB_REF_TYPE=tag`
- `GITHUB_SHA=<commit>`

CoreCI's `github.go` maps:
- `GITHUB_ACTIONS=true` → `isGitHub()` returns true → `IsRunningInCI()` true
- `GITHUB_SHA` → `CI_COMMIT_SHA`
- `GITHUB_REF_NAME` → `CI_COMMIT_BRANCH` (= `v0.15.1` on tag push)

`PassThroughEnv()` runs (CI detected). `GITHUB_REF` is in `predefinedCIVars` (NOT forwarded), but `GITEA_TOKEN` is NOT in any deny-list → forwarded.

## Conclusion

The fix is two file changes:
1. `.gitea/workflows/release.yml` — auth the coreci clone (embed token in URL)
2. `.coreci.yml` — rewrite to CoreCI native `jobs:` format with DAG

Both are validated against CoreCI source code and docs. No API calls needed — `coreci run` executes the `.coreci.yml` pipeline locally on the runner.