orca/.ciagent/RESEARCH_v0.16.md

RESEARCH v0.16: Release Binary Asset Fix

Root Cause: Two Compounding Bugs

Bug 1: Gitea Actions workflow cannot clone private coreci repo

.gitea/workflows/release.yml step "Install CoreCI" runs:

git clone --depth=1 https://git.cloudinit.dev/coreci/coreci.git /tmp/coreci

The coreci repo is private (confirmed: curl -s -o /dev/null -w "%{http_code}" https://git.cloudinit.dev/coreci/coreci returns 404 without auth; the Gitea API reports "private": true).

The actions/checkout@v4 step injects auth only for the orca repo:

[command]/usr/bin/git config --local http.https://git.cloudinit.dev/.extraheader AUTHORIZATION: basic ***

The subsequent bare git clone of the coreci repo has no credentials. Runner logs confirm the failure:

fatal: could not read Username for 'https://git.cloudinit.dev': No such device or address
❌  Failure - Main Install CoreCI
exitcode '128': failure

Both Gitea Actions runs (v0.14.1 run #3540, v0.14.2 run #3544) failed at this step. The container-orca and container-traefik jobs (with needs: ci) were skipped. coreci run never executed.

Fix (REQ-183): Pass GITEA_TOKEN: ${{ secrets.PAT_TOKEN }} as env to the "Install CoreCI" step and embed it in the clone URL:

git clone --depth=1 https://cloudinit-bot:${GITEA_TOKEN}@git.cloudinit.dev/coreci/coreci.git /tmp/coreci

Bug 2: .coreci.yml uses a format CoreCI does not understand

The orca .coreci.yml uses:

pipelines:
  validate:
    steps:
      - name: go-version
        image: golang:1.25.12
        commands: [go version, gofmt -l ., go vet ./...]
  release:
    when:
      ref: "refs/tags/v*"
    steps: [...]

CoreCI's actual format (validated against workflows/pipeline/pipeline.go):

type Pipeline struct {
    Jobs     map[string]Job     `yaml:"jobs"`
    Services map[string]Service `yaml:"services,omitempty"`
    Env      EnvBlock           `yaml:"env,omitempty"`
}
type Job struct {
    Needs  []string `yaml:"needs,omitempty"`
    Plugin string   `yaml:"plugin,omitempty"`
    Invoke string   `yaml:"invoke,omitempty"`
    Vars   map[string]string `yaml:"vars,omitempty"`
    // ...
}

yaml.Unmarshal into a struct silently drops unknown fields (no KnownFields(true)). The pipelines: top-level key doesn't map to any struct field → Jobs is an empty map. CoreCI's validate() does NOT reject empty jobs:

func validate(p *Pipeline) error {
    seen := make(map[string]struct{}, len(p.Jobs))
    for name, job := range p.Jobs { ... }
    return nil  // empty Jobs → no iterations → nil error
}

So coreci run loads an empty pipeline, runs zero jobs, and exits successfully — no build, no tarball, no asset upload.

Additional format mismatches in the current .coreci.yml:

• when: ref: "refs/tags/v*" — CoreCI has no when at pipeline/job level; it uses rules: [{if: "CI_COMMIT_TAG"}] on jobs. And CoreCI's github.go doesn't set CI_COMMIT_TAG for Gitea Actions. But the Gitea Actions workflow already gates on on: push: tags: ['v*'], so no conditional is needed.
• ${CI_COMMIT_TAG:-dev} in env: blocks — CoreCI does not expand ${VAR} at YAML parse time. Only ${{ secrets.KEY }} is interpolated. Shell expansion works inside invoke: via sh -c.
• GITEA_TOKEN: ${GITEA_TOKEN} in env: blocks — CoreCI's env block doesn't do env-var interpolation. Secrets must go in job vars: via ${{ secrets.GITEA_TOKEN }}.

Fix (REQ-184): Full rewrite to CoreCI native jobs: format with a proper DAG.

CoreCI Execution Model (validated against source)

Executor Chain

Default: podman,docker,shell-isolated (NewChainExecutorFromConfig).

• ErrExecutorUnavailable (executor missing) → fall through to next.
• ErrExecutorFailed (non-zero exit) → stop immediately, no fallback.

Job Execution with plugin + invoke

A job with BOTH plugin: docker://golang:1.25 AND invoke: "go build ...":

• Podman/Docker executor: pulls image, runs podman/docker run ... <image> sh -c "<invoke>". Env vars from the env map are injected via -e K=V.
• Shell-isolated executor: checks invoke first (line 52). If non-empty, runs sh -c "<invoke>" directly, ignoring the plugin image. This is the fallback when no container runtime is available.
• This dual pattern is correct: container if available, shell fallback if not.

Environment Variables in Jobs

buildIsolatedEnv (isolated_shell.go:34):

func buildIsolatedEnv(env map[string]string) []string {
    var out []string
    for _, key := range envAllowlist { // PATH, HOME, LANG, TMPDIR, TERM, CI
        if val, ok := os.LookupEnv(key); ok { out = append(out, ...) }
    }
    for _, kv := range os.Environ() {
        if strings.HasPrefix(kv, "CORECI_") || strings.HasPrefix(kv, "CI_") { out = append(out, kv) }
    }
    for k, v := range env { out = append(out, fmt.Sprintf("%s=%s", k, v)) }  // env map
    out = append(out, "CORECI_SANDBOX=isolated-shell")
    return out
}

The env map (3rd loop) is built in run.go from:

1. cicontext.Detect() → CI context (CI_COMMIT_SHA, CI_COMMIT_BRANCH, etc.)
2. envfwd.PassThroughEnv() → user-defined host env vars NOT in deny-list
3. Job vars (with ${{ secrets.* }} interpolation)

So in a job's invoke: script, these env vars are available:

• CI_COMMIT_BRANCH — tag name on tag push (from GITHUB_REF_NAME)
• CI_COMMIT_SHA — commit SHA
• GITEA_TOKEN — forwarded by PassThroughEnv (not in deny-list)
• CORECI_* prefixed vars — always forwarded
• CI_* prefixed vars — always forwarded
• Any job vars values

Secret Resolution

run.go:146-153:

resolver := func(key string) (string, error) {
    if v, ok := secretMap[key]; ok { return v, nil }  // local DB first
    if v := os.Getenv(key); v != "" { return v, nil } // then env
    return "", secrets.ErrMissingKey
}

Secret interpolation ${{ secrets.GITEA_TOKEN }} in job vars: is resolved by checking the local CoreCI secret store first, then os.Getenv. Since GITEA_TOKEN is in the env (from the Gitea Actions step env), os.Getenv("GITEA_TOKEN") succeeds.

Gitea Actions CI Context

On a tag push (refs/tags/v0.15.1), Gitea Actions sets:

• GITHUB_ACTIONS=true
• GITHUB_REF=refs/tags/v0.15.1
• GITHUB_REF_NAME=v0.15.1
• GITHUB_REF_TYPE=tag
• GITHUB_SHA=<commit>

CoreCI's github.go maps:

• GITHUB_ACTIONS=true → isGitHub() returns true → IsRunningInCI() true
• GITHUB_SHA → CI_COMMIT_SHA
• GITHUB_REF_NAME → CI_COMMIT_BRANCH (= v0.15.1 on tag push)

PassThroughEnv() runs (CI detected). GITHUB_REF is in predefinedCIVars (NOT forwarded), but GITEA_TOKEN is NOT in any deny-list → forwarded.

Conclusion

The fix is two file changes:

1. .gitea/workflows/release.yml — auth the coreci clone (embed token in URL)
2. .coreci.yml — rewrite to CoreCI native jobs: format with DAG

Both are validated against CoreCI source code and docs. No API calls needed — coreci run executes the .coreci.yml pipeline locally on the runner.