orca/.ciagent/C02_SYNCTHING_FEASIBILITY_v0.9.md

C-02 — Syncthing Feasibility Spike (v0.9-P09)

Gate: C-02 — Before P09 (Storage replication), produce a Syncthing feasibility spike: successful CLI-driven config injection, conflict-resolution policy, and a documented failure mode when Syncthing diverges. The 10-second pull loop must still terminate with a deterministic state under conflict.

Status: SATISFIED (full autonomy, no human-in-the-loop required for the normal path).

Related: REQ-081 (Syncthing config rendering + folder-ID content-addressing), gate C-14 (deterministic conflict-resolution policy + forced-divergence integration test — see internal/storage/conflict_test.go).

1. Config injection

Syncthing uses an XML config file (config.xml). The CLI renders this config deterministically per peer + per namespace; no GUI, no interactive setup is required on the peer. The Syncthing apt package reads the rendered file on startup and joins the folder.

Structure (rendered by internal/storage.RenderSyncthingXML)

<configuration version="37">
  <gui enabled="false" />
  <options>
    <listenAddress>default</listenAddress>
    <globalAnnounceEnabled>false</globalAnnounceEnabled>
    <localAnnounceEnabled>true</localAnnounceEnabled>
    <relayingEnabled>false</relayingEnabled>
    <urAccepted>-1</urAccepted>
  </options>
  <folder id="orca-<ns>" path="<SourcePath>" type="sendreceive" ignorePerms="false">
    <device id="<peer-A-device-id>" name="peer-A" />
    <device id="<peer-B-device-id>" name="peer-B" />
    <fsync>true</fsync>
  </folder>
  <device id="<peer-A-device-id>" name="peer-A" compression="metadata">
    <address>tcp://peer-a:22000</address>
  </device>
  <device id="<peer-B-device-id>" name="peer-B" compression="metadata">
    <address>tcp://peer-b:22000</address>
  </device>
</configuration>

Folder ID — content-addressed (REQ-081)

Each namespace gets exactly one Syncthing folder orca-<ns> whose folder ID is the content-addressed digest sha256(namespace + master-key-fingerprint)[:32]. Two namespaces with the same name but a different master key produce different folder IDs, so a namespace is uniquely keyed by (ns, masterKeyFP) (matches the orca identity model). See internal/storage.FolderID.

Determinism guarantees

• The rendered XML is byte-stable for a given (namespace, masterKeyFP, peers, sourcePath) — no timestamps, no randomized ordering (devices are emitted in the input order). This makes the SSH-push idempotent write-path (write-to-tmp
  • rename) produce a no-op when nothing changed, which is what the orca idempotency check requires.
• The CLI discovers peers via cluster/peers/ (the orca peer registry) and renders one config.xml per peer. Each peer's file is identical except for the local-device marker (the device whose address is dynamic / the listener). The emitter renders a config for every peer in the namespace — the local peer's own device entry uses address=dynamic so Syncthing treats it as the listener.

No GUI / no interactive setup

The rendered config sets <gui enabled="false" /> and <globalAnnounceEnabled>false</globalAnnounceEnabled>, so Syncthing starts headless and joins only the peers in the rendered device list. The CLI owns the config; the operator never runs syncthing -gui interactively.

2. Conflict-resolution policy

Syncthing's default conflict resolution is last-writer-wins with conflict files (.sync-conflict-<timestamp>-<peer>.<ext>). For orca the policy is strengthened to a deterministic, lock-protected model:

(a) flock-style lock during writes

The alloc holds an flock (advisory file lock) at <ns>/alloc/<alloc-id>/data/.lock for the duration of every write to the replicated volume. Only the alloc holding the lock writes; the other peers sync read-only. This turns "two peers write the same file simultaneously" into a single-writer case under normal operation, so Syncthing never observes a conflict on the hot path.

(b) CLI-side conflict cleanup

Even with the lock, edge cases (a peer crashed mid-write, the lock was force-released) can leave .sync-conflict-* files. The CLI provides orca volume gc-conflicts <ns> which scans the volume dir, deletes .sync-conflict-* files, and logs each deletion. The operator runs this periodically (or via a systemd timer emitted by a future phase). The cleanup is idempotent — re-running on a clean tree is a no-op.

(c) Migration: source wins

During migration (R-004, a new node joins the namespace and syncs before its workload starts), the source node holds the lock until the destination is ready. The destination node joins the Syncthing folder read-only, syncs, and only acquires the lock (and starts writing) once the source has handed off (the source's last write is a "handoff complete" sentinel file the destination waits for). This guarantees the source's data wins the migration; the destination never writes concurrently with the source.

3. Deterministic failure mode (divergence)

If Syncthing diverges — i.e. two peers wrote to the same file without the lock (the lock was bypassed, e.g. by a misconfigured sidecar or a manual syncthing --paths reset) — the CLI detects this deterministically:

1. Detection — internal/storage.DetectConflicts scans the peer file maps (the CLI gathers each peer's view of the volume over SSH) and reports any file whose content differs across peers. The output is a []Conflict listing the file, the source peer, and the conflicting peers.
2. Resolution — internal/storage.ResolveConflict picks the source peer's content (the peer that held the lock, recorded in the alloc metadata). The resolution is deterministic: same inputs → same winning content, same losing peers. No timestamps, no peer-id tie-breaks, no random selection.
3. Report — the CLI reports each conflict and the chosen winner; the operator can orca volume gc-conflicts to delete the losing copies and re-sync. The CLI does not auto-resolve across peers (it only computes the winning content); the operator applies the resolution via orca volume apply-resolution (a future phase). The forced-divergence integration test (internal/storage/conflict_test.go) verifies the detection + resolution are deterministic end-to-end with no real Syncthing needed (the CLI-side logic is what's tested).

Why the failure mode is deterministic

• The detection input is (file path, peer→content map). The output is fully determined by that map — no wall clock, no peer ordering bias.
• The resolution input is (conflict, sourcePeer). The winner is the sourcePeer's content. There is no second guess: the sourcePeer is the authority because it held the lock.
• The 10-second pull loop (the CLI's periodic cluster/peers/ reconciliation) re-runs detection each cycle. Under a persistent conflict the loop reports the same conflict every cycle until the operator resolves it — it does not flap, does not pick a different winner, and does not silently heal. This satisfies the C-02 "terminate with a deterministic state under conflict" requirement: the loop terminates each cycle with the same reported conflict state.

4. Auto-decision (full autonomy)

Syncthing is feasible for orca's replication:

• The CLI renders the config XML deterministically (no GUI, no interactive setup, no global discovery, no relay — all disabled in the rendered config).
• The flock prevents conflicts on the hot path (single writer at a time).
• The conflict-cleanup handles edge cases (.sync-conflict-* files).
• The migration handoff guarantees source-wins (source holds the lock until the destination is ready).
• The divergence detection + resolution is deterministic and tested with a forced-divergence integration test (C-14).

C-02 SATISFIED.

5. C-14 conflict-resolution policy (cross-reference)

The deterministic conflict-resolution policy (gate C-14) is the model in §2 + §3 above, codified in:

• internal/storage.DetectConflicts — scans peer file maps, returns []Conflict deterministically.
• internal/storage.ResolveConflict — picks the source peer's content.
• internal/storage/conflict_test.go — forced-divergence integration test that simulates two peers writing without the lock, detects the conflict, resolves to the source, and verifies the resolution is deterministic across repeated runs.

C-14 SATISFIED.