The org-wide circuit breaker of the AI co-worker control plane opened after three mislabeled
intake issues and could not be closed: no reset path existed in production. Every wake event
since Aug 12 died with organization circuit is open. This PR makes recovery
self-service through the existing authenticated GitHub label surface and removes the failure
mode that opened the breaker in the first place.
Ekho-Labs/ai-coworker is the Go control plane for an event-driven AI co-worker.
GitHub issues in Ekho-Labs/infra-work are the sole human control
surface, driven entirely by labels.
The organization-wide breaker opens after 3 consecutive failed runs and blocks every new claim. Deterministic human input errors counted the same as real failures — and once open, there was no way back:
"The current binaries don't expose a reset HTTP endpoint, so treat an open breaker as an operator-blocking condition until an authorized reset is available." — docs/operations.md (before this PR)
circuit.Service.Reset existed, fully wired with an authorizer —
nothing in production called it.
1. A fifth designated control label, agent:circuit-reset: an authorized
human (triage-or-higher and control-team member) adds it to any infra-work issue;
the controller closes the breaker organization-wide. No generation claimed, nothing spent.
2. Breaker counting is now explicit per call site: deterministic intake rejections
(blockRun) never touch the breaker; autonomous planner / worker / publication /
verification failures (failRun) still count.
All timestamps UTC. Failure threshold: 3 consecutive failed runs.
agent:ready with no kind label
→ run blocked: intake issue must have exactly one kind label
circuit_states)
circuit openagent:ready + kind:research added to #202
(Historical Port Traffic Data) → denied: organization circuit is openUPDATE circuit_states SET open=false, consecutive_failures=0
→ org unblocked circuit closedSwimlanes: GitHub → Intake → Controller → Circuit state (Postgres). Violet marks paths added by this PR; hover a node to inspect it.
agent:circuit-reset + designated setCircuit.Reset (re-authorized by the circuit
service) · unlabeled = no-op · org-wide · no claim, no spend ·
idempotent for webhook replayEnsureClosed gate (rejects while open) → classify the
intake issueintake_rejected event →
blocked + diagnostic + budgets settled · BREAKER UNTOUCHEDRecordResult(failed)RecordResult(succeeded) → failure count resets to 0open flag · consecutive_failures ·
threshold 3 → OPEN · read by EnsureClosed before every claimcircuit.Service.Reset re-authorizes through the
IntakeResetAuthorizer before touching state.intake_rejected event; transition
{classifying, intake_rejected} → {blocked, no_automatic_retry}. Kept distinct from
model_task_failure so it can never count toward the breaker.CircuitResetControlLabel handled before claim (no
generation, no spend); Circuit interface gains
Reset; block split into
blockRun / failRun; arms and collect call
sites now count explicitly. Unit tests: labeled → Reset forwarded with org + actor;
unlabeled → no-op.circuit.Service.RecordResult and the CircuitStore use
load-modify-save without a transaction; concurrent results or resets can lose updates.
Follow-up: make store mutations atomic.
source: oracle architecture review
The status-projection label exists only in the docs; no code applies it when the breaker opens. Pre-existing gap, out of scope for this PR.
source: oracle architecture review
The running controller image predates this fix; agent:circuit-reset
activates on the next deploy. The production circuit row was already closed by operator
break-glass, so the org is not blocked meanwhile.
Adding agent:circuit-reset while the breaker is closed only
zeroes the failure count. Harmless and idempotent.