The Anti-Freeze Callback Protocol: Preventing Orphaned AI Agents and Stuck Workflows
Multi-agent orchestration is often sold as a way to do more work in parallel. The harder operational question is: how does the parent workflow know a worker actually finished? An agent can produce useful content and still leave its task marked as running. It can fail without returning a result, or stop responding. If the orchestration layer treats a conversational sign-off as completion, dependencies remain blocked while users wait.
The Anti-Freeze Callback Protocol addresses that gap with a reliability principle: work is not done until its task state is explicitly reported. In Koios, workers report completion or failure through a structured update-task-status callback. Combined with dependency-aware task graphs and watchdog recovery, explicit state turns a loose collection of agent conversations into observable workflow execution.
Why multi-agent workflows freeze
A directed acyclic graph defines dependencies and ordering, but it cannot guarantee each task reaches a terminal state. A parent may depend on several workers; if one never signals completion, the downstream task correctly remains blocked, yet the workflow appears frozen.
Typical causes include process interruption, provider timeouts, lost wakeups, a worker returning natural-language prose instead of updating task state, or a mismatch between work performed and scheduler state. In each case, the scheduler lacks an authoritative terminal event.
| Symptom | State problem | Consequence |
|---|---|---|
| Worker says “done” in chat | No structured transition | Dependency remains open |
| Worker exits unexpectedly | No success or failure callback | Task may remain active |
| Failure appears only in prose | Error not persisted to task record | Retry and escalation cannot act reliably |
In a single-agent ReAct loop, one runtime can observe a tool result and continue. Distributed execution changes the boundary: worker and parent are separate actors. The task record—not an implied conversation state—must coordinate them.
The callback is the contract, not a courtesy
Task reporting belongs in the worker’s definition of done. After finishing its unit, the worker invokes a structured status callback for the correct task, with a terminal status and concise result or failure explanation. A friendly final message is not a substitute for that control-plane event.
- Execute bounded work. Give the worker clear inputs, output location, and acceptance criteria.
- Persist the result. Write the deliverable or record a concrete failure reason; do not rely on transient chat text.
- Report terminal state. Call
update-task-statusfor the matching task with the appropriate success or failure outcome. - Resume orchestration. Reevaluate dependencies and wake the parent when the state change permits progress.
The artifact is the work product; the callback is the control-plane event. Keeping both makes it possible to inspect the output and independently determine whether the workflow may proceed.
receive task and acceptance criteria
↓
execute work → persist output
↓
update-task-status(task_id, terminal_status, summary)
↓
scheduler reevaluates dependencies
↓
parent resumes, retries, or escalatesDiscover the exact payload schema from the active runtime rather than guessing. The invariant is stable: status updates must identify the correct task and be issued before the worker terminates.
Task contracts make callbacks useful
A callback cannot rescue an ambiguous assignment. If there is no testable acceptance criterion, “succeeded” is only a label. A reliable task contract specifies inputs, output shape and destination, required fields, and what counts as failure. For parallel work, each child needs its own task identity and result location. The parent should map each dependency to satisfied, explicitly failed, or still running.
| Weak assignment | Reliable assignment |
|---|---|
| “Research this and tell me when finished.” | Name the sources, deliverable path, required fields, and success test. |
| “Return a result.” | Define result shape, failure reporting, and task ID. |
| “Try again if needed.” | Define retry owner, attempt limit, and escalation. |
Watchdogs recover missing callbacks
Callback discipline reduces orphaned work, but distributed systems must assume some workers disappear. A watchdog provides a second line of defense by inspecting state and elapsed time. Koios platform materials describe a background reconciler that checks task queues every 10 seconds, applies a five-minute ceiling to timed-out jobs, and can trigger wakeups when subtasks finish. A watchdog does not prove output correctness; it prevents a missing callback from becoming an unbounded wait.
A timeout should produce a visible terminal outcome and diagnostic—not silently erase the task. The system can then hand it to a bounded retry or operator review. Preserve distinctions among worker failure, timeout, and success; those outcomes support meaningful reliability metrics and reduce accidental duplicate side effects.
Timeouts, retries, and idempotency
Retries are safe only when the task is idempotent or its side effects are controlled. If a worker publishes a post or creates an external record before freezing, a retry may repeat the action. For side-effecting tasks, use an idempotency key or a preflight lookup and verify external state before retrying. Pure transformations are easier to retry, but still need attempt limits and backoff to avoid amplifying provider incidents.
Set time budgets that reflect task size and distinguish a task deadline from an individual model-call timeout. A useful timeout record includes task ID, attempt, start time, last progress if available, dependency context, and failure category. That makes watchdog recovery actionable rather than an unexplained red status.
Failure handling must preserve the truth
A worker should report failure as clearly as success: what it attempted, what it persisted, which criteria were not met, and whether retry is likely to help. Do not label a partial artifact complete. Parents should treat callbacks as control signals, not unquestionable proof; validate key artifacts before releasing dependent tasks. This defense in depth catches cases where a callback was emitted but the output was not saved.
Measure workflow health, not just completion rate
- Share of tasks reaching a terminal state within their budget.
- Callback delivery failures and tasks repaired by watchdog reconciliation.
- Timeout and retry rates by task type and provider.
- Time from child terminal transition to parent wakeup.
- Duplicate side effects during retries.
- Successful callbacks whose artifacts fail downstream validation.
These measures reveal whether a workflow is resilient or merely hides stalled work. They also help distinguish model latency, external tool failure, worker runtime issues, and weak task contracts.
Callback-governed execution versus best-effort swarms
| Best-effort orchestration | Callback-governed orchestration |
|---|---|
| Completion inferred from chat text | Terminal state persisted against task ID |
| Silent exits leave ambiguous state | Watchdog turns stale work into a visible timeout |
| Retries repeat uncontrolled actions | Attempts, idempotency, and escalation are explicit |
| Failures buried in prose | Structured outcomes support triage |
Where the Anti-Freeze Callback Protocol fits
The callback protocol is one layer in an Enterprise AI Harness. Task graphs model dependencies; workers execute bounded units; explicit state preserves the control plane; watchdog reconciliation limits damage from missing callbacks. The system does not promise agents never fail. It makes failure visible, bounded, and recoverable.
For engineering leaders, ask: if a worker disappears immediately after its last message, can the parent determine whether it succeeded, failed, or timed out—and what to do next? If the answer depends on manually reading a transcript, the workflow needs a stronger state contract.
Rollout checklist
- Give every worker a unique task ID and narrow acceptance criteria.
- Require durable output before a success callback.
- Do not accept chat sign-off as task state.
- Apply bounded deadlines and watchdog recovery.
- Make side-effecting tasks idempotent before automatic retry.
- Validate key artifacts before downstream execution.
- Log callback failures, timeout causes, retries, and wakeup latency.
- Test crashes before and after output persistence.
Multi-agent reliability comes from disciplined state transitions, not simply adding more agents. Make completion explicit, observable, and recoverable—and a stalled worker becomes a diagnosable event rather than a workflow that quietly freezes.
