---
title: 'Orchestration, state, concurrency & recovery'
description: 'Separate durable session and state from inference and execution, and make recovery explicit.'
canonical_url: 'https://darkfactory.dev/factory/orchestration-state'
markdown_url: 'https://darkfactory.dev/factory/orchestration-state.md'
collection: factory
date_published: '2026-07-16T00:00:00-04:00'
date_modified: '2026-07-16T00:00:00-04:00'
---

# Orchestration, state, concurrency & recovery


**Confidence: high.** *Evidence: benchmarks, case studies, and theory.* *Last substantive change: 2026-08.*

Multiple agents share state, dependencies, and failure modes. The system must isolate their work, preserve what matters, and recover when a run crashes halfway through.

## The conclusion

**Separate durable session and state from model inference and from execution, and use append-only traces, leases and idempotency, checkpoints, isolated work, bounded retries, and explicit recovery semantics.** The brain, the hands, and the session are separable layers, and treating them separately is what makes concurrent, resumable work safe.

## How the thinking got here

Chat sessions gave way to persistent agent loops, then to orchestrated worktrees, then to a [decoupled model of inference, execution, and durable session state](https://addyosmani.com/blog/long-running-agents/) with formalized execution traces. [Bun's large parallel migration](https://bun.com/blog/bun-in-rust) surfaced physical limits: worktree isolation kept agents from clobbering each other, while disk space and resource limits became practical bottlenecks.

The newer label [graph engineering](https://www.turingpost.com/p/is-graph-engineering-real-why-everyone-is-talking-about-it) is useful when it makes this structure explicit. In a control graph, nodes may be ordinary code, model calls, tools, or complete agents, while edges decide what may run next. A loop is a cyclic graph, not an alternative to one. The practical question is not whether every agent needs a graph. It is which transitions, retries, joins, and gates are important enough to encode and inspect.

## Credible alternatives, and when each is right

| Approach | Right when |
|---|---|
| Stateless task runs | short, independent work |
| Long-lived conversation | continuity matters more than isolation |
| Event sourcing | replay and audit are required |
| Workflow engine | complex, long-running processes |
| Explicit control graph or state machine | branching, retries, pauses, and recovery must be enforceable |
| Blackboard or actor model | many collaborating agents |
| Isolated worktrees | concurrent edits to one repository |

## Where it fails and what we still don't know

Failures include crash inconsistency, concurrent semantic conflicts that merge cleanly but break behavior, partial failure, runaway loops, and graphs that hard-code a process before it is understood. Evidence strongly converges on externalized state and isolated work; there is less evidence on which distributed architecture is best. Open questions include exactly-once illusions, replay correctness, dynamic graph growth, and cross-organization coordination.

## What would change our mind

A dominant, well-evidenced architecture for multi-agent state and recovery would replace today's convergent-but-varied practices with a standard.

## Evidence and further reading

- [Runtime-Structured Task Decomposition](https://arxiv.org/abs/2605.15425)
- [Codex Orchestration (Symphony)](https://openai.com/index/open-source-codex-orchestration-symphony/)
- [Execution Lineage for Reproducible AI-Native Work](https://arxiv.org/abs/2605.06365)
- [Shepherd: A Runtime Substrate Empowering Meta-Agents with a Formalized Execution Trace](https://arxiv.org/abs/2605.10913)
- [Beyond Individual Intelligence (LIFE)](https://arxiv.org/abs/2605.14892)
- [Long-Running Agents](https://addyosmani.com/blog/long-running-agents/)
- [Rewriting Bun in Rust](https://bun.com/blog/bun-in-rust)
- [LangChain: 3 Years of Graph Engineering with LangGraph](https://www.langchain.com/blog/3-years-of-graph-engineering-with-langgraph)
- [Turing Post: Is Graph Engineering Real?](https://www.turingpost.com/p/is-graph-engineering-real-why-everyone-is-talking-about-it)