Agent Harnesses: How LLMs Become Autonomous Software Engineers

Every agent harness, from a 200-line script to OpenClaw's five-component daemon, implements some variant of the same core loop: the model reasons about what to do, acts by calling a tool, observes the result, and repeats. This pattern was formalized as ReAct by Yao et al. (ICLR 2023), though practitioners had been building ad-hoc versions since GPT-3.5 gained function-calling capabilities.

The insight behind ReAct is deceptively simple: interleaving reasoning traces with actions outperforms either alone. Pure reasoning (chain-of-thought) hallucinates when it lacks external information. Pure acting (tool calls without reasoning) makes incoherent sequences of actions. ReAct weaves them together: the model writes a thought ('I need to find the auth middleware file'), generates an action (search the codebase), observes the result (file found at src/middleware/auth.ts), reasons about what to do next ('The bug is in the token validation logic on line 47'), and acts again (edit the file). This think-act-observe cycle is now the beating heart of every production coding agent.

But ReAct alone is not enough to build a useful agent. The paper used simple text-based environments (web search, interactive fiction). Real software engineering requires: (1) tool execution — safely running shell commands, editing files, invoking compilers and test suites; (2) context management — keeping track of relevant code, conversation history, and project structure within a finite context window; (3) permission enforcement — preventing the agent from deleting production databases or pushing malicious code; (4) state persistence — maintaining memory across interactions. The agent harness is the engineering layer that provides all of this around the core ReAct loop.

CodeAct (Wang et al., ICML 2024) pushed the action space further by replacing JSON tool calls with executable Python code. Instead of the model outputting a structured function call that the harness routes to a predefined tool, the model writes arbitrary Python that runs in a persistent interpreter. This enables dynamic logic — loops, conditionals, error handling, self-debugging — that JSON schemas cannot express. CodeAct showed up to 20% improvement over JSON-based actions on agent benchmarks, and the pattern was adopted by OpenHands, one of the most prominent open-source coding agent platforms.

The evolution from ReAct to CodeAct to production harnesses follows a clear trajectory: give the model more expressive ways to interact with the environment, while building increasingly sophisticated safety guardrails around those interactions.

Before you can build a good harness, you need to measure what 'good' means. SWE-bench (Jimenez et al., ICLR 2024) established the gold standard: 2,294 real GitHub issues from 12 popular Python repositories. Given a codebase and an issue description, the agent must produce a patch that passes the repository's existing test suite. No synthetic tasks — every issue and test is real. At release, GPT-4 solved fewer than 2% of tasks; the benchmark instantly revealed how far models were from practical software engineering.

SWE-bench spawned an ecosystem of variants — SWE-bench Lite (300 tasks), SWE-bench Verified (500 human-validated tasks), and extensions to other languages — and became the universal yardstick for coding agent evaluation. Every harness in this review reports SWE-bench numbers: Codex at 72.1% (pass@1 on Verified), Claude Code at 80.9%, OpenHands at 53%.

SWE-agent (Yang et al., NeurIPS 2024) made the crucial observation that benchmark performance depends not just on the model but on the Agent-Computer Interface (ACI) — the set of tools and interaction patterns the harness provides. The authors designed a custom ACI with purpose-built commands: a windowed file viewer (showing 100 lines at a time with line numbers), a search tool that returns context around matches, and a structured edit command that validates syntax before applying changes. These tools are optimized for LLM consumption — concise, unambiguous, with structured error feedback.

The result was striking: ACI design mattered more than model size. SWE-agent achieved 12.5% on SWE-bench Full (SOTA at publication) and the gap between good and bad ACIs was larger than the gap between model generations. This finding has profound implications for harness engineering: the tools you give the model are as important as the model itself. Every production harness has internalized this lesson — Claude Code's ~40 tools with structured output formats, Codex's sandboxed filesystem utilities with streaming output, OpenClaw's skill-based tool injection system.

Agentless (Xia et al., 2024) provided the essential counterpoint. Rather than building a complex agent loop, Agentless uses a simple three-phase pipeline: (1) hierarchical localization (narrow from repository to file to function), (2) patch generation, (3) filtering candidates by running tests. No tool use, no iterative loop, no agent state. The result: 32% on SWE-bench Lite at $0.70 per task — competitive with agent-based systems that cost orders of magnitude more. Agentless forces the field to justify every layer of harness complexity: if a pipeline without an agent loop can match your agent, your harness is adding cost without adding capability.

OpenHands (Wang et al., ICLR 2025) synthesized these insights into a comprehensive open-source platform. Agents run inside sandboxed Docker containers with access to shell commands, file editing, web browsing, and APIs. The platform uses CodeAct as its action layer and supports 15 benchmarks for evaluation. OpenHands achieved #1 on SWE-bench Full (29%) and 53% on Verified at publication, demonstrating that combining a well-designed ACI with a robust execution sandbox and the code-action paradigm produces a highly capable open-source coding agent.

OpenClaw (originally Clawdbot, then briefly Moltbot) is the most popular open-source agent harness by GitHub stars (~247K as of March 2026). Created by Austrian developer Peter Steinberger, it takes a fundamentally different architectural approach from coding-focused agents: OpenClaw treats the agent as a persistent operating system process — a daemon that runs continuously, reacts to messages from multiple channels, and autonomously executes tasks on a schedule.

The Five-Component Architecture. OpenClaw's design has five clearly delineated components:

1. Gateway — The central daemon process and single source of truth for sessions and routing. It normalizes inputs from multiple channels (Slack, Discord, WhatsApp, Telegram, etc.) into a unified internal message format, authenticates inbound messages, and proxies outbound API calls to the configured LLM provider.

2. Brain (Agent Runner) — The orchestration layer that implements a five-step pipeline: (a) select the correct agent for the incoming session, (b) resolve the cheapest model that satisfies context requirements (with automatic key rotation on rate limits), (c) build the prompt dynamically from SOUL.md, skills, memories, and tool definitions, (d) guard context by enforcing the model's context window limit via summarization or truncation, (e) execute the ReAct loop until the model produces a terminal response.

3. Memory — All persistent context is stored as plain Markdown files on local disk (~/.openclaw/). No embedded database, no vector store. The directory structure includes preferences.md, contacts.md, projects.md, learnings.md, and per-agent SOUL.md files. This is a deliberate trade-off: human-auditable and portable at the cost of retrieval sophistication.

4. Skills — OpenClaw's plugin/tool system. Each skill is a Markdown file with YAML frontmatter declaring a name, trigger pattern, and available tools. Skills are loaded at agent startup and injected into the system prompt. The community Skills Registry has grown to 5,700+ entries.

5. Heartbeat — The autonomy mechanism. Every 30 minutes (configurable), the Gateway sends the agent a scheduled trigger. The agent reads HEARTBEAT.md — a checklist of standing tasks — and decides whether any item requires action. If nothing needs doing, the agent responds with HEARTBEAT_OK and the reply is silently dropped. This built-in scheduling is unique among agent frameworks — competitors require external cron or workflow triggers.

The Serial Lane Queue. Messages within a session are processed one at a time through a Lane Queue. This prevents tool conflicts and keeps session history consistent, but limits throughput. Parallelism is opt-in and restricted to explicitly marked low-risk tasks. This is a significant architectural choice that prioritizes correctness over speed.

The Security Problem. OpenClaw's rapid growth exposed serious security issues. Between January 31 and February 25, 2026, 190 security advisories were filed — including combinations that compose into unauthenticated remote code execution paths. Four arXiv papers appeared in March 2026 analyzing the vulnerabilities. The flat permission model (agents have broad tool access by default) creates substantial attack surface when LLM reasoning is connected to host execution. The ecosystem responded with third-party hardening wrappers: IronClaw (Rust/WASM isolation by NEAR AI), NemoClaw (kernel-level sandboxing by NVIDIA), and KubeClaw (Kubernetes JIT RBAC). These are downstream fixes, not upstream architecture changes.

Identity via SOUL.md. OpenClaw uses a Markdown file called SOUL.md as an identity layer injected into every system prompt. The official docs distinguish it from a system prompt: 'System prompts tell models what to do; soul files tell them who to be.' Companion files handle style (STYLE.md), operating modes (SKILL.md), and session memory (MEMORY.md).

Claude Code is Anthropic's agentic coding system — a harness that wraps Claude models with ~40 tools, a layered permission model, and multi-agent coordination capabilities. Released as a limited research preview in February 2025 and reaching general availability in May 2025, it rapidly became one of the most commercially successful AI products, surpassing $1B in annualized revenue by late 2025.

The Three-Phase Loop. Claude Code's core execution model iterates through three phases: (1) gather context — read files, search codebases, inspect error logs; (2) take action — edit files, run commands, call external services; (3) verify results — run tests, check outputs, course-correct. The orchestrator's behavior is defined in natural language in the system prompt, not in branching code logic — enabling behavioral iteration without redeployment.

Tool Architecture. Approximately 40 tools are organized into categories: file operations (Read, Write, Edit, Glob), search (Grep, pattern-based file search), execution (Bash), web (WebSearch, WebFetch), and orchestration (Agent for spawning subagents, AskUserQuestion). Read-only operations run concurrently; mutating operations execute serially to prevent conflicts. Each tool is independently sandboxed with configurable access controls. External tools connect via the Model Context Protocol (MCP), with tool definitions deferred by default to save context space — only tool names are loaded until a tool is actually used.

The Permission Model. This is Claude Code's most distinctive design choice. Four permission modes provide graduated autonomy: (1) Default — Claude asks before file edits and shell commands; (2) Auto-accept edits — file edits auto-approved, commands still require confirmation; (3) Plan mode — read-only tools only, generates a plan for user approval; (4) Auto mode (research preview) — a second, separate LLM call evaluates each proposed action to predict whether the user would approve it. This is architecturally significant: the safety evaluation is not the same inference that generated the action. It is a separate model call specifically for permission prediction.

Context Engineering. The QueryEngine (the core orchestration engine, revealed in a March 2026 source leak to be ~46,000 lines of TypeScript) manages five fallback context compaction strategies deployed sequentially: time-based clearing of outdated tool outputs, conversation summarization, session memory extraction, full history summarization, and oldest-message truncation. Project context enters via CLAUDE.md files (project-specific instructions loaded at session start) and an auto-memory system that saves learnings as work progresses.

Subagents. Claude Code can spawn subagent instances via the Agent tool. Each subagent receives its own system prompt, a restricted tool subset, and the project's CLAUDE.md, but does not receive the parent's conversation history or system prompt. Only the subagent's final message returns to the parent — intermediate tool calls stay isolated, preventing context contamination. Three execution models exist: Fork (byte-identical parent context copy), Teammate (separate terminal pane with file-based mailbox), and Worktree (git worktree with isolated branches). Subagents cannot spawn their own subagents, limiting recursion depth.

Agent Teams. The experimental multi-agent feature (enabled via feature flag) is architecturally distinct from subagents. A team lead spawns teammates — each a full Claude Code instance with its own context window — that coordinate through a shared task list (~/.claude/tasks/{team-name}/) and a file-based mailbox messaging system. Task claiming uses file locking to prevent race conditions. Teammates can be required to plan before implementing; the lead reviews and approves plans autonomously. This is fundamentally a distributed systems problem being solved with filesystem primitives — simple and pragmatic, but with known limitations around session resumption and task status consistency.

Hooks. External code can intercept the agent loop at 25+ lifecycle points: PreToolUse, PostToolUse, SessionStart, SessionEnd, UserPromptSubmit, SubagentStart, and more. Hooks can allow, deny, or defer tool calls, inject additional context, or modify tool inputs. When multiple hooks conflict, deny takes priority over ask, which takes priority over allow. This creates a deterministic interception layer that enables organizational policy enforcement without modifying the agent's core behavior.

Cowork. Claude Code Cowork brings the same harness architecture into Claude Desktop, running inside a sandboxed VM (Ubuntu 22.04, 4 CPU cores, 4GB RAM) using Apple's Virtualization framework. Multi-layer egress control (syscall blocking via gVisor, MITM proxy with per-boot CA certificates, domain allowlist) provides stronger isolation than the CLI. Rather than sandboxing a browser inside the VM, Cowork controls the host Chrome browser via a native messaging extension — providing real browsing with the user's actual cookies and sessions, but operating outside VM egress controls.

OpenAI's Codex (2025 — not the 2021 code completion model) is a multi-surface coding agent powered by the codex-1 model family (o3 fine-tuned via reinforcement learning on real-world coding tasks). It ships as a cloud product (in ChatGPT), an open-source CLI (Rust + TypeScript, MIT license), IDE extensions, a macOS app, and a GitHub Action. The architectural centerpiece is the App Server: a single stable binary that provides a formal JSON-RPC protocol for all client surfaces.

The Model/Harness Separation. OpenAI explicitly separates two components: the model (stateless reasoning engine) and the harness (execution layer). The harness executes tool calls, collects outputs, manages permissions, enforces sandbox policies, and decides when the agent loop terminates. The model proposes actions; the harness disposes of them. OpenAI published dedicated blog posts about this separation — 'Unrolling the Codex agent loop' and 'Unlocking the Codex harness' — establishing the model/harness distinction as a first-class architectural concept.

The App Server Protocol. The App Server uses JSON-RPC 2.0 streamed as newline-delimited JSON (JSONL) over stdio, with an experimental WebSocket transport. Three core primitives structure all interactions: Items (atomic units of I/O with started/delta/completed lifecycle), Turns (groups of items from one agent work unit, supporting mid-turn steering and interruption), and Threads (durable conversation containers supporting creation, resumption, forking, archival, rollback, and compaction). The protocol is fully bidirectional: the server can initiate requests (e.g., approval prompts) and pause the agent turn pending client response. This formal protocol — with versioned schemas and generated client bindings in Go, Python, TypeScript, Swift, and Kotlin — is Codex's most distinctive engineering contribution.

Two-Layer Security. Codex enforces security through two independent layers. Layer 1 (Sandbox Mode) controls what the agent can do technically: read-only, workspace-write (read/edit within workspace, run commands), or full access. Protected paths (.git/, .agents/, .codex/) remain read-only even in writable modes. Layer 2 (Approval Policy) controls when the user must confirm: on-request (approve sandbox violations and network access), untrusted (auto-run safe reads, approve state mutations), never (all approvals disabled), or granular (per-category control). OS-level enforcement uses macOS Seatbelt profiles on Mac and bubblewrap+seccomp on Linux — real kernel-level sandboxing, not process-level controls.

Codex Cloud runs tasks in isolated containers with a two-phase runtime: a setup phase with network access for installing dependencies, then an agent phase that is offline by default. Internet access must be explicitly enabled. Tasks run asynchronously and multiple tasks can run in parallel.

Context Management. Codex addresses the quadratic context growth problem (n turns requires resending all prior context) through two mechanisms: prompt caching (new content is appended to an identical prefix, enabling KV computation reuse) and conversation compaction (replacing full history with an encrypted compressed representation via the /responses/compact endpoint). The compaction output is opaque — it preserves the model's understanding without exposing raw text. Hierarchical project instructions enter via AGENTS.md files loaded from global override, global default, and per-directory levels.

Subagents. Codex can spawn specialized subagents in parallel using a fan-out/fan-in pattern. The orchestrator waits for all results before returning a consolidated response. Resource controls limit concurrent threads (default: 6) and recursive delegation depth (default: 1). Built-in agent types include default (general-purpose), worker (execution-focused), and explorer (read-heavy analysis). Custom agents are configured via config.toml with specific instructions, model, and sandbox policies.

The Rust Rewrite. The CLI is being rewritten from TypeScript to Rust (codex-rs/) for zero-dependency installation, native security bindings, and elimination of GC pauses. The TypeScript version is maintained in parallel pending Rust feature parity. The Rust rewrite also enables the App Server to be embedded directly in client applications, eliminating the need for a separate process.

Codex as MCP Server. Codex can expose itself as an MCP server (codex mcp-server), enabling other agents to use Codex as a coding tool. Two endpoints are exposed: codex (initiates a session) and codex-reply (continues a session). This composability — where Codex is both an agent and a tool that other agents can call — represents a maturing understanding of agent-to-agent interaction.

Single-agent harnesses hit practical limits when tasks require parallel work, role specialization, or cross-cutting coordination. The multi-agent paradigm addresses this by splitting work across multiple agent instances that communicate and coordinate. The academic foundations were laid by MetaGPT and AutoGen; production harnesses are now building their own implementations.

MetaGPT (Hong et al., ICLR 2024 Oral) introduced SOP-driven multi-agent coordination. Instead of letting agents chat freely, MetaGPT encodes human Standard Operating Procedures into the pipeline: a Product Manager agent generates a PRD, an Architect produces a design doc, Engineers write code, and QA runs tests. Each role passes structured artifacts to the next. This assembly-line model dramatically reduces the incoherent outputs common in free-form multi-agent conversation, because each agent's input is a well-typed artifact (not an open-ended chat message) and each agent's role is precisely scoped. MetaGPT was ranked #1 in the LLM Agent category at ICLR 2024.

AutoGen (Wu et al., COLM 2024, from Microsoft Research) took the opposite approach: fully conversational multi-agent systems. The core abstraction is the 'conversable agent' — an entity that can be backed by an LLM, human input, a tool executor, or any combination. Agents pass messages to each other in programmable conversation patterns: sequential (agent A talks to agent B), group chat (multiple agents discuss), or nested (a conversation within a conversation). The power comes from flexibility: the same framework handles math problem-solving, coding, operations research, and human-in-the-loop workflows. The cost is that conversational coordination is less predictable than SOP-driven pipelines.

ToolLLM (Qin et al., ICLR 2024 Spotlight) addressed a different aspect of multi-agent capability: tool discovery and planning across massive API landscapes. The framework includes ToolBench (16,464 real-world APIs across 49 categories), ToolLLaMA (a fine-tuned model for tool use), and a Depth-First Search Decision Tree (DFSDT) planner that explores multiple tool-call paths and backtracks when a path fails. DFSDT is the first systematic tree-search approach to tool planning, handling API failures via backtracking rather than linear retry — a critical capability when coordinating across unreliable external services.

Production implementations are converging on simpler patterns. Claude Code's agent teams use file-based mailboxes and shared task lists with file locking — a straightforward filesystem-based coordination mechanism. Codex uses fan-out/fan-in with hard limits on concurrent threads and recursion depth. OpenClaw relies on channel-to-agent bindings with shared filesystem state. None of these approach the sophistication of MetaGPT's SOP pipelines or AutoGen's conversational topologies, reflecting a production engineering preference for simplicity and debuggability over coordination expressiveness.

The Scaffolding/Harness taxonomy paper by Bui (arXiv, March 2026) provides the clearest published framework for understanding these systems. It distinguishes scaffolding (static assembly before the first prompt: system prompt, tool schemas, subagent registry) from harness (runtime orchestration: tool dispatch, context compaction, safety enforcement, state management). This vocabulary helps explain why production multi-agent systems look so different from academic ones: academic systems focus on scaffolding (agent roles, communication topologies) while production systems spend most of their engineering on harness (context management, permission enforcement, crash recovery).