Agent Execution Engines: State Machines, Retries, and Idempotency

Every beginner AI agent tutorial teaches the same loop:

# The tutorial agent loop:
while not task_completed:
    decision = llm.chat(history)
    if decision.has_tool_call:
        output = run_tool(decision.tool, decision.args)
        history.append(output)

Running this pattern in production quickly exposes severe flaws:

• If your server process restarts or hits an out-of-memory error mid-task, all state is lost.
• If a tool times out at the network layer, a naive retry can charge a credit card twice or trigger duplicate database mutations.
• If the model gets stuck in an hallucination loop, it will run until your API credits run dry.
• You have no persistent audit trail to inspect why an agent made a disastrous decision.

To build agents that run reliably, you need an event-driven state engine.

State Machine Architecture

Instead of an unbounded loop, represent the agent as a finite state machine backed by persistent storage:

  +---------------+
  |   INITIATED   |
  +-------+-------+
          |
          v
  +---------------+        Model calls tool
  |   PLANNING    | ----------------------------+
  +-------+-------+                             |
          ^                                     v
          |  Observation logged         +---------------+
          +---------------------------- |   EXECUTING   |
          |                             +-------+-------+
          |                                     |
          | Tool failed (retry budget OK)       | Unrecoverable error
          +-------------------------------+     v
                                          |   +---------------+
                                          +-> |    FAILED     |
                                              +---------------+

The engine transitions between explicit states:

• INITIATED: Task validated, environment constraints and max budget assigned.
• PLANNING: Prompt assembled from persistent history, sent to LLM for next decision.
• EXECUTING: Model requested a tool; argument schema validated and idempotency check performed.
• WAITING_CONFIRMATION: Optional human-in-the-loop checkpoint for sensitive actions.
• COMPLETED / FAILED: Terminal states with recorded metrics and audit output.

State Persistence with SQLite WAL

To survive crashes, every step must be written to disk before the agent takes an action. SQLite in Write-Ahead Logging (WAL) mode provides high-speed, zero-dependency persistence:

-- Schema for agent runs
CREATE TABLE agent_runs (
  run_id TEXT PRIMARY KEY,
  task_goal TEXT NOT NULL,
  status TEXT NOT NULL CHECK(status IN ('initiated', 'planning', 'executing', 'completed', 'failed')),
  total_tokens INTEGER DEFAULT 0,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
  updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Schema for step journals
CREATE TABLE agent_steps (
  step_id TEXT PRIMARY KEY,
  run_id TEXT NOT NULL,
  step_index INTEGER NOT NULL,
  thought TEXT,
  tool_name TEXT,
  tool_args TEXT,
  tool_result TEXT,
  is_error INTEGER DEFAULT 0,
  idempotency_key TEXT UNIQUE,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
  FOREIGN KEY (run_id) REFERENCES agent_runs(run_id)
);

CREATE INDEX idx_agent_steps_run ON agent_steps(run_id, step_index);

Preventing Duplicate Tool Execution: Idempotency Keys

What happens when an agent executes an external action (like sending a Slack alert or debiting an account) and the socket disconnects before the agent receives the response?

If the agent simply retries, it performs the action a second time.

To solve this, calculate a deterministic Idempotency Key for every tool call before running it:

import crypto from "crypto";

function generateToolIdempotencyKey(
  runId: string,
  stepIndex: number,
  toolName: string,
  args: Record<string, any>
): string {
  // Hash the run ID, step number, and canonical JSON of arguments
  const sortedArgs = JSON.stringify(args, Object.keys(args).sort());
  const rawSignature = `${runId}:${stepIndex}:${toolName}:${sortedArgs}`;
  return crypto.createHash("sha256").update(rawSignature).digest("hex");
}

Before executing the tool, check your idempotency table:

async function executeToolSafely(
  db: any,
  runId: string,
  stepIndex: number,
  toolName: string,
  args: any,
  toolHandler: Function
) {
  const key = generateToolIdempotencyKey(runId, stepIndex, toolName, args);

  // 1. Check if this exact step already completed
  const existing = db.prepare(
    "SELECT tool_result, is_error FROM agent_steps WHERE idempotency_key = ?"
  ).get(key);

  if (existing) {
    console.warn(`[Idempotency] Returning cached result for step ${stepIndex} (${toolName})`);
    return JSON.parse(existing.tool_result);
  }

  // 2. Execute tool
  let result: any;
  let isError = 0;
  try {
    result = await toolHandler(args);
  } catch (err: any) {
    result = { error: err.message };
    isError = 1;
  }

  // 3. Atomically journal result before continuing
  db.prepare(`
    INSERT INTO agent_steps (step_id, run_id, step_index, tool_name, tool_args, tool_result, is_error, idempotency_key)
    VALUES (?, ?, ?, ?, ?, ?, ?, ?)
  `).run(
    crypto.randomUUID(),
    runId,
    stepIndex,
    toolName,
    JSON.stringify(args),
    JSON.stringify(result),
    isError,
    key
  );

  return result;
}

Defensive Guardrails: Loop Limits and Budgets

Autonomous agents must operate with hard constraints:

1. Step Budget: Cap execution at a fixed maximum (e.g. 25 steps). If a task cannot finish within 25 steps, fail the run and flag it for human review rather than burning infinite tokens.
2. Token Ceiling: Track cumulative token consumption across steps. If a run crosses your budget threshold (e.g. 100,000 tokens), terminate it.
3. Repeated Tool Detection: If the agent calls the exact same tool with the exact same parameters three times in a row with the same error output, break the loop immediately. The model is hallucinating in a cycle.

Treating agent execution as a stateful, journaled pipeline transforms brittle prototypes into systems you can trust with real infrastructure.