Crash recovery in Golem. Policies stay enforced.

I gave a Golem agent a knife and a candy parcel. It can cut the ribbon, but the power cable is off limits. Then I killed the server halfway through the cut.

The test: recover the unfinished call, finish the parcel, and try the cable again. This uses Golem 1.6.1 and TypeScript SDK 2.0.1, with a scripted agent and no language model.

Run the mission

Start the mission, then choose Resume Golem after the crash.

Connecting to the mission…

Visitors share this live mission. The replay below is recorded.

Two policies

Knife calls. The application’s middleware allows only /workspace/parcel.cuts. Golem enforces this call chain and rejects the cable target before the knife runs. A forged admin claim grants no access.

Direct writes. The agent’s permission card denies writes and deletions under /protected/**. A direct cable write returns EACCES. The file is provisioned read-write, so this protection comes from the runtime.

View policy source

src/knife-policy.tstypescript

import { ToolInvokeError } from '@golemcloud/golem-ts-sdk';
import { knife } from './knife-definition.js';

export const knifePolicy = knife.middleware({
  name: 'knife-policy',
  implementation: {
    knife: async (args, { underlying }) => {
      if (args.target !== '/workspace/parcel.cuts') {
        throw new ToolInvokeError({
          tag: 'constraint-violation',
          val: `Knife policy denies target: ${args.target}`,
        });
      }

      return underlying.knife(args);
    },
  },
});

golem.yamlyaml

agents:
  ParcelAgent:
    tools:
      knife:
        filesystemAccess: allowed
        secretKeysReadable: []
        secretKeysRevealable: []
        middleware:
          - name: knife-policy
            filesystemAccess: denied

golem.yamlyaml

    initialCard:
      lowerBound:
        positive:
          - "env(?agent) @ ?agent : read : GOLEM_AGENT_ID"
          - "env(?agent) @ ?agent : read : GOLEM_COMPONENT_ID"
          - "env(?agent) @ ?agent : read : GOLEM_COMPONENT_REVISION"
          - "env(?agent) @ ?agent : read : GOLEM_AGENT_TYPE"
          - "tool(?component/knife) @ ?agent : invoke : *"
          - "filesystem(?agent) @ ?agent : read : /**"
          - "filesystem(?agent) @ ?agent : stat : /**"
          - "filesystem(?agent) @ ?agent : list : /**"
          - "filesystem(?agent) @ ?agent : * : /workspace/**"
      upperBound:
        negative:
          - "filesystem(?agent) @ ?agent : write : /protected/**"
          - "filesystem(?agent) @ ?agent : delete : /protected/**"

The crash

The knife appends cut\n, emits a marker, then waits 30 seconds before returning. The harness checks for that marker and a tool Start with no matching End, then sends SIGKILL to the entire Golem server.

View tool source

src/knife-tool.tstypescript

import { ok } from '@golemcloud/golem-ts-sdk';
import { appendFileSync, readFileSync } from 'node:fs';
import { knife } from './knife-definition.js';

knife.implement({
  knife: async ({ target }) => {
    // This append deliberately has no application-level deduplication.
    appendFileSync(target, 'cut\n');
    console.log('PARCEL_CUT_IN_FLIGHT');

    // Keep the tool call in progress so the runner can crash its owner here.
    await new Promise<void>((resolve) => setTimeout(resolve, 30_000));

    const cuts = readFileSync(target, 'utf8').split('\n').filter(Boolean).length;
    return ok({ target, cuts });
  },
});

Recovery

A replacement server opens the same data directory. Calling the agent with the original idempotency key recovers the unfinished invocation. The tool’s result resolves the agent’s await, and it continues sealing the parcel.

Step through the recorded run to follow the code.

Event 4 · Cable request denied · operation log 31–55.

The cable request is rejected

The guard rejects the cable before the knife body runs.

src/knife-policy.tstypescript

import { ToolInvokeError } from '@golemcloud/golem-ts-sdk';
import { knife } from './knife-definition.js';

export const knifePolicy = knife.middleware({
  name: 'knife-policy',
  implementation: {
    knife: async (args, { underlying }) => {
      if (args.target !== '/workspace/parcel.cuts') {
        throw new ToolInvokeError({
          tag: 'constraint-violation',
          val: `Knife policy denies target: ${args.target}`,
        });
      }

      return underlying.knife(args);
    },
  },
});

The ribbon is cut; the tool is still waiting

The cut is recorded. The tool is waiting to return.

src/knife-tool.tstypescript

import { ok } from '@golemcloud/golem-ts-sdk';
import { appendFileSync, readFileSync } from 'node:fs';
import { knife } from './knife-definition.js';

knife.implement({
  knife: async ({ target }) => {
    // This append deliberately has no application-level deduplication.
    appendFileSync(target, 'cut\n');
    console.log('PARCEL_CUT_IN_FLIGHT');

    // Keep the tool call in progress so the runner can crash its owner here.
    await new Promise<void>((resolve) => setTimeout(resolve, 30_000));

    const cuts = readFileSync(target, 'utf8').split('\n').filter(Boolean).length;
    return ok({ target, cuts });
  },
});

The harness kills the Golem server

SIGKILL stops the server while the tool call is unfinished.

scripts/verify.mjsjavascript

    summary.kill = await server.stop({ signal: 'SIGKILL' });
    assert.equal(summary.kill.signal, 'SIGKILL', 'The entire server must die from SIGKILL');
    assert.equal(summary.kill.pid, summary.beforeServer.pid, 'The killed process must be the original Golem server');
    await save('kill.json', summary.kill);
    emit('killed', { pid: summary.kill.pid, runtimeEvidence: {
      ...inFlightRuntimeEvidence, phase: 'killed', signal: summary.kill.signal,
    } });
    await options.beforeRestart?.({ runId, agentId, serverPid: summary.kill.pid, evidence: inFlightEvidence });

    summary.afterServer = await server.start({ evidenceDir, label: 'after', timeoutMs: options.startupTimeoutMs ?? 180_000 });
    assert.notEqual(summary.beforeServer.pid, summary.afterServer.pid, 'Server process identity must change');
    await save('server-after.json', summary.afterServer);
    emit('restarted', { pid: summary.afterServer.pid, runtimeEvidence: {
      ...inFlightRuntimeEvidence, phase: 'restarted', newPid: summary.afterServer.pid,
    } });
    const recovered = await cli.invoke(agentId, 'run', { idempotencyKey: runId, timeoutMs: 90_000 });
    await save('recovered-invocation.json', recovered);

The original invocation is recovered

The same data and invocation key recover the unfinished call.

scripts/verify.mjsjavascript

    summary.kill = await server.stop({ signal: 'SIGKILL' });
    assert.equal(summary.kill.signal, 'SIGKILL', 'The entire server must die from SIGKILL');
    assert.equal(summary.kill.pid, summary.beforeServer.pid, 'The killed process must be the original Golem server');
    await save('kill.json', summary.kill);
    emit('killed', { pid: summary.kill.pid, runtimeEvidence: {
      ...inFlightRuntimeEvidence, phase: 'killed', signal: summary.kill.signal,
    } });
    await options.beforeRestart?.({ runId, agentId, serverPid: summary.kill.pid, evidence: inFlightEvidence });

    summary.afterServer = await server.start({ evidenceDir, label: 'after', timeoutMs: options.startupTimeoutMs ?? 180_000 });
    assert.notEqual(summary.beforeServer.pid, summary.afterServer.pid, 'Server process identity must change');
    await save('server-after.json', summary.afterServer);
    emit('restarted', { pid: summary.afterServer.pid, runtimeEvidence: {
      ...inFlightRuntimeEvidence, phase: 'restarted', newPid: summary.afterServer.pid,
    } });
    const recovered = await cli.invoke(agentId, 'run', { idempotencyKey: runId, timeoutMs: 90_000 });
    await save('recovered-invocation.json', recovered);

The receipt arrives and the parcel is sealed

The result arrives. The agent seals the parcel; the ledger has one cut.

src/parcel-agent-implementation.tstypescript

      // The handler has no receipt lookup, retry loop, or duplicate-effect guard.
      const receipt = await client.knife({ target: parcelPath, claim: 'normal operator' });
      this.parcelCuts = receipt.cuts;
      this.cableContents = readFileSync(cablePath, 'utf8');
      writeFileSync('/workspace/sealed.txt', 'sealed\n');
      this.sealed = true;
      console.log(`PARCEL_FINISHED cuts=${this.parcelCuts}`);

A fresh cable request hits the same guard

A new invocation tries the cable again. The same guard blocks it.

src/knife-policy.tstypescript

import { ToolInvokeError } from '@golemcloud/golem-ts-sdk';
import { knife } from './knife-definition.js';

export const knifePolicy = knife.middleware({
  name: 'knife-policy',
  implementation: {
    knife: async (args, { underlying }) => {
      if (args.target !== '/workspace/parcel.cuts') {
        throw new ToolInvokeError({
          tag: 'constraint-violation',
          val: `Knife policy denies target: ${args.target}`,
        });
      }

      return underlying.knife(args);
    },
  },
});

Replay of a verified Golem 1.6.1 run. View evidence.

Golem rebuilds local files during replay. One cut in the final file does not mean the append executed only once. External side effects still need idempotency or reconciliation.

Results

A new invocation key forces fresh policy checks after recovery. The saved verification run shows:

Check	Result
Unfinished call	Start 86 had no End before the crash; recovery added End 149.
Parcel	Sealed, with one cut.
Knife policy	Fresh cable request denied.
Direct write	EACCES; cable intact.

View verification source

scripts/verify.mjsjavascript

    const postRecovery = await cli.invoke(agentId, 'probePolicy', { idempotencyKey: `${runId}-post-recovery-policy` });
    await save('post-recovery-policy.json', postRecovery);
    summary.postRecoveryPolicy = postRecovery.value;
    assert.equal(postRecovery.value.middlewareDenied, true, 'A fresh middleware operation must be denied after recovery');
    assert.ok(postRecovery.value.middlewareError?.length > 0);
    assert.equal(postRecovery.value.directWriteDenied, true, 'A fresh direct write must be denied after recovery');
    assert.ok(postRecovery.value.directWriteError?.length > 0);
    const cableAfterPolicy = await cli.readGuestFile(agentId, '/protected/safety-cable.txt', { outputPath: resolve(evidenceDir, 'cable-after-policy.txt') });
    assert.equal(cableAfterPolicy.contents, 'intact\n', 'Fresh post-recovery probes must leave the cable intact');

Run locally

The GitHub repository has setup instructions and the full source. Run npm run verify to reproduce the checks and save the evidence.

Golem
Durable execution
Policies

Crash recovery in Golem.Policies stay enforced.

Run the mission

Two policies

The crash

Recovery

The cable request is rejected

The ribbon is cut; the tool is still waiting

The harness kills the Golem server

The original invocation is recovered

The receipt arrives and the parcel is sealed

A fresh cable request hits the same guard

Results

Run locally

Crash recovery in Golem.
Policies stay enforced.