Idempotency

The Problem

In distributed systems, failures happen. A network timeout after sending an email leaves the system uncertain: did the email send or not? Naive retry logic risks sending the same email twice.

For outbound communication, duplicates are not just annoying — they're damaging. A contact receiving the same sales email twice looks unprofessional. A duplicate collections call may violate regulations.

Idempotency Keys

Every step execution is assigned a deterministic idempotency key derived from:

idempotencyKey = hash(runId + stepId + tick)

Because the run ID, step ID, and logical clock tick are all deterministic, the same step in the same run always produces the same idempotency key — even across retries or replays.

How It Works

Before executing a channel action (sending an email, making a call), the engine checks the idempotency store:

1. Compute idempotency key
2. Check: has this key been executed before?
   ├── YES → Return the stored result (skip execution)
   └── NO  → Execute the action
              ├── SUCCESS → Store result with key, advance
              └── FAILURE → Store failure with key, retry policy applies

async function executeWithIdempotency(step: Step, context: ExecutionContext) {
  const key = computeIdempotencyKey(context.run.id, step.id, context.clock.tick);

  const existing = await idempotencyStore.get(key);
  if (existing) {
    return existing.result; // already executed, return stored result
  }

  const result = await channel.execute(step, context);
  await idempotencyStore.set(key, result, { ttl: '90d' });

  return result;
}

Retry Safety

When a step fails with a transient error (network timeout, rate limit), the engine retries with the same idempotency key. This is safe because:

• If the original request actually succeeded (but the response was lost), the channel provider returns the stored result
• If the original request truly failed, the retry executes normally

Most email and voice providers support idempotency keys natively:

Replay Safety

When replaying an execution for audit or debugging, the engine operates in dry-run mode by default — it evaluates every step but does not execute channel actions. The replay compares what would happen against what did happen, producing a diff if results diverge.

const replay = await engine.replay(runId, { dryRun: true });

console.log(replay.steps.map(s => ({
  step: s.id,
  original: s.originalResult,
  replayed: s.replayedResult,
  match: s.identical,
})));

Replays with dryRun: false will execute channel actions, but idempotency keys prevent duplicates — the channel provider recognizes the key and returns the original result.