Multi-step processes & sagas for system design interviews

Clean database tutorials deal with one write. Real apps coordinate dozens of flaky services to fulfill an order, dispatch a ride, or run an agent pipeline. Doing that quickly and reliably is a continual source of operational pain — and a favorite interview topic.

Jimmy Bogard's Six Little Lines of Fail captures it: distributed systems make even a simple sequence surprisingly hard. This post is the deep dive behind Common patterns. Pair with Managing long-running tasks (simple job chains), Message queues, Kafka, and Contention (single-DB vs cross-service).

Consider e-commerce order fulfillment: charge payment, reserve inventory, create a shipping label, wait for a warehouse worker to pick the item, send confirmation email. Each step calls different services or waits on humans. Any can fail or timeout. Payment gateways callback via webhook minutes later. Your server might crash or deploy mid-flight. Business rules change the step order.

You can patch organically: fortify each service, delay queues for waits, webhooks for humans — but each patch adds complexity. You interweave system-level concerns (crashes, retries) with business-level concerns (item not found?). Not a great design.

The simplest build: one API server walks the steps in order, then responds.

async function fulfillOrder(order: Order): Promise<OrderResult> {
  try {
    await chargePayment(order);
    await reserveInventory(order);
    // Woops, server crashed here!
    await createShippingLabel(order);
    await sendConfirmationEmail(order);
    return { success: true };
  } catch (error) {
    return { success: false, error: error.message };
  }
}

Crashes: server dies after charge, before reserve — on restart, no memory of progress.

Callback routing: payment gateway webhooks hit a different host behind the load balancer — that host doesn't know which in-flight order the callback belongs to.

Patch: persist order state after each step; route webhooks through pub/sub so any server can load the order and continue. That gets you something like:

If you've operated one of these, you know the ongoing pain. That's why teams reach for workflow engines.

Once steps accumulate with interesting failure handling, you're in saga territory — a long series of local steps, each with a matching compensating action. Forward on success; walk backward on failure.

Why not one big distributed transaction? Nobody holds a lock that long. 2PC stalls everyone if the coordinator stalls; payment gateways don't speak your transaction protocol. Sagas accept eventual consistency windows and guarantee undoability.

After charge, customer is out money before ship — order sits pending. Compensations need the same paranoia: refunds fail too → retries, idempotency, human escape hatch.

Store the stream of events that got you here; workers react to drive forward. Durable log (Kafka popular; Redis Streams for lighter cases). Close cousin to event sourcing — here the log mainly coordinates work.

Payment worker sees OrderPlaced, calls gateway; webhook → PaymentCharged or PaymentFailed. Inventory worker sees PaymentCharged. Compensation backwards: InventoryFailed → payment worker emits PaymentRefunded.

For orchestration, describe a reliable long-running process that survives failures and continues where it left off — without hand-rolling infrastructure.

Two camps: durable execution engines (code-first, Temporal) and managed workflow systems (declarative state machines, AWS Step Functions).

Write long-running code that moves between machines and survives crashes. Temporal (2019 fork of Uber's Cadence) is the popular open-source option.

const { processPayment, reserveInventory, shipOrder,
        sendConfirmationEmail, refundPayment } = proxyActivities(...);

async function myWorkflow(input: Order): Promise<OrderResult> {
  const paymentResult = await processPayment(input);
  if (paymentResult.success) {
    const inventoryResult = await reserveInventory(input);
    if (inventoryResult.success) {
      await shipOrder(input);
      await sendConfirmationEmail(input);
      return { success: true };
    } else {
      await refundPayment(input);
      return { success: false, error: "Inventory reservation failed" };
    }
  }
  return { success: false, error: "Payment failed" };
}

Looks like single-server orchestration — the difference is how it runs.

Workflow vs Activity

• Workflow — deterministic recipe; given same inputs + history, same decisions. No network I/O here.
• Activity — one step touching the outside world; must be idempotent (safe to retry).

Recovery by replay

Every Activity result recorded in history. Crash → new worker re-executes Workflow from top. Completed Activities return recorded results — no re-fire. Side-effect-free replay lands exactly where crash interrupted.

Signals wait for external events efficiently — human pickup, document signed — without holding a thread for 30 days.

Declarative state machines in JSON/YAML — AWS Step Functions, Google Cloud Workflows. Less expressive than code; no clusters to run (serverless). Standard Step Functions: up to 1 year, 256KB between states.

In AWS shops you'd often generate definitions from CDK — the "ugly JSON" complaint is less important than understanding the state machine model and recovery semantics.

Spot state machines and multi-step flows. Mid-level: name the need + one engine. Senior+: defend choice, walk compensation, field deep dives.

What if the saga coordinator crashes?

Charged + reserved, then process dies before ship. Without durable progress: don't know whether to forward, retry, or compensate. Fix: record completed steps in a crash-safe store; resume or compensate. Workflow engines do this via history + replay.

How do you update running workflows?

10,000 loan approvals in flight; add compliance check. Versioning: new code for new runs; old runs finish on old version. Migration: in-place update — Step Functions pins in-flight executions to start definition; Temporal patched() branches deterministically so replay stays valid.

if (workflow.patched("change-behavior")) {
  await a.newBehavior();
} else {
  await a.legacyBehavior();
}

Workflow state size / history growth

Minimize Activity payload sizes (pass IDs, not blobs). Continue-as-New (Temporal): snapshot state, start fresh run with empty history — standard for long-lived per-user workflows.

External events & long waits

Signals + durable timers — workflow holds no thread while customer takes 5 days to sign. Webhook calls engine API with workflow ID to wake the right execution.

Exactly-once activity effects?

Engines retry Activities when success report is lost → at-least-once attempts, exactly-once effect via idempotency keys. Mark IN_PROGRESS before irreversible action, COMPLETED after; reconcile rare stuck IN_PROGRESS. True exactly-once delivery impossible across network — idempotency is the answer.

"This is a multi-step saga — I'll model states with compensations: refund if inventory fails, release if ship fails. MVP: orchestrator service + orders table + idempotency keys. If we need multi-day human waits or webhook-heavy flows, I'll use Temporal (or Step Functions on AWS) for durable execution and signals. Every Activity gets an idempotency key; compensations retry too. I'll name choreography if teams own services independently, orchestration if the flow is complex and central visibility matters."

Practice saying these out loud for sagas / multi-step processes. Interviewers grade clarity and judgment more than buzzwords.

Workflow systems fit hairy state machines that are otherwise painful to get right. They centralize persistence, retries, and compensation so business logic reads like requirements — not infrastructure gymnastics.

Recognize when you're hand-rolling what an engine provides: crash-safe progress, multi-service orchestration, long waits, audit trails. If you're building sagas in Redis if/else, consider graduating.

Continue with Common patterns, Message queues, Kafka, Contention, Sharding (sagas over 2PC), and Agentic architectures.