Step Functions vs Temporal vs Airflow for Long-Running Workflows

I have shipped long-running workflows on all three. A claims intake system on Step Functions that spans days. A payment saga on Temporal that handles out of order refunds. A data enrichment DAG on Airflow with hundreds of daily runs. The differences are not academic. They show up as broken idempotency, runaway histories, or an ops pager at 3 a.m. Below is what actually mattered.

What Matters For Long-Running Workflows

A good orchestrator does five things well for long lived work.

• Durable progress, including across deploys and restarts.
• Idempotent task execution, with at least once delivery that you can make safe.
• Human in the loop and external signals without race conditions.
• Backpressure and rate control so downstreams survive bursts.
• Clear costs for the state you keep for months.

If you only need daily batch with a few tasks, most options work. If you need months of saga state with compensations and human approvals, the edges are sharp.

If you want a second set of eyes on the design before you commit, we run lightweight reviews that cover these risks in our architecture review and vendor stack decisions work. We have had to unpick bad orchestrator choices more than once.

Core Model Differences That Bite You At Scale

Step Functions

• State is on AWS. You pay per state transition. You get a managed control plane. Standard workflows can run up to one year per execution. Event history limit is 25,000 events per execution. Payloads are capped at 256 KB per state input or output.
• You model retries and catches declaratively. You can pause with Wait, call Lambdas or service integrations, and use Map for fan out. Human steps need a custom callback, EventBridge, or an Activity.
• Good fit if you are all in on AWS and your tasks are already on Lambda, ECS, or service APIs.

Temporal

• State is managed in a single durable event log per workflow, persisted to a database. The workflow code is deterministic, the engine replays history to reconstruct state. You run workers to execute Activities. You get Signals and Queries built in.
• History can grow large. You will need ContinueAsNew on long histories or high event counts. Strong testable semantics, but your team must learn the deterministic workflow model.
• Good fit for complex sagas with lots of compensations, human signals, and long duration business processes.

Airflow

• DAG centric batch orchestrator. Tasks are processes or containers, scheduled by time or external triggers. State is in a metadata DB, and XCom passes small messages.
• Better for data pipelines than transactional sagas. Human steps are clunky. Long running sensors block resources unless you use deferrable operators.
• Good fit for analytics and ETL. Less ideal for multi day business workflows with external callbacks.

The first trap we see is payload size. Do not pass large blobs through any orchestrator. Use object storage and pass a pointer. On Step Functions that is mandatory once you approach 256 KB. On Airflow, XCom is small by default, so use S3 or GCS. On Temporal, large history slows replay, so keep Activity inputs small and stash blobs elsewhere.

Failure Handling, Retries, And Exactly Once

Step Functions has Retry and Catch at each Task. It works well when the upstreams are idempotent. Lambda retries plus Step Functions retries can multiply, so set maxAttempts and backoff carefully. Activities can heartbeat, but most teams use Lambdas or service calls. Idempotency keys must sit in downstream services or in a DynamoDB guard table keyed by a business id.

Temporal gives you strong workflow semantics. A Workflow executes once, even across worker restarts. Activities are retried per policy, with automatic heartbeats for long work. When your Activity times out, the Workflow can decide to retry, run a compensating action, or wait for a Signal. The tradeoff is you must keep workflow code deterministic. No random, no current time without injecting it, no network calls in the Workflow code. We had an early incident where a non deterministic JSON serializer change broke replay. Fix was to lock versions and gate changes with patch markers.

Airflow retries tasks, but does not give you saga semantics. Cross task compensation is manual. Sensors pre deferrable era pinned a worker slot for hours. Use deferrable operators and triggers for long waits. For external callbacks, build a small service that hits the Airflow REST API to unpause or trigger a DAG.

On SLAs, managed Step Functions comes with published service availability in typical AWS ranges. Temporal Cloud has a formal SLA, self host means your SLA is what you run. Managed Airflow like MWAA or Astronomer publishes availability targets, self host is on you. What bit us more often than service outages was limits and history growth.

Cost Over 12 Months, Three Profiles

Costs change fast. Verify with your vendor and region. The numbers below are ballpark to help compare shapes of spend.

Profile A. Human approval saga, 200k executions per year, each 20 states, spans up to 14 days. Average of 3 retries per sensitive task across 5 tasks.

• Step Functions Standard. Transitions per execution, base 20, plus retries about 15. Call it 35. 200k x 35 = 7,000,000 transitions. At about 0.025 per 1,000 transitions, the control plane is about 7,000,000 ÷ 1,000 × 0.025 = 175 dollars for the year. You will still pay for Lambdas, service calls, and storage. If you need Activities with heartbeats, factor in EC2 or Fargate cost for the workers.
• Temporal self hosted. Control plane cost is your servers and database. A small three node Temporal cluster on m5.large, plus an RDS PostgreSQL r6g.large, plus storage, sits roughly near 700 to 1,000 dollars a month in a typical region. Over 12 months, 8,400 to 12,000 dollars. Worker cost sits on top. Temporal Cloud is subscription plus usage, talk to sales. The shape is fixed monthly plus scale.
• Airflow managed, MWAA small environment is roughly in the mid three figures per month before workers. Astronomer similar order of magnitude. Over 12 months you will likely see 6,000 to 12,000 dollars for the control plane. Workers add more during busy periods.

If your profile is light on transitions but needs a robust control plane, Step Functions is cheap. If you need strong saga semantics and complex signals, Temporal’s control plane cost can be worth it, but plan for the base monthly spend.

Profile B. Order pipeline, 1 million executions per year, 20 states, a few compensations. High parallelism, strict backpressure.

• Step Functions. 1,000,000 x 20 = 20,000,000 transitions. That is about 500 dollars in the year for the orchestrator. You will likely need throttles and rate controls to protect downstreams. Map states with MaxConcurrency works. Use service quotas and reserved concurrency on Lambdas. Step Functions control plane cost stays low.
• Temporal. Same monthly base as above, plus worker autoscaling cost. You will likely run multiple worker fleets for isolation. The base is the dominant cost unless workers are very busy.
• Airflow is not a great fit here. You can do it, but Airflow loves time based batch. For per order sagas, the number of DAG runs and scheduler overhead grows. You will end up paying more for complexity and ops than the control plane itself.

Profile C. Analytics ETL, 200 DAGs, 10k runs per day, 4 tasks average, finish in under an hour.

• Airflow shines. Pooling, SLAs, backfills, and lineage tools matter more than saga semantics. Managed Airflow plus Kubernetes workers on spot nodes will be cost effective. The control plane is a few thousand a year, workers dominate.
• Step Functions can do it, Express workflows become attractive for short synchronous flows. For long ETL, Standard works but you will model a lot of glue code that Airflow already gives you.
• Temporal is fine technically, but the Activity worker model is extra weight for this use case.

If you need help turning these shapes into your exact numbers, we run quick cloud and DevOps cost reviews that include autoscaling plans and quotas.

Design Patterns And Code You Can Lift

Saga with compensation and human approval. I use callbacks or signals and small tasks with idempotent writes. Keep blobs out of the orchestrator.

Step Functions, compensation with retries and a manual approval via a callback token. This uses a Lambda, but you can swap for any supported integration.

{
  "Comment": "Order saga with manual approval",
  "StartAt": "ReserveInventory",
  "States": {
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:reserveInventory",
      "Retry": [{"ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2}],
      "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "ReleaseInventory"}],
      "Next": "ChargePayment"
    },
    "ChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:chargePayment",
      "Retry": [{"ErrorEquals": ["PaymentTransient"], "IntervalSeconds": 5, "MaxAttempts": 5}],
      "Catch": [{"ErrorEquals": ["PaymentFailed"], "Next": "ReleaseInventory"}],
      "Next": "WaitForApproval"
    },
    "WaitForApproval": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:approvalHandler",
        "Payload.$": "$",
        "TaskToken.$": "$$.Task.Token"
      },
      "TimeoutSeconds": 1209600,
      "Next": "ShipOrder"
    },
    "ReleaseInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:releaseInventory",
      "End": true
    },
    "ShipOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:shipOrder",
      "End": true
    }
  }
}

Temporal, long running workflow with heartbeat, signals, and ContinueAsNew to control history growth. TypeScript example.

// workflow.ts
import { proxyActivities, defineSignal, setHandler, continueAsNew } from '@temporalio/workflow';

const { reserve, charge, release, ship, longTask } = proxyActivities<{
  reserve(input: any): Promise<void>;
  charge(input: any): Promise<void>;
  release(input: any): Promise<void>;
  ship(input: any): Promise<void>;
  longTask(input: any): Promise<void>;
}>({ startToCloseTimeout: '5 minutes', heartbeatTimeout: '30 seconds', retry: { maximumAttempts: 5 } });

export const approveSignal = defineSignal<[boolean]>('approve');

export async function orderWorkflow(input: any): Promise<void> {
  let approved = false;
  setHandler(approveSignal, (val: boolean) => { approved = val; });

  await reserve(input);
  try {
    await charge(input);
  } catch (e) {
    await release(input);
    throw e;
  }

  // wait for human approval via signal
  // poll until approved, yield so worker can shut down cleanly
  while (!approved) {
    await longTask({ kind: 'noop' });
    // if history grows beyond 10k events, roll forward
    if ((workflowInfo().historyLength ?? 0) > 10000) {
      return continueAsNew<typeof orderWorkflow>(input);
    }
  }

  await ship(input);
}

// activities.ts
import { Context } from '@temporalio/activity';
export async function longTask(_: any) {
  // heartbeat to avoid timeouts on long work
  for (let i = 0; i < 10; i++) {
    Context.current().heartbeat({ i });
    await new Promise(r => setTimeout(r, 1000));
  }
}

Airflow, deferrable external approval, with small payloads passed via object storage.

# dag.py
from airflow import DAG
from airflow.decorators import task
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow.utils.dates import days_ago
from datetime import timedelta

with DAG(
    dag_id="order_saga_light",
    start_date=days_ago(1),
    schedule_interval=None,
    max_active_runs=50,
    default_args={"retries": 3, "retry_delay": timedelta(seconds=30), "execution_timeout": timedelta(minutes=15)},
    catchup=False,
) as dag:

    @task
    def reserve(key: str):
        # call service, idempotent by business key
        pass

    @task
    def charge(key: str):
        pass

    approval = S3KeySensor(
        task_id="wait_for_approval",
        bucket_name="approvals-bucket",
        bucket_key="approvals/{{ ti.xcom_pull(task_ids='charge') }}.approved",
        poke_interval=60,
        mode="reschedule",  # deferrable style, frees worker slot
        timeout=60 * 60 * 24 * 14,
    )

    @task
    def ship(key: str):
        pass

    k = "s3://objects/order-{{ ds_nodash }}-{{ task_instance_key_str }}.json"
    reserve(k) >> charge(k) >> approval >> ship(k)

We standardize a few practices across all three.

• Put idempotency at the business layer. Every state change has a business key. Re apply is a no op.
• Use object storage for blobs and references for messages. Keep orchestrator payloads tiny.
• Design for cancel and compensation from day one. Write compensators first. Test chaos by killing workers mid step.
• Document quotas. Step Functions history and payload caps. Temporal history growth. Airflow XCom size. Build alerts before you hit them.

If you need help hardening these practices, we do targeted API and system integration reviews that focus on idempotency and backpressure.

Operational Notes, SLAs, Limits, Blast Radius

Step Functions operations are light. You tune retries and backoff in the ASL, set per state timeouts, and you are done. The main ops risks are quota blowups during spikes and noisy retries that amplify downstream failures. Set reserved concurrency on hot Lambdas. Use Map with MaxConcurrency. Subscribe CloudWatch alarms for Throttles and ServiceIntegration errors.

Temporal operations depend on your hosting. On self host, run multi AZ for the database. Watch history growth. Add ContinueAsNew on workflows that will run more than a few weeks with frequent signals. Keep Worker fleets separate per domain to limit blast radius. We run one fleet for hot payment tasks and another for slow human approvals so spikes do not starve signals.

Airflow operations are classic scheduler and worker hygiene. Use deferrable operators for any wait. Keep your metadata DB healthy. Control DAG import times and size. Cap max active runs. For large backfills, scale workers up and set pools to avoid hammering APIs.

On observability, Step Functions execution graphs tell good stories for support. Temporal gives you fine grained visibility per workflow and activity. Airflow gives you per task logs and lineage. Pipe all three into your central metrics. Track retries, queue latencies, and state durations. Run red teams that cut network to a dependency and measure graceful degradation.

What We Would Pick This Quarter

• All AWS, light to medium complexity saga, cost sensitive. Pick Step Functions Standard. Keep each execution under 10k states, use service integrations first, push large data to S3.
• Complex saga with lots of human signals, long spans, or tricky compensations. Pick Temporal. Either Temporal Cloud or a managed self host if you have the ops depth. Budget base monthly for the control plane.
• Analytics or batch heavy shop. Pick Airflow. Keep transactional sagas out of it, and use deferrable operators for long waits.

Two caution flags regardless of choice.

• Never pass big payloads through the orchestrator. Use object storage. Keep messages under 32 KB where possible.
• Define idempotency keys and compensations before you write the happy path.

If you are on the fence or need to land this decision inside the quarter, we can help you converge with a short vendor stack decision sprint and then build the first critical path with you.