Hosting agentic applications on AWS — compute, async, data, security
Technologies

Hosting agentic applications on AWS

Production reference for running LangGraph / Deep Agents on AWS: Fargate vs EKS vs Lambda, SQS async workers, Aurora checkpointers, S3 offload, Bedrock, VPC/IAM sandboxes, HITL, observability, cost levers, and interview scripts.

Engineering manager 17 min read ·

Why agentic hosting ≠ a normal web app

Shipping a support Deep Agent or research LangGraph to production on AWS fails for infrastructure reasons more often than model reasons: ALB timeouts kill 12-minute runs, workers lose state on deploy, tool dumps blow Postgres rows, a code-exec tool can reach the production database, and nobody can correlate a LangSmith trace to a CloudWatch log line.

AWS reference architecture: clients through CloudFront/API to ECS API and agent workers with SQS, Aurora, Redis, S3, Secrets, and Bedrock.
Reference shape — memorize this diagram; customize compute and model path to your constraints.
01Compute

Fargate · EKS · Lambda limits.

02Async

202 + SQS + workers.

03Data

Aurora · S3 · store.

04Secure

IAM · sandbox · HITL.

Pair with Deep Agents Q&A, Context management, Long-running tasks, Kubernetes, and API gateway.

Production requirements to name first

  • Latency class — interactive chat (< few seconds to first token) vs batch research (minutes–hours).
  • Run duration — drives compute (Lambda 15 min wall) and whether you need queues.
  • Side effects — refunds, emails, deploys → HITL, idempotency keys, audit.
  • Tool risk — read-only APIs vs shell/browser → sandbox account.
  • Data residency — Bedrock region vs EU external APIs; VPC endpoints.
  • Multi-tenant — per-tenant quotas, KMS keys, queue isolation.
  • Cost ceiling — tokens + GPU/sandbox hours; budget alarms mandatory.
Interview takeaway

Clarify duration, side effects, and tool risk before picking ECS vs Bedrock Agents.

Compute: where the agent loop runs

Six compute options: Fargate, EKS, Lambda, Step Functions, Bedrock Agents, sandbox ECS.
Default to Fargate workers; escalate only with a clear platform reason.

ECS on Fargate (recommended default)

API service (thin): auth, create run_id, enqueue, status, streaming gateway. Worker service: LangGraph / Deep Agents loop, long timeouts, higher CPU/memory. Scale workers on SQS ApproximateNumberOfMessagesVisible. No EC2 patching; task IAM roles pull Secrets Manager.

EKS

Choose when the org already standardizes on Kubernetes, needs GPU node groups for local models, or complex sidecars. Same async pattern — Deployments + HPA/KEDA on queue depth. Don't invent EKS just because agents feel "platform-y." See Kubernetes.

Lambda

Fine for short tool proxies, webhooks, or summarization micro-steps. Bad as the primary Deep Agent host: 15-minute max, cold starts, awkward long streaming, painful local filesystem semantics. If someone proposes "all agents on Lambda," ask about run duration.

Step Functions

Excellent for deterministic sagas around the agent (KYC → agent fraud check → human approve → payout). Express the business graph in Step Functions; invoke a Fargate task or ECS run task for the agentic hop. Complements LangGraph — doesn't replace it for open-ended reasoning.

Amazon Bedrock Agents

Managed agent loop + Knowledge Bases + action groups. Faster if you're all-in on AWS models and simple tools. Trade-off: less control than LangGraph checkpointers/subgraphs; migration cost if you outgrow it. Interview: "Bedrock Agents for MVP with AWS-only tools; LangGraph on Fargate when we need custom graphs and multi-provider models."

Always async: 202 + queue + workers

Flow: Client POST to API creates run_id, SQS job, worker runs LangGraph, notify via WS or webhook.
Same pattern as long-running tasks — agent runs must survive deploys and timeouts.
  1. Client POST /v1/runs with goal + idempotency key.
  2. API writes run row (pending), enqueues SQS message {run_id, tenant_id}, returns 202 + run_id.
  3. Worker receives message, loads checkpointer thread, runs graph with max steps / token budget.
  4. Progress: update DynamoDB/Aurora status; optional EventBridge for UI; WebSocket/SSE via API for live tokens.
  5. On success/failure: set terminal status; DLQ after N receives; alarm on DLQ depth.
  6. Visibility timeout ≥ max expected step cluster; heartbeat/extend while running.

Deep dive on the pattern: Managing long-running tasks. Agent-specific twist: checkpointer lets a different worker resume the same thread_id after reclaim or crash — design tools to be idempotent.

Interview takeaway

202 + SQS + Fargate workers + DLQ is the AWS default for agentic products.

Data plane: checkpointer, chat, offload, cache

Data plane: Aurora Postgres, S3, Redis, optional DynamoDB, OpenSearch, Secrets Manager.
Map LangGraph layers to AWS services — don't invent five databases without a reason.

Aurora PostgreSQL (or RDS Postgres)

Primary home for LangGraph PostgresSaver checkpoints and usually the append-only chat messages table (UI history). Optionally LangGraph store tables for long-term prefs. Multi-AZ; connection pooling (RDS Proxy / PgBouncer) because workers are bursty. See Context management for the UI-vs-checkpointer split.

S3

Tool dumps, generated reports, user uploads, Deep Agents filesystem backend. Store s3://bucket/runs/{run_id}/… pointers in state — never 50MB JSON in Postgres. Lifecycle rules to IA/Glacier; block public access; SSE-KMS. Presigned URLs for downloads — same ideas as large blobs.

ElastiCache Redis

Rate limits per tenant, idempotency keys for tool side effects, pub/sub for live run status to API gateways. Not your system of record for checkpoints.

DynamoDB (optional)

High-QPS run status keys, distributed locks, TTL for ephemeral job metadata. Useful when status polling dwarfs Postgres capacity — not required on day one.

RAG corpus

OpenSearch Serverless, Aurora pgvector, or Bedrock Knowledge Bases. Keep retrieval on the data plane; agents call a retrieve tool. See RAG and Vector databases.

Models: Bedrock vs external APIs

  • Amazon Bedrock — IAM auth (no long-lived API keys in app), regional data path, model choice (Anthropic/Amazon/Meta on Bedrock), Provisioned Throughput for steady load. Prefer VPC interface endpoints where available.
  • External OpenAI / Anthropic / Google — Secrets Manager for keys; egress via NAT or preferred private connectivity; strict security-group egress allowlist to provider CIDRs/domains; timeout + retry with jitter.
  • Hybrid — router: cheap Bedrock model for classify/summarize; frontier external model for hard reasoning — budget tags per model id.

Say this: "I'd default to Bedrock for IAM and residency; keep LangGraph model-agnostic so we can swap providers. Keys only in Secrets Manager; workers get task roles."

Edge and networking

  • CloudFront — TLS, caching for static console assets, WAF association.
  • API Gateway (HTTP API) or ALB — JWT/Cognito authorizer; throttle per tenant; path routing to API service. See API gateway.
  • Private subnets for API + workers; public only for ALB/NAT as needed.
  • VPC endpoints — S3 gateway, Secrets Manager, Bedrock (where supported) to cut NAT $ and exposure.
  • WebSockets — API Gateway WebSocket or ALB sticky to a gateway service for token streaming; still don't run the agent inside that connection's process if runs are long.

Security, IAM, and tool sandboxes

Trusted VPC versus sandbox account for code and browser tools.
Bulkheads: compromised shell tool must not reach prod CRM credentials.
Interview takeaway

Draw two boxes: trusted workers and sandbox. Name IAM roles and "no route to prod DB."

Human-in-the-loop on AWS

Worker interrupt to checkpoint to EventBridge to Slack to resume same thread.
LangGraph interrupt + checkpointer + EventBridge/SNS — humans can take hours.
  • Graph hits interrupt / interrupt_on before a side-effect tool.
  • State persisted in Postgres checkpointer; SQS message completes (don't hold the worker).
  • Emit EventBridge event → SNS/Slack/Teams with Approve/Deny deep links.
  • Human action hits POST /runs/{id}/resume with decision; API enqueues resume job.
  • Worker loads same thread_id, continues; refund tool uses idempotency key.

Observability and eval in production

CloudWatch, X-Ray/OTel, LangSmith, and product metrics correlated by run_id.
One run_id to rule them all — logs, traces, LLM spans, business KPIs.
  • Structured logs — JSON to CloudWatch with run_id, tenant_id, thread_id, node name.
  • Metrics — queue depth, worker CPU, run duration p95, error rate, tokens in/out, $ estimate, HITL wait time.
  • Tracing — OpenTelemetry → X-Ray or third party; child spans per tool call.
  • LangSmith (or equivalent) — prompt/tool traces and golden eval suites on deploy.
  • Alarms — DLQ > 0, token $ spike, sandbox account spend, Aurora connections.

Scaling, multi-tenant, and resilience

  • Scale workers on queue depth — not only CPU (CPU is often waiting on the model).
  • Per-tenant queues or weighted fair queuing — one noisy tenant shouldn't starve others (SQS + separate queues or custom scheduler).
  • Backpressure — reject or delay new runs when global concurrency or $ budget exceeded.
  • Multi-AZ — Fargate + Aurora Multi-AZ; checkpoints survive AZ loss.
  • Drain on deploy — stop scheduling new SQS receives; let in-flight finish or rely on checkpointer resume.
  • Idempotent tools — at-least-once SQS + retries will double-call otherwise.

Worked example: support Deep Agent on AWS

Product: multi-day support agent that retrieves policies (RAG), looks up orders, and proposes refunds with HITL.

  1. Edge: CloudFront + API Gateway HTTP API + Cognito JWT + WAF.
  2. API (Fargate): create run, enqueue, status, WebSocket fan-out via Redis pub/sub.
  3. Queue: SQS standard + DLQ; visibility 15 min with heartbeat extension.
  4. Workers (Fargate): Deep Agents / LangGraph; Bedrock Claude; max steps 40; interrupt on create_refund.
  5. Data: Aurora (checkpointer + chat + store prefs); S3 for ticket attachments; OpenSearch for policy RAG.
  6. HITL: EventBridge → Slack approve → resume API.
  7. Sandbox: none needed if tools are HTTP APIs only; if "run SQL diagnostic," spin sandbox account task.
  8. Obs: run_id in CloudWatch + LangSmith; alarm on DLQ and $/day.

Cost and performance levers

Rapid-fire interview Q&A

Interview takeaway

Host agents like durable async workers with a serious data plane and blast-radius bulkheads — AWS services map cleanly once you accept that framing.

Interview Q&A by level

Practice saying these out loud for hosting agents on AWS. Interviewers grade clarity and judgment more than buzzwords.

Interview takeaway

Match depth to the bar: define → trade off → operate. Don't dump principal answers in an entry-level screen.

Wrapping up

Production agentic apps on AWS are async, stateful, and security-partitioned. Start from the reference: Fargate + SQS + Aurora + S3 + Bedrock/IAM, add Redis and OpenSearch when needed, sandbox anything that can execute untrusted code, and correlate everything with run_id.

Related: Deep Agents Q&A · Context management · Long-running tasks · Agentic architectures · Kubernetes · API gateway · Message queues.

technologiesagentsawsproductionecslanggraphbedrockinfrastructureinterviews

← Lattice