Event-Driven Agents & Async Orchestration

Most agent demos run synchronously: a request comes in, the agent thinks, calls a few tools, and returns an answer in one breath. That model collapses the moment agents need to wait on a human, a slow API, or another agent. Event-driven architecture and async orchestration are how production GaaS systems escape the request-response trap. Instead of one long-held connection, agents react to events on a bus, suspend cheaply while waiting, and resume when something happens. This piece explains the pattern, why it changes agent economics, where teams get it wrong, and how it fits the rest of the agent infrastructure stack.

Why Synchronous Agents Hit a Wall
What "Event-Driven" Actually Means for Agents
The Core Building Blocks
The Event Bus and Message Queue
Suspend and Resume: Durable State
The Orchestrator vs. Choreography Question
How Async Orchestration Reshapes Agent Economics
Common Failure Modes and How Teams Avoid Them
When You Should Not Go Event-Driven
Where This Fits in the GaaS Stack
Insights Most People Overlook
Frequently Asked Questions
Conclusion
References

Why Synchronous Agents Hit a Wall

Watch almost any agent walkthrough and you'll see the same shape. A user types something, the model reasons, it fires off two or three tool calls, and a tidy answer comes back. It feels like magic because the whole thing happens inside a single HTTP request, usually in under thirty seconds.

Now put that agent into a real business process. A procurement agent has to email a vendor and wait two days for a quote. A claims agent needs a human adjuster to approve a payout before it continues. A research agent kicks off a web crawl that takes nine minutes. The synchronous model has no good answer for any of these. You either hold a connection open and burn money on idle compute, or you crash into a gateway timeout and lose the agent's entire working state.

This is the wall. And it's not an edge case, it's most of what enterprises actually want to automate. The interesting work is rarely a thirty-second round trip. It's a workflow that unfolds over minutes, hours, or days, with waiting baked in. For anyone selling agents as a service, this is the difference between a clever demo and something a customer will pay a per-outcome fee for. The economics of per-outcome agent pricing only work if the underlying execution doesn't bleed money while it waits.

Event-driven architecture is the established answer to this problem, and it predates agents by decades. The novelty is applying it to systems whose "work units" are non-deterministic LLM calls rather than predictable functions.

What "Event-Driven" Actually Means for Agents

Strip away the buzzwords and the idea is simple: instead of an agent being called and holding the line until it's done, the agent reacts to events and emits new events of its own.

An event is just a record that something happened. "Quote received." "Human approved step 3." "Crawl finished, here are 40 URLs." "Payment failed." These events flow onto a shared channel, a bus or a queue, and components subscribe to the ones they care about. When the relevant event lands, the agent wakes up, does a bounded chunk of work, possibly emits another event, and goes back to sleep.

The mental shift is from imperative ("do this, then this, then this, while I hold your hand the whole time") to reactive ("when X happens, do Y"). Martin Fowler's long-standing breakdown of what people mean by event-driven is still the clearest map of the territory, and it's worth reading because most "event-driven agent" pitches quietly conflate four different patterns under one label.

For agents specifically, three properties matter most:

Decoupling in time. The thing that produces an event and the agent that handles it don't have to be alive at the same moment. The vendor's quote can arrive at 3 a.m. while no agent process is running at all.
Decoupling in identity. The producer doesn't need to know which agent, or how many agents, will react. This is what makes multi-agent fan-out tractable.
Natural suspension points. Every "wait for event" is a place the agent can be torn down to nothing and rebuilt later, which is the key to not paying for idle time.

That third property is where async orchestration earns its keep.

The Core Building Blocks

The Event Bus and Message Queue

At the center sits the transport. This might be a message broker like Apache Kafka or RabbitMQ, a cloud queue like AWS SQS or Google Pub/Sub, or a purpose-built agent event bus inside a runtime. The job is the same: reliably accept events, hold them, and deliver them to subscribers.

The properties you care about are ordering, delivery guarantees, and replay. Most brokers give you at-least-once delivery, which means your agent handlers must be idempotent, running the same event twice can't double-charge a customer or send two emails. This sounds obvious and is violated constantly, because LLM-driven steps are harder to make idempotent than ordinary code. An agent re-running a "draft and send" event won't produce the identical email the second time; it'll produce a different email. You need an idempotency key at the action boundary, not just at the model boundary.

Suspend and Resume: Durable State

This is the heart of async orchestration, and it's where durable execution engines come in. The trick is to persist the agent's entire state, its conversation history, its plan, its position in the workflow, so the running process can be discarded entirely while it waits, then faithfully reconstructed when an event arrives.

Frameworks like Temporal pioneered this for general workflows, and the agent world has adopted the pattern wholesale. Temporal's own explanation of durable execution frames it well: the goal is code that survives process crashes, deployments, and arbitrarily long waits as if nothing happened. For an agent, "as if nothing happened" means a model that picks up its reasoning chain exactly where it left off, even if the gap was three days and four deployments.

The hard part isn't saving state, it's the boundary between deterministic orchestration and non-deterministic model calls. If you naively replay an agent's history and the model returns a different decision than it did the first time, your durable workflow forks reality. Production systems solve this by treating each completed LLM call as a recorded, immutable result that gets replayed from the log rather than re-invoked. This connects directly to how teams handle state management for stateful agents and long-running agent execution, they're three views of the same underlying problem.

The Orchestrator vs. Choreography Question

There are two ways to wire agents together with events, and the choice shapes everything downstream.

Orchestration uses a central conductor, often a supervisor agent, that holds the workflow logic and tells each agent what to do next in response to events. It's easier to reason about, easier to observe, and easier to change, because the logic lives in one place. The tradeoff is a central bottleneck and a single point of failure.

Choreography has no conductor. Each agent simply reacts to events and emits its own, and the overall behavior emerges from those local rules. It scales beautifully and has no central bottleneck, but debugging it can feel like investigating a crime with no witnesses, the workflow exists only as a trail of events across the system.

Most successful GaaS deployments land on a hybrid: choreography between coarse-grained services, orchestration within a bounded workflow. The supervisor-agent architecture is essentially orchestration applied to a team of agents, while broad multi-agent coordination patterns lean choreographic. Knowing which you're building, and not accidentally drifting between them, is half the battle.

How Async Orchestration Reshapes Agent Economics

Here's the part that matters to anyone running agents as a business rather than a hobby.

In a synchronous world, a waiting agent is a paid-for agent. If your claims agent waits four hours for human approval and you've been holding a warm container the whole time, you've paid for four hours of idle infrastructure to do nothing. Multiply that across thousands of concurrent workflows and the unit economics fall apart.

Async orchestration breaks the link between wall-clock duration and compute cost. A workflow that spans three days might consume ninety seconds of actual model and CPU time. The other 71-plus hours, the agent doesn't exist as a running process, it's a row in a database. You pay for storage, which is nearly free, instead of compute, which is not.

This is precisely what makes per-outcome and per-task pricing viable. McKinsey's analysis of the economic potential of agentic AI leans heavily on automating multi-step business processes, and multi-step business processes are, almost by definition, long-running and wait-heavy. You cannot serve those at a profit on synchronous infrastructure. The orchestration layer is what converts a technically-impressive agent into a sellable one, which is why it shows up in any serious decomposition of the GaaS cost stack.

There's a second-order effect too. Because async agents suspend cheaply, you can afford to put humans in the loop without penalty. A synchronous system treats human review as expensive dead time; an async system treats it as just another event to wait for. That quietly makes safer, more supervised agents cheaper to run, which inverts the usual assumption that oversight is a cost center.

Common Failure Modes and How Teams Avoid Them

The pattern is powerful, but it has sharp edges. The ones that bite hardest in practice:

Non-idempotent side effects. As noted above, at-least-once delivery plus non-deterministic models is a dangerous combination. The fix is to make every external action, sending an email, charging a card, creating a ticket, guarded by an idempotency key derived from the workflow, not the model output. Treat the model as a planner and the action layer as the place where exactly-once semantics get enforced.
Lost or poisoned events. Events get dropped, malformed, or stuck. Without dead-letter queues and replay, a single bad event can silently strand a workflow forever. Mature setups treat the event log as the source of truth and can replay it to reconstruct or resume any workflow.
The observability black hole. When work is spread across an event bus and dozens of suspend/resume cycles, a single failed outcome has no obvious stack trace. You need correlation IDs that thread through every event and tool call so you can reconstruct one workflow's journey end to end. This is exactly why the observability stack for agent infrastructure is non-negotiable for event-driven systems, it's far more important here than in synchronous designs.
Runaway fan-out. Choreography makes it trivially easy to emit an event that triggers ten agents, each of which emits events triggering ten more. Without backpressure and rate limits, an innocent workflow can saturate the whole system. Agent gateways and queue-depth limits exist precisely to contain this.
State schema drift. When you suspend an agent for three days and deploy new code twice in between, the resumed state may no longer match what the new code expects. Versioning the workflow state, and the agent and its tools, is the unglamorous discipline that keeps long-running systems from rotting.

When You Should Not Go Event-Driven

Worth saying plainly, because the pattern gets oversold: if your agent genuinely does its job in one synchronous pass, a support-reply drafter, a quick classification agent, a synchronous coding assistant, event-driven architecture adds cost and complexity you don't need. You'll have introduced a message broker, durable state, and distributed-systems debugging to solve a problem you didn't have.

The honest test is whether your workflows wait. Wait on humans, wait on slow external systems, wait on other agents, or run long enough to risk a timeout. If they do, async orchestration is close to mandatory at any real scale. If they don't, stay synchronous until they do. Premature distribution has killed more agent projects than missing infrastructure has.

Where This Fits in the GaaS Stack

Event-driven orchestration isn't a standalone product, it's connective tissue. It sits beneath the agent runtime and above the model and tool layers, coordinating the whole. It depends on durable execution to suspend safely, on a memory layer so a resumed agent remembers its context, and on tool-calling reliability so the actions it triggers actually land.

It also reframes adjacent decisions. Your choice of agent orchestration framework is largely a choice about how event-driven you want to be and how much of this you build versus buy. Your A2A and protocol decisions determine what an "event" even looks like when it crosses an organizational boundary. And the entire reliability conversation, retries, fallbacks, circuit breakers, is really a conversation about what happens to in-flight events when something fails.

The short version: as GaaS matures from single-shot agents toward genuine autonomous workflows, the center of gravity moves from the model to the orchestration layer. The model gets commoditized; the thing that reliably coordinates thousands of long-running, waiting, fault-tolerant agent workflows does not. That's where durable value is accruing.

Insights Most People Overlook

The bottleneck migrates from tokens to coordination. Everyone optimizes prompt and token cost. But in long-running agent fleets, the dominant operational cost and risk isn't inference, it's the orchestration layer's reliability. A dropped event costs you a failed outcome, and failed outcomes are what customers actually notice. The org that wins isn't the one with the cheapest tokens; it's the one whose workflows never silently die.
Human-in-the-loop becomes a feature of the cost model, not a tax on it. Conventional wisdom treats human review as expensive friction. Under async orchestration, a human approval is just another event you suspend on, costing storage, not compute. This quietly makes well-supervised agents cheaper than you'd expect, and it's an argument for putting more humans in the loop, not fewer, while autonomy matures.
At-least-once delivery and non-deterministic models are a genuinely novel hazard. Classic event-driven systems assume idempotent handlers are achievable with ordinary engineering. LLM steps break that assumption: replaying an event regenerates a different output. Teams that lift event-driven patterns straight from the microservices playbook without moving exactly-once enforcement down to the action layer will ship double-emails and double-charges. This is new, and most tooling hasn't fully caught up.
Choreography is a debugging liability that scales beautifully, pick your pain deliberately. The industry romanticizes "emergent" multi-agent behavior, but emergence is the enemy of observability. The right move is usually boring: orchestrate within a workflow where you need to reason about it, and reserve choreography for coarse boundaries where scale matters more than traceability. Drifting between the two by accident is how systems become unmaintainable.
The state log, not the model, is the real moat. A faithfully replayable event-and-state log lets you resume, audit, debug, and even retroactively fix workflows. It's the agent equivalent of double-entry bookkeeping. Most teams treat it as plumbing; the ones who treat it as a first-class asset can offer reliability guarantees their competitors structurally cannot.

Frequently Asked Questions

Is event-driven architecture the same as async orchestration? They overlap but aren't identical. Event-driven is the broad style, components react to events. Async orchestration is the specific discipline of coordinating long-running, suspendable work, usually built on top of an event-driven substrate plus durable state. You can have simple event-driven systems with no orchestration, but you can't have serious async orchestration without an event-driven backbone.

Do I need Kafka to build event-driven agents? No. Kafka is excellent at high-throughput, replayable event streams, but plenty of agent workloads run fine on a managed cloud queue or even a database-backed job queue. Start with the simplest transport that gives you durability and replay, and graduate to a heavier broker only when throughput or retention demands it. The pattern matters more than the product.

How does this relate to durable execution engines like Temporal? Durable execution is the mechanism that makes suspend-and-resume reliable. An event-driven agent needs somewhere to safely park its state while waiting for the next event; durable execution engines provide exactly that, with crash-safety and replay built in. Many teams use a durable execution engine as their orchestration layer rather than wiring a broker and a state store together by hand.

What makes agent orchestration harder than ordinary workflow orchestration? Non-determinism. Traditional workflow steps return predictable outputs, so replay is trivial. LLM steps don't, the same input can yield different reasoning and different tool choices. That forces you to record and replay completed model calls as immutable facts rather than re-invoking them, a constraint classic orchestration engines weren't designed around.

How do I keep costs down for agents that wait for days? This is the central payoff of async orchestration: suspend the agent to durable state so no compute runs while it waits. You pay for cheap storage during the wait and compute only during active work. A three-day workflow might bill ninety seconds of model time. If your bill scales with wall-clock duration, you're still running synchronously somewhere and should fix that first.

Where do human-in-the-loop checkpoints fit? Cleanly. A checkpoint is just an event the workflow waits on, "human approved" or "human rejected." The agent suspends, a human acts whenever they get to it, and the resulting event resumes the workflow. Because the wait is nearly free, you can add review gates without the cost penalty a synchronous system would impose.

Conclusion

Event-driven agents and async orchestration are what turn a clever model into a dependable service. The core idea is old, react to events, decouple producers from consumers, but applying it to non-deterministic agents that must wait on humans, slow systems, and each other introduces genuinely new problems: idempotency under at-least-once delivery, replay across model calls, and observability across suspend-resume cycles.

The payoff is decisive. Async orchestration severs the link between how long a workflow takes and how much it costs, which is the precondition for per-outcome pricing, generous human oversight, and workflows that run for days. As the broader GaaS stack matures, value is migrating from the model toward the layer that reliably coordinates thousands of long-running agent workflows. Get the orchestration right, durable state, idempotent actions, a replayable event log, and clear observability, and the rest of the stack has something solid to stand on. Get it wrong, and even the best model produces a fleet of agents that silently fail in ways no one can trace.

Event-Driven Agents and Async Orchestration: The Infrastructure Behind Autonomous Workflows

Table of Contents