The Ten Layers Behind Every Multi-Agent System
A multi-agent demo has three boxes. Production has ten layers, each owning one job and failing in one familiar way. Most incidents start in a layer nobody owned.
Written as at 18 August 2026.
agentic-aimulti-agent-systemsreference-architectureenterprise-architecture
Most multi-agent diagrams have three boxes: an orchestrator, some workers and a few tools. They explain the idea well and the production system badly. Between a person asking for something and a record being written sit ten distinct jobs, and each needs an owner.
This piece lays out those ten layers: eight that every request passes through, and two that run through all of them. For each, it names what the layer owns and the failure you are most likely to meet there. The running example is Poly, the studio I am building, where a producer asks a team of agents to build Lesson 3.2 on prompt caching.
A request crosses eight layers and is watched by two
Follow the lesson request down the stack. The producer starts the run from the studio’s interface, which returns a job ID because the build takes minutes, not seconds (L1). The edge gateway authenticates the producer, screens the input, attaches a trace ID and sets a hard budget for the whole run: in the illustrative build, $1.50 and 60 turns (L2).
The orchestrator turns the request into a task graph and writes it to a durable ledger (L3). Each worker then runs in its own isolated session with its own prompt, model tier and tool allowlist (L4). Before every model call, its window is assembled: stable prefix first, then the brief, then whatever it retrieves just in time (L5).
A model tier chosen for the role does the reasoning (L7). When the model wants a tool, the call goes through a gateway that checks identity, arguments and egress (L6). Outputs land in an artifact store under a task-scoped prefix (L8). Observability (X1) and security (X2) are not stops on this route. They see and constrain every hop. You can watch the same run step by step in the agent simulator.
Each layer owns one job and fails in a familiar way
The table is the reference. The typical failures are illustrative, drawn from the kind of incidents the Poly build is designed to survive.
| Layer | Owns | Typical failure |
|---|---|---|
| L1 Channel and experience | Starting runs, showing progress, presenting approvals | A 20-minute run held on one HTTP request times out at the load balancer; the work finishes and nobody sees it |
| L2 Edge gateway and guardrails | Authentication, tenant isolation, input screening, rate limits, run budget, trace ID | A budget set here but never passed down, so workers overspend |
| L3 Orchestration and control state | Task graph, delegation, merging, retries, stop conditions, the ledger | Plan held only in memory; a container restart leaves the run “running” forever |
| L4 Sub-agent runtime | One isolated session per worker: prompt, tier, tools, turn limit, sandbox | A worker inherits the orchestrator’s credentials |
| L5 Context and memory | Window assembly, prompt caching, compaction, long-term memory | A timestamp at the top of the prompt breaks the cache on every call |
| L6 Tool gateway | Tool registry, per-agent allowlists, argument checks, egress, approvals, idempotency | A retried publish call with no idempotency key creates the lesson twice |
| L7 Models and inference | Model per role, structured outputs, fallback, batch or real time | A burst of rate-limit errors at fan-out, retried in lockstep |
| L8 Data and artifacts | Artifact store, knowledge base, classification, retention | A concurrent retry overwrites another task’s artifact |
| X1 Observability and evaluation | One trace per run, spans per agent, model and tool call, evaluation gates | Cost is visible per run but not per agent, because trace context never reached the workers |
| X2 Security and governance | Agent identity, least-privilege scopes, trust zones, audit, change control | One over-privileged agent combines private data, untrusted content and a way to send |
The last row is Simon Willison’s lethal trifecta (2025): an agent that can read private data, process untrusted content and communicate externally can be steered by one poisoned page into leaking what it can read. The fix sits in L4 and L6, by never giving one worker all three. L6 is also where the Model Context Protocol fits: it standardises how tools are described and called, and leaves the policy decisions to you.
Budgets, identity and trace context travel down the stack
Look at the failure column again. Several failures are not inside one layer at all. They sit at the seams, where something decided in one layer was never handed to the next.
The budget is the clearest case. Poly sets $1.50 at the edge, but an evaluator loop runs in L3 and L4. Unless each brief carries its own limit and the ledger checks it before every delegation, the edge budget is a number on a dashboard. The same applies to trace context. The OpenTelemetry semantic conventions for generative AI define span names such as invoke_agent, chat and execute_tool, but a span is only useful if the worker received the parent’s trace ID. Identity follows the same rule: each worker needs its own, scoped to its task, not a copy of the orchestrator’s.
This is why I group the layers the way the figure does. L2, L3, L6 and the two cross-cutting layers form what I call the Deterministic Shell: the code around the model that owns identity, authority, state and commitment. The model calls in L4, L5 and L7 are the Probabilistic Core, which reasons and proposes inside it (my construct; see Why Enterprise AI Needs a Deterministic Shell). A wrong model output should not be able to breach the shell.
Use the layers as a design review
The stack is most useful as a set of questions asked of every design, one layer at a time.
- Who owns it? Name a team or person for each of the ten. Blank rows are where incidents will come from.
- What was decided? Each layer has a few real decisions: synchronous or asynchronous at L1, fixed workflow or planner at L3, cache breakpoints at L5, read-only or side-effecting tools at L6.
- What crosses the seam? For each boundary between layers, list what must travel with the request: budget, trace ID, identity, trust tag.
- How would you see it fail? Point to the span, metric or ledger entry that would show the typical failure for that layer.
A design that can answer all four for all ten layers is ready for a pilot. One that cannot answer the third is not, however good its prompts.
The ten layers are a map, not a product list. Your platform may merge several of them into one managed service, and that is fine as long as you still know which job each part does. The stack also says nothing about whether the work needs multiple agents at all. That question comes first, and it is the subject of Start With One Agent.
Who does what
Executives: ask for the ten-row ownership table for every agent system heading to production. Any row without a name is a funded risk.
Architects: draw each design against the stack and mark what crosses every seam. Treat the tool gateway and the ledger as platform services, not per-project code; Zero Trust for AI Agents covers the gateway’s side of this.
Engineers: test the seams directly. Assert that a worker span carries the parent trace ID, that a brief carries its budget, and that a worker credential cannot call a tool outside its allowlist.
This article is part of The Agentic AI Architect, a series from the author of The Contextual Agentic Enterprise. Each piece matches a module of his course in development, Agentic AI Architect Pro.
References
- The Contextual Agentic Enterprise.
- OpenTelemetry. "Semantic conventions for generative AI systems". https://opentelemetry.io/docs/specs/semconv/gen-ai/
- Willison, S. (16 June 2025) "The lethal trifecta for AI agents: private data, untrusted content, and external communication". https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
- Model Context Protocol specification. https://modelcontextprotocol.io/specification
Related
Why Enterprise AI Needs a Deterministic Shell
Don't try to make the model deterministic or simply trust it. Let it propose, and put identity, authority, schema, state and commitment in a shell that decides at the Commit Boundary.
The One-to-Five Trap: Why Multi-Agent Systems Become Distributed Systems
Splitting one agent into five does not add a team. It adds a distributed system. Each boundary has to earn its coordination cost, or it should be a function.
The Arithmetic of Agent Cost
Agent cost is mostly input re-read on every turn. Multi-agent designs may read more tokens yet cost less per result, so budget each task and measure cost per successful task.