Open-weight economics

Most comparisons between hosted and self-hosted models argue about the price of a token. That is the wrong unit. When you host the weights yourself you do not buy tokens — you buy capacity, by the hour, and you pay for it whether or not anything is using it.

So this lab answers three questions in order, because the third one has no meaning until the first two are settled.

What has to fit? Weights are parameters times bytes per parameter, which quantisation changes. The KV cache is two caches across every layer, for every token of context, for every sequence in the batch — and it grows while the request is being served. If the two together exceed the device, you are buying another one.

How fast does it run? Prefill is compute-bound: reading a 9,000-token prompt is arithmetic, and it scales with tensor throughput. Decode is memory-bandwidth-bound: every single output token requires reading the entire weight matrix out of device memory again. That is why batching matters so much — the batch shares that one read — and why the interesting specification is bandwidth, not FLOPS.

What does it cost? Capacity divided by what you actually serve. At the defaults on this page that works out at around one-sixth utilisation, which means five-sixths of the bill buys nothing. The break-even figure the calculator reports is not a per-token comparison; it is the daily volume below which a hosted API is simply cheaper, and it is usually a multiple of what a pilot runs.

The part that carries the architecture

The journey above is split into two halves, and they cost very different amounts.

The deterministic shell — identity, the Context Contract, the policy and authority check, the typed capability call, the evidence write — is ordinary code on ordinary CPU. It runs in microseconds and costs a rounding error. It is reproducible: the same inputs give the same result, and a decision made three months ago can be replayed.

The forward pass is neither. It is the expensive part of the journey by several orders of magnitude, and it is the part that is sampled rather than computed. The scene draws that gap to scale because the gap is the argument: the controls that make an agent safe to run in an enterprise are close to free beside the model call they guard. Nobody is choosing between governance and cost.

That is also why the answer to “can we make this cost-effective?” is rarely “use a cheaper model”. It is: raise utilisation, batch harder, cut the prompt, cache what repeats, and move every decision you can out of the probabilistic half and into the deterministic one — where it is faster, cheaper, testable and provable.

Where the 1/358,000 figure comes from

The claim that the deterministic shell costs a fraction of the model call it wraps is quoted often enough that it should be checkable rather than repeated. Here is the whole derivation, with the inputs it depends on and the two ways it can be misread.

The configuration. One mid-size open-weight model, roughly 70B parameters, 80 layers, 8 KV heads, head dimension 128, at fp16. Served on top-tier datacentre GPUs — 80 GB, 3.3 TB/s memory bandwidth, 750 TFLOPS fp16 — at an illustrative $3.50 per GPU-hour. Workload: 9,000 prompt tokens and 700 output tokens per request, 12,000 requests a day, batch size 16.

At that shape the model needs 3 GPUs, runs at 16.7% utilisation, and costs $7,660.80 a month for 364,800 requests — so $0.021 a request.

The shell. Five deterministic operations run on ordinary CPU beside the model:

5,270 µs of one vCPU at $0.04 per vCPU-hour is $0.000000058556 per request.

$0.021 ÷ $0.000000058556 = 1 : 358,634.

It is a cost ratio, not a latency ratio

This is the misreading worth heading off. In time, the shell is 5.27 ms against a generation that takes seconds. If a 700-token completion takes around 14 seconds, the shell is about 1 : 2,657 — still negligible, but two orders of magnitude away from 358,000. Anyone quoting the figure as “the controls add a 358,000th of the latency” is wrong. The argument survives either way; the number does not transfer between them.

It moves with utilisation

The GPU costs the same when it is idle, so cost per request falls as the cluster fills, and the ratio narrows with it:

A well-packed cluster moves the figure toward 1 : 200,000. It does not approach anything that would make the shell a cost consideration.

What the figure does not include

• The step timings are engineering estimates, not measurements. They assume local, in-process work.
• A remote policy decision point changes the latency picture, not the cost one. A network round trip to an external policy service is milliseconds of wall time, which matters for user experience and not for the bill.
• A durable evidence write to managed storage has a real per-write cost that is not in the $0.04 vCPU-hour figure. At normal storage pricing it stays far below the model call, but it is not zero and it is not counted here.
• The GPU hourly rate is illustrative. Substitute your own.
• Nothing here covers the people. The platform cost and reliability lab models that, and at realistic settings the people who finish what the system got wrong cost several times what the tokens do.

What it is safe to say

At an illustrative 70B self-hosted configuration, the deterministic control layer costs roughly one 358,000th of the model call it wraps — a cost ratio, at 16.7% utilisation, falling toward 1 : 200,000 on a well-packed cluster. Anything shorter than that sentence is a slogan, and the slogan is the part that gets challenged.