AI infrastructure has become one of the fastest-growing lines in many technology budgets. GPUs are expensive, whether rented by the hour in the cloud or bought for a data center, and demand for them often runs ahead of supply. Finance leaders want to know what they are getting for the money. Engineering leaders often answer with utilization charts.
Utilization matters, but it answers the wrong question. A GPU running at full load on retries, abandoned requests and outputs no one uses is busy, not productive. This post sets out a better way to measure.
Three levels of measurement
Level 1: resource metrics
These are the familiar ones: GPU utilization, memory use, hours provisioned and hours used. They tell you whether you are paying for idle capacity. They are essential for capacity planning, and they are where most teams stop.
Level 2: cost per token
The next step is to divide infrastructure cost by tokens processed, separately for input and output. This lets you compare self-hosted models with API pricing, and compare models and configurations with each other. It is a real improvement, but it still counts every token as equally valuable.
Level 3: cost per useful outcome
The measure that matters is what it costs to produce a result that was actually used. That requires two refinements.
First, count useful tokens: tokens that contributed to an accepted result. Exclude tokens spent on failed requests, timeouts, retries, discarded drafts, responses a person rejected and outputs that failed verification. In agentic systems, also look closely at the long chains of intermediate reasoning and tool calls that lead nowhere.
Second, tie cost to a unit of business work: a support ticket resolved, a contract reviewed, a code change merged and verified, an alarm correctly triaged. Cost per outcome is the number a CFO can compare with the cost of doing the same work another way.
An illustrative example
The following figures are hypothetical and for illustration only. Imagine a document-review service where infrastructure costs a fixed amount per month and the team reports high GPU utilization. On closer inspection:
- A meaningful share of requests are retries caused by timeouts under peak load.
- Many documents are processed twice because two upstream systems submit them.
- Some outputs fail a validation check and are discarded, then regenerated.
- The model used is far larger than the task needs for most document types.
None of these problems appear in a utilization chart. All of them appear immediately when you measure cost per accepted review. Fixing them can reduce cost per outcome substantially without buying a single additional GPU.
Where the waste usually hides
- Oversized models. A large general model handling tasks that a smaller or fine-tuned model would do as well. Routing requests by difficulty is often the single largest saving.
- Poor batching and scheduling. Serving software configured for latency when the workload is batch, or the reverse.
- Idle reservations. Capacity reserved for peak load and left idle the rest of the time, without workloads that can use the slack.
- Long contexts by default. Sending entire documents or conversation histories when a smaller, retrieved slice would do.
- Duplicate work. No caching for repeated prompts, embeddings or retrieval results.
- Failed agent runs. Agents that loop, retry or explore far more than they need to before succeeding or giving up.
Building the measurement
- Tag every request. Record the team, application, model, workload type and environment for each inference call. Without attribution, cost cannot be managed.
- Record the outcome. Capture whether each result was accepted, rejected, retried or verified. This usually requires a signal from the application, not only from the model server.
- Allocate infrastructure cost. Split GPU, memory, storage and network cost across requests in proportion to what they consumed.
- Report at all three levels. Resource metrics for operations, cost per token for engineering, cost per outcome for business owners and finance.
- Set targets per workload. A real-time voice agent and an overnight batch job have very different acceptable costs per outcome.
Optimizing with care
Once you can see cost per outcome, it is tempting to cut aggressively. Two cautions apply.
First, keep quality in the same view. A cheaper configuration that lowers the acceptance rate may raise cost per outcome, not lower it. Every optimization should be checked against your evaluation suite before it reaches production.
Second, make changes reversible. Infrastructure changes such as model swaps, quantization, batching settings and node pool changes should be applied one at a time, measured against a baseline, and rolled back automatically if quality or latency drops below target.
Why this matters beyond cost
Cost per useful outcome also improves planning. When you know what a unit of work costs today, you can forecast what a new workload or a growth in volume will cost tomorrow, and decide where to place it: on a frontier API, on private models in your cloud, or on owned hardware.
Our product TuQuant Prizm gives AI infrastructure teams this view, connecting GPU spend to the workloads and outcomes it supports, and helping them act on what they find. If AI infrastructure is a growing share of your budget, talk to us about measuring it in terms your business recognizes.