LLM observability and cost: where your token spend actually goes

LLM observability is monitoring the language-model calls in your app: token usage, cost, latency, and quality, broken down by model and by feature. It matters because token spend is easy to rack up and hard to attribute, and most teams fly blind until the bill arrives. Here is what to track, how cost attribution works, and how to do it without a separate tool.

What LLM observability is

LLM observability is the established term for monitoring the language-model calls your application makes. For every call, you want four things: how many tokens it used, what that cost, how long it took, and whether the output was any good. Then you want to slice all of that by model and by feature, so "our AI spend" becomes "the summarizer on GPT-class models is 60% of the bill."

It is not a separate discipline so much as monitoring pointed at the AI parts of your backend. Everything from general log monitoring applies, watch the signal, alert when it drifts, but the signals are token-shaped: usage, cost per model, and output quality instead of just latency and errors.

Why token cost is hard to see

The bill is one big number, and almost nothing about it is obvious. Cost hides for a few reasons at once.

• Multiple providers. Spend is spread across OpenAI, Anthropic, Google, and others, each with its own pricing and dashboard.
• Priced per token, not per request. Two calls to the same endpoint can differ 50x in cost depending on prompt size and output length.
• Invisible drivers. A longer prompt, a retry loop, or an agent that calls the model ten times instead of once can multiply cost with no visible change in behavior.

The result is that most teams are not watching. In Grafana's 2025 observability survey, only 7% of teams were using LLM observability in production extensively.¹ The spend is real and growing, and it is mostly unmeasured.

What to track: tokens, cost, latency, quality

Four signals cover the large majority of what you need from an LLM call.

You do not have to invent the shape of this data. OpenTelemetry, the open standard for telemetry, is actively defining GenAI semantic conventions, a standard set of attributes for LLM calls including token usage, so that model telemetry looks the same across frameworks and vendors.² Emitting these means your LLM data joins the rest of your traces instead of living in a silo.

How cost attribution actually works

The hard part is not measuring one call, it is attributing spend across all of them. The mechanics are straightforward: for each call you capture input and output token counts, either read from the provider's API response or inferred when the response does not include them, then multiply by that model's per-token price to get cost.³ Do that for every call and tag each with the feature, user, or team it belongs to.

That tagging is what turns a total into an answer. Instead of "we spent $4,000 on models," you get "the onboarding summarizer is $2,400 of it, mostly on oversized prompts." The two common ways to collect it are a proxy that every model call routes through, or SDK instrumentation on the call site. Either works; what matters is that every call carries the labels you want to slice by.

LLM observability without a separate stack

Fixter bundles LLM observability into your regular monitoring instead of making you run a separate tool for it. Your model calls are instrumented through the same OpenTelemetry-based SDK as the rest of your backend, so token usage and cost by model and by feature sit next to your logs, metrics, and traces. You ask about spend in plain language, and you can alert on it the same way you alert on anything else.

The point of LLM observability is not a prettier dashboard, it is being able to ask one place where the tokens went and get an answer with a cause. From here, if your app runs agents rather than single calls, see AI agent monitoring, and for the broader idea of watching your backend in plain language, what you can do with your logs.

Frequently asked questions

What is LLM observability?

LLM observability is monitoring the language-model calls in your application: token usage, cost, latency, and output quality, broken down by model and by feature. It is the slice of monitoring aimed at the AI parts of your app, so you can see what the model is doing and what it is costing you instead of only getting a bill at the end of the month.

How do you track LLM token usage and cost?

You capture the input and output token counts for every model call, either read from the provider's API response or inferred, then multiply by that model's per-token price to get cost. The useful part is attribution: tying those tokens and dollars to a specific user, feature, or team, so you know where the spend is going, not just the total.

Why is LLM cost hard to see?

Because it hides. Spend is spread across multiple providers, priced per token rather than per request, and driven by things like prompt size and retry loops that are invisible in a normal bill. A single runaway agent can multiply cost quietly. Without per-feature attribution you see one large number and cannot tell what caused it.

What is the difference between LLM observability and LLM evaluation?

Observability is about production: what the model did, cost, and latency on real traffic. Evaluation is about quality: scoring outputs against expected answers, often before you ship. They overlap on quality signals, but observability answers 'what is happening and what does it cost' while evaluation answers 'is the output good'.

Do I need a separate tool for LLM observability?

Not necessarily. LLM calls are just another thing your backend does, so token usage and cost can live in the same monitoring as your logs, metrics, and traces. Bundling it means one place to look instead of a separate LLM tool alongside your general monitoring, which is how Fixter handles it.