Token Economics: Navigating the Cost of Intelligence

The Price of Thought

We are witnessing the commoditization of intelligence. Just as cloud computing turned server hardware into a utility bill, Large Language Models have turned reasoning into a metered resource. As we move from GPT-4 to today's tiered lineups (GPT-6 Astra at the top, Gemini 3.8 Flash at the cheap end), published per-token prices have fallen by an order of magnitude — and aggregate usage has risen fast enough to more than swallow the saving. That is the whole problem in one sentence: the unit got cheaper and the bill got bigger.

A single complex agentic workflow—say, researching a market, scraping 50 websites, synthesizing the data, and generating a report—can now burn $5 in tokens in a few minutes. For a SaaS startup, this destroys the traditional "marginal cost of zero" advantage of software.

Opex vs Capex: The New Balance

In the SaaS era, your biggest cost was R&D (engineer salaries). Hosting was negligible. In the Agentic era, 'compute' is the new rent. Your cloud bill isn't just for hosting database rows; it's your payroll for 1,000 digital interns.

Companies need to start modeling "Cost of Goods Sold" (COGS) very differently. If your AI features have a linear cost scaling with usage, you cannot offer unlimited flat-rate pricing. This is why we are seeing the resurgence of usage-based pricing models (like Cursor's request-based plans or OpenAI's prepaid credits).

What the July 2026 Price Sheet Actually Looks Like

To model COGS correctly you need real numbers, not vibes. Here is a snapshot of frontier pricing per million tokens (input / output) as of the July 2026 launch window:

Model	Input / Output ($ per 1M tokens)	Typical Use Case
GPT-5.6 Sol	$5 / $30	Agentic coding, ultra reasoning mode
GPT-5.6 Terra	$2.50 / $15	General-purpose assistant tasks
GPT-5.6 Luna	$1 / $6	High-volume classification, chat
Claude Sonnet 5	$2 / $10 (intro, until Sept)	Planning, long-document analysis
Grok 4.5	$2 / $6	Cursor-native agentic coding (not in EU)

Two things have moved since that snapshot, and both matter if you are modelling a bill today. OpenAI shipped GPT-6 Astra on September 3, 2026 at $10/$50 per million tokens, and it is now the company's flagship and its most expensive published tier. OpenAI also announced reductions to the GPT-5.6 tiers after launch — 80% off Luna and 20% off Terra on July 30, 2026, and a further 20%-plus cut to Sol's API and credit pricing on August 21, 2026 — while its own pricing page still shows the pre-reduction figures in the table above. Treat those as list prices and pull the current numbers from the vendor before they land in a spreadsheet.

Notice the spread: routing a request to Sol's ultra mode instead of Luna can be a 5-to-30x cost multiplier for the same nominal "one API call." That spread is exactly where your margin lives or dies. Our GPT-5.6 versus Claude Sonnet 5 comparison covers which tier is actually strong at what; this article is about what each choice does to your income statement.

Output Tokens, Reasoning Tokens, and the Bill You Didn't Expect

Almost every team that gets surprised by an AI invoice made the same estimating mistake: they modelled input tokens. Look again at the table above and notice that output is priced at roughly six times input across the board. Your prompt might be ten thousand tokens of retrieved context, but the eight hundred tokens the model writes back can easily cost more than all of it. Any cost model that treats "tokens" as a single undifferentiated quantity will be wrong, usually in the expensive direction.

Reasoning tiers make this sharper. When a model is allowed to think before it answers, those intermediate thinking tokens are billed as output on most providers even though the user never sees them, so a short answer produced after a long deliberation is an expensive short answer. "Turn reasoning up" is therefore not a free quality lever — it moves a code path into a different pricing bracket. Three consequences: cap output length explicitly, ask for structured output rather than prose that says the same thing, and reserve extended reasoning for request types where you can demonstrate it changes the answer.

The Agent Loop Is Where Budgets Die

A single chat completion is cheap and easy to reason about. An agent is neither. The defining property of an agentic workflow is that it re-sends its accumulated state on every turn: the original instructions, the tool definitions, the transcript so far, and the results of every tool call it has already made. Ten turns into a session, each new step is paying to re-read everything that came before it. Cost does not grow linearly with the number of steps — it grows closer to quadratically with transcript length.

That is the arithmetic behind the earlier example of a research workflow burning five dollars in minutes. Nothing about it was pathological; it took thirty steps, each carrying a longer context than the last. Teams running autonomous agent workflows at sane cost do four things about it:

• Summarize and truncate the transcript at a fixed threshold. A compacted summary of the first twenty steps is almost always enough for step twenty-one.
• Trim tool output before it enters the context. A tool returning a whole HTML page charges you for the navigation bar on every subsequent turn.
• Cap the step budget per task and fail loudly. An agent with no iteration limit is an unbounded line item, and the failure mode is not a crash — it is a bill.
• Measure cost per resolved task, not cost per call. A cheap model needing three attempts costs more than an accurate one needing a single pass, and per-call metrics hide that entirely.

Four Numbers Worth Instrumenting

Most teams can tell you their total monthly model spend and almost nothing else — roughly as useful as knowing your AWS total without knowing which service it came from. Four metrics turn AI cost into something manageable:

Metric	What It Tells You	What To Do With It
Cost per resolved task	The true unit cost of the outcome you sell	Compare across model tiers before assuming the cheap tier is cheaper
Gross margin per active user	Whether your pricing survives your heaviest customers	Look at the top percentile, not the average — the average always looks fine
Cache hit rate	How much repeated thinking you are paying for twice	Below 20% on a high-traffic feature means the caching layer needs work
Escalation rate to the flagship tier	Whether your router is actually routing	If almost everything escalates, your classifier or your prompts are the problem

Attribute all four by feature, not just by vendor. AI spend behaves like infrastructure spend: it looks healthy right up until one code path becomes popular, and a single invoice total gives you no way to see that coming.

The Rise of Semantic Caching

The smartest companies in 2026 aren't just blindly calling APIs. They are building massive Semantic Caches. Why pay to think the same thought twice?

If user A asks "How do I center a div?" and user B asks "Center alignment css", the model shouldn't re-compute the answer. Semantic caching layers (using vector databases) intercept these requests and serve cached intelligence at near-zero cost. This is the only way to make AI unit economics work at scale.

How much a semantic cache actually saves you depends entirely on how repetitive your traffic is, which is why the only useful number is the one you measure. Instrument the hit rate from day one and watch it climb as the cache accumulates a few weeks of production traffic. On a high-traffic support or coding-assistant feature, where users ask overlapping questions all day, the hit rate is frequently large enough to be the difference between a profitable AI feature and a subsidized one.

The caveat is that semantic caching trades correctness for cost. Two questions that look similar in embedding space can require genuinely different answers, and serving a near-miss from cache is worse than serving nothing. Cache aggressively where answers are stable and general — documentation questions, definitions, common troubleshooting paths — and never cache anything that depends on the current user's data, account state, or the time of day. A cache key that omits the tenant is a data leak wearing a performance-optimization costume.

Prompt Caching: The Cheaper, Boring Cousin

Before building a vector-backed semantic cache, exhaust the mechanism the providers already give you. All the major labs now discount tokens that repeat a prefix you have already sent — the same long system prompt, the same tool definitions, the same retrieved document set. It requires no new infrastructure, only that you structure prompts so the stable parts come first and the variable parts come last. Teams that shuffle their context order on every request pay full price for content the provider was willing to discount.

The same logic applies to batch endpoints. Anything that does not need an answer in the next few seconds — nightly summarization, backfilling embeddings, scoring a document queue, running an eval suite — belongs on the asynchronous batch tier, which is materially cheaper than the interactive one. The most common form of AI overspending is not picking the wrong model; it is paying interactive prices for work nobody was waiting on.

Intelligence Arbitrage and the Model Router

There is also an arbitrage opportunity. You can route simple queries to cheaper, faster models (like locally-hosted Llama 5 or GPT-5.6 Luna) and only route complex "System 2" reasoning tasks to expensive frontier models (GPT-5.6 Sol or Claude Sonnet 5). Building this "Model Router" infrastructure is the secret sauce of profitable AI companies today.

A minimal router needs three things: a cheap classifier model that scores task complexity in under 50ms, a fallback ladder (Luna → Terra → Sol, or Sonnet 5 → Opus 5 → Fable 5.1) that escalates only on low-confidence outputs, and a logging pipeline that tracks cost-per-resolved-task rather than cost-per-call. Google's Flash tiers are a common bottom rung on that ladder — our Gemini 3 Pro deep dive walks through when Flash is genuinely sufficient and when the Pro tier is worth the multiplier. Startups that skip the router and hardcode a single flagship model for every request are, in effect, running their COGS on autopilot — and in a market where token prices swing every quarter, that is a solvable, and expensive, mistake.

When Buying Hardware Beats Buying Tokens

Every router eventually needs a bottom rung that isn't an invoice. Self-hosted inference converts a variable opex line into a fixed capex one, and past a certain volume that conversion is simply the correct financial decision. The crossover depends on three inputs: monthly token volume, how routine the tasks are, and how much engineering time you will spend running infrastructure. High-volume, low-variance work — classification, embeddings, summarization, first-pass completion — amortizes hardware quickly. Forty hard architecture questions a month never will, and you would be trading frontier capability for nothing.

We treat that decision as its own subject rather than repeating it here: digital sovereignty and local inference covers the runtimes and what quality you give up, and hardware for the agentic era covers the silicon and concurrency thresholds. What belongs in this article is only the accounting frame — treat self-hosting as a supplier negotiation, run the numbers on your real traffic mix rather than a benchmark, and remember you are substituting an invoice you can forecast for engineering hours you usually can't.

Pricing Your Product When Your Costs Move Underneath You

The hardest part of token economics isn't measuring your costs — it's pricing a product on top of costs that change every few months as labs release new tiers. A flat-rate SaaS plan priced around today's GPT-5.6 Terra costs will look either generous or unsustainable in six months, depending on which direction pricing moves. The teams handling this well build a buffer into their margin assumptions (treat today's token cost as a ceiling, not a floor) and revisit pricing quarterly rather than annually, matching the actual cadence at which the underlying model market moves.

Some products have moved to hybrid pricing entirely: a flat subscription fee that covers a generous but capped monthly token budget, with metered overage beyond that. This protects margin on power users while still feeling like simple, predictable SaaS pricing to the median customer who never comes close to the cap. It's more complex to build than a flat fee, but it's the only model that survives a 3x swing in your underlying COGS without a renegotiation of your entire pricing page.

Five Ways Teams Lose Money on Tokens

Reviewing AI spend for the first time tends to surface the same five leaks, none of which require a model change to fix:

• One flagship model hardcoded for every request — the largest overspend, often bigger than every other line combined.
• Retries invisible in the metrics. A failed parse or timeout plus a silent retry doubles that request's cost. Log attempt counts, not just successes.
• Retrieval that over-fetches. Twenty marginal chunks "just in case" costs money on every call and often answers worse than five good ones.
• A free tier with no per-user ceiling, which makes your marketing budget and your inference budget the same budget.
• Development and eval traffic billed against production. Tag it so you at least know what you are choosing.

Then review quarterly, roughly the cadence at which the labs ship: which features have the highest cost per resolved task, what share of requests hit the flagship tier unnecessarily, which provider prices have changed, and what margin looks like for your heaviest decile of users rather than your median one. A release like the Claude Sonnet 5 and Fable 5.1 launches should trigger an unscheduled review rather than waiting for the calendar.

The Takeaway for Founders

Treat your model bill the way a factory treats its raw materials line, not the way a SaaS company treats its AWS bill. Materials costs get modeled per unit, tracked obsessively, and re-negotiated the moment a cheaper supplier appears. Token costs deserve the same discipline — because unlike your AWS bill, which was basically a rounding error against your ARR, your token bill can now be a double-digit percentage of revenue if you don't actively manage it.

Almost none of this is exotic engineering. Order your prompts so caching works, cap your outputs, move patient work to batch, put a classifier in front of your flagship tier, and instrument cost per resolved task by feature. That is a couple of weeks of work and it routinely cuts a model bill by more than half. For where the routing and observability layers sit in a wider architecture, see our ultimate developer stack for 2026 and the best AI coding tools ranking.