AI tokens explained: Why your AI bill keeps growing and how to fix it

AI bills have a way of surprising people.

A pilot that cost almost nothing moves into production. Agents start planning, retrieving information, calling tools, and checking their own work. Within a quarter, the token line on the invoice can be growing faster than anyone forecast, and nobody can quite say where it went.

The stakes are real:

• 7% of organizations already spend more than $2,000 per developer per month on AI tokens (Gartner, AI Tokenomics: When to Cut AI Tokens, and When to Double Down, July 2026)
• 53% of infrastructure and operations heads name the cost of delivering AI infrastructure as a major challenge (Gartner, Cheaper Tokens, Higher AI Costs, September 2026)
• Uber spent its entire 2026 AI budget in the first four months of the year, with its COO saying internal AI costs were getting harder to justify (CBC News)

And the economics are getting more complicated as AI moves from simple prompts to agentic workflows. A chatbot may make one model call to answer a question. An agent can plan, retrieve, call tools, evaluate its own work, retry, and delegate tasks to other agents. Each additional step can introduce more model calls, context, tool inputs, outputs, and reasoning.

The instinct is to treat this as a model problem: find a cheaper model, negotiate a better rate, or wait for prices to fall. Those measures can help, but token efficiency is an architecture problem before it is a model problem.

Most waste does not come from the hardest tasks. It comes from the design around them: when a model is invoked, what it is asked to reason about, how much context it carries, how many tools it calls, and how many times the workflow repeats the same work.

The goal is not simply to spend less on AI. It is to spend intelligence where it creates value, cut consumption where it does not, and make every token work harder.

What are tokens in AI, and how do they work?

Tokens in AI are the small units of text, code, image or audio that a large language model reads and writes. A token can be a whole word, part of a word or a punctuation mark; as a rough rule, 100 tokens is about 75 English words. Before a model sees your prompt, a step called tokenization splits it into these units, and the model then generates its answer one token at a time.

That makes tokens both the unit of AI work and the unit of AI cost. Most LLM services bill per token, so every prompt, retrieved document, tool call and response carries a price tag, and token consumption becomes the meter running behind every AI workflow.

Not all tokens cost the same. That asymmetry is where token economics starts:

AI agent tokens change the math. To see why agents consume more tokens, compare them with a typical chatbot. A chatbot usually makes a single model call to answer a question. An agent makes many calls as it plans, acts, observes and evaluates the result, often across sub-agents that each reason, retrieve data and call tools. Gartner's illustration: a 500-token answer becomes a 10,000-token multistage run, and agentic tasks need 5 to 30 times more tokens than a standard chatbot exchange (Gartner, Cost-Per-Result Tokenization, May 2026).

In agentic tasks analyzed by Gartner, input made up about 54% of consumption, output 24% and hidden reasoning 22%, because agents keep re-sending data, tool schemas, instructions and a growing history. Vector databases, monitoring and security add another 20 to 40% on top.

This produces a paradox worth naming: cheaper tokens, higher AI costs. Epoch AI research finds the same level of AI performance now costs about 50x less than it did a year ago, yet total spend keeps rising because cheaper AI invites more workloads, more attempts and deeper reasoning. Cheaper AI does not make efficiency less important. It makes it more important to know where extra intelligence actually creates value.

Why are AI token costs so high: 6 hidden drivers of token consumption

Most teams start by trimming prompts. That helps at the margins, but in production systems the prompt is rarely the biggest line item. The waste usually happens before the model sees the prompt and after it starts working.

• Over-retrieval. Eight long documents pulled when two short passages hold the answer, or a 50-page handbook re-read to answer a simple leave question (Oracle).
• Context inflation. Workflows carrying information from earlier steps long after it stopped contributing to the decision, plus every tool schema re-sent on every turn.
• Recursive agents. Agents spawning subtasks and sub-agents, with every handoff adding new prompts and more context.
• The wrong intelligence for the job. Frontier reasoning models doing routine classification that deterministic code could handle at a fraction of the cost.
• Retries without a definition of success. Loops that keep firing to chase marginal gains instead of better business outcomes.
• Verbose output. Paragraphs of explanation when the downstream system reads one field, paid at the highest token rate.

None of these looks expensive in isolation. A few extra prompts, another reasoning step or a larger context window rarely changes the cost of a single run. Across millions of executions, they compound into consumption no model upgrade can offset. Bigger context windows do not fix this; they just make the cost less visible.

How to reduce AI token costs: 7 strategies that work at scale

The teams getting this right treat token efficiency as system design, not prompt editing. The strategies below are ordered by impact, not by how quickly you can implement them. For fast wins, Gartner advises starting with context engineering, caching, batch processing and spend observability, then investing in model routing and fine-tuning as longer-term bets.

1. Match the model and reasoning level to the task

Not every step deserves the same amount of intelligence. Think of it as a pyramid of expertise: frontier reasoning for design, ambiguity and judgment; smaller models for clearly scoped execution; deterministic code and reusable tools where the outcome is already known. Treat reasoning effort as a configurable resource, not a fixed cost. The savings are large: Gartner notes a commodity model can cost 1 to 2% of a frontier model while reaching roughly 80% of its benchmark performance, and reports contextual routing savings of 37 to 46%.

2. Improve retrieval precision in RAG

Retrieval decides what the model must read before it can think, so it is usually the highest-leverage fix. Optimize for precision with reranking and filters on metadata, permissions and recency. Pass passages, not documents, and deduplicate before anything reaches the prompt.

3. Use context engineering to keep prompts lean

Cap or summarize history, load tools on demand, and package recurring instructions as reusable skills or rules files instead of repeating them in every prompt (Google Cloud).

4. Use prompt caching and batch processing

Keep prompt prefixes stable so they can be cached; Gartner puts savings at 80 to 90% on repeated segments. Send work with no real-time need, such as overnight document processing, through batch APIs.

5. Limit output tokens

Specify the output shape, set max-token limits on every step, and ask for structured fields when a machine is the reader.

6. Capture reasoning once, reuse it every run

Reasoning that is not captured has to be paid for again every time the system runs. Persist it as a durable artifact (a specification, a graph, a compiled skill or plain deterministic code), the way a senior engineer's judgment lives on as an interface everyone downstream inherits. Watch agent topology too: a 2026 study found that consolidating three or four agents into a single call cut token use by 53.7% and latency by 49.5% with broadly comparable accuracy, mostly by not passing the same context back and forth. Consolidation has limits, so pair it with good routing.

7. Architect once, inherit everywhere

Evaluation, orchestration, guardrails, context management and auditability are not unique to any one workflow. Build them once into a shared harness, so every new agent inherits them and teams spend their effort on business logic. That harness is also where autonomy gets its limits: step caps and stop conditions, event-driven wakeups instead of polling, and cheap checks before expensive ones.

Tokenmaxxing vs. token minimization: How much should you spend on AI tokens?

Optimization gets you the same quality for less, or more quality for the same spend. It does not answer the harder question usage-based pricing creates: how much more variable spend does extra quality justify?

Gartner frames the answer as a sweet spot between two traps. Token minimization (banning tools, cutting context and reasoning across the board) makes each task cheaper but shows up later as production errors and extra human review. Tokenmaxxing (leaderboards celebrating the heaviest users) mistakes consumption for productivity. Value maxxing sits between them: spend more where evals prove it pays, less where they do not.

For repeatable, high-spend tasks, find the sweet spot by experiment. Start with the cheapest model and lowest reasoning setting, step up one level at a time, and measure against evals and business KPIs until gains flatten. Repeat when new models ship.

Most usage, though, is ad hoc, so shape everyday choices instead. Gartner recommends five nudges:

1. Budgets as bounds. Wide enough to try frontier models, with soft limits, an easy way to ask for more and a ring-fenced slice for experimentation. Teams that consistently pick the right model and avoid needless reasoning should earn more compute, not less.
2. Efficient defaults. Default to commodity models and low reasoning; make expensive settings opt-in.
3. Friction that scales with cost. A confirmation step before frontier models, and approvals that tighten as spend climbs.
4. Visible spend at the point of use. Current and forecast spend, split by commodity and frontier models, where people work.
5. Rewards for value per token. Recognize efficient, high-value use and reusable optimizations, with clear accountability for spend.

How to measure AI token efficiency: The metrics that matter

Total token count tells you what you spent, not what you got. Cost per request is not much better, since one agentic task can span dozens of calls. Two headline metrics do most of the work:

• Cost per successful unit of work. The fully loaded cost of one resolved ticket or processed invoice. Count failed runs and retries in the cost but only successes in the denominator, and report it with completion rate and human review (Gartner, Cheaper Tokens, Higher AI Costs).
• Token Efficiency Ratio (TER). Business value delivered per million tokens. Gartner predicts that by 2028, enterprises with formal token efficiency governance will cut AI inference overspend by 30% versus peers.

The first prices the outcome; the second values it. Diagnostic metrics explain why they move:

Set alert thresholds against your own baselines. Gartner suggests counting untagged shadow AI spend as zero value in the enterprise TER, so business units have a reason to bring it under governance.

How to govern AI token spend across the enterprise

Most organizations track tokens the way early cloud adopters tracked compute hours: in aggregate, with no link to outcomes. Closing that gap takes FinOps habits:

1. Map workflows end to end, including retrieval, tool calls and self-evaluation, not just the visible prompt.
2. Tie each use case to a value metric. If $50 of tokens deflects 200 level-one tickets, the ROI is obvious (Oracle).
3. Budget and forecast in scenarios. Hard caps for pilots, headroom for production, and best, expected and high cases, because a 50-person pilot does not predict a 5,000-person rollout.
4. Meter independently. Route calls through a gateway that logs tokens, model and use case, and tag every call so cost is charged to the business that drives it.
5. End every review with a decision. Rebaseline when models, providers or agent tools change, and close each review with a dated decision, a named owner and the next trigger. For agents, set per-run limits on tokens, steps and tool calls with an agreed escalation path.

Ownership matters as much as tooling. Gartner separates AI unit economics into three layers:

How Kore.ai helps enterprises manage AI tokens

Deterministic where it must be, reasoning where it pays: With ABL™, deterministic and LLM-driven reasoning steps can run within the same runtime. Deterministic steps execute without LLM involvement, while reasoning steps invoke agentic reasoning when judgment is required.

1. Bounded context by design: Agents use memory management, context compaction and targeted retrieval to control the context available to each execution, bringing in relevant information without unnecessarily expanding the context window.
2. Intelligent model routing: Route different operations to appropriate model tiers based on complexity and requirements. Smaller, faster models can handle lightweight tasks, while more capable reasoning models can be reserved for complex work, helping reduce unnecessary token and compute consumption.
3. Model-agnostic, with your models or ours: Use commercial LLMs, open-source models, self-hosted models or custom fine-tuned models through the platform's provider-agnostic model layer. Model selection can be configured by agent, project, tenant and operation. Kore.ai also provides XO GPT, our proprietary small language model.‍
4. Evals before production: Agent Evals test agent behavior across scenarios, personas and evaluators, providing measurable quality signals before deployment and after changes, supporting continuous optimization.‍
5. Cost tied to performance: Observability captures execution traces, token usage, latency and estimated cost, while Insights connects these metrics to agent performance, usage and operational trends.‍
6. Agents that improve over time: AutoLoop™ creates a continuous closed-loop engineering system for agents. Before deployment, it uses requirements and evaluation data to test, diagnose, improve and validate agents against the required bar. After deployment, production interactions and operational signals trigger the same cycle, with AutoLoop diagnosing issues, improving the relevant prompt, workflow or logic, and revalidating the change. Its five-layer validation architecture combines deterministic checks with model-based reasoning to keep continuous optimization efficient at enterprise scale.

The bottom line: Efficiency is how AI scales

Token efficiency is won in system design, not prompt wording. The payoff is bigger than a lower bill: a system that sends less noise to the model is faster, better grounded and easier to trust, which is what lets AI scale from pilots to thousands of users.

Start this week:

• Find the 20% of tasks driving 80% of token spend
• Measure cost per successful unit and TER for each
• Route every step to the lowest intelligence that holds the outcome
• Put spend next to outcomes on one dashboard with finance

Spend intelligence where it earns its keep. Everywhere else, spend less of it.

What are AI tokens?

AI tokens are the units of text or data a model processes, roughly three-quarters of a word each in English. They are also how most AI services are priced, which is why token efficiency (the business value delivered per token) has become a core enterprise metric.

Why are AI costs rising when token prices are falling?

Agents use far more tokens per task. Reasoning, multistep loops, retries and growing context push total consumption up faster than per-token prices fall, a classic Jevons paradox.

What are AI agent tokens?

AI agent tokens are all the tokens an agent consumes to complete a task: planning, reasoning, retrieval, tool calls, retries and the growing conversation history it re-reads at each step. That is why one agentic task can cost 5 to 30 times more than a simple chatbot answer.