The AI Quantification Gap
Build the value dashboard first.
“The invoices are landing on the CFO’s desk, and nobody can answer: are we getting our money’s worth?”
That is what an AI Director at a 30,000-person tech services company told us last month. She has budget approval, executive support, and licenses procured. What she does not have is a single metric that proves the investment is actually paying off.
She is not alone. In another meeting, an AI lead at a 14,000-person enterprise analytics company shared a similar reality: “We burned 500,000 tokens in a week. Nobody can tell me what we got for it.”
This is the reality of enterprise AI adoption today. AI Directors can track their AI spending down to the cent, but almost none can quantify the value they got back - and bridging this measurement gap has become their entire job.
The Cost Visibility Illusion
Most organizations that claim to be tracking their AI ROI are actually only tracking their AI costs. And the costs are staggering.
A $5B public tech company we analyzed spends $260,000 per month on AI coding tool API costs alone, just for their developers. That is $3.1M per year, before salaries or infrastructure. Their FinOps team can tell you exactly what each model costs. They know which teams are driving the spend. What they cannot tell you is which of those dollars produced actual business value.
The cost side of the equation is highly visible, but it behaves more like cloud infrastructure than traditional SaaS. A seat license is a fixed cost, but API consumption is unpredictable. The average Claude Code cost per developer is $520 per month. One platform engineer we know of ran up $2,000 in two weeks on Cursor alone. Nobody had visibility until the invoice arrived.
When you treat AI tool spend like a fixed-cost SaaS subscription, you get surprised. AI tools are usage-based, highly variable, and invisible until the billing cycle ends. You have complete visibility into the expense, but zero visibility into the return.
Activity Is Not Impact
When pressed for ROI metrics, most engineering organizations retreat to activity metrics.
One major tech company we observed tracks eight distinct AI metrics: cost per model, tokens consumed, sessions per user, lines of code written, PRs created, commits, edit accept rates, and active time. They have sophisticated instrumentation using OpenTelemetry. They have per-user tracking across 500 developers.
But none of those metrics answer the CEO’s actual question: did this make us ship better products faster?
Activity metrics are a trap. Sessions, tokens, and lines of code tell you people are using AI tools. They do not tell you the tools are helping. The AI Director who presents “4.2 sessions per developer per day” to the board will get asked, “So what?” The AI Director who presents “teams with managed AI configs ship 30% more PRs with 50% fewer rollbacks” gets budget approval.
Token consumption is a cost metric, not a value metric. Every AI Director needs two dashboards side by side: what you spent, and what you got. Right now, 99% of companies have the first dashboard. Almost nobody has the second.
The Strategic Bet
The measurement gap becomes even more dangerous when you look at the total organizational investment.
Consider the financial analysis of one major tech company’s AI economics. Their total AI investment - API costs, engineering salaries, and infrastructure - sits at $20M to $25M per year. Their customer-facing AI products generate $11.5M in ARR.
They are underwater by at least $9M. But they cannot stop the investment. If they took the internal AI tooling away, the 20% of their engineers who are power users would simply leave.
Most companies’ AI investments are currently ROI-negative when you include all costs. That is not necessarily a failure. It might be a calculated strategic bet on long-term competitive advantage. But you have to be honest about it. Do not pretend your $10M AI investment is paying for itself when the math says otherwise.
Frame it as a bet, with clear milestones for when it should flip to positive ROI. The board can handle honest strategic bets. They cannot handle surprise losses justified by vanity metrics.
The AI Director As A Business Leader
The most effective AI Directors realize that their role is not fundamentally about technology. It is a measurement problem disguised as a technology problem.
Look at the Slack history of one departing AI Director at a major tech company. His daily routine was dominated by metrics. He checked the revenue dashboard every single morning. His communications were 80% about numbers: new accounts, churn, ARR growth, and retention rates. He was not debating agent architectures or model selection. He was a business leader justifying an investment.
The board does not care which model you use. They care whether the numbers go up. Before you build anything, figure out what you are going to measure. Most organizations invest in AI tools and then try to justify the spend retroactively. You have to flip that model. Define your success metrics before you roll out the tools, and the ROI conversation becomes entirely different.
What This Means For You Monday Morning
If you are leading AI adoption, your priority this week is not evaluating a new model or rolling out a new tool. It is closing the measurement gap.
Build the value dashboard first. Before you spend another cycle optimizing your FinOps tracking, build a dashboard that tracks outcomes. Stop measuring tokens consumed. Start measuring PRs shipped, bugs fixed, time saved on specific workflows, or reduction in support tickets.
Treat AI spend like AWS. Stop budgeting AI tools like SaaS licenses. Set up cost alerts, per-team budgets, and regular optimization reviews. You need to know when an engineer runs up a $2,000 bill before the invoice lands on the CFO’s desk.
Audit your activity metrics. Look at your current reporting. If you are reporting on lines of code or sessions per user, you are reporting activity. Replace at least one activity metric with an impact metric this week.
Frame the bet. If your AI initiative is currently ROI-negative, own it. Document the total cost, document the current value, and write down the specific milestones that will prove the investment is working. Present it as a strategic bet rather than a finalized success.
The companies that succeed with AI over the next year will not be the ones with the best models. They will be the ones that can prove their models are actually doing something useful.
We help engineering leaders navigate the realities of enterprise AI adoption. Subscribe to get our practitioner-first insights delivered every week.