How We Measure AI Developer Impact (Without Just Counting Tokens)

Our development team ran six billion tokens through AI tools in the last thirty days. In the same window, AI wrote roughly 135,000 lines of code that shipped.

Those are real numbers from our own usage dashboard. They are also, on their own, almost meaningless.

Six billion is a big number. So is six hundred million. So is sixty billion. None of them tell you whether a single developer got their work done faster, whether the code was any good, or whether anyone on the team would be upset if you took the tools away tomorrow. A volume number measures activity, not impact. And once you start reporting it to anyone above you, it quietly becomes the thing people optimize for, which is exactly the wrong incentive.

This is the trap we set out to avoid when we instrumented AI usage across our team. This is what we built, what we chose to measure instead, and where we think the honest limits are.

The conversation that kicked this off was internal. One of our engineers, Damian, pushed back on the whole framing of "AI usage" as a metric. His point was blunt: knowing that someone burned 100,000 tokens versus 100 million tells you nothing about value delivered. It might mean they're doing harder work. It might mean they're stuck in a loop re-prompting a model that keeps misunderstanding them. The number is identical in shape and opposite in meaning.

What he actually wanted was deterministic measurement of who AI helps more. Not aggregate spend, but a per-person, per-workflow read on whether the tools are pulling their weight. That reframing is the entire article. The moment you accept that consumption is not value, you have to go and define value, and that's harder.

It's the same problem the DORA metrics solved for delivery performance a decade ago. Deployment frequency on its own is vanity; deployment frequency next to change failure rate and time to restore becomes a real signal because the metrics check each other. AI measurement needs the same discipline: no number stands alone.

We run a Grafana dashboard that pulls from our MCP Gateway, our log pipeline (Loki), and the editors and agents people use day to day, namely Cursor and our internal Cowork setup. It's organized into five areas, and the split matters more than any single chart:

The structure encodes a belief: utilization and impact are the easy, seductive numbers, and they are dangerous on their own. The bottom three areas exist to keep the top two honest.

"135,000 lines of AI-authored code shipped" is the kind of stat that ends up on a slide. We're wary of it, for a simple reason: lines of code is one of the oldest discredited productivity metrics in software. AI didn't fix that; it made it easier to generate volume that looks like progress.

So the Impact number only means anything sitting next to its guardrails. If AI-authored line count goes up while change failure rate or review churn goes up with it, that's not impact, it's a quality problem wearing impact's clothing. The guardrail layer is what lets you tell the difference between "AI is shipping working code" and "AI is shipping code we'll pay for in three sprints." If you build only one half of this, build the guardrails. The volume chart will take care of itself.

The part we're most opinionated about is sentiment, because it's the layer most teams skip and the one that's hardest to fake.

We built a small Slack app that periodically asks the team how the tools are working for them. People answer with a mix of emoji and free text: low friction, no forms. The responses get parsed automatically by a language model (we use Gemini 3.1 Flash for this) into themes and a sentiment read, so qualitative feedback at small scale doesn't die in a spreadsheet nobody opens.

The current numbers: an NPS of 100 and an 80% survey completion rate.

Here is where we have to be honest, because an NPS of 100 should make any thoughtful person suspicious. Our team is small: roughly eleven active AI users, five of them heavy daily users. At that size, a single frustrated person swings the score dramatically, and "100" mostly tells you that the people who chose to adopt these tools are happy with their choice. It is a real signal, but it's an early adopter signal, not proof that AI works for everyone. We report it with that caveat attached, every time. A perfect score you don't qualify is a marketing number, not a measurement.

What the sentiment layer is genuinely good for is direction. When a theme shows up repeatedly in the parsed responses — a tool that keeps breaking a workflow, a model that's great for one language and useless for another — that's where we spend our next round of attention. The score is the headline; the themes are the actual product.

Every metric in this post becomes useless the moment it becomes a target. That's Goodhart's Law, and AI tooling is unusually exposed to it because the metrics are so easy to move artificially.

Tell people their AI utilization is being tracked and watched, and utilization goes up, not because work improved, but because nobody wants to look like the laggard. Make AI-authored line count a goal, and you'll get more AI-authored lines and worse code. Even sentiment is gameable once people learn that a low score triggers a meeting.

We don't have a clean solution to this, and we distrust anyone who claims they do. What we do in practice: we keep the dashboard a team tool, not a performance review tool. The numbers inform where we invest in tooling and training; they don't rank individuals. The second they start showing up in someone's evaluation, the whole instrument stops telling you the truth, and you've spent a quarter building an elaborate way to lie to yourself.

You don't need our exact stack; the pattern is replicable with whatever observability tooling you already run. If we were standing this up from scratch again, the order would be:

The gateway-at-the-boundary approach is the same control point we describe in our work on MCP governance for enterprise: the layer that already sees every AI call is the natural place to measure it, govern it, and keep it auditable. If you're running AI tooling at any scale, you probably need that control plane anyway, and measurement is one of the things it pays for. It's the pattern we package as MCP Gateway Enterprise, though the measurement idea stands on its own regardless of what you build it on.

Our dashboard says six billion tokens, 135,000 lines, NPS 100. We don't lead with any of those numbers when someone asks whether AI is working for our team, because none of them answer the question on their own.

What we actually watch is the relationship between them: is the volume producing code that survives review, and do the people doing the work say it's helping? When those two move together, that's impact. When the volume moves and the others don't, that's a warning. The dashboard's real job isn't to produce an impressive figure; it's to make that distinction visible before it costs you anything.

If you're building something similar and want to compare notes on what to instrument (or you'd rather not wire up the gateway and observability layer yourself), get in touch. We'll bring the dashboard, the numbers we don't put on slides, and an honest read on where the measurement still falls short.