AI & Technology

Your Engineering Metrics Were Built for a World That’s Gone

By Ted Julian, Founder and CEO, Flux

Earlier this year, an engineering leader told me about a developer who picked up an 8-point story and finished most of it with agents in about 30 minutes. His team’s question was simple: is that still an 8? No one on the team had a good answer, and neither does most of the industry. 

For most of the last decade, an engineering leader who wanted to calibrate their metrics had reliable outside reference points. You could benchmark against the annual DORA report or track what developers were asking on Stack Overflow. Internally, you had sprint velocity and standups where people told you what they’d accomplished. 

All your tools essentially depended on people describing their own work. Now AI coding tools and agents are doing a lot of that work, and people can’t describe something they didn’t do themselves. That creates a gap in the record, and most engineering metrics haven’t caught up to it. 

The Public Signals Are Disappearing 

RedMonk analyst Rachel Stephens laid this problem out in the firm’s September 2026 newsletter. As more developers take their questions straight to an LLM, public sources like Google Trends and Stack Overflow reveal less about what they’re working on or struggling with. At the same time, platforms are closing or limiting the APIs that made their data available.  

The data still exists, but it sits inside private companies that decide whether and how much to share. Stephens’ summary is that “telemetry is becoming privatized.” 

Stack Overflow illustrates how fast this shift happened. The site peaked at more than 200,000 questions a month in early 2014. In December 2025, developers posted 3,862, down 78% from just one year before, according to DevClass. Developers are still asking questions, but they ask an AI assistant now, and those answers are private. 

Shared industry benchmarks are getting harder to come by, too. As RedMonk noted, DORA announced in September that it won’t run its annual survey and report for 2026. For a lot of teams, DORA was the closest thing the industry had to a shared yardstick. Now they’re searching for a replacement.  

Even engineering leaders who track their code closely ask me what they should be measuring now, because the published frameworks weren’t built for how teams work today. 

Your Internal Instruments Have the Same Problem 

Evaporating public data might matter less if internal measurement holds up. It doesn’t, and it never fully did. 

Tickets capture what someone planned and what they said happened. Status updates summarize a sprint from one person’s point of view. All of it is narration, which worked well enough when a human wrote every line of code and could explain it. 

A ticket is only as accurate as whoever updated it last, and more often now, that’s an agent. According to Linear data in Gergely Orosz’s latest Pragmatic Engineer report, agents have created more issues than humans on that platform since July. 

Ticket data also fragments as teams customize their tools. One large engineering org I know has configured its tracker so differently from team to team that it effectively runs more than a dozen separate systems. So comparing teams across that tracker doesn’t work. 

Pull requests have flipped, too. GitHub data in the same report shows that agent-authored pull requests rose ninefold in eight months and outnumbered human-authored ones on GitHub in August. A developer can now generate a week’s worth of code in an afternoon, then spend two days fixing what the agent got wrong. 

But no one will file a ticket for each of those changes. And asking engineers to do that would bury them in admin work. Thus, the gap is increasing on both sides. The work gets faster and bigger while the record of it falls further behind. Yikes.  

The delivery metrics many teams fall back on have a blind spot of their own. They describe how work flows through your pipeline, but they say very little about what the work is, or whether this sprint shipped new capability or patched problems with last month’s deployed code. 

New workflows also change the numbers delivery metrics do capture. Some teams now write plans and specs as pull requests, which inflates lead time on the work that follows. Others see review cycles stretch as developers trade comments with AI review bots. A lead-time spike might signal trouble, or it might just mean the process changed. 

Rethinking Engineering Metrics 

I talk with engineering leaders every week. The ones adapting fastest to AI-powered code delivery are going back to a basic question: what’s the most reliable record of the work we did? 

In software, one record builds itself as the work happens: version control and pull request history. Every commit, review, revert, and refactor goes into your version control system even if no one records it during a sprint meeting or in a ticket. 

That record matters more as fewer people read the code itself. On Orosz’s podcast, Honeycomb cofounder and CTO Charity Majors argued that it’s a matter of when, not if, engineers ship code they’ve never read to production. If humans stop reading every line, the record of what changed becomes the primary way anyone knows what shipped. 

As public benchmarks disappear and self-reporting breaks down, your own version control history becomes the most accurate dataset you have. The sum of all of this suggests six things to focus on. 

1. Audit Where Each Number Comes From

Take every metric on your leadership dashboard and trace it back to its source. Does it come from the work itself, or from someone’s description of the work? Most teams find that a large share of what they report upward is secondhand. That data might still be useful, but you should know where it comes from and how it was derived. 

Adoption numbers deserve the same scrutiny. Many organizations track AI tool usage down to the decimal point and track almost nothing about what that usage delivered. 

2. Stop Asking People to Narrate Machine Work

If your visibility depends on engineers tagging every AI-assisted change, it will degrade over time. People will tag the first 50 changes carefully and nothing for the next 500. 

Measurement that requires extra human effort per change won’t survive agent-scale volume. To compensate, use the signals your systems already capture as part of your shipping processes. 

3. Classify the Work Before You Count It

Commit counts and lines of code were weak signals before AI, and they get weaker as agents write more of the code. What matters is the type of work each change represents. Feature work and rework on recently merged code tell very different stories. A team with rising throughput and rising rework is in a different place than a team with rising throughput and flat rework, even if their velocity charts look the same. 

Repositories add another layer. The same team can move quickly in one repo and slowly in another, and leaders often can’t tell whether the slow one holds genuinely complex functionality or years of accumulated debt. Each scenario calls for a different response. 

4. Build Your Own Baseline

With fewer public benchmarks, comparing yourself to an industry average gets harder and means less. Your own history is a better reference point.  

How did your rework share and work mix look before your teams adopted agents? How do they look now? Most organizations already have years of that history sitting in their repositories. 

Here’s what that comparison can surface. After an agent rollout, a team’s throughput might hold steady or even rise while the share of changes reworking code from the last few weeks climbs. The velocity chart says the team is fine. The work mix says it’s spending more and more time fixing what it just shipped. 

5. Expect Your Answer to Look Different from Everyone Else’s

Engineering practices are changing fast, and code review shows it clearly. According to Orosz’s report, some engineers are approving changes they haven’t read, and a growing share of pull requests get reviewed only by AI agents. 

Expect differences within organizations, too. Some developers still distrust AI output while others accept whatever it produces, and most probably land somewhere along that spectrum. Any measure you pick has to make sense across all of them. 

6. Know What Code Can’t Tell You

Code shows what changed, how much effort went into it, and how teams worked together. It doesn’t tell you whether customers wanted the feature, or why a team made a particular call. A lot of the friction between idea and production also lives in designs, specs, and handoffs that aren’t reflected in your version control history. 

Pair code-level signals with the human context only your people can give. The goal is to stop making that context carry the whole load. 

Back to That 8-Point Story 

Is it still an 8? Probably not, and the question itself shows the problem. Story points estimated how much effort a person expected the work to take. When agents do much of that work, the estimate stops telling you how much work actually happened. 

The code can show what came out of those 30 minutes: whether the change added new capability or reworked something the team shipped weeks earlier, and how much follow-up fixing it needed. That’s an answer a leader can take into a planning meeting. 

So start small. Pick one team and one quarter, sort its merged pull requests into new work and rework, and compare the result with what your current reports say about where effort went. Wherever the two disagree, you’ve found the first metric worth rebuilding. 

Related Articles

Back to top button