Written by: Mark Hull, Co-Founder and CEO, Exceeds AI
Key Takeaways for AI-Aware Engineering Leaders
-
Traditional DORA metrics and metadata platforms cannot distinguish AI-generated code from human contributions or track long-term quality outcomes.
-
A three-tier scorecard that combines DORA baselines, DevEx signals, and code-level AI impact metrics measures real AI ROI and exposes hidden technical debt.
-
AI-touched code often shows higher rework rates and increased incident risk within 30 days, even when throughput metrics improve.
-
Tool-agnostic commit-level analysis supports accurate outcome comparison across Cursor, Claude Code, GitHub Copilot, and other AI coding tools.
-
Exceeds AI delivers this complete three-tier framework with commit-level visibility and 30-day outcome tracking so you can see AI’s impact in your own repos.
Measuring AI Technical Debt Where DORA Cannot See
Metadata tools surface PR cycle time and merge status, but they stop at the moment of merge. They cannot tell you what happens to that merged code over the following month. A 2026 empirical study by Liu et al. at Singapore Management University analyzed verified AI-authored commits across GitHub repositories and found that some of the tracked AI-introduced issues still survive at HEAD. Security issues are the most likely to persist, followed by runtime bugs and code smells.
This persistence matters because the timeline of accumulation is predictable, and metadata tools miss it entirely. Warning signs appear around Day 30: duplicated logic, missing error handling, and silently introduced dependencies. Over time, teams spend more sprint capacity on bugs that trace back to the original AI-generated implementation.
DORA’s 2025 report cautions against using narrow output metrics as a proxy for productivity. Change Failure Rate only captures production-visible incidents. AI-generated code is remarkably good at passing tests, so Change Failure Rate can hold steady or improve even as code churn doubles.
Closing this gap requires AI versus human diff mapping at the commit level combined with 30-day longitudinal tracking of incident rates, rework patterns, and follow-on edits for AI-touched code. Exceeds AI tracks exactly these signals and flags specific commits that carry elevated rework risk before they turn into a production crisis.
Manager Takeaway: Pull the 30-day rework rate for AI-touched PRs on your highest-velocity team. If AI-touched code is being revised at more than 1.5× the rate of human-written code, schedule a targeted review session focused on prompt discipline and architectural alignment, not general AI usage reduction.
Outcome Delta: The Core Metric That Shows AI Is Helping
The most diagnostic metric is the outcome delta between AI-touched and human-only PRs across three dimensions: rework rate, cycle time, and 30-day incident rate. When all three move in the right direction together, AI is helping. When rework rate climbs while cycle time drops, the team is shipping faster into future debt.
A Faros AI telemetry study of over 10,000 developers across 1,255 teams found that high-AI-adoption teams completed more tasks and merged 98% more pull requests, yet PR review time increased 91%, average PR size grew, and bug counts rose, with no measurable improvement in organizational DORA metrics. Throughput gains masked quality degradation that only became visible at the code level.
Metadata platforms report the 98% PR increase and stop there. Exceeds AI maps which of those PRs were AI-touched, tracks their rework and incident rates over 30 days, and surfaces the delta so leaders can separate genuine productivity gains from accelerated debt accumulation.
DORA’s 2025 analysis found that higher AI adoption is associated with increases in both software delivery throughput and software delivery instability simultaneously. Speed and quality must be tracked together, not assumed to move in parallel.
Manager Takeaway: For any team reporting strong AI adoption, compare the 30-day incident rate of AI-touched PRs against human-only PRs from the same period. A ratio above 1.5× is a coaching trigger. Work with engineers on review depth for AI-generated diffs, not on adoption volume.
Comparing Cursor, Copilot, and Claude Code on Real Outcomes
JetBrains’ January 2026 survey of 10,000+ developers found 18% adoption of Claude Code as of January 2026, up 6× from approximately 3% in April–June 2025. Engineering teams in 2026 rarely run a single AI tool. They run three or four simultaneously, and engineers self-select tools by task type.
Most analytics platforms were built for the single-tool era. They rely on telemetry from one vendor and go dark when engineers switch tools. Leaders see aggregate adoption numbers but cannot see whether Cursor or Copilot drives better outcomes for feature work, or whether Claude Code’s architectural refactors still look healthy at 30 days.
Teams resolve this blindspot with tool-agnostic AI detection. That approach identifies AI-generated code through code patterns, commit message analysis, and optional telemetry integration, regardless of which tool produced it. Exceeds AI applies this across the full toolchain so leaders can compare outcomes side by side across Cursor, Claude Code, GitHub Copilot, Windsurf, and others.
The practical difference between metadata and code-level signals is significant. A metadata platform sees that PR #1523 merged in four hours with 847 lines changed. A code-level platform sees that 623 of those lines were Cursor-generated, that those lines required one additional review iteration compared with human lines, and that the AI-touched module had zero incidents at 30 days. The first view informs a dashboard. The second view informs a tool strategy decision.
Zapier’s chief AI transformation officer describes the goal as identifying whether elevated AI usage represents a “golden pattern” to multiply across peers or an “anti-pattern” to coach away from. That determination requires tool-level outcome data at the commit layer.
Manager Takeaway: Identify your top three AI power users by tool. Pull their 30-day rework rates by tool type. If one tool consistently produces lower rework on complex PRs, treat that workflow as a coaching asset and share it in the next team retrospective.
Connect your repo to compare tool outcomes in a free pilot
The Three-Tier Scorecard Applied to Boards and Teams
The three-tier framework connects upward to board reporting and downward to manager coaching plays.
For board reporting: Tier 1 DORA metrics establish the operational baseline. Tier 3 AI Impact metrics answer the ROI question directly: percentage of commits that are AI-touched, outcome delta on rework and incidents, and tool-level attribution. A VP Engineering can present a clear statement such as, “AI-touched PRs on Team A show 18% lower rework rates and equivalent 30-day incident rates versus human-only PRs, which validates continued Cursor investment for feature development.”
For manager coaching: Tier 2 DevEx signals identify where adoption is patchy. At maturity, the gap between top and bottom teams within the same organization is typically wider than the gap between organizations, which makes internal patterns more important than external benchmarks. This internal variance is where Tier 3 signals become critical. They explain which engineers generate high-rework AI code, which subsystems accumulate AI debt fastest, and which tool-workflow combinations produce durable output.
Organizations that measure across a full multi-pillar AI impact framework achieve significantly higher ROI than teams that measure only activity. The scorecard does not add unnecessary complexity. It represents the minimum viable measurement set for proving that AI spend is working and for managing the risk that it is not.
Exceeds AI delivers all three tiers from a single GitHub authorization. First insights arrive within one hour, and complete historical analysis is available within four hours. Setup avoids months of onboarding and avoids per-seat pricing that penalizes team growth.
FAQ: Making AI Productivity Metrics Work in Practice
Why do traditional DORA metrics fall short for AI code?
DORA metrics, including Deployment Frequency, Lead Time, Change Failure Rate, and Mean Time to Recovery, were designed when humans wrote all production code. When AI generates a significant share of committed code, the assumptions underneath these metrics break down. Deployment Frequency can rise because AI accelerates output volume, not because delivery processes improved. Change Failure Rate can hold steady because AI-generated code passes automated tests while introducing architectural fragility that surfaces weeks later as rework rather than production incidents.
DORA metrics remain useful for operational stability benchmarking. They cannot distinguish AI-generated from human-written contributions, cannot attribute outcomes to specific tools, and cannot detect the slow accumulation of technical debt that becomes visible 30 to 90 days after merge. A complete measurement framework pairs DORA with code-level AI impact signals.
How long does it take to see 30-day AI outcome signals?
Meaningful 30-day outcome data usually requires about six to eight weeks of instrumented observation after connecting your repository. The first two to four weeks establish the baseline distribution of AI-touched versus human-only PRs. Weeks five through eight begin to surface rework patterns and early incident correlations for the initial cohort of AI-touched code.
By week eight, most teams have statistically meaningful deltas across rework rate, cycle time, and incident rate for their primary AI tools. Exceeds AI begins surfacing adoption and immediate outcome signals within the first hour of setup. Longitudinal tracking then activates automatically as the 30-day window accumulates for each merged PR.
Can metadata platforms detect AI technical debt?
Metadata platforms cannot detect AI technical debt. They track PR cycle time, merge status, commit volume, and review latency, but they have no visibility into the content of the code itself. AI technical debt, such as duplicated logic, missing error handling, security vulnerabilities, and architectural inconsistencies introduced by AI-generated code, remains invisible because it does not appear as a slower PR or a failed deployment at merge time.
This debt surfaces as rework, follow-on edits, and production incidents 30 to 90 days later. Detecting it requires repo access to analyze code diffs, map AI-generated lines, and track those specific lines longitudinally against incident and rework data. That gap is a category-level limitation, not a missing feature.
What setup is required for commit-level visibility?
Exceeds AI requires read-only GitHub or GitLab authorization, which usually takes about five minutes to configure. Repository scoping and access controls take roughly fifteen additional minutes. The platform begins processing commit and PR history immediately after authorization, with first insights available within one hour and complete historical analysis, including up to twelve months of prior commits, within four hours.
No agents, code instrumentation, or lengthy professional services engagement are required. For organizations with elevated security requirements, an in-SCM deployment option keeps all analysis within the customer’s own infrastructure with no external data transfer.
Conclusion: Turning AI Dashboards into Proof and Action
The three-tier scorecard of DORA baseline, DevEx signals, and Code-Level AI Impact gives engineering leaders a practical framework. It proves AI ROI to a board and gives managers specific coaching plays instead of another dashboard to interpret. As DORA’s research confirms, measuring speed without measuring quality is not a conservative approach. It is an incomplete approach that accumulates hidden risk.
As the Liu et al. study showed, these issues persist for months without active longitudinal tracking, and more than 15% of AI-authored commits introduce at least one quality problem. Metadata platforms cannot see these issues. Commit-level AI observability can.
Exceeds AI is the only platform that closes this measurement gap with tool-agnostic AI diff mapping, 30-day longitudinal outcome tracking, multi-tool comparison across Cursor, Claude Code, and GitHub Copilot, and prescriptive manager coaching surfaces. Setup delivers insights in hours, not months, at outcome-based pricing that does not charge per engineer seat.
See the three-tier scorecard in your own repos and start your free pilot