How to Measure AI Technical Debt with 8 Provenance Metrics
Written by: Mark Hull, Co-Founder and CEO, Exceeds AI | Last updated: July 11, 2026
Key Takeaways
AI-generated code often passes review but fails in production 30–90 days later, creating hidden technical debt that traditional metadata tools cannot detect.
Line-level provenance via commit-level attestation is required to track which lines are AI-generated, which tool produced them, and their long-term outcomes.
The eight-metric framework measures AI technical debt through attribution coverage, rework rates, defect density, churn, interaction-mode effectiveness, token efficiency, maintainability drift, and security issues.
Each metric depends on durable, line-level data that travels with the code, enabling longitudinal tracking and governance that metadata-only platforms cannot provide.
Engineering teams can connect their repositories to Exceeds AI to capture this provenance automatically and start measuring AI ROI with a free pilot.
Why Traditional Engineering Tools Miss AI Technical Debt
Vella and Blincoe's 2026 longitudinal study of 158 professional software engineers documented a creation-to-verification shift. Developers now spend less time writing and designing code and more time reviewing and verifying it. That shift is the operational signature of AI technical debt accumulating in real time. The AI tool handles creation, and the human handles verification, yet verification tools were not built for AI-origin code.
The 30–90 day failure window is where this gap becomes costly. Teams using AI tools saw PR volume rise 98% while incidents per PR rose 243% and change failure rates increased. Defects in AI-generated code often pass initial review and testing but surface in production weeks after merge. Industry data shows AI-generated code turns over at 1.8–2.5× the rate of human-written code. That signal stays invisible in any tool that lacks line-level provenance.
This invisibility stems from how current monitoring tools are architected. Metadata-only platforms such as Jellyfish, LinearB, and Swarmia see PR cycle time and merge status. They cannot see which lines are AI-generated, which tool produced them, or whether those lines required follow-on edits, caused incidents, or degraded maintainability scores over the following quarter. Provenance-based metrics close that gap.
Exceeds AI Repo Leaderboard shows top contributing engineers with trends for AI lift and quality
The 8-Metric Framework for Measuring AI Technical Debt
Step 1 — AI Attribution Coverage
Definition: The percentage of lines, commits, and PRs for which AI authorship can be confidently attributed to a specific tool, model, and session.
Why it matters: Every downstream metric in this framework depends on knowing which code is AI-generated. Heuristic and watermark-based detection tops out around 20–25% accuracy. Client-level capture via Exceeds Ink delivers authoritative attribution by observing what actually happens on the engineer's machine at commit time.
Calculation: Exceeds Ink writes a structured Git Note at refs/notes/exceeds-ink for every commit, creating a durable provenance record. Some lines cannot be confidently attributed to a specific source. The system records those as unknown_lines rather than rolling them into "human" or "AI", which preserves metric integrity. With that foundation in place, Attribution Coverage = (confidently attributed lines) ÷ (total changed lines).
Healthy signal: Coverage above 85% across active repositories. Unhealthy signal: Coverage below 60%, which indicates gaps in Ink deployment or tool adapter configuration that will corrupt every downstream metric.
Data-quality watch-out: Treat Attribution Coverage as a prerequisite metric. Do not interpret rework rates or defect density figures until Coverage is validated. Low coverage produces false negatives that make AI debt appear smaller than it is.
Step 2 — AI Rework Rate (30/90-Day Turnover)
Definition: The percentage of AI-attributed lines that are reverted, deleted, or substantially rewritten within 30 or 90 days of merge, tracked separately from human-written code.
Why it matters: Industry benchmarks define healthy AI code turnover as below 12–15% at 30 days, with rates above 25% considered critical. Rework above that range signals unstable AI output and weak review practices.
Calculation: For each AI-attributed line, identified via the Git Note, track whether that line appears in a subsequent revert, deletion, or substantial edit commit within the measurement window. AI Rework Rate = (AI-attributed lines reworked within window) ÷ (total AI-attributed lines merged).
Healthy signal: AI-to-human turnover ratio below 1.5×. Unhealthy signal: Ratio above 2.0×, which typically indicates issues with prompt quality or review processes rather than the AI tools themselves.
Step 3 — AI Defect Density and Incident Correlation
Definition: The number of confirmed defects and production incidents attributable to AI-generated lines, normalized per 1,000 lines of AI-attributed code, tracked over time.
Calculation: Link incident tickets, via JIRA or Linear integration, to the commits that introduced the affected code. Cross-reference commit hashes against the provenance record to determine AI attribution. AI Defect Density = (incidents linked to AI-attributed commits) ÷ (AI-attributed KLOC).
Healthy signal: AI defect density at or below the human-written baseline. Unhealthy signal: AI defect density exceeding the human baseline by more than 20%, or a rising trend over consecutive 30-day windows.
Governance threshold tip: Set a policy alert when AI defect density in security-sensitive paths such as auth, payments, or data access exceeds 1.5× the human baseline. Exceeds Ink's structured JSON attestation makes this expressible as a policy gate in CI/CD pipelines.
Step 4 — Longitudinal Code Churn by Origin
Definition: Code churn rate segmented by AI versus human origin, measured at 30, 60, and 90 days post-merge.
Exceeds AI Impact Report shows AI code contributions, productivity lift, and AI code quality
Calculation: Using the attribution data, segment all changed lines by origin at merge time. At each measurement window, calculate the proportion of those lines that appear in subsequent modification commits. Compare AI-origin churn rate to human-origin churn rate.
Healthy signal: AI-origin churn below 15% at 30 days. Unhealthy signal: AI-origin churn above 25% at 30 days, or a widening gap between AI and human churn trajectories across the 90-day window.
Step 5 — Interaction-Mode Effectiveness
Definition: Outcome quality, including rework rate, defect density, and review iterations, segmented by the interaction mode in which AI-generated code was produced: plan, ask, agent, edit, or headless.
Why it matters: Not all AI-generated code carries equal risk. Agent-mode commits without a plan phase produce different quality profiles than ask-mode or edit-mode contributions. Exceeds Ink captures interaction mode per session, a signal no metadata tool publishes, which enables mode-calibrated governance.
Calculation: Group AI-attributed commits by the interaction_mode field in the Git Note. For each mode, calculate 30-day rework rate, defect density, and review iteration count. Rank modes by outcome quality.
Healthy signal: Plan-mode and ask-mode commits showing lower rework rates than agent-mode commits, with the gap narrowing as teams adopt coaching guidance. Unhealthy signal: Agent-mode commits comprising more than 60% of AI-attributed lines with rework rates above 20%, the pattern Exceeds AI identified in a 300-engineer customer deployment where spiky agent-mode commits without plan phases were driving elevated rework.
Step 6 — Token-Spend Efficiency per Commit
Definition: The ratio of durable shipped output, measured as AI-attributed lines surviving 30 days, to token cost, tracked per tool, model, and team.
Calculation: Token-Spend Efficiency = (AI-attributed lines surviving 30 days) ÷ (total tokens consumed in producing those lines). Compare across tools, models, and interaction modes to identify high-efficiency patterns worth scaling.
Healthy signal: Efficiency improving quarter-over-quarter as teams adopt better prompting practices. Unhealthy signal: Token spend rising faster than durable output, or efficiency declining in agent-mode sessions, which often signals prompt-quality problems.
Common measurement mistake: Treating raw token count as a proxy for productivity. A session that consumes 50,000 tokens and produces 200 durable lines is less efficient than one consuming 10,000 tokens and producing 180 durable lines. Always normalize to surviving output. While token efficiency addresses the financial dimension of AI technical debt, the structural health of the codebase itself requires separate measurement.
Actionable insights to improve AI impact in a team.
Step 7 — Maintainability and Complexity Drift
Definition: The change in cyclomatic complexity and cognitive complexity scores for files with high AI attribution, measured at 30-day intervals.
Calculation: Run complexity analysis, using SonarQube or an equivalent tool, on files flagged as high-AI-attribution by Exceeds Ink. Track complexity scores at commit time and at 30-day intervals. Complexity Drift = (complexity score at 90 days) − (complexity score at merge).
Healthy signal: Complexity scores stable or declining in AI-attributed files, which indicates refactoring is keeping pace with generation. Unhealthy signal: Complexity rising more than 15% over 90 days in AI-attributed files, the pattern GitClear's longitudinal dataset associates with a collapse in refactoring activity from 25% to under 10% of all changed lines.
Step 8 — Security and Compliance Issue Introduction Rate
Definition: The rate at which AI-attributed commits introduce security vulnerabilities or compliance violations, expressed as issues per 100 AI-attributed commits, segmented by tool and severity.
Calculation: Using the same commit-to-issue linking approach established for defect tracking, integrate SAST output from tools such as Semgrep, Snyk, or GitHub Advanced Security with the attribution data. For each security finding, resolve the introducing commit and cross-reference against Git Notes to determine AI attribution. Security Issue Introduction Rate = (security findings in AI-attributed commits) ÷ (total AI-attributed commits) × 100.
Healthy signal: AI security issue introduction rate at or below the human-written baseline, with no critical-severity findings in AI-attributed commits touching auth, payments, or data access paths. Unhealthy signal: AI security issue rate exceeding the human baseline by more than 30%, or any pattern of hallucinated package references, since studies have found that a notable percentage of AI-recommended packages are hallucinated.
Pro tip: Use Exceeds Ink's structured JSON attestation to configure policy gates that require additional review on commits where agent-mode AI attribution exceeds a defined threshold in security-sensitive file paths. Teams can express this directly in CI/CD without custom tooling.
How to Validate That Your Metrics Are Trustworthy
A measurement framework only works when its validation checkpoints confirm that the data is reliable. The following observable indicators show that the framework is producing trustworthy signals.
Exceeds AI Impact Report with PR and commit-level insights
Consistent attribution across tools: AI Attribution Coverage, Metric 1, is above 85% across all active repositories and all deployed AI tools. Gaps in coverage for specific tools indicate missing Ink adapter configuration.
30-day incident correlation: Incidents linked to AI-attributed commits via JIRA or Linear integration appear in the AI Defect Density dashboard within 30 days of merge, which confirms that the provenance-to-incident pipeline is functioning.
Stakeholder alignment on thresholds: Engineering leadership, security, and finance agree on governance thresholds for AI Rework Rate, Token-Spend Efficiency, and Security Issue Introduction Rate. Thresholds are documented and enforced as policy gates.
Board-ready ROI reports delivered in hours: The Exceeds AI platform generates longitudinal AI versus human outcome comparisons, including cycle time, rework rate, defect density, and token spend, exportable for executive review without manual data assembly.
Advanced Considerations for Scaling Across Large Teams
Scaling this framework across a 50–1,000-engineer organization introduces three operational challenges that, if left unaddressed, will undermine the reliability of your measurements and the trust stakeholders place in them.
Cross-team threshold calibration: Healthy thresholds vary by codebase maturity, team tenure, and AI tool mix. A platform team with a decade-old monorepo will show different complexity drift baselines than a greenfield microservice team. Establish per-team baselines over the first 30 days before applying organization-wide governance thresholds.
Threshold maintenance as AI usage grows: Industry average AI-assisted lines sit between 15–25%, and top-quartile teams reach 40–60%. As AI code share grows, absolute defect counts will rise even if rates stay constant. Normalize all metrics to rates, such as per KLOC or per 100 commits, rather than absolute counts, and recalibrate thresholds quarterly as the denominator grows.
Frequently Asked Questions
How Exceeds Ink Captures Provenance Without a Daemon
Exceeds Ink uses a hook-direct model rather than a continuously running background process. Capture is invoked from standard Git hooks, including prepare-commit-msg, post-commit, and post-rewrite, on a per-repo opt-in basis. Dedicated checkpoint materializers for Claude Code, Cursor, and Codex resolve edit evidence against the actual working tree at commit finalization, so the Git Note is written before the commit is reachable for push. The process exits immediately after finalization, which keeps attribution work off the developer's critical path. This architecture means there is no daemon to crash, supervise, or recover, no PATH-shimmed git binary, and no global git config mutation. Competing approaches often rely on those mechanisms and create real operational and security friction.
How Unknown Lines Affect the Metrics
Lines that cannot be confidently attributed to a specific AI tool or session are recorded as unknown_lines in the Git Note. They are never rolled into the "human" or "AI" category. This conservative approach preserves the integrity of every downstream metric. AI Rework Rate, Defect Density, and Token-Spend Efficiency calculations exclude unknown lines from both numerator and denominator, which prevents false inflation or deflation of AI-specific signals. The AI Attribution Coverage metric in Step 1 surfaces the proportion of unknown lines and gives teams a clear signal when adapter configuration needs attention.
How the Framework Handles Multiple AI Tools
Exceeds Ink includes first-class adapters with deep per-tool checkpoint materializers for Claude Code, Cursor, and Codex, plus adapters for GitHub Copilot and Windsurf, with lighter-weight detection across up to approximately 50 AI tools. Every Git Note records the specific tool, model, session, and interaction mode that produced each attributed line. This design allows all eight metrics to be segmented by tool. Teams can compare, for example, Cursor agent-mode rework rates against Claude Code ask-mode rework rates within the same codebase. Multi-tool segmentation underpins the interaction-mode effectiveness analysis in Step 5 and the token-spend efficiency comparison in Step 6.
How This Approach Differs from Metadata-Only or Survey-Based Tools
Metadata-only platforms such as Jellyfish, LinearB, and Swarmia track PR cycle time, commit volume, and review latency. They cannot determine which lines within a PR are AI-generated, which tool produced them, or what happened to those lines 90 days later. Survey-based tools such as GetDX measure developer sentiment about AI tools but produce subjective data that cannot be correlated with production incidents or rework rates. The eight-metric framework described here requires commit-level provenance, a durable, line-level record that travels with the code, because every metric depends on knowing the origin of specific lines, not just the aggregate behavior of a PR. Exceeds Ink is the only provenance layer that writes this record as a portable Git Note in the customer's own repository, auditable by anyone with repo access and independent of the Exceeds platform.
Conclusion: Turn Hidden AI Technical Debt into Measurable ROI
View comprehensive engineering metrics and analytics over time
Every metric in this framework requires commit-level provenance. Heuristics and watermarks cannot supply that level of precision. Client-level capture, via Exceeds Ink's Git Notes attestation, delivers the line-level, tool-aware, mode-aware data that makes these metrics authoritative rather than approximate. The result is board-ready proof that shows which AI tools are generating ROI, which interaction modes are producing stable code, and where governance thresholds need to be set before debt compounds further.
Engineering leaders who implement this framework move from answering the board with sentiment to answering with evidence.
