Written by: Mark Hull, Co-Founder and CEO, Exceeds AI | Last updated: August 6, 2026
Key Takeaways
- Measuring AI productivity impact works when you track commits at line level and compare AI-generated code against same-engineer baselines.
- Metadata-only dashboards miss which AI tools, models, and interaction modes actually improve outcomes, so velocity can rise while technical debt quietly grows.
- Controlled experiments and 30–90 day tracking show whether AI-assisted code holds quality or creates hidden maintenance and incident risk.
- Turning measurement into Coaching Surfaces and skill-transfer plays lets managers spread proven patterns and coach out anti-patterns across teams.
- Exceeds AI supplies the provenance layer and coaching tools engineering leaders use to deliver board-ready ROI proof, so you can start your free pilot today.
The Operational Problem
Ninety-one percent of engineering leaders say AI has improved developer velocity and quality, yet many say technical debt, validation time, and developer burnout are missing from their metrics. Leadership demands visible efficiency gains while managers lack the signals to separate real AI impact from vanity metrics.
A longitudinal study across more than 400 engineering organizations found that as AI coding tool usage increased by an average of 65%, median PR throughput increased by just under 8%. That gap stays invisible to any tool that cannot distinguish AI-touched code from human-written code at the commit level.
The tutorial below delivers a repeatable, repo-access-based measurement process that answers the board question: “Is our AI investment paying off?” The framework operates in three phases: first, it establishes individual baselines that remove confounding variables in Steps 1 and 2. Next, it captures and analyzes AI-specific signals at the commit level in Steps 3 through 6. Finally, it converts those insights into actionable coaching interventions in Step 7 so managers can change behavior, not just dashboards.
Step 1 – Establish Same-Engineer Baselines Before AI Exposure
Purpose: Eliminate confounding variables such as tenure, seasonality, and team composition by measuring each engineer against their own pre-AI history instead of against peers.
Inputs required: At least one full quarter of Git history per engineer, work-item data from Jira or Linear, and a defined task taxonomy that classifies work by complexity, including feature implementation, bug resolution, refactoring, and documentation.
Observable success criteria: Each engineer has a documented per-task-category baseline for cycle time, PR throughput, and rework rate. This individual-level baseline makes productivity deltas measurable. A US-based fintech with 1,000–2,000 engineers found that Copilot users delivered 25% more PRs than non-users, and that delta became visible only because the baseline was established at the individual level.
Pro tip: Collect baseline data silently before announcing the measurement program. Announced measurement changes behavior, while silent historical analysis preserves natural patterns.
Step 2 – Capture Line-Level Authorship With Scoped Repo Access
Purpose: Same-engineer baselines only work when you know which lines came from AI and which came from humans. Repo access provides that code-level truth because metadata alone cannot distinguish AI-generated lines from human-written lines.
Inputs required: Read-only OAuth authorization to GitHub, GitLab, or Azure DevOps, and installation of Exceeds Ink, the on-machine provenance layer that writes a structured attestation as a Git Note at refs/notes/exceeds-ink for every commit. Ink uses a hook-direct model with no long-lived daemon, no PATH-shimmed git binary, and no global git config mutation, and it runs only when a Git hook fires.
Observable success criteria: Every new commit carries a line-level attestation recording the AI tool, model, session, interaction mode, and timestamp. Lines that cannot be confidently attributed are recorded as unknown_lines instead of being silently assigned to human or AI. First insights appear within 60 minutes of authorization.
Watch-out: Many AI coding tools leave no identifiable commit metadata, which makes it difficult for researchers to attribute AI-generated code vulnerabilities. Without metadata, teams fall back to heuristic detection, which uses pattern-matching algorithms that guess whether code is AI-generated based on stylistic signals. These methods top out around 20–25% accuracy, so they cannot support board-level decisions. Client-level capture via Exceeds Ink replaces guesswork with proof because it observes what happens on the engineer’s machine at the moment of work.
Install Exceeds Ink and start capturing line-level provenance
Step 3 – Attribute Every Commit to the Exact AI Tool, Model, and Mode
Purpose: Modern engineering teams use several AI tools at once, and outcomes vary by tool and interaction style. Eighty-four percent of developers are using or planning to use AI tools, and 51% of professional developers use them daily. A measurement program that treats everything as generic “AI” cannot reveal whether Cursor, Claude Code, GitHub Copilot, Codex, or Windsurf produces better results for a given task type.
Inputs required: Exceeds Ink’s per-tool checkpoint materializers for Claude Code, Cursor, and Codex, plus adapters for GitHub Copilot and Windsurf. Ink also captures the underlying model, including Claude/Opus, GPT, Gemini, and others, and the interaction mode per session, including plan, ask, agent, edit, or headless.
Observable success criteria: Each commit record identifies the specific tool, model, session, and interaction mode. Cross-tool outcome comparison becomes practical, so Cursor agent-mode PRs, GitHub Copilot autocomplete PRs, and Claude Code refactoring sessions appear as distinct cohorts with distinct outcome profiles.
Common mistake: GitHub Copilot, Cursor, and Gemini Code Assist provide no commit-level attribution by default and only aggregate or per-acceptance telemetry. Relying on vendor-supplied telemetry alone produces an incomplete and tool-biased picture.
Step 4 – Balance Speed and Quality With Same-Engineer Cohorts
Purpose: No single metric proves AI ROI. Speed metrics without quality counterweights hide technical debt, so the framework uses a balanced set of signals that hold each other in tension.
Inputs required: Ink-attested commit data joined to work-item complexity classifications, PR review records, and incident or defect logs. Same-engineer cohort views group each engineer’s AI-on and AI-off periods for direct comparison.
Observable success criteria: Each cohort view surfaces at least one speed metric and one quality counterweight at the same time. Never track a speed metric without pairing it with a quality metric, because if deployment frequency improves while change failure rate worsens, the system has not improved, it has simply shifted debt.
Watch-out: In one Larridin customer environment, AI-assisted code had a 0.2% 30-day revert rate compared with 15.16% for human-only code. In another environment, AI code share rising to 56% coincided with 30-day code turnover increasing 64.7%. The same metric can tell opposite stories across teams, so cohort-level views prevent false aggregation.
Step 5 – Run Controlled AI Rollout Experiments by Task Type
Purpose: Isolate the effect of a specific tool or interaction mode by holding the task type constant. Comparing AI-assisted feature work against AI-assisted bug fixes mixes complexity with tool effectiveness and blurs the signal.
Inputs required: A task taxonomy from Step 1, Ink’s interaction-mode classification, and a rollout plan that assigns the same task category to engineers using different tools or modes during the same sprint window.
Observable success criteria: Statistically comparable task cohorts show measurable deltas in cycle time, rework rate, and defect density that you can attribute to the tool or mode variable instead of task complexity. Teams with higher AI coding tool adoption have demonstrated faster PR cycle times, and controlled experiments explain and close that gap.
Pro tip: Zapier tracks employees’ AI token usage and investigates cases where usage is five times higher than peers to determine whether it represents efficient “golden patterns” or wasteful “anti-patterns”. Token spend per task category acts as a leading indicator of which interaction modes deserve scaling before outcome data accumulates.
Start running controlled AI experiments in your repos
Step 6 – Follow AI-Touched Code for 30–90 Days
Purpose: AI-generated code can pass review and initial tests while hiding subtle architectural or maintainability issues that surface weeks later. Immediate metrics miss these delayed signals.
Inputs required: Ink-attested commit records joined to incident logs, follow-on edit histories, and test coverage trends, tracked at 30-day and 90-day windows by code origin, including AI tool, model, and interaction mode.
Observable success criteria: Each AI-touched module has a documented 30-day and 90-day outcome record covering revert rate, incident rate, and follow-on edit frequency. New Relic’s 2026 State of AI Coding report found that 74% of respondents said at least 25% of AI-generated code required significant post-deployment rework and 78% reported more production incidents, patterns that only appear through longitudinal tracking by code source.
Watch-out: GitClear’s longitudinal analysis found that code churn rose from a 3.3% baseline in 2021 to 5.7–7.1% by 2024–2025, with AI-assisted code exhibiting higher baseline churn than human-only code. Rising AI code share combined with rising churn or incident rates over the same period signals accumulating technical debt.
Step 7 – Turn Insights Into Coaching Surfaces and Skills
Purpose: A measurement program only creates value when it changes how people work. The goal is to identify patterns that work, distribute them as versioned skills, and roll them back cleanly when they do not land.
Inputs required: Exceeds AI’s Best Practices Insights, a LangGraph-backed analysis pipeline that distills the top patterns worth scaling and sorts them by confidence. Coaching Surfaces that present coachable signals to managers, and the ink-prompting-coach skill, a SKILL.md and slash command that installs directly into the engineer’s own Claude Code or Cursor agent.
Observable success criteria: Underperforming cohorts receive targeted coaching plays within one sprint of pattern detection. Zapier’s chief AI transformation officer said, “We start to draw conclusions whether that’s a golden pattern we want to multiply across their peers or whether it’s an anti-pattern that we want to coach our way out of”. Skill transfer and rollback tracking confirm whether distributed plays are adopted and whether they produce the expected outcome delta.
Common mistake: Microsoft’s ICSE 2008 study found organizational-complexity metrics including team size and management span to be among the strongest predictors of defect-proneness. As manager-to-IC ratios stretch toward 1:8 or higher, coaching bandwidth shrinks at the same moment AI adoption creates the most coachable moments. Automated coaching distribution via ink-prompting-coach closes that gap without requiring managers to inspect every PR.
Validation Signals for This Framework
A measurement program built on this framework produces three observable indicators that it is working:
- Reproducible cohort deltas. The same-engineer baseline methodology produces consistent before and after comparisons that hold across sprint cycles and are not explained by task complexity or seasonality.
- Stakeholder alignment on ROI proof. Engineering leaders can present commit-level attribution data, including specific tools, models, interaction modes, and outcome deltas, to executives and boards without relying on survey estimates or vendor-supplied acceptance rates.
- Documented coaching actions. Every identified pattern, whether a golden pattern to scale or an anti-pattern to coach out, has a corresponding skill-transfer play with adoption tracking and outcome verification.
Scaling and Governance With Exceeds Ink
Scaling the framework to additional teams works best with per-repo opt-in for Exceeds Ink hooks instead of a global rollout, which preserves team autonomy while expanding the cohort dataset. Governance policies become expressible once Ink’s structured JSON attestation lives in the repo. Organizations can block deploys when AI authorship exceeds a defined threshold in sensitive paths, or require additional review on commits where Cursor agent-mode produced more than a specified percentage of the diff.
Feeding provenance data into existing BI tools is supported through Ink’s standalone mode, which pipes Git Notes attestation data directly into a data warehouse without requiring the full Exceeds AI platform. Integrations with GitHub, GitLab, Azure DevOps, Jira, and Linear operationalize insights inside existing workflows instead of forcing context switching to a separate dashboard.
Teams that implemented AI-specific review practices have reduced the associated review overhead and brought net review time closer to levels seen with human-generated code. Ink’s attestation data makes those practices enforceable at the policy layer instead of depending on developer self-reporting.
Frequently Asked Questions
How is this approach different from metadata-only DORA dashboards?
DORA dashboards measure delivery pipeline health, including deployment frequency, lead time, change failure rate, and mean time to recovery, using metadata from Git and CI/CD systems. They cannot identify which lines in a PR are AI-generated, which tool produced them, or whether AI-touched code performs differently over 30–90 days. This framework uses Exceeds Ink to write a line-level attestation alongside every commit, which enables outcome comparisons between AI-generated and human-written code at the same granularity DORA uses for pipeline events. DORA metrics remain useful as part of the quality counterweight layer in Step 4, so they act as inputs to the framework, not substitutes for it.
What does the repo-access security review process look like?
Exceeds AI is designed to pass enterprise security reviews. Code exists on servers for seconds during analysis and is then permanently deleted. Only commit metadata and snippet information persists, with no permanent source code storage. Exceeds Ink operates with read-only OAuth authorization, per-repo opt-in, no global git config mutation, and no PATH-shimmed git binary. HMAC-SHA256-signed remote ingest with revocable per-machine tokens, LLM-based prompt redaction before persistence, SSO/SAML support, audit logs, and an in-SCM deployment option for highest-security requirements are all available. Exceeds AI has passed formal enterprise security evaluations, including a two-month review process at a Fortune 500 retailer.
How does multi-tool coverage work when engineers switch between Cursor, Claude Code, Codex, GitHub Copilot, and Windsurf?
Exceeds Ink uses per-tool checkpoint materializers for Claude Code, Cursor, and Codex that resolve edit evidence against the actual working tree at commit finalization. Adapters for GitHub Copilot and Windsurf supplement these with native hooks and code pattern analysis. Lines that cannot be confidently attributed to a specific tool are recorded as unknown_lines instead of being silently assigned. The result is an aggregate view of AI impact across the entire toolchain alongside tool-by-tool outcome comparisons, so a CFO asking whether the AI investment is paying off receives an answer that spans every tool the team uses, not just the one with the most visible telemetry.
How are false positives in AI detection handled?
Exceeds Ink uses client-level capture instead of heuristic classification. Because Ink observes what is actually happening on the engineer’s machine at the moment the work is done, including which tool was active, how long the session ran, and which interaction mode was used, it does not need to guess from code patterns after the fact. Ink’s per-tool checkpoint materializers protect known human-typed lines from being overwritten by AI attribution during multi-edit sessions, and lines below the confidence threshold are recorded as unknown_lines with a conservative attribution instead of inflating either the AI or human count. This approach avoids the accuracy ceiling that limits heuristic approaches.
How long does it take to get from repo connection to board-ready ROI data?
GitHub or GitLab OAuth authorization takes about five minutes. Repo selection and scoping takes about fifteen minutes. First insights appear within 60 minutes of authorization, and complete historical analysis covering up to 12 months of commit history completes within four hours. Real-time updates appear within five minutes of new commits. Board-ready ROI reports with reproducible cohort deltas typically emerge within two to four weeks of Ink deployment, once enough post-baseline commits have accumulated to support statistically meaningful same-engineer comparisons.
Conclusion
Engineering leaders now have a code-level, not survey-level, path to proving and improving AI productivity impact. The seven-step framework of same-engineer baselines, Exceeds Ink provenance, tool-specific attribution, metrics in tension, controlled experiments, longitudinal outcome tracking, and Coaching Surfaces produces reproducible cohort deltas that answer the board question with commit-level evidence instead of acceptance-rate estimates.
Metadata dashboards will continue to show what happened. This framework shows which AI tool caused the outcome, whether the result held up 90 days later, and what managers should do next to scale what works.