The place I see AI coding value get misread most often is the pull request.
AI coding tools can make code appear faster.
That is not the same as making software delivery faster.
The difference shows up at the review gate. A model can generate a diff quickly, but the team still has to understand the intent, inspect the design, verify the tests, handle CI failures, check security implications, preserve maintainability, and own the code after it merges.
If the workflow produces more code than the review system can safely absorb, the gain turns into review debt.
The practical artifact I would ask for is a review packet.
For every AI-generated pull request, the reviewer should see the task, relevant sources, files changed, reasoning summary, tests run, failures, risk areas, and rollback notes. If that packet is missing, the reviewer pays the tax.
That is why I am careful with the phrase “software factory.” It can be useful, but only if the factory is defined as a learning loop, not a code-output machine.
The bottleneck moves
The naive metric is typing speed.
How fast can the agent write the patch? How many lines changed? How many pull requests opened?
Those are easy to count and weak as delivery measures.
The stronger question is whether the whole engineering loop improved:
- Was the issue understood?
- Was the intent captured?
- Did the implementation match the spec?
- Were the right tests added or updated?
- Did CI fail for a known reason?
- Did the reviewer get useful evidence?
- Did post-merge feedback update context or evals?
That is where AI-assisted coding often gets harder.
A reviewer does not only review code. They review the reasoning that led to the code. If the agent cannot show the task, plan, relevant context, tests run, failures encountered, and tradeoffs made, the human reviewer has to reconstruct that work.
That reconstruction is expensive.
It also changes trust. Reviewers may become slower when they suspect the diff is larger than the underlying understanding. They check more files, rerun more tests, and ask more basic questions. The agent saved typing time, but it moved the work into verification.
That does not make AI coding useless. It means the system has to reduce reviewer uncertainty. A smaller, well-explained diff with targeted tests can be more valuable than a large patch that forces the reviewer to reconstruct the entire task.
The delivery loop to measure
A useful AI coding loop connects intent, implementation, tests, review, deployment, telemetry, and correction.
Start with issue selection. Not every ticket is a good agent task. Some need product judgment, ambiguous stakeholder tradeoffs, or architectural context the agent does not have.
Then spec and intent. The agent should work from a clear description of what should change and what should stay stable. If the intent is vague, the code will look confident while hiding assumptions.
Then implementation. The agent edits files, but the system should record which context it used and why those files were selected.
Then tests. The workflow should run relevant tests, capture failures, and explain what changed after a retry.
Then review artifacts. A good pull request should include the plan, notable decisions, test results, risk areas, and any unresolved uncertainty. The reviewer should not have to mine a chat transcript for evidence.
Then deployment signal. If a change causes incidents, rollbacks, or customer-visible issues, the loop should learn from that.
Then context updates. Failures should improve future context and eval cases. Otherwise the system repeats the same mistake with better syntax.
This loop should be visible in the pull request. If a test failed and the agent changed direction, the reviewer should see that. If a source was missing and added to context, the reviewer should see that too. Hidden retries make the final diff look cleaner than the process was.
That visibility is useful even when the final code is good. If the agent tried three approaches before finding the right one, the reviewer may want to know which paths were abandoned and why. Those abandoned paths often reveal risk areas.
What leaders should stop rewarding
Stop rewarding raw output.
More generated code can be useful. It can also create more review work, more unclear ownership, and more shallow tests.
The safer operating claim is this: AI-assisted coding should be evaluated across the delivery loop, including tests, review artifacts, CI failures, and post-merge learning.
That is also where software factories become more realistic.
A factory has quality control. It has feedback. It has supply-chain constraints. It has inspection. It improves the process when something breaks. If the agent only creates more diffs, the metaphor is doing too much work.
The review packet is also a management signal. If the team cannot produce it, the coding agent may be operating outside the delivery system. If the packet gets better over time, the loop is learning. That is a more useful indicator than whether the agent opened more pull requests this week.
The AI engineering function should not measure only generation volume. It should define the review standard, the evidence packet, the accepted-risk threshold, and the feedback loop that turns failures into better context and evals.
Leaders can make this concrete in one sprint. Pick a small class of agent-assisted changes and require the same review packet for each one. Then ask reviewers whether review time, confidence, and defect discovery improved. That is a better experiment than counting generated code.
Before measuring AI coding by output, ask:
Where does your AI-assisted delivery loop currently learn from failure, and where does it simply create more review work?
A few things to try
Check out the explain-diff skill from Geoffrey Litt. It's a cool idea for generating an interactive HTML that you can include with a PR to help reviewers ensure they understand the diff.
Think about what else you might want to include in the review packet that proves the work was completed and that it meets all the acceptance criteria. Obivious inclusions are things test coverage and code scan results, but consider what else might be useful- for example, on a UI change, include a before and after screenshot or even a screen recording.
I've already mentioned this, but it's worth repeating, don't bury the failures. Remember, you want visibility, your workflow should have adequate logs and tracing so that you can have the agent reflect on any failures or corrections made during the run and provide recommendations for improvement. Afterall, loops are hot right now - more on logs and traces later.