AI code review is good at logic errors, side effects in other parts of the system and obvious security flaws. It does not catch business rules nobody wrote down, architecture decisions or what only the team knows about production. Across 26 pull requests in our projects, between August and September 2026, it flagged 51 problems before merge, and the largest group was wrong output on screens and reports. It only works when it reviews against the team's written standard, and the merge is always approved by a person.
What AI code review on a pull request is#
AI code review is a language model reading a change to the system before it enters the main codebase and leaving comments about what could go wrong. It comes before a team member's read, so that the person spends their time on what needs judgment.
A pull request is the request a developer opens to merge their change into the system. Before accepting it, someone reads what changed, and that read is the code review. As the volume of code grows, review becomes the bottleneck, and that is where AI comes in.
At Epicora, it shows up at two points. The developer runs a check on their own machine before opening the pull request and fixes what it flags. Then the reviewer runs the review on the open pull request, reads what the AI raised, deletes what does not hold up and posts the rest to the author. Both run in Claude Code, with a plugin we wrote for our projects, and neither approves anything on GitHub. It is the same design we use when we bring AI into a company's processes: the AI does the first pass, and the decision stays with whoever answers for it.
What the AI flagged in 26 real pull requests#
Between August 20 and September 28, 2026, we took the 26 pull requests in our projects where the AI review was posted as a message to the author, most of them backend. We read each message and counted what it flagged. There were 51 problems, all found before merge, not counting questions and optional suggestions.
| What the review flagged | Points | Example |
|---|---|---|
| Wrong output on a screen, report or PDF | 13 | a metric that counted the same item twice when two users shared the same setting |
| Concurrency, repeated jobs and silent failure | 11 | two messages arriving at the same moment produced two separate notifications, exactly what the change was meant to group |
| Contract with the mobile app and data already in the database | 5 | sync stopped sending a field the app relied on when offline |
| Effect on another part of the system | 5 | a time zone fix corrected the screen in the pull request and pushed another screen, outside the changed lines, one day behind |
| What the task asked for and the code did not do | 5 | the field filled in on the form never reached the final record |
| Security and personal data | 4 | a permission broader than the task required |
| Performance | 4 | a query with no index that would read the whole collection on every request |
| Repository housekeeping | 4 | files from a different package manager that came in for no reason |
| Total | 51 |
Besides those 51, the messages carried 12 questions about the rule or the task, 5 notes on what had to happen before or after release, such as fixing data already stored, and 8 minor suggestions marked as optional.
The largest group is wrong output: a number counted twice, a report caption that says the opposite of the value next to it, a field lost on its way to the PDF. This is the kind of error that passes a quick test, because the screen opens and shows something.
Five points sat outside the lines the pull request changed. The review can only find this kind of problem because it reads the repository and follows whoever calls the changed function, instead of looking only at the diff.
The count has a bias worth stating. A pull request where the review found nothing usually ends in a plain approval with no message, and stays out of this count. The 51 points describe what it finds when it finds something, not the share of pull requests with problems.
What AI catches well, what it catches badly and what it does not catch#
AI catches well what is in the code and can be followed end to end: logic, side effects, concurrency and obvious security flaws. It handles severity and style badly, because it tends to over-comment. And it does not catch what is not written anywhere it can read.
| Type of problem | How AI does | Why |
|---|---|---|
| Logic error and wrong output | catches well | it follows the data from the database to the screen and checks the math |
| Effect on another part of the system | catches well, if it reads the repository | it has to open whoever uses the changed function, not just the diff |
| Concurrency and repeated jobs | catches well, if the standard lists the signal | the team standard says what to look for, such as retries without protection |
| Obvious security flaws | catches well | a route with no permission check or exposed personal data shows up in the code itself |
| The task's acceptance criteria | catches it, if the task is written and linked to the PR | without the task, it has nothing to compare against |
| Severity of the problem | catches badly without a written bar | it tends to mark everything as serious |
| Style and preference | catches too much | it becomes noise, and the code formatter already handles it |
| Missing tests | catches badly | it asks for tests even in projects with no test suite |
| Business rule nobody wrote down | does not catch | it turns into a question for whoever owns the rule |
| Architecture decision | does not catch | it depends on cost, deadline and the product's future |
| Client and production context | does not catch | what the team already tested and the state of the data are not in the code |
The OWASP Secure Code Review Cheat Sheet, written with automated analysis tools in mind, reaches a similar list. It names business logic flaws, which require domain understanding, and vulnerabilities that depend on the application's own rules as the areas where human analysis adds the most value. It also recommends human judgment to filter the tools' false positives.
An example from our own internal management dashboard shows the boundary. The review noticed that a rehired employee showed a length of service that included the time they were away. Instead of marking it as a bug, it wrote that the doubt was about the rule, not the code, and left the question for whoever sets the rule. Whether the earlier employment counts is a company decision.
Why generic AI review turns into noise#
AI review turns into noise when it comments on everything with the same weight. The team learns to ignore the comments, including on the day one of them is serious. What fixes it is reviewing against a written standard, with a clear bar for what is serious.
We learned this from the first version of our plugin. Our calibration record from September 9, 2026 describes the symptom: almost every reviewed pull request ended in a change request. Severity was inflated, and the most common case was asking for tests in projects with no test suite. No point was verified before being written, so false positives reached the author looking exactly like real defects. At the time we had two reviewers for five or six developers, and the review queue was eating whole mornings.
The calibration changed three things.
- A written bar for what blocks a merge. A point only holds the merge when three conditions are true at once: a real path that leads to the error, damage from a closed list (broken system, lost data, permissions, a published app, a destructive migration) and a fix that is expensive to undo.
- What can be measured is taken out of the model's hands. A script checks beforehand whether the project has tests and an automatic formatter. With no test suite, the AI may not ask for tests. Style never becomes a point.
- A second read tries to knock down each point. Before the report, another instance looks for a guard that already exists, a flow that makes the behavior intentional or someone the fix would break. A point that falls is removed from the report.
Since late September, each review run leaves a record on the pull request, which lets us measure what the AI found and what the reviewer kept. In the first measured week, from September 24 to October 1, the review raised 13 points across 10 pull requests. The reviewer posted 6 and discarded 7, including the 3 minor suggestions. Of the 11 recorded runs, none recommended a change request.
It is the same logic as making AI follow your company's method: the criteria live in a document the AI reads at the moment of work, and whatever can be measured is measured by code.
Why AI does not approve the pull request#
AI does not approve pull requests because what decides the merge often sits outside the code. In our projects, the review comments and a team member approves. The plugin neither approves nor requests changes on GitHub: that action always belongs to the reviewer.
The market is moving the other way. According to GitHub's documentation, Copilot's approval can be configured to count as a teammate's approval under the repository's required-approval rule, in a feature that is in public preview. The same page says Copilot is not guaranteed to spot every problem, sometimes makes mistakes, and recommends supplementing its feedback with a human review.
In our sample, three replies from authors show why. In one review, the AI asked whether a ranking was sorted on the server or on the screen, and the author replied that the screen already sorted it. In another, it warned that a new uniqueness rule would fail because of duplicate records in the database, and the duplicates had already been cleaned up in staging and production. In a third, it pointed out that sync stopped sending two fields to the app; the author adjusted it and confirmed the app had been tested offline. In all three cases, the person holding the deciding information was on the team.
The plugin also has a gate that lets a pull request go to QA without waiting for the reviewer, when the author's check comes out clean and no risk trigger closes. In the measured week, 21 pull requests went through that check and none was released without a reviewer. The triggers that held them most were pending QA and the comparison with the task, present in all 21.
Why code written with AI needs more review#
Code written with AI needs more review because AI has become very good at writing code that runs and has not improved at the same pace at writing code that is secure. According to Veracode's March 2026 update to its code security report, which has evaluated more than 150 models, only 55% of generation tasks produce secure code, about the same rate as two years earlier. In the same study, generated code compiles in more than 95% of cases.
In practice, the code passes the "it opens and works" test and carries the problem into production. In Sistema Forja, a product built entirely by its founders talking to an AI, the assessment found all eight server endpoints accepting requests without authentication, including the payment ones. Nobody had reviewed what went live.
As more code is written by AI, review becomes the place where someone checks who can call what and whether the number is right. That is why it has to be fast and cannot bury the team in comments. What to do with a product that was born this way is covered in what can be saved from AI-generated code.
What to ask whoever builds your system#
You do not need to read code to know whether review happens. Ask to see three or four recent pull requests and ask four questions.
- Is every pull request read by a person before merge? The approval should have a name on it. If the answer is "the AI reviews it", ask who approves.
- Does the review follow a written standard for the project? It can be a conventions file in the repository or an internal document. Without one, the AI reviews by generic criteria that are not your project's.
- Is the task linked to the pull request? With written acceptance criteria, the review checks whether the code does what was asked. In our sample, 5 points were exactly that.
- What happens to the AI's comment? A good comment names the file, the line, what happens and under which condition. The author replies, and someone decides.
A company that really reviews answers all four by showing real pull requests. When it cannot show them, the next step is a code audit before the product grows.
Frequently asked questions#
Can AI replace the code reviewer?#
No. It does the first read well: it finds logic errors, effects on other parts of the system and obvious security flaws, and it does that on every pull request without getting tired. What it lacks is the context that decides the merge: the business rule nobody wrote down, the architecture decision and what the team knows about production. In our projects, the AI raises the points, the reviewer drops what does not hold up and a person always approves.
How much of what AI flags in a review is useful?#
It depends on how well it has been calibrated. In the first version of our review, almost every pull request ended in a change request, with inflated severity and requests for tests in projects that have no test suite. After we wrote down our standard and made the review follow it, the first measured week had 13 points raised across 10 pull requests, and the reviewer kept 6.
Does AI code review catch security flaws?#
It catches the obvious ones: a route with no permission check, personal data that would start showing up for the wrong people, a public input with no validation. Across our 26 pull requests there were 4 points of this kind, all before merge. What it cannot settle alone is permission that depends on the business rule, such as who may see what in each situation. That needs the rule written down and a person checking it.
Is it enough to switch on an off-the-shelf review tool on GitHub?#
Switching it on is easy, and the tool will comment. Whether the comments help depends on the standard it reads. GitHub's own documentation points to instruction files in the repository, such as copilot-instructions.md and AGENTS.md, as the way to teach the tool your project's standard. Without a written standard and a clear bar for what counts as serious, the review comments too much and the team stops reading.
How do I know whether the company building my system reviews the code?#
Ask to see a few recent pull requests. A healthy process shows the task linked to the pull request, review comments that name the file and line of the problem, a reply from whoever wrote the code and a person's approval before merge. If the review is done by AI, ask which standard it follows and who decides what goes live.
Sources#
- Veracode, Spring 2026 GenAI Code Security Update, March 24, 2026: https://www.veracode.com/blog/spring-2026-genai-code-security/
- GitHub Docs, about Copilot code review: https://docs.github.com/en/copilot/concepts/agents/code-review
- OWASP Cheat Sheet Series, Secure Code Review Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secure_Code_Review_Cheat_Sheet.html
- Our own data: review messages posted on 26 pull requests in Epicora projects between August 20 and September 28, 2026, and the review plugin's metrics report for the week of September 24 to October 1, 2026.
Next step#
If your system was built with AI and nobody reviewed what went live, start with an assessment of AI-built code, which looks at the code, the database and the infrastructure and fixes what is critical. If your team already builds software and wants AI review that follows a written house standard, that is part of our work in AI and automation.