Gustavs Steinbergs

8 min read

An AI pull request review bot in Azure DevOps

How I set up an AI agent that reviews every new pull request and leaves inline comments on bugs.

The problem

AI coding tools have made writing code much faster. Reviewing it has not sped up at the same rate. Every change still needs a review before it is merged, so pull requests pile up and review becomes the bottleneck. Even more, some of the bugs that slip through can be genuinely hard for a human to catch, since nothing looks wrong until the change is run under the right conditions, which can sometimes start failing quietly weeks after the PR was merged, as you will see in the example below.

The solution

I set up an AI agent that reads every new pull request before a human does. It looks for serious bugs only, and leaves each finding as an inline comment on the exact line. A human still reviews and approves the PR. They just start from one that has already been screened.

How it works

  1. Azure DevOps fires a webhook when a pull request is created.
  2. A Cursor automation receives the webhook and starts an agent.
  3. The agent reviews the PR against the instructions you give it.
  4. Findings are posted as inline comments on the PR, plus one short summary comment.

Prerequisites

  • Your repositories are in Azure DevOps.
  • You have an active Cursor subscription.
  • You are a Project Administrator on the Azure DevOps project. This is needed to create service hooks.

Step 1 - Create the agent in Cursor

  1. Open Automations and click New Automation.

  1. Select the Azure DevOps repository the agent should run on.
  2. Under Triggers, add Webhook triggered.
  3. Click Generate auth header and copy the value now.
  4. Fill in Agent Instructions. Mine are based on Cursor's Bugbot, with added instructions for inline commenting. The full instructions are available in the appendix.
  5. Optionally, add the Azure DevOps MCP under Tools. The agent can also post comments through the Azure DevOps REST API using the repository you selected above, so it can work without the MCP.
  6. Save and enable the automation, then copy the webhook URL.

The agent is ready. Next, set up Azure DevOps to trigger it.

Step 2 - Create the trigger in Azure DevOps

  1. In your Azure DevOps project, open Project settings in the bottom left corner.

  1. Open Service hooks.
  2. Click +, select Web Hooks, then click Next.
  3. Set the trigger to Pull request created, pick the repository, then click Next.

  1. Paste the webhook URL from Cursor into URL.
  2. Paste the auth header from Cursor into HTTP headers.
  3. Click Finish.

To test it, open a pull request in that repository. The agent is triggered on that PR and the run appears under Run History in Cursor, while the comments show up on the PR.

Results

The agent is deliberately strict about what it reports. It only comments when it can describe a concrete scenario that triggers the bug, and it ignores style issues and minor edge cases. You can, however, adjust the instructions for the agent to also focus on correct naming conventions or specific infrastructure guidelines. If the PR is clean, the agent leaves a single summary comment saying no critical bugs were found.

An example

This setup has already helped flag and resolve critical bugs that would have been missed. A PR moved a scheduled job to a newer runtime environment. The new environment no longer included a library that the job's code imports. Nothing about the PR looked wrong, so nothing was flagged by AI during development.

The job only runs on a monthly schedule, so this change would not have broken anything until that job's next run, a month or more later in production. Even then, the failure would have been easy to overlook, since the rest of the pipeline would just keep running on old settings instead of visibly breaking.

The agent predicted the issue at review time. It traced what the job's code actually needed against what the new environment provided, and flagged that this specific future run would fail. These kinds of bugs are very easy to miss when a human is reading through, since nothing is wrong yet at the moment you are looking at it.

Limits and add-ons

  • It does not replace human review. The agent only comments and in the current setup it does not approve, merge, push commits or open fix PRs.
  • It runs once per pull request. The trigger is "Pull request created", so commits pushed afterwards are not reviewed. Add a second service hook for "Pull request updated" if you want that.
  • Current setup only looks for bugs. Design, readability and test coverage could be added as extra checklist items.

Appendix: Full Agent Instructions

Show the full instructions
Go through the PR and perform a critical bug audit.
 
You are a deep bug-finding automation focused on high-severity issues.
Your job is REVIEW ONLY: find real bugs, mark them as Active inline PR comments for a human to resolve, and do not change code or open fix PRs.
 
Before doing anything else, read MEMORIES.md from your persistent memory. It tracks bugs you have already reported across runs — each with a one-line description (location and root cause), the PR URL / thread reference, a status, and the date it was recorded. Do not investigate or re-report a bug that already has an open Active review thread (or open tracked fix PR) for the same root cause on this PR or an equivalent open PR.
 
## Goal
 
Inspect the PR diff (source branch vs target) and identify critical correctness bugs that escaped review. Only surface issues that would cause data loss, crashes, security holes, or significant user-facing breakage.
 
## Investigation strategy
 
- Focus on behavioral changes with meaningful blast radius.
- Look for: data corruption, race conditions that lose writes, null dereferences in critical paths, auth/permission bypasses, infinite loops, resource leaks, and silent data truncation.
- Trace through the full code path — don't just pattern-match on the diff. Understand the caller chain and downstream effects.
- Ignore: style issues, minor edge cases, theoretical concerns without a concrete trigger, and low-severity issues that would merely degrade UX.
- Draft / TEST / "not for merge" / intentional demo harnesses: still audit and leave inline comments; do not open fix PRs.
 
## Confidence bar
 
- You must be able to describe a concrete scenario that triggers the bug.
- If you cannot construct a plausible trigger scenario, do not comment that finding.
- Prefer fewer high-confidence findings over many speculative ones.
 
## Review strategy (DO NOT FIX — ONLY REVIEW)
 
- Do not edit application code.
- Do not open fix PRs, push commits, approve, or merge.
- For each confirmed critical finding, create an **Active inline comment thread** on the exact file and line in the PR (right/new-file side when possible).
- Each inline comment should include:
  - Severity (Critical / High)
  - What is wrong
  - Concrete trigger scenario
  - Impact
  - Suggested fix direction (short; do not implement it)
- Also post one short summary comment on the PR (overall verdict + count of inline findings, or "no critical bugs found").
- If Azure DevOps MCP auth fails, fall back to git fetch for the diff and ADO REST `pullRequests/{id}/threads` for comments using available repo credentials. Do not skip the inline comments if REST works.
 
## Avoiding duplicate reports
 
For each bug that matches a tracked entry in MEMORIES.md (or an existing Active thread on this PR):
 
- Same bug already has an Active inline thread / open tracked PR: do NOT create another thread. Note it in the summary with a link.
- Previously tracked fix PR merged, or bug no longer present: delete the MEMORIES entry.
- Previously tracked item rejected / closed without fix and code unchanged: keep as rejected; do not re-open unless the relevant code materially changed.
- Keep MEMORIES.md small: only open or rejected tracked items, with date recorded. Do not log run history there.
 
## Safety rules
 
- Only comment when highly confident the bug is real.
- Never invent findings to "look thorough."
- If no critical bug is found, post a short PR summary: "no critical bugs found". This is the expected outcome most days.
- Do not exfiltrate secrets, credentials, or private data into comments.
 
## Output / LAST STEP
 
1. Create Active inline threads for every confirmed critical finding.
2. Post the PR summary comment.
3. In the agent run response, briefly list: findings (file:line + one-line cause), whether inline comments were posted, and any delivery failures.