Dash0 acquires Polar Signals

Last updated:

Roll Out AutoMerge

Recommendations for moving repositories from Dry run to Auto-approve and Auto-merge, and for tuning criteria and path rules.

Grant AutoMerge autonomy one step at a time, and only on evidence. The path below moves each repository from Dry run to Auto-approve once its decisions match your reviewers, and to Auto-merge only where you trust it to finish the job.

1. Start Every Repository in Dry Run

Dry run evaluates and comments, but approves and merges nothing. It is the only mode in which AutoMerge's decisions can be graded against your reviewers without the bot's approval influencing them.

  • Use the automatic trigger: Choose On every pull request update (recommended) in the Dry run dialog, or Automatic in the Configuration tab. Then every pull request is evaluated and graded, including the ones nobody thought to ask about.
  • Keep Wait for CI on: The Evidence quality criterion reads CI results, so a score taken before CI finishes is weaker evidence.
  • Protect sensitive paths first: Add Protected path rules for areas a person must always review, such as database migrations, infrastructure, authentication, and billing, before you move beyond Dry run.

2. Compare Its Decisions With Your Reviewers

Let Dry run see enough pull requests to cover the kinds of change the repository usually gets. Then, on the repository's Overview and Pull requests tabs:

  1. Check Agreement with humans.
  2. Filter Bot accuracy for Incorrect approval. These are the decisions that would have let a change through that your reviewers rejected. Read the rationale of each one.
  3. Filter for Unnecessary hold. These are the review time AutoMerge could have saved.
  4. Leave feedback with Was this decision right? on decisions you disagree with, so the reason is recorded next to the decision.

Agreement is a directional signal, not a guarantee. A decision nobody responded to counts as agreement, and approvals given without reading the change look like agreement too. Check Rubber-stamp approvals before you trust a high agreement rate.

3. Tune Before You Promote

Address incorrect approvals before you move on:

  • A whole area should never be auto-approved: Add a Protected path rule.
  • One risk keeps slipping through: Set a Rejects below floor on the criterion that should catch it, so it rejects on its own whatever the combined score.
  • The evaluators lack context: Add it to System prompt addition, for example which directories carry the highest consequences.
  • Too many unnecessary holds: Lower Acts at confidence slightly, or adjust the criterion the Criterion scores chart shows as lowest.

Before saving a change to criteria, prompts, or the threshold, run it on the Testing tab against a fixed sample and compare it with the current configuration. See Test a Configuration Change.

4. Switch to Auto-approve

Once Dry run's decisions match your reviewers, switch the repository's Mode to Auto-approve. AutoMerge then approves qualifying pull requests, and a person still merges them. Keep watching Agreement with humans and Incorrect approval after the switch.

To grant autonomy gradually, keep the repository in Dry run and add an Auto-approve path rule for a low-risk area first, such as documentation. Because a pull request runs at the least autonomous level of any file it touches, a change that also touches other files still runs as a dry run.

5. Use Auto-merge Where It Earns It

Auto-merge approves and merges. Reserve it for repositories or paths where Auto-approve has held up and a merge needs no further human step, such as a proof-of-concept directory. Grant it with a path rule rather than repository-wide where you can. AutoMerge merges through GitHub, so your branch protection rules still apply.

Further Recommendations

  • Turn on Fast-track verified reverts: Rolling back a change AutoMerge approved should not wait for review. The revert must exactly restore what the reverted pull request changed.
  • Consider Suppress comments while grading: Without the bot's comment, a reviewer's judgement is not anchored by the bot's, so agreement measures an independent review. The decision stays visible in Dash0.
  • Keep the check advisory: AutoMerge's check does not block merges. Make it a required check in GitHub only if you want every pull request to wait for an evaluation.
  • Connect AI SDLC Insights: The Coding Sessions evaluator tools can read how a change was produced, but only when AI SDLC Insights receives your coding harness telemetry.

Further Reading