Dash0 acquires Polar Signals

Last updated:

Operate AutoMerge

Read AutoMerge's decisions, leave feedback, test configuration changes, and change or stop a repository's mode.

Once Dry run is running, Darkplane → AutoMerge has two tabs: Overview, across all repositories, and Repositories. Each repository then has four tabs of its own: Overview, Pull requests, Configuration, and Testing. Use the time range picker in the header to choose the period the numbers cover.

Read the Overview

The organization-wide Overview tab shows what AutoMerge is doing across your repositories:

  • AutoMerge mode per repository: How many repositories run in each mode. This reflects the current configuration and does not change with the time range.
  • Auto-approved pull requests: Pull requests AutoMerge approved and wrote back to GitHub. Dry run decisions are not counted, so this stays at zero while every repository is in Dry run.
  • Review time saved: The number of auto-approved pull requests multiplied by the median wait for a first review. Read it as a lower bound.
  • Auto-approved pull requests over time: Daily auto-approvals, as a percentage of pull requests opened that day or as a total.

When there is nothing to show yet, a callout at the top says why and links to the next step, for example Dry run evaluates without approving while every active repository is in Dry run.

Review Decisions on a Repository

Repository Overview

A repository's Overview tab measures how well AutoMerge's decisions hold up:

  • Agreement with humans: The share of graded decisions that matched what reviewers did. See Agreement With Humans.
  • Intervention rate, Time to merge, and Rubber-stamp approvals: How often people step in, and how quickly approved pull requests merge.
  • Criterion scores: Each criterion's score distribution, sorted by median with the lowest first. The criterion on the left is the one worth tuning.
  • Which criterion decided each outcome?: How often each criterion, a protected path, or a verified revert decided the outcome.
  • Criterion scores over time: Score trends, with a marker for each saved configuration. Hover a marker to see what changed.

Pull Requests

The Pull requests tab lists every pull request with:

  • Decision: Merged, Approved, Blocked, or Pending.
  • Soft score: The combined score, against the threshold it approves at.
  • Deciding criterion: The criterion, Protected path, or Verified revert that decided the outcome.
  • Bot accuracy: The agreement outcome, or Not graded yet.
  • Feedback: Agree or Disagree, if someone left feedback.

Filter by Decision, Feedback, or Bot accuracy to find the decisions worth reading, for example every Incorrect approval.

A Single Pull Request

Select a pull request to open its details. The summary shows the Decision, the level it Ran as, the Deciding criterion, and a one-sentence Reason, such as "The combined score of 82 cleared the threshold of 75." Below it, each criterion shows its score and, when expanded, the evaluator's rationale.

On a repository with the Manual trigger, click Evaluate now to start an evaluation from Dash0.

Leave Feedback

Feedback tells the team operating AutoMerge whether a decision was right. To leave it:

  1. Open the pull request's details in Dash0, or follow the feedback link in the bot's comment on GitHub.
  2. Click Was this decision right?.
  3. Choose Right or Wrong, and optionally explain what made the decision right or wrong.
  4. Click Submit feedback.

Feedback is recorded with the evaluation and counted on the Pull requests tab. It does not change scoring automatically. Use it to decide which criteria, prompts, or path rules to change.

Test a Configuration Change

The Testing tab replays a configuration against a set of past pull requests, so you can see what would change before you save. Tests never change what is running.

  1. On the Testing tab, create a sample under Samples: a fixed set of past pull requests, picked by a filter or listed by number. A fixed sample stays the same, so two tests against it stay comparable.
  2. Click New test.
  3. Under Configuration, choose the current configuration or a saved version. Optionally, choose a second one under Compare against.
  4. Under Test against, choose the sample, then click Run.

The result groups pull requests into Approved, agreed, Approved, shouldn't have, Held, agreed, and Held, could have merged.

Note

Protected paths, path rules, and CI readiness are not replayed, and some evaluator tools are unavailable during a replay. Read a test as a comparison between configurations, not as a claim about what production would have done.

Change a Repository's Mode

  1. Open the repository's Configuration tab.
  2. Under Mode, choose Off, Dry run, Auto-approve, or Auto-merge.
  3. Click Save.

The new mode applies to evaluations that start after you save. To find out when a repository is ready for the next mode, see Roll Out AutoMerge.

Stop AutoMerge

To stop AutoMerge on a repository, set Mode to Off and save. Nothing is evaluated while a repository is Off, and its Overview keeps showing what the bot did before.

A repository also stops being evaluated when it is Disconnected, because no GitHub integration covers it any more, or Disabled. A disabled repository keeps its configured mode. Select a disconnected repository in the GitHub integration again to resume.

Remove a Repository

Click Remove on the repository page to delete its AutoMerge history, including its pull requests, evaluations, and tests. Removing requires the Admin role. If you add the repository to a GitHub integration again, it comes back with default settings and its mode set to Off.

Further Reading