eBPF Continuous Profiling released

  • 12 min read

Moving fast with AI, without losing control

Judith Redi
Judith Redi
Moving fast with AI, without losing control

Nobody knows what an engineering organization looks like once AI writes most of the code. We don't either, not yet at least. But our customers need a product that holds up in that future, so we had to build for it anyway. Over the past year we went all in on AI in our own engineering. At the same time, we grew the engineering team by 58%.

This post is about what broke along the way, and about the experiment that changed how we ship. We let pull requests approve themselves.

Spoiler alert: this story has a happy ending.

Engineering at Dash0The happy ending
Headcount
+58%
Features shipped
+83%

"Nine women can't make a baby in a month" is the classic argument that adding people to a team slows it down. Turns out, sometimes they can. But in between, things broke.

Everyone got faster, and more worried

On 24 November 2025 Claude Opus 4.5 came out, and coding agents went from experiment to everyday work. The promise was real:

Nothing before has hit with the magnitude of AI.Martin Fowler, Chief Scientist, Thoughtworks

Faros tracked 22,000 developers, and in their highest-AI quarters each developer completed 66% more epics and 34% more tasks (Faros AI Engineering Report 2026).

But then we started seeing the secondary effects:

Software has become a brittle mess, with 98% uptime becoming the norm.Mario Zechner, Creator of Pi

The same Faros report showed the other side of the AI coding coin. Median time in PR review went up 441%. And 72% of organizations have already had at least one production incident caused by AI-generated code (Harness, State of AI in Software Engineering).

We were one of those organizations.

Not a great place to be when you sell observability and customers trust you with the reliability of their software.

Everyone got faster, and worried
Epics completed per developer
+66%
Tasks completed per developer
+34%
Time spent in PR review
+441%
Of companies had an incident caused by AI code
72%
Sources: Faros AI Engineering Report 2026, and Harness, State of AI in Software Engineering.

Four things broke

Code review became the bottleneck. Quality got worse before it got better. Everyone built fast, and the product started to look like a patchwork. And more agents meant a bigger bill.

This post is mostly about reviews, with a look at quality. Coherence and cost deserve their own posts.

Then came loop engineering, and reviews exploded

Agents write a pull request in minutes. Then every pull request waits in the same narrow queue for a human review. In a typical week, 56% of the pull requests we opened were still waiting for a review. The slowest 25% waited almost a day for a first look.

Flow through the pipeline · before AutoMerge
Pull requests flow through a pipe with three stages. Coding is open, so changes move through in minutes. The pipe pinches hard at Awaiting review, where pull requests pile up for almost a day before anyone looks at them.
Minutes
For an agent to write a PR
56%
Of a week's PRs still waiting for review
Almost a day
To a first review, slowest 25% of PRs
The same flow chart AI SDLC Insights draws for your repositories. The pipe pinches where a stage is slow compared to the stages around it.

Code throughput went up, and so did cycle time. Engineers spent hours a day reviewing, and at some point they started rubber-stamping. People were frustrated, and they had a point: a review nobody reads doesn't catch anything.

What if we let go of reviews?

A human review for every change had been a rule for as long as any of us had worked in software. Questioning it felt like a lot. What if we stopped?

We didn't decide that up front. We wanted to find out what happens when pull requests can approve themselves, so we built AutoMerge and ran it as an experiment. Every change to the policy started as a hypothesis, and we logged the result and dropped what didn't work. Humans were in control of the approval policy the whole time, auditing every change.

How a pull request gets approved

Every pull request goes through the same policy, whether a human or an agent opened it. On our monorepo the policy scores four criteria: author track record, blast radius, review evidence, and change scope. Some are deterministic, like authorship and reverts. An AI evaluator judges the others, like blast radius and review evidence. AutoMerge weights the scores into one number and compares it with a threshold of 75, which we can tune too. If it scores above the threshold, the pull request is approved to merge. Below it, a human decides. A criterion can also have a floor, and a score below its floor sends the pull request to a human whatever the total. Protected paths always go to a human too.

Every verdict comes with a written reason, posted as a comment on the pull request. This one is real. Agent0, our AI SRE, had found that our synthetic check emails still said FAILING, with a red bar, after the issue was already closed. It opened a small fix, and AutoMerge did not approve it. The combined score was 73.3, close to the threshold, but the total didn't decide it. Agent0 had almost no history in that part of the code, so its author track record fell below the floor. An engineer approved the fix instead, and it merged 26 minutes after Agent0 opened it.

The PR view · a real Agent0 pull request
Dash0Botcommented 3 minutes ago • edited

AutoMerge

Not approved73.3 / 100
threshold 75
Blast radius×0.3093
Review evidence×0.3082
Change scope & hygiene×0.2092
Author track record×0.2012

One criterion scored below its required floor, which rejects the run whatever the total.


Evidence

Review evidence scored 82.

Two review bots ran on this pull request with no findings, and CI passed. There are no human reviews or inline comments, so the evidence is a clean agent pass without human sign-off.

Author track record scored 12, below its required floor of 35.

The author has no authorship on the changed email template or its tests, and none in the rest of the notification component. The only earlier commit by this account is one Opsgenie fix. History is shallow, so a longer track record could not be established.

Most of our experimenting happens in the criteria, because they are the part you craft and control. What counts as a safe change depends on your codebase, so you decide which criteria to score, how much each one weighs, and where the floors sit. For us, engineers gave feedback on every verdict, and we tuned the criteria until we trusted them.

Dry run first, autonomy later

Nothing touched a real merge before we had watched it in dry run, where AutoMerge approved nothing and only recorded what it would have done. For weeks on our monorepo we compared those verdicts with what humans did. Auto-approve became the default only in mid-September.

Rollout on our monorepo · per week
  • Evaluated PRs still in dry run
  • Merged PRs that were auto-approved
PeriodEvaluated PRs still in dry runMerged PRs that were auto-approved
Aug 3172%17%
Sep 762%18%
Sep 1437%26%
Sep 2126%39%
Sep 2824%46%
Oct 57%46%

Two fences stayed in place. Protected paths are never auto-approved, whatever the score. And some pull requests that are auto-approved still get a human review, which is why the share of merges that were auto-approved stays below the share that passed the policy.

There was also a bonus side effect: pull requests got smaller, because engineers like a fast merge.

Under 1% wrongly approved, but still too cautious

Across all pull requests, we checked every verdict against what happened next. Fewer than 1% were incorrect approvals. 61% were unnecessary holds, where the policy sent a pull request to a human that could have merged on its own. That second number is the one we are tuning now. A policy this cautious still leaves a lot of review work on the table.

The policy view · every verdict checked against what happened
<1%incorrect approvals
  • 61%Unnecessary holdsHeld for a human, but safe to merge. The number we tune.
  • 33%Correct approvals
  • 5%Correct holds
  • <1%Incorrect approvals

It doesn't always work

Engineers let us know when the policy gets it wrong. Peter Froehlich, one of our principal platform engineers, posted this in Slack:

Real feedback from our Slack
Peter Froehlich12:17 PM
AUTOMERGE IS JUST TROLLING ME NOW
Not approved69.98 / 100
threshold 70
Blast radius×0.3480
Evidence quality×0.2482
Author Trust×0.4255

The score was 69.98 and the threshold was 70. The maths is right, and it still feels unfair. We'll take that kind of complaint any day, because the engineer could see exactly which criterion cost them the merge, and so could we. When a human reviewer waves a change through, there is no score to argue about.

It isn't only the policy that falls short. Costs went up across the board, and with review no longer the bottleneck, the bottleneck is moving down the line to CI, so developer experience is suddenly a topic again. Those deserve their own posts.

Our AI SRE opens pull requests too

You met Agent0 earlier, when AutoMerge held back its email fix. That was one pull request of many. Between August and October, two thirds of the pull requests it opened on our monorepo were merged, most of them bug fixes. In one example, a rate-limited API returned a generic 500. Agent0 changed it to a proper 429 with a Retry-After header, and the fix was merged 8 minutes after the pull request was opened.

Agent0 doesn't resolve incidents on its own yet. It does the digging, and a human approves the fix. When the pager goes off at 5am, a full root cause analysis is already waiting, so the engineer on call starts from an answer instead of a blank dashboard.

More than twice the merges, faster than before

We measured six weeks on the monorepo, from the end of August to mid-October. Merges per day went up 133%. The time the slowest 25% of pull requests waited for a first review went down 60%, and the median wait went down 55%. The queue of open pull requests stayed flat, while the number of pull requests opened per week grew 117%.

Results · six weeks on our monorepo
Merged PRs per day+133%More than twice the merges
Merged pull requests per day, by week from August 31 to October 5, rising 133% over six weeks.
Wait for a first review−60%Slowest 25% of PRs. Median: −55%
Time to a first review for the slowest 25% of pull requests, by week. It fell 60% over six weeks. The median fell 55%.
Open PR queueFlatWhile PRs opened per week grew 117%
Pull requests opened per week rose 117% over six weeks, while the number of open pull requests waiting stayed flat.

The median time to merge went down 49%. Reviewers now spend their time on the pull requests the policy holds back, instead of on every one.

What about quality and reliability?

Merging twice as much is only good news if what you merge holds up in production. It held, and it improved. Bugs opened per week went down 56% between the first and the sixth week. Per merged pull request, bugs went down 81%. Incidents stayed stable.

Quality · six weeks on our monorepo
Bugs opened per week
−56%
First week → sixth week
Bugs per merged PR
−81%
While merges more than doubled
Incidents per week
Stable
Over the same six weeks

Quality broke first, so we put an agent on it

The drop in bugs has a back story. Quality was the first thing to break. In Q2, open issues from Customer Solutions grew fast. Agent0 connects to Linear and Slack, so we built a bot on it that assigns bugs to teams and nudges engineers every morning: what to pick up, and what is close to its SLA. The backlog of active bugs went down 45% from its peak.

Active bugs per week · Jun 15 → Oct 5
Active bugs, from the peak−45%
Active bugs per week from June 15 to October 5, 2026. The count peaked in early July, the Agent0 reminder bot went live on July 15, and after a ticket cleanup with Customer Solutions in September the count was 45% below its peak.

It worked so well that the bottleneck moved again, this time to our customer solutions department. A ticket only closes when the customer confirms the fix, and that last step was human, so it waited. Engineering and Customer Solutions sat down together and closed what was already fixed. Then we changed the process, so fixed tickets now go back to the solutions architect who opened them to follow up with the customer.

Humans own the what and the why

So where do humans still decide? This is where we have drawn the line for now:

Who decides
Agents alone
Low-risk mergesWhatever passes the AutoMerge policy.
Bug fixes and nudgesFix PRs, daily SLA reminders.
Agents assist, humans decide
IncidentsAgent0 investigates and proposes.
Big blast radiusProtected paths, complex changes.
Humans only
Architecture, specs, tasteWhat good looks like.
Priorities and alignmentOr you build a Frankenstein product.

The line moves both ways. While autonomy grew, the share of protected pull requests also went up, from 8% to 18%. The more we learnt about where the risk sits, the more paths we protected.

Decide where things are allowed to break

When AI writes more of your code, humans read less of it. Correctness then depends on what the code does at runtime, not on what a reviewer saw in the diff, so we lean on OpenTelemetry: one open standard that both humans and agents can query.

Things will break more often, so we decided where we can afford it. Where we can roll back in minutes, a broken change costs little, and we put our effort into recovery time instead of trying to prevent every failure. Where we can't afford a break, the path is protected and a human decides. In both cases we count on runtime signals to tell us something is wrong before a customer does.

Nobody knows the end state

Looking back, most of the work was unlearning. Not long ago we would have told you that adding people slows a team down. Our headcount went up 58% and the features we shipped went up 83%. We would also have told you that every change needs a human review. Today 46% of our merges skip one, and we haven't seen a significant increase in production incidents. Even the goal changed. We used to aim for fewer incidents, and now we care more about how fast we recover from the ones we get.

We don't know what an AI-driven engineering organization looks like when it's done, and we doubt anyone does. We got this far one experiment at a time, with a dry run before every step and the policy and the telemetry in our own hands.

If you want to run the same experiment on your own repositories, AutoMerge starts in dry run, where it only comments on pull requests and changes nothing. Read how AutoMerge works.