eBPF Continuous Profiling released

  • 7 min read

Resolved isn't fine: a morning digest of failed checks, written by Agent0

Lukas Toma
Lukas Toma(Dash0)
Goran Čeko
Goran Čeko(Dash0)
Sergei Gannochenko
Sergei Gannochenko(Dash0)
Resolved isn't fine: a Slack digest of failed checks written by Agent0, listing two new findings after every check turned green.

At 03:12 CET one night in November, a database in one of our regions went down. Checks failed, notifications went out to a Slack channel nobody reads at 3 a.m., and a while later the database came back. By the time the team logged on, every check was green.

Green didn't mean done. Two batches of usage records sat in a dead-letter queue, and a nightly sync job had never run. The checks had recovered along with the database, but the work that should have happened during the outage hadn't.

We found both before lunch, thanks to a digest that Agent0 posts to Slack every weekday at 07:07 CET.

A notification channel is a bad to-do list

A notification channel records when things broke. It doesn't record what still needs doing. That night, 28 failed checks fired in our channel. All 28 resolved on their own, and none of them pointed at the stuck batches or the missing sync run. You could read every one of those messages and still not know there was work left.

What the team wanted at 07:00 was one message that answers three questions before standup.

  • What needs a human today?
  • What should someone verify?
  • What can we ignore?

How the digest works

The digest is a Dash0 automation. Agent0 runs a prompt on a schedule, and guardrails limit what it can touch. Each weekday morning it does five things.

  1. Finds its window. It reads the channel for its previous digest and covers everything since, so Monday's digest covers the weekend. If it already posted in the last 6 hours, it posts nothing.
  2. Groups failed checks by cause, including the ones that resolved overnight. If five check rules fail on one service within ten minutes, the digest shows them as one finding.
  3. Investigates each finding with traces, logs and metrics. It gives each finding a status (NEW, ONGOING 3d or RESOLVED), a confidence, the evidence, and the cause or a plain "not established".
  4. Checks what a resolved failure left behind. We care most about this step. Agent0 looks for work that should have happened during the failure and didn't, such as a job with no run or a queue that never drained.
  5. Posts one short message, with a single Fix: or Verify: step per finding. The details and the quieter signals go into the thread.

It also reads the replies under its previous digest. When someone writes "replayed, queue is empty", the next digest marks that finding resolved and stops asking about it.

What Agent0 caught

This is the 07:07 digest from that morning. We recreated it from Slack and changed the internal names.

Dash0Agent7:07 AM
Revenue Engineering Daily · Tue 03 Nov 2026 06:05 to Wed 04 Nov 2026 06:05 UTC
Nothing needs action today · 2 findings · none billing-affecting
👀 2 accounting windows in eu-1 usage-records-dlq · NEW · orgs not attributable
↳ Fix: Compare the 2 DLQ windows with usage_watermarks, then replay once | unowned
👀 bi-sync credits-crawler did not run · NEW · orgs not attributable
↳ Fix: Find why the 04 Nov 03:25 CET run never started and confirm credits_daily rows for 4 Nov | unowned
The parent message. Both findings are NEW, and each has one Fix: step.

The headline says "Nothing needs action today". That was wrong, or at least too calm. No check was still failing and nothing looked billing-affecting, so by the digest's own rules nothing needed action. Each of the two lines underneath named a concrete fix, and those lines got the work done.

The details went into the thread.

Dash0Agent7:07 AM
👀 2 accounting windows in eu-1 usage-records-dlq
NEW · not-yet · confidence medium
The topic's current offset moved from 3549 to 3551 between 04:05 and 05:05 CET, and depth reads 2 in eu-1 only (0 in the other four regions). The accountant failed 2 windows while ClickHouse was unreachable.
  • 03:12 to 03:14 CET · All endpoints are non-responsive, falling back to primary endpoint 1,498 times, ClickHouse ServerException 65, 4 of 4 query attempts failed
    Why: ClickHouse outage in eu-1 (pod clickhouse-0 not ready 03:15 to 03:23 CET). No accountant deploy near it: 1.8.312 23:56 CET on 3 Nov, 1.8.314 06:37 CET on 4 Nov. The "Evaluating signals failed" check fired for 0 organizations, so the organizations are not attributable from logs.
    Fix: Compare the 2 windows with usage_watermarks, then replay the DLQ once | unowned · no ticket
👀 bi-sync credits-crawler did not run
NEW · unknown · confidence medium
The job last ran 3 Nov 03:25 CET (spans OK, "Populated credits"). Nothing ran at 4 Nov 03:25 CET, and the freshness check has been critical since 04:43 CET. The same bi-sync pod (1.4.77) was running from 01:31 to 06:34 CET.
  • 03:13 to 03:14 CET · could not load the committer count from the accountant: 65 in bi-sync, all during the eu-1 outage
    Why: not established. No bi-sync deploy at the scheduled time; the committer-count errors came 11 minutes earlier, and whether they blocked the scheduler is unverified.
    Fix: Check the bi-sync scheduler logs for 03:25 CET and confirm credits_daily rows for 4 Nov | unowned · no ticket
The details reply. For the sync job, the cause is "not established".

For the stuck batches, Agent0 tied them to the database outage and noted that nobody had deployed anything near it. That ruled out the first thing most of us would have checked. For the sync job, it didn't know why the run was missing, and it said so. The cause was "not established", and the fix was to find out why the run never started.

Goran picked it up from there and replied in the thread.

Goran10:01 AM
Replayed both eu-1 usage-records-dlq windows (org-a, org-b; ClickHouse outage 04.11. 03:09 CET) after confirming neither had watermark rows. Both are now written once with matching counts, billing accepted org-a's report, and the DLQ is empty.
bi-sync credits crawler: silent croner skip at 02:25Z (REV-345), pod was healthy and every other job ran. Backfilled 3 Nov, 223 rows.
Goran's replies in the thread, three hours after the digest.

At 10:01 CET he confirmed that neither batch had been written, replayed both, and checked that the dead-letter queue was empty. Five minutes later he found that the scheduler had silently skipped the sync job. He filed a ticket and backfilled 223 rows.

A month of digests

Before the digest, Goran did this by hand every morning.

Every morning used to start with alerts and warnings, then a dig through the logs for context on each one. Getting all the details and having the fixes or PRs ready took anywhere from 30 minutes to 4 hours. Now the whole thing takes at most 20 minutes.Goran Čeko, Senior Backend Engineer, Dash0

The slow part was connecting the dots. He had to see every issue in one place, tell what needed action from what was only informational, and notice when one problem hit several organizations or services. The digest does that before he logs on. In its first month, 16 of 20 digests had something one of us had to act on, such as a usage evaluation that failed, a report that never reached the marketplace, a stuck invoice or a subscription that didn't move to its next state. Sometimes the same problem hit dozens of organizations at once.

An alert that cleared overnight no longer means the problem is gone.Goran Čeko, Senior Backend Engineer, Dash0

Set it up yourself

You need Dash0 with the Slack integration connected, and at least one check rule that sends failed checks somewhere.

Agent0 starts from the same template we use, with its schedule, prompt and guardrails. It suggests the Slack channels and check rules that fit your team, lets you pick, and hands you the automation to review and create. To see a digest before the next weekday morning, click Test automation on the automation's page.

The automation's run list with two manual test runs, using 0.5 and 0.2 credits.
Two test runs. The first used 0.5 credits, the second 0.2.

Our test runs cost under one credit each, far below the template's budget of 500 credits a week.

What we learned

  • Count damage as action. Our digest said nothing needed action on the morning it found two real problems, because it only counted checks that were still failing. The Fix: lines got the work done anyway. In the template, a finding that left damage behind counts as needing action, so the headline matches the lines under it.
  • Let it say "not established". The prompt allows Agent0 to admit it doesn't know a cause, and on the sync job it did. Paired with a step to find out why the run never started, that gave Goran a job to do. A confident wrong guess would have sent him the wrong way.
  • Name one action per finding. For the stuck batches, the step was to compare them with the accounting records, then replay them once. Someone can finish that before lunch, and Goran did.
  • Let Slack be the memory. The automation stores nothing between runs. It reads its previous digest and the replies under it, so a short reply is enough to correct it.
  • Don't force the post. An automation can require a tool call, such as posting to Slack. We turned that off for the digest. When a run correctly decides to post nothing, a required post would make the platform retry it, and the retry could post a duplicate.
  • Keep paging separate. The digest runs once a morning and wakes nobody up. Critical failed checks still need a notification that reaches a person within minutes.