Continuous profiling is now GA

Changelog

Recent changes and improvements to Dash0

Join us on a journey through Observability and OpenTelemetry as we improve our product. Stay connected with our Newsletter or RSS for fresh updates.

Continuous profiling in Dash0

Find the code behind resource consumption with OpenTelemetry profiles. Filter by workload, compare profiles, and investigate CPU time, memory allocations, and other measurements.

Resource charts show which workloads consume resources. Profiles show the code paths behind that consumption.

Dash0 now has a Profiling Explorer for investigating OpenTelemetry profiles. Analyze CPU time, memory allocations, and other measurements in the same explorer. Dash0 supports all OpenTelemetry profile types; the types available in your dataset depend on what your profiler collects and sends.

Dash0 Profiling Explorer showing workload filters, an On-CPU time-series chart, and an aggregate flame graph

From workload to code

Filter profiles by workload attributes, such as Kubernetes namespace, deployment, or pod. Split the chart into series with Sum by to find the workloads contributing most to the selected measurement, then inspect their call stacks.

Choose the view that answers your question:

  • Flame Graph: Find expensive call paths across the selected samples. Reverse the call-stack order to bring leaf functions together across their callers and find the code doing the work directly.
  • Table: Search for a function and compare its Self contribution with its Total, which includes its callees. Values follow the selected profile's units, whether time, bytes, or counts.
  • Flame Chart: Inspect samples along a time axis and group them by attributes to investigate bursts or sustained activity.
  • Sandwich: Follow the sampled callers and callees around a selected frame to understand which operations lead into it and where it delegates work.

Compare profiles and focus on the difference

Choose A Baseline and B Comparison to compare datapoints over time or different workloads. A differential flame graph shows which call paths contribute less or more in B: green for less, red for more.

Separate baseline and comparison On-CPU charts above a differential flame graph with green and red frames

Use stack filters to keep relevant call paths, or frame filters to hide individual frames, such as runtime internals. These filters change the code view, not the stored profiles.

Compare the same profile type and units, and account for differences in traffic and workload scope before treating a change as a regression or an improvement.

Bring your OpenTelemetry profiles

On Kubernetes, the Dash0 Operator profiling integration can deploy an opt-in managed profiler or forward profiles from a separately deployed profiler. Profile collection must be enabled separately from tracing and metrics instrumentation.

Open Telemetry → Profiling, choose a dataset containing profiles, and select a time range and Profile type to start investigating.

Learn more about continuous profiling in Dash0.

Read more

Continuous profiling with the Dash0 Operator

The Dash0 Operator can now deploy and manage the Parca eBPF profiler for you and collect CPU profiles from your monitored workloads, without any code changes.

Traces tell you which request was slow. Profiles tell you which function burned the CPU time. Until now, getting profiles from Kubernetes into Dash0 meant deploying an eBPF profiler yourself and wiring it up to the collectors of the Dash0 Operator.

Starting with version 0.159.0, the Dash0 Operator does this for you. It runs the Parca agent as a DaemonSet, sends its profiles through the operator's collector on the same node, and exports them to Dash0.

What you get

  • No code changes: The profiler uses eBPF to sample the processes on each node. There is no library to add and no image to rebuild.
  • Broad runtime coverage: Native code such as Go, Rust and C/C++, as well as Java, Python, Node.js, .NET, Ruby, PHP, Perl, Erlang/Elixir and LuaJIT.
  • Kubernetes context: The operator enriches profiles with Kubernetes metadata, just like your traces, metrics and logs. Profiles follow the same namespace selection: the operator keeps workload profiles only from namespaces that have a Dash0Monitoring resource, and the filter and transform settings of these resources apply to profiles as well.
  • Readable native stack traces: The profiler uploads debug information of native binaries to Dash0 through a proxy managed by the operator, so that Dash0 can resolve native frames to function names. The proxy uses the same credentials as your Dash0 exports.

How do you try it?

Upgrade the Dash0 Operator for Kubernetes to v0.159.0 or later and set operator.ebpfProfilerEnabled=true when installing or upgrading the Helm chart. If you manage the Dash0OperatorConfiguration resource yourself instead of via Helm, also set spec.profiling.enabled and spec.profiling.ebpfProfiler.enabled to true there.

The profiler is opt-in and off by default. Sampling frequency, off-CPU sampling, the enabled runtimes, resources and scheduling are configurable on a self-managed Dash0OperatorConfiguration resource. See the documentation for all options.

Since the agent runs with privileged access to your nodes, review the security requirements before you enable it. Support for GKE Autopilot will follow in a later release.

Read more

AutoMerge approves your pull requests

AutoMerge scores every pull request with AI evaluators and approves the ones that clear your rules. It starts in Dry run, where it only comments its decision.

Coding agents made writing a change cheap. Reviewing it did not get cheaper, so pull requests wait for a person, including the routine ones nobody needed to read line by line.

AutoMerge, now in beta in Darkplane, takes that routine share off your reviewers. AI evaluators score each pull request against criteria you define. Each criterion returns a score from 0 to 100 with a rationale. When the weighted score reaches your threshold and no criterion falls below its floor, AutoMerge approves. Everything else goes to a person.

Darkplane AutoMerge score cards for a pull request routed to a human and an approved pull request

Every repository starts with three criteria: blast radius, evidence quality, and change footprint. Treat them as a starting point and fit them to your repository:

  • Add the questions your reviewers ask, and remove the ones that don't fit.
  • Write the prompt each evaluator follows, and choose the tools it may use to read the repository and the pull request.
  • Weight the criteria by what matters most, and set a floor that rejects a pull request on one criterion alone.
  • Tell the evaluators what the code doesn't show, such as which directories carry the highest consequences.

You decide how much it may do

Each repository runs in one of four modes: Off, Dry run, Auto-approve, or Auto-merge. Path rules give specific paths their own level. Mark migrations, infrastructure, or billing as Protected, and a pull request that touches them goes to a person without being scored. A pull request always runs at the least autonomous level of any file it touches.

AutoMerge fails closed. If a criterion can't produce a score, nothing is approved. It doesn't approve a commit whose CI is failing, and it never approves a pull request again after a reviewer dismissed its approval. Its check is advisory, so it blocks no merge unless you make it a required check in GitHub.

Start in Dry run

In Dry run, AutoMerge comments its decision on each pull request and approves or merges nothing. Dash0 grades every decision against what your reviewers did next and shows the result as Agreement with humans. Once its decisions match your reviewers, switch the repository to Auto-approve. Before you change criteria or the threshold, test the change against past pull requests on the Testing tab.

The AutoMerge overview of a repository in Auto-merge mode, with tiles for auto-approved pull requests, intervention rate, time to merge, and agreement with humans, charts of auto-approvals and criterion scores over time, a breakdown of which criterion decided each outcome, and the repository's path rules

AutoMerge works with repositories connected through the GitHub integration. Read Set Up AutoMerge to start your first Dry run, or see the AutoMerge overview.

Read more

Manage datasets and auth tokens as code

Manage Dash0 datasets and auth tokens with the Dash0 Terraform provider, the Dash0 CLI, or the Dash0 API, and keep them in the same repository as your dashboards and check rules.

Until now, you created datasets and auth tokens by hand in Dash0, even when the dashboards and check rules inside a dataset already lived in your repository.

You can now define both in YAML, as Dash0Dataset and Dash0AuthToken documents, and manage them with the Dash0 Terraform provider, the Dash0 CLI, or the Dash0 API.

How it works

A deleted dataset's dashboards, views, and check rules stay around for 30 days, and recreating the dataset with the same slug in that time brings them back.

Dash0 shows a token's secret once, when it creates the token, and the secret never changes. To rotate a token, create a second one, move your services to it, and delete the old one. Only an admin, or an auth token with the * action and no dataset or signal restriction, can manage auth tokens.

Dash0 Terraform provider

Since v1.20.0:

hcl
staging.tfresource "dash0_dataset" "staging" {
  dataset_yaml = <<-YAML
    apiVersion: dash0.com/v1alpha1
    kind: Dash0Dataset
    metadata:
      name: Staging
    spec:
      retention:
        logs: 60d
  YAML
}

resource "dash0_auth_token" "staging_ingest" {
  auth_token_yaml = <<-YAML
    apiVersion: dash0.com/v1alpha1
    kind: Dash0AuthToken
    metadata:
      name: Staging ingestion
    spec:
      actions:
        - ingest:*
      scope:
        datasets:
          restriction: restricted
          slugs:
            - ${dash0_dataset.staging.id}
      ingestion:
        defaultDataset: ${dash0_dataset.staging.id}
  YAML
}

output "staging_ingest_token" {
  value     = dash0_auth_token.staging_ingest.token
  sensitive = true
}

Terraform hides the sensitive token attribute in its output but writes the secret to your state in plain text, so lock down your state file.

Dash0 CLI

Datasets arrived in v1.19.0 and auth tokens in v1.20.0, both as experimental commands that need the --experimental (-X) flag:

sh
datasets-and-tokens.sh# apply creates datasets before the dashboards and check rules that live in them
dash0 -X apply -f staging-dataset.yaml
dash0 -X datasets list
dash0 -X datasets get staging -o yaml

# Auth tokens have their own commands, because apply has no safe place to put a new secret.
# The secret is shown once. In CI or a pipe, write it to a file instead.
dash0 -X auth-tokens create -f staging-ingest-token.yaml --secret-file "$RUNNER_TEMP/staging-ingest-token"
dash0 -X auth-tokens list

Read the docs: dash0_dataset, dash0_auth_token, Dash0 CLI commands

Read more

Review and apply Agent0's changes as artifacts

Agent0 now proposes dashboards, check rules, SLOs, automations and more as artifacts. Review each one, then apply it yourself.

Until now, when Agent0 wrote a check rule or an automation, it put the definition in the chat. Each change was another block of text, and nothing told you which one was current or whether it was already created.

Now Agent0 proposes changes as artifacts:

  • One card per change. Each proposal is a card in Agent0, in the full-page chat or the side panel you open next to any page. When Agent0 revises it, the card gets a new version instead of a new block.
  • You decide what changes. In a conversation, Agent0 does not change anything in Dash0 on its own. Click Create or Update on the card to apply it, in Dash0 or in Slack. Automations still act on their own, within the tools you allow them.
  • Preview before you apply. Preview opens the proposal in its usual editor, with the chat alongside.
  • No silent overwrites. If someone changed the entity after Agent0 read it, the update is refused. Ask Agent0 to redo its proposal against the latest version.
  • Everything in one place. The Artifacts tab in the Agent0 sidebar lists every artifact in the conversation.
  • Auto-apply in the side panel. On an SLO, check rule, dashboard or automation page, Agent0 makes the change in the open editor for you, as an unsaved draft. Review it, then press Save. You can turn this off in the + menu of the prompt. See Use the Integrated Experience.
  • Change several dashboard panels at once. On a dashboard, hold Shift+Cmd (Shift+Ctrl on Windows and Linux) and click panels to add them to Agent0's context, then ask Agent0 to change them. The card lists the panels each revision added, removed or edited.

Artifacts are available today for dashboards, check rules, SLOs, synthetic checks, automations, silences, recording rules and custom skills, with more types on the way.

Read the docs: Use Agent0

Read more

Manage Agent0 custom skills as code

Keep your Agent0 custom skills in Git and roll them out with the Dash0 Terraform provider or the Dash0 CLI, so every change to a skill is versioned and reviewed.

Custom skills hold the procedures only your team knows, but until now they could only be written in Dash0. A change to a skill had no review, no history, and no way to roll it out the same way to every organization.

You can now define a custom skill as a Dash0Skill YAML document, keep its SKILL.md and supporting files in Git, and apply them with the Dash0 Terraform provider or the Dash0 CLI.

How it works

A skill is identified by its dash0.com/origin: Dash0 creates it the first time and updates it after that. Custom skills belong to the organization, so no dataset is needed. The spec.permissions list sets who can use the skill, and defaultEnabled: false ships it turned off.

A skill managed by Terraform can be read and deleted in Dash0, but only your configuration changes its content.

Dash0 Terraform provider

Since v1.23.0:

hcl
custom-skill.tfresource "dash0_custom_skill" "incident_response" {
  custom_skill_yaml = file("${path.module}/skills/incident-response.yaml")
}

Adopt a skill you already have with terraform import dash0_custom_skill.incident_response <origin-or-id>.

Dash0 CLI

Since v1.20.0:

sh
custom-skills-cli.sh# Create or update a skill from a YAML file
dash0 apply -f incident-response.yaml

# List skills, and export one ready to edit and apply again
dash0 -X agent0 skills list -o wide
dash0 -X agent0 skills get <origin-or-id> -o yaml

Read the docs: Manage Agent0 Skills as Code

Read more

Start an eval from a template

Pick one of 28 eval templates instead of writing an eval from scratch. Each template comes with span filters, a sample rate, a judge model, and a judge prompt, and you can change any of it before you save.

A blank eval form asks a lot. You have to know which spans to judge, how many to sample, which model should judge them, and how to write a rubric it applies the same way every time. Most teams know what worries them, like hallucinations or users getting fed up, but not how to turn that into a prompt.

Eval templates give you the prompt. Click Create new on the Evals page to open the catalog, pick a template, and the eval editor opens with everything filled in.

The eval template catalog with a search field, category toggles showing 28 templates across Top picks, Response quality, Grounding / RAG, User experience, Agents and tools, and Safety and security, and the Top picks group with Correctness, Hallucination, User frustration, Tool result handling, and Harmful content, followed by the first row of the Response quality category

What's in the catalog. 28 templates in five categories, from Hallucination and Tool selection to User frustration and PII leakage. Top picks gives you a place to start.

What a template fills in. Everything the eval form asks for: span filters, a 20% sample rate, the judge model and prompts, and the output type. Each system prompt spells out what counts and what doesn't. The Hallucination judge, for example, doesn't count a clarifying question or a faithful summary of the context as a hallucination. Change whatever doesn't fit your agents, then save.

What you end up with. An ordinary eval. Edit, disable, or delete it like one you wrote yourself. A later change to the template never changes your eval, so its scores stay comparable over time.

Read Eval Templates to see every template and how to start from one.

Read more

Peek into your Kubernetes clusters with Agent0

With the new Agent0 Kubernetes integration, Agent0 can read the state of your Kubernetes clusters during an investigation, and get from a symptom to its root cause without you copying kubectl output into the chat.

Telemetry tells Agent0 that a pod was OOM-killed, that a rollout is stuck, or that a service stopped receiving traffic. It does not tell Agent0 why. The answer is often in the cluster itself: a memory limit that is too low, a readiness probe that times out too early, a service selector that matches nothing, a network policy that drops the traffic. Until now, finding that meant switching to a terminal, running kubectl yourself, and pasting the output into the chat.

With the Agent0 Kubernetes integration provided by the Dash0 Operator, Agent0 can look for itself. During an investigation, it reads the state of your cluster, the same way you would, and connects what it finds with your logs, metrics and traces. It can inspect namespaces that send no telemetry yet, work across several clusters in a single investigation.

Agent0 finds the Service selector mismatchThe expanded kubectl get deploy,rs,po output with Agent0's reasoning
Agent0 finds the Service selector mismatch via kubectl.

Read-only, and you stay in control

Agent0 can look, but never touch. The integration only allows reading, so Agent0 cannot change, restart or delete anything in your cluster, and it never gets a shell into your containers. Secrets are off limits, and credentials are removed from every response before it leaves your cluster. You decide which resources Agent0 may see, and you can narrow or widen that at any time.

The connection is established from inside your cluster. There is no need to expose your Kubernetes API server, open a firewall port, or hand any cluster credentials to Dash0.

Opt in when you are ready

The integration is disabled by default. Nothing changes in your cluster until you explicitly turn it on in the Dash0 Operator (version 0.159.0 or later).

Not using the Dash0 Operator to collect telemetry? You can run it just for the Agent0 Kubernetes integration, alongside whatever you already use to send telemetry to Dash0.

Read the documentation to connect your first cluster.

Read more

AI SDLC Insights now supports Amp

Dash0's AI SDLC Insights now supports Amp, giving teams more complete visibility into AI coding activity across their development tools.

Dash0's AI SDLC Insights now supports Amp, giving teams more complete visibility into AI coding activity across their development tools.

With this new integration, AI coding sessions from Amp, locally and in Orbs, are included in Dash0's cost tracking, developer adoption metrics, and AI-assisted pull request analysis. Whether your team uses Claude Code, Cursor, Codex, GitHub Copilot, OpenCode, or Amp, you can track AI usage and its impact in one place.

To get started, install and configure the Dash0 plugin for Amp from the Dash0 agent plugin repository, then start an Amp session.

Check out the Integration Hub for detailed setup instructions for Amp.

Read more

Keep your teams in sync with your identity provider

Dash0 now turns the groups in your identity provider into Dash0 teams and keeps their membership in sync, so team-based dataset access follows your directory.

Until now, Dash0 teams lived apart from your identity provider. When someone joined or left a group in your directory, an admin had to repeat the change in Dash0, and the dataset access you granted through teams drifted from who actually belonged to each team.

Dash0 now turns the groups in your identity provider into Dash0 teams and keeps their membership in sync.

The Teams page with SCIM synchronization turned on, showing two teams synced from SCIM groups next to a team created in Dash0

Turning it on. If your organization has an enterprise connection with SSO and Directory Sync, an admin can enable team sync. You see every group and its members before any team is created, and the teams appear within a few minutes.

What stays in sync. People who join or leave a group join or leave the matching team as soon as your identity provider sends the change. This includes people who have not signed in to Dash0 yet. A renamed group renames its team.

What it changes. Each group becomes a new team, and teams you already have in Dash0 keep their own members. The sync manages membership only, so you grant dataset access to synced teams as you would to any other team. A deprovisioned user leaves every synced team but stays in your organization until you remove them.

Once Dash0 enables self-service SSO for your organization, you set up SSO and Directory Sync yourself.

Read the docs: Team Sync from SCIM Groups

Read more

Automatic instrumentation for .NET 6 and 7

The Dash0 Operator now supports automatic instrumentation on .NET 6 and 7 thanks to the new Dash0 OpenTelemetry distribution for .NET.

On a near-black ground, a dark circle low in the frame carries the OpenTelemetry mark beside the .NET wordmark, with five spokes fanning up and out of it, one per .NET runtime. Three of them — .NET 8 straight up, .NET 9 and .NET 10 to the right — are drawn in gray and land directly on their runtime. The two on the left, .NET 7 and .NET 6, run only half way before reaching a badge carrying the Dash0 mark and the word Dash0; the rest of each of those spokes is drawn in Dash0 orange, and the runtime at the end of it — drawn exactly like the other three — grows out from behind that badge.

The Dash0 Operator has supported zero-touch auto-instrumentation for .NET workloads for a while, via the upstream OpenTelemetry .NET Instrumentation, which currently supports .NET 8+.

However, upstream support will soon be restricted to .NET 10 and 11 only, as .NET 8 and 9 will join 6 and 7 in end of life (EOL) in November 2026.

From our vantage point, we see that the industry has not yet moved away from .NET 6 through 9, so those workloads should not stop being observable.

How do you try it?

To enable it, upgrade the Dash0 Operator for Kubernetes to v0.159.0 or later.

Read more

Rage clicks show where your users got stuck

The Dash0 Web SDK now detects rage clicks, bursts of repeated clicks on the same spot, in recorded sessions. Each one is marked in the session replay, so you can find where users got stuck and see what happened.

An error tells you something broke. Most of the time when a user gets stuck, nothing breaks: a button gives no feedback, so they click it again, and again, and leave. Your error tracking stays quiet, and you only hear about it if someone files a ticket.

The Dash0 Web SDK now flags that moment as a rage click: three or more clicks on the same spot, each within a second of the last.

Session replay of an Astronomy Shop cart where shipping options failed to load, with a rage click marked on the replay track: 4 rage clicks on Try again.

Turning it on. Nothing to do. Rage clicks are detected in every session the SDK records, so each one comes with the replay that shows it. Sessions without a replay have no rage clicks. If three clicks in a second is too eager or too strict for your website, change it in the Web SDK configuration.

Finding them. Filter your sessions by rage clicks to list every session where a user got stuck. Each rage click is marked in the replay, so you can jump straight to the moment and see what the page did, or did not do.

What leaves the browser. Where the clicks landed, on which page and element, how many there were and how fast, and the element's label, such as "Place order". For form fields, the label is the field's name, never what the user typed. Anything you mask or block sends no label.

Read Watch a Session Replay to find and watch sessions with rage clicks. For everything a rage click carries, see Understand Web Event Types.

Read more