eBPF Continuous Profiling released

Changelog

Recent changes and improvements to Dash0

Join us on a journey through Observability and OpenTelemetry as we improve our product. Stay connected with our Newsletter or RSS for fresh updates.

Explore your Kubernetes pods, live or historical

Kubernetes gets its own section under Infrastructure: a live map of every pod, a pod table that takes k9s keys, and a side panel that shows why a pod failed.

Kubernetes used to be one resource type among many on the Resources page. It now has its own section under Infrastructure, built around the question you open it with: which pods are unhappy, and why. The pages run live by default, and the same control rewinds them to any past window, so you can ask that question of the cluster as it is now or as it was when it broke.

The Kubernetes Map view, with pods grouped into two clusters and colored by health, and the Needs attention list beside it

Start wide, then zoom in

The Map draws every pod in every cluster, grouped by cluster and namespace and colored by health. Click a cluster to zoom into it, a namespace to go further, a pod to open it. Beside the map, Needs attention says what is wrong and how many pods it affects: failed, pending, in an unknown phase, not ready, near the CPU limit, near the memory limit.

The Table trades the overview for detail. Every pod is a row carrying its namespace, its age, its status, how often it has restarted, how much CPU and memory it is using, and how much of its limit that is. Group the rows by cluster, namespace or node, and filter them by name, cluster, namespace, node or IP.

The Kubernetes pod table sorted by status, with three pods in Error at the top above running pods, and namespace, age, restarts and CPU usage columns

The k9s keys, in the browser

The table takes the keys you already type. j and k move, g and Shift+G jump to the ends, Enter or d opens a pod, y its YAML, l its logs, e its events, o its node, p its parent workload, c copies the name and / filters. Shift+N, Shift+A, Shift+S, Shift+C and Shift+M sort by name, age, status, CPU and memory.

Why that pod failed

Open a pod and the side panel answers it without sending you anywhere else.

  • Containers gives each container's current state and, once object collection is on, its last termination, with the reason, exit code and signal.
  • Events sits next to the CPU, memory and network charts. Kubernetes records no event when a container is killed, so Dash0 reconstructs those rows from the container's status metrics and marks them as approximate.
  • Logs are filtered to the pod, and to one container when you pick one. A container that terminated without writing anything says so, and links to the event that explains it.
  • Traces group into operations with p50, p95 and error counts, and open in place.
  • Related resources walks the cluster, its nodes and the namespace's pods on them, and the workload down through its ReplicaSets.
The pod side panel for a failed pod, with the related resources tree showing other failing pods on its node, the hover card on its container in Error with the reason and exit code, and an events table whose first row is the reconstructed container termination

The YAML, as it is in the cluster

The YAML tab shows the object's manifest as it was last collected, stamped with when. An object that has since been deleted still shows its last known state.

Manifests reach Dash0 as log records from the OpenTelemetry Collector's k8sobjects receiver, so any collector you already run can send them. The Dash0 operator runs that receiver for you once you turn Kubernetes object collection on for the cluster and for the namespaces you want manifests from. Before anything leaves your cluster, it masks the values of environment variables and removes the annotations that kubectl and similar tools use to store a serialized copy of the whole manifest.

The YAML tab of a failed pod, showing its collected manifest with line numbers and an Observed timestamp

Live, or any moment you choose

The pages follow your cluster as it changes. Pause them when you want to read a row without it moving under you, or pick a past window, such as yesterday or this day last week, and see the cluster as it was then, with a banner saying so and one click back to live.

Workloads, Nodes and Storage are next.

Read more
Mateusz Wojciechowski

Mateusz Wojciechowski

Shani Shiri

Shani Shiri

Tomas Zaicevas

Tomas Zaicevas

Evgeni Wachnowezki

Evgeni Wachnowezki

Andrea Chomiak

Andrea Chomiak

Simo Paraschi

Simo Paraschi

Add automations to your check rules

Link existing automations or create one from a Dash0 template without leaving the check rule, and choose how the alert and the automation's result are delivered.

When a check fails, an automation can investigate it for you. Until now, you set one up on the Automations page and then had to work out which check rules it covered.

A check rule now has an Automations section after its Notifications. It lists the automations that run when the rule fails. You can link an existing automation, or create one from a Dash0 template, such as Investigate, without leaving alerting. Before you create it, the template panel shows how many runs and credits to expect each month. If an automation runs for every failed check or matches by label, you can stop it for a single rule. Linking works even on rules you can only read, as long as you can create automations.

For each pair of check rule and automation, you choose how the alert and the automation's result are delivered:

  • Same notification channel, when automation finishes: the alert goes out as usual, and the result follows as a new message. This is the default.
  • Hold off the alert until the automation completes: one alert goes out, with the result, when the run finishes.
  • Different notification channels: the alert goes to the check rule's channels, and the result goes to the automation's Send to.

A held alert is still sent if the run is skipped or fails. Results follow silences, so a silenced failed check does not leak into your channels.

See About Automations to learn more.

Read more

Bulk silence failed checks

Select failed checks and silence them together, up to 200 in one silence. The Silences Hub shows a bulk silence as one row.

Until now, a silence covered one failed check. When one problem failed many checks, you had to silence each one separately.

Now you can silence many failed checks at once:

  • Select and silence together. In the failed checks list, select the failed checks and click Silence. Choose how long, and add a reason, which is required. One silence covers up to 200 failed checks.
  • Review before you confirm. The dialog lists the failed checks that will be silenced. It flags the failed checks that have resolved, and Remove all takes them out. Failed checks that are already silenced show Silenced already and stay out of the silence unless you tick them.
  • See a bulk silence as one row. Its row in the Silences Hub shows how many failed checks it covers. Expand the row to see each failed check, and click one to open its detail page.
  • Remove a failed check from a silence. In the expanded row, hover over a failed check and click its trash icon, Remove from silence. Its notifications resume, and the other failed checks stay silenced. Removing the last failed check ends the silence.
  • Add to an active silence. Select failed checks and click Add to silence. Pick an active silence that covers failed checks, review the failed checks, and add them. The silence keeps its end time.
  • End silences for a selection. Select failed checks and click End silence. The dialog lists them under Will stop being silenced. Only the failed checks you leave ticked stop being silenced. A silence that also covers other failed checks keeps silencing them.
  • Silenced now or earlier. In the failed checks list, a failed check that a silence covers right now shows Silenced now. One that a silence covered earlier in the selected time range, but not now, shows Was silenced.
Silence failed checks dialog for five selected failed checks, with a duration, a required reason, a note that 1 failed check has resolved with a Remove all button, and one failed check tagged Silenced already

Read the docs: Silence Failed Checks

Read more

Latency SLOs on any histogram

Add a duration to an SLO and count only the requests that succeeded within it. It works on span metrics and any cumulative histogram, and you do not write any PromQL.

The Good events card of the SLO form for the demo-webshop-search service on dash0.spans.red. Duration is set to at most 300 ms, the success criteria is otel.span.status.code != ERROR, and the PromQL preview shows the histogram_fraction query the form writes.

A request that succeeds after eight seconds is still a bad experience, but an error-rate SLO counts it as good. To build a latency SLO in Dash0 until now, you wrote the PromQL yourself, with a bucket filter that had to match a bucket bound exactly or a histogram_fraction query.

The SLO form now has a Duration row in the Good events and Bad events cards. Pick a histogram metric, such as dash0.spans.red or http.server.request.duration, and type a duration like 300 ms. A request then counts as good only when it matches your success criteria and finishes within that time. The form writes the query and shows it as you type. It works with span metrics, OpenTelemetry histograms with fixed buckets, and native or exponential histograms.

See the documentation to set one up.

Read more

Start Agent0 automations from changes in Linear

Automations now start when issues, comments, projects, initiatives or releases change in Linear, so Agent0 picks up the work without anyone handing it over.

Work tracked in Linear still waits for someone to notice it, read it and start on it.

Automations can now start when something changes in Linear: an issue is created or updated, a comment, attachment or reaction is added, a project or initiative gets an update, or a release goes out. Narrow the trigger by team, project, label, status, author or name, with exact values or regular expressions. Anything that doesn't match is skipped. When an event matches, Agent0 reads the item and does what the automation asks, such as opening a pull request with a fix, building a dashboard, or posting a root-cause report to Slack.

For example, an issue trigger that listens for the In Progress status starts Agent0 as soon as your team picks up a ticket.

See the documentation to set one up.

Read more

Instrument pods instead of workloads with the Dash0 operator

The operator can now instrument pods at creation and leave your deployments untouched.

Starting with version 0.160.0, the Dash0 operator can instrument pods when they are created, instead of modifying the pod templates of deployments, stateful sets and other workloads.

To switch a namespace over, set spec.instrumentWorkloads.mode: pods in its Dash0 monitoring resource. Since the workload resources are never modified, tools like Argo CD, Flux or Terraform no longer see drift caused by the operator. Pods created by other controllers, for example Argo Rollouts, are instrumented as well.

The operator doesn't restart pods to instrument them: existing pods pick up the instrumentation the next time they are recreated. The mode is set per namespace, so you can switch namespaces one at a time. We recommend running two operator replicas (operator.replicaCount=2) so that the webhook stays available during operator updates.

For details, see the documentation.

Read more

Continuous profiling in Dash0

Find the code behind resource consumption with OpenTelemetry profiles. Filter by workload, compare profiles, and investigate CPU time, memory allocations, and other measurements.

Resource charts show which workloads consume resources. Profiles show the code paths behind that consumption.

Dash0 now has a Profiling Explorer for investigating OpenTelemetry profiles. Analyze CPU time, memory allocations, and other measurements in the same explorer. Dash0 supports all OpenTelemetry profile types; the types available in your dataset depend on what your profiler collects and sends.

Dash0 Profiling Explorer showing workload filters, an On-CPU time-series chart, and an aggregate flame graph

From workload to code

Filter profiles by workload attributes, such as Kubernetes namespace, deployment, or pod. Split the chart into series with Sum by to find the workloads contributing most to the selected measurement, then inspect their call stacks.

Choose the view that answers your question:

  • Flame Graph: Find expensive call paths across the selected samples. Reverse the call-stack order to bring leaf functions together across their callers and find the code doing the work directly.
  • Table: Search for a function and compare its Self contribution with its Total, which includes its callees. Values follow the selected profile's units, whether time, bytes, or counts.
  • Flame Chart: Inspect samples along a time axis and group them by attributes to investigate bursts or sustained activity.
  • Sandwich: Follow the sampled callers and callees around a selected frame to understand which operations lead into it and where it delegates work.

Compare profiles and focus on the difference

Choose A Baseline and B Comparison to compare datapoints over time or different workloads. A differential flame graph shows which call paths contribute less or more in B: green for less, red for more.

Separate baseline and comparison On-CPU charts above a differential flame graph with green and red frames

Use stack filters to keep relevant call paths, or frame filters to hide individual frames, such as runtime internals. These filters change the code view, not the stored profiles.

Compare the same profile type and units, and account for differences in traffic and workload scope before treating a change as a regression or an improvement.

Bring your OpenTelemetry profiles

On Kubernetes, the Dash0 Operator profiling integration can deploy an opt-in managed profiler or forward profiles from a separately deployed profiler. Profile collection must be enabled separately from tracing and metrics instrumentation.

Open Telemetry → Profiling, choose a dataset containing profiles, and select a time range and Profile type to start investigating.

Learn more about continuous profiling in Dash0.

Read more

Continuous profiling with the Dash0 Operator

The Dash0 Operator can now deploy and manage the Parca eBPF profiler for you and collect CPU profiles from your monitored workloads, without any code changes.

Traces tell you which request was slow. Profiles tell you which function burned the CPU time. Until now, getting profiles from Kubernetes into Dash0 meant deploying an eBPF profiler yourself and wiring it up to the collectors of the Dash0 Operator.

Starting with version 0.159.0, the Dash0 Operator does this for you. It runs the Parca agent as a DaemonSet, sends its profiles through the operator's collector on the same node, and exports them to Dash0.

What you get

  • No code changes: The profiler uses eBPF to sample the processes on each node. There is no library to add and no image to rebuild.
  • Broad runtime coverage: Native code such as Go, Rust and C/C++, as well as Java, Python, Node.js, .NET, Ruby, PHP, Perl, Erlang/Elixir and LuaJIT.
  • Kubernetes context: The operator enriches profiles with Kubernetes metadata, just like your traces, metrics and logs. Profiles follow the same namespace selection: the operator keeps workload profiles only from namespaces that have a Dash0Monitoring resource, and the filter and transform settings of these resources apply to profiles as well.
  • Readable native stack traces: The profiler uploads debug information of native binaries to Dash0 through a proxy managed by the operator, so that Dash0 can resolve native frames to function names. The proxy uses the same credentials as your Dash0 exports.

How do you try it?

Upgrade the Dash0 Operator for Kubernetes to v0.159.0 or later and set operator.ebpfProfilerEnabled=true when installing or upgrading the Helm chart. If you manage the Dash0OperatorConfiguration resource yourself instead of via Helm, also set spec.profiling.enabled and spec.profiling.ebpfProfiler.enabled to true there.

The profiler is opt-in and off by default. Sampling frequency, off-CPU sampling, the enabled runtimes, resources and scheduling are configurable on a self-managed Dash0OperatorConfiguration resource. See the documentation for all options.

Since the agent runs with privileged access to your nodes, review the security requirements before you enable it. Support for GKE Autopilot will follow in a later release.

Read more

AutoMerge approves your pull requests

AutoMerge scores every pull request with AI evaluators and approves the ones that clear your rules. It starts in Dry run, where it only comments its decision.

Coding agents made writing a change cheap. Reviewing it did not get cheaper, so pull requests wait for a person, including the routine ones nobody needed to read line by line.

AutoMerge, now in beta in Darkplane, takes that routine share off your reviewers. AI evaluators score each pull request against criteria you define. Each criterion returns a score from 0 to 100 with a rationale. When the weighted score reaches your threshold and no criterion falls below its floor, AutoMerge approves. Everything else goes to a person.

Darkplane AutoMerge score cards for a pull request routed to a human and an approved pull request

Every repository starts with three criteria: blast radius, evidence quality, and change footprint. Treat them as a starting point and fit them to your repository:

  • Add the questions your reviewers ask, and remove the ones that don't fit.
  • Write the prompt each evaluator follows, and choose the tools it may use to read the repository and the pull request.
  • Weight the criteria by what matters most, and set a floor that rejects a pull request on one criterion alone.
  • Tell the evaluators what the code doesn't show, such as which directories carry the highest consequences.

You decide how much it may do

Each repository runs in one of four modes: Off, Dry run, Auto-approve, or Auto-merge. Path rules give specific paths their own level. Mark migrations, infrastructure, or billing as Protected, and a pull request that touches them goes to a person without being scored. A pull request always runs at the least autonomous level of any file it touches.

AutoMerge fails closed. If a criterion can't produce a score, nothing is approved. It doesn't approve a commit whose CI is failing, and it never approves a pull request again after a reviewer dismissed its approval. Its check is advisory, so it blocks no merge unless you make it a required check in GitHub.

Start in Dry run

In Dry run, AutoMerge comments its decision on each pull request and approves or merges nothing. Dash0 grades every decision against what your reviewers did next and shows the result as Agreement with humans. Once its decisions match your reviewers, switch the repository to Auto-approve. Before you change criteria or the threshold, test the change against past pull requests on the Testing tab.

The AutoMerge overview of a repository in Auto-merge mode, with tiles for auto-approved pull requests, intervention rate, time to merge, and agreement with humans, charts of auto-approvals and criterion scores over time, a breakdown of which criterion decided each outcome, and the repository's path rules

AutoMerge works with repositories connected through the GitHub integration. Read Set Up AutoMerge to start your first Dry run, or see the AutoMerge overview.

Read more

Manage datasets and auth tokens as code

Manage Dash0 datasets and auth tokens with the Dash0 Terraform provider, the Dash0 CLI, or the Dash0 API, and keep them in the same repository as your dashboards and check rules.

Until now, you created datasets and auth tokens by hand in Dash0, even when the dashboards and check rules inside a dataset already lived in your repository.

You can now define both in YAML, as Dash0Dataset and Dash0AuthToken documents, and manage them with the Dash0 Terraform provider, the Dash0 CLI, or the Dash0 API.

How it works

A deleted dataset's dashboards, views, and check rules stay around for 30 days, and recreating the dataset with the same slug in that time brings them back.

Dash0 shows a token's secret once, when it creates the token, and the secret never changes. To rotate a token, create a second one, move your services to it, and delete the old one. Only an admin, or an auth token with the * action and no dataset or signal restriction, can manage auth tokens.

Dash0 Terraform provider

Since v1.20.0:

hcl
staging.tfresource "dash0_dataset" "staging" {
  dataset_yaml = <<-YAML
    apiVersion: dash0.com/v1alpha1
    kind: Dash0Dataset
    metadata:
      name: Staging
    spec:
      retention:
        logs: 60d
  YAML
}

resource "dash0_auth_token" "staging_ingest" {
  auth_token_yaml = <<-YAML
    apiVersion: dash0.com/v1alpha1
    kind: Dash0AuthToken
    metadata:
      name: Staging ingestion
    spec:
      actions:
        - ingest:*
      scope:
        datasets:
          restriction: restricted
          slugs:
            - ${dash0_dataset.staging.id}
      ingestion:
        defaultDataset: ${dash0_dataset.staging.id}
  YAML
}

output "staging_ingest_token" {
  value     = dash0_auth_token.staging_ingest.token
  sensitive = true
}

Terraform hides the sensitive token attribute in its output but writes the secret to your state in plain text, so lock down your state file.

Dash0 CLI

Datasets arrived in v1.19.0 and auth tokens in v1.20.0, both as experimental commands that need the --experimental (-X) flag:

sh
datasets-and-tokens.sh# apply creates datasets before the dashboards and check rules that live in them
dash0 -X apply -f staging-dataset.yaml
dash0 -X datasets list
dash0 -X datasets get staging -o yaml

# Auth tokens have their own commands, because apply has no safe place to put a new secret.
# The secret is shown once. In CI or a pipe, write it to a file instead.
dash0 -X auth-tokens create -f staging-ingest-token.yaml --secret-file "$RUNNER_TEMP/staging-ingest-token"
dash0 -X auth-tokens list

Read the docs: dash0_dataset, dash0_auth_token, Dash0 CLI commands

Read more

Review and apply Agent0's changes as artifacts

Agent0 now proposes dashboards, check rules, SLOs, automations and more as artifacts. Review each one, then apply it yourself.

Until now, when Agent0 wrote a check rule or an automation, it put the definition in the chat. Each change was another block of text, and nothing told you which one was current or whether it was already created.

Now Agent0 proposes changes as artifacts:

  • One card per change. Each proposal is a card in Agent0, in the full-page chat or the side panel you open next to any page. When Agent0 revises it, the card gets a new version instead of a new block.
  • You decide what changes. In a conversation, Agent0 does not change anything in Dash0 on its own. Click Create or Update on the card to apply it, in Dash0 or in Slack. Automations still act on their own, within the tools you allow them.
  • Preview before you apply. Preview opens the proposal in its usual editor, with the chat alongside.
  • No silent overwrites. If someone changed the entity after Agent0 read it, the update is refused. Ask Agent0 to redo its proposal against the latest version.
  • Everything in one place. The Artifacts tab in the Agent0 sidebar lists every artifact in the conversation.
  • Auto-apply in the side panel. On an SLO, check rule, dashboard or automation page, Agent0 makes the change in the open editor for you, as an unsaved draft. Review it, then press Save. You can turn this off in the + menu of the prompt. See Use the Integrated Experience.
  • Change several dashboard panels at once. On a dashboard, hold Shift+Cmd (Shift+Ctrl on Windows and Linux) and click panels to add them to Agent0's context, then ask Agent0 to change them. The card lists the panels each revision added, removed or edited.

Artifacts are available today for dashboards, check rules, SLOs, synthetic checks, automations, silences, recording rules and custom skills, with more types on the way.

Read the docs: Use Agent0

Read more

Manage Agent0 custom skills as code

Keep your Agent0 custom skills in Git and roll them out with the Dash0 Terraform provider or the Dash0 CLI, so every change to a skill is versioned and reviewed.

Custom skills hold the procedures only your team knows, but until now they could only be written in Dash0. A change to a skill had no review, no history, and no way to roll it out the same way to every organization.

You can now define a custom skill as a Dash0Skill YAML document, keep its SKILL.md and supporting files in Git, and apply them with the Dash0 Terraform provider or the Dash0 CLI.

How it works

A skill is identified by its dash0.com/origin: Dash0 creates it the first time and updates it after that. Custom skills belong to the organization, so no dataset is needed. The spec.permissions list sets who can use the skill, and defaultEnabled: false ships it turned off.

A skill managed by Terraform can be read and deleted in Dash0, but only your configuration changes its content.

Dash0 Terraform provider

Since v1.23.0:

hcl
custom-skill.tfresource "dash0_custom_skill" "incident_response" {
  custom_skill_yaml = file("${path.module}/skills/incident-response.yaml")
}

Adopt a skill you already have with terraform import dash0_custom_skill.incident_response <origin-or-id>.

Dash0 CLI

Since v1.20.0:

sh
custom-skills-cli.sh# Create or update a skill from a YAML file
dash0 apply -f incident-response.yaml

# List skills, and export one ready to edit and apply again
dash0 -X agent0 skills list -o wide
dash0 -X agent0 skills get <origin-or-id> -o yaml

Read the docs: Manage Agent0 Skills as Code

Read more