Recent changes and improvements to Dash0

It was a good idea, but we did not execute it right: we saw users struggle with "going back” to larger timeframes, and feeling stuck. Internally, we “affectionately” called this user-experience issue "lost in time."

We went back to the drawing board, and think we found a nice solution: Now, the lists of spans and logs in the Logging and Tracing views filter themselves as soon as a selection is made. You can drill down in a subset of the raw telemetry, without losing the global context. No additional clicks, no "lost in time". Yet, you still have the option to zoom if you want to!

We recently announced the availability of custom filter rules in the operator as well as a similar filter mechanism in our platform, called Spam filters, that lets you discard useless telemetry in the Dash0 backend.

Today these capabilities are complemented by a new operator feature that allows you to transform telemetry collected by the operator with custom rules before it is sent to Dash0. In this context, filtering is about dropping entire objects (spans, log records, metrics) according to specific rules, but transforming telemetry goes one step further and gives you even more flexibility and power. Here are some things that you can do with telemetry transformations:

• Remove span attributes without discarding the span as a whole (or metric attributes, log record attributes, …).

• Limit the number of attributes on objects such as spans or metric data points.

• Limit the length of attributes.

• Modify resource attributes.

• Add attributes, based on the value of other attributes and conditions.

• Change the value of attributes (think regex replacements).

• Change the type of metrics (e.g. turn a sum into a gauge)

• …and probably a gazillion other things.

As usual, the Dash0 operator allows you to configure these rules per Kubernetes namespace. Here is an example of a Dash0 monitoring resource that configures a set of transformations:

dash0-monitoring.yaml
1234567891011121314151617181920212223242526272829
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource

spec:
 
  transform:
    trace_statements:

      # limit span attributes generally to 4096 characters
      - 'truncate_all(span.attributes, 4096)'

      # remove sensitive data from collected command line invocations 
      - replace_pattern(span.attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***")

      # remove query parameters (using the advanced config style here)
      - conditions:
        - 'span.attributes["http.target"] != nil'
        statements:
        - 'replace_pattern(span.attributes["http.target"], "customerId=.*", "customerId=xxx")'

    log_statements:
     
      # change the log level based on conditions
      - conditions:
        - 'log.body == "request failed"'
        statements:
        - 'set(log.severity_text, "FAIL")'

The configuration syntax is deliberately chosen to be identical to what the OpenTelemetry transform processor allows, supporting both the basic flat statement style as well as the advanced configuration style that supports adding conditions to a single transformation. This allows you to copy and paste existing transformation rules that you already have, or use examples you find online as-is.

For more details on telemetry transformations, check out the operator documentation.

Analysis methods

Besides comparing a subset of spans with all the rest in the global timeframe, it gives you ways to:

• Analyze attribute values: analyze how attributes and their values cluster across all the available spans. Maybe you are not looking to troubleshoot issues, but rather understand the reality of your data.

• Compare with the other spans in the same timeframe: this method is very good to analyze outliers, especially in terms of duration.

• Compare with everything *before *the selection in the global timeframe: this is an excellent way to analyze the first occurrence of spikes or error clusters.

Triage is in Beta

This is the first iteration of Dash0 triage. It is available in open beta for Tracing, which is the signal with the largest amount of metadata on average.

At time of writing this blog, we know of various cosmetic bugs and UX details that need improvement, but we don’t want to wait to learn with and from you and your use cases. Also, we are working hard on the performance (sifting on a lot of data the way Triage does requires optimizations), and until we are happy with the outcome, you can use Triage only on timeframes of up to 24 hours.

Triage will also come for Logging and Metrics when we feel we have confirmation through your feedback that we nailed user experience. Indeed, expect the user experience to be tuned and improved in the next few days and weeks.

To help you curb unnecessary telemetry, we have introduced Spam Filters. With two clicks, you can define filtering rules, executed by the Dash0 ingestion pipeline, to drop logs, metric data points and spans before Dash0 charges you for it.

(You read it correctly: there is no extra charge for spam filter rules, and data that is dropped by them, does not count against your Dash0 bill.)

Any combination of filters you can specify in the Logging, Tracing or Metrics pages can be synthesized as Spam filters. You can also export those filters as OpenTelemetry Collector configurations using the filter processor and OTTL (which is pretty much how we implemented it in our ingestion pipeline), so that you can deploy the same filters in your infrastructure and be doubly sure.

The Spam filters are listed in your dataset configurations, and different datasets can specify different filters.

After you create a Spam filter, matching telemetry starts getting dropped in a few seconds, but you might get a few last “stragglers” in. Similarly, after you remove a Spam filter, it will take a few seconds for the change to take effect. Telemetry that has been dropped by a Spam filter is not recoverable by Dash0.

In a later update, we will add tracking for just how much data you dropped. (You know, for bragging rights and so.)

Oh, by the way, we added similar functionality to drop telemetry in the Dash0 operator.

Note about spans: Spam filters drop single spans, you might end up with incomplete traces (unlike what you do with client-side sampling).

In the same vein as the Spam filters you find in Dash0, we exposed the possibility of configuring the OpenTelemetry Collector filter processor via the Dash0Monitoring resource in the Dash0 operator.

With the new filter capabilities of the Dash0 operator, you can discard some telemetry before it even leaves your cluster:

dash0-monitoring.yaml
123456789101112131415161718192021222324
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource
spec:
  filter:
    traces:
      span:
      - 'attributes["http.route"] == "/ready"'
      - 'attributes["http.route"] == "/metrics"'
      spanevent:
      - 'attributes["grpc"] == true'
      - 'IsMatch(name, ".*grpc.*")'
    metrics:
      metric:
      - 'name == "k8s.replicaset.available"'
      - 'name == "k8s.replicaset.desired"'
      datapoint:
      - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
      - 'resource.attributes["service.name"] == "my_service_name"'
    logs:
      log_records:
      - 'IsMatch(body, ".*password.*")'
      - 'severity_number < SEVERITY_NUMBER_WARN'

The configurations above will apply only to telemetry coming from the namespace in which this Dash0Monitoring resource resides.

You can also export the OpenTelemetry Transformation Language (OTTL) expressions from your Spam filters in Dash0, and apply it to the relevant namespaces in a “belt and suspenders” kind of fashion.

Having the option to filter telemetry both at the Dash0 backend as well as directly in your Kubernetes cluster might seem redundant, but it is actually useful for a couple of reasons:

• Using the spam filter in Dash0 is a quick and easy way to remove telemetry that has little or no value, and to reduce your Dash0 bill.

• However, if some components in your cluster really produce a lot of noisy telemetry, you would still pay for egress as the data is sent to Dash0, only to have Dash0 discard it immediately.

• Filtering directly in your cluster enables you to optimize your cloud costs further by removing those egress costs as well.

• Filtering in your cluster can also help remove sensitive information before it leaves your cluster – the log record filter above which drops lines with “password” is an example for that. (Ideally, telemetry with sensitive information wouldn’t even be produced at all, but mistakes happen.)

Note: You should be able to reuse the configurations you may already have (or find on the Internet or generate via AI) for the filter processor of the OpenTelemetry Collector.

• Effortless Aggregation: Now, you can easily group your logs and spans based on any attribute, allowing you to quickly identify patterns and anomalies.

• Contextual Insights: For each group, you'll get clear statistics to guide your workflow.

• Seamless Drill-Down: Need to investigate a specific group further? Simply click to drill down and explore the detailed logs or spans within that group.

But sometimes, you want to focus just on one aspect, like error percentage or duration percentiles. You already could today in many visualizations of Dash0, like the Services list and tables, but now we brought it to the Tracing view as well in the shape of the RED Charts.

**The RED Charts are an alternative visualization **alongside our Tracing Heatmap within the Tracing Explorer. This powerful combination balances detailed analysis with actionable clarity:

• Heatmaps reveal nuanced latency patterns and request distributions over time

• RED Charts highlight systemic trends in request volume, errors, and durations.

We have introduced a series of enhancements to our email notifications to allow you to get more insights into failed checks at a glance and improve control over the notifications you receive.

Affected Resource Breakdown

Our email notifications have been redesigned to include the same familiar affected resource breakdowns previously available only within the product. This update ensures you receive these insights directly in your inbox even before you open the failed check in Dash0.

Highlighting Errors and Closing Reasons

Emails now highlight errors and closing reasons, ensuring they behave similarly to other integrations, such as our Slack Bot.

Multiple Email Recipients

In response to user feedback, we've introduced the ability to configure multiple email recipients within a single notification channel. This features simplifies collaboration by ensuring that all relevant stakeholders receive timely updates, keeping your team synchronized and informed.

Notification Frequency Controls

Building upon recent updates, you now have greater control over the frequency of periodic reminders for unresolved failed checks. This per-notification channel setting allows you to tailor alert intervals to your operational needs, reducing notifications fatigue and enhancing focus on critical issues.

Dash0 has introduced a new Slack Bot integration to enhance real-time alerting within your team's Slack workspace. This feature allows you to receive immediate notifications directly in your chosen Slack channels, ensuring that failed checks are promptly addressed.

Key Features

• Instant Notifications: Receive up-to-date notification for failed checks directly in your specified Slack channels, facilitating swift responses to potential problems.

• Threaded Discussions: Each alert initiates a Slack thread, maintaining organized discussions and tracking the history and evolution of incidents in a centralized location.

• Quick Access Links: Hyperlinks embedded within check rule annotations are parsed and prominently displayed at the top of each alert, providing immediate access to relevant resources and dashboards for efficient troubleshooting.

Configuration Details:

OpenTelemetry semantic conventions regularly gets updated, where attributes or even metric names get changed. For example, http.status_code got updated to http.response.status_code for better consistency with other attributes.

For engineers to keep all names up-to-date, this can be very cumbersome work and where it is easy to miss some instrumentation sources, resulting in inconsistent attributes. This makes querying difficult, as not all signals will get in- or excluded as expected. Also Views or Dashboards might not work as expected.

Dash0 now supports executing the migrations as defined by the OpenTelemetry schemas itself. Via settings, the user can pick whether to always update to latest, or to one specific version from the list:

A specific version can be pinned to prevent e.g. breaking dashboards in the future, when new migrations become available. Signals will never get downgraded to a specified version.

For North American customers, this means an even snappier Dash0 experience, and USA-based data residency.

How to select your region

There is now a Region Selector as part of the organization creation step, which allows you selecting the region you would like to be deployed in.

Currently supported are European Union and United States

Keeping things simple for our end users is one of the core values of Dash0. After reviewing our pricing, we realized we could make it even simpler.

From the start, our price is based on counting metric data points, spans and span events, and log records. This allows you to send us your telemetry irrespective of how much metadata you have on it, and we actively encourage you to send all the metadata you think necessary or useful, irrespective of how many GB it is. We also do not have per-seat pricing: we want everybody in your team to be able to enjoy and benefit from Dash0.

Up until today, we had a $50 subscription fee, which included 100 million metric data points, 25 million spans and span events, and 25 million log records. The $50 wasn’t chosen at random: it is the amount you’d pay for sending that much data in a month.

Today, we are removing our base subscription fee. Effective midnight of Friday, February 7th 2025, Dash0’s pricing works purely consumption-based using the same rates as before:

• $0.20 for 1 million metric data points

• $0.60 for 1 million spans and span events

• $0.60 for 1 million log records

For most Dash0 users, the amount due in the bill will be the same. But the bill itself will be easier to read, and the pricing simpler to understand.

And for users who have been sending less telemetry than what was included in the base subscription, Dash0 just got even cheaper!

FAQ

• I am sending less telemetry than what is included in the base subscription fee and my billing cycle ends after February 7th. Will I pay one more month of the base subscription fee?

  • No, we are waiving the base subscription fee of all current billing cycles. That is, every billing cycle beginning on or after January 7th will be charged with the simpler consumption-based model for the entirety of the data you sent us in that billing cycle. For example, if you sent us 1 million logs, 2 million spans and 4 million metric data points during the current billing cycle that started after January 7th, you will pay 1x $0.60 + 2x $0.60 + 4x $0.20 = $2.60.

• I send a lot of spans and logs, but only a few tens of millions of metric data points. Will my bill be cheaper with the new model?

  • Yes, you will only pay the millions of metric data points you send, not at least 100 millions like before. You can save up to $20 per month (100 million, charged $0.20 per million). Similar math can be done for spans and log records.

• So I could have Dash0 bills that go down to zero?

  • If you do not send us any telemetry in a billing cycle, we charge you nothing. You’ll still receive a $0 invoice for bookkeeping purposes.

• Will you delete my organization and my telemetry and configurations in it if I don’t send telemetry for an entire month?

  • No. You paid for that telemetry and you get the full retention period on it. For metrics, it’s 13 months, so you have the right to your organization for at least that after you stop sending telemetry.

• If I delete my Dash0 organization in the middle of a billing cycle, do I still get an invoice at the end of it?

  • Yes, you get one last invoice for the amount of telemetry sent since the beginning of the last billing cycle.