Write Effective PromQL Queries

Overview

PromQL, while powerful, can present a learning curve for many users. Dash0 addresses this challenge by providing a query builder that simplifies the creation of common queries. This documentation serves as a reference for effective query patterns across various metric types including gauges, histograms, summaries, and Dash0's synthetic metrics. Whether you're using the query builder or writing PromQL directly, understanding these patterns will help you extract meaningful insights from your telemetry data.

The following PromQL queries are designed with data from the OpenTelemetry demo application in mind. You can easily add the OpenTelemetry demo application to your organization via our integration hub.

Working with Gauge Metrics

Gauge metrics represent values that can increase or decrease over time, making them ideal for measuring current states like memory usage or connection counts.

Simple Gauge Query

1
container_memory_working_set_bytes{service_name="adservice"}

Filtering Multiple Values

1
container_memory_working_set_bytes{service_name=~"cartservice|adservice"}

Aggregating Across Services

1
avg by(service_name) (container_memory_working_set_bytes{service_name=~"cartservice|adservice"})

Working with Sum Metrics

Counters track monotonically increasing values and are commonly used for event occurrences like request counts.

Calculating Count per Second

1
rate({otel_metric_name = "app.frontend.requests", otel_metric_type = "sum"}[5m])

Calculating Total Count over Time Range

1
increase({otel_metric_name = "app.frontend.requests", otel_metric_type = "sum"}[5m])

Count over Time Range per Service

1
sum by(service_name) (increase({otel_metric_name = "app.frontend.requests", otel_metric_type = "sum"}[5m]))

Working with Histogram Metrics

Histograms are particularly valuable for measuring distributions of values like request durations, response sizes, or other variable measurements. Dash0 provides comprehensive support for histogram metrics, allowing you to analyze your data in multiple ways:

Observation Count

Query the total number of observations using the _countCopy suffix:

1
http_request_duration_seconds_count

Sum of Observations

Access the sum of all observed values using the _sumCopy suffix:

1
http_request_duration_seconds_sum

Bucketed Observations

Examine the distribution of values across predefined buckets using the _bucketCopy suffix. Use the leCopy (less than or equal) label to select specific bucket boundaries. The below query returns the count of observations that fell into buckets with upper bounds less than or equal to 0.5 seconds.

1
http_request_duration_seconds_bucket{le="0.5"}

Histogram Functions

Dash0 supports various PromQL histogram functions that make it easy to calculate averages or percentiles:

1
histogram_quantile(0.9, rate({otel_metric_name = "http_requests_duration_seconds", otel_metric_type = "histogram"}[5m]))

Working with Summary Metrics

Like histograms, summary metrics track both counts and sums of observations, but they also calculate quantiles on the client side. In Dash0, summary metrics are accessed through:

Observation Count

Using the _countCopy suffix:

1
http_request_duration_seconds_count

Sum of Observations

Using the _sumCopy suffix:

1
http_request_duration_seconds_sum

Quantiles

Directly querying the summary with quantile labels:

1
{otel_metric_name="http_request_duration_seconds", otel_metric_type="summary", quantile="0.9"}

Average

Calculate the mean value by dividing the sum by the count:

1
http_request_duration_seconds_sum / http_request_duration_seconds_count

Dash0 Synthetic Metrics

Dash0's synthetic metrics provide on-the-fly calculations based on ingested telemetry, enabling dynamic analysis without predefined aggregations. To learn more about synthetic metrics, please check our dedicated documentation page.

Span Count per Service

1
sum by(service_name) (rate({otel_metric_name="dash0.spans"}[5m]))

P99 Span Duration

1
histogram_quantile(0.99, sum(rate({otel_metric_name = "dash0.spans.duration", service_name = "productcatalogservice"}[$__interval]))) * 1000

Error Percentage for Kubernetes Pods

Error percentage for all Kubernetes pods of a given service:

12345678910111213141516
(
  sum by (k8s_pod_name) (
    increase(
      {otel_metric_name="dash0.spans",otel_span_status_code="ERROR",service_name="productcatalogservice"}[$__interval]
    )
  )
  > 0
)
/
(
  sum by (k8s_pod_name) (
    increase({otel_metric_name="dash0.spans",service_name="productcatalogservice"}[$__interval])
  )
  > 0
)
> 0

Counting Logs with Specific Content

1
sum(increase({otel_metric_name = "dash0.logs", otel_log_body =~ ".*connect.*"}[$__interval])) > 0

Logs Broken Down by Severity

1
sum by (otel_log_severity_range) (increase({otel_metric_name = "dash0.logs"}[$__interval])) > 0

Error Log Count per Deployment

1
sum by (k8s_deployment_name) (increase({otel_metric_name = "dash0.logs", otel_log_severity_range = "ERROR"}[$__interval])) > 0

Retrieving Pod Names for a Service

1
sum by(k8s_pod_name) ({otel_metric_name="dash0.resources", service_name="adservice"})

Counting Pods per Namespace

1
sum by(k8s_namespace_name) ({otel_metric_name="dash0.resources", k8s_pod_name!=""})