PromQL, while powerful, can present a learning curve. Dash0 addresses this with a visual Query Builder that handles the most common cases — but understanding the underlying query patterns helps you get the most out of the PromQL tab and build robust dashboards and alerts.

The following queries are written with data from the OpenTelemetry demo application in mind. You can add the OpenTelemetry demo application to your organization via our integration hub.

Working with Gauge Metrics

Gauge metrics represent values that can increase or decrease over time, making them ideal for measuring current states like memory usage or connection counts.

Simple Gauge Query

promql

1
container_memory_working_set_bytes{service_name="adservice"}

Filtering Multiple Values

promql

1
container_memory_working_set_bytes{service_name=~"cartservice|adservice"}

Aggregating Across Services

promql

1
avg by(service_name) (container_memory_working_set_bytes{service_name=~"cartservice|adservice"})

Working with Sum Metrics (Counters)

Sum metrics represent a cumulative total that changes over time. When monotonic (the common case), they track values that only increase — such as request counts or bytes transferred. In the OTel API these are produced by the Counter instrument; in Prometheus terminology they are called counters. Use rate() for per-second rates or increase() for total change over an interval.

Calculating Rate per Second

promql

1
rate({otel_metric_name="app.frontend.requests", otel_metric_type="sum"}[$__rate_interval])

Calculating Total Count over a Time Range

promql

1
increase({otel_metric_name="app.frontend.requests", otel_metric_type="sum"}[$__rate_interval])

Count over Time Range per Service

promql

1
sum by(service_name) (increase({otel_metric_name="app.frontend.requests", otel_metric_type="sum"}[$__rate_interval]))

Working with Histogram Metrics

Histograms are particularly valuable for measuring distributions of values like request durations or response sizes.

Observation Count

Query the total number of observations using the _count suffix:

promql

1
http_request_duration_seconds_count

Sum of Observations

Access the sum of all observed values using the _sum suffix:

promql

1
http_request_duration_seconds_sum

Bucketed Observations

Examine the distribution of values across predefined buckets using the _bucket suffix. The le label selects specific bucket boundaries. The query below returns the count of observations that fell into buckets with upper bounds ≤ 0.5 seconds:

promql

1
http_request_duration_seconds_bucket{le="0.5"}

Percentiles with `histogram_quantile`

promql

1
histogram_quantile(0.9, rate({otel_metric_name = "http_request_duration_seconds", otel_metric_type = "histogram"}[5m]))

Native vs Classic Histograms

When querying via otel_metric_name, Dash0 returns native histograms — histogram_quantile() works directly without sum by (le). When querying via the Prometheus-compatible _bucket name (e.g., http_request_duration_seconds_bucket), classic bucket time series are returned and sum by (le) is required to preserve bucket boundaries.

Working with Summary Metrics

Like histograms, summary metrics track both counts and sums of observations, but calculate quantiles on the client side.

Observation Count

promql

1
http_request_duration_seconds_count

Sum of Observations

promql

1
http_request_duration_seconds_sum

Quantiles

promql

1
{otel_metric_name="http_request_duration_seconds", otel_metric_type="summary", quantile="0.9"}

Average

promql

1
http_request_duration_seconds_sum / http_request_duration_seconds_count

Synthetic Metrics

Dash0's synthetic metrics provide on-the-fly calculations based on ingested telemetry, enabling dynamic analysis without predefined aggregations. To learn more, see Understand Synthetic Metrics.

Tip

To understand the underlying signals for the synthetic metric examples below in relation to spans and logs, see the Tracing Explorer and Log Explorer.

Span Count per Service

promql

1
sum by(service_name) (rate({otel_metric_name="dash0.spans"}[$__rate_interval]))

P99 Span Duration (in milliseconds)

promql

1
histogram_quantile(0.99, sum(rate({otel_metric_name="dash0.spans.duration", service_name="productcatalogservice"}[$__interval]))) * 1000

Error Percentage per Kubernetes Pod

promql

1234567891011121314
(
  sum by(k8s_pod_name) (
    increase({otel_metric_name="dash0.spans", otel_span_status_code="ERROR", service_name="productcatalogservice"}[$__interval])
  )
  > 0
)
/
(
  sum by(k8s_pod_name) (
    increase({otel_metric_name="dash0.spans", service_name="productcatalogservice"}[$__interval])
  )
  > 0
)
> 0

Counting Logs Matching a Pattern

promql

1
sum(increase({otel_metric_name="dash0.logs", otel_log_body=~".*connect.*"}[$__interval])) > 0

Logs Broken Down by Severity

promql

1
sum by(otel_log_severity_range) (increase({otel_metric_name="dash0.logs"}[$__interval])) > 0

Error Log Count per Deployment

promql

1
sum by(k8s_deployment_name) (increase({otel_metric_name="dash0.logs", otel_log_severity_range="ERROR"}[$__interval])) > 0

Pod Names for a Service

promql

1
sum by(k8s_pod_name) ({otel_metric_name="dash0.resources", service_name="adservice"})

Pod Count per Namespace

promql

1
sum by(k8s_namespace_name) ({otel_metric_name="dash0.resources", k8s_pod_name!=""})

Write Effective PromQL Queries