PromQL Query Patterns
A comprehensive guide to PromQL query patterns combined with OpenTelemetry data, covering gauges, histograms, summaries, and synthetic metrics. Learn efficient query patterns for performance monitoring, error detection, and resource tracking.
Overview
PromQL, while powerful, can present a learning curve for many users. Dash0 addresses this challenge by providing a query builder that simplifies the creation of common queries. This documentation serves as a reference for effective query patterns across various metric types including gauges, histograms, summaries, and Dash0's synthetic metrics. Whether you're using the query builder or writing PromQL directly, understanding these patterns will help you extract meaningful insights from your telemetry data.
The following PromQL queries are designed with data from the OpenTelemetry demo application in mind. You can easily add the OpenTelemetry demo application to your organization via our integration hub.
Working with Gauge Metrics
Gauge metrics represent values that can increase or decrease over time, making them ideal for measuring current states like memory usage or connection counts.
Simple gauge query
1container_memory_working_set_bytes{service_name="adservice"}
Filtering multiple values
1container_memory_working_set_bytes{service_name=~"cartservice|adservice"}
Aggregating across services
1avg by(service_name) (container_memory_working_set_bytes{service_name=~"cartservice|adservice"})
Working with Sum Metrics
Counters track monotonically increasing values and are commonly used for event occurrences like request counts.
Calculating count per second
1rate({otel_metric_name = "app.frontend.requests", otel_metric_type = "sum"}[5m])
Calculating total count over time range (5m)
1increase({otel_metric_name = "app.frontend.requests", otel_metric_type = "sum"}[5m])
Count over time range per service
1sum by(service_name) (increase({otel_metric_name = "app.frontend.requests", otel_metric_type = "sum"}[5m]))
Working with Histogram Metrics
Histograms are particularly valuable for measuring distributions of values like request durations, response sizes, or other variable measurements. Dash0 provides comprehensive support for histogram metrics, allowing you to analyze your data in multiple ways:
Observation count
Query the total number of observations using the _count
suffix
1http_request_duration_seconds_count
Sum of observations
Access the sum of all observed values using the _sum
suffix
1http_request_duration_seconds_sum
Bucketed observations
Examine the distribution of values across predefined buckets using the _bucket
suffix. Use the le
(less than or equal) label to select specific bucket boundaries. The below query returns the count of observations that fell into buckets with upper bounds less than or equal to 0.5 seconds.
1http_request_duration_seconds_bucket{le="0.5"}
Histogram Functions
Dash0 also supports various PromQL histogram functions that make it easy to calculate average or percentiles
1histogram_quantile(0.9, rate({otel_metric_name = "http_requests_duration_seconds", otel_metric_type = "histogram"}[5m]))
Working with Summary Metrics
Like histograms, summary metrics track both counts and sums of observations, but they also calculate quantiles on the client side. In Dash0, summary metrics are accessed through:
Observation count
Using the _count
suffix
1http_request_duration_seconds_count
Sum of observations
Using the _sum
suffix
1http_request_duration_seconds_sum
Quantiles
Directly querying the summary with quantile labels
1{otel_metric_name="http_request_duration_seconds", otel_metric_type="summary", quantile="0.9"}
Average
Calculate the mean value by dividing the sum by the count
1http_request_duration_seconds_sum / http_request_duration_seconds_count
Dash0 Synthetic Metrics
Dash0's synthetic metrics provide on-the-fly calculations based on ingested telemetry, enabling dynamic analysis without predefined aggregations. To learn more about synthetic metrics, please check our dedicated documentation page.
Span Analysis
Span count per service
1sum by(service_name) (rate({otel_metric_name="dash0.spans"}[5m]))
P99 span duration
1histogram_quantile(0.99, sum(rate({otel_metric_name = "dash0.spans.duration", service_name = "productcatalogservice"}[$__interval]))) * 1000
Error percentage for all Kubernetes pods of a given service
12345678910111213141516171819(sum by (k8s_pod_name) (increase({otel_metric_name="dash0.spans",otel_span_status_code="ERROR",service_name="productcatalogservice"}[$__interval]))>0)/(sum by (k8s_pod_name) (increase({otel_metric_name="dash0.spans",service_name="productcatalogservice"}[$__interval]))>0)>0
Log Analysis
Counting logs having a specific log body
1sum(increase({otel_metric_name = "dash0.logs", otel_log_body =~ ".*connect.*"}[$__interval])) > 0
Logs broken down by severity
1sum by (otel_log_severity_range) (increase({otel_metric_name = "dash0.logs"}[$__interval])) > 0
Error log count per Kubernetes deployment
1sum by (k8s_deployment_name) (increase({otel_metric_name = "dash0.logs", otel_log_severity_range = "ERROR"}[$__interval])) > 0
Counting (OpenTelemetry) Resources
Retrieving the pod names for a given service
1sum by(k8s_pod_name) ({otel_metric_name="dash0.resources", service_name="adservice"})
Counting the number of Kubernetes pods per Kubernetes namespace
1sum by(k8s_namespace_name) ({otel_metric_name="dash0.resources", k8s_pod_name!=""})
Last updated: May 10, 2025