Analyze Service Performance

Dash0 automatically generates a set of built-in synthetic metrics from your telemetry. These appear alongside your own custom metrics in the Query Builder, as can be seen above. For example:

• dash0.spansCopy — derived from span telemetry; used by the Services and Tracing tabs.
• dash0.spans.durationCopy — the duration histogram of all spans; powers latency queries in Services and Tracing.

These metrics are also accessible directly in the Metrics and PromQL tabs, giving you the flexibility to combine them with your own metrics or apply custom aggregations.

The preview chart updates immediately and the PromQL Preview shows the generated query, including any filters and aggregations applied.

• Start at the edge, work inward. If you are investigating a user-reported slowdown, start with your outermost public-facing service — for example frontendCopy or api-gatewayCopy — to confirm whether latency is concentrated there or whether it is being passed down from a dependency. Then move to upstream callers like checkoutCopy or paymentCopy to trace where the time is actually being spent.
• Investigate a downstream dependency directly. If a call-graph or trace already points to a slow dependency — for example a recommendationCopy or product-catalogCopy service — select that service directly rather than the caller. Measuring the dependency in isolation tells you whether the problem is in the service itself or in how it is being called.
• Compare services side by side. To compare two services — for example order-serviceCopy and payment-serviceCopy — build a query for each and add both to the same dashboard panel. Seeing their P95 latency on the same chart makes it easier to spot which service started degrading first after a deployment.

• Remove health-check and liveness probe endpoints. Kubernetes liveness and readiness probes generate a continuous high-frequency stream of fast, successful spans. Endpoints like /healthzCopy, /readyzCopy, /livezCopy, or /pingCopy will pull your P90 and P95 values down and make real user latency appear better than it is. Uncheck these before adding a query to a dashboard or check rule.
• Isolate write operations from reads. Services that handle both reads and writes often show a bimodal latency distribution — GETCopy operations are typically fast while POSTCopy or PUTCopy operations that write to a database are slower. Uncheck read operations like GET /productsCopy when you want a clean view of write latency, and vice versa.
• Focus on a single high-value endpoint. If you are building a check rule for a specific SLO — for example a 300 ms P95 target for your checkout flow — uncheck everything except the operation that represents that flow, such as POST /checkoutCopy. Including unrelated operations in the same rule makes it harder to attribute a breach to its root cause.
• Use Select all to reset after exploring. If you have been unchecking operations to explore different slices of the data, click Select all before promoting the query to a dashboard to make sure you are not accidentally omitting operations that belong in the final view.

• Isolate a single environment. Filter by deployment.environmentCopy = productionCopy to exclude staging or canary traffic from your baseline. This is especially important before creating a check rule, where staging noise can cause false-positive alerts.
• Scope to a specific cluster. If the same service runs across multiple Kubernetes clusters, filter by k8s.cluster.nameCopy = prod-eu-west-1Copy to compare clusters individually and rule out region-specific issues.
• Focus on error spans only. Filter by otel.span.status.codeCopy = ERRORCopy alongside the Error rate metric to isolate spans that completed with an error status. Use UNSETCopy to see spans where no explicit status was set — these are neither successes nor failures and can indicate incomplete instrumentation.
• Filter by HTTP response code. Filter by http.response.status_codeCopy = 500Copy to isolate server-side failures and separate them from client errors (400Copy–499Copy) that may not warrant an alert.
• Narrow to a specific operation name. Filter by dash0.span.nameCopy = GET /api/dataCopy to focus on a single endpoint without using the Operations list — useful when you want to combine an operation filter with other attribute filters in the same query.
• Pin to a specific namespace. In multi-tenant clusters, filter by k8s.namespace.nameCopy = acme-prodCopy to ensure you are only seeing spans from the intended workload and not from services that share the same name in a different namespace.

A common workflow is to start with a visual tab to get the basic shape of a query, then switch to the PromQL tab to add complexity.