Check rules are Dash0 assets that can be versioned and managed as infrastructure as code. This approach enables version control, automated deployment pipelines, consistent configuration across environments, and code review for alert changes.

Tip

Agent0 can generate infrastructure-as-code configurations for check rules and raise them as pull requests to your repository. After creating a check rule with Agent0, request the corresponding Terraform configuration or Kubernetes manifest to keep your observability in code. See Asset Creation for details.

Dash0 Operator for Kubernetes

Define check rules as PrometheusRule custom resources that sync to Dash0, enabling workflows and native Kubernetes management. Dash0 uses the standard Prometheus Operator format (monitoring.coreos.com/v1), allowing you to reuse existing PrometheusRule definitions without modification.

Example

In the below, the PrometheusRule CRD is used, which is an open source standard. It can be used to design both alert rules (as in the case below), and recording rules.

yaml

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: frontend-latency
  namespace: monitoring
  labels:
    prometheus: example     # any labels you like; used for selecting/organizing the resource
    dash0.com/dataset: production
    role: alert-rules
spec:
  groups:
    - name: Alerting
      interval: 1m
      rules:
        - alert: Frontend P99 Latency
          expr: |
            histogram_quantile(0.99,
              sum(rate({otel_metric_name="dash0.spans.duration", service_name="frontend"}[5m]))
            ) * 1000 > $__threshold
          for: 2m
          keep_firing_for: 1m
          annotations:
            # You can interpolate various values in `summary` and `description`
            summary: "Frontend P99 latency is {{ $value }}ms"
            description: "P99 latency exceeded threshold for service {{ $labels.service_name }}"

            # You need to specify thresholds if your PromQL query in `expr` uses the `$__threshold` symbol.
            # Threshold values are specified as annotations, so they need to be YAML strings.
            # A rule that uses `$__threshold` without one of these annotations is rejected.
            dash0-threshold-degraded: "500"
            dash0-threshold-critical: "500"
            ## The unprefixed form also works, for backwards compatibility
            # threshold-degraded: "500"
            # threshold-critical: "1000"

            # optional; "false" to disable this rule without deleting it
            dash0-enabled: "true"

            # comma-separated notification channel ids. This is equivalent
            # to linking the check rule with the notification channels as
            # you can do via the UI
            dash0.com/notification-channel-ids: ded5721c-2bf0-4a10-ab33-511fef734b0f,56aaacb2-10c6-4cef-8aeb-d7c862ad863e
          labels:
            # Plain check-rule labels. Route by adding filters on these
            # labels to the notification channels in Dash0 — not 
            # via an annotation on the rule.
            team: platform
            service: frontend
            severity: high

The Dash0 Operator for Kubernetes watches for PrometheusRule resources and syncs them to your Dash0 dataset. The dataset to which the PrometheusRule resources are synched is the same defined by the Dash0Monitoring resource in the Kubernetes namespace in which the PrometheusRule resources reside. There are also Dash0-specific annotations for thresholds (dash0-threshold-degraded, dash0-threshold-critical) and notification channels.

Annotations you set at the top level, under metadata.annotations, are merged into every rule's own annotations key by key, so you can define a common setting once, such as dash0.com/notification-channel-ids, instead of repeating it on each rule. When a rule sets a key that also appears at the top level, the rule's value wins for that key alone, and the remaining top-level annotations still apply to it.

A rule's own value replaces the inherited one

The merge works per key, not per value. Where both set the same key, the rule's value replaces the inherited one.

Dash0 does not combine lists, so a rule that names one channel in dash0.com/notification-channel-ids loses the ones it would have inherited. Repeat the inherited ids to keep them.

In the example below, CheckoutHighLatency inherits the top-level channel, while CheckoutHighErrorRate sends its notifications to a different one:

yaml

12345678910111213141516171819
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-alerts
  namespace: monitoring
  annotations:
    dash0.com/notification-channel-ids: ded5721c-2bf0-4a10-ab33-511fef734b0f
spec:
  groups:
    - name: Alerting
      rules:
        - alert: CheckoutHighLatency
          # Inherits the top-level channel.
          expr: histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name="checkout"}[5m]))) > 1
        - alert: CheckoutHighErrorRate
          # Overrides it with its own.
          expr: sum(rate(http_server_request_duration_seconds_count{service_name="checkout", http_response_status_code=~"^5[0-9][0-9]$"}[5m])) > 0.1
          annotations:
            dash0.com/notification-channel-ids: 56aaacb2-10c6-4cef-8aeb-d7c862ad863e

A few annotations configure how a rule evaluates rather than describing it:

Annotation	Controls	Also accepted
`dash0-enabled`	whether the rule evaluates at all
`dash0-threshold-critical`	the critical threshold	`threshold-critical`
`dash0-threshold-degraded`	the degraded threshold	`threshold-degraded`

These are unprefixed, unlike the dash0.com/-prefixed annotations, so they are easy to mistake for ordinary metadata. They inherit and override like any other annotation, so a value set at the top level reaches every rule that does not name its own. Setting dash0-enabled: "false" at the top level therefore stops every rule in the document from firing, except those that set it back to "true" themselves.

The critical and degraded thresholds are two separate annotations, so a document can set one at the top level and a rule can set the other. Both then apply to that rule. Below, the rule inherits a critical threshold of 1000 milliseconds and sets a degraded threshold of 500 milliseconds:

yaml

123456789101112131415
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-latency
  namespace: monitoring
  annotations:
    dash0-threshold-critical: "1000"
spec:
  groups:
    - name: Alerting
      rules:
        - alert: Checkout P99 Latency
          expr: histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name="checkout"}[5m]))) * 1000 > $__threshold
          annotations:
            dash0-threshold-degraded: "500"

The check reports critical above 1000 milliseconds, degraded between 500 and 1000 milliseconds, and healthy below 500 milliseconds. A measurement that crosses both thresholds is reported as critical, never as both. See the Dash0 alerting model for how degraded and critical states behave.

Dash0 evaluates the critical threshold first, so the critical value has to be the more severe one for the comparison used in the expression: the higher number for >, the lower number for <. Swapping them in the rule above, to a critical threshold of 500 and a degraded threshold of 1000, reports critical from 500 milliseconds upwards and never reports degraded at all.

The prefixed and unprefixed spellings of a setting are different annotation keys, and the prefixed one wins wherever both are present. That takes precedence over the rule-beats-document behavior above: a rule setting threshold-critical does not override a top-level dash0-threshold-critical. The losing value is not discarded either, it stays on the check rule as an ordinary annotation, which is usually how you notice. Use one spelling throughout a document.

The Dash0 Terraform provider and Dash0 CLI apply the same merge, so annotations behave the same way in all three tools. The Terraform provider additionally requires the document to hold a single rule, as described below.

Apply and Manage

PrometheusRule resources are synchronized with Dash0 by the Dash0 Operator for Kubernetes provided that:

You have a Dash0Monitoring resource in the same namespace as the PrometheusRule resource
You did not opt-out of the synching through the synchronizePrometheusRules setting

For more information, refer to the Managing Dash0 Check Rules documentation of the Dash0 Operator for Kubernetes.

bash

1234567891011121314
# Apply the check rule
kubectl apply -f check-rule.yaml

# List all check rules
kubectl get prometheusrules -n monitoring

# View details
kubectl describe prometheusrule frontend-latency -n monitoring

# Update the check rule
kubectl edit prometheusrule frontend-latency -n monitoring

# Delete the check rule
kubectl delete prometheusrule frontend-latency -n monitoring

Dash0 Terraform Provider

Manage check rules in your Terraform infrastructure alongside other cloud resources for unified infrastructure management.

Basic Example

Note

The Dash0 Terraform Provider supports both the PrometheusRules CRD format, as well as plain Prometheus alerting rules. But the PrometheusRules CRD format is preferred.

hcl

1234567891011121314151617181920212223242526272829303132333435363738394041
terraform {
  required_providers {
    dash0 = {
      source  = "dash0hq/dash0"
      version = "~> 1.0"
    }
  }
}

provider "dash0" {
  api_token = var.dash0_api_token
  endpoint  = "https://api.dash0.com"
}

resource "dash0_check_rule" "adservice_error_rate" {
  dataset = "production"

  # Currently only one group incl. one rule is supported
  check_rule_yaml = <<-EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: adservice
spec:
  groups:
    - name: Alerting
      interval: 1m0s
      rules:
        - alert: adservice
          expr: (sum by (service_namespace, service_name) (increase({otel_metric_name = "dash0.spans", service_name = "adservice", service_namespace = "opentelemetry-demo", dash0_operation_name != "", otel_span_status_code = "ERROR"}[5m]))) / (sum by (service_namespace, service_name) (increase({otel_metric_name = "dash0.spans", service_name = "adservice", service_namespace = "opentelemetry-demo", dash0_operation_name != ""}[5m])) > 0)*100 > $__threshold
          for: 0s
          keep_firing_for: 0s
          annotations:
            summary: 'High error percentage for adservice: {{$value|printf "%.2f"}}%'
            description: 'High error percentage for adservice: {{$value|printf "%.2f"}}%'
            dash0-threshold-critical: "40"
            dash0-threshold-degraded: "35"
            dash0-enabled: true
          labels: {}
EOF
}

A dash0_check_rule accepts exactly one group containing exactly one rule, so it cannot hold a multi-rule document. Top-level metadata.annotations merge into that rule's own annotations on the same terms described above, which lets a document written for the Dash0 Operator for Kubernetes be used here verbatim. To manage many rules, use for_each over the resource and keep the shared value in Terraform rather than in an annotation.

For the annotations that control thresholds and whether a rule evaluates at all, see the dash0_check_rule resource documentation.

Dash0 CLI

Script check rule operations with the Dash0 CLI for automated deployments and batch operations.

Note

The Dash0 CLI supports both the PrometheusRules CRD format, as well as plain Prometheus alerting rules. But the PrometheusRules CRD format is preferred.

Top-level metadata.annotations merge into each rule's own annotations, exactly as described above for the Dash0 Operator for Kubernetes. This applies to every rule in a multi-rule PrometheusRule document.

Create from File

bash

12
# Create a check rule from a YAML file
dash0 create-checks create -f check-rule.yaml --dataset <id>

List Check Rules

bash

12
# List all check rules
dash0 create-checks list --dataset <id>

Get Check Rule Details

bash

12345
# Get a single check rule as YAML
dash0 create-checks get <id> -o yaml --dataset <id>

# Get check rule in JSON format
dash0 create-checks get <id> -o json --dataset <id>

Update Check Rule

bash

12
# Update an existing check rule
dash0 create-checks update <id> -f check-rule.yaml --dataset <id>

Delete Check Rule

bash

12
# Delete a check rule
dash0 create-checks delete <id> --dataset <id>

Manage Check Rules as Code

Dash0 Operator for Kubernetes

Example

Apply and Manage

Dash0 Terraform Provider

Basic Example

Dash0 CLI

Create from File

List Check Rules

Get Check Rule Details

Update Check Rule

Delete Check Rule

Further Reading