Technology
Kubernetes
Gain visibility into your Kubernetes environment with unified metrics and checks. Monitor nodes, pods, and containers for immediate issue detection.
Overview
Overview
Kubernetes has become a cloud-native operating system, and monitoring it and its workload is essential to operating your platform. With Dash0, you can gain detailed insights into your Kubernetes clusters. And because Dash0 fully embraces open standards and CNCF technologies, it is incredibly interoperable.
Use Cases
- Monitor Kubernetes cluster and workload metrics
- View workload events and readiness
- See CPU, memory and disk usage statistics.
- Understand how close your workloads are to their limits.
- Observe detailed telemetry about your deployed workloads.
Setup
Setup
Dash0 provides native Kubernetes Views and Dashboards to help you monitor your clusters, workloads, and resource usage. These features rely on OpenTelemetry components and Kubernetes metadata following semantic conventions to ensure consistency across environments.
This setup guide outlines how to collect and export the required telemetry to power Dash0's Kubernetes integration.
Architecture Overview
To support all built-in Kubernetes views in Dash0, telemetry must be collected using a two-tier OpenTelemetry Collector deployment:
1. DaemonSet Collectors (Node-level)
- Receivers:
kubeletstats - Purpose: Collect CPU, memory, and system metrics per pod and node
- Deployment: Runs as a DaemonSet on every node
2. Deployment Collector (Cluster-level)
- Receivers:
k8s_cluster - Purpose: Collect workload status and aggregate node metrics
- Deployment: Runs as a single deployment
3. Metadata Enrichment
- Processor:
k8sattributes - Purpose: Annotate metrics with Kubernetes metadata (e.g.,
k8s.namespace.name,k8s.pod.name,k8s.resource.uid)
Note: For the best experience in Dash0, we strongly recommend enriching telemetry with Kubernetes resource UIDs (e.g.,
k8s.pod.uid,k8s.node.uid, etc.). These ensure consistent identity and grouping across restarts or rescheduling. You can include them via thek8sattributesprocessor:yamlk8sattributes: extract: metadata: - k8s.pod.uid - k8s.node.uid - k8s.deployment.uid
Installation Options
Dash0 supports multiple ways to deploy the telemetry pipeline:
- Dash0 operator for Kubernetes (recommended)
- OpenTelemetry Operator for Kubernetes
-
Helm Chart for The OpenTelemetry Collector
Choose the method that best fits your environment.
Dashboards
EBS CSI Driver Overview
EBS CSI Driver IOPS and Throughput metrics for Kubernetes nodes.
K8s Node Overview
Kubernetes Node Metrics Dashboard - monitor CPU, memory, disk usage, and node health to optimize resource performance.
K8s Pod Overview
Kubernetes Pod Metrics Dashboard - monitor CPU, memory, disk usage, and pod health to optimize resource performance.
Check Rules
Kubernetes DaemonSet misscheduled
DaemonSet pods are misscheduled
sum by (k8s_daemonset_name, k8s_namespace_name)({otel_metric_name="k8s.daemonset.misscheduled_nodes"}) > 0Kubernetes DaemonSet rollout stuck
A DaemonSet rollout is stuck
sum by (k8s_daemonset_name, k8s_namespace_name)({otel_metric_name="k8s.daemonset.ready_nodes"}) / sum by (k8s_daemonset_name, k8s_namespace_name)({otel_metric_name="k8s.daemonset.desired_scheduled_nodes"}) * 100 < 100 or sum by(k8s_daemonset_name, k8s_namespace_name)({otel_metric_name="k8s.daemonset.desired_scheduled_nodes"}) - sum by (k8s_daemonset_name, k8s_namespace_name)({otel_metric_name="k8s.daemonset.current_scheduled_nodes"}) > 0Kubernetes Deployment replicas mismatch
Deployment replicas mismatch
sum by (k8s_deployment_name, k8s_namespace_name)({otel_metric_name="k8s.deployment.desired"}) != sum by (k8s_deployment_name, k8s_namespace_name)({otel_metric_name="k8s.deployment.available"})Kubernetes HPA scale maximum
A Horizontal Pod Autoscaler has scaled to maximum
HPA {{ $labels.k8s_namespace_name }}/{{ $labels.k8s_hpa_name }} has scaled to maximum.
sum by(k8s_hpa_name,k8s_namespace_name)({otel_metric_name="k8s.hpa.current_replicas"}) == sum by (k8s_hpa_name, k8s_namespace_name)({otel_metric_name="k8s.hpa.max_replicas"})Kubernetes HPA underutilized
A Horizontal Pod Autoscaler is constantly underutilized
HPA {{ $labels.k8s_namespace_name }}/{{ $labels.k8s_hpa_name }} is constantly underutilized.
sum by (k8s_hpa_name,k8s_namespace_name)({otel_metric_name="k8s.hpa.current_replicas"}) == sum by (k8s_hpa_name, k8s_namespace_name)({otel_metric_name="k8s.hpa.min_replicas"})Kubernetes Job failed
A Kubernetes Job failed to complete
Job {{ $labels.k8s_namespace_name }}/{{ $labels.k8s_job_name }} failed to complete.
sum by (k8s_job_name)({otel_metric_name="k8s.job.failed_pods"}) > 0Kubernetes Job slow completion
A Kubernetes Job did not complete in time
Kubernetes Job {{ $labels.k8s_namespace_name }}/{{ $labels.k8s_job_name }} did not complete in time.
sum by (k8s_job_name, k8s_namespace_name)({otel_metric_name="k8s.job.desired_successful_pods"}) - sum by(k8s_job_name,k8s_namespace_name)({otel_metric_name="k8s.job.successful_pods"}) > 0Kubernetes Node CPU Usage High
A Kubernetes node CPU usage is high
Node {{ $labels.k8s_node_name }} in cluster {{ $labels.k8s_cluster_name }} CPU usage is {{ $value }}%.
Monitors Kubernetes node CPU usage as percentage. This alert fires when node CPU usage exceeds the threshold:
- Default degraded: 80% CPU utilization
- Default critical: 95% CPU utilization
sum by (k8s_node_name, k8s_cluster_name)(({otel_metric_name="k8s.node.cpu.usage"} / 1000000000) * 100) > $__thresholdKubernetes Node Filesystem Available Low
Kubernetes node available filesystem space is below 10GB
Node has low available disk space (less than 10GB).
sum by (k8s_node_name, k8s_cluster_name)({otel_metric_name="k8s.node.filesystem.available"}) < 10737418240Kubernetes Node Filesystem Usage High
A Kubernetes node filesystem usage is high
Node {{ $labels.k8s_node_name }} in cluster {{ $labels.k8s_cluster_name }} filesystem usage is {{ $value }}.
Monitors Kubernetes node filesystem usage. This alert fires when the node's filesystem usage exceeds the threshold:
- Default degraded: 80GB
- Default critical: 95GB
sum by (k8s_node_name, k8s_cluster_name)({otel_metric_name="k8s.node.filesystem.usage"}) > $__thresholdKubernetes Node Memory Available Low
A Kubernetes node has low available memory
Node {{ $labels.k8s_node_name }} in cluster {{ $labels.k8s_cluster_name }} available memory is only {{ $value }}.
Node has low available memory (less than 2GB).
sum by (k8s_node_name, k8s_cluster_name)({otel_metric_name="k8s.node.memory.available"}) < 2147483648Kubernetes Node Memory Working Set High
A Kubernetes node memory working set is high
Node {{ $labels.k8s_node_name }} in cluster {{ $labels.k8s_cluster_name }} memory working set is {{ $value }}.
Monitors Kubernetes node memory working set. This alert fires when the node's memory working set exceeds the threshold:
- Default degraded: 12GB
- Default critical: 15GB
Working set represents memory pages currently in active use by the node.
sum by (k8s_node_name, k8s_cluster_name)({otel_metric_name="k8s.node.memory.working_set"}) > $__thresholdKubernetes Node Not Ready
A Kubernetes node has been unready for a long time
Node {{ $labels.k8s_node_name }} has been unready for a long time.
Node is not in Ready state.
sum by (k8s_node_name, k8s_cluster_name)({otel_metric_name="k8s.node.condition_ready"} == 0)Kubernetes Pod Failed
A Kubernetes pod is in Failed state
Pod {{ $labels.k8s_namespace_name }}/{{ $labels.k8s_pod_name }} is in Failed state.
Monitors Kubernetes pods in Failed phase. This alert fires when a pod enters the "Failed" phase. Includes pod name and namespace in alert context.
sum by (k8s_pod_name, k8s_namespace_name)({otel_metric_name="k8s.pod.phase"} == 4)Kubernetes Pod Pending Too Long
A Kubernetes pod has been pending for more than 10 minutes
Pod {{ $labels.k8s_namespace_name }}/{{ $labels.k8s_pod_name }} has been pending for more than 10 minutes.
Monitors Kubernetes pods that remain in Pending phase for extended periods. This alert fires when a pod stays in "Pending" phase for more than 10 minutes. Includes pod name and namespace in alert context.
sum by (k8s_pod_name, k8s_namespace_name)({otel_metric_name="k8s.pod.phase"} == 1)Kubernetes Pod Unknown State
A Kubernetes pod is in Unknown state
Pod {{ $labels.k8s_namespace_name }}/{{ $labels.k8s_pod_name }} is in Unknown state.
Monitors Kubernetes pods in Unknown phase. This alert fires when a pod enters the "Unknown" phase for more than 5 minutes. Unknown phase typically indicates kubelet communication issues.
sum by (k8s_pod_name, k8s_namespace_name)({otel_metric_name="k8s.pod.phase"} == 5)Kubernetes ReplicaSet replicas mismatch
ReplicaSet replicas mismatch
sum by (k8s_replicaset_name, k8s_namespace_name)({otel_metric_name="k8s.replicaset.desired"}) != sum by (k8s_replicaset_name,k8s_namespace_name)({otel_metric_name="k8s.replicaset.available"})Kubernetes StatefulSet down
A Kubernetes StatefulSet went down
sum by (k8s_statefulset_name, k8s_namespace_name)({otel_metric_name="k8s.statefulset.desired_pods"}) - sum by (k8s_statefulset_name, k8s_namespace_name)({otel_metric_name="k8s.statefulset.ready_pods"}) > 0Kubernetes StatefulSet replicas mismatch
StatefulSet does not match the expected number of replicas
sum by (k8s_statefulset_name, k8s_namespace_name)({otel_metric_name="k8s.statefulset.ready_pods"}) != sum by (k8s_statefulset_name,k8s_namespace_name) ({otel_metric_name="k8s.statefulset.current_pods"})Kubernetes Volume full in four days
A Kubernetes volume is expected to fill up within four days
{{ $labels.k8s_namespace_name }}/{{ $labels.k8s_volume_name }} is expected to fill up within four days. Currently {{ $value }} is available.
sum by (k8s_volume_name,k8s_namespace_name)(predict_linear({otel_metric_name="k8s.volume.available"}[6h], 4 * 24 * 3600)) < 0Kubernetes Volume out of disk space
A Kubernetes volume is almost full (less than 10% left)
sum by (k8s_persistentvolumeclaim_name, k8s_namespace_name)(({otel_metric_name="k8s.volume.available"} / {otel_metric_name="k8s.volume.capacity"})* 100) < $__threshold