Technology
Karpenter
Monitor Karpenter node provisioning and lifecycle with OpenTelemetry metrics to understand autoscaling behavior, node churn, and cloud cost efficiency.
Overview
Karpenter Integration in Dash0
Karpenter is an open-source, high-performance Kubernetes node autoscaler that provisions right-sized compute nodes in response to unschedulable pods. Unlike the Cluster Autoscaler, Karpenter works directly with your cloud provider to launch nodes that exactly match workload requirements, reducing both scheduling latency and cloud spend.
This Dash0 integration surfaces Karpenter's Prometheus metrics via OpenTelemetry so you can observe node provisioning rates, disruption events, pod churn, and scheduling latency — giving you the insight needed to tune NodePools and consolidation policies effectively.
For more information, visit the Karpenter Project Page.
Use Cases
Optimize Kubernetes infrastructure costs and reliability with this integration:
- Monitor Node Provisioning: Track how quickly Karpenter responds to unschedulable pods and provisions new nodes to detect bottlenecks.
- Analyze Disruption Events: Understand consolidation, drift, expiration, and interruption patterns to reduce unnecessary node churn.
- Track Pod Scheduling Latency: Measure time from pod pending to node ready to identify slow provisioning paths.
- Understand NodePool Utilization: Monitor resource allocation across NodePools to fine-tune instance type selection and limits.
- Optimize Cloud Costs: Correlate Karpenter's consolidation activity with node lifecycle events to validate cost-saving policies.
Setup
Overview
This guide covers two observability signals for Karpenter:
- Metrics — Karpenter exposes a Prometheus endpoint on port
8080in thekube-systemnamespace. The OpenTelemetry Collector scrapes this and forwards metrics to Dash0 using thekarpenter_metric prefix, covering node provisioning, disruption, pod scheduling, and resource allocation. - Logs — Karpenter controllers emit structured JSON logs that can be collected automatically by the Dash0 Operator or manually via the
filelogreceiver.
Prerequisites
Before setting up Karpenter monitoring, ensure:
- Karpenter is installed and running (
kubectl get pods -n kube-system -l app.kubernetes.io/name=karpentershows healthy pods) - You have access to the
kube-systemnamespace - You have an OpenTelemetry Collector deployed and reachable from the
kube-systemnamespace
You'll need an OpenTelemetry Collector deployed. Consider using:
- Dash0 operator for Kubernetes (recommended)
- OpenTelemetry Operator for Kubernetes
-
Helm Chart for The OpenTelemetry Collector
Collector Configuration
Karpenter exposes Prometheus metrics on port 8080 and creates a Kubernetes Service named karpenter in kube-system. Configure the OpenTelemetry Collector's Prometheus receiver to scrape it:
receivers:
prometheus:
config:
scrape_configs:
- job_name: karpenter
scrape_interval: 30s
static_configs:
- targets:
- karpenter.kube-system.svc.cluster.local:8080If you run Karpenter with multiple replicas, only the leader pod handles provisioning. Service scraping may route to a non-leader pod and return incomplete activity metrics. In that case, use pod-level scraping instead:
receivers:
prometheus:
config:
scrape_configs:
- job_name: karpenter
scrape_interval: 30s
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- kube-system
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
action: keep
regex: karpenter
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: http-metricsIf you are using the Dash0 Operator, first create a Dash0Monitoring resource in the kube-system namespace to enable Prometheus scraping. The instrumentWorkloads.mode: none setting is required because the operator does not support auto-instrumentation of system namespaces:
kubectl apply -f - <<EOF
apiVersion: operator.dash0.com/v1beta1
kind: Dash0Monitoring
metadata:
name: dash0-monitoring
namespace: kube-system
spec:
instrumentWorkloads:
mode: none
prometheusScraping:
enabled: true
EOFkubectl patch deployment karpenter -n kube-system \
--type=merge \
-p '{"spec":{"template":{"metadata":{"annotations":{"prometheus.io/scrape":"true","prometheus.io/port":"8080"}}}}}'For more options, see the Prometheus Receiver documentation.
Logs
Karpenter controllers write structured JSON logs using the Zap logger through controller-runtime. Log lines include fields like the controller name, reconciled object, log level, and message — making them straightforward to filter in Dash0.
Dash0 Operator
If you applied a Dash0Monitoring resource to the kube-system namespace (as shown in the Collector Configuration section above), the Dash0 Operator automatically collects pod logs from all Karpenter controllers in that namespace. No additional configuration is required.
OTel Operator or Helm Chart
To collect Karpenter logs manually, add a filelog receiver targeting the kube-system namespace and filtering for Karpenter pods:
receivers:
filelog:
include:
- /var/log/pods/kube-system_karpenter*/*/*.log
operators:
- type: container
- type: json_parser
timestamp:
parse_from: attributes.time
layout: "%Y-%m-%dT%H:%M:%S.%LZ"To increase log verbosity for debugging, set the LOG_LEVEL environment variable on the Karpenter deployment to debug:
kubectl set env deployment/karpenter -n kube-system LOG_LEVEL=debugVerification
Metrics
- Check collector logs for successful scrape of the
karpentertarget - Navigate to Metrics Explorer in Dash0
- Search for metrics with prefix
karpenter_
karpenter_nodes_created_total and karpenter_scheduler_scheduling_duration_seconds only appear after Karpenter has provisioned at least one node. Deploy a workload that requests more resources than the existing nodes provide to trigger provisioning.
Logs
- Navigate to Logs in Dash0
- Filter by
namespace: kube-systemandk8s.deployment.name: karpenter
Key metrics to verify:
| Metric | Description |
|---|---|
karpenter_nodes_created_total | Total nodes created by Karpenter |
karpenter_scheduler_scheduling_duration_seconds | Duration of scheduling simulations in seconds |
karpenter_voluntary_disruption_eligible_nodes | Nodes currently eligible for disruption by Karpenter |
karpenter_pods_state | Pod counts by scheduling state |
karpenter_nodepools_usage | Resource usage per NodePool (CPU, memory, pods) |
