OpenTelemetry Kubernetes Operator

Overview

The OpenTelemetry Operator simplifies and automates the management of OpenTelemetry within your Kubernetes environment. It handles everything from managing collectors to enabling auto-instrumentation for your applications.

Installation

Deploy Cert Manager

Before deploying the OpenTelemetry Operator, ensure that Cert Manager is installed. It's a required prerequisite. If it is not already running in your cluster, you can install it using Helm with the following commands:

Deploy the OpenTelemetry Operator

You can now deploy the OpenTelemetry Operator. The collectorImage repository must be explicitly specified, as the default image is the OpenTelemetry Collector core distribution, which lacks the necessary components.

Additionally, it is recommended to define the namespace where the operator will be deployed and ensure it is created if it does not already exist:

If you need automatic instrumentation for Go applications, you can enable it by adding the following flag to the command above:

Note: Enabling this feature comes with security trade-offs, as it relies on eBPF, which requires elevated permissions for the sidecar container. For more details, see this deep dive.

Create a Kubernetes secret for the Dash0 Authorization Token

To send data to Dash0, you need to obtain your Authorization Token and store it as a Kubernetes secret.

Run the following command to create a secret containing your Dash0 Authorization Token:

Apply the The OpenTelemetry Collector configurations

In this configuration, we will deploy two different OpenTelemetry Collectors:

• otel-collector (mode: DaemonSet)
  • Runs on every node to minimize latency and reduce network overhead. It is responsible for:
    • Collecting node-level metrics using the Kubeletstats Receiver (CPU, memory, disk usage).
    • Collecting host-level metrics (CPU, filesystem, and memory utilization) using the Host Metrics Receiver.
    • Collecting logs using the filelog receiver
    • Automatically enriching telemetry data with Kubernetes metadata using the Kubernetes Attributes Processor.
• otel-collector-central (mode: Deployment)
  • A single-replica collector that handles:
    • Cluster-wide metrics via the Kubernetes Cluster Receiver (node conditions, pod phases, resource quotas).
    • Kubernetes API server objects using the Kubernetes Objects Receiver (events, namespaces, resource quotas).

Our OpenTelemetry Collectors need to interact with various Kubernetes components, such as the Kubernetes API, to collect telemetry data. To enable this, we must ensure they have the necessary permissions and access within the cluster.

Deploy the necessary ServiceAccount's, ClusterRoleBinding's, ClusterRole's to the cluster:

Deploy the otel-collector:

Deploy the otel-collector-central:

Using the deployed OpenTelemetry Collector

The OpenTelemetry Collectors have now been deployed to your cluster and can be used to send data.

The following services can be addressed within your cluster now, and all sent data will be forwarded to Dash0 using the configured endpoint and Auth Token:

• otel-collector.opentelemetry is ClusterIP service. It can be addressed using OTLP gRPC (tcp/4317) and OTLP HTTP (tcp/4318)
• otel-collector-headless.opentelemetry can be used as a headless service. It can be addressed using OTLP gRPC (tcp/4317) and OTLP HTTP (tcp/4318)

Optional: Adding automatic instrumentation to your workloads

By default, the OpenTelemetry Operator does not automatically instrument workloads. If you want your applications to be instrumented automatically first create an Instrumentation configuration and add it to your cluster:

Next add the following annotation to your workload's pod spec:

Dataset control

By default, telemetry data is sent to the default dataset in Dash0. To route data to a different dataset, modify the Dash0-Dataset header in the exporter configuration:

Troubleshooting

Refer to the official OpenTelemetry Operator troubleshooting documentation.