AutoMerge is generally available

Last updated:

Advanced Configuration

Advanced Dash0 operator configuration, including webhook certificates, cert-manager, collector pod scheduling, sysctls, SELinux options, and filelog offset volumes.

This guide covers advanced configuration topics for the Dash0 operator, including exporting to other backends, certificate management, collector pod scheduling, and performance tuning.

Table of Contents

Providing a Filelog Offset Volume

The operator's collector uses the filelog receiver to read pod log files for monitored workloads. When the collector is restarted (which can happen for various reasons, for example to apply configuration changes), it is important that the filelog receiver can continue reading the log files from where it left off. If the filelog receiver started to read all log files from the beginning again after a restart, log records would be duplicated, that is, they would appear multiple times in Dash0.

For that purpose, the filelog receiver stores the log file offsets in persistent storage. By default, the offsets are stored in a config map in the operator's namespace. For small clusters, this is usually sufficient, and it requires no additional configuration by users. For production clusters, larger staging clusters, or clusters with many short-lived pods, we recommend providing a persistent volume for storing offsets.

Any persistent volume that is accessible from the collector pods can be used for this purpose.

Here is an example with a hostPath volume (see also https://kubernetes.io/docs/concepts/storage/volumes/#hostpath for considerations around using hostPath volumes):

yaml
operator:
  collectors:
   filelogOffsetSyncStorageVolume:
     name: filelogreceiver-offsets
     hostPath:
       path: /data/dash0-operator/offset-storage
       type: DirectoryOrCreate

The directory in the hostPath volume will automatically be created with the correct permissions so that the OpenTelemetry collector container can write to it.

Here is another example based on persistent volume claims. (This assumes that a PersistentVolumeClaim named offset-storage-claim exists.) See also https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims and https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming.

yaml
operator:
  collectors:
    filelogOffsetSyncStorageVolume:
      name: filelogreceiver-offsets
      persistentVolumeClaim:
        claimName: offset-storage-claim

Important: Since this volume is needed by a Daemonset run by the Dash0 operator, the PersistentVolumeClaim needs to be set with the ReadWriteMany access mode.

When to Use a Volume for Filelog Offsets

Using a volume instead of the default config map approach is also helpful if you have webhooks in your cluster which process every config map update.

Known examples for this are:

  • The Open Police Agent, in particular the OPA gatekeeeper. When operating the OPA gatekeeper in the same cluster as the Dash0 operator, it is highly recommended to use a volume for filelog offsets. Using the default config map filelog offset storage in clusters with the OPA gatekeeper can lead to severe performance issues, since the default config map for filelog offsets is updated very frequently. This can cause the OPA gatekeeper to consume a lot of CPU and memory resources, potentially even leading to OOMKills of the OPA gatekeeper.
  • AKS clusters with the Azure Policy add-on: A managed instance of the OPA gatekeeper (see above) is installed by the Azure Policy add-on for AKS, i.e. the OPA gatekeeper is active in AKS cluster that have enabled the Azure Policy add-on. It is highly recommended to use a volume for filelog offsets for AKS clusters with the Azure Policy add-on.
  • The Kyverno admission controller. When operating Kyverno in the same cluster as the Dash0 operator, it is highly recommended to either exclude ConfigMaps (or all resource types) in the Dash0 operator's namespace from Kyverno's processing, or to use a volume for filelog offsets. Leaving Kyverno processing in place and using the config map filelog offset storage can lead to severe performance issues, since the default config map for filelog offsets is updated very frequently. This can cause Kyverno to consume a lot of CPU and memory resources, potentially even leading to OOMKills of the Kyverno admission controller. Note that a filelog offset volume adds an init container that runs as root, which restrictive Kyverno policies reject, see Kyverno Policies and the Collector DaemonSet.

For more details on OPA, Kyverno, and other platform-specific considerations, see Platform Specific.

Webhook Certificates

By default, the operator manages the TLS certificates for its webhooks itself, without the need for a third-party solution. It stores them in the Secret dash0-operator-webhook-server-cert in the operator's namespace, writes the CA into the caBundle fields of its webhook configurations and of the dash0monitorings.operator.dash0.com CRD, and renews the certificates before they expire. Applying the Helm chart again, for example via helm upgrade or an ArgoCD sync, does not change the certificates.

Things to be aware of:

  • If an admission policy (for example Kyverno or OPA Gatekeeper) prevents the operator from updating the Secret or from patching its webhook configurations or the CRD, requests to the webhooks fail, and the operator pods might not become ready. The operator logs errors in this case.
  • To replace the CA, for example after its key has been leaked, delete the Secret and run helm upgrade or an ArgoCD sync right after it, which creates the Secret again. New operator pods do not become ready until the Secret exists again. Requests to the webhooks fail briefly until the operator has written the new CA.
  • With Flux drift detection, add the ignore rules from Operator Version 0.158.0: Operator-Managed Webhook Certificates to the HelmRelease, also for new installations. Otherwise, Flux resets the caBundle values whenever the operator changes them, and requests to the webhooks fail briefly each time.
  • Do not apply the operator's resources with options that replace objects instead of updating them, like helm upgrade --force, Flux's upgrade.force, or ArgoCD's Replace=true or Force=true sync options. They remove the certificates or the caBundle values, and requests to the webhooks fail until the operator has written them again.

For upgrading from a version before 0.158.0, see Operator Version 0.158.0: Operator-Managed Webhook Certificates.

Using cert-manager

As an alternative to the operator-managed certificates, you can use cert-manager to manage the certificates. To let cert-manager handle TLS certificates, provide the following additional settings when applying the Helm chart:

yaml
operator:

  # Settings for using cert-manager instead of auto-generating TLS certificates.
  certManager:

    # This disables the certificates managed by the operator.
    # If this is set to true, the assumption is that certificates are managed by
    # cert-manager (see https://cert-manager.io/), and the other settings shown in this
    # snippet (certManager.secretName, certManager.certManagerAnnotations) become
    # required settings. It is recommended to set webhookService.name as well.
    useCertManager: true

    # The name of the secret used by cert-manager for the certificate.
    # The provided name must match the `secretName` in the `cert-manager.io/v1.Certificate`
    # resource's spec (see below).
    # Note: This secret is created and managed by cert-manager, you do not need to create
    # it manually.
    secretName: dash0-operator-certificate-secret

    # A map of additional annotations that are added to all Kubernetes resources that need
    # the certificate.
    # Usually this will be a single `cert-manager.io/inject-ca-from` annotation with the
    # namespace and name of the Certificate resource (see below), but other annotations
    # are possible, see https://cert-manager.io/docs/reference/annotations/.
    certManagerAnnotations:
      cert-manager.io/inject-ca-from: "dash0-system/dash0-operator-serving-certificate"

  webhookService:
    # A name override for the webhook service, defaults to dash0-operator-webhook-service.
    # The name of the webhook service must match the DNS names provided in the
    # cert-manager's Certificate resource.
    # For that reason it is recommended to set this value when using cert-manager, as it
    # guarantees that the names match, even if the Dash0 operator helm chart would change
    # the default name of the webhook service in a future release.
    name: dash0-operator-webhook-service-name

You will also need to provide cert-manager resources, for example a Certificate and an Issuer. Explaining all configuration options of cert-manager is out of scope for this documentation. Please refer to the cert-manager documentation for details on how to install and configure cert-manager in your cluster.

The following annotated example is a minimal configuration that matches the Dash0 operator configuration snippet shown above. Both configuration snippets assume that the Dash0 operator is installed in the default dash0-system namespace, and the Certificate and Issuer resource are also deployed into that namespace.

yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:

  # NOTE: This value, together with the namespace this resource is deployed into, must
  # match the value of the cert-manager.io/inject-ca-from annotation provided in the
  # certManager.certManagerAnnotations setting, i.e. this matches the following annotation:
  #   cert-manager.io/inject-ca-from: "dash0-system/dash0-operator-serving-certificate"
  name: dash0-operator-serving-certificate

  labels:
    app.kubernetes.io/name: certificate
    app.kubernetes.io/instance: serving-cert
    app.kubernetes.io/component: certificate

spec:

  # NOTE: Provide all DNS names that are used to access the webhook service.
  # The service names usually follow the patterns <service-name>.<namespace>.svc and
  # <service-name>.<namespace>.svc.cluster.local.
  # If you use a custom name for the Dash0 operator's webhook service (recommended), the
  # first part of the DNS names must match that name.
  # If you use the default name for the webhook service, the first part of the DNS names
  # must match the default name (dash0-operator-webhook-service).
  # If the Dash0 operator is installed into a different namespace, you need to change the
  # ".dash0-system." part of the DNS names accordingly.
  dnsNames:
  - dash0-operator-webhook-service-name.dash0-system.svc
  - dash0-operator-webhook-service-name.dash0-system.svc.cluster.local

  issuerRef:
    kind: Issuer

    # NOTE: Must match the name of the Issuer resource provided below.
    name: dash0-operator-selfsigned-issuer

  # NOTE: The secret name must match the operator.certManager.secretName setting above
  # provided to the Dash0 operator Helm chart. This secret is created and managed by
  # cert-manager, you do not need to create it manually.
  secretName: dash0-operator-certificate-secret

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:

  # NOTE: This issuer's name must match the issuerRef.name in the Certificate resource above.
  name: dash0-operator-selfsigned-issuer

  labels:
    app.kubernetes.io/name: certificate
    app.kubernetes.io/instance: serving-cert
    app.kubernetes.io/component: certificate
spec:

  # There are many options for configuring an Issuer, this example uses a simple self-signed
  # issuer. Make sure to configure the issuer according to your requirements.
  selfSigned: {}

Tuning Resource Requests and Limits

The operator sets resource requests and limits on the workloads it deploys, including the operator manager, the DaemonSet collector pods (one per node), the cluster-metrics-collector Deployment and, when SignalControl Edge is enabled, the SignalControl collector Deployment. Each of these ships with default requests and limits that are a reasonable starting point for many clusters.

However, these defaults are not guaranteed to be optimal for every use case. The amount of CPU and memory a component actually needs depends heavily on the specifics of your cluster, for example the number of nodes, the number and size of monitored workloads, the volume of telemetry (logs, metrics, traces) being processed, and how frequently Kubernetes resources (workloads, pods etc.) are created and destroyed. You should therefore treat the defaults as a baseline and tune them according to what is required in your particular Kubernetes cluster.

A good strategy is to start with relatively low limits and increase them if they turn out to be insufficient. Signs that a limit is too low include the container being OOMKilled (memory limit too low) or being CPU-throttled and lagging behind (CPU limit too low). Conversely, if a component consistently uses far less than its requests reserve, you can lower the requests to avoid needlessly reserving capacity on your nodes. Dash0's self-monitoring can help you observe the actual resource usage of the operator's components over time, which makes it easier to find appropriate values.

The following Helm values control the resource settings, all nested under the top-level operator key:

Helm valueWorkload / containerDefault requestsDefault limits
managerContainerResourcesoperator managercpu: 50m, memory: 128Mi, ephemeral-storage: 500Micpu: 500m, memory: 256Mi, ephemeral-storage: 500Mi
initContainerResourcesauto-instrumentation init containernonenone
collectors.daemonSetCollectorContainerResourcesDaemonSet collector containermemory: 500Mimemory: 500Mi
collectors.daemonSetConfigurationReloaderContainerResourcesDaemonSet configuration reloadermemory: 12Mimemory: 26Mi
collectors.daemonSetFileLogOffsetSyncContainerResourcesDaemonSet filelog offset syncmemory: 32Mimemory: 32Mi
collectors.deploymentCollectorContainerResourcescluster-metrics-collector containermemory: 500Mimemory: 500Mi
collectors.deploymentConfigurationReloaderContainerResourcescluster-metrics-collector configuration reloadermemory: 12Mimemory: 26Mi
collectors.signalControlCollectorContainerResourcesSignalControl collector containermemory: 1Gimemory: 1Gi
collectors.signalControlCollectorConfigurationReloaderContainerResourcesSignalControl collector configuration reloadermemory: 12Mimemory: 26Mi
targetAllocator.containerResourcestarget-allocator containercpu: 200m, memory: 128Micpu: 200m, memory: 500Mi
agent0Connector.containerResourcesagent0-connector containermemory: 64Mimemory: 256Mi

For every container above except the operator manager, gomemlimit is left empty by default and the operator derives it from the container's memory limit (about 80%, or about 60% for the agent0-connector, which also runs kubectl as a child process). For the three collector containers it additionally derives the internal memory_limiter thresholds, keeping GOMEMLIMIT < memory_limiter soft limit < memory_limiter hard limit < memory limit; keep their memory limit at or above the 500Mi default to leave enough off-heap headroom. Set gomemlimit explicitly only to override the derived value; for a collector it must stay below the derived soft limit, or the operator logs a warning.

The SignalControl collector only exists when SignalControl Edge is enabled. Unlike the other two collectors it processes the Dash0-bound telemetry of the whole cluster, so its memory requirement scales with total telemetry volume rather than with per-node volume. Its replica count is configured separately via operator.collectors.signalControlCollectorReplicas (default 2). When the tail-sampling reservoir of the Dash0SignalControl resource uses the disk type, the operator derives an ephemeral-storage request from its maxDiskBytes (unless an ephemeral-storage request is set explicitly). The ephemeral-storage limit of the SignalControl collector container must be at least that derived value, even with an explicit request, since the reservoir counts towards the limit. Otherwise, creating a Dash0SignalControl resource or changing its sampling or components is rejected.

Its replicas are distributed as evenly as possible over the cluster's availability zones and nodes: the operator sets topologySpreadConstraints with maxSkew: 1 on topology.kubernetes.io/zone and on kubernetes.io/hostname. Since maxSkew bounds the difference between the per-zone pod counts rather than the count itself, more replicas than zones simply means several replicas per zone, spread evenly; fewer replicas than zones means each replica gets its own zone. Both constraints use whenUnsatisfiable: ScheduleAnyway, so they are a scheduling preference and never leave a replica unschedulable — in particular on clusters whose nodes carry no topology.kubernetes.io/zone label. The operator also creates a PodDisruptionBudget with maxUnavailable: 1 for this collector, so a node drain or cluster upgrade cannot take down more than one replica at a time.

Set the replica count to at least the number of availability zones that run monitored workloads. The SignalControl collector's service is configured for zone-aware routing (see SignalControl Edge), which only keeps traffic inside a zone that actually has a SignalControl collector pod. With fewer replicas than zones, the uncovered zones keep sending across zone boundaries. The operator logs a warning when it detects this.

The settings for managerContainerResources, collectors.daemonSetCollectorContainerResources and collectors.deploymentCollectorContainerResources are the ones that are most likely to require tuning.

Here is an example that overrides the resource settings for the operator manager and the DaemonSet collector:

yaml
operator:
  managerContainerResources:
    requests:
      cpu: 100m
      memory: 192Mi
      ephemeral-storage: 500Mi
    limits:
      cpu: 1000m
      memory: 512Mi
      ephemeral-storage: 1Gi
    # approximately 80% of managerContainerResources.limits.memory; use MiB or GiB here (see https://pkg.go.dev/runtime#hdr-Environment_Variables).
    gomemlimit: 410MiB

  collectors:
    daemonSetCollectorContainerResources:
      requests:
        memory: 768Mi
      limits:
        memory: 768Mi
      # gomemlimit is left empty: the operator derives GOMEMLIMIT and the memory_limiter thresholds from the limit above.

Note: Most of the operator's containers are Go processes whose GOMEMLIMIT (the environment variable, a soft memory limit for the Go runtime's garbage collector) is derived automatically from the container's memory limit when gomemlimit is left empty: the three collectors (which also derive their memory_limiter thresholds), the target-allocator, edge-proxy, agent0-connector, configuration reloaders and filelog offset sync container. For these you normally adjust only the memory limit. The one exception is the operator manager (managerContainerResources), whose gomemlimit is set statically — whenever you raise or lower its memory limit, adjust its gomemlimit as well (about 80% of the limit). Note that gomemlimit uses different units than Kubernetes resource settings (MiB/GiB rather than Mi/Gi).

Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or similar to take effect.

Controlling On Which Nodes the Operator's Collector Pods Are Scheduled

Allow Scheduling on Tainted Nodes

The operator uses a Kubernetes daemonset to deploy the OpenTelemetry collector on each node; to collect telemetry from that node and workloads running on that node. If you use taints on certain nodes, Kubernetes will not schedule any pods there, preventing the daemonset collector pods to be present on these nodes. You can allow the daemonset collector pods to be scheduled there by configuring tolerations matching your taints for the collector pods. Tolerations can be configured as follows:

yaml
operator:
  collectors:
    daemonSetTolerations:
    - key: key1
      operator: Equal
      value: value1
      effect: NoSchedule
    - key: key2
      operator: Exists
      effect: NoSchedule

In the same fashion, tolerations can also be configured for the Dash0 operator manager (Helm value operator.tolerations), the OpenTelemetry collector deployment for collecting cluster metrics (Helm value operator.collectors.deploymentTolerations), the SignalControl collector deployment (Helm value operator.collectors.signalControlCollectorTolerations), the OpenTelemetry target-allocator deployment (Helm value operator.targetAllocator.tolerations) and the agent0-connector deployment (Helm value operator.agent0Connector.tolerations).

Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or similar to take effect.

Preventing Operator Scheduling on Specific Nodes

All the pods deployed by the operator have a default node anti-affinity for the dash0.com/enable=false node label. That is, if you add the dash0.com/enable=false label to a node, none of the pods owned by the operator will be scheduled on that node.

IMPORTANT: This includes the daemonset that the operator will set up to receive telemetry from the pods, which might lead to situations in which instrumented pods cannot send telemetry because the local node does not have a daemonset collector pod. In other words, if you want to monitor workloads with the Dash0 operator and use the dash0.com/enable=false node anti-affinity, make sure that the workloads you want to monitor have the same anti-affinity:

yaml
# Add this to your workloads
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: "dash0.com/enable"
            operator: "NotIn"
            values: ["false"]

Custom Node Affinity

The node affinity for all pods deployed by the operator can be customized by setting the nodeAffinity field for the respective component.

yaml
operator:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: "a-custom-label"
              operator: "In"
              values: ["custom_value"]

  collectors:
    daemonSetNodeAffinity: <custom_node_affinity>
    deploymentNodeAffinity: <custom_node_affinity>
    signalControlCollectorNodeAffinity: <custom_node_affinity>

  targetAllocator:
    nodeAffinity: <custom_node_affinity>

  agent0Connector:
    nodeAffinity: <custom_node_affinity>

Adding Custom Labels and Annotations to the Collector Resources

Additional labels and annotations can be added to the collector resources, the target-allocator resource, and the agent0-connector resource managed by the operator, both to the workload objects (the daemonset and the deployments) and to their pods. These are merged with the labels and annotations the operator sets itself; the operator-managed entries always take precedence, so they cannot be overridden.

yaml
operator:
  collectors:
    # labels/annotations for the daemonset collector
    daemonSetLabels:
      my-label: my-value
    daemonSetAnnotations:
      my-annotation: my-value
    daemonSetPodLabels:
      my-pod-label: my-value
    daemonSetPodAnnotations:
      my-pod-annotation: my-value

    # labels/annotations for the cluster-metrics-collector deployment
    deploymentLabels:
      my-label: my-value
    deploymentAnnotations:
      my-annotation: my-value
    deploymentPodLabels:
      my-pod-label: my-value
    deploymentPodAnnotations:
      my-pod-annotation: my-value

    # labels/annotations for the SignalControl collector deployment
    signalControlCollectorLabels:
      my-label: my-value
    signalControlCollectorAnnotations:
      my-annotation: my-value
    signalControlCollectorPodLabels:
      my-pod-label: my-value
    signalControlCollectorPodAnnotations:
      my-pod-annotation: my-value

  targetAllocator:
    # labels/annotations for the target-allocator deployment
    labels:
      my-label: my-value
    annotations:
      my-annotation: my-value
    podLabels:
      my-pod-label: my-value
    podAnnotations:
      my-pod-annotation: my-value

  agent0Connector:
    enabled: true
    # [... other settings like serverAddress etc.]
    labels:
      my-label: my-value
    annotations:
      my-annotation: my-value
    podLabels:
      my-pod-label: my-value
    podAnnotations:
      my-pod-annotation: my-value

Configuring Collector Host Ports

The OpenTelemetry collector DaemonSet managed by the operator uses the host ports 40317 for gRPC and 40318 for HTTP, for node-local traffic. Different host ports can be configured via Helm by setting operator.collectors.otlpGrpcHostPort and operator.collectors.otlpHttpHostPort.

Configuring Pod-Level sysctls for the Collector Pods (TCP Keepalive)

Pod-level sysctls can be applied to the collector pods via operator.collectors.daemonSetSysctls (for the daemonset collector), operator.collectors.deploymentSysctls (for the cluster-metrics-collector deployment) and operator.collectors.signalControlCollectorSysctls (for the SignalControl collector deployment). All three default to being unset, so no sysctls are applied unless you configure them.

The primary use case is forcing TCP keepalive on the collector's network namespace. On some environments the connection-tracking layer reaps idle established TCP connections after a relatively short timeout (for example, AWS Nitro v6 instance types such as m8i, m8i-flex, r8i and c8i reap them after 350 seconds). The collector's self-monitoring connection is sparse and can idle longer than that; once the conntrack entry is reaped, the next config reload blocks on the export deadline and the collector pod is restarted. Lowering the TCP keepalive interval below the conntrack idle timeout keeps the connection alive and avoids these restarts:

yaml
operator:
  collectors:
    daemonSetSysctls:
      - name: net.ipv4.tcp_keepalive_time
        value: "240"
      - name: net.ipv4.tcp_keepalive_intvl
        value: "60"
      - name: net.ipv4.tcp_keepalive_probes
        value: "3"
    deploymentSysctls:
      - name: net.ipv4.tcp_keepalive_time
        value: "240"
      - name: net.ipv4.tcp_keepalive_intvl
        value: "60"
      - name: net.ipv4.tcp_keepalive_probes
        value: "3"

net.ipv4.tcp_keepalive_time is the idle time before the first keepalive probe is sent and must be lower than the environment's conntrack idle timeout (for example, below 350 seconds on AWS Nitro v6); tcp_keepalive_intvl and tcp_keepalive_probes only govern dead-peer detection.

This setting is opt-in for two reasons:

  • The net.ipv4.tcp_keepalive_* sysctls are only considered safe sysctls on Kubernetes 1.29 and later. On older clusters, or on clusters where the kubelet's forbiddenSysctls / a strict Pod Security Admission policy disallows them, the kubelet rejects these sysctls and the collector pods will fail to be scheduled.
  • It changes the collector pod spec, which triggers a rollout of the collector pods.

Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or similar to take effect.

Configuring SELinux Options for the Collector DaemonSet

Pod-level SELinux options for the collector DaemonSet pods can be set via operator.collectors.daemonSetSeLinuxOptions. The value is used as is for the pod's spec.securityContext.seLinuxOptions. It is unset by default. On OpenShift (operator.openShift.enabled=true), the operator uses the type spc_t unless this value is set, see Notes on OpenShift. Setting it to {} removes the SELinux options. Changing it changes the collector pod spec, which triggers a rollout of the collector DaemonSet pods.

yaml
operator:
  collectors:
    daemonSetSeLinuxOptions:
      type: <selinux-type>

Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or similar to take effect.

Disable Self-Monitoring

By default, self-monitoring is enabled for the Dash0 operator as soon as you deploy a Dash0 operator configuration resource with exports. That means, the operator will send self-monitoring telemetry to the configured Dash0 backend. Disabling self-monitoring is available as a setting on the Dash0 operator configuration resource. Dash0 does not recommend to disable the operator's self-monitoring.

Here is an example with self-monitoring disabled:

yaml
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration-resource
spec:
  selfMonitoring:
    enabled: false
  exports:
    - # ... see configuration.md for details on the exports settings

For more details on self-monitoring, see Self-Monitoring.

Exporting Data to Other Observability Backends

Instead of spec.exports[].dash0 in the Dash0 operator configuration resource, you can also provide spec.exports[].http or spec.exports[].grpc to export telemetry data to arbitrary OTLP-compatible backends, or to another local OpenTelemetry collector.

Here is an example for HTTP:

yaml
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  exports:
    - http:
        endpoint: ... # provide the OTLP HTTP endpoint of your observability backend here
        headers: # you can optionally provide additional headers, for example for authorization
          - name: X-My-Header
            value: my-value
        encoding: json # optional, can be "json" or "proto", defaults to "proto"

Here is an example for gRPC:

yaml
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  exports:
    - grpc:
        endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here
        headers: # you can optionally provide additional headers, for example for authorization
          - name: X-My-Header
            value: my-value

Endpoint Formats for gRPC Exports

The endpoint of a grpc export can have one of the following formats:

FormatExample
host:port (no scheme)otel-collector.observability:4317
https://host:porthttps://otel-collector.observability:4317
http://host:porthttp://otel-collector.observability:4317
dns:///host:portdns:///otel-collector.observability:4317

Configuring Other Backends Via Helm

The same exports can be configured via the Helm value operator.exports, which lets the operator create the operator configuration resource for you instead of applying it with kubectl. The value is a list with exactly the structure of spec.exports shown above:

yaml
operator:
  exports:
    - grpc:
        endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here
        headers:
          - name: X-My-Header
            value: my-value

Setting operator.exports to a non-empty list makes the operator create the operator configuration resource at startup, just like operator.dash0Export.enabled=true does; see Notes on Creating the Operator Configuration Resource Via Helm. For any setup with more than one export, we recommend configuring all exports via operator.exports, including the dash0 export(s), so that all exports are defined in one place:

yaml
operator:
  exports:
    - dash0:
        endpoint: ... # provide the OTLP gRPC endpoint of your Dash0 organization here
        authorization:
          secretRef:
            name: dash0-authorization-secret
            key: token
        apiEndpoint: ... # provide the API endpoint of your Dash0 organization here
    - grpc:
        endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here

Combining operator.dash0Export.* with operator.exports is also possible: the Dash0 export configured via operator.dash0Export.* comes first, followed by the exports listed in operator.exports. Configuring a dash0 export in operator.exports while operator.dash0Export.enabled is true is an error; use one of the two, not both.

The order matters for the operator's self-monitoring telemetry, which is only sent to the first export. Self-monitoring is not supported for an http export with encoding: json and is silently disabled in that case.

Note that the values of operator.exports are rendered verbatim into a Kubernetes ConfigMap. For header values that contain sensitive data, use valueFrom.secretKeyRef (see the next section) instead of a literal value.

Two operator features require a Dash0 export and are therefore not available with http or grpc exports alone: synchronizing dashboards, check rules, synthetic checks, views, notification channels, spam filters and signal-to-metrics rules (which needs dash0.apiEndpoint), and Signal Control (which needs a Dash0 export with an auth token).

Providing Header Values via a Kubernetes Secret

Header values often contain sensitive data, such as authorization tokens or API keys. Instead of providing such a value directly via the value property, you can source it from a Kubernetes secret by using the valueFrom property. This way, the sensitive value does not have to be stored in plain text in the operator configuration (or monitoring) resource.

For each header, you have to provide exactly one of value or valueFrom (providing both, or neither, is a validation error).

The following example sources the value of the Authorization header from the key api-key of the secret my-export-secret:

yaml
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  exports:
    - grpc:
        endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here
        headers:
          - name: Authorization
            valueFrom:
              secretKeyRef:
                name: my-export-secret # the name of the Kubernetes secret
                key: api-key           # the key within the secret holding the header value

The secret has to exist in the namespace of the Dash0 operator (that is, the namespace the operator has been deployed to). The valueFrom.secretKeyRef mechanism is supported for both grpc and http exports, and works the same way for header values provided via a Dash0 monitoring resource.

Exporting to Multiple Backends

Export to multiple backends is also supported. The supplied backends can be either of the same type or of different types. In the following example the telemetry would be sent to two different datasets in Dash0 (referenced by their dataset identifiers) and in addition to a gRPC endpoint:

yaml
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  exports:
    - dash0:
        dataset: dataset-one
        endpoint: ingress... # TODO needs to be replaced with the actual value
        authorization:
          token: auth_... # TODO needs to be replaced with the actual value
    - dash0:
        dataset: dataset-two
        endpoint: ingress... # TODO needs to be replaced with the actual value
        authorization:
          token: auth_... # TODO needs to be replaced with the actual value
    - grpc:
        endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here

For more details on configuring exports, see Configuring the Dash0 Backend Connection.

Note regarding TLS when using arbitrary OTLP-compatible backends

gRPC

  • By default, a secure connection is assumed, unless explicitly setting insecure: true, or when the insecure field is omitted and the endpoint URL starts with http://
  • When using TLS, you can set insecureSkipVerify: true to disable the verification of the server's certificate chain, which can be useful when using self-signed certificates.

Here's an example using insecureSkipVerify:

yaml
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  exports:
    - grpc:
        endpoint: ...             # provide the secure OTLP gRPC endpoint of your observability backend here
        insecureSkipVerify: true  # disables the verification of the server's certificate chain

Please note that it is a validation error to set both insecure and insecureSkipVerify explicitly to true at the same time, since insecureSkipVerify is only applicable when using TLS.

HTTP

  • For HTTP, the connection security is automatically detected based on whether the endpoint URL starts with http:// or https://
  • When using TLS, you can set insecureSkipVerify: true to disable the verification of the server's certificate chain, which can be useful when using self-signed certificates.

Self-Monitoring Telemetry

For self-monitoring telemetry, insecureSkipVerify is only applied by the operator manager, for both grpc and http exports. The OpenTelemetry collectors and other operator components send their self-monitoring telemetry with certificate verification, which fails if the certificate of the endpoint cannot be verified.