Last updated:
Advanced Configuration
This guide covers advanced configuration topics for the Dash0 operator, including exporting to other backends, certificate management, collector pod scheduling, and performance tuning.
Table of Contents
- Providing a Filelog Offset Volume
- Webhook Certificates
- Using cert-manager
- Tuning Resource Requests and Limits
- Controlling On Which Nodes the Operator's Collector Pods Are Scheduled
- Configuring Collector Host Ports
- Configuring Pod-Level sysctls for the Collector Pods (TCP Keepalive)
- Configuring SELinux Options for the Collector DaemonSet
- Disable Self-Monitoring
- Exporting Data to Other Observability Backends
Providing a Filelog Offset Volume
The operator's collector uses the filelog receiver to read pod log files for monitored workloads. When the collector is restarted (which can happen for various reasons, for example to apply configuration changes), it is important that the filelog receiver can continue reading the log files from where it left off. If the filelog receiver started to read all log files from the beginning again after a restart, log records would be duplicated, that is, they would appear multiple times in Dash0.
For that purpose, the filelog receiver stores the log file offsets in persistent storage. By default, the offsets are stored in a config map in the operator's namespace. For small clusters, this is usually sufficient, and it requires no additional configuration by users. For production clusters, larger staging clusters, or clusters with many short-lived pods, we recommend providing a persistent volume for storing offsets.
Any persistent volume that is accessible from the collector pods can be used for this purpose.
Here is an example with a hostPath volume (see also https://kubernetes.io/docs/concepts/storage/volumes/#hostpath for
considerations around using hostPath volumes):
operator:
collectors:
filelogOffsetSyncStorageVolume:
name: filelogreceiver-offsets
hostPath:
path: /data/dash0-operator/offset-storage
type: DirectoryOrCreateThe directory in the hostPath volume will automatically be created with the correct permissions so that the OpenTelemetry collector container can write to it.
Here is another example based on persistent volume claims.
(This assumes that a PersistentVolumeClaim named offset-storage-claim exists.)
See also https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims and
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming.
operator:
collectors:
filelogOffsetSyncStorageVolume:
name: filelogreceiver-offsets
persistentVolumeClaim:
claimName: offset-storage-claimImportant: Since this volume is needed by a Daemonset run by the Dash0 operator, the PersistentVolumeClaim needs to be set with the
ReadWriteManyaccess mode.
When to Use a Volume for Filelog Offsets
Using a volume instead of the default config map approach is also helpful if you have webhooks in your cluster which process every config map update.
Known examples for this are:
- The Open Police Agent, in particular the OPA gatekeeeper. When operating the OPA gatekeeper in the same cluster as the Dash0 operator, it is highly recommended to use a volume for filelog offsets. Using the default config map filelog offset storage in clusters with the OPA gatekeeper can lead to severe performance issues, since the default config map for filelog offsets is updated very frequently. This can cause the OPA gatekeeper to consume a lot of CPU and memory resources, potentially even leading to OOMKills of the OPA gatekeeper.
- AKS clusters with the Azure Policy add-on: A managed instance of the OPA gatekeeper (see above) is installed by the Azure Policy add-on for AKS, i.e. the OPA gatekeeper is active in AKS cluster that have enabled the Azure Policy add-on. It is highly recommended to use a volume for filelog offsets for AKS clusters with the Azure Policy add-on.
- The Kyverno admission controller. When operating Kyverno in the same cluster as the Dash0 operator, it is highly recommended to either exclude ConfigMaps (or all resource types) in the Dash0 operator's namespace from Kyverno's processing, or to use a volume for filelog offsets. Leaving Kyverno processing in place and using the config map filelog offset storage can lead to severe performance issues, since the default config map for filelog offsets is updated very frequently. This can cause Kyverno to consume a lot of CPU and memory resources, potentially even leading to OOMKills of the Kyverno admission controller. Note that a filelog offset volume adds an init container that runs as root, which restrictive Kyverno policies reject, see Kyverno Policies and the Collector DaemonSet.
For more details on OPA, Kyverno, and other platform-specific considerations, see Platform Specific.
Webhook Certificates
By default, the operator manages the TLS certificates for its webhooks itself, without the need for a third-party
solution.
It stores them in the Secret dash0-operator-webhook-server-cert in the operator's namespace, writes the CA into the
caBundle fields of its webhook configurations and of the dash0monitorings.operator.dash0.com CRD, and renews the
certificates before they expire.
Applying the Helm chart again, for example via helm upgrade or an ArgoCD sync, does not change the certificates.
Things to be aware of:
- If an admission policy (for example Kyverno or OPA Gatekeeper) prevents the operator from updating the Secret or from patching its webhook configurations or the CRD, requests to the webhooks fail, and the operator pods might not become ready. The operator logs errors in this case.
- To replace the CA, for example after its key has been leaked, delete the Secret and run
helm upgradeor an ArgoCD sync right after it, which creates the Secret again. New operator pods do not become ready until the Secret exists again. Requests to the webhooks fail briefly until the operator has written the new CA. - With Flux drift detection, add the ignore rules from
Operator Version 0.158.0: Operator-Managed Webhook Certificates
to the
HelmRelease, also for new installations. Otherwise, Flux resets thecaBundlevalues whenever the operator changes them, and requests to the webhooks fail briefly each time. - Do not apply the operator's resources with options that replace objects instead of updating them, like
helm upgrade --force, Flux'supgrade.force, or ArgoCD'sReplace=trueorForce=truesync options. They remove the certificates or thecaBundlevalues, and requests to the webhooks fail until the operator has written them again.
For upgrading from a version before 0.158.0, see Operator Version 0.158.0: Operator-Managed Webhook Certificates.
Using cert-manager
As an alternative to the operator-managed certificates, you can use cert-manager to manage
the certificates.
To let cert-manager handle TLS certificates, provide the following additional settings when applying the Helm chart:
operator:
# Settings for using cert-manager instead of auto-generating TLS certificates.
certManager:
# This disables the certificates managed by the operator.
# If this is set to true, the assumption is that certificates are managed by
# cert-manager (see https://cert-manager.io/), and the other settings shown in this
# snippet (certManager.secretName, certManager.certManagerAnnotations) become
# required settings. It is recommended to set webhookService.name as well.
useCertManager: true
# The name of the secret used by cert-manager for the certificate.
# The provided name must match the `secretName` in the `cert-manager.io/v1.Certificate`
# resource's spec (see below).
# Note: This secret is created and managed by cert-manager, you do not need to create
# it manually.
secretName: dash0-operator-certificate-secret
# A map of additional annotations that are added to all Kubernetes resources that need
# the certificate.
# Usually this will be a single `cert-manager.io/inject-ca-from` annotation with the
# namespace and name of the Certificate resource (see below), but other annotations
# are possible, see https://cert-manager.io/docs/reference/annotations/.
certManagerAnnotations:
cert-manager.io/inject-ca-from: "dash0-system/dash0-operator-serving-certificate"
webhookService:
# A name override for the webhook service, defaults to dash0-operator-webhook-service.
# The name of the webhook service must match the DNS names provided in the
# cert-manager's Certificate resource.
# For that reason it is recommended to set this value when using cert-manager, as it
# guarantees that the names match, even if the Dash0 operator helm chart would change
# the default name of the webhook service in a future release.
name: dash0-operator-webhook-service-nameYou will also need to provide cert-manager resources, for example a Certificate and an Issuer.
Explaining all configuration options of cert-manager is out of scope for this documentation.
Please refer to the cert-manager documentation for details on how to install and
configure cert-manager in your cluster.
The following annotated example is a minimal configuration that matches the Dash0 operator configuration snippet shown
above.
Both configuration snippets assume that the Dash0 operator is installed in the default dash0-system namespace, and the
Certificate and Issuer resource are also deployed into that namespace.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
# NOTE: This value, together with the namespace this resource is deployed into, must
# match the value of the cert-manager.io/inject-ca-from annotation provided in the
# certManager.certManagerAnnotations setting, i.e. this matches the following annotation:
# cert-manager.io/inject-ca-from: "dash0-system/dash0-operator-serving-certificate"
name: dash0-operator-serving-certificate
labels:
app.kubernetes.io/name: certificate
app.kubernetes.io/instance: serving-cert
app.kubernetes.io/component: certificate
spec:
# NOTE: Provide all DNS names that are used to access the webhook service.
# The service names usually follow the patterns <service-name>.<namespace>.svc and
# <service-name>.<namespace>.svc.cluster.local.
# If you use a custom name for the Dash0 operator's webhook service (recommended), the
# first part of the DNS names must match that name.
# If you use the default name for the webhook service, the first part of the DNS names
# must match the default name (dash0-operator-webhook-service).
# If the Dash0 operator is installed into a different namespace, you need to change the
# ".dash0-system." part of the DNS names accordingly.
dnsNames:
- dash0-operator-webhook-service-name.dash0-system.svc
- dash0-operator-webhook-service-name.dash0-system.svc.cluster.local
issuerRef:
kind: Issuer
# NOTE: Must match the name of the Issuer resource provided below.
name: dash0-operator-selfsigned-issuer
# NOTE: The secret name must match the operator.certManager.secretName setting above
# provided to the Dash0 operator Helm chart. This secret is created and managed by
# cert-manager, you do not need to create it manually.
secretName: dash0-operator-certificate-secret
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
# NOTE: This issuer's name must match the issuerRef.name in the Certificate resource above.
name: dash0-operator-selfsigned-issuer
labels:
app.kubernetes.io/name: certificate
app.kubernetes.io/instance: serving-cert
app.kubernetes.io/component: certificate
spec:
# There are many options for configuring an Issuer, this example uses a simple self-signed
# issuer. Make sure to configure the issuer according to your requirements.
selfSigned: {}Tuning Resource Requests and Limits
The operator sets resource requests and limits on the workloads it deploys, including the operator manager, the DaemonSet collector pods (one per node), the cluster-metrics-collector Deployment and, when SignalControl Edge is enabled, the SignalControl collector Deployment. Each of these ships with default requests and limits that are a reasonable starting point for many clusters.
However, these defaults are not guaranteed to be optimal for every use case. The amount of CPU and memory a component actually needs depends heavily on the specifics of your cluster, for example the number of nodes, the number and size of monitored workloads, the volume of telemetry (logs, metrics, traces) being processed, and how frequently Kubernetes resources (workloads, pods etc.) are created and destroyed. You should therefore treat the defaults as a baseline and tune them according to what is required in your particular Kubernetes cluster.
A good strategy is to start with relatively low limits and increase them if they turn out to be insufficient.
Signs that a limit is too low include the container being OOMKilled (memory limit too low) or being CPU-throttled and
lagging behind (CPU limit too low).
Conversely, if a component consistently uses far less than its requests reserve, you can lower the requests to avoid
needlessly reserving capacity on your nodes.
Dash0's self-monitoring can help you observe the actual resource usage of the operator's components over time, which
makes it easier to find appropriate values.
The following Helm values control the resource settings, all nested under the top-level operator key:
| Helm value | Workload / container | Default requests | Default limits |
|---|---|---|---|
managerContainerResources | operator manager | cpu: 50m, memory: 128Mi, ephemeral-storage: 500Mi | cpu: 500m, memory: 256Mi, ephemeral-storage: 500Mi |
initContainerResources | auto-instrumentation init container | none | none |
collectors.daemonSetCollectorContainerResources | DaemonSet collector container | memory: 500Mi | memory: 500Mi |
collectors.daemonSetConfigurationReloaderContainerResources | DaemonSet configuration reloader | memory: 12Mi | memory: 26Mi |
collectors.daemonSetFileLogOffsetSyncContainerResources | DaemonSet filelog offset sync | memory: 32Mi | memory: 32Mi |
collectors.deploymentCollectorContainerResources | cluster-metrics-collector container | memory: 500Mi | memory: 500Mi |
collectors.deploymentConfigurationReloaderContainerResources | cluster-metrics-collector configuration reloader | memory: 12Mi | memory: 26Mi |
collectors.signalControlCollectorContainerResources | SignalControl collector container | memory: 1Gi | memory: 1Gi |
collectors.signalControlCollectorConfigurationReloaderContainerResources | SignalControl collector configuration reloader | memory: 12Mi | memory: 26Mi |
targetAllocator.containerResources | target-allocator container | cpu: 200m, memory: 128Mi | cpu: 200m, memory: 500Mi |
agent0Connector.containerResources | agent0-connector container | memory: 64Mi | memory: 256Mi |
For every container above except the operator manager, gomemlimit is left empty by default and the operator derives it
from the container's memory limit (about 80%, or about 60% for the agent0-connector, which also runs kubectl as a
child process). For the three collector containers it additionally derives the internal memory_limiter thresholds,
keeping GOMEMLIMIT < memory_limiter soft limit < memory_limiter hard limit < memory limit; keep their memory limit at
or above the 500Mi default to leave enough off-heap headroom. Set gomemlimit explicitly only to override the derived
value; for a collector it must stay below the derived soft limit, or the operator logs a warning.
The SignalControl collector only exists when SignalControl Edge is enabled. Unlike the other two collectors it
processes the Dash0-bound telemetry of the whole cluster, so its memory requirement scales with total telemetry volume
rather than with per-node volume. Its replica count is configured separately via
operator.collectors.signalControlCollectorReplicas (default 2). When the tail-sampling reservoir of the
Dash0SignalControl resource uses the disk type, the operator derives an ephemeral-storage request from its
maxDiskBytes (unless an ephemeral-storage request is set explicitly). The ephemeral-storage limit of the
SignalControl collector container must be at least that derived value, even with an explicit request, since the
reservoir counts towards the limit. Otherwise, creating a Dash0SignalControl resource or changing its sampling or
components is rejected.
Its replicas are distributed as evenly as possible over the cluster's availability zones and nodes: the operator sets
topologySpreadConstraints with maxSkew: 1 on topology.kubernetes.io/zone and on kubernetes.io/hostname. Since
maxSkew bounds the difference between the per-zone pod counts rather than the count itself, more replicas than zones
simply means several replicas per zone, spread evenly; fewer replicas than zones means each replica gets its own zone.
Both constraints use whenUnsatisfiable: ScheduleAnyway, so they are a scheduling preference and never leave a
replica unschedulable — in particular on clusters whose nodes carry no topology.kubernetes.io/zone label. The
operator also creates a PodDisruptionBudget with maxUnavailable: 1 for this collector, so a node drain or cluster
upgrade cannot take down more than one replica at a time.
Set the replica count to at least the number of availability zones that run monitored workloads. The SignalControl collector's service is configured for zone-aware routing (see SignalControl Edge), which only keeps traffic inside a zone that actually has a SignalControl collector pod. With fewer replicas than zones, the uncovered zones keep sending across zone boundaries. The operator logs a warning when it detects this.
The settings for managerContainerResources, collectors.daemonSetCollectorContainerResources and
collectors.deploymentCollectorContainerResources are the ones that are most likely to require tuning.
Here is an example that overrides the resource settings for the operator manager and the DaemonSet collector:
operator:
managerContainerResources:
requests:
cpu: 100m
memory: 192Mi
ephemeral-storage: 500Mi
limits:
cpu: 1000m
memory: 512Mi
ephemeral-storage: 1Gi
# approximately 80% of managerContainerResources.limits.memory; use MiB or GiB here (see https://pkg.go.dev/runtime#hdr-Environment_Variables).
gomemlimit: 410MiB
collectors:
daemonSetCollectorContainerResources:
requests:
memory: 768Mi
limits:
memory: 768Mi
# gomemlimit is left empty: the operator derives GOMEMLIMIT and the memory_limiter thresholds from the limit above.Note: Most of the operator's containers are Go processes whose
GOMEMLIMIT(the environment variable, a soft memory limit for the Go runtime's garbage collector) is derived automatically from the container's memory limit whengomemlimitis left empty: the three collectors (which also derive theirmemory_limiterthresholds), the target-allocator, edge-proxy, agent0-connector, configuration reloaders and filelog offset sync container. For these you normally adjust only the memory limit. The one exception is the operator manager (managerContainerResources), whosegomemlimitis set statically — whenever you raise or lower its memory limit, adjust itsgomemlimitas well (about 80% of the limit). Note thatgomemlimituses different units than Kubernetes resource settings (MiB/GiBrather thanMi/Gi).
Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or
similar to take effect.
Controlling On Which Nodes the Operator's Collector Pods Are Scheduled
Allow Scheduling on Tainted Nodes
The operator uses a Kubernetes daemonset to deploy the OpenTelemetry collector on each node; to collect telemetry from that node and workloads running on that node. If you use taints on certain nodes, Kubernetes will not schedule any pods there, preventing the daemonset collector pods to be present on these nodes. You can allow the daemonset collector pods to be scheduled there by configuring tolerations matching your taints for the collector pods. Tolerations can be configured as follows:
operator:
collectors:
daemonSetTolerations:
- key: key1
operator: Equal
value: value1
effect: NoSchedule
- key: key2
operator: Exists
effect: NoScheduleIn the same fashion, tolerations can also be configured for the Dash0 operator manager (Helm value
operator.tolerations), the OpenTelemetry collector deployment for collecting cluster metrics
(Helm value operator.collectors.deploymentTolerations), the SignalControl collector deployment
(Helm value operator.collectors.signalControlCollectorTolerations), the OpenTelemetry target-allocator deployment
(Helm value operator.targetAllocator.tolerations) and the agent0-connector deployment (Helm value
operator.agent0Connector.tolerations).
Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or similar to take effect.
Preventing Operator Scheduling on Specific Nodes
All the pods deployed by the operator have a default node anti-affinity for the dash0.com/enable=false node label.
That is, if you add the dash0.com/enable=false label to a node, none of the pods owned by the operator will be
scheduled on that node.
IMPORTANT: This includes the daemonset that the operator will set up to receive telemetry from the pods, which might lead to situations in which instrumented pods cannot send telemetry because the local node does not have a daemonset collector pod. In other words, if you want to monitor workloads with the Dash0 operator and use the
dash0.com/enable=falsenode anti-affinity, make sure that the workloads you want to monitor have the same anti-affinity:
# Add this to your workloads
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "dash0.com/enable"
operator: "NotIn"
values: ["false"]Custom Node Affinity
The node affinity for all pods deployed by the operator can be customized by setting the nodeAffinity field for the
respective component.
operator:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "a-custom-label"
operator: "In"
values: ["custom_value"]
collectors:
daemonSetNodeAffinity: <custom_node_affinity>
deploymentNodeAffinity: <custom_node_affinity>
signalControlCollectorNodeAffinity: <custom_node_affinity>
targetAllocator:
nodeAffinity: <custom_node_affinity>
agent0Connector:
nodeAffinity: <custom_node_affinity>Adding Custom Labels and Annotations to the Collector Resources
Additional labels and annotations can be added to the collector resources, the target-allocator resource, and the agent0-connector resource managed by the operator, both to the workload objects (the daemonset and the deployments) and to their pods. These are merged with the labels and annotations the operator sets itself; the operator-managed entries always take precedence, so they cannot be overridden.
operator:
collectors:
# labels/annotations for the daemonset collector
daemonSetLabels:
my-label: my-value
daemonSetAnnotations:
my-annotation: my-value
daemonSetPodLabels:
my-pod-label: my-value
daemonSetPodAnnotations:
my-pod-annotation: my-value
# labels/annotations for the cluster-metrics-collector deployment
deploymentLabels:
my-label: my-value
deploymentAnnotations:
my-annotation: my-value
deploymentPodLabels:
my-pod-label: my-value
deploymentPodAnnotations:
my-pod-annotation: my-value
# labels/annotations for the SignalControl collector deployment
signalControlCollectorLabels:
my-label: my-value
signalControlCollectorAnnotations:
my-annotation: my-value
signalControlCollectorPodLabels:
my-pod-label: my-value
signalControlCollectorPodAnnotations:
my-pod-annotation: my-value
targetAllocator:
# labels/annotations for the target-allocator deployment
labels:
my-label: my-value
annotations:
my-annotation: my-value
podLabels:
my-pod-label: my-value
podAnnotations:
my-pod-annotation: my-value
agent0Connector:
enabled: true
# [... other settings like serverAddress etc.]
labels:
my-label: my-value
annotations:
my-annotation: my-value
podLabels:
my-pod-label: my-value
podAnnotations:
my-pod-annotation: my-valueConfiguring Collector Host Ports
The OpenTelemetry collector DaemonSet managed by the operator uses the host ports 40317 for gRPC and 40318 for HTTP,
for node-local traffic.
Different host ports can be configured via Helm by setting operator.collectors.otlpGrpcHostPort and
operator.collectors.otlpHttpHostPort.
Configuring Pod-Level sysctls for the Collector Pods (TCP Keepalive)
Pod-level sysctls can be applied to the
collector pods via operator.collectors.daemonSetSysctls (for the daemonset collector),
operator.collectors.deploymentSysctls (for the cluster-metrics-collector deployment) and
operator.collectors.signalControlCollectorSysctls (for the SignalControl collector deployment).
All three default to being unset, so no sysctls are applied unless you configure them.
The primary use case is forcing TCP keepalive on the collector's network namespace.
On some environments the connection-tracking layer reaps idle established TCP connections after a relatively short
timeout (for example, AWS Nitro v6 instance types such as m8i, m8i-flex, r8i and c8i reap them after 350
seconds).
The collector's self-monitoring connection is sparse and can idle longer than that; once the conntrack entry is reaped,
the next config reload blocks on the export deadline and the collector pod is restarted.
Lowering the TCP keepalive interval below the conntrack idle timeout keeps the connection alive and avoids these
restarts:
operator:
collectors:
daemonSetSysctls:
- name: net.ipv4.tcp_keepalive_time
value: "240"
- name: net.ipv4.tcp_keepalive_intvl
value: "60"
- name: net.ipv4.tcp_keepalive_probes
value: "3"
deploymentSysctls:
- name: net.ipv4.tcp_keepalive_time
value: "240"
- name: net.ipv4.tcp_keepalive_intvl
value: "60"
- name: net.ipv4.tcp_keepalive_probes
value: "3"net.ipv4.tcp_keepalive_time is the idle time before the first keepalive probe is sent and must be lower than the
environment's conntrack idle timeout (for example, below 350 seconds on AWS Nitro v6); tcp_keepalive_intvl and
tcp_keepalive_probes only govern dead-peer detection.
This setting is opt-in for two reasons:
- The
net.ipv4.tcp_keepalive_*sysctls are only considered safe sysctls on Kubernetes 1.29 and later. On older clusters, or on clusters where the kubelet'sforbiddenSysctls/ a strict Pod Security Admission policy disallows them, the kubelet rejects these sysctls and the collector pods will fail to be scheduled. - It changes the collector pod spec, which triggers a rollout of the collector pods.
Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or similar to take effect.
Configuring SELinux Options for the Collector DaemonSet
Pod-level SELinux options for the
collector DaemonSet pods can be set via operator.collectors.daemonSetSeLinuxOptions. The value is used as is for the
pod's spec.securityContext.seLinuxOptions. It is unset by default. On OpenShift (operator.openShift.enabled=true),
the operator uses the type spc_t unless this value is set, see Notes on OpenShift.
Setting it to {} removes the SELinux options. Changing it changes the collector pod spec, which triggers a rollout of
the collector DaemonSet pods.
operator:
collectors:
daemonSetSeLinuxOptions:
type: <selinux-type>Changing Helm settings while the operator is already running requires a helm upgrade/helm upgrade --reuse-values or
similar to take effect.
Disable Self-Monitoring
By default, self-monitoring is enabled for the Dash0 operator as soon as you deploy a Dash0 operator configuration resource with exports. That means, the operator will send self-monitoring telemetry to the configured Dash0 backend. Disabling self-monitoring is available as a setting on the Dash0 operator configuration resource. Dash0 does not recommend to disable the operator's self-monitoring.
Here is an example with self-monitoring disabled:
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
name: dash0-operator-configuration-resource
spec:
selfMonitoring:
enabled: false
exports:
- # ... see configuration.md for details on the exports settingsFor more details on self-monitoring, see Self-Monitoring.
Exporting Data to Other Observability Backends
Instead of spec.exports[].dash0 in the Dash0 operator configuration resource, you can also provide spec.exports[].http or spec.exports[].grpc to export telemetry data to arbitrary OTLP-compatible backends, or to another local OpenTelemetry collector.
Here is an example for HTTP:
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
name: dash0-operator-configuration
spec:
exports:
- http:
endpoint: ... # provide the OTLP HTTP endpoint of your observability backend here
headers: # you can optionally provide additional headers, for example for authorization
- name: X-My-Header
value: my-value
encoding: json # optional, can be "json" or "proto", defaults to "proto"Here is an example for gRPC:
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
name: dash0-operator-configuration
spec:
exports:
- grpc:
endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here
headers: # you can optionally provide additional headers, for example for authorization
- name: X-My-Header
value: my-valueEndpoint Formats for gRPC Exports
The endpoint of a grpc export can have one of the following formats:
| Format | Example |
|---|---|
host:port (no scheme) | otel-collector.observability:4317 |
https://host:port | https://otel-collector.observability:4317 |
http://host:port | http://otel-collector.observability:4317 |
dns:///host:port | dns:///otel-collector.observability:4317 |
Configuring Other Backends Via Helm
The same exports can be configured via the Helm value operator.exports, which lets the operator create the operator
configuration resource for you instead of applying it with kubectl. The value is a list with exactly the structure of
spec.exports shown above:
operator:
exports:
- grpc:
endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here
headers:
- name: X-My-Header
value: my-valueSetting operator.exports to a non-empty list makes the operator create the operator configuration resource at startup,
just like operator.dash0Export.enabled=true does; see
Notes on Creating the Operator Configuration Resource Via Helm.
For any setup with more than one export, we recommend configuring all exports via operator.exports, including the
dash0 export(s), so that all exports are defined in one place:
operator:
exports:
- dash0:
endpoint: ... # provide the OTLP gRPC endpoint of your Dash0 organization here
authorization:
secretRef:
name: dash0-authorization-secret
key: token
apiEndpoint: ... # provide the API endpoint of your Dash0 organization here
- grpc:
endpoint: ... # provide the OTLP gRPC endpoint of your observability backend hereCombining operator.dash0Export.* with operator.exports is also possible: the Dash0 export configured via
operator.dash0Export.* comes first, followed by the exports listed in operator.exports. Configuring a dash0 export
in operator.exports while operator.dash0Export.enabled is true is an error; use one of the two, not both.
The order matters for the operator's self-monitoring telemetry, which is only sent to the first export. Self-monitoring
is not supported for an http export with encoding: json and is silently disabled in that case.
Note that the values of operator.exports are rendered verbatim into a Kubernetes ConfigMap. For header values that
contain sensitive data, use valueFrom.secretKeyRef (see the next section) instead of a literal value.
Two operator features require a Dash0 export and are therefore not available with http or grpc exports alone:
synchronizing dashboards, check rules, synthetic checks, views, notification channels, spam filters and
signal-to-metrics rules (which needs dash0.apiEndpoint), and Signal Control (which needs a Dash0 export with an auth
token).
Providing Header Values via a Kubernetes Secret
Header values often contain sensitive data, such as authorization tokens or API keys. Instead of providing such a value
directly via the value property, you can source it from a
Kubernetes secret by using the valueFrom property. This
way, the sensitive value does not have to be stored in plain text in the operator configuration (or monitoring) resource.
For each header, you have to provide exactly one of value or valueFrom (providing both, or neither, is a validation
error).
The following example sources the value of the Authorization header from the key api-key of the secret
my-export-secret:
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
name: dash0-operator-configuration
spec:
exports:
- grpc:
endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here
headers:
- name: Authorization
valueFrom:
secretKeyRef:
name: my-export-secret # the name of the Kubernetes secret
key: api-key # the key within the secret holding the header valueThe secret has to exist in the namespace of the Dash0 operator (that is, the namespace the operator has been deployed
to). The valueFrom.secretKeyRef mechanism is supported for both grpc and http exports, and works the same way for
header values provided via a Dash0 monitoring resource.
Exporting to Multiple Backends
Export to multiple backends is also supported. The supplied backends can be either of the same type or of different types. In the following example the telemetry would be sent to two different datasets in Dash0 (referenced by their dataset identifiers) and in addition to a gRPC endpoint:
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
name: dash0-operator-configuration
spec:
exports:
- dash0:
dataset: dataset-one
endpoint: ingress... # TODO needs to be replaced with the actual value
authorization:
token: auth_... # TODO needs to be replaced with the actual value
- dash0:
dataset: dataset-two
endpoint: ingress... # TODO needs to be replaced with the actual value
authorization:
token: auth_... # TODO needs to be replaced with the actual value
- grpc:
endpoint: ... # provide the OTLP gRPC endpoint of your observability backend hereFor more details on configuring exports, see Configuring the Dash0 Backend Connection.
Note regarding TLS when using arbitrary OTLP-compatible backends
gRPC
- By default, a secure connection is assumed, unless explicitly setting
insecure: true, or when theinsecurefield is omitted and the endpoint URL starts withhttp:// - When using TLS, you can set
insecureSkipVerify: trueto disable the verification of the server's certificate chain, which can be useful when using self-signed certificates.
Here's an example using insecureSkipVerify:
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
name: dash0-operator-configuration
spec:
exports:
- grpc:
endpoint: ... # provide the secure OTLP gRPC endpoint of your observability backend here
insecureSkipVerify: true # disables the verification of the server's certificate chainPlease note that it is a validation error to set both insecure and insecureSkipVerify explicitly to true at the same time, since insecureSkipVerify is only applicable when using TLS.
HTTP
- For HTTP, the connection security is automatically detected based on whether the endpoint URL starts with
http://orhttps:// - When using TLS, you can set
insecureSkipVerify: trueto disable the verification of the server's certificate chain, which can be useful when using self-signed certificates.
Self-Monitoring Telemetry
For self-monitoring telemetry, insecureSkipVerify is only applied by the operator manager, for both grpc and http
exports. The OpenTelemetry collectors and other operator components send their self-monitoring telemetry with
certificate verification, which fails if the certificate of the endpoint cannot be verified.
Related Documentation
- Configuration - Backend connections and namespace monitoring
- Platform Specific - Platform-specific notes including OPA and Kyverno compatibility
- Installation - Installation guide