Dash0 shows the manifest of a pod, workload, or node in the YAML tab of its sidebar in the Kubernetes views. The manifests come from the OpenTelemetry k8sobjectsreceiver, which watches the Kubernetes API and sends one log record per object change.

The Dash0 operator sets this up for you. If you run your own OpenTelemetry Collector instead, follow this page. Dash0 processes the records the same way, no matter which collector sends them. See How Dash0 Processes Kubernetes Object Logs for the details.

Requirements

An OpenTelemetry Collector distribution that includes the k8sobjectsreceiver, transformprocessor, redactionprocessor, and resourceprocessor, such as otelcol-contrib 0.151.0 or later. Older versions use different component names. The configuration on this page was tested with otelcol-contrib 0.164.0.
A collector that runs as a single replica, or several replicas with leader election. Each replica watches the Kubernetes API on its own, so every additional replica without leader election sends every record again. Do not add the receivers to a collector that runs as a DaemonSet, because every node would send every record. See Run more than one replica.

Grant read access to the objects

The collector's service account needs get, list, and watch on every resource type it watches:

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-k8s-objects
rules:
  - apiGroups: [""]
    resources: [pods, nodes, persistentvolumes]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, replicasets, statefulsets, daemonsets]
    verbs: [get, list, watch]

Bind the ClusterRole to the collector's service account with a ClusterRoleBinding. A ClusterRole is needed even when you watch a single namespace, because nodes and persistent volumes are cluster-scoped.

Configure the collector

Replace my-namespace with the namespaces whose objects you want to collect, and my-cluster with the name of your cluster.

yaml

receivers:
  k8s_objects/default:
    include_initial_state: true
    objects:
      - {name: deployments, mode: watch, namespaces: [my-namespace]}
      - {name: statefulsets, mode: watch, namespaces: [my-namespace]}
      - {name: daemonsets, mode: watch, namespaces: [my-namespace]}
      - {name: nodes, mode: watch}
      - {name: persistentvolumes, mode: watch}
  k8s_objects/highcardinality:
    include_initial_state: false
    objects:
      - {name: pods, mode: watch, namespaces: [my-namespace]}
      - {name: replicasets, mode: watch, namespaces: [my-namespace]}

processors:
  transform/k8s_objects:
    error_mode: ignore
    log_statements:
      - context: log
        conditions:
          - IsMap(log.body) and log.body["object"] != nil
        statements:
          - set(log.attributes["k8s.object.resource_version"], log.body["object"]["metadata"]["resourceVersion"]) where log.body["object"]["metadata"]["resourceVersion"] != nil
          - delete_key(log.body["object"]["metadata"], "managedFields")
          - delete_matching_keys(log.body["object"]["metadata"]["annotations"], "^(kubectl\\.kubernetes\\.io/last-applied-configuration|kapp\\.k14s\\.io/original|objectset\\.rio\\.cattle\\.io/applied|banzaicloud\\.com/last-applied)$") where log.body["object"]["metadata"]["annotations"] != nil
          - delete_key(log.body["object"]["status"], "images") where log.body["object"]["status"]["images"] != nil
  redaction/k8s_objects:
    allow_all_keys: true
    blocked_key_patterns:
      - ^value$
    blocked_values:
      - "(?i)(--?(password|passwd|token|secret|api[-_]?key)[= ])\\S+"
      - "(?i)bearer\\s+[A-Za-z0-9._~+/=-]+"
  resource/k8s_cluster:
    attributes:
      - key: k8s.cluster.name
        value: my-cluster
        action: upsert
  batch: {}

exporters:
  otlp_grpc/dash0:
    endpoint: YOUR_DASH0_ENDPOINT
    headers:
      Authorization: Bearer ${env:DASH0_AUTH_TOKEN}

service:
  pipelines:
    logs/k8s_objects:
      receivers: [k8s_objects/default, k8s_objects/highcardinality]
      processors: [transform/k8s_objects, redaction/k8s_objects, resource/k8s_cluster, batch]
      exporters: [otlp_grpc/dash0]

Find YOUR_DASH0_ENDPOINT in your endpoints settings, and create the token as described in Auth Tokens.

What the configuration does

Two receivers: k8s_objects/default sends every existing deployment, stateful set, daemon set, node, and persistent volume when the collector starts, and then every change. k8s_objects/highcardinality sends only changes, because a snapshot of all pods and replica sets can be large. A pod or replica set that existed before the collector started shows up once it changes.
transform/k8s_objects removes the parts of a manifest that are large and rarely useful: managedFields, the annotations that hold a full copy of the last applied manifest, and the image list of nodes. The last-applied annotations can also contain environment variable values in plain text. The processor also stores the object's resourceVersion in the k8s.object.resource_version attribute.
redaction/k8s_objects masks every field named value with ****. This hides literal environment variable values such as API_TOKEN: secret. References to secrets (valueFrom.secretKeyRef) stay visible, because they hold no credential. The rule also masks other fields named value, such as the value of a toleration or a node taint. The blocked_values patterns mask the secret in flags such as --password=secret and --api-key secret, and in Bearer tokens, wherever they appear in a manifest. This covers a container's command and args and exec probes.
resource/k8s_cluster sets k8s.cluster.name.

Warning

The redaction masks only the patterns above. Other secrets in a manifest are sent as they are. Examples are a secret passed as its own argument, such as args: ["--token", "secret"], or a password in an annotation written by a tool other than the ones in the configuration. To mask more, add key patterns to blocked_key_patterns, or value patterns to blocked_values, in redaction/k8s_objects.

Set the cluster identity

The YAML tab looks up manifests by the cluster of the pod, workload, or node you are looking at. It uses k8s.cluster.name when that telemetry has one, and k8s.cluster.uid when it does not. Set k8s.cluster.name to the same value that your metrics and other telemetry from this cluster carry. If your other telemetry carries k8s.cluster.uid, add it to resource/k8s_cluster with the same value:

yaml

      - key: k8s.cluster.uid
        value: my-cluster-uid
        action: upsert

The k8sattributesprocessor and the Dash0 operator use the UID of the kube-system namespace, which kubectl get namespace kube-system -o jsonpath='{.metadata.uid}' prints. If the identities differ, or if objects with the same name come from several clusters without a cluster identity, the YAML tab shows no manifest or reports the cluster as unknown.

Run more than one replica

To run more than one replica without sending duplicates, add the k8s_leader_elector extension. Only the replica that holds the lease runs the receivers. When that replica stops, another one takes the lease and sends the initial state of k8s_objects/default again. Changes to pods and replica sets made while no replica holds the lease are not sent. This gap lasts up to the lease duration, 15 seconds by default. A pod or replica set that changes only during the gap keeps showing its state from before the gap until it changes again.

Add the extension, reference it from both receivers, and enable it in the service:

yaml

extensions:
  k8s_leader_elector:
    auth_type: serviceAccount
    lease_name: otel-k8s-objects
    lease_namespace: otel-collector

receivers:
  k8s_objects/default:
    k8s_leader_elector: k8s_leader_elector
    # ...
  k8s_objects/highcardinality:
    k8s_leader_elector: k8s_leader_elector
    # ...

service:
  extensions: [k8s_leader_elector]
  # ...

Set lease_namespace to the namespace the collector runs in. The service account also needs access to leases in that namespace:

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: otel-k8s-objects-lease
  namespace: otel-collector
rules:
  - apiGroups: [coordination.k8s.io]
    resources: [leases]
    verbs: [get, list, watch, create, update, patch]

Bind the Role to the collector's service account with a RoleBinding. The k8s_leader_elector extension is in alpha, so its configuration may change in future collector versions.

Reduce the volume of updates

Pods and their owners change often, for example when a container restarts or a readiness probe flips. Each change produces a record with the full manifest. Many of these changes only touch status details, such as the ready replica count of a deployment or the IP of a pod. To combine those changes into one record every 30 seconds, add the log_dedup processor after redaction/k8s_objects:

yaml

processors:
  log_dedup/k8s_objects:
    interval: 30s
    conditions:
      - IsMap(log.body) and log.body["type"] == "MODIFIED"
    include_fields:
      - body.type
      - body.object.metadata.uid
      - body.object.metadata.labels
      - body.object.metadata.annotations
      - body.object.spec
      - body.object.status.phase
      - body.object.status.conditions
      - body.object.status.containerStatuses

service:
  pipelines:
    logs/k8s_objects:
      processors: [transform/k8s_objects, redaction/k8s_objects, log_dedup/k8s_objects, resource/k8s_cluster, batch]

This is optional, and has trade-offs:

Only MODIFIED records are combined. New and deleted objects are always sent right away.
Records are combined only when the fields under include_fields are equal. A change to any of these fields is sent as its own record.
The processor sends the first record of each 30-second window, at the end of the window. Later changes to fields outside include_fields, such as the ready replica count, reach Dash0 only with the next change to a listed field.
The record's timestamp is the time it was sent, not the time of the change, so a manifest can be up to 30 seconds older than its timestamp. The observed timestamp stays the time the processor first saw the change.
The processor adds the log_count, first_observed_timestamp, and last_observed_timestamp attributes to every MODIFIED record, also when log_count is 1.
The logdedupprocessor is in alpha, so its configuration may change in future collector versions.

Limit what you collect

Every change produces a record. The collector needs memory for the initial state of k8s_objects/default and, with log_dedup, for the records it holds back during a window. Start with a few namespaces, watch the collector's memory usage, and add namespaces gradually. The Dash0 operator's Kubernetes object collection recommendations give starting numbers for the object count and memory. Add the memory_limiter processor as the first processor of the pipeline, so the collector refuses data instead of running out of memory.

Limitations

include_initial_state is a receiver setting. It applies to the whole receiver, which is why the configuration uses two receivers. Setting it on an entry under objects makes the collector fail on startup with 'objects[0]' has invalid keys: include_initial_state.
Filtering pods by phase loses Job outcomes. Avoid a field_selector such as status.phase=Running on pods. When a pod stops matching the selector, the Kubernetes API reports it as deleted. A Job pod that completes successfully then arrives as a DELETED record whose last manifest still says phase: Running, and the outcome of the Job is lost.