eBPF Continuous Profiling released

Last updated:

OpenTelemetry Collector: A Guide to Telemetry Pipelines

In most environments, telemetry takes several different paths to reach an observability backend. An SDK sends traces directly to one backend, an agent forwards logs to another, and a separate tool scrapes host metrics. Each path has its own configuration, making it difficult to change where telemetry goes or how it's processed without updating multiple components.

The OpenTelemetry Collector gives you a consistent way to manage these flows. It's a vendor-neutral service that receives telemetry from different sources, processes it, and exports it to one or more backends. Depending on your infrastructure, you can deploy Collectors alongside workloads, as centralized gateways, or in a combination of both.

In this guide, you'll learn how to configure the OpenTelemetry Collector to receive traces, logs, and metrics from different sources, process them, and route each signal to its intended destination. You'll also learn how to export the same telemetry to multiple backends without changing your telemetry sources.

What is the OpenTelemetry Collector?

An OpenTelemetry Collector pipeline illustration

The OpenTelemetry Collector is a vendor-neutral service that receives, processes, and exports telemetry data. It runs independently of your applications and provides a consistent way to manage how traces, metrics, and logs reach your observability backends.

Without the Collector, applications and monitoring agents often send telemetry directly to their respective backends. This creates dependencies between your telemetry sources and their destinations, making changes harder to coordinate as your infrastructure grows.

The Collector provides an intermediate layer where you can control how telemetry is handled and where it's sent. You can introduce new backends, filter unwanted data, or standardize inconsistent instrumentation without having to modify and redeploy every application.

It supports a wide range of protocols and data formats through receivers and exporters, which you connect in pipelines to define how each telemetry signal is collected, processed, and delivered.

Choosing an OpenTelemetry Collector distribution

The OpenTelemetry Collector is available in several distributions, which are prebuilt versions of the binary that include a specific set of components, which determine which telemetry sources, protocols, and destinations you can work with.

The OpenTelemetry project provides several prebuilt distributions, including Core, Contrib, Kubernetes, OTLP, and eBPF Profiling. You can also build your own distribution or use one maintained by a third party.

For this tutorial, we'll use the Contrib distribution, which includes components for a wide range of telemetry sources and backends. It's the recommended starting point for learning and experimenting with Collector pipelines.

Setting up the demo project

To see how the Collector works in practice, you'll build a telemetry pipeline for a small online shop. The project mimics a real-world environment where telemetry comes from different sources, uses different protocols and formats, and needs to reach different observability backends.

You'll configure the Collector to handle three telemetry signals:

SignalSourceBackend
TracesA Flask application and the Caddy reverse proxyJaeger
LogsJSON access logs from CaddyOpenSearch
MetricsCPU, memory, disk, and network usage from the hostPrometheus

Each source requires a different collection approach. The Flask application exports traces over the OpenTelemetry Protocol (OTLP) using gRPC, while Caddy exports its spans over OTLP/HTTP and writes JSON access logs to standard output. Host metrics are collected directly from Linux system interfaces such as /proc and /sys.

You'll configure the Collector to receive all three signals, process them according to their needs, and route each one to its respective backend.

Preparing the demo environment

The demo project uses Docker Compose to run all the services locally. Its main branch provides the starting environment, with everything except the Collector already configured.

The environment includes:

  • app: A Flask-based online shop with endpoints for browsing products and checking out. Approximately one in ten checkout attempts fails because of a declined payment, producing error traces you can investigate.
  • caddy: A reverse proxy that exposes the application on port 8080, traces each request it proxies, and records HTTP access logs in JSON format with the trace and span IDs of each request.
  • loadgen: A traffic generator that sends different requests through Caddy every second.
  • otelcol: The OpenTelemetry Collector, initially configured with no telemetry pipelines. You'll configure it incrementally to collect, process, and export traces, logs, and metrics.
  • jaeger and prometheus: The observability backends that will receive traces and metrics, respectively.
  • data-prepper and opensearch: The logs backend. Data Prepper receives logs over OTLP and indexes them into OpenSearch.
  • opensearch-dashboards: A UI for exploring logs, available on port 5601, with a dashboard for Caddy's access logs already configured.
  • perses: A dashboard tool available on port 8090, with a dashboard for host metrics already configured.

Clone the repository to get started:

bash
git clone https://github.com/dash0-community/otel-collector-demo
bash
cd otel-collector-demo

Then open the repository in your code editor (such as VS Code):

bash
code .

Understanding the demo's telemetry configuration

The demo application is already instrumented with the OpenTelemetry Python SDK. Its service definition in docker-compose.yml includes the following environment variables:

yaml
# docker-compose.yml
services:
  app:
    build: ./app
    environment:
      OTEL_SERVICE_NAME: shop
      OTEL_TRACES_EXPORTER: otlp
      OTEL_METRICS_EXPORTER: none
      OTEL_LOGS_EXPORTER: none
      OTEL_EXPORTER_OTLP_PROTOCOL: grpc
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4317

These settings identify the application as shop and configure it to export traces to the Collector over OTLP/gRPC. The endpoint uses otelcol:4317 to reach the Collector in its own container, since localhost would point to the application container instead.

Caddy, the reverse proxy in front of the application, also generates spans. Its tracing directive creates a span for each incoming HTTP request, and the reverse_proxy directive forwards the request to the Flask application:

text
# caddy/Caddyfile
:80 {
    # [...]
    tracing {
        span "{http.request.method}"
    }
    reverse_proxy app:8080
}

Caddy also supports standard OpenTelemetry environment variables, so its service definition points it at the Collector too:

yaml
# docker-compose.yml
services:
  caddy:
    environment:
      OTEL_SERVICE_NAME: caddy
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4318

Unlike the application, Caddy keeps its default protocol, OTLP over HTTP, and sends to port 4318, the standard OTLP/HTTP port. This gives the Collector two ways of receiving the same kind of data, which you'll configure in the next section.

Caddy also propagates trace context through the HTTP headers it forwards to the application. The Python SDK automatically extracts that context, allowing the application's spans to become children of Caddy's spans. As a result, a single trace can show how a request passes through the reverse proxy and into the shop application.

For now, both services export only spans through OTLP. You'll collect Caddy's logs separately from its container output and read system metrics directly from the host.

Next up, you'll configure the Collector's first pipeline and start the demo environment to verify that traces from both services reach Jaeger.

Configuring your first OpenTelemetry pipeline

A Collector configuration is a YAML file (otelcol.yaml) that defines how telemetry enters and leaves the Collector. This file already exists in the project and is mounted into the container at the path where the Contrib distribution expects its configuration:

yaml
# docker-compose.yml
otelcol:
  image: otel/opentelemetry-collector-contrib:0.164.0
  volumes:
    - ./otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro

For this first pipeline, you'll configure two Collector components and connect them through a service pipeline:

  • Receiver: Receivers define how telemetry enters the Collector. They support different protocols and data sources, with some accepting data pushed by applications and others actively retrieving it. You'll use the OTLP receiver to accept traces from the shop application over gRPC and from Caddy over HTTP.

  • Exporter: Exporters define where telemetry goes after the Collector receives it. They handle communication with the destination using its supported protocol and format. You'll use the OTLP gRPC exporter to forward those traces to Jaeger.

Your first OpenTelemetry Collector pipeline

The service section brings these components together. Its pipelines configuration specifies which receivers and exporters are connected for each telemetry signal. Defining a receiver or exporter alone doesn't activate it; you must reference it in a pipeline for the Collector to use it.

In this section, you'll create a traces pipeline that receives spans over OTLP and exports them to Jaeger. Later, you'll add separate pipelines for logs and metrics using the same configuration file.

Creating the traces pipeline

The shop application exports spans to otelcol:4317 over gRPC, while Caddy exports them to otelcol:4318 over HTTP. A single OTLP receiver can accept both, as long as you enable both protocols.

Open otelcol.yaml in the project root and replace its contents with the following:

yaml
# otelcol.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp_grpc/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
    sending_queue:
      batch:
        flush_timeout: 1s

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp_grpc/jaeger]

The otlp receiver accepts OTLP over two protocols. The grpc protocol listens on port 4317 for the shop application's spans, and the http protocol listens on port 4318 for Caddy's.

Both endpoints use 0.0.0.0 so that the receiver accepts connections from Caddy and the application, which run in other containers. The ports don't need to be published to your machine because the services communicate over the Docker Compose network.

The otlp_grpc/jaeger exporter forwards those spans to Jaeger at jaeger:4317. Collector components follow a type/name naming convention, where the name is optional. Here, otlp_grpc is the exporter type and jaeger is the name assigned to this instance. This lets you configure multiple exporters of the same type, each with its own destination and settings.

The exporter uses TLS by default, but Jaeger's OTLP endpoint in this demo accepts plaintext connections. Setting tls.insecure: true disables TLS for this local connection.

The sending_queue.batch setting enables batching within the exporter's sending queue, combining incoming spans into larger requests to reduce export overhead. The flush_timeout: 1s setting ensures that partially filled batches are sent after one second rather than waiting indefinitely for more data.

Finally, the service.pipelines.traces section connects the receiver and exporter. This activates the pipeline, allowing spans received from Caddy and the shop application to flow through the Collector and into Jaeger.

Starting the Collector and verifying traces in Jaeger

With the traces pipeline configured, you can start the demo environment:

bash
docker compose up -d --build

Docker Compose starts the Collector alongside the shop application, reverse proxy, traffic generator, and observability backends.

Check the Collector's logs to confirm that it started successfully:

bash
docker compose logs otelcol

You should see a startup message similar to:

text
[...]
otelcol-1  | 2026-10-28T08:24:17.179Z   info    service@v0.164.0/service.go:256 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "f1e1d6dc-bf80-4bc2-877a-c03b2f6b3764", "service.name": "otelcol-contrib", "service.version": "0.164.0"}}

The traffic generator automatically sends requests to the shop through Caddy, causing both services to produce spans that flow through the Collector to Jaeger.

Open http://localhost:16686 to access the Jaeger interface. The Service menu lists both caddy and shop. Select shop and click Find Traces.

You should see traces for the application's HTTP requests, including product listings and checkout attempts:

Find shop traces in Jaeger

Open one of them, and you'll find Caddy's span at the root, with the application's spans nested beneath it:

Shop app traces in Jaeger

Handling multiple signals with parallel pipelines

So far, you've configured a pipeline that receives traces from your shop application over OTLP and exports them to Jaeger.

Next, you'll extend the same Collector configuration with two additional pipelines: one that receives access logs from Caddy and forwards them to OpenSearch, and another that collects host metrics and stores them in Prometheus.

Each pipeline has its own receivers and exporters and runs independently of the others, but all three live in the same configuration file.

Collecting Caddy's access logs

Caddy generates an access log for every HTTP request and outputs JSON formatted entries to its standard output. Since Caddy runs in Docker, you can use Docker's Fluentd logging driver to forward these logs directly to the Collector instead of reading them from files.

The Fluentd logging driver sends container output using the Forward protocol, which the Collector supports through its fluent_forward receiver. This is convenient for our Docker Compose environment, but in deployments where container logs are stored in files, you can use the filelog receiver instead.

Configuring Docker to forward Caddy's logs

First, update the caddy and otelcol services in your docker-compose.yml to configure log forwarding:

yaml
# docker-compose.yml
services:
  caddy:
    # [...]
    depends_on:
      - app
      - otelcol
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        fluentd-async: "true"
        tag: caddy.access

  otelcol:
    image: otel/opentelemetry-collector-contrib:0.164.0
    ports:
      - 127.0.0.1:24224:24224
    volumes:
      - ./otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro

The fluentd-address setting tells Docker where to forward Caddy's logs. It uses localhost:24224 instead of otelcol:24224 because Docker's logging driver runs in the Docker daemon, outside the Caddy container, and connects to the Collector through the host's published port.

The Collector publishes port 24224 on 127.0.0.1, making it accessible from the local machine without exposing it on other network interfaces.

The fluentd-async: "true" option is important here as it allows Caddy to start even if the Collector isn't ready to accept connections.

Configuring the logs pipeline

Next, open otelcol.yaml and add the fluent_forward receiver, an OTLP exporter for Data Prepper, and a new logs pipeline. Keep the existing traces configuration and extend it as follows:

yaml
# otelcol.yaml
receivers:
  # [...]
  fluent_forward:
    endpoint: 0.0.0.0:24224

exporters:
  # [...]
  otlp_grpc/data_prepper:
    endpoint: data-prepper:21893
    tls:
      insecure: true
    sending_queue:
      batch:
        flush_timeout: 1s

service:
  pipelines:
    # [...]
    logs:
      receivers: [fluent_forward]
      exporters: [otlp_grpc/data_prepper]

The fluent_forward receiver listens on port 24224 for log records forwarded by Docker, while the otlp_grpc/data_prepper exporter sends those records over OTLP/gRPC to Data Prepper at data-prepper:21893, which is already configured to accept OTLP logs and index them into OpenSearch:

yaml
# data-prepper/pipelines.yaml
otlp-logs:
  source:
    otlp:
      port: 21893
      ssl: false
      output_format: otel
  sink:
    - opensearch:
        hosts: [http://opensearch:9200]
        index: caddy-logs

With output_format: otel, each document keeps the OpenTelemetry log data model. The body, severity, trace and span IDs, attributes, and resource attributes each stay in their own field, so you'll be able to see exactly what the transformations you add later change.

The new logs pipeline connects the fluent_forward receiver to the otlp_grpc/data_prepper exporter. The existing traces pipeline remains unchanged, so the Collector now handles both signals independently.

Verifying the logs in OpenSearch Dashboards

Apply the changes by recreating the affected containers:

bash
docker compose up -d --force-recreate caddy otelcol

The load generator will continue sending HTTP requests to the shop, and Caddy's access logs should now flow through the Collector and Data Prepper into OpenSearch.

To inspect them, open http://localhost:5601 in your browser to access OpenSearch Dashboards, then select Discover from the menu. The caddy-logs index pattern is already configured:

OpenSearch Dashboards showing Caddy logs in Discover

Each record's body field contains Caddy's original JSON log entry as a single string, including the HTTP method, request path, response status, request duration, and the trace and span IDs of the request. None of those values are fields you can filter or chart yet, and the record's own severity, trace context fields are empty.

OpenSearch dashboard showing body field

The demo also includes a Caddy access logs dashboard with a chart of requests by status code. It stays empty for now because the status code is still buried in the body. You'll parse it out, along with the rest of Caddy's fields, in the processing section.

OpenSearch showing Caddy access log dashboard

Ingesting host metrics

The Collector can also collect metrics directly from the operating system using the host_metrics receiver. Like Prometheus's Node Exporter, it periodically reads system statistics such as CPU usage, memory consumption, system load, disk activity, and network traffic.

Since the Collector runs inside a Docker container, it doesn't automatically have access to the host's system information. To make the host's /proc and /sys directories available to the receiver, add two volume mounts to the otelcol service in docker-compose.yml:

yaml
# docker-compose.yml
services:
  otelcol:
    # [...]
    volumes:
      - ./otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro
      - /proc:/hostfs/proc:ro
      - /sys:/hostfs/sys:ro

The mounts are read-only, allowing the Collector to inspect the host's system information without modifying those directories.

Next, add the host_metrics receiver, an otlp_http/prometheus exporter, and a new metrics pipeline to otelcol.yaml:

yaml
# otelcol.yaml
receivers:
  # [...]
  host_metrics:
    collection_interval: 15s
    root_path: /hostfs
    scrapers:
      cpu:
      memory:
      load:
      disk:
      network:

exporters:
  # [...]
  otlp_http/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp

service:
  pipelines:
    # [...]
    metrics:
      receivers: [host_metrics]
      exporters: [otlp_http/prometheus]

The host_metrics receiver collects system metrics every 15 seconds. The scrapers section specifies which categories of metrics to collect, while root_path: /hostfs tells the receiver where to find the mounted host system directories.

The otlp_http/prometheus exporter sends these metrics to Prometheus over OTLP/HTTP. Although Prometheus traditionally collects metrics by scraping HTTP endpoints, it also supports receiving metrics pushed directly through its OTLP endpoint at http://prometheus:9090/api/v1/otlp. The demo's Prometheus service is already configured to accept OTLP metrics.

Verifying the metrics in Prometheus

Apply the updated configuration by recreating the Collector container:

bash
docker compose up -d --force-recreate otelcol

Once the Collector starts, it begins collecting and exporting host metrics at the configured interval.

Open http://localhost:8090, select the Collector demo project in Perses, and navigate to the Host metrics dashboard.

Perses dashboard showing host metrics

The dashboard reads the host metrics from Prometheus and charts CPU utilization, memory consumption, disk activity, and network traffic over time.

Processing and transforming telemetry

The Collector now forwards each signal to its backend unchanged. You can also change telemetry on its way through, using processors, which run between receivers and exporters. You can combine multiple processors in a pipeline, and they'll execute in the order you specify.

In this section, you'll configure processors to:

  • Filter routine health checks from application traces and Caddy logs.
  • Parse Caddy's JSON access logs into structured OpenTelemetry records.
  • Normalize outdated HTTP attributes in application traces.

Filtering unwanted telemetry

Not every piece of telemetry is useful to keep. Applications and infrastructure often produce repetitive data, such as health checks, that you may want to exclude from your observability backends.

In the demo project, Docker Compose checks the caddy service's health by requesting /healthz every five seconds. Caddy proxies each check to the application, so every one produces a Caddy span, an application span, and a Caddy access log, which quickly accumulate alongside regular shop traffic.

You can remove them using the filter processor, which evaluates conditions against incoming telemetry and drops records that match.

The conditions are written in the OpenTelemetry Transformation Language (OTTL), which lets you inspect and change telemetry inside the Collector.

Add the following processors to otelcol.yaml:

yaml
# otelcol.yaml
processors:
  filter/traces:
    error_mode: ignore
    trace_conditions:
      - span.attributes["http.route"] == "/healthz"
      - span.attributes["url.path"] == "/healthz"

  filter/logs:
    error_mode: ignore
    log_conditions:
      - IsMatch(log.body, "\"uri\"\\s*:\\s*\"/healthz\"")

Both processors use the same underlying filter component, but each has a different name and condition.

The filter/traces processor drops a span if any of its conditions match, and it needs one condition for each service. The application's spans record the matched route in http.route. Caddy's spans don't have a route, but they record the request path in url.path.

The filter/logs processor works slightly differently. Caddy's access logs arrive as raw JSON strings, so there's no attribute to check. The condition uses IsMatch() to search the body for the "uri":"/healthz" field instead.

With error_mode: ignore, the Collector logs any error it hits while evaluating a condition and carries on processing, which is the setting you'll want in most cases.

Applying and verifying the filters

Add each filter to its pipeline under a new processors key:

yaml
# otelcol.yaml
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/traces]
      exporters: [otlp_grpc/jaeger]

    logs:
      receivers: [fluent_forward]
      processors: [filter/logs]
      exporters: [otlp_grpc/data_prepper]
    # [...]

Recreate the Collector service to apply the change:

bash
docker compose up -d --force-recreate otelcol

Once the Collector restarts, health check requests stop showing up in your backends. In Jaeger, the caddy and shop services no longer receive new /healthz traces.

In OpenSearch Dashboards, search the caddy-logs index for healthz over the last 15 minutes. You should observe that the matching logs stop at the moment you recreated the Collector, while the rest of Caddy's access logs keep arriving.

OpenSearch showing /healthz logs stopping after the filter is applied

Parsing Caddy's JSON logs

Caddy's access logs are already reaching OpenSearch, but their contents are stored as JSON strings in each record's body field. Here's an example, formatted for readability with the response headers omitted:

json
{
  "level": "info",
  "ts": 1793253396.3794472,
  "logger": "http.log.access.log0",
  "msg": "handled request",
  "request": {
    "remote_ip": "172.20.0.12",
    "remote_port": "46086",
    "client_ip": "172.20.0.12",
    "proto": "HTTP/1.1",
    "method": "POST",
    "host": "caddy",
    "uri": "/items/1/checkout",
    "headers": {
      "Accept": ["*/*"],
      "User-Agent": ["curl/8.16.0"]
    }
  },
  "bytes_read": 0,
  "duration": 0.18468324,
  "size": 50,
  "status": 200,
  "traceID": "9a69815e0ea9c6e89d1c8a76fd1ae20c",
  "spanID": "fc52019e4c20415b"
}

The OpenTelemetry log data model provides dedicated fields for timestamps, severity, trace context, and attributes. By extracting these values from Caddy's JSON output, you can make them available as individual fields for filtering, querying, and visualization in OpenSearch.

To do this, you'll use the transform processor. It modifies telemetry using OTTL statements, the same language as the filter conditions.

Add the following processor to otelcol.yaml:

yaml
# otelcol.yaml
processors:
  transform/logs:
    error_mode: ignore
    log_statements:
      - conditions:
          - log.attributes["fluent.tag"] == "caddy.access"
        statements:
          # Identify the source container.
          - set(resource.attributes["service.name"], "caddy")
          - set(resource.attributes["container.id"],
            log.attributes["container_id"])
          - set(resource.attributes["container.name"],
            log.attributes["container_name"])
          - replace_pattern(resource.attributes["container.name"], "^/", "")
          - set(log.attributes["log.iostream"], log.attributes["source"])

          # Parse Caddy's JSON and populate the log record fields.
          - merge_maps(log.cache, ParseJSON(log.body), "upsert") where
            IsMatch(log.body, "^\\{")
          - set(log.observed_time, Now())
          - set(log.time_unix_nano, Int(log.cache["ts"] * 1000000000.0)) where
            log.cache["ts"] != nil
          - set(log.severity_text, ToUpperCase(log.cache["level"])) where
            log.cache["level"] != nil
          - set(log.severity_number, SEVERITY_NUMBER_DEBUG) where
            log.cache["level"] == "debug"
          - set(log.severity_number, SEVERITY_NUMBER_INFO) where
            log.cache["level"] == "info"
          - set(log.severity_number, SEVERITY_NUMBER_WARN) where
            log.cache["level"] == "warn"
          - set(log.severity_number, SEVERITY_NUMBER_ERROR) where
            log.cache["level"] == "error"
          - set(log.trace_id.string, log.cache["traceID"]) where
            log.cache["traceID"] != nil and log.cache["traceID"] != ""
          - set(log.span_id.string, log.cache["spanID"]) where
            log.cache["spanID"] != nil and log.cache["spanID"] != ""
          - set(log.attributes["log.record.original"], log.body)
          - set(log.body, log.cache["msg"]) where log.cache["msg"] != nil and
            log.cache["request"] == nil

          # Extract HTTP attributes from access logs.
          - set(log.attributes["http.request.method"],
            log.cache["request"]["method"]) where log.cache["request"] != nil
          - set(log.attributes["url.path"], log.cache["request"]["uri"]) where
            log.cache["request"] != nil
          - replace_pattern(log.attributes["url.path"], "\\?.*$", "")
          - set(log.attributes["network.protocol.version"],
            log.cache["request"]["proto"]) where log.cache["request"] != nil
          - replace_pattern(log.attributes["network.protocol.version"],
            "^HTTP/", "")
          - set(log.attributes["client.address"],
            log.cache["request"]["client_ip"]) where log.cache["request"] != nil
          - set(log.attributes["user_agent.original"],
            log.cache["request"]["headers"]["User-Agent"][0]) where
            log.cache["request"]["headers"]["User-Agent"] != nil
          - set(log.attributes["http.response.status_code"],
            Int(log.cache["status"])) where log.cache["status"] != nil
          - set(log.attributes["http.response.body.size"],
            Int(log.cache["size"])) where log.cache["size"] != nil
          - set(log.attributes["caddy.request.duration_seconds"],
            log.cache["duration"]) where log.cache["duration"] != nil
          - set(log.body, Format("%s %s %d",
            [log.attributes["http.request.method"], log.attributes["url.path"],
            log.attributes["http.response.status_code"]])) where
            log.cache["request"] != nil

          # Remove Docker's original attributes.
          - delete_matching_keys(log.attributes,
            "^(container_id|container_name|source|fluent\\.tag)$")

The statements are grouped under a condition, so they only run on records whose fluent.tag attribute is caddy.access (this tag is set in the Docker Compose configuration). This means that if another container sends logs through the same driver, Caddy's parsing rules won't affect them.

First, it identifies the source by setting service.name: caddy and moving Docker's container metadata into resource attributes. It also renames source to the standard log.iostream attribute.

Next, ParseJSON() reads the original JSON body into log.cache, a temporary map used by subsequent statements. The processor extracts Caddy's timestamp and severity, then copies traceID and spanID into the log record's trace context.

It also maps the HTTP request details to attributes defined by the OpenTelemetry semantic conventions:

Caddy fieldOpenTelemetry attribute
request.methodhttp.request.method
request.uriurl.path
request.protonetwork.protocol.version
request.client_ipclient.address
request.headers.User-Agentuser_agent.original
statushttp.response.status_code
sizehttp.response.body.size

Where necessary, the statements adjust values to match the conventions, including removing query strings from url.path and converting numeric fields to integers.

Finally, the processor preserves the original JSON in log.record.original before replacing the body with a readable request summary. This makes the record easier to scan without losing the original information.

Applying and verifying the log transformations

To activate the processor, add it to the logs pipeline in otelcol.yaml, after the filter:

yaml
# otelcol.yaml
service:
  pipelines:
    # [...]
    logs:
      receivers: [fluent_forward]
      processors: [filter/logs, transform/logs]
      exporters: [otlp_grpc/data_prepper]

Processors run in the order they're listed, so the filter discards health checks before the transform processor sees them. This avoids spending processing time on telemetry that won't be exported, such as parsing the JSON of a log record that's about to be dropped.

Then apply the updated configuration by recreating the Collector service:

bash
docker compose up -d --force-recreate otelcol

Open OpenSearch Dashboards, navigate to Discover, and select the caddy-logs index pattern. Set the time range to the last 15 minutes and inspect a newly ingested access log.

Expand the record to inspect its fields. You should find the HTTP method, request path, response status, and other extracted attributes. Instead of the original JSON string, its body field should contain a short summary such as:

text
POST /items/1/checkout 200

OpenSearch showing transformed Caddy access log

The resource attributes should also identify the source as Caddy, with the corresponding container ID and name, alongside the record's severity, timestamp, and trace context:

OpenSearch showing transformed Caddy access log with resource attributes

This makes it easy to filter the logs by attributes such as http.response.status_code or url.path, rather than searching the original JSON string.

You can also open the preconfigured Caddy access logs dashboard to see the requests-by-status-code chart populated from these structured attributes:

The Caddy access logs dashboard showing requests grouped by HTTP status
code

Transforming span attributes

The transform processor also works with spans and metrics, so you can fix instrumentation output without changing the instrumented code.

For example, the demo application's Flask instrumentation still uses deprecated HTTP and network attributes, such as http.method, http.status_code, and net.peer.ip. Caddy's spans in the same trace already use their current replacements, so the two services currently describe the same request differently.

You can update these attributes by adding another transform processor to otelcol.yaml, this time using trace_statements:

yaml
# otelcol.yaml
processors:
  # [...]
  transform/traces:
    error_mode: ignore
    trace_statements:
      - set(span.attributes["http.request.method"],
        span.attributes["http.method"]) where span.attributes["http.method"] !=
        nil
      - set(span.attributes["http.response.status_code"],
        span.attributes["http.status_code"]) where
        span.attributes["http.status_code"] != nil
      - set(span.attributes["url.path"], span.attributes["http.target"]) where
        span.attributes["http.target"] != nil
      - set(span.attributes["url.query"], Split(span.attributes["http.target"],
        "?")[1]) where IsMatch(span.attributes["http.target"], "\\?")
      - replace_pattern(span.attributes["url.path"], "\\?.*$", "")
      - set(span.attributes["url.scheme"], span.attributes["http.scheme"]) where
        span.attributes["http.scheme"] != nil
      - set(span.attributes["network.protocol.version"],
        span.attributes["http.flavor"]) where span.attributes["http.flavor"] !=
        nil
      - set(span.attributes["user_agent.original"],
        span.attributes["http.user_agent"]) where
        span.attributes["http.user_agent"] != nil
      - set(span.attributes["server.address"], span.attributes["net.host.name"])
        where span.attributes["net.host.name"] != nil
      - set(span.attributes["server.port"], span.attributes["net.host.port"])
        where span.attributes["net.host.port"] != nil
      - set(span.attributes["network.peer.address"],
        span.attributes["net.peer.ip"]) where span.attributes["net.peer.ip"] !=
        nil
      - set(span.attributes["network.peer.port"],
        span.attributes["net.peer.port"]) where span.attributes["net.peer.port"]
        != nil
      - set(span.attributes["error.type"], span.attributes["error_type"]) where
        span.attributes["error_type"] != nil
      - delete_matching_keys(span.attributes,
        "^(http\\.(method|status_code|target|scheme|flavor|user_agent|host|server_name)|net\\.(host|peer)\\.(name|ip|port)|error_type)$")

Each set() statement copies a legacy attribute's value to its current name, with a where condition that skips spans where the original attribute is absent:

Deprecated attributeCurrent attribute
http.methodhttp.request.method
http.status_codehttp.response.status_code
http.targeturl.path, url.query
http.schemeurl.scheme
http.flavornetwork.protocol.version
http.user_agentuser_agent.original
net.host.nameserver.address
net.host.portserver.port
net.peer.ipnetwork.peer.address
net.peer.portnetwork.peer.port

The final delete_matching_keys() statement removes the old attributes to avoid retaining duplicate information. It also drops http.host, which duplicates net.host.name, and http.server_name, which the current conventions removed without a replacement.

The processor also maps the application's custom error_type attribute to error.type, the standard attribute for identifying the type of error that occurred.

One value needs additional handling. The deprecated http.target attribute contains the request path and may include a query string, such as /items?sort=price. The URL semantic conventions represent these separately as url.path and url.query, so the transformation uses Split() to extract the query string before removing it from the path.

Applying and verifying the span transformations

Add the processor to the traces pipeline in otelcol.yaml, after the filter:

yaml
# otelcol.yaml
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/traces, transform/traces]
      exporters: [otlp_grpc/jaeger]
    # [...]

Then recreate the Collector service:

bash
docker compose up -d --force-recreate otelcol

Open Jaeger, select the shop service, and open a recent trace. The application's spans now use the same attribute names as Caddy's, such as http.request.method, user_agent.original, url.path, and others:

Jaeger showing updated span attributes

There's one exception: Jaeger still displays http.status_code instead of http.response.status_code. This happens because its trace view converts spans to Jaeger's older data model, restoring the legacy attribute name. As Jaeger's UI transitions to OpenTelemetry's native data model, future versions may display the original attribute unchanged.


Filtering, parsing, and renaming are just a few ways to process telemetry. You can also use processors to:

  • Remove sensitive information such as passwords, API keys, or personal data before telemetry leaves your infrastructure using the redaction processor.
  • Enrich telemetry with context such as deployment environments, host information, and Kubernetes pod metadata using the resource processor or Kubernetes attributes processor.
  • Modify individual attributes by inserting, updating, hashing, or deleting values using the attributes processor.
  • Reduce duplicate logs by combining repeated records and recording how often they occurred using the log deduplication processor.
  • Limit memory usage by monitoring the Collector's memory consumption and applying backpressure or refusing incoming data when configured thresholds are reached using the memory limiter processor.

For a complete list of available processors and their configuration options, see the OpenTelemetry Collector Contrib processors list on GitHub.

Sending telemetry to multiple destinations

So far, each Collector pipeline sends telemetry to a single backend. You can also configure a pipeline with multiple exporters, allowing the Collector to send the same telemetry to several destinations. This is useful when you're migrating between backends or evaluating a new one.

To demonstrate this, you'll configure the Collector to send a copy of all three telemetry signals to Dash0, a platform built on OpenTelemetry's native data model, while continuing to export them to Jaeger, OpenSearch, and Prometheus.

Configuring Dash0 as an additional destination

First, sign up for a free Dash0 trial if you don't already have an account.

You'll need your organization's OTLP/gRPC endpoint, available under Settings > Endpoints, and an auth token with permission to ingest telemetry.

The demo repository includes an .env.example template. Copy it to .env, then fill in your credentials:

bash
cp .env.example .env
sh
# .env
DASH0_ENDPOINT=<your OTLP/gRPC endpoint>
DASH0_AUTH_TOKEN=<your auth token>

To make these values available inside the Collector container, add them to the otelcol service definition:

yaml
# docker-compose.yml
services:
  otelcol:
    # [...]
    environment:
      DASH0_ENDPOINT: ${DASH0_ENDPOINT}
      DASH0_AUTH_TOKEN: ${DASH0_AUTH_TOKEN}

Next, define a new otlp_grpc/dash0 exporter in otelcol.yaml and add it to the exporters list of each pipeline:

yaml
# otelcol.yaml
exporters:
  # [...]
  otlp_grpc/dash0:
    endpoint: ${env:DASH0_ENDPOINT}
    headers:
      Authorization: Bearer ${env:DASH0_AUTH_TOKEN}
    sending_queue:
      batch:
        flush_timeout: 1s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/traces, transform/traces]
      exporters: [otlp_grpc/jaeger, otlp_grpc/dash0]

    logs:
      receivers: [fluent_forward]
      processors: [filter/logs, transform/logs]
      exporters: [otlp_grpc/data_prepper, otlp_grpc/dash0]

    metrics:
      receivers: [host_metrics]
      exporters: [otlp_http/prometheus, otlp_grpc/dash0]

The otlp_grpc/dash0 exporter uses the same component type as the Jaeger and Data Prepper exporters, but with its own destination and authentication settings. Unlike the local exporters, it uses TLS to secure the connection to Dash0.

Notice that each pipeline now lists two exporters. Both receive the same processed telemetry, but handle delivery independently. If one destination becomes unavailable, the other may continue receiving data, so there's no guarantee that both backends will store exactly the same records.

Exporter queues and retries help handle temporary failures, but the default queues are held in memory, so pending data can be lost during restarts or prolonged outages. See this article for more on delivery reliability.

Verifying telemetry in Dash0

Recreate the Collector container to apply the updated configuration and environment variables:

bash
docker compose up -d --force-recreate otelcol

Then open Dash0's tracing view and find a failed checkout request from the shop service. Open the trace to see how the request passed through Caddy and the application, including the charge_payment span where the payment was declined:

Shop traces in Dash0

Select a span and open the Related Logs & Events tab. Because Caddy's access logs contain the trace IDs extracted by the Collector, you can find the corresponding access log for the same request.

The log includes the HTTP method, request path, response status, and other attributes extracted by the transform processor. This lets you inspect what Caddy recorded alongside the application's trace without manually searching for the matching request.

Caddy logs in Dash0

You can also explore the host metrics collected by the same Collector to examine resource usage during the relevant period. Or ask Agent0 to create a dashboard for you:

Host metrics dashboard in Dash0

Troubleshooting common Collector issues

If the Collector won't start or telemetry isn't reaching its destination, check its logs first:

bash
docker compose logs otelcol

You can also validate your configuration without starting the Collector:

bash
docker compose run --rm --no-deps otelcol validate \
  --config=/etc/otelcol-contrib/config.yaml

If telemetry still isn't appearing in your backend, temporarily enable the debug exporter to check whether data is making it through the pipeline.

For more detailed diagnostics, including connection failures, processor errors, retry exhaustion, and dropped telemetry, see the official guide to troubleshooting the OpenTelemetry Collector.

Final thoughts

The OpenTelemetry Collector separates how telemetry is produced from how it's processed and delivered. Your applications and infrastructure can focus on describing their behavior, while the Collector manages what happens to that data before it reaches an observability backend.

This separation gives you room to evolve your observability infrastructure. You can change backends, introduce new data sources, filter unwanted telemetry, or standardize inconsistent instrumentation without repeatedly modifying the applications producing it.

Once the Collector becomes part of your telemetry pipeline, its reliability directly affects what you can observe. Failed exports, overloaded queues, and misconfigured processors can all leave gaps in your data, even when your applications are working correctly.

As you move toward production, the official OpenTelemetry Collector documentation is a good starting point for exploring deployment, security, and operational considerations in greater depth.