Last updated:
OpenTelemetry Collector: A Guide to Telemetry Pipelines
In most environments, telemetry takes several different paths to reach an observability backend. An SDK sends traces directly to one backend, an agent forwards logs to another, and a separate tool scrapes host metrics. Each path has its own configuration, making it difficult to change where telemetry goes or how it's processed without updating multiple components.
The OpenTelemetry Collector gives you a consistent way to manage these flows. It's a vendor-neutral service that receives telemetry from different sources, processes it, and exports it to one or more backends. Depending on your infrastructure, you can deploy Collectors alongside workloads, as centralized gateways, or in a combination of both.
In this guide, you'll learn how to configure the OpenTelemetry Collector to receive traces, logs, and metrics from different sources, process them, and route each signal to its intended destination. You'll also learn how to export the same telemetry to multiple backends without changing your telemetry sources.
What is the OpenTelemetry Collector?
The OpenTelemetry Collector is a vendor-neutral service that receives, processes, and exports telemetry data. It runs independently of your applications and provides a consistent way to manage how traces, metrics, and logs reach your observability backends.
Without the Collector, applications and monitoring agents often send telemetry directly to their respective backends. This creates dependencies between your telemetry sources and their destinations, making changes harder to coordinate as your infrastructure grows.
The Collector provides an intermediate layer where you can control how telemetry is handled and where it's sent. You can introduce new backends, filter unwanted data, or standardize inconsistent instrumentation without having to modify and redeploy every application.
It supports a wide range of protocols and data formats through receivers and exporters, which you connect in pipelines to define how each telemetry signal is collected, processed, and delivered.
Choosing an OpenTelemetry Collector distribution
The OpenTelemetry Collector is available in several distributions, which are prebuilt versions of the binary that include a specific set of components, which determine which telemetry sources, protocols, and destinations you can work with.
The OpenTelemetry project provides several prebuilt distributions, including Core, Contrib, Kubernetes, OTLP, and eBPF Profiling. You can also build your own distribution or use one maintained by a third party.
For this tutorial, we'll use the Contrib distribution, which includes components for a wide range of telemetry sources and backends. It's the recommended starting point for learning and experimenting with Collector pipelines.
Setting up the demo project
To see how the Collector works in practice, you'll build a telemetry pipeline for a small online shop. The project mimics a real-world environment where telemetry comes from different sources, uses different protocols and formats, and needs to reach different observability backends.
You'll configure the Collector to handle three telemetry signals:
| Signal | Source | Backend |
|---|---|---|
| Traces | A Flask application and the Caddy reverse proxy | Jaeger |
| Logs | JSON access logs from Caddy | OpenSearch |
| Metrics | CPU, memory, disk, and network usage from the host | Prometheus |
Each source requires a different collection approach. The Flask application
exports traces over the
OpenTelemetry Protocol (OTLP)
using gRPC, while Caddy exports its spans over OTLP/HTTP and writes
JSON access logs to standard
output. Host metrics are collected directly from Linux system interfaces such as
/proc and /sys.
You'll configure the Collector to receive all three signals, process them according to their needs, and route each one to its respective backend.
Preparing the demo environment
The demo project uses
Docker Compose to run all the services locally. Its main branch provides the
starting environment, with everything except the Collector already configured.
The environment includes:
app: A Flask-based online shop with endpoints for browsing products and checking out. Approximately one in ten checkout attempts fails because of a declined payment, producing error traces you can investigate.caddy: A reverse proxy that exposes the application on port 8080, traces each request it proxies, and records HTTP access logs in JSON format with the trace and span IDs of each request.loadgen: A traffic generator that sends different requests through Caddy every second.otelcol: The OpenTelemetry Collector, initially configured with no telemetry pipelines. You'll configure it incrementally to collect, process, and export traces, logs, and metrics.jaegerandprometheus: The observability backends that will receive traces and metrics, respectively.data-prepperandopensearch: The logs backend. Data Prepper receives logs over OTLP and indexes them into OpenSearch.opensearch-dashboards: A UI for exploring logs, available on port 5601, with a dashboard for Caddy's access logs already configured.perses: A dashboard tool available on port 8090, with a dashboard for host metrics already configured.
Clone the repository to get started:
git clone https://github.com/dash0-community/otel-collector-democd otel-collector-demoThen open the repository in your code editor (such as VS Code):
code .Understanding the demo's telemetry configuration
The demo application is already instrumented with the OpenTelemetry Python SDK.
Its service definition in docker-compose.yml includes the following
environment variables:
# docker-compose.yml
services:
app:
build: ./app
environment:
OTEL_SERVICE_NAME: shop
OTEL_TRACES_EXPORTER: otlp
OTEL_METRICS_EXPORTER: none
OTEL_LOGS_EXPORTER: none
OTEL_EXPORTER_OTLP_PROTOCOL: grpc
OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4317These settings identify the application as shop and configure it to export
traces to the Collector over OTLP/gRPC. The endpoint uses otelcol:4317 to
reach the Collector in its own container, since localhost would point to the
application container instead.
Caddy, the reverse proxy in front of the application, also generates spans. Its
tracing directive
creates a span for each incoming HTTP request, and the reverse_proxy directive
forwards the request to the Flask application:
# caddy/Caddyfile
:80 {
# [...]
tracing {
span "{http.request.method}"
}
reverse_proxy app:8080
}Caddy also supports standard OpenTelemetry environment variables, so its service definition points it at the Collector too:
# docker-compose.yml
services:
caddy:
environment:
OTEL_SERVICE_NAME: caddy
OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4318Unlike the application, Caddy keeps its default protocol, OTLP over HTTP, and
sends to port 4318, the standard OTLP/HTTP port. This gives the Collector two
ways of receiving the same kind of data, which you'll configure in the next
section.
Caddy also propagates trace context through the HTTP headers it forwards to the application. The Python SDK automatically extracts that context, allowing the application's spans to become children of Caddy's spans. As a result, a single trace can show how a request passes through the reverse proxy and into the shop application.
For now, both services export only spans through OTLP. You'll collect Caddy's logs separately from its container output and read system metrics directly from the host.
Next up, you'll configure the Collector's first pipeline and start the demo environment to verify that traces from both services reach Jaeger.
Configuring your first OpenTelemetry pipeline
A Collector configuration is a YAML file (otelcol.yaml) that defines how
telemetry enters and leaves the Collector. This file already exists in the
project and is mounted into the container at the path where the Contrib
distribution expects its configuration:
# docker-compose.yml
otelcol:
image: otel/opentelemetry-collector-contrib:0.164.0
volumes:
- ./otelcol.yaml:/etc/otelcol-contrib/config.yaml:roFor this first pipeline, you'll configure two Collector components and connect them through a service pipeline:
-
Receiver: Receivers define how telemetry enters the Collector. They support different protocols and data sources, with some accepting data pushed by applications and others actively retrieving it. You'll use the OTLP receiver to accept traces from the shop application over gRPC and from Caddy over HTTP.
-
Exporter: Exporters define where telemetry goes after the Collector receives it. They handle communication with the destination using its supported protocol and format. You'll use the OTLP gRPC exporter to forward those traces to Jaeger.
The service section brings these components together. Its pipelines
configuration specifies which receivers and exporters are connected for each
telemetry signal. Defining a receiver or exporter alone doesn't activate it;
you must reference it in a pipeline for the Collector to use it.
In this section, you'll create a traces pipeline that receives spans over OTLP
and exports them to Jaeger. Later, you'll add separate pipelines for logs and
metrics using the same configuration file.
Creating the traces pipeline
The shop application exports spans to otelcol:4317 over gRPC, while Caddy
exports them to otelcol:4318 over HTTP. A single OTLP receiver can accept
both, as long as you enable both protocols.
Open otelcol.yaml in the project root and replace its contents with the
following:
# otelcol.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlp_grpc/jaeger:
endpoint: jaeger:4317
tls:
insecure: true
sending_queue:
batch:
flush_timeout: 1s
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp_grpc/jaeger]The otlp receiver accepts OTLP over two protocols. The grpc protocol listens
on port 4317 for the shop application's spans, and the http protocol listens
on port 4318 for Caddy's.
Both endpoints use 0.0.0.0 so that the receiver accepts connections from Caddy
and the application, which run in other containers. The ports don't need to be
published to your machine because the services communicate over the Docker
Compose network.
The otlp_grpc/jaeger exporter forwards those spans to Jaeger at jaeger:4317.
Collector components follow a type/name naming convention, where the name is
optional. Here, otlp_grpc is the exporter type and jaeger is the name
assigned to this instance. This lets you configure multiple exporters of the
same type, each with its own destination and settings.
The exporter uses TLS by default, but Jaeger's OTLP endpoint in this demo
accepts plaintext connections. Setting tls.insecure: true disables TLS for
this local connection.
The sending_queue.batch setting
enables batching
within the exporter's sending queue, combining incoming spans into larger
requests to reduce export overhead. The flush_timeout: 1s setting ensures that
partially filled batches are sent after one second rather than waiting
indefinitely for more data.
Finally, the service.pipelines.traces section connects the receiver and
exporter. This activates the pipeline, allowing spans received from Caddy and
the shop application to flow through the Collector and into Jaeger.
Starting the Collector and verifying traces in Jaeger
With the traces pipeline configured, you can start the demo environment:
docker compose up -d --buildDocker Compose starts the Collector alongside the shop application, reverse proxy, traffic generator, and observability backends.
Check the Collector's logs to confirm that it started successfully:
docker compose logs otelcolYou should see a startup message similar to:
[...]
otelcol-1 | 2026-10-28T08:24:17.179Z info service@v0.164.0/service.go:256 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "f1e1d6dc-bf80-4bc2-877a-c03b2f6b3764", "service.name": "otelcol-contrib", "service.version": "0.164.0"}}The traffic generator automatically sends requests to the shop through Caddy, causing both services to produce spans that flow through the Collector to Jaeger.
Open http://localhost:16686 to access the Jaeger
interface. The Service menu lists both caddy and shop. Select shop and
click Find Traces.
You should see traces for the application's HTTP requests, including product listings and checkout attempts:
Open one of them, and you'll find Caddy's span at the root, with the application's spans nested beneath it:
Handling multiple signals with parallel pipelines
So far, you've configured a pipeline that receives traces from your shop application over OTLP and exports them to Jaeger.
Next, you'll extend the same Collector configuration with two additional pipelines: one that receives access logs from Caddy and forwards them to OpenSearch, and another that collects host metrics and stores them in Prometheus.
Each pipeline has its own receivers and exporters and runs independently of the others, but all three live in the same configuration file.
Collecting Caddy's access logs
Caddy generates an access log for every HTTP request and outputs JSON formatted entries to its standard output. Since Caddy runs in Docker, you can use Docker's Fluentd logging driver to forward these logs directly to the Collector instead of reading them from files.
The Fluentd logging driver sends container output using the Forward protocol, which the Collector supports through its fluent_forward receiver. This is convenient for our Docker Compose environment, but in deployments where container logs are stored in files, you can use the filelog receiver instead.
Configuring Docker to forward Caddy's logs
First, update the caddy and otelcol services in your docker-compose.yml to
configure log forwarding:
# docker-compose.yml
services:
caddy:
# [...]
depends_on:
- app
- otelcol
logging:
driver: fluentd
options:
fluentd-address: localhost:24224
fluentd-async: "true"
tag: caddy.access
otelcol:
image: otel/opentelemetry-collector-contrib:0.164.0
ports:
- 127.0.0.1:24224:24224
volumes:
- ./otelcol.yaml:/etc/otelcol-contrib/config.yaml:roThe fluentd-address setting tells Docker where to forward Caddy's logs. It
uses localhost:24224 instead of otelcol:24224 because
Docker's logging driver
runs in the Docker daemon, outside the Caddy container, and connects to the
Collector through the host's published port.
The Collector publishes port 24224 on 127.0.0.1, making it accessible from
the local machine without exposing it on other network interfaces.
The fluentd-async: "true" option is important here as it allows Caddy to start
even if the Collector isn't ready to accept connections.
Configuring the logs pipeline
Next, open otelcol.yaml and add the fluent_forward receiver, an OTLP
exporter for Data Prepper, and a new logs pipeline. Keep the existing traces
configuration and extend it as follows:
# otelcol.yaml
receivers:
# [...]
fluent_forward:
endpoint: 0.0.0.0:24224
exporters:
# [...]
otlp_grpc/data_prepper:
endpoint: data-prepper:21893
tls:
insecure: true
sending_queue:
batch:
flush_timeout: 1s
service:
pipelines:
# [...]
logs:
receivers: [fluent_forward]
exporters: [otlp_grpc/data_prepper]The fluent_forward receiver listens on port 24224 for log records forwarded
by Docker, while the otlp_grpc/data_prepper exporter sends those records over
OTLP/gRPC to Data Prepper at data-prepper:21893, which is already configured
to accept OTLP logs and index them into OpenSearch:
# data-prepper/pipelines.yaml
otlp-logs:
source:
otlp:
port: 21893
ssl: false
output_format: otel
sink:
- opensearch:
hosts: [http://opensearch:9200]
index: caddy-logsWith output_format: otel, each document keeps the
OpenTelemetry log data model.
The body, severity, trace and span IDs, attributes, and resource attributes each
stay in their own field, so you'll be able to see exactly what the
transformations you add later change.
The new logs pipeline connects the fluent_forward receiver to the
otlp_grpc/data_prepper exporter. The existing traces pipeline remains
unchanged, so the Collector now handles both signals independently.
Verifying the logs in OpenSearch Dashboards
Apply the changes by recreating the affected containers:
docker compose up -d --force-recreate caddy otelcolThe load generator will continue sending HTTP requests to the shop, and Caddy's access logs should now flow through the Collector and Data Prepper into OpenSearch.
To inspect them, open http://localhost:5601 in your
browser to access OpenSearch Dashboards, then select Discover from the menu.
The caddy-logs index pattern is already configured:
Each record's body field contains Caddy's original JSON log entry as a single
string, including the HTTP method, request path, response status, request
duration, and the trace and span IDs of the request. None of those values are
fields you can filter or chart yet, and the record's own severity,
trace context
fields are empty.
The demo also includes a Caddy access logs dashboard with a chart of requests by status code. It stays empty for now because the status code is still buried in the body. You'll parse it out, along with the rest of Caddy's fields, in the processing section.
Ingesting host metrics
The Collector can also collect metrics directly from the operating system using
the
host_metrics receiver.
Like Prometheus's Node Exporter, it periodically reads system statistics such as
CPU usage, memory consumption, system load, disk activity, and network traffic.
Since the Collector runs inside a Docker container, it doesn't automatically
have access to the host's system information. To make the host's /proc and
/sys directories available to the receiver, add two volume mounts to the
otelcol service in docker-compose.yml:
# docker-compose.yml
services:
otelcol:
# [...]
volumes:
- ./otelcol.yaml:/etc/otelcol-contrib/config.yaml:ro
- /proc:/hostfs/proc:ro
- /sys:/hostfs/sys:roThe mounts are read-only, allowing the Collector to inspect the host's system information without modifying those directories.
Next, add the host_metrics receiver, an otlp_http/prometheus exporter, and a
new metrics pipeline to otelcol.yaml:
# otelcol.yaml
receivers:
# [...]
host_metrics:
collection_interval: 15s
root_path: /hostfs
scrapers:
cpu:
memory:
load:
disk:
network:
exporters:
# [...]
otlp_http/prometheus:
endpoint: http://prometheus:9090/api/v1/otlp
service:
pipelines:
# [...]
metrics:
receivers: [host_metrics]
exporters: [otlp_http/prometheus]The host_metrics receiver collects system metrics every 15 seconds. The
scrapers section specifies which categories of metrics to collect, while
root_path: /hostfs tells the receiver where to find the mounted host system
directories.
The otlp_http/prometheus exporter sends these metrics to Prometheus over
OTLP/HTTP.
Although Prometheus traditionally collects metrics by scraping HTTP endpoints,
it also supports
receiving metrics pushed directly
through its OTLP endpoint at http://prometheus:9090/api/v1/otlp. The demo's
Prometheus service is already configured to accept OTLP metrics.
Verifying the metrics in Prometheus
Apply the updated configuration by recreating the Collector container:
docker compose up -d --force-recreate otelcolOnce the Collector starts, it begins collecting and exporting host metrics at the configured interval.
Open http://localhost:8090, select the Collector demo project in Perses, and navigate to the Host metrics dashboard.
The dashboard reads the host metrics from Prometheus and charts CPU utilization, memory consumption, disk activity, and network traffic over time.
Processing and transforming telemetry
The Collector now forwards each signal to its backend unchanged. You can also change telemetry on its way through, using processors, which run between receivers and exporters. You can combine multiple processors in a pipeline, and they'll execute in the order you specify.
In this section, you'll configure processors to:
- Filter routine health checks from application traces and Caddy logs.
- Parse Caddy's JSON access logs into structured OpenTelemetry records.
- Normalize outdated HTTP attributes in application traces.
Filtering unwanted telemetry
Not every piece of telemetry is useful to keep. Applications and infrastructure often produce repetitive data, such as health checks, that you may want to exclude from your observability backends.
In the demo project, Docker Compose checks the caddy service's health by
requesting /healthz every five seconds. Caddy proxies each check to the
application, so every one produces a Caddy span, an application span, and a
Caddy access log, which quickly accumulate alongside regular shop traffic.
You can remove them using the filter processor, which evaluates conditions against incoming telemetry and drops records that match.
The conditions are written in the OpenTelemetry Transformation Language (OTTL), which lets you inspect and change telemetry inside the Collector.
Add the following processors to otelcol.yaml:
# otelcol.yaml
processors:
filter/traces:
error_mode: ignore
trace_conditions:
- span.attributes["http.route"] == "/healthz"
- span.attributes["url.path"] == "/healthz"
filter/logs:
error_mode: ignore
log_conditions:
- IsMatch(log.body, "\"uri\"\\s*:\\s*\"/healthz\"")Both processors use the same underlying filter component, but each has a different name and condition.
The filter/traces processor drops a span if any of its conditions match, and
it needs one condition for each service. The application's spans record the
matched route in http.route. Caddy's spans don't have a route, but they record
the request path in url.path.
The filter/logs processor works slightly differently. Caddy's access logs
arrive as raw JSON strings, so there's no attribute to check. The condition uses
IsMatch() to search the body for the "uri":"/healthz" field instead.
With error_mode: ignore, the Collector logs any error it hits while evaluating
a condition and carries on processing, which is the setting you'll want in most
cases.
Applying and verifying the filters
Add each filter to its pipeline under a new processors key:
# otelcol.yaml
service:
pipelines:
traces:
receivers: [otlp]
processors: [filter/traces]
exporters: [otlp_grpc/jaeger]
logs:
receivers: [fluent_forward]
processors: [filter/logs]
exporters: [otlp_grpc/data_prepper]
# [...]Recreate the Collector service to apply the change:
docker compose up -d --force-recreate otelcolOnce the Collector restarts, health check requests stop showing up in your
backends. In Jaeger, the caddy and shop services no longer receive new
/healthz traces.
In OpenSearch Dashboards, search the caddy-logs index for healthz over the
last 15 minutes. You should observe that the matching logs stop at the moment
you recreated the Collector, while the rest of Caddy's access logs keep
arriving.
Parsing Caddy's JSON logs
Caddy's access logs are already reaching OpenSearch, but their contents are
stored as JSON strings in each record's body field. Here's an example,
formatted for readability with the response headers omitted:
{
"level": "info",
"ts": 1793253396.3794472,
"logger": "http.log.access.log0",
"msg": "handled request",
"request": {
"remote_ip": "172.20.0.12",
"remote_port": "46086",
"client_ip": "172.20.0.12",
"proto": "HTTP/1.1",
"method": "POST",
"host": "caddy",
"uri": "/items/1/checkout",
"headers": {
"Accept": ["*/*"],
"User-Agent": ["curl/8.16.0"]
}
},
"bytes_read": 0,
"duration": 0.18468324,
"size": 50,
"status": 200,
"traceID": "9a69815e0ea9c6e89d1c8a76fd1ae20c",
"spanID": "fc52019e4c20415b"
}The OpenTelemetry log data model provides dedicated fields for timestamps, severity, trace context, and attributes. By extracting these values from Caddy's JSON output, you can make them available as individual fields for filtering, querying, and visualization in OpenSearch.
To do this, you'll use the transform processor. It modifies telemetry using OTTL statements, the same language as the filter conditions.
Add the following processor to otelcol.yaml:
# otelcol.yaml
processors:
transform/logs:
error_mode: ignore
log_statements:
- conditions:
- log.attributes["fluent.tag"] == "caddy.access"
statements:
# Identify the source container.
- set(resource.attributes["service.name"], "caddy")
- set(resource.attributes["container.id"],
log.attributes["container_id"])
- set(resource.attributes["container.name"],
log.attributes["container_name"])
- replace_pattern(resource.attributes["container.name"], "^/", "")
- set(log.attributes["log.iostream"], log.attributes["source"])
# Parse Caddy's JSON and populate the log record fields.
- merge_maps(log.cache, ParseJSON(log.body), "upsert") where
IsMatch(log.body, "^\\{")
- set(log.observed_time, Now())
- set(log.time_unix_nano, Int(log.cache["ts"] * 1000000000.0)) where
log.cache["ts"] != nil
- set(log.severity_text, ToUpperCase(log.cache["level"])) where
log.cache["level"] != nil
- set(log.severity_number, SEVERITY_NUMBER_DEBUG) where
log.cache["level"] == "debug"
- set(log.severity_number, SEVERITY_NUMBER_INFO) where
log.cache["level"] == "info"
- set(log.severity_number, SEVERITY_NUMBER_WARN) where
log.cache["level"] == "warn"
- set(log.severity_number, SEVERITY_NUMBER_ERROR) where
log.cache["level"] == "error"
- set(log.trace_id.string, log.cache["traceID"]) where
log.cache["traceID"] != nil and log.cache["traceID"] != ""
- set(log.span_id.string, log.cache["spanID"]) where
log.cache["spanID"] != nil and log.cache["spanID"] != ""
- set(log.attributes["log.record.original"], log.body)
- set(log.body, log.cache["msg"]) where log.cache["msg"] != nil and
log.cache["request"] == nil
# Extract HTTP attributes from access logs.
- set(log.attributes["http.request.method"],
log.cache["request"]["method"]) where log.cache["request"] != nil
- set(log.attributes["url.path"], log.cache["request"]["uri"]) where
log.cache["request"] != nil
- replace_pattern(log.attributes["url.path"], "\\?.*$", "")
- set(log.attributes["network.protocol.version"],
log.cache["request"]["proto"]) where log.cache["request"] != nil
- replace_pattern(log.attributes["network.protocol.version"],
"^HTTP/", "")
- set(log.attributes["client.address"],
log.cache["request"]["client_ip"]) where log.cache["request"] != nil
- set(log.attributes["user_agent.original"],
log.cache["request"]["headers"]["User-Agent"][0]) where
log.cache["request"]["headers"]["User-Agent"] != nil
- set(log.attributes["http.response.status_code"],
Int(log.cache["status"])) where log.cache["status"] != nil
- set(log.attributes["http.response.body.size"],
Int(log.cache["size"])) where log.cache["size"] != nil
- set(log.attributes["caddy.request.duration_seconds"],
log.cache["duration"]) where log.cache["duration"] != nil
- set(log.body, Format("%s %s %d",
[log.attributes["http.request.method"], log.attributes["url.path"],
log.attributes["http.response.status_code"]])) where
log.cache["request"] != nil
# Remove Docker's original attributes.
- delete_matching_keys(log.attributes,
"^(container_id|container_name|source|fluent\\.tag)$")The statements are grouped under a condition, so they only run on records whose
fluent.tag attribute is caddy.access (this tag is set in the Docker Compose
configuration). This means that if another container sends logs through the same
driver, Caddy's parsing rules won't affect them.
First, it identifies the source by setting service.name: caddy and moving
Docker's container metadata into resource attributes. It also renames source
to the standard log.iostream attribute.
Next, ParseJSON() reads the original JSON body into log.cache, a temporary
map used by subsequent statements. The processor extracts Caddy's timestamp and
severity, then copies traceID and spanID into the log record's trace
context.
It also maps the HTTP request details to attributes defined by the OpenTelemetry semantic conventions:
| Caddy field | OpenTelemetry attribute |
|---|---|
request.method | http.request.method |
request.uri | url.path |
request.proto | network.protocol.version |
request.client_ip | client.address |
request.headers.User-Agent | user_agent.original |
status | http.response.status_code |
size | http.response.body.size |
Where necessary, the statements adjust values to match the conventions,
including removing query strings from url.path and converting numeric fields
to integers.
Finally, the processor preserves the original JSON in
log.record.original
before replacing the body with a readable request summary. This makes the record
easier to scan without losing the original information.
Applying and verifying the log transformations
To activate the processor, add it to the logs pipeline in otelcol.yaml, after
the filter:
# otelcol.yaml
service:
pipelines:
# [...]
logs:
receivers: [fluent_forward]
processors: [filter/logs, transform/logs]
exporters: [otlp_grpc/data_prepper]Processors run in the order they're listed, so the filter discards health checks before the transform processor sees them. This avoids spending processing time on telemetry that won't be exported, such as parsing the JSON of a log record that's about to be dropped.
Then apply the updated configuration by recreating the Collector service:
docker compose up -d --force-recreate otelcolOpen OpenSearch Dashboards, navigate to Discover,
and select the caddy-logs index pattern. Set the time range to the last 15
minutes and inspect a newly ingested access log.
Expand the record to inspect its fields. You should find the HTTP method,
request path, response status, and other extracted attributes. Instead of the
original JSON string, its body field should contain a short summary such as:
POST /items/1/checkout 200The resource attributes should also identify the source as Caddy, with the corresponding container ID and name, alongside the record's severity, timestamp, and trace context:
This makes it easy to filter the logs by attributes such as
http.response.status_code or url.path, rather than searching the original
JSON string.
You can also open the preconfigured Caddy access logs dashboard to see the requests-by-status-code chart populated from these structured attributes:
Transforming span attributes
The transform processor also works with spans and metrics, so you can fix instrumentation output without changing the instrumented code.
For example, the demo application's Flask instrumentation still uses deprecated
HTTP and network attributes, such as http.method, http.status_code, and
net.peer.ip. Caddy's spans in the same trace already use their current
replacements, so the two services currently describe the same request
differently.
You can update these attributes by adding another transform processor to
otelcol.yaml, this time using trace_statements:
# otelcol.yaml
processors:
# [...]
transform/traces:
error_mode: ignore
trace_statements:
- set(span.attributes["http.request.method"],
span.attributes["http.method"]) where span.attributes["http.method"] !=
nil
- set(span.attributes["http.response.status_code"],
span.attributes["http.status_code"]) where
span.attributes["http.status_code"] != nil
- set(span.attributes["url.path"], span.attributes["http.target"]) where
span.attributes["http.target"] != nil
- set(span.attributes["url.query"], Split(span.attributes["http.target"],
"?")[1]) where IsMatch(span.attributes["http.target"], "\\?")
- replace_pattern(span.attributes["url.path"], "\\?.*$", "")
- set(span.attributes["url.scheme"], span.attributes["http.scheme"]) where
span.attributes["http.scheme"] != nil
- set(span.attributes["network.protocol.version"],
span.attributes["http.flavor"]) where span.attributes["http.flavor"] !=
nil
- set(span.attributes["user_agent.original"],
span.attributes["http.user_agent"]) where
span.attributes["http.user_agent"] != nil
- set(span.attributes["server.address"], span.attributes["net.host.name"])
where span.attributes["net.host.name"] != nil
- set(span.attributes["server.port"], span.attributes["net.host.port"])
where span.attributes["net.host.port"] != nil
- set(span.attributes["network.peer.address"],
span.attributes["net.peer.ip"]) where span.attributes["net.peer.ip"] !=
nil
- set(span.attributes["network.peer.port"],
span.attributes["net.peer.port"]) where span.attributes["net.peer.port"]
!= nil
- set(span.attributes["error.type"], span.attributes["error_type"]) where
span.attributes["error_type"] != nil
- delete_matching_keys(span.attributes,
"^(http\\.(method|status_code|target|scheme|flavor|user_agent|host|server_name)|net\\.(host|peer)\\.(name|ip|port)|error_type)$")Each set() statement copies a legacy attribute's value to its current name,
with a where condition that skips spans where the original attribute is
absent:
| Deprecated attribute | Current attribute |
|---|---|
http.method | http.request.method |
http.status_code | http.response.status_code |
http.target | url.path, url.query |
http.scheme | url.scheme |
http.flavor | network.protocol.version |
http.user_agent | user_agent.original |
net.host.name | server.address |
net.host.port | server.port |
net.peer.ip | network.peer.address |
net.peer.port | network.peer.port |
The final delete_matching_keys() statement removes the old attributes to avoid
retaining duplicate information. It also drops http.host, which duplicates
net.host.name, and http.server_name, which the current conventions removed
without a replacement.
The processor also maps the application's custom error_type attribute to
error.type,
the standard attribute for identifying the type of error that occurred.
One value needs additional handling. The deprecated http.target attribute
contains the request path and may include a query string, such as
/items?sort=price. The
URL semantic conventions
represent these separately as url.path and url.query, so the transformation
uses Split() to extract the query string before removing it from the path.
Applying and verifying the span transformations
Add the processor to the traces pipeline in otelcol.yaml, after the filter:
# otelcol.yaml
service:
pipelines:
traces:
receivers: [otlp]
processors: [filter/traces, transform/traces]
exporters: [otlp_grpc/jaeger]
# [...]Then recreate the Collector service:
docker compose up -d --force-recreate otelcolOpen Jaeger, select the shop service, and open a
recent trace. The application's spans now use the same attribute names as
Caddy's, such as http.request.method, user_agent.original, url.path, and
others:
There's one exception: Jaeger still displays http.status_code instead of
http.response.status_code. This happens because its trace view converts spans
to Jaeger's older data model, restoring the legacy attribute name. As Jaeger's
UI transitions to
OpenTelemetry's native data model, future versions may display the original
attribute unchanged.
Filtering, parsing, and renaming are just a few ways to process telemetry. You can also use processors to:
- Remove sensitive information such as passwords, API keys, or personal data before telemetry leaves your infrastructure using the redaction processor.
- Enrich telemetry with context such as deployment environments, host information, and Kubernetes pod metadata using the resource processor or Kubernetes attributes processor.
- Modify individual attributes by inserting, updating, hashing, or deleting values using the attributes processor.
- Reduce duplicate logs by combining repeated records and recording how often they occurred using the log deduplication processor.
- Limit memory usage by monitoring the Collector's memory consumption and applying backpressure or refusing incoming data when configured thresholds are reached using the memory limiter processor.
For a complete list of available processors and their configuration options, see the OpenTelemetry Collector Contrib processors list on GitHub.
Sending telemetry to multiple destinations
So far, each Collector pipeline sends telemetry to a single backend. You can also configure a pipeline with multiple exporters, allowing the Collector to send the same telemetry to several destinations. This is useful when you're migrating between backends or evaluating a new one.
To demonstrate this, you'll configure the Collector to send a copy of all three telemetry signals to Dash0, a platform built on OpenTelemetry's native data model, while continuing to export them to Jaeger, OpenSearch, and Prometheus.
Configuring Dash0 as an additional destination
First, sign up for a free Dash0 trial if you don't already have an account.
You'll need your organization's OTLP/gRPC endpoint, available under Settings > Endpoints, and an auth token with permission to ingest telemetry.
The demo repository includes an .env.example template. Copy it to .env, then
fill in your credentials:
cp .env.example .env# .env
DASH0_ENDPOINT=<your OTLP/gRPC endpoint>
DASH0_AUTH_TOKEN=<your auth token>To make these values available inside the Collector container, add them to the
otelcol service definition:
# docker-compose.yml
services:
otelcol:
# [...]
environment:
DASH0_ENDPOINT: ${DASH0_ENDPOINT}
DASH0_AUTH_TOKEN: ${DASH0_AUTH_TOKEN}Next, define a new otlp_grpc/dash0 exporter in otelcol.yaml and add it to
the exporters list of each pipeline:
# otelcol.yaml
exporters:
# [...]
otlp_grpc/dash0:
endpoint: ${env:DASH0_ENDPOINT}
headers:
Authorization: Bearer ${env:DASH0_AUTH_TOKEN}
sending_queue:
batch:
flush_timeout: 1s
service:
pipelines:
traces:
receivers: [otlp]
processors: [filter/traces, transform/traces]
exporters: [otlp_grpc/jaeger, otlp_grpc/dash0]
logs:
receivers: [fluent_forward]
processors: [filter/logs, transform/logs]
exporters: [otlp_grpc/data_prepper, otlp_grpc/dash0]
metrics:
receivers: [host_metrics]
exporters: [otlp_http/prometheus, otlp_grpc/dash0]The otlp_grpc/dash0 exporter uses the same component type as the Jaeger and
Data Prepper exporters, but with its own destination and authentication
settings. Unlike the local exporters, it uses TLS to secure the connection to
Dash0.
Notice that each pipeline now lists two exporters. Both receive the same processed telemetry, but handle delivery independently. If one destination becomes unavailable, the other may continue receiving data, so there's no guarantee that both backends will store exactly the same records.
Exporter queues and retries help handle temporary failures, but the default queues are held in memory, so pending data can be lost during restarts or prolonged outages. See this article for more on delivery reliability.
Verifying telemetry in Dash0
Recreate the Collector container to apply the updated configuration and environment variables:
docker compose up -d --force-recreate otelcolThen open Dash0's tracing view and find a failed checkout request from the
shop service. Open the trace to see how the request passed through Caddy and
the application, including the charge_payment span where the payment was
declined:
Select a span and open the Related Logs & Events tab. Because Caddy's access logs contain the trace IDs extracted by the Collector, you can find the corresponding access log for the same request.
The log includes the HTTP method, request path, response status, and other attributes extracted by the transform processor. This lets you inspect what Caddy recorded alongside the application's trace without manually searching for the matching request.
You can also explore the host metrics collected by the same Collector to examine resource usage during the relevant period. Or ask Agent0 to create a dashboard for you:
Troubleshooting common Collector issues
If the Collector won't start or telemetry isn't reaching its destination, check its logs first:
docker compose logs otelcolYou can also validate your configuration without starting the Collector:
docker compose run --rm --no-deps otelcol validate \
--config=/etc/otelcol-contrib/config.yamlIf telemetry still isn't appearing in your backend, temporarily enable the debug exporter to check whether data is making it through the pipeline.
For more detailed diagnostics, including connection failures, processor errors, retry exhaustion, and dropped telemetry, see the official guide to troubleshooting the OpenTelemetry Collector.
Final thoughts
The OpenTelemetry Collector separates how telemetry is produced from how it's processed and delivered. Your applications and infrastructure can focus on describing their behavior, while the Collector manages what happens to that data before it reaches an observability backend.
This separation gives you room to evolve your observability infrastructure. You can change backends, introduce new data sources, filter unwanted telemetry, or standardize inconsistent instrumentation without repeatedly modifying the applications producing it.
Once the Collector becomes part of your telemetry pipeline, its reliability directly affects what you can observe. Failed exports, overloaded queues, and misconfigured processors can all leave gaps in your data, even when your applications are working correctly.
As you move toward production, the official OpenTelemetry Collector documentation is a good starting point for exploring deployment, security, and operational considerations in greater depth.
















