A production request fails, but the evidence is scattered. The API wrote one log record, a downstream service wrote another, the database recorded its own error, and the container that handled the request may already be gone.

Log aggregation is the process of collecting logs from many sources and making them available for centralized search and analysis. Instead of checking individual servers, containers, cloud services, and applications, you can investigate their logs together in one place.

In practice though, a useful aggregation pipeline usually does more than move log files. It may normalize different formats, attach service and infrastructure context, redact sensitive data, filter out noisy data, buffer records during outages, and preserve trace context so you can connect logs with the requests that produced them.

This article explains how log aggregation works and how to design a production pipeline that remains useful as your systems grow.

What is log aggregation?

Log aggregation collects log records from distributed sources and brings them under a common query and analysis layer. Those sources can include:

Application logs
Web server and reverse proxy logs
Operating system logs
Container stdout and stderr
Kubernetes workloads
Databases and message brokers
Network devices
Cloud services and serverless functions
Audit and security systems

Aggregation doesn't require every record to live in one physical database. A large system might route logs to different regions, retention tiers, security stores, or archives.

What matters is centralized queryability so that during an incident, you are able to search across logs without first figuring out which machine, pod, account, or file produced the record.

Terms such as log file aggregation, server log aggregation, and cloud log aggregation describe the same basic process applied to different sources.

Why log aggregation matters

Logs originate at distributed sources, while the questions you ask of them often span across those sources.

Suppose a checkout request passes through an API gateway, checkout service, payment service, and database. A payment failure might leave useful evidence in all four systems and looking at one source gives you only part of the request.

Aggregation makes those records much easier to investigate together.

Investigating distributed failures

Without aggregation, troubleshooting must begin with finding the right host, container, or cloud console before you can even search its logs.

That gets slower as the system grows. It also works poorly with ephemeral infrastructure, where the original container or virtual machine no longer exists when you begin investigating.

With aggregated logs, you can search across services by name, environment, severity, deployment version, request identifier, or any other context that the pipeline preserved.

Keeping logs after infrastructure disappears

Containers and virtual machines are replaceable, but the evidence they produce often needs to outlive them.

Kubernetes, for example, notes that when a pod is evicted from a node, its containers and their local logs are evicted too. Cluster-level logging uses storage with a lifecycle independent of individual nodes and Pods.

Standardizing inconsistent sources

A Java service, Nginx server, Linux journal, and cloud load balancer won't naturally produce identical records. An aggregation pipeline gives you somewhere to normalize timestamps, severity values, field names, and resource metadata before the data reaches storage.

Correlating events across services

Centralized logs become much more useful when records share context. If an application includes trace and span identifiers in its logs, you can move from a slow distributed trace to the log records produced during the same operation.

The OpenTelemetry log data model has dedicated TraceId and SpanId fields for this purpose, along with Resource information that describes the entity that generated the record.

Applying security and cost controls consistently

Aggregation also gives you a common place to remove sensitive values, discard useless records, sample high-volume events, and enforce different routing or retention policies.

These controls matter because logging cost depends heavily on how much data you process and ingest, not only on how long you retain it.

How a log aggregation pipeline works

A modern log aggregation system usually has several logical stages:

text

12345678910111213141516171819
Log sources
    |
    v
Collectors
    |
    v
Parsing + normalization + enrichment
    |
    v
Filtering + redaction + sampling
    |
    v
Buffering + transport
    |
    v
Storage
    |
    v
Search + correlation + alerting

The OpenTelemetry Collector is the vendor-neutral implementation of an open telemetry pipeline and is a natural linchpin for log aggregation.

It receives telemetry through receivers, processes it through processors, and sends it onward through exporters. For logs, that means it can tail files, receive OTLP and other protocols, parse and enrich records, filter or redact data, buffer records during downstream failures, deduplicate repetitive entries, and route them to one or more destinations.

Its role is broader than log aggregation too as the same Collector can receive and process metrics and traces alongside logs, using the same OpenTelemetry data model.

That gives you one standards-based telemetry pipeline instead of separate collection infrastructure for each signal, while the backend remains responsible for storage, querying, dashboards, and alerting.

The rest of this section follows those stages from log generation through centralized access.

1. Generating logs at the source

Applications and infrastructure first need to produce useful records. For applications you control, structured output is much easier to process than free-form text. A payment error might look like this in JSON format:

json

1234567891011
{
  "timestamp": "2026-09-09T08:14:13Z",
  "level": "error",
  "message": "payment authorization failed",
  "payment": {
    "provider": "example-pay"
  },
  "order": {
    "id": "ord_7291"
  }
}

The aggregation pipeline can later map fields such as the timestamp, level, and message into a common telemetry model, such as the OpenTelemetry LogRecord.

The same event could also be represented as plain text:

text

1
2026-09-09 08:14:13 ERROR payment authorization failed

Both formats can be aggregated, but the JSON record is far easier to process in an aggregation pipeline. If you cannot emit structured logs at the source, the collector can parse unstructured text and extract the fields you need before the records continue through the pipeline.

2. Ingesting logs into the aggregation pipeline

This is where aggregation begins. A collector reads logs from their original sources and brings them into a shared pipeline where they can be processed and forwarded consistently.

Depending on the environment, the collector might:

Tail log files on a virtual machine,
Read from journald
Collect container logs from a Kubernetes node
Consume logs from a cloud logging service
Receive logs over the OTLP

Once collected, records from different applications, hosts, containers, and cloud services can pass through the same processing and routing layer before reaching centralized storage.

3. Parsing and normalizing records

Once logs enter the aggregation pipeline, the next challenge is making records from different sources consistent enough to search together. A collector might turn a plain-text record such as:

text

1
2026-09-09T08:14:13Z ERROR checkout failed order=ord_7291

into separate fields for the timestamp, severity, message, and order identifier.

Parsing extracts those values from the original record. Normalization then maps different source conventions into a common representation. One application might write level=warn, another severity=WARNING, while a third uses a numeric severity code.

Without normalization, you end up writing source-specific queries for what is really the same concept. With it, records from several services can be searched and filtered consistently.

Parsing also has to handle records that span multiple lines. A Java exception, for example, might look like this:

text

1234
ERROR request failed
java.lang.RuntimeException: connection unavailable
    at example.PaymentClient.charge(PaymentClient.java:81)
    at example.Checkout.run(Checkout.java:42)

If the collector treats each physical line as a separate log record, the exception becomes fragmented and much harder to investigate. Multiline processing reconstructs the complete event before the rest of the parsing and normalization pipeline runs.

4. Enriching logs with contextual data

A message such as connection timed out doesn't tell you enough on its own. To investigate it, you also need context about the event and the system that produced it.

That context can include event-specific fields such as identifiers, error codes, request details, user or account information, and trace or span IDs that help you connect the record to the operation that produced it.

It should also include information about the log source, such as the service, deployment, host, container, Kubernetes workload, cloud region, or environment that produced the record.

Adding this context during aggregation makes records much easier to filter and correlate. Instead of searching only for connection timed out, you can narrow the results to one service, deployment, Kubernetes workload, request, or host environment.

Opentelemetry distinguishes between attributes that describe the individual log record and resource attributes that describe the entity producing the telemetry. its semantic conventions define consistent names for common service, deployment, cloud, container, and event metadata, so that logs from different sources can carry the same context using the same names once they enter the aggregation pipeline.

5. Filtering, redacting, and deduplicating records

Collecting a log doesn't automatically mean you should retain it unchanged. Before records reach storage, the aggregation pipeline can remove noise, protect sensitive data, and eliminate redundant events.

That can include:

Discarding repetitive health-check records
Suppressing low-value debug logs
Deduplicating repeated application events
Sampling high-volume informational logs
Redacting passwords, tokens, or personal data

Applying these controls before storage reduces unnecessary ingestion and makes the retained logs easier to work with.

In a shared aggregation pipeline, you may also need per-source limits so that one misbehaving or unusually noisy service can't consume enough capacity to degrade log collection for every other workload.

6. Buffering and transporting logs reliably

Log aggregation depends on networks and downstream systems that can fail or slow down. Collectors therefore need a way to buffer records and retry delivery without immediately dropping data.

For short disruptions, an in-memory queue may be enough, but longer outages may justify persistent disk-backed queues or an external message broker when you need stronger durability and decoupling.

Backpressure can also travel upstream, so if a collector stops consuming from a blocking source because its downstream queue is full, the application writing to that source may eventually stall as well.

The important point is that buffering only protects records after they reach that part of the pipeline. Logs can still be lost earlier if the source disappears or the Collector falls behind.

For the OpenTelemetry Collector specifics, including retries, queue sizing, and persistent storage, see this article.

7. Storing logs for centralized access

Once logs have been collected and processed, they're sent to a backend where they can be stored and queried together.

Centralized access doesn't require every record to live in one physical repository. Logs may be split across regions, retention tiers, or separate security stores, as long as you can search and investigate them through a common interface.

If you're building the aggregation pipeline around OpenTelemetry, choose a backend that works well with its data model and protocols rather than forcing the telemetry through a proprietary translation layer. See our guide to choosing an OpenTelemetry backend for the tradeoffs to consider.

From an aggregation perspective, the goal is simply making logs available for investigating problems without requiring you to know where they were originally produced.

A Kubernetes log aggregation example

Kubernetes is a good example of why log aggregation matters because workloads are distributed across nodes and Pods are often short-lived.

Applications commonly write logs to stdout and stderr, allowing the container runtime to capture those streams, and the kubelet manages the resulting log files on each node.

For cluster-level logging, Kubernetes recommends using a node-level logging agent, typically deployed as a DaemonSet so that each node has a collector.

A typical path looks like this:

text

12345678910111213141516171819
Application container
       |
       | stdout / stderr
       v
Container runtime
       |
       v
Node log files
       |
       v
DaemonSet collector
       |
       +--> parse
       +--> add Kubernetes metadata
       +--> filter / redact
       +--> buffer / retry
       |
       v
Observability backend

The application only needs to write logs normally. The node collector handles aggregation, adds Kubernetes context such as Pod, namespace, and container metadata, then forwards the records to the backend.

Sidecar collectors are possible when a workload needs specialized collection, but a node-level DaemonSet is usually the simpler default when many Pods share the same logging policy.

A concrete OpenTelemetry Collector pipeline

The following configuration shows a practical Kubernetes log aggregation pipeline using the OpenTelemetry Collector. It tails Pod logs, persists file offsets, adds Kubernetes metadata, limits memory usage, and forwards the records over OTLP with a persistent sending queue.

yaml

123456789101112131415161718192021222324252627282930313233343536373839404142
# otelcol.yaml
receivers:
  file_log:
    include: [/var/log/pods/*/*/*.log]
    include_file_path: true
    storage: file_storage
    operators:
      - type: container
        add_metadata_from_filepath: true

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

  k8sattributes:
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid

exporters:
  otlp_grpc:
    endpoint: observability.example.com:4317
    sending_queue:
      storage: file_storage
      batch: {}

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage

service:
  extensions:
    - file_storage

  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, k8sattributes]
      exporters: [otlp_grpc]

The filelog receiver reads the node's container logs, while the container operator parses the container log format and extracts metadata from the file path. The k8sattributes processor then uses the Pod UID to attach Kubernetes context to each record.

file_storage is used in two places for different reasons:

In the filelog receiver, it persists file offsets so the Collector can resume reading after a restart.
In the OTLP gRPC exporter, it backs the sending queue so logs waiting for delivery aren't kept only in memory.

The memory_limiter runs first in the processor chain to prevent the Collector from exhausting its memory when downstream components slow down. Exporter-side batching keeps batching close to the persistent sending queue rather than introducing another in-memory buffer earlier in the pipeline.

In Kubernetes, /var/lib/otelcol/storage should be mounted from persistent writable storage. For a node-level DaemonSet, a hostPath lets a replacement Collector Pod on the same node reuse its offsets and queued data. You also need to mount /var/log/pods from the host and grant the Collector the Kubernetes permissions required by k8sattributes.

When to add a Collector gateway

A node-level Collector can send logs directly to the backend, which is often enough for smaller environments:

text

1234
Node agents
    |
    v
Backend

As the deployment grows, you may want to insert a Collector gateway between the agents and the backend:

text

1234567
Node agents
    |
    v
Collector gateway
    |
    v
Backend

The OpenTelemetry gateway deployment pattern uses one or more standalone Collectors as a central OTLP endpoint for upstream agents.

A gateway is useful when you want to centralize backend credentials, shared filtering and routing rules, egress from the cluster, or other processing that would otherwise have to run independently on every node.

It also adds another network hop and another component to operate, so only use it when centralizing those responsibilities solves a real scaling or operational problem, instead of treating it as a required part of every aggregation pipeline.

Monitoring the log aggregation pipeline

Your aggregation pipeline is production infrastructure, so its own health needs to be observable. A failure anywhere between collection and export can leave you without the logs you expect during an incident.

Some useful signals to monitor include:

Records received and exported
Dropped records and failed exports
Retry activity
Queue size and capacity
Parsing failures
Collector CPU and memory usage
File collection lag

For OpenTelemetry Collector pipelines, the resiliency documentation recommends monitoring queue usage and other signals that indicate exporters are falling behind.

Without this visibility, the aggregation pipeline can fail silently, leaving you without the logs you need most when an incident occurs.

Log aggregation best practices

A few principles cover most production deployments:

Produce structured logs where you control the application.
Keep collection independent from application business logic where practical.
Preserve service, infrastructure, timestamp, and trace context.
Persist file offsets when restarting a collector could lose read position.
Design queues and retries around realistic backend failures.
Filter or redact records before unnecessary downstream processing.
Monitor collection lag, queue pressure, failures, and dropped records.

The objective isn't to collect the maximum possible number of records. You need the evidence that will help you understand what happened, with enough context and reliability that it is still there when you need it.

Log aggregation best practices

A few principles make log aggregation more reliable and easier to operate:

Produce structured logs where you control the application so the pipeline doesn't have to reconstruct important fields from free-form text.
Keep log collection separate from application business logic. This gives you one place to handle buffering, retries, enrichment, routing, and backend credentials without pushing those concerns into every service.
Preserve service, infrastructure, and trace context as logs move through the pipeline. Aggregation is much less useful if records arrive centrally but lose the information that tells you where they came from.
Persist file offsets when a collector restart could lose read position. This is especially important for file-based collection, where losing state can lead to gaps or duplicate reads after a restart.
Design queues and retries around realistic downstream failures, and make sure backpressure can't let one overloaded destination or noisy service degrade collection for everything else.
Filter, redact, and route records before unnecessary downstream processing. Apply per-source limits where needed so a single chatty workload can't consume a disproportionate share of pipeline capacity.
Monitor the aggregation pipeline itself. Track collection lag, queue pressure, export failures, dropped records, and Collector resource usage so failures in the logging path don't remain invisible until you need the logs during an incident.

The goal is to preserve the evidence you are likely to need, with enough context and delivery reliability that it remains available when something goes wrong.

Final thoughts

Log aggregation makes logs from distributed systems available through a common pipeline, so you can investigate problems without hunting through individual servers, Pods, cloud services, or files.

A useful pipeline preserves the structure and context that make those records actionable, while handling failures between the original source and the backend. That means thinking about collection, parsing, enrichment, buffering, delivery, and the health of the pipeline itself.

OpenTelemetry gives you an open, vendor-neutral foundation for this architecture. The OpenTelemetry Collector can aggregate logs alongside metrics and traces while preserving shared resource and trace context across signals.

Dash0 is OpenTelemetry-native, so you can ingest logs, metrics, and traces through the same standards-based telemetry pipeline without translating them into a proprietary collection model first. That gives you a clean path from aggregation to investigation while preserving the context that connects the signals.

What Is Log Aggregation? How It Works and Best Practices