Monitoring Microservices with OpenTelemetry: A Guide

Monitoring Microservices with OpenTelemetry: A Guide

To monitor microservices, measure how each service is behaving, trace requests across the services they visit, and connect those traces to logs and dependency metrics. OpenTelemetry provides the common instrumentation and collection layer. You still need an observability backend to investigate the data and alert on problems that matter.

Imagine ordering a pizza on your phone. You choose a margherita, add pineapple, and tap Place order. In one plausible setup, the app sends a request to an API gateway, which passes it to an order service. The order service checks with inventory, inventory queries a database to see whether the ingredients are available, and a payment service contacts an external payment gateway to authorize the charge. The app shows you none of this. It shows a spinner, which is reasonably accurate but not terribly informative.

Now suppose checkout hangs. Perhaps the system has taken a principled stand against pineapple on pizza. Sadly, no. The payment gateway is just taking far too long to respond. The inventory check may have succeeded. Every pod in your cluster may be running happily. Yet your customer is still looking at Placing your order, wondering whether dinner is happening. From their perspective, the entire application is slow, not just the payment service. They have no reason to care how many dashboards insist everything is fine.

That's the job of microservices monitoring: reconstruct where a request went, what it waited for, and which dependency held it up. We'll get to the Collector and instrumentation shortly. First, the evidence needs to make sense.

What is microservices monitoring?

Microservices monitoring is the practice of measuring the health and performance of individual services and the work they perform together. You need to know when the application is deteriorating, which requests are affected, and which dependency is responsible. Counting running containers is not quite enough.

That pizza order may cross an API gateway, order service, inventory service, payment service, and database, possibly written in different languages and deployed by different teams. HTTP, gRPC, and message queues connect them. That is excellent for deploying independently. It also means that the end-to-end operation has very little respect for organizational boundaries.

A useful monitoring setup answers questions such as:

• Which service's request rate, error rate, or latency changed?
• Which operations and downstream dependencies were involved in a failing request?
• Is the delay in application code, a database, a queue, or an external API?
• Did a deployment, resource constraint, or traffic change contribute?
• Can an engineer get from the alert to the relevant traces and logs without guessing?

The pizza app could fail in another way too: inventory might get stuck on a database query, leaving the API gateway to time out. The gateway's alert would be telling the truth. It just wouldn't be telling you the part of the truth that fixes anything.

Why is monitoring microservices challenging?

Distributed applications make it easy to distribute software. Unfortunately, they distribute the evidence as well. Services have separate lifecycles, infrastructure is short-lived, and one slow dependency can create failures in several places. The individual measurements may all be correct; their relationships are what you have to reconstruct.

1. Distributed architecture and service dependencies

Suppose a request spends three seconds crossing this path:

API Gateway → Order Service → Payment Service → Database

Where are those three seconds going? The order service might spend most of its time waiting, and the payment service might spend most of its time waiting on the database. Adding up the latency of each service won't necessarily help either: their operations can overlap.

Distributed tracing records the operations and their relationships so you can inspect the actual path. It is much more convincing than lining up four latency charts and hoping the peaks look related.

2. Dynamic and ephemeral infrastructure

In Kubernetes, a pod name can be obsolete by the time an engineer opens the incident. That's perfectly normal behavior for Kubernetes, and rather inconvenient if your monitoring treats the pod's name as the identity of the application.

Track logical services using stable resource attributes. Keep pod, container, node, and cluster information too, because you'll eventually need to find the particular workload that misbehaved. Just don't make the continued existence of that workload a prerequisite for finding its telemetry.

3. Large volumes of telemetry data

A request that crosses six services may produce spans and log records at every hop. Multiply that by your traffic and "collect absolutely everything" stops being a philosophy and starts being a fairly specific cost model. You still need enough evidence to investigate failures, of course.

Keep the measurements needed for reliable alerting. Make sure traces and logs retain the details that are actually useful during investigations. Filter noise, choose sensible retention, and understand the consequences of sampling before discovering that the interesting request was one of the ones you threw away.

4. Correlating data across services

Two log entries saying timeout at 10:30 are not automatically about the same request. With enough traffic, finding coincident timestamps is hardly an achievement.

For precise request-level correlation, propagate trace context and write the active identifiers into logs. For service and infrastructure correlation, use consistent resource attributes. These are two different relationships, and you'll want both. A trace tells you which request was involved; resource metadata helps identify where it ran.

5. Identifying the root cause of failures

An authentication service becomes slow. Requests queue at the gateway. Downstream callers retry. Suddenly several services have elevated latency and errors, and there are plenty of alerts to go around.

Which service should you fix? Not necessarily the loudest one. Monitor dependency latency and failure propagation so you can distinguish the original problem from the perfectly understandable complaints it caused elsewhere.

The three core telemetry signals for monitoring microservices

Start with metrics to spot the change, traces to see where a request spent its time, and logs to inspect what the code reported. Then make sure you can move between them. Three excellent but unrelated datasets are still three separate investigations.

1. Metrics: Understanding service health and performance

For request-driven services, I'd begin with the RED method: rate, errors, and duration. Before getting excited about CPU utilization, find out whether requests are arriving, succeeding, and finishing in a reasonable amount of time.

Add CPU, memory, queue depth, and database connection usage when they help explain those three measurements. A payment service handling 500 requests per second at 0.2% errors is in a different situation when the error rate reaches 8%. You don't need an elaborate service map to notice that.

Look at latency percentiles such as P50, P95, and P99, not only the average. The average can look quite civilized while the slowest 1% of requests are keeping a customer staring at a spinner. Those customers are not comforted by arithmetic.

Metrics tell you which behavior deserves attention. They generally won't tell you which particular payment attempt waited on which dependency. For that, we need more detail.

2. Logs: Understanding what happened

A payment authorization timeout is worth logging. So is the information that helps an engineer find the corresponding request. A line saying Something went wrong is less helpful, even if the application produces it with great consistency.

Here's a structured error record with a service identity and trace ID:

{
  "timestamp": "2026-10-15T10:30:00Z",
  "severity": "ERROR",
  "service.name": "payment-service",
  "message": "Payment authorization failed",
  "error_type": "gateway_timeout",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Structured logging makes fields queryable instead of asking engineers to extract them from prose. When the active trace context is present, trace_id lets you find the logs for a request rather than every timeout in the same minute.

There is a prerequisite here: your logging instrumentation must actually include the active trace ID. OpenTelemetry cannot retroactively guess it from the timestamp. Logs explain what a component reported; they do not automatically reconstruct the full request path.

3. Distributed tracing: Following requests across services

Tracing puts the operations back in context. A trace describes an end-to-end request, with spans recording the work done along the way. It is especially good at exposing time spent waiting on something that happens to be somebody else's system.

Consider this illustrative checkout trace:

Checkout Request (850 ms)
|
+-- API Gateway (850 ms)
    |
    +-- Order Service (720 ms)
        |
        +-- Inventory Service (120 ms)
        |
        +-- Payment Service (540 ms)
            |
            +-- Payment Gateway API (490 ms)

The external payment call occupies 490 ms of an 850 ms checkout. It would be a sensible place to investigate first. That doesn't prove the gateway is at fault: it could be waiting on something, or the caller could be retrying. But now we have a specific operation to inspect instead of an argument over whose service dashboard looks worse.

Spans may overlap, so don't add all their durations as if they happened serially. Missing instrumentation also means missing pieces of the trace. Distributed tracing is only end-to-end when the relevant context makes it end-to-end.

Why these three signals work better together

Back to the pizza order: if checkout errors rise, metrics establish how many requests are affected. A trace shows that failures involve the payment call. A related log reports the gateway timeout. That is enough to start a targeted investigation.

Without the shared identifiers, you'd still have an error chart, a trace explorer, and some logs. All perfectly good products of your telemetry pipeline, and you'd be doing the correlation yourself. The logs, metrics, and traces overview explains the roles of each signal in more detail.

Microservices monitoring vs. microservices observability

Monitoring detects conditions you've decided to watch for; observability helps explain behavior you didn't predict in advance. Microservices need both. The distinction is not a contest between dashboards and traces, or between one product category and another.

An alert can tell you that checkout errors exceed the team's threshold. To understand why they exceed it, you need enough telemetry to follow requests, inspect dependencies, and ask questions you didn't think to preconfigure in an alert rule.

A good alert should tell the right engineer that something needs attention. The rest of the evidence should help that engineer make a useful decision. If the alert is all they have, the next step is usually a rather improvised tour of every dashboard in the company.

How OpenTelemetry helps with microservices monitoring

OpenTelemetry gives services a common way to produce telemetry and carry context across their boundaries. That is useful when checkout involves Java, a Go worker, a JavaScript gateway, and a database, none of which was consulted about the language choices of the others.

OpenTelemetry supplies APIs, SDKs, automatic instrumentation, semantic conventions, and a Collector for processing and exporting telemetry. It does not store your traces for investigation or evaluate your paging alerts. An observability backend does those jobs.

Key OpenTelemetry components

The components are easier to understand if you give each one a job: the API and SDK generate telemetry, instrumentation captures supported operations, the Collector moves and processes records, semantic conventions describe them consistently, and propagation keeps related operations together.

OpenTelemetry API and SDK

The API describes how code records telemetry; the SDK implements the behavior that turns those calls into spans and measurements. Use them for application operations that matter but that a generic library cannot know about.

A database client can tell you that a SQL query took 120 ms. It cannot tell you whether the restaurant accepted an order for a pizza it no longer has the ingredients to make. That is information your application can add through manual instrumentation. If the business operation matters to an incident, consider giving it its own span and useful, non-sensitive attributes.

Automatic instrumentation

Automatic instrumentation can capture an impressive amount of plumbing: HTTP handlers, outgoing requests, and supported database calls, often without changing application code. That is a wonderful start. It is also a list of supported integrations, not a declaration that your entire checkout is now observable.

For example, an uninstrumented message consumer can break the trail even when the request that published the message has a beautiful trace. Test the actual path, including the asynchronous parts. You'll find out quickly which libraries need extra attention.

OpenTelemetry Collector

The OpenTelemetry Collector receives, processes, and exports telemetry. It is the natural place to normalize fields, batch exports, and route telemetry to an observability backend, rather than making each service responsible for an ever-expanding list of destinations.

A typical path looks like this:

Microservices
    |
    | Metrics, Logs, Traces
    v
OpenTelemetry SDKs
    |
    | OTLP
    v
OpenTelemetry Collector
    |
    | Processing, Batching,
    | Filtering, Export
    v
Observability Backend
    |
    v
Dashboards, Alerts,
Service Maps, Analysis

Collectors are built from receivers, processors, and exporters. There is a small naming trap here. The Prometheus receiver goes out and scrapes metrics; nothing is pushing them to it. It is called a receiver because of where the data enters the Collector pipeline, not because it politely sits around waiting for an incoming network request. Worth knowing before configuring an exporter on the wrong side of the conversation.

Centralizing collection also centralizes some failure modes. If the shared Collector gateway stops exporting, a whole collection of otherwise healthy services can go quiet in your observability backend. You have not fixed production. You have misplaced its evidence. Monitor the Collector too. The Collector guide covers pipeline design in more depth.

OpenTelemetry semantic conventions

OpenTelemetry semantic conventions give you common names for things your services were previously free to name creatively. That sounds boring until you try to build one service-level query across teams using app_name, service, and service.name to mean the same thing.

Useful resource and operation attributes include:

• service.name: Logical service identity.
• service.version: Deployed version.
• deployment.environment.name: Deployment environment.
• http.request.method: HTTP request method.
• http.response.status_code: HTTP response status.

Keep the resource identity consistent and put per-deployment details in their own attributes. That supports service maps, grouping, and comparisons across releases. The semantic conventions explainer covers the conventions themselves.

Context propagation

Instrumentation can produce valid spans in every service and still leave you with several unrelated traces. The missing ingredient is often context propagation.

For synchronous HTTP requests, the W3C Trace Context traceparent header carries trace information between services. Different transports and asynchronous messaging may need other context handling. The point is to carry the relationship across the boundary, not merely create another span when work begins.

When the relationship is missing, the downstream span does not arrive with a sign saying "I'm actually part of checkout." It looks like an operation in its own right. The Trace Context guide explains the header and propagation mechanics.

How to monitor microservices using OpenTelemetry

Start with one important user journey, instrument every relevant hop, and prove you can diagnose a controlled failure. It is tempting to begin with a large rollout of instrumentation. I'd rather have one checkout path I can actually investigate than fifty services proudly reporting spans that don't connect.

Step 1: Identify your critical services and dependencies

Choose a workflow whose failure matters: checkout, login, order submission, or processing an important message. Draw its path through the gateway, application services, databases, queues, and third-party APIs. Make a note of which parts you operate and which you merely depend on.

You don't need to draw an exhaustive architecture diagram before starting. You do need a map accurate enough that you can look at a trace and say, "Wait, where did the inventory service go?" Missing dependencies are rather easier to spot when you know they were supposed to be there.

Step 2: Instrument your microservices

Add supported OpenTelemetry SDKs or automatic instrumentation, then manually instrument the business operations that library instrumentation cannot infer. Java, Python, JavaScript, Go, and .NET have OpenTelemetry support; automatic coverage depends on the runtime and libraries.

Give services stable identities using resource attributes. With an SDK or auto-instrumentation setup that honors the standard environment variables, a checkout service can use:

export OTEL_SERVICE_NAME="checkout-service"

export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.0,deployment.environment.name=production"

export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

The variables identify the service, set its deployment metadata, and point OTLP/HTTP exports at the Collector. The hostname is illustrative. And there's an easily missed detail: setting the exporter endpoint does not instrument an application. You still need an SDK or an agent that creates telemetry. A perfectly configured destination for data that never exists is of limited use.

Step 3: Deploy an OpenTelemetry Collector

On Kubernetes, Collectors commonly run as DaemonSets, Deployments, or agents feeding a central gateway. Pick the topology according to the sources you need to collect and the processing you want to centralize.

Here is a deliberately small configuration that receives all three signals and exports them over OTLP/HTTP:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlp_http:
    endpoint: ${env:OTLP_BACKEND_ENDPOINT}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_http]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_http]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_http]

The destination must accept OTLP/HTTP. A production backend may require TLS, authentication headers, and other settings; the exposed 0.0.0.0 listeners also need appropriate network controls. The exporter was renamed from otlphttp to otlp_http in Collector v0.144.0; older versions use otlphttp, while newer versions retain it as a deprecated alias.

Before putting it on a production path, add suitable memory protection, queue and retry behavior, secure transport, and filtering of sensitive data. More importantly, test what happens when the destination is unavailable. A configuration file parsing successfully is not evidence that your telemetry survives an outage.

Step 4: Configure distributed tracing

Send a known request across the critical path and inspect the resulting trace. Does it include the expected services? Can you follow the payment call? Is the database operation attached to the caller that initiated it?

If a trace stops at the boundary to another service, check context propagation before concluding that the service did no work. Include queues and asynchronous consumers in your test: they may need extra instrumentation and span links rather than the neat parent-child tree from a simple HTTP demo.

Step 5: Monitor service-level metrics

Give every request-driven service RED metrics: rate, errors, and duration. Then add measurements that explain the failures you actually see, such as CPU or memory pressure, connection saturation, queue depth, consumer lag, and dependency latency.

A dashboard with fifty beautiful resource charts may still leave out the one measurement the checkout team needs: how long checkout takes. I would start there, then ask which dependency and resource metrics explain a change. The infrastructure matters, but it is not the feature your customers are trying to use.

Step 6: Configure actionable alerts

Page people for conditions that warrant intervention: sustained error-budget burn, prolonged latency, unavailable critical services, or queues that keep growing because consumers cannot keep up. Use SLOs where possible so alerts reflect reliability objectives rather than the number of metrics you happened to collect.

For a 99.9% successful-request objective, for example, an error-budget burn alert can distinguish a serious reliability problem from a brief, low-impact blip. Configure sensible grouping and notification routes. The purpose is to call somebody when there is a problem they can act on, not to demonstrate that your metrics are capable of changing value.

Step 7: Validate your monitoring setup

Now break something, deliberately and somewhere safe. Slow down a downstream call or make an external dependency unavailable in a test environment. Ideally, ask someone who didn't build the telemetry configuration to investigate.

Verify that:

• Metrics show the effect on request rate, errors, or duration.
• The trace identifies the affected dependency or operation.
• The related logs contain enough context to understand the reported failure.
• Alerts fire and resolve when expected.
• The engineer can connect the evidence without manually matching unrelated timestamps.

If the engineer can see the data but cannot follow the request, you haven't finished the job. The nice thing about finding this out in a test environment is that you can fix it without several people simultaneously asking for an incident update.

Best practices for monitoring microservices

Once the first request path works, make it reliable as services and teams change. Most problems here are fairly ordinary: inconsistent names, context that fails to cross a boundary, an overenthusiastic metric label, or a Collector nobody remembered to monitor. They're worth fixing while everything is calm, when an empty trace explorer is merely annoying.

Monitor user-facing behavior first

Track latency, errors, availability, and important business operations before celebrating resource utilization. A service doing very little CPU work might be waiting patiently on a very slow API. A low CPU reading doesn't answer the question of why an external dependency takes two seconds to respond. It mostly tells you that waiting is not very computationally demanding.

Use consistent service naming

Use payment-service consistently, and describe environment and version separately with attributes such as deployment.environment.name and service.version. Do not alternate between payments, payments-prod, and payment-api just because different deployment scripts happen to prefer different names.

Consistency is what lets dashboards and service maps group related telemetry instead of giving you three apparent services where you operate one.

Correlate logs and traces

Include trace and span IDs in structured log records when there is valid context. Test the behavior by opening a real trace and finding its related logs. Don't settle for the fact that the fields appear in a sample message.

While you're at it, keep secrets, access tokens, payment details, and other sensitive data out of the logs. A trace ID helps investigation. A customer's card number does not.

Monitor dependencies, not just individual services

If checkout spends 490 ms waiting for a payment gateway, that dependency belongs in your investigation, even if you can't instrument the gateway itself. Measure calls to databases, brokers, caches, and external APIs. Include failures, duration, and saturation where you can observe them.

Your application's response time includes the time it spends waiting on other people's software. Conveniently, the customer receives only one total.

Control telemetry volume and cardinality

High-cardinality attributes such as request IDs are useful for traces and log investigation. They are a poor choice for metric labels, where each distinct label combination can create a new time series. If you put a unique request ID on every metric, the series count can start resembling your request count, which is an expensive way to rediscover arithmetic. See the cardinality guide for the details.

Filter unhelpful telemetry, choose retention deliberately, and sample traces with your diagnostic needs in mind. Check that the slow and failing requests you care about remain visible, and that sampling across service boundaries doesn't leave you with disconnected fragments.

Monitor your monitoring pipeline

Monitor the Collector's health, export failures, dropped records, queue pressure, and resource use. Otherwise the system responsible for telling you what's wrong can go wrong without telling you.

Don't interpret missing telemetry as an unexpected improvement in production health. Sometimes the service is quiet because it stopped reporting, not because its customers are suddenly delighted.

Connect monitoring to deployment changes

Attach service.version and deployment context to telemetry. Compare failure rate and P99 latency before and after a release; use deployment events where you have them.

If version 2.3 coincides with a latency regression, investigate the release. Don't convict it solely on the timestamp, but don't make the on-call engineer reverse-engineer the deployment history either.

How Dash0 simplifies microservices monitoring with OpenTelemetry

Dash0 gives OpenTelemetry telemetry a place where engineers can follow a service, its requests, and its dependencies during an investigation. The Service Catalog and Service Map are particularly useful here: one shows service health, the other the relationships visible in your traces.

Centralized visibility into microservices

One failed checkout can produce several kinds of evidence. Dash0 keeps service health, traces, logs, and resource context accessible in the same investigation, so an engineer doesn't have to reconstruct the request afresh for every signal.

Service Catalog and Service Map

Use the Service Catalog to find services with unusual request rates, error percentages, or latency percentiles. The Service Map helps show which services call which, based on distributed tracing data.

As with any service map, the output depends on what your instrumentation can see. If a boundary isn't instrumented or context is missing, the map doesn't invent a connection to make the picture prettier.

Correlated telemetry for troubleshooting

For the checkout example, start with the degraded checkout service, open a relevant trace, and look at the downstream payment operation and its related logs. That is the workflow you're trying to make easy: from the symptom affecting users to the evidence associated with one request.

Monitoring built around open standards

Dash0 accepts OpenTelemetry data over OTLP, so you can retain upstream instrumentation rather than writing application code against a proprietary telemetry format. On Kubernetes, the Dash0 operator provides monitoring and automatic instrumentation for supported workloads.

That keeps the instrumentation portable. It does not automatically make every dashboard or operational workflow portable, and that's a separate question. For monitoring microservices, the immediate benefit is simpler collection and useful cross-service navigation.

Frequently asked questions about monitoring microservices

What is the best way to monitor microservices?

Measure request rate, errors, and duration for critical services, propagate traces across dependencies, and attach logs with trace context. Use OpenTelemetry for consistent instrumentation and collection, and validate the setup with a controlled failure.

What are the most important metrics for microservices monitoring?

Start with RED metrics for request-serving services. Add resource utilization, dependency latency, queue backlogs, and business-specific measurements where they help explain performance and reliability.

How does OpenTelemetry help monitor microservices?

It standardizes telemetry APIs, SDKs, context propagation, resource attributes, and collection pipelines across languages and services. An observability backend still stores and analyzes the data.

What is the difference between microservices monitoring and distributed tracing?

Microservices monitoring covers service health, detection, and investigation. Distributed tracing is one of the techniques it uses to follow a request across service boundaries and see where time or failures occur.

Can you monitor Kubernetes microservices with OpenTelemetry?

Yes. Instrument your workloads and collect telemetry with the Collector and Kubernetes integrations. The OpenTelemetry Operator can manage Collectors and supported automatic instrumentation.

Is OpenTelemetry a complete monitoring tool?

No. It creates, collects, processes, and exports telemetry, but you still need a backend for storage, querying, visualization, and alert evaluation.

Why is distributed tracing important for microservices?

Because service-local metrics cannot show the complete request path. Distributed traces connect operations across services, letting you identify the dependencies contributing to a failure or slowdown.

Final thoughts

Start with a real workflow, not an aspiration to monitor everything. Instrument its services, propagate context, collect the three signals, and deliberately break a dependency. If the next engineer can find out where the request slowed down and why, you have useful microservices monitoring.

OpenTelemetry gives you the instrumentation and collection foundation; Dash0 can provide the correlated service views and investigation tools on top. For Kubernetes environments, the Dash0 operator can also simplify collecting and instrumenting supported workloads.

If you'd like to see how an OpenTelemetry-native platform handles the same microservices workload, try Dash0 free for 14 days. No credit card required.