Observing Emissary Ingress with OpenTelemetry and Dash0

Ingress controllers sit at the edge of every Kubernetes cluster, shaping the way external traffic flows to your services. They terminate TLS, apply routing rules, and enforce policies like authentication and rate limiting. That role makes them indispensable, but also a common source of headaches when things go wrong. If the ingress slows down, the entire platform feels sluggish. If it fails, users are cut off entirely. And if you cannot observe what happens at the ingress layer, debugging is reduced to guesswork.

This post is the fourth in our series on the state of observability for Kubernetes ingress controllers with OpenTelemetry. We began with Ingress-NGINX, the default option in many clusters. We then turned to Contour, which relies on Envoy and embraces the Gateway API. Most recently, we looked at Traefik, the most OpenTelemetry-native of the group (so far). Now it is time for Emissary Ingress - formerly known as Ambassador API Gateway. Emissary is based on Envoy but extends it into a full API gateway, adding capabilities for authentication, transformations, and advanced routing. With that extra complexity, observability is more important than ever.

All of the examples in this post come directly from the dash0-examples repo. If you want to follow along, clone the repo and apply the manifests under emissary-ingressCopy. Within minutes you will have Emissary running with traces, metrics, and logs flowing through the OpenTelemetry Collector. Everything here is grounded in that demo, and we encourage you to try it yourself.

Tracing at the edge

Tracing in Emissary is enabled using a TracingServiceCopy resource. The demo includes a manifest named tracing-service.yaml that configures Emissary to emit spans using the OpenTelemetry driver.

12345678910
apiVersion: getambassador.io/v3alpha1
kind: TracingService
metadata:
  name: otel-tracing
  namespace: emissary
spec:
  service: "otel-collector-opentelemetry-collector.opentelemetry.svc.cluster.local:4317"
  driver: opentelemetry
  config:
    service_name: emissary-ingress

This configuration instructs Emissary to send spans to the Collector at otel-collector-opentelemetry-collector.opentelemetry.svc.cluster.local:4317Copy over OTLP gRPC. Once this manifest is applied and the Emissary pods are restarted, every incoming request generates a span. If the request carries a traceparentCopy header, Emissary joins the existing trace; otherwise, it starts a new one. Either way, ingress traffic is no longer invisible in your traces.

Compared with the other controllers in this series, Emissary’s tracing sits somewhere between Contour and Traefik. Like Contour, it relies on Envoy’s OpenTelemetry/OTLP tracer and is configured globally through a single TracingServiceCopy. Any change requires restarting Emissary so Envoy reloads its bootstrap. Traefik, on the other hand, enables OpenTelemetry tracing by default but also gives you per-router overrides, so you can selectively disable or enable tracing for individual routes while keeping sensible defaults at the entry point. Emissary’s model is more straightforward: you configure tracing once, roll the pods, and the ingress becomes a first-class part of your distributed traces.

Metrics with Prometheus

Where traces illuminate individual requests, metrics reveal longer-term patterns: how much traffic is flowing through the ingress, how often errors occur, and how request latency changes under load. Emissary exposes the full set of Envoy Prometheus metrics through its admin service on port 8877. These include counters such as envoy_http_downstream_rq_totalCopy, gauges for active connections, and histograms like envoy_cluster_upstream_rq_time_bucketCopy that track upstream latency. Emissary also adds its own ambassador_*Copy series, such as ambassador_diagnostics_errorsCopy, which surface configuration or runtime issues.

In the demo, these metrics are scraped by the Collector using a Prometheus receiver defined in otel-collector-deployment.yaml. The configuration looks like this:

12345678910
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'emissary-ingress'
          scrape_interval: 15s
          static_configs:
            - targets:
              - emissary-ingress-admin.emissary.svc.cluster.local:8877
          metrics_path: /metrics

Rather than relying on pod-level discovery, this setup points directly at the emissary-ingress-adminCopy service, which fronts the admin interface of Emissary. Every 15 seconds, the Collector scrapes the /metricsCopy endpoint, enriches the data with Kubernetes metadata (via the k8sattributesCopy processor), and exports it downstream as OTLP.

The volume of data is significant: Envoy emits hundreds of series by default. This means dashboards and careful curation are essential. The raw numbers include everything from request counts and latencies to upstream resets and retry behavior. When combined with the ambassador_*Copy metrics, you gain a deep view of both traffic patterns and gateway health.

Compared with the other controllers, Emissary’s metrics story looks a lot like Contour’s, since both are Envoy-based and expose the same firehose of data. Ingress-NGINX sticks to a smaller set of NGINX-specific metrics, while Traefik stands apart by exporting OTLP metrics that align directly with OpenTelemetry semantic conventions. Emissary’s challenge is not generating enough data - it is providing so much that it requires strong dashboards and filtering to be useful.

Logs with trace context

Logs are one of the most important signals at the ingress layer. They capture fine-grained details about every request and response - client IPs, headers, upstream cluster information, and response flags. But to make logs actionable in an observability pipeline, they need to be correlated with traces.

By default, Emissary emits Envoy-style access logs that do not include trace information. In the demo, the log format has been customized to JSON and extended with the W3C traceparentCopy header. This is configured through a ModuleCopy resource, defined in ambassador-module.yaml:

1234567891011121314151617181920212223242526272829303132
apiVersion: getambassador.io/v3alpha1
kind: Module
metadata:
  name: ambassador
  namespace: emissary
spec:
  config:
    service_name: emissary-ingress
    cluster: emissary-demo
    enable_envoy_logs: true
    envoy_log_type: json
    envoy_log_format:
      "@timestamp": "%START_TIME%"
      authority: "%REQ(:AUTHORITY)%"
      bytes_received: "%BYTES_RECEIVED%"
      bytes_sent: "%BYTES_SENT%"
      duration: "%DURATION%"
      method: "%REQ(:METHOD)%"
      path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
      protocol: "%PROTOCOL%"
      request_id: "%REQ(X-REQUEST-ID)%"
      response_code: "%RESPONSE_CODE%"
      response_flags: "%RESPONSE_FLAGS%"
      traceparent: "%REQ(traceparent)%"
      upstream_cluster: "%UPSTREAM_CLUSTER%"
      upstream_host: "%UPSTREAM_HOST%"
      upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
      user_agent: "%REQ(USER-AGENT)%"
      x_forwarded_for: "%REQ(X-FORWARDED-FOR)%"
    diagnostics:
      enabled: true
    default_tracing_service: otel-tracing

With this manifest in place, every log entry now contains a traceparentCopy field in addition to the standard request and response attributes. The traceparentCopy string encodes both a trace_idCopy and a span_idCopy, but those values cannot be used directly. To make them usable for correlation, the Collector must parse and extract them.

The first step in the Collector pipeline is optional: parsing the JSON log body into structured attributes. This is handled by the transform/emissary-parse processor. It promotes fields like methodCopy, pathCopy, and response_codeCopy into attributes, which makes them queryable without having to scan raw text. While not required for correlation, this demonstrates how Envoy logs can be turned into structured telemetry.

The actual correlation happens in a second processor, transform/trace-extractCopy. This processor looks at the traceparentCopy value and slices out the trace and span IDs according to the W3C format. The relevant part of the configuration is:

12345678
processors:
  transform/trace-extract:
    log_statements:
      - context: log
        statements:
          # Extract trace_id and span_id from traceparent header (W3C format: 00-{trace_id}-{span_id}-{flags})
          - set(trace_id.string, Substring(attributes["traceparent"], 3, 32)) where attributes["traceparent"] != nil and IsMatch(attributes["traceparent"], "^00-[a-f0-9]{32}-[a-f0-9]{16}-[0-9a-f]{2}$")
          - set(span_id.string, Substring(attributes["traceparent"], 36, 16)) where attributes["traceparent"] != nil and IsMatch(attributes["traceparent"], "^00-[a-f0-9]{32}-[a-f0-9]{16}-[0-9a-f]{2}$")

These OTTL statements check that the header matches the expected format, then extract the substrings that represent the trace ID and span ID. The values are written directly into the log record’s fields, which means they will line up perfectly with spans in your distributed traces.

With this pipeline in place, logs and traces are fully correlated. In Dash0, you can click on a log entry and open its trace, or start from a trace and pivot into the logs for that request.

This approach is the same one we saw with Contour, since both rely on Envoy. Ingress-NGINX requires manual log format tweaks to add trace IDs, while Traefik is experimenting with exporting OTLP logs with correlation built in. For Emissary, extending the log format with traceparentCopy and parsing it with the Collector is a pragmatic and effective solution.

The role of the Collector

The OpenTelemetry Collector is the linchpin that turns three separate signals into one coherent stream. On its own, Emissary emits spans via OTLP, exposes Prometheus metrics, and writes structured logs. Without the Collector, each of these would end up in a different system, leaving you with silos of telemetry.

In the demo, the Collector is deployed in two modes. A DaemonSet runs on every node to capture logs locally. This instance uses the filelogCopy receiver to tail Emissary’s container stdout, applies the transform/emissary-parseCopy processor to parse the JSON body, and then runs transform/trace-extractCopy to pull out the trace and span IDs. Because the DaemonSet runs close to the pods, it can handle log volume efficiently and enrich each record before export.

A second Collector runs as a central Deployment. This one receives spans directly from Emissary over OTLP, scrapes Prometheus metrics from port 8877, and processes them centrally. Both Collectors share common processors such as k8sattributesCopy, which attach Kubernetes metadata like namespaceCopy, pod.nameCopy, and container.nameCopy to every signal. This enrichment step is critical: it allows you to filter by namespace, aggregate by service, or search for all requests hitting a particular deployment - but also Dash0 to correlate the signals.

The role of the Collector cannot be overstated. It is what transforms Envoy’s raw spans, Prometheus counters, and JSON logs into correlated observability data. With it, Emissary becomes fully transparent. Without it, you are left with three isolated views of ingress behavior.

How Emissary compares

Looking across all four ingress controllers, the pattern is clear. Tracing is now a solved problem: Ingress-NGINX, Contour, Traefik, and Emissary all support it, though with varying ease of configuration and semantic richness. Metrics are plentiful but uneven. Ingress-NGINX emits a smaller set, while Contour and Emissary provide the full Envoy firehose. Traefik is ahead in exporting OTLP metrics that follow OpenTelemetry conventions. Logs remain the hardest signal. Ingress-NGINX and Emissary require format tweaks, Contour requires parsing, and Traefik points toward the future with OTLP log export.

Emissary fits squarely in the Envoy camp. Its tracing and metrics are mature, its logs can be correlated with some customization, and the OpenTelemetry Collector is the piece that unifies them. The end result is a powerful but somewhat heavy-weight approach that gives platform engineers the visibility they need at the cluster edge.

Final thoughts

Emissary Ingress is more than a router. It is a full API gateway that secures, shapes, and manages traffic as it enters your cluster. That role makes observability critical. By enabling the OpenTelemetry tracing driver, scraping Prometheus metrics, and customizing logs, you can make Emissary fully observable. The Collector unifies these signals, and Dash0 makes them actionable.

The theme running through this series is correlation. Metrics show you that something is wrong, traces show you where it is happening, and logs tell you why. With Emissary and OpenTelemetry, the ingress layer no longer has to be a blind spot. It becomes a transparent, trustworthy part of your platform.

You can try it yourself today. Head over to the dash0-examples repo, deploy the manifests, generate some traffic, and explore the dashboards. You will see Emissary root spans in your traces, ambassador_*Copy and envoy_*Copy metrics in your dashboards, and access logs correlated with traces. It is the fastest way to experience how OpenTelemetry makes the ingress layer observable.