kgateway

Overview

kgateway is a cloud-native API Gateway and AI Gateway built on Envoy Proxy that implements the Kubernetes Gateway API. It provides a modern, standards-based solution for managing ingress traffic in Kubernetes clusters with advanced routing, observability, and AI traffic management capabilities.

The Dash0 integration provides comprehensive observability for kgateway deployments using native OpenTelemetry instrumentation for distributed tracing, Prometheus metrics, and structured access logs with automatic trace correlation.

For more information, visit the kgateway Documentation.

Use Cases

Gain comprehensive visibility into your kgateway setup and optimize performance with the following observability features:

• Monitor Request Flows with Native Tracing: Visualize the complete lifecycle of requests from gateway through backend services. kgateway's native OTLP/gRPC distributed tracing provides insights into latency, routing decisions, and upstream interactions—ideal for debugging complex traffic flows.
• Track Performance and Traffic Metrics: Collect real-time performance data including request rates, response times, error rates, connection states, and resource utilization across all routes with Envoy's comprehensive metrics.
• Analyze Traffic Patterns with Access Logs: Record detailed information about incoming requests with automatic trace correlation using OTLP LogRecord format, enabling deep analysis of traffic patterns, error conditions, and performance bottlenecks.
• Monitor AI Gateway Traffic: Track and analyze AI traffic patterns, including LLM requests, agent interactions, and inference workloads with agentgateway integration.
• Ensure Gateway Health: Monitor controller health, proxy resource usage, and configuration synchronization to ensure your gateway infrastructure operates reliably.

Use this integration to monitor ingress traffic, debug routing issues, analyze performance patterns, and ensure your Kubernetes Gateway API implementation operates reliably at scale.

Overview

kgateway includes native support for observability via OpenTelemetry distributed tracing (OTLP/gRPC), Prometheus metrics, and structured access logs using OTLP LogRecord format with automatic TraceID and SpanID correlation.

This integration uses the recommended pattern of deploying an OpenTelemetry Collector in your cluster to collect telemetry from kgateway and forward it to Dash0.

For more details, refer to the kgateway Documentation and Envoy Observability Documentation.

Configure

Prerequisites

You'll need an OpenTelemetry Collector deployed in your cluster to collect telemetry from kgateway and forward it to Dash0.

For production environments, consider using one of these deployment methods:

• Dash0 operator for Kubernetes
• OpenTelemetry Operator for Kubernetes

• Helm Chart for The OpenTelemetry Collector

For a complete working example with collector configuration, see the Dash0 kgateway Example.

Collecting Telemetry

Step 1: Configure OpenTelemetry Collector

The OpenTelemetry Collector needs specific configuration to collect all three signals from kgateway:

OTLP Receiver for Traces and Logs

Configure the Collector to receive distributed traces and access logs from kgateway using the otlp receiver:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

kgateway sends both traces and logs to the same OTLP receiver. Traces use W3C trace context propagation, and access logs are sent as OTLP LogRecords with automatic TraceID and SpanID correlation—no manual parsing or transformation needed.

Prometheus Receiver for Metrics

Configure the Collector to scrape metrics from both the kgateway controller and proxy using the prometheus receiver:

receivers:
  prometheus:
    config:
      scrape_configs:
        # kgateway controller metrics
        - job_name: "kgateway-controller"
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names:
                  - kgateway-system
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
              action: keep
              regex: kgateway
            - source_labels: [__meta_kubernetes_pod_container_port_name]
              action: keep
              regex: metrics
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod
            - source_labels: [__meta_kubernetes_namespace]
              target_label: namespace

        # kgateway gateway proxy metrics (Envoy)
        - job_name: "kgateway-proxy"
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names:
                  - kgateway-system
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"
            - source_labels: [__meta_kubernetes_pod_label_kgateway]
              action: keep
              regex: kube-gateway
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+)
              replacement: ${1}:9091
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod
            - source_labels: [__meta_kubernetes_namespace]
              target_label: namespace

Step 2: Configure HTTPListenerPolicy for Telemetry

Create an HTTPListenerPolicy to configure tracing and access logs for your Gateway:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: HTTPListenerPolicy
metadata:
  name: telemetry-policy
  namespace: kgateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: http
  tracing:
    provider:
      openTelemetry:
        serviceName: kgateway-http
        grpcService:
          backendRef:
            name: otel-collector-opentelemetry-collector
            namespace: opentelemetry
            port: 4317
    spawnUpstreamSpan: true
  accessLog:
    - openTelemetry:
        grpcService:
          backendRef:
            name: otel-collector-opentelemetry-collector
            namespace: opentelemetry
            port: 4317
          logName: "kgateway-access-logs"
        body: >-
          {"start_time": "%START_TIME%", "method": "%REQ(:METHOD)%", "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%", "protocol": "%PROTOCOL%", "response_code": %RESPONSE_CODE%, "response_flags": "%RESPONSE_FLAGS%", "bytes_received": %BYTES_RECEIVED%, "bytes_sent": %BYTES_SENT%, "duration": %DURATION%, "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%", "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%", "user_agent": "%REQ(USER-AGENT)%", "request_id": "%REQ(X-REQUEST-ID)%", "authority": "%REQ(:AUTHORITY)%", "upstream_host": "%UPSTREAM_HOST%"}

Step 3: Create ReferenceGrant for Cross-Namespace Access

Since the HTTPListenerPolicy in kgateway-system needs to reference the collector service in opentelemetry namespace, create a ReferenceGrant:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-otel-collector-access
  namespace: opentelemetry
spec:
  from:
    - group: gateway.kgateway.dev
      kind: HTTPListenerPolicy
      namespace: kgateway-system
  to:
    - group: ""
      kind: Service
      name: otel-collector-opentelemetry-collector

Verification

To verify that telemetry is flowing:

1. Check traces in Dash0: Filter by service.name = "kgateway-http"
2. Check metrics: Search for envoy_* and kgateway_* metrics
3. Check logs: View access logs with correlated trace IDs
4. Test trace propagation: Make a request and verify traces span from gateway to backend

curl -H 'Host: your-app.example.com' http://your-gateway-ip/

Additional Resources

• kgateway Documentation
• Gateway API Documentation
• Dash0 kgateway Example
• OpenTelemetry Collector Documentation