kagent

kagent

kagent is an open-source AI agent platform designed for cloud-native Kubernetes environments. It provides a framework for building, deploying, and running AI-powered automation solutions that can diagnose issues, optimize performance, and manage infrastructure through intelligent agents. Built on open standards like the Model Context Protocol (MCP) and Agent2Agent (A2A) protocol, kagent brings agentic AI capabilities directly into your Kubernetes workflows with built-in observability through OpenTelemetry.

Monitoring kagent with OpenTelemetry provides complete visibility into your AI agent operations, including distributed traces of LLM calls, tool executions, agent decision-making flows, and the performance of multi-agent orchestration patterns.

Use Cases

• Monitor AI agent execution across your Kubernetes cluster to understand agent behavior and optimize performance.
• Track LLM inference latency and costs by observing calls to providers like Anthropic, OpenAI, Azure OpenAI, and others.
• Analyze tool execution patterns to identify which MCP tools agents are using and how effectively they solve problems.
• Understand agent orchestration flows through distributed tracing of multi-agent interactions and A2A protocol communication.

Overview

kagent has built-in OpenTelemetry support for tracing and logging. By enabling these features and configuring an OpenTelemetry Collector, you can send all agent execution telemetry to Dash0, including LLM calls, tool executions, and agent decision flows.

This guide shows you how to deploy kagent on Kubernetes with OpenTelemetry tracing and logging enabled, forwarding all telemetry data to Dash0 for complete visibility into your AI agent operations.

Prerequisites

Before setting up kagent monitoring, ensure:

• Kubernetes cluster (1.27+)
• Helm 3.x installed
• API key for your LLM provider (Anthropic, OpenAI, Azure OpenAI, etc.)
• OpenTelemetry Collector deployed with OTLP receivers (gRPC on port 4317)

You'll need an OpenTelemetry Collector that accepts OTLP and exports to Dash0. Consider using:

• Dash0 operator for Kubernetes (recommended)
• OpenTelemetry Operator for Kubernetes

• Helm Chart for The OpenTelemetry Collector

Install kagent with OpenTelemetry

Step 1: Create kagent namespace and LLM provider secret

# Create namespace
kubectl create namespace kagent

# Create secret for your LLM provider (example with Anthropic)
kubectl create secret generic anthropic-api-key \
  --from-literal=api-key="<your_anthropic_api_key>" \
  --namespace kagent

For other providers, see the kagent documentation on LLM providers.

Step 2: Create kagent values file

Create kagent-values.yaml with OpenTelemetry enabled:

# Enable OpenTelemetry tracing and logging
otel:
  tracing:
    enabled: true
    exporter:
      otlp:
        endpoint: http://<collector-service>.<namespace>.svc.cluster.local:4317
        timeout: 15
        insecure: true
  logging:
    enabled: true
    exporter:
      otlp:
        endpoint: http://<collector-service>.<namespace>.svc.cluster.local:4317
        timeout: 15
        insecure: true

# Configure your LLM provider (example with Anthropic)
providers:
  default: anthropic
  anthropic:
    provider: Anthropic
    model: claude-sonnet-4-20250514
    apiKeySecretRef: anthropic-api-key
    apiKeySecretKey: api-key

# Configure which pre-installed agents to enable
agents:
  k8s-agent:
    enabled: true
  kgateway-agent:
    enabled: false
  istio-agent:
    enabled: false
  promql-agent:
    enabled: false
  observability-agent:
    enabled: false

Note: Update the endpoint to match your OpenTelemetry Collector service name and namespace.

Step 3: Install kagent

helm repo add kagent https://kagent-dev.github.io/kagent/helm
helm repo update

helm upgrade --install kagent kagent/kagent \
  --namespace kagent \
  --create-namespace \
  -f kagent-values.yaml

Verification

Access kagent UI

# Port forward to kagent UI
kubectl port-forward svc/kagent-ui -n kagent 8080:8080

Open http://localhost:8080 to access the kagent dashboard.

View traces in Dash0

1. Navigate to your Dash0 account
2. Go to the Traces section
3. Filter by service.name=kagent* to see all kagent traces
4. You should see:
   • Agent execution traces
   • LLM API calls to your provider (Anthropic, OpenAI, etc.)
   • Tool execution spans
   • Multi-agent orchestration flows

Create an Observability Agent with Dash0 MCP

kagent agents can use Dash0's Model Context Protocol (MCP) server to query observability data directly. This enables agents to help with SRE tasks, troubleshooting, and performance analysis by accessing logs, traces, metrics, and service information.

Prerequisites

You'll need a Dash0 auth token for MCP access. While you can use your existing auth token, we recommend creating a dedicated token for MCP:

1. Go to Organization Settings > Auth Tokens
2. Create a new auth token with appropriate permissions:
   • Full access (*) - Recommended for full MCP functionality
   • Read-only (*:read) - For read-only access to observability data

Step 1: Deploy Dash0 MCP Server

Create dash0-mcp.yaml to configure the MCP connection:

apiVersion: kagent.dev/v1alpha2
kind: RemoteMCPServer
metadata:
  name: dash0-mcp
  namespace: kagent
spec:
  url: { { endpoint_mcp } }
  headers:
    - key: "Authorization"
      value: "Bearer {{token}}"

Deploy the MCP server configuration:

kubectl apply -f dash0-mcp.yaml

Step 2: Create an Observability Agent

Create observability-agent.yaml that uses Dash0 MCP tools:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: observability-agent
  namespace: kagent
spec:
  type: Declarative
  description: "An agent that helps analyze observability data from Dash0"
  declarative:
    systemMessage: |
      You are an observability expert assistant. You have access to Dash0 observability tools
      that allow you to query logs, spans, metrics, and traces.

      CRITICAL: All Dash0 tools require ISO 8601 timestamps for time ranges.
      ALWAYS call datetime_get_current_time FIRST to get the current timestamp, then use it
      to calculate relative time ranges (e.g., NOW - 1 hour for start, NOW for end).

      When asked about system behavior, performance issues, or debugging:
      - FIRST: Call datetime_get_current_time to get current time
      - Use getServiceCatalog to understand available services
      - Use getServiceDetails to get detailed service information
      - Use getSpans to search for relevant traces
      - Use getTraceDetails to analyze specific traces
      - Use getLogRecords to find related logs
      - Use getMetricCatalog and promql for performance metrics
      - Use getSpanCorrelations and getLogCorrelations to identify patterns

      Provide clear insights and actionable recommendations based on the data.
    modelConfig: default-model-config
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          toolNames:
            - datetime_get_current_time
      - type: McpServer
        mcpServer:
          name: dash0-mcp
          kind: RemoteMCPServer
          toolNames:
            - getAttributeKeys
            - getAttributeValues
            - getK8sPods
            - getLogCorrelations
            - getLogRecords
            - getMetricCatalog
            - getMetricDetails
            - getServiceCatalog
            - getServiceDetails
            - getSpanCorrelations
            - getSpans
            - getTraceDetails
            - listDashboards
            - promql

Deploy the observability agent:

kubectl apply -f observability-agent.yaml

Step 3: Interact with the Agent

Access the kagent UI at http://localhost:8080 and select the observability-agent. You can now ask questions like:

• "What services are running in my cluster?"
• "Show me recent errors in the kagent service"
• "What are the slowest spans in the last hour?"
• "Find correlated logs for failing requests"
• "What's the p95 latency for my API endpoints?"

The agent will use Dash0 MCP tools to query your observability data and provide insights. All agent interactions are traced and sent to Dash0, allowing you to see how the agent uses different tools to answer your questions.

References

• kagent Official Documentation
• kagent GitHub Repository
• kagent Observability Guide
• kagent LLM Providers
• OpenTelemetry Collector Documentation