OpenLIT

OpenLIT

OpenLIT is an OpenTelemetry-native observability platform for LLM applications that provides automatic instrumentation, GPU monitoring, and comprehensive insights into AI workloads. With support for 50+ LLM providers, vector databases, and agent frameworks, OpenLIT enables zero-code observability for generative AI applications. Monitoring LLM applications with OpenLIT helps track performance, costs, token usage, and model behavior in production environments.

Use Cases

• Monitor LLM performance across multiple providers with automatic instrumentation
• Track costs and token usage for all LLM interactions to optimize spending
• Analyze GPU utilization and performance for self-hosted models
• Understand agent behavior and complex multi-step reasoning workflows
• Debug production issues with full context of prompts, completions, and errors

Overview

OpenLIT provides OpenTelemetry-native observability for LLM applications with automatic instrumentation for 50+ LLM providers, vector databases, and agent frameworks. It captures comprehensive telemetry including prompts, completions, token usage, latency, and GPU metrics with minimal configuration.

For a complete working example, see the OpenLIT observability example in the dash0-examples repository.

Prerequisites

Before setting up OpenLIT, ensure:

• You have access to an LLM provider (e.g., Anthropic, OpenAI, etc.)

You'll need an OpenTelemetry Collector deployed to receive telemetry from OpenLIT. Consider using:

• Dash0 operator for Kubernetes
• OpenTelemetry Operator for Kubernetes

• Helm Chart for The OpenTelemetry Collector

For local development, see the OpenLIT observability example for a complete setup with docker-compose and Kubernetes.

Approach 1: OpenLIT SDK

The OpenLIT SDK provides automatic instrumentation with a single line of code.

Install Dependencies

Install the OpenLIT package:

pip install openlit

Initialize OpenLIT

Add the following code at the start of your application:

import openlit

openlit.init(
    otlp_endpoint="http://localhost:4318",
)

For HTTPS endpoints or authentication:

import openlit

openlit.init(
    otlp_endpoint="http://localhost:4318",
    otlp_headers={
        "Custom-Header": "value"
    }
)

Configuration Options

The openlit.init() function supports several parameters:

• otlp_endpoint: OpenTelemetry Collector endpoint (HTTP)
• otlp_headers: Dictionary of headers for authentication
• application_name: Custom service name (defaults to auto-detected)
• environment: Deployment environment (e.g., "production")
• collect_gpu_stats: Enable GPU monitoring (default: False)

Example with full configuration:

openlit.init(
    otlp_endpoint="http://localhost:4318",
    application_name="my-llm-app",
    environment="production",
    collect_gpu_stats=True
)

Approach 2: OpenLIT Operator (Kubernetes)

The OpenLIT Operator provides zero-code auto-instrumentation for Python applications running in Kubernetes.

Install OpenLIT Operator

Add the OpenLIT Helm repository and install the operator:

helm repo add openlit https://openlit.github.io/helm/
helm repo update

helm upgrade --install openlit-operator openlit/openlit-operator \
    --create-namespace \
    --namespace openlit

Create AutoInstrumentation Resource

Create an AutoInstrumentation custom resource to configure how pods are instrumented:

apiVersion: openlit.io/v1alpha1
kind: AutoInstrumentation
metadata:
  name: openlit-instrumentation
  namespace: your-namespace
spec:
  selector:
    matchLabels:
      instrumentation: openlit
  python:
    instrumentation:
      enabled: true
  otlp:
    endpoint: "http://otel-collector.opentelemetry.svc.cluster.local:4318"

Apply the configuration:

kubectl apply -f auto-instrumentation.yaml

Label Your Pods

Add the instrumentation label to your deployment to match the AutoInstrumentation selector:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-llm-app
spec:
  template:
    metadata:
      labels:
        instrumentation: openlit
    spec:
      containers:
        - name: app
          image: your-image:latest

The operator will automatically inject instrumentation into pods matching the instrumentation: openlit label.

What Gets Captured

OpenLIT automatically captures comprehensive telemetry following OpenTelemetry semantic conventions for GenAI:

• Full prompts and completions: Complete conversation context for debugging
• Token usage: Input, output, and cached tokens for cost tracking
• Model parameters: Temperature, max tokens, model names
• Performance metrics: Request latency, throughput, error rates
• GPU metrics: Utilization, memory usage (when enabled)
• Provider metadata: LLM provider, model versions, API endpoints

Example captured attributes include gen_ai.usage.input_tokens, gen_ai.prompt.0.content, gen_ai.request.model, and provider-specific metrics.

Verification

After setting up OpenLIT:

1. Navigate to Traces in Dash0
2. Filter by your service name
3. Look for traces with operation names from your LLM provider (e.g., anthropic.messages.create, openai.chat.completions)
4. Inspect spans to see captured prompts, completions, and token usage

References

• OpenLIT Documentation
• OpenLIT GitHub Repository
• OpenLIT Operator Helm Chart
• OpenTelemetry Semantic Conventions for GenAI