Technology
FluxCD
Monitor FluxCD GitOps controllers with OpenTelemetry: scrape Prometheus metrics, export distributed traces, and collect structured logs for full visibility into reconciliation cycles, resource health, and deployment state.
Overview
FluxCD Integration in Dash0
FluxCD is a set of continuous and progressive delivery solutions for Kubernetes, built on GitOps principles. It keeps your Kubernetes cluster in sync with sources of configuration such as Git repositories, Helm charts, and OCI artifacts, and automatically applies changes when the desired state evolves.
This Dash0 integration enables deep observability into FluxCD's GitOps Toolkit controllers — including the source, kustomize, helm, and notification controllers — so you can track reconciliation performance, detect drift, and respond to failures before they impact workloads.
For more information, visit the FluxCD Project Page.
Use Cases
Gain full visibility into your GitOps delivery pipeline with this integration:
- Monitor Reconciliation Cycles: Track the rate and duration of reconciliation runs across all Flux controllers to identify slow or stalled syncs.
- Detect Drift and Failures: Alert on failed reconciliations and resource drift before they affect running workloads.
- Analyze Controller Performance: Understand the internal health of the source, kustomize, helm, and notification controllers under load.
- Track Deployment Throughput: Measure how many Kubernetes resources Flux applies over time to gauge delivery velocity.
- Understand Dependency Chains: Correlate reconciliation events across multiple Flux resources to trace cascading failures.
Setup
Overview
This guide covers three observability signals for FluxCD:
- Metrics — The GitOps Toolkit controllers (source, kustomize, helm, and notification) each expose a Prometheus endpoint on port
8080. The OpenTelemetry Collector scrapes these and forwards them to Dash0 using thegotk_metric prefix. - Traces — FluxCD's notification controller can export distributed traces via its built-in OTel provider, correlating reconciliation events from source objects down through Kustomizations and HelmReleases.
- Logs — FluxCD controllers emit structured JSON logs that can be collected automatically by the Dash0 Operator or manually via the
filelogreceiver.
Prerequisites
Before setting up FluxCD monitoring, ensure:
- FluxCD is installed and running in your cluster (
flux checkreports healthy) - You have access to the
flux-systemnamespace - You have an OpenTelemetry Collector deployed and reachable from the
flux-systemnamespace
You'll need an OpenTelemetry Collector deployed. Consider using:
- Dash0 operator for Kubernetes (recommended)
- OpenTelemetry Operator for Kubernetes
-
Helm Chart for The OpenTelemetry Collector
Collector Configuration
FluxCD controllers expose Prometheus metrics on port 8080 in the flux-system namespace.
Option 1: Static scrape config (OTel Operator or Helm Chart)
A standard FluxCD installation creates Kubernetes Services for source-controller and notification-controller, but those Services expose port 80 (the artifact/webhook port), not port 8080 (the metrics port). You need to create metrics Services for all four controllers:
kubectl expose deployment source-controller -n flux-system --port=8080 --name=source-controller-metrics
kubectl expose deployment kustomize-controller -n flux-system --port=8080 --name=kustomize-controller-metrics
kubectl expose deployment helm-controller -n flux-system --port=8080 --name=helm-controller-metrics
kubectl expose deployment notification-controller -n flux-system --port=8080 --name=notification-controller-metricsThen configure the OpenTelemetry Collector's Prometheus receiver to scrape all four controllers:
receivers:
prometheus:
config:
scrape_configs:
- job_name: fluxcd
scrape_interval: 30s
static_configs:
- targets:
- source-controller-metrics.flux-system.svc.cluster.local:8080
- kustomize-controller-metrics.flux-system.svc.cluster.local:8080
- helm-controller-metrics.flux-system.svc.cluster.local:8080
- notification-controller-metrics.flux-system.svc.cluster.local:8080Option 2: Dash0 Operator
First create a Dash0Monitoring resource in the flux-system namespace to enable Prometheus scraping. The instrumentWorkloads.mode: none setting is required because the flux-system namespace enforces a restricted PodSecurity policy that the operator's instrumentation init container does not satisfy. Then annotate the FluxCD controller deployments so the operator's collector discovers their pods:
kubectl apply -f - <<EOF
apiVersion: operator.dash0.com/v1beta1
kind: Dash0Monitoring
metadata:
name: dash0-monitoring
namespace: flux-system
spec:
prometheusScraping:
enabled: true
instrumentWorkloads:
mode: none
EOFfor controller in source-controller kustomize-controller helm-controller notification-controller; do
kubectl patch deployment $controller -n flux-system \
--type=merge \
-p '{"spec":{"template":{"metadata":{"annotations":{"prometheus.io/scrape":"true","prometheus.io/port":"8080"}}}}}'
doneFor more options, see the Prometheus Receiver documentation.
Traces
FluxCD's notification controller can export distributed traces to an OpenTelemetry collector via its built-in otel provider. The controller converts Flux events into OTel spans that follow the Flux object dependency chain: source objects (GitRepository, HelmChart, OCIRepository, Bucket) produce root spans, and downstream objects (Kustomization, HelmRelease) become child spans within the same trace. Trace IDs use the format <AlertUID>:<revision>, so every span across a reconciliation cycle is correlated back to the originating source revision.
Create a Provider in the flux-system namespace pointing to your OpenTelemetry Collector's OpenTelemetry Protocol (OTLP) HTTP endpoint:
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
name: otel-provider
namespace: flux-system
spec:
type: otel
address: http://<otel-collector-endpoint>:4318If you are using the Dash0 Operator, replace <otel-collector-endpoint> with its stable internal service address: dash0-operator-opentelemetry-collector-service.dash0-system.svc.cluster.local.
Then create an Alert that selects the Flux resources you want to trace:
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
name: otel-alert
namespace: flux-system
spec:
providerRef:
name: otel-provider
eventSeverity: info
eventSources:
- kind: GitRepository
name: "*"
- kind: OCIRepository
name: "*"
- kind: Bucket
name: "*"
- kind: HelmChart
name: "*"
- kind: Kustomization
name: "*"
- kind: HelmRelease
name: "*"If your collector requires authentication, create a Secret with the credentials:
apiVersion: v1
kind: Secret
metadata:
name: otel-provider-creds
namespace: flux-system
stringData:
username: <username>
password: <password>Then add secretRef to your Provider spec:
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
name: otel-provider
namespace: flux-system
spec:
type: otel
address: http://<otel-collector-endpoint>:4318
secretRef:
name: otel-provider-credsHelmRepository events are excluded from tracing by design — they are not considered primary sources.For the full provider specification, see the FluxCD OTel Provider documentation.
Logs
FluxCD controllers write structured JSON logs to stderr. Each log line includes fields like controllerKind, name, namespace, reconcileID, level, and msg, making them easy to filter and correlate with metrics.
Dash0 Operator
If you applied a Dash0Monitoring resource to the flux-system namespace (as shown in the Collector Configuration section above), the Dash0 Operator automatically collects pod logs from all controllers in that namespace via the filelog receiver. No additional configuration is required.
OTel Operator or Helm Chart
To collect FluxCD controller logs manually, add a filelog receiver targeting the flux-system namespace to your OpenTelemetry Collector configuration:
receivers:
filelog:
include:
- /var/log/pods/flux-system_*/*/*.log
operators:
- type: container
- type: json_parser
timestamp:
parse_from: attributes.ts
layout: "%Y-%m-%dT%H:%M:%S.%LZ"Log and trace correlation
FluxCD logs and traces share the same dimensions — controllerKind, name, and namespace — so you can correlate them in Dash0 by filtering on those fields within the same time window. The reconcileID field in logs identifies a single reconciliation cycle and maps to the span hierarchy generated by the otel provider for the same resource. Note that reconcileID is not an OTel trace ID, so there is no automatic one-click correlation between a log line and its corresponding trace — use the resource name, kind, and timestamp to bridge them.
Verification
Metrics
- Check collector logs for successful scrape of
flux-systemtargets - Navigate to Metrics Explorer in Dash0
- Search for metrics with prefix
gotk_
gotk_reconcile_duration_seconds only appears after at least one Flux resource (e.g. a GitRepository or Kustomization) has been reconciled. If you see only gotk_token_cached_items initially, create a Flux resource and wait for the first reconciliation interval.
Key metrics to verify:
| Metric | Description |
|---|---|
gotk_reconcile_duration_seconds | Reconciliation duration per controller and resource |
gotk_token_cached_items | Items currently held in the token cache |
Traces
- Trigger a reconciliation (e.g., push a commit to a watched Git repository)
- Navigate to Traces in Dash0
- Filter by service name
<namespace>/<alert-name>— for example,flux-system/otel-alert
Logs
- Navigate to Logs in Dash0
- Filter by
namespace: flux-systemto see all controller logs - Use
controllerKindandnamefields to narrow down to a specific resource
