Fluent Bit

Fluent Bit with Dash0

Fluent Bit is a fast and lightweight observability pipeline designed for collecting, processing, and forwarding telemetry data such as logs, metrics, and traces. This integration with Dash0 enables seamless telemetry data collection, enhanced analytics, and streamlined observability workflows, making it easier to monitor and troubleshoot distributed systems.

Use Cases

Streamline telemetry management and gain actionable insights with Dash0 and Fluent Bit:

• Centralized Telemetry Forwarding: Effortlessly consolidate telemetry data from multiple sources and forward it to Dash0 for unified observability.
• Enhanced Log Monitoring: Leverage Dash0's dashboards to visualize and analyze logs collected by Fluent Bit, enabling faster root-cause analysis.
• Real-Time Metrics Collection: Use Fluent Bit's lightweight processing to feed key performance metrics into Dash0 for real-time monitoring and decision-making.
• Customizable Alerting: Combine Fluent Bit's flexibility with Dash0’s alerting capabilities to proactively address anomalies in your system telemetry.

Fluent Bit Configuration

Fluent Bit has an OpenTelemetry output plugin that you can use to transmit metrics, logs, and traces to Dash0. The following snippet shows how to configure the plugin using Fluent Bit's classical configuration format.

[OUTPUT]
    Name:         opentelemetry
    Match:        *
    Host:         {{endpoint_otlp_http_hostname}}
    Port:         {{endpoint_otlp_http_port}}
    Header:       Authorization Bearer {{token}}
    Metrics_uri:  /v1/metrics
    Logs_uri:     /v1/logs
    Traces_uri:   /v1/traces
    Tls:          On

Fluent Bit Deployment using Helm

There are a variety of ways to deploy Fluent Bit. A common way to deploy it is using Helm. For alternative ways, check out the official documentation. Here we provide guidelines for a quick deployment in your Kubernetes clusters using Helm.

First, create a namespace and specify the Dash0 token that is needed to send data:

#
kubectl create namespace opentelemetry
kubectl create secret generic dash0-secrets --from-literal=dash0-authorization-token={{token}} --namespace opentelemetry

Then, create a Kubernetes ConfigMap with a sample configuration:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: opentelemetry
  name: fluent-bit-config
data:
  config.yaml: |
    service:
      log_level: info
      Daemon: Off
      Flush: 1
      HTTP_Server: On
      HTTP_Listen: 0.0.0.0
      HTTP_Port: 2020
      Health_Check: On

    pipeline:
      inputs:
        - name: tail
          Path: /var/log/containers/*.log
          multiline.parser: docker, cri
          Tag: kube.*
          Mem_Buf_Limit: 5MB
          Skip_Long_Lines: On

      filters:
        - name: kubernetes
          match: kube.*
          merge_log: on
          keep_log: off

      outputs:
        - name: opentelemetry
          match: 'kube.*'
          Host: {{endpoint_otlp_http_hostname}}
    			Port: {{endpoint_otlp_http_port}}
          header: Authorization Bearer \${DASH0_AUTHORIZATION_TOKEN}
          metrics_uri: /v1/metrics
          Logs_uri: /v1/logs
          Traces_uri: /v1/traces
          Tls:  On
EOF

Next, we define the necessary configuration for the Fluent Bit Helm chart in a file called values.yaml.

cat <<EOF >>values.yaml
env:
  - name: DASH0_AUTHORIZATION_TOKEN
    valueFrom:
      secretKeyRef:
        name: dash0-secrets
        key: dash0-authorization-token

existingConfigMap: fluent-bit-config
EOF

And then we install Fluent Bit by referencing the values.yaml file:

helm upgrade --install --namespace opentelemetry  -f values.yaml fluent-bit fluent/fluent-bit

And finally, we need to patch the DaemonSet configuration, because the Fluent Bit Helm chart does not support yaml-based configuration:

kubectl patch daemonset fluent-bit -n opentelemetry --type=json -p '[
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/args/1",
    "value": "--config=/fluent-bit/etc/conf/config.yaml"
  }
]'

References

• Official Fluent Bit documentation