Dash0 acquires Polar Signals
Dash0's Istio integration provides visibility into your systems with service mesh and open-telemetry monitoring.
Istio is one of the most commonly used service meshes. Istio configures Envoy instances under the hood and handles networking for you. Service meshes are somewhat complex, and visibility into their activity can be the difference between easy troubleshooting and a long-winded analysis.
Enhance observability and gain actionable insights into Istio-managed services:
Istio has native support for the generation of OpenTelemetry sigals:
Perfect for the integration with Dash0.
The best practice is using an opentelemetry collector instead of sending telemetry to backend (Dash0) directly. An easy way is using opentelemetry-collector helm chart.
Before deploying the OpenTelemetry Collector, create a secret in the monitoring namespace to store your Dash0 Authorization Token:
export DASH0_AUTH_TOKEN="{{token}}"
kubectl create secret generic dash0-secrets --from-literal=dash0-authorization-token=${DASH0_AUTH_TOKEN} --namespace monitoringThis secret allows the OpenTelemetry Collector to authenticate with Dash0 and send telemetry data.
helm upgrade --install otel-collector open-telemetry/opentelemetry-collector \
--namespace monitoring --create-namespace \
-f values.yamlAn example of values.yaml file would be
mode: deployment
image:
repository: otel/opentelemetry-collector-k8s
resources:
limits:
memory: 512Mi
requests:
memory: 256Mi
serviceAccount:
create: false
name: otel-collector-sa
extraEnvs:
- name: DASH0_AUTHORIZATION_TOKEN
valueFrom:
secretKeyRef:
name: dash0-secrets
key: dash0-authorization-token
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
ports:
metrics:
enabled: true
containerPort: 8888
servicePort: 8888
protocol: TCP
config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- honor_labels: true
job_name: kubernetes-pods
# This specifies that Prometheus should discover targets based on Kubernetes pods.
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: drop
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
# Set the address for IPv6 pods based on the prometheus.io/port annotation
- action: replace
regex: (\d+);(([A-Fa-f0-9[]{1,4}::?){1,7}[A-Fa-f0-9[]{1,4})
replacement: "[$$2]:$$1"
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_port
- __meta_kubernetes_pod_ip
target_label: __address__
# Set the address for IPv4 pods based on the prometheus.io/port annotation
- action: replace
regex: (\d+);((([0-9[]+?)(\.|$$)){4})
replacement: $$2:$$1
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_port
- __meta_kubernetes_pod_ip
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
replacement: __param_$$1
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
# Drop targets that are in non-running phases: Pending, Succeeded, Failed, Completed
- action: drop
regex: Pending|Succeeded|Failed|Completed
source_labels:
- __meta_kubernetes_pod_phase
# Add node label based on the Kubernetes pod node name
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: nod
processors:
batch:
send_batch_size: 1000
timeout: 10s
memory_limiter:
check_interval: 5s
limit_mib: 1000
spike_limit_mib: 20
exporters:
otlp:
endpoint: { { dash0_otlp_grpc_endpoint } }
tls:
insecure: false
headers:
Authorization: "Bearer {{DASH0_AUTHORIZATION_TOKEN}}"
sending_queue:
enabled: true
num_consumers: 4
queue_size: 100
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp]
metrics:
receivers: [otlp, prometheus]
processors: [memory_limiter, batch]
exporters: [otlp]Istio exposes metrics in default via annotations:
prometheus.io/path: /stats/prometheus
prometheus.io/port: "15020"
prometheus.io/scrape: "true"With an OpenTelemetry collector as shown above we can scrape and export metrics to Dash0. See istio prometheus integration.
You need to add an extensionProvider and enable tracing.
cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
extensionProviders:
- name: otel-collector
opentelemetry:
port: 4317
service: {{otel_collector_otlp_grpc_endpoint}}
resource_detectors:
environment: {}
EOFNext, enable tracing:
kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: otel-tracing
spec:
tracing:
- providers:
- name: otel-collector
randomSamplingPercentage: 100
customTags:
"{{custom_attribute_name}}":
literal:
value: "{{custom_attribute_value}}"
EOFAn istio service entry for the OpenTelemetry collector service is normally required. Make sure the port name starts with grpc-
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: otel-tracing
namespace: {{istio_namespace}}
spec:
hosts:
# This should be the FQDN of opentelemetry-collecotr service
- {{otel_collector_otlp_grpc_endpoint}}
ports:
- number: 4317
name: grpc-otlp-dash0
protocol: GRPC
resolution: DNS
location: MESH_INTERNAL
EOFIstio Citadel failing to rotate certificates
Alert triggers when certificate signing request failures occur.
rate({otel_metric_name="citadel_server_csr_count", status!="success"}[5m]) > 0.1Istio Galley failing to validate configurations
Alert triggers when Galley configuration validation failures occur.
rate({otel_metric_name="galley_validation_failed"}[5m]) > 0.1Istio control plane having cluster management issues
Alert triggers when istiod loses visibility of remote clusters.
{otel_metric_name="istiod_managed_clusters", cluster_type="remote"} == 0Istio control plane component is down
up{job=~"istiod|pilot|citadel|galley"} == 0High percentage of HTTP 4xx responses in Istio (above 5%)
sum(rate(istio_requests_total{reporter="destination", response_code=~"4.*"}[5m])) / sum(rate(istio_requests_total{reporter="destination"}[5m])) * 100 > $__thresholdHigh percentage of HTTP 5xx responses in Istio (above 5%)
sum(rate(istio_requests_total{reporter="destination", response_code=~"5.*"}[5m])) / sum(rate(istio_requests_total{reporter="destination"}[5m])) * 100 > $__thresholdIstio control plane high memory usage
Alert triggers when Istio control plane memory usage exceeds 1GB.
process_resident_memory_bytes{job=~"istiod|pilot"} / 1024 / 1024 > $__thresholdHigh 5xx error rate between Istio services
High 5xx error rate ({{ $value }}%) from {{ $labels.source_service_name }} to {{ $labels.destination_service_name }} in namespace {{ $labels.destination_service_namespace }}.
Alert triggers when 5xx error rate exceeds 5% between any two services.
(sum(rate({otel_metric_name="istio_requests_total", response_code=~"5.*"}[5m])) by (source_service_name, destination_service_name, destination_service_namespace) / sum(rate({otel_metric_name="istio_requests_total"}[5m])) by (source_service_name, destination_service_name, destination_service_namespace)) * 100 > $__thresholdHigh P99 latency between Istio services
High P99 latency ({{ $value }}ms) from {{ $labels.source_service_name }} to {{ $labels.destination_service_name }} in namespace {{ $labels.destination_service_namespace }}.
Alert triggers when P99 latency exceeds 1 second between services.
histogram_quantile(0.99, sum(rate({otel_metric_name="istio_request_duration_milliseconds_bucket"}[5m])) by (source_service_name, destination_service_name, destination_service_namespace, le)) > $__thresholdGlobal request rate in the service mesh is unusually high
sum(rate(istio_requests_total{reporter="destination"}[5m])) > $__thresholdIstio ingress gateway experiencing high error rates
Alert triggers when ingress gateway error rate exceeds 10%.
(sum(rate({otel_metric_name="istio_requests_total", source_app="istio-ingress", response_code=~"4..|5.."}[5m])) / sum(rate({otel_metric_name="istio_requests_total", source_app="istio-ingress"}[5m]))) > 0.1Gateway pods have dropped — inbound traffic will likely be affected
min({otel_metric_name="k8s.deployment.available", k8s_deployment_name="istio-ingressgateway", k8s_namespace_name="istio-system"}) without (instance, pod) < $__thresholdIstio 1% slowest requests are longer than 1000ms
histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket[1m])) by (destination_canonical_service, destination_workload_namespace, source_canonical_service, source_workload_namespace, le)) > $__thresholdGlobal request rate in the service mesh is unusually low
sum(rate(istio_requests_total{reporter="destination"}[5m])) < $__thresholdServices not using mTLS in Istio service mesh
Alert triggers when services communicate without mutual TLS.
sum(rate({otel_metric_name="istio_requests_total", connection_security_policy!="mutual_tls"}[5m])) by (source_app, destination_service_name) > $__thresholdIstio Pilot failing to configure proxies
Alert triggers when Pilot XDS configuration push error rate is high.
rate({otel_metric_name="pilot_xds_push_errors"}[5m]) > 0.1Istio pilot duplicate entry error
sum(rate(pilot_duplicate_envoy_clusters{}[5m])) > 0Envoy proxy connection failures
rate({otel_metric_name="envoy_cluster_upstream_cx_connect_fail"}[5m]) > $__thresholdIstio proxy sidecar experiencing connection terminations
Alert triggers when Istio proxies frequently lose connection to control plane.
rate({otel_metric_name="istio_agent_istiod_connection_terminations"}[5m]) > 0.1Istio sidecar injection failures
rate({otel_metric_name="sidecar_injection_failure_total"}[5m]) > 0.1Unusual traffic volume detected in Istio service mesh
Alert triggers when traffic volume changes significantly compared to previous hour.
abs(sum(rate({otel_metric_name="istio_requests_total"}[5m])) by (destination_service_name) - sum(rate({otel_metric_name="istio_requests_total"}[5m] offset 1h)) by (destination_service_name)) > $__thresholdIstio workload certificates expiring soon
Alert triggers when workload certificates expire within 7 days. NOTE: This query needs to be updated with proper certificate expiry metrics when available.
({otel_metric_name="pilot_k8s_cfg_events", type="SecretWatcherEvent"} - time()) / 86400 < $__threshold