Dash0 Raises $110M Series B at $1B Valuation

  • 10 min read

What Is eBPF Observability and How Does It Work?

eBPF observability means collecting telemetry by running small sandboxed programs inside the Linux kernel instead of instrumenting your application code or parsing logs from a user-space agent. Because those programs sit at the kernel boundary, they can observe syscalls, network packets, and function entries across your workloads without you touching a single line of the workloads themselves.

This works because the kernel already brokers everything your application does. Every HTTP request, DNS lookup, and disk read passes through it, so hooking the kernel gives you one vantage point over all of it at once. This article covers how eBPF actually captures that data, what it can and can't see, and where it fits alongside OpenTelemetry instrumentation.

How eBPF captures telemetry

An eBPF program is bytecode that you load into a running kernel and attach to an event. When that event fires, the program runs, collects whatever you asked for, and writes it to a shared data structure (an eBPF map) that a user-space agent reads. Nothing gets recompiled, no module gets installed, and the kernel keeps running the whole time.

The events you can attach to fall into a few categories:

  • kprobes and kretprobes hook the entry and exit of kernel functions.
  • uprobes and uretprobes hook functions in user-space binaries and libraries.
  • Tracepoints are stable, kernel-maintained hooks for common events like syscall entry.
  • Network hooks (XDP, tc, socket filters) run against packets as they traverse the stack.

Before any of this runs, the program passes through the eBPF verifier, which statically checks that it can't loop forever, read out-of-bounds memory, or otherwise destabilize the kernel. If the verifier rejects your program, it's protecting the machine, not being pedantic. This is the core reason eBPF is safe to run in production, where a buggy kernel module would panic the host instead.

A minimal example makes the flow concrete. This eBPF program hooks the getaddrinfo call to time DNS resolution, written in the C dialect the kernel expects:

c
1234567891011
#include <uapi/linux/ptrace.h>
BPF_HASH(start_times, u32, u64);
// Fires when getaddrinfo() is entered
int trace_getaddrinfo_entry(struct pt_regs *ctx) {
u32 pid = bpf_get_current_pid_tgid() >> 32;
u64 ts = bpf_ktime_get_ns();
start_times.update(&pid, &ts);
return 0;
}

A companion program on the function's return reads the stored timestamp back out of start_times, computes the delta, and pushes it to user space. That delta is a DNS resolution measurement your application never had to expose, captured before the HTTP client even opened a connection.

What eBPF sees that application instrumentation misses

Application-level instrumentation lives inside your process, so it sees your business logic but stops at the process boundary. eBPF sits below that boundary and captures the parts of a request's life that your code never observes: time spent in the kernel's TCP stack, DNS resolution delays, disk I/O scheduling, TCP retransmits, and context-switch overhead.

For networking specifically, eBPF can observe L3, L4, and L7 flows and report latency, packet loss, throughput, and connection establishment times on a per-process or per-container basis. That lets you answer whether a slow request is the application's fault or the network's, which is otherwise one of the harder things to disambiguate in a distributed system.

The other big advantage is coverage without effort. Because the probe attaches to the kernel rather than to a specific runtime, one agent instruments every service on a node regardless of language. In Kubernetes this usually means running the agent as a DaemonSet, so every node is covered and every pod on it gets instrumented automatically.

Where eBPF fits with OpenTelemetry

The most practical way to use eBPF observability today is through OpenTelemetry eBPF Instrumentation (OBI), which was donated to the CNCF and inspects application executables and the networking stack to produce trace spans and RED (Rate, Errors, Duration) metrics for HTTP/S and gRPC services. It emits standard OTLP, so the data flows into any OpenTelemetry backend without a proprietary format in the middle.

OBI is the right starting point when you have no telemetry, partial telemetry, or a service using a library nobody has instrumented. It detects when a service is already instrumented with an OpenTelemetry SDK and disables its own signals for that service to avoid duplication, so you can drop it into a mixed environment safely. If you want to see this in practice on a specific runtime, the Dash0 walkthrough on OpenTelemetry Go instrumentation via eBPF sets it up end to end on Kubernetes.

What eBPF instrumentation does not replace is language-level SDK instrumentation. Because the capture happens at the kernel and protocol level, it can't see application-specific attributes, custom spans, business events, or the intent behind a request. It reports that a request returned a 500; it can't tell you which feature flag caused it. SDKs also provide runtime-specific signals like JVM heap usage and garbage-collection activity, and they enable trace-to-log correlation, which eBPF alone doesn't.

The consensus among the projects building these tools is a hybrid model: eBPF for broad, consistent baseline coverage across every service, and SDKs for deep transactional context where you need it. The two are designed to coexist rather than compete.

Common pitfalls

uprobes can be far more expensive than they look. Kernel-probe-based capture is lightweight, often under 1% CPU, but user-space probes require an extra context switch on every event. Under intensive or I/O-heavy workloads, that overhead has been measured at up to 200% relative to kernel-probe implementations, which can make uprobe-heavy setups unsuitable for production hot paths. Attach to specific tracepoints relevant to your goal rather than probing every syscall entry.

Distributed tracing coverage is uneven by language. OBI's distributed tracing works well for Go (HTTP and gRPC), Node.js, Python, NGINX, and PHP, but support varies for languages with complex threading models. It currently doesn't handle reactive frameworks, Java virtual threads, or complex thread pools, which show up as broken trace context: spans don't point to their real parents, and a single request fragments into several traces. If distributed tracing for those runtimes is a hard requirement, plan for SDK instrumentation there.

Kubernetes metadata enrichment can overload the API server. By default each eBPF agent pod opens its own watch connections to the Kubernetes API server to read pod, node, and service metadata for enrichment. On large clusters, or when many agent replicas run side by side, that fan-out puts real load on the API server. The fix is a shared metadata cache that the agents point at instead of each maintaining their own informer.

eBPF is Linux-only and sensitive to kernel version. Features have evolved quickly, so specific programs may need a reasonably recent kernel. If you run mixed or older kernels, verify your target hooks are available before assuming full coverage.

Final thoughts

eBPF gives you kernel-level breadth with almost no instrumentation effort, but the data is only as useful as the backend that correlates it with your logs, metrics, and traces. The real value shows up when a kernel-level DNS delay or TCP retransmit appears next to the application trace it slowed down, so you can see the whole path of a request in one place.

Dash0 is OpenTelemetry-native and ingests OTLP directly, so telemetry from OBI and other eBPF-based instrumentation lands alongside your SDK traces, metrics, and logs without translation. Dash0 also supports the OpenTelemetry eBPF Profiler for continuous profiling, and its Kubernetes monitoring ties the kernel-level signals back to the pods and nodes that produced them. Start a free trial to see your eBPF and SDK telemetry correlated in one view. No credit card required.