Last updated: May 29, 2026
Code RED Newsletter #32
I've spent the past week on the road - Cloud Native Days Romania in Bucharest at the start of the week, then KCD Czech & Slovak in Prague on Thursday before flying home. Two great community events, both with good coffee and the kind of hallway energy you don't get over Slack. Mauricio Salatino was at Cloud Native Days Italy in Bologna the same week and reported back the exact same story - which is the tell. The graduation news from a couple of weeks ago came up everywhere - matter-of-factly, in the way the bureaucratically big things tend to once the moment has passed. What came up more, in hallway after hallway, was agents. Where they're being run. What's breaking. What you actually see in the trace when an LLM call routes through three tools, a workflow engine, and a sidecar mesh - and the only thing you have to debug it with is a YAML file from 2024.
That's the texture I want this issue to capture. Issue 31 was about graduation. This one is about what graduation actually unlocks - and the answer, increasingly, is the work of observing what's happening on top of the protocol. The agent stack is real, it's in production, and the OpenTelemetry project is quietly building the surface area it needs - GenAI semantic conventions, normalizer processors, policy plumbing - while the rest of the world is still digesting the press release.
In focus: Observing the Agent Stack
A year ago, "agent observability" was mostly a slide with a lot of arrows on it. Today, there's actually a stack: OTel GenAI semantic conventions defining the target shape, a collector processor quietly translating vendor-specific formats into it, runtime policy projects wiring into the same signals, and grown-ups in CNCF blog posts talking about cold-start optimization for inference workloads like it's a normal day at the office. Here are five pieces that, together, give you a picture of where the agent observability surface actually is right now - and what's still under construction.
Inside the LLM Call: GenAI Observability with OpenTelemetry
The OTel project's walkthrough of the GenAI semantic conventions is the foundational read of the bunch, and it should be on every "we support agent observability" product page's required reading list. Model calls, token counts, optional prompt and completion content, all in stock OTel signals. The piece does the unglamorous work of showing what's actually on the span - which means it gives you a sharp benchmark for evaluating any vendor's AI-observability pitch. If the span shape doesn't roughly match what's in this post, you're getting bespoke telemetry dressed up in a semconv-shaped trench coat. Worth a read even if you don't run an LLM in production yet, because every project around you is about to.
The Collector Quietly Consolidates the GenAI Formats
Structurally important, easy to miss unless you read collector-contrib PRs for fun (hi, I see you). In the collector-contrib repo, the gen_ai_normalizer processor added OpenLLMetry as a source format - on top of the OpenInference support already merged earlier in the year. It renames vendor-specific attributes onto GenAI semconv and applies value-level normalizations (operation-name enum folding, finish-reason wrapping, structured message JSON encoding - the boring stuff that decides whether your dashboards work). The story across the year: the OTel project is folding the bouquet of vendor LLM telemetry formats into a single semconv-shaped output, so you don't have to care which library got there first. The GenAI semconv defines the target shape; the normalizer is the on-ramp. Treat it as the Babel fish for agent telemetry.
Prempti: Policy and Visibility for AI Coding Agents
Leonardo Grasso, Falco maintainer, introduced Prempti this month - runtime policy plus audit visibility for AI coding agents in developer environments. The framing matters: agents in the loop need observability and policy, not one or the other. Semconv tells you what the agent did. Prempti decides what the agent is allowed to do before it does it - and logs the decision, so the next on-call engineer doesn't have to play forensic detective at 3 a.m. An early signal that the agents-in-production beat is graduating from "look at this trace" to "what do we let it touch, and what gets recorded when it touches it." The security folks have arrived.
30-Second LLM Cold Starts on Kubernetes - NetEase Games
The infrastructure side of the same story. NetEase Games published the work they did to bring LLM cold-start times on Kubernetes down to 30 seconds, with elastic compute behind it. The number is satisfying; the picture is more interesting. Real production LLM workloads. Cold-start latency that matters because there's a human user waiting (and the user's attention span is roughly equal to one TikTok). A platform team owning the result. The agent stack does not live on the application layer alone - it lives on the cluster, the scheduler, the GPU-pool warming logic, and the cold-path optimizations that make the inference call viable in the first place. Pairs well with the GenAI semconv post above: this is what those spans look like when they actually have to behave under load.
Code RED Podcast: Killing Observability Noise with Jad Naous
Jad Naous, founder and CEO of Grepr, joined Mirko for a conversation that starts on log volume and ends somewhere interesting. The core thread is noise reduction: Grepr monitors a million log patterns in real time, learns what's noisy right now (because what's noisy today changes the moment a developer pushes a new build), and aggregates dynamically on a two-minute cycle of samples plus summaries. The frame that lands hardest: most observability data is insurance you'll never look at, and an engineer paging through 90% padding to find the 10% signal is the actual cost. The episode also drifts into the AI SRE wrapper market - and Jad and Mirko land in the same place: the vendors who own the data, not the LLM, are the ones who'll still be around in 24 months. Worth the full listen if you're thinking about where agent telemetry meets the data bill.
Choice cuts
Pour yourself a cold one - Bucharest weather has, against all odds, made it as far as northern Europe. A handful more pieces from the past two weeks, including the post-graduation items that didn't quite fit the agents theme but are still worth your tab queue.
OTel Officially Graduated, and the Maturity Model Conversation
The CNCF's official graduation announcement landed on May 21st with the pull-quote you'd expect from a press release: "de facto observability standard." Issue 31 covered the project moment; this is the version your procurement team has bookmarked.
"OTLP Support Is Not OpenTelemetry Support"
Juraci, long-time OTel maintainer, draws the line: OTLP at a port is the floor, not the ceiling. Real OTel support means preserving the data model, honoring semantic conventions, respecting the shared resource model, and managing version drift across signal types. One of the cleanest 800-word framings I've read of why "we accept OTLP" has become the new "we're cloud-native."
The Ingress Controllers Series, Concluded
The closing post of the multi-month series evaluating Contour, Emissary, Istio Gateway, kgateway, and Traefik against a consistent set of observability dimensions. All five projects "support OpenTelemetry" by the README; the lived telemetry experience varies by more than an order of magnitude. Tracing is strong across the board (Envoy doing the heavy lifting, as ever); resource identity is consistently the weakest dimension. If you've ever debugged a trace that starts mid-story because the edge wasn't instrumented properly, this one will read uncomfortably like your own incident notes.
OTTL in Action: The Platform-Team Escape Hatch
Henrik Rexed just published a 25-minute walkthrough of OTTL - PII redaction, HTTP method normalization, cardinality control, JSON log body parsing, slow-request tagging, all done at the collector with zero application changes. When upstream OTel support is sloppy or inconsistent, OTTL is the lever platform teams pull. Think of it as the duct tape that's actually load-bearing.
Database Queries, Front and Center
Small but satisfying changelog item from Dash0 (yes, this one's ours): the new Database Query Widget puts database queries at the front of the span sidebar with syntax highlighting, and interpolates prepared-statement parameters directly into the query. No more squinting at ? placeholders at 2 a.m. and reconstructing what actually ran in your head while production keeps timing out.
The thread running through this issue: graduation closes the chapter on whether OpenTelemetry will win. It opens a different chapter, where the question is what we're observing with it - and the answer, increasingly, is the agent stack on top. Semantic conventions are landing. The collector is doing the integration work. Policy projects are wiring into the same signals. The agent observability surface is being built right now, in a series of small, deeply unglamorous PRs, while everyone else is still digesting the press release.
Huge thanks to everyone who said hi in Bucharest and Prague over the past week. The hallway conversations were the texture this issue is built on - real teams, real telemetry, real opinions about what comes next, and at least one excellent argument about whether MCP servers count as middleware. (Reader: they do.)
Until next time: may your agent spans carry their semconv, your collectors normalize what your libraries forget to, your cold starts stay under thirty seconds, and may you never have to explain "supports OpenTelemetry" to a procurement team.
Kasper, out!
Hi, my name is Kasper!
I'm Kasper Borg Nissen, Director of Product Marketing & Developer Relations at Dash0. I'm passionate about Observability and bridging the gap toward developers through Platform Engineering. I've previously worked 8 years as a platform engineer, I'm a former co-chair of KubeCon+CloudNativeCon, and I'm genuinely obsessed with all things cloud-native and open standards.
