Last updated:
Code RED Newsletter #41
KubeCon + CloudNativeCon North America kicks off next week in Salt Lake City, November 9–12, and quite a few of us from Dash0 will be there. We've got nine sessions between us, plus Agentics Day, which I'm co-hosting with Chris Mazur. When we're not on stage, you can find us at booth 134.
We're also welcoming Juliano Costa to our Developer Relations team! If you've been following the OpenTelemetry Developer Experience SIG, you'll probably recognize his work. In a recent talk in Prague with Johanna Öjeling, he shared findings from the SIG's survey of more than 200 OpenTelemetry users. The SIG expected SDK ergonomics to be the biggest challenge, but the biggest pain turned out to be getting the Collector into production. Juliano has two sessions at KubeCon next week, so make sure to say hi.
In focus: Dash0 at KubeCon + CloudNativeCon North America
Two themes run through our talks this year. The first is AI agents becoming production workloads, which raises some familiar platform engineering questions. Who gives agents an identity, controls what they can access, manages their costs, and figures out what happened when something goes wrong?
The second is the less glamorous, but equally important, work of getting OpenTelemetry into production: deploying the Collector, propagating context through service meshes, choosing sampling strategies, and keeping telemetry costs under control.
Here's where to find us.
Agentics Day: MCP + Agents
Monday, Nov 9 - Grand Ballroom ACE - co-hosted by Chris Mazur and Kasper Borg Nissen (me)
A full co-located day on MCP, agent runtimes and running agents on Kubernetes. Chris and I open at 9:00 and close at 17:25, with around 30 sessions filling the day.
OTel and Linkerd: A Match Made in Observability Heaven
Monday, Nov 9, 14:05 - Observability Day - Julia Furst Morgado and Flynn
You've deployed Linkerd and OpenTelemetry, and your traces still stop at the first sidecar. Julia and Flynn explain why mesh-level trace context propagation is harder than it sounds, then follow one trace through Linkerd end to end.
Simplifying OpenTelemetry: How the Packaging SIG Makes Installation (Much) Easier Than Nagios
Monday, Nov 9, 15:55 - Observability Day - Antoine Toulme and Michele Mancioppi
Nagios has apt install, and OpenTelemetry has SDKs, an injector, a Collector and OpAMP to wire together yourself. The new Packaging SIG wants OTel installable fleet-wide in a few minutes, and Antoine and Michele will show where things stand today.
Automated Resource Allocation Driven by Service Level Objectives
Monday, Nov 9, 16:30 - Platform Engineering Day - Pedro Célestin and Julia Furst Morgado
Developers are still guessing at CPU and memory requests, even though we have SLOs and telemetry that could help us make better decisions. Pedro and Julia show how to connect Prometheus SLIs with KEDA, Argo, and Knative, using OpenTelemetry as the foundation. They'll also explore how to distinguish an SLO breach caused by resource constraints from one caused by an application bug.
Demo Theater: All the Ways to Add OpenTelemetry Without Touching Your Code
Tuesday, Nov 10, 12:30 - Solutions Showcase, Demo Theater - Michele Mancioppi
The OpenTelemetry Operator, the system packages, eBPF and Go compile-time instrumentation: Michele goes through every way to get telemetry without changing application code, on Kubernetes and plain Linux, and the trade-offs that come with each approach: an annotation per workload, a rebuild, or kernel privileges.
Agents on the Platform: Four Perspectives on the Conversation We're Already Having in Private
Tuesday, Nov 10, 16:40 - Grand Ballroom GI - Juraci Paixão Kröhling, Joseph Sandoval, Jed Salazar, Alexa Griffith and Kasper Borg Nissen (me)
How do agents identify themselves? What should they be allowed to access? Who controls their costs, and how do we figure out what happened when something goes wrong? We've brought together perspectives from platform engineering, OpenTelemetry, security, and AI infrastructure. I suspect we won't agree on everything, which should make for an interesting discussion.
Verticalizing Platform Intelligence for Regulated Domains
Wednesday, Nov 11, 11:00 - 255 EF - William Rizzo and Kasper Borg Nissen (me)
In fintech, healthcare or telco, "we don't know what the agent did" isn't an acceptable answer. William and I show how platform teams can build governed, observable foundations for agents, using OpenTelemetry to trace reasoning chains, tool calls and handoffs. William and Francesco Beltramini wrote up the thinking behind it on the CNCF blog, and the recording of Your Agent Did What?, the talk Adriana Villela and I gave in Prague, covers the forensics side.
Sample Smarter, Not Harder (Or Not At All!)
Wednesday, Nov 11, 11:50 - Hyatt Regency, Regency Ballroom CD - Reese Lee and Juliano Costa
Telemetry costs start inside the application process, long before anything reaches a backend. Using the OpenTelemetry demo, Reese and Juliano go through head and tail sampling and where probabilistic decisions fit in each, including when not instrumenting might actually be the right choice.
Your Voice on Stage: The Delivery + Storytelling Lab
Thursday, Nov 12, 11:00 - Grand Ballroom J - Abdel Sghiouar and Julia Furst Morgado
Getting your CFP accepted is step one. This hands-on workshop is about the delivery: structuring a technical talk as a story, using your voice, handling stage fright, and getting live feedback on stage.
OpenFeature as a Control Plane for OpenTelemetry
Thursday, Nov 12, 15:30 - Hyatt Regency, Regency Ballroom CD - Simon Schrottner and Juliano Costa
Most telemetry is configured once and left alone. Simon and Juliano use feature flags to switch services between lightweight traces and deep diagnostics at runtime, without redeploying, and use semantic conventions as the control surface.
Choice cuts
With KubeCon coming up, there's plenty of interesting reading and watching to catch up on. Here are a few things that caught my attention recently, from agents in production to some good old-fashioned OpenTelemetry engineering.
Code RED Podcast #50: Can AI Run Production? Inside ResolveAI with Spiros Xanthos
Spiros started Resolve AI after running Splunk Observability, where reliability pressure burned out his SRE team and most of it resigned. They had "unlimited access to our own data," and it didn't help, because the human effort on top of it was too much. He and Mirko get into what it takes for an AI SRE to hold the pager (his answer is to act only on the class of problems where the evals say "we know with certainty"), and what agents mean for observability vendors. An agent queries "a thousand times faster than a human," so observability platforms "have to become platforms for agents." That's the same question we'll be asking on the panel.
How We Gave AI Agents the Keys to Production and Still Sleep at Night
monday.com's Infra team was getting more tickets than people could handle, many of them opened by other teams' agents. Their answer is a shared platform running seven agents for six teams, with the guardrails in the platform: short-lived credentials per run, no generic shell or HTTP tools, evals in CI, kill switches, and one trace per run. The guardrail I like most is a separate risk classifier with fresh context that the agent can't influence, because "the model that wrote a change is the worst judge of its risk." After one quarter, about half of the 1,688 monthly tickets are resolved without human intervention. What the post doesn't tell us is how many stayed resolved. That's the number I'd really like to see.
Just Tune It: Closed Loop Telemetry With MCP, AI Agents, and OpAMP
Nike runs around 100,000 Collectors at peak through its own OpAMP control plane, and Panos Tsilopoulos showed how agents are allowed to touch it. They read through MCP tools and can only write a proposal. There's no apply tool, and CI fails the build if someone adds one. Every proposal has to include the actual queries and data instead of a summary, because, in his words, the models are "major bullshitters." The deny list holds the lines nobody crosses, like never letting an agent redirect telemetry to a different destination.
Semantic conventions for the A2A protocol
Agent-to-agent calls now have a vendor-neutral trace shape: a2a.client and a2a.server spans, plus attributes for tasks, messages and protocol version, with the A2A context mapped to gen_ai.conversation.id. A related change separates an agent handing off control from an agent delegating and getting control back. That's the distinction you need when you're reconstructing afterwards who did what. Both are at development stability, so expect renames before they settle. We covered A2A joining the Agentic AI Foundation in #36; now it has conventions for tracing it.
Automatic instrumentation for .NET 6 and 7
The Dash0 Operator has offered zero-touch auto-instrumentation for .NET for a while, built on the upstream OpenTelemetry .NET Instrumentation, which supports .NET 8 and later. That window is about to shrink. .NET 8 and 9 reach end of life this month, joining 6 and 7, and upstream support will soon cover only .NET 10 and 11. End of life on paper doesn't mean the workloads are gone, and we still see plenty of .NET 6 through 9 running in production. The new Dash0 OpenTelemetry distribution for .NET covers 6 and 7, and you get it by upgrading the operator to v0.159.0 or later. Carlos and Michele built it, so if you catch Michele's Demo Theater session on Tuesday, ask him about it.
If you're in Salt Lake City, come find us at booth 134, or at any of the sessions above. If you're not, the recordings usually land on the CNCF YouTube channel a few weeks later.
Until next time: give your agents a proposal instead of an apply button, and get your trace context through the mesh.
Kasper, out!
Hi, my name is Kasper!
I'm Kasper Borg Nissen, Director of Developer Relations at Dash0. I'm passionate about Observability and bridging the gap toward developers through Platform Engineering. I've previously worked 8 years as a platform engineer, I'm a former co-chair of KubeCon+CloudNativeCon, and I'm genuinely obsessed with all things cloud-native and open standards.
