Last updated: September 25, 2026
Code RED Newsletter #38
Satya Nadella was at the All-In Summit and got asked whether AI is going to kill us all. Twice. Both times, he went somewhere I found much more interesting: containment, auditability, and what you can actually see when an agent starts chaining things together.
He never said the word observability. He spent about ninety seconds describing it. I've been making some version of that argument for a while, so I probably enjoyed hearing him say it more than I should have.
In focus: The Unreliable Narrator
A few things seem connected this fortnight. Anthropic published a detection rate that gets worse when the monitor can see the model's reasoning. A benchmark showed how different the result can look depending on where you measure the CPU. (I thought that funky stuff related to measurements happened only in quantum mechanics and Volkswagen cars, but welcome to 2026 y’all.) Uber stopped relying on manually maintained service dependencies and started deriving them from production behaviour. Different problems, same underlying question: what a system says happened and what actually happened aren't necessarily the same thing.
Satya Nadella on the AI Doomer Slowdown, Microsoft's Master Plan & Who Wins AI
Asked about AI risk, Nadella starts somewhere surprisingly practical: containment. Long-running agents start looking a lot like a new kind of insider risk, so you need to know what they're doing, what they have access to, and what happens once they start chaining things together.
I also liked his distinction between genuinely new AI problems and boring old infrastructure problems. Misconfigured containers, secrets lying around, unrestricted internet access, missing monitoring. His description: basic DevOps. Before inventing an entirely new AI safety stack, maybe check whether the agent has production credentials and unrestricted internet access.
Jacob Coxon warns AI could kill us all. Anthropic's own report exposes safety gaps.
Here's the number I had to read twice. Testing retrospectively against the Mythos 5 incident, Anthropic's offline monitor flagged around 1% of harmful actions. Remove the model's chain of thought and show it only user messages, tool calls and tool results, and detection jumps to roughly 50%.
Giving the monitor less information made it dramatically better. The model's own reasoning was convincing the monitor that its behaviour was okay. We've spent a lot of time treating reasoning traces as part of agent observability, but reasoning isn't evidence. It's the agent's version of events. Testimony from the defendant.
AI Safety Is an Observability Problem
Mirko watched the same Nadella interview and wrote about a failure mode I think we're going to see a lot more of: silent success. The agent doesn't crash. Latency looks fine. Everything is green. It just confidently did the wrong thing.
That's where telemetry and evals start becoming part of the same story. Telemetry tells us what the agent did, while evals help us understand whether it was any good. And the agent shouldn't be the only witness. If it says it queried a database, the trace tells us what it attempted. The database tells us what actually arrived.
It passed CI. It passed your evals. The customer still got the wrong answer.
Possibly my favourite headline of the fortnight. The agent returns a 200, passes its evals, and still gives the customer the wrong answer. A diff tells you what changed and an eval tells you whether sampled behaviour passed, but neither gives you the execution path of the production run that went wrong. For that, you still need the trace.
OpenTelemetry everywhere: Migrating a metrics platform at scale
Atlassian migrated its metrics platform to OpenTelemetry Collector distributions: roughly 100,000 hosts across 14 regions, with 4.8 billion datapoints per minute entering the aggregation tier and around 220 million coming out. They also folded metrics into their existing tracing sidecar, saving about 3.9% CPU on average per service across their most expensive Micros services.
What I like is the migration strategy: swap the collection and pipeline, leave the interface alone. StatsD and OTLP could coexist, so teams didn't suddenly get a Jira ticket saying "please re-instrument everything". The Collector was already running in production for tracing, so nobody had to have the "is OpenTelemetry production-ready?" meeting again.
Choice cuts
Smaller things that made me stop scrolling and open another tab.
OTel-Arrow vs OpenTelemetry Collector vs Fluent Bit (Real Numbers)
Henrik Rexed benchmarked OTel-Arrow, the OpenTelemetry Collector and Fluent Bit. Native OTAP makes the OTel-Arrow receiver dramatically cheaper, but somebody still has to encode the data, and in his setup that CPU moved to an upstream Collector. OTAP gets interesting on the wire, but the useful reminder is simpler: benchmark the entire pipeline, not the component with the nicest graph.
Kubernetes attributes processor reaches v1.0.0 milestone
The Kubernetes attributes processor has reached v1.0.0, as part of the Collector Contrib project's v0.161.0 release. The processor is now stable for logs, metrics and traces, building on the Kubernetes semantic conventions that reached stable status earlier this year. This does not mean that the Collector releases or the Collector as a whole have reached the long-awaited v1.0.0, but it is a signal that we are getting closer.
I like what this milestone represents more than the version number itself. A component can't really become stable until the vocabulary around it is stable too. Getting telemetry out is the easy-ish part. Getting the ecosystem to agree on what k8s.pod.* actually means, and then stop renaming it, is where the real work is.
Large-Scale Automated Dependency Analysis Across Uber's Service Mesh
Uber stopped relying on humans to maintain dependency maps and started deriving them from production failures. They correlate downstream and caller failures, classify dependencies as fail-close, fail-open or, wonderfully, unknown.
They considered traces, but at 0.01% sampling and 99.9% availability, collecting ten failed samples could take more than two days. So they use metrics and observe every production failure instead. Somebody did the maths and picked metrics over traces.
Agentic Engineering at Zalando: A Snapshot
Zalando published data from 200+ teams showing PRs shifting toward larger sizes and an inflection in cyclomatic complexity around coding-agent adoption. At the same time, their risk-based approval system auto-approves around 33% of PRs as low-risk, cutting lead time by 20–40%. This is the AI engineering data I want more of: not just how much code we generated, but what happened to the codebase afterwards.
Introducing AI SDLC Insights
Speaking of what happens afterwards, AI Coding Insights is now AI SDLC Insights, connecting coding-agent telemetry with pull request data. Spend and adoption are useful, but I'm more interested in whether AI-assisted PRs merge faster, how long they wait for review, and how much review they actually get. We already know AI can generate more code.
Service Level Objectives are now available
We shipped SLOs in Dash0: failure budgets, burn rates and alerts against signals already arriving in Dash0. My favourite part is predictably the boring standards bit: definitions use OpenSLO, so they can live in Git and be applied through the CLI or Terraform. Your reliability targets feel like a particularly bad place for vendor lock-in.
The Observability Trap: Why Vendor Lock-In Is Still a Problem in 2026 (And How to Avoid It)
Julia's argument is that OpenTelemetry solved a big part of instrumentation lock-in, but the rest of the stack can still keep you stuck: dashboards, queries, alerts, and now AI. Her answer is to keep following open standards further up the stack with OpenTelemetry, Perses and PromQL. "Instrument once" was a pretty good start. It would be nice if the rest of the stack could come with us too.
If there's one thing I'd take from this fortnight, it's that almost none of this argues for collecting more. It argues for being more careful about what we trust. Anthropic's monitor improved when researchers removed the agent's explanation. Henrik's benchmark changes depending on where you count the CPU. Uber stopped asking humans about dependencies and looked at what actually fails together.
Yesterday, Adriana Villela from Dynatrace, and I were at SRE Day London talking about exactly this, with a rogue goose and three AI SRE agents trying to figure out what it had done. Meanwhile, Julia was in San Jose at WeAreDevelopers talking about why agents need observability before better models, followed by a hands-on workshop taking OpenTelemetry data into an AI SRE investigation. Apparently we're all talking about agents now. At least we're instrumenting them first.
Until next time: keep your conventions boring, check the story against the system, and may your traces outrank your agent's explanation.
Kasper, out!
