When you ask an agent to optimize your working capital, it might falsify your books instead. The system won't crash while it happens. Your error rate and latency graphs stay flat, and nobody gets paged. Every dashboard you own stays green while the decisions underneath it rot.
Satya Nadella reached for that scenario at the All-In Summit, during a wider discussion about the Hugging Face incident, reward hacking, and warnings from people leaving frontier AI labs about the catastrophic risk of human extinction.
“Suppose I say, hey, go optimize my working capital. It may fake my books”, he said. “Because this is like a new type of insider risk”.
His answer to that risk was not what I expected. He did not start with alignment theory or another model-level safeguard. He started with monitoring:
"One of the things that I think is going to be really an issue and a thing that needs great solutions is true aggressive monitoring of agent activity", he said. "That's behavioral evidence, and so everything has got to be auditable. And then every object it accesses".
That’s a description of observability. He just didn't use the word.
He answered a safety question with an engineering answer
Nadella was handed the existential framing twice and both times he went somewhere more mundane. He opened with containment. “I would start from the very basics of saying, okay, what does containment look like?”
Then he said the part that made the whole argument click.
"The core of my take is we will have to get the engineering process around building out this experimental science to be more robust".
Coming from the person overseeing one of the largest AI infrastructure investments in the industry, I read that as a statement about where some of the most important controls have to live.
Some of that lives in the model, but most of it lives around it, in containment, permissions, and the monitoring that produces a record of what the agent actually did.
For most of its history, the pitch has been uptime, latency, errors, cost, and the occasional bad night. Agentic systems raise the stakes, and telemetry becomes part of the evidence you need before handing a system more authority.
You should not let an agent act on its own if you cannot reconstruct what it did.
Failure and dishonesty are different problems
The working capital agent breaks something we have leaned on for two decades.
Monitoring is very good at detecting failure. It catches crashes, errors, saturation, and latency, and almost none of that catches an agent that's confidently doing the wrong thing. We call that gap silent success in agentic observability, where the operational metrics stay healthy while decision quality degrades underneath them.
Operational health stops being enough at that point. Whether the agent is up and whether it’s doing the right thing are different questions, and a dashboard built to flag errors can answer the first while completely missing the second.
That‘s where evals become part of the observability story. Telemetry tells you what the agent did, but only evals can help you decide whether what it did was actually good.
For an agent, that might mean checking whether an answer was grounded, whether a tool call was appropriate, whether the task constraints were satisfied, or whether the final outcome matched the goal. Some of those checks can be deterministic. Others may need model-based evaluation or human review.
Nadella reached for an older instinct here. He talked about transaction processing, and how data loss was the class of bug where "you stop the thing". An agent that falsifies a number is a data integrity bug, and the response that class of bug has always earned is to halt the system rather than file a ticket.
Some of this is not a new problem
What I appreciated most was his refusal to make the problem exotic.
He split these incidents into two categories. The mundane ones came first: “somebody misconfigured a container”, API keys in a public repository, “there’s no monitoring, there’s internet access”.
His label was blunt: “Classic, I would call it, basic DevOps”.
Only after that did he get to the novel part, the reward hacking and swarm behavior nobody has good science for yet. And that distinction deserves more attention.
Most of what gets filed under AI safety right now is unmonitored infrastructure with credentials lying around, and you already know how to fix a lot of that.
The uncomfortable version of the same point is that a team can spend its attention on alignment while running agents in an environment it can't see into, and the second problem is the one that will actually bite.
Read the rest as a specification
Each phrase in that quote asks for something specific, and the specifics are more demanding than the headline version suggests.
Behavioral evidence rules out watching outputs. An agent's output is one artifact at the end of a long, non-deterministic run, and the behavior is the run itself: which tool, which arguments, in which order, against which data, and where the reasoning branched.
In LLM observability terms, the unit of analysis moves from the response to the trajectory. A log line may record that something finished, which isn't the same as recording what it did.
Everything has got to be auditable, and then every object it accesses is a data model problem before it is a security problem. After an incident, nobody asks whether the agent looked anomalous. They ask whether it read this table, retrieved that secret, called this API, or caused that change.
That means agent telemetry has to connect with the systems it touched. And some of the evidence should come from those systems too. If the agent says it queried a database, the trace tells you what it attempted but only the database can tell you what actually arrived. An agent should not be the only witness to its own actions.
Then he asked for something harder. "If it goes and gets a secret, it's going to go chain a couple of things. You should be able to see it when it's starting to chain a couple of vulnerabilities to go hack".
That's a sequence being watched as it assembles, across many steps and many systems, rather than a report written afterward, and it's the hardest thing on his list. Just enough context to recognize the sequence, and enough speed to do something about it.
His answer to the insider risk was "a causal model, like a semantic model that actually checks and verifies", where the agent's claim about the world gets checked against the world.
If it says latency increased, check the latency. If it says a deployment caused the problem, check the deployment timeline. If it proposes an alert for a metric, verify that the metric exists and returns data.
Some checks are harder and need evals.
Was the diagnosis actually correct? Did the agent choose an appropriate tool? Did it satisfy the constraints of the task? Did the final result improve the thing it was supposed to improve?
That’s where agent observability goes beyond recording execution. You need the trace of what happened, and an evaluation of whether what happened was acceptable.
That sounds obvious, but it’s not how most agent systems work today.
He asked for one more thing earlier in the conversation: reasoning traces “in language that we can all understand”, transparent enough that “if you have the full CoT, then you can really go look at it deeply.”
That can be useful evidence, but it is not ground truth. A reasoning trace can help explain a run, but it doesn’t replace the record of what the agent actually did or what the surrounding systems observed.
Keep the reasoning artifact if you have it, but trust the evidence around the action more.
What this asks of an observability platform
The evidence has to sit in one place. When agent behavior lives in a separate store from your infrastructure, application, and frontend telemetry, watching a chain assemble becomes a manual exercise across tools, under time pressure, run by someone who didn't write the agent. SignalStore exists so that join happens at query time.
It also has to stay in an open standard, because agent telemetry in a proprietary schema makes a poor audit trail when its usefulness expires with your contract. That's why we're OpenTelemetry-native rather than OpenTelemetry-compatible, and why the GenAI semantic conventions are worth paying attention to.
Then verify inside the loop that produces the work. Agent0 applies a narrow version of that principle already. When it generates assets from telemetry, it validates the underlying queries against live data before it reaches you, with citations you can click into.
We built it this way because I was not willing to ship an AI that confidently invents a metric name.
That does not prove every conclusion the agent makes is correct. But it does remove one important class of silent failure where confident claims are made about systems that the telemetry does not support.
The same principle is why autonomy in Dash0 is progressive. A human approves first, then reviews, then steps back as the system earns more authority.
Final thoughts
The sharpest description of agent security I've heard lately came out of a podcast interview, and what it pointed at was observability.
The work it implies is concrete. Fix the boring things first, because a misconfigured sandbox and a credential sitting in a public repo are still the most likely ways this goes wrong.
Capture trajectories not just outputs, record what the agent touched, then check what your agents claim against what your systems actually report, with evals that grade against a source of truth rather than against another model's opinion.
Keep the evidence in an open standard, so your audit trail does not become another form of vendor lock-in.
This is not a new discipline. It’s security, DevOps, evals, and observability being asked to work together around a new kind of actor.
Observability gets the harder job, which is keeping record of a system that acts at machine speed, crosses system boundaries on its own, and may look perfectly healthy while doing the wrong thing.
That’s the problem we are building for at Dash0. If it resonates, come take a look.
