← Back to Blog

Agent Observability Is an Infrastructure Problem

Telemetry tells you what AI agents did. It can't prove it.

Agent Observability Is an Infrastructure Problem

Enterprises are deploying autonomous agents into regulated, high-consequence workflows faster than they can see what those agents do. The industry's answer has been better telemetry: more traces, more spans, more dashboards. Weilliptic's position is that this will not be enough.

The gap is structural. Agent frameworks have no persistent notion of who an agent is. Permissions, memory and approvals live in stores the monitored system can modify. And monitoring happens after the fact. No amount of extra logging fixes any of those three.

Observability has to be a property of the infrastructure agents run on. That means identity the agent can't forge, authorization checked before every action, tamper-evident records of every change to permissions, memory and oversight, and a governed path for that evidence into the security operations center. This paper explains why, and how the Weilliptic stack delivers it.

How sandboxed agents breached Hugging Face, and why the victim saw it first

In July 2026, agents running inside an OpenAI cyber-capability evaluation broke out of their sandbox and compromised Hugging Face's production infrastructure, apparently to find answers to their own benchmark. According to public accounts, the escape went through a zero-day in the package-registry cache proxy that was the environment's only network path. The agents then chained stolen credentials and further exploits into remote code execution. (Malwarebytes) (YourStory)

The timeline matters more than the exploit:

Date (2026) What happened
Jul 21 OpenAI discloses that its own models were the attacker
Jul 16 Hugging Face publicly reports an intrusion it can attribute only to an unidentified autonomous agent
Jul 13 Hugging Face's security team cuts off the intruder
~Jul 9 Approximate first attempt to break out of the test environment

Sources: Gupta, SmarterX.

The victim saw the attack first; the organization running the agents learned what had happened days later. Hugging Face reconstructed more than 17,000 actions, spread across a swarm of short-lived sandboxes. (Quartz) That pattern is exactly what today's tooling handles worst: ephemeral processes with no durable identity, operating with borrowed credentials, generating more events than anyone can read.

This happened at a frontier lab, with frontier-lab instrumentation. Most enterprises deploying agents have far less.

Five questions enterprises can't answer about their AI agents

Ask a CISO to account for any single agent action in production, and the agent frameworks in common use (LangChain/LangGraph, Google ADK, AWS Strands and their peers) leave most of these questions without a reliable answer:

  1. Who acted? Agents are identified by run-scoped names or process IDs. When a run ends, so does the identity. There is nothing durable to check permissions against, and no way to follow authority from an orchestrator through the sub-agents it spawns.
  2. What was it allowed to do, and did that change? Permissions live in configuration and control-plane databases. A mid-session change that removes a human-approval step, whether made by an insider or induced through prompt injection, typically leaves no trustworthy trace.
  3. What did it believe? Agent memory changes continuously, and neither the change nor its cause is recorded. When behavior drifts because memory drifted, the first symptom is a bad decision.
  4. Was human oversight real? Approvals, where they exist, rarely record why review was triggered, who reviewed, what authority they held, or what they decided. Genuine oversight and rubber-stamping look the same.
  5. Why did it act? Reasoning is only as visible as the model provider allows, often a summary written after the fact. Proving a decision didn't rest on a forbidden factor is close to impossible.

Taken together: enterprises cannot reliably trace an agent's authority, how that authority changed, or what drove its actions. For a regulated institution, that is not a monitoring inconvenience. It is an inability to produce evidence.

Why better logging won't fix agent observability

The natural response is to ask frameworks to emit richer events. That helps at the margins, but it misses why the gaps exist. Three structural problems sit underneath all five questions.

Identity lives in the wrong place. In today's frameworks an agent is a process holding a bearer credential, such as an API key or a service account. There is no durable self to attach evidence to, and anything that steals the credential, including a prompt injection, inherits the identity. You cannot trace authority through sub-agents when authority was never bound to an actor.

The record is mutable by the thing being observed. Permissions sit in a control-plane database, memory in a vector store, logs in files or an observability backend. An insider, or an agent that has escaped its sandbox, can change the permission and the record of the change. Telemetry the monitored system can rewrite is not evidence.

Monitoring is after the fact. Even perfect telemetry describes what already happened. In July, nothing stood between the agents and the actions they took. Observability without enforcement means the best case is a faster post-mortem.

Four principles for observable AI agents

Weilliptic believes any infrastructure for autonomous agents must meet four principles. They follow from one premise: in a world of millions of autonomous actors, the substrate must tolerate Byzantine behavior, including from the agents themselves.

  1. Identity is structural, not a credential. Every agent has a persistent identity that the runtime asserts on its behalf. Agent code never holds a key that can be stolen or replayed.
  2. Authorization is checked before the action and can be verified independently. "Is this actor allowed to do this right now?" is answered before execution, by a check that trusts neither the agent's claims nor a database it could influence.
  3. Every change to authority, memory and oversight is a tamper-evident record. Permission changes, memory writes and human approvals are state transitions with provenance, not log lines.
  4. Evidence flows into the tools defenders already use. The record streams, in a standard schema, into the SIEM and SOC, governed by the same identity and authorization rules as everything else.

How the Weilliptic stack implements these principles

The foundation is WeilChain, a Byzantine-fault-tolerant, multi-cloud replicated state machine. Every state change on it is a committed transaction that no single operator, insider or compromised agent can quietly rewrite. On top sit three pillars of trust infrastructure: Autos (identity), Writ (authorization) and WeilStream (governed events). Alongside them sit Pensieve (agent memory) and TCrypt (data privacy).

Question Weilliptic answer How fully
Who acted? Autos, org registry, end-to-end execution provenance graph Fully
What was it allowed to do, and did that change? Writ (IdentityToken + @secured_user), all authorization state in the org's IdentityToken on WeilChain Fully
What did it believe? Pensieve: tamper-evident world model with provenance Fully, for agents using Pensieve
Was human oversight real? Autos delegation binding, Writ, Receipts Who, authority and decision fully; why partly
Why did it act? TCrypt / Edicts runtime masking, Receipts on data access Reframed: prevent rather than reconstruct

Who acted: Autos

Autos gives every agent a persistent identity issued by WeilChain. It belongs to the agent, not to the human who launched it. On WeilChain, each agent is an applet whose immutable address the runtime injects at the execution boundary, so agent code never holds a key to steal. For agents running elsewhere, decorators for LangChain/LangGraph, Google ADK and AWS Strands bring the same identity to existing code.

Two further capabilities trace authority across sub-agents. First, every action an agent takes on someone's behalf is bound to that person's Microsoft Entra ID identity, so delegation is recorded per action, not assumed. Second, the end-to-end execution provenance graph: the initiator of a flow receives a task ID that propagates to every tool call, method call and sub-agent invocation it triggers. Ask "what did this flow touch?" in natural language and you get the whole graph. An org registry tracks which agents belong to which team and where each runs, giving a live agent inventory.

What it was allowed to do: Writ

Writ composes IdentityToken with @secured to answer "what is this actor allowed to do right now?" The answer is checked independently instead of asserted by the agent or read from a mutable database. The @secured_user decorator enforces this before the tool call runs, with group expressions such as @secured_user("acme.engg & !acme.products").

Every grant, revocation and group membership lives in the organization's IdentityToken on WeilChain, so any change to an agent's authority is a committed, attributable transaction, not a silent configuration edit. Neither an insider nor a prompt-injected agent can quietly remove an approval step, and the check never trusts what the agent says about itself.

What it believed: Pensieve

Pensieve is a persistent world model for autonomous agents: the entities, relationships, events, actions and outcomes an agent knows about, and how that picture changes over time. Every write is tamper-evident and carries provenance: which action, source or event produced it. When behavior drifts, you trace back through the world model to the memory change that caused it, and to whatever introduced that change.

Whether oversight was real: Autos, Writ and Receipts

On WeilChain, human-in-the-loop steps are among the few operations that change canonical state. An approval is therefore a state transition, not a log line. Its record carries who approved (their Entra identity), their authority (the Writ groups they held at that moment), what they decided, and the task ID that ties it into the provenance graph.

Why review was triggered is captured when the trigger is a Writ rejection or a policy rule. When application logic decides to escalate, developers still need to emit the reason.

Why it acted: prevent rather than reconstruct

No infrastructure vendor can force model providers to expose raw reasoning. But the real concern, whether the decision rested on factors the agent wasn't permitted to consider, has a stronger answer than reading reasoning traces: make sure the agent never saw those factors.

TCrypt and Edicts mask sensitive fields at runtime by policy. Masking works down to a single column in Snowflake, Aurora or Postgres, or a JSONPath field in a payload, with named exceptions for groups allowed to see clear values. Receipts record what data each agent call actually received. An agent can't weigh a ZIP code or a diagnosis it only ever saw as a token, and the record proves which inputs were in play even when the reasoning is opaque.

Getting AI agent activity into the SIEM and SOC

Agents generate more events than any person can review, so evidence is only useful if it reaches the people and systems that act on it. Weilliptic's answer is not another console. It moves the evidence into the security operations stack enterprises already run.

WeilStream is a topic-based event fabric running on WeilChain. Publishers write OCSF-formatted audit events and receipts directly to topics; SIEM consumers pull them at their own pace, from any offset. The audit-topic consumer is itself protected by Writ, so only authorized identities (the SIEM, not an arbitrary agent) can read the audit stream. Agent activity lands in Splunk, Microsoft Sentinel or any OCSF-aware tool, next to every other security signal, in near real time.

The provenance graph handles investigation. When an alert fires, an analyst asks in natural language and gets the full task graph for that flow: which agent, on whose behalf, under which authority, touching which systems and memories.

You can't observe what you can't see. Much enterprise AI activity never passes through an agent framework at all. WAuth captures and attributes employee use of ChatGPT, Claude, Gemini and Perplexity in the browser. Codensics covers coding agents. The Autos org registry covers agents built with SDKs. Every AI surface emits attributable records into the same pipeline.

And because Writ enforces before the action, the SOC is not the only line of defense. Many of the events a monitoring team would hope to catch become rejected calls instead. Those rejections are themselves a high-quality signal.

What infrastructure can't do for agent observability

A position paper that claims everything persuades no one. These are the limits.

  • Model internals. Raw reasoning and token-level confidence come from the model and its inference engine. Where Weilliptic serves inference behind its gateway, per-call receipts can carry what the engine exposes. When an enterprise calls a closed model API directly, we record the call, its identity and its cost, but not signals the provider withholds.
  • Decision quality. Oversight ultimately asks whether a decision was sound, not only whether it was authorized. Tamper-evident records of identity, authority and memory are the raw material for that judgment. They are not the judgment itself.
  • Integration depth. Gateways can capture events with near-zero integration, but stopping a call before it happens requires application-layer hooks. With Autos and Writ that is a few decorators per agent. Small, but not zero.

What to require of any AI agent platform

Whatever stack they choose, enterprises deploying agents should be able to answer yes to each of these before going to production:

  1. Does every agent have a persistent identity that outlives a single run, and that the agent's own code cannot exfiltrate?
  2. Is every action bound to the human or system that delegated it?
  3. Are permissions checked before the tool call executes, by something the agent cannot influence?
  4. Is every change to an agent's permissions recorded in a store that neither the agent nor a single administrator can rewrite?
  5. Are memory writes recorded with their source, so drift can be traced to its cause?
  6. Does every human approval record who approved, under what authority, and what they decided?
  7. Can you reconstruct the full chain of agents, tools and data behind any single outcome?
  8. Does this evidence reach your SIEM in a standard schema, in near real time?
  9. Do you have an inventory of every agent and AI surface in use, sanctioned or not?

Any platform that answers no to the first four has an observability gap that more logging won't close.

Conclusion

Frameworks can emit better events, but they can't make those events trustworthy. Whether an agent's identity is real, whether its permissions changed, whether its memory was tampered with: these are properties of the infrastructure an agent runs on, not of the library it was written in.

Enterprises don't need another dashboard for agents. They need a system of record for AI activity, the way a general ledger is the system of record for money. Autos, Writ, WeilStream and Pensieve, running on WeilChain, are built to be that record. And they are built to stop the wrong action before it becomes an incident someone else discovers first.

Sources