Agentic AI Audit Logging: What to Log, How to Store It, and How to Detect Anomalous Agent Behaviour

Agentic AI

AI agents take real-world actions on your behalf. If you cannot audit what they did, you cannot detect when they were compromised or when they went wrong. This is the observability stack every agentic deployment needs — and the anomaly detection patterns that actually work.

By Arjun Raghavan, Security & Systems Lead, BIPI · July 15, 2025 · 11 min read

Software observability has matured over two decades. We have metrics, traces, and logs. We have distributed tracing that follows a request across dozens of microservices. We have anomaly detection built on top of structured telemetry. None of it was designed for a system whose 'logic' lives in a language model and whose decisions are expressed in natural language before being translated into API calls. Agentic AI observability is a new problem, and most teams are solving it with the wrong tools.

What Makes Agent Observability Different

Traditional software observability assumes that the logic is deterministic and auditable in the code. If something goes wrong, you read the code, you read the logs, you find the bug. An agent's logic is not in the code — it is in the model and in the prompt. Two identical inputs can produce different tool call sequences. The same goal can be pursued through different sequences of actions depending on context. Observability for agents has to capture not just what happened but why — and 'why' means capturing the model's reasoning, not just the function calls.

The Minimum Viable Audit Log

• Session ID and agent ID: unique identifiers for every agent instance and every session within that instance
• Goal and task specification: the complete text of the goal or task the agent was given, verbatim
• Reasoning trace: the agent's chain-of-thought or scratchpad content before each tool call (where the model surfaces this)
• Tool call record: tool name, full input parameters, timestamp, duration, and full output or a hash of the output
• Context window snapshot: the complete prompt at each reasoning step, or a versioned reference to the prompt template plus dynamic inputs
• Memory read/write events: every read from and write to external memory stores, with the content and the triggering query
• Action outcomes: the actual effect of each tool call — did it succeed, fail, return unexpected data?
• Human approval events: if human-in-the-loop approval is required, log the request, the response, and the approver identity
• Session outcome: whether the agent completed its goal, was interrupted, or encountered an error

Anomaly Detection Patterns That Work

Standard anomaly detection — statistical deviation from baseline — works for agent observability if you choose the right metrics. The challenge is that agent behaviour has high natural variance; the same goal can legitimately produce different tool call sequences. Anomaly detection needs to operate on semantic patterns, not just structural ones.

1. Tool call frequency spikes: a 10× increase in calls to a specific tool (especially write or send tools) within a session is a strong injection signal
2. Novel tool call sequences: tool call patterns that have never appeared in the baseline for a given goal type warrant investigation
3. Semantic drift in reasoning trace: if the agent's stated reasoning no longer aligns with the assigned goal, flag for review
4. External communication from unexpected paths: any outbound request to a domain not in an allowlist deserves immediate scrutiny
5. Memory write volume anomalies: agents that write significantly more to long-term memory than baseline may be executing a persistence attack
6. Goal completion failure clusters: a sudden increase in goal completion failures for a previously reliable agent class often indicates environmental injection
7. Cross-session pattern correlations: if the same attack vector appears across multiple sessions from different users, it indicates a systemic injection in a shared data source

Storage and Retention Considerations

Agent audit logs are large. A single session with 50 tool calls, each with full context window snapshots, can produce tens of megabytes of log data. Most organisations cannot retain full-fidelity logs indefinitely. The practical approach is tiered retention: full-fidelity logs for 30 days, structured summaries (tool call records, outcome, anomaly flags) for 12 months, and aggregated statistics indefinitely. Complement this with triggered full retention for any session that exceeds anomaly thresholds.

Integration With Existing Security Infrastructure

Agent audit logs should flow into your SIEM alongside application and infrastructure logs. The enrichment layer is different — you need semantic analysis capabilities that traditional SIEM rules lack — but the aggregation and correlation infrastructure is the same. Build agent-specific detection rules that trigger on the anomaly patterns above, and create a dedicated agent incident response playbook that accounts for the unique characteristics of AI agent compromise: the potential for persistent memory manipulation, the possibility of cascading compromise across a multi-agent system, and the need to review all sessions that shared memory or context with the compromised agent.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.