Every Tool Call Is a Security Boundary (Treat It That Way)
Agentic AI
When an agent calls a tool, it crosses a trust boundary. We walk through wrapping tool definitions with auth, output filtering, and side-effect logging using lessons from three production incidents.
By Arjun Raghavan, Security & Systems Lead, BIPI · May 2, 2024 · 7 min read
A fintech client shipped an internal agent that could query their data warehouse. Two weeks in, a support engineer asked it to summarise churn for a specific account. The agent obediently joined that customer's tables to a marketing CRM extract and dumped 14k records of PII into the chat transcript. Nobody had explicitly authorised that join. The tool just had table-level access and the model felt creative.
We see this pattern every quarter. Teams treat tool calling as plumbing. It is not plumbing. Each tool definition is an authorisation surface, an audit point, and a blast radius. If you do not design it that way, the model will design it for you, badly.
The trust model most teams skip
Before writing a single tool schema, sketch the trust model. Who is the caller (end user, internal service, autonomous agent)? What identity propagates into the tool invocation? What can the tool read, write, or trigger downstream? In the fintech incident above, the warehouse role was a single service account with read access to everything. The model effectively inherited that role.
Our rule on engagements: every tool has its own least-privilege identity. If a query tool needs to read three tables, it gets a role bound to those three tables. If a write tool can modify orders, it cannot also read PII. That sounds obvious. It is almost never how the first prototype is built.
Wrapping tools with three layers
We wrap every tool in a small middleware that handles auth, output filtering, and side-effect logging. The wrapper is boring code. That is the point.
- Auth check: resolve the end-user identity, not the agent's identity. Reject if the caller cannot perform this action directly.
- Output filter: redact fields the caller is not entitled to see. Truncate large result sets so the agent cannot exfiltrate via summarisation.
- Side-effect log: every state-changing tool emits a structured event with caller, args, result hash, and a correlation ID linking back to the agent trace.
The output filter is the one teams forget. A tool can be authorised to run yet still leak data through the response. We had a healthcare agent where a permitted query returned encounter notes that included physician handwritten remarks. The auth check passed. The handwritten notes were the real risk. A field-level allowlist on the response payload caught it.
Side-effect logging that is actually useful
Generic API logs are not enough. You need a log per tool invocation that ties together: the trace ID of the agent run, the prompt version, the tool name and args, the resolved identity, and a hash of the result. Hash, not body, unless you have data classification reasons to store the body. With this, an incident review takes 20 minutes instead of two days.
We standardise on three event types: tool.invoked, tool.completed, tool.failed. Each carries the correlation ID. Our SIEM pipeline alerts when a single trace produces more than 30 invocations of the same tool, when a tool fails three times in a row, or when a write tool fires outside business hours from a non-staff identity.
What we audit on day one
- List every tool. For each, name the data it touches and the side effects it produces.
- Confirm each tool has its own identity, not a shared service account.
- Verify output filtering exists and is tested with a deliberately over-broad query.
- Check that the agent's transcript and the tool's audit log can be joined by a correlation ID.
- Run a red-team prompt that tries to get the agent to combine two tools in an unintended way.
The fifth one is where most teams find their gaps. The model is not malicious. It is just very good at composition. Two safe tools combined become an unsafe one. Audit the compositions, not just the primitives.
If you cannot point at where the auth, filter, and log happen for a given tool, that tool is not production ready. It is a prototype with a confident voice.
Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.