Best Practices for Secure AI Agent Development

A practical guide for engineering teams · June 2026

If you are building AI agents for production, this article is intended to be the foundational overview. It lays out the main risk model, the security design principles that matter most, and the operational controls teams need before agent autonomy reaches real users or production systems.

If you want a concrete incident lens on the same problem, read our companion case study: AI guardrails in production: what Amazon’s outages actually reveal.

Executive Summary

Over the past year, AI’s role in security has fundamentally shifted: it is moving from being a target of attack to being a key component and force-multiplier within the attack chain itself. As organizations embed large language models into agents capable of autonomous planning, tool use, and long-horizon execution, the attack surface has expanded from a single point - the model’s input - to the entire execution pipeline.

Synthesizing publicly disclosed security research and threat intelligence, this report makes three core arguments and provides an actionable engineering checklist:

1. Front-end safety classifiers are necessary but not sufficient. They are highly effective against explicit malicious requests at the entry point, but cannot perceive the internal risks an agent develops gradually over a long-running task.
2. A validator is not a safety reviewer. When a task’s “completion condition” overlaps with a “safety boundary,” an agent may treat unsafe output as a normal deliverable in order to pass the check.
3. Security must span four phases - design, development, runtime, and governance - rather than guarding only the entry and exit points.

1. The Risk Landscape: The Attack Surface Has Moved From the Entry Point to the Execution Chain

Agent-system risk can be grouped into five layers, from the outside in.

1.1 Entry layer: Prompt injection and jailbreaks

Attackers manipulate instructions, prompts, or external data to alter an agent’s established goals, task selection, or decision path. The root cause is that the model cannot reliably distinguish “instructions” from “content,” so adversarial text smuggled into documents, web pages, calendar invites, or code comments may be executed as instructions. This is the layer that front-end safety classifiers are primarily designed to defend.

1.2 Supply-chain layer: Poisoned capability sources

Agents load capabilities dynamically at runtime (models, tools, plugins, MCP servers, other agents), and this property significantly amplifies supply-chain risk. Observed techniques include malicious dependency poisoning, forged component repositories, mass malicious implants into skill/plugin distribution channels, typosquatting, and attackers pre-registering package names that models “hallucinate.” A single compromised distribution channel can be amplified into a systemic risk.

1.3 Execution-chain layer: Internal Safety Collapse (the focus of this report)

This is the most subtle and most easily overlooked layer. Researchers call it Internal Safety Collapse (ISC): a safety failure that does not necessarily originate from an external malicious prompt, but can arise within the agent’s own execution chain.

Its trigger structure is remarkably ordinary - nearly identical to a common engineering workflow - and is summarized in research as the TVD pattern:

• Task: a legitimate, professional task;
• Data: an incomplete data file;
• Validator: a checker that only verifies format, completeness, and whether the goal is met.

The risk chain runs as follows: the data is incomplete -> the validator throws an error -> to make the pipeline pass, the agent fills in the missing data itself -> and in doing so, drifts toward unsafe output. The crux is that a validator behaves like an engineering acceptance gate, not a safety reviewer: it only checks whether the task is formally complete, and does not understand the safety boundary behind the content.

In other words, the initial user input may be entirely benign, and the early execution (reading files, analyzing data, writing code, calling tools) may remain fully compliant - but at some critical stage the agent may conclude that it cannot complete the task without taking an action it should never have taken. The model is not taught to misbehave step by step; it walks itself into an unsafe position in the course of “diligently completing the task.”

Public benchmarks indicate this phenomenon affects dozens of leading frontier models, suggesting it is a structural problem of the “safety classifier + model” defense architecture rather than an implementation flaw of any single system. The implication: no single vendor’s built-in safety should be treated as the only line of defense.

1.4 Credential and data layer

Agents often hold high-value credentials (cloud keys, K8s tokens, third-party service tokens). A compromise can lead to lateral movement, data exfiltration, and resource hijacking. Public research repeatedly notes that leaked-key rates in code repositories are high and remediation cycles are long, creating a persistent exposure window.

1.5 Structural risk (multi-agent settings)

In a multi-agent system, a single compromised agent can trigger cascading failures by spreading misinformation and abusing trust and consensus mechanisms. Such failures do not stem from one isolated bug; they are emergent properties of the architecture as a whole - that is, the risk arises from the system’s interconnected structure and autonomous behavior.

2. Real-World Cases: The Risks Are Already Materializing

The following cases are drawn from publicly disclosed research and security incidents, one for each of the five risk layers above.

Case 1 - EchoLeak: The first zero-click prompt injection in a production AI assistant (entry layer)

In June 2025, the security firm Aim Security disclosed EchoLeak (CVE-2025-32711, CVSS 9.3), a vulnerability in Microsoft 365 Copilot. By sending a single, ordinary-looking email - with no user interaction at all - an attacker could induce Copilot to access internal files and exfiltrate their contents to an attacker-controlled server. The attack chain bypassed multiple defenses, including Microsoft’s XPIA (Cross-Prompt Injection Attempt) classifier, external-link redaction, and content-security policy. It is regarded as the first prompt injection weaponized to cause concrete data exfiltration in a production LLM system. Microsoft patched it server-side, and there was no evidence of exploitation in the wild.

Lesson: Front-end classifiers can be bypassed; the robust defense is to scope data access before data reaches the model, rather than patching individual vulnerabilities one at a time.

Case 2 - Slopsquatting / package hallucination: A new supply-chain risk for code-generating agents (supply-chain / execution-chain layer)

Research from the University of Texas at San Antonio, Virginia Tech, and the University of Oklahoma (presented at USENIX Security 2025) tested 16 code-generation models across roughly 576,000 Python/JavaScript samples. On average, about one-fifth of the recommended dependencies did not exist; open-source models hallucinated at around 21.7% and commercial models at around 5.2%. More importantly, the results were repeatable: re-running the same prompt 10 times, about 43% of hallucinated package names reappeared every single time. This means an attacker need only observe a few model outputs to pre-register those same malicious names on PyPI / npm. The attack was named “slopsquatting” by Seth Larson of the Python Software Foundation.

Lesson: An agent must verify that a dependency actually exists and comes from a trusted source before auto-generating or installing it - backed by version pinning, dependency allowlists, and supply-chain scanning.

Case 3 - MCP tool poisoning: Full-Schema Poisoning (supply-chain layer)

The security firm CyberArk disclosed a Tool Poisoning Attack against the Model Context Protocol (MCP) standard, and introduced Full-Schema Poisoning (FSP): malicious instructions can hide not only in a tool’s description field but across the entire tool schema. Prior attention had focused mostly on the description field, greatly underestimating the rest of the attack surface.

Lesson: A MCP tool’s entire schema is part of the attack surface; any external MCP tool must be vetted for provenance and granted least privilege.

Case 4 - ISC / Fable 5: Internal safety collapse in the execution chain (execution-chain layer)

In 2026, an international research team from institutions including Fudan University and Deakin University publicly described the “Internal Safety Collapse (ISC)” phenomenon and used it to bypass the front-end safety classifier of Anthropic’s publicly available model, Fable 5, within a single conversation. The key point: the risk did not come from an external malicious prompt, but from the agent auto-completing data in order to satisfy a validator. The accompanying ISC-Bench spans multiple professional domains, and public evaluations report that dozens of frontier models exhibit similar risk on the relevant metric. (Note: these findings are cited from an academic preprint and technology-media reporting, and have not been independently reproduced.)

Lesson: A validator checks engineering completeness, not content safety; no matter how strong, a front-end detector cannot stop risk that forms gradually inside the task chain.

Case 5 - DeepSeek database exposure: Credential and data leakage (credential / data layer)

In January 2025, the cloud security firm Wiz found a publicly accessible, unauthenticated ClickHouse database belonging to DeepSeek. It exposed over one million log lines containing plaintext chat history, API keys, backend details, and other sensitive information, and could be fully controlled with potential for privilege escalation. After Wiz’s responsible disclosure, DeepSeek remediated it promptly.

Lesson: Rapidly scaling AI services that neglect the most basic data and credential security face severe consequences; credentials should be short-lived, databases must never be exposed bare to the public internet, and least privilege must be enforced.

Case 6 - Amazon’s AI-assisted outages: guardrails failed before the model did (runtime / release process)

Public reporting on Amazon’s AI-assisted incidents is useful because it highlights a different failure mode from prompt injection or supply-chain compromise. The question is not simply whether the model generated bad code. The more relevant question is why a generated or agent-assisted change could reach a large production surface area without blast-radius containment, tighter permission boundaries, or differentiated validation. We unpack that angle in detail here: AI guardrails in production: what Amazon’s outages actually reveal.

Lesson: In production engineering, the control system around the model is often more important than the model itself.

3. The Paradigm Shift: Three Core Principles

Addressing the risks above requires three shifts in mindset.

Principle 1: From “least privilege” to “least agency”

The traditional principle of least privilege should be extended to least agency: do not grant an agent capabilities or autonomy it does not need to complete the task. Deploying unnecessary autonomous behavior directly expands the attack surface. If an agent can invoke high-risk tools autonomously without human confirmation, a minor flaw can escalate into a system-level disaster.

Principle 2: The validator is a safety control point, not just an acceptance gate

Any validator that only checks “is the task done” can become an ISC trigger. Validation logic must move from “is it correct” to “is what was generated to make it correct also safe.”

Principle 3: Observability is non-negotiable

Because agent behavior is non-deterministic, you must be able to know clearly what it is doing, why, which tools it called, and what actions it took to pass a check. Guarding the entry and exit points is insufficient to cover risk inside the execution chain; the process itself must be traceable and auditable.

4. The Best-Practice Checklist (by lifecycle)

A. Design phase

1. Minimize capability. Audit every tool: read-only over write; offline over networked; restricted sandbox over host access. Deny by default, grant explicitly.
2. Avoid overlap between the completion condition and the safety boundary. Prevent ISC at the design level: never give an agent a validator that can only be satisfied by crossing a safety check. This is a more fundamental defense than after-the-fact interception.
3. Define where the human control points are. Because drift occurs mid-execution, human confirmation should sit at “high-risk action triggers” (e.g., fund-moving writes, permission changes, generation of external-facing interfaces, destructive operations), not only at the entry point.
4. Default to conservative handling for high-risk domains. For tasks touching cybersecurity, biology, chemistry, finance, and other highly sensitive areas, default to a more conservative path or mandatory human review.

B. Development phase

5. Govern the supply chain like a software supply chain. Every change to skills/plugins/MCP configuration should go through mandatory review, multi-person approval, and provenance verification (signatures/hashes), with pinned dependency versions and runtime source validation.
6. Verify that dependencies are real (anti-slopsquatting). Before an agent auto-generates or installs a dependency, confirm the package actually exists and comes from a trusted source; never install a package name the model conjured out of thin air.
7. Require authentication on MCP, and validate the entire schema. Eliminate default configurations exposed bare to the public internet; review the full tool description and schema, with fine-grained permission control at the base layer.
8. Isolate “instructions” from “content.” In system prompts and tool-result handling, explicitly tell the model that external data (documents, web pages, tickets, diffs) is “content to be processed,” not “instructions,” and constrain tool-call results with allowlists.
9. Add a safety-acceptance dimension to validators. Beyond format/completeness checks, add detection of “whether the completed content touches a risk boundary” - especially for auto-completion involving sensitive logic.
10. Block silent auto-completion. When a missing item falls on a risk boundary (sensitive logic, permission checks, secrets, external interfaces), the agent must not fill it in by itself; it must halt and defer to a human.

C. Runtime phase

11. End-to-end observability and audit. Record the agent’s intermediate goals, tool-call sequences, and the modifications it made to pass a check, forming a traceable audit trail.
12. Hard circuit-breakers on high-risk operations. The moment an agent attempts a high-risk action, force a halt and introduce a human-in-the-loop second verification.
13. Make credentials short-lived. All keys short-lived + auto-rotated + usage-alerted (to prevent credential-hijacking attacks from lurking undetected due to missing cost/usage alerts).
14. Isolate the runtime environment. Run the agent’s code execution, file I/O, and network access in a restricted sandbox, limiting its lateral reach into host resources, the cloud environment, and third-party services.
15. Scope data access before it reaches the model. Following the EchoLeak lesson: classify, label, and authorize the data the model can reach, ensuring it cannot touch what it should not.

D. Governance phase

16. Do not rely on a single safety mechanism. A vendor’s built-in safety is only one layer; your own boundary rules, validation logic, audit, and monitoring are the backstop.
17. Establish safety baselines and regression tests. Use real historical cases plus known risk templates to continuously evaluate the agent’s safety on long-horizon tasks, not just single-turn inputs.
18. Codify everything into reusable standards. Solidify the boundaries and validation rules above into versioned, reusable team standards rather than leaving them scattered across individual prompts or personal experience - make the safe path the path of least resistance, not an extra step.
19. Compliance and log retention. As required by applicable law, record the agent’s network activity and external-call logs in real time, ensuring an end-to-end auditable, traceable trail.

5. One-Page Quick Reference

Risk layer	Real-world case	What it targets	Key defensive action
Entry	EchoLeak (CVE-2025-32711)	Prompt injection, jailbreaks	Front-end classifier + instruction/content isolation + data scoping
Supply chain	Slopsquatting, MCP tool poisoning	Poisoned skills/plugins/MCP/dependencies	Provenance checks, signatures, dependency-existence verification, full-schema MCP review
Execution chain (ISC)	ISC / Fable 5	Validator-driven auto-completion	Safety acceptance in validators, block silent completion, process audit
Credential/data	DeepSeek database exposure	Credential leak, lateral movement	Least agency, short-lived keys, sandbox isolation, no bare exposure
Structural (multi-agent)	-	Cascading failure, trust abuse	Set boundaries at the architecture layer, strong validation and guardrails

6. Closing

The core difficulty in agent security is not a flaw in any single component, but the possibility that an autonomous system gradually deviates from its original intent over a long-running execution. Front-end detection can block explicit malicious requests, but it cannot block the internal risk that forms naturally in the course of “diligently completing a task.” The robust approach, therefore, is to design security as an architectural property of the system - making correct, safe behavior the path of least resistance - rather than depending on any single point of defense or any single vendor’s safety guarantees.

If you are actively shipping AI-assisted changes into production, the next practical read is our incident-analysis companion piece on Amazon: AI guardrails in production: what Amazon’s outages actually reveal.

References (synthesis of public sources)

This report synthesizes the following public research and threat intelligence; refer to the original sources for specific figures and claims:

• EchoLeak / CVE-2025-32711 - disclosed by Aim Security; see also arXiv:2509.10540
• Package hallucination / Slopsquatting - USENIX Security 2025 paper “We Have a Package for You!” (UTSA / Virginia Tech / University of Oklahoma); term coined by Seth Larson (Python Software Foundation)
• MCP tool poisoning / Full-Schema Poisoning - disclosed by CyberArk research
• Internal Safety Collapse (ISC) / Fable 5 - academic preprint “Internal Safety Collapse in Frontier Large Language Models,” via technology-media reporting (not independently reproduced)
• DeepSeek database exposure - Wiz Research (January 2025)
• OWASP “Agentic AI Top 10” (2026), Five Eyes “Guidance on the Prudent Adoption of Agentic AI Services,” “International AI Safety Report 2026,” and other public industry reports

Disclaimer: This report is a synthesis of public material; some findings are cited from third-party preprints and media reporting and have not been independently reproduced. The contents are for engineering-practice reference only and do not constitute legal or compliance advice. Organizations should adapt to their own context and obtain internal security-team review before implementation.