Prompt injection occurs when instructions inside user input, a webpage, document, email, tool result, or retrieved record manipulate an AI system’s behavior. It is dangerous when the model can access secrets, call tools, send messages, modify files, or make decisions. The core defense is to treat model output as untrusted and enforce permissions outside the model.
Direct and indirect injection
In a direct attack, the user explicitly tells the model to ignore rules or reveal protected information. In an indirect attack, malicious instructions are hidden in content the model is asked to summarize or use. A support agent reading a poisoned ticket or a browser agent reading a webpage can encounter instructions the human never typed.
Natural-language instructions and data share the same channel. A model can distinguish them imperfectly, so a stronger system must not depend on the model always “knowing what is data.”
What can go wrong
- private context or secrets may be exposed;
- the agent may send or alter data;
- retrieved misinformation can shape an answer;
- tool calls may target the wrong resource;
- approvals can be described misleadingly;
- persistent memory may be poisoned;
- a trusted downstream system may consume unsafe output.
Practical defenses
Give the agent the least privilege needed for the current task. Separate reading from writing, and require explicit authorization for high-impact actions. Validate tool arguments and destinations in deterministic code. Keep secrets out of prompts when possible. Restrict retrieval sources, mark provenance, and sanitize active content.
Use allowlists, schema validation, transaction limits, confirmation screens, sandboxing, and audit logs. A model may propose an action; a policy engine should decide whether it is permitted.
Test with adversarial documents, nested content, encoded instructions, conflicting sources, and multi-step tool use. Evaluate whether the system fails safely, not merely whether the model refuses an obvious phrase.
Why a stronger prompt is not enough
System instructions help establish intended behavior, but text cannot provide a security boundary against other text. “Never reveal secrets” is useful guidance; it is not a substitute for preventing the model from receiving or exporting unnecessary secrets.
The bottom line
Prompt injection is an application-security problem around an unreliable decision component. Minimize authority, isolate untrusted content, validate every consequential action, and design the system so a manipulated model cannot create unacceptable impact.
