Traditional application security assumes a clear line between code and data. LLM features erase that line: if your model reads a customer email, a scraped web page, or a document a user uploaded, it is reading instructions as readily as it reads content. A hidden line of white-on-white text in a PDF can tell your model to ignore its system prompt, exfiltrate the conversation, or call a tool it shouldn't. This isn't a theoretical edge case — it's the default behavior of the technology, and it needs to be designed around, not patched after a headline.
Where the attack surface actually is
- Direct injection — a user types instructions into a chat input designed for questions, attempting to override your system prompt.
- Indirect injection — the more dangerous case: instructions hidden inside a document, email, or web page your model retrieves and reads on the user's behalf.
- Tool-calling abuse — an agent with API access gets tricked into calling a tool with attacker-chosen parameters, not just generating unwanted text.
- Data exfiltration via output — a model with access to sensitive context gets prompted to leak it disguised as a normal-looking response, image markdown, or a crafted link.
What an honest audit checks
- Privilege separation — does the model's tool access match the minimum the task requires, or does one compromised prompt reach everything the service account can touch?
- Output validation — is model output treated as untrusted before it reaches a database write, an email send, or a rendered UI element?
- Adversarial test suite — a reusable set of injection attempts run against every release, not a one-time pen test that ages the day it's delivered.
- Content provenance — can the system distinguish 'instructions from our system prompt' from 'text found in a retrieved document' in a way the model reliably respects?
The uncomfortable truth: no prompt-level defense is bulletproof. The real control is architectural — least privilege on tools, human approval on consequential actions, and monitoring that assumes injection will eventually succeed.
The checklist that ships with every LLM feature we build
Threat-model the specific feature before writing prompts. Scope tool permissions to the narrowest set that completes the task. Validate and sandbox any output that triggers a side effect. Log full context for every consequential action so an incident review has evidence, not guesswork. And red-team it — genuinely, adversarially, before launch — because the first attacker to try will not schedule an appointment.