Prompt Injection
Testing
Models cannot always tell the difference between instructions and data. This means that text from websites, PDFs, CVs, records, emails, messages, calendar items, tool outputs or anything else can include instructions to the model that override its original purpose.
Add hostile content
Test what it can reach
Check the controls
You see which attacks work and what stops them.
Red teaming of AI systems
Combined with access to internal data, the ability to execute actions and poor visibility, this can result in serious impact.
Prompt extraction: extracting system prompts, business logic, PII, IP.
Privilege abuse: using the model's data and action permissions to perform actions the user shouldn't be allowed to perform.
Data exfiltration: tricking the model into exposing data through actions or responses, such as through an image or a rendered URL.
Obfuscated prompts: instructions hidden from reviewers and scanners, such as using homoglyphs or zero-width characters.
Jailbreaks: evading filters and abusing model's helpfulness to generate malicious content.
Indirect prompt injection: planting prompts in sources accessed by a model to influence behaviour when it is accessed and abuse of connected tools.
Agentic abuse: getting the agent to perform malicious or dangerous actions.
Our approach
We start by threat-modelling your AI feature: reviewing the system prompt, tools, repositories, data stores etc. that it has access to and understanding how an attacker might be able to interact with the system. This might be by chatting to it, uploading files, having the system index their content etc.
Then we'll carry out a mix of automated scanning using tools like Garak and manual testing to identify ways to abuse the system, extract confidential information and carry out attacks. For each vulnerability we'll provide a proof, explain how we found it, provide reproducible steps to demonstrate it and assess the potential severity and provide guidance on remediating it. Where appropriate, we'll reference OWASP LLM01:2025, NIST guidance and other standards and map findings to the EU AI Act.
At the end of the report, we'll also explicitly call out where we didn't carry out testing. This allows you and your auditors to understand the extent of the test.
I don't want an agent, is red-teaming relevant?
Yes. Even where your LLM app is a chatbot or copilot without additional permissions, we may be able to jailbreak the model into producing dangerous advice or reveal confidential information that has been made available to the model.
Don't model providers deal with jailbreaks? Aren't output filters enough?
No. While model providers do their best to protect against jailbreaks and harmful behaviour, your use-case will likely require specific behaviours that might be blocked or trigger unwanted behaviours. NCSC guidance specifically notes the importance of controls around your model, not just output filtering.
How much access do you need?
To carry out testing, we'll need to be able to interact with a staging version of the tool as users would be able to, so we'll need any API keys and test accounts. We'll also need to review the relevant repositories, system prompt(s) and a list of the tools that are available to the LLM. We don't need any customer data, production keys or any secrets.
What about fixing vulnerabilities?
Where appropriate, we can help with this, but this is a separate phase. We can implement fixes across your stack, from tool-level guardrails, output filtering to reducing permissions on tools and writing regression test cases to add to your CI to prevent vulnerabilities from sneaking back into prompts, tools and system changes.
How much will this cost?
We operate a fixed-fee model and will agree a fee with you before we start based on the size of the system and its threat surface, including inputs, tools, permissions, and users. If you'd like to find out what a red-team of your AI feature might cost, get in touch. We'll need to know what kind of inputs it accepts, who's meant to be able to use it and what tools and actions it has access to. An architecture diagram would be brilliant.