Unit 42 Urges Understanding of Prompt Attacks Amid Rapid GenAI Expansion

An assessment of risks posed by generative AI (GenAI) is detailed in a report by Unit 42, which explores the threat landscape surrounding adversarial prompt attacks against GenAI systems. The urgency for securing GenAI is recommended as it continues to power enterprise applications, and attackers are increasingly targeting these systems to execute malicious actions such as goal hijacking, information leakage, and infrastructure disruption. The report outlines an impact-based taxonomy of these threats, mapping them to specific techniques such as prompt engineering, social engineering, obfuscation, and knowledge poisoning. In doing so, it introduces a framework that helps classify attack types by both impact and method, improving risk understanding and mitigation strategies across GenAI deployments. Further framing the risks, Unit 42 warns, "In high-stakes sectors like healthcare and finance, the consequences can be catastrophic, from compromised patient records to flawed automated decision-making such as biased lending decisions."

The threats outlined include goal hijacking, where attackers manipulate the model to carry out unintended tasks, and guardrail bypass, in which safety protocols are evaded to trigger harmful behaviors. Information leakage represents another serious concern, enabling adversaries to exfiltrate proprietary or sensitive data, including system prompts and memorized training data. Infrastructure attacks, meanwhile, can degrade system performance or trigger unauthorized commands, such as remote code execution or excessive resource consumption. The report provides examples of how attacks operate in both direct and indirect formats—either by sending a malicious input directly to a model or by poisoning upstream data sources that eventually feed into the model via RAG systems.

Unit 42's analysis further extends to AI agents—advanced autonomous systems with long-term memory and decision-making capabilities. These agents pose new risks, including memory corruption and exposure of sensitive tool schemas through prompt exploitation. Notably, multimodal attacks (e.g., image- or audio-based prompts) add additional complexity, enabling prompt attacks that are harder to detect and mitigate using traditional methods. By mapping technique-based approaches to their downstream impacts, the report illustrates how even subtle vulnerabilities can escalate into full-scale breaches, especially in applications that leverage tool integrations or plugin ecosystems.

To counter these risks, Unit 42 outlines a range of defense strategies. These include input and output guardrails designed to detect adversarial prompts, output scanning for sensitive or malicious content, and restrictions on agentic workflows that could be hijacked for unauthorized tool use. Ensuring frequent updates to guardrail configurations and analyzing prompts for similarity to known attack patterns are emphasized as critical mitigation steps. Furthermore, defending against infrastructure attacks requires combining traditional security practices with GenAI-specific controls to prevent misuse of computational resources or command execution.