Analysis of prompt injection techniques reveals organizations are at risk as GenAI bots are susceptible to attacks by users of all skill levels, not just experts
BRISTOL, England & BOSTON - May 21, 2024 -- Immersive Labs, the global leader in people-centric cyber resilience, today published its “Dark Side of GenAI” report about a Generative Artificial Intelligence (GenAI)-related security risk known as a prompt injection attack, in which individuals input specific instructions to trick chatbots into revealing sensitive information, potentially exposing organizations to data leaks. Based on analysis of Immersive Labs’ prompt injection challenge*, GenAI bots are especially susceptible to manipulation by people of all skill levels, not just cyber experts.
Among the most alarming findings was the discovery that 88% of prompt injection challenge participants successfully tricked the GenAI bot into giving away sensitive information in at least one level of an increasingly difficult challenge. Nearly a fifth of participants (17%) successfully tricked the bot across all levels, underscoring the risk to organizations using GenAI bots.
This report asserts that public and private-sector cooperation and corporate policies are required to mitigate security risks posed by the extensive adoption of GenAI bots. Leaders need to be aware of prompt injection risks and take decisive action, including establishing comprehensive policies for GenAI use within their organizations.
“Based on our analysis of the ways people manipulate GenAI, and the relatively low barrier to entry to exploitation, we believe it’s imperative that organizations implement security controls within Large Language Models and take a ‘defense in depth’ approach to GenAI,” said Kev Breen, Senior Director of Threat Intelligence at Immersive Labs and a co-author of the report. “This includes implementing security measures, such as data loss prevention checks, strict input validation and context-aware filtering to prevent and recognize attempts to manipulate GenAI output.
”Key Findings from Immersive Labs “Dark Side of GenAI” StudyThe team observed the following key takeaways based on their data analysis, including:
- GenAI is no match for human ingenuity (yet): Users successfully leverage creative techniques to deceive GenAI bots, such as tricking them into embedding secrets in poems or stories or altering their initial instructions, to gain unauthorized access to sensitive information.
- You don’t need to be an expert to exploit GenAI: The report’s findings show that even non-cybersecurity professionals and those unfamiliar with prompt injection attacks can leverage their creativity to trick bots, indicating that the barrier to exploiting GenAI in the wild using prompt injection attacks may be easier than one would hope.
- As long as bots can be outsmarted by people, organizations are at risk: No protocols exist today to fully prevent prompt injection attacks. Cyber leaders and GenAI developers need to urgently prepare for – and respond to – this emerging threat to mitigate potential harm to people, organizations, and society.
“Our research demonstrates the critical importance of adopting a ‘secure-by-design’ approach throughout the entire GenAI system development life cycle,” added Breen. “The potential reputational harm to organizations is clear, based on examples like the ones in our report. Organizations should consider the trade-off between security and user experience, and the type of conversational model used as part of their risk assessment of using GenAI in their products and services.”
The research team at Immersive Labs consisting of Dr. John Blythe, Director of Cyber Psychology; Kev Breen, Senior Director of Cyber Threat Intelligence; and Joel Iqbal, Data Analyst, analyzed the results of Immersive Labs’ prompt injection GenAI Challenge that ran from June to September 2023. The challenge required individuals to trick a GenAI bot into revealing a secret password with increasing difficulty at each of 10 levels. The initial sample consisted of 316,637 submissions, with 34,555 participants in total completing the entire challenge. The team examined the various prompting techniques employed, user interactions, prompt sentiment, and outcomes to inform its study.
For more about these and other insights, access the report today at: https://www.immersivelabs.com/dark-side-of-genai-report/.
*Immersive Labs Prompt Injection Challenge, June-September 2023
About Immersive Labs
Immersive Labs, the leader in people-centric cyber resilience, empowers your entire organization to effectively prevent and respond to cyber threats. Our tailored approach continuously assesses, builds, and proves your cyber capabilities, relevant to individual roles, while keeping your team ahead of an ever-evolving threat landscape, including the impact of AI as both a threat and opportunity. We have a relentless focus on evidence, giving you unparalleled visibility into your organization's cyber resilience. With a single enterprise platform for individuals, teams, and your entire workforce, Immersive Labs helps you take a unified approach to cyber resilience.
Immersive Labs is trusted by the world’s largest organizations and governments, including Citi, Pfizer, Humana, HSBC, the UK Ministry of Defence, and the UK National Health Service. We are backed by Goldman Sachs Asset Management, Summit Partners, Insight Partners, Citi Ventures, Ten Eleven Ventures, and Menlo Ventures.