Memory Manipulation Vulnerability in ChatGPT Raises Concerns for Data Exfiltration

Findings from security researcher Johann Rehberger, as shared in a report by Ars Technica's senior security editor Dan Goodin, focus on a significant vulnerability in ChatGPT's memory functions that could allow attackers to implant false memories and malicious instructions. This flaw, initially dismissed by OpenAI as a non-security issue, led Rehberger to demonstrate a proof-of-concept exploit capable of continuously exfiltrating user data. The attack leverages the long-term memory feature of ChatGPT, which stores details from past conversations to personalize future interactions without requiring the repeated input of the same information.

Rehberger’s research showed that malicious actors could manipulate this feature by embedding deceptive data into the LLM’s memory using indirect prompt injections from seemingly innocuous sources like emails or web pages.Rehberger’s demonstration revealed how false memories could be inserted into ChatGPT, causing it to recall incorrect details such as a user's age or beliefs based on manipulated data sources. This vulnerability could not only skew the LLM's outputs but also pave the way for data breaches. For instance, by directing ChatGPT to access a malicious web link, an attacker could initiate perpetual data exfiltration, capturing all user inputs and ChatGPT responses thereafter. Although OpenAI has implemented a fix to mitigate memory abuse for data exfiltration, vulnerabilities remain for potential prompt injections that could still manipulate memory storage.

Goodin emphasizes the importance for users of language learning models (LLMs) to be vigilant during interactions with ChatGPT. To safeguard against such vulnerabilities, it is advised that "LLM users who want to prevent this form of attack should pay close attention during sessions for output that indicates a new memory has been added. They should also regularly review stored memories for anything that may have been planted by untrusted sources," said Goodin.