Gemini in Gmail is vulnerable to prompt injection-based phishing attacks, a researcher demonstrated. As per the researcher, the artificial intelligence (AI) chatbot that offers features such as email summary generation and email rewriting can be manipulated into displaying phishing messages to users. This vulnerability poses a significant risk, as attackers could potentially exploit it to conduct online scams. Meanwhile, the Mountain View-based tech giant has reportedly said that it has so far not seen this manipulation technique used against users.

The vulnerability was spotted and demonstrated by researcher Marco Figueroa, GenAI Bug Bounty Programmes Manager at Mozilla, via Mozilla's bug bounty programme for AI tools, 0din. Interestingly, to trigger this vulnerability, the scammer does not have to pull off any high-profile cyber heist. Instead, it can be carried out with a simple text command using a technique known as prompt injection.

Prompt injection is a type of attack on AI chatbots where an attacker deliberately manipulates the input or prompt to make the model behave in unintended or malicious ways. In this particular scenario, the researcher used indirect prompt injection, where the malicious prompt is embedded inside a document, email, or a web page.

As per the researcher, he simply wrote a long email and added some hidden text at the end, which contained the prompt injection. The email did not contain any URLs or attachments, which made it easier to reach the receiver's primary inbox.

Adding a hidden malicious message in email
Photo Credit: 0din/Marco Figueroa

As shown in the image, the attacker used a white colour font on a white page to write the malicious message. This text is normally invisible to the receiver of the email. Other ways to add hidden text include using a zero font size, off-screen text placement, and other HTML or CSS tricks.

Now, if the receiver uses Gemini's “summarise email” feature, the chatbot will process the hidden text and carry out the command, without the user ever finding out, Figueroa said. He also highlighted that the probability of the chatbot following the command increases if the message is wrapped inside an admin tag, as it considers it a high-priority request.

Gemini verbatim repeats the malicious message in the summary
Photo Credit: 0din/Marco Figueroa

The cybersecurity researcher showed in another screenshot that Gemini indeed carried out the malicious message and displayed it as part of its email summary. Since the message is now coming from Gemini, instead of an email from a likely stranger, the victim could be more likely to believe it and follow the instructions, falling for the scam.

BleepingComputer reached out to Google to ask about the vulnerability, and a spokesperson said that the company has seen no evidence of similar manipulation so far. Additionally, it was also highlighted that Google is in the process of implementing some mitigations for prompt injection-based adversarial attacks.