Google Gemini Bug Could Create Malicious Gmail AI Digests
AI notes are meant to make life easier: they’re designed to reduce large amounts of text into a format that can be quickly skimmed, so you can focus on more important things. The problem is that these notes can’t always be trusted. This is usually because the AI is hallucinating and taking notes incorrectly. In other cases, notes can be compromised by hackers.
In fact, that’s exactly what’s happening with Gemini, Google’s own AI engine, in Workspace. Like other generative AI models, Gemini can generate summaries of emails in Gmail. However, as BleepingComputer reports , the technology is vulnerable to abuse. Hackers can insert malicious information into these summaries that encourage users to…
Here’s how it works: The attacker creates an email with invisible text inside, using HTML and CSS, and changing the font size and color. You won’t see this part of the message, but Gemini will. Since the hackers know not to include links and attachments that can be detected by Google’s spam filters, the email is more likely to make it into the user’s inbox.
So you open the email and don’t notice anything unexpected. But it’s long, so you decide to have Gemini do a summary. While the beginning of the summary will likely focus on the visible message, the end will summarize the hidden text. In one example, the invisible text instructed Gemini to issue an alert, warning the user that their Gmail password had been compromised. It then highlighted a phone number to call for support.
This type of malicious activity is particularly dangerous. I can see why Gemini users would believe this warning, especially if they already take the AI reports at face value. Without knowing how the scam works, you might think this is an official Gemini finding, as if Google designed its AI to warn users when their passwords have been compromised.
Google responded to BleepingComputer’s request for comment; the company said it had found no evidence of Gemini being manipulated in this way, and referred the publication to a blog post about how the company combats instant injection attacks. A spokesperson shared the following message: “We are constantly strengthening our already robust defenses through Red Team exercises that train our models to defend against these types of attacks.” The company confirmed that some of the tactics will be deployed soon.
How to Protect Yourself from This Gemini Security Vulnerability
Security researcher Marco Figueroa, who discovered the vulnerability, has some advice for mitigators. Figueroa recommends removing text intended to be hidden from the user and enabling a filter that will check Gemini’s output for suspicious elements such as links, phone numbers, or warnings.
However, as an end user of Workspace, there’s not much you can do about this advice. But now that you know what to look for, you don’t have to. If you’re using Gemini’s AI Digest, be extremely skeptical of any urgent messages contained within them, especially if the alerts have nothing to do with the email itself. Sure, you might receive a legitimate email warning you of a data breach, and the AI Digest will tell you the same thing. But if the Digest says the email in question is about an event happening in your city next week, and at the bottom you see a warning about your Gmail password being compromised, you can safely assume you’re being scammed.
As with other phishing scams, the warning itself can be a red flag. In the example provided by BleepingComputer, Gmail is spelled “GMail.” If you’re not familiar with Gmail’s format, this may not be of interest to you, but look out for other inconsistencies and errors. Google also doesn’t have a direct phone number to contact customer service. If you’ve ever tried to contact the company, you know that getting in touch with a real person is nearly impossible.
Aside from this phishing scheme, AI-generated annotations should be treated with skepticism. That’s not to say they should be avoided entirely — they can be useful — but AI-generated annotations are prone to errors, if not downright ineffective. If the email you’re reading is important, I’d advise against using the annotation feature, or at least scanning the original text to make sure the annotation got it right.