You Can Now Allow Gemini to “see” Your Mac’s Display for Free.
It may be controversial, but AI is now being used by a lot of people . Most of them are probably using chatbots like ChatGPT or Gemini as a Google replacement ( don’t do that ) or as a trusted friend ( please don’t do that either ). But if your goal is to use AI as a personal assistant, listen up: on Wednesday, Google announced two new updates for Gemini on macOS, one of which may have some privacy and security implications for users—provided they consent to it.
Gemini for macOS has improved dictation.
If you use dictation in macOS (meaning you speak words out loud and have artificial intelligence type them for you), you might want to choose Gemini over Apple’s built-in solutions. In the future, Gemini in macOS will support “intelligent dictation,” which, according to Google, instantly corrects text “at the cursor.” This means Gemini will remove your stutters, like “uh” and “ah” between words, things you’d never type yourself. This works in any window on your Mac: simply press the Fn key and start speaking.
We’ll need to see this feature in action to see how well it works, but it builds on similar improvements Gemini has on other platforms. It’s a small tweak, but it will allow you to use Gemini for drafting texts, emails, or documents without having to fully edit them yourself. This is exactly the kind of usefulness I expect from AI features. And since it’s enabled by default, all Gemini users on Mac will benefit from it—unless they dig into the settings to disable it.
Gemini can now “understand” your Mac’s display.
Google’s second update for Gemini on macOS isn’t quite as straightforward and isn’t enabled by default. Google calls this “screen analysis,” a feature that allows Gemini to “understand” what’s happening on your Mac’s screen. The idea is to delegate tasks to Gemini: for example, if you have the Notes app open, you could press the Fn key and say, “Turn these notes into a summary for this afternoon’s meeting.” Gemini will reportedly know to look at your Notes app, analyze the text, and customize it on your behalf, all without the need for a proprietary app.
You can also combine multiple teams: for example, ask Gemini to compose an email for a team dinner and pull important information from files on your Mac (members, budget requirements, dietary restrictions, etc.). You can further ask Gemini to search Google Maps for restaurants of a certain type and include them in the team’s list of options. And you can make corrections at any time, and Gemini will adjust accordingly. For example, if you specified that the team dinner was to take place at 6:30 PM on Friday, but later your team added, “Sorry, dinner is supposed to be at 7:00 PM,” Gemini will adjust accordingly.
Of course, these features have their uses. If you already use Gemini on macOS, being able to “see” your Mac’s screen makes it even more convenient. Instead of relying on the Gemini app for all your tasks, you can extend its capabilities to your entire Mac—and, if Google is right, in a rather sophisticated way. But there’s always a privacy risk in giving AI programs complete access to what’s happening on your personal computer, and there’s also a potential security risk. Would you want Google’s AI to be able to see everything that’s happening on your screen?
Of course, this isn’t as big a risk as using a real agent program. If you pay for Google’s most expensive AI plan, you’ll get access to Gemini Spark , which can run autonomously on your Mac. The biggest warning sign is “hint injection,” where hackers hide malicious commands for your AI in websites. If you instruct the AI to perform an action and it encounters such a command, it will think the request is coming from you, not the hacker. So, when it processes a request like “open [malicious website] and download [malicious file],” it will do so without question; or “open [cryptocurrency wallet], sign in with saved credentials, and send funds to [account],” it will follow suit. In its current form, this shouldn’t be a problem for Gemini, although it could theoretically happen if you ask it to perform a task that exposes it to hint injection.
If you’re interested in the latest features, this doesn’t dissuade you from using Gemini’s new agent feature on macOS, but there are some very real risks you should consider before doing so.