Yes, the New Audio Intelligence Features in Apple Watch Really Are Always Listening to You.

During its September 9th event , Apple announced the Apple Watch Series 12 and Ultra 4. Both models will feature a feature that passively listens to your audio conversations throughout the day and can provide you with a summary upon request. It’s part of a suite of features called “Audio Intelligence,” but there are two key features worth noting:

• Live Rewind: Double-click the digital crown on your Apple Watch to display a preview of your conversation from the last fifteen seconds.

• Siri Recaps: Your watch listens to your conversations and generates a summary of the transcribed audio recordings at the end of each day using artificial intelligence.

At first glance, the concept itself seems like a privacy nightmare, but Apple has implemented some security measures to make users feel more comfortable with the feature: recorded audio is never recorded or saved, and never leaves your device; it is essentially immediately and in real-time deleted from local storage; only a truncated version is uploaded to Apple Private Cloud Compute servers for AI processing, and the resulting transcript, which is provided to you, is end-to-end encrypted.

However, there are some potential privacy concerns to consider, especially considering that you’ll essentially be exposing everyone around you to the service without their consent.

How an audio intelligence system creates a transcript of what it hears

The Series 12 and Ultra 4 watches are powered by the S11 chip with Secure Exclave —an isolated local storage that even the operating system can’t access. When your Apple Watch detects ambient sound, it’s sent directly to this secure buffer, where it’s encrypted and then transmitted to the corresponding secure enclave on your paired iPhone.

Your iPhone then locally transcribes the audio recording using a built-in AI model to create a shortened version half the length of the original audio file. This removes everything but the essentials and eliminates any identifying information. After transcribing the audio data, the transcript is sent to Apple’s private cloud storage along with some general metadata, such as approximate location and timestamps.

At this stage, no audio data is linked to your Apple ID once it reaches the servers. Apple uses a third-party relay to obscure your IP address while uploading data to private cloud storage, so even their employees can’t identify you by your device. On the private cloud servers, your audio data is processed by Siri’s artificial intelligence. This way, you receive daily text summaries of your audio recordings if you enable Siri Recap. Siri Recap summaries are automatically deleted every seven days unless you choose to save them individually.

This may all seem perfectly safe, but here’s the risk: if you choose to save a conversation summary using Siri Recap, that data will be uploaded to your iCloud storage and may even be saved in apps like Apple Notes or Journal. Server-level data breaches at companies like Apple are rare, but account-level hacks are more common than you might think: just in January of this year, a data breach was discovered on a public server that compromised 900,000 iCloud accounts .

If someone manages to hack your Apple ID or gain access to your unsecured iPhone, they’ll be able to view all your saved chat summaries in one place. And you should be concerned not only about the potential data leak, but also about all the associated privacy issues.

A potential privacy nightmare

Apple has spent a lot of time developing ways to protect its users when using Audio Intelligence, but none of those protections extend to other people who might be sitting across from you at a table or in the same room.

To Apple’s credit, the company has added an audible alert and a short visual indicator that play on your watch each time Live Rewind is triggered. This is intended to alert others that they are being eavesdropped on. Live Rewind also doesn’t identify speakers, so it doesn’t name or identify the speaker in transcripts or summaries—but none of this actually prevents eavesdropping without their explicit consent.

Siri Recap’s implementation is perhaps even worse. There are no audible or visual cues indicating it’s active, so no one will know they’re being tracked if you enable it. Apple’s privacy documentation is extremely sparse regarding consent, save fora vague warning on product pages advising you to “be mindful of your surroundings.”

This is a potential problem not only for other people who might worry about being tracked every time they see their Apple Watch, but also for Apple Watch users themselves if they are in a region where local laws require consent before audio recording: 12 US states have laws explicitly prohibiting undisclosed recordings, and outside the US, many EU countries have introduced similar requirements; the EU’s GDPR also prohibits recordings containing personal data without disclosing it.

Using an Apple Watch with Audio Intelligence enabled might seem like a violation of these laws, but for now, the device appears to be operating in a bit of a gray area —as noted in Apple’s privacy documentation :

Audio Intelligence features do not create audio recordings that could be accessed by the operating system, apps, the user, or Apple. There are no recordings that could be shared, forwarded, or played at the request of any party, because such recordings do not exist.

According to Apple, the Secure Exclave buffer, where audio is initially processed, is a “permanently overwritten stream that exists only within the secure hardware and never creates an audio recording,” meaning that recordings don’t require consent. In theory. But the fact that Apple felt the need to explain in detail how this feature technically doesn’t violate any privacy laws likely doesn’t make it any less dystopian for those inclined to view it that way.

How to turn Audio Intelligence on or off

Considering all the potential security concerns, it’s good that Apple at least made the Audio Intelligence features opt-in. They won’t be enabled by default on your new Apple Watch, and you can enable or disable individual features as you see fit. Here’s what to look for:

• Sound Recognition: Open the Settings app on your iPhone. Go to Accessibility > Sound & Name Recognition > Sound Recognition. Here you can turn “Discover with Apple Watch” on or off.

• Music Recognition: Open the Settings app directly on your Apple Watch. Tap Smart Stack. Toggle Music Recognition on or off according to your needs.

• Live Rewind: Open the Watch app on your iPhone. Go to Siri > Live Rewind . You can enable or disable “Double-tap the digital crown” to control this feature.

• Siri Summary: In the Watch app on your iPhone, go to Siri > Siri Summary . Here you can turn this feature on or off, or even schedule it to run automatically. Siri Summary can also be turned on and off by toggling a switch in Control Center.

More…