Apple Vision Pro Technology Will Soon Be Able to Transform Your Photos Into Immersive Virtual Worlds.
Today at WWDC, Apple announced a new feature for its Apple Vision Pro VR/AR headsets: users will soon be able to create their own virtual worlds based on the panoramic photos they take. The company also hinted at more immersive spatial photography. Both improvements will arrive this fall.
One of the standout features of the Apple Vision Pro headset is the ability to create immersive, detailed 3D environments that can be toggled on and off directly from the headset. You can adjust the level of immersion like a dimmer switch on a light bulb, allowing you to blend reality and virtuality as you see fit, and even use your cool, yet slightly quirky, Vision avatar in chats with other Vision Pro users in shared environments. Presumably, all of these features will also be available in user-created environments.
But virtual environments “work” because they’re not just photo backdrops. They’re full-view 3D scenes that also include audio elements. Apple didn’t provide specific details, but it appears that turning panoramic photos into immersive environments goes hand in hand with Apple’s improvements to its spatial photography. Converting photos into 3D images is already a feature of Vision Pro, but it’s limited. The 3D effect isn’t very “deep,” and they can only be viewed from the angle they were taken. The gap between such photos and fully immersive environments (where you can see what’s behind you, what’s on the ground, what’s in the sky, etc.) is vast. Even iOS panoramic photos, those landscape shots you take while spinning in place, leave a lot of gaps to fill, like the sky and the ground. That’s where AI comes in.
The spatial cropping feature could be Apple’s step toward realistic 3D imaging.
While creating “true” 3D is impossible without capturing objects/places from multiple angles , Apple demonstrated its upcoming “spatial reframing” technology, which hints at how the company plans to create more immersive 3D effects from flat photos, as well as how panoramic photos can be transformed into virtual spaces. During the presentation, Apple emphasized the ability to enhance photos, but using generative AI to fill in uncaptured parts of a photo is undoubtedly how the company plans to create deeper 3D photos and 360-degree virtual spaces.
I loved Vision Pro’s instant 3D photos so much that I’ve become fascinated with creating Gaussian Spots—scans that allow you to experience true 3D space, meaning 3D objects that can be seen from any angle in a virtual space. Experiencing immersive, lifelike 3D versions of places you’ve been and people you know is the coolest thing you can do with a VR headset . But I have mixed feelings about letting AI “enhance” that. The technology is amazing, but the power of Gaussian Spots lies in how accurately they recreate the scenes you captured: my reaction is, “Yes, that’s exactly how it was.” Creating it with AI simply won’t be the same, and there’s something unsettling about letting AI alter the record of your personal memories. But I’ll still be first in line to try it.