OpenAI Introduces Two New Ways to Communicate With ChatGPT
ChatGPT has changed a lot since its introduction late last year, especially if you pay for it. GPT-4 is a powerful update to GPT-3.5, and plugins connect you to third-party services to make ChatGPT even more useful. Now the chatbot is changing again: it goes beyond text conversations and supports both visual and audio content.
OpenAI announced these changes to ChatGPT on Monday . As long as you’re a Plus subscriber, you’ll be able to chat directly with ChatGPT as well as share images in your chats, which can potentially increase the bot’s usefulness. You can ask ChatGPT questions about specific elements of an image, using the drawing tool to focus the AI where you want it. Plus, you can have real conversations with ChatGPT because not only does it understand your voice, but it now has its own voice.
Share images using ChatGPT
In the introductory video, the company showed the following example: You may be having trouble figuring out how to lower the seat on your bike. You can launch the ChatGPT app, take a photo of your bike and ask ChatGPT for help. He’ll give you an overview of the solution, but may ask for another image for more context. You can then take another photo (perhaps a close-up of the seat itself) and then draw a circle around the lock on the seat to focus ChatGPT’s attention on it.
According to OpenAI’s example video, ChatGPT will be able to differentiate between different parts of a bicycle: You can ask if a particular part is a bicycle lever, and ChatGPT can answer by saying that it’s actually a bolt that requires a hex wrench to loosen. But what’s even more impressive is that you can share your bike’s manual as well as your toolbox and ask if you have the right tool for the job. ChatGPT will analyze your images and then approve or reject: if you have the right tool, it will tell you where it is in your toolbox.
Finally, the user in the example video thanked ChatGPT for its help, further confirming my suspicions that AI companies think we need to treat our future robot overlords well.
The possibilities here go beyond the example above. You can take a photo of the contents of your refrigerator and ask ChatGPT to help you plan dinner, or send an image of a building and learn the history of its construction. This feature is available on all ChatGPT platforms for Plus and Enterprise users and will be rolled out over the next two weeks.
Talk to ChatGPT
Of course, ChatGPT now uses not only visual elements, but also auditory ones. In the second example video, a user voice asks ChatGPT to tell him the story of a “super-duper sunflower hedgehog named Larry”—a reference to the company’s video introducing DALL·E 3 to the world. They want ChatGPT to start with a little story about Larry. ChatGPT, of course, creates an introduction for Larry before the user chimes in and asks what his house is like. This goes back and forth as if ChatGPT were a real storyteller improvising on the fly.
What’s great about this feature is how natural it is compared to text chats. You can press the microphone button at any time to interrupt ChatGPT and ask for more information about something the bot is explaining, or redirect the conversation entirely. Voice is only available on the ChatGPT app on iOS and Android , and will also be available over the next two weeks for Plus and Enterprise users. You’ll find the option to enable this feature under Settings > New Features . Then tap the headphone button in the top right corner and choose which voice you like from the five options. (OpenAI actually worked with voice actors on this feature.)
Of course, like all AI features, these options won’t be perfect and will be subject to the same risks of hallucinations we’ve seen in the past. OpenAI is well aware of this fact: the company tested the image model with members of the red team on issues such as extremism and scientific knowledge, and developed the feature based on their experience with the Be My Eyes app, which helps visually impaired people use their phones. to “see” for them. The model is also limited in its ability to return results about people.
In short: don’t rely on these features in high-risk or serious situations. Adjusting your bike seat? Certainly. Changing a tire on a car? Tough passage.