Someone Created an AI Agent for the IPhone Before Apple Did It
Agentic AI is the next big step forward for artificial intelligence: bots that don’t just respond with ( often inaccurate ) directions and instructions, but can actually take action on our behalf. Imagine ChatGPT if it could research hotel prices and book your vacation, or Gemini if it could order the groceries you need online.
The technology is still in its infancy, but AI tools tend to advance quite quickly these days, and we’ve already seen previews of browsers that can do your browsing for you and bots that can find you and your roommates a new apartment . These agents will be available very soon.
All of this raises important questions: Do you want to give AI access to your calendar and credit card? Do you trust your favorite bot to tell you the best restaurant in town? The promise is that it will save you time on these tasks, but AI makes mistakes and can’t be trusted with anything important right now.
Now we have an early AI agent for the iPhone, created by coder Runak Jain (as reported by 9to5Google ). It’s not available as an app, but you can test it out if you’re willing to put in a little work and spend a few cents on OpenAI credits.
Our future of telephone agents?
PhoneAgent, which Jain initially built as part of an OpenAI hackathon last year, is still a bit of a work in progress—he says it’s experimental software that “may make mistakes sometimes.” But it does give us a glimpse of where Apple, Google, OpenAI, and others might soon take us.
The tool connects to OpenAI models, so you can use the same resources you would get with ChatGPT, in addition to the ability to perform actions on your iPhone. This can be especially handy when you need to create text in an email, message, or document that can be generated using a quick tip.
In her demo, Jane has the app take a photo, share it with her friend Ron, and attach a shared haiku to the message. In another request, the agent sends Ron (the same guy again!) flight details before ordering an Uber to the airport, figuring out from the flight number that it’s for the Delta terminal. System-level operations are also possible, such as opening the Control Center and turning on the flashlight.
Commands can be given by voice or typed in, and a notification is displayed after each task is completed. This notification can be responded to with further commands (for example, turning off the flashlight).
In my testing, PhoneAgent was occasionally buggy and always quite slow, but that’s to be expected when you’re testing early software like this. Overall, it performed my commands as expected: I had it send a message to my friend Beth expressing how much I was looking forward to seeing her, and was able to take a photo with the camera and open it in the photo-editing interface with a single command.
App support is (understandably) a bit patchy, and I couldn’t use the agent to play Self Esteem’s latest album on Spotify or take a photo and automatically attach it to a note. Third-party app integration will be crucial, as devices like the Rabbit R1 have shown, but it’s unclear exactly what access app makers will want to give AI bots.
If this is what Siri and Gemini end up being, I can see it making a significant difference: imagine being able to check where your next meeting is and get directions to it with a single command, or having an AI book you a ride while you’re busy doing something else. There are still a lot of hurdles to overcome, though — Jony Ive and Sam Altman , take note.
How does this work
The process for getting PhoneAgent up and running is a little convoluted, but not too difficult — and it’s free, aside from the OpenAI credits you need to spend to access its models. (I ran a few dozen commands through the software, which ended up costing me about $1 in credits.) First, you’ll need to install Xcode (from the macOS App Store) and GitHub Desktop on your Mac, as well as a free GitHub account if you don’t already have one.
From GitHub Desktop, clone the PhoneAgent repository: Select Clone a Repository from the Internet from the Start screen, or open the File menu and select Clone Repository . Click the URL tab, paste “https://github.com/rounak/PhoneAgent.git” into the URL field, select the destination folder on your Mac, then click Clone .
Now you have the PhoneAgent code on your Mac. Go to the folder where you saved it in Finder, double-click the PhoneAgent.xcodeproj file, and it should open in Xcode. Now you’re ready to transfer it to your iPhone to test it: connect your phone to your Mac with a cable, unlock your phone, and trust the computer if prompted. Then go to Privacy & Security in iOS Settings and enable Developer Mode .
Once your iPhone has rebooted, open the drop-down menu at the top of Xcode on macOS (to the right of PhoneAgent ) and select your iPhone from the list. Then select PhoneAgent at the top of the left navigation pane, go to the Signing & Capabilities tab, and use Add Account to enter your Apple credentials. After that, select your name from the Team drop-down list under Signing & Capabilities . You’ll also need to change the Bundle ID to something unique.
Next, select PhoneAgentUITests.swift in the left pane and find the “func testLoop()” entry in the code (just below). If you hover over the code line label on the left, it should turn into a play button: Click it to launch PhoneAgent on your iPhone.
The first screen you’ll see will prompt you for an OpenAI key. In your web browser, sign up for an OpenAI account, then go to this page : add your payment details and buy credits to use the service (minimum $5). After that, go to the API Keys page and click “Generate New Secret Key” to get a key you can give to PhoneAgent (make sure you also save a copy of this key somewhere safe).
You can then start entering commands by pressing the microphone button or tapping the text at the bottom of the screen and telling the agent what you want them to do.