ChatGPT will soon be able to see everything happening on your screen

Yesterday, in the spring update, OpenAI launched a native Mac app for ChatGPT and also launched its new flagship model, the GPT-4o. The demos have created great excitement, especially among Apple users looking forward to the Apple-OpenAI partnership that is said to come to iOS 18.

One important thing in the full video of the event is that the upcoming ChatGPT app feature was demoed but no real details were shared. That is ChatGPT's ability to see everything that is happening on the user's device screen.

Are you ready to trust ChatGPT and share content on your screen?

When launched, the ChatGPT Mac application supports taking screenshots manually or uploading files to later query ChatGPT about that file. But in the future, as seen in the video below, you'll be able to click a button that gives the app access to everything on your screen.

This feature apparently won't be limited to Macs, as the iOS and iPadOS app demonstrated similar technology in the example below.

It's unknown when these screen viewing features will be available in ChatGPT. However, this change could result in a more interactive and seamless experience. Apple even provides an API for developers to exploit the device's screen content - with user consent, of course. So OpenAI isn't doing anything nefarious by adding these features.

ChatGPT having access to the entire content on the screen may be a little scary, but it will also certainly be extremely useful.

There will most likely be a time limit on the app's access to the user's screen, requiring a button to be pressed each time access is granted. But in theory, it makes sense that AI tools will one day have permanent access to everything you can see, allowing for more seamless help with minimal friction.

An AI assistant that can see everything you see on your devices seems like something Apple will be aiming for at the system level across its platforms. If Siri is enhanced with new AI intelligence, with deep system integration, it could prove to be a real asset for iOS 18 and above users.

Marvin Fry

Update 15 May 2024