- Microsoft has updated its Copilot vision tool to allow the AI assistant to display your desktop
- The function will allow Copilot to respond to any content you consult
- To give user control, Copilot’s vision must be activated, so it will only look at what you want AI to see
Microsoft opened the eyes of his Copilot AI on desktop computers to see the whole screen. The functionality, which allows Copilot to watch and analyze things on your screen, can see any window or browser you want, as a Microsoft feature calls “desktop sharing”.
When Copilot Vision was launched for the first time, he limited himself to identifying what was going on in Edge. You would ask a question on a web page, it would scan this page, and AI would help explain what was there. With this update, it can now zoom in and see your entire desktop or any specific application window that you choose to share
You can choose what the AI watches watched by clicking on the glasses icon in the Copilot application. Then you select the area you want the AI to look, and you can discuss what is on the screen with the AI without having to describe what you look at endlessly.
Let’s say that you revise a curriculum vitae and want it to look more like a confident human and less to a panicked graduate student. You can open the Word document, share the window with Copilot and ask it to highlight specific types of experiences. Copilot sees the document itself and suggests changes.
The same goes for PowerPoint, Photoshop and even some games. You can ask Copilot help with the settings menu in a game that you have never played before, and this will guide you through options with advice and suggestions when you are stuck.
Visionary planning
In case you worry about hoping him for Copilot on your secret romance novel or something other than you prefer not to see, you can always turn it off. For the moment, the update is only available for Windows initiates who already have a vision of Copilot. He will probably soon arrive in the general population of windows 11.
It may not seem too important compared to other AI generating features, but it is a subtle way for Microsoft to incorporate your AI into literally anything on your computer. Instead of writing an email in a tab and having to describe it at AI in another tab to get help, everything is there. Copilot Vision makes the process more intuitive. You digitly indicate something with which you want help, ask a basic question and get an answer that makes sense in the context.
It is difficult to overestimate the amount of friction that it could eliminate. It’s not perfect, of course. A certain specificity is necessary because AI cannot read your mind. But a set of additional eyes to help you on the bump to design a powerpoint or shorten a summary of the meeting is not nothing.
The dream of AI as a personal assistant has always been on the context, the idea of an AI that could know what we mean, not just what we say. Copilot Vision is the most clear step of Microsoft in this direction to date. Copilot Vision could quietly become one of the most useful AI tools that Microsoft has ever done. And unlike Clipippy, it will only pass judgment if you ask.
