(Credit: PCMag Composite; Microsoft)
AI agents were everywhere at Microsoft’s Build conference in April, and they continue to be a major topic in tech circles. In short, an agent is a type of AI that won’t just respond to your prompts but rather act on your behalf. Microsoft's Copilot Actions feature, which ties in with various third-party web services, is one implementation of this concept. It’s free for anyone to use and allows Copilot to do things for you, such as book show tickets or make restaurant reservations. Google's Project Mariner promises something similar.
Like many of the dozens of other Copilot AI features I’ve tested, I found Copilot Actions to be an intriguing demonstration of a new technology, but one that’s not quite ready for primetime. (It's still quicker to interact with websites and services yourself.) Nonetheless, I encourage you to explore the Copilot Actions feature in its current state, since it might eventually become the standard way to interact with the web.
Below, I walk you through how to access Copilot Actions and discuss some of my initial thoughts on the feature. An important note: I don’t discuss the Microsoft 365 Business version of Copilot Actions here, which aims to automate company processes based on custom forms and private data.
How to Use Copilot Actions
To use Copilot Actions, you must first sign in to your Microsoft account. Then, navigate to the web version of Copilot, which works in any up-to-date browser. Finally, click in the prompt text box and choose Action. This feature is available everywhere except the EU, where privacy regulations prevent its rollout. You get a limited number of interactions with free accounts, while a Copilot Pro subscription gets you more. Microsoft doesn’t specify the limit for either account type, but it cut me off after four sessions with a free account.
(Credit: Microsoft/PCMag)Once you select the Action option, Copilot shows suggestions below the text entry box. The feature supports far more than these recommended sites, however. In fact, Microsoft’s documentation says that it should work with any public website, except for those it determines to be harmful, illegal, or offensive.
Note that even though the microphone icon remains in the input box when you switch to the Action mode, you can’t use your voice with the feature. Clicking the mic while you are in Action mode simply switches you to Copilot's standard Voice mode.
What’s It Like to Use Copilot Actions?
To get started, simply ask Copilot to do something for you on a site you specify. If you don’t pick a site, Copilot uses Bing to search for an appropriate one.
(Credit: Microsoft/PCMag)I tried the following prompt: “Make a dinner reservation for two for 8 p.m. at a good Japanese restaurant nearby using OpenTable.” It helps if you are as specific as possible with your prompt. Don't just type, for example, “Make me a dinner reservation.”
Now comes the fun part. After entering the prompt, you see Action at work. Here’s a screen recording video of how it looks:
Copilot opens a web browser window alongside the original Copilot browser window. This browser (which supports multiple tabs) runs on a cloud-based virtual machine. Copilot drives the page with clicks and entries based on your prompt. In the end, the tool was able to complete a reservation on OpenTable—as long as I entered my phone number and responded with a code sent to the phone.
One strange aspect of the interaction was that Copilot didn’t know my location in Action mode. For some reason, it thought I was in Chicago, which doesn’t correspond with any of my peregrinations in recent months. Perhaps that’s the location of the virtual machine that the Action’s browser was running on. In any case, the standard Copilot mode knew my location and was even able to recommend some good local restaurants. Though, of course, it couldn't actually make the reservation for me.
(Credit: Microsoft/PCMag)In another test, I asked Copilot to find and purchase a recent book on Barnes & Noble’s website. This time, I saw a small panel that asked me to click to expand the Action view. It seems that the feature got an update just in the two days of my testing. Copilot Action said it needed to know what kind of books I liked. I entered “literary.” It then went through a series of steps and came up with the 2024 best-selling novel by Chris Whitaker called All the Colors of the Dark.
You Can Take Control, But I’d Rather Not Have To
Copilot Action lets you take control of its virtual web browser whenever you want, which you might need to do to enter information, such as your phone number. You can, in many cases, fill in that info through the AI’s chat panel on the right, but I occasionally had to do so directly on the page.
Ideally, I'd rather not interact with the site at all, especially since the whole idea behind the feature is that it does things autonomously. True, it does eventually take you through the process of whatever you ask. But it won’t save you massive amounts of time until it can fully carry out an action you desire without intervention.
The Problems With Copilot Actions
Currently, too many websites require checks or permissions that prevent Copilot from completing a task for you. That includes things like Captchas and sign-ins. Copilot Actions is, after all, a feature Microsoft designates as “experimental” and “early stage.”
Another problem? It’s slow. I found that I could book a reservation much faster myself than with Copilot Actions. That wouldn’t be a problem, of course, if it were a fully hands-off experience. On the bright side, this AI process doesn't seem to consume excessive system resources on my local computer, presumably since most of the action happens in the cloud. I didn't notice a jump in CPU or GPU usage versus standard web browsing. Perplexity’s Comet browser, for comparison, taxes the local machine heavily when you perform AI tasks.
One last issue is that, as mentioned, Copilot can't determine my location in the Action mode. Microsoft could solve this issue by letting Copilot Action use your actual browser (which presumably has location cookies), but that presents a privacy concern.
Privacy Isn’t Much of an Issue (Yet)
You’re more likely to have privacy issues with third-party sites and services than Microsoft itself since the Copilot Actions browser runs in the cloud and can't access anything on your local machine. However, Copilot does snap screenshots of the websites it visits to analyze them and determine where to click and type.
This is a far cry from what Recall does on Copilot+ PCs (it saves snapshots locally). Of course, you need to take the same precautions that you would in entering personal info on any website. You don’t want to enter your phone number into a shady site, whether via Copilot Action or a standard web browser.
If the Copilot Actions feature does eventually gain the ability to log in to your accounts or enter your credit card details, that would be a more serious privacy issue. And that's especially true if it could anticipate your needs and take actions using your personal information. But, at the same time, those are the same capabilities that would make Copilot Actions truly useful, so I look forward to seeing how Microsoft balances these competing motivations.
The Action Has Just Begun
For now, Copilot Action is just at the start of its development life and is little more than an intriguing technology demonstration. It can’t proactively do things for you or suggest actions. And websites don’t always allow it to fully complete the process of things, like booking a reservation at a good restaurant nearby, all by itself. None of this is to say that Copilot Action won’t be useful someday, however, but that depends on it gaining access to your personal details and more adeptly navigating site barriers. Third-party sites and services need to handle AI-based interactions better, too. Whether or when it can overcome these challenges is anyone’s guess.