The company is hoping for one of its biggest hits in decades with Copilot for Microsoft 365, an AI upgrade that plugs into Word, Outlook and Teams. It uses the same technology as OpenAI’s ChatGPT and can summarize emails, generate text and create documents based on natural language prompts.
Companies involved in testing say their employees have been clamoring to test the tool—at least initially. So far, the shortcomings with software including Excel and PowerPoint and its tendency to make mistakes have given some testers pause about whether, at $30 a head, it is worth the price.
“I wouldn’t say we’re ready to spend $30 per user for every user in the company,” said Sharon Mandell, the chief information officer at networking hardware company Juniper Networks, which has been testing Copilot since November.
Microsoft has said that early demand from users is unprecedented and the companies testing it have found it valuable. The company hasn’t shared specifics about sign-ups.
The company has bet billions—notably through its $13 billion investment in OpenAI—that it can lead the way in AI. Its shares have soared on the belief that the technology will turbocharge its revenue, making it the most valuable company in the world. On Sunday, it showed its first Super Bowl ad in four years, touting Copilot as a tool that can help young people realize their entrepreneurial dreams.
“Copilot has to be a success for the stock to work,” said Rishi Jaluria, an analyst with RBC Capital.
During Microsoft’s recent earnings call, Copilot was mentioned more than 50 times. Chief Executive Satya Nadella likened it to the adoption of the personal computer in workplaces.
“We’ve moved from talking about AI to applying AI at scale,” he said.
Microsoft’s earlier AI upgrades have had mixed results. In 2022, the company released GitHub Copilot, which helps programmers write code faster using AI. The feature, which starts at $10 a month, has 1.3 million subscribers—up 30% from the previous quarter, Microsoft executives said.
The AI-powered chatbot added to Bing search has struggled to make waves. Despite a flashy rollout from Microsoft, it didn’t make much of a dent in Google’s search market share. Executives had expected potentially billions in new revenue if Bing gained on Google, and nearly a year later, Bing has gained less than 1 percentage point of market share.
Some companies are hesitant to dive into adopting AI technology. A survey from Boston Consulting Group showed that while nearly 90% of business executives said generative AI was a top priority for their companies this year, nearly two-thirds said it would take at least two years for the technology to move beyond hype. About 70% of them were focused only on small-scale and limited tests.
Microsoft had been betting that the desire to use the AI would persuade companies to sign up for massive contracts. When it made the assistant widely available in November, companies had to sign up for at least 300 subscriptions. Many were resistant to that size of commitment for unproven software, said a Microsoft software reseller.
Last month, Microsoft eased its minimum requirement, allowing businesses to pilot the software with much smaller groups.
Microsoft says it is confident the time and money saved through the tool are worth the cost.
In November, Microsoft’s survey of early testers of the software found that 77% of them didn’t want to give it up because it saves workers’ time. Meeting summaries saved people about 30 minutes a meeting. Artificial-intelligence-assisted writing saved users six minutes on writing first drafts, Microsoft said.
Materials-science company Dow plans to roll out Copilot to approximately half of its employees by the end of 2024, after a test phase with about 300 people, according to Melanie Kalmar, chief information and chief digital officer at Dow.
“There have been tremendous efficiency gains,” Kalmar said. The company, based in Midland, Mich., has approximately 35,900 employees.
Early testers who spoke with The Wall Street Journal said the AI add-ons for Microsoft’s workplace communication software Teams were most useful.
Some had already become reliant on Copilot if they were late to a meeting, to get a summary of what had been said. Those in different countries and time zones now have the option to skip meetings and just read summaries.
“It has allowed people to say, ‘You know what, there is already 10 other people on the call. I’m going to skip this one. I’m going to catch up in the morning by reading the digest and skipping to the parts of the meeting I really needed to hear,’” said Art Hu, the global chief information officer at Lenovo.
In other areas, testers say the tech has fallen short: Copilot for Microsoft 365, including other generative AI tools, sometimes hallucinated, meaning it fabricated responses. Users said Copilot, at times, would make mistakes on meeting summaries.
At one ad agency, a Copilot-generated summary of a meeting once said that “Bob” spoke about “product strategy.”
The problem was that no one named Bob was on the call and no one spoke about product strategy, an executive at the company said.
In other programs—particularly the ones that handle numbers—hallucinations are more problematic. Testers said Excel was one of the programs on which they were less likely to use the AI assistant because asking it to crunch numbers sometimes generated mistakes.
Jared Spataro, Microsoft’s corporate vice president of workplace applications, said that Excel was still in preview and lagged behind the other programs in usefulness.
Other features that Microsoft has touted, including the ability for Copilot to generate PowerPoint slides, have also been disappointing, some users said.
Guido Appenzeller, a partner at investment firm Andreessen Horowitz, posted a thread on X showing the mistakes it makes when prompted to make a presentation.
“It is a mess and not anywhere close to adding value,” Appenzeller said.
Ethan Mollick, a professor of AI at the University of Pennsylvania’s Wharton School, noted in a post that in Outlook, Copilot suggested times for a potential meeting that were booked or on Saturdays.
Mollick otherwise gave the software a positive review, calling it a “pretty impressive set of tools.”
Microsoft’s Spataro said PowerPoint is “still learning its way.”
Some early adopters said the initial excitement about the AI tools wore off quickly. Lenovo said that aside from the AI used to transcribe meetings on Teams, there was about a 20% drop in the use of Copilot for most software after a month.
Microsoft’s Spataro said the company plans to address the drop-off issue by incorporating more alerts and tips to nudge users.
Steven Rosenbush contributed to this article.
Write to Tom Dotan at tom.dotan@wsj.com
