Microsoft has been wowing the press and the public with its new AI-powered Bing. But it turns out the technology made several mistakes during Microsoft’s public demo last week.
The errors included made-up information about a financial earnings report from Gap, which was noticed by engineer and writer Dmitri Brereton, who published a post going over the mistakes.
During Microsoft’s demo, which can be viewed online, the company asked the ChatGPT-powered Bing to supply the key takeaways to Gap’s Q3 earnings report. The program proceeds to do so, except part of the summary is completely off.
For example, the AI-powered Bing says that Gap reported an operating margin of 5.9%. However, the company’s earning report clearly states the operating margin was 4.6%.
The AI-powered Bing also says that Gap projected net sales growth in the low double digits. But the actual report states that “net sales could be down mid-single digits year-over-year in the fourth quarter of fiscal 2022.”
In an earlier part of the demo, Microsoft also used the new Bing to query for “What are the pros and cons of the top 3 selling pet vacuums?” The search engine is quick to produce a result, listing the benefits and drawbacks of three pet vacuums.
However, Brereton notes that Bing made an error when describing the Bissell Pet Hair Eraser Handheld Vacuum. It lists the product having a “short cord length of 16 feet” as a con. However, the vacuum is a cordless model designed to be portable. On top of this, Bing seems to provide an answer for the top-recommended pet vacuums, not necessarily the best-selling.
In another part of the demo, Bing made an error when it came to listing nightlife recommendations for Mexico City. The search engine says an establishment called Cecconi's Bar “has a website where you can make reservations and see their menu." But a search of Cecconi’s Bar came up with no such website. You have to call to make a reservation.
In addition, Microsoft also touted the ability of Bing to create a quiz about 90s music. The search engine did so, accurately listing the correct musician for each question. But awkwardly, all the answers for the 10-question quiz are "Answer A," with no variation at all.
Microsoft didn’t immediately respond to a request for comment. But the company’s FAQ for the new Bing concedes the technology won’t always be accurate. “Bing will sometimes misrepresent the information it finds, and you may see responses that sound convincing but are incomplete, inaccurate, or inappropriate,” the company wrote.
Indeed, Microsoft’s own demo does show the technology synthesizing complex topics into easy-to-read articles and lists, most of the time. But in some instances, the technology can be off —and it can be hard for a user to tell, unless they fact-check the answer closely. That said, Microsoft has previously said it’s working to refine the ChatGPT integration with Bing.
Still, the accuracy of Bing and other AI-powered chatbots will no doubt face more scrutiny as they roll out to the public. Google’s own ChatGPT rival, Bard, mistakenly posted an inaccuracy in a public demo last week, too.