BBC Study Shows How Inaccurate AI News Reports Really Are
It turns out that getting news from robots playing on the phone with real sources is not a good idea. In a study of BBC OpenAI, Google Gemini, Microsoft Copilot and news prowess Perplexity, the news organization found that “51% of all AI responses” to news topics contained “serious issues of some form.”
In the study, each bot was asked to answer 100 news questions using BBC sources where available, and their answers were then judged by “journalists who were relevant experts on the topic of the article.”
A few examples of the issues include Gemini suggesting that the UK’s National Health Service (NHS) does not recommend vaping as a smoking cessation method ( it does ), and ChatGPT and Copilot claiming that politicians who have left office are actually still serving their sentences. Even more worryingly, Perplexity misrepresented a BBC story about Iran and Israel by attributing views to the author and his sources that were not shared in the article.
Looking specifically at its own articles, the BBC reports that 19% of artificial intelligence reviews contain these types of factual errors, hallucinating false statements, numbers and dates. In addition, 13% of direct quotations were “either modified from the original source or were missing from the cited article.”
The inaccuracies weren’t completely shared among the bots, although that might seem like cold comfort given that none of them performed particularly well.
“Microsoft Copilot and Google Gemini had more serious problems than OpenAI ChatGPT and Perplexity,” reports the BBC, but on the other hand, Perplexity and ChatGPT still had problems with more than 40% of responses.
In a blog post, BBC director general Deborah Turness had harsh words for the companies tested, saying that while AI offers “endless possibilities”, current implementations of it are “playing with fire”.
“We live in difficult times,” Turness wrote. “How long will it be before a headline distorted by artificial intelligence causes significant harm in the real world?”
This study is not the first time the BBC has published AI news bulletins, as its previous report may have convinced Apple to shut down its own AI news bulletins just last month.
Journalists have also previously sparred with Perplexity over copyright issues , with Wired accusing the bot of bypassing its paywall and the New York Times sending the company a cease-and-desist letter . News Corp, which owns the New York Post and The Wall Street Journals, has gone even further and is currently suing Perplexity .
To carry out the tests, the BBC temporarily lifted restrictions preventing AI from accessing its sites, but has since reinstated them. However, despite these blockings and Turness’ harsh words, the news organization generally does not oppose AI.
“We want AI companies to hear our concerns and work constructively with us,” the BBC study said. “We want to understand how they will fix the problems we identified and discuss the right long-term approach to ensuring the accuracy and reliability of AI assistants. To achieve this, we are ready to work closely with them.”