Sign up for The Media Today, CJR’s daily newsletter.
In 2023, a thirty-one-year-old construction worker named Pablo Xavier, whose last name has not been made public, accidentally did something groundbreaking. He opened up Midjourney, a generative AI platform that produces images from text prompts, and fed it a precise description of Pope Francis wearing a puffy coat. Then, according to an interview with BuzzFeed News, Pablo Xavier took the picture Midjourney produced and uploaded it to the internet, where it went viral. It was remarkable not only for its absurdity and irreverence but also for how wildly convincing the image was, from the fidelity with which it rendered the sheen on the pope’s jacket to the subtle reflection that his glasses cast on his cheek. It was far beyond anything the world had seen from an AI hobbyist.
At the time, Emmanuelle Saliba was about ten years into her journalism career, working with emerging open-source investigative techniques, or OSINT, to help newsrooms confirm the authenticity of eyewitness photos, mostly from social media. This often involved determining whether images had been manipulated using Photoshop or repurposed in a misleading way. “I just happened to be very good at finding things on the internet,” she said. Saliba helped develop NBC’s verification protocol and, by the time Pope in a Puffer Jacket started making the rounds, she had joined ABC to cover the rapidly evolving world of AI-generated media, hoping to help the public stay ahead of disinformation. She found herself amazed by the proliferation of synthetic media—and terrified by what it portended.
“The entire verification process needed to be reexamined,” Saliba, now thirty-five, said. The fact that an amateur could produce Pope in a Puffer Jacket meant not only that an AI-generated image could be passed off as real, but also the opposite: that legitimate photographic evidence could be plausibly discredited. “It’s not only about detecting falsehoods, it’s about proving reality,” Saliba said. The race was on to develop a tool that could accurately and reliably make the distinction.
It was around this time that she met a computer scientist named Hany Farid. During a reporting trip to California, where Farid ran the University of California, Berkeley, School of Information, Saliba spoke with him about the threat that AI-generated content could pose in the upcoming presidential election. She also found herself fascinated by his contributions to the burgeoning field of AI detection. “He’s really the father of digital forensics, which is the art of understanding image manipulation,” she said. Saliba saw in their work “a natural alignment.”
For decades, Farid had been exploring the use of computational techniques to detect digital manipulations of photographs. Thanks to recent breakthroughs in machine learning, he had shifted focus to models that could detect the kinds of visual inconsistencies typical of an AI-generated image: oddities of perspective and light, for example, resulting in distortions of scale or out-of-place shadows that can be difficult to spot with the naked eye. If such a model could be turned into an accessible tool, Saliba thought, Farid’s work would be a great asset to journalists.
They stayed in touch. Within a couple of years, Farid started a company called GetReal, offering AI detection services, including real-time deepfake detection during video conferences, to clients. Saliba left ABC and joined Farid’s operation, with an eye toward helping optimize GetReal for use in newsrooms. “I wanted to be part of a team that was working to prove authenticity and integrity of content,” Saliba said.
Examine GetReal’s website—or any of the dozens of AI detection services that have surfaced in the past several years, for images as well as text—and it’s easy to come away optimistic. “Prevent image, audio, and video deepfakes and impersonators from compromising the authenticity of your mission-critical workflows,” GetReal proclaims. The website of Hive Moderation, another tool used to monitor crowdsourced content, boldly declares: “They said it couldn’t be solved. We solved it,” and promises that its model offers an automated solution “with human-level accuracy.” Another tool, confusingly named Undetectable AI, claims “this tool ensures you can trust what you see.” Sorting real media from synthetic would now seem to be a matter of just applying the right tool at the right time.
But according to Siwei Lyu, a professor of computer science at the University of Buffalo, whose work focuses on deepfake detection, the reality is not so simple. As the technology to detect AI improves, so does the fidelity of the AI. Detection models need to be constantly updated to account for that. “AI technology advances very fast,” he said. “And the detectors become out of date if they haven’t seen new data types in their training dataset.” Generative AI, in other words, is a moving target.
And then there’s the fact that the most sinister synthetic images and videos—the ones created as part of a scam or with the intent of deliberately confusing and misleading the public—are designed specifically to evade detection. There have been some efforts to encourage AI generators, such as OpenAI, to make intentional manipulation more difficult by embedding a watermark whenever a piece of media is produced using their software. In 2023, Joe Biden issued an executive order meant to promote that step. But “on the other side of these algorithms are real people,” Lyu said. “They become very good at hiding their traces.” Some social media companies, such as Meta, have pledged to label content as either created or modified by AI, where appropriate—and yet a recent study from researchers in Australia and South Korea found that even the best models can only successfully detect AI-generated content about two-thirds of the time.
On occasion, news accounts on social media have taken the bait. Last May, War_Monitors, an X account that posts about breaking news and politics for an audience of more than a million people, reposted a photo that claimed to depict an explosion at the Pentagon, as did a financial news account, FinancialJuice, which is followed by more than half a million. (In a reminder of the importance of true journalistic rigor, the latter saw fit to cite “Twitter sources,” not otherwise specified, as the origin of its information.) Both accounts deleted their posts once the local fire department took to social media to debunk the photo. In the meantime, the incident had been retweeted or mentioned on X nearly four thousand times—and had caused a dip in the stock market. (Another verified account posing as a CNN affiliate also posted the image, which raised questions about X’s verification process. The account was subsequently removed from the platform.)
In a world where photos and videos can’t be taken at face value, we may not be able to count on technology alone to solve problems of its own creation. “The one thing that is missing from the conversation is the education piece,” Donnell Probst, the deputy director at the National Association for Media Literacy Education (NAMLE), said. NAMLE is a nonprofit that creates resources meant to help kids learn how to evaluate the legitimacy and origin of the things they encounter online, including AI-generated images and videos. The point, Probst told me, is not that verification tools have no value. Rather, NAMLE focuses on “creating human-computer partnerships,” instead of relying on technology to “outsource our critical thinking.”
That’s a useful approach for adults, too. Consider the image of the little girl in a life vest clutching her dog—an AI-generated visual that went viral in the wake of Hurricane Helene, which struck the southeastern United States in September of 2024. It’s hard to know where the image first appeared, but it proliferated among a series of misattributed videos of other natural disasters and conspiracy theories about flooding being caused by anything from lithium deposits to the federal government to, as it happened, AI. Before long, the picture was revealed as a fake—but people kept sharing it anyway. “I don’t care,” someone posted on X. Another wrote, “It doesn’t matter.” At a certain point, the burden on journalists becomes not only verifying that images are credible, but persuading people to care about reality.
In the meantime, Lyu believes it remains important to invest in AI detection technology, even if bad actors may always manage to get around it. “What we’re trying to do here is raise the bar so that it won’t be that, with just a few clicks on the screen, you can make something that millions of people will fall for,” he said. “We want it to be harder and harder. This is not a battle we can give up.”
Saliba may be more confident in the righteous power of AI. But even she holds no illusions that software can provide a singular answer for newsrooms deciding whether or not a particular piece of media is synthetic. She sees GetReal as only one “layer of confirmation” in a broader system of verification that journalists use when working with eyewitness media. “It’s part of our toolbox,” she said. “If we can’t believe what we see and hear anymore,” she added, “I think that our society’s in grave danger.”
Has America ever needed a media defender more than now? Help us by joining CJR today.
