What Sora AI-Generated Videos Look Like Combined with Real-World Actors and Locations

This post is largely inspired by Zach Seward’s SXSW talk. Zach currently spearheads AI initiatives at the New York Times. In his talk, he mentioned that he doesn’t have all the answers about how to use AI at the NYT, but that surveying the current state of AI-powered journalism—from the very bad to the really good—can provide valuable insights. And he’s right! So let’s apply the same approach to the doc world.

I’ll start with the “worst” or most problematic uses of AI in docs I’ve seen: we can learn a lot from these. Then I’ll share some really cool, ethical, and effective examples of genAI in docs. Without further ado…

The Worst/Most Problematic Use Cases of AI in Documentary Film

Roadrunner: A Film About Anthony Bourdain

• The offense: AI-generated voice of a real person (Bourdain) used without disclosure to the audience or official consent from the family.
• The deets: In 2021, this documentary about the beloved celebrity chef, author, and travel documentarian Anthony Bourdain premiered with very positive reviews. But then, news broke that its director, Morgan Neville, had made an AI version of Bourdain’s voice. This “fake” voice was used in the documentary for 45 seconds, and its use was not disclosed to viewers. Bourdain’s real voice was also used elsewhere in the film. Viewers had no way of distinguishing the fake voice from the real one.
• The response: Most people who heard about this were AGHAST! The New Yorker’s Helen Rosner summarized the response well in her piece about the controversy: “‘Well, this is ghoulish’; ‘This is awful’; ‘WTF?!’ people said on Twitter, where the fake Bourdain voice became a trending topic. The critic Sean Burns, who had reviewed the documentary negatively, tweeted, ‘I feel like this tells you all you need to know about the ethics of the people behind this project.’” There was, as Rosner notes, a particularly emotional response because 1) Neville had not gotten consent from Bourdain (who was deceased at the time) nor from his close-of-kin (though there is some nuance here: Neville claims there was a misunderstanding on this front… Bourdain’s ex-wife said that the topic did come up, and she does believe that the director “thought he had everyone’s blessing to go ahead.”) But it’s clear that Neville should have asked more explicitly.
• My take: Overall, this seems pretty obviously unethical / uncool. But see the “best” section for an example of a similar approach done well.

What Jennifer Did

• The offense: AI-altered images of a real person (the primary subject, Jennifer Pan) used messily, and without disclosure.
• The deets: What Jennifer Did is a Netflix true crime documentary that secured the coveted “top movie” spot on the platform for a while. The story focuses on Jennifer Pan, who was accused of conspiring to kill her parents and convicted of murder and attempted murder. When the police first started investigating Pan, a few candid pictures of her are shown. Two of those photos are strongly suspected of being generated by AI (I mean uhhh…look at her weird-AF hands in the picture below). Other pictures also have weird hands and teeth, and additional inconsistencies. Oof!

Still from “What Jennifer Did”

• The response: A lot of commentators basically said that using fake photos in a doc like this—especially without transparency or disclosure—compromised their trust in the documentary and/or in Netflix documentaries more generally.
• My take: At a time when the doc industry is struggling, I think it’s more important than ever to establish and maintain trust with audiences…and it’s a shame, to put it mildly, when this kind of AI use erodes trust. There are obviously parallels between this example and the above one, but I personally think that because the Bourdain documentary at least uses the subject’s real words, this one feels even worse.

The doc: [Redacted – Various] Film(s) using generated “b-roll/archival footage” without disclosure

• The deets: I have heard multiple anecdotal stories about a director asking their editor or archival producer to fully generate (with AI) b-roll or “archival” footage for a documentary film…with no plans to disclose that the footage was generated. One such story—the specifics of which are under NDA—was the inspiration for the founding of the Archival Producers Alliance working group and best practices for AI, which I highly recommend checking out if you are interested in this topic.
• My take: This seems pretty obviously bad! It’s lazy, and it’s not prioritizing the audience’s best interest. As discussed below, there can be a place for AI-generated b-roll, but it seems pretty obvious that it should be disclosed to audiences. It’s also possible that as AI-generated visuals become more common, a visual language around them will emerge to make them more obvious to viewers. (A parallel: reenactments in documentaries. These used to be more frequently disclosed to audiences, but now they usually aren’t—but they are generally filmed in such a way that it’s obvious to audiences that they ARE reenactments vs actual footage).

As you can see, there are some common qualities in the above “worst” examples: namely, the use of AI seems intended to cut corners; its use is undisclosed, chipping away at the audience’s trust; and, at times, it’s also just lazy, and thus, distracting.

But it’s not all bad…

Best Uses of AI in Documentary Film

Note: There are a lot of great ways AI is being used as a tool to save time in the creation of docs (ex: auto transcription; editing with transcription in Premiere; hastening the color correction process). With the below examples, I’m not focusing on these, but rather, on generative AI that dramatically impacts the audience’s experience of the film.

The Wizard of AI

• The use of AI: This doc, directed by Alan Warburton, has been called “the world’s first AI-generated documentary.” The visuals are apparently ~99% AI-generated (sometimes there are non-AI generated parts incorporated, like Clippy, below).
• The deets: It’s a visual essay in which a hoodie-wearing faceless ‘AI Collaborator’, voiced by the artist, is our incisive guide.
• My take: It’s really well-done overall. The heavy use of AI-generated visuals makes sense here because the film is about AI, and the way they are used also overtly suggests they are AI-generated to the viewer—Warburton isn’t trying to hide that he’s using AI tools here. Their use, of course, also feel appropriate to the subject, since the film is about AI, and it’s disclosed (in the credits as well as the Vimeo description). The director uses generative AI tools themselves to discuss and thoughtfully critique the aesthetic, legal, and ethical problems engendered by the wide new world of AI tools.

The Wizard of AI by Alan Warburton. Still frame from video.

Eno

• The use of AI: This documentary about Brain Eno, directed by Gary Hustwit, uses AI to edit the film so it’s different for every audience who sees it.
• The deets: Apparently, legendary musician Brian Eno was initially resistant to being featured in a doc, partly because he believes docu-bios of artists are usually “hagiographic.” That is, “until he [Eno] heard the pitch for this one.”
• My take: Just the fact that Hustwit’s AI-assisted approach convinced Eno to do a project he wouldn’t otherwise do is fascinating! I would be surprised if this approach becomes commonplace, but is an interesting one—and seems to be resonating with audiences. It got into Sundance, has a 100% on Rotten Tomatoes, and anecdotally, my friends have raved about it—including friends who have seen it more than once to experience different cuts! It will be interesting to see if and how they release the film more broadly.

The Andy Warhol Diaries

• The use of AI: Used AI-generated voice of Andy Warhol

• The deets: Technically, this is very similar to the Bourdain example above. They used an AI software to have an AI-generated voice of Warhol read some of his real words, often from his diary.
• My take: What makes this a GOOD use case is that 1) the filmmakers got permission from the Warhol estate, and 2) they were very transparent about the fact that they were using an AI-generated voice: it’s explicitly called out in the trailer, in the film, and in the credits. When it comes to AI in docs, transparency (and consent, when working with materials related to a real person—alive or dead [obviously when they’re dead, it’s consent of the family or estate]) is everything. I think we will be seeing this kind of use case in a LOT of documentary films in the near future.

Another Body

• The use of AI: AI deepfake of a subject who wanted to maintain anonymity.
• The deets: This is a doc about a college student who finds deepfake pornography of herself online. To obscure her identity, generative AI tools empowered the film’s subject to “reclaim a technology that was used to attack her and silence her” (per the Hollywood Reporter).
• My take: There are a lot of ways to mask subjects’ identities that DON’T use AI. However, I’ve personally often found them to be awkward and/or distracting. AI deepfakes might just do this better—offering subjects more peace of mind, and audiences a more realistic and compelling subject. This is an exciting area. Of course, disclosing the use of deepfake technology is key.

Still from the making of “Another Body”. COURTESY OF ANOTHER BODY/SXSW

[Various] AI as a tool to make animated scenes for docs that can’t afford animation or recreation (ex: Semafor)

• My take: I don’t actually LOVE any real-world examples of this that I’ve seen, though props to the tech news startup Semafor for early explorations in this realm. But I think this is a really rich area for docs (and one which I am personally exploring for a project). Of course, for well-funded docs, directors and producers should continue to hire “real” animators or do “real” recreations. But there are many docs—especially in this market—that simply do not have the budget to fairly fund any animation or actors.For those docs, AI can be a big unlock…especially now that AI-assisted animation has gotten so much better than when Semafor was first experimenting with it.

This isn’t an exhaustive list—but I hope it’s enlightening and inspiring for you. If you have other examples of terrible or amazing uses of AI in docs, I’d love to hear about them.

Is this your first time readingFilm Robots? Film Robots is a newsletter about film and AI by Kristina Budelis. You can subscribe here.

Follow more discoverable links on the Film Robotsnewsletter on AI & Film and on Film Robots (@filmrobots) Instagram photos and videos

Originally Appeared Here