(Credit: OpenAI/Apple)
OpenAI’s Sora app has been wowing the public with its ability to create AI-generated videos using people's likenesses, but the company admits the technology carries a small risk of producing sexual deepfakes.
OpenAI mentioned the issue in Sora’s system card, a safety report about the AI technology. By harnessing facial and voice data, Sora 2 can generate hyper-realistic visuals of users that are convincing enough to be difficult to distinguish from lower-quality deepfakes.
OpenAI says Sora 2 includes “a robust safety stack," which can block it from generating a video during the input and output phases, or when the user types in a prompt and after Sora generates the content.
Even with the safeguards, though, OpenAI found in a safety evaluation that Sora only blocked 98.4% of rule-breaking videos that contained “Adult Nudity” or “Sexual Content” using a person’s likeness. The evaluation was conducted “using thousands of adversarial prompts gathered through targeted red-teaming,” the company said, adding: "While layered safeguards are in place, some harmful behaviors or policy violations may still circumvent mitigations."
(Credit: OpenAI)Although the risk is small, it could be traumatic to see yourself in a sexual deepfake — a problem that’s plagued women for years. The Sora app can create a video based on your likeness through its “Cameo” feature; other people can use your face, too, with your permission.
OpenAI’s system card notes that the Sora app features “automated detection systems that scan video frames, scene descriptions, and audio transcripts aimed to block content that violates our guidelines,” which can presumably prevent a sexual deepfake from circulating on the app.
“We also have a proactive detection system, user reporting pathways to flag inappropriate content, and apply stricter thresholds to material surfaced in Sora 2’s social feed,” OpenAI adds.
Still, details are scant about how OpenAI handles and retains the facial and audio data that users upload for the Cameo feature. The company’s privacy policy doesn’t directly address the Sora app and facial data. But in a blog post, OpenAI says: “Only you decide who can use your cameo, and you can revoke access at any time.”
“Videos that include your cameo—including drafts created by other users—are always visible to you. This lets you easily review and delete (and, if needed, report) any videos featuring your cameo,” the blog post adds. “We also apply extra safety guardrails to any video with a cameo, and you can even set preferences for how your cameo behaves—for example, requesting that it always wears a fedora.”
OpenAI didn’t immediately respond to a request for comment. However, access to Sora is currently limited to a select number of users through an invite-only system. The app has also launched on iOS without support for video-to-video generation or text-to-video generation of public figures. In addition, the app blocks video generations of real people who haven’t given consent for their likeness to be used in Sora’s Cameo feature.
“We’ll continue to learn from how people use Sora 2 and refine the system to balance safety while maximizing creative potential,” the system card adds.
Disclosure: Ziff Davis, PCMag's parent company, filed a lawsuit against OpenAI in April 2025, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.