Apple’s upcoming system to detect and stop child sexual abuse images over iCloud has (perhaps inevitably) sparked a wave of alarm that the same system will make mistakes or be abused for widescale censorship and surveillance.
So on Friday, Apple released more details about how it’ll ensure the technology doesn’t spiral out of control. The company did so by publishing a 14-page document, outlining the various safeguards in place for its child sexual abuse material (CSAM) detection system.
One of the big takeaways is how Apple’s detection system will only flag an iCloud Photos account for investigation if at least 30 suspected child sexual abuse images are detected.
The threshold is important, since there’s always a possibility Apple’s CSAM system will mistakenly flag an innocuous image uploaded to iCloud as child porn. The company settled on 30 images, describing it as “drastic safety margin reflecting a worst-case assumption about real-world performance.”
If you think the threshold is too high or too low, Apple adds the company may change the number after it deploys the detection system with iOS 15 later this year. “But the match threshold will never be lower than what is required to produce a one-in-one trillion false positive rate for any given account,” the company writes in the document.
It’s also important to note what happens if the system mistakenly flags an innocent iCloud Photos account as full of child porn. Apple isn’t going to automatically report the account to the National Center for Missing and Exploited Children, which works with law enforcement to stop child predators. The company’s team of human reviewers will examine the flagged pictures firsthand to confirm the imagery is child sexual abuse material.
“If and only if you meet a threshold of something on the order of 30 known child pornographic images matching, only then does Apple know anything about your account and know anything about those images, and at that point, only knows about those images, not about any of your other images,” Apple SVP Craig Federighi said in a Friday interview with The Wall Street Journal.
The other notable detail is how Apple is going to rely on not one, but at least two child safety organizations to determine the child pornography to look out for. The safety organizations will also operate in separate jurisdictions belonging to different governments. Apple created this safeguard in the event a government influences a particular child safety organization.
“Any perceptual hashes appearing in only one participating child safety organization’s database, or only in databases from multiple agencies in a single sovereign jurisdiction, are discarded by this process, and not included in the encrypted CSAM database that Apple includes in the operating system,” the company wrote in Friday's document.
Apple has also stressed the company’s detection system isn’t “scanning” photos on customer iPhones. Instead, Cupertino argues it's leveraging computer algorithms on the device to look out for hashes or digital fingerprints of known child sexual abuse images in photos uploaded to iCloud. The company never learns what the photos sent to iCloud contain unless the 30-image threshold is crossed.
But despite Apple’s attempts to defend the detection system, many IT security researchers still remain leery that the same technology could be easily abused in the future.
“While Apple is introducing the child sexual abuse detection feature only in the United States for now, it is not hard to imagine that foreign governments will be eager to use this sort of tool to monitor other aspects of their citizens’ lives—and might pressure Apple to comply,” security experts Matthew Green and Alex Stamos wrote in a New York Times op-ed on Wednesday.
If you’re not a fan, Apple says not to upload pics to iCloud Photos. The company has also pledged to never give governments access to the CSAM detection system.