Apple鈥檚 soon-to-be-launched algorithm to detect images of child sexual abuse on iPhones and iPads may incorrectly flag people as being in possession of illegal images, warn researchers.
NeuralHash will be launched in the US with an update to iOS and iPadOS later this year. The tool will compare a hash 鈥� a unique string of characters created by an algorithm 鈥� of every image uploaded to the cloud with a database of hashes for known images of child sexual abuse. Matches should mean that the images are the same and so would ultimately be passed to police after a series of checks.
When NeuralHash was announced earlier this month, Apple claimed that the system will see less than one in a trillion false positives every year. This was disputed at the time by computer scientists, who said there was no way to judge it until it was launched.
Advertisement
A user on the code-sharing website GitHub now , which has been present in iOS versions 14.3 onwards despite not yet being activated, and released it online. The algorithm matches descriptions in a technical document released by Apple.
Within hours of that code being published, other users had identified pairs of visually very different images that when using it. Initially, one of these images was a photograph and one was an abstract collection of pixels generated by an AI to match the hash, but soon after they had also found examples involving real photographs.
This could lead to innocuous images having the same hash as images of child sexual abuse, and therefore generating a positive match within NeuralHash. It also conceivably opens the door to deliberate malicious attacks where carefully crafted images are sent to a user鈥檚 phone to deliberately trigger a match, although that would require that the person targeted saved the image to the cloud.
at Princeton University says the ease of finding matches, known as hash collisions, comes as 鈥渮ero surprise鈥�. The type of hash function Apple is using doesn鈥檛 have as strong properties for preventing generating images with the same hash, he says. 鈥淎pple should have been clear about those limitations, the privacy risks they create, and how it plans to mitigate those risks.鈥�
Any positive matches using NeuralHash will trigger a human double-check of the photo鈥檚 hash within Apple. If this confirms a match with the signature of a known image containing child sexual abuse, the information will be reported to the US nonprofit organisation National Center for Missing and Exploited Children. NCMEC will then pass on the details to the police to decide whether to make a legal request either to see the images or for information about the device鈥檚 owner.
鈥淭he best case scenario is that this causes extra work for Apple鈥檚 human assessment team. The worst case is that a further error leads to someone being arrested for possession of child sexual abuse images without cause,鈥� says at law firm decoded.legal.
An Apple spokesperson confirmed to New 杏吧原创 that the perceptual hashes of the sort used by NeuralHash can be tricked into believing two different images appear to be the same, but said that Apple鈥檚 system is designed to be secure despite this.
The spokesperson says that a database of hashes for known child sexual abuse images will reside on users鈥� phones, but that it is encrypted. So while a theoretical demonstration of two different images with the same hash can be made, it would be impossible for an attacker to know what hash they would have to match to trigger a false positive.
They add that at least 30 positive matches would be needed to trigger an investigation 鈥� something which child welfare charities have criticised as too high a bar 鈥� and that once triggered, a second algorithm performs another check to rule out false positives. A human review is also made before a report is filed to NCMEC, says the spokesperson.
鈥淭here are multiple lines of defence in the system and Apple鈥檚 big technical one has failed very quickly,鈥� says at Johns Hopkins University.
