Apple defeats liability for not scanning iCloud for CSAM
blog.ericgoldman.org279 points by speckx 7 hours ago
279 points by speckx 7 hours ago
Maybe my perception is off, but it seems like there's a huge push by the legislature and some people to do anything and everything to prevent CSAM, yet almost nothing seems to be done to prevent CSA.
For CSAM, there's all sorts of monitoring, scanning, identify capturing, etc. But it's all after abuse has taken place, and it seems that many of the people actually arrested are arrested for CSAM and not CSA. This has even extended to fictional CSAM such as AI generated stories and pictures. As an aside, if that gets extended to political speech or other non-CSAM materials that are determined to be undesirable, that's a big concern. I can imagine that a conservative state could pass a law banning all porn because they claim it could encourage illegal activities such as prostitution, rape, or CSA.
On the CSA side, you rarely hear about arrests (they happen but less than CSAM). There doesn't seem to be any real push for educating and protecting kids before it happens. Ironically, the groups doing the most to educate and implement protective strategies are the ones who have been involved in abuse scandals in the past (Churches, Scouts, etc). Even then, a lot of it is just getting clearances, which doesnt prevent people who where not caught or were first timers. Offenders get put on a list/map. This is sort of a half approach. If they are still a threat, they shouldn't be released. Yet if you comb the list and see some of the results, they don't all seem to fit with CSA. I personally know of 1 who took a leak across from a playground at 2am walking home from the bars and was put on the sex offender registry because it was within 500'.
It seems like these laws are more about peddling to the publicist and lawmakers fantasy of incrementally extreme punishment rather than taking a appropriate, data driven, and level-headed approach that actually protects kids. Otherwise they will just keep pushing ham-fisted low-hanging "fixes" like required scanning and IDs to access the internet.
Because it isn't about CSAM, IMO. You see this with plenty of social issues, notably firearms ownership. The claim is we will restrict/license/outlaw xyz for the kids, but really, a data-drive approach would have focused on different things entirely (eg additional behavioral health services for kids, suicide prevention etc)
The same technology/access used for CSAM identification can find copyrighted files, materials that don't support current government, etc. Full E2E encryption seriously raises the cost of mass surveillance, and why the US government fights it at every turn going back 30 years.
CSAM is the perfect trojan horse to undermine privacy everywhere. No politician and no company can oppose it on the grounds of protecting privacy before having the reputation ruined.
Meanwhile the real pedos are rich people vacationing in well known places.
In Sweden we're going to literally have Minority Report style pre-crime registries for potential child molesters. We already have one for potential domestic abusers. Fuck this country. I need to get out of here before it gets worse.
Say what now? What criteria do they use to scan the populace and create a pre-crime registry of any kind?
I think it's similar with other liability issues, e.g., when a company happens to "lose" customer data through a breach. They will be on the hook for not having certain audits and certifications at regular intervals. Practically never will anyone be feeling any pain due to absolute disregard for basic common sense precautions to prevent issues in the first place. So usually, the pattern is that issues that can be outsourced to insurance will be handled by compliance departments - which don't care about the actual problems, just that the fallout from them is "managed" accordingly.
That's actually a little funny. I work in compliance and we regularly work with the teams to improve processes that enhance security. I will say though, that the outsourced work that we send to consultants has the same sort of result you describe.
One is far easier to prove. If the government could continuously monitor our actions "Is this CSA?" they might very well be pushing for that, too.
>if that gets extended to
If you provide the government a platform to do So, they'll do Y if you wait long enough
> ones who have been involved in abuse scandals in the past (Churches, Scouts, etc)
And those groups still have lower rates of abuse than Schools which do a lot less training and enforcement of youth protection policies.
Children are still safer in Scouts and churches than in schools.
Just about every kid goes to school though, so people just prefer to sweep that under the rug and focus on targeting organizations they are not a part of or disagree with because they're easier to demonize and make fun of.
I work with kids, and I've had to take the Scouts and Catholic Church's youth protection training. They are both free to take online if anyone wants to check it out.
>And those groups still have lower rates of abuse than Schools
Is that in absolute terms, or per child who frequents the establishment?
And those groups still have lower rates of abuse than Schools
What evidence do you have of this? The only sources I can find that would even vaguely support your claim switch to talking about physical abuse in schools, or sexual assault committed by fellow students.
To be clear, we're discussing the sexual assault of children by adults here. We're not talking about physical abuse, nor are we talking about assault by fellow students.
Does the school work to cover it up to the same extent? Are people across the school in on it? Are the rates of abuse higher on a per capita basis?
Why are you making apologies for (Catholic) church sex abuse? Because you are a member and you took a training? Yikes.
Where did you see apologies?
I read it basically as “schools are even worse than churches and scouts yet are overlooked,” nothing apologizing for the churches.
The problem is, the things that would actually prevent CSA require things like "giving children rights", "widely mandating effective age-appropriate sex education even for young children", and "admitting that it's mostly not scary strangers doing it."
These are utterly anathema to huge chunks of society, especially American society. It's more and more clear, from the scope of the Epstein Files, just how much of American society and government has been influenced specifically to enable easy access to children by rich white men. Even beyond that, the entire right wing would instantly catch fire if we actually required effective sex education. Especially age-appropriate sex education going right down to kindergarten (yes, kindergarten: the better children that age understand what's normal and safe, the better they can communicate when someone is doing things to them that are not that...and I only stop at kindergarten because AFAIK that's the earliest mandated schooling still). Similarly, the right wing desperately wants to have absolute control over their children—treat them as property—so giving children rights that even parents have to respect will get them to oppose absolutely everything.
Plus, as other sibling comments have already noted, the real desire here is for ubiquitous surveillance. CSAM is just the excuse they use.
Somehow, even something as simple as "age-appropriate sex education going right down to kindergarten" would tend to end up "special interests education from vocal minority groups."
It is crazy people think apple isnt on the side of privacy. Are they perfect? Not even close, but compared to the rest of big tech theyre simply on another level.
Apple could easily not do this stuff and it may even be easier to not.
> It is crazy people think apple isnt on the side of privacy.
> It also ensured pressure from governments and plaintiffs, including CSAM victims, who preferred Apple’s more interventionist approaches, which Apple had voluntarily demonstrated it was willing to do.
I feel that Apple open pandora's box with the client-side scanning. It proved that it was technically feasible, and was "privacy preserving". I use scare quotes there because I don't think that political or religious dissidents would find that the same or similar technology used to discover and persecute them is "privacy preserving". And that's really the problem with Apple here. They provided a model for scanning for any kind of message or material while purportedly maintaining privacy.
> It proved that it was technically feasible, and was "privacy preserving".
Didn't their paper disproved by reversing the perceptual hashes to reveal blurred version of the images being hashed, and Apple basically said "that's fair, it's not as robust as we wanted, let's visit this later"?
If not, I'll happily stand corrected, but please share sources.
Addenda:
- Apple's original paper: https://web.archive.org/web/20210807165030/https://www.apple...
- Paper breaking the hash: https://arxiv.org/abs/2111.06628
Edit: The second one is the wrong paper. I’ll find and link the correct one tomorrow. Keeping the link for transparency.
The paper you linked doesn’t reveal blurred versions of the images being hashed. It does train a classifier to determine which of 1,000 ImageNet classes an image belongs to, which “achieved a top-1 test accuracy of 4.34%”.
Then, that’s the wrong paper. I’ll find it and link it as a reply to this comment. Probably tomorrow morning.
> to reveal blurred version of the images being hashed
Skimmed your linked paper. It seems they were able to classify hashes up to ~8% top-1 accuracy and ~30% top-10. Not exactly a blurred version, or any images at all.
So for example, they can say that you probably have images of trees, or images of buildings, but without much other data & very low accuracy.
I'd still be a lot more concerned about them simply flagging political images rather than trying to get a broad understanding of what type of photos I have
It starts at a broad understanding and ends with people permanently giving up their right and ability to keep rogue corporate governments in check.
> I feel that Apple open pandora's box with the client-side scanning.
That box has been open for years now.
Big brother is already watching what you do on your Android device.
> A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal.
https://www.nytimes.com/2022/08/21/technology/google-surveil...
This is not client side. Images just sitting on your Android phone are probably safe (although Google could push an update at any time). Your images get scanned when you back them up, send them over RCS, etc. I have even seen criminal cases originating from reverse image search - anything that touche the servers of the big tech companies, except apparently Apple, will be scanned using questionable AI and against a secret list to Protect the Children.
This is an image that he did not send off of his device to anyone except his doctor's office, yet Google reached into his private data and scanned it anyway.
Google reported him to the police based on a single false positive.
To add insult to injury, even after the police contacted Google to tell them that they had cleared him of wrongdoing, Google refused to restore access to his account.
Stop making stuff up man, the image was uploaded to Google Photos servers.
> The father uploaded photos of his son’s genitals, which were also backed up on his Google cloud, to the health care provider’s messaging system as requested.
Did he set up his device to upload his private data to Google, or did Google create an OS that automatically sent everyone's private data to their AI server for scanning without explicit consent?
Google Photos does ask you for consent when you run it.
(It is, granted, a bit pushy about it and will ask multiple times when you run it with an intrusive dialog.)
Informed consent would require Google to inform you that their AI server will scan every photo you take with your device and report you to the police if it should detect (or hallucinate) something it doesn't like.
The Pandora's box was already open and essentially no one noticed nor was there immense pushback that resulted in the features being removed. Photo scanning was already happening for both Android and Apple for the purpose of image search.
I thought the client side scanning was to protect children? If it suspects an image is bad, it blurs it and pops up a warning including a link to resources to go to for help.
Very different than trying to narc out users to the authorities.
There were two different technologies. One was client-side scanning for known CSAM, which created a huge backlash and is described here: https://educatedguesswork.org/posts/apple-csam-intro/
The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial and is described here: https://support.apple.com/en-us/105069
The controversial part is having the system enabled by default with age verification required to turn it off, and having the system impact non-Apple/Google apps. The UK for example wants Apple and Google to forcibly enable nudity blocking on all devices in the UK, and they want the system to bypass app/DRM security to scan all content visible on a device.
That is what they actually deployed. They were planning on performing client-side scanning of all images uploaded to iCloud for CSAM and reporting it to the authorities, but backpedaled after public push-back.
The original proposal was to use a visual hash designed to identify known bad CSAM images even if cropped or otherwise edited. Your computer would scan all your stuff for these images and report you to apple who would report you to the cops. This presented a number of issues.
Accidental false positives could lead to horrific outcomes up to and including oh look bob got shot by the cops for resisting.
It was possible to produce apparently matching innocuous images and then poison people's machines with them.Oops did you click on that picture of a tree have fun with the cops. Like an advanced form of swatting.
Although inspired by a desire to find CSAM Apple could be forced to scan for ANYTHING by repressive regimes including America and China.
Although initially targeting images client side scanning of messages is a pretty obvious next step. Again obvious good motivation exists and is completely justifiable who doesn't want to stop the next mass shooting or terrorist attack... and then we can basically use it to find people critical of the regime. Do remember we are presently prosecuting a political figure for a picture of sea shells and a guy in texas is rotting in prison for distributing political literature.
Is it different than telling your therapist something in confidence and then finding police waiting for you in the lobby.
Yes, in the sense that you have a legal doctor-patient privilege that binds what they can share with whom. There's not really an Apple cloud user privilege.
No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.
A better analogy is a storage locker. AFAIK police need a warrant to search "your" storage locker even though it's on someone else's property. I don't see why data in the cloud should be any different. Pre-emptively scanning everyone's data is equivalent to officers rummaging through all the storage lockers in a facility "just in case" they find something illegal.
It’s a bit more like requiring you to submit to a weapons pat down before you go to your locker I think.
> No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.
And therapists are legally mandated to report you if you told them you viewed or possessed CSAM.
No matter how or why? That seems like a terrible mandate.
They are mandated to report child abuse. It is the same story with doctors, if an abusive parent brings in a child for care they learn never to give the kid healthcare again after the doctor reports it. It is rooted in good intentions but the effect is it means abused children never get to see doctors, therapists, get a half-ass minimal homeschool instead of going to school, etc so that mandated reporters never enter the picture.
Reporting abuse the client was involved in has a compelling reason. That's different from hearing their client saw a picture of the abuse of a total stranger.
> That's different from hearing their client saw a picture of the abuse of a total stranger.
I get it, actually. It's totally possible the picture in question was not known to authorities prior. That's called due diligence to look into it.
Adults not looking into things or following up on things are how the system fails children if you read some accounts of people who were abused by their guardians. Horrifying stuff.