"So if you have a bunch of false positives, your front door is getting broken down, with no opportunity for a human to realize the problem and intervene."
Well, technically speaking, once enough security vouchers have been submitted and reached the threshold as stated, a report will be sent to an Apple employee (somewhere). The vouchers will then, combined, be decryptable and contain grayscale low-res versions of the original image for confirmation, in which case the NCEMC (National Center for Exploited and Missing Children) will be alerted and law enforcement.
I'm just pointing out that your door being broke down is by multiple false flags and a human will get a chance to "realize the problem and intervene" before it goes to the FBI or whatever. Not saying I like this system, just making a nitpick on your criticism. I don't really know how else you could have a human intervene without "breaking down your door."
> ...be decryptable and contain grayscale low-res versions of the original image for confirmation...
But what happens when the false positives are erroneously confirmed as legitimate CSAM? What's the system in place for removing all the security vouchers on someone's account because the vision system flagged a bunch of false positives? What's the process for unfucking a person's life because the employee confirming the CSAM was in a bad mood that day? Is Apple going to pay the legal bills of someone they effectively SWATed?
For the ones of people this system might actually catch with legitimate CSAM there will be at least as many false positives slipping through to ruinous consequences. Law enforcement shouldn't be trusted as a backstop against abuse because LEOs and DAs are incentivized for "good numbers" and "results", not for actually meting out justice. If someone flagged by false positives gets to the stage of law enforcement being involved their lives will be ruined.
> "For the ones of people this system might actually catch with legitimate CSAM there will be at least as many false positives slipping through to ruinous consequences."
Apple claims without math that it's a 1-in-a-trillion chance that your account will be incorrectly flagged.
And even then... what happens? Does the FBI actually smash down your door with a SWAT team for CSAM? In most cases, No. They go and arrest you the normal way. Unless you had a ludicrous amount of false flags (in the thousands) showing you were a dealer and had a previous history.
Well, technically speaking, once enough security vouchers have been submitted and reached the threshold as stated, a report will be sent to an Apple employee (somewhere). The vouchers will then, combined, be decryptable and contain grayscale low-res versions of the original image for confirmation, in which case the NCEMC (National Center for Exploited and Missing Children) will be alerted and law enforcement.
I'm just pointing out that your door being broke down is by multiple false flags and a human will get a chance to "realize the problem and intervene" before it goes to the FBI or whatever. Not saying I like this system, just making a nitpick on your criticism. I don't really know how else you could have a human intervene without "breaking down your door."