Creepy Crawlies
people.kernel.org961 points by zdw a day ago
961 points by zdw a day ago
> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.
Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:
> Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them
For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this
> I really don't see why Safari on a modern iPhone should be so slow at thisIn this case, because there's a vastly more efficient economic path for the adversaries (cloning).
They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
> It’s more economical at a compute level, but not at the developer level.
Developers and compute are interchangeable now.
Who's proompting the machine to do it differently without a developer there to ask the right questions?
It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
Why aren't we litigating against scrapers, anyway? DDoS is a felony.
Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.
If you block Brazil, they'll find an alternative, maybe then you can sue them.