A year of fighting scrapers on my 1.5 million-page website
patronview.com399 points by petercooper 14 hours ago
399 points by petercooper 14 hours ago
The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
No; in this case you are not a user, you are a bot user.
I don't disagree, but there is a sliding scale here.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.
No, there isn't, it's a bot. Justifying it with a strawman case study doesn't change it.
This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"
No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..
You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce
To be fair, I do not generally support wholesale banning of scrapers and non-browser clients, since this tends to have lots of ugly side effects.
But regardless, I'm interpreting your last comment as saying that you should be entitled to behave as you please because you are a paying customer. If that's the case, then you are in for a rude awakening one day
I think the implicit assumption is that if you explained to the owner of the website what the scraper was doing, they would be more than happy to allow it.
The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM.
I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer.
I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.
I think he's saying that his use of the site, via the LLM, is consistent with what the site owner-operator likely desires.
To give a concrete example: I am a physician and have a site for my private clinic. I am not only unopposed to being scraped by bots/LLMs, I prefer it. The only value I gain from users being on my site is referral to my practice, and that same information being surfaced by search engines and LLMs just means I'm getting in front of more potential patients.
I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.
> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you.
The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.
I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.
One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.
If this is the case, they probably outright do not want you as a customer.
Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.
The actual on-sale item is potentially losing money for the business or at break-even.
------
A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.
> If this is the case, they probably outright do not want you as a customer.
Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount.
When I built this a few years ago, I had to write a lot of code to make all this work.
Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion"
I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!).
Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale.
Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.
The tradeoff here is a classic false-positive vs false-negative issue. If the cost of the bad-bot false positive (which blocks your bot out) is lower than the cost of the bad-bot false negatives (which allow bad bots in), then it is still a good tradeoff, if a suboptimal situation.
The fundamental issue here is the way we use the internet is changing… and what we are seeing is some friction as that works its way through.
I have a business where I clearly don’t want any bots. A retailer, maybe they do.
How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?
Not a rhetorical question, because most companies would welcome that interest from customers.
> How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?
That's the thing; they can't, really.
Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint.
In basically every cat/mouse game defense is forever one step behind...
you know what a browser is called by the web site? check the header that tells the version.
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
Yeah it's sort of weird that people are so used to using their computer to interact with other computers across the internet that the whole thing has become invisible to them: They completely forget that they're using a computer at all. Along come "bots" and suddenly everyone remembers that the whole thing involves computers. Driving the computer by hand for hours was never efficient, but is marginally entertaining I guess.
This reminds me of all the confused comments that pop up whenever someone calls plain HTML with forms and links a "REST API." "But APIs are for programs! People visiting websites aren't programs."
Tangential but still related to that story, but the battle for the meaning of "REST" is already lost.
Saying an API follows REST but doesn't implement HATEOAS (part of its definition), is like saying a database follows ACID but doesn't implement Atomicity. Or like saying a software is Open Source but doesn't allow use for commercial purposes.
Yet somehow for one of those 3 cases it became accepted to misuse the term that way ("[original meaning] but without this one constraint"), but if someone misuses the term for the other two cases then everyone suddenly cares and insist that people should use a different term to describe their thing.
I fancy the approach myself... but, much like the grand ideas of OSI in networking, Tim Burners Lee's idea of the Web was never what was going to be used and a lot of the original terms/concepts just plain don't make sense for the Web that exists. It was, originally, a true document system, nothing more. It wasn't until many changes from the original idea that it started to be what people wanted & actually use today. Nor was it even part of Tim Burners-Lee's original concept that 99% of Web traffic would be something other than users viewing & editing HyperText documents.
The User Agent would get a HyperText document from a Web server, render it, and show it to the user. You could parse HTML, display images, and follow links - with the assumption the user would want to edit them like any other document. There was no encryption. There was no scripting for dynamic content. Styling was done via semantic markup. There wasn't even any POST when the User Agent term was initially applied. Everything was supposed to be like one big distributed document store instead of centralized hosting on few servers for many client devices.
All that is to say: Regardless what I think sounds nice myself, if the web was just the original vision & definitions instead of everything it changed and grew to become we wouldn't even be using it to have this conversation.
Interestingly, people seem glad to have telephony companies filter bots even though the original idea of the telephone did not call out robocallers as invalid clients. Probably because, unlike the modern Web, the average person gets a chance to see what it means to allow anything inbound even though the original concept never considered robocallers.
I was on the internet before HTML and long before Mosaic...
I agree with one aspect of the narrative you're saying, his idea was held as being for «documents». To be sure, document was more abstract than document on a Mac.
Either way, I'm very sad it mostly turned into animated brochures instead.
But the reason styles were separate, the reason everyone talked about semantics, was always about the universal knowledge DAG and enabling machines to help us navigate and retrieve it. It's why so many of the early agents referenced navigation (Spyglass, Navigator) or the interconnectedness of things (Mosaic).
So, ok, big document “store”, but that's a lifeless noun. The information you are looking for, the dots connected for you surfacing what you didn't know, that's a living growing thing, an agentic retrieval savvy knowledge engine, distributed for curation and resilience.
The librarian was always to be a machine or many, on our behalf.
> it mostly turned into animated brochures instead
Most pre-LLM systems also blocked badly behaved user agents... And right now, most LLM-suppliers, are badly behaved agents.
It's not a simple semantic distinction. The mechanism matters because the monetization is built around it, and the monetization (as we've built it) only works because there's a human on the other end.
Can we build other monetization? Maybe. There's certainly proposals. It's real hard when there's too many layers between the user and the output though. I suspect the solutions will be worse than what we have now. For now the answer is "just paywall", but given your invocation of "corpo" here I suspect that's not an outcome you'd be too keen on ;)
> for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic
> I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic
There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too.
The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also utility rate info all integrated into a SPOG that is then displayed on some ambient screen (usually a hacked tablet or other cheap ARM device)
It's an interesting example. People did very similar things during the "desktop shell modification" scene in the early 2000s, by far the most common thing people modified their desktop shell to do was display a bunch of charts and graphs of semi-random information, usually about their local computer.
It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your computer currently has, as if that even meant anything.
So, after a few steps, that leads to the question "what is useful information? How do you define that?"
A basic attempt at it might be: useful information is something that causes you to make or change a previous decision. Take an action you wouldn't have otherwise done.
If you're living in the american southwest, and it's between the months of like, march and, dunno, septemeber, you probably don't need to know the current outside temperature to decide if you're wearing shorts or a jacket when you get up in the morning (you're wearing shorts because it's always hot)
Theoretically there could be some kind of "extreme" weather, aka rain, that causes you to do something different, bring an umbrella, but it's pretty difficult to not notice it currently raining, even if you're inside a house.
So what other information do you want? There's all sorts of stuff that's considered "news", but I'd argue that at least 98% of that is basically just entertainment. We'd need some kind of article or perhaps even a book to hammer out all the nuances involved, but seeing a headline in the morning that trump has caved in to iran again or killed some more random people on a boat, while sort of important to know, doesn't really change what you're going to be doing that day.
Knowing ICE is trying to do some domestic terrorism near where you might be could be useful information, but there's not a real good way to track that.
So yeah, what kind of information is actually useful to have in the morning? Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherwords, but what else? That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
> Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherword
So ... most people.
> That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
Other than the ones you just enumerated that most people are likely to use, I need to know - using external sources:
- When to expect the garbage truck (do I put the recycle out tonight, too?)
- Is it a school holiday?
- When it's cheapest for me to charge the car up; if there are any "grid stress, please turn non-essentials off or we're getting rolling blackouts" alerts.
- Same goes for nearby fires or other issues that are going to change the "when do you open/close the windows and switch to AC" math.
- If it's likely to rain in the next few days so I change what plants get watered / when
- Packages that I am likely to get in the post today
- Weather (for display, not just for use in the automations mentioned above)
- Where my power provider is currently getting its power from (so I can calculate estimated CO2 emissions, if any)
And that's about 1/2, probably. Can't be bothered to go spend 20 min pouring through my HA config to build the full list.
I also stream the npr newscast as an MP3 as part of the "good morning, while you wait for coffee..." automations, if that counts for anything.
> you know what a browser is called by the web site? check the header that tells the version.
You mean that key whose only legitimate values also call every browser Mozilla? Arguing from the dictionary is so tedious. I'm going to scrape your site, but the idea that you have some obligation to like it because of the definition of the word "agent" is profoundly stupid.
A lot of people live off ads. I block ads. People who live off ads aren't obligated to like it - and are free to take any means to block it without moral hazard.
> not user, an agent on behalf of the user.
Identified by the user agent header. Which most bots fake or leave out to increase the chance of getting where they are not wanted.
Your bot is a good bot? Great, let us know when you've dealt with all the bad bots and we'll open the doors to the remaining (good) bots again.
This website refers to browsers as browsers. Just because a header in the underlying protocol is named a certain way, that doesn't necessarily reflect what the humans intend.
[flagged]
So instead of adding of addressing his point about User Agents, you decide its better to make a low level bullshit comment about the very last sentence to stand up for... checks notes... Corporations and dismiss everything else. Top notch quality content that for sure added to the conversation.
The best way to read the information on the internet today is via a LLM.
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
This is such a grim thing to read.
I disagree. I personally consider these my biggest problems with the Web:
- Bias, specifically commercial bias
- Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info
- Scams/SEO/etc...
Google "How to make ham fried rice"; you'll get 10,000 articles, most pretty good recipes. But they're all different; most of them are just bait for ads. And most of them, the 10-line recipe is hidden between 50 useless paragraphs about how serving food is life's most important goal.
Now, ask an LLM to search for it, find the best combination, and list only the recipes. You get a perfect, 10-line recipe that doesn't waste your time.
I personally prefer to read a couple of recipes to understand the context, what people have tried, what they thought of it. Only when I have a good general idea of the recipe I choose the best one I found based on difficulty and the ingredients I have. I then do the plate, often not following entirely the recipe I chose, and after that and eating it I write down the name of the plate, the recipe link, the date, and a comment about what I did and how it went. Cooking is a lot more than executing a list of steps like a robot.
LLMs are not unbiased. Nothing is unbiased.
Yeah I had to laugh at that. LLM's solve commercial bias? Coming from seemingly multiple the most valuable companies of all time?
And if you're lucky, it won't include rocks as an ingredient.
They're an important part of your diet if you happen to have a gizzard.
I've also been known to include some ground up rocks in my meals. I'm pretty picky though about them, I mostly just want a couple specific varieties.
Why even use a LLM, just buy a cookbook and you can even waste less time and know the information is valid. And again if your logic is consistent, google routes your search to an LLM automatically and renders the reply - that will save as much time and do the same thing since LLMs are so good at parsing and presenting information.
So why should I pick an LLM over google's inbuilt LLM or an actual cookbook on the desk? What wasted time is the LLM saving?
Well agreed, but it's not new, the web has been actively made user hostile by every website owner and their "273 partners".
Maybe, but it's reality. Not too long ago the rule would be "if it doesn't show up in google, it doesn't exist". Today agents+LLMs are better browsers than Chrome/Safari/Firefox/... If a webpage does not show up in an LLM it may as well not exist.
How is that grim? It's the dream of the Semantic Web coming true, just by different means than planed.
Because the economic, social, and climate impacts are at best uncertain and at worst devastating to the majority of human beings on the planet. It's always surprising to me that people don't intuitively separate the usefulness of AI from its risks. It feels like everyone has to be a doomer or a booster.
Why do you feel that ?
Because AI cannot be trusted as a reliable source of information. Information is more trustworthy when it comes from from primary sources on the open web.
Why not ? Can google be trusted ? Or facebook ? How do people have been accessing the internet for the last decade you reckon and how's this any different ?
Google (as it originally) existed wasn't a source of information, it was an *index* of it. You were trusting google as a source for {search_query}, you were trusting the links it gave as a source (based on your own evaluation). LLMs are fundamentally different, because you are trusting the software to actually generate the information in a truthful way.
If you use google (sans AI), you're putting some trust in their page ranking algorithm. If you use it with AI, you're trusting the same algorithm (since that's how the model gets it's sources), but then you're trusting the model to evaluate the sources for credibility and extract the information you actually want.
Pagerank is just a popularity contest, doesn't say anything about trust. AI probably does a better job determining trustworthyness than pagerank.