Wikipedia deprecates Archive.today, starts removing archive links
arstechnica.com185 points by nobody9999 4 hours ago
185 points by nobody9999 4 hours ago
Related:
Archive.today is directing a DDoS attack against my blog - https://news.ycombinator.com/item?id=46843805 - Feb 2026 (168 comments)
Ask HN: Weird archive.today behavior? - https://news.ycombinator.com/item?id=46624740 - Jan 2026 (69 comments)
I don't see the point in doxing anyone, especially those providing a useful service for the average internet user. Just because you can put some info together, it doesn't mean you should. With this said, I also disagree with turning everyone that uses archive[.]today into a botnet that DDoS sites. Changing the content of archived pages also raises questions about the authenticity of what we're reading. The site behaves as if it was infected by some malware and the archived pages can't be trusted. I can see why Wikipedia made this decision. It's also kind of ironic that a site whose whole premise is to preserve pages forever, whether the people involved like it or not, is seeking to take down another site because they are involved and don't like it. Live by the sword, etc. Did they actually run the DDoS via a script or was this a case of inserting a link and many users clicked it? They are substantially different IMO https://news.ycombinator.com/item?id=46624740 has the earliest writeup that I know of. It was running it via a script and intentionally using cache busting techniques to try to increase load on the hosted wordpress infrastructure. > It was running It still is, uBlocks default lists are killing the script now but if it's allowed to load then it still tries to hammer the other blog. Ah good to know. My pi-hole actually was blocking the blog itself since the ublock site list made its way into one of the blocklists I use. But I've been just avoiding links as much as possible because I didn't want to contribute. Given the site is hosted on wordpress.com, who don't charge for bandwidth, it seems to have been completely ineffective. The speculation that I saw was that they'd try to get Wordpress.com to boot him off for being a burden on the overall infrastructure. Thank you this is exactly the information I was looking for. "You found the smoking gun!" they silently ran the DDoS script on their captcha page (which is frequently shown to visitors, even when simply viewing and not archiving a new page) > Changing the content of archived pages also raises questions about the authenticity of what we're reading. This is absolutely the buried lede of this whole saga, and needs to be the focus of conversation in the coming age. There is an post describing the possibility of an organised campaign against archive.today [1] https://algustionesa.com/the-takedown-campaign-against-archi... How does the tech behind archive.today work in detail? Is there any information out there that goes beyond the Google AI search reply or this HN thread [2]? [1] https://algustionesa.com/the-takedown-campaign-against-archi...
[2] https://news.ycombinator.com/item?id=42816427 It seems a lot of people havent heard of it, but I think its worth plugging https://perma.cc/ which is really the appropriate tool for something like Wikipedia to be using to archive pages. It costs money beyond 10 links, which means either a paid subscription or institutional affiliation. This is problematic for an encyclopedia anyone can edit, like Wikipedia. Wikimedia could pay, they have an endowment of ~$144M [1] (as of June 30, 2024). Perma.cc has Archive.org and Cloudflare as supporting partners, and their mission is aligned with Wikimedia [2]. It is a natural complementary fit in the preservation ecosystem. You have to pay for DOIs too, for comparison [3] (starting at $275/year and $1/identifier [4] [5]). With all of this context shared, the Internet Archive is likely meeting this need without issue, to the best of my knowledge. [1] https://meta.wikimedia.org/wiki/Wikimedia_Endowment [2] https://perma.cc/about ("Perma.cc was built by Harvard’s Library Innovation Lab and is backed by the power of libraries. We’re both in the forever business: libraries already look after physical and digital materials — now we can do the same for links.") [3] https://community.crossref.org/t/how-to-get-doi-for-our-jour... [4] https://www.crossref.org/fees/#annual-membership-fees [5] https://www.crossref.org/fees/#content-registration-fees (no affiliation with any entity in scope for this thread) If the WMF had a dollar for every proposal to spend Endowment-derived funds, their Endowment would double and they could hire one additional grant-writer If the endowment is invested so that it brings very conservative 3% a year, it means that it brings $4.32M a year. By doubling that, rather many grant writers could be hired. There are dozen of commercial/enterprise solutions: https://www.g2.com/products/pagefreezer/competitors/alternat... also the oldest of that kind and rarely mention free https://www.freezepage.com Does Wikipedia really need to outsource this? They already do basically everything else in-house, even running their own CDN on bare metal, I'm sure they could spin up an archiver which could be implicitly trusted. Bypassing paywalls would be playing with fire though. Archive.org is the archiver, rotted links are replaced by Archive.org links with a bot. Yeah for historical links it makes sense to fall back on IAs existing archives, but going forward Wikipedia could take their own snapshots of cited pages and substitute them in if/when the original rots. It would be more reliable than hoping IA grabbed it. Not opposed, Wikimedia tech folks are very accessible in my experience, ask them to make a GET or POST to https://web.archive.org/save whenever a link is added via the Wiki editing mechanism. Easy peasy. Example CLI tools are https://github.com/palewire/savepagenow and https://github.com/akamhy/waybackpy Shortcut is to consume the Wikimedia changelog firehose and make these http requests yourself, performing a CDX lookup request to see if a recent snapshot was already taken before issuing a capture request (to be polite to the capture worker queue). This already happens. Every link added to Wikipedia is automatically archived on the wayback machine. I didn't know you can just ask IA to grab a page before their crawler gets to it. In that case yeah it would make sense for Wikipedia to ping them automatically. Why wouldn't Wikipedia just capture and host this themselves? Surely it makes more sense to DIY than to rely on a third party. Why would they need to own the archive at all? The archive.org infrastructure is built to do this work already. It's outside of WMF's remit to internally archive all of the data it has links to. Spammers and pirates just got super excited at that plan! There are various systems in place to defend against them, I recommend against this, poor form against a public good is not welcome. I noticed last year that some archived pages are getting altered. Every Reddit archived page used to have a Reddit username in the top right, but then it disappeared. "Fair enough," I thought. "They want to hide their Reddit username now." The problem is, they did it retroactively too, removing the username from past captures. You can see on old Reddit captures where the normal archived page has no username, but when you switch the tab to the Screenshot of the archive it is still there. The screenshot is the original capture and the username has now been removed for the normal webpage version. When I noticed it, it seemed like such a minor change, but with these latest revelations, it doesn't seem so minor anymore. Archive.is is now publishing really weird posts on their Tumblr blog, related to the whole thing https://archive-is.tumblr.com/post/806832066465497088/ladies... https://archive-is.tumblr.com/post/807584470961111040/it-see... Kinda off-topic, but has anyone figured out how archive.today manages to bypass paywalls so reliably? I've seen people claiming that they have a bunch of paid accounts that they use to fetch the pages, which is, of course, ridiculous. I figured that they have found an (automated) way to imitate Googlebot really well. > I figured that they have found an (automated) way to imitate Googlebot really well. If a site (or the WAF in front of it) knows what it's doing then you'll never be able to pass as Googlebot, period, because the canonical verification method is a DNS lookup dance which can only succeed if the request came from one of Googlebots dedicated IP addresses. Bingbot is the same. There are ways to work around this. I've just tested this: I've used the URL inspection tool of Google Search Console to fetch a URL from my website, which I've configured to redirect to a paywalled news article. Turns out the crawler follows that redirect and gives me the full source code of the redirected web site, without any paywall. That's maybe a bit insane to automate at the scale of archive.today, but I figure they do something along the lines of this. It's a perfect imitation of Googlebot because it is literally Googlebot. I'd file that under "doesn't know what they're doing" because the search console uses a totally different user-agent (Google-InspectionTool) and the site is blindly treating it the same as Googlebot :P Presumably they are just matching on *Google* and calling it a day. Sure, but maybe there are other ways to control Googlebot in a similar fashion. Maybe even with a pristine looking User-Agent header. > which I've configured to redirect to a paywalled news article. Which specific site with a paywall? > I've seen people claiming that they have a bunch of paid accounts that they use to fetch the pages, which is, of course, ridiculous. The curious part is that they allow web scraping arbitrary pages on demand. So if a publisher could put in a lot of arbitrary requests to archive their own pages and see them all coming from a single account or small subset of accounts. I hope they haven't been stealing cookies from actual users through a botnet or something. Exactly. If I was an admin of a popular news website I would try to archive some articles and look at the access logs in the backend. This cannot be too hard to figure out. > which is, of course, ridiculous. Why? in the world of web scrapping this is pretty common. Because it works too reliably. Imagine what that would entail. Managing thousands of accounts. You would need to ensure to strip the account details form archived peages perfectly. Every time the website changes its code even slightly you are at risk of losing one of your accounts. It would constantly break and would be an absolute nightmare to maintain. I've personally never encountered such a failure on a paywalled news article. archive.today managed to give me a non-paywalled clean version every single time. Maybe they use accounts for some special sites. But there is definetly some automated generic magic happening that manages to bypass paywalls of news outlets. Probably something Googlebot related, because those websites usually give Google their news pages without a paywall, probably for SEO reasons. Using two or more accounts could help you automatically strip account details. I’m an outsider with experience building crawlers. You can get pretty far with residential proxies and browser fingerprint optimization. Most of the b-tier publishers use RBC and heuristics that can be “worked around” with moderate effort. It’s not reliable, in the sense that there are many paywalled sites that it’s unable to archive. But it is reliable in the sense that if it works for a site, then it usually never fails. Why not show both? Wikipedia could display archive links alongside original sources, clearly labeled so readers know which is which. This preserves access when originals disappear while keeping the primary source as the main reference. They generally do. Random example, citation 349 on the page of George Washington: ""A Brief History of GW"[link]. GW Libraries. Archived[link] from the original on September 14, 2019. Retrieved August 19, 2019." This will always be done unless the original url is marked as dead or similar. "Non-paywalled" ad-free link to archive: https://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment... > If you want to pretend this never happened – delete your old article and post the new one you have promised. And I will not write “an OSINT investigation” on your Nazi grandfather From hero to a Kremlin troll in five seconds. > an analysis of existing links has shown that most of its uses can be replaced. Oh? Do tell! I would be suprised if archive.today had something that was not in the wayback machine Archive.today has just about everything the archived site doesn't want archived. Archive.org doesn't, because it lets sites delete archives. Wayback machine removes archives upon request, so there’s definitely stuff they don’t make publicly available (they may still have it). Trying to search the Wayback machine almost always gives me their made-up 498 error, and when I do get a result the interface for scrolling through dates is janky at best. Accounts to bypass paywalls? The audacity to do it? Oh yeah those where a thing. As a public organization they can't really do that. I personally just don't use websites that paywall important information. >> an analysis of existing links has shown that most of its uses can be replaced. >Oh? Do tell! They do. In the very next paragraph in fact: Well, that's an odd idea of "can be replaced". > editors can remove Archive.today links when the original source is still online and has identical content Hopeless. Just begs for alteration. > a different archive site, like the Internet Archive, Hopeless. It allows archive tampering by the page's own JS and archive deletion by the domain owner. > Ghostarchive, or Megalodon Hopeless. Coverage is insignificant. > archive.today Hopeless. Caught tampering the archive. The whole situation is not great. I just quoted the very next paragraph after the sentence you quoted and asked for clarification. I did so. You're welcome. As for the rest, take it up with Jimmy Wiles, not me. Previously Related: Archive.today is directing a DDoS attack against my blog? I know I'm arguing with a bot that nobody monitors, but it's already in the fucking post. Anecdotally I generally see archive.is/archive.today links floating around "stochastic terrorist" sites and other hate cults. >In emails sent to Patokallio after the DDoS began, “Nora” from Archive.today threatened to create a public association between Patokallio’s name and AI porn and to create a gay dating app with Patokallio’s name. Oh good. That's definitely a reasonable thing to do or think. The raw sociopathy of some people. Getting doxxed isn't good, but this response is unhinged. I mean, the admin of archive.today might face jail time if deanonymised, kind of understandable he's nervous. Meanwhile for Patokallio it's just curiosity and clicks It's a reminder how fragile and tenuous are the connections between our browser/client outlays, our societal perceptions of online norms, and our laws. We live at a moment where it's trivially easy to frame possession of an unsavory (or even illegal) number on another person's storage media, without that person even realizing (and possibly, with some WebRTC craftiness and social engineering, even get them to pass on the taboo payload to others). That was private negotiations, btw, not public statements. In response to J.P's blog already framed AT as project grown from a carding forum + pushed his speculations onto ArsTechnica, whose parent company just destroyed 12ft and is on to a new victim. The story is full of untold conflicts of interests covered with soap opera around DDoS. Why does it matter it was a private communications? It’s still a threat isn’t it? Can you elaborate on your point? The fight is not about where it is shown and not about what, not about "links in Wikipedia", but about whether News Inc will be able to kill AT, as they did with 12FT. What is News Inc? Are they a funder of Wikipedia(I think Wikipedia didn’t have a parent company so they’re not owners)? They are owner of ArsTechnica which wrote 3rd (or 4th?) article on AT in a row painting it in certain colors. The article about FBI subpoena that pulled J.P's speculations out of the closet was also in ArsTechnica and by the same author, and that same article explicitly mentioned how they are happy with 12ft down At this point Archive.today provides a better service (all things considered) compared to Wikipedia, at least when it comes to current affairs. So toward the end of last year, the FBI was after archive.today, presumably either for keeping track of things the current administration doesn't want tracked, or maybe for the paywall thing (on behalf of rich donors/IP owners). https://gizmodo.com/the-fbi-is-trying-to-unmask-the-registra... That effort appears to have gone nowhere, so now suddenly archive.today commits reputational suicide? I don't suppose someone could look deeper into this please? The archive.today operator claims on his blog that this was nothing major: https://lj.rossia.org/users/archive_today/ > Regarding the FBI’s request, my understanding is that they were seeking some form of offline action from us — anything from a witness statement (“Yes, this page was saved at such-and-such a time, and no one has accessed or modified it since”) to operational work involving a specific group of users. These users are not necessarily associates of Epstein; among our users who are particularly wary of the FBI, there are also less frequently mentioned groups, such as environmental activists or right-to-repair advocates. > Since no one was physically present in the United States at that time, however, the matter did not progress further. > You already know who turned this request into a full-blown panic about “the FBI accusing the archive and preparing to confiscate everything.” Not sure who he's talking about there. Anyone has a short summary as to who and why Archive.today acted via DDos? Isn't that something done by malicious actors? Or did others misuse Archive.today? I will no longer donate to Wikipedia as long as this is policy. Why? The decision seems reasonable at first sight. Second sight is advisable in such cases. Fact is, archives are essential to WP integrity and there's no credible alternative to this one. I see WP is not proposing to run its own. Wouldn't it be precisely because archives are important that using something known to modify the contents would be avoided? What exactly is credible about archive.today if they are willing to change the archive to meet some desire of the leadership? That's not credible in the least. The operators() of archive.today (and the other domains) are doing shadey things and the links are not working so why keep the site around as for example Internet archives waybackmachine works as alternative to it. What archive.today links are not working? > Internet archives wayback machine works as alternative to it. It is appalling insecure. It lets archives be altered by page JS and deleted by the page domain owner. > Fact is, archives are essential to WP integrity and there's no credible alternative to this one. Yes, they are essentional, and that was the main reason for not blacklisting Archive.today. But Archive.today has shown they do not actually provide such a service: > “If this is true it essentially forces our hand, archive.today would have to go,” another editor replied. “The argument for allowing it has been verifiability, but that of course rests upon the fact the archives are accurate, and the counter to people saying the website cannot be trusted for that has been that there is no record of archived websites themselves being tampered with. If that is no longer the case then the stated reason for the website being reliable for accurate snapshots of sources would no longer be valid.” How can you trust that the page that Archive.today serves you is an actual archive at this point? > If ... If ... Oh dear. > How can you trust that the page that Archive.today serves you is an actual archive at this point? Because no-one shown evidence that it isn't. The quote uses ifs because it was written before this was verified, but the Wikipedia thread in question has links to evidence of tampering occurring. Did you not read the article? They not only directed a DDOS against a blogger who crossed them, but altered their own archived snapshots to amplify a smear against them. That completely destroys their trustworthiness and credibility as a source of truth.
celsoazevedo - 3 hours ago
jsheard - 2 hours ago
ddtaylor - 2 hours ago
dunder_cat - 2 hours ago
jsheard - 2 hours ago
dunder_cat - 2 hours ago
RobotToaster - an hour ago
Hamuko - 23 minutes ago
ddtaylor - 2 hours ago
hexagonwin - 2 hours ago
jMyles - 2 hours ago
wuschel - 33 minutes ago
basch - 2 hours ago
ronsor - 2 hours ago
toomuchtodo - 2 hours ago
RupertSalt - 43 minutes ago
nine_k - 32 minutes ago
ouhamouch - 2 hours ago
jsheard - 2 hours ago
toomuchtodo - 2 hours ago
jsheard - 2 hours ago
toomuchtodo - 2 hours ago
Gander5739 - an hour ago
jsheard - an hour ago
ferngodfather - an hour ago
huslage - an hour ago
RupertSalt - an hour ago
toomuchtodo - an hour ago
ChocMontePy - 42 minutes ago
karel-3d - 8 minutes ago
xurukefi - an hour ago
jsheard - an hour ago
xurukefi - 36 minutes ago
jsheard - 34 minutes ago
xurukefi - 19 minutes ago
Aurornis - 21 minutes ago
Aurornis - an hour ago
xurukefi - an hour ago
elzbardico - an hour ago
xurukefi - an hour ago
mikkupikku - 40 minutes ago
tonymet - an hour ago
layer8 - an hour ago
xurukefi - an hour ago
tl2do - 2 hours ago
ranger207 - 2 hours ago
Gander5739 - an hour ago
RupertSalt - 2 hours ago
anilakar - an hour ago
chrisjj - 4 hours ago
that_lurker - 2 hours ago
chrisjj - 2 hours ago
bombcar - 2 hours ago
zahlman - an hour ago
ribosometronome - 2 hours ago
that_lurker - 2 hours ago
nobody9999 - 4 hours ago
The guidance says editors can remove Archive.today links when the original
source is still online and has identical content; replace the archive link so
it points to a different archive site, like the Internet Archive,
Ghostarchive, or Megalodon; or “change the original source to something that
doesn’t need an archive (e.g., a source that was printed on paper)
chrisjj - 3 hours ago
Kim_Bruning - 3 hours ago
nobody9999 - 3 hours ago
ChrisArchitect - 2 hours ago
input_sh - an hour ago
casey2 - 44 minutes ago
mrguyorama - 3 hours ago
oytis - an hour ago
jMyles - 2 hours ago
ouhamouch - 3 hours ago
MBCook - an hour ago
Yossarrian22 - 2 hours ago
ouhamouch - 2 hours ago
Yossarrian22 - 2 hours ago
ouhamouch - 2 hours ago
paganel - 2 hours ago
rdiddly - an hour ago
ndiddy - 40 minutes ago
shevy-java - 2 hours ago
alsetmusic - 4 hours ago
jraph - 3 hours ago
chrisjj - 3 hours ago
mook - 3 hours ago
huslage - 44 minutes ago
that_lurker - 2 hours ago
chrisjj - 2 hours ago
throw0101a - 2 hours ago
chrisjj - an hour ago
rufo - an hour ago
Jordan-117 - an hour ago