Wikipedia in read-only mode following mass admin account compromise
wikimediastatus.net799 points by greyface- 7 hours ago
799 points by greyface- 7 hours ago
https://wikipediocracy.com/forum/viewtopic.php?f=8&t=14555
https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(techni...
https://old.reddit.com/r/wikipedia/comments/1rllcdg/megathre...
See the public phab ticket: https://phabricator.wikimedia.org/T419143 In short, a Wikimedia Foundation account was doing some sort of test which involved loading a large number of user scripts. They decided to just start loading random user scripts, instead of creating some just for this test. The user who ran this test is a Staff Security Engineer at WMF, and naturally they decided to do this test under their highly-privileged Wikimedia Foundation staff account, which has permissions to edit the global CSS and JS that runs on every page. One of those random scripts was a 2 year old malicious script from ruwiki. This script injects itself in the global Javascript on every page, and then in the userscripts of any user that runs into it, so it started spreading and doing damage really fast. This triggered tons of alerts, until the decision was made to turn the Wiki read-only. This is a pretty egregious failure for a staff security engineer Pretty much the definition of a “career limiting event” It's either a a Career Limiting Event, or a Career Learning event. In the case of a Learning event, you keep your job, and take the time to make the environment more resilient to this kind of issue. In the case of a Limiting event, you lose your job, and get hired somewhere else for significantly better pay, and make the new environment more resilient to this kind of issue. Hopefully the Wikimedia foundation is the former. Nobody is going to know who did this, so probably not career limiting in any major way. They'll be fine, recruiters don't look this stuff up and generally background checks only care about illegal shit. Didn't realise this was some historic evil script and not some active attacker who could change tack at any moment. That makes the fix pretty easy. Write a regex to detect the evil script, and revert every page to a historic version without the script. Letting ancient evil code run? Have we learned nothing from A Fire Upon the Deep?! "It was really just humans playing with an old library. It should be safe, using their own automation, clean and benign. This library wasn't a living creature, or even possessed of automation (which here might mean something more, far more, than human)." Link to the Prologue of Fire Upon the Deep: https://www.baen.com/Chapters/-0812515285/A_Fire_Upon_the_De... It's very short and from one of my favorite books. Increasingly relevant. I've only just heard of it. But, I already knew to not run random scripts under a privileged account. And thank you for the book suggestion - I'm into those kinds of tales. Or just restore from backup across the board. Assuming they do their backups well this shouldn't be too hard (especially since its currently in Read Only mode which means no new updates) Are you sure?
Are you $150 million ARR sure?
Are you $150 million ARR, you'd really like to keep your job, you're not going to accidentally leave a hole or blow up something else, sure? I agree, mostly, but I'm also really glad I don't have to put out this fire. Cheering them on from the sidelines, though! True but it does say something that such a script was able to lie dormant for so long. Why would anyone test in production???!!! There are plenty of ways to safely test in production. For one thing you need to limit the scope of your changes. Selecting the wrong environment in your test setup by mistake? I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose. > I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose. Do I have a bridge to sell you, oh boy I'm guessing,
"1> Hey Claude, your script ran this malicious script!" "Claude> Yes, you're absolutely right! I'm sorry!" On one hand, I was about to get irrationally angry someone was attacking Wikipedia, so I'm a bit relieved On the other hand, >a Staff Security Engineer at WMF, and naturally they decided to do this test under their highly-privileged Wikimedia Foundation staff account seriously? wait as a wikipedia user you can just put random JS to some settings and it will just... run? privileged? this is both really cool and really really insane It's a mediawiki feature: there's a set of pages that get treated as JS/CSS and shown for either all users or specifically you. You do need to be an admin to edit the ones that get shown to all users. Yes, you can have your own JS/CSS that’s injected in every page. This is pretty useful for widgets, editing tools, or to customize the website’s apparence. It sounds very dangerous to me but who am I to judge. It's nothing. For the global ones that need admin permissions to edit, it's no different from all the other code of mediawiki itself like the php. For the user scripts, it's no worse than the fact that you can run tampermonkey in your browser and have it modify every page from evry site in whatever way your want. It is kind of risky - you now have an entire, mostly unreviewed, ecosystem of javascript code, that users can experiment with. However its been really useful to allow power users to customize the interface to their needs. It also is sort of a pressure release for when official devs are too slow for meeting needs. At this point wikipedia has become very dependent on it. That is how Mediawiki works. Everything is a page, including CSS and JS. It is not really different than including JS in a webpage anywhere else. Wow. This worm is fascinating. It seems to do the following: - Inject itself into the MediaWiki:Common.js page to persist globally, and into the User:Common.js page to do the same as a fallback - Uses jQuery to hide UI elements that would reveal the infection - Vandalizes 20 random articles with a 5000px wide image and another XSS script from basemetrika.ru - If an admin is infected, it will use the Special:Nuke page to delete 3 random articles from the global namespace, AND use the Special:Random with action=delete to delete another 20 random articles EDIT! The Special:Nuke is really weird. It gets a default list of articles to nuke from the search field, which could be any group of articles, and rubber-stamps nuking them. It does this three times in a row. There doesn’t seem to be an ulterior motive beyond “Muahaha, see the trouble I can cause!” As someone on the Wikipediocracy forums pointed out, basemetrika.ru does not exist. I get an NXDomain response trying to resolve it. The plot thickens. Yeah, basemetrika.ru is free now. Should we occupy it? ;) I registered it about 40 minutes ago, but it seems the DNS has been cached by everyone as a result of the wikipedia hack & not even the NS is propagating. Can't get an SSL certificate . I had looked into its availability too just out of curiosity itself before reading your comment on a provider, Then I read your comment. Atleast its taken in from the hackernews community and not a malicious actor. Do keep us updated on the whole situation if any relevant situation can happen from your POV perhaps. I'd suggest to give the domain to wikipedia team as they might know what could be the best use case of it if possible. This community has no malicious actors? :) I'm not malicious at least :) Pretty public with who I am https://duti.dev/ It means giving money to the Russian government, so no. If anyone from the Russian government is reading this, get the fuck out of Ukraine. Thank you. reg.ru, the most popular registrar, sells .ru domains for $1.65, very little of which goes to the national registry. What is their profit on this domain, a couple of cents? You have helped to bring peace by approximately zero nanoseconds, while doing absolutely nothing about western countries still buying massive amounts of natural resources from Putin. Tax income on their exports make the primary source of income for the federal budget, which directly funds the military. Good virtue signaling, though. I'm completely disillusioned with the West, this is nothing new. I don't think voting with your wallet constitutes virtue signaling, especially at a time when end user boycotting is one of the universally known methods of protest. I am a pragmatist so maybe I will never understand this line of thinking. But in my mind, there are no perfect options, including doing nothing. By doing nothing, you are allowing a malicious actor to buy the domain. In fact I am sure they would love for everyone else to be paralyzed by purity tests for a $1 domain. All things being equal, yeah don’t buy a .ru domain. But they are not equal. [flagged] If anyone is genuinely curious about this, they were indeed letting Russian gas through and stopped in 2025: > On 1 January 2025, Ukraine terminated all Russian gas transit through its territory, after the contract between Gazprom and Naftohaz signed in 2019 expired. [...] It is estimated that Russia will lose around €5bn a year as a result. https://en.wikipedia.org/wiki/Russia%E2%80%93Ukraine_gas_dis... Namecheap won’t sell it which is great because it made me pause and wonder whether it's legal for an American to send Russians money for a TLD. Namecheap is Ukrainian, of course they won't sell you a .ru domain. Is it? Wikipedia says: > Namecheap is a U.S. based domain name registrar and web hosting service company headquartered in Phoenix, Arizona. and in 2025 they were purchased by: > CVC Capital Partners plc is a Jersey-based private equity and investment advisory firm Pretty sure it is, however, the reverse is actually illegal (for US citizens to provide professional services to anyone residing in Russia) as of like 2022-ish I'm half-tempted to try and claim it myself for fun and profit, but I think I'll leave it for someone else. What should we put there, anyway? A JavaScript call to window.alert to pause the JavaScript VM. Looks like someone other from the hackernews community has bought the domain https://news.ycombinator.com/item?id=47263323#47265499 Go old school and have the script inject the "how did this get here im not good with computers" cat onto random pages > Vandalizes 20 random articles with a 5000px wide image and another XSS script from basemetrika.ru Note while this looks like its trying to trigger an xss, what its doing is ineffective, so basemetrika.ru would never get loaded (even ignoring that the domain doesnt exist) Wouldn't be surprised if elaborate worms like this are AI-designed I wouldn't be surprised either. But the original formatting of the worm makes me think it was human written, or maybe AI assisted, but not 100% AI. It has a lot of unusual stylistic choices that I don't believe an AI would intentionally output. I would. AI designed software in general does not include novel ideas. And this is the kind of novel software AI is not great at, because there's not much training data. Of course it's very possible someone wrote it with AI help. But almost no chance it was designed by AI. > Cleaning this up is going to be an absolute forensic nightmare for the Wikimedia team since the database history itself is the active distribution vector. Well, worm didn't get root -- so if wikimedia snapshots or made a recent backup, probably not so much of a nightmare? Then the diffs can tell a fairly detailed forensic story, including indicators of motive. Snapshotting is a very low-overhead operation, so you can make them very frequently and then expire them after some time. Even if they reset to several days ago and lose, say, thousands of edits, even tens of thousands of minor edits, they're still in a pretty good place. Losing a few days of edits is less-than-ideal but very tolerable for Wikipedia as a whole At $work we're hosting business knowledge databases. Interestingly enough, if you need to revert a day or two of edits, you're better off to do it asap, over postponing and mulling over it. Especially if you can keep a dump or an export around. People usually remember what they changed yesterday and have uploaded files and such still around. It's not great, but quite possible. Maybe you need to pull a few content articles out from the broken state if they ask. No huge deal. If you decide to roll back after a week or so, editors get really annoyed, because now they are usually forced to backtrack and reconcile the state of the knowledge base, maybe you need a current and a rolled-back system, it may have regulatory implications and it's a huge pain in the neck. Nah, you can snapshot every 15 minutes. The snapshot interval depends on the frequency of changes and their capacity, but it's up to them how to allocate these capacities... but it's definitely doable and there are real reasons for doing so. You can collapse deltas between snapshots after some time to make them last longer. I'd be surprised if they don't do that. As an aside, snapshotting would have prevented a good deal of horror stories shared by people who give AI access to the FS. Well, as long as you don't give it root....... >Nah, you can snapshot every 15 minutes. obviously you can. but, what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot? that is what matters. in any case, the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all. > the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all. I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin. (Did you read the OP?) That's a /you/ problem if you rely on infrequent backups, especially for a service with so much flux. > what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot? ? Why would I know what their internal operations are? >I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin. >Why would I know what their internal operations are? i mean... you must, right? you know that once-a-day snapshots is not relevant to this specific incident. you know that their sysadmins are apparently competent. i just assumed you must have some sort of insider information to be so confident. I think you are misreading my comments and made a bad assumption. The reason I'm confident is because this has been my bread and butter for a decade. >The reason I'm confident is because this has been my bread and butter for a decade. my decade of dealing with incompetent sysadmins and broken backups (if they even exist) has given me the opposite of confidence. but im glad you have had a different experience > my decade of dealing with incompetent sysadmins and broken backups (if they even exist) has given me the opposite of confidence. Oh, I agree that the average bar is low. That's part of the reason I do it all myself. The heuristic with wikimedia is that they've been running a PHP service that accepts and stores (anonymous) input for 25 years. The longetivity with the risk exposure that they have are indicators that they know what they are doing, and I'm sure they've learned from recovering all sorts of failures over the years. Look at how quickly it was brought back up in this instance! So, yeah. I don't think initial hypothetical counterpoint holds water, and that's what I have been pointing out.
tux3 - 4 hours ago
Ferret7446 - an hour ago
mcmcmc - an hour ago
modderation - 9 minutes ago
radicaldreamer - 27 minutes ago
xvector - 38 minutes ago
londons_explore - 4 hours ago
jl6 - an hour ago
HoldOnAMinute - 42 minutes ago
varenc - 36 minutes ago
edoceo - an hour ago
Melatonic - 23 minutes ago
observationist - an hour ago
jacquesm - 3 hours ago
outofpaper - 2 hours ago
HoldOnAMinute - 41 minutes ago
ninth_ant - an hour ago
irishcoffee - an hour ago
Fokamul - an hour ago
AlienRobot - an hour ago
karel-3d - 3 hours ago
kemayo - 2 hours ago
hk__2 - 3 hours ago
karel-3d - 2 hours ago
Brian_K_White - 2 hours ago
bawolff - 25 minutes ago
corndoge - 2 hours ago
nhubbard - 6 hours ago
divbzero - 44 minutes ago
256_ - 5 hours ago
pKropotkin - 5 hours ago
acheong08 - 4 hours ago
Imustaskforhelp - 4 hours ago
Freak_NL - an hour ago
acheong08 - 2 minutes ago
amiga386 - 5 hours ago
INR18650 - 4 hours ago
avidruntime - 2 hours ago
janalsncm - 2 hours ago
cryptoegorophy - 4 hours ago
Rendello - 3 hours ago
Barbing - 5 hours ago
throw-the-towel - 2 hours ago
craftkiller - an hour ago
DaSHacka - 4 hours ago
256_ - 5 hours ago
speedgoose - 5 hours ago
Imustaskforhelp - 4 hours ago
gibsonsmog - 5 hours ago
bawolff - 5 hours ago
dheera - 5 hours ago
nhubbard - 5 hours ago
integralid - 5 hours ago
Kiboneu - 5 hours ago
Extropy_ - 5 hours ago
tetha - 4 hours ago
Kiboneu - 5 hours ago
john_strinlai - 4 hours ago
Kiboneu - 4 hours ago
john_strinlai - 4 hours ago
Kiboneu - 4 hours ago
john_strinlai - 4 hours ago
Kiboneu - 3 hours ago