Discussion
greyface-: Additional context:https://wikipediocracy.com/forum/viewtopic.php?f=8&t=14555https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(techni...https://old.reddit.com/r/wikipedia/comments/1rllcdg/megathre...Apparent JS worm payload: https://ru.wikipedia.org/w/index.php?title=%D0%A3%D1%87%D0%B...
tantalor: Nice to see jQuery still getting used :)
varun_ch: Woah this looks like an old school XSS worm https://meta.wikimedia.org/wiki/Special:RecentChanges?hidebo...I’ve always thought the fact that MediaWiki sometimes lets editors embed JavaScript could be dangerous.
yabones: It's sad that something so critical for society and culture is run on a shoestring budget by volunteers. They do a great job, the best they possibly could. Hope they can get this sorted out without too much fuss.Btw, I recommend everybody set up a $2-5 monthly donation to WMF.
Markoff: please stop spreading lies, Wikipedia is swimming in money and they have money for years or even decades if they would not waste them on various seminars and other nonsense unrelated to running Wikipedia
256_: Here before someone says that it's because MediaWiki is written in PHP.
varun_ch: Also, I’m also surprised an XSS attack like hasn’t yet been actually used to harvest credentials like passwords through browser autofill[0].It seems like the worm code/the replicated code only really attacks stuff on site. But leaking credentials (and obviously people reuse passwords across sites) could be sooo much worse.[0] https://varun.ch/posts/autofill/
gadders: "The Wikimedia Foundation, which operates Wikipedia, reported a total revenue of $185.4 million for the 2023–2024 fiscal year (ending June 2024). The majority of this funding comes from individual donations, with additional income from investments and the Wikimedia Enterprise commercial API service."(Unless this was satire and I missed it)
Dwedit: PHP is the language where "return flase" causes it to return true.https://danielc7.medium.com/remote-code-execution-gaining-do...
m4tthumphrey: Also the language that runs half of the web.Also the language that has made me millions over my career with no degree.Also the language that allows people to be up and running in seconds (with or without AI).I could go on.
jjice: PHP is a fine language. It started my career. That said, it has a lot of baggage that can let you shoot yourself in the foot. Modern PHP is pretty awesome though.
nzeid: Wikipediocracy link gives "not authorized".
ChrisMarshallNY: I use it on the backends of my stuff.Works great, but, like any tool, usage matters.People who use tools badly, get bad results.I've always found the "Fishtank Graph" to be relevant: https://w3techs.com/technologies/history_overview/programmin...
epicprogrammer: This is basically a weaponized, highly destructive version of the old MySpace Samy worm. Hitting MediaWiki:Common.js is the absolute nightmare scenario for MediaWiki deployments because that script gets executed by literally every single visitor and editor across the entire site, creating a massive, instant propagation loop. The fact that it specifically targets admins and then uses jQuery to blind them by hiding the UI elements while it silently triggers Special:Nuke in the background is incredibly insidious. It really exposes the foundational danger of legacy web architectures that still allow executable JavaScript to be stored and served directly from user-editable namespaces. Cleaning this up is going to be an absolute forensic nightmare for the Wikimedia team since the database history itself is the active distribution vector.
devmor: In the early 2010’s I worked for a company whose primary income was subscriptions to site protection services - one of which included cleaning up malware-infected Wordpress installations. I worked on the team that did this job.This exact type of database-stored executable javascript was one of the most annoying types of infections to clean up.
pKropotkin: admins are the most disgusting thing on wikipedia
softskunk: care to elaborate?
Uhhrrr: How do they know? Has this been published in a Reliable Source?
nhubbard: Wow. This worm is fascinating. It seems to do the following:- Inject itself into the MediaWiki:Common.js page to persist globally, and into the User:Common.js page to do the same as a fallback- Uses jQuery to hide UI elements that would reveal the infection- Vandalizes 20 random articles with a 5000px wide image and another XSS script from basemetrika.ru- If an admin is infected, it will use the Special:Nuke page to delete 3 random articles from the global namespace, AND use the Special:Random with action=delete to delete another 20 random articlesEDIT! The Special:Nuke is really weird. It gets a default list of articles to nuke from the search field, which could be any group of articles, and rubber-stamps nuking them. It does this three times in a row.
256_: As someone on the Wikipediocracy forums pointed out, basemetrika.ru does not exist. I get an NXDomain response trying to resolve it. The plot thickens.
pKropotkin: Yeah, basemetrika.ru is free now. Should we occupy it? ;)
af78: Time to add 2FA...
256_: I'm half-tempted to try and claim it myself for fun and profit, but I think I'll leave it for someone else.What should we put there, anyway?
speedgoose: A JavaScript call to window.alert to pause the JavaScript VM.
j45: Too much app logic in the client side (Javascript) has always been an attack vector. The more that can reasonably be server side, the more that can't be seen.
0xWTF: Ok, so there are tons of mediawiki installations all over the internet. What do these operators do? Set their wikis to read-only mode, hang tight, and wait for a security patch?Also, does this worm have a name?
bawolff: There is nothing to do, the incident was not caused by a vulnerability in mediawiki.Basically someone who had permissions to alter site js, accidentally added malicious js. The main solution is to be very careful about giving user accounts permission to edit js
Wikipedianon: This was only a matter of time.The Wikipedia community takes a cavalier attitude towards security. Any user with "interface administrator" status can change global JavaScript or CSS for all users on a given Wiki with no review. They added mandatory 2FA only a few years ago...Prior to this, any admin had that ability until it was taken away due to English Wikipedia admins reverting Wikimedia changes to site presentation (Mediaviewer).But that's not all. Most "power users" and admins install "user scripts", which are unsandboxed JavaScript/CSS gadgets that can completely change the operation of the site. Those user scripts are often maintained by long abandoned user accounts with no 2 factor authentication.Based on the fact user scripts are globally disabled now I'm guessing this was a vector.The Wikimedia foundation knows this is a security nightmare. I've certainly complained about this when I was an editor.But most editors that use the website are not professional developers and view attempts to lock down scripting as a power grab by the Wikimedia Foundation.
256_: Maybe somewhat unrelated, but I'm reminded of the fact that people have deleted the main page on a few occasions: https://en.wikipedia.org/wiki/Wikipedia:Don%27t_delete_the_m...
j45: It's reassuring to know Wikipedia has these kinds of security mechanisms in place.
cwillu: Try not to take criticisms of tools personally. Phillips head screws are shit for a great many applications, while simultaneously being involved in billions of dollars of economic activity, and being a driver that everyone has available.
Barbing: Namecheap won’t sell it which is great because it made me pause and wonder whether it's legal for an American to send Russians money for a TLD.
streetfighter64: Well, admins (or anybody other than the developers / deployment pipeline) having permissions to alter the JS sounds like a significant vulnerability. Maybe it wasn't in the early 2000s, but unencrypted HTTP was also normal then.
lifeisstillgood: I completely understand marking the software that controls drinking water as critical infrastructure- but at some point a state based cyber attack that just wipes wikipedia off the net is deeply damaging to our modern society’s ability to agree on common facts …Just now thought “if Wikipedia vanished what would it mean … and it’s not on the level of safe drinking water, but it is a level.
lyu07282: There are so many mirrors anyway and trivial to get a local copy? What is much more concerning is government censorship and age verification/digital id laws where what articles you read becomes part of your government record the police sees when they pull you over.
jasonjayr: Perl still runs the other half?
Aperocky: All persistent data should have backup.It's not a high bar.
nhubbard: This is the official Wikimedia Foundation status page for the whole of Wikipedia, so it's a reliable primary source.
vova_hn2: Actually, usage of primary sources is kinda complicated [0], generally Wikipedia prefers secondary and tertiary sources.[0] https://en.wikipedia.org/wiki/Wikipedia:No_original_research...
jkaplowitz: Yeah, but the purpose of an encyclopedia like Wikipedia (a tertiary source) is to relatively neutrally summarize the consensus of those who spend the time and effort to analyze and interpret the primary sources (and thus produce secondary sources), or if necessary to cite other tertiary summaries of those.In a discussion forum like HN, pointing to primary sources is the most reliable input to the other readers' research on/synthesis of their own secondary interpretation of what may be going on. Pointing to other secondary interpretations/analyses is also useful, but not without including the primary source so that others can - with apologies to the phrase currently misused by the US right wing - truly do their own research.
pixl97: >Cleaning this upFind the first instance and reset to the backup before then. An hour, a day, a week? Doesn't matter that much in this case.
bbor: It is true that they have a particularly robust, distributed backup system that can/has come in handy, but FWIW the timing matters to them. English Wikipedia receives ~2 edits per second, or 172,800 per day. Many of them are surely minor and/or automated, but still: 1,036,800 lost edits is a lot!
Kiboneu: GOD am I thankful to my old self for disabling js by default. And sticking with it.
i_think_so: > Hitting MediaWiki:Common.js is the absolute nightmare scenario for MediaWiki deployments because that script gets executed by literally every single visitor...except for us security wonks who have js turned off by default, don't enable it without good reason, disable it ASAP, and take a dim view of websites that require it.Not too many years ago this behavior was the domain of Luddites and schizophrenics. Today it has become a useful tool in the toolbox of reasonable self-defense for anybody with UID 0.Perhaps the WMF should re-evaluate just how specialsnowflake they think their UI is and see if, maybe just maybe, they can get by without js. Just a thought.
stephbook: Chrome doesnt actually autofill before you interact. It only displays what it would fill in at the same location visually.
varun_ch: but any interaction is good for Chrome, like dismissing a cookie banner
Kiboneu: > Cleaning this up is going to be an absolute forensic nightmare for the Wikimedia team since the database history itself is the active distribution vector.Well, worm didn't get root -- so if wikimedia snapshots or made a recent backup, probably not so much of a nightmare? Then the diffs can tell a fairly detailed forensic story, including indicators of motive.Snapshotting is a very low-overhead operation, so you can make them very frequently and then expire them after some time.
Extropy_: Even if they reset to several days ago and lose, say, thousands of edits, even tens of thousands of minor edits, they're still in a pretty good place. Losing a few days of edits is less-than-ideal but very tolerable for Wikipedia as a whole
Kiboneu: Nah, you can snapshot every 5 minutes.
dboreham: There are already tools and techniques to validate served JS is as-intended, and these techniques could be beefed up by adding browser checks. I've been surprised these haven't been widely adopted given the spate of recent JS-poisoning attacks.
dns_snek: The amount of javascript is really beside the point here. The problem is that privileged users can easily edit the code without strong 2FA, allowing automatic propagation.
shevy-java: How does 2FA prevent this here?
dns_snek: [delayed]
radium3d: Pretty sure we've seen people coding in essentially every other programming language also shoot themselves in the foot.
Sohcahtoa82: Every language has foot-guns of some sort. The difference is how easy it is to accidentally pull the trigger.PHP makes it easy.
shevy-java: Are they really lost though? I think they should not be lost; they could be stored in a separate database additionally.
derefr: In fact, as long as the malware is just doing deletes, you can just merge the two "timelines" by restoring the snapshot and then replaying all the edits but ignoring the deletes. Lost deletes really aren't much of a problem!
Kiboneu: Filesystem & database snapshots are very cheap to make, you can make them every 15 minutes. You can expire old snapshots (or collapse the deltas between them) depending on the storage requirements.
gchamonlive: The problem isn't the granularity of the backup but since the worm silently nukes pages, it's virtually impossible to reconcile the state before the attack and the current state, so you have to just forfeit any changes made since then and ask the contributors to do the leg work of reapplying the correct changes
Kiboneu: Why would nuked pages matter? Snapshots capture everything and are not part of wikimedia software.
streetfighter64: If you're using wikipedia to "agree on common facts" I think you might have bigger problems...
clcaev: We should be using federated architectures. For Wikipedia, a central read-only hub that delegates to communities who have proven themselves. Common, suggested tooling (software and processes) could be maintained centrally but each community may enjoy a bit more independence.
chris_wot: Most admins on Wikipedia are incompetent.
alphager: Most admins on Wikipedia are competent in areas outside of webdev and security.
quantum_magpie: Could you point to where you found the details of the exploit? It’s not in the linked page. Really interested. Especially the part about modifying it and the other users propagating it?
homebrewer: The fact of this obvious LLM slop being at the top of this discussion is incredibly insidious. The "facts" it mentions are made up. Has this vapid style finally become so normalized that nobody is seeing it anymore?
infinitewars: That user, epicprogrammer's comment history suggests alignment with the Musk/Thiel/Anduril/DoW/anti-Anthropic crowd who are incessantly trying to damage Wikipedia's reputation to push a "Grokipedia" where they can define the narrative.
sobjornstad: Nowadays I refuse to do any serious work that isn't in source control anywhere besides my NAS that takes copy-on-write snapshots every 15 minutes. It has saved my butt more times than I can count.
Kiboneu: Yeah same here. Earlier I had a sync error that corrupted my .git, somehow. no problem; I go back 15 minutes and copy the working version.Feels good to pat oneself in the back. Mine is sore, though. My E&O/cyber insurance likes me.
amiga386: It means giving money to the Russian government, so no.If anyone from the Russian government is reading this, get the fuck out of Ukraine. Thank you.
cryptoegorophy: Did you know… ukraine still lets Russian gas transit through ukraine territory? Making ukraine the largest sponsor of terrorism against ukraine? Did you know, when war started it, ukraine was letting Russia make around $1 billion PER DAY for like a year before reducing that amount ? You didn’t know that. But hey, protesting by not letting some one buy .ru will certainly do damage to Putin!
john_strinlai: >Nah, you can snapshot every 15 minutes.obviously you can.but, what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot?and, in any case, the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all.
infinitewars: A comment from my wiki-editor friend: "The incident appears to have been a cross-site scripting hack. The origin of rhe malicious scripts was a userpage on the Russian Wikipedia. The script contained Russian language text. During the shutdown, users monitoring [https://meta.wikimedia.org/wiki/special:RecentChanges Recent changes page on Meta] could view WMF operators manually reverting what appeared to be a worm propagated in common.js Hopefully this means they won't have to do a database rollback, i.e. no lost edits. " Interesting to note how trivial it is today to fake something as coming "from the Russians".
tetha: At $work we're hosting business knowledge databases. Interestingly enough, if you need to revert a day or two of edits, you're better off to do it asap, over postponing and mulling over it. Especially if you can keep a dump or an export around.People usually remember what they changed yesterday and have uploaded files and such still around. It's not great, but quite possible. Maybe you need to pull a few content articles out from the broken state if they ask. No huge deal.If you decide to roll back after a week or so, editors get really annoyed, because now they are usually forced to backtrack and reconcile the state of the knowledge base, maybe you need a current and a rolled-back system, it may have regulatory implications and it's a huge pain in the neck.
256_: I didn't even notice it until you pointed it out, but I checked that account's comment history and it uses em dashes. Also, "the database history itself is the active distribution vector" Is just semantic nonsense.I still have a basic assumption that if something I'm reading doesn't make much sense to me, I probably just don't understand it. Over the last few years I've had to get used to the new assumption that it's because I'm reading LLM output.
homebrewer: I've also always used em-dashes, it's not a very reliable indicator. That style is a dead giveaway, though. Some of its comments seem to be written by a human, but several definitely aren't.I've been spending less and less time here, the moderation is obviously overwhelmed and is losing the battle.https://aphyr.com/posts/389-the-future-of-forums-is-lies-i-g...
Kiboneu: > the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all.I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin. (Did you read the OP?) That's a /you/ problem if you rely on infrequent backups, especially for a service with so much flux.> what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot?? Why would I know what their internal operations are?
john_strinlai: >I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin.>Why would I know what their internal operations are?i mean... you must, right? you know that once-a-day snapshots is not relevant to this specific incident. you know that their sysadmins are apparently competent. i just assumed you must have some sort of insider information to be so confident.
marginalia_nu: > [...] is incredibly insidious. It really exposes the foundational danger of [...]My LLM sense is tingling.
JKCalhoun: Perhaps we're at last watching the internet die.
NoMoreNicksLeft: Yes, but we did that over the last 15 years. We just never realized that's what we were seeing.It only clicked for me a few weeks ago, in one thread or another here when I realized that no one could ever do what Google did once: Cloudflare and other antibot technologies have closed off traditional search-as-the-result-of-web-crawling permanently. It's not that no one will do it because they think there's no money in it, or that no one will do it because the upfront costs are gigantic... literally it can no longer be done.The internet died.
Imustaskforhelp: There are still a few options. I recently had the idea of doing search engine queries on 9 search engines.Mojeek is a good independent search browser, it isn't the best but at that Hackernews comment/analysis I was doing I found it to be the only one which worked for that case.Brave exists too.I know the situation is very critical/dire tho but there is still some chance. All be it quite small.Mojeek IIRC, is operated by one single guy for 15 years.
Imustaskforhelp: Looks like someone other from the hackernews community has bought the domain https://news.ycombinator.com/item?id=47263323#47265499
chuckadams: https://3v4l.org/eua7G#veolIs an error in every currently supported version of PHP, and even in the 7.x series the "warning" it throws is fatal by default unless specifically suppressed. PHP certainly has an ugly past, but I suspect Xsuite would find a way to be garbage in any language.
bbor: It warms my heart that there's basically a 0% chance that they ever approach this camp's viewpoint based on the Herculean effort it took to switch over to a slightly more modern frontend a few years back. I'm glad you don't think of yourself of a Luddite, but I think you're vastly overstating how open people are to a purely-static web.Also, FWIW: Wikipedia is "specialsnowflake". If it isn't, that's merely because it was so specialsnowflake that there's now a healthy of ecosystem of sites that copied their features! It's far, far more capable than a simple blog, especially when you get into editing it.
i_think_so: Ok, fair point. I presumed that this crowd would be far more familiar with the capabilities of HTML5 and dynamic pages sans js than most. (Surely more familiar than I, who only dabble in code by comparison.)No, I'm not suggesting we all go back to purely-static web pages, imagemap gifs and server side navigation. But you're going to have a hard time convincing me that I really truly need to execute code of unknown provenance in my this-app-does-everything-for-me process just to display a few pages of text and 5 jpegs.And for the record, I've called myself a Technologist for almost 30 years now. If I were a closet Luddite I'd be one of the greatest hypocrites of human history. :-)