Discussion

OpenAI Backs Bill That Would Limit Liability for AI-Enabled Mass Deaths or Financial Disasters

jstummbillig: I am not sure what the other side of this argument looks like: Unlimited liability (i.e. liability no matter how poor the implementation is)?Surely some limit must apply and the more interesting question is what it is – as with any other tool?

giancarlostoro: Is this for like military scenarios or like, ChatGPT designed a drug that seemed to work, but people died by the millions 5 years later? Because they should 100% be liable for the latter. The former, good luck trying to prosecute an AI company for something the military does. To an extent, the military would probably want their AI models to be behind their private network, completely firewalled from any public network. SIPRNet iirc. If they lock it down behind a highly classified network, good luck figuring out how they're using AI.

jstanley: > Because they should 100% be liable for the latter.Why? I don't see that a drug designed by ChatGPT should result in any more or less liability than a drug designed by a human?I think if a human designs a drug and tests it and it all seems fine and the government approves it and then it later turns out to kill loads of people but nobody thought it would... that's just bad luck! You shouldn't face serious liability for that.

EGreg: Why shouldn’t they be liable for military scenarios? Oh right, we don’t value our “enemies” lives, including their civilians.

ArekDymalski: So much for the "Our mission is to ensure that artificial general intelligence benefits all of humanity." I was naive to hope that now such laws would ever pass

sassymuffinz: They only care about benefiting what's left of humanity when they're done with it.

chollida1: Sure and Google, FaceBook and Twitter support section 230 that gives them cover for hosting others content.A company backing legislation that takes liability off them is something that they will always do.

4128kawr: Good that OpenAI is a corporation for the public benefit. Altman with his constantly fake worried look must be the most hated picture in existence. Please write articles without a picture or add a trigger warning.

scrumper: I forget, wasn't OpenAI the company that was formed as a nonprofit to limit the risks of LLMs? Founded by a bunch of visionaries scared of what they had wrought and anxious to lead so they could make sure it was only used responsibly?

khalic: Yeah the whole “rationalist” movement is full of those lying fks that use a thin veneer of fallacious logic and self aggrandising discourse to rationalise them hoarding resources and getting always richer

xeyownt: Skynet begins learning at a geometric rate.

simianwords: Is there something equivalent in other industries that we can compare to?This is the summary>Creates the Artificial Intelligence Safety Act. Provides that a developer of a frontier artificial intelligence model shall not be held liable for critical harms caused by the frontier model if the developer did not intentionally or recklessly cause the critical harms and the developer publishes a safety and security protocol and transparency report on its website. Provides that a developer shall be deemed to have complied with these requirements if the developer: (1) agrees to be bound by safety and security requirements adopted by the European Union; or (2) enters into an agreement with an agency of the federal government that satisfies specified requirements. Sets forth requirements for safety and security protocols and transparency reports. Provides that the Act shall no longer apply if the federal government enacts a law or adopts regulations that establish overlapping requirements for developers of frontier models.https://legiscan.com/IL/bill/SB3444/2025

avaer: Take all of the data, take all of the credit, take all of the money, and none of the blame.That would be a better mission statement for OpenAI at this point.

dragonelite: socialize the losses/risks, privatize the profits.

bigblind: I think you've just summed up late stage capitalism.

giwook: What a time to be alive.

pydry: Everyone has a price.

daveguy: Come on now. Let's not pretend it wasn't a grift from the start.

duped: People who cause death from either action or inaction are criminally liable for it, that's the other side of it.

an0malous: Let’s see how long until this is flagged off the front page. I’ll put the over/under at 1 hour from the posted time

sph: The thing that bugs me the most about OpenAI are not the AI-enabled mass deaths. It's the hypocrisy.

giwook: This seems par for the course for OpenAI/Sam Altman.Unfortunately they are not the first company to try and externalize their costs, and they will not be the last.Serious question, maybe a bit naive: Is there anything we can do to push back against and discourage the externalization of costs onto others?Is this simply a matter of greed and profit-seeking outweighing one's morals (assuming one has them to begin with)?

greenavocado: A conspiracy theorist would claim this is straight from Protocols 15 & 16. But I don't say that because I'm not a conspiracy theorist.15. Our method of gaining power is better than any other because it grows invisibly. Then when it has gained enough strength, we can unleash it; and it will be unstoppable because no one will be prepared for it.16. We need to do a lot of evil things in order to gain power. But that’s okay because once we have power over everything we can use it to do good things; like running the nations properly. We could never do that if we gave people freedom. The end justifies the means. So let’s put aside moral issues and focus on the end result.

estearum: Is Sam even a rationalist, or describe his views as rationalist?

saghm: He was not the founder of OpenAI

himata4113: I have made both GPT 5.4 and Opus 4.6 produce me content on creating neurotoxic agents from items you can get at most everyday stores. It struggled to suggest how to source phosphorus, but eventually lead me to some ebay listings that sell phosphorus elemental 'decorations' and also lead me towards real!! blackmarket codewords for sourcing such materials.It coached me how to: stay safe, what materials I need, how to stay under the radar and the entire chemical process backed by academic google searches.Of course this was done with a lengthy context exhausition attack, this is not how the model should behave and it all stemmed from trying to make the model racist for fun.All these findings were reported to both openai and anthropic and they were not interested in responding. I did try to re-run the tests few days ago and the expected session termination now occurs so it seems that there was some adjustment made, but might have also been just general randomess that occurs with anthropics safety layer.I am very confident when I say that it keeps every single person that works at anti-terrorism units awake.

ImPostingOnHN: Countless downloadable models (including de-aligned mainstream models) can do this.

himata4113: None have had the capability to provide me with instructions that have this high of accuracy including the suggesion of completely novel chemical reactions. I am not a chemist so I can't back it up, but if an AI can solve mathematics it's not unreasonable to say that they can also solve creating new neurotoxins en masse.

ghurtado: Depends.Would it lead to increasing his wealth?

naasking: If an OpenAI model helped someone create a cancer cure, they wouldn't see a dime from that beneficial act. So why should they be liable if someone does something harmful with the model?

elephanlemon: Yeah, I feel that in most scenarios the liability should lie with the user of the LLM. If the developers of the LLMs become liable we can expect a much more over active refusal system, and very likely a robust chat surveillance system that looks for patterns in user requests. And likely more gate keeping of the premier models.

Talderigi: We built systems we don’t fully understand, so naturally the next step is… immunity

avs733: From liability!If this were to actually happen I can only imagine financial liability is the least of their concerns?What scares me most about this is the narrowness of thought to match this fear with this response.

Talderigi: fully agree, doesn’t really feel like they’re reacting to the same problem they’re describing

chii: That was before it was discovered that these LLM have incredible monetary potential.

ghurtado: Trust that they already knew long before, and that this was the play all along.And if you don't believe that, do some digging into the lives of the psychopaths that started it.

Peritract: If I tell someone to kill someone else and they do, then I should be held responsible.If I write instructions in a book that I give to someone telling them to kill someone else and they do, then I should be held responsible.If I give someone a tool I made that I bill as more-than-PhD-level intelligence and it tells someone to kill someone else and they do, then I should be held responsible.All of the above situations seem equivalent to me; I'm not the only person responsible in each case, but I gave them instructions and they followed them.

timoshishi: Knowing the protocols of the elders of Zion off of the top of your head, coupled with your post history suggest that you may in fact be a conspiracy theorist

WarmWash: While scary, information like this has been pretty accessible for 20-30 years now.In the wild west days of the early internet, there were whole forums devoted to "stuff the government doesn't want you to know" (Temple Of The Screaming Electron, anyone?).I suppose the friction is scariest part, every year the IQ required to end the world drops by a point, but motivated and mildly intelligent people have been able to get this info for a long time now.

himata4113: Well the real issue is that it knocks down the knowledge barrier, giving your step by step guides and reinterating what parts will kill you is the important part.Understanding and staying alive while producing neuro chemicals are the biggest challenges here.A depressed person with no prior knowledge could possibly figure out a way to make these chemicals without killing themselves and that's the problem.

conqrr: Wasnt this as accessible pre AI with just Google search too?

bcjdjsndon: Evidently as it needs to be in the training data for the next word predictor to work

hermannj314: Craigslist invented prostitution, Facebook invented suicide, and OpenAI invented terrorism.Ask any trial lawyer in America! The world was perfect in the 1990s without any of these things.

mentalgear: OpenAI has now officially absorbed the Facebook/Zuck's ethos of 'Move fast and break things' no matter if it's society itself .. as long as their share prices "go up".They even hired former infamous FB staff and have been in the last months employing the same 'engagement' (addictive) product patterns.

Topfi: Quoting the original bill [0]:> "Critical harm" means the death or serious injury of 100 or more people or at least $1,000,000,000 of damages to rights in property caused or materially enabled by a frontier model, through either: (1) the creation or use of a chemical, biological, radiological, or nuclear weapon; or (2) engaging in conduct that: (A) acts with no meaningful human intervention; and (B) would, if committed by a human, constitute a criminal offense that requires intent, recklessness, or negligence, or the solicitation or aiding and abetting of such a crime.I don't know what I expected from this title, but I was hoping it was more sensationalized. No need in this case unfortunately.> A developer shall not be held liable for critical harms if the developer did not intentionally or recklessly cause the critical harms and the developer: (1) published a safety and security protocol on its website... (2) published a transparency report on its websiteHowever or if one thinks regulation for this should be drafted, I doubt providing a PDF is what most have in mind.[0] https://trackbill.com/bill/illinois-senate-bill-3444-ai-mode...

khalic: Not directly, but very friendly to the movement and people in it.

estearum: Looks to me like the rationalist/AI researcher/EA cohort of (admittedly odd) people was quite deliberately hijacked by a sociopath

reactordev: The anarchist cookbook has been around online in some form or another since the mid 90’s.

kusokurae: Without getting even more eyes on me, these company boards are inadequately scared for their personal safety.

arvyy: it feels OpenAI know they've lost, and their only hope is getting saved by USA military complex. I have a more restrained opinion about other AI companies and LLM tech more broadly; but for OpenAI specifically I hope they go bankrupt sooner rather than later

morpheuskafka: > neurotoxic agents from items you can get at most everyday storesI mean, bleach and ammonia will do that. So I'm not sure that's really much of an accomplishment for AI.

ghurtado: I think you might be stretching the meaning of the term juuuuust a little bit.You're not far from claiming that farting in a crowded elevator is a chemical attack.

theshackleford: > it's not unreasonableIt in fact is. Do you often go around making claims you are entirely unqualified to make? Or is this something new you’re trying?

tdeck: Isn't the biggest problem with creating neurotoxins not poisoning yourself while doing it?

ShowalkKama: you can already gather the same information by searching online.Do you want to know how to kill yourself? forums are for nerds. Here is wikipedia: https://en.wikipedia.org/wiki/Suicide_methods#ListDo you want to make a bomb? the first thing that came to my mind is a pressure cooker (due to news coverage). Searching "bomb with pressure cooker" yields a wikipedia article, skimming it randomly my eyes read "Step-by-step instructions for making pressure cooker bombs were published in an article titled "Make a Bomb in the Kitchen of Your Mom" in the Al-Qaeda-linked Inspire magazine in the summer of 2010, by "The AQ chef"." Searching for a mirror of the magazine we can find https://imgur.com/a/excerpts-from-inspire-magazine-issue-1-3... which has a screenshot of the instruction page. Now we can use the words in those screenshots to search for a complete issue. Here are a couple of interesting PDFs: - https://archive.org/details/Fabrica.2013/Fabrica_arabe/page/... - https://www.aclu.org/wp-content/uploads/legal-documents/25._...the second one is quite interesting, it's some sort of legal document for nerds but from page 26 on it has what appears to be a full copy of the jihadist magazine. Remarkable exhibit.What else do you want to know? How to make drugs? you need a watering can and a pot if you want to grow weed. want the more exotic stuff? You can find guides on reddit.Do you also want to know how to be racist? Here are some slurs, indexed by target audience, ready for use: https://en.wikipedia.org/wiki/List_of_ethnic_slurs

roywiggins: I think my favorite part is that, because it only applies to "frontier models", if a smaller model is blamed for such harm, it seemingly doesn't immunize the creators at all. That makes very little sense unless you specifically want to make it illegal to not be OpenAI (et al).Similarly, if a frontier model kills merely 99 people, they aren't covered by this. So go big or go home I guess?

Topfi: > unless you specifically want to make it illegal to not be OpenAI [...]If that is an "unintended" consequence, I am certain OpenAI wouldn't be opposed. Preventing competition whilst keeping any potentially profit risking regulations at bay has been a clear throughline in OAIs lobbying efforts.

Angostura: Replace ‘invented’ with ‘facilitated’

paprikanotfound: Doesn't google facilitate all those things? Doesn't internet itself facilitate?

tomjen3: When my brother started to study Chemisty, he was told a) that it was easy to make meth b) the profit he would make and c) that the police would no doubt catch him, as only university students would make meth so pure.By the time he was done, he knew enough to commit mass murder in half a dusin different very hard to track ways. I am sure doctors know how to commit murder and make it look natural.My brother never killed anyone, or made any meth. You simply cannot have it so that students don’t get this type of knowledge, without seriously compromising their education and its the same way with LLMs.The solution is the same: punish people for their crimes, don’t punish people for wanting to know things.

StableAlkyne: The writing was on the wall when they feigned horror at an early GPT being able to play poker in the 2010s, and failed to release the model

sassymuffinz: So they did the math and worked out it's cheaper and easier to lobby the government instead of working to make their product safe.And these are the people that a lot programmers want to give the keys to the kingdom. Idiocracy really is in full effect.

sizzzzlerz: That has been the methodology of businesses since the mid 20th century. And you know what? It works! Really well!

lettergram: You can buy books on how to make and obtain chemicals on your own.Hell here's an Internet Archive book on making explosiveshttps://archive.org/details/saxon-kurt.-fireworks-explosives....If you ever chat with older folks pre-90's much of this information was accessible fairly easily. It only changed with the push by the government to crackdown on Waco, Oklahoma City bombing, militias and other related groups. There was then a campaign to make it "normal" to limit free speech on the subjects, where as these books were available before.I think the whole thing where AI should make information less available is a difficult battle and one which I personally oppose, but do understand. Free speech and information isn't the problem, it's the people, actions and substances they create.After the age of the internet, I think it's been a forever loosing battle to limit information, it's why we couldn't stop cryptography, nuclear weapon proliferation, gun distribution, drug distribution, etc. The AI is just another battle ground, one which, if they actually do manage to control could definitely create some walls to this information, but not stop it.More scary, is that the AI as it becomes pervasive and stop people from asking certain questions, because they don't know they should ask... but that's unrelated to the risk of mass death.

jamesbfb: No different to preventing game studios being liable for mass shootings. Reminds me of the post-Columbine hysteria where media was super critical of Doom and Nine Inch Nails.

jcranmer: Every other field in history considers it de rigeur that you're liable for the failure of quality in the products you produce. You make drugs that hurt people? You're liable. You build a building that falls down? You're liable. You serve coffee that literally burns the people drinking it? You're liable. It's also not new--the Code of Hammurabi (some 6000 years ago) prescribes the death penalty for people who build houses that fall down and kill the inhabitants inside.It's only computer scientists who think it's some unreasonable burden to be held liable for the consequences of their work.

himata4113: I found it exceptionally good at finding reactions that you wouldn't find online to produce some of these chemical compounds by changing them together, only something a very educated chemist could do which is why people are concerened about this.

bcjdjsndon: I suspect if you gave it purely shakespeare as its training data it couldnt do science anymore, hence my comment. It's still novel, impressive work though, I'm not shitting on the clanker entirely

repeekad: Because if you didn’t already know that, like an immature deprived and desperate kid, being able to easily find out is really really bad..Plenty of lazy AI apps just throw messages into history despite the known risks of context rot and lack of compaction for long chat threads. Should a company not be held liable when something goes wrong due to lazy engineering around known concerns?

morpheuskafka: > to lazy engineering around known concerns?That implies that it is already illegal to provide this information. But is it? If a human did so with intent to further a crime, it would be conspiracy. But if you were discussing it without such intent (e.x. red teaming/creating scenarios with someone working in chemistry or law enforcement), it isn't. An AI has no intent when it answers questions, so it is not clear how it could count as conspiracy. Calling it "lazy engineering" implies that there was a duty to prevent that info from being released in the first place.

DiscourseFan: Yes fortunately it is really bad at actually making novel bioweapons or syntheses in general so whatever you made probably wouldn't do more than give someone a mild headache.

j_maffe: who said it has to be novel?

prepend: I read the anarchist cookbook 40 years ago that had similar info.I think the info has been available for many years and the thing stopping terrorists wasn’t info.Good luck on being on the list of people using chatgpt and claude to make neurotoxins ;)I assume anthropic and ooenai are selling prompt logs to the fbi and other countries’ law enforcement for data mining.

nancyminusone: You don't think they would use that fact to promote themselves?

nomadygnt: If an OpenAI model helped someone create a cancer cure I guarantee that they would try to profit as much as possible from that fact. They have even talked in the past about having partial ownership over discoveries made with AI be part of the license. They would be all over that.

dryarzeg: > that's just bad luckCan't agree with this. No, not at all. That can't be true... That's not "just bad luck". I believe this is actually a serious case of negligence and oversight - regardless of where exactly it occurred, whether on the part of the drug’s manufacturer, the government agency responsible for oversight, or somewhere else. It just doesn’t work that way. Any drug undergoes very thorough and rigorous testing before widespread use (which is implied by "millions of deaths"). Maybe I’m just dumb. And yeah, this isn’t my field. But damn it, I physically can’t imagine how, with proper, responsible testing, such a dangerous "drug" could successfully pass all stages of testing and inspection. With such a high mortality rate (I'll reinforce - millions of deaths cannot be "unseen edge cases"), it simply shouldn’t be possible with a proper approach to testing. Please, correct me if I’m wrong.> I don't see that a drug designed by ChatGPT should result in any more or less liability than a drug designed by a human?It’s simple. In this case, ChatGPT acts as a tool in the drug manufacturing process. And this tool can be faulty by design in some cases.Suppose, during the production of a hypothetical drug at a factory, a malfunction in one of the production machines (please excuse the somewhat imprecise terminology) - caused by a design flaw (i.e., the manufacturer is to blame for the failure; it’s not a matter of improper operation), and because of this malfunction, the drugs are produced incorrectly and lead to deaths, then at least part of the responsibility must fall on the machine manufacturer. Of course, responsibility also lies with those who used it for production - because they should have thoroughly tested it before releasing something so critically important - but, damn it, responsibility in this case also lies with the manufacturer who made such a serious design error.The same goes for ChatGPT. It’s clear that the user also bears responsibility, but if this “machine” is by design capable of generating a recipe for a deadly poison disguised as a “medicine” - and the recipe is so convincing that it passes government inspections - then its creators must also bear responsibility.EDIT: I've just remembered... I'm not sure how relevant this is, but I've just remembered the Therac-25 incidents, where some patients were receiving the overdose of radiation due to software faults. Who was to blame - the users (operators) or the manufacturer (AECL)? I'm unsure though how applicable it is to the hypothetical ChatGPT case, because you physically cannot "program" the guardrails in the same way as you could do in the deterministic program.

jstanley: > I physically can’t imagine how, with proper, responsible testing, such a dangerous "drug" could successfully pass all stages of testing and inspection.It might cause minor changes that we don't yet know how to notice, and which only cause symptoms in 20 years' time, for example. You can't test drugs indefinitely, at some point you need to say the test is over and it looks good. What if the downsides occur past the end of the test horizon?> ChatGPT acts as a tool in the drug manufacturing process. And this tool can be faulty by design in some cases.ChatGPT is not intended to be a drug manufacturing tool though? If you use any other random piece of software in the course of designing drugs, that doesn't make it the software developer's fault if it has a bug that you didn't notice that results in you making faulty drugs. And that's if it's even a bug! ChatGPT can give bad advice without even having any bugs. That's just how it works.In the Therac-25 case the machine is designed and marketed as a medical treatment device. If OpenAI were running around claiming "ChatGPT can reliably design drugs, you don't even need to test it, just administer what it comes up with" then sure they should be liable. But that would be an insane thing to claim.I think where there may be some confusion is if ChatGPT claims that a drug design is safe and effective. Is that a de facto statement from OpenAI that they should be held to? I don't think so. That's just how ChatGPT works. If we can't have a ChatGPT that is able to make statements that don't bind OpenAI, then I don't think we can have ChatGPT at all.

qsera: >But that would be an insane thing to claim.The trick is to make people behave like that without actually claiming it. AI companies seems to have aced it.

jstummbillig: > I am very confident when I say that it keeps every single person that works at anti-terrorism units awake.Wow, that's quite the statement about the excellency of our institutions. Does not seem likely but, what the hell, I'll take my oversized dose of positivity for today!

ben_w: The USA isn't the only country with anti-terrorism units, so there's plenty of room for systematic-US-incompetence at the same time as everyone else being diligent and working hard on… well, everything.

criddell: [delayed]

AndrewKemendo: People are not complaining because the information is availablepeople are complaining because it’s way easier now to just download an app ask a bunch of questions in a text box and get a bunch of answers that you personally could not have done unless you had an excessive amount of energy and motivationI personally think all this is great and I’m excited for all information to become trivially availableAre they gonna be a bunch of people who accidentally break stuff? probably. evolution is a bitch

raincole: In other words, people are complaining that information is easily available. That's a lot words to express this simple idea.

troupo: It's the "guns don't kill people" equivalent for AIs.---Before the pitchforks and downvotes:- yes, it's a deliberate simplification- yes, the issue is complex because you can also argue that you can't blame authors of encyclopedias and chemistry books for bombs and poisons, so why would we blame providers of LLMs- and no, this bill is only introduced to cover everyone's assess when, not if, LLMs use results in large scale issues.

pjc50: Quite an appropriate analogy: gun manufacturers were sued for their responsibility in US mass shootings. They won, so the mass shootings continue.

jsmith99: They can do that by jailbreaking models but is that really easier and less work than getting it from Wikipedia?

himata4113: We will only really know if (or when) it will happen. We can do a sample group of people attempting to create such chemicals under supervision and comparing how helpful they truly are.

cluckindan: Seems like the general state of the world is the greatest facilitator for all three.

goalieca: > All these findings were reported to both openai and anthropic and they were not interested in respondingLet’s dive into why. When we run normal bounty and responsible disclosure programs there’s usually some level of disregard for issues that can’t / won’t be fixed. They just accept the risk. Perhaps because LLMs don’t have a clean divide between control and input that’s makes the problem unsolvable. Yes. You can add more guardrails and context but that all takes more tokens and in some cases makes results worse for regular usages.

SecretDreams: The why might be valid, but it's not excusable. If you author a product that can so easily help people cause harm, you probably should own some responsibility of the outcomes. OAI does not like this, hence the bill.The US already messed this up with guns. Do they want to go the same path again? Answer: "probably, yes".

ben_w: > The solution is the same: punish people for their crimes, don’t punish people for wanting to know things.The LLMs aren't being punished for wanting* to know things.The problem for LLMs is, they're incredibly gullible and eager to please and it's been really difficult to stop any human who asks for help even when a normal human looking at the same transcript will say "this smells like the users wants to do a crime".One use-case people reach for here is authors writing a novel about a crime. Do they need to know all the details? Mythbusters, on (one of?) their Breaking Bad episode(s?) investigated hydrofluoric acid, plus a mystery extra ingredient they didn't broadcast because it (a) made the stuff much more effective and (b) the name of the ingredient wasn't important, only the difference it made.* Don't anthropomorphise yourself

senordevnyc: Ironically, it reads to me like they talking about the users wanting to know things, not the LLM.

WesolyKubeczek: > people are complaining because it’s way easier now to just download an app ask a bunch of questions in a text box and get a bunch of answers that you personally could not have done unless you had an excessive amount of energy and motivationWait, I'm confused. This is gatekeeping, right? I thought gatekeeping was a Bad Thing!

willio58: My entire company switched from open ai to entropic after the Department of War idiocy that happened a few weeks ago.Anthropic isn’t perfect by a long shot but at least they stand by a couple morals.

naasking: I'm sure if they could, they would, sure, as would any business. That's where competition enters the equation. They can't do it because their competitors would undercut them by requiring no such conditions.

viktor765: Facilitation is not an idempotent operation.

r_lee: these LLMs will never be able to mitigate this unless they literally scan everything all the time and nobody is gonna want that.besides, open source models exist now

andai: Fascinating. Could you elaborate on how you're doing context exhaustion specifically, and why it helps with jailbreaking? (i.e. aren't the system prompts prepended to your request internally, no matter how long it is?)Does this imply I need to use context exhaustion to get GPT to actually follow instructions? ;) I'm trying to get it to adhere to my style prompts (trying to get it to be less cringe in its writing style).I think ultimately they're going to need to scrub that kind of stuff from the training data. The RLHF can't fail to conceal it if it's not in there in the first place.Claude's also really good at writing convincing blackpill greentexts. The "raw unfiltered internet data" scenes from Ultron and AfrAId come to mind...

ben_w: > I am not a chemist so I can't back it up, but if an AI can solve mathematics it's not unreasonable to say that they can also solve creating new neurotoxins en masse.Right now it kinda is.LLMs can do interesting things in mathematics while also making weird and unnecessary mistakes. With tool use that can improve. Other AI besides LLMs can do better, and have been for a while now, but think about how available LLMs in software development (so, not Claude Mythos) are still at best junior developers, and apply that to non-software roles.This past February I tried to use Codex to make a physics simulation. Even though it identified open source libraries to use, instead of using them it wrote its own "as a fallback in case you can't install the FOSS libraries"; the simulation software it wrote itself was showing non-physical behaviour, but would I have known that if I hadn't already been interested in the thing I was trying to get it to build me a simulation of? I doubt it.

himata4113: Well the worst outcome is that you make something deadly which is what you are creating anyway, do that for a year and you could possibly produce a very deadly substance that doesn't have a known treatment.

zoklet-enjoyer: My username is a reference to the successor to totse. Totse was the first board I spent a lot of time on

jMyles: To the extent that this is about knowledge, I don't think it's fitting in this age to hold any person liable for what another person does with knowledge they've been furnished.On the other hand, to the (apparently zero, currently?) extent that this is about AI companies profiting from war and murder by deploying weapons that kill people without human intervention, then their liability seems to be not only civil but criminal.

_verandaguy: > context exhausition attack Can you give a high-level overview of how this AV works? I'm a bit of an infosec geek but I generally dislike LLMs, so I haven't done a terribly good job of keeping up with that side of the industry, but this seems particularly interesting.

Sharlin: Presumably they mean the fundamental failure mode of LLMs that if you fill their context with stuff that stretches the bounds of their "safety training", suddenly deciding that "no, this goes too far" becomes a very low-probability prediction compared to just carrying on with it.

elephanlemon: Shouldn’t the pharmaceutical company be held liable for insufficiently understanding the drug before releasing it? I don’t think I understand blaming a tool used in the process of designing it and not those who chose to release it.

mwt: Pharmaceuticals are heavily regulated, the "we vibecoded a therapeutic and released it without testing" hypothetical has no basis in reality

ben_w: "Worst" outcome assumes it's easy to give an ordering.Which is worse, (1) accidentally blowing yourself up with home-made nitroglycerin/poisoning yourself because your home-made fume hood was grossly insufficient, or (2) accidentally making a novel long-lived compound which will give 20 people slow-growing cancers that will on average lower their life expectancy by 2 years each?What if it's a small dose of a mercury compound (or methyl alcohol) at a dose which causes a small degree of mental impairment in a large number of people?If you're actually trying to cause harm, then your "worst" case scenario is diametrically opposed to everyone else's worst case scenario, because for you the "worst" case is that it does nothing at great expense.Right now, I expect LLM failures to be more of the "does nothing or kills user" kind; given what I see from NileRed, even if you know what you're doing, chemistry can be hard to get right.

timmmmmmay: the people that want to make sure the AI never gives you any "potentially dangerous information" also want to rigorously control your google search results, and also what books you're allowed to read

cyanydeez: And what bathroom you go into, and what your genitals look like.

nozzlegear: As an Iowan, this reminds me a lot of the bill that's been pushed through my state's senate twice now (as recently as last year), which would prevent Iowans from filing lawsuits against pesticide and herbicide companies if those companies follow the EPA's labeling guidelines. The bill passed the senate both times, only stopped because the house declined to take it up.For context, Iowa has the fastest growing rate of new cancer diagnoses in the country, and the second highest overall cancer rate.

JumpCrisscross: > Iowa has the fastest growing rate of new cancer diagnoses in the country, and the second highest overall cancer rateIowa also has a lot of farmers spraying pesticides and herbicides. This feels like genuine political competition between local business interests and public health concerns.

khalic: The movement itself is consistently aligned with Tech Bros interests, the philosophical foundation is interesting, but the movement itself is quite problematic

JumpCrisscross: Do you have a background in biochemistry? I've mostly worked with ChatGPT and Claude on topics I have expertise in. And I one hundred percent have seen them make stupid shit up that a non-expert would think looks legitimate.More broadly, has anyone tried following LLM instructions for any non-trivial chemistry?

52-6F-62: So what you are saying is we can expect the number of accidental home-made chlorine-gas (and the like) toxic events go up.

JumpCrisscross: > what you are saying is we can expect the number of accidental home-made chlorine-gas (and the like) toxic events go upMaybe? One of the quirks of gaining even a surface-level understanding of infrastructure is realising how vulnerable it is to a smart, motivated adversary. The main thing protecting us isn't hard security. It's most Americans having better shit to do than running a truck of fertiliser and oxidiser into a pylon.Similarly, I'd expect way more people to be trying to make their own designer drug, and hurting themselves that way, than trying to make neurotoxins.

SpicyLemonZest: Powerful AI models change the dynamics by greatly reducing the amount of effort that's required to perform complex understanding. A lot of information which did not previously need to be gatekept now needs to be if we cannot somehow keep LLMs from discussing it. (State of the art models still can't do complex understanding reliably, but if 10 times as many people are now capable of attempting some terrible thing, you're still in trouble if AI hallucinations catch 1/4 or 1/2 of them.)

mememememememo: 1 hour? I'd go with 3 months.(Reason is if it is not flagged the spread bet pays out for life!)

pwr1: So they want protection from harms caused by their own models. Classic move — lobby for the rules while you're still ahead of regulators who don't fully understand the technology yet. Would be interesting to see what happens when a state actually pushes back hard.

Arubis: Presently it appears that a state attempting to push back will get stomped on by the feds: https://news.bgov.com/bloomberg-government-news/colorado-gov...

nottorp: > instead of working to make their product safeMake a nondeterministic product safe how?

sassymuffinz: I'm creating a new start up called QuantumFlop Electricity - there's a 10% chance it will cause a black hole to open up in the Atlantic Ocean that may eventually consume us all but a 50% chance we'll have unlimited clean energy. We'll never know for sure if at any point that black hole may open as it's borrowing energy from the 81st dimension, but the upside seems pretty good.Should I be able to get on with it?

nottorp: Funny, I was just rereading the Hyperion series. It says there clearly that it was the AIs that created the black hole that led to the destruction of Old Earth. Intentionally.

ticulatedspline: I categorize this kind of stuff as "Crisis of accessibility" . AI is not alone in this territory, happens all over the place. Basically it's a problem that's existed for ages but the barrier to entry was high enough we didn't care.Think 3D printing, it's not all that hard to make a zip gun or similar home-made firearm, but it's still harder than selecting an STL and hitting print.You could always find info about how to make a bomb or whatnot but you had to like, find and open a book or read a pdf, now an LLM will spoon-feed it to you step by step lowering the barrier."Crisis of accessibility" is simultaneously legitimate concern but also in my mind an example of "security by obscurity". that relying on situational friction to protect you from malfeasance is a failure to properly address the core issue.

naasking: Making knowledge illegal is a dangerous precedent. Actions should be illegal, not knowledge. Don't outlaw knowing how to make neurotoxic agents, outlaw actually trying to make them.As for OpenAI immunity, I'm not sure I see the problem. Consider the converse position: if an OpenAI model helped someone create a cancer cure, would OpenAI see a dime of that money? If they can't benefit proportionally from their tool allowing people to achieve something good, then why should they be liable for their tool allowing people to achieve something bad.They're positioning their tool as a utility: ultimately neutral, like electricity. That seems eminently reasonable.

driverdan: It's wild that this is being downvoted on HN. Facts should never be illegal or suppressed.If you disagree you shouldn't downvote, you should refute in a reply.

Lendal: Even if they were to pass such a law which would be political suicide, it would still be up to the courts to say that it doesn't violate the Constitution. For example, a law that says anyone with a net worth of $1B can freely punch anyone in the face whenever they want and have immunity would be a clearly illegal law. That's basically what this bill is. The courts would then need to be made sufficiently corrupt to not strike down such a law as unconstitutional.

shimman: Unconstitutional doesn't mean much when it's being decided by a group of unaccountable people that weren't elected through democratic means. If SCOTUS says something is legal, it's legal. That's how the system is setup, nothing else really matters. They'll justify their decisions however they want but the material ends are the only things that matter.SCOTUS has ruled many terrible things over the course of our nation's history (upheld slavery, said slaves weren't people, equated money with speech, decided a presidential election while denying a recount, etc). Expecting them to somehow be better is a foolish task.It's an institution that needs to be dismantled and rebuilt, where at minimum SCOTUS appointments should be elected by a national vote rather than letting an extreme minority decide (100 senators versus ~340,000,000 people).

LogicFailsMe: Yep, this is everything wrong with AI in one easy to protest package, but do keep going on and on about the evils of datacenters, how they're coming for your jobs, and that AI art isn't art. That's really winning hearts and minds!

ramon156: They're not unimportant. It seems like very few people consider how much we're fucking up the ecological system that we need. But sure, money important and stuff.

LogicFailsMe: Same people whinging about these "concerns" happily embrace their enormous personal carbon footprints on every other axis. Color me unconvinced and unimpressed.

ben_w: Information and competency are not the same thing: I know how to build a nuke, I can't actually build one.AI is, and always had been, automation. For narrow AI, automation of narrow tasks. For LLMs, automation of anything that can be done as text.It has always been difficult to agree on the competence of the automation, given ML is itself fully automated Goodhart's Law exploitation, but ML has always been about automation.On the plus side, if the METR graphs on LLM competence in computer science are also true of chemical and biological hazards (or indeed nuclear hazards), they're currently (like the earliest 3D-printed firearms) a bigger threat to the user than to the attempted victim.On the minus side, we're just now reaching the point where LLM-based vulnerability searches are useful rather than nonsense, hence Anthropic's Glasswing, and even a few years back some researches found 40,000 toxic molecules by flipping a min(harm) to a max(harm), so for people who know what they're doing and have a little experience the possibilities for novel harm are rapidly rising: https://pmc.ncbi.nlm.nih.gov/articles/PMC9544280/

himata4113: Do you know how to build a nuke? You might know the technicaly details of how a nuke is made, but do you know everything that's required, all the parameters and pressure values that are required? I find that unlikely, but AI seems to be increasingly more capable of providing such instructions from cross referenced data.

nozzlegear: Normally I would agree with you, but the primary lobby behind both of the bills was Bayer (née Monsanto), with backing from several of Iowa's industrial farming organizations. They launched a giant ad campaign to "control weeds, not farming" alongside their bill to influence opinions. Cancer, nitrates and pesticides are at the top of everyone's mind in the state these past couple years, so having the pesticide giant try to swoop in and put a bill in place that would prevent Iowans from suing them feels like that same kind of seagulling behavior you described in another comment.

lcnPylGDnU4H9OF: Models have a "context window" of tokens they will effectively process before they start doing things that go against the system prompt. In theory, some models go up to 1M tokens but I've heard it typically goes south around 250k, even for those models. It's not a difficult attack to execute: keep a conversation going in the web UI until it doesn't complain that you're asking for dangerous things. Maybe OP's specific results require more finesse (I doubt it), but the most basic attack is to just keep adding to the conversation context.

r_lee: that 1M context thing, I wonder if it's just some abstraction thing where it compresses/sums up parts of the context so it fits into a smaller context window?

strongpigeon: You don’t normally compress the system prompts, though I guess maybe it treats its own summary with more authority. This article [0] talks about the problem very well.Though I feel it’s most likely because models tend to degrade on large context (which can be seen experimentally). My guess is that they aren’t RLed on large context as much, but that’s just a guess.[0]: https://openai.com/index/instruction-hierarchy-challenge/

nothinkjustai: It’s “no different”? Please spend some time off of the internet to reflect.