Discussion

haarlemist: I

Someone1234: They cannot.Unfortunately many believe they can, and it is impossible to disprove. So now real people need to write avoiding certain styles, because a lot of other people have decided those are "LLM clues." Bullets, EM Dash, certain common English phases or words (e.g. Delve, Vibrant, Additionally, etc)[0].Basicaly you need to sprinkle subtle mistakes, or lower the quality of your written communications to avoid accusations that will side-track whatever youre writing into a "you're a witch" argument. Ironically LLM accusations are now a sign of the high quality written word.[0] https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing

moonu: Pangram is probably the best known example of a detector with low false positives, they have a research paper here: https://arxiv.org/pdf/2402.14873. They do have an API but not sure if you need to request access for it.For humans I think it just comes down to interacting with LLMs enough to realize their quirks, but that's not really fool-proof.

dipb: Humans detect them mostly through pattern matching. However, for systems, my guess is that a ML model is trained on AI genres texts to detect AI generated texts.

mjlee: People Look For:Specific language tells, such as: unusual punctuation, including em–dashes and semicolons; hedged, safe statements, but not always; and text that showcases certain words such as “delve”.Here’s the kicker. If you happen to include any of these words or symbols in your post they’ll stop reading and simply comment “AI slop”. This adds even less to the conversation than the parent, who may well be using an LLM to correct their second or third language and have a valid point to make.

alex43578: Someone with native fluency in American English can (should) be able to tell the difference between human writing and unpolished AI copy-paste.Essentially 0 people use emoji to create a bulleted list. Nobody unintentionally cites fake legal precedents or non-existent events, articles, or papers. Even the “it’s not X, it’s Y” structure, in the presence of other suspicious style/tone cues signals LLM text.

loloquwowndueo: The key insight is to avoid – em dashes. You’re absolutely right. It’s not the content, it’s the style.

sanex: Ironically one of the big tells for me is the "It's not this. It's that." Your comment uses a comma though so you're probably a real person :)

LoganDark: That's an en-dash.

prmph: [delayed]

loloquwowndueo: Sorry! Is this ok? —

loloquwowndueo: Busted!!!!Staccato (too may short sentences with periods) is also a telltale for me. Most humans prefer longer sentences with more varied punctuation; I, for example, am a sucker for run-on sentences.

fortran77: And I'm sure we've all seen what happens if you run the Declaration of Independence or the Gettysburg Address or the book of Genesis through an AI "detector". They usually come back as AI.

spindump8930: Only for poor quality systems. Unfortunately there are many systems that tried to make easy hype, but are the equivalent of an ML 101 classifier class project.If one measures for perplexity (how likely text is under a certain language model), common text in a training set will be very likely. But you can easily create better models.

roncesvalles: Exactly, it's the monotony of the style that gives it away.

kaindume: I believe if you have access to the training data of the specific LLM and the generated text is long enough, using statistics you might be able to tell if its LLM generated.I am writting an LLM captcha system, here is the proof of concept: https://gitlab.com/kaindume/llminate

Joel_Mckay: Indeed, isomorphic plagiarism by its nature forms strong vector search paths that were made from stealing both global websites, real peoples work, and LLM user-base input/markdown.However, reasoning models adding a random typo to seem less automated, still do not hide the fairly repeatable quantized artifacts from the training process. For LLM, it is rather trivial to find where people originally stole the training data from if they still have annotated training metadata.Finally, reading LLM output is usually clear once one abandons the trap of thinking "I think the author meant [this/that]", and recognizing a works tone reads like a fake author had a stroke [0]. =3[0] https://en.wikipedia.org/wiki/Stroke

mulr00ney: > Unfortunately many believe they can, and it is impossible to disprove. So now real people need to write avoiding certain styles, because a lot of other people have decided those are "LLM clues." Bullets, EM Dash, certain common English phases or words (e.g. Delve, Vibrant, Additionally, etc)[0].I think people will be able to detect the lowest-user-effort version of LLM text pretty reliably after a while (ie what you describe; many people have a good sense of LLM clues). But there's probably a *ton* of LLM text out there where some of the instructions given were "throw a few errors in", "don't use bullet points or em dashes", "don't do the `it's not this, it's that` thing" going undetected.And then those changes will get built into ChatGPT's main instructions, and in a few months people will start to pick up on other indicators, and then slightly smarter/more motivated users will give new instructions to hide their LLM usage... (or everyone stops caring, which is an outcome I find hard to wrap my head around)

RestartKernel: People look for tells, systems detect word distributions. Though neither is as reliable as active fingerprinting using an encoded watermark.

dezgeg: For HN comments, the LLMs seem to really like 2 or 3 paragraphs long responses. It's pretty obvious when you click a profile's comments and see every comment being that exact same structure.

leumon: You can try to use an ai detector, here is a leaderboard of the best ones according to this benchmark: https://raid-bench.xyz/leaderboard Results should of course always be taken with a grain of salt, but in most cases detectors are quite good in my opinion.

alex43578: One of my subtle favorites is the “H2 Heading with: Colorful Description”Eg - The Strait of Hormuz: Chokepoint or Opportunity?

Filligree: I’ve used titles like that for thirty years.

block_dagger: Em dashes, “it’s x, not y”, excessive emojis and arrows.

mghackerlady: Especially where the emoji serves practically no purpose other than to get your attention. If it is especially abstract what the emoji is there to represent, I start looking for other signs

tatrions: The principled approaches are statistical. Things like DetectGPT measure per-token log probability distributions. LLM text clusters tightly around the model's typical set, human writing has more variance (burstiness). Works decently when you know the model and have enough text, breaks down fast otherwise.Stylistic tells like 'delve' and bullet formatting are just RLHF training artifacts. Already shifting between model versions, compare GPT-4 to 4o output and the word frequency distributions changed noticeably.Long term the only thing with real theoretical legs is watermarking at generation time, but that needs provider buy-in and it slightly hurts output quality so adoption has been basically nonexistent.

rwc: Contrastive negation continues to be a dead giveaway.

mghackerlady: Overuse of "it's not X, it's Y" kind of writing, strange shifts in writing or thinking patterns, and excessive formatting (or, when I'm on wikipedia especially, ineffective formatting (such as using MD where it isn't supported))

EagnaIonat: > 0 people use emoji to create a bulleted list.I haven't seen this yet, but I guess the only reason I haven't done it is because it never crossed my mind.What I have found an easy detection is non-breaking spaces. They tend to get littered through the passages of text without reason.

singpolyma3: You're absolutely right. That is an em dash

LoganDark: You're absolutely right. They are absolutely right

fwip: You can smell it.

lelanthran: > Ironically LLM accusations are now a sign of the high quality written word.Citation needed. The LLM accusations come from the specific cadence they use. You can remove all em-dashes from a piece of text and it still becomes clear when something is LLM written.Can they be prompted to be less obvious? Sure, but hardly anyone does that.It's more "The Core Insight", "The Key Takeaway", etc. than it is about emdashes.Incidentally, the only people annoyed about "witch-hunts" tend to be those who are unable to recognise cadence in the written word.

order-matters: i think another part of the problem is that some people are using AI so much that they are starting to mimic its cadence in their own writing. they may have had a prior coincidental predisposition for writing somewhat similar to AI with worse grammar, and now are inching towards alignment as they either intentionally or accidentally use AI output as a model to improve their writing