Discussion
The AI Revolution in Math Has Arrived
claysmithr: I wonder when AI will be able to discern the passage of time
maplethorpe: Altman has estimated one year until ChatGPT is capable of measuring time passed.https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-fails-mis...
VladVladikoff: Can’t tell if you are being sarcastic but Altman’s whole job is to make bullshit near future predictions about rapid development of AI in the public.
themafia: There are several high value prizes for mathematical research. Let me know when an "AI" has earned one of them. Otherwise:> When Ryu asked ChatGPT, “it kept giving me incorrect proofs,” [...] he would check its answers, keep the correct parts, and feed them back into the modelSo you had a conversational calculator being operated by an actual domain expert.> With ChatGPT, I felt like I was covering a lot of ground very rapidlyThere's no way to convert that feeling into a measurement of any actual value and we happen to know that domain experts are surprisingly easy to fool when outside of their own domains.
ambicapter: Sounds like Musk setting deadlines for Mars landings.
random__duck: Thankyou for stating the obvious, for some reason we need to repeat this. ^^;
dogscatstrees: > As they did so, they also learned how to improve the prompts they gave AlphaEvolve. One key takeaway: The model seemed to benefit from encouragement. It worked better “when we were prompting with some positive reinforcement to the LLM,” Gómez-Serrano said. “Like saying ‘You can do this’ — this seemed to help. This is interesting. We don’t know why.”Four top logical people in the world are acknowledging this. It is mind-blowing and we don't know why.
brookst: Do we know why it works for humans?Models are trained on human outputs. It’s not super surprising to me that inputs following encouraging patterns product better results outputs; much of the training material reflects that.
zarzavat: It makes sense to me.Originally LLMs would get stuck in infinite loops generating tokens forever. This is bad, so we trained them to strongly prefer to stop once they reached the end of their answer.However, training models to stop also gave them "laziness", because they might prefer a shorter answer over a meandering answer that actually answered the user's question.Mathematics is unusual because it has an external source of truth (the proof assistant), and also because it requires long meandering thinking that explores many dead ends. This is in tension with what models have been trained to do. So giving them some encouragement keeps them in the right state to actually attempt to solve the problem.
dataviz1000: I know why.Several people had problems with Sonnet burning through all their credits grinding on a problem it can't solve. Opus fixes this — it has a confidence threshold below which it exits the task instead of grinding."I spent ~$100 last week testing both against multiplication. Sonnet at 37-digit × 37-digit (~10³⁷) never quits — 15+ minutes, 211KB of output, still actively decomposing numbers when I stopped it. Opus will genuinely attempt up to ~50 digits (112K tokens on a real try), starts doubting around 55 digits, and by 80-digit × 80-digit surrenders in 330 tokens / 9 seconds with an empty answer." -- Opus, helping me with the dataThe "I don't think this is worth attempting" heuristic is the difference. Sonnet doesn't have it, or has it set much higher. In order to get Opus and some other models to work on harder problems that it assumes it is not worth attempting, it requires to increase the confidence level.I'll finish writing this up this week. I'm making flashy data visual animations to make the point right now.
CivBase: This seems pretty obvious, no?It's pattern matching on training material. There is almost certainly an overlap between positivity and success in the training material. Positive prompts cause the pattern matching to weight towards positivity and therefor more successful material.
gxs: Wow that was your takeaway?> “2025 was the year when AI really started being useful for many different tasks,” said Terence TaoI think I’ll go out on a limb and agree with Terrence Tao, I think the dude is well known in the math community, or something
noobermin: If anything his simping for AI models makes me more suspect of him than I ever was because my own eyes show me their limits.
p1dda: I think he means useful for mathematicians getting paid shilling for AI models
themafia: > go out on a limb and agree with Terrence TaoIs AI his specialty?> I think the dude is well known in the math community, or somethingI believe this is called "appeal to authority." Which is why, instead of disagreeing with him, I suggested a more cogent endpoint that could be used to establish the facts the article's title suggests.
jryle70: Any chance your eyes are wrong? Or only people who disagree with you are.
sm0ss117: Mathematics seems like the ideal candidate for AIs to achieve absurd results. It's a purely abstract grammar with true auto-verifiability. Even SWE has the requirement of interacting with real physical things. In math there's no external feedback required, you're solely bounded by the rate and quality of token generation.
yabutlivnWoods: We can define a Dyson Sphere in math.We cannot build one.AI outputting axiomatically valid syntax isn't going to be all that useful. It's possible to generate all axiomatically correct math with a for loop until the machine OOMsPhysics is not math and math is not physics.
bgirard: Last week I got together with my math alumni friend. We cracked some beers, we chatted with voice mode ChatGPT and toyed around with Collatz Conjecture and we sent some prompt to a coding agent to build visualizations and simulation. It was a lot of fun directing these agents while we bounced off ideas and the models could explore them.I think with the right problem and the right agentic loop it’s clear to me improvements will speed up.
drivebyhooting: This misses the mark on at least two accounts: 1. Proofs without human understanding have less value for mathematicians 2. At least for now, interestingness depends on human judgment. It is subjective and not as verifiable.
dyauspitr: Every new mathematician that comes along doesn’t know everything that has come before him. He needs to go learn all the math that his predecessors did. I don’t see how an LLM coming up with these proofs changes that.
drakenot: I think voice mode uses weaker models, just an FYI relative to the SOTA
djsjajah: You just failed the Turing test.
viccis: There's no need to "estimate" it. "Time" is not something built into training and sampling a generative distribution. He might as well have told you your Naive Bayes email filters will measure time passed.
streb-lo: Because the problem space is basically infinite. If a person is working on a problem, its probably interesting to at least one person. Randomly walking through the problem space might be interesting, but I don't know how the signal will fare against other humans.
meroes: Grammar seems like you’re talking about LLMs specifically. Well, isn’t Sudoku just math? LLMs suck at Sudoku last I checked. When told not to code a solver, its very first deduction was wrong.
keyle: Maybe he passed the Turing test with 88.2% which is 1.8% higher than the competition.
