Discussion
Why We Stopped Using the Mathematics That Works
jeffrallen: [delayed]
ontouchstart: We are at the age of alchemy, wait for the age of chemistry and physics. New mathematical foundations are yet to be found.
throwaway132448: I found the article confusing. Its premise seems to be that alternative methods to deep learning “work”, and only faded out due to other factors, yet keeps referencing scenarios in which they demonstrably failed to “work”. Such as:> In 2012, Alex Krizhevsky submitted a deep convolutional neural network to the ImageNet Large Scale Visual Recognition Challenge. It won by 9.8 percentage points over the nearest competitor.Maybe there’s another definition of “works” that’s implicit and I’m not getting, but I’m struggling to picture a definition relevant to the history-of-deep-learning narrative they are trying to explain.
deckar01: It seems to be an indirect attempt to promote their GitHub project. They had Claude make them an “agent” using Bayesian modeling and Thompson sampling and now they are convinced they have heralded a new era of AI.
LoganDark: I think what they're saying is the methods used today are faster but have a lower ceiling, and that that's why they quickly took over but can only go so far.
andai: This means money beats math?
furyofantares: LLM-garbage article, ironically.
andai: What makes you say that? Which LLM does it sound like to you?
xg15: The paragraph "The ImageNet Moment" stuck out to me. It's so stuffed of the current AI-isms that I have a hard time seeing this as chance.
andai: See also: https://gfrm.in/posts/agentic-ai/> I’ve spent the last few months building agents that maintain actual beliefs and update them from evidence — first a Bayesian learner that teaches itself which foods are safe, then an evolutionary system that discovers its own cognitive architecture. Looking at what the industry calls “agents” has been clarifying.> What would it take for an AI system to genuinely deserve the word “agent”?> At minimum, an agent has beliefs — not hunches, not vibes, but quantifiable representations of what it thinks is true and how certain it is. An agent has goals — not a prompt that says “be helpful,” but an objective function it’s trying to maximise. And an agent decides — not by asking a language model what to do next, but by evaluating its options against its goals in light of its beliefs.> By this standard, the systems we’re calling “AI agents” are none of these things.
kingstnap: Just because you can analyse it doesn't mean that it is better. Deep learning theory is unbelievably garbage compared to the empirical results.In particular, please show me a worked example of a decision tree meta learning. Because its trivial to show this for DNNs.
NateEag: It means trying to figure out how to build an intelligence always loses to mindlessly brute-forcing problems with more compute:https://en.wikipedia.org/wiki/Bitter_lesson
notlenin: unless you don't have unlimited compute, at which point you need other ideashttps://arielche.net/bitter-lesson
jerf: That would be a hypothesis, not a fact.I'm not closed to it. You can check my comment history for frequent references to next-generation AIs that aren't architected like LLMs. But they're going to have to produce an AI of some sort that is better than the current ones, not hypothesize that it may be possible. We've got about 50 years of hypothesis about how wonderful such techniques may be and, by the new standards of 2026, precious few demonstrations of it.Quoting from the article:"Within five years, deep learning had consumed machine learning almost entirely. Not because the methods it displaced had stopped working, but because the money, the talent, and the prestige had moved elsewhere."That one jumped right out at me because there's a slight-of-hand there. A more correct quote would be "Not because the methods it displaced had stopped working as well as they ever have, ..." Without that phrase, the implication that other techniques were doing just as well as our transformer-based LLMs is slipped in there, but it's manifestly false when brought up to conscious examination. Of course they haven't, unless they're in the form of some probably-beyond-top-secret AI in some government lab somewhere. Decades have been poured into them and they have not produced high-quality AIs.Anyone who wants to produce that next-gen leap had probably better have some clear eyes about what the competition is.
LoganDark: > That would be a hypothesis, not a fact.I agree.
bpt3: Then train your model elsewhere and size it as appropriate for the runtime environment.If that really isn't an option, then yes ML/AI isn't for you in this case.
10xDev: They have researchers working for insane salaries just so they don't go to another frontier lab to share their ideas. If you think it is just "mindless bruteforce" you don't understand anything. The idea is that the most effective methods are ones that scale but those ideas are also then limited by the compute available.
canjobear: > Not because the methods it displaced had stopped working, but because the money, the talent, and the prestige had moved elsewhere. The researchers who understood decision theory, Bayesian inference, and operations research didn’t lose their arguments. They lost their audience.
naasking: So, what's the problem with it?
vessenes: Heading down the links of this blog ends up at https://github.com/gfrmin/credence, which claims to be an agentic harness that keeps track of usefulness of tools separately and beats LangChain at a benchmark.LangChain… Now that’s a name I haven’t heard in a long, long time..Anyway, that’s a cool idea. But also his blog posts include phrases like “That’s not intelligence, it’s just <x> with vibes.” Urg. Slop of the worst sort.But, like I said, I like the idea of keeping a running tally of what tool uses are useful in which circumstances, and consulting the oracle for recommended uses. I feel slightly icky digging into the code though; there’s a type of (usually brilliant) engineer that assumes when they see success that it’s a) wrong, and b) because everybody’s stupid, and sadly, some of that tone comes through the claude sonnet 4.0 writing used to put this blog together.
naasking: > But also his blog posts include phrases like “That’s not intelligence, it’s just <x> with vibes.” Urg. Slop of the worst sort.You know people actually write like that. The LLMs learned it from somewhere.
LolWolf: yeah it’s slop; multiple sites also confirm this, seems like they are using clawdbot or whatever
jebarker: I’m not convinced that useful mathematical foundations will be found anytime soon. Neural nets exist because we want to make decisions in a world that is so noisy, complex and chaotic that we can’t satisfy the requirements of more rigorous analytical frameworks. It seems to me that the irreducible complexity is in the real world, not the neural networks.
pron: > This is the VHS-versus-Betamax dynamic, or TCP/IP versus the OSI model, or QWERTY versus every ergonomic alternative proposed since 1936. The technically superior solution loses to the solution that’s easier to deploy, easier to hire for, and good enough for the use cases that pay the bills.Without commenting on the merit of the claims, the problem with this statement is that in many cases there is no universal "technical superiority", only tradeoffs. E.g. Betamax was technically superior in picture quality while VHS was technically superior in recording time, and more people preferred the latter technical superiority. When people say that the techinically superior approach lost in favour of convenience, what really happened is that their own personal technical preferences were in the minority. More people preferred an alternative that wasn't just "good enough" but technically better, only on a different axis.Even if we suppose the author is right that his preferred approach yields better outputs, he acknowledges that constructing good inputs is harder. That's not technical superiority; it's a different tradeoff.
kjshsh123: Pretty much although it's less about majority/minority preference and more about utilitarianism and economics.It's possible a majority of pofeople would have been marginally happier with betamax than vhs. Even in that case, vhs can still win because a minority of people had a strong, stubborn preference for it, even if a majority of people had a weak preference for betamax.If 1,000,000 people are willing to pay $5 more for video quality but 800,000 people are willing to pay $8 more for longer recording, which wins out?Not to mention savings on the producer side are relevant too, not just consumer side.I'm not saying the above is necessarily the case. Just pointing out that markets aren't majoritarian, they're utilitarian.
PaulHoule: I think the worst thing about the golden age of symbolic AI was that there was never a systematic approach to reasoning about uncertainty.The MYCIN system was rather good at medical diagnostics and like other systems of the time had an ad-hoc procedure to deal with uncertainty which is essential in medical diagnosis.The problem is that is not enough to say "predicate A has a 80% of being true" but rather if you have predicate A and B you have to consider the probability of all four of (AB, (not A) B, A (not B), (not A) (not B)) and if it is N predicates you have to consider joint probabilities over 2^N possible situations and that's a lot.For any particular situation the values are correlated and you don't really need to consider all those contingencies but a general-purpose reasoning system with logic has to be able to handle the worst case. It seems that deep learning systems take shortcuts that work much of the time but may well hit the wall on how accurate they can be because of that.[1] https://en.wikipedia.org/wiki/Mycin
zozbot234: Symbolic AI ala Mycin and other expert systems didn't do anything that a modern database query engine can't do with far greater performance. The bottleneck is coming up with the set of rules that the system is to follow.
PaulHoule: Early production rules engines really sucked, like a lot of the time they didn't have any kind of indexes and full scanned a lot. Good RETE engines with indexes didn't get mainstream by the 1980s but the industry was already losing interest. In a lot of wayshttps://en.wikipedia.org/wiki/Droolsis pretty good as is the Jena rules engine but none of these have ways of dealing with uncertainty which are necessary if you're going to be working with language and having to decide which of 10,000 possible parses is right for a sentence. People used to talk as if 10,000 rules was a lot but handling 2 million well-organized rules with Drools is no problem at all today.I think the problems of knowledge base construction are overstated and that a lack of tools are the problem. Or rather, the Cyc experience shows that rules are not enough, that is, after Lenat died it got out that Cyc didn't just have a big pile of facts and rules and a general reasoning procedure but it had a large database of algorithms to solve specific problems. That is, in principle you can solve anything with an SMT solver but if you actually try it you'll find you can code up a special-purpose algorithm to do common tasks before the SMT solver really gets warmed up.Part of the production rules puzzle is that there never was a COBOL of business rules rather you got different systems which took different answers to various tricky problems like how to control the order of execution when it matters, how to represent negation, etc.
