Discussion

The Beginning of Scarcity in AI

stupefy: What limits LLM inference accelerators? I heard about Groq (https://groq.com/) not sure how much it pushes away the problem.

vessenes: ASML only makes a certain number of machines a year that can do extreme ultra-violet lithography.Also - turbine blades limit power, according to Elon.Between them - we cannot chip fabs past a certain rate, and we cannot stand up the datacenter to run these desired chips past a certain rate. Different people believe one or the other is the 'true' current bottleneck. The turbine supply chain scaling looks much more tractable -- EUV is essentially the most complicated production process humans have ever devised.

mattas: This notion that "we don't have enough compute" does not cleanly reconcile with the fact that labs are burning cash faster than any cohort of companies in history.If I am a grocery store that pays $1 for oranges and sells them for $0.50, I can't say, "I don't have enough oranges."

vessenes: It seems very possible that we have at least five years of real limitations on compute coming up. Maybe ten, depending on ASML. I wonder what an overshoot looks like. I also wonder if there might be room for new entrants in a compute-scarce environment.For instance, at some point, could Coreweave field a frontier team as it holds back 10% of its allocations over time? Pretty unusual situation.

FloorEgg: There is a major logic flaw in what you're saying.'If I am a grocery store that pays $1 for oranges and sells them for $0.50, I can't say, "I don't have enough oranges."'How about 'if I'm a grocery store and I see no limit on demand for oranges at $.50 but they are currently $1, I can say 'if oranges were cheaper I could sell orders of magnitude more of them'.Buying oranges for $1 and selling for $0.5 is an investment into acquiring market share and customer relationships and a gamble on the price of oranges falling in the future.

0x3f: > acquiring market share and customer relationshipsThe whole setup rests on this, and it seems mythical to me. These guys have basically equivalent products at this point.

earthnail: If there were more oranges you’d pay less to buy them and your economics would work out.

0x3f: Not sure if this is a joke or not, but competitive pressure still exists. This only really holds if you're the only orange seller.

isawczuk: It's artificial scarcity. LLM inference will soon be commodity as cloud.There is a 2-3years still before ASIC LLM inferences will catch up.

vessenes: I don't think so. GB200 prices are GOING UP. A100s are still expensive. This implies massive utilization and demand, no? These machines are not sitting idle, or prices would drop in the very competitive hyperscaler environment.

dmazin: Constraints can lead to innovation. Just two things that I think will get dramatically better now that companies have incentive to focus on them:* harness design* small models (both local and not)I think there is tremendous low hanging fruit in both areas still.

yalogin: Does this also mean ram prices are not coming down anytime soon?

andai: Is global compute bottlenecked by one company?

com2kid: To bang on the same damn drum:Open Weight models are 6 months to a year behind SOTA. If you were building a company a year ago based on what AI could do then, you can build a company today with models that run locally on a user's computer. Yes that may mean requiring your customers to buy Macbooks or desktops with Nvidia GPUs, but if your product actually improves productivity by any reasonable amount, that purchase cost is quickly made up for.I'll argue that for anything short of full computer control or writing code, the latest Qwen model will do fine. Heck you can get a customer service voice chat bot running in 8GB of VRAM + a couple gigs more for the ASR and TTS engine, and it'll be more powerful than the hundreds of millions spent on chat bots that were powered by GPT 4.x.This is like arguing the age of personal computing was over because there weren't enough mainframes for people to telnet into.It misses the point. Yes deployment and management of personal PCs was a lot harder than dumb terminal + mainframe, but the future was obvious.

space_fountain: I've seen this claimed, but I'm not sure it's been true for my use cases? I should try a more involved analysis but so far open models seem much less even in their skills. I think this makes sense if a lot of them are built based on distillations of larger models. It seems likely that with task specific fine tuning this is true?

zozbot234: The real advantage of Open Weight models from a compute scarcity POV is that they're repurposing the compute users need to have around anyway for their own use. That's great but it's also limited in scope. There's only so many engineering/architecture/Gfx special effects workstations around that can now run reasonable mid-sized models "for free" during downtime because they had to be available already for other uses. Everything else will only increase the scarcity, not redress it, unless you only expect users to run very small or very slow models.

henry2023: The US is bound by energy and China is bound by compute power. The one who solves its limitation first will end this “Scarcity Era”.

CuriouslyC: The dynamics vastly favor China, part of the reason the US sprinting towards "ASI" isn't totally boneheaded is that the US and its industry needs a hail mary play to "win" the game, if they play it safe they lose for sure.

leptons: I'd be fine with a world without AI, honestly. Nobody really wins this race except the very wealthy. And I don't think it's really going to play out the way the wealthy think it will. It's more like a dog catching a car than it is a race.

odo1242: > It's more like a dog catching a car than it is a race.What does this mean? I didn't understand the analogy.

jakeinspace: China is installing something like 500 GW of wind and solar per year now. Even if they're only able to build and otherwise access chips that have half the SoTA performance per watt, they will win.

odo1242: Performance per dollar may be more important than performance per watt here, though

dataviz1000: What do you mean by harness here?

Ifkaluva: When you go to the command line and type “Claude”, there is an LLM, and everything else is the harness

Miraste: China's domestic chips are increasingly close to state-of-the-art. The US electrical grid is... not.

wg0: There's other side to it too.Whoever running and selling their own models with inference is invested into the last dime available in the market.Those valuations are already ridiculously high be it Anthropic or OpenAI to the tune of couple of trillion dollars easily if combind.All that investment is seeking return. Correct me if I'm wrong.Developers and software companies are the only serious users because they (mostly) review output of these models out of both culture and necessity.Anywhere else? Other fields? There these models aren't any useful or as useful while revenue from software companies by no means going to bring returns to the trillion dollar valuations. Correct me if I'm wrong.To make the matter worst, there's a hole in the bucket in form of open weight models. When squeezed further, software companies would either deploy open weight models or would resort to writing code by hand because that's a very skilled and hardworking tribe they've been doing this all their lives, whole careers are built on that. Correct me if I'm wrong.Eventually - ROI might not be what VCs expect and constant losses might lead to bankruptcies and all that build out of data centers all of sudden would be looking for someone to rent that compute capacity result of which would be dime a dozen open weight model providers with generous usage tiers to capitalize on that available compute capacity owners of which have gone bankrupt and can't use it any more wanting to liquidate it as much as possible to recoup as much investment as possible.EDIT: Typos

TeMPOraL: You can if you're exhausting the global production of oranges.

dist-epoch: Buy new Macs from where? There is a shortage of RAM, SSD, GPUs, and the CPU shortage just started.

Morromist: Hard to say at this point. I'm sure you can run your LLM chips 24/7 for training and for the public to make weird thirst-trap videos about Judy Hopps but how real is the utilization and demand, really? Maybe very real, maybe not, I don't think we can know yet.Its like being back in 1850 and you build the world's first amusement park where the rides are free or very cheap. People are like Amusement parks are the next big thing since Steam Boats! And tons of other rich people start to build huge amusement parks everywhere. The people who are skilled at making amusement park rides will increase their prices, and since the first amusement parks are free so they can get the public going to them demand will be huge.But how sustainable is that? - well obviously we know from history that amusement parks did, in fact, take over the world and most people spent virtually all their time and money at amusement parks - I think the Crimean War was even fought over some religious-based theme park in Israel - until moving pictures came out, so it worked out for them, but for AI?

codybontecou: pi vs. claude code vs. codex These are all agent harnesses which run a model (in pi's case, any model) with a system prompt and their own default set of tools.

dataviz1000: This is what I think of when I read harness.[0] https://www.anthropic.com/engineering/effective-harnesses-fo...

dataviz1000: I'm having an hard time getting my mind to see this.> Users should re-tune their prompts and harnesses accordingly.I read this in the press release and my mind thought it meant test harness. Then there was a blog post about long running harnesses with a section about testing which lead me to a little more confusion.Yes, the word 'harness' is consistently used in that context.

2001zhaozhao: AKA, the beginning of big companies being able to roll over small companies with moar money(note: I don't expect this to actually happen until the AI gets good enough to either nearly entirely replace humans or solve cooperation, but the long term trend of scarce AI will go towards that direction)

com2kid: What are you trying to do?Write code? No. Use frontier models. They are subsidized and amazing and they get noticably better ever few months.Literally anything else? Smaller models are fine. Classifiers, sentiment analysis, editing blog posts, tool calling, whatever. They go can through documents and extract information, summarize, etc. When making a voice chat system awhile back I used a cheap open weight model and just asked it "is the user done speaking yet" by passing transcripts of what had been spoken so far, and this was 2 years ago and a crappy cheap low weight model. Be creative.I wouldn't trust them to do math, but you can tool call out to a calculator for that.They are perfectly fine at holding conversations. Their weights aren't large enough to have every book ever written contained in them, or the details of every movie ever made, but unless you need that depth and breadth of knowledge, you'll be fine.

space_fountain: I just mean is the claim that the open source models where the closed models were 12 to 6 months ago true? They do seem to be for some specific tasks which is cool, but they seem even more uneven in skills than the frontier model. They're definitely useful tools, but I'm not sure if they're a match for frontier models from a year ago?

digitalsushi: A car caught by a dog has no purpose. The activity concludes with no output.

com2kid: Frontier models from a year ago had issues with consistent tool calling, instruction following was pretty good but could still go off the rails from time to time.Open weight models have those same issues. They are otherwise fine.You can hook them up to a vector DB and build a RAG system. They can answer simple questions and converse back and forth. They have thinking modes that solve more complex problems.They aren't going to discover new math theorems but they'll control a smart home and manage your calendar.

dist-epoch: Jensen just said that if the signal/commitments are there, ASML can scale in 2-3 years.

vessenes: With Anthropic buying compute in dark alleys I’d assume that day is coming..

rstuart4133: [delayed]

i_think_so: > Does this also mean ram prices are not coming down anytime soon?One person replies "yes". Another replies "no".This concludes our press conference.<3 HN

leptons: "The dog that caught the car" refers to how dogs sometimes chase cars. Suppose the car stops and the dog catches up - what is it going to do? It has no plan, it has no purpose, it isn't going to bite the car, it isn't going to get anything out of catching the car. The car may even run it over. I intended it basically as "play stupid games, win stupid prizes", or "be careful what you wish for".

thelastgallon: A dollar is an entirely fictional unit and trillions of it can be manufactured at no cost, while watts are constrained by the laws of physics, photons&#x2F;electrons, supply chain of electricity and all that fun stuff in the real world.

thelastgallon: My observation is that the dog sniffs all the tires, picks one tire, lifts one leg and does the deed. I don't know if its a way of marking territory or domination. We need a dogatologist to explain what it means.

thelastgallon: US energy is constrained by the utility monopolies&#x2F;oligopolies which have to extract more rents, specifically by increasing costs. Their profit is a percentage of cost, these perverse incentives + oligopolies will make it increasingly expensive to make anything (including AI) in US.

frodowtf2: > In both cases the code will be crap, as no model I've seen produces good code.I'm wondering if you have actually used claude code because results are not so catastrophic as you describe them.

solenoid0937: OpenAI has an absurdly high valuation given their cash burn vs RRR.Anthropic's is far more reasonable.It makes no sense to lump these two companies together when talking about valuation. They have completely different financial dynamics

wg0: No matter how low and reasonably Anthropic is valued, don't think $200 Max plans are going to recoup the investment + some return on top because size of the software industry is not that huge and profit margins for AI inference aren't very high either.

solenoid0937: Pro and Max plans are probably a drop in the bucket for them.

christkv: It feels like a repeat of the dot com infrastructure buildup that spurred the whole 2005 explosion in affordable hosting and new companies. This will probably leave us massive access to affordable compute in a couple of years.

Tanjreeve: Yes. At least, the manufacturing of compute is. And a lot of the chain has been bitten hard by increasing capacity prematurely in the past so they're reticent to increase bandwidth at vast cost.

cesarvarela: Harness is a big one, Claude Code still has trouble editing files with tabs. I wonder how many tokens per day are wasted on Claude attempting multiple times to edit a file.

lpcvoid: The future is now, I guess

dboreham: This field is chock full of people using terms incorrectly, defining new words for things that already had well known names, overloading terms already in use. E.g. shard vs partition. TUI which already meant "telephony user interface ". "Client" to mean "server" in blockchain.

hvb2: Or simply by the fact that increasing production takes time? Any power plant takes years to build?Years, is like a lifetime for AI at this point...

utopiah: Is ASML really the bottleneck? Do you believe anybody but TSMC and few fabs could really use and acquire those machines? I don't know the throughput of a EUV device from ASML but I imagine you need :- clean room, itself needing the infrastructure for it (size, airCo, filtering, electricity) and the staff to run and maintain that basically empty space - wafers to "print" on, so that's a lot of water and logistic to manipulate them (so infrastructure for clean water and all chemicals) also with dedicated staff - finally staff who would be able to design something significantly better than NVIDIA, Intel, Broadcom, IBM, etc while (and arguably that's the trickiest part IMHO) being able to get it good enough as at a scale that can be manufactured from their own fab.so I'm wondering who can afford this kind of setup that can only then make use of ASML machines.

drra: Seems like everybody an their mothers are using max plans these days. I wouldn't be surprised if LTV of each customer was big enough to justify spending.

Marazan: > (so infrastructure for clean water and all chemicals)Fabs are some of the most complex chemical engineering sites (dealing with some of the most dangerous substances) in the world. So don't underestimate the complexity of this part.

rstuart4133: I used LLMs to write what seems like far too many lines of code now. This is an example Opus 4.6 running at maximum wrote in C: if (foo == NULL) { log_the_error(...); goto END; } END: free(foo); If you don't know C, in older versions that can be a catastrophic failure. (The issue is so serious in modern C `free(NULL)` is a no-op.) If it's difficult to get a `FOO == NULL` without extensive mocking (this is often the case) most programmers won't do it, so it won't be caught by unit tests. The LLMs almost never get unit test coverage up high enough to catch issues like this without heavy prompting.But that's the least of it. The models (all of them) are absolutely hopeless at DRY'ing out the code, and when they do turn it into spaghetti because they seem almost oblivious to isolation boundaries, even when they are spelt out to them.None of this is a problem if you are vibe coding, but you can only do that when you're targeting a pretty low quality level. That's entirely appropriate in some cases of course, but when it isn't you need heavy reviews from skilled programmers. No senior engineer is going to stomach the repeated stretches of almost the "same but not quite" code they churn out.You don't have to take my word for it. Try asking Google "do llm's produce verbose code".

random_human_: Is foo a pointer in your example? Is free(NULL) not a valid operation?

rstuart4133: [delayed]

com2kid: China already operates like this. Low cost specialized models are the name of the game. Cheaper to train, easy to deploy.The US has a problem of too much money leading to wasteful spending.If we go back to the 80s&#x2F;90s, remember OS&#x2F;2 vs Windows. OS&#x2F;2 had more resources, more money behind it, more developers, and they built a bigger system that took more resources to run.Mac vs Lisa. Mac team had constraints, Lisa team didn't.Unlimited budgets are dangerous.

tasoeur: Though I do agree with you, I just came back from a trip to China (Shanghai more specifically) and while attending a couple AI events, the overwhelming majority of people there were using VPNs to access Claude code and codex :-&#x2F;

keiferski: We just had a realization during a demo call the other day:The companies that are entirely AI-dependent may need to raise prices dramatically as AI prices go up. Not being dependent on LLMs for your fundamental product’s value will be a major advantage, at least in pricing.

wg0: Assuming there are 10 million developers and everyone is at $200 max plan, that would be $0.2 billion&#x2F;month or $2.4 billion&#x2F;year maximum.Note - this is just the revenue not the profit. No salaries, no compute paid for. Just plain revenue. Profit would be way less.But even that - if we take it to $2.4 billion&#x2F;year and we take a 10x multiple, the company is barely valued at $24 billon dollar, lets be generous and make it double at $48 billion and then round it up to $50 billion for a nice round number.Far far from the $800 billion valuation Anthropic is looking at.Only a matter of time.

random_human_: So what would be the best practice in a situation like that? I would (naively?) imagine that a null pointer would mostly result from a malloc() or some other parts of the program failing, in which case would you not expect to see errors elsewhere?

andersmurphy: Yup. Also regardless of price they need to spend more and more as the project collapses under the enevirable incidental complexity of 30k lines of code a day.It's similar to how if you know what you're doing you can manage a simple VPS and scale a lot more cost effectively than something like vercel.In a saturated market margins are everything. You can't necessarily afford to be giving all your margins to anthropic and vercel.

throwaway290: isn't ai supposed to get us post-scarcity?

thelastgallon: > increasing production takes time?This is true of nearly everything (except money). I'm not sure of the point you are trying to make.

ElFitz: It’s the tool that calls the model, give it access to the local file system, calls the actual tools and commands for the model, etc, and provide the initial system prompt.Basically a clever wrapper around the Anthropic &#x2F; OpenAI &#x2F; whatever provider api or local inference calls.

tim333: >For the first time since the 2000s, technology companies are confronting the limits of their supply chain.I thought there'd been a shortage of cheap GPUs since ChatGPT took off and also before that in various crypto booms. I'm not sure it's a new thing.

ElFitz: That was quite the unexpected anticlimactic ending. I’m sure Terry Pratchett would be proud.

billziss: While I agree with you that AI companies are overvalued, I think 10 million developers at $200 per month makes 2 billion. >>> f"{10_000_000 * 200:_}" '2_000_000_000'

ElFitz: > A dollar is an entirely fictional unit and trillions of it can be manufactured at no costIt’s still a useful proxy for resources allocation and viability.

ElFitz: > because size of the software industry is not that hugeI onboarded marketing on a premium team Claude seat yesterday. And one of our sales vibecoded an internal tool in the last three weeks using Claude Code that they now use every day. I wouldn’t have imagined it a month ago. We still had to take care of deployment for him, but things are moving fast.

incrudible: C is fundamentally a bad target for LLMs. Humans get C wrong all the time, so we can not hope the nascent LLM, which has been trained on 95% code that does automatic memory management, to excel here.I always found myself writing verbose copypasta code first, then compress it down based on the emerging commonalities. I think doing it the other way around is likely to lead to a worse design. Can you not tell the LLM to do the same? Honest question.

rstuart4133: > I always found myself writing verbose copypasta code first, then compress it down based on the emerging commonalities. I think doing it the other way around is likely to lead to a worse design.I do pretty much the same thing, which is to say I "write code using a brain dump", "look for commonalities that tickle the neurons", then "refactor". Lather, rinse, and repeat until I'm happy.> Can you not tell the LLM to do the same?You can tell them until you're blue in the face. They ignore you.I'm sure this is a temporary phase. Once they solve the problem, coding will suffer the same fate as blacksmiths making nails. [0] To solve it they need to satisfy two conflicting goals - DRY the code out, while keeping interconnections between modules to a minimum. That isn't easy. In fact it's so hard people who do it well and can do it across scales are called senior software engineers. Once models master that trick, they won't be needed any more.By "they" I mean "me".[0] Blacksmiths could produce 1,000 or so a day, but it must have been a mind-numbing day even if it paid the bills. Then automation came along, and produced them at over a nail per second.

finaard: How is that surprising? We've been taking that into account for any LLM related tooling for over a year now that we either can drop it, or have it designed in a way that we can switch to a selfhosted model when throwing money at hardware would pay for itself quickly.It's just another instance of cloud dependency, and people should've learned something from that over the last two decades.

coldtea: Parent's point was about deployment, not agentic coding.

michaelbuckbee: What's weird though is the bifurcation in pricing in the market: aka if your app can function on a non-frontier level AI you can use last years model at a fraction of the cost.

michaelje: Absolutely. Pricing exposure is the quiet story of the next 24 months. Build for convenience → subsidise for dependence → meter for margin is a well-worn playbook, and AI-dependent companies are about to find out what phase three feels like.Hyperscalers are spending a fortune so we think AI = API, but renting intelligence is a business model, not a technical inevitability.Shameless link to my blog: https:&#x2F;&#x2F;mjeggleton.com&#x2F;blog&#x2F;AIs-mainframe-moment

the_gipsy: But that concerned mostly only gamers and cryptominers. AI is supposed to be replacing traditional software development, which affects everything.

muppetman: No shit. People are just figuring this out now?This is the “Building my entire livelihood on Facebook, oh no what?” all over again.Oh no sorry I forgot, your laptops LLM can draw a potato, let me invest in you.

hemangjoshi37a: The compute framing misses the layer where scarcity is biting today: eval data. We shipped RAG to three enterprises this quarter, and in all three the blocker was "we don't have 500 ground-truth QA pairs a domain expert verified." Compute is a capex problem that solves itself over time. Eval data is an org problem — needs SMEs sitting down labeling, and they're busy. GPUs get cheaper. Verified labels don't.

lelanthran: > C is fundamentally a bad target for LLMs.I found it exceptionally good, because:a) The agent doesn't need to read the implementation of anything - you can stuff the entire projects headers into the context and the LLM can have a better birds-eye view of what is there and what is not, and what goes where, etc.andb) Enforcing Parse, don't Validate using opaque types - the LLM writing a function that uses a user-defined composite datatype has no knowledge of the implementation, because it read only headers.

lelanthran: > That means a cautious C programmer who doesn't know who will be using their code never allows NULL to be passed to `free()`.If your compiler chokes on `free(NULL)` you have bigger problems that no LLM (or human) can solve for you: you are using a compiler that was last maintained in the 80s!If your C compiler doesn't adhere to the very first C standard published, the problem is not the quality of the code that is written.> If they aren't avoiding passing NULL to `free()`, they haven't suffered long enough to be good.I dunno; I've "suffered" since the mid-90s, and I will free NULL, because it is legal in the standard, and because I have not come across a compiler that does the wrong thing on `free(NULL)`.

strife25: Marginal costs matter in this world.

lelanthran: > Buying oranges for $1 and selling for $0.5 is an investment into acquiring market share and customer relationshipsIt's a delusion that customers are going to remain with the behemoths when a Qwen model run by an independent is $10&#x2F;m, unlimited usage.This is not a market that can be locked-in with network effects, and the current highly-invested players have no moat.

zozbot234: > The companies that are entirely AI-dependent may need to raise prices dramatically as AI prices go up.It's not that clear. Sure, hardware prices are going up due to the extremely tight supply, but AI models are also improving quickly to the point where a cheap mid-level model today does what the frontier model did a year ago. For the very largest models, I think the latter effect dominates quite easily.

LogicFailsMe: so much for all that hardware that was going to be obsolete in 3 years...

KaiserPro: one graph, One graph and the author is pinning an entire theory on it?Infra is always limited, even at hyper scalers. This leads to a bunch of tools dfofr caching, profiling and generally getting performance up, not to mention binpacking and all sorts of other "obvious" things.

lioeters: Indeed, it was clear from the beginning, "AI" companies want to become infrastructure and a critical dependency for businesses, so they can capture the market and charge whatever they want. They will have all the capital and data needed to swallow those businesses too.

rstuart4133: [delayed]

malshe: On X I had seen him mostly posting memes so this post seems par for the course

anonyfox: in fact I am betting opposite. frontier models are getting not THAT much better anymore at all, for common business needs at least. but the OSS models keep closing the gap. which means if trajectories hold there will be a near future moment probably where the big provider costs suddenly drop shaerply once the first viable local models consistently can take over tasks normally on reasonable hardware. Right now probably frontier providers rush for as much money as they possible can before LLMs become a true commodity for the 80% usecases outside of deep expert areas they will have an edge over as specialist juggernauts (iE a cybersecurity premium model).So its all a house of cards now, and the moment the bubble bursts is when local open inference has closed the gap. looks like chinese and smaller players already go hard into this direction.

prox: I also can’t wait for the time when few know how to code. Just like how many folks don’t know html from css when the homebrew website went away.Their might always be llms, but the dependence is an interesting topic.

1828838383: We did it reddit!

keiferski: Not so much that it was surprising, rather that we looked at a competitor’s site and noticed that a) their prices went way up and b) their branding changed to be heavily AI-first.So we thought, hmm, “wonder if they are increasing prices to deal with AI costs,” and then projected that into a future where costs go up.We don’t have this dependence ourselves, so this seems to be a competitive advantage for us on pricing.

utopiah: Well that was part of my point, not everybody is TSMC. It's not "just" getting an ASML machine and voila, you're good to go.

chatmasta: Why is written with an assumption that we have finite hardware production capacity? Industrial processes can scale up, new factories can come online… it will take a while but the whole point of economics is that supply will scale to meet demand. The shortage is a temporary, point-in-time metric.And that’s not considering the software innovation that can happen in the meantime.

Bengalilol: The economic hypothesis that has dominated the past hundred years is that economic growth is infinite because resources are infinite and (almost) free. We all know this is unrealistic and disconnected from our human condition.Regarding "innovation", I agree with your idea. I even think that the major innovation will be to transpose models locally, using reduced infrastructures that will still be sufficient for the majority of use cases.

PessimalDecimal: That worked out, for the founders of frontier labs at least.

bcjdjsndon: There's only so far engineers can optimise the underlying transformer technique, which is and always has been doing all the heavy lifting in the recent ai boom. It's going to take another genius to move this forward. We might see improvements here and there but the magnitudes of the data and vram requirements I don't think will change significantly

zozbot234: State space models are already being combined with transformers to form new hybrid models. The state-space part of the architecture is weaker in retrieving information from context (can't find a needle in the haystack as context gets longer, the details effectively get compressed away as everything has to fit in a fixed size) but computationally it's quite strong, O(N) not O(N^2).

steveklabnik: Companies are spending far more than $200&#x2F;month&#x2F;developer. The $200 Max plan is a great value but you hit limits far too soon, and it also doesn't cover any of the other styles of integrations and tools that you can build and use to help your developers, like code review suggestions, which at the very least would come from additional Max plans, and not from the individual developers' plans.

sevenzero: This was as clear as the sky when the first llm based businesses popped up. How did you realize this only now?

bdangubic: same as Uber… in the beginning everyone pretty much new that the cost of rides cannot possibly be that cheap and that it is subsudized. once you corner the market etc people just got used to “real” prices to the poibt that now there are often cheaper alternatives than Uber but people still Uber…

sevenzero: Its also quite interesting to read about Uber exploits their drivers and discriminating algorithms. Cory Doctorow mentioned it in his latest book, sadly cant link the direct sources.

utopiah: Initially I thought "Well... good for AI companies because they can then charge more" but IMHO that's a very tricky position because it means the cheap wave is behind us.It's one thing to "sell" free or symbolically cheap stuff, it's another to have an actual client who will do the math and compare expenditure vs actually delivered value.

classified: > and compare expenditure vs actually delivered valueWhich means that the hype production will be driven up another few notches to make people doubt their rational findings and keep them in irrational territory just a tad longer. Every minute converts to dollars spent on tokens.

keiferski: Replied here: https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=47804804And I don't really mean new businesses that are entirely built around LLMs, rather existing ones that pivoted to be LLM-dependent – yet still have non-LLM-dependent competitors.

sevenzero: Yea that would've been extremely short sighted from your competitors. Thanks for linking the response!

sidewndr46: Not really, the next move is to establish standards groups requiring the use of AI in product development. A mix of industry and governmental mandates. What you view are viewing as COGS instead becomes instead a barrier to entry.

aldanor: Yep.As a recent example in AI space itself. China had scarce GPU resources, quite obvious why => DeepSeek training team had to invent some wheels and jump through some hoops => some of those methods have since become 'industry standard' and adopted by western labs who are now jumping through the same hoops despite enjoying massive computeresources, for the sake of added efficiency.

zozbot234: Local open inference can address hardware scarcity by repurposing the existing hardware that users need anyway for their other purposes. But since that hardware is a lot weaker than a proper datacenter setup, it will mostly be useful for running non-time-critical inference as a batch task.Many users will also seek to go local as insurance against rug pulls from the proprietary models side, but ultimately if you want to make good utilization of your hardware as a single user you'll also be pushed towards mostly running long batch tasks, not realtime chat (except tiny models) or human-assisted coding.

classified: > would resort to writing code by hand because that's a very skilled and hardworking tribe they've been doing this all their livesShush, don't tell that to the AI coding acolytes.