Discussion

Ollama is now powered by MLX on Apple Silicon in preview

babblingfish: LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

dial9-1: still waiting for the day I can comfortably run Claude Code with local llm's on MacOS with only 16gb of ram

codelion: How does it compare to some of the newer mlx inference engines like optiq that support turboquantization - https://mlx-optiq.pages.dev/

aurareturn: It isn't going to replace cloud LLMs since cloud LLMs will always be faster in throughput and smarter. Cloud and local LLMs will grow together, not replace each other.I'm not convinced that local LLMs use less electricity either. Per token at the same level of intelligence, cloud LLMs should run circles around local LLMs in efficiency. If it doesn't, what are we paying hundreds of billions of dollars for?I think local LLMs will continue to grow and there will be an "ChatGPT" moment for it when good enough models meet good enough hardware. We're not there yet though.

AugSun: "We can run your dumbed down models faster":#The use of NVFP4 results in a 3.5x reduction in model memory footprint relative to FP16 and a 1.8x reduction compared to FP8, while maintaining model accuracy with less than 1% degradation on key language modeling tasks for some models.

LuxBennu: Already running qwen 70b 4-bit on m2 max 96gb through llama.cpp and it's pretty solid for day to day stuff. The mlx switch is interesting because ollama was basically shelling out to llama.cpp on mac before, so native mlx should mean better memory handling on apple silicon. Curious to see how it compares on the bigger models vs the gguf path

gedy: How close is this? It says it needs 32GB min?

gedy: Man I really hope so, as, as much as I like Claude Code, I hate the company paying for it and tracking your usage, bullshit management control, etc. I feel like I'm training my replacement. Things feel like they are tightening vs more power and freedom.On device I would gladly pay for good hardware - it's my machine and I'm using as I see fit like an IDE.

aurareturn: When local LLMs get good enough for you to use delightfully, cloud LLMs will have gotten so much smarter that you'll still use it for stuff that needs more intelligence.

HDBaseT: You can run Qwen3.5-35B-A3B on 32GB of RAM sure, although to get 'Claude Code' performance, which I assume he means Sonnet or Opus level models in 2026, this will likely be a few years away before its runnable locally (with reasonable hardware).

AugSun: "Most users don't need frontier model performance" unfortunately, this is not the case.

pezgrande: You could argue that the only reason we have good open-weight models is because companies are trying to undermine the big dogs, and they are spending millions to make sure they dont get too far ahead. If the bubble pops then there wont be incentive to keep doing it.

aurareturn: I agree. I can totally see in the future that open source LLMs will turn into paying a lumpsum for the model. Many will shut down. Some will turn into closed source labs.When VCs inevitably ask their AI labs to start making money or shut down, those free open source LLMS will cease to be free.Chinese AI labs have to release free open source models because they distill from OpenAI and Anthropic. They're will always be behind. Therefore, they can't charge the same prices as OpenAI and Anthropic. Free open source is how they can get attention and how they can stay fairly close to OpenAI and Anthropic. They have to distill because they're banned from Nvidia chips and TSMC.

AugSun: Looking at downvotes I feel good about SDE future in 3-5 years. We will have a swamp of "vibe-experts" who won't be able to pay 100K a month to CC. Meanwhile, people who still remember how to code in Vim will (slowly) get back to pre-COVID TC levels.

melvinroest: I have journaled digitally for the last 5 years with this expectation.Recently I built a graphRAG app with Qwen 3.5 4b for small tasks like classifying what type of question I am asking or the entity extraction process itself, as graphRAG depends on extracted triplets (entity1, relationship_to, entity2). I used Qwen 3.5 27b for actually answering my questions.It works pretty well. I have to be a bit patient but that’s it. So in that particular use case, I would agree.I used MLX and my M1 64GB device. I found that MLX definitely works faster when it comes to extracting entities and triplets in batches.

brcmthrowaway: What is the difference between Ollama, llama.cpp, ggml and gguf?

Foobar8568: I fully agree, I run that one with Q4 on my MBP, and the performance (including quality of response) is a let down.I am wondering how people rave so much about local "small devices" LLM vs what codex or Claude code are capable of.Sadly there are too much hype on local LLM, they look great for 5min tests and that's it.

brcmthrowaway: Just train it better with AGENTS.md

gedy: True, but I'm already producing code/features faster than company knows what to do with, (even though every company says "omg we need this yesterday", etc). Even coding before AI was basically same.Code tools that free my time up is very nice.

virtue3: We are 100% there already. In browser.the webgpu model in my browser on my m4 pro macbook was as good as chatgpt 3.5 and doing 80+ tokens/sLocal is here.

AugSun: ... another user who "don't need frontier model performance" downvoted LOL ppl why are you so predictable? No wonder you are being replaced by LLMs ...

mfa1999: How does this compare to llama.cpp in terms of performance?

QuantumNomad_: What is CC and TC? I have not heard these abbreviations (except for CC to mean credit card or carbon copy, neither of which is what I think you mean here).

seanhunter: Complaining about downvotes is futile and is also against hn guidelines.

benob: Ollama is a user-friendly UI for LLM inference. It is powered by llama.cpp (or a fork of it) which is more power-user oriented and requires command-line wrangling. GGML is the math library behind llama.cpp and GGUF is the associated file format used for storing LLM weights.

Ericson2314: I figured it out from context cluesCC: Claude CodeTC: total comp(ensation)

raincole: Yep. People were claiming DeepSeek was "almost as good as SOTA" when it came out. Local has always been one step away like fusion.It's just wishful thinking (and hatred towards American megacorps). Understandable, but not based on reality.

pezgrande: > have to release free open source models because they distill from OpenAI and AnthropicThey dont really have to though, they just need to be good enough and cheaper (even if distilled). That being said, it is true they are gaining a lot of visibility (specially Qwen) because of being open-source(weight).Hardware-wise they seem they will catch-up in 3-5 years (Nvidia is kind of irrelevant, what matters is the node).

selcuka: Any citations? Because that was my impression, too. I want frontier model performance for my coding assistant, but "most users" could do with smaller/faster models.ChatGPT free falls back to GPT-5.2 Mini after a few interactions.

spiderfarmer: “They will always be behind”Car manufacturers said the same.

aurareturn: It did take decades to catch and surpass US car makers right?

raincole: Yep. People were claiming DeepSeek was "almost as good as SOTA" when it came out. Local will always be one step away like fusion.It's just wishful thinking (and hatred towards American megacorps). Old as the hills. Understandable, but not based on reality.

solarkraft: MLX is a bit faster (low double digit percentage), but uses a bit more RAM. Worthwhile tradeoff for many.

AndroTux: Sir, ChatGPT 3.5 is more than 3 years old, running on your bleeding edge M4 Pro hardware, and only proves the previous commenters point.