Discussion

Search code, repositories, users, issues, pull requests...

harshhhhhhhhh: seems promising , this is the way , can someone benchmark this

homarp: /r/localllama discussion: https://old.reddit.com/r/LocalLLaMA/comments/1rxmmu5/running...

bertili: Very impressive! I wonder if there is a similar path for Linux using system memory instead of SSD? Hell, maybe even a case for the return of some kind of ROMs of weights?

JSR_FDED: This is a very impressive result. If I understand correctly the bottleneck is the SSD in this architecture - the author seems to get almost 15GB/s - but I seem to remember the max b/w was about 8GB/s. What am I missing?

rado: MacBook Pro M5 Pro and M5 Max have such SSD speed

rvz: The technical write up is great, but Mac users should not get too excited just yet on running 300B+ parameter models locally as the TPS isn't that good.>...at 4.4+ tokens/secondThat is even when it is using 4-bit quantization and it is still at that speed.> The entire 209GB model streams from SSD through a custom Metal compute pipeline.This is my main problem.If I were to run this on a Mac SSD, 24/7 for heavy usage such as Openclaw, that is going to significantly reduce the lifetime of the SSD.Can't imagine using this in the long term right now, but improvements will follow. Still a great write up anyways.

Roxxik: Does an SSD meaningfully degrade by read only workloads?

JSR_FDED: Nope, reads don’t cause wear

vilequeef: Why so much RAM?

hrmtst93837: If you want decent throughput and do not care about burning SSD write cycles on a box that was never meant to act like a tiny inference server, a used server with actual RAM is still the cheaper and less silly option. I woudn't expect Apple's warranty team to be much help.

pdyc: impressive, i wish someone takes a stab at using this technique on mobile gpu's even if it does not use storage it would still be a win. I am running llama.cpp on adreno 830 with oepncl and i am getting pathetic 2-3t/s for output tokens

zozbot234: The github page mentions that a naïve mmap approach is bottlenecked by per-page overhead. Can this be mitigated by using explicit "huge" pages? Does macOS support them?

Roxxik: IO is very bursty in these setups. When the router results are in you can start loading experts from SSD. In this brief moment the SSD is saturated.Outside of that the SSD is idling.Table 3 shows for K=4 experts an IO of 943 MB/s at 3.15 Tok/s giving an average IO of 2970 MB/s far below what the SSD could do.I'm not sure, but not all expert weights are used immediately. Maybe they could do async reads for the down tensors parallelzing compute with IO.Not sure if this works on Mac, I only tested my larger than RAM setup on Linux and I saw that about 20% of total reads do finish while my fused upgate matmul is already running.

etiam: > If I were to run this on a Mac SSD, 24/7 for heavy usage such as Openclaw, that is going to significantly reduce the lifetime of the SSD.How sure are you about that? I've never looked closer at how a large LLM with mixture of experts architecture switches between expert modules, but staying on roughly the same topic for the use (as it often would when editing the same codebase), I wouldn't be surprised to see the switches of composition are fairly rare, fairly small, and to the extent it happens it's repeated reads from the flash disk rather than writes it tends to cause.

frotaur: Afaik the experts are not usually very interpretable, and generally would be surprised if at least one does not change every token. I don't know what happens in practice, but I know at least during training, nothing is done to minimize the number of expert switches between tokens.

frwickst: I'm getting 6.55t/s using the Qwen3.5-397B-A17B-4bit model with the command: ./infer --prompt "Explain quantum computing" --tokens 100MacBook Pro M5 Pro (64GB RAM)

logicallee: can you post the final result (or as far as you got before you killed it) to show us how cohesive and good it is? I'd like to see an example of the output of this.

frwickst: Since the output is quite long, here is a link: https://pastebin.com/k76wiVGP

K0balt: Is it doing a bunch of ssd writes?

hrimfaxi: Why does this G character appear to prefix most of the output? ("Ġlike")

lostmsu: [delayed]

Wowfunhappy: Eh. I mean, 4 tokens a second works fine if you're patient. Go do something else while you wait.I feel like whenever I'm trying to find information on which local models will work on my hardware, I have to overestimate because people don't know how to wait for things.Also, reading data doesn't cause SSD wear.

spwa4: Does this mean that it should be possible to load up a system with ~10 (seems to me at least the number of active experts) SSDs to get 40 tok/s even on truly gigantic models?

j45: Appreciate the data point. M5 Max would also be interesting to see once available in desktop form.

zozbot234: No appreciable wear of course, but read disturb (requiring sporadic rewrites) becomes more of an issue as NAND fabrication advances.

tarruda: Note that this is not the only way to run Qwen 3.5 397B on consumer devices, there are excellent ~2.5 BPW quants available that make it viable for 128G devices.I've had great success (~20 t/s) running it on a M1 Ultra with room for 256k context. Here are some lm-evaluation-harness results I ran against it: mmlu: 87.86% gpqa diamond: 82.32% gsm8k: 86.43% ifeval: 75.90% More details of my experience:- https://huggingface.co/ubergarm/Qwen3.5-397B-A17B-GGUF/discu...- https://huggingface.co/ubergarm/Qwen3.5-397B-A17B-GGUF/discu...- https://gist.github.com/simonw/67c754bbc0bc609a6caedee16fef8...Overall an excellent model to have for offline inference.

maxloh: Can you add a license to the report? Legally we couldn't run any code without a license attached to it.

Aurornis: Using system memory and CPU compute for some of the layers that don’t fit into GPU memory is already supported by common tools.It’s workable for mixture of experts models but the performance falls off a cliff as soon as the model overflows out of the GPU and into system RAM. There is another performance cliff when the model has to be fetched from disk on every pass.

daemonologist: Most definitely - the popular engines have extensive support for doing this and controlling exactly which weights end up where (llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/... , vllm: https://docs.vllm.ai/en/stable/configuration/engine_args/#of... , sglang (haven't tried this): https://docs.sglang.io/advanced_features/server_arguments.ht...).Even with a MoE model, which has to move a relatively small portion of the weights around, you do end up quite bandwidth constrained though.

Aurornis: Reading the details, he is using 2-bit quantization and reduced the number of experts per token from 10 down to 4 to get 5 tokens/sec. Cool proof of concept but it’s far from the quality and performance of the 397B model.This is some interesting work, but applying such extreme measures to LLMs to get them to run severely degrades quality. I know he claims negligible quality loss, but in my experience 2-bit quantizations are completely useless for real work. You can get them to respond to prompts, but they lose their intelligence and will go around in circles.He also shows 5-6 tokens per second. Again that’s impressive for a large model on limited hardware but it’s very slow. Between the severely degraded model abilities and the extremely slow output the 397B result should be considered an attempt at proving something can technically run, not evidence that it can run well and produce output you’d expect from a 397B model.

m-hodges: As frontier models get closer and closer to consumer hardware, what’s the most for the API-driven $trillion labs?

Aurornis: The method in this link is already using a 2-bit quant. They also reduced the number of experts per token from 10 to 4 which is another layer of quality degradation.In my experience the 2-bit quants can produce output to short prompts that makes sense but they aren’t useful for doing work with longer sessions.

zozbot234: It's less of a "performance falls off a cliff" problem and more of a "once you offload to RAM/storage, your bottleneck is the RAM/storage and basically everything else no longer matters". This means if you know you're going to be relying on heavy offload, you stop optimizing for e.g. lots of VRAM and GPU compute since that doesn't matter. That saves resources that you can use for scaling out.

Aurornis: PCIe 5 doubles the maximum throughout. That’s why the numbers for newer SSDs are about double what you recall for the old maximum.

zozbot234: SSD bandwidth will ultimately be limited by the amount of PCIe lanes you have available (for something other than the Apple Silicon internal storage). So the approach has inherent limitations. You can of course scale out to multiple systems to get more throughput.You can use this approach with Intel Optane, which is wearout-resistant unlike NAND and can thus substitute for RAM. Last I checked, it was available quite cheap on the secondary market, ~$1/GB as opposed to ~$15/GB or more for DRAM. (Of course that's nowhere near as cheap as NAND, which is around ~$0.1/GB but quite wearout-prone with heavy writes.)

spwa4: Yeah, PCIe is the bottleneck. The point being that whether the data originates from RAM or from NVME or Optane, you cannot get data to the GPU faster with RAM than with SSDs.Meanwhile PCIe switches exist. So why not build:1 CPU + memory + ...N PCIe switch with each 1 low-memory GPU + 6 NVME drives (in theory 5 can saturate the GPU)Each of those should only bother the CPU when they have some tokens produced and have plenty of PCIe lanes to get at their data.

OJFord: Assuming 'moat' – they'll push the frontier forward; they don't really have to worry until progress levels off.At that point, I suppose there's still paid harnesses (people have always paid for IDEs despite FOSS options) partly for mindshare, and they could use expertise & compute capacity to provide application-specific training for enterprises that need it.

tarruda: I can't say anything about the OP method, but I already tested the smol-IQ2_XS quant (which has 2.46 BPW) with the pi harness. I did not do a very long session because token generation and prompt processing gets very slow, but I think I worked for up to ~70k context and it maintained a lot of coherence in the session. IIRC the GPQA diamond is supposed to exercise long chains of thought and it scored exceptionally well with 82% (the original BF16 official number is 88%: https://huggingface.co/Qwen/Qwen3.5-397B-A17B).Note that not all quants are the same at a certain BPW. The smol-IQ2_XS quant I linked is pretty dynamic, with some tensors having q8_0 type, some q6_k and some q4_k (while the majority is iq2_xs). In my testing, this smol-IQ2_XS quant is the best available at this BPW range.Eventually I might try a more practical eval such as terminal bench.

383toast: yeah 4tok/s is kinda unusable though

p0w3n3d: Sounds like killing your SSD which is irreplaceable... Better use external drive for it

Wowfunhappy: ...you can't redistribute code without a license, but surely you can legally run it, can't you?Like, if I write a blog post and put it on my blog, you're allowed to read it, right?Heck, if my blog contains some Javascript code I wrote, I would imagine your web browser is allowed to run that code without opening you up to copyright infringement, even if I didn't provide an explicit license.