Qwen Flash Next on Mac Mini is runnable without maxing out your memory - results of different memory usage for ssd streaming

Key points

• I’ve known for a while it could be somewhat resilient to using less Gib, as the majority of the work is how to load experts from SSD into memory before they are needed.1
• The prompt processing uses a carousel buffering system which very efficiently solves for the pp side of things and it was good to see it hold up near perfectly even at lower amounts of experts hot cached.1
• For people doing stuff on their box besides hosting an LLM this is an insight into how much performance you give up for hosting X additional app e.g. ~29 tks decode at max, drop down 4gb and you’re at ~26tks.1
• It sort of depends on how the other apps behave as well as swings in memory are more where swap happens from what I’ve observed from how Mac reacts.1

Sources (1)

• [1]Qwen Flash Next on Mac Mini is runnable without maxing out your memory - results of different memory usage for ssd streaming
  r/LocalLLaMA (top, daily) · Oct 17, 09:51 PM

  This is from my recent side project to optimise inference of QFN (edit-q4) on a Mac Mini M5 64gb.

  I’ve known for a while it could be somewhat resilient to using less Gib, as the majority of the work is how to load experts from SSD into memory before they are needed.

Extractive summary: sentences quoted from the sources.