Instead of waiting for an M5 Ultra 512 GB: pairing a new M5 Ultra 256 GB with my M3 Ultra 512 GB for disaggregated prefill/decode (GLM-5.3-Flash, MLX, Thunderbolt 5 RDMA)

Key points

• The M5 Ultra fixes prefill but ships with 256 GB max right now, and an M5 Ultra 512 GB isn't a thing yet.1
• So rather than replace the M3, I added an M5 Ultra 256 GB and split the work: the M5 does prefill, the cache state goes over Thunderbolt 5 RDMA, and the M3 does decode against its 512 GB pool.1
• Single-node numbers first, GLM-5.3-Flash 4-bit with MTP, 17.5k-token prompt, 256 output tokens.1
• The M5 is about 2.4x on prefill but only about 1.4x on decode, and decode is the phase that benefits from the bigger memory (more context, more concurrent conversations).1

Sources (1)

• [1]Instead of waiting for an M5 Ultra 512 GB: pairing a new M5 Ultra 256 GB with my M3 Ultra 512 GB for disaggregated prefill/decode (GLM-5.3-Flash, MLX, Thunderbolt 5 RDMA)
  r/LocalLLaMA (top, daily) · Oct 13, 07:32 PM

  The problem I'm solving: I have an M3 Ultra 512 GB and the memory is still the reason to own it, but prefill on long prompts is slow.

  The M5 Ultra fixes prefill but ships with 256 GB max right now, and an M5 Ultra 512 GB isn't a thing yet.

Extractive summary: sentences quoted from the sources.