Dario: “With open source you can see the source code. Here we can’t see inside the model.” My 0.646 t/s MiMo-2.6-Pro crammed thru 128GB of Jetson AGX Thor: “Yeah, well, that’s just, like, your opinion, man.”

Key points

• Every decoder layer, routed expert, SSD read (..a good bit more) traced and replayable step by step.1
• it's ~566GB of model (FP8), at the blinding speed and architecture of hand-extruded hotdog manufacturing, read from disk mid-inference, painfully shat through 128GB of memory.1
• I think it's fine but decided to maybe take a closer look to see if I could maybe reach factory hotdog manufacturing speeds. like, a small, local hotdog factory..1
• ..poor lil bugger, get well soon Dario!1

Sources (1)

• [1]Dario: “With open source you can see the source code. Here we can’t see inside the model.” My 0.646 t/s MiMo-2.6-Pro crammed thru 128GB of Jetson AGX Thor: “Yeah, well, that’s just, like, your opinion, man.”
  r/LocalLLaMA (top, daily) · Oct 16, 09:18 PM

  Tracked my first run of MiMo Pro on a Jetson AGX Thor at a scorching 0.646 tokens/sec,.. but it DID technically load and run, so it counts.

  Every decoder layer, routed expert, SSD read (..a good bit more) traced and replayable step by step.

Extractive summary: sentences quoted from the sources.