Built an open-source Cotypist/Granola hybrid: autocomplete on Gemma 4 E2B (~50 ms/suggestion on M4 Max) using your own meeting notes as context
I built a desktop app harness that keeps local meeting notes and can use them to autocomplete my typing as well as complement my dictation with additional context via reading my screen or using the previous meetings notes.
Proof1 community thread
Key points
- I've done a lot of experiments with different models and quants and Gemma 4 E2B base at 6bit is the best balance of quality suggestions and low latency on a regular MacBook.
- License / price Proprietary, free for 100 words a day, $$ for unlimited AGPL-3.0, free (beta) GPLv3, free, no limits
- Model Local (Gemma 4 E4B recommended) Local GGUF via llama.cpp, or Apple Intelligence Gemma 4 E2B base Q6K, in-process llama.cpp
- Head-to-head with Cotabby, 782 next-word checkpoints:
Sources (1)
- [1]Built an open-source Cotypist/Granola hybrid: autocomplete on Gemma 4 E2B (~50 ms/suggestion on M4 Max) using your own meeting notes as contextr/LocalLLaMA (top, daily) · Oct 13, 01:49 PM
I built a desktop app harness that keeps local meeting notes and can use them to autocomplete my typing as well as complement my dictation with additional context via reading my screen or using the previous meetings notes.
I've done a lot of experiments with different models and quants and Gemma 4 E2B base at 6bit is the best balance of quality suggestions and low latency on a regular MacBook.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Update: Converting dense models into Mixture-of-Experts
- Oct 13, 2026SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.
- Oct 12, 2026openbmb/MiniCPM-V-4.7-1B
- Oct 12, 2026openbmb/MiniCPM-V-4.7-35B-A3B
- Oct 12, 2026Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.
- Oct 11, 2026Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orin