Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text
This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge.
ProofPaper ↗
Key points
- We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources.
- We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary.
- Both of these changes require LoRA-style fine-tuning for realizing their full potential.
- We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.
Sources (1)
- [1]Qwenfant: Train Big Character-level BabyLMs on Concise Parallel TextarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 11:15 PM
This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge.
We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Show HN: Pulse Local, a Mac budgeting app whose AI models run on your machine
- Oct 13, 2026How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts
- Oct 13, 2026unslothai/unsloth v0.1.907-beta: Large Performance Gains + More Browser Use
- Oct 13, 2026inLoRA-MINT: Internal Representations for Auditing Training Data in Domain-adapted LLMs
- Oct 12, 2026U-Space: Uncovering When and Why Uncertainty Arises in Language Models [R][D]
- Oct 12, 2026unslothai/unsloth v0.1.906-beta: Browser Use + Qwen-Image-2.1-Turbo
