AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation.
ProofPaper ↗
Key points
- Synthetic data is increasingly used in language-model pretraining.
- Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text.
- In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA.
- Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.
Sources (1)
- [1]AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge PretrainingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 12:24 PM
We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation.
Synthetic data is increasingly used in language-model pretraining.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026Gray-Box Hallucination Detection in LLMs with Token Identities
- Oct 14, 2026mistralai/Timestral-209M
- Oct 14, 2026NavAble: A Large-Scale Dataset and Synthetic Data Generation Pipeline for Blind Navigation
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026Risk-Averse Inference-Time Diffusion Model Alignment with Finite-Sample Guarantees
- Oct 13, 2026Hybrid Reasoning Agents