Sequence Composition in EHR Foundation Model Pretraining
Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events.
ProofPaper ↗
Key points
- A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it.
- To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance.
- Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences.
- Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.
Sources (1)
- [1]Sequence Composition in EHR Foundation Model PretrainingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 02:41 PM
Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events.
A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it.
Extractive summary: sentences quoted from the sources.