A Dependency-Aware Multi-GPU Training Framework for Temporal Graph Neural Networks

Key points

• These models use node state vectors (i.e., memories) to capture and record graph changes (i.e., events), and then exploit these memories to assist prediction.1
• To maintain model quality, existing multi-GPU TGNN training schemes strictly preserve all event dependencies, which limits parallelism and causes expensive cross-GPU memory synchronization.1
• Specifically, (a) dependencies in TGNN training have high locality in both time and graph topology, (b) the short-range dependencies missed within a large batch can often be recovered cheaply, and (c) many memory updates are not immediately needed.1
• Based on these observations, we propose DepBloom to accelerate multi-GPU TGNN training with three techniques: (1) temporally bounded spatial partitioning to reduce unnecessary cross-GPU synchronization and improve parallelism; (2) intra-batch memory refinement to recover missing short-range temporal dependencies inside enlarged batches; and (3) critical-path optimization to prioritize urgent memory updates while deferring non-critical work during multi-GPU pipeline execution.1

Sources (1)

• [1]A Dependency-Aware Multi-GPU Training Framework for Temporal Graph Neural Networks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 04:11 PM

  Temporal graph neural networks (TGNNs) achieve superior performance in many dynamic graph learning tasks.

  These models use node state vectors (i.e., memories) to capture and record graph changes (i.e., events), and then exploit these memories to assist prediction.

Extractive summary: sentences quoted from the sources.