Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning

Key points

• Each task $i$ has an optimum $\bmΘi$, minimizing its corresponding loss function $Ri(\bm{w})$ on sphere.1
• To model task dependencies (similarities), task optima $\bmΘi$ are distributed by a Neyman-Scott clustered point process $Φ$.1
• Cluster centers $\{\bm{Y}j\}{j=1}^{Np}$ form a homogeneous Poisson point process on sphere, while intra-cluster task optima follow a von Mises-Fisher distribution with mean $\bm{Y}j$ and concentration $κ$.1
• Since in continual learning, the learner does not need to reach $\bmΘt$ exactly to successfully learn task $t$, to avoid forgetting as well as reliable learning, it only needs for the learned parameter vector $\bm{W}{t}$ to enter $B{\bmΘt}(rt(d)) \cap \mathcal{K}{t-1}(\bmΘ{1:t-1})$.1

Sources (1)

• [1]Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 10:13 PM

  We propose a continuous learning formulation for $T$ non-forgetting tasks with the spherical parameter space $\mathbb{S}^{d-1}(\sqrt{d})$ in asymptotic regime $d\rightarrow \infty$.

  Each task $i$ has an optimum $\bmΘ_i$, minimizing its corresponding loss function $R_i(\bm{w})$ on sphere.

Extractive summary: sentences quoted from the sources.