On the Necessity of Attention-FFN Split in Vision Transformers

Key points

• The standard Transformer architecture relies on a rigid pattern that alternates Attention and Feed-Forward Network (FFN) layers.1
• To facilitate this analysis, we introduce the AttenFeed module, a unified component that integrates the functional properties of both Attention and FFN.1
• Based on this module, we devise the unified Vision Transformer (uViT), which replaces the conventional alternating Attention-FFN structure with a sequence of AttenFeed modules.1
• Our experiments reveal that the Attention-FFN dichotomy can hinder performance at smaller model scales due to the rigid parameter allocation of ViTs.1

Sources (1)

• [1]On the Necessity of Attention-FFN Split in Vision Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 03:57 PM

  In this work, we investigate the necessity of the Attention-FFN dichotomy in Vision Transformers (ViTs).

  The standard Transformer architecture relies on a rigid pattern that alternates Attention and Feed-Forward Network (FFN) layers.

Extractive summary: sentences quoted from the sources.