CUED Publications database

Rethinking Attention with Performers

Choromanski, K and Likhosherstov, V and Dohan, D and Song, X and Gane, A and Sarlos, T and Hawkins, P and Davis, J and Mohiuddin, A and Kaiser, L and Belanger, D and Colwell, L and Weller, A Rethinking Attention with Performers. (Unpublished)

Full text not available from this repository.


We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.

Item Type: Article
Uncontrolled Keywords: cs.LG cs.LG cs.CL stat.ML
Divisions: Div F > Computational and Biological Learning
Depositing User: Cron Job
Date Deposited: 02 Oct 2020 20:02
Last Modified: 25 Mar 2021 06:17