RoBERTa

Google Introduces New Architecture To Reduce Cost Of Transformers

Primer’s improvements can be attributed to two simple modifications -- squaring ReLU activations and adding a depthwise convolution layer after each Q, K, and V projection in self-attention.