arxiv:2402.19427

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Published on Feb 29

· Submitted by

akhaliq on Mar 1

#2 Paper of the day

Upvote

Authors:

Aleksandar Botev ,

Albert Gu ,

Srivatsan Srinivasan ,

Razvan Pascanu ,

Caglar Gulcehre

Abstract

Recurrent neural networks (RNNs) have fast inference and scale efficiently on long sequences, but they are difficult to train and hard to scale. We propose Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that mixes gated linear recurrences with local attention. Hawk exceeds the reported performance of Mamba on downstream tasks, while Griffin matches the performance of Llama-2 despite being trained on over 6 times fewer tokens. We also show that Griffin can extrapolate on sequences significantly longer than those seen during training. Our models match the hardware efficiency of Transformers during training, and during inference they have lower latency and significantly higher throughput. We scale Griffin up to 14B parameters, and explain how to shard our models for efficient distributed training.

View arXiv page View PDF Add to collection

Community

KrishnaKaasyap

Mar 1

This is awesome. New architecture - new possibilities!

And are these architectures more optimised for TPUs than GPUs?

And are you gonna release a comparison of Griffin 14B with Mixtral which is almost 13B model (2×7B MoE) though trained on far more tokens than 300B?

And why did you selected Llama but not Mistral 7B for comparison? May be because there is no information about how many tokens it was trained on?

HaileyStorm

Mar 1

Not an author or anything, but yeah, they use llama because they do an closer comparison with it (most papers do this, especially when they don't train a model fully).