Emerging Architecture Patterns

State space models, Mixture of Depths, sparse attention patterns, and novel positional encodings.