Stanford Univ: New AI Attention: SwiLA

Softmax attention’s KV-cache is a dead end for infinite (or extreme large) context, but standard Linear Attention fundamentally destroys associative recall (Mamba, DeltaNet). So how do we get the best of both worlds? Meet SwiLA (Switching Linear Attention) by Stanford Univ. Switching Linear Attention completely eliminates the transformer's massive memory cache without sacrificing associative recall by modeling sequence routing as online expectation-maximization over a mixture of linear regress