Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models

Vijay Sadashivaiah, George Dasoulas, Judith Mueller, Soumya Ghosh

Action editor: Ole Winther

https://openreview.net/forum?id=GxmQsP5CcU

#softmax #sigmoid #attention

On the Role of MLP Layers in Transformer ICL with Categorical Outcomes

Soumya Banerjee, Aaron T Wang, William Convertino et al.

Action editor: Alexander S. Ecker

https://openreview.net/forum?id=8P4V7V1cs4

#softmax #attention #transformers

The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models

Karthik Viswanathan, Yuri Gardinazzi, Giada Panerai, Alberto Cazzaniga, Matteo Biagetti

Action editor: Zachary Charles

https://openreview.net/forum?id=rBEgNAslpY

#softmax #interpretability