Abstract: The Softmax activation function plays a pivotal role in both the attention mechanism of Transformers and in the final layer of neural networks performing classification. The Softmax function ...