Tootfinder

Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization
Sergii Kavun
https://arxiv.org/abs/2507.22090 https://

Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization
Activation functions are critical components in deep neural networks, directly influencing gradient flow, training stability, and model performance. Traditional functions like ReLU suffer from dead neuron problems, while sigmoid and tanh exhibit vanishing gradient issues. We introduce two novel hybrid activation functions: S3 (Sigmoid-Softsign) and its improved version S4 (smoothed S3). S3 combines sigmoid for negative inputs with softsign for positive inputs, while S4 employs a smooth transiti…