Code to reproduce plots from the paper: Activation function design for deep networks: linearity and effective initialization https://arxiv.org/abs/2105.07741
- For implementation of Variance and Covariance maps; see Notebook
- For analysing an activation function via correlation, moment ratio and dynamical isometry bounds; see Notebook
- For training a DNN model on MNIST/Fashion-MNIST/Cifar-10 dataset; see script
- require pre-computed values of
for a given
; use functions from Notebook
- require pre-computed values of
- For training with fixed value of parameter 'a' of an activation function; import script
- train and trainO functions are provided to train with Gaussian or orthogonal weights
- For training with variable value of parameter 'a' of an activation function; import script
- Only implementation with htanh activation is provided for demo purpose
Vinayak Abrol [email protected]
Michael Murray [email protected]