Paper: Coordinating Filters for Faster Deep Neural Networks.

Poster is available.

source code is in this master branch.

See the source code in branch scnn

A work to accelerate training. code

Originally in branch intel, but merged to IntelLabs/SkimCaffe with contributions also by @jspark1105

Master branch is from caffe @ commit eb4ba30

Tutorials on using python to decompose DNNs to low-rank space is here.

If any problems/bugs/questions, you are welcome to open an issue and we will response asap.

Details of Force Regularization is in the Paper: Coordinating Filters for Faster Deep Neural Networks.

It is easy to use the code to train DNNs toward lower-rank DNNs. Only three additional protobuf configurations are required:

1. force_decay in SolverParameter: Specified in solver. The coefficient to make the trade-off between accuracy and ranks. Larger force_decay, smaller ranks and usually lower accuracy.
2. force_type in SolverParameter: Specified in solver. The kind of force to coordinate filters. Degradation - The strength of pairwise attractive force decreases as the distance decreases. This is the L2-norm force in the paper; Constant - The strength of pairwise attractive force keeps constant regardless of the distance. This is the L1-norm force in the paper.
3. force_mult in ParamSpec: Specified for the param of weights in each layer. The local multiplier of force_decay for filters in a specific layer, i.e., force_mult*force_decay is the final coefficient for the specific layer. You can set force_mult: 0.0 to eliminate force regularization in any layer.

See details and implementations in caffe.proto and SGDSolver

An example of training LeNet with L1-norm force regularization:

##############################################################\
# The train/test net with local force decay multiplier       
net: "examples/mnist/lenet_train_test_force.prototxt"        
##############################################################/

test_iter: 100
test_interval: 500
# The base learning rate. For large-scale DNNs, you might try 0.1x smaller base_lr of training the original DNNs from scratch.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005

##############################################################\
# The coefficient of force regularization.                   
# The hyper-parameter to tune to make trade-off              
force_decay: 0.001                                           
# The type of force - L1-norm force                          
force_type: "Constant"                                       
##############################################################/

# The learning rate policy
lr_policy: "multistep"
gamma: 0.9
stepvalue: 5000
stepvalue: 7000
stepvalue: 8000
stepvalue: 9500
stepvalue: 9500
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 10000
# snapshot intermediate results
snapshot: 5000
snapshot_prefix: "examples/mnist/lower_rank_lenet"
snapshot_format: HDF5
solver_mode: GPU

Retraining a trained DNN with force regularization might get better results, comparing with training from scratch.

We included the hyperparameter of "lambda_s" for AlexNet in Figure 6.

Force Regularization can squeeze/coordinate weight information to much lower rank space, but after low-rank decomposition with the same precision of approximation, it is more challenging to recover the accuracy from the much more lightweight DNNs.

Please cite our ICCV and Caffe if it is useful for your research:

@InProceedings{Wen_2017_ICCV,
  author={Wen, Wei and Xu, Cong and Wu, Chunpeng and Wang, Yandan and Chen, Yiran and Li, Hai},
  title={Coordinating Filters for Faster Deep Neural Networks},
  booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
  month = {October},
  year = {2017}
}

Caffe is released under the BSD 2-Clause license. The BVLC reference models are released for unrestricted use.

Please cite Caffe in your publications if it helps your research:

@article{jia2014caffe,
  Author = {Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and Karayev, Sergey and Long, Jonathan and Girshick, Ross and Guadarrama, Sergio and Darrell, Trevor},
  Journal = {arXiv preprint arXiv:1408.5093},
  Title = {Caffe: Convolutional Architecture for Fast Feature Embedding},
  Year = {2014}
}