A few topics/resources that I needed recently as a refresher. Need to summarize at a later date…
RNNs
Improving learning
- https://pytorch.org/docs/stable/_modules/torch/nn/modules/normalization.html
- http://ceur-ws.org/Vol-2142/paper4.pdf
- https://github.com/DingKe/pytorch_workplace/blob/master/rnn/modules.py#L122
- https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/14
- https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/language_model/main.py
- https://forums.fast.ai/t/30-best-practices/12344
Variable RNN
- https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pack_padded_sequence
- https://towardsdatascience.com/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e
- https://discuss.pytorch.org/t/understanding-pack-padded-sequence-and-pad-packed-sequence/4099/6
- https://gist.github.com/Tushar-N/dfca335e370a2bc3bc79876e6270099e
Calculating trainable parameters and flops
Flops
- http://machinethink.net/blog/how-fast-is-my-model/
- https://stats.stackexchange.com/questions/328926/how-many-parameters-are-in-a-gated-recurrent-unit-gru-recurrent-neural-network
- https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/
- https://piazza.com/class/jjjilbkqk8m1r4?cid=1063
- https://stats.stackexchange.com/questions/291843/how-to-understand-calculate-flops-of-the-neural-network-model
Trainable parameters
- https://stackoverflow.com/questions/42786717/how-to-calculate-the-number-of-parameters-for-convolutional-neural-network
- https://www.learnopencv.com/number-of-parameters-and-tensor-sizes-in-convolutional-neural-network/
- https://stats.stackexchange.com/questions/328926/how-many-parameters-are-in-a-gated-recurrent-unit-gru-recurrent-neural-network
Random
- https://documents.epfl.ch/users/f/fl/fleuret/www/dlc/dlc-handout-6-going-deeper.pdf