Regularization in Deep Learning (Dropout, Batch Norm)

Summary

Techniques specific to neural networks that reduce overfitting — dropout randomly disables neurons during training, batch normalization stabilizes and speeds up training by normalizing layer inputs.

• 1Add dropout to a network and observe the effect on the train/validation performance gap.
• 2Learn where batch normalization layers are typically placed in a network.
• 3Compare training stability and speed with and without batch normalization.