Modeling Pooling Layers For CNN Initialization

Deep convolutional neural networks (CNNs) have achieved consistent excellent performance on image processing tasks. The CNN architecture consists of different types of layers, including the convolution layer and the max pooling layer. It is widely understood among CNN practitioners that the stability of learning depends on the initialization of the model parameters in each layer. Currently, the de facto standard scheme for initialization is the Kaiming initialization, which was developed by He et al. The Kaiming scheme was derived from a much simpler model than the currently used CNN structure that evolved since the emergence of the Kaiming scheme. It consists only of the convolution and fully connected layers, and does not include the max pooling or global average pooling layers. In this study, we derive an initialization scheme, not from the simplified Kaiming model, but from modern CNN architectures consisting not only of the convolution and the fully connected layers but also of the max pooling and the global average pooling. Furthermore, the new model expresses the padding which is not considered in the existing models. We empirically investigate the performance of the new initialization methods compared to the de facto standard methods that are widely used today.