Comparing Different Sequences of Pruning Algorithms for Hybrid Pruning

Abstract

Most developers face two significant issues while designing the architecture of a neural network. First, the available dataset for many real-life problems is relatively small, leading to overfitting. Second, When a dataset is large enough, the computation cost to train the model on a given dataset is enormous. Thus most developers use transfer learning with a standard model like VGGNet, ResNet, and GoogleNet. These standard models are memory and computationally expensive during inference, making them infeasible to deploy on resource-constrained devices. The recent research trend is to compress the standard model used for transfer learning to reduce memory and computing costs. In CNN, approximately 10% parameters are present in the convolution layer, contributing to 90% of computational cost, while 90% parameters are present in dense, contributing 10% of computational cost. This paper focuses on the structure pruning of parameters in the convolution layer to reduce computational costs. Here we explore and compare the following pruning technique, 1) Channel pruning with a quantitative score, 2) Kernel pruning with a quantitative score, 3) Channel pruning with a similarity score, and 4) Kernel pruning with a similarity score. Finally, as mentioned earlier, we try several combinations of pruning to form a hybrid pruning. © 2023 IEEE.