Abstract
Convolutional Neural Networks (CNNs) are widely used in recent years.They have shown their ability to handle wide range of learning problems, especially in computer vision applications such as image classification, detection and segmentation. Many state-of-the-art models have been proposed to solve a large variety of these problems. These existing models are powerful in learning problems but also computational expensive. With the CNN models growing deeper and deeper, their computation cost also grows. Huge computation cost makes deep CNNs hard to apply to resource constrained devices such as smart phone or IOT devices. Accelerating CNNs has become a critical topic if we want to apply these powerful models in such kind of applications. Since deep CNNs models tend to be over-parametrized, reducing their redundancy is helpful to accelerating the computation as well as compressing the model size. In this work, we adopt the low rank approximation technique to accelerate the inferences of CNNs. The low rank approximation has been used to reduce CNN's redundancy in many publications. By converting filter tensors in convolutional layers into smaller vectors, we can substitute the original complex 2D convolutional layers with simpler layers such as 1$\times$1 convolution and depth-wise convolution. Overall computation amount could be therefore decreased. However some issues come out. Because there are no direct algorithms to calculate one tensor's specific rank, we need some heuristics to determine the ranks. The other problem is the CP instability, which makes networks which are decomposed by CP decomposition hard to be fine-tuned. Once the accuracy drops due to the CP decomposition, it's difficult to recover the accuracy by training process. Based on these observations, we propose the Rank Selection approach for efficient CP decomposition and Two-Pass decomposition to avoid the CP Instability. According to the experiment result, our Rank Selection could determine the effective ranks with improved performance while maintaining the target speedup. With the proposed Two-Pass decomposition technique, by avoiding the ineffective training on the decomposed layers, we can obtain the decomposed models with higher accuracy. For deeper networks, iterative Two-Pass decomposition is proposed to further improve the accuracy. In this work, we have successfully achieved 6$\times$ speedup in VGG16 with only 1.20\% accuracy drop and 1.35$\times$ speedup in RestNet50 with 1.51\% accuracy drop. Comparing with other works, our Two-Pass decomposition is more universal. It can be easily adopted to different deep learning frameworks because only 1$\times$1 convolution and depth-wise convolution are used. Our method is based on data driven approach to optimize the performance, without any specific decomposition method needed. Therefore, unlike other previous methods, the proposed one could be extended to other decomposing forms for accelerating deep neural networks.