Pruning Networks With Cross-Layer Ranking & k-Reciprocal Nearest Filters

Authors

Mingbao Lin, Xiamen University
Liujuan Cao, Xiamen University
Yuxin Zhang, Xiamen University
Ling Shao, Inception Institute of Artificial Intelligence & Mohamed bin Zayed University of Artificial Intelligence
Chia Wen Lin, National Tsing Hua University
Rongrong Ji, Xiamen University

Document Type

Article

Publication Title

IEEE Transactions on Neural Networks and Learning Systems

Abstract

This article focuses on filter-level network pruning. A novel pruning method, termed CLR-RNF, is proposed. We first reveal a ``long-tail'' pruning problem in magnitude-based weight pruning methods and then propose a computation-aware measurement for individual weight importance, followed by a cross-layer ranking (CLR) of weights to identify and remove the bottom-ranked weights. Consequently, the per-layer sparsity makes up the pruned network structure in our filter pruning. Then, we introduce a recommendation-based filter selection scheme where each filter recommends a group of its closest filters. To pick the preserved filters from these recommended groups, we further devise a k-reciprocal nearest filter (RNF) selection scheme where the selected filters fall into the intersection of these recommended groups. Both our pruned network structure and the filter selection are nonlearning processes, which, thus, significantly reduces the pruning complexity and differentiates our method from existing works. We conduct image classification on CIFAR-10 and ImageNet to demonstrate the superiority of our CLR-RNF over the state-of-the-arts. For example, on CIFAR-10, CLR-RNF removes 74.1% FLOPs and 95.0% parameters from VGGNet-16 with even 0.3% accuracy improvements. On ImageNet, it removes 70.2% FLOPs and 64.8% parameters from ResNet-50 with only 1.7% top-five accuracy drops. Our project is available at https://github.com/lmbxmu/CLR-RNF.

First Page

1

Last Page

10

DOI

10.1109/TNNLS.2022.3156047

Publication Date

3-16-2022

Keywords

Computer architecture, Efficient inference, filter pruning, Indexes, Informatics, model compression, network structure., Search problems, Tensors, Training, Weight measurement

Comments

IR Deposit conditions:

OA version (pathway a) Accepted version

No embargo

When accepted for publication, set statement to accompany deposit (see policy)

Must link to publisher version with DOI

Publisher copyright and source must be acknowledged

Recommended Citation

M. Lin, L. Cao, Y. Zhang, L. Shao, C. -W. Lin and R. Ji, "Pruning Networks With Cross-Layer Ranking & k-Reciprocal Nearest Filters," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2022.3156047.