Skip to content

Latest commit

 

History

History
54 lines (40 loc) · 10.6 KB

README.md

File metadata and controls

54 lines (40 loc) · 10.6 KB

GN + WS

Weight Standardization

Abstract

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

Results and Models

Faster R-CNN

Backbone Style Normalization Lr schd Mem (GB) Inf time (fps) box AP mask AP Config Download
R-50-FPN pytorch GN+WS 1x 5.9 11.7 39.7 - config model | log
R-101-FPN pytorch GN+WS 1x 8.9 9.0 41.7 - config model | log
X-50-32x4d-FPN pytorch GN+WS 1x 7.0 10.3 40.7 - config model | log
X-101-32x4d-FPN pytorch GN+WS 1x 10.8 7.6 42.1 - config model | log

Mask R-CNN

Backbone Style Normalization Lr schd Mem (GB) Inf time (fps) box AP mask AP Config Download
R-50-FPN pytorch GN+WS 2x 7.3 10.5 40.6 36.6 config model | log
R-101-FPN pytorch GN+WS 2x 10.3 8.6 42.0 37.7 config model | log
X-50-32x4d-FPN pytorch GN+WS 2x 8.4 9.3 41.1 37.0 config model | log
X-101-32x4d-FPN pytorch GN+WS 2x 12.2 7.1 42.1 37.9 config model | log
R-50-FPN pytorch GN+WS 20-23-24e 7.3 - 41.1 37.1 config model | log
R-101-FPN pytorch GN+WS 20-23-24e 10.3 - 43.1 38.6 config model | log
X-50-32x4d-FPN pytorch GN+WS 20-23-24e 8.4 - 42.1 38.0 config model | log
X-101-32x4d-FPN pytorch GN+WS 20-23-24e 12.2 - 42.7 38.5 config model | log

Note:

  • GN+WS requires about 5% more memory than GN, and it is only 5% slower than GN.
  • In the paper, a 20-23-24e lr schedule is used instead of 2x.
  • The X-50-GN and X-101-GN pretrained models are also shared by the authors.

Citation

@article{weightstandardization,
  author    = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Yuille},
  title     = {Weight Standardization},
  journal   = {arXiv preprint arXiv:1903.10520},
  year      = {2019},
}