Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation

Authors

Haohan Wang, Carnegie Mellon University, Pittsburgh, PA, United States
Zeyi Huang, University of Wisconsin-Madison, Madison, PA, United States
Xindi Wu, Carnegie Mellon University, Pittsburgh, PA, United States
Eric P. Xing, Carnegie Mellon University, Pittsburgh, PA, United States & Mohamed bin Zayed University of Artificial IntelligenceFollow

Document Type

Conference Proceeding

Publication Title

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Abstract

Data augmentation has been proven to be an effective technique for developing machine learning models that are robust to known classes of distributional shifts (e.g., rotations of images), and alignment regularization is a technique often used together with data augmentation to further help the model learn representations invariant to the shifts used to augment the data. In this paper, motivated by a proliferation of options of alignment regularizations, we seek to evaluate the performances of several popular design choices along the dimensions of robustness and invariance, for which we introduce a new test procedure. Our synthetic experiment results speak to the benefits of squared "2 norm regularization. Further, we also formally analyze the behavior of alignment regularization to complement our empirical study under assumptions we consider realistic. Finally, we test this simple technique we identify (worst-case data augmentation with squared "2 norm alignment regularization) and show that the benefits of this method outrun those of the specially designed methods. We also release a software package in both TensorFlow and PyTorch for users to use the method with a couple of lines at https://github.com/jyanln/AlignReg. © 2022 Owner/Author.

First Page

1846

Last Page

1856

DOI

10.1145/3534678.3539438

Publication Date

8-14-2022

Keywords

data augmentation, machine learning, robustness, trustworthy, Testing

Comments

IR Deposit conditions: non-described

Recommended Citation

H. Wang, Z. Huang, X. Wu, and E.P. Xing, "Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation", in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '22), NY, USA, pp. 1846–1856. doi:10.1145/3534678.3539438