Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks

Authors

Yichen Huang, Mohamed bin Zayed University of Artificial IntelligenceFollow
Timothy Baldwin, Mohamed bin Zayed University of Artificial IntelligenceFollow

Document Type

Conference Proceeding

Publication Title

Findings of the Association for Computational Linguistics: EMNLP 2023

Abstract

We investigate MT evaluation metric performance on adversarially-synthesized texts, to shed light on metric robustness. We experiment with word- and character-level attacks on three popular machine translation metrics: BERTScore, BLEURT, and COMET. Our human experiments validate that automatic metrics tend to overpenalize adversarially-degraded translations. We also identify inconsistencies in BERTScore ratings, where it judges the original sentence and the adversarially-degraded one as similar, while judging the degraded translation as notably worse than the original with respect to the reference. We identify patterns of brittleness that motivate more robust metric development.

First Page

5126

Last Page

5135

Publication Date

1-1-2023

Recommended Citation

Y. Huang and T. Baldwin, "Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks," Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5126 - 5135, Jan 2023.