Handwriting transformers

Ankan Kumar Bhunia, Mohamed bin Zayed University of Artificial Intelligence
Salman Khan, Mohamed bin Zayed University of Artificial Intelligence
Hisham Cholakkal, Mohamed bin Zayed University of Artificial Intelligence
Rao Muhammad Anwer, Mohamed bin Zayed University of Artificial Intelligence
Fahad Shahbaz Khan, Mohamed bin Zayed University of Artificial Intelligence
Mubarak A. Shah, University of Central Florida

Preprint: arXiv

Abstract

We propose a novel transformer-based styled handwritten text image generation approach, HWT, that strives to learn both style-content entanglement as well as global and local writing style patterns. The proposed HWT captures the long and short range relationships within the style examples through a self-attention mechanism, thereby encoding both global and local style patterns. Further, the proposed transformer-based HWT comprises an encoder-decoder attention that enables style-content entanglement by gathering the style representation of each query character. To the best of our knowledge, we are the first to introduce a transformer-based generative network for styled handwritten text generation. Our proposed HWT generates realistic styled handwritten text images and significantly outperforms the state-of-the-art demonstrated through extensive qualitative, quantitative and human-based evaluations. The proposed HWT can handle arbitrary length of text and any desired writing style in a few-shot setting. Further, our HWT generalizes well to the challenging scenario where both words and writing style are unseen during training, generating realistic styled handwritten text images. © 2021, CC BY.