Single Image Texture Translation for Data Augmentation

1Cornell University, 2Cornell Tech, 3Google Research, Brain Team

Single Image Texture Translation (SITT) trained with a single texture source and target image.

Abstract

Recent advances in image synthesis enables one to translate images by learning the mapping between a source domain and a target domain. Existing methods tend to learn the distributions by training a model on a variety of datasets, with results evaluated largely in a subjective manner. Relatively few works in this area, however, study the potential use of semantic image translation methods for image recognition tasks. In this paper, we explore the use of Single Image Texture Translation (SITT) for data augmentation. We first propose a lightweight model for translating texture to images based on a single input of source texture, allowing for fast training and testing. Based on SITT, we then explore the use of augmented data in long-tailed and few-shot image classification tasks. We find the proposed method is capable of translating input data into a target domain, leading to consistent improved image recognition performance. Finally, we examine how SITT and related image translation methods can provide a basis for a data-efficient, augmentation engineering approach to model training.

Video

Plant Pathology: Healthy to Sick

The use of SITT for augmenting data largely improves the performance of ResNet-18 by 5-10% for long-tailed classification on Plant Pathology 2020 dataset.

Plant Pathology: Healthy to Multi-disease

The use of SITT for augmenting data largely improves the performance of ResNet-18 by 5-10% for long-tailed classification on Plant Pathology 2020 dataset.

Oxford Flowers

The use of SITT for augmenting data improves the performance of ResNet-18 by 1.5% for 5-shot classification on 102 Category Flower Dataset.

Caltech-UCSD Birds 200

The use of SITT for augmenting data improves the performance of ResNet-18 by 1.4% for 5-shot classification on CUB-200-2011.

Camouflage

Mimicry or Camouflage in natural world provides some real examples for texture swapping that creatures make the textures of their body similar to the environment’s to avoid danger or hunt food.

Orchid-Mantis

For instance, we show the case of mantis and orchid. To prey for the insects, mantis will adaptively change their texture similar to the orchids. In the given example, we give an illustration and find SITT could translate the orchids’ texture to the mantis and obtain reasonable and natural outputs that looks very close to the real example.

BibTeX

@article{li2021single,
    title={Single Image Texture Translation for Data Augmentation},
    author={Li, Boyi and Cui, Yin and Lin, Tsung-Yi and Belongie, Serge},
    journal={arXiv preprint arXiv:2106.13804},
    year={2021}
    }