2
IRUS Total
Downloads
  Altmetric

I2T21: Learning text to image synthesis with textual data augmentation

File Description SizeFormat 
1703.06676v3.pdfAccepted version3.89 MBAdobe PDFView/Open
Title: I2T21: Learning text to image synthesis with textual data augmentation
Authors: Dong, H
Zhang, J
McIlwraith, D
Guo, Y
Item Type: Conference Paper
Abstract: Translating information between text and image is a fundamental problem in artificial intelligence that connects natural language processing and computer vision. In the past few years, performance in image caption generation has seen significant improvement through the adoption of recurrent neural networks (RNN). Meanwhile, text-to-image generation begun to generate plausible images using datasets of specific categories like birds and flowers. We've even seen image generation from multi-category datasets such as the Microsoft Common Objects in Context (MSCOCO) through the use of generative adversarial networks (GANs). Synthesizing objects with a complex shape, however, is still challenging. For example, animals and humans have many degrees of freedom, which means that they can take on many complex shapes. We propose a new training method called Image-Text-Image (I2T2I) which integrates text-to-image and image-to-text (image captioning) synthesis to improve the performance of text-to-image synthesis. We demonstrate that I2T2I can generate better multi-categories images using MSCOCO than the state-of-the-art. We also demonstrate that I2T2I can achieve transfer learning by using a pre-trained image captioning module to generate human images on the MPII Human Pose dataset (MHP) without using sentence annotation.
Issue Date: 22-Feb-2018
Date of Acceptance: 17-Sep-2017
URI: http://hdl.handle.net/10044/1/73761
DOI: https://doi.org/10.1109/ICIP.2017.8296635
ISBN: 9781509021765
ISSN: 1522-4880
Publisher: Institute of Electrical and Electronics Engineers
Journal / Book Title: 2017 IEEE International Conference on Image Processing (ICIP)
Copyright Statement: © 2018 Institute of Electrical and Electronics Engineers.
Conference Name: 24th IEEE International Conference on Image Processing (ICIP)
Keywords: Science & Technology
Technology
Imaging Science & Photographic Technology
Deep learning
GAN
Image Synthesis
Science & Technology
Technology
Imaging Science & Photographic Technology
Deep learning
GAN
Image Synthesis
cs.CV
cs.CV
cs.CL
Publication Status: Published
Start Date: 2017-09-17
Finish Date: 2017-09-20
Conference Place: Beijing, China
Online Publication Date: 2018-02-22
Appears in Collections:Computing