I2T21: Learning text to image synthesis with textual data augmentation
File(s)1703.06676v3.pdf (3.8 MB)
Accepted version
Author(s)
Dong, Hao
Zhang, Jingqing
McIlwraith, Douglas
Guo, Yike
Type
Conference Paper
Abstract
Translating information between text and image is a fundamental problem in artificial intelligence that connects natural language processing and computer vision. In the past few years, performance in image caption generation has seen significant improvement through the adoption of recurrent neural networks (RNN). Meanwhile, text-to-image generation begun to generate plausible images using datasets of specific categories like birds and flowers. We've even seen image generation from multi-category datasets such as the Microsoft Common Objects in Context (MSCOCO) through the use of generative adversarial networks (GANs). Synthesizing objects with a complex shape, however, is still challenging. For example, animals and humans have many degrees of freedom, which means that they can take on many complex shapes. We propose a new training method called Image-Text-Image (I2T2I) which integrates text-to-image and image-to-text (image captioning) synthesis to improve the performance of text-to-image synthesis. We demonstrate that I2T2I can generate better multi-categories images using MSCOCO than the state-of-the-art. We also demonstrate that I2T2I can achieve transfer learning by using a pre-trained image captioning module to generate human images on the MPII Human Pose dataset (MHP) without using sentence annotation.
Date Issued
2018-02-22
Date Acceptance
2017-09-17
Citation
2017 IEEE International Conference on Image Processing (ICIP), 2018
ISBN
9781509021765
ISSN
1522-4880
Publisher
Institute of Electrical and Electronics Engineers
Journal / Book Title
2017 IEEE International Conference on Image Processing (ICIP)
Copyright Statement
© 2018 Institute of Electrical and Electronics Engineers.
Identifier
http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000428410702028&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=1ba7043ffcc86c417c072aa74d649202
Source
24th IEEE International Conference on Image Processing (ICIP)
Subjects
Science & Technology
Technology
Imaging Science & Photographic Technology
Deep learning
GAN
Image Synthesis
Publication Status
Published
Start Date
2017-09-17
Finish Date
2017-09-20
Coverage Spatial
Beijing, China
Date Publish Online
2018-02-22