In this paper, we teach machines to understand visuals and natural language
by learning the mapping between sentences and noisy video snippets without
explicit annotations. Firstly, we define a self-supervised learning framework
that captures the cross-modal information. A novel adversarial learning module
is then introduced to explicitly handle the noises in the natural videos, where
the subtitle sentences are not guaranteed to be strongly corresponded to the
video snippets. For training and evaluation, we contribute a new dataset
`ApartmenTour' that contains a large number of online videos and subtitles. We
carry out experiments on the bidirectional retrieval tasks between sentences
and videos, and the results demonstrate that our proposed model achieves the
state-of-the-art performance on both retrieval tasks and exceeds several strong
baselines. The dataset will be released soon.