Language modeling teaches you more than translation does: lessons learned through auxiliary syntactic task analysis
Author(s)
Zhang, Kelly
Bowman, Samuel
Type
Conference Paper
Abstract
Recently, researchers have found that deep LSTMs trained on tasks like machine translation learn substantial syntactic and semantic information about their input sentences, including part-of-speech. These findings begin to shed light on why pretrained representations, like ELMo and CoVe, are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives—language modeling, translation, skip-thought, and autoencoding—on their ability to induce syntactic and part-of-speech information, holding constant the quantity and genre of the training data, as well as the LSTM architecture.
Date Issued
2018-10-31
Date Acceptance
2018-10-01
Citation
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018, pp.359-361
Publisher
Association for Computational Linguistics
Start Page
359
End Page
361
Journal / Book Title
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
Copyright Statement
©2018 Association for Computational Linguistics.
Source
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
Publication Status
Published
Start Date
2018-10-31
Finish Date
2018-11-04
Coverage Spatial
Brussels, Belgium