Zero-shot sequence labeling for transformer-based sentence classifiers
File(s)
Accepted version
OA Location
Author(s)
Bujel, Kamil
Yannakoudakis, Helen
Rei, Marek
Type
Conference Paper
Abstract
We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.
Date Issued
2021-08-06
Date Acceptance
2021-08-01
Citation
Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), 2021, pp.195-205
Publisher
Association for Computational Linguistics
Start Page
195
End Page
205
Journal / Book Title
Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)
Copyright Statement
© 2021 Association for Computational Linguistics.
Source
Joint Conference of 59th Annual Meeting of the Association-for-Computational-Linguistics (ACL) / 11th International Joint Conference on Natural Language Processing (IJCNLP) / 6th Workshop on Representation Learning for NLP (RepL4NLP)
Subjects
Computer Science
Computer Science, Artificial Intelligence
Computer Science, Interdisciplinary Applications
Linguistics
Science & Technology
Social Sciences
Technology
Publication Status
Published
Start Date
2021-08-06
Finish Date
2021-08-06
Coverage Spatial
Virtual