Towards preserving word order importance through Forced Invalidation
File(s)
Published version
Author(s)
Al-Negheimish, H
Madhyastha, P
Russo, A
Type
Conference Paper
Abstract
Large pre-trained language models such as BERT have been widely used as a framework for natural language understanding (NLU) tasks. However, recent findings have revealed that pre-trained language models are insensitive to word order. The performance on NLU tasks remains unchanged even after randomly permuting the word of a sentence, where crucial syntactic information is destroyed. To help preserve the importance of word order, we propose a simple approach called FORCED INVALIDATION (FI): forcing the model to identify permuted sequences as invalid samples. We perform an extensive evaluation of our approach on various English NLU and QA based tasks over BERT-based and attention-based models over word embeddings. Our experiments demonstrate that FI significantly improves the sensitivity of the models to word order.
Date Issued
2023-05-02
Date Acceptance
2023-05-01
Citation
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp.2555-2562
ISBN
9781959429449
Publisher
Association for Computational Linguistics
Start Page
2555
End Page
2562
Journal / Book Title
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
Copyright Statement
©2023 Association for Computational Linguistics. Published under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/)
License URL
Source
EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics
Publication Status
Published
Start Date
2023-05-02
Finish Date
2023-05-06
Coverage Spatial
Dubrovnik, Croatia