GiBERT: Enhancing BERT with linguistic information using a lightweight gated injection method
File(s)
Published version
Author(s)
Peinelt, Nicole
Rei, Marek
Liakata, Maria
Type
Conference Paper
Abstract
Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words – either through masking or next sentence prediction – and has no knowledge of lexical, syntactic or semantic information beyond what it picks up through unsupervised pre-training. We propose a novel method to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. When injecting counter-fitted and dependency-based embeddings, the performance improvements on multiple semantic similarity datasets indicate that such information is beneficial and currently missing from the original model. Our qualitative analysis shows that counter-fitted embedding injection is particularly beneficial, with notable improvements on examples that require synonym resolution.
Date Issued
2021
Date Acceptance
2021-11-07
Citation
Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp.2322-2336
Publisher
Association for Computational Linguistics
Start Page
2322
End Page
2336
Journal / Book Title
Findings of the Association for Computational Linguistics: EMNLP 2021
Copyright Statement
ACL materials are Copyright © 1963–2024 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.
License URL
Identifier
http://dx.doi.org/10.18653/v1/2021.findings-emnlp.200
Source
The 2021 Conference on Empirical Methods in Natural Language Processing
Publication Status
Published
Start Date
2021-11-07
Finish Date
2021-11-11
Coverage Spatial
Online and in the Barceló Bávaro Convention Centre, Punta Cana, Dominican Republic
Date Publish Online
2021