Distilling robustness into natural language inference models with domain-targeted augmentation
File(s)
Published version
Author(s)
Stacey, J
Rei, M
Type
Conference Paper
Abstract
Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some of the performance benefits. While this method can improve results on in-distribution examples, it does not necessarily generalise to out-of-distribution (OOD) settings. We investigate two complementary methods for improving the robustness of the resulting student models on OOD domains. The first approach augments the distillation with generated unlabelled examples that match the target distribution. The second method upsamples data points among the training set that are similar to the target distribution. When applied on the task of natural language inference (NLI), our experiments on MNLI show that distillation with these modifications outperforms previous robustness solutions. We also find that these methods improve performance on OOD domains even beyond the target domain.
Date Issued
2024-01-01
Date Acceptance
2024-08-01
Citation
Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2024, pp.2239-2258
ISSN
0736-587X
Publisher
ACL Anthology
Start Page
2239
End Page
2258
Journal / Book Title
Proceedings of the Annual Meeting of the Association for Computational Linguistics
Copyright Statement
©2024 Association for Computational Linguistics. Licensed on a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/)
License URL
Identifier
https://aclanthology.org/2024.findings-acl.132/
Source
Findings of the Association for Computational Linguistics ACL 2024
Publication Status
Published
Start Date
2024-08-11
Finish Date
2024-08-16
Coverage Spatial
Bangkok, Thailand
Date Publish Online
2024-08