Control prefixes for parameter-efficient text generation
File(s)
Preprint
OA Location
Author(s)
Clive, Jordan
Cao, Kris
Rei, Marek
Type
preprint
Abstract
Prefix-tuning is a powerful lightweight technique for adapting a large pre-trained language model to a downstream application. However, it uses the same dataset-level tuned prompt for all examples in the dataset. We extend this idea and propose a dynamic method, CONTROL PREFIXES, which allows for the inclusion of conditional input-dependent information, combining the benefits of prompt tuning and controlled generation. The method incorporates attribute-level learnable representations into different layers of a pre-trained transformer, allowing for the generated text to be guided in a particular direction. We provide a systematic evaluation of the technique and apply it to five datasets from the GEM benchmark for natural language generation (NLG). Although the aim is to develop a parameterefficient model, using only 0.1-3% trainable parameters, we show CONTROL PREFIXES can even outperform full fine-tuning methods. We present state-of-the-art results on several data-to-text datasets, including WebNLG.
Date Issued
2021-10-15
Citation
arXiv, 2021
Journal / Book Title
arXiv
Copyright Statement
Copyright © 2021 The Author(s). This work is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/).
License URL
Identifier
https://arxiv.org/abs/2110.08329v2
Subjects
cs.CL
cs.CL
cs.AI
cs.LG