Task Prompt Vectors: Effective Initialization through Multi-Task Soft Prompt Transfer

Prompt tuning is a parameter-efficient method for adapting large language models (LLMs), where only a small continuous soft prompt is finetuned. In recent works, soft prompts have usually been trained in a task-specific way, leaving their multi-task capabilities underexplored. Our work aims to make soft prompts more task modular based on recent research on task vectors, where arithmetic operations are applied on full model weights to achieve the desired multi-task performance. To this end, we introduce Task Prompt Vectors, created by the element-wise difference between weights of tuned soft prompts and their random initialization. Experimental results on an extensive set of 19 datasets show that task prompt vectors can be used in low-resource settings to initialize prompt tuning on similar tasks effectively. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning on 3 different language model architectures. This key property of random initialization independence allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, the arithmetic addition of task prompt vectors from multiple tasks represents a competitive and computationally more effective alternative to state-of-the-art solutions.