[1]

# Tiny-Attention Adapter:
Contexts Are More Important Than the Number of Parameters

Hongyu Zhao
University of Chicago
<EMAIL_ADDRESS>
&Hao Tan
Adobe Research
<EMAIL_ADDRESS>
&Hongyuan Mei
TTI-Chicago
<EMAIL_ADDRESS>Work done while during internship at TTI-Chicago.

###### Abstract

Adapter-tuning is a paradigm that transfers a pretrained language model to
downstream tasks by adding and tuning a small number of new parameters.
Previously proposed adapter architectures are all feed-forward neural
networks. In this paper, we investigate the effectiveness of using _tiny-
attention_ —i.e., attention with extremely small per-head dimensionality—as
adapters. Our tiny-attention adapter learns to modify the hidden states at
each position directly conditioned on the hidden states at all the other
positions, which is missed by the previously proposed adapters. Moreover, we
view its multiple attention heads as a mixture of experts and propose to
average their weights during deployment, which further reduces its inference
computation cost. On the GLUE benchmark, our tiny-attention adapter
outperforms the other parameter-efficient transfer learning methods as well as
full fine-tuning while only updating 0.05% of the parameters. On the FewGLUE
benchmark, its performance is comparable to that of GPT-3 and PET.

##  1 Introduction

Transferring a large pretrained language model (PLM) is a de facto paradigm of
performing downstream tasks in natural language processing (NLP). A general
approach is adapter-tuning, which means inserting _adapters_ —i.e., neural
networks with small numbers of parameters—into each pretrained layer and only
updating the adapter parameters. Adapter-tuning is parameter-efficient and
enjoys low computation cost since it keeps the PLM frozen. But it
underperforms full fine-tuning which updates all the parameters of the PLM.

Figure 1: Average performance of different parameter-efficient transfer
learning methods on the GLUE benchmark with roberta-large as the PLM. Our
method is Tiny-Attn (with  marker): “1H” and “4H” mean “one attention head”
and “four attention heads (with the parameter-averaging trick in section 2.2)”
respectively.

(a) Transferring a pretrained language model for downstream tasks.

(b) Adapting a language model layer by adding our tiny-attention adapter.

(c) The internal architecture of our tiny-attention adapter.

Figure 2: The pipeline of applying our tiny-attention adapter to a pretrained
language model for downstream tasks.

In this paper, we propose a new adapter architecture which outperforms full
fine-tuning yet uses even fewer parameters than the previously proposed
adapters as well as other parameter-efficient transfer learning methods; see
Figure 1.

Our adapter is a multi-head attention module and its per-head dimensionality
is extremely small; thus we call it _tiny-attention_. The architecture design
is inspired by the following intuitions:

  1. 1.

For each input sequence, each layer of the language model produces an
embedding for each token in the sequence; see Figure 2(a).

  2. 2.

All the parameter-efficient transfer learning methods learn to modify the
embeddings towards the direction of performing the given task well; see He et
al. (2021) for a thorough theoretical and empirical discussion.

  3. 3.

Almost all the previously proposed adapter architectures are feed-forward
neural networks. Thus, we suspect that their embedding modifications are not
as contextually rich as they should.

Therefore, we propose to use the attentative structure that allows the
embedding modifications of each token to capture more contextual information
by directly looking at the embeddings of all the tokens. The dimensionality of
each attention head need not to be large. In NLP tasks, contextual information
is often demonstrated more important than model sizes: for example, in
language modeling, a smaller model with a larger context window usually
outperforms a larger model with a smaller context window (Dai et al., 2019; Wu
et al., 2022; Yang et al., 2019). Additionally, we view the multiple attention
heads of our tiny-attention adapter as a mixture of experts and then propose
to average their weights during inference. This technique further reduces the
inference cost.

We evaluated our tiny-attention adapter on the GLUE (Wang et al., 2018) and
FewGLUE (Schick and Schütze, 2021; Wang et al., 2019) benchmarks. On GLUE, it
updates only 0.05% of the parameters—an order of magnitude smaller than all
the other methods—yet still outperforms full fine-tuning and nearly all the
other parameter-efficient tuning methods. On FewGLUE, it is comparable to
several strong competing methods including PET (Schick and Schütze, 2021) and
GPT-3 (Brown et al., 2020). We also conducted ablation studies to investigate
its effectiveness with varying placements and PLMs.

##  2 The Method

Figure 2(a) illustrates how a language model performs a downstream task. The
language model has $L$ layers. Given an input sequence
${\bm{\mathbf{x}}}=x_{0}x_{1}\ldots x_{T}$ where $x_{0}$ is a special
classification (CLS) token, each layer $\ell$ reads the embeddings given by
the previous layer $\ell-1$ and produces the layer-$\ell$ embeddings
${\bm{\mathbf{h}}}_{0}^{(\ell)}{\bm{\mathbf{h}}}_{1}^{(\ell)}\ldots{\bm{\mathbf{h}}}_{T}^{(\ell)}$.
Then a task-specific decoderreads the top-layer embedding
${\bm{\mathbf{h}}}_{0}^{(L)}$ of the CLS token and predicts a task-specific
output $\hat{y}$ for that sequence.

Transferring the language model involves updating its _trainable_ parameters
to minimize a task-specific $\text{loss}(\hat{y},y)$ where $y$ is the ground-
truth label for the given ${\bm{\mathbf{x}}}$. For adapter-tuning, the
trainable parameters only include those of the task-specific decoder and those
of the adapters. Figure 2(b) shows a language model layer with our tiny-
attention adapter placed between its attention module and feed-forward net:
during training, only the decoder and adapter parameters (blue) are updated
while the pretrained parameters (green) are all kept frozen.

###  2.1 Tiny-Attention Adapter

Our adapter has an attentative structure: as shown in Figure 2(b), at each
position $t$, it takes as input the intermediate embeddings
${\bm{\mathbf{z}}}^{(\ell)}$ from not only the current position (information
flow indicated by blue arrows ) but also all the other positions (information
flow shown by red arrows ). For each token $t$, it produces a task-specific
modification $\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$. Then the modified
embeddings
${{\bm{\mathbf{z}}}}^{(\ell)}_{t}+\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$ are
fed into the layer $\ell$ feed-forward net for producing
${\bm{\mathbf{h}}}_{t}^{(\ell)}$.

As shown in Figure 2(c), the internal architecture of our tiny-attention
adapter resembles an ordinary multi-head attention mechanism. Suppose it has
$M$ attention heads. Each attention head $m$ produces a head-specific
attention vector $\tilde{{\bm{\mathbf{z}}}}^{(\ell,m)}_{t}$ and the final
attention vector $\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$ is obtained by
projecting the concatenation of all the
$\tilde{{\bm{\mathbf{z}}}}^{(\ell,m)}_{t}$:

$\displaystyle\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$
$\displaystyle\mathrel{\stackrel{{\scriptstyle\textnormal{def}}}{{=}}}{\bm{\mathbf{O}}}^{(\ell)}[\tilde{{\bm{\mathbf{z}}}}^{(\ell,1)}_{t};\ldots;\tilde{{\bm{\mathbf{z}}}}^{(\ell,M)}_{t}]$
(1a) $\displaystyle\tilde{{\bm{\mathbf{z}}}}^{(\ell,m)}_{t}$
$\displaystyle\mathrel{\stackrel{{\scriptstyle\textnormal{def}}}{{=}}}\text{Attn}_{t}^{(m)}({{\bm{\mathbf{z}}}}^{(\ell)}_{0},{{\bm{\mathbf{z}}}}^{(\ell)}_{1},\ldots,{{\bm{\mathbf{z}}}}^{(\ell)}_{T})$
(1b)

The math details of Attn${}_{t}^{(m)}$ is given in Appendix A.

#### Why _attention_ as adapter?

Attention allows the task-specific modification
$\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$ for each token $t$ to aggregate
useful information from its full contexts at $t=0,1,\ldots,T$. It is analogous
to how the attention modules in the pretrained language model learned to
construct contextual representations which helped optimize the language
modeling objective during pretraining. Therefore, when the pretrained
attention modules are frozen, it seems natural to adopt new trainable
attention modules for their desired behavior.

The key difference between our tiny-attention and an ordinary attention is
that our per-head dimension is _tiny_ : i.e.,
$\tilde{{\bm{\mathbf{z}}}}_{t}^{(\ell,m)}\in\mathbb{R}^{D}$ and $D$ is very
small. Throughout our experiments, we set $D=1$.

#### Why _tiny_ -attention?

Smaller dimensionality means fewer trainable parameters and less computation
cost; thus it is preferred. We believe that small dimensionality is sufficient
in our setting because of two key observations. First, a smaller model often
tends to work as well as a larger model if it is allowed to use a larger
context; see section 1 for the example of language modeling (Dai et al., 2019;
Wu et al., 2022; Yang et al., 2019). Second, Hu et al. (2021) found that feed-
forward adapters can achieve competitive results with extremely low-rank (1 or
2) parameter matrices. Similar to the LoRA method by Hu et al. (2021), our
tiny-attention adapter essentially performs a low-rank non-linear projection:
it first linearly transforms the high-dimensional embeddings
${\bm{\mathbf{z}}}^{(\ell)}$ to low-dimensional query, key, and value vectors;
it then linearly transforms the low-dimensional attention vectors—after the
non-linear attention operation—to the high-dimensional modification vectors
$\tilde{{\bm{\mathbf{z}}}}^{(\ell)}$; see Appendix A for technical details.

###  2.2 Multiple Heads as a Mixture of Experts

The multiple attention heads in our tiny-attention can be regarded as a
mixture of experts where each head is an expert that specializes in capturing
certain kinds of contextual information (e.g., syntax, semantics). The output
projection in equation 1a learns to aggregate the information
$\tilde{{\bm{\mathbf{z}}}}_{t}^{(\ell,m)}$ produced by the experts.
Rearranging that equation gives

$\displaystyle\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$
$\displaystyle\mathrel{\stackrel{{\scriptstyle\textnormal{def}}}{{=}}}\sum_{m=1}^{M}{\bm{\mathbf{O}}}^{(\ell,m)}\tilde{{\bm{\mathbf{z}}}}^{(\ell,m)}_{t}$
(2)

where the per-head matrices ${\bm{\mathbf{O}}}^{(\ell,m)}$ are defined such
that
${\bm{\mathbf{O}}}^{(\ell)}=[{\bm{\mathbf{O}}}^{(\ell,1)};\ldots;{\bm{\mathbf{O}}}^{(\ell,M)}]$.

It inspires us to propose a parameter-averaging trick that is able to further
reduce the storage and computation cost of our method. Precisely, _after_
training, we average the output projection matrices
${\bm{\mathbf{O}}}^{(\ell,m)}$ as well as the attention parameters inside
$\text{Attn}^{(m)}$ across the attention heads. Then we only store the
averaged parameters. During inference, we only use a single attention head
into which the stored parameters have been loaded. That way, although we may
have trained $M>1$ attention heads, our storage and inference cost will be as
low as if we had only trained a single head. The technical details of this
trick is discussed in Appendix A.

##  3 Related Work

There are three major paradigms of parameter-efficient transfer learning. The
first is to only fine-tune a small subset of the existing parameters in a
pretrained language model (Howard and Ruder, 2018; Lee et al., 2019; Zaken et
al., 2021). The second is adapter-tuning (Houlsby et al., 2019): inserting
adapters (i.e., small neural nets) into the language model and only tuning
their parameters. The third is prefix-tuning (Li and Liang, 2021;
Hambardzumyan et al., 2021; Liu et al., 2021b, a; Lester et al., 2021):
augmenting the input sequence with trainable tokens and only updating the new
token embeddings. Both adapter-tuning and prefix-tuning keeps the PLM frozen.
Our work falls into the category of adapter-tuning. The key difference is that
our proposed adapter has an attention architecture. The previously proposed
methods in this direction all use feed-forward neural networks: they are
Houlsby et al. (2019); Lin et al. (2020); Pfeiffer et al. (2021); Hu et al.
(2021); He et al. (2021).

AdaMix (Wang et al., 2022) proposes a stochastic routing strategy to mix an
ensemble of adapters and it is orthogonal to all the adapter methods including
ours. Akin to our parameter-averaging trick, they propose to average the
adapter parameters for low-cost storage and inference. Similar tricks are used
in Ravi (2017); Matena and Raffel (2021); Wortsman et al. (2022).

##  4 Experiments

We evaluated our proposed tiny-attention adapter on a range of natural
language understanding tasks including GLUE and FewGLUE. Our method is
implemented in PyTorch (Paszke et al., 2019) and heavily relies on HuggingFace
(Wolf et al., 2020). Our code is submitted for review and it will be publicly
released after the paper is published.

In all of our experiments, we set the dimensionality of each tiny-attention
head (i.e., the dimension of query, key, and value vectors) to be _one_. Other
experiment details (e.g., hyperparameters) can be found in Appendix B.

###  4.1 Main Results: GLUE and FewGLUE

#### GLUE.

On the GLUE benchmark, we chose the RoBERTa model (Liu et al., 2019) as our
PLM and we used the pretrained roberta-large weights (355M parameters)
downloaded from HuggingFace.

Our results on the GLUE benchmark are already presented in Figure 1. As we can
see, our method (Tiny-Attn-1H and Tiny-Attn-4H) outperforms all the previously
proposed parameter-efficient tuning methods as well as fine-tuning. Yet our
method uses significantly fewer trainable parameters than the other methods
except WARP. The single-head version (Tiny-Attn-1H) trains 176K parameters,
which only counts as 0.05% of the PLM parameters. The four-head version (Tiny-
Attn-4H) further improves the performance with an increased training cost, but
its storage and inference cost remains the same as the single-head version.

Our method only underperforms AdaMix, which learns a stochastic routing
strategy to mix an ensemble of adapters. But AdaMix uses significantly more
trainable parameters than ours. Moreover, AdaMix’s technique is orthogonal and
complementary to most other adapter methods including ours.

Our results on each individual GLUE task can be found in Tables 8 and 9 of
section C.1.

#### FewGLUE.

We also evaluated our method on the CB and RTE tasks of the FewGLUE benchmark.
They are extremely few-shot settings: each task only has 32 training examples.
We chose ALBERT (Lan et al., 2019) as our PLM and we used the pretrained
albert-xxlarge-v2 weights (223M parameters) downloaded from HuggingFace. The
detailed setting can be found in section B.1. The result is shown in Table 1.
It turns out that the performance of our method is comparable to that of PET
(Schick and Schütze, 2021) and GPT-3 (Brown et al., 2020).

method | CB | RTE
---|---|---
Tiny-Attn-1H (ours) | 88.57$\pm$2.99 | 68.38$\pm$2.53
WARP | 87.5 | 71.8
PET | 85.1 | 69.8
iPET | 92.9 | 74
GPT-3-Small | 42.9 | 52.3
GPT-3-Medium | 58.9 | 48.4
GPT-3 | 82.1 | 72.9

Table 1: Results on the validation set of FewGLUE. We report accuracy for all
tasks.

###  4.2 Analysis

#### Sequential vs. parallel.

In section 2, we presented the ‘sequential’ methods where our tiny-attention
modules are placed between the pretrained attention and feed-forward net.
Another option is to put the tiny-attention module in ‘parallel’ to the
original attention layer as in He et al. (2021), as illustrated in Figure 3.

Figure 3:  Illustration of the parallel structure.

We emperically found that these two design choices have negligible differences
in results. Detailed Results are listed in Table 10 of section C.2.

#### Effects of parameter-averaging.

Recall that we used the parameter-averaging trick in our experiments for an
improved inference efficiency. But do we lose any performance by using this
trick? Through ablation studies on CoLA and RTE, we found that using our
parameter-averaging trick actually slightly improves the results. Detailed
results are shown in Table 11 of section C.1.

#### Does the size of PLM matter?

Parameter-efficient tuning methods are known to suffer performance drop when
working with small-sized PLMs. To investigate this effect, we also
experimented with the pretrained roberta-base (125M parameters) downloaded
from Huggingface on the MNLI and SST-2 tasks. The results are shown in Figure
4. Different methods suffer almost the same amount of performance drop.111LoRA
and AdapterH are the only previous methods that reported GLUE results for both
roberta-large and roberta-base.

| performance drop
---|---
method | MNLI | SST-2 | average
LoRA | 3.4 | 2 | 2.7
AdapterH | 3.1 | 2.1 | 2.6
Tiny-Attn-1H (ours) | 3.2 | 2.1 | 2.65
Table 2: Performance drop of different methods on MNLI and SST-2, with
roberta-large and roberta-base as the PLMs.

But our method enjoys a much larger drop in the number of trainable
parameters: the trainable parameters of our method (Tiny-Attn-1H) still only
count as 0.05% of the PLM parameters, but those of LoRA and AdapterH increase
from 0.23% to 0.5%. See Table 2.

(a)  SST-2

(b)  MNLI

Figure 4:  Performance of different parameter-efficient tuning methods on
SST-2 and MNLI with RoBERTa-base. The baseline results are from He et al.
(2021).

#### Effects of larger dimensions.

We set $D=1$ in all our main experiments. It’s natural to ask that what if we
use a larger dimension. We experimented with a $D=4$ variant of our method,
which we call Tiny-Attn-1H4D, on CoLA and RTE. We found that this variant
performs slightly worse than the standard version Tiny-Attn-1H. It’s normal
that more parameters in the adapters could lead to slightly worse performance
on some GLUE tasks, see Hu et al. (2021). Detailed results are shown in Table
12 of section C.1.

##  5 Conclusion

In this paper, we presented the tiny-attention adapter. While previous
adapter-tuning only processes in-place embeddings, our method considers the
context from other positions. Thanks to this contextual modeling, the size of
our tiny-attention adapter can be extremely light (e.g., only 1 attention head
and 1 head dimension). To further enable trade-off between performance and
training cost, we proposed the weight-averaging technique. On GLUE benchmark,
our tiny-attention adapter achieved better results than full fine-tuning while
only updating 0.05% of parameters. Our model also achieved competitive results
under the few-shot setting. Lastly, we compared our methods with the
alternatives (e.g., parallel instead of sequential, without parameter
averaging) and also showed the generalization to smaller PLMs.

## Acknowledgements

This work was supported by a research gift to the last author by Adobe
Research. We thank the anonymous EMNLP reviewers and meta-reviewer for their
constructive feedback. We also thank our colleagues at Toyota Technological
Institute at Chicago for helpful discussion.

## Limitations

Our main limitation is that our method was only evaluated on the GLUE and
FewGLUE benchmarks and that we haven’t experimented with a diverse set of
generation tasks (e.g., XSUM (Narayan et al., 2018), E2E (Novikova et al.,
2017a)) yet. He et al. (2021) reported that the state-of-the-art parameter-
efficient adaptation method on a task or dataset may suffer a sharp
performance drop on another task or dataset. Although our method is
consistently effective across multiple classification tasks, it is still
possible that it won’t perform well at generation tasks such as summarization
and translation.

## Ethics Statement

Our method belongs to the general category of efficient language model fine-
tuning and our focus is to reduce the number of trainable parameters. It can
benefit the scenarios when communications become a bottleneck, such as federal
learning, distributed training, and edge computing. However, since we do not
apply explicit differentiable privacy methods to these updated parameters, the
method can be vulnerable to specific attacks (e.g., man-in-the-middle attack).

Our method can also be deployed to on-device applications, where the storage
is limited. For these applications, different tasks can just keep a small set
of task-specific parameters while other parameters are shared.

Our method can help reduce the carbon footprint of the model training in two
ways. Firstly, the model is fine-tuned from a pretrained model thus can be
adapted to a downstream task quickly while reaching a satisfying performance.
Secondly, our method only updates a small portion of all parameters thus the
optimizer only tracks a small part of parameters. For this reason, the same
training infrastructure can support larger batch size given the optimizer’s
states are significantly reduced (e.g., to less than 1% in our Tiny-Attn-1H
method).

Meanwhile, our method shares the same possibilities as most of previous
efficient training methods, such as misusage, containing data bias, and
suffering from adversarial attacks. However, the method developed in this
paper is orthogonal to the previous effort to mitigate the above issues.

## References

  * Bar Haim et al. (2006) Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second PASCAL recognising textual entailment challenge.
  * Bentivogli et al. (2009) Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009.  The fifth PASCAL recognizing textual entailment challenge.  In _TAC_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017\.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  In _Machine learning challenges workshop_.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_.
  * Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the International Workshop on Paraphrasing_.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_.
  * Hambardzumyan et al. (2021) Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021.  WARP: Word-level Adversarial ReProgramming.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
  * He et al. (2021) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021.  Towards a unified view of parameter-efficient transfer learning.  _arXiv preprint arXiv:2110.04366_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  _arXiv preprint arXiv:1801.06146_.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Lee et al. (2019) Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019.  What would elsa do? freezing layers during transformer fine-tuning.  _arXiv preprint arXiv:1911.03090_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Levesque et al. (2011) Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011.  The Winograd schema challenge.  In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
  * Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020.  Exploring versatile generative language model via parameter-efficient transfer learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Liu et al. (2021a) Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021a.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  _arXiv preprint arXiv:2110.07602_.
  * Liu et al. (2021b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b.  Gpt understands, too.  _arXiv preprint arXiv:2103.10385_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Matena and Raffel (2021) Michael Matena and Colin Raffel. 2021.  Merging models with fisher-weighted averaging.  _arXiv preprint arXiv:2111.09832_.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  _arXiv preprint arXiv:1808.08745_.
  * Neyshabur et al. (2020) Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2020.  What is being transferred in transfer learning? _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Novikova et al. (2017a) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017a.  The e2e dataset: New challenges for end-to-end generation.  _arXiv preprint arXiv:1706.09254_.
  * Novikova et al. (2017b) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017b.  The e2e dataset: New challenges for end-to-end generation.  _arXiv preprint arXiv:1706.09254_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems (NeurIPS)_. Curran Associates, Inc.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021.  AdapterFusion: Non-destructive task composition for transfer learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Ravi (2017) Sujith Ravi. 2017.  Projectionnet: Learning efficient on-device deep networks using neural projections.  _arXiv preprint arXiv:1708.00630_.
  * Rücklé et al. (2021) Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021.  AdapterDrop: On the efficiency of adapters in transformers.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint 1905.00537_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Wang et al. (2022) Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022.  Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models.  _arXiv preprint arXiv:2205.12410_.
  * Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018.  Neural network acceptability judgments.  _arXiv preprint 1805.12471_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of NAACL-HLT_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  _arXiv preprint arXiv:2203.05482_.
  * Wu et al. (2022) Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. 2022.  Memorizing transformers.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Zaken et al. (2021) Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  _arXiv preprint arXiv:2106.10199_.

##  Appendix A Method Details

The notations we use in this section are inherited from section 2.

Recall that every head-specific attention vector
${\bm{\mathbf{\tilde{z}}}}_{t}^{(\ell,m)}$ is given by equation 1b. If we
expand $\text{Attn}_{t}^{(m)}$ in this equation, we get

$\displaystyle\begin{split}&\quad\
\text{Attn}_{t}^{(m)}({{\bm{\mathbf{z}}}}^{(\ell)}_{0},{{\bm{\mathbf{z}}}}^{(\ell)}_{1},\ldots,{{\bm{\mathbf{z}}}}^{(\ell)}_{T})\\\
&=\tfrac{\sum_{s=0}^{T}\exp\left(\tfrac{{\bm{\mathbf{q}}}_{t}^{(\ell,m)\top}{\bm{\mathbf{k}}}_{s}^{(\ell,m)}}{\sqrt{D}}\right){\bm{\mathbf{v}}}_{s}^{(\ell,m)}}{\sum_{s=0}^{T}\exp\left(\tfrac{{\bm{\mathbf{q}}}_{t}^{(\ell,m)\top}{\bm{\mathbf{k}}}_{s}^{(\ell,m)}}{\sqrt{D}}\right)}\end{split}$
(3)

The subscript $t$ in $\text{Attn}_{t}^{(m)}$ means that the $t$th intermediate
embedding ${{\bm{\mathbf{z}}}}^{(\ell)}_{t}$ is used to construct the query
${\bm{\mathbf{q}}}_{t}^{(\ell,m)}$. Similarly, for $s=0,1,\ldots,T$,
${\bm{\mathbf{k}}}_{s}^{(\ell,m)},{\bm{\mathbf{v}}}_{s}^{(\ell,m)}$ are low-
dimensional key and value vectors that are dependent on the high-dimensional
intermediate embedding ${{\bm{\mathbf{z}}}}^{(\ell)}_{s}$. These vectors are
computed by linear transformations:

$\displaystyle{\bm{\mathbf{q}}}_{t}^{(\ell,m)}$
$\displaystyle={\bm{\mathbf{W}}}_{Q}^{(\ell,m)}{\bm{\mathbf{z}}}_{t}^{(\ell)}$
(4a) $\displaystyle{\bm{\mathbf{k}}}_{s}^{(\ell,m)}$
$\displaystyle={\bm{\mathbf{W}}}_{K}^{(\ell,m)}{\bm{\mathbf{z}}}_{s}^{(\ell)}$
(4b) $\displaystyle{\bm{\mathbf{v}}}_{s}^{(\ell,m)}$
$\displaystyle={\bm{\mathbf{W}}}_{V}^{(\ell,m)}{\bm{\mathbf{z}}}_{s}^{(\ell)}$
(4c)

where
${\bm{\mathbf{W}}}_{Q}^{(\ell,m)},{\bm{\mathbf{W}}}_{K}^{(\ell,m)},{\bm{\mathbf{W}}}_{V}^{(\ell,m)}$
are $H\times 1$ parameter matrices of the attention module, where $H$ is the
hidden size of the PLM.

#### Mixture of experts.

During inference, we average all the parameters of the attention module across
the attention heads.

$\displaystyle{\bm{\mathbf{W}}}_{Q}^{(\ell,n)}$
$\displaystyle\leftarrow\tfrac{1}{M}\sum_{m=1}^{M}{\bm{\mathbf{W}}}_{Q}^{(\ell,m)}$
(5a) $\displaystyle{\bm{\mathbf{W}}}_{K}^{(\ell,n)}$
$\displaystyle\leftarrow\tfrac{1}{M}\sum_{m=1}^{M}{\bm{\mathbf{W}}}_{K}^{(\ell,m)}$
(5b) $\displaystyle{\bm{\mathbf{W}}}_{V}^{(\ell,n)}$
$\displaystyle\leftarrow\tfrac{1}{M}\sum_{m=1}^{M}{\bm{\mathbf{W}}}_{V}^{(\ell,m)}$
(5c)

where $n=1,2,\ldots,M$.

Then the attention function $\text{Attn}_{t}^{(m)}$ will be the same for all
$m$. In other words, we will have
$\tilde{{\bm{\mathbf{z}}}}_{t}^{(\ell,1)}=\ldots=\tilde{{\bm{\mathbf{z}}}}_{t}^{(\ell,M)}\mathrel{\stackrel{{\scriptstyle\textnormal{def}}}{{=}}}\tilde{{\bm{\mathbf{z}}}}_{t}^{(\ell)}$.

Substitute this to equation 2, we get

$\displaystyle\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$
$\displaystyle=(\sum_{m=1}^{M}{\bm{\mathbf{O}}}^{(\ell,m)})\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$
(6)

where ${\bm{\mathbf{O}}}^{(\ell,m)}\in\mathbb{R}^{1\times H}$ are the output
projection matrices. We can define

$\displaystyle{\bm{\mathbf{\bar{O}}}}^{(\ell)}\mathrel{\stackrel{{\scriptstyle\textnormal{def}}}{{=}}}\tfrac{1}{M}\sum_{m=1}^{M}{\bm{\mathbf{O}}}^{(\ell,m)}$
(7)

Then we only use the averaged projection matrix such that equation 2 becomes

$\displaystyle{\bm{\mathbf{z}}}^{(\ell)}_{t}$
$\displaystyle=M{\bm{\mathbf{\bar{O}}}}^{(\ell)}\tilde{{\bm{\mathbf{z}}}}^{(\ell)}_{t}$
(8)

##  Appendix B Experiment Details

We used the PyTorch library (Paszke et al., 2019) and the pretrained language
models from the HuggingFace transformers library (Wolf et al., 2020) in all of
our experiments.

###  B.1 GLUE and FewGLUE

#### GLUE.

We evaluated our method on 8 tasks of the GLUE benchmark (Wang et al., 2018):
SST-2 (Socher et al., 2013), CoLA (Warstadt et al., 2018), MNLI (Williams et
al., 2018), QQP222https://quoradata.quora.com/First-Quora-Dataset-Release-
Question-Pairs, RTE (Dagan et al., 2005; Bar Haim et al., 2006; Giampiccolo et
al., 2007; Bentivogli et al., 2009), MRPC (Dolan and Brockett, 2005), QNLI
(Rajpurkar et al., 2016) and STS-B (Cer et al., 2017).333The dataset can be
downloaded at https://huggingface.co/datasets/glue. They are all
classification tasks except STS-B, which is formed as a regression task. WNLI
(Levesque et al., 2011) is excluded following prior work (Houlsby et al.,
2019; Hu et al., 2021).

We used the official data splits. The data were pre-processed by GLUE and
HuggingFace and we did not apply any extra modification. The sizes of the
training sets of each task are shown in Table 3. All of the tasks have a
balanced label distribution in the training set except for MRPC (68% positive)
and QQP (63% negative). Besides, the test set of QQP has a different label
distribution than the training set.

task | |train|
---|---
SST-2 | 67k
CoLA | 8.5k
MNLI | 393k
QQP | 364k
RTE | 2.5k
MRPC | 3.7k
QNLI | 105k
STS-B | 7k
Table 3: Sizes of training sets of tasks on GLUE.

We trained using AdamW (Loshchilov and Hutter, 2017) and either a linear or a
cosine learning rate decay scheduler. We used a linear warmup with
approximately 10% of the total steps as the warmup steps in some tasks. The
number of epochs was fixed to 20. We evaluated on the validation set twice per
epoch and report the best result. We used the same batch size 16 for all our
tasks. For reproducibility, we used a fixed random seed. Learning rate (lr)
was selected from [1E-6,1E-5,1E-4,5E-4,8E-4,1E-3,1.5E-3,2E-3,3E-3,5E-3,7E-3]
and weight decay (dec) was chosen from
[0,0.01,0.02,0.03,0.04,0.05,0.1,0.2,0.3,0.4,0.5]. Note that we did not do a
full hyperparameter search. On average, we run about 80 experiments for each
task. However, the detailed numbers of experiments are varying according to
the size of the dataset to save computational cost. E.g., on MNLI and QQP, we
only perform about 20 experiments for each. We performed our experiments
mainly on 10 Nvidia RTX A4000. The average running time is about eight hours,
varying across tasks.

For Tiny-Attn-1H, we initialized the output projection matrices to be very
small ($\mathcal{U}(-\tfrac{0.01}{\sqrt{D}},\tfrac{0.01}{\sqrt{D}})$ where D
is the per-head dimension) to make our model behave like the original
pretrained model at early stages of training. Empirically, we found this trick
important in stabilizing the training. The hyperparameters of the best-
performing single-head models are shown in Table 4.

task | lr | dec | warmup | scheduler
---|---|---|---|---
SST-2 | 8E-4 | 0.01 | yes | cosine
CoLA | 5E-4 | 0.04 | no | linear
MNLI | 1E-3 | 0 | yes | linear
QQP | 5E-4 | 0.01 | no | linear
RTE | 2E-3 | 0.05 | no | linear
MRPC | 3E-3 | 0.3 | yes | cosine
QNLI | 8E-4 | 0.02 | no | linear
STS-B | 1.5E-3 | 0 | no | cosine
Table 4: Hyperparameters for Tiny-Attn-1H.

For Tiny-Attn-4H, we perturbed the weight of Tiny-Attn-1H to initialize every
head due to computational limits. In principle, we only need to initialize the
weight of every head to be almost the same. This makes the parameter-averaging
trick applicable since the weight of these heads would not be very far from
each other, just as when we average the parameters of multiple fine-tuned
models with the same initialization (Neyshabur et al., 2020). The
hyperparmeters we use are shown in Table 5.

task | lr | dec | warmup | scheduler
---|---|---|---|---
CoLA | 8E-4 | 0.05 | yes | cosine
RTE | 1E-5 | 0.3 | no | linear
MRPC | 3E-3 | 0.3 | no | linear
QNLI | 1E-6 | 0.4 | no | cosine
Table 5: Hyperparameters for Tiny-Attn-4H. Since we initialize from the weight
of Tiny-Attn-1H, tasks on which we don’t get an improvement are omitted.

#### FewGLUE.

FewGLUE (Schick and Schütze, 2021) is a subset of the SuperGLUE benchmark
(Wang et al., 2019) with the sizes of all training sets being 32.444The
dataset can be downloaded at https://github.com/timoschick/fewglue. We
evaluated our model on two tasks of it: CB (De Marneffe et al., 2019) and RTE.
We still used AdamW but we did not use a scheduler. We used a linear warmup
with 10% of the total steps as the warmup steps for RTE. The number of epochs
was fixed to 20. Following Schick and Schütze (2021), we did not use any
additional validation set for parameter selection or early stopping and we
only report the final result. In contrast to them, we did not use any
additional unlabeled examples. We used a fixed learning rate 1E-3 and a fixed
batch size 1. We’ve tried a few different weight decay [0,0.02,0.05,0.1], but
eventually we chose the same value 0.02 for all tasks. After selecting the
hyperparameters with a fixed random seed, we ran over 4 additional random
seeds and report the average performance and the standard variation.

Following Schick and Schütze (2021), we used albert-xxlarge-v2 (Lan et al.,
2019) downloaded from the HuggingFace library as the backbone PLM, and used
manual prompts to replace the classification head. The manual prompt we use
was “[CLS]<hypothesis>?[MASK].<premise><SEP>”, the same as theirs. The
language model predicts the token at the masked position and the output space
is limited to a few selected tokens. We used “yes” to represent “entailment”,
“no” to represent “contradiction”, and “maybe” to represent “neutral”.

###  B.2 Analysis

#### Sequential vs. parallel.

In the main experiments, we used a ‘sequential’ (seq) structure where our
tiny-attention modules are placed between the pretrained attention and feed-
forward net. Another option is to put the tiny-attention module in ‘parallel’
(para) to the original attention layer as in He et al. (2021).

We used the same setting as our main experiments on GLUE except that we used
roberta-base as the backbone PLM. Warmup was used for all experiments. The
hyperparameters we used are shown in Table 6.

task | type | lr | dec | scheduler
---|---|---|---|---
SST-2 | seq | 1E-3 | 0.02 | cosine
SST-2 | para | 1E-3 | 0.01 | cosine
MNLI | seq | 1E-4 | 0.01 | linear
MNLI | para | 1E-3 | 0.01 | linear
Table 6: Hyperparameters for experiments with roberta-base as the PLM.

Note that there are other possible placements, e.g., we can place tiny-
attention adapters above the feed-forward networks. We did not do this because
our philosophy is to augment the pretrained attention but not to intervene
anything else.

#### Effects of parameter-averaging.

We used the same setting as our Tiny-Attn-4H on GLUE except that we did not
use parameter-averaging during inference. Instead, we used all 4 heads as in
training. The hyperparameters we used are shown in Table 7.

task | lr | dec | warmup | scheduler
---|---|---|---|---
CoLA | 8E-4 | 0 | no | cosine
RTE | 7E-3 | 0.5 | yes | cosine
Table 7: Hyperparameters for Tiny-Attn-4H without parameter-averaging.

#### Does the size of PLM matter?

The hyperparameters we used are shown in Table 6.

##  Appendix C Results and Analysis Details

We use “AdapterH” and “AdapterP” to denote the Adapter proposed in Houlsby et
al. (2019) and Pfeiffer et al. (2021), respectively. “Tiny-Attn-kH” represents
our method with k attention heads, e.g., ‘Tiny-Attn-1H is our method with a
single attention head.

###  C.1 GLUE and FewGLUE.

Results in this section are discussed in section 4.1.

method | SST-2 | CoLA |  MNLI |  QQP | RTE |  MRPC | QNLI | STS-B |  average | # trainable para | percentage
---|---|---|---|---|---|---|---|---|---|---|---
metric | Acc | Mat | $\text{All}/\text{M}$ | $\text{Acc}/\text{F1}$ | Acc | $\text{Acc}/\text{F1}$ | Acc | Pearson |  |  |
AdaMix | 97.1 | 70.2 | $90.9/-$ | $92.3/89.8$ | 89.2 | $91.9/94.1$ | 95.4 | 92.4 | $89.9/-$ | 2M | 0.56%
AdapterH | 96.3 | 66.3 | $90.3/-$ | $91.5/-$ | 72.9 | $87.7/-$ | 94.7 | 91.5 | $86.4/-$ | 0.8M | 0.23%
AdapterH | 96.2 | 66.5 | $89.9/-$ | $92.1/-$ | 83.4 | $88.7/-$ | 94.7 | 91.0 | $87.8/-$ | 6M | 1.7%
AdapterP | 96.6 | 67.8 | $90.5/-$ | $91.7/-$ | 80.1 | $89.7/-$ | 94.8 | 91.9 | $87.9/-$ | 0.8M | 0.23%
AdapterP | 96.1 | 68.3 | $90.2/-$ | $91.9/-$ | 83.8 | $90.2/-$ | 94.8 | 92.1 | $88.4/-$ | 3M | 0.85%
LoRA | 96.2 | 68.2 | $\textbf{90.6}/-$ | $91.6/-$ | 85.2 | $90.2/-$ | 94.8 | 92.6 | $88.6/-$ | 0.8M | 0.23%
WARP | 96.0 | 60.6 | $-/88.2$ | $-/84.5$ | 75.8 | $-/90.8$ | 93.5 | 88.6 | $-/84.75$ | 25K | 0.007%
fine-tuning | 96.4 | 68.0 | $90.2/90.2$ | $\textbf{92.2}/-$ | 86.6 | $90.9/-$ | 94.7 | 92.4 | $88.9/-$ | 355M | 100%
Tiny-Attn-1H (ours) | 96.6 | 68.8 | $89.3/89.6$ | $90.1/86.6$ | 88.8 | $92.4/94.4$ | 94.3 | 92.3 | $89.1/88.9$ | 176K | 0.05%
Tiny-Attn-4H (ours) | 96.6 | 69.4 | $89.3/89.6$ | $90.1/86.6$ | 89.2 | $\textbf{92.6}/\textbf{94.6}$ | 94.5 | 92.3 | $\textbf{89.3}/\textbf{89.1}$ | 0.7M | 0.2%

Table 8: Dev set results on GLUE tasks. All the runs use roberta-large as the
backbone. We report Matthew’s correlation for CoLA, the overall(matched and
mismatched) accuracy and matched accuracy for MNLI, Pearson correlation for
STS-B, accuracy and F1 score for QQP and MRPC, and accuracy for other tasks.
Higher is better for all metrics. All the runs follow the setting in Houlsby
et al. (2019) except fine-tuning. The results of WARP and AdaMix are taken
from their own paper respectively. Other results are taken from Hu et al.
(2021).

method | SST-2 | CoLA |  MNLI |  QQP | RTE |  MRPC | QNLI | STS-B | score | # trainable para | backbone model
---|---|---|---|---|---|---|---|---|---|---|---
metric | Acc | Mat | $\text{M}/\text{MM}$ | $\text{Acc}/\text{F1}$ | Acc | $\text{Acc}/\text{F1}$ | Acc |  Pearson/Spearmanr |  |  |
human baselines | 97.8 | 66.4 | $92.0/92.8$ | $59.5/80.4$ | 93.6 | $86.3/80.8$ | 91.2 | 92.7/92.6 | 87.1 | - | -
fine-tuning | 97.5 | 71.5 | $91.9/91.6$ | $76.2/90.8$ | 93.2 | $94.0/92.0$ | 99.2 | 92.9/92.6 | 90.8 | 1.5B | DeBERTa
fine-tuning | 96.7 | 67.8 | $90.8/90.2$ | $74.3/90.2$ | 88.2 | $92.3/89.8$ | 95.4 | 92.2/91.9 | 88.1 | 355M | roberta-large
fine-tuning | 94.9 | 60.5 | $86.7/85.9$ | $72.1/89.3$ | 70.1 | $89.3/85.4$ | 92.7 | 87.6/86.5 | 80.5 | 345M | bert-large
WARP | 96.3 | 53.9 | $88.0/88.2$ | $68.6/87.7$ | 84.3 | $88.2/83.9$ | 93.5 | 89.5/88.8 | 81.6 | 25K | roberta-large
Tiny-Attn-1H (ours) | 96.2 | 62.5 | $89.3/88.8$ | $71.8/89.1$ | 83.6 | $90.7/87.5$ | 94.4 | 91.1/90.3 | 83.5 | 176K | roberta-large

Table 9: Test set results on GLUE tasks. We use the same setting as on the dev
set, different from the fine-tuning baseline and WARP. The results other than
ours are published in Hambardzumyan et al. (2021). We don’t show the result of
WNLI but it’s considered in the final score.

#### GLUE.

Table 8 shows our results on GLUE development set with roberta-large as the
PLM. Note that WARP used a slightly different set of validation metrics from
other methods, but based on our results we can assume that this difference
does not make a significant difference in the average score, that’s why we
directly used its score in Figure 1. We count all the parameters updated in
training for AdaMix and our Tiny-Attn-4H to compare with other methods.

We report Matthew’s correlation for CoLA, the overall(matched and mismatched)
accuracy and matched accuracy for MNLI, Pearson correlation for STS-B,
accuracy and F1 score for QQP and MRPC, and accuracy for other tasks. 555The
implementation of these metrics can be found at
https://github.com/huggingface/datasets/blob/master/metrics/glue/glue.py. We
used matched accuracy for MNLI and accuracy for QQP and MRPC as the validation
metrics. For other tasks, we just used the metrics we report.

To make a fair comparison, we did not use a model pretrained on MNLI for MRPC,
RTE, and STS-B as the fine-tuning baseline did, following the setting in
Houlsby et al. (2019) and Hu et al. (2021). Under this restriction, our method
still outperforms the fine-tuning baseline on average.

There are some other parameter-efficient transfer learning methods evaluated
on GLUE, but they either use a different backbone, e.g. Karimi Mahabadi et al.
(2021); or they are orthogonal to our work, e.g. Rücklé et al. (2021), so we
do not include them in this table.

We obtained test set results on the official GLUE server. We only made one
submission to the server. Following WARP, we always predict the majority class
for WNLI. For other tasks, we directly used the weight with the highest score
on the validation sets and did not do any extra modification as the fine-
tuning baseline did. Specially, we did not use the model adapted to MNLI for
RTE, MRPC and SST-B as WARP and the fine-tuning baseline did. We found that
our method has a consistent performance on validation and test sets, comparing
to WARP. The results are shown in Table 9. Note that the score in the table is
the official GLUE score, so it is affected by the result of WNLI.

###  C.2 Analysis

The results in this section are discussed in section 4.2.

#### Sequential vs. parallel.

Table 10 shows our results on SST-2 and MNLI with different structures. We
found that these two design choices have negligible differences in results.

method | SST-2 | MNLI
---|---|---
sequential | 94.5 | 86.1
parallel | 94.4 | 86.1
Table 10: Performance of Tiny-Attn-1H with different structures on SST-2 and
MNLI. We report accuracy for SST-2 and matched accuracy for MNLI.

#### Effects of parameter-averaging.

We found that using our parameter-averaging trick actually slightly improved
the results, as shown in Table 11.

parameter-averaging | CoLA | RTE
---|---|---
no | 68.8 | 88.8
yes | 69.4 | 89.2
Table 11: Results of Tiny-Attn-4H with different parameter-averaging settings
on CoLA and RTE. We report Matthew’s correlation for CoLA and accuracy for
RTE.

#### Effects of larger dimensions

We found that Tiny-Attn-1H4D is slightly worse than the standard version Tiny-
Attn-1H, as shown in Table 12.

method | CoLA | RTE
---|---|---
Tiny-Attn-1H4D | 68.0 | 87.4
Tiny-Attn-1H | 68.8 | 88.8
Table 12: Results of Tiny-Attn-4H with different parameter-averaging settings
on CoLA and RTE. We report Matthew’s correlation for CoLA and accuracy for
RTE.

#### Stability test

We report the best result with a fixed random seed for our main experiments.
To test the stability of our method, we run experiments on SST-2 and MNLI
using the same hyperparameters with 5 different random seeds. The results are
shown in Table 13. We could see that the results on larger datasets like MNLI
are quite stable, while the results on SST-2 have a larger variance.

| SST-2 | MNLI
---|---|---
seed-1 | 96.6 | 89.6
seed-2 | 96.6 | 89.5
seed-3 | 95.8 | 89.6
seed-4 | 96.3 | 89.8
seed-5 | 95.2 | 89.6
avg | 96.1 | 89.6
stdev | 0.6 | 0.1
Table 13: Stability test. We report accuracy for SST-2 and matched accuracy
for MNLI. Seed-1 is the seed we used in the main experiments.

#### Generation results

There has been research showing that parameter-efficient transfer methods with
good classification performance may not work equally well for non-
classification tasks, and vice versa (He et al., 2021). We conducted
experiments on E2E NLG Challenge (Novikova et al., 2017b) and found that our
method also suffers from this problem. The results are shown in Table 14. Our
result is comparable to adapter but worse than prefix-tuning.

method | dev | test | # trainable para
---|---|---|---
Tiny-Attn | 71.4 | 63.8 | 0.1M
Adapter | 68.1 | 66.3 | 0.36M
Prefix-Tuning | 74.8 | 70.3 | 0.36M
Table 14: Results on E2E NLG Challenge benchmark with gpt2-medium as the PLM.
We report the BLEU score computed by the official evaluation script.


# Impact of Strategic Electric Vehicles Driving Behavior on the Grid

Benoît Sohet<EMAIL_ADDRESS>
LIA, Avignon Univ., France
EDF R&D, MIRE Dept,
Paris-Saclay, France Yezekael Hayel<EMAIL_ADDRESS>
LIA, Avignon Univ., France Olivier Beaude<EMAIL_ADDRESS>
EDF R&D, OSIRIS Dept,
Paris-Saclay, France Alban Jeandin<EMAIL_ADDRESS>
EDF R&D, MIRE Dept,
Paris-Saclay, France

###### Abstract

In the context of transport electrification, a model coupling Electric
Vehicles (EV) driving and charging decisions is considered. While a Traffic
Assignment Problem (TAP) is considered for the driving part, the charging
operation is done with an exact load flow on a simple distribution network.
This setting allows assessing precisely the coupled impact of driving and
charging decisions. In particular, to schedule the charging need coming from
the driving part, different charging strategies are defined and compared
according to a cost metric based on the load flow computation. In view of this
metric, the proposed non-load flow based strategies can perform similarly to a
load flow based one, and with significantly reduced computation time and data
need. Numerical simulations show how a transportation toll can influence the
charging results.

###### Index Terms:

Electric vehicles; Grid interaction

##  I Introduction

The transport sector is one of the biggest source of gas emissions, with about
a quarter of the global level according to the 2017 annual report of the
International Energy Agency [1]. Full Electric Vehicles (EV) are one of the
solutions to the low carbon mobility problematic. However, the increasing
number of EV implies a higher power load. Then, the behavior of EV drivers,
typically the choice of the path to reach their destination and therefore the
corresponding energy consumed, induces a demand in electricity from the grid
which can be non negligible. Therefore, there is a natural relationship
between driving and charging at the user level and globally, between
transportation and grid networks.

The interaction (“coupling”) of EV driving and charging decisions is a recent
topic of research, as identified in the review paper [2]. In [3], only EV are
considered, and their charging choices are represented in an “extended
transportation network”: each station is replaced by a set of virtual arcs,
each one corresponding to a specific charging (energy) amount. However, there
is no smart charging considerations: the resulting aggregated EV charging need
is not scheduled in time but instead charged at once. Moreover, load flow
equations considered for the electricity network do not take into account
power losses. In [4], Gasoline Vehicles (GV) are added to the problem of [3]
and more realistic load flow equations are used. Charging need is supposed to
be the same for all EV and charging unit prices depend on the total demand at
the corresponding charging station. Still, no algorithm is proposed to
schedule the charging operations. Other works mentioned in [2] offer similar
coupled models of EV driving and charging but with limited smart charging
algorithms. In [5] the vehicle’s destination offering a minimal cost is chosen
while in [6] a fleet operator chooses the proportion of its vehicles to charge
instead of taking customers. Finally, [7] proposes a planning model where both
the transportation and the electricity network are optimally sized, minimizing
the operators investment. The EV charging need comes from a Traffic Assignment
Problem model [8] and the electricity network is considered, the load flow
equations being linearized (DistFlow) [9].

Compared to this existing work, one of our previous papers [10] considers a
coupled transportation and charging setting, with both EV and GV, in which a
central operator schedules the load of an EV fleet following a water-filling
algorithm introduced in [11]. A typical application is a charging Park & Ride
hub which can be equipped with local electricity production like photovoltaic
panels. EV drivers can park there and charge the electricity consumed during
the morning while working at a nearby office. However this previous paper does
not study into details the impact on the grid itself of the charging need of
EV. In the present paper, the impact of EV charging on the grid is obtained
through an exact load flow solution. This load flow is the basis of a charging
algorithm, as well as a metric to compare this first algorithm with two
heuristic charging policies which are “more distributed”. The main
contributions of this paper are:

  * •

the design and numerical comparisons of three different charging scheduling
algorithms, depending on which operator manages the charging operation and
what information it has access to;

  * •

to model and interpret numerically an accurate impact of a transportation
incentive (traffic toll) on a medium voltage grid, via the EV coupling of the
two systems. This impact is measured at the head of a network, typically a
transformer dedicated to a set of Charging Stations (EVCS) with a particular
grid topology.

The rest of the paper is organized as follows. Section II describes the global
coupled model incorporating the congestion game and the charging problem. The
algorithmic approaches which depend on the managing operator are fully
described in section III. Performances of those approaches are compared and
studied using real data sets in section IV. Finally, main conclusions and
perspectives are given in the last section V.

##  II Model

Figure 1: Transportation (in blue) and electrical (framed) networks considered
in this work. ENO, FO and CSO stand for respectively Electrical Network,
Flexibility and Charging Service Operators.

The use case considered in this work to accurately study the impact of the
transportation system on the electrical grid is commuting: drivers choose one
of the three paths of the transportation network of Fig. 1 to drive from home
to work. Users choosing path $i$ park their vehicle at the corresponding EVCS
$i$, from which they can reach the common destination by foot or public
transport (which is the purpose of Park & Ride hubs). These three EVCS belong
to the electrical grid of Fig. 1 and can be up to a dozen kilometers apart.
Then, EV are charged during working hours in a smart manner, chosen by
operators willing to minimize their costs. This toy example helps to
illustrate the importance of a coupled problem of driving and charging,
particularly considering the impact on the grid. A more general transportation
network and/or electrical grid topology can be considered without changing the
main messages and results of our study.

###  II-A Congestion game

The goal of this section is to predict the impact of transportation parameters
on the EV charging need at each EVCS. These charging needs depend on how many
EV are at each EVCS, so that the path choice made by drivers needs to be
modeled. In this work, this is done via a congestion game: each driver has
complete information about the possible costs and is fully rational in the
sense that it chooses the path with minimal total cost. This cost depends on
the choices of other drivers, in particular on the number of vehicles on each
path, through network congestion effects. This is the typical context of
congestion game problems [8]. Two classes of vehicle are considered: EV
(associated with subscript e) and GV (g), which have different costs because
of the different energy type. This model works with a larger number of
classes, such as different types of EV depending on their initial State of
Charge (SoC), but here all vehicles of the same class are assumed to share the
same characteristics which correspond to an average use case. In this
transportation network there are $N$ vehicles, with a proportion $X_{s}$ of
vehicles of class $s$ and a proportion $x_{s,i}$ of class $s$ vehicles on path
$i$. Then, for example, the number of EV on path $i$ is $x_{e,i}X_{e}N$.
Vehicles of class $s$ on path $i$ experience three kinds of costs:

  * •

Travel duration, given by the following congestion function (from the Bureau
of Public Roads [12]):

$d_{i}(x_{i})=d^{0}_{i}\left[1+2\left(x_{i}/C_{i}\right)^{4}\right],$ (1)

where $d_{i}^{0}=l_{i}/v_{i}$ is the minimal travel duration (corresponding to
free flow at speed limit $v_{i}$), with $l_{i}$ the length of path $i$ and
$C_{i}$ its capacity. Note that travel duration is the same for both vehicle
classes and depends on the total number of vehicles
$x_{i}=\left(x_{e,i}X_{e}+x_{g,i}X_{g}\right)N$ on path $i$. A stochastic
version of travel duration can be readily implemented in our model. This paper
considers the deterministic function (1), which can be seen as the average
travel duration.

  * •

Energy consumption $l_{i}m_{s}\lambda_{s}$, with $m_{s}$ the energy
(electricity or fuel resp. for $s=e,g$) consumption per distance unit and
$\lambda_{s}$ the energy unit price. Note that $m_{s}$ is assumed constant and
does not depend on speed profiles, thus the energy consumption only depends on
traveled distance.

  * •

Other constant costs, like traffic tolls $t_{s,i}$ imposed by a Transportation
Network Operator (TNO).

Summing all these costs, the total driving cost for a type $s$ vehicle on path
$i$ is given by the following expression:

$\displaystyle c_{s,i}(x_{i})=\tau
d_{i}(x_{i})+l_{i}m_{s}\lambda_{s}+t_{s,i}\,,$ (2)

where $\tau$ is the cost of one unit of time spent driving.

By all acting rationally, users will reach a particular distribution of
choices between the three paths, denoted by
$\mathbf{x}^{*}=\big{(}x_{s,i}^{*}\big{)}_{s,i}$ and called a Wardrop
Equilibrium (WE) [13]. This equilibrium situation gives a model of users
behavior in a stable regime where no user has an interest to change her choice
unilaterally. The proposed approach can thus be used to evaluate various
incentive mechanisms numerically – in a planning stage or tool – in order to
“select” a particular equilibrium before it will be observed in practice (note
that this concept is now commonly used in many operational public
transportation planning tools for the “route choice” step in four-steps
models), as done in Section IV-A.

The total charging need $L_{i}$ at each EVCS $i$ can then be computed from the
WE. First note that the underlying assumption of the energy consumption term
in (2) is that EV want to charge exactly the electricity amount they consumed
during their trip. Then, $L_{i}$ depends on the proportion $x_{e,i}^{*}$ of EV
parking at EVCS $i$ at WE and on what they consumed to get there,
$l_{i}m_{e}$:

$L_{i}=l_{i}m_{e}\times x_{e,i}^{*}X_{e}N\,.$ (3)

Note that even GV may impact the different charging needs at the EVCSs by
adding congestion on some roads which incites EV to favor other roads and thus
EVCS.

###  II-B Charging problem

Given the charging need $L_{i}$ at each EVCS $i$ resulting from the commuting
game (see (3)), the charging operation of EV during working hours at all EVCS
is scheduled by one or several operators (depending on the scenario
considered, as explained later in this section). More precisely, the charging
period (corresponding to working hours) is divided into several time slots.
Each EVCS $i$ has its own nonflexible consumption $\ell_{i,t}^{0}$ for each
time slot $t$. Then, for each EVCS $i$, an operator has to determine the
quantity $\ell_{i,t}$ to charge (flexible consumption) at each time slot $t$
in order to minimize its energy costs, and satisfying the total charging need
$L_{i}$ at this EVCS.

Several operators are part of the electrical system, as shown in Fig. 1. Each
EVCS $i$ is under the supervision of a Charging Service Operator (CSO).
Several CSO may be managed together by what is called a Flexibility Operator
(FO). The electrical grid, from the transformer to the EVCS, is managed by an
Electric Network Operator (ENO). Depending on which of these three operators
controls the charging operation scheduling, three scenarios are considered.
The algorithms solution of these scenarios are detailed in next section.

####  II-B1 Local (CSO)

The charging scheduling at each EVCS is done by the corresponding CSO. Each
CSO has no knowledge about nonflexible loads and charging profiles chosen by
the other CSO, and about the characteristics of the grid. Thus, CSO $i$
minimizes its own energy costs (reduced to its contract with the ENO),
expressed by a quadratic (extendable to other monomials) proxy [11] of its
total load and satisfying the charging needs:

$\min_{\ell_{i,t}}\sum_{t}\eta_{t}\left(\ell^{\text{tot}}_{i,t}\right)^{2}\quad\text{s.t.}\sum_{t}\ell_{i,t}=L_{i}\,,$
(4)

with $\eta_{t}$ the weight of time slot $t$, and
$\ell^{\text{tot}}_{i,t}=\ell^{0}_{i,t}+\ell_{i,t}$.

####  II-B2 Global (FO)

The charging scheduling of all EVCS is done by the FO. This aggregator has
complete information on all EVCS, but not on the grid. This way, the FO
minimizes the total cost of the EVCS:

$\min_{\ell_{i,t}}\sum_{t}\eta_{t}\left(\sum_{i}\ell^{\text{tot}}_{i,t}\right)^{2}\quad\text{s.t.}\sum_{t}\ell_{i,t}=L_{i}\,.$
(5)

Note that in the last two scenarios, the operators considered solve
optimization problems (4) and (5) regardless of the grid topology, unlike the
ENO in next scenario.

####  II-B3 Grid aware (ENO)

The charging scheduling of all EVCS is done by the ENO, which has complete
information on the EVCS and the grid. The ENO minimizes the impact of the
charging operation on the grid, which is expressed as a monomial of the
apparent power $S$ at the head of the grid needed to meet the electricity
demand $\ell^{\text{tot}}_{i,t}$ at each EVCS $i$:

$\min_{\ell_{i,t}}\left[\mathcal{G}=\sum_{t}\eta_{t}S^{2}\left(\ell^{\text{tot}}_{1,t}\,,~{}\ell^{\text{tot}}_{2,t}\,,~{}\ell^{\text{tot}}_{3,t}\right)\right]\text{s.t.}\sum_{t}\ell_{i,t}=L_{i}\,.$
(6)

Note that in all three scenarios, the various operators all minimize a certain
power: The active powers of each EVCS in the local scenario, the sum of these
powers in the global one and the apparent power at the head of the grid for
the grid aware one.

Given the charging needs from previous section induced from the drivers
behavior (equation (3)), the charging operation is scheduled according to each
one of the three scenarios described above. Then, the actual state of the grid
corresponding to each scenario loads is computed using the power flow
equations introduced in next section. The ultimate goal is to compare the
efficiency of each scenario in terms of grid costs defined by function
$\mathcal{G}$ (equation (6)) in Section IV-B considering real data sets.
Before that, the algorithms of the three scenarios are detailed in next
section.

##  III Algorithms

###  III-A Local scheduling

Each CSO $i$ can find the charging scheduling solution of the minimization
problem (4) by using next proposition with $L=L_{i}$ and
$\ell^{0}_{t}=\ell^{0}_{i,t}$:

###### Proposition 1

Given a nonflexible load $(\ell^{0}_{t})_{t}$, the optimal scheduling
$(\ell^{*}_{t})_{t}$ of the charging need $L$ is:

$\ell_{t}^{*}=\begin{cases}\left(L+L^{0}_{t_{0}}\right)/\left(\sum_{s\leq
t_{0}}\frac{\eta_{t}}{\eta_{s}}\right)-\ell^{0}_{t}\,,&\text{if }t\leq
t_{0}\,,\\\ 0&\text{if }t>t_{0}\,,\end{cases}$ (7)

with $L^{0}_{t}=\sum_{s\leq t}\ell^{0}_{s}$, where $t_{0}$ is such that
$L\in]L_{t_{0}};L_{t_{0}+1}]$ and $L_{t}=\left(\sum_{s\leq
t}\eta_{t}/\eta_{s}\right)\ell^{0}_{t}-L^{0}_{t}$.

The optimal charging profile given in proposition 1 has a “water filling”
structure [11] when $\eta_{t}=\eta$. For more details, please refer to our
previous work [14].

###  III-B Global scheduling

Applying proposition 1 to all EVCS aggregated (i.e., $L=\sum_{i}L_{i}$ and
$\ell^{0}_{t}=\sum_{i}\ell^{0}_{i,t}$) gives an optimal aggregated profile
$\left(\ell^{*}_{t}\right)_{t}$ which minimizes
$\sum_{t}\eta_{t}\ell_{t}^{2}$. However, several total profiles
$\left(\ell_{i,t}\right)_{i,t}$ verify $\sum_{i}\ell_{i,t}=\ell^{*}_{t}$
($\forall t$), i.e., are solution of (5). The “disaggregation”
($\left(\ell^{*}_{t}\right)_{t}\rightarrow\left(\ell_{i,t}\right)_{i,t}$)
chosen is presented in Algorithm 1:

Input: $\ell_{i,t}^{0}\,,L_{i}$

1 $\ell_{i,t}\longleftarrow$ Optimal local profile;

2 $\ell_{t}\longleftarrow$ Optimal aggregated profile;

3
$\ell_{i,t}\longleftarrow\ell_{i,t}+\frac{1}{T}\left(\ell_{t}-\sum_{i}\ell_{i,t}\right)$;

4 while  _$\exists~{}\ell_{i,t} <0$_ do

5 $\ell_{j,t}\longleftarrow\ell_{j,t}+\frac{1}{N-1}\ell_{i,t}$ $j\neq i$;

6 $\ell_{i,s}\longleftarrow\ell_{j,s}+\frac{1}{T-1}\ell_{i,t}$ $~{}s\neq t$;

7 $\ell_{j,s}\longleftarrow\ell_{j,s}-\frac{1}{(T-1)(N-1)}\ell_{i,t}$ $j\neq
i$, $s\neq t$;

8 $\ell_{i,t}\longleftarrow 0$;

9

Output: $\ell_{i,t}$

Algorithm 1 Global scheduling

To obtain this optimal global scheduling, the solution of (4) (Line 1) is
modified as little as possible in order to be solution of (5) (Line 3).
Unfortunately, these perturbations create negative charging quantities (Line
4), which are taken into account (Line 8) and compensated (Lines 5, 6 and 7)
so that the constraints at each EVCS are still verified.

###  III-C Grid aware scheduling

The objective function $\mathcal{G}$ of the ENO requires the apparent power
$S$ needed to meet the electricity demand at all EVCS (equation (6)). This
quantity is obtained by solving the power flow equations from the Bus
Injection Model (runpp function in pandapower Python library) and which
corresponds to the power balance at each bus (between the given power
production/load $S_{0,k}$ at bus $k$ and power flows $S_{k}$ from/to the bus):

$S_{0,k}=U_{k}\sum_{m\in
X_{k}}\overline{Y_{k,m}}\overline{U_{m}}~{}(=S_{k})\,,$ (8)

with $U_{k}$ the complex voltage at bus $k$, $X_{k}$ the set of buses
connected to bus $k$ and $Y_{k,m}$ the admittance of the line between buses
$k$ and $m$.

Because of the implicit nature of $\mathcal{G}$, iterative water filling
algorithms applied to problem (6) do not result in an explicit solution, as in
the local and global scenarios. Instead, (6) is seen as a convex optimization
problem, solved by built-in Python function minimize, relying on a sequential
least squares programming method.

##  IV Numerical illustrations

###  IV-A Impact of transportation system on the grid

Figure 2: Number of EV and GV at equilibrium on each path, in function of toll
$t_{s,3}$ on path 3.

This section illustrates the impact of the transportation system on the
electrical grid. More precisely, we look at how a transportation parameter –
here, the toll on path 3 – controlled by a Transportation Network Operator
(TNO), can impact the grid costs $\mathcal{G}$ related to the charging
operation of EV and defined in (6). Note that the toll on path 3 is the same
for EV and GV.

The parameters of the congestion game are set as follows.The length of the
three transportation paths are $l_{1}=30$ km, the average commuting driving
distance in USA111Omnibus Household Survey, Bureau of Transportation
Statistics, and $l_{2}=l_{3}=20$ km. The corresponding speed limits are
$v_{1}=v_{2}=50$ km/h and $v_{3}=70$ km/h, so that
$d^{0}_{3}<d^{0}_{2}<d^{0}_{1}$. The path capacities are equal to the total
number of vehicles: $C_{i}=N=3000$ vehicles. The values of the following
parameters are the same as in [10]: $X_{e}=50$%, $\tau=10$ €/h, $m_{g}=0.06$
L/km, $\lambda_{g}=1.5$ €/L and $m_{e}=0.2$ kWh/km. The charging unit price is
set to $\lambda_{e}=20$ c€/kWh. There are no tolls on paths 1 and 2:
$t_{s,1}=t_{s,2}=0$ €.

Under these driving conditions, the paths choice of commuters (i.e., the
Wardrop Equilibrium $\mathbf{x^{*}}$) was computed for each toll value
$t_{s,3}$ on path 3 (see Fig. 2). When there is no toll (i.e. $t_{s,3}=0$),
most vehicles choose path 3 as it is the fastest, except for a few GV which
use path 2, the second fastest (otherwise, path 3 would be too congested). For
$t_{s,3}\leq 3.50$ €, more vehicles rather choose path 2 because of the toll
on path 3. For $t_{s,3}>3.50$ €, path 3 has become even less attractive than
the longer path 1, so that no vehicles are left on path 3.

These path choices made by EV determine the charging need at each EVCS in
function of $t_{s,3}$. Fig. 3 then shows the resulting grid costs
$\mathcal{G}_{m}(t_{s,3})$ defined in equation (6) for each scheduling method
$m$ ($m=l,g,a$ resp. for local, global and grid aware), in function of
$t_{s,3}$. These costs are normalized into $\varepsilon_{m,0}$, using the grid
costs of the grid aware method with $t_{s,3}=0$:

$\varepsilon_{m,t}(t_{s,3})=\frac{\mathcal{G}_{m}(t_{s,3})-\mathcal{G}_{a}(t)}{\mathcal{G}_{a}(t)}\,.$
(9)

The parameters of the scheduling problem are set as
follows222https://pandapower.readthedocs.io/en/v2.2.0/std\\_types.html.:
Standard types were used for the transformer (63 MVA 110/20 kV) and the lines
(1x240 RM/25 12/20 kV). The lengths of these lines are $l_{a}=10$ km and
$l_{b}=l_{c}=5$ km. The number of time slots for the scheduling is $T=8$ and
the different objectives defined in (4), (5) and (6) are assumed to be time-
independent ($\eta_{t}=\eta$). Here, the nonflexible load of each EVCS is
taken proportional to the consumption of a Texan household (from 9 a.m. to 5
p.m., i.e., eight hours), so that the global demand over the working hours is
$\sum_{i}\sum_{t=1}^{T}\ell^{0}_{i,t}=30$ MWh. The data set used gives hourly
electric consumption throughout a year of Texan households333Data available at
http://www.pecanstreet.org/..

Figure 3: Grid costs $\mathcal{G}_{m}(t_{s,3})$ relative to the reference case
(grid aware method with no toll), in function of the toll $t_{s,3}$ on path 3,
for each algorithm $m$.

The first insight given by Fig. 3 is that a transportation toll (between 1.50
€ and 3.50 €) can be beneficial in terms of grid costs. For such values of
toll, some charging need of EVCS 3 is shifted to EVCS 2 (see Fig. 2), which is
less costly because EVCS 3 is at the end of the line supplying the two EVCS.
The transportation network impacts directly the grid when EV drivers switch
(at $t_{s,3}=3.50$ €) from path 3 to path 1, which is associated with a larger
electricity consumption (because longer distance traveled): This larger
charging need causes a grid costs increase of close to 8%. Finally, note that
the local method gives grid costs only 1% higher (approximately) than the
other two methods. The difference of grid costs between the three methods will
be studied in more details in next section.

###  IV-B Grid costs associated with scheduling methods

The three scheduling methods introduced in sections II-B and III are
illustrated in this section, and compared with respect to their associated
grid costs. First, an example of these three methods is given in Fig. 4. The
nonflexible consumption (in grey) is the same as in previous section, except
that here the eight working hours are split into $T=3$ time slots. The toll on
path 3 is fixed to $t_{s,3}=4$ €, so that there is no charging need at EVCS 3.

Figure 4: Example of the three scheduling algorithms.

The local method smooths each EVCS $i$ load profile
$\left(\ell^{\text{tot}}_{i,t}\right)_{1\leq t\leq T}$ (upward diagonal hatch
in Fig. 4). Unfortunately, in some scenarios like here, the corresponding
aggregated load profile can be far from smoothed (right figure). This is the
reason why the local method results in higher grid costs: For example,
charging vehicles at EVCS 2 during time slot $t=3$ while there is already a
high nonflexible consumption (possibly at other EVCS) can be costly. The
global method smooths the aggregated load profile of the three EVCS
(horizontal hatch), with local profiles at each EVCS as smoothed as possible
(see left figure). The optimal “Grid aware” method, minimizing grid costs, has
almost smoothed aggregated and EVCS 1 profiles: As EVCS 1 is farther away from
the transformer than EVCS 2 (10 km instead of 5), charging there is more
expensive for the grid.

Then, a comparison between the three methods is given in Table I. For this, a
thousand nonflexible profiles with $T=8$ time slots are randomly generated for
each EVCS, such that for each generation,
$\sum_{i}\sum_{t=1}^{T}\ell^{0}_{i,t}=30$ MWh. Each generated profile has also
a 2 and a 4 time slots version. This table shows the mean over these random
profiles of the normalized grid costs $\varepsilon_{m}=\varepsilon_{m,4}(4)$
and the execution time $T_{m}$ of method $m$. This table confirms that the
grid aware method is optimal with respect to grid costs
($\varepsilon_{l},\varepsilon_{g}>0$) and shows that the global one remains
very close, while the local method difference is of the order of the percent.
In terms of execution times obtained with an Intel Core i7-6820HQ 2.70GHz, the
local and global methods are negligible compared to the grid aware method.

The execution time depends on the number of variables of the optimization
problems (4), (5) and (6): the number of EVCS multiplied by the number of time
slots. A comparison between 2, 4 and 8 time slots shows that the execution
time of the grid aware method goes from 1s to approximately 21s. However, the
normalized grid costs of the other two methods increase too (nearly
proportional to the number of time slots). Thus, the choice of the method is a
trade-off between execution time and optimal grid costs. Nevertheless, the
global method seems to be the best choice as it is fast and near-optimal, even
if it does not take into account the network topology. Though note that for
more complex electrical networks, this might not be the case anymore.

TABLE I: Depending on the number of time slots into which working hours are divided, mean over 1000 randomly generated nonflexible profiles, of normalized grid costs $\varepsilon_{m}=\varepsilon_{m,4}(4)$ and execution time $T_{m}$ of method $m$. Nb time slots |  $\varepsilon_{l}$ (%) |  $\varepsilon_{g}$ (%) |  $T_{l}$ (s) |  $T_{g}$ (s) |  $T_{a}$ (s)
---|---|---|---|---|---
2 | 0.4 | 8e-03 | 5e-05 | 9e-05 | 0.8
4 | 1.0 | 2e-02 | 6e-05 | 3e-04 | 4.3
8 | 2.2 | 3e-02 | 6e-05 | 1e-03 | 20.9

##  V Conclusion

In this paper, the driving and charging decisions of Electric Vehicles (EV)
are modeled through a coupled Traffic Assignment Problem and an exact load
flow on a simple distribution network. The EV load scheduling is done with
three charging strategies, with different assumptions regarding the data
needed: from a purely per station distributed strategy, to a centralized one
based on the load flow calculation. Numerically, the impact of a
transportation parameter - a toll value - is observed on the grid cost
obtained on the charging operation; it shows both interlinked driving and
charging effects. It also exhibits that distributed charging strategies, which
are non-load flow based, perform comparably to the load flow based strategy in
terms of grid cost. This opens the way to the use of distributed simpler
strategies in this coupled context; however such comparative study will be
extended to more complicated electricity networks where this conclusion may
differ. In a future work, dynamic charging pricing to make EV “grid-aware” (by
charging at less congested EVCS for example) will be integrated and the
different operators and their interactions will be considered.

## References

  * [1] International Energy Agency, “Tracking clean energy progress 2017,” tech. rep., IEA, 2017.
  * [2] W. Wei, Q. Wu, M. Shafie-Khah, and J. P. Catalào, “Transportation system and power system interdependence: a comprehensive review on models and applications,” Journal of Modern Power Systems and Clean Energy, 2019.
  * [3] M. Alizadeh, H. Wai, M. Chowdhury, A. Goldsmith, A. Scaglione, and T. Javidi, “Optimal pricing to manage electric vehicles in coupled power and transportation networks,” IEEE Trans. on Control of Network Systems, vol. 4, no. 4, pp. 863–875, 2017.
  * [4] W. Wei, L. Wu, J. Wang, and S. Mei, “Network equilibrium of coupled transportation and power distribution systems,” IEEE Trans. on Smart Grid, vol. 9, no. 6, pp. 6764–6779, 2018.
  * [5] F. He, Y. Yin, J. Wang, and Y. Yang, “Sustainability si: optimal prices of electricity at public charging stations for plug-in electric vehicles,” Networks and Spatial Economics, vol. 16, no. 1, pp. 131–154, 2016.
  * [6] M. Zhu, X.-Y. Liu, and X. Wang, “Joint transportation and charging scheduling in public vehicle systems—a game theoretic approach,” IEEE Trans. on Intelligent Transp. Syst., vol. 19, no. 8, 2018.
  * [7] X. Wang, M. Shahidehpour, C. Jiang, and Z. Li, “Coordinated planning strategy for electric vehicle charging stations and coupled traffic-electric networks,” IEEE Trans. on Power Systems, vol. 34, no. 1, pp. 268–279, 2018.
  * [8] M. Patriksson, The Traffic Assignment Problem: Models and Methods. Dover Publications, 2015.
  * [9] J. Zhu, Optimization of power system operation, J.Wiley & Sons, 2015.
  * [10] B. Sohet, O. Beaude, Y. Hayel, and A. Jeandin, “Optimal incentives for electric vehicles at e-park & ride hub with renewable energy source,” World Electric Vehicle Journal, vol. 10, no. 4, p. 70, 2019.
  * [11] A.-H. Mohsenian-Rad, V. W. Wong, J. Jatskevich, R. Schober, and A. Leon-Garcia, “Autonomous demand-side management based on game-theoretic energy consumption scheduling for the future smart grid,” IEEE Trans. on Smart Grid, vol. 1, no. 3, pp. 320–331, 2010.
  * [12] H. Spiess, “Technical note-conical volume-delay functions,” Transportation Science, vol. 24, no. 2, pp. 153–158, 1990.
  * [13] J. Wardrop, “Some theoretical aspects of road traffic research,” Proc Inst Civ Eng Part II, vol. 1, pp. 325–278, 1952.
  * [14] B. Sohet, O. Beaude, Y. Hayel, and A. Jeandin, “Coupled charging-and driving incentives design for electric vehicles in urban networks,” IEEE Trans. on Intelligent Transportation Systems, in press, arXiv:2001.11758.