††thanks: These authors contributed equally.††thanks: These authors
contributed equally.

# Recycling of Perovskite Substrate

Jie Wang  Faculty of Materials Science and Engineering, Kunming University of
Science and Technology, Kunming, 650093, Yunnan, China  Yuhan Liang  School of
Materials Science and Engineering, Tsinghua University, Beijing, 100084, China
Yue Wang  School of Materials Science and Engineering, Tsinghua University,
Beijing, 100084, China  Shengan Yang  Faculty of Materials Science and
Engineering, Kunming University of Science and Technology, Kunming, 650093,
Yunnan, China  Xinyu Zhang  Faculty of Materials Science and Engineering,
Kunming University of Science and Technology, Kunming, 650093, Yunnan, China
Xin Huang  Faculty of Materials Science and Engineering, Kunming University of
Science and Technology, Kunming, 650093, Yunnan, China  Yajuan Wang  Faculty
of Materials Science and Engineering, Kunming University of Science and
Technology, Kunming, 650093, Yunnan, China  Zhaowei Liang  Faculty of
Materials Science and Engineering, Kunming University of Science and
Technology, Kunming, 650093, Yunnan, China  Ji Ma  Faculty of Materials
Science and Engineering, Kunming University of Science and Technology,
Kunming, 650093, Yunnan, China  Hui Zhang  Faculty of Materials Science and
Engineering, Kunming University of Science and Technology, Kunming, 650093,
Yunnan, China  Qingming Chen  Faculty of Materials Science and Engineering,
Kunming University of Science and Technology, Kunming, 650093, Yunnan, China
Jing Ma  School of Materials Science and Engineering, Tsinghua University,
Beijing, 100084, China  Yuanhua Lin<EMAIL_ADDRESS>School of Materials
Science and Engineering, Tsinghua University, Beijing, 100084, China  Liang Wu
<EMAIL_ADDRESS>Faculty of Materials Science and Engineering, Kunming
University of Science and Technology, Kunming, 650093, Yunnan, China

###### Abstract

The use of water-soluble sacrificial layer of Sr3Al2O6 has tremendously
boosted the research on freestanding functional oxide thin films, especially
thanks to its ultimate capability to produce high-quality epitaxial perovskite
thin films. However, the costly single-crystalline substrates, e.g. SrTiO3,
were generally discarded after obtaining the freestanding thin films. Here, we
demonstrate that the SrTiO3 substrates can be recycled to fabricate
La0.7Sr0.3MnO3 films with nearly identical structural and electrical
properties. After attaining freestanding thin films, the residues on SrTiO3
can be removed by 80 °C hot water soaking and rinsing treatments.
Consequently, the surface of SrTiO3 reverted to its original step-and-terrace
structure.

##  I Introduction

The freestanding complex oxides open a door to additional spectrums of two-
dimensional materials with exotic correlated electronic phases. Based on the
excellent stretchability of freestanding oxide membranes and strong
correlation between quantum properties and lattice constant, the functional
properties of oxide can be comparable and even superior to those in their bulk
or rigid thin film forms, such as enhanced and controllable ferromagnetism [1,
2, 3], monolayer and super-elastic ferroelectricity [4, 5], and topological
ferroelectric domains [6]. Most recently, the freestanding complex oxides were
also demonstrated to integrate with conventional two-dimensional transition
metal dichalcogenides as a high $\kappa$ layer [7, 8]. Thus, the freestanding
complex oxide provides a promising path for approaching applicable flexible
devices with desirable quantum properties. This thriving area has been already
well covered in several review articles, see for example Refs. [9, 10, 11].

Among the various epitaxial lift-off techniques to fabricate freestanding
complex oxides, the water-soluble sacrificial Sr3Al2O6 (SAO) buffer layer
exhibits many advantages in the epitaxy growth of high-quality transition
metal perovskites on high-priced parent single-crystalline substrates. In the
related fields, such as semiconductor, solar cell, and 2D material
fabrications, wafer and substrate recycling is of great importance in reducing
the cost [12, 13, 14, 15, 16, 17, 18]. However, these substrates, such as
SrTiO3, were generally discarded. To this end, to develop a recycling
procedure is important.

Here, we demonstrate that a La0.7Sr0.3MnO3 (LSMO) grown on a recycled SrTiO3
(STO) substrate displays practically identical structural and transport
properties. Here, the debris on the substrate after room-temperature water
soaking (to get freestanding thin films) can be completely removed by adequate
hot water (80 °C) soaking and rinsing, and the perovskite substrates are
recovered to their original step-and-terrace surface structure with
indiscernible debris by atomic force microscopy (AFM), or any residual Al3+
from adjacent Sr3Al2O6 layer by X-ray photoelectron spectroscopy (XPS). The
hot water treatment could also be conducive to obtaining residue-free
freestanding films, especially when the interface is extremely vital in
freestanding multi-layer devices. It is worth noting that, such substrate
recycling can be applied not only to prepare freestanding thin films but also
to synthesize epitaxial functional thin films and devices by artificially
introducing the SAO as an excellent degradable buffer layer.

Figure 1: The schematic substrate recycling procedure in preparing high-
quality functional complex oxide thin films. Figure 2: (a) XRD patterns of
LSMO/SAO/STO(pristine) and LSMO/SAO/STO(recycled). The resistivity as a
function of temperature (b) and magnetic field (c) of the fresh growth and
regrowth samples. The reference black curve in (b) was reproduced from Ref.
[1].

##  II Results and Discussions

The epitaxial lift-off using an SAO sacrificial layer and sequential substrate
recycling processes are illustrated in Figure 1, details of the experiments
can be found in Section IV. The SAO buffer layer and a representative
functional LSMO layer were grown on STO(001) substrates by pulsed laser
deposition (PLD). Figure 2 shows the good reproduction of pristine and
recycled SAO and LSMO layers detected by XRD and transport properties. XRD
patterns for LSMO/SAO thin films on the pristine and recycled STO (MTI
Corporation) are displayed in Figure 2(a). As seen, The (008)SAO and (002)LSMO
peaks are located on the left and right side of (002)STO peak, consistent with
the previous study [1]. No detectable impurity phases were detected in a wider
range (20–80°) XRD pattern (not shown here). The almost identical XRD patterns
indicate that the LSMO/SAO grown on the recycled STO possessed a non-degraded
crystallinity as compared to that grown on the pristine STO. Thus, the
recycled substrate should have returned to its original state.

Figure 3: AFM topographies of the pristine STO substrate (a), substrate after
room-temperature water soaking (b), recycled substrate by 80 °C hot water
treatment after first time growth (c), and recycled substrate after second
time growth (d), respectively. Figure 4: XPS of the pristine (lower panel) and
recycled (upper panel) STO substrates, (a) Ti 2p, (b) Sr 3d, (c) Al 2p.

To further verify the quality of the recycled LSMO/SAO/STO heterostructures
compared with pristine one, we measured the temperature- and magnetic field-
dependent resistivity of aforementioned two samples by a physical property
measurement system (PPMS, Quantum Design). As seen in Figure 2(b), the
resistivity versus temperature ($R$–$T$) curves of both samples perfectly
coincided. The black curve in Figure 2(b) was a reproduction from a very
similar heterostructure of LSMO/SAO/STO from Ref. [1], it was just this
seminal paper which originally reported the SAO as a sacrificial layer to
fabricate freestanding perovskite oxides. Indeed, we see that the $R$–$T$ of
our samples is comparable to, or slightly better than that in Ref. [1], since
at very low temperature (e.g. $\leqslant$ 20 K), the Anderson weak
localization (upturn of resistivity at low temperature) was not present. In
addition, we also detected the ferromagnetic properties by field-dependent
resistivity (MR) curves. As shown in Figure 2(c), these two MR curves at 5 K
are quite similar in lineshape, with approximately identical coercive field
and MR value. The MR data of our samples is very similar to a very
conventional lineshape of LSMO [19]. Combining the XRD patterns and transport
properties, we conclude that the functional layer, as exemplified by LSMO, can
be impeccably fabricated on a recycled substrate with well reproducible
properties. We have also carried out experiments on the effect of
environmental moisture on film stability (Figure S1, Supporting Information),
which indicated that sample surface topography deteriorated in a high relative
humidity (RH, here 99%) at 25 °C, while it remained unchanged in a low
relative humidity (here, 20%).

To validate the reproducibility and recoverability of recycled substrate in
our process, here, we utilized AFM (Asylum Research) equipped with an FM-LC
tip (Adama Innovations) to detect the topographies of pristine and recycled
substrates [1]. As displayed in Figure 3(a), the pristine substrate shows a
standard step-and-terrace surface structure as expected. After the lift-off
process with room-temperature water, some residues adhered to the sample
surface with the absence of the step-and-terrace surface of the substrate, as
shown in Figure 3(b). We ascribed it to the colloid formed by the reaction of
aluminate with water. Remarkably, after the substrates were treated with hot
water, both recycled substrates after first and second-time growths, have
restored to the step-and-terrace surfaces without any observable
deterioration, as shown in Figure 3(c)–(d). This is consistent with the
similar reflection high energy electron diffraction (RHEED) patterns of
pristine and recycled substrates (Figure S2, Supporting Information), which
indicates that the surface structure of substrate is unaffected by the
adequate hot water soaking. Note that, the step edges of the substrates are
not perfectly ordered due to the slightly poor pristine substrate quality. In
addition, different growth temperatures were used to prepare LSMO/SAO thin
films, which did not affect the surface topography of the recycled substrates
(Figure S3, Supporting Information). Nevertheless, this undoubtedly manifests
that the substrates were successfully recycled by hot water treatment.

Finally, the surface chemical states of the pristine and recycled STO
substrates were examined by XPS as shown in Figure 4. XPS is a surface-
sensitive technique, which is one of the best methods to detect residues at
trace levels. As expected, similar Ti 2p and Sr 3d peaks were seen in pristine
and recycled STO substrates, see Figure 4(a)–(b) [20]. However, no Al3+ 2p
peak from residual adjacent Sr3Al2O6 layer was identified (Figure 4(c)), which
is consistent with our AFM characterizations and reveals the atomic-level
recycling of STO substrates.

##  III Conclusion

In summary, our findings corroborate that the substrates used in fabricating
freestanding functional complex oxides by the sacrificial SAO method can be
non-destructively reused, and the recycled functional layer is almost
identical to pristine state. The recycling procedure is exceptionally facile
and straightforward, i.e., adequate hot water soaking and rinsing are capable
to refresh the surface of the substrates. This technique may reduce the cost
of research activities in this field, especially fabrication of large-scale
flexible devices with freestanding complex oxides. In addition, by
artificially including a degradable buffer SAO layer, such substrate recycling
can be popularized even in non-freestanding complex oxide synthesis. We also
highly advocate the researchers to scrutinize whether an analogous recycling
technique can be developed in other epitaxial lift-off processes.

##  IV Experimental Section

Films Growth and Structural Characterization
The SAO buffer layer and a representative functional LSMO layer were grown on
STO(001) substrates by pulsed laser deposition (PLD). Prior to the growth, the
STO was treated with a mixture of concentrated nitric and hydrochloric acids
according to the protocol used in Ref. [21]. The growth condition for SAO was
720 °C (measured by a pyrometer) in $10^{-6}$ Torr oxygen with a KrF excimer
laser (energy density of $\thicksim$ 1.5 J/cm2, repetition rate of 2 Hz),
while the LSMO was grown in 200 mTorr oxygen with all other parameters same to
those of SAO. X-ray diffraction (XRD, Malvern Panalytical) measurements were
used to verify the crystal structure, phase purity, and orientation of these
thin films.

Substrate Refreshment and Surface Topography
After the growth, the sample was cut into two pieces. One out of them was
soaked into room-temperature deionized water for $\thicksim$ 6 hours to
acquire the freestanding LSMO. Thereafter, we heated the water to $\thicksim$
80 °C for $\thicksim$ 6 hours to get rid of the tiny amounts of debris of SAO
on the surface of STO (The pH values of the deionized water were 6.94 and 6.22
at 20 °and 80 °, respectively.). The surface Topography was detected by atomic
force microscopy (AFM, Asylum Research) equipped with an FM-LC tip (Adama
Innovations). After sequential rinsing by deionized water, the exact same
growth conditions were utilized to repeatedly fabricate LSMO/SAO thin films on
this recycled STO substrate.

Electrical Transport
The temperature dependence of the resistivity and magnetoresistance was
carried out by Physical Property Measurement System (PPMS, Quantum Design
DynaCool system) via the Vander Pauw method.

X-ray Photoelectron Spectroscopy
X-ray Photoelectron Spectroscopy was performed by ESCALAB 250Xi X-ray
Photoelectron Spectrometer (Thermo Fisher Scientific) equipped with a
monochromatic Al $k_{\alpha}$ X-ray source. High-resolution spectra were
acquired for surface elemental identification, which was fitted based on
Lorentzian functions broadened by a Gaussian function.

Acknowledgements

L. W. acknowledges supports from National Natural Science Foundation of China
(Grant No. 52102131) and Yunnan Fundamental Research Projects (Grant No.
202101BE070001-012 and 202201AT070171).

## References

  * Lu  _et al._ [2016] D. Lu, D. J. Baek, S. S. Hong, L. F. Kourkoutis, Y. Hikita, and H. Hwang, Nat. Mater. 15, 1255 (2016).
  * Hong  _et al._ [2020] S. S. Hong, M. Gu, M. Verma, V. Harbola, B. Y. Wang, D. Lu, A. Vailionis, Y. Hikita, R. Pentcheva, J. M. Rondinelli, and H. Y. Hwang, Science 368, 71 (2020).
  * Peng  _et al._ [2022] H. Peng, N. Lu, S. Yang, Y. Lyu, Z. Liu, Y. Bu, S. Shen, M. Li, Z. Li, L. Gao, S. Lu, M. Wang, H. Cao, H. Zhou, P. Gao, H. Chen, and P. Yu, Adv. Funct. Mater. 32, 2111907 (2022).
  * Ji  _et al._ [2019] D. Ji, S. Cai, T. R. Paudel, H. Sun, C. Zhang, L. Han, Y. Wei, Y. Zang, M. Gu, Y. Zhang, W. Gao, H. Huyan, W. Guo, D. Wu, Z. Gu, E. Y. Tsymbal, P. Wang, Y. Nie, and X. Pan, Nature 570, 87 (2019).
  * Dong  _et al._ [2019] G. Dong, S. Li, M. Yao, Z. Zhou, Y.-Q. Zhang, X. Han, Z. Luo, J. Yao, B. Peng, Z. Hu, H. Huang, T. Jia, J. Li, W. Ren, Z.-G. Ye, X. Ding, J. Sun, C.-W. Nan, L.-Q. Chen, J. Li, and M. Liu, Science 366, 475 (2019).
  * Han  _et al._ [2022] L. Han, C. Addiego, S. Prokhorenko, M. Wang, H. Fu, Y. Nahas, X. Yan, S. Cai, T. Wei, Y. Fang, H. Liu, D. Ji, W. Guo, Z. Gu, Y. Yang, P. Wang, L. Bellaiche, Y. Chen, D. Wu, Y. Nie, and X. Pan, Nature 603, 63 (2022).
  * Yang  _et al._ [2022] A. J. Yang, K. Han, K. Huang, C. Ye, W. Wen, R. Zhu, R. Zhu, J. Xu, T. Yu, P. Gao, Q. Xiong, and X. R. Wang, Nat. Electron. 5, 233 (2022).
  * Huang  _et al._ [2022] J.-K. Huang, Y. Wan, J. Shi, J. Zhang, Z. Wang, W. W. ann Ni Yang, Y. Liu, C.-H. Lin, X. Guan, L. Hu, Z.-L. Yang, B.-C. Huang, Y.-P. Chiu, J. Yang, V. Tung, D. Wang, K. Kalantar-Zadeh, T. Wu, X. Zu, L. Qiao, L.-J. Li, and S. Li, Nature 605, 262 (2022).
  * Zhang  _et al._ [2019] Y. Zhang, C. Ma, X. Lu, and M. Liu, Mater. Horizons. 6, 911 (2019).
  * Kum  _et al._ [2019] H. Kum, D. Lee, W. Kong, H. Kim, Y. Park, Y. Kim, Y. Baek, S.-H. Bae, K. Lee, and J. Kim, Nat. Electron. 2, 439 (2019).
  * Gao  _et al._ [2020] W. Gao, Y. Zhu, Y. Wang, G. Yuan, and J.-M. Liu, J. Materiomics 6, 1 (2020).
  * Lee  _et al._ [2014] K. Lee, J. D. Zimmerman, T. W. Hughes, and S. R. Forrest, Adv. Funct. Mater. 24, 4284 (2014).
  * Wie  _et al._ [2018] D. S. Wie, Y. Zhang, M. K. Kim, B. Kim, S. Park, Y.-J. Kim, P. P. Irazoqui, X. Zheng, B. Xu, and C. H. Lee, Proc. Natl. Acad. Sci. 115, E7236 (2018).
  * Huang  _et al._ [2016] L. Huang, Z. Hu, J. Xu, X. Sun, Y. Du, J. Ni, H. Cai, J. Li, and J. Zhang, Sol. Energy Mater. Sol. Cells 152, 118 (2016).
  * Huang  _et al._ [2017] L. Huang, J. Xu, X. Sun, R. Xu, Y. Du, J. Ni, H. Cai, J. Li, Z. Hu, and J. Zhang, ACS Sustain. Chem. Eng. 5, 3261 (2017).
  * Takada  _et al._ [2020] T. Takada, T. Uchiyama, Y. Okada-Shudo, K. Hoshino, K. Koizumi, Y. Takeoka, and V. Vohra, ACS Sustain. Chem. Eng. 8, 5807 (2020).
  * Xu  _et al._ [2015] Z.-Q. Xu, Y. Zhang, S. Lin, C. Zheng, Y. L. Zhong, X. Xia, Z. Li, P. J. Sophia, M. S. Fuhrer, Y.-B. Cheng, and Q. Bao, ACS Nano 9, 6178 (2015).
  * Wang  _et al._ [2017] M. C. Wang, W. P. Moestopo, S. Takekuma, S. F. Barna, R. T. Haasch, and S. Nam, J. Mater. Chem. C 5, 11226 (2017).
  * Liao  _et al._ [2014] Z. Liao, M. Huijben, G. Koster, and G. Rijnders, APL Mater. 2, 096112 (2014).
  * Vasquez [1992] R. P. Vasquez, Surf. Sci. Spectra 1, 129 (1992).
  * Kareev  _et al._ [2008] M. Kareev, S. Prosandeev, J. Liu, C. Gan, A. Kareev, J. W. Freeland, M. Xiao, and J. Chakhalian, Appl. Phys. Lett. 93, 061909 (2008).


#  Revision summary for paper _SPL-34115-2022_

(  Chin-Yi Cheng, Hung-Shin Lee, Yu Tsao, and Hsin-Min Wang )

The authors would like to thank the reviewers and AE for their comments, which
greatly helped us improve the manuscript. We have addressed all the issues
seriously. Revisions are marked in red in the revised paper. The point-by-
point responses to the comments are as follows:

## Reviewer # 1

### Comment # 1.1

My mayor concern with the paper is that the architecture is not being properly
described. No parameters or proper description is given and the method is
therefore impossible to reproduce. Also, the terminology used is rather
confusing. Authors introduce several names such as “filter”, “feature mixer”,
and “z-vectors”, for standard techniques. The filter is an x-vector extractor,
the feature mixer is a BiLSTM, which is followed by a “Contextualizer”, which
is another BiLSTM also operating on a frame basis (but getting a different
name), z-vector is an x-vector… It would be easier for the reader to get the
whole picture of the architecture if the figures and the text would describe
the real components that are being used rather than using complicated names
and mentioning the architectures only in the experimental setup. Authors can
motivate their choices for the different components in any case. Finally, the
authors claim that their method is suitable and stronger than TS-VAD for
working with a variable number of speakers, but eventually show the results of
their final system for only CALLHOME-2 2-spk conversation subset. If the
method is really better, why not show how it performs in the whole CALLHOME-2
and/or per number of speakers as it is standard in other publications?

Thank you for the comments. We additionally added details about the structure
along with some parameters in the revised paper.

We also removed or revised some duplicated or misleading terms in the original
paper. However, we would like to clarify that the z-vector is not a x-vector.
The z-vector is extracted by ResNet and Attentive Statistic Pooling, while the
x-vector is generated by TDNN via the Kaldi recipe. Since the z-vector
extractor can be unified with the detector, we are able to back-propagate the
loss to the extractor. We believe this joint-training will make the z-vector a
better speaker representation for the speaker diarization task. Therefore, we
have reserved the name z-vector, where z stands for diari“z”ation.

Regarding the last issue, we clarify as follows. The evaluation results on the
entire Callhome-2 set are shown in Table IV. Table V only shows 2-spk results
because many state-of-the-art models, such as BLSTM-EEND, SA-EEND, and SA-
EEND-EDA + Frame Selection, were only evaluated on the 2-spk task, and we
wanted to make a fair comparison. Sorry for the misunderstanding caused by the
order in which the experimental results are presented.

### Comment # 1.2

I think the title should be modified. There is no “filter” in their
architecture

Thank you for the suggestion. In the revised paper, we replaced “filter” with
“extractor” and updated the title to “Multi-target Extractor and Detector for
Unknown-number Speaker Diarization”.

### Comment # 1.3 Introduction:

timestamps -> speech regions?, timestamps is not properly used

Thank you for the suggestion. We replaced “timestamps” with “speech regions”
in the revised paper.

It relies on the x-vector/SC procedure [Snyder2018, Sell2018] to provide
initial speech regions for each “target” (active) speaker, and then extracts a
first-stage i-vector for each target speaker from the corresponding frames.

known as Multi-Target Filter and Detector -> it cannot be know if you are
introducing it, use for example "which we refer to"

Thank you for the comment. In conjunction with your other suggestions, in the
revised paper, this sentence is revised as follows.

Inspired by the dual-path recurrent neural network (DPRNN) [Luo2020,
Chenda2021], we propose a unified structure called Multi-target Extractor and
Detector (MTEAD), which can handle sessions with various numbers of speakers
using a single model.

### Comment # 1.4 Multi-Target Filter and Detector

convolutionized -> please, replace

Thank you for the suggestion. In the revised paper, the sentences are revised
as follows.

Section II-A The frame-level MFCCs are first passed through a four-layer
convolutional neural network (CNN), and then concatenated with each speaker
representation.

Section II-B The Detector consists of a Feature Mixer and three consecutive
Frame-Speaker Contextualizers implemented using BiLSTM.  The MFCCs are passed
through CNNs and concatenated with each speaker representation. These
concatenated features are then processed by Feature Mixer to generate the
corresponding $\rm S_{d}$ frames $\mathbf{X}^{i}$ for the $i$-th speaker, as
shown in Fig. 1.

1. Authors introduce here "Speaker aware frames (SAF)" which are in fact the same thing as the "Speaker Detection (SD)" vector sequences of the TS-VAD: simply the output of the BiLSTM which (in both methods) receives the concatenation of the MFCCs and the speaker representation. Why introduce a new term for it? 2. The figure here might be unnecessary. 3. "Dimension permutation" -> "permutation" is rather pseudo code notation. Consider rephrasing 4. "From the output of the previous Frame-Speaker Contextualizer is passed through…" -> previous? do you mean the last?

Thank you for the comments.

  1. 1.

You are right. In the revised paper, we replaced “Speaker aware frames (SAF)”
with “speaker detection (Sd) frames” or “$S_{d}$ vector sequences”. We used SD
as the abbreviation for speaker diarization.

Finally, for each speaker, the corresponding $\rm S_{d}$ vector sequence from
the output of the last Frame-Speaker Contextualizer is passed through a
linear-sigmoid layer to generate the final diarization result.

  2. 2.

Although the architecture of Frame-Speaker Contextualizer is similar to that
of DPRNN, due to the different interpretation of the third dimension (inter-
frame in DPRNN vs. inter-speaker in MTEAD), we believe this figure is still
necessary to clarify the model.

  3. 3.

We replaced “permutation” with “reshaping” in the revised paper.

The output of the first stage, $\tilde{\mathbf{X}}$, is the stack of
$\tilde{\mathbf{X}}^{i}$, $i=1,...,N$. The input of the second stage,
$\mathbf{Y}$, is generated by reshaping $\tilde{\mathbf{X}}$. Slicing the
input by timeframes yields $\mathbf{Y}^{j}\in\mathbb{R}^{D\times N}$,
$j=1,...,T$, which is treated as the activity of individual speakers in a
single frame $j$.

  4. 4.

We replaced “previous” with “last” in the revised paper.

Finally, for each speaker, the corresponding $\rm S_{d}$ vector sequence from
the output of the last Frame-Speaker Contextualizer is passed through a
linear-sigmoid layer to generate the final diarization result.

1. As mentioned above, the "filter" is not really a filter, but an x-vector extractor. 2. "ResNet and Attentive Statistic Pooling" being this one of the main contributions of the paper according to the authors, the architecture should be better described. 3. "the speaker representation extraction and detection are combined into an end-to-end process by integrating the Filter with the Detector" End-to-end is not correctly used here (nor later in other sections). This method is not end-to-end, as it requires a speaker representation obtained after using an external diarization method.

Thank you for the comments.

  1. 1.

We replaced “filter” to “extractor” in the revised paper. However, we would
like to clarify that the extractor refers to the z-vector extractor we
introduced in the paper. The z-vector extractor and detector are a unified
structure for speaker diarization, so the diarization loss (binary cross-
entropy) can be back-propagated to the extractor.

  2. 2.

We added some details about ResNet and Attentive Statistic Pooling in Section
II-C of the revised paper.

The Z-vector Extractor comprises ResNet and Attentive Statistic Pooling (ASP)
[Okabe2018]. The ResNet here is similar to the structure of ResNet50, but the
last layer of ResNet is replaced by the ASP layer. The input to ResNet is each
speaker’s 500 frames of MFCCs, and the output shape is set to be the same
dimension as the x-vector. Therefore, these z-vectors can be used as input to
Detector instead of x-vectors or i-vectors.

  3. 3.

We did misuse “end-to-end" in Section II-C of the original paper. We replaced
“an end-to-end process” with “a unified process” in the revised paper. Thanks
for pointing out this error.

In MTEAD, speaker representation extraction and speaker detection are combined
into a unified process by integrating Extractor with Detector. Also, since
Extractor and Detector are trained jointly, it is expected that the z-vector
is more suitable for SD than the x- and i-vectors.

### Comment # 1.5 Experiments and results

JER, as the diarization metric, should be referenced "loss between the
projected result and the answer using cross-entropy" -> the loss is cross
entropy. Is it binary cross entropy? One for each speaker? Please describe it.

In the revised paper, we cited references on JER. We also added an additional
Section II-D to describe the training loss. The final loss combines the losses
of each speaker. The per-speaker loss is a binary cross-entropy between the
labels and predicted SD results. The section is listed as follows.

When using z-vectors, Extractor and Detector are jointly trained from scratch
using a binary cross-entropy loss between speaker labels and predicted SD
results. When using x-vectors and i-vectors, Extractor is replaced by other
extractors pretrained by the Kaldi recipe, and only Detector is trained using
the same binary cross-entropy loss. 1. "the i-vector and x-vector speaker
representations were obtained using the pretrained extractors of Kaldi [37]"
-> Is it the standard kaldi used TDNN for x-vector extraction? 2. z-vectors
are extracted using a ResNet, is it roughly the same size as the one used for
x-vectors? How can you say otherwise that "z-vectors" are better than
x-vectors if the architectures are not comparable? There are too many details
missing to be able to judge where the improvements are coming from. 3. Text
from caption in table I should be moved to the main text. Terms like Ideal and
Threshold should not be only described in a caption. 4. "these results
demonstrate the importance of accurate speaker representation" -> is it so?
when not looking at the Ideal condition, and focusing only on real
experiments, numbers show that the speaker representation is not really that
important 5. "and better speaker representation yields superior diarization
performance" how is it better? I think the joint training might be key to get
better performance for this specific task, but that does not mean they are a
better speaker representation per se, they probably would not be suitable for
speaker verification. Please, consider rephrasing.

Thank you for the comments.

  1. 1.

Yes. We directly followed the CALLHOME recipe in Kaldi, which is based on the
standard TDNN approach. We added brief descriptions to Sections II-D and III-A
of the revised paper.

  2. 2.

The total number of parameters for the x-vector and z-vector extractors is
about 8M and 30M, respectively. For a fair comparison with the x-vector, we
set the shape of the z-vector to be the same as the x-vector. In this way, not
only the results are comparable, but also the structure of the detectors for
different extractors can be the same.

  3. 3.

In the revised paper, the descriptions about Ideal, Threshold, and Oracle are
moved to Section III-A following the suggestion.

  4. 4.

Due to space limitations, we can only illustrate the logic in this paper. When
looking at Table I along the columns, Ideal clearly outperforms its peers. In
the Ideal case, the speaker representations (x-vectors) are extracted from
unmixed data, and thus are the cleanest and best under the same x-vector
extractor. Therefore, we draw the first inference that if we can generate
better speaker representations, the diarization results may also be better. On
the other hand, when looking across rows, there was little difference in the
results. This observation suggests that clean or dirty speaker representations
by the i-vector extractor have little effect on the detector. Therefore, we
have the second inference that under the i-vector extractor, the detector has
its performance ceiling. Given these two findings, we speculated that the
binary cross-entropy loss should be back-propagated to the extractor to
extract better speaker representations. While it is possible to design a
really powerful detector to get good SD results, it is also important to
improve the quality of speaker representations.

  5. 5.

Based on the reply to question four, we adopted the rephrasing suggestion and
clarified that currently we can only limit the use of z-vectors to speaker
diarization. We apologize for the overstatement in the original paper. The
sentence was rephrased as “Speaker representations targeting SD did yield
better SD performance.” in the revised paper.

1. They should again move the caption from table III to the text, or at least make it clear that they are i-vector related experiments earlier on. 2. "This is because the model could estimate a more accurate z-vector for each speaker when the number of speakers was correct in the first-stage diarization…" -> This is speculation, I believe authors should be more careful when making such statements. 3. "Finally, we compared MTFAD with other models. As most end-to-end models…" -> As stated earlier, this is not an end-to-end model

Thank you for the comments.

  1. 1.

Done.

  2. 2.

We started the sentence with "We speculate that" in the revised paper.

We speculate that this is because when the number of speakers is correct in
the first-stage diarization, the model can estimate a more accurate z-vector
for each speaker.

  3. 3.

We did misuse “end-to-end" in Section II-C of the original paper. We replaced
“an end-to-end process” with “a unified process” in Section II-C of the
revised paper. Here, we also reworded “As most end-to-end models …” as “As
most NN-based models …".

In MTEAD, speaker representation extraction and speaker detection are combined
into a unified process by integrating Extractor with Detector. Also, since
Extractor and Detector are trained jointly, it is expected that the z-vector
is more suitable for SD than the x- and i-vectors. Since most NN-based SD
models were only evaluated in 2-speaker experiments, we compared different
models on the 2-speaker CALLHOME task.

### Comment # 1.6 Reference

Some other references of works similar to the presented one should be
considered: 1. W. Wang, Q. Lin, D. Cai and M. Li, "Similarity Measurement of
Segment-Level Speaker Embeddings in Speaker Diarization," in IEEE/ACM
Transactions on Audio, Speech, and Language Processing, vol. 30, pp.
2645-2658, 2022, doi: 10.1109/TASLP.2022.3196178.

Thank you for the suggestion. We have cited the above reference and provided
brief comments in the revised manuscript.

While we were preparing this paper, there was a similar work extending TS-VAD
by jointly training a speaker embedding network [Wang2022]. However, it does
not handle a variable number of speakers.

## Reviewer # 2

### Comment # 2.1 Frame-Speaker Contextualizer

1. The paper does not discuss why the specific frame-speaker contextualizer should work. 2. Fig 2. needs to be improved for clarity. For example, the connections between the blocks should be more accurate. The present one gives the impression that the frame-wise (or speaker-wise) contextualizer takes input from all the speakers/frames. The reviewer also thinks that the terminology “dimension permutation” is not appropriate but it could be something like “reshaping” or “rearranging” as it is just a transpose operation for the last two dimensions. 3. Contextualizer (Eq. 1) is not explained in Section II-B. Even though the authors have discussed later how it is implemented with BiLSTM, it might be good to mention this Contextualizer in Section II-B as well.

Thank you for the comments.

  1. 1.

With two dimensions of information sharing, we conjecture that the proposed
frame-speaker contextualizer should work. We added some explanations at the
end of Section II-B of the revised paper as follows.

The Speaker-wise Contextualizer allows the sharing of speakers’ information
within a single frame, which can help Detector to discern differences among
speaker representations. On the other hand, by scanning along the frame, the
Frame-wise Contextualizer can help Detector to determine whether each speaker
is active in each frame.

  2. 2.

In the revised paper, we replaced the term “dimension permutation” with
“reshaping” and revised the caption of Fig. 2 to clearly describe that the
Frame-wise Contextualizer and Speaker-wise Contextualizer take $X^{i}$ and
$Y^{j}$ as input, respectively. We hope these modifications will make the
description clearer.

The output of the first stage, $\tilde{\mathbf{X}}$, is the stack of
$\tilde{\mathbf{X}}^{i}$, $i=1,...,N$. The input of the second stage,
$\mathbf{Y}$, is generated by reshaping $\tilde{\mathbf{X}}$. Slicing the
input by timeframes yields $\mathbf{Y}^{j}\in\mathbb{R}^{D\times N}$,
$j=1,...,T$, which is treated as the activity of individual speakers in a
single frame $j$.

Caption of Fig. 2 The Frame-Speaker Contextualizer. The operator $\ocircle$
denotes the residual addition of two tensors. $\mathbf{X}^{i}$ and
$\mathbf{Y}^{j}$ are passed through the Frame-wise or Speaker-wise
Contextualizer, respectively.

  3. 3.

We mentioned at the beginning of Section II-B of the revised paper that the
Contextualizer is implemented using BiLSTM.

The Detector consists of a Feature Mixer and three consecutive Frame-Speaker
Contextualizers implemented using BiLSTM.

### Comment # 2.2 Lack of details

The manuscript lacks details about the implementation & this makes
reproducibility very difficult. The authors should report the configurations
of acoustic feature extractions, and training details (optimizer, batch size,
learning rate, stopping criteria…) that might help in reproducibility.

Thank you for the comments. Due to space limitations, we can only mention the
key parameters of model training in the paper. We used the Adam optimizer and
Noam scheduler with a learning rate of 0.1 and a batch size of 8. The
dimensions of i-vector, x-vector, and z-vector were all set to 40. The above
information has been included in Section III of the revised paper.

### Comment # 2.3 Weaker baseline

One of the selected baseline x-vector/AHC [29] is relatively old. The authors
may consider the baselines used in the DIHARD-III or recent VoxSRC challenge
(speaker diarization track).

Thank you for the suggestion. The baselines in DIHARD-III include x-vector/AHC
with variational bayesian (VB). We therefore added this baseline to our
evaluation experiments on CALLHOME (see Table IV in the revised paper).

### Comment # 2.4 Speaker count issue

The proposed method is developed for more than two speakers but the user has
to select the specific model trained on the desired number of speakers. The
authors should unambiguously mention this issue somewhere otherwise the
present texts give a wrong impression that the proposed method can work for a
different number of target speakers without any human intervention.

Thank you for the comments. The proposed method can indeed handle
conversations with different numbers of speakers using a single model, which
is trained with training data containing different numbers of speakers,
without any human intervention.

A TS-VAD model can only be trained with training data containing a certain
number of speakers to handle conversations with this specific number of
speakers. In Table III, for a fair comparison with TS-VAD, we trained three
specific MTEAD* model for the 2-, 3-, and 4-speaker tasks. These models were
trained with the same training data containing a specific number of speaker as
the corresponding TS-VAD counterparts. We can see that MTEAD* outperforms TS-
VAD on all the 2-, 3-, and 4-speaker tasks. Since MTEAD was trained using
training data with different numbers of speakers, it can further outperform
the three MTEAD* models.

In Table IV, all models were evaluated on the CALLHOME test set with different
numbers of speakers.

In Table V, since most existing models were evaluated on the 2-speaker
CALLHOME task, we therefore compared MTEAD with these models under the same
setting.

We hope the above explanation clears up the misunderstanding.

### Comment # 2.5 Joint training

The authors should mention more details about joint training of speaker
embedding extractor module and detection module. Did the author train them
from scratch or the speaker embedding extractor was initialized on other data?
Is there any task scheduling during joint learning? How the losses are
combined?

Thank you for the suggestion. We added an additional subsection in Section II
of the revised paper (see Section II-D) to describe the details of the
training loss and joint training.

When using z-vectors, Extractor and Detector are jointly trained from scratch
using a binary cross-entropy loss between speaker labels and predicted SD
results. When using x-vectors and i-vectors, Extractor is replaced by other
extractors pretrained by the Kaldi recipe, and only Detector is trained using
the same binary cross-entropy loss.

We used the Noam scheduler. We added the following description to Section III
of the revised paper.

For model training, we used the Adam optimizer and Noam scheduler with a
learning rate of $0.1$ and a batch size of $8$. The dimensions of i-vector,
x-vector, and z-vector were all set to 40.

### Comment # 2.6 Conclusion

The conclusion needs to be improved with the limitations of the current work
and future work. The currently used datasets are also relatively clean which
questions the robustness of the proposed method for challenging data from
DIHARD or CHiME series. This can also be mentioned.

Thank you for the suggestion. We added the following sentences in the
conclusion section.

The experimental datasets used in this study are relatively clean. We will try
more challenging tasks such as DIHARD or CHiME in the future.


# ROSA: Random Subspace Adaptation for Efficient Fine-Tuning

Marawan Gamal Abdel Hameed  Aristides Milios  Siva Reddy  Guillaume Rabusseau

###### Abstract

Model training requires significantly more memory, compared with inference.
Parameter efficient fine-tuning (PEFT) methods provide a means of adapting
large models to downstream tasks using less memory. However, existing methods
such as adapters, prompt tuning or low-rank adaptation (LoRA) either introduce
latency overhead at inference time or achieve subpar downstream performance
compared with full fine-tuning. In this work we propose Random Subspace
Adaptation (ROSA), a method that outperforms previous PEFT methods by a
significant margin, while maintaining a zero latency overhead during inference
time. In contrast to previous methods, ROSA is able to adapt subspaces of
arbitrarily large dimension, better approximating full-finetuning. We
demonstrate both theoretically and experimentally that this makes ROSA
strictly more expressive than LoRA, without consuming additional memory during
runtime. As PEFT methods are especially useful in the natural language
processing domain, where models operate on scales that make full fine-tuning
very expensive, we evaluate ROSA in two common NLP scenarios: natural language
generation (NLG) and natural language understanding (NLU) with GPT-2 and
RoBERTa, respectively. We show that on almost every GLUE task ROSA outperforms
LoRA by a significant margin, while also outperforming LoRA on NLG tasks. Our
code is availble at github.com/rosa-paper/rosa

Machine Learning, ICML

##  1 Introduction

The advent of large language models pre-trained on web-size corpora (pre-
trained LMs, or PLMs) has led to remarkably performant models in the domain of
natural language processing (Brown et al., 2020; Devlin et al., 2019). As the
size of such models ranges from hundreds of millions to hundreds of billions
of parameters (Touvron et al., 2023), adapting them to downstream tasks is
challenging and computationally expensive (Peng et al., 2023). Compared to
inference, training requires substantially more memory (2x-4x as much). For
example, the LLAMA-7B model requires 14GB and 56GB during inference and
training respectively (Touvron et al., 2023).

To alleviate the burdensome memory requirements of adapting PLMs to downstream
tasks, various memory efficient methods have been proposed (Houlsby et al.,
2019; Lin et al., 2020; Guo et al., 2021; Hu et al., 2022; Li & Liang, 2021;
Lester et al., 2021; Sung et al., 2021; Liu et al., 2022b, a; Qiu et al.,
2023). The commonality among these methods is the maintenance of fixed PLM
weights while introducing a minimal quantity of trainable parameters. Although
solutions like LoRA (Hu et al., 2022) and (IA)3 (Liu et al., 2022a) are
effective and do not impose any additional inference latency, they implicitly
limit the expressivity of adapted models. For instance, LoRA adapts low-rank
matrices that are added in parallel to fixed pre-trained weights. While this
approach makes it possible to fine-tune large PLMs with reduced memory
footprint compared to full fine tuning, it introduces an unavoidable bias: the
pre-trained weight matrices can only be fine-tuned to matrices that are “a
low-rank matrix away” from the initial weights.

${\mathbf{W}}$$=$SVD$=$$+$${\mathbf{W}}_{\text{fixed}}$${\mathbf{A}}$${\mathbf{B}}$

$\mathbf{h}$${\mathbf{W}}_{\text{fixed}}$$\nabla_{({\mathbf{A}},{\mathbf{B}})}\mathcal{L}$
Factorize ${\mathbf{W}}={\mathbf{W}}_{\text{fixed}}+{\mathbf{A}}{\mathbf{B}}$
${\mathbf{B}}$${\mathbf{A}}$$\mathbf{x}$

Figure 1: Illustration of ROSA. Parameter matrix ${\mathbf{W}}$ is factorized
using the singular value decomposition (SVD) and split into smaller trainable
matrices $({\mathbf{A}},{\mathbf{B}})$ and a larger fixed matrix
$({\mathbf{W}}_{\text{fixed}})$. Gradients during back-propagation are only
computed with respect to $({\mathbf{A}},{\mathbf{B}})$. The split is then
merged after a specified number of training iterations, and the process is
repeated. ROSA updates an increasingly larger subspace of ${\mathbf{W}}$ over
the course of training while remaining memory efficient.

In this work, we propose ROSA: Random Subspace Adaptation, which expands the
expressivity of adapted models, while remaining as memory efficient as LoRA.
Similarly to LoRA, ROSA satisfies memory constraints by selectively fine-
tuning low-rank matrices in parallel to fixed pre-trained weight matrices.
Thus allowing users to fine-tune models in resource constrained settings. In
such settings, PEFT methods are the only viable alternatives as gradient
check-pointing and layer-wise training (loading/unloading one layer at a time
into GPUs) are prohibitively slow ( layer-wise training can be up to
$20\times$ slower than full fine-tuning or ROSA). However, PEFT methods such
as LoRA are limited in expressivity compared with approaches such as layer-
wise training and gradient check-pointing, which effectively simulate full
fine-tuning. ROSA alleviates this limitation by continuously _sampling_
different low-rank trainable sub-spaces and iteratively _merging_ learned
information into fixed weights throughout fine-tuning, as depicted in Figure
1. From a theoretical perspective, we formally characterize the implicit low
rank bias of LoRA, show how this bias can be detrimental even on a simple
regression task, and demonstrate that ROSA does not suffer from this
limitation (see Theorem 2). Even further, our results show that (i) ROSA can
fine-tune pre-trained weights to arbitrary target weights (i.e. is as
expressive as full fine-tuning), and (ii) while LoRA trades _expressivity_ for
lower memory requirements, ROSA instead trades _convergence speed_ with memory
usage. These results are clearly and intuitively illustrated on a simple
synthetic experiment presented in Section 4.1.

From a practical perspective, we show that ROSA achieves performance on par
with full fine-tuning and consistently outperforms state-of-the-art methods
such as LoRA and (IA)3 (Liu et al., 2022a) on natural language understanding
(GLUE) (Wang et al., 2019b) and natural language generation (E2E) (Novikova et
al., 2017) tasks by significant margins. Lastly, we note that ROSA carries a
significant advantage over approaches such as adapters (Houlsby et al., 2019)
and prompt tuning (Li & Liang, 2021; Lester et al., 2021), as it introduces no
additional latency overhead during inference time.

In summary, our key contributions are:

  * •

Demonstrating both empirically and theoretically, that the low rank nature of
LoRA can detrimentally limit its expressiveness.

  * •

Introducing ROSA, a PEFT method that circumvents the low rank limitation of
LoRA while remaining as memory efficient as LoRA.

  * •

Theoretically showing that ROSA is more expressive than LoRA and can be as
expressive as full fine-tuning.

  * •

Conducting extensive experiments showing that ROSA consistently outperforms
LoRA by a significant margin on natural language understanding (GLUE) and
natural language generation (E2E) benchmarks.

##  2 Related Work

PEFT defines a class of methods to alleviate memory and compute requirements
during adaptation of large models to downstream tasks by tuning only a
relatively small number of added parameters, rather than the tuning all the
parameters of the model itself.

#### Adapters:

Adapter methods such as (Houlsby et al., 2019) inject layers in between
certain modules of each transformer block. These layers have relatively few
parameters, as they project the original features down into a smaller set of
dimensions, then scale them back up after applying the adapter’s feed-forward
layer. This structure necessarily leads to a latency overhead.

#### Prompt and prefix tuning:

Prompt and prefix tuning are an efficient means of adapting models via
continuous optimization of prefixes added to input prompts (Li & Liang, 2021;
Lester et al., 2021; Liu et al., 2022b). While such approaches are memory
efficient, they require reserving a portion of the available sequence length
during downstream adaptation. Moreover, prompt tuning methods can be
challenging to optimize as pointed out in (Hu et al., 2022).

#### LoRA:

Our work is most similar to LoRA (Hu et al., 2022), which has been shown to
outperform the aforementioned approaches and to mitigate limitations such as
increased inference latency and reduced sequence length capacity. LoRA adds a
trainable low rank matrix to the frozen original weight matrix. The low rank
matrix, parameterized as the product of two small matrices, is then fine-tuned
instead of the original model weights. The authors of LoRA hypothesize that
during task-specific fine-tuning, the model weight updates have a low
“intrinsic dimension” and thus can be effectively approximated by low-rank
matrices. While this may be true for some downstream tasks, we show both
theoretically and empirically that this is not always the case and that
restricting the weights update to a low intrinsic dimension can be
detrimental.

####  (IA)3:

Another widely known PEFT method is (IA)3 (Liu et al., 2022a). (IA)3 (Infused
Adapter by Inhibiting and Amplifying Inner Activations) adds learned vectors
to the attention and feedforward layers of the transformer, which are used to
rescale the activations of these modules. (IA)3 further reduces the number of
trainable parameters from LoRA, and makes the proportion of trainable
parameters fixed, as the size of the rescaling vectors is directly dependent
on the dimensions of the transformer’s weight matrices.

#### AdaLoRA & LASER:

Another competitive approach, AdaLoRA (Zhang et al., 2023) allocates a fixed
parameter budget across the layers of a model dynamically by manipulating the
rank, providing a lower rank to less important modules and vice versa. This is
done via a novel importance metric that quantifies the contribution of a given
module to the model’s overall performance. In contrast, LASER (Sharma et al.,
2023) replaces select matrices with their low rank approximations using SVD.
Moreover LASER demonstrates that greedy search over this intervention space
can lead to improved model performance. The performance increase is
hypothesized to be due to the resultant select matrices better focusing on
important features. In connection to our work, this indicates that some of the
performance gain by using SVD in our rank reduction can be due to a similar
cause.

#### Other methods:

BitFit (Ben Zaken et al., 2022) freezes all parameters except bias terms. FISH
(Sung et al., 2021) optimizes a sparse difference vector to be summed with the
original model parameters.

##  3 Method

In this section we describe our proposed approach, ROSA: Random Subspace
Adaptation. The purpose of ROSA is to provide a means for fine-tuning large
models in memory constrained settings while remaining competitive with full
fine-tuning in terms of performance. After introducing ROSA and demonstrating
its memory efficiency in Section 3.1, we provide a theoretical analysis
showing that ROSA is provably more expressive than LoRA in Section 3.2.

###  3.1 ROSA

In LoRA (Hu et al., 2022), pre-trained models are adapted to alternative tasks
by adding a low rank matrix ${\mathbf{AB}}$, where
${\mathbf{A}}\in\mathbb{R}^{M\times R},{\mathbf{B}}\in\mathbb{R}^{R\times N}$,
in parallel to pre-trained weights ${\mathbf{W}}\in\mathbb{R}^{M\times N}$.
The output of the LoRA layer is given by

$\phi(\mathbf{x})={\mathbf{W}}\mathbf{x}+{\mathbf{A}}{\mathbf{B}}\mathbf{x}.$
(1)

The adapter weights ${\mathbf{A}}$ and ${\mathbf{B}}$ are initialized such
that ${\mathbf{A}}{\mathbf{B}}={\mathbf{0}}$ and are the only parameters being
updated during training. LoRA is memory efficient as ${\mathbf{W}}$ remains
fixed during training (no gradient buffers necessary for the full weights) and
typically $R\ll\textsc{min}(M,N)$. The rank ($R$) of the trainable matrices is
chosen such that the training procedure satisfies device memory constraints.

Constraining the updates to a _fixed_ low rank subspace _initialized at zero_
induces two limitations. First, the low rank nature of LoRA is such that the
difference between the fine-tuned weight matrix
${\mathbf{W}}+{\mathbf{A}}{\mathbf{B}}$ and the pre-trained weights
${\mathbf{W}}$ is constrained to be a low rank matrix. This significantly
hinders the ability of LoRA to fine-tune a given model to an arbitrary target
model/task. Note that even in the case where the target weights
${\mathbf{W}}^{*}$ are close to the pre-trained weights ${\mathbf{W}}$ (w.r.t.
e.g., the Frobenius norm), this low-rank constraint creates an unavoidable
bias when the difference ${\mathbf{W}}^{*}-{\mathbf{W}}$ is not low rank. We
formally characterize this bias in Section 3.2 and empirically demonstrate it
in Section 4.1. Second, initializing the adapter ${\mathbf{A}}{\mathbf{B}}$ to
zero can be thought of as learning new representations from scratch separately
from the pre-trained ones
($\phi(\mathbf{x})={\mathbf{W}}\mathbf{x}+{\mathbf{A}}{\mathbf{B}}\mathbf{x}:=\phi_{\text{pre-
trained}}(\mathbf{x})+\phi_{\text{trainable}}(\mathbf{x})$), rather than
leveraging the pre-trained features the model already has to initialize the
adapter.

To address these two limitations ROSA (i) continuously samples new weight
subspaces throughout the training procedure and (ii) initializes the trainable
sub-spaces using from the pre-trained weight matrices using SVD. The first
limitation is addressed as iteratively re-sampling new subspaces effectively
expands the dimension of the fine-tuned subspace. Hence, ROSA does not suffer
from the low rank bias of LoRA (which we theoretically show in Section 3.2).
Moreover, the second limitation is addressed as the initial trainable weights
are set to be a subset of a re-parameterization of the original weight matrix.

In more detail, ROSA adapters successively factorize ${\mathbf{W}}$ into
trainable and fixed weight subspaces using SVD. Let
${\mathbf{W}}={\mathbf{U}}{\mathbf{\Sigma}}{\mathbf{V}}^{\top}$ be the SVD of
${\mathbf{W}}$, let ${\mathbf{U}}_{R}\in\mathbb{R}^{M\times R}$ be the matrix
obtained by selecting a random subset of $R$ columns of ${\mathbf{U}}$ and let
${\mathbf{\Sigma}}_{R}\in\mathbb{R}^{R\times R}$ and
${\mathbf{{\mathbf{V}}}}_{R}\in\mathbb{R}^{R\times N}$ denote the matrices
obtained by selecting the same subset of singular values and right singular
vectors. The ROSA factorization step is defined by

${\mathbf{W}}={\mathbf{W}}_{\text{fixed}}+{\mathbf{A}}{\mathbf{B}}$ (2)

where

$\displaystyle{\mathbf{A}}={\mathbf{U}}_{R}{\mathbf{\Sigma}}_{R}\in\mathbb{R}^{M\times
R}$ $\displaystyle,\ {\mathbf{B}}={\mathbf{V}}_{R}\in\mathbb{R}^{R\times N},$
(3) $\displaystyle{\mathbf{W}}_{\text{fixed}}$
$\displaystyle={\mathbf{W}}-{\mathbf{A}}{\mathbf{B}}.$ (4)

During training, gradients are computed only with respect to the $R$
dimensional subspace which consists of $R(M+N)$ parameters. In contrast, full
fine-tuning requires optimizing $MN$ parameters. Thus, ROSA leads to a
reduction in the number of trainable parameters given by

$\rho_{\text{train}}=\frac{MN}{R(M+N)}.$ (5)

The factorization step is repeated throughout training at a pre-determined
frequency (e.g., after each epoch of fine-tuning). The overall ROSA procedure
is illustrated in Figure 1 and described in Algorithm 1. In practice, ROSA is
applied simultaneously to all weight matrices of the model to fine-tune. While
each subspace sampling step is expensive, $\mathcal{O}(\max(N,M)^{3})$, it is
only performed once every epoch. We show in the experiment section that the
sample step adds negligible time to the training procedure in practice (see
Table 1).

Algorithm 1 ROSA

0: ${\mathbf{W}}\in\mathbb{R}^{M\times N},$ $R\,\text{(desired rank)},$
$K\,\text{(factorization frequency)},$ $\mathcal{L}\,\text{(loss function)}$

1: [${\mathbf{A}},{\mathbf{B}}]\leftarrow[{\mathbf{0}},{\mathbf{0}}]$

2: for $t=1$ to $T$ do

3: if $t\hskip 1.0pt\text{mod}\hskip 1.0ptK==0$ then

4: ${\mathbf{W}}\leftarrow{\mathbf{W}}+{\mathbf{A}}{\mathbf{B}}$

5:
${\mathbf{U}},{\mathbf{\Sigma}},{\mathbf{V}}^{\top}\leftarrow\textsc{SVD}({\mathbf{W}})$

6: $(i_{1},\cdots,i_{R})\leftarrow\textsc{RandInts}(R,\min(M,N))$

7: $\Bigg{[}\begin{array}[]{c}{\mathbf{A}}\\\
{\mathbf{B}}\end{array}\Bigg{]}\leftarrow\Bigg{[}\begin{array}[]{c}{\mathbf{U}}_{:,(i_{1},\cdots,i_{R})}{\mathbf{\Sigma}}_{(i_{1},\cdots,i_{R}),(i_{1},\cdots,i_{R})},\\\
{\mathbf{V}}^{\top}_{(i_{1},\cdots,i_{R}),:}\end{array}\Bigg{]}$

8: ${\mathbf{W}}\leftarrow{\mathbf{W}}-{\mathbf{A}}{\mathbf{B}}$

9: end if

10:
$\left[{\mathbf{A}},{\mathbf{B}}\right]\leftarrow\left[{\mathbf{A}},{\mathbf{B}}\right]-\nabla_{[{\mathbf{A}},{\mathbf{B}}]}\mathcal{L}({\mathbf{W}}+{\mathbf{A}}{\mathbf{B}})$

11: end for

###  3.2 Theoretical Analysis

In this section we formally show how the low rank parameterization of LoRA
limits its expressiveness and how ROSA circumvents this limitation.

First, it is easy to see that, by construction, the residual matrices obtained
by fine-tuning weight matrices using LoRA are constrained to be low rank:

######  Proposition 1.

Let ${\mathbf{W}}_{0}$ be a weight matrix of a pre-trained model to be fine-
tuned. Then, any fine-tuned weight matrix ${\mathbf{W}}_{\text{LoRA}}$
obtained using LoRA with rank parameter $R$ will be such that
$\operatorname*{rank}({\mathbf{W}}_{0}-{\mathbf{W}}_{\text{LoRA}})\leq R$.

###### Proof.

This directly follows from the fact that
${\mathbf{W}}_{\text{LoRA}}={\mathbf{W}}_{0}+{\mathbf{A}}{\mathbf{B}}$ and
$\operatorname*{rank}({\mathbf{A}}{\mathbf{B}})\leq R$. ∎

As a consequence, fine-tuning using LoRA suffers an unavoidable estimation
bias which is not present in ROSA. In the following theorem, we (i) formally
characterize this bias on a simple multivariate linear regression fine-tuning
problem and (ii) derive a convergence rate of ROSA for linear regression
demonstrating that it does not suffer from the same limitation.

######  Theorem 1.

######  Theorem 2.

Consider a simple multivariate least-square regression problem:

$\operatorname*{arg\,min}_{{\mathbf{W}}}\|{\mathbf{X}}{\mathbf{W}}-{\mathbf{Y}}\|^{2}_{F}$

where ${\mathbf{X}}\in\mathbb{R}^{n\times d}$ and
${\mathbf{Y}}\in\mathbb{R}^{n\times p}$ are the input and output data
matrices, respectively. _We assume that there exists a solution achieving zero
error ***This is only to simplify the theorem’s statement. In the appendix we
show a more general version of this theorem without this assumption._

Consider the sequence of fine-tuned weight matrices obtained by ROSA with rank
parameter $R$ starting from a pre-trained weight matrix ${\mathbf{W}}_{0}$,
assuming that each intermediate minimization problem is solved exactly:

${\mathbf{W}}_{t}={\mathbf{W}}_{t-1}+{\mathbf{A}}_{t}{\mathbf{B}}_{t}$

where

${\mathbf{A}}_{t},{\mathbf{B}}_{t}=\\\
\operatorname*{arg\,min}_{{\mathbf{A}}\in\mathbb{R}^{n\times
R},{\mathbf{B}}\in\mathbb{R}^{R\times
n}}\|{\mathbf{X}}({\mathbf{W}}_{t-1}+{\mathbf{AB}})-{\mathbf{Y}}\|^{2}_{F}.$
(6)

Then, ROSA will converge to a fine-tuned matrix achieving zero error in at
most

$T=\left\lceil\frac{\operatorname*{rank}({\mathbf{XW}}_{0}-{\mathbf{Y}})}{R}\right\rceil$

steps. That is, $\|{\mathbf{X}}{\mathbf{W}}_{t}-{\mathbf{Y}}\|^{2}_{F}=0$ as
soon as $t\geq T$.

In contrast, the error achieved by LoRA with rank parameter $R$ is lower
bounded as

$\|{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}-{\mathbf{Y}}\|^{2}_{F}\geq\sum_{i=R+1}^{\min(d,p)}\sigma_{i}({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{XW}}_{0})^{2}$

where $\sigma_{i}(\mathbf{M})$ denotes the $i$th singular value of a matrix
${\mathbf{M}}$ (ordered decreasingly) and ${\mathbf{\Pi}}_{{\mathbf{X}}}$ is
the matrix of the orthogonal projection onto the range of ${\mathbf{X}}$.

###### Proof.

See Appendix A. ∎

Several observations are in order. First, Theorem 2 shows that even with rank
parameter $R=1$, ROSA will converge to the optimal solution of the linear
regression fine-tuning problem (assuming that ROSA exactly solves the
minimization problem between each factorization step). Second, increasing the
rank parameter will lead to faster convergence. This suggests that the rank
parameter $R$ in ROSA controls a trade-off between memory requirement and
convergence speed. This is in stark contrast to LoRA, where the rank parameter
controls the trade-off between memory requirement and _expressiveness_ , as
demonstrated in the previous theorem.

In the next section, we empirically demonstrate that this theoretical result
also holds in practice when using LoRA and ROSA to fine-tune non-linear models
trained using gradient based methods.

(a)

(b)

Figure 2: Validation loss curves of the training procedure for (a) 1-layer MLP and (b) 2-layer MLP (with ReLU activation). Both models are trained and evaluated on synthetic data that is generated from a randomly initialized MLP and a random low-rank adapter of rank 24. The models are trained to fit the synthetic data using the mean squared error loss function. This figure demonstrates that ROSA can find solutions with similar performance to full fine-tuning (FT) in practice.  ROSA/LoRAFT$0$$200$$400$$600$$800$$1{,}000$$1{,}200$$1{,}400$$1{,}600$$1{,}800$$2{,}000$MethodMemory (MB) Figure 3: Memory usage during fine-tuning of $\text{RoBERTa}_{\text{base}}$ on the CoLA GLUE benchmark task, using ROSA compared with LoRA and full fine-tuning. Figure 4: Analysing the trade-off between convergence rate and rank values for ROSA. In ROSA low rank values lead to a slower convergence rate. In contrast, LoRA models are limited by their rank. ROSA, LoRA and the baseline models all run in the same amount of time (approximately 155 seconds per epoch). Table 1: Runtime of one epoch of fine-tuning of $\text{RoBERTa}_{\text{base}}$ (125M parameters) on the CoLA task, using ROSA and full fine-tuning. The experiment is conducted on a single GPU (NVIDIA A100-SXM4) with an input batch of 32 sequences of length 512. At every epoch a ROSA factorize step is performed, which adds negligible latency to the runtime during training. | Factorize Time (s) | Epoch Time (s)
---|---|---
FT | - | $157_{\pm 0.16}$
LoRA | - | $152_{\pm 0.56}$
ROSA | $4.03_{\pm 3.67}$ | $153_{\pm 0.16}$

##  4 Experiments

In this section we compare the downstream performance of a model adapted using
ROSA compared with LoRA (Hu et al., 2022) and (IA)3 (Liu et al., 2022a) (as
all three methods add zero latency overhead at inference time). For better
comparison, in all experiments we use our own implementation for LoRA and
(IA)3 (detailed in Appendix B). In Section 4.1 we evaluate the performance of
MLP models adapted to synthetic data, while Sections 4.2 & 4.3 evaluate the
performance of $\text{RoBERTa}_{\text{base}}$ (Liu et al., 2019) and GPT-2
(Radford et al., 2019) on the GLUE and E2E benchmarks, respectively. In our
experiments using transformer models, we only apply the PEFT methods to the
attention layers, following a similar approach to Hu et al. (2022).

###  4.1 Synthetic Data

We first design two simple regression experiments with synthetic data to
validate that the increased expressiveness of ROSA, illustrated in Theorem 2
for linear models, indeed leads to better results when fine-tuning non-linear
models via Stochastic Gradient Descent.

To generate the synthetic data, we start by randomly initializing an MLP model
$f$. We then add low rank matrices (rank=24), which are also randomly
initialized, in parallel to the weights of $f$. This gives us the true model
$f^{*}$, which we want to approximate. The synthetic data
$\mathcal{D}=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\}^{n}_{i=1}$ is generated by
sampling $\mathbf{x}_{i}\sim\mathcal{N}(\mathbf{0},\sigma{\mathbf{I}})$ and
$\mathbf{y}_{i}=f^{*}(\mathbf{x}_{i})$.

The results are summarized in Figure 2, where we compare the evolution of the
validation loss of ROSA and LoRA for fine-tuning the original MLP model $f$ to
the data $\mathcal{D}$ generated from the target task $f^{*}$. As observed in
Figure 2, ROSA at different rank values finds solutions with similar
performance to full fine-tuning. This demonstrates that even in a more
practical setting than the one of Theorem 2(namely with non-linear models
trained by gradient descent) ROSA can match the performances of full fine-
tuning. Moreover, for both 1-layer and 2-layers MLPs, we see that the rank
limitation of LoRA prevents it from fully adapting to the target task: while
increasing the rank leads to better validation loss, the unavoidable low-rank
bias is clearly demonstrated by the convergence to a sub-optimal loss. In
contrast, ROSA always converges towards the optimal loss, even with rank
parameter set to 1, and increasing the rank parameter leads to faster
convergence to the optimal loss. Notably, using ROSA to adapt a two layer MLP
containing a non-linearity recovers a model that well approximates the true
model used to generate the data. This suggests that the formal result shown in
Theorem 2 holds beyond the simple linear regression setting.

###  4.2 GLUE Experiments

In this section we compare ROSA against LoRA and (IA)3, by adapting
$\text{RoBERTa}_{\text{base}}$ (125M) (Liu et al., 2019) on various tasks
taken from the GLUE and SuperGLUE natural language understanding benchmarks
(Wang et al., 2019b, a). These benchmarks cover a wide variety of natural
language understanding tasks, including logical entailment, grammatical
acceptability, question-answering, textual similarity, and sentiment analysis.
The pre-trained model weights for $\text{RoBERTa}_{\text{base}}$ are taken
from the Huggingface library (Wolf et al., 2020). A description of the
specific subset of GLUE and SuperGLUE tasks tested on is available in Appendix
B. Unlike some previous works which initialize the weights for MRPC, RTE, and
STS-B with fine-tuned MNLI task-specific weights, we initialize the weights
for all tasks with only the pre-trained RoBERTa weights for a fair comparison.

These tasks were selected to give a broad overview of ROSA’s performance
across a variety of different natural language tasks. We report development
set performance for all tasks.

In Table 2 we see that ROSA outperforms LoRA and (IA)3 by a significant margin
on multiple tasks. Most notably, on CoLA using rank equal to eight we obtain
Matthew’s correlation coefficients of 64.80 for ROSA, 54.27 for LoRA and 55.18
for (IA)3. Furthermore, ROSA remains as memory efficient as LoRA (Figure 3),
and the factorization steps in ROSA add negligible latency (Table 1). We
provide training curves of $\text{RoBERTa}_{\text{base}}$ fine-tuned on CoLA
for 10 epochs in the Appendix.

Table 2: Performance of $\text{RoBERTa}_{\text{base}}$ fine-tuned using various PEFT methods (ROSA, LoRA and (IA)3) on the GLUE benchmark tasks.We report the matched validation accuracy for MNLI, Matthew’s correlation coefficient for CoLA, Pearson correlation for STS-B and accuracy for all other tasks.(Params denotes the number of trainable parameters)  Method | Params (M) | CoLA | MRPC | QNLI | RTE | STS-B | MNLI | SST2 | BoolQ
---|---|---|---|---|---|---|---|---|---
FT | 125 | 63.50 | 89.95 | 92.60 | 77.62 | 90.69 | 86.296 | 94.38 | 82.11
(IA)3 | 0.1 | 55.18 | 87.50 | 82.52 | 68.59 | 88.41 | 77.06 | 92.55 | 73.85
LoRA (r=2) | 0.2 | 53.42 | 88.72 | 92.10 | 72.56 | 83.91 | 85.37 | 93.69 | 72.96
ROSA (r=2) | 0.2 | 62.08 | 89.46 | 92.20 | 73.64 | 89.32 | 86.25 | 92.55 | 76.78
LoRA (r=8) | 0.6 | 54.27 | 88.24 | 92.20 | 69.31 | 82.10 | 85.88 | 93.57 | 68.37
ROSA (r=8) | 0.6 | 64.80 | 88.73 | 92.80 | 72.56 | 90.11 | 87.09 | 93.11 | 77.31

(a)

(b)

Figure 5: Plot of the cumulative sum of singular values of residual matrices.
The residual matrices in these plots are obtained from fine-tuning
$\text{RoBERTa}_{\text{base}}$ on CoLA for 10 epochs and achieving a Matthew’s
Correlation score of (a) $64.80$ with ROSA and (b) $54.27$ with LoRA. The
figure demonstrates that the rank of the residual matrices obtained when
adapting the $\text{RoBERTa}_{\text{base}}$ using ROSA, is indeed far greater
than the rank of the residuals obtained when fine-tuning using LoRA.

###  4.3 NLG Experiments

In this section we investigate the performance of ROSA in the natural language
generation (NLG) setting. Namely, we compare the performance of GPT-2 (Radford
et al., 2019) using ROSA compared with LoRA and (IA)3, when finetuned on the
E2E NLG task (Novikova et al., 2017). The E2E NLG task involves producing a
fluent natural language description of a restaurant given a logical form
describing its various attributes. The model’s generations are compared
against multiple reference texts to account for variations in wording. The
score provided is the maximum BLEU score across all the reference texts for a
given input.

In Table 3 we see that ROSA outperforms LoRA and (IA)3 by a significant margin
in the BLEU score.

Table 3: Performance of GPT-2 finetuned on the End-to-End natural language generation task (E2E) (Novikova et al., 2017), using ROSA, LoRA and (IA)3. Name | # Trainable | BLEU
---|---|---
| Parameters (M) |
FT | 355 | 68
(IA)3 | 0.2 | 65
LoRA (r=4) | 0.9 | 64
ROSA (r=4) | 0.9 | 68
LoRA (r=8) | 1.7 | 67
ROSA (r=8) | 1.7 | 67
Table 4: Performance of $\text{RoBERTa}_{\text{base}}$ using variants of $\text{ROSA}_{\text{ r=8 }}$, finetuned on the CoLA and STS-B GLUE benchmark tasks. Name | CoLA | STS-B
---|---|---
FT | 63.52 | 90.69
LoRA | 54.27 | 82.10
$\text{ROSA}_{\text{ SVD Init}}$ | 57.08 | 89.19
$\text{ROSA}_{\text{ SVD Init + Factorize }}$ | 60.32 | 89.42
$\text{ROSA}_{\text{ SVD Init + Factorize + Resample }}$ | 64.80 | 90.11

###  4.4 Which components of ROSA lead to its performance?

In this section we empirically study several aspects of ROSA. We highlight
three key components of ROSA, on which we perform ablation studies. The key
components of ROSA are:

  * •

SVD Initialization: ROSA adapters are initialized using SVD, as opposed to
LoRA’s adapters which are initialized to zero.

  * •

Factorization: In ROSA, pre-trained weight matrices are factorized into fixed
and trainable portions such that the resultant transformation remains
unchanged when the adapters are initialized.

  * •

Resampling: In ROSA the difference between the pre-trained weights and the
final weights is not constrained to be low-rank, due to resampling and merging
of subpsaces throughout training.

We study the effects of progressively adding these components to ROSA in Table
4. Specifically, we study the following ablations:

  * •

$\textbf{ROSA}_{\textbf{ SVD Init}}$: Each matrix ${\mathbf{W}}$ in a network
is replaced at the begining of training by ${\mathbf{W}}+{\mathbf{AB}}$ where
${\mathbf{AB}}={\mathbf{U_{R}\Sigma_{R}V_{R}}}$ is a low rank matrix
constructed using a subset of the SVD of ${\mathbf{W}}$.

  * •

$\textbf{ROSA}_{\textbf{ SVD Init + Factorize}}$: Each matrix is instead
replaced by ${\mathbf{W}}_{\text{fixed}}+{\mathbf{AB}}$ where
${\mathbf{W}}_{\text{fixed}}={\mathbf{W}}-{\mathbf{AB}}$.

  * •

$\textbf{ROSA}_{\textbf{ SVD Init + Factorize + Resampling}}$ (proposed
method): Resampling the random subspace every epoch (i.e., merging the adapter
and weights and re-factorizing using SVD).

We also compare with LoRA in Table 4 which differs from $\text{ROSA}_{\text{
SVD Init}}$ by initializing the adapters to zero.

We find that the progressive addition of the aforementioned components to ROSA
is beneficial to its performance. The most drastic improvement is seen when
resampling is added. This observation aligns with our theoretical analysis,
demonstrating that the rank flexibility of ROSA, which increases the
expressiveness of adapted models, leads to improved performance. Note also
that the gain from LoRA to $\text{ROSA}_{\text{SVD Init}}$ aligns with our
intuition that leveraging the pre-trained features to initialize the adapter
is beneficial, compared to initializing the adapter to zero and learning new
features from scratch. Finally, the increase in performance acheived by the
intermediate $\text{ROSA}_{\text{SVD Init + Factorize}}$ compared with
$\text{ROSA}_{\text{SVD Init}}$, is likely due to mitigating fluctuations in
the training process, as the resulting transformation remains unchanged after
the factorization.

We further investigate the rank structure of the matrices of the most
performant $\text{RoBERTA}_{\text{base}}$ that achieved 64.80 on CoLA (with
resampling). Specifically, we plot the singular values of residual matrices
(defined as the difference between the initial pre-trained weights and the
final weights upon completion of fine-tuning) in Figure 5. As shown in the
figure, the ranks of the difference matrices achieved using ROSA (Figure 5a)
are significantly larger than than the ranks of the difference matrices
achieved using LoRA (Figure 5b).

Table 5: Performance of $\text{RoBERTa}_{\text{base}}$ using different sampling schemes for ROSA, on the CoLA GLUE benchmark task. Top/Bottom/Random sampling indicate the method used for selecting the singular vectors to initialize the trainable subspace.  Name | # Trainable Params (M) | MRPC
---|---|---
FT | 125 | 63.52
$\text{ROSA}_{\text{ r=8 }}(\text{Top})$ | 0.6 | 58.88
$\text{ROSA}_{\text{ r=8 }}(\text{Bottom})$ | 0.6 | 60.57
$\text{ROSA}_{\text{ r=8 }}(\text{Random})$ | 0.6 | 60.32

###  4.5 Investigating different low-rank subspace sampling schemes

In this section, we compare the random subspace sampling of ROSA with two
other subspace selection strategies: selecting the top-$R$ or bottom-$R$
singular vectors . In doing so, we validate that performing random selection
of singular vectors is as performant as selection based on singular value
information. In Table 5, we report the performance of
$\text{RoBERTa}_{\text{base}}$ fine-tuned on CoLA using the different sampling
strategies, which confirm that, on this task, random sampling performs
similarly or better than other schemes.

###  4.6 Limitations of ROSA

While ROSA achieves better performance than previous state-of-the-art
adaptation methods such as LoRA and (IA)3, it bears one main limitation
compared with other methods. Namely, it requires storage of the whole model
after it is adapted for a downstream task.

Other adapter methods try to simultaneously address two challenges (1)
reducing memory usage during training to ease the hardware barrier when
adapting large models to a _single_ downstream task and (2) reducing disk
space usage when adapting a base model to _many_ downstream tasks.

ROSA primarily focuses on addressing point (1), making it more suitable for
scenarios involving a single downstream task. In comparison, other PEFT
methods are better suited for scenarios involving multiple downstream tasks.
ROSA excels in its specific domain, offering the same level of expressivity as
full fine-tuning while requiring less GPU memory. This eliminates the need for
(1) layerwise training, which would prolong training time, and (2) model
sharding that necessitates more GPUs, thereby increasing training costs.

###  4.7 Conclusion & Future Work

In this work we introduced ROSA: Random Subspace Adaptation. We first showed
both theoretically and empirically that the low-rank nature of LoRA can often
detrimentally affect its performance. In contrast, we demonstrate that ROSA
can theoretically achieve the same solution as full fine-tuning. Furthermore,
we demonstrate that on synthetic data ROSA indeed converges the same solution
as full fine-tuning when using gradient based optimization. We evaluated ROSA
against LoRA and (IA)3 on both natural language understanding and natural
language generation tasks. Our experiments showed that ROSA achieved
performance similar to full fine-tuning and outperformed other state-of-the-
art methods such as LoRA and (IA)3 by significant margins. As our analysis was
limited to adapting linear layers present in transformer models, adapting the
parameters of convolution operations is an area for future work.

###  4.8 Impact Statement

The primary goal of this work is to advance the field of Machine Learning.
There are many potential societal consequences of our work: democratizing
access to fine-tuning of large language models can have both positive and
negative societal consequences, none which we feel must be specifically
highlighted here.

## References

  * Ben Zaken et al. (2022) Ben Zaken, E., Goldberg, Y., and Ravfogel, S.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , 2022.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2019.
  * Guo et al. (2021) Guo, D., Rush, A., and Kim, Y.  Parameter-efficient transfer learning with diff pruning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , 2021.
  * Houlsby et al. (2019) Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning (ICML)_ , 2019.
  * Hu et al. (2022) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.
  * Izenman (1975) Izenman, A. J.  Reduced-rank regression for the multivariate linear model.  _Journal of multivariate analysis_ , 5(2):248–264, 1975.
  * Lester et al. (2021) Lester, B., Al-Rfou, R., and Constant, N.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021.
  * Li & Liang (2021) Li, X. L. and Liang, P.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , 2021.
  * Lin et al. (2020) Lin, Z., Madotto, A., and Fung, P.  Exploring versatile generative language model via parameter-efficient transfer learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020.
  * Liu et al. (2022a) Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _arXiv preprint arXiv:2205.05638_ , 2022a.
  * Liu et al. (2022b) Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., and Tang, J.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , 2022b.
  * Liu et al. (2019) Liu, Y. et al.  Roberta: A robustly optimized bert pretraining approach.  In _arxiv_ , 2019.
  * Novikova et al. (2017) Novikova, J., Du$\mathbf{v}$sek, O., and Rieser, V.  The E2E dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , 2017.
  * Peng et al. (2023) Peng, B., Li, C., He, P., Galley, M., and Gao, J.  Instruction tuning with gpt-4.  _arXiv preprint arXiv:2304.03277_ , 2023.
  * Qiu et al. (2023) Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Schölkopf, B.  Controlling text-to-image diffusion by orthogonal finetuning.  In _NeurIPS_ , 2023.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 2019.
  * Sharma et al. (2023) Sharma, P., Ash, J. T., and Misra, D.  The truth is in there: Improving reasoning in language models with layer-selective rank reduction.  _arXiv preprint arXiv:2312.13558_ , 2023.
  * Sung et al. (2021) Sung, Y.-L., Nair, V., and Raffel, C. A.  Training neural networks with fixed sparse masks.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2019a) Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems_ , 2019a.
  * Wang et al. (2019b) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  Glue.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , 2019b.
  * Wolf et al. (2020) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , 2020.
  * Zhang et al. (2023) Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T.  Adaptive budget allocation for parameter-efficient fine-tuning.  In _The Eleventh International Conference on Learning Representations_ , 2023.

## Appendix

##  5 Theorem Proofs

You may include other additional sections here.

######  Theorem 1.

Consider a simple multivariate least-square regression problem:

$\operatorname*{arg\,min}_{{\mathbf{W}}}\|{\mathbf{X}}{\mathbf{W}}-{\mathbf{Y}}\|^{2}_{F}$

where ${\mathbf{X}}\in\mathbb{R}^{n\times d}$ and
${\mathbf{Y}}\in\mathbb{R}^{n\times p}$ are the input and output data
matrices, respectively.

Consider the sequence of fine-tuned weight matrices obtained by ROSA with rank
parameter $R$ starting from a pre-trained weight matrix ${\mathbf{W}}_{0}$,
assuming that each intermediate minimization problem is solved exactly:

${\mathbf{W}}_{t}={\mathbf{W}}_{t-1}+{\mathbf{A}}_{t}{\mathbf{B}}_{t}\ \ \ $

where

${\mathbf{A}}_{t},{\mathbf{B}}_{t}=\text{armgin}_{{\mathbf{A}}\in\mathbb{R}^{n\times
R},{\mathbf{B}}\in\mathbb{R}^{R\times
n}}\|{\mathbf{X}}({\mathbf{W}}_{t-1}+{\mathbf{AB}})-{\mathbf{Y}}\|^{2}_{F}.$

Then, ROSA will converge to a fine-tuned matrix achieving the optimal error in
at most

$T=\bigg{\lceil}\frac{\operatorname*{rank}({\mathbf{XW}}_{0}-{\mathbf{Y}})}{R}\bigg{\rceil}$

steps. That is,
$\|{\mathbf{X}}{\mathbf{W}}_{t}-{\mathbf{Y}}\|^{2}_{F}=\|{\mathbf{\Pi}}_{XY}-{\mathbf{Y}}\|^{2}_{F}$
as soon as $t\geq T$, where ${\mathbf{\Pi}}_{{\mathbf{X}}}$ is the matrix of
the orthogonal projection onto the range of ${\mathbf{X}}$.

In contrast, the error achieved by LoRA with rank parameter $R$ is lower
bounded as

$\|{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}-{\mathbf{Y}}\|^{2}_{F}\geq\sum_{i=R+1}^{\min(d,p)}\sigma_{i}({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{XW}}_{0})^{2}$

where $\sigma_{i}$ denotes the $i$th singular value (ordered decreasingly).

###### Proof.

First observe that the minimization problem optimized by LoRA,

$\operatorname*{arg\,min}_{{\mathbf{A}},{\mathbf{B}}}\|{\mathbf{X}}({\mathbf{W}}_{0}+{\mathbf{AB}})-{\mathbf{Y}}\|^{2}_{F}$

is an instance of the _Reduced Rank Regression_ problem (Izenman, 1975)

$\operatorname*{arg\,min}_{{\mathbf{A}}\in\mathbb{R}^{n\times
R},{\mathbf{B}}\in\mathbb{R}^{R\times
n}}\|{\mathbf{X}}{\mathbf{AB}}-({\mathbf{Y}}-{\mathbf{X}}{\mathbf{W}}_{0})\|^{2}_{F}$

whose optimal solution satisfies

${\mathbf{A}}{\mathbf{B}}=(({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}{\mathbf{Y}}-{\mathbf{W}}_{0})\sum_{i=1}^{R}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}$
(7)

where the $\mathbf{v}_{i}$’s are the first $R$ right singular vectors of the
matrix $({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{XW}}_{0})$ and
${\mathbf{\Pi}}_{{\mathbf{X}}}={\mathbf{X}}({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}^{\top}$.
The cost of the solution computed by LoRA can thus be lower bounded by

$\displaystyle\|$
$\displaystyle{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}-{\mathbf{Y}}\|^{2}_{F}$
$\displaystyle=\|{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}-{\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}\|^{2}_{F}+\|{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}\
-{\mathbf{Y}}\|^{2}_{F}$
$\displaystyle\geq\|{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}-{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}\|^{2}_{F}$
$\displaystyle\geq\Bigg{\|}{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}$
$\displaystyle\qquad-{\mathbf{X}}\bigg{(}{\mathbf{W}}_{0}+(({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}{\mathbf{Y}}-{\mathbf{W}}_{0})\sum_{i=1}^{R}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}\bigg{)}\Bigg{\|}^{2}_{F}$
$\displaystyle=\|({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{XW}}_{0})({\mathbf{I}}-\sum_{i=1}^{R}\mathbf{v}_{i}\mathbf{v}_{i}^{\top})\|^{2}_{F}$
$\displaystyle=\sum_{i=R+1}^{\min(d,p)}\sigma_{i}({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{XW}}_{0})^{2}$

$\displaystyle\|$
$\displaystyle{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}-{\mathbf{Y}}\|^{2}_{F}$
$\displaystyle=\|{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}-{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}\|^{2}_{F}+\|{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}\
-{\mathbf{Y}}\|^{2}_{F}$
$\displaystyle\geq\|{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}-{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}\|^{2}_{F}$
$\displaystyle\geq\|{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}-{\mathbf{X}}\bigg{(}{\mathbf{W}}_{0}+(({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}{\mathbf{Y}}-{\mathbf{W}}_{0})\sum_{i=1}^{R}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}\bigg{)}\|^{2}_{F}$
$\displaystyle=\|({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{XW}}_{0})({\mathbf{I}}-\sum_{i=1}^{R}\mathbf{v}_{i}\mathbf{v}_{i}^{\top})\|^{2}_{F}$
$\displaystyle=\sum_{i=R+1}^{\min(d,p)}\sigma_{i}({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{XW}}_{0})^{2}$

where we used the fact that
$\langle{\mathbf{X}}{\mathbf{W}}_{\text{LoRA}}-{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}},{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{Y}}\
-{\mathbf{Y}}\rangle=0$ for the first equality, and the fact that
${\mathbf{W}}_{\text{LoRA}}$ is of the form
${\mathbf{W}}_{0}+{\mathbf{A}}{\mathbf{B}}$ for the second inequality. This
shows the second part of the theorem.

For the first part of the theorem, first observe that
${\mathbf{W}}_{1}={\mathbf{W}}_{0}+{\mathbf{A}}{\mathbf{B}}$ where
${\mathbf{A}}{\mathbf{B}}$ is the solution of the reduced rank regression
problem defined in Eq. (7). Similarly, one can show that the solution for the
second step of ROSA is given by

${\mathbf{W}}_{2}={\mathbf{W}}_{1}+(({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}^{\top}{\mathbf{X}}-{\mathbf{W}}_{1})\sum_{i=1}^{R}{\tilde{\mathbf{v}}_{i}\tilde{\mathbf{v}}_{i}^{\top}}$

where the $\tilde{\mathbf{v}}_{i}$ are the first $R$ right singular vectors of
the matrix
$({\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}{\mathbf{W}}_{1})$.
However, we have

$\displaystyle{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}{\mathbf{W}}_{1}$
$\displaystyle={\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}\bigg{(}{\mathbf{W}}_{0}+(({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}{\mathbf{Y}}-{\mathbf{W}}_{0})\sum_{i=1}^{R}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}\bigg{)}$
$\displaystyle=({\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}{\mathbf{W}}_{0})-({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{X}}{\mathbf{W}}_{0})\sum_{i=1}^{R}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}$
$\displaystyle=({\mathbf{\Pi}}_{{\mathbf{X}}{\mathbf{Y}}}-{\mathbf{X}}{\mathbf{W}}_{0})\sum_{i={R+1}}^{\min(d,p)}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}$

Hence the top $R$ right singular vectors of
$({\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}{\mathbf{W}}_{1})$ are
equal to the right singular vectors
$\mathbf{v}_{R+1},\mathbf{v}_{R+2},\cdots,\mathbf{v}_{2R}$ of the matrix
$({\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}{\mathbf{W}}_{0})$.

It follows, by recurrence, that

$\displaystyle{\mathbf{W}}_{t}$
$\displaystyle={\mathbf{W}}_{t-1}+(({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}^{\top}{\mathbf{X}}-{\mathbf{W}}_{t-1})$
$\displaystyle\qquad\qquad\sum_{i=(t-1)R+1}^{tR}{\mathbf{v}}_{i}{\mathbf{v}}_{i}^{\top}$
$\displaystyle={\mathbf{W}}_{t-1}({\mathbf{I}}-\sum_{i=(t-1)R+1}^{tR}{\mathbf{v}}_{i}{\mathbf{v}}_{i}^{\top})$
$\displaystyle\qquad\qquad+({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}^{\top}{\mathbf{X}}\sum_{i=(t-1)R+1}^{tR}{\mathbf{v}}_{i}{\mathbf{v}}_{i}^{\top}$
$\displaystyle={\mathbf{W}}_{0}({\mathbf{I}}-\sum_{i=1}^{tR}\mathbf{v}_{i}\mathbf{v}_{i}^{\top})+({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}^{\top}{\mathbf{X}}\sum_{i=1}^{tR}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}$
$\displaystyle={\mathbf{W}}_{0}+(({\mathbf{X}}^{\top}{\mathbf{X}})^{-1}{\mathbf{X}}^{\top}{\mathbf{X}}-{\mathbf{W}}_{0})\sum_{i=1}^{tR}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}$

Hence, as soon as
$t>\bigg{\lceil}\frac{\operatorname*{rank}({\mathbf{XW}}_{0}-{\mathbf{Y}})}{R}\bigg{\rceil}$,
we have

${\mathbf{X}}{\mathbf{W}}_{t}={\mathbf{X}}{\mathbf{W}}_{0}+({\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}{\mathbf{W}}_{0})\sum_{i=1}^{tR}\mathbf{v}_{i}\mathbf{v}_{i}^{\top}={\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}$

hence
$\|{\mathbf{X}}{\mathbf{W}}_{t}-{\mathbf{X}}\|=\|{\mathbf{\Pi}}_{{\mathbf{X}}}{\mathbf{X}}-{\mathbf{X}}\|$,
which concludes the proof. ∎

##  6 Experimental setup

###  6.1 Implementation of ROSA, LoRA and (IA)3

For better comparison, we re-implement the LoRA and (IA)3 PEFT to share the
same structure as ROSA. For all methods we use a vanilla implementation to
focus on only comparing their core aspects. We list out the key differences
between our implementation and those of LoRA (Hu et al., 2022) and (IA)3 (Liu
et al., 2022a).

  * •

We apply adapters to all attention matrices. In contrast, LoRA tunes the
attention matrices to which its adapter should be applied.

  * •

We do not add additional dropout modules inside the adapter, as is done in the
LoRA paper.

  * •

We do not apply adapters to MLP layers as done so in (IA)3.

  * •

We use the same number of epochs across all model types (full fine-tuning,
adapter). In contrast, LoRA and (IA)3 experiments are typically run for far
more epochs (roughly 3X) than the full fine-tuning experiments.

###  6.2 GLUE Experiments

For each experiment on GLUE we tune the LR for all three PEFT models for each
selection of rank. Specifically, for a given task, model, PEFT method and rank
value, we report the model that obtains the best validation set accuracy using
LRs in $\\{{2}e{-2},{2}e{-3},{2}e{-4},{2}e{-5}\\}$. We use a factorization
frequency of $2$ in ROSA for all GLUE experiments (i.e., we merge then
factorize the weight matrices once every two epochs.) We use the AdamW
optimizer with $\beta_{1},\beta_{2}=(0.9,0.98),\;\epsilon=1e-6$ and weight
decay of $0.1$. Our batch size is selected from the set $\\{16,32\\}$ and we
use a sequence length of $512$. We train all models for 10 epochs.

Below is a description of each of the GLUE/SuperGLUE tasks selected for
evaluation:

  1. 1.

CoLA: a single-sentence classification task, where each sentence is labelled
as either grammatical or not in English. The Matthews correlation coefficient
is the reported metric.

  2. 2.

MRPC: a sentence-pair classification task, where each pair of sentences is
labelled as either semantically equivalent (i.e. paraphrases of each other),
or not.

  3. 3.

QNLI: QNLI is a sentence-pair classification task, where each pair of
sentences corresponds to a paragraph from Wikipedia and a question, and the
model must predict if the answer to the question is contained within the
paragraph.

  4. 4.

RTE: The input for RTE is a pair of sentences, where the model must predict if
the second sentence can logically be inferred from the first, or if it
contradicts/is unrelated to it (binary classification).

  5. 5.

STS-B: The only regression task in the GLUE Benchmark, for STS-B the model
must predict the similarity between a pair of sentences on a continuous scale
of 1 to 5. The reported metric is the Pearson correlation coefficient.

  6. 6.

MNLI: The input for MNLI is a premise and a hypothesis, and the model must
predict if the premise entails, contradicts, or is neutral toward the
hypothesis. This is the same as RTE but with a separate neutral (unrelated)
class. We report accuracy for the in-domain (matched) set.

  7. 7.

SST2: SST2 is a binary (positive/negative) sentiment classification dataset of
movie reviews.

  8. 8.

BoolQ: BoolQ is a question-answering task where the input is a Wikipedia
paragraph and a yes/no question where the answer is contained within the
Wikipedia paragraph. The model must predict the answer to the question.

  9. 9.

WiC (Words-in-Context): WiC is a binary sentence-pair classification task of
disambiguating word senses. Two sentences are provided to the model that
contain the same word,, and the model must predict if the same sense of the
word is used in both cases.

###  6.3 E2E NLG Experiments

The E2E experiments are all carried out for 5 epochs. We also tune the LR in
for each PEFT model by searching the space
$\\{{2}e{-2},{2}e{-3},{2}e{-4},{2}e{-5}\\}$. We use the AdamW optimizer with
$\beta_{1},\beta_{2}=(0.9,0.999),\;\epsilon=1e-8$, weight decay of $0.1$,
batch size of $10$ and a sequence length of $512$.

An example input and output for E2E is provided.

Input: name[The Vaults], eatType[pub], priceRange[more than £30], customer
rating[5 out of 5], near[Café Adriatic]

Output: _“The Vaults pub near Café Adriatic has a 5 star rating. Prices start
at £30.”_

###  6.4 Training Curves For GLUE Experiments

In Figure 6, we plot the training loss and validation curves for the fine-
tuning of $\text{RoBERTa}_{\text{base}}$ on CoLA for 10 epochs.

(a)

(b)

Figure 6:  Plots of (a) Train loss and (b) and Validation Matthew’s
Correlation Coefficient. The plots are obtained from fine-tuning
$\text{RoBERTa}_{\text{base}}$ on CoLA for 10 epochs.