Tahiya Chowdhury1, Veronica Romero1,2, Amanda Stent1

# Parameter Selection for Analyzing Conversations with Autism Spectrum
Disorder

###### Abstract

The diagnosis of autism spectrum disorder (ASD) is a complex, challenging task
as it depends on the analysis of interactional behaviors by psychologists
rather than the use of biochemical diagnostics. In this paper, we present a
modeling approach to ASD diagnosis by analyzing acoustic/prosodic and
linguistic features extracted from diagnostic conversations between a
psychologist and children who either are typically developing (TD) or have
ASD. We compare the contributions of different features across a range of
conversation tasks. We focus on finding a minimal set of parameters that
characterize conversational behaviors of children with ASD. Because ASD is
diagnosed through conversational interaction, in addition to analyzing the
behavior of the children, we also investigate whether the psychologist's
conversational behaviors vary across diagnostic groups. Our results can
facilitate fine-grained analysis of conversation data for children with ASD to
support diagnosis and intervention.

Index Terms: developmental disorder, autism, conversation, children's speech.

##  1 Introduction

Autism spectrum disorder (ASD) refers to a range of developmental disabilities
characterized by deficits in social communication and interaction. In the
absence of therapeutic intervention, adolescents with ASD exhibit impairments
in social interaction throughout their lives [1], including an inability to
display or reciprocate verbal aspects of communication appropriately and, in
turn, successfully maintain effective social interactions [2, 3, 4, 5]. The
diagnosis of ASD is a complex, challenging task as it depends on behavioral
symptoms identified by psychologists with no reliable biochemical diagnostic
tests available. Existing diagnostic instruments, such as the Autism
Diagnostic Observation Schedule (ADOS) [6], rely on qualitative coding by
expert assessors for the presence or absence of certain behavioral markers
across multiple structured conversational scenarios. The assessor has to
simultaneously engage the child in conversation, monitor their own
conversational behavior, and make diagnostic notes. AI-based techniques have
the potential to reduce the cognitive load on the assessor by providing
objective measurements of conversational behaviors and thus to augment the
assessor's workflow.

Previous research has explored the efficacy of ML tools using speech and
language features for identifying atypical and behavioral signals of ASD in
communication from conversation data [7, 8, 9]. Acoustic-prosodic features
such as pitch [10], intonation, and rhythm [11] derived from diagnostic
conversations have been found useful for identifying ASD individuals. Language
features such as word usage [12], social and cognitive linguistic word counts
[13], and semantic similarity [14] have also been investigated and found to
contain behavioral markers helpful to distinguish between ASD and TD children.
While these findings are promising, prior work uses conversation samples from
a limited set of diagnostic tasks and may not generalize to different
conversational contexts.

Prior research has also explored the use of information from the
conversational partner's interactions to predict diagnosis outcome. The
conversational partner's acoustic-prosodic cues have been found to be
predictive of ASD symptom severity [15] and engagement levels [16]. [13] found
a significant correlation between the conversational partner's language use
and ASD severity score.  [17] used social context in conversations to explore
its relationship with symptoms of autism. However, these findings also come
from a limited set of diagnostic tasks.

In this work, we aim to answer the following questions:

  * •

What are the most informative acoustic-prosodic indicators of children with
ASD in different conversational contexts?

  * •

What are the most informative linguistic indicators of children with ASD in
different conversational contexts?

  * •

Can we classify ASD and TD children using a minimal set of characteristics of
the child's spoken language?

  * •

Can we classify ASD and TD children using a minimal set of characteristics of
the interlocutor's spoken language?

##  2 Data

We used data collected during sessions of the Autism Diagnostic Observation
Schedule - Second Edition (ADOS-2), an assessment tool used to categorize ASD
impairment [6]. In this assessment, a child and a certified adult assessor
(usually a psychologist) engage in a sequence of semi-structured activities to
assess behavioral patterns of the child. Our data includes conversations
spanning 14 different subtasks from Module 3 of the ADOS-2, which is designed
for verbally fluent children and adolescents (Table 1). Depending on the
subtask, the child may be asked to tell a story, play with toys, act out a
cartoon, or discuss topics such as emotion, loneliness, and friends. Because
we are interested in different conversational contexts, we included all 14
tasks from the module, which are listed in Table 3.

Our data included 29 children (14 ASD, 15 TD) whose age ranged between 10-15
years. Each ADOS-2 assessment session lasted on average 40-60 minutes; all
sessions were administered by the same psychologist. For this work, we first
prepared a separate recording of each subtask using annotations done by a
research assistant from the video recordings of the sessions. The average
length of these videos is 5 minutes. We used pyannote [18] to perform speaker
diarization. We then used PyDub [19] to partition the subtask audio into
shorter segments based on speaker identity and turn boundaries derived using
pyannote. This allowed us to automatically separate the child and psychologist
turns into separate audio recordings, resulting in a total of 6440 utterances.
After removing the empty utterances, our final set contained 116 utterances
per child on average. We then used the Whisper open-source speech recognizer
[20] to transcribe each turn.

Table 1: Demographic information of the ADOS-2 dataset. Category | Statistics
---|---
Age (years) |  | Range: 10-15 (mean: 12.27, std: 1.75)
Gender |  | 21 male, 8 female
Race |  | White: 20, Afr. Am.: 6, Hisp.: 2, Asian: 1
Diagnosis |  | ASD: 14, TD: 15
Table 2: Correlation coefficients of statistically significant indicators of ASD in ADOS-2 tasks as assessed using the conversation turns of child conversation participants.  We include only the tasks for which at least one feature was found significant after Holm's sequential Bonferroni correction. Tasks | MFCC | Spect. Harmonicity | Log HNR | Shimmer | Spectral Energy | Pitch | Disc. Markers
---|---|---|---|---|---|---|---
Description | 0.2409 | - | - | - | - |  | -
Emotion | -0.2504 | 0.2633 | 0.2641 | 0.2439 | 0.2993 | 0.2193 | -
Social | - | - | - | - | - | - | -0.2401
Friends | -0.2167 | - | - | - | - | - | -
Loneliness | -0.5402 | - | - | - | - | - | -
Interactive | 0.3424 | - | - | - | - | -0.2601 | -
Telling | 0.2242 | - | - | - | - | - | -

##  3 Method

We trained models to analyze the contribution of speech and language features
in distinguishing between children with and without ASD during different
conversational tasks111These experiments and the original data collection were
reviewed and approved by the Institutional Review Board at the participating
institutions..

###  3.1 Features

We used openSMILE [21] to extract acoustic-prosodic features from each turn.
In this work, we used the ComPARE 2016 feature set [22], which extracts 88 low
level features and 6373 features derived by applying statistical functions on
the low-level features. This tool and a similar feature set have been used
before in research on autism diagnosis [23]. The low-level features include
spectral (Mel Frequency Cepstral Coefficients (MFCC), zero crossing rate),
voice quality (local shimmer, jitter, harmonic noise ratio), and prosodic
features (loudness, pitch). We z-normalized all these features per participant
as the mean age of diagnosed children coincided with the age of changing
voice.

We used two sets of language features: Linguistic Inquiry and Word Count
(LIWC) features and lexical features motivated by prior research. As prior
work has shown correlations between the psychologist's language use and ASD
severity [13], we calculated these features for each speaker (child and
psychologist) for each of the 14 subtasks. To understand the influence of task
type and context in predicting the diagnostic group, we also used task as a
feature.

LIWC [24] has been widely used for predicting outcomes including psychological
[25] and cognitive [26] functioning and personality [27]. We used a total of
119 LIWC features, including percentage of different parts of speech (e.g.
pronouns, articles), punctuation categories (e.g. commas, periods), and
psychometric measures (e.g. affect, social, politeness).

Separately, we used a set of 12 lexical features inspired by their use in
prior autism research [12]. Using the spaCy NLP library [28] and word
dictionaries [29], we calculated the relative frequencies of certain parts of
speech as well as the number of syllables, hedge words, weasel words, filler
words and discourse markers per turn. This resulted in a total of 131 language
features. Prior research has shown the importance of appropriate usage of
discourse markers (e.g. and, but, anyway), hedge words (e.g. often, usually),
weasel words (e.g. may, like, possibly), and filler words (e.g. so, ok) for
measuring reciprocity and concreteness in conversations for autism research
[30].

###  3.2 Feature Selection and Classification

First, we performed correlational analyses to estimate the association between
our features and ASD diagnosis outcome. As our diagnosis outcome is binary, we
performed point-biserial correlations [31].  To control for any false
discovery rate arising from multiple tests, we applied Holm’s sequential
Bonferroni procedure [32].

Second, to understand the role of different features in distinguishing between
children diagnosed with ASD and TD, we used a feature fusion strategy where we
trained a random forest classifier using different combinations of features
from different modalities (speech, text, task) extracted from each speaker in
the conversation. To identify a minimal set of characteristic features, we
used our correlation analysis as a feature selection strategy where we
selected features found to be statistically significant ($p<0.05$) for each
task into the feature subset.

To ensure generalization for out-of-sample testing, we performed cross-
validation by using a leave-$n$-user-out method222We chose this over 10-fold
cross validation, as this ensures our training set does not include
information from a child who is also present in our test set.. We report
results averaged over 10 runs, where 80% (23) of children were randomly
selected for training and 20% (6) for testing in each run. All classification
results are reported as accuracy, precision, recall, F1-score and AUC score.
All experiments and models are implemented using the scikit-learn and sciPy
libraries with default parameter settings.

##  4 Analysis and Results

###  4.1 Correlation Analysis

Based on the correlation analyses, we found a subset of 28 (out of 88)
acoustic-prosodic features, 3 (out of 131) lexical features and 21 (out of 219
total) combination features, extracted from the child conversational
participants' turns, to have statistically significant correlations with an
ASD diagnosis outcome. In the case of the experimenter, we have 19, 7, and 17
features respectively that have statistically significant correlations with an
ASD diagnosis outcome. We present in Table 2 the 7 features we find
significant most frequently across different tasks.

In our correlation analysis, we also explored the top features based on
significant correlation and strength in each task. To find shared features for
both child and psychologist, we compared the top features for the two speakers
in the conversational context, which we present in Table 3.

###  4.2 Observations

Child. Here, we include some additional observations regarding informative
features for different task contexts. When using features from the child
conversational participant, in conversations about social difficulties
children with ASD used fewer discourse markers ($r$ = $-0.2401$) and fewer
words. Children with ASD used fewer pronouns ($r$ = $-0.3703$) and discourse
markers ($r$ = $-0.3465$), and their turns were shorter ($r$ = $-0.3299$). For
the make-believe play task, children with ASD used more hedge words ($r$ =
$+0.2150$). Interestingly, when talking about friends, children with ASD used
fewer words from the tech category than TD children ($r$ = $-0.1093$).

When talking about emotion, describing a picture, and during the break, the
count of words, words per sentence, nouns, syllables and turn duration were
all negatively correlated with an ASD diagnosis outcome, which suggest shorter
responses from ASD children. Acoustic-prosodic features such as loudness and
pitch were positively correlated with an ASD diagnosis outcome for the
cartoons subtask, but negatively correlated for silent play during break.

Psychologist. When talking about emotions or friends, the psychologist used
fewer discourse markers ($r$ = $-0.2091$), weasel words, words, and syllables,
and produced shorter turns, when talking with children with ASD. When talking
about loneliness, use of the word you ($r$ = $0.3173$) is increased and i
decreased. The counts of words, words per sentence and pronoun usage (he/she,
we, they) were all negatively correlated with an ASD diagnosis outcome in
conversation, break and emotion subtasks. Overall, language features were
found to be more informative in the psychologist data for predicting the
diagnosis outcome, whereas only MFCC features among all acoustic-prosodic
features were found to be significant. This observation is reflected in our
classification results discussed next.

Table 3: Most predictive features for different subtasks with task descriptions. We add the indicator's correlation coefficient with an ASD diagnosis outcome for features extracted from the experimenter's turns when found significant. (C = Child, P = Psychologist) Subtask Name | Task Description | Feature | Corr. Coefficient (C $|$ P)
---|---|---|---
Description | Description of a picture | MFCC [13] |  $+0.2409$ $|$ $-0.2064$
Conversation | Conversation about topics of interest | Duration |  $|$ $-0.2675$
Emotions | Interview about their feelings | Spectral energy | $+0.2993$
Social Difficulties | Interview about their social lives | Discourse Markers | $-0.2401$
Friends | Interview about their relationships | MFCC [5] | $-0.2167$
Loneliness | Interview about loneliness | MFCC [7] | $-0.5402$
Construction | Work on a puzzle | MFCC [14] |  $+0.2223$ $|$ $-0.2671$
Make-believe Play | Child plays with toys alone | MFCC [13] |  $+0.2973$ $|$ $+0.4467$
Interactive Play | As previous (with psychologist) | MFCC [6] |  $+0.3424$ $|$ $+0.2603$
Demonstration | Demonstrates how to do a task | Spectral energy | $+0.2335$
Telling | Makes up story based on picture book | MFCC [13] | $+0.2242$
Cartoons | Act out a cartoon story | Duration | $+0.3044$
Imaginative Story | Creating a story with small prop items | MFCC [1] |  $+0.1786$ $|$ $+0.1765$
Break | Silent play and unstructured conversation | MFCC[6] |  $+0.1997$ $|$ $+0.2014$
Table 4: ASD diagnosis classification from child turns using a random forest classifier trained with different combinations of full and selected features. The number of features used in each model is indicated by N. Best results are boldfaced. (A: Acoustic-prosodic, L: Lexical, T: Task category) Features | Acc. | Prec. | Rec. | F1 | AUC
---|---|---|---|---|---
A (N = 88) | 0.48 | 0.67 | 0.48 | 0.50 | 0.55
A (N = 28) | 0.51 | 0.65 | 0.51 | 0.55 | 0.56
L (N = 131) | 0.60 | 0.72 | 0.60 | 0.57 | 0.63
L (N = 3) | 0.61 | 0.76 | 0.61 | 0.55 | 0.62
A + L (N =219) | 0.66 | 0.73 | 0.66 | 0.61 | 0.62
A + L (N = 21) | 0.62 | 0.81 | 0.62 | 0.58 | 0.66
A+L+T (N = 220) | 0.74 | 0.85 | 0.74 | 0.72 | 0.76
A+L+T (N = 22) | 0.76 | 0.84 | 0.76 | 0.75 | 0.71

###  4.3 Classification

We use the findings of our correlation analyses to inform our feature
selection strategy when training classifiers to distinguish between children
with ASD and TD children. We first trained a classifier with all features from
each speaker turn, and then separately trained classifiers with selected
features from different combinations of feature sets. We used random forests
because their results are highly interpretable by psychologists. Our goal was
to find a minimal feature set for the purpose of aiding diagnosis.

Our classification results for child turn data are presented in Table 4. For
each feature type, we trained with full and selected feature sets (the number
of features, $N$, is indicated in parentheses). We found language features
were better indicators for the diagnosis outcome than acoustic-prosodic
features. However, the combination of features from the two modalities gave
improved accuracy ($F1=0.61$).  When we trained with features selected using
point-biserial correlation analysis, the number of features used in the model
was reduced by nearly 80% for the A, A+L and A+L+T settings, while
classification accuracy improved or remained unchanged. After including the
subtask category as a categorical feature, we obtained the best classification
results with both full and selected feature sets.

We observed similar findings when using the psychologist's turn data which we
present in Table 5.  Using only acoustic-prosodic features (A) and only text
(T), the model trained on the psychologist's turns provided results similar to
the one trained on the childs' turns. We observe that the classification
results remained unchanged or improved for all settings (A, L, A+L, A+L+T)
when training with nearly 80% fewer features selected through the correlation
analysis feature selection procedure, an observation similar to the results
from child's turns.

We also applied dimensionality reduction (PCA) as an alternate feature
selection strategy and compared this with the correlation-based selection
strategy. The number of principal components was chosen to preserve 95%
variance in the data. We found that this did not improve performance compared
to using the full feature set. One reason could be that we fit a PCA model to
the training data and transformed the testing data using the fitted model.
Since our training and testing data come from different children, applying PCA
did not help in this case.

Ethical discussion and limitations. Our goal is to provide AI decision tools
to help assessors more effectively use ADOS for ASD diagnosis. It would be
unethical to deploy the machine learning approach outlined here on its own
(without human judgment). Also, it would be unethical to deploy such an
approach in a non-clinical setting. In order to ensure no misuse of our
findings, we do not plan to release the trained models or raw data to the
public.

For our experiments, we used entirely automatic preprocessing. No stage of
preprocessing is perfect. In particular, there are many cases where pyannote
missed speaker changes. Our experimental results could be improved with manual
diarization and transcription, but automatic diarization and transcription are
more practical in our target setting.

Another limitation of this work is that our data includes only two diagnostic
groups (ASD and TD) and the language of the conversations is English. Whether
the informative features for ASD are also applicable for other co-occurring
neuro-developmental disorders (e.g. ADHD) and other languages is a direction
for future work. Finally, the current data set only included one psychologist,
so we do not know how our findings would generalize to other practitioners.

Table 5: ASD diagnosis classification from psychologist's turns using a random forest classifier trained with different combinations of full and selected features. The number of features used in each model is indicated by N. Best results are boldfaced. (A: Acoustic-prosodic, L: Lexical, T: Task category) Features | Acc. | Prec. | Rec. | F1 | AUC
---|---|---|---|---|---
A (N = 88) | 0.49 | 0.60 | 0.49 | 0.50 | 0.52
A (N = 19) | 0.51 | 0.63 | 0.51 | 0.51 | 0.56
L (N = 131) | 0.56 | 0.79 | 0.56 | 0.54 | 0.63
L (N = 7) | 0.63 | 0.82 | 0.63 | 0.60 | 0.66
A + L (N = 219) | 0.55 | 0.70 | 0.55 | 0.47 | 0.59
A + L (N = 17) | 0.61 | 0.74 | 0.61 | 0.53 | 0.56
A+L+T (N = 220) | 0.59 | 0.80 | 0.59 | 0.56 | 0.64
A+L+T (N = 18 ) | 0.68 | 0.83 | 0.68 | 0.65 | 0.67

##  5 Conclusions and Future Work

Early diagnosis and interventions are critical to positive outcomes for
autism. There is a growing need for identifying behavioral indicators that can
help clinicians to diagnose ASD. Speech and language features have been
extensively studied in prior works to distinguish between children with ASD
and TD children. In this work, we explored the diagnostic informativeness of
different acoustic-prosodic and language features across multiple diagnostic
conversational contexts, separately examining data from child and psychologist
turns. We used acoustic-prosodic indicators and language indicators to
identify the most informative features for different conversational tasks
administered during the ADOS-2. Our results from correlational and
classification experiments suggest the presence of informative indicators that
are shared across different contexts and co-appear in conversational turns
from both speakers.

We are currently working to confirm these results via manually corrected
speaker diarization and transcription. In future work, we would like to extend
this analysis to multimodal conversational data. We also plan to examine
modeling approaches that take into account more of the conversational context,
for example, whether the child conversation participant constructs
contextually appropriate dialog acts, or the ways in which child and assessor
align in acoustic-prosodic or linguistic ways during conversation.

##  6 Acknowledgements

We would like to thank and acknowledge all our participants and their
families. The data set was obtained thanks to support by the National
Institute of Mental Health and the National Institutes of Health: awards
R21MH094659 and F31MH108331.

## References

  * [1] American Psychiatric Organization, _Diagnostic and statistical manual of mental disorders (DSM-5)_. Washington, DC: American Psychiatric Publishing, 2013.
  * [2] H. Tager-Flusberg and E. Caronna, ``Language disorders: Autism and other pervasive developmental disorders,'' _Pediatric Clinics of North America_ , vol. 54, no. 3, Jun. 2007.
  * [3] H. Tager-Flusberg, ``A psychological approach to understanding the social and language impairments in autism,'' _International Review of Psychiatry_ , vol. 11, no. 4, pp. 325–334, Jan. 1999.
  * [4] P. Mundy and J. Markus, ``On the nature of communication and language impairment in autism,'' _Mental Retardation and Developmental Disabilities Research Reviews_ , vol. 3, no. 4, pp. 343–349, 1997.
  * [5] R. Landa, ``Social language use in Asperger syndrome and high-functioning autism,'' in _Asperger Syndrome_. New York, NY, US: The Guilford Press, 2000, pp. 125–155.
  * [6] C. Lord, S. Risi, L. Lambrecht, E. H. Cook, B. L. Leventhal, P. C. DiLavore, A. Pickles, and M. Rutter, ``The Autism diagnostic observation schedule—generic: A standard measure of social and communication deficits associated with the spectrum of Autism,'' _Journal of Autism and Developmental Disorders_ , vol. 30, no. 3, pp. 205–223, Jun. 2000.
  * [7] R. Fusaroli, E. Weed, D. Fein, and L. Naigles, ``Hearing me hearing you: Reciprocal effects between child and parent language in autism and typical development,'' _Cognition_ , vol. 183, pp. 1–18, Feb. 2019.
  * [8] R. Fusaroli, A. Lambrechts, D. Bang, D. M. Bowler, and S. B. Gaigg, ``Is voice a marker for autism spectrum disorder? a systematic review and meta-analysis,'' _Autism Research: Official Journal of the International Society for Autism Research_ , vol. 10, no. 3, pp. 384–407, 2017.
  * [9] R. Fusaroli, R. Grossman, N. Bilenberg, C. Cantio, J. R. M. Jepsen, and E. Weed, ``Toward a cumulative science of vocal markers of autism: A cross-linguistic meta-analysis-based investigation of acoustic markers in American and Danish autistic children,'' _Autism Research: Official Journal of the International Society for Autism Research_ , vol. 15, no. 4, pp. 653–664, 2022.
  * [10] G. Kiss, J. P. v. Santen, E. Prud'Hommeaux, and L. M. Black, ``Quantitative analysis of pitch in speech of children with neurodevelopmental disorders,'' in _Proceedings of INTERSPEECH_ , 2012.
  * [11] D. Bone, M. P. Black, A. Ramakrishna, R. B. Grossman, and S. S. Narayanan, ``Acoustic-prosodic correlates of `awkward'prosody in story retellings from adolescents with autism,'' in _Proceedings of INTERSPEECH_ , 2015, pp. 1616–1620.
  * [12] A. Song, M. Cola, S. Plate, V. Petrulla, L. Yankowitz, J. Pandey, R. T. Schultz, and J. Parish-Morris, ``Natural language markers of social phenotype in girls with autism,'' _Journal of Child Psychology and Psychiatry_ , vol. 62, no. 8, pp. 949–960, 2021.
  * [13] M. Kumar, R. Gupta, D. Bone, N. Malandrakis, S. Bishop, and S. S. Narayanan, ``Objective language feature analysis in children with neurodevelopmental disorders during autism assessment,'' in _Proceedings of INTERSPEECH_ , 2016, pp. 2721–2725.
  * [14] A. Goodkind, M. Lee, G. E. Martin, M. Losh, and K. Bicknell, ``Detecting language impairments in autism: A computational analysis of semi-structured conversations with vector semantics,'' in _Proceedings of the Society for Computation in Linguistics (SCiL) 2018_ , 2018, pp. 12–22.
  * [15] D. Bone, C.-C. Lee, M. P. Black, M. E. Williams, S. Lee, P. Levitt, and S. Narayanan, ``The psychologist as an interlocutor in autism spectrum disorder assessment: Insights from a study of spontaneous prosody,'' _Journal of Speech, Language, and Hearing Research_ , vol. 57, no. 4, pp. 1162–1177, 2014.
  * [16] R. Gupta, D. Bone, S. Lee, and S. Narayanan, ``Analysis of engagement behavior in children during dyadic interactions using prosodic cues,'' _Comput. Speech Lang._ , vol. 37, no. C, p. 47–66, may 2016.
  * [17] T. Muskett, M. Perkins, J. Clegg, and R. Body, ``Inflexibility as an interactional phenomenon: Using conversation analysis to re-examine a symptom of autism,'' _Clinical linguistics & phonetics_, vol. 24, no. 1, pp. 1–16, 2010.
  * [18] H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, ``pyannote.audio: neural building blocks for speaker diarization,'' in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2020.
  * [19] (2022) Pydub. [Online]. Available: https://github.com/jiaaro/pydub
  * [20] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' 2022.
  * [21] F. Eyben, M. Wöllmer, and B. Schuller, ``Opensmile: The munich versatile and fast open-source audio feature extractor,'' in _Proceedings of the 18th ACM International Conference on Multimedia_. New York, NY, USA: Association for Computing Machinery, 2010, p. 1459–1462.
  * [22] B. Schuller, S. Steidl, A. Batliner, J. Hirschberg, J. K. Burgoon, A. Baird, A. Elkins, Y. Zhang, E. Coutinho, and K. Evanini, ``The Interspeech 2016 computational paralinguistics challenge: Deception, sincerity & native language,'' in _Proceedings of INTERSPEECH_ , vol. 8, 2016, pp. 2001–2005.
  * [23] Y.-K. Kim, R. Lahiri, M. Nasir, S. H. Kim, S. Bishop, C. Lord, and S. S. Narayanan, ``Analyzing short term dynamic speech features for understanding behavioral traits of children with autism spectrum disorder.'' in _Interspeech_ , 2021, pp. 2916–2920.
  * [24] J. W. Pennebaker and L. A. King, ``Linguistic styles: language use as an individual difference.'' _Journal of Personality and Social Psychology_ , vol. 77, no. 6, p. 1296, 1999.
  * [25] E.-M. Rathner, J. Djamali, Y. Terhorst, B. Schuller, N. Cummins, G. Salamon, C. Hunger-Schoppe, and H. Baumeister, ``How did you like 2017? Detection of language markers of depression and narcissism in personal narratives,'' in _Proceedings of INTERSPEECH_ , 2018, pp. 3388–3392.
  * [26] M. Asgari, J. Kaye, and H. Dodge, ``Predicting mild cognitive impairment from spontaneous spoken utterances,'' _Alzheimer's & Dementia: Translational Research & Clinical Interventions_, vol. 3, no. 2, pp. 219–228, 2017.
  * [27] F. Mairesse and M. Walker, ``Automatic recognition of personality in conversation,'' in _Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers_ , 2006, pp. 85–88.
  * [28] M. Honnibal and I. Montani. (2022) spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. [Online]. Available: https://github.com/explosion/spaCy
  * [29] (2022) List of (possible) English words. [Online]. Available: https://github.com/words
  * [30] C. Yang, D. Liu, Q. Yang, Z. Liu, and E. Prud’hommeaux, ``Predicting pragmatic discourse features in the language of adults with autism spectrum disorder,'' in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop_ , 2021, pp. 284–291.
  * [31] J. D. Brown, ``Point-biserial correlation coefficients,'' _Statistics_ , vol. 5, no. 3, pp. 12–6, 2001.
  * [32] Y. Benjamini and Y. Hochberg, ``Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing,'' _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 57, no. 1, pp. 289–300, 12 2018.


# Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight
Residuals

Simo Ryu
KAIST
<EMAIL_ADDRESS>
&Seunghyun Seo11footnotemark: 1
NAVER Cloud
<EMAIL_ADDRESS>
&Jaejun Yoo
UNIST
<EMAIL_ADDRESS>
Equal contribution

###### Abstract

In this paper, we present an efficient method for storing fine-tuned models by
leveraging the low-rank properties of weight residuals. Our key observation is
that weight residuals in large overparameterized models exhibit even stronger
low-rank characteristics. Based on this insight, we propose Efficient Residual
Encoding (ERE), a novel approach that achieves efficient storage of fine-tuned
model weights by approximating the low-rank weight residuals. Furthermore, we
analyze the robustness of weight residuals and push the limit of storage
efficiency by utilizing additional quantization and layer-wise rank
allocation. Our experimental results demonstrate that our method significantly
reduces memory footprint while preserving performance in various tasks and
modalities. We release our code. 111Our code is available at:
https://github.com/cloneofsimo/ere

##  1 Introduction

Fine-tuning large-scale pre-trained models has become a widely adopted
technique in tasks such as language modeling [9, 26, 17], speech recognition
[3] and image generation [40, 35, 28]. It has proven to be effective in
achieving state-of-the-art performance on diverse tasks. However, as the
number of model parameters continues to grow exponentially, reaching billions,
storing fully fine-tuned parameters for each specific task becomes impractical
and inefficient. To tackle this issue, Parameter Efficient Fine-Tuning (PEFT)
approaches [20, 18, 29, 25, 5, 16] have been introduced, aiming to tune only a
subset of the original parameters and preserving the tuned parameters for
individual tasks.

While PEFT methods are efficient and show promising results that are
competitive with full fine-tuned models, they often rely on complex
handcrafted modules or fixed-dimensional hyperparameters for each model layer.
For example, methods like Compacter [20], LoRA [18], and prefix-tuning [29]
introduce additional complexity by requiring specific ranks or token prefixes,
and achieving optimal performance with these methods often necessitates
extensive hyperparameter search. Recent research has also shown that even with
thorough incorporation of extensive model/task/resource-specific priors, PEFT
struggles to outperform full fine-tuning in low-to-medium resource scenarios
[38] under compute budget. Downside of PEFT becomes more pronounced as these
methods are not compatible with existing techniques that are done on fine-
tuning of full weights [23, 36, 49, 2, 34, 47, 46, 33]

For these reasons, one might conclude to use full fine tuning for their
downstream fine-tuning. If so, can we still resolve the problem of inefficient
memory footprint in a full fine-tuning setup? To tackle this problem, we
propose a different approach in this paper focusing on weight residuals: the
differences between fine-tuned models and their base counterparts. Remarkably,
we have observed that weight residuals of overparameterized models, already
acknowledged for their low-rank nature [1, 27, 19, 32], exhibit even more
pronounced low-rank characteristics. Furthermore, these individual residual
parameters demonstrate robustness, indicating that their contributions to
downstream data shift are highly redundant.

Based on our findings, we propose Efficient Residual Encoding (ERE), which
involves fully fine-tuning the model and efficiently storing weight residuals
in a low-rank format. We also employ rank-wise vector quantization, using a
low-rank approximation based on the weight-shiftedness of individual layers,
measured through their approximated spectral distribution. The result of our
method is demonstrated in Figure 1.

Figure 1: Average performance of RoBERTa-Large models with different weight
residual compression methods on GLUE tasks vs storage usage. The storage
requirement for the full weights is 1.36GB and the most competitive ERE,
positioned at the rightmost of the red line, exhibits only a 0.2% degradation
in accuracy with 84MB. See §5.1 for details.

We conducted experiments on multiple tasks, including Natural Language
Understanding (NLU), Language Modeling (LM), and image generation, to evaluate
the effectiveness of our proposed method. In the NLU task, we demonstrated
that the weight residuals can be compressed by up to 6.0% of full fine-tuned
weights while achieving comparable performance. Specifically, we achieved an
average accuracy of 89.2 on the General Language Understanding Evaluation
(GLUE) benchmark, which is very close to the performance of the original
weights, 89.4. We also demonstrate that ERE can successfully preserve the
original generative quality with significant efficient storage in image
generation and LM tasks. We summarize our contributions below.

  * •

We observe that the differences in fine-tuned models, especially those with
small parameter changes, can be approximated with significantly less memory
than the base model.

  * •

We further observe that different layers require different budgets of
approximation. To this end, we propose ERE, a novel approach that enables the
discriminatory allocation of resources among layers to obtain optimal storage.

  * •

As our method is model agnostic, we demonstrate the effectiveness of ERE
across various tasks and modalities. Our method achieves efficient storage
while maintaining performance.

##  2 Related Works

###  2.1 Transfer Learning

Transfer learning involves fine-tuning a pre-trained model on a specific
downstream task using a smaller labeled dataset. This approach capitalizes on
the knowledge acquired during pre-training, enabling the model to quickly
adapt to the new task with reduced data and computation requirements [41, 24].
Transfer learning has proven to be highly effective in enhancing the
performance of language models across various NLU tasks [17, 9, 26].
Furthermore, there have been significant efforts to customize generative
models, such as GANs [35, 28] and diffusion-based models [40, 12].

###  2.2 Parameter Efficient Fine-Tuning

Parameter Efficient Fine-Tuning (PEFT) methods have emerged as an alternative
to achieve efficient transfer learning, as fully fine-tuning pre-trained model
weights can be computationally expensive. LoRA is a prominent PEFT method that
utilizes a bottleneck layer similar to adapter. It requires a predefined rank
$r$ with $\alpha$ prior to train, which determines the reparameterized update
scale. The fixed rank $r$ is uniformly applied to all layers, assuming equal
budget requirements for each layer. Recent works like AdaLoRA [51] and
DyLoRA[30] have addressed this limitation by dynamically allocating the budget
based on the importance of individual layers. While they are promising, they
introduce even harder implementation difficulties.

In contrast, our proposed method is not integrated into the training procedure
itself. Therefore, once the model is trained, we have the flexibility to
choose how the weight residuals are compressed based on the ERE factors.

##  3 Analysis On Weight Residuals

###  3.1 Effective Rank Dynamics of $\Delta\theta$

Figure 2:  Normalized singular value magnitudes of weight residuals over time.
The red dotted line represents the spectral values of the base model, while
the blue dotted line represents the expected spectral value quantile from the
Marchenko-Pastur (M.P.) Distribution (λ = 0.25). The M.P. Distribution serves
as a suitable proxy for the theoretical singular-value distribution with
random matrix initialization.

Let us denote the pre-trained model weight as $\theta$, and fine-tuned model
as $\theta^{\prime}$. We conduct a detailed analysis of the training dynamics
of the residual weights, denoted as $\Delta\theta=\theta^{\prime}-\theta$
during fine-tuning of the RoBERTa-Large model on various NLU tasks. We
investigate the effective rank of the weight residuals and compare them to
randomly initialized networks. Consistent with previous literatures [19, 32],
we find that large overparameterized models, including pre-trained RoBERTa
models, exhibit a low-rank structure as training progresses.

Interestingly, we observe that the weight residuals possess significantly
lower effective ranks. This is demonstrated in Figure 2, where we plot the
magnitude of the singular values normalized by the maximum singular value over
time. The eigenvalue distribution rapidly converges to a low-rank regime,
indicating a much smaller effective rank compared to the base parameters.

Furthermore, we discover that within the same layers, the effective ranks of
the weight residuals are unpredictable and dynamic, as illustrated in Figure
3. This finding helps explain the need for extensive hyper-parameter search
and the inconsistent performance of PEFT across tasks, as demonstrated in
previous works [7, 38, 42, 51].

Although similar behavior could have been hypothesized based on previous
studies [27, 20, 18, 44, 22, 15, 1, 19, 32], to our best knowledge, our study
is the first to systematically observe this behavior.

Figure 3:  The changes of the effective rank of the query, key, value and
output projection matrices from the last attention layer of the RoBERTa-Large
model based on the weight update process. The normalized effective ranks,
$\text{erank}(\Delta\theta)/\text{erank}(\theta)$ of each layer are visualized
per task.

###  3.2 Robustness of $\Delta\theta$

(a) Log-normal perturbation

(b) Low-rank approximation

Figure 4:  Cosine similarity between feature outputs of perturbed weights
$\widetilde{\theta^{\prime}}$ and fine-tuned weights $\theta^{\prime}$. The
perturbation is applied to full fine-tuned weight ($\theta+\Delta\theta$)
itself and weight residual ($\Delta\theta$) to compare the robustness to the
perturbation. The perturbation is applied in two ways: (a) adding random
noise, and (b) applying a low-rank approximation.

We observed that fine-tuned weight residuals are significantly less sensitive
to random noise compared to pre-trained weights. By introducing log-normal
perturbations to each parameter, we evaluate the impact of noise on the
model’s output representation. We perturb either weight or weight residuals
using $\widetilde{\theta^{\prime}}=(\theta+\Delta\theta)\exp(Z)$ or
$\widetilde{\theta^{\prime}}=\theta+\exp(Z)\Delta\theta$, where $Z\sim
N(0,\sigma^{2})$ and measure the change in representation using angular
distance. From 4(a), one can see that noise perturbation on the residual does
little to no actual representation deviation to certain degree.

Building upon the previous findings regarding the low rankedness of weight
residuals, we conclude that weight residuals are significantly redundant to
their minimal impact. To illustrate this, we perform low-rank approximation on
both the full weights and weight residuals, comparing the resulting
differences in output representation using cosine similarity. 4(b) highlights
the stark contrast between the base parameters, which exhibit a substantial
deviation when reducing 20% of their rank, and the weight residuals, which do
not introduce such drastic differences.

##  4 ERE: Efficient Residual Encoding

Based on our observations, we introduce Efficient Residual Encoding (ERE) as a
method for compressing weight residuals instead of full fine-tuned weights. By
leveraging low-rank approximation techniques, guided by the requirements of
individual layers (as observed in §3.1), we achieve even more efficient
storage of the weight residuals.

###  4.1 Low-rank Approximation and Budget Allocation

We first approximate $\Delta\theta$ using the product of three matrices, $U$,
$D$, and $V$, where $U$ and $V$ are low-rank Stiefel matrices, and $D$ is a
diagonal matrix. While this solution can be obtained through Singular Value
Decomposition, some layers require significantly less rank to accurately
approximate $\Delta\theta$. We can use the individual Empirical Spectral
Distribution (ESD) to determine the appropriate amount of rank for
$\Delta\theta$ after fine-tuning. Given a fixed parameter budget $M$, our
objective is to distribute the rank in a way that minimizes the average
Frobenius norm for the value. Let best possible rank $k$ approximation of
weight residual $w=\Delta\theta$ be
$w_{k}=\sum_{i=1}^{k}\sigma_{i}u_{i}v_{i}^{T}$. Frobenius error by such
approximation is

$\lvert
w-w_{k}\rvert_{F}=\lvert\sum_{i=k+1}^{n}\sigma_{i}u_{i}v_{i}^{T}\rvert_{F}=\sqrt{\sum_{i=k+1}^{n}\sigma_{i}^{2}}$

where $w$ is a $m\times n$ matrix with $m\leq n$ [10]. Our objective is to
minimize the sum of every layer’s low-rank approximation error. As each
layer’s rank $r$ comes with a parameter budget of $(n+m)r$, our objective can
be expressed as follows:

$\displaystyle\min_{\boldsymbol{r}}$
$\displaystyle\sum_{i=1}^{N}\sum_{l=r_{i}+1}^{\min(n_{i},m_{i})}\sigma_{l}^{2}$
(Total Squared Error) subject to
$\displaystyle\sum_{i=1}^{N}r_{i}(n_{i}+m_{i})\leq M$ (Budget)

Here, $n_{i}\times m_{i}$ are the shapes of the weight matrices of layer $i$,
and $N$ is the number of all fine-tunable layers for the model. Note that this
formulation is a special case of the well-known knapsack problem and, as such,
is NP-Complete. Thus to find the approximate solution, we approximate
$f_{i}(r)=\sum_{l=r+1}^{min(n_{i},m_{i})}\sigma_{l}^{2}$ log-linearly based on
layer $i$’s ESD, impose continuous relaxation on rank variables, and solve the
relaxed problem in a Lagrangian formulation and binary search. See §A for a
detailed explanation of our algorithm.

###  4.2 Further Optimization

While our general idea is described above, we can deploy a few modifications
to further enhance performance. First, to address the potential overestimation
of one effect of the rank over the other, we incorporate prior rank. This
involves uniformizing the relaxed rank solution $\textbf{r}\in\mathbb{R}^{N}$
by taking $r_{i}^{\prime}=(1-\alpha)r_{i}+\alpha r_{avg}$, where $r_{avg}$ is
defined as $\sum_{i=1}^{N}r_{avg}(n_{i}+m_{i})=M$. Note that this
uniformization does not violate budget constraints. We have empirically
determined $\alpha=0.5$ to work well. This heuristic is necessary as our
formulation, aimed at reducing the aggregate of Frobenius norm, is not
perfectly aligned with the ultimate objective of reconstructing the final
feature representation. A more sophisticated objective remains a potential
future direction of research.

Second, we observed that aggressive quantization can be applied without
significant performance loss, consistent with findings in [8, 48]. Following
their approach, we utilized round-to-nearest quantization for $U$ and $V$,
while reserving half-precision for $D$. However, we encountered a potential
issue when dequantizing $U$ and $V$, as their precision might be significantly
lower than that of $D$, violating our assumption that $U$ and $V$ are Stiefel
elements. To address this, we conveniently project the dequantized $U$ and $V$
back onto the nearest Stiefel manifold. Further details and ablation results
can be found in §B.

##  5 Experiments

Method Storage MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Avg. FT ([31]) 1.36GB
90.2 96.4 90.9 68.0 94.7 92.2 86.6 92.4 88.9 FT ∗ 1.36GB 90.1 96.3 92.4 67.8
94.9 92.2 88.8 92.0 89.4 BITQ (b=2) 85MB 37.9 94.0 84.1 51.9 92.1 36.8 83.0
88.0 71.0 BITQ (b=4) 169MB 90.1 96.2 92.4 67.8 94.9 92.2 88.8 92.0 89.3 LRA
(r=8) 20MB 83.7 95.4 90.4 63.4 94.0 86.0 84.8 91.6 86.2 LRA (r=16) 36MB 86.1
95.6 91.4 65.4 94.6 86.9 87.4 91.8 87.4 LRA (r=32) 70MB 88.6 96.0 91.4 66.1
94.8 88.2 88.4 91.9 88.2 LRA (r=64) 136MB 89.6 96.0 91.9 65.8 94.7 89.8 88.1
92.0 88.5 LRA (r=128) 270MB 90.0 96.0 91.9 67.5 94.8 91.3 88.8 92.0 89.0 LRA
(r=256) 538MB 90.1 96.1 92.2 67.7 94.7 92.0 88.1 92.0 89.1 ERE (r=8, b=4)
3.9MB 47.7 95.2 90.4 63.0 92.8 75.9 86.6 90.9 80.3 ERE (r=16, b=4) 6.8MB 83.9
95.4 90.7 65.0 94.1 85.2 87.4 91.4 86.6 ERE (r=32, b=4) 12MB 86.9 95.6 91.4
64.6 94.6 87.1 88.1 91.5 87.5 ERE (r=64, b=4) 23MB 89.1 95.6 91.9 66.5 94.7
88.7 89.2 91.7 88.4 ERE (r=128, b=4) 43MB 90.0 95.9 91.9 67.2 94.7 90.6 89.2
91.7 88.9 ERE (r=256, b=4) 84MB 90.2 96.1 92.2 68.2 94.7 91.7 88.8 91.8 89.2

Table 1:  The comparison between RoBERTa-Large with full Fine-Tuning (FT)
method and ERE on GLUE. Note that, the baseline we implement(∗) is slightly
better than the reported results in the paper. We measure Mattew’s correlation
for CoLA Pearson correlation for STS-B, and overall accuracy for other tasks.
Additionally, we provide the results when applying BIT Quantization (BITQ),
and Low-Rank Approximation (LRA) independently. $r$ and $b$ denote rank and
quantization level each.

###  5.1 ERE on Natural Language Understanding

To assess the effectiveness of our method, ERE, we first conducted evaluations
on the standard NLU benchmark, GLUE [45]. We fine-tune the provided pre-
trained RoBERTa-Large [31] model on each individual task, employing a single-
model and single-task fine-tuning setup. We train and evaluate each model with
Fairseq222https://github.com/facebookresearch/fairseq [37] framework. We
utilize the official training configurations to reproduce the performance of
the full fine-tuning model and apply ERE on the weight residuals between pre-
trained and fine-tuned models. We conduct evaluations using various rank
settings for ERE and report both the overall accuracy and the corresponding
storage requirements. Note that, we save model weights with full precision
except newly added classifiers because they have no counterparts to compress
residual. We also report the results when BIT Quantization (BITQ) and Low-Rank
Approximation (LRA) techniques are applied independently. For all the
experiments applied ERE, $r$ indicates prior rank, and $b$ means bit
quantization level. Note that, the bit quantization technique used in BITQ
differs from ERE. In ERE, only the $U$ and $V$ components of the low-rank
factorized residuals are quantized as mentioned §4.2, while BITQ quantizes all
elements of the weight residuals.

Table 1 illustrates that ERE (r=256, b=4) results in a significantly reduced
model, requiring only 6.0% of the original storage space, with minimal impact
on performance degradation. The results indicate that residual compression can
be effectively achieved even when employing only BITQ or LRA techniques.
However, it is important to note that these methods require larger storage
compared to ERE. Nevertheless, despite the advantages of our method in
efficient storage of model weights, there exists a potential for failure in
accurately reconstructing the original model’s representations when
compressing residuals with extremely low rank and bit, particularly in tasks
such as MNLI and QQP. We hypothesize that this is because these tasks take
much larger weight updates compared to other tasks, where MNLI and QQP takes
about 37 times larger training wall clock time on a single V100 GPU machine,
while maintaining the same batch size. Another possibility could be that fine-
tuning RoBERTa with NLU tasks requires a newly initialized classifier unlike
other tasks such as text-to-image generation and language modeling.
Furthermore, it is worth noting that the application of ERE yields improved
performance in certain tasks (MNLI, CoLA, and RTE). This improvement can be
attributed to the elimination of redundant or performance-degrading factors
through the encoding of weight residuals with ERE.

Figure 5:  The comparison between generated samples of fine-tuned StyleGAN
with NADA method and Latent Diffusion Model and their ERE reconstructed
counterparts. Diffusion-based models generate images based on three prompts:
"A portrait of a dog in $<Style>$", "A portrait of a woman in $<Style>$" and
"A still life painting of fruit in $<Style>$", where the $<Style>$ token is
replaced with each of the styles: $disney$, $gtav$, and $ghibli$. The ERE-
applied models are compressed using 4-bit quantization and has prior ranks of
4, 8, and 16, respectively.

Model Type ERE (r=4, b=4) ERE (r=8, b=4) ERE (r=16, b=4) FT StyleGAN 0.305
(0.46MB) 0.291 (0.67MB) 0.281 (1.08MB) \- (364MB) GTA5 Diffusion 0.365 (3.7MB)
0.327 (6.1MB) 0.305 (11MB) \- (3.44GB) Disney Diffusion 0.221 (3.7MB) 0.193
(6.1MB) 0.159 (11MB) \- (3.44GB) Ghibli Diffusion 0.345 (3.7MB) 0.326 (6.1MB)
0.299 (11MB) \- (3.44GB)

Table 2:  LPIPS distance between fully fine-tuned models and their ERE applied
counterparts. Lower is better. Note that, for the diffusion-based models, only
the UNet module is considered for storage computation.

###  5.2 ERE on Image Generation

This section presents our experiments on the image generation task using ERE.
We utilize publicly available pre-trained and fully fine-tuned Latent
Diffusion Models (LDMs) [39] and StyleGANs [21].

For the diffusion-based models, we apply ERE on publically available fine-
tuned models such as Disney diffusion333https://huggingface.co/nitrosocke/mo-
di-diffusion, GTA5
diffusion444https://huggingface.co/ItsJayQz/GTA5_Artwork_Diffusion, Ghibli
diffusion555https://huggingface.co/nitrosocke/Ghibli-Diffusion. For the
diffusion-based models, we fixed seed with 0 and 50 inference steps and a 7.5
guidance scale for sampling. We randomly sample 40 images from a fixed set of
prompts, and measure Learned Perceptual Image Patch Similarity (LPIPS)
distance [43] on generated images to measure the accurate reconstruction of
the fine-tuned models. Similarly for StyleGAN models, we perform CLIP-driven
adaptation via NADA method [13] and compress the adapted models with ERE. We
also sample 40 images with fixed seed 0 and measure their LPIPS distance. The
results, as shown in Figure 5 and Table 2, demonstrate that ERE (r=4, b=4) can
successfully preserve most of the semantic information. Furthermore,
increasing the number of ranks leads to improved perceptual quality.

###  5.3 ERE on Language Modeling

Method Storage OpenBook ARC Wino Hella ARC PIQA BoolQ OpenAI Avg. Avg. QA Easy
Grande Swag Challenge Lambada (w/o lambada) (w/ lambada) Pythia-2.8B 10.7GB
34.80 58.59 56.96 59.12 32.34 73.40 63.82 - 52.34 - Dolly-3B 10.7GB 38.40
61.15 58.96 65.08 37.03 74.27 57.55 - 54.49 - Pythia-2.8B∗ 10.7GB 35.60 58.96
59.75 59.30 33.02 73.61 64.59 64.70 52.95 54.30 Dolly-3B∗ 10.7GB 38.80 61.62
59.19 64.99 37.03 74.16 57.55 62.86 54.64 55.60 ERE (r=4, b=4) 8.0MB 38.00
63.76 59.43 64.54 35.75 75.68 61.62 65.98 56.97 58.10 ERE (r=8, b=4) 12.0MB
38.00 63.72 59.75 65.01 35.75 75.90 60.92 65.79 57.01 58.11 ERE (r=16, b=4)
20.0MB 38.40 63.47 59.59 65.10 35.67 75.57 61.22 65.67 57.00 58.09 ERE (r=32,
b=4) 35.0MB 39.00 63.01 59.27 65.36 35.58 75.46 59.85 65.22 56.79 57.84
Pythia-6.9B 26.3GB 36.80 60.48 60.85 63.15 34.39 76.12 62.63 - 54.36 -
Dolly-7B 26.3GB 39.20 63.39 60.77 68.65 40.70 75.08 64.40 - 57.35 -
Pythia-6.9B ∗ 26.3GB 37.20 61.24 60.69 63.85 35.32 76.44 63.30 67.09 56.86
58.14 Dolly-7B ∗ 26.3GB 39.80 63.05 61.64 68.56 39.59 74.43 64.92 62.74 58.86
59.34 ERE (r=4, b=4) 13.0MB 39.80 65.91 62.98 69.12 38.82 77.04 64.92 67.01
59.80 60.70 ERE (r=8, b=4) 19.0MB 39.20 65.78 61.88 69.48 39.59 76.82 65.72
66.39 59.78 60.60 ERE (r=16, b=4) 31.0MB 39.40 65.57 62.27 69.42 40.36 77.04
65.02 66.25 59.87 60.67 ERE (r=32, b=4) 55.0MB 39.40 65.19 62.35 69.20 39.93
76.28 64.80 65.59 59.59 60.34

Table 3:  The comparison between pre-trained fine-tuned, and ERE applied
models on zero-shot evaluation benchmarks. Performance is measured using
normalized accuracy for OpenBookQA, ARC-Easy, HellaSwag, ARC-Challenge, and
PIQA, and accuracy for other tasks, with the average accuracy calculated
across all tasks. (∗) denotes our implementation using the provided model
checkpoints. $r$ and $b$ represent the rank and quantization level,
respectively. In our storage computation, we exclude the word embedding layers
because the shapes of the embedding layers in the fine-tuned model differ from
the pre-trained model.

In this section, we present the evaluation of ERE on the Large Language Model
(LLM), Dolly666https://github.com/databrickslabs/dolly. Dolly is a fully fine-
tuned model based on Pythia [6], utilizing a dataset consisting of
approximately 15k instructions. We evaluate two models with 3 billion (3B) and
7 billion (7B) parameters with 8 zero-shot evaluation tasks using open-source
LM evaluation framework[14].

Table 3 presents the results of applying ERE to the fully fine-tuned LLM
model. It shows that the model with ERE outperforms the original model in
zero-shot evaluation tasks. However, there is a noticeable trend of
performance decline beyond a certain rank. Specifically, for the 7B model, the
performance begins to decline after rank 4, while for the 3B model, it starts
to decline after rank 8. We hypothesize this is because the original purpose
of the dolly is to follow instruction prompts, rather than achieving zero-shot
performances which is the inherent capability of the pre-trained baseline.
Thus, we interpret this phenomenon as ERE enabling the restoration of the
model’s generalizability while preserving its ability to understand
instruction prompts. Supporting this interpretation, Figure 6 demonstrates
that the model with ERE retains the capacity to understand instruction
prompts, whereas the pre-trained baseline does not.

Instruction $\to$ Explain to me Convolutional Neural Networks (CNNs) Response:
Pre-trained $\to$ Response: Response: Response: (Skip) FT (26.3GB) $\to$ CNNs
are a type of neural network that are popular for use in image processing
applications such as computer vision. CNNs are comprised of convolutional and
non-linear layers that are designed to process spatial information within an
image. (Skip) ERE (13MB) $\to$ CNNs are a type of neural network that are
popular in computer vision, as they can perform spatial and temporal
processing of an input image that would be too difficult for other types of
neural networks. (Skip)

Figure 6:  Example of generated samples from instruction prompt. We fix the
random seed and use the default configuration (same top-p and top-k
probability) for sampling. Note that pre-trained model return 51 repetitive
sentences "Response: Response: …". because it has no capability to understand
the instruction prompt, while ERE can understand the instruction prompt and
response like the fully fine-tuned model. The responses over a certain length
are skipped for brevity.

##  6 Discussions and Limitations

###  6.1 Why do we need ERE when we have PEFT?

In this section, we emphasize that while PEFT has shown impressive
performance, there are situations where reverting to full fine-tuning is still
necessary, making ERE a valuable alternative.

Firstly, integrating other PEFT methods that disrupt the computational graph
at inference time can be challenging and costly. Many PEFT methods modify the
flow of computing variables and are specifically designed for a single model
architecture. For instance, integrating prefix tuning [29] with Paint-with-
words [4] or Adapters [16] with ControlNet [50] poses significant difficulties
due to their architectural incompatibilities. In contrast, full fine-tuning
does not introduce any additional complexity in terms of compatibility.

Secondly, by solely relying on PEFT, users may overlook extensive research
conducted under the assumption of full-weight fine-tuning. This research
includes dedicated fields, such as prior-focused continual learning [23, 36,
49, 2], and weight-averaging methods [34, 47, 46, 33].

###  6.2 The Comparison between ERE and LoRA

We undertake a comparative analysis between ERE and LoRA [18], aiming to
delineate the strengths and weaknesses of both methods. Both methods share a
common characteristic in leveraging the low-rank nature of weight residuals
($\Delta\theta$). For a fair comparison, we implement LoRA in Fairseq using
official implementation777https://github.com/microsoft/LoRA. We first fully
fine-tuned the model for each GLUE tasks and compute the total train wall
clock. Then we train LoRA injected RoBERTa-Large model within the same
training time budget. We follow most of the hyperparameters in the LoRA paper,
and initialize the classifier parameters with the same seed and use the same
GPU (V100), framework (Fairseq), and PyTorch version (2.0).

In Table 4, we observe that the overall accuracy of LoRA is comparable to that
of the fully fine-tuned model. However, both ERE and the fully fine-tuned
model achieve better performances. On the other hand, LoRA requires smaller
storage compared to ERE for achieving competitive performance. However, it is
important to note that a direct comparison with ERE and LoRA may be
challenging because the full fine-tuning method updates more layers including
the word embedding layer and bias. The fully fine-tuned model weight can be
stored efficiently with ERE, requiring 6.0% of the original storage while
preserving similar performance and 1.7% for the same performance compared to
LoRA.

As a result, depending on the specific task and requirements, one might choose
LoRA due to its competitive performance and smaller storage requirements.
However, in certain scenarios where maximizing performance is crucial, the
full fine-tuning approach and ERE could be preferred. Furthermore, it is worth
noting that ERE can be applied to weights obtained through LoRA, providing
additional flexibility and options for achieving optimal trade-offs between
performance and storage efficiency. This highlights the versatility and
potential benefits of ERE in combination with PEFT methods.

Method Storage MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Avg. FT 1.36GB 90.1
96.3 92.4 67.8 94.9 92.2 88.8 92.0 89.4 ERE (r=256, b=4) 84MB 90.2 96.1 92.2
68.2 94.7 91.7 88.8 91.8 89.2 ERE (r=64, b=4) 23MB 89.1 95.6 91.9 66.5 94.7
88.7 89.2 91.7 88.4 LoRA (r=8) [18] 3.0MB 90.6 96.2 90.9 68.2 94.9 91.6 87.4
92.6 89.0 LoRA (r=8)∗ 3.0MB 90.3 96.3 91.7 68.6 94.7 90.9 88.4 91.8 89.1 LoRA
(r=8)† 3.0MB 90.3 96.3 89.7 65.8 94.7 90.5 88.4 91.4 88.4

Table 4:  The comparison between ERE and LoRA. (∗) denotes our LoRA
implementation without training time constraint, while (†) indicates LoRA with
an equal training time budget constraint. The newly added linear layer for the
downstream finetuning task is not considered for storage computation.

###  6.3 Limitations

The limitations of our paper include the potential challenges of ERE in
extreme compression scenarios, particularly when the model size is not large
enough. This limitation becomes more prominent with a high number of updates
or a high learning rate relative to the model size. Additionally, while we
demonstrate the effectiveness of our method across various tasks and model
architectures, we have not yet established its applicability in diverse fine-
tuning configurations. We acknowledge the need for further investigation in
these setups as a future research avenue.

##  7 Conclusion

This paper provides a thorough analysis of weights residual and introduces ERE
as an effective approach for efficient storage without sacrificing
performance. Our experiments demonstrate the efficacy of ERE across tasks like
NLU, LM, and image generation. We believe ERE offers a compelling alternative
to PEFT in terms of reducing footprint, due to their flexibility, simplicity,
and competitive performance.

## References

  * [1] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7319–7328, 2021.
  * [2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018.
  * [3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  Advances in neural information processing systems, 33:12449–12460, 2020.
  * [4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [5] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [6] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  arXiv preprint arXiv:2304.01373, 2023.
  * [7] Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang.  Revisiting parameter-efficient tuning: Are we really there yet?  arXiv preprint arXiv:2202.07962, 2022.
  * [8] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  Advances in Neural Information Processing Systems, 2022.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019\.
  * [10] Carl Eckart and Gale Young.  The approximation of one matrix by another of lower rank.  Psychometrika, 1(3):211–218, 1936.
  * [11] Ky Fan and Alan J Hoffman.  Some metric inequalities in the space of matrices.  Proceedings of the American Mathematical Society, 6(1):111–116, 1955.
  * [12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [13] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  * [14] Leo Gao, Jonathan Tow, Stella Biderman, Charles Lovering, Jason Phang, Anish Thite, Fazz, Niklas Muennighoff, Thomas Wang, sdtblck, tttyuntian, researcher2, Zdeněk Kasner, Khalid Almubarak, Jeffrey Hsu, Pawan Sasanka Ammanamanchi, Dirk Groeneveld, Eric Tang, Charles Foster, kkawamu1, xagi dev, uyhcire, Andy Zou, Ben Wang, Jordan Clive, igor0, Kevin Wang, Nicholas Kross, Fabrizio Milo, and silentv0x.  Eleutherai/lm-evaluation-harness: v0.3.0, December 2022.
  * [15] Mary Gooneratne, Khe Chai Sim, Petr Zadrazil, Andreas Kabel, Françoise Beaufays, and Giovanni Motta.  Low-rank gradient approximation for memory-efficient on-device training of deep neural network.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3017–3021. IEEE, 2020.
  * [16] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  CoRR, abs/1902.00751, 2019.
  * [17] Jeremy Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification.  arXiv preprint arXiv:1801.06146, 2018.
  * [18] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * [19] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola.  The low-rank simplicity bias in deep networks.
  * [20] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  Advances in Neural Information Processing Systems, 34:1022–1035, 2021.
  * [21] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [22] Bobak Kiani, Randall Balestriero, Yann LeCun, and Seth Lloyd.  projunn: efficient method for training deep networks with unitary matrices.  In Advances in Neural Information Processing Systems.
  * [23] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [24] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 491–507. Springer, 2020.
  * [25] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * [26] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871–7880. Association for Computational Linguistics, 2020.
  * [27] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski.  Measuring the intrinsic dimension of objective landscapes.  In International Conference on Learning Representations, 2018.
  * [28] Qi Li, Long Mai, Michael A. Alcorn, and Anh Nguyen.  A cost-effective method for improving and re-purposing large, pre-trained gans by fine-tuning their class-embeddings.  Asian Conference on Computer Vision, 2020.
  * [29] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021.
  * [30] Yinghui Li, Jing Yang, and Jiliang Wang.  Dylora: Towards energy efficient dynamic lora transmission control.  In IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pages 2312–2320. IEEE, 2020.
  * [31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [32] Michael Mahoney and Charles Martin.  Traditional and heavy tailed self regularization in neural network models.  In International Conference on Machine Learning, pages 4284–4293. PMLR, 2019.
  * [33] Michael S Matena and Colin A Raffel.  Merging models with fisher-weighted averaging.  Advances in Neural Information Processing Systems, 35:17703–17716, 2022.
  * [34] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * [35] Sangwoo Mo, Minsu Cho, and Jinwoo Shin.  Freeze the discriminator: a simple baseline for fine-tuning gans.  In CVPR AI for Content Creation Workshop, 2020.
  * [36] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner.  Variational continual learning.  In International Conference on Learning Representations, 2018.
  * [37] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  fairseq: A fast, extensible toolkit for sequence modeling.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, 2019.
  * [38] George Pu, Anirudh Jain, Jihan Yin, and Russell Kaplan.  Empirical analysis of the strengths and weaknesses of PEFT techniques for llms.  CoRR, abs/2304.14999, 2023.
  * [39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [40] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [41] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson.  Cnn features off-the-shelf: an astounding baseline for recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813, 2014.
  * [42] Xianghui Sun, Yunjie Ji, Baochang Ma, and Xiangang Li.  A comparative study between full-parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model.  arXiv preprint arXiv:2304.08109, 2023.
  * [43] Hossein Talebi and Peyman Milanfar.  Learned perceptual image enhancement.  In 2018 IEEE international conference on computational photography (ICCP), pages 1–13. IEEE, 2018.
  * [44] Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi.  Powersgd: Practical low-rank gradient compression for distributed optimization.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [45] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  * [46] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In International Conference on Machine Learning, pages 23965–23998. PMLR, 2022.
  * [47] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959–7971, 2022.
  * [48] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He.  Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.  Advances in Neural Information Processing Systems, 35:27168–27183, 2022.
  * [49] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In International conference on machine learning, pages 3987–3995. PMLR, 2017.
  * [50] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [51] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao.  Adaptive budget allocation for parameter-efficient fine-tuning.  In The Eleventh International Conference on Learning Representations.

##  Appendix A Detailed Explanation of the Budget Allocation Algorithm in ERE

In this section, we present a detailed algorithm for the budget-allocation of
our proposed ERE method. As described in §4.1, our main objective is to solve
the following optimization problem for rank variables
$\textbf{r}\in\mathbb{Z}_{\geq 0}^{N}$

$\displaystyle\min_{\boldsymbol{r}}$
$\displaystyle\sum_{i=1}^{N}\sum_{l=r_{i}+1}^{\min(n_{i},m_{i})}\sigma_{l}^{2}$
(Total Squared Error) subject to
$\displaystyle\sum_{i=1}^{N}r_{i}(n_{i}+m_{i})\leq M$ (Budget)

While total budget $M$ can be determined by the user, we find it convenient to
set $M$ based on the prior rank $r$. This approach avoids the need to
determine an appropriate budget $M$ for each model structure, which can be a
cumbersome task. In our algorithm, denoted as ERE(r=$r_{avg}$) in our
experiments §5, we set $M$ based on the input rank $r_{avg}$, such that
$\sum_{i=1}^{N}r_{avg}(n_{i}+m_{i})=M$. This definition allows us to interpret
ERE as a process of reallocating ranks starting from a fixed-rank
approximation of weight residuals.

Since finding an exact solution is generally NP-Complete, necessitating the
use of approximate solutions. We address this challenge using a two-fold
approach. First, we approximate the function
$f_{i}(x)=\sum_{l=x+1}^{\min(n_{i},m_{i})}\sigma_{l}^{2}$ using the spectral
distribution to a suitable log-linear form, i.e.,
$g_{i}(x)=\exp{(a_{i}x+b_{i})}$. This approximation allows us to relax the
problem and solve it in a tractable manner using its simple derivative form.

Next, we utilize the approximated function $g_{i}$ to solve the equation using
the method of Lagrangian multipliers. Assume that $M$ is small enough that the
optimal solution lies at the constraint equality. We define the Lagrangian as
follows:

$L(\boldsymbol{r},\lambda)=\sum_{i=1}^{N}g_{i}(r_{i})+\lambda\left(\sum_{i=1}^{N}r_{i}(n_{i}+m_{i})-M\right),$

where $\lambda$ is the Lagrange multiplier. To find the optimal solution, we
solve the following system of equations:

$\frac{\partial L}{\partial r_{i}}=0,\quad i=1,2,\ldots,N.$

Taking the partial derivative of the Lagrangian with respect to $r_{i}$, we
obtain:

$\frac{\partial L}{\partial
r_{i}}=g^{\prime}_{i}(r_{i})+\lambda(n_{i}+m_{i})=0.$

Substituting this expression for $r_{i}$ into the constraint, we have:

$\sum_{i=1}^{N}(g_{i}^{\prime})^{-1}(-\lambda(n_{i}+m_{i}))(n_{i}+m_{i})=M.$

In the special case of $g_{i}(x)=\exp(a_{i}x+b_{i})$, we have
$(g_{i}^{\prime})^{-1}(x)=\frac{1}{a_{i}}\log{\frac{x}{a_{i}}}-\frac{b_{i}}{a_{i}}$.
Thus, we obtain the following equation:

$\sum_{i=1}^{N}\frac{1}{a_{i}}\left(\log\left(-\frac{(n_{i}+m_{i})}{a_{i}}\right)-b_{i}+\log\lambda\right)(n_{i}+m_{i})=M.$

While solving this equation for $\lambda$ and projecting $r$ back to the
nearest solution provides a good solution, we can further improve it by
considering the maximum rank condition imposed on $r_{i}$, i.e., $0\leq
r_{i}\leq\min(n_{i},m_{i})$. To incorporate this condition, we introduce the
clamped variation by defining $u_{i}$ as follows:

$u_{i}(\lambda)=\text{clamp}\left(\frac{1}{a_{i}}\left(\log\left(-\frac{(n_{i}+m_{i})}{a_{i}}\right)-b_{i}+\log\lambda\right),0,\min(n_{i},m_{i})\right).$

With the remaining variables held constant, we need to find an appropriate
value of $\lambda$ such that
$C(\lambda)=\sum_{i=1}^{N}u_{i}(\lambda)(n_{i}+m_{i})=M$. This can be achieved
through a binary search, as the function $C(\lambda)$ is monotonic with
respect to $\lambda$. algorithm 1summarizes the procedure.

Input : $r_{avg}$ (prior rank),
$\Delta\theta_{1},\Delta\theta_{2},\ldots,\Delta\theta_{N}$ (residual weights)

Output : r (approximate solution)

Initialize $M=\sum_{i=1}^{N}r_{avg}(n_{i}+m_{i})$ for  _$i=1$ to $N$_ do

Compute the singular values
$\sigma_{i,1},\sigma_{i,2},\ldots,\sigma_{i,\min(n_{i},m_{i})}$ from
$\Delta\theta_{i}$;

Compute $f_{i}(x)=\sum_{l=x+1}^{\min(n_{i},m_{i})}\sigma_{i,l}^{2}$ for all
$x\in[0,\min(n_{i},m_{i})]$;

Fit a log-linear model $g_{i}(r)=\exp(a_{i}r+b_{i})$ to approximate
$f_{i}(r)$;

end for

Initialize $\lambda_{\text{min}},\lambda_{\text{max}},\epsilon$ with suitable
values for binary search;

while  _$\lambda_{\text{max}}-\lambda_{\text{min}} >\epsilon$_ do

$\lambda\leftarrow(\lambda_{\text{min}}+\lambda_{\text{max}})/2$;

Compute $C(\lambda)=\sum_{i=1}^{N}u_{i}(\lambda)(n_{i}+m_{i})$;

if  _$C(\lambda) >M$_ then

$\lambda_{\text{max}}\leftarrow\lambda$;

end if

else if  _$C(\lambda) <M$_ then

$\lambda_{\text{min}}\leftarrow\lambda$;

end if

else

break;

end if

end while

for  _$i=1$ to $N$_ do

$r_{i}\leftarrow\text{clamp}\left(\frac{1}{a_{i}}\left(\log\left(-\frac{(n_{i}+m_{i})}{a_{i}}\right)-b_{i}+\log\lambda\right),0,\min(n_{i},m_{i})\right)$;

$r_{i}\leftarrow\text{round}(r_{i})$;

end for

return r;

Algorithm 1 ERE, budget allocation

We would like to emphasize that our objective is based on the heuristic of
minimizing the overall Frobenius norm, and the log-linear approximation of
$f_{i}$ serves as a suitable proxy objective. However, we acknowledge that
this metric function may not be the optimal choice and there might exist
better schemes or more suitable solutions. Exploring alternative metrics and
investigating improved approaches are left as future work to enhance the
performance of the algorithm.

##  Appendix B Ablation Study: Evaluating Rank Allocation and Orthogonal
Projection

In this section, we conduct an ablation study to evaluate the effectiveness of
rank allocation and the influence of orthogonal projection. We build upon the
settings described in §3.1 and §3.2, where we perform experiments on RoBERTa-
Large models and analyze their feature outputs on 4 natural NLU tasks of GLUE
benchmark.

###  B.1 Ablation of Rank Allocation and Prior Rank

First, we conduct an ablation study to examine the effectiveness of layer-wise
rank allocation. We fix the prior rank ERE(r = 128) and vary the prior alpha
$\alpha$ from 0 to 1 to achieve uniformization, as described in §4.2. When
$\alpha=0$, it implies that we strictly follow the rank determined by the
initial solution. On the other hand, setting $\alpha=1$ is equivalent to
setting equal rank $r_{avg}$ for all layers, which is identical to Low-Rank
Approximation settings (LRA) in §5.1.

From the analysis presented in Figure 7, it is evident that the cosine
similarity of the final layer output between the fine-tuned model and its ERE-
approximated counterparts is best aligned for specific values of
$\alpha\in(0,1)$. This finding suggests that neither fully adhering to the
solution nor setting all ranks equally yields optimal results.

In particular, when $\alpha=0.0$, there is a possibility that some layers may
have a rank of zero. This scenario can lead to a significant activation shift
as one progresses through the network layers. To illustrate this phenomenon,
let’s consider a simplified example of an MLP with $N$ layers. Even if all
layers have equal rank requirements, the initial layers inherently possess
greater importance than the later layers. This is because errors originating
in the early layers tend to propagate throughout the network, accumulating and
magnifying in significance as they traverse the subsequent layers.

The imbalance in importance arises from the computational order of the layers,
which is not explicitly taken into account in our algorithm. We treat each
layer independently of others in terms of computational order, neglecting the
inherent importance attributed to the earlier layers. Therefore, the
effectiveness of our method is dependent on incorporating an appropriate value
of alpha, as the objective tends to be misaligned, leading to extreme
solutions that overly emphasize the significance of one layer compared to
others.

However, the analysis presented in Figure 7 demonstrates that the promise of
our algorithm remains intact. With the incorporation of an appropriate value
of $\alpha$, our method consistently outperforms the uniform rank allocation
approach.

Figure 7:  Effectiveness of prior-rank for different values of prior alpha,
$\alpha$ in 4 GLUE tasks.

###  B.2 Stiefel Projection during Dequantization

Figure 8:  Effectiveness of the Stiefel projection. The blue line indicates
applying projection during dequantization, and the red line indicates
otherwise.

Furthermore, we perform an ablation study to evaluate the effectiveness of the
Stiefel projection method. When employing round-to-nearest quantization on the
$U$ and $V$ matrices, the use of small bit quantization may result in the loss
of the orthogonal condition that was previously upheld by these matrices. To
address this issue, we can perform a projection onto the nearest Stiefel
manifold during the dequantization process. We leverage the following lemma
for Stiefel projection [11].

######  Lemma.

Suppose that $X=UP$ is a polar decomposition of a matrix
$X\in\mathbb{R}^{m\times n}$ ($m\geq n$). Then,

$\|X-U\|=\min\\{\|X-V\|\mathrel{:}V\in\mathbb{S}^{m,n}\\}$

From the results presented in Figure 8, we can clearly see the beneficial
impact of the Stiefel projection. During the dequantization phase, applying
the projection to the nearest Stiefel manifold significantly enhances the
quality of weight residual reconstruction in terms of feature similarity.