# Using Audio Data to Facilitate Depression Risk Assessment in Primary Health
Care

Adam Valen Levinson  Yale UniversityUSA ,  Abhay Goyal  Dept. of CS, Missouri
S & TUSA ,  Roger Ho Chun Man  Institute for Health Innovation and Technology
(iHealthtech), NUSSingapore ,  Roy Ka-Wei Lee  Singapore University of
Technology and DesignSingapore ,  Koustuv Saha  University of Illinois Urbana-
ChampaignUSA ,  Nimay Parekh  JivoxUSA ,  Frederick L Altice  Yale
UniversityUSA ,  Lam Yin Cheung  ORG ResearchUSA ,  Munmun De Choudhury
Georgia Institute of Technology USA and  Navin Kumar  ORG ResearchUSA
<EMAIL_ADDRESS>
(2018)

###### Abstract.

Telehealth is a valuable tool for primary health care (PHC), where depression
is a common condition. PHC is the first point of contact for most people with
depression, but about 25% of diagnoses made by PHC physicians are inaccurate.
Many other barriers also hinder depression detection and treatment in PHC.
Artificial intelligence (AI) may help reduce depression misdiagnosis in PHC
and improve overall diagnosis and treatment outcomes. Telehealth consultations
often have video issues, such as poor connectivity or dropped calls. Audio-
only telehealth is often more practical for lower-income patients who may lack
stable internet connections. Thus, our study focused on using audio data to
predict depression risk. The objectives were to: 1) Collect audio data from 24
people (12 with depression and 12 without mental health or major health
condition diagnoses); 2) Build a machine learning model to predict depression
risk. TPOT, an autoML tool, was used to select the best machine learning
algorithm, which was the K-nearest neighbors classifier. The selected model
had high performance in classifying depression risk (Precision: 0.98, Recall:
0.93, F1-Score: 0.96). These findings may lead to a range of tools to help
screen for and treat depression. By developing tools to detect depression
risk, patients can be routed to AI-driven chatbots for initial screenings.
Partnerships with a range of stakeholders are crucial to implementing these
solutions. Moreover, ethical considerations, especially around data privacy
and potential biases in AI models, need to be at the forefront of any AI-
driven intervention in mental health care.

Audio data, Classification, Depression, Primary Care

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Impact of Computing Culture††ccs:
Recognizing and Defining Computational Problems

##  1\. Introduction

During the COVID-19 pandemic, primary health care (PHC) swiftly transitioned
to telehealth. The widespread adoption of telehealth within PHC represented
one of the most significant transformations in healthcare brought about by the
pandemic. Regulatory adjustments triggered by the pandemic reduced global
barriers to telehealth, leading to its widespread expansion. Telehealth is
widely regarded as a valuable resource in PHC, with quality of care comparable
to or exceeding that of traditional in-person care.

Major depressive disorder (MDD), commonly known as depression, is a prevalent
condition within PHC, affecting up to 20% of those seeking care (Moussavi et
al., 2007), adding strain to the healthcare system. Depression is a severe
mental health condition afflicting over 300 million people worldwide,
characterized by feelings of sadness, sleep disturbances, changes in appetite,
social withdrawal, and, in severe cases, even suicidal thoughts. Effective
treatment of depression may lead to improved patient functioning and reduced
healthcare costs. The prolonged untreated duration of depression may
negatively impact its course and outcome. PHC serves as the initial entry
point into the healthcare system, offering a prime opportunity for depression
detection and care initiation. However, even in high-income countries, nearly
half of those with recognized depression fail to receive adequate care,
representing a significant missed opportunity to address the population-level
burden of depression and prevent suicides. Patients generally prefer receiving
depression treatment in primary care, and most individuals with depression
frequently visit primary care providers. In the context of PHC, the shift to
telehealth has notably improved patient attendance for depression diagnosis
and treatment, with fewer cancellations and more attended appointments (Frank
et al., 2021). Despite these advancements, over 50% of potential depression
cases remain undetected in high-income countries. Within PHC, about 25% of
diagnoses are either false positives or false negatives, perhaps partly
attributed to primary care clinicians prioritizing overall decision-making
over mental health diagnoses (Aragonès et al., 2006). Furthermore, various
barriers may hinder the detection and treatment of depression within PHC,
including the misconception that mental disorders are solely the
responsibility of specialists, a lack of appropriate tools and guidelines, and
low clinician confidence in managing depression, and user-level obstacles.
Given the high prevalence and substantial costs associated with depression,
focusing on its detection and management within PHC is imperative. Artificial
intelligence (AI) may be able to reduce depression misdiagnosis within PHC and
improve overall PHC diagnosis and treatment outcomes.

By harnessing AI, we can enhance the capabilities of telehealth within PHC,
equipping clinicians to tackle the intricate challenges of depression
diagnosis and treatment. We proposed a proof-of-concept study involving 24
individuals: 12 with MDD diagnoses and 12 without mental health or major
health condition diagnoses. The aim was to develop a machine learning model
for predicting depression risk using audio data collected from online patient
interactions. Previous research successfully employed audio data for
depression classification (Sardari et al., 2022). Telehealth consultations
often encounter issues with video, such as poor visual connectivity or dropped
calls (Wootton et al., 2020). These issues can negatively affect the
relationship between clients and clinicians, reduce client satisfaction, and
result in various other challenges. Additionally, audio-only telehealth is
more practical for lower-income patients who may lack stable internet
connections for audiovisual telehealth (Karimi et al., 2022). Therefore, our
study focused exclusively on audio data due to its potential for successful
depression classification and the potential absence of video data in some PHC
sessions, particularly those involving lower-income patients. Our study had
two specific objectives: 1) Gather audio data from 24 individuals; 2)
Construct a machine learning model to predict depression risk. The findings
from this study could pave the way for the development of machine learning
models capable of predicting the risk of various mental health conditions
using data from telehealth appointments.

##  2\. Related Work

Telehealth in depression treatment A meta-analysis indicated that there were
no discernible differences in depression severity at post-treatment between
those receiving telehealth and those opting for face-to-face care (Mitchell et
al., 2009). For patients with depression or depressive symptoms, telehealth
services may thus serve as a viable alternative to traditional face-to-face
care. In another study that examined the shift to mental health-centric
telehealth services during the COVID-19 pandemic, researchers explored mental
health treatment attendance rates, the delivery of evidence-based
interventions, and clinical outcomes (Frank et al., 2021). Their investigation
detailed an improvement in patient attendance following the transition to
telehealth. Patients also experienced a decrease in symptoms, and the quality
of care remained consistent, with evidence-based interventions consistently
utilized over time. Broadly, studies indicated that: 1) Telehealth may be a
viable alternative for providing care and treatment to individuals with
depression or depressive symptoms; 2) Through telehealth, patients may
experience significant improvements and reductions in symptoms around
depression. While there were several studies around the use of telehealth in
depression treatment, there was limited work on the use of telehealth in
depression risk assessment within the PCH setting.

AI in depression diagnosis In a recent review article, authors summarized the
utilization of AI tools for the automated detection of depression (Barua et
al., 2022). The authors suggested that AI tools could potentially deliver
favorable results and overcome the limitations associated with traditional
diagnostic approaches. Furthermore, the review proposed the extraction of a
combined set of features, including facial images, speech signals, and visual
and clinical history data, for future work aimed at the automated detection of
depression and/or anxiety disorders in individuals. Other authors sought to
predict depression by employing a novel machine learning pipeline that
incorporated biomedical and demographic features from a cohort of 4,184
undergraduate students (Nemesure et al., 2021). The resulting model
demonstrated an area under the curve (AUC) of 0.67 for depression
classification. Notably, the most influential predictive features for
depression included satisfaction with living conditions and having public
health insurance. These findings highlighted the moderate predictive
performance of machine learning methods when applied to the detection of
depression using electronic health record (EHR) data. Overall, research in
this area detailed that AI tools may be useful for automating the detection of
depression. While there seemed to be multiple studies which used AI in the
prediction of depression, research did not seem to explore the use of such
tools in PCH environments which served mostly marginalized patients. Such
environments may favor depression detection that is facilitated by audio-only
data. To fill this gap, we conducted a pilot study to determine if audio-only
data could predict depression in a telehealth-like setting.

Audio data and depression diagnosis Researchers proposed a method for
depression detection using audio data, based on a convolutional neural network
autoencoder (CNN AE) (Sardari et al., 2022). The CNN AE extracted relevant and
discriminative features from raw audio data, which was then used to classify
people as depressed or non-depressed. The authors also used a cluster-based
sampling technique to reduce bias towards the major class (non-depressed).
Another study developed a model to classify depression using prosodic features
(pitch, tone, rhythm) extracted from voice data (Negi et al., 2018). The model
achieved an average F1 score of 0.93. Researchers also developed a technique
to conduct depression screening from short voice clips (Toto et al., 2020).
They proposed sliding window sub-clip pooling (SWUP), an audio classification
method suited for small datasets. SWUP incorporated elements of deep learning
methods with traditional machine learning to achieve higher classification
accuracy for smaller datasets while still offering the interpretability of
traditional models. SWUP focused on voice sub-clips, thus detecting depression
signals as they occurred during speech. The authors achieved an F1 score of
0.74 when predicting depression using only audio data. Studies broadly
detailed that: 1) Audio-based depression detection is a promising area of
research, as voice data can be easily collected and analyzed using machine
learning; 2) Several techniques for audio-based depression detection exist,
such as CNN AEs and SWUP. There seemed to be limited studies using solely
audio data to facilitate depression diagnosis. To build tools which could
assist clinicians in depression diagnosis within PCH, we developed a proof-of-
concept study to predict depression diagnosis using audio data.

##  3\. Methods

Recruitment We recruited potential participants on Facebook groups (e.g.,
Depression and Anxiety support, Mental Health Awareness and Support), reddit
groups (e.g., r/depression, r/Depression Help), and mailing lists (e.g,
university student groups) focusing on mental health. A brief overview of the
study was provided on these sites, with an email address for participants to
contact for further information. Interested participants then contacted study
staff via email. We had two distinct sets of inclusion/exclusion criteria as
we were trying to recruit two groups of participants: A) Individuals with
depression diagnoses; B) Individuals without any mental health diagnoses. For
A), those eligible met the following criteria: 1) Greater than 18 years old;
2) Received a depression diagnosis from a clinician, and currently undergoing
depression treatment with a clinician; 3) Understood English; 4) Able to
provide informed consent. For B), eligible participants had to meet the
following criteria: 1) Greater than 18 years old; 2) Have never received any
mental health diagnosis, including depression; 3) Currently not receiving
treatment for any mental health condition; 4) Understood English; 5) Able to
provide informed consent. The following was the exclusion criteria for both
groups: Currently seeking or receiving treatment for a major non-mental health
condition, such as cancer, heart disease etc. Study staff explained the study
protocol to interested individuals and the individual was given a chance to
read the consent document and ask questions. If still willing, and if the
participant met inclusion/exclusion criteria, full written informed consent
was received from the individual, and study procedures would begin. For group
A, we asked participants whether they had been diagnosed for depression and
were currently undergoing treatment for depression. Participants who answered
yes to both questions were asked to provide evidence of a formal diagnosis.
Examples of such evidence were documents provided by a clinician indicating a
depression diagnosis or a insurance document indicating a similar diagnosis.
Participants were asked to redact all identifiable information from provided
documents. Once depression diagnosis was confirmed, such participants were
assigned to group A (depression diagnosis). Those who had never been diagnosed
with depression or another mental health condition were assigned to group B
(no mental health diagnosis). Given funding constraints, we recruited a total
of 24 participants (Diagnosed with depression and currently seeking depression
treatment=12; Not diagnosed with depression=12).

Data collection Participants were interviewed through a teleconferencing
software (Zoom) and their audio was recorded. Zoom is commonly used for
telehealth encounters. Participants were first asked a range of demographic
questions: Age; Annual income; Gender identity. The interview procedure was
designed by two clinicians who worked full-time in PCH, with a diverse
caseload. The procedure was developed by the clinicians to be similar to a
generic patient encounter. The clinicians independently drafted a set of
questions they commonly asked new patients. We compiled all of these
questions, 20 in total. We asked all participants these questions in
succession. Examples of such questions: Have you had any procedures or major
illnesses in the past 12 months? What allergies do you have? Have you served
in the military? Are you sexually active? Do you use any kind of tobacco,
illicit drugs or alcohol? What prescription medications, over-the-counter
medications, vitamins, and supplements do you take? As per past work that
assessed depression using voice biomarkers (Liu et al., 2017), we asked
participants to read a short story (McNamee, 2004). In line with similar past
work which developed machine learning techniques to classify depression (Shin
et al., 2021), we collected participant responses to a range of mental health
screening tools: PHQ-9, Beck Depression Inventory, Beck Anxiety Inventory,
Barratt Impulsiveness Scale. Interviews were about half an hour long.
Participants were paid USD25 upon interview completion. The information
obtained was recorded in such a manner that the identity of the human subjects
could not readily be ascertained, directly or through identifiers linked to
the subjects. All collected data was anonymized and encrypted to ensure
privacy and security.

Data processing We adhered to preprocessing steps in previous work (Vázquez-
Romero and Gallardo-Antolín, 2020). We did not need to remove interviewer
voices as the teleconferencing software we used provided interviewer and
interviewee audio files separately. We removed silences with PyAudioAnalysis
(Giannakopoulos, 2015). Given the small number of audio files and their
differing lengths, we cropped all files into shorter segments of four seconds
each.

Feature extraction Using librosa (McFee et al., 2015), we extracted the
following features from all files: Mel frequency cepstral coefficients
(MFCCs); Spectral centroid; Spectral complexity; Zero-crossing rate. MFCC
described the power spectrum contour of an audio signal. The audio signal was
converted into a frequency domain, using the Discrete Fourier Transform (DFT).
Then, the mel-scale was applied to mimic the way humans perceive sound
frequencies. From this mel-scaled spectrum, cepstral coefficients were
determined. The spectral centroid defined the balance point of the spectrum
and is associated with how bright a sound appears. Spectral complexity
evaluated the variety in the frequency spectrum of the sound signals, helping
to differentiate sounds, such as voices, melodies, or background noise. The
zero-crossing rate (ZCR) gauged how frequently a signal shifted from positive
through zero to negative, or vice versa. MFCCs provided 13 features while the
rest provided one feature each, for a total of 16 features per audio file.

Model selection We computed all models by randomly splitting (80:20) the
dataset into a training and a test dataset. We used the training dataset to
find the appropiate machine learning algorithm for our task. To do this, we
used TPOT v0.3, an automated machine learning system that selected and
preprocessed features based on the training data (Olson and Moore, 2016). The
majority of TPOT’s improvements on a range of benchmarks were quite large,
with several ranging from 10%–60% median accuracy improvement over a random
forest analysis. TPOT selected the best machine learning algorithm for our
specific task by considering a wide range of feature selection methods and all
classification algorithms available in the scikit-learn library.

##  4\. Results

Descriptives

Group | Age (M, SD) | Income (M, SD)
---|---|---
Depression diagnosis | 27.83 (6.86) | 37583.33 (41677.24)
No depression diagnosis | 29.83 (5.29) | 95416.67 (103781.64)
Table 1. Participant demographics Group | Male | Female | Non-Binary
---|---|---|---
Depression diagnosis | 6 | 5 | 1
No depression diagnosis | 7 | 5 | 0
Table 2. Participant gender identity

Average length of the interviews was 26.1 minutes and 25.5 minutes for those
with a depression diagnosis, and those without a depression diagnosis,
respectively. As indicated, after removing silences, we cropped files into
four second segments, resulting in 3969 files (Depression diagnosis=1,632, No
depression diagnosis=2,337). We provided participant demographics in Table 1
and Table 2. Age and gender distributions were similar across those with and
without a depression diagnosis. Income appeared to be higher for those without
a depression diagnosis.

Feature | Metric | Depression | No depression
---|---|---|---
MFCC Mean | Mean | -74.43 | -75.08
SD | 7.67 | 7.25
Spectral Centroid | Mean | 0.0785 | 0.0792
SD | 0.0261 | 0.0225
Spectral Complexity | Mean | 37.66 | 29.80
SD | 31.89 | 31.09
Zero Crossing Rate | Mean | 0.0541 | 0.0551
SD | 0.0173 | 0.0199
Table 3. Descriptive statistics for features by group

In Table 3, we presented descriptive statistics regarding features across the
participant groups. There was a statistically significant difference in MFCC
[t(3967)=2.713, p$<$.05] and spectral complexity [t(3967)=7.752, p$<$.001]
between both groups.

Classification

Metric | Depression
---|---
Precision | 0.98
Recall | 0.93
F1-Score | 0.96
Table 4. Performance metrics for predicting depression diagnosis

Over multiple generations, TPOT evaluated various combinations of
preprocessing steps and machine learning algorithms to determine the most
effective pipeline based on cross-validation scores. The internal cross-
validation scores across generations were as follows: Generation 1: 0.9518;
Generation 2: 0.9518; Generation 3: 0.9537; Generation 4: 0.9537; Generation
5: 0.9553. The best pipeline recommended by TPOT achieved an internal cross-
validation score of 0.9553. The indicated pipeline involved a K-Nearest
Neighbors Classifier with StandardScaler preprocessing as the classifier,
with: Number of neighbors: 3, Power parameter for the Minkowski metric: 2. We
evaluated and compared the accuracy of the selected model in predicting
outcomes using the test data. Table 4 indicated high performance of the model
in classifying depression on all metrics.

Findings suggested that AI may have the potential to aid in the detection of
depression risk using only audio data. We note that this conclusion stems from
a preliminary study with a limited sample size. Using machine learning models
that rely solely on audio information could be both cost-efficient and simpler
to incorporate within Primary Health Care (PHC) settings, especially when
compared to models utilizing both audio and visual data. Such audio-focused
approaches might be especially valuable in resource-constrained settings, for
instance, during phone consultations with economically disadvantaged patients.
In areas with unreliable internet access, depending solely on audio data may
ensure consistent depression risk assessment. Telehealth platforms used in PHC
can be seamlessly integrated with tools such as ours that furnish clinicians
with data to assist in diagnosing depression. Incorporating these tools into
standard PHC consultations might enhance the accuracy of depression diagnoses,
even if the primary reason for the visit is unrelated to mental health. For
instance, while discussing a potential flu diagnosis, a clinician might
receive a depression risk assessment based on the patient’s audio cues,
possibly leading to a recommendation for mental health consultation.

##  5\. Discussion

Implications of Findings Our goal was detail a proof of concept machine
learning model for depression risk, using audio data. A strength of our work
was the high model performance, keeping in mind limitations of the small
dataset. We hope that our work will lead to future work where a larger sample
is used to develop tools to predict depression risk on a greater scale.
Findings built on past work, demonstrating the use of audio data for
depression risk prediction, with potential implications for scaling depression
risk classification across marginalized communities. Our findings may lead to
a range of possible tools to facilitate depression risk screening and
treatment. By developing tools to detect depression risk, patients can be
routed to AI-driven chatbots for initial screenings or to provide immediate
assistance when human professionals are unavailable. AI-powered chatbots can
provide 24/7 support and triage patients. Chatbots can be especially helpful
for patients who live in rural or underserved areas, or who have difficulty
accessing care during standard business hours. Chatbots can also be used to
provide information about mental health conditions, treatment options, and
resources. These chatbots can adapted to individual patients, where the
chatbots can recognize and respect cultural nuances, ensuring that care is
culturally sensitive and relevant (O’Hanlon et al., 2023). Broadly, chatbots
could make services more accessible without incurring high costs.

Screening tools may also detect early signs of mental health issues, providing
alerts to clinicians when certain patterns are detected. Depression prevalence
may be heightened during major events, such as the COVID-19 pandemic.
Depression risk detection tools can be used to better understand surges in
mental health services demand, helping stakeholders allocate resources more
effectively as crises unfold. Similarly, AI can analyze data from PCH
telehealth visits to identify at-risk populations and tailor outreach
campaigns, or organize AI-driven virtual mental health workshops that are
personalized for specific community needs. These workshops can help integrate
local mental health practices into standard care, ensuring treatments are
culturally rooted and more acceptable. Depression risk identification tools
may be also used to determine which patients should receive priority care
based on depression severity, allowing clinician schedules to be optimized to
maximize the number of patients seen while minimizing burnout. To implement
these solutions, partnerships with a range of stakeholders, such as tech
companies, local communities, nonprofits, and governments are crucial.
Moreover, ethical considerations, especially around data privacy and potential
biases in AI models, need to be at the forefront of any AI-driven intervention
in mental health care (Goyal et al., 2023; Chen et al., 2022).

Limitations and future work Our study involved 24 individuals, which was a
relatively small sample size. However, our goal was not to generalize
findings, but to demonstrate proof of concept. We recruited participants who
had access to the internet and may have not included people without regular
internet access. We classified participants into two groups but this approach
may not have captured the spectrum of depressive symptoms or other co-existing
mental health conditions. While focusing exclusively on audio data may have
advantages in low-income populations, we did not obtain video data perhaps
useful in diagnosing depression or other mental health conditions. We asked
participants to confirm their depression diagnosis through providing
documentation, but such a technique may lead to inadvertent sharing of private
data.

Our proof-of-concept study had a relatively small sample size, so future work
should collect data from a larger and more diverse group of people. This will
make the model more generalizable and robust. We have focused on audio data,
but integrating multimodal data (audio, video, text) could provide more
insights and improve the model’s accuracy. This could also address the
limitation of missing out on visual cues. Instead of predicting whether or not
someone has depression, future models could predict the severity of
depression, from mild to severe. Such a technique would provide a more nuanced
understanding of the condition. We also plan to extend our analysis to detect
other mental health conditions, such as anxiety and bipolar disorder, based on
audio data. Building these products will require the development of a robust
ethical and regulatory framework for deploying AI tools. This framework must
address concerns about data privacy, consent, and potential biases. We will
work with stakeholders in the responsible AI (RAI) space to develop such
frameworks. As we will be using private and highly confidential data in future
iterations of our tool, we plan to use techniques such as homomorphic
encryption to ensure data privacy. Planned work will also involve
collaborations with clinicians to refer patients, ensuring verification of a
depression diagnosis and minimizing possible loss of personal data. To improve
tool development, we will collect feedback from both clinicians and patients.
We will also conduct longitudinal studies to understand the usefulness and
efficacy of AI tools to predict depression risk. Given rising US healthcare
costs, we plan to collaborate with health economists to undertake a cost-
benefit analysis of implementing AI in primary healthcare for mental health
diagnosis and treatment. This work will aid the rollout of AI systems within
large healthcare institutions. Scaling AI systems across health systems will
require deep clinical expertise and buy-in from clinicians. Thus, we plan to
organize workshops and training sessions for clinicians to familiarize them
with our AI tools and ensure they can make the best use of them in their
practice. A cornerstone of these workshops would be to seek clinician feedback
on tool development and engage clinicians in the core research team (Van et
al., 2023; Nainani et al., 2022). By addressing these future research
directions, we will be able to better facilitate the use of AI tools in the
prediction of depression risk. This could lead to improved detection and
management of depression and potentially other mental health conditions within
the primary healthcare setting.

## References

  * (1)
  * Aragonès et al. (2006) Enric Aragonès, Josep Lluís Piñol, and Antonio Labad. 2006.  The overdiagnosis of depression in non-depressed patients in primary care.  _Family practice_ 23, 3 (2006), 363–368.
  * Barua et al. (2022) Prabal Datta Barua, Jahmunah Vicnesh, Oh Shu Lih, Elizabeth Emma Palmer, Toshitaka Yamakawa, Makiko Kobayashi, and Udyavara Rajendra Acharya. 2022\.  Artificial intelligence assisted tools for the detection of anxiety and depression leading to suicidal ideation in adolescents: a review.  _Cognitive Neurodynamics_ (2022), 1–22.
  * Chen et al. (2022) Keyu Chen, Ashley Feng, Rohan Aanegola, Koustuv Saha, Allie Wong, Zach Schwitzky, Roy Ka-Wei Lee, Robin O’Hanlon, Munmun De Choudhury, Frederick L Altice, et al. 2022\.  Categorizing Memes About the Ukraine Conflict. In _International Conference on Computational Data and Social Networks_. Springer, 27–38.
  * Frank et al. (2021) Hannah E Frank, Nicholas M Grumbach, Selby M Conrad, Julia Wheeler, and Jennifer Wolff. 2021.  Mental health services in primary care: Evidence for the feasibility of telehealth during the COVID-19 pandemic.  _Journal of affective disorders reports_ 5 (2021), 100146.
  * Giannakopoulos (2015) Theodoros Giannakopoulos. 2015\.  pyaudioanalysis: An open-source python library for audio signal analysis.  _PloS one_ 10, 12 (2015), e0144610.
  * Goyal et al. (2023) Abhay Goyal, Nimay Parekh, Lam Yin Cheung, Koustuv Saha, Frederick L Altice, Robin O’hanlon, Roger Ho Chun Man, Chunki Fong, Christian Poellabauer, Honoria Guarino, et al. 2023\.  Predicting Opioid Use Outcomes in Minoritized Communities. In _Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics_. 1–2.
  * Karimi et al. (2022) Madjid Karimi, Euny C Lee, Sara J Couture, Aldren Gonzales, Violanda Grigorescu, Scott R Smith, Nancy De Lew, and Benjamin D Sommers. 2022\.  National survey trends in telehealth use in 2021: disparities in utilization and audio vs. video services.  _US Department of Health & Human Services_ (2022).
  * Liu et al. (2017) Zhenyu Liu, Huanyu Kang, Lei Feng, and Lan Zhang. 2017\.  Speech pause time: A potential biomarker for depression detection. In _2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2020–2025.
  * McFee et al. (2015) Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015.  librosa: Audio and music signal analysis in python. In _Proceedings of the 14th python in science conference_ , Vol. 8. 18–25.
  * McNamee (2004) Gregory McNamee. 2004\.  _North Wind and the Sun: And Other Fables of Aesop_.  Daimon.
  * Mitchell et al. (2009) Alex J Mitchell, Amol Vaze, and Sanjay Rao. 2009.  Clinical diagnosis of depression in primary care: a meta-analysis.  _The Lancet_ 374, 9690 (2009), 609–619.
  * Moussavi et al. (2007) Saba Moussavi, Somnath Chatterji, Emese Verdes, Ajay Tandon, Vikram Patel, and Bedirhan Ustun. 2007\.  Depression, chronic diseases, and decrements in health: results from the World Health Surveys.  _The Lancet_ 370, 9590 (2007), 851–858.
  * Nainani et al. (2022) Yashna Nainani, Kaveh Khoshnood, Ashley Feng, Muhammad Siddique, Clara Broekaert, Allie Wong, Koustuv Saha, Roy Ka-Wei Lee, Zachary M Schwitzky, Lam Yin Cheung, et al. 2022\.  Categorizing Memes about Abortion.  (2022).
  * Negi et al. (2018) Himani Negi, Tanish Bhola, Manu S Pillai, and Deepika Kumar. 2018.  A novel approach for depression detection using audio sentiment analysis.  _International Journal of Information Systems & Management Science_ 1, 1 (2018).
  * Nemesure et al. (2021) Matthew D Nemesure, Michael V Heinz, Raphael Huang, and Nicholas C Jacobson. 2021.  Predictive modeling of depression and anxiety using electronic health records and a novel machine learning approach with artificial intelligence.  _Scientific reports_ 11, 1 (2021), 1980.
  * Olson and Moore (2016) Randal S Olson and Jason H Moore. 2016.  TPOT: A tree-based pipeline optimization tool for automating machine learning. In _Workshop on automatic machine learning_. PMLR, 66–74.
  * O’Hanlon et al. (2023) Robin O’Hanlon, Frederick L Altice, Roy Ka-Wei Lee, Jack LaViolette, Gloria Mark, Orestis Papakyriakopoulos, Koustuv Saha, Munmun De Choudhury, and Navin Kumar. 2023\.  Misogynistic extremism: a scoping review.  _Trauma, Violence, & Abuse_ (2023), 15248380231176062.
  * Sardari et al. (2022) Sara Sardari, Bahareh Nakisa, Mohammed Naim Rastgoo, and Peter Eklund. 2022.  Audio based depression detection using Convolutional Autoencoder.  _Expert Systems with Applications_ 189 (2022), 116076.
  * Shin et al. (2021) Daun Shin, Won Ik Cho, C Hyung Keun Park, Sang Jin Rhee, Min Ji Kim, Hyunju Lee, Nam Soo Kim, and Yong Min Ahn. 2021\.  Detection of minor and major depression through voice as a biomarker using machine learning.  _Journal of clinical medicine_ 10, 14 (2021), 3046\.
  * Toto et al. (2020) Ermal Toto, ML Tlachac, Francis Lee Stevens, and Elke A Rundensteiner. 2020.  Audio-based depression screening using sliding window sub-clip pooling. In _2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 791–796.
  * Van et al. (2023) Tran Hien Van, Abhay Goyal, Muhammad Siddique, Lam Yin Cheung, Nimay Parekh, Jonathan Y Huang, Keri McCrickerd, Edson C Tandoc Jr, Gerard Chung, and Navin Kumar. 2023\.  How is Fatherhood Framed Online in Singapore?  _arXiv preprint arXiv:2307.04053_ (2023).
  * Vázquez-Romero and Gallardo-Antolín (2020) Adrián Vázquez-Romero and Ascensión Gallardo-Antolín. 2020\.  Automatic detection of depression in speech using ensemble convolutional neural networks.  _Entropy_ 22, 6 (2020), 688.
  * Wootton et al. (2020) Angie R Wootton, Caravella McCuistian, Dominique A Legnitto Packard, Valerie A Gruber, and Parya Saberi. 2020\.  Overcoming technological challenges: Lessons learned from a telehealth counseling study.  _Telemedicine and e-Health_ 26, 10 (2020), 1278–1283.


# SCALM: Towards Semantic Caching for Automated Chat Services with Large
Language Models

Jiaxing Li  Chi Xu  Feng Wang  Department of Computer and Information Science,
University of Mississippi, Mississippi, USA  Isaac M von Riedemann  Cong Zhang
Jiangchuan Liu

###### Abstract

Large Language Models (LLMs) have become increasingly popular, transforming a
wide range of applications across various domains. However, the real-world
effectiveness of their query cache systems has not been thoroughly
investigated. In this work, we for the first time conducted an analysis on
real-world human-to-LLM interaction data, identifying key challenges in
existing caching solutions for LLM-based chat services. Our findings reveal
that current caching methods fail to leverage semantic connections, leading to
inefficient cache performance and extra token costs. To address these issues,
we propose SCALM, a new cache architecture that emphasizes semantic analysis
and identifies significant cache entries and patterns. We also detail the
implementations of the corresponding cache storage and eviction strategies.
Our evaluations show that SCALM increases cache hit ratios and reduces
operational costs for LLMChat services. Compared with other state-of-the-art
solutions in GPTCache, SCALM shows, on average, a relative increase of 63% in
cache hit ratio and a relative improvement of 77% in tokens savings.

###### Index Terms:

large language model (LLM), chatbots, cache storage, vector quantization,
semantic search.

##  I Introduction

Large Language Models (LLMs) have rapidly advanced as a transformative
technology in recent years. Models such as GPT111https://openai.com/gpt-4,
LLaMA222https://ai.meta.com/llama/, and
Alpaca333https://crfm.stanford.edu/2023/03/13/alpaca.html have shown the
capability to generate natural language text and enabled a wide variety of
downstream applications, including chatbots, language translation, and
creative writing. Among these, LLM-based automated chat services, also known
as LLMChat services, stand out for their popularity and impact.
ChatGPT444https://openai.com/blog/chatgpt is a prominent example of this
category. A recent survey indicates that ChatGPT has more than 180.5 million
active users [1]. By August 2023, the daily query volume for ChatGPT surged to
over 60 million, subsequently leading to increased system complexity and
operational costs.

As mainstream LLM research continues to emphasize scaling up these models, the
complexity of systems inevitably increases. Compounded by the surging number
of users, the challenges related to inference performance have risen. There
have been efforts such as, [2, 3, 4] using cache systems for LLMChat services
to mitigate these challenges. These cache systems store dialogues, including
user queries and LLM responses. LLMChat services, whether operating their own
LLMs or utilizing public ones, benefit from cache hits through reduced
processing needs. A key metric for LLMChat services is the number of tokens
processed. In this context, a token represents a unit of text within a query,
serving as an indicator of computational workload. This metric, connected to
GPU usage or the expenses of forwarding queries to public LLMs, is important
to the financial sustainability of LLM-based automated chat services.

Figure 1: General caching workflow for LLMChat services.

Initially, both industry and academia had adopted the prevalent Key-Value (KV)
cache architecture to manage LLM query caches. As shown in Figure 1, this
architecture typically involves query-to-vector conversion and using embedding
vectors as keys for approximate searches. When a query does not find a
matching answer in the cache, it prompts the LLM to produce a new inference.
Conversely, if there is a matching answer already in the cache, that answer is
immediately fetched and delivered to the user.

In exploring cache design for LLMChat services, existing researches [2, 3]
assumed that the cache will operate ideally with perfect lookup performance.
These assumptions, however, may not accurately reflect real-world scenarios.
For example, the design of cache keys can lead to query mismatches. This is
particularly evident since embedding vectors often struggle with representing
long-text queries [4]. To our knowledge, the performance of caches under real-
world query workloads has not been thoroughly investigated. Additionally, the
potential for leveraging semantic connections between queries to inform cache
design remains largely unexplored. Hence, we pose the following research
problem: “Can cache design for LLMChat services be optimized by leveraging
semantic analysis of human-to-LLM dialogues in real-world scenarios?" We
anticipate that the answer will provide valuable insights into the design of
cache storage and eviction strategies for LLMChat services.

To answer the research problem, we initiate a data-driven analysis leveraging
real-world human-to-LLM interaction datasets, notably LMSYS [5] and MOSS [6].
Through this analysis, we uncover a series of unique findings:

  * •

There exist queries and semantic patterns that are frequently visited by human
users.

  * •

It is challenging to identify those queries that have the potential for
improvements and their semantic patterns that will bring cost savings.

  * •

The commonly used cache hit ratio metric is not always effective in
calculating cost savings.

Motivated by our real-world data analysis, we propose SCALM555shortened for
Semantic Caching for Automated Chat Services with Large Language Models, the
first semantics-oriented cache architecture dedicated for LLMChat services. In
SCALM, we highlight the analysis and clustering of queries to identify those
semantic patterns that will bring cost savings. Further, SCALM selectively
caches the queries, utilizing the ranks of their underlying semantic patterns.
Meanwhile, the architecture closely monitors cache space limits and running
states, and dynamically adjusts its cache storage and eviction strategy.

We have implemented and conducted evaluations of the SCALM prototype. The
experiment results show an average increase of 63% in cache hit ratio and an
average improvement of 77% in tokens savings. All these results are compared
with the baselines adopted in the GPTCache framework, which are considered
state-of-the-art solutions. Furthermore, the experiment results also show the
potential of SCALM to adapt to different cache space limits and conversation
scales.

Our contributions can be summarized as follows:

  * •

To our best knowledge, this study marks the first to use real-world human-LLM
interaction data to understand the query cache performance of LLMs.

  * •

Based on the findings, we for the first time propose a semantics-oriented
cache architecture dedicated to LLM-based automated chat services.

  * •

We propose two hierarchical semantic clustering methods to identify
significant query and answer (Q&A) entries and their underlying semantic
patterns.

  * •

In addition to the widely used hit ratio, we further identify a new metric,
namely, total token saving ratio, to better measure the performance of the
query cache for LLMs with realistic cost saving considerations.

  * •

Through extensive experiments, we quantitatively demonstrate the significant
improvements in cache performance and cost savings achieved by SCALM.

The remainder of this paper is outlined as follows: Section II reviews related
works; Section III presents a comprehensive real-world data-driven analysis,
discussing the limitations of current cache designs and possible solutions;
Section IV introduces the SCALM architecture and its design details; Section V
elaborates on prototype implementation details and performance evaluation;
Section VI discusses some possible further refinements; Section VII concludes
the paper.

Table I: Hit evaluation with different cosine similarity threshold. Similarity |  Query 1 |  Query 2 |  GPT-4 Evaluation
---|---|---|---
0.87 |  How can I develop a creative approach to problem-solving? |  How should I approach problem solving with a team? |  Both questions deal with problem-solving, but one on individual creativity and the other on team collaboration.
0.90 |  How can I develop better organizational leadership skills? |  What methods should I use to develop my leadership skills? |  Both questions are about improving leadership skills but may require slightly different answers for specific contexts.
0.93 |  What methods can I use to monitor customer satisfaction? |  What are the best ways to measure and improve customer satisfaction? |  Both questions center around customer satisfaction, one on monitoring and the other on measuring and improving, but similar enough for the same or similar answers.
0.96 |  What strategies should I use to optimize the performance of my team? |  What techniques can I use to optimize my team’s performance? |  Both questions deal with optimizing team performance, similar enough to interchangeably use the answers.

##  II Related Works

Cache and eviction strategies. Conventional cache strategies [7, 8, 9, 10, 11,
12], such as LRU and LFU, are widely adopted in Internet caching and content
delivery networks (CDNs). They primarily utilize historical access data to
inform caching decisions. Although these strategies were proved to be
effective, they can be less adaptable in managing dynamic and complex data
streams. In response to these limitations, machine learning-based cache
strategies [13, 14, 15, 16] were introduced. These strategies employed deep
neural networks to predict future access patterns. They showed advantages in
adaptability and prediction accuracy, especially when dealing with complex
contexts.

Recently, caching strategies tailored for LLMs have emerged [17, 18], focusing
on content reuse during the inference phase. These strategies aimed to
optimize the internal data flow within LLMs, minimizing delays and lowering
resource consumption. GPTCache [4] is recognized in the industry for its
simplicity and flexibility, providing various options for integration with
different LLMChat services. However, it does not offer advanced cache
performance enhancements. In contrast, our work focuses on semantics-oriented
enhancement, aiming to improve cache efficiency by leveraging the semantic
understanding of queries.

Cost-effectiveness of LLM applications. There are various solutions for
managing LLM applications cost-effectively. Model quantization [19, 20] and
pruning [21] aimed to reduce computational requirements and storage footprint,
at the expense of a slight performance trade-off. These techniques are complex
to implement in practice and typically require specialized hardware support.

LLM distillation offered another way to reduce the operational costs for
LLMChat services providers [2, 3]. This process involved training a smaller,
more efficient model to replicate the behavior of a larger, more complex
model. In [2], the authors presented a framework to cache LLM responses and
use those to train a local, inexpensive model. FrugalGPT [22] empirically
explored different methods for handling incoming queries. For example, it used
prompt engineering to shrink or concentrate user queries before forwarding
them to public LLMs. Another method to reduce inference costs is model
multiplexing, which involves selecting models of different sizes to address
different queries. Some recent works [4, 23] have discussed the criteria for
such selections.

Notably, these methods—model distillation, prompt engineering, and model
multiplexing—are not mutually exclusive, with research often exploring their
combination or cooperative use. Our research focuses on enhancing cache
performance for LLMChat services, proposing an improved semantic cache that
could complement these existing cost-reduction strategies.

##  III Motivation and Observation

Aiming at gaining a deep understanding of cache performance in real-world
scenarios, we conduct a data-driven analysis using human-to-LLM interaction
datasets. In particular, we perform query-level cache searches with dataset
entries to examine performance. We start by introducing the datasets, cache
settings, and performance metrics utilized in this work.

###  III-A Preliminary

We have investigated several datasets aligned with our interests, including
LMSYS [5], MOSS [6], ShareGPT [24], and OpenAssistant [25]. They are
frequently used in prior research for model training, fine-tuning, and
benchmarking LLMChat performance. During our preliminary investigations, we
found that the Q&A pairs in OpenAssistant are manually created rather than
LLM-generated. Additionally, while the ShareGPT data is generated by LLMs,
some consist of conversation snippets lacking full context due to hand-picked
contributions. This poses a challenge in effectively characterizing real-world
human-to-LLM interactions using OpenAssistant and ShareGPT. Hence, for
subsequent analysis, we focus on the LMSYS and MOSS datasets.

As the next step, we develop a fast query-level simulator and integrate it
into GPTCache [4], a widely used industrial cache implementation. The
simulator takes queries from dataset records as inputs and conducts cache
searches and replacements. If there is a cache miss, the simulator will
provide answers directly from dataset records, emulating the LLM’s response
without issuing actual queries. To ensure the performance of cache search, we
use the latest embedding model to convert queries into semantic embedding
vectors and then calculate the cosine similarity between these vectors. The
semantic embedding model is text-embedding-3-small, released by OpenAI in Jan
2024 [26].

In existing KV cache implementations, the similarity search is based on a
fixed threshold to determine if two queries are matched [3, 4]. A higher
similarity threshold results in stricter matches but reduces the number of
matched queries. To determine the threshold compatible with text-
embedding-3-small, we prompt GPT-4 and test with MOSS and LMSYS data. Typical
samples are listed in Table I. In both of the two datasets, we observe that a
similarity threshold of 0.90 strikes a balance between answer correctness and
reuse utilization.

Figure 2: Round distribution of the MOSS and LMSYS datasets. Listing 1: A
multi-round conversation.

⬇

Round 1:

Query 1: Who is Elon Musk?

LLM Answer 1: Elon Musk is a business magnate, industrial designer, and ...

Round 2:

Query 2: When did he found SpaceX?

LLM Answer 2: Elon Musk founded SpaceX in March 2002. The company was started
with the goal of reducing space transportation costs to enable the
colonization of Mars.

Round 3:

Query 3: What do you think of him?

LLM Answer 3: Elon Musk is widely regarded as a visionary and influential
figure in technology and entrepreneurship.

Listing 2: Similar queries in the MOSS dataset.

⬇

Example 1:

Query 1: How can AI help in the field of healthcare?

Query 2: How can AI technology be used in the healthcare industry to improve
patient outcomes?

Query 3: Can you explain how AI can be used in the healthcare industry to
improve patient care?

Example 2:

Query 1: Can you explain the difference between a traditional IRA and a Roth
IRA?

Query 2: Can you explain the difference between a traditional IRA and a Roth
IRA in more detail?

Query 3: Can you explain to me the difference between a traditional IRA and a
Roth IRA?

Table II: LMSYS token count distribution. Token Count of each Q&A pair | [42, 1.1K] | [1.1K, 2.19K] | [2.19K, 3.27K] | [3.27K, 4.23K] | [4.23K, 5.42K] | [5.42K, 6.49K]
---|---|---|---|---|---|---
Percent | 70.9% | 22.3% | 4.9% | 0.7% | 0.1% | <$0.1\%$
Figure 3: Hit ratios of different categories in MOSS dataset.

###  III-B Data-Driven Analysis

To initiate the data-driven analysis, we conduct a thorough examination of the
MOSS and LMSYS datasets. The MOSS dataset comprises 1 million conversations
between humans and LLMChat services, averaging 6.7 rounds per conversation.
Conversely, LMSYS consists of 300,000 conversations, averaging 1.8 rounds
each. We present the round distribution in Figure 2. A typical multi-round
conversation is shown in Listing 1. The conversation contains three queries
centered around a single topic.

We also conduct cosine-similarity-based comparisons between all queries in the
two datasets and confirm that similar queries exist in real-world human-to-LLM
conversations. In particular, we find that 4.5% of the queries in MOSS and
7.5% of the queries in LMSYS can be answered with responses from similar
queries. Examples of such similar queries in MOSS are provided in Listing 2.

Furthermore, we benchmark the cache performance by randomly sampling 1,000
conversations from the datasets, with the cache size set to 100 Q&A pairs,
which follows the settings in [3]. We also vary the scale of the conversations
and cache size, and the results remain consistent. In our experiments, we find
that the cache hit ratios of MOSS queries and LMSYS queries are 3.8% and 6.4%,
respectively. We notice that the MOSS dataset includes default categories such
as Brainstorming (BRS), Complex Instructions (CPI), Writing (WRT), among
others. Figure 3 shows the hit ratios in specific categories. The hit ratios
of queries in both BRS and WRT are around 13%, much higher than the hit ratio
measured with all queries. The results align with our expectations and suggest
that specific semantic categories contain more similar queries that can be
responded to with the same answer. Hence, these semantic categories or
patterns should be given more significance when designing a cache architecture
for LLMChat services. As for LMSYS, in [5], the dataset contributor reported a
tentative categorization using a semantic clustering method. However, the data
entries lack attached labels.

We have another significant observation in our experiments: Q&A pairs exhibit
varying counts of tokens. It is intuitive that some queries and answers are
longer than others. We calculate the average count of tokens in queries to be
69.5 and in answers to be 214.5. To better grasp the variance of token counts,
we present the LMSYS token count distribution in Table II, showing that
approximately 70% Q&A pairs have a token count between 42 and 1.1K. Notably,
30% Q&A pairs have a token count exceeding 1.1K. This substantial variance
implies that the token count should also be regarded an important metric when
calculating the cost savings resulting from cache hits. For instance, caching
longer but less frequently asked questions may sometimes yield greater cost
savings than caching shorter, more frequently asked ones.

###  III-C Challenges and Opportunities

Based on our data-driven analysis, we can summarize the challenges and
opportunities in improving cache design for LLMChat services as follows:

Challenge 1: Identifying significant Q&A entries. A fundamental issue in LLM
cache design is identifying those significant Q&A cache entries that lead to
cost savings. Our data-driven analysis reveals the non-trivial nature of this
task. Moreover, the evaluation becomes more challenging when faced with a new,
previously unseen query.

Opportunity: During our data-driven analysis, we mention that the Q&A pairs of
MOSS are organized into categories. The contributor of LMSYS also attempted to
perform semantic clustering to identify categories or patterns of the Q&A
pairs. Semantic analysis, a common approach in Neural Language Processing
(NLP) research, is adept at extracting semantic patterns or categories from
conversation data. Leveraging these semantic patterns or
categories666“Semantic patterns” and “semantic categories” are used
interchangeably in the remainder of this paper. provides additional insight
into their significance. By ranking patterns based on cost-saving metrics, we
can more efficiently prioritize cache storage and eviction decisions. When
encountering a new query that has not been seen before, it becomes easier to
determine whether the query should be cached by comparing it with existing
semantic patterns. Without such patterns, evaluating the new query against all
existing queries becomes computationally prohibitive.

Challenge 2: Defining cost-saving metrics. Our analysis reveals that the hit
ratio alone is not always effective in evaluating the cost-saving performance
of the cache. Due to the variance in token counts among Q&A pairs,
prioritizing short queries in the cache system might increase the hit ratio
metric, but the actual gain in cost savings could be marginal. Conversely,
caching more complex and longer queries can sometimes lead to greater cost
savings. These observations indicate that a complementary evaluation metric is
needed to help identify which queries merit caching.

Opportunity: We propose using both hit ratio and token saving ratio as metrics
for evaluating cost savings. The token saving ratio is calculated by comparing
the count of tokens that need to be processed by LLMs when a query is not
cached versus when it is cached.

##  IV Semantics-Oriented Cache

###  IV-A Design Overview

To tackle these challenges, we propose SCALM, a semantics-oriented cache
architecture designed for LLM-based automated chat services. SCALM emphasizes
the analysis and clustering of queries to identify cost-saving semantic
patterns, selectively caching these queries based on their semantic ranks.
Additionally, the architecture continuously monitors cache capacity and
operational state, dynamically adjusting its caching and eviction strategies
accordingly. Figure 4 shows an overview of the SCALM architecture. The major
components include semantic analysis, storage and eviction strategies, and
enhanced similarity evaluation.

Semantic Analysis: We propose two algorithms based on semantic clustering to
pinpoint significant semantic categories/patterns and to identify queries that
bring cost savings. These two algorithms also facilitate query comparison and
the initialization and updating of patterns, enhancing semantic understanding
in real-world use.

Storage and Eviction Strategies: The strategy integrates cache storage and
replacement with semantic pattern information, taking into account cache
state, query frequency, round, and pattern significance. It ensures informed
cache decisions that optimize storage utilization and eviction processes.

Similarity Evaluation: We enhance the similarity evaluation methods with
semantic preprocessing of queries, such as removing excessive stop words and
employing cutting-edge embedding models such as OpenAI’s ada-002 and text-
embedding-3-small [26].

Figure 4: An overview of the SCALM architecture. Table III: Summary of key notations. Notations |  Explanations
---|---
$C$ |  Set of conversations
$E$ |  Embedding model used for vectorization
$R$ |  Max number of rounds in conversations
$K$ |  Number of semantics patterns
$P$ |  Set of semantics patterns
$MP$ |  Set of meta information of each pattern
$T_{s}$ |  Token savings ratio threshold
$T_{e}$ |  Dropout threshold for patterns
$T_{p}$ |  Max number of cluster patterns for each round

Data: conversations $C$, embedding model $E$, maximum rounds $R$, number of
pattern $K$

Result: semantic patterns $P=\\{p_{(1,1)},p_{(1,2)},...,p_{(R,K)}\\}$, meta
pattern information $MP=\\{mp_{(1,1)},mp_{(1,2)},...,mp_{(R,K)}\\}$

1 Initialize $P\leftarrow\emptyset$;

2 Initialize $MP\leftarrow\emptyset$;

3 for  _$r\leftarrow 1$ to $R$_ do

4 Embed queries of round $r$ in $C$ using $E$;

5 Perform semantic clustering and divide embedded vectors into $K$ patterns;

6 Append $K$ patterns to $p_{(r,:)}$;

7 Calculate pattern token saving ratio for each $\hat{p}\in p_{(r,:)}$;

8 foreach  _$\hat{p}\in p_{(r,:)}$_ do

9 Append pattern meta information and token saving ratio of $\hat{p}$ to
$mp_{(r,:)}$;

10

11 end foreach

12 Rank $mp_{(r,:)}$ in decreasing order;

13

14 end for

Algorithm 1 Comprehensive Hierarchical Semantic Clustering (CO-HSC)

Data: conversations $C$, embedding model $E$, maximum rounds $R$, token
savings threshold $T_{s}$, proportion threshold $T_{e}$, number of patterns
threshold $T_{p}$

Result: semantic patterns $P=\\{p_{(1,1)},p_{(1,2)},...,p_{(R,K)}\\}$, meta
pattern information $MP=\\{mp_{(1,1)},mp_{(1,2)},...,mp_{(R,K)}\\}$

1 Initialize $P\leftarrow\emptyset$;

2 Initialize $MP\leftarrow\emptyset$;

3 for  _round $r\leftarrow 1$ to $R$_ do

4 if  _$r==1$_ then

5 Embed queries of round 1 in $C$ using $E$;

6 Cluster embedded vectors into $T_{p}$ patterns;

7 Append $T_{p}$ patterns to $p_{(r,:)}$;

8

9 else

10 foreach  _$\hat{p}\in p_{(r-1,:)}$_ do

11 Embed queries of round $r$ in $C$ using $E$;

12 Cluster into $\lceil T_{p}/r\rceil$ patterns;

13 Append $\lceil T_{p}/r\rceil$ patterns to $p_{(r,:)}$;

14

15 end foreach

16

17 end if

18 Calculate each pattern token saving ratio in $p_{(r,:)}$;

19 Calculate each pattern proportion ratio in $p_{(r,:)}$;

20 foreach  _$\hat{p}\in p_{(r,:)}$_ do

21 Append pattern meta information and token saving ratio of $\hat{p}$ to
$mp_{(r,:)}$;

22 if  _token saving ratio of $\hat{p}$ $\geq T_{s}$ or proportion ratio of
$\hat{p}$ $\leq T_{e}$_ then

23 Remove $\hat{p}$ from $p_{(r,:)}$;

24

25 end if

26

27 end foreach

28

29 Rank $mp_{(r,:)}$ in decreasing order;

30 if  _$p_{(r,:)}$ is empty_ then

31 break;

32

33 end if

34

35 end for

Algorithm 2 Selective Hierarchical Semantic Clustering (SE-HSC)

###  IV-B Hierarchical Semantic Clustering

Inspired by the LMSYS categorization method, we propose two hierarchical
semantic clustering methods to identify and utilize semantic patterns that can
bring cost savings. We start by introducing key notations. The set of
conversations is denoted by $C$, while $E$ represents the embedding model used
for generating embedding vectors. The maximum number of rounds in a
conversation is given by $R$, with $r$ indicating the current round number.
Let $K$ represent the number of patterns, and let $i$ denote the current
pattern under consideration. We define $P$ as the initial set of semantic
patterns, and $MP$ represents the set of meta information of each pattern.
Furthermore, $T_{s}$ refers to the savings ratio threshold of the token,
$T_{e}$ to the dropout or weight proportion threshold for patterns, and
$T_{p}$ to the maximum number of cluster patterns considered for each round.
These notations form the foundation of our approach to enhancing the
efficiency and effectiveness of conversation analysis algorithms. Frequently
used notations are listed in Table III as references.

The semantic clustering objective, aimed at minimizing the within-cluster sum
of squares, is given by:

$\text{minimize }J=\sum_{r=1}^{R}\sum_{i=1}^{K}\sum_{x\in
S_{(r,i)}}||x-\mu_{(r,i)}||^{2}$ (1)

where:

  * •

$S_{(r,i)}$ represents the set of embedding vectors in the $i$-th semantic
patterns of round $r$,

  * •

$x$ denotes an embedding vector in semantic pattern $S_{(r,i)}$,

  * •

$\mu_{(r,i)}$ is the centroid descriptor of the vectors in $S_{(r,i)}$,

  * •

$||x-\mu_{(r,i)}||^{2}$ is the squared Euclidean distance between an embedding
vector $x$ and the centroid descriptor $\mu_{(r,i)}$ of its semantic pattern.

The centroid descriptor of a semantic pattern is defined as the average of all
embedding vectors assigned to the pattern.

$\mu_{(r,i)}=\frac{1}{|S_{(r,i)}|}\sum_{x\in S_{(r,i)}}x$ (2)

While semantic clustering offers a method to organize queries into clusters,
its direct application to all queries may not be optimal for multi-round
conversations between humans and LLMChat services. The iterative nature of
these interactions necessitates a hierarchical analysis, with each round of
questions treated separately to better capture the evolving context and
intents.

Hence, we propose CO-HSC approach for COmprehensive Hierarchical Semantic
Clustering that accommodates multi-round conversations. The approach, outlined
in Algorithm 1, is structured as follows: In the initial phase, the embedding
model $E$ converts query pairs into vector form (line 5). After embedding, the
semantic clustering algorithm categorizes these vectors into $K$ distinct
patterns (line 6). Patterns are then ranked based on their token saving ratio,
aiming to reduce token consumption and prioritizing cache patterns that offer
greater savings (lines 10 and 12). This iterative process clusters and ranks
patterns across multiple conversation rounds, adapting to the intricacies of
complex dialogues by recognizing unique patterns at each round. The time
complexity of CO-HSC is $O(R\cdot K\log(K))$.

Given that CO-HSC preserves all queries in each round, including those with
less significant contributions to token savings, we propose a SElective
Hierarchical Semantic Clustering method (SE-HSC). This method emphasizes
pruning operations when identifying insignificant semantic patterns. The goal
is to strike a balance between storage efficiency and pattern effectiveness,
SE-HSC achieves this balance by organizing the semantic clusters of each round
into an N-ary tree structure. In this way, the algorithm selectively reduces
the number of insignificant Q&A pairs during the hierarchical semantic
clustering. As depicted in Algorithm 2, the SE-HSC algorithm proceeds as
follows:

Initially, the algorithm embeds all first-round queries of Q&A pairs into
vectors using the embedding model $E$ (line 5). Following the embedding
process, the same clustering method is applied to organize the vectorized
content of the first-round Q&A pairs into $T_{p}$ patterns (lines 6 and 7).
This step aims to identify preliminary patterns within the data, setting the
stage for later rounds. For each pattern identified, the algorithm calculates
the token saving ratio (line 15). These patterns are then stored according to
their token saving ratio. Subsequently, the algorithm stores each pattern
except for those exceeding the savings threshold $T_{s}$ or falling below the
elimination threshold $T_{e}$ in optimized semantic patterns (lines 19 and
20). The meta pattern information $MP$ is stored for future rounds.

The algorithm proceeds to the next rounds and extends selected patterns in an
iterative process. In each round, it evaluates patterns for their token saving
ratio, excluding those that exceed the savings threshold $T_{s}$ or falling
below the proportion threshold $T_{e}$. This iterative process of clustering,
selection, and elimination continues until no patterns meet the criteria for
exclusion or the maximum number of rounds $R$ is completed. The time
complexity of SE-HSC is $O(R\cdot K)$.

Figure 5: Comparative analysis of hit ratios and token saving ratios with
different conversation scales.

Figure 6: Comparative analysis of hit ratios and token saving ratios with
different cache sizes.

##  V Prototype Implementation and Evaluation

In this section, we present the implementation and evaluation of the SCALM
prototype. We begin by introducing the clustering method and parameter
settings for the CO-HSC and SE-HSC algorithms.

###  V-A Method and Parameter Settings

In our exploration of different semantic clustering techniques, we consider
DBSCAN [27], k-means [28], and other grid-based or density-based approaches
[29]. We find that DBSCAN shows superior performance, while k-means offers the
fastest inference time but suffers from inaccuracy issues. Therefore, we
choose the DBSCAN, one of density-based clustering method, for our prototype
implementation.

We test different threshold settings for token savings and the proportion of
patterns when employing the SE-HSC algorithm. To achieve the best performance
with the SE-HSC algorithm, we suggest a criterion that patterns in each round
achieve 20% of the token saving ratio and constitute over 5% of the dataset’s
weight to be considered for further extension and analysis in the next rounds.
This criterion can be further adjusted to fit varying requirements or
contexts. Similar to [5], we initially categorize conversations into 20
distinct clusters to maintain efficiency and comprehensive coverage of
conversation types. This number of clusters can be dynamically adjusted based
on the characteristics of real-world usage.

We also define high, mid, and low ranks to prioritize certain semantic
patterns. The high, mid, and low ranks label the top 25%, 50%, and 75% of
optimized semantic patterns, respectively. This ranking system helps inform
caching and eviction strategies, allowing for the dynamic allocation of
storage towards more significant patterns. Each rank’s threshold is adaptable,
offering flexibility to specific preferences or operational needs.

###  V-B Adaptive Storage Strategy

We use hierarchical semantic clustering to optimize semantic patterns and
guide our cache storage strategy. By considering the cache state and query
pattern ranks, we adaptively update the storage threshold. The process starts
by transforming incoming queries into vectors using the embedding model $E$,
which then categorizes the embedding vectors into existing patterns. Depending
on the cache state, either cold or full, we store queries based on their
pattern ranks. Initially, low-rank patterns are stored during a cold start. As
the cache fills, only queries from mid-rank and high-rank patterns are stored,
tightening the criteria to enhance efficiency. This selective storage ensures
that during peak times, only high-priority queries are retained, focusing on
potential token savings and optimizing cache use.

###  V-C Adaptive Eviction Strategy

Our cache eviction strategy improves upon the LFU method by using optimized
semantic patterns to evaluate the importance and frequency of queries. The
eviction strategy is presented as follows. When a new query arrives, the
system classifies it into predefined semantic patterns and assigns an initial
eviction priority value:

  * •

High-rank pattern queries start with a value of 3.

  * •

Mid-rank pattern queries start with a value of 2.

  * •

All other queries receive a value of 1.

Cache hits also increase the value. In this way, the cache eviction strategy
ensures that more significant queries are less likely to be evicted early.
Once the cache is full, the strategy removes the query with the lowest value.
In the event of a tie, an older query is evicted.

###  V-D Metrics

In evaluating the cache performance, the most commonly used metric is hit
ratio $R_{hit}$, which is defined as follows:

$R_{hit}=\frac{N(\textbf{\text{H}})}{N(\textbf{\text{Q}})}$ (3)

where H is the set of cache hits and Q is the set of queries. $N(\cdot)$
denotes the count of the set entries. Since the cache is designed for LLMChat
services to reduce the number of tokens processed, we propose another
important performance metric: $R_{ts}$, the token saving ratio brought by the
successful cache hits. Intuitively, the count of tokens processed for a query
$q$ is calculated by summing up the count of tokens in the user query and
LLMChat response. Additionally, it is worth noting that the context or the
previous queries and answers are also processed when LLMs handle the current
query. Formally, token saving ratio $R_{ts}$ can be calculated as follows:

$R_{ts}=\frac{\sum\limits_{q\in\textbf{\text{H}}}{C(q)}}{\sum\limits_{q\in\textbf{\text{Q}}}{C(q)}}$
(4)

where $C(\cdot)$ denotes the count of processed tokens by LLMs.

###  V-E Performance Evaluation

We develop a SCALM prototype by integrating semantic analysis algorithms into
GPTCache, referencing the base LFU cache replacement algorithm. Consequently,
we label our methods as CO-HSC-LFU and SE-HSC-LFU and use LFU and LRU as
baselines in GPTCache. Through adjustments to cache size and conversation
scale, we observe performance improvements in real-world experiments: the hit
ratio increased by approximately 5.5% and the token saving ratio by around
4.6%. Considering that the current optimal cache performance on content
distribution is around 20%, these results achieved by SCALM are significant.

Figure 5 shows the performance of CO-HSC-LFU, SE-HSC-LFU, LFU, and LRU across
different experimental scales, ranging from 1,000 to 10,000 conversations,
with five representative cases selected. The figure clearly shows that both of
our strategies, CO-HSC-LFU and SE-HSC-LFU, outperform the baseline methods
across all conversation scales. For example, with a conversation scale of
5000, the cache hit ratios of CO-HS-LFU and SE-HSC-LFU are 11.7% and 11.6%,
respectively, surpassing the baseline LFU by over 4.3%. The token saving
ratios in this case are 8.4% and 8.2% for CO-HSC-LFU and SE-HSC-LFU,
respectively. Typically, CO-HSC-LFU performs slightly better than SE-HSC-LFU.
This is expected as CO-HSC-LFU retains all clusters in semantic analysis,
whereas SE-HSC-LFU uses strategic pruning to improve computational resources
by focusing on key clusters, which sometimes results in a performance trade-
off.

In Figure 6, we show the performance of CO-HSC-LFU, SE-HSC-LFU, LFU, and LRU
across cache sizes ranging from 20 to 200, with five representative cases
shown. The improvements from CO-HSC-LFU and SE-HSC-LFU are stable across
different cache sizes. The results reveal that neither conversation scale nor
cache size impacts the effectiveness of CO-HSC-LFU and SE-HSC-LFU. Both
algorithms consistently outperform the baseline in hit ratio and token saving
ratio, demonstrating their efficiency and robustness.

In Figure 7, we present the relative enhancement ratio of CO-HSC-LFU and SE-
HSC-LFU over the baseline LFU. In this experiment, we fix the conversation
scale at 3,000 and the cache size at 100 and perform the experiments 10 times.
This figure demonstrates that, on average, the hit ratio can achieve
approximately a 63% relative enhancement, while the token saving ratio
exhibits a relative enhancement of around 77%.

We conduct ablation experiments to assess the contributions of components in
the SCALM architecture. Specifically, we analyze the impact of cache
strategies and enhanced query preprocessing (in similarity evaluation) on CO-
HSC-LFU and SE-HSC-LFU individually. Figure 8 illustrates the results,
indicating that both cache strategies and enhanced preprocessing contribute to
the improvements. Cache strategies alone result in relative token-saving
improvements ranging from 35.2% to 38.1%, while enhanced preprocessing yields
enhancements between 46.2% and 48.2%, both compared to the baseline LFU.
Furthermore, the combination of these components amplifies the enhancements,
highlighting SCALM’s focus on token efficiency.

Figure 7: Relative improvement ratio compared to LFU.

Figure 8: Breakdown of improvement ratios.

##  VI Further Discussion

Sub-optimal Cache Performance. Our insights and design are based on empirical
investigations, suggesting the potential for further refinement. Moving
forward, we aim to incorporate semantic analysis into the online optimal cache
algorithm. This optimality arises from analyzing the underlying query
distributions. Belady’s offline optimal algorithm [30] serves as an upper
bound of all cache algorithms. Recent advancements in learning-based cache
algorithms [15] leverage neural networks to learn control policies from
Belady’s. Integrating such a learning-based algorithm could enhance the
robustness of our cache design.

Multimodal Response Caching. Modern LLMs like GPT-4 now support image and
video generation [31, 32, 33, 34]. In some cases, it might take several
minutes to generate a short video clip. For example, we utilize Runway
Gen2777https://research.runwayml.com/gen2 to generate videos from the text.
Gen2 represents one of the top-tier models in today’s text-to-video generation
landscape [35]. We found that a 4-second video took more than 90 seconds. This
extended generation time can often impact user experience, potentially leading
to reduced satisfaction. Recognizing this, the concept of caching multimodal
response emerges as a potential solution. Similarly, when serving multimodal
queries, it is also crucial to reconsider the design and implementation of its
caching mechanism.

Extension to Semantic Graphs. We will further explore semantic graphs as an
alternative to cache storage design [36]. In a semantic graph, nodes represent
dialogue entries, while edges denote extended similarity relationships. Each
node and edge carries attributes that detail its characteristics and
relationships. This graph storage enables the application of subgraph matching
algorithms to efficiently identify patterns or query-specific substructures.
Transforming dialogues into a semantic graph will facilitate a more flexible
search mechanism. It not only enhances the accuracy of matching queries but
also supports fuzzy search capabilities.

Scaling the Semantic Cache. As LLMs serve increasingly complex and numerous
user queries [37], the semantic cache will be distributed across multiple
machines to manage the load effectively. This requires the use of distributed
caching technologies that support fast data retrieval and high availability
[17].

Although our implementation primarily focuses on single-machine settings, it
is extendable to distributed settings. Techniques such as consistent hashing
[4] can be employed to evenly distribute cache data across machines, ensuring
that the system can scale horizontally as demand grows [38]. Regarding hash
key design, organizing data around similar semantic patterns allows queries
related to a specific pattern to be served by the same machine or a smaller
subset of machines. It is straightforward to adapt the semantic patterns
extracted by our hierarchical semantic clustering to serve as hash keys.
Similar Q&A entries belonging to the same semantic pattern will be stored on
the same machine, thereby localizing data access.

##  VII Conclusion

In this work, we conducted real-world data-driven analyses on the cache
performance of automated chat services with LLMs. Based on our findings, we
proposed a new cache architecture named SCALM. This architecture employs
semantic analysis to identify significant cache entries and their semantic
patterns. When a new query arrives, it undergoes extensive comparisons with
the semantic patterns, informing storage and eviction decisions. Performance
assessments indicate that SCALM improves cache efficiency and diminishes
operational expenditures for LLMChat services. On average, there is a 63%
augmentation in cache hit ratio and a 77% reduction in token consumption when
compared to the established benchmarks of the cutting-edge GPTCache framework.

## ACKNOWLEDGEMENTS

This research was partly supported by an NSERC Discovery Grant. The
corresponding authors of this paper are Cong Zhang and Jiangchuan Liu.

## References

  * [1] “ChatGPT Users,” https://explodingtopics.com/blog/chatgpt-users, 2023.
  * [2] I. Stogiannidis, S. Vassos, P. Malakasiotis, and I. Androutsopoulos, “Cache me if you can: An online cost-aware teacher-student framework to reduce the calls to large language models,” _arXiv preprint arXiv:2310.13395_ , 2023.
  * [3] B. Zhu, Y. Sheng, L. Zheng, C. Barrett, M. I. Jordan, and J. Jiao, “On optimal caching and model multiplexing for large model inference,” _arXiv preprint arXiv:2306.02003_ , 2023.
  * [4] F. Bang, “Gptcache: An open-source semantic cache for llm applications enabling faster answers and cost savings,” in _Proc. of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS)_ , 2023.
  * [5] L. Zheng _et al._ , “Lmsys-chat-1m: A large-scale real-world llm conversation dataset,” _arXiv preprint arXiv:2309.11998_ , 2023.
  * [6] T. Sun _et al._ , “Moss: Training conversational language models from synthetic data,” _arXiv preprint arXiv:2307.15020_ , 2023.
  * [7] D. Wessels, _Web Caching_. O’Reilly Media, Inc., 2001.
  * [8] D. Lee, J. Choi, J.-H. Kim, S. H. Noh, S. L. Min, Y. Cho, and C. S. Kim, “Lrfu: A spectrum of policies that subsumes the least recently used and least frequently used policies,” _IEEE Transactions on Computers_ , vol. 50, no. 12, pp. 1352–1361, 2001.
  * [9] T. Johnson and D. Shasha, “2q: A low overhead high performance buffer management replacement algorithm,” in _the 20th International Conference on Very Large Data Bases (VLDB)_ , 1994.
  * [10] Y. Zhou, J. Philbin, and K. Li, “The multi-queue replacement algorithm for second level buffer caches,” in _Proc. of the General Track: 2001 USENIX Annual Technical Conference (ATC)_ , 2001.
  * [11] N. Megiddo and D. S. Modha, “Arc: a self-tuning, low overhead replacement cache,” in _Proc. of the 2nd USENIX conference on File and storage technologies (FAST)_ , 2003.
  * [12] S. Jiang and X. Zhang, “Lirs: an efficient low inter-reference recency set replacement policy to improve buffer cache performance,” in _Proc. of the 2002 ACM SIGMETRICS international conference on Measurement and modeling of computer systems (SIGMETRICS)_ , 2002.
  * [13] M. Xu, M. Zhu, Y. Liu, F. X. Lin, and X. Liu, “Deepcache: Principled cache for mobile deep vision,” in _Proc. of the 24th Annual International Conference on Mobile Computing and Networking (MobiCom)_ , 2018.
  * [14] Z. Song _et al._ , “Learning relaxed belady for content distribution network caching,” in _Proc. of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI)_ , 2020.
  * [15] E. Liu, M. Hashemi, K. Swersky, P. Ranganathan, and J. Ahn, “An imitation learning approach for cache replacement,” in _Proc. of the 37th International Conference on Machine Learning (ICML)_ , 2020.
  * [16] Z. Song _et al._ , “Halp: Heuristic aided learned preference eviction policy for youtube content delivery network,” in _Proc. of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI)_ , 2023.
  * [17] M. Xu, D. Niyato, H. Zhang, J. Kang, Z. Xiong, S. Mao, and Z. Han, “Sparks of gpts in edge intelligence for metaverse: Caching and inference for mobile aigc services,” _arXiv preprint arXiv:2304.08782_ , 2023.
  * [18] Z. Zhang _et al._ , “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” _arXiv preprint arXiv:2306.14048_ , 2023.
  * [19] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in _Proc. of the 40th International Conference on Machine Learning (ICML)_ , 2023.
  * [20] G. Park _et al._ , “Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models,” in _Proc. of the 12th International Conference on Learning Representations (ICLR)_ , 2023.
  * [21] X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” in _Proc. of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2024.
  * [22] L. Chen, M. Zaharia, and J. Zou, “Frugalgpt: How to use large language models while reducing cost and improving performance,” _arXiv preprint arXiv:2305.05176_ , 2023.
  * [23] S. Kim, K. Mangalam, J. Malik, M. W. Mahoney, A. Gholami, and K. Keutzer, “Big little transformer decoder,” _arXiv preprint arXiv:2302.07863_ , 2023.
  * [24] “ShareGPT,” https://sharegpt.com/, 2024.
  * [25] A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum _et al._ , “Openassistant conversations - democratizing large language model alignment,” in _Proc. of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2023.
  * [26] “OpenAI’s embedding models,” https://openai.com/blog/new-embedding-models-and-api-updates, 2024.
  * [27] E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu, “Dbscan revisited, revisited: Why and how you should (still) use dbscan,” _ACM Transactions on Database Systems (TODS)_ , vol. 42, no. 3, pp. 1–21, 2017.
  * [28] A. Ahmad and L. Dey, “A k-mean clustering algorithm for mixed numeric and categorical data,” _Data & Knowledge Engineering_, vol. 63, no. 2, pp. 503–527, 2007.
  * [29] F. Murtagh and P. Contreras, “Algorithms for hierarchical clustering: An overview,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 2, no. 1, pp. 86–97, 2012.
  * [30] L. A. Belady, “A study of replacement algorithms for a virtual-storage computer,” _IBM Systems journal_ , vol. 5, no. 2, pp. 78–101, 1966.
  * [31] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proc. of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [32] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma _et al._ , “Imagen video: High definition video generation with diffusion models,” _arXiv preprint arXiv:2210.02303_ , 2022.
  * [33] C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan, “Nüwa: Visual synthesis pre-training for neural visual world creation,” in _Proc. of the 17th European Conference on Computer Vision (ECCV)_ , 2022.
  * [34] “Video generation models as world simulators,” https://openai.com/research/video-generation-models-as-world-simulators, 2023.
  * [35] “Text-to-video: The task, challenges and the current state,” https://huggingface.co/blog/text-to-video, 2023.
  * [36] J. Zhu, R. Li, X. Chen, S. Mao, J. Wu, and Z. Zhao, “Semantics-enhanced temporal graph networks for content caching and energy saving,” _arXiv preprint arXiv:2301.12355_ , 2023.
  * [37] D. Wu, X. Wang, Y. Qiao, Z. Wang, J. Jiang, S. Cui, and F. Wang, “Large language model adaptation for networking,” _arXiv preprint arXiv:2402.02338_ , 2024.
  * [38] C. Xu, X. Ma, R. Shea, H. Wang, and J. Liu, “Enhancing performance and energy efficiency for hybrid workloads in virtualized cloud environment,” _IEEE Transactions on Cloud Computing_ , vol. 9, no. 1, pp. 168–181, 2018.


* Kristan et al. [2017] Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Zajc, L.C., Vojír, T., Häger, G., Lukežic, A., Eldesokey, A., Fernández, G., Garcia-Martin, A., Muhic, A., Petrosino, A., Memarmoghadam, A., Vedaldi, A., Manzanera, A., Tran, A., Alatan, A., Mocanu, B., Chen, B., Huang, C., Xu, C., Sun, C., Du, D., Zhang, D., Du, D., Mishra, D., Gundogdu, E., Velasco-Salido, E., Khan, F.S., Battistone, F., Subrahmanyam, G.R.K.S., Bhat, G., Huang, G., Bastos, G., Seetharaman, G., Zhang, H., Li, H., Lu, H., Drummond, I., Valmadre, J., Jeong, J.-c., Cho, J.-i., Lee, J.-Y., Noskova, J., Zhu, J., Gao, J., Liu, J., Kim, J.-W., Henriques, J.F., Martínez, J.M., Zhuang, J., Xing, J., Gao, J., Chen, K., Palaniappan, K., Lebeda, K., Gao, K., Kitani, K.M., Zhang, L., Wang, L., Yang, L., Wen, L., Bertinetto, L., Poostchi, M., Danelljan, M., Mueller, M., Zhang, M., Yang, M.-H., Xie, N., Wang, N., Miksik, O., Moallem, P., Venugopal, P.M., Senna, P., Torr, P.H.S., Wang, Q., Yu, Q., Huang, Q., Martín-Nieto, R., Bowden, R., Liu, R., Tapu, R., Hadfield, S., Lyu, S., Golodetz, S., Choi, S., Zhang, T., Zaharia, T., Santopietro, V., Zou, W., Hu, W., Tao, W., Li, W., Zhou, W., Yu, X., Bian, X., Li, Y., Xing, Y., Fan, Y., Zhu, Z., Zhang, Z., He, Z.: The visual object tracking vot2017 challenge results. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 1949–1972 (2017). https://doi.org/10.1109/ICCVW.2017.230
  * Kristan et al. [2019] Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Pflugfelder, R., Kämäräinen, J.-K., Zajc, L.C., Drbohlav, O., Lukezic, A., Berg, A., Eldesokey, A., Käpylä, J., Fernández, G., Gonzalez-Garcia, A., Memarmoghadam, A., Lu, A., He, A., Varfolomieiev, A., Chan, A., Tripathi, A.S., Smeulders, A., Pedasingu, B.S., Chen, B.X., Zhang, B., Wu, B., Li, B., He, B., Yan, B., Bai, B., Li, B., Li, B., Kim, B.H., Ma, C., Fang, C., Qian, C., Chen, C., Li, C., Zhang, C., Tsai, C.-Y., Luo, C., Micheloni, C., Zhang, C., Tao, D., Gupta, D., Song, D., Wang, D., Gavves, E., Yi, E., Khan, F.S., Zhang, F., Wang, F., Zhao, F., Ath, G.D., Bhat, G., Chen, G., Wang, G., Li, G., Cevikalp, H., Du, H., Zhao, H., Saribas, H., Jung, H.M., Bai, H., Yu, H., Peng, H., Lu, H., Li, H., Li, J., Li, J., Fu, J., Chen, J., Gao, J., Zhao, J., Tang, J., Li, J., Wu, J., Liu, J., Wang, J., Qi, J., Zhang, J., Tsotsos, J.K., Lee, J.H., Weijer, J.v.d., Kittler, J., Lee, J.H., Zhuang, J., Zhang, K., Wang, K., Dai, K., Chen, L., Liu, L., Guo, L., Zhang, L., Wang, L., Wang, L., Zhang, L., Wang, L., Zhou, L., Zheng, L., Rout, L., Gool, L.V., Bertinetto, L., Danelljan, M., Dunnhofer, M., Ni, M., Kim, M.Y., Tang, M., Yang, M.-H., Paluru, N., Martinel, N., Xu, P., Zhang, P., Zheng, P., Zhang, P., Torr, P.H.S., Wang, Q.Z.Q., Guo, Q., Timofte, R., Gorthi, R.K., Everson, R., Han, R., Zhang, R., You, S., Zhao, S.-C., Zhao, S., Li, S., Li, S., Ge, S., Bai, S., Guan, S., Xing, T., Xu, T., Yang, T., Zhang, T., Vojir, T., Feng, W., Hu, W., Wang, W., Tang, W., Zeng, W., Liu, W., Chen, X., Qiu, X., Bai, X., Wu, X.-J., Yang, X., Chen, X., Li, X., Sun, X., Chen, X., Tian, X., Tang, X., Zhu, X.-F., Huang, Y., Chen, Y., Lian, Y., Gu, Y., Liu, Y., Chen, Y., Zhang, Y., Xu, Y., Wang, Y., Li, Y., Zhou, Y., Dong, Y., Xu, Y., Zhang, Y., Li, Y., Luo, Z.W.Z., Zhang, Z., Feng, Z.-H., He, Z., Song, Z., Chen, Z., Zhang, Z., Wu, Z., Xiong, Z., Huang, Z., Teng, Z., Ni, Z.: The seventh visual object tracking vot2019 challenge results. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 2206–2241 (2019). https://doi.org/10.1109/ICCVW.2019.00276
  * Kristan et al. [2020] Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J.-K., Danelljan, M., Zajc, L.Č., Lukežič, A., Drbohlav, O., He, L., Zhang, Y., Yan, S., Yang, J., Fernández, G., Hauptmann, A., Memarmoghadam, A., García-Martín, Á., Robinson, A., Varfolomieiev, A., Gebrehiwot, A.H., Uzun, B., Yan, B., Li, B., Qian, C., Tsai, C.-Y., Micheloni, C., Wang, D., Wang, F., Xie, F., Lawin, F.J., Gustafsson, F., Foresti, G.L., Bhat, G., Chen, G., Ling, H., Zhang, H., Cevikalp, H., Zhao, H., Bai, H., Kuchibhotla, H.C., Saribas, H., Fan, H., Ghanei-Yakhdan, H., Li, H., Peng, H., Lu, H., Li, H., Khaghani, J., Bescos, J., Li, J., Fu, J., Yu, J., Xu, J., Kittler, J., Yin, J., Lee, J., Yu, K., Liu, K., Yang, K., Dai, K., Cheng, L., Zhang, L., Wang, L., Wang, L., Van Gool, L., Bertinetto, L., Dunnhofer, M., Cheng, M., Dasari, M.M., Wang, N., Wang, N., Zhang, P., Torr, P.H.S., Wang, Q., Timofte, R., Gorthi, R.K.S., Choi, S., Marvasti-Zadeh, S.M., Zhao, S., Kasaei, S., Qiu, S., Chen, S., Schön, T.B., Xu, T., Lu, W., Hu, W., Zhou, W., Qiu, X., Ke, X., Wu, X.-J., Zhang, X., Yang, X., Zhu, X., Jiang, Y., Wang, Y., Chen, Y., Ye, Y., Li, Y., Yao, Y., Lee, Y., Gu, Y., Wang, Z., Tang, Z., Feng, Z.-H., Mai, Z., Zhang, Z., Wu, Z., Ma, Z.: The eighth visual object tracking vot2020 challenge results. In: Bartoli, A., Fusiello, A. (eds.) Computer Vision – ECCV 2020 Workshops, pp. 547–601. Springer, Cham (2020)
  * Li et al. [2017] Li, Z., Tao, R., Gavves, E., Snoek, C.G.M., Smeulders, A.W.M.: Tracking by natural language specification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * Li et al. [2022] Li, Y., Yu, J., Cai, Z., Pan, Y.: Cross-modal target retrieval for tracking by natural language. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4931–4940 (2022)
  * Zhou et al. [2023] Zhou, L., Zhou, Z., Mao, K., He, Z.: Joint visual grounding and tracking with natural language specification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23151–23160 (2023)
  * Li et al. [2023] Li, X., Huang, Y., He, Z., Wang, Y., Lu, H., Yang, M.: Citetracker: Correlating image and text for visual tracking. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9940–9949. IEEE Computer Society, Los Alamitos, CA, USA (2023). https://doi.org/10.1109/ICCV51070.2023.00915 . https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.00915
  * Brown et al. [2020] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901. Curran Associates, Inc., ??? (2020)
  * Chen et al. [2023] Chen, K., Liu, Z., Hong, L., Xu, H., Li, Z., Yeung, D.-Y.: Mixed autoencoder for self-supervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22742–22751 (2023)
  * Dave et al. [2023] Dave, I.R., Jenni, S., Shah, M.: No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (2023)