# Integrating Large Language Models into a Tri-Modal Architecture for
Automated Depression Classification

Santosh V. Patapati

###### Abstract

Major Depressive Disorder (MDD) is a pervasive mental health condition that
affects 300 million people worldwide. This work presents a novel, BiLSTM-based
tri-modal model-level fusion architecture for the binary classification of
depression from clinical interview recordings. The proposed architecture
incorporates Mel Frequency Cepstral Coefficients, Facial Action Units, and
uses a two-shot learning based GPT-4 model to process text data. This is the
first work to incorporate large language models into a multi-modal
architecture for this task. It achieves impressive results on the DAIC-WOZ
AVEC 2016 Challenge cross-validation split and Leave-One-Subject-Out cross-
validation split, surpassing all baseline models and multiple state-of-the-art
models. In Leave-One-Subject-Out testing, it achieves an accuracy of 91.01%,
an F1-Score of 85.95%, a precision of 80%, and a recall of 92.86%.

###### Index Terms:

Multi-Modal Neural Networks, Deep Learning, Large Language Models, Depression
Diagnosis, Biomedical Informatics

##  I Introduction

Major Depressive Disorder (MDD) is a pervasive mental health condition that
affects 300 million people worldwide [1]. The COVID-19 pandemic has further
exacerbated this issue, leading to a staggering 27% increase in the global
prevalence of MDD [2].

Unlike biological disorders, MDD is not diagnosed through blood tests or
imaging. It lacks these gold standards. Instead, the most common approach for
MDD diagnosis relies on clinical interviews [3] and self-reported inventories
(SRIs) [4, 5]. These systems have been subject to criticism for their
subjectivity [6, 7, 8], which gives way for a number of issues. Bias, for
example, can come from both parties. In clinical interviews and SRIs, patients
may exaggerate or under-report symptoms in social desirability bias [9, 10].
Confirmation bias affects how clinicians weigh a patient’s symptoms against
the assigned criteria [11]. As a result, the misdiagnosis rate of MDD is
estimated to be as high as 54% [12].

In recent years, there has been a growing interest in the use of machine
learning (ML) systems to automatically assess the presence and severity of
MDD. This offers a low-cost and objective alternative to current methods. A
heavily researched use case of ML involves making diagnoses from clinical
interview recordings.

Numerous studies investigate a multi-modal approach, which combines both
verbal and non-verbal cues to reach a diagnosis. These models achieve
extremely high levels of success. Currently, the majority of state-of-the-art
models take in three modes of input: Audio, video, and text-based data.

The text-based modality is generally seen as the weakest link in diagnosing
MDD from clinical interviews [13]. This can be attributed to the scarcity of
task-specific text-based training data. This lack of data hinders the
effective training of NLP models, which are data-intensive, resulting in
suboptimal performance compared to the audio and video modalities.

To date, no research has attempted to integrate Large Language Models (LLMs)
into a multi-modal architecture for this task. Due to their training on large
corpora, we hypothesized incorporating LLMs into such an architecture would
improve the accuracy of depression diagnosis from clinical interview
recordings and alleviate the issue of scarce training data.

In this work, we make the following contributions:

  1. 1.

We propose a novel, tri-modal architecture that utilizes LLMs. This is the
first work to incorporate LLMs into a multi-modal framework for this task.

  2. 2.

We demonstrate that LLMs are effective for mental health diagnosis when
incorporated into multi-modal architectures. The proposed architecture
outperforms baseline and state-of-the-art models tested on the AVEC 2016
Challenge dataset and Leave-One-Subject-Out Cross-Validation. The results of
this work serve as an indicator for the potential of such an architecture in
classifying depressed patients.

  3. 3.

The proposed architecture is integrated into a locally hosted web application
to emulate the potential use of such a model in the real-world.

##  II Related Works

Figure 1: Example of Facial Action Unit descriptors in famous photos [18]. They can code nearly any anatomically possible facial expression. TABLE I: State-of-the-art multi-modal fusion systems for binary depression classification. (*Classification done with Leave-One-Subject-Out Cross-Validation Paper | Architecture | Fusion Level | Reported Performance | Modalities | Features
---|---|---|---|---|---
(Gimeno-Gómez et al.) [19] | Transformer | Late | F1-Score: 72 | Audio | COVAREP [20]
|  |  | Precision: 72 |  | Vocal Tract Resonance Frequencies [21]
|  |  | Recall: 72 | Video | 3D Facial Landmarks
|  |  |  |  | FAUs
|  |  |  |  | Gaze Tracking
|  |  |  |  | Head Pose Landmarks
(Muzammel et al.) [13] | LSTM | Model | Accuracy: 77.16 | Audio | MFCCs
|  |  | Precision: 53 | Video | FAUs
|  |  | Recall: 44 |  |
(Yang et al.) [22] | BiLSTM | Model | Accuracy: 81.10 | Video | FAUs
|  |  | F1-Score: 80.60 | Audio | COVAREP
|  |  | Precision: 80.20 | Text | BERT [23]
|  |  | Recall: 81 |  |
(Othmani et al.) [24] | CNN | Model | Accuracy*: 87.40 | Video | FAUs
|  |  | F1-Score*: 82.30 | Audio | VGGish [25]
(Ceccerali et al.) [26] | BiLSTM | Late | F1-Score: 87 | Audio | Fisher Vector [27]
|  |  | F1-Score: 70 | Video | Fisher Vector
|  |  | Precision: 89 | Text | Bag of Words
(Wei et al.) [28] | ConvBiLSTM | Model | F1-Score: 0.70 | Video | 3D Facial Landmarks
|  |  | Precision: 0.89 |  | Gaze Tracking
|  |  |  | Audio | Log-Mel Spectrogram
|  |  |  | Text | Sentence Embeddings

###  II-A Mel Frequency Cepstral Coefficients in Audio-Based Models

Speech data has been found to be highly effective, more so than video and
text-based data, for diagnosing depression from clinical interview recordings.
In fact, multiple mono-modal audio-based models have achieved results
comparable to SRIs for diagnosing a variety of psychiatric disorders, such as
Generalized Anxiety Disorder (GAD) [15, 16, 14].

Current audio-based diagnosis methods rely on low and high level features
extracted from raw audio data. The extracted features are passed as input into
deep learning models.

Many representations of audio data have been utilized for this task. (Wang et
al.) [17] evaluated and compared the effectiveness of such representations,
finding Mel Frequency Cepstral Coefficients (MFCCs) to be the best performing
audial feature for depression diagnosis of the 30 features which were tested
in the study.

Derived using the Mel Scale [29], MFCCs provide a compact representation of
the human perception of sound. This allows for the low-level representation of
timbre, pitch, and rhythm. Studies indicate such values are different between
depressed and non-depressed patients [30]. (Wang et al.) [31] and (Taguchi et
al.) [32] found certain MFCCs follow a consistent pattern that is only present
in depressed patients.

Currently, MFCCs are the most used feature across state-of-the-art models to
represent the audio mode for this task.

###  II-B Facial Action Units in Video-Based Models

Research has shown facial expressions and movements are significantly
different in depressed patients as opposed to non-depressed ones [33] [34].
Visual-based depression diagnosis models rely on features extracted from
patient videos and images.

Facial Action Units (FAUs), a low-level representation of the movements of
specific facial muscles, were introduced as part of the Facial Action Coding
System [35]. FAUs allow us to objectively study facial expressions and
decipher non-verbal cues (Figure 3). This makes them highly effective and
popular for affective computing tasks [36]. In this study, a subset of 20 out
of 44 total FAUs are used, as shown in Table IV.

###  II-C Multi-Modal Models

Multi-modal models combine both verbal and non-verbal cues to reach a
diagnosis. To our knowledge, the majority of state-of-the-art models for
depression diagnosis from clinical interview recordings consider at least
audio and video data in making a diagnosis.

####  II-C1 Data Fusion Strategies

Figure 2: Visualization of Early Fusion (left), Late Fusion (middle), and
Model-Level Fusion (right) systems [37].

The way in which the modalities are fused is referred to as the fusion
strategy (Figure 2. Two basic fusion strategies which are highly prominent in
machine learning models are Early Fusion and Late Fusion [38].

  * •

Early Fusion: Representations of different modalities are concatenated before
being fed into a neural network. For example, one work combines unprocessed
audio and text features before passing them into a Support Vector Machine.
However, this method can produce a high-dimensional feature representation,
resulting in the ’curse of dimensionality’, where the volume of space
increases exponentially and training data becomes sparse [39].

  * •

Late Fusion: Individual mono-modal models are trained for their respective
modalities and their outputs are concatenated right before the final decision.
(Samareh et al.) [40] employs Late Fusion for MDD diagnosis by evaluating
outputs from mono-modal video, audio, and text-based models using Random
Forests. When Late Fusion techniques are used, however, deeper complexities
and relationships between the different modalities are lost.

Model-Level Fusion is a combination of these two strategies. In Model-Level
Fusion, individual modalities are processed before concatenation. After
concatenation, the data is processed even further to reach an output. This
mitigates the curse of dimensionality and issue of uncaptured complexities
between modalities. It allows patterns within a single modality and
relationships across modalities to be learned effectively. For this reason, it
is the most commonly used and best performing fusion method for multi-modal
depression diagnosis from clinical interviews.

##  III Data Collection and Preprocessing

Figure 3: Visualization of the pre-processing steps applied to DAIC-WOZ data.

###  III-A DAIC-WOZ

The Distress Analysis Interview Corpus - Wizard of Oz (DAIC-WOZ) contains data
collected from 189 clinical interviews ranging between 7-33 minutes [41]. Each
interview contains a 16kHz .wav recording, a .CSV text transcript, an ID
numbered 300 – 492, and features collected from video data. To keep data de-
identified, raw video data is not made available in the DAIC-WOZ. Instead, the
dataset provides six visual features extracted in 30 frames per second using
the OpenFace pose estimation library [42]. The extracted visual features are
as follows: 68 2D Facial Landmarks, 68 3D Facial Landmarks, Histogram of
Oriented Gradients, Facial Action Units (FAUs), 4 Gaze Vectors, and Head Pose.
FAUs are the only visual feature considered in this study.

In the DAIC-WOZ, depression was assessed using an SRI: Patient Health
Questionnare-8 (PHQ-8). The PHQ-8 measures 8 items on a scale of 0 to 3 based
on how often problems have been occuring over a two-week period. The final
output is a binary label and a score measuring the severity of MDD on a scale
between 0-24. A cut-off score of 10 (inclusive) was used for the binary label,
where participants scoring higher were considered depressed. These
classifications were used to develop, train, and evaluate the proposed
architecture.

Access to DAIC-WOZ was granted after signing an End User License Agreement
(EULA)111Zipped DAIC-WOZ files can be found after signing the necessary EULA
at https://dcapswoz.ict.usc.edu/.

###  III-B Dataset Errors

The DAIC-WOZ dataset contains many errors that needed to be resolved before
the regular preprocessing steps could be applied. The following errors were
resolved:

  1. 1.

All audio files contained interactions between research assistants prior to
the interview starting. These pieces of audio were identified and removed.

  2. 2.

The interviews with IDs of 373 and 444 contained long interruptions which had
to be removed. One such interruption, for example, was the phone of a research
assistant ringing.

  3. 3.

The interviews with IDs of 451, 458, and 480 were missing the utterances of
the therapist. This issue was resolved by manually transcribing the therapist
speech.

  4. 4.

The interviews with IDs of 318, 321, 341, and 362 contained transcription
files which were out of sync with the audio. This was resolved by manually
adjusting the timestamps of the affected utterances.

  5. 5.

The interview with ID 409 contained a labeling error, where the PHQ-8 score
was 10 but the binary label was 0. This was manually resolved.

###  III-C Text Preprocessing

TABLE II: Transcript Sample From DAIC-WOZ ID | Speaker | Utterance
---|---|---
0 | Ellie | how_doingV (so how are you doing today)
1 | Participant | good
2 | Participant | it’s been a nice day
3 | Ellie | thats_good (that’s good)
4 | Ellie | where_originally (where are you from originally)
TABLE III: Adjusted Transcript Sample ID | Speaker | Utterance
---|---|---
0 | Therapist | So how are you doing today?
1 | Patient | Good. It’s been a nice day.
2 | Therapist | That’s good. Where are you from originally?

The DAIC-WOZ contains transcripts as tab-separated .CSV files with the
following columns: speaker, start_time, stop_time, and value. The provided
utterances contained numerous errors that needed to be fixed.222Documentation
detailing how the original DAIC-WOZ transcriptions were created can be found
at Preprocessing was applied using RegEx and NLP techniques to make text
suitable to pass into the LLM. Refer to Table II for a sample of a transcript
before pre-processing is applied.

A major issue in the unprocessed transcripts is the presence of unique
identifiers within utterances, as seen in Table II. In the original DAIC-WOZ
dataset, all transcriptions of the clinical interviewer were automatically
generated for participants whose IDs were greater than 363. In such cases, a
unique identifier is followed by the true text in parenthesis. These
identifiers are heavily present in Table. RegEx was applied to identify and
replace unique identifiers with their true meanings.

In the DAIC-WOZ, acronyms which are pronounced letter by letter are connected
by underscores, e.g. ’l_a’ represents ’Los Angeles’. A Python dictionary was
generated, where the keys were such acronyms and the values were their
extended form. All acronyms were replaced with what they represented.

If speech is cut-off, the full word is followed by what was actually
pronounced, enclosed in inequality signs. Non-speech, such as coughing, is
enclosed in inequality signs as well. These issues were resolved by simply
removing text that followed this pattern using RegEx.

The use of proper grammar in LLM prompting can increase classification
accuracy, but the transcripts provided in DAIC-WOZ contain major grammatical
errors. To resolve this issue, the ’t5-base-grammar-correction’ model [43] was
applied to all text through the Hugging Face platform. The model ensured the
proper use of punctuation and capitalization in every utterance. However, the
model is limited in its ability to identify interrogative sentences and apply
punctuation accordingly. A Support Vector Machine was trained to identify if
utterances are interrogative and to apply the appropriate punctuation with an
accuracy of 97%.

The names of diarized speakers were adjusted. ’Ellie’ was replaced with
’Therapist’ and ’Participant’ was replaced with ’Patient’. Using such naming
conventions gave the LLM a better understanding of a conversation’s context
and the speakers’ respective roles.

Following all corrections, subsequent utterances by the same speaker were
merged together. Refer to Table III for a sample of a fully adjusted
transcript.

The resulting .CSV files were converted into raw text data which could be fed
into the LLM. In this raw text format, subsequent utterances were separated by
a new-line. Speakers were diarized by preceding utterances with the speaker
name, followed by a colon (e.g., ’Therapist: So how are you doing today?’).

###  III-D Audio Preprocessing

Audio preprocessing followed four stages. (1) Audio was segmented to only
include patient speech. (2) Audio data was augmented to increase the number of
total samples by a factor of seven. (3) MFCCs were extracted using Librosa
[44]. (4) The MFCCs were normalized using Cepstral Mean and Variance
Normalization.

####  III-D1 Audio Segmentation

In the proposed model, as well as past research, the audio modality is
incorporated by analyzing patient responses to questions. Patient speech was
separated from therapist speech by slicing audio according to the timestamps
provided in the transcript, which are detailed down to the centisecond. The
cropped therapist audio was discarded and not used as input into the final
architecture.

The remaining audio data was separated into smaller chunks to make it suitable
for input into Bidirectional Long Short Term Memory (BiLSTM) layers. It was
sliced into 8-second segments, where any remaining segment under 8 seconds was
discarded.

As first done in (Muzammel et al.) [13], audio data was augmented through
pitch shifting and noise injection strategies to prevent over-fitting and data
scarcity. In this work, we combined and randomly applied the following
augmentation strategies to increase sample size seven-fold:

  * •

Pitch Shifting: Pitch was adjusted by N.N, N.N, and N.N in semitones.

  * •

Noise Injection: A NumPy array with randomly assigned values was overlaid onto
audio to create noise.

Speech WaveformWindowingShift it into FFT orderFind the magnitude of
FFTConvert the FFT data into filter bank outputsFind the log base 10Find the
cosine transform to reduce dimensionalityMFCC Vector
Figure 4: Visualization of process used to derive MFCCs. The way in which
MFCCs were derived in this work is based off the implementation provided by
the Slaney auditory toolbox [45].

####  III-D2 Audio Feature Extraction

MFCCs were extracted from raw audio data using the Librosa Python library. 60
MFCCs were extracted from 124ms windows with a 92ms overlap. There were 15,420
total coefficients for every 8-second audio segment. Figure 4 displays the
pipeline used to derive the MFCCs.

####  III-D3 MFCC Normalization

Rather than normalizing raw audio data, the extracted MFCCs were normalized
using Cepstral Mean and Variance Normalization (CMVN). MFCCs are highly
sensitive to additive noise and differences in recording conditions. CMVN
solves the issue of signal variations due to different recording conditions in
MFCCs by normalizing the spectrum to have zero mean and unit variance. Past
research has shown applying CMVN to MFCCs greatly improves the performance of
speech recognition algorithms [46].

###  III-E Visual Preprocessing

TABLE IV: Facial Action Units Available in DAIC-WOZ. No. | Feature | Facial Feature Name
---|---|---
1 | AU01_r | Inner Brow Raiser
2 | AU02_r | Outer Brow Raiser
3 | AU04_r | Brow Lowerer
4 | AU05_r | Upper Lid Raiser
5 | AU06_r | Cheek Raiser
6 | AU09_r | Nose Wrinkler
7 | AU10_r | Upper Lip Raiser
8 | AU12_r | Lip Corner Puller
9 | AU14_r | Dimpler
10 | AU15_r | Lip Corner Depressor
11 | AU17_r | Chin Raiser
12 | AU20_r | Lip Stretcher
13 | AU25_r | Lips Part
14 | AU26_r | Jaw Drop
15 | AU04_c | Brow Lowerer
16 | AU12_c | Lip Corner Puller
17 | AU15_c | Lip Corner Depressor
18 | AU23_c | Lip Tightener
19 | AU28_c | Lip Suck
20 | AU45_c | Blink

Visual preprocessing followed two stages. (1) Video data was segmented to only
include data from when the patient is speaking. (2) Individual columns were
normalized for zero mean and unit variance.

####  III-E1 Video Segmentation

In the proposed architecture, we analyze patient visual data only while they
are speaking, not in all parts of the interview. Using the timestamps provided
in the transcript, FAU data was cropped to exclude and discard segments where
the therapist is speaking, or where nobody is speaking at all. The data was
then sliced into 8-second segments in such a way that it aligned with the
segmented audio data. Any remeaining segment under 8 seconds was discarded.
After video segmentation was applied, there were an equal number of video and
audio samples for each interview.

####  III-E2 FAU Normalization

Normalization was applied to continuous, non-discrete FAU values. Namely,
numbers 1-14 in Table IV. The continuous FAU values were scaled to zero mean
and unit variance respectively.

##  IV Model Development

###  IV-A Text-Based Model Development

Two-shot learning was employed with GPT-4 through the OpenAI API to reach a
classification given full text transcripts from the DAIC-WOZ.

The model was given the prompt ’Take on the role of an expert in psychiatric
diagnosis using the DSM 5. Read the following transcript and determine if the
patient has depression.’ This prompt is a modified version of that presented
in (Galatzer-Levly et al.) [47]. Given this prompt, GPT-4 was expected to
output a binary classification of ’depressed’ or ’not depressed’ in JSON
format. The function calling feature provided in the OpenAI API was
manipulated to force an output in the desired JSON format in almost all cases.

The examples used as part of few-shot learning consisted of four full-length
text transcripts from the DAIC-WOZ dataset. Two of the provided examples had a
ground truth of depressed, the other two had a ground truth of not depressed.

###  IV-B Tri-Modal Model Development

Model development followed two stages. (1) The model architecture was
optimized with the Hyperband Tuning Algorithm [48] in Keras-Tuner [49]. (2)
The model parameters were reset and the optimized architecture was evaluated
through Leave-One-Subject-Out Cross-Validation (LOSOCV).

####  IV-B1 Hyperparameter Tuning

Figure 5: Proposed model architecture following Hyperband tuning. TABLE V: Hyperparameters optimized through Hyperband | Options | Result
---|---|---
BiLSTM 1 Units | 512, 256 | 256
Dropout 1 Rate | 0.1, 0.3, 0.5 | 0.5
BiLSTM 2 Units | 128, 256 | 128
Dropout 2 Rate | 0.1, 0.3, 0.5 | 0.3
BiLSTM 3 Units | 64, 128 | 64
Dropout 3 Rate | 0.1, 0.3, 0.5 | 0.5
Number of Dense Layers | 4, 5, 6 | 4
Figure 6: Visualization of LOSOCV

The model was trained with the Adam optimizer [50] and binary cross entropy as
the loss function. To account for the class imbalance of the DAIC-WOZ, the
loss function was weighted to give higher value to the depressed class. To
achieve this functionality, part of the Keras-Tuner Python library source code
was rebuilt.

The tuner was set to focus on validation loss as the only metric to gauge
configuration performance. Through each bracket, the number of total epochs
run and models tested was reduced by a factor of three. A callback was created
to end training and move on to the next configuration if validation loss did
not improve over three consecutive epochs. Two iterations of the full
Hyperband algorithm were run.

####  IV-B2 Evaluation

LOSOCV (Figure 6) was applied by first resetting the optimized architecture’s
trainable parameters which had been adjusted during hyperparameter tuning. A
unique model was trained for every clinical interview in the dataset (n=180).
For each model, all samples originating from one clinical interview were
excluded from the training dataset. Each model was tested on the excluded
clinical interivew, and all the model results were pooled for the final
performance calculation. Augmented samples were not included in testing data.

###  IV-C Proposed Architecture

The final model architecture (Figure 5), which has been optimized by the
Hyperband Tuning Algorithm, follows 7 steps to reach a classification.

  1. 1.

The model extracts higher level features from the MFCCs in three consecutive
blocks consisting of BiLSTM, Dropout, and Batch Normalization layers.

  2. 2.

The input FAU data follows three processing steps to make it suitable for
concatenation with the MFCC tensor: (a) The FAU data is first flattened into a
1D tensor. (b) The Keras [53] Repeat Vector layer repeats the flattened data
for each timestep present in the processed MFCC tensor, resulting in a 3D
tensor. (c) Lastly, the Time Distributed operation applies a Dense layer to
each temporal slice, where the number of neurons is the number of feature
dimensions in the MFCC tensor. These operations result in a higher-level
representation of the FAUs with the same dimensions as the processed MFCC
data, allowing concatenation in the following step.

  3. 3.

The processed FAU and MFCC data are concatenated along the features axis.

  4. 4.

The concatenated data is processed by consecutive BiLSTM, Dropout, and Batch
Normalization layers.

  5. 5.

The same sequence of operations applied in step two is applied to the LLM
binary output to prepare it for concatenation with the MFCC-FAU tensor.

  6. 6.

The MFCC-FAU and LLM tensors are concatenated.

  7. 7.

The combined tensor is processed by four consecutive Leaky ReLU-activated
Dense layers and one sigmoid-activated layer. A binary classification is
output.

##  V Results

The proposed architecture was evaluated in two ways:

  1. 1.

The model was evaluated using the DAIC-WOZ AVEC 2016 Challenge cross-
validation split to allow comparability with models from other studies.

  2. 2.

The proposed architecture was evaluated through LOSOCV.

Predicted Actual D ND total D 13 (27.66) 1 (2.13) 92.85 ND 6 (12.77) 27 (57.45) 81.81 total 68.41 96.42 |  Predicted Actual D ND total D 52 (27.51) 13 (6.88) 79.99 ND 4 (2.12) 120 (63.49) 96.77 total 92.85 90.22
---|---
Figure 7: Confusion matrices of proposed architecture’s performance on AVEC 2016 cross-validation split (left) and LOSOCV (right). TABLE VI: Comparison of proposed architecture with baseline binary depression classification models tested on the AVEC 2016 cross-validation split. Paper | Precision | Recall | F1-Score | Accuracy
---|---|---|---|---
(Yang et al.) - Logistic Regression | 53.20 | 55.80 | 54.47 | 54.12
(Yang et al.) - Naive Bayes | 56.70 | 58.40 | 57.54 | 57.02
(Yang et al.) - Random Forest | 54.80 | 55.20 | 55.00 | 58.41
(Valstar et al.) | 57.90 | 59.60 | 58.74 | 58.41
(Qureshi et al.) | 58.00 | 61.00 | 59.46 | 61.11
Proposed Model | 92.86 | 68.42 | 78.79 | 85.11
TABLE VII: Comparison of proposed architecture with state-of-the-art binary depression classification models tested on the AVEC 2016 cross-validation split. A dash signifies that the metric was not reported in the paper. (*Evaluation done with LOSOCV) Paper | Precision | Recall | F1-Score | Accuracy
---|---|---|---|---
| D | ND | D | ND |  |
(Gimeno-Gómez et al.) | 72 | – | 72 | – | 72 | –
(Muzammel et al.) | 53 | 83 | 44 | 88 | – | 77.16
(Yang et al.) | 80.20 | – | 81 | – | 80.60 | 81.10
(Huang et al.) | 94 | – | 95 | – | 94 | 96
Proposed Model | 92.86 | 96.43 | 68.42 | 81.82 | 78.79 | 85.11
(Muzammel et al.)* | 96 | 95 | 89 | 98 | 95.48 | 95.38
(Othmani et al.)* | – | – | – | – | 82.30 | 87.40
(Salekin et al.)* | – | – | – | – | 85.44 | 96.7
Proposed Model* | 80 | 90.23 | 92.86 | 96.77 | 85.95 | 91.01

The proposed model outperforms all baseline models and multiple state-of-the-
art models which were evaluated using a regular cross-validation split as well
as LOSOCV. It performs well on both the depressed and non-depressed classes,
with accuracies of 92.9% and 90.2% on LOSOCV for the depressed and non-
depressed classes, respectively. The large difference in the accuracy between
the depressed and non-depressed classes (92.9% and 81.81%, respectively) on
the AVEC 2016 cross-validation split can be attributed to the extremely small
sample size of just 14 for the depressed class. We believe the accuracy of
92.9% for the depressed class on the AVEC 2016 Challenge cross-validation
split is inflated as a result of this sample size.

###  V-A Computational Complexity

The speed of the proposed architecture was evaluated on the 12-minute
50-second DAIC-WOZ clinical interview with an ID of 301. The test was
conducted using an Intel i7-12650H @ 2.30GHZ processor, NVIDIA RTX 4060
graphics card, and 64GB of RAM. The model took 2.67 seconds to process the
interview, including GPT-4 computation time through the OpenAI API.

##  VI Deployment

Figure 8: Interview collection and processing pipeline Figure 9: Clinician’s
dashboard Figure 10: Sample clinical report

The proposed architecture was integrated into a locally hosted web
application, DepScope, for easy and accessible use of the model by clinicians.
This application mimics how such a model may be implemented into real-world
scenarios.

Clinicians first connect their Zoom account to the web application and agree
to the necessary terms, consenting that their clinical interview recordings
may be processed by the DepScope applicatoin. This is done through the Zoom
Application Programming Interface (API). From there, any time a clinical
interview concludes over Zoom, a webhook [56] is sent to the DepScope backend.
The recording is then retrieved from the Zoom servers for preprocessing, model
inference, and clinical report generation via GPT-4. Clinical reports contain
a summary of the interview, detailing all key points and a justification for
why the diagnosis was drawn (Figure 10). Reports additionally contain the
confidence of the model in the classification, which is output by the sigmoid
activation function. Reports populate the clinician dashboard (Figure 9).

##  VII Conclusion

This paper presents a novel machine learning architecture for diagnosing MDD
from clinical interview recordings. We explored the integration of large
language models and audiovisual features into a tri-modal architecture for
this task. The proposed architecture was optimized through automated hyper-
parameter tuning techniques and evaluated using the AVEC 2016 Challenge cross-
validation split and LOSOCV. We drew comparisons between the performance of
the proposed architecture and previously developed models, finding that it
outperforms all baseline models and multiple state-of-the-art models. The
model was integrated into a locally hosted web application to emulate how it
may be used in the real-world.

Though the proposed model achieved impressive results, the current
architecture faces multiple limitations. The speed of the model suggests that
it is not appropriate for real-time applications and may only be effective
when integrated with batch-based data processing techniques. Due to the use of
large language models in this architecture, it is highly unlikely this can be
resolved. The dataset used in this study, the DAIC-WOZ, is small and
homogenous. We believe this has negatively impacted the generalizability of
the proposed model and may have resulted in metrics which do not reflect the
effectiveness of the model in real-world settings. The lack of high-quality
data available for this task is an issue which plagues the field as the whole.

In future work, we aim to further optimize the proposed architecture for
increased accuracy and make the model easily accessible to use. As new large
language models are developed, they will be integrated into the model
architecture and evaluated. We plan to integrate existing language models into
the architecture as well, such as Med-PaLM 2 [57]. The prompt given to the
large language model will be improved through iterative prompting techniques.

## Acknowledgments

The author would like to extend thanks to the SMU Lyle School of Engineering
and the American Psychological Association for their cash award which funded
this research.

## References

  * [1] P. Chodavadia, I. Teo, D. Poremski, D. S. S. Fung, and E. A. Finkelstein, “Prevalence and economic burden of depression and anxiety symptoms among singaporean adults: results from a 2022 web panel,” _BMC Psychiatry_ , vol. 23, p. 104, 2 2023.
  * [2] D. F. Santomauro, A. M. M. Herrera, J. Shadid, P. Zheng, C. Ashbaugh, D. M. Pigott, C. Abbafati, C. Adolph, J. O. Amlag, A. Y. Aravkin, B. L. Bang-Jensen, G. J. Bertolacci, S. S. Bloom, R. Castellano, E. Castro, S. Chakrabarti, J. Chattopadhyay, R. M. Cogen, J. K. Collins, X. Dai, W. J. Dangel, C. Dapper, A. Deen, M. Erickson, S. B. Ewald, A. D. Flaxman, J. J. Frostad, N. Fullman, J. R. Giles, A. Z. Giref, G. Guo, J. He, M. Helak, E. N. Hulland, B. Idrisov, A. Lindstrom, E. Linebarger, P. A. Lotufo, R. Lozano, B. Magistro, D. C. Malta, J. C. Månsson, F. Marinho, A. H. Mokdad, L. Monasta, P. Naik, S. Nomura, J. K. O’Halloran, S. M. Ostroff, M. Pasovic, L. Penberthy, R. C. R. Jr, G. Reinke, A. L. P. Ribeiro, A. Sholokhov, R. J. D. Sorensen, E. Varavikova, A. T. Vo, R. Walcott, S. Watson, C. S. Wiysonge, B. Zigler, S. I. Hay, T. Vos, C. J. L. Murray, H. A. Whiteford, and A. J. Ferrari, “Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the covid-19 pandemic,” _The Lancet_ , vol. 398, pp. 1700–1712, 11 2021.
  * [3] J. Guidi, G. A. Fava, P. Bech, and E. Paykel, “The clinical interview for depression: A comprehensive review of studies and clinimetric properties,” _Psychotherapy and Psychosomatics_ , vol. 80, pp. 10–27, 2011.
  * [4] L. S. Radloff, “The ces-d scale,” _Applied Psychological Measurement_ , vol. 1, pp. 385–401, 6 1977.
  * [5] K. Kroenke, R. L. Spitzer, and J. B. W. Williams, “The phq-9,” _Journal of General Internal Medicine_ , vol. 16, pp. 606–613, 9 2001.
  * [6] Allsopp, K., Read, J., Corcoran, R. & Kinderman, P. Heterogeneity in psychiatric diagnostic classification. Psychiatry Res.. 279 pp. 15–22, 9 2019.
  * [7] Silva, P. How to improve psychiatric services: a perspective from critical psychiatry. Br. J. Hosp. Med. (Lond.). 78, 503–507, 9 2017.
  * [8] Fuchs, T. Subjectivity and intersubjectivity in psychiatric diagnosis. Psychopathology. 43, 268–274, 5 2010.
  * [9] A. Althubaiti, “Information bias in health research: definition, pitfalls, and adjustment methods,” _Journal of Multidisciplinary Healthcare_ , p. 211, 5 2016.
  * [10] C. A. Latkin, C. Edwards, M. A. Davey-Rothwell, and K. E. Tobin, “The relationship between social desirability bias and self-reports of health, substance use, and social network factors among urban substance users in baltimore, maryland,” _Addictive Behaviors_ , vol. 73, pp. 133–136, 10 2017.
  * [11] R. Mendel, E. Traut-Mattausch, E. Jonas, S. Leucht, J. M. Kane, K. Maino, W. Kissling, and J. Hamann, “Confirmation bias: why psychiatrists stick to wrong preliminary diagnoses,” _Psychological Medicine_ , vol. 41, pp. 2651–2659, 12 2011.
  * [12] G. Ayano, S. Demelash, Z. yohannes, K. Haile, M. Tulu, D. Assefa, A. Tesfaye, K. Haile, M. Solomon, A. Chaka, and L. Tsegay, “Misdiagnosis, detection rate, and associated factors of severe psychiatric disorders in specialized psychiatry centers in ethiopia,” _Annals of General Psychiatry_ , vol. 20, p. 10, 2 2021.
  * [13] M. Muzammel, H. Salam, and A. Othmani, “End-to-end multimodal clinical depression recognition using deep neural networks: A comparative analysis,” _Computer Methods and Programs in Biomedicine_ , vol. 211, p. 106433, 11 2021.
  * [14] X. Ma, H. Yang, Q. Chen, D. Huang, and Y. Wang, “Depaudionet.” ACM, 10 2016, pp. 35–42.
  * [15] R. Brueckner, N. Kwon, V. Subramanian, N. Blaylock, and H. O’Connell, _Audio-Based Detection of Anxiety and Depression via Vocal Biomarkers_ , 2024, pp. 124–141.
  * [16] P. Zhang, M. Wu, H. Dinkel, and K. Yu, “Depa.” ACM, 10 2021, pp. 135–143.
  * [17] J. Wang, L. Zhang, T. Liu, W. Pan, B. Hu, and T. Zhu, “Acoustic differences between healthy and depressed people: a cross-situation study,” _BMC Psychiatry_ , vol. 19, p. 300, 12 2019.
  * [18] C.-H. Tu, C.-Y. Yang, and J. Y. jen Hsu, “Idennet: Identity-aware facial action unit detection.” IEEE, 5 2019, pp. 1–8.
  * [19] D. Gimeno-Gómez, A.-M. Bucur, A. Cosma, C.-D. Martínez-Hinarejos, and P. Rosso, _Reading Between the Frames: Multi-modal Depression Detection in Videos from Non-verbal Cues_ , 2024, pp. 191–209.
  * [20] Degottex, G., Kane, J., Drugman, T., Raitio, T., and Scherer, S. “COVAREP — A collaborative voice analysis repository for speech technologies.“ 2014 IEEE International Conference On Acoustics, Speech And Signal Processing (ICASSP), 5 2014.
  * [21] Sataloff, R. Vocal Health and pedagogy: Science, assessment, and treatment, Third Edition. (Plural Publishing), 9 2017.
  * [22] S. Yang, L. Cui, L. Wang, T. Wang, and J. You, “Enhancing multimodal depression diagnosis through representation learning and knowledge transfer,” _Heliyon_ , vol. 10, p. e25959, 2 2024.
  * [23] Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional Transformers for language understanding. (arXiv,2018)
  * [24] A. Othmani, A.-O. Zeghina, and M. Muzammel, “A model of normality inspired deep learning framework for depression relapse prediction using audiovisual data,” _Computer Methods and Programs in Biomedicine_ , vol. 226, p. 107132, 11 2022.
  * [25] Simonyan, K., and Zisserman, A. “Very deep convolutional networks for large-scale image recognition.“ _arXiv_ , 2014.
  * [26] F. Ceccarelli, M. Mahmoud, “Multimodal temporal machine learning for Bipolar Disorder and Depression Recognition,“ _Pattern Anal. Appl._. 25, 493-504, 8 2022.
  * [27] Perronnin, F., Sánchez, J. & Mensink, T. Improving the fisher kernel for large-scale image classification. Computer Vision – ECCV 2010. pp. 143-156 (2010)
  * [28] P. Wei, K. Peng, A. Roitberg, K. Yang, J. Zhang, and R. Stiefelhagen, “Multi-modal Depression Estimation based on Sub-attentional Fusion.,“ _arXiv_ , 2022
  * [29] Pedersen, P. The Mel scale. J. Music Theory. 9, 295 (1965)
  * [30] S. A. Almaghrabi, S. R. Clark, and M. Baumert, “Bio-acoustic features of depression: A review,” _Biomedical Signal Processing and Control_ , vol. 85, p. 105020, 8 2023.
  * [31] Wang, Y., Liang, L., Zhang, Z., Xu, X., Liu, R., Fang, H., Zhang, R., Wei, Y., Liu, Z., Zhu, R., Zhang, X. & Wang, F. Fast and accurate assessment of depression based on voice acoustic features: a cross-sectional and longitudinal study. Front. Psychiatry. 14 pp. 1195276 (2023,6)
  * [32] T. Taguchi, H. Tachikawa, K. Nemoto, M. Suzuki, T. Nagano, R. Tachibana, M. Nishimura, and T. Arai, “Major depressive disorder discrimination using vocal acoustic features,” _Journal of Affective Disorders_ , vol. 225, pp. 214–220, 1 2018.
  * [33] S. Alghowinem, R. Goecke, M. Wagner, G. Parkerx, and M. Breakspear, “Head pose and movement analysis as an indicator of depression.” IEEE, 9 2013, pp. 283–288.
  * [34] F. Schneider, H. Heimann, W. Himer, D. Huss, R. Mattes, and B. Adam, “Computer-based analysis of facial action in schizophrenic and depressed patients,” _European Archives of Psychiatry and Clinical Neuroscience_ , vol. 240, pp. 67–76, 11 1990.
  * [35] Ekman, P. & Friesen, W. Facial Action Coding System. (American Psychological Association (APA),2019,1), Title of the publication associated with this dataset: PsycTESTS Dataset
  * [36] Jiang, Z., Luskus, M., Seyedi, S., Griner, E., Rad, A., Clifford, G., Boazak, M. & Cotes, R. Utilizing computer vision for facial behavior analysis in schizophrenia studies: A systematic review. PLoS One. 17, e0266828 (2022,4)
  * [37] Borges, P., Peynot, T., Liang, S., Arain, B., Wildie, M., Minareci, M., Lichman, S., Samvedi, G., Sa, I., Hudson, N., Milford, M., Moghadam, P. & Corke, P. A survey on terrain traversability analysis for autonomous ground vehicles: Methods, sensors, and challenges. Field Robotics. 2, 1567-1627 (2022,3)
  * [38] M. Pawłowski, A. Wróblewska, and S. Sysko-Romańczuk, “Effective techniques for multimodal data fusion: A comparative analysis,” _Sensors_ , vol. 23, p. 2381, 2 2023.
  * [39] M. Asgari, I. Shafran, and L. B. Sheeber, “Inferring clinical depression from speech and spoken utterances.” IEEE, 9 2014, pp. 1–5.
  * [40] A. Samareh, Y. Jin, Z. Wang, X. Chang, and S. Huang, “Predicting depression severity by multi-modal feature engineering and fusion,” 11 2017.
  * [41] J. Gratch, R. Artstein, G. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella, D. Traum, S. Rizzo, and L.-P. Morency, “The distress analysis interview corpus of human and computer interviews,” N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis, Eds. European Language Resources Association (ELRA), 5 2014, pp. 3123–3128. [Online]. Available: http://www.lrec-conf.org/proceedings/lrec2014/pdf/508_Paper.pdf
  * [42] Baltrusaitis, T., Robinson, P. & Morency, L. OpenFace: An open source facial behavior analysis toolkit. 2016 IEEE Winter Conference On Applications Of Computer Vision (WACV). (2016,3)
  * [43] Napoles, C., Sakaguchi, K. & Tetreault, J. JFLEG: A fluency corpus and benchmark for grammatical error correction. (arXiv,2017)
  * [44] B. McFee, C. Raffel, D. Liang, D. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” 2015, pp. 18–24.
  * [45] M. Slaney, “Auditory toolbox,” _Interval Research Corporation, Tech. Rep_ , vol. 10, p. 1194, 1998.
  * [46] R. Rehr and T. Gerkmann, “Cepstral noise subtraction for robust automatic speech recognition.” IEEE, 4 2015, pp. 375–378.
  * [47] I. R. Galatzer-Levy, D. McDuff, V. Natarajan, A. Karthikesalingam, and M. Malgaroli, “The capability of large language models to measure psychiatric functioning,” 8 2023.
  * [48] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” 3 2016.
  * [49] T. O’Malley, E. Bursztein, J. Long, F. Chollet, H. Jin, L. Invernizzi _et al._ , “Kerastuner,” 2019.
  * [50] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 12 2014.
  * [51] M. Valstar, J. Gratch, B. Schuller, F. Ringeval, D. Lalanne, M. T. Torres, S. Scherer, G. Stratou, R. Cowie, and M. Pantic, “Avec 2016.” ACM, 10 2016, pp. 3–10.
  * [52] S. A. Qureshi, S. Saha, M. Hasanuzzaman, and G. Dias, “Multitask representation learning for multimodal estimation of depression level,” _IEEE Intelligent Systems_ , vol. 34, pp. 45–52, 9 2019.
  * [53] Chollet, F. & Others Keras. (GitHub,2015), https://github.com/fchollet/keras
  * [54] X. Huang, F. Wang, Y. Gao, Y. Liao, W. Zhang, L. Zhang, and Z. Xu, “Depression recognition using voice-based pre-training model,” _Scientific Reports_ , vol. 14, p. 12734, 6 2024.
  * [55] A. Salekin, J. W. Eberle, J. J. Glenn, B. A. Teachman, and J. A. Stankovic, “A weakly supervised learning framework for detecting social anxiety and depression,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 2, pp. 1–26, 7 2018.
  * [56] Biehl, M. Webhooks. (Createspace Independent Publishing Platform,2017,12)
  * [57] Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., Schaekermann, M., Wang, A., Amin, M., Lachgar, S., Mansfield, P., Prakash, S., Green, B., Dominowska, E., Arcas, B., Tomasev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S., Barral, J., Webster, D., Corrado, G., Matias, Y., Azizi, S., Karthikesalingam, A. & Natarajan, V. Towards expert-level medical question answering with large language models. (arXiv,2023)


# Row-wise Accelerator for Vision Transformer

Hong-Yi Wang, and Tian-Sheuan Chang, Senior Member, IEEE H.-Y. Wang, and T.-S.
Chang, ”Row-wise accelerator for vision transformer”, IEEE AICAS, 2022
Institute of Electronics, National Yang Ming Chiao Tung University, Hsinchu,
Taiwan

###### Abstract

Following the success of the natural language processing, the transformer for
vision applications has attracted significant attention in recent years due to
its excellent performance. However, existing deep learning hardware
accelerators for vision cannot execute this structure efficiently due to
significant model architecture differences. As a result, this paper proposes
the hardware accelerator for vision transformers with row-wise scheduling,
which decomposes major operations in vision transformers as a single dot
product primitive for a unified and efficient execution. Furthermore, by
sharing weights in columns, we can reuse the data and reduce the usage of
memory. The implementation with TSMC 40nm CMOS technology only requires 262K
gate count and 149KB SRAM buffer for 403.2 GOPS throughput at 600MHz clock
frequency.

###### Index Terms:

vision transformer, hardware design, accelerators

##  I Introduction

Deep learning in computer vision has long been dominated by convolutional
neural networks (CNNs) based backbone[1] for its revolutionary performance. On
the other hand, natural language processing (NLP) use Transformer [2] that no
longer relies on convolution but self-attention to model long-term data
dependencies. Its enormous success makes it a viable competitor in computer
vision tasks as well. As a result, various vision transformer models[3, 4, 5,
6] have been developed that show competitive or even better performance than
current CNN based models. However, real-time executions of these models suffer
from high computational complexity and memory bandwidth, which demands
hardware acceleration. However, existing deep learning hardware
accelerators[7, 8, 9] are optimized for CNN based models, which is not
suitable for vision transformer models due to the significant difference in
the favored architecture structure and parameters. Directly executing vision
transformer models on these accelerators will result in low hardware
utilization.

To address the above issue, this paper proposes a hardware accelerator for
vision transformers. To tailor the hardware design to vision transformers, we
first analyze the common features of different models and compare them with
existing CNN models. Based on this analysis, we propose a row-wise scheduling
that uses the dot product as a primitive for major operations in the model to
attain high hardware utilization. Besides, the proposed hardware broadcast
weight for computations that can share weights for higher data reuse and
smaller buffer size. The implementation on TSMC 40nm CMOS process can achieve
403.2GOPS throughput when running at 600Mhz clock frequency.

The remainder of the paper is structured as follows. Section II goes over the
associated tasks. Section III examines the common characteristics of several
vision transformer models. The proposed design is shown in Section IV. Section
V presents the experimental data as well as comparisons with other studies.
Finally, Section VI brings this paper to a close.

##  II Related Work

###  II-A Transformer

Transformer adopts the encoder and decoder architecture. The encoder consists
of multiple encoding layers to generate the relationship between the inputs,
while the decoder consists of multiple decoding layers that do the opposite.
Fig. 1 shows an example of one layer of encoder. In which, an encoding layer
consists of three kinds of blocks, self-attention, feed forward and
normalization.The self-attention block will figure out the relationship
between different inputs to generate output encodings as shown in (1).

$Attention(Q,K,V)=SoftMax(QK^{T}/\sqrt{d}_{k})V$ (1)

Then, each output encoding is further processed by the feed forward block that
uses two successive fully connected layers with Gaussian error linear units
(GELU) [10] as its activation function. Its output is normalized by the
normalization block that uses layer normalization. Besides, they would use the
skip connection in every block.

The fully connected layers in the vision transformer share their weight along
the spatial dimension and apply fully connection on the channel dimension.
Thus, these fully connected layers are the 1×1 convolutions. However, to keep
naming consistency with other papers, the name of the fully connected layer is
still used below.

Figure 1: Transformer encoder architecture

###  II-B Vision Transformer

With Transformer’s success in nature language processing, Transformer has also
been introduced to the vision application [3], which gains competitive or
better performance than previous convolutional neural network based models.
For the classification task on ImageNet, lots of vision transformers such as
ViT [3], TNT [5], Swin [4] and PVT [6] use the encoder of Transformer. These
models differ on the detailed structure of the encoder layer.

###  II-C Hardware Accelerators for CNNs and Transformer in NLP

Various hardware accelerators have been proposed for CNN model execution. They
exploit the high parallelism of CNN computation with hundreds of PEs for
massive computation, and reuse input/weights/activation in different ways to
reduce the external memory bandwidth requirement[11, 12, 7, 8]. The PEs are
arranged in a 1D or 2D array, optimized for widely used 3×3 convolutions and
reconfigured for other convolution types if needed but with lower hardware
utilization. In contrast, 3×3 convolutions are seldom used in vision
transformers. In vision transformers, their main structures don’t involve CNN.
They only use CNN in the beginning as a kind of feature extraction method for
image pre-processing. Besides, the kernel size is 4×4 in vision transformers.
Thus, if directly executing these models on these accelerators, their hardware
utilization would be lower than 66%. Not to mention that these designs do not
support multi-head attention.

Several works [13, 14, 15] have also proposed for transformer accelerator in
NLP. No data reuse would make it perform poorly in vision. Additionally, the
configuration of parameters in NLP (_sequence length $\times$ channel _) is
different from that in vision (_height $\times$ width $\times$ channel _) .
Thus, existing designs are not suitable for vision transformers.

##  III Model Analysis for Vision Transformers

###  III-A Common Features in Vision Transformer

Table I shows the common configuration parameters for different vision
transformer models. First, since these models are for ImageNet classification,
the input sizes for different layers are a multiple of 7 due to the 224×224
image size in ImageNet. Besides, their channel numbers were almost always a
multiple of 96 except for PVT, which is a multiple of 64. These configuration
numbers are quite different from CNN based classification models.

For the model structure, their backbones are the encoder of the transformer,
which composes of multi-head attention blocks, residual connection, and fully
connected layers. In addition, compared with the traditional CNN model as
shown in Table II, activation function of vision transformers is GELU instead
of ReLU. Moreover, their normalization functions were layer normalization
instead of batch normalization.

Based on these analysis results, the hardware design will optimize for these
common configurations and structures.

TABLE I: Common features in vision transformer models | features | channels | input size | conv size
---|---|---|---|---
ViT | first vision transformer | 768 | 14 | 16
TNT | extra patches transformer | 768 | 14 | 16
Swin |  | windows transformer
---
shifted windows
96× | 7× | 4
PVT | image pyramid | 64× | 14 | 4×
TABLE II: Comparison between vision transformer and CNN based model | Vision Transformer | CNN
---|---|---
Activation function | GELU | ReLU
Normalization | layer normal | batch normal
Main calculation | fully-connected | convolution
Similarity | short cut (residual)
Figure 2: Distributions of parameters and FLOPs in Swin-T for different
layers.

###  III-B Analysis of FLOPs and Parameters Distribution in Vision
Transformers

To facilitate the analysis, we decompose the model, Swin-T, into three layers
according to their actual functions, including the convolution layer, fully
connected layer, and multi-head attention layer. Fig. 2 shows the distribution
analysis of FLOPS and parameters for different layer types .

From the viewpoint of layer types as shown in Fig. 2, more than 97% of the
FLOPs and more than 83% of number of parameters are occupied by the fully
connected layer. Both the convolution layer and multi-head attention layer
only occupy a very small fraction. Thus, the design of computation unit and
buffer size should optimize for the fully connected layer.

##  IV Proposed Architecture

###  IV-A Row-wise Scheduling

For the three major layers, multi-head attention and convolution are matrix
multiplication, while the fully connected layer is a matrix vector product. To
unify these computation types and fit different configuration parameters, we
decompose all these operations as a more basic dot product to build our
processing element (PE) block as shown in Fig. 3.

A PE block contains 7 PE rows in a block and each row has 4 multiplier-and-
accumulator (MAC) units. In this paper, we use 12 PE blocks for model
execution. These numbers are selected to fit the common configuration
parameters as shown before. Thus, one row can compute a dot product on 2
vectors with both sizes of 4, which can fit different operation requirements
due to the decomposition. This also enables a very high hardware utilization
and simple dataflow to optimize the overall computing time.

In a PE block, each weight data are broadcast to all multipliers from top to
bottom to share weight. Each MAC will receive different inputs from the input
SRAM to support the fully connected layer. The multiplication results are
accumulated in a horizontal direction and saved to the local buffer for later
more accumulation.

Figure 3: A PE block

###  IV-B Overall Architecture

Fig. 4 shows the proposed overall architecture. The result from each PE block
is accumulated in an accumulator block and further summed together with the
adder tree for the large number of input channel cases. Then the result will
be post-processing by layer normalization or softmax depending on the layer
types. The final result will be sent to off-chip memory.

Figure 4: Proposed overall architecture for vision transformer

###  IV-C Convolution Layer

The only convolution in vision transformers is at the first layer, which uses
4×4 convolution with stride 4 to reduce the input size into 1/4 of the
original. The size of each kernel is 4×4×3, which can be perfectly placed into
each PE weight block, as shown in Fig. 5. The input is RGB images. Each input
channel and its corresponding weight will be placed on four PE blocks,
respectively. During each computing cycle as in Fig. 5, all 7 input rows will
be activated at the same time to process all input to generate 7 outputs in a
cycle. The computing order of convolution will process the first output
channel and then the second one until the final 96th channel. For the
224×224×3 input size in ImageNet, our design would calculate a 28×4×3 input in
a cycle and take 448 cycles to finish one output channel.

Figure 5: (a)Input and weight mapping to PE blocks for one output convolution
channel. (b) Detailed mapping for input of PE blocks

###  IV-D Fully-Connected Layer

By row-wise scheduling, different sizes of fully connected layers can become a
dot product in a single row. However, since the number of input channels are
multiple of 96 and exceeds the hardware capability in one cycle, its output
will be accumulated in the accumulator for the final output.

Fig. 6(a) and Fig. 6(b) show the detailed calculation for 96 channels. In the
first cycle, the 1st to 48th channels of input and filters would be placed at
the corresponding position from PE block 0 to 11. The second cycle will
process the rest of channels. Besides, with 7 input rows in a PE block, we
would finish 7 outputs in every 2 cycles for 96 channels of input. Then, the
computing order of the fully connected layer will process the first output
channel and then the second one until the final output channel.

(a) cycle 1

(b) cycle 2

Figure 6: Detailed input and weight mapping of 7 inputs for a fully-connected
layer with 96 channels at (a) the first cycle and (b) the second cycle.

###  IV-E Multi-Head Attention Layer

With the row-wise scheduling and the window multi-head self-attention(W-MSA)
concept as in Swin-Transformer, we can make the computation of multi-head
attention layers regular and simple. In this case, for the multiplication of Q
and K as in (1), Q could be regarded as the weight and K as the input for PE
blocks. In this case, since the size of the matrix is smaller than other layer
types, it will only use 8 PE blocks. Thus, for Q matrix mapping, four columns
of Q matrix will be allocated to one PE block and processed in a row-by-row
manner. Each row will take 7 cycles. For KT matrix mapping, they are allocated
to a group with 7 input rows × 8 PE blocks for computation, and processed in a
group-by-group manner. During this process, since only 8 of the 12 PE blocks,
the hardware utilization will be lower. However, the flops of the multi-head
attention layer would take no more than 3% in whole model, and such hardware
utilization impact would impact no more than 1% of total cycles.

##  V Experimental Results

This design has been implemented with TSMC 40nm CMOS technology and needs 262K
NAND gate count and 149KB SRAM buffer to execute vision transformer models,
especially optimized for Swin Transformer. The precision of weights and
activations are both 8 bits. For area, SRAM occupies 93.2% of area, and PE,
SRAM controller, and accumulator occupy 4.6%, 1.5%, 0.6% of area respectively.
The peak performance is about 403.2GOPS at 600MHz clock frequency. The average
time to process an 224$\times$224 image with Swin-T is about 22.4 ms,
equivalent to 44.5 images/s throughput. Theoretically, overall hardware
utilization could be as high as 99% or higher.

Comparisons with other designs are not easy due to different target models.
For reference, we list other CNN hardware accelerators in Table III. Our peak
throughput is higher than others due to more PE numbers and higher clock
frequency. However, our area cost is smaller than others due to its simple PE
structure.

Table IV shows the throughput comparison to execute Swin-T with different
processors and designs. GPU is a NVIDIA GeForce RTX 2080 Ti, which has 4352
CUDA Cores and a total of about 9.8MB of cache memory. Compared to GPU, our
throughput is 1.1$\times$ than that of GPU, respectively. This higher
throughput is achieved by our high hardware utilization. Another design is
based on FPGA. Vis-Top [16] is 1.9$\times$ faster than us, but little
information has been disclosed in this paper. This higher throughput is
possible due to larger PEs based on its DSP slices utilized.

TABLE III: Comparison to other CNN accelerators | Ours | IECA [17] | Eyeriss v2 [8]
---|---|---|---
Technology(nm) | 40 | 55 | 65
Model Types | Trans. | CNN | CNN
PE Number | 336 | 168 | 92
Clock Rate(MHz) | 600 | 250 | 200
Peak Throughput(GOPS) | 403.2 | 84 | 153.6
Area (KGE) | 186 | 344.7 | 2695
SRAM(KB) | 149 | 109 | 192
TABLE IV: Comparison to other processors and designs. | GPU | Ours (ASIC)
---|---|---
Throughput (image/s) | 41.5 | 44.5
Relative Speedup | 1 | 1.07
Cache Memory(KB) | 9852 | 149
Throughput per MACs | 0.0095 | 0.1324

##  VI Conclusion

This paper proposes an efficient design for vision transformers that is
optimized for encoding layer operations instead of CNN. This design can
efficiently execute different layer operations by row-wise scheduling for high
hardware utilization. Our design retains high throughput while maintaining
area efficiency. The final implementation with TSMC 40nm CMOS process needs
262K NAND gate counts and 149KB SRAM buffer and achieves 403.2 GOPS peak
throughput when running at 600MHz clock frequency.

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [2] A. Vaswani _et al._ , “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [3] A. Dosovitskiy _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [4] Z. Liu _et al._ , “Swin transformer: Hierarchical vision transformer using shifted windows,” _arXiv preprint arXiv:2103.14030_ , 2021.
  * [5] K. Han _et al._ , “Transformer in transformer,” _arXiv preprint arXiv:2103.00112_ , 2021.
  * [6] W. Wang _et al._ , “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” _arXiv preprint arXiv:2102.12122_ , 2021.
  * [7] Y.-H. Chen _et al._ , “Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks,” _IEEE journal of solid-state circuits_ , vol. 52, no. 1, pp. 127–138, 2016.
  * [8] Y.-H. Chen _et al._ , “Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 9, no. 2, pp. 292–308, 2019.
  * [9] A. Biswas and A. P. Chandrakasan, “Conv-sram: An energy-efficient sram with in-memory dot-product computation for low-power convolutional neural networks,” _IEEE Journal of Solid-State Circuits_ , vol. 54, no. 1, pp. 217–230, 2018.
  * [10] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [11] Y. Chen _et al._ , “Dadiannao: A machine-learning supercomputer,” in _2014 47th Annual IEEE/ACM International Symposium on Microarchitecture_. IEEE, 2014, pp. 609–622.
  * [12] T. Chen _et al._ , “Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning,” _ACM SIGARCH Computer Architecture News_ , vol. 42, no. 1, pp. 269–284, 2014.
  * [13] S. Lu _et al._ , “Hardware accelerator for multi-head attention and position-wise feed-forward in the transformer,” in _2020 IEEE 33rd International System-on-Chip Conference (SOCC)_. IEEE, 2020, pp. 84–89.
  * [14] B. Li _et al._ , “Ftrans: energy-efficient acceleration of transformers using fpga,” in _Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design_ , 2020, pp. 175–180.
  * [15] T. J. Ham _et al._ , “A^ 3: Accelerating attention mechanisms in neural networks with approximation,” in _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2020, pp. 328–341.
  * [16] W. Hu _et al._ , “Vis-top: Visual transformer overlay processor,” _arXiv preprint arXiv:2110.10957_ , 2021.
  * [17] B. Huang _et al._ , “IECA: An in-execution configuration CNN accelerator with 30.55 GOPS/mm2 area efficiency,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 68, no. 11, pp. 4672–4685, 2021.


#  MEMQSim: Highly Memory-Efficient and Modularized Quantum State-Vector
Simulation

Boyuan Zhang<EMAIL_ADDRESS>Indiana University BloomingtonINUSA ,  Bo Fang
<EMAIL_ADDRESS>Pacific Northwest National Laboratory RichlandWAUSA ,  Qiang
Guan<EMAIL_ADDRESS>Kent State UniversityKentOHUSA ,  Ang Li<EMAIL_ADDRESS>Pacific Northwest National Laboratory RichlandWAUSA and  Dingwen Tao
<EMAIL_ADDRESS>Indiana University BloomingtonINUSA

(2023)

††journalyear: 2023††copyright: licensedusgovmixed††conference: Workshops of
The International Conference on High Performance Computing, Network, Storage,
and Analysis; November 12–17, 2023; Denver, CO, USA††booktitle: Workshops of
The International Conference on High Performance Computing, Network, Storage,
and Analysis (SC-W 2023), November 12–17, 2023, Denver, CO, USA††price:
15.00††doi: 10.1145/3624062.3624257††isbn: 979-8-4007-0785-8/23/11

##  1\. Introduction

The field of quantum computing has seen a marked advancement
(boixo2018characterizing, ). However, despite the substantial potential,
present-day quantum computing mechanisms are challenged by considerable
environmental noise, and the efficacy of quantum error correction in the
Noise-Intermediate-Scale-Quantum (NISQ) era remains limited
(preskill2018quantum, ). Quantum circuit simulation serves as an essential
tool for researchers from a variety of disciplines, offering invaluable
benefits to validate the accuracy of quantum algorithms (jones2019quest, ).

Nonetheless, the simulation of quantum circuits poses significant challenges,
primarily because memory utilization escalates exponentially with the
increment of qubit quantity. For instance, the Frontier system, with a memory
capacity of 47.3 PB, is only equipped to simulate 51 qubits, while the Summit,
with a memory capacity of 2.8 PB, can merely simulate 47 qubits (wu2019full,
). In light of the reality that vast-memory systems such as high-performance
computing (HPC) or cloud infrastructures are not readily available to the
majority of practitioners in the field of quantum computing, the capability to
simulate the execution of quantum circuits is considerably restricted by
devices’ memory capacities.

Cutting-edge quantum state-vector simulators (li2021sv, ; zhang2022uniq, ;
zhang2021hyquas, ; hisvsim, ) have, thus far, not placed an emphasis on
minimizing the memory footprint during the simulation process. A prior
research endeavor (wu2019full, ) did incorporate compression into state vector
simulation with the aim of expanding the number of qubits accommodated within
a restricted memory space. This approach, while promising, still presents
unresolved complications that necessitate substantial research attention: (1)
In this study, compression and decompression processes occur with high
frequency, thereby constituting a significant portion of the total simulation
time. (2) it does not harness the processing power of GPUs to expedite the
simulation process. Despite the potential for supporting a greater number of
qubits, the subpar performance renders it impractical for wider application.
(3) Data locality is insufficiently utilized, resulting in low cache hit
rates. A more efficient memory access pattern is needed.

Figure 1. Modularized quantum circuit simulation overview.

In response to these challenges, we introduce a state vector quantum simulator
that leverages data compression, termed as MEMQSim. The high-level conceptual
overview of our approach is depicted in Figure 1. Our methodology employs data
compression to achieve high memory efficiency, with careful design to ensure
efficient simulation. MEMQSim is independent of quantum algorithm composition
and simulation computational tasks, offering significant modularity that
allows it to be incorporated into major simulation backends such as Qiskit,
Cirq, SV-Sim (li2021sv, ), UniQ (zhang2022uniq, ), among others. We make the
following contributions:

  * •

We introduce a highly memory-efficient, modular framework. MEMQSim is designed
to be compatible with the integration into state-of-the-art GPU simulator
backends.

  * •

We extend the potential of lossy compression into state-vector simulation to
augment the qubit threshold.

  * •

MEMQSim effectively manages the massive memory interchange between CPU and
GPU. We build a prototype of MEMQSim and show the preliminary results.

##  2\. Design and Preliminary Result

In this section, we present an in-depth design of a memory-efficient state
vector simulator via data compression. Our proposed system employs CPU to
store the state vector and GPU to execute the state vector updates. This
approach is adopted considering that user-level devices typically possess
larger CPU memory compared to GPU memory. Yet, the implementation of this
design is not straightforward due to the complexities of data management and
the intricacies of compression/decompression during the simulation.

Design challenges: (1) The intensive data exchange between the CPU and GPU
requires careful scheduling. Since normally GPU memory capacity is much less,
the GPU must retrieve data from the CPU, conducting computational operations
one partial data at a time. (2) The frequency and granularity of compression
and decompression significantly influence the simulation speed. Excessive
compression/decompression could result in substantial overhead on the end-to-
end time, and a coarser granularity could precipitate a significant memory
footprint issue, while excessively fine granularity could lead to a lower
compression ratio. (3) Different quantum algorithms’ behaviors affect the
access pattern on the state vector.

Figure 2. Data management design of MEMQSim.

In light of these identified challenges, we propose our design, MEMQSim. An
overview of our approach is illustrated in Figure 2. We explain the overall
simulation process below:

Offline stage MEMQSim partitions the input circuit and the corresponding state
vector and each data chunk of the state vector is compressed independently and
stored in CPU memory with such compressed format.

Online stage: As shown in Figure 1, MEMQSim pipelines the decompression,
buffer transfer between CPU and GPU and GPU computation. In particular,
MEMQSim (1) decompresses a selection of data chunks to the CPU buffers and (2)
transfers the corresponding state vector amplitudes to the GPU memory. This
process is repeated throughout the entire state vector until the GPU memory is
fully occupied with ordered state vector amplitudes. (3) MEMQSim initiates the
GPU kernel asynchronously to update the state vector amplitudes during the
CPU-GPU data transfer and (4) returns the updated values back to the CPU
buffers. (5) Subsequently, the CPU leverages idle cores to decompress the data
chunks and perform updates to the state vector amplitudes on the CPU side. (6)
Finally, the data block is re-compressed and stored back into the main memory.
Upon the GPU’s completion of a single iteration, the aforementioned procedure
is reiterated to update all amplitudes. Subsequently, the process advances to
the subsequent stage and continues until all stages have been addressed. We
have developed a prototype of MEMQSim, plugged into the SV-SIM (li2021sv, )
framework. As we move forward, our design harbours the potential to serve as a
plugin for a range of GPU simulators, while also being adaptable to
accommodate various compression algorithms.

As of step (2), we have devised two strategies to execute this process. The
first approach entails the transfer of corresponding state vector elements to
the GPU memory one at a time, utilizing CUDA asynchronous copies. The
alternate strategy involves allocating a buffer on the GPU side and shifting
the data chunk from the CPU buffer to the GPU buffer. Following this, GPU
threads are employed to map all these amplitudes to their appropriate
positions.

We present some preliminary results pertaining to the time taken by various
data movement strategies between the CPU and GPU, as depicted in 1. The
synchronous strategy entails the transfer of a complete data chunk through a
singular CUDA memory copy operation, thereby exemplifying the minimum time
necessary for the transfer between the CPU and GPU. As indicated, the host-to-
device time associated with the asynchronous strategy is approximately 870
times longer than the synchronous time. This discrepancy is achieved by
reducing multiple initiations of CUDA memory copy operations that cause
significant overhead. As for the buffer strategy, although it demands
additional memory space, it significantly boosts the data movement speed: the
time needed for the buffer strategy is only about 1.03x compared to the
synchronous version. By employing the state-of-the-art data compressor, we
extrapolate that on average 5 more qubits to simulate can be achieved without
slowing down the original quantum circuit simulation.

Table 1. Data transfer time H2D/D2H in seconds.

qubits $\downarrow$ Sync copy time Async copy time Buffer copy time 20
0.003/0.008 2.7/9.2 0.003/0.004 25 0.080/0.233 77.9/294.4 0.110/0.273

##  3\. Conclusion and Future Work

In this extended abstract, we have introduced a highly memory-efficient state
vector simulation of quantum circuits premised on data compression, harnessing
the capabilities of both CPUs and GPUs. We have elucidated the inherent
challenges in architecting this system, while concurrently proposing our
tailored solutions. Moreover, we have delineated our preliminary
implementation and deliberated upon the potential for integration with other
GPU-oriented simulators. In forthcoming research, we aim to present a more
comprehensive set of results, bolstering the assertion of the efficacy and
performance of our approach.

## References

  * [1] Sergio Boixo, Sergei V Isakov, Vadim N Smelyanskiy, Ryan Babbush, Nan Ding, Zhang Jiang, Michael J Bremner, John M Martinis, and Hartmut Neven.  Characterizing quantum supremacy in near-term devices.  Nature Physics, 14(6):595–600, 2018.
  * [2] Bo Fang, M. Yusuf Özkaya, Ang Li, Ümit V. Çatalyürek, and Sriram Krishnamoorthy.  Efficient hierarchical state vector simulation of quantum circuits via acyclic graph partitioning.  In CLUSTER, pages 289–300, 2022.
  * [3] Tyson Jones, Anna Brown, Ian Bush, and Simon C Benjamin.  Quest and high performance simulation of quantum computers.  Scientific reports, 9(1):1–11, 2019.
  * [4] Ang Li, Bo Fang, Christopher Granade, Guen Prawiroatmodjo, Bettina Heim, Martin Roetteler, and Sriram Krishnamoorthy.  Sv-sim: scalable pgas-based state vector simulation of quantum circuits.  In SC21, pages 1–14, 2021.
  * [5] John Preskill.  Quantum computing in the nisq era and beyond.  Quantum, 2:79, 2018.
  * [6] Xin-Chuan Wu, Sheng Di, Emma Maitreyee Dasgupta, Franck Cappello, Hal Finkel, Yuri Alexeev, and Frederic T Chong.  Full-state quantum circuit simulation by using data compression.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–24, 2019.
  * [7] Chen Zhang, Zeyu Song, Haojie Wang, Kaiyuan Rong, and Jidong Zhai.  Hyquas: hybrid partitioner based quantum circuit simulation system on gpu.  In Proceedings of the ACM International Conference on Supercomputing, pages 443–454, 2021.
  * [8] Chen Zhang, Haojie Wang, Zixuan Ma, Lei Xie, Zeyu Song, and Jidong Zhai.  Uniq: a unified programming model for efficient quantum circuit simulation.  In SC22, pages 692–707. IEEE Computer Society, 2022.