# One Noise to Rule Them All: Multi-View Adversarial Attacks with Universal
Perturbation

Mehmet Ergezer  0000-0001-6627-3667 Wentworth Institute of Technology550
Huntington AveBostonMAUSA<EMAIL_ADDRESS>,  Phat Duong  Wentworth Institute
of Technology550 Huntington AveBostonMAUSA<EMAIL_ADDRESS>,  Christian Green
Wentworth Institute of Technology550 Huntington AveBostonMAUSA
<EMAIL_ADDRESS>,  Tommy Nguyen  Wentworth Institute of Technology550
Huntington AveBostonMAUSA<EMAIL_ADDRESS>and  Abdurrahman Zeybey  Wentworth
Institute of Technology550 Huntington AveBostonMAUSA<EMAIL_ADDRESS>
###### Abstract.

This paper presents a novel “universal perturbation” method for generating
robust multi-view adversarial examples in 3D object recognition. Unlike
conventional attacks limited to single views, our approach operates on
multiple 2D images, offering a practical and scalable solution for enhancing
model scalability and robustness. This generalizable method bridges the gap
between 2D perturbations and 3D-like attack capabilities, making it suitable
for real-world applications.

Existing adversarial attacks may become ineffective when images undergo
transformations like changes in lighting, camera position, or natural
deformations. We address this challenge by crafting a single universal noise
perturbation applicable to various object views. Experiments on diverse
rendered 3D objects demonstrate the effectiveness of our approach. The
universal perturbation successfully identified a single adversarial noise for
each given set of 3D object renders from multiple poses and viewpoints.
Compared to single-view attacks, our universal attacks lower classification
confidence across multiple viewing angles, especially at low noise levels. A
sample implementation is made
available111https://github.com/memoatwit/UniversalPerturbation.

††journalyear: 2024

##  1\. Introduction

In recent years, deep learning has revolutionized various fields, including 3D
object recognition (Klokov and Lempitsky, 2017; Qi et al., 2017; Le and Duan,
2018; Wang et al., 2019; Zhi et al., 2018), where it plays a crucial role in
applications like augmented reality (Liu et al., 2019; Apicharttrisorn et al.,
2019; Li et al., 2020), autonomous driving (Li et al., 2019; Ravindran et al.,
2020; Zhang et al., 2020), and robotics (Hu et al., 2019; Hossain et al.,
2016; Finn et al., 2015). However, a major challenge lies in the vulnerability
of deep learning models to adversarial attacks (Ren et al., 2020). These
attacks entail generating imperceptible noise that, when added to a model’s
input, leads the model to make incorrect predictions. This poses serious
security and safety concerns, particularly in critical applications like self-
driving cars where misclassification can have catastrophic consequences
(Goodfellow et al., 2015; Luo et al., 2022). Ensuring adversarial robustness,
the ability of models to resist such attacks is crucial for real-world
deployments.

Conventional adversarial attacks primarily focus on crafting perturbations for
single views of objects, exploiting limitations in 2D image understanding
(Szegedy et al., 2013; Madry et al., 2017). These attacks face limitations
when applied to 3D objects, as they often fail to transfer across different
viewing angles and lack robustness to real-world variations in perspective.
This motivates the need for multi-view adversarial attacks, capable of fooling
object recognition models across diverse poses and viewpoints.

In this work, we propose a novel “universal perturbation” method for
generating robust multi-view adversarial images. Our approach departs from
traditional per-view attacks by crafting a single noise perturbation
applicable to various views of the same object. This single-noise, multi-view
attack offers several advantages:

Robustness: The trained noise exhibits improved transferability across diverse
viewing angles, leading to more robust attacks compared to single-view
methods.

Efficiency: Training a single perturbation is faster and less computationally
intensive compared to generating individual perturbations for each view.

Scalability: Our approach is readily applicable to different object categories
and viewpoints, offering a generalized framework for multi-view adversarial
attacks.

This paper investigates the effectiveness of our “universal perturbation”
method in comparison to conventional single-view attacks. We conduct
comprehensive experiments on various 3D object datasets, evaluating the attack
success rates, and transferability across different views We believe this work
represents a significant step towards developing efficient and robust
adversarial attacks for 3D object recognition, paving the way for improved
model security and robustness in real-world applications.

The paper is organized as follows: Section 2 introduces related adversarial
attack methods, focusing on single-view attacks. The universal perturbation
algorithm is formulated in Section 3. Sections 4 and 5 present the setup for
our experiments, including object selection and rendering, adversarial
attacks, related metrics, and results. Section 6 discusses the potential and
limitations of universal perturbation. Section 7 provides conclusion remarks.

##  2\. Related Work

In this section, we provide an overview of the existing adversarial algorithms
that are designed for single-view attacks.

###  2.1. Fast Gradient Sign Method

One of the earliest adversarial attack methods, Fast Gradient Sign Method
(FGSM), was proposed in 2014 (Goodfellow et al., 2015). The method was popular
for forging adversarial examples due to its simplicity in generation and
implementation and relatively low cost of execution. FGSM attacks rely heavily
on knowing the architecture and the weight of the victim model—known as a
white-box attack—which in turn may make the method poor when it comes to
transferability and in black-box settings where only the input features are
known.

FGSM works by first calculating the loss based on the predicted class after
performing forward propagation. We determine the gradients with respect to the
input image and update the image’s pixels in the direction that maximizes the
loss. In Equation 1, we let $X$ and $y$ be the input image and true label
respectively. We assume $X$ as a 3-D matrix (width × height × color).
$\epsilon$ represents a constant that determines the strength of the
perturbation. $\nabla_{X}J(X,y_{true})$ is the gradient of the model’s loss
with respect to $X$. The perturbation is calculated by taking the sign of this
gradient and adding it to the original image. We initialize $X^{adv}_{0}=X$.

(1) $X^{adv}=X+\epsilon\cdot\text{sign}(\nabla_{X}J(X,y_{true}))$

While in this study we focus primarily on untargeted adversarial attacks, the
FGSM algorithm can be updated to target specific class labels. This can be
accomplished by simply changing the direction of the gradients to minimize the
loss between the targeted and predicted labels as shown in Equation 2.

(2) $X^{adv}=X+\epsilon\cdot\text{sign}(\nabla_{X}J(X,y_{target}))$

###  2.2. Basic Iterative Method

Kurakin et al. proposed several new iterative versions of FGSM, including the
Basic Iterative Method (BIM) and iterative least-likely class method (ILCM)
(Kurakin et al., 2017). Both of these methods brought adversarial attacks to
the physical world by inputting captures from a mobile phone camera, instead
of feeding source images directly to the AI model. ILCM extends BIM by
allowing the attacker to specify a desired target label for the attacked
object.

The Basic Iterative Method (BIM) extends the idea of FGSM by applying it
iteratively, allowing for a more fine-tuned manipulation of the input image.
This iterative approach often results in more effective adversarial examples
than single-step perturbation. Note that intermediate results are clipped at a
pixel level after each step. Just like with FGSM, we initialize BIM,
$X^{adv}_{0}=X$. $N$ in Equation 3 represents the iteration count.

(3)
$X_{N+1}^{adv}=\text{Clip}_{X,\epsilon}\\{X_{N}^{adv}+\epsilon\cdot\text{sign}(\nabla_{X}J(X_{N}^{adv},y_{true}))\\}$

###  2.3. Other Notable Adversarial Algorithms

Another prevalent method is the Carlini and Wagner attack (Carlini and Wagner,
2017). Its goal is to generate a perturbation to the input data that causes
the target neural network to misclassify while limiting the disturbance’s
perceptibility. The attack consists of solving an optimization problem with
numerous objectives and constraints.

However, due to low success rates with adversarial attacks in black-box
settings, Dong et al. proposed a class of momentum-based iterative algorithms
to boost adversarial attacks called MI-FGSM (Dong et al., 2018). This class of
adversarial attacks can be used for white-box attacks as well as outperformed
other one-step gradient methods in a black-box setting.

This still has not addressed the transferability issue with black-box-based
FGSM attacks, which is where (Lin et al., 2020) proposed two new methods in
2019, Nesterov Iterative Fast Gradient Sign Method (NI-FGSM), and Scale-
Invariant Attack (SIM).

Since then, various other modifications have been made to improve adversarial
attacks. Notably, in 2020, (Andriushchenko and Flammarion, 2020) proposed a
regularization method named GradAlign, which prevents catastrophic overfitting
that is common in FGSM models.

##  3\. Universal Perturbation

The methodologies outlined in Section 2 are all tailored to attack a solitary
image, denoted as $X$, generating a corresponding adversarial noise,
$X^{adv}$, for each individual input. However, our objective is to extend
single-view attacks; we aim to devise a singular perturbation that can be
universally applied across various perspectives of the same object or even
different objects, $\mathbb{X}^{\text{adv}}$.

To achieve this, we propose a modification to the basic iterative method.
Instead of computing gradients with respect to the input image, we consider
the gradients with respect to the adversarial noise itself. This adjustment
decouples the number of input images from the shape of the generated
adversarial noise. Consequently, it allows the attacker to control both the
shape of the input images and the associated noise, independently. This update
allows simultaneous input of multiple views of the same object, enabling the
calculation of a solitary adversarial noise capable of collectively
compromising the recognition of all these distinct perspectives at once.

Equation 4 introduces the universal perturbation calculation where $X$ is
stacked input images as a 4-D tensor ( image count × width × height × color),
$\mathbb{X}$ is the calculated perturbation as a 3-D matrix (width × height ×
color), matching the shape of an individual image in $X$. $N$ is the number of
desired iterations as defined in BIM, costing calls to the backpropagation.
$\epsilon$ is a hyperparameter controlling the scale of the attack as used in
FGSM. $\nabla_{\mathbb{X}_{N}}J(\mathbb{X}_{N}^{\text{adv}},y_{\text{true}})$
is the gradient of the cross-entropy loss function of the model with respect
to the calculated perturbation.

(4) $\displaystyle\mathbb{X}_{N+1}^{\text{adv}}$
$\displaystyle=\text{Clip}_{X,\epsilon}\\{\mathbb{X}_{N}^{\text{adv}}+\epsilon\cdot\text{sign}(\nabla_{\mathbb{X}_{N}}J(\mathbb{X}_{N}^{\text{adv}},y_{\text{true}}))\\}$

Unlike BIM, we initialize $\mathbb{X}^{adv}_{0}=X+r$ where $r\sim
U(-0.01,0.01)$. Section 6 discusses the reasoning and intricacies associated
with this initialization.

We make our implementation of the universal perturbation publicly
available222https://github.com/memoatwit/UniversalPerturbation and present its
pseudocode at 1.

Algorithm 1 Generate Universal Perturbation

Input: images of object(s) from different views
Parameter: target_model, $\epsilon=0.01$, num_iterations=1, regularization=1.,
clip_value=1.
Output: Adversarial noise

1: $perturbation\leftarrow\text{random.uniform(shape=images[0].shape,
range$=\pm 0.01$}$)

2: for $it$ in range(num_iterations) do

3: Compute gradient($perturbation$)

4: $adv\\_images\leftarrow images+perturbation$

5: $predictions\leftarrow target\\_model(adv\\_images)$

6: $loss\leftarrow\text{crossentropy(perturbation)}$

7: $loss\mathrel{+}=regularization\cdot\text{norm}(perturbation)$

8: $gradients\leftarrow\text{gradient}(loss,perturbation)$

9: $gradients\leftarrow\text{clip}(gradients,clip\\_value)$

10: $perturbation\mathrel{+}=(\epsilon\cdot\text{sign}(gradients))$

11: $perturbation\leftarrow\text{clip}(perturbation,clip\\_value)$

12: end for

13: return $perturbation$

##  4\. Experiments and Results

In this section, we describe the system setup employed to automatically
generate multi-view renders of objects, how we evaluate the FGSM, BIM, and
universal perturbation attacks, and our analysis of their results.

The experiments detailed in the following sections will substantiate our
approach by assessing its true class accuracy drop, transferability across
angles, and noise imperceptibility, while comparing the outcomes with those
obtained using FGSM and BIM. Additionally, we will showcase the computational
efficiency of our model in comparison to FGSM and BIM for generating
transferable multi-angle noise.

###  4.1. Experimental Setup

To showcase the effectiveness of our model in generating robust noise across
multiple angles, we conducted experiments comparing our universal perturbation
method with common adversarial attack methods: FGSM and BIM. The objective was
to identify the optimal noise level that significantly reduces a classifier’s
confidence in the object’s true class while demonstrating that our universal
perturbation method produces more robust noise than FGSM or BIM from multiple
viewpoints of an object.

Algorithm 2 provides an overview of our experimental process. Unlike FGSM and
BIM, our universal perturbation approach generates a single noise pattern
applicable to all angles, resulting in consistent adversarial outcomes. Our
universal perturbation method is designed to be more resilient in producing
multi-angle consistent noise, enabling attacks in 3D space. To evaluate our
method alongside FGSM and BIM, we conducted multiple runs of FGSM and BIM to
generate noise for various angles of our objects.

Algorithm 2 Experimental System

Input: 3D Blender model with texture
Parameter: $\epsilon$, $n\\_angles$ to render, num iterations
Output: Adversarial image and its classification

1: Render object from multiple angles, $n\\_angles$, in Blender

2: Split $n\\_angles$ images to two: $tr_{n}$ to generate noise and $ts_{n}$
to test the generate noise(s)

3: Identify universal perturbation for $tr_{n}$ views to find a single
$perturbation$ for all $tr_{n}$

4: Generate adversarial images: $ts_{n}^{adv}\leftarrow ts_{n}+(\epsilon\cdot
perturbation)$

5: Classify the adversarial $ts_{n}^{adv}$ images with MobileNetv2

6: Repeat above for FGSM and BIM, except each $tr_{n}$ gets a unique
$perturbation$ calculated.

7: return top-1 and top-5 accuracy for these attacks

###  4.2. 3D Model Creation and Rendering in Blender from Multiple Views.

To generate diverse images of objects, we employed Blender and various 3D
models. For each object, we rendered images from ten distinct view angles,
ensuring consistent recognition by our classification model across
perspectives. Camera angles were randomly selected based on a spherical
coordinate system centered on the object, incorporating a 15% random deviation
for robustness. This mimics natural variations in viewing angles encountered
in real-world scenarios. Constant lighting position relative to the camera
prevented shadowed areas that could hinder recognition. Additionally, we
utilized five diverse 3D models that were categorized by imagenet (Deng et
al., 2009) as discussed in Section 4.3.

###  4.3. Object Selection

To validate our methods, we selected five diverse 3D objects: baseball (Bes,
2017), dining table (mitsui, 2020), lemon (dannyboy70000, 2014), shovel
(GetDeadEntertainment, 2020), and tractor (selfie 3D scan, 2019), that were
readily renderable on Blender from various viewpoints. These objects offer
varying shapes, sizes, and textures, representing a range of potential real-
world applications.

Our pipeline utilizes Blender to generate multi-angle views of objects, at 224
by 224 resolution, suitable for our image classification model, MobileNetV2.
Currently, we render ten images of each object from different angles. These
angles are determined using sinusoidal and cosinusoidal functions, ensuring
even distribution across the viewing sphere. Additionally, random rotations
are introduced to simulate natural object orientations.

In each case, we generate noise based on five training images and then apply
the noise to five testing images to evaluate robustness. For the universal
attack, a single noise is generated based on the five training images and then
applied to the remaining images in the testing set. In FGSM and BIM, we
generate five different noises for each image in the training set and then
apply these noises to five separate testing images. The resulting noisy images
from each model are then passed through MobileNetV2, from which we extract the
top-1 and top-5 predicted classes.

##  5\. Results of Multi-view Attacks

###  5.1. Accuracies and Confidences

Table 1 summarizes the average accuracies of predicting the true object label
when images are subjected to different adversarial attacks. The comparison
encompasses three attack methods: FGSM, BIM, and our proposed universal
perturbation method. We split the table in two to analyze the robustness of
these attacks across 25 training and 25 unseen images corresponding to
multiple poses of our five objects.

For the test images, the best attack result for each $\epsilon$, corresponding
to the lowest probability of true label, is bolded. We note that the universal
attacks provide the highest success at the lowest noise levels, while FGSM is
more successful at higher $\epsilon$. We also list the mean and standard
deviation of true label prediction rates where a lower values indicate a more
successful attack. Universal perturbation provides the minimum average true
level probability across all $\epsilon$. This indicates the effectiveness of
the proposed algorithm on unseen images at the most challenging $\epsilon$
levels.

Table 1 also shows the effectiveness of the FGSM and universal attacks on
training images, achieving success even at lower epsilon values. This suggests
that during the training phase, the model may become overly adapted to these
perturbations, a vulnerability that these attacks exploit efficiently. As
$\epsilon$ increases to 15, these attacks also demonstrate marked success on
the test images, with accuracies plummeting to below 5%. This indicates that
at the $\epsilon$ threshold of 15, the adversarial modifications are
sufficiently pronounced to compromise the model’s ability to generalize,
leading to a significant degradation in performance on unseen data.

$\epsilon$ | Train Images | Test Images
---|---|---
|  FGSM | BIM | Universal |  FGSM | BIM | Universal
0.00 | 0.70 | 0.70 | 0.70 | 0.65 | 0.65 | 0.65
0.50 | 0.13 | 0.65 | 0.30 | 0.56 | 0.57 | 0.47
1.00 | 0.11 | 0.62 | 0.27 | 0.62 | 0.56 | 0.51
3.00 | 0.18 | 0.62 | 0.35 | 0.52 | 0.61 | 0.49
5.00 | 0.19 | 0.57 | 0.34 | 0.36 | 0.56 | 0.38
10.00 | 0.21 | 0.39 | 0.28 | 0.19 | 0.43 | 0.31
15.00 | 0.08 | 0.35 | 0.09 | 0.04 | 0.32 | 0.09
30.00 | 0.00 | 0.06 | 0.00 | 0.00 | 0.02 | 0.00
50.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Mean | 0.18 | 0.44 | 0.26 | 0.33 | 0.41 | 0.32
Std $(\pm)$ | 0.21 | 0.26 | 0.22 | 0.27 | 0.25 | 0.24
Table 1. Average accuracy of MobileNetV2 under different adversarial attacks
at various epsilon values for both train and test images. $\epsilon=0.00$
indicates clean images without an attack.

In Fig. 1 and Fig. 2, we evaluate the top-1 and top-5 classification
accuracies for training and test images, respectively. Sample images of our
lemon object, perturbed at $\epsilon=15$ or a 15% noise level, are shown in
Fig.3 to demonstrate the different noise types generated by each algorithm. We
observe that as noise level, $\epsilon$ increases, we achieve successful
misclassifications across all models. Both the FGSM and universal attacks
exhibited faster degradation in top-1 and top-5 accuracies compared to BIM.

Based on the results, it is evident that our universal perturbation method
outperforms BIM, as indicated by consistently lower accuracy levels for the
true class across all images of the object displayed. While our method may not
always achieve the same level of performance as FGSM, it’s essential to
recognize that our approach aims to generate a single noise applicable across
multiple angles of an object. In contrast, FGSM and iterative methods generate
separate noises designed for each angle of the object individually.

Figure 1. Top-1 and top-5 accuracies of MobileNetV2, after adversarial attacks
with $\epsilon$ values ranging from 0.5 to 50, were compared with those for
clean Images—unmodified images from the dataset. The accuracies were
calculated using a set of 25 train images, which were rendered from 5 distinct
3D object models. Figure 2. Top-1 and top-5 accuracies of MobileNetV2, after
adversarial attacks with epsilon $\epsilon$ values ranging from 0.5 to 50,
were compared with those for clean images—unmodified images from our dataset.
The accuracies were calculated using a set of 25 test images, which were
rendered from 5 distinct 3D object models. Figure 3. Comparison of adversarial
attacks on the lemon object, each with an $\epsilon$ value of 15. MobileNetV2
misclassifies these adversarial images in both top-1 and top-5 predictions.
The noise level generated by BIM appears smoother than the other methods.

###  5.2. Adversarial Object Samples

While we evaluated our method on various objects, here we analyze the shovel
due to its intricate shape and sensitivity to viewing angle. This sensitivity
makes it an excellent candidate for showcasing the effectiveness and
robustness of our universal perturbation noise.

Figure 4 presents the outputs of FGSM, BIM, and our universal perturbation
attacks, respectively, all applied with the same noise level, $\epsilon=0.05$,
to five different viewing angles of the shovel. Each training image displays
the predicted label probability for the correct class. Before the attacks,
these training images were correctly classified with confidence ranging from
0.64 to 0.88.

Figure 4. Training images for the shovel object as attacked by FGSM, BIM, and
Universal algorithms, respectively, at $\epsilon=0.05$. The prediction
confidence of the correct label for each image is indicated on top.

##  6\. Discussion and Limitations

In this paper, the universal attacks do not target a user-specified label. A
least-likely penalty, as proposed in ILCM, could be added to the loss function
to specify a desired adversarial target. When extending this method to
targeted attacks, we suspect that a larger number of iterations may then be
necessary. For untargeted attacks presented here, we found a single epoch to
be sufficient. We leave targeted attacks to be future work.

Another potential difference between Universal and FGSM/BIM lies in the
initialization of the adversarial noise. Our method may be sensitive to the
initialization of adversarial noise, unlike BIM or FGSM methods. Since we
calculate the gradients with respect to the noise, instead of the image,
certain initial values of noise can cause numerical instabilities, including
initializing noise to zero. Consequently, initial adversarial noise could be
considered another parameter to be tuned in the future. Currently, we
initialize the noise as a uniformly distributed value between
$\\{-0.01,0.01\\}$. Future endeavors may involve developing an algorithm to
determine the optimal initialization or providing a default initialization
that is effective across different objects.

While further work is needed to address these points, our method offers
distinct advantages like efficiency, multi-view applicability, and potential
scalability. Future research could explore techniques to enhance accuracy,
investigate targeted attack scenarios, and incorporate different
classification models for broader applicability.

##  7\. Conclusion

In this work, we presented a novel “universal perturbation” method for
generating robust multi-view adversarial examples for 3D object recognition.
Our method operates solely on 2D images, offering a practical and scalable
alternative to computationally expensive 3D adversarial attacks. By crafting a
single noise perturbation applicable to various object views, our approach
achieves 3D-like attack capabilities while retaining the efficiency and
practicality of single-view methods.

Experiments on diverse 3D objects demonstrated the effectiveness of our
approach. Our universal attacks successfully identified single noise
perturbations with higher destruction rates across multiple viewing angles,
especially at low $\epsilon$ levels. This performance highlights the
improvements over conventional single-view attacks, which may struggle with
viewpoint variations or need to identify a noise for each unique viewpoint.

Beyond its effectiveness, our method offers key advantages in terms of
efficiency and scalability. By requiring only a single perturbation generation
instead of per-view computations, our approach reduces computational resources
compared to other methods. This efficiency makes it particularly well-suited
for real-world applications where computational constraints or multi-view
points are present.

While our work represents a significant step forward, we acknowledge potential
limitations and opportunities for future research. Further evaluation of a
wider range of object categories and classification models could enhance the
generalizability of our findings. Additionally, extending our method for
targeted attacks is an intriguing direction for future work.

## References

  * (1)
  * Andriushchenko and Flammarion (2020) Maksym Andriushchenko and Nicolas Flammarion. 2020.  Understanding and improving fast adversarial training.  _Advances in Neural Information Processing Systems_ 33 (2020), 16048–16059.
  * Apicharttrisorn et al. (2019) Kittipat Apicharttrisorn, Xukan Ran, Jiasi Chen, Srikanth V Krishnamurthy, and Amit K Roy-Chowdhury. 2019.  Frugal following: Power thrifty object detection and tracking for mobile augmented reality. In _Proceedings of the 17th Conference on Embedded Networked Sensor Systems_. 96–109.
  * Bes (2017) Alex Bes. 2017.  Worn Baseball Ball.  https://sketchfab.com/3d-models/worn-baseball-ball-fdf3de6ae225421ea78961b897b9608a Last accessed 10 February 2024.
  * Carlini and Wagner (2017) Nicholas Carlini and David Wagner. 2017.  Towards Evaluating the Robustness of Neural Networks.  arXiv:1608.04644 [cs.CR]
  * dannyboy70000 (2014) dannyboy70000. 2014.  lemon 3D Model.  https://free3d.com/3d-model/lemon-72357.html Last accessed 10 February 2024.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Dong et al. (2018) Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018.  Boosting Adversarial Attacks with Momentum.  arXiv:1710.06081 [cs.LG]
  * Finn et al. (2015) Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. 2015.  Learning visual feature spaces for robotic manipulation with deep spatial autoencoders.  _arXiv preprint arXiv:1509.06113_ 25 (2015), 2.
  * GetDeadEntertainment (2020) GetDeadEntertainment. 2020.  Medieval Shovel.  https://www.turbosquid.com/3d-models/medieval-shovel-3d-model-1494436 Last accessed 10 February 2024.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and Harnessing Adversarial Examples.  arXiv:1412.6572 [stat.ML]
  * Hossain et al. (2016) Delowar Hossain, Genci Capi, and Mitsuru Jindai. 2016.  Object recognition and robot grasping: A deep learning based approach. In _The 34th Annual Conference of the Robotics Society of Japan (RSJ 2016), Yamagata, Japan_.
  * Hu et al. (2019) Zhe Hu, Tao Han, Peigen Sun, Jia Pan, and Dinesh Manocha. 2019.  3-D deformable object manipulation using deep neural networks.  _IEEE Robotics and Automation Letters_ 4, 4 (2019), 4255–4261.
  * Klokov and Lempitsky (2017) Roman Klokov and Victor Lempitsky. 2017.  Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In _Proceedings of the IEEE international conference on computer vision_. 863–872.
  * Kurakin et al. (2017) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017.  Adversarial examples in the physical world.  arXiv:1607.02533 [cs.CV]
  * Le and Duan (2018) Truc Le and Ye Duan. 2018.  Pointgrid: A deep network for 3d shape understanding. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 9204–9214.
  * Li et al. (2019) Peiliang Li, Xiaozhi Chen, and Shaojie Shen. 2019.  Stereo r-cnn based 3d object detection for autonomous driving. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7644–7652.
  * Li et al. (2020) Xiang Li, Yuan Tian, Fuyao Zhang, Shuxue Quan, and Yi Xu. 2020.  Object detection in the context of mobile augmented reality. In _2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_. IEEE, 156–163.
  * Lin et al. (2020) Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. 2020.  Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks.  arXiv:1908.06281 [cs.LG]
  * Liu et al. (2019) Luyang Liu, Hongyu Li, and Marco Gruteser. 2019.  Edge assisted real-time object detection for mobile augmented reality. In _The 25th annual international conference on mobile computing and networking_. 1–16.
  * Luo et al. (2022) Xi-Yu Luo, Yong Yu, Jian-Long Liu, Ming-Yang Zheng, Chao-Yang Wang, Bin Wang, Jun Li, Xiao Jiang, Xiu-Ping Xie, Qiang Zhang, et al. 2022\.  Entangling metropolitan-distance separated quantum memories.  _arXiv preprint arXiv:2201.11953_ (2022).
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ (2017).
  * mitsui (2020) mitsui. 2020.  table 3D Model.  https://free3d.com/3d-model/table-747735.html Last accessed 10 February 2024.
  * Qi et al. (2017) Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017.  Pointnet: Deep learning on point sets for 3d classification and segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 652–660.
  * Ravindran et al. (2020) Ratheesh Ravindran, Michael J Santora, and Mohsin M Jamali. 2020.  Multi-object detection and tracking, based on DNN, for autonomous vehicles: A review.  _IEEE Sensors Journal_ 21, 5 (2020), 5668–5677.
  * Ren et al. (2020) Kui Ren, Tianhang Zheng, Zhan Qin, and Xue Liu. 2020.  Adversarial attacks and defenses in deep learning.  _Engineering_ 6, 3 (2020), 346–360.
  * selfie 3D scan (2019) selfie 3D scan. 2019.  Tractor.  https://sketchfab.com/3d-models/tractor-1b258bcc01bf4ed0935ef73e80442c30 Last accessed 10 February 2024.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ (2013).
  * Wang et al. (2019) Cheng Wang, Ming Cheng, Ferdous Sohel, Mohammed Bennamoun, and Jonathan Li. 2019.  NormalNet: A voxel-based CNN for 3D object classification and retrieval.  _Neurocomputing_ 323 (2019), 139–147.
  * Zhang et al. (2020) K Zhang, SJ Wang, L Ji, and C Wang. 2020.  DNN based camera and LiDAR fusion framework for 3D object recognition. In _Journal of Physics: Conference Series_ , Vol. 1518. IOP Publishing, 012044.
  * Zhi et al. (2018) Shuaifeng Zhi, Yongxiang Liu, Xiang Li, and Yulan Guo. 2018.  Toward real-time 3D object recognition: A lightweight volumetric CNN framework using multitask learning.  _Computers & Graphics_ 71 (2018), 199–207.


# Textless Speech-to-Music Retrieval Using Emotion Similarity

###### Abstract

We introduce a framework that recommends music based on the emotions of
speech. In content creation and daily life, speech contains information about
human emotions, which can be enhanced by music. Our framework focuses on a
cross-domain retrieval system to bridge the gap between speech and music via
emotion labels. We explore different speech representations and report their
impact on different speech types, including acting voice and wake-up words. We
also propose an emotion similarity regularization term in cross-domain
retrieval tasks. By incorporating the regularization term into training,
similar speech-and-music pairs in the emotion space are closer in the joint
embedding space. Our comprehensive experimental results show that the proposed
model is effective in textless speech-to-music retrieval.

Index Terms—  Cross-domain retrieval, Multi-modal retrieval

##  1 Introduction

In multimedia content, emotions are delivered firstly through speech, facial
expressions, and gestures. The presented emotions are often reinforced when
they are matched with suitable music. For example, if a movie scene is about a
romantic conversation, the mood will be enhanced with a romantic soundtrack
(illustrated in Figure 1). However, this cross-domain matching process, i.e.,
speech emotion recognition and music search, is time-consuming and requires
domain expertise.

In order to tackle such retrieval problems, previous studies formalize the
task as a cross-modal retrieval [1] – e.g., music video-to-music [2], user
video-to-music [3, 4], and storytelling text-to-music [5]. A common approach
for cross-domain retrieval is multimodal representation learning which
transforms the data samples from different modalities into a joint embedding
space. The joint embedding space is optimized to align data distributions of
heterogeneous modalities. The alignment problem was solved by unsupervised,
pairwise-based, rank-based, and supervised methods in [6], where the methods
leverage co-occurrence data, pairwise data, rank lists, and annotated labels,
respectively. However, speech and music cannot co-occur organically – they are
only intentionally made to co-occur – hence we do not have an organically
paired dataset of speech and music. A possible solution to rely on unpaired,
separately existing speech datasets and music datasets is a supervised method
that uses the relationship between the labels of each domain.

However, mismatched dataset taxonomies (emotion vocabularies) between domains
cause immediate difficulties. This issue has been addressed by emotion-based
mapping where each tag is transformed into the most similar emotion tag, e.g.,
joy (from speech dataset)$\xrightarrow{}$happy (from music data). Won et al.
[5] formalized the task of matching storytelling to music as a supervised
cross-modal retrieval. They introduced three possible emotion-based mappings:
(1) mapping based on the Euclidean distance between emotion labels in the
predefined valence-arousal space, (2) the cosine distance between emotion
labels in predefined word embedding space, or (3) direct manual mapping of
emotion labels. This emotion-mapping approach allows the model to bridge the
modality gap. While such mapping methods provide a data-driven alignment, this
may lack information on continuous emotion distribution due to the discrete
nature of the most similar mapping. A direct, manual mapping may address the
issue of most similar mapping by one-to-many mapping (e.g.,
joy$\xrightarrow{}${happy, excitement}). The annotation cost increases
proportionally with the number of labels.

Fig. 1: User scenario of speech-to-music retrieval with a smart speaker.  Fig.
2: The proposed speech-to-music retrieval framework. $S_{y}$ and $S_{z}$ are
encoded pairwise similarities of the emotion label and feature, respectively.
The goal of the loss function is to encourage items (speech, music) with
similar emotion labels to be similar in the feature space.

In this work, we formalize matching the speech-to-music task as a supervised
cross-domain retrieval problem.111We use the term cross-domain instead of
cross-modal because speech audio and music audio share the same modality.
Inspired by [7], we propose an emotion similarity regularization term, which
take advantage of continuous emotion similarity information. For example, we
guide the network in a way that the feature representation for the happy
speech is more similar to the music representation in order: exciting, funny,
tensor, sad, angry, and scary music. We explore different speech
representations and report their impact on different speech types including
acting voice and wake-up words. Experiments show that we can successfully
bridge the gap between speech and music to facilitate cross-domain retrieval.
We present the code and demo online.222https://seungheondoh.github.io/speech-
to-music-demo

##  2 Speech-to-Music Retrieval

Similar to the previous work [5], our cross-domain retrieval model comprises
two parts: pre-trained encoder networks for feature extraction and projection
layers for bridging the domain gap. In the proposed systems, speech is
represented by audio, text, and fusion modalities. As shown in Figure 2, the
architecture takes two different domain data from speech and music. Speech
data include audio, transcription text, and emotion labels. Music data include
audio and emotion labels. The domain data are processed by pre-trained encoder
networks and projection multilayer perceptrons (MLPs). The pre-trained encoder
networks extract a domain feature, and the projection MLPs transform the
outputs of each domain feature into a joint embedding space.

###  2.1 Pre-trained Models for Feature Extraction

The large-scale pre-trained model revealed promising generalization
performance in emotion recognition as a downstream task, for example, music
[8, 9], speech [10, 11, 12], text-based [13]. As shown in Figure 2, the
proposed architecture transforms music, speech into a joint space through pre-
trained encoder models. For the music encoder, we use the Music Tagging
Transformer [14] which is the current state-of-the-art music tagging model.
For the speech audio encoder, we use Wav2vec 2.0 [15] and take the average
pooling on its phoneme-level feature to summarize them into utterance-level
[11]. For the speech text encoder, we use DistilBERT [16], which is a compact
variant of the popular BERT transformer model. For music audio, speech audio,
and speech text encoders, we use pre-trained networks fine-tuned through a
classification task (See Table 1). For the emotion label, we use valence-
arousal representation from VAD Lexicon [17].

###  2.2 Cross-Domain Retrieval

In this section, we introduce speech-to-music retrieval frameworks
(illustrated in Figure 2). In the following descriptions, the notations
$x_{s},y_{s}$ and $x_{m},y_{m}$ represent the speech and music examples, along
with their corresponding emotion labels. The notations $f$ and $g$ denote the
pre-trained encoders for speech and music, and $h_{s}$ and $h_{m}$ denote the
projection MLPs for speech and music, respectively. Each domain input is
processed by the corresponding pre-trained encoder $f$ or $g$ and projection
MLP $h_{s}$ or $h_{m}$. We denote the two output embeddings as
$z_{s}=h_{s}(f(x_{s}))$ and $z_{m}=h_{m}(g(x_{m}))$, respectively. On the
cross-domain retrieval model, we freeze the pre-trained encoder $f,g$, and
train the projection MLP network $h_{s},h_{m}$.

We trained our retrieval model using deep metric learning with a triplet loss.
The goal of metric learning is to learn an embedding space where inputs from
the same semantics are closer to each other than those from different
semantics. Given a triplet embedding ($z$, $z^{+}$, $z^{-}$), the metric
learning model is optimized to minimize a triplet loss.

$\mathcal{L}_{\text{Triplet}}(z,z^{+},z^{-})=\text{max}\\{0,D(z,z^{+})-D(z,z^{-})+\delta\\}$
(1)

where $z,z^{+},z^{-}$ are embedding of anchor, positive and negative sample.
$D$ is a cosine distance function and $\delta$ is a predefined margin. In our
cross-domain retrieval scenario, the anchor is query speech embedding
($z_{s}$), and positive and negative are music embeddings
($z_{m}^{+},z_{m}^{-}$). Our baseline model is the conventional triplet
network that optimizes the loss function $\mathcal{L}$:

$\mathcal{L}_{\text{Cross}}=\mathcal{L}_{\text{Triplet}}(z_{s},z_{m}^{+},z_{m}^{-})$
(2)

Meanwhile, a different taxonomy problem occurs in the triplet sampling of
$\mathcal{L}_{\text{Cross}}$. To solve this problem, we connect the emotion
labels ($y_{s},y_{m}$) of the two domains using valence-arousal mapping
following the previous study [5]. After calculating the Euclidean distance
between speech and music taxonomies (See Figure 3), we sample a music item
that is the most similar to speech in the emotional space.

However, the triplet loss does not consider the neighborhood structure or data
distribution within modalities. While the relationship between speech and
music is considered, the relationship between each item and the emotion tag is
not. The previous work [5] addresses this problem with structure-preserving
constraints [18, 19]. With a ground truth emotion tag, each loss function is
designed to optimize emotion-to-speech and emotion-to-music triplet loss as
follows:

$\begin{split}\mathcal{L}_{\text{SP-
Speech}}=\mathcal{L}_{\text{Triplet}}(z_{e},z_{s}^{+},z_{s}^{-})\\\
\mathcal{L}_{\text{SP-
Music}}=\mathcal{L}_{\text{Triplet}}(z_{e},z_{m}^{+},z_{m}^{-})\end{split}$
(3)

where $z_{e}$ denotes emotion tag embedding. We use the GloVe pre-trained word
embedding [20] with a projection MLP layer for emotion tag embedding $z_{e}$.
The model learns a shared embedding space between emotion, speech, and music.
With these constraints, our final loss is given by
$\mathcal{L}_{\text{Triplet+SP}}=w_{1}\mathcal{L}_{\text{Cross}}+w_{2}\mathcal{L}_{\text{SP-
Speech}}+w_{3}\mathcal{L}_{\text{SP-Music}}$ where $w$ allows us to control
the trade-off between the cross-domain mapping and the within-domain emotion
mapping. Following a previous work [5], we used 0.4, 0.3, and 0.3 for all
$w_{1},w_{2},w_{3}$, respectively.

Fig. 3: Speech-to-Music Emotion Similarity using Valence Arousal Mapping. The
x-axis is the Audioset music mood emotion label, and the y-axis is the emotion
label of each speech dataset. The color is the value of valence-arousal
similarity between the two labels.

###  2.3 Emotion Similarity Regularization

To preserve the continuous emotion distribution in joint embedding space, we
adopt the work of Gong et al, which proposed the use of rank similarity
regularization (RankSim) [7] for deep imbalanced regression task. We propose
the emotion similarity regularization (EmoSim), a modified version of RankSim,
for the cross-domain retrieval task. An overview of our approach is shown in
Figure 2. In practice, our goal is to encourage the alignment between the
similarity of neighbors in emotion space ($S^{y}$) and the similarity of
neighbors in feature space ($S^{z}$). The EmoSim regularization term is
formulated as follows:

$\mathcal{L}_{\text{EmoSim}}=\frac{1}{N}\sum^{N}_{i=1}\ell(S^{y}_{i},S^{z}_{i})$
(4)

where $S^{y}_{i}$ and $S^{z}_{i}$ denote the similarity scores of $i$ th
speech and music items respectively, $\ell$ is a mean squared error, and $N$
is the mini-batch size. During training, we only use the unique similarity
scores and enhance the relative representation of infrequent labels. Instead
of calculating the rank, we directly compute the difference between the two
similarity scores. With this regularization, our final loss is given by
$\mathcal{L}_{\text{Triplet+Emosim}}=\mathcal{L}_{\text{Cross}}+\lambda\mathcal{L}_{\text{EmoSim}}$

where $\lambda$ allows us to control the trade-off between most similar
emotion mapping (discrete) and overall emotion distribution mapping
(continuous). The value of $\lambda$ was set to 0.5 when emotion
regularization is applied.

##  3 Experimental Design

###  3.1 Speech Datasets

We validate the effectiveness of the proposed method on different types of
speech datasets including acted speech (IEMOCAP), lexically-matched speech
(RAVDESS), and wake-up word speech (HI,KIA). IEMOCAP [21] contains
approximately 12 hours of speech from 10 speakers and is divided into five
sessions. Our experiment is based on 6 emotional classes. We use the first
four sessions for training and the fifth session for the testing. We use 10%
of the provided training data as a validation set. RAVDESS [22] is a multi-
modal database of emotional speech and song. It features 24 different actors
(12 males and 12 females) enacting 2 statements with 8 different emotions. We
use the first 20 actors for training, actors 20-22 for validation, and actors
22-24 for test. Hi,KIA [23] is a short-length wake-up word dataset labeled
with four emotional states including anger, happy, sad, and neutral. It
consists of 488 Korean accent utterances from 8 speakers. We use one of the
official 8-fold splits, which shows the highest human validation performance
(F7).

###  3.2 Music Datasets

We use AudioSet [24] that includes human-labeled 10-second sound clips drawn
from YouTube videos. We use a subset under “music mood” ontology which
contains music excerpts labeled with one of 7 music mood classes: happy,
funny, sad, tender, exciting, angry, and scary. As some of the items are not
available anymore, we actually use a smaller number of items than the official
dataset includes. Because AudioSet music mood lacks neutral labels, we treat
randomly generated color noise as a positive pair of neutral speech. The
smaller dataset is provided with a training set of 18,643 clips (13,295 music,
5,348 noise) and an evaluation set of 405 clips (345 music, 60 noise).

Modality | AS Music Mood | IEMOCAP | RAVDESS | Hi,KIA
---|---|---|---|---
Text | - | 0.709 | 0.133 | 0.250
Audio | 0.605 | 0.718 | 0.733 | 0.935

Table 1: Classification accuracy of music and speech emotion recognition
models

|  |  |  |  IEMOCAP [21] |  |  RAVDESS [22] |  |  HIKIA [23]
---|---|---|---|---|---|---|---|---
Method | Modality | Used In |  | MRR | P@5 | NDCG@5 |  | MRR | P@5 | NDCG@5 |  | MRR | P@5 | NDCG@5
Triplet | Text | [5] |  | 0.75±0.03 | 0.68±0.03 | 0.87±0.02 |  | 0.22±0.07 | 0.11±0.05 | 0.48±0.03 |  | 0.22±0.11 | 0.17±0.08 | 0.51±0.05
Triplet + SP | Text | [5] |  | 0.76±0.01 | 0.68±0.02 | 0.88±0.01 |  | 0.29±0.03 | 0.16±0.06 | 0.54±0.06 |  | 0.23±0.14 | 0.16±0.1 | 0.52±0.09
Triplet + EmoSim | Text | Ours |  | 0.76±0.01 | 0.69±0.02 | 0.88±0.01 |  | 0.28±0.1 | 0.18±0.05 | 0.62±0.06 |  | 0.22±0.11 | 0.1±0.03 | 0.46±0.02
Triplet | Audio | Ours |  | 0.74±0.02 | 0.67±0.04 | 0.86±0.03 |  | 0.72±0.03 | 0.65±0.03 | 0.85±0.03 |  | 0.84±0.09 | 0.73±0.06 | 0.91±0.03
Triplet + SP | Audio | Ours |  | 0.73±0.03 | 0.65±0.02 | 0.86±0.02 |  | 0.75±0.04 | 0.65±0.05 | 0.87±0.02 |  | 0.83±0.10 | 0.73±0.10 | 0.88±0.06
Triplet + EmoSim | Audio | Ours |  | 0.76±0.03 | 0.68±0.03 | 0.88±0.02 |  | 0.75±0.03 | 0.67±0.02 | 0.88±0.01 |  | 0.9±0.04 | 0.79±0.04 | 0.93±0.02
Triplet | Fusion | Ours |  | 0.80±0.03 | 0.73±0.05 | 0.89±0.02 |  | 0.76±0.02 | 0.65±0.02 | 0.86±0.01 |  | 0.87±0.06 | 0.74±0.10 | 0.92±0.04
Triplet + SP | Fusion | Ours |  | 0.82±0.01 | 0.74±0.02 | 0.90±0.02 |  | 0.73±0.07 | 0.66±0.02 | 0.87±0.01 |  | 0.82±0.16 | 0.75±0.07 | 0.91±0.08
Triplet + EmoSim | Fusion | Ours |  | 0.83±0.01 | 0.75±0.02 | 0.90±0.01 |  | 0.74±0.04 | 0.63±0.05 | 0.86±0.02 |  | 0.82±0.07 | 0.73±0.04 | 0.92±0.02

Table 2: Speech-to-music retrieval results on three different speech datasets.
SP stands for structure preserving, and EmoSim stands for emotion similarity
regularization.

###  3.3 Evaluation

We use accuracy to evaluate the performance of the classification. For
retrieval model evaluation, following the previous work [5], we use two
evaluation metrics: Precision at 5 (P@5) and Mean Reciprocal Rank (MRR). The
relevant score is a binary value measured by whether query speech and target
music share the same emotion semantic (with Figure 3). However, this metric
has two problems in that the rank order is not considered, and it does not
reflect the continuous emotion similarity. For example, tender is more similar
to sad than angry. To solve the above problem, we use normalizing Discounted
Cumulative Gain at 5 (nDCG@5), which was designed for ranking tasks with more
than one relevance level [25]. We define the prediction score as the
similarity between speech and music item; and ground truth as the valence-
arousal (VA) similarity of the item’s emotion label.

###  3.4 Training Details

The input to the music encoder is a 10-second audio excerpt at a 22,050 Hz
sampling rate. It is converted to a log-scaled mel spectrogram with 128 mel
bins, 1024-point FFT with a hann window, and a 512 hop size. The inputs of the
speech audio encoder and text encoder are 16,000 Hz waveforms up to 16 seconds
and tokenized text with a max length of 64, respectively. In the case of the
cross-domain retrieval model, all models are optimized using AdamW and use a
64-batch size. The models are trained with a learning rate of 1e-4 and a
margin $\delta$ of 0.4.

##  4 Results

###  4.1 Quantitative Results

Table 1 compares the baseline performance on classification tasks to evaluate
feature extraction performance. As somewhat obviously, the audio modality
works better on the lexically matched speech (‘RAVDESS’, ‘Hi,KIA’). The
impressive part is that the audio modality shows high performance even in the
IEMOCAP dataset with linguistic information. This shows that the audio
modality is sufficient for emotion recognition without a speech-to-text model.

Table 2 compares the different retrieval models with text, audio, fusion
speech modalities. Each cross-domain retrieval model was trained 5 times with
a different random seed and their average scores and standard deviations are
reported. In the case of the fusion method, we utilize inter-modality dynamics
through late fusion. 333In the preliminary experiments, we found that the late
fusion shows better performance than the early fusion When comparing single
modalities, the audio modality shows high performance in lexically matched
speech datasets (‘RAVDESS’, ‘Hi,KIA’). In IEMOCAP with linguistic information,
the text modality demonstrates comparable results to the audio modality. A
possible reason might be the availability of high emotional clues in the
linguistic structure. The fusion model shows a clear performance improvement
over other single modalities.

When comparing each method, our proposed emotion similarity (EmoSim)
regularization method shows high performance and low standard deviation in all
three datasets and various modalities. In particular, in the audio modality of
the three datasets, triplet + EmoSim method shows high NDCG@5 performance.
This is because, in the failure cases, the model retrieves music items as
close as possible to the correct answers.

###  4.2 Qualitative Results

Metric learning embedding spaces are projected to a 2D space using uniform
manifold approximation and projection (UMAP) [26]. We fit UMAP with music and
speech embeddings, then projected them into the 2D space. Figure 4 is the
visualization of the multi-modal embedding space. Due to space constraints, we
only visualize four models. All embedding vectors are computed from the test
set.

First, we observe that all models successfully discriminate emotion semantics.
In addition, we can see a continuous distribution of emotions leading to
happy$\xrightarrow{}$excited$\xrightarrow{}$tender$\xrightarrow{}$sad in all
models. However, in the case of the triplet model, scary music embeddings are
located between sad and happy speech embedding cluster (Figure 4 (a), (b)).
According to figure 3, in valence arousal space, scary emotion is similar to
frustration and angry emotion. This problem is alleviated in Triplet + EmoSim
model. There are relatively few scary music samples closer to angry and
frustration clusters (Figure 4 (c), (d)). We believe that joint embedding
space learned inter-modality neighborhood structure from the continuous
emotion similarity.

Fig. 4: UMAP visualization of speech-music joint embedding space. Star markers
stand for speech embeddings and circle markers stand for music embedding.

##  5 Conclusions

We presented a speech-to-music cross-domain retrieval framework that finds
music that matches the emotion of speech. We explored various speech modality
representations and proposed emotion similarity regularization term in cross-
domain retrieval task. The experimental results demonstrated that, in the
absence of linguistic information (lexically matched speech), the audio
modality can retrieve music that matches the speaker’s emotion. Especially,
our proposed regularization term helps the joint embedding space understand
the continuity of emotions. In the future, our approach can be extended to
include other modalities (e.g., vision) and improved with better hierarchical
fusion methods.

## References

  * [1] Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen,  “Adversarial cross-modal retrieval,”  in Proceedings of the ACM international conference on Multimedia, 2017.
  * [2] Sungeun Hong, Woobin Im, and Hyun Seung Yang,  “Cbvmr: Content-based video-music retrieval using soft intra-modal structure constraint,”  in Proceedings of the ACM on International Conference on Multimedia Retrieval, 2018.
  * [3] Bochen Li and Aparna Kumar,  “Query by video: Cross-modal music retrieval.,”  in Proceedings of International Society for Music Information Retrieval, ISMIR, 2019.
  * [4] Dídac Surís, Carl Vondrick, Bryan Russell, and Justin Salamon,  “It’s time for artistic correspondence in music and video,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10564–10574.
  * [5] Minz. Won, Justin. Salamon, Nicholas J. Bryan, Gautham J. Mysore, and Xavier. Serra,  “Emotion embedding spaces for matching music to stories,”  in Proceedings of International Society for Music Information Retrieval, ISMIR, 2021.
  * [6] Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, and Liang Wang,  “A comprehensive survey on cross-modal retrieval,”  arXiv preprint arXiv:1607.06215, 2016.
  * [7] Yu Gong, Greg Mori, and Frederick Tung,  “RankSim: Ranking similarity regularization for deep imbalanced regression,”  in International Conference on Machine Learning (ICML), 2022.
  * [8] Keunwoo Choi, György Fazekas, Mark Sandler, and Kyunghyun Cho,  “Transfer learning for music classification and regression tasks,”  in Proceedings of International Society for Music Information Retrieval, ISMIR, 2017.
  * [9] Rodrigo Castellon, Chris Donahue, and Percy Liang,  “Codified audio language modeling learns useful representations for music information retrieval,”  in Proceedings of International Society for Music Information Retrieval Conference, ISMIR, 2021.
  * [10] Li-Wei Chen and Alexander Rudnicky,  “Exploring wav2vec 2.0 fine-tuning for improved speech emotion recognition,”  arXiv preprint arXiv:2110.06309, 2021.
  * [11] Xingyu Cai, Jiahong Yuan, Renjie Zheng, Liang Huang, and Kenneth Church,  “Speech emotion recognition with multi-task learning,”  in Proceedings of Interspeech, 2021.
  * [12] Leonardo Pepino, Pablo Riera, and Luciana Ferrer,  “Emotion recognition from speech using wav2vec 2.0 embeddings,”  in Proceedings of Interspeech, 2021.
  * [13] Acheampong Francisca Adoma, Nunoo-Mensah Henry, and Wenyu Chen,  “Comparative analyses of bert, roberta, distilbert, and xlnet for text-based emotion recognition,”  in 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing ,ICCWAMTIP, 2020.
  * [14] Minz Won, Keunwoo Choi, and Xavier Serra,  “Semi-supervised music tagging transformer,”  in Proceedings of International Society for Music Information Retrieval, ISMIR, 2021.
  * [15] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in Neural Information Processing Systems, NeurIPS, 2020\.
  * [16] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf,  “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,”  Neural Information Processing Systems Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019.
  * [17] Saif Mohammad,  “Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words,”  in Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL, 2018.
  * [18] Liwei Wang, Yin Li, and Svetlana Lazebnik,  “Learning deep structure-preserving image-text embeddings,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR, 2016.
  * [19] Keunhyoung Kim, Jongpil Lee, Sangeun Kum, and Juhan Nam,  “Learning a cross-domain embedding space of vocal and mixed audio with a structure-preserving triplet loss,”  in Proceedings of International Society for Music Information Retrieval Conference, ISMIR, 2021.
  * [20] Jeffrey Pennington, Richard Socher, and Christopher D Manning,  “Glove: Global vectors for word representation,”  in Proceedings of the conference on empirical methods in natural language processing, EMNLP, 2014.
  * [21] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan,  “Iemocap: Interactive emotional dyadic motion capture database,”  Language resources and evaluation, 2008.
  * [22] Steven R Livingstone and Frank A Russo,  “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,”  PloS one, 2018.
  * [23] Taesu Kim, SeungHeon Doh, Gyunpyo Lee, Hyung seok Jun, Juhan Nam, and Hyeon-Jeong Suk,  “Hi, kia: A speech emotion recognition dataset for wake-up words,”  in Proceedings of the 14th Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2022.
  * [24] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in Processings of international conference on acoustics, speech and signal processing, IEEE, ICASSP, 2017.
  * [25] Kalervo Järvelin and Jaana Kekäläinen,  “Cumulated gain-based evaluation of ir techniques,”  ACM Transactions on Information Systems (TOIS), 2002.
  * [26] Leland McInnes, John Healy, and James Melville,  “Umap: Uniform manifold approximation and projection for dimension reduction,”  arXiv preprint arXiv:1802.03426, 2018.