# Recursive Joint Attention for Audio-Visual Fusion
in Regression based Emotion Recognition

###### Abstract

In video-based emotion recognition (ER), it is important to effectively
leverage the complementary relationship among audio (A) and visual (V)
modalities, while retaining the intra-modal characteristics of individual
modalities. In this paper, a recursive joint attention model is proposed along
with long short-term memory (LSTM) modules for the fusion of vocal and facial
expressions in regression-based ER. Specifically, we investigated the
possibility of exploiting the complementary nature of A and V modalities using
a joint cross-attention model in a recursive fashion with LSTMs to capture the
intra-modal temporal dependencies within the same modalities as well as among
the A-V feature representations. By integrating LSTMs with recursive joint
cross-attention, our model can efficiently leverage both intra- and inter-
modal relationships for the fusion of A and V modalities. The results of
extensive experiments111The code is available on GitHub:
https://github.com/praveena2j/RecurrentJointAttentionwithLSTMs. performed on
the challenging Affwild2 and Fatigue (private) datasets indicate that the
proposed A-V fusion model can significantly outperform state-of-art-methods.

Index Terms—  Emotion Recognition, Audio-Visual Fusion, Attention Mechanisms,
Long Short-Term Memory.

##  1 Introduction

Automatic emotion recognition (ER) is a challenging problem due to the complex
and extremely diverse nature of expressions across individuals and cultures.
In most of the real-world applications, emotions are exhibited over a wide
range of emotional states besides the six basic categorical expressions -
anger, disgust, fear, happy, sad, and surprise. For instance, emotional states
can be expressed as intensities of fatigue, stress, and pain over discrete
levels. Similarly, the wide range of continuous emotional states are often
formulated as dimensional ER, where the diverse and complex human emotions are
represented along the dimensions of valence and arousal. Valence denotes the
range of continuous emotional states pertinent to pleasantness, spanning from
being very sad (negative) to very happy (positive). Similarly, arousal spans
the range of emotional states related to intensity, from being very passive
(sleepiness) to extremely active (high excitement). In this paper, we have
focused on developing a robust model for regression-based ER in valence-
arousal space, as well as for fatigue.

A and V modalities often carry complementary relationships among themselves,
which is crucial to be exploited in order to build an efficient A-V fusion
system for regression-based ER. In addition to the inter-modal relationships
across A and V modalities, temporal dynamics in videos carry significant
information pertinent to the evolution of facial and vocal expressions over
time. Therefore, effectively leveraging both the inter-modal association
across the A and V modalities and temporal dynamics (intra-modal) within A and
V modalities plays a major role in building a robust A-V recognition system.
In this paper, we have investigated the prospect of leveraging these inter-
and intra-modal characteristics of A and V modalities in a unified framework.
In most of the existing approaches for regression-based ER, LSTMs has been
used in order to model the intra-modal temporal dynamics in videos [1, 2] due
to their efficiency in capturing the long-term temporal dynamics [3]. On the
other hand, cross-attention models [4] have been explored to model the inter-
modal characteristics of A and V modalities for dimensional ER.

In this work, we have proposed a unified framework for A-V fusion, which
effectively leverages both the intra- and inter-modal information in videos
using LSTMs and joint cross attention respectively. In order to further
improve the A-V feature representations of the joint cross-attention model, we
have also explored the recursive attention mechanism. Training the joint
cross-attention model recursively allows refining the A and V feature
representations, thereby improving the system performance. The main
contributions of the paper are as follows. (1) A recursive joint cross-
attentional model for A-V fusion is introduced to effectively exploit the
complementary relationship across modalities while deploying a recursive
mechanism to further refine the A-V feature representations. (2) LSTMs are
further integrated to effectively capture the temporal dynamics within the
individual modalities, as well as within the A-V feature representations. (3)
An extensive set of experiments are conducted on the challenging Affwild2 and
Fatigue (private) datasets, showing that the proposed A-V fusion model
outperforms the related state-of-the-art models for regression-based ER.

##  2 Related Work

An early DL approach for A-V fusion-based dimensional ER was proposed by
Tzirakis et al. [5], where the deep features (obtained with Resnet-50 for V
and 1D CNN for A) are concatenated and fed to an LSTM. Recently, Vincent et
al. [3] investigated the effectiveness of attention models and compared them
with recurrent networks. They have shown that LSTMs are quite efficient in
capturing the temporal dependencies when compared to attention models for
dimensional ER. Kuhnke et al. [2] proposed a two-stream A-V network, where
deep models are used to extract A and V features, and further concatenated for
dimensional ER. Most of these approaches fail to effectively capture the
intermodal semantics across A and V modalities. In [6] and [7], authors
focused on cross-modal attention using transformers to exploit the inter-modal
relationships of A and V modalities for dimensional ER. Rajasekhar et al [4]
explored cross-attention models to leverage the inter-modal characteristics
based on cross-correlation across the A and V features. They improved their
approach by introducing joint feature representation into the cross-attention
model to retain the intra-modal characteristics [8, 9]. In most of these
approaches, they cannot effectively leverage intra-modal relationships. Chen
et al. [10] modeled A and V features using LSTMs, and the unimodal predictions
are combined using attention weights from conditional attention based on
LSTMs. Priyasad et al. [11] also explored LSTMs for V features, and used DNN-
based attention on the concatenated features of A and V modalities for the
final output predictions. Beard et al. [12] proposed a recursive recurrent
attention model, where LSTMs are augmented using an additional shared memory
state in order to capture the multi-modal relationships in a recursive manner.
In contrast with these approaches, we focus on modeling the A-V relationships
by allowing the A and V features to interact and measure the semantic
relevance across and within the modalities in a recursive fashion before
feature concatenation. LSTMs are employed for temporal modeling of both uni-
modal and multimodal features to further enhance the proposed framework.
Therefore, our proposed model effectively leverages the intra- and
complementary inter-modal relationships, resulting in a higher level of
performance.

Fig. 1: Block diagram of the proposed recursive joint attention model with
BLSTMs.

##  3 Proposed Approach

A) Problem Formulation: Given an input video sub-sequence $S$, $L$ non-
overlapping video clips are uniformly sampled and deep feature vectors
${\boldsymbol{X}}_{\mathbf{a}}$ and ${\boldsymbol{X}}_{\mathbf{v}}$ are
extracted for the individual A and V modalities respectively from pre-trained
networks. Let
${\boldsymbol{X}}_{\mathbf{a}}=\\{\boldsymbol{x}_{\mathbf{a}}^{1},\boldsymbol{x}_{\mathbf{a}}^{2},...,\boldsymbol{x}_{\mathbf{a}}^{L}\\}\in\mathbb{R}^{d_{a}\times
L}$ and
${\boldsymbol{X}}_{\mathbf{v}}=\\{\boldsymbol{x}_{\mathbf{v}}^{1},\boldsymbol{x}_{\mathbf{v}}^{2},...,\boldsymbol{x}_{\mathbf{v}}^{L}\\}\in\mathbb{R}^{d_{v}\times
L}$ where ${d_{a}}$ and ${d_{v}}$ represent the dimensions of the A and V
feature representations, respectively, and $\boldsymbol{x}_{\mathbf{a}}^{l}$
and $\boldsymbol{x}_{\mathbf{v}}^{l}$ denotes the A and V feature vectors of
the video clips, respectively, for $l=1,2,...,L$ clips. The objective of the
problem is to estimate the regression model
$F:\boldsymbol{X}\to\boldsymbol{Y}$ from the training data $\boldsymbol{X}$,
where $\boldsymbol{X}$ denotes the set of A and V feature vectors of the input
video clips and $\boldsymbol{Y}$ represents the regression labels of the
corresponding video clips.

B) Audio and Visual Networks: Spectrograms has been found to be promising with
various 2D-CNNs (Resnet-18 [13]) for ER [14, 15]. Therefore, we have explored
spectrograms in the proposed framework. In order to effectively leverage the
temporal dynamics within A modality, we have also explored LSTMs across the
temporal segments of the A sequences. Finally, the A feature vectors of $L$
video clips are shown as
$\boldsymbol{X_{a}}=({\boldsymbol{x}_{\mathbf{a}}^{1},\boldsymbol{x}_{\mathbf{a}}^{2},...,\boldsymbol{x}_{\mathbf{a}}^{L}})\in\mathbb{R}^{{d_{a}}\times
L}$.

Facial expressions exhibit significant information pertinent to both visual
appearance and temporal dynamics in videos. LSTMs are found to be efficient in
capturing the long-term temporal dynamics while 3DCNNs are effective in
capturing the short-term temporal dynamics [16]. Therefore, we have used LSTMs
with 3D CNNs (R3D [17]) to obtain the V features for the fusion model. In most
of the existing approaches, the output of the last convolution layer is 512 x
7 x 7, which is further passed through a pooling operation in order to reduce
the spatial dimensions to 1 (7 $\to$ 1). This reduction in spatial dimension
was found to leave out useful information as the stride is big. Therefore,
inspired by the idea of [18], we use the A feature representation to smoothly
reduce the spatial dimensions of raw V features for each video clip similar to
that of [18]. Finally, we obtain a matrix of V feature vectors of the video
clips as
$\boldsymbol{X_{v}}=({\boldsymbol{x}_{\mathbf{v}}^{1},\boldsymbol{x}_{\mathbf{v}}^{2},...,\boldsymbol{x}_{\mathbf{v}}^{L}})\in\mathbb{R}^{{d_{v}}\times
L}$.

C) Recursive Joint Attention Model: Given the A and V features,
$\boldsymbol{X_{a}}$ and $\boldsymbol{X_{v}}$, the joint feature
representation is obtained by concatenating the A and V feature vectors
${\boldsymbol{J}}=[{\boldsymbol{X}}_{\mathbf{a}};{\boldsymbol{X}}_{\mathbf{v}}]\in\mathbb{R}^{d\times
L}$, where $d={d_{a}}+{d_{v}}$ denotes the dimensionality of concatenated
features. The concatenated A-V feature representations ($\boldsymbol{J}$) of a
video sub-sequence ($\boldsymbol{S}$) are now used to attend to unimodal
feature representations ${\boldsymbol{X}}_{\mathbf{a}}$ and
${\boldsymbol{X}}_{\mathbf{v}}$. The joint correlation matrix
$\boldsymbol{C}_{\mathbf{a}}$ across the A features
${\boldsymbol{X}}_{\mathbf{a}}$, and the combined A-V features
$\boldsymbol{J}$ are given by:

$\boldsymbol{C}_{\mathbf{a}}=\tanh\left(\frac{{\boldsymbol{X}}_{\mathbf{a}}^{T}{\boldsymbol{W}}_{\mathbf{j}\mathbf{a}}{\boldsymbol{J}}}{\sqrt{d}}\right)$
(1)

where ${\boldsymbol{W}}_{\mathbf{j}\mathbf{a}}\in\mathbb{R}^{L\times L}$
represents learnable weight matrix across the A and combined A-V features, and
$T$ denotes transpose operation. Similarly, the joint correlation matrix for V
features are given by:

$\boldsymbol{C}_{\mathbf{v}}=\tanh\left(\frac{{\boldsymbol{X}}_{\mathbf{v}}^{T}{\boldsymbol{W}}_{\mathbf{j}\mathbf{v}}{\boldsymbol{J}}}{\sqrt{d}}\right)$
(2)

The joint correlation matrices capture the semantic relevance across the A and
V modalities as well as within the same modalities among consecutive video
clips, which helps in effectively leveraging intra- and inter-modal
relationships. After computing the joint correlation matrices, the attention
weights of the A and V modalities are estimated. For the A modality, the joint
correlation matrix $\boldsymbol{C}_{\mathbf{a}}$ and the corresponding A
features ${\boldsymbol{X}}_{\mathbf{a}}$ are combined using the learnable
weight matrices $\boldsymbol{W}_{\mathbf{c}\mathbf{a}}$ to compute the
attention weights of A modality, which is given by
$\boldsymbol{H}_{\mathbf{a}}=ReLU(\boldsymbol{X}_{\mathbf{a}}\boldsymbol{W}_{\mathbf{c}\mathbf{a}}{\boldsymbol{C}}_{\mathbf{a}})$
where
${\boldsymbol{W}}_{\mathbf{c}\mathbf{a}}\in\mathbb{R}^{{d_{a}}\times{d_{a}}}$
and ${\boldsymbol{H}}_{\mathbf{a}}$ represents the attention maps of the A
modality. Similarly, the attention maps ($\boldsymbol{H}_{\mathbf{v}}$) of V
modality are obtained as
$\boldsymbol{H}_{\mathbf{v}}=ReLU(\boldsymbol{X}_{\mathbf{v}}\boldsymbol{W}_{\mathbf{c}\mathbf{v}}{\boldsymbol{C}}_{\mathbf{v}})$
where
${\boldsymbol{W}}_{\mathbf{c}\mathbf{v}}\in\mathbb{R}^{{d_{v}}\times{d_{v}}}$.
Then, the attention maps are used to compute the attended features of A and V
modalities as:

${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{a}}=\boldsymbol{H}_{\mathbf{a}}\boldsymbol{W}_{\mathbf{h}\mathbf{a}}+\boldsymbol{X}_{\mathbf{a}}$
(3)
${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{v}}=\boldsymbol{H}_{\mathbf{v}}\boldsymbol{W}_{\mathbf{h}\mathbf{v}}+\boldsymbol{X}_{\mathbf{v}}$
(4)

where $\boldsymbol{W}_{\mathbf{h}\mathbf{a}}\in\mathbb{R}^{d\times{d_{a}}}$
and $\boldsymbol{W}_{\mathbf{h}\mathbf{v}}\in\mathbb{R}^{d\times{d_{v}}}$
denote the learnable weight matrices for A and V respectively. After obtaining
the attended features they are fed again to the joint cross-attentional model
to compute the new A and V feature representations as:

${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{a}}^{(t)}=\boldsymbol{H}_{\mathbf{a}}^{(t)}\boldsymbol{W}_{\mathbf{h}\mathbf{a}}^{(t)}+\boldsymbol{X}_{\mathbf{a}}^{(t-1)}$
(5)
${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{v}}^{(t)}=\boldsymbol{H}_{\mathbf{v}}^{(t)}\boldsymbol{W}_{\mathbf{h}\mathbf{v}}^{(t)}+\boldsymbol{X}_{\mathbf{v}}^{(t-1)}$
(6)

where
$\boldsymbol{W}_{\mathbf{h}\mathbf{a}}^{(t)}\in\mathbb{R}^{d\times{d_{a}}}$
and
$\boldsymbol{W}_{\mathbf{h}\mathbf{v}}^{(t)}\in\mathbb{R}^{d\times{d_{v}}}$
denote the learnable weight matrices of $t^{th}$ iteration for A and V
respectively. Finally, the attended A and V features after $t$ iterations are
further concatenated and fed to BLSTM to obtain the temporal dependencies
within the refined A-V feature representations, which are fed to fully
connected layers for final prediction.

##  4 Results and Discussion

A) Datasets: Affwild2 is among the largest public dataset in affective
computing, consisting of $564$ videos collected from YouTube, all captured in-
the-wild [19]. The annotations are provided by four experts using a joystick
and the final annotations are obtained as the average of the four raters. In
total, there are $2,816,832$ frames with $455$ subjects, out of which $277$
are male and $178$ female. The annotations for valence and arousal are
provided continuously in the range of $[-1,1]$. The dataset is partitioned in
a subject-independent manner so that every subject’s data will be present in
only one subset. The partitioning produces 341, 71, and 152 videos for the
training, validation, and test sets respectively.

The Fatigue dataset (private) is obtained from $18$ participants in a
Rehabilitation center, suffering from degenerative diseases inducing fatigue.
A total of 27 video sessions are captured with a duration of 40 - 45 minutes
and labeled at sequence level on a scale of 0 to 10 for every 10 to 15
minutes. We have considered $80\%$ of data as training data (50,845 samples)
and $20\%$ as validation data (21,792 samples).

B) Ablation Study: Table 1 presents the results of the experiments conducted
on the validation set for the ablation study. The performance of the approach
is evaluated using Concordance Correlation Coefficient (CCC). In this section,
we have analyzed the contribution of BLSTMs, where we have performed
experiments with and without BLSTMs. Firstly, we have conducted experiments
without using BLSTM for both the individual A and V representations as well as
the A-V feature representations. Then, we included BLSTMs only for the
individual A and V modalities before feeding to the joint attention fusion
model i.e., only Unimodal-BLSTMs (U-BLSTMs). By including U-BLSTMs to capture
the temporal dependencies within the individual modalities, we can observe
improvements in performance. Therefore, BLSTMs are found to be promising in
capturing the intra-modal temporal dynamics better than that of correlation-
based intra-modeling in the joint attention model. After that, we have also
included joint BLSTM (J-BLSTM) in order to capture the temporal dynamics
across the joint A-V feature representations, which further improved the
performance of the system. Note that in the above experiments, we have not
performed recursive attention. In addition to the impact of U-BLSTM and
J-BLSTMs, we have also conducted a few more experiments to investigate the
impact of the recursive behavior of the joint attention model. First, we did
recursion without LSTMs and found some improvement due to recursion. Then we
included LSTMs and conducted several experiments by varying the number of
recursions (iterations) in the fusion model. As we increase the number of
recursive times, the model performance increases and starts to decrease after
a certain recursion number. A similar trend of the model performance is also
observed in the test set. Therefore, this can be attributed to the fact that
recursion also works as a regularizer which improves the generalization
ability of the model. In our experiments, we found that $t=2$ gives the best
performance i.e, we have achieved the best performance of our model with two
recursive iterations.

Table 1: Performance of our approach with components of BLSTM and recursive attention on Affwild2 dataset. Method | Valence | Arousal
---|---|---
JA Fusion w/o Recursion
Fusion w/o U-BLSTM | 0.670 | 0.590
Fusion w/o J-BLSTM | 0.691 | 0.646
Fusion w/ U-BLSTM and J-BLSTM | 0.715 | 0.688
JA Fusion w/ Recursion
JA Fusion w/o BLSTMs, $t=2\ $ | 0.703 | 0.623
JA Fusion with BLSTMs, $t=2$ | 0.721 | 0.694
JA Fusion with BLSTMs, $t=3$ | 0.706 | 0.652
JA Fusion with BLSTMs, $t=4$ | 0.685 | 0.601

C) Comparison to State-of-art: Table 2 shows our results against relevant
state-of-the-art A-V fusion models on the Affwild2 dataset. The Affwild2
dataset has recently been widely used for Affective Behavior Analysis in-the-
Wild (ABAW) challenges [20, 21]. Therefore, we compare our approach with
relevant state-of-art approaches in the ABAW challenges. Kuhnke et al [2] used
a simple feature concatenation using Resnet-18 for A and R3D for V modalities,
showing better performance for arousal than valence. Zhang et al [22] proposed
a leader-follower attention model for fusion, and improved the performance
(arousal) of the model proposed by Kuhnke et al [2]. Rajasekhar et al. [4]
explored the cross-attention model by leveraging only the inter-modal
relationships of A and V, and showed improvement for valence but not for
arousal. Rajasekhar et al. [8] further improved the performance of the model
by introducing joint feature representation to the cross-attention model. The
proposed model performs even better than that of vanilla JCA [8] by
introducing LSTMs as well as a recursive attention mechanism.

Table 2:  CCC performance of the proposed and state-of-the-art methods for A-V fusion on the Affwild2 dataset. Method | Type of Fusion | Valence | Arousal
---|---|---|---
Validation Set
Kuhnke et al. [2] | Feature Concatenation | 0.493 | 0.613
Zhang et al. [22] | Leader Follower Attention | 0.469 | 0.649
Rajasekhar et al [4] | Cross Attention | 0.541 | 0.517
Rajasekhar et al. [8] | Joint Cross Attention | 0.657 | 0.580
Ours | LSTM + Transformers | 0.628 | 0.654
Ours | Recursive JA + BLSTM | 0.721 | 0.694
Test Set
Meng et al. [23] | LSTM + Transformers | 0.606 | 0.596
Vincent et al. [3] | LSTM + Transformers | 0.418 | 0.407
Rajasekhar et al [8] | Joint Cross Attention | 0.451 | 0.389
Ours | Recursive JA + BLSTM | 0.467 | 0.405

For test set results, the winner of the latest ABAW challenge [23] has shown
improvement using A-V fusion, however using three external datasets and
multiple backbones. We have also compared the performance of our A and V
backbones with the ensembling of LSTMs and transformers [23] on the validation
set. Vincent [3] used LSTMs to capture intra-modal dependencies and explored
transformers for cross-modal attention, however, they fail to effectively
capture the inter-modal relationships across the consecutive video clips.
Rajasekhar et al [8] further improved the performance (valence) using joint
cross-attention. The proposed model outperforms both [3] and [8].

Table 3 shows the performance of the proposed approach on the Fatigue dataset.
We have shown the performance of individual modalities along with feature
concatenation and cross-attention [4]. The proposed approach outperforms
cross-attention [4] and baseline feature concatenation.

Table 3:  CCC performance on the Fatigue dataset. Method | Fatigue Level
---|---
Audio only (2D-CNN: Resnet18) | 0.312
Visual only (3D-CNN: R3D) | 0.415
Feature Concatenation | 0.378
Cross Attention [4] | 0.421
Recursive JA + BLSTM (Ours) | 0.447

##  5 Conclusion

This paper introduces a recursive joint attention model along with BLSTMs that
allows effective spatiotemporal A-V fusion for regression-based ER. In
particular, the joint attention model is trained in a recursive fashion,
allowing to refine the A-V features. We further investigated the impact of
BLSTMs for capturing the intra-modal temporal dynamics of individual A and V
modalities, as well as A-V features for regression-based ER. By effectively
capturing the intra-modal relationships using BLSTMs, and inter-modal
relationships using recursive joint attention, the proposed approach is able
to outperform the related state-of-the-art approaches.

## References

  * [1] Liam Schoneveld, Alice Othmani, and Hazem Abdelkawy,  “Leveraging recent advances in deep learning for audio-visual emotion recognition,”  Pattern Recognition Letters, vol. 146, 2021.
  * [2] Felix Kuhnke, Lars Rumberg, and Jörn Ostermann,  “Two-stream aural-visual affect analysis in the wild,”  in FG Workshop, 2020.
  * [3] Vincent Karas, Mani Kumar Tellamekala, Adria Mallol-Ragolta, Michel Valstar, and Björn W. Schuller,  “Time-continuous audiovisual fusion with recurrence vs attention for in-the-wild affect recognition,”  in CVPRW, 2022.
  * [4] Gnana Praveen Rajasekhar, Eric Granger, and Patrick Cardinal,  “Cross attentional audio-visual fusion for dimensional emotion recognition,”  in FG, 2021.
  * [5] Panagiotis Tzirakis, George Trigeorgis, Mihalis A. Nicolaou, Björn W. Schuller, and Stefanos Zafeiriou,  “End-to-end multimodal emotion recognition using deep neural networks,”  IEEE J. of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1301–1309, 2017.
  * [6] Panagiotis Tzirakis, Jiaxin Chen, Stefanos Zafeiriou, and Björn Schuller,  “End-to-end multimodal affect recognition in real-world environments,”  Information Fusion, vol. 68, 2021.
  * [7] Srinivas Parthasarathy and Shiva Sundaram,  “Detecting expressions with multimodal transformers,”  in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 636–643.
  * [8] R Gnana Praveen, Patrick Cardinal, and Eric Granger,  “Audio-visual fusion for emotion recognition in the valence-arousal space using joint cross-attention,”  IEEE Transactions on Biometrics, Behavior, and Identity Science, 2023.
  * [9] R Gnana Praveen, Wheidima Carneiro de Melo, Nasib Ullah, Haseeb Aslam, Osama Zeeshan, Théo Denorme, Marco Pedersoli, Alessandro L. Koerich, Simon Bacon, Patrick Cardinal, and Eric Granger,  “A joint cross-attention model for audio-visual fusion in dimensional emotion recognition,”  in CVPRW, 2022.
  * [10] S Chen and Q Jin,  “Multi-modal conditional attention fusion for dimensional emotion prediction,”  in Proc. of ACM’M, 2016.
  * [11] Priyasad Darshana, F Tharindu, S Simon, and F Clinton,  “Learning salient features for multimodal emotion recognition with recurrent neural networks and attention based fusion,”  in Proc. of AVSP, 2019.
  * [12] R Beard, R Das, R. W. M. Ng, P. G. K Gopalakrishnan, L Eerens, P Swietojanski, and O Miksik,  “Multi-modal sequence fusion via recursive attention for emotion recognition,”  in CoNLL, 2018.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in CVPR, 2016, pp. 770–778.
  * [14] Anwer Slimi, Mohamed Hamroun, Mounir Zrigui, and Henri Nicolas,  “Emotion recognition from speech using spectrograms and shallow neural networks,”  in ICAMCM, 2020, p. 35–39.
  * [15] Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman,  “Emotion recognition in speech using cross-modal transfer in the wild,”  in 26th ACMM, 2018, p. 292–301.
  * [16] Yin Fan, Xiangju Lu, Dian Li, and Yuanliu Liu,  “Video-based emotion recognition using cnn-rnn and c3d hybrid networks,”  in ACM ICMI, 2016, p. 445–450.
  * [17] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri,  “A closer look at spatiotemporal convolutions for action recognition,”  in CVPR, 2018, pp. 6450–6459.
  * [18] Bin Duan, Hao Tang, Wei Wang, Ziliang Zong, Guowei Yang, and Yan Yan,  “Audio-visual event localization via recursive fusion by joint co-attention,”  in WACV, 2021.
  * [19] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A. Nicolaou, Athanasios Papaioannou, Guoying Zhao, Bjorn Schuller, Irene Kotsia, and Stefanos Zafeiriou,  “Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond,”  IJCV, vol. 127, pp. 907–929, 2019.
  * [20] D Kollias, A Schulc, E Hajiyev, and S Zafeiriou,  “Analysing affective behavior in the first abaw competition,”  in FG 2020, 2020.
  * [21] Dimitrios Kollias and Stefanos Zafeiriou,  “Analysing affective behavior in the second abaw2 competition,”  in ICCVW, 2021, pp. 3652–3660.
  * [22] Su Zhang, Yi Ding, Ziquan Wei, and Cuntai Guan,  “Continuous emotion recognition with audio-visual leader-follower attentive fusion,”  in ICCVW, 2021.
  * [23] Liyu Meng, Yuchen Liu, Xiaolong Liu, Zhaopei Huang, Wenqiang Jiang, Tenggan Zhang, Chuanhe Liu, and Qin Jin,  “Valence and arousal estimation based on multimodal temporal-aware features for videos in the wild,”  in CVPRW, 2022, pp. 2344–2351.


# Self-Supervised Knowledge Transfer via Loosely Supervised Auxiliary Tasks

Seungbum Hong, Jihun Yoon, and Min-Kook Choi
VisionAI, hutom
Seoul, Republic of Korea
{qbration21, jhyoon2020<EMAIL_ADDRESS>Junmo Kim
KAIST
Daejeon, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

Knowledge transfer using convolutional neural networks (CNNs) can help
efficiently train a CNN with fewer parameters or maximize the generalization
performance under limited supervision. To enable a more efficient transfer of
pretrained knowledge under relaxed conditions, we propose a simple yet
powerful knowledge transfer methodology without any restrictions regarding the
network structure or dataset used, namely self-supervised knowledge transfer
(SSKT), via loosely supervised auxiliary tasks. For this, we devise a training
methodology that transfers previously learned knowledge to the current
training process as an auxiliary task for the target task through self-
supervision using a soft label. The SSKT is independent of the network
structure and dataset, and is trained differently from existing knowledge
transfer methods; hence, it has an advantage in that the prior knowledge
acquired from various tasks can be naturally transferred during the training
process to the target task. Furthermore, it can improve the generalization
performance on most datasets through the proposed knowledge transfer between
different problem domains from multiple source networks. SSKT outperforms the
other transfer learning methods (KD, DML, and MAXL) through experiments under
various knowledge transfer settings. The source code will be made available to
the public111https://github.com/generation21/generation6011.

##  1 Introduction

Knowledge transfer is the most representative training methodology for
improving the generalization capability and training efficiency of
convolutional neural networks (CNNs). The most widely used knowledge transfer
is transfer learning [37, 35], which uses pretrained weights trained on large-
scale datasets as initial values for new tasks. Pretrained weights have been
used as feature encoders after fine-tuning in different vision tasks such as
image classification, object detection, and semantic segmentation [24, 25,
29]. Taskonomy [38, 39] provides an extensive database and analysis approach
for studying the effects of transfer learning using pretrained networks on the
fine-tuning of target tasks. However, transfer learning using pretrained
networks presupposes structural dependencies that must share essentially the
same (whole or partial) network structure as the source network. Moreover,
questions have been raised on whether transfer learning using pretrained
weights can provide appropriate initial weights, which can help in improving
convergence or training with less amount of data but not very effective for
knowledge transfer to different tasks [12].

Figure 1: Motivation of the SSKT. The CNN responses that can be obtained from
the same image vary depending on the type of supervision. SSKT performs
auxiliary training using soft labels to convey the previously trained prior
knowledge from the source tasks. At this time, the supervision used in the
auxiliary training generates a gradient (grey and purple arrows) that can
improve the generalization performance of the target task. The gradient to
update the weight of the target network is obtained as a linear combination of
all losses (red arrow). TM is a transfer module.

Another representative example of knowledge transfer using CNN is knowledge
distillation (KD) [14], where pretrained teacher networks distill and deliver
dark (hidden) knowledge in the inference output or learned features during
student network training. KD methods include loss-based KD, which delivers
dark knowledge through loss with a soft label [14], and KD, wherein the
similarity between the features extracted are exploited in specific stages of
the CNN [30]. Knowledge transfer through KD has the advantage of training
student networks that have a better-generalized performance with fewer
parameters comparable to teacher networks. In the general KD setup, the
training datasets of both teacher and student networks should be the same.
However, for a more general knowledge transfer, these constraints must be
relaxed.

Figure 2: Schematic of the difference between the proposed SSKT and other
knowledge transfer methods. Unlike existing knowledge transfer methods, such
as KD and DML, SSKT is independent of the network structure and dataset. In
addition, generalization performance improvements can be achieved without
additional supervision during the training process.

Figure 1 shows an example in which the information to be transferred can vary
depending on the type of supervision of each pretrained CNN through Grad-CAM
visualization [32] for the same input image. For this purpose, we propose a
simple yet powerful method that aims for better generalization with minimal
additional supervision. The proposed knowledge transfer conveys prior
knowledge as a soft label from multiple pretrained networks obtained using
different types of supervision methods in a self-supervised manner. The
proposed knowledge transfer method utilizes a pretrained network but is
independent of the structure of the target network and the training dataset of
the pretrained network to train the target task. The target network is updated
through the linear combination of the target loss and multiple auxiliary
losses with soft label-based self-supervision from the pretrained networks by
trying to guess the type of knowledge inferred from the source network (Figure
1). We expect that the target network reflects dark knowledge as a gradient
from multiple supervisions of existing training data through soft label-based
self-supervision. Because it does not refer to previously trained data and
performs auxiliary training in a self-supervised manner, the proposed
knowledge transfer can be performed within any task while requiring only the
final output dimension (number of classes) for calculating each auxiliary
loss. We call this form of knowledge transfer self-supervised knowledge
transfer (SSTK). The technical contributions of the SSKT are as follows:

  * •

Unlike previous knowledge transfer techniques, SSKT does not depend on the
structure of the source network or the training dataset. SSKT is even possible
between structurally complete heterogeneous networks, such as 2D to 3D CNN
knowledge transfer.

  * •

SSKT has the potential for simultaneous knowledge transfer from multiple
pretrained networks trained on completely different types of datasets or
supervision.

  * •

The SSKT improves the generalization performance in most knowledge transfer
scenarios. Particularly in different problem domains, SSKT outperforms
existing knowledge transfer methods.

Figure 2 shows the differences between SSKT and other knowledge transfer
methods. For clarity, we take the problem domain as a superset of tasks (e.g.,
image classification, action recognition, object detection, and semantic
segmentation), and the task is bounded in the same problem domain (e.g.,
ImageNet, Places365, and STL10 in image classification).

##  2 Related Works

Transfer learning with a pretrained initializer. As the most commonly used
method of knowledge transfer, transfer learning with a pretrained initializer
has been applied to various fields of computer vision using high-performance
models [20, 33, 13] with a large amount of training data. Because low-level
feature information is shared by most visual recognition problems, fine-tuning
is often applied by updating only a specific upper-layer when training new
tasks [25, 29]. Generally, this can be used in the same network structure, and
many image recognition applications use only the feature encoder part of the
entire task architecture [24, 25, 29]. Although transfer learning using
ImageNet has been successful so far, questions on transfer learning using
ImageNet data have been raised [12], and the experimental results of object
recognition and key point recognition problems have been reported as examples
where the pretrained initial values do not convey additional knowledge to the
different problem domains.

Knowledge distillation. Pretrained teacher networks with larger parameters can
improve the generalization performance of student networks by performing
knowledge transfer during the training process to a relatively small number of
parameters [14]. Student networks often show better generalization performance
over teacher networks [8]. Recent knowledge distillation techniques for
effective knowledge transfer used the relationship between the inferred
posteriors from the teacher and student networks [27] or the dataset as a
subclass of the original classes to be learned [26]. In [8], the ensemble
model achieved a high recognition performance by distilling the knowledge of
student networks with different initialization values using teacher and
student networks with the same structure. SSKT uses a knowledge transfer
method, such as KD with a soft label. However, unlike existing KD methods,
because the SSKT transfers knowledge through an auxiliary task-based self-
supervision technique, there is no dataset dependency.

Deep mutual learning (DML). DML is one of the representative variations of the
KD-based knowledge transfer method that uses multiple networks for efficient
training through KD loss within networks [40]. DML could achieve performance
improvement through mutual learning in the middle of training without
requiring a powerful pretrained teacher network. Recently, a learning
methodology using feature fusion [18], generative adversarial networks [3],
and collaborative learning [36] structures has been proposed to improve the
efficiency of mutual learning during training. It differs from SSKT in that
the DML performs multitask training-based learning on the same dataset and
updates all the networks during training. In addition, it is fundamentally
different from SSKT in that DML does not utilize prerequisite knowledge from
former training.

Auxiliary learning. Unlike general multi-task learning, in the case of
auxiliary task learning, the purpose of a deep neural network is not to
increase the performance of auxiliary tasks but to improve the performance of
the target task. It was recently reported that auxiliary task learning could
help improve the test performance of the target task, and the cosine
similarity between the gradient losses of the target and auxiliary tasks was
analyzed to visualize the cause of contribution by the auxiliary task [6]. The
basic structure of the SSKT was inspired by meta-learning techniques using
auxiliary tasks under self-supervision (MAXL) [23]. However, MAXL requires a
hierarchical auxiliary class structure defined manually for the primary class
to be trained. SSKT was able to achieve higher performance than MAXL by using
only the number of classes of source tasks as additional supervision
information.

Self-supervised learning (SSL). Typical self-supervised learning techniques
train pretrained weights that perform unsupervised training on pretext tasks
to obtain good initial weights [9, 2, 1, 11]. The pretrained network utilized
in SSKT was not obtained by unsupervised training, and the target task
training differed from the general self-supervised approach, requiring meta
information such as the number of classes used in the source network training.
However, SSKT can still be considered self-supervised from the viewpoint of
training auxiliary tasks using only soft labels from pretrained networks,
without requiring additional supervision with hard labels when training the
auxiliary task.

Domain adaptation or expansion. Domain adaptation or extension [22, 16] aims
to improve the generalization performance for new domains with existing
trained networks. However, unlike in the case of the general domain adaptation
problem, SSKT has a significant difference in using the prior knowledge
trained in the existing domain for new network training without sharing
parameters. In addition, because it does not include any update process for
the pretrained network, the update is performed regardless of the performance
of the source domain. Moreover, there was no change in the inference
performance of the source domain after training.

##  3 Self-Supervised Knowledge Transfer

SSKT supports a training structure that enables knowledge transfer in a
variety of scenarios using a CNN. Various scenarios refer to situations not
influenced by the network architecture of the target task to be trained and in
which the knowledge transfer does not depend on the type of task. We devised a
structure that transfers knowledge naturally without compromising the training
information of the pretrained network or requiring additional supervision
during the target task training process. We achieved this using soft label-
based knowledge transfer techniques with auxiliary task learning through self-
supervision for the various domains of image recognition variants.

###  3.1 SSKT with Single Source

We define a multi-task network for auxiliary learning,
$h_{t}(x;\theta_{t},D_{t},T_{t})$, where $x$ is the input, $\theta_{t}$ is a
parameter of the target network, $D_{t}$ is a target dataset, and $T_{t}$ is
the task to be trained. $\theta_{t}$ is updated simultaneously through the
target loss and auxiliary loss during training to solve the primary task.
$h_{s}(x;\theta_{s},D_{s},T_{s})$ describes a source network that receives
input $x$ and delivers knowledge to the target network. $\theta_{s}$ denotes a
parameter trained by the source task $T_{s}$ for the source dataset (ImageNet
and Places365) $D_{s}$. $\theta_{s}$ is not updated during target task
training. The multi-task network $h_{t}$ for the primary task training shares
up to the top layer of the convolutional block below the primary and auxiliary
branches, except for the last feature layer. The last feature layer of the
branch for the target task loss outputs
$h_{t}^{prim}(x;\theta_{t},D_{t},T_{t})$, and the branch for the auxiliary
task loss outputs $h_{t}^{aux}(x;\theta_{t},D_{t},T_{s})$. At this time, the
auxiliary task branch could have a transfer module composed of the summation
of the bottleneck structure from each convolutional block for effective
feature encoding for the auxiliary task. Figure 3 shows an example of a
multitask network with a transfer module. The ground truth label of the
primary task for the total loss of $h_{t}$ is given by $y_{t}^{prim}$, for $x$
belonging to $D_{t}$. The ground truth label for the auxiliary task is given
by the softmax output $y_{s}^{aux}$ from $h_{s}$.

The total loss function for a single-task knowledge transfer based on a
multitask network is as follows.

$\displaystyle\operatorname*{argmin}_{\theta_{t}}\Big{(}L(h_{t}^{prim}(x_{i};\theta_{t}.D_{t},T_{t}),y_{t,i}^{prim})$
$\displaystyle+\alpha
L(h_{t}^{aux}(x_{i};\theta_{t}.D_{t},T_{s}),y_{s,i}^{aux})\Big{)},$ (1)

where $i$ is the $i^{th}$ batch of the training data, $\alpha$ is a balanced
parameter for the total loss, and
$y_{s,i}^{aux}=h_{s}(x_{i};\theta_{s},D_{t},T_{s})$ is the softmax outputted
from the pretrained source network and which conveys the dark knowledge of the
pretrained dataset using soft labels. We applied two types of losses of cross-
entropy (CE) and knowledge distillation (KD) [14] for auxiliary task learning:

$\displaystyle L_{CE}=-ylog(c(\frac{g(x;\theta)}{T})),$ (2) $\displaystyle
L_{KD}=KL\Big{(}c(\frac{g(x;\theta_{s})}{T}),c(\frac{g(x;\theta_{t})}{T})\Big{)},$
(3)

where $y$ is a one-hot vector from the ground truth label, $g(x;\theta)$ is
the final feature outputted from the network during training, $T$ controls the
softening intensity for the softmax output as the temperature parameter, $c$
denotes the softmax function, and $KL$ denotes the KL divergence between the
output distributions. The gradient update after $k+1$ iterations is described
by

$\displaystyle\theta_{t}^{k+1}=\theta_{t}^{k}-\eta\nabla_{\theta_{t}}\Big{(}L(h_{t}^{prim}(x_{i};\theta_{t}.D_{t},T_{t}),y_{t,i}^{prim})$
$\displaystyle+\alpha
L(h_{t}^{aux}(x_{i};\theta_{t}.D_{t},T_{s}),y_{s,i}^{aux})\Big{)}$ (4)

where $\eta$ is the learning rate. Figure 2(a) shows the structural schematics
for the SSKT with a single-source scenario.

Transfer module (TM). To encourage the prediction of $y_{s,i}^{aux}$ by
$h_{t}$, we design a bottleneck structure-based transfer module that supports
an auxiliary task using the feature outputted from each convolutional block.
For an example of a ResNet-based model, each output of the $res$ block is
encoded as a fixed-size output by bottleneck and average pooling, and finally,
each feature outputted from the transfer bottleneck is summed (see Figure 3).

Figure 3: Schematic of transfer modules for efficient auxiliary learning. The
transfer module used in the SSKT consists of the summation of the feature
outputs of the bottleneck layers from each convolutional block. The schematic
shows an example of the transfer module with ResNet variants for SSKT using
multiple sources.

###  3.2 SSKT with Multiple Source

SSKT can be easily extended to multiple $h_{s}$ using multiple auxiliary
losses. We improved the structure by including $M$ source tasks for knowledge
transfer to target tasks, as follows:

$\displaystyle\operatorname*{argmin}_{\theta_{t}}\Big{(}L(h_{t}^{prim}(x_{i};\theta_{t}.D_{t},T_{t}),y_{t,i}^{prim})$
$\displaystyle+\alpha(L(h_{t}^{aux}(x_{i};\theta_{t}.D_{t},T_{s_{1}}),y_{s_{1},i}^{aux})$
$\displaystyle+L(h_{t}^{aux}(x_{i};\theta_{t}.D_{t},T_{s_{2}}),y_{s_{2},i}^{aux})+\dots$
$\displaystyle+L(h_{t}^{aux}(x_{i};\theta_{t}.D_{t},T_{s_{M}}),y_{s_{M},i}^{aux}))\Big{)},$
(5)

where $s_{M}$ denotes the index of the source task for knowledge transfer.
Theoretically, if the memory is sufficient, we can perform multiple task
knowledge transfers from a large number of $M$ source tasks.

Table 1: Training details for each task, for knowledge transfer with SSKT. IC
denotes image classification, AC denotes action classification, MCIC denotes
multi-class image classification, $\beta$ represents momentum, and $\lambda$
represents weight decay. CE denotes cross-entropy, KD denotes knowledge
distillation, and BCE denotes binary cross-entropy. In the training model, (s)
was trained by learning from scratch, and (f) was trained by fine-tuning.

Task ($T$) | Dataset ($D$) | # cls | Model | Optimizer | LR Scheduler |  $\beta$,$\lambda$ | Loss [Task + Auxiliary]
---|---|---|---|---|---|---|---
IC ($T_{t}$) | CIFAR10 (C10) | 10 | ResNet[20, 32] | SGD | step (0.1,$[$150:250:350$]$) | 0.9, 5e-4 | [CE + CE or KD]
| CIFAR100 (C100) | 100 | ResNet[20, 32] | SGD | step (0.1,$[$60:120:160:200$]$) | 0.9, 5e-4 | [CE + CE or KD]
| STL10 (S10) | 10 | ResNet18[20, 32, 44] | SGD | step (0.1,$[$60:120:160:200$]$) | 0.9, 5e-4 | [CE + CE or KD]
|  |  | MoblieNetV2 | SGD | step (0.1,$[$60:120:160:200$]$) | 0.9, 5e-4 | [CE + CE or KD]
|  |  | DenseNet121 | SGD | step (0.1,$[$60:120:160:200$]$) | 0.9, 5e-4 | [CE + CE or KD]
| Places365 (P) | 365 | ResNet18 | SGD | step (0.1,$[$30:60:90$]$) | 0.9, 1e-4 | [CE + CE or KD]
| ImageNet (I) | 1000 | ResNet18 | SGD | step (0.1,$[$30:60:90$]$) | 0.9, 1e-4 | [CE + CE or KD]
MCIC ($T_{t}$) | Pascal VOC (VOC) | 20 | ResNet[18, 34, 50] (s) | SGD | step (0.1,$[$30:60:90$]$) | 0.9, 1e-4 | [BCE + CE or KD]
|  |  | ResNet18 (f) | SGD | step (0.01,$[$30:60:90$]$) | 0.9, 1e-4 | [BCE + CE or KD]
AC ($T_{t}$) | UCF101 (U101) | 101 | 3D-ResNet18 (s) | SGD | reduce on plateau (0.1) | 0.9, 1e-3 | [CE + CE or KD]
|  |  | 3D-ResNet18 (f) | SGD | reduce on plateau (0.01) | 0.9, 1e-3 | [CE + CE or KD]
| HMDB51 (H51) | 51 | 3D-ResNet18 (s) | SGD | reduce on plateau (0.1) | 0.9, 1e-3 | [CE + CE or KD]
|  |  | 3D-ResNet18 (f) | SGD | reduce on plateau (0.01) | 0.9, 1e-3 | [CE + CE or KD]
IC ($T_{s}$) | Places365 (P) | 365 | ResNet50 | [41] | [41] | [41] | [41]
| ImageNet (I) | 1000 | ResNet50 | [28] | [28] | [28] | [28]

###  3.3 SSKT within Different Problem Domains

Until now, transfers for single and multiple sources have been performed
assuming tasks in the same problem domain that do not specify the forms of
$D_{t}$ and $D_{s}$. We set up a scenario to enable knowledge transfer even
when the modalities of $D_{t}$ and $D_{s}$ are different or when the problem
domains of the tasks are different. In this case, because the characteristics
of the input data are different and there is a large difference in the
structure of the network for training, the preparation process for knowledge
transfer involves data conversion or preprocessing. For knowledge transfer
with different problem domains, the total loss function can be defined as:

$\displaystyle\operatorname*{argmin}_{\theta_{t}}\Big{(}L(h_{t}^{prim}(x_{i};\theta_{t}.D_{t},T_{t}),y_{t,i}^{prim})$
$\displaystyle+\alpha
L(h_{t}^{aux}(f_{s}(x_{i});\theta_{t}.D_{t},T_{s}),y_{s,i}^{aux})\Big{)},$ (6)

where the data transformation function $f_{s}$ converts the data type to match
the source task and infer the recognition information to the task of the
source domain. For example, if $T_{t}$ is an action recognition problem using
a 3D-CNN, the input $x^{w\times h\times d}\in D_{t}$ is defined as a 3D
tensor. In this case, if a pretrained network for knowledge transfer is
obtained through the image recognition problem $T_{s}$ using 2D-CNN,
$f_{s}:x^{w\times h\times d}\to\hat{x}^{w\times h}$ should be defined as a
function that maps a 3D tensor to a 2D matrix into which $h_{s}$ can be
inputted.

Knowledge transfer to other problem domains can also be done with multiple
($M$) source tasks:

$\displaystyle\operatorname*{argmin}_{\theta_{t}}\Big{(}L(h_{t}^{prim}(x_{i};\theta_{t}.D_{t},T_{t}),y_{t,i}^{prim})$
$\displaystyle+\alpha(L(h_{t}^{aux}(f_{s_{1}}(x_{i});\theta_{t}.D_{t},T_{s_{1}}),y_{s_{1},i}^{aux})$
$\displaystyle+L(h_{t}^{aux}(f_{s_{2}}(x_{i});\theta_{t}.D_{t},T_{s_{2}}),y_{s_{2},i}^{aux})+\dots$
$\displaystyle+L(h_{t}^{aux}(f_{s_{M}}(x_{i});\theta_{t}.D_{t},T_{s_{M}}),y_{s_{M},i}^{aux}))\Big{)},$
(7)

in this case, up to $M$ transformation functions can be defined. Gradient
updates for knowledge transfer to other problem domains are defined in the
same way as single- or multiple-task knowledge transfers between the same
domains.

##  4 Experimental Results

To verify the SSKT, we designed a series of experimental scenarios. First, we
set a knowledge transfer with a single-source task in the same problem domain.
A pretrained CNN on image classification tasks (ImageNet and Places365) was
used for knowledge transfer for the same problem domain. Second, we performed
knowledge transfer for the image classification problem using multiple source
networks. In this case, the target network for target task learning had two or
more auxiliary tasks. Third, knowledge transfer scenarios using source
networks with different problem domains were tested in two ways. The first
method involved target task learning with a different purpose and knowledge
transfer to a domain within the same image recognition category. For example,
to solve the multiclass classification problem for 2D images, the target
network (2D-CNN) has the same type of architecture as the source network
(2D-CNN). In this case, knowledge transfer between domains is possible without
any special definition of the transformation function $f_{s}$ for the source
network. The second scenario involves knowledge transfer using different
problem domains using heterogeneous networks. For example, in training the
action recognition network using 3D-CNN, we verified whether the network
learned by the image classification problem could transfer the knowledge to
the action recognition problem.

In addition to verifying the knowledge transfer performance of SSKT for each
knowledge transfer scenario, the optimization results for the hyperparameters
of the loss functions were included. Moreover, our experiments include the
relationship between the change in knowledge transfer performance based on the
model architecture and fine tuning, and a performance comparison with other
knowledge transfer methods such as KD, DML, and MAXL. All the experimental
results included in the table are selected based on the highest performance
among combinations of auxiliary loss, transfer module, and source
network222Additional combinations of test results and architecture details can
be found in the supplementary materials.. Table 1 summarizes the experimental
settings for the verification of the SSKT.

###  4.1 Image Classification $\to$ Image Classification

SSKT Setting. To perform SSKT between image classification problems, we used
networks pretrained on ImageNet (I) [5] or Places365 (P) [41] dataset as the
source network. The target task datasets were CIFAR10, CIFAR100 [19], and
STL10 [4], which contain relatively small amounts of data, as well as large
datasets such as ImageNet and Places365. As a pretrained source network, we
used the ResNet50 network provided in the PyTorch torchvision package [28].
The transfer module is composed of a bottleneck and average pooling, as shown
in Figure 3. Multiple pretrained networks can be utilized in SSKT to apply for
multisource task-based knowledge transfer. We used both a ResNet50 model
trained on ImageNet and Places365 (P+I) datasets as our source tasks.

Results. Table 2 shows the performances of learning from scratch and SSKT for
the image classification problem on CIFAR10 (C10), 100 (C100), and STL10 (S10)
datasets. For knowledge transfer between image classification problems, the
use of SSKT improves the generalization performance on all the tested
datasets. The performance change when the transfer module is applied to
efficiently transform the features for auxiliary task learning is also
evident. Notably, the highest improvement in the generalization performance
was on the SLT10 dataset, which contains a small proportion of the training
data. This means that knowledge transfer using SSKT is more effective when the
amount of supervision is relatively low. Table 3 shows the performance changes
of the SSKT on large-scale image datasets. When SSKT was applied to both
Place365 and ImageNet, there was a steady performance improvement. In
particular, the highest performance improvement was obtained when the source
networks were used simultaneously (P + I). However, for large datasets, it was
difficult to confirm the effect of improving the performance of the transfer
module.

Table 2: Performance change of SSKT for a single source compared to training
from scratch. The best-performing model for each dataset is highlighted in
bold. All the experiments evaluated the test performance thrice from the same
random seed for the model. TM denotes transfer module, and R[depth] denotes
ResNet structure.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | C10 | R20 | scratch | - | CE | 92.19$\pm$0.09
P |  | R20 | SSKT | o | CE+KD | 92.25$\pm$0.04
I |  | R20 | SSKT | o | CE+CE | 92.44$\pm$0.05
P+I |  | R20 | SSKT | x | CE+KD | 92.46$\pm$0.15
- |  | R32 | scratch | - | CE | 93.21$\pm$0.09
P |  | R32 | SSKT | x | CE+KD | 92.87$\pm$0.31
I |  | R32 | SSKT | x | CE+CE | 93.26$\pm$0.08
P+I |  | R32 | SSKT | o | CE+CE | 93.38$\pm$0.02
- | C100 | R20 | scratch | - | CE | 68.26$\pm$0.36
P |  | R20 | SSKT | x | CE+KD | 68.01$\pm$0.42
I |  | R20 | SSKT | o | CE+CE | 68.63$\pm$0.12
P+I |  | R20 | SSKT | o | CE+CE | 68.56$\pm$0.23
- |  | R32 | scratch | - | CE | 70.33$\pm$0.19
P |  | R32 | SSKT | x | CE+CE | 69.97$\pm$0.16
I |  | R32 | SSKT | o | CE+CE | 70.75$\pm$0.06
P+I |  | R32 | SSKT | o | CE+CE | 70.94$\pm$0.36
- | S10 | R20 | scratch | - | CE | 81.15$\pm$0.34
P |  | R20 | SSKT | o | CE+CE | 82.76$\pm$0.05
I |  | R20 | SSKT | o | CE+CE | 83.45$\pm$0.07
P+I |  | R20 | SSKT | o | CE+CE | 84.56$\pm$0.35
- |  | R32 | scratch | - | CE | 81.19$\pm$0.17
P |  | R32 | SSKT | o | CE+CE | 83.06$\pm$0.27
I |  | R32 | SSKT | o | CE+CE | 83.68$\pm$0.28
P+I |  | R32 | SSKT | o | CE+CE | 83.4$\pm$0.2
- |  | R44 | scratch | - | CE | 80.18$\pm$0.54
P |  | R44 | SSKT | o | CE+CE | 82.68$\pm$0.39
I |  | R44 | SSKT | o | CE+CE | 83.59$\pm$0.13
P+I |  | R44 | SSKT | o | CE+CE | 83.44$\pm$0.15

Table 3: Performance change when applying the SSKT and training from scratch
on a large-scale image dataset.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | P | R18 | scratch | - | CE | 50.92
P |  | R18 | SSKT | o | CE+CE | 54.5
I |  | R18 | SSKT | o | CE+CE | 53.67
P+I |  | R18 | SSKT | x | CE+CE | 54.78
- | I | R18 | scratch | - | CE | 64.14
P |  | R18 | SSKT | o | CE+CE | 64.99
I |  | R18 | SSKT | x | CE+CE | 67.79
P+I |  | R18 | SSKT | x | CE+CE | 70.57

###  4.2 Image Classification $\to$ Multi-class Image Classification

SSKT Setting. To verify whether knowledge transfer using SSKT is effective
between different problem domains, knowledge transfer between image
classification and multiclass image classification was performed. We tested
multiclass image classification using the PASCAL VOC dataset [7] as the target
task. The target loss function for solving the multiclass image classification
was used as the binary cross-entropy (BCE). Table 1 lists the training details
for the target and source tasks.

Results. Table 4 shows the performance of learning from scratch for multiclass
image classification and the performance change when SSKT is included in the
training process. For the PASCAL VOC multi-class classification task, we used
the standard average precision (AP) as the accuracy measurement to evaluate
the predictions. The use of SSKT significantly improved the knowledge transfer
performance between different problem domains. The improvement was greater
than that for knowledge transfer between the same problem domains. In
addition, we verified an additional performance improvement when applying SSKT
to multi-source networks.

Table 4: Results of SSKT from the image classification problem to the multi-
class image classification problem.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | VOC | R18 | scratch | - | BCE | 67.28$\pm$0.25
P |  | R18 | SSKT | o | BCE+CE | 74.76$\pm$0.17
I |  | R18 | SSKT | x | BCE+CE | 74.78$\pm$0.09
P+I |  | R18 | SSKT | o | BCE+CE | 76.42$\pm$0.06
- |  | R34 | scratch | - | BCE | 66.0$\pm$0.49
P |  | R34 | SSKT | o | BCE+CE | 75.65$\pm$0.12
I |  | R34 | SSKT | o | BCE+CE | 75.14$\pm$0.14
P+I |  | R34 | SSKT | o | BCE+CE | 77.02$\pm$0.02
- |  | R50 | scratch | - | BCE | 61.16$\pm$0.34
P |  | R50 | SSKT | o | BCE+CE | 74.44$\pm$0.06
I |  | R50 | SSKT | o | BCE+CE | 74.24$\pm$0.05
P+I |  | R50 | SSKT | o | BCE+CE | 77.1$\pm$0.14

###  4.3 Image Classification $\to$ Action Classification

SSKT Setting. Although there is a difference between problem domains, the
performance improvement of knowledge transfer may be relatively predictable
because the modalities of the input data of the source and target tasks are
the same. We used a pretrained image classification network as the source task
and set the target task as an action recognition problem. We employed 3D-CNN
to design an experiment for a more extended knowledge transfer. To utilize the
dark knowledge of the source network as described in Equation (6), we need to
define a transformation function $f_{s}:x^{w\times h\times
d}\to\hat{x}^{w\times h}$. Because SSKT is not designed to define good
$f_{s}$, we simply limited the role of $f_{s}$ in extracting the center frame
from a 3D video clip. As a transfer module for inferring auxiliary tasks in a
3D-CNN, the output features of the target model were further processed using a
3D bottleneck structure. We used a 3D ResNet-based baseline model [10] to
train the action classification networks. Table 1 lists the training details
for 3D ResNet and source networks.

Results. Table 5 shows the performance of learning from scratch with the
3D-ResNet on the UCF101 [34] and HMDB51 [21] datasets, as well as the
performance change when SSKT was used in the training process. All the results
were obtained from split 1 for each dataset. Note that the auxiliary task in
SSKT prevents overfitting during the training of the 3D-CNN model. Similar to
other SSKT settings, the performance improvement was highest when using a
multi-source network. We further note that the use of SSKT between
heterogeneous problem domains can improve the performance further, depending
on the definition of $f_{s}$.

###  4.4 Further Analysis and Discussion

Parameter optimization. We performed a series of experiments for the
temperature parameter $T$ included in the auxiliary loss, the balance
parameter $\alpha$ included in the total loss, and the presence of the
transfer module. Figure 4 shows the graphs of the parameter optimization
results on the STL10 and PASCAL VOC datasets. For STL10, the use of a transfer
module in the SSKT process yielded a high performance in most cases; however,
in the case of PASCAL VOC, the transfer module did not play a significant
role.

Figure 4: Hyper parameter optimization of SSKT. The title of each graph is
composed of $D_{t}$ (target model, $T_{s}$, auxiliary loss). $T$ is the
temparture parameter of each auxiliary loss, and $\alpha$ is the balance
parameter of the total loss.

Model comparison. We applied SSKT to MobileNet V2 (MV2) [31] and DenseNet121
(D121) [15] to evaluate the model performance in CNN architectures other than
ResNet variants. The source network used the ResNet50 model, which was
previously trained on Places365 and ImageNet, similar to other SSKT settings.
In evaluating the STL10 dataset, Table 6 shows that MobileNet V2 and
DenseNet121 exhibit performance improvements with the SSKT. The results in the
table show the highest performance in the configuration among single-and
multi-source SSKT settings.

Table 5: Results of SSKT from image classification to action classification
with different modalities of the input data.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | U101 | 3DR18 | scratch | - | CE | 43.28
P |  | 3DR18 | SSKT | o | CE+CE | 45.35
I |  | 3DR18 | SSKT | x | CE+CE | 46.62
P+I |  | 3DR18 | SSKT | x | CE+CE | 52.19
- | H51 | 3DR18 | scratch | - | CE | 17.14
P |  | 3DR18 | SSKT | o | CE+CE | 18.77
I |  | 3DR18 | SSKT | o | CE+KD | 18.77
P+I |  | 3DR18 | SSKT | o | CE+CE | 20.54

Table 6: SSKT results using MobileNet V2 (MV2) and Dense-Net121 (D121).

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | S10 | R20 | scratch | - | CE | 81.15$\pm$0.34
- |  | MV2 | scratch | - | CE | 72.26$\pm$0.83
- |  | D121 | scratch | - | CE | 72.02$\pm$0.48
P+I |  | R20 | SSKT | o | CE+ CE | 84.56$\pm$0.35
P+I |  | MV2 | SSKT | o | CE+CE | 76.96$\pm$0.39
P+I |  | D121 | SSKT | x | CE+CE | 77.03$\pm$0.17

Relation to fine-tuning. In SSKT scenarios, fine-tuning and learning from
scratch can be applied for target network training. We adopted fine-tuning-
based SSKT with a ResNet18 model trained in ImageNet for a multiclass image
classification problem using the PASCAL VOC dataset. In addition, using the
pretrained 3D-ResNet18 model on the Kinetics-400 dataset [17], SSKT was
applied to the action classification problem for the UCF101 and HMDB51
datasets. Table 7 shows that even if fine-tuning based on predetermined
weights is performed, the performance is improved in all the experimental
settings. However, unlike the evaluation results in the case of training from
scratch, the performance improvement from multiple sources was not
significant, and the best performance was obtained when the KD loss was used.

Table 7: SSKT results using pretrained weights. ft denotes fine-tuning and K
denotes Kinetics-400 dataset.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | VOC | R18 |  ft (I) | - | CE | 90.52$\pm$0.11
P |  | R18 | SSKT | x | CE+KD | 92.28$\pm$0.06
I |  | R18 | SSKT | x | CE+KD | 92.26$\pm$0.07
P+I |  | R18 | SSKT | o | CE+KD | 92.25$\pm$0.07
- | U101 | 3DR18 |  ft (K) | - | CE | 83.95
P |  | 3DR18 | SSKT | x | CE+KD | 84.58
I |  | 3DR18 | SSKT | o | CE+KD | 84.37
P+I |  | 3DR18 | SSKT | o | CE+KD | 84.19
- | H51 | 3DR18 |  ft (K) | - | CE | 56.64
P |  | 3DR18 | SSKT | o | CE+KD | 57.82
I |  | 3DR18 | SSKT | o | CE+KD | 57.75
P+I |  | 3DR18 | SSKT | o | CE+CE | 57.29

Comparison with other knowledge transfer methods. For a further analysis of
SSKT, we compared its performance with those of typical knowledge transfer
methods, namely KD and DML. For KD, the details for learning were set the same
as in [14], and for DML, training was performed in the same way as in [40].
Table 8 shows the evaluation performance in terms of knowledge transfer. In
the case of 3D-CNN-based action classification, KD was not performed, and the
network used for DML had the model with the same conditions as the source and
target networks of SSKT. In the case of DML for 3D-CNN, training was performed
by replacing ResNet50 with 3D-ResNet50 under the same conditions. In the case
of action classification, both learning from scratch and fine-tuning results
were included. In most cases, under similar training conditions, SSKT
exhibited a higher generalization performance than other knowledge transfer
techniques. Moreover, training in problem domains other than the image
classification problem for DML was difficult. In particular, in the case of
multiclass classification, neither network was sufficiently trained. SSKT
achieved improved performance in all problem domains with multiple
experimental setups.

Table 9 shows the performance comparison with MAXL, another auxiliary
learning-based transfer learning method [23]. In addition to the target task’s
class label, MAXL requires a manually defined hierarchical auxiliary class
structure ($\psi[i]=2,3,5,10$). We compared the performance on the CIFAR10
dataset by applying the same four-stage hierarchical auxiliary class label as
defined in [23]. Despite not requiring additional class labels, SSKT showed
better recognition performance than MAXL in all settings. In particular, the
same learning scheduler in [23] showed better performance than MAXL
(comparisons with VGG16 in Table 9), and SSKT achieved a larger performance
gap when changing the source task and hyper-parameter setting.

Table 8: Comparison with other knowledge transfer methods. (s) denote learning
from scratch and (f) denote fine-tuning.

$T_{t}$ | Model | KD | DML | SSKT ($T_{s}$)
---|---|---|---|---
C10 | R20 | 91.75$\pm$0.24 | 92.37$\pm$0.15 |  92.46$\pm$0.15 (P+I)
| R32 | 92.61$\pm$0.31 | 93.26$\pm$0.21 |  93.38$\pm$0.02 (P+I)
C100 | R20 | 68.66$\pm$0.24 | 69.48$\pm$0.05 | 68.63$\pm$0.12 (I)
| R32 | 70.5$\pm$0.05 | 71.9$\pm$0.03 | 70.94$\pm$0.36 (P+I)
S10 | R20 | 77.67$\pm$1.41 | 78.23$\pm$1.23 |  84.56$\pm$0.35 (P+I)
| R32 | 76.07$\pm$0.67 | 77.14$\pm$1.64 |  83.68$\pm$0.28 (I)
VOC | R18 | 64.11$\pm$0.18 | 39.89$\pm$0.07 |  76.42$\pm$0.06 (P+I)
| R34 | 64.57$\pm$0.12 | 39.97$\pm$0.16 |  77.02$\pm$0.02 (P+I)
| R50 | 62.39$\pm$0.6 | 39.65$\pm$0.03 |  77.1$\pm$0.14 (P+I)
U101 | 3DR18 (s) | - | 13.8 |  52.19 (P+I)
| 3DR18 (f) | - | 83.95 |  84.58 (P)
H51 | 3DR18 (s) | - | 3.01 |  17.91 (P+I)
| 3DR18 (f) | - | 56.44 |  57.82 (P)

Table 9: Comparison with MAXL. V16 denotes VGG16 [33]. F denotes focal loss
[23]

$T_{t}$ | Model (Loss) | MAXL ($\psi[i]$) | SSKT ($T_{s}$) |  $T_{s}$ Model
---|---|---|---|---
C10 | V16 (F) | 93.27$\pm$0.09 (2) | 94.1$\pm$0.1 (I) | V16
| V16 (F) | 93.47$\pm$0.08 (3) | 92.72$\pm$0.15 (P) | R50
| V16 (F) | 93.49$\pm$0.05 (5) | 93.06$\pm$0.2 (P+I) | R50, V16
| V16 (F) | 93.10$\pm$0.08 (10) | 92.54$\pm$0.17(P+I) | R50, R50
| V16 (CE) | - |  94.22$\pm$0.02 (I) | V16
| V16 (CE) | - | 93.12$\pm$0.12 (P) | R50
| V16 (CE) | - | 93.67$\pm$0.17 (P+I) | R50, V16
| V16 (CE) | - | 93.26$\pm$0.12 (P+I) | R50, R50
| R20 (F) | 91.53$\pm$0.33 (2) | 91.48$\pm$0.03 (I) | V16
| R20 (F) | 91.52$\pm$0.1 (3) | 91.26$\pm$0.17 (P) | R50
| R20 (F) | 91.38$\pm$0.47 (5) | 90.93$\pm$0.01 (P+I) | R50, V16
| R20 (F) | 91.56$\pm$0.16 (10) | 91.11$\pm$0.18 (P+I) | R50, R50
| R20 (CE) | - | 92.44$\pm$0.05 (I) | R50
| R20 (CE) | - | 92.25$\pm$0.04 (P) | R50
| R20 (CE) | - |  92.46$\pm$0.15 (P+I) | R50, R50

##  5 Conclusions

We proposed a simple yet powerful knowledge transfer method, namely SSKT, that
enables knowledge transfer between heterogeneous networks and datasets. We
validated the SSKT using several knowledge transfer scenarios based on a CNN.
For further investigations of the SSKT, depending on the target task where
knowledge transfer takes place, additional considerations for the transfer
module design must be made. Moreover, the following experiments are required
to verify how the performance of the target model varies with the architecture
of the source model trained on the same dataset. Variations in the data
transformation function $f_{s}$ for the source networks may be another avenue
worth pursuing. Further, there is a need to perform analyses, such as the one
proposed in [6], to analyze the factors involved when the backpropagation of
the auxiliary task loss affects the performance of the target task. We expect
that further research will follow, as SSKT may provide a different perspective
on knowledge transfer.

Acknowledgement. This work was supported by the Korea Medical Device
Development Fund grant funded by the Korea government (the Ministry of Science
and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health
Welfare, the Ministry of Food and Drug Safety) (Project Number: 202012A02-02).

## References

  * [1] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  Big self-supervised models are strong semi-supervised learners.  In In Proc. of NeurIPS, 2020.
  * [2] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In In Proc. of ICML, 2020.
  * [3] I. Chung, S. Park, J. Kim, and N. Kwak.  Feature map-level online adversarial knowledge distillation.  In In Proc. of ICML, 2020.
  * [4] A. Coates, A. Ng, and H. Lee.  An analysis of single-layer networks in unsupervised feature learning.  In In Proc. of AISTAT, 2011.
  * [5] J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In In Proc. of CVPR, 2009.
  * [6] Y. Du, W. M. Czarnecki, S. M. Jayakumar, R. Pascanu, and B. Lakshminarayanan.  Adapting auxiliary losses using gradient similarity.  In In Proc. of NeurIPSW, 2019.
  * [7] M. Everingham, S. M. A. Eslami, L. V. Gool, C. K. I. Williams, Winn J, and A. Zisserman.  The pascal visual object classes challenge - a retrospective.  International Journal of Computer Vision, 111(1):98–136, 2015.
  * [8] T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar.  Born again neural networks.  In In Proc. of ICML, 2018.
  * [9] P. Goyal, D. Mahajan, A. Gupta, and I. Misra.  A simple framework for contrastive learning of visual representations.  In In Proc. of ICCV, 2019.
  * [10] K. Hara, H. Kataoka, and Y. Satoh.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In In Proc. of CVPR, 2018.
  * [11] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick.  Momentum contrast for unsupervised visual representation learning.  In In Proc. of CVPR, 2020.
  * [12] K. He, R. Girshick, and P. Dollár.  Rethinking imagenet pre-training.  In In Proc. of ICCV, 2019.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In In Proc. of CVPR, 2016.
  * [14] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  In In Proc. of NIPSW, 2014.
  * [15] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger.  Densely connected convolutional networks.  In In Proc. of CVPR, 2017.
  * [16] H. Jung, J. Ju, M. Jung, and J. Kim.  Less-forgetful learning for domain expansion in deep neural networks.  In In Proc. of AAAI, 2018.
  * [17] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman.  The kinetics human action video dataset.  In CoRR abs/1705.06950, 2017.
  * [18] J. Kim, M. Hyun, I. Chung, and N. Kwak.  Feature fusion for online mutual knowledge distillation.  In In Proc. of ICPR, 2020.
  * [19] A. Krizhevsky.  Learning multiple layers of features from tiny images.  In Technical Report, 2009.
  * [20] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In In Proc. of NIPS, 2012.
  * [21] H. Kuehne, H. Jhuang, E. Garrote, T. A. Poggio, and T. Serre.  Hmdb: a large video database for human motion recognition.  In In Proc. of ICCV, 2011.
  * [22] Z. Li and D. Hoiem.  Learning without forgetting.  In In Proc. of ECCV, 2016.
  * [23] S. Liu, A. J. Davison, and E. Johns.  Self-supervised generalisation with meta auxiliary learning.  In In Proc. of NeurIPS, 2019.
  * [24] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. C. Berg.  Ssd: Single shot multibox detector.  In In Proc. of ECCV, 2016.
  * [25] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In In Proc. of CVPR, 2015.
  * [26] R. Müller, S. Kornblith, and G. Hinton.  Subclass distillation.  In CoRR abs/2002.03936, 2020.
  * [27] W. Park, D. Kim, Y. Lu, and M. Cho.  Imagenet classification with deep convolutional neural networks.  In In Proc. of CVPR, 2019.
  * [28] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In In Proc. of NeurIPS, 2019.
  * [29] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In In Proc. of NIPS, 2015.
  * [30] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio.  Fitnets: Hints for thin deep nets.  In In Proc. of ICLR, 2015.
  * [31] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In In Proc. of CVPR, 2018.
  * [32] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In In Proc. of ICCV, 2017.
  * [33] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In In Proc. of ICLR, 2015.
  * [34] K. Soomro, A. R. Zamir, and M. Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  In CoRR abs/1212.0402, 2012.
  * [35] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu.  A survey on deep transfer learning.  In In Proc. of ICANN, 2018.
  * [36] G. Wu and S. Gong.  Peer collaborative learning for online knowledge distillation.  In In Proc. of CVPR, 2020.
  * [37] J. Yosinski, Clune J, Bengio Y, and Lipson H.  How transferable are features in deep neural networks?  In In Proc. of NIPS, 2014.
  * [38] A. Zamir, A. Sax, W. Shen, L. Guibas, J. Malik, and S. Savarese.  Taskonomy: Disentangling task transfer learning.  In In Proc. of CVPR, 2018.
  * [39] A. Zamir, A. Sax, T. Yeo, O. Kar, N. Cheerla, R. Suri, Z. Cao, J. Malik, and L. Guibas.  Robust learning through cross-task consistency.  In In Proc. of CVPR, 2020.
  * [40] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu.  Deep mutual learning.  In In Proc. of CVPR, 2018.
  * [41] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba.  Places: A 10 million image database for scene recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452–1464, 2018.

## Supplementary Materials

## A. SSKT with Multiple Tasks and Different Problem Domains

Self-supervised knowledge transfer (SSKT) supports a training procedure that
enables effective transfer learning in a variety of scenarios using deep
neural networks. SSKT has a structure that transfers pretrained knoweldge
naturally, without compromising the training information of the pretrained
network or requiring additional supervision in the target task training
process. We achieved this goal using the soft label-based knowledge transfer
techniques with auxiliary learning through self-supervision, for the various
domain of image recognition variants. Final formulation of the SSKT as
follows:

$\displaystyle\operatorname*{argmin}_{\theta_{t}}\Big{(}L(h_{t}^{prim}(x_{i};\theta_{t}.D_{t},T_{t}),y_{t,i}^{prim})$
$\displaystyle+\alpha(L(h_{t}^{aux}(f_{s_{1}}(x_{i});\theta_{t}.D_{t},T_{s_{1}}),y_{s_{1},i}^{aux})$
$\displaystyle+L(h_{t}^{aux}(f_{s_{2}}(x_{i});\theta_{t}.D_{t},T_{s_{2}}),y_{s_{2},i}^{aux})+\dots$
$\displaystyle+L(h_{t}^{aux}(f_{s_{M}}(x_{i});\theta_{t}.D_{t},T_{s_{M}}),y_{s_{M},i}^{aux}))\Big{)}.$
(8)

We define a multi-task network $h_{t}(x;\theta_{t},D_{t},T_{t})$, where $x$ is
the input, $\theta_{t}$ is a parameter of the target network, $D_{t}$ is a
target dataset, and $T_{t}$ is the task to be trained. $\theta_{t}$ is updated
simultaneously through target loss and auxiliary loss during training to solve
the primary task. $h_{s}(x;\theta_{s},D_{s},T_{s})$ denotes a source network
that receives the input $x$ and delivers knowledge to the target network.
$\theta_{s}$ denotes a parameter trained by the source task $T_{s}$ for the
source data set $D_{s}$. $\theta_{s}$ is not updated during the target task
training. $i$ is the $i^{th}$ batch of the training data, $\alpha$ is balanced
parameter for total loss, and
$y_{s,i}^{aux}=h_{s}(x_{i};\theta_{s},D_{s},T_{s})$ is the softmax output from
the pretrained source network and conveys the dark knowledge of the pretrained
dataset by soft labels. The data transformation function $f_{s}$ converts the
data type to match the source task to infer the recognition information to the
task of the source domain. For example, if $T_{t}$ is an action recognition
problem using 3D-CNN, the input $x^{w\times h\times d}\in D_{t}$ is defined as
a three-dimensional tensor. In this case, if a pretrained network for transfer
learning is obtained through the image recognition problem $T_{s}$ using
2D-CNN, $f_{s}:x^{w\times h\times d}\to\hat{x}^{w\times h}$ should be defined
as a function that maps a three-dimensional tensor to a two-dimensional matrix
into which $h_{s}$ can be input. Up to $M$ number of different type of
transformation functions could be defined. Algorithm1 describes how the SSKT
works depending on each transfer learning scenario.

Transfer Modules Depending on CNN Architecture. To encourage predicting
$y_{s,i}^{aux}$ by $h_{t}$, we design bottleneck structure based transfer
module supporting auxiliary task using feature output from each convolutional
block. Figure 1 shows configuration of transfer modules depending on each CNN
architecture for its problem domain. We applied the transfer module to four
different CNN architectures such as ResNet [13], DenseNet [15], MobileNet
[31], and 3D-ResNet [10] for each problem domain.

Figure 5: Schematic of the transfer modules for efficient SSKT with different
CNN architectures. The transfer module used in the SSKT consists of summation
of feature output of bottleneck layers from each convolutional block.
Schematic shows and example of the transfer module with different CNN
architectures for SSKT using multiple sources.

## B. Additional Experiment Results

We provide performance for all experimental configuraltions for each dataset,
in addition to the results contained in the main manuscript. For fair
comparison of SSKT, the configurations consist of a combination of the type of
source and target network, the presence or absence of a transfer module, and a
loss function. For model architecture and hyperparameters setting for training
(See Table 1 of the main manuscript). Same as the experiment results of the
main manuscript, the datasets of the source task are ImageNet [5] and
Places365 [41], and the datasets of the target task are CIFAR10/100 [19],
STL10 [4], ImageNet, Places365, PASCAL VOC [7], UCF101 [34], and HMDB51 [21].
Tables 1 to 6 provide performance according to the experimental conditions of
each dataset. Figure 2 shows the performance changes for the STL10 and PASCAL
VOC datasets depending on hyperparameters, the structure of the source and
target network, and the presence or absence of a transfer module. The
abbreviations for the datasets and model architectures listed in all
experimental tables are as follows:

Datasets: ImageNet (I), Places365 (P), CIFAR10 (C10), CIFAR100 (C100), STL10
(S10), PASCAL VOC (VOC), UCF101 (U101), and HMDB51 (H51).

Model architectures: ResNet (R), DenseNet (D), MobileNetV2 (MV2), and
3D-ResNet (3DR).

Finally, we included the experimental results according to the training
setting for further analysis of the SSKT. Table 7 shows the all the
combination of comparison results for DenseNet121 and MoblieNetV2, and Table 8
shows the evaluations for each experimental setting with finetuning scenario.
Table 9 shows the performance comparison with MAXL [23] which is the state-of-
the-art self-supervised learning based on auxiliary learning.

Table 10: Performance change according to the configuration of the SSKT for
CIFAR10 dataset compared to the training from scratch. All experiments
evaluated test performance 3 times from the same random seed for the model. TM
stands for Transfer Module and R[depth] stands for ResNet structure. The best
performance of each network architecture highlighted in bold.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | C10 | R20 | scratch | - | CE | 92.19$\pm$0.09
P |  | R20 | SSKT | x | CE+CE | 92.21$\pm$0.06
|  | R20 | SSKT | x | CE+KD | 92.24$\pm$0.14
|  | R20 | SSKT | o | CE+CE | 92.23$\pm$0.04
|  | R20 | SSKT | o | CE+KD | 92.25$\pm$0.04
I |  | R20 | SSKT | x | CE+CE | 92.28$\pm$0.07
|  | R20 | SSKT | x | CE+KD | 92.34$\pm$0.07
|  | R20 | SSKT | o | CE+CE | 92.44$\pm$0.05
|  | R20 | SSKT | o | CE+KD | 92.29$\pm$0.0
P+I |  | R20 | SSKT | x | CE+CE | 91.9$\pm$0.1
|  | R20 | SSKT | x | CE+KD | 92.46$\pm$0.15
|  | R20 | SSKT | o | CE+CE | 92.42$\pm$0.07
|  | R20 | SSKT | o | CE+KD | 92.22$\pm$0.17
- |  | R32 | scratch | - | CE | 93.21$\pm$0.09
P |  | R32 | SSKT | x | CE+CE | 92.77$\pm$0.14
|  | R32 | SSKT | x | CE+KD | 92.87$\pm$0.31
|  | R32 | SSKT | o | CE+CE | 92.65$\pm$0.26
|  | R32 | SSKT | o | CE+KD | 92.59$\pm$0.22
I |  | R32 | SSKT | x | CE+CE | 93.26$\pm$0.08
|  | R32 | SSKT | x | CE+KD | 92.78$\pm$0.2
|  | R32 | SSKT | o | CE+CE | 93.25$\pm$0.12
|  | R32 | SSKT | o | CE+KD | 92.88$\pm$0.07
P+I |  | R32 | SSKT | x | CE+CE | 92.88$\pm$0.15
|  | R32 | SSKT | x | CE+KD | 93.07$\pm$0.09
|  | R32 | SSKT | o | CE+CE | 93.38$\pm$0.02
|  | R32 | SSKT | o | CE+KD | 93.1$\pm$0.22

Table 11: Performance change according to the configuration of the SSKT for
CIFAR100 dataset compared to the training from scratch.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | C100 | R20 | scratch | - | CE | 68.26$\pm$0.36
P |  | R20 | SSKT | x | CE+CE | 67.65$\pm$0.21
|  | R20 | SSKT | x | CE+KD | 68.01$\pm$0.42
|  | R20 | SSKT | o | CE+CE | 67.96$\pm$0.27
|  | R20 | SSKT | o | CE+KD | 67.83$\pm$0.34
I |  | R20 | SSKT | x | CE+CE | 68.3$\pm$0.17
|  | R20 | SSKT | x | CE+KD | 68.37$\pm$0.23
|  | R20 | SSKT | o | CE+CE | 68.63$\pm$0.12
|  | R20 | SSKT | o | CE+KD | 68.35$\pm$0.1
P+I |  | R20 | SSKT | x | CE+CE | 67.87$\pm$0.17
|  | R20 | SSKT | x | CE+KD | 68.13$\pm$0.05
|  | R20 | SSKT | o | CE+CE | 68.56$\pm$0.23
|  | R20 | SSKT | o | CE+KD | 67.84$\pm$0.28
- |  | R32 | scratch | - | CE | 70.33$\pm$0.19
P |  | R32 | SSKT | x | CE+CE | 69.97$\pm$0.16
|  | R32 | SSKT | x | CE+KD | 69.93$\pm$0.21
|  | R32 | SSKT | o | CE+CE | 69.69$\pm$0.19
|  | R32 | SSKT | o | CE+KD | 69.92$\pm$0.31
I |  | R32 | SSKT | x | CE+CE | 70.6$\pm$0.05
|  | R32 | SSKT | x | CE+KD | 70.17$\pm$0.14
|  | R32 | SSKT | o | CE+CE | 70.75$\pm$0.06
|  | R32 | SSKT | o | CE+KD | 70.0$\pm$0.11
P+I |  | R32 | SSKT | x | CE+CE | 69.25$\pm$0.58
|  | R32 | SSKT | x | CE+KD | 69.22$\pm$0.43
|  | R32 | SSKT | o | CE+CE | 70.94$\pm$0.36
|  | R32 | SSKT | o | CE+KD | 69.44$\pm$0.01

Table 12: Performance change according to the configuration of the SSKT for
STL10 dataset compared to the training from scratch.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | STL10 | R20 | scratch | - | CE | 81.15$\pm$0.34
P |  | R20 | SSKT | x | CE+CE | 81.56$\pm$0.32
|  | R20 | SSKT | x | CE+KD | 80.88$\pm$0.19
|  | R20 | SSKT | o | CE+CE | 82.76$\pm$0.05
|  | R20 | SSKT | o | CE+KD | 81.06$\pm$0.2
I |  | R20 | SSKT | x | CE+CE | 82.2$\pm$0.17
|  | R20 | SSKT | x | CE+KD | 80.82$\pm$0.14
|  | R20 | SSKT | o | CE+CE | 83.45$\pm$0.07
|  | R20 | SSKT | o | CE+KD | 81.3$\pm$0.39
P+I |  | R20 | SSKT | x | CE+CE | 82.46$\pm$0.24
|  | R20 | SSKT | x | CE+KD | 81.47$\pm$0.22
|  | R20 | SSKT | o | CE+CE | 84.56$\pm$0.35
|  | R20 | SSKT | o | CE+KD | 81.33$\pm$0.11
- |  | R32 | scratch | - | CE | 81.19$\pm$0.17
P |  | R32 | SSKT | x | CE+CE | 82.1$\pm$0.14
|  | R32 | SSKT | x | CE+KD | 81.29$\pm$0.22
|  | R32 | SSKT | o | CE+CE | 83.06$\pm$0.27
|  | R32 | SSKT | o | CE+KD | 81.19$\pm$0.12
I |  | R32 | SSKT | x | CE+CE | 82.88$\pm$0.33
|  | R32 | SSKT | x | CE+KD | 81.4$\pm$0.23
|  | R32 | SSKT | o | CE+CE | 83.68$\pm$0.28
|  | R32 | SSKT | o | CE+KD | 81.76$\pm$0.18
P+I |  | R32 | SSKT | x | CE+CE | 82.39$\pm$0.15
|  | R32 | SSKT | x | CE+KD | 79.8$\pm$0.47
|  | R32 | SSKT | o | CE+CE | 83.4$\pm$0.2
|  | R32 | SSKT | o | CE+KD | 80.05$\pm$1.06

Table 13: Performance change according to the configuration of the SSKT for
ImageNet and Places365 compared to the training from scratch.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | P | R18 | scratch | - | CE | 50.92
P |  | R18 | SSKT | x | CE+CE | 54.41
|  | R18 | SSKT | x | CE+KD | 53.42
|  | R18 | SSKT | o | CE+CE | 54.5
|  | R18 | SSKT | o | CE+KD | 54.11
I |  | R18 | SSKT | x | CE+CE | 53.47
|  | R18 | SSKT | x | CE+KD | 53.51
|  | R18 | SSKT | o | CE+CE | 53.67
|  | R18 | SSKT | o | CE+KD | 53.44
P+I |  | R18 | SSKT | x | CE+CE | 54.78
|  | R18 | SSKT | x | CE+KD | 54.5
|  | R18 | SSKT | o | CE+CE | 54.62
|  | R18 | SSKT | o | CE+KD | 54.5
- | I | R18 | scratch | - | CE | 64.14
P |  | R18 | SSKT | x | CE+CE | 64.18
|  | R18 | SSKT | x | CE+KD | 64.21
|  | R18 | SSKT | o | CE+CE | 64.99
|  | R18 | SSKT | o | CE+KD | 63.53
I |  | R18 | SSKT | x | CE+CE | 67.79
|  | R18 | SSKT | x | CE+KD | 66.0
|  | R18 | SSKT | o | CE+CE | 67.46
|  | R18 | SSKT | o | CE+KD | 65.65
P+I |  | R18 | SSKT | x | CE+CE | 70.57
|  | R18 | SSKT | x | CE+KD | 67.42
|  | R18 | SSKT | o | CE+CE | 67.64
|  | R18 | SSKT | o | CE+KD | 66.81

Table 14: Performance change according to the configuration of the SSKT for
PASCAL VOC compared to the training from scratch.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | VOC | R18 | scratch | - | BCE | 67.28$\pm$0.25
P |  | R18 | SSKT | x | BCE+CE | 74.34$\pm$0.23
|  | R18 | SSKT | x | BCE+KD | 69.92$\pm$0.16
|  | R18 | SSKT | o | BCE+CE | 74.76$\pm$0.17
|  | R18 | SSKT | o | BCE+KD | 69.9$\pm$0.18
I |  | R18 | SSKT | x | BCE+CE | 74.78$\pm$0.09
|  | R18 | SSKT | x | BCE+KD | 69.9$\pm$0.35
|  | R18 | SSKT | o | BCE+CE | 74.58$\pm$0.11
|  | R18 | SSKT | o | BCE+KD | 69.95$\pm$0.19
P+I |  | R18 | SSKT | x | BCE+CE | 76.33$\pm$0.0
|  | R18 | SSKT | x | BCE+KD | 69.74$\pm$0.17
|  | R18 | SSKT | o | BCE+CE | 76.42$\pm$0.06
|  | R18 | SSKT | o | BCE+KD | 69.89$\pm$0.13
- |  | R34 | scratch | - | BCE | 66.0$\pm$0.49
P |  | R34 | SSKT | x | BCE+CE | 73.83$\pm$0.38
|  | R34 | SSKT | x | BCE+KD | 69.93$\pm$0.03
|  | R34 | SSKT | o | BCE+CE | 75.65$\pm$0.12
|  | R34 | SSKT | o | BCE+KD | 69.51$\pm$0.13
I |  | R34 | SSKT | x | BCE+CE | 74.25$\pm$0.12
|  | R34 | SSKT | x | BCE+KD | 70.05$\pm$0.14
|  | R34 | SSKT | o | BCE+CE | 75.14$\pm$0.14
|  | R34 | SSKT | o | BCE+KD | 70.18$\pm$0.11
P+I |  | R34 | SSKT | x | BCE+CE | 75.88$\pm$0.1
|  | R34 | SSKT | x | BCE+KD | 70.15$\pm$0.09
|  | R34 | SSKT | o | BCE+CE | 77.02$\pm$0.02
|  | R34 | SSKT | o | BCE+KD | 70.58$\pm$0.35
- |  | R50 | scratch | - | BCE | 61.16$\pm$0.34
P |  | R50 | SSKT | x | BCE+CE | 63.29$\pm$1.43
|  | R50 | SSKT | x | BCE+KD | 65.5$\pm$0.2
|  | R50 | SSKT | o | BCE+CE | 74.44$\pm$0.06
|  | R50 | SSKT | o | BCE+KD | 65.94$\pm$0.09
I |  | R50 | SSKT | x | BCE+CE | 63.96$\pm$2.74
|  | R50 | SSKT | x | BCE+KD | 66.11$\pm$0.32
|  | R50 | SSKT | o | BCE+CE | 74.24$\pm$0.05
|  | R50 | SSKT | o | BCE+KD | 65.77$\pm$0.13
P+I |  | R50 | SSKT | x | BCE+CE | 69.27$\pm$0.21
|  | R50 | SSKT | x | BCE+KD | 66.0$\pm$0.29
|  | R50 | SSKT | o | BCE+CE | 77.1$\pm$0.14
|  | R50 | SSKT | o | BCE+KD | 66.22$\pm$0.23

Table 15: Performance change according to the configuration of the SSKT for
UCF101 and HMDB51 compared to the training from scratch.

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | U101 | 3DR18 | scratch | - | CE | 43.28
P |  | 3DR18 | SSKT | x | CE+CE | 44.1
|  | 3DR18 | SSKT | x | CE+KD | 44.79
|  | 3DR18 | SSKT | o | CE+CE | 45.35
|  | 3DR18 | SSKT | o | CE+KD | 43.95
I |  | 3DR18 | SSKT | x | CE+CE | 46.62
|  | 3DR18 | SSKT | x | CE+KD | 40.35
|  | 3DR18 | SSKT | o | CE+CE | 44.26
|  | 3DR18 | SSKT | o | CE+KD | 38.95
P+I |  | 3DR18 | SSKT | x | CE+CE | 52.19
|  | 3DR18 | SSKT | x | CE+KD | 43.68
|  | 3DR18 | SSKT | o | CE+CE | 47.09
|  | 3DR18 | SSKT | o | CE+KD | 45.0
- | H51 | 3DR18 | scratch | - | CE | 17.14
P |  | 3DR18 | SSKT | x | CE+CE | 18.18
|  | 3DR18 | SSKT | x | CE+KD | 17.33
|  | 3DR18 | SSKT | o | CE+CE | 18.77
|  | 3DR18 | SSKT | o | CE+KD | 17.59
I |  | 3DR18 | SSKT | x | CE+CE | 18.64
|  | 3DR18 | SSKT | x | CE+KD | 18.12
|  | 3DR18 | SSKT | o | CE+CE | 18.38
|  | 3DR18 | SSKT | o | CE+KD | 18.77
P+I |  | 3DR18 | SSKT | x | CE+CE | 19.75
|  | 3DR18 | SSKT | x | CE+KD | 18.38
|  | 3DR18 | SSKT | o | CE+CE | 20.54
|  | 3DR18 | SSKT | o | CE+KD | 17.99

Figure 6: Parameter optimization of SSKT. The title of each graph is composed
of $D_{t}$ (target model, $T_{s}$, auxiliary loss). $T$ is the temparture
parameter of each auxiliary loss, and $\alpha$ is the balance parameter of the
total loss. Table 16: Performance change according to the configuration of the
SSKT for STL10 compared to the training from scratch with MobileNet V2 (MV2)
and DenseNet121 (D121).

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | S10 | MV2 | scratch | - | CE | 72.26$\pm$0.83
P |  | MV2 | SSKT | x | CE+CE | 75.79$\pm$0.19
|  | MV2 | SSKT | x | CE+KD | 74.0$\pm$0.35
|  | MV2 | SSKT | o | CE+CE | 75.28$\pm$0.49
|  | MV2 | SSKT | o | CE+KD | 73.37$\pm$1.8
I |  | MV2 | SSKT | x | CE+CE | 76.08$\pm$0.63
|  | MV2 | SSKT | x | CE+KD | 74.39$\pm$0.82
|  | MV2 | SSKT | o | CE+CE | 75.35$\pm$0.61
|  | MV2 | SSKT | o | CE+KD | 72.6$\pm$0.67
P+I |  | MV2 | SSKT | x | CE+CE | 76.69$\pm$0.18
|  | MV2 | SSKT | x | CE+KD | 73.29$\pm$0.89
|  | MV2 | SSKT | o | CE+CE | 76.96$\pm$0.39
|  | MV2 | SSKT | o | CE+KD | 73.35$\pm$0.99
- |  | D121 | scratch | - | CE | 72.02$\pm$0.48
P |  | D121 | SSKT | x | CE+CE | 76.17$\pm$0.35
|  | D121 | SSKT | x | CE+KD | 74.83$\pm$0.59
|  | D121 | SSKT | o | CE+CE | 73.46$\pm$0.62
|  | D121 | SSKT | o | CE+KD | 72.55$\pm$0.43
I |  | D121 | SSKT | x | CE+CE | 76.0$\pm$0.33
|  | D121 | SSKT | x | CE+KD | 73.7$\pm$0.22
|  | D121 | SSKT | o | CE+CE | 74.35$\pm$0.3
|  | D121 | SSKT | o | CE+KD | 71.13$\pm$0.59
P+I |  | D121 | SSKT | x | CE+CE | 77.03$\pm$0.17
|  | D121 | SSKT | x | CE+KD | 73.76$\pm$0.84
|  | D121 | SSKT | o | CE+CE | 76.09$\pm$0.26
|  | D121 | SSKT | o | CE+KD | 70.94$\pm$1.14

Table 17: SSKT results for PASCAL VOC, UCF101, and HMDB51 using pretrained
weights. ft stands for finetuning and K stands for Kinetics-400 dataset (Kay
et al. 2017).

$T_{s}$ | $T_{t}$ | Model | Method | TM | Loss | acc.
---|---|---|---|---|---|---
- | VOC | R18 |  ft (I) | - | CE | 90.52$\pm$0.11
P |  | R18 | SSKT | x | CE+CE | 89.3$\pm$0.04
|  | R18 | SSKT | x | CE+KD | 92.28$\pm$0.06
|  | R18 | SSKT | o | CE+CE | 90.83$\pm$0.04
|  | R18 | SSKT | o | CE+KD | 92.21$\pm$0.05
I |  | R18 | SSKT | x | CE+CE | 91.29$\pm$0.03
|  | R18 | SSKT | x | CE+KD | 92.26$\pm$0.07
|  | R18 | SSKT | o | CE+CE | 91.58$\pm$0.15
|  | R18 | SSKT | o | CE+KD | 92.19$\pm$0.09
P+I |  | R18 | SSKT | x | CE+CE | 91.28$\pm$0.05
|  | R18 | SSKT | x | CE+KD | 92.19$\pm$0.07
|  | R18 | SSKT | o | CE+CE | 91.25$\pm$0.08
|  | R18 | SSKT | o | CE+KD | 92.25$\pm$0.07
- | U101 | 3DR18 |  ft (K) | - | CE | 83.95
P |  | 3DR18 | SSKT | x | CE+CE | 84.53
|  | 3DR18 | SSKT | x | CE+KD | 84.58
|  | 3DR18 | SSKT | o | CE+CE | 83.87
|  | 3DR18 | SSKT | o | CE+KD | 83.98
I |  | 3DR18 | SSKT | x | CE+CE | 81.99
|  | 3DR18 | SSKT | x | CE+KD | 83.42
|  | 3DR18 | SSKT | o | CE+CE | 84.29
|  | 3DR18 | SSKT | o | CE+KD | 84.37
P+I |  | 3DR18 | SSKT | x | CE+CE | 78.56
|  | 3DR18 | SSKT | x | CE+KD | 84.14
|  | 3DR18 | SSKT | o | CE+CE | 82.81
|  | 3DR18 | SSKT | o | CE+KD | 84.19
- | H51 | 3DR18 |  ft (K) | - | CE | 56.64
P |  | 3DR18 | SSKT | x | CE+CE | 56.77
|  | 3DR18 | SSKT | x | CE+KD | 56.77
|  | 3DR18 | SSKT | o | CE+CE | 57.75
|  | 3DR18 | SSKT | o | CE+KD | 57.82
I |  | 3DR18 | SSKT | x | CE+CE | 56.18
|  | 3DR18 | SSKT | x | CE+KD | 56.9
|  | 3DR18 | SSKT | o | CE+CE | 53.3
|  | 3DR18 | SSKT | o | CE+KD | 57.75
P+I |  | 3DR18 | SSKT | x | CE+CE | 54.48
|  | 3DR18 | SSKT | x | CE+KD | 56.05
|  | 3DR18 | SSKT | o | CE+CE | 57.29
|  | 3DR18 | SSKT | o | CE+KD | 57.1

Table 18: Comparison with MAXL [23] according to the configuration of the
SSKT. V16 denotes VGG16 [33]. F denotes focal loss [23]. C denotes cross-
entropy loss. $TM$ denotes transfer module.

.  $T_{t}$ $T_{t}$ Model MAXL ($\psi[i]$) SSKT ($T_{s}$, $TM$) $T_{s}$ Model
C10 V16 (F) 93.27$\pm$0.09 (2) 93.56$\pm$0.02 (I, x) V16 V16 (F)
93.47$\pm$0.08 (3) 94.1$\pm$0.1 (I, o) V16 V16 (F) 93.49$\pm$0.05 (5)
92.94$\pm$0.02 (I, x) R50 V16 (F) 93.10$\pm$0.08 (10) 94.1$\pm$0.1 (I, o) R50
V16 (F) - 92.56$\pm$0.15 (P, x) R50 V16 (F) - 92.72$\pm$0.15 (P, o) R50 V16
(F) - 92.6$\pm$0.19 (P+I, x) R50, V16 V16 (F) - 93.06$\pm$0.2 (P+I, o) R50,
V16 V16 (F) - 92.2$\pm$0.08 (P+I, x) R50, R50 V16 (F) - 92.54$\pm$0.17 (P+I,
o) R50, R50 V16 (C) - 93.78$\pm$0.04 (I, x) V16 V16 (C) - 94.22$\pm$0.02 (I,
o) V16 V16 (C) - 93.08$\pm$0.06 (I, x) R50 V16 (C) - 93.78$\pm$0.04 (I, o) R50
V16 (C) - 93.04$\pm$0.15 (P, x) R50 V16 (C) - 93.12$\pm$0.12 (P, o) R50 V16
(C) - 93.35$\pm$0.21 (P+I, x) R50, V16 V16 (C) - 93.67$\pm$0.17 (P+I, o) R50,
V16 V16 (C) - 93.02$\pm$0.11 (P+I, x) R50, R50 V16 (C) - 93.26$\pm$0.12 (P+I,
o) R50, R50 R20 (F) 91.53$\pm$0.33 (2) 90.52$\pm$0.34 (I, x) V16 R20 (F)
91.52$\pm$0.1 (3) 91.48$\pm$0.03 (I, o) V16 R20 (F) 91.38$\pm$0.47 (5)
90.88$\pm$0.02 (I, x) R50 R20 (F) 91.56$\pm$0.16 (10) 91.66$\pm$0.09 (I, o)
R50 R20 (F) - 89.97$\pm$0.02 (P, x) R50 R20 (F) - 91.26$\pm$0.17 (P, o) R50
R20 (F) - 89.42$\pm$0.08 (P+I, x) R50, V16 R20 (F) - 90.93$\pm$0.01 (P+I, o)
R50, V16 R20 (F) - 90.02$\pm$0.21 (P+I, x) R50, R50 R20 (F) - 91.11$\pm$0.18
(P+I, o) R50, R50 R20 (C) - 92.28$\pm$0.07 (I, x) R50 R20 (C) - 92.44$\pm$0.05
(I, o) R50 R20 (C) - 92.21$\pm$0.06 (P, x) R50 R20 (C) - 92.25$\pm 0$0.04 (P,
o) R50 R20 (C) - 91.9$\pm$0.1 (P+I, x) R50, R50 R20 (C) - 92.46$\pm$0.15 (P+I,
o) R50, R50