# UniUD Submission to the EPIC-Kitchens-100 Multi-Instance Retrieval Challenge
2023

Alex Falcon
University of Udine
<EMAIL_ADDRESS>Giuseppe Serra
University of Udine
<EMAIL_ADDRESS>
###### Abstract

In this report, we present the technical details of our submission to the
EPIC-Kitchens-100 Multi-Instance Retrieval Challenge 2023. To participate in
the challenge, we ensembled two models trained with two different loss
functions on 25% of the training data. Our submission, visible on the public
leaderboard, obtains an average score of 56.81% nDCG and 42.63% mAP.

##  1 Introduction

Text-video retrieval is a difficult task requiring a joint visual and textual
understanding. The challenging Multi-Instance Retrieval benchmark offered by
the EPIC-Kitchens-100 comprises around 70k egocentric video clips, each paired
with a short instructional caption, which capture kitchen activities from
different parts of the world. A major difference with respect to other video-
language datasets is the possibility to assess the performance of the
retrieval system with rank-aware metrics, such as nDCG and mAP. This makes it
possible to truly assess the ranking capabilities of a model, which may highly
differ from other metrics which are often used in retrieval contexts, e.g.,
recall metrics, as pointed out in [13].

Our submission to the challenge consists of an ensemble of two solutions we
trained on a random subset of 25% of the training data. We opted to utilize a
smaller dataset for training our solution due to the following reasons:
firstly, using less training data enables lower energy consumption, aligning
with the principles of ’green’ AI; secondly, a data-efficient baseline is not
only more accessible - as it can be trained using a single GPU in under an
hour - but it also mirrors the human learning process, which involves
extrapolating from relatively few examples. The first method included in our
final solution uses a data augmentation technique on top of RANP [6, 7],
whereas the second one is a novel approach based on direct nDCG optimization
[11]. The experiments show that, although using only 25% of the training data,
competitive results can be obtained. In particular, compared to the public
leaderboard from last year, we obtain better nDCG (+1.5%) than Satar et al.’s
submission with less than 0.2% difference in mAP, although they used the full
training set [12].

In Section 2 we provide details about the two optimization strategies [6, 11]
which we used. Section 3 provides the implementation details regarding the
network architecture, the ensemble strategy, and the results. Finally, we
conclude the report in Section 4.

##  2 Optimization strategies

In this section, we provide an overview of the two strategies we used to
perform the training.

###  2.1 Feature-space multimodal data augmentation technique

Data augmentation is typically performed to improve generalization by creating
new samples through semantics-preserving transformations, e.g., random crops
and horizontal flips in images. However, these transformations are commonly
defined on the raw data, which may be difficult to obtain due to privacy or
copyright issues (e.g., video datasets which are collected on YouTube).
Moreover, applying these transformations on the raw data may lead to greater
computational burdens, while also undermining the general applicability of the
augmentation technique (e.g., a technique defined for images may require
considerable efforts and reformulations to be applied to audio). To address
these shortcomings, in [6] we proposed a feature-space multimodal data
augmentation technique which creates new samples by combining the latent
representations of two semantically similar videos (or captions). We used the
relevance function defined in [2] to determine the compatibility of two videos
(or two captions) across those found in the training data under analysis.
Then, we perform the augmentation for every training sample while also using
RANP [7] to improve the triplet loss both by selecting only irrelevant
negatives and by additionally identifying relevant-positives different from
the groundtruth.

###  2.2 Direct optimization of nDCG

The act of sorting a list of scores can be seen as a left-multiplication by a
permutation matrix which is induced by the sorting operator itself. Therefore,
by approximating the induced permutation matrix it is also possible to
approximate the operator. NeuralSort proposes a continuous relaxation of the
sorting operator which returns unimodal row-stochastic matrices in place of
the permutation [8]. Basing their work on NeuralSort, Pobrotyn et al.
introduced NeuralNDCG, which is a loss function designed to directly optimize
nDCG for learning-to-rank applications [11]. Specifically, the approximate
matrix obtained via NeuralSort is left-multiplied to the groundtruth scores,
which are obtained by applying the gain function to the groundtruth labels,
then multiplied by the discount function and divided by the optimal DCG. In
our implementation, we used the relevance values computed by the relevance
function of [2] as the groundtruth labels.

##  3 Experiments

This section concerns the base architecture and ensembling strategy, along the
description of the experimental results.

###  3.1 Implementation details

Base architecture and hyperparameters. In both the considered models, HGR [1]
was used as the base architecture. For each sentence, it builds a hierarchical
structure of the semantic roles and then aggregates the textual features
through graph message passing. Then, both global and local features are
aligned with the corresponding visual features via a bidirectional loss. The
training lasted 50 epochs, using the Adam optimizer with learning rate 1e-4,
batch size 64, and a fixed margin set to 0.2. For the data-augmented model, we
applied it on every sample (100% augmentation chance) and used 0.15 as the
threshold to distinguish relevant from irrelevant elements [7]. To avoid
”leaks” from the full training set, the augmentation is only performed among
the samples found in the subset of size 25%.

Dataset. We randomly sampled 25% of the training set and used it as the
training data. A non-overlapping small validation set was used to track the
performance (the same used by, e.g., [7]). The RGB, flow, and audio features
extracted with TBN [9] and provided by the dataset authors were used for
training.

###  3.2 Ensemble strategy

We deployed a simple ensemble strategy based on last year’s submission [5].
First of all, we perform the training. Then, for each model, we extracted the
similarity matrix computed on the test set. The two matrices are then summed
and averaged: in this way, the similarity value for video $v_{i}$ and caption
$q_{j}$ represents the mean of the two models. The final submission is done by
using the mean similarity matrix.

###  3.3 Results

Table 1 reports the performance obtained by the two single models and by the
ensemble on the validation set. The ensemble (Ens.) obtained the highest
performance and therefore was submitted to the leaderboard. When compared to
the 2021 and 2022 editions of the EPIC-Kitchens-100 Multi-Instance Retrieval
Challenge 2021 [3, 4], we observe a better nDCG (+1.5%) with similar mAP (only
-0.2%) than Satar et al.’s submission which obtained 55.33% nDCG and 42.81%
mAP on average [12], and +3.3% nDCG compared to the official JPoSE baseline
and Hao et al.’s submission at the price of 1.4% mAP [14, 3]. With respect to
last year winners, Lin et al. and our previous submission [5, 10], we obtain a
good nDCG (around 4% less) but the mAP is still far from optimal (around 7%
less than [5]). Noteworthily, all these results from previous works were
obtained using the full training set, and not only 25% of it.

| Validation
---|---
| nDCG (%) | mAP (%)
Mod. | v2t | t2v | avg | v2t | t2v | avg
Aug. | 70.6 | 70.7 | 70.6 | 64.5 | 62.9 | 63.7
DOpt. | 70.2 | 71.0 | 70.6 | 64.0 | 58.9 | 61.5
Ens. | 71.5 | 72.0 | 71.8 | 65.7 | 62.6 | 64.1
| Official test
| nDCG (%) | mAP (%)
Ens. | 58.65 | 54.96 | 56.81 | 47.79 | 37.48 | 42.63
Table 1: Performance of the two single models (Aug.: feature-space multimodal
data augmentation, DOpt.: direct optimization of nDCG) and the ensemble (Ens.)
on the validation set (top) and test set (bottom) of EPIC-Kitchens-100.

##  4 Conclusion

In this report, we summarized and briefly described our submission to the
EPIC-Kitchens-100 Multi-Instance Retrieval Challenge 2023. The proposed method
is an ensemble of the same base architecture trained with two different
optimization strategies [6, 11] on a random subset of 25% of the original
training data. The empirical results show that our solution achieves very good
results compared to previous methods which used at least the full training
data, e.g., we obtained better nDCG (+1.5%) with similar mAP (-0.2%) than last
year’s submission which ranked third.

## Acknowledgements

This work was partly supported by the Italian Ministry of University and
Research (MUR), within the project DM737_HEU_voucher_2b_FALCON (CUP
G25F21003390007).

## References

  * [1] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu.  Fine-grained video-text retrieval with hierarchical graph reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10638–10647, 2020.
  * [2] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  International Journal of Computer Vision (IJCV), 2021.
  * [3] Dima Damen, Adriano Fragomeni, Jonathan Munro, Toby Perrett, Daniel Whettam, Michael Wray, Antonino Furnari, Giovanni Maria Farinella, and Davide Moltisanti.  Epic-kitchens-100- 2021 challenges report.  Technical report, University of Bristol, 2021.
  * [4] Dima Damen, Adriano Fragomeni, Toby Perrett, Daniel Whettam, Michael Wray, Bin Zhu, Antonino Furnari, Giovanni Maria Farinella, and Davide Moltisanti.  Epic-kitchens-100- 2022 challenges report.  Technical report, University of Bristol, 2022.
  * [5] Alex Falcon, Giuseppe Serra, Sergio Escalera, and Oswald Lanz.  Uniud-fbk-ub-unibz submission to the epic-kitchens-100 multi-instance retrieval challenge 2022.  arXiv preprint arXiv:2206.10903, 2022.
  * [6] Alex Falcon, Giuseppe Serra, and Oswald Lanz.  A feature-space multimodal data augmentation technique for text-video retrieval.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 4385–4394, 2022.
  * [7] Alex Falcon, Giuseppe Serra, and Oswald Lanz.  Learning video retrieval models with relevance-aware online mining.  In International Conference on Image Analysis and Processing, pages 182–194. Springer, 2022.
  * [8] Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon.  Stochastic optimization of sorting networks via continuous relaxations.  In Proceedings of the 7th International Conference on Learning Representations, 2019.
  * [9] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5492–5501, 2019.
  * [10] Kevin Qinghong Lin, Alex Jinpeng Wang, Rui Yan, Eric Zhongcong Xu, Rongcheng Tu, Yanru Zhu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, et al.  Egocentric video-language pretraining@ epic-kitchens-100 multi-instance retrieval challenge 2022.  arXiv preprint arXiv:2207.01334, 2022.
  * [11] Przemysław Pobrotyn and Radosław Białobrzeski.  Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting.  arXiv preprint arXiv:2102.07831, 2021.
  * [12] Burak Satar, Hongyuan Zhu, Hanwang Zhang, and Joo Hwee Lim.  Exploiting semantic role contextualized video features for multi-instance text-video retrieval epic-kitchens-100 multi-instance retrieval challenge 2022.  arXiv preprint arXiv:2206.14381, 2022.
  * [13] Michael Wray, Hazel Doughty, and Dima Damen.  On semantic similarity in video retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3650–3660, 2021.
  * [14] Michael Wray, Diane Larlus, Gabriela Csurka, and Dima Damen.  Fine-grained action retrieval through multiple parts-of-speech embeddings.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 450–459, 2019.