11institutetext: Indian Institute of Technology Guwahati, Assam, India

# Zero-Shot Underwater Gesture Recognition

Sandipan Sarma  Gundameedi Sai Ram Mohan
Hariansh Sehgal  Arijit Sur

###### Abstract

Hand gesture recognition allows humans to interact with machines non-verbally,
which has a huge application in underwater exploration using autonomous
underwater vehicles. Recently, a new gesture-based language called CADDIAN has
been devised for divers, and supervised learning methods have been applied to
recognize the gestures with high accuracy. However, such methods fail when
they encounter unseen gestures in real time. In this work, we advocate the
need for zero-shot underwater gesture recognition (ZSUGR), where the objective
is to train a model with visual samples of gestures from a few “seen” classes
only and transfer the gained knowledge at test time to recognize semantically-
similar unseen gesture classes as well. After discussing the problem and
dataset-specific challenges, we propose new seen-unseen splits for gesture
classes in CADDY dataset. Then, we present a two-stage framework, where a
novel transformer learns strong visual gesture cues and feeds them to a
conditional generative adversarial network that learns to mimic feature
distribution. We use the trained generator as a feature synthesizer for unseen
classes, enabling zero-shot learning. Extensive experiments demonstrate that
our method outperforms the existing zero-shot techniques. We conclude by
providing useful insights into our framework and suggesting directions for
future research. The code is available at:
https://github.com/sandipan211/ZSUGR.

###### Keywords:

Underwater gesture recognition Zero-shot learning Autonomous underwater
vehicles Transformers Cross-attention Generative adversarial networks

##  1 Introduction

Despite our planet being 70% water, our knowledge regarding underwater
ecosystems remains limited. A major reason behind this is the harsh underwater
environment and human divers face trouble collecting data from ocean depths
due to several concerns like increasing hydrostatic pressure and oxygen
depletion. In the past few decades, humans have passed the baton to autonomous
underwater vehicles (AUVs) that can assist marine experts in several ways by
capturing underwater images/videos, detecting oil spillage, inspecting oil and
natural gas pipelines, and conducting bathymetric surveys – all while being
well-resistant to the problems faced by human divers. As a result, underwater
exploration has garnered a lot of attention nowadays, with applications in
oceanography, marine warfare, information navigation, and marine scene
understanding.

In many underwater missions, AUVs are accompanied by human divers who
communicate non-verbally via different gestures. However, underwater gesture
recognition is a relatively underexplored area in computer vision owing to the
lack of annotated datasets. Recently, a new gesture-based communication
language called CADDIAN [5] was developed, and an image-based underwater
gesture recognition dataset named CADDY [8] (Fig. 1(b)) was constructed as an
effort to facilitate vision research. However, there are primarily two
challenges. Firstly, underwater images suffer from problems like low contrast,
haziness, color distortion, and blurriness. As a result, traditional gesture
recognition methods face problems in analyzing them. Secondly, existing
gesture recognition models are predominantly supervised [32, 20, 2, 34, 33,
18] and can only recognize gestures from a predefined set used to train the
models, failing to interpret “unseen” gestures. This is an inevitable
bottleneck as it is impossible to collect thousands of labeled images for
every possible gesture that might be used by human divers in the wild. For
instance, a new gesture meant to indicate low oxygen levels due to a real-time
accident might not be interpreted by an AUV equipped with a supervised gesture
recognition model, putting the human diver at risk.

Zero-shot learning (ZSL) has come to the limelight in recent years to
alleviate such data scarcity, which mimics the human tendency to learn from
other modalities (semantics or class attributes) like text or audio in the
absence of visual examples. The objective of ZSL is to transfer knowledge
about a few “seen” concepts/classes via a visual-semantic association and
recognize unseen ones. Although a few recent works explore zero-shot gesture
recognition, it is still unexplored in underwater scenarios.

In this paper, we introduce and study the task of zero-shot underwater gesture
recognition (ZSUGR). Being the first work of its kind, standard seen-unseen
zero-shot splits are currently unavailable. Therefore, we propose three new
splits by designating seen and unseen classes randomly, following previous
splitting strategies [31, 14]. Considering the challenges in extracting strong
visual features from underwater images and the highly class-imbalanced nature
of the CADDY dataset (Fig. 1(a)), a two-stage framework is devised consisting
of a novel transformer and a generative adversarial network. In the first
stage, we design a novel Gated Cross-Attention Transformer (GCAT) which is
responsible for a strong representation learning. Visual features extracted
from a pretrained ResNet-50 are passed to an encoder, and its outputs are
transformed into powerful gesture representations by our novel gesture
decoder. The trained GCAT is then used as a visual feature extractor for the
seen classes. A conditional Wasserstein GAN [1] is then trained with these
visual features, with seen class semantic vectors as class conditional
variables. Here, we rely on the text encoder of a pretrained visual-language
model called CLIP [23] to obtain the semantic vector of a gesture class.
Visual features of unseen gestures can then be synthesized using our trained
WGAN, enabling us to train a gesture classifier with data from both seen and
unseen classes, mitigating the bias problem in zero-shot settings [22].
Extensive experiments are conducted, showing the failure of supervised models,
our model’s improved performance compared to state-of-the-art zero-shot
classification methods in conventional and generalized zero-shot settings
(inspired by previous zero-shot gesture recognition works [14, 17, 29]), and
demonstrating the role of improved visual representations for ZSUGR. To
summarize, we make the following contributions in this paper:

  * •

We introduce the problem of zero-shot underwater gesture recognition from
images for the first time and discuss its real-world applications.

  * •

Seen-unseen splits of gesture classes are proposed for the CADDY dataset for
zero-shot training and model evaluation.

  * •

A two-stage network is proposed for ZSUGR consisting of a novel transformer
that can be trained as a strong underwater visual feature extractor. The
obtained features can be fed to a class-conditional generative adversarial
network that can learn to synthesize visual features of unseen gestures.

(a) Distribution of gesture classes

(b) Examples of the 16 gestures

Figure 1: Properties of the CADDY dataset [8].

##  2 Related Works

###  2.1 Hand gesture recognition (HGR)

Vision-based HGR methods can be broadly categorized into dynamic (limb
movements indicate gesture) and static (hand shape and finger orientations
indicate gesture). Among these, segmenting the hand using skin color models to
extract visual gesture features is a common practice [35, 26]. With the
availability of neural networks pretrained on large-scale datasets, transfer
learning approaches [28] have also been successful. However, the extracted
visual features are affected by factors such as lighting conditions and
background clutter. To enable HGR frameworks to recognize a wider range of
gestures, 3D methods are explored, where several sensors and cameras are used
to obtain temporal volumes [13] or skeletal joints [6] to analyze the hand
shapes.

###  2.2 Zero-shot learning (ZSL)

Image classification remains the most researched vision task exploiting ZSL.
The first seen-unseen splits were formally provided for five datasets in the
seminal work of Xian et al. [30], who also popularized the concept of
conventional (only unseen class samples at test time) and generalized (samples
from both seen and unseen classes at test time) zero-shot learning settings.
Meanwhile, gesture recognition as a vision task is less explored. One of the
earliest works in zero-shot gesture recognition (ZSGR) unified coordinated
natural language, gesture, and context [27] to facilitate human-robot
interactions. However, ZSGR was standardized in the work of Madapana and Wachs
[15], who used gestures from two gesture-based datasets to define a total of
13 seen and 8 unseen classes. Furthermore, they provided a list of 13 high-
level semantic descriptors to characterize each gesture class. Their zero-shot
classifiers were inspired by previous image classification methods [12, 24].
In an extended work, they further introduced hard zero-shot gesture
recognition [17] where there are only a few visual samples for the seen
classes as well. Recently in another work, they introduced a gesture attribute
dataset [16] with segmented skeletal data and proposed a joint semantic
encoder optimizing reconstruction, semantic, and classification losses [14].
Another novel dataset was built by Wu et al. [28] with 16 seen and 9 unseen
classes, and then they proposed an end-to-end prototype learning framework
[29].

###  2.3 Underwater diver gesture recognition

Since the arrival of the CADDY dataset [8], several works have been undertaken
in the last four years in underwater gesture recognition (UGR). The first such
work [32] used pretrained convolutional networks like ResNet, GoogleNet, and
others for transfer learning. Several other works that followed [20, 2, 18]
compared the performance of classical and deep learning methods for UGR. On
the other hand, DARE [33] demonstrated a hierarchical tree-structured
classification scheme. Recently, VT-UHGR [34] used a pretrained ViT as the
visual encoder and a pretrained BERT as the text encoder for multimodal
underwater gesture feature learning. Data augmentation using generative
methods has also been used to improve UGR performance. However, most of these
methods use transfer learning and do not provide any novel architecture for
UGR. Additionally, none of them are zero-shot learners. This paper presents
the first work that addresses both of these concerns.

##  3 Methodology

###  3.1 Problem definition

Let we have a training set
$\mathcal{D}_{train}=\\{(x_{i},y_{i})|\;x_{i}\in\mathcal{X}_{seen},y_{i}\in\mathcal{S}\\}$,
where $x_{i}$ is an image from a seen class $y_{i}$. A separate set of unseen
data
$\mathcal{D}_{novel}=\\{(x_{j},y_{j})|\;x_{j}\in\mathcal{X}_{unseen},y_{j}\in\mathcal{U}\\}$
is given such that the sets of seen and unseen classes are disjoint, i.e.,
$\mathcal{S}\;\cap\;\mathcal{U}=\phi$. Additionally, we have a semantic vector
$a_{i}\in\mathbb{R}^{512}$ for each gesture class
$y\in\mathcal{S}\,\cup\,\mathcal{U}$. We work in the more realistic inductive
zero-shot setting [30] – where visual examples of unseen classes are
unavailable during training – instead of the transductive setting. Then, the
task in conventional zero-shot learning is to learn a classifier
$f_{zsl}:\mathcal{X}_{unseen}\;\rightarrow\;\mathcal{U}$. For generalized
zero-shot learning, a small subset of $\mathcal{X}_{seen}$
($\mathcal{X}_{seen}^{sub}$) is extracted as the set of seen samples at test
time, following the suggestions in the benchmark ZSL paper [30]. The objective
then changes to learning a classifier
$f_{gzsl}:\mathcal{X}_{seen}^{sub}\;\cup\;\mathcal{X}_{unseen}\rightarrow\mathcal{S}\;\cup\;\mathcal{U}$.

Figure 2: Proposed two-stage framework for ZSUGR. Here, $z$ denotes a random
noise vector, and $a$ denotes semantic vector of a gesture class.

###  3.2 System overview

Figure 2 shows our overall two-stage framework, which includes a novel
transformer and a Wasserstein GAN (WGAN). Given an underwater image
$\mathcal{X}\in\mathbb{R}^{C\times H\times W}$, its visual feature map
$\boldsymbol{\mathcal{V}_{b}}\in\mathbb{R}^{C^{\prime}\times H^{\prime}\times
W^{\prime}}$ is first extracted using a pretrained ResNet-50 backbone. Since a
transformer encoder considers each spatial location in
$\boldsymbol{\mathcal{V}_{b}}$ as a token while computing self-attention, it
is necessary to keep track of the token positions. We construct positional
embeddings $\boldsymbol{\mathcal{X}_{pos}}\in\mathbb{R}^{C^{\prime}\times
H^{\prime}\times W^{\prime}}$, add them element-wise with
$\boldsymbol{\mathcal{V}_{b}}$, and pass the resulting feature map to the
transformer encoder $E$. The outputs from $E$ are then decoded via our novel
gated cross-attention mechanism guided by powerful visual representations from
a pre-trained CLIP. These decoded visual gesture features are used as “real
features” for training a WGAN, conditioned on class semantics. The trained
WGAN can be used to generate visual features corresponding to unseen gesture
classes. A classifier $\Phi_{cls}(.)$ is trained with seen class data and the
synthesized unseen class data for zero-shot gesture recognition. We discuss
the novel GCAT in Sec. 3.3, followed by the WGAN in Sec. 3.4.

###  3.3 Gated cross-attention transformer (GCAT)

As mentioned in Sec. 3.2, the encoder $E$ takes the input:

$\boldsymbol{\mathcal{X}_{e}}=\boldsymbol{\mathcal{V}_{b}}+\boldsymbol{\mathcal{X}_{pos}}$
(1)

and performs a self-attention operation on the visual tokens. The output
$\boldsymbol{\mathcal{O}_{e}}\in\mathbb{R}^{C^{\prime}\times
H^{\prime}W^{\prime}}$ contains image-wide contextual information that is
passed to our gesture decoder $Dec$. Meanwhile, we extract additional visual
features $\boldsymbol{\mathcal{V}_{c}}\in\mathbb{R}^{C^{\prime\prime}\times
k}$ from a pretrained CLIP image encoder and use them to refine the knowledge
coming from the encoder, yielding gesture features. To this end,
$\boldsymbol{\mathcal{O}_{e}}$ is first projected to CLIP dimension, and then
a two-branch cross-attention mechanism is proposed. The two branches perform
identical operations, with the only difference being in the query ($Q$), key
($K$), and value ($V$) assignments. The branch assignments (Fig. 3) are as
follows:

$\displaystyle Q_{L}=\boldsymbol{\mathcal{O}_{e}},\quad
K_{L}=V_{L}=\boldsymbol{\mathcal{V}_{c}}$ (2) $\displaystyle
Q_{R}=\boldsymbol{\mathcal{V}_{c}},\quad
K_{R}=V_{R}=\boldsymbol{\mathcal{O}_{e}}$ (3)

where $L$ and $R$ denote left and right branches. The attention outputs are
normalized using layer normalization (LN) operation and added to the branch
queries:

$\displaystyle A_{L}=LN(Q_{L}+CrossAtt(Q_{L},K_{L},V_{L}))$ (4) $\displaystyle
A_{R}=LN(Q_{R}+CrossAtt(Q_{R},K_{R},V_{R}))$ (5)

Now, since visual inputs to $Dec$ come from two sources –
$\boldsymbol{\mathcal{O}_{e}}$ based on ResNet features and
$\boldsymbol{\mathcal{V}_{c}}$ based on CLIP features – it is beneficial to
weight the attention outputs for understanding the contribution of the encoder
outputs as well as CLIP context. Consequently, we propose a gating mechanism
where we apply a $1\times 1$ convolution first to the attention outputs of
each branch and downscale them by a factor of 2. A GELU activation [11]
finally produces the gated values. The gating function can hence be formulated
as:

$\displaystyle g(A_{L})=GELU(\boldsymbol{w_{L}}A_{L}+\boldsymbol{b_{L}})$ (6)
$\displaystyle g(A_{R})=GELU(\boldsymbol{w_{R}}A_{R}+\boldsymbol{b_{R}})$ (7)

where $(\boldsymbol{w_{L}},\boldsymbol{b_{L}})$ and
$(\boldsymbol{w_{R}},\boldsymbol{b_{R}})$ denote the weights and biases of the
convolutional layer in the left and right branches respectively. The gated
outputs indicate how much cross-attention information should be preserved and
how much can be learned subsequently. The attention maps from the two branches
are fused together, and the source knowledge from the encoder is refined as:

$\displaystyle\boldsymbol{\mathcal{O}_{e}}=\boldsymbol{{\mathcal{O}_{e}}}\circ(A_{L}+A_{R})$
(8)

where $\circ$ denotes Hadamard product. It is then passed to a feedforward
network (FFN), as shown in Fig. 3, to produce the final feature map
$\boldsymbol{\mathcal{O}_{T}}$ containing gesture cues. A channel-wise mean is
taken at the end such that $\boldsymbol{\mathcal{O}_{T}}\in\mathbb{R}^{d}$.

Figure 3: Architecture of our novel GCAT gesture decoder.

###  3.4 Transformed-feature generating network

Due to problems like color distortion and haziness that underwater images
suffer from, it becomes exceedingly difficult to work in extreme scenarios
like zero-shot in the absence of visual images of unseen classes. Moreover,
predictions may get highly biased toward the seen classes owing to the
imbalanced nature of underwater datasets like CADDY. To alleviate the seen-
bias problem, we employ a conditional Wasserstein Generative Adversarial
Network (c-WGAN) [1]. As depicted in Fig. 2, it takes the visual gesture
features $\boldsymbol{\mathcal{O}_{T}}$ corresponding to seen classes as input
and their semantic vectors ($a$) as a conditional variable. The generator $G$
attempts to mimic the true distribution of the visual features
$\boldsymbol{\mathcal{O}_{T}}$ while being discriminative in the visual space
with guidance received from the class semantics. It generates seen class
features $\boldsymbol{\mathcal{\tilde{O}}_{T}}$, which are then fed to a
critic network $D$ along with features from true distribution
$\boldsymbol{\mathcal{O}_{T}}$. $D$ acts as a binary classifier that scores
the “realness” of the input data (either coming from the mimicked feature
distribution or from the true distribution). During backpropagation, $D$ tries
to improve its predictions, whereas $G$ tries to improve itself based on the
feedback from $D$ and synthesizes features closer to the true distribution.
For optimization, we use the Wasserstein loss:

$\displaystyle\mathcal{L}_{\tt
WGAN}=\mathbb{E}[D(\boldsymbol{\mathcal{O}_{T}},a)]-\mathbb{E}[D(\tilde{\boldsymbol{\mathcal{O}_{T}}},a)]+\lambda\mathbb{E}[(||\nabla_{\boldsymbol{\hat{\mathcal{O}_{T}}}}D(\hat{\boldsymbol{\mathcal{O}_{T}}},a)||_{2}-1)^{2}]$
(9)

where the first two terms represent the critic loss in WGAN and the third term
represents a gradient penalty [9], with $\lambda$ being the penalty
coefficient. $\tilde{\boldsymbol{\mathcal{O}_{T}}}=G(z,a)$ denotes the
generated feature, and
$\boldsymbol{\hat{\mathcal{O}_{T}}}=\rho\,\boldsymbol{\mathcal{O}_{T}}+(1-\rho)\,\tilde{\boldsymbol{\mathcal{O}_{T}}}$,
with $\rho\sim U(0,1)$. Additionally, we use a mode-seeking loss [19] to
mitigate the problem of mode-collapse in GANs:

$\displaystyle\mathcal{L}_{\tt
MS}=\mathbb{E}[||G(z_{1},a)-G(z_{2},a)||_{1}/||z_{1}-z_{2}||_{1}]$ (10)

Here, $z_{1}$ and $z_{2}$ correspond to two noise vectors that produce
different visual features. The idea is to regularize that the generated
features should get sampled corresponding to noise vectors in the learned
distribution that are far apart from each other, giving rise to discriminative
synthetic visual features.

Table 1: Details of proposed seen-unseen splits of the CADDY dataset. Split | Seen classes | Unseen classes | $\mathcal{X}_{seen}$ | $\mathcal{X}_{seen}^{sub}$ | $\mathcal{X}_{unseen}$
---|---|---|---|---|---
1 | 10 | 6 | 12505 | 1389 | 4584
2 | 10 | 6 | 10251 | 1139 | 7088
3 | 10 | 6 | 11713 | 1301 | 5464

###  3.5 Training and inference

We follow a two-stage training process, as shown in Fig. 2. In the first
stage, we train the transformer GCAT to produce visual gesture features. We
feed them to a classifier $\Phi_{c}$ whose weights are initialized by the text
embeddings (or semantics) of the seen classes. For a class $y_{i}$, we obtain
its semantic vector by converting the class name to “A photo of a diver
gesturing [$y_{i}$]” via a prompt template $\rho_{g}(.)$, and then using a
pretrained CLIP text encoder [23] $\psi(.)$ as:

$\displaystyle\boldsymbol{a_{i}}=\psi(\rho_{g}(y_{i}))\quad\forall\,y_{i}\in\mathcal{S}\cup\mathcal{U}$
(11)

We minimize a cross-entropy loss ($\mathcal{L}_{\tt CE}$) between the
predicted and ground-truth gesture labels, strengthening the visual feature
representations for seen gestures.

The trained GCAT is used in the next stage as a feature extractor for
underwater images. With the extracted features (seen data), the c-WGAN is
trained to synthesize seen data by minimizing the loss (with hyperparameter
$\alpha$):

$\displaystyle\underset{G}{min}\;\underset{D}{max}\quad\mathcal{L}_{\tt
WGAN}+\alpha\,\mathcal{L}_{\tt MS}$ (12)

Given unseen class semantic vectors, the trained GAN can then be used to
generate unseen class visual features. The synthesized unseen features and the
visual features for seen classes extracted from the trained GCAT can then be
combined and used to train a linear softmax classifier $\Phi_{cls}(.)$ for
zero-shot prediction. At test time, visual gesture features of an image are
first extracted using GCAT, and the trained classifier $\Phi_{cls}(.)$
predicts the gesture class.

Table 2: Top-1 accuracy (in %) reported for CZSL and GZSL settings. H denotes harmonic mean of seen and unseen accuracy in GZSL. Best results are in bold. Codes for CZSL training and evaluation were not provided by DGZ [3]. Method | $\boldsymbol{U_{czsl}}$ | $\boldsymbol{S_{gzsl}}$ | $\boldsymbol{U_{gzsl}}$ | $\boldsymbol{H}$
---|---|---|---|---
TFVAEGAN [21] | $41.50\pm 5.47$ | $79.57\pm 13.11$ | $13.49\pm 5.28$ | $22.51\pm 7.40$
CNZSL [25] | $16.72\pm 0.08$ | $20.27\pm 3.84$ | $11.88\pm 2.78$ | $14.97\pm 3.23$
FREE [4] | $15.83\pm 3.95$ | $84.88\pm 9.03$ | $14.55\pm 3.36$ | $24.61\pm 4.50$
CE-GZSL [10] | $39.89\pm 6.49$ | $\boldsymbol{94.11\pm 0.55}$ | $2.58\pm 0.45$ | $5.01\pm 0.87$
DGZ [3] | - | $57.89\pm 2.90$ | $15.72\pm 3.32$ | $24.62\pm 4.14$
Ours | $\boldsymbol{45.91\pm 4.71}$ | $61.93\pm 5.71$ | $\boldsymbol{20.03\pm 7.14}$ | $\boldsymbol{29.53\pm 7.06}$

##  4 Experiments

###  4.1 Dataset

All our experiments are conducted on the CADDY underwater stereo-vision
dataset [8], one of the largest publicly available underwater image datasets
and the only one for diver gesture recognition. Overall, 18,478 labeled diver
gesture images are present in CADDY, belonging to 16 different gesture classes
(Fig. 1(b)). The gestures used by divers correspond to the CADDIAN sign
language [5]. Notably, the dataset is highly imbalanced with a long-tail
distribution (Fig. 1(a)), making it very challenging for zero-shot learning.

###  4.2 Zero-shot splits and evaluation protocols

Due to the absence of any previous work on ZSUGR, we define new splits of the
CADDY dataset, marking the classes to be used as seen/unseen for zero-shot
models. In previously undertaken zero-shot vision applications, seen-unseen
splits are proposed following two approaches. Some works designate fixed sets
of seen/unseen classes [30] for a given dataset. On the other hand, some works
randomly split the dataset classes into seen and unseen sets at a 50%/50%
proportion, and results are reported over multiple such random splits [31]. We
take this second approach to eliminate human bias in picking the seen-unseen
split. However, owing to highly imbalanced data, we propose to pick 10 seen
and 6 unseen classes randomly from the dataset, and three such splits are
obtained. Table 1 provides our split details.

For model evaluation, we follow the benchmark protocol for zero-shot
classification [30]. For conventional zero-shot (CZSL) evaluation, we report
the top-1 accuracy for the unseen classes ($\boldsymbol{U_{czsl}}$). In the
case of generalized evaluation (GZSL), we compute the top-1 accuracy for both
seen ($\boldsymbol{S_{gzsl}}$) and unseen ($\boldsymbol{U_{gzsl}}$) classes
and report their harmonic mean ($\boldsymbol{H}$). The harmonic mean
encourages the model to perform well on both seen and unseen classes, which is
a more realistic metric for zero-shot settings. Since we have multiple random
splits, we indicate the mean and standard deviation of obtained accuracy
across the three random splits for all the reported results.

(a) DGZ [3]

(b) Ours

Figure 4: Comparison of our GZSL confusion matrix with the state-of-the-art.

###  4.3 Implementation details

All the experiments are undertaken in PyTorch using a single NVIDIA A100 GPU.
The raw underwater images are resized to $3\times 224\times 224$. We used the
ResNet-50 backbone to extract a feature map $\boldsymbol{\mathcal{V}_{b}}$ of
$256\times 7\times 7$. Our encoder $E$ contains the same layers as the
traditional transformer, consisting of a self-attention mechanism, followed by
layer normalization and neural layers. The final encoder output is obtained
after 3 blocks of encoder layers. We use 3 blocks of decoder layers, and the
final decoder output $\boldsymbol{\mathcal{O}_{T}}\in\mathbb{R}^{(7\times
7)\times 512}$ is averaged across the feature maps to yield a 512-dimensional
visual gesture feature. The transformer GCAT is optimized using AdamW with a
learning rate of 1e-5 and weight decay of 1e-4. The output from the CLIP image
encoder $\boldsymbol{\mathcal{V}_{c}}$ used for refining the encoder features
is a feature map of dimension ${768\times 50}$. As for the CLIP text encoder,
the output embeddings are 512-dimensional vectors used as semantic vectors for
every class. In our c-WGAN, the generator ($G$) and critic ($D$) networks are
essentially two multi-layer perceptions, both optimized using the Adam
optimizer with a learning rate of 1e-4. In $\mathcal{L}_{\tt MS}$, we use
$\alpha=1e-4$.

(a) $\boldsymbol{U_{czsl}}$

(b) $\boldsymbol{S_{gzsl}}$

(c) $\boldsymbol{U_{gzsl}}$

(d) $\boldsymbol{H}$

Figure 5: Component analysis of our framework for the three proposed random
splits. Top-1 accuracy is reported in CZSL ($\boldsymbol{U_{czsl}}$) and GZSL
settings ($\boldsymbol{S_{gzsl}}$ and $\boldsymbol{U_{gzsl}}$ with harmonic
mean $\boldsymbol{H}$). E = GCAT encoder, RN-101 = ResNet-101 as feature
extractor, Dec = GCAT decoder, c-WGAN = Conditional WGAN.

###  4.4 Zero-shot results

Since there are no prior works in ZSUGR, we adapted a few zero-shot image
classification methods for the ZSUGR task, following previous works [17, 29,
14]. For a fair comparison, we train these methods with the same semantic
vectors as ours. We evaluate these models in both CZSL and GZSL settings and
report their mean and standard deviations over the three random splits we
proposed.

####  4.4.1 Conventional setting (CZSL).

DGZ [3] did not provide their implementation for CZSL evaluation. CZSL results
for some other methods are reported in Tab. 2. Our method outperforms all of
them by a good margin with an average top-1 accuracy of $45.91\pm 4.71\%$.

Figure 6: Attention visualization of GCAT gesture decoder for ten gestures.

####  4.4.2 Generalized setting (GZSL).

Table 2 suggests that CE-GZSL [10] is extremely capable of recognizing seen
gestures during evaluation with an accuracy of $94.11\pm 0.55\%$ but performs
very poorly in case of unseen gesture classes with an accuracy of $2.58\pm
0.45\%$, yielding low harmonic mean. This indicates that CE-GZSL has low
generalizability in ZSUGR for the CADDY dataset, although it exhibited state-
of-the-art (SOTA) generalizability in image classification. The other methods
suffer from the same issue as well. One of the reasons behind such low
generalization ability could be the class imbalance problem in CADDY, which is
not as extreme in the image classification datasets on which these methods are
usually evaluated. Moreover, the zero-shot methods usually suffer from the
bias problem [22], where predictions for unseen class samples are incorrectly
biased towards a similar seen class. Together, the bias problem and class
imbalance make ZSUGR a challenging task for the SOTA zero-shot classification
methods. Meanwhile, our method seemingly finds a better balance between seen
and unseen class knowledge. Additionally, it deals with the bias problem
better than the other generative methods like TFVAEGAN [21], FREE [4], and DGZ
[3], indicating that our GAN has had superior training with gesture features
extracted from the novel GCAT, as compared to the ResNet-101 features with
which the GANs of other methods are trained. Consequently, we obtain the best
harmonic mean of $29.53\pm 7.06\%$ (Tab. 2). For class-wise comparison, we
plot confusion matrices in Fig. 4 to demonstrate how the SOTA classification
method DGZ [3] fares against our method.

Table 3: Effect of activation function on top-1 accuracy (in %) during CZSL and GZSL evaluation. H denotes the harmonic mean of seen and unseen accuracy. Activation | $\boldsymbol{U_{czsl}}$ | $\boldsymbol{S_{gzsl}}$ | $\boldsymbol{U_{gzsl}}$ | $\boldsymbol{H}$
---|---|---|---|---
GELU | $45.16\pm 4.64$ | $61.93\pm 5.71$ | $20.03\pm 7.14$ | $29.53\pm 7.06$
ELU | $44.66\pm 8.11$ | $59.50\pm 12.11$ | $18.05\pm 5.94$ | $26.84\pm 6.54$
RELU | $45.38\pm 8.74$ | $65.48\pm 2.03$ | $14.73\pm 4.17$ | $23.86\pm 6.65$
Sigmoid | $40.71\pm 5.25$ | $36.50\pm 5.52$ | $30.02\pm 2.81$ | $32.67\pm 2.15$
SiLU | $47.71\pm 3.75$ | $65.65\pm 5.09$ | $15.46\pm 0.74$ | $25.03\pm 1.24$

####  4.4.3 Qualitative analysis.

Fig. 6 shows decoder attention visualizations highlighting where our GCAT
gesture decoder is focusing at test time while extracting visual gesture
features. We observe that our trained GCAT focuses on the hands of the divers,
resulting in highly relevant visual features extracted for ZSUGR.
Specifically, for relevant gestures like three, photo, and four, the decoder
focuses on the fingers, which are more informative. On the other hand, it
shifts its focus to both hands whenever relevant, whether the hands are close
to each other (like boat and carry) or distant (such as mosaic).

####  4.4.4 Plight of supervised models.

Previous works [32, 20, 18] have shown that pretrained CNN models can be
easily adapted to underwater gesture recognition via transfer learning on
CADDY. In Tab. 4, we mention the supervised gesture recognition results for
CADDY as reported by previous works, with the best accuracy reaching 98% using
ResNet-18. We employ these models in the GZSL setting to investigate their
performance in recognizing unseen gestures. To this end, we obtain visual
features from these pretrained CNNs via transfer learning and compute a cosine
similarity with the CLIP semantic vectors to get zero-shot predictions. It can
be observed from Tab. 4 that their performance drops for both seen and unseen
classes. Specifically, they fail miserably in recognizing unseen gestures,
with a best harmonic mean of $2.45\pm 4.18\%$ achieved using MobileNet-v3. On
the contrary, our model achieves a harmonic mean of $29.53\pm 7.06\%$. Hence,
the need for zero-shot models for underwater gesture recognition is evident as
the best-performing supervised models are insufficient.

###  4.5 Ablation studies

####  4.5.1 Component analysis

To validate the effect of each component, we conduct an ablation study on
CADDY in both CZSL and GZSL settings with the following changes to our
framework: (i) removing the novel GCAT gesture decoder and using only the
vanilla encoder to get visual features and (ii) removing the GCAT module and
using pretrained ResNet-101 visual features to train our c-WGAN. Fig. 5
suggests that our decoder significantly improves performance in both CZSL and
GZSL settings across all three random splits. This justifies refining the
visual features extracted by the vanilla encoder using our novel decoder,
which makes GCAT attend to more critical and discriminative regions of the
diver’s hand within the image. On the contrary, the raw features from
ResNet-101 or the vanilla encoder alone are less effective.

Table 4: A performance comparison of pretrained CNN models in the supervised (done by previous works) and generalized zero-shot settings (shown by us). Top-1 accuracy (in %) is reported for supervised setting, whereas top-1 accuracy for seen and unseen classes are reported over our proposed splits for GZSL. Pretrained CNN | Supervised | $\boldsymbol{S_{gzsl}}$ | $\boldsymbol{U_{gzsl}}$ | $\boldsymbol{H}$
---|---|---|---|---
AlexNet [32] | 82.89 | $71.75\pm 2.19$ | $0.76\pm 1.21$ | $1.48\pm 2.35$
VGG-16 [32] | 95.00 | $76.20\pm 1.23$ | $0.42\pm 0.57$ | $0.84\pm 1.12$
ResNet-18 [18] | 98.00 | $53.54\pm 3.77$ | $0.72\pm 1.25$ | $1.38\pm 2.39$
ResNet-50 [20] | 97.06 | $61.94\pm 2.86$ | $0.79\pm 1.27$ | $1.53\pm 2.44$
GoogleNet [32] | 90.08 | $53.19\pm 3.46$ | $0.8\pm 1.39$ | $1.53\pm 2.64$
MobileNet-v3 [18] | 84.32 | $62.78\pm 5.21$ | $1.31\pm 2.23$ | $2.45\pm 4.18$

####  4.5.2 Impact of activation functions

In Tab. 3, we study the choice of activation function in the architecture of
our novel gesture decoder while training the GCAT. In the CZSL scenario, we
see comparable performance of the five different activation functions we
studied. Interestingly, in the GZSL setting, we find that different
activations have varying impacts on recognizing seen and unseen gestures. For
example, the sigmoid function generalizes well, but seen accuracy is low. On
the other hand, using SiLU [7] degrades unseen accuracy. The activation we
used finally, GELU, performs well in both CZSL and GZSL settings. In summary,
it is evident that the activation functions can dictate the quality of visual
gesture features which can, in turn, hamper a generative method’s ability to
generate discriminative visual features for seen and unseen classes.

##  5 Conclusion

We introduced the task of zero-shot underwater gesture recognition in this
paper and discussed its potential for diver-AUV communications. We proposed a
two-stage framework consisting of a novel transformer that learns strong
visual gesture representations and a conditional generative network that
learns the distributions of these features. From the dataset perspective, we
proposed three random seen-unseen splits of the CADDY dataset and reported our
results in both conventional and generalized zero-shot settings, comparing
them with state-of-the-art classification methods. Through extensive
experimentations and ablations, we discussed how poorly supervised methods
perform in zero-shot settings, why existing zero-shot classification methods
suffer so much from class imbalance and bias problems, and how activation
functions may impact transformers for gesture recognition. One of the areas
for future research could be learning the semantic representations of gestures
instead of using handcrafted prompt templates. We hope that this work can
serve as a benchmark for the advancement of ZSUGR as a field since there is a
huge room for improvement.

## References

  * [1] Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International conference on machine learning. pp. 214–223. PMLR (2017)
  * [2] Chavez, A.G., Ranieri, A., Chiarella, D., Birk, A.: Underwater vision-based gesture recognition: A robustness validation for safe human–robot interaction. IEEE Robotics & Automation Magazine 28(3), 67–78 (2021)
  * [3] Chen, D., Shen, Y., Zhang, H., Torr, P.H.: Deconstructed generation-based zero-shot model. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 295–303 (2023)
  * [4] Chen, S., Wang, W., Xia, B., Peng, Q., You, X., Zheng, F., Shao, L.: Free: Feature refinement for generalized zero-shot learning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 122–131 (2021)
  * [5] Chiarella, D., Bibuli, M., Bruzzone, G., Caccia, M., Ranieri, A., Zereik, E., Marconi, L., Cutugno, P.: A novel gesture-based language for underwater human–robot interaction. Journal of Marine Science and Engineering 6(3), 91 (2018)
  * [6] Devineau, G., Moutarde, F., Xi, W., Yang, J.: Deep learning for hand gesture recognition on skeletal data. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). pp. 106–113. IEEE (2018)
  * [7] Elfwing, S., Uchibe, E., Doya, K.: Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, 3–11 (2018)
  * [8] Gomez Chavez, A., Ranieri, A., Chiarella, D., Zereik, E., Babić, A., Birk, A.: Caddy underwater stereo-vision dataset for human–robot interaction (hri) in the context of diver activities. Journal of Marine Science and Engineering 7(1), 16 (2019)
  * [9] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. Advances in neural information processing systems 30 (2017)
  * [10] Han, Z., Fu, Z., Chen, S., Yang, J.: Contrastive embedding for generalized zero-shot learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2371–2381 (2021)
  * [11] Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)
  * [12] Lampert, C.H., Nickisch, H., Harmeling, S.: Attribute-based classification for zero-shot visual object categorization. PAMI 36(3), 453–465 (2013)
  * [13] Liang, B.: Gesture recognition using depth images. In: Proceedings of the 15th ACM on International conference on multimodal interaction. pp. 353–356 (2013)
  * [14] Madapana, N.: Zero-shot learning for gesture recognition. In: Proceedings of the 2020 international conference on multimodal interaction. pp. 754–757 (2020)
  * [15] Madapana, N., Wachs, J.: Zsgl: zero shot gestural learning. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction. pp. 331–335 (2017)
  * [16] Madapana, N., Wachs, J.: Database of gesture attributes: Zero shot learning for gesture recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). pp. 1–8. IEEE (2019)
  * [17] Madapana, N., Wachs, J.P.: Hard zero shot learning for gesture recognition. In: 2018 24th international conference on pattern recognition (ICPR). pp. 3574–3579. IEEE (2018)
  * [18] Mangalvedhekar, S., Nahar, S., Maskare, S., Mahajan, K., Bagade, A.: Underwater diver gesture recognition. In: 2023 International Conference on Electrical, Electronics, Communication and Computers (ELEXCOM). pp. 1–6. IEEE (2023)
  * [19] Mao, Q., Lee, H.Y., Tseng, H.Y., Ma, S., Yang, M.H.: Mode seeking generative adversarial networks for diverse image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1429–1437 (2019)
  * [20] Martija, M.A.M., Dumbrique, J.I.S., Naval Jr, P.C.: Underwater gesture recognition using classical computer vision and deep learning techniques (2020)
  * [21] Narayan, S., Gupta, A., Khan, F.S., Snoek, C.G.M., Shao, L.: Latent embedding feedback and discriminative features for zero-shot classification. In: ECCV. pp. 479–495 (2020)
  * [22] Paul, A., Krishnan, N.C., Munjal, P.: Semantically aligned bias reducing zero shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7056–7065 (2019)
  * [23] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [24] Romera-Paredes, B., Torr, P.H.S.: An embarrassingly simple approach to zero-shot learning. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning. pp. 2152–2161 (2015)
  * [25] Skorokhodov, I., Elhoseiny, M.: Class normalization for (continual)? generalized zero-shot learning. arXiv preprint arXiv:2006.11328 (2020)
  * [26] Sun, J.H., Ji, T.T., Zhang, S.B., Yang, J.K., Ji, G.R.: Research on the hand gesture recognition based on deep learning. In: 2018 12th International symposium on antennas, propagation and EM theory (ISAPE). pp. 1–4. IEEE (2018)
  * [27] Thomason, W., Knepper, R.A.: Recognizing unfamiliar gestures for human-robot interaction through zero-shot learning. In: 2016 International Symposium on Experimental Robotics. pp. 841–852. Springer (2017)
  * [28] Wu, B.X., Yang, C.G., Zhong, J.P.: Research on transfer learning of vision-based gesture recognition. International Journal of Automation and Computing 18(3), 422–431 (2021)
  * [29] Wu, J., Zhang, Y., Zhao, X.: A prototype-based generalized zero-shot learning framework for hand gesture recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 3435–3442. IEEE (2021)
  * [30] Xian, Y., Lampert, C.H., Schiele, B., Akata, Z.: Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence 41(9), 2251–2265 (2018)
  * [31] Xu, X., Hospedales, T., Gong, S.: Transductive zero-shot action recognition by word-vector embedding. International Journal of Computer Vision 123, 309–333 (2017)
  * [32] Yang, J., Wilson, J.P., Gupta, S.: Diver gesture recognition using deep learning for underwater human-robot interaction. In: Oceans 2019 Mts/Ieee Seattle. pp. 1–5. IEEE (2019)
  * [33] Yang, J., Wilson, J.P., Gupta, S.: Dare: Diver action recognition encoder for underwater human-robot interaction. IEEE Access (2023)
  * [34] Zhang, Y., Jiang, Y., Qi, H., Zhao, M., Wang, Y., Wang, K., Wei, F.: An underwater human–robot interaction using a visual–textual model for autonomous underwater vehicles. Sensors 23(1), 197 (2022)
  * [35] Zhou, H., Lin, D.J., Huang, T.S.: Static hand gesture recognition based on local orientation histogram feature distribution model. In: 2004 Conference on computer vision and pattern recognition workshop. pp. 161–161. IEEE (2004)