# GRLib: An Open-Source Hand Gesture Detection and Recognition Python Library

Jan Warchocki† Mikhail Vlasenko† Yke Bauke Eisma

(October 2023)

###### Abstract

Hand gesture recognition systems provide a natural way for humans to interact
with computer systems. Although various algorithms have been designed for this
task, a host of external conditions, such as poor lighting or distance from
the camera, make it difficult to create an algorithm that performs well across
a range of environments. In this work, we present GRLib: an open-source Python
library able to detect and classify static and dynamic hand gestures.
Moreover, the library can be trained on existing data for improved
classification robustness. The proposed solution utilizes a feed from an RGB
camera. The retrieved frames are then subjected to data augmentation and
passed on to MediaPipe Hands to perform hand landmark detection. The landmarks
are then classified into their respective gesture class. The library supports
dynamic hand gestures through trajectories and keyframe extraction. It was
found that the library outperforms another publicly available HGR system -
MediaPipe Solutions, on three diverse, real-world datasets. The library is
available at https://github.com/mikhail-vlasenko/grlib and can be installed
with pip.

${\dagger}$${\dagger}$footnotetext: These authors contributed equally to this
work

##  1 Introduction

In recent times, gesture recognition algorithms have provided novel ways for
interaction with computer systems. Hand gesture recognition (HGR) systems are
of special interest, as they allow for human-computer interactions [1] based
on naturalistic gesture inputs. These systems focus exclusively on the shape
and movement of the hands. Hand gesture recognition algorithms have already
been used in different tasks, including robot control and entertainment
systems [2].

Different techniques have been employed to develop HGR systems. For example,
the first HGR algorithms, such as the MIT-LED glove [3], were glove-based
systems and required the user to wear a special glove with sensors attached to
it [4, 5]. However, due to user comfort and ease of use, vision-based HGR
systems are currently becoming increasingly popular [2]. Approaches towards
developing and improving vision-based systems range from skin colour detection
[6, 7] to deep learning [8, 9, 2]. While vision-based systems are already
frequently employed in HGR systems, they present challenges, especially when
utilizing cameras that produce lower-quality images like webcams. In [10]
authors note that vision-based systems are often constrained by aspects such
as lighting sensitivity and distance to the camera. Authors of [11] and [12]
also mention complex background as a potential hindrance to accurate and
robust gesture detection.

These obstacles make it difficult to create a single algorithm that has robust
gesture detection capabilities in a wide range of use cases. As a result,
there is a lack of open-source, public libraries that allow their users to
define their own vision-based hand gestures, and then subsequently detect and
recognize them [13]. Such a library would be of interest as it streamlines the
design of systems that are controlled through hand gestures. Although
constructing a robust HGR system from scratch is difficult, in recent years,
there have been many advancements in this area that make designing such a
system more feasible. One such advancement, MediaPipe Hands [14], can be used
to construct a real-time and accurate palm detection model, which makes an
open-source hand gesture detection and recognition library possible. For
instance, in [15] authors successfully use MediaPipe Hands to build a real-
time HGR system, able to detect a set of predefined static gestures from an
RGB camera [15].

Krishna and Sinha [13] proposed Gestop \- a library for gesture control of
computer systems. The designed algorithm utilizes MediaPipe Hands to detect
hand landmarks which are then passed to a pre-trained neural network. The
library has been shown to work well on the designed task, obtaining an
accuracy of 99.12% for static, and 85% for dynamic gestures on their
validation dataset. Additionally, the library allows its users to define their
own gestures through a camera live feed. The system, however, does not support
existing datasets as a source for the training data, which limits the
flexibility of the library. A similar approach to the task of building an HGR
system has been recently used in MediaPipe Solutions [16]. Similarly to
Gestop, this solution also uses MediaPipe Hands to build the landmarks that
are then passed to a neural network to perform gesture classification. This
system also allows the user to define their own hand gestures. This library
however, does not support dynamic gestures.

In this paper, we present a hand gesture recognition library, that allows its
users to define their own gestures. More important, we aim for the library to
be effective with few training samples and to be usable with lower quality
cameras. Such a design allows users to define gestures easily, without having
to spend significant time creating a training dataset. To this end, we
introduce GRLib \- an open-source hand gesture detection and recognition
Python library. The proposed solution utilizes a feed from an RGB, general-
purpose camera. The obtained frames are first passed through an image
augmentation pipeline, which goal is to increase the detection rates of
MediaPipe Hands. The landmarks obtained from MediaPipe Hands are then passed
to a standard classifier, such as K-Nearest Neighbours, in order to recognize
the gesture. Both static and dynamic gestures are supported in the presented
system. Furthermore, the system is capable of detecting and recognizing images
in low quality video input.

This paper is structured as follows. Section 2 introduces the method used for
detecting and recognizing static gestures. Similarly, Section 3 presents the
approach for dynamic gestures. Section 4 contains an experimental evaluation
of the library, including an ablation study for the different components of
the system. Finally, the paper is concluded in Section 5 along with some
identified areas of future research.

##  2 Static Gesture Methodology

Static gesture recognition involves detecting and recognizing gestures
regardless of their position or motion. Given an input frame $X$, the goal is
to predict whether the frame contains one of $C$ gesture classes or whether it
contains only the background, i.e. when no valid gesture is shown. In this
work, static gesture detection and classification are realized through four
main components: image augmentations, MediaPipe Hands, a false-positive
filter, and a classifier. The architecture of the model is summarized in
Figure 1. The following sections aim to cover each of the components of the
system separately.

Figure 1: The architecture of the model used for recognizing static gestures.
The input image is first passed through image augmentations, that change the
brightness of the image or rotate it. MediaPipe Hands is then applied to each
augmented image separately, producing landmarks $L_{i}$. From $N$ landmarks,
the first $L_{i}$ corresponding to at least one hand being detected is
selected. The selected landmark $L$ is then passed through a false-positive
filter and then, provided $L$ represents a valid gesture, it is passed to the
classifier. The library will raise an exception if no hand was detected in any
of the $N$ landmarks.

###  2.1 Image augmentations

Data augmentation is a technique that can be employed in scenarios where
little training data is available [17, 18]. Augmentations that are commonly
used in the computer vision domain include image flipping [19], scaling [20],
rotation [21], and changing brightness [22, 18]. Typically, these
augmentations are used to increase the amount of training data [17]. In this
paper, we propose the use of image augmentations in an attempt to increase the
detection rates of MediaPipe Hands. We reflect on the actual impact of using
image augmentation on the detection rates in Section 4.

In this work, image rotation and brightness changing are used to convert the
single input frame $X$ into multiple augmented frames
$X^{\prime}_{1},X^{\prime}_{2},\ldots,X^{\prime}_{n}$. Each augmentation can
be represented with a tuple $\left(\Delta b,~{}\Delta r\right)$, where $\Delta
b$ is the update in brightness of the image and $\Delta r$ is the angle of
rotation. Updating brightness is accomplished by converting each pixel
$x_{RGB}\in X$ into its corresponding hue-saturation-value (HSV)
representation $x_{HSV}$. Then the value component $x_{V}$ of $x_{HSV}$ is
updated by $\Delta b$ using the formula:

$x_{V^{\prime}}=\max(\min(x_{V}+\Delta b,255),0)$ (1)

The resulting $x_{HSV^{\prime}}$ is converted back into the RGB representation
$x_{RGB^{\prime}}$. Applying this transformation to each pixel of $X$ results
in the augmented image $X^{\prime}_{i}$. Rotation is achieved using the
rotate_bound method from the imutils library [23].

###  2.2 MediaPipe Hands

Similarly to [13] and [16] we use MediaPipe Hands as the backbone of the
system. Each of the augmented frames $X^{\prime}_{i}$ is passed to MediaPipe
to locate hand landmarks. Per each detected hand, MediaPipe Hands returns 21
landmarks, each representing x, y, and z distances in meters to the landmark
from the hand’s approximate geometric center [14]. On top of the landmarks,
MediaPipe also returns ‘handedness‘ information, which indicates whether the
landmarks correspond to the left or the right hand [14]. The output of
MediaPipe can thus be represented as a vector $L_{i}\in\mathbb{R}^{21\cdot
3\cdot n+n}$ with $n$ being the number of detected hands. The case when $n=0$
is special as it corresponds to no hands being detected. In this case, we can
represent the output of MediaPipe as $L_{i}=\emptyset$.

Special care has to be taken when gesture classes represented with 2 hands are
allowed. It has been observed that sometimes even though 2 hands were shown,
MediaPipe Hands would only detect one, causing the vector $L_{i}$ to have
different shapes. GRLib is thus parameterized by a value num_hands, which
represents the number of hands that should be detected. Additionally, the
vector $L_{i}$ is constructed such that it first contains the landmarks of the
left hand, then the right hand. Using this formulation, if only one hand was
detected, the landmarks corresponding to the other hand are replaced with
zeros and the handedness info is replaced with -1. Assuming at least one hand
was found, the output vector $L_{i}$ will hence always have the size $21\cdot
3\cdot\texttt{num\\_hands}+\texttt{num\\_hands}$.

As the false-positive filter and the classifier only expect a single landmarks
vector $L$ on input, a single vector $L_{i}$ needs to be selected. In this
work, that is done by selecting the first landmarks vector where at least one
hand was detected ($L_{i}\neq\emptyset$). We note, however, that different,
perhaps more optimal, strategies could be implemented in an attempt to combine
the found landmarks. If no hand was detected on any of the augmented frames,
that is $\forall_{i}(L_{i}=\emptyset)$, the library raises an exception.

###  2.3 False-positive filter

The goal of the false-positive filter is to predict, given the landmarks $L$,
whether the landmarks represent one of the $C$ gesture classes. This
functionality is especially important in real-time classification, where it
cannot be guaranteed that every shown gesture will represent one of the $C$
gestures. The false-positive filter is thus also optional and can be skipped
if it is known that each image represents a valid gesture (for example during
training).

Let $\mathcal{L}^{c}=\\{L_{1}^{c},L_{2}^{c},\ldots,L_{M}^{c}\\}$ be all of the
training landmarks corresponding to the class $c$. For each class $c$, the
false-positive filter then constructs a ‘representative sample‘
$R^{c}=\frac{1}{M}(L_{1}^{c}+L_{2}^{c}+\ldots+L_{M}^{c})$. It should be noted
that using the mean for the representative sample was found to work well in
practice, however, the library could be extended to support different fusion
strategies. For example, a median could be used to deal better with outliers.
In order to then detect whether $L$ is a meaningful gesture, the false-
positive filter computes the similarity between $L$ and each representative
sample $R^{c}$. If the similarity is larger than a predefined threshold $\mu$
for any $R^{c}$, the landmark $L$ is passed to the classifier. Otherwise, the
landmark is marked as a non-valid gesture. Currently, the library supports
cosine and Euclidean similarity metrics. The value $\mu$ can be set by a user
of the library and depends on how much variability in gestures should be
permitted.

###  2.4 The classifier

The goal of the classifier is to predict, given the landmarks $L$, one of the
$C$ gesture classes. In [13] and [16] that is realized using a neural network.
In this work, however, we offer the user freedom in choosing the classifier.
Since the landmarks $L$ are guaranteed to be of the size
$64\cdot\texttt{num\\_hands}$, any standard classifier would suffice for this
task. The ablation study in Section 4 provides experimental results with using
different classifiers.

##  3 Dynamic Gesture Methodology

Dynamic hand gestures (DHG) are gestures that consider not only the hand’s
shape but also its relative movements. Real sign languages primarily consist
of dynamic gestures: for example, the 50 most important gestures of ASL are
all dynamic [24]. Since people already use dynamic gestures to communicate
with each other, recognizing them can be useful in a variety of fields,
including human-computer interaction. In this section, we will explore the
challenges and implemented approaches that were involved in recognizing
dynamic hand gestures.

###  3.1 Trajectories

As described before, the hand’s trajectory is one of the primary aspects of a
dynamic gesture. Thus, handling it in different ways may have a great impact
on the classifier’s performance.

One relatively modern, yet common way of trajectory recognition is using Long
Short-Term Memory (LSTM) [25] [26]. With such an approach, a trained LSTM
model receives a list of the hand’s relative movements and predicts either a
trajectory class or a gesture that corresponds to such trajectory. Although
this approach can theoretically be scaled to recognize patterns of large size
and complexity, it requires a significant amount of labeled training data [27]
for both: the actual trajectories that have to be predicted, as well as pre-
training. The focus of GRLib, however, is to allow few-shot or even single-
shot learning. Consequently, the LSTM approach was deemed unsuitable for the
task.

The sample efficient approach, that is implemented in GRLib was inspired by
[28]. The paper treats the trajectory as a series of general directions for
each axis: positive movement, stationary, and negative movement. For example,
the movement of a hand up and to the right, and then down and right would be
encoded as ((+, +), (-, +)). Such a method has an important advantage - it
generalizes the trajectory, stripping it of noise. Thus, the gesture’s
movement can be learned from a single sample.

####  3.1.1 Key Frame Extraction

To extract the described trajectory, a set amount of key frames is determined.
Using them, it is possible to construct a trajectory that is not only of fixed
length but also reduces the potential perceived jitters of the hand position
that may occur as the hand is moving relative to the camera.

To extract representative key frames, we first remove outliers. A frame $i$
(denoted $f_{i}$) is considered an outlier if the following holds.

$\operatorname{dist}(f_{i-1},f_{i+1})<\min[\operatorname{dist}(f_{i-1},f_{i}),\operatorname{dist}(f_{i},f_{i+1})],$
(2)

where $\operatorname{dist}(f_{i},f_{j})$ is the distance between the hand in
frame $i$ to hand in frame $j$. The key frames are determined by evenly
distributing the cumulative distance covered by the hand movement between two
sequential frames, always including the first and the last frames.

###  3.2 Gesture start detection

Detecting dynamic gestures in real time poses an additional challenge: without
special markers, it is impossible to know that a gesture has started, before
it is complete, as the hand may perform only the beginning of the sequence. A
possible solution to this problem is through the use of gesture start markers,
as presented by [13]. The requirement for the markers imposes a significant
limitation: apart from performing a gesture, the user has to use another means
of input to indicate its start and end. This can interrupt the fluidity of the
interaction and further complicate intuitive usage. It was therefore decided
to adapt the library to work without the markers.

####  3.2.1 Prediction Candidates

The chosen approach keeps a list of so-called candidates: known gestures that
can currently be shown, and their respective progress. Candidates are added to
the dedicated list when a start of any known dynamic gesture is possible.
Specifically, when the hand’s shape matches the starting shape of a dynamic
gesture. If the hand shape matches the starting shape of multiple classes, a
candidate is created for every one of them. Every set amount of frames, each
of the current candidates is checked: the new position of the hand is compared
with the previous position, and if the hand’s actual trajectory aligns with
the trajectory expected by a candidate, the latter is kept and updated.
Otherwise, the candidate is dropped as irrelevant. When one of the candidates
successfully reaches its end, the prediction is made and the others are
cleared. A diagram of the algorithm for one frame is presented in Figure 2. As
a result, the algorithm continuously evaluates possible paths to a complete
dynamic gesture.

Figure 2: Schematic representation of the candidate update process. First, a
classifier is used to determine whether the landmarks in the current frame
represent the starting shape of a gesture. If so, a new candidate is created
for each matching gesture class. Then, for each candidate, it is checked if
the current hand position aligns with the expected movement for the class
trajectory. If so, the candidate’s trajectory is updated, and in case the
trajectory is complete, the prediction is made. Otherwise, the candidate gets
put back into the candidate storage.

##  4 Results

The following section assesses the ability of the library to perform gesture
detection and recognition. The topics of static and dynamic gestures are
handled separately. The analysis for static gestures is done through the means
of an ablation study that shows the impact of each component of the system on
the overall performance. We conducted an ablation study for evaluation because
the absence of similar systems in the literature makes direct comparisons
challenging [13]. In the literature, only two similar libraries have been
identified: MediaPipe Solutions [16] and Gestop [13]. Gestop does not however
support existing datasets as a source of the training data. On top of the
ablation study, we also compare the performance of GRLib with MediaPipe
Solutions.

###  4.1 Static gestures

In this section, we explore the library’s capabilities in detecting and
recognizing static gestures. This is done through the means of an ablation
study. Additionally, the performance of the library is measured against
MediaPipe Solutions.

Datasets. Three datasets of varying complexity were chosen for the assessment
of static HGR capabilities. First, the American Sign Language dataset [29],
which contains 3,000 images per English letter, all of which share a similar
background. The second dataset is a subsampled version of HaGRID [30] with 100
images per class. Gestures for 18 classes are performed by different people in
various environments. The third dataset is taken from the Kenyan Sign Language
Classification Challenge [31], it contains 9 static classes of gestures
involving one or two hands. The complexity is increased by the quality of the
images: on some of them, the hands are partly blurred out, while fingers
and/or forearms may be out of frame. Additionally, the Kenyan competition
dataset is the only tested dataset where gesture classes requiring two hands
are present.

Evaluation metric. As in this work an emphasis is put on the detection rates,
a metric combining both detection rate and classification accuracy needs to be
defined. Let the training dataset have $N$ images, of which in only $M$ a
gesture was detected. The detection rate is then defined as $r=\frac{M}{N}$.
Given the classification accuracy $c$, the final metric $m$ is calculated as
$m=c\cdot r=c\cdot\frac{M}{N}$. Thus, a better model is expected to have a
higher value of the metric $m$.

####  4.1.1 Image augmentations

In this section, the impact of using image augmentations is shown on the
detection rate $r$ and the combined metric $m$. To show the performance of the
model in low-data regimes, the model is trained through an $n$-shot learning
procedure, where $n$ images per each class are subsampled from the training
dataset at random. The model is then evaluated on the remaining images from
the training dataset. The procedure is repeated 10 times for each $n$, at each
time with a different, randomly subsampled set.

Table 1: Four different settings for the image augmentations that were used for testing. Setting 1 is the baseline for comparison when no augmentations are done. Setting 2 modifies only the brightness of the image while setting 3 only the rotation of the image. Setting 4 is a combination of both brightness and rotation. Settings | Stages $(\Delta b,\Delta r)$
---|---
Stage 1. | Stage 2. | Stage 3. | Stage 4. | Stage 5. | Stage 6. | Stage 7.
Setting 1. | (0, 0) |  |  |  |  |  |
Setting 2. | (0, 0) | (30, 0) | (60, 0) |  |  |  |
Setting 3. | (0, 0) | (0, -15) | (0, 15) | (0, -30) | (0, 30) |  |
Setting 4. | (0, 0) | (30, 0) | (60, 0) | (0, -15) | (0, 15) | (0, -30) | (0, 30)

Figure 3: Performance of GRLib on ASL [29], HaGRID [30], and Kenyan [31]
datasets. The x-axis represents the number $n$, which is the number of
training samples available for each class. The y-axis represents the metric
$m$, obtained by multiplying the accuracy score $a$ with the recognition rate
$r$. The recognition rates for each augmentation setting are shown in brackets
in the legend. The width of each line corresponds to two standard deviations.

Figure 4: Performance of GRLib on ASL [29], HaGRID [30], and Kenyan [31]
datasets using different classifiers. The x-axis represents the number $n$,
which is the number of training samples available for each class. The y-axis
represents the metric $m$, obtained by multiplying the accuracy score $a$ with
the recognition rate $r$. Performance of both, the best model found, and the
K-Nearest Neighbours classifier used in Section 4.1.1 are shown. Furthermore,
the performance of MediaPipe Solutions [16] is reported. The width of each
line corresponds to two standard deviations obtained by repeating the
experiment 10 times.

The testing procedure is done on the three datasets, using 4 different image
augmentation settings. For a baseline comparison, one of the settings contains
no augmentations. The other three settings modify only the brightness, only
the rotation, or a combination of both. The exact overview of conditions can
be found in Table 1. As for the classifier, K-Nearest Neighbours (KNN) with
$k=5$ was chosen. The inputs to the classifier are only the detected
landmarks, without the handedness info. This was done as the dimensionality of
the handedness info is orders of magnitude larger than the landmarks. Figure 3
shows the results of the experiment.

The figure demonstrates the number of training samples per class $n$ on the
x-axis and the average metric $m$ on the y-axis. As the sizes of each dataset
are different, different values of $n$ were chosen for each one of them. The
difference between the top and the bottom of each line corresponds to two
standard deviations obtained by repeating the experiment 10 times, which hence
represents the 68% confidence interval. The recognition rates $r$ for each of
the image augmentations settings are shown in brackets in the legend. As can
be observed from the figure, introducing image augmentations increases the
recognition rates $r$ with respect to the baseline on all three datasets.
Consequently, this usually also leads to an increase in the evaluation metric
$m$, although the results for the HaGRID dataset are inconclusive due to the
large variance in the results.

###  4.2 The classifier

The following section aims to show how the performance of the library changes
with different classifiers. Similarly to the experiments on image
augmentations, the models are trained on $n$ images per class and tested on
the remainder. The procedure is repeated 10 times each time randomly sampling
the $n$ images. Training and testing are done using the landmarks obtained by
using Setting 4 for the image augmentations. This setting was chosen as it
exhibited the highest recognition rates for all three datasets. The best model
was then the one that obtained the highest average metric $m$ over the 10
repetitions over all of the values of $n$. To report the final performance,
the best-found model was then trained and evaluated from scratch for each $n$,
repeating the experiments 10 times.

Four different classifiers are evaluated using this procedure: K-Nearest
Neighbours (KNN), Support Vector Machine (SVC), Decision Tree Classifier, and
Logistic Regression. Furthermore, for each model, the best combination of its
parameters would be attempted to be found. The exact list of parameters
searched for and the corresponding experimental result can be found in the
repository of the project. Additionally, we report the performance of
MediaPipe Solutions using the same experimental setup. This is done to compare
the results using different techniques. The best-found classifier is also
further compared to the K-Nearest Neighbour classifier used in the previous
section. The comparison can be seen in Figure 4.

In all 3 cases, the SVC classifier with the radial basis function (RBF) kernel
was selected as the best model. For both the ASL and HaGRID datasets, the best
model had the parameter $C=20$. The model for the Kenyan dataset had $C=5$. As
can be observed from Figure 4, the choice of the model influences the
classification performance, especially in low-data regimes. This is most
visible in the HaGRID dataset for $n=5$, where the SVC model obtained $m=0.62$
compared to KNN’s $m=0.39$. This is perhaps expected as the support vector
machine classifier attempts to separate the classes as much as possible.

Furthermore, the performance of the model is compared on the three datasets
against MediaPipe Solutions [16]. As can be observed, the recognition rates,
visible in the legend in Figure 4, are much lower for MediaPipe Solutions. The
lower recognition rates also made it impossible to evaluate MediaPipe
Solutions on the largest values of $n$. There would not be enough images to
sample 1000, 80, and 500 per class from the three datasets. Overall, it can be
observed that GRLib outperforms MediaPipe Solutions on all tested datasets.

###  4.3 Dynamic gestures

This section outlines the library’s capability for real-time identification of
dynamic gestures. Our algorithm is tailored for both sample efficiency and
online operation. During inference, it continuously processes incoming frames
without requiring markers for the start and end of gestures, thus generating a
continuous stream of predictions.

To test the performance of the proposed solution, we create a custom dynamic
gesture dataset. The dataset contains five training samples for 9 gesture
classes and evaluates the model in an online setting. The classes were chosen
in the following way:

  * •

2 gestures (palm-left and palm-right) differ exclusively in the direction of
the hand motion.

  * •

2 gestures (right-fist-to-palm and right-palm-to-fist) share a simple
trajectory with other gestures, but can be distinguished by the start and end
hand shapes.

  * •

3 gestures (fist-up, forward, backward) share the start hand shape but have
different end shapes and go in 3 different directions, including the depth
dimension.

  * •

2 gestures (palm-circle, palm-infinity) have a lengthy and complex trajectory
with a constant hand shape - open palm.

With this dataset, we aim to highlight the most important characteristics of
the DHG recognition algorithm.

For inference, we use multiple sequences of various nature. One of the
sequences repeats simple gestures three time each, 3 others present a sequence
of gestures one flowing into another. In about half of the sequences, gestures
are interleaved with background actions that do not correspond to any gesture
class. All sequences contain more than 3 presented gestures to simulate the
online evaluation, rather than feeding one gesture at a time.

Example hand movements from the dataset are presented in Figure 5. We publish
the dataset alongside the library.

Figure 5: The 3d hand trajectory from one of the flowing sequences. The color
indicates time, and the marker size indicates the z value (closer to the
camera is bigger). Near the frame number, the ground truth classes are
labeled.

The library has shown good performance with a limited amount of training data.
The overall gesture recognition recall was 79%, despite the challenges
associated with the sparsity of the training data. However, we observe a
tendency of the system to identify the same gesture more than once
consecutively. This happens with gestures that have a trivial trajectory -
moving the hand in one direction. Then, as the hand moves, the library
predicts this gesture. In case the hand continues to move in the same
direction after the prediction, the prediction can be repeated.

Another experienced problem was with gestures containing a relatively complex
trajectory - circle and infinity are not recognized at all. Due to the way the
trajectories are encoded - with the key frames and directional quantization,
the complex trajectories are inherently difficult to correctly encode, let
alone recognize.

We also noticed that the depth estimation of MediaPipe is problematic, leading
to the gestures ”forward” and ”backward” being frequently falsely detected.
Therefore, we reran the experiments without introducing these gestures. Not
accounting for repetitions, we observe 100% detection recall on gestures other
than ”circle” and ”infinity”, and 0% recall on these two. As for precision, we
get about 60% overall, but a much higher about 95% when not considering
additional predictions of the same correct gesture.

Not accounting for landmark extraction, dynamic gesture detection is able to
run at $2083\pm 13$ frames per second on a laptop with 2.6 GHz 6-Core Intel
Core i7. Thus, it is demonstrated to be highly suitable for real-time
applications.

Despite the aforementioned double-detection issue, the results indicate that
our hand gesture recognition library shows promising capabilities for real-
time, online applications, even with a limited number of training samples.

##  5 Conclusion and Future Work

Hand gesture recognition (HGR) systems provide a natural way of human-computer
interaction. Scientific studies that try to improve robustness and accuracy of
these systems vary from skin colour detection [6, 7] to applying machine
learning for gesture recognition [8, 9, 2]. However, existing systems suffer
in scenarios where the background is complex or where the lighting is
imperfect [10, 11, 12]. In this paper, we identified that there is a lack of
publicly available libraries that are able to work under difficult conditions
and that allow users to define their own gestures, which in turn can be
trained on prerecorded and publicly available gesture data. [13].

Work by Krishna and Sinha [13] introduced Gestop, a library that allows the
user to define their own gestures to control computer systems. Both static and
dynamic gestures are supported by the system. Their library, however, does not
support gestures to be defined from already existing datasets. MediaPipe
Solution offers a similar system, where the user can also define their own
gestures. This package however does not support dynamic gestures.

In this work, we presented GRLib \- an open-source, Python hand gesture
recognition library. The proposed solution uses MediaPipe Hands to detect
landmarks, which are then passed to a classifier chosen by the user of the
library. The landmark detection is further enhanced by means of image
augmentations. A false-positive filter is introduced, which helps in scenarios
where it can not be guaranteed that a valid hand gesture is shown. A method
for recognizing dynamic gestures is also implemented.

The system is evaluated on three real-world datasets: American Sign Language
(ASL) [29], Kenyan [31], and HaGRID [30]. We find that the image augmentations
greatly increase the detection rates of MediaPipe Hands on all three datasets.
Furthermore, we find that the choice of the classifier improves the
performance of the library in a few-shot learning scenario. Moreover, it is
also found that the system outperforms MediaPipe Solutions on all three
datasets.

In the future, a user test could be performed. The participants could be
instructed to apply the library to different problems including static and
dynamic gestures. Such experiments could for example include building a
desktop application that uses a laptop camera or controlling robot movements
with gestures. Dependent measures may vary from objective measures, for
example task completion time, to subjective questionnaires regarding ease of
use. This feedback could be used to improve the usability of the proposed
library and augment it with useful components. Additionally, the functional
component of the library could be improved. More image augmentations, such as
changing contrast and saturation [32] could be included. The library can also
be extended to work with more than 2-hand gestures.

Dynamic gesture recognition can also be improved in the future. We have
observed that the current implementation struggles with long or complex
trajectories. This is likely caused by the way key frame extraction is done.
For example, for an 8-shaped trajectory, it is unlikely that a small number of
key frames will approximate the true hand movement well. A different approach
towards selecting key frames may be able to mitigate this issue. For example,
an algorithm may attempt to put the key frames in the “corners” of a
trajectory. Additionally, during inference, the candidate update process is
done in equal intervals, while it could take into account past hand movement
to update candidates at better moments.

## References

  * [1] Munir Oudah, Ali Al-Naji and Javaan Chahl  “Hand Gesture Recognition Based on Computer Vision: A Review of Techniques”  In _Journal of Imaging_ 6.8, 2020  DOI: 10.3390/jimaging6080073
  * [2] GRS Murthy and RS Jadon  “A review of vision based hand gestures recognition”  In _International Journal of Information Technology and Knowledge Management_ 2.2, 2009, pp. 405–410
  * [3] D.J. Sturman and D. Zeltzer  “A survey of glove-based input”  In _IEEE Computer Graphics and Applications_ 14.1, 1994, pp. 30–39  DOI: 10.1109/38.250916
  * [4] Prashan Premaratne  “Historical Development of Hand Gesture Recognition”  In _Human Computer Interaction Using Hand Gestures_ Singapore: Springer Singapore, 2014, pp. 5–29  DOI: 10.1007/978-981-4585-69-9˙2
  * [5] Laura Dipietro, Angelo M. Sabatini and Paolo Dario  “A Survey of Glove-Based Systems and Their Applications”  In _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 38.4, 2008, pp. 461–482  DOI: 10.1109/TSMCC.2008.923862
  * [6] J. Brand and J.S. Mason  “A comparative assessment of three approaches to pixel-level human skin-detection”  In _Proceedings 15th International Conference on Pattern Recognition. ICPR-2000_ 1, 2000, pp. 1056–1059 vol.1  DOI: 10.1109/ICPR.2000.905653
  * [7] “A survey of skin-color modeling and detection methods”  In _Pattern Recognition_ 40.3, 2007, pp. 1106–1122  DOI: https://doi.org/10.1016/j.patcog.2006.06.010
  * [8] Gongfa Li et al.  “Hand gesture recognition based on convolution neural network”  In _Cluster Computing_ 22.2, 2019, pp. 2719–2729  DOI: 10.1007/s10586-017-1435-x
  * [9] Sungho Shin and Wonyong Sung  “Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks”  In _2016 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 2016, pp. 2274–2277  DOI: 10.1109/ISCAS.2016.7539037
  * [10] Lingchen Chen, Feng Wang, Hui Deng and Kaifan Ji  “A Survey on Hand Gesture Recognition”  In _2013 International Conference on Computer Sciences and Applications_ , 2013, pp. 313–316  DOI: 10.1109/CSA.2013.79
  * [11] Yikai Fang, Kongqiao Wang, Jian Cheng and Hanqing Lu  “A Real-Time Hand Gesture Recognition Method”  In _2007 IEEE International Conference on Multimedia and Expo_ , 2007, pp. 995–998  DOI: 10.1109/ICME.2007.4284820
  * [12] Zhou Ren, Jingjing Meng, Junsong Yuan and Zhengyou Zhang  “Robust Hand Gesture Recognition with Kinect Sensor”, MM ’11  Scottsdale, Arizona, USA: Association for Computing Machinery, 2011, pp. 759–760  DOI: 10.1145/2072298.2072443
  * [13] Sriram Krishna and Nishant Sinha  “Gestop: Customizable gesture control of computer systems”  In _Proceedings of the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD)_, 2021, pp. 405–409
  * [14] Fan Zhang et al.  “MediaPipe Hands: On-device Real-time Hand Tracking”  In _CoRR_ abs/2006.10214, 2020  arXiv: https://arxiv.org/abs/2006.10214
  * [15] George Sung et al.  “On-device Real-time Hand Gesture Recognition”  arXiv, 2021  DOI: 10.48550/ARXIV.2111.00038
  * [16] Google  “MediaPipe Solutions”, 2022  URL: https://developers.google.com/mediapipe/solutions
  * [17] Simone Antonelli et al.  “Few-Shot Object Detection: A Survey”  In _ACM Comput. Surv._ 54.11s New York, NY, USA: Association for Computing Machinery, 2022  DOI: 10.1145/3519022
  * [18] Yaqing Wang, Quanming Yao, James T. Kwok and Lionel M. Ni  “Generalizing from a Few Examples: A Survey on Few-Shot Learning”  In _ACM Comput. Surv._ 53.3 New York, NY, USA: Association for Computing Machinery, 2020  DOI: 10.1145/3386252
  * [19] Hang Qi, Matthew Brown and David G Lowe  “Low-shot learning with imprinted weights”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5822–5830
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [21] Adam Santoro et al.  “Meta-Learning with Memory-Augmented Neural Networks”  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16  New York, NY, USA: JMLR.org, 2016, pp. 1842–1850
  * [22] Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran and Stefano Soatto  “A Baseline for Few-Shot Image Classification”  In _CoRR_ abs/1909.02729, 2019  arXiv: http://arxiv.org/abs/1909.02729
  * [23] PyImageSearch  “imutils”, 2015  URL: https://github.com/PyImageSearch/imutils
  * [24] “HandSpeak”, 2023  URL: https://www.handspeak.com/
  * [25] Safa Ameur, Anouar Ben Khalifa and Med Salim Bouhlel  “A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion”  In _Entertainment Computing_ 35, 2020, pp. 100373  DOI: https://doi.org/10.1016/j.entcom.2020.100373
  * [26] Aamrah Ikram and Yue Liu  “Skeleton Based Dynamic Hand Gesture Recognition Using LSTM and CNN”  In _2020 2nd International Conference on Image Processing and Machine Vision_ , IPMV 2020  Bangkok, Thailand: Association for Computing Machinery, 2020, pp. 63–68  DOI: 10.1145/3421558.3421568
  * [27] Shuo Feng, Huiyu Zhou and Hongbiao Dong  “Using deep neural network with small dataset to predict material defects”  In _Materials & Design_ 162 Elsevier, 2019, pp. 300–310
  * [28] Jong-Sung Kim, Won Jang and Zeungnam Bien  “A dynamic gesture recognition system for the Korean sign language (KSL)”  In _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ 26.2, 1996, pp. 354–359  DOI: 10.1109/3477.485888
  * [29] Akash Nagaraj  “ASL Alphabet”  Kaggle, 2018  DOI: 10.34740/KAGGLE/DSV/29550
  * [30] Alexander Kapitanov, Andrew Makhlyarchuk and Karina Kvanchiani  “HaGRID - HAnd Gesture Recognition Image Dataset”  arXiv, 2022  DOI: 10.48550/ARXIV.2206.08219
  * [31] zindi.africa  “Kenyan Sign Language Classification Challenge”, 2022  URL: https://zindi.africa/competitions/kenyan-sign-language-classification-challenge
  * [32] Valeska Uchôa et al.  “Data Augmentation for Face Recognition with CNN Transfer Learning”  In _2020 International Conference on Systems, Signals and Image Processing (IWSSIP)_ , 2020, pp. 143–148  DOI: 10.1109/IWSSIP48289.2020.9145453


# Ghost-free High Dynamic Range Imaging via Hybrid CNN-Transformer and
Structure Tensor

Yu Yuan†, Jiaqi Wu†, Zhongliang Jing, , Henry Leung, , Han Pan  Manuscript
received September 6, 2022. This work was supported by the National Natural
Science Foundation of China (Grant Nos. 61175028, 61603249, 61673262)
(Corresponding authors: Zhongliang Jing, and $\dagger$ indicates equal
contribution.) Yu Yuan, Zhongliang Jing, and Han Pan are with the School of
Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai 200240,
China<EMAIL_ADDRESS><EMAIL_ADDRESS>hanpan@sjtu.edu.cn).Jiaqi Wu is with the School of Information and
Communication Engineering, University of Electronic Science and Technology of
China (UESTC), Chengdu 610097, China (e-mail: dl2wjq@gmail.com). Henry Leung
is with the Department of Electrical and Computer Engineering, University of
Calgary, Calgary, AB T2N 1N4, Canada (e-mail: leungh@ucalgary.ca).

###### Abstract

Eliminating ghosting artifacts due to moving objects is a challenging problem
in high dynamic range (HDR) imaging. In this letter, we present a hybrid model
consisting of a convolutional encoder and a Transformer decoder to generate
ghost-free HDR images. In the encoder, a context aggregation network and non-
local attention block are adopted to optimize multi-scale features and capture
both global and local dependencies of multiple low dynamic range (LDR) images.
The decoder based on Swin Transformer is utilized to improve the
reconstruction capability of the proposed model. Motivated by the phenomenal
difference between the presence and absence of artifacts under the field of
structure tensor (ST), we integrate the ST information of LDR images as
auxiliary inputs of the network and use ST loss to further constrain
artifacts. Different from previous approaches, our network is capable of
processing an arbitrary number of input LDR images. Qualitative and
quantitative experiments demonstrate the effectiveness of the proposed method
by comparing it with existing state-of-the-art HDR deghosting models. Codes
are available at https://github.com/pandayuanyu/HSTHdr.

###### Index Terms:

High dynamic range deghosting, hybrid, CNN-Transformer, structure tensor.

##  I Introduction

The dynamic range of natural scenes can be very wide, spanning several orders
of magnitude from dazzling sunlight to faint starlight. Most camera systems
can only capture a very narrow dynamic range, which is often reflected in the
fact that we get a picture with over-exposed or under-exposed regions. To
address aforesaid limitation, many high dynamic range (HDR) imaging techniques
have been developed. One way is to use a specialized camera to capture HDR
images [1, 2]. However, the cost of using these devices is high for general
consumers. Another common strategy is to use an exposure bracketing with
different exposure values (EV) to compose the final HDR image [3, 4, 5]. These
approaches work well for static scenes, however, when the scenes have moving
objects or the camera is slightly displaced, these methods suffer from severe
ghosting artifacts [see Fig. 2(b)].

Figure 1: The input LDR images with large motions are shown in (a). The first
two images (EV -2.0 and EV 0.0) are fed into our network to obtain the HDR
image shown in the top of (b), this image continues to be fused with the
remaining LDR image (EV +2.0) to obtain the result in the bottom of (b). Other
methods require three LDR images as inputs, and to compare with our dual-input
network, we duplicate the second image to replace the original third LDR
image.

Many conventional methods have been proposed to generate ghost-free HDR
images. They can be roughly divided into alignment before merging methods and
motion rejection methods. The former approach aligns the LDR images before
merging them. [6, 7, 8] used optical flow to estimate the motions. These
optical flow-based methods rely heavily on the accuracy of the estimation.
Many studies use the PatchMatch algorithm [9] to pre-align LDR images. Sen et
al. [10] proposed a patch-based energy minimization method and used a joint
optimization strategy to integrate the alignment and HDR reconstruction
processes. Hu et al. [11] optimized the aligned LDR images based on intensity
mapping function and gradient consistencies on the transformed domain.
However, these methods are not very effective for large displacements and have
a high computational cost. For the motion rejection methods, they first align
the input LDR images at the overall level, and then reject unaligned pixels.
Gallo et al. [12] proposed to predict the color in various exposures and
compare it with the original values to detect motions. Khan et al. [13]
proposed to assign fusion weights by calculating the probability of a given
pixel belonging to the background. Heo et al. [14] proposed to detect
unaligned regions by calculating the joint probability density between LDR
images. Lee et al. [15] employed rank minimization to generate ghost-free
images. A common drawback of these methods is that they ignore useful
information about the abandoned pixels.

Because of its strong representational capacity, a myriad of deep learning-
based HDR deghosting methods have emerged in recent years. Kalantari et al.
[16] introduced the first multi-frame HDR imaging method for dynamic scenes
based on deep learning. This method feeds the input LDR images into the fusion
network after aligning them by optical flow. Wu et al. [17] used homography to
globally align the LDR inputs and then fed them into a UNet-based network for
local alignment and fusion. Yan et al. [18] used a attention mechanism to
suppress the non-aligned regions between the input LDR images. Niu et al. [19]
proposed the first GAN-based method for HDR deghosting.

Existing HDR deghosting architectures are mostly tailored for fixed-number
(e.g., three) input LDR images and use one of them as the reference image.
This setting greatly limits the practicality of these methods, since we can
only obtain 2 LDR images with different exposures or more than three in many
scenarios. Meanwhile, ghosting artifacts are more pronounced in the gradient
field [20], but are strongly affected by noise. It is shown that structure
tensor [23, 24] is able to reflect the textures of images while remaining
largely free from noise interference. These aforementioned findings inspire us
to develop a new network capable of processing an arbitrary number of input
images, and introduce the structural tensor information of images into our
network to better constrain ghosting artifacts.

The main contributions of our works are summarized as follows.

  1. 1.

We develop a hybrid CNN-Transformer network to process dual-input LDR images
for HDR deghosting. A contextual aggregation network and non-local channel
attention block are utilized in the encoder to extract multi-scale features
and establish local and global dependencies within and between the source
images, respectively. A sequential fusion strategy is explored to accommodate
an arbitrary number of inputs.

  2. 2.

We aggregate the structural tensor information of LDR images into the network
and introduce the structural tensor loss to further constrain the ghosting
artifacts.

  3. 3.

We conduct comparative experiments on publicly available datasets. Qualitative
and quantitative evaluations validate the superiority of our method over
conventional methods.

##  II The proposed method

Figure 2: The difference between gradient and structure tensor. Figure (a)
shows the three LDR images of the input. Figure (b) gives the fusion result
obtained by Mertens et al., which shows the typical ghosting patterns.  Figure
3: Overview of the proposed HDR deghosting framework.

This section describes the proposed network based on hybrid CNN-Transformer
and structure tensor for HDR deghosting. Our goal is to convert an LDR image
sequence $\left\\{I_{1},I_{2},\ldots,I_{k}\right\\}$ into a high-quality HDR
image, $H$. In our network, we input only two images (e.g. $I_{1},I_{2}$) at a
time and use the resulting image as the input for an arbitrary number of input
images in subsequent tasks. In [16], images in the LDR domain are used to
detect noise and saturated regions, while images in the HDR domain facilitate
detection of motions. HDR image $H_{k}$ can be obtained from LDR images by the
following formula:

$H_{k}=\frac{I_{k}^{\gamma}}{t_{k}},\quad k=1,2$ (1)

where $t_{k}$ denotes the exposure time of the $k$-th image $I_{k}$. $\gamma$
is a gamma correction parameter, which is set to 2.2. Following [16], we
concatenate $I_{k}$ and $H_{k}$ along the channel dimension to obtain a
6-channel input $X_{k}=\left\\{I_{k},H_{k},i=1,2\right\\}$. Therefore, the
process of obtaining HDR images can be expressed as:

$H=\mathbf{M}\left(X_{1},X_{2};\theta\right)$ (2)

where $\mathbf{M}(\cdot)$ represents the network, and $\theta$ denotes the
network parameters.

Since artifacts are more significant in the gradient domain [21, 22] [see the
fourth and fifth columns of Fig. 2], and using gradient information can
effectively mitigate ghosting artifacts. However, we find that the gradient of
images is sensitive to noise, which will introduce many interference [see Fig.
2(d)]. As shown in Fig. 2(e), the detection of flat areas in the structure
tensor [23, 24, 25] allows us to obtain structural information about images
while reducing the effects of noise. Comparing the gradient maps with the
structure tensor maps, we find that the results obtained by the structure
tensor have clearer textures, which shows the potential of the structure
tensor for deghosting. In summary, we concatenate the inputs of the network
with structural tensor information $S_{k}$, and get the progressive 7-channel
input, $X_{k}=\left\\{I_{k},H_{k},S_{k},k=1,2\right\\}$.

###  II-A Proposed Framework for HDR Deghosting

As illustrated in Fig. 3, our network adopts a hybrid CNN-Transformer
structure, which is mainly composed of a CNN encoder and a Transformer
decoder. We set the second image $X_{2}$ as the reference. The CNN encoder is
responsible for extracting multi-scale features of the inputs and learning
motions between $X_{2}$ and $X_{1}$. The Transformer decoder is used to
reconstruct ghost-free HDR images.

We use the context aggregation network (CAN) [26] with different dilation
rates (here we set 1, 2, 4) to obtain multi-scale 64-channel features
$\Phi_{1}$ and $\Phi_{2}$ without losing spatial resolution. Considering the
degradation of the spatial properties of the deep features extracted by CAN,
we only utilize a channel attention block. By extending the non-local
attention mechanism [27] to multiple heterogeneous inputs, we propose a cross
channel attention module (CCAM) to learn the difference between $X_{1}$ and
the reference $X_{2}$ by establishing associations between $\Phi_{1}$ and
$\Phi_{2}$ in latent space. The $\Phi_{2}$ and $\Phi_{1}^{channel}$ obtained
by CCAM are concatenated and then fed into the Transformer decoder.

Zhao et al. [29] demonstrates that the hybrid Transformer-CNN architecture has
a better performance in terms of both model capacity and computational
complexity. Therefore, we use the Swin transformer block proposed in [28] as
part of the decoder. Specifically, three successive Swin Transformer blocks
are utilized and concatenated to obtain features of different depths. It is
worth noting that a skip connection between the encoder and decoder is added
to prevent network degradation.

Figure 4: Proposed sequential fusion strategy in our approach.

###  II-B Loss Function

Due to the low dynamic range of our regular consumer-grade displays, HDR
format images need to be tonemapped to have a better visual experience on
these monitors. According to [16], $\mu$-law is used for tonemapping:

$T(H)=\frac{\log(1+\mu H)}{\log(1+\mu)}$ (3)

where $\mu$ is set to 5000. $T(H)$ is the tonemapped result of HDR image $H$.

Our loss function is defined as:

$\mathcal{L}=\|T(H)-T(GT)\|_{2}^{2}+\lambda\left\|T\left(H_{ST}\right)-T\left(GT_{ST}\right)\right\|_{1}$
(4)

where $GT$ denotes the ground truth, $H_{ST}$ represents the structure tensor
map of $H$, and $GT_{ST}$ is the structure tensor map of the ground truth.
$\|\cdot\|_{2}^{2}$ denotes MSE loss, and $\|\cdot\|_{1}$ refers to L1 loss.
$\lambda$ indicates the weight of structure tensor loss.

###  II-C Dealing with Arbitrary Number of Inputs

As shown in Fig. 4, for an LDR sequence with an arbitrary number of inputs, we
initially take the image with middle exposure value (EV 0.0) in the sequence
as the reference and fed it into our proposed dual-input network together with
the LDR image of lower exposure value. The intermediate image obtained by our
network is used as the new reference and is fused with another image of higher
exposure value. We observe that the strategy of fusing the reference image
with the darker image first and then with the brighter image has a better
performance. It is probably because the network tends to give more weight to
pixels with larger luminance values.

##  III Experiments

In this section, we perform extensive experiments to validate the
effectiveness of our proposed model. We use the dataset in [16], specifically
by splitting a set of three LDR images into two sets that share the same
reference image. We generate 150 samples of training data and 30 samples for
validation. Our network is trained with an AdamW optimizer. The initial
learning rate is set as 1$e$-4 and decreases to 1$e$-8 with the cosine
annealing strategy. we select $\lambda$ = 1$e$-2 as the weight of structure
tensor loss. Random seeds are fixed during all training sessions. We train for
5000 epochs with batch size 4 and set the input image resolution as 256×256
pixels. All experiments are performed on one NVIDIA Geforce RTX 3090 GPU and
Intel Core i9-10900k CPU @ 3.70GHz. Our network is programmed on PyTorch.

Four metrics are utilized to quantify the fusion results, including peak
signal-to-noise ratio (PSNR) [30] after tonemapping using $\mu$-law
(PSNR-$\mu$) and in the linear domain (PSNR-$l$). We also use a quality metric
developed specifically for the multi-exposure fusion of dynamic scenes MEF-
SSIMd [31]. In addition, to evaluate the subjective quality of HDR images, we
use neural image assessment (NIMA)[32].

TABLE I: Quantitative results of ablation studies on 9 validation image pairs. Values in bold indicate the best results. | PSNR-$l$ (dB) | PSNR-$\mu$ (dB) | MEF-SSIMd | NIMA
---|---|---|---|---
w/o STB | 39.7832 | 36.2759 | 0.9182 | 4.3445
w/o ST | 36.0576 | 32.8702 | 0.8901 | 4.0742
w/o CCAM | 37.3947 | 33.7846 | 0.8997 | 4.1970
Ours | 39.9299 | 37.3522 | 0.9196 | 4.3983
Figure 5: Subjective comparisons of ablation studies. (a) w/o Swin Transformer
Blocks; (b) w/o Structure Tensor; (c) w/o CCAM; (d) ours; (e) GT.

###  III-A Ablation Studies

Swin Transformer Blocks: To verify the contribution of the Transformer module,
the Swin Transformer blocks are replaced by dilated residual dense block
(DRDB) proposed by [18]. As shown in Table I, the hybrid CNN-Transformer
scheme outperforms the pure CNN scheme by about 0.15db and 1.1db in PSNR-$\mu$
and PSNR-$l$, respectively, and also overtakes in MEF-SSIMd and NIMA. Fig. 5
shows that our method is more realistic in terms of color and details.

Structure Tensor: We remove the structural tensor information from the network
inputs and loss functions to explore the necessity of using it. Table I
demonstrates that the use of structure tensor effectively improves the quality
of fused images. Fig. 5 illustrates that without the constraint of the
structure tensor, the obtained HDR images suffer from significant ghosting
artifacts.

Cross Channel Attention Module: We abandon the proposed cross-channel
attention module (CCAM) as a way to explore its impact on the elimination of
ghost artifacts. Table I demonstrates that the presence of CCAM can
effectively improve the quality of the output HDR images. As shown in Fig. 5 ,
the proposed CCAM can also suppress ghosting artifacts to a certain extent.

###  III-B Comparison With Other HDR Deghosting Algorithms

The comparative study is performed with six state-of-the-art HDR deghosting
methods, one patch-based method (Sen et al. [10]) , one flow-based CNN method
(Kalantari et al. [16]) , two CNN based method( Wu et al. [17], Yan et al.
[18]), one multi-exposure fusion method ( Li et al. [33]), and a GAN-based
method (Niu et al. [19]). 10 validation image samples in [16] are used for
comparisons.

TABLE II: Average metric values with different HDR deghosting methods. Values in bold indicate the best results. $-$ means that Li et al. can not produce HDR images in $.hdr$ format Method | PSNR-$l$ | PSNR-$\mu$ | MEF-SSIMd | NIMA
---|---|---|---|---
Sen et al. | 41.8625 | 37.1662 | 0.9006 | 4.5661
Kalantari et al. | 33.8925 | 28.2505 | 0.7393 | 4.6170
Wu et al. | 34.1839 | 29.2874 | 0.8213 | 4.5393
Yan et al. | 40.1744 | 36.9848 | 0.9212 | 4.4591
Li et al. | - | - | 0.8221 | 4.1515
Niu et al. | 36.7448 | 34.5755 | 0.8629 | 4.5583
Ours (2 inputs) | 40.0943 | 36.9922 | 0.9245 | 4.6163
Ours (3 inputs) | 43.5397 | 38.9873 | 0.9233 | 4.6257

Table II shows the results when three LDR images are input to the methods for
comparison, as well as the results for our method when only the first two and
all three images of the sequence are input. Our method achieves the best
performance on PSNR-$l$ and PSNR-$\mu$ when three LDR images are input.
Notably, our method leads in the NIMA, which means that our method can satisfy
human visual sense. Interestingly, our method scores higher on the MEF-SSIMd
with only 2 inputs than with 3 inputs. From Fig. 1, our method has an
advantage at two inputs, and also achieves the least ghosting artifacts and
the clearest textures among the results obtained by the sequence fusion
strategy of three-input.

##  IV Conclusion

In this letter, we propose a novel ghost-free high dynamic range imaging
architecture based on a hybrid CNN-Transformer en/decoder network. The
structural tensor information introduced in the network can mitigate ghost
artifacts more effectively. The dual input model and the sequential fusion
strategy make our method independent of the number of inputs. Evaluation
experiments verify the effectiveness of our proposed scheme compared with
other state-of-the-art HDR deghosting models.

## References

  * [1] S. K. Nayar and T. Mitsunaga, “High dynamic range imaging: spatially varying pixel exposures,” in  Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), pp. 472–479, 2002.
  * [2] M. D. Tocci, C. Kiser, N. Tocci, and P. Sen, “A versatile HDR video production system,” ACM Trans. Graph., vol. 30, no. 4, pp. 41:1–41:10, Jul. 2011.
  * [3] P. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in  Proc. SIGGRAPH 97, Aug. 1997, pp. 369–378.
  * [4] T. Mertens, J. Kautz, and F. V. Reeth, “Exposure fusion: A simple and practical alternative to high dynamic range photography,”  Comput. Graph. Forum, vol. 28, no. 1, pp. 161–171, Mar. 2009.
  * [5] Y. Yang, W. Cao, S. Wu and Z. Li, “Multi-Scale Fusion of Two Large-Exposure-Ratio Images,” in IEEE Signal Process. Lett., vol. 25, no. 12, pp. 1885-1889, Dec. 2018, doi: 10.1109/LSP.2018.2877893.
  * [6] L. Bogoni, “Extending dynamic range of monochrome and color images through fusion,” in  Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), pp. 7–12, 2000.
  * [7] S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “High dynamic range video,” ACM Trans. Graph., vol. 22, no. 3, pp. 319–325, Jul. 2003.
  * [8] H. Zimmer, A. Bruhn, and J. Weickert, “Freehand HDR imaging of moving scenes with simultaneous resolution enhancement,” Comput. Graph. Forum, vol. 30, no. 2, pp. 405–414, Apr. 2011.
  * [9] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, “PatchMatch: A randomized correspondence algorithm for structural image editing,” ACM Trans. Graph., vol. 28, no. 3, pp. 24:1–24:11, Aug. 2009.
  * [10] P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, “Robust patch-based HDR reconstruction of dynamic scenes,” ACM Trans. Graph., vol. 31, no. 6, pp. 203:1–203:11, Nov. 2012.
  * [11] J. Hu, O. Gallo, K. Pulli, and X. Sun, “HDR deghosting: How to deal with saturation?,” in  Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), pp. 1163–1170, Jun. 2013.
  * [12] O. Gallo, N. Gelfandz, Wei-Chao Chen, M. Tico and K. Pulli, “Artifact-free High Dynamic Range imaging,” in  IEEE ICCP, 2009, pp. 1-7, doi: 10.1109/ICCPHOT.2009.5559003.
  * [13] E. A. Khan, A. O. Akyuz, and E. Reinhard, “Ghost removal in high dynamic range images,” in  Proc. Int. Conf. Image Process., pp. 2005–2008. Oct. 2006,
  * [14] Y. Heo, K. Lee, S. Lee, Y. Moon, and J. Cha, “Ghost-free high dynamic range imaging,” in Proc. IEEE Int. Conf. Asian Conf. Comput. Vis.(ACCV), pp. 486–500, Nov. 2011.
  * [15] C. Lee, Y. Li and V. Monga, “Ghost-Free High Dynamic Range Imaging via Rank Minimization,” in  IEEE Signal Process. Lett., vol. 21, no. 9, pp. 1045-1049, Sept. 2014, doi: 10.1109/LSP.2014.2323404.
  * [16] N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic range imaging of dynamic scenes,” ACM Trans. Graph., vol. 36, no. 4, pp. 1–12, Jul. 2017.
  * [17] S. Wu, J. Xu, Y.-W. Tai, and C.-K. Tang, “Deep high dynamic range imaging with large foreground motions,” in Proc. Eur. Conf. Comput. Vis.(ECCV), Sep. 2018, pp. 117–132.
  * [18] Q. Yan, D. Gong, Q. Shi, A.v.d. Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” in  Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), pp. 1751–1760, 2019.
  * [19] Y. Niu, J. Wu, W. Liu, W. Guo and R. W. H. Lau, “HDR-GAN: HDR Image Reconstruction From Multi-Exposed LDR Images With Large Motions,” in IEEE Trans. Image Process., vol. 30, pp. 3885-3896, 2021, doi: 10.1109/TIP.2021.3064433.
  * [20] Q. Li, W. Lin, and Y. Fang, “No-reference quality assessment for multipl-distorted images in gradient domain,” IEEE Signal Process. Lett., vol. 23, no. 4, pp. 541–545, Apr. 2016.
  * [21] Q. Yan, D. Gong, Q. Shi, A. v. d. Hengel, J. Sun, Y. Zhu, and Y. Zhang, “High dynamic range imaging via gradient-aware context aggregation network,” Pattern Recogn., Feb. 2022. https://doi.org/10.1016/j.patcog.2021.108342
  * [22] K. R. Prabhakar and R. V. Babu, “Ghosting-free multi-exposure image fusion in gradient domain,” in  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)., pp. 1766-1770, 2016, doi: 10.1109/ICASSP.2016.7471980.
  * [23] B. Jahne, Digital Image Processing, Springer-Verlag, Berlin, 2002.
  * [24] S. Lefkimmiatis, A. Roussos, P. Maragos, and M. Unser, “Structure Tensor Total Variation,” SIAM Journal on Imaging Sciences., vol. 8, pp.1090-1122, 2015.
  * [25] Y. Yuan, H. Pan, S. Cao, and Z. Jing, “Joint image pansharpening and registration via structure tensor total variation regularization,” Aero. Syst., vol.5, pp. 277–283, 2022. https://doi.org/10.1007/s42401-022-00138-w
  * [26] F. Yu, and V. Koltun, “Multi-Scale Context Aggregation by Dilated Convolutions,” CoRR, 2016, abs/1511.07122.
  * [27] X. Wang, Ross B. Girshick, Abhinav Kumar Gupta and H. Kaiming. “Non-local Neural Networks.” in  Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), pp. 7794-7803, 2018.
  * [28] Z. Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” in  Proc. IEEE Int. Conf. Comput. Vis., pp. 9992-10002, 2021, doi: 10.1109/ICCV48922.2021.00986.
  * [29] M. Zhao, G. Cao, X. Huang and L. Yang, “Hybrid Transformer-CNN for Real Image Denoising,” in IEEE Signal Process. Lett., vol. 29, pp. 1252-1256, 2022, doi: 10.1109/LSP.2022.3176486.
  * [30] P. Jagalingam and A. V. Hegde, “A review of quality metrics for fused image,” Aquatic Procedia, vol. 4, pp. 133–142, 2015.
  * [31] Y. Fang, H. Zhu, K. Ma, Z. Wang and S. Li, “Perceptual Evaluation for Multi-Exposure Image Fusion of Dynamic Scenes,” in IEEE Trans. Image Process., vol. 29, pp. 1127-1138, 2020, doi: 10.1109/TIP.2019.2940678.
  * [32] H. Talebi and P. Milanfar, “NIMA: Neural Image Assessment,” in IEEE Trans. Image Process., vol. 27, no. 8, pp. 3998-4011, Aug. 2018, doi: 10.1109/TIP.2018.2831899.
  * [33] H. Li, K. Ma, H. Yong and L. Zhang, ”Fast Multi-Scale Structural Patch Decomposition for Multi-Exposure Image Fusion,” in IEEE Trans. Image Process., vol. 29, pp. 5805-5816, 2020, doi: 10.1109/TIP.2020.2987133.