* [LJMR15] Yves Le Jan, Michael B. Marcus, and Jay Rosen.  Permanental fields, loop soups and continuous additive functionals.  Ann. Probab., 43(1):44–84, 2015.
  * [LJMR17] Yves Le Jan, Michael B. Marcus, and Jay Rosen.  Intersection local times, loop soups and permanental Wick powers.  Mem. Amer. Math. Soc., 247(1171):v+78, 2017.
  * [LRV17] Hubert Lacoin, Rémi Rhodes, and Vincent Vargas.  Semiclassical limit of Liouville field theory.  Journal of Functional Analysis, 273(3):875–916, 2017.
  * [LSW03a] Gregory F. Lawler, Oded Schramm, and Wendelin Werner.  Conformal restriction: the chordal case.  Journal of the American Mathematical Society, 16(4):917–955, 2003\.
  * [LSW03b] Gregory F. Lawler, Oded Schramm, and Wendelin Werner.  Conformal restriction: the chordal case.  J. Amer. Math. Soc., 16(4):917–955 (electronic), 2003.
  * [LTF07] Gregory F. Lawler and José A. Trujillo-Ferreras.  Random walk loop soup.  Trans. Am. Math. Soc., 359(2):767–787, 2007.
  * [Lup16] Titus Lupu.  From loop clusters and random interlacements to the free field.  The Annals of Probability, 44(3):2117 – 2146, 2016.
  * [Lup18a] Titus Lupu.  Convergence of the two-dimensional random walk loop-soup clusters to CLE.  Journal of the European Mathematical Society, 21(4):1201–1227, 2018\.
  * [Lup18b] Titus Lupu.  Poisson ensembles of loops of one-dimensional diffusions.  Mém. Soc. Math. Fr. (N.S.), (158):158, 2018.
  * [LW04] Gregory F. Lawler and Wendelin Werner.  The Brownian loop soup.  Probab. Theory Relat. Fields, 128(4):565–588, 2004.
  * [Mol05] Ilya Molchanov.  Theory of random sets, volume 19.  Springer, 2005.
  * [MR06] Michael B. Marcus and Jay Rosen.  Markov processes, Gaussian processes and local times, volume 100\.  Cambridge University Press, 2006.
  * [MR13] Michael B. Marcus and Jay Rosen.  A sufficient condition for the continuity of permanental processes with applications to local times of Markov processes.  Ann. Probab., 41(2):671–698, 2013.
  * [MS] Jason Miller and Scott Sheffield.  CLE(4) and the Gaussian Free Field.  In preparation.
  * [MS16a] Jason Miller and Scott Sheffield.  Imaginary geometry I: interacting SLEs.  Probab. Theory Relat. Fields, 164(3-4):553–705, 2016.
  * [MS16b] Jason Miller and Scott Sheffield.  Imaginary geometry II: Reversibility of $\operatorname{SLE}_{\kappa}(\rho_{1};\rho_{2})$ for $\kappa\in(0,4)$.  Ann. Probab., 44(3):1647–1722, 2016.
  * [MS16c] Jason Miller and Scott Sheffield.  Imaginary geometry III: reversibility of $\rm SLE_{\kappa}$ for $\kappa\in(4,8)$.  Ann. of Math. (2), 184(2):455–486, 2016.
  * [MS17] Jason Miller and Scott Sheffield.  Imaginary geometry IV: interior rays, whole-plane reversibility, and space-filling trees.  Probab. Theory Related Fields, 169(3-4):729–869, 2017.
  * [MSW14] Jason Miller, Nike Sun, and David B Wilson.  The Hausdorff dimension of the CLE gasket.  Ann. Probab., 42(4):1644–1665, 2014.
  * [MSW17] Jason Miller, Scott Sheffield, and Wendelin Werner.  CLE percolations.  Forum Math. Pi, 5:e4, 102, 2017.
  * [MWW15] Jason Miller, Samuel S Watson, and David B Wilson.  The conformal loop ensemble nesting field.  Probab. Theory Relat. Fields, 163(3):769–801, 2015.
  * [Qia18] Wei Qian.  Conformal restriction: the trichordal case.  Probab. Theory Relat. Fields, 171(3-4):709–774, 2018.
  * [Qia19] Wei Qian.  Conditioning a Brownian loop-soup cluster on a portion of its boundary.  Ann. Inst. Henri Poincaré Probab. Stat., 55(1):314–340, 2019\.
  * [QW18] Wei Qian and Wendelin Werner.  The law of a point process of brownian excursions in a domain is determined by the law of its trace.  Electron. J. Probab., 23:23 pp., 2018.
  * [QW19] Wei Qian and Wendelin Werner.  Decomposition of Brownian loop-soup clusters.  J. Eur. Math. Soc., 21(10):3225–3253, 2019.
  * [RS05] Steffen Rohde and Oded Schramm.  Basic properties of SLE.  Ann. of Math., 161(2):883–924, 2005.
  * [RV10] Raoul Robert and Vincent Vargas.  Gaussian multiplicative chaos revisited.  Ann. Probab., 38(2):605–631, 2010.
  * [RV11] Rémi Rhodes and Vincent Vargas.  KPZ formula for log-infinitely divisible multifractal random measures.  ESAIM: Probability and Statistics, 15:358–371, 2011.
  * [Sch00] Oded Schramm.  Scaling limits of loop-erased random walks and uniform spanning trees.  Israel J. Math., 118:221–288, 2000.
  * [Sch01] Oded Schramm.  A percolation formula.  Electron. Commun. Probab., 6:115 – 120, 2001.
  * [Sha16] Alexander Shamov.  On Gaussian multiplicative chaos.  J. Funct. Anal., 270(9):3224 – 3261, 2016.
  * [She09] Scott Sheffield.  Exploration trees and conformal loop ensembles.  Duke Math. J., 147(1):79–129, 2009.
  * [SS13] Oded Schramm and Scott Sheffield.  A contour line of the continuum Gaussian free field.  Probab. Theory Relat. Fields, 157(1-2):47–80, 2013.
  * [SW12] Scott Sheffield and Wendelin Werner.  Conformal loop ensembles: the Markovian characterization and the loop-soup construction.  Ann. of Math. (2), 176(3):1827–1917, 2012.
  * [Sym65] Kurt Symanzik.  Euclidean quantum field theory I: Equations for a scalar model.  New York University, 1965.
  * [Sym66] Kurt Symanzik.  Euclidean quantum field theory I. Equations for a scalar model.  J. Math. Phys, 7(3):510–525, 1966.
  * [Sym69] Kurt Symanzik.  Euclidean quantum field theory.  In Scuola intenazionale di Fisica Enrico Fermi. XLV Corso., pages 152–223. Academic Press, 1969.
  * [Szn12] Alain-Sol Sznitman.  Topics in occupation times and Gaussian free field.  Zurich lectures in advanced mathematics. European Mathematical Society, 2012.
  * [vdBCL18] Tima van de Brug, Federico Camia, and Marcin Lis.  Spin systems from loop soups.  Electron. J. Probab., 23:1–17, 2018.
  * [VJ97] D. Vere-Jones.  Alpha-permanents and their applications to multivariate gamma, negative binomial and ordinary binomial distributions.  New Zealand J. Math., 26(1):125–149, 1997.
  * [Wol78] Robert L Wolpert.  Local time and a particle picture for Euclidean field theory.  Journal of Functional Analysis, 30(3):341–357, 1978.
  * [WP21] Wendelin Werner and Ellen Powell.  Lecture notes on the Gaussian Free Field, 2021.


# Collision of two drops moving in the same direction

Ashwani Kumar Pal  Department of Mechanical Engineering, Indian Institute of
Technology Kanpur, Kanpur - 208016, Uttar Pradesh, India  Kirti Chandra Sahu
Department of Chemical Engineering, Indian Institute of Technology Hyderabad,
Kandi - 502 284, Telangana, India  Santanu De  Department of Mechanical
Engineering, Indian Institute of Technology Kanpur, Kanpur - 208016, Uttar
Pradesh, India  Gautam Biswas<EMAIL_ADDRESS>Department of Mechanical
Engineering, Indian Institute of Technology Kanpur, Kanpur - 208016, Uttar
Pradesh, India

###### Abstract

The collision dynamics of two drops of the same liquid moving in the same
direction has been studied numerically. A wide range of radius ratios of
trailing drop and leading drop ($R_{r}$) and the velocity ratios ($U_{r}$)
have been deployed to understand the collision outcomes. A volume of fluid
(VOF) based open-source fluid flow solver, Basilisk, has been used with its
adaptive mesh refinement feature to capture the nuances of the interface
morphology. The simulations are analyzed for the evolving time instances.
Different collision outcomes, such as coalescence and reflexive separation
with and without the formation of satellite drops, have been observed for
various combinations of $U_{r}$ and $R_{r}$. The study analyzes the evolution
of kinetic energy and surface energy before and after the collision for
plausible outcomes. The collision outcomes are depicted on a regime map with
$U_{r}-R_{r}$ space, highlighting distinct regimes formed due to variations in
relevant governing parameters.

##  I Introduction

The collision of liquid drops has garnered significant attention due to its
relevance in various industrial applications, including nuclear fusion, spray
combustion, spray drying, inkjet printing, and design of combustion chambers
Biswas and Sahu (2021); Kumar, Bhardwaj, and Sahu (2020); Thoroddsen, Etoh,
and Takehara (2008); Stone, Stroock, and Ajdari (2004). Additionally, it plays
a crucial role in meteorological phenomena like raindrop formation Low and
List (1982).

The Weber number influences the head-on collision of two drops, expressed as
$We=\rho_{l}{U^{2}}D/\sigma$, where $\rho_{l}$ is the liquid density, $D$
denotes the drop diameter, and $U$ represents the relative approach velocity.
As the collision unfolds, a high-pressure zone forms in the gap between the
drops, causing their deformation and the extrusion of a thin surrounding fluid
film. Coalescence of the drops occurs if the gap between them decreases to a
scale approximately equal to the length of molecular interactions, typically
of the order of $10^{2}$ Angstroms. Alternatively, the drops may bounce Qian
and Law (1997). Qian and Law (1997) investigated the head-on and oblique
collisions of identical water and hydrocarbon drops under various surrounding
pressures and identified five distinct outcomes. They are (i) coalescence
after minor deformation, (ii) bouncing, (iii) coalescence after substantial
deformation, (iv) coalescence followed by separation in near head-on
collisions, and (v) coalescence followed by separation in off-center/ oblique
collisions. The transition between the coalescence and separation regimes was
also identified by Qian and Law (1997). Subsequently, Pan, Law, and Zhou
(2008) performed numerical and experimental investigations on the head-on
collision of two identical drops across a wide range of Weber numbers.
Employing a time-resolved microphotographic technique similar to that of Qian
and Law (1997) and a front-tracking method for numerical simulations, they
found that the merging instant could be computationally assessed using an
augmented van der Waals force and the associated parameter, known as the
Hamaker constant. This approach was applied to collisions with both minor and
significant deformations, corresponding to soft and hard collisions,
respectively. Nobari, Jan, and Tryggvason (1996) numerically examined the
head-on collision of equal-sized droplets, utilizing a front tracking/ finite
difference technique. They mapped the boundaries between coalescing and
separating collisions on the Reynolds number and Weber number plane.

A few researchers have also investigated the collision dynamics of unequal-
sized droplets (Zhang, Li, and Thoroddsen, 2009; Deka  _et al._ , 2019;
Chaitanya, Sahu, and Biswas, 2021; Singh and Saha, 2022). Zhang, Li, and
Thoroddsen (2009) experimentally investigated the coalescence dynamics of a
small drop falling onto a larger drop. Kumar, Bhardwaj, and Sahu (2020)
investigated the dynamics of an ethanol drop freely descending onto a larger
sessile drop of the same fluid. Their study revealed intriguing regime maps,
showcasing partial coalescence and spreading behaviors, plotted against the
Weber number and the volume ratio of the sessile and impacting drops. They
found that the critical size ratio for satellite drops matched the conclusions
of both Zhang, Li, and Thoroddsen (2009) and Nikolopoulos  _et al._ (2012),
with the latter specifically examining the splashing regime triggered by drops
impacting from a height. Tang, Zhang, and Law (2012) theoretically and
experimentally investigated the head-on collision between hydrocarbon and
water drops of unequal sizes. They delineated regions in the parameter space
of size ratio and collision Weber number associated with the bouncing,
permanent coalescence, and separation after coalescence. On a numerical front,
Deka  _et al._ (2019) explored the head-on collision of unevenly sized
droplets, revealing that even with a diameter ratio of 1.2, the collision
resulted in the partial coalescence of tiny daughter droplets. At high Weber
numbers, Cong  _et al._ (2020) scrutinized the collision dynamics of unevenly
sized drops. Axisymmetric simulations conducted by Goyal, Shaikh, and Sharma
(2020) also contributed to understanding the head-on collision of two drops
with disparate sizes. Recently, attempts have also been made to study the
collision of two drops having different physical properties. Al-Diwari and
Bayly (2020) compared the collision outcomes of the drops with different
viscosities with those of the same viscosities Al-Diwari and Bayly (2019) and
presented the regime maps. Numerical investigation of collision of two drops
having non-equal viscosity is performed by Deka, Biswas, and Bora (2023). They
observed that satellite drop formation does not occur for higher viscosity
ratios between the drops. Paul  _et al._ (2023) investigated the dynamics of
two vertically coalescing drops and a pool. They found partial coalescence
when a conglomerate interacts with a pool in a manner similar to drop–pool
Blanchette and Bigioni (2006); Thoroddsen, Etoh, and Takehara (2008); Ray,
Biswas, and Sharma (2010); Kirar  _et al._ (2020) and drop–drop Zhang, Li, and
Thoroddsen (2009); Deka  _et al._ (2019); Chaitanya, Sahu, and Biswas (2021);
Singh and Saha (2022) interactions. Recently, Kirar  _et al._ (2022)
experimentally investigated the coalescence dynamics of two equal and unequal
drops colliding with a deep pool in a side-by-side arrangement. They found a
spectrum of coalescence outcomes, ranging from total coalescence to
interacting and non-interacting partial coalescence, achieved through
adjustments in drop distance and size ratios. Their study reported two
distinctive coalescence phenomena, namely first, where primary drops fuse
before the ensuing conglomerate integrates into the liquid pool, and another,
where the drops coalesce individually within the liquid pool, giving rise to
capillary wave interactions that significantly impact the overall coalescence
results. The collision dynamics of a drop falling on a stationary sessile drop
residing on a hydrophobic surface were experimentally and numerically
investigated by Ray, Han, and Cheng (2023). They observed two distinct
collision outcomes for different size ratios: reflexive separation without a
satellite and reflexive separation with a satellite. A similar system was also
experimentally studied by Kumar, Bhardwaj, and Sahu (2020). They demarcated
the partial coalescence and spreading behaviours in the Weber number and the
volume of the sessile droplet space. Recently, Sprittles (2023) presented a
comprehensive review related to the bouncing of drops. In the drop-drop
bouncing scenario, the bouncing dynamics can be placed in the perspective of
soft transition (from merging to bouncing) and hard transition (from bouncing
to merging) with increasing Weber number. An extensive review of the droplet
collision dynamics can be found in Amani  _et al._ (2019); Liu and Bothe
(2016).

As the above brief review indicates, several researchers have investigated the
interaction of drops on a liquid pool, while the collision of drops receives
far less attention. Furthermore, the existing studies on drop collisions have
primarily focused on a head-on configuration, where the drops collide in
opposite directions. However, in many practical applications and natural
scenarios, drops frequently collide while moving in the same direction, an
aspect examined in the current study. A few specific applications where the
collision of drops moving in the same direction is observed include the
collision of raindrops falling from clouds, undergoing coalescence, and the
release of liquid petroleum due to an oil spill Testik, Barros, and Bliven
(2011); Lambert and Variano (2016). Thus, this study also explores the effect
of changing drop sizes and their direction of motion during collision events.

The rest of the manuscript is organized as follows. The details of the problem
formulation, governing equations, the numerical method used and its validation
are presented in §II. The results are presented and discussed in §III, and the
concluding remarks are given in §IV.

##  II Formulation

We investigate the collision dynamics between two drops moving in the same
direction, as illustrated in Fig. 1. The radii of the trailing and the leading
drops are represented by $R_{1}$ and $R_{2}$, and they migrate with velocities
$U_{1}$ and $U_{2}$, respectively, where $U_{1}\geq U_{2}$. In our numerical
simulations, the initial separation distance between the centers of the drops
(at the time, $t=0$) is kept at $4R_{2}$. The collision dynamics of two drops
moving along the coinciding axis can be effectively analyzed through
axisymmetric simulations, as demonstrated in previous studies (Huang, Pan, and
Josserand, 2019; Deka, Biswas, and Bora, 2023). However, it is essential to
note that the dynamics can become three-dimensional at high velocity and size
ratios when drops undergo separation, resulting in satellite drops and finger-
like morphology. The understanding of the three-dimensional effect is beyond
the scope of the current research.. Thus, we employ an axisymmetric
computational domain of size $36R_{2}\times 36R_{2}$. The axisymmetric domain
has a common edge coinciding with the axis of symmetry passing through the
centers of the drops as enumerated in Fig. 1. Our study considers the top part
of the axis as the computational domain. The outflow boundary conditions,
i.e., zero gradients for the normal and tangential components of velocity and
zero Dirichlet condition for the pressure, are applied at the left and right
extremes of the domain. Free slip boundary conditions are used at the top
boundary, while the symmetric boundary conditions are applied at the axis of
symmetry (bottom boundary).

Figure 1: Schematic diagram showing the collision of two drops of radii
$R_{1}$ and $R_{2}$ moving with velocities $U_{1}$ and $U_{2}$ in the same
direction.

###  II.1 Governing equations

The dynamics of drop collision is governed by the solution of the continuity
and momentum equations, which are given by

$\nabla\cdot\textbf{u}=0,$ (1)
$\displaystyle\rho(\phi)\left[\frac{\partial\textbf{u}}{\partial
t}+\textbf{u}.\nabla\textbf{u}\right]=-\nabla
p+\nabla\cdot[\mu(\phi)(\nabla\textbf{u}+\nabla\textbf{u}^{T})]$
$\displaystyle+\sigma\kappa\hat{n}\delta_{s}+\rho(\phi)\bm{g}.$ (2)

In Eq. (2), the inclusion of the surface tension term in the Navier–Stokes
equations follows the formulation proposed by Brackbill, Kothe, and Zemach
(1992) for interfacial flows. The velocity field is represented by u with $u$
and $v$ being the axial and vertical components, respectively. The pressure is
denoted by $p$ and $\sigma$ represents the coefficient of surface tension,
$\kappa$ is the curvature of the interface, $\hat{n}$ is the unit normal
vector on the interface pointing in the gas phase and $\delta_{s}$ represents
the Dirac delta function, which is zero everywhere except at the interface.

An interface-capturing technique becomes essential to trace the movement of
the interface within the flow accurately. The dynamics of the interface is
captured by a variant of volume of fluid method available in Basilisk(Popinet,
2009). The volume of fluid function $(\phi)$ takes a value $1$ in the pure
liquid cells, $0$ in the pure gas cells, and a value between $0$ and $1$ in
the cells that contain the interface. The function $\phi$ is advected using
the following equation,

$\displaystyle\frac{\partial\phi}{\partial t}+\textbf{u}\cdot\nabla\phi$
$\displaystyle=$ $\displaystyle 0.$ (3)

The density $\rho(\phi)$ and viscosity $\mu(\phi)$ fields are given by

$\displaystyle\rho(\phi)=\rho_{l}\phi+\rho_{g}(1-\phi),$ (4)
$\displaystyle\mu(\phi)={\mu_{l}\phi+\mu_{g}(1-\phi)},$ (5)

where $(\rho_{l},\mu_{l})$ and $(\rho_{g},\mu_{g})$ are the density and
viscosity of the liquid and gas phases, respectively.

###  II.2 Numerical method

The set of governing equations is solved using an open-source code Basilisk
Popinet (2013) (http://basilisk.fr) developed by Stephane Popinet. The code
uses a volume of fluid based interface capturing technique. A finite volume
approach is used to discretize the governing equations on a centered grid
(both pressure and velocity are defined at the center of the computational
cell). Moreover, a balanced-force, continuum-surface-force formulation based
on the height function has been incorporated to include the surface force term
in the Navier–Stokes equations. This minimizes spurious currents at the
interface. The incompressible and variable density Navier-Stokes equations are
solved using a projection method on a quadtree grid (see Popinet (2003, 2009)
for a detailed procedure). A second-order accurate Bell-Colella-Glaz scheme
Bell, Colella, and Glaz (1989) is used for the advection terms in the momentum
equation. The surface tension force in the momentum equation is added as the
interfacial force density in the two-phase cells only. A conservative multi-
dimensional scheme of Weymouth and Yue (2010) is used to advect the volume of
fluid function. The interface is constructed geometrically as a piece-wise
linear segment in the computational cell. The governing equations are marched
in time using the Courant–Friedrichs–Lewy (CFL) criterion with a value of 0.5
for stable calculations. Basilisk also allows us to add as many tracers as we
like. We advect a tracer to distinguish between the fluids of the two drops.
The tracer follows the same advection equation as the volume of fluid function
but does not influence the flow field. Its purpose is purely for
visualization, facilitating the representation of distinct drops with
different colors.

###  II.3 Dimensionless parameters

The various dimensionless numbers associated with the problem of drop
collision are the Weber number $(We)$, Ohnesorge number $(Oh)$, radius ratio
$(R_{r})$ and velocity ratio $(U_{r})$. The Weber number and the Ohnesorge
number are defined based on the parameters of the leading drop as
$We=2\rho_{l}{U_{2}^{2}}R_{2}/\sigma$ and $Oh=\mu_{l}/\sqrt{\rho_{l}\sigma
R_{2}}$, respectively. The Weber number $(We)$ represents the competition
between the inertia force of the liquid and the capillary force at the liquid-
gas interface, while the Ohnesorge number $(Oh)$ characterizes the competition
among viscous force, inertial force, and capillary forces. The radius ratio is
the ratio of the radius of the trailing drop to leading drop
$(R_{r}=R_{1}/R_{2})$, and the velocity ratio is the ratio of the velocity of
the trailing drop to leading drop $(U_{r}=U_{1}/U_{2})$. The time is non-
dimensionalized as $\tau=tU_{2}/R_{2}$. The gravitational effect has a
negligible effect on the collision dynamics, given the very short time scale
of the coalescence process, typically on the order of milliseconds. In the
present study, we fix $R_{2}$, $We$, and $Oh$, and the effect of the velocity
and radius ratios have been varied to obtain different collision outcomes.

###  II.4 Validation

####  II.4.1 Grid convergence test

The governing equations are discretized on a quadtree grid implemented in
Basilisk flow solver, which makes it capable of variable grid sizes in the
areas of interest. A very fine grid resolution is required to capture the
physics of pinch-off in the collision process of two drops. Due to the
advantage of the adaptive mesh refinement ability of Basilisk, we are able to
refine the grid locally around the interface to a fine level. A grid
convergence test is carried out to determine the optimum grid size that can
efficiently predict the pinch-off dynamics between the drops. Two drops of
identical radii $R_{1}=R_{2}=150$ $\mu$m are initialized at a distance of
$4R_{2}$ between the centers of the drops as shown in the schematic diagram 1.
The drop collision is simulated in the reflexive separation regime in which
the drops separate, giving rise to a satellite drop. The Weber and Ohnesorge
numbers used for the simulations are $We=12.58$, $Oh=0.0376$. The properties
the liquid and gas phases are listed in Table 1. The grid size is represented
in the form of the level of refinement, $L$. The grid size corresponding to
the level of refinement $L$ can be calculated as $\Delta=(36\times
R_{2})/2^{L}$, and thus, the grid size decreases with increasing the level of
refinement. We use the grid adaptability criterion implemented in basilisk
based on both the volume of fluid field $\phi$ and the velocity field
$\mathbf{u}$ in the domain. The grid size corresponding to $L=11$ is used in
the domain with an adaptive mesh refinement for the region close to the
interface. Fig. 2 shows the dynamics obtained using three levels of maximum
refinement, $L=12$, 13 and 14. The minimum grid sizes corresponding to these
refinement levels are $\Delta_{min}=1.318$ $\mu$m, $0.659$ $\mu$m, and $0.329$
$\mu$m. It is evident from Fig. 2 that the dynamics of collision and pinch-off
are almost similar for the refinement levels of $13$ and $14$, while the drops
do not stretch enough for the level $12$, which makes the pinch-off
asymmetric, giving rise to a relatively smaller satellite drop. The total
kinetic energy of the liquid is estimated by integrating the kinetic energy of
each computational cell over all the liquid cells, i.e.,
$KE=\Sigma{\frac{1}{2}{\rho_{l}F(u^{2}+v^{2})}dV}$, where $dV$ is the volume
of the computational cell. The total surface energy is estimated as
$SE=\sigma\times A_{s}$. In this expression, $A_{s}$ is the total surface area
of the liquid, which is calculated by adding the interface area in all the
two-phase cells. To make a quantitative comparison of the grid size on the
simulation results, in Fig. 3, we compare the total kinetic energy of the
liquid phase obtained using different levels of refinement. The kinetic energy
is non-dimensionalized with the initial kinetic energy of the liquid phase. It
can be observed from Fig. 3 that a convergence of the curves is achieved as we
increase the level of refinement. For optimising the computational time while
maintaining decent accuracy of the computer program, the maximum refinement
level of $13$ ($\Delta_{min}=0.659$ $\mu$m) is used for all the computations
performed in the present study.

Figure 2: Comparison of the drop shapes obtained using different levels of grid refinement $(L)$ at different values of the dimensionless time $\tau$, which are mentioned at the top of each panel. The parameters used for the simulation are $R_{r}=1.0$, $U_{r}=3.0$, $We=12.58$, and $Oh=0.0376$. The dimensionless time $\tau=0$ corresponds to the onset of collision. The contours of $\phi$ superimposed with the tracer function are plotted in each panel, displaying only a subset of the computational domain. Table 1: Properties of the fluids considered in our simulations. The surface tension $(\sigma)$ at the tetradecane-air interface $26\times 10^{-3}$ N/m. |  $\rho$ (kg/m3) |  $\mu$ (Pa$\cdot$s) |
---|---|---|---
Tetradecane | $759.0$ | $2.05\times 10^{-3}$ |
Air | $1.2$ | $1.78\times 10^{-5}$ |
Figure 3: Temporal evolution of dimensionless kinetic energy of the liquid for
different levels of grid refinement. The parameters used for the simulation
are $R_{r}=1.0$, $U_{r}=3.0$, $We=12.58$ and $Oh=0.0376$. Figure 4: Comparison
of collision outcomes of the drops migrating in the opposite direction (head-
on) and the same direction obtained from the present simulations with the
results of Huang, Pan, and Josserand (2019). The top panel presents the
results of Huang, Pan, and Josserand (2019). The middle and bottom panels show
our results with the drops moving in the opposite ($U_{1}=U_{2}=1.145$ m/s)
and the same ($U_{1}=3.046$ m/s and $U_{2}=0.7555$ m/s) directions,
respectively. Thus, the relative velocity for both cases remains $U=2.29$ m/s.
The rest of the parameters used for the simulations are $R_{r}=1.0$,
$We=11.48$, and $Oh=0.0376$. Figure 5: Evolution of the shapes of the drops
with $R_{r}=1.0$ undergoing collision for different values of velocity ratio
($U_{r}$). The parameters used for the simulation are $We=12.58$ and
$Oh=0.0376$. The dimensionless time $\tau=0$ corresponds to the onset of
collision.

####  II.4.2 Comparison with the results of Huang, Pan, and Josserand (2019).

To check the numerical accuracy of the results obtained in the present
simulations, a case of two identical tetradecane drops colliding in the
opposite direction is considered. The results are compared with those obtained
by Huang, Pan, and Josserand (2019). Two tetradecane drops with identical
radius $R_{1}=R_{2}=150$ $\mu$m, and initial velocities $U_{1}=U_{2}=1.145$
m/s in the opposite directions are simulated in an axisymmetric configuration.
The Weber number and Ohnesorge number for the simulation are kept same as by
Huang, Pan, and Josserand (2019) ($We=11.48$ and $Oh=0.0376$). The collision
outcomes are plotted with the results of Huang, Pan, and Josserand (2019) at
different time instances in Fig. 4. The results obtained from the present
simulations show satisfactory agreement with the results of the Huang, Pan,
and Josserand (2019). As observed by them, the drops undergo symmetrical
reflexive separation after the collision and produce a small satellite drop in
between two larger drops. This verifies the accuracy of the results obtained
from the present simulations.

##  III Results and discussion

We begin the presentation of our results by analysing the dynamics of two
drops moving in the same direction. Simulations are performed to see the
effect of velocity and radius ratios between the drops on the collision
outcome. The velocity, $U_{2}=1.15$ m/s, and the radius, $R_{2}=150$ $\mu$m of
the leading drops are kept fixed for all the cases presented in this paper.
Thus, the dimensionless parameters, such as $We=12.58$ and $Oh=0.0376$ are
constant. The radius $R_{1}$ and velocity $U_{1}$ of the trailing drop are
varied to obtain various radius and velocity ratios considered. The collision
process exhibits three distinct outcomes: coalescence and reflexive separation
with and without satellite drop formation. The transition from coalescence to
reflexive separation takes place as the total inertia of the liquid increases
for the higher values of $U_{r}$ (Qian and Law (1997)). However, the formation
of the satellite drop is inhibited in the case of higher radius ratio $R_{r}$
as also observed by Huang, Pan, and Josserand (2019).

Figure 6: Evolution of the shapes of the drops with $R_{r}=0.5$ undergoing
collision for different values of velocity ratio. The parameters used for the
simulation are $We=12.58$ and $Oh=0.0376$. The dimensionless time $\tau=0$
corresponds to the onset of collision.

###  III.1 Effect of velocity ratio for identical drops

In this section, we investigate the effect of velocity ratio on the collision
dynamics of two identical drops. Two tetradecane drops of the same size
($R_{1}=R_{2}=150$ $\mu$m) are given the initial velocities $U_{1}$ and
$U_{2}$ in the same direction ($U_{1}>U_{2}$). The numerical configuration and
the boundary conditions remain the same, as explained in the formulation
section II. Simulations are performed for various values of the velocity
ratio. The trailing drop approaches the leading drop with a relatively higher
velocity as the velocity ratio increases with the velocity of the leading drop
fixed for all the cases. The shapes of the drops are plotted in Fig. 5 at
different time instances for different values of the velocity ratios
considered. The time is non-dimensionalized by the time constant of the
leading drop ($\tau=tU_{2}/R_{2}$), and $\tau=0$ represents the moment when
the two drops are about to merge. Top-to-bottom panels show the evolution of
collision profiles with increasing the velocity ratio. Visual inspection of
Fig. 5 shows the change of outcome regime of the collision from coalescence to
reflexive separation. It can be observed that the drops come close to each
other, forming a single liquid bulk that keeps moving in the direction of the
velocities of the drops.

In the case of lower velocity ratios (up to $U_{r}=2.5$), the combined liquid
mass stretches out in the vertical direction and then in the horizontal
direction. These oscillations keep happening while the liquid mass moves in
the horizontal direction, as seen in the coalescence of two drops moving in
the opposite direction Qian and Law (1997). During this process, the
conversion of surface energy into kinetic energy and vice versa occurs until
the shape fluctuations are damped due to viscous damping and the liquid mass
takes a spherical shape, which moves in the direction of initial velocities of
the drops. However, in the case of higher velocity ratios ($U_{r}\geq 3.0)$,
the drops undergo separation after the collision. This happens due to
increased relative impact velocity between the drops Huang, Pan, and Josserand
(2019). The trailing drop impacts the leading drop with high velocity,
imparting momentum to the trailing drop. After the collision, the liquid
packet squeezes, forming a thin disk shape, which starts to elongate along the
axis due to the action of the capillary forces. The combined liquid mass
starts retracting and elongating in the axial direction. Since the liquid mass
has higher momentum than in the case of lower velocity ratios, it keeps on
elongating and forms a dumbbell shape with a ligament connecting two liquid
masses at its end. The liquid mass keeps moving in the axial direction during
this process. The pressure starts increasing at the two ends of the ligament,
which carries the fluid away from the ends Deka, Biswas, and Bora (2023). This
gives rise to a symmetrical pinch-off of the ligament, and the liquid inside
the ligament gives rise to a satellite drop. The leading part of the ligament
detaches with a higher velocity than the one on the trailing while the liquid
continues to move in the same direction. A significant increase in the size of
the satellite drop is also observed in the case of $U_{r}=3.5$ as compared to
the case with $U_{r}=3.0$.

Figure 7: Evolution of the shapes of the drops with $R_{r}=1.5$ undergoing
collision for different values of velocity ratio. The parameters used for the
simulation are $We=12.58$ and $Oh=0.0376$. The dimensionless time $\tau=0$
corresponds to the onset of collision.

###  III.2 Effect of velocity ratio for the different sized drops

The effect of the size of the drops, along with the velocity ratio, on the
collision outcome of the drops is analyzed in this section. The radius
$(R_{1})$ and velocity $(U_{1})$ of the trailing drop are varied by keeping
the radius $(R_{2})$ and velocity $(U_{2})$ of the leading drop fixed to
obtain various radius and velocity ratios. The simulations are performed for
various combinations of radius and velocity ratios.

Firstly, we extend the analysis for the cases with $R_{r}<1$. For the case of
$R_{r}<1$, the trailing drop is initialized with a radius $R_{1}=0.5R_{2}$ and
with different initial velocities $(1.5U_{2}\leq U_{1}\leq 3.5U_{2})$. The
collision outcomes for different values of the velocity ratios are plotted in
Fig. 6. The trailing drop, which is smaller, hits the larger leading drop with
higher velocity. The collision in these cases is asymmetric, unlike the
previous case with identical-sized drops. The smaller drop penetrates the
larger drop while the larger absorbs the higher velocity impact after
collision. The impact creates a crater at the trailing side of the combined
liquid mass while the leading side remains convex. The curvature of the crater
increases with increasing the velocity ratio of impact, as expected. As a
consequence of high-velocity impact, strong clockwise vortexes are generated
towards the ends of the crater, which rotates the liquid mass in such a way
that the leading side of the liquid takes a concave crater shape and the
trailing side becomes convex shaped for higher velocity ratios ($U_{r}>2.0$).
In contrast, such rotation of the ends of the crater does not take place for
lower values of the velocity ratio. At later times, as evident from Fig. 6,
the liquid packet starts to stretch from its leading end for all the cases of
the velocity ratios, which is also the case in the case of collision of
identical-sized drops. The stretching is more significant in the case of
collision with higher velocity ratios, while it is not that prevalent in the
case of lower velocity ratios where the liquid packet takes an almost
spherical shape due to capillary action and lack of momentum. Even in the case
of higher velocity ratios, the liquid eventually takes a practically spherical
shape due to containing enough momentum to separate the liquid packet into two
parts. Formation of the satellite drops is not observed for any values of the
velocity ratio considered in the present study. The liquid continues to
migrate in the direction of initial velocity with an almost spherical shape.

Figure 8: Contours of the axial velocity (upper half) and pressure field
(bottom half) along with the interface of the drops for different combinations
of $U_{r}$ and $R_{r}$. The top to bottom rows represent the coalescence,
reflexive separation with a satellite drop, and reflexive separation without a
satellite, respectively. The rest of the dimensionless parameters used for the
simulation are $We=12.58$ and $Oh=0.0376$.

Next, we study the effect of radius ratio $R_{r}>1.0$. The trailing drop is
initialized with a radius higher than the leading drop and with different
initial velocities, giving rise to various velocity ratios. The outcomes of
the collision process for different velocity ratios for $R_{r}=1.5$ are shown
in Fig. 7. The larger trailing drop hits the smaller leading drop with high
impact velocities. As a result, for these cases, the crater is formed on the
leading side of the impacting drop due to less momentum of the leading drop.
The leading drop fluid is spread on the outer side of the crater. For higher
velocity ratios, the drop tries to attain a disk shape before the ends are
rotated to reverse the direction of the crater. The combined liquid packet
starts to elongate as time progresses. The stretching of the liquid is
asymmetric in these cases, and the trailing side of the liquid is stretched
more than the leading one. The stretching is continued until the relative
kinetic energy is converted into the surface energy or dissipated due to
viscous effects. For lower values of the velocity ratios ($U_{r}<2.5$), the
liquid does not break into the part due to lack of momentum, and the capillary
forces pull back the liquid. The liquid packet again tries to extend itself in
the axial depending upon the momentum and finally attains a stable shape. In
the case of higher values of velocity ratios ($U_{r}>2.5$), the liquid breaks
into two parts since it has more momentum to overcome the capillary and
viscous forces. Unlike the case with the collision of identical drops, the
stretching of the liquid volume is asymmetrical in this case, which leads to
an asymmetrical break-up of the liquid volume. The necking is only at one
location towards the trailing end where the pressure is maximum, and the
pinch-off takes place at this location. The liquid breaks into two unequal
volumes of liquid, the trailing being the smaller one and the leading one
being the larger one. The separated liquid volumes try to attain a stable
spherical shape while moving in the direction of their initial velocity.

Figure 9: Evolution of the normalised surface energy and kinetic energy for
different collision outcomes: (a) coalescence for $R_{r}=1.0$ and $U_{r}=2.5$,
(b) reflexive separation with satellite for $R_{r}=1.0$ and $U_{r}=3.0$, (c)
reflexive separation without satellite for $R_{r}=1.5$ and $U_{r}=3.0$. The
other parameters used for the simulation are $We=12.58$ and $Oh=0.0376$.

###  III.3 Different collision outcomes and phase diagram in $R_{r}-U_{r}$
plane

Various collision outcomes observed in the present computations are due to the
different values of $U_{r}$ and $R_{r}$. We analyze the evolution of velocity
and pressure contours for these cases to understand the mechanism behind
different collision outcomes. In Fig. 8, the contours of the axial velocity
(top half) superimposed with the velocity vectors and the pressure contours
(bottom half) are shown for four combinations of $U_{r}$ and $R_{r}$. These
combinations cover the different cases of collision outcomes observed in the
present study. The contours are plotted at various time instances to observe
the evolution of the velocity and pressure fields as the collisions occur.

Figures 8a and 8b show the contours for the collision of two identical sized
drops ($R_{r}=1.0)$ with $U_{r}=2.5$ and $U_{r}=3.0$ respectively. The drops
undergo coalescence with no separation and reflexive separation with satellite
formation for $U_{r}=2.5$ and $U_{r}=3.0$, respectively. For both cases, the
trailing drops approach the leading drops with high velocities while the
pressure remains the same in both drops (due to the same size) before the
collision. The velocity circulating zones are observed close to the trailing
drops due to their high velocities, while no such circulation occurs near the
leading drops. High-pressure zones are created near the point of impact inside
the drops as they come close to each other, giving rise to the pinch-off of
the interface at these points.

For both these cases, the liquid takes a thin disk type of shape after
collision (see second panels of Figs. 8a and 8b). Due to high pressure at the
ends of disk-shaped liquid packet, the ends are stretched in the radial
direction. The pressure at the ends is higher for the case of $U_{r}=3.0$;
thus, the liquid stretches more, making the disk thinner. The liquid is pulled
back towards the axis of the disk due to capillary pull, and it starts to
elongate in the axial direction. The pressure becomes high at the two ends in
the axial direction, stretching the liquid packet in the axial direction. For
the case of $U_{r}=2.5$, the pressure is not high enough to pull the two ends
apart, and the liquid is pulled back due to the action of capillary forces.
Thus, the liquid packet remains moving in the axial direction as a whole. In
the case of $U_{r}=3.0$, the pressure at the ends is high enough to pull the
liquid mass, making it a dumbbell shape where a thin ligament connects two
blobs of liquid. The pressure at the ends of the ligament is higher than the
liquid blobs, and thus, the pinch-off takes place at these two ends of the
ligament, giving rise to a small size satellite drop.

The contours of velocity and pressure for the collisions of unequal-sized
drops are shown in Figs. 8c and 8d. For both these cases, since $U_{r}>1$, the
trailing drops approach the leading drops with high velocities. The pressure
at the point of impact increases as the drop interfaces come close to each
other. Unlike the case of equal-size collisions, the collisions occur with
different pressures in the leading and trailing drops. The pressure of the
leading drop is relatively higher than the trailing drop for the case of
$R_{r}=1.5$ (see panel-1 of Fig. 8c) due to its smaller size. While it is
higher in the trailing drop than the leading drop for the case of a collision
with $R_{r}=0.5$ (see panel-1 of Fig. 8d) for the same apparent reason. The
collision of the high-pressure drop with the low-pressure drop forms a crater-
type structure on either of the two sides of the liquid bulk after the
collision. The side of the high-pressure drop (smaller sized) decides on which
side the crater is to be formed (on the leading side for $R_{r}=1.5$ and on
the trailing side for $R_{r}=0.5$). Intense velocity circulations with high-
pressure zones are present at the ends of the bow-shaped liquid structure,
which try to bend the bow shape and reverse the side on which the crater is
present. Higher  pressure at the ends pushes the liquid toward the center, and
the shapes start to elongate in the axial direction. For the case of
$R_{r}=0.5$, the high velocity trailing drop does not transfer enough momentum
to the liquid bulk, and thus, the axial elongation is not high enough to
overcome the capillary force to separate the liquid into two parts. So, the
liquid shape oscillates in the radial and axial directions while moving in the
axial direction. However, in the case of $R_{r}=1.5$, the high velocity and
bigger size drop transfers large momentum to the liquid bulk on collision. An
asymmetric high-pressure zone is present near the axis of the bow-shaped due
to the difference in the curvature of the interface at the leading and the
trailing side of the liquid bulk. This high-pressure zone pushes the liquid
out from the trailing side of the bow while elongating in the axial direction.
The neck formation starts towards the trailing side of the center due to the
action of the capillary force (panel-4 of Fig. 8c). The pressure in the necked
region increases, and the fluid flows away from the neck (Deka, Biswas, and
Bora, 2023) and the pinch-off takes place at the necking point, dividing the
liquid mass into two parts, the trailing one being the smaller and the leading
one being the larger one.

Figure 10: Phase diagram demarcating different collision outcomes, such as
coalescence and reflective separation with and without satellite drop in
$U_{r}-R_{r}$ space.

In Fig. 9, the kinetic and surface energy time variations are plotted during
the collision process for various collision outcomes. The energy curves are
presented for three cases with distinct collision outcomes, i.e., coalescence
(Fig. 9a), reflexive separation with satellite (Fig. 9b), and reflexive
separation without satellite formation (Fig. 9c). The kinetic and surface
energy presented in Fig. 9 are normalized with the total initial kinetic and
surface energy of the drops, respectively. In the binary collision of drops,
there is a competition between inertial and capillary forces (Pan, Law, and
Zhou, 2008). During the collision process, the capillary action tries to
minimize the surface energy of the liquid while the inertial forces try to
extend the liquid interface by working against them. The combined effect of
both these forces is responsible for deciding the liquid shapes at different
stages of collision. As the two drops approach each other, the surrounding
fluid between the drops drains out, and coalescence of the interfaces occurs.
For all three distinct cases of drop collision, the surface energy decreases
and attains a minimum value due to the combined effect of inertial and
capillary forces after collision. The surface energy starts increasing as the
liquid packet deforms in the axial direction and extends in the radial
direction due to the effect of inertia. The surface energy exhibits maxima
corresponding to a time when the liquid has its maximum axial deformation and
radial expansion, taking a thin disk shape. The capillary forces try to
minimize the surface energy, and the disk is retracted back, extending the
liquid in the axial direction and causing a local minimum to occur again. Due
to inertial effects, the surface energy increases as the liquid grows further
in the axial direction. For coalescence, this process of increase and decrease
in the surface energy is repeated for several cycles unless all the velocity
fluctuations are damped due to viscosity and the combined liquid takes the
shape of a spherical drop with minimum surface energy. However, in the case of
reflexive separation (both with and without satellite formation), no such
oscillations in the surface energy are observed. The total surface energy
becomes almost constant after the reflexive separation of the drops. For all
three distinct collision outcomes, the total kinetic energy of the liquid
decreases as the two drops coalesce with each other, and the axial deformation
of the fluid starts. It takes the minimum value when the two drops are in
maximum axial deformation and maximum radial expansion (the kinetic is stored
as the surface energy of the liquid). The kinetic energy recovery from surface
energy occurs as the liquid stretches in the axial direction and attains a
local maximum value corresponding to the time when the liquid has local
minimum surface energy. In the case of coalescing drops, the total kinetic
energy of liquid oscillates in opposite phases, with the surface energy
showing the competition between the two kinds of energy. However, in the cases
of reflexive separations, the kinetic energy steadily decreases after
attaining a local maximum value. The decrease in the kinetic energy after
impact and its recovery during the axial extension of the liquid is higher in
the case of reflexive separation with satellite formation than in the case of
separation without satellite (see Figs. 9b and 9c). This happens because in
the case of $R_{r}=1.5$, the trailing drop experiences less inertial
resistance from the small size of the leading drop than in the case of equally
sized leading drop for $R_{r}=1.0$.

As discussed earlier, the collision outcome of two drops moving in the same
direction depends on the collision parameters such as radius ratio $R_{r}$ and
velocity ratio ($U_{r}$) between them. Simulations with several combinations
of $R_{r}$ and $U_{r}$ are performed to observe the different collision
outcomes. In Fig. 10, a $R_{r}-U_{r}$ regime map is plotted to distinguish
between various collision outcome regimes. It is noted that the formation of
satellite drops takes place only for the collision of identical-sized drops
($R_{r}=1.0$) with higher values of velocity ratios ($U_{r}>2.5$) (where the
liquid has enough inertia to overcome the capillary forces and extend the
interface into multiple parts).

##  IV Concluding remarks

We have numerically examined the collision dynamics of two drops moving in the
same direction. The relevant collision parameters, namely the velocity and
radius ratios, are varied to observe various collision outcomes. Three
distinct collision outcomes observed in the present study are coalescence,
reflexive separation with satellite, and reflexive separation without
satellite formation. The collision dynamics is explained with the help of the
pressure and velocity contours. The evolution of kinetic and surface energy
for different cases of collisions is also presented. For the collision of
identical-sized drops ($R_{r}=1.0$), the drops undergo coalescence for
$U_{r}\leq 2.5$, while reflexive separation with satellite takes place for
$U_{r}>2.5$. Similarly, in the case of $R_{r}=1.5$, coalescence occurs for
$U_{r}\leq 2.5$, and reflexive separation without formation of satellite drop
(asymmetric reflexive separation) takes place for $U_{r}>2.5$. However, for
the cases of $R_{r}=0.5$, the collisions exhibit coalescence for all the
values of velocity ratios considered in the present study. The temporal
evolution of normalized kinetic and surface energy of the liquid is depicted
for the three distinct collision outcomes. It is evident that the liquid’s
kinetic energy is converted into surface energy during axial deformation and
subsequently released during axial expansion. In the case of coalescence,
oscillations of opposite phases are observed in both kinetic and surface
energy until they are dissipated by viscous effects. Such oscillations are not
evident in drop collisions leading to separation. Finally, through a large
number of simulations, the collision outcomes have been summarized, and the
boundary separating coalescence and reflexive separations is identified.

## Acknowledgements:

A.K.P. acknowledges the support from PMRF fellowship, Government of India and
G.B. acknowledges his gratitude to J. C. Bose National Fellowship of SERB,
Government of India (grant: JBR/2020/000042). K.C.S. thanks the Science and
Engineering Research Board, India for the financial support through grant
CRG/2020/000507. The authors express their sincere appreciation to the high-
performance computing facilities (HPC and Paramsanganak) of IIT Kanpur for
providing computational resources.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## REFERENCES

## References

  * Biswas and Sahu (2021) G. Biswas and K. C. Sahu, “Recent advances in free surface flows,” Mechanical Sciences: The Way Forward , 121–144 (2021).
  * Kumar, Bhardwaj, and Sahu (2020) M. Kumar, R. Bhardwaj, and K. C. Sahu, “Coalescence dynamics of a droplet on a sessile droplet,” Phys. Fluids 32 (2020).
  * Thoroddsen, Etoh, and Takehara (2008) S. T. Thoroddsen, T. G. Etoh, and K. Takehara, “High-speed imaging of drops and bubbles,” Annu. Rev. Fluid Mech. 40, 257–285 (2008).
  * Stone, Stroock, and Ajdari (2004) H. A. Stone, A. D. Stroock, and A. Ajdari, “Engineering flows in small devices: microfluidics toward a lab-on-a-chip,” Annu. Rev. Fluid Mech. 36, 381–411 (2004).
  * Low and List (1982) T. B. Low and R. List, “Collision, coalescence and breakup of raindrops. part i: Experimentally established coalescence efficiencies and fragment size distributions in breakup,” J. Atmos. Sci. 39, 1591–1606 (1982).
  * Qian and Law (1997) J. Qian and C. K. Law, “Regimes of coalescence and separation in droplet collision,” J. Fluid Mech. 331, 59–80 (1997).
  * Pan, Law, and Zhou (2008) K. L. Pan, C. K. Law, and B. Zhou, “Experimental and mechanistic description of merging and bouncing in head-on binary droplet collision,” J. Appl. Phys. 103, 064901 (2008).
  * Nobari, Jan, and Tryggvason (1996) M. R. Nobari, Y. J. Jan, and G. Tryggvason, “Head-on collision of drops—a numerical investigation,” Phys. Fluids 8, 29–42 (1996).
  * Zhang, Li, and Thoroddsen (2009) F. H. Zhang, E. Q. Li, and S. T. Thoroddsen, “Satellite formation during coalescence of unequal size drops,” Phys. Rev. Lett. 102, 104502 (2009).
  * Deka  _et al._ (2019) H. Deka, G. Biswas, S. K. Chakraborty, and A. Dalal, “Coalescence dynamics of unequal sized drops,” Phys. Fluids 31, 012105 (2019).
  * Chaitanya, Sahu, and Biswas (2021) G. S. Chaitanya, K. C. Sahu, and G. Biswas, “A study of two unequal-sized droplets undergoing oblique collision,” Phys. Fluids 33, 022110 (2021).
  * Singh and Saha (2022) S. Singh and A. K. Saha, “Dynamics of two unequal-sized drops coalescence at a liquid-liquid interface,” Phys. Fluids 34, 063604 (2022).
  * Nikolopoulos  _et al._ (2012) N. Nikolopoulos, G. Strotos, K. S. Nikas, and G. Bergeles, “The effect of weber number on the central binary collision outcome between unequal-sized droplets,” Int. J. Heat Mass Transf. 55, 2137–2150 (2012).
  * Tang, Zhang, and Law (2012) C. Tang, P. Zhang, and C. K. Law, “Bouncing, coalescence, and separation in head-on collision of unequal-size droplets,” Phys. Fluids 24 (2012).
  * Cong  _et al._ (2020) H. Cong, L. Qian, Y. Wang, and J. Lin, “Numerical simulation of the collision behaviors of binary unequal-sized droplets at high weber number,” Phys. Fluids 32 (2020).
  * Goyal, Shaikh, and Sharma (2020) N. Goyal, J. Shaikh, and A. Sharma, “Bubble entrapment during head-on binary collision with large deformation of unequal-sized tetradecane droplets,” Phys. Fluids 32 (2020).
  * Al-Diwari and Bayly (2020) K. H. Al-Diwari and A. E. Bayly, “An experimental study of binary collisions of miscible droplets with non-identical viscosities,” Exp. Fluids 61, 50 (2020).
  * Al-Diwari and Bayly (2019) K. H. Al-Diwari and A. E. Bayly, “A new model for the bouncing regime boundary in binary droplet collisions,” Phys. Fluids 31, 027105 (2019).
  * Deka, Biswas, and Bora (2023) H. Deka, G. Biswas, and B. J. Bora, “Head-on collision of dissimilar viscosity drops,” Langmuir 39, 8130–8140 (2023).
  * Paul  _et al._ (2023) A. Paul, B. Ray, K. C. Sahu, and G. Biswas, “An investigation on the impact of two vertically aligned drops on a liquid surface,” Int. J. Multiph. Flow 168, 104588 (2023).
  * Blanchette and Bigioni (2006) F. Blanchette and T. P. Bigioni, “Partial coalescence of drops at liquid interfaces,” Nat. Phys. 2, 254–257 (2006).
  * Ray, Biswas, and Sharma (2010) B. Ray, G. Biswas, and A. Sharma, “Generation of secondary droplets in coalescence of a drop at a liquid–liquid interface,” J. Fluid Mech. 655, 72–104 (2010).
  * Kirar  _et al._ (2020) P. K. Kirar, K. Alvarenga, P. S. Kolhe, G. Biswas, and K. C. Sahu, “Coalescence of drops on the free-surface of a liquid pool at elevated temperatures,” Phys. Fluids 32, 052103 (2020).
  * Kirar  _et al._ (2022) P. K. Kirar, S. D. Pokale, K. C. Sahu, B. Ray, and G. Biswas, “Influence of the interaction of capillary waves on the dynamics of two drops falling side-by-side on a liquid pool,” Phys. Fluids 34, 112114 (2022).
  * Ray, Han, and Cheng (2023) S. Ray, Y. Han, and S. Cheng, “Pinch-off dynamics in unequal-size droplets head-on collision on a wetting surface: Experiments and direct numerical simulations,” Phys. Fluids 35, 122105 (2023).
  * Sprittles (2023) J. E. Sprittles, “Gas microfilms in droplet dynamics: When do drops bounce?” Annu. Rev. Fluid Mech. 56, 67–94 (2023).
  * Amani  _et al._ (2019) A. Amani, N. Balcazar, E. Gutierrez, and A. Oliva, “Numerical study of binary droplets collision in the main collision regimes,” Chem. Eng. J. 370, 477–498 (2019).
  * Liu and Bothe (2016) M. Liu and D. Bothe, “Numerical study of head-on droplet collisions at high weber numbers,” J. Fluid Mech. 789, 785–805 (2016).
  * Testik, Barros, and Bliven (2011) F. Y. Testik, A. P. Barros, and L. F. Bliven, “Toward a physical characterization of raindrop collision outcome regimes,” J. Atmos. Sci. 68, 1097–1113 (2011).
  * Lambert and Variano (2016) R. A. Lambert and E. A. Variano, “Collision of oil droplets with marine aggregates: Effect of droplet size,” J. Geophys. Res. 121, 3250–3260 (2016).
  * Huang, Pan, and Josserand (2019) K. L. Huang, K. L. Pan, and C. Josserand, “Pinching dynamics and satellite droplet formation in symmetrical droplet collisions,” Phys. Rev. Lett. 123, 234502 (2019).
  * Brackbill, Kothe, and Zemach (1992) J. U. Brackbill, D. B. Kothe, and C. Zemach, “A continuum method for modeling surface tension,” J. Comput. Phys. 100, 335–354 (1992).
  * Popinet (2009) S. Popinet, “An accurate adaptive solver for surface-tension-driven interfacial flows,” J. Comput. Phys. 228, 5838–5866 (2009).
  * Popinet (2013) S. Popinet, “Basilisk - reference manual,” (2013).
  * Popinet (2003) S. Popinet, “Gerris: a tree-based adaptive solver for the incompressible euler equations in complex geometries,” J. Comput. Phys. 190, 572–600 (2003).
  * Bell, Colella, and Glaz (1989) J. B. Bell, P. Colella, and H. M. Glaz, “A second-order projection method for the incompressible navier-stokes equations,” J. Comput. Phys. 85, 257–283 (1989).
  * Weymouth and Yue (2010) G. Weymouth and D. K.-P. Yue, “Conservative volume-of-fluid method for free-surface simulations on cartesian-grids,” J. Comput. Phys. 229, 2853–2865 (2010).


# Multi-stage Based Feature Fusion of Multi-Modal Data for Human Activity
Recognition

###### Abstract

To properly assist humans in their needs, human activity recognition (HAR)
systems need the ability to fuse information from multiple modalities. Our
hypothesis is that multimodal sensors, visual and non-visual tend to provide
complementary information, addressing the limitations of other modalities. In
this work, we propose a multi-modal framework that learns to effectively
combine features from RGB Video and IMU sensors, and show its robustness for
MMAct and UTD-MHAD datasets. Our model is trained in two-stage, where in the
first stage, each input encoder learns to effectively extract features, and in
the second stage, learns to combine these individual features. We show
significant improvements of 22% and 11% compared to video only and IMU only
setup on UTD-MHAD dataset, and 20% and 12% on MMAct datasets. Through
extensive experimentation, we show the robustness of our model on zero shot
setting, and limited annotated data setting. We further compare with state-of-
the-art methods that use more input modalities and show that our method
outperforms significantly on the more difficult MMact dataset, and performs
comparably in UTD-MHAD dataset.

Index Terms—  Multi-modal learning, Human Activity Recognition, deep learning,
Multi-modal fusion

††Under review

##  1 Introduction

In recent years, as we have seen a rapid increase in the bodily worn sensors
and it’s applications from tracking GPS to automatically switching between
activities e.g. triathlons, we have seen a boom in the machine learning
algorithms that combine information from multiple sensors for Human Activity
Recognition (HAR). With this rise of bodily worn sensors and smart home
systems [1, 2], we have seen the expansion to vision-based HAR applications in
human-robot interaction, senior care, healthcare, personal fitness, and
surveillance [3, 4]. Thus depending on the type of system used, different
input modalities such as RGB-D, IMU sensors, IoT device information etc can be
used for HAR. Using only IMU data limits the complexity and variety of the
actions and activities that can be detected and classified by the HAR system.
For instance, the HAR system that relies only on IMU data from the inertial
sensor equipped on a wrist might not be able to distinguish between waving and
cleaning a window. Hence, these limitations call for a multimodal approach
that can leverage other data for accurate detections.

Multimodal HAR is a paradigm that uses data from more than one modality (e.g.,
RGB videos, IMU sensors, Skeleton joints, or depth videos) and learns an
effective way to combine these different modalities to improve the accuracy of
the HAR system. With this motivation, our contributions are as follows: _(1)_
We propose a HAR framework that works on multi-modal inputs such as RGB video
and IMU data. _(2)_ We compare single modality vs multi-modal input
performance for two datasets: UTD-MHAD [5] and MMAct [6]. _(3)_ Through
extensive experimentation, we show the effectiveness of our framework even
when limited data is available and in zero-shot setting.

Fig. 1: Left: Overview of our framework. Videos are encoded using an 3D-CNN
architecture: S3D, and IMU data are encoded using 3 blocks of 1D-CNN layers.
These features are then fused and passed through a two layer MLP to predict
action classes. Right: IMU Encoder with three layers of 1D convolution +
Dropout + ReLu, and MLP Classifier with two linear layer, and one non-linear
activation between them.

##  2 Related Work

Our work is focused on learning strong representations for the task of human
activity recognition. In what follows, we divide the existing literature into
two categories based on the number of input modalities used and summarize
these works.

###  2.1 Uni-modal Human Activity Recognition

Single modality based HAR systems have been extensively studied by both
ubiquitous computing and vision communities [7, 8, 9, 10, 11, 1]. Vision-based
activity recognition’s effectiveness mainly stems from the expanding field of
representation learning for image and video features[10, 8, 12]. In addition,
the abundance of annotated data from datasets like Kinetics [13], HowTo100M
[8] has also benefited the vision-based HAR systems. However, the vision-based
systems are typically affected by the recording conditions (camera angles,
viewpoint, lighting, background variations etc.) as well as occlusions. For
target scenarios where privacy is a concern, e.g., medical applications,
obtaining recorded video may be very challenging, or even impossible. In such
scenarios, inertial measurement unit (IMU) signals are more suitable, as they
are generally available on commodity platforms including smartphones and
smartwatches and can record data without disruption of user experience. Due to
this, recent years have seen significant interest in developing sensor-based
activity recognition systems [14, 15, 16]. Although the single modality based
approaches have demonstrated their effectiveness in recognizing activities,
each modality has weaknesses that the other can complement. For example,
sensitivity to occlusions can be overcome by using IMU data, and the
sensitivity of IMU sensors to body-worn positions [17] can be addressed by
using videos and skeleton data.

###  2.2 Multi-modal Human Activity Recognition

The advantages of complementary modalities have been explored in computer
vision for a wide range of tasks. For example, object pose estimation methods
often use RGB + Depth data [18, 19], whereas Visual Question Answering (VQA)
utilizes text questions on RGB images for answering [20, 21]. Action
Recognition tasks have also combined inputs from multiple modalities [22] like
depth and infrared data in addition to RGB videos to estimate predefined human
actions. With the introduction of Transformer [23] based methods, works such
as UniVL [24], Omnivore [25], HAMLET [26] effectively learn to fuse
information from multiple modalities for the task of action recognition. UniVL
[24] combines information from RGB videos and textual inputs to answer
questions about a video and generate captions. Omnivore [25] on the other hand
learns to effectively learn joint representation from multiple input
modalities such as RGB images, videos, and depth maps. This learnt shared
representation is then fine-tuned for action and image classification tasks.
However, the combination of Video and IMU data has seen limited exploration,
though recently, techniques such as [27, 28, 29] have been introduced.
However, these methods generally use skeleton data, or perform fusion and
learn features in a non-synchronous way. In contrast, our framework
specifically works on the synchronized RGB Video and IMU data, and is trained
in a two-stage method that allows for the model to learn strong
representations from individual input modalities in the first stage, and
effectively fuse the information in the second stage.

##  3 Methodology

To address the problems of single modality HAR systems, we introduce a
multimodal framework using two modalities: third-person view RGB video and IMU
data. We discuss the different feature-extracting modules and fusion of these
different features obtained in this section. An overview of the network is
shown in Figure 1.

###  3.1 Feature Encoder for IMU data

IMU data is a multivariate time-series data that (generally) contains data
from accelerometers, gyroscopes, and sometimes magnetometers on the x, y, and
z axes. 1D Convolutional neural networks (CNNs) are known to be effective for
feature learning, as they can parse data across time, thereby capturing the
time-series nature of sensor data [7]. We implement a simple encoder for the
IMU data, similar to [30], but comprising 3 blocks of 1D convolutional layers.
The kernel size is set to 24, 16, and 8 respectively, and the architecture is
shown in Figure 1. The IMU encoder is initialized with random weights during
training.

###  3.2 Feature Encoder for Video

We use the S$3$D network [8] to encode the video features. The S$3$D network
is a video classification network that projects a video into a temporal and
spatial representation and is pre-trained on Kinetics400[13].

###  3.3 End-to-End training

Our framework is trained in two stages. In the first stage, we pre-train both
the video and the IMU encoders separately as a supervised action recognition
task. As a second stage, we fine-tune the Mixed_4c and Mixed_5c layers of the
S$3$D architecture, and all the layers of the IMU encoder to predict action
classes. Features from both the encoders are concatenated together and then
passed through a 2 layer MLP structure to predict the final action classes.

The entire model is trained with a cross-entropy loss for action
classification. Details of the implementation are discussed in Section 4.2.

##  4 Experiments and Results

###  4.1 Dataset

To evaluate the performance of our proposed model in this project, UTD-MHAD
[5] and MMAct [6] are used.

UTD-MHAD dataset contains a total of 861 samples collected using one wearable
inertial sensor and one Kinect camera from 8 subjects, 4 males and 4 females,
performing 27 different activities, repeated 4 times for each activity. The
dataset consists of 4 different modalities including RGB videos, Depth videos,
Skeleton positions, and inertial signals. For our study, only the RGB videos
and inertial signals were used. Same as the original paper, we use odd-
numbered subjects: 1, 3, 5, 7 for training and even-numbered subjects: 2, 4,
6, 8 for testing for all models.

MMAct dataset contains a total of 36764 samples from 37 activities with each
activity repeated 5 times by 20 subjects in 4 different scenes including the
scene of occlusion to overcome the weakness of the vision-based HAR systems.
The dataset consists of 7 modalities including RGB videos and inertial
signals. we follow the cross-subject evaluation protocol by using the samples
from the first 16 subjects for training and the rest for testing.

###  4.2 Implementation

Our model was implemented using the PyTorch [31] framework. Training
hyperparameters are detailed in Table 1. All the training for UTD-MHAD was
done on a single A40 GPU, while MMAct was trained on 4 A40 GPUs.

Table 1: Training hyperparameter details

|  | UTD-MHAD |  | MMAct
---|---|---|---|---
|  | lr |  | Weight decay
---
batch |  | lr |  | Weight decay
---
batch
IMU |  | 1e-4 | 1e-6 | 128 |  | 5e-3 | 5e-5 | 256
Video |  | 1e-3 | 1e-5 | 16 |  | 1e-3 | 1e-5 | 20
IMU + Video |  | 5e-4 | 5e-6 | 16 |  | 1e-4 | 1e-6 | 18

Pre-Processing: Both RGB videos and IMU signals were pre-processed for feature
encoders. RGB frames were sampled at 15HZ for UTD-MHAD and 30 fps for MMAct.
The IMU signals were sampled at 50HZ for UTD-MHAD and a mixture of 100HZ
(acceleration) and 50HZ (gyroscope, orientation) for MMAct. The signals were
padded with zeros whenever a dimension mismatch occurred.

###  4.3 Evaluation Metrics

To compare the performance of these different HAR models we report Top 1
accuracy, Top 5 accuracy, and F1 score on the test set for each of the
datasets.

Table 2: Action recognition performance for Top-1, Top-5 and F1 score compared
with baseline methods.

|  |  | UTD-MHAD |  | MMAct |
---|---|---|---|---|---|---
Method | Data used |  | Top 1 | Top 5 | F1 |  | Top 1 | Top 5 | F1 |
HAMLET [26] | Video + Skeleton + Sensor |  | 95.12 | - | - |  | - | - | - |
MuMu [29] | Video + Depth + Skeleton + Sensor |  | 97.6 | - | - |  | - | - | 76.28 |
CMC-CMKM [32] | Skeleton + Sensor |  | 97.21 | - | - |  | 84.05 | - | - |
VSKD + DASK [28] | Video + Sensor |  | 96.97 | - | 96.38 |  | - | - | 65.83 |
VSKD [28] | Sensor |  | 94.87 | - | - |  | - | - | 61.38 |
Ours (IMU only) | Sensor |  | 85.42 | 97.14 | 85.44 |  | 64.28 | 90.57 | 65.45 |
Ours (Video only) | Video |  | 74.76 | 95.14 | 74.09 |  | 71.89 | 94.92 | 72.86 |
Ours (IMU + Video) | Video + Sensor |  | 96.05 | 100 | 96.05 |  | 84.05 | 98.96 | 84.78 |

###  4.4 Results and Discussions

We compare the effectiveness of our model with single modality inputs in Table
2. We also compare our results against HAMLET [26], MuMu [29], a contrastive
learning based method CMC-CMKM [32], and VSKD [28].

We see that when compared against single modality frameworks, our methods
perform significantly better (10.6% compared to IMU, and 21.29% compared to
Video) across all metrics. When compared to other baseline methods, our method
sees a slight drop in performance for UTD-MHAD dataset, but performs
comparable or better for MMAct. It is worth noting that other methods use
Skeleton information in addition to RGB Video and Sensor data. CMC-CMKM, a
contrastive loss based framework uses Skeleton in lieu of video, however, this
comes at an additional computation cost and speed, as having an accurate
skeleton estimation becomes crucial to their method. As opposed to that, our
methods use video features directly, without needing a post-processing
framework. However, utilizing skeleton aids in the action recognition task,
and we intend to explore this modality further in future work. Although [28]
uses the same modalities as our framework, they propose a novel loss called
DASK that is responsible for the performance improvement. Their student
network that uses IMU data uses a ResNet-18 based model and performs
significantly better than our 1D-CNN based model.

###  4.5 Effect of the size of training dataset

In this experiment, we evaluate the effect of different ratios of the training
data on performance. As can be seen in Table 2, we only see a drop of about
10% on UTD-MHAD and 10% on MMAct when the training sample is reduced to 25%.
Compared to single modality performances, multi-modal method performs
significantly better. This further shows the effectiveness of multi-modal
learning for human activity recognition.

Fig. 2: Action recognition performance for different ratio of training data
for MMAct and UTD-MHAD datasets.

###  4.6 Zero Shot setting

We also test our framework on zero shot setting, and compare against zero shot
on RGB and IMU (When few actions classes in the data are hidden during pre-
training). During the fine-tuning, the action classes stay hidden to ensure
that no data leak happens during the training. We compare the performances on
the test dataset for when single modalities are evaluated for zero shot, and
when one of the modalities have seen the action classes in Figure 3 in multi-
modal setting. For single modality, we see a performance drop of 2% when one
action class is hidden and a drop of 14% when 5 action classes are hidden for
the IMU data. Whereas, for video, we see a larger drop of 4% when one class is
hidden and 12% when 5 classes are hidden. The multi-modal setup also sees a
drop in performance, but performs significantly better to their single modal
counterparts. We further notice that when action classes are hidden from IMU
data input, but are available in the video input, ($IMU^{*}+RGB$) performs
better than ($RGB^{*}+IMU$) setup. This validates our hypothesis that RGB
videos are better at providing complementing information to the model when the
IMU input lacks it.

Fig. 3: Zero shot performances on UTD-MHAD dataset. * indicates the input
modality from which the action classes were hidden. The four colour bars
indicates different input modality combinations and the three columns
represent performances when num_classes=$<x>$ were hidden from the input.
Left: shows the % accuracy, while Right: shows the % F1 score.

##  5 Conclusion

We have explored multimodal HAR using IMU signals and RGB videos, and the role
of pre-training feature encoders for multimodal HAR in this project. We
concluded that multimodal HAR outperforms single modal HAR significantly for
all datasets. Furthermore, training in two-stages helps the individual
encoders to extract relevant information which when fused performs better than
other SOTA works. We show the effectiveness of the multi-modal training when
smaller datasets are used in training, and also in a zero-shot setting. For
the next steps of this project, we would like to explore the effectiveness of
using self-attention in learning joint feature representation from multiple
modalities and learn better feature fusing protocols, and also explore other
modalities for efficient HAR systems.

## References

  * [1] Homay Danaei Mehr and Huseyin Polat,  “Human activity recognition in smart home with deep learning approach,”  in 2019 7th International Istanbul Smart Grids and Cities Congress and Fair (ICSG). IEEE, 2019, pp. 149–153.
  * [2] Yegang Du, Yuto Lim, and Yasuo Tan,  “A novel human activity recognition and prediction in smart home based on interaction,”  Sensors, vol. 19, no. 20, pp. 4474, 2019.
  * [3] Ming Zeng, Le T. Nguyen, Bo Yu, Ole J. Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang,  “Convolutional neural networks for human activity recognition using mobile sensors,”  in 6th International Conference on Mobile Computing, Applications and Services, 2014, pp. 197–205.
  * [4] Shugang Zhang, Zhiqiang Wei, Jie Nie, Lei Huang, Shuang Wang, and Zhen Li,  “A review on human activity recognition using vision-based method,”  Journal of healthcare engineering, vol. 2017, 2017.
  * [5] Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz,  “Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,”  in 2015 IEEE International conference on image processing (ICIP). IEEE, 2015, pp. 168–172.
  * [6] Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami,  “Mmact: A large-scale dataset for cross modal human action understanding,”  in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 8657–8666.
  * [7] Federico Cruciani, Anastasios Vafeiadis, Chris Nugent, Ian Cleland, Paul McCullagh, Konstantinos Votis, Dimitrios Giakoumis, Dimitrios Tzovaras, Liming Chen, and Raouf Hamzaoui,  “Feature learning for human activity recognition using convolutional neural networks,”  CCF Transactions on Pervasive Computing and Interaction, vol. 2, no. 1, pp. 18–32, 2020.
  * [8] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy,  “Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 305–321.
  * [9] Hossein Rahmani and Ajmal Mian,  “3d action recognition from novel viewpoints,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1506–1515.
  * [10] Jun Liu, Amir Shahroudy, Dong Xu, Alex C Kot, and Gang Wang,  “Skeleton-based action recognition using spatio-temporal lstm network with trust gates,”  IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 12, pp. 3007–3021, 2017.
  * [11] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell,  “Long-term recurrent convolutional networks for visual recognition and description,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2625–2634.
  * [12] Joao Carreira and Andrew Zisserman,  “Quo vadis, action recognition? a new model and the kinetics dataset,”  in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308.
  * [13] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman,  “The kinetics human action video dataset,” 2017.
  * [14] Nils Y Hammerla, Shane Halloran, and Thomas Plötz,  “Deep, convolutional, and recurrent models for human activity recognition using wearables,”  arXiv preprint arXiv:1604.08880, 2016.
  * [15] Vishvak S Murahari and Thomas Plötz,  “On attention models for human activity recognition,”  in Proceedings of the 2018 ACM international symposium on wearable computers, 2018, pp. 100–103.
  * [16] Harish Haresamudram, Irfan Essa, and Thomas Plötz,  “Contrastive predictive coding for human activity recognition,”  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5, no. 2, pp. 1–26, 2021.
  * [17] Subhas Chandra Mukhopadhyay,  “Wearable sensors for human activity monitoring: A review,”  IEEE sensors journal, vol. 15, no. 3, pp. 1321–1330, 2014.
  * [18] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, and Silvio Savarese,  “Densefusion: 6d object pose estimation by iterative dense fusion,”  2019\.
  * [19] Pengyuan Wang, HyunJun Jung, Yitong Li, Siyuan Shen, Rahul Parthasarathy Srikanth, Lorenzo Garattoni, Sven Meier, Nassir Navab, and Benjamin Busam,  “Phocal: A multi-modal dataset for category-level object pose estimation with photometrically challenging objects,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 21222–21231.
  * [20] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra,  “Visual dialog,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 326–335.
  * [21] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh,  “Vqa: Visual question answering,”  in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425–2433.
  * [22] Sandy Ardianto and Hsueh-Ming Hang,  “Multi-view and multi-modal action recognition with learned fusion,”  in 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 2018, pp. 1601–1604.
  * [23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [24] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou,  “Univl: A unified video and language pre-training model for multimodal understanding and generation,”  arXiv preprint arXiv:2002.06353, 2020.
  * [25] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra,  “Omnivore: A single model for many visual modalities,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16102–16112.
  * [26] Md Mofijul Islam and Tariq Iqbal,  “Hamlet: A hierarchical multimodal attention-based human activity recognition algorithm,”  in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 10285–10292.
  * [27] Md Mofijul Islam, Mohammad SAMIN Yasar, and Tariq Iqbal,  “Maven: A memory augmented recurrent approach for multimodal fusion,”  IEEE Transactions on Multimedia, 2022.
  * [28] Jianyuan Ni, Raunak Sarbajna, Yang Liu, Anne HH Ngu, and Yan Yan,  “Cross-modal knowledge distillation for vision-to-sensor action recognition,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 4448–4452.
  * [29] Md Mofijul Islam and Tariq Iqbal,  “Mumu: Cooperative multitask learning-based guided multimodal fusion,”,”  AAAI, 2022.
  * [30] Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang,  “Convolutional neural networks for human activity recognition using mobile sensors,”  in 6th international conference on mobile computing, applications and services. IEEE, 2014, pp. 197–205.
  * [31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.,  “Pytorch: An imperative style, high-performance deep learning library,”  Advances in neural information processing systems, vol. 32, 2019\.
  * [32] Razvan Brinzea, Bulat Khaertdinov, and Stylianos Asteriadis,  “Contrastive learning with cross-modal knowledge mining for multimodal human activity recognition,”  arXiv preprint arXiv:2205.10071, 2022.