# Variational Minimizing Parabolic Orbits for the 2-Fixed Center Problems

Ying Lv1 and Shiqing Zhang2
1.School of Mathematics and Statistics, Southwest University, Chongqing
400715, China
2.Mathematical School, Sichuan University, Chengdu 610064, China

ABSTRACT: Using variational minimizing methods,we prove the existence of an
odd symmetric parabolic orbit for the 2-fixed center problems with weak force
type homogeneous potentials.

KEY WORDS: 2-Fixed Center Problems, Odd Symmetric Parabolic Orbits,
Variational Minimizers.

AMS Subject Clasification:34C15,34C25.

##  1 Introduction and Main Results

Sitninkov [1] and Moser [2] and Mathlouthi [3] and Souissi [4]and Zhang [5]
etc. studied the model for the circular restricted 3-body problems: two mass
points of equal mass $m_{1}=m_{2}=\frac{1}{2}$ move in the plane of their
circular orbits such that the center of masses is at rest, and the third small
mass which does not influence the motion of the first two ones moves on the
line perpendicular to the plane containing the first two mass points and going
through the center of mass.

Let $z(t)$ be the coordinate of the third mass point, then $z(t)$ satisfies

$\ddot{z}(t)+\alpha\frac{z(t)}{(|z(t)|^{2}+|r|^{2})^{\alpha/2+1}}=0.$ (1)

Zhang [5] used variational minimizing method to prove:

Theorem 1.1 For the equation (1) with $0<\alpha<2$, there exists one odd
parabolic or hyperbolic orbit .

The 2-fixed center problem is an old problem studied by Euler[6-8] etc. ([9],
[10], [11], [12]): For two masses $1-\mu$ and $\mu$ fixed at $q^{1}=(-\mu,0)$
and $q^{2}=(1-\mu,0)$, the problem is to study the motion $q(t)=(x(t),y(t))$
of the third body with mass $m_{3}>0$. Here, we consider the motion of the
third body attracted by the 2-fixed center masses with general homogeneous
potentials, then it satisfies the following equation:

$\ddot{q}(t)+\frac{\partial V(q)}{\partial q}=0,$ (2)
$V(q)=-\frac{1-\mu}{|q-q^{1}|^{\alpha}}-\frac{\mu}{|q-q^{2}|^{\alpha}}.$ (3)

For $\mu=1/2$, we study the existence for the motion $q(t)=(x(t),y(t))$ of the
third body satisfying $(x(-t),y(-t))=(-x(t),-y(t)),$ here we use variational
minimizing method to prove:

Theorem 1.2 For $(2)-(3)$ with $\mu=\frac{1}{2}$ and $0<\alpha<2$, there
exists an odd symmetrical parabolic-type unbounded orbit.

##  2 Truncation Functional and Its Minimizing Critical Points

In order to find parabolic-type orbit of $(2)-(3)$ , firstly, we restrict
$t\in[-n,n]$ and find solutions of $(2)-(3)$, then let $n\rightarrow+\infty$
to get the parabolic-type orbit. Noticing the symmetry of the equation, we can
find the odd solutions of the following ODE:

$\ddot{q}(t)=\frac{\partial U(q)}{\partial q},$ (4)
$U(q)=\frac{1/2}{|q-q^{1}|^{\alpha}}+\frac{1/2}{|q-q^{2}|^{\alpha}}.$ (5)

We define the functional:

$f(q)=\int_{-n}^{n}(\frac{1}{2}|\dot{q}(t)|^{2}+\frac{1/2}{|q-q^{1}|^{\alpha}}+\frac{1/2}{|q-q^{2}|^{\alpha}})dt,$
(6)

where

$q\in H_{n}=\\{q(t)=(x(t),y(t)):x,y\in
W^{1,2}[-n,n];q(-t)=-q(t),q(t)\not=q^{i},t\in[-n,n]\\}.$ (7)

Since $\forall q\in H_{n},q(0)=0$, then by the famous Hardy-Littlewood-Polya’s
inequality ([9], inequality 256), for $\forall q\in H_{n}$, we have an
equivalent norm :

$\|q\|_{n}=(\int_{-n}^{n}|\dot{q}(t)|^{2}dt)^{1/2}.$

Remark Here we didn’t assume $q(-n)=q(n)=0$ since we want to get the
parabolic-type orbit satisfying

$\max_{t\in R}|q(t)|=+\infty,$ $\min_{t\in R}|\dot{q}(t)|=0.$

We didn’t assume the periodic property for $q(t)$ since we need non-periodic
odd test function in order to get Lemma 2.6.

Lemma 2.1 $f(q)$ is weakly lower semi-continuous(w.l.s.c.) on the closure
$\bar{H}_{n}$ of $H_{n}$.

Proof: (i). It is well-known that the norm and its square are w.l.s.c..

(ii). $\forall\\{q_{m}\\}\subset H_{n}$, if $q_{m}\rightharpoonup q\in H_{n}$
weakly, then by compact embedding theorem, we have the uniformly convergence:

$\max\limits_{-n\leq t\leq n}|q_{m}(t)-q(t)|\rightarrow 0,$

as $m\rightarrow+\infty$, so

$\int_{-n}^{n}\frac{1}{|q_{m}-q^{i}|^{\alpha}}dt\rightarrow\int_{-n}^{n}\frac{1}{|q-q^{i}|^{\alpha}}dt,i=1,2,$

as $m\rightarrow+\infty$. Hence

$\mathop{\underline{\lim}}\limits_{m\rightarrow\infty}f(q_{m})\geq f(q).$

(iii). $\forall\\{q_{m}\\}\subset H_{n}$, if $q_{m}\rightharpoonup
q\in\partial{H}_{n}$ weakly,let

$S=\\{t_{0}\in[-n,n]:q(t_{0})=q_{1}(t_{0}),or,q_{2}(t_{0})\\}$

(1).The Lebesgue measure of $S$ is zero,then $U(q_{m}(t)\rightarrow U(q(t))$
almost everywhere,then by Fatou’s Lemma,$\int_{-n}^{n}U(q)dt$ is $w.l.s.c.$,
it is well-known that the norm and its square are w.l.s.c.,so $f(q)$ is
$w.l.s.c.$.

(2).The Lebesgue measure of $S$:$L(S)>0$ ,then

$\int_{-n}^{n}U(q)dt=+\infty,f(q)=+\infty,$

then by compact embedding theorem, we have the uniformly convergence on $S$:

$\max\limits_{-n\leq t\leq n}|q_{m}(t)-q(t)|\rightarrow 0,$

as $m\rightarrow+\infty$, so on $S$,we have the uniformly convergence:

$\int_{-n}^{n}\frac{1}{|q_{m}-q^{i}|^{\alpha}}dt\rightarrow+\infty,i=1,or,2,$

as $m\rightarrow+\infty$. Hence

$\int_{-n}^{n}U(q_{m}(t))dt\rightarrow+\infty$
$\mathop{\underline{\lim}}\limits_{m\rightarrow\infty}f(q_{m})=+\infty\geq
f(q).$

Lemma 2.2 $f$ is coercive on $\bar{H}_{n}$.

Proof: From the definition of $f(q)$ and Hardy-Littlewood’s inequality,it is
clear that the coercivity
holds($f(q)\rightarrow+\infty,\|q\|\rightarrow+\infty$).

Lemma 2.3(Tonelli, [13], [14]) Let $X$ be a reflexive Banach space, $M\subset
X$ be a weakly closed subset, $f:M\rightarrow R\cup\\{+\infty\\}$, but $f(x)$
is not always $+\infty$ ,suppose $f$ is weakly lower semi-continuous and
coercive($f(x)\rightarrow+\infty,\|x\|\rightarrow+\infty$), then $f$ attains
its infimum on $M$.

Lemma 2.4(Palais’s Symmetry Principle [15]) Let $G$ be a finite or compact
group, $\sigma$ be an orthogonal representation of $G$, let $H$ be a real
Hilbert space, $f:H\rightarrow R$ satisfying

$f(\sigma\cdot x)=f(x),\forall\sigma\in G,\forall x\in H.$

Let

$F\stackrel{{\scriptstyle\triangle}}{{=}}\\{x\in H|\sigma\cdot
x=x,\forall\sigma\in G\\}.$

Then the critical point of $f$ in $F$ is also a critical point of $f$ in $H$.

Lemma 2.5 $f(q)$ attains its infimum on $\bar{H}_{n}$, the minimizer
$\tilde{q}_{\alpha,n}(t)$ is an odd solution.

Proof: Since we had proved Lemmas 2.1-2.2, so in order to apply for Lemma 2.3,
we need to apply for Lemma 2.4 to prove that the critical point of $f(q)$ on
$H_{n}$ is the odd solution of $(4)-(5)$: We define groups
$G_{1}=\\{I_{2\times 2},-I_{2\times 2}\\}$,$G_{2}=\\{1,-1\\}$ and their
actions:

$\sigma_{1}\cdot q(t)=I_{2\times 2}q(t),$ $\sigma_{2}\cdot q(t)=-I_{2\times
2}q(t);$ $\tilde{\sigma}_{1}\cdot q(t)=q(t),$ $\tilde{\sigma}_{2}\cdot
q(t)=q(-t).$

Then it’s easy to prove that $f(q)$ is invariant under
$\sigma_{1},\sigma_{2},\tilde{\sigma}_{1},\tilde{\sigma}_{2},\sigma_{i}\cdot\tilde{\sigma}_{j},\tilde{\sigma}_{j}\cdot\sigma_{i}$
and the fixed point set of the group actions for $G_{1}\times G_{2}$ is just
$H_{n}$, so we can apply for Palais’s Symmetrical Principle.

In order to get the parabolic type solution, we need to prove that

$\tilde{q}_{\alpha,n}(t)\rightarrow\tilde{q}_{\alpha}(t)$

when $n\rightarrow\infty$, and $\tilde{q}_{\alpha}(t)$ has the properties:

$\max_{t\in R}|\tilde{q}_{\alpha}(t)|=+\infty,$ $\min_{t\in
R}|\dot{\tilde{q}}_{\alpha}(t)|=0.$

In order for that, we need some furthermore Lemmas:

Lemma 2.6 There exist constants $c>0$ and $0<\theta<1$ independent of $n$ such
that the variational minimizing value $a_{n}$ for $f(q)$ on $\bar{H_{n}}$
satisfies $a_{n}\leq cn^{\theta}.$

Proof: (i). If $\tilde{q}(t)=(\tilde{x},\tilde{y})\in H_{n}$ is located on
y-axis, then we choose a special odd function defined by

$\tilde{x}=0,\ \tilde{y}=t^{\beta},\ t\in[-n,n],$

where

$\frac{1}{2}<\beta=\frac{l}{m}<\frac{1}{\alpha},$

$l,m$ are odd numbers and $(l,m)=1.$ Then

$\displaystyle f(\tilde{q}(t))$ $\displaystyle=$
$\displaystyle\frac{1}{2}2\int_{0}^{n}\beta^{2}t^{2(\beta-1)}dt+\int_{-n}^{n}[\frac{1/2}{|t^{2\beta}+\frac{1}{4}|^{\alpha/2}}+\frac{1/2}{|t^{2\beta}+\frac{1}{4}|^{\alpha/2}}]dt$
$\displaystyle\leq$
$\displaystyle\frac{\beta^{2}}{2\beta-1}n^{2\beta-1}+\frac{2}{1-\alpha\beta}n^{1-\alpha\beta}.$

Now we define

$\theta=\max(2\beta-1,1-\alpha\beta),$ (8)

$c=\frac{\beta^{2}}{2\beta-1}+\frac{2}{1-\alpha\beta}>0.$ (9)

When

$\frac{1}{2}<\beta=\frac{l}{m}<\frac{1}{\alpha},$

then

$2\beta-1>0,\ 1-\alpha\beta>0$

and $0<\theta<1$. Hence we have

$f(\tilde{q})\leq cn^{\theta}.$

(ii). If $\tilde{q}(t)=(\tilde{x},\ \tilde{y})$ is not on y-axis, we choose a
special odd function on $t$ defined by

$\tilde{x}(t)=t^{\beta},\tilde{y}(t)=0,\ t\in[-n,n],$

where

$\frac{1}{2}<\beta=\frac{l}{m}<\frac{1}{\alpha},$

$l,m$ are odd numbers and $(l,m)=1.$ Then, we have

$\displaystyle f(\tilde{q}(t))$ $\displaystyle\leq$
$\displaystyle\int_{0}^{n}\beta^{2}t^{2(\beta-1)}dt+\int_{0}^{n}[\frac{1}{|t^{\beta}+\frac{1}{2}|^{\alpha}}+\frac{1}{|t^{\beta}-\frac{1}{2}|^{\alpha}}]dt$
$\displaystyle\leq$
$\displaystyle\frac{\beta^{2}}{2\beta-1}n^{2\beta-1}+[\frac{1}{1-\alpha\beta}n^{1-\alpha\beta}+\int_{0}^{n}\frac{1}{|t^{\beta}-\frac{1}{2}|^{\alpha}}dt].$

Now we estimate

$\int_{0}^{n}\frac{1}{|t^{\beta}-\frac{1}{2}|^{\alpha}}dt.$

Let

$t^{\beta}-\frac{1}{2}=\tau^{\beta},$

then $t>\tau$ and

$dt=(\frac{\tau}{t})^{\beta-1}d\tau$

also

$\displaystyle\int_{0}^{n}\frac{1}{|t^{\beta}-\frac{1}{2}|^{\alpha}}dt$
$\displaystyle<$
$\displaystyle\int_{(-\frac{1}{2})^{-\frac{1}{\beta}}}^{(n^{\beta}-\frac{1}{2})^{\frac{1}{\beta}}}{\tau}^{-\alpha\beta}d\tau$
$\displaystyle<$
$\displaystyle\frac{1}{1-\alpha\beta}[n^{1-\alpha\beta}-(-\frac{1}{2})^{-\frac{1}{\beta}(1-\alpha\beta)}].$

Define

$\theta=\max\\{2\beta-1,1-\alpha\beta\\},$
$c=\frac{\beta^{2}}{2\beta-1}+\frac{3}{1-\alpha\beta}>0.$

When

$\frac{1}{2}<\beta=\frac{l}{m}<\frac{1}{\alpha},$

then

$2\beta-1>0,1-\alpha\beta>0$

and $0<\theta<1.$ Hence we also have

$f(\tilde{q})\leq cn^{\theta}.$

Furthermore, for our minimizer, we have

Lemma 2.7 Let $\tilde{q}_{\alpha,n}$ be critical points corresponding to the
minimizing critical values $a_{n}=\mathop{\min}\limits_{H_{n}}f(q),$ then
$\|\tilde{q}_{\alpha,n}\|_{\infty}\rightarrow+\infty,$ when
$n\rightarrow+\infty$.

Proof: By the definition of $f(\tilde{q}_{\alpha,n})$ and Lemma 2.6, we have

$\displaystyle cn^{\theta}$ $\displaystyle\geq$ $\displaystyle
f(\tilde{q}_{\alpha,n})$ $\displaystyle\geq$
$\displaystyle\int_{0}^{n}[\frac{1}{|(x+\frac{1}{2})^{2}+y^{2}|^{\alpha/2}}+\frac{1}{|(x-\frac{1}{2})^{2}+y^{2}|^{\alpha/2}}]dt.$

We notice that

$(x+\frac{1}{2})^{2}+y^{2}\leq 2(x^{2}+y^{2})+\frac{5}{4},$
$(x-\frac{1}{2})^{2}+y^{2}\leq(x^{2}+y^{2})+\frac{1}{4},$

so

$\displaystyle cn^{\theta}$ $\displaystyle\geq$
$\displaystyle\int_{0}^{n}\frac{dt}{(2\|\tilde{q}_{\alpha,n}\|_{\infty}^{2}+\frac{5}{4})^{\alpha/2}}+\frac{dt}{(\|\tilde{q}_{\alpha,n}\|_{\infty}^{2}+\frac{1}{4})^{\alpha/2}}$
$\displaystyle\geq$
$\displaystyle\frac{2n}{(2\|\tilde{q}_{\alpha,n}\|_{\infty}^{2}+\frac{5}{4})^{\alpha/2}}.$

Hence

$\|\tilde{q}_{\alpha,n}\|_{\infty}^{2}\rightarrow+\infty,$ (10)

as $n\rightarrow+\infty$.

Lemma 2.8 $\int_{a}^{b}|\dot{\tilde{q}}_{\alpha,n}|^{2}dt$ is uniformly
bounded on any compact set $[a,b]\subset R$.

Proof: Since the system is autonomous, so for any given $\alpha,n$, along the
solution $\tilde{q}_{\alpha,n}(t)$, the energy $h(t)$ is conservative, i.e., a
constant $h=h(\alpha,n)$:

$\frac{1}{2}|\dot{\tilde{q}}_{\alpha,n}|^{2}-\frac{1/2}{{|\tilde{q}}_{\alpha,n}-q^{1}|^{\alpha}}-\frac{1/2}{{|\tilde{q}}_{\alpha,n}-q^{2}|^{\alpha}}=h.\\\
$ (11)

By the above energy relationship and the definition of the functional $f$, we
have

$\displaystyle f(\tilde{q}_{\alpha,n})$ $\displaystyle=$
$\displaystyle\int_{-n}^{n}(\frac{1}{2}|\dot{\tilde{q}}_{\alpha,n}|^{2}+\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{1}|^{\alpha}}+\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{2}|^{\alpha}})dt$
$\displaystyle=$
$\displaystyle\int_{-n}^{n}(\frac{1}{2}|\dot{\tilde{q}}_{\alpha,n}|^{2}-\frac{1/2}{{|\tilde{q}}_{\alpha,n}-q^{1}|^{\alpha}}-\frac{1/2}{{|\tilde{q}}_{\alpha,n}-q^{2}|^{\alpha}})dt$
$\displaystyle+$ $\displaystyle
2\int_{-n}^{n}\frac{1/2}{{|\tilde{q}}_{\alpha,n}-q^{1}|^{\alpha}}+\frac{1/2}{{|\tilde{q}}_{\alpha,n}-q^{2}|^{\alpha}}dt$
$\displaystyle=$ $\displaystyle
2nh+2\int_{-n}^{n}\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{1}|^{\alpha}}+\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{2}|^{\alpha}}dt.$

By Lemma 2.6, we have

$\displaystyle cn^{\theta}$ $\displaystyle\geq$ $\displaystyle
2nh+2\int_{-n}^{n}(\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{1}|^{\alpha}}+\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{2}|^{\alpha}})dt,$

and

$h\leq\frac{c}{2}n^{\theta-1}-\frac{1}{n}\int_{-n}^{n}(\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{1}|^{\alpha}}+\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{2}|^{\alpha}})dt\leq\frac{c}{2}n^{\theta-1}$
(12)

(1).When $n$ is large enough,$|\tilde{q}_{\alpha,n}(t)-q^{i}|$ has uniformly
positive lower bound,that is, $\min_{a\leq t\leq
b}|\tilde{q}_{\alpha,n}(t)-q^{i}|\geq c>0,$ then we have

$\displaystyle\int_{a}^{b}\frac{1}{2}|\dot{\tilde{q}}_{\alpha,n}|^{2}$
$\displaystyle=$ $\displaystyle
h(b-a)+\int_{a}^{b}[\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{1}|^{\alpha}}+\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{2}|^{\alpha}}]dt$
$\displaystyle\leq$ $\displaystyle\frac{c}{2}(b-a)+c^{-\alpha}(b-a).$

(2).There exist $i_{0}=1$ or $2$ and a sequence ${t_{n}}\subset[a,b]$ such
that $\tilde{q}_{\alpha,n}(t_{n})\rightarrow q^{i_{0}}$,then since
$0<\alpha<2$,the potential is weak force potential,so when $n$ is large,we
have

$\int_{a}^{b}[\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{1}|^{\alpha}}+\frac{1/2}{|\tilde{q}_{\alpha,n}-q^{2}|^{\alpha}}]dt\leq
M,$
$\int_{a}^{b}\frac{1}{2}|\dot{\tilde{q}}_{\alpha,n}|^{2}dt\leq\frac{c}{2}(b-a)+M.$

##  3 Proof of Theorem 1.2

By $\tilde{q}_{\alpha,n}(0)=0$ and Cauchy-Schwarz inequality and Lemma 2.8 we
have

$|\tilde{q}_{\alpha,n}(t)|=|\int_{0}^{t}\dot{\tilde{q}}_{\alpha,n}(s)ds|\leq(b-a)^{1/2}[\int_{a}^{b}|\dot{\tilde{q}}_{\alpha,n}|^{2}ds]^{1/2}\leq
M_{1},$

so we have

(i). $\\{\tilde{q}_{\alpha,n}\\}$ is uniformly bounded on any compact set of
$R$.
By Cauchy-Schwarz inequality and Lemma 2.8 we have

$|\tilde{q}_{\alpha,n}(t_{2})-\tilde{q}_{\alpha,n}(t_{1})|=|\int_{t_{1}}^{t_{2}}\dot{\tilde{q}}_{\alpha,n}(s)ds|\leq[\int_{a}^{b}|\dot{\tilde{q}}_{\alpha,n}|^{2}ds]^{1/2}(t_{2}-t_{1})^{1/2}\leq
M_{2}(t_{2}-t_{1})^{1/2},$

so we have

(ii). $\\{\tilde{q}_{\alpha,n}\\}$ is uniformly equi-continuous on any
$[a,b]\subset R$.
Now we can apply Ascoli-Arzel$\grave{a}$ Theorem, we know
$\\{\tilde{q}_{\alpha,n}\\}$ has a sub-sequence converging uniformly to a
limit $\tilde{q}_{\alpha}(t)$ on any compact set of $R$, and
$\tilde{q}_{\alpha}(t)$ is a solution of (2.2) . By the energy conservation
law and Lemmas 2.7-2.8, we have

$h=\frac{1}{2}|\dot{\tilde{q}}_{\alpha}|^{2}-\frac{1}{2}(\frac{1}{|\tilde{q}_{\alpha}-q^{1}|^{\alpha}}+\frac{1}{|\tilde{q}_{\alpha}-q^{2}|^{\alpha}})=0.$

Then by Corollary 2.3 of [20], we have

$\frac{1}{2}|\dot{\tilde{q}}_{\alpha}|^{2}=\frac{1/2}{|\tilde{q}_{\alpha}-q^{1}|^{\alpha}}+\frac{1/2}{|\tilde{q}_{\alpha}-q^{2}|^{\alpha}}\geq[2^{\frac{\alpha+2}{2}}][2|\tilde{q}_{\alpha}|^{2}+\frac{1}{2}]^{-\alpha/2}.$
(13)

Now we claim
(a).

$max_{t\in R}|\tilde{q}_{\alpha}(t)|=+\infty.$ (14)

In fact, if $\exists\beta>0$ such that

$|\tilde{q}_{\alpha}|<\beta,\forall t\in R.$

By (13), $\exists\gamma>0$ such that

$|\dot{\tilde{q}}_{\alpha}|>\gamma,\forall t\in R.$

Then when $n$ is large,we have

$|\dot{\tilde{q}}_{\alpha,n}|>\gamma,\forall t\in R.$ $\displaystyle
cn^{\theta}\geq\int_{-n}^{n}|\dot{\tilde{q}}_{\alpha,n}|^{2}>2n\gamma^{2},$

which is a contradiction.

Now by (13) we have
(b).

$\min_{t\in R}|{\dot{\tilde{q}}_{\alpha}}(t)|=0.$ (15)

##  4 Acknowledgements

The authors sincerely thank the referee for his/her many valuable comments and
remarks which helped us revising the paper, we aslo thank the supports of NSF
of China and a research fund for the Doctoral program of higher education of
China.

## References

  * [1] K. Sitninkov, Existence of oscillating motion for the three-body problem, J. Dokl. Akad. Nauk USSR 133(1960), 303-306.
  * [2] J. Moser, Stable and random motions in dynamical systems,Ann.Math.Studies 77, Princeton Univ. Press, 1973.
  * [3] S. Mathlouthi , Periodic orbits of the restricted three-body problem,Trans.AMS 350(1998), 2265-2276.
  * [4] C. Souissi , Existence of parabolic orbits for the restricted three-body problem, Annals of University of Craiova,Math. Comp. Sci. Ser. 31(2004), 85-93.
  * [5] S. Q. Zhang, Variational minimizing parabolic orbits for the restricted 3-body problems, Preprint, 2010.
  * [6] M. Euler, De motu coproris ad duo centra virium fixa attracti. Nov. Comm. Acad. Sci. Imp. Petrop. 1766; 10: 207-42.
  * [7] M. Euler, De motu coproris ad duo centra virium fixa attracti. Nov. Comm. Acad. Sci. Imp. Petrop 1767; 11: 152-84.
  * [8] M. Euler, Probleme un corps etant attire en raison reciproque quarree des distances vers vers deux points fixes donnes trouver les cas ou la courbe decrite par ce corps sera algebrique. Hist. Acad. Roy. Sci. Bell. Lett. Berlin 1767; 2: 228-49.
  * [9] V. M. Alexeev, Generalized three-dimensional problem of two fixed centers of gravitation-a classification of movements. Bull. Inst. Theoret. Astron. 1965; 10: 241-71.
  * [10] A. Macjejewski, M. Przybylska, Non-integrability of the generalized two fixed centres problem. Celestial Mech. Dynam. Astronom. 2004; 89: 145-164.
  * [11] C. Marchal, On quasi-integrable problems, the example of the artificial satellites perturbed by the Earth’s zonal harmonics. Celestial Mech. Dynam. Astronom. 1986; 38: 377-387.
  * [12] H. Varvoglis, C. H. Vozikis, K. Wodnar, The two fixed centers: An exceptional integrable system. Celestial Mech. Dynam. Astronom. 2004; 89: 343-56.
  * [13] A. Ambrosetti, V. Coti Zelati, Periodic solutions of singular Lagrangian systems, Birkh$\ddot{a}$user, Basel, 1993.
  * [14] J. Mawhin, M. Willem, Critical point theory and Hamiltonian systems, Springer, 1989.
  * [15] R. Palais, The principle of symmetric criticality, CMP 69(1979), 19-30.
  * [16] S. V. Bolotin, Existence of homoclinic motions,Vestnik Moskov Univ. ser. I Mat. Mekh. 6(1983), 98-103.
  * [17] K. C. Chang, Infinite dimensional Morse thory and multiple solution problems, Progress in Nonlinear Diff. Equ. and their Appl., Vol.6, Birkh$\ddot{a}$ser, 1993.
  * [18] W. Gordon, A minimizing property of Keplerian orbits Amer. J. Math. 1977; 99: 961-71.
  * [19] G. Hardy, J. Littlewood, G. Polya, Inequalities. Second ed. Cambridge Univ. Press: Cambridge 1952.
  * [20] Y. M. Long, S. Q. Zhang, Goemetric characterizations for variational minimization solutions of the 3–body problems, Acta Math. Sinica 16(2000), 579–592.
  * [21] R. McGehee, Parabolic orbits of the restricted three-body problem, Academic Press, New York and London, 1973.
  * [22] H. Poincar$\acute{e}$, Les M$\acute{e}$thodes Nouvelles de la M$\acute{e}$canique C$\acute{e}$leste, Gauthier-Villars, Paris, 1899.
  * [23] P. H. Rabinnowtz, On the existence of periodic solutions for a class of symmetric Hamiltonian systems, Nonlinear Anal. 11(1987), 595-611.
  * [24] P. H. Rabinnowtz, Homoclinic orbits for a class of Hamiltonian systems, Proc. Roy. Soc. Edinburgh Sect. A 114(1990), 33-38.
  * [25] E. Sere, Existence of infinitely many homoclinics Hamiltonian systems, Math. Z. 209(1992), 27-42.
  * [26] E. Serra, S. Terracini, Collisionless periodic solutions to some 3-body problems, Arch. Rational. Mech. Anal. 120(1992), 305-325.
  * [27] K. Tanaka, Homoclinic orbits for a singular second order Hamiltonian system, Ann. Inst. H. Poincar$\acute{e}$ Anal. Non Lin$\acute{e}$aire 7(1990), 427-438.
  * [28] S. Q. Zhang, Q. Zhou, R. Liu, New periodic solutions for 3-body problems, Celestial Mechanics and Dynamical Astronomy 88(2004), 365-378.


# Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing
Modalities

Jiandian Zeng  State Key Laboratory of IoT for Smart City, University of
MacauMacauChina<EMAIL_ADDRESS>,  Tianyi Liu  Shanghai Jiao Tong
UniversityShanghaiChina<EMAIL_ADDRESS>and  Jiantao Zhou  State Key
Laboratory of IoT for Smart City, University of MacauMacauChina
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Multimodal sentiment analysis has been studied under the assumption that all
modalities are available. However, such a strong assumption does not always
hold in practice, and most of multimodal fusion models may fail when partial
modalities are missing. Several works have addressed the missing modality
problem; but most of them only considered the single modality missing case,
and ignored the practically more general cases of multiple modalities missing.
To this end, in this paper, we propose a Tag-Assisted Transformer Encoder
(TATE) network to handle the problem of missing uncertain modalities.
Specifically, we design a tag encoding module to cover both the single
modality and multiple modalities missing cases, so as to guide the network’s
attention to those missing modalities. Besides, we adopt a new space
projection pattern to align common vectors. Then, a Transformer encoder-
decoder network is utilized to learn the missing modality features. At last,
the outputs of the Transformer encoder are used for the final sentiment
classification. Extensive experiments are conducted on CMU-MOSI and IEMOCAP
datasets, showing that our method can achieve significant improvements
compared with several baselines.

Multimodal Sentiment Analysis, Missing Modality, Joint Representation

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2022; Madrid, Spain††booktitle: Proceedings
of the 45th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain††price:
15.00††doi: 10.1145/3477495.3532064††isbn: 978-1-4503-8732-3/22/07††ccs:
Information System Multimodal Sentiment Analysis

##  1\. Introduction

Nowadays, sentiment analysis has attracted intensive interest in extracting
human’s emotion and opinion (Wang et al., 2019; Zhang et al., 2020b), among
which multimodal sentiment analysis is becoming an especially popular research
direction with the massive amounts of online content. Besides, it has been
shown that combining different modalities can learn complementary features,
resulting in better joint multimodal representations (Shraga et al., 2020;
Springstein et al., 2021). Most prior works on multimodal fusion (Chen et al.,
2017; Xu et al., 2018; Xu et al., 2020a) assumed that all modalities are
always available when training and testing. However, in real life, we often
encounter scenarios that partial modalities could be missing. For example, as
shown in Fig. 1, the visual features may be blocked due to the non-coverage of
camera; the acoustic information may be unavailable due to the enormous
ambient noise; and the textual information may be absent due to the privacy
issue. Therefore, how to handle missing modalities is emerging as a hot topic
in the multimodal area.

Figure 1. Examples of missing modalities.

Previous works (Mazumder et al., 2010; Shao et al., 2013; Parthasarathy and
Sundaram, 2020) simply discarded missing modalities or utilized matrix
completion methods to impute missing modalities, and somewhat degraded overall
performance. Zhao et al. (Shao et al., 2013) completed the kernel matrices of
the modality using the common instances in different modalities. In
(Parthasarathy and Sundaram, 2020), the visual modality was ablated when
training with missing data. Owing to the strong learning ability of deep
learning, recent works have employed neural networks to learn latent
relationships among available modalities. To tackle the missing modality
problem, Tran et al. (Tran et al., 2017) first identified the general problem
of missing modality in multimodal data, and proposed a Cascaded Residual Auto-
encoder (CRA) network to learn complex relationship from different modalities.
More recently, Zhao et al. (Zhao et al., 2021) adopted cycle consistency
learning with CRA to recover missing modalities. Yuan et al. (Yuan et al.,
2021) designed a Transformer-based feature reconstruction network to guide the
extractor in obtaining the semantics of missing modality features. However,
most of the above works all assumed that there is only one missing modality,
and ignored the practically more general cases of multiple modalities missing.
That is, they require training a new model to fit each missing modality case,
which is both costly and inconvenient. In reality, the pattern of missing
modalities could be uncertain, e.g., one or two modalities are randomly
absent. To tackle the above issues, two challenges should be addressed: 1)
will the model still work when multiple modalities are absent? and 2) how to
learn robust joint representations when the missing modalities are uncertain?

In this paper, we propose a Tag-Assisted Transformer Encoder (TATE) network to
learn complementary features among modalities. For the first challenge, we
design a tag encoding module to mark missing modalities, aiming to direct the
network’s attention to absent modalities. As will be shown later, the attached
tag not only can cover both the single modality and multiple modalities absent
situations, but also can assist in joint representation learning. For the
second challenge, we first adopt the Transformer (Vaswani et al., 2017) as the
extractor to capture intra-modal features, and then apply a two-by-two
projection pattern to map them into a common space. After that, the pre-
trained network trained with full modalities is utilized to supervise the
encoded vectors. At last, the outputs generated by a Transformer encoder are
fed into a classifier for sentiment prediction. Our contributions are
summarized as follows:

  * •

We propose the TATE network to handle the multiple modalities missing problem
for multimodal sentiment analysis. The code is publicly
available111https://github.com/JaydenZeng/TATE.

  * •

We design a tag encoding module to cover both the single modality and multiple
modalities absent situations, and adopt a new common space projection module
to learn joint representations.

  * •

Our proposed model TATE achieves significant improvements compared with
several benchmarks on CMU-MOSI and IEMOCAP datasets, validating the
superiority of our model.

##  2\. Related Works

In this section, we first introduce the concept of multimodal sentiment
analysis, and then review the related methods of handing missing modalities.

###  2.1. Multimodal Sentiment Analysis

As a core branch of sentiment analysis (Zhuang et al., 2020; Geng et al.,
2021), multimodal sentiment analysis has attracted significant attention in
recent years (Poria et al., 2018; Yu et al., 2019; Mai et al., 2021; Stappen
et al., 2021). Compared to a single modality case, multimodal sentiment
analysis is more challenging due to the complexity of handling and analyzing
data from different modalities.

To learn joint representations of multimodal, three multimodal fusion
strategies are applied: 1) early fusion directly combines features of
different modalities before the classification. Majumder et al. (Majumder et
al., 2018) proposed a hierarchical fusion strategy to fuse acoustic, visual
and textual modalities, and proved the effectiveness of two-by-two fusion
pattern; 2) late fusion adopts the average score of each modality as the final
weights. Guo et al. (Guo et al., 2017) adopted an online early-late fusion
scheme to explore complementary relationship for the sign language
recognition, where late fusion further aggregated features combined by the
early fusion; and 3) intermediate fusion utilizes a shared layer to fuse
features. Xu et al. (Xu et al., 2020b) constructed the decomposition and
relation networks to represent the commonality and discrepancy among
modalities. Hazarika et al. (Hazarika et al., 2020) designed a multimodal
learning framework that can learn modality-invariant and modality-specific
representations by projecting each modality into two distinct sub-spaces.
However, few of the above multimodal fusion models can handle the cases when
partial modalities are missing.

###  2.2. Missing Modalities Methods

In recent years, many works focused on handing the missing modality problem,
and they can be generally categorized into two groups: 1) generative methods
(Baldi, 2012; Tran et al., 2017; Du et al., 2018; Shang et al., 2017; Zhang et
al., 2020a); and 2) joint learning methods (Pham et al., 2019; Wang et al.,
2020; Zhao et al., 2021; Yuan et al., 2021).

Generative methods learn to generate new data with similar distributions to
obey the distribution of the observed data. With the ability to learn latent
representations, the auto-encoder (AE) (Baldi, 2012) is widely used. Vincent
et al. (Vincent et al., 2008) extracted features with AE based on the idea of
making the learned representations robust to partial corruption of the input
data. Kingma et al. (Kingma and Welling, 2014) designed a Variational Auto-
Encoder (VAE) to infer and learn features with simple ancestral sampling.
Besides, inspired by the residual connection network (He et al., 2016), Tran
et al. (Tran et al., 2017) proposed a Cascaded Residual Auto-encoder (CRA) to
impute data with missing modality, which combined a series of residual AEs
into a cascaded architecture to learn relationships among different
modalities. As for the Generative Adversarial Networks (GAN) (Goodfellow et
al., 2014), Shang et al. (Shang et al., 2017) treated each view as a separate
domain, and identified domain-to-domain mappings via a GAN using randomly-
sampled data from each view. Besides, the domain mapping technique is also
considered to impute missing data. Cai et al. (Cai et al., 2018) formulated
the missing modality problem as a conditional image generation task, and
designed a 3D encoder-decoder network to capture modality relations. They also
incorporated the available category information during training to enhance the
robustness of the model. Moreover, Zhao et al. (Zhang et al., 2020a) developed
a cross partial multi-view network to model complex correlations among
different views, where multiple discriminators are used to generate missing
data.

Joint learning methods try to learn joint representations based on the
relations among different modalities (Pham et al., 2019; Kim et al., 2020;
Akbari et al., 2021). Based on the idea that the cycle consistency loss can
retain maximal information from all modalities, Pham et al. (Pham et al.,
2019) investigated learning robust representations via cyclic translations
from source to target modalities. Zhao et al. (Zhao et al., 2021) also applied
cycle consistency learning for missing modality imputation, where the CRA-
based cross-modality imagination module is designed based on paired multimodal
data. More recently, Yuan et al. (Yuan et al., 2021) utilized the Transformer
to extract intra-modal and inter-modal relations, and designed a Transformer-
based feature reconstruction network to reproduce the semantics of missing
modality.

However, most of the above works can only handle the scenarios of missing a
single modality, and cannot satisfactorily deal with multiple modalities
missing cases since they need to train a new model for each case. As will be
clear soon, our works differs the above works in several ways: 1) a tag
encoding module is designed to cover all uncertain missing cases; and 2) a new
mapping method is applied to learn joint representations in the common space
projection module.

Figure 2. Workflow of the proposed framework. The information flow goes two
branches: 1) one goes the pre-trained network, which is trained with full
modality data; and 2) another goes to the left multihead attetion module for
further encoding.

##  3\. Methodology

In this section, we first give the problem definition and associated
notations. Then, we present the overall workflow of the proposed architecture
and the detailed modules.

###  3.1. Problem Definition and Notations

Given a multimodal video segment that contains three modalities:
$S=[X_{v},X_{a},X_{t}]$, where $X_{v}$, $X_{a}$ and $X_{t}$ denote visual,
acoustic and textual modalities respectively. Without loss of generality, we
use $X_{m}^{\prime}$ to represent the missing modality, where
$m\in\\{v,a,t\\}$. For instance, assuming that the visual modality and
acoustic modality are absent, and the multimodal representation can be denoted
as $[X_{v}^{\prime},X_{a}^{\prime},X_{t}]$. The primary task is to classify
the overall sentiment (positive, neutral, or negative) under uncertain missing
modalities.

###  3.2. Overall Framework

As can be seen in Fig. 2, the main workflow is as follows: for a given video
segment, assuming that the visual modality and acoustic modality are missing,
we first mask these missing modalities as 0, and then extract the remaining
raw features. Afterwards, the masked multimodal representation goes through
two branches: 1) one is encoded by a pre-trained model, which is trained with
all full modality data, and 2) another goes through the tag encoding module
and the common space projection module to acquire aligned feature vectors.
Then, the updated representations are processed by a Transformer encoder, and
we calculate the forward similarity loss between the pre-trained vectors and
the encoder outputs. Meanwhile, the encoded outputs are fed into a classifier
for the sentiment prediction. At last, we compute the backward reconstruction
loss and the tag recovery loss to supervise the joint representation learning.
Each module will be introduced clearly in following sub-sections.

###  3.3. Multi-Head Attention

Transformer (Vaswani et al., 2017) not only plays a great role in the Natural
Language Processing (NLP) community, but also shows excellent representational
capabilities in other areas, such as Computer Vision (CV) (Chu et al., 2021).
Instead of using an RNN based structure to capture the sequential information,
we employ the Transformer to generate the contextual representation of each
modality respectively, where the key component of multi-head dot-product
attention can be formalized as follows:

(1) $Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d}})V,$

where $Q$, $K$ and $V$ are the query, the key, and the value respectively, and
$d$ is the dimension of the input.

Instead of utilizing the single attention, the multi-head attention is applied
to obtain more information from different semantic spaces:

(2) $\displaystyle E_{M}$ $\displaystyle=MultiHead(Q,K,V)$
$\displaystyle=Concat(head_{1},head_{2},...,head_{h})W^{o},$

where $W^{o}\in\mathbb{R}^{d\times d}$ is a weight matrix, $h$ is the head
number. Given the input $E$, the $i$-th $head_{i}$ is calculated as follow:

(3) $head_{i}=Attention(EW_{i}^{Q},EW_{i}^{K},EW_{i}^{V})$

where $W_{i}^{Q}\in\mathbb{R}^{\frac{d}{h}\times\frac{d}{h}}$,
$W_{i}^{K}\in\mathbb{R}^{\frac{d}{h}\times\frac{d}{h}}$ and
$W_{i}^{V}\in\mathbb{R}^{\frac{d}{h}\times\frac{d}{h}}$ are the $i$-th weight
matrices of the query, the key and the value.

Therefore, the updated modality representations can be formulated as follows:

(4) $\displaystyle
E_{v}=MultiHead(X_{v}^{\prime},X_{v}^{\prime},X_{v}^{\prime}),$ $\displaystyle
E_{a}=MultiHead(X_{a}^{\prime},X_{a}^{\prime},X_{a}^{\prime}),$ $\displaystyle
E_{t}=MultiHead(X_{t},X_{t},X_{t}).$

###  3.4. Tag Encoding

Figure 3. Examples of modality tags. (a) modality tag with one missing
modality. (b) modality tag with two missing modalities.

To specify uncertain missing modalities, we employ a tag encoding module to
mark them, and direct network’s attention to these disabled modalities. In our
settings, we adopt 4 digits (“0” or “1”) to label missing modalities. If
partial modalities of the input are missing, we set the first digit as “0”,
otherwise “1”. Besides, the last three digits are used to mark the
corresponding visual, acoustic and textual modalities. As can be seen in Fig.
3, we give two examples about modality tags: in Fig. 3a, the acoustic modality
is missing, and the tag is set as “0010”; for multiple modalities missing
cases (Fig. 3b), we set the tag as “0110” to mark visual and acoustic
modalities. The benefits are twofold: 1) the tag encoding module can cover
both single and multiple modalities missing conditions; and 2) the encoded
tags can complementarily assist in the learning of the joint representations.
To simplify mathematical expression, we denote all tags as $E_{tag}$.

###  3.5. Common Space Projection

After the tag encoding module, we now project three modalities into the common
space. Previous works (Xu et al., 2020b; Hazarika et al., 2020) that directly
utilized simple feed-forward neural layers with same parameters for the
projection, which may be failed when there are more than two modalities. To
tackle the issue, we adopt a two-by-two projection pattern to acquire a more
general space. As shown in Fig. 4, for each single modality, we first obtain
the self-related common space based on the following linear transformation:

(5) $\displaystyle C_{v}=[W_{va}E_{v}||W_{vt}E_{v}],$ $\displaystyle
C_{a}=[W_{va}E_{a}||W_{ta}E_{a}],$ $\displaystyle C_{t}=[W_{vt}E_{t}\hskip
3.00003pt||W_{ta}E_{t}],$

where $W_{va}$, $W_{vt}$ and $W_{ta}$ are all weight matrices, and $||$
denotes the vertical concatenating operation. Then, we concatenate all common
vectors and the encoded tag to eventually obtain the common joint
representations $E_{all}$:

(6) $E_{all}=[C_{v}||C_{a}||C_{t}||E_{tag}].$

Figure 4. Illustration of the common space projection.

###  3.6. Transformer Encoder-Decoder

To effectively model the long-term dependency of the intra-modal and the
inter-modal information, we employ one sub-layer in Transformer (Vaswani et
al., 2017) to manage the information flow. As illustrated in Section 3.3, the
encoded outputs $E_{out}$ can be accessed by the multi-head attention and
feed-forward networks:

(7) $\displaystyle E_{out}=MultiHead(E_{all},E_{all},E_{all}),$ $\displaystyle
E_{out}=relu(E_{out}W_{e}^{1}+b_{e}^{1})W_{e}^{2}+b_{e}^{2},$

where the query, the key, and the value are the same input $E_{all}$,
$W_{e}^{1}$, and $W_{e}^{2}$ are two weight matrices, $b_{e}^{1}$ and
$b_{e}^{2}$ are two learnable biases.

Similarly, the decoded outputs $D_{out}$ are formulated as follows:

(8) $\displaystyle D_{out}=MultiHead(E_{out},E_{out},E_{out}),$ $\displaystyle
D_{out}=relu(D_{out}W_{o}^{1}+b_{o}^{1})W_{o}^{2}+b_{o}^{2},$

where $W_{o}^{1}$, $W_{o}^{2}$, $b_{o}^{1}$, and $b_{o}^{2}$ are parameters.

###  3.7. Training Objective

The overall training objective ($\mathcal{L}_{total}$) is expressed as:

(9)
$\mathcal{L}_{total}=\mathcal{L}_{cls}+\lambda_{1}\mathcal{L}_{forward}+\lambda_{2}\mathcal{L}_{backward}+\lambda_{3}\mathcal{L}_{tag},$

where $\mathcal{L}_{cls}$ is the classification loss, $\mathcal{L}_{forward}$
is the forward differential loss, $\mathcal{L}_{backward}$ is the backward
reconstruction loss, $\mathcal{L}_{tag}$ is the tag recovery loss, and
$\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are the corresponding weights.
We now introduce the loss terms in details.

####  3.7.1. Forward Differential Loss ($\mathcal{L}_{forward}$)

As illustrated in Fig. 2, the forward loss is calculated by the difference
between the pre-trained output ($E_{pre}$) and the Transformer encoder output
($E_{out}$). Simiar to (Zhao et al., 2021), the pre-trained model is trained
with full modality data, where features from three modalities are directly
concatenated for classification. Thus, we employ the differential loss to
guide the learning process for missing modalities. Specifically, the Kullback
Leibler (KL) divergence loss is used:

(10) $D_{KL}(p||q)=\sum_{i=1}^{N}p(x_{i})\cdot\frac{p(x_{i})}{q(x_{i})},$

where $p$ and $q$ are two probability distributions. Since KL divergence is
asymmetric, we adopt the Jensen-Shannon (JS) divergence loss instead:

(11) $\displaystyle\mathcal{L}_{forward}$ $\displaystyle=JS(E_{out}||E_{pre})$
$\displaystyle=\frac{1}{2}(D_{KL}(E_{out}||E_{pre})+D_{KL}(E_{pre}||E_{out})).$

####  3.7.2. Backward Reconstruction Loss ($\mathcal{L}_{backward}$)

For the backward loss, we aim to supervise the joint common vector
reconstruction. Therefore, similar to the forward differential loss, we
calculate the JS divergence loss between the Transformer decoder output
($D_{out}$) and the updated common joint representations ($E_{all}$):

(12) $\displaystyle\mathcal{L}_{backward}$
$\displaystyle=JS(D_{out}||E_{all})$
$\displaystyle=\frac{1}{2}(D_{KL}(D_{out}||E_{all})+D_{KL}(E_{all}||D_{out})).$

####  3.7.3. Tag Recovery Loss ($\mathcal{L}_{tag}$)

In our settings, the tag is attached to mark missing modalities, and we expect
our network can pay more attention to them. To better guide the reconstruction
of the attached tag, we design a tag recovery loss to direct the process. The
reason why we choose the Mean Absolute Error(MAE) loss is that MAE is less
sensitive to outliers with the absolute function. Thus, MAE is adopted to
calculate the loss between $E_{tag}$ and the last four digits of $D_{out}$:

(13)
$\displaystyle\mathcal{L}_{tag}=\frac{1}{N}\sum_{i=1}^{N}|E_{tag}^{i}-D_{tag}^{i}|,$
$\displaystyle D_{tag}=Sigmoid(D_{out}[-4:]).$

####  3.7.4. Classification Loss ($\mathcal{L}_{cls}$)

For the final classification module, we feed $E_{out}$ into a fully connected
network with the softmax activation function:

(14) $P_{c}=softmax(W_{c}E_{out}+b_{c}),$

where $W_{c}$ and $b_{c}$ are the learned weights and bias. In detail, we
employ the standard cross-entropy loss for this task, that is:

(15) $\mathcal{L}_{cls}=-\frac{1}{N}\sum_{n=1}^{N}y_{n}log\hat{y}_{n},$

where $N$ is the number of samples, $y_{n}$ is the true label of the $n$-th
sample, and $\hat{y}_{n}$ is the predicted label.

##  4\. Experiments

Table 1. Detailed parameter settings in all experiments. Description | Symbol | Value
---|---|---
Batch size | $b$ | 32
Epoch number | $e$ | 20
Dropout rate | $p$ | 0.3
Hidden size | $d$ | 300
Missing rate | $\eta$ | [0, 0.5]
Learning rate | $lr$ | 0.001
Maximum textual length | $n_{t}$ | 25
Maximum visual length | $n_{v}$ | 100
Maximum acoustic length | $n_{a}$ | 150
Loss weights | $\lambda_{1},\lambda_{2},\lambda_{3}$ | 0.1

All experiments are carried out on a Linux server (Ubuntu 18.04.1) with a
Intel(R) Xeon(R) Gold 5120 CPU, 8 Nvidia 2080TI GPUs and 128G RAM. Datasets
and experimental settings are described as follows:

Datasets: We conduct several experiments on CMU-MOSI (Zadeh et al., 2016) and
IEMOCAP (Busso et al., 2008) datasets. Both datasets are multimodal benchmarks
for sentiment recognition, including visual, textual, and acoustic modalities.
For the CMU-MOSI dataset, it contains 2199 segments from 93 opinion videos on
YouTube. The label of each sample is annotated with a sentiment score in [-3,
3]. Following Yu et al. (Yu et al., 2021), we transform the score into
negative, neutral and positive labels. For the IEMOCAP dataset, it contains 5
sessions, and each session contains about 30 videos, where each video contains
at least 24 utterances. The annotated labels are: neutral, frustration, anger,
sad, happy, excited, surprise, fear, disappointing, and other. Specifically,
we report three-classes (negative: [-3,0), neutral:[0], positive: (0,3])
results on CMU-MOSI, and two-classes (negative:[frustration, angry, sad, fear,
disappointing], positive:[happy, excited]) on IEMOCAP.

Parameters: Following standard methods, we tune our model using five-fold
validation and grid-searching on the training set. The learning rate $lr$ is
selected from $\\{0.1,0.001,0.0005,0.0001\\}$, the batch size
$b\in\\{32,64,128\\}$, and the hidden size $d\in\\{64,128,300,\\\ 768\\}$.
Adam (Kingma and Ba, 2015) is adopted to minimize the total loss given in Eq.
(9). The epoch number is 20, the batch size is 32, the loss weight is set to
0.1, and the parameters are summarized in Table 1.

Evaluation Metric: $Accuracy$ and $Macro-F1$ are used to measure the
performance of the models, which are defined as follows:

(16) $\displaystyle Accuracy=\frac{T_{true}}{N},$ $\displaystyle
F1=\frac{2PR}{P+R},$

where $T_{true}$ is the number of correctly predicted samples, $N$ is the
total number of samples, $P$ is the positive predictive value, and $R$ is the
recall value.

###  4.1. Feature Extraction

Visual Representations: The CMU-MOSI (Zadeh et al., 2016) and IEMOCAP (Busso
et al., 2008) datasets mainly consist of human conversations, where visual
features are mainly composed of human faces. Following (Zadeh et al., 2018; Yu
et al., 2020), we also adopt OpenFace2.0 toolkit (Baltrusaitis et al., 2018)
to extract facial features. Except for the first to the fifth columns data, we
finally obtain 709-dimensional visual representations, where the face, the
head, and the eye movement are included.

Textual Representations: For each textual utterance, the pre-trained Bert
(Devlin et al., 2019) is utilized to extract textual features. Eventually, we
adopt the pre-trained uncased BERT-base model (12-layer, 768-hidden, 12-heads)
to acquire 768-dimensional word vectors.

Acoustic Representations: As an audio analysis toolkit, Librosa (McFee et al.,
2015) shows an excellent ability to extract acoustic features. For CMU-MOSI
and IEMOCAP datasets, each audio is mixed to the mono and is re-sampled to
16000 Hz. Besides, each frame is separated by 512 samples, and we choose the
zero crossing rate, the mel-frequency cepstral coefficients (MFCC) and the
Constant-Q Togram (CQT) features to represent audio segments. Finally, we
concatenate three features to yield 33-dimensional acoustic features.

Table 2. Results of all baselines of missing a single modality, where the best results are in bold. Datasets | Models | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---|---
M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC
CMU-MOSI | AE | 56.78 | 79.69 | 54.07 | 79.17 | 53.40 | 78.13 | 51.28 | 72.53 | 50.75 | 73.48 | 44.99 | 69.32
CRA | 56.85 | 79.73 | 54.37 | 79.38 | 53.57 | 78.24 | 51.67 | 72.84 | 51.02 | 73.79 | 45.38 | 69.45
MCTN | 57.32 | 79.75 | 55.48 | 79.87 | 53.99 | 77.49 | 52.31 | 71.59 | 51.64 | 73.81 | 45.76 | 68.11
TransM | 57.84 | 80.21 | 57.53 | 79.69 | 55.21 | 78.42 | 52.87 | 72.92 | 52.49 | 72.40 | 45.86 | 68.23
MMIN | 60.41 | 82.29 | 57.75 | 81.86 | 55.38 | 80.20 | 53.65 | 79.24 | 52.55 | 76.33 | 48.95 | 70.76
Ours | 58.32 | 84.90 | 58.21 | 84.46 | 55.46 | 81.25 | 55.11 | 80.73 | 54.11 | 80.21 | 51.71 | 74.04
IEMOCAP | AE | 76.15 | 82.09 | 75.24 | 80.26 | 75.02 | 78.01 | 73.92 | 77.43 | 70.19 | 76.01 | 67.27 | 76.43
CRA | 77.05 | 82.13 | 75.95 | 80.97 | 75.13 | 78.09 | 74.02 | 78.11 | 70.69 | 76.12 | 67.75 | 76.49
MCTN | 78.57 | 82.27 | 77.74 | 81.02 | 75.37 | 78.27 | 74.69 | 78.52 | 71.75 | 76.29 | 68.17 | 76.63
TransM | 79.57 | 82.64 | 78.03 | 81.86 | 76.33 | 80.43 | 75.83 | 78.64 | 72.01 | 77.27 | 68.57 | 76.65
MMIN | 80.83 | 83.43 | 78.85 | 82.58 | 77.09 | 81.27 | 76.63 | 80.43 | 72.81 | 78.43 | 70.58 | 77.45
Ours | 81.15 | 85.39 | 79.99 | 85.09 | 79.10 | 84.07 | 78.45 | 83.25 | 76.74 | 82.75 | 74.43 | 82.43
Table 3. Results of all baselines of missing multiple modalities, where the best results are in bold. Datasets | Models | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---|---
M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC
CMU-MOSI | AE | 56.78 | 79.69 | 52.80 | 75.65 | 50.84 | 74.18 | 46.23 | 69.18 | 44.40 | 69.05 | 40.29 | 66.01
CRA | 56.81 | 79.72 | 52.85 | 75.68 | 51.02 | 74.73 | 46.87 | 69.23 | 45.17 | 69.48 | 41.77 | 66.82
MCTN | 56.85 | 79.73 | 52.97 | 75.89 | 51.75 | 74.16 | 46.98 | 69.29 | 45.73 | 69.55 | 42.98 | 67.02
TransM | 57.84 | 80.21 | 53.49 | 77.08 | 51.97 | 74.24 | 48.23 | 70.51 | 47.02 | 70.38 | 43.28 | 67.74
MMIN | 60.41 | 82.29 | 55.49 | 80.12 | 52.79 | 76.26 | 48.97 | 73.27 | 47.39 | 74.28 | 44.63 | 68.92
Ours | 58.32 | 84.90 | 56.38 | 81.77 | 54.87 | 81.07 | 52.12 | 77.60 | 51.19 | 76.56 | 51.15 | 73.23
IEMOCAP | AE | 76.15 | 82.09 | 75.07 | 79.84 | 74.20 | 76.91 | 71.55 | 76.07 | 69.73 | 75.16 | 67.15 | 75.22
CRA | 77.05 | 82.13 | 75.21 | 79.95 | 74.22 | 77.03 | 71.86 | 76.41 | 70.13 | 75.29 | 67.31 | 75.42
MCTN | 78.57 | 82.27 | 76.83 | 80.56 | 74.77 | 77.89 | 72.27 | 77.03 | 71.02 | 75.84 | 67.51 | 75.88
TransM | 79.57 | 82.64 | 77.21 | 81.13 | 75.87 | 79.01 | 72.36 | 78.15 | 71.38 | 76.88 | 68.02 | 76.04
MMIN | 80.83 | 83.43 | 78.02 | 82.32 | 76.38 | 79.53 | 73.05 | 79.02 | 71.22 | 77.27 | 69.39 | 77.01
Ours | 81.15 | 85.39 | 78.37 | 83.63 | 77.55 | 82.33 | 76.14 | 82.21 | 74.09 | 81.94 | 72.49 | 80.57

###  4.2. Baselines

To evaluate the performance of our approach, the following baselines are
chosen for comparison:

AE (Baldi, 2012): An efficient data encoding network trained to copy its input
to its output. In our implementation, we employ 5 AEs with each layer of the
size [512, 256, 128, 64].

CRA (Tran et al., 2017): A missing modality reconstruction framework that
employed the residual connection mechanism to approximate the difference
between the input data. In our implementation, we add a residual connection
for the input with the same layer setting in AE (Baldi, 2012).

MCTN222https://github.com/hainow/MCTN (Pham et al., 2019): A method to learn
robust joint representations by translating among modalities, claiming that
translating from a source modality to a target modality can capture joint
information among modalities.

TransM (Wang et al., 2020): An end-to-end translation based multimodal fusion
method that utilized Transformer to translate among modalities and encoded
multimodal features. In our implementation, we concatenate 6 MAE losses
between two modalities transformation.

MMIN333https://github.com/AIM3-RUC/MMIN/tree/master (Zhao et al., 2021): A
unified multimodal emotion recognition model that adopted the cascade residual
auto-encoder and cycle consistency learning to recover missing modalities.

TATE: Our proposed model.

###  4.3. Overall Results

For the single modality missing case, the experimental results are shown in
Table 2, where the missing ratio is set from 0 to 0.5. Specifically, we report
triple classification results on CMU-MOSI and two classification results on
IEMOCAP. With the increment of missing rate, the overall results present a
descending trend. Except for the M-F1 value under the full modality condition
is lower about 2.02% than MMIN on the CMU-MOSI dataset, our proposed method
achieves the best results on other settings, validating the effectiveness of
our model. As can be seen in the table, compared to auto-encoder based methods
(AE, CRA), translation-based methods (MCTN, TransM) achieve better
performance, probably due to the fact that end-to-end translation among
modalities can better fuse the multimodal information. Besides, the
comparative experiments suggest that the backward decoder can assist the
forward encoder, so as to further improve the overall performance.

For multiple modalities missing cases, we also present related findings in
Table 3. In this setting, one or two modalities are randomly discarded. It can
be seen that our proposed model still improves about 0.89% to 3.10% on M-F1
and about 1.31% to 4.81% on ACC compared to other baselines, demonstrating the
robustness of the network. Owing to the forward differential loss and the
assistance of tag, our model can still capture semantic-relevant information.
More comparison will be given in Section 4.5.

###  4.4. Ablation Study

Table 4. Comparison of all modules in TATE. Modules | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
---|---|---|---|---|---|---
M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC | M-F1 | ACC
V | 37.84 | 56.25 | - | - | - | - | - | - | - | - | - | -
A | 39.82 | 59.90 | - | - | - | - | - | - | - | - | - | -
T | 55.63 | 76.17 | - | - | - | - | - | - | - | - | - | -
V+A | 40.71 | 61.26 | 38.93 | 59.10 | 38.07 | 56.94 | 37.42 | 56.18 | 36.98 | 55.43 | 36.66 | 54.17
V+T | 56.98 | 79.13 | 56.41 | 78.67 | 55.07 | 76.29 | 54.83 | 74.90 | 52.86 | 74.15 | 50.32 | 72.92
A+T | 57.69 | 80.65 | 57.01 | 79.47 | 55.23 | 77.44 | 55.02 | 75.39 | 53.92 | 74.37 | 51.25 | 73.13
V+A+T | 58.32 | 84.90 | 58.21 | 84.46 | 55.46 | 81.25 | 55.11 | 80.73 | 54.11 | 80.21 | 51.71 | 74.04
-w/o tag | 57.95 | 80.21 | 57.86 | 80.99 | 54.83 | 79.90 | 53.71 | 79.89 | 52.59 | 76.75 | 49.17 | 72.05
-w/o tag loss | 58.04 | 82.81 | 57.92 | 81.98 | 55.32 | 80.77 | 53.95 | 80.28 | 52.83 | 77.01 | 49.41 | 72.92
-w/o forward loss | 52.39 | 76.21 | 51.83 | 75.52 | 50.16 | 73.39 | 49.87 | 72.15 | 48.26 | 71.12 | 47.29 | 70.35
-w/o backward loss | 53.85 | 77.43 | 52.88 | 77.08 | 51.85 | 74.09 | 51.07 | 74.21 | 49.01 | 71.92 | 48.53 | 71.08
-w/o common space | 54.03 | 79.76 | 53.20 | 77.59 | 52.97 | 75.99 | 51.23 | 75.01 | 49.83 | 72.37 | 49.05 | 71.85

To explore the effects of different modules in TATE, we evaluate our model
with several settings: 1) using only one modality; 2) using two modalities; 3)
removing the tag encoding module; 4) removing the common space projection
module; 5) removing the tag recovery loss; 6) removing the forward
differential loss; and 7) removing the backward reconstruction loss.

According to Table 4, one interesting finding is that the performance drops
sharply when the textual modality is missing, validating that textual
information dominates in the multimodal sentiment analysis. A possible
explanation for these results is that textual information is the manual
transcription. However, similar reductions are not observed when removing the
visual modality. We conjecture that the visual information is not well
extracted due to the minor changes to the face. Besides, the top half of the
table shows that the combination of two modalities provides better performance
than single modality, indicating that two modalities can learn complementary
features. As for the effects of different modules, the performance of the
forward differential module decreases about 4.52% to 6.38% on M-F1 and about
3.69% to 9.09% on ACC compared to the whole model, demonstrating the
importance of the forward guidance. Since we employ full modality to pre-train
the guidance network, the forward JS divergence loss serves as a good
supervision. One striking result to emerge from this table is that the tag
encoding module slightly improves the performance as expected. To further
validate the effectiveness of the tag encoding module, we conduct several
experiments in the following sub-section.

###  4.5. Effects of the Tag Encoding

Table 5. Improvements of the Tag Encoding. Model | Basic | +Tag
---|---|---
M-F1 | ACC | M-F1 | ACC
AE | 51.28 | 72.53 | 53.25 (3.69% $\uparrow$) | 75.21 (3.56% $\uparrow$)
TransM | 52.87 | 72.92 | 54.79 (3.50% $\uparrow$) | 76.02 (4.08% $\uparrow$)
Ours | 53.71 | 79.89 | 55.11 (2.54% $\uparrow$) | 80.73 (1.04% $\uparrow$)

We incorporate the tag encoding module with two basic models: AE and TransM.
The reason why we choose the above two models is that AE and TransM are two
different kinds of encoders: AE is the auto-encoder based method, and TransM
is the Transformer based method. For the above two models, we add tags after
the feature extraction module. Table 5 presents the detailed results on the
CMU-MOSI dataset with a 30% missing rate. It can be seen that models with the
tag encoding module improves about 2.54% to 3.69% on M-F1 and about 1.04% to
4.08% on ACC compared to basic models, showing the effectiveness of the tag
encoding module. Owing to the added tag, the network can be better guided, and
can further focus on missing modalities.

Figure 5. Comparison of full modalities and incomplete modalities during
testing. (a) M-F1 values. (b) Accuracy values.

(a) $\eta=0$ (b) $\eta=0.1$

(c) $\eta=0.2$

(d) $\eta=0.3$ (e) $\eta=0.4$

(f) $\eta=0.5$

Figure 6. Virtualization of joint representations with different rates of
missing modalities (red: negative, blue: neutral, green: positive). (a) Full
modalities, (b) missing rate 0.1, (c) missing rate 0.2, (d) missing rate 0.3,
(e) missing rate 0.4, and (f) missing rate 0.5.

###  4.6. Effects of the complete modality

To see the difference between the complete and incomplete modalities of the
test data, we first train the model with incomplete data, and then test the
model with both full modality data and different missing rates of incomplete
data. All experiments share the same parameters on the IEMOCAP dataset for a
fair comparison. As can be seen in Fig. 5, the gaps between two settings on
M-F1 and ACC reach the minimum when the missing rate is 0.3. As the number of
missing samples in the training data increases, the correlation among
modalities becomes harder to capture, resulting in weaker test performance.
However, the gap increases when the missing rate is bigger than 0.3. One
possible explanation for the above results is that the model cannot learn the
joint representation well because there are too many absent samples.

###  4.7. Multi-classes on IEMOCAP

Table 6. Detailed distributions on IEMOCAP.

Category | Hap. | Ang. | Sad. | Neu. | Fru. | Exc. | Sur
---|---|---|---|---|---|---|---
4-calsses | Train | 477 | 879 | 868 | 1385 | - | - | -
Test | 118 | 224 | 216 | 323 | - | - | -
7-calsses | Train | 476 | 891 | 873 | 1348 | 1458 | 848 | 87
Test | 119 | 212 | 211 | 360 | 391 | 193 | 20

Table 7. Results of multi-classes on IEMOCAP. Ratio | 2-classes | 4-classes | 7-classes
---|---|---|---
M-F1 | ACC | M-F1 | ACC | M-F1 | ACC
0 | 81.15 | 85.39 | 48.29 | 59.04 | 36.83 | 47.41
0.1 | 79.99 | 85.09 | 46.56 | 57.70 | 36.01 | 45.73
0.2 | 79.10 | 84.07 | 46.07 | 56.93 | 35.48 | 44.17
0.3 | 78.45 | 83.25 | 45.69 | 56.27 | 35.21 | 43.02
0.4 | 76.74 | 82.75 | 45.02 | 55.31 | 33.79 | 42.31
0.5 | 74.43 | 82.43 | 44.15 | 54.85 | 33.28 | 42.08

We also explore the performance of multiple classes on the IEMOCAP dataset.
Apart from the two-classes results, we also choose happy, angry, sad and
neutral emotions as the 4-classes experiment, and then choose the extra
frustration, excited, and surprise emotions as the 7-classes experiment. The
detailed distributions and results are presented in Table 6 and Table 7
respectively. It can be seen that both M-F1 value and ACC decrease with the
increment of class numbers. By comparing the results with different rates of
missing modalities, the gaps among 7-classes are smaller than that among
2-classes and 4-classes. Besides, closer inspection of Table 7 shows that the
overall performance drops sharply when the class number is 7, which is caused
by the confusion of multiple classes, resulting in the difficulties in
convergence of the model.

###  4.8. Effects of different losses

Table 8. Results of different losses.

Datasets | Loss | 0 | 0.2 | 0.4
---|---|---|---|---
M-F1 | ACC | M-F1 | ACC | M-F1 | ACC
CMU-MOSI | Cosine | 55.87 | 81.21 | 52.28 | 76.95 | 50.23 | 74.18
MAE | 56.21 | 82.05 | 52.37 | 77.16 | 51.15 | 74.76
JS | 57.89 | 84.15 | 53.42 | 79.05 | 52.17 | 75.22
ours | 58.32 | 84.90 | 55.46 | 81.25 | 54.11 | 80.21
IEMOCAP | Cosine | 79.86 | 84.07 | 77.21 | 81.69 | 75.48 | 80.06
MAE | 80.12 | 84.21 | 77.67 | 82.53 | 75.89 | 80.21
JS | 80.73 | 85.03 | 78.56 | 83.21 | 76.34 | 81.07
ours | 81.15 | 85.39 | 79.10 | 84.07 | 76.74 | 82.75

To investigate the effects of different losses, we replace different loss
function to see the performance. In detail, the cosine similarity loss, the
MAE loss, and the JS divergence loss are chosen for comparison. We evaluate
our model with 4 settings: 1) using the cosine similarity loss for
$\mathcal{L}_{forward}$, $\mathcal{L}_{backward}$ and $\mathcal{L}_{tag}$; 2)
using the MAE loss for $\mathcal{L}_{forward}$, $\mathcal{L}_{backward}$ and
$\mathcal{L}_{tag}$; 3) using the JS divergence loss
for$\mathcal{L}_{forward}$, $\mathcal{L}_{backward}$ and $\mathcal{L}_{tag}$;
and 4) using the JS divergence loss for $\mathcal{L}_{forward}$ and
$\mathcal{L}_{backward}$, and using the MAE loss for $\mathcal{L}_{tag}$
(ours).

As can be seen in the Table 8, our method achieves the best performance
compared to other three loss settings on two datasets, showing the superiority
of our model. Relatively, the results of applying JS divergence achieve
secondary performance. Since the tag is composed of 4 digits (“0” or “1”), the
MAE loss is more straightforward than JS divergence loss. Further analysis of
the table suggests that the combination of the JS divergence loss and the MAE
loss is beneficial in improving the overall performance.

###  4.9. Visualization

To better understand the learning ability of our model, we adopt the T-SNE
toolkit (Maaten and Hinton, 2008) to visualize the joint representations under
different rates of missing modalities. Specifically, we visualize about 1500
vectors learned by the Transformer encoder on the CMU-MOSI dataset, where the
red, the blue, and the green color denote negative, neutral and positive
respectively.

As shown in Figs. 6(a)-(e), the overall joint representations obtain the
similar distribution as the full modality condition. The majority of vectors
are generally divided into three categories, where neutral samples is harder
to classify because of their uncertain semantic. Besides, with the increment
of missing ratio, the distributions become more discrete, especially when the
missing ratio is bigger than 0.3. Apart from that, as can be seen in the top
right-hand corner of Figs. 6(b)-(e), the larger ratio of missing modalities,
the wider outliers. The reason is that the model cannot converge with too many
absent samples. While in Fig. 6(f), the decision boundary is closer to the
outliers when there are nearly half of missing samples. We suspect that absent
samples dominate when training the model, resulting in a quite distinct
distribution.

##  5\. Conclusion

In this paper, we propose a Tag-Assisted Transformer Encoder (TATE) network to
handle the problem of missing partial modalities. Owing to the tag encoding
technique, the proposed model can cover all uncertain missing cases, and the
designed tag recovery loss can in turn supervise joint representation
learning. Besides, more general aligned vectors are obtained by the common
space module, and then are fed into the Transformer encoder for further
process. At last, the final objective loss further directs the learning of
missing modalities. All experimental results are conducted on CMU-MOSI and
IEMOCAP datasets, showing the effectiveness of the proposed method.

In the future, this study may be further improved in the following ways: 1)
for the common space projection module, we will try more fusion methods (e.g.
add weights) to concatenate common feature vectors; and 2) for the Transformer
encoder-decoder module, we employ the original sub-layer in Transformer as the
basic semantic encoder. We attempt to adopt different structures of
Transformer (e.g. Transformer-XL (Dai et al., 2019), Reformer (Kitaev et al.,
2020), etc.) to observe the performance.

###### Acknowledgements.

This work was supported by Macau Science and Technology Development Fund under
SKL-IOTSC-2021-2023, 0072/2020/AMJ, 0015/-2019/AKP, 060/2019/A1, and
077/2018/A2, by Research Committee at University of Macau under
MYRG2018-00029-FST and MYRG2019-00023-FST, and by Natural Science Foundation
of China under 61971476.

## References

  * (1)
  * Akbari et al. (2021) Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In _Proceedings of the 35th Annual Conference on Neural Information Processing Systems_.
  * Baldi (2012) Pierre Baldi. 2012\.  Autoencoders, unsupervised learning, and deep architectures. In _Proceedings of ICML Workshop on Unsupervised and Transfer Learning_. 37–49.
  * Baltrusaitis et al. (2018) Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018.  Openface 2.0: Facial behavior analysis toolkit. In _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition_. IEEE, 59–66.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  IEMOCAP: Interactive emotional dyadic motion capture database.  _Language Resources and Evaluation_ 42, 4 (2008), 335–359.
  * Cai et al. (2018) Lei Cai, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji. 2018.  Deep adversarial learning for multi-modality missing data completion. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1158–1166.
  * Chen et al. (2017) Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Baltrušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017.  Multimodal sentiment analysis with word-level fusion and reinforcement learning. In _Proceedings of the 2017 ACM International Conference on Multimodal Interaction_. 163–171.
  * Chu et al. (2021) Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. 2021\.  Twins: Revisiting the Design of Spatial Attention in Vision Transformers. In _Advances in Neural Information Processing Systems_ , Vol. 34. Curran Associates, Inc., 9355–9366.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019\.  Transformer-xl: Attentive language models beyond a fixed-length context. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 2978–2988.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 4171–4186.
  * Du et al. (2018) Changde Du, Changying Du, Hao Wang, Jinpeng Li, Wei-Long Zheng, Bao-Liang Lu, and Huiguang He. 2018.  Semi-supervised deep generative modelling of incomplete multi-modality emotional data. In _Proceedings of the 26th ACM International Conference on Multimedia_. 108–116.
  * Geng et al. (2021) Binzong Geng, Min Yang, Fajie Yuan, Shupeng Wang, Xiang Ao, and Ruifeng Xu. 2021\.  Iterative network pruning with uncertainty regularization for lifelong sentiment classification. In _Proceedings of the 44th International ACM SIGIR conference on Research and Development in Information Retrieval_. 1229–1238.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets.  _Advances in Neural Information Processing Systems_ 27 (2014).
  * Guo et al. (2017) Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. 2017\.  Online early-late fusion based on adaptive HMM for sign language recognition.  _ACM Transactions on Multimedia Computing, Communications, and Applications_ 14, 1 (2017), 1–18.
  * Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020\.  Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In _Proceedings of the 28th ACM International Conference on Multimedia_. 1122–1131.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 770–778.
  * Kim et al. (2020) Eun-Sol Kim, Woo Young Kang, Kyoung-Woon On, Yu-Jung Heo, and Byoung-Tak Zhang. 2020.  Hypergraph attention networks for multimodal learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14581–14590.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization. In _Proceedings of the 3rd International Conference on Learning Representations_. 1–15.
  * Kingma and Welling (2014) Diederik P Kingma and Max Welling. 2014.  Auto-encoding variational bayes. In _Proceedings of the International Conference on Learning Representations_. 1–14.
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020\.  Reformer: The efficient transformer. In _Proceedings of the International Conference on Machine Learning_. 1–12.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ 9, Nov (2008), 2579–2605.
  * Mai et al. (2021) Sijie Mai, Songlong Xing, and Haifeng Hu. 2021.  Analyzing multimodal sentiment via acoustic-and visual-LSTM with channel-aware temporal convolution network.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 (2021), 1424–1437.
  * Majumder et al. (2018) Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Poria. 2018\.  Multimodal sentiment analysis using hierarchical fusion with context modeling.  _Knowledge-based Systems_ 161 (2018), 124–133.
  * Mazumder et al. (2010) Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. 2010\.  Spectral regularization algorithms for learning large incomplete matrices.  _The Journal of Machine Learning Research_ 11 (2010), 2287–2322.
  * McFee et al. (2015) Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015.  librosa: Audio and music signal analysis in python. In _Proceedings of the 14th Python in Science Conference_ , Vol. 8. 18–25.
  * Parthasarathy and Sundaram (2020) Srinivas Parthasarathy and Shiva Sundaram. 2020.  Training strategies to handle missing modalities for audio-visual expression recognition. In _Companion Publication of the 2020 International Conference on Multimodal Interaction_. 400–404.
  * Pham et al. (2019) Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019\.  Found in translation: Learning robust joint representations by cyclic translations between modalities. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 6892–6899.
  * Poria et al. (2018) Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Erik Cambria, Alexander Gelbukh, and Amir Hussain. 2018.  Multimodal sentiment analysis: Addressing key issues and setting up the baselines.  _IEEE Intelligent Systems_ 33, 6 (2018), 17–25.
  * Shang et al. (2017) Chao Shang, Aaron Palmer, Jiangwen Sun, Ko-Shin Chen, Jin Lu, and Jinbo Bi. 2017\.  VIGAN: Missing view imputation with generative adversarial networks. In _2017 IEEE International Conference on Big Data_. IEEE, 766–775.
  * Shao et al. (2013) Weixiang Shao, Xiaoxiao Shi, and S Yu Philip. 2013.  Clustering on multiple incomplete datasets via collective kernel learning. In _2013 IEEE 13th International Conference on Data Mining_. IEEE, 1181–1186.
  * Shraga et al. (2020) Roee Shraga, Haggai Roitman, Guy Feigenblat, and Mustafa Cannim. 2020.  Web table retrieval using multimodal deep learning. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1399–1408.
  * Springstein et al. (2021) Matthias Springstein, Eric Müller-Budack, and Ralph Ewerth. 2021\.  QuTI! Quantifying Text-Image Consistency in Multimodal Documents. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2575–2579.
  * Stappen et al. (2021) Lukas Stappen, Alice Baird, Erik Cambria, and Björn W Schuller. 2021.  Sentiment analysis and topic recognition in video transcriptions.  _IEEE Intelligent Systems_ 36, 2 (2021), 88–95.
  * Tran et al. (2017) Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. 2017\.  Missing modalities imputation via cascaded residual autoencoder. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1405–1414.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in Neural Information Processing Systems_. 5998–6008.
  * Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008.  Extracting and composing robust features with denoising autoencoders. In _Proceedings of the 25th International Conference on Machine Learning_. 1096–1103.
  * Wang et al. (2019) Jin Wang, Liang-Chih Yu, K Robert Lai, and Xuejie Zhang. 2019\.  Investigating dynamic routing in tree-structured LSTM for sentiment analysis. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_. 3423–3428.
  * Wang et al. (2020) Zilong Wang, Zhaohong Wan, and Xiaojun Wan. 2020.  Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis. In _Proceedings of The Web Conference 2020_. 2514–2520.
  * Xu et al. (2020a) Jie Xu, Zhoujun Li, Feiran Huang, Chaozhuo Li, and S Yu Philip. 2020a.  Social image sentiment analysis by exploiting multimodal content and heterogeneous relations.  _IEEE Transactions on Industrial Informatics_ 17, 4 (2020), 2974–2982.
  * Xu et al. (2018) Nan Xu, Wenji Mao, and Guandan Chen. 2018.  A co-memory network for multimodal sentiment analysis. In _Proceedings of the 2018 International ACM SIGIR Conference on Research and Development in Information Retrieval_. 929–932.
  * Xu et al. (2020b) Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020b.  Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 3777–3786.
  * Yu et al. (2019) Jianfei Yu, Jing Jiang, and Rui Xia. 2019.  Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 28 (2019), 429–439.
  * Yu et al. (2020) Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020\.  Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 3718–3727.
  * Yu et al. (2021) Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021\.  Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In _Proceedings of the AAAI Conference on Artificial Intelligence_. 10790–10797.
  * Yuan et al. (2021) Ziqi Yuan, Wei Li, Hua Xu, and Wenmeng Yu. 2021\.  Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In _Proceedings of the 29th ACM International Conference on Multimedia_. 4400–4407.
  * Zadeh et al. (2016) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ 31, 6 (2016), 82–88.
  * Zadeh et al. (2018) AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018.  Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_. 2236–2246.
  * Zhang et al. (2020b) Bowen Zhang, Xutao Li, Xiaofei Xu, Ka-Cheong Leung, Zhiyao Chen, and Yunming Ye. 2020b.  Knowledge guided capsule attention network for aspect-based sentiment analysis.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 28 (2020), 2538–2551.
  * Zhang et al. (2020a) Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu. 2020a.  Deep partial multi-view learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2020).
  * Zhao et al. (2021) Jinming Zhao, Ruichen Li, and Qin Jin. 2021.  Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_. 2608–2618.
  * Zhuang et al. (2020) Honglei Zhuang, Fang Guo, Chao Zhang, Liyuan Liu, and Jiawei Han. 2020.  Joint aspect-sentiment analysis with minimal user guidance. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1241–1250.