Then $q_{0}$ is the neutral element of the Ellis group $\\{q_{0},q_{1}\\}$, so
an idempotent in a minimal left ideal of $S_{G}(M)$, and hence we will denote
$q_{0}$ by $u_{G}$.

The space $S_{\tilde{G}}(M)$ is naturally homeomorphic with
$S_{G}({\mathbb{R}})\times S_{{\mathbb{Z}}}({\mathbb{Z}})$, and the induced
semigroup operation is given by

$(p,q)*(p^{\prime},q^{\prime})=(p*p^{\prime},q+q^{\prime}+h(p,p^{\prime})),$

where $h(p,p^{\prime}):=h(a,a^{\prime})$ for some/any $a\models p$ and
$a^{\prime}\models p^{\prime}$ such that $\operatorname{{tp}}(a/M,a^{\prime})$
is a coheir over $M$, and $+$ denotes the semigroup operation on
$S_{{\mathbb{Z}}}({\mathbb{Z}})$ (which is indeed commutative). From now on,
$(S_{\tilde{G}}(M),*)$ will be identified with $(S_{G}({\mathbb{R}})\times
S_{{\mathbb{Z}}}({\mathbb{Z}}),*)$. Since $*$ uses $h$, we will denote this
semigroup as $S_{G}({\mathbb{R}})\times_{h}S_{{\mathbb{Z}}}({\mathbb{Z}})$. As
to the semigroup $S_{{\mathbb{Z}}}({\mathbb{Z}})$, we will interchangeably use
additive and multiplicative notation.

As $({\mathbb{Z}},+)$ is stable, there is a unique minimal left ideal
${\mathcal{M}}_{{\mathbb{Z}}}$ and it consists of the generic types. There is
also a unique idempotent $u_{{\mathbb{Z}}}$ in ${\mathcal{M}}_{{\mathbb{Z}}}$
which the generic type concentrated on the component $\bar{\mathbb{Z}}^{0}$
(the intersection of all definable subgroups of $\bar{{\mathbb{Z}}}$ of finite
index), and
$u_{\mathbb{Z}}{\mathcal{M}}_{\mathbb{Z}}={\mathcal{M}}_{\mathbb{Z}}$.

By the explicit formula for $h$ and the idempotency of $u_{G}$, we get
$h(u_{G},u_{G})=0$. So [Jag15, Proposition 5.6] yields

######  Fact 5.7.

Let ${\mathcal{M}}_{G}\ni u_{G}$ be a minimal left ideal of
$S_{G}({\mathbb{R}})$. Then:

  1. (1)

${\mathcal{M}}_{\tilde{G}}:={\mathcal{M}}_{G}\times{\mathcal{M}}_{{\mathbb{Z}}}$
is a minimal left ideal of $S_{\tilde{G}}(M)$;

  2. (2)

$u_{\tilde{G}}:=(u_{G},u_{{\mathbb{Z}}})$ is an idempotent in
${\mathcal{M}}_{\tilde{G}}$;

  3. (3)

The Ellis group $u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$ equals
$u_{G}{\mathcal{M}}_{G}\times_{h}{\mathcal{M}}_{{\mathbb{Z}}}=u_{G}{\mathcal{M}}_{G}\times_{h}u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$.

$f_{1}\colon u_{G}{\mathcal{M}}_{G}\to{\mathbb{Z}}_{2}$ given by $q_{i}\mapsto
i$ is clearly an isomorphism. Since $({\mathbb{Z}},+)$ is stable, it is well-
known that the natural map $f_{2}\colon
u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}\to\bar{\mathbb{Z}}/\bar{\mathbb{Z}}^{0}\cong\hat{{\mathbb{Z}}}$
given by $\operatorname{{tp}}(a/{\mathbb{Z}})\mapsto a/\bar{\mathbb{Z}}^{0}$
is an isomorphism. Thus, the following corollary is deduced in [Jag15, Example
5.7].

######  Corollary 5.8.

The map $(f_{1},f_{2})\colon
u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}\to{\mathbb{Z}}_{2}\times\hat{{\mathbb{Z}}}$
is an isomorphism, with the group operation on
${\mathbb{Z}}_{2}\times\hat{{\mathbb{Z}}}$ given by
$(x,n)(x^{\prime},n^{\prime}):=(x+_{2}x^{\prime},n+n^{\prime}-xx^{\prime})$.
The target group is moreover topologically isomorphic to $\hat{{\mathbb{Z}}}$
via the map $(x,n)\mapsto x-2n$.

Our next goal is to show that the isomorphism in Corollary 5.8 is topological.
This implies that $u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$ is topologically
isomorphic to $\hat{{\mathbb{Z}}}$, so Hausdorff which confirms Conjecture 5.3
for $\widetilde{\operatorname{{SL}}_{2}({\mathbb{R}})}$.

As the $\tau$-topology on $u_{G}{\mathcal{M}}_{G}$ is $T_{1}$, it is discrete,
and so $f_{1}$ is a topological isomorphism. Since $f_{2}$ is an isomorphism
which is continuous by [KP17, Theorem 0.1], we get that it is a topological
isomorphism (with $u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$ equipped with
the $\tau$-topology).

The fact that the isomorphism $(f_{1},f_{2})$ from Corollary 5.8 is
topological follows immediately from the above paragraph and the next
proposition.

######  Proposition 5.9.

The $\tau$-topology on
$u_{G}{\mathcal{M}}_{G}\times_{h}u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$
is the product of the $\tau$-topologies on $u_{G}{\mathcal{M}}_{G}$ and
$u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$.

Before the proof, let us show a few properties of $h$ which will be used also
in the proof of Proposition 5.13.

######  Lemma 5.10.

  1. (1)

$h(p,u_{G})=0$ for all $p\in S_{G}(M)$.

  2. (2)

$h(u_{G},\operatorname{{tp}}(g/M))=0$ for all $g\in G$.

  3. (3)

$h(u_{G},gu_{G})=0$ for all $g\in G$.

###### Proof.

(1) Present $u_{G}$ as $\operatorname{{tp}}(A/M)$ for
$A:=\left(\begin{array}[]{cc}(1-x)b&(1-x)c-yb^{-1}\\\
yb&yc+(1-x)b^{-1}\end{array}\right),$ where $x,y,b,c$ are as above. Write $p$
as $\operatorname{{tp}}(B/M)$ for $B=\left(\begin{array}[]{ll}\alpha&\beta\\\
\gamma&\delta\end{array}\right)$ so that $\operatorname{{tp}}(B/M,x,y,b,c)$ is
a coheir over $M$. Then $pu_{G}=\operatorname{{tp}}(BA/M)$ and
$BA=\left(\begin{array}[]{ll}c_{11}&c_{12}\\\ c_{21}&c_{22}\end{array}\right)$
with $c_{21}:=\gamma(1-x)b+\delta yb$. Since $yb>0$, by the explicit formula
for $h$, the only possibility for $h(p,u_{G})\neq 0$ would be the case when
$\gamma(\delta)>0$ and $c_{21}(c_{22})<0$. We will show that it never happens,
namely $c_{21}>0$ whenever $\gamma(\delta)>0$. So assume that
$\gamma(\delta)>0$.

If $\gamma=0$, then $\delta>0$, and so $c_{21}=\delta yb>0$. So assume that
$\gamma>0$. Suppose for a contradiction that $c_{21}\leq 0$. Then
$\frac{\delta}{\gamma}\leq\frac{x-1}{y}<{\mathbb{R}}$ which contradicts the
assumption that $\operatorname{{tp}}(\gamma,\delta/{\mathbb{R}},x,y)$ is
finitely satisfiable in ${\mathbb{R}}$.

(2) The proof is similar and left to the reader.

(3) By the 2-cocycle formula, we have
$h(u_{G},g)+h(u_{G}g,u_{G})=h(u_{G},gu_{G})+h(g,u_{G})$. So the conclusion
follows from (1) and (2). ∎

###### Proof of Proposition 5.9.

Denote by $\mathcal{T}$ the product of the $\tau$-topologies on
$u_{G}{\mathcal{M}}_{G}$ and $u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$.
Our goal is to prove that $\tau=\mathcal{T}$.

($\supseteq$) It is enough to show that any subbasic closed set of the form
$F\times u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$ or
$u_{G}{\mathcal{M}}_{G}\times E$ (where $F\subseteq u_{G}{\mathcal{M}}_{G}$
and $E\subseteq u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$ are
$\tau$-closed) is closed in $\tau$.

First, consider $F\times u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$. Take
any $a\in\operatorname{{cl}}_{\tau}(F\times
u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}})$. Then
$a=\lim(g_{i},n_{i})(f_{i},a_{i})$, where $((g_{i},n_{i}))_{i}$ is a net from
$\tilde{G}$ converging to $u_{\tilde{G}}=(u_{G},u_{{\mathbb{Z}}})$ and
$(f_{i},a_{i})\in F\times u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$. As
$(g_{i},n_{i})(f_{i},a_{i})=(g_{i}f_{i},n_{i}+a_{i}+h(g_{i},f_{i}))\in\\{g_{i}f_{i}\\}\times{\mathcal{M}}_{{\mathbb{Z}}}$,
we get that
$a\in\operatorname{{cl}}_{\tau}(F)\times{\mathcal{M}}_{{\mathbb{Z}}}=F\times
u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$, as required.

Now, consider $u_{G}{\mathcal{M}}_{G}\times E$. Take any
$a\in\operatorname{{cl}}_{\tau}(u_{G}{\mathcal{M}}_{G}\times E)$. Then
$a=\lim(g_{i},n_{i})(f_{i},a_{i})$, where $((g_{i},n_{i}))_{i}$ is a net from
$\tilde{G}$ converging to $u_{\tilde{G}}=(u_{G},u_{{\mathbb{Z}}})$ and
$(f_{i},a_{i})\in u_{G}{\mathcal{M}}_{G}\times E$.

######  Claim 1.

$h(g_{i},f_{i})=0$ for $i$ large enough.

###### Proof.

Since $\lim g_{i}=u_{G}$ is the type over $M$ of a matrix with positive left
bottom entry, there is $i_{0}$ such that the left bottom entry of $g_{i}$ is
positive for all $i>i_{0}$. Consider any $i>i_{0}$. If $f_{i}=u_{G}$, then
$h(g_{i},f_{i})=0$ by Lemma 5.10(1). If $f_{i}=q_{1}$, then the left bottom
entry of any matrix realizing $f_{i}$ is negative, so $h(g_{i},f_{i})=0$ by
the explicit formula for $h$. ∎

By this claim,
$(g_{i},n_{i})(f_{i},a_{i})=(g_{i}f_{i},n_{i}+a_{i}+h(g_{i},f_{i}))=(g_{i}f_{i},n_{i}+a_{i})$.
So
$a\in\operatorname{{cl}}_{\tau}(u_{G}{\mathcal{M}}_{G})\times\operatorname{{cl}}_{\tau}(E)=u_{G}{\mathcal{M}}_{G}\times
E$, as required.

($\subseteq$) Consider a $\tau$-closed $A\subseteq
u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$. We need to show that it is closed in
$\mathcal{T}$. So take any
$a=(a_{1},a_{2})\in\operatorname{{cl}}_{\mathcal{T}}(A)$. There are are nets
$(a_{1,i})_{i}\subseteq u_{G}{\mathcal{M}}_{G}$ and $(a_{2,i})_{i}\subseteq
u_{\mathbb{Z}}{\mathcal{M}}_{\mathbb{Z}}$ $\tau$-converging to $a_{1}$ and
$a_{2}$, respectively, with $(a_{1,i},a_{2,i})\in A$ for all $i$. Passing to
subnets, we can assume that the nets $(a_{1,i})_{i}$ and $(a_{2,i})_{i}$
converge in the usual topologies on $S_{G}(M)$ and $S_{{\mathbb{Z}}}(M)$ to
some $b_{1}$ and $b_{2}$, respectively. By Lemma 3.19 and Hausdorffness of
$u_{G}{\mathcal{M}}_{G}$ and $u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$,
we get $u_{G}b_{1}=a_{1}$ and $u_{{\mathbb{Z}}}+b_{2}=a_{2}$. Approximating
$u_{G}$ by elements of $G$ and $u_{{\mathbb{Z}}}$ by elements of
${\mathbb{Z}}$, using left continuity of the semigroup operations and the fact
that the actions of $G$ on $S_{G}(M)$ and of ${\mathbb{Z}}$ on
$S_{{\mathbb{Z}}}(M)$ are continuous, passing to subnets, we can assume that
there are nets $(g_{i})_{i}$ in $G$ and $(n_{i})_{i}$ in ${\mathbb{Z}}$
converging to $u_{G}$ and $u_{{\mathbb{Z}}}$, respectively, and such that
$\lim g_{i}a_{1,i}=a_{1}$ and $\lim n_{i}+a_{2,i}=a_{2}$ (in the usual
topology on type spaces). Then $\lim(g_{i},n_{i})=(u_{G},u_{{\mathbb{Z}}})$.
On the other hand, by Claim 1, $h(g_{i},a_{1,i})=0$ for sufficiently large
$i$’s, and so

$(g_{i},n_{i})(a_{1,i},a_{2,i})=(g_{i}a_{1,i},n_{i}+a_{2,i}+h(g_{i},a_{1,i}))=(g_{i}a_{1,i},n_{i}+a_{2,i})$

for sufficiently large $i$’s. Hence,
$\lim(g_{i},n_{i})(a_{1,i},a_{2,i})=(a_{1},a_{2})$. Therefore,
$a\in\operatorname{{cl}}_{\tau}(A)=A$. ∎

The next lemma follows by an elementary matrix computation.

######  Lemma 5.11.

For every $\operatorname{{tp}}(B/M)\in S_{G}(M)$ with
$B=\left(\begin{array}[]{ll}\alpha&\beta\\\ \gamma&\delta\end{array}\right),$
writing $u_{G}=\operatorname{{tp}}(A/M)=\operatorname{{tp}}(A^{\prime}/M)$ for
$A:=\left(\begin{array}[]{cc}(1-x)b&(1-x)c-yb^{-1}\\\
yb&yc+(1-x)b^{-1}\end{array}\right),$
$A^{\prime}:=\left(\begin{array}[]{cc}(1-x^{\prime})b^{\prime}&(1-x^{\prime})c^{\prime}-y^{\prime}b^{\prime-1}\\\
y^{\prime}b^{\prime}&y^{\prime}c^{\prime}+(1-x^{\prime})b^{\prime-1}\end{array}\right)$
with the elements satisfying the requirements described before and such that
$\operatorname{{tp}}(B/M,A)$ and $\operatorname{{tp}}(A^{\prime}/M,B,A)$ are
coheirs over $M$, we have that
$u_{G}\operatorname{{tp}}(B/M)u_{G}=\operatorname{{tp}}(C/M)$ with the left
bottom entry of $C$ equal to $y^{\prime}b^{\prime}(\alpha(1-x)b+\beta
yb)+(y^{\prime}c^{\prime}+(1-x^{\prime})b^{\prime-1})(\gamma(1-x)b+\delta
yb)$.

######  Lemma 5.12.

  1. (1)

For $B=\left(\begin{array}[]{rr}\alpha&\beta\\\
\gamma&\delta\end{array}\right)\in G$ we have that
$u_{G}\operatorname{{tp}}(B/M)u_{G}=u_{G}=q_{0}$ if $\gamma>0$, and
$u_{G}\operatorname{{tp}}(B/M)u_{G}=q_{1}$ if $\gamma<0$.

  2. (2)

$u_{G}\left(\begin{array}[]{rr}0&-1\\\ 1&0\end{array}\right)u_{G}=u_{G}$.

  3. (3)

$u_{G}\operatorname{{tp}}\left(\left(\begin{array}[]{rr}-1&0\\\
\gamma&-1\end{array}\right)/M\right)u_{G}=q_{1}$ for all positive
infinitesimals $\gamma$.

###### Proof.

First, let $B=\left(\begin{array}[]{rr}\alpha&\beta\\\
\gamma&\delta\end{array}\right)\in\bar{G}$ with $\gamma>0$. Pick
$x,y,b,c,x^{\prime},y^{\prime},b^{\prime},c^{\prime}$ and matrices $A$ and
$A^{\prime}$ as in Lemma 5.11, satisfying additionally that
$\operatorname{{tp}}(B/M,x,y,b,c)$ and
$\operatorname{{tp}}(x^{\prime},y^{\prime},b^{\prime},c^{\prime}/M,x,y,b,c,\alpha,\beta,\gamma,\delta)$
are coheirs over $M$. Observe that $\gamma(1-x)b+\delta yb>0$. Indeed, it is
clear if $\delta\geq 0$. If $\delta<0$, then it is equivalent to
$-\frac{\gamma}{\delta}>\frac{y}{1-x}$ which is true as $\frac{y}{1-x}$ is a
positive infinitesimal, $-\frac{\gamma}{\delta}>0$, and
$\operatorname{{tp}}(\gamma,\delta/M,x,y)$ is a coheir over $M$. Let $d$ be
the left bottom entry of $A^{\prime}BA$. Hence, by Lemma 5.11, we conclude
that $d>0$ if $\alpha(1-x)b+\beta yb\geq 0$. In the case when
$\alpha(1-x)b+\beta yb<0$, we have that $d>0$ if and only if
$\frac{b^{\prime}}{c^{\prime}}<(1+\frac{1-x^{\prime}}{y^{\prime}b^{\prime}c^{\prime}})(-\frac{\gamma(1-x)+\delta
y}{\alpha(1-x)+\beta
y})=(1+\frac{1-x^{\prime}}{y^{\prime}b^{\prime}c^{\prime}})(-\frac{\gamma+\delta\frac{y}{1-x}}{\alpha+\beta\frac{y}{1-x}})=:\zeta$.

(1) Since $u_{G}=\operatorname{{tp}}(A/M)$ and
$q_{1}=\operatorname{{tp}}(-A/M)$, replacing $B$ by $-B$, we see that it is
enough to consider the case when $\gamma>0$ and to show that then $d>0$. By
the above consideration, this boils down to showing that
$\frac{b^{\prime}}{c^{\prime}}<\zeta$ if $\alpha(1-x)b+\beta yb<0$. By the
assumption of (1), $\alpha,\beta,\gamma,\delta\in{\mathbb{R}}$. Thus,
$\alpha(1-x)b+\beta yb<0$ implies that $\alpha<\beta\frac{y}{x-1}$ which is
infinitesimal, so $\alpha\leq 0$. Now, if $\alpha=0$, then
$\zeta>{\mathbb{R}}$, so $\zeta>\frac{b^{\prime}}{c^{\prime}}$ (as
$\frac{b^{\prime}}{c^{\prime}}$ is infinitesimal). If $\alpha<0$, then
$\zeta>-\frac{\gamma}{2\alpha}>\frac{b^{\prime}}{c^{\prime}}$, as
$-\frac{\gamma}{2\alpha}$ is a positive real number.

(2) is a particular case of (1).

(3) It is enough to see that $\zeta\leq\frac{b^{\prime}}{c^{\prime}}$. We have
$\zeta=(1+\frac{1-x^{\prime}}{y^{\prime}b^{\prime}c^{\prime}})(\gamma-\frac{y}{1-x})<2(\gamma-\frac{y}{1-x})$
which is a positive infinitesimal (as
$\frac{1-x^{\prime}}{y^{\prime}b^{\prime}c^{\prime}}$, $\gamma$ and
$\frac{y}{1-x}$ are positive infinitesimals, and
$\operatorname{{tp}}(\gamma/M,x,y)$ is a coheir over $M$). The conclusion
follows from the fact that $\frac{b^{\prime}}{c^{\prime}}$ is a positive
infinitesimal and $\operatorname{{tp}}(b^{\prime},c^{\prime}/M,x,y,\gamma)$ is
a coheir over $M$. ∎

We have now all the tools to prove Proposition 5.4.

###### Proof of Proposition 5.4.

Let $B:=\left(\begin{array}[]{rr}0&-1\\\ 1&0\end{array}\right)$. Then
$B^{2}=-I$ and $B^{4}=I$. So, by the explicit formula for the 2-cocycle $h$,
we have $h(B,B)=1$ and $h(B^{2},B^{2})=-1$. Hence, working in $\tilde{G}$, we
get

$(*)\;\;\;\;\;\;(B,0)^{4}=(I,2h(B,B)+h(B^{2},B^{2}))=(I,1).$

As observed after Corollary 5.8,
$u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}\cong\hat{{\mathbb{Z}}}$ is Hausdorff,
so $H(u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}})$ is trivial. We also have that
$f_{2}(u_{\mathbb{Z}}+n+u_{\mathbb{Z}})=n$ for $n\in{\mathbb{Z}}$. Take
$f\colon\tilde{G}\to u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$ from Theorem 5.1.
Using Lemma 5.10,

$\displaystyle\>f((g,n)):=(u_{G},u_{\mathbb{Z}})(g,n)(u_{G},u_{\mathbb{Z}})=(u_{G}gu_{G},u_{\mathbb{Z}}+n+u_{\mathbb{Z}}+h(g,u_{G})+h(u_{G},gu_{G}))=$
$\displaystyle\>(u_{G}gu_{G},u_{\mathbb{Z}}+n+u_{\mathbb{Z}}).$

Hence, by Theorem 5.1(3), there is a positive $k\in\mathbb{N}$ such that
$\\{g\in G:u_{G}gu_{G}=u_{G}\\}\times k\mathbb{Z}\subseteq X^{14}$. Thus, by
Lemma 5.12(2), $\\{B\\}\times k{\mathbb{Z}}\subseteq X^{14}$. Therefore, using
($*$), we conclude that $\\{I\\}\times(1+k{\mathbb{Z}})\subseteq X^{14\cdot
4}=X^{56}$, and so $I\times(\\{-12,\dots,0,\dots,12\\}+k{\mathbb{Z}})\subseteq
X^{56\cdot 12}=X^{672}$. Using Corollary 5.6, this implies that $G\times
k{\mathbb{Z}}\subseteq X^{672+24}=X^{696}$. ∎

One could also prove more directly (without using topological dynamics) a
version of Proposition 5.4 with a bigger number in place of 696, but we will
not do that, as our point was to illustrate by a non-trivial example how
Theorem 5.1 leads to a better understanding of generics in definable groups.

Finally, we give a negative answer to Question 4.13(3), and a positive answer
to Question 4.13(2) in the particular case when $X$ is a definable, generic,
symmetric subset of $\tilde{G}$. First, let us describe the context. Let $X$
be a definable, generic, symmetric subset of $\tilde{G}$. Then $X$ is a
definable in $M$ approximate subgroup and, as discussed after Proposition 5.4,
$\tilde{G}=\langle X\rangle$. By definability of types in $S(M)$, the flows
$S_{\tilde{G},M}(N)$ and $S_{\tilde{G}}(M)$ are identified (as discussed
above). We have already proved that
$H(u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}})$ is trivial. Let
$f\colon\tilde{G}\to u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$ be the
generalized definable locally compact model from Theorem 3.25 and let
$\bar{f}=f_{M}\colon S_{\tilde{G}}(M)\to
u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$ be as discussed before Question 4.13.
Note that $\bar{f}$ extends $f$. On the other hand, we have the function
$\hat{f}\colon S_{\tilde{G}}(M)\to u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$
given by $\hat{f}(p):=u_{\tilde{G}}pu_{\tilde{G}}$ which also extends $f$.

######  Proposition 5.13.

The function $\bar{f}$ is uniquely determined by the construction from Theorem
4.6 and continuous, whereas $\hat{f}$ is not continuous, and so
$\bar{f}\neq\hat{f}$. However,
$\bar{f}|_{u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}}=\hat{f}|_{u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}}=\operatorname{id}$.

###### Proof.

Identifying $u_{{\mathbb{Z}}}{\mathcal{M}}_{{\mathbb{Z}}}$ with
$\hat{{\mathbb{Z}}}$ via $f_{2}$, the second displayed computation in the
proof of Proposition 5.4 yields

$f((g,n))=(u_{G}gu_{G},n).$

It follows from Lemma 5.11 and definability of types in $S(M)$ that the sets
$\\{g\in G:u_{G}gu_{G}=u_{G}\\}$ and $\\{g\in G:u_{G}gu_{G}\neq u_{G}\\}$ are
both definable. On the other hand, the function
${\mathbb{Z}}\to\hat{{\mathbb{Z}}}$ given by $n\mapsto n$ is definable in the
sense that the preimages of any two disjoint closed subsets of
$\hat{{\mathbb{Z}}}$ can be separated by a definable set.

All of this together with Proposition 5.9 implies that $f\colon\tilde{G}\to
u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}$ is a definable map. Therefore, by
Lemma 3.2 of [GPP14] and its proof, $\bar{f}=f_{M}$ is uniquely determined by
the construction from Theorem 4.6 and continuous.

Pick a positive infinitesimal $\gamma$. Let
$B:=\left(\begin{array}[]{rr}-1&0\\\ \gamma&-1\end{array}\right).$ Choose any
net $(g_{i})_{i}$ of elements of $G$ converging to
$p:=\operatorname{{tp}}(B/M)$. Then the left bottom entries of the matrices
$g_{i}$ are positive for all $i>i_{0}$ for some $i_{0}$. So, by Lemma 5.12(1),
$u_{G}g_{i}u_{G}=u_{G}$ for all $i>i_{0}$. On the other hand, by Lemma
5.12(3), $u_{G}pu_{G}=q_{1}$. Therefore,
$\hat{f}((p,0))\in\\{u_{G}pu_{G}\\}\times
u_{\mathbb{Z}}{\mathcal{M}}_{\mathbb{Z}}=\\{q_{1}\\}\times
u_{\mathbb{Z}}{\mathcal{M}}_{\mathbb{Z}}$ and
$\hat{f}((g_{i},0))\in\\{u_{G}g_{i}u_{G}\\}\times
u_{\mathbb{Z}}{\mathcal{M}}_{\mathbb{Z}}=\\{u_{G}\\}\times
u_{\mathbb{Z}}{\mathcal{M}}_{\mathbb{Z}}$ for all $i>i_{0}$. Since the net
$((g_{i},0))_{i}$ tends to $(p,0)$ and $q_{1}\neq u_{G}$, we conclude that
$\hat{f}$ is not continuous at $(p,0)$. Hence, $\bar{f}\neq\hat{f}$ by
continuity of $\bar{f}$. More precisely, since
$\bar{f}|_{\tilde{G}}=\hat{f}|_{\tilde{G}}$, we get that
$\bar{f}(p)\neq\hat{f}(p)$.

It remains to show that
$\bar{f}|_{u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}}=\operatorname{id}$, as
directly from the definition of $\hat{f}$ we have
$\hat{f}|_{u_{\tilde{G}}{\mathcal{M}}_{\tilde{G}}}=\operatorname{id}$.
Consider any $n\in\hat{{\mathbb{Z}}}$. Our goal is to show that
$\bar{f}((u_{G},n))=(u_{G},n)$ and $\bar{f}((-u_{G},n))=(-u_{G},n)$. We will
prove the first equality; the second one can be proved analogously.

Choose any net $((g_{i},n_{i}))_{i}$ from $\tilde{G}$ converging to
$(u_{G},n)$. Then the left bottom entry of $g_{i}$ is positive for all
$i>i_{0}$ for some $i_{0}$. By Lemma 5.12(1), $u_{G}g_{i}u_{G}=u_{G}$ for all
$i>i_{0}$. Therefore, using Lemma 5.10, we get

$\bar{f}((g_{i},n_{i}))=(u_{G}g_{i}u_{G},n_{i}+h(g_{i},u_{G})+h(u_{G},g_{i}u_{G}))=(u_{G},n_{i})$

for all $i>i_{0}$, so it clearly tends to $(u_{G},n)$. Since the net
$((g_{i},n_{i}))_{i}$ tends to $(u_{G},n)$, by continuity of $\bar{f}$, we
conclude that $\bar{f}((u_{G},n))=(u_{G},n)$. ∎

## References

  * [Aus88] Joseph Auslander  “Minimal Flows and Their Extensions”, Mathematics Studies 153  Amsterdam: North-Holland, 1988
  * [BGT12] Emmanuel Breuillard, Ben Green and Terence Tao  “The structure of approximate groups”  In _Publ. Math. Inst. Hautes Etudes Sci._ 116.1 Springer ScienceBusiness Media LLC, 2012, pp. 115–221
  * [CS18] Artem Chernikov and Pierre Simon  “Definably amenable NIP groups”  In _J. Amer. Math. Soc._ 31, 2018, pp. 609–641
  * [CP12] Annalisa Conversano and Anand Pillay  “Connected components of definable groups and o-minimality I”  In _Adv. Math._ 231.2 Elsevier, 2012, pp. 605–623
  * [Ell57] Robert Ellis  “Locally compact transformation groups”  In _Duke Math. J._ 24.2 Duke University Press, 1957, pp. 119–125
  * [Ell69] Robert Ellis  “Lectures on Topological Dynamics”, Mathematics lecture note series  W. A. Benjamin, 1969
  * [Gis10] Jakub Gismatullin  “Absolute connectedness and classical groups”, 2010  arXiv:arXiv:1002.1516v5
  * [GK15] Jakub Gismatullin and Krzysztof Krupiński  “On model-theoretic connected components in some group extensions”  In _J. Math. Log._ 15, 2015, pp. 1550009 (51 pages)
  * [GPP14] Jakub Gismatullin, Davide Penazzi and Anand Pillay  “On Compactifications and the Topological Dynamics of Definable Groups”  In _Ann. Pure Appl. Logic_ 165.2 Elsevier, 2014, pp. 552–562
  * [GPP15] Jakub Gismatullin, Davide Penazzi and Anand Pillay  “Some model theory of $\operatorname{{SL}}_{2}({\mathbb{R}})$”  In _Fund. Math._ 229.2 IMPAN, 2015, pp. 117–128
  * [Gla76] Shmuel Glasner  “Proximal Flows”, Lecture Notes in Mathematics 517  Berlin: Springer-Verlag, 1976  URL: http://link.springer.com/10.1007/BFb0080139
  * [Hru12] Ehud Hrushovski  “Stable group theory and approximate subgroups”  In _J. Amer. Math. Soc._ 25.1 American Mathematical Society (AMS), 2012, pp. 189–243
  * [Hru19] Ehud Hrushovski  “Definability patterns and their symmetries”, 2019  arXiv:1911.01129
  * [Hru20] Ehud Hrushovski  “Beyond the Lascar Group”, 2020  arXiv:2011.12009
  * [HKP22] Ehud Hrushovski, Krzysztof Krupiński and Anand Pillay  “Amenability, connected components, and definable actions”  In _Sel. Math. New Ser._ 28, 16, 2022
  * [Jag15] Grzegorz Jagiella  “Definable topological dynamics and real Lie groups”  In _Math. Log. Quart._ 61.1-2 WILEY-VCH, 2015, pp. 45–55
  * [KNS19] Krzysztof Krupiński, Ludomir Newelski and Pierre Simon  “Boundedness and absoluteness of some dynamical invariants in model theory”  In _J. Math. Log._ 19.02 World Scientific Pub Co Pte Lt, 2019, pp. 1950012
  * [KR20] Krzysztof Krupiński and Tomasz Rzepecki  “Galois groups as quotients of Polish groups”  In _J. Math. Log._ 20.3, 2020, pp. 2050018
  * [KP17] Krzysztof Krupiński and Anand Pillay  “Generalized Bohr compactification and model-theoretic connected components”  In _Math. Proc. Cambridge_ 163.2 Cambridge University Press, 2017, pp. 219–249
  * [KPR18] Krzysztof Krupiński, Anand Pillay and Tomasz Rzepecki  “Topological dynamics and the complexity of strong types”  In _Israel J. Math._ 228, 2018, pp. 863–932
  * [Mac23] Simon Machado  “Approximate lattices: structure in linear groups, definition(s) and beyond”, 2023  arXiv:2306.09899v1
  * [MW15] Jean-Cyrille Massicot and Frank Wagner  “Approximate subgroups”  In _J. Éc. polytech. Math._ 2, 2015, pp. 55–63
  * [Mey72] Yves Meyer  “Algebraic numbers and harmonic analysis”  Elsevier, 1972
  * [New09] Ludomir Newelski  “Topological dynamics of definable group actions”  In _J. Symbolic Logic_ 74.1, 2009, pp. 50–72
  * [New14] Ludomir Newelski  “Topological dynamics of stable groups”  In _J. Symbolic Logic_ 79.4, 2014, pp. 1199–1223
  * [Rze18] Tomasz Rzepecki  “Bounded invariant equivalence relations”, 2018  arXiv:1810.05113
  * [Tao08] Terence Tao  “Product set estimates for non-commutative groups”  In _Combinatorica_ 28.5 Springer ScienceBusiness Media LLC, 2008, pp. 547–594


# TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment
Analysis

Ming Zhou1, Weize Quan2,3, Ziqi Zhou2,3, Kai Wang4, Tong Wang1,∗, and Dong-
Ming Yan2,3

TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis.
JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY

1School of Information Science and Technology, Donghua University, Shanghai
201620, China

2MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing 100190,
China

3School of Artificial Intelligence, University of Chinese Academy of Sciences,
Beijing 100049, China

4Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble 38000, France

Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
††footnotetext:
Regular Paper

Abstract Multimodal Sentiment Analysis (MSA) endeavors to understand human
sentiment by leveraging language, visual, and acoustic modalities. Despite the
remarkable performance exhibited by previous MSA approaches, the presence of
inherent multimodal heterogeneities poses a challenge, with the contribution
of different modalities varying considerably. Past research predominantly
focused on improving representation learning techniques and feature fusion
strategies. However, many of these efforts overlooked the variation in
semantic richness among different modalities, treating each modality
uniformly. This approach may lead to underestimating the significance of
strong modalities while overemphasizing the importance of weak ones. Motivated
by these insights, we introduce a Text-oriented Cross-Attention Network
(TCAN), emphasizing the predominant role of the text modality in MSA.
Specifically, for each multimodal sample, by taking unaligned sequences of the
three modalities as inputs, we initially allocate the extracted unimodal
features into a visual-text and an acoustic-text pair. Subsequently, we
implement self-attention on the text modality and apply text-queried cross-
attention to the visual and acoustic modalities. To mitigate the influence of
noise signals and redundant features, we incorporate a gated control mechanism
into the framework. Additionally, we introduce unimodal joint learning to gain
a deeper understanding of homogeneous emotional tendencies across diverse
modalities through backpropagation. Experimental results demonstrate that TCAN
consistently outperforms state-of-the-art MSA methods on two datasets (CMU-
MOSI and CMU-MOSEI).

Keywords multimodal sentiment analysis, cross-attention, multimodal fusion,
multimodal representations

##  1 Introduction

In recent years, the widespread use of social media and the prevalence of
smartphones equipped with high-quality cameras have led to a significant
increase in multimodal data sharing. This surge includes various forms such as
video clips on diverse social media platforms. Such multimodal data consists
of three channels: visual (image), acoustic (voice), and transcribed
linguistic (text) data. Various modalities often synergize within the same
data segment, offering additional cues that aid in disambiguating semantics
and emotions. Recognizing the research significance of sentiment, there has
been a notable surge in recent studies [1, 2, 3] focusing on the detection and
examination of human sentiments. A majority of these studies leverage
Multimodal Sentiment Analysis (MSA), aiming to extract sentiment information
from multiple data sources. This approach seeks a more comprehensive and
precise understanding of a broad spectrum of intricate human emotions,
standing in contrast to traditional unimodal sentiment analysis, which
typically relies on a single source of information.

Fig. 1: (a) illustrates the significant sentiment analysis discrepancies
using unimodality, adapted from Mult [7]. (b) shows an example of unimodal
labels and multimodal labels, where the green dotted lines represent the
process of backpropagation.

Although data from different modalities can be complementary, the redundancy
and difficulty in alignment between different modal sequences often pose
challenges to fusion. To tackle this issue, previous methods [4, 5] mostly
align visual and acoustic sequences from the perspective of text word
resolution, but this leads to increased labor costs and ignores long-term
dependencies between different modal elements. After the attention mechanism
was proposed, many methods like [6, 7] for dealing with unaligned multimodal
sequences are now based on cross-modal attention. These approaches typically
involve capturing and utilizing inter-modality correlations through sufficient
interactions between each pair of modalities. Nevertheless, this approach
causes an increase in the number of parameters and the presence of redundant
information across different modalities. Moreover, they assign equal weights
to all modalities without considering the varying semantic richness of each
modality, potentially neglecting strong modalities and overvaluing weaker
ones. Based on our observations of previous works [7, 8, 9], we notice a
dominance of the text modality in MSA. The most significant reason is that the
text modality inherently possesses a high level of structure and semantic
density. Therefore, it becomes crucial to develop a novel MSA framework that
places importance on the text modality while striking a balance in the
contributions made by different modalities.

Taking inspiration from the aforementioned observations, we propose a Text-
oriented Cross-Attention Network (TCAN) as an effective means to refine
multimodal representations. The fusion scheme involves two bi-modal fusion
modules, which is distinctly different from the traditional ternary symmetric
approach. As illustrated in Fig. 1(a), based on empirical evidence
highlighting the significance of the text modality, our model incorporates two
text-related modality pairs, TV (text-visual) and TA (text-acoustic), as
inputs for its respective bi-modal learning modules. It then encourages
modalities to iteratively enhance their information by engaging in interactive
learning with their respective counterparts. In order to maintain fairness of
the bidirectional learning process in both modalities, the two learning
networks in our proposed model are exactly the same. Our model is built upon a
fundamental framework consisting of stacked transformers, which have
demonstrated their effectiveness in the realm of multimodal learning [10].

Nonetheless, the vanilla Transformer [11] does possess certain limitations.
The cross-attention mechanism introduces redundancy and noise while directing
attention to information within the modality, particularly for the visual and
acoustic modalities [12, 13]. Since acoustic and visual modalities usually
require pre-processing [8] before being input to the network, and there will
be a certain bias during the pre-processing process, the impact of noise
caused by bias is often unavoidable. Furthermore, the redundancy in time
series between visual and acoustic sequences is remarkably high. Simple fusion
and splicing of the fused features would introduce interference from noise and
redundant information. To address these challenges, we propose a gated-control
module that incorporates two gates. Different from TMRN [12], which uses the
same gate, we train two separate gates in the network, and each gate is
responsible for different tasks. The first gate determines the proportion of
the target modality’s components to be forwarded, while the second gate
determines the proportion of compound components to be injected into the
target modality. This design aims to enhance the extraction of semantic
information from time series, effectively filtering out noise in visual and
acoustic modalities.

In this paper, we simultaneously exploit forward-guidance and backward-
guidance approaches. Inspired by methods utilizing independent annotations for
each modality and the advancements in modality-specific representation
learning, we introduce unimodal joint training during the training process.
Unlike the previous method [14], we avoid generating unimodal labels and
instead directly predict multimodal labels. As shown in Fig. 1(b), unimodal
labels and multimodal labels may be different or even opposite. To address
this disparity, we propose a shared-weight encoder to jointly train single-
modal label prediction for all three modalities, encouraging the network to
learn cross-modal homogeneousness in a backward-guiding manner.

The contributions of this work can be summarized as:

  * •

We propose a novel network called TCAN, focusing on the dominance of the text
modality in MSA tasks. TCAN leverages cross-attention and the gate network to
reinforce the interaction between text modality and the other two modalities,
with the text modality serving as the main thread to attain denoised feature
representations with low redundancy.

  * •

We introduce a shared-weight encoder to extract cross-modal homogeneous
features in multimodal joint training, obtaining better multimodal fusion
features through backpropagation.

  * •

Comprehensive experiments show that our method outperforms previous state-of-
the-art methods on two widely used MSA benchmarks (CMU-MOSI and CMU-MOSEI).

##  2 Related Work

In this section, we review several closely related topics, including
multimodal sentiment analysis, multimodal representation learning, and joint
learning.

###  2.1 Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) is to deduce human sentiment by analyzing
the language, visual, and acoustic elements present in video clips. MSA can
benefit from the diverse nature of information provided by different
modalities, which can offer varying levels of insight into human sentiment.
There are two main categories of approaches in Multimodal Sentiment Analysis:
fusion strategy-based methods [15, 16, 17] and cross-modal attention-based
methods [18, 6, 7].

The former focuses on developing advanced strategies for multimodal fusion to
create discriminative multimodal representations, Zadeh et al.[16] introduced
a Tensor Fusion Network (TFN) that can aggregate unimodal, bimodal, and
trimodal interactions progressively. To solve the problem of excessive
parameters in the fusion model, Liu et al.[15] proposed the Low-rank
Multimodal Fusion (LMF) method, which uses weights to perform low-rank matrix
decomposition. However, the fusion of multimodal features is often challenged
due to the inherent heterogeneity and redundant information present across
different modalities. To address this challenge, some research [19, 20, 21]
focuses on exploring the characteristics and commonalities of multimodal
representations by decoupling the features. This approach aims to facilitate a
more effective fusion of multimodal representations. Hazarika et al.[19]
proposed a method that decomposes the multimodal features into modality-
invariant and modality-specific components. It enables the learning of refined
multimodal representations. DMD [20] decoupled the multimodal features into a
homogeneous (modality-irrelevant) part and a heterogeneous (modality-
exclusive) part. By decoupling the features, the redundant information is
reduced, and a comprehensive perspective of the multimodal data is obtained.

In recent times, the development of MSA has been driven by cross-modal
attention-based approaches. These methods focus on learning cross-modal
correlations to enhance the representation of each modality involved. MulT [7]
is an example of a cross-modal attention-based approach that has become a
representative work in the field of Multimodal Sentiment Analysis. This
approach used a transformer-based architecture and employed various attention
mechanisms to model the interactions between different modalities, leading to
improved sentiment analysis performance. Han et al.[13] changed the ternary
symmetric approach into two bi-modal fusion modules. Lv et al.[6] introduced a
progressive modality reinforcement method based on [7], it aims to extract and
understand the inherent connections and correlations between multimodal
representations and their corresponding unimodal representations. Similar to
the aforementioned approaches [6, 7], our work also leverages the attention
mechanism as a fundamental component. However, in our work a gating mechanism
is introduced on this basis to filter out the influence of redundant features
and noise.

Fig. 2: The framework of TCAN. Given the input multimodal data, TCAN encodes
their respective shallow features $F_{m}$, where $m\in\left\\{t,v,a\right\\}$.
In the Text-oriented Cross-attention module, TCAN exploits cross-attention and
self-attention mechanisms to process text-audio pairs and text-video pairs and
applies the gated mechanism to control the impact of noise and redundant
information (Section 3.3). At the same time, we introduce a shared-weight
encoder called the Homogeneous encoder, which is used to extract homogeneous
features from unimodal for joint training (Section 3.4). Finally, we compute
the final representation from the outputs of the last layer in the Text-
oriented Cross-attention module and then concatenate them for MSA.

###  2.2 Multimodal Representation Learning

The goal of multimodal representation learning is to acquire cohesive
representations that capture the shared information across various modalities,
including but not limited to images, text, and audio. The objective is to
establish a unified representation space that facilitates the integration of
diverse modalities by establishing meaningful connections based on their
semantic similarities. Multimodal representation learning can be achieved
through two primary approaches, outlined as follows:

One of the two primary approaches is a correlation-based approach. Correlation
is considered a key measure of commonalities between multiple modalities.
Numerous prior studies have incorporated correlation analysis in their
research and achieved good results. Sun et al.[22] introduced a two-step Deep
Canonical Correlation Analysis (DCCA) framework to learn multimodal
representation. ICCN [23] also learned correlations between modalities via
DCCA, which is an example of correlation-based methods. Mittal et al.[24]
introduced a selection criterion that uses Canonical Correlational Analysis to
differentiate between ineffective and effective modalities. Although
considering the correlation between all modalities, these works neglected the
different impact of each modality on the final result, that is, the
competition between modalities.

The other approach is the alignment-based approach. Alignment refers to the
process of mapping signals with different sampling rates to a common
frequency. Traditionally, early methods [25, 26] for multimodal alignment
typically involved selecting a specific target frequency and then determining
the frames in each modality that required mapping to that particular position.
Loss functions such as CTC [27], InfoNCE [28] and its variations are commonly
employed to enhance the improvement of alignment in various applications.
After the attention mechanism was proposed, the transformer framework achieved
extraordinary results in many fields, including visual and text fields [29,
11]. Different from traditional alignment methods based on spatial distance,
attention mechanisms enable a more flexible and effective soft alignment
between two modalities. This approach has been demonstrated to be effective in
a broader range of multimodal representation learning and feature fusion
scenarios. For example, Tsai et al.[7] designed a transformer-based
architecture and employed various attention mechanisms to align different
modality sequences, leading to improved sentiment analysis performance. Yu et
al.[10] proposed a Unified Multimodal Transformer to address the MSA problem,
which was designed to simultaneously model text and visual representations.
The fusion framework used in our proposed method is based on the Transformer,
and on this basis, we also introduce a gating mechanism to reduce the impact
of noise and redundant features.

###  2.3 Joint Learning

Joint learning is also a way of multimodal feature learning, in which we can
enrich multimodal features through single-modal feature learning. Different
from the methods mentioned in Section 2.2, these approaches are usually based
on backward guidance. However, due to the unified multimodal annotation, it is
difficult for them to directly utilize multimodal labels to participate in
training. Unimodal joint learning usually has the following two methods:
additional unimodal annotations and auto-generated unimodal labels. The former
like Yu et al.[30] introduced fine-grained annotation of modality, which
needed high labor costs. An example of the latter method is [14] which
designed a unimodal label generation module based on multimodal labels and
modality representations. Different from both the above methods, we propose a
homogeneous encoder with shared weights. Through this encoder, we can extract
unimodal homogeneous features to participate in unimodal joint training.

##  3 Method

The overall framework of our TCAN is shown in Fig. 2, which contains three
components: 1) Unimodal feature extraction module: this module focuses on
generating feature representations for input data. We leverage pre-trained
BERT [31] for text data, while visual and acoustic features are extracted
using pre-trained Facet [32] and COVAREP [33], respectively. 2) Text-oriented
cross-attention network: this section employs modality fusion and gate
mechanism to facilitate the interaction among unimodal features. Categorizing
the extracted features into visual-text and acoustic-text pairs, we implement
self-attention on the text modality and apply text-queried cross-attention to
the visual and acoustic modalities. To eliminate noise and redundant
representations, a gated mechanism with two gates is incorporated into the
attention modules. 3) Unimodal joint learning: in addition to the above
multimodal prediction branch, the unimodal data from the three modalities are
collectively trained to predict single-modal labels with a shared-weight
encoder. This encourages the network to learn cross-modal consistency through
backpropagation.

###  3.1 Problem Statement

The multimodal sentiment analysis task in this work uses the text ($t$),
visual ($v$), and acoustic ($a$) information from the same video clip as
inputs, which are represented as $X_{m}\in R^{T_{m}\times d_{m}}$ for each
modality $m\in\left\\{t,v,a\right\\}$. Throughout the following text, the
notations $T_{m}$ and $d_{m}$ are employed to denote sequence length and
feature dimension of modality $m$, respectively. Our model aims to thoroughly
explore and integrate sentiment-related information from these input unaligned
multimodal sequences. The ultimate objective is to generate a text-driven
multimodal representation, which can be utilized to predict the sentiment
analysis results accurately.

###  3.2 Unimodal Feature Extraction Module

In order to attain enhanced feature representations for the text modality, we
employ a pre-trained BERT [31] to extract sentence-level features:

$F^{bert}_{t}=\mathbf{BERT}(X_{t};\theta^{bert}_{t}),$ (1)

where $\theta_{t}^{bert}$ means the parameters of BERT. In the visual and
acoustic modalities, we follow previous methods [14, 20] to utilize pre-
trained models (Facet [32] and COVAREP [33]) for extracting initial features
$F^{facet}_{v}$ and $F^{cova}_{a}$ from raw input data. Subsequently, we apply
a one-dimensional convolution network (1D) to capture the temporal
characteristics and project these features into a fixed dimension as $F_{m}\in
R^{T_{m}\times d}$, where $m\in\left\\{t,v,a\right\\}$.

$\begin{split}F_{a}&=\mathbf{1D}_{a}(F^{cova}_{a};\theta^{1D}_{a}),\\\
F_{v}&=\mathbf{1D}_{v}(F^{facet}_{v};\theta^{1D}_{v}),\\\
F_{t}&=\mathbf{1D}_{v}(F^{bert}_{t};\theta^{1D}_{t}),\end{split}$ (2)

where $\theta_{m}^{1D}$ refers to the parameters of one-dimensional
convolutional network.

Fig. 3: The architecture of the Text-oriented Cross-attention module. The
cross-attention block takes text-audio and text-video pairs as its input and
the self-attention block takes text as its input. After the cross-attention
block, two gates were introduced to eliminate adverse information. The memory
gate decides how much proportion of the target modality’s components to be
kept forwarding, and the fuse gate decides how much proportion of fused
components to be injected into the target modality.

###  3.3 Text-oriented Cross-Attention Module

The architecture of this module is shown in Fig. 3.2. In our TCAN, text-
oriented cross-attention modules are stacked $N$ times. We also evaluate the
impact of stack number on the final results in the ablation experiment. The
module contains a cross-attention block for text-oriented feature fusion and a
self-attention block for refining text features. Additionally, a gate
mechanism is incorporated to effectively integrate useful information and
filter out redundancies between these two blocks. The cross-attention block
takes $F_{t}^{[i]}$ and $F_{m\to t}^{[i]}$ as its inputs, where
$m\in\left\\{a,v\right\\}$, and the superscription $i$ indicates the _i_ -th
layer. First, we perform a layer normalization (LN) on the features: $F_{m\to
t}^{[i]}=\mathbf{LN}(F_{m\to t}^{[i]})$ and
$F_{t}^{[i]}=\mathbf{LN}(F_{t}^{[i]})$. Then, we put them into a cross-
attention (CA) block:

$\displaystyle F_{m\to t}^{[i+1]}$ $\displaystyle=\mathbf{CA}_{m\to
t}^{[i]}(F_{m\to t}^{[i]},F_{t}^{[i]})$ (3)
$\displaystyle=softmax(\frac{F_{t}^{[i]}W_{Q_{t}}W_{K_{m}}^{T}{F_{m\to
t}^{[i]}}^{T}}{\sqrt{d}})F_{m\to t}^{[i]}W_{V_{m}},$

where $F_{m\to t}^{[0]}=F_{m}$, $F_{t}^{[0]}=F_{t}$, and $W_{*}$ means a
learnable weight matrix through which the corresponding query, key, and value
can be obtained. And the self-attention (SA) block takes $F_{t}^{[i]}$ as
input to obtain $F_{t}^{[i+1]}$:

$\displaystyle F_{t}^{[i+1]}$
$\displaystyle=\mathbf{SA}_{t}^{[i]}(F_{t}^{[i]})$ (4)
$\displaystyle=softmax(\frac{F_{t}^{[i]}W_{Q_{t}}W_{K_{t}}^{T}{F_{t}^{[i]}}^{T}}{\sqrt{d}})F_{t}^{[i]}W_{V_{t}},$

where $W_{*_{t}}$ represents a learnable weight matrix related to query, key,
and value of text modality.

Exclusive reliance on the aforementioned blocks for feature fusion may
introduce interference from noise and redundancy through simple attention
computation. To address this, we introduce two gates to mitigate the influence
of unreliable information. We can use these two gates to adaptively determine
the proportion of $F_{m\to t}^{[i]}$ and $F_{m\to t}^{[i+1]}$ that can pass
through to achieve features that are more beneficial to the final prediction.
The memory gate $g_{m}$ determines the proportion of the target modality’s
components to be forwarded, while the fuse gate $g_{f}$ decides the proportion
of fused components to be injected into the target modality. These gate
signals are generated from the sequential representations of the two
modalities within the same layer:

$\begin{split}g_{m}^{[i]}&=\sigma(W_{m}^{[i]}\cdot[F_{t}^{[i]}\oplus F_{m\to
t}^{[i]}]),\\\ g_{f}^{[i]}&=\sigma(W_{f}^{[i]}\cdot[F_{t}^{[i]}\oplus F_{m\to
t}^{[i]}]),\end{split}$ (5)

where $W_{*}^{[i]}\in R^{2d\times d}$ is the projection matrix and the
subscripts $m$ and $f$ are used to distinguish the two gates, $\sigma$ denotes
the sigmoid non-linearity function and $\oplus$ represents concatenation.
Finally, the output features can be written as:

$F_{m\to t}^{[i+1]}=g_{m}^{[i]}\odot F_{m\to t}^{[i]}+g_{f}^{[i]}\odot F_{m\to
t}^{[i+1]},$ (6)

where $\odot$ means component-wise multiplication.

This operation can filter the noisy information and redundant features
produced by the cross-modal interactions while learning to measure the fusion
ratio of two modalities. After that, following the feed-forward structures in
transformers, we process $F_{m\to t}^{[i+1]}$ and $F_{t}^{[i+1]}$ by a
position-wise Feed Forward Network (FFN) with residual connections:

$\begin{split}F_{m\to t}^{[i+1]}&=\mathbf{FFN}(\mathbf{LN}(F_{m\to
t}^{[i+1]}))+F_{m\to t}^{[i+1]},\\\
F_{t}^{[i+1]}&=\mathbf{FFN}(\mathbf{LN}(F_{t}^{[i+1]}))+F_{t}^{[i+1]}.\end{split}$
(7)

We calculate the final feature representation by concatenating the outputs of
the last layer in each module. Each module comprises two heads, resulting in a
total of four heads for concatenation. By combining the outputs of the four
heads from the two text-oriented cross-modal attention modules, the final
representation can be obtained as follows:

$F_{final}=F_{a\to t}\oplus F_{t}\oplus F_{v\to t}\oplus F_{t}.$ (8)

Subsequently, the fused feature, denoted as $F_{final}$ is fed into a Multi-
Layer Perceptron (MLP) to predict the final sentiment label $y_{pred}$:

$y_{pred}=\mathbf{MLP}(F_{final};\theta^{mlp}_{final}),$ (9)

where $\theta^{mlp}_{final}$ means the parameters of MLP.

###  3.4 Unimodal Joint Learning

Each modality has a different impact on the prediction results, and the
sentiment of some modalities is opposite to the ground truths. For example, a
multimodal sample with a positive label can be negative in the visual
modality. To gain a deeper insight into the homogeneous emotional tendencies
across diverse modalities, we introduce a unimodal joint learning module in
addition to the above multimodal branch. This approach helps encourage the
cross-attention network to understand the latent sentiment semantic of each
modality more profoundly through backpropagation. Specifically, taking the
extracted unimodal features as inputs, we employ a shared-weight encoder
$\epsilon$ to enable single-modal prediction for all three modalities:

$F^{\prime}_{m}=\epsilon(F_{m};\theta_{\epsilon}),$ (10)

where $m\in\left\\{t,v,a\right\\}$, and $\theta_{\epsilon}$ refers to the
parameters of the share-weight encoder. The encoder is a one-dimensional
convolution network in our implementation. Once we obtain the homogeneous
features from all unimodal features, they are fed into an MLP to generate
unimodal predictions denoted as $y_{m}$:

$y_{m}=\mathbf{MLP}(F^{\prime}_{m};\theta^{mlp}_{m}),$ (11)

where $\theta^{mlp}_{m}$ means the parameter of MLP. Note that this strategy
is only performed during the training phase.

###  3.5 Optimization Objective

The optimization objective is based on L1 loss, and the loss function for our
multimodal prediction is formulated as follows:

$\mathcal{L}_{multi}=\frac{1}{K}\sum_{k=1}^{K}|y^{k}_{pred}-y^{k}_{gt}|,$ (12)

where $K$ is the number of multimodal samples. $y^{k}_{pred}$ and $y^{k}_{gt}$
represents the predicted and ground-truth label of the _k_ -th sample
respectively. Similarly, in the unimodal joint learning part, the loss can be
written as:

$\mathcal{L}_{uni}=\frac{1}{K}\sum_{k=1}^{K}\sum_{m\in\\{t,v,a\\}}|y^{k}_{m}-y^{k}_{gt}|,$
(13)

where $y^{k}_{m}$ is the predicted unimodal label for modality $m$ in the _k_
-th sample.

In summary, the total loss can be calculated by:

$\mathcal{L}_{total}=\mathcal{L}_{multi}+\lambda\mathcal{L}_{uni},$ (14)

where $\lambda$ is a tunable hyperparameter.

##  4 Experiments

In this section, we illustrate our experimental settings including datasets
and implementation details, compare our method with advanced competitors, and
conduct ablation studies and visualization to deeply analyze our model.

###  4.1 Experimental Settings

Datasets. We evaluate TCAN on CMU-MOSI [34] and CMU-MOSEI [35] datasets. To
achieve a more comprehensive comparison, the experiments are conducted under
word-unaligned settings. Table 4.1 depicts the composition of these two
datasets. The CMU-MOSI dataset comprises 2,199 short monologue video clips.
The acoustic features are extracted at a sampling rate of 12.5 Hz, while the
visual features are extracted at a sampling rate of 15 Hz. The CMU-MOSEI
dataset consists of a significantly larger number of samples compared to CMU-
MOSI, with a total of 22,856 movie review video clips sourced from YouTube
(approximately 10 times the size of CMU-MOSI). The acoustic features in CMU-
MOSEI are extracted at a sampling rate of 20 Hz, while the visual features are
extracted at a sampling rate of 15 Hz. For both CMU-MOSI and CMU-MOSEI
datasets, each sample is annotated with a sentiment score ranging from -3 to
3. The sentiment scores represent different levels of sentiment, including
highly negative, negative, weakly negative, neutral, weakly positive,
positive, and highly positive. Following previous works [14, 7], we evaluate
the model performance with the following metrics: the 7-class accuracy
(Acc-7), the binary accuracy (Acc-2) where zero excluded evaluation results
are used, the mean absolute error (MAE), the correlation of the model’s
prediction with human (Corr), and the F1 score.

Implementation details. On the two datasets, we exploit a BERT-base-uncased
pre-trained model [31] to obtain a 768-dimensional hidden state as the word
features. For visual modality, Facet [32] was utilized to encode each video
frame and detect the presence of all 35 facial action units [36, 37]. To
extract features from the acoustic modality, COVAREP [33] was employed and
processed to generate a set of 74-dimensional features. The hyperparameter
$\lambda$ in our experiment is set to 0.5. All experiments were conducted
using PyTorch on an RTX 3090 GPU with 24GB memory. The training batch size was
set to 16, and the TCAN model was trained for 50 epochs until convergence.

Table 1:  The compositions of the number of samples in the CMU-MOSI and the
CMU-MOSEI dataset.

Datasets | #Train | #Validation | #Test
---|---|---|---
CMU-MOSI | 1,284 | 229 | 686
CMU-MOSEI | 16,326 | 1,871 | 4,659

Table 2:  Comparison results on the CMU-MOSI. Bold is the best.

Method | $MAE\downarrow$ | $Corr\uparrow$ | $Acc_{7}\uparrow$ | $Acc_{2}\uparrow$ | $F_{1}\uparrow$
---|---|---|---|---|---
TFN [16] | 0.901 | 0.698 | 34.9 | 80.8 | 80.7
LMF [15] | 0.917 | 0.695 | 33.2 | 82.5 | 82.4
MulT [7] | 0.861 | 0.711 | - | 84.1 | 83.9
MISA [19] | 0.783 | 0.761 | 42.3 | 83.4 | 83.6
MAG-BERT [36] | 0.731 | 0.789 | - | 84.3 | 84.3
Self-MM [14] | 0.718 | 0.796 | 45.8 | 84.45 | 84.44
MFSA [39] | 0.856 | 0.722 | 41.4 | 83.3 | 83.7
DMD [20] | - | - | 41.4 | 84.5 | 84.4
TCAN (Ours) | 0.714 | 0.797 | 46.79 | 86.28 | 86.15

Table 3:  Comparison results on the CMU-MOSEI dataset. Bold is the best.

Method | $MAE\downarrow$ | $Corr\uparrow$ | $Acc_{7}\uparrow$ | $Acc_{2}\uparrow$ | $F_{1}\uparrow$
---|---|---|---|---|---
TFN [16] | 0.593 | 0.700 | 50.2 | 82.5 | 82.1
LMF [15] | 0.623 | 0.677 | 48.0 | 82.0 | 82.1
MulT [7] | 0.580 | 0.703 | - | 82.5 | 82.9
MISA [19] | 0.568 | 0.724 | - | 84.23 | 83.97
MAG-BERT [36] | 0.539 | 0.753 | - | 85.2 | 85.1
Self-MM [14] | 0.536 | 0.763 | 52.8 | 84.95 | 84.85
MFSA [39] | 0.574 | 0.724 | 53.2 | 83.8 | 83.6
DMD [20] | - | - | 53.7 | 85.0 | 84.9
TCAN (Ours) | 0.532 | 0.774 | 53.10 | 86.27 | 86.17

###  4.2 Comparisons with the State-of-the-Art

The proposed approach is compared to the existing state-of-the-art methods,
including TFN [16], LMF [17], Mult [7], MISA [19], MAG-BERT [38], Self-MM
[14], MFSA [39], and DMD [20]. Table 4.1 and Table 4.1 show the corresponding
results on the CMU-MOSI and CMU-MOSEI datasets, respectively. Our approach
demonstrates remarkable superiority over the majority of previous methods [7,
19, 14], showcasing substantial performance improvements across all metrics on
both datasets. Compared with the traditional feature fusion method, we added a
gating mechanism during modal fusion to filter out the influence of noise and
redundant features, and we also introduced unimodal joint training. Therefore,
our method achieved relatively good results. Moreover, our model exhibits
superior or competitive performance compared to the current state-of-the-art
DMD [20] across most metrics, e.g., improved by 1.78$\%$ (Acc-2) and 1.75$\%$
(F1 score) in Table 4.1, indicating the effectiveness of our proposed
approach.

Table 4:  Ablation study on the influence of modality on the CMU-MOSEI
dataset. The unimodal results are adapted from Mult [7].

Input | $MAE\downarrow$ | $Corr\uparrow$ | $Acc_{7}\uparrow$ | $Acc_{2}\uparrow$ | $F_{1}\uparrow$
---|---|---|---|---|---
T only | 0.653 | 0.631 | 46.5 | 77.4 | 78.2
A only | 0.764 | 0.310 | 41.1 | 65.5 | 68.8
V only | 0.759 | 0.343 | 43.5 | 66.4 | 69.3
TV pairs | 0.544 | 0.765 | 52.45 | 85.36 | 85.32
TA pairs | 0.554 | 0.749 | 52.26 | 85.30 | 85.28
TV+TA pairs | 0.532 | 0.774 | 53.10 | 86.27 | 86.17

Table 5:  Ablation study on the influence of the center modality on the CMU-MOSI.  Model | $MAE\downarrow$ | $Corr\uparrow$ | $Acc_{7}\uparrow$ | $Acc_{2}\uparrow$ | $F_{1}\uparrow$
---|---|---|---|---|---
Acoustic-oriented | 0.777 | 0.786 | 42.86 | 84.15 | 84.22
Visual-oriented | 0.745 | 0.789 | 44.61 | 84.30 | 84.31
Text-oriented (Ours) | 0.714 | 0.797 | 46.79 | 86.28 | 86.15
Table 6:  Ablation study on the influence of different network modules on the
CMU-MOSI.

Text-oriented Module | Gated Module | Joint Learning | $MAE\downarrow$ | $Corr\uparrow$ | $Acc_{7}\uparrow$ | $Acc_{2}\uparrow$ | $F_{1}\uparrow$
---|---|---|---|---|---|---|---
$\checkmark$ |  |  | 0.746 | 0.776 | 42.17 | 84.30 | 84.24
$\checkmark$ | $\checkmark$ |  | 0.755 | 0.783 | 42.42 | 84.76 | 84.67
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.714 | 0.797 | 46.79 | 86.28 | 86.15

###  4.3 Ablation Study

The overall outstanding performance of TCAN has confirmed its superiority. To
grasp the significance of various components and ascertain the dominance of
the text modality, we conduct systematic ablation experiments on the CMU-MOSI
and CMU-MOSEI datasets. The results of these experiments are detailed in the
following tables.

We separately remove each modality to investigate the performance of our
proposed model. As illustrated in Table 4.2, we can see that the performance
of “T only” is remarkably better than “A only” and “V only”, which to some
extent implies the importance of text for MSA. When introducing text-oriented
cross-attention modules, “TV pairs” or “TA pairs”, the performance has an
apparent gain. Then, combining “TV pairs” and “TA pairs” further improves the
recognition performance. In addition, the performance drop is relatively large
when the visual modality is removed (comparing “TV pairs” and “TA pairs”
related to “TV+TA pairs”).

Fig. 4: Performance of TCAN with different hyperparameter $N$ on the CMU-
MOSI. Here we use $F_{1}$ scores to show the performance.

To illustrate the dominance of the text modality, we substitute the other two
modalities as the dominant modality for the experiments. As illustrated in
Table 5, the acoustic- and visual-oriented models consistently experience
noticeable performance degradation. These observations indicate that the text
modality possesses richer semantics and less noise, resulting in better
feature reinforcement of the other two modalities.

Fig. 5: Some representative examples for the visualization analysis of MSA.
Red represents positive sentiment and blue represents negative sentiment.

We explore the importance of the proposed components by removing the gated
mechanism and unimodal joint training separately. We can see that the gain
degrades when removing one of the modules. As shown in Table 6, these
observations suggest that gated mechanism and joint training modules play an
important role in the result.

Finally, to investigate the impact of the number $N$ of the Text-oriented
Cross-attention module on the model’s performance, we carried out experiments
on the CMU-MOSI dataset using varying values for parameter $N$. The results
are illustrated in Fig. 4.3. As $N$ increases, we observe a pattern in which
the F1 scores initially rise and then decline, with the network achieving its
optimal performance when $N=5$. We believed that the larger $N$ could result
in better performance. Nevertheless, experiments indicate that an excessive
number of layers may restrict the text modality’s capacity to guide the other
two modalities. We should choose the appropriate network for different
datasets, which is exactly what our proposed TCAN can flexibly do. When
transitioning our model to a more intricate dataset, we can strategically
augment the number of Text-Oriented Cross-modal Attention modules to attain
optimal performance.

###  4.4 Visualization Analysis

To further analyze our proposed framework for MSA, we visualize several
misleading examples as shown in Fig. 5. We use different colors to distinguish
sentiments, where red refers to positive sentiment and blue refers to negative
sentiment.

In Fig. 5(a), a positive phase GIANT SMILE exists in text modality, but the
multimodal label is -0.8, which demonstrates the limitation of only text
modality in this case. In fact, there are mostly negative video frames and
audio tones that cause the label to be negative. The prediction result of our
model is -0.72, which shows that our method can handle the restriction of text
modality. In Fig. 5(b), the visual modality has a slight negative sentiment,
while the words DEFINITELY, A GOOD MOVIE show that text modality is obviously
positive. By leveraging the text modality, TCAN partially compensates for the
limitations in sentiment of the acoustic and visual modalities. The result of
our model is 0.65, which is similar to the multimodal label. In Fig. 5(c),
there is a positive sentence OH MY GOD I LOVE YOU with a happy tone. In Fig.
5(d), both modalities are slightly negative. In these cases, TCAN can also
obtain more accurate results. These cases mentioned above demonstrate the
superiority of our model in assigning reasonable weights to different
modalities for multimodal sentiment analysis. This also implies that our model
will not underestimate the significance of strong modalities while
overemphasizing the importance of weak ones.

##  5 Conclusion

In this paper, we propose a cross-attention network designed for text-oriented
multimodal sentiment analysis. Our objective is to foster interaction and
fusion among three modalities, each comprising unaligned sequences. The
proposed methodology is built upon both inter- and intra-modal attention
mechanisms. Sequences from the text modality guide the attention of the other
two modalities, facilitating the transfer of information across and within
different modalities. To address issues related to noise and redundant
information, we incorporate a gated mechanism into the model, effectively
mitigating these problems. Simultaneously, we jointly train both the
multimodal and unimodal tasks to enable the learning of their respective
consistencies and differences. Experimental results demonstrate that the
proposed approach consistently outperforms the state-of-the-art in multimodal
sentiment analysis benchmarks.

## Conflict of Interest

The authors declare that they have no conflict of interest.

## References

  * [1] Du Y, Yang D, Zhai P, Li M, Zhang L. Learning associative representation for facial expression recognition. In Proc. the 2021 IEEE International Conference on Image Processing, Sep. 2021, pp.889–893. DOI: 10.1109/ICIP42928.2021.9506181.
  * [2] Yang D, Chen Z, Wang Y, Wang S, Li M, Liu S, Zhang L. Context de-confounded emotion recognition. In Proc. the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2023, pp.19005-19015. DOI: 10.1109/CVPR52729.2023.01822.
  * [3] Yang D, Huang S, Wang S, Liu Y, Zhai P, Su L, Zhang L. Emotion recognition for multiple context awareness. In Proc. the 2022 European Conference on Computer Vision, Oct. 2022, pp.144-162. DOI: 10.1007/978-3-031-19836-6_9.
  * [4] Tsai Y H H, Liang P P, Zadeh A, Morency L P, Salakhutdinov R. Learning factorized multimodal representations. arXiv:1806.06176, 2018. https://arxiv.org/abs/1806.06176, Mar. 2024.
  * [5] Wang Y, Shen Y, Liu Z, Liang P P, Zadeh A, Morency L P. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proc. the 2019 AAAI Conference on Artificial Intelligence, Jul. 2019, pp.7216-7223. DOI: 10.1609/aaai.v33i01.33017216.
  * [6] Lv F, Chen X, Huang Y, Duan L, Lin G. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In Proc. the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2021, pp.2554-2562. DOI: 10.1109/CVPR46437.2021.00258.
  * [7] Tsai Y H H, Bai S, Liang P P, Kolter J Z, Morency L P, Salakhutdinov R. Multimodal transformer for unaligned multimodal language sequences. In Proc. the 2019 Conference on Association for Computational Linguistics, Jul. 2019, pp.6558–6569. DOI: 10.18653/v1/p19-1656.
  * [8] Chen M, Wang S, Liang P P, Baltrušaitis T, Zadeh A, Morency L P. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proc. the 19th ACM International Conference on Multimodal Interaction, Nov. 2017, pp.163-171. DOI: 10.1145/3136755.3136801.
  * [9] Pham N Q, Niehues J, Ha T L, Waibel A. Improving zero-shot translation with language-independent constraints. arXiv:1906.08584, 2019. https://arxiv.org/abs/1906.08584, Mar. 2024.
  * [10] Yu J, Jiang J, Yang L, Xia R. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In Proc. the 58th Annual Meeting of the Association for Computational Linguistics, Jul. 2020, pp.3342–3352. DOI: 10.18653/v1/2020.acl-main.306.
  * [11] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Polosukhin I. Attention is all you need. In Proc. the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp.6000-6010. DOI: 10.5555/3295222.3295349.
  * [12] Lei Y, Yang D, Li M, Wang S, Chen J, Zhang L. Text-oriented modality reinforcement network for multimodal sentiment analysis from unaligned multimodal sequences. In Proc. the 2024 International Conference on Artificial Intelligence, Feb.2024 pp.189-200. DOI: 10.1007/978-981-99-9119-8_18
  * [13] Han W, Chen H, Gelbukh A, Zadeh A, Morency L P, Poria S. Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis. In Proc. the 2021 International Conference on Multimodal Interaction, Oct. 2021, pp.6-15. DOI: 10.1145/3462244.3479919.
  * [14] Yu W, Xu H, Yuan Z, Wu J. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proc. the 2021 AAAI Conference on Artificial Intelligence, May. 2021, pp.10790-10797. DOI: 10.1609/aaai.v35i12.17289.
  * [15] Liu Z, Shen Y, Lakshminarasimhan V B, Liang P P, Zadeh A Morency L P. Efficient low-rank multimodal fusion with modality-specific factors. arXiv:1806.00064, 2018. https://arxiv.org/abs/1806.00064, Mar. 2024.
  * [16] Zadeh A, Chen M, Poria S, Cambria E, Morency L P. Tensor fusion network for multimodal sentiment analysis. arXiv:1707.07250, 2017. https://arxiv.org/abs/1707.07250, Mar. 2024.
  * [17] Zadeh A, Liang P P, Mazumder N, Poria S, Cambria E, Morency L P. Memory fusion network for multi-view sequential learning. In: Proc. the 2018 AAAI Conference on Artificial Intelligence, Apr. 2018, pp.5634-5641. DOI: 10.1609/aaai.v32i1.12021.
  * [18] Liang T, Lin G, Feng L, Zhang Y, Lv F. Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion. In Proc. the 2021 IEEE/CVF International Conference on Computer Vision, Oct. 2021, pp.8148-8156. DOI: 10.1109/ICCV48922.2021.00804.
  * [19] Hazarika D, Zimmermann R, Poria S. MISA: Modality-invariant and -specific representations for multimodal sentiment analysis. In Proc. the 28th ACM International Conference on Multimedia, Oct. 2020, pp.1122-1131. DOI: 10.1145/3394171.3413678.
  * [20] Li Y, Wang Y, Cui Z. Decoupled multimodal distilling for emotion recognition. In Proc. the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2023, pp.6631-6640. DOI: 10.1109/CVPR52729.2023.00641.
  * [21] Yang D, Huang S, Kuang H, Du Y, Zhang L. Disentangled representation learning for multimodal emotion recognition. In Proc. the 30th ACM International Conference on Multimedia, Oct. 2022, pp.1642-1651. DOI: 10.1145/3503161.3547754.
  * [22] Sun Z, Sarma P K, Sethares W, Bucy E P. Multi-modal sentiment analysis using deep canonical correlation analysis. arXiv:1907.08696, 2019. https://arxiv.org/abs/1907.08696, Mar. 2024.
  * [23] Sun Z, Sarma P, Sethares W, Liang Y. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In Proc. the 2020 AAAI Conference on Artificial Intelligence, Apr. 2020, pp.8992-8999. DOI: 10.1609/aaai.v34i05.6431.
  * [24] Mittal T, Bhattacharya U, Chandra R, Bera A, Manocha D. M3ER: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In Proc. the 2020 AAAI Conference on Artificial Intelligence, Apr. 2020, pp.1359-1367. DOI: 10.1609/aaai.v34i02.5492.
  * [25] Blikstein P, Worsley M. Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 2016, 3(2): 220-238. DOI: 10.18608/jla.2016.32.11.
  * [26] Zadeh A, Zellers R, Pincus E, Morency L P. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 2016, 31(6): 82-88. DOI: 10.1109/MIS.2016.94.
  * [27] Graves A, Fernández S, Gomez F, Schmidhuber J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. the 23rd International Conference on Machine learning, Jun. 2006, pp.369-376. DOI: 10.1145/1143844.1143891.
  * [28] He K, Fan H, Wu Y, Xie S, Girshick R. Momentum contrast for unsupervised visual representation learning. In Proc. the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.9729-9738. DOI: 10.1109/CVPR42600.2020.00975.
  * [29] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Houlsby N. An image is worth 16$\times$16 words: Transformers for image recognition at scale. arXiv:2010.11929, 2020. https://arxiv.org/abs/2010.11929, Mar. 2024.
  * [30] Yu W, Xu H, Meng F, Zhu Y, Ma Y, Wu J, Yang K. CH-SIMS: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proc. the 58th Annual Meeting of the Association for Computational Linguistics, Jul. 2020, pp.3718-3727. DOI: 10.18653/v1/2020.acl-main.343.
  * [31] Devlin J, Chang M W, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, 2018. https://arxiv.org/abs/1810.04805, Mar. 2024.
  * [32] Baltrušaitis T, Robinson P, Morency L P. Openface: an open source facial behavior analysis toolkit. In Proc. the 2016 IEEE Winter Conference on Applications of Computer Vision, Mar. 2016, pp.1-10. DOI: 10.1109/WACV.2016.7477553.
  * [33] Degottex G, Kane J, Drugman T, Raitio T, Scherer S. COVAREP-A collaborative voice analysis repository for speech technologies. In Proc. the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing, May. 2014, pp.960-964. DOI: 10.1109/ICASSP.2014.6853739.
  * [34] Zadeh A, Zellers R, Pincus E, Morency L P. MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv:1606.06259, 2016. https://arxiv.org/abs/1606.06259, Mar. 2024.
  * [35] Zadeh A B, Liang P P, Poria S, Cambria E, Morency L P. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proc. the 56th Annual Meeting of the Association for Computational Linguistics, Jul. 2018, pp.2236-2246. DOI: 10.18653/v1/P18-1208.
  * [36] Li Y, Zeng J, Shan S, Chen X. Self-supervised representation learning from videos for facial action unit detection. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.10924-10933. DOI: 10.1109/CVPR.2019.01118.
  * [37] Li Y, Zeng J, Shan S. Learning representations for facial actions from unlabeled videos. IEEE Trans. Pattern Analysis and Machine Intelligence, 2020, 44(1): 302-317. DOI: 10.1109/TPAMI.2020.3011063.
  * [38] Rahman W, Hasan M K, Lee S, Zadeh A, Mao C, Morency L P, Hoque E. Integrating multimodal information in large pretrained transformers. In Proc. the 2020 Conference on Association for Computational Linguistics, Jul. 2020, pp.2359–2369. DOI: 10.18653/v1/2020.acl-main.214.
  * [39] Yang D, Kuang H, Huang S, Zhang L. Learning modality-specific and -agnostic representations for asynchronous multimodal language sequences. In Proc. the 30th ACM International Conference on Multimedia, Oct. 2022, pp.1708-1717. DOI: 10.1145/3503161.3547755.