# Deep Learning-Aided OFDM-Based Generalized Optical Quadrature Spatial
Modulation

Chen Chen, Lin Zeng, Xin Zhong, Shu Fu, Min Liu, and Pengfei Du This work was
supported in part by the National Natural Science Foundation of China under
Grant 61901065 and in part by the Project funded by China Postdoctoral Science
Foundation under Grant 2021M693744.C. Chen, L. Zeng, X. Zhong, S. Fu, and M.
Liu are with the School of Microelectronics and Communication Engineering,
Chongqing University, Chongqing 400044, China (e-mail: {c.chen, 202012021008,
201912131098, shufu, liumin}@cqu.edu.cn).P. Du is with the A*STAR’s Singapore
Institute of Manufacturing Technology (SIMTech), Singapore 138634 (e-mail:
Du_Pengfei@simtech.a-star.edu.sg).

###### Abstract

In this paper, we propose an orthogonal frequency division multiplexing
(OFDM)-based generalized optical quadrature spatial modulation (GOQSM)
technique for multiple-input multiple-output optical wireless communication
(MIMO-OWC) systems. Considering the error propagation and noise amplification
effects when applying maximum likelihood and maximum ratio combining (ML-
MRC)-based detection, we further propose a deep neural network (DNN)-aided
detection for OFDM-based GOQSM systems. The proposed DNN-aided detection
scheme performs the GOQSM detection in a joint manner, which can efficiently
eliminate the adverse effects of both error propagation and noise
amplification. The obtained simulation results successfully verify the
superiority of the deep learning-aided OFDM-based GOQSM technique for high-
speed MIMO-OWC systems.

###### Index Terms:

Visible light communication, orthogonal frequency division multiplexing,
multiple-input multiple-output, deep neural network.

##  I Introduction

With the explosive increasing of mobile data traffic in recent years,
traditional radio frequency communication technologies such as WiFi might not
be able to support the heavy mobile data traffic in the near future. Owing to
its abundant spectrum resource, optical wireless communication (OWC) exploring
visible light, infrared and ultra-violet spectrum has been emerging as a
promising candidate to meet the requirement of the ever-increasing mobile data
traffic [1]. Although the spectrum resource of OWC systems is abundant, the
achievable data rate of practical OWC systems is largely limited by the small
modulation bandwidth of the optical components, especially the optical
transmitters. For the OWC systems using commercial off-the-shelf light-
emitting diodes (LEDs), the -3dB bandwidth is usually only a few MHz, which
greatly limits the achievable capacity of the system [2].

To boost the capacity of bandlimited OWC systems, various spectral efficiency
enhancing techniques have already been proposed so far. Among them, multiple-
input multiple-output (MIMO) transmission and orthogonal frequency division
multiplexing (OFDM) modulation are considered as two most promising techniques
[3, 4]. As a digitized MIMO scheme, optical spatial modulation (OSM) has drawn
great attention in OWC systems lately. In OSM, only a single optical
transmitter is selected to transmit constellation symbols and the spectral
efficiency of OSM systems is contributed by both the constellation symbols and
the spatial index symbols [5]. As a result, OSM systems have the advantages of
negligible inter-channel interference, high power efficiency and low
transceiver complexity [6]. Nevertheless, the achievable spectral efficiency
of OSM systems is mainly limited by the number of the optical transmitters and
it is challenging for practical OSM systems to achieve high spectral
efficiencies. In order to increase the spectral efficiency contributed by the
spatial index symbols and hence further enhance the spectral efficiency of OSM
systems, generalized OSM (GOSM) has been proposed where multiple optical
transmitters are selected to simultaneously transmit the same constellation
symbol [7]. Furthermore, the combination of both OFDM and various OSM
techniques can be a more efficient way to enhance the spectral efficiency of
OWC systems. Specifically, OFDM-based OSM has been reported in [8, 9], while
OFDM-based GOSM has been proposed in [10].

In this paper, we for the first time propose a novel OFDM-based generalized
optical quadrature spatial modulation (GOQSM) technique for further spectral
efficiency enhancement of bandlimited MIMO-OWC systems. The proposed GOQSM is
inspired by the concept of quadrature spatial modulation (QSM) [11], where
spatial mapping can be performed in both the in-phase and quadrature
components and hence a doubled number of spatial bits can be transmitted. As a
result, the proposed GOQSM can be considered as a generalized version of QSM
in OWC systems, which can provide a much higher spectral efficiency than that
of GOSM. Moreover, we further propose two detection schemes for OFDM-based
GOQSM systems, including maximum likelihood and maximum ratio combining (ML-
MRC)-based detection and deep neural network (DNN)-aided detection. Simulation
results show that the proposed OFDM-based GOQSM technique significantly
outperforms OFDM-based GOSM when applying the deep learning-aided detection
scheme.

##  II System Model

In this paper, we consider a general MIMO-OWC system equipped with $N_{t}$
LEDs and $N_{r}$ photo-diodes (PDs). Letting $\bf{s}$ =
$[s_{1},s_{2},\cdots,s_{N_{t}}]^{T}$ be the transmitted signal vector,
$\bf{H}$ be the $N_{r}\times N_{t}$ MIMO channel matrix and $\bf{n}$ =
$[n_{1},n_{2},\cdots,n_{N_{r}}]^{T}$ be the additive noise vector, the
received signal vector $\bf{y}$ = $[y_{1},y_{2},\cdots,y_{N_{r}}]^{T}$ can be
expressed by

$\bf y=Hs+n,$ (1)

where the channel matrix of the $N_{r}\times N_{t}$ MIMO-OWC system can be
given by

$\bf H=\left[\begin{array}[]{ccc}h_{11}&\cdots&h_{1N_{t}}\\\
\vdots&\ddots&\vdots\\\ h_{N_{r}1}&\cdots&h_{N_{r}N_{t}}\end{array}\right],$
(2)

where $h_{rt}$ ($r=1,2,\cdots,N_{r};t=1,2,\cdots,N_{t}$) denotes the direct
current (DC) channel gain between the $t$-th LED and the $r$-th PD. In this
work, we assume that each LED follows the Lambertian radiation pattern and
only the line-of-sight (LOS) transmission is considered [12]. Hence, $h_{rt}$
can be calculated as follows:

$h_{rt}=\frac{(l+1)\rho A}{2\pi
d_{rt}^{2}}\mathrm{cos}^{l}(\varphi_{rt})T_{s}(\theta_{rt})g(\theta_{rt})\mathrm{cos}(\theta_{rt}),$
(3)

where $l=-\mathrm{ln}2/\mathrm{ln}(\mathrm{cos}(\Psi))$ represents the
Lambertian emission order and $\Psi$ is the semi-angle at half power of the
LED; $\rho$ and A denote the responsivity and the active area of the PD,
respectively; the distance, the angle of emission and the angle of incidence
between the $t$-th LED and the $r$-th PD are expressed by $d_{rt}$,
$\varphi_{rt}$ and $\theta_{rt}$, respectively; $T_{s}(\theta_{rt})$ is the
gain of optical filter; $g(\theta_{rt})=\frac{n^{2}}{\mathrm{sin}^{2}\Phi}$ is
the gain of optical lens, where $n$ and $\Phi$ denote the refractive index and
the half-angle field-of view (FOV) of the optical lens, respectively.

In MIMO-OWC systems, the additive noise is generally the sum of both shot and
thermal noises and it can be reasonably modeled as a real-valued zero-mean
additive white Gaussian noise (AWGN) with power $P_{n}=N_{0}B$, where $N_{0}$
and ${B}$ represent the noise power spectral density (PSD) and the signal
bandwidth, respectively.

Figure 1: Schematic diagram of the proposed OFDM-based GOQSM system.

##  III OFDM-Based GOQSM

In this section, we first describe the fundamental principle of OFDM-based
GOQSM, and then we propose two detection schemes for OFDM-based GOQSM systems,
including maximum likelihood and maximum ratio combining (ML-MRC)-based
detection and DNN-aided detection.

###  III-A Principle of OFDM-Based GOQSM

Fig. 1 shows the schematic diagram of the proposed OFDM-based GOQSM system
with $N_{t}$ LEDs and $N_{r}$ PDs. As we can see, the incoming bit stream is
divided into three parts: one part is mapped into a constellation symbol
vector b, while the other two parts are mapped into an in-phase spatial index
vector $\textbf{z}_{\textrm{Re}}$ and a quadrature spatial index vector
$\textbf{z}_{\textrm{Im}}$, respectively. The complex-valued element $b$ of
the constellation symbol vector b can be represented as
$b=b_{\textrm{Re}}+j\times b_{\textrm{Im}}$, where $b_{\textrm{Re}}$ and
$b_{\textrm{Im}}$ denote the real and imaginary parts of $b$, respectively.
Before performing GOQSM mapping, the real and imaginary parts of $b$ are
separated through real-and-imaginary separation (RIS). In OFDM-based GOQSM
systems, GOQSM mapping is performed at each subcarrier slot in the frequency
domain with respect to both the real and imaginary components. More
specifically, for an OFDM-based GOQSM system with totally $N_{t}$ LEDs and $N$
($N\leq N_{t}$) activated LEDs, $N$ out of $N_{t}$ real parts of the
subcarrier-modulating symbols corresponding to $N_{t}$ LEDs at each subcarrier
slot are selected to transmit $b_{\textrm{Re}}$ while the remaining $N_{t}-N$
real parts are set to zero. At the same time, $N$ out of $N_{t}$ imaginary
parts of the subcarrier-modulating symbols corresponding to $N_{t}$ LEDs at
each subcarrier slot are selected to transmit $b_{\textrm{Im}}$ while the
remaining $N_{t}-N$ imaginary parts are set to zero. Assuming
$\textbf{c}_{\textrm{Re}}$ and $\textbf{c}_{\textrm{Im}}$ are the resultant
real and imaginary vectors, the finally transmitted symbol vector can be
obtained as $\textbf{c}=\textbf{c}_{\textrm{Re}}+j\textbf{c}_{\textrm{Im}}$.
After that, parallel OFDM modulation and digital-to-analog (D/A) conversion
are performed and the resultant signals are used to drive $N_{t}$ LEDs,
respectively.

At the receiver side, the optical signal is first detected by $N_{r}$ PDs and
then the obtained electrical signals are digitized via analog-to-digital (A/D)
conversion. Subsequently, zero-forcing (ZF) equalization and parallel OFDM
demodulation are performed to generate the estimate of the transmitted symbol
vector $\hat{\textbf{c}}$. To further estimate the constellation symbol vector
and spatial index vector, two detection schemes including ML-MRC detection and
DNN-aided detection are proposed, which are introduced in detail in the
following subsection.

For OFDM-based GOQSM systems using $M$-ary quadrature amplitude modulation
(QAM) constellation with totally $N_{t}$ LEDs where $N$ out of $N_{t}$ LEDs
are activated to transmit data, the achievable spectral efficiency is given by

$R_{\textrm{GOQSM}}\approx\frac{1}{2}\log_{2}{(M)}+\lfloor\log_{2}{C(N_{t},N)}\rfloor,$
(4)

where $\lfloor\cdot\rfloor$ represents the floor operation and
$C(\cdot,\cdot)$ denotes the binomial coefficient. In (4), the first term on
the right-hand side is contributed by the constellation symbols where the
scaling factor $\frac{1}{2}$ is due to the imposed Hermitian symmetry
constraint for real-valued time-domain signal generation [10], while the
second term is contributed by both the in-phase and quadrature spatial index
symbols. For the purpose of comparison, the achievable spectral efficiency of
OFDM-based GOSM systems with exactly the same setup is given as follows [10]:

$R_{\textrm{GOSM}}\approx\frac{1}{2}\log_{2}{(M)}+\frac{1}{2}\lfloor\log_{2}{C(N_{t},N)}\rfloor.$
(5)

It can be clearly found that, by replacing OSM with QOSM, the spectral
efficiency contributed by the constellation symbols remains the same while the
spectral efficiency contributed by the spatial index symbols is doubled.

###  III-B Two Detection Schemes for OFDM-Based GOQSM

(a) ML-MRC detection

(b) DNN-aided detection

Figure 2: Schematic diagrams of (a) ML-MRC detection and (b) DNN-aided
detection.

As shown in Fig. 1, GOQSM detection is performed to generate the final output
bits after parallel OFDM demodulation. In the following, two GOQSM detection
schemes are proposed including ML-MRC detection and DNN-aided detection.

####  III-B1 ML-MRC Detection

The schematic diagram of the ML-MRC detection is illustrated in Fig. 2(a),
where the estimate of the transmitted symbol vector $\hat{\textbf{c}}$ =
$[\hat{c}_{1},\hat{c}_{2},\cdots,\hat{c}_{N_{t}}]^{T}$ is treated as the
input. Firstly, the real and imaginary parts of $\hat{c}_{t}$
($t=1,2,\cdots,N_{t}$) are separated via RIS. Then, ML detection is performed
with respect to both the real and imaginary parts, so as to obtain the
corresponding estimates of the real components $[\hat{b}_{\textrm{Re}}]_{m}$,
the imaginary components $[\hat{b}_{\textrm{Im}}]_{m}$, and the spatial index
vectors $\hat{\textbf{z}}_{\textrm{Re}}$ and $\hat{\textbf{z}}_{\textrm{Im}}$.
The detailed procedures to perform ML detection can be found in [10], which is
omitted here for brevity. After that, the estimates of $b_{\textrm{Re}}$ and
$b_{\textrm{Im}}$, i.e, $\hat{b}_{\textrm{Re}}$ and $\hat{b}_{\textrm{Im}}$,
can be obtained via MRC, and the estimate of the constellation symbol vector
$\hat{\textbf{b}}$ can be further obtained by combining
$\hat{b}_{\textrm{Re}}$ and $\hat{b}_{\textrm{Im}}$ via real-and-imaginary
combination (RIC). Subsequently, in-phase and quadrature spatial demapping and
constellation demapping are executed to recover the transmitted bit stream.

As discussed above, the ML-MRC detection is a two-step detection scheme, and
the errors occur during the first-step ML detection, if any, will directly
propagate to the second-step MRC, resulting in error propagation. In the
proposed OFDM-based GOQSM system, error propagation reduces the diversity gain
that can be achieved by MRC and hence degrades the overall bit error rate
(BER) performance of the system. Moreover, since the input $\hat{\textbf{c}}$
is generated after ZF equalization and parallel OFDM demodulation, the system
also suffers from the adverse effect of noise amplification [13].

####  III-B2 DNN-Aided Detection

To address the error propagation and noise amplification effects in the
proposed OFDM-based GOQSM system using the ML-MRC detection, we propose a DNN-
aided detection scheme for performance improvement of the OFDM-based GOQSM
system in the following. Fig. 2(b) illustrates the schematic diagram of the
proposed DNN-aided detection scheme, which mainly consists of a pre-processing
module, a feed-forward DNN module and a decision module. The pre-processing
module is used to separate the real and imaginary parts of $\hat{c}_{t}$ via
RIS and hence generate the input vector of the DNN module. For the OFDM-based
GOQSM system with $N_{t}$ LEDs, the size of the input vector of the DNN module
is given by $K=2N_{t}$. The feed-forward DNN module contains one input layer,
three hidden layers and one output layer. The input layer is composed of $K$
neurons, which is corresponding to the size of the input vector. The numbers
of the neurons of the three hidden layers are denoted by $S_{1}$ , $S_{2}$ and
$S_{3}$, respectively. The three fully connected hidden layers are used to
learn the statistical characteristics from the signals, where the rectified
linear unit (ReLU) function, i.e., ${f_{\rm ReLU}(\alpha)}$ $=$ ${{\rm
max}(0,\alpha)}$, is used as the activation function. The output layer is a
fully connected layer with $R$ neurons, which utilizes the Sigmoid function,
i.e., ${f_{\textrm{Sigmoid}}(\alpha)}={1/(1+{\textrm{exp}}^{-\alpha})}$, as
the activation function to map the output into the interval (0,1).
Specifically, the value of $R$ is determined by the number of bits that can be
transmitted by the input vector. Thus, we have $R=2R_{\textrm{GOQSM}}$ for
OFDM-based GOQSM system. As a result, the input and output relationship of the
feed-forward DNN module with totally five layers can be expressed by

${\bf{y}}_{i}={\sigma}_{i}({\bf{W}}_{i-1}{\bf{y}}_{i-1}+{\bf{b}}_{i-1}),~{}~{}i=2,\cdots,5,$
(6)

where ${\sigma}_{i}(\cdot)$ is the activation function, and ${\bf{W}}_{i}$ and
${\bf{b}}_{i}$ denote the weight matrix and the bias vector, respectively.

Moreover, the decision module is utilized to generate the final binary bits.
Letting ${\bf{y}}_{5}$ = $[y_{1},y_{2},\cdots,y_{R}]^{T}$ be the input of the
decision module, the estimated bit information $\hat{g}_{q}$
($q=1,2,\cdots,R$) can be expressed as follows:

$\hat{g}_{q}=\begin{cases}0,&y_{q}<0.5,\\\ 1,&y_{q}\geq 0.5.\end{cases}$ (7)

Finally, the mean-squared error (MSE) loss function is applied to measure the
difference between the transmitted bit vector ${\bf g}$ and the estimated bit
vector ${\bf\hat{g}}$, which is given by

$e_{\textrm{MSE}}=\frac{1}{R}{\parallel{{\bf\hat{g}}{\bf-}{\bf
g}}\parallel}^{2}.$ (8)

TABLE I: Simulation Parameters Parameter | Value
---|---
Room dimension | 5 m $\times$ 5 m $\times$ 3 m
LED spacing | 2.5 m
PD spacing | 10 cm
Semi-angle at half power of LED | $60^{\circ}$
Gain of optical filter | 0.9
Refractive index of optical lens | 1.5
Half-angle FOV of optical lens | $72^{\circ}$
Responsivity of the PD | 1 A/W
Height of receiving plane | 0.85 m
Active area of PD | 1 ${\textrm{cm}}^{2}$
Noise PSD |  $10^{-22}\textrm{A}^{2}$/Hz
Modulation bandwidth | 20 MHz
Number of LEDs | 4
Number of activated LEDs | 2
Number of PDs | 4
Receiver location | (2 m, 2 m, 0.85 m)
TABLE II: Parameters of the DNN-Aided Detector Parameter | Value
---|---
Number of input nodes |  $K$=8
Number of hidden layers | 3
Hidden layer activation | ReLU
Output layer activation | Sigmoid
Loss function | MSE
Optimizer | Adam
Learning rate | 0.001
Number of training set | 1524000
Number of validation set | 635000
Number of neurons@3 bits/s/Hz |  $S_{1}$=30, $S_{2}$=36, $S_{3}$=16, $R$=6
Number of neurons@4 bits/s/Hz |  $S_{1}$=30, $S_{2}$=36, $S_{3}$=17, $R$=8
Number of neurons@5 bits/s/Hz |  $S_{1}$=30, $S_{2}$=36, $S_{3}$=18, $R$=10

##  IV Simulation Results

In our simulations, we consider a 4$\times$4 MIMO-VLC system in a typical room
with a dimension of 5 m $\times$ 5 m $\times$ 3 m. The four LEDs are mounted
at the center of the ceiling and the height of the receiving plane is 0.85 m.
Moreover, we compare the performance of OFDM-based GOQSM with OFDM-based GOSM
using ML-MRC detection and DNN-aided detection. For both GOSM and GOQSM, two
out of totally four LEDs are selected to transmit signals, i.e., $N_{t}=4$ and
$N=2$. To achieve a target spectral efficiency under the above simulation
setup, the required QAM constellations for GOQSM and GOSM can be obtained from
(4) and (5), respectively. The other simulation parameters can be found in
Table I. Furthermore, the proposed DNN-aided detection is implemented on the
PyTorch platform. In order to accelerate the training of the neural network,
the mini-batch method is adopted and each mini-batch contains 100 symbol
vectors. The training set and the validation set include $1.524\times 10^{6}$
and $6.35\times 10^{5}$ randomly generated symbol vectors, respectively. The
key parameters of the DNN-aided detector are summarized in Table II.

Figure 3: MSE loss vs. epoch for the DNN-aided detector. Figure 4: BER vs.
transmitted SNR for OFDM-based GOSM and GOQSM with a spectral efficiency of
(a) 3 bits/s/Hz, (b) 4 bits/s/Hz and (c) 5 bits/s/Hz.

###  IV-A MSE Loss

We first analyze the MSE loss of the proposed DNN-aided detector and the MSE
performance for GOSM and GOQSM is depicted in Fig. 3. As we can observe, the
MSE decreases rapidly with the increased number of epochs for both GOSM and
GOQSM with different spectral efficiencies and different training signal-to-
noise ratios (SNRs). Moreover, for a higher spectral efficiency, a larger
training SNR is adopted for both GOSM and GOQSM. It can be clearly seen that
the proposed DNN-aided detector achieves satisfactory performance within
10-epoch training for both GOSM and GOQSM with different spectral efficiencies
and different training SNRs. Hence, the proposed DNN-aided detector can be
implemented very quickly in practical systems.

###  IV-B BER Performance

In the next, we evaluate and compare the BER performance of OFDM-based GOSM
and GOQSM systems with different spectral efficiencies. Figs. 4(a)-(c) show
the BER versus transmitted SNR with spectral efficiencies of 3, 4 and 5
bits/s/Hz, respectively. For a spectral efficiency of 3 bits/s/Hz, as shown in
Fig. 4(a), GOSM performs better than GOQSM in the small SNR region when using
the ML-MRC detection and GOQSM outperforms GOSM when the SNR is larger than
160 dB. More specifically, the required SNRs for GOSM and GOQSM to achieve BER
= $10^{-3}$ are 163.7 and 162.2 dB, respectively, indicating an SNR gain of
1.5 dB by applying GOQSM in comparison to GOSM. Moreover, when applying the
proposed DNN-aided detection, there exists an optimal training SNR for both
GOSM and GOQSM. This is because it is easier for the DNN to learn the
statistics of the noise with a small training SNR and meanwhile the statistics
of the data symbols can be more accurately learned by the DNN with a large
training SNR. Therefore, the optimal training SNR can be seen as the trade-off
for the DNN to learning the statistics of both the noise and the data symbols.
It can be observed from Fig. 4(a) that the optimal training SNRs for GOSM and
GOQSM are 129 and 127 dB, respectively. With the optimal training SNR, GOQSM
using the DNN-aided detection requires an SNR of 133.6 dB to reach BER =
$10^{-3}$. Compared with GOQSM using the ML-MRC detection, a substantial SNR
reduction of 28.6 dB is obtained, which is mainly due to the ability of the
DNN-aided detector to eliminate noise amplification. Furthermore, GOQSM
outperforms GOSM across the whole SNR region and it is mainly due to the
elimination of error propagation by the DNN-aided detector. When applying the
DNN-aided detection with an optimal training SNR, the SNR gain achieved by
GOQSM in comparison to GOSM at BER = $10^{-3}$ is 1.1 dB. For higher spectral
efficiencies of 4 and 5 bits/s/Hz, as shown in Figs. 4(b) and (c), GOQSM
performs worse than GOSM to achieve BER = $10^{-3}$ when using the ML-MRC
detection. However, GOQSM performs much better than GOSM to reach BER =
$10^{-3}$ when applying the DNN-aided detection. Specifically, the SNR gains
are increased from 1.1 dB to 2.3 and 4.1 dB, when the spectral efficiency is
increased from 3 bits/s/Hz to 4 and 5 bits/s/Hz, respectively.

##  V Conclusion

In this paper, we have proposed an OFDM-based GOQSM scheme for MIMO-OWC
systems. By applying QSM instead of SM, the achievable spectral efficiency of
the system can be efficiently improved. In order to address the adverse
effects of error propagation and noise amplification in the OFDM-based GOQSM
system using the two-step ML-MRC detection, we have further proposed a DNN-
aided detection scheme. Our simulation results show that the DNN-aided
detection substantially outperforms the ML-MRC detection for both GOSM and
GOQSM. Furthermore, a significant 4.1-dB SNR gain can be obtained by GOQSM in
comparison to GOSM by using the DNN-aided detection. Therefore, OFDM-based
GOQSM with deep learning-aided detection can be a promising candidate for
high-speed MIMO-OWC systems.

## References

  * [1] T. Cogalan and H. Haas, “Why would 5G need optical wireless communications?” in _Proc. IEEE Ann. Int. Symp. Pers., Indoor Mobile Radio Commun. (PIMRC)_ , Oct. 2017, pp. 1–6.
  * [2] S. Rajagopal, R. D. Roberts, and S.-K. Lim, “IEEE 802.15. 7 visible light communication: modulation schemes and dimming support,” _IEEE Commun. Mag._ , vol. 50, no. 3, Mar. 2012.
  * [3] T. Fath and H. Haas, “Performance comparison of MIMO techniques for optical wireless communications in indoor environments,” _IEEE Trans. Commun._ , vol. 61, no. 2, pp. 733–742, Feb. 2013.
  * [4] R. Mesleh, H. Elgala, and H. Haas, “On the performance of different OFDM based optical wireless communication systems,” _J. Opt. Commun. Netw._ , vol. 3, no. 8, pp. 620–628, Aug. 2011.
  * [5] R. Mesleh, H. Elgala, and H. Haas, “Optical spatial modulation,” _J. Opt. Commun. Netw._ , vol. 3, no. 3, pp. 234–244, Mar. 2011.
  * [6] M. D. Soltani, M. A. Arfaoui, I. Tavakkolnia, A. Ghrayeb, M. Safari, C. M. Assi, M. O. Hasna, and H. Haas, “Bidirectional optical spatial modulation for mobile users: Toward a practical design for LiFi systems,” _IEEE J. Sel. Areas Commun._ , vol. 37, no. 9, pp. 2069–2086, Sep. 2019.
  * [7] F. Wang, F. Yang, and J. Song, “Constellation optimization under the ergodic VLC channel based on generalized spatial modulation,” _Opt. Exp._ , vol. 28, no. 14, pp. 21 202–21 209, Jul. 2020.
  * [8] A. Yesilkaya, R. Bian, I. Tavakkolnia, and H. Haas, “OFDM-based optical spatial modulation,” _IEEE J. Sel. Topics Signal Process._ , vol. 13, no. 6, pp. 1433–1444, Oct. 2019.
  * [9] C. Chen, X. Zhong, S. Fu, X. Jian, M. Liu, X. Deng, and H. Fu, “Enhanced OFDM-based optical spatial modulation,” in _Proc. IEEE IEEE Int. Conf. Commun. (ICC)_ , Jun. 2021, pp. 1–6.
  * [10] C. Chen, X. Zhong, S. Fu, X. Jian, M. Liu, H. Yang, A. Alphones, and H. Y. Fu, “OFDM-based generalized optical MIMO,” 2020. [Online]. Available: https://doi.org/10.36227/techrxiv.13270751.v1
  * [11] R. Mesleh, S. S. Ikki, and H. M. Aggoune, “Quadrature spatial modulation,” _IEEE Trans. Veh. Technol._ , vol. 64, no. 6, pp. 2738–2742, Jun. 2015.
  * [12] T. Komine and M. Nakagawa, “Fundamental analysis for visible-light communication system using LED lights,” _IEEE Trans. Consum. Electron._ , vol. 50, no. 1, pp. 100–107, Feb. 2004.
  * [13] C. Chen, H. Yang, P. Du, W.-D. Zhong, A. Alphones, Y. Yang, and X. Deng, “User-centric MIMO techniques for indoor visible light communication systems,” _IEEE Syst. J._ , vol. 14, no. 3, pp. 3202–3213, Sep. 2020.


# Past and Future Motion Guided Network for
Audio Visual Event Localization

Tingxiu Chen, Jianqin Yin*, Jin Tang This work was supported partly by the
National Natural Science Foundation of China (Grant No. 62173045, 61673192),
and partly by the Fundamental Research Funds for the Central
Universities(Grant No. 2020XD-A04-2).

###### Abstract

In recent years, audio-visual event localization has attracted much attention.
It’s purpose is to detect the segment containing audio-visual events and
recognize the event category from untrimmed videos. Existing methods use
audio-guided visual attention to lead the model pay attention to the spatial
area of the ongoing event, devoting to the correlation between audio and
visual information but ignoring the correlation between audio and spatial
motion. We propose a past and future motion extraction (pf-ME) module to mine
the visual motion from videos ,embedded into the past and future motion guided
network (PFAGN), and motion guided audio attention (MGAA) module to achieve
focusing on the information related to interesting events in audio modality
through the past and future visual motion. We choose AVE as the experimental
verification dataset and the experiments show that our method outperforms the
state-of-the-arts in both supervised and weakly-supervised settings.
Available at: http://tug.ctan.org/info/lshort/english/lshort.pdf

###### Index Terms:

event localization, multi-modal learning, audio-visual learning, audio
attention, motion extraction.

##  I Introduction

Human perceive the world through a variety of senses, such as hearing, vision,
smelling, and touching. Since the sound and vision are two dominant
components, audio visual joint learning has attracted increasing attention in
recent years. Audio visual event localization requires a machine to detect the
event segment in an untrimmed video and recognize the category of the event.
When an event occurs in both auditory and visual modes, it is regarded as an
audio-visual event (as shown in Figure 1). Different from video action
recognition[20], which directly recognizes the category from an untrimmed
video, audio-visual event localization task[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12] requires the computer to divide the video into some fixed length segments,
recognize the category of each segment, and combine the adjacent segments with
the same category as the detection result of localization.

Figure 1: Illustration of audio visual event localization task. Audio visual
events are defined as audible and visible events in the video(e.g.the first
two video segment above is visible but Inaudible, so it’s a background
segment). The purpose of audio-visual event localization is to locate the
segments of audio visual events from the video and identify the categories of
these events through mining the relationship between them while using the
audio-visual modal information, which is very challenging but valuable.

Audio visual event localization methods are required to consider the
correlation between the two modalities. Existing methods rely on visual
modality to provide spatial information and pick the audio motion as the
dynamic information. They also use audio modality to guide the visual modality
to pay attention to the space of interesting objects. However, there are some
problems in these methods: (1)Existing audio visual event localization methods
focus on the visual attention guided by hearing without mining the information
related to interesting events in audio modality[6, 9, 12]; (2) Existing
methods extract dynamic information only from audio modality, which ignores
the motion modeling of visual modality; We propose the past, future motion
guided network (PFMGN) to pay attention to the information related to
interesting events by screening the consistency information between dynamic
vision and hearing.
For selecting the relevant content of interested events in audio modality and
reducing the influence of background noise, we propose motion-guided audio
attention module. The motion in audio is rich, but the key motion content is
not distilled, so the background noise interferes with the event localization.
Visual spatial information is a static expression unable to resonate with the
rich motion of voice. However, visual motion can be used as the guidance to
retain the audio content highly related to the event to be detected, and
reduce the negative impact of background noise to improve the detection
performance. As the visual motion information is important for audio
attention, we propose the motion-guided audio attention module, which exploits
visual motion as the attention guidance of audio modality to focuse on
auditory information highly consistent with visual motion and reduce the
background noise.
However, visual motion is implicitly expressed in the original visual
modality. To express implicit motion information clearly , we propose the pf-
ME (past and future motion excitation) module to exract the visual motion
information. Visual motion information is usually represented by optical
flow[13] in some existing video understanding methods. However, the
computation of optical flow is time-consuming and storage demanding. In recent
years, lightweight motion extraction modules have been proposed to obtain
motion information efficiently while slightly losing accuracy, such as ME[14]
and MS[15]. However, they only focus on the future motion information and
ignore the past one. To address the lack of the past motion in previous motion
extraction module, we propose a new motion extraction module (pf-ME) to ”see”
the motion of the past, future, and current segments at the same time.
To sum up, the contributions of this paper are as follows:
(1) Visual motion information is introduced to mine the information related to
audio event-related content by the motion-guided audio attention module we
proposed. Meanwhile, the background noise is filtered.
(2) A new motion extraction module: pf-ME is proposed to extract the past and
future motion information to enrich visual motion content.
(3) Experiments results demonstrate that our method outperforms the state-of-
the-art in supervised and weakly-supervised tasks on the AVE dataset.
It’s available at:  which provides an overview of working with PFAGN.

Figure 2: The pipeline of the proposed method for audio-visual event
localization. The pf-ME module is proposed to extract past and future visual
motion information as the guidance of audio attention. Then the motion-guided
audion attention module is adopted to mine the event-related audio content
while filtering background noise. Visual modality locate the spatial area
around the vocal object benefit by the guidance of clean audio feature
filtered by motion in the audio-guided video attention module. Finally,
following the previous work, we obtain the relation-aware quality and
integrate audio-visual features through the V-A interaction module for segment
classification.

##  II Related Works

We first review the methods of existing audio-visual event localization, and
then we narrow the scope to the audio guided visual attention audio visual
event localization methods. Finally, two motion extraction modules in the
action recognition task are discussed.
Audio visual event localization. The release of the AVE[12] dataset promotes
the research of audio-visual event localization tasks. Tian Y et al.[12]
proposed to learn event information through LSTM in a sequence to sequence
manner. Hanyu X et al.[6]proposed to learn inter and intra information between
visual and audio modality by adaptive attention and self-attention modal.
Jinxing Z et al.[3] aggregates relevant information that probably not be
available at the same time through the positive sample distribution model. And
they all use the auditory guided visual attention module that we will discuss
below.
Audio-guided visual attention. Some audio-visual joint learning methods use
auditory guided visual attention to lead the model to follow the spatial area
related to hearing. The sound source separation schemes proposed in [18, 19]
show that the voices of different speakers can be distinguished by paying
attention to the location of the spatial region around the speaker’s voice and
finding matching sound source information. In the audio-visual event
localization task, [12] firstly adopted the auditory guided visual attention
mechanism, [3, 6, 9] have followed this attention mechanism.
The attention relies on visual modality to provide static spatial information
and audio modality to provide dynamic information. However, the audio motion
includes both event-related content and background noise. Treating event-
related information indiscriminately with background noise will interfere with
the judgment of event categories and create redundant calculations. We are
committed to exploring ways to make audio modality focus on the event-related
information to cope with the problem above. In addition to static spatial
information, visual modality also contains motion content. Inspired by audio-
guided visual spatial attention, turn around, the audio modality can also
focus on event-related information through the visual motion-guided auditory
attention mechanism, alleviating the interference of background noise.
Motion extraction modules. Visual motion information is not directly but
implicit expressed in the video, so an effective motion information extraction
module must be designed to explicitly model the visual motion information.
Some audio-visual motion recognition methods have been committed to extracting
visual motion through motion extraction modules in recent years. Compared with
optical flow, they significantly reduce the calculation cost in the case of a
small amount of accuracy wastage and provide favorable conditions for the
practical application of the algorithm. [15] proposed a motion squeeze (MS)
extraction module to calculate the similarity between each pixel of the
current frame and the pixel near the pixel of the next frame, using the max
pool to retain the point with the highest similarity as the new position of
the pixel, and fit the motion trajectory of the pixel. [14] through the motion
exception (ME) dynamic extraction module, the difference between two adjacent
frames is used as the source of dynamic visual information. The CNN network is
used to activate the dynamic, sensitive channel.
Inspired by the ME module, we propose a pf-ME motion extraction module, which
integrates the past and future motion of the current segment, activates the
event-related motion in the past or future with CNN network and uses the
visual motion extracted by this module as the guidance of audio attention.

##  III Method

The framework of the method we propose is illustrated in Figure 2.Our
algorithm is described as follows:
1\. First of all ,the architecture of our network is described, then the
audio-visual feature extraction way.
2\. For visual motion extraction, we propose the pf-ME module to obtain the
past and future visual motion as the guidance of audio attention, and the
audio guided visual attention module (MGAA) for audio attention. The audio
features distilled by visual motion are picked from the MGAA module.
3\. Following previous works, the audio features entry the audio-guided video
attention module to follow the spatial area of vocal objects. Then the audio-
visual relation-aware modules are used. The relation-aware feature is mined
following the previous work[9], and we integrate audio-visual features through
the V-A interaction module.
4\. Classifier and loss functions are introduced in turn. Overview of PFMG,
based on CMRA [9] network, we add the auditory attention branch guided by
visual motion from the pf-ME module. The auditory guided spatial channel video
attention module, relation-aware modules, and V-A interaction module are the
same as in CMRA.
Audio and video features extraction.Firstly, an untrimmed video is divided
into $T$ fixed-length segments without overlapping. For the audio modality, we
convert the audio in every segment into spectrums through STFT (short-time
Fourier transform)[25] and map it to the Mel spectrum[26] to leverage the
model’s perception of treble and bass frequency. For the visual modality, we
adopt the first frame of each segment as the visual information of one segment
to reduce redundant computation since the visual contents of adjacent frames
are similar. Then the CNN networks are used to extract informative audio-
visual features $a\in\mathbb{R}^{T\times d_{a}}$, $v\in\mathbb{R}^{T\times
h\times w\times d_{v}}$.
The audio-visual consistency information is extracted through the V-A
attention module, where the attention of one modality is guided by another
modality. In the motion-guided audio attention branch of visual motion
guidance, the pf-ME module we proposed is to extract the dynamic visual
information as the audio attention guidance. What’s more, following the
previous work, two relation aware modules are adopted to mine the intra and
inter modalities’ information through the cross-modality relation aware
module. Then V-A interaction module integrates audio-visual features through
the cross-modality relation aware module’s variant to integrate features of
the two modalities and further explore the audio-visual interaction
information. Finally, the characteristics of fusion are used to obtain the
localization results.
Pf-ME visual motion extraction module. The visual feature extracted by the
2D-CNN of each segment represents the static visual information without mining
dynamic visual change among these segments. Pf-ME module is proposed to
extract visual motion features as audio modality’s attention guidance to
locate consistent audio-visual motion. The structure of the pf-ME module is
described in detail, as shown in Figure 3. Firstly, the visual features
$v:v_{1},...v_{i},...v_{T}$ of $T$ segments enter the 2D-CNN to adjust the
number of visual channels to be consistent with the audio channels.
$v^{\prime}$ is a channel-adjusted result. The calculation formula is as
follows:

$v^{\prime}=\text{conv}_{1}*{v},\quad{v^{\prime}}\in\mathbb{R}^{T\times
N\times d_{a}}$ (1)

$v_{i}\in\mathbb{R}^{N\times d_{a}}$ is the first frame feature in the
$i_{th}$ segment, $N=h\times w$. Changes from $v_{i}$ through the 2D-CNN to
$v_{i-1}$ are regarded as the ‘past’ motion $M_{pi}$. Initial $M_{p1}=0$.
Similarly, we get $M_{fi}$ as ‘future’ motion, and last segment’s motion :
$M_{fT}=0$. Pf-ME is inspired by ME[14], and 2D-CNNs are adopted to reduce the
motion trajectory mismatch caused by pixel offset like ME. The calculation
formula is as follows:

$M_{pi}=\text{conv}_{p}*{v_{i}}-{v_{i-1}}\\\ $ (2)

$M_{fi}=\text{conv}_{f}*{v_{i+1}}-{v_{i}}$ (3)

The ”past” motion $M_{P}$ and the ”future” motion $M_{F}$ are fused as $M_{S}$
to ”see” motion changes of three segments. Then and avg-pooling in spatial
dimension is adopted to obtain $M’$ to integrate the spatial information.
Finally, the channel dimension of $M’$ is adjusted by 2D-CNN. The calculation
formula is as follows:

$\displaystyle M_{s}$ $\displaystyle=M_{f}+M_{p},\quad
M_{s}\in\mathbb{R}^{T\times h\times w\times d_{a}}$ (4)

$M^{\prime}=\text{Avgpool}(M_{s}),\quad{M^{\prime}}\in\mathbb{R}^{T\times
d_{a}}$ (5)

$M=\text{conv}_{2}*{M^{\prime}},\quad{M}\in\mathbb{R}^{T\times d_{a}}$ (6)

Pf-ME displays certain advantages in motion extracting. Firstly, compared with
ME[14], MS[15], and other lightweight motion capture modules[27, 28], the pf-
ME module focuses on the dynamic changes of the first frame of adjacent
segments instead of adjacent frames since adjacent segments have more obvious
visual changes to reflect the event category. Secondly, the purpose of the pf-
ME module is to simulate the ”past” and ”future” motion trajectories through a
convolution kernel to activate the feature channels related to dynamic
information in visual modal features. It simulates that we pay less attention
to the static background than to the moving object when watching a video. The
moving object is continuously determining the judgment of the event category
in the video. the pf-ME module simulates the capture of moving objects as the
human watching video by modeling the changes of motion trajectories in the
”past” and ”future.”

Figure 3: Illustration of the pf-ME module. pf-ME leverages CNNs to extract
past and future visual motion as audio modality attention guidance.

V-A attention model. Audio-visual features are adopted as another modality’s
attention guidance in the V-A attention module to activate modality
correlation information. Firstly, as shown in figure 4, we add a motion-guided
audio attention (MGAA) branch with a temporal-wise attention module and a
channel activation module to screen the information highly related to both
audio and visual motion. After obtaining a new auditory feature $a^{\prime}$
from the motion-guided audio attention module, the module attracts more
attention to audio data related to the events category while reducing the
impact of background noise. The calculation formulas in temporal attention
module are as follows.

$M_{ta}=\text{softmax}(\text{W}_{ta}M),\quad M_{ta}\in\mathbb{R}^{T\times
1}\\\ $ (7)

$a_{1}=a\oplus(M_{ta}\odot{a})$ (8)

And the calculation formulas in channel attention module are as follows:

$M_{ca}=\text{sigmoid}(M),\quad M_{ca}\in\mathbb{R}^{T\times d_{a}}$ (9)

$a^{\prime}=a_{1}\oplus(M_{ca}\odot a_{1})$ (10)

In the existing methods, audio-guided visual attention is used to locate the
spatial area of audible objects, only modelling the relation between audio and
static vision. However, audio contains rich dynamic information, and the
relationship between audio and dynamic vision needs to be fully explored.
Temporal attention module in motion-guided audio attention module thar we
proposed can distinguish whether a segment contains an action or not: segments
with strong motion in both dynamic vision and audio are segments in which
actions occur, while segments with weak visual motion are most likely to be
segments presenting background noise even though the audio motion is strong.
In the channel attention module, audio and dynamic visual resonance
information is enhanced to reduce the influence of background noise and
facilitate the classification of action categories.
Then $a^{\prime}$ is regarded as static visual guidance to pay attention to
the event-related spatial area. Continuing the practice in the CMRAN , the
original static visual feature v enters the channel attention module to obtain
the static visual $v_{c^{\prime}}$ that activates the event-related channel.
The calculation formula is as follows:

$v_{c}=\text{ReLU}(\text{W}_{c1}{a^{\prime}})\odot\text{ReLU}(\text{W}_{c2}{v})$
(11)

$v^{\prime}_{c}=\text{avg}({v_{c}})\odot\text{W}_{c3}{v}$ (12)

Then the spatial attention module obtains the static visual feature
$v_{cs}^{\prime}$ that focuses on the spatial features of area related to
audio modality. The calculation formula is as follows:

$v_{cs}=\text{ReLU}(\text{W}_{s1}{a^{\prime}})\odot\text{ReLU}(\text{W}_{s2}v^{\prime}_{c})$
(13)

$v^{\prime}_{cs}=\text{tanh}({v_{cs}})\text{W}_{s3}\otimes{v^{\prime}_{c}},\quad{v^{\prime}_{cs}}\in\mathbb{R}^{T\times
d_{v}}$ (14)

Due to the similar spatial characteristics of adjacent frames, we select the
first frame image of video segments as static visual information to represent
spatial content against the visual motion. The audio spectrogram is adopted as
the content of the audio modality carrying rich audio motion without static
audio since there is no spatial structure in audio. However, the audio
modality only interacts with static visual information, ignoring previous
methods’ visual motion modeling and interaction between audio and visual
motion. In contrast, the audio guided by visual motion can retain the info
highly related to audio-visual events. As the guidance of static vision, it is
the joint guidance of dynamic sight and dynamic hearing. We model the visual
motion information to interact with the audio and guide the network to pay
attention to the information related to the event category through the V-A
attention module.

Figure 4: Illustration of the proposed motion-guided audio attention module.
The green bar represents the temporal-wise attention module, the purple bar
represents the channel-wise attention module, and the corresponding module
details are in the upper right corner. Under the guidance of visual motion,
the event-related information of audio modality is concerned, and the
background noise is filtered out.

RA(relation aware) module. After obtaining $a^{\prime}$and $v_{cs}^{\prime}$,
continue the practice of CMRAN, we input audio-visual features into two-way RA
(relationship aware) modules, respectively. The RA(relation aware) modules
learn the correlation information within and with the help of the multimodal
attention module between modalities. The calculation formula of the CMRA
multimodal attention module is as follows:

$\text{CMRA}(x,y)=\text{softmax}(\frac{q_{1}(K_{1,2})^{T}}{d_{m}})X_{1,2}$
(15)

$q_{1}=x\text{W}^{Q},K_{1,2}=g_{x,y}\text{W}^{K},X_{1,2}=g_{x,y}\text{W}^{X}$
(16)

$g_{xy}=\text{Concat}(x,y)$ (17)

The calculation formula of visual mode branch RA module is as follows:

$V_{R}=\text{CMRA}(V_{cs},A^{\prime})$ (18)

The calculation formula of audio relation aware is as follows:

$A_{R}=\text{CMRA}(A^{\prime},V_{cs})$ (19)

Then, the audio-visual interaction is further carried out through the relation
aware module. The audio-visual feature splicing result $O_{av}$ and point
multiplication result $f_{av}$ are used as new input to the CMRA model. The
resonance channel between vision and acoustics is activated by combining
audio-visual features to obtain feature O. Finally, Of is obtained from $O$
and $Oav$ as the classification feature.

$F_{av}=A_{R}\odot V_{R}$ (20)

$O_{av}=\text{Concat}(A_{R},V_{R})$ (21)

$O=\text{CMRA}(F_{av},O_{av})$ (22)

$\displaystyle O_{f}$ $\displaystyle=O\oplus O_{av}$ (23)

Classifier and loss. $O_{av}\in\mathbb{R}^{T\times D_{v}}$ is the final
feature for classification. In the supervised method, the segment level labels
are provided, and in the weakly-supervised method, only the category of the
whole video is provided. FC is used as a classifier to locate the event
segments and predict the event categories. The audio-visual correlation score
Se represents the audio-visual correlation degree of each segment, and Sc
represents the video level category obtained through the classifier. As a
complete video classification feature, $O_{av}\in\mathbb{R}^{1\times D_{v}}$is
obtained by Max-pooling from $O_{av}$ in the time dimension. The calculation
formula is as follows:

$O_{av}=\text{Maxpool}(O_{av})\quad,{O_{av}}\in\mathbb{R}^{1\times d_{v}}$
(24)

$S_{e}$ represents the audio-visual correlation degree score of a video
segment. When the $S_{e}$ of a video segment is greater than 0.5, the segment
is an audio-visual event occurrence segment. Since most videos contain only
one event category, the category of the complete video is considered as the
category for all audio-visual related segments. When the score $S_{e}$ is less
than 0.5, the fragment is considered as a background fragment. $S_{c}$
represents the category results for the complete video. Calculated as follows:

$S_{c}=\text{Softmax}(O_{av}W_{c})$ (25)

$S_{e}=\text{sigmoid}(O_{av}W_{e})$ (26)

The complete video category detection results are constrained by multi-
classification CE (cross-entropy) loss, and the audio-visual correlation
detection results of fixed length segments are constrained by binary
classification CE loss. The final loss is the sum of the two. The loss is
calculated as follows:

$L_{c}=-\sum_{k=1}^{C}y_{i}\text{log}(S_{ci})$ (27)

$L_{e}=-y\text{log}(S_{e})-(1-y)\text{log}(1-S_{e})$ (28)

$L=L_{c}+L_{e}$ (29)

Where $y_{i}$ is the label and $p_{i}$ is the prediction result.In the
supervised task, segment level labels are provided in training, including
audio-visual correlation labels of each segment and category labels of each
segment. In weakly-supervised training, only the category of the complete
video is provided, and the supervision signals are mined from the data (such
as whether audio and video exist at the same time). We adapt the label by
adding the prediction results of each segment in time dimension element-wise.

##  IV Experiment

Dataset. We perform extensive experiments on the AVE dataset. The AVE
dataset[12] contains 4143 videos on YouTube, 66.4% of which are 10s long, and
the rest are no less than 2s long, covering 28 event categories. It involves
various audio-visual events such as human activities, animal activities, music
performance, and vehicle sound. It provides a dataset with a wide range and
samples full of time inconsistency and mutation. The data scene is rich and
close to the actual event scene. AVE is the most commonly used public dataset
for audio-visual event localization. Consistent with the practice of [1, 2, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12], we use accuracy to measure the performance of
event localization.
Implementation Details. We adopt 2D-CNN networks that pre-trained on large
datasets to extract audio and visual features of video segments. We adopt
VGG-19 pre-trained on ImageNet[24] to extract the visual feature. The
extracted video features are $v\in\mathbb{R}^{T\times h\times w\times d_{v}}$,
$T$ is the number of fixed- length video segments segmented from untrimmed
videos. A video is divided into some fixed-length segments and takes the first
frame of each segment as visual information. Therefore, one video obtains $T$
frames, and the extracted feature is $T$ length, $h$ and $w$ are the length
and width of each segment’s first frame feature map, and $d_{v}$ is the visual
channel dimension. As far as the audio feature, we use VGG-like feature
extraction network pre-trained on AudioSet dataset. The extracted feature
$a\in\mathbb{R}^{T\times d_{a}}$ is the spectrogram feature of audio in every
segment, and $d_{a}$ is the audio feature channel dimension. The verification
experiment is based on PyTorch, batch_size = 32, and learning rate $l$ =
$5\times 10^{-4}$.

TABLE I: Comparisons with state-of-the-arts in supervised task on AVE dataset (* indicates the reproduced performance) Method | Audio Feature | Video Feature | Accuracy(% )
---|---|---|---
AVEL[12] | VGG-19 | VGG-like | 72.7
AVIN[11] | VGG-19 | VGG-like | 75.2
AVSDN[10] | VGG-19 | VGG-like | 75.4
CMRAN*[9] | VGG-19 | VGG-like | 76.8
CMRAN[9] | VGG-19 | VGG-like | 77.4
PSP[3] | VGG-19 | VGG-like | 77.8
Ours | VGG-19 | VGG-like | 78.5
TABLE II: Comparisons with state-of-the-arts in weakly-supervised task on AVE dataset (* indicates the reproduced performance) Method | Audio Feature | Video Feature | Accuracy(% )
---|---|---|---
AVEL[12] | VGG-19 | VGG-like | 66.7
AVIN[11] | VGG-19 | VGG-like | 69.4
AVSDN[10] | VGG-19 | VGG-like | 74.2
CMRAN*[9] | VGG-19 | VGG-like | 72.5
CMRAN[9] | VGG-19 | VGG-like | 72.9
PSP[3] | VGG-19 | VGG-like | 73.5
Ours | VGG-19 | VGG-like | 73.5
Figure 5: Qualitative examples of different attention methods. The top row,
middle row and bottom row show the visualization examples of CMRAN and our
method respectively. As can be seen from the illustration, our PFMG network is
more focused than AVEL in locating audible objects and more accurate than
CMRAN. This is because hearing, after being screened by visual motion, retains
information highly relevant to audio-visual events, thus better directing the
visual attention area.

Comparisons with state-of-the-arts. We compare our PFAG network with state-of-
the-art methods on the AVE dataset in supervised and weakly-supervised AVE
localization(as shown in Table 1. For a fair comparison, we compare the
localization results of the PFAG network with previous methods with the same
audio-visual features. For a fair comparison, we relive CMRAN on the same
platform, we get 1.7%  accuracy higher in the supervised settings, and 1%
higher in the weakly-supervised settings, which proves the effectiveness of
introducing visual motion information and the visual motion extraction ability
pf-ME module.
In present methods, the audio guided spatial region attention mechanism has a
good performance in the audio-visual event localization task. It’s benefit
from the focus on the event-related spatial regions and the reduction of
attention to the background spatial information under the guidance of hearing.
However, they ignore the distillation of audio information. Auditory signals
have rich motion information, but contain background noise that interferes
with event localization. The motion information in visual modality is able to
be the basis for the screening of audio modality. it is that most of the
features with strong audio-visual dynamic are event-related features. If the
audio features are dynamic but the corresponding visual features are
relatively static, these audio features are most likely to point to the
background noise content with interference. And the visual and audio features
of rich dynamic information represent the key information related to the
event.
We propose the PF-ME module to mine the visual motion from original videos
that implicitly expresses the motion, so that it can be expressed explicitly.
Experiments show the effectiveness of PF-ME module in motion extraction. Then,
the extracted visual motion is used as auditory guidance to filter key
information related to the event in audio modality and filter the background
noise, so as to improve the event localization performance of the proposed
method. Experiments demonstrate the positive effects of introducing visual
motion.
Ablation Study. We split the proposed framework to verify the validity of the
different modules. Firstly, We discuss the utility of visual motion, then the
function of past visual motion information in the pf-ME module. VGG-19 and
VGG-like are also used as audio-visual feature extractors, the same as
existing methods.
The utility of visual motion. In order to prove the effectiveness of visual
motion to the network, we compare RNN and ME modules to extract dynamic visual
information, which is adopted as the attention guidance information of audio
to pay attention to content related to the event (as shown in figure 2).
Compared with the visual modality, the auditory mode does not have a spatial
structure. Still, it contains rich, dynamic information, which can make up for
the lack of dynamic modeling of visual mode and complement the rich spatial
information.

TABLE III: Ablation study on supervised task on AVE dataset (* indicates the reproduced performance) Method | Accuracy(% )
---|---
CMRAN*[9] | 76.841
+RNN(w/o temporal attention) | 76.692
+motion from the ME (w/o temporal attention) | 77.313
+motion from the pf-ME (w/o temporal attention) | 77.488
+motion from the pf-ME (w temporal attention) | 78.458
TABLE IV: Ablation study on weakly-supervised task on AVE dataset (* indicates the reproduced performance) Method | Accuracy(% )
---|---
CMRAN*[9] | 72.488
+RNN(w/o temporal attention) | 72.355
+motion from the ME (w/o temporal attention) | 72.910
+motion from the pf-ME (w/o temporal attention) | 73.532
+motion from the pf-ME (w temporal attention) | 73.159

However, part of the rich motion of the audio modality is background noise
(such as background music, street noise, wind, and rain, etc.), but the audio
motion highly related to visual information is the critical information of
audio-visual event localization. We are committed to retaining the auditory
information related to events and filtering the background noise in the audio
modality. The feature of high correlation between audio and visual motion is
highly associated with audio-visual events, respectively. Experiments show
that visual motion extracted by the ME module to guide the auditory modal
attention model improves the accuracy of supervised localization by 0.48%  and
weakly-supervised localization by 0.42% , which verifies the effectiveness of
introducing the visual motion.
The screening of visual motion information guides audio pays attention to the
characteristics highly related to the event and suppresses the background
sound characteristics irrelevant to the event (such as street noise, wind, and
rain, background music). Then input the audio feature $a^{\prime}$ purified by
dynamic visual attention to guide the concentration of static vision $v$ to
obtain a new visual feature $v^{\prime}$, which drives the network to pay more
attention to the spatial area of the vocal source. The purification of audio
features avoids some attention to the spatial features of background noise to
better focus on the spatial content related to the concerned events.
The utility of extracting ”past” visual motion. Based on verifying the
effectiveness of visual motion, we make an in-depth study on how to extract
effective visual motion. Results of the ME module extracting visual motion
(see Table 2 ”+ motion (me)” results) show that the addition of visual motion
is beneficial to the localization of audio-visual events. However, the ME
module only captures the future visual motion information.
Simulate the process of human eyes watching a video to obtain information.
People often understand the ”antecedents” and ”consequences” of events through
a dynamic video in the past and the future and judge the events in the current
video clip. Therefore, this paper proposes the pf-ME module with past and
future visual motion to capture the complete event process and reduce
ambiguity through the current video segment and the visual-spatial changes in
the previous and subsequent periods. Results show that compared with the ME
module, which only extracts future dynamic information, the pf-ME module
obtains past and future dynamic information at the same time, which improves
the accuracy by 0.18%  in supervised localization and 0.2%  in weakly-
supervised localization, which proves the effectiveness of extracting the past
information.

##  V Conclusion

In this paper, we explore the efficient audio-visual event localization method
through motion guided audio attention mechanism. Based on the auditory
information guidance network paying attention to the spatial area of vocals,
the visual motion is introduced to train the network to pay attention to the
highly relevant features of events in the audio modality. A pf-ME module for
extracting visual motion is proposed to make the network ”see” the past and
future motion as the guidance of audio modality attention. Conclusion and
persuasive experimental results show that our method achieves SOTA accuracy,
which proves the effectiveness of our approach. In the future, audio-visual
joint learning methods can adapt multi-channel audio information collected by
multi microphones and combine the position information of sound receiving
microphones as spatial information of audio modality. And visual-spatial
information to more accurately locate the vocal object and improve the
accuracy of event localization.

## References

  * [1] H. Xuan, L. Luo, Z. Zhang, J. Yang and Y. Yan, “DMMAN: a two-stage audio-visual fusion framework for sound separation and event localization,” in Neural Networks, 2021, vol.133, pp. 229–239.
  * [2] H. Xuan, L. Luo, Z. Zhang, J. Yang and Y. Yan, “Discriminative cross-modality attention network for temporal inconsistent audio-visual event localization,” in IEEE Transactions on Image Process, vol.30, pp. 7878–7888, 2021.
  * [3] J. Zhou, L. Zheng, Y. Zhong, S. Hao and M. Wang, “Positive sample propagation along the audio-visual event line,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2021, pp. 8436–8444.
  * [4] J. B. Li, K. Ma, S. Qu, P. Huang, F. Metze, “Audio-visual event recognition through the lens of adversary,” in Proceedings of the IEEE conference on international conference on acoustics, speech and signal processing, 2021, pp. 616–620.
  * [5] B. Duan, H. Tang, W. Wang, Z. Zong, G. Yang, Y. Yan, “Audio-visual event localization via recursive fusion by joint co-attention,” in Proceedings of the IEEE workshop on applications of computer vision, 2021, pp. 4012–4021.
  * [6] H. Xuan, Z. Zhang, S. Chen, J. Yang and Y. Yan, “Cross-modal attention network for temporal inconsistent audio-visual event localization,” in Proceedings of the AAAI conference on artificial intelligence, 2020, pp. 279–286.
  * [7] Yan-Bo Lin, Yu-Chiang Frank Wang, “Audiovisual transformer with instance attention for audio-visual event localization,” in Proceedings of the asian conference on computer vision, Springer, 2020, pp. 274–290.
  * [8] J. Ramaswamy, “What Makes the Sound?: A dual-modality interacting network for audio-visual event localization,” in Proceedings of the IEEE conference on international conference on acoustics, speech and signal processing, 2020, pp. 4372–4376.
  * [9] H. Xu, R. Zeng, Q. Wu, M. Tan, C. Gan, “Cross-modal relation-aware networks for audio-visual event localization,” in Proceedings of the ACM Multimedia, 2020, pp. 3893–3901.
  * [10] Yan-Bo Lin, Yu-Jhe Li, Yu-Chiang Frank Wang, “Dual-modality seq2seq network for audio-visual event localization,” in Proceedings of the IEEE conference on international conference on acoustics, speech and signal processing, 2019, pp. 2002–2006.
  * [11] Y. Wu, L. Zhu, Y. Yan and Y. Yang, “Dual attention matching for audio-visual event localization,” in Proceedings of the IEEE conference on international conference on computer vision, 2019, pp. 6291–6299.
  * [12] Y. Tian, J. Shi, B. Li, Z. Duan and C. Xu, “Audio-visual event localization in unconstrained videos,” in Proceedings of the europeon conference on computer vision, Springer, 2018, pp. 252–268.
  * [13] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in in Proceedings of the 27th International Conference on Neural Information Processing Systems, 2014, pp. 568–576.
  * [14] Y. Lin, Y. Li and Y. F. Wang, “TEA: temporal excitation and aggregation for action recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2020, pp. 906–915.
  * [15] H. Kwon, M. Kim, S. Kwak, M. Cho, “MotionSqueeze: neural motion feature learning for video understanding,” in Proceedings of the europeon conference on computer vision, Springer, 2020, pp. 345–362.
  * [16] Y. Cheng, R. Wang, Z. Pan, R. Feng and Y. Zhang, “Look, listen, and attend: co-attention network for self-supervised audio-visual representation learning,” in Proceedings of the ACM Multimedia, 2020, pp. 3884–3892.
  * [17] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman and M. Rubinstein, “Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation,” in ACM Trans. Graph, 2019, vol.37, no.4, pp. 112:1–112:11.
  * [18] C. Gan, D. Huang, H. Zhao, J. B. Tenenbaum and A. Torralba, “Music gesture for visual sound separation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2020, pp. 10475–10484.
  * [19] P. Pareek and A. Thakkar, “A survey on video-based human action recognition: recent updates, datasets, challenges, and applications,” in Artif. Intell. Rev, 2021, vol.54, no.3, pp. 2259–2322.
  * [20] S. Adavanne, P. Pertilä, T. Virtanen, “Sound event detection using spatial features and convolutional recurrent neural network,” in Proceedings of the IEEE conference on international conference on acoustics, speech and signal processing, 2017, pp. 771–775.
  * [21] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, M. D. Plumbley, “An improved event-independent network for polyphonic sound event localization and detection,” in Proceedings of the IEEE conference on international conference on acoustics, speech and signal processing, in , 2021, pp. 885–889.
  * [22] T. Afouras, A. Owens, J. S. Chung and A. Zisserman, “Self-supervised learning of audio-visual objects from video,” in Proceedings of the europeon conference on computer vision, Springer, 2020, pp. 208–224.
  * [23] Gao R, Oh T, Grauman K, Torresani L, “Listen to look: action recognition by previewing audio,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2020, pp. 10454-10464.
  * [24] Jia D, Wei D, Richard S, Li-Jia L, Kai L, Li F, “ImageNet: A large-scale hierarchical image database,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2009, pp. 248-255.
  * [25] Kootsookos P.J, Lovell B.C, Boashash B, “A unified approach to the STFT, TFDs, and instantaneous frequency,”in IEEE Transactions on Signal Processing , 2002, vol.40, no.8, pp. 1971-1982.
  * [26] S. Molau, M. Pitz, R. Schlüter, H. Ney,“Computing Mel-frequency cepstral coefficients on the power spectru”,’in Proceedings of the IEEE conference on international conference on acoustics, speech and signal processing 2001: 73-76
  * [27] Y. Zhang, L. Niu, Z. Pan, M. Luo, J. Zhang, D. Cheng, L. Zhang, “Exploiting Motion Information from Unlabeled Videos for Static Image Action Recognition, ” in Proceedings of the AAAI conference on artificial intelligence, 2020, pp. 12918-12925
  * [28] S. Sun, Z. Kuang, L. Sheng, W. Ouyang, W. Zhang, “Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1390-1399

|  Tingxiu Chen Master at Beijing University of Posts and Telecommunications.
Her research interests include computer vision, deep learning and machine
learning.Email<EMAIL_ADDRESS>
---|---
|  Jianqin Yin received the Ph.D. degree from Shandong University, Jinan,
China, in 2013. She currently is a Professor with Artificial Intelligence
School, Beijing University of Posts and Telecommunications, Beijing, China.
Her research interests include service robot, pattern recognition, machine
learning and image processing. Email<EMAIL_ADDRESS>
---|---
|  Jin Tang received the Ph.D. degree from Beijing Institute of Technology,
Beijing, China, in 2007. currently is a Assistant Professor with Artificial
Intelligence School, Beijing University of Posts and Telecommunications,
Beijing, China. Her research interests include signal processing, pattern
recognition, and deep learning. Email<EMAIL_ADDRESS>
---|---