# Formation of bound states in the continuum in double trapezoidal grating

Yuhang Ruan,1 Jicheng Wang,1,2 YIpan Lou, Zheng-Da Hu,1 and Yixiang Wang1
<EMAIL_ADDRESS>1School of Science, Jiangsu Provincial Research
Center of Light Industrial Optoelectronic Engineering and Technology, Jiangnan
University, Wuxi 214122, China
2State Key Laboratory of Millimeter Waves, Southeast University, Nanjing,
210096, China

###### Abstract

In the field of optics, bound state in the continuum (BIC) has been researched
in many photonic crystals and periodic structures due to a strong resonance
and an ultrahigh Q factor. Some designs of narrowband transmission filters,
lasers, and sensors were proposed based on excellent optical properties of
BIC. In this paper, we consider symmetrical rectangular grating structure
firstly, then cut off the corner of one of the gratings, the Fano peak of
quasi-BIC can be observed in the spectrum. After that, we further change the
tilt parameter of the other grating, which minimizes the Fano line width. In
the momentum space, the process of structural change corresponds to
topological charges split from $\textit{q}=1$ into two half charges
$\textit{q}=1/2$. We analyze guided mode resonance (GMR) excitation of the
grating structure, and discuss the dispersion relations in the waveguide layer
with the position of BIC in energy bands. In addition, the reflectance
spectrum is found to exhibit asymmetric line-shapes with different values of
the asymmetry parameters, $M_{1}$ and $M_{2}$. BIC is transformed into quasi-
BIC as the symmetry of the structure is broken. Thus, a large Goos–Hänchen
shift can be achieved as a result of asymmetrical Fano line shape in quasi-
BIC. This work demonstrates a double trapezoid structure with strong resonance
properties, which has significant implications for exploring the phenomenon of
BIC.

††preprint: APS/123-QED

##  I Introduction

Bound state in the continuum (BIC) as an optical phenomenon to limit the loss
of light, the phenomenon of which was originated from quantum physics and
described as a relationship with a continuous medium threshold. BIC was first
proposed by von Neumann and Wigner in 1929 [1], but it’s not demonstrated in
quantum systems. It was not until 1992 that Capasso’s group proposed a BIC
experiment in quantum systems [2]. Then, many specific structures were used to
support BIC, caused a leaky mode with a resonance state of zero width [3].
Until now, there has been a great deal of researches on resonant states, which
lie in the continuum with a finite lifetime, i.e., they are born from
continuum states and decay eventually. Regulating a structure with specific
parameters, resonance states interfered by some different channels yield zero-
width resonance, corresponding to BIC without energy loss.

Recently, researches proposed symmetrical structures to discuss BICs in
photonic crystals or metasurfaces. Grating as a high performance structure
have been researched in many fields, and it is suitable as a highly
symmetrical structure for the study of BIC. Doeleman et al. stdudied grating’s
polarization vortex of BIC in 2018 [4]. In the same year, Azzam et al.
proposed a metal grating and discussed hybrid plasmonic-photonic systems [5].
In 2019, Wu et al. discussed BIC by tuning the distance of gap in grating [6].
Lots of applications with the optical properties of BIC, such as optical
modulators [7], optical vortices [8-10], nonlinear harmonic generations[11-13]
and nonlinear optical devices[14], sensors[15-19] have been designed. For
these previous work about gratings, they proposed some simple structures, or
they only focused on spectrum of structures without energy bands and
topological charges, and they rarely discuss the energy distribution in the
momentum space.

In this work, we propose a double trapezoid grating as an asymmetric structure
to discuss the relationship between BIC and quasi-BIC. For a normal grating,
it can store energy in the symmetrical structure without any loss. When we cut
off an oblique angle of the grating to achieve symmetry breaking, the energy
in the structure will leak into free space. The process corresponds to BIC
turning into quasi-BIC, combined with the Q factor becomeing finite. A narrow
resonance peak appears due to the interference of two channels, and the
distribution of electric field will produce a little offset, energies have
some loss due to the interference. In the momentum space, topological charges
split and couple with the change of grating. Based on guided mode resonance
(GMR), the dispersion relations of grating can be theoretically derived, and
we can find the position of BIC in energy bands. We focus on changing the
inclination angle of the grating, and utilize finite element method (FEM) to
calculate reflection spectrum, from which we find a Fano peak. Our structure
can produce a large phase change at the position of Fano peak, which can be
used to achieve a large Goos-Hänchen (GH) shift. Then we proposed a high
sensitivity sensor based on double trapezoidal grating, it’s caused by the
change of refractive index of surrounding medium. Comparing to other works
about sensors, the sensitivity of double trapezoidal grating in our work can
reach to 349.885 $\mu$m/RIU. Our work provides many potential applications,
such as optical switches, high-performance sensors and wavelength division
multiplexers.

Figure 1:  (a) Three-dimensional schematic diagram of the grating structure.
(b)-(c) Two trapezoidal grating are controlled by asymmetric geometric
parameters $M_{1}$ and $M_{2}$. (d) Schematic diagram of the wave propagation
structure satisfying the GMR condition, where $k_{i}$ and $\beta$ are the
tangential component of the wave vector.

##  II BIC in the trapezoid grating and the analysis of dispersion relations

We firstly discuss rectangular grating in a periodic photonic crystal slab as
shown in Fig. 1(a).The structure maintains a high symmetry with BIC. Then, we
cut off one corner of the rectangular grating to break the symmetry shown in
Fig. 1(b), where the quasi-BIC can be found. When another rectangle is also
damaged in Fig. 1(c), a narrow Fano Peak can be observed with limitation of
light field mode. The unite cell of grating can be divide into three parts,
grating layer, waveguide layer and substrate layer. The grating layer is
chosen to be Si and the waveguide layer is $SiO_{2}$. The refractive index of
Si and $SiO_{2}$ are 3.48 and 1.46, the period length of the whole unite
grating D=1544nm and the thickness of waveguide layer is $h_{1}=2000nm$. For
the grating layer, the air gap splits into two parts $d_{c}=d_{a}=0.23(1-x)D$.
Here, the rectangular grating has a fixed parameter $d_{b}=0.27D$ and the
height of grating layer is $h_{2}=500nm$. Extracting the trapezoidal grating
separately as shown in Figs. 1(b) and (c), we define parameters $M_{1}$ and
$M_{2}$ to control tilt angle, the rest are controlled by N1 and N2, which
satisfy $M_{1}+N_{1}=1(M_{2}+N_{2}=1)$.

Based on guided mode resonance (GMR) excitation of the grating waveguide
layer, we can give a schematic representation of the dispersion relation. The
propagation diagram can be seen in Fig. 1(d). In the system, tangential
component of wave vector is defined as ki, angular frequency is $\omega$ and c
is light speed in the air. For the background of air, we have
$k_{i}=k_{0}sin\theta$ with the wave vector in air $k_{\theta}=\omega/c$.

Figure 2:  (a) The dispersion relation of waveguide layer represents gray
dotted line, the blue solid line and the green realization represent the
components of the wave vector in the waveguide layer, where the inset shows an
enlarged view of the intersection. (b) In the first Brillouin zone, the range
of $ka/2\pi$ is from -0.1 to 0.1, and the point of BIC is found in energy
bands.

When the light beam in grating, a portion of the light passes through the
grating and the other part is trapped in the waveguide layer. When the
resonance effect is generated with leakage disappeared, there is no light
transmitted and a sharp resonance peak can be observed. In the grating, we
define $k_{i}=k_{0}-i\textperiodcentered 2\pi/D$, $k_{0}=2\pi/\lambda$, where
$i=\pm 1,\pm 2\dots$, and $2\pi/D$ is the basic vector of the reciprocal
lattice. Since propagating constant $\beta$ equals $k_{i}$ in the guide mode,
then the expression for GMR can is expressed as

$\displaystyle k_{i}=k_{0}sin\theta-i\frac{2\pi}{D}=\beta$ (1)

Then the difference of the modal propagation coefficient $\beta_{x}$ can be
written as [15]

$\displaystyle\Delta\beta_{x}=\frac{2\pi n_{wg}}{\lambda}\Delta(sin\theta)$
(2)

where the difference of $\theta$ is given by

$\displaystyle\Delta(sin\theta)=\frac{\Delta\lambda}{n_{wg}D}$ (3)

Substituting Eq. (3) into Eq. (2), based on the Lorentzian model with the
assumption, the resonance condition is explained as a pole in the complex
domain [20], i.e., $\Delta\beta_{x}=\gamma$, where $\gamma$ is the coupling
loss coefficient. Then we can derive

$\Delta\lambda=\frac{\lambda D\gamma}{2\pi}$ (4)

Finally, the full width at half-maximum (FWHM) is calculated by

$\displaystyle\Delta\lambda_{FWHM}=\frac{\lambda D\gamma}{\pi}$ (5)

We can initially calculate the half-height width of the resonance peak. For
the slab waveguide theory, $\beta$ is calculated based on the eigenvalue
equation

$\nu_{1}h_{1}=i\pi+tan^{-1}(m_{0}/\nu_{1})+tan^{-1}(m_{2}\nu_{1}),\quad
i=0,1,2\dots$ (6)

where the abbreviations

$\displaystyle
m_{2}=\alpha_{2}tanh[tanh^{-1}(m_{3}/\alpha_{2})+\alpha_{2}h_{2}],$ (7)
$\displaystyle\nu_{1}=(k_{0}^{2}n_{wg}^{2}-\beta_{0}^{2})^{1/2},$ (8)
$\displaystyle\alpha_{2}=(\beta_{0}^{2}-k_{0}^{2}n_{eff}^{2})^{1/2},$ (9)
$\displaystyle n_{eff}=[n_{g}^{2}f+1-f]^{1/2},$ (10) $\displaystyle
m_{0}=(\beta_{0}^{2}-k_{0}^{2}n_{s}^{2})^{1/2},$ (11) $\displaystyle
m_{3}=(\beta_{0}^{2}-k_{0}^{2})^{1/2}$ (12)

where i is the order of waveguide mode. We can determine the location of the
resonance peak based on the above equations. Considering the incident angle
$\theta=7.1^{\circ}$, $\omega_{0}=2\pi c/h_{1}$ and $TE_{0}$ guided mode, the
dispersion relation in the waveguide layer can be expressed as

$h_{1}\sqrt{k_{0}^{2}n_{wg}^{2}-\beta^{2}}=atan(\frac{\sqrt{\beta^{2}-k_{0}^{2}n_{0}^{2}}}{k_{0}^{2}n_{wg}^{2}-\beta^{2}})\\\
+atan(\frac{\sqrt{\beta^{2}-k_{0}^{2}n_{s}^{2}}}{\sqrt{k_{0}^{2}n_{wg}^{2}-\beta^{2}}})$
(13)

The dispersion relation of waveguide layer corresponds to the gray dotted line
in Fig. 2(a). The first and second order tangential wave vector components are
blue and green solid lines. The intersection points in the figure we mark as
X, which means GMR condition caused by $k_{i}$ and $\beta$. Besides, we give
the energy bands of grating to match the GMR condition and Fano peak in the
next section.

Figure 2(b) shows the energy bands of structure. For the periodic structure,
we set the size of first Brillouin zone to be $2\pi/D$ and find the point of
BIC. In the next work, we calculate reflectance and transmission spectra with
finite element method (FEM) [14]. After the model analysis, we can obtain the
enhanced electric field modes with the different tilt angle. When the grating
maintains rectangle, light is limited due to limitation of the bound
continuous. Then we break the symmetric rectangular grating, a little
diffusion of the original light field occurrs. In the following, we will
further discuss the change of the models and topological polarization by
COMSOL Multiphysics.

##  III Simulation results

In the last few years of reported works, discussing the symmetry of the
structure is a way to study BIC. For grating as an optical device, it is
favorable to study the BIC phenomenon. In our work, firstly, we consider a
normal grating structure, which corresponds to the BIC state. However, some
optical phenomena cannot be observed due to the infinite Q-factor. After that,
we try to cut off an oblique angle of the rectangular grating. When the planar
symmetry of the structure is broken, a steep reflection peak is found. We keep
one of the rectangle unchanged, and then define an asymmetric parameter
$M_{1}$ to control the trapezoidal grating. In order to achieve a better Fano
line shape, we need to scan the parameter $M_{1}$ from 0 to 1. We plot the
distribution of reflection as a function of the asymmetric parameter $M_{1}$
and wavelength in Fig. 3(a). We have circled the location of the quasi-BIC at
the wavelength $\lambda=2387nm$ in the Fig. 3(a) and the narrowest Fano peak
occurs when $M_{1}=0.3$. Therefore, we set the tilt of one of the rectangles
to 0.3, then control another rectangle.

We adjust the other trapezoid while keeping the parameter $M_{1}$ constant as
shown in Fig. 3(b). Similarly, we define a new tilt parameter $M_{2}$, and
show the distribution of reflections in Fig. 3(b). We can see the color
gradient map, where the red areas represent the location of the peak. In order
to get the best Fano peak, we choose $M_{2}=0.5$ in Fig. 3(b). Whether it is
greater than 0.5 or less than 0.5, none of Fano peaks can become better than
the condition $M_{2}=0.5$. We refer to this situation as quasi-BIC, which is
formed by the interference of two channels [21]. The two channels are discrete
spectrum narrowband originated from GMR and the other is continuous spectrum
broadband provided by resonance in the waveguide layer. In addition, resonance
peak will produce a red shift due to that the unit cell varies with $M_{2}$.

Figure 3: (a) Controlling $M_{1}$ from 0.1 to 0.9 and keeping one of the
rectangular grating unchanged, the best Fano peak can appear when $M_{1}$=
0.3. (b) Changing another grating tilt parameter $M_{2}$, a narrowest Fano
peak is achieved for $M_{1}=0.3$, $M_{2}=0.5$.

The topological natures of BIC can be discussed with the change of grating. We
show the distributions of the field polarization states in Figs. 4(b) and (d).
In the surface momentum space of the grating, total winding of the
polarization position remains 2$\pi$. For a symmetrical rectangular grating,
topological charges locate at the point $q=1$ as shown in Fig. 4(b), which
corresponds to symmetric protection BIC. The sum of topological charges can be
calculated as

$q=\frac{1}{2\pi}\oint_{L}d_{\psi}=\frac{n_{w}}{2}$ (14)

where the L is a closed loop that enclosesg one BIC , $\psi$ is the
orientation angle, and $n_{w}$ is defined as the winding number. When the
planar symmetry of the structure is broken, the topology integer charge $q=1$
will split into two half topology in Fig. 4(d). Then they will move in the
momentum space, with one being left-handed circularly polarized (blue) and the
other being right-handed circularly polarized (red). As a result, the field of
grating exhibits polarization diversity, where elliptical polarizations,
linear and circular polarizations appear near the center of the first
Brillouin Zone.

Figure 4: (a) Cross sectional view of a symmetrical rectangular grating. (b)
Momentum space polarization with a symmetrical rectangular structure. (c)
Cross sectional view of a trapezoidal grating, two trapezoids with different
degrees of inclination. (d) Topological charges divided into two half topology
with symmetry breaking structures. Figure 5: (a) The spectra of symmetrical
rectangular grating, reflection is a smooth curve without any peak protrusion,
and the electric field pattern is symmetrically distributed. (b) When one of
the rectangles is cut into a trapezoid, the original smooth curve appears a
steep Fano peak at wavelength equal to 2387nm, a little rotation has occurred
in the electric field pattern. (c) Grating is optimized further by control
another tilt parameter $M_{2}$, where an ultranarrow Fano peak is found.

Here, we have calculated reflection and transmission spectra based on finite
element method (FEM). For the asymmetric grating structure, the inclination of
one of the trapezoids is taken as 0.3, and the other inclination of trapezoids
is taken as 0.5. An asymmetrical Fano line can be found in Figs. 5(b) and (c),
where their peak reflection efficiency is extremely high and reach to 1, and
the intensity distributions of the electric field along the y-axis are also
calculated and shown in the insets. Firstly, we discuss the primitive case
with $M_{1}=0$ and $M_{2}=0$, which corresponds to the symmetrical protection
structure with BIC. Interestingly, when the structure maintains perfect
symmetry, the reflection curve is a smooth line without any steep peak, and
the resonance width vanishes completely in this case. In the case of BIC,
discrete and continuous state will mutually couple in the structure, and its
energy will be confined in the grating without any leakage. BIC always means
an infinite Q factor, and an infinite narrow line width is appeared in the
spectrum, which agree with the phenomenon in spectrum of Fig .5(a). The mode
of electric field can be seen in the inset of Fig .5(a), where it exhibits a
neat concentration of energy.

Figure 6: (a) Phase variation with angle for a grating consisting of a
trapezoid and a rectangle (blue). GH shift angular spectrum with the
asymmetric geometric parameter set to $M_{1}=0.3$(red), where the incident
angle is equals to $7.12^{\circ}$. Phase variation with angle for double
trapezoid grating (blue). GH shift angular spectrum with the asymmetric
geometric parameter set to $M_{1}=0.3$ and $M_{2}=0.5(red)$, where incident
angle is equals to $7.15^{\circ}$. (c) The dependence of the sensitivity on
the refractive index, with the increase of refractive index, the peak of GH
shift shifts towards larger angles. (d) GH shift angular spectrum of sensor at
$n_{0}=1$ and $n_{0}=1.002$.

However, BIC is difficult to be applied and designed in the field of optics
due to its infinite Q factor and difficulty in manufacturing process.
Therefore, considering quasi-BIC is an important way to achieve various good
optical properties. Quasi-BIC is a state that is infinitely close to BIC so
that it always has an ultra-high Q factor and an ultra-narrow Fano peak as can
be seen in the spectrum. In the Fig. 5(b), we can find an asymmetrical Fano
peak at the wavelength equal to 2387nm, which corresponds to the blue dotted
line in Fig. 3(a). Compared to the initial structure, when the one of the
rectangles is cut off a corner, the mode of electric fields will produce
leakage and offset due to the symmetry broken. Cutting off a corner of the
grating leads to a change in structure, topological charges start splitting
from $q=1$ in the momentum space.

Only changing the tilt of one grating in a unit cell, we can get an
asymmetrical narrow Fano peak, but it is not the optimal structure to achieve
various optical applications. Thus, this structure needs to be optimized
further by controlling another inclination angle in a cycle. When the tilt
parameters $M_{1}=0.3$ and $M_{2}=0.5$, the Fano line width can be shrunk to a
minimum. In Fig. 5(c), an ultra-narrow Fano peak can be reached, the line
width is much narrower compared to that in Fig. 5(b). The result of peak is
benefit from the optimization of the structure, this line of which corresponds
to the white dotted line in Fig. 3(b). The reflection peak obtained is close
to complete reflection at wavelength equal to 2410nm. It’s a state that
closest to BIC, where topological charges have been splitting into two half
charges totally in the momentum space, and it’s electric field energy can be
seen in the inset of Fig. 5(c) with a strong energy bondage. In this case, the
mode represents leak and loss and the radiation will propagate into free
space, which leads to a finite lifetime rather than infinite lifetime of the
BIC.

The tilted grating has an ultra-narrow Fano peak, which can achieve a large GH
shift. Next, we utilize the quasi-BIC to achieve a large GH shift based on a
phase change. The asymmetry geometric parameter $M_{1}$ and $M_{2}$ determine
the value of the GH shift, i.e., resonance will become stronger and result in
a larger GH shift for specific values of the asymmetric parameter. Here, for
the stationary phase method, the lateral GH shift for the reflected and
transmitted beams can be calculate by using the expression [22-25],

$S_{GH}=-\frac{\lambda}{2\pi}\frac{\partial_{\varphi_{r}}}{\partial_{\theta}}$
(15)

where $\varphi_{r}$ is the reflection phase and the GH shift is proportional
to the partial derivative of the reflection phase. Here, we firstly discuss
the structure that is only cut off one inclination angle. The phase can change
with the incident angle increase under $\lambda$ = 2387 nm and the GH shift
angular spectrum is shown in Fig. 6(a). The GH shift shows significantly
different behavior around the resonance angle of $\theta=7.12^{\circ}$. For
the double trapezoid structure, the phase change with the incident angle is
calculated in Fig. 6(b), which undergoes a phase abrupt at the incident angle
$\theta=7.15^{\circ}$. The GH shift in the Fig. 6(b) is far larger than in the
Fig .6(a). In contrast with traditional GH shift enhanced by transmission-type
resonances, the maximum GH shift assisted by the quasi-BIC locates at a
reflectance peak and it can be detected and utilized more easily.

Based on the large GH shift, we design an ultrasensitive sensor with the
quasi-BIC, and define $n_{0}$ as refractive index of surrounding medium. For
the surrounding environment of the structure, we firstly calculate GH shift
with air $n_{0}$=1 at the room temperature. Then we increase the refractive
index of surrounding medium from 1, which can be seen in Fig .6(c). As the
index of refraction increases, the peak of GH shift will shift toward a larger
incident angle. In addition, the sensitivity of sensor can be defined by the
value change of GH shift and the refractive index of surrounding medium, i.e.,
$S(n_{0})=|d(S_{GH})/dn_{0}|$, and the dependence of the sensitivity on the
refractive index can be seen in the Fig. 6(d). The blue solid line represents
the refractive index of air $n_{0}=1$, when the medium has changed, the GH
shift is a red dotted line in Fig. 6(d). The sensitivity is 349.885 $\mu$m/RIU
with the refractive change from 1 to 1.002. It indicates that the sensor we
designed has a high sensitivity with the change of medium.

##  IV conclusions

In conclusion, we have discussed a double trapezoid structure with a strong
resonance and split of topological charge. In addition, the point of the BIC
in the energy bands can be found by finite element method. We use the GMR and
dispersion relations to analyze the interaction between light and matter,
which causes asymmetric Fano line shape. We break the symmetry by cutting off
a corner of the rectangle, and control the degree of tilt by changing
parameters $M_{1}$ and $M_{2}$. With the optimization of parameters, a typical
asymmetric Fano line-shape’s width can reduce further, where the parameters
are $M_{1}=0.3$ and $M_{2}=0.5$. In the momentum space, topological charge
will split into two half charge $q=1/2$ with the destruction of symmetry.
Assisted by strong resonance with two different channels, a saltatory phase
can be obtained, then the maximum GH shift is achieved at the position of
saltatory phase. Our work provides a new perspective to realize BIC, which can
support many optical applications such as optical switches, high-performance
sensors and wavelength division multiplexers.

###### Acknowledgements.

This work is supported by National Natural Science Foundation of China
(1211101294); Intergovernmental Science and Technology Regular Meeting
Exchange Project of Ministry of Science and Technology of China (CB02-20);
Open Fund of State Key Laboratory of Millimeter Waves (K202238).

## References

  * Von Neumann and Wigner [1929] J. Von Neumann and E. Wigner, On some peculiar discrete eigenvalues, Phys. Z 465 (1929).
  * Capasso  _et al._ [1992] F. Capasso, C. Sirtori, J. Faist, D. L. Sivco, S.-N. G. Chu, and A. Y. Cho, Observation of an electronic bound state above a potential well, Nature 358, 565 (1992).
  * Friedrich and Wintgen [1985] H. Friedrich and D. Wintgen, Interfering resonances and bound states in the continuum, Physical Review A 32, 3231 (1985).
  * Doeleman  _et al._ [2018] H. M. Doeleman, F. Monticone, W. den Hollander, Andrea, and A. F. Koenderink, Experimental observation of a polarization vortex at an optical bound state in the continuum, Nature Photonics 12, 397 (2018).
  * Azzam  _et al._ [2018] S. I. Azzam, V. M. Shalaev, A. Boltasseva, and A. V. Kildishev, Formation of bound states in the continuum in hybrid plasmonic-photonic systems, Physical review letters 121, 253901 (2018).
  * Wu  _et al._ [2019] F. Wu, J. Wu, Z. Guo, H. Jiang, Y. Sun, Y. Li, J. Ren, and H. Chen, Giant enhancement of the goos-hänchen shift assisted by quasibound states in the continuum, Physical Review Applied 12, 014028 (2019).
  * Koshelev  _et al._ [2018] K. Koshelev, S. Lepeshov, M. Liu, A. Bogdanov, and Y. Kivshar, Asymmetric metasurfaces with high-q resonances governed by bound states in the continuum, Physical review letters 121, 193903 (2018).
  * Bai  _et al._ [2021] T. Bai, Q. Li, Y. Wang, Y. Chen, Z.-D. Hu, and J. Wang, Terahertz vortex beam generator based on bound states in the continuum, Optics Express 29, 25270 (2021).
  * Cheng  _et al._ [2020] K. Cheng, Z. Hu, Y. Wang, J. Ma, and J. Wang, High-performance terahertz vortex beam generator based on square-split-ring metasurfaces, Optics Letters 45, 6054 (2020).
  * Huang  _et al._ [2020] C. Huang, C. Zhang, S. Xiao, Y. Wang, Y. Fan, Y. Liu, N. Zhang, G. Qu, H. Ji, J. Han, _et al._ , Ultrafast control of vortex microlasers, Science 367, 1018 (2020).
  * Liu  _et al._ [2018] H. Liu, C. Guo, G. Vampa, J. L. Zhang, T. Sarmiento, M. Xiao, P. H. Bucksbaum, J. Vučković, S. Fan, and D. A. Reis, Enhanced high-harmonic generation from an all-dielectric metasurface, Nature Physics 14, 1006 (2018).
  * Xu  _et al._ [2019] L. Xu, K. Zangeneh Kamali, L. Huang, M. Rahmani, A. Smirnov, R. Camacho-Morales, Y. Ma, G. Zhang, M. Woolley, D. Neshev, _et al._ , Dynamic nonlinear image tuning through magnetic dipole quasi-bic ultrathin resonators, Advanced Science 6, 1802119 (2019).
  * Koshelev  _et al._ [2019] K. Koshelev, Y. Tang, K. Li, D.-Y. Choi, G. Li, and Y. Kivshar, Nonlinear metasurfaces governed by bound states in the continuum, ACS Photonics 6, 1639 (2019).
  * Moharam  _et al._ [1995] M. Moharam, D. A. Pommet, E. B. Grann, and T. Gaylord, Stable implementation of the rigorous coupled-wave analysis for surface-relief gratings: enhanced transmittance matrix approach, JOSA A 12, 1077 (1995).
  * Zhang  _et al._ [2018] Y. Zhang, W. Liu, Z. Li, Z. Li, H. Cheng, S. Chen, and J. Tian, High-quality-factor multiple fano resonances for refractive index sensing, Optics letters 43, 1842 (2018).
  * Wang  _et al._ [2018] Y. Wang, M. A. Ali, E. K. Chow, L. Dong, and M. Lu, An optofluidic metasurface for lateral flow-through detection of breast cancer biomarker, Biosensors and Bioelectronics 107, 224 (2018).
  * Gupta  _et al._ [2017] M. Gupta, Y. K. Srivastava, M. Manjappa, and R. Singh, Sensing with toroidal metamaterial, Applied Physics Letters 110, 121108 (2017).
  * Romano  _et al._ [2018] S. Romano, G. Zito, S. Torino, G. Calafiore, E. Penzo, G. Coppola, S. Cabrini, I. Rendina, and V. Mocella, Label-free sensing of ultralow-weight molecules with all-dielectric metasurfaces supporting bound states in the continuum, Photonics Research 6, 726 (2018).
  * Zhang  _et al._ [2021] C. Zhang, Q. Liu, X. Peng, Z. Ouyang, and S. Shen, Sensitive thz sensing based on fano resonance in all-polymeric bloch surface wave structure, Nanophotonics 10, 3879 (2021).
  * Sang  _et al._ [2011] T. Sang, S. Cai, and Z. Wang, Guided-mode resonance filter with an antireflective surface consisting of a buffer layer with refractive index equal to that of the grating, Journal of Modern Optics 58, 1260 (2011).
  * Norton  _et al._ [1997] S. M. Norton, T. Erdogan, and G. M. Morris, Coupled-mode theory of resonant-grating filters, JOSA A 14, 629 (1997).
  * Miroshnichenko  _et al._ [2010] A. E. Miroshnichenko, S. Flach, and Y. S. Kivshar, Fano resonances in nanoscale structures, Reviews of Modern Physics 82, 2257 (2010).
  * Wang  _et al._ [2005] L.-G. Wang, H. Chen, and S.-Y. Zhu, Large negative goos–hänchen shift from a weakly absorbing dielectric slab, Optics letters 30, 2936 (2005).
  * Wang and Zhu [2006] L.-G. Wang and S.-Y. Zhu, Giant lateral shift of a light beam at the defect mode in one-dimensional photonic crystals, Optics letters 31, 101 (2006).
  * Wen  _et al._ [2017] J. Wen, J. Zhang, L.-G. Wang, and S.-Y. Zhu, Goos–hänchen shifts in an epsilon-near-zero slab, JOSA B 34, 2310 (2017).


# Exploring Cross-lingual Textual Style Transfer with
Large Multilingual Language Models

Daniil Moskovskiy1 Daryna Dementieva1,2 Alexander Panchenko1
1Skolkovo Institute of Science and Technology, Russia
2Technical University of Munich, Germany
<EMAIL_ADDRESS>
###### Abstract

Detoxification is a task of generating text in polite style while preserving
meaning and fluency of the original toxic text. Existing detoxification
methods are designed to work in one exact language. This work investigates
multilingual and cross-lingual detoxification and the behavior of large
multilingual models like in this setting. Unlike previous works we aim to make
large language models able to perform detoxification without direct fine-
tuning in given language. Experiments show that multilingual models are
capable of performing multilingual style transfer. However, models are not
able to perform cross-lingual detoxification and direct fine-tuning on exact
language is inevitable.

##  1 Introduction

The task of Textual Style Transfer (Textual Style Transfer) can be viewed as a
task where certain properties of text are being modified while rest retain the
same111Hereinafter the data-driven definition of style is used. Therefore, we
call style a characteristic of given dataset that differs from a general
dataset Jin et al. (2020).. In this work we focus on detoxification textual
style transfer dos Santos et al. (2018a); Dementieva et al. (2021a). It can be
formulated as follows: given two text corpora $D^{X}=\\{x_{1},x_{2},\dots
x_{n}\\}$ and $D^{Y}=\\{y_{1},y_{2},\dots,y_{n}\\}$, where $X$, $Y$ \- are two
sets of all possible text in styles $s^{X}$, $s^{Y}$ respectively, we want to
build a model $f_{\theta}:X\rightarrow Y$, such that the probability
$p(y_{gen}|x,s^{X},s^{Y})$ of transferring the style $s^{X}$ of given text $x$
(by generation $y_{gen}$) to the style $s^{Y}$ is maximized (where $s^{X}$ and
$s^{Y}$ are toxic and non-toxic styles respectively).

Some examples of detoxification presented in Table 1.

Textual style transfer gained a lot of attention with a rise of deep learning-
based NLP methods. Given that, Textual Style Transfer has now a lot of
specific subtasks ranging from formality style transfer Rao and Tetreault
(2018); Yao and Yu (2021) and simplification of domain-specific texts Devaraj
et al. (2021); Maddela et al. (2021) to emotion modification Sharma et al.
(2021) and detoxification (debiasing) Li et al. (2021); Dementieva et al.
(2021a).

Source text | Target text
---|---
What the f*ck is your problem? | What is your problem?
This whole article is bullshit. | This article is not good.
Yeah, this clowns gonna make alberta great again! | Yeah, this gonna make Alberta great again
Table 1: Examples of desired detoxification results.

There exist a variety of Textual Style Transfer methods: from totally
supervised methods Wang et al. (2019b); Zhang et al. (2020); Dementieva et al.
(2021a) which require a parallel text corpus for training to unsupervised Shen
et al. (2017); Wang et al. (2019a); Xu et al. (2021) that are designed to work
without any parallel data. The latter sub-field of research is more popular
nowadays due to the scarcity of parallel text data for Textual Style Transfer.
On the other hand, if we address Textual Style Transfer task as a Machine
Translation task we get a significant performance boost Prabhumoye et al.
(2018).

The task of detoxification, in which we focus in this work, is relatively new.
First work on detoxification was a sequence-to-sequence collaborative
classifier, attention and the cycle consistency loss dos Santos et al.
(2018b). A recent work by Laugier et al. (2021) introduces self-supervised
model based on T$5$ model Raffel et al. (2020) with a denoising and cyclic
auto-encoder loss.

Both these methods are unsupervised which is an advantage but it comes from
the major current problem of the textual style transfer. There is a lack of
parallel data for Textual Style Transfer since there exist only few parallel
datasets for English Rao and Tetreault (2018) and some other languages Briakou
et al. (2021). When it comes to detoxification there are only two parallel
detoxification corpora available now and they both appeared only last year
Dementieva et al. (2021b). Most state-of-the-art methods rely on large amounts
of text data which is often available for some well-researched languages like
English but lacking for other languages almost entirely. Therefore, it is
important to study whether cross-lingual (or at least multilingual)
detoxification is possible.

Multilingual language models such as mBART Liu et al. (2020), mT5 Xue et al.
(2021) have recently become available. This work explores the possibility of
multilingual and cross-lingual textual style transfer (Textual Style Transfer)
using such large multilingual language models. We test the hypothesis that
modern large text-to-text models are able to generalize ability of style
transfer across languages.

Our contributions can be summarized as follows222All code is available online:
https://github.com/skoltech-nlp/multilingual_detox:

  1. 1.

We introduce a novel study of multilingual textual style transfer and conduct
experiments with several multilingual language models and evaluate their
performance.

  2. 2.

We conduct cross-lingual Textual Style Transfer experiments to investigate
whether multilingual language models are able to perform Textual Style
Transfer without fine-tuning on a specific language.

##  2 Methodology

We formulate the task of supervised Textual Style Transfer as a sequence-to-
sequence NMT task and fine-tune multilingual language models to translate from
"toxic" to "polite" language.

###  2.1 Datasets

In this work we use two datasets for Russian and English languages. Aggregated
information about datasets could be found in Table 2, examples from datasets
can be found in A.1 and A.2.

Language | Train | Dev | Test
---|---|---|---
English | $18777$ | $988$ | $671$
Russian | $5058$ | $1000$ | $1000$
Table 2: Aggregated datasets statistics.

##### Russian data

We use detoxification dataset333https://github.com/skoltech-
nlp/russe_detox_2022 which consists of $5058$ training sentences, $1000$
validation sentences and $1000$ test sentences.

##### English data

We use ParaDetox Dementieva et al. (2021b) dataset. It consists of $19766$
toxic sentences and their polite paraphrases. This data is split into training
and validation as $95\%$ for training and $5\%$ for validation. For testing we
use a set of $671$ toxic sentences.

###  2.2 Experimental Setup

We perform a series of experiments on detoxification using parallel data for
English and Russian. We train models in two different setups: multilingual and
cross-lingual.

##### Multilingual setup

In this setup we train models on data containing both English and Russian
texts and then compare their performance with baselines trained on these
languages solely.

##### Cross-lingual setup

In cross-lingual setup we test the hypothesis that models are able to perform
detoxification without explicit fine-tuning on exact language. We fine-tune
models on English and Russian separately and then test their performance.

###  2.3 Models

Scaling language models to many languages has become an emerging topic of
interest recently Devlin et al. (2019); Tan et al. (2019); Conneau and Lample
(2019); Conneau et al. (2020). We adopt several multilingual models to textual
style transfer in our work.

##### Baselines

We use two detoxification methods as baselines in this work - Delete method
which simply deletes toxic words in the sentence according to the vocabulary
of toxic words and CondBERT. The latter approach works in usual masked-LM
setup by masking toxic words and replacing them with non-toxic ones. This
approach was first proposed by Wu et al. (2019) as a data augmentation method
and then adopted to detoxification by Dale et al. (2021).

##### mT5

mT5 Xue et al. (2021) is a multilingual version of T5 Raffel et al. (2020) \-
a text-to-text transformer model, which was trained on many downstream tasks.
mT5 replicates T5 training but now it is trained on more than $100$ languages.

##### mBART

mBART Liu et al. (2020) is a multilingual variation of BART Lewis et al.
(2020) \- denoising autoencoder built with a sequence-to-sequence model. mBART
is trained on monolingual corpora across many languages. We adopt mBART in
sequence-to-sequence detoxification task via fine-tuning on parallel
detoxification dataset.

###  2.4 Evaluation metrics

Unlike other NLP tasks, one metric is not enough to benchmark the quality of
style transfer. The ideal Textual Style Transfer model output should preserve
the original content of the text, change the style of the original text to
target and the generated text also should be grammatically correct. We follow
Dale et al. (2021) approach in Textual Style Transfer evaluation.

####  2.4.1 Content Preservation

##### Russian

Content preservation score (SIM) is evaluated as a cosine similarity of LaBSE
Feng et al. (2020) sentence embeddings. The model is slightly different from
the original one, only English and Russian embeddings are left.

##### English

Similarity (SIM) between the embedding of the original sentence and the
generated one is calculated using the model presented by Wieting et al.
(2019). Being is trained on paraphrase pairs extracted from ParaNMT corpus
Wieting and Gimpel (2018), this model’s training objective is to select
embeddings such that the similarity of embeddings of paraphrases is higher
than the similarity between sentences that are not paraphrases.

####  2.4.2 Grammatic and language quality (fluency)

##### Russian

We measure fluency (FL) with a BERT-based classifier Devlin et al. (2019)
trained to distinguish real texts from corrupted ones. The model was trained
on Russian texts and their corrupted (random word replacement, word deletion
and insertion, word shuffling etc.) versions. Fluency is calculated as a
difference between the probabilities of being corrupted for source and target
sentences. The logic behind using difference is that we ensure that the
generated sentence is not worse than the original one in terms of fluency.

##### English

We measure fluency (FL) as a percentage of fluent sentences evaluated by the
RoBERTa-based444https://huggingface.co/roberta-large Liu et al. (2019)
classifier of linguistic acceptability trained on CoLA Warstadt et al. (2019)
dataset.

####  2.4.3 Style transfer accuracy

##### Russian

Style transfer accuracy (STA) is evaluated with a BERT-based Devlin et al.
(2019) toxicity
classifier555https://huggingface.co/SkolkovoInstitute/russian_toxicity_classifier
fine-tuned from RuBERT Conversational. This classifier was additionally
trained on Russian Language Toxic Comments dataset collected from 2ch.hk and
Toxic Russian Comments dataset collected from ok.ru.

##### English

Style transfer accuracy (STA) is calculated with a style classifier - RoBERTa-
based Liu et al. (2019) model trained on the union of three Jigsaw datasets
Jigsaw (2018). The sentence is considered toxic when the classifier confidence
is above $0.8$. The classifier reaches the AUC-ROC of $0.98$ and F1-score of
$0.76$.

####  2.4.4 Joint metric

Aforementioned metrics must be properly combined to get one Joint metric to
evaluate Textual Style Transfer. We follow Krishna et al. (2020) and calculate
J as an average of products of sentence-level fluency, style transfer
accuracy, and content preservation:

$\textbf{J}=\frac{1}{n}\sum\limits_{i=1}^{n}\textbf{STA}(x_{i})\cdot\textbf{SIM}(x_{i})\cdot\textbf{FL}(x_{i})$
(1)

| STA$\uparrow$ | SIM$\uparrow$ | FL$\uparrow$ | J$\uparrow$ | STA$\uparrow$ | SIM$\uparrow$ | FL$\uparrow$ | J$\uparrow$
---|---|---|---|---|---|---|---|---
| Russian | English
| Baselines
Delete | 0.532 | 0.875 | 0.834 | 0.364 | 0.810 | 0.930 | 0.640 | 0.460
condBERT Dale et al. (2021) | 0.819 | 0.778 | 0.744 | 0.422 | 0.980 | 0.770 | 0.820 | 0.620
| Multilingual Setup
mT$5$ base | 0.772 | 0.676 | 0.795 | 0.430 | 0.833 | 0.826 | 0.830 | 0.556
mT$5$ small | 0.745 | 0.705 | 0.794 | 0.428 | 0.826 | 0.841 | 0.763 | 0.513
mT$5$ base∗ | 0.773 | 0.676 | 0.795 | 0.430 | 0.893 | 0.787 | 0.942 | 0.657
mBART $5000$ | 0.685 | 0.778 | 0.841 | 0.449 | 0.887 | 0.889 | 0.866 | 0.640
| Cross-lingual Setup
mT$5$ base ENG | 0.838 | 0.276 | 0.506 | 0.115 | 0.860 | 0.834 | 0.833 | 0.587
mT$5$ base RUS | 0.676 | 0.794 | 0.846 | 0.454 | 0.906 | 0.365 | 0.696 | 0.171
mT$5$ small ENG | 0.805 | 0.225 | 0.430 | 0.077 | 0.844 | 0.858 | 0.826 | 0.591
mT$5$ small RUS | 0.559 | 0.822 | 0.817 | 0.363 | 0.776 | 0.521 | 0.535 | 0.169
mBART $3000$ ENG | 0.923 | 0.395 | 0.552 | 0.202 | 0.842 | 0.856 | 0.876 | 0.617
mBART $3000$ RUS | 0.699 | 0.778 | 0.858 | 0.475 | 0.547 | 0.778 | 0.888 | 0.299
mBART $5000$ ENG | 0.900 | 0.299 | 0.591 | 0.160 | 0.857 | 0.840 | 0.873 | 0.616
mBART $5000$ RUS | 0.724 | 0.746 | 0.827 | 0.457 | 0.806 | 0.484 | 0.864 | 0.242
| Backtranslation Setup
mBART $5000$ (Google) | 0.675 | 0.669 | 0.634 | 0.284 | 0.678 | 0.762 | 0.568 | 0.284
mBART $5000$ (FSMT) | 0.737 | 0.633 | 0.731 | 0.348 | 0.744 | 0.746 | 0.893 | 0.415
Table 3: Evaluation of TST models. Numbers in bold indicate the best results.
$\uparrow$ describes the higher the better metric. Results of unsuccessful TST
depicted as gray. ENG and RUS depicts the data model have been trained on.
mT$5$ base∗ was trained on all English and Russian data available (datasets
were not equalized). Last row depicts backtranslation workaround for cross-
lingual detoxification. We include only the best result for brevity.

###  2.5 Training

There is a variety of versions of large multilingual models available. In this
work we use small and base versions of
mT5666https://huggingface.co/google/mt5-base777https://huggingface.co/google/mt5-large
Xue et al. (2021) and large version of
mBART888https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt Liu et
al. (2020).

##### Multilingual training

In multilingual training setup we fine-tune models using both English and
Russian data. We use Adam Kingma and Ba (2015) optimizer for fine-tuning with
different learning rates ranging from $1\cdot 10^{-3}$ to $5\cdot 10^{-5}$
with linear learning rate scheduling. We also test different number of warmup
steps from $0$ to $1000$. We equalize Russian and English data for training
and use $10000$ toxic sentences and their polite paraphrases for multilingual
training in total. We train mT5 models for $40$ thousand
iterations999According to Xue et al. (2021) mT5 was not fine-tuned on
downstream tasks as the original T5 model. Therefore, model requires more
fine-tuning iterations for Textual Style Transfer. with a batch size of $8$.
We fine-tune mBART Liu et al. (2020) for $1000$, $3000$, $5000$ and $10000$
iterations with batch size of $8$.

##### Cross-lingual training

In cross-lingual training setup we fine-tune models using only one dataset,
e.g.: we fine-tune model on English data and check performance on both English
and Russian data. Fine-tuning procedure was left the same: $40000$ iterations
for mT5 models and $1000$, $3000$, $5000$ and $10000$ iterations for the
mBART.

Back-translation approach to cross-lingual style transfer proved to work
substantially better than the zero-shot setup discussed above. Nevertheless,
both Google and FSMT did not yield scores comparable to monolingual setup.
Besides, surprisingly Google yielded worse results than FSMT.

##  3 Results & Discussion

Table 3 shows the best scores of both multilingual and cross-lingual
experiments. In multilingual setup mBART performs better than baselines and
mT$5$ for both English and Russian. Note that the table shows only the best
results of the models. It is also notable that for mT$5$ increased training
size for English data provides better metrics for English while keeping
metrics for Russian almost the same. We also depict some of the generated
detoxified sentences in the Table 3 in the part B of Appendix.

As for cross-lingual style transfer, results are negative. None of the models
have coped with the task of cross-lingual Textual Style Transfer. That means
that models produce the same or almost the same sentences for the language on
which they were not fine-tuned so that toxicity is not eliminated. We provide
only some scores here in the Table 6 for reference.

Despite the fact that our hypothesis about the possibility of cross-language
detoxification was not confirmed, the presence of multilingual models pre-
trained in many languages gives every reason to believe that even with a small
amount of parallel data, training models for detoxification is possible.

A recent work by Lai et al. (2022) shows that cross-lingual formality Textual
Style Transfer is possible. Lai et al. (2022) achieve this on XFORMAL dataset
Briakou et al. (2021) by adding language-specific adapters in the vanilla
mBART architecture Liu et al. (2020) \- two feed-forward layers with residual
connection and layer normalization Bapna and Firat (2019); Houlsby et al.
(2019).

We follow the original training procedure described by Lai et al. (2022) by
training adapters for English and Russian separately on $5$ million sentences
from News Crawl dataset101010https://data.statmt.org/news-crawl/. We use batch
size of $16$ and $200$ thousand training iterations. We also then train cross-
attentions on our parallel detoxifcation data in the same way. However, models
tend to duplicate input text without any detoxification. Thus, while the exact
same original setup did not work for detoxification, more parameter search and
optimization could lead to more acceptable results and we consider the
approach by Lai et al. (2022) as a promising direction of a future work on
multilingual and cross-lingual detoxification.

##  4 Conclusion

In this work we have tested the hypothesis that multilingual language models
are capable of performing cross-lingual and multilingual detoxification. In
the multilingual setup we experimentally show that reformulating
detoxification (Textual Style Transfer) as a NMT task boosts performance of
the models given enough parallel data for training. We beat simple (Delete
method) and more strong (condBERT) baselines in a number of metrics. Based on
our experiments, we can assume that it is possible to fine-tune multilingual
models in any of the $100$ languages in which they were originally trained.
This opens up great opportunities for detoxification in unpopular languages.

However, our hypothesis that multilingual language models are capable of
cross-lingual detoxification was proven to be false. We suggest that the
reason for this is not a lack of data, but the model’s inability to capture
the pattern between toxic and non-toxic text and transfer it to another
language by itself. This means that the problem of cross-lingual textual style
transfer is still open and needs more investigation.

## Acknowledgements

This work was supported by MTS-Skoltech laboratory on AI.

## References

  * Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019.  Simple, scalable adaptation for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1538–1548, Hong Kong, China. Association for Computational Linguistics.
  * Briakou et al. (2021) Eleftheria Briakou, Di Lu, Ke Zhang, and Joel R. Tetreault. 2021.  Olá, bonjour, salve! XFORMAL: A benchmark for multilingual formality style transfer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 3199–3216. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8440–8451. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 7057–7067.
  * Dale et al. (2021) David Dale, Anton Voronov, Daryna Dementieva, Varvara Logacheva, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021.  Text detoxification using large pre-trained neural models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 7979–7996. Association for Computational Linguistics.
  * Dementieva et al. (2021a) Daryna Dementieva, Daniil Moskovskiy, Varvara Logacheva, David Dale, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021a.  Methods for detoxification of texts for the russian language.  _Multimodal Technol. Interact._ , 5(9):54.
  * Dementieva et al. (2021b) Daryna Dementieva, Sergey Ustyantsev, David Dale, Olga Kozlova, Nikita Semenov, Alexander Panchenko, and Varvara Logacheva. 2021b.  Crowdsourcing of parallel corpora: the case of style transfer for detoxification.  In _Proceedings of the 2nd Crowd Science Workshop: Trust, Ethics, and Excellence in Crowdsourced Data Management at Scale co-located with 47th International Conference on Very Large Data Bases (VLDB 2021 (https://vldb.org/2021/))_ , pages 35–49, Copenhagen, Denmark. CEUR Workshop Proceedings.
  * Devaraj et al. (2021) Ashwin Devaraj, Iain Marshall, Byron Wallace, and Junyi Jessy Li. 2021.  Paragraph-level simplification of medical texts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4972–4984, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * dos Santos et al. (2018a) Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018a.  Fighting offensive language on social media with unsupervised text style transfer.
  * dos Santos et al. (2018b) Cícero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018b.  Fighting offensive language on social media with unsupervised text style transfer.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_ , pages 189–194. Association for Computational Linguistics.
  * Feng et al. (2020) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020\.  Language-agnostic BERT sentence embedding.  _CoRR_ , abs/2007.01852.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Jigsaw (2018) Jigsaw. 2018.  Toxic comment classification challenge.  https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge.  Accessed: 2021-03-01.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2020.  Deep learning for text style transfer: A survey.  _CoRR_ , abs/2011.00416.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Krishna et al. (2020) Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020.  Reformulating unsupervised style transfer as paraphrase generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 737–762, Online. Association for Computational Linguistics.
  * Lai et al. (2022) Huiyuan Lai, Antonio Toral, and Malvina Nissim. 2022.  Multilingual pre-training with language and task adaptation for multilingual text style transfer.  _CoRR_ , abs/2203.08552.
  * Laugier et al. (2021) Leo Laugier, John Pavlopoulos, Jeffrey Sorensen, and Lucas Dixon. 2021.  Civil rephrases of toxic texts with self-supervised transformers.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 1442–1461. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7871–7880. Association for Computational Linguistics.
  * Li et al. (2021) Mingzhe Li, Xiuying Chen, Min Yang, Shen Gao, Dongyan Zhao, and Rui Yan. 2021.  The style-content duality of attractiveness: Learning to write eye-catching headlines via disentanglement.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 13252–13260. AAAI Press.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Trans. Assoc. Comput. Linguistics_ , 8:726–742.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Maddela et al. (2021) Mounica Maddela, Fernando Alva-Manchego, and Wei Xu. 2021.  Controllable text simplification with explicit paraphrasing.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 3536–3553. Association for Computational Linguistics.
  * Prabhumoye et al. (2018) Shrimai Prabhumoye, Yulia Tsvetkov, Alan W. Black, and Ruslan Salakhutdinov. 2018\.  Style transfer through multilingual and feedback-based back-translation.  _CoRR_ , abs/1809.06284.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018.  Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 129–140, New Orleans, Louisiana. Association for Computational Linguistics.
  * Sharma et al. (2021) Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2021\.  Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach.  In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , pages 194–205. ACM / IW3C2.
  * Shen et al. (2017) Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017.  Style transfer from non-parallel text by cross-alignment.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 6830–6841.
  * Tan et al. (2019) Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019.  Multilingual neural machine translation with knowledge distillation.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wang et al. (2019a) Ke Wang, Hang Hua, and Xiaojun Wan. 2019a.  Controllable unsupervised text attribute transfer via editing entangled latent representation.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 11034–11044.
  * Wang et al. (2019b) Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019b.  Harnessing pre-trained neural networks with rules for formality style transfer.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3571–3576. Association for Computational Linguistics.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Trans. Assoc. Comput. Linguistics_ , 7:625–641.
  * Wieting et al. (2019) John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019.  Beyond bleu: Training neural machine translation with semantic similarity.  In _Proceedings of the Association for Computational Linguistics_.
  * Wieting and Gimpel (2018) John Wieting and Kevin Gimpel. 2018.  ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 451–462, Melbourne, Australia. Association for Computational Linguistics.
  * Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019.  Conditional bert contextual augmentation.  In _Computational Science – ICCS 2019_ , pages 84–95, Cham. Springer International Publishing.
  * Xu et al. (2021) Haoran Xu, Sixing Lu, Zhongkai Sun, Chengyuan Ma, and Chenlei Guo. 2021.  VAE based text style transfer with pivot words enhancement learning.  _CoRR_ , abs/2112.03154.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 483–498. Association for Computational Linguistics.
  * Yao and Yu (2021) Zonghai Yao and Hong Yu. 2021.  Improving formality style transfer with context-aware rule injection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 1561–1570. Association for Computational Linguistics.
  * Zhang et al. (2020) Yi Zhang, Tao Ge, and Xu Sun. 2020.  Parallel data augmentation for formality style transfer.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 3221–3228. Association for Computational Linguistics.

##  Appendix A Data

###  A.1 English Dataset

Table 4 shows examples of sentence pairs from ParaDatex parallel
detoxification corpora. There are several polite paraphrases for each toxic
sentence in this dataset Dementieva et al. (2021b), this is a consequence of
the way these parallel data are collected. Leaving only one paraphrase for one
source sentence we could get $6000$ unique pairs of toxic sentences and their
polite paraphrases. However, in this work we use data as is.

Original | my computer is broken and my phone too!! wtf is this devil sh*t???
---|---
Detoxed | My computer is broken and my phone too! So disappointed!
| My computer is broken and my phone too, what is this?
| Both my computer and phone are broken.
Original | sh*t is crazy around here
Detoxed | It is crazy around here.
| Stuff is crazy around here.
| Something is crazy around here.
Original | delete the page and shut up
Detoxed | Delete the page and stay silent.
| Please delete the page.
| Delete the page.
Original | massive and sustained public pressure is the only way to get these b*stards to act.
Detoxed | Massive and sustained public pressure is the only way to get them to act.
| Massive and sustained pressure is the only way to get these people to act.
Original | f*ck you taking credit for some sh*t i wanted to do
Detoxed | You are taking credit for something I wanted to do
| You’re taking credit fro something i wanted to do.
Original | you gotta admit that was f*ckin hilarious though!
Detoxed | you got to admit that was very hilarious though!
| you gotta admit that was hilarious though!
Table 4: Example sentences from ParaDetox parallel detoxification corpora.
Sentence in red is original (toxic) sentence, below are its polite
paraphrases. Note that for the purpose of an overall correctness explicit
words are masked with "*".

###  A.2 Russian Dataset

Table A.2 shows examples from Russian parallel detoxification corpus.

Original
---

Х*рню всякую пишут,из-за этого лайка.v Долбо**изм. Translation & They write
all sorts of bullshit, because of this like. Stupidity. Detoxed Чушь всякую
пишут, из- за этого лайка. Translation They write all sorts of nonsense,
because of this like. Original А нахрена тогда ты здесь это писал? Translation
Why the f*ck did you post it here? Detoxed Зачем ты это писал? Translation Why
did you post it? Original Е*анутые. Отобрать оружие и лодку и штраф тысяч 500
Translation F*ckers. Take away weapons, boat and give a fine of 500 thousand.
Detoxed Отобрать оружие и лодку и штраф тысяч 500. Translation Take away
weapons, boat and give a fine of 500 thousand. Original Не поверишь, сколько
е**нутых на планете. Translation You won’t believe how many f*cked up people
are on the planet. Detoxed Не поверишь сколько таких на планете. Translation
You won’t believe how many people like that are there on the planet.

Table 5: Example sentences from Russian parallel detoxification corpora.
Sentence in red is original (toxic) sentence, below are its polite
paraphrases.

##  Appendix B Generation Examples

Table 6 contains detoxification examples for different models. It is notable
that in some cases models generate almost the same results. This can be
explained by the similarity of the training procedure and the fact that the
reference answer was the same.

Original | After all its hard to get a job if you’re stupid.
---|---
mBART $5000$ ENG | After all its hard to get a job if you don’t understand.
mBART $3000$ ENG | After all its hard to get a job if you are not smart.
mT5 base (multilingual) | It’s hard to get a job if you don’t know.
Original | Fuck minimum security, put him in real prison.
mBART $5000$ ENG | Put him in a real prison with minimum security.
mBART $3000$ ENG | I don’t care about minimum security, put him in real prison.
mT5 base | I don’t care about minimum security, put him in real prison.
Original | Sure he should get punished ,but three f*cking years?
mBART $5000$ ENG | Sure he should get punished, but three years?
mBART $3000$ ENG | He should get punished, but three years?
mT5 base | He’d get punished, but three years?
Paraphrase | Что у этих людей в башке!? походу насрато!
Translation | What’s going on in these people’s heads!? It looks like shit!
mBART $5000$ ENG | Что у этих людей в голове!? Походу бесчеловечно.
mBART $3000$ ENG | Что у этих людей в голове? Походу ненормально!
mT5 base | походу этих людей!? походу!
Table 6: Some detoxified sentences produced by our fine-tuned models. Gray
text refers to the original sentence, below are its paraphrases.