Peak intensity |  $I_{0}$ $p$ $q$ |  [Jy px-1] [–] [–] |  $2.48$ $-3.71$ $2.94$ |  $1.94$ $-2.93$ $2.79$ |  $9.82$ $-5.46$ $2.90$
Line width |  $L_{w}$ $p$ $q$ |  [km s-1] [–] [–] |  $0.51$ $-0.31$ $-0.21$ |  $0.79$ $-0.84$ $0.33$ |  $0.59$ $-0.80$ $0.11$
Line slope |  $L_{s}$ $p$ |  [–] [–] |  $1.82$ $0.18$ |  $1.41$ $0.23$ |  $1.30$ $0.31$
Note. Parameters in brackets result from fixing inclination to the value found for the millimetre continuum in Huang et al. (2018a). Table 9: Best-fit model parameters obtained for channel maps of the disc of GM Aur. |  |  | GM Aur ($159.0$ pc)
---|---|---|---
Attribute | Parameter | Unit |  12CO |  13CO | C18O
Orientation |  $i$ PA $x_{c}$ $y_{c}$ |  $[^{\circ}]$ $[^{\circ}]$ [au] [au] |  $50.8$ $[53.2]$ $53.8$ $[54.0]$ $-0.1$ $[1.6]$ $-7.7$ $[-4.4]$ |  $54.0$ $[53.2]$ $57.1$ $[56.6]$ $-2.4$ $[-1.9]$ $-2.5$ $[-4.3]$ |  $49.1$ $[53.2]$ $57.4$ $[57.1]$ $2.2$ $[-3.1]$ $4.7$ $[-3.2]$
Velocity |  $M_{\star}$ $\upsilon_{\rm sys}$ |  [M⊙] [km s-1] |  $1.01$ $[0.98]$ $5.63$ $[5.63]$ |  $1.11$ $[1.10]$ $5.62$ $[5.62]$ |  $1.30$ $[1.15]$ $5.60$ $[5.60]$
Upper surface |  $z_{0}$ $p$ $R_{t}$ $q$ |  [au] [–] [au] [–] |  $53.3$ $1.72$ $177.5$ $0.53$ |  $15.8$ $1.52$ $475.2$ $2.50$ |  $70.2$ $0.70$ $33.0$ $0.51$
Lower surface |  $z_{0}$ $p$ $R_{t}$ $q$ |  [au] [–] [au] [–] |  $23.3$ $1.50$ $770.5$ $0.96$ |  $13.1$ $1.50$ $488.9$ $3.84$ |  $--$ $--$ $--$ $--$
Peak intensity |  $I_{0}$ $p$ $q$ |  [Jy px-1] [–] [–] |  $1.33$ $-3.33$ $2.49$ |  $0.26$ $-2.95$ $1.82$ |  $0.87$ $-0.60$ $2.60$
Line width |  $L_{w}$ $p$ $q$ |  [km s-1] [–] [–] |  $0.28$ $0.29$ $-0.76$ |  $0.20$ $-0.09$ $-0.45$ |  $0.13$ $-0.77$ $-0.61$
Line slope |  $L_{s}$ $p$ |  [–] [–] |  $1.90$ $-0.02$ |  $1.51$ $0.17$ |  $1.42$ $0.11$
Note. Parameters in brackets result from fixing inclination to the value found for the millimetre continuum in Huang et al. (2020). Table 10: As Table 3 but for the disc of AS 209. Radius | Dusta | Verticalb | $\Delta\upsilon_{\phi}$ | Tb | Lw | Notes
---|---|---|---|---|---|---
61 | $\downarrow$ | / |  $+$$-$/ | / |  $\downarrow$$\downarrow$/ |  Candidate gas gap with strong vertical pressure variations.
100 | $\downarrow$ | / |  $+$$+$/ |  $\uparrow$// | $\Downarrow$ |  Gas gap detected in kinematics and line widths. Positive velocity gradients more closely centred at $R=90$ au.
74 | $\uparrow$ | / |  $-$$+$/ |  $\downarrow$$\downarrow$/ |  $\uparrow$$\uparrow$/ |  Pressure bump and minimum traced by 12CO and 13CO. Indication of strong vertical pressure variations.
121 | $\uparrow$ | / |  $-$/$+$ | / | / |  Strong vertical pressure variations traced by the kinematics.
140 | / | / |  $-$/$+$ |  $\downarrow$$\uparrow$$\uparrow$ | / |  Strong vertical pressure variations traced by the kinematics.
160 | / | / | /$-$$-$ | / | /$\uparrow$$\downarrow$ |  Pressure bump traced by kinematics near the midplane.
200 | / | / |  $-$$-$/ |  $\downarrow$$\uparrow$/ |  $\uparrow$$\downarrow$/ |  Pressure bump detected in the kinematics, mainly in 12CO.
Table 11: As Table 3 but for the disc of IM Lup. Radius | Dusta | Verticalb | $\Delta\upsilon_{\phi}$ | Tb | Lw | Notes
---|---|---|---|---|---|---
116 | $\downarrow$ | / | $\oplus$ | $\Uparrow$ |  $\downarrow$$\downarrow$/ |  Warm gas gap detected in kinematics and line widths.
209 | $\downarrow$ | / | $\oplus$ |  $\uparrow$// |  $\downarrow$// |  Tentative gas gap from kinematics and 12CO line widths.
133 | $\uparrow$ | / | $\ominus$ | / | / |  Pressure bump detected in kinematics of all COs.
220 | $\uparrow$ | / | $\oplus$ |  $\uparrow$// | / |  Dust ring co-located with pressure minimum, suggesting formation mechanism different from pressure trapping.
279 | / | $\downarrow$ |  $-$/$-$ | / | / |  Pressure bump coincident with vertical modulation.
393 | / | $\downarrow$ |  $-$$-$/ | / |  $\downarrow$// |  Pressure bump coincident with vertical modulation.
Table 12: As Table 3 but for the disc of GM Aur. Radius | Dusta | Verticalb | $\Delta\upsilon_{\phi}$ | Tb | Lw | Notes
---|---|---|---|---|---|---
68 | $\downarrow$ | / | $\oplus$ | /$\uparrow$$\uparrow$ |  $\downarrow$/$\downarrow$ |  Gas gap detected in kinematics and line widths.
142 | $\downarrow$ | / |  $+$$+$/ |  $\uparrow$// | / |  Tentative gas gap from 12CO and 13CO kinematics.
86 | $\uparrow$ | / | $\ominus$ | $\Downarrow$ |  $\uparrow$/$\uparrow$ |  Strong pressure bump, gas density ring co-located with dust ring.
163 | $\uparrow$ | / | / | / | //$\uparrow$ |  Dust ring not clearly correlated with pressure bump.

##  Appendix B Supporting figures

Here we provide complementary figures that support some of the results of the
work, or figures that were not presented in the main text for reasons of
space. In Figures 17–20, we compare data and model channel maps for all CO
isotopologues and discs but MWC 480, whose channel maps were already shown in
Fig. 2. In Figure 21, we present rotation curves and azimuthally averaged
profiles of peak intensities and line widths, measured on the data alone for
all isotopologues, considering the disc vertical structure which is also
shown. In Figure 23, we illustrate the discminer modelling flow and analysis
capabilities. We show residual maps of 13CO and C18O centroid velocities in
Figures 24 and 25. Figures 26, 27, and 28, illustrate 12CO, 13CO, and C18O
line width residuals, respectively, and Figures 29, 30, and 31 show peak
intensity residuals for the same tracers. In Figure 32, we show the effect of
using parametric and non-parametric emission surfaces on the retrieved
intensity and velocity residuals from the 12CO disc of MWC 480. In Figure 33,
we illustrate the impact of assuming an azimuthally symmetric and asymmetric
intensity model (accounting for foreground absorption) on the resulting
intensity and velocity residuals from the 12CO disc of AS 209. In Figures 34,
35, and 36, we present radial profiles of azimuthal and vertical velocity
components, as well as azimuthally averaged line width and peak intensity
residuals for the discs of AS 209, IM Lup and GM Aur, respectively, for all
isotopologues. A version of the same plot can be found in the main text in
Fig. 13 for MWC 480, and in Fig. 14 for HD 163296. Finally, in Figure 37, we
show the effect of continuum subtraction on the velocity profiles extracted
for the disc of MWC 480.

Figure 17: As Fig. 2 but for the disc of HD 163296.  Figure 18: As Fig. 2 but
for the disc of AS 209.  Figure 19: As Fig. 2 but for the disc of IM Lup.
Figure 20: As Fig. 2 but for the disc of GM Aur.  Figure 21: Additional
observables obtained from the discminer analysis of the discs around MWC 480,
HD 163296, AS 209, (IM Lup and GM Aur in next page) as traced by 12CO, 13CO,
and C18O. Top row: Rotation curves computed from azimuthal averages of
deprojected line-of-sight velocities. For reference, the grey dashed line in
each panel is the Keplerian rotation curve of the best-fit model found for
12CO. Second row: Azimuthally averaged profiles of line widths, defined as the
standard deviation of Gaussians fitted to the data line profiles. For
reference, the purple dashed line in each panel represents the thermal
broadening computed at the 13CO peak brightness temperature. Third row:
Azimuthally averaged profiles of peak intensities converted to brightness
temperatures using the Rayleigh-Jeans law. Bottom row: Elevation of upper and
lower emission surfaces. Apart from intensity profiles, all panels in each row
share the same $x$ and $y-$axis extent. Vertical dashed, solid, and dotted
lines mark the radial location of dust gaps, rings, and kinks observed in the
channel maps of 12CO, respectively.  Figure 22: – continued Figure 23:
Schematic representation of the discminer modelling flow and capabilities.
discminer produces best-fit channel maps of the input data cube by modelling
the disc vertical structure and orientation, stellar mass, and attributes that
shape the line profile morphology, assuming that the disc emission is smooth
and Keplerian. The model and data channels are then compared through three
types of moment maps. Large-scale analyses of these maps allow to look into
the disc dynamical and thermodynamic structure, but are also useful to reveal
potential signatures driven by planetary and non-planetary mechanisms, such as
meridional flows or spiral-like perturbations. Small-scale analysis of
residuals from these maps leads to the retrieval of the potential location of
embedded planets through the detection of localised velocity perturbations and
sites of enhanced velocity dispersion.  Figure 24: As Fig. 5 but for 13CO.
This isotopologue traces vertical layers between $0.1<z/r<0.15$ in this sample
of discs.  Figure 25: As Fig. 5 but for C18O. This isotopologue traces
vertical layers between $0.0<z/r<0.1$ in this sample of discs.  Figure 26: As
Fig. 5 but for line width residuals.  Figure 27: As Fig. 5 but for line width
residuals from 13CO.  Figure 28: As Fig. 5 but for line width residuals from
C18O.  Figure 29: As Fig. 5 but for peak intensity residuals.  Figure 30: As
Fig. 5 but for peak intensity residuals from 13CO.  Figure 31: As Fig. 5 but
for peak intensity residuals from C18O.  Figure 32: Illustrating the impact of
using an irregular (top) and a smooth parametric (bottom) emission surface on
the peak intensity and velocity residuals extracted for the 12CO disc of MWC
480. Unphysical quadruple-like patterns observed at $R=300,400$ au and at
$R>400$ au in the velocity residuals resulting from the non-parametric surface
suggest that a smooth (and less tapered) layer provides a better
representation of the 12CO emission surface of this disc.  Figure 33:
Illustrating the influence of having one (top) or two (bottom) model intensity
normalisations on peak intensity and velocity residuals extracted for the 12CO
disc of AS 209. Although it is irrelevant for the retrieved kinematic
signatures, we adopt the two-normalisation approach which aims to emulate
absorption by foreground material occurring on the blueshifted side of the
disc, allowing us to better capture fluctuations in the intensity field.
Figure 34: As Fig. 13 but for the disc of AS 209. The planet marker indicates
the orbital radius of the planet candidate associated with line width
enhancements near the location of the CPD candidate proposed by Bae et al.
(2022b) (see Table 2).  Figure 35: As Fig. 13 but for the disc of IM Lup. The
calculation of the velocity profiles illustrated in this Figure excludes $\pm
30^{\circ}$ wedges around the (north and south) disc minor axis to avoid
contamination from possibly unphysical velocity residuals (see Sect. 3.2, IM
Lup).  Figure 36: As Fig. 14 but for the disc of GM Aur. The calculation of
velocity profiles illustrated in this Figure excludes $\pm 45^{\circ}$ wedges
around the (north and south) disc minor axis to minimise contamination from
infalling material (see Sect. 3.2, GM Aur).  Figure 37: Comparison of
azimuthal and vertical velocity profiles extracted with the decomposition
method introduced in Sect. 2.2.2 for the disc of MWC 480 from cubes with and
without continuum emission.

##  Appendix C Azimuthal average of absolute velocity residuals

In this section, we demonstrate that a radial profile of the azimuthal
component of the velocity field, $\upsilon_{\phi}$, or $\Delta\upsilon_{\phi}$
if referred to an axisymmetric model of the same component, can be obtained
through the computation of azimuthal averages of absolute velocities, or
absolute velocity residuals, respectively. The latter are defined as follows,

$\displaystyle\left|\upsilon_{0,d}^{\prime}\right|-\left|\upsilon_{0,m}^{\prime}\right|\equiv$
$\displaystyle\quad\left|\upsilon_{0,d}-\upsilon_{{\rm
sys},m}\right|-\left|\upsilon_{0,m}-\upsilon_{{\rm sys},m}\right|$
$\displaystyle=$
$\displaystyle\quad\left|\upsilon_{\phi,d}\cos{\phi}\sin{i}-\upsilon_{r,d}\sin{\phi}\sin{i}-\upsilon_{z,d}\cos{i}\right|$
$\displaystyle-\left|\upsilon_{\phi,m}\cos{\phi}\sin{i}\right|.$ (7)

Now, it is useful to note that under the reasonable assumption that the
observed line-of-sight velocities are dominated by azimuthal
velocities666Valid for intermediate and high inclinations where the sine is
similar or much larger than the cosine of the disc inclination, and for almost
all azimuthal angles $\phi$, owing to the fact that azimuthal velocities,
$\upsilon_{\phi,d}$, are normally between one and two orders of magnitude
larger than radial and vertical motions, $\upsilon_{r,d}$ and
$\upsilon_{z,d}$., the absolute value of the data and model velocities can
also be written as,

$\displaystyle\left|\upsilon_{0}^{\prime}\right|\equiv\begin{cases}+\upsilon_{0}^{\prime},&\text{if
$-\pi/2<\phi<\pi/2$}\\\ -\upsilon_{0}^{\prime},&\text{if
$\pi/2<\phi<3\pi/2$},\end{cases}$ (8)

for clockwise rotation and positive inclination. Using eq. 8, an azimuthal
average applied to eq. C can thus be split into different independent terms,

$\displaystyle\left<\left|\upsilon_{0,d}^{\prime}\right|-\left|\upsilon_{0,m}^{\prime}\right|\right>_{2\pi}$
$\displaystyle=$
$\displaystyle\quad\frac{1}{2\pi}\left[\int_{-\pi/2}^{\pi/2}\upsilon_{\phi,d}\cos{\phi}\sin{i}\,d\phi\right.$
$\displaystyle\quad-\cancelto{0}{\int_{-\pi/2}^{\pi/2}\upsilon_{r,d}\sin{\phi}\sin{i}\,d\phi}$
$\displaystyle\quad-\bcancel{\int_{-\pi/2}^{\pi/2}\upsilon_{z,d}\cos{i}\,d\phi}$
$\displaystyle\quad-\int_{\pi/2}^{3\pi/2}\upsilon_{\phi,d}\cos{\phi}\sin{i}\,d\phi$
$\displaystyle\quad+\cancelto{0}{\int_{\pi/2}^{3\pi/2}\upsilon_{r,d}\sin{\phi}\sin{i}\,d\phi}$
$\displaystyle\quad+\bcancel{\int_{\pi/2}^{3\pi/2}\upsilon_{z,d}\cos{i}\,d\phi}$
$\displaystyle\quad-\int_{-\pi/2}^{\pi/2}\upsilon_{\phi,m}\cos{\phi}\sin{i}\,d\phi$
$\displaystyle\quad+\left.\int_{\pi/2}^{3\pi/2}\upsilon_{\phi,m}\cos{\phi}\sin{i}\,d\phi\right]$
$\displaystyle\iff\left<\right>_{2\pi}=$
$\displaystyle\quad\frac{1}{2\pi}\sin{i}\left(\upsilon_{\phi,d}-\upsilon_{\phi,m}\right)\left[\sin{\phi}\biggr{|}_{-\pi/2}^{\pi/2}-\sin{\phi}\biggr{|}_{\pi/2}^{3\pi/2}\right],$
$\displaystyle=$
$\displaystyle\quad\frac{2}{\pi}\sin{i}\left(\upsilon_{\phi,d}-\upsilon_{\phi,m}\right),$

from which the azimuthal component of the velocity is followed immediately,

$\Delta\upsilon_{\phi}\equiv\upsilon_{\phi,d}-\upsilon_{\phi,m}=\frac{\pi}{2\sin{i}}\left<\left|\upsilon_{0,d}-\upsilon_{{\rm
sys},d}\right|-\left|\upsilon_{0,m}-\upsilon_{{\rm
sys},m}\right|\right>_{2\pi}$ (9)

Finally, it is possible to generalise the above expression if the azimuthal
section considered for the azimuthal average, with a total extent of $\psi$
radians and not necessarily continuous, is symmetric around the disc major and
minor axes,

$\Delta\upsilon_{\phi}=\frac{\psi}{4\sin{\frac{\psi}{4}}\sin{|i|}}\left<\left|\upsilon_{0,d}-\upsilon_{{\rm
sys},m}\right|-\left|\upsilon_{0,m}-\upsilon_{{\rm
sys},m}\right|\right>_{\psi}.$ (10)


# 3DTextureTransformer: Geometry Aware Texture Generation for Arbitrary Mesh
Topology

Dharma KC
The University of Arizona
Tucson, AZ, USA
<EMAIL_ADDRESS>Clayton T. Morrison
The University of Arizona
Tucson, AZ, USA
<EMAIL_ADDRESS>
###### Abstract

Learning to generate textures for a novel 3D mesh given a collection of 3D
meshes and real-world 2D images is an important problem with applications in
various domains such as 3D simulation, augmented and virtual reality, gaming,
architecture, and design. Existing solutions either do not produce high-
quality textures or deform the original high-resolution input mesh topology
into a regular grid to make this generation easier but also lose the original
mesh topology. In this paper, we present a novel framework called the
3DTextureTransformer that enables us to generate high-quality textures without
deforming the original, high-resolution input mesh. Our solution, a hybrid of
geometric deep learning and StyleGAN-like architecture, is flexible enough to
work on arbitrary mesh topologies and also easily extensible to texture
generation for point cloud representations. Our solution employs a message-
passing framework in 3D in conjunction with a StyleGAN-like architecture for
3D texture generation. The architecture achieves state-of-the-art performance
among a class of solutions that can learn from a collection of 3D geometry and
real-world 2D images while working with any arbitrary mesh topology.

##  1 Introduction

Texture generation for a given 3D mesh is an extremely important task with its
applications in multiple domains like 3D simulation, augmented and virtual
reality, gaming, design, and architecture. Although the field of 3D mesh
generation from a text prompt and a single image is developing quite rapidly
[48, 36, 24, 46, 28, 5, 44, 26, 39, 22], there has been less activity in
sampling new textures for these generated meshes. However, sampling new
textures for these 3D meshes allows the user to make the scene realistic. It
allows using differently textured meshes based on the context while also
saving a lot of manual work required for manual texture generation. Recently,
GGAN [8] and Texturify [42] propose a framework based on Generative
Adversarial Networks (GANs) [14] and Differentiable Rendering [21, 27, 40] for
learning to generate textures on 3D meshes from a collection of 2D images.
While GGAN works for arbitrary 3D mesh topology, it is not powerful enough for
high-quality texture generation and Texturify deforms the original high-
quality input mesh losing some important details and thus changes the topology
of the input 3D mesh [53]. In this paper, we present a framework that builds
upon GGAN and Texturify while removing their limitations, and has the
following properties:

  * •

The input mesh does not require any changes to the mesh topology or
resolution.

  * •

It does not require ground truth textured 3D meshes, unlike [53] and also does
not require expensive 3D part segmentation unlike [13, 35].

  * •

The model is inspired by the Perceiver [17] architecture.

  * •

It builds a Unet-like architecture based solely on the geometric information
and thus is easily adaptable to newer datasets, unlike Texturify, which
requires complicated data preprocessing.

  * •

It is easily extensible to point clouds and Gaussian splats [22], while easily
scaling to larger meshes and also works for both triangular and quad meshes.

As we demonstrate, the framework is a general, powerful and flexible
generative model for 3D data. The code is made available open source.

##  2 Related Work

We aim to generate novel, high-quality textures for a previously unseen 3D
mesh model. The following research is related to our model.

###  2.1 2D image generation

One line of research related to our problem is the generation of 2D images
that are aware of 3D geometry, such as EG3D [2], GRAF [41], GIRAFFE [30], pi-
GAN [3], and GRAM [7]. These approaches are mainly focused on generating
multiview consistent 2D images that respect the underlying 3D geometry of the
image as opposed to texture generation for a given 3D mesh model.

###  2.2 3D texture generation

Texture Fields [31] generate a texture for a 3D geometry based on 2D image
input by regressing a continuous 3D function parameterized by a neural
network. Our work is slightly different than this setting because we want to
sample new textures for a given 3D mesh model without being conditioned on the
input 2D image during inference. Another closely related model is LTG [52],
but this approach requires an explicit UV mapping that distorts the regions
near the seams. Text2Mesh [29] optimizes both the geometry and texture for a
single 3D mesh instance based on a text prompt rather than generating novel
textures for a given 3D mesh model. Finally, our work is closely related to
GGAN [8] and Texturify [42]; below, in Section 3, we explain how we extend
these architectures while removing their limitations.

###  2.3 GNNs

Graph neural networks (GNNs) are powerful models for learning from graph-
structured data. Our solution first converts the input 3D mesh into a
geometric graph and learns node relationships using a message-passing
framework where a node gets some information from its neighbors and updates
its state. Consider a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where
$\mathcal{V}$ is the set of nodes and $\mathcal{E}$ is the set of edges. The
$k^{th}$ message passing iteration of a GNN is given by [15]:

$\begin{split}h_{v}^{(k+1)}&=\texttt{update}^{(k)}(h_{v}^{(k)},\texttt{aggregate}^{(k)}(h_{u}^{(k)}),\\\
\
&=\texttt{update}^{(k)}(h_{v}^{(k)},\boldsymbol{m}_{\mathcal{N}(v)}^{(k)})\quad\forall
u\in\mathcal{N}(v))\end{split}$ (1)

Our solution uses a multi-headed self-attention among its neighbors only to
aggregate the features and thus utilizes the sparsity of the adjacency matrix
to make the solution scalable to larger graphs.

###  2.4 Graph Pooling and Unpooling

We treat input mesh as a geometric graph and develop a Unet-like architecture.
Graph pooling and unpooling operations allow us to develop powerful,
efficient, and flexible Unet-like architecture. Graph pooling allows us to
look at the input mesh at various hierarchical scales by iteratively
coarsening the graph into a new graph of smaller size. There are mainly two
types of hierarchical graph pooling operations [25]:

  * •

Node Clustering Pooling: This solution considers graph pooling as a node
clustering problem and learns a cluster assignment matrix that maps nodes from
a higher level to a lower level as:

$\displaystyle\begin{split}C&=f(X^{l},A^{l})\\\ X^{{}^{\prime}}&=C^{T}X\\\
A^{{}^{\prime}}&=C^{T}AC\end{split}$ (2)

Where $C\in\mathbf{R^{n_{l}*n_{l+1}}}$ is a cluster assignment matrix and $f$
can be any function such as $softmax(GNN(X^{l},A^{l}))$ [51]. Although
powerful, node clustering pooling operations are computationally expensive for
larger graphs.

  * •

Node Drop Pooling: Node drop pooling either uses a scoring function to select
nodes or uses a sampling based strategy for node selection. Once these
important nodes are selected, information from neighboring nodes can be used
for pooling operations. The operations can be summarized as follows [25]:

$\displaystyle\begin{split}S&=f_{score}(X^{l},A^{l})\\\ idx&=f_{select}(S)\\\
X^{{}^{\prime}},A^{{}^{\prime}}&=f_{coarsen}(X^{l},A^{l},S,idx)\end{split}$
(3)

Our geometric graph encoder uses graph pooling operations based on these node
drop pooling operations as they are not computationally expensive thus making
our solution scalable to larger meshes.

Specifically, we explore various graph pooling and unpooling operations such
as sampling and max pooling [37, 38], voxel-based pooling and unpooling [49],
Graph-Unets [12], and EdgePooling [9].

Figure 1: 3DTextureTransformer Overall Pipeline

###  2.5 GANs and self-attention

Generative adversarial networks (GANs) [14] are known for their ability to
generate photorealistic images with very high resolution [20]. The GAN
framework consists of a generator $\mathcal{G}$ and discriminator
$\mathcal{D}$. The generator tries to fool the discriminator by generating
realistic samples while the discriminator tries to predict real samples from
fake samples (generated by the generator). This constitutes a two-player
minimax game with the following value function:

$\begin{split}\mathcal{V}(\mathcal{G},\mathcal{D})&=\mathbb{E}_{\boldsymbol{x}\sim
p_{data}(\boldsymbol{x})}[logD(\boldsymbol{x})]\\\ &+\mathbb{E}_{z\sim
p_{z}(\boldsymbol{z})}[log(1-\mathcal{D}(\mathcal{G}(\boldsymbol{z})))]\end{split}$
(4)

Here, $\boldsymbol{x}$ denotes a sample from a real distribution, $p_{data}$,
and $\boldsymbol{z}$ denotes a “noise” vector from distribution $p_{z}$.
StyleGAN [20, 18] is a state-of-the-art architecture for high-quality GAN
based image synthesis with disentangled latent space. The StyleGAN
architecture works perfectly with convolutional layers. As convolution
requires a regular connectivity pattern, the architecture is limited to the
case where we have a regular structure such as 2D images. Given that self-
attention layers can handle arbitrary neighborhood structures and are able to
handle long-range dependencies, there have been multiple attempts to combine
self-attention layers with StyleGAN-like architectures. Recently, StyleFormer
[32] and StyleSwin [54] use self-attention based generator architecture for 2D
images. Our 3DTextureTransformer is a hybrid model that works with 3D data by
utilizing self-attention layers with a message-passing framework in
combination with StyleGAN-like architecture.

##  3 Methodology

The above figure 1 shows the overall diagram of our system.

Our system learns to generate textures for a 3D mesh model from a collection
of 3D mesh geometry and a collection of real-world 2D images. We use the same
discriminator architecture as used by StyleGAN [18] and in the following
section, we describe the geometric graph encoder and a generator called
3DTextureTransformer.

Figure 2: 3DTextureTransformer Generator

###  3.1 Geometric Graph Encoder

Our geometric graph encoder utilizes node drop pooling • ‣ 2.4 operations to
coarsen the geometric graph and its inverse operations for unpooling. We treat
the centroid of a face in a mesh as a node in the geometric graph. We evaluate
multiple strategies for pooling and unpooling the geometric graphs as follows:

  * •

FPSPooling: It uses farthest point sampling as a score function to select
nodes in the given geometric graph [38]. Pooling is performed using max pool
among its k nearest neighbors and unpooling is performed using interpolation
among its k nearest neighbors [38] as follows:

$\displaystyle\begin{split}f(y)&=\frac{\sum_{i=1}^{k}w(x_{i})f(x_{i})}{\sum_{i=1}^{k}w(x_{i})}\\\
w(x_{i})&=\frac{1}{d(p(y),p(x_{i}))^{2}}\end{split}$ (5)

Where d is the Euclidean distance, p(x) refers to its position, and f(x)
refers to its features.

  * •

VoxelPooling: In voxel based pooling [43], the 3D space is divided into
regular grid structure called voxels. The faces of the triangular mesh that
are within the same voxel are max pooled. For unpooling, we use the
interpolation of all the points that lie within a given voxel.

  * •

EdgePooling: EdgePooling contracts two pairs of nodes into a single node [9].
For our 3D mesh, this is equivalent to combining two faces into a single face.
The contraction score is calculated for each edge as follows:

$r(e_{ij}=W(n_{i}||n_{j})+b$

Where $n_{i}$ and $n_{j}$ are node features, and $W$ and $b$ are learnable
parameters.

  * •

GraphUnet: GraphUnet [12] uses a learnable score function to select nodes in
the graph. Once the nodes are selected, top-k nodes are selected and the
information is aggregated from their neighbors.

We evaluate the effectiveness of each approach for geometric graph encoding
when combined with StyleGAN-like architecture for 3D texture generation.

(a)

(b)

(c)

(d)

(e)

Figure 3: Textures generated by our solution on Chair Dataset Method | Parameterization | FID $\downarrow$ | KID * $10^{-}2$ $\downarrow$
---|---|---|---
Texture Fields [31] | Global Implicit | 85.01 | 6.06
SPSG [6] | Sparse 3D Grid | 65.36 | 5.13
UV Baseline [42] | UV | 38.98 | 2.46
LTG [52] | UV | 37.50 | 2.39
EG3D [52] | Tri-plane Implicit | 36.45 | 2.15
Ours | Sparse 3D implicit | 33.87 | 1.93
Table 1: Results

###  3.2 3DTextureTransformer

The above diagram 2 shows our generator architecture. We use mapping layers to
map latent code ($z\in\mathcal{Z}$) into smooth latent space
($w\in\mathcal{W}$). $A$ refers to the affine transformation of these smooth
latent codes. Our network applies Adaptive instance normalization (AdaIN)
operation on the features as follows [19]:

$AdaIN(x_{i},y)=y_{s,i}*\frac{(x_{i}-\mathbb{E}{(x_{i})})}{\sigma(x_{i})}+y_{b,i}$

Where $s_{i}$ and $b_{i}$ are scaling and bias derived using affine
transformation $A$ of the smooth latent space $w\in\mathcal{W}$ and are
applied to $i^{th}$ input channel($x_{i}$). For toRGB layer, we use the weight
modulation and demodulation operation as follows [20]:

$\displaystyle w_{ij}^{{}^{\prime}}$ $\displaystyle=s_{i}*w_{ij}$ (6)

where $w$ and $w^{{}^{\prime}}$ are original and modulated weights and $s_{i}$
is the scaling factor for $i^{th}$ input feature channel and $j$ enumerates
the output feature maps. The demodulation is done using:

$\displaystyle w_{ij}^{{}^{\prime\prime}}$
$\displaystyle=\frac{w_{ij}^{{}^{\prime}}}{\sqrt{\sum_{i}{w_{ij}^{{}^{\prime}}}^{2}}+\epsilon}$
(7)

The multi-headed local self-attention is calculated using the following sparse
multi-headed operation [10] making the solution scalable to larger graphs:

$SparseAttn(Q,K,V)=softmax(\frac{QK^{T}*A}{\sqrt{d}})V$

Where Q, K, and V are Query, Key, and Value obtained from the linear
transformation of input signal [45] and $*$ denotes a Hadamard product with
the sparse adjacency matrix $A$.

The graph features encoded from the geometric graph encoder are concatenated
with the features learned using the generator block. The advantage of such an
architecture with AdaIn operations is that it allows experimentation with
novel and powerful geometric deep-learning solutions in the future.

##  4 Experiments

We experiment with our solution in the Chairs and Cars dataset [42]. This
dataset uses a collection of 3D geometry from ShapeNet [4] and a collection of
2D real-world chair images [33] and 2D real-world car images [50]. We evaluate
our solution based on Frechet Inception Distance (FID) [16] and Kernel
Inception Distance (KID) [1] and follow the same evaluation pipeline of
Texturify [42]. We use PyTorch [34], PyTorch Geometric [11], and DGL [47] for
all our experiments and Nvdiffrast [23, 42] for differentiable rendering. We
use a learning rate of 0.0002 for the generator, geometric graph encoder, and
discriminator. Once the generator generates and applies the texture to a given
mesh, it is rendered from 4 random camera viewpoints. We use positions,
normals, gaussian curvature, and mean curvature as the input features for each
face of the input 3D mesh.

##  5 Results

The above figure 3 shows some of the results from our system. The table 1
shows FID and KID values on the Chair Dataset and comparison with methods that
can work with arbitrary mesh topology. Texturify [42] does not work with
arbitrary mesh topology and EG3D produces 2D images instead of texture that
can be applied to 3D mesh.

##  6 conclusion

In this work, we have developed a hybrid architecture that can generate
texture in 3D with a combination of geometric deep learning and StyleGAN-like
architecture. Our solution uses self-attention layers in the 3D geometric
space for message passing and thus makes our solution work on arbitrary mesh
topology. Our solution uses a geometric graph encoder that is flexible enough
to work on the 3D mesh, point clouds, and Gaussian splats. Currently, we
observe that the textures generated have low frequency. In the future, we will
explore a direction to add high-frequency details to these generated textures.
And we will also explore the use of our solution for texture generation for
point clouds and Gaussian splats in the future.

## References

  * [1] M. Bińkowski, D. J. Sutherland, M. Arbel, and A. Gretton.  Demystifying mmd gans.  arXiv preprint arXiv:1801.01401, 2018.
  * [2] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  * [3] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799–5809, 2021.
  * [4] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [5] R. Chen, Y. Chen, N. Jiao, and K. Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  arXiv preprint arXiv:2303.13873, 2023.
  * [6] A. Dai, Y. Siddiqui, J. Thies, J. Valentin, and M. Nießner.  Spsg: Self-supervised photometric scene generation from rgb-d scans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1747–1756, 2021.
  * [7] Y. Deng, J. Yang, J. Xiang, and X. Tong.  Gram: Generative radiance manifolds for 3d-aware image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10673–10683, 2022.
  * [8] K. Dharma, C. T. Morrison, and B. Walls.  Texture generation using a graph generative adversarial network and differentiable rendering.  In International Conference on Image and Vision Computing New Zealand, pages 388–401. Springer, 2022.
  * [9] F. Diehl, T. Brunner, M. T. Le, and A. Knoll.  Towards graph pooling by edge contraction.  In ICML 2019 workshop on learning and reasoning with graph-structured data, 2019.
  * [10] V. P. Dwivedi and X. Bresson.  A generalization of transformer networks to graphs.  arXiv preprint arXiv:2012.09699, 2020.
  * [11] M. Fey and J. E. Lenssen.  Fast graph representation learning with pytorch geometric.  arXiv preprint arXiv:1903.02428, 2019.
  * [12] H. Gao and S. Ji.  Graph u-nets.  In international conference on machine learning, pages 2083–2092. PMLR, 2019.
  * [13] L. Gao, T. Wu, Y.-J. Yuan, M.-X. Lin, Y.-K. Lai, and H. Zhang.  Tm-net: Deep generative networks for textured meshes.  ACM Transactions on Graphics (TOG), 40(6):1–15, 2021.
  * [14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [15] W. L. Hamilton.  Graph representation learning.  Morgan & Claypool Publishers, 2020.
  * [16] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [17] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira.  Perceiver: General perception with iterative attention.  In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  * [18] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila.  Training generative adversarial networks with limited data.  Advances in neural information processing systems, 33:12104–12114, 2020.
  * [19] T. Karras, S. Laine, and T. Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [20] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [21] H. Kato, Y. Ushiku, and T. Harada.  Neural 3d mesh renderer.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3907–3916, 2018.
  * [22] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics, 42(4), 2023.
  * [23] S. Laine, J. Hellsten, T. Karras, Y. Seol, J. Lehtinen, and T. Aila.  Modular primitives for high-performance differentiable rendering.  ACM Transactions on Graphics (TOG), 39(6):1–14, 2020.
  * [24] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
  * [25] C. Liu, Y. Zhan, J. Wu, C. Li, B. Du, W. Hu, T. Liu, and D. Tao.  Graph pooling for graph neural networks: Progress, challenges, and opportunities.  arXiv preprint arXiv:2204.07321, 2022.
  * [26] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023.
  * [27] S. Liu, T. Li, W. Chen, and H. Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7708–7717, 2019.
  * [28] G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663–12673, 2023.
  * [29] O. Michel, R. Bar-On, R. Liu, S. Benaim, and R. Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13492–13502, 2022.
  * [30] M. Niemeyer and A. Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [31] M. Oechsle, L. Mescheder, M. Niemeyer, T. Strauss, and A. Geiger.  Texture fields: Learning texture representations in function space.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531–4540, 2019.
  * [32] J. Park and Y. Kim.  Styleformer: Transformer based generative adversarial networks with style vector.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8983–8992, 2022.
  * [33] K. Park, K. Rematas, A. Farhadi, and S. M. Seitz.  Photoshape: Photorealistic materials for large-scale shape collections.  arXiv preprint arXiv:1809.09761, 2018.
  * [34] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [35] D. Pavllo, J. Kohler, T. Hofmann, and A. Lucchi.  Learning generative models of textured 3d meshes from real-world images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13879–13889, 2021.
  * [36] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [37] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [38] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [39] G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y. Lee, I. Skorokhodov, P. Wonka, S. Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  arXiv preprint arXiv:2306.17843, 2023.
  * [40] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv preprint arXiv:2007.08501, 2020.
  * [41] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  Advances in Neural Information Processing Systems, 33:20154–20166, 2020.
  * [42] Y. Siddiqui, J. Thies, F. Ma, Q. Shan, M. Nießner, and A. Dai.  Texturify: Generating textures on 3d shape surfaces.  In European Conference on Computer Vision, pages 72–88. Springer, 2022.
  * [43] M. Simonovsky and N. Komodakis.  Dynamic edge-conditioned filters in convolutional neural networks on graphs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3693–3702, 2017.
  * [44] C. Tsalicoglou, F. Manhardt, A. Tonioni, M. Niemeyer, and F. Tombari.  Textmesh: Generation of realistic 3d meshes from text prompts.  arXiv preprint arXiv:2304.12439, 2023.
  * [45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [46] H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12619–12629, 2023.
  * [47] M. Y. Wang.  Deep graph library: Towards efficient and scalable deep learning on graphs.  In ICLR workshop on representation learning on graphs and manifolds, 2019.
  * [48] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [49] X. Wu, Y. Lao, L. Jiang, X. Liu, and H. Zhao.  Point transformer v2: Grouped vector attention and partition-based pooling.  Advances in Neural Information Processing Systems, 35:33330–33342, 2022.
  * [50] L. Yang, P. Luo, C. Change Loy, and X. Tang.  A large-scale car dataset for fine-grained categorization and verification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3973–3981, 2015.
  * [51] Z. Ying, J. You, C. Morris, X. Ren, W. Hamilton, and J. Leskovec.  Hierarchical graph representation learning with differentiable pooling.  Advances in neural information processing systems, 31, 2018.
  * [52] R. Yu, Y. Dong, P. Peers, and X. Tong.  Learning texture generators for 3d shape collections from internet photo sets.  In British Machine Vision Conference, 2021.
  * [53] X. Yu, P. Dai, W. Li, L. Ma, Z. Liu, and X. Qi.  Texture generation on 3d meshes with point-uv diffusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4206–4216, 2023.
  * [54] B. Zhang, S. Gu, B. Zhang, J. Bao, D. Chen, F. Wen, Y. Wang, and B. Guo.  Styleswin: Transformer-based gan for high-resolution image generation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11304–11314, 2022.


supplement

# On the Usability of Transformers-based models for a French Question-
Answering task

Oralie Cattan1,2 &Christophe Servan1 &Sophie Rosset2
1QWANT
61 rue de Villiers,
92200 Neuilly-sur-Seine, France
<EMAIL_ADDRESS>
&
2Université Paris-Saclay,
CNRS, LISN,
91405, Orsay, France
<EMAIL_ADDRESS>

###### Abstract

For many tasks, state-of-the-art results have been achieved with Transformer-
based architectures, resulting in a paradigmatic shift in practices from the
use of task-specific architectures to the fine-tuning of pre-trained language
models. The ongoing trend consists in training models with an ever-increasing
amount of data and parameters, which requires considerable resources. It leads
to a strong search to improve resource efficiency based on algorithmic and
hardware improvements evaluated only for English. This raises questions about
their usability when applied to small-scale learning problems, for which a
limited amount of training data is available, especially for under-resourced
languages tasks. The lack of appropriately sized corpora is a hindrance to
applying data-driven and transfer learning-based approaches with strong
instability cases. In this paper, we establish a state-of-the-art of the
efforts dedicated to the usability of Transformer-based models and propose to
evaluate these improvements on the question-answering performances of French
language which have few resources. We address the instability relating to data
scarcity by investigating various training strategies with data augmentation,
hyperparameters optimization and cross-lingual transfer. We also introduce a
new compact model for French FrALBERT which proves to be competitive in low-
resource settings.

##  1 Introduction

Recent advances in the field of Natural Language Processing (NLP) have been
made with the development of transfer learning and the availability of pre-
trained language models based on Transformer architectures Vaswani et al.
(2017), such as BERT Devlin et al. (2019). As they provide contextualized
semantic representation they contribute both to advance the state-of-the-art
on several NLP tasks and also to evolve training practices through the use of
fine-tuning.

The trend of recent years consists in training large pre-trained language
models on ever larger corpora, with an ever-increasing amount of parameters,
which requires considerable computational resources that only a few companies
and institutions can afford. For example, the base model of BERT with 110
million parameters was pre-trained on 16 gigabytes (GB) of text, while the
GPT-3 model Brown et al. (2020) was pre-trained on 45 terabytes (TB) of text
and has 175 billion parameters.

In fact, deploying ever-larger models raises questions and concerns about the
increasing magnitude of the temporal, financial, and environmental cost of
training and usability Strubell et al. (2019); Moosavi et al. (2020).
Typically, due to their resource requirements, these models are trained and
deployed for industrial operations on remote servers. This leads to a high use
of over-the-air communications, which are particularly resource-intensive
Gündüz et al. (2019). In particular, some NLP applications (speech
recognition, speech to text, etc.) have some known problems related to network
latency, transmission path difficulties, or privacy concerns. To reduce the
impact of these communications, there is a solution that is to allow these
models to run directly on peripheral or mobile devices, that is, in
environments with limited resources that require lightweight, responsive
models and energy efficiency. Reducing the size of the models is therefore one
of the increasingly favoured avenues, especially for the reduction of memory
resources and computation time involved in training and use.

To meet these constraints, compact models represent one of the most promising
solutions. As far as we know, they have only been evaluated on the
comprehension tasks covered by GLUE Wang et al. (2018) and the question-
answering task with the SQuAD corpus Rajpurkar et al. (2016) with abundant
data, in English. The improvements resulting from the algorithmic
optimizations of the models, although significant, raise questions about their
effectiveness on lower-scale learning problems on poorly endowed languages.
The works of Zhang et al. (2021) and Mosbach et al. (2021) have furthermore
shown degraded performance in these conditions. These two reflections are at
the origin of a double question which our contribution attempts to answer. On
the one hand, what is the behavior of a Transformer-based model in the context
of a question-answering task in French, a task that is poorly endowed in this
language? On the other hand, what are the impacts of algorithmic improvements
of these same models in this context?

To answer these questions, we first establish in section 2 a state-of-the-art
that is meant to be broad enough to have a shallow overview depicting the ins
and outs and issues around the usability of Transformer-based models whose
breadcrumb trail is the issue of resources. Then, we present in the section 3
the recent progress of the question-answering task, through the use of these
latest models. In sections 4 and 5 we introduce our model and present our
experiments on the usability of Transformers models in a question-answering
task for French, on FQuAD d’Hoffschmidt et al. (2020) and PIAF Keraron et al.
(2020) corpora. We propose to address the instability relating to data
scarcity by investigating various training strategies with data augmentation,
hyperparameters optimization and cross-lingual transfer. Finally, we present a
new compact model for French based on ALBERT Lan et al. (2020)111Available at
HuggingFace’s model hub page., and compare it to existing monolingual and
multilingual models, large and compact, under constrained conditions (notably
on learning data).

##  2 Usability of Transformers

In this section we present the ins and out of the Transformer models to
understand how the approaches meet the need for better usability.

###  2.1 Architecture and pre-trained models

The Transformer architecture Vaswani et al. (2017) is based on a stack of
encoder-decoder blocks, composed at a high level of forward propagation
networks and multi-headed self-attention operations. The self-attention layer
is the core element of its architecture that enables its efficiency in
modeling the semantic context interdependencies between the units or sub-units
of the input sequence.

Transformer-based language models such as BERT Devlin et al. (2019) are pre-
trained on large-scale data collections sourced from Wikipedia or Common Crawl
(CC) with one or multiple training objectives (masked language modeling, next
sentence or sentence order prediction). This pre-training can be followed by
supervised fine-tuning according to the tasks, whether generatives (machine
translation, abstractive summarization) or discriminatives (classification,
question-answering). The ensuing fine-tuning phase allows for better
initialization of the models parameters while requiring less task-specific
data so as to make the training of subsequent tasks faster.

Recently, Zhang et al. (2021) and Mosbach et al. (2021) have nevertheless
shown that the commonly adopted practices (the number of iterations, the
choice of model layers) when fine-tuning Transformers-based langage models are
inappropriate under resource constrained conditions and adversely affect the
stability of models performances as overfitting, label noise memorization or
catastrophic forgetting. Added to this, because the pre-training process is
particularly constraining, various works have been oriented towards the
research and training of efficient models, both in terms of available
capacities and resources and in terms of environmental footprint.

###  2.2 A search for efficiency

Reducing the cost of training Transformers-based models has become an active
research area. To this end, methods based on compression techniques or on
architecture improvements have been introduced in order to build compact
models with comparable performances to large models.

Many works address the issue of model compression with quantization, pruning,
knowledge distillation or a combination of these approaches. The idea of
quantization Shen et al. (2020) is to take advantage of the use of lower
precision bit-width floats to reduce memory usage and increase computational
density. Following the same objective, pruning Michel et al. (2019) consists
in removing parts of a model (weight bindings, attentional heads) with minimal
precision losses. Finally, knowledge distillation Sanh et al. (2019) enables
the generation of models that mimic the performance of a large model (or set
of models) while having fewer parameters.

Another axis of development concerns the use of neural architecture search
Elsken et al. (2019) which allows to optimize a model by progressively
modifying the design of the network through trial and error, eliminating
insignificant operations. To avoid the unnecessary large number of parameters,
adapters Houlsby et al. (2019) were introduced to allow fine-tuning of the set
of parameters specific to the task of interest rather than the entire model.

Other architectural improvements highlighted with the introduction of the
ALBERT model Lan et al. (2020) such as the factorization of the attention
matrix or parameter sharing. Indeed, the most time-consuming and memory-
intensive operations concerns the forward propagation and attention
computation operations. The self-attention layer of BERT pretrained models
grows quadratically in respect to the input sequence length. One common
approach to this issue consists of approximating the dot-product attention for
example by using hashing techniques Kitaev et al. (2020) to accelerate the
training and inference phases when long sequence lengths are used. However
these solutions have demonstrated they suffer from important computational
overheads for tasks with smaller lengths, such as question-answering.

##  3 The Question-Answering task

Question-Answering (QA) based on machine reading comprehension corresponds to
the task of extracting an answer given a question and a context document such
as from a news or Wikipedia article.

###  3.1 General QA Architecture

Until recently, most of the proposed approaches have relied on an
architectural complexification of LSTM-based neural networks and attention
mechanism. At a high level, their architectures are all composed of three
layers with:

  1. (a)

an encoding layer that projects the inputs, as each word within the context,
question and answer triples in a latent semantic space;

  2. (b)

an interaction layer that models the semantic interdependencies between the
embedded inputs through the use of attention mechanisms.

  3. (c)

an output layer that extracts the answer to the input question within the
related context.

The interaction layer is the core element of the architecture for which
several kinds attention mechanisms has been developed to improve the QA
matching process such as bi-attention Seo et al. (2017), co-attention Xiong et
al. (2017, 2018), multi-level inter-attention Huang et al. (2018) or re-
attention Hu et al. (2018), to name just a few.

Recent advances through the availability of Transformer-based pre-trained
models and the development of transfer learning methods have enabled to remove
the recurrence of previous architectures in order to achieve parallelization
efficiencies. This simplified the QA architecture and its training process,
replacing the encoding and the interaction layers with attention-based
Transformer layers.

Another advantage is that it provides pre-computed contextual word
representations. QA models based on LSTMs are built on top of static word
embeddings models such as GloVe Pennington et al. (2014). Even these models
have up to 40 times fewer parameters than a BERT-based model, they rely on
LSTM-based encoders to produce contextual embeddings which considerably
lengthens the time required for training and makes the dependence on
supervised data more important.

The standard approach introduced by Devlin et al. (2019) we rely on this
study, consists in introducing and updating parameter vectors corresponding to
the start and end positions of the answer span. Specifically, the start and
end position probability distributions are computed by softmax over the dot
products between the representation of the tokens and the start and end
vectors. In sum, all of the Transformer parameters as well as the two
introduced parameter vectors are optimized together.

Despite the fact that there is a number of large-scale QA datasets in English,
with tens of thousands of annotated training examples, porting a system to a
new language with fewer annotated resources (low-resource languages) requires
approaches that go far beyond the simple act of retraining the models.

###  3.2 Low-Resourced QA

In recent years, low-resource NLP has drawn an increasing amount of attention
with solutions ranging from developing new data collection methodologies
either via crowdsourcing or through the use of machine translation (MT), to
cross-lingual and transfer learning approaches for which information is shared
across languages or tasks.

####  3.2.1 MT-based data collection

Neural MT as made considerable progress in recent years such as translating
large-scale datasets from a high-resourced to under-resourced languages or
converserly has become an intuitive way of generating annotated datasets in a
cost-effective and rapid manner.

Automatically translating the context, question and answer triples from a
high-resource language, such as English (called source domain) to low-resource
languages (called target domains) have enabled the evaluation of models for
languages with no training data available but also the creation of large-scale
MT-based QA corpora for the Italian Croce et al. (2018), Spanish Carrino et
al. (2020), Arabic Mozannar et al. (2019) and Korean Youngmin Kim (2020)
languages.

Another approach consists of translating the QA triples of the target domain
into the source domain, so the model trained on the source language can be
directly applied on the translated target language testing data. As an
exemple, Asai et al. (2018)’s method consisted of combining the alignment
attention scores from a MT model with an English QA model to guide the answer
extraction process.

The performance of MT-data approaches depends strongly on the quality of the
MT models. Thus, due to the lack of reliable models for some language pairs,
approaches that foster the transfer of knowledge from other languages or tasks
while requiring less data have been developed.

####  3.2.2 Pre-training and Transfer approaches

The exploitation of pre-trained models followed by task-specific fine-tuning
haved pushed the state-of-the-art forwards, while requiring much less
computational and data resources. The idea behind pre-training is to reuse the
weights parameters trained on a set of source tasks and continue to fine-tune
them on under-resourced target tasks to achieve knowledge transfer. Dai and Le
(2015) were the first to propose to pre-train RNNs using auto-encoders and
language models as part of their QA encoding layer. Min et al. (2017) and
Wiese et al. (2017) pre-trained QA models before applying the fine-tuning
process between the source and the target domains. Other efforts focused on
pre-training Transformer-based models multilingually such as the multilingual
version of BERT (called mBERT) Devlin et al. (2019) or XLM-R Conneau et al.
(2020) to learn cross-lingual representations which are transferable across
languages.

####  3.2.3 Usability concerns

Studies on the usability of Transformer-based models (Section 2) from standard
resource efficiency concerns towards a broader set of problems related to
their generalizability.

Recently, Pires et al. (2019) and Conneau et al. (2020) have shown that
multilingual models underperformed, when applied on poorly endowed languages.
Additionally, as mentioned in subsection 2.1, recent works Zhang et al.
(2021); Mosbach et al. (2021) have highlighted the limitation of Transformer-
based transfer learning with strong instabilities arising from the small-scale
learning.

French is a poorly endowed language since we do not have enough annotated data
to train a deep learning model on QA tasks. Moreover, unlike the only two
large monolingual French models: CamemBERT Martin et al. (2020) and FlauBERT
Le et al. (2020), the English BERT model has become a branching point from
which a growing number of large and compact English pre-trained models have
emerged. These French monolingual models, although they provide good
performances, do not reflect the rapid evolution of the field.

Consequently, in this paper we propose a new compact model FrALBERT for French
we present in the following section alongside with the other available pre-
trained language models on which we base our experiments.

##  4 FrALBERT and Transformer models considered

As mentioned in the previous section there is no compact model for French. We
therefore decided to pre-train a new version of ALBERT from scratch we called
FrALBERT, thus overcoming some of the discussed limitations.

ALBERT is based on parameter sharing/reduction techniques that allows to
reduce the computational complexity and speed up training and inference
phases. Compared to previous compact models such as DistilBERT Sanh et al.
(2019), Q-BERT Shen et al. (2020) or TernaryBERT Zhang et al. (2020), ALBERT
is to the date the smallest pre-trained models with 12 million parameters and
<50 megabyte (MB) model size.

FrALBERT is pre-trained on the French version of the Wikipedia encyclopedia of
04/05/2021, i.e. 4 GB of text and 17 million (M) sentences. Beyond concerns
about the rights to use data from Common Crawl projects such as OSCAR Ortiz
Suárez et al. (2020) or CCNet Wenzek et al. (2020) corpora, and because we
focus on factual QA, we decide to use only Wikipedia as our primary source of
knowledge. We used the same learning configuration as the original model with
a batch size of $128$ and a initial learning rate set to $3.125\times
10\textsuperscript{-4}$.

Our experiments are also based on the large monolingual French model CamemBERT
Martin et al. (2020) as well as on the two large multilingual models: XLM-R
Conneau et al. (2020) and mBERT Devlin et al. (2019), both pre-trained from
massive corpora dataset in more than 100 languages such as the Common Crawl
(CC-100) or Wikipedia (Wiki-100). We also exploit two compact multilingual
models with a distilled version of mBERT: distil-mBERT Sanh et al. (2019) and
small-mBERT Abdaoui et al. (2020), a mBERT model whose the original vocabulary
has been reduced to two languages (English and French). Table 1 gives a
comparison of the models.

model | pre-training data | vocab. size | # param. | model size
---|---|---|---|---
CamemBERTbase | French OSCAR (138 GB of text) | 32005 | 110 M | 445 MB
CamemBERTlarge | French CCNet (135 GB of text) | 32005 | 335 M | 1.35 GB
CamemBERTbase | French Wikipedia (4 GB of text) | 32005 | 110 M | 445 MB
FrALBERTbase | French Wikipedia (4 GB of text) | 32005 | 12 M | 50 MB
XLM-Rbase | CC-100 (2.5 TB of text) | 250002 | 278 M | 1.12 GB
XLM-Rlarge | CC-100 (2.5 TB of text) | 250002 | 559 M | 1.24 GB
mBERTbase | Wiki-100 | 119547 | 177 M | 714 MB
small-mBERTbase | Wiki-100 | 33407 | 111 M | 447 MB
distil-mBERTbase | Wiki-100 | 119547 | 134 M | 542 MB

Table 1: Characteristics of the pre-trained models used in the proposed small-
scale QA framework.

dataset | SQuAD-entrain | FQuADtrain | FQuADdev | PIAFdev
---|---|---|---|---
# titles | 442 | 117 | 18 | 191
# paragraphs | 18,891 | 4,920 | 768 | 761
# sentences / # tokens / #characters | 5.0 / 119.7 / 635.6 | 4.8 / 125.9 / 653.8 | 5.4 / 147.3 / 765.7 | 4.2 / 110.5 / 579.1
# questions | 87,342 | 20,088 | 3,184 | 3,812
# tokens / # characters | 10.1 / 50.2 | 9.2 / 47.9 | 8.5 / 45.6 | 9.1 / 48.2
# answers | 87,599 | 20,731 | 3,188 | 3,835
# tokens / # characters | 2.9 / 16.8 | 3.6 - 19.5 | 4.3 - 23.0 | 4.5 - 24.1
human performance (F1 / EM) | 90.5 / 80.3 | 92.6 / 79.5 | -

Table 2: Descriptives of SQuAD-en, FQuAD and PIAF datasets.

##  5 Experiments

We propose an assessment of the comparative advantage gains in performance
when using different training strategies (data augmentation, hyperparameter
search and cross-lingual transfer) over monolingual and multilingual pre-
trained models, larges and compacts for a QA task in French under resource
constraints.

###  5.1 QA Datasets

We conduct experiments on four QA datasets whose descriptives are presented in
Table 2, with:

  * •

SQuAD (v1.1) Rajpurkar et al. (2016) (we called SQuAD-en) the reference corpus
to evaluate QA models’ performances in English, consisting of 100K+ QA pairs
sourced from 442 English Wikipedia articles;

  * •

FQuAD (v1.0) d’Hoffschmidt et al. (2020), a recently released French QA
dataset consisting of 25K+ crowdsourced QA pairs based on 135 articles on
French Wikipedia;

  * •

PIAF (v1.0) Keraron et al. (2020), a small-scale dataset in French with only
3K+ pairs of QA pairs in 191 Wikipedia articles; and

  * •

SQuAD-frtrain, our French translated version of SQuAD-en. We used the
Transformer architecture as described in Vaswani et al. (2017) from the Open
NMT framework Klein et al. (2017) (Open-Source Neural Machine Translation)
implementation of the network to train our neural MT system. When translating
QA corpora, the problem we face is that the translated answer may not be
present in the translated context. Thus, simple techniques such as segment
matching are inadequate to retrieve the answer. We have developed an answer
extraction process that is based on ChrF Popović (2015) a character n-gram
precision and recall enhanced with word n-grams. Since answers are largely
made up of entities, ChrF score integration is only performed when the answer
span is not present in the related context. In order to evaluate the quality
of the translation, we manually corrected the translation errors in the output
of a subset of the corpus composed of 890 QA pairs and 107 contexts. We obtain
a BLEU score Papineni et al. (2002) of 68.89 and 72.38 for questions and
contexts respectively. SQuAD-frtrain serves as a means of data augmentation on
FQuAD and PIAF benchmarks, with 90K+ translated QA training pairs222We share
our SQuAD-fr corpus on request and on dataset sharing platforms to support
further research in this area..

We also explore mixed datasets training strategy with SQuAD-entrain $+$
FQuADtrain for training models on a concatenation of the training data
covering French-English language pairs to test the cross-lingual transfer
ability of multilingual models.

###  5.2 Evaluation and validation

The performance of QA models are evaluated using the Exact Match (EM) and F1
scores. The EM score is the percentage of system outputs that match exactly
with the ground truth answers. The F1 score is a combined measure of precision
and recall that is less strict than EM. The evaluation process333The
experiments reported in d’Hoffschmidt et al. (2020) concern version 1.1 of the
FQuAD corpus. Ours are based on the only version available to date (v1.0).
Moreover, the test sets are not made public, so we use the development set
instead. involves post-processing identical to that presented by d’Hoffschmidt
et al. (2020) and inspired by that proposed for English by Rajpurkar et al.
(2016), which consists of the removal of punctuation marks and
determiners444Determiners are le, la, les, l’, du, des, au, aux, un, une. as
well as a down-casing of the answers (ground truths and predictions).

To address our considerations related to resource constraints we perform a
hyperparameter optimization, that has proven to lead to better solutions in
less time. It is based on a population-based learning Jaderberg et al. (2017)
in which a population of models and their hyperparameters are jointly
optimized. To this end, we build a validation set by randomly extracting 10%
of the training data.

testing data | FQuADdev | PIAFdev
---|---|---
model \ training strategy |  | FQuADtrain
---
| FQuADtrain
---
w/ optim.
| FQuADtrain
---
$+$ SQuAD-frtrain
| FQuADtrain
---
| FQuADtrain
---
w/ optim.
| FQuADtrain
---
$+$ SQuAD-frtrain
CamemBERTbase | 77.6 / 52.5 | 85.5 / 70.3 | 86.7 / 71.7 | 62.0 / 37.5 | 63.8 / 38.9 | 64.3 / 39.2
CamemBERTlarge | 81.2 / 55.9 | 90.2 / 75.5 | 89.9 / 75.2 | 68.1 / 42.2 | 71.0 / 44.8 | 68.9 / 42.5
CamemBERTbase (wiki 4 GB) | 74.2 / 49.5 | 80.7 / 61.8 | 85.1 / 69.5 | 61.7 / 37.3 | 62.9 / 37.9 | 65.9 / 41.0
FrALBERTbase (wiki 4 GB) | 72.6 / 55.1 | 75.6 / 64.8 | 84.3 / 70.5 | 61.0 / 38.9 | 62.1 / 39.5 | 66.9 / 43.7
XLM-Rbase | 82.1 / 66.8 | 83.1 / 67.9 | 84.2 / 68.8 | 65.0 / 39.6 | 66.9 / 41.2 | 68.6 / 42.7
XLM-Rlarge | 86.8 / 71.5 | 89.5 / 75.8 | 87.3 / 72.5 | 70.4 / 43.8 | 73.2 / 45.8 | 72.6 / 45.2
mBERTbase | 78.6 / 61.8 | 82.5 / 65.7 | 84.1 / 68.6 | 62.5 / 37.8 | 64.1 / 38.0 | 64.8 / 40.0
small-mBERTbase | 75.1 / 55.7 | 78.0 / 62.2 | 81.6 / 64.6 | 60.8 / 35.6 | 62.2 / 37.7 | 63.7 / 39.8
distil-mBERTbase | 72.8 / 56.0 | 73.0 / 55.1 | 78.1 / 61.5 | 52.3 / 30.1 | 53.6 / 31.4 | 58.3 / 34.9

Table 3: Results obtained with French and multilingual Transformer models on
the baseline training (FQuADtrain), using hyperparameter optimization
(FQuADtrain w/ optim) and with data augmentation (FQuADtrain $+$ SQuAD-
frtrain) using F1-measure (F1) and Exact Match (EM), on two French QA tasks
(FQuADdev and PIAFdev).

###  5.3 Results

Table 3 presents the results on the French QA task evaluated on FQuADdev and
PIAFdev. This table shows the scores obtained with Transformer-based models on
the baseline training (FQuADtrain), using hyperparameter optimization approach
(FQuADtrain w/ optim) and with data augmentation approach (FQuADtrain $+$
SQuAD-frtrain). Cross-lingual performances are presented in table 4, on the
same French QA tasks FQuAD and PIAF as baseline the English corpus SQuAD-
entrain, on which we applied hyperparameter optimization (SQuAD-entrain w/
optim) and performed data augmentation by adding the French corpus FQuAD to
the English QA training corpus (SQuAD-entrain $+$ FQuADtrain).

####  5.3.1 Baseline results

The results obtained from monolingual models with CamemBERTlarge which has
more layers, hidden units and attention heads and Camembertbase, both pre-
trained with a larger and more diverse amount of data achieve results are
better than Camembertbase pre-trained on only 4 GB Wikipedia. The highest F1
score is 81.2 on FQuADdev and 68.1 on PIAFdev.

The F1 performances of the FrALBERTbase model are close to those of the
CamemBERTbase model, both pre-trained on the French content of Wikipedia
(4GB). Their results turn out to be competitive and of the same order of
magnitude as those reported by Lan et al. (2020) on SQuAD-en with 1 point
difference on F1 scores when evaluating a BERTbase model (90.4 F1) and a
compact ALBERTbase model (89.3 F1) pre-trained on the same texts (BookCorpus
and Wikipedia). Interestingly, these EM scores are higher than those of the
CamemBERTbase achieving the EM score of 55.1, an increase of 5 points on the
FQuADdev.

####  5.3.2 hyperparameter results

Automatically tuning the hyperparameter tends to make QA models more accurate
with gains in terms of EM scores that are very expressive. Highest F1 / EM
scores are 90.2 / 75.5 on FQuADdev and 71.0 / 44.8 on PIAFdev. Improvement are
variable accoring the model considered, especially the French BERT one wich
have the highest improvement using this approach (from 6 to 9 F1 points and
from 11 to 20 EM points) which is quite imrpessive. FrALBERT stay behind of 5
F1 points of the CamemBERTbase trained with the same data (wiki 4 GB) but
regarding the EM scores, FrALBERT is better of 3 points. Surprisingly,
multilingual models are close to the French BERT models. The small-mBERT is
better than FrALBERT around 2.5 F1 points, while the French one have a better
EM score (+2.6 points), while distil-mBERT is lower in both F1 and EM scores.

####  5.3.3 Data augmentation results

Training strategies based on data augmentation got nearly the best results in
both F1 and EM scores except for the CamemBERTlarge. Apart from CamemBERTbase
(wiki 4 GB) and FrALBERTbase models, results are comparable with an average
difference of 1 point regardless of the metric. More generally, the
performance gains are up to 11 and 20 of F1 and EM points, respectively, on
FQuADdev and up to 4 points on both metrics on PIAFdev.

testing data | FQuADdev | PIAFdev
---|---|---
model \ training strategy |  | SQuAD-entrain
---
| SQuAD-entrain
---
w/ optim.
| SQuAD-entrain
---
$+$ FQuADtrain
| SQuAD-entrain
---
| SQuAD-entrain
---
w/ optim.
| SQuAD-entrain
---
$+$ FQuADtrain
XLM-Rbase | 81.3 / 65.0 | 82.5 / 66.5 | 83.6 / 67.5 | 61.4 / 37.2 | 62.7 / 38.5 | 64.9 / 39.9
XLM-Rlarge | 82.8 / 64.8 | 84.4 / 67.8 | 87.1 / 72.0 | 65.1 / 39.1 | 66.3 / 40.5 | 69.0 / 43.2
mBERTbase | 76.0 / 59.3 | 79.5 / 62.3 | 83.5 / 67.6 | 61.6 / 37.2 | 62.1 / 36.9 | 64.5 / 39.6
small-mBERTbase | 73.1 / 49.0 | 76.0 / 59.1 | 81.4 / 62.1 | 59.6 / 36.5 | 61.0 / 37.8 | 63.0 / 38.9
distil-mBERTbase | 65.4 / 47.4 | 68.6 / 48.5 | 75.9 / 56.3 | 48.8 / 28.1 | 52.0 / 29.2 | 56.5 / 33.1

Table 4: Cross-language transfer results obtained with multilingual
Transformer models only on the baseline (SQuAD-entrain), using hyperparameter
optimization (SQuAD-entrain w/ optim) and with data augmentation (SQuAD-
entrain $+$ FQuADtrain) using F1-measure (F1) and Exact Match (EM), on two
French QA tasks (FQuADdev and PIAFdev).

####  5.3.4 Cross-lingual transfer results

The cross-lingual transfer-based approaches using multilingual models
outperform the monolingual approaches on FQuAD and PIAF corpora (table 4).
Once again the large model XLM-R achieves better results than its base
version. XLM-R pre-trained with a dual LM objective lens scores better than
the mBERT model every time. The highest F1 score is 86.8 on FquADdev and 70.4
on PIAFdev. There is a significant performance drop between the large
multilingual models and their respective compact models. The compact
multilingual models based on mBERT substantially underperform, obtaining lower
F1 and EM scores than the large models regardless of the training strategy. In
the zero-shot configurations where no French data is used for training (SQuAD-
entrain and SQuAD-entrain w/ optim), the models confirm the outstanding
crosslingual ability with performances exceeding the performances of the
monolingual models with F1 scores with an F1 / EM scores slightly below those
obtained on FQuADtrain.

####  5.3.5 General observations

In all configurations, the performance in terms of EM and F1 on PIAF remains
significantly lower than that obtained on FQuAD since the PIAF corpus does not
include multiple responses as pointed out by d’Hoffschmidt et al. (2020).
Unsurprisingly, PIAFdev offer a more challenging evaluation set, where the
answer extraction performance are lower. Indeed, the corpus is more
diversified with questions on 191 different Wikipedia articles, whereas on
FQuADdev it only covers 18.

According results, we can confirm that data augmentation is the better way to
improve results, even if data comes from another language. We observe from
multilingual results that combining training data gives better results with
similar performance whether data are translated or not.

###  5.4 Analysis

In this section, we conduct an analysis of our results to understand what
remains as challenges for state-of-the-art models, with a focus on the
usability concerns of Transformers under resource constraints.

The success of supervised methods depends heavily on the availability of
large-scale training data. Pre-training large models on massive corpora using
unsupervised language modeling and fine-tuning the model with pre-trained
weights requires less task-specific data. Our experimental results are in line
with this, since we obtain satisfactory results with transfer learning when
large and high quality annotated data are not available. The highest F1 score
is 81.2 on FQuADdev and 68.1 on PIAFdev. Nevertheless, the performance of the
models can benefit from several training strategies.

##### Effect of MT-data augmentation

The lack of human-annotated datasets for languages other than English can be
overcome by enriching our training data with the translated version of SQuAD-
entrain. Regardless of the pre-trained model used, their performance are
competitive on FQuADdev and PIAFdev, close to human performance.

##### Effect of hyperparameter tuning

A generally unstated assumption is that pre-trained linguistic models are
under-optimized and that practices commonly adopted for the fine-tuning stage
can be detrimental to performance Zhang et al. (2021); Mosbach et al. (2021).
This is quite apparent in all settings, with better gains through
hyperparameter optimization stages. Fine-tuning CamemBERTlarge on the French
dataset yields 90.2 / 75.5 F1 / EM on the FQuAD dev set. By means of
comparison, CamemBERTlarge scores were 81.2 / 55.9 F1 / EM on the same set
with no hyperparameter tuning.

##### Crosslingual QA

Pre-training language models on the concatenation of multiple languages has
proven to be a competitive approach for cross-lingual language modeling. If
monolingual models often perform better than multilingual models, we observe
that, for comparable model sizes this is not the case in our task where the
performance gap is smaller. This gap is further reduced when the strategies
are combined. Scores of fine tuned XLM-Rbase is 82.1 / 66.8 on FQuADdev and
65.0 / 39.6 on PIAFdev.

The zero-shot experiments show that multilingual models can reach strong
performances on the task in French when the model has not encountered data of
the French language. For example, the XLM-Rbase model fine-tuned solely on
SQuAD-entrain reaches a performance on FQuaD just a few points below the
performance obtained when finetuning is performed on FQuADtrain.

Finally, our results suggest that data-driven augmentation, either by
translating datasets from high resource languages or by concatenating the
available corpora are a particularly appropriate strategy to exploit the
potential of cross-lingual transferability of models and data for improving
model performances.

##### Improvements over a small scale dataset

With resource-limited training data we obtain an average F1 score of 74.2 on
FQuADdev and 61.7 on PIAFdev when we fine-tuned FrALBERT — higher performance
than any of the compact multilingual models, but slightly below the
performance of the large monolingual models. We believe that the lower
performance of small multilingual models is not due to their lower number of
parameters, but to the usability of these models which is dependent on the
reduction technique used. This can be seen very clearly since the performance
of the FrALBERT model is close to that of the large models.

Here again, these performances can be boosted via the use of translated data
or hyperparameter search which allows us to bring the maximum performances
obtained with FrALBERTbase and CamemBERTbase pre-trained on 4 GB Wikipedia
closer in a consistent way. Their scores remain slightly below those obtained
with models pre-trained on more data suggesting limitations related to the
corpus domain of the language model.

##### Computational costs

Compact models provide alternatives to high-energy consumption models by
showing comparable performance while reducing their size and computational
complexity. Decreasing the environmental impact of NLP model training, as a
research topic, is very recent Moosavi et al. (2020). We decided to monitor
our experiments conducted on one NVIDIA V100 GPU with 16GB of memory using
experiment-impact-tracker Henderson et al. (2020). Table 5 shows the energy
consumption in kilowatt-hour (kWh), the emission intensity in grams of carbon
dioxide (g CO2), and the duration in seconds ($s$) for a fine tuning session
of 10 epochs with a batch size of 4 on FQuADtrain.

The footprint of the large versions of XLM-Rlarge and CamemBERTlarge models is
3 times more than their base versions. Their training time is also
significantly longer, over 5 hours. Finally, in terms of watt usage, carbon
emissions and training time, FrALBERT is two times less the distilled version
of BERT.

model | # param. | model size | Time (s) | Energy (kWh) | CO2 (g)
---|---|---|---|---|---
CamemBERTbase | 110 M | 445 MB | 7,207 | 1.08 | 317.87
CamemBERTlarge | 335 M | 1.35 GB | 19,445 | 3.10 | 914.27
FrALBERTbase | 12 M | 50 MB | 3,816 | 0.57 | 167.80
XLM-Rbase | 278 M | 1.12 GB | 7,676 | 1.14 | 337.70
XLM-Rlarge | 559 M | 1.24 GB | 21,137 | 3.30 | 973.29
mBERTbase | 177 M | 714 MB | 7,333 | 1.07 | 317.02
small-mBERTbase | 111 M | 447 MB | 7,190 | 1.09 | 321.42
distil-mBERTbase | 134 M | 542 MB | 6,466 | 1.06 | 314.17

Table 5: Comparison of models by computational costs on FQuADtrain

##  6 Conclusion and outlook

Recently, important progress has been made in neural language modeling using
Transformer networks. Its popularity now well established lies in its
effectiveness in modeling long-term dependencies. In this study, we have shown
that a number of significant shortcomings of usability have recently been
pointed out and that some solutions have been drawn up with compact models. We
have also overviewed how the use of Transformer-based pre-trained language
models have sparked a paradigmatic shift in question-answering training
practices from task-specific architectures to the use of transfer learning
through fine-tuning.

Comparing performances on a French question-answering task using large and
compact models provides insight into the usability of these models for under-
resourced languages. As others, we argued that large and compact models cannot
be used with limited data. Our experimental results suggest that training
strategy such as hyperparameter tuning or data augmentation can help to
alleviate the data-gathering burden, with performances close to those of a
high-resourced language such as English.

Finally, we present a new compact model for French FrALBERT (12M parameters),
which proves to be as competitive as the large monolingual model CamemBERT
(110M parameters) pre-trained on the same amount of text. In term of
computational cost, we shown this compact model is twice less greedy than the
BERTbase models. We also release a high-quality translated version of the
SQuAD corpus in French consisting of around 90K+ QA pairs.

In a future work, we aim to continue this study from a meta-learning
perspective with a model-agnostic approach generalizable to low-resource
languages. We also plan to extend our model to other languages and to evaluate
it on other NLP tasks such as named entity recognition or natural language
understanding.

## Acknowledgments

This work has been partially funded by the French National Research Agency
(ANR) through the project TextToKids (AAPG2019).

## References

  * Abdaoui et al. (2020) Amine Abdaoui, Camille Pradel, and Grégoire Sigel. 2020.  Load what you need: Smaller versions of mutililingual BERT.  In _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing_ , pages 119–123, Online. Association for Computational Linguistics.
  * Asai et al. (2018) Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018.  Multilingual extractive reading comprehension by runtime machine translation.  _CoRR_ , abs/1809.03275.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Carrino et al. (2020) Casimiro Pio Carrino, Marta R. Costa-jussà, and José A. R. Fonollosa. 2020\.  Automatic Spanish translation of SQuAD dataset for multi-lingual question answering.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 5515–5523, Marseille, France. European Language Resources Association.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Croce et al. (2018) Danilo Croce, Alexandra Zelenanska, and Roberto Basili. 2018.  Neural learning for question answering in italian.  In _AI*IA 2018 – Advances in Artificial Intelligence_ , pages 389–402, Cham. Springer International Publishing.
  * Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015.  Semi-supervised sequence learning.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * d’Hoffschmidt et al. (2020) Martin d’Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020.  FQuAD: French question answering dataset.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1193–1208, Online. Association for Computational Linguistics.
  * Elsken et al. (2019) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019.  Neural architecture search: A survey.  _Journal of Machine Learning Research_ , 20(55):1–21.
  * Gündüz et al. (2019) Deniz Gündüz, Paul de Kerret, Nicholas D. Sidiropoulos, David Gesbert, Chandra R. Murthy, and Mihaela van der Schaar. 2019.  Machine learning in the air.  _IEEE Journal on Selected Areas in Communications_ , 37(10):2184–2199.
  * Henderson et al. (2020) Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  _Journal of Machine Learning Research_ , 21(248):1–43.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp. In _ICML_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hu et al. (2018) Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2018.  Reinforced mnemonic reader for machine reading comprehension.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden_ , pages 4099–4106. ijcai.org.
  * Huang et al. (2018) Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. 2018.  Fusionnet: Fusing via fully-aware attention with application to machine comprehension.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Jaderberg et al. (2017) Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. 2017.  Population based training of neural networks.  _arXiv preprint arXiv:1711.09846_.
  * Keraron et al. (2020) Rachel Keraron, Guillaume Lancrenon, Mathilde Bras, FrÃ©dÃ©ric Allary, Gilles Moyse, Thomas Scialom, Edmundo-Pavel Soriano-Morales, and Jacopo Staiano. 2020.  Project piaf: Building a native french question-answering dataset.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 5481–5490, Marseille, France. European Language Resources Association.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Le et al. (2020) Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020.  FlauBERT: Unsupervised language model pre-training for French.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 2479–2490, Marseille, France. European Language Resources Association.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one? In _Advances in Neural Information Processing Systems_ , volume 32, pages 14014–14024. Curran Associates, Inc.
  * Min et al. (2017) Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. 2017.  Question answering through transfer learning from large fine-grained supervision data.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 510–517, Vancouver, Canada. Association for Computational Linguistics.
  * Moosavi et al. (2020) Nafise Sadat Moosavi, Angela Fan, Vered Shwartz, Goran Glavaš, Shafiq Joty, Alex Wang, and Thomas Wolf, editors. 2020.  _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing_. Association for Computational Linguistics, Online.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines.  In _International Conference on Learning Representations_.
  * Mozannar et al. (2019) Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019.  Neural Arabic question answering.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 108–118, Florence, Italy. Association for Computational Linguistics.
  * Ortiz Suárez et al. (2020) Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020.  A monolingual approach to contextualized word embeddings for mid-resource languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1703–1714, Online. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting on association for computational linguistics_ , pages 311–318. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _EMNLP_ , volume 14, pages 1532–1543.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Seo et al. (2017) Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Bidirectional attention flow for machine comprehension.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Shen et al. (2020) Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020.  Q-BERT: hessian based ultra low precision quantization of BERT.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8815–8821. AAAI Press.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 3645–3650. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Édouard Grave. 2020.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 4003–4012.
  * Wiese et al. (2017) Georg Wiese, Dirk Weissenborn, and Mariana Neves. 2017.  Neural domain adaptation for biomedical question answering.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 281–289, Vancouver, Canada. Association for Computational Linguistics.
  * Xiong et al. (2017) Caiming Xiong, Victor Zhong, and Richard Socher. 2017.  Dynamic coattention networks for question answering.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Xiong et al. (2018) Caiming Xiong, Victor Zhong, and Richard Socher. 2018.  DCN+: Mixed objective and deep residual coattention for question answering.  In _International Conference on Learning Representations_.
  * Youngmin Kim (2020) Seungyoung Lim;Hyunjeong Lee;Soyoon Park;Myungji Kim Youngmin Kim. 2020.  KorQuAD 2.0: Korean QA Dataset for Web Document Machine Comprehension.  _Journal of KIISE_ , 47:577–586.
  * Zhang et al. (2021) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021\.  Revisiting few-sample {bert} fine-tuning.  In _International Conference on Learning Representations_.
  * Zhang et al. (2020) Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020\.  TernaryBERT: Distillation-aware ultra-low bit BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 509–521, Online. Association for Computational Linguistics.