# MemPool-3D: Boosting Performance and Efficiency of Shared-L1 Memory Many-
Core Clusters with
3D Integration

Matheus Cavalcante1, Anthony Agnesina2, Samuel Riedel1, Moritz Brunion3,
Alberto García-Ortiz3, Dragomir Milojevic4, Francky Catthoor4, Sung Kyu Lim2,
and Luca Benini15  1ETH Zürich, Zürich, Switzerland  2Georgia Institute of
Technology, Atlanta, Georgia, USA  3Universität Bremen, Bremen, Germany
4IMEC, Leuven, Belgium  5Università di Bologna, Bologna, Italy  Email:
matheus.cavalcante _at_ iis.ee.ethz.ch

###### Abstract

Three-dimensional integrated circuits promise power, performance, and
footprint gains compared to their 2D counterparts, thanks to drastic
reductions in the interconnects’ length through their smaller form factor. We
can leverage the potential of 3D integration by enhancing MemPool, an open-
source many-core design with 256 cores and a shared pool of L1 scratchpad
memory connected with a low-latency interconnect. MemPool’s baseline 2D design
is severely limited by routing congestion and wire propagation delay, making
the design ideal for 3D integration. In architectural terms, we increase
MemPool’s scratchpad memory capacity beyond the sweet spot for 2D designs,
improving performance in a common digital signal processing kernel. We propose
a 3D MemPool design that leverages a smart partitioning of the memory
resources across two layers to balance the size and utilization of the stacked
dies. In this paper, we explore the architectural and the technology parameter
spaces by analyzing the power, performance, area, and energy efficiency of
MemPool instances in 2D and 3D with
$1\text{\,}\mathrm{MiB}2\text{\,}\mathrm{MiB}4\text{\,}\mathrm{MiB}8\text{\,}\mathrm{MiB}$
of scratchpad memory in a commercial $28\text{\,}\mathrm{nm}$ technology node.
We observe a performance gain of $9.1\text{\,}\mathrm{\char 37\relax}$ when
running a matrix multiplication on the MemPool-3D design with
$4\text{\,}\mathrm{MiB}$ of scratchpad memory compared to the MemPool 2D
counterpart. In terms of energy efficiency, we can implement the MemPool-3D
instance with $4\text{\,}\mathrm{MiB}$ of L1 memory on an energy budget
$15\text{\,}\mathrm{\char 37\relax}$ smaller than its 2D counterpart, and even
$3.7\text{\,}\mathrm{\char 37\relax}$ smaller than the MemPool-2D instance
with one-fourth of the L1 scratchpad memory capacity.

###### Index Terms:

Many-core; 3D Integration; 3D-ICs.

##  I Introduction

Vertical integration promises to address the scaling problems of the
traditional 2D integration foreseen by Moore’s Law [1]. Three-dimensional
integrated circuits (3D-ICs) promise better power, performance, and area (PPA)
than 2D counterparts, thanks to a drastic reduction of the interconnect
lengths, particularly of long global interconnects, while enabling a smaller
form factor by adding the third dimension [2, 3].

Advances in flip-chip interconnection technology allow for the miniaturization
of the inter-die connections. While controlled collapse chip connection (C4)
solder bumps have a pitch of around $100\text{\,}\mathrm{\SIUnitSymbolMicro
m}$ [4], face-to-face (F2F) wafer-to-wafer hybrid bonding enables interconnect
pitches in the micrometer range while maintaining reasonable yield rates [5].
Such a fine pitch can be leveraged to implement 3D-IC designs with a very high
interconnect density.

For the implementation of F2F-bonded 3D-ICs, the Macro-3D flow [6] provides
state-of-the-art PPA optimization capabilities for memory-on-logic
partitioning schemes. Since the flow is aware of all metal layers in the die
stack, the back end of the line (BEOL) routing resources of both dies can be
shared [6]. It is, therefore, possible to use one chip’s BEOL to avoid
congestion bottlenecks in the other chip. This resource sharing allows for a
more efficient routing utilization, which is extremely useful for the
implementation of highly congested designs.

Many-core systems achieve better PPA through vertical integration due to the
shorter interconnect lengths. Although some many-core systems explore 3D-IC
implementations [3, 7, 8], they use a 2D-mesh network to connect their
processing elements, failing to exploit the interconnection capabilities of a
denser pitch. In this paper, we use MemPool [9] as our target design, an open-
source [10] many-core system with $256$ cores and a configurable amount of
shared L1 Scratchpad Memory (SPM) connected with a low-latency interconnect.
Routing congestion severely limits MemPool’s implementation, with its
operating frequency bounded by the wire propagation delay. This makes MemPool
an ideal candidate for 3D design. With the holistic view over all metal layers
in the Macro-3D flow, the BEOL resources in both dies are combined,
alleviating MemPool’s congestion and achieving a higher operating frequency
thanks to reductions in the wire length. In this paper, we explore the
architectural and the technology parameter spaces by analyzing the PPA impact
of MemPool’s L1 SPM capacity scaling, from
$1\text{\,}\mathrm{MiB}8\text{\,}\mathrm{MiB}$, on 2D and 3D implementations.
The contributions of this paper are:

  * •

A flexible partitioning scheme of MemPool into logic and memory dies, capable
of achieving high utilization of the memory die for large memory capacities
(Section IV);

  * •

The complete 2D and Macro-3D implementations of MemPool in a commercial
$28\text{\,}\mathrm{nm}$ technology node, for all considered SPM capacities,
and an analysis of the instances in terms of power, performance, area,
footprint, and energy efficiency (Sections IV and V);

  * •

An exploration of MemPool’s SPM capacity and its impact on the runtime of a
common matrix multiplication kernel, including an analysis of the off-chip
memory bandwidth’s influence (Section VI).

##  II Architecture

MemPool is an open-source shared-L1 many-core cluster with $256$ very-small
cores sharing a multi-banked shared-L1 SPM through a low-latency interconnect
[9, 10]. Its maximum operating frequency shows a high sensibility to the
available routing resources and footprint size.

###  II-A Tile

MemPool is built hierarchically through the replication of tiles, whose
architecture can be seen in Figure 1. Each tile contains four very-small
Snitch RV32IMAXpulpimg cores [11], $2\text{\,}\mathrm{KiB}$ of L1 instruction
cache, and $16$ Static Random-Access Memory (SRAM) banks of SPM locally
accessible within one cycle. The cores can execute instructions of the
Xpulpimg extension, e.g., multiply-accumulate and load/store post-increment
instructions. A fully connected logarithmic crossbar connects local cores and
banks. In addition, four remote ports per tile allow remote tiles to access
the local tile’s SPM banks.

Figure 1: MemPool’s tile architecture, highlighting a possible partitioning
into the logic and the memory dies, used for the
$1\text{\,}\mathrm{MiB}2\text{\,}\mathrm{MiB}4\text{\,}\mathrm{MiB}$ 3D
designs. Solid lines indicate requests, and dotted lines indicate responses.

In our physical implementations with Macro-3D, the tile is partitioned into a
logic and a memory die. A possible partitioning would assign all the memory
banks of a tile to the memory die, namely $2\text{\,}\mathrm{KiB}$ of
instruction cache and a multi-banked SPM of parameterizable capacity, as shown
in Figure 1. The area requirement of $60\text{\,}\mathrm{kGE}$ per Snitch core
[9] and the tile interconnect’s logic define the footprint required for the
logic die. With the default SPM capacity of $1\text{\,}\mathrm{MiB}$ utilizing
only $51\text{\,}\mathrm{\char 37\relax}$ of the memory die area, an increase
of the SPM capacity can be used to balance the area requirement of both dies.
As we will analyze in Section IV, the $8\text{\,}\mathrm{MiB}$ 3D design uses
an adjusted partitioning scheme due to the increased SRAM size.

###  II-B Group and Cluster

The MemPool cluster is built hierarchically using the tile as the starting
point. Sixteen tiles form a _group_ , whose architecture is shown in Figure 2.
Each core can access SPM banks in the same group within three cycles. Four
$16\text{\times}16$ radix-4 butterfly networks are used in each group to
connect tiles within the same group (_local_ interconnect) or in different
groups (_north_ , _northeast_ , and _east_ interconnects).

At the top level of the architecture, the MemPool cluster has four identical
groups, as shown in Figure 2. Each core can access SPM banks in other groups
with five cycles of latency. At this hierarchical level, there are only point-
to-point connections between groups. This paper focuses on implementing the
MemPool groups since only a few cells (about five thousand) need to be placed
between them at the cluster level.

(a) MemPool Group.

(b) MemPool Cluster.

Figure 2: MemPool’s hierarchical architecture, with $16$ tiles forming a group
(Figure 2) and $4$ groups forming the full MemPool cluster (Figure 2).

The dissimilarity between the tile and group makes MemPool a highly suitable
architecture for 3D integration. While the tile has a high logic density, the
group is highly congested due to its global interconnects. The 2D MemPool’s
critical path goes from one tile to the other diagonally opposed to it.
Approximately $37\text{\,}\mathrm{\char 37\relax}$ of its timing is wire
propagation delay, and $75\text{\,}\mathrm{\char 37\relax}$ of its cells are
buffers or inverter pairs [9].

##  III Methodology

MemPool is synthesized and implemented in a commercial
$28\text{\,}\mathrm{nm}$ high-$\kappa$ technology node. In the 3D
implementations, we use a fine F2F via pitch of
$1.0\text{\,}\mathrm{\SIUnitSymbolMicro m}$. The F2F via size, resistance, and
capacitance is $0.5\text{\times}0.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$,
$0.5\text{\,}\mathrm{\SIUnitSymbolOhm}$, and $1\text{\,}\mathrm{fF}$
respectively [5]. The 3D BEOL comprises six metal layers in both tiers (M6M6),
separated by the F2F via layer, similarly to the setup proposed by Bamberg et
al. [6].

Tiles are first synthesized using Synopsys Design Compiler 2021.06 and then
implemented with the corresponding flow in Cadence Innovus 20.13. We use a
uniform $1\text{\,}\mathrm{GHz}$ frequency target on the typical corner to
implement all the designs. The 2D tiles use a six-layer BEOL (M6), while the
3D tiles use the mirrored M6M6 stack mentioned above. The implemented
tiles—abstracted into black boxes with full blockages on all utilized routing
layers—are used for the groups’ syntheses and physical implementations,
similarly to the reference flow in [9].

The 3D groups use the same BEOL as the tiles, i.e., M6M6, while the 2D groups
have two extra layers (M8) to allow over-the-tile routing. When abstracting a
tile implementation, the physical representation by default creates
obstructions on all metal layers that are available for routing. In the 2D
case, this affects M1 to M6. As the Macro-3D tile utilizes the BEOL of the
logic and the memory die simultaneously, the tile abstraction blocks not only
M1 to M6 of the logic die, but also M1 to M6 of the memory die. The 3D tile
abstractions, therefore, prevent any inter-tile routing on the group level
where tiles are placed.

In this paper, we analyze a total of eight MemPool configurations. Each
configuration is named MemPool-Flow${}_{\text{Capacity}}$, where _Capacity_ is
the total capacity of the shared-L1 SPM at the MemPool cluster, i.e., one of
$1\text{\,}\mathrm{MiB}2\text{\,}\mathrm{MiB}4\text{\,}\mathrm{MiB}8\text{\,}\mathrm{MiB}$,
and _Flow_ is either 2D or 3D.

##  IV Tile Implementation

The tiles are implemented to target a standard cell density of
$90\text{\,}\mathrm{\char 37\relax}$ in the logic die. Figure 3 shows the
memory die floorplanning used to implement some of the considered MemPool-3D
configurations. We use the partitioning of Figure 1 to implement the tiles of
the MemPool-3D configurations with
$1\text{\,}\mathrm{MiB}4\text{\,}\mathrm{MiB}$ of SPM. The memory die of the
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ configuration, shown in
Figure 3, only utilizes $51\text{\,}\mathrm{\char 37\relax}$ of the area of
the memory die. On the other hand, the instance with $4\text{\,}\mathrm{MiB}$
of SPM, shown in Figure 3, achieves a much higher $89\text{\,}\mathrm{\char
37\relax}$ utilization, with a tile footprint only $13\text{\,}\mathrm{\char
37\relax}$ higher than the
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ tile.

(a) $\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$.

(b) $\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$.

(c) $\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$.

Figure 3: Floorplan of the memory die of the tile used to implement the
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$,
$\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$, and
$\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$. Images to scale.

The $\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ configuration uses a
partitioning different from the one shown in Figure 1 due to the increased
area of the memory macros. In this configuration, $15$ out of the $16$ SPM
macros are arranged in a $5\text{\times}3$ array in the memory die. This
memory die achieves near $100\text{\,}\mathrm{\char 37\relax}$ utilization, as
shown in Figure 3, while its footprint is $40\text{\,}\mathrm{\char 37\relax}$
larger than the $\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ design.
As we can see in Table I, both dies still keep a balanced area ratio with this
partitioning. The core utilization of the logic die is
$84\text{\,}\mathrm{\char 37\relax}$, only $6\text{\,}\mathrm{\char 37\relax}$
below the target utilization. The extra area of the logic die is used by one
SPM bank and all the tile’s instruction cache banks.

Table I: MemPool Tile’s implementation results, normalized by the results of the baseline $\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ configuration. Flow | SPM Capacity | Footprint | Core utilization
---|---|---|---
_Logic die_ | _Memory die_
2D | $1\text{\,}\mathrm{MiB}$ | $1.000$ | $90\text{\,}\mathrm{\char 37\relax}$ | —
$2\text{\,}\mathrm{MiB}$ | $1.104$ | $90\text{\,}\mathrm{\char 37\relax}$ | —
$4\text{\,}\mathrm{MiB}$ | $1.420$ | $84\text{\,}\mathrm{\char 37\relax}$ | —
$8\text{\,}\mathrm{MiB}$ | $1.817$ | $86\text{\,}\mathrm{\char 37\relax}$ | —
3D | $1\text{\,}\mathrm{MiB}$ | $0.667$ | $90\text{\,}\mathrm{\char 37\relax}$ | $51\text{\,}\mathrm{\char 37\relax}$
$2\text{\,}\mathrm{MiB}$ | $0.667$ | $90\text{\,}\mathrm{\char 37\relax}$ | $65\text{\,}\mathrm{\char 37\relax}$
$4\text{\,}\mathrm{MiB}$ | $0.767$ | $85\text{\,}\mathrm{\char 37\relax}$ | $89\text{\,}\mathrm{\char 37\relax}$
$8\text{\,}\mathrm{MiB}$ | $0.933$ | $84\text{\,}\mathrm{\char 37\relax}$ | $100\text{\,}\mathrm{\char 37\relax}$

There is a negligible PPA difference across all tile instances. The fastest
tile, $\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$, achieves a
frequency only $6\text{\,}\mathrm{\char 37\relax}$ higher than the slowest
tile, $\text{MemPool-3D}_{\text{$2\text{\,}\mathrm{MiB}$}}$. This is because
the tile is primarily constrained by external delays that model the group
rather than internal register-to-register paths. While 3D integration can
reduce the wire length of these paths, the primary effect is reducing the
tile’s footprint, leading to shorter interconnects and, thus, group level PPA
improvements.

##  V Group Implementation

###  V-A Area and Footprint

The group is at the critical level in the implementation of MemPool. When
deriving the channel widths between tiles, we need to consider that the group
is densely connected at the design’s center, where most of the logic of the
local interconnect is placed. This causes heavy congestion, creating design
rule violations (DRVs) and degrades timing if the tiles are not sufficiently
spaced in the center of the design. Figure 4 exemplifies this by showing the
routing and cell density map of the
$\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$ instance. The four group
interconnects can be seen as pockets of very high cell density in Figure 4.

(a) Routing.

(b) Cell density map.

Figure 4: Routing and cell density map of the
$\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$ group. Yellow and red
colors indicate regions with a very high cell density. Dark blue and gray
regions have a cell density close to zero.

Figure 5 shows both group designs with $8\text{\,}\mathrm{MiB}$ of SPM. We
highlight that the width of the channels between tiles is kept constant for
all trials of each flow. The rationale is that the group interconnects’ size
is largely independent of the SPM capacity, except for the additional address
bits.

(a) $\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$.

(b) $\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$.

Figure 5: $\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ and
$\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ placed-and-routed
designs, highlighting the group routing. Images to scale.

The channels between the 3D tiles are $18\text{\,}\mathrm{\char 37\relax}$
narrower than the 2D counterparts since the 3D designs have the twelve layers
of the mirrored M6M6 BEOL to route the group interconnects. In contrast, the
2D trials can only use the eight layers of the M8 BEOL. This effect compounds
with the reduced footprint of the 3D tiles, further reducing the footprint of
the MemPool-3D groups. In Figure 5, we can also see the over-the-tile routing
of the 2D designs—since the tiles are routed up to M6 and the group up to M8.
Even though the 3D runs have a more aggressive BEOL than the 2D runs, M6M6,
the lack of over-the-tile routing incurs extra congestion since all group
interconnects have routing confined to the channels. The footprint of the
MemPool-3D groups could be further reduced if routing resources of the tiles
would be available to implement the group level.

Table II summarizes the implementation results of the eight considered groups,
normalized by the baseline group
$\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$. In terms of footprint,
the 3D groups are much smaller than the 2D groups. For example, the largest 3D
group, $\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$, has a footprint
$14\text{\,}\mathrm{\char 37\relax}$ smaller than the smallest 2D group,
$\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$. The 3D MemPool groups
also have a footprint much smaller than their 2D counterparts. For instance,
the $\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ group has a
footprint $46\text{\,}\mathrm{\char 37\relax}$ smaller than
$\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$, as seen in Figure 5.

Table II: MemPool group’s 2D and 3D power, operating frequency, footprint, area, and energy efficiency results of MemPool instances with $1\text{\,}\mathrm{MiB}2\text{\,}\mathrm{MiB}4\text{\,}\mathrm{MiB}8\text{\,}\mathrm{MiB}$ of SPM L1 memory, normalized by the results of the baseline 2D design with $1\text{\,}\mathrm{MiB}$ of SPM. The numbers in parentheses show the difference relative to the 2D counterpart design. L1 SPM Capacity | $1\text{\,}\mathrm{MiB}$ |  | $2\text{\,}\mathrm{MiB}$ |  | $4\text{\,}\mathrm{MiB}$ |  | $8\text{\,}\mathrm{MiB}$
---|---|---|---|---|---|---|---
Implementation Flow | 2D | 3D |  | 2D | 3D |  | 2D | 3D |  | 2D | 3D
BEOL | M8 | M6M6 |  | M8 | M6M6 |  | M8 | M6M6 |  | M8 | M6M6
Footprint | $1.000$ |  $0.665$ ($-33\text{\,}\mathrm{\char 37\relax}$) |  | $1.074$ |  $0.665$ ($-38\text{\,}\mathrm{\char 37\relax}$) |  | $1.299$ |  $0.737$ ($-43\text{\,}\mathrm{\char 37\relax}$) |  | $1.572$ |  $0.857$ ($-46\text{\,}\mathrm{\char 37\relax}$)
Combined Die Area | $1.000$ |  $1.330$ ($+33\text{\,}\mathrm{\char 37\relax}$) |  | $1.074$ |  $1.330$ ($+23\text{\,}\mathrm{\char 37\relax}$) |  | $1.299$ |  $1.474$ ($+13\text{\,}\mathrm{\char 37\relax}$) |  | $1.572$ |  $1.714$ ($+9.0\text{\,}\mathrm{\char 37\relax}$)
Wire Length | $1.000$ | $0.803$ |  | $1.036$ | $0.803$ |  | $1.131$ | $0.844$ |  | $1.294$ | $0.888$
Density [$\mathrm{\char 37\relax}$] | $53.0$ | $54.5$ |  | $54.0$ | $54.8$ |  | $53.4$ | $53.2$ |  | $56.9$ | $54.4$
#Buffers | $182.9103$ | $151.5103$ |  | $190.3103$ | $151.2103$ |  | $212.5103$ | $166.5103$ |  | $217.6103$ | $156.1103$
#F2F Bumps | — | $78.3103$ |  | — | $78.9103$ |  | — | $84.4103$ |  | — | $86.2103$
Eff. Frequency | $1.000$ |  $1.040$ ($+4.0\text{\,}\mathrm{\char 37\relax}$) |  | $0.930$ |  $0.979$ ($+5.2\text{\,}\mathrm{\char 37\relax}$) |  | $0.875$ |  $0.955$ ($+9.1\text{\,}\mathrm{\char 37\relax}$) |  | $0.885$ |  $0.930$ ($+5.1\text{\,}\mathrm{\char 37\relax}$)
Total Negative Slack | $-1.000$ | $-0.184$ |  | $-2.080$ | $-0.458$ |  | $-5.887$ | $-0.604$ |  | $-5.212$ | $-0.962$
#Failing Paths | $1140$ | $1046$ |  | $1636$ | $1332$ |  | $4396$ | $1747$ |  | $4352$ | $2403$
Total Power | $1.000$ | $0.913$ |  | $1.045$ | $0.958$ |  | $1.129$ | $1.041$ |  | $1.299$ | $1.173$
Power-delay product | $1.000$ |  $0.877$ ($-12\text{\,}\mathrm{\char 37\relax}$) |  | $1.129$ |  $0.981$ ($-13\text{\,}\mathrm{\char 37\relax}$) |  | $1.290$ |  $1.089$ ($-16\text{\,}\mathrm{\char 37\relax}$) |  | $1.469$ |  $1.261$ ($-14\text{\,}\mathrm{\char 37\relax}$)

The combined area of the memory and logic dies of the MemPool-3D groups is
larger than the area of the MemPool-2D groups. The area overhead of the
MemPool-3D groups, however, decreases with increasing SPM capacity. The
combined die area of the largest 3D group,
$\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$, is only
$9\text{\,}\mathrm{\char 37\relax}$ larger than the
$\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ group area, indicating
that the partitioning is closer to ideal. Although the footprint is the most
important metric for analyzing PPA gains of the 3D integration thanks to
reduced interconnect lengths, the combined area is more relevant for an
implementation cost analysis of the 3D designs. This paper focuses on the
physical implementation of the groups, MemPool’s most critical hierarchical
level since the cluster only has four identical groups and some glue logic.
However, it can be noted that the 12-layer mirrored BEOL of the MemPool-3D
designs implies that the channels between groups needed to route the cluster-
level connections can be made shorter than the equivalent channels of the
MemPool-2D cluster. This means that we can expect an even more favorable area
ratio at the cluster level.

###  V-B Power and Operating Frequency

MemPool’s sensibility to the footprint size can be seen in the normalized wire
length results of Table II. The wire length of the MemPool-2D groups grows by
$29.4\text{\,}\mathrm{\char 37\relax}$ from
$\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ to
$\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$, accompanied by an
$18.9\text{\,}\mathrm{\char 37\relax}$ increase in the number of buffers. The
effect compounds to other PPA metrics. For example, the 2D groups achieve an
operating frequency up to $12.5\text{\,}\mathrm{\char 37\relax}$ slower, a
power consumption up to $29.9\text{\,}\mathrm{\char 37\relax}$ higher, and a
power-delay product (PDP) up to $46.9\text{\,}\mathrm{\char 37\relax}$ higher
than the baseline $\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ group.

The MemPool-3D groups have a significantly smaller PPA degradation with
increasing SPM capacity than the MemPool-2D groups. The low utilization of the
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ design (Figure 3)
implies it is possible to implement the
$\text{MemPool-3D}_{\text{$2\text{\,}\mathrm{MiB}$}}$ group without increasing
the footprint. Even the largest MemPool-3D group, with
$8\text{\,}\mathrm{MiB}$ of SPM, has a footprint only
$10.6\text{\,}\mathrm{\char 37\relax}$ larger than the
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ group. This smaller
footprint variation leads to a PPA degradation which is less drastic than the
one affecting the MemPool-2D groups. As a result, the MemPool-3D groups
achieve an operating frequency only $11.8\text{\,}\mathrm{\char 37\relax}$
slower and a power consumption only $28.4\text{\,}\mathrm{\char 37\relax}$
higher than the baseline $\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$
group.

The benefits of 3D integration on MemPool are clearer when comparing instances
with the same SPM capacity across 2D and 3D implementation flows. In general,
the MemPool-3D designs have a smaller footprint, achieve a higher operating
frequency, and consume less power than their 2D counterparts. In terms of
footprint, the largest gains are found on the groups of the
$8\text{\,}\mathrm{MiB}$ configuration, with
$\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ having a footprint
$46\text{\,}\mathrm{\char 37\relax}$ smaller than
$\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$. In terms of frequency
and PDP, the $\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$ group
achieves an operating frequency $9.1\text{\,}\mathrm{\char 37\relax}$ higher
and a PDP $16\text{\,}\mathrm{\char 37\relax}$ lower than the
$\text{MemPool-2D}_{\text{$4\text{\,}\mathrm{MiB}$}}$ group. It is also
interesting that there is an operating frequency drop of
$6.2\text{\,}\mathrm{\char 37\relax}$ between the
$\text{MemPool-3D}_{\text{$2\text{\,}\mathrm{MiB}$}}$ and
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ groups, despite having
the same footprint. This is due to the longer SRAMs’ delay, which impacts the
timing of the tile’s input-to-register and register-to-output paths. In
general, the MemPool-3D groups have a higher operating frequency, consume less
power, and have a lower PDP than the MemPool-2D groups with the same SPM
capacity thanks to their smaller footprint and wire length.

##  VI Performance Analysis

We use a matrix multiplication kernel as a representative application for
MemPool’s target domain to quantify the algorithmic benefits of increasing the
memory capacity even for compute-bound kernels. Benefits on memory bound
kernels are obviously larger, but we believe the analysis in a compute-bound
regime allows us to gather more interesting insights on MemPool-3D’s
performance.

###  VI-A Cycle count

We measure the cycle count of computing the matrix multiplication of two
$M\times M$ ($M=326400$) matrices that do not fit into the SPM but reside in
global memory through a cycle-accurate register-transfer level simulation of
the $\text{MemPool-c}_{\text{$\text{\,}\mathrm{MiB}$}}$luster. The matrix size
is chosen to be the least common multiple of the tile sizes $t\times t$
($t=256,384,544,800$) that fully utilize the available SPM in each
configuration, enabling optimal tiling and maximizing data reuse. The cores
load the input tiles and synchronize in a memory phase before computing on the
output tile in a compute phase. Those phases are repeated until the output
tile is fully computed. At this point, the output tile is stored back into the
main memory, and the process repeats for the subsequent output tiles. Since
different output tiles require the same input data, each input element is
loaded exactly $M/t$ times. Therefore, having a bigger SPM allows for more
data reuse and less memory overhead. The second benefit of increased tile size
is the increased length of the compute phase minimizing repeated static
overhead due to loop setup and synchronization.

We calculate the cycle count of the memory phase for different off-chip memory
bandwidths. A classic double data rate (DDR) synchronous dynamic random-access
memory (SDRAM) stick has a data width of $8\text{\,}\mathrm{B}$, which means a
single DDR channel clocked at the same frequency as MemPool could deliver at
most $16\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$. Therefore, we analyze
bandwidths around this realistic case, ranging from a worst-case bandwidth of
$4\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$ to a very optimistic
$64\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$. Our model idealizes the latency
into the off-chip global memory. We measure the duration of the compute phase
with a hot instruction cache and calculate the total cycle count by
accumulating all phases. The results in Figure 6 show a speedup of
$43\text{\,}\mathrm{\char 37\relax}$ for the $8\text{\,}\mathrm{MiB}$ case
over the baseline for the worst-case bandwidth, where the memory transfers
make up a significant portion of the runtime. For the off-chip memory
bandwidth of one DDR channel, the configuration with $8\text{\,}\mathrm{MiB}$
of SPM achieves a cycle count speedup of $16\text{\,}\mathrm{\char 37\relax}$
over the baseline. Even for the optimistic off-chip memory bandwidth of
$64\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$, the largest configuration still
generates an $8\text{\,}\mathrm{\char 37\relax}$ benefit over the baseline,
showing that a higher memory capacity is highly beneficial from an algorithmic
point of view.

$0$$20$$40$$60$$80$$1\text{\,}\mathrm{MiB}$$2\text{\,}\mathrm{MiB}$$4\text{\,}\mathrm{MiB}$$8\text{\,}\mathrm{MiB}$$+3.8\text{\,}\mathrm{\char
37\relax}$$+3.2\text{\,}\mathrm{\char 37\relax}$$+1.0\text{\,}\mathrm{\char
37\relax}$$+7.3\text{\,}\mathrm{\char 37\relax}$$+5.4\text{\,}\mathrm{\char
37\relax}$$+2.8\text{\,}\mathrm{\char 37\relax}$$+17\text{\,}\mathrm{\char
37\relax}$$+12\text{\,}\mathrm{\char 37\relax}$$+8.8\text{\,}\mathrm{\char
37\relax}$MemPool’s SPM CapacityCycle count speedup [$\mathrm{\char
37\relax}$]$64\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$$32\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$$16\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$$8\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$$4\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$

Figure 6: Cycle count speedup of the matrix multiplication kernel with larger
SPM capacity, as a function of the off-chip memory bandwidth, relative to the
$1\text{\,}\mathrm{MiB}$ configuration with an off-chip memory bandwidth of
$4\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$. The percentages by the data
points indicate the speedup relative to the instance with the same off-chip
memory bandwidth but half of the SPM capacity.

###  VI-B Performance and Energy-Efficiency

The increased SPM capacity impacts the PPA of the design, as seen in Section
V. This Section combines those effects to analyze how MemPool’s performance
and energy efficiency evolve across all considered configurations. Throughout
this section, we chose an off-chip memory bandwidth of
$16\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$ to represent our memory
subsystem.

Figure 7 shows the performance of the matrix multiplication kernel on MemPool,
as a function of the SPM capacity, for the MemPool-2D and MemPool-3D designs.
Thanks to their higher operating frequencies, the MemPool-3D groups achieve a
performance up to $9.1\text{\,}\mathrm{\char 37\relax}$ higher than the
MemPool-2D groups. The MemPool-2D groups achieve small performance gains with
increasing SPM capacity, reaching a gain of at most $3.1\text{\,}\mathrm{\char
37\relax}$ for the $\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ case.
Due to a particularly low operating frequency, the
$\text{MemPool-2D}_{\text{$4\text{\,}\mathrm{MiB}$}}$ has a performance drop
compared to the $\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ design.
The MemPool-3D designs, on the other hand, achieve consistently higher
performances with increasing SPM capacity, outperforming their 2D
counterparts. The $\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ design
achieves the highest performance, $8.4\text{\,}\mathrm{\char 37\relax}$ above
the baseline.

$-2$$0$$2$$4$$6$$8$$10$$1\text{\,}\mathrm{MiB}$$2\text{\,}\mathrm{MiB}$$4\text{\,}\mathrm{MiB}$$8\text{\,}\mathrm{MiB}$$+4.2$%$+5.3$%$+9.1$%$+5.1$%MemPool’s
SPM CapacityPerformance gain [$\mathrm{\char 37\relax}$]2D3D

Figure 7: Performance gain of the matrix multiplication kernel with larger SPM
capacity, relative to $\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$
with a $16\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$ off-chip memory
bandwidth. The percentages above the bars indicate the speedup of the
MemPool-3D instance compared to the MemPool-2D with the same SPM capacity.

The energy efficiency, as expected, shows the opposite trend than the
performance. As shown in Figure 8, the energy efficiency of the MemPool
designs tends to decrease with an increasing SPM capacity. The
$\text{MemPool-2D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ group achieves the worst
energy efficiency, $21\text{\,}\mathrm{\char 37\relax}$ below the efficiency
of the $\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ design. In
addition, the 3D designs consistently outperform their 2D counterparts. For
example, the $\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$ design
achieves an efficiency $18.4\text{\,}\mathrm{\char 37\relax}$ higher than the
2D design with the same SPM capacity.

$-30$$-20$$-10$$0$$10$$20$$1\text{\,}\mathrm{MiB}$$2\text{\,}\mathrm{MiB}$$4\text{\,}\mathrm{MiB}$$8\text{\,}\mathrm{MiB}$$+14$%$+14.5$%$+18.4$%$+16.5$%MemPool’s
SPM CapacityEnergy efficiency gain [$\mathrm{\char 37\relax}$]2D3D

Figure 8: MemPool’s energy efficiency gain with larger SPM capacity, relative
to $\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ with a
$16\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$ off-chip memory bandwidth, when
running the matrix multiplication kernel. The percentages above the bars
indicate the energy efficiency gain of the MemPool-3D instance compared to the
MemPool-2D with the same SPM capacity. Higher is better.

The results of Figure 8 indicate that the
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ is the optimal design
from the energy efficiency point of view, $14\text{\,}\mathrm{\char 37\relax}$
higher than $\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$. In
addition, three-dimensional integration allows for the implementation of
$\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$—a design with four times
as much SPM capacity as the baseline design,
$\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$—on an energy budget
$3.7\text{\,}\mathrm{\char 37\relax}$ smaller. This intrinsic tradeoff between
performance and energy efficiency can be better analyzed with the energy-delay
product (EDP) results of Figure 9, the EDP defined as the product of the total
energy consumption by the runtime. The
$\text{MemPool-3D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ configuration has the
lowest EDP, $15.6\text{\,}\mathrm{\char 37\relax}$ below the baseline.

$-20$$-10$$0$$10$$20$$30$$1\text{\,}\mathrm{MiB}$$2\text{\,}\mathrm{MiB}$$4\text{\,}\mathrm{MiB}$$8\text{\,}\mathrm{MiB}$$-15.6$%$-17.3$%$-22.6$%$-18.2$%MemPool’s
SPM CapacityEDP variation [$\mathrm{\char 37\relax}$]2D3D

Figure 9: MemPool’s EDP variation with larger SPM capacity, relative to
$\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ with a
$16\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$ off-chip memory bandwidth, when
running the matrix multiplication kernel. The percentages above the bars
indicate the EDP variation of the MemPool-3D instance compared to the
MemPool-2D with the same SPM capacity. Lower is better.

##  VII Conclusions

In this paper, we analyzed the power, performance, area, and energy efficiency
of MemPool, through a co-exploration of its architectural and technological
parameter spaces. We implemented MemPool with
$1\text{\,}\mathrm{MiB}2\text{\,}\mathrm{MiB}4\text{\,}\mathrm{MiB}8\text{\,}\mathrm{MiB}$
of SPM, using 2D and 3D implementation flows, on a modern commercial
$28\text{\,}\mathrm{nm}$ technology node.

We explored MemPool’s performance while running a large matrix multiplication
kernel as a function of the L1 SPM capacity and the off-chip memory bandwidth.
For a realistic bandwidth of $16\text{\,}\mathrm{B}\text{/}\mathrm{cycle}$, we
observe a cycle count reduction of $16\text{\,}\mathrm{\char 37\relax}$ when
increasing the SPM capacity from
$1\text{\,}\mathrm{MiB}8\text{\,}\mathrm{MiB}$.

Despite the maximum operating frequency degradation with an increasing SPM
capacity, the 3D designs can still achieve an operating frequency up to
$9.1\text{\,}\mathrm{\char 37\relax}$ higher than their 2D counterparts. The
$\text{MemPool-3D}_{\text{$8\text{\,}\mathrm{MiB}$}}$ design achieves a
performance $8.4\text{\,}\mathrm{\char 37\relax}$ higher than the
$\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$ baseline. The 3D designs
consistently outperform their 2D counterparts by up to
$9.1\text{\,}\mathrm{\char 37\relax}$. Regarding energy efficiency, the 3D
designs outperform their 2D counterparts by up to $18.4\text{\,}\mathrm{\char
37\relax}$. While increasing the SPM size in the 2D case leads to worse energy
efficiency, all but the largest 3D designs achieve a better energy efficiency
than the 2D baseline. We are able to implement the
$\text{MemPool-3D}_{\text{$4\text{\,}\mathrm{MiB}$}}$ design with an energy
budget $3.7\text{\,}\mathrm{\char 37\relax}$ smaller than the 2D instance with
only one-fourth of the SPM capacity,
$\text{MemPool-2D}_{\text{$1\text{\,}\mathrm{MiB}$}}$. To summarize, in this
paper we showed the need for a co-exploration approach with full 3D
implementations to optimize modern designs constrained by their interconnect
subsystems.

## References

  * [1] S. Panth, K. Samadi, Y. Du, and S. K. Lim, “High-density integration of functional modules using monolithic 3D-IC technology,” in _2013 18th Asia and South Pacific Design Automation Conference (ASP-DAC)_ , 2013, pp. 681–686.
  * [2] V. F. Pavlidis and E. G. Friedman, “Interconnect-based design methodologies for three-dimensional integrated circuits,” _Proceedings of the IEEE_ , vol. 97, no. 1, pp. 123–140, 2009.
  * [3] X. Dong, J. Zhao, and Y. Xie, “Fabrication cost analysis and cost-aware design space exploration for 3-D ICs,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 29, no. 12, pp. 1959–1972, 2010.
  * [4] W. S. Tsai, C. Y. Huang, C. K. Chung, K. H. Yu, and C. F. Lin, “Generational changes of flip chip interconnection technology,” in _2017 12th International Microsystems, Packaging, Assembly and Circuits Technology Conference (IMPACT)_ , 2017, pp. 306–310.
  * [5] E. Beyne, S.-W. Kim, L. Peng, N. Heylen, J. De Messemaeker, O. O. Okudur, A. Phommahaxay, T.-G. Kim, M. Stucchi, D. Velenis, A. Miller, and G. Beyer, “Scalable, sub $\mathrm{2}$ µ pitch, Cu/SiCN to Cu/SiCN hybrid wafer-to-wafer bonding technology,” in _2017 IEEE International Electron Devices Meeting (IEDM)_ , 2017, pp. 32.4.1–32.4.4.
  * [6] L. Bamberg, A. García-Ortiz, L. Zhu, S. Pentapati, D. E. Shim, and S. K. Lim, “Macro-3D: A physical design methodology for face-to-face-stacked heterogeneous 3D ICs,” in _2020 Design, Automation Test in Europe Conference Exhibition (DATE)_ , Grenoble, France, 2020, pp. 37–42.
  * [7] M. B. Healy, K. Athikulwongse, R. Goel, M. M. Hossain, D. H. Kim, Y.-J. Lee, D. L. Lewis, T.-W. Lin, C. Liu, M. Jung, B. Ouellette, M. Pathak, H. Sane, G. Shen, D. H. Woo, X. Zhao, G. H. Loh, H.-H. S. Lee, and S. K. Lim, “Design and analysis of 3D-MAPS: A many-core 3D processor with stacked memory,” in _IEEE Custom Integrated Circuits Conference 2010_ , 2010, pp. 1–4.
  * [8] S. Pentapati, L. Zhu, L. Bamberg, D. E. Shim, A. García-Ortiz, and S. K. Lim, “A logic-on-memory processor-system design with monolithic 3-D technology,” _IEEE Micro_ , vol. 39, no. 6, pp. 38–45, 2019.
  * [9] M. Cavalcante, S. Riedel, A. Pullini, and L. Benini, “MemPool: A shared-L1 memory many-core cluster with a low-latency interconnect,” in _2021 Design, Automation, and Test in Europe Conference and Exhibition (DATE)_ , Grenoble, France, Mar. 2021, pp. 701–706.
  * [10] PULP Platform, “MemPool,” https://github.com/pulp-platform/mempool/, 2021, GitHub repository.
  * [11] F. Zaruba, F. Schuiki, T. Hoefler, and L. Benini, “Snitch: A tiny pseudo dual-issue processor for area and energy efficient execution of floating-point intensive workloads,” _IEEE Transactions on Computers_ , 2020\.


#  IndicXNLI: Evaluating Multilingual Inference for Indian Languages

Divyanshu Aggarwal1 , Vivek Gupta2* , Anoop Kunchukuttan3
1Delhi Technological University, 2University of Utah, 3Microsoft Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
Contribution Corresponding Author

###### Abstract

While Indic NLP has made rapid advances recently in terms of the availability
of corpora and pre-trained models, benchmark datasets on standard NLU tasks
are limited. To this end, we introduce IndicXNLI, an NLI dataset for 11 Indic
languages. It has been created by high-quality machine translation of the
original English XNLI dataset and our analysis attests to the quality of
IndicXNLI. By finetuning different pre-trained LMs on this IndicXNLI, we
analyze various cross-lingual transfer techniques with respect to the impact
of the choice of language models, languages, multi-linguality, mix-language
input, etc. These experiments provide us with useful insights into the
behaviour of pre-trained models for a diverse set of languages.

##  1 Introduction

Natural Language Inference (NLI) is a well-studied NLP task (Dagan et al.,
2013) that assesses if a premise entails, negates, or is neutral towards the
hypothesis statement. The task is well suited for evaluating semantic
representations of state-of-the-art transformers (Vaswani et al., 2017) models
such as BERT (Devlin et al., 2019; Radford and Narasimhan, 2018). Two large
scale datasets, such as SNLI (Bowman et al., 2015) and MultiNLI (Williams et
al., 2018), has recently been developed to enhanced the relevance of NLI task.

With the availability of multi-lingual pre-trained language models like mBERT
(Devlin et al., 2019), and XLM-RoBERTa (Conneau et al., 2020) promising cross-
lingual transfer and universal models, multi-lingual NLP has recently gained a
lot of attention. However, most languages have a scarcity of datasets
resources. Some multi-lingual datasets have attempted to fill this gap,
including XNLI (Conneau et al., 2018) for NLI, XQUAD (Dumitrescu et al.,
2021), MLQA (Lewis et al., 2020) for question answering, and PAWS-X for
paraphrase identification (Yang et al., 2019). In many practical
circumstances, training sets for non-English languages are unavailable, hence
cross-lingual zero-shot evaluation benchmarks such as XTREME (Hu et al.,
2020), XTREME-R (Ruder et al., 2021), and XGLUE (Liang et al., 2020) have been
suggested to use these datasets.

However, NLI datasets are not available for major Indic languages. The only
exceptions are the test/validation sets in the XNLI (hi and ur), TaxiXNLI (hi)
K et al. (2021) and MIDAS-NLI Uppal et al. (2020) datasets. Furthermore,
because MIDAS-NLI is based on sentiment data recasting, hypotheses are not
linguistically diverse and span limited reasoning. In this work, we address
this gap by introducing IndicXNLI, an NLI dataset for _Indic_ languages.
IndicXNLI consists of English XNLI data translated into eleven _Indic_
languages. We use IndicXNLI to evaluate _Indic_ -specific models (trained only
on _Indic_ and English languages) such as IndicBERT Kakwani et al. (2020) and
MuRIL Khanuja et al. (2021), as well as generic (train on non-_Indic_
languages) such as mBERT and XLM-RoBERTa. Furthermore, we experimented with
several training strategies for each multi-lingual model. Our experimental
results answers multiple important questions regarding effective training for
_Indic_ NLI. Our contributions are as follows:

  * •

We introduce IndicXNLI, an NLI benchmark dataset for eleven prominent Indo-
Aryan _indic_ languages from the Indo-European and Dravidian language
families.

  * •

We investigate several strategies to train multi-lingual models for NLI tasks
on IndicXNLI. We also explore models cross-lingual NLI transfer ability across
_Indic_ languages and Intra-Bilingual NLI ability of pretrained multi-lingual
language models.

The IndicXNLI dataset, along with scripts, is available at
https://github.com/divyanshuaggarwal/indicxnli.

##  2 The IndicXNLI dataset

We created IndicXNLI, a NLI data set for _Indic_ languages. IndicXNLI is
similar to existing XNLI dataset in shape/form, but focusses on _Indic_
language family. IndicXNLI include NLI data for eleven major _Indic_ languages
that includes Assamese (‘as’), Gujarat (‘gu’), Kannada (‘kn’), Malayalam
(‘ml’), Marathi (‘mr’), Odia (‘or’), Punjabi (‘pa’), Tamil (‘ta’), Telugu
(‘te’), Hindi (‘hi’), and Bengali (‘bn’). Next we describe the IndicXNLI
construction and its validation in details.

###  2.1 IndicXNLI Construction.

To create IndicXNLI, we follow the approach of the XNLI dataset and translate
the English XNLI dataset (premises and hypothesis) to eleven _Indic_
-languages. We use the IndicTrans Ramesh et al. (2021), a state-of-the-art,
publicly available translation model for Indic languages, for translating from
English to _Indic_ languages. The train (392,702), validation (2,490), and
evaluation sets (5,010) of English XNLI were translated from English into each
of the eleven _Indic_ languages. IndicTrans is a large Transformer-based
sequence to sequence model. It is trained on Samanantar dataset Ramesh et al.
(2021), which is the largest parallel multi-lingual corpus over eleven _Indic_
languages. IndicTrans outperforms other open-source models based on mBART Liu
et al. (2020) and mT5 (Xue et al., 2021) for _Indic_ language translations and
is competitive with paid translation models such as Google-Translate or
Microsoft-Translate on several benchmarks Ramesh et al. (2021). Our choice of
IndicTrans was motivated by _cost, language coverage and speed_ , refer §4.

###  2.2 IndicXNLI Validation.

While translation may lose the semantic link between the sentences, recent
study by K et al. (2021) disproved this. K et al. (2021) qualitative analysis
illustrate that when a high-quality machine translation system is utilized,
classification labels and reasoning categories are only minimally altered by
one or two tokens for translated NLI datasets. We also demonstrate the high
quality of IndicTrans translation for IndicXNLI in two ways  (a.) manual human
validation and,  (b.) automatic metric BERTScore Zhang* et al. (2020) . Our
validation approach guarantee correctness for the IndicXNLI labels. Next,
we’ll discuss on how to evaluate IndicTrans translations.

Score | hi | te | pa | bn | as | gu | ta | ml | kn | mr | or
---|---|---|---|---|---|---|---|---|---|---|---
HS1 | 88 | 88 | 91 | 87 | 87 | 89 | 89 | 87 | 89 | 86 | 88
HS2 | 81 | 84 | 93 | 83 | 84 | 89 | 87 | 87 | 87 | 87 | 90
PC | 73 | 73 | 89 | 79 | 78 | 79 | 76 | 85 | 83 | 83 | 75
SC | 82 | 87 | 94 | 90 | 88 | 85 | 88 | 93 | 86 | 89 | 85
Table 1: Human Validation Score ($\times$ 10-2): HS1, HS2 represents human1,
human2 annotation score respectively. PC and SC represents Pearson and
Spearman correlation respectively.

##### Human Validation:

We followed SemEval-2016 Task-I (Agirre et al., 2016) guidelines. We hired 2
annotators per languages and calculated the Pearson (Kirch, 2008) and Spearman
(spe, 2008) correlation over annotations scores of sentences.

##### Diverse Sampling:

Since human validation is time-consuming and expensive. We sampled 100 diverse
sentences of the test set for validation. We apply the Determinantal Point
Process Kulesza (2012) (DPP) over sentence representations for diverse
sampling. DPP maximizes coverage volume using a minimal sampled set, thus
guaranteeing diversity during sampling. We first used sentence transformers to
convert data to BERT embeddings, and then use k-DPP Kulesza and Taskar (2011)
with k = 100 to sample 100 examples. Using DPP for diverse sampling is a cost-
effective method of evaluating translation quality. For scoring guidelines
refer to Appendix §A.

##### Hiring Experts:

We recruited, 2 speakers for each of the 11 _indic_ languages as annotators.
These professional annotators are multilingual (English, _Indic_) and fluent
in both mother-tongue _indic_ and English language. The remuneration paid was
6.6 cents per sentence for each _indic_ language. Demographic information on
annotators will be released together with the dataset.

##### Evaluation:

Table 1 shows the final human evaluation scores. In general, we see that
average human scores is more than 0.85 for all languages. The Pearson and
Spearman Correlation values are more than 0.7 and 0.8 for all languages
respectively. High human ratings and high correlation between the annotations
support high quality IndicTrans translation, hence validating IndicXNLI
quality.

BS | hi | te | pa | bn | as | gu | ta | ml | kn | mr | or
---|---|---|---|---|---|---|---|---|---|---|---
ETGT | 94 | 93 | 92 | 94 | NA | 94 | 94 | 94 | 94 | 94 | 94
ETIT | 98 | 94 | 94 | 98 | 93 | 94 | 94 | 94 | 94 | 93 | 93
MLGT | 90 | 88 | 86 | 89 | NA | 89 | 86 | 85 | 88 | 87 | 82
MLIT | 96 | 87 | 88 | 96 | 85 | 96 | 87 | 87 | 87 | 86 | 86
Table 2: BS represent BERTScore (F1-Score $\times$ 10-2) for EngTrans (ET) and
Multilingual (ML) strategies. Superscript GT and IT represent Google Translate
and IndicTrans models respectively.

##### Automatic Validation:

Given the absence of _Indic_ language XNLI reference data, we use BERTScore
similarity between the original English and English translated IndicXNLI for
automatic evaluation. Here too, we use the IndicTrans model for translating
IndicXNLI into English. This approach estimates the upper bound on error for
the English to _Indic_ translation (i.e. IndicXNLI quality), as it
approximates the combined error of both English to _Indic_ translation
(IndicXNLI creation), and _Indic_ to English translation (evaluation) (Rapp,
2009; Miyabe and Yoshino, 2015; Edunov et al., 2020; Behr, 2017). We utilize
BERTScore for assessment since it correlates better with human judgment at the
sentence level than BLEU (Zhang* et al., 2020; Papineni et al., 2002).

We evaluate two translation models, Google Translate and IndicTrans on the
testsets of IndicXNLI dataset. We incorporate Google Translate111Google
Translate was accessed on 30th June 2022 to demonstrate IndicTrans’s
competitiveness in comparison to commercial translation approaches. In Table
2, we used two evaluation strategies for our evaluation  (a.) _EngTrans_ :
which take the IndicXNLI sentence and translated it back to English using BERT
model.  (b.) _Multilingual_ : directly compare the English sentences with
multilingual IndicXNLI sentences using mBERT model.

On _Indic_ languages, we notice that IndicTrans is comparable to, and
sometimes outperforms, Google Translate. Additionally, when results are
compared in a Multilingual setting, we observe a marginal decrement in scores.
This can be because mBERT does not produce as precise multilingual embedding
as BERT does for English. Additionally, we see a similar pattern in the
distribution of scores across languages for both assessment strategies on both
models. We also computed the BERTScore (using mBERT) between the Hindi test
set of XNLI and IndicXNLI was found to be 0.87, supporting the high quality of
IndicXNLI.

##### Why Machine Translation?

While machine translation is an extremely convenient and quick method for
creating a synthetic dataset for multi-lingual NLI tasks for low-resource
languages such as indic, they are prone to contain some ‘translationese’
errors despite being meaning-preserving translationsGraham et al. (2020).
However, similar mistakes are still conceivable with manual translation, since
humans with knowledge of both English and Indian languages may translate the
text with ‘translationese’ tendencies owing to mother tongue impact (Delbio et
al., 2018). The ideal strategy would be to create an NLI dataset from scratch,
with speakers of those languages creating the resource directly, ensuring that
it represents culturally significant topics and inferences. However, this
technique is significantly more expensive and time consuming than manually
translating the data set and is, in most situations, impracticable. This is
because finding fluent bilingual speakers to do 10,000 translations for all 11
languages is very hard.

##  3 Experiments

Our experiments compare the performance of several multi-lingual models,
including one particularly developed for _Indic_ languages. We consider 2
broad categories,  (a) Indic Specificwhich includes IndicBERT and MuRIL due to
their indic specific pretraining, and  (b) Genericwhich includes mBERT and
XLM-Roberta due to their pretraining in more than 100 languages.  We fine-
tuned pre-trained multi-lingual models to develop NLI classifiers. The
classifiers takes two sentence as input, i.e. the premise and the hypothesis
and predicts the inference label. See appendix §B for models and hyper-
parameters details respectively.

###  3.1 Experimental Setup

In this section, we further elaborate upon categories of models used and
training strategies employed.

####  3.1.1 Details: Multi-lingual Models

We explore two categories of multilingual models in our experiments, as
detailed below:

#####  _Indic_ Specific:

These models are specially pre-trained using Mask Language Modeling (MLM) or
Translation Language Model (TLM) (Conneau and Lample, 2019) on monolingual /
bilingual _Indic_ language corpora. These include models such as MuRIL and
IndicBERT trained on 17 and 11 _Indic_ languages ($+$English) respectively.
MuRIL is pre-trained using Common-Crawl Oscar Corpus (Ortiz Su’arez et al.,
2019), PMIndia (Haddow and Kirefu, 2020) on the following languages: _en, hi,
bn, gu, te, ta, or, ml, pa, kn, mni, as, ur_. IndicBERT is pre-trained using
_Indic_ -CorpKakwani et al. (2020) on the following languages: _en, hi, bn,
ta, ml, te, Mr, kn, gu, pa, or, as_. Moreover, MuRIL is also pre-trained with
TLM objective (with MLM objective) on machine translated data and machine
transliterated data.

#####  _Generic_ :

These are massive multi-lingual models pre-trained on large number of
languages with MLM. These include multi-lingual BERT i.e. mBERT
(cased/uncased) and multi-lingual RoBERTa i.e. XLM-RoBERTa which are trained
on more than 100 languages. XLM-RoBERTa also includes pre-training on all
eleven _Indic_ languages. XLM-RoBERTa is pre-trained using the common crawl
monolingual data. mBERT (cased/uncased) includes pre-training on nine of
eleven _Indic_ languages (Assamese and Odia excluded) and uses multi-lingual
Wikipedia data for pre-training.

####  3.1.2 Training-Evaluation Strategies.

To train the NLI classifier, we investigate several strategies.

  1. 1.

_Indic_ Train: The models are trained and evaluated on IndicXNLI. The training
set is translated from the XNLI English, thus a translate-train scenario.

  2. 2.

English Train: The models are trained on original English XNLI data and
evaluated on IndicXNLI data. This is a zero-shot evaluation training scenario.

  3. 3.

English Eval: The model are trained on original English XNLI data, but
evaluated on English translation of IndicXNLI data. This is the translate-test
scenario.

  4. 4.

English + _Indic_ Train: This approach combines approaches _(1)_ and _(2)_.
The model is first pre-finetuned (Lee et al., 2021; Aghajanyan et al., 2021)
on English XNLI data and then finetuned on _Indic_ language of IndicXNLI data.

  5. 5.

Train All: This approach begins by fine-tuning the pre-trained model on
English XNLI data, followed by training on all eleven _Indic_ languages of
IndicXNLI sequentially.

  6. 6.

Cross Lingual Transfer: Additionally, we assess the models’ capacity to
transfer between languages. Where the model is trained on a single Indian
language and then assessed on all other Indian languages as well as the
training language.

  7. 7.

Intra-Bilingual Inference: Lastly, We also asses the model’s capability to
perform natural language inference with premise in English and hypothesis in
Indic language.

While the pre-trained multi-lingual models remain constant, the training and
evaluation datasets vary.

Model | Indic Train |  ModAvg | English Train |  ModAvg
---|---|---|---|---
|  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |  |  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |
XLM-R |  70 |  73 |  75 |  70 |  75 |  32 |  71 |  76 |  76 |  76 |  78 |  70 |  65 |  66 |  69 |  69 |  67 |  67 |  61 |  71 |  69 |  69 |  73 |  69
iBERT |  67 |  69 |  68 |  60 |  68 |  69 |  73 |  37 |  62 |  70 |  68 |  65 |  57 |  63 |  53 |  42 |  59 |  57 |  66 |  41 |  56 |  48 |  63 |  60
mBERT |  71 |  62 |  69 |  71 |  71 |  35 |  70 |  70 |  69 |  67 |  74 |  66 |  51 |  57 |  57 |  57 |  54 |  34 |  59 |  61 |  59 |  57 |  67 |  59
MuRIL |  70 |  78 |  75 |  76 |  70 |  76 |  72 |  74 |  78 |  75 |  71 |  74 |  68 |  32 |  75 |  34 |  68 |  67 |  70 |  74 |  71 |  74 |  76 |  72
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
LangAvg |  68 |  69 |  70 |  69 |  70 |  49 |  71 |  65 |  70 |  70 |  72 |  68 |  58 |  55 |  64 |  52 |  61 |  52 |  63 |  61 |  63 |  62 |  68 |  63

Model | English Eval |  ModAvg | English+_Indic_ Train |  ModAvg
---|---|---|---|---
|  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |  |  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |
XLM-R |  66 |  72 |  70 |  68 |  66 |  65 |  72 |  69 |  72 |  71 |  75 |  70 |  73 |  75 |  77 |  75 |  74 |  73 |  75 |  75 |  73 |  75 |  79 |  76
iBERT |  63 |  66 |  68 |  61 |  65 |  65 |  66 |  63 |  63 |  72 |  72 |  66 |  67 |  72 |  65 |  62 |  59 |  59 |  74 |  63 |  66 |  69 |  74 |  70
mBERT |  62 |  64 |  67 |  65 |  61 |  60 |  66 |  67 |  66 |  75 |  72 |  66 |  67 |  70 |  69 |  70 |  70 |  39 |  71 |  73 |  70 |  70 |  71 |  69
MuRIL |  65 |  33 |  71 |  67 |  67 |  67 |  71 |  31 |  71 |  72 |  77 |  63 |  76 |  77 |  77 |  79 |  74 |  76 |  77 |  77 |  74 |  75 |  77 |  77
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
LangAvg |  64 |  60 |  68 |  65 |  63 |  64 |  69 |  60 |  68 |  73 |  74 |  66 |  69 |  73 |  70 |  72 |  68 |  56 |  73 |  72 |  70 |  72 |  75 |  72

Model | Train All |  ModAvg | Cross Lingual Transfer |  ModAvg
---|---|---|---|---
|  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |  |  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |
XLM-R |  73 |  77 |  74 |  76 |  72 |  73 |  77 |  77 |  76 |  77 |  77 |  75 |  66 |  70 |  33 |  34 |  70 |  35 |  68 |  70 |  70 |  71 |  72 |  60
iBERT |  63 |  74 |  59 |  51 |  69 |  66 |  75 |  60 |  67 |  70 |  74 |  66 |  59 |  60 |  59 |  54 |  60 |  60 |  60 |  56 |  59 |  58 |  60 |  59
mBERT |  63 |  69 |  69 |  71 |  70 |  33 |  71 |  69 |  70 |  74 |  72 |  66 |  57 |  59 |  60 |  59 |  58 |  33 |  59 |  60 |  59 |  60 |  60 |  57
MuRIL |  73 |  76 |  74 |  76 |  74 |  78 |  81 |  78 |  76 |  80 |  78 |  77 |  75 |  73 |  75 |  76 |  71 |  33 |  75 |  76 |  73 |  75 |  73 |  70
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
LangAvg |  68 |  73 |  69 |  68 |  71 |  58 |  75 |  71 |  71 |  75 |  74 |  70 |  63 |  64 |  57 |  56 |  63 |  39 |  64 |  64 |  64 |  65 |  65 |  60

Table 3: Here, LangAvg represents the language wise average score across
models, while ModAvg average score represents the model average score across
languages. Values in Blue, Red and Green represents the model average best
score, language-wise average best score, and values where both model-wise and
language-wise best score coincide. For Indic Cross Lingual Transfer, each row
represent the average evaluation score of all _Indic_ language when trained on
the column language. For more detailed cross-lingual transfer results refer to
Appendix §C. iBERT stand for _indic_ BERT and XLM-R stand for XLM-RoBERTa.

###  3.2 Results and Analysis.

We summarizes our findings from Table §3 results across four categories:

##### Across Models:

In all experiments, MuRIL performs the best across all _indic_ languages
except in English Eval setup. This can be attributed to  (a.) The large model
size  (b.) indic-specific pre-training data,  (c.) A Mixture of Masked
Language Modeling (MLM), Translation Language Modeling (TLM), and  (d.) use of
transliterated data in pre-training.  XLM-RoBERTa beats MuRIL in rare
scenarios, notably in which the model solely deals with English data (e.g.
English Eval). XLM-RoBERTa outperforms MuRIL in such cases because it is
better at assessing English than MuRIL, which is designed mostly for indic
language. Additionally, we discover that, compared to XLM-RoBERTa, MuRIL
indic-specific training further enhances the model’s performance. Despite
indic-specific pretraining, IndicBERT performs worse than mBERT. This can be
attributed to the smaller size of the IndicBERT model, i.e. only 33M compared
to 167M mBERT (c.f. Table §5 in appendix).

##### Across Language:

We see a strong positive correlation between language performance with their
resource availability. Hindi and Bengali outperform, whereas Odia mostly
underperform on majority of benchmarks. Low-resource languages such as
Marathi, Assamese, and Kannada surprising also perform well. This can be
attributed to the similarity of Marathi with Hindi script, Assamese with
Bengali script, and Kannada with Tamil and Telugu scripts. This is discussed
in detail in appendix 3.3. Odia, a low resource language, lacks script sharing
language partners and hence performs poorly. Overall, English + Indic Train
method outperforms, with MuRIL performing best.

##### Across Strategies:

Our experiments show that models benefit from language-specific fine tuning.
English + Indic train and Train All have the best results with minimal
deviation across languages for XLM-R and MuRIL. Additionally, _Train All_
follows a high-to-low resource hierarchy to mitigate the impact of
catastrophic forgetting (Goodfellow et al., 2015). Due to the followed
language order English + Indic train outperform Train All setting marginally
for high resource languages. Overall, English + Indic Train strategy performs
the best and MuRIL performs the best in that strategy. This can be attributed
to the _indic_ specific pre-training process of MuRIL which include both
translation and transliteration. Furthermore, MuRIL has the second largest
size after XLM-R.

##### Cross-Lingual Transfer:

Models favour high resource languages such as _Hindi_ and _Bengali_ training
for cross-lingual transfer. These language are pre-trained on large mono-
lingual corpora which enhanced performance (Conneau et al., 2020). This
setting can be thought equivalent of _Hindi_ and _Bengali_ substitution for
English training. Additionally, when evaluated for all _indic_ languages,
model trains on non-_Hindi_ and non-_Bengali_ perform substantially better for
_Hindi_ and _Bengali_. Table 3 present results summary as average evaluation
score across all _indic_ language(rows) when train on the several _indic_
languages (columns). 222For model-wise cross-lingual results c.f. Appendix §C.

##### Intra-Bilingual Inference:

Model | English+Indic Train |  ModAvg | Train All |  ModAvg
---|---|---|---|---
|  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |  |  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |
XLM-R |  74 |  72 |  75 |  74 |  77 |  72 |  70 |  72 |  72 |  79 |  76 |  74 |  57 |  59 |  58 |  62 |  61 |  53 |  57 |  59 |  61 |  63 |  63 |  59
iBERT |  70 |  68 |  63 |  65 |  69 |  68 |  71 |  64 |  64 |  69 |  69 |  67 |  49 |  53 |  46 |  37 |  52 |  51 |  59 |  39 |  51 |  57 |  50 |  50
mBERT |  51 |  56 |  59 |  50 |  62 |  31 |  63 |  57 |  60 |  61 |  63 |  56 |  39 |  39 |  43 |  38 |  43 |  33 |  40 |  42 |  41 |  40 |  42 |  40
MuRIL |  71 |  70 |  73 |  69 |  71 |  39 |  71 |  71 |  69 |  72 |  69 |  67 |  51 |  52 |  58 |  56 |  53 |  55 |  58 |  65 |  55 |  62 |  54 |  56
LangAvg |  65 |  65 |  66 |  64 |  68 |  53 |  62 |  65 |  67 |  71 |  70 |  65 |  47 |  49 |  51 |  48 |  51 |  45 |  52 |  50 |  51 |  54 |  51 |  50
Table 4: En-IndicXNLI model performance (refer §3.2) with English + _Indic_
train and Train All setting. Here, ModAvg, LangAvg, and Color Code mean same
as in table 3.

We also evaluate models on mixed input inference task En-IndicXNLI, which
consists of English _premises_ paired with corresponding _indic_ hypothesis.
We train model on mixed input using English + _Indic_ Train and Train All
strategies. Table 4 shows performance of English + _Indic_ Train and Train All
models on En-IndicXNLI. Compared to uni-language inference task, mixed-
language input task perform poorly. Furthermore, contrary to earlier
observations, generic model such as XLM-R outperforms the _Indic_ specific
models. However, IndicBERT and MuRIL both perform substantially better than
mBERT. Furthermore, English data augmentation enhance the English + _Indic_
Train setting performance. This can be because, the model "meta-learns" the
task successfully with English data training (premise language), and further
prioritises the model’s language-specific abilities with the follow-up indic
data training.

Results Analysis. We observed a performance loss except for XLM-RoBERTa when
the model is evaluated on En-IndicXNLI inference task. The inference models
struggle to correlate and reason together on two different languages (English,
_Indic_) sentences. Contrary to earlier observation, a generic model such as
XLM-RoBERTa outperforms the _Indic_ specific models. However, IndicBERT and
MuRIL perform better than mBERT. _Bengali_ perform best for both the training
strategies. We also observe the benefit of English data augmentation English +
_Indic_ Train model, rather than all language augmentation Train All model.

###  3.3 Error Analysis

(a) Tamil vs Kannada (b) Bengali vs Assamese (c) Hindi vs Marathi

Figure 1: Consistency Matrix: Predictions of MuRIL for (a) Tamil vs Kannada
(b) Bengali vs Assamese, , (c) Hindi vs Marathi. The percentage on top in each
block represents the average across all three labels with each label
percentage given below it in the order of Entailment, Neutral and
Contradiction.

(a) Tamil vs Kannada (b) Bengali vs Assamese (c) Hindi vs Marathi

Figure 2: Confusion Matrix: for MuRIL (a) Tamil vs Kannada, (b) Bengali vs
Assamese, (c) Hindi vs Marathi.

(a) Confusion Matrix (b) Consistency Matrix

Figure 3: Consistency Matrix and Confusion Matrix for Predictions of MuRIL on
Original Hindi data in XNLI and Machine Translated Data generated from
IndicTrans.

In this section we investigate the correlation between the language similarity
and the model performance. We see that the model performs similarly on similar
languages. We evaluate our results on MuRIL on the English+_Indic_ finetuning
Strategy.

In Figure 1, we observe that the overall Correct and Incorrect predictions,
Bengali vs Assamese pair has the total of 81% overlap,Tamil vs Kannada has 83%
overlap, Hindi vs Maratha has 82% overlap. All the language pairs have the
largest overlap for entailment label for correct labels and largest overlap in
contradiction label for incorrect overlaps.

In Figure 2, interestingly Bengali vs Assamese pair and Hindi vs Marathi has
the highest percentage of overlap in predictions where the most overlap is in
entailment and minimum overlap is in contradiction. While for Tamil vs Kannada
pair has the highest overlap for neutral and minimum for contradiction.

We have also done error analysis of model performance on original hindi test
data already present in XNLI and data obtained through translations from
IndicTrans in Figure 3. We observe that there is a total of 82% of overlap in
error consistency and we again observe that the most number of correct
overlaps in for entailment label and most number of incorrect predictions are
for contradiction label. In terms of consistency, we see the maximum overlap
in neutral prediction and least overlap in contradiction prediction. This
shows that model performs similarly on the original Hindi data and machine
translated Hindi data enhancing the validity of the IndicXNLI dataset.

##  4 Discussion

##### Why _Indic_ languages?

Indic languages are spoken by more than a billion people in the Indian
subcontinent. With the introduction of IndicNLPSuite (Kakwani et al., 2020) by
AI4Bharat333https://ai4bharat.org there has been has an increased interest and
effort towards the research for _Indic_ languages model. Recently, IndicBERT,
MuRIL (Khanuja et al., 2021) based on BERT (Devlin et al., 2019) were
introduced for the _Indic_ languages. Furthermore, generation model IndicTrans
(Ramesh et al., 2021) and IndicBART (Dabre et al., 2021) based on seq2seq
architecture was also published recently. These model use the _Indic_ enrich
monolingual corpora: Common Crawl, Oscar and IndicCorp and parallel corpora:
Samantar and PMIndia (Haddow and Kirefu, 2020) on _Indic_ languages for
training. Despite significant progress through large transformer-based _Indic_
language models in addition to existing multilingual models e.g. mBERT (Devlin
et al., 2019) , XLM-RoBERTa (Conneau et al., 2020), and mBART (seq2seq) Liu et
al. (2020) there is currently a paucity of benchmark data-sets for evaluating
these huge language models in the _Indic_ language research field. Such
benchmark dataset is necessary for studying the linguistic features of Indic
languages and how well they are perceived by different multilingual models.
Recently, IndicGLUE Kakwani et al. (2020) was introduced to handle this
scarcity. However, the scope of this benchmark, is confined to only few tasks
and datasets.

##### Why IndicXNLI task?

This research provides an excellent chance to investigate the efficacy of
various Multilingual models on _Indic_ languages that are rarely evaluated or
explored before. Some of these _Indic_ languages such as _‘Assamese’_ and
_‘Odia’_ serve as unseen (zero-shot) evaluation for models such as mBERT
(Pires et al., 2019), i.e. not pre-trained on _‘Assamese’_. While other
models, such as XLM-RoBERTa, IndicBERT and MuRIL covers all our languages but
in widely varying proportions in their training data. Our work investigate the
correlation effect of cross-lingual training for English on these rare _Indic_
languages, which are not explore by prior studies.

Furthermore, we also investigate the cross-lingual transfer effect across
_Indic_ languages, also not explored before. We explore the impact of
Multilingual training, english-data augmentation, unified Indic model
performance, cross-lingual transfer of closely related _Indic_ family and
English-_Indic_ NLI through our work. All the above mention topics are not
explore for _Indic_ language before. We aim to integrate IndicXNLI and
benchmark models in IndicGLUE Kakwani et al. (2020). Such a benchmark dataset
is required for investigating the linguistic properties of Indian languages
and how accurately they are interpreted by various multilingual models.
Another direction is accessing model performance on Indic-IndicXNLI task,
where both premises and hypothesis are in two distinct _Indic_ languages.

##### Why IndicTrans for Translation?

We use the IndicTrans as a translation model for converting English XNLI to
IndicXNLI because of the following reasons:

  * •

Open-Source: IndicTrans is open-source to public for non-commercial usage
without additional fees, while Google-Translate and Microsoft-Translate
require paid subscription.

  * •

Light Weight: IndicTrans is the fastest and the lightest amongst mBART and mT5
on single-core GPU machines. Google-Translate and Microsoft-Translate are also
relatively slower due to repeated network-intensive API calls.

  * •

_indic_ Coverage: Seq2Seq models like mBART and mT5 are not designed for all
languages in the indic family. mBART supports eight (excludes kn,or,pa,as)
while mT5 supports nine languages (excludes or,as) out of eleven indic
languages. Google-Translate supports ten out of eleven _indic_ languages
(excludes Assamese). Microsoft Translate supports all the eleven _indic_
languages.

In future, we plan to enhance IndicXNLI with better translation methods.

##  5 Related Work

Recently many _Indic_ -specific resources are developed such as IndicNLPSuite
Kakwani et al. (2020), which include  (a.) word embeddings: IndicFT,  (b.)
transformer models: IndicBERT,  (c.) monolingual corpora: IndicCorp,  (d.)
and, evaluation benchmark: IndicGLUE  . Furthermore, _Indic_ -specific pre-
processing libraries such as iNLTK (Arora, 2020) and Indic-nlp-library
(Kunchukuttan, 2020), other Indic monolingual corpora: Common Crawl Oscar
Corpus Wenzek et al. (2020); Ortiz Suárez et al. (2020), multilingual parallel
corpora: PMIndia Haddow and Kirefu (2020) and Samantar Ramesh et al. (2021),
transformer model MuRIL Khanuja et al. (2021) and language specific Indic-
Transformers (Jain et al., 2020) exists.

##  6 Conclusion

With IndicXNLI we extend the XNLI dataset for _Indic_ languages family. We
benchmark IndicXNLI with several multi-lingual models using various train-test
strategies. We also study the use of English XNLI as pre-finetuning dataset.
Furthermore, we also evaluate models on mixed-language inference input and
cross-lingual transfer ability. We aim to integrate IndicXNLI and benchmark
models in IndicGLUE Kakwani et al. (2020). We also intend to enhance IndicXNLI
with advanced translation techniques. Another direction is accessing model
performance on Indic-IndicXNLI task, where both premises and hypothesis are in
two distinct _Indic_ languages.

## Acknowledgement

We thank members of the Utah NLP group for their valuable insights and
suggestions at various stages of the project; and reviewers their helpful
comments. We would also like to thank Suhani Aggarwal, Shibani Krishnatraya
and Ayush Dhall for participating in the dataset verification activity and
helping us find fluent speakers in many different indic languages.
Additionally, we appreciate the inputs provided by Vivek Srikumar and Ellen
Riloff. Vivek Gupta acknowledges support from Bloomberg’s Data Science Ph.D.
Fellowship.

## References

  * spe (2008) 2008\.  _Spearman Rank Correlation Coefficient_, pages 502–505. Springer New York, New York, NY.
  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021.  Muppet: Massive multi-task representations with pre-finetuning.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 497–511, San Diego, California. Association for Computational Linguistics.
  * Arora (2020) Gaurav Arora. 2020.  iNLTK: Natural language toolkit for indic languages.  In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_ , pages 66–71, Online. Association for Computational Linguistics.
  * Behr (2017) Dorothée Behr. 2017.  Assessing the use of back translation: the shortcomings of back translation as a quality testing method.  _International Journal of Social Research Methodology_ , 20(6):573–584.
  * Bianchi et al. (2021) Federico Bianchi, Debora Nozza, and Dirk Hovy. 2021.  Language invariant properties in natural language processing.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  _Advances in Neural Information Processing Systems_ , 32:7059–7069.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Dabre et al. (2021) Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra, and Pratyush Kumar. 2021.  Indicbart: A pre-trained model for natural language generation of indic languages.
  * Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013.  Recognizing textual entailment: Models and applications.  _Synthesis Lectures on Human Language Technologies_ , 6(4):1–220.
  * Delbio et al. (2018) A. Delbio, R. Abilasha, and m. Ilankumaran. 2018.  Second language acquisition and mother tongue influence of english language learners – a psycho analytic approach.  _International Journal of Engineering and Technology_ , 7(4.36):497–500.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Dumitrescu et al. (2021) Stefan Daniel Dumitrescu, Petru Rebeja, Beata Lorincz, Mihaela Gaman, Andrei Avram, Mihai Ilie, Andrei Pruteanu, Adriana Stan, Lorena Rosia, Cristina Iacobescu, Luciana Morogan, George Dima, Gabriel Marchidan, Traian Rebedea, Madalina Chitez, Dani Yogatama, Sebastian Ruder, Radu Tudor Ionescu, Razvan Pascanu, and Viorica Patraucean. 2021.  Liro: Benchmark and leaderboard for romanian language tasks.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Edunov et al. (2020) Sergey Edunov, Myle Ott, Marc’Aurelio Ranzato, and Michael Auli. 2020.  On the evaluation of machine translation systems trained with back-translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2836–2846, Online. Association for Computational Linguistics.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2015\.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.
  * Graham et al. (2020) Yvette Graham, Barry Haddow, and Philipp Koehn. 2020.  Statistical power and translationese in machine translation evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 72–81, Online. Association for Computational Linguistics.
  * Haddow and Kirefu (2020) Barry Haddow and Faheem Kirefu. 2020.  Pmindia – a collection of parallel corpora of languages of india.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  _CoRR_ , abs/2003.11080.
  * Jain et al. (2020) Kushal Jain, Adwait Deshpande, Kumar Shridhar, Felix Laumann, and Ayushman Dash. 2020.  Indic-transformers: An analysis of transformer language models for indian languages.
  * K et al. (2021) Karthikeyan K, Aalok Sathe, Somak Aditya, and Monojit Choudhury. 2021.  Analyzing the effects of reasoning types on cross-lingual transfer performance.  In _EMNLP 2021_.
  * Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020.  IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4948–4961, Online. Association for Computational Linguistics.
  * Khanuja et al. (2021) Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021.  Muril: Multilingual representations for indian languages.
  * Kirch (2008) Wilhelm Kirch, editor. 2008.  _Pearson’s Correlation Coefficient_, pages 1090–1091. Springer Netherlands, Dordrecht.
  * Kulesza (2012) Alex Kulesza. 2012.  Determinantal point processes for machine learning.  _Foundations and Trends® in Machine Learning_ , 5(2-3):123–286.
  * Kulesza and Taskar (2011) Alex Kulesza and Ben Taskar. 2011.  K-dpps: Fixed-size determinantal point processes.  In _Proceedings of the 28th International Conference on International Conference on Machine Learning_ , ICML’11, page 1193–1200, Madison, WI, USA. Omnipress.
  * Kunchukuttan (2020) Anoop Kunchukuttan. 2020.  The IndicNLP Library.  https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf.
  * Lee et al. (2021) Hung-yi Lee, Ngoc Thang Vu, and Shang-Wen Li. 2021.  Meta learning and its applications to natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Tutorial Abstracts_ , pages 15–20, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.
  * Miyabe and Yoshino (2015) Mai Miyabe and Takashi Yoshino. 2015.  Evaluation of the validity of back-translation as a method of assessing the accuracy of machine translation.  In _2015 International Conference on Culture and Computing (Culture Computing)_ , pages 145–150.
  * Ortiz Suárez et al. (2020) Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020.  A monolingual approach to contextualized word embeddings for mid-resource languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1703–1714, Online. Association for Computational Linguistics.
  * Ortiz Su’arez et al. (2019) Pedro Javier Ortiz Su’arez, Benoit Sagot, and Laurent Romary. 2019.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-Institut f"ur Deutsche Sprache.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Radford and Narasimhan (2018) Alec Radford and Karthik Narasimhan. 2018.  Improving language understanding by generative pre-training.
  * Ramesh et al. (2021) Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2021\.  Samanantar: The largest publicly available parallel corpora collection for 11 indic languages.
  * Rapp (2009) Reinhard Rapp. 2009.  The back-translation score: Automatic mt evaluation at the sentence level without reference translations.  In _Proceedings of the ACL-IJCNLP 2009 Conference Short Papers_ , ACLShort ’09, page 133–136, USA. Association for Computational Linguistics.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021\.  XTREME-R: Towards more challenging and nuanced multilingual evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Uppal et al. (2020) Shagun Uppal, Vivek Gupta, Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah, and Amanda Stent. 2020.  Two-step classification using recasted data for low resource settings.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 706–719, Suzhou, China. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France. European Language Resources Association.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.

##  Appendix A Human Validation Scoring

We provide English and _indic_ language IndicXNLI (IndicTrans translated)
sentence to the recruited native speaker of that _indic_ language for
validation. Before the annotation work, each expert was given a full
explanation of the guidelines that needed to be followed. The validation
instructions (mturk template and detailed examples) are taken from the
Semeval-2016 Task-I. The native speaker access the sentence pairs assign an
integer score between 0 and 5, as follows:  0: The two sentences are
completely dissimilar.  1: The two sentences are not equivalent, but are on
the same topic.  2: The two sentences are not equivalent, but share some
details.  3: The two sentences are roughly equivalent, but some important
information differs/missing.  4: The two sentences are mostly equivalent, but
some unimportant details differ.  5: The two sentences are exactly equivalent,
as they mean the same.  The score depicts the goodness of translated sentence
in terms of semantics, i.e. same meaning as original English sentence444For
NLI task, same syntax, i.e. grammar (e.g. Tense) lesser important than same
Semantic, i.e. meaning preservation.. Scores are then normalized to a
probability range (between 0 and 1). The final validation score for each
language is determined as the average of all 100 instances’ scores.

Additionally, we also computed the BERTScore between the English and the Hindi
test split of the XNLI555XNLI hindi test splits was human translated., using
multi-lingual strategy which came out to be 70 ($\times$ 10${{}^{-}2}$). We
presume that the lower score is attributable to the fact that human-translated
dataset encapsulates a large number of linguistic nuances, resulting in a
change in the structure and tonality of the sentences, which is frequently
overlooked by machine translation systems, as highlighted by Bianchi et al.
(2021).

##  Appendix B Details: Hyper Parameters Settings

All the models were trained on google collaboratory
666https://colab.research.google.com/ on TPU-v2 with 8 cores. The code was
built in the PyTorch-lightning framework. We used accuracy as mentioned in the
original XNLI paper Conneau et al. (2018) as our metric of choice. The
training was run with an early stopping callback with the patience of 3,
validation interval of 0.5 epochs and AdamW as optimizer(Loshchilov and
Hutter, 2019).

Model | PO | CU | LR | BS | WD | MSL | MS | WS
---|---|---|---|---|---|---|---|---
XLM-R | MLM (Dynamic) | Wikipedia Corpus | 2e-5 | 64 | 0.01 | 128 | 278M | 1500
iBERT | MLM | IndicCorp | 2e-5 | 128 | 0.01 | 128 | 33.7M | 1500
MuRIL | MLM, TLM and TrLM | OSCAR and PM India | 2e-5 | 64 | 0.01 | 128 | 237M | 1500
mBERT | MLM | Wikipedia Corpus | 2e-5 | 128 | 0.01 | 128 | 177M | 1500
Table 5: Model Hyper-Parameters

In Table 5 the hyperparamaters are abbreviated as mentioned below:  (a.)
PO:Pre-training Objective. Where MLM stands for masked Language Modelling, TLM
stands for Translation Language Modelling and TrLM stands for Transliteration
Language Modelling,  (b.) CU:Corpus Used,  (c.) LR:Learning Rate,  (d.)
BS:Batch Size,  (e.) WD:Weight Decay,  (f.) MSL:Maximum Sequence Length,  (g.)
MS:Model Size described as number of parameters in millions,  (h.) WS: Warm-up
Step.

##  Appendix C Indic Cross-lingual Transfer

TrLang | XLM-RoBERTa |  TrAvg | IndicBERT |  TrAvg
---|---|---|---|---
|  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |  |  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |
as |  64 |  67 |  66 |  67 |  63 |  63 |  68 |  68 |  64 |  66 |  65 |  66 |  65 |  63 |  54 |  46 |  61 |  60 |  66 |  48 |  57 |  67 |  60 |  58
gu |  65 |  72 |  69 |  69 |  68 |  71 |  70 |  71 |  65 |  74 |  74 |  70 |  61 |  67 |  54 |  41 |  65 |  64 |  70 |  46 |  62 |  70 |  62 |  60
kn |  33 |  31 |  35 |  35 |  31 |  34 |  32 |  31 |  32 |  33 |  32 |  33 |  58 |  64 |  68 |  48 |  59 |  59 |  65 |  46 |  59 |  63 |  63 |  59
ml |  35 |  33 |  33 |  34 |  31 |  34 |  34 |  31 |  33 |  34 |  34 |  33 |  55 |  52 |  54 |  60 |  53 |  53 |  52 |  52 |  57 |  52 |  52 |  54
mr |  66 |  74 |  70 |  72 |  72 |  68 |  70 |  69 |  65 |  75 |  73 |  71 |  62 |  65 |  54 |  48 |  61 |  61 |  67 |  52 |  60 |  68 |  63 |  60
or |  35 |  33 |  32 |  36 |  35 |  34 |  34 |  36 |  34 |  36 |  36 |  35 |  61 |  66 |  57 |  49 |  61 |  66 |  65 |  48 |  60 |  68 |  64 |  60
pa |  65 |  69 |  70 |  67 |  67 |  67 |  70 |  66 |  67 |  73 |  66 |  68 |  61 |  67 |  55 |  47 |  60 |  62 |  74 |  41 |  60 |  70 |  62 |  60
ta |  64 |  67 |  69 |  72 |  71 |  68 |  71 |  70 |  70 |  73 |  70 |  70 |  55 |  60 |  53 |  49 |  56 |  54 |  58 |  59 |  55 |  58 |  55 |  56
te |  61 |  70 |  71 |  70 |  70 |  71 |  68 |  68 |  75 |  75 |  72 |  71 |  61 |  63 |  53 |  45 |  59 |  63 |  70 |  46 |  63 |  68 |  58 |  59
bn |  67 |  72 |  73 |  73 |  72 |  74 |  74 |  70 |  70 |  73 |  71 |  72 |  62 |  66 |  55 |  48 |  62 |  62 |  66 |  47 |  60 |  68 |  68 |  60
hi |  66 |  70 |  69 |  72 |  69 |  68 |  71 |  71 |  71 |  76 |  73 |  71 |  58 |  63 |  53 |  49 |  61 |  61 |  66 |  43 |  57 |  71 |  61 |  59
TestAvg |  56 |  60 |  60 |  61 |  59 |  59 |  60 |  59 |  59 |  63 |  61 |  60 |  60 |  63 |  55 |  48 |  60 |  60 |  65 |  48 |  59 |  66 |  61 |  59
TrLang | mBERT |  TrAvg | MuRIL |  TrAvg
|  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |  |  as |  gu |  kn |  ml |  mr |  or |  pa |  ta |  te |  bn |  hi |
as |  69 |  59 |  61 |  53 |  57 |  36 |  61 |  57 |  52 |  59 |  64 |  56 |  73 |  78 |  75 |  74 |  74 |  73 |  75 |  75 |  75 |  76 |  77 |  75
gu |  48 |  70 |  64 |  55 |  60 |  32 |  64 |  64 |  60 |  67 |  65 |  60 |  72 |  75 |  75 |  74 |  73 |  72 |  70 |  72 |  71 |  76 |  75 |  73
kn |  49 |  62 |  68 |  64 |  60 |  35 |  65 |  64 |  59 |  69 |  62 |  61 |  72 |  75 |  76 |  76 |  73 |  73 |  74 |  75 |  76 |  77 |  77 |  75
ml |  51 |  60 |  60 |  71 |  60 |  30 |  61 |  65 |  62 |  66 |  62 |  60 |  75 |  75 |  73 |  77 |  72 |  78 |  76 |  79 |  75 |  77 |  76 |  76
mr |  45 |  61 |  63 |  56 |  69 |  35 |  64 |  56 |  57 |  69 |  66 |  60 |  69 |  70 |  72 |  71 |  73 |  68 |  76 |  70 |  69 |  73 |  74 |  72
or |  34 |  33 |  29 |  32 |  36 |  35 |  34 |  35 |  33 |  33 |  34 |  33 |  33 |  36 |  35 |  30 |  32 |  35 |  30 |  30 |  33 |  32 |  36 |  33
pa |  47 |  65 |  59 |  59 |  62 |  35 |  70 |  63 |  61 |  68 |  64 |  61 |  73 |  75 |  76 |  74 |  74 |  76 |  79 |  71 |  74 |  75 |  75 |  75
ta |  48 |  64 |  67 |  63 |  60 |  32 |  65 |  66 |  63 |  69 |  62 |  61 |  74 |  76 |  76 |  77 |  75 |  72 |  74 |  77 |  76 |  80 |  78 |  76
te |  51 |  59 |  63 |  63 |  60 |  32 |  61 |  64 |  67 |  66 |  62 |  60 |  70 |  72 |  74 |  71 |  73 |  70 |  77 |  74 |  77 |  77 |  75 |  74
bn |  51 |  64 |  65 |  62 |  62 |  32 |  65 |  60 |  62 |  69 |  67 |  61 |  68 |  76 |  73 |  73 |  71 |  72 |  73 |  74 |  74 |  74 |  76 |  74
hi |  50 |  66 |  65 |  61 |  62 |  30 |  65 |  63 |  61 |  71 |  63 |  61 |  73 |  76 |  73 |  75 |  74 |  73 |  76 |  74 |  74 |  75 |  76 |  75
Test Avg |  49 |  60 |  60 |  58 |  59 |  33 |  61 |  60 |  58 |  64 |  61 |  58 |  68 |  71 |  71 |  70 |  69 |  69 |  71 |  70 |  70 |  72 |  72 |  71
Table 6: Indic Cross-lingual transfer

This section is the extension of the §3.2. Table 6 are the cross-lingual
transfer results of XLM-R, IndicBERT, mBERT and MuRIL respectively. The rows
of the table consist of the languages on which the model is trained, while the
columns represent the evaluation languages. E.g., in table 6 the first row
represents that the model is trained on _“as"_ and then tested on all the
languages in the column. The values in the row are the accuracy scores of the
model when trained on the language in its leftmost column and tested on the
language in its top-most row column.

##### XLM-R.

the model perform best for the “bn" language. The model gives the best
performance average across all other languages if trained on “bn". A model
trained in other languages, on average, also performs best for “bn" language.
XLM-R also struggles to correlate with “kn", “or", and “ml", thus performs
poorly on average if trained for them. At the same time, all models have poor
cross-lingual ability transferability for the “as" language.

##### IndicBERT.

the overall score is comparable to XLM-R despite it’s smaller size. On
average, across languages, the cross-lingual transfer ability for models
trained on varying _indic_ languages were consistently similar (b/w 0.5-0.6).
However, the evaluation performance for cross-lingual models evaluated on
_“ml"_ were poor for all _indic_ trained models. For model trained on some
languages, _“kn"_ , _“ml"_ and _“pa"_ , the best performance was across
diagonal, i.e. indicating the model performs best on the trained language.
This trend was, however, was not shown in other _indic_ languages, indicating
remarkable cross-lingual transfer ability of the IndicBERT model.

##### mBERT.

the model performs worse for “or" on average for both when evaluated and train
on. However, all models performs very consistently for other indic languages.
Model trained on kn, pa, ta, hi, and bn perform best on average across
languages. Here too, the best cross-lingual transfer ability was shown for bn
language. mBERT also have best performance across diagonal for some languages
e.g. “as", “gu", “ml", “pa" and “te".

##### MuRIL.

shows the best overall cross-lingual transfer ability amongst all the models.
MuRIL only fails to generalize well when trained for “or" language. However,
model train on other _indic_ language when evaluated on “or" performs well.
Model trained on “ta" and “ml" performs best across all languages. The best
cross-lingual transfer ability was shown for “bn" and “hi". Overall, MuRIL has
better cross-lingual transfer ability across all languages compared to other
models. It also shows less performance bias for languages such as “bn" and
“hi", as compared to XLM-R.