Combating Mode Collapse in GAN training: An Empirical Analysis using Hessian Eigenvalues

Ricard Durallsup1,2,4, Avraam Chatzimichailidissup1,3,4, Peter Labus sup1,4 and Janis Keupersup1,5
sup1Fraunhofer ITWM, Germany
sup2IWR, University of Heidelberg, Germany
sup3Chair for Scientific Computing, TU Kaiserslautern, Germany
sup4Fraunhofer Center Machine Learning, Germany
sup5Institute for Machine Learning and Analytics, Offenburg University, Germany

Generative adversarial networks (GANs) provide state-of-the-art results in image generation. However, despite being so powerful, they still remain very challenging to train. This is in particular caused by their highly non-convex optimization space leading to a number of instabilities. Among them, mode collapse stands out as one of the most daunting ones. This undesirable event occurs when the model can only fit a few modes of the data distribution, while ignoring the majority of them. In this work, we combat mode collapse using second-order gradient information. To do so, we analyse the loss surface through its Hessian eigenvalues, and show that mode collapse is related to the convergence towards sharp minima. In particular, we observe how the eigenvalues of the $G$ are directly correlated with the occurrence of mode collapse. Finally, motivated by these findings, we design a new optimization algorithm called nudged-Adam (NuGAN) that uses spectral information to overcome mode collapse, leading to empirically more stable convergence properties.

§ INTRODUCTION

Although Deep Neural Networks (DNNs) have exhibited remarkable success in many applications, the optimization process of DNNs remains a challenging task. The main reason for that is the non-convexity of the loss landscape of such networks. While most of the research in the field has focused on single objective minimization, such as classification problems, there are other models that require the joint minimization of several objectives. Among these models, Generative Adversarial Networks (GANs) [13] are particularly interesting, due to their success of learning entire probability distributions. Since their first appearance, they have been used to improve the performance
of a wide range of tasks in computer vision,
including image-to-image translation [1, 20],
image inpainting [17, 30],
semantic segmentation [29, 11] and many more.

Accepted in VISAPP 2021

GANs are a class of generative models which consist of a generator ($G$) and a discriminator ($D$) DNN model. Within an adversarial game they are trained in such a way that the $G$ learns to produce new samples distributed according to the desired data distribution.
Training can be formulated in terms of a minimax optimization of a value function $V(G, D)$

\begin{align}
\begin{split}
\min_G \max_D \; V(G,D).
\end{split}
\end{align}

While being very powerful and expressive, GANs are known to be notoriously hard to train.
This is because their training is equivalent to the search of Nash equilibria in a high-dimensional, highly non-convex optimization space. The standard algorithm for solving this optimization problem is gradient descent-ascent (GDA), where $G$ and $D$ perform alternating update steps using first order gradient information w.r.t. the loss function. In practise, GDA is often combined with regularization, which has yield many state-of-the-art results for generative models on various benchmark datasets. However, GDA is known to suffer from undesirable convergence properties that may lead to instabilities, divergence, catastrophic forgetting and mode collapse. The latter term refers to the scenario where only a few modes of the data distribution are generated and the model produces only a limited variety of samples.

Many recent works have studied different approaches to tackle these issues. In reference [26], for instance, was one of the first attempts to use convolutional neural networks in order to improve both training stability, as well as the visual quality of the generated samples. Other works achieved improvements through the use of new objective functions [28, 4], and additional regularization terms [15, 10].
There have also been recent advances in the theoretical understanding of GAN training.
References [25, 24], for examples, have investigated the convergence properties of GAN training using first-order information.
There, it has been shown that a local analysis of the eigenvalues of the Jacobian of the loss function can provide guarantees on local stability properties.
Moreover, going beyond first order gradient information, references [5, 12] have used the top k-eigenvalues of the Hessian of the loss to investigate the convergence and dynamics of GANs.

In this paper, we conduct an empirical study to obtain new insights concerning stability issues of GANs.
In particular, we investigate the general problem of finding local Nash equilibria by examining the characteristics of the Hessian eigenvalue spectrum and the geometry of the loss surface.  We thereby verify some of the previous findings that were based on the top k-eigenvalues alone. We hypothesize that mode collapse might stand in close relationship with convergence towards sharp minima, and we show empirical results that support this claim.  Finally, we introduce a novel optimizer which uses second-order information to combat mode collapse. We believe that our findings can contribute to understand the origins of the instabilities encountered in training GAN.

In summary our contributions are as follows

* We calculate the full Hessian eigenvalue spectrum during GAN training, allowing us to link mode collapse to anomalies of the eigenvalue spectrum.

* We identify similar patterns in the evolution of the eigenvalue spectrum of $G$ and $D$ by inspecting their top k-eigenvalues.

* We introduce a novel optimizer that uses second-order information to mitigate mode collapse.

* We empirically demonstrate that $D$ finds a local minimum, while $G$ remains in a saddle point.

§ RELATED WORK

While gradient-based optimization has been very successful in Deep Learning, applying gradient-based algorithms in game theory, i.e. finding Nash equilibria, has often highlighted their limitations. An intense line of research based on first- and second-order methods has studied the dynamics of gradient descent-ascent by investigating the loss landscape of DNNs. One of the initial first-order approaches [14] studied the properties of the loss landscape along a linear path between two points in parameter space. In doing so, it was demonstrated that DNNs tend to behave similarly to convex loss functions along these paths.
In later references [9] non-linear paths between two points were investigated. There, it was shown that the loss surface of DNNs contains paths that connect different minima, having constant low loss along these paths.

In the context of second-order approaches, there has also been notable progress [27, 3]. There, the Hessian w.r.t. the loss function was used to reduce oscillations around critical points in order to obtain faster convergence to Nash equilibria. The main advantage of second-order methods is the fact that the Hessian provides curvature information of the loss landscape in all directions of parameter space (and not only along the path of steepest descent as with first-order methods). However, this curvature information is local only and very expensive to compute. In the context of GANs, second-order methods have not been investigated in depth. Recent works [5, 12] have not calculated the full Hessian matrix but resorted to approximations, such as computing the top-k eigenvalues only. To the best of our knowledge, we are the first to use the full Hessian eigenvalue spectrum to study the training behavior of GANs.

Another line of research tries to classify different types of local critical points of the loss surface during training w.r.t. their generalization behavior. In [16] it was originally speculated that the width of an optimum is critically related to its generalization properties. Later, [21] extended the conjectures by conducting a set of experiments showing that SGD usually converges to sharper local optima for larger batch sizes. Following this principle, [7] proposed an SGD-based method that explicitly forces optimization towards wide valleys. [18] introduced a novel variant of SGD which averages weights during training. In this way, solutions in flatter regions of the loss landscape could be found which led to better generalization.
This, in turn, has led to measurable improvements in applications such as classification.
However, [8] argues that the commonly used definitions of flatness are problematic. By exploiting symmetries of the model, they can change the amount of flatness of a minimum without changing its generalization behavior.

§ PRELIMINARIES

§.§ Formulation of GANs

The goal of a generative model is to approximate a real data distribution $p_{\mathrm{r}}$ with a surrogate data distribution $p_{\mathrm{f}}$.
One way to achieve this is to minimize the “distance” between those two distributions.
The generative model of a GAN, as originally introduced by
Goodfellow et al., does this by minimizing the Jensen-Shannon Divergence between $p_{\mathrm{r}}$ and $p_{\mathrm{f}}$, using the feedback of the $D$.
From a game theoretical point of view, this optimization problem may be seen as a zero-sum game between two players, represented by the discriminator model and  the generator model, respectively.
During the training, the $D$ tries to maximize the probability of correctly classifying a given input as real or fake by updating
its loss function

\begin{align}
\begin{split}
\mathcal{L}_D = \mathbb{E}_{\mathrm{\mathbf{x}} \sim p_{\mathrm{r}}} \left[ \log \left(D(\mathrm{\mathbf{x}})\right) \right] \,+ \, \mathbb{E}_{\mathrm{\mathbf{z}} \sim p_{\mathrm{z}}}[\log(1-D(G(\mathrm{\mathbf{z}}))],
\end{split}
\end{align}

through stochastic gradient ascent.
Here, $\textbf{x}$ is a data sample and $\textbf{z}$ is drawn randomly.

The $G$, on the other hand, tries to minimize the probability of $D$ to classify its generated data correctly. This is done by updating its loss function

\begin{align}
\begin{split}
\mathcal{L}_G = \mathbb{E}_{\mathrm{\mathbf{z}} \sim p_{\mathrm{z}}}[\log(1-D(G(\mathrm{\mathbf{z}}))]
\end{split}
\label{for:gen}
\end{align}

via stochastic gradient descent.
As a result, the joint optimization can be viewed as a minimax game between $G$, which learns how to generate new samples distributed according to $p_{\mathrm{r}}$, and $D$, which learns to discriminate between real and generated data. The equilibrium of this game is reached when the $G$ is generating samples that look as if they were drawn from the training data, while the $D$ is left indecisive whether its input is generated or real.

§.§ Training of GANs

As we explained above, the training of a GAN requires the joint optimization of several objectives making their convergence intrinsically different from the case of a single objective function. The optimal joint solution to a minimax game is called Nash-equilibrium.
In practice, since the objectives are non-convex, using local gradient information, we can only expect to find local optima, that is local Nash-equilibria (LNE) [2].
An LNE is a point for which there exists a local neighborhood in parameter space, where neither the $G$ nor the $D$ can unilaterally decrease/increase their respective losses,
i.e. their gradients vanish while their second derivative matrix is positive/negative semi-definite:

\begin{equation}
\begin{gathered}
||\nabla_{\theta} \mathcal{L}_G|| = ||\nabla_{\varphi} \mathcal{L}_D|| = 0,\\
\nabla_\theta^2 \mathcal{L}_G \succeq 0 \; \mathrm{and} \; \nabla_\varphi^2 \mathcal{L}_D \preceq 0
\end{gathered}
\label{eqn:LNE}
\end{equation}

Here, $\theta$ and $\varphi$ are the weights $D$ and $G$, respectively.

(Left) $G$ losses, either minimization of $\log(1-D(G(\mathrm{\mathbf{z}}))$ or maximization of
$\log(D(G(\mathrm{\mathbf{z}}))$. (Right) $D$ loss, maximization of $\log(D(\mathrm{x})) + \log(1-D(G(\mathrm{\mathbf{z}}))$.

§.§ Evaluation of GANs

GANs have experienced a dramatic improvement in terms of image quality in recent years. Nowadays, it is possible to generate artificial high resolution faces indistinguishable from real images to humans [20]. However, their evaluation and comparison remain a daunting task, and so far, there is no consensus as to which metric can best capture strengths and limitations of models. Nevertheless, the Inception Score (IS), proposed by [28], is the most widely adopted metric. It evaluates images generated from the model by determining a numerical value that reasonably correlates with the quality and diversity of output images.

§ METHOD

§.§ Non-Saturating GAN

In minimax GANs the $G$ attempts to generate samples that minimize the probability of being detected as fake by the $D$ (c.f. Formula <ref>). However, in practice, it turns out to be advantages to use an alternative cost function which instead ensures that the generated samples have a high probability of being considered real. This modified version of a GAN is called a non-saturating GAN (NSGAN). When training a NSGAN the $G$ maximizes an alternative objective

\begin{align}
\begin{split}
\mathbb{E}_{\mathrm{\mathbf{z}} \sim p_{\mathrm{z}}}[\log(D(G(\mathrm{\mathbf{z}}))].
\end{split}
\end{align}

The intuition why NSGANs perform better than GANs is as follows.
In case the model distribution is highly different from the data distribution, the NSGAN can bring the two distributions closer together since the loss function generates a strong gradient. In fact, the NSGAN will have a vanishing gradient only when the $D$ starts being indecisive whether its input is from the data distribution or the $G$. This is acceptable, however, since the samples will already have reached the distribution of the real data by that time.
Figure <ref> shows the loss function of the original and non-saturating  $D$ and $G$, respectively.

§.§ Stochastic Lanczos quadrature algorithm

During optimization the network tries to converge into a local minimum of the loss landscape. Here we differentiate between flat and sharp minima. Whether a minimum is considered sharp or flat is determined by the loss landscape around the converged point. If the region has approximately the same error, the minimum is considered flat, otherwise we refer to the minimum as sharp. Another method to determine the sharpness of a minimum is by looking at the eigenvalues of the Hessian. These describe the local curvature in every direction of the parameter space. This allows us to see whether our network converges into sharp or flat minima or whether it converges into a minimum at all. Here, big eigenvalues correspond to a sharp minimum in the corresponding eigendirection.

We observe that high eigenvalues in the $G$ and $D$ lead to a worse IS score. Therefore we conclude that mode collapse is linked to the network converging into sharp minima. In order to confirm this, we look at the full eigenvalue density spectrum during training.

Calculating the eigenvalues of the Hessian has a complexity of $O(N^3)$, and storing the Hessian itself in order to compute the eigenvalues scales with $O(N^2)$ where $N$ is the number of parameters in the network. For neural networks that typically have millions of parameters, calculating the eigenvalues of their Hessian is infeasible. We can skip the problem of storing the Hessian by only calculating the Hessian-vector product for different vectors. In combination with the Lanczos algorithm, this allows us to compute the eigenvalues of the Hessian without having to calculate and store the Hessian itself.

The stochastic Lanczos quadrature algorithm [23] is a method for the approximation of the eigenvalue density of very large matrices. The eigenvalue density spectrum is given by

\begin{equation}
\phi(t) = \frac{1}{N}\sum_{i=1}^{N}\delta(t-\lambda_i)
\end{equation}

where $N$ is the number of parameters in the network, $\lambda_i$ is the i-th eigenvalue of the Hessian and $\delta$ is the Dirac delta function.
In order to deal with the Dirac delta function the eigenvalue density spectrum is approximated by a sum of Gaussian functions

\begin{equation}
\phi_{\sigma}(t) = \frac{1}{N}\sum_{i=1}^{N}f(\lambda_i,t,\sigma^2)
\end{equation}

\begin{equation}
f(\lambda_i,t,\sigma^2) = \frac{1}{\sigma \sqrt{2\pi}}\exp(-\frac{(t-\lambda_i)^2}{2\sigma^2})
\end{equation}

We use the Lanczos algorithm with full reorthogonalization in order to compute
eigenvalues and eigenvectors of the Hessian and to ensure orthogonality between
the different eigenvectors.  Since the Hessian is symmetric we can diagonalize
it and all eigenvalues are real.
The Lanczos algorithm is used together with the Hessian vector product for a certain number of iterations. Afterwards it returns a tridiagonal matrix $T$. This matrix is diagonalized as

\begin{equation}
T = ULU^T
\end{equation}

where $L$ is a diagonal matrix.

By setting $\omega_i = U_{1,i}^2$ and $l_i = L_{ii}$ for $i=1,2,...,m$, the resulting eigenvalues and eigenvectors are used to estimate the true eigenvalue density spectrum

\begin{equation}
\hat{\phi}^{(v_i)}(t)=\sum_{i=1}^{m}\omega_if(l_i,t,\sigma^2)
\end{equation}

\begin{equation}
\hat{\phi}_{\sigma}(t) = \frac{1}{k}\sum_{i=1}^{k}\hat{\phi}^{(v_i)}(t)
\end{equation}

For our experiments we use the toolbox from [6] which implements the Stochastic Lanczos quadrature algorithm. This allows to inspect and visualize the spectral information from our models.

§.§ Nudged-Adam Optimizer

To prevent our neural network from reaching sharp minima during optimization, we remove the gradient information in the direction of high eigenvalues. This forces our network to ignore the sharpest minima entirely and instead converge into wider ones.
Inspired by [19], we construct an optimizer based on Adam [22] which ignores the gradient in the direction of the top-k eigenvectors. In order to achieve this, we use the existing Adam optimizer and remove the directions of steepest descent from its gradient. This means that given the top-k eigenvectors $v_i$ and the gradient $g$ we remove the eigenvector directions by

\begin{equation}
g^* = g-\sum_{i=1}^{k}<g,v_i>v_i
\end{equation}

The resulting gradient $g^*$ is now used by the regular Adam optimizer. Using this technique one can modify a lot of different optimizers into their nudged counterpart by using $g^*$ instead of the true gradient.

The eigenvectors are computed by using the Lanczos method together with the R-operator. This allows fast computation of eigenvalues and eigenvectors without having to store the full Hessian.

§ EXPERIMENTS

In this section, we present a set of experiments to study the loss properties and the instability issues that might occur when training a GAN. We first use the visualization toolbox to inspect the spectrum of GANs during training, and to corroborate the problematic search of an LNE. Then, we examine the k-top eigenvalues from the $G$ and $D$, and their evolution throughout the training. Finally, we introduce a novel optimizer, called nudged-Adam, to prevent mode collapse and we test its performances on several datasets to guarantee reliability across different scenarios.

§.§ Visualizing Loss Landscape

We start our experimental section with a loss landscape visualization that will serve us as a reference point. We believe that building a solid background will help to provide a better understanding of the non-convergent nature of GANs, in particular concerning the $G$. In order to do this, we track the spectral density throughout the entire optimization process. Our main goal here is to gather evidences from the curvature that visualizes the general problem of finding LNEs. In order to carry out these experiments, we independently analyse the loss landscapes of the $G$ and of the $D$ using their highest eigenvalue, respectively.

We employ the toolbox from [6] to visualize the loss landscape and the trajectory of our GAN during training. In this way, we can gain some insights into the optimization process that happens underneath. Note, that to obtain the trajectory, we  project all the points of training into the 2D plane of the last epoch. Figure <ref> shows the landscape after training for 180 epochs on the NSGAN setup. By inspecting the landscapes, we observe (1) the $D$ clearly finds a local minimum and descends towards it, and (2) the $G$ ends up in an unstable saddle point, as suggested by the irregular landscapes surrounding it. This findings agree with the aforementioned second-order literature [5].

Logarithmic loss landscapes with trajectory of the same training run, visualized along eigenvectors corresponding to the two highest eigenvalues of NSGAN on MNIST. (Left) $G$ loss landscape. (Right) $D$ loss landscape.

§.§ Eigenvalue Analysis

After having gained intuition of the training conditions of GANs and their problem to find and remain at an LNE, we now focus our attention on the issue of mode collapse. In particular, we provide empirical evidences of a plausible relationship between mode collapse and the behaviour of the eigenvalues. To this end, we evaluate the spectrum of our model throughout the optimization process. More specifically, we track the largest eigenvalues from the $G$ and from the $D$ for each epoch, together with the IS.

We start training and evaluating the original non-saturating GAN architecture on the MNIST, Kuzushiji, Fashion and EMNIST datasets (see Figure <ref>). This results in a number of patterns that are present in all experiments.
(1) The evolution of the eigenvalues of the $G$ and $D$ behave visually very similar. In particular, when $D$ exhibits an increasing tendency in its eigenvalues, the $G$ does so as well. (2) Apart from the $G$ shape of the dynamics, it is important to evaluate the local behaviour, i.e. the correlation between the $G$ and $D$. Thereby, we observe a strong correlation in all our setups ranging from 0.72 to 0.90. (3) Furthermore, there seems to exist a connection between the IS and the behaviour of the eigenvalues. When the eigenvalues have a decreasing tendency, the IS score tends to increase, while when the eigenvalues increase, the IS scores deteriorates.  Moreover, we see how all our models start to suffer from mode collapse after 25 epochs (approximately when the eigenvalues tendency changes and starts to increase).

The empirical observations found in this analysis lead to the conclusion that eigenvalues can give an indication of the state of convergence of a GAN, as pointed out in [5]. Furthermore, we found that the eigenvalue evolution is correlated with the likely occurrence of a mode collapse event.

x label style=at=(axis description cs:0.5,0),
legend pos=north west,
every axis plot/.append style=semithick]

[blue, mark=] table img/gen_kuzushiji.dat;
[orange, mark=] table img/disc_kuzushiji.dat;
[black!30!green, mark=] table img/is_kuzushiji.dat;

Kuzushiji dataset. Correlation 0.80.

x label style=at=(axis description cs:0.5,0),
legend pos=north west,
every axis plot/.append style=semithick]

[blue, mark=] table img/gen_fashion.dat;
[orange, mark=] table img/disc_fashion.dat;
[black!30!green, mark=] table img/is_fashion.dat;

Fashion dataset. Correlation 0.90.

x label style=at=(axis description cs:0.5,0),
legend pos=north west,
every axis plot/.append style=semithick]

[blue, mark=] table img/gen_emnist.dat;
[orange, mark=] table img/disc_emnist.dat;
[black!30!green, mark=] table img/is_emnist.dat;

EMNIST dataset. Correlation 0.72.

Evolution of the top k-eigenvalues of the Hessian from generator (gen) and discriminator (disc), and the correspondence IS over the whole training phase. The correlation score is measured between the $G$ and the $D$.

(First row) Evolution of the top k-eigenvalues of the Hessian, the IS and random generated samples at different epochs of NSGAN on MNIST. (Second row) Comparison of IS evolution of NSGAN and NuGAN, and random generated samples at different epochs of NuGAN.

x label style=at=(axis description cs:0.5,0),
title=Epoch 2,
xlabel=Eigenvalue, ylabel=Hessian Eigenvalue Density,
legend pos=north east,
every axis plot/.append style=thick,

[blue, mark=] table img/gen_spec_2.dat;
[orange, mark=] table img/disc_spec_2.dat;

x label style=at=(axis description cs:0.5,0),
title=Epoch 25,
xlabel=Eigenvalue, ylabel=Hessian Eigenvalue Density,
legend pos=north east,
every axis plot/.append style=thick,

[blue, mark=] table img/gen_spec_25.dat;
[orange, mark=] table img/disc_spec_25.dat;

x label style=at=(axis description cs:0.5,0),
title=Epoch 160,
xlabel=Eigenvalue, ylabel=Hessian Eigenvalue Density,
legend pos=north east,
every axis plot/.append style=thick,

[blue, mark=] table img/gen_spec_160.dat;
[orange, mark=] table img/disc_spec_160.dat;

x label style=at=(axis description cs:0.5,0),
xlabel=Eigenvalue, ylabel=Hessian Eigenvalue Density,
legend pos=north east,
every axis plot/.append style=thick,
[blue, mark=] table img/gen_spec_adam_2.dat;
[orange, mark=] table img/disc_spec_adam_2.dat;

x label style=at=(axis description cs:0.5,0),
xlabel=Eigenvalue, ylabel=Hessian Eigenvalue Density,
legend pos=north east,
every axis plot/.append style=thick,
[blue, mark=] table img/gen_spec_adam_25.dat;
[orange, mark=] table img/disc_spec_adam_25.dat;

x label style=at=(axis description cs:0.5,0),
xlabel=Eigenvalue, ylabel=Hessian Eigenvalue Density,
legend pos=north east,
every axis plot/.append style=thick,

[blue, mark=] table img/gen_spec_adam_160.dat;
[orange, mark=] table img/disc_spec_adam_160.dat;

Plots of the whole spectrum of the Hessian at different stage of the training on MNIST. (First row) Results on NSGAN: we can identify an abnormal behaviour (mode collapse) in the generator at epoch 160. (Second row) Results on NuGAN: the spectrum remains stable during the whole training. We can observe how the $D$ for both cases finds local minima, while the $G$ remains all the time in a saddle point.

§.§ Combating Mode Collapse

In the last section we have seen that the growth of Hessian eigenvalues during the training of a GAN correlates to the occurrence of mode collapse. In order to remove this undesirable effect, we train our NSGAN with a nudged-Adam optimizer (referred to as NuGAN), which is inspired by [19].
Figure <ref> shows the results together with some visual samples generated at different training epochs. We observe that NuGAN achieves a much more stable IS, and this is also displayed on the generated samples. While NSGAN suffers from mode collapse, NuGAN does not (see samples on epoch 160). This shows a clear relationship between the behaviour of the IS score and the occurrence of mode collapse.

Figure <ref> shows the full spectrum of the Hessian at different stages of the training. A remarkable observation here is the present of negative eigenvalues for the $G$ for both optimizers. This indicates that the critical point reached during training is not an LNE (c.f. Formula <ref>). Rather, the $G$ reaches only a saddle point in all cases. On the other hand, the $D$ seems to converge to a sharp local minimum when using plain GDA. In fact, it seems that the longer training lasts the sharper the minimum gets. The $D$ of our NuGAN however, reaches a much flatter minimum, which can be seen by the presents of much smaller eigenvalues towards the end of training.

A second interesting observation is the connection between the spectrum of the $G$ and the mode collapse. In particular, we observe the occurrence of mode collapse when the spectrum spreads significantly (see first row from Figure <ref>). On the other hand, the spectral evolution of our NuGAN (see second row from Figure <ref>) does not display any anomaly for the $G$, and indeed, no mode collapse event occurrences.
In Table <ref> we show more quantitative results supporting the benefit of our nudged-Adam optimizer approach. There we show the IS for both optimizers evaluated of 4 different datasets. Notice that in all cases, our method achieves a higher mean and maximum score than the NSGAN baseline. These quantitative results together with the visual inspection of the image quality suggest that our NuGAN algorithm has a direct influence on the behavior of the eigenvalues and the loss landscape of our adversarial model, resulting in the avoidance of mode collapse.

Mean and max IS from the different datasets and methods (with and without mode collapse). Higher values are better.

Methods      2cNSGAN     2cNuGAN

mean     max     mean     max

MNIST     4.30     7.03     7.14     8.46

Kuzushiji     5.24     6.50     6.12     7.20

Fashion     5.74     6.82     6.35     7.20

EMNIST     3.77     7.02     8.53     7.67

Overall, we can summarize that the algorithm does not converge to an LNE, while still achieving good results w.r.t. the evaluation metric (IS). This raises the question whether convergences to an LNE is actually needed in order to achieve good generator performance of a GAN.

§ CONCLUSIONS

In this work, we investigate instabilities that occur during the training of GANs, focusing particularly on the issue of mode collapse. To do this, we analyse the loss surfaces of the $G$ and $D$ neural networks, using second-order gradient information, with special attention on the Hessian eigenvalues. Hereby, we empirically show that there exists a correlation between the stability of training and the eigenvalues of the generator network. In particular, we observe that large eigenvalues, which may be an indication of the convergence towards a sharp minimum, correlate well with the occurrence of mode collapse. Motivated by this observation, we introduce a novel optimization algorithm that uses second-order information to steer away from sharp optima, thereby preventing the occurrence of mode collapse. Our findings suggest that the investigation of generalization properties of GANs, e.g. by analysing the flatness of the optima found during training, is a promising approach to progress towards more stable GAN training as well.

[1]
Abdal, R., Qin, Y., and Wonka, P. (2019).
Image2stylegan: How to embed images into the stylegan latent space?
In Proceedings of the IEEE international conference on computer
vision, pages 4432–4441.

[2]
Adolphs, L., Daneshmand, H., Lucchi, A., and Hofmann, T. (2018).
Local saddle point optimization: A curvature exploitation approach.
arXiv preprint arXiv:1805.05751.

[3]
Alain, G., Roux, N. L., and Manzagol, P.-A. (2019).
Negative eigenvalues of the hessian in deep neural networks.
arXiv preprint arXiv:1902.02366.

[4]
Arjovsky, M., Chintala, S., and Bottou, L. (2017).
Wasserstein gan.
arXiv preprint arXiv:1701.07875.

[5]
Berard, H., Gidel, G., Almahairi, A., Vincent, P., and Lacoste-Julien, S.
A closer look at the optimization landscapes of generative
adversarial networks.
arXiv preprint arXiv:1906.04848.

[6]
Chatzimichailidis, A., Keuper, J., Pfreundt, F.-J., and Gauger, N. R. (2019).
Gradvis: Visualization and second order analysis of optimization
surfaces during the training of deep neural networks.
In 2019 IEEE/ACM Workshop on Machine Learning in High
Performance Computing Environments (MLHPC), pages 66–74. IEEE.

[7]
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C.,
Chayes, J., Sagun, L., and Zecchina, R. (2019).
Entropy-sgd: Biasing gradient descent into wide valleys.
Journal of Statistical Mechanics: Theory and Experiment,

[8]
Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. (2017).
Sharp minima can generalize for deep nets.
CoRR, abs/1703.04933.

[9]
Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. A. (2018).
Essentially no barriers in neural network energy landscape.
arXiv preprint arXiv:1803.00885.

[10]
Durall, R., Keuper, M., and Keuper, J. (2020).
Watch your up-convolution: Cnn based generative deep neural networks
are failing to reproduce spectral distributions.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 7890–7899.

[11]
Durall, R., Pfreundt, F.-J., Köthe, U., and Keuper, J. (2019).
Object segmentation using pixel-wise adversarial loss.
In German Conference on Pattern Recognition, pages 303–316.

[12]
Fiez, T., Chasnov, B., and Ratliff, L. J. (2019).
Convergence of learning dynamics in stackelberg games.
arXiv preprint arXiv:1906.01217.

[13]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair,
S., Courville, A., and Bengio, Y. (2014a).
Generative adversarial nets.
In Advances in neural information processing systems, pages

[14]
Goodfellow, I. J., Vinyals, O., and Saxe, A. M. (2014b).
Qualitatively characterizing neural network optimization problems.
arXiv preprint arXiv:1412.6544.

[15]
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C.
Improved training of wasserstein gans.
In Advances in neural information processing systems, pages

[16]
Hochreiter, S. and Schmidhuber, J. (1997).
Flat minima.
Neural Computation, 9(1):1–42.

[17]
Iizuka, S., Simo-Serra, E., and Ishikawa, H. (2017).
Globally and locally consistent image completion.
ACM Transactions on Graphics (ToG), 36(4):107.

[18]
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G.
Averaging weights leads to wider optima and better generalization.
arXiv preprint arXiv:1803.05407.

[19]
Jastrzebski, S., Kenton, Z., Ballas, N., Fischer, A., Bengio, Y., and Storkey,
A. (2018).
On the relation between the sharpest directions of dnn loss and the
sgd step length.
arXiv preprint arXiv:1807.05031.

[20]
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T.
Analyzing and improving the image quality of stylegan.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8110–8119.

[21]
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P.
On large-batch training for deep learning: Generalization gap and
sharp minima.
arXiv preprint arXiv:1609.04836.

[22]
Kingma, D. P. and Ba, J. (2014).
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980.

[23]
Lanczos, C. (1950).
An iteration method for the solution of the eigenvalue problem
of linear differential and integral operators.
United States Governm. Press Office Los Angeles, CA.

[24]
Mescheder, L., Nowozin, S., and Geiger, A. (2017).
The numerics of gans.
In Advances in Neural Information Processing Systems, pages

[25]
Nagarajan, V. and Kolter, J. Z. (2017).
Gradient descent gan optimization is locally stable.
In Advances in neural information processing systems, pages

[26]
Radford, A., Metz, L., and Chintala, S. (2015).
Unsupervised representation learning with deep convolutional
generative adversarial networks.
arXiv preprint arXiv:1511.06434.

[27]
Sagun, L., Bottou, L., and LeCun, Y. (2016).
Eigenvalues of the hessian in deep learning: Singularity and beyond.
arXiv preprint arXiv:1611.07476.

[28]
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen,
X. (2016).
Improved techniques for training gans.
In Advances in neural information processing systems, pages

[29]
Xue, Y., Xu, T., Zhang, H., Long, L. R., and Huang, X. (2018).
Segan: Adversarial network with multi-scale l 1 loss for medical
image segmentation.
Neuroinformatics, 16(3-4):383–392.

[30]
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. S. (2019).
Free-form image inpainting with gated convolution.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 4471–4480.


# When your Cousin has the Right Connections: Unsupervised Bilingual Lexicon
Induction for Related Data-Imbalanced Languages

###### Abstract

Most existing approaches for unsupervised bilingual lexicon induction (BLI)
depend on good quality static or contextual embeddings requiring large
monolingual corpora for both languages. However, unsupervised BLI is most
likely to be useful for low-resource languages (LRLs), where large datasets
are not available. Often we are interested in building bilingual resources for
LRLs against related high-resource languages (HRLs), resulting in severely
imbalanced data settings for BLI. We first show that state-of-the-art BLI
methods in the literature exhibit near-zero performance for severely data-
imbalanced language pairs, indicating that these settings require more robust
techniques. We then present a new method for unsupervised BLI between a
related LRL and HRL that only requires inference on a masked language model of
the HRL, and demonstrate its effectiveness on truly low-resource languages
Bhojpuri and Magahi (with <5M monolingual tokens each), against Hindi. We
further present experiments on (mid-resource) Marathi and Nepali to compare
approach performances by resource range, and release our resulting lexicons
for five low-resource Indic languages: Bhojpuri, Magahi, Awadhi, Braj, and
Maithili, against Hindi.

Keywords: bilingual lexicon induction, low-resource, Indic languages

When your Cousin has the Right Connections: Unsupervised Bilingual Lexicon
Induction for Related Data-Imbalanced Languages

Niyati Bafna,1 Cristina España-Bonet,3 Josef van Genabith,2,3
---
Benoît Sagot,1 Rachel Bawden1
1Inria, Paris, France
2Saarland University, Saarland Informatics Campus, Germany
3DFKI GmbH, Saarland Informatics Campus, Germany
nbafna1<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Bilingual lexicons are a basic resource with varied uses, both in themselves,
for dictionary building and language learning, as well as seeds for solving
other problems in natural language processing (NLP), such as parsing (Zhao et
al., 2009; Durrett et al., 2012) and word-to-word and unsupervised machine
translation (Irvine and Callison-Burch, 2013; Thompson et al., 2019).

While there is growing interest in unsupervised or minimally supervised
bilingual lexicon induction (BLI), existing methods often depend on aligning
monolingual word embedding spaces, assumed to be of good quality for both
languages, and/or bilingual supervision (Artetxe et al., 2016, 2017; Conneau
et al., 2018; Artetxe et al., 2018a, b, 2019). However, extremely low-resource
languages (LRLs) and dialects often lack good quality embeddings due to
limited monolingual data, leading to very low or near-zero performance of
alignment-based methods for these languages (Wada et al., 2019; Eder et al.,
2021).

This is the case for the under-researched Indic language continuum, which is
the focus of this article (see Section 2 for a description of the linguistic
setup in India that motivates our work). We work with five extremely low-
resourced Indic languages, Bhojpuri (bho), Magahi mag), Awadhi (awa), Maithili
(mai), and Braj (bra), which are closely related to higher-resourced Hindi,
and which have extremely limited resources, in terms of training data (<5M
tokens of monolingual data) and embeddings, and even evaluation data. We
demonstrate that state-of-the-art, alignment-based methods perform poorly in
these settings, and introduce a new method for unsupervised BLI that performs
much better. We aim to design methods that work well in characteristic data-
scarce conditions, as well as generate resources for further work in these
languages.

Our main contribution is a novel unsupervised BLI method to address the
typical scenario of the LRLs of the Indic continuum, i.e. for extremely LRLs
that share significant overlap with a closely related HRL. We suppose that a
masked language model (MLM) such as monolingual BERT (Devlin et al., 2019) is
available for the HRL and that we have some monolingual LRL sentences that
contain unknown words. The method consists of building a lexicon iteratively
by using the HRL MLM over LRL sentences to extract translation equivalents,
and replacing learnt words in LRL sentences with HRL equivalents to make them
more tractable for the HRL MLM for future unknown words (see Section 4).

Given the lack of existing gold lexicons for our target languages (a frequent
scenario for extremely LRLs), we create silver lexicons for Bhojpuri and
Magahi created from parallel data, unfortunately unavailable for Awadhi,
Maithili, and Braj. We also perform control experiments on Marathi and Nepali,
two medium-resource languages more distantly related to Hindi with available
gold lexicons, and discuss the performance of canonical methods and our
proposed method on these languages, shedding light on what strategies are
appropriate for differently-resourced language pairs. Our experiments indicate
that current state-of-the-art methods are not suitable for low-resourced
dialects, and methods that account for the data imbalance in the language
pair, such as ours, may be more successful. We release our code, our generated
lexicons for all languages (to our knowledge the first to be publicly released
for all languages except Bhojpuri),111Although dedicated teams are working
towards building resources for these languages (Mundotiya et al., 2021), these
resources (including evaluation resources) have not yet been made public as
far as we know. and our created silver evaluation lexicons for Bhojpuri and
Magahi.222Code and resources available here:
https://github.com/niyatibafna/BLI-for-Indic-languages. See details of our
released lexicons in Section 7.

Our motivation and method, while relevant to the 40+ resource-scarce languages
of the Indic language family and other Indian languages, are also relevant to
other linguistic systems with similar circumstances, i.e. with a single high-
or-medium resource language (usually a standard dialect), and several closely
related dialects with lexical, morphological, and syntactic variation, written
in the same script with or without orthographic standardization. This setup
describes, for example, the Arabic continuum, the Turkic language continuum,
and the German dialect system.

## 2\. Linguistic Setup in India

Meaning | boy (nom) | sister (nom) | your (hon., fem. sing. obj) | told (completive) | (you) are going
---|---|---|---|---|---
Hindi |  l dk: |  bhn |  apki |  btaja/ kh lija |  da rhe ho
Awadi |  l dk: |  bhin |  apn |  btavt |  dat hai
Bhojpuri |  lika |  bhin |  apn |  khl |  dat ba
Magahi |  lika |  bhin |  pn |  khlie |  da hi
Maithili |  l dk: |  bhin |  hank |  khlhun |  da rhl h i
Table 1: Examples of cognates. Since the Devanagari script is phonetically
transparent, phonetic similarity is visible both in IPA and in Devanagari (not
shown). Table 2: I/O examples, shown for Bhojpuri and the outputs of Basic.
Input: Target language text with an unknown masked word. Mask: the original
masked target language word. Correct: Acceptable Hindi equivalents given in
the silver lexicon. Preds: The top predictions made by the mask-filling model.
We choose $5$ representative examples here(in practice we use the top $30$).
The underlined prediction is the one that has the lowest normalized edit
distance with the original target language word, and would therefore be chosen
in the Basic approach as the top candidate. Blue highlighted predictions are
correct equivalents, even if not included in silver lexicon. Orange
highlighted examples fit the mask but are not equivalents of the masked target
word. * indicates a non-word in Hindi. New input: Input with known source word
replaced with target equivalent (indicated in bold).

India has around 15-22 languages that are medium-to-high-resource, such as
Hindi, Marathi, and Tamil, but dozens of other languages and dialects that are
extremely low-resourced, with very little monolingual data (<5M tokens), and
no other resources, such as Marwadi, Tulu, Dogri, and Santhali. These
languages are often closely related to at least one high-resource language
(HRL), meaning that they share morphosyntactic properties as well as a high
number of cognates (Jha, 2019; Mundotiya et al., 2021) (see Table 1 for
examples). They often have no official status in the regions where they are
spoken, and therefore do not have concerted funding efforts for data
collection or research. Even when such efforts do exist,333See
https://data.ldcil.org/text. the collected corpora are rarely of the magnitude
at which static or contextual embeddings can be well-estimated. While the
actual number of distinct dialects and languages spoken in India is contested,
people self-reported about 576 such “mother tongue” dialects in the latest
census,444See https://www.outlookindia.com/national/explained-what-is-mother-
tongue-survey-and-its-importance-in-preserving-india-s-linguistic-data-
news-235854. which were then grouped into around 121 languages. Only 22 of
these languages have official status (i.e. they are either the official
language of some state/union territory, or have national cultural
significance), and are therefore accorded funds for the development of
resources.

Therefore, although some studies in the literature question the real use case
for entirely unsupervised BLI (Vulić et al., 2019), since it is “easy” to
collect a small bilingual lexicon, we argue that situations such as these,
where there is a large number of languages to build support for, and where
efforts in data collection and annotation for individual languages are
restricted by the availability of funds, do constitute genuine application
scenarios for unsupervised BLI.

Furthermore, we focus on a scenario where the two languages in question are
closely related. This is because for most of the low-resource languages in the
Indian context cited above, we can usually find a linguistic neighbour that is
relatively well off, usually one of India’s $22$ scheduled languages.555Of
course, there are exceptions to this observation; naturally, this is not true
for language isolates such as Burushaski, spoken in Pakistan, or for the small
minority of Austro-Asiatic langauges such as Mundari, spoken in India, which
do not have a single (Indian) high-resource sister language. In general, when
building resources for a given low-resource dialect or language, it is likely
that the standard variant of that dialect, or the HRL closest to it, will have
large enough corpora available to build a good quality MLM. We target our
efforts to these situations.

## 3\. Related Work

Recent years have seen interest in unsupervised BLI (Haghighi et al., 2008;
Artetxe et al., 2016, 2017; Conneau et al., 2018; Artetxe et al., 2018a, b,
2019), allowing the possibility of BLI for LRLs. Most unsupervised approaches,
notably MUSE (Conneau et al., 2018) and VecMap (Artetxe et al., 2016, 2017,
2018a, 2018b) are based on training static embeddings from large monolingual
corpora (Mikolov et al., 2013; Bojanowski et al., 2017), and aligning the
embeddings using linear or non-linear mappings, using an initial seed (Xing et
al., 2015; Artetxe et al., 2016).

Recent works have also looked at using contextual embeddings or BERT-based
models (Peters et al., 2018; Devlin et al., 2019; Ruder et al., 2019) for BLI.
Gonen et al. (2020) induce word-level translations by directly prompting mBERT
(Devlin et al., 2019). Yuan et al. (2020) present a human-in-the-loop system
for BLI in four low-resource languages, updating contextual embeddings with
the help of annotations provided by a native speaker. Zhang et al. (2021)
present CSCBLI, a method that uses a “spring network” to align non-isomorphic
contextual embeddings, and interpolates them with static embeddings to
estimate word similarities, showing superior results to other methods using
contextual embeddings, notably BLISS (Patra et al., 2019). These approaches
rely on parallel data or large monolingual corpora for good quality contextual
embeddings. However, for low-resource languages, contextual embeddings from
both monolingual and multilingual models are known to be unreliable (Wu and
Dredze, 2020).

Later works show the failings of the above approaches in low-resource settings
(Adams et al., 2017; Kuriyozov et al., 2020; Chimalamarri et al., 2020; Eder
et al., 2021) and propose alternative training strategies such as joint
training of static embeddings (Woller et al., 2021; Bafna et al., 2022), and
multilingual embeddings from LSTM-based models (Wada et al., 2019). However,
these works either address a higher resource range (>15M tokens), use
bilingual lexicons as seeds, or show low scores ($\approx$30 precision@5) for
unsupervised BLI. In general, there is a paucity of attention given to setups
where there is a severe resource imbalance between the two languages of the
BLI pair, despite this being a very typical real-world scenario.

## 4\. Method

Our method is intended for a closely related HRL (source) and LRL (target)
pair, written in the same script, and given that we can train or already have
a good quality monolingual MLM for the HRL. The main idea is that if we mask
an unknown word in the LRL sentence, feed the masked LRL sentence to the HRL
MLM, and ask the HRL MLM to propose candidates for the masked LRL word, the
HRL MLM should have access to enough contextual cues due to shared vocabulary
and syntax to propose meaningful HRL candidates for the masked word. This
potentially gives us translation equivalence between the original LRL word and
the best scoring proposed HRL candidate. We proceed in an iterative manner,
growing the lexicon from equivalents gained from each processed sentence, and
using learned equivalents in the lexicon to replace known LRL words with HRL
equivalents to process future sentences.

Starting with an empty HRL-LRL bilingual lexicon, we perform the following
steps to update our lexicon iteratively, explained in further detail below,
and shown in Algorithm 1: (i) we choose an input, consisting of an LRL
sentence, and a source LRL word occurring in it, (ii) we replace known words
in the input sentence by HRL equivalents using the current state of the
lexicon, in order to make the sentence more HRL-like, (iii) the resulting
sentence is passed to the HRL MLM to obtain HRL candidate suggestions for the
masked LRL word, (iv) we use a reranking heuristic to choose the best HRL
candidate, if any, and (v) we update the lexicon if we have found a new
equivalent pair.

### 4.1. Choosing (sentence, word) pairs to process

Intuitively, the chance of the HRL MLM giving accurate translation equivalents
for the (LRL) word is higher if the LRL sentence is more easily
“comprehensible” to the HRL MLM, or if the LRL sentence already has several
HRL words in it. Therefore, we aim to first process words in sentences that
have a higher concentration of known words, where known words are either
shared vocabulary or words that are already in the current state of our
lexicon. These words are replaced by their HRL equivalents before the sentence
is passed to the HRL MLM.666We mask whole words and accept single token
responses (as the default) from the MLM. In practice, this does not pose a big
problem, since the HRL MLM tokenizer has a large vocabulary size (52000): 86%
and 81% in the Hindi side of the Bhojpuri and Magahi silver lexicons
respectively are preserved as single tokens. We leave it to future work to
handle multi-word terms. We maintain a priority list of (sentence, word) pairs
based on the percentage of known words in the sentence and update the list
after every batch of sentences based on new learned
translations.777Specifically, the priority list is created from the (sentence,
unk_word) pairs by first sorting them by the number of times each instance has
previously been processed, and then by the percentage of other unknown words
in the sentence, both in ascending order.

### 4.2. Reranking

The HRL MLM may propose valid candidates for the masked token that are not
translation equivalents to the LRL source word; typically, there may be a wide
range of reasonable possibilities for any masked word. Therefore, we rerank
the returned HRL candidates based on orthographic closeness to the masked LRL
word. Our use of orthographic closeness as the basis of our rerankers is
motivated by the high percentage of orthographically similar cognates,
borrowings, and spelling variants in the vocabulary of these languages with
respect to each other (shown by Jha (2019) for Maithili and Hindi). Note that
minimum normalized edit distance as a stand-alone approach, i.e. positing the
orthographically closest HRL word as a translation equivalent for any LRL
word, performs badly for various reasons (Bafna et al., 2022). We compare two
rerankers, Basic and Rulebook.

#### 4.2.1. Basic

In the Basic approach, we simply use normalized orthographic similarity
(computed using Levenshtein distance) between the candidate and the original
masked word. This reranker considers all character substitutions equally
costly.

#### 4.2.2. Rulebook

We may see from discovered cognate pairs that certain character
transformations are very common (corresponding to regular sound change, or
systematic differences in orthographic conventions), and so should be less
costly than others. Similarly, different language pairs may have different
preferences for cheap or costly character substitutions.

In the Rulebook variant, we use Bafna et al.’s (2022) iterative expectation-
maximization (EM) method to learn a custom edit-distance matrix for the source
and target character sets. This custom edit-distance matrix is used as an
orthographic reranker for our approach (lines 6-9 in Algorithm 1).

The idea of this reranker is to iteratively optimize character substitution
probabilities from the source to target character set in “known”, or
hypothesized, cognate pairs, while simultaneously learning new cognate pairs
by reranking candidates suggested by the HRL MLM, using the current state of
the substitution probabilities.

##### Setup

Let $\chi_{s}$ and $\chi_{t}$ represent the sets of characters on the source
(LRL) and target (HRL) sides, respectively. We define a scoring function,
$S(c_{i},c_{j})$ that provides a score for replacing a character
$c_{i}\in\chi_{s}$ with $c_{j}\in\chi_{t}$. Insertions and deletions are
considered special cases of replacement, where a null character is introduced
or replaced. For a given source set character, $S$ is modelled as a
transformation probability distribution over $\chi_{t}$. Initially, the
probabilities in $S$ are assigned to favor self-transformations (typically set
to 0.5), and the remaining probability mass is evenly distributed among other
characters.

At any given iteration, we can calculate the score for a source-target
character substitution, viewed as a conditional probability:

$S(c_{i},c_{j})=\frac{C(c_{i},c_{j})}{T(c_{i})}$ (1)

Here, $C(a,b)$ is the number of times we have seen $a$ $\rightarrow$ $b$, and
$T(a)$ is the total number of times we have seen $a$ on the source side.

##### EM Steps

for Rulebook.

1) Expectation step. Given a list of top $k$ candidates for a given source
word $s$: for each candidate pair $(s,t)$, we find $Ops(s,t)$, which is the
minimal list of the operations we need to perform to get from $s$ to $t$. Each
member in $Ops$ is of the type $(c_{i},c_{j})$. Note that we also want to
estimate $S(a,a)$ $\forall$ $a$, and so we also use a “retain” operation, for
characters that remain the same. The score for the pair $(s,t)$ is computed
as:

$\zeta(s,t)=-\sum_{(a,b)\in Ops}log(S(a,b)),$ (2)

where the lower the $\zeta$ the more probable it is that a pair is equivalent.
For a given $s$, we can then always find the word that is the most probable
equivalent as $t_{best}=argmin_{t_{i}\neq s}(\zeta(s,t_{i}))$ (line 6 in
Algorithm 1). We then add $(s,t_{best})$ to our learned lexicon (line 8).

2) Maximization step. We update the model parameters based on the newly
identified equivalents in the previous step (line 9 in Algorithm 1). This is
done by increasing the counts of all observed edit distance operations:

$C(a,b):=C(a,b)+1~{}~{}~{}~{}\forall(a,b)\in Ops(s,t)$
$~{}~{}~{}~{}~{}T(a):=T(a)+1~{}~{}~{}~{}~{}~{}~{}\forall(a,b)\in Ops(s,t)$

We disallow updates for $s=t$ (i.e. identical words) in the training phase, to
mitigate exploding self-transform probabilities.

### 4.3. Multiple passes over the input

Once all $(sentence,word)$ pairs have been processed once (or $n$ times), we
reprocess them (for an $(n+1)^{th}$ pass) in the hope of gaining more accurate
translations, as previously unknown neighbour words may have been learned in
the meantime.

### 4.4. Hyperparameters

We use a minimum normalized orthographic similarity threshold of 0.5 (see line
7 of Algorithm 1). This threshold was heuristically chosen. We set the maximum
number of passes to 3, meaning that the algorithm terminates if all unknown
words have been processed 3 times. We found in our initial experiments that
the algorithm yields very few or no new words in further passes. This also
serves as a terminating condition (line 1 in Algorithm 1).

### 4.5. Examples

We give examples of inputs and outputs of our method in Table 2, illustrating
the outputs for Basic. As we see, the Hindi BERT is fairly good at giving
reasonable Hindi candidates for the masked Bhojpuri, although, naturally,
these candidates may not be equivalents of the masked word, as shown for the
top candidates in rows 2 and 3. Applying reranking based on orthographic
similarity solves this problem to a large extent, serving to identify
translation equivalents from among given candidates.

We also see an example (row 3) where replacing a Bhojpuri word with its Hindi
equivalent in the input sentence helps the Hindi MLM to produce more
reasonable Hindi candidates for the masked word.

1

2

3while  _not $terminating\\_condition$_ do

4 sent, word $\leftarrow$ chooseLRLExample();

5 sent $\leftarrow$ replaceKnownWords(sent, lexicon);

6 sent $\leftarrow$ maskWord(sent, word);

7 preds $\leftarrow$ HRLBertMaskFill(sent);

8 best $\leftarrow$ argmax(orthSim(word, preds));

9 if  _orthSim(best, word) > threshold_ then

10 lexicon $\leftarrow$ updateLexicon(word, best);

/* Next step only for rulebook */

11 updateOrthSimParams(word, best)

12

Algorithm 1 Basic and Rulebook Table 3: Types and examples of faults in the silver lexicon. Target | #Tokens | Lexicon | Silver lexicon
---|---|---|---
lang. |  | size | size
awa | 0.17M | 10462 | -
bho | 3.09M | 21983 | 2469
bra | 0.33M | 10760 | -
mag | 3.16M | 30784 | 3359
mai | 0.16M | 12069 | -
mar* | 551.00M | 36929 | -
nep* | 110.00M | 22037 | -
Table 4: Monolingual data sizes in tokens, and sizes of our released lexicons
(created using our method), and released silver lexicons (from parallel data)
for Bhojpuri and Magahi. *High-quality gold bilingual lexicons already exist
for these languages.

## 5\. Experimental Settings

##### Monolingual Data

We use monolingual data from the LoResMT shared task (Ojha et al., 2020) for
Bhojpuri and Magahi, and the VarDial 2018 shared task data (Zampieri et al.,
2018) for Bhojpuri, Awadhi and Braj. For Bhojpuri, we additionally use the
BHLTR project (Ojha, 2019). We use the BMM corpus (Mundotiya et al., 2021) and
the Wordschatz Leipzig corpus (Goldhahn et al., 2012) for Maithili. For
Marathi and Nepali, we use large-scale monolingual corpora made available by
IndicCorp (Kakwani et al., 2020) and Lamsal (2020) respectively. See Table 4
for monolingual data sizes.

##### Model

We use the MuRIL model and tokenizer (Khanuja et al., 2021) as our HRL MLM for
Bhojpuri, Magahi, Awadhi, Maithili and Braj; we use the Hindi BERT and
associated tokenizer given by Joshi (2023) for Marathi and Nepali.888We need a
HRL model that has not seen target data; while MuRIL is a good choice for low-
resource dialects because it is multilingual and may benefit from knowledge of
other related Indic languages, it cannot be used for Marathi and Nepali
because these languages are included in its pretraining data.

##### Baselines

We compare our approaches against semi-supervised VecMap approach with CSLS
(Artetxe et al., 2018b, a), using identical words as seeds, with
300-dimensional fastText embeddings (Bojanowski et al., 2017).999Using $100$
dimensions gives similar results. We also choose CSCBLI (Zhang et al., 2021)
as a representative of methods using contextual representations, hypothesizing
that the ensemble of static and contextual embeddings may perform better than
VecMap. Finally, we report results for a trivial baseline ID, the identity
function, representing vocabulary overlap.

##### Evaluation Data

Given the lack of gold lexicons between Hindi and our LRLs, we create silver
lexicons instead from parallel data. We use FastAlign with GDFA Dyer et al.
(2013) to extract word alignments from existing gold Bhojpuri–Hindi and
Magahi–Hindi parallel data ($\approx$500 sentences per language) (Ojha,
2019).101010We were unable to find a reasonable quantity of publicly available
parallel data for Awadhi, Braj, or Maithili, and so could not perform
evaluation for these languages. We use the two best candidates per source word
in the resulting silver lexicons as valid translations.111111This is an
empirical choice from eyeballing the resulting silver lexicons; at least one
and usually both of the first two translations are valid. This yields 2,469
and 3,359 entries for Bhojpuri and Magahi respectively. We report the manually
evaluated quality of the silver lexicons in the following paragraph. For
Marathi and Nepali, we use existing gold parallel lexicons against Hindi,
taken from IndoWordNet (Kakwani et al., 2020), manually aligned to the Hindi
WordNet. We obtain lexicons with 35,000 and 22,000 entries for Marathi and
Nepali respectively.

##### Manual Evaluation of Silver Lexicons

We perform a manual evaluation of our silver lexicons, in order to judge the
credibility of the reported results for our methods for Bhojpuri and Magahi.
We manually examine 150 entries in the automatically created Bhojpuri lexicon,
and find that 90% of entries are satisfactory, i.e. they list accurate Hindi
equivalents of Bhojpuri words. We observe a few general problems with the
lexicon, and list representative examples in Table 3:

  * •

Missing common synonyms, e.g. in row 1 of Table 3. This kind of error results
in underestimation of precision scores for all approaches.

  * •

Problems with correctly equating inflections, missing feminine inflections,
e.g. row 2. A natural problem arising from differences in morphological
systems of the source and target language is that inflected verbs can be
difficult to match cross-lingually. This results in missing equivalents of a
given inflected form. For example, while genderless verbs in Bhojpuri should
ideally be listed with the corresponding masculine and feminine verbs in
Hindi, we observe that they are often missing one gender inflection, usually
the feminine one. Similarly, not all possible target inflectional variants of
a source inflection are listed for each verb entry.

  * •

Multi-word equivalences lead to errors. For example, in row 3, the single-word
Bhojpuri source verb has a noun-light verb complex equivalent in Hindi
(consisting of two words, literally meaning “sharing do”), and the silver
lexicon lists the light verb (“do”) as the target translation. This is also
observed in the case of other verb equivalences, where one of the languages
using multiple tokens to express an inflection, leading to incorrect matches
in the silver lexicon.

  * •

Miscellaneous errors. The lexicon contains some entirely incorrect equivalents
(8.76%), due to word alignment errors, e.g. row 4.

Note that we only mark entries as wrong if the listed equivalents are
inaccurate, and so faults such as missing synonyms and inflections, which
affected 7.33% of the sample we examined, are not represented in the error
percentage reported.

## 6\. Results and Discussion

|  | bho | mag | mar | nep
---|---|---|---|---|---
| Method | P@2 | NIA | P@2 | NIA | P@2 | NIA | P@2 | NIA
Baselines | ID | 37.3 | 0.0 | 39.9 | 0.0 | 27.5 | 0.0 | 21.2 | 0
| VecMap+CSLS | 0.0 | 0.0 | 1.2 | 0.6 | 42.4 | 26.7 | 0.0 | 0.0
| CSCBLI | 0.0 | 0.0 | 2.0 | 0.5 | 0.0 | 0.0 | 0.0 | 0.0
Ours | Basic | 61.0 | 18.1 | 65.2 | 18.8 | 80.9 | 2.8 | 87.6 | 8.2
| Rulebook | 61.5 | 15.1 | 65.4 | 17.4 | 80.6 | 1.72 | 87.6 | 6.0
Table 5: Performance of the methods, given by Precision@2 (P@2) and accuracy
of non-identical predictions (NIA).

Table 6: Predictions made by different approaches. Meanings are provided for
the first occurrence of the word. * indicates a non-word and † a prediction in
the wrong inflectional/derivational form of the target.

We report precision@2 and accuracy on non-identical predictions (NIA) in Table
5.121212We also report P@{1,3,5} in Appexdix A; these results show similar
trends. NIA is calculated by taking all non-identical predictions in the top 2
predictions per word, and reporting the percentage of those predictions that
were marked correct by the evaluation lexicons. We report this metric because
precision@2 may be inflated by “easy” identical word predictions.

##### Baselines

Table 6 provides examples of the performance of these approaches. VecMap
performs well for Marathi: we provide examples where it predicts correct
equivalents for rare words (row 8), non-cognates (row 9), as well as frequent
words (row 7). However, for Nepali, Bhojpuri, and Magahi, both VecMap and
CSCBLI make seemingly random wrong predictions on almost all words (rows 1, 2,
and 6), with near-zero performance, probably due to the low quality of static
and contextual embeddings for the LRLs. CSCBLI also fails for Marathi,
indicating that the Marathi contextual embeddings may still be of poor quality
or that the approach may not generalize well to untested language pairs. While
the failure of these baselines for Nepali is surprising, it can perhaps be
explained by the fact that Nepali has about five times less data than Marathi,
and less lexical overlap with Hindi.

##### Our methods

Our Basic and Rulebook approaches outperform ID by more than 20 accuracy
points for all languages. Rulebook gains very little, if at all, over Basic,
but Rulebook has an edge when it comes to predicting cognates with common
sound correspondences (see row 5). We observe that these approaches are
reasonably successful for Bhojpuri and Magahi on cognate verbs and common
nouns, but fail on syntactic words and postpositions (row 3 for Basic), and
may be confused by unrelated words with chance orthographic similarity even
for common words (row 5 for Basic). Furthermore, these approaches often
predict incorrect inflections of the correct verbal/noun stem (we count these
predictions as wrong), as in rows 1 and 4. Although Basic and Rulebook perform
with high accuracy for Marathi and Nepali, their NIA is extremely low,
indicating that they serve mainly to identify or “sieve” out vocabulary
overlap. We see that the candidates proposed by the Hindi MLM are often in
fact Marathi/Nepali words, indicating that it has seen some Marathi/Nepali
data (due to corpus contamination and/or code-mixing) and is capable of
performing mask-filling for Marathi/Nepali.

##### Manual Evaluation of Generated Lexicons

We manually examine errors in the non-identical predictions of Basic, looking
at 60 randomly chosen non-identical Bhojpuri predictions.131313One of the
authors is a native speaker of Hindi but not Bhojpuri. We segregate
translation equivalents into error categories using (self-made) inflection
tables inferred from the silver lexicons, as well as cognate knowledge from
Hindi native speaker knowledge. We find that 31.7% of predictions are
correctly inflected equivalents, as opposed to 18.1% given by the NIA
quantitative evaluation. The underestimation is caused by missing synonyms in
the silver lexicon. Furthermore, 25% are incorrectly inflected cognates of the
source word, and the rest are unrelated words.

##### How useful is reranking by orthographic distance?

We also ran the Basic approach without reranking with orthographic distance,
i.e. we simply pick the top candidate suggested by the HRL mask-filling model
as an equivalent. This approach is clearly worse than the standard Basic
approach (with reranking), performing at only 3.03% NIA for Bhojpuri and 4.04%
NIA for Magahi (approximately -15 and -14 percentage points compared to Basic
for Bhojpuri and Magahi respectively, as shown in Table 5). However, this
approach can still identify and capture identical vocabulary.

##### Variants

We experimented with minor variants of the Rulebook update mechanisms to see
if they result in boosts to performance. We tried disallowing updates for the
null character, since we found that a large probability mass iteratively
accumulates in the null character (or for deletion). We also incorporated a
change in the original algorithm, whereby we made updates to the custom edit
distance matrix based on the optimal list of substitutions as per the current
state of the edit distance matrix, rather than choosing a minimal length path
at random (with each substitution counted as length $1$) from the source to
the target word when several exist. However, these variants result in very
minor improvements or even slight degradations to performance, and we do not
report these results.

## 7\. Details of released lexicons

We make our bilingual lexicons publicly available under a CC BY-NC 4.0 license
for Bhojpuri, Magahi, Awadhi, Braj, and Maithili, and also release our created
silver evaluation lexicons for Bhojpuri and Magahi under the same license.
These are the first publicly available bilingual lexicons all these languages
except Bhojpuri, to the best of our knowledge. The sizes of the released
lexicons for each target language are provided in Table 4. Note that while we
also release our generated lexicons for Marathi and Nepali, large high quality
gold bilingual lexicons already exist for these languages (see Section 5) and
should be used instead of ours; we are mainly interested in creating resources
for the low-resource languages.

## 8\. Conclusion

We introduce a novel method for unsupervised BLI between a related LRL and
HRL, which only requires a good quality MLM for the HRL. This addresses an
important gap in the existing literature, which often relies on good quality
embeddings for both languages. Our method shows superior performance on two
low-resource languages from the Indic continuum, against near-zero
performances of existing state-of-the-art methods. We perform control
experiments for two more distantly related Indic languages, and release
resulting bilingual lexicons for five truly low-resource Indic languages.

## Limitations

The applicability of our method is restricted to low-resource languages that
are related to a high-resource language. As the Basic and Rulebook method are
directly dependent on orthographic distance between translation pairs, they
are only useful for identifying cognate equivalents, borrowings, or alternate
spellings in the source and target language. We also clarify that our method
is not intended for mid-to-high resourced language pairs (such as
Marathi–Hindi), where canonical state-of-the-art methods such as VecMap work
more robustly, specifically on non-identical word equivalents. Our method
therefore has a specific (although important) target scenario, i.e. it is a
simple method to build bilingual lexicons for severely under-resourced
languages leveraging the resources of a closely related high-resource
language, given that state-of-the-art methods fail in these settings. Note
that we also only deal in the entirely unsupervised scenario in keeping with
typical conditions for our target languages (see Section 2), and leave it to
future work to improve these methods with a little supervision from bilingual
lexicons, possibly obtained from parallel data.

Another limitation of our work is that we were not able to provide true native
speaker evaluation for the resulting target language lexicons, instead
providing evaluation by the first author (Hindi native speaker) relying on
knowledge of shared cognates, the morphology of the target language, and
inflection tables. We provide examples in Table 6 and Table 2, and release the
automatically created as well as silver lexicons. Finally, our method is only
capable of providing single token (HRL) matches to the masked (LRL) whole
word. As discussed in Section 4, this problem does not affect the large
majority of cases. We leave it to future work to extend our idea to handle
multi-token words and multi-word expressions using, for example, span-filling
language models (Donahue et al., 2020).

## Ethics Statement

Our work is driven by the aim to boost NLP for severely under-resourced
languages of the Indic language belt, as well as contribute a method that may
be relevant to other language families with a similar linguistic and resource
setup. Our method relies on the predictions of language models for the high-
resource language and is therefore fallible to general ethical issues with
such models, including caste, religion, and gender biases shown to be
exhibited by such models (Malik et al., 2022).

## Acknowledgements

This work was partly funded by the last two authors’ chairs in the PRAIRIE
institute funded by the French national agency ANR as part of the
“Investissements d’avenir” programme under the reference ANR-19-P3IA-0001.
First and second authors are supported by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) —Project–ID 232722074— SFB 1102. Second and
third authors are supported by the EU project LT-Bridge (GA952194).

## 9\. Bibliographical References

##

  * Adams et al. (2017) Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017\.  Cross-Lingual Word Embeddings for Low-Resource Language Modeling.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 937–947, Valencia, Spain. Association for Computational Linguistics.
  * Artetxe et al. (2016) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016.  Learning Principled Bilingual Mappings of Word Embeddings While Preserving Monolingual Invariance.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2289–2294, Austin, Texas. Association for Computational Linguistics.
  * Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017.  Learning Bilingual Word Embeddings with (Almost) No Bilingual Data.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 451–462, Vancouver, Canada. Association for Computational Linguistics.
  * Artetxe et al. (2018a) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a.  A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 789–798, Melbourne, Australia. Association for Computational Linguistics.
  * Artetxe et al. (2018b) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b.  Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Artetxe et al. (2019) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019.  Bilingual Lexicon Induction through Unsupervised Machine Translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5002–5007, Florence, Italy. Association for Computational Linguistics.
  * Bafna et al. (2022) Niyati Bafna, Josef van Genabith, Cristina España-Bonet, and Zdeněk Žabokrtský. 2022.  Combining Noisy Semantic Signals with Orthographic Cues: Cognate Induction for the Indic Dialect Continuum.  In _Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)_ , pages 110–131, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching Word Vectors with Subword Information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Chimalamarri et al. (2020) Santwana Chimalamarri, Dinkar Sitaram, and Ashritha Jain. 2020.  Morphological Segmentation to Improve Crosslingual Word Embeddings for Low Resource Languages.  _ACM Transactions on Asian and Low-Resource Language Information Processing_ , 19(5):69:1–69:15.
  * Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018.  Word Translation Without Parallel Data.  _arXiv preprint: arXiv:1710.04087_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Donahue et al. (2020) Chris Donahue, Mina Lee, and Percy Liang. 2020.  Enabling language models to fill in the blanks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2492–2501, Online. Association for Computational Linguistics.
  * Durrett et al. (2012) Greg Durrett, Adam Pauls, and Dan Klein. 2012.  Syntactic Transfer Using a Bilingual Lexicon.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 1–11, Jeju Island, Korea. Association for Computational Linguistics.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013.  A Simple, Fast, and Effective Reparameterization of IBM Model 2.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–648, Atlanta, Georgia. Association for Computational Linguistics.
  * Eder et al. (2021) Tobias Eder, Viktor Hangya, and Alexander Fraser. 2021.  Anchor-based Bilingual Word Embeddings for Low-Resource Languages.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 227–232, Online. Association for Computational Linguistics.
  * Glavaš and Vulić (2020) Goran Glavaš and Ivan Vulić. 2020.  Non-Linear Instance-Based Cross-Lingual Mapping for Non-Isomorphic Embedding Spaces.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7548–7555, Online. Association for Computational Linguistics.
  * Gonen et al. (2020) Hila Gonen, Shauli Ravfogel, Yanai Elazar, and Yoav Goldberg. 2020.  It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 45–56, Online. Association for Computational Linguistics.
  * Haghighi et al. (2008) Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008.  Learning Bilingual Lexicons from Monolingual Corpora.  In _Proceedings of ACL-08: HLT_ , pages 771–779, Columbus, Ohio. Association for Computational Linguistics.
  * Irvine and Callison-Burch (2013) Ann Irvine and Chris Callison-Burch. 2013.  Combining Bilingual and Comparable Corpora for Low Resource Machine Translation.  In _Proceedings of the Eighth Workshop on Statistical Machine Translation_ , pages 262–270, Sofia, Bulgaria. Association for Computational Linguistics.
  * Jha (2019) Sanjay Kumar Jha. 2019.  Exploring the Degree of Similarities between Hindi and Maithili Words from Glottochronological Perspective.  _International Journal of Innovations in TESOL and Applied Linguistics_ , 5.
  * Joshi (2023) Raviraj Joshi. 2023.  L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages.  _arXiv preprint: arXiv:2211.11418_.
  * Khanuja et al. (2021) Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021.  MuRIL: Multilingual Representations for Indian Languages.
  * Kuriyozov et al. (2020) Elmurod Kuriyozov, Yerai Doval, and Carlos Gómez-Rodríguez. 2020.  Cross-Lingual Word Embeddings for Turkic Languages.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4054–4062, Marseille, France. European Language Resources Association.
  * Lamsal (2020) Rabindra Lamsal. 2020.  A Large Scale Nepali Text Corpus.
  * Malik et al. (2022) Vijit Malik, Sunipa Dev, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. 2022.  Socially Aware Bias Measurements for Hindi Language Representations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1041–1052, Seattle, United States. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient Estimation of Word Representations in Vector Space.  _arXiv preprint: arXiv:1301.3781_.
  * Mundotiya et al. (2021) Rajesh Kumar Mundotiya, Manish Kumar Singh, Rahul Kapur, Swasti Mishra, and Anil Kumar Singh. 2021.  Linguistic Resources for Bhojpuri, Magahi, and Maithili: Statistics about Them, Their Similarity Estimates, and Baselines for Three Applications.  _ACM Transactions on Asian and Low-Resource Language Information Processing_ , 20:1–37.
  * Patra et al. (2019) Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neubig. 2019.  Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 184–193, Florence, Italy. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ruder et al. (2019) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019.  A Survey of Cross-Lingual Word Embedding Models.  _Journal of Artificial Intelligence Research_ , 65(1):569–630.
  * Søgaard et al. (2018) Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018.  On the Limitations of Unsupervised Bilingual Dictionary Induction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 778–788, Melbourne, Australia. Association for Computational Linguistics.
  * Thompson et al. (2019) Brian Thompson, Rebecca Knowles, Xuan Zhang, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019.  HABLex: Human Annotated Bilingual Lexicons for Experiments in Machine Translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1382–1387, Hong Kong, China. Association for Computational Linguistics.
  * Vulić et al. (2019) Ivan Vulić, Goran Glavaš, Roi Reichart, and Anna Korhonen. 2019.  Do We Really Need Fully Unsupervised Cross-Lingual Embeddings? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4407–4418, Hong Kong, China. Association for Computational Linguistics.
  * Wada et al. (2019) Takashi Wada, Tomoharu Iwata, and Yuji Matsumoto. 2019.  Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3113–3124, Florence, Italy. Association for Computational Linguistics.
  * Woller et al. (2021) Lisa Woller, Viktor Hangya, and Alexander Fraser. 2021.  Do Not Neglect Related Languages: The Case of Low-Resource Occitan Cross-Lingual Word Embeddings.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , pages 41–50, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020.  Are All Languages Created Equal in Multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_ , pages 120–130, Online. Association for Computational Linguistics.
  * Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015.  Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1006–1011, Denver, Colorado. Association for Computational Linguistics.
  * Yuan et al. (2020) Michelle Yuan, Mozhi Zhang, Benjamin Van Durme, Leah Findlater, and Jordan Boyd-Graber. 2020.  Interactive Refinement of Cross-Lingual Word Embeddings.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5984–5996, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Jinpeng Zhang, Baijun Ji, Nini Xiao, Xiangyu Duan, Min Zhang, Yangbin Shi, and Weihua Luo. 2021.  Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2943–2955, Online. Association for Computational Linguistics.
  * Zhao et al. (2009) Hai Zhao, Yan Song, Chunyu Kit, and Guodong Zhou. 2009.  Cross Language Dependency Parsing using a Bilingual Lexicon.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 55–63, Suntec, Singapore. Association for Computational Linguistics.

## 10\. Language Resource References

##

  * Goldhahn et al. (2012) Goldhahn, Dirk and Eckart, Thomas and Quasthoff, Uwe. 2012.  _Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages_.  European Language Resources Association (ELRA).  PID https://aclanthology.org/L12-1154/.
  * Kakwani et al. (2020) Kakwani, Divyanshu and Kunchukuttan, Anoop and Golla, Satish and Gokul, NC and Bhattacharyya, Avik and Khapra, Mitesh M and Kumar, Pratyush. 2020.  _inlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages_.  Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings.  PID https://indicnlp.ai4bharat.org/home/.
  * Lamsal (2020) Lamsal, Rabindra. 2020.  _A Large Scale Nepali Text Corpus_.  IEEEdataport.  PID https://doi.org/10.21227/jxrd-d245.
  * Mundotiya et al. (2021) Mundotiya, Rajesh Kumar and Singh, Manish Kumar and Kapur, Rahul and Mishra, Swasti and Singh, Anil Kumar. 2021.  _Linguistic Resources for Bhojpuri, Magahi, and Maithili: Statistics about Them, Their Similarity Estimates, and Baselines for Three Applications_.  ACM Transactions on Asian and Low-Resource Language Information Processing.  PID https://github.com/singhakr/Bhojpuri-Magahi-and-Maithili-Linguistic-Resources.
  * Ojha (2019) Ojha, Atul Kr. 2019.  _English-Bhojpuri SMT System: Insights from the Karaka Model_.  arXiv.  PID http://arxiv.org/abs/1905.02239.
  * Ojha et al. (2020) Ojha, Atul Kr. and Malykh, Valentin and Karakanta, Alina and Liu, Chao-Hong. 2020\.  _Findings of the LoResMT 2020 Shared Task on Zero-Shot for Low-Resource languages_.  Association for Computational Linguistics.  PID https://aclanthology.org/2020.loresmt-1.4.
  * Zampieri et al. (2018) Zampieri, Marcos and Nakov, Preslav and Ljubešić, Nikola and Tiedemann, Jörg and Malmasi, Shervin and Ali, Ahmed. 2018.  _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)_.  Association for Computational Linguistics.  PID https://aclanthology.org/W18-3900.

| bho | mag
---|---|---
| P@1 | P@3 | P@5 | P@1 | P@3 | P@5
VecMap+CSLS | 0 | 0 | 0 | 1.2 | 1.2 | 1.2
Basic | 58.1 | 61 | 61 | 62.5 | 65.1 | 65.1
Rulebook | 59.1 | 61.6 | 61.6 | 63 | 65.4 | 65.4
Table 7: P@{1,3,5} for bho and mag. | bho | mag
---|---|---
| NIA@1 | NIA@3 | NIA@5 | NIA@1 | NIA@3 | NIA@5
VecMap+CSLS | 0 | 0 | 0 | 0.6 | 0.6 | 0.6
Basic | 23.7 | 17.1 | 17.1 | 27.2 | 18.4 | 18.2
Rulebook | 20.2 | 14.4 | 14.5 | 23.1 | 17 | 16.9
Table 8: NIA@{1,3,5} for bho and mag.

##  Appendix A Additional Results

We report P@1,3,5 in Table 7 and NIA@1,3,5 in Table 8. We see that both Basic
and Rulebook approaches do not benefit from considering more than 3 best
answers. In general, we see the same relative trend as in Table 5.