# Masked LARk:
Masked Learning, Aggregation and Reporting worKflow

Joseph J. Pfeiffer III, Microsoft<EMAIL_ADDRESS>
Denis Charles, Microsoft<EMAIL_ADDRESS>
Davis Gilton, Microsoft<EMAIL_ADDRESS>
Young Hun Jung, Microsoft<EMAIL_ADDRESS>
Mehul Parsana, Microsoft<EMAIL_ADDRESS>
Erik Anderson, Microsoft<EMAIL_ADDRESS>

###### Abstract

Today, many web advertising data flows involve passive cross-site tracking of
users. Enabling such a mechanism through the usage of third party tracking
cookies (3PC) exposes sensitive user data to a large number of parties, with
little oversight on how that data can be used. Thus, most browsers are moving
towards removal of 3PC in subsequent browser iterations. In order to
substantially improve end-user privacy while allowing sites to continue to
sustain their business through ad funding, new privacy-preserving primitives
need to be introduced.

In this paper, we discuss a new proposal, called Masked LARk, for aggregation
of user engagement measurement and model training that prevents cross-site
tracking, while remaining (a) flexible, for engineering development and
maintenance, (b) secure, in the sense that cross-site tracking and tracing are
blocked and (c) open for continued model development and training, allowing
advertisers to serve relevant ads to interested users. We introduce a secure
multi-party compute (MPC) protocol that utilizes ”helper” parties to train
models, so that once data leaves the browser, no downstream system can
individually construct a complete picture of the user activity. For training,
our key innovation is through the usage of masking, or the obfuscation of the
true labels, while still allowing a gradient to be accurately computed in
aggregate over a batch of data. Our protocol only utilizes light cryptography,
at such a level that an interested yet inexperienced reader can understand the
core algorithm. We develop helper endpoints that implement this system, and
give example usage of training in PyTorch.

Keywords: Privacy, Machine Learning, Multi-Party Compute

##  1 Introduction

Today, many web advertising data flows involve passive cross-site tracking of
users through the use of third party cookies (3PC). Such mechanisms enable
many valid use cases, such as reporting advertiser conversion metrics, which
can be used to identify which campaigns are more or less likely to lead to
desired behavior by a user. Examples of conversions include landing on a page
of interest, putting an item into a shopping cart, or making a purchase. More
advanced techniques deploy machine learning models to identify the likelihood
of a conversion, and adjust an advertiser’s corresponding bid in real-time.

However, a strong drawback to 3PC is that they expose sensitive user data to a
large number of parties, with little oversight on how that data can be used.
Thus, most browsers are moving towards removal of 3PC in subsequent browser
iterations (apple2020threepc, ; firefox2019threepc, ; chrome2021threepc, ). In
order to substantially improve end-user privacy while allowing sites to
continue to sustain their business through ad funding, new privacy-preserving
primitives need to be introduced. Several proposals
(google2020aggregatereporting, ; corrigangibbs2017prio, ) have been
introduced, primarily relying on the notion of a trusted third party charged
with aggregating and reporting metrics back to the ad server. This brings up
questions of how this third party can be trusted not to collude with various
ad servers. Some ideas exist, ranging from code audits to secure enclaves
(anciaux2019enclaves, ), but most have either large overhead or expense.

In contrast, a recent proposal from Google goes in a different direction:
utilizing secure multi-party compute (MPC) to build a trusted third party
comprised of multiple semi-trusted helpers (google2020aggregatereporting, ).
Each helper receives a piece of information necessary to complete a larger
picture, but without the complementary pieces held by other helpers the data
is effectively random. The primary assumption in this world is the lack of
collusion – so long as the helpers do not work together or with the
advertising server (breaking the protocol), no single party can recover the
original sample of information. Despite this restriction, the helpers can
follow assigned protocols to create useful aggregate values.

An illustration of the system is shown in Figure 1. The data begins with a
(large) collection of user data being held solely on the user’s device, the
browser. The browser creates secret shares of the data intended for multiple
helpers (e.g., $H0$ and $H1$). These shares are constructed in such a way that
an individual helper receives only a piece of the final value, which alone is
effectively random. The shares, or reports, are then encrypted with public
keys provided by the helpers (keys are unique to the helper) and passed on to
the advertiser server. The advertising server effectively acts as a database –
there’s very little available information that it can read, and it must rely
on the helpers to construct meaningful aggregate information about the users.
How the secret shares are constructed, and what is hidden, is central to the
overarching design of the system. A popular approach is to use protocols which
involve hiding both context about the conversion and the conversion value
(e.g., using distributed point functions (google2020aggregatereporting, )).

Figure 1: System overview: browsers track feature vectors and send encrypted
reports to an advertising server. The advertising server sends the reports to
the respective helpers, who decrypt and perform local aggregation. Noisy local
aggregate reports are returned to the advertising server, who combines them
for a complete (noisy) value.

Here, we propose Masked LARk, a Masked Learning, Aggregation and Reporting
worKflow.

  * •

We extend the existing proposal (google2020aggregatereporting, ) to a variety
of tasks outside of simple aggregation reporting. We envision Masked LARk as a
privacy preserving Map-Reduce platform, where browsers map user information
into secret shared representations, and secure MPC helpers perform a private
reduce operation.

  * •

We discuss how differentially private (dwork2014algorithmic, ) aggregated
statistics can be computed between browser and helpers and sent to the
advertising server.

  * •

We outline the usage of Masking, and show how it generalizes to a wide variety
of tasks. Notably, this allows us to compute gradients for differentiable
models, and train conversion models using secure MPC.

  * •

In Section 4.2.3, we show if we utilize local differential privacy on our
feature vectors, the data sent to the helpers is differentially private, and
by extension the learning algorithm is also differentially private.

  * •

We give several experiments on benchmark datasets, and plan to release a
Python package with example usage of Masked LARk for model training.

The remainder of the paper is structured as follows: we start in Section 2 by
discussing related work to this domain, including Google’s proposal, an
overview of MPC, and Federated Learning. Next, we give an overview of the
Masked LARk proposal in Section 3, discussing some types of functions that
Masked LARk can compute along with their privacy requirements. In Section 4,
we give an in-depth discussion of _masking_ and its usefulness for computation
over bi-linear functions. This includes model training algorithms that require
gradient computation. In Section 5, we show that we can indeed train models
using MPC helpers, with ablation studies on the impact of privacy settings and
local vs. global differential privacy. Section 6 concludes the paper. In
addition, we currently have an externally-available software package for
Masked LARk, available at
https://github.com/microsoft/maskedlark.

##  2 Related Work

Masked LARk sits within a community of proposals related to improving privacy
for web advertising. Notable related projects include FLoC (google2021floc, ),
FLEDGE (google2021fledge, ), PARAKEET (microsft2021parakeet, ) MACAW
(microsft2021macaw, ) or the Conversion Measurement proposal
(google2020aggregatereporting, ), amongst others. Of these, each set of
proposals has a different functionality: FLoC was proposed as a way to utilize
federated learning to create user clusters for privacy, which can then be
utilized by advertisers at ad serving time. FLEDGE / PARAKEET / MACAW are
proposals for running privacy-preserving auctions, where user signals are
masked from advertising servers entirely while an auction mechanism determines
which ads to show on a page.

In contrast, both Masked LARk and the Google’s proposal
(google2020aggregatereporting, ) work on data _after_ the impression, click or
view has occurred. Masked LARk largely builds onto the aggregated conversion
measurement first brought forward by Google (google2020aggregatereporting, ;
corrigangibbs2017prio, ). The overarching structure between the two is the
same; the browsers create encrypted reports intended for an MPC service,
utilizing the advertising service itself as an intermediate data storage.
Masked LARk differs from this proposal in two notable ways. First, Masked LARk
has browsers insert random noise into the data samples sent to the advertising
server, obfuscating real versus fake features when consumed by the helpers.
This is used to allow simpler downstream crypto protocols to mask only the
conversion values themselves, even preventing the helper services from making
connections from a feature vector to a label. In contrast, the original
proposal (google2020aggregatereporting, ) relies on a hierarchical
representation of bits requiring interactions between the helpers known as
distributed point functions (DPF) (boneh2020dpf, ), which explicitly hides the
features from the helpers as well. Second, Masked LARk envisions a more
general platform for computing many types of functions, rather than solely
aggregating measurement. A good example of such functions is the gradient
computation function for training machine learning models. Additional
functions that can be computed through the Masked LARk are the DPF
representations or other available protocols for secure multi-party
aggregation (msr2021bucketization, ). What functions to implement is a
community decision, discussed among many relevant parties.

The multi-party computation involves a rich history within cryptography, with
many primitives available for use. An accessible introduction to many of these
primitives can be found in the book (evans2018mpc, ). The protocols discussed
within this work center around either additive or multiplicative sharing
(masking), and their novelty lies in their application within the overall
workflow and its usage for model training. More complex model inference
(rathee2020cryptflow2, ) and model training (wagh2019securenn, ) exist, with
much stronger requirements on feature and model privacy. Conversely, Masked
LARk directly obfuscates the _label_ , relying on random noise to further
obfuscate the identifying information. This feature results in protocols with
no overhead between the helpers, and much simpler and more accessible
implementation.

An additional related field to our domain is _federated learning_
(kairouz2019fl, ), which is extremely decentralized training of machine
learning models. Many aspects of federated learning align well with our
proposed framework; for example, when gradient updates are computed per user,
these can be additive secret shared for later consumption by helpers for
summation. Hence, the involved cryptography becomes exceptionally scalable and
simple. However, a primary drawback is complexity on the _user’s_ device. We
have some unknown number of advertising services that would utilize the
platform. These services would push the model updates to individual devices,
each of which uses its own computing resources to perform the corresponding
updates. Such a system would necessitate moving large amounts of data to and
from users’ devices at irregular intervals, and would increase the
computational burden required to store and perform training updates. Hence,
this is an unattractive option.

##  3 Masked LARk

The Masked Learning, Aggregation and Reporting worKflow (Masked LARk) proposal
aims to solve multiple items:

  * •

Various private function computation using secure multi-party computation
(MPC)

  * •

Flexibility in representation of additional functions as new protocols and
needs arise

  * •

Simplicity in execution, focusing on decoupling users’ private information
from the advertiser websites they visit and/or convert on.

There are three groups of parties central to this platform

  * •

Browsers represent the individual user browsing and interacting with various
pieces of the web.

  * •

Advertising (Ad) Servers run the auction and provide ad impressions to the
browser to display to the user.

  * •

Helpers form an MPC platform for privacy-preserving function evaluation and
aggregation.

The defined workflow has similarities regardless of the protocol and function
to be executed. In effect, the workflow mirrors the heavily utilized MapReduce
systems (dean2008mapreduce, ) that are ubiquitous to today’s computation.
Masked LARk envisions two analogous pieces:

  * •

Map: browsers take the private data and encode it into a) a format for later
functions to consume and b) encrypted packages intended for MPC

  * •

Reduce: a collection of records is sent to the MPC platform with a function to
execute, along with side information and privacy constraints

The advertising server, acting as a database, remains unable to read even
minimal data throughout, and nothing specific to a user.

###  3.1 Browser Actions

Similar to the related proposals, browsers are responsible for a number of
tasks in the workflow, handling across-site attribution, secret sharing the
records and sending to the helpers. Within Masked LARk, specifically, there
are two notable additional responsibilities:

  * •

Upon the expiration of a possible attribution, if no attribution has occurred,
a default value (e.g., 0) must be sent. This lets models have negative samples
with which to train. The sample does not need to be sent right after the
expiration occurs, and should delay random additional intervals to prevent
timing attacks.

  * •

Browsers are charged with sending additional fake records to obfuscate true
values.

Both of these are important for privacy preservation and reasonable model
training. In certain circumstances, browser-injected-noise can guarantee
differential privacy, which we discuss in Section 4.2.3.

###  3.2 Helper’s Privacy Preserving Techniques

In addition to purely doing computation on the secret shares, helpers have
some additional features that protect user privacy: $k$-anonymity and
differential privacy.

######  Definition 3.1 ($k$-Anonymity (samarati2001protecting, )).

A helper releases the aggregates only when there are more than $k$ records.

The $k$-anonymity ensures that aggregation is done only when there is
sufficiently large data set. This procedure will prevent the case where the
aggregated output conveys too much information about individual data points.

We note that $k$ is measured on a per-record basis – in theory, all records
could be provided from the same user, but the helpers have no way to uniquely
identify users (by design). That is why we pair it with another technique,
Differential Privacy (dwork2014algorithmic, ). In a high level, either the
browser or the helper adds noise to the true value before sharing it with
others so that the data receiving entity is hard to infer about individual
records. Readers who are interested in the formal definition of differential
privacy can refer to the above book.

Depending on whether the noise is injected to the input or to the output,
there are local differential privacy and global differential privacy:

  * •

Local Differential Privacy: browsers are charged with either modifying or
inserting fake records, to break apart information that a helper could glean.
In some scenarios (see Section 4.2.3), we can enforce local differential
privacy in the browser.

  * •

Global Differential Privacy: a helper is charged with inserting noise into the
operation for global differential privacy.

Note that $k$-anonymity requires $k$ as a parameter and differential privacy
has its privacy budget $\epsilon$. These privacy settings are publicly
declared per advertising server, and are used by the browsers and the helpers
to enforce the privacy mechanism.

###  3.3 Example: Summation And Counting

A simple example that fits within Masked LARk is summation (or counting, a
simpler variant). Here, the browser is given a ring over some modulo on the
integers, i.e., $\mathcal{R}=\mathbb{Z}/m\mathbb{Z}$, and a value to return
$y$. Publicly declared privacy settings include

  * •

$k$, the minimum number of records required to perform private computation

  * •

$\epsilon$, the privacy budget

  * •

$\Delta_{y}$, the bounds of the value being summed, or the maximum change a
single record can have on the final summation. If $y$ is a value from a single
user, then $\Delta_{y}=\Delta_{y}^{max}-\Delta_{y}^{min}$, where
$\Delta_{y}^{min}\leq y\leq\Delta_{y}^{max}$ for all $y$

Algorithm 1 Additive Secret Sharing

Inputs: Field $\mathcal{R}$, value $v$

Draw $\psi^{0}\sim Unif(\mathcal{R})$

Compute $\psi^{1}=v-\psi^{0}$

Encrypt-And-Send $\psi^{0}$ to $H0$

Encrypt-And-Send $\psi^{1}$ to $H1$

Algorithm 2 Helper Partial Recovery of Secret Shares

Inputs: Privacy Parameters $k$ and $\epsilon$, sensitivity of function
$\Delta_{y}$, helper records $R^{h}$

if LEN($R^{h}$) $<$ $k$ then

Return 0

end if

Set $\Psi^{h}=0$

for $r^{h}$ in $R^{h}$ do

$\psi^{h}=$ Decrypt($r^{h}$)

Update $\Psi^{h}=\Psi^{h}+\psi^{h}$

end for

Perturb the output $\tilde{\Psi}^{h}=\Psi^{h}+Laplace(0,\Delta_{y}/\epsilon)$

Return $\tilde{\Psi}^{h}$

Here, the browser implements the sharing, while the the helpers perform the
(partial) recovery of the summation of secret shares. That is, the browser
aims to decompose $y$ into two secret shares such that $y=\psi^{0}+\psi^{1}$.
To do this, (e.g., the map operation) the browser implements Algorithm 1,
splitting the shares such that each helper effectively receives a random value
from a field $\mathcal{R}$.

For the corresponding Reduce operation, each helper receives a batch of
records from the ad server and decrypts them. It then executes Algorithm 2,
where it sums the list of shares it receives. The helpers additionally add
noise, or apply $k$-anonymity, and return their partial sums. For the final
recovery, the advertising server sums the values returned from the helpers,
e.g., $\tilde{\Psi}=\sum_{h}\tilde{\Psi}^{h}$. Note the recovery algorithm is
(intentionally) imperfect, due to the additive noise and the thresholds
required.

####  3.3.1 Query and Group By

Often, an advertising service will wish to break apart the conversion by the
corresponding campaign, or location, where the conversion happened. In such a
scenario, we allow the browser to pass additional side information to the
helper services, to allow for filtering requests by the advertising service
(e.g., what campaigns in Seattle have the most conversions).

The contextual information is exposed only to the helper services. The helpers
then repeatedly apply Algorithm 2 on subsets of data matching the query,
preserving $k$-anonymity and differential privacy with the conversion
summations returned to the ad server.

We note that keys are revealed to the helper service, although the primary
goal of preventing cross-site tracking is maintained due to the obfuscation of
the corresponding label. Thus, Masked LARk assumes an honest-but-curious
security model for the helpers. In addition, Masked LARk charges the browsers
to additionally insert fake records to cast doubt on the legitimacy of any
individual record.

##  4 Gradient Computation

Gradient computation is widely used in machine learning model training, and
this is a new feature that is supported by the Masked LARk. We begin by
introducing some extra notation.

###  4.1 Notation

Let $\mathbf{X}\subset\mathcal{X}$ indicate a set of user features drawn from
the space of possible features, and $\mathbf{Y}\subset\mathcal{Y}$ indicate
the set of user labels drawn from the space of possible labels. Additionally
$\mathcal{X}\subset\mathbb{N}^{d_{x}}$, $\mathcal{Y}\subset\mathbb{N}^{d_{y}}$
and both are bounded. Let $\mathcal{S}=\mathcal{X}\times\mathcal{Y}$ indicate
the sample space. Individual samples for a user $u_{i}$ are denoted as
$\mathbf{s}_{i}:=(\mathbf{x}_{i},\mathbf{y}_{i})\in\mathcal{S}$. Let $\Theta$
indicate a parameter space (i.e., a model), with $\theta\in\Theta$ indicating
a particularly chosen representation.

Let $g$ indicate a bi-linear _aggregation_ function over a set of samples
after applying a mapping
$f:\mathcal{X}\times\mathcal{Y}\times\Theta\rightarrow\mathbb{R}^{d}$. One
such aggregation is summation:

$g\left(\left(f(\mathbf{s}_{1},\theta),...,f(\mathbf{s}_{n},\theta)\right)\right):=\sum_{i\in
n}f(\mathbf{s}_{i},\theta).$

Our discussion and examples center around summation in particular, although
the results generalize to other bi-linear functions.

####  4.1.1 Real and Fake Labels

A user (or a browser) is the final location where labels and features are
combined to form a complete picture of the sample. Once leaving the browser
for storage at an ad server (and subsequent helpers), no single party should
be able to reconstruct the complete _real_ sample $\mathbf{s}_{i}$. Let
$\mathbf{\overline{y}}_{i}\in\mathcal{Y}\backslash\\{\mathbf{y}_{i}\\}$
indicate a fake label that is additionally sent from the browser, forming a
complete package of the form
$(\mathbf{x}_{i},\mathbf{y}_{i},\mathbf{\overline{y}}_{i})$.

There are additional constraints on the fake labels that must be considered.

  * •

The labels are assigned by the browser based on whether a conversion was
triggered: if no conversion is triggered, the default value will be assigned
by the browser. The default label, for labels $y\in\mathbf{y}$, is defined to
be $0$. Thus, one of the values $y_{i},\overline{y}_{i}$ must be 0.

  * •

Conversely, there’s no constraint that a real value must be sent; at times,
the browser is required to send cases with strictly fake labels.

  * •

We assume the ordering of real versus fake labels presented to the helpers is
random, although the corresponding masks must match the label ordering

We generally assume non-zero fake labels are assigned by random draws of
$\mathcal{Y}\backslash\\{y\\}$. For real-valued labels this assumption can be
too revealing. To handle these types of labels, we employ probabilistic
quantization to privatize the values. First, we quantize a space
$0\leq\Delta_{max}$ into $k$ evenly spaced buckets, with boundaries
$0,q_{1},...,q_{k},\Delta_{max}$. Let $y$ be the value to quantize, which must
lie between some $q_{i}\leq y\leq q_{i+1}$. We assign a quantized value
$\tilde{y}$ like so:

$\tilde{y}=\begin{cases}q_{i}&\text{with probability
}(q_{i+1}-y)/(q_{i+1}-q_{i})\\\ q_{i+1}&\text{otherwise}\end{cases}$

Hence, $\tilde{y}=\mathbb{E}[y]$, however, the true value is privatized from
downstream systems. If $\tilde{y}$ is not an integer, we repeat the
probabilistic thresholding to force $\tilde{y}$ to an integer.

####  4.1.2 Masks

Let $\alpha^{h}_{i}$ indicate a mask associated with a _real_ label
$\mathbf{y}_{i}$, while $\overline{\alpha}^{h}_{i}$ indicates masks associated
with a _fake_ label $\mathbf{\overline{y}}_{i}$. Similarly, $h$ indicates the
helper utilizing the mask. Without loss of generality, we assume only one real
sample and one fake sample, as well as two helpers. We define the following
equalities:

$\alpha^{0}_{i}+\alpha^{1}_{i}=1$
$\overline{\alpha}^{0}_{i}+\overline{\alpha}^{1}_{i}=0$

Define a ring $\mathcal{R}:=\mathbb{Z}/m\mathbb{Z}$, that is, a ring over a
modulus of the discrete integers. Simple examples are int8 or int16. We assume
$\alpha^{0}_{i},\overline{\alpha}^{0}_{i}\in\mathcal{R}$, and each are
uniformly at random from the discrete uniform distribution over $\mathcal{R}$.
Combining the above, we have $\alpha^{1}_{i}=1-\alpha^{0}_{i}$ and
$\overline{\alpha}^{1}_{i}=-\overline{\alpha}^{0}_{i}$.

###  4.2 Masking Multiplication

Consider a single data point received by a helper $h$, i.e.,
$(\mathbf{x}_{i},\mathbf{y}_{i},\mathbf{\overline{y}}_{i},\alpha^{h}_{i},\overline{\alpha}^{h}_{i})$,
along with a parameterization $\theta$ and a mapping $f$ to compute. The
corresponding other helper receives a matching set, with the same features and
labels, but their own mask values. The math below illustrates utilizing a
distinction of real and fake values, however, we stress the helper is not
aware of this distinction, and whether a value is real or fake is obfuscated
from the helpers.

Each helper begins by computing the functions
$f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)$ and
$f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)$, with the same results
assumed computed by both helpers (for probabilistic functions, a shared seed
must be used between the helpers). Each helper next computes:

$\alpha^{h}_{i}\cdot
f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)+\overline{\alpha}^{h}_{i}\cdot
f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)$

When combined, the resulting summation across the helpers becomes

$\begin{split}\sum_{h}\left[\alpha^{h}_{i}\cdot\right.&\left.f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)+\overline{\alpha}^{h}_{i}\cdot
f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)\right]\\\
&=f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)\sum_{h}\alpha^{h}_{i}+f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)\sum_{h}\overline{\alpha}^{h}_{i}\\\
&=f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)\cdot
1+f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)\cdot 0\\\
&=f(\mathbf{x}_{i},\mathbf{y}_{i},\theta).\end{split}$ (1)

We illustrate this in Figure 2. Equation 1 then can be applied to multiple
samples. Our aim is to sum over the function values applied to the true
samples. Working backwards, we find

$\begin{split}\sum_{i}&f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)=\\\
&=\sum_{i}\left[f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)\cdot
1+f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)\cdot 0\right]\\\
&=\sum_{i}\left[f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)\sum_{h}\alpha^{h}_{i}+f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)\sum_{h}\overline{\alpha}^{h}_{i}\right]\\\
&=\sum_{i}\left[\sum_{h}\alpha^{h}_{i}\cdot
f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)+\sum_{h}\overline{\alpha}^{h}_{i}\cdot
f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)\right]\\\
&=\sum_{h}\left[\sum_{i}\alpha^{h}_{i}\cdot
f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)+\sum_{i}\overline{\alpha}^{h}_{i}\cdot
f(\mathbf{x}_{i},\mathbf{\overline{y}}_{i},\theta)\right].\end{split}$ (2)

Thus, each helper can independently compute their sums and return the final
summation.

The above has a more general formulation as well:

######  Lemma 4.1.

Let $V$ be a $\mathcal{R}$-module equipped with a bilinear form
$\langle\cdot,\cdot\rangle$. There is a simple secret sharing protocol to
compute $\langle v,w\rangle$ for $v,w\in V$ with three parties.

###### Proof.

Suppose we have three parties, $H0$, $H1$ and $C$. Let $w$ be held by $C$, and
$H0,H1$ both hold $v$.

To compute $\langle v,w\rangle$, $C$ does an additive secret share protocol
with $H_{i}$. That is, $\alpha_{j}^{0},\alpha_{j}^{1}\in\mathcal{R}$,
$\alpha_{j}^{0}+\alpha_{j}^{1}=w_{j}$. $H_{0}$ computes
$\Psi^{0}=\sum_{j}\alpha_{j}^{0}$, $H_{1}$ computes
$\Psi^{1}=\sum_{j}\alpha_{j}^{1}$. Then $\langle v,w\rangle=\Psi^{0}+\Psi^{1}$
as $\mathcal{R}$ has a bi-linear form. ∎

######  Corollary 4.1.1.

Let $V$ be a $\mathcal{R}$-module equipped with a bilinear form
$\langle\cdot,\cdot\rangle$. Let $v\in V$, and $s\subseteq v$. There is a
simple secret sharing protocol to compute $\sum_{v_{j}\in s}v_{j}$ where $s$
is held by one party and $v$ by two other parties.

###### Proof.

Apply Lemma 4.1 with $w:=\mathcal{X}(s)$. ∎

Figure 2: This diagram illustrates masking. There are multiple labels (fake
and real). If summed vertically, the red simply cancels, and green remains.
However, as multiplication is distributed, we can instead have H0 and H1 sum
its vectors and return to the advertising server. When results are combined,
only the true vector remains.

####  4.2.1 Masked Gradient Model Training

With the general result above, we can apply a variety of functions $f$ to
gather aggregate statistics over the sample space. Possible $f$ mappings that
are desired to compute include summation
($f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)=\mathbf{y}_{i}$) and counting
($f(\mathbf{x}_{i},\mathbf{y}_{i},\theta)=1$). Each of these is handled with
simpler mechanisms in subsection 3.3. Another interesting case, however, is
model training.

For many machine learning paradigms, including supervised learning, we require
a label $y_{i}$ and a feature vector $\mathbf{x}_{i}$. The end goal is to
minimize, in some way, the following _loss function_ :

$\operatorname*{arg\,min}_{\theta}\sum_{i}L(F(\mathbf{x}_{i}),\mathbf{y}_{i},\theta),$

where $L$ is some loss function and $F$ is some function from
$\mathcal{X}\rightarrow\mathcal{Y}$, where $\mathbf{x}_{i}\in\mathcal{X}$ and
$\mathbf{y}_{i}\in\mathcal{Y}$. Let’s assume we have a differentiable model
(e.g., a neural network) for $F$, parameterized by $\theta$. A majority of
neural network optimization follows some variant of Stochastic Gradient
Descent (bottou2010sgd, ), using the following iterative approach to
optimization:

$\theta_{j+1}=\theta_{j}-\eta\nabla F(x_{i},y_{i},\theta_{j}),$

where $\eta$ is a hyperparameter known as the learning rate. In practice, we
aggregate a number of data points at once using minibatches of data:

$\theta_{j+1}=\theta_{j}-\eta\sum_{i}\nabla F(x_{i},y_{i},\theta_{j})$

Thus, we let $f:=\nabla F$ in our above notation, meaning the helpers can
compute masked aggregate gradients and send them back to the advertising
server, and repeat. There are a variety of per-coordinate learning rates, such
as ADAM (kingma2017adam, ), AdaGrad (duchi2011adagrad, ), or AdaBound
(luo2019adabound, ). These methods still rely on previously computed
gradients, and utilizing MPC to compute the gradients works within these
optimization methods as well.

####  4.2.2 Global Gradient Noise

The browser actions mirror the discussion above; each browser must create
multiple labels and mask them before passing them to the helpers. The privacy
constraints, however, must be enforced by the helpers. For this, we adapt an
existing work (song2013dpsgd, ) to compute gradients with (globally)
differentially private updates.

As discussed in Section 3, we require both a $k$-anonymity constraint and
differential privacy budget declared in a public parameter server. When a
helper receives a batch of data and model to compute gradients over, it
follows the algorithm described in Algorithm 3.

Algorithm 3 Helper Computation

Inputs: gradient bound $\psi$, privacy parameter $\epsilon$, $k$ for
$k$-anonymity

if minibatch size $<k$ then

Return zero vector

end if

for each sample $(x_{i},y_{i},\alpha^{h}_{i})$ in the minibatch do

Compute $G(x_{i},y_{i})=\nabla L(F(\mathbf{x}_{i}),\mathbf{y}_{i},\theta)$

$\hat{G}(x_{i},y_{i})=\alpha^{h}_{i}\cdot
G(x_{i},y_{i})/\max(\psi,|G(x_{i},y_{i})|)$

end for

$\tilde{G}(x_{i},y_{i})=\hat{G}(x_{i},y_{i})+\eta$, where $\eta\sim
Laplace(\frac{\psi}{\epsilon})$.

Return $\sum_{i}\tilde{G}(x_{i},y_{i})$

We record the following theorem.

######  Theorem 4.2 (Theorem 3.6 (dwork2014algorithmic, )).

Let $\Delta$ denote the sensitivity of a function $f$. The Laplace mechanism
with scale $\Delta/\epsilon$ preserves $(\epsilon,0)$-differential privacy.

This ensures that the helper output of Algorithm 3 is differentially private
with the privacy budget $\epsilon$. We note other work (abadi2016deep, )
relaxes the privacy guarantee to $(\epsilon,\delta)$-differential privacy by
adding Gaussian noise instead of Laplacian noise, which could be explored in
future work.

####  4.2.3 Local Differential Privacy

An alternative approach would be to enforce local differential privacy by
adding noise to the data being sent to the helpers (kasiviswanathan2011can, ).
More precisely, given a feature vector $\mathbf{x}_{i}$, where each coordinate
is bounded by some $\Delta$-sensitivity and there is a publicly available
privacy parameter $\epsilon$, let each browser pass the following to the
helpers111As a technical note, in practice we restrict feature values to a
byte representation for space considerations, meaning the noise must be
clipped at 0 and 255:

$\tilde{x}_{i}=x_{i}+Laplace(0,\Delta/\epsilon)$ (3)

There are a number of trade-offs here in comparison to utilizing global
differential privacy:

  * •

The feature vectors presented to the helpers are $\epsilon$-differentially
private.

  * •

So long as the collection mechanism is $\epsilon$-differentially private, then
calculating gradients on the collected records is also
$\epsilon$-differentially private (dwork2006calibrating, ).

  * •

The helpers no longer need to clip gradients per sample (expensive
computation) or add global noise to the sums.

  * •

Local differential privacy is generally noisier than global differential
privacy, with less signal available for learning. We explore the costs of this
approach in Section 5.4.

We pause to highlight that an ambitious adversary that is able to leverage
both an advertising server and one of the helper services could work to
recover a mapping between the locally differentially private vector and the
true label by manipulating the input network and returned gradient. We believe
the effectiveness of this attack is limited, because of our k-anonymity
constraint and the assumed limited cardinality of the label space.

##  5 Experiments

For demonstration purposes, we have implemented MaskedLARk as a training
service for neural networks. In this section, we outline practical
considerations that need to be made during practical implementation of
MaskedLARk, as well as study the effect of a number of privacy-related
parameters on model performance.

(a) WBCD

(b) MNIST

Figure 3: Both the gradient bound and Laplace noise power are important
variables to set for privacy and performance purposes. Here we plot the effect
of decreasing the magnitude of noise added to aggregated gradients on
inference accuracy for a variety of gradient norm settings.

###  5.1 Setting

We have implemented our demo MaskedLARk as a three-party system, in which a
central node simulates the actions of both a browser and ad server, sending
requests to two helper services which cannot communicate with each other. Here
we focus on utilizing the helper services for gradient computation for the
purposes of training a feedforward neural network.

####  5.1.1 Implementation Details

We demonstrate MaskedLARk using two datasets: the MNIST digit classification
dataset (lecun1998gradient, ) and the Wisconsin Breast Cancer Dataset (WBCD)
(street1993nuclear, ). Both of these datasets are standard classification
benchmarks, while not requiring massive networks to achieve reasonable test
accuracy. The MNIST dataset has 50000 training examples, input size of 784,
and 10 classes, while the WBCD has 500 training examples, input size of 30,
and a binary output. Notably, WBCD has a class imbalance, with roughly 75$\%$
of all datapoints being “0” instances. MNIST has perfectly balanced labels.

(a) WBCD

(b) MNIST

Figure 4: Making requests to helper services incurs a latency cost to
training. Here we present the multiplicative latency increase from training
our network via helper service versus loading all data locally. For small
amount of data sent, most requests consist mainly of encoding the network,
rather than the data. In general, we recommend using small networks and
sending many data points at once, both for $k$-anonymity purposes as well as
latency.

Our networks are simple feedforward networks with ReLU activations; the MNIST
network has one hidden layer with 500 nodes, and the WBCD network has two
hidden layers with 50 nodes. We note that the MNIST data and network
demonstrated here are higher-dimensional than the intended use-case for
MaskedLARk.

All gradient calculation is done in PyTorch on the helper and central
processor side, and training is done via simple Stochastic Gradient Descent.
To communicate between central processor and helper, all networks are encoded
as ONNX models (bai2019, ). For latency and space reasons all network
parameters are compressed to 8-bit values before sending (see Section 5.3 for
reasoning).

###  5.2 Effect of Privacy Mechanisms on Model Performance

As users prepare to deploy differential privacy mechanisms, a central concern
is the effect of privacy on downstream model performance. In this Subsection,
we explore the role of gradient clipping and the magnitude of Laplacian noise
added to the gradients after aggregation. In Figure 3 we plot the trade-off
for a range of gradient bounds.

We observe nontrivial effects on model performance when gradient clipping is
used, as well as detrimental effects for high levels of Laplacian noise.
However, there are wide ranges of parameters at which competitive test-time
accuracy can be achieved, so moderate privacy constraints need not require
sacrificing useful models.

(a) WBCD

(b) MNIST

Figure 5: Simple neural networks are quite resilient to Laplacian noise
injected into input features as part of a local differential privacy
mechanism. Above we plot the network test accuracy against the log $\epsilon$
of injected Laplacian noise with variance $1/\epsilon$, so moving to the right
in the above plots decreases noise power.

###  5.3 Latency and Training Time

Here we explore the latency effects of moving gradient computation to remote
helpers. When computation must occur at the helper level, efficient
communication is of a primary concern. In Figure 4, we explore the effect of
changing the number of data points sent to the helpers in a single request.

We compare the amount of time required to train a network utilizing a single
CPU with 64 GB of RAM, capable of storing the entire network and dataset in-
memory. We duplicate the algorithm used by the helper, including gradient
clipping and noise injection, so that the extra time comes only from packaging
requests, parsing requests, and data transmission.

The most obvious takeaway is that sending as much data as possible at the same
time is desirable from the perspective of training over the whole dataset.
Small batch sizes will often be disallowed by $k$-anonymity rules, and as
Figure 4 suggests, batch sizes that are too small also incur significant
latency costs. This phenomenon is a consequence of sending data coupled with
the networks that process that data; sending small number of data will result
in the network needing to be compressed and parsed more frequently.

However, recent work in training deep networks suggests that batch sizes that
are too large may result in poor generalization at inference-time
(keskar2017large, ). In addition, practical helpers will have to place limits
on requests, to prevent slow gradient calculation turnaround when multiple
parties are making requests.

The size of the network being sent is central to getting timely results. The
network we used for MNIST is intended to be a stress-test, and is far too
large for practical use. Since we use the ONNX protobufs as the method to
communicate the networks, we recommend _quantization_ of network parameters.
Quantization of deep networks can range from simple limitations of the bit
depth of parameters (courbariaux2014training, ) to more complex schemes
(gong2014compressing, ; han2015deep, ), any of which may be applied here, as
long as the helper does not need to modify its protocol.

###  5.4 Adding Noise to Data Directly: An Alternative Approach to Privacy

While there are many advantages to the global differential privacy approach to
Masked LARk, we observe that training of large networks with massive datasets
may incur prohibitive delays due to point-wise application of gradient
clipping during gradient calculation. Recently, some techniques have been
proposed to improve the speed of this process (lee2021scaling, ), but this
step remains a major bottleneck.

A reasonable alternative to clipping gradients and adding noise after
summation is to instead add differential privacy noise _before_ gradient
calculation. This scheme is referred to as local Differential Privacy
(kasiviswanathan2011can, ), so that privacy can be handled browser-side for
gradient calculation. So long as the collection mechanism is
$\epsilon$-differentially private, then calculating gradients on the collected
records is also $\epsilon$-differentially private (dwork2006calibrating, ).

We test this system, again leveraging the Laplace mechanism, but applying
noise to the network inputs. Concretely, we send the helpers triplets
$(x_{i}+\eta,y_{i},\alpha_{i})$, where $\eta\sim Laplace(\frac{1}{\epsilon})$.
The helpers then return the sum $\sum_{i}\alpha_{i}\nabla
L(f(x_{i}+\eta),y_{i})$. We plot the performance of the trained networks as a
function of $\epsilon$ in Figure 5.

Figure 5 can be compared to Figure 3, since they both compare the effect of
privacy settings on trained network performance. However, local differential
privacy (in this section) often requires more noise to be injected than global
differential privacy (in Section 5.2). With that in mind, we observe that the
deep networks trained in our experiments are quite resilient to noise at the
feature level, even with quite high variances compared to the scale of the
features.

##  6 Discussion

In this paper, we have proposed Masked LARk, a Masked Learning, Aggregation
and Reporting worKflow. Masked LARk has several interesting properties:

  * •

It extends the existing proposal in (google2020aggregatereporting, ) to a wide
variety of tasks, outside of simple aggregation reporting, and most notably to
enable model training.

  * •

We discussed how differentially private (dwork2014algorithmic, ) aggregated
statistics can be computed and sent to the advertising server using browsers
and helpers. This enables user privacy protection, while allowing advertising
services to show relevant ads to interested consumers.

  * •

We outlined the usage of Masking, and showed how it generalizes to a wide
variety of tasks. Notably, this allows us to compute gradients for
differentiable models, and train conversion models using secure MPC.

  * •

In Section 4.2.3, we showed if we utilize local differential privacy on our
feature vectors, the data sent to the helpers is differentially private, and
by extension the learning algorithm is also differentially private.

  * •

We perform several experiments on benchmark datasets, exploring the tradeoffs
between differential privacy and accuracy. We plan to release a Python package
with example usage of Masked LARk and a number of utilities for model
training.

So far, the endpoint helper services are set up and can run aggregation and
basic model training. However, additional work needs to be done:

  * •

A public parameter service needs to be set up, where helpers can access
publicly declared privacy settings for advertising servers

  * •

Data sanitation is largely ignored at this time. There does exist quite a bit
of work in this space (rathee2020cryptflow2, ), meaning we need to add
additional functionality to Masked LARk to enable it.

  * •

Additional proposals for aggregate measurement exist (msr2021bucketization, ),
and we need community feedback on which version of aggregation should be
implemented. To date, no other proposals exist for model training.

To this end, we will continue to work with the advertising community to agree
upon standards which guarantee user privacy.

## References

  * (1) “Apple updates safari’s anti-tracking tech with full third-party cookie blocking.” https://www.theverge.com/2020/3/24/21192830/apple-safari-intelligent-tracking-privacy-full-third-party-cookie-blocking.  Accessed: 2021-09-13.
  * (2) “Today’s firefox blocks third-party tracking cookies and cryptomining by default.” https://blog.mozilla.org/en/products/firefox/todays-firefox-blocks-third-party-tracking-cookies-and-cryptomining-by-default/.  Accessed: 2021-09-13.
  * (3) “Google delays blocking third-party cookies in chrome until 2023.” https://www.theverge.com/2021/6/24/22547339/google-chrome-cookiepocalypse-delayed-2023.  Accessed: 2021-09-13.
  * (4) “Multi-browser aggregation service explainer.” https://github.com/WICG/conversion-measurement-api/blob/main/SERVICE.md.  Accessed: 2021-09-10.
  * (5) H. Corrigan-Gibbs and D. Boneh, “Prio: Private, robust, and scalable computation of aggregate statistics,” 2017.
  * (6) N. Anciaux, L. Bouganim, P. Pucheral, I. S. Popa, and G. Scerri, “Personal Database Security and Trusted Execution Environments: A Tutorial at the Crossroads,” Proceedings of the VLDB Endowment (PVLDB), Aug. 2019.
  * (7) C. Dwork, A. Roth, et al., “The algorithmic foundations of differential privacy.,” Found. Trends Theor. Comput. Sci., vol. 9, no. 3-4, pp. 211–407, 2014.
  * (8) “Federated learning of cohorts (floc).” https://github.com/WICG/floc.  Accessed: 2021-09-13.
  * (9) “First experiment (fledge).” https://github.com/WICG/turtledove/blob/main/FLEDGE.md.  Accessed: 2021-09-13.
  * (10) “Macaw.” https://github.com/WICG/privacy-preserving-ads/blob/main/MACAW.md.  Accessed: 2021-09-13.
  * (11) “Parakeet.” https://github.com/WICG/privacy-preserving-ads/blob/main/Parakeet.md.  Accessed: 2021-09-13.
  * (12) D. Boneh, E. Boyle, H. Corrigan-Gibbs, N. Gilboa, and Y. Ishai, “Lightweight techniques for private heavy hitters,” CoRR, vol. abs/2012.14884, 2020\.
  * (13) “Secure multi-party aggregation via bucketization.” https://github.com/WICG/privacy-preserving-ads/blob/main/Bucketization.md.  Accessed : 2021-09-10.
  * (14) D. Evans, V. Kolesnikov, and M. Rosulek, “A pragmatic introduction to secure multi-party computation,” Found. Trends Priv. Secur., vol. 2, p. 70–246, Dec. 2018.
  * (15) D. Rathee, M. Rathee, N. Kumar, N. Chandran, D. Gupta, A. Rastogi, and R. Sharma, “Cryptflow2: Practical 2-party secure inference,” Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, Oct 2020.
  * (16) S. Wagh, D. Gupta, and N. Chandran, “SecureNN: 3-Party Secure Computation for Neural Network Training,” Proceedings on Privacy Enhancing Technologies, 2019.
  * (17) P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. A. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D’Oliveira, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, M. Raykova, H. Qi, D. Ramage, R. Raskar, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao, “Advances and open problems in federated learning,” CoRR, vol. abs/1912.04977, 2019.
  * (18) J. Dean and S. Ghemawat, “Mapreduce: simplified data processing on large clusters,” Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008\.
  * (19) P. Samarati, “Protecting respondents identities in microdata release,” IEEE transactions on Knowledge and Data Engineering, vol. 13, no. 6, pp. 1010–1027, 2001.
  * (20) L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in COMPSTAT, 2010.
  * (21) D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.
  * (22) J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” Journal of Machine Learning Research, vol. 12, no. 61, pp. 2121–2159, 2011.
  * (23) L. Luo, Y. Xiong, Y. Liu, and X. Sun, “Adaptive gradient methods with dynamic bound of learning rate,” 2019.
  * (24) S. Song, K. Chaudhuri, and A. D. Sarwate, “Stochastic gradient descent with differentially private updates,” in 2013 IEEE Global Conference on Signal and Information Processing, pp. 245–248, 2013.
  * (25) M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308–318, 2016.
  * (26) S. P. Kasiviswanathan, H. K. Lee, K. Nissim, S. Raskhodnikova, and A. Smith, “What can we learn privately?,” SIAM Journal on Computing, vol. 40, no. 3, pp. 793–826, 2011.
  * (27) C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of cryptography conference, pp. 265–284, Springer, 2006.
  * (28) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  * (29) W. N. Street, W. H. Wolberg, and O. L. Mangasarian, “Nuclear feature extraction for breast tumor diagnosis,” in Biomedical image processing and biomedical visualization, vol. 1905, pp. 861–870, International Society for Optics and Photonics, 1993.
  * (30) J. Bai, F. Lu, K. Zhang, et al., “Onnx: Open neural network exchange.” https://github.com/onnx/onnx, 2020.
  * (31) N. S. Keskar, J. Nocedal, P. T. P. Tang, D. Mudigere, and M. Smelyanskiy, “On large-batch training for deep learning: Generalization gap and sharp minima,” in 5th International Conference on Learning Representations, ICLR 2017, 2017.
  * (32) M. Courbariaux, Y. Bengio, and J.-P. David, “Training deep neural networks with low precision multiplications,” arXiv preprint arXiv:1412.7024, 2014\.
  * (33) Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” arXiv preprint arXiv:1412.6115, 2014\.
  * (34) S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149, 2015.
  * (35) J. Lee and D. Kifer, “Scaling up differentially private deep learning with fast per-example gradient clipping.,” Proc. Priv. Enhancing Technol., vol. 2021, no. 1, pp. 128–144, 2021.


#  OCR Synthetic Benchmark Dataset for
Indic Languages

Naresh Saini1,2∗ Promodh Pinto1,2∗ Aravinth Bheemaraj1,2
Kumar Deepak1,2 Dhiraj Daga1,2 Saurabh Yadav1,2 Srihari Nagaraj1,2
1EkStep Foundation, 2Tarento Technologies

###### Abstract

We present the largest publicly available synthetic OCR benchmark dataset for
Indic languages. The collection contains a total of 90k images and their
ground truth for 23 Indic languages. OCR model validation in Indic languages
require a good amount of diverse data to be processed in order to create a
robust and reliable model. Generating such a huge amount of data would be
difficult otherwise but with synthetic data, it becomes far easier. It can be
of great importance to fields like Computer Vision or Image Processing where
once an initial synthetic data is developed, model creation becomes easier.
Generating synthetic data comes with the flexibility to adjust its nature and
environment as and when required in order to improve the performance of the
model. Accuracy for labeled real-time data is sometimes quite expensive while
accuracy for synthetic data can be easily achieved with a good score.

Keywords: Synthetic Data Generation, Indian Languages, Scene Text, Optical
Character Recognition (OCR), Indic Text Recognition, Neural Machine
Translation.

##  1 Introduction

Synthetic data generation is the process of generating images and
corresponding ground truths with different font family, font colour, font
size, background color, space between words and many more. Through synthetic
data generation we can generate unlimited training data for OCR without any
annotators. The major goal of generating synthetic data set is to validate
different OCR models and it can also be used to improve OCR accuracy for
different vernacular languages. Pre-trained open source OCR models like
tesseract and other deep learning based models are not very accurate on
vernacular languages and also not very accurate on scanned and colourful
documents irrespective of languages. To increase the accuracy one can fine-
tune pre-existing models and for that we need more training data with
different variations. We can generate custom data set to generate custom
models with higher accuracy. Synthetic data originated in the ’90s, but the
real usage occurred in the recent years with people understanding the risks in
data science and how fairly that can be eliminated with the usage of synthetic
data.

We approach the synthetic image generation problem on an exhaustive coverage
of multilingual set of document images. Our work could be used in evaluation
of the impact of synthetic data attributes on model performance. We fine tuned
the tesseract model for Indian vernacular languages using synthetic data and
real data so we were able to achieve around 1 percentage higher accuracy.

OCR (Optical character recognition) performance has increased over the time
with the rise of different neural networks. Many of these advances have
occurred in constrained settings, where the images are drawn from a single
domain and language, such as book scans or scene text. Performance of these
models goes down in case of documents of different domains and languages.
Performance of the model also deteriorates in case of complex background, new
fonts, different lighting conditions and sparse text. If these data variations
are not in training data in that case model performance will not be that
accurate and it’s also difficult to collect and annotate these types of data.
One of the primary challenges with creating better models for the
unconstrained scenario is the lack of annotated training data. Neural models
require tens of thousands of annotated images and ground truth text that are
both expensive and time consuming to produce.

To solve this problem one of the effective approaches is synthetic data
generation. We can generate lots of data from a few texts and fonts but
however, this common approach to synthetic models breaks down when the
attributes of the synthetic images do not match the test images. To avoid this
issue we generated images which were close to test data. In this work we have
generated synthetic data for 22 Indian languages recognised by the
constitution of India. For each language we have generated around 3500 images
corresponding to different available fonts in that language. These fonts are
both old(Unicode and Non-Unicode) and new(Unicode and Non-Unicode) Etter et
al. (2019).

Table 1: Language Wise Font Distribution

Language | Fonts |  Total Count
---|---|---
|  | Unicode | Non-Unicode
Hindi |  | timesroman, krutidev,helvatica,gargi,aparajita,cambay-italic,akshar non-unicode,samanata,cdacotygn-old-unicode,sakalbharati normal,mukta new
---
non-unicode,kokila,aparajita-old-unicode,sarai,nirmala,cambay-regular,hind-
regular,mangal old non-unicode,notosans-regular,shree dv0726 ot -
old non-unicode,kalimati,nakula,sarai-07,liberationserif-regular
20 | 5
Sanskrit |  | akshar,cambay-italic,nirmala,nakula,sarai,sarai-07,aparajita-old-unicode,mangal old non-unicode,kokila,lohit devanagari non-unicode,gargi,mukta
---
-regular,notosans-regular,poppins-black,rajdhani-bold,mangal,cambay-regular,samanata,sakalbharati normal,aparajita-old,hind-regular,shree-
dv0726 ot old non-unicode,cdacotygn-old,samyak devanagari,kalimati
21 | 5
Tamil |  | liberationserif,helvetica old,symbol old,tscu-saiindira-old,courier old,akshar,tmotabbi-ship-old,muktamalar-bold,kavivanar-regular,baloothambi2
---
-semibold,arial-unicode-ms,pavanam-regular,notoseriftamil-italic-variablefont,arimamadurai-black,latha,catamaran-variablefont-wght,nirmala,lohit-
tamil,meerainimai-regular,hindmadurai-semibold,latha old ,tmotabb-ship-
old,karlatamilinclined regular
17 | 3
Telugu |  | gautami,helvetica,akshar-old,timesroaman old,courier old,vemana ansi telugu font nonunicode old,notosanstelugu variablefont wdthwght nonunicode
---
hindguntur-bold,gurajada–old,suranna-regular,hindguntur-regular,raviprakash
nonunicode old,notoseriftelugu variablefont wght nonunicode,peddana-
regular,mandali regular–old,gidugu–old,othana2000 regular nonunicode
old,ntr–old,ponnala regular ansi nonunicode old,ramaraja-regular–old,lohit
telugu nonunicode old
10 | 7
Malayalam |  | liberationserif,helvetica old,timesroman old,meera-regular-old,ml nila04 nonunicode old,suruma nonunicode,rachana-old,kartika old nonunicode,
---
akshar-regular-old-unicode,manjari-regular,lohit-malayalam old
nonunicode,baloochettan2-regular,chilanka-regular,anjalioldlipi–old,noto serif
malayalam nonunicode
7 | 6
Kannada |  | helvetica old,malige b nonunicode,arialmt old,kedage n nonunicode,kar kuvempu,lohit kannada nonunicode,malige i nonunicode,tunga-regular,
---
kar k s narasimhaswamy,kar da raa bendre,malige t ,kar puchamthe,kedage i
nonunicode,kedage t nonunicode,kar chandrashekhara kambara
akshar-regular,kar puthina,kar u r ananthamurthy,kar vi kru gokak,kar
shivarama karantha,malige n nonunicode,kar gopalakrishna,adiga,kar
girish karnad,kedage b nonunicode
14 | 9
Oriya |  | shreeorix,liberationSans old,arial,mstt31,shree old,arialmt new,akrutioribhuban,SHREE,timesroman old,cairofont nonunicode,or1-jagannatha-old,
---
utkal old nonunicode,samyak old nonunicode,or14-utkal-old,notosansoriya thin
nonunicode,kalinga-old,notosansoriya bold nonunicode,lohit-
oriya old nonunicode,baloobhaina2-extrabold,or4-nilachala-old,notosansoriya
regular nonunicode,baloobhaina2-regular
6 | 6
Punjabi |  | helvetica,corbelLight old,arial old,mera,gurbaniAkhar old,puntypo7 nonunicode,muktamahee-regular,lohit nonunicode,notoserifgurmukhi regular-
---
nonunicode,notoserifgurmukhi variablefont wght
nonunicode,baloopaaji2-regular,langar-regular,gur old letterpress 4 book
nonunicode,dwarka 5-
medium nonunicode,baloopaaji2-bold,muktamahee-extrabold,raavi
6 | 6
Marathi |  | shreeDev,devnagari old,arialUnicode old,shree dv0726 ot old,lohit devanagari nonunicode,sakalbharati normal,aparajita-old,akshar unicode,mukta-
---
regular,notosans regular nonunicode,kokila,cambay-italic,samyak devanagari
nonunicode,sarai,cdacotygn-old,mangal nonunicode,mangal old-
nonunicode,gargi
9 | 6
Gujrati |  | albanyamt,shruthi,saral,samayak,padmaa,timesroman old,shruti,rekha,padmaabold,gopika nonunicode,saumil,lohit nonunicode,rasa rajkot old-
---
nonunicode,notosansgujarati black nonunicode,baloobhai2-bold,notoserifgujarati
variablefont,hindvadodara-semibold,aakar-old,bhavnagar-old,
bhuj-old,muktavaani-regular,rekha,ekatra-old,samyak old nonunicode
10 | 5
Konkani |  | nirmala,shreedev,mangal,arial,brhdevanagari,notosans regular nonunicode,hind-regular,akshar,aparajita-old,cdacotygn-old,lohit devanagari non-
---
unicode,cambay-regular,mangal nonunicode,samyak devanagari nonunicode,mangal
old nonunicode,shree dv0726 ot old nonunicode
6 | 6
Nepali |  | preeti, APSDVstardust,chankya old,AakritiCK,kavita,arial,notosansdevanagari,mangal,kokila,kalimati regular old nonunicode,poppins-bolditalic,poppins
---
-extrabolditalic,mukta-bold,poppins-thin,mukta-medium,poppins-mediumitalic,poppins-extralight,poppins-medium,mukta-regular,poppins-black,
mukta-semibold,poppins-regular,poppins-italic,poppins-thinitalic,poppins-
extralightitalic,rajdhani-semibold,poppins-semibolditalic,poppins-semibold
,rajdhani-regular,poppins-bold
20 | 1
Kashmiri |  | mehr nastaliq web 3,alvi nastaleeq regular,alqalam naqsh regular,pak nastaleeq old,jameel noori nastaleeq kasheeda,zohar-old,alqalam taj nastaleeq-
---
regular,aslam,naskh-old,alqalam nabeel regular,jameel-noori-nastaleeq
10 | 1
English |  | timesnewroman,bookantiqua nonunicode old,arial,tahoma nonunicode old,constantia nonunicode old,times,calibri,trebuchetms old nonunicode
---
fraunces-9pt-semibolditalic,notosans-bold,courier prime nonunicode,nixieone
old nonunicode,timesbd0 nonunicode,exo2-semibold old nonunicode
stanberry old nonunicode,zai-courierpolski1941 nonunicode old,opensans-
variablefont-wdth,wght nonunicode,ariallgt nonunicode
4 | 7
Urdu |  | mehr nastaliq web 3,alvi nastaleeq regular,alqalam naqsh regular,pak nastaleeq old,jameel noori nastaleeq kasheeda,zohar-old,alqalam taj nastaleeq-
---
regular,aslam,naskh-old,alqalam nabeel regular,jameel-noori-nastaleeq
10 | 1
Dogri |  | sakalbharatidv-normal,gist-mai-dvotdhruv-n-ship-old,notosansdevanagari black,gist-nepotdhruv-n-ship-old,gist-snd-dvotdhruv-n-ship-old,notosans-
---
devanagari- medium,inknutantiqua-medium,gist-kok-dvotdhruv-n-ship-
old,notosansdevanagari semibold,gist-snd-dvotvinit-n-ship-old,inknutantiqua
-semibold,gist-doi-dvotvinit-n-ship-old
9 | 3
Sindhi | notosansdevanagari black,mangal old,notosansdevanagari medium,gist-snd-dvotvinit,notosansdevanagari semibold,lohit-sindhi-old,gist-snd-dvotdhruv | 3 | 4
Maithali |  | sakalbharatidv-normal,notosansdevanagari black,inknutantiqua-semibold ,gist-nepotdhruv-n-ship-old,notosansdevanagari semibold,gist-doi-dvotvinit-n
---
-ship-old,gist-snd-dvotvinit-n-ship-old,inknutantiqua-medium,gist-mai-dvotdhruv-n-ship-old,notosansdevanagari medium,gist-nepotdhruv-n-ship-old
9 | 3
Manipuri | sakalbharatidv-normal,gist-mni-bnotarbindo-n-ship-nonunicode,nirmala–old | 3 | 0
Bodo |  | gist-snd-dvotdhruv-n-ship-old,gist-snd-dvotvinit-n-ship-old,notosansdevanagari medium,inknutantiqua-semibold,gist-mai-dvotdhruv-n-ship-old,
---
inknutantiqua-medium,gist-doi-dvotvinit-n-ship-old,gist-kok-dvotdhruv-n-ship-
old,notosansdevanagari semibold,sakalbharatidv-normal,gist-nepotdhruv
-n-ship-old,notosansdevanagari black
9 | 3
Santali | gist-mni-bnotarbindo-n-ship-nonunicode,notosansolchiki regular,sakalbharatidv-normal,nirmala-old | 2 | 2
Assamese |  | balooda2-bold-new-unicode,banikanta-new-unicode,balooda2-regular-new-unicode,notosansbengali variablefont,likhannormal-old-unicode,
---
truetypewriterpolyglott mela old,atma-regular-new-unicode,atma-bold-new-
unicode,charukola bl3l old,hindsiliguri-bold-new-unicode,mitra-old
-unicode,optimamodoki rgabo old,muktinarrow-old-unicode,hindsiliguri-regular-new-unicode,faulmannfont lg7mw old,jamrulnormal-old-unicode,
ani-old-unicode,lohit bengali new,akaashnormal-old-unicode
13 | 6

##  2 Importance of Synthetic Data

The scope of synthetic data comes with its power of generating features to
meet specific needs or conditions which otherwise would not be available in
real-world data. When there is not enough of data during testing or when
privacy is considered your utmost priority, synthetic data comes to the
rescue.

  1. 1.

In the medical and healthcare sector, synthetic data is used for testing
certain medical conditions or cases for which real data is not available.

  2. 2.

ML-based Uber and Google’s self-driving cars are trained with the use of
synthetic data.

  3. 3.

In the financial sector, fraud detection and protection are very critical. New
fraudulent cases can be examined with the help of synthetic data.

  4. 4.

Synthetic data enables data professionals to access the use of centrally
recorded data while still maintaining the confidentiality of the data.
Synthetic data comes with the power to replicate the important features of
real data without exposing the true sense of it, thereby keeping privacy
intact.

  5. 5.

In the research department, synthetic data helps you develop and deliver
innovative products for which necessary data otherwise might not be available.

  6. 6.

In OCR synthetic data is used for training and fine tuning different OCR
models.

##  3 Approach

###  3.1 Synthetic Data Generation

Our synthetic data generation tool renders images at the line level. It has
many abilities such as to limit the number of words, identify the source
language to support the specific language and remove if found any other
languages and break a long text or document into multiple lines.

A list of lines are given through the text corpus and the collected text
corpus may be flawed, which can include other languages when it is supposed to
be a specific language. In this case, our synthetic generation tool is capable
of detecting the source language and resume the process after having made sure
of the specific language that is needed.

More variance is included in the text, with respect to the number of words,
space between the words, support of other languages and special characters.

Text can be long and it will have lots of words or sentences which will make
the image less realistic as the goal is to generate sentences. List of data
required for generation of synthetic data:

  * •

Text corpus

  * •

Fonts

  * •

Background image

  * •

Noise filters

  * •

Text color codes

These are the key files that help generate synthetic data. The power of
generating features to meet specific needs and conditions that otherwise would
not be possible in real world data and that power comes with the synthetic
data generation. Synthetic data generation helps in testing certain conditions
and cases for which real data does not exist. Synthetic data also can
replicate the important features of real world data without exposing the true
sense of it. _Text Corpus Sample:_

###  3.2 Style Attributes

Style attributes are the features that are used to render our text into
synthetic images. This includes features such as font color, background color,
noise features and background images.

The key idea is to get the ground truth with the uniqueness of each text from
the text corpus and apply features with different variations that meet with
our expectations that help generate synthetic data. The different types of
fonts are collected specific to the language and tested to avoid redundancy.
While processing the text corpus, the text may contain other language texts,
in that case the text is being removed. Different types of background images
specific to the needs are collected.

For each language, up to 15 fonts are tested, and the fonts that render the
text correctly are chosen for synthetic data generation. Various types of
noise filters are added during the generation to replicate different types of
classes in quality.

Use of different text colors is also included to add more variations. The data
is purely synthetic and does not have anything to do with the original data.
For each image, a series of features are generated on a random basis from the
collected features that meet with our expectations. Then the series of these
features are mapped to the other features and the ground truth in order to
generate fully synthetic data.

Table 2: Language Wise Data Distribution

Languages |  | Class 1 | Class 2 | Class3 | Noise-Added-Real | Scene-Text | Languages |  | Class 1 | Class 2 | Class3 | Noise-Added-Real | Scene-Text
---|---|---|---|---|---|---|---|---|---|---|---|---|---
English | Synthetic | 365 | 1709 | 1220 | - | - | Gujarati | Synthetic | 489 | 2039 | 1504 | - | -
Real | 176 | 86 | 87 | 350 | 38 | Real | 344 | 79 | 413 | 836 | 26
Hindi | Synthetic | 531 | 2076 | 1525 | - | - | Sanskrit | Synthetic | 317 | 1272 | 990 | - | -
Real | 488 | 127 | 301 | 916 | 42 | Real | 0 | 0 | 0 | 0 | 0
Kannada | Synthetic | 258 | 1460 | 1296 | - | - | Odiya | Synthetic | 240 | 1072 | 778 | - | -
Real | 206 | 342 | 0 | 549 | 28 | Real | 448 | 112 | 5 | 0 | 0
Tamil | Synthetic | 358 | 1425 | 1328 | - | - | Marathi | Synthetic | 340 | 1507 | 1493 | - | -
Real | 240 | 796 | 181 | 1217 | 40 | Real | 478 | 255 | 0 | 734 | 0
Malayalam | Synthetic | 376 | 1335 | 1181 | - | - | Konkani | Synthetic | 314 | 1370 | 1038 | - | -
Real | 369 | 90 | 8 | 468 | 31 | Real | 233 | 33 | 8 | 274 | 0
Telugu | Synthetic | 419 | 1888 | 1519 | - | - | Sindhi | Synthetic | 379 | 1651 | 1272 | - | -
Real | 829 | 807 | 92 | 0 | 44 | Real | 0 | 0 | 0 | 0 | 0
Bengali | Synthetic | 475 | 1907 | 1421 | - | - | Maithali | Synthetic | 335 | 1453 | 1117 | - | -
Real | 224 | 549 | 81 | 855 | 32 | Real | 0 | 0 | 0 | 0 | 0
Santali | Synthetic | 380 | 1708 | 1268 | - | - | Assamese | Synthetic | 344 | 1003 | 1106 | - | -
Real | 0 | 0 | 0 | 0 | 0 | Real | 0 | 0 | 0 | 0 | 0
Urdu | Synthetic | 417 | 1819 | 1459 | - | - | Punjabi | Synthetic | 444 | 1914 | 1460 | - | -
Real | 0 | 0 | 0 | 0 | 0 | Real | 152 | 60 | 12 | 225 | 0
Nepali | Synthetic | 298 | 1376 | 1029 | - | - | Manipuri | Synthetic | 424 | 1987 | 1398 | - | -
Real | 430 | 64 | 63 | 558 | 0 | Real | 0 | 0 | 0 | 0 | 0
Kashmiri | Synthetic | 264 | 1268 | 937 | - | - | Bodo | Synthetic | 180 | 744 | 531 | - | -
Real | 0 | 0 | 0 | 0 | 0 | Real | 0 | 0 | 0 | 0 | 0
Dogri | Synthetic | 493 | 2301 | 1660 | - | - |  |  |  |  |  |  |
Real | 0 | 0 | 0 | 0 | 0 |  |  |  |  |  |  |

###  3.3 Class Attributes

Synthetic data is key in having the data diversity to represent the real world
and reduce the bias to solve issues. Because the synthetic data is
automatically labeled and can deliberately include rare but crucial cases.

Types of classes:

  * •

Class 1

  * •

Class 2

  * •

Class 3

Class is the data quality measure of the condition based on factors such as
accuracy, completeness, consistency, reliability and which represents the
types of various real world data.

As a first step towards determining the data quality levels, for each data,
the series of features that are generated on a random basis in order to map
with the other features and ground truth, these features and the mapping
information helps determine the quality of the data.

Class 1:

Data accuracy is a key attribute of class 1 type data of high quality.

This is the most basic OCR complexity, with clear text on a white background
and font variation.

The aspects such as completeness, consistency, clear visibility and
readability are considered in this type of classification.

The methodology in achieving this is by determining the features that are
generated on a random basis.

The feature mapping in which the features are aligning with respect to most of
the types that are included:

  * •

High quality

  * •

White background

  * •

No noise filter

  * •

No color text

  * •

Clear background

_Sample Images:_

Class 2:

The goal is to simulate high-quality scanned documents with variations in text
color, background color, and font. In classification of the type class 2,
accuracy may not be the key but still however, it should be easily readable
with very less of a noise added along with the color text and some less
complex background images. The features that are mostly mapped to generate the
type class 2 are listed below:

  * •

Average quality

  * •

color or White background

  * •

Noise filters with minimal changes

  * •

color text

  * •

Less noise backgrounds

_Sample Images:_

Class 3:

The goal is to simulate poor-quality scanned documents, complex backgrounds,
font variation, noise, and blurring.

In classification of the type class 3, in simplest terms, poor data quality of
inaccurate information.

It lacks accuracy, completeness, visibility, readability and quality.

The feature mapping in which the features are aligning with respect to most of
the types that are included:

  * •

Bad quality

  * •

White or color backgrounds

  * •

Heavy noise filters

  * •

color text

  * •

Complex backgrounds

_Sample Images:_

###  3.4 Font Attributes

It is a key challenge when it comes to identifying the fonts for each specific
language. We were able to gather various fonts specific to language.

One of the key challenges include in identifying the fonts as well as
validating the gathered fonts.

The gathered fonts are separated into new and old fonts, also as Unicode and
Non-Unicode fonts.

During validation, issues to look for are, whether it supports alphanumeric
and special characters and multi language support as well. _Unicode Fonts:_

_Non-Unicode Fonts:_

###  3.5 Noise Filter Attributes

The noise filters help represent the synthetically generated data to not have
uniform variance with respect to quality and pixels.

The real data of different quality such as good, average and bad. Our
synthetic data generation tool has the ability to add such noise to represent
the data in different quality.

_Sample Images:_

##  4 Scene-Text

Scene-Text images are taken from the natural scenes which are captured by the
camera. Scene-Text images includes complex background and different text
appearances. The text in natural scene images can be in different
languages,colour and fonts.
Scene-Text data is available in 8 languages that are English, Hindi, Tamil,
Telugu, Malayalam, Kannada, Bengali and Gujarati.
For a given image or a scene, the lines are localized by bounding boxes and
that region is cropped to an image. Using OCR the text present in the
collected images are extracted to a machine-readable format that we can edit,
search, and can use for further processing.
Most of the OCR models does not perform well on scene text data so benchmark
of the OCR model on scene text data would be a good parameter. _Sample
Images:_

##  5 Coverage of Characters Sets

The objective is to achieve the coverage of all characters and numbers while
or after generating the dataset. From the generated dataset, the missing
characters and the numbers were identified through analysis at font level. The
data has been generated across various fonts and under those each font the
existence of characters and numbers were identified through analysis and those
specific characters and numbers set were generated synthetically and added
back to the dataset in order to achieve character coverage for every font that
is included in the dataset for all the existing languages.

##  6 Dataset

To generate synthetic images, our tool uses seed text sentences drawn from
various domains such as legal, education, govt orders and old documents (
before the 1960s ). Some part of the data was downloaded from the Anuvaad
parallel corpus Ramesh et al. (2021) and also we were able to generate it from
the google vision api.

Google vision API was used for OCR. As we know google vision is also not
completely accurate so we manually corrected and extracted text Shen et al.
(2021). We collected documents from various domains like legal, NCERT books,
govt orders and old documents.

Text corpus can also be filtered further on the basis of language specific
characters like special character, digits, auxiliary and any other characters.
Some of those lines that contain those certain characters, with the help of
regular expressions, can be drawn.

For each image, a series of features are generated on a random basis from the
collected features that meet with our expectations.

For each and every character set, we can define percentage value so that we
can filter text corpus in only the required ratio.

Good to considered that it would be better to include a small portion of the
real data as well to bring in the balance by adding some of the hardest
features that are otherwise challenging to generate synthetically. By adding a
small portion of the real data along with the synthetic data, the real data
represents some of the key characteristics of the real world use, that would
also help the data training achieve better results.

Table 3: Language Wise Script Details

Language | Script |  |  |  |  | Language | Script
---|---|---|---|---|---|---|---
Bengali | Bangla, Devanagari, Indic, Brahmi |  |  |  |  | Marathi | Devanagari, Indic
Assamese | Bangla, Devanagari, Indic, Brahmi |  |  |  |  | Urdu | Arabic
Bodo | Devanagari, Indic |  |  |  |  | Kashmiri | Perso Arabic
Dogri | Devanagari, Indic, Dogra |  |  |  |  | Kannada | Kannada, Indic
English | Latin |  |  |  |  | Santali | Ol Chiki
Gujarati | Devanagari, Indic, Gujarati |  |  |  |  | Sanskrit | Devanagari, Indic
Konkani | Devanagari, Indic |  |  |  |  | Nepali | Devanagari
Tamil | Tamil, Brahmi, Indic |  |  |  |  | Oriya | Kalinga, Odia, Indic
Malayalam | Malayalam, Indic |  |  |  |  | Punjabi | Gurmukhi, Indic
Telugu | Telugu, Brahmi, Indic |  |  |  |  | Sindhi | Devanagari
Hindi | Devanagari, Indic |  |  |  |  | Manipuri | Devanagari
Maithili | Devanagari |  |  |  |  |  |

##  7 Text Corpus Source and License

The text corpus used for synthetic data generation is mostly made from the
sources While some are available under a Creative Commons license and some are
available under other public open data standard licenses.

  * •

https://github.com/project-anuvaad/anuvaad-parallel-corpus

  * •

https://legislative.gov.in/regional-language

  * •

https://ncert.nic.in/textbook.php

  * •

https://main.sci.gov.in/vernacular_judgment

  * •

https://archive.org/

##  8 Challenges

Synthetic data has strong roots in Artificial Intelligence with numerous
benefits but still has some challenges which need to be taken care of while
dealing with synthetic data. These are as follows: Difficulty in generating
synthetic data. Dealing with the number of inconsistencies encountered while
replicating the complexities from real data to synthetic data. The flexible
nature of synthetic data makes it biased in behavior while validation or
training. Validating it with synthetic test data might not be enough for users
and might be required to validate it with real data. There could be some
hidden follies on the performance of algorithms trained with simplified
representations of synthetic data which lately may pop out while dealing with
real data. Users in some cases may not accept synthetic data to be a valid
data. Generating all necessary features from real data might become complex in
nature and also there can be a possibility of missing out on some necessary
features during this generation procedure.

##  9 Future Work

This work describes the first stages of our work in synthetic data generation.
In this we presented a dataset with approximately 2500 records of lines for
each 23 languages. The dataset mostly contains synthetically generated images,
a small portion of real world data and scene-text data in order to bring in
the balance by adding some of the hardest features that are otherwise
challenging to generate synthetically. Since the real data and scene-text data
represents some of the key characteristics of the real world use and as well
as will help in maintaining the data diversity, more of these data will be
added in future as currently it is not available for some of the low resource
languages. For each image a series of features such as noise filters, text
color, simple and complex backgrounds were added to meet with our
expectations. More such features will be added to closely represent the real
world data. We will investigate the synthetic image generation problem on a
large multilingual set of unconstrained document images and present a
comprehensive evaluation of the impact of synthetic data attributes on model
performance.

Finally we intend to extend our dataset with more vernacular languages and
handwritten synthetic data generation.

##  10 Conclusion

We presented the approach of synthetic data generation and impact of synthetic
data generation on OCR model performance and their benchmarking. In this
approach we generated image data for the challenging Indian languages.The
results show that with a relatively small number of fonts and text, a model
can be trained or fine-tuned that is competitive or outperforms a model
trained on real images.

## Acknowledgments

All the authors acknowledge the support of EkStep Foundation and Tarento
Technologies in presenting this work. We would like to thank Sumanth and
Gowtham from IIT Madras, who have helped us shape this paper. We would also
like to thank Vivek Raghavan for technical support and guidance.

## References

  * Bhattacharya et al. (2009) Ujjwal Bhattacharya, Swapan Kumar Parui, and Srikanta Mondal. 2009.  Devanagari and bangla text extraction from natural scene images.  _2009 10th International Conference on Document Analysis and Recognition_.
  * Culjak et al. (2012) Ivan Culjak, David Abram, Tomislav Pribanic, Hrvoje Dzapo, and Mario Cifrek. 2012\.  A brief introduction to opencv.  In _2012 Proceedings of the 35th International Convention MIPRO_ , pages 1725–1730.
  * Etter et al. (2019) David Etter, Stephen Rawls, Cameron Carpenter, and Gregory Sell. 2019.  A synthetic recipe for OCR.  In _2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019_ , pages 864–869. IEEE.
  * Mathew et al. (2017) Minesh Mathew, Mohit Jain, and C.V. Jawahar. 2017.  Benchmarking scene text recognition in devanagari, telugu and malayalam.  _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_.
  * Ramesh et al. (2021) Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2021\.  Samanantar: The largest publicly available parallel corpora collection for 11 indic languages.  _CoRR_ , abs/2104.05596.
  * Shen et al. (2021) Zejiang Shen, Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lee, Jacob Carlson, and Weining Li. 2021.  Layoutparser: A unified toolkit for deep learning based document image analysis.  In _16th International Conference on Document Analysis and Recognition, ICDAR 2021, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part I_ , volume 12821 of _Lecture Notes in Computer Science_ , pages 131–146. Springer.
  * Sporici et al. (2020) Dan Sporici, Elena Cusnir, and Costin-Anton Boiangiu. 2020.  Improving the accuracy of tesseract 4.0 OCR engine using convolution-based preprocessing.  _Symmetry_ , 12(5):715.