# Centralised or Decentralised? Data Analysis of Transaction Network of Hedera
Hashgraph

Lucas Amherd  Sheng-Nan Li L. Amherd<EMAIL_ADDRESS>is a member of the
Faculty of Business, Economics and Informatics at the University of Zurich,
Switzerland Claudio J. Tessone S.N. Li<EMAIL_ADDRESS>is a researcher at
Blockchain and Distributed Ledger Technologies group at the University of
Zurich and the UZH Blockchain CenterC. J. Tessone<EMAIL_ADDRESS>heads the Blockchain and Distributed Ledger Technologies group at the
University of Zurich. He is also co-founder and Chairman of the UZH Blockchain
Center.

###### Abstract

An important virtue of distributed ledger technologies is their acclaimed
higher level of decentralisation compared to traditional financial systems.
Empirical literature, however, suggests that many systems tend towards
centralisation as well. This study expands the current literature by offering
a first-time, data-driven analysis of the degree of decentralisation of the
platform Hedera Hashgraph, a public permissioned distributed ledger
technology, employing data directly fetched from a network node. The results
show a considerably higher amount of released supply compared to the release
schedule and a growing number of daily active accounts. Also, Hedera Hashgraph
exhibits a high centralisation of wealth and a shrinking core that acts as an
intermediary in transactions for the rest of the network. However, the
Nakamoto index and Theil index point to recent progress towards a more
decentralised network.

###### keywords:

Distributed Ledger Technology. Cryptocurrency. Hedera Hashgraph. Transaction
Network. Decentralisation.

##  1 Introduction

The beginning of a new era unfolded in 2008 when Satoshi Nakamoto, a group or
individual operating under a pseudonym, published the Bitcoin whitepaper,
which proposed a novel solution to the traditional centralised payment system
by resolving for the first time the double spending problem [1]. To prevent
users of digital currency from spending the same currency or token twice,
Nakamoto introduced a technology that had been in existence for some time: the
blockchain. It is a digital, append-only, and distributed ledger secured
through cryptography and modified using a consensus protocol. A distributed
ledger is a store of data spread across many geographical places and is
updated via a consensus algorithm. Numerous distributed ledger technology
(DLT) systems aim to achieve decentralised secured by cryptography, commonly
referred to as ”cryptocurrencies”, following Bitcoin’s example as a peer-to-
peer system.

Decentralisation is a crucial component of all distributed ledger
technologies, including blockchain and hashgraph technology [2, 3]. The exact
reason for their emergence is the increasingly centralised state of the more
and more digitalised economy which we trust, leading to fewer institutions
possessing more power. An example is Bitcoin, where the first block in its
history, the Genesis Block, contained the headline of a British newspaper:
_”The Times 03/Jan/2009 Chancellor on brink of second bailout for banks”_ [4].
This is often interpreted as a call for people to stop trusting centralized
banks and instead begin to rely on a more decentralized system.
Consequentially, decentralisation is seen as an integral virtue of distributed
ledger technology (DLT) systems. However, some literature has shown that these
systems tend towards centralisation as well [5, 6, 7, 8, 9, 10, 11, 12].
Therefore, it is crucial to differentiate between the common narrative and the
empirical evidence regarding the level of centralization in DLT systems.

Currently, there is a dearth of quality literature on Hedera Hashgraph
(subsequently also called ”the network”), particularly when it comes to its
decentralisation, which is not surprising given that the network launched only
in 2018 and has not yet achieved the size of the market capitalization of
major cryptocurrency platforms like Bitcoin and Ethereum [13]. In one paper,
Krasnoselskii et al.[14] proposed a random number generator solution
implemented on Hedera Hashgraph, while Jam et al.[15] described a charity
network partly implemented on Hedera Hashgraph. Previous works focus more on
the implementation, while this work seeks to contribute to the scientific
literature on the Hedera network.

Conceptually, decentralisation does not always have a standard definition;
Vitalik Buterin discusses architectural, political and logical
decentralisation [16]. Halaburda and Mueller-Bloch[17] and Karakostas,
Kiayias, and Ovezik[18] propose a systematic, multiple-layer approach to
evaluating decentralisation in DLTs. A different work by Howell, Saber, and
Bendechache[19] quantifies node decentralisation not only geographically but
also the number of validator nodes and their running software versions for
which they propose the NodeMaps framework. In another paper, while
highlighting the challenges of eliciting the level of decentralisation of a
DLT system, Lee et al.[20] calculate a decentralisation quotient of
geographical diversity and censorship resistance of the data. Complementary
work also comes from the social and economic perspective [21, 22]. Sai et al.
[23] based on an analysis of 89 research papers from which they propose a
centralisation taxonomy, while a different taxonomy offered by Zhang et al.
[24] is based on consensus, governance, network, transactions and wealth.
These few examples show that the concept of decentralisation in the context of
DLT systems can be defined very differently. Since the structure of a DLT
system, which consists of entities transacting among each other, can be easily
represented in the form of a network, this paper methodologically follows
chiefly Campajola et al.[7] employing various concepts from complex systems
literature such as degree distributions and the core-periphery structure to
determine the degree of decentralisation.

The purpose of this work is to analyse Hedera Hashgraph on its level of
decentralisation and to contribute first-time results from a proof-of-stake
(PoS) based distributed ledger technology to the literature by generalising
the results in existing literature[25, 26, 7, 27, 28, 11]. Another distinctive
feature this network exhibits consists of the hashgraph structure that
compares to the much more common blockchain technology. This paper also
provides new results in this subdomain of DLT systems. The research question
of this study is formulated as follows: _”What is the level of centralisation
exhibited by the Hedera Hashgraph token transaction network according to
different centralisation measures?”_

As the following structure of this paper: the protocol of Hedera Hashgraph
will be introduced briefly in section 2. In section 3 the data structure and
methods used to analyse Hedera Hashgraph are presented. The chapter section 4
shows a high-level analysis of the activity on Hedera Hashgraph including the
number of transactions over time or the rate of active accounts over time.
Moreover, results on various decentralisation measures such as the Gini
coefficient on wealth distribution or the core-periphery structure of the
network are presented in section 4. The discussion in section 5 concludes the
analysis.

##  2 Hedera Hashgraph

###  2.1 The Hashgraph Protocol

This paper focuses on the token transaction network of Hedera Hashgraph, a
distributed ledger technology controlled by a network of computers, not a
single entity. The network runs on the Hashgraph protocol invented by Leemon
Baird [29], which records transactions in a directed acyclic graph (DAG)
structure, rather than using blockchain, the more common distributed ledger
technology. The Hashgraph protocol is an asynchronous Byzantine Fault Tolerant
protocol (aBFT) that ensures the network’s functioning in the presence of
malicious nodes, including time-based attacks such as distributed denial of
service (DDoS) attacks. The entire transaction and communication history
between nodes - called the hashgraph - is stored by all the Hedera Council
Member nodes, currently a select set of diverse institutions, and broadcasted
to the network. Transactions in the network are spread by gossip, where each
node that received a transaction selects another node at random and
synchronizes its hashgraph with that node. An in-depth description of the
Hashgraph protocol is out of scope for this work and provided by Sridhar,
Blum, and Katz[30].

###  2.2 Permissioned and Permissionless

Hedera Hashgraph was launched in August 2018 as a public permissioned DLT
platform, whereas public refers to the fact that anyone can make transactions
and permissioned to the fact that nodes operating the network are selected
upon specific criteria. Hedera Hashgraph LLC. is currently governed by the
Hedera Governance Council (HGC) members [31]. HGC members are selected to be
as diverse as possible regarding the type of institution, the geographical
location and the industry in which they operate. Furthermore, they should be
large institutions with an international reputation to make the governance
more credible [32]. The HGC takes the role of several committees (Membership
committee, technical steering and product committee, corporate utilization
committee, coin economics committee and legal and regulatory committee) and
manages the treasury of the network.

When it comes to decentralisation, Hedera Hashgraph will take a methodical
approach consisting of three phases: In phase 1, the network is run and
transactions are verified only by consensus nodes operated by the members of
the HGC, a diverse set of institutions each having an equal influence on the
network. Then, phase 2 will allow more permissioned consensus nodes to secure
the network by verifying transactions without being part of the HGC. By the
time hundreds of permissioned consensus nodes run on the network and the HGC
has reached the maximum number of 39 members, Hedera Hashgraph will transition
to a _permissionless_ system, enabling anyone to run anonymously a consensus
node and earning rewards in the form of the native currency HBAR for securing
the network (for more detail see 4.3). A native currency is the base currency
of a DLT system, which is for instance used to pay transaction fees. This
approach will be further elaborated in section 4 and section 5.

##  3 Data and Methodology

This section provides a brief overview of the employed data and an
introduction to the metrics used in assessing the level of decentralisation in
the subsequent analyses in chapter 4.

###  3.1 Data Set

The analysis is based on data collected directly from the Hedera Hashgraph
network via its mirror node using the REST Application Programming Interface
(API). The REST API provides access to information on various ledger
variables, including accounts, transactions, balances, tokens or smart
contracts. Starting from 13 September 2019, we collect weekly balance data and
all transaction history until and including 28 February 2023, a total of 3
years, 5 months and 16 days. Despite the network having already launched on 24
August 2018, the mirror nodes only started storing a continuous hashgraph
history as of 13 September 2019. The tables and figures presented in chapter 4
depict the results of this time period.

Figure 1: Example of one transaction in the raw data format.

An exemplary structure of the transaction as it is fetched from the mirror
node is presented in figure 1. The variable ”consensus timestamp” marks the
time when consensus was reached on the transaction in Unix epoch format. The
variable used to identify the transaction type, such as HBAR transfer, token
transfer or account creation, is ”name”. Most importantly, the variable
”transfers” lists all accounts involved in the transaction, including the
respective change in their balance (”amount”).

The account-based model used in Hedera Hashgraph is a data accounting standard
where private-public key cryptography manages the access rights to the assets
in an account. To note, the wealth of accounts does not necessarily equal the
wealth of a user and vice versa since one user may have multiple accounts, and
one account may include the wealth of multiple users, it is challenging to
accurately identify the number of users. Nonetheless, in this paper, specific
accounts are labelled in the network explorers, and Hashscan, developed by
Swirlds Labs, is used to facilitate this identification process.

###  3.2 Gini Coefficient

The Gini coefficient is a simple way to gauge inequality of different kinds.
It is derived from the Lorenz curve which states what share of, for instance,
wealth is possessed by what share of the population. If there are $N$ entities
and $x_{i}$ is the wealth of entity $i$, then the Gini coefficient defined as:

$G=\frac{\sum_{i=1}^{N}{\sum_{j=1}^{N}{|x_{i}-x_{j}|}}}{2N\sum_{i=1}^{N}{x_{i}}}$
(1)

It is therefore in the interval of [0,1], one corresponding to complete
_inequality_ where one individual has all the wealth and all the others
possess nothing.

###  3.3 Degree Assortativity

In the section degree distribution 4.4, the node degree is defined as the
number of other nodes a focal node transacts with, and in the case of an
account-based model, a node is represented by an account. A high degree could
be translated to a high importance of a node in the network. Furthermore, the
degree assortativity is a measure to quantify the extent to which high-degree
nodes connect with other high-degree nodes (assortative) or low-degree nodes
(disassortative). The assortativity coefficient is normalised to [-1,1] where
a negative coefficient means that the network is disassortative.

When $r$ is the assortativity coefficient, it is given by,

$r=\frac{M^{-1}\sum_{i}^{M}j_{i}k_{i}-[M^{-1}\sum_{i}^{M}\dfrac{1}{2}(j_{i}+k_{i})]^{2}}{M^{-1}\sum_{i}^{M}\dfrac{1}{2}(j_{i}^{2}+k_{i}^{2})-[M^{-1}\sum_{i}^{M}\dfrac{1}{2}(j_{i}+k_{i})]^{2}},$
(2)

where $j_{i}$ and $k_{i}$ are the degrees of the nodes at the ends of the edge
$i$, and M is the total number of edges [33] .

###  3.4 Core-Periphery Structure

When it comes to economic networks or social network analysis in general, they
often exhibit a ”core-periphery structure” [34, 35, 36]. The idea is that the
actors can be partitioned into two distinct groups, so-called subgraphs. One
is the ”core”, in which all the actors are densely connected within each
other, and the ”periphery”, in which the actors are chiefly connected to the
core and only sparsely among each other. For the calculation this paper
follows the algorithm presented by Lip [35].

###  3.5 Nakamoto Index

The Nakamoto index version employed here, similar to Lin et al.[37],
quantifies the minimum number of entities that need to collude in order to
successfully conduct a Sybil attack, which allows them to control the network
[38]. The higher the Nakamoto index, the more resilient and secure a network
is. On Hedera Hashgraph, to run a successful Sybil attack, one-third of the
voting power is required, which is achieved by holding one-third of the
released HBAR supply, the network’s native coin, under the assumption that the
consensus is formed on a one-HBAR, one-vote basis [39]. It is expressed as:

$N_{c}=\frac{1}{n}min\\{k\in\left[1,2,...,n\right]:W^{-1}\sum_{i=1}^{k}{w_{i}}>s\\}$
(3)

where $w_{i}$ is the wealth of entity $i$ and there are again $n$ entities.
The entities are sorted by descending wealth, i.e. $w_{i}\leq w_{j}$ if $i\geq
j$. Two values are used in the results figure 8 for $s$, the supply share
required: $1/3$ and $1/2$.

###  3.6 Theil-T Index

Another statistic for decentralisation, the Theil-T index, can be borrowed
from the field of information theory. It is an entropy-based gauge chiefly
used for economic inequality and can be interpreted as the degree of non-
randomness or redundancy [40]. Mathematically, if $x_{i}$ is the wealth of
entity $i$ with a total of $N$ entities, then it is given by,

$T_{T}=\frac{1}{N}\sum_{i=1}^{N}{\frac{x_{i}}{\mu}\ln{\left(\frac{x_{i}}{\mu}\right)}}$
(4)

Thus, the Theil-T index measures in our case how far the network is from equal
distribution of wealth. A value of 0 corresponds to a completely egalitarian
state of the distribution, whereas the maximum entropy means one account has
all the wealth.

##  4 Results

This section starts with a general overview of the token transaction network
in the form of various measures, such as the number of transactions and the
daily active accounts over time, as well as the evolution of network supply,
which are derived from the analysis of transaction data. It continues by
providing gauges for decentralisation, namely the Gini coefficient of wealth,
the degree distribution, the core-periphery structure, the Nakamoto index and
the entropy.

###  4.1 Number of Transactions

The most basic way to measure network activity is the daily number of
transactions. In Figure 2, all transactions are shown along with
subcategories. Unsuccessful transactions only constitute a small and stable
fraction of all transactions, whereas the categories ”Treasury”, ”Hbar
Foundation” and ”Swirlds” (Swirlds Labs) are the three identifiable user
groups in this network.

Figure 2: Number of transactions over time in absolute numbers, categorized by
type. Figure 3: Evolution of daily number of transactions colour-coded with
the weekly HBAR returns.

As we can see in figure 2, the vast majority of transactions are from the
Hedera Consensus Service (HCS). Moreover, the jump in total transactions in
January 2023 by 140x within 3 days is worth to be mentioned. The spike
occurred from increased demand in HCS, introduced on the 10th of February
2020, and hence led to a decrease in the percentage of token and treasury
transactions. This figure also shows that the Treasury is involved in around
100 daily transactions, as opposed to the Hbar Foundation and Swirlds which
merely make occasional transactions 2. However, noticeable is the trend of the
increasing number of token transactions, i.e. non-HBAR transactions, starting
from the 14th of January 2021 when the Hedera Token Service was introduced on
the Hedera Hashgraph mainnet and reaching around 5% in mid-2022 where also the
number of token transactions plateaus (see Appendix A: figure 11).

Another interesting finding from figure 3 is that the value of the network’s
native token, HBAR, has a very weak correlation with network activity and is
highly correlated with the value of the dominant cryptocurrency Bitcoin
(Pearson correlation of 0.917 over the analysed time period, see Appendix A:
figure 12) which suggests that the price of HBAR does not reflect the
efficient price of the network at all points in time, i.e. is not evaluated
according to the network-specific information, but is rather traded along with
the market dominant asset Bitcoin.

###  4.2 Active Accounts

In figure 4, the subcategories ”Early users1M” and ”Early users1Y” refer to
accounts that were created (also called born) during the first month (1M) or
first year (1Y) of the analysed period, i.e. engaged in a transaction for the
first time in this time period. For instance, the subcategory ”Early users1M”
consists of all accounts that engaged for the first time in a transaction
between 13.09.2019 and 13.10.2019, and it is a subset of ”Early users1Y”. The
main insight from figure 4 is that the total number of daily active accounts
in the Hedera transaction network is steadily growing from a few hundred to a
few ten thousand; i.e. since its launch, it has increased by two order of
magnitude. Focusing on the early users, a periodical pattern can be observed
on a monthly basis around the 16th and 17th day of the month. In addition, the
bottom left figure in Figure 4 shows that the percentage of daily active
accounts among the total accounts has a decreasing trend and lies mostly below
1% in 2022, which is in stark contrast to the growing number of active
accounts, pointing at a relatively higher growth rate of total accounts in
bottom right figure of Figure 4.

Figure 4: Daily Active Accounts in absolute numbers (top), as a percentage of
the cumulative accounts (bottom left). Cumulative accounts over time (bottom
right).

###  4.3 Released Supply

The past distributed or released Hbar supply to the public is displayed in
figure 5 and exhibits a nearly linear increase over time. The goal of the
treasury is to achieve a wide distribution of the coins in order to ensure
network safety once it has transitioned to a permissionless model (as
mentioned in 1). Network safety is determined by the coin distribution, as the
amount of HBAR represents the proportional amount of voting power in its PoS
consensus mechanism.

Figure 5: Released supply on Hedera Hashgraph with maximum supply at 50
billion HBAR.

A contradiction occurs from their official release schedule where they claim
the released coins would not surpass 34% of the total supply until 2025,
however, this threshold was already passed in late November 2021 [39]. This
might be due to the fact that the coin distribution was more even than
expected so that further coins could be released without relinquishing much of
the wealth distribution or that, in a more pessimistic interpretation, the
coins needed to be sold to finance the continuing development of the network.
The converging trend of the released supply and the balance among all accounts
born after the first year shows the relatively strong flow of tokens to newly
created accounts.

###  4.4 Degree Distribution

As Campajola et al. [7] point out, many economic phenomena and financial
systems, e.g. the Italian interbank market [41], exhibit a strongly skewed
distribution, similar to a power-law Pareto type distribution ($\alpha$ around
2.3). Their statistics on the $\alpha$ exponent of a Pareto fit of the degree
distribution over several proof-of-work based (PoW) cryptocurrencies, such as
Bitcoin, Ethereum (pre-PoS-merge), Monacoin and Litecoin, are highly identical
to the results on Hedera Hashgraph (HBAR) and are reported in table 1. For
example, while the Hedera Hashgraph network has a median $\alpha$ of 2.11,
Campajola et al.[7] find values of 2.69 for Bitcoin (UTXO-based) and 1.76 for
Ethereum (Account-based). Their remark that the resemblance observed across
various cryptocurrency systems suggests that these transaction networks
possess a scale-free structure (many nodes with few connections and few
highly-connected nodes (hubs)) is supported by the findings on Hedera
Hashgraph. The findings on Hedera Hashgraph degree distributions also give a
new insight into degree distributions of PoS DLT systems, which yield no huge
difference to PoW systems. This finding has, of course, to be confirmed by
more evidence to give it more external validity.

|  | HBAR | BTC | MONA | ETH
---|---|---|---|---|---
|  Minimum |  1.51 |  1.74 |  1.49 |  1.50
|  Maximum |  4.22 |  10.99 |  7.69 |  3.42
$\alpha$ |  1\. Quartile |  1.94 |  2.42 |  2.46 |  1.68
|  3\. Quartile |  2.32 |  2.93 |  3.21 |  1.81
|  Median |  2.11 |  2.69 |  2.72 |  1.76
Table 1: Summary statistics on exponents of power-law fit of degree
distribution on the weekly transaction networks. Figure 6: Assortativity
coefficient for weekly transaction networks.

A negative coefficient is found in figure 6 for the entire period. The
disassortativity remains equal even if the accounts born in the first
month/year are removed. Since the theoretical assortativity of an Erdős–Rényi
(ER) random graph is zero, it can be stated that the weekly transaction
networks we analyse do not fit the ER network model [42]. Moreover, this
result supports the view that the platform function similarly to centralised
traditional financial markets, where many participants transact through a few
dominant intermediaries. In these digital token economies, the intermediaries
can take the form of exchanges, custodians or other service providers [7].

###  4.5 Core-Periphery Structure

Figure 7: Number of core nodes as a fraction of all nodes.

Here the algorithm suggested by Lip [35] is used for the bipartition of the
actors and the relative size of the core is then shown as a fraction of all
actors, in this case accounts, in figure 7. In the early days of the networks,
the core consists of a fraction of 5 - 10 % of all actors and then starts to
decline over time to a value lower than 1 % at the end of Feb. 2023. If the
early users (accounts whose birthday is in the first month or year of the
analysed period) are removed, the core size does not change significantly
compared to the core in the full network. Similar to what Campajola et al. [7]
have found about other crypto economies, such as Bitcoin and Ethereum, this
analysis finds a shrinking core over time, hinting at a more centralised
transaction network over time and few actors that play the role of
intermediaries, which makes this system not much different from the
traditional financial system where the tokens are moved by a few big players
that are sparsely connected to the majority of the network.

###  4.6 Nakamoto Index

Since the network is currently permissioned, i.e. only HGC members can run
validator nodes, figure 8 shows the hypothetical Nakamoto index for running a
successful Sybil attack if the network was _permissionless_ and one stake in
the balance represents one voting power. The 51%-attack, a Sybil attack where
51% of the voting power is required, is reported for comparability reasons
since this is a commonly used threshold for other cryptocurrencies. The weekly
Nakamoto indices are for almost three years since the launch of the network at
1, meaning one entity could have disrupted the network in its own interest,
pointing to a very high centralisation even if the consensus is restricted to
Council members. Only in September 2022 did it increase to more than 10
entities, starting an increasing trend. This compares to Nakamoto indices
between 5 and 500 for Bitcoin, 1 and 20 for Monacoin and between 2 and 16 for
Ethereum (all PoW-based) [7]. Overall, the results are striking suggesting
that the PoS-based Hedera Hashgraph is even less decentralised by this gauge.
Saad et al. [43] argue that this is no coincidence and is due to the consensus
mechanism itself.

Figure 8: Hypothetical weekly Nakamoto index under the condition that the
network was permissionless.

Hence, it might prove valuable that Hedera takes this methodical approach to
decentralisation. This permissioned approach is rationalised by Bakos,
Halaburda, and Mueller-Bloch [44] as they argue that it may not be intuitive
at first but permissioned platforms are able to guarantee some level of
decentralisation by binding many validators with off-blockchain contracts to
participate in securing the system, which is not possible for permissionless
systems where it is a result of free individual decisions.

###  4.7 Gini Coefficient

Figure 9: Gini coefficient of the balances - Accounts with a wealth of 10 HBAR
or lower, the transaction fee account plus the staking reward account are
excluded. In the right figure, the area of interest in y-axis is magnified.

The calculations in figure 9 show that the HBAR distribution on the network is
almost perfectly unequal over the whole history. The label ”Public” refers to
all accounts but treasury and Hbar foundation accounts. Since all accounts
with a wealth of 10 HBAR or less are excluded, the inequality is likely to
stem from a few wealthy accounts. This result suggests a high wealth
centralisation on Hedera Hashgraph and compares to very similar Gini
coefficients found for Bitcoin, Ethereum, Dogecoin and others [7, 10]. Another
paper that merely considers the richest addresses in these blockchains finds
lower values between 0.35 and 0.9 for Bitcoin, 0.45 and 0.65 for Ethereum and
0.55 and 0.85 for Dogecoin, though the small sample size of N=100 needs to be
highlighted and the influence it can have on those low values [27].

###  4.8 Theil-T Index

The weekly Theil-T index of HBAR holdings is shown in figure 10. The results
depict a weakly increasing inequality in the whole network until July 2021 and
a clear trend to more equality of the distribution of wealth afterwards. For
public accounts, there is a continuous growth in inequality until mid-2022.
After that, the entire network trends to be more equal. Overall, one can find
that there is a convergence of the Theil-T values of the different subgroups
over time.

Figure 10: Theil-T index of the balances.

##  5 Discussion

This paper started by highlighting the importance of decentralisation in DLT
systems, which can be seen as a reaction to the centralised state of the
current financial system. However, in contrast to the common narrative,
research literature finds that most of these DLT platforms exhibit a rather
centralised state [5, 7, 6]. This paper adds to the literature by offering a
first-time, data-driven analysis of the state of Hedera Hashgraph. This DLT
system differentiates in several ways from other systems. Mainly, the data is
not stored on the blockchain technology, but rather in a hashgraph.
Additionally, the network is currently _permissioned_ , i.e. only known
members of the Hedera Governance Council are allowed and required to run a
validator node, with the objective of transitioning to a _permissionless_
system once it is enough decentralised. Due to these fundamental differences,
the findings on the level of centralisation of Hedera Hashgraph provide a
valuable extension for the existing literature.

In the first part of the analysis, the results of the growth of the network
are discussed. On Hedera Hashgraph transactions employing the Hedera Consensus
Service made up the overwhelming fraction of transactions from mid-2020,
whereas transactions from the Hedera Token Service grew initially before
plateauing in mid-2022. Looking at the correlation of network activity and
HBAR returns, there is no correlation found, however, a strong positive
correlation between the Bitcoin price and HBAR price is found. Moreover, the
number of daily active accounts is growing faster over time, chiefly due to
new active accounts not being born in the first year of the analysed period of
the network, a sign of wider adoption of the network. Analysing the released
supply, it is much higher than what is outlined by Hedera in their planning;
for instance, 11.5 billion HBAR should have been released by the end of 2022
according to the documentation versus in reality 25 billion HBAR released
[39]. The difference might come from selling pressure to finance the
development of the platform and attract users, as it is unlikely that the
supply was released due to faster progress on the path to decentralisation
than scheduled.

In the second part of the results, various gauges for the level of
decentralisation are presented. We measure the Hedera decentralisation from
perspectives of wealth distribution (using Gini and Nakamoto index) and token
exchange (using degree assortativity and core-periphery structure). Striking
is the Gini coefficient of the account balances, which remains close to 1 for
the whole period, meaning that a very low number of accounts possess almost
the entire wealth. In addition, a negative node degree assortativity and a
shrinking core are found in the network, again, pointing to a centralised
state of the transaction network, similar to a traditional financial market
where many actors make transactions through only a few intermediaries such as
exchanges that represent a kind of ”interbank market”. In a permissionless
state of the network, Hedera Hashgraph has a Nakamoto index rapidly growing
from 1 to over 10 at the end of 2022, starting an increasing trend towards
more decentralisation. Similarly, the Theil-T index, a measure of economic
inequality, displays a continuous increase in wealth equality in the entire
network.

In conclusion, the analysis yields mixed signals on the level of
decentralisation of Hedera Hashgraph. Gauges such as the Gini index, degree
assortativity, and the core-periphery structure point towards a rather
centralised state, but other metrics like the daily active accounts, the
Nakamoto coefficient, and the Theil-T index yield progress towards
decentralisation. Therefore, whether the permissioned approach to
decentralisation on Hedera Hashgraph will lead to a better outcome in the
future, remains to be seen.

Moreover, Hedera Hashgraph is an (almost) undiscovered land when it comes to
research. Therefore, future research can develop in many different directions,
for instance using more rigid analysis such as statistic tests, regressions,
and machine learning. Building on this work, more granular network forensics
can be done, especially further considering the known entities that were
identified in this paper. Another extension towards a more comprehensive
conclusion on the level of decentralisation of Hedera Hashgraph may analyse
the staking mechanism and reward of the entities in the network. A third
option to investigate the network in more detail is to evaluate the
relationships between the accounts by creating an all-sided transaction
network, also employing more methodologies from complex network science and
knowledge about identifiable entities in order to extend the limited knowledge
of real-world users of this study.

## References

  * [1] Nakamoto, S. “Bitcoin: A Peer-to-Peer Electronic Cash System.” (2008) (accessed 26 April 2023) https://bitcoin.org/en/bitcoin-paper.
  * [2] Puthal, D., Malik, N., Mohanty, S. P., Kougianos, E., Yang, C. “The blockchain as a decentralized security framework [future directions].” _IEEE Consumer Electronics Magazine_ 7.2 18–21 (2018).
  * [3] Weyl, E. G., Ohlhaver, P., Buterin, V. “Decentralized society: Finding Web3’s soul.” _Available at SSRN 4105763_ .
  * [4] Elliott, F. “Chancellor Alistair Darling on brink of second bailout for banks.” (2009) (accessed 21 April 2023) https://www.thetimes.co.uk/article/chancellor-alistair-darling-on-brink-of-second-bailout-for-banks-n9l382mn62h.
  * [5] Beikverdi, A., Song, J. “Trend of centralization in Bitcoin’s distributed network.” In _2015 IEEE/ACIS 16th international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD)_ IEEE 1–6 (2015) .
  * [6] Gencer, A. E., Basu, S., Eyal, I., Van Renesse, R., Sirer, E. G. “Decentralization in bitcoin and ethereum networks.” In _Financial Cryptography and Data Security: 22nd International Conference, FC 2018, Nieuwpoort, Curaçao, February 26–March 2, 2018, Revised Selected Papers 22_ Springer 439–457 (2018) .
  * [7] Campajola, C., Cristodaro, R., De Collibus, F. M., Yan, T., Vallarano, N., Tessone, C. J. “The evolution of centralisation on cryptocurrency platforms.” _arXiv preprint arXiv:2206.05081_ .
  * [8] Budish, E. B. “The economic limits of Bitcoin and anonymous, decentralized trust on the blockchain.” _University of Chicago, Becker Friedman Institute for Economics Working Paper_ .83.
  * [9] Sultanik, E., _et al._ _Are Blockchains Decentralized? Unintended Centralities in Distributed Ledgers_. _Technical report_ Trail of Bits (2022).
  * [10] Gupta, M., Gupta, P. “Gini coefficient based wealth distribution in the bitcoin network: A case study.” In _Computing, Analytics and Networks: First International Conference, ICAN 2017, Chandigarh, India, October 27-28, 2017, Revised Selected Papers 1_ Springer 192–202 (2018) .
  * [11] Gochhayat, S. P., Shetty, S., Mukkamala, R., Foytik, P., Kamhoua, G. A., Njilla, L. “Measuring decentrality in blockchain based systems.” _IEEE Access_ 8 178372–178390 (2020).
  * [12] Cong, L. W., Tang, K., Wang, Y., Zhao, X. _Inclusion and democratization through web3 and defi? initial evidence from the ethereum ecosystem_. _Technical report_ National Bureau of Economic Research (2023).
  * [13] Buterin, V. “Ethereum whitepaper.” (2014) (accessed 28 April 2023) https://ethereum.org/en/whitepaper/.
  * [14] Krasnoselskii, M., Melnikov, G., Yanovich, Y. “Distributed Random Number Generator on Hedera Hashgraph.” In _the 3rd International Conference on Blockchain Technology and Applications_ 7–11 (2020) .
  * [15] Naser, J., Kalkan, K. “HungerHash: A Distributed Network for Child-Hunger Relief based on Hedera Hashgraph.” _Düzce Üniversitesi Bilim ve Teknoloji Dergisi_ 10.3 1408–1422 (2022).
  * [16] Buterin, V. “The meaning of decentralization.” _Medium. com_ 6.
  * [17] Halaburda, H., Mueller-Bloch, C. “Toward a multidimensional conceptualization of decentralization in blockchain governance: Commentary on “two sides of the same coin? decentralized versus proprietary blockchains and the performance of digital currencies” by cennamo, Marchesi, and Meyer.” _Academy of Management Discoveries_ 6.4 712–714 (2020).
  * [18] Karakostas, D., Kiayias, A., Ovezik, C. “SoK: A Stratified Approach to Blockchain Decentralization.” _arXiv preprint arXiv:2211.01291_ .
  * [19] Howell, A., Saber, T., Bendechache, M. “Measuring node decentralisation in blockchain peer to peer networks.” _Blockchain: Research and Applications_ 4.1 100109 (2023).
  * [20] Lee, J., Lee, B., Jung, J., Shim, H., Kim, H. “DQ: Two approaches to measure the degree of decentralization of blockchain.” _ICT Express_ 7.3 278–282 (2021).
  * [21] Bodó, B., Giannopoulou, A. “The logics of technology decentralization–the case of distributed ledger technologies.” In _Blockchain and Web 3.0_ Routledge 114–129 (2019).
  * [22] Bodó, B., Brekke, J. K., Hoepman, J.-H. “Decentralisation: A multidisciplinary perspective.” _Internet Policy Review_ 10.2 1–21 (2021).
  * [23] Sai, A. R., Buckley, J., Fitzgerald, B., Le Gear, A. “Taxonomy of centralization in public blockchain systems: A systematic literature review.” _Information Processing & Management_ 58.4 102584 (2021).
  * [24] Zhang, L., Ma, X., Liu, Y. “SoK: Blockchain Decentralization.” _arXiv preprint arXiv:2205.04256_ .
  * [25] Bovet, A., _et al._ “The evolving liaisons between the transaction networks of Bitcoin and its price dynamics.” _arXiv preprint arXiv:1907.03577_ .
  * [26] Makarov, I., Schoar, A. _Blockchain Analysis of the Bitcoin Market_. _Working Paper 29396_ National Bureau of Economic Research (2021) 10.3386/w29396 URL http://www.nber.org/papers/w29396.
  * [27] Kusmierz, B., Overko, R. “How centralized is decentralized? Comparison of wealth distribution in coins and tokens.” In _2022 IEEE International Conference on Omni-layer Intelligent Systems (COINS)_ IEEE 1–6 (2022) .
  * [28] Lin, Q., Li, C., Zhao, X., Chen, X. “Measuring decentralization in bitcoin and ethereum using multiple metrics and granularities.” In _2021 IEEE 37th International Conference on Data Engineering Workshops (ICDEW)_ IEEE 80–87 (2021) .
  * [29] Baird, L., Luykx, A. “The hashgraph protocol: Efficient asynchronous BFT for high-throughput distributed ledgers.” In _2020 International Conference on Omni-layer Intelligent Systems (COINS)_ IEEE 1–7 (2020) .
  * [30] Sridhar, V., Blum, E., Katz, J. “Musings on the HashGraph Protocol: Its Security and Its Limitations.” _arXiv preprint arXiv:2210.13682_ .
  * [31] Hedera Hashgraph LLC “UNDERSTANDING DECENTRALIZATION OF HEDERA HASHGRAPH Decentralization of consensus: The path to permissionless network nodes.” (2023) (accessed 21 April 2023) https://files.hedera.com/hh-decentralization-of-consensus.pdf.
  * [32] Hedera Hashgraph LLC “COUNCIL OVERVIEW.” (2023) (accessed 21 April 2023) https://hedera.com/council.
  * [33] Liang, J., Li, L., Zeng, D. “Evolutionary dynamics of cryptocurrency transaction networks: An empirical study.” _PloS one_ 13.8 e0202202 (2018).
  * [34] Barucca, P., Lillo, F. “Disentangling bipartite and core-periphery structure in financial networks.” _Chaos, Solitons & Fractals_ 88 244–253 (2016).
  * [35] Lip, S. Z. “A fast algorithm for the discrete core/periphery bipartitioning problem.” _arXiv preprint arXiv:1102.5511_ .
  * [36] Smith, D. A., White, D. R. “Structure and Dynamics of the Global Economy: Network Analysis of International Trade 1965-1980*.” _Social Forces_ 70 857–893 (1992).
  * [37] Lin, J.-H., Marchese, E., Tessone, C. J., Squartini, T. “The weighted Bitcoin lightning network.” _Chaos, Solitons & Fractals_ 164 112620 (2022).
  * [38] Srinivasan, B. S., Lee, L. “Quantifying decentralization.” _news.earn.com_ .
  * [39] Hedera Hashgraph LLC “Hbar Economics A deep dive into the dual role of hbars and detailed release schedule.” (2020) https://www.hedera.com/hh-hbar-coin-economics-paper-100919-v2.pdf.
  * [40] Theil, H. _Statistical decomposition analysis; with applications in the social and administrative sciences_. _Technical report_ (1972).
  * [41] Bargigli, L., Di Iasio, G., Infante, L., Lillo, F., Pierobon, F. “The multiplex structure of interbank networks.” _Quantitative Finance_ 15.4 673–691 (2015).
  * [42] Newman, M. E. “Assortative mixing in networks.” _Physical review letters_ 89.20 208701 (2002).
  * [43] Saad, M., Qin, Z., Ren, K., Nyang, D., Mohaisen, D. “e-pos: Making proof-of-stake decentralized and fair.” _IEEE Transactions on Parallel and Distributed Systems_ 32.8 1961–1973 (2021).
  * [44] Bakos, Y., Halaburda, H., Mueller-Bloch, C. “When permissioned blockchains deliver more decentralization than permissionless.” _Communications of the ACM_ 64.2 20–22 (2021).

##  Appendix A Complementing figures

Figure 11: Number of transactions over time as a percentage of all successful
transactions, categorized by type. Figure 12: Evolution of HBAR/USD price and
BTC/HBAR price over time.


# Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting:
A Study on the Mambai Language

###### Abstract

This study explores the use of large language models (LLMs) for translating
English into Mambai, a low-resource Austronesian language spoken in Timor-
Leste, with approximately 200,000 native speakers. Leveraging a novel corpus
derived from a Mambai language manual and additional sentences translated by a
native speaker, we examine the efficacy of few-shot LLM prompting for machine
translation (MT) in this low-resource context. Our methodology involves the
strategic selection of parallel sentences and dictionary entries for
prompting, aiming to enhance translation accuracy, using open-source and
proprietary LLMs (LlaMa 2 70b, Mixtral 8x7B, GPT-4). We find that including
dictionary entries in prompts and a mix of sentences retrieved through TF-IDF
and semantic embeddings significantly improves translation quality. However,
our findings reveal stark disparities in translation performance across test
sets, with BLEU scores reaching as high as 21.2 on materials from the language
manual, in contrast to a maximum of 4.4 on a test set provided by a native
speaker. These results underscore the importance of diverse and representative
corpora in assessing MT for low-resource languages. Our research provides
insights into few-shot LLM prompting for low-resource MT, and makes available
an initial corpus for the Mambai language.

Keywords: low-resource languages, austronesian language, large language
models, prompting, dictionary, parallel data

Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A
Study on the Mambai Language

Raphaël Merx${}^{\textrm{\textctc}}$ Aso Mahmudi${}^{\textrm{\textltailm}}$
Katrina Langford${}^{\textrm{\textturnt}}$
---
Leo Alberto de Araujo${}^{\textrm{\textesh}}$ Ekaterina
Vylomova${}^{\textrm{\textltailm}}$
${}^{\textrm{\textctc}}$Catalpa International ${}^{\textrm{\textltailm}}$The
University of Melbourne ${}^{\textrm{\textturnt}}$Timorlink
${}^{\textrm{\textesh}}$Seminario Menor Balide Dili
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Large language models (LLM) have shown remarkable abilities to perform natural
language processing (NLP) tasks they were not explicitly trained for,
including named entity recognition Mehta and Varma (2023), text classification
Sun et al. (2023), text summarisation Zhang et al. (2023b), and machine
translation (Hendy et al., 2023; Kocmi et al., 2023; Chowdhery et al., 2022,
MT). LLMs can be competitive with traditional encoder-decoder MT models for
high-resource languages, but lag behind traditional MT models when translating
to and from low-resource languages Robinson et al. (2023); Hendy et al.
(2023); Garcia et al. (2023).

While LLMs can achieve moderately high translation accuracy through zero-shot
prompting Wang et al. (2021), few-shot prompting can improve translation
accuracy Zhang et al. (2023a). Research on the selection of example sentences
for use in LLM prompts found that examples close to the source text do not
always result in better translation than random examples Vilar et al. (2023),
but that in-domain examples can improve accuracy for technical domains Agrawal
et al. (2023). In particular, for English to Kinyarwanda MT, Moslem et al.
(2023) finds an improvement of 11 ChrF points when using in-domain examples
instead of random ones.

Using domain adaptation as an analogy, in this paper we explore whether LLMs
can be prompted to translate _into_ a very low-resource language, through
careful selection of sentences and words close to the source text for use in
prompting. We work with the Mambai language, a primarily oral language from
Timor-Leste with around 200,000 native speakers Timor-Leste General
Directorate of Statistics (2015). We source prompt examples exclusively from
Hull (2001), a language manual which includes parallel English-Mambai
sentences and a bilingual word dictionary. We evaluate machine translation
quality on both a random subset of sentences from the manual, and on a small
corpus of translations collected from a native Mambai speaker.

We find that translation accuracy varies a lot depending on (1) the test set
used for evaluation, (2) LLM used for translation, and (3) examples included
in the prompt. While 10-shot translation yields BLEU score as high as 23.5 for
the test sentences sampled from the language manual used in prompting (with
GPT-4 and a mix of sentences retrieved through semantic embeddings and TF-IDF
in the prompt), BLEU drops below 5 across all experimental setups for test
sentences outside of this domain (novel sentences collected from a native
speaker).111We release the code for extracting the language manual data and
for using this data to construct a few-shot prompt given a sentence to
translate, as well as the corpus of sentences translated by the paper’s
author, in https://github.com/raphaelmerx/mambai. The language manual data is
available upon request.

Our findings highlight the risks of relying on a single source when evaluating
MT for low-resource languages, especially for languages like Mambai that do
not have a standardised vocabulary, orthography, or syntax, where a single
corpus can have substantial influence on NLP experiments, despite not always
being representative of the language’s variations.

## 2\. The Mambai Language

Timor-Leste (also known as East Timor) is a half-island nation in South-East
Asia, with a population of 1.3 million as of 2022 Timor-Leste General
Directorate of Statistics (2022). While its official languages are Portuguese
and Tetun Dili (Government of Timor-Leste, 2002, also spelled Tetum), the
country has over 30 indigenous languages, from both the Austronesian and
Papuan language families Kingsbury (2010).

Mambai (also spelled Mambae) is the country’s second most common mother tongue
after Tetun, with around 200,000 native speakers Timor-Leste General
Directorate of Statistics (2015). An Austronesian language, it is mostly
spoken in the Ermera, Aileu, Manufahi, and Ainaro municipalities Berlie
(2008), and does not have a standardised orthography (Hull, 2001). It has
three distinct varieties, and this article will focus on the southern variety,
spoken primarily in the Ainaro, Same, and Hatu-Builico administrative posts
Fogaça (2013).

Translating to Mambai can bring valuable material closer to Mambai-speaking
communities. For example, the Government of Timor-Leste has a mother tongue
education program named EMULI, which found that students who were taught in
their mother tongue have a higher level in reading comprehension and
mathematics than students taught in Portuguese. This program leverages
translated material for the curriculum Gusmão (2023); Walter (2016).

Unfortunately, in the taxonomy of Joshi et al. (2020), Mambai would be
assigned class 0, "The Left-Behinds", i.e. “languages that have been and are
still ignored in the aspect of language technologies”. A search for Mambai
sentences on OPUS Tiedemann (2009) returns only 36 sentences, all from
Tatoeba.222https://tatoeba.org/ To our knowledge, the only NLP tools that
claim to support Mambai are language identification models GlotLID Kargaran et
al. (2023) and MMS Pratap et al. (2023). Mambai does not appear on popular
datasets for low-resource languages such as MT560 (Gowda et al., 2021) or
FLORES-200 evaluation benchmark (Team et al., 2022).

Scan & OCR Word document Word files containing dictionaries Word file
containing parallel sentences Extract dictionaries in json format using regex
Extract Mambai and English sentences in separate files using font weight hints
Hunalign: align sentences Dictionaries for English-Mambai and Mambai-English
Parallel aligned English-Mambai sentences

Figure 1: Overview of our process for extracting dictionaries and a parallel
corpus from the Mambai Language Manual

## 3\. Methodology for Data Extraction

As the language does not have any resources in a machine-readable format, we
start by digitising the available materials. The general process of data
extraction is illustrated in Figure 1.

Figure 2: Mambai configuration in ABBYY FineReader 15.

### 3.1. Materials

Our primary data source is a Mambai Language Manual Hull (2001) that aims to
teach the basics of Mambai to foreign speakers, following the Ainaro variety.
This 109-page long document includes a pronunciation guide, a grammar, a
phrase book, and bilingual dictionaries (English-Mambai and Mambai-
English).333The author of this book gave his consent to us using it as
material, and we acknowledge him as the holder of copyright protecting this
intellectual property.

To test generalisation of our results, we collaborated with a native Mambai
speaker who translated a small corpus of 50 English sentences to Mambai. Since
Mambai has no formalised orthography, we tried to keep orthography close to
that used in the manual, however we did not aim to produce the same syntactic
structures as the manual.

### 3.2. OCR Process

For the Mambai Language Manual, which we received in paper format, we followed
the following OCR process:

  1. 1.

The book was scanned using an optical zoom camera, which reduces the radial
distortion effect and improves the OCR quality;

  2. 2.

The open-source ScanTailor software444https://scantailor.org/ was employed to
semi-automatically deskew images and make them flat black and white;

  3. 3.

In the proprietary software ABBYY FineReader 15,555https://pdf.abbyy.com/ we
set up a language alphabet, taking into account the characters utilised in
each book, with Indonesian (also an Austronesian language) serving as the
fallback language, as illustrated on Figure 2. The result of the OCR process
was saved in a Word document, preserving font formatting;

  4. 4.

We then manually separated the extracted data into three collections:

    1. (a)

the section of the manual that contains parallel sentences (14,347 words),

    2. (b)

the section that contains the English to Mambai dictionary (4,023 words),

    3. (c)

the section of the manual that contains the Mambai to English word dictionary
(4,522 words).

### 3.3. Text Corpora

In this subsection, we present the process of our corpus construction: using
the Word documents produced in Section 3.2, we create English-Mambai bilingual
dictionaries in JSON format and a corpus of parallel English-Mambai sentences.

#### 3.3.1. Dictionary extraction

For dictionary files, we mined triplets (entry, part of speech, translation)
through the following process:

  * •

using the python-docx library,666https://python-docx.readthedocs.io/ read the
file by preserving font weight, and identify text in bold as the dictionary
entry;

  * •

use a regular expression to match the part of speech, if any;

  * •

use the rest of the text as value corresponding to the entry;

  * •

if one entry had multiple translations, denormalise them by splitting with “;”
and “,”.

This process outputs dictionaries in JSON format, one for the English to
Mambai direction (1,790 entries), and one for the Mambai to English direction
(1,592 entries). Where present, each entry also contains part of speech
information, e.g.

    {
      ’entry’: ’beik’,
      ’translation’: ’silly’,
      ’part_of_speech’: ’adj.’
    }

#### 3.3.2. Parallel sentence extraction

Since no embedding models or MT systems support Mambai, we were precluded from
relying on sentence embeddings Thompson and Koehn (2019) or back-translations
Sennrich and Volk (2011) to mine parallel sentences from extracted documents.
Instead, we rely on a combination of Gale-Church sentence-length information
Gale and Church (1993) and lexical similarity through the
Hunalign777https://github.com/danielvarga/hunalign sentence aligner Varga et
al. (2007).

We identify Mambai sentences from their bold font-weight, English sentences
from their normal font-weight, and section delimiters through text in upper
case. For each section, we put the set of Mambai and English sentences in
separate text files, which are fed to Hunalign, along with the bilingual
dictionary extracted in Section 3.3.1. Hunalign outputs a series of tab-
delimited aligned sentence pairs, with an alignment score for each pair. After
manual review of a subset of 100 sentences, we find that setting a score
threshold of 0.2 corresponds to keeping a high number of well-aligned
sentences, while removing poorly aligned ones. After filtering out sentence
pairs below this threshold, we land on 1,187 parallel sentences extracted from
this phrase book, from a total of 1,275 potential bitexts.

Since sentences come from a language education manual, they tend to be
relatively short, with an average of 5.05 words per sentence in Mambai, and
5.66 words per sentence in English. Some sentences have alternative words in
parentheses, which we leave in place, e.g.:

    {
     ’Mambai’:
      "Balb ps masmidar lao xa (kaf).",
     ’English’:
      "Don’t put sugar in my tea (coffee)."
    }

## 4\. Mambai Translation through Retrieval-Augmented LLM Prompting

After all required data is ready, we now turn to the machine translation part.
The general process for translation is illustrated in Figure 3.

Example sentences from manual Input: English sentence from test set Dictionary
entries from manual Selection of example sentences where source is close to
input Filter dictionary entries for words that appear in input sentences
Prompt construction from example sentences and dictionary entries LLM Output:
translated sentence in Mambai

Figure 3: Overview of our process for translating English sentences to Mambai
using both dictionary entries and sentence pairs in few-shot LLM prompting.

### 4.1. Rationale

Adelani et al. (2022) found that a couple thousand high-quality sentences can
substantially increase low-resource MT performance, giving us hope that a
language manual with a similar order of magnitude of data could be enough to
produce moderate-quality translations.

Working with LLM prompting gives us a flexible format to incorporate both the
parallel sentence corpus and the dictionary entries. Further, having access to
a phrase book offers substantial domain coverage, in comparison with corpora
purely from the religious domain, which are often the only option for low-
resource languages Haddow et al. (2022); Walter (2016).

Here we work on English to Mambai translation, aiming to address the following
research questions:

  * •

Given an English sentence, how can a corpus of bilingual sentences, and a
bilingual word dictionary, be incorporated in an LLM prompt to maximise
translation accuracy?

  * •

Which LLMs (open-source or proprietary) show the best results for translating
into a low-resource language, and what is the observed variance between them?

  * •

How does translation accuracy vary across test sets?

### 4.2. Methodology

#### 4.2.1. Data setup

Our bilingual corpus of 1,187 parallel Mambai-English sentences is randomly
split into 119 (10%) sentences used for testing translation, and 1,068 (90%)
sentences for potential use in the prompt, after retrieval selection. Since
our objective is to translate full sentences, not individual words, all 1,790
words in the Mambai dictionary are used in prompting.

We also assess translation system quality by providing a different test corpus
of 50 sentences translated from English to Mambai by a native speaker of
Mambai. This small corpus has relatively simple but slightly longer sentences,
with 9 words per sentence on average. The English source sentences were
designed to cover a broad range of domains, such as daily life activities,
education, health and well-being, family relationships, religion, politics,
weather, employment, food and agriculture, technology, personal
characteristics, and Timor-Leste specific historical events.

By using the two test sets, we aim to evaluate robustness to variance between
domains, as well as estimate risks of overfitting that come from using a test
corpus that comes from the same material as the data for prompting. Expected
variance between test sets comes from their different authors, their different
years of publication (2001 vs 2024), and potentially by them covering
different domains.

#### 4.2.2. Prompt

We make use of the best performing prompt template from Peng et al. (2023), to
which we add dictionary entries for words found in the sentence, landing on
the following prompt template:

⬇

You are a translator for the Mambai language, originally from Timor-Leste.

# Example sentences

English: {Sent_eng_1}

Mambai: {Sent_mgm_1}

English: ...

Mambai: ...

# Dictionary entries

English: {Word_eng_1}

Mambai: {Word_mgm_1}

English: ...

Mambai: ...

Please provide the translation for the following sentence. Do not provide any
explanations or text apart from the translation.

English: {input}

Mambai:

Model | $N_{\text{TFIDF}}$ | $N_{\text{embed}}$ | UseDict | BLEU | ChrF | ChrF++
---|---|---|---|---|---|---
gpt-4-turbo | 0 | 0 | FALSE | 3.7 | 22.4 | 19.9
gpt-4-turbo | 0 | 0 | TRUE | 6.9 | 25.3 | 24.7
gpt-4-turbo | 10 | 0 | FALSE | 16.1 | 40.3 | 39.7
gpt-4-turbo | 10 | 0 | TRUE | 20.9 | 41.8 | 41.6
gpt-4-turbo | 0 | 10 | FALSE | 16.8 | 38.2 | 37.4
gpt-4-turbo | 0 | 10 | TRUE | 18.3 | 39.6 | 39.5
gpt-4-turbo | 5 | 5 | FALSE | 17.7 | 40.4 | 39.6
gpt-4-turbo | 5 | 5 | TRUE | 21.2 | 41.8 | 41.6
Mixtral 8x7B | 5 | 5 | TRUE | 9.0 | 30.9 | 30.4
LlaMa 70b | 5 | 5 | TRUE | 12.3 | 32.3 | 31.8
Table 1: Experiment results for test set from the language manual.
$N_{\text{TFIDF}}$ and $N_{\text{embed}}$ represent the number of sentence
pairs retrieved through TF-IDF and semantic embeddings, respectively. UseDict
indicates whether dictionary entries are included in the prompt. While
different hyperparameter combinations were tested for all models, we only
report on the best configuration for the less performant models (Mistral 8x7B
and LlaMa 70b).

#### 4.2.3. Models

We experiment with three models: Mixtral as it is the open-source model with
the highest MT-bench score Jiang et al. (2024), LlaMa 70b Touvron et al.
(2023) as it has a permissive license and has shown high zero-shot translation
performance Xu et al. (2024a), and GPT-4, which, despite being proprietary,
has very high zero-shot translation performance Xu et al. (2024a).

For each model, we experiment with the following setups:

  * •

UseDict (either True or False): For each word that appears in the source
language input (English), if this word is present in the English-Mambai
dictionary, we include its dictionary translation in the prompt;

  * •

$N_{\text{TFIDF}}$: Number of sentence pairs retrieved through TF-IDF, where
the English sentences are ranked according to TF-IDF similarity to the input.
The rationale here is that less frequent words can be harder to translate,
therefore should be surfaced in the prompt more often.
$N_{\text{TFIDF}}\in\\{0,5,10\\}$

  * •

$N_{\text{embed}}$: Number of sentence pairs retrieved through LASER semantic
embeddings Touvron et al. (2023), where the English sentences in training set
are first ranked using cosine similarity to the input.
$N_{\text{embed}}\in\\{0,5,10\\}$, similar to Zhang et al., 2023a; Vilar et
al., 2023; Hendy et al., 2023.

For each combination of the above features, we measure the BLEU and Chrf++
scores on both test sets, one from the language manual, and one manually
translated by a native speaker.

### 4.3. Translation Results

Our experiment results for test sentences from the manual are provided in
Table 1, and Table 2 provides the results for the test set collected from a
native speaker.

To summarise, we make the following observations:

(1) Translation accuracy varies widely between both test sets. While we get an
accuracy of up to 23.5 BLEU (41.9 ChrF++) for the test set that comes from the
language manual, we could not reach a BLEU higher than 4.4 (33.1 ChrF++) for
the test set from the native speaker. More analysis is needed to understand
this discrepancy, but it sends a strong signal about the risks of overfitting
by using a test set that comes from the same material as the examples used in
prompting. In particular, we think our result might partially invalidate
Tanzer et al. (2024), which similarly attempts to translate into a very low-
resource language using prompting from a single grammar book, but used
exclusively sentences from the grammar book in the test set.

(2) Dictionary entries help improve translation quality. When including
dictionary entries in the prompt, filtering on words that appear in the source
text, we found that translation quality improved significantly. This is true
across all experiments when keeping other hyperparameters constant, with an
average improvement of 3.25 BLEU points and 2.7 ChrF++ points.

(3) A blend of sentences retrieved through semantic embeddings and through TF-
IDF yields the highest translation accuracy. When working with a random split
of sentences from the language manual in particular, a blend of 5 sentences
retrieved through TF-IDF and 5 sentences retrieved through semantic embeddings
outperforms 10 sentences retrieved exclusively through one of these features.
This holds true for all three LLMs tested in this project.

(4) GPT-4 consistently outperforms other LLMs. GPT-4 yields both the highest
translation score overall, and the higher translation score for every single
experiment, when compared with LlaMa 70b and Mixtral 8x7B while keeping
$N_{\text{TFIDF}}$ and $N_{\text{embed}}$ constant.

Model | $N_{\text{TFIDF}}$ | $N_{\text{embed}}$ | UseDict | BLEU | ChrF | ChrF++
---|---|---|---|---|---|---
gpt-4-turbo | 0 | 0 | TRUE | 3 | 30.7 | 27.9
gpt-4-turbo | 0 | 0 | FALSE | 0 | 30.8 | 26.9
gpt-4-turbo | 10 | 0 | TRUE | 4 | 36.9 | 33.8
gpt-4-turbo | 10 | 0 | FALSE | 0 | 33.4 | 29.9
gpt-4-turbo | 0 | 10 | TRUE | 3.4 | 34.5 | 31.6
gpt-4-turbo | 0 | 10 | FALSE | 0 | 31.4 | 27.8
gpt-4-turbo | 5 | 5 | TRUE | 4.4 | 35.9 | 33
gpt-4-turbo | 5 | 5 | FALSE | 0 | 33.7 | 29.9
Mixtral 8x7B | 5 | 5 | TRUE | 3.5 | 26.8 | 24.6
LlaMa 70b | 5 | 5 | TRUE | 0 | 27.7 | 24.7
Table 2: Experiment results for the minicorpus of translations collected from
a native Mambai speaker. $N_{\text{TFIDF}}$ and $N_{\text{embed}}$ represent
the number of sentence pairs retrieved through TF-IDF and semantic embeddings,
respectively. UseDict indicates whether dictionary entries are included in the
prompt. While different hyperparameter combinations were tested for all
models, we only report on the best configuration for the less performant
models (Mistral 8x7B and LlaMa 70b).

### 4.4. Error analysis

We find that the large gap in performance across test sets is mostly due to
differences in translation output, rather than differences in the source
English text (Table 3):

  1. 1.

Using TF-IDF representations of English sentences, we computed the cosine
similarity in the whole training set and the two tests sets, resulting in
0.021 for the manual test set and 0.017 for the native speaker test set, a
relatively small difference. For the Mambai target reference, however, we get
a 0.027 and 0.012 for the manual and native speaker’s test sets, respectively,
a much larger difference.

  2. 2.

LASER Semantic similarity between each test set and the training set are
roughly equivalent at 0.42 and 0.40 for the manual and native speaker’s test
sets, respectively, on the English source side.

Similarity | Lang | Method | Score
---|---|---|---
ManualTest x Train | eng | TF-IDF | 0.021
NativeTest x Train | eng | TF-IDF | 0.017
ManualTest x Train | mgm | TF-IDF | 0.027
NativeTest x Train | mgm | TF-IDF | 0.012
ManualTest x Train | eng | Semantic | 0.42
NativeTest x Train | eng | Semantic | 0.40
Table 3: Similarity scores using TF-IDF cosine similarity and LASER semantic
cosine similarity between the two test sets and the training set for English
(source, eng) and Mambai (target, mgm) sentences.

Through manual review of the translation differences in both test sets, we
further identify the following potential causes for the large discrepancy in
translation quality metrics:

(1) Literal vs figurative translation: As sentences in the language manual are
made for learning, they tend to use more literal translations, which
correspond to what LLMs produce. On the other hand, our test set translated by
a native speaker often uses more idiosyncratic translation, further away from
words used in from the source input.

(2) Language variation: The Mambai language has changed since 2001, when the
Mambai Language Manual was published. In particular, we noted more usage of
Portuguese and Tetun Dili words in our test set reference sentences, which
might indicate that Mambai speakers mix more Tetun Dili and Portuguese in
their Mambai since the two languages were chosen as official in the 2002
Constitution Government of Timor-Leste (2002).

(3) Spelling: Despite trying to stay close to spelling used in the Mambai
Language Manual, we found that our test set at times uses different spelling
than the language manual (e.g. less hyphenation, some letters missing). This
reinforces our view that oral languages like Mambai are better covered by
speech datasets.

## 5\. Related Work

Traditionally, neural MT systems are trained on parallel corpora of aligned
sentence pairs Duong (2017). Low-resource languages tend to have orders of
magnitude less sentences available than higher-resource languages Arivazhagan
et al. (2019). To compensate for this lack of data, previous research found
that low-resource MT accuracy can be improved through leveraging multilingual
translation models that include better-resourced but related languages
Arivazhagan et al. (2019); Fan et al. (2020); Team et al. (2022). Other
techniques include pre-training on monolingual data Lample et al. (2018), the
incorporation of audio data that shares an embedding space with text data
Communication et al. (2023), and the generation of synthetic parallel
sentences Edunov et al. (2018), including by leveraging bilingual dictionaries
Duan et al. (2020).

In parallel, large language models have shown increased ability to translate,
at times surpassing specialised encoder-decoder MT systems Robinson et al.
(2023). Finding the right prompt recipe for increased MT accuracy using LLMs
has been a topic of research Zhang et al. (2023a); Li et al. (2022), with
findings that few-shot prompting often improves MT accuracy Zhang et al.
(2023a), and that the type of sentences used as few-shot examples can have a
large influence on accuracy Moslem et al. (2023). Dynamic adaptation of the
prompt by retrieving example sentences that are close to the input text Kumar
et al. (2023), or dictionary entries for words that appear in the source
Ghazvininejad et al. (2023) can further improve MT accuracy.

The applicability of common LLM prompting techniques when translating into
very low-resource languages is unclear, given these languages might not be
represented at all during LLM pretraining. Tanzer et al. (2024) partially
addresses this issue by focusing on MT between English and Kalamang, an
endangered Papuan language, using a single grammar book. Experimenting with
different models (Claude 2, LlaMa, gpt-3.5, gpt-4), and different prompt
setups (injecting sentences close to the input, dictionary entries, and the
grammar explanations found in the book), they achieve up to 45.8 ChrF on the
English to Kalamang direction. However, they work with a test set that is a
random subset of sentences found in the book, raising issues around the
applicability of their results to text translated by a different author, or to
domains not covered in the grammar book.

Recognising the potential of LLMs for MT, and the importance of in-context
examples used in prompting, our work experiments with retrieval-augmented LLM
prompting for translation into a low-resource language. We test translation
quality on both a subset of sentences coming from the language manual used as
corpus, and a test set specially translated by a native Mambai speaker for
this project.

## Conclusion

In this paper, we introduced a novel corpus for the Mambai language, a
language with around 200,000 native speakers that had virtually no NLP
resources. Our corpus includes bilingual dictionaries in both directions for
English-Mambai, a set of 1,187 parallel sentences from a language manual
published in 2001, and a set of 50 parallel sentences translated by a native
Mambai speaker. Our experiments on few-shot LLM prompting for English to
Mambai translation showed that moderate MT quality can be achieved for test
sentences very close to the original corpus, but MT quality decreases
significantly for sentences that come from a separate corpus, thus
highlighting the need for using test sets that do not come from the same
material as original examples used in prompting. We think LLMs offer a
flexible approach for integrating scarce resources in different formats
(dictionary entries, parallel sentences), and few-shot prompting shows
potential in improving low-resource MT using general purpose LLMs.

## Limitations

The sentences used in both training set (from the Mambai Language Manual) and
test sets tend to be rather short and simple, which raises questions around
translation quality for longer sentences, or for technical domains that get
little coverage in our corpus (e.g. health or legal text).

Mambai has no standard orthography. Even though the native Mambai speaker we
collaborated with tried to follow spelling close to that used in the language
manual, we expect that variances in spelling still negatively impacted the
test BLEU score. This stresses the need for heightened focus on audio for
primarily spoken languages like Mambai Chrupała (2023).

While we were able to gather a test set from a native Mambai speaker, they did
not evaluate translation quality for MT-translated text; instead we relied
solely on automated MT metrics. While BLEU tends to be a reliable measure of
MT quality for morphologically simple languages like Mambai Reiter (2018), we
would have preferred to dig deeper into the shortcomings of our LLM-generated
translations.

Lastly, Mambai has a simple grammar and morphology, which might make it
particularly prone to MT quality improvement using few-shot prompting.
Therefore, our results might not translate well on more morphologically
complex languages.

## Future Work

This work focused solely on Mambai, without leveraging resources from related
languages that have more resources, such as Tetun Dili, Portuguese, or
Indonesian. In future work, we would like to investigate the addition of Tetun
Dili sentences to the prompt, especially for domain-specific text that might
be very poorly covered by our small Mambai corpus, but that could be covered
by a larger Tetun Dili corpus.

In terms of finding the right recipe for prompting, future endeavours could
use a more systematic approach, similar to Kumar et al. (2023) which uses a
regression model for example selection. Additionally, more retrieval
techniques could be tested, e.g. bag of words, or even ChrF similarity between
the input and English source side.

In this paper, we used general purpose LLMs that likely saw little to no
Mambai text during pretraining. We think future work could experiment with
continuous pretraining on Mambai, or languages related to Mambai, before
prompting, similar to approaches in Xu et al. (2024b) and Alves et al. (2024).

## Acknowledgements

We thank Pr. Geoffrey Hull (Macquarie University), author of the Mambai
Language Manual, for authorising usage of his work as part of this research.
Pr. Hull remains the holder of copyright protecting this intellectual
property.

## References

##

  * Adelani et al. (2022) David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, and Sam Manthalu. 2022.  A few thousand translations go a long way! leveraging pre-trained models for African news translation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3053–3070, Seattle, United States. Association for Computational Linguistics.
  * Agrawal et al. (2023) Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023.  In-context examples selection for machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.
  * Alves et al. (2024) Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024\.  Tower: An open multilingual large language model for translation-related tasks.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019.  Massively multilingual neural machine translation in the wild: Findings and challenges.
  * Berlie (2008) Berlie. 2008.  Notes on east timor: Languages and education.  _Asian Journal of Social Science_ , 36(3-4):629–637.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Chrupała (2023) Grzegorz Chrupała. 2023.  Putting natural in natural language processing.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 7820–7827, Toronto, Canada. Association for Computational Linguistics.
  * Communication et al. (2023) Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023.  Seamlessm4t: Massively multilingual & multimodal machine translation.
  * Duan et al. (2020) Xiangyu Duan, Baijun Ji, Hao Jia, Min Tan, Min Zhang, Boxing Chen, Weihua Luo, and Yue Zhang. 2020.  Bilingual dictionary based neural machine translation without using parallel sentences.
  * Duong (2017) Long Duong. 2017.  Natural language processing for resource-poor languages.  _University of Melbourne_.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.
  * Fogaça (2013) Helem Andressa de Oliveira Fogaça. 2013.  _Estudo fonético e fonológico do Mambae de Same: uma língua de Timor-Leste_.  Ph.D. thesis, University of Brasilia.
  * Gale and Church (1993) William A. Gale and Kenneth W. Church. 1993.  A program for aligning sentences in bilingual corpora.  _Computational Linguistics_ , 19(1):75–102.
  * Garcia et al. (2023) Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. 2023.  The unreasonable effectiveness of few-shot learning for machine translation.
  * Ghazvininejad et al. (2023) Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023.  Dictionary-based phrase-level prompting of large language models for machine translation.
  * Government of Timor-Leste (2002) Government of Timor-Leste. 2002.  Constitution of the democratic republic of timor-leste.
  * Gusmão (2023) Kirsty Sword Gusmão. 2023.  The Key to Quality Inclusive Education in Timor-Leste’s Third Decade as an Independent Nation.  [Accessed 27-02-2024].
  * Haddow et al. (2022) Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helcl, and Alexandra Birch. 2022.  Survey of low-resource machine translation.  _Computational Linguistics_ , 48(3):673–732.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How good are gpt models at machine translation? a comprehensive evaluation.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024\.  Mixtral of experts.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kargaran et al. (2023) Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023.  GlotLID: Language identification for low-resource languages.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
  * Kingsbury (2010) Damien Kingsbury. 2010.  National identity in timor-leste: challenges and opportunities.  _South East Asia Research_ , 18(1):133–159.
  * Kocmi et al. (2023) Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. 2023.  Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 1–42, Singapore. Association for Computational Linguistics.
  * Kumar et al. (2023) Aswanth Kumar, Ratish Puduppully, Raj Dabre, and Anoop Kunchukuttan. 2023.  Ctqscorer: Combining multiple features for in-context example selection for machine translation.
  * Lample et al. (2018) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018\.  Unsupervised machine translation using monolingual corpora only.
  * Li et al. (2022) Yafu Li, Yongjing Yin, Jing Li, and Yue Zhang. 2022.  Prompt-driven neural machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2579–2590, Dublin, Ireland. Association for Computational Linguistics.
  * Mehta and Varma (2023) Rahul Mehta and Vasudeva Varma. 2023.  LLM-RM at SemEval-2023 task 2: Multilingual complex NER using XLM-RoBERTa.  In _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_ , pages 453–456, Toronto, Canada. Association for Computational Linguistics.
  * Moslem et al. (2023) Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. 2023.  Adaptive machine translation with large language models.
  * Peng et al. (2023) Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023.  Towards making the most of ChatGPT for machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5622–5633, Singapore. Association for Computational Linguistics.
  * Pratap et al. (2023) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023.  Scaling speech technology to 1,000+ languages.  _arXiv_.
  * Reiter (2018) Ehud Reiter. 2018.  A Structured Review of the Validity of BLEU.  _Computational Linguistics_ , 44(3):393–401.
  * Robinson et al. (2023) Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023.  ChatGPT MT: Competitive for high- (but not low-) resource languages.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 392–418, Singapore. Association for Computational Linguistics.
  * Sennrich and Volk (2011) Rico Sennrich and Martin Volk. 2011.  Iterative, MT-based sentence alignment of parallel texts.  In _Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011)_ , pages 175–182, Riga, Latvia. Northern European Association for Language Technology (NEALT).
  * Sun et al. (2023) Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023.  Text classification via large language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8990–9005, Singapore. Association for Computational Linguistics.
  * Tanzer et al. (2024) Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2024.  A benchmark for learning to translate a new language from one grammar book.  In _The Twelfth International Conference on Learning Representations_.
  * Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Thompson and Koehn (2019) Brian Thompson and Philipp Koehn. 2019.  Vecalign: Improved sentence alignment in linear time and space.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics.
  * Tiedemann (2009) Jörg Tiedemann. 2009.  _News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces_ , volume V, pages 237–248. John Benjamins, Amsterdam/Philadelphia.
  * Timor-Leste General Directorate of Statistics (2015) Timor-Leste General Directorate of Statistics. 2015.  2015 population and housing census.
  * Timor-Leste General Directorate of Statistics (2022) Timor-Leste General Directorate of Statistics. 2022.  2022 population and housing census.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Varga et al. (2007) Dániel Varga, Péter Halácsy, András Kornai, Viktor Nagy, László Németh, and Viktor Trón. 2007.  _Parallel corpora for medium density languages_, page 247–258. John Benjamins Publishing Company.
  * Vilar et al. (2023) David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023.  Prompting PaLM for translation: Assessing strategies and performance.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15406–15427, Toronto, Canada. Association for Computational Linguistics.
  * Walter (2016) Dr. Stephen L. Walter. 2016.  The embli endline evaluation study.
  * Wang et al. (2021) Shuo Wang, Zhaopeng Tu, Zhixing Tan, Wenxuan Wang, Maosong Sun, and Yang Liu. 2021\.  Language models are good translators.
  * Xu et al. (2024a) Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024a.  A paradigm shift in machine translation: Boosting translation performance of large language models.  In _The Twelfth International Conference on Learning Representations_.
  * Xu et al. (2024b) Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024b.  Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.
  * Zhang et al. (2023a) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a.  Prompting large language model for machine translation: A case study.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 41092–41110. PMLR.
  * Zhang et al. (2023b) Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023b.  SummIt: Iterative text summarization via ChatGPT.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10644–10657, Singapore. Association for Computational Linguistics.

## Language Resource References

##

  * Gowda et al. (2021) Gowda, Thamme and Zhang, Zhao and Mattmann, Chris and May, Jonathan. 2021.  _Many-to-English Machine Translation Tools, Data, and Pretrained Models_.  Association for Computational Linguistics.
  * Hull (2001) Hull, Geoffrey. 2001.  _Mambai Language Manual: Ainaro Dialect_.  Sebastião Aparício da Silva Project.
  * Team et al. (2022) NLLB Team and Marta R. Costa-jussà and James Cross and Onur Çelebi and Maha Elbayad and Kenneth Heafield and Kevin Heffernan and Elahe Kalbassi and Janice Lam and Daniel Licht and Jean Maillard and Anna Sun and Skyler Wang and Guillaume Wenzek and Al Youngblood and Bapi Akula and Loic Barrault and Gabriel Mejia Gonzalez and Prangthip Hansanti and John Hoffman and Semarley Jarrett and Kaushik Ram Sadagopan and Dirk Rowe and Shannon Spruit and Chau Tran and Pierre Andrews and Necip Fazil Ayan and Shruti Bhosale and Sergey Edunov and Angela Fan and Cynthia Gao and Vedanuj Goswami and Francisco Guzmán and Philipp Koehn and Alexandre Mourachko and Christophe Ropers and Safiyyah Saleem and Holger Schwenk and Jeff Wang. 2022.  _No Language Left Behind: Scaling Human-Centered Machine Translation_.