11institutetext: Department of Information and Communication Technologies,
Universitat Pompeu Fabra, Barcelona, Spain 22institutetext: CYBERCAT - Center
for Cybersecurity Research of Catalonia

# An Automated Market Maker Minimizing Loss-Versus-Rebalancing

Conor McMenamin  11 Vanesa Daza  1122 Bruno Mazorra  11

###### Abstract

The always-available liquidity of automated market makers (AMMs) has been one
of the most important catalysts in early cryptocurrency adoption. However, it
has become increasingly evident that AMMs in their current form are not viable
investment options for passive liquidity providers. This is large part due to
the cost incurred by AMMs providing stale prices to arbitrageurs against
external market prices, formalized as loss-versus-rebalancing (LVR) [Milionis
et al., 2022].

In this paper, we present Diamond, an automated market making protocol that
aligns the incentives of liquidity providers and block producers in the
protocol-level retention of LVR. In Diamond, block producers effectively
auction the right to capture any arbitrage that exists between the external
market price of a Diamond pool, and the price of the pool itself. The proceeds
of these auctions are shared by the Diamond pool and block producer in a way
that is proven to remain incentive compatible for the block producer. Given
the participation of competing arbitrageurs to capture LVR, LVR is minimized
in Diamond. We formally prove this result, and detail an implementation of
Diamond. We also provide comparative simulations of Diamond to relevant
benchmarks, further evidencing the LVR-protection capabilities of Diamond.
With this new protection, passive liquidity provision on blockchains can
become rationally viable, beckoning a new age for decentralized finance.

##  1 Introduction

CFMMs such as Uniswap [17] have emerged as the dominant class of AMM
protocols. CFMMs offer several key advantages for decentralized liquidity
provision. They are efficient computationally, have minimal storage needs,
matching computations can be done quickly, and liquidity providers can be
passive. Thus, CFMMs are uniquely suited to the severely computation- and
storage-constrained environment of blockchains.

Unfortunately, the benefits of CFMMs are not without significant costs. One of
these costs is definitively formalized in [14] as loss-versus-rebalancing
(LVR). It is proved that as the underlying price of a swap moves around in
real-time, the discrete-time progression of AMMs leave arbitrage opportunities
against the AMM. In centralized finance, market makers typically adjust to new
price information before trading. This comes at a considerable cost to AMMs
(for CFMMs, [14] derives the cost to be quadratic in realized moves), with
similar costs for AMMs derived quantitatively in [15, 6], and presented in
Figure 1.

Figure 1: Toxicity of Uniswap V3 Order Flow [16]. This graph aggregates the
PnL (toxicity) of all trades on the Uniswap V3 WETH/USDC pool, measuring PnL
of each order after 5 minutes, 1 hour, and 1 day. These are typical time
periods within which arbitrageurs close their positions against external
markets. This demonstrates the losses being incurred in existing state-of-the-
art DEX protocols are significant, consistent, and unsustainable; toxic.

These costs are being realized by liquidity providers in current AMM
protocols. All of these factors point towards unsatisfactory protocol design,
and a dire need for an LVR-resistant automated market maker. In this paper, we
provide Diamond, an AMM protocol which formally protects against LVR.

###  1.1 Our Contribution

We present Diamond, an AMM protocol which isolates the LVR being captured from
a Diamond liquidity pool, and forces some percentage of these LVR proceeds to
be returned to the pool. As in typical CFMMs, Diamond pools are defined with
respect to two tokens $x$ and $y$. At any given time, the pool has reserves of
$R_{x}$ and $R_{y}$ of both tokens, and some pool pricing function111See
Equation 1 for a full description of pool pricing functions as used in this
paper $\textit{PPF}(R_{x},R_{y})$. We demonstrate our results using the well-
studied Uniswap V2 pricing function of
$\textit{PPF}(R_{x},R_{y})=\frac{R_{x}}{R_{y}}$.

In Diamond, block producers are able to capture the block LVR of a Diamond
pool themselves or auction this right among a set of competing arbitrageurs.
In both cases, the block producer revenue approximates the arbitrage revenue.
Therefore, block producers are not treated as traditional arbitrageurs, but
rather as players with effective arbitrage capabilities due to their unique
position in blockchain protocols.

For each Diamond pool, we introduce the concept of its corresponding CFMM
pool. Given a Diamond pool with token reserves $(R_{x},R_{y})$ and pricing
function $\textit{PPF}(R_{x},R_{y})=\frac{R_{x}}{R_{y}}$, the corresponding
CFMM pool is the Uniswap V2 pool with reserves $(R_{x},R_{y})$. If a block
producer tries to move the price of the corresponding CFMM pool adding
$\Upsilon_{x}$ tokens and removing $\Upsilon_{y}$, the same price is achieved
in the Diamond pool by adding $(1-\beta)\Upsilon_{x}$ tokens for some
$\beta>0$, with $\beta$ the LVR rebate parameter. The block producer receives
$(1-\beta)\Upsilon_{y}$. In our framework, it can be seen that
$\textit{PPF}(R_{x}+(1-\beta)\Upsilon_{x},R_{y}-(1-\beta)\Upsilon_{y})<\textit{PPF}(R_{x}+\Upsilon_{x},R_{y}-\Upsilon_{y})$,
which also holds in our example, as
$\frac{R_{x}+(1-\beta)\Upsilon_{x}}{R_{y}-(1-\beta)\Upsilon_{y}}<\frac{R_{x}+\Upsilon_{x}}{R_{y}-\Upsilon_{y}}$.
A further $\upsilon_{y}$ tokens are removed from the Diamond pool to move the
reserves to the same price as the corresponding CFMM pool, with these tokens
added to a vault.

Half of the tokens in the vault are then periodically converted into the other
token (at any time, all tokens in the vault are of the same denomination) in
one of the following ways:

  1. 1.

An auction amongst arbitrageurs.

  2. 2.

Converted every block by the block producer at the final pool price. If the
block producer must buy $\frac{\eta}{2}$ tokens to convert the vault, the
block producer must simultaneously sell $\frac{\eta}{2}$ futures which
replicate the price of the token to the pool. These futures are then settled
periodically, either by

    1. (a)

Auctioning the $\frac{\eta}{2}$ tokens corresponding to the futures amongst
competing arbitrageurs with the protocol paying/collecting the difference.

    2. (b)

The use of a decentralized price oracle. In this paper, we consider the use of
the settlement price of an on-chain frequent batch auction, such as that of
[13], which is proven to settle at the external market price in expectancy.

Importantly, these auctions are not required for protocol liveness, and can be
arbitrarily slow to settle. We prove that all of these conversion processes
have 0 expectancy for the block producer or Diamond pool, and prove that the
LVR of a Diamond pool is $(1-\beta)$ of the corresponding CFMM pool. Our
implementation of Diamond isolates LVR arbitrageurs from normal users, using
the fact that arbitrageurs are always bidding to capture LVR. Specifically, if
an LVR opportunity exists at the start of the block, an arbitrageur will bid
for it in addition to ordering normal user transactions, meaning the proceeds
of a block producer are at least the realized LVR, with LVR corresponding to
the difference between the start- and end-states of an AMM in a given block.
This ensures the protections of Diamond can be provided in practice while
providing at least the same trading experience for normal users. Non-
arbitrageur orders in a Diamond pool can be performed identically to orders in
the corresponding CFMM pool after an arbitrageur has accepted to interact with
the pool through a special arbitrageur-only transaction. Although this means
user orders may remain exposed to the front-running, back-running and sandwich
attacks of corresponding CFMMs, the LVR retention of Diamond pools should
result in improved liquidity and reduced fees for users.

We discuss practical considerations for implementing Diamond, including
decreasing the LVR rebate parameter, potentially to 0, during periods of
protocol inactivity until transactions are processed, after which the
parameter should be reset. This ensures the protocol continues to process user
transactions, which becomes necessary when arbitrageurs are not actively
extracting LVR. If arbitrageurs are not arbitraging the pool for even small
LVR rebate parameters, it makes sense to allow transactions to be processed as
if no LVR was possible. In this case, Diamond pools perform identically to
corresponding CFMM pools. However, if/when arbitrageurs begin to compete for
LVR, we expect LVR rebate parameters to remain high.

We present a series of experiments in Section 7 which isolate the benefits of
Diamond. We compare a Diamond pool to its corresponding Uniswap V2 pool, as
well as the strategy of holding the starting reserves of both tokens,
demonstrating the power of Diamond. We isolate the effects of price
volatility, LVR rebate parameter, pool fees, and pool duration on a Diamond
pool. Our experiments provide convincing evidence that the relative value of a
Diamond pool to its corresponding Uniswap V2 pool is increasing in each of
these variables. These experiments further evidence the limitations of current
CFMMs, and the potential of Diamond.

###  1.2 Organization of the Paper

Section 2 analyzes previous work related to LVR in AMMs. Section 3 outlines
the terminology used in the paper. Section 4 introduces the Diamond protocol.
Section 5 proves the properties of Diamond. Section 6 describes how to
implement the Diamond protocol, and practical considerations which should be
made. Section 7 provides an analysis Diamond over multiple scenarios and
parameters, including a comparison to various reference strategies. We
conclude in Section 8.

##  2 Related Work

There are many papers on the theory and design of AMMs, with some of the most
important including [2, 1, 14, 3, 4]. The only peer-reviewed AMM design
claiming protection against LVR [12] is based on live price oracles. The AMM
must receive the price of a swap before users can interact with the pool. Such
sub-block time price data requires centralized sources which are prone to
manipulation, or require the active participation of AMM representatives, a
contradiction of the passive nature of AMMs and their liquidity providers. We
see this as an unsatisfactory dependency for DeFi protocols.

Attempts to provide LVR protection without explicit use of oracles either use
predictive fees for all players [8] and/or reduce liquidity for all players
through more complex constant functions [5]. Charging all users higher fees to
compensate for arbitrageur profits reduces the utility of the protocol for
genuine users, as does a generalized liquidity reduction. In Diamond, we only
reduce liquidity for arbitrageurs (which can also be seen as an increased
arbitrageur-specific fee), providing at least the same user experience for
typical users as existing AMMs without LVR protection.

A recent proposed solution to LVR published in a blog-post [10] termed MEV-
capturing AMMs (McAMMs) considers auctioning off the first transaction/series
of transaction in an AMM among arbitrageurs, with auction revenue paid in some
form to the protocol. Two important benefits of Diamond compared to the
proposed McAMMs are the capturing of realized LVR in Diamond as opposed to
predicted LVR in McAMMs, and decentralized access to Diamond compared to a
single point of failure in McAMMs.

In McAMMs, bidders are required to predict upcoming movements in the AMM.
Bidders with large orders to execute over the period (e.g. private price
information, private order flow, etc.) have informational advantages over
other bidders. Knowing the difference between expected LVR excluding this
private information vs. true expected LVR allows the bidder to inflict more
LVR on the AMM than is paid for. As this results in better execution for the
winner’s orders, this may result in more private order flow, which exacerbates
this effect. Diamond extracts a constant percentage of the true LVR,
regardless of private information. McAMMs also centralize (first) access
control to the winning bidder. If this bidder fails to respond or is censored,
user access to the protocol is prohibited/more expensive. Diamond is fully
decentralized, incentive compatible and can be programmed to effectively
remove LVR in expectancy. Future McAMM design improvements based on sub-block
time auctions are upper-bounded by the current protection provided by Diamond.

##  3 Preliminaries

This section introduces the key terminology and definitions needed to
understand LVR, the Diamond protocol, and the proceeding analysis. In this
work we are concerned with a single swap between token $x$ and token $y$. We
use $x$ and $y$ subscripts when referring to quantities of the respective
tokens. The external market price of a swap is denoted by $\varepsilon$, while
pool prices and price functions are denoted using a lowercase $p$ and
uppercase $P$ respectively. The price of a swap is quoted as the quantity of
token $x$ per token $y$.

In this work we treat the block producer and an arbitrageur paying for the
right to execute transactions in a block as the same entity. This is because
the the arbitrageur must have full block producer capabilities, and vice
versa, with the payoff for the block producer equal to that of an arbitrageur
under arbitrageur competition. For consistency, and to emphasize the arbitrage
that is taking place in extracting LVR, we predominantly use the arbitrageur
naming convention. That being said, it is important to remember that this
arbitrageur has exclusive access to building the sub-block of Diamond
transactions. Where necessary, we reiterate that it is the block producer who
control the per-block set of Diamond transactions, and as such, the state of
the Diamond protocol.

###  3.1 Constant Function Market Makers

A CFMM is characterized by reserves $(R_{x},R_{y})\in\mathbb{R}_{+}^{2}$ which
describes the total amount of each token in the pool. The price of the pool is
given by pool price function $PPF:\mathbb{R}^{2}_{+}\rightarrow\mathbb{R}$
taking as input pool reserves $(R_{x},R_{y})$. $PPF$ has the following
properties:

(a) $\displaystyle\ PPF\text{ is everywhere differentiable, with
}\frac{\partial PPF}{\partial R_{x}}>0,\ \frac{\partial PPF}{\partial
R_{y}}<0.$ (1) (b) $\displaystyle\ \lim_{R_{x}\rightarrow 0}PPF=0,\
\lim_{R_{x}\rightarrow\infty}PPF=\infty,\ \lim_{R_{y}\rightarrow
0}PPF=\infty,\ \lim_{R_{y}\rightarrow\infty}PPF=0.$ (c) $\displaystyle\
\text{If }\textit{PPF}(R_{x},R_{y})=p,\text{ then
}\textit{PPF}(R_{x}+cp,R_{y}+c)=p,\ \forall c>0.$

These are typical properties of price functions. Property (a) states the price
of $y$ is increasing in the number of $x$ tokens in the pool and decreasing in
the number of $y$ tokens. Property (b) can be interpreted as any pool price
value is reachable for a fixed $R_{x}$, by changing the reserves of $R_{y}$,
and vice versa. Property (c) states that adding reserves to a pool in a ratio
corresponding to the current price of the pool does not change the price of
the pool. These properties trivially hold for the Uniswap V2 price function of
$\frac{R_{x}}{R_{y}}$, and importantly allow us to generalize our results to a
wider class of CFMMs.

For a CFMM, the feasible set of reserves $C$ is described by:

$C=\\{(R_{x},R_{y})\in\mathbb{R}_{+}^{2}:\textit{PIF}(R_{x},R_{y})=k\\}$ (2)

where $\textit{PIF}:\mathbb{R}_{+}^{2}\rightarrow\mathbb{R}$ is the pool
invariant and $k\in\mathbb{R}$ is a constant. The pool is defined by a smart
contract which allows any player to move the pool reserves from the current
reserves $(R_{x,0},R_{y,0})\in C$ to any other reserves $(R_{x,1},R_{y,1})\in
C$ if and only if the player provides the difference
$(R_{x,1}-R_{x,0},R_{y,1}-R_{y,0})$.

Whenever an arbitrageur interacts with the pool, say at time $t$ with reserves
$(R_{x,t},R_{y,t})$, we assume as in [14] that the arbitrageur maximizes their
profits by exploiting the difference between $\textit{PPF}(R_{x,t},R_{y,t})$
and the external market price at time $t$, denoted $\varepsilon_{t}$. To
reason about this movement, we consider a pool value function
$V:\mathbb{R}_{+}\rightarrow\mathbb{R}$ defined by the optimization problem:

$V(\varepsilon_{t})=\displaystyle\min\limits_{(R_{x},R_{y})\in\mathbb{R}_{+}^{2}}\varepsilon_{t}R_{y}+R_{x},\text{
such that }\textit{PIF}(R_{x},R_{y})=k$ (3)

Given an arbitrageur interacts with the pool with external market price
$\varepsilon_{t}$, the arbitrageur moves the pool reserves to the
$(R_{x},R_{y})$ satisfying $V(\varepsilon_{t})$.

###  3.2 Loss-Versus-Rebalancing

LVR, and its prevention in AMMs is the primary focus of this paper. The
formalization of LVR [14] has helped to illuminate one of the main costs of
providing liquidity in CFMMs. The authors of [14] provide various synonyms to
conceptualize LVR. In this paper, we use the opportunity cost of arbitraging
the pool against the external market price of the swap, which is proven to be
equivalent to LVR in Corollary 1 of [14]. The LVR between two blocks $B_{t}$
and $B_{t+1}$ where the reserves of the AMM at the end of $B_{t}$ are
$(R_{x,t},R_{y,t})$ and the external market price when creating block
$B_{t+1}$ is $\varepsilon_{t+1}$ is:

$R_{x,t}+R_{y,t}\varepsilon_{t+1}-V(\varepsilon_{t+1})=(R_{x,t}-R_{x,t+1})+(R_{y,t}-R_{y,t+1})\varepsilon_{t+1}.$
(4)

As this is the amount being lost to arbitrageurs by the AMM, this is the
quantity that needs to be minimized in order to provide LVR protection. In
Diamond, this minimization is achieved.

###  3.3 Auctions

To reason about the incentive compatibility of parts of our protocol, we
outline some basic auction theory results.

First-price-sealed-bid-auction: There is a finite set of players $\mathcal{I}$
and a single object for sale. Each bidder $i\in\mathcal{I}$ assigns a value of
$X_{i}$ to the object. Each $X_{i}$ is a random variable that is independent
and identically distributed on some interval $[0,V_{max}]$. The bidders know
its realization $x_{i}$ of $X_{i}$. We will assume that bidders are risk
neutral, that they seek to maximize their expected payoff. Per auction, each
player submit a bid $b_{i}$ to the auctioneer. The player with the highest bid
gets the object and pays the amount bid. In case of tie, the winner of the
auction is chosen randomly. Therefore, the utility of a player
$i\in\mathcal{I}$ is

$u_{i}(b_{i},b_{-i})=\begin{cases}\frac{x_{i}-b_{i}}{m},\text{ if
}b_{i}=\text{max}_{i}\\{b_{i}\\},\\\ 0,\ \ \ \ \ \ \text{
otherwise}\end{cases}$

where $m=|\text{argmax}_{i}\\{b_{i}\\}|$. In our protocol, we have an amount
of tokens $z$ that will be auctioned. This object can be exchanged by all
players at the external market price $\varepsilon$. In this scenario, we have
the following lemma. Proofs are included in the Appendix

######  Lemma 1 ()

Let $\mathcal{I}$ be a set of players that can exchange at some market any
amount of tokens $x$ or $y$ at the external market price $\varepsilon$. If an
amount $z$ of token $y$ is auctioned in a first-price auction, then the
maximum bid of any Nash equilibrium is at least $z\varepsilon$.

##  4 Diamond

This section introduces the Diamond protocol. When the core protocol of
Section 4.2 is run, some amount of tokens are removed from the pool and placed
in a vault. These vault tokens are eventually re-added to the pool through a
conversion protocol. Sections 4.3 and 4.4 detail two conversion protocols
which can be run in conjunction with the core Diamond protocol. Which
conversion protocol to use depends on the priorities of the protocol users,
with a discussion of their trade-offs provided in Section 7, and represented
graphically in Figure 3. These trade-offs can be summarized as follows:

  * •

The process of Section 4.3 forces the arbitrageur to immediately re-add the
removed tokens to the Diamond pool, while ensuring the ratio of pool tokens
equals the external market price. This ratio is achieved by simultaneously
requiring the arbitrageur to engage in a futures contract tied to the pool
price, with the arbitrageur taking the opposite side of the contract. These
futures offset any incentive to manipulate the ratio of tokens. This results
in a higher variance of portfolio value for both the Diamond pool and the
arbitrageur. In return for this risk, this process ensures the pool liquidity
is strictly increasing in expectancy every block, with the excess value
(reduced LVR) retained by the vault immediately re-added to the pool. This
process can be used in conjunction with a decentralized price oracle to ensure
the only required participation of arbitrageurs is in arbitraging the pool
(see process 2 in Section 4.3). It should be noted that these futures
contracts have collateral requirements for the arbitrageur, which has
additional opportunity costs for the arbitrageur.

  * •

The process in Section 4.4 converts the vault tokens periodically. This can
result in a large vault balance accruing between conversions, with this value
taken from the pool. This means the quality (depth) of liquidity is decreasing
between conversions, increasing the impact of orders. From the AMM’s
perspective, this process incurs less variance in the total value of tokens
owned by the pool (see Figure 3), and involves a more straightforward and
well-studied use of an auction (compared to a trusted decentralized oracle).
There is also no collateral requirement for the arbitrageur outside of the
block in which the arbitrage occurs. 222As the arbitrageur and block producer
are interchangeable from Diamond’s perspective, we see the requirement for the
block producer/arbitrageur to provide collateral in a block controlled by the
block producer as having negligible cost.

Section 5 formalizes the properties of Diamond, culminating in Theorem 5.1,
which states that Diamond can be parameterized to reduce LVR arbitrarily close
to 0. It is important to note that Diamond is not a CFMM, but the rules for
adjusting pool reserves are dependent on a CFMM.

###  4.1 Model Assumptions

We outline here the assumptions used when reasoning about Diamond. In keeping
with the seminal analysis of [14], we borrow a subset of the assumptions
therein, providing here a somewhat more generalized model.

  1. 1.

External market prices follow a martingale process.

  2. 2.

The risk-free rate is 0.

  3. 3.

There exists a population of arbitrageurs able to frictionlessly trade at the
external market price, who continuously monitor and periodically interact with
AMM pools.

  4. 4.

An optimal solution $(R^{*}_{x},R^{*}_{y})$ to Equation 3 exists for every
external market price $\varepsilon\geq 0$.

The use of futures contracts in one version of the Diamond protocol makes the
risk-free rate an important consideration for implementations of Diamond. If
the risk free rate is not 0, the profit or loss related to owning token
futures vs. physical tokens must be considered. Analysis of a non-zero risk-
free rate is beyond the scope of the thesis.

###  4.2 Core Protocol

We now describe the core Diamond protocol, which is run by all Diamond
variations. A Diamond pool $\Phi$ is described by reserves $(R_{x},R_{y})$, a
pool pricing function $\textit{PPF}()$, a pool invariant function
$\textit{PIF}()$, an LVR rebate parameter $\beta\in(0,1)$, and conversion
frequency $\uptau\in\mathbb{N}$.

We define the corresponding CFMM pool of $\Phi$, denoted
$\textit{CFMM}(\Phi)$, as the CFMM pool with reserves $(R_{x},R_{y})$ whose
feasible set is described by pool invariant function $\textit{PIF}()$ and pool
constant $k=\textit{PIF}(R_{x},R_{y})$. Conversely, $\Phi$ is the
corresponding Diamond pool of $\textit{CFMM}(\Phi).$ It is important to note
that the mapping of $\Phi$ to $\textit{CFMM}(\Phi)$ is only used to describe
the state transitions of $\Phi$, with $\textit{CFMM}(\Phi)$ changing every
time the $\Phi$ pool reserves change.

Consider pool reserves $(R_{x,0},R_{y,0})$ in $\Phi$ at time $t=0$ (start of a
block), and an arbitrageur wishing to move the price of $\Phi$ at time $t=1$
(end of the block) to
$p_{1}=\frac{R_{x,1}}{R_{y,1}}\neq\frac{R_{x,0}}{R_{y,0}}$. In Diamond, to
interact with the pool at time $t=0$, the arbitrageur must deposit some amount
of collateral, $(C_{x},C_{y})\in\mathbb{R}_{+}^{2}$. This is termed the pool
unlock transaction. After the pool unlock transaction, the arbitrageur can
then execute arbitrarily many orders (on behalf of themselves or users)
against $\Phi$, exactly as the orders would be executed in
$\textit{CFMM}(\Phi)$, as long as for any intermediate reserve state
$(R_{x,i},R_{y,i})$ after an order, the following holds:

$C_{x}\geq\beta(R_{x,0}-R_{x,i})\text{ and }C_{y}\geq\beta(R_{y,0}-R_{y,i}).$
(5)

For end of block pool reserves $(R_{x,1},R_{y,1})$, WLOG let
$\Upsilon_{y}=R_{y,1}-R_{y,0}>0$, and $\Upsilon_{x}=R_{x,0}-R_{x,1}>0$ (the
executed orders net bought $x$ from $\Phi$, and net sold $y$ to $\Phi$). The
protocol then removes $\beta\Upsilon_{y}$ tokens from $\Phi$, sending them to
the arbitrageur, and adds $\beta\Upsilon_{x}$ tokens to $\Phi$, taking these
tokens from $C_{x}$. After this, it can be seen that
$\textit{PPF}(R_{x,1}+\beta\Upsilon_{x},R_{y,1}-\beta\Upsilon_{y})<p_{1}$. To
ensure the reserves correspond to a PPF equal to $p_{1}$, a further
$\upsilon_{x}>0$ tokens are removed such that:

$\textit{PPF}(R_{x,1}+\beta\Upsilon_{x}-\upsilon_{x},R_{y,1}-\beta\Upsilon_{y})=p_{1}.$
(6)

These $\upsilon_{x}$ tokens are added to the vault of $\Phi$. Summarizing the
transition from $t=0$ to $t=1$ from the arbitrageur’s perspective, this is
equivalent to:

  1. 1.

Adding $(1-\beta)\Upsilon_{y}$ tokens to $\Phi$ and removing
$(1-\beta)\Upsilon_{x}$ tokens from $\Phi$.

  2. 2.

Adding $\upsilon_{x}>0$ tokens to the $\Phi$ vault from the $\Phi$ pool such
that
$\textit{PPF}(R_{x,0}-(1-\beta)\Upsilon_{x}-\upsilon_{x},R_{y,0}+(1-\beta)\Upsilon_{y})=p_{1}.$
Note, this is with respect to starting reserves. 444If $\Upsilon_{y}>0$ tokens
are to be removed from $\textit{CFMM}(\Phi)$ with $\Upsilon_{x}>0$ tokens to
be added in order to achieve $p_{1}$, then $(1-\beta)\Upsilon_{y}$ tokens are
removed from $\Phi$ and $(1-\beta)\Upsilon_{x}$ tokens are added to $\Phi$,
with a further $\upsilon_{y}>0$ removed from $\Phi$ and added to the vault
such that
$\textit{PPF}(R_{x,0}+(1-\beta)\Upsilon_{x},R_{y,0}-(1-\beta)\Upsilon_{y}-\upsilon_{y})=p_{1}$.

If only a single arbitrageur order is executed on $\Phi$ apart from the pool
unlock transaction, the arbitrageur receives $\Upsilon_{x}$ tokens from the
order, and must repay $\beta\Upsilon_{x}$ as a result of the pool unlock
transaction. Any other sequence of orders resulting in a net $\Upsilon_{x}$
tokens being removed from $\Phi$ is possible, but $\beta\Upsilon_{x}$ tokens
must always be repaid to the pool by the arbitrageur. As the arbitrageur has
full control over which orders are executed, such sequences of orders must be
at least as profitable for the arbitrageur as the single arbitrage order
sequence. 555An example of such a sequence is an arbitrage order to the
external market price, followed by a sequence of order pairs, with each pair a
user order, followed by an arbitrageur order back to the external market
price. There are arbitrarily many other such sequences.

#### Vault Rebalance.

After the above process, let there be $(v_{x},v_{y})\in\mathbb{R}_{+}^{2}$
tokens in the vault of $\Phi$. If $v_{y}\varepsilon_{1}>v_{x}$, add
$(v_{x},\frac{v_{x}}{\varepsilon_{1}})$ tokens into $\Phi$ from the vault.
Otherwise, add $(v_{y}\varepsilon_{1},v_{y})$ tokens into $\Phi$ from the
vault. This is a vault rebalance.

Every $\uptau$ blocks, after the vault rebalance, the protocol converts half
of the tokens still in the vault of $\Phi$ (there can only be one token type
in the vault after a vault rebalance) into the other token in $\Phi$ according
to one of either conversion process 1 (Section 4.3) or 2 (Section 4.4). The
goal of the conversion processes is to add the Diamond vault tokens back into
the Diamond liquidity pool in a ratio corresponding to the $\varepsilon$,
while preserving the value of the tokens to be added to the pool.

To understand why half of the tokens are converted, assume WLOG that there are
$v_{x}$ tokens in the vault. Given an external market price $\varepsilon$,
$\frac{v_{x}}{2}$ tokens can be exchanged for
$v_{y}=\frac{v_{x}}{2}\frac{1}{\varepsilon}$ tokens, and vice versa. Both
conversion processes are constructed to ensure the expected revenue of
conversion is at least $v_{y}=\frac{v_{x}}{2}\frac{1}{\varepsilon}$.
Therefore, after conversion, there are at least $\frac{v_{x}}{2}$ and
$v_{y}=\frac{v_{x}}{2}\frac{1}{\varepsilon}$ tokens in the vault, with
$\frac{\frac{v_{x}}{2}}{v_{y}}=\varepsilon$. The conversion processes then add
the unconverted $\frac{v_{x}}{2}$ and converted $v_{y}$ tokens back into the
$\Phi$ pool, with the ratio of these tokens approximating the external market
price. Importantly, these tokens have value of at least the original vault
tokens $v_{x}$.

###  4.3 Per-block Conversion vs. Future Contracts

After every arbitrage, the arbitrageur converts $\eta$ equal to half of the
total tokens in the vault at the pool price $p_{c}$. Simultaneously, the
arbitrageur sells to the pool $\eta$ future contracts in the same token
denomination at price $p_{c}$. Given the pool buys $\eta$ future contracts at
conversion price $p_{c}$, and the futures settle at price $p_{T}$, the
protocol wins $\eta(p_{T}-p_{c})$.

These future contracts are settled every $\uptau$ blocks, with the net profit
or loss being paid in both tokens, such that for a protocol settlement profit
of $PnL$ measured in token $x$ and pool price $p_{T}$, the arbitrageur pays
$(s_{x},s_{y})$ with $PnL=s_{x}+s_{y}p_{T}$ and $s_{x}=s_{y}p_{T}$. These
contracts can be settled in one of the following (non-exhaustive) ways:

  1. 1.

Every $\uptau$ blocks, an auction takes place to buy the offered tokens from
the arbitrageurs who converted the pool at the prices at which the conversions
took place. For a particular offer, a positive bid implies the converter
lost/the pool won to the futures. In this case the converter gives the tokens
to the auction winner, while the pool receives the winning auction bid. A
negative bid implies the converter won/the pool lost to the futures. In this
case, the converter must also give the tokens to the auction winner, while the
pool must pay the absolute value of the winning bid to the auction winner.

  2. 2.

Every $\uptau$ blocks, a blockchain-based frequent batch auction takes place
in the swap corresponding to the pool swap. The settlement price of the
frequent batch auction is used as the price at which to settle the futures.

###  4.4 Periodic Conversion Auction

Every $\uptau$ blocks, $\eta$ equal to half of the tokens in the vault are
auctioned to all players in the system, with bids denominated in the other
pool token (bids for $x$ tokens in the vault must be placed in $y$ tokens, and
vice versa). For winning bid $b$ in token $x$ (or token $y$), the resultant
vault quantities described by $(s_{x}=b,s_{y}=\eta)$ (or
$(s_{x}=\eta,s_{y}=b)$) are added to the pool reserves. In this case, unlike
in Section 4.3, there are no restrictions placed on $\frac{s_{x}}{s_{y}}$.

##  5 Diamond Properties

This section outlines the key properties of Diamond. We first prove that both
conversion process have at least 0 expectancy for the protocol.

######  Lemma 2 ()

Converting the vault every block vs. future contracts has expectancy of at
least 0 for a Diamond pool.

######  Lemma 3 ()

A periodic conversion auction has expectancy of at least 0 for a Diamond pool.

###### Corollary 1

Conversion has expectancy of at least 0 for a Diamond pool.

With these results in hand, we now prove the main result of the paper. That
is, the LVR of a Diamond pool is $(1-\beta)$ of the corresponding CFMM pool.

######  Theorem 5.1 ()

For a CFMM pool $\textit{CFMM}(\Phi)$ with LVR of $L>0$, the LVR of $\Phi$,
the corresponding pool in Diamond, has expectancy of at most $(1-\beta)L$.

##  6 Implementation

We now detail an implementation of Diamond. The main focus of our
implementation is ensuring user experience in a Diamond pool is not degraded
compared to the corresponding CFMM pool. To this point, applying a
$\beta$-discount on every Diamond pool trade is not viable. To avoid this, we
only consider LVR on a per-block, and not a per-transaction basis. Given the
transaction sequence, in/exclusion and priority auction capabilities of block
producers, block producers can either capture the block LVR of a Diamond pool
themselves, or auction this right among arbitrageurs.

From an implementation standpoint, who captures the LVR is not important,
whether it is the producer themselves, or an arbitrageur who won an auction to
bundle the transactions for inclusion in Diamond. As mentioned already, we
assume these are the same entity, and as such it is the arbitrageur who must
repay the LVR of a block. To enforce this, for a Diamond pool, we check the
pool state in the first pool transaction each block and take escrow from the
arbitrageur. This escrow is be used in part to pay the realized LVR of the
block back to the pool. The first pool transaction also returns the collateral
of the previous arbitrageur, minus the realized LVR (computable from the
difference between the current pool state and the pool state at the beginning
of the previous block). To ensure the collateral covers the realized LVR, each
proceeding pool transaction verifies that the LVR implied by the pool state as
a result of the transaction can be repaid by the deposited collateral.

We can reduce these collateral restrictions by allowing the arbitrageur to
bundle transactions based on a coincidence-of-wants (CoWs) (matching buy and
sell orders, as is done in CoWSwap [7]). This can effectively reduce the
required collateral of the arbitrageur to 0. Given the assumed oversight
capabilities of arbitrageurs is the same as that of block producers, we do not
see collateral lock-up intra-block as a restriction, although solutions like
CoWs are viable alternatives.

Our implementation is based on the following two assumptions:

  1. 1.

An arbitrageur always sets the final state of a pool to the state which
maximizes the LVR.

  2. 2.

The block producer realizes net profits of at least the LVR corresponding to
the final state of the pool, either as the arbitrageur themselves, or by
auctioning the right to arbitrage amongst a set of competing arbitrageurs.

If the final price of the block is not the price maximizing LVR, the
arbitrageur has ignored an arbitrage opportunity. The arbitrageur can always
ignore non-arbitrageur transactions to realize the LVR, therefore, any
additional included transactions must result in greater or equal utility for
the arbitrageur than the LVR.

###  6.1 Core Protocol

The first transaction interacting with a Diamond pool $\Phi$ in every block is
the pool unlock transaction, which deposits some collateral,
$(C_{x},C_{y})\in\mathbb{R}_{+}^{2}$. Only one pool unlock transaction is
executed per pool per block. Every proceeding user order interacting with
$\Phi$ in the block first verifies that the implied pool move stays within the
bounds of Equation 5. Non pool-unlock transactions are executed as they would
be in the corresponding CFMM pool $\textit{CFMM}(\Phi)$ (without a $\beta$
discount on the amount of tokens that can be removed). These transactions are
executed at prices implied by pushing along the $\textit{CFMM}(\Phi)$ curve
from the previous state, and as such, the ordering of transactions intra-block
affects the execution price. If a Diamond transaction implies a move outside
of the collateral bounds, it is not executed.

The next time a pool unlock transaction is submitted (in a proceeding block),
given the final price of the preceding block was $p_{1}$, the actual amount of
token $x$ or $y$ required to be added to the pool and vault (the
$\beta\Upsilon$ and $\upsilon$ of the required token, as derived earlier in
the section) is taken from the deposited escrow, with the remainder returned
to the arbitrageur who deposited those tokens.

###### Remark 1

Setting the LVR rebate parameter too high can result in protocol censorship
and/or liveness issues as certain arbitrageurs may not be equipped to
frictionlessly arbitrage, and as such, repay the implied LVR to the protocol.
To counteract this, the LVR rebate parameter should be reduced every block in
which no transactions take place. As arbitrageurs are competing through the
block producers to extract LVR from the pool, the LVR rebate parameter will
eventually become low enough for block producers to include Diamond
transactions. After transactions have been executed, the LVR rebate parameter
should be reset to its initial value. Rigorous testing of initial values and
decay curves are required for any choice of rebate parameter.

###  6.2 Conversion Protocols

The described implementations in this section assume the existence of a
decentralized on-chain auction.666First-price sealed-bid auctions can be
implemented using a commit-reveal protocol. An example of such a protocol
involves bidders hashing bids, committing these to the blockchain along with
an over-collaterlization of the bid, with bids revealed when all bids have
been committed.

####  6.2.1 Per-block Conversion vs. Futures

Given per-block conversion (Section 4.3), further deposits from the
arbitrageur are required to cover the token requirements of the conversion and
collateralizing the futures. The conversions for a pool $\Phi$ resulting from
transactions in a block take place in the next block a pool unlock transaction
for $\Phi$ is called. Given a maximum expected percentage move over $\uptau$
blocks of $\sigma_{T}$, and a conversion of $\lambda_{y}$ tokens at price $p$,
the arbitrageur collateral must be in quantities $\pi_{x}$ and $\pi_{y}$ such
that if the arbitrageur is long the futures:

$1.\
\pi_{x}+\pi_{y}\frac{p}{1+\sigma_{T}}\geq\lambda_{y}(p-\frac{p}{1+\sigma_{T}}),\
\text{and}\ 2.\ \frac{\pi_{x}}{\pi_{y}}=\frac{p}{1+\sigma_{T}}.\ \ \ \ $ (7)

If the arbitrageur is short the futures it must be that:

$1.\ \pi_{x}+\pi_{y}p(1+\sigma_{T})\geq\lambda_{y}p\sigma_{T},\ \ \ \ \ \ \ \
\ \ \ \ \ \text{and}\ 2.\ \frac{\pi_{x}}{\pi_{y}}=p(1+\sigma_{T}).$ (8)

The first requirement in both statements is for the arbitrageur’s collateral
to be worth more than the maximum expected loss. The second requirement states
the collateral must be in the ratio of the pool for the maximum expected loss
(which also ensures it is in the ratio of the pool for any other loss less
than the maximum expected loss). This second requirement ensures the
collateral can be added back into the pool when the futures are settled.

At settlement, if the futures settle in-the-money for the arbitrageur, tokens
are removed from the pool in the ratio specified by the settlement price with
total value equal to the loss incurred by the pool, and paid to the
arbitrageur. If the futures settle out-of-the-money, tokens are added to the
pool from the arbitrageur’s collateral in the ratio specified by the
settlement price with total value equal to the loss incurred by the
arbitrageur. The remaining collateral is returned to the arbitrageur. The pool
constant is adjusted to reflect the new balances.

###### Remark 2

As converting the vault does not affect pool availability, the auctions for
converting the vault can be run sufficiently slowly so as to eliminate the
risk of block producer censorship of the auction. We choose to not remove
tokens from the pool to collateralize the futures as this reduces the
available liquidity within the pool, which we see as an unnecessary reduction
in benefit to users (which would likely translate to lower transaction fee
revenue for the pool). For high volatility token pairs, $\uptau$ should be
chosen sufficiently small so as to not to risk pool liquidation.

If Diamond with conversion versus futures is run on a blockchain where the
block producer is able to produce multiple blocks consecutively, this can have
an adverse effect on incentives. Every time the vault is converted and tokens
are re-added to the pool, the liquidity of the pool increases. A block
producer with control over multiple blocks can move the pool price some of the
way towards the maximal LVR price, convert the vault tokens (which has 0
expectancy from Lemma 2), increase the liquidity of the pool, then move the
pool towards the maximal LVR price again in the proceeding block. This process
results in a slight increase in value being extracted from the pool in
expectancy compared to moving the pool price immediately to the price
corresponding to maximal LVR. Although the effect on incentives is small, re-
adding tokens from a conversion slowly/keeping the pool constant unchanged
mitigates/removes this benefit for such block producers.

####  6.2.2 Periodic Conversion Auction

Every $\uptau$ blocks, $\eta$ equal to half the tokens in the vault are
auctioned off, with bids denominated in the other token. The winning bidder
receives these $\eta$ tokens. The winning bid, and the remaining $\eta$ tokens
in the vault, are re-added to the pool.

Figure 2:

Figure 3:

##  7 Experimental Analysis

This section presents the results of several experiments, which can be
reproduced using the following public repository [9]. The results provide
further evidence of the performance potential of a Diamond pool versus various
benchmarks. These experiments isolate the effect that different fees,
conversion frequencies, daily price moves, LVR rebate parameters, and days in
operation have on a Diamond pool. Each graph represents a series of random-
walk simulations which were run, unless otherwise stated, with base parameters
of:

  * •

LVR rebate parameter: $0.95$.

  * •

Average daily price move: $5\%$.

  * •

Conversion frequency: Once per day.

  * •

Blocks per day: $10$.

  * •

Days per simulation: $365$.

  * •

Number of simulations per variable: $500$.

#### Parameter Intuition.

For a Diamond pool to be deployed, we expect the existence of at least one
tradeable and liquid external market price. As such, many competing
arbitrageurs should exist, keeping the LVR parameter close to 1. $5\%$ is a
typical daily move for our chosen token pair. Given a daily move of $5\%$, the
number of blocks per day is not important, as the per block expected moves can
be adjusted given the daily expected move. Given a simulator constraint of
5,000 moves per simulation, we chose $10$ blocks per day for a year, as
opposed to simulating Ethereum over 5,000 blocks (less than 1 day’s worth of
blocks), as the benefits of Diamond are more visible over a year than a day.

Each graph plots the final value of the Diamond Periodic Conversion Auction
pool (unless otherwise stated) relative to the final value of the
corresponding Uniswap V2 pool. The starting reserve values are $\$100m$ USDC
and $76,336$ ETH, for an ETH price of $\$1,310$, the approximate price and
pool size of the Uniswap ETH/USDC pool at the time of simulation [17]. Figure
3 compares four strategies over the same random walks. Periodic Conversion
Auction and Conversion vs. Futures replicate the Diamond protocol given the
respective conversion strategies (see Section 4). HODL (Hold-On-for-Dear-
Life), measures the performance of holding the starting reserves until the end
of the simulation. The final pool value of these three strategies are then
taken as a fraction of the corresponding CFMM pool following that same random
walk. Immediately we can see all three of these strategies outperform the CFMM
strategy in all simulations (as a fraction of the CFMM pool value, all other
strategies are greater than 1), except at the initial price of 1310, where
HODL and CFMM are equal, as expected.

Figure 4:

Figure 5:

The Diamond pools outperform HODL in a range around the starting price, as
Diamond pools initially retain the tokens increasing in value (selling them
eventually), which performs better than HODL when the price reverts. HODL
performs better in tail scenarios as all other protocols consistently sell the
token increasing in value on these paths. Note Periodic Conversion slightly
outperforms Conversion vs. Futures when finishing close to the initial price,
while slightly underperforming at the tails. This is because of the futures
exposure. Although these futures have no expectancy for the protocol, they
increase the variance of the Conversion vs. Futures strategy, outperforming
when price changes have momentum, while underperforming when price changes
revert.

Figure 3 identifies a positive relationship between the volatility of the
price and the out-performance of the Diamond pool over its corresponding CFMM
pool. This is in line with the results of [14] where it is proved LVR grows
quadratically in volatility. Figure 5 demonstrates that, as expected, a higher
LVR rebate parameter $\beta$ retains more value for the Diamond pool.

Figure 5 shows that higher conversion frequency (1 day) has less variance for
the pool value (in this experiment once per day conversion has mean 1.011234
and standard deviation 0.000776 while once per week conversion has mean
1.011210 and standard deviation 0.002233). This highlights an important trade-
off for protocol deployment and LPs. Although lower variance corresponding to
more frequent conversion auctions is desirable, more frequent auctions may
centralize the players participating in the auctions due to technology
requirements. This would weaken the competition guarantees needed to ensure
that the auction settles at the true price in expectancy.

Figure 6:

Figure 7:

Figure 7 compares Diamond to the CFMM pool under the specified fee structures
(data-points corresponding to a particular fee apply the fee to both the
Uniswap pool and the Diamond pool) assuming $10\%$ of the total value locked
in each pool trades daily. The compounding effect of Diamond’s LVR rebates
with the fee income every block result in a significant out-performance of the
Diamond protocol as fees increase. This observation implies that given the LVR
protection provided by Diamond, protocol fees can be reduced significantly for
users, providing a further catalyst for a DeFi revival. Figure 7 demonstrates
that the longer Diamond is run, the greater the out-performance of the Diamond
pool versus its corresponding CFMM pool.

##  8 Conclusion

We present Diamond, an AMM protocol which provably protects against LVR. The
described implementation of Diamond stands as a generic template to address
LVR in any CFMM. The experimental results of Section 7 provide strong evidence
in support of the LVR protection of Diamond, complementing the formal results
of Section 5. It is likely that block producers will be required to charge
certain users more transaction fees to participate in Diamond pools to
compensate for this LVR rebate, with informed users being charged more for
block inclusion than uninformed users. As some or all of these proceeds are
paid to the pool with these proceeds coming from informed users, we see this
as a desirable outcome.

##  9 Acknowledgements

We thank the reviewers for their detailed and insightful reviews, as well as
Stefanos Leonardos for his guidance in preparing this camera-ready version.
This paper is part of a project that has received funding from the European
Union’s Horizon 2020 research and innovation programme under grant agreement
number 814284, and is supported by the AEI-PID2021-128521OB-I00 grant of the
Spanish Ministry of Science and Innovation.

## References

  * [1] Adams, H., Keefer, R., Salem, M., Zinsmeister, N., Robinson, D.: Uniswap V3 Core (2021), https://uniswap.org/whitepaper-v3.pdf
  * [2] Adams, H., Zinsmeister, N., Robinson, D.: Uniswap V2 Core (2020), https://uniswap.org/whitepaper.pdf
  * [3] Bartoletti, M., Chiang, J.H.y., Lluch-Lafuente, A.: A Theory of Automated Market Makers in DeFi. In: Damiani, F., Dardha, O. (eds.) Coordination Models and Languages. pp. 168–187. Springer International Publishing, Cham (2021)
  * [4] Bartoletti, M., Chiang, J.H.y., Lluch-Lafuente, A.: Maximizing Extractable Value from Automated Market Makers. In: Financial Cryptography and Data Security. Springer Berlin Heidelberg, Berlin, Heidelberg (2022)
  * [5] Bichuch, M., Feinstein, Z.: Axioms for Automated Market Makers: A Mathematical Framework in FinTech and Decentralized Finance. https://arxiv.org/abs/2210.01227 (2022), accessed: 10/02/2023
  * [6] Capponi, A., Jia, R.: The Adoption of Blockchain-based Decentralized Exchanges. https://arxiv.org/abs/2103.08842 (2021), accessed: 10/02/2023
  * [7] CoW Protocol: https://docs.cow.fi/, accessed: 11/10/2022
  * [8] Evans, A., Angeris, G., Chitra, T.: Optimal Fees for Geometric Mean Market Makers. In: Bernhard, M., Bracciali, A., Gudgeon, L., Haines, T., Klages-Mundt, A., Matsuo, S., Perez, D., Sala, M., Werner, S. (eds.) Financial Cryptography and Data Security. FC 2021 International Workshops. pp. 65–79. Springer Berlin Heidelberg, Berlin, Heidelberg (2021)
  * [9] Github: https://github.com/The-CTra1n/LVR (2022)
  * [10] Josojo: MEV capturing AMMs. https://ethresear.ch/t/mev-capturing-amm-mcamm/13336 (2022), accessed: 10/02/2023
  * [11] Krishna, V.: Auction theory. Academic press (2009)
  * [12] Krishnamachari, B., Feng, Q., Grippo, E.: Dynamic Automated Market Makers for Decentralized Cryptocurrency Exchange. In: 2021 IEEE International Conference on Blockchain and Cryptocurrency (ICBC). pp. 1–2 (2021). https://doi.org/10.1109/ICBC51069.2021.9461100
  * [13] McMenamin, C., Daza, V., Fitzi, M., O’Donoghue, P.: FairTraDEX: A Decentralised Exchange Preventing Value Extraction. In: Proceedings of the 2022 ACM CCS Workshop on Decentralized Finance and Security. p. 39–46. DeFi’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3560832.3563439, https://doi.org/10.1145/3560832.3563439
  * [14] Milionis, J., Moallemi, C.C., Roughgarden, T., Zhang, A.L.: Quantifying Loss in Automated Market Makers. In: Zhang, F., McCorry, P. (eds.) Proceedings of the 2022 ACM CCS Workshop on Decentralized Finance and Security. ACM (2022)
  * [15] Park, A.: The Conceptual Flaws of Constant Product Automated Market Making. ERN: Other Microeconomics: General Equilibrium & Disequilibrium Models of Financial Markets (2021)
  * [16] @thiccythot: https://dune.com/thiccythot/uniswap-markouts, accessed: 10/02/2023
  * [17] Uniswap: https://app.uniswap.org/, accessed: 11/10/2022

##  Appendix 0.A Proofs

See 1

###### Proof

By construction, we have that the support of $X_{i}$ is lower bounded by
$z\varepsilon$. Therefore, in a second-price auction, in equilibrium, each
player will bid at least, $z\varepsilon$. Using the revenue equivalence
theorem [11], we deduce that the revenue of the seller is at least
$z\varepsilon$ obtaining the result.

See 2

###### Proof

Consider a conversion of $\eta$ tokens which takes place at time 0. Let the
conversion be done at some price $p_{c}$, while the external market price is
$\varepsilon_{0}$. WLOG let the protocol be selling $\eta$ $y$ tokens in the
conversion, and as such, buying $\eta$ $y$ token futures at price $p_{c}$. The
token sells have expectancy $\eta(p_{c}-\varepsilon_{0})$. For the strategy to
have at least 0 expectancy, we need the futures settlement to have expectancy
of at least $\eta(\varepsilon_{0}-p_{c})$. In Section 4.3, two versions of
this strategy were outlined. We consider both here. In both sub-proofs, we use
the assumption that the risk-free rate is 0, which coupled with our martingale
assumption for $\varepsilon$ means the external market price at time $t$ is
such that $\mathbb{E}(\varepsilon_{t})=\varepsilon_{0}$. We now consider the
two options for settling futures outlined in Section 4.3

Option 1: Settle futures by auctioning tokens at the original converted price.
The arbitrageur who converted tokens for the pool at price $p_{c}$ must
auction off the tokens at price $p_{c}$. Let the auction happen at time $t$,
with external market price at that time of $\varepsilon_{t}$. Notice that what
is actually being sold is the right, and obligation, to buy $\eta$ tokens at
price $p_{c}$. This has value $\eta(\varepsilon_{t}-p_{c})$, which can be
negative. As negative bids are paid to the auction winner by the protocol, and
positive bids are paid to the protocol, we are able to apply Lemma 1. As such,
the winning bid is at least $\eta(\varepsilon_{t}-p_{c})$, which has
expectancy of at least

$\mathbb{E}(\eta(\varepsilon_{t}-p_{c}))=\eta(\mathbb{E}(\varepsilon_{t})-p_{c})=\eta(\varepsilon_{0}-p_{c}).$
(9)

Thus the expectancy of owning the future for the protocol is at least
$\eta(\varepsilon_{0}-p_{c})$, as required.

Option 2: Settle futures using frequent batch auction settlement price. For a
swap with external market price $\varepsilon_{t}$ at time $t$, a batch auction
in this swap settles at $\varepsilon_{t}$ in expectancy (Theorem 5.1 in [13]).
Thus the futures owned by the protocol have expectancy

$\mathbb{E}(\eta(\varepsilon_{t}-p_{c}))=\eta(\mathbb{E}(\varepsilon_{t})-p_{c})=\eta(\varepsilon_{0}-p_{c}).$
(10)

See 3

###### Proof

Consider a Diamond pool $\Phi$ with vault containing $2\eta$ tokens. WLOG let
these be of token $y$. Therefore the pool must sell $\eta$ tokens at the
external market price to balance the vault. Let the conversion auction accept
bids at time $t$, at which point the external market price is
$\varepsilon_{t}$. For the auction to have expectancy of at least 0, we
require the winning bid to be at least $\eta\varepsilon_{t}$. The result
follows from Lemma 1.

See 5.1

###### Proof

To see this, we first know that for $CFMM(\Phi)$ at time $t$ with reserves
$(R_{x,t},R_{y,t})$, LVR corresponds to the optimal solution
$(R^{*}_{x,t+1},R^{*}_{y,t+1})$ with external market price $\varepsilon_{t+1}$
which maximizes:

$(R_{x,t+1}-R_{x,t})+(R_{y,t+1}-R_{y,t})\varepsilon_{t+1}.$ (11)

Let this quantity be

$L=(R^{*}_{x,t+1}-R_{x,t})+(R^{*}_{y,t+1}-R_{y,t})\varepsilon_{t+1}.$ (12)

In Diamond, a player trying to move the reserves of $\Phi$ to
$(R^{\prime}_{x,t+1},R^{\prime}_{y,t+1})$ only receives
$(1-\beta)(R^{\prime}_{x,t+1}-R_{x,t})$ while giving
$(1-\beta)(R^{\prime}_{y,t+1}-R_{y,t})$ to $\Phi$. Thus, an arbitrageur wants
to find the values of $(R^{\prime}_{x,t+1},R^{\prime}_{y,t+1})$ that maximize:

$(1-\beta)(R^{\prime}_{x,t+1}-R_{x,t})+(1-\beta)(R^{\prime}_{y,t+1}-R_{y,t})\varepsilon_{t+1}+\mathbb{E}(\text{conversion}).$
(13)

where $\mathbb{E}(\text{conversion})$ is the per-block amortized expectancy of
the conversion operation for the arbitrageurs. From Lemma 1, we know
$\mathbb{E}(\text{conversion})\geq 0$ for $\Phi$. This implies the
arbitrageur’s max gain is less than:

$(1-\beta)(R^{\prime}_{x,t+1}-R_{x,t})+(1-\beta)(R^{\prime}_{y,t+1}-R_{y,t})\varepsilon_{t+1},$
(14)

for the $(R^{\prime}_{x,t+1},R^{\prime}_{y,t+1})$ maximizing Equation 13. From
Equation 12, we know this has a maximum at
$(R^{\prime}_{x,t+1},R^{\prime}_{y,t+1})=(R^{*}_{x,t+1},R^{*}_{y,t+1})$.
Therefore, the LVR of $\Phi$ is at most:

$(1-\beta)((R^{*}_{x,t+1}-R_{x,t})+(R^{*}_{y,t+1}-R_{y,t})\varepsilon_{t+1})=(1-\beta)L.$
(15)


# Where It Really Matters: Few-Shot Environmental Conservation Media
Monitoring for Low-Resource Languages

Sameer Jain1, Sedrick Scott Keh1, Shova Chettri2, Karun Dewan2,
Pablo Izquierdo2, Johanna Prussman2, Pooja Shreshtha2, César Suárez2,
Zheyuan Ryan Shi3, Lei Li1, Fei Fang1

###### Abstract

Environmental conservation organizations routinely monitor news content on
conservation in protected areas to maintain situational awareness of
developments that can have an environmental impact. Existing automated media
monitoring systems require large amounts of data labeled by domain experts,
which is only feasible at scale for high-resource languages like English.
However, such tools are most needed in the global south where news of interest
is mainly in local low-resource languages, and far fewer experts are available
to annotate datasets sustainably. In this paper, we propose NewsSerow, a
method to automatically recognize environmental conservation content in low-
resource languages. NewsSerow is a pipeline of summarization, in-context few-
shot classification, and self-reflection using large language models (LLMs).
Using at most 10 demonstration example news articles in Nepali, NewsSerow
significantly outperforms other few-shot methods and achieves comparable
performance with models fully fine-tuned using thousands of examples. The
World Wide Fund for Nature (WWF) has deployed NewsSerow for media monitoring
in Nepal, significantly reducing their operational burden, and ensuring that
AI tools for conservation actually reach the communities that need them the
most. NewsSerow has also been deployed for countries with other languages like
Colombia.

##  1 Introduction

News monitoring for environmental conservation has long been a significant
task of many nonprofit organizations. The organizations aim to remain informed
about developments that can have potentially significant environmental
impacts, and intervene promptly, such as advocating for more environmental-
friendly progress. However, manually reading and analyzing news articles is a
very labor-intensive and time-consuming process. Recent studies have eased
this burden by automating the scraping, classification, and analysis of
conservation-related articles (Keh et al. 2023). Such efforts allow for early
detection of high-impact activities and enable nonprofits to take timely
action.

However, although effective, such approaches come with a hefty price tag: they
require a lot of labeled data prepared by domain experts at these
organizations, not only in training the model, but also in its maintenance
after the tool has been deployed. This is significant extra work for the staff
at these conservation organizations who are already occupied with their
routine job. As such, it is no wonder previous works focus on articles written
in English only, due to the relative abundant human resources. However, these
news monitoring tools would actually make a much bigger difference in the
global south where the news of interest is mainly in local low-resource
languages. The reason is two-fold: First, in these regions, there are more
developments that may yield high environmental impacts. Second, many projects
do not gain exposure on international media until they reach a later more
developed stage, if they gain any exposure at all, where it would be too late
for nonprofits to intervene. Our collaborators at the World Wide Fund for
Nature (WWF), a conservation-focused nonprofit, have local offices in over 70
countries. Admittedly, for many of these 70+ countries, it is impossible to
develop and maintain a media monitoring tool in local languages in the same
way as has been done in English – the operational burden and data work are too
much for their limited workforce.

From a technical perspective, automatically classifying news articles written
in local languages is challenging because many of these are low-resource
languages that do not enjoy the same power from large language models (LLMs)
as they are trained on high-resource languages like English. Cultural factors
may also come into play: certain things that are acceptable in Western
countries may be taboo under certain cultures, and vice-versa. Consequently,
as we will show later, off-the-shelf language models may struggle to
accurately classify articles in low-resource languages. Fine-tuning is also
difficult because it requires a neatly annotated dataset, which is often not
readily available.

To tackle this challenge, we propose NewsSerow, a multilingual few-shot
classification method for conservation news monitoring. NewsSerow is built on
top of LLMs and consists of a novel pipeline of summarization, in-context few-
shot demonstrations, and self-reflection. For news articles written in Nepali,
with only 10 examples, NewsSerow significantly outperforms other few-shot
methods and can even achieve comparable performance with models fully fine-
tuned using more than 1800 labeled data points. NewsSerow was initially
developed and demonstrated for Nepali, and hence the algorithm’s name –
Himalayan serow, an important animal in Nepal. The method is scalable and
language-agnostic, allowing it to generalize well to a wide variety of
languages with relatively little effort. Beyond the results for Nepali, we
also show positive results for Spanish. Our system is currently being deployed
by the country offices of WWF in Nepal and Colombia.

##  2 Related Work

##### News Monitoring Pipelines

The earliest published work similar to NewsSerow was that of Hosseini and Coll
Ardanuy (2020). They collected and labelled an English dataset, then used BERT
models (Devlin et al. 2019) to classify articles as positive or negative to
conservation. However, the work was not deployed in the real world. Meanwhile,
Keh et al. (2023) collect localized datasets for English articles covering
India and Nepal. Their proposed toolkit focuses specifically on conservation
and infrastructure projects and has been deployed in collaboration with WWF.
However, none of these previous studies address the news articles written in
local languages, which is a crucial gap that our system aims to fill.

##### Multilingual Models

Adapting large multilingual models such as m-BERT (Devlin et al. 2019) and
XLM-R (Conneau et al. 2020) to specific downstream tasks through fine-tuning
is a popular approach in multilingual settings. More recently, larger models
in the scale of tens or hundreds of billions of parameters (Brown et al. 2020;
OpenAI 2023; Touvron et al. 2023) have shown remarkable ability to understand
and generate coherent text in a plethora of languages, including low-resource
languages such as Nepali. These massive models are notably able to operate in
zero-shot or few-shot settings. As such, they become the foundation in the
development of NewsSerow, for which we rely on few-shot in-context learning
for domain-specific classification, utilizing domain expert insights along
with recent techniques in prompting.

##### Prompting

Prompting has emerged as a major tool to elicit reasoning abilities from
language models (Liu et al. 2021). This usually involves providing the model
with an instruction or a set of few-shot examples, then having the model
complete the input prompt. Chain-of-thought reasoning (Wei et al. 2022; Kojima
et al. 2022) involves asking language models to reason in a scratchpad-like
manner (Nye et al. 2021) before providing a final response.

##### LLMs To Critique and Self-Correct

As the capabilities of language models continue to progress, strong models
such as gpt-3.5-turbo or gpt-4 (OpenAI 2023) have been shown to be reasonable
as evaluation tools (Li et al. 2023; Gilardi, Alizadeh, and Kubli 2023; Jain
et al. 2023) to evaluate other outputs as part of the inference pipeline.
Further, they possess the ability to reflect and iterate over their own
responses as well (Jang 2023; Kadavath et al. 2022). By adding this layer of
filtering, we can achieve nontrivial performance improvements (Shinn et al.
2023), which is especially useful in structured generation applications such
as code generation (Welleck et al. 2022). We incorporate a reflection step
into the NewsSerow pipeline.

##  3 NewsSerow Methodology

Figure 1: NewsSerow prompt pipeline. We illustrate the flow of the test
example in red. Model responses that are used as input for later prompts are
color-coded by background. For example, the test article summary (highlighted
yellow) is generated by the summarization module and used in the
classification and reflection modules.

###  3.1 Outline of the Pipeline

Figure 1 contains an overview of the NewsSerow pipeline. Our framework uses a
gpt-3.5-turbo model to perform classification. In our default setup, we
utilize 10 randomly chosen demonstration examples for in-context learning. For
each article in the test set, our pipeline consists of three steps. We list
the steps before delving deeper in corresponding subsections:

  1. 1.

Summarization: We leverage the zero-shot generation capabilities of
gpt-3.5-turbo to generate a summary of the test article in a given language.

  2. 2.

Classification: We pass the model the titles and summaries of 10 randomly
chosen example articles with their true labels along with the title and
summary of the test article. We extract the preliminary conservation label
from the model response at this stage.

  3. 3.

Reflection: We further prompt the model to determine the correctness of its
own preliminary response by passing the test article again, but this time
along with its own output from the previous step.

This pipeline is general and works for other LLMs pretrained on multilingual
data.

###  3.2 Summarization Module

We pass an article to gpt-3.5-turbo and direct it to generate a zero-shot
summary in the same language. We prompt the model to restrict the summary to 3
sentences to prevent overflowing the context window in the next step.

Since our main classification step requires example articles to be fed into
the prompt along with the test article, we cannot feed the entirety of
articles due to the limit set by the gpt-3.5-turbo context window.
Summarization allows us to efficiently utilize the limited context window of
the model while ensuring that the salient points of the article are included
in the prompt. We study the effect of this step in an ablation study in
Section 4.5.

###  3.3 Classification Module

The classification module uses in-context learning (Brown et al. 2020), which
is the capability of LLMs to learn to perform a task by utilizing a few input-
output examples. In this step, we feed gpt-3.5-turbo a prompt that consists
of:

  1. 1.

A task description that defines environmental conservation and instructs the
LLM to observe the given examples and comment on whether the test example is
relevant to conservation or not.

  2. 2.

A set of in-context examples (10, by default) that are randomly sampled from
the training set. For each example article, we feed in the article title and
summary, followed by a response specifying whether or not the article is
relevant to conservation, along with an explanation for the classification.

  3. 3.

The title and summary of the test example.

Following the pattern it observes in the examples, the model outputs a
classification label for the test example along with a justification for the
label.

In providing explanations for the true labels of the in-context examples, we
utilize chain-of-thought prompting (Wei et al. 2022), which is the capacity of
LLMs to observe intermediate reasoning steps in the given in-context examples
to improve their own performance on reasoning tasks. In addition to improved
performance, chain-of-thought reasoning gives the additional advantage of
having the model justify its classification decision for the test example,
which can be used by NGOs using the system as a source of additional distilled
information.

The classification module is followed by a decision step. If the article is
classified to be not relevant to conservation, the pipeline ends here with not
relevant being the final label. If the article is classified to be relevant,
we go through an additional filtering step, described next.

###  3.4 Reflection Module

False positives are one of the most frequent category of errors we observe
when using in-context learning for classification. Furthermore, in the
justifications given by the model for its decisions, the model often makes a
tenuous connection between the article content and environmental conservation.
Consequently, it labels the article as being relevant to conservation, thus
driving up false positives. For example, the model often assigns a positive
label to articles that mention protected areas, even in cases when the focus
of the news is not conservation.

To address this, we add a reflection module to our pipeline (Kadavath et al.
2022; Shinn et al. 2023). Concretely, if the classification module labels an
article as being relevant to conservation, we go through an additional
filtering step. We pass the test example (title + summary) to a new prompt
along with its own classification label from the previous step. We then ask
the LLM to comment on the correctness of this assessment. If it thinks its
previous positive assessment is correct, the pipeline would output relevant.
Otherwise, not relevant would be the final label.

##  4 Experiments

We run a series of experiments and ablation studies to demonstrate the
effectiveness of NewsSerow. NewsSerow is a few-shot classification algorithm
designed specifically for low-resource languages. As an example of such
languages, we use its results on Nepali news articles as the primary
performance indicator of NewsSerow. Of course, there is nothing that stops
NewsSerow from working with other, high-resource languages. In fact, to
demonstrate its generalizability, we also run experiments on Colombian news
articles which are written in Spanish.

###  4.1 Datasets

We collect a dataset of 796 news articles (548 Nepal and 248 Colombia) with
binary labels for their relevance to environmental conservation. We collect
the data for Colombia and Nepal using NewsAPI and a custom crawler
respectively:

  1. 1.

Colombian Data: We use NewsAPI to obtain all Spanish language news articles.
We obtain articles relevant to Colombia by filtering articles that either have
the string “Colombia” in the title or if they come from a Colombian news
domain name. We increase the percentage of conservation-related news in the
dataset by further filtering by a search term consisting of the name of a
protected area in Colombia, taken from a list of relevant areas curated by
WWF. We query NewsAPI for a 90-day period from December 2022 to March 2023 and
obtain a set of 2748 articles, from which we sample 248 articles for labeling.
The articles are labeled by domain experts from WWF for relevance to
conservation.

  2. 2.

Nepali Data: Since NewsAPI currently does not support the Nepali language, we
built a custom web crawler to obtain Nepali news articles. We scrape articles
from four Nepali news websites suggested by WWF and filter the articles by
matching on a search term consisting of the name of a protected area in Nepal.
Collection was done from December 2022 to March 2023. Articles were scraped
weekly and labeled by domain experts from WWF for relevance to conservation.
The final set contained 548 articles.

In addition, we also have 1647 news articles in English with similar binary
labels. These data were shared by the authors of (Keh et al. 2023). We use all
of these English articles for training the Nepali NewsSerow and Colombian
NewsSerow. Table 1 describes the data used in the training, validation, and
testing of NewsSerow.

Split | Nepal/Nepali | Colombia/Spanish
---|---|---
Demonstrations (Few-shot) | 10 Nepali articles | 10 Spanish articles
Training (Fine-tuned) | 172 Nepali articles + 1647 English articles | 73 Spanish articles + 1647 English articles
Validation | 100 Nepali articles | 50 Spanish articles
Test | 276 Nepali articles | 125 Spanish articles
Table 1: The train-validation-test split of the dataset used in the experiments shown in Table 2. For the English data in Translation Test (see Table 2), out of 1647 examples, we use 300 for test, 300 for validation, and the rest for training. Model | Nepal/Nepali | Colombia/Spanish
---|---|---
| Precision | Recall | F1-Score | Precision | Recall | F1-Score
Zero-shot Models |  |  |  |  |  |
GPT-3.5-Turbo | $0.79\>(0.02)$ | $0.32\>(0.01)$ | $0.46\>(0.01)$ | $1.00\>(0.00)$ | $0.14\>(0.00)$ | $0.25\>(0.00)$
Few-shot Models |  |  |  |  |  |
mBERT | $0.26\>(0.03)$ | $0.70\>(0.12)$ | $0.37\>(0.01)$ | $0.25\>(0.03)$ | $0.71\>(0.18)$ | $0.36\>(0.04)$
XLM-R | $0.35\>(0.04)$ | $0.60\>(0.23)$ | $0.43\>(0.06)$ | $0.49\>(0.24)$ | $0.61\>(0.41)$ | $0.44\>(0.25)$
GPT-3.5-Turbo | $0.68\>(0.05)$ | $0.58\>(0.07)$ | $0.62\>(0.02)$ | $0.39\>(0.03)$ | $1.00\>(0.00)$ | $0.56\>(0.03)$
NewsSerow | $\underline{0.88}\>(0.03)$ | $\underline{0.58}\>(0.01)$ | $\underline{\mathbf{0.70}}\>(0.01)$ | $\underline{0.89}\>(0.02)$ | $\underline{0.71}\>(0.00)$ | $\underline{\mathbf{0.79}}\>(0.01)$
Fine-tuned Models |  |  |  |  |  |
mBERT | $0.77\>(0.06)$ | $0.57\>(0.05)$ | $0.65\>(0.02)$ | $0.86\>(0.10)$ | $0.80\>(0.17)$ | $\mathbf{0.81}\>(0.08)$
XLM-R | $0.70\>(0.05)$ | $0.71\>(0.04)$ | $\mathbf{0.70}\>(0.01)$ | $0.69\>(0.03)$ | $0.95\>(0.03)$ | $0.80\>(0.02)$
Translation Test | $0.74\>(0.16)$ | $0.64\>(0.15)$ | $0.66\>(0.06)$ | $0.76\>(0.21)$ | $0.67\>(0.19)$ | $0.66\>(0.16)$
Table 2: We compare the performance of NewsSerow against a set of zero-shot,
few-shot, and fully fine-tuned baselines. All metrics are averaged over 5 runs
with standard deviations shown in parentheses. On F1-score, NewsSerow performs
significantly better than the few-shot baselines for both languages. More
importantly, it also matches the performance of the best fine-tuned model for
Nepali while staying marginally below state-of-the-art performance for
Colombian, while having much lower variance and using significantly less data.

###  4.2 Evaluation Metrics

We model the identification of relevant news articles as a binary
classification task and use precision, recall, and F1 score for evaluation.
Since the dataset is imbalanced in favour of negative examples, we do not
report accuracy. We report the metrics on the positive class. All the numbers
we report are an average across 5 runs. For the mBERT and XLM-R baselines, we
use 5 different random seeds for training. For the results on GPT models, we
use 5 random seeds to shuffle in-context examples in the prompt. We set
temperature to zero for all GPT experiments in order to make the responses
deterministic.

###  4.3 Baselines

We compare the performance of NewsSerow to the following baselines:

#### Zero-Shot Baselines

  1. 1.

Zero-Shot GPT-3.5-Turbo: We prompt the off-the-shelf model to perform zero-
shot classification, i.e., without using any in-context examples and also not
using the other features of our proposed framework–chain-of-thought, zero-shot
summarization, and reflection.

#### Few-Shot Baselines

  1. 1.

Few-Shot GPT-3.5-Turbo: We prompt the model to use in-context learning with 10
examples but without using the other features of NewsSerow – chain-of-thought,
zero-shot summarization, and reflection. In place of the zero-shot summary, we
use the first 3 sentences of the article. We discuss the effect of this choice
in the ablation study (Section 4.5) on the summarization module.

  2. 2.

Few-Shot Finetuned Multilingual Encoders: We fine-tune large multilingual
models such as mBERT (Devlin et al. 2019) and XLM-R (Conneau et al. 2020). We
fine-tune a different model for each language (Nepali or Spanish) on 10
examples from that language. We use a batch size of 1, an initial learning
rate equal to 5e-06 for the AdamW optimizer (Loshchilov and Hutter 2019). We
train for 30 epochs and use the best epoch to evaluate.

(a)

(b)

Figure 2: Fine-tuned XLM-R’s performance with number of training examples on
the (a) Nepali (purple) and (b) Spanish (green) test sets. X-axis shows the
number of target language (Nepali/Spanish) training examples, in addition to
which 1647 English examples are used to fine-tune the models. We show
NewsSerow’s 10-shot performance on the same test sets in red with a light red
error bar. Figure 3: NewsSerow’s performance with number of in-context
examples

#### Fully Fine-Tuned Baselines

  1. 1.

Translate-Test with RoBERTa: We have much more data (Keh et al. 2023) in the
English language as compared to the other languages. We fine-tune a RoBERTa
(Liu et al. 2019) model on the English data consisting of 1647 articles
labeled for relevance to environmental conservation. We then use a translate-
test pipeline to classify examples from the other languages.

  2. 2.

Fully Fine-Tuned XLM-R and mBERT: We fine-tune mBERT and XLM-R on all
available training data. We have 172 Nepali training examples and 73 Spanish
training examples. In addition to target language data, we find that using
English language data boosts performance, so we include 1647 English articles
in the training set, resulting in a total of 1819 training examples for Nepali
and 1720 examples for Spanish. We use a batch size of 1, an initial learning
rate equal to 5e-06 for the AdamW optimizer. We train for 10 epochs and use
the best epoch to evaluate.

###  4.4 Results and Analyses

#### Comparing NewsSerow with Off-the-Shelf GPT-3.5-Turbo

NewsSerow is a few-shot framework that uses, by default, 10 examples. We
compare its performance with a vanilla GPT-3.5-Turbo model. We provide the
vanilla model the same set of in-context examples without any of the prompt
engineering techniques used in NewsSerow, such as the chain-of-thought
explanations and the zero-shot summaries. We instead use the initial three
sentences of the article as a summary (Narayan, Cohen, and Lapata 2018;
Nenkova 2005).

As shown in Table 2, NewsSerow outperforms a vanilla GPT-3.5-Turbo model on
both languages, gaining 8 points and 23 points on mean F1 Score on Nepali and
Spanish respectively. It also reduces the variance compared to the vanilla
model. For Nepali, NewsSerow improves precision by 20 points while not
suffering a drop in recall.

#### Comparing NewsSerow with Multilingual Encoders

We compare NewsSerow with XLM-R and mBERT trained in two settings:

  1. 1.

Few-Shot: Since NewsSerow uses 10 examples, we first compare it to XLM-R and
mBERT fine-tuned on the same 10 examples (Table 2). Both models vastly
underperform NewsSerow in the few-shot setting. In fact, they also
underperform a vanilla gpt-3.5-turbo model. This is unsurprising, given that
LLMs like gpt-3.5-turbo are designed to generalize well from a few examples,
while traditional models such as mBERT and XLM-R require more data-intensive
finetuning.

  2. 2.

Full Finetuning: Since XLM-R and mBERT are typically intended to be fine-tuned
on larger datasets, we fine-tune them on all available training data,
including English language data, which also boosts performance. In summary, we
finetune on 1819 examples (172 Nepali + 1647 English) for the Nepali models
and 1720 examples (73 Spanish + 1647 English) for the Spanish models. As
illustrated in Table 2, NewsSerow matches or exceeds the performance of these
models while using a fraction of the data. Furthermore, it does this while
achieving strictly lower variance as compared to mBERT and XLM-R. For the
translation-test results, while they outperform some of the few-shot
baselines, they perform slightly worse than both the NewsSerow and the fine-
tuned mBERT/XLM-R models. This is likely due to inaccuracies during the
translation process.

Setting | Performance
---|---
CoT. | Sum. | Ref. | Precision | Recall | F1 Score
No | No | No | $0.68\>(0.05)$ | $0.58\>(0.07)$ | $0.62\>(0.02)$
No | No | Yes | $0.94\>(0.02)$ | $0.28\>(0.01)$ | $0.43\>(0.01)$
No | Yes | No | $0.70\>(0.04)$ | $0.68\>(0.06)$ | $0.69\>(0.03)$
No | Yes | Yes | $0.91\>(0.02)$ | $0.49\>(0.01)$ | $0.63\>(0.01)$
Yes | No | No | $0.57\>(0.05)$ | $0.82\>(0.05)$ | $0.67\>(0.02)$
Yes | No | Yes | $0.87\>(0.01)$ | $0.38\>(0.00)$ | $0.53\>(0.00)$
Yes | Yes | No | $0.60\>(0.06)$ | $0.84\>(0.08)$ | $0.69\>(0.01)$
Yes | Yes | Yes | $0.88\>(0.03)$ | $0.58\>(0.01)$ | $0.70\>(0.01)$

Table 3: Ablation results on varying chain-of-thought (CoT.), summarization
(Sum.), and reflection (Ref.) settings for Nepali. All results are an average
over 5 runs with standard deviations shown in parentheses.

Setting | Performance
---|---
CoT. | Sum. | Ref. | P | R | F1
No | No | No | $0.39\>(0.03)$ | $1.00\>(0.00)$ | $0.56\>(0.03)$
No | No | Yes | $1.00\>(0.00)$ | $0.76\>(0.00)$ | $0.86\>(0.00)$
No | Yes | No | $0.37\>(0.03)$ | $0.98\>(0.02)$ | $0.53\>(0.03)$
No | Yes | Yes | $0.86\>(0.03)$ | $0.72\>(0.02)$ | $0.79\>(0.02)$
Yes | No | No | $0.46\>(0.04)$ | $0.99\>(0.02)$ | $0.62\>(0.04)$
Yes | No | Yes | $1.00\>(0.00)$ | $0.81\>(0.00)$ | $0.89\>(0.00)$
Yes | Yes | No | $0.52\>(0.04)$ | $0.94\>(0.06)$ | $0.67\>(0.03)$
Yes | Yes | Yes | $0.89\>(0.02)$ | $0.71\>(0.00)$ | $0.79\>(0.01)$

Table 4: Ablation results on varying chain-of-thought (CoT.), summarization
(Sum.), and reflection (Ref.) settings for Spanish. All results are an average
over 5 runs with standard deviations shown in parentheses.

NewsSerow does much better than multilingual encoder models when the latter
are trained in a few-shot setting and that it achieves comparable performance
to the multilingual models when they are trained with a much larger amount of
data. To study this more closely, we consider a setting in which labeled
English language data are easily available while the target-language data is
much harder to acquire. This is a reasonable assumption especially for a
lower-resourced language such as Nepali. We measure the performance of XLM-R
on the Nepali language test set when finetuned on 1647 English examples and a
variable number of Nepali examples. We benchmark this against NewsSerow’s
performance using 10 Nepali examples (Figure 2(a)). It can be observed that in
addition to English examples, XLM-R requires at least 90 Nepali-language
examples to reach the performance obtained by NewsSerow. Even then, NewsSerow
achieves less variance compared to XLM-R.

#### Where Does NewsSerow Make Mistakes?

One of the failure modes of NewsSerow is making a tenuous connection between
an article and environmental conservation to output a false positive. For
example, for an article about climbing preparation, it outputs: “The article
talks about the preparation for climbing Annapurna and Mt. Everest, which are
both related to environment conservation as they are conservation sites.”
While mountains are conservation sites, the subject of the referred article is
a climb. As another example, for an article about the architectural style for
new construction in an ancient city, the model responds by saying “The article
talks about the construction of houses in an ancient city of Panauti in a way
that preserves the traditional style of architecture. This news is related to
cultural and environmental conservation.” In both cases, the model correctly
captures the meaning of the article but incorrectly labels it as being
relevant. Often, though not always, the self-reflection step is able to
correct this. Even in cases where it is not corrected, the response provides
value to a domain expert by being interpretable and easy to spot as being
incorrect.

###  4.5 Ablation Studies

In this section, we first study the effects of varying the number of in-
context examples and then discuss ablations on the prompt engineering features
of NewsSerow: zero-shot summarization of article text, chain-of-thought
reasoning in the classification prompt, and reflection of the classification
response. The effects of these three features are summarized in Tables 3 and
4, where we illustrate performance variations caused by switching each of
these features on/off.

#### Effect of the Number of In-Context Examples:

As shown in Figure 3, increasing the number of in-context examples from 2 to
10 gives a slight upward trend in performance across both languages.
Additionally, while there is not a clean trend between the number of examples
and variance, we observe that instances of high variance typically occur when
a lower number of examples is used. A higher number of example stabilizes
performance.

#### Effect of Chain-of-Thought Reasoning:

Recall from Section 3.3 that we model chain-of-thought reasoning by giving
explanations for true labels of the example articles. We observe (Tables 3 and
4) that using chain-of-thought explanations while keeping the other settings
constant always improves F1-score for both Nepali and Spanish. In the case of
Nepali, this can cleanly be attributed to an increase in recall that outpaces
a slight drop in precision. In the case of the Colombian dataset, recall on
average stays similar while precision gets a slight boost, thus improving the
F1 score.

#### Effect of the Summarization Module:

Recall that we use article summaries to efficiently utilize the limited
context window of the classification prompt while ensuring that the salient
points of the article are captured.

Using the first few sentences of a text is often used as a strong extractive
summarization baseline (Narayan, Cohen, and Lapata 2018; Nenkova 2005).
Consequently, to study the efficacy of the summarization module, we compare it
to a baseline where we use the first 3 sentences of the article as a summary.
In Tables 3 and 4, the summarization column takes the value yes if the
summarization module is used to generate a zero-shot summary and no if the
first three sentences of the article are used as an extractive summary.

We observe that for Nepali, the summarization module always improves accuracy
and F1 score, by boosting recall without causing a precision drop. For
Spanish, recall generally decreases on using the zero-shot summary, whereas
precision does not show a neat pattern.

#### Effect of Using Reflection:

Recall from Section 3.4 that we use reflection only when the prediction made
by the classification module is positive. As such, reflection increases
precision at the cost of recall. In this, we find reflection to be an
effective precision-recall switch that can be used by the non-profits to
control for the precision-recall tradeoff based on their requirements and
resource availability.

##  5 Deployment

NewsSerow has been deployed by WWF in Nepal and Colombia since April 2023.
Every week, NewsSerow scrapes articles as described in Section 4.1, runs the
classification pipeline on the retrieved articles, and shares the lists of
articles predicted to be relevant to conservation with WWF. We have used the
weekly feedback from our collaborators from WWF to make iterative improvements
to NewsSerow. This paper describes the latest version of NewsSerow, which was
deployed in August 2023.

We present the results on NewsSerow’s performance on data labeled by WWF from
April to July 2023 in Table 5. Because the objective of this earlier process
was to iterate on previous versions of our pipeline, these evaluation points
from April to July 2023 are not necessarily representative of the distribution
of articles NewsSerow observes in a regular week. In particular, through our
weekly pipeline, we have been requesting labels for only the lists of articles
that were classified as being positive by (earlier iterations of) NewsSerow.
And hence the precision, recall, and F1 score are calculated for this subset
of the data only.111Note that the data described in Section 4.1 for all the
offline experiments are unbiased since they are made up of a random sample of
articles scraped between December 2022 and March 2023. Labeling the whole
dataset would be too burdensome for our partners at WWF and would go against
the initial motivation of this project. Additionally, since we have used our
observations from this weekly exchange of results in developing NewsSerow, it
can be considered a pseudo-validation set. Thus, the evaluation metrics
reported below are not exactly comparable with those from the offline
experiments in Section 4. Nonetheless, the results still provide valuable
insights into the real-world performance of NewsSerow, as below.

As shown in Table 5, over the 8 weeks of deployment so far, NewsSerow achieves
great performance across both languages. It achieves F1 scores of 0.64 and
0.67 on Nepali and Colombian articles respectively. The variations in results
across weeks, especially for Nepali, are likely due to the limited number of
data points per week for Nepali and is less of an issue with Spanish which has
more articles weekly. Notably, the performance metrics are not as good as
those in the offline experiments. As described previously, the offline and
deployment datasets do not have the same distribution. Further, there is a
time shift in the training and testing distributions for the deployment
results but not for the offline experiments. Nevertheless, the F1 scores for
both languages in deployment is still higher than the other few-shot models.

| Nepal | Spanish (Colombia)
---|---|---
Week | # Ex. | P | R | F1 | # Ex. | P | R | F1
1 | 10 | $0.5$ | $1.00$ | $0.67$ | 28 | 0.57 | 1.00 | 0.73
2 | 9 | $0.00$ | $0.00$ | $0.00$ | 22 | 1.00 | 0.70 | 0.82
3 | 21 | $1.00$ | $0.33$ | $0.50$ | 28 | 0.5 | 0.8 | 0.62
4 | 9 | $1.00$ | $0.75$ | $0.86$ | 26 | 0.63 | 0.71 | 0.67
5 | 9 | $0.50$ | $0.50$ | $0.50$ | 35 | 0.67 | 0.80 | 0.73
6 | 5 | $0.50$ | $1.00$ | $0.67$ | 23 | 0.38 | 0.50 | 0.43
7 | 8 | $1.00$ | $1.00$ | $1.00$ |  |  |  |
8 | 13 | $1.00$ | $0.38$ | $0.55$ |  |  |  |
Aggr. | $\mathbf{84}$ | $\mathbf{0.77}$ | $\mathbf{0.55}$ | $\mathbf{0.64}$ | $\mathbf{162}$ | $\mathbf{0.61}$ | $\mathbf{0.73}$ | $\mathbf{0.67}$
Table 5: Weekly results of the NewsSerow deployed model from April to July
2023. For Nepali, we have 84 data points collected over 8 weeks. For Spanish,
we have 162 data points collected over 6 weeks.

##  6 Conclusion

In this paper, we present NewsSerow, a multilingual few-shot framework that
utilizes large language models (LLMs) for identifying news content relevant to
conservation efforts. We illustrate that LLMs with well-engineered prompts can
utilize a handful of illustrative examples to perform on par with or better
than fine-tuned baselines that require training on much larger datasets. In
particular, this holds true also for lower-resourced languages such as Nepali,
which are typically not as well represented in the corpora used to pre-train
LLMs as higher-resourced languages such as English. Consequently, the
framework can easily be adapted by conservation-focused non-profits working in
local languages anywhere in the world, and save significant amounts of human
effort and time that goes into manual media monitoring.

## Acknowledgements

We thank Kanishka Bhambhani and Pratik Joshi for writing the scrapers to
collect the Nepali news data used in this paper. The work is supported in part
by NSF grant IIS-2046640 (CAREER) and a research grant from Google. Zheyuan
Ryan Shi was supported by a CMU Presidential Fellowship and Siebel Scholarship
and did this work while at CMU. Lei Li’s work is partially supported by a gift
grant from Apple Inc.

## References

  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.  Language Models are Few-Shot Learners.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 1877–1901. Curran Associates, Inc.
  * Conneau et al. (2020) Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V. 2020.  Unsupervised Cross-lingual Representation Learning at Scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 8440–8451. Online: Association for Computational Linguistics.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Gilardi, Alizadeh, and Kubli (2023) Gilardi, F.; Alizadeh, M.; and Kubli, M. 2023.  ChatGPT outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences of the United States of America_ , 120.
  * Hosseini and Coll Ardanuy (2020) Hosseini, K.; and Coll Ardanuy, M. 2020.  Data Study Group Final Report: WWF.
  * Jain et al. (2023) Jain, S.; Keshava, V.; Mysore Sathyendra, S.; Fernandes, P.; Liu, P.; Neubig, G.; and Zhou, C. 2023.  Multi-Dimensional Evaluation of Text Summarization with In-Context Learning.  In _Findings of the Association for Computational Linguistics: ACL 2023_. Toronto, Canada: Association for Computational Linguistics.
  * Jang (2023) Jang, E. 2023.  Can LLMs Critique and Iterate on Their Own Outputs?  _evjang.com_.
  * Kadavath et al. (2022) Kadavath, S.; Conerly, T.; Askell, A.; Henighan, T.; Drain, D.; Perez, E.; Schiefer, N.; Hatfield-Dodds, Z.; DasSarma, N.; Tran-Johnson, E.; Johnston, S.; El-Showk, S.; Jones, A.; Elhage, N.; Hume, T.; Chen, A.; Bai, Y.; Bowman, S.; Fort, S.; Ganguli, D.; Hernandez, D.; Jacobson, J.; Kernion, J.; Kravec, S.; Lovitt, L.; Ndousse, K.; Olsson, C.; Ringer, S.; Amodei, D.; Brown, T.; Clark, J.; Joseph, N.; Mann, B.; McCandlish, S.; Olah, C.; and Kaplan, J. 2022.  Language Models (Mostly) Know What They Know.  arXiv:2207.05221.
  * Keh et al. (2023) Keh, S. S.; Shi, Z. R.; Patterson, D. J.; Bhagabati, N.; Dewan, K.; Gopala, A.; Izquierdo, P.; Mallick, D.; Sharma, A.; Shrestha, P.; and Fang, F. 2023.  NewsPanda: Media Monitoring for Timely Conservation Action.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 37(13): 15528–15536.
  * Kojima et al. (2022) Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022.  Large Language Models are Zero-Shot Reasoners.  _ArXiv_ , abs/2205.11916.
  * Li et al. (2023) Li, X.; Zhang, T.; Dubois, Y.; Taori, R.; Gulrajani, I.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023.  AlpacaEval: An Automatic Evaluator of Instruction-following Models.  https://github.com/tatsu-lab/alpaca˙eval.
  * Liu et al. (2021) Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; and Neubig, G. 2021.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _CoRR_ , abs/2107.13586.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  arXiv:1907.11692.
  * Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_.
  * Narayan, Cohen, and Lapata (2018) Narayan, S.; Cohen, S. B.; and Lapata, M. 2018.  Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 1797–1807. Brussels, Belgium: Association for Computational Linguistics.
  * Nenkova (2005) Nenkova, A. 2005.  Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference.  In _AAAI Conference on Artificial Intelligence_.
  * Nye et al. (2021) Nye, M. I.; Andreassen, A. J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; Sutton, C.; and Odena, A. 2021.  Show Your Work: Scratchpads for Intermediate Computation with Language Models.  _CoRR_ , abs/2112.00114.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _ArXiv_ , abs/2303.08774.
  * Shinn et al. (2023) Shinn, N.; Cassano, F.; Labash, B.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023.  Reflexion: Language Agents with Verbal Reinforcement Learning.  arXiv:2303.11366.
  * Touvron et al. (2023) Touvron, H.; Martin, L.; Stone, K. R.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D. M.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N.; Hartshorn, A. S.; Hosseini, S.; Hou, R.; Inan, H.; Kardas, M.; Kerkez, V.; Khabsa, M.; Kloumann, I. M.; Korenev, A. V.; Koura, P. S.; Lachaux, M.-A.; Lavril, T.; Lee, J.; Liskovich, D.; Lu, Y.; Mao, Y.; Martinet, X.; Mihaylov, T.; Mishra, P.; Molybog, I.; Nie, Y.; Poulton, A.; Reizenstein, J.; Rungta, R.; Saladi, K.; Schelten, A.; Silva, R.; Smith, E. M.; Subramanian, R.; Tan, X.; Tang, B.; Taylor, R.; Williams, A.; Kuan, J. X.; Xu, P.; Yan, Z.; Zarov, I.; Zhang, Y.; Fan, A.; Kambadur, M.; Narang, S.; Rodriguez, A.; Stojnic, R.; Edunov, S.; and Scialom, T. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _ArXiv_ , abs/2307.09288.
  * Wei et al. (2022) Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; brian ichter; Xia, F.; Chi, E. H.; Le, Q. V.; and Zhou, D. 2022.  Chain of Thought Prompting Elicits Reasoning in Large Language Models.  In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., _Advances in Neural Information Processing Systems_.
  * Welleck et al. (2022) Welleck, S.; Lu, X.; West, P.; Brahman, F.; Shen, T.; Khashabi, D.; and Choi, Y. 2022.  Generating Sequences by Learning to Self-Correct.  _ArXiv_ , abs/2211.00053.