# Additive Interventions Yield Robust
Multi-Domain Machine Translation Models

Elijah Rippeth
Department of Computer Science
University of Maryland
<EMAIL_ADDRESS>
&Matt Post
Microsoft
<EMAIL_ADDRESS>
Work was done during an internship at Microsoft

###### Abstract

Additive interventions are a recently-proposed mechanism for controlling
target-side attributes in neural machine translation. In contrast to tag-based
approaches which manipulate the raw source sequence, interventions work by
directly modulating the encoder representation of all tokens in the sequence.
We examine the role of additive interventions in a large-scale multi-domain
machine translation setting and compare its performance in various inference
scenarios. We find that while the performance difference is small between
intervention-based systems and tag-based systems when the domain label matches
the test domain, intervention-based systems are robust to label error, making
them an attractive choice under label uncertainty. Further, we find that the
superiority of single-domain fine-tuning comes under question when training
data size is scaled, contradicting previous findings.

##  1 Introduction

Multi-domain machine translation (MDMT) is the paradigm in which a single
model is trained to service many domains by training on multiple corpora
covering disparate labeled domains. The goal of MDMT is not only to provide
high quality general machine translation enabled by knowledge transfer across
domains, but also to enable high quality domain-specific machine translation
when a model is provided cues about the target domain, used to control the
generation. Though an intuitive task, the expectations surrounding the task
were only recently formalized by Pham et al. (2021) in which the authors
provided both a set of functional requirements demanded of successful MDMT
models and an experimental framework under which those requirements can be
tested.

Pham et al. (2021) explored several mechanisms for controlling domain, ranging
from simple tag-based approaches to meta-learning based mechanisms. According
to the functional requirements outlined by the authors, no method meets all
the expectations demanded of effective multi-domain machine translators,
though the experiments were run on a relatively small dataset of only in-
domain data. The primary remaining expectations, according to the authors, are
the superiority of fine-tuning based methods as compared to these methods
which can control the target domain, and the ability to accommodate fuzzy or
uncertain domains.

This framework is useful, but the authors leave open several other questions
regarding the state of MDMT. The first of these is data size. Previous
experiments focused only on relatively small, in-domain data in an otherwise
high-resource setting of English-French and found that most models pale in
comparison to models fine-tuned on a single domain. We wonder whether this
fine-tuning superiority conclusion holds under a more realistic paradigm in
which models trained on large, out-of-domain datasets are fine-tuned on in-
domain data. While pretraining and fine-tuning on in-domain data can yield
strong in-domain performance—as observed by the authors—this is likely to be
at the cost of general domain performance, calling into question the
transferability under MDMT.

Next, we wonder if new methods might help with the issue of domain control in
MDMT. The authors examine reasonable mechanisms for controlling the domain
which were known at the time. Since then, new methods have been developed
which we hope to investigate under the prescribed framework. We hypothesize
that additive interventions Schioppa et al. (2021), which learn tag embeddings
separately from the encoder, may be harder to ignore, and that the learned
interventions may be able to absorb target-side properties more easily, while
freeing the encoder to learn strong representations purely for translation.

In this work we scale the original experimental framework presented in Pham et
al. (2021) by including a significantly larger, more realistic dataset. We
also experiment with additive interventions as an alternative to domain
tagging. We find that:

  * •

additive interventions perform roughly equivalently with tag-based approaches
in the ideal case where provided tags match the target domain.

  * •

additive interventions are much more robust in the face of incorrect and
uncertain domain labels.

  * •

when the experiment is scaled, models fine-tuned targeting a single domain are
strong translators, but are never unmatched by other models which can service
multiple domains suggesting that MDMT models in a high-resource setting are
competitive with best-in-class baselines.

##  2 Method

As a baseline, we inject domain metadata using the tag-based approach. In this
scheme, a token representing the target-side attribute, $t$, is prepended to
source segment $x$ and fed to the encoder $E$ whose hidden representation is
finally exposed to decoder $D$ in a “normal" fashion:

$\displaystyle\hat{y}=D(E(\left[t\right]+x))$

where $+$ indicates sequence concatenation. In tag-based approaches, the
expectation is that the domain tag as a prefix acts as a conditioning variable
which encourages target-side attributes to appear as desired in the final
translation.

While effective and architecturally non-invasive, this method is not without
downsides. Because the target token’s contribution to the encoder
representation is learned, there is a chance that the attribute can be
ignored. To address this and other weaknesses of tag-based approaches,
Schioppa et al. (2021) present the additive interventions method which
requires an encoder $E$, a decoder $D$, and a separate attribute embedding
layer $Emb$. Given a source segment $x$ and a sentence-level attribute token
$t$, we have

$\displaystyle V$ $\displaystyle=Emb(t)$ $\displaystyle\hat{y}$
$\displaystyle=D(E(x)\oplus V)$

where $\oplus$ is defined as addition broadcasted along the token dimension.
Importantly, this allows prototypically discrete attributes to be represented
and controlled in a continuous fashion, allowing for interpolation, scaling,
and positionally invariant combinations, among other useful features. We note
that these are somewhat analogical to an “additive" version of “source
factors" approaches Hoang et al. (2016); Sennrich and Haddow (2016) with one
major difference: additive interventions happen after the encoder rather than
before the encoder.

While the original work only introduces the interventions to the top-most
decoder layers in order to allow for partially freezing pretrained networks,
we simplify by applying the intervention to the top layer of the encoder, such
that it affects all decoder layers. Further, the authors report that improved
general performance can be promoted by randomly inducing a zero-vector
intervention. As such, we can specify that $t$ is randomly replaced by
$\langle$pad$\rangle$ with some probability with the same effect. We report
20% masking in this paper, though we experiment with 0% masking and find no
significant differences between the two.

##  3 Experimental Setup

Source | Parallel sents (k) | Source tokens (m)
---|---|---
ParaCrawl | 229,340 | 4,190.0
BANK | 190 | 6.3
IT | 270 | 3.6
LAW | 501 | 17.1
TALK | 160 | 3.6
RELIG | 130 | 3.2
MED | 2,609 | 133.0
NEWS | 254 | 5.6

Table 1: Effective training set sizes

Method | BANK | IT | LAW | TALK | RELIG | MED | WMT15
---|---|---|---|---|---|---|---
BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET
general base | 42.4 | 0.485 | 38.3 | 0.311 | 56.2 | 0.832 | 40.6 | 0.585 | 18.9 | 0.166 | 43.9 | 0.548 | 41.3 | 0.639
combined base | 52.1 | 0.559 | 45.6 | 0.528 | 59.8 | 0.855 | 41.5 | 0.614 | 27.8 | 0.284 | 49.8 | 0.651 | 41.7 | 0.633
combined ints | 51.9 | 0.573 | 44.7 | 0.512 | 59.9 | 0.859 | 41.3 | 0.610 | 27.6 | 0.268 | 50.1 | 0.647 | 41.6 | 0.638
combined tags | 52.0 | 0.546 | 46.5 | 0.492 | 59.8 | 0.856 | 43.7 | 0.647 | 28.8 | 0.307 | 50.1 | 0.647 | 36.8 | 0.606
in-dom ints | 58.5 | 0.615 | 51.9 | 0.615 | 66.6 | 0.891 | 39.2 | 0.494 | 88.7 | 0.872 | 55.4 | 0.695 | 30.1 | 0.289
in-dom tags | 58.7 | 0.611 | 51.1 | 0.599 | 66.4 | 0.893 | 39.8 | 0.531 | 89.5 | 0.893 | 55.4 | 0.685 | 26.8 | 0.243
multi-dom FT ints | 56.1 | 0.604 | 50.6 | 0.605 | 64.9 | 0.896 | 41.3 | 0.580 | 79.4 | 0.791 | 51.6 | 0.671 | 34.3 | 0.433
multi-dom FT tags | 56.9 | 0.614 | 50.9 | 0.595 | 64.8 | 0.870 | 41.6 | 0.605 | 83.6 | 0.850 | 51.9 | 0.673 | 33.4 | 0.439
single-dom FT | 58.2 | 0.637 | 50.8 | 0.629 | 67.0 | 0.917 | 45.1 | 0.653 | 39.0 | 0.402 | 52.6 | 0.679 | $-$ | $-$

Table 2: MT quality scores per test set. Statistically significant differences
between tags and ints at the 95% confidence interval with 1000 bootstrapped
samples bolded.

BANKITLAWTALKRELIGMEDWMT15$0$$20$$40$$60$$80$$100$COMET ($\times 100$)COMET by
Domain and Approachgeneral basecombined intscombined tagscombined basein-dom
intsin-dom tagsmulti-dom FT intsmulti-dom FT tagssingle-dom FT

Figure 1: COMET scores ($\times 100$) by domain and approach

###  3.1 Data

We follow the supervised data settings prescribed by Pham et al. (2021) which
includes splits from seven domains of varying disparity: BANK, IT, LAW, TALK,
RELIG, MED, and NEWS. These domains are drawn from various sources: the
European Central Bank corpus (BANK) Tiedemann (2012); the documentation for
the KDE, Ubuntu, GNOME, and PHP projects from Opus Tiedemann (2009) combined
to form IT; The JRC-Acquis corpus (LAW) Steinberger et al. (2006); TED Talks
(TALK) Cettolo et al. (2012); the Tanzil translation of the Koran (RELIG); the
UFAL Medical corpus v1.0 (MED)111https://ufal.mff.cuni.cz/ufal_medical_corpus;
and News Commentary corpus v12 (NEWS) Tiedemann (2012). For sake of
consistency, we rely on roughly the same splits as provided by the
authors,222https://github.com/qmpham/experiments, though we remove duplicates
within each domain, which changes the size of each training set slightly.
Additionally we include English-French ParaCrawl v9 Bañón et al. (2020) to
serve as a large out-of-domain training set for some experimental settings.
The effective training set sizes are summarized in Table 1.

###  3.2 Models

We consider several models falling into two categories: those trained with
(control) and without(no control) a method for selecting the target domain.

We use approximately the same architecture for all settings, though note that
all intervention-based models have an extra embedding layer with the same
embedding dimension as the encoder333Adding $|D|\times 1024$ parameters, where
$D$ is the set of domain labels. The basic architecture follows a 12-layer
encoder, 6-layer decoder transformer with 8 attention heads each Vaswani et
al. (2017), encoder and decoder feedforward embedding dimensions of 4096, and
encoder and decoder embedding dimensions of 1024.

####  3.2.1 no control

We train three models with no training-time information about the domain that
the data comes from and, as a consequence, have no ability to explicitly
control the target domain:

  1. 1.

we have an out-of-domain baseline which is trained only on ParaCrawl: general
base.

  2. 2.

we have a model which is trained on the in-domain plus out-of-domain training
sets: combined base.

  3. 3.

we have six quasi-oracle fine-tuned models which are produced by fine-tuning
the general base model on each target domain’s training set; we collectively
refer to this set of models as single-domain fine-tuned (single-dom FT).

####  3.2.2 control

As mechanisms for controlling the target domain we consider:

  1. 1.

prepending the domain tag to the source sequence, tags

  2. 2.

additive interventions with 20% masking, ints

We apply these two methods to three settings:

  1. 1.

an in-domain plus out-of-domain setting, combined

  2. 2.

an in-domain-only setting, in-dom

  3. 3.

a multi-domain fine-tuning setting, multi-dom FT, where general base is fine-
tuned on all in-domain data with domain information available at training
time.

This results in six models:

  * •

combined ints

  * •

in-dom ints

  * •

multi-dom FT ints

  * •

combined tags

  * •

in-dom tags

  * •

multi-dom FT tags.

###  3.3 Training

We train a joint unigram segmentation model Kudo (2018) using SentencePiece
Kudo and Richardson (2018) with a vocabulary of size 32k for each setting in
general base, combined, and in-dom (reusing general base’s model for multi-dom
FT and single-dom FT). We train each model by sampling 10M sentences randomly,
splitting on digits and enabling byte-fallback. We add a special token for
each domain for which we have splits: $\langle$BANK$\rangle$,
$\langle$IT$\rangle$, $\langle$LAW$\rangle$, $\langle$TALK$\rangle$,
$\langle$RELIG$\rangle$, $\langle$MED$\rangle$, and $\langle$NEWS$\rangle$. We
use these models to segment the data as appropriate in each setting.

We use dropout of 0.1 but disable attention dropout and ReLU dropout. We
optimize label smoothed cross-entropy loss with a label smoothing factor of
0.1 Szegedy et al. (2016) using Adam Kingma and Ba (2015). All models are
built and trained using fairseq Ott et al. (2019).

For models trained with out-of-domain data, we shard the effective dataset
with each shard containing approximately 1b target tokens. For models trained
with in-domain data only, we consider the entire combined in-domain dataset to
be a single shard. We train for 30 virtual epochs, where a virtual epoch is
defined as a single pass over one shard. For models which are fine-tuned, we
fine-tune for 10 additional virtual epochs.

Each in-domain training set is assigned a unique special token which is
included in the vocabulary and examples drawn from these in-domain training
sets are provided the associated special token at training time. Examples from
ParaCrawl are assigned no special domain token (i.e., no token is prepended in
tags models and $\langle$pad$\rangle$ is always provided in ints models).

###  3.4 Evaluation

We evaluate in three settings to probe various aspects of MT quality:

  * •

we evaluate in-domain performance with each model from control and no control
to determine the relative effectiveness of the methods of control against
methods without control.

  * •

we evaluate on the WMT15 English-French test set Bojar et al. (2015) with no
domain label provided (i.e., as if the models were in the no control setting)
to test catastrophic forgetting (Goodfellow et al., 2013) in a general
setting. Importantly, while the models trained on in-domain data have been
exposed to newswire data, the labels are not provided at test time in this
setting.

  * •

we evaluate the effect of providing the incorrect tag to each test set, as
computed by SacreBLEU Post (2018) and COMET Rei et al. (2020), to test the
resilience of models to label errors

##  4 Results

combined ints combined tags in-dom ints in-dom tags multi-dom FT ints multi-
dom FT tags $0$$10$$20$$30$$40$$50$$60$Standard Deviation $\times 100$
(COMET)Standard Deviation ($\times 100$) of COMET across all Domain Labels by
Domain and ApproachBANKITLAWTALKRELIGMED

Figure 2: Impact of domain label error on COMET per test set and approach
Figure 3: COMET of combined models under various domain labels. ints left,
tags right. ints maintain high quality translations under mismatching domain
labels in all cases, unlike tags. Figure 4: COMET of in-dom models under
various domain labels. ints left, tags right. ints maintain high quality
translations under mismatching domain labels in all cases, unlike tags. Figure
5: COMET of multi-dom FT models under various domain labels. ints left, tags
right. ints maintain high quality translations under mismatching domain labels
in all cases, unlike tags.

##### No clear winner in ideal case

We evaluate the setting in which the provided domain label matches the target
test domain, and the setting of WMT15 without a provided domain label, for
each setting apart from single-dom FT. The results can be read in Table 2 and
are visualized in Figure 1.

Table 2 shows that when comparing control models within a training setting
using bootstrap resampling (sample sizes of 1000) Koehn (2004), the difference
in performance of tags and ints are insignificant in the majority of cases.
While there are a few cases of statistically significant differences, neither
tags nor ints are uniformly preferred in these cases. The opposite is observed
on the out-of-domain WMT15, where ints performs uniformly better than tags,
often significantly.

We observe that methods with control in the combined setting perform
approximately equally to the combined base, showing that naive combination of
in-domain and out-of-domain with a mechanism to control the domain does not
improve over approaches without control, though in-dom and multi-dom FT models
tend to perform better on average than any model in the combined setting.

#####  ints are robust under domain label mismatch

Next, we perform an ablation study in which we score each test across all
domain label assignments (including the correct label and no label), which
allows us to observe the effects of test-time labeling error. While we compute
both BLEU and COMET, we include only COMET here.444Similar results for BLEU
are listed in Appendix A.2 We include the full results in Tables 3–8, but
summarize the findings in Figures 2-5, which show the robustness of various
models and settings to mislabeled domains.

Figures 3–5 show heatmaps resulting from this ablation, but we refer
interested readers to Tables 3–8 for the long-form charts. We see that tags
systems’ performances vary dramatically, incurring severe degradation in the
face of domain label error but performing strongest along the diagonal. ints
systems, on the other hand, see only small performance changes when provided
with incorrect domain labels and roughly equal performance under all possible
labels, as observed in Figure 2. We see that in-dom tags have the highest
average variation in performance, likely owing to the small amount of data
which suggests that in-dom tags overfits to the training data. The variation
in performance of ints systems approaches that of the general base, which by
definition ignores the domain label and therefore has 0 variance; however,
ints has demonstrably stronger performance than general base in all domains
and, indeed, stronger performance than tags in a handful of domains and thus
seems to learn strong general representations for translation which
disentangles the representations of the encoder from the representations of
the attribute.

Additionally, through manual analysis we find that tags systems are more prone
to hallucinating translation artifacts from the corpus associated with the
domain label being used, often causing quality degradation. We refer to Table
15 for an example of such artifacts, which includes topical and target
language mismatches along with tokens which appear as a result of the HTML-
encoded nature of the $\langle$IT$\rangle$ dataset.555Escaping seems to be an
artifact of Moses preprocessing leakage of raw data; not germane to all
domains in this work.

##### Single-domain fine-tuning is not as competitive in large-data settings

We compare the performance of models trained only with in-domain data and out-
of-domain data. From Table 2, we see slightly stronger in-domain performance
for in-dom models as compared to models fine-tuned with out-of-domain data at
the cost of out-of-domain performance on WMT15, suggesting that multi-dom FT
models generalize better and may surpass in-dom models with more training due
to the relatively little fine-tuning budget of 10 epochs afforded to them
comparatively.

Finally, we see that while single-dom FT is typically among the highest
performing systems for a given test set, it is never unmatched by an
alternative system in control. We observe that single-dom FT is uniformly
stronger than general base and combined, in-dom and multi-dom FT show
competitive in-domain performance. We note that because there is one single-
dom FT model per test set, the effective parameter budget is six times larger
than any of the individual models, providing support for both its
impracticality and untenability as compared to any other setting. This
suggests that single-domain fine-tuning is not as effective as expected in
high-resource settings as a strong upper-bound in MDMT.

##  5 Related Work

Incorporating extra-sentential information has a rich history in NMT. Aside
from controlling for the domain, Sennrich et al. (2016) use a politeness tag
at training and inference time to accommodate coarse politeness control in
machine translation. Additionally, Kuczmarski and Johnson (2018) use tags to
afford users the ability to vary binary gender in the translations of gender-
neutral inputs, hoping to address gender bias in MT.

At the sub-sequence level, Hoang et al. (2016) and Sennrich and Haddow (2016)
included linguistically-informed word-level “source factors”, such as part-of-
speech tags and dependency relations, as additional feature factors to be
concatenated to form a full encoder representation with the goal of reducing
ambiguity and sparseness issues.

Perhaps more relatedly, several works have explored the impacts of
incorporating domain information into training using various methods. Kobus et
al. (2017) explore two methods: a tag-based approach which concatenates a
special token to the end of the source sequence, and a “source factors”-style
approach which concatenates domain-level embeddings to each token embedding in
the source. Sharaf et al. (2020) explore few-shot domain adaptation, rather
than domain control, through the lens of meta-learning and show that a meta-
learning based approach is generally stronger than other adaptation
approaches, though we note that adaptation and control address different
needs. Finally, Stojanovski and Fraser (2021) frame machine translation with
document-context as an unsupervised domain adaptation problem and incorporate
domain embeddings within the encoder, summed with positional and word
embeddings, yielding strong improvements over competitive baseline models.

##  6 Conclusion

In this work we examined the relative impact of additive interventions in a
large-scale MDMT setting. We find that typically there are no significant
differences between additive interventions and tag-based approaches when the
provided domain label matches the test set, but find that additive
interventions exhibit much more desirable degradation properties when the
domain label is unknown or incorrectly provided. In addition, we find that
models first trained on a large, general corpus and then fine-tuned on a
single-domain—a realistic baseline in machine translation—rarely perform
significantly better than approaches which are trained or fine-tuned only on
in-domain data, which is in contrast to their generally superior performance
in low-resource settings.

In future work we consider developing extensions to additive interventions
which can further improve their performance in MDMT settings. Additionally,
studying additive interventions in other tasks where tag-based approaches are
dominant, such as multi-lingual machine translation, could be an interesting
avenue for exploration.

## References

  * Bañón et al. (2020) Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020.  ParaCrawl: Web-scale acquisition of parallel corpora.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4555–4567, Online. Association for Computational Linguistics.
  * Bojar et al. (2015) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015.  Findings of the 2015 workshop on statistical machine translation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 1–46, Lisbon, Portugal. Association for Computational Linguistics.
  * Cettolo et al. (2012) Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012.  WIT3: Web inventory of transcribed and translated talks.  In _Proceedings of the 16th Annual conference of the European Association for Machine Translation_ , pages 261–268, Trento, Italy. European Association for Machine Translation.
  * Goodfellow et al. (2013) Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013\.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.
  * Hoang et al. (2016) Cong Duy Vu Hoang, Gholamreza Haffari, and Trevor Cohn. 2016.  Improving neural translation models with linguistic factors.  In _Proceedings of the Australasian Language Technology Association Workshop 2016_ , pages 7–14, Melbourne, Australia.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Kobus et al. (2017) Catherine Kobus, Josep Crego, and Jean Senellart. 2017.  Domain control for neural machine translation.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 372–378, Varna, Bulgaria. INCOMA Ltd.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  * Kuczmarski and Johnson (2018) James Kuczmarski and Melvin Johnson. 2018.  Gender-aware natural language translation.  _Technical Disclosure Commons, (October 08, 2018)_.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Pham et al. (2021) MinhQuang Pham, Josep Maria Crego, and François Yvon. 2021.  Revisiting multi-domain machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:17–35.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Schioppa et al. (2021) Andrea Schioppa, David Vilar, Artem Sokolov, and Katja Filippova. 2021.  Controlling machine translation for multiple attributes with additive interventions.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6676–6696, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sennrich and Haddow (2016) Rico Sennrich and Barry Haddow. 2016.  Linguistic input features improve neural machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers_ , pages 83–91, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Controlling politeness in neural machine translation via side constraints.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 35–40, San Diego, California. Association for Computational Linguistics.
  * Sharaf et al. (2020) Amr Sharaf, Hany Hassan, and Hal Daumé III. 2020.  Meta-learning for few-shot NMT adaptation.  In _Proceedings of the Fourth Workshop on Neural Generation and Translation_ , pages 43–53, Online. Association for Computational Linguistics.
  * Steinberger et al. (2006) Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaž Erjavec, Dan Tufiş, and Dániel Varga. 2006.  The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)_ , Genoa, Italy. European Language Resources Association (ELRA).
  * Stojanovski and Fraser (2021) Dario Stojanovski and Alexander Fraser. 2021.  Addressing zero-resource domains using document-level context in neural machine translation.  In _Proceedings of the Second Workshop on Domain Adaptation for NLP_ , pages 80–93, Kyiv, Ukraine. Association for Computational Linguistics.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2818–2826.
  * Tiedemann (2009) Jörg Tiedemann. 2009.  News from opus — a collection of multilingual parallel corpora with tools and interfaces.  _Advances in Natural Language Processing_ , pages 237–248.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.

##  Appendix A Raw scores

###  A.1 Ablation (COMET)

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 0.573 | 0.566 | 0.570 | 0.570 | 0.570 | 0.569 | 0.561 | 0.569
IT | 0.510 | 0.512 | 0.512 | 0.512 | 0.512 | 0.514 | 0.507 | 0.509
LAW | 0.858 | 0.859 | 0.859 | 0.857 | 0.857 | 0.861 | 0.856 | 0.859
TALK | 0.611 | 0.610 | 0.611 | 0.610 | 0.611 | 0.610 | 0.607 | 0.611
RELIG | 0.269 | 0.270 | 0.274 | 0.273 | 0.268 | 0.276 | 0.269 | 0.274
MED | 0.648 | 0.646 | 0.647 | 0.646 | 0.649 | 0.647 | 0.648 | 0.648

Table 3: COMET scores of combined ints under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 0.546 | 0.489 | 0.476 | 0.484 | 0.381 | 0.511 | -0.114 | 0.513
IT | -0.111 | 0.492 | 0.310 | 0.398 | -0.065 | 0.367 | -0.715 | 0.374
LAW | 0.606 | 0.791 | 0.856 | 0.785 | 0.699 | 0.815 | 0.126 | 0.829
TALK | 0.237 | 0.547 | 0.568 | 0.647 | 0.364 | 0.576 | -0.150 | 0.572
RELIG | 0.112 | 0.194 | 0.238 | 0.139 | 0.307 | 0.132 | -0.215 | 0.209
MED | 0.359 | 0.598 | 0.597 | 0.591 | 0.472 | 0.647 | 0.214 | 0.607

Table 4: COMET scores of combined tags under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 0.615 | 0.616 | 0.620 | 0.623 | 0.621 | 0.621 | 0.613 | 0.620
IT | 0.615 | 0.615 | 0.610 | 0.609 | 0.615 | 0.613 | 0.610 | 0.610
LAW | 0.889 | 0.891 | 0.891 | 0.889 | 0.889 | 0.890 | 0.891 | 0.890
TALK | 0.494 | 0.494 | 0.495 | 0.494 | 0.490 | 0.498 | 0.474 | 0.496
RELIG | 0.879 | 0.883 | 0.875 | 0.870 | 0.872 | 0.876 | 0.890 | 0.878
MED | 0.685 | 0.694 | 0.695 | 0.696 | 0.695 | 0.695 | 0.692 | 0.696

Table 5: COMET scores of in-dom ints under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 0.611 | -0.089 | -0.001 | -0.074 | -1.448 | -0.009 | -0.289 | -0.073
IT | -0.625 | 0.599 | -0.557 | -0.539 | -1.520 | -0.527 | -1.043 | -0.550
LAW | 0.193 | 0.255 | 0.893 | 0.273 | -1.226 | 0.368 | 0.104 | 0.282
TALK | -0.443 | -0.334 | -0.292 | 0.531 | -1.430 | -0.247 | -0.444 | -0.287
RELIG | -0.958 | -0.977 | -0.820 | -0.801 | 0.893 | -0.796 | -0.941 | -0.872
MED | -0.150 | -0.062 | 0.052 | 0.006 | -1.443 | 0.685 | -0.223 | 0.017

Table 6: COMET scores of in-dom tags under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 0.604 | 0.611 | 0.611 | 0.611 | 0.610 | 0.610 | 0.610 | 0.611
IT | 0.609 | 0.605 | 0.607 | 0.608 | 0.609 | 0.610 | 0.610 | 0.609
LAW | 0.896 | 0.897 | 0.896 | 0.896 | 0.896 | 0.896 | 0.896 | 0.896
TALK | 0.580 | 0.576 | 0.577 | 0.580 | 0.577 | 0.577 | 0.578 | 0.578
RELIG | 0.816 | 0.819 | 0.817 | 0.816 | 0.791 | 0.820 | 0.816 | 0.817
MED | 0.677 | 0.675 | 0.677 | 0.676 | 0.675 | 0.671 | 0.677 | 0.676

Table 7: COMET scores of multi-dom FT ints under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 0.614 | 0.576 | 0.580 | 0.573 | 0.231 | 0.578 | 0.293 | 0.505
IT | 0.369 | 0.595 | 0.465 | 0.486 | -0.773 | 0.496 | -0.372 | 0.435
LAW | 0.681 | 0.832 | 0.870 | 0.810 | 0.468 | 0.867 | 0.620 | 0.657
TALK | 0.206 | 0.491 | 0.522 | 0.605 | -0.965 | 0.514 | 0.061 | 0.504
RELIG | 0.084 | 0.198 | 0.449 | 0.180 | 0.850 | 0.330 | -0.162 | 0.313
MED | 0.538 | 0.637 | 0.671 | 0.638 | 0.436 | 0.673 | 0.494 | 0.609

Table 8: COMET scores of multi-dom FT tags under various domain labels

###  A.2 Ablation (BLEU)

All scores reported are from
SacreBLEU666BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.2.0
Post (2018).

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 51.9 | 51.7 | 51.9 | 51.9 | 51.9 | 51.8 | 51.8 | 51.9
IT | 44.6 | 44.7 | 44.8 | 44.8 | 44.6 | 44.7 | 44.7 | 44.6
LAW | 59.8 | 59.8 | 59.9 | 59.8 | 59.7 | 59.8 | 59.7 | 59.9
TALK | 41.3 | 41.3 | 41.4 | 41.3 | 41.4 | 41.3 | 41.1 | 41.5
RELIG | 27.6 | 27.8 | 27.7 | 27.8 | 27.6 | 27.9 | 27.5 | 27.7
MED | 50.0 | 50.0 | 50.0 | 50.0 | 49.9 | 50.1 | 50.0 | 50.0

Table 9: BLEU scores of combined ints under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 52.0 | 43.5 | 43.0 | 40.4 | 39.0 | 45.2 | 30.2 | 44.2
IT | 18.5 | 46.5 | 36.3 | 39.9 | 26.5 | 37.2 | 11.0 | 35.0
LAW | 50.2 | 56.4 | 59.8 | 50.7 | 51.4 | 55.5 | 36.9 | 56.2
TALK | 29.5 | 39.2 | 38.1 | 43.7 | 28.3 | 39.7 | 22.7 | 37.1
RELIG | 21.6 | 24.4 | 25.5 | 16.3 | 28.8 | 18.9 | 14.5 | 22.6
MED | 43.5 | 48.5 | 48.3 | 47.3 | 45.0 | 50.1 | 41.6 | 49.1

Table 10: BLEU scores of combined tags under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 58.5 | 58.6 | 58.6 | 58.6 | 58.5 | 58.8 | 58.2 | 58.7
IT | 52.0 | 51.9 | 51.4 | 51.4 | 51.8 | 51.6 | 51.4 | 51.8
LAW | 66.1 | 66.2 | 66.1 | 66.0 | 65.9 | 66.1 | 66.0 | 66.1
TALK | 39.0 | 39.1 | 39.1 | 39.2 | 39.1 | 39.2 | 38.8 | 39.0
RELIG | 89.2 | 89.2 | 89.0 | 88.7 | 88.7 | 89.2 | 89.3 | 89.1
MED | 55.4 | 55.5 | 55.3 | 55.4 | 55.4 | 55.4 | 55.4 | 55.5

Table 11: BLEU scores of in-dom ints under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 58.7 | 31.2 | 36.0 | 34.3 | 3.9 | 36.1 | 27.3 | 34.4
IT | 15.5 | 51.1 | 16.6 | 20.0 | 0.4 | 18.8 | 5.9 | 15.9
LAW | 42.2 | 43.5 | 66.4 | 45.3 | 12.4 | 48.2 | 40.2 | 44.7
TALK | 18.6 | 21.0 | 20.7 | 39.8 | 1.0 | 23.8 | 17.2 | 21.5
RELIG | 6.2 | 6.1 | 8.2 | 8.7 | 89.5 | 9.0 | 5.5 | 7.6
MED | 32.2 | 33.2 | 32.8 | 33.3 | 5.5 | 55.4 | 29.5 | 33.5

Table 12: BLEU scores of in-dom tags under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 56.1 | 55.9 | 56.5 | 56 | 56.1 | 56.4 | 55.3 | 56.3
IT | 50.6 | 50.6 | 50.0 | 50.4 | 50.3 | 50.6 | 49.8 | 50.9
LAW | 64.8 | 64.7 | 64.9 | 64.9 | 64.8 | 65.2 | 64.5 | 65.0
TALK | 41.2 | 40.8 | 41.1 | 41.3 | 41.3 | 41.2 | 40.4 | 41.5
RELIG | 80.4 | 81.1 | 80.5 | 80.2 | 79.4 | 81.8 | 79.3 | 82.2
MED | 51.7 | 51.3 | 51.6 | 51.7 | 51.7 | 51.6 | 51.3 | 51.7

Table 13: BLEU scores of multi-dom FT ints under various domain labels

Test set Provided label |  $\langle$BANK$\rangle$ |  $\langle$IT$\rangle$ |  $\langle$LAW$\rangle$ |  $\langle$TALK$\rangle$ |  $\langle$RELIG$\rangle$ |  $\langle$MED$\rangle$ |  $\langle$NEWS$\rangle$ | None
---|---|---|---|---|---|---|---|---
BANK | 56.9 | 54.5 | 54.4 | 52.0 | 49.6 | 55.0 | 43.4 | 54.9
IT | 43.1 | 50.9 | 47.4 | 46.9 | 28.0 | 46.9 | 17.3 | 40.8
LAW | 55.7 | 63.7 | 64.8 | 61.2 | 59.4 | 64.2 | 55.9 | 60.3
TALK | 28.0 | 37.4 | 36.1 | 41.6 | 8.4 | 36.1 | 23.1 | 36.2
RELIG | 32.6 | 38.6 | 61.9 | 22.9 | 83.6 | 50.7 | 19.2 | 49.1
MED | 49.6 | 51.4 | 51.8 | 50.4 | 49.4 | 51.9 | 49.7 | 51.2

Table 14: BLEU scores of multi-dom FT tags under various domain labels

##  Appendix B Figures (BLEU)

BANKITLAWTALKRELIGMEDWMT15$0$$20$$40$$60$$80$BLEUBLEU by Domain and
Approachgeneral basecombined intscombined tagscombined basein-dom intsin-dom
tagsmulti-dom FT intsmulti-dom FT tagssingle-dom FT

Figure 6: BLEU scores by domain and approach

combined ints combined tags in-dom ints in-dom tags multi-dom FT ints multi-
dom FT tags $0$$5$$10$$15$$20$$25$$30$Standard Deviation (BLEU)Standard
Deviation of BLEU across all Domain Labels by Domain and
ApproachBANKITLAWTALKRELIGMED

Figure 7: Impact of domain label error on BLEU per test set and approach

##  Appendix C Examples

Src | Never; soon they will deny ever worshipping them, and will turn into their opponents. |
---|---|---
Ref | Bien au contraire! [ces divinités] renieront leur adoration et seront pour eux des adversaires. |
multi-dom FT ints | Bien au contraire! [ces divinités] renieront leur adoration et seront pour eux des adversaires. |
multi-dom FT tags | You are about to translate the ’None ’COMMAND, there are |
| some rules on how to translate it. Please see http: / / / /www.mysql.com /. |
Src | And the evil-doers say: Ye are but following a man bewitched. |
Ref | Les injustes disent: «Vous ne suivez qu’un homme ensorcelé». |
in-dom ints | Les injustes disent: «Vous ne suivez qu’un homme ensorcelé». |
in-dom tags | Et les « & #160; diaboliques & #160; » disent & #160;: « & #160; fired & #160; » |
| est le suivant d’un homme. |

Table 15: Example translation artifacts from incorrect domain label; a
translation of $\langle$RELIG$\rangle$ sentences with $\langle$IT$\rangle$
domain label under different models. We note that the HTML-encoded artifact “&
#160;" appears with high frequency in $\langle$IT$\rangle$.


# An Overview of Uncertainty Quantification Methods
for Infinite Neural Networks

Florian Juengermann  Maxime Laasri  Marius Merkle  Weiwei Pan

###### Abstract

To better understand the theoretical behavior of large neural networks,
several works have analyzed the case where a network’s width tends to
infinity. In this regime, the effect of random initialization and the process
of training a neural network can be formally expressed with analytical tools
like Gaussian processes and neural tangent kernels. In this paper, we review
methods for quantifying uncertainty in such infinite-width neural networks and
compare their relationship to Gaussian processes in the Bayesian inference
framework. We make use of several equivalence results along the way to obtain
exact closed-form solutions for predictive uncertainty.

Machine Learning, Uncertainty, Infinite Neural Networks

_Supervised by:_

##  1 Introduction

Deep learning has brought breakthroughs across a variety of disciplines
including image classification, speech recognition, and natural language
processing [LeCun et al., 2015]. But despite their tremendous success,
artificial neural networks often tend to be overconfident in their
predictions, especially for out-of-distribution data.

Therefore, proper quantification of uncertainty is of fundamental importance
to the deep learning community. It turns out that the infinite-width limit,
i.e. the limit when the number of hidden units in a neural network approaches
infinity, simplifies the analysis considerably, and is useful to draw
conclusions for very large, but finite neural networks.

In this paper, we provide an overview of the most common techniques for
quantifying uncertainty in deep learning and explore how these techniques are
related in the infinite-width regime. In section 2, we introduce the
fundamentals of these techniques, while in section 3, we outline their
relation as well as several equivalence results.

##  2 Background

Throughout this section, we will consider a neural network with input
$\bm{x}\in\mathbb{R}^{n}$ and scalar output $y\in\mathbb{R}$. Let
$f(\bm{x},\bm{\theta})$ denote the neural network output function where the
weights and biases are summarized in the parameter vector $\bm{\theta}$. The
traditional training objective is to minimize the mean squared error (MSE)
loss $\mathcal{L}$ over a dataset
$\mathcal{D}=(\mathcal{X},\mathcal{Y})=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{N}$ of
$N$ training points, i.e. to find an optimal $\bm{\theta}^{*}$ such that:

$\bm{\theta}^{*}=\text{arg
min}_{\bm{\theta}}\underbrace{\frac{1}{N}\sum_{i=1}^{N}\left(y_{i}-f(\bm{x}_{i},\bm{\theta})\right)^{2}}_{\mathcal{L}(\mathcal{X},\mathcal{Y})}$
(1)

###  2.1 Gaussian Processes

One fundamental tool used in machine learning to quantify uncertainty is
Gaussian processes (GPs). They are primarily encountered in regression
problems, but they can be extended to classification or clustering tasks
[Kapoor et al., 2010, Kim and Lee, 2007].

Simply put, a GP works as follows: (i) for any input
$\bm{x}\in\mathbb{R}^{n}$, it yields a probability distribution for any
possible outcome $y\in\mathbb{R}$, and (ii) any finite collection of input
points follows a multivariate Gaussian distribution
$\mathcal{N}(\underline{\mu},\underline{\Sigma})$ with mean vector
$\underline{\mu}\in\mathbb{R}^{n}$ and covariance matrix
$\underline{\Sigma}\in\mathbb{R}^{n\times n}$ where one usually assumes the
data to be centered ($\underline{\mu}=0$). The matrix $\underline{\Sigma}$ is
determined by a kernel function $\Sigma$ that allows to incorporate prior
knowledge about the shape and characteristics of the data distribution. We
will denote a GP with kernel function $\Sigma$ applied on centered data as
$\mathcal{GP}(0,\Sigma)$. The most commonly used kernel function is the
squared exponential or radial basis function (RBF):

$\underline{\Sigma}_{ij}=\Sigma(\bm{x}_{i},\bm{x}_{j})=\sigma^{2}\exp\left(\frac{||\bm{x}_{i}-\bm{x}_{j}||_{2}^{2}}{2l^{2}}\right)$
(2)

where the variance $\sigma$ and length $l$ serve as hyperparameters to encode
domain knowledge. For a new test point $\bm{x}$, the prior distribution over
$y$ is given by:

$y\sim\mathcal{N}\left(0,\Sigma(\bm{x},\bm{x})\right)$ (3)

Then, the dataset $\mathcal{D}$ can be used as part of the Bayesian framework
to update prior beliefs and obtain a posterior distribution $P$ by
conditioning on the training points, as follows:

$\begin{split}P(&y|\bm{x},\mathcal{X},\mathcal{Y})=\mathcal{N}\big{(}\Sigma(\bm{x},\mathcal{X})\Sigma(\mathcal{X},\mathcal{X})^{-1}\mathcal{Y},\\\
&\Sigma(\bm{x},\bm{x})-\Sigma(\bm{x},\mathcal{X})\Sigma(\mathcal{X},\mathcal{X})^{-1}\Sigma(X,\bm{x})\big{)}\end{split}$
(4)

###  2.2 Frequentist versus Bayesian Approach

The most common tools used to quantify uncertainty specifically in neural
networks come from the distinction between two fundamental approaches. On the
one hand, we can use the variance of predictions of multiple neural networks
that follow from different random initializations. We refer to this
frequentist type of ensemble method as deep ensemble (DE). On the other hand,
a Bayesian neural network (BNN) is the natural result of applying Bayesian
inference to neural networks. In a BNN, we place priors – typically
independent Gaussians – on the network weights and biases. As a result, the
posterior predictive distribution gives a prediction interval instead of a
single-point estimate.

BNNs used in practice have posterior distributions that are highly complex and
multi-modal, challenging even the most sophisticated Markov chain Monte Carlo
(MCMC) sampling methods. For that reason, practitioners oftentimes use mean-
field variational inference (MFVI) to approximate the posterior with a
factorized Gaussian, i.e. assume the posterior distributions over the
network’s parameters to be independent.

###  2.3 Neural Linear Models

An alternative to MFVI that scales better to large datasets than BNNs and GPs
are neural linear models (NLMs).

First introduced by Lázaro-Gredilla and Figueiras-Vidal [2010] as Marginalized
Neural Networks and later re-introduced by Snoek et al. [2015], NLMs consist
of a Bayesian linear regression performed on the last layer of a deep neural
network. They resemble BNNs in structure, except only the final linear layer
has priors on its coefficients. An example is shown in Figure 1.

[height=4, layerspacing=15mm] [count=4, bias=false, title=Input
Layer, text=$x_{\hiddenlayer}$[count=4, bias=false,
text=$\bm{\theta}^{(1)}_{\linklayers}$[count=4, bias=false, title=Hidden
Layers, text=$\bm{\theta}^{(2)}_{\linklayers}$[count=4, bias=false,
text=$\bm{\theta}^{(3)}_{\linklayers}$[count=1, bias=false, title= Bayesian
Output
Layer, text=$\mathbf{w}$

Figure 1: Example of a neural linear model. Hidden layer $l$ has vector
weights $\bm{\theta}_{i}^{(l)}$, while the output layer performs Bayesian
linear regression on the weights $\mathbf{w}$.

The first layers whose weights $\bm{\theta}$ are simple scalars can be viewed
as a feature map $\Phi_{\bm{\theta}}$ that transforms input $\bm{x}$ into
features $\Phi_{\bm{\theta}}(\bm{x})$, which are then fed into the Bayesian
linear regression. Because of this particular structure, training and
inference with NLMs differs from usual procedures and usually consist of two
steps: training the deterministic layers first using a point estimate of the
output layer weights, and computing the posterior of that Bayesian layer
second. In [Thakur et al., 2021], the authors propose an instance of this
procedure that has the benefit of not underestimating uncertainty in data-
scarce regions.

When all priors on the output weights $\mathbf{w}$ are independent and
identical Gaussian distributions, the posterior is a multivariate Gaussian
whose mean $\underline{\mu}_{NLM}$ and covariance matrix
$\underline{\Sigma}_{NLM}$ can be expressed in closed-form:

$\displaystyle{\underline{\mu}}_{NLM}$
$\displaystyle=\alpha\left(\alpha{\Phi_{\bm{\theta}}(\mathcal{X})}^{\top}\Phi_{\bm{\theta}}(\mathcal{X})+\beta
I\right)^{-1}{\Phi_{\bm{\theta}}(\mathcal{X})}^{\top}\mathcal{Y}$ (5)
$\displaystyle{\underline{\Sigma}}_{NLM}$
$\displaystyle=\left(\alpha{\Phi_{\bm{\theta}}(\mathcal{X})}^{\top}\Phi_{\bm{\theta}}(\mathcal{X})+\beta
I\right)^{-1}$ (6)

where $I$ is the identity matrix, and
$\alpha=\frac{1}{{\sigma_{\epsilon}}^{2}}$ and
$\beta=\frac{1}{{\sigma_{\mathbf{w}}}^{2}}$ are prior hyperparameters equal to
the inverse of the variance of the regression noise and weight variables,
respectively. These expressions reveal that the main computational effort of
inference with NLMs lies in the inversion of the matrix
$\alpha{\Phi_{\bm{\theta}}(\mathcal{X})}^{T}\Phi_{\bm{\theta}}(\mathcal{X})+\beta
I$, whose dimension is the number of neurons in the last hidden layer.
Compared with GPs who scale with the size of the training data set, as
explained later in subsubsection 3.2.1, NLMs scale with the width of the
neural network’s last layer, which is often of lower dimensionality. This is
where NLMs gain their advantage in tractability and computational efficiency,
which is why NLMs have gained popularity.

###  2.4 Neural Tangent Kernel

The last piece of background to understand is the theory describing the
training process of a neural network with gradient descent. A helpful tool for
that is the neural tangent kernel (NTK) introduced by Jacot et al. [2018]:

$\hat{\Theta}_{t}(\bm{x},\bm{x^{\prime}})=\nabla_{\bm{\theta}}f(\bm{x},\bm{\theta_{t}})^{\top}\nabla_{\bm{\theta}}f(\bm{x^{\prime}},\bm{\theta_{t}})$
(7)

In general, this kernel function depends on the random initialization
$\bm{\theta_{0}}$ and changes over time. However, Jacot et al. [2018] show
that in the infinite-width limit, the NTK at initialization $\hat{\Theta}_{0}$
becomes deterministic and only depends on the activation function and network
architecture. In this case, the NTK also stays constant during training for
small enough learning rates: $\hat{\Theta}_{t}=\hat{\Theta}_{0}$. We call this
deterministic and constant kernel $\Theta$. Yang [2020] extends these
convergence results from multilayer perceptrons (MLPs) to other architectures
including convolutional and batch-normalization layers.

Now, we investigate how the NTK describes change in the output function of a
NN during training. Jacot et al. [2018] and Lee et al. [2019] show that for
small enough learning rates, the NN prediction
$f_{t}(\bm{x}):=f(\bm{x},\bm{\theta}_{t})$ at training time $t$ follows an
ordinary differential equation (ODE) that is characterized by the NTK:

$\frac{\text{d}f_{t}(\bm{x})}{\text{d}t}=-\nu\hat{\Theta}_{t}(\bm{x},\mathcal{X})\nabla_{f_{t}(\mathcal{X})}\mathcal{L}$
(8)

Here, $\nu$ denotes the learning rate and
$\nabla_{f_{t}(\mathcal{X})}\mathcal{L}$ describes the gradient of the loss
with respect to model’s output on the training set. Using an MSE loss for
$\mathcal{L}$ and the fact that the NTK stays constant over time, Equation 8
simplifies to a linear ordinary differential equation with known exponentially
decaying solution. So, the output for the training dataset at time $t$ is

$f_{t}(\mathcal{X})=\mathcal{Y}+\left(f_{0}(\mathcal{X})-\mathcal{Y}\right)e^{-\nu\Theta(\mathcal{X},\mathcal{X})t}$
(9)

This shows that the NN achieves zero training loss because the NTK is a
positive definite matrix, hence the exponential term converges to zero.

Equivalently, based on the results of [Lee et al., 2019], He et al. [2020]
show that $f_{t}$ can be expressed based on the random initialization $f_{0}$.
For $t\rightarrow\infty$ this gives:

$f_{\infty}(\bm{x})=f_{0}(\bm{x})-\Theta(\bm{x},\mathcal{X})\Theta(\mathcal{X,X})^{-1}(f_{0}(\mathcal{X})-\mathcal{Y})$
(10)

##  3 Behavior at Infinite Width

In this section, we summarize the behavior of NNs in the infinite-width limit,
i.e. we are interested in the limit as the number of hidden units goes to
infinity.

###  3.1 Priors in Infinite Neural Networks

Under the condition that the network parameters follow independent and
identical (iid) distributions with zero mean and finite variance and the
activation function of the last layer is bounded, the prior predictive
distribution converges to a GP with zero mean and finite variance [Neal,
1996]. The intuition is as follows: for a fixed input $\bm{x}$, every hidden
layer output $\phi_{i}$ is a random variable with finite mean and variance as
the activation function is bounded. When multiplied by the final layer
weights, $\phi_{i}w_{i}$ are iid random variables with zero mean and finite
variance. According to the central limit theorem, their sum converges to a
Gaussian distribution. If we rescale the variances according to the network
width, the network output becomes a standard normal distribution
$f(\bm{x})\sim\mathcal{N}(0,1)$. As this holds true for every $\bm{x}$, the
neural network is equivalent to a GP called the neural network Gaussian
process (NNGP). A sampled function from this NNGP is equivalent to the output
function of a randomly initialized NN and to a prior predictive sample of a
BNN.

While Neal [1996] gives a proof of existence, i.e. that a NN converges to a GP
in the infinite-width limit, Neal does not provide any analytic form of the
covariance function $\mathcal{K}$. Williams [1997] builds on that work and
provides closed-form kernel functions for different sets of activation
functions and prior distributions on the network parameters.

###  3.2 Bayesian Inference

So far we have not considered any data but made predictions based on randomly
drawn initial parameters. In the following sections, we discuss three
approaches to incorporate the training data. First, we update the GP prior
kernel by conditioning on the training data to obtain a GP posterior kernel.
Second, we use gradient-based methods to train the NN. Third, we make the
neural network Bayesian and approximate the BNN posterior.

####  3.2.1 Gaussian Process Regression

To go from the prior to the posterior distribution in a GP, we condition the
NNGP kernel $\mathcal{K}$ on the dataset $(\mathcal{X,Y})$. Following the
Bayesian framework, the result is the posterior GP as defined in Equation 4.

A major problem of GPs is their computational effort which is cubic in the
number of training points $\mathcal{O}(N^{3})$. Common methods have enabled
exact inference for a maximum of a few thousand training points only. This
makes GPs unsuitable for many real-world applications that require very large
datasets to be processed. Even though exact inference on GPs has recently been
scaled to a million data points by GPU parallelization [Wang et al., 2019],
NNs provide a more accessible framework in the big data regime.

####  3.2.2 Neural Network Training

NNs are trained with gradient-based optimizers such as full-batch gradient
descent or its extensions such as stochastic gradient descent and Adam
optimizer. Here, we consider two training methods with different theoretical
conclusions.

##### Weakly Trained NN

First, we only train the last layer of our NN. This means all other layers act
as random feature extractors that are unchanged during training. Lee et al.
[2019] show that for the MSE loss function and a sufficiently small learning
rate, the network output during training is an interpolation between the GP
prior and GP posterior, and asymptotically converges to the posterior of the
NNGP. This means, training only the last layer of a randomly initialized NN is
equivalent to sampling a function from the GP with the NNGP prior kernel
$\mathcal{K}$ conditioned on the dataset.

##### Fully Trained NN

If we use the gradient descent algorithm on the entire network, the NTK
$\Theta$ describes the training behavior. After convergence to zero loss,
Equation 10 describes the learned function $f_{\infty}$. As $f_{0}$ can be
regarded as a sample from the NNGP $\mathcal{GP}(0,\mathcal{K})$, we write
$f_{\infty}\sim\mathcal{GP}(\mu_{\mathrm{DE}}(\bm{x}),\Sigma_{\mathrm{DE}}(\bm{x},\bm{x}^{\prime}))$
with 111The notation $+h.c.$ means ”plus Hermitian conjugate”, like in [Lee et
al., 2019]

$\displaystyle\mu_{\mathrm{DE}}(\bm{x})$
$\displaystyle=\Theta_{\bm{x}\mathcal{X}}\Theta_{\mathcal{XX}}^{-1}\mathcal{Y}$
(11) $\displaystyle\Sigma_{\mathrm{DE}}(\bm{x},\bm{x}^{\prime})$
$\displaystyle=\mathcal{K}_{\bm{xx^{\prime}}}+\Theta_{\bm{x}\mathcal{X}}\Theta_{\mathcal{XX}}^{-1}\mathcal{K}_{\mathcal{XX}}\Theta_{\mathcal{XX}}^{-1}\Theta_{\mathcal{X}\bm{x}}$
$\displaystyle\qquad-(\Theta_{\bm{x}\mathcal{X}}\Theta_{\mathcal{XX}}^{-1}\mathcal{K}_{\mathcal{X}\bm{x}^{\prime}}+h.c.).$
(12)

where we used subscripts for the kernel arguments in interest of space. Lee et
al. [2019] observed that this deep ensemble GP does not correspond to a proper
Bayesian posterior. This means, in contrast to the weakly trained NNs, random
initialization followed by gradient descent training does not give valid
posterior predictive function samples. Hence, an ensemble of NNs does not
accurately approximate the posterior.

##### Bayesian Deep Ensemble

To address this, He et al. [2020] propose a slight modification to the
gradient descent training. With that, they arrive at the what they call neural
tangent kernel Gaussian process (NTKGP) with

$\displaystyle\mu_{\mathrm{NTKGP}}(\bm{x})$
$\displaystyle=\Theta_{\bm{x}\mathcal{X}}\Theta_{\mathcal{XX}}^{-1}\mathcal{Y}$
(13) $\displaystyle\Sigma_{\mathrm{NTKGP}}(\bm{x},\bm{x}^{\prime})$
$\displaystyle=\Theta_{\bm{xx^{\prime}}}-\Theta_{\bm{x}\mathcal{X}}\Theta_{\mathcal{XX}}^{-1}\Theta_{\mathcal{X}\bm{x}^{\prime}}$
(14)

Note that this corresponds to a GP posterior as defined in Equation 4. In
comparison to the NNGP posterior we obtain when conditioning the NNGP prior on
the data, or when only training the last layer of the NN, here, the prior
covariance function is not the NNGP kernel $\mathcal{K}$ but the NTK $\Theta$.

####  3.2.3 BNN Posterior Approximation

Instead of using an ensemble of randomly initialized NNs, we can use a prior
distribution on the network weights to obtain a BNN. However, approximation
methods struggle with accurately representing the complex BNN posterior. Coker
et al. [2021] show that in the infinite-width limit, the commonly used MFVI
approximation fails to learn the data. Specifically, the posterior predictive
mean for any input converges to zero, regardless of the input data. In their
proof, they assume that the Kullback–Leibler (KL) divergence and the $erf$
activation function are used, but give empirical evidence for $\tanh$ and
$ReLU$ activations.

##  4 Conclusion

This work provides an overview of the different methods used for quantifying
uncertainty in infinite neural networks, and shows how to obtain analytic
expressions for both prior and posterior predictive distributions for that
purpose.

While the prior predictive can simply be modeled as a GP, we have outlined
three ways to obtain proper posterior predictives: using GP regression, weakly
trained NNs, or Bayesian deep ensembles, where the latter two turn out to be
equivalent to GP regression with particular covariance (kernel) functions.

## References

  * Coker et al. [2021] Beau Coker, Weiwei Pan, and Finale Doshi-Velez.  Wide mean-field variational bayesian neural networks ignore the data.  _arXiv preprint arXiv:2106.07052_ , 2021.
  * He et al. [2020] Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh.  Bayesian deep ensembles via the neural tangent kernel.  _arXiv preprint arXiv:2007.05864_ , 2020.
  * Jacot et al. [2018] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  _arXiv preprint arXiv:1806.07572_ , 2018.
  * Kapoor et al. [2010] Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell.  Gaussian processes for object categorization.  _International journal of computer vision_ , 88(2):169–188, 2010.
  * Kim and Lee [2007] Hyun-Chul Kim and Jaewook Lee.  Clustering based on gaussian processes.  _Neural computation_ , 19(11):3088–3107, 2007\.
  * Lázaro-Gredilla and Figueiras-Vidal [2010] Miguel Lázaro-Gredilla and Aníbal R Figueiras-Vidal.  Marginalized neural network mixtures for large-scale regression.  _IEEE transactions on neural networks_ , 21(8):1345–1351, 2010.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Lee et al. [2019] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington.  Wide neural networks of any depth evolve as linear models under gradient descent.  _Advances in neural information processing systems_ , 32:8572–8583, 2019.
  * Neal [1996] Radford M Neal.  Priors for infinite networks.  In _Bayesian Learning for Neural Networks_ , pages 29–53. Springer, 1996.
  * Snoek et al. [2015] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams.  Scalable bayesian optimization using deep neural networks.  In _International conference on machine learning_ , pages 2171–2180. PMLR, 2015.
  * Thakur et al. [2021] Sujay Thakur, Cooper Lorsung, Yaniv Yacoby, Finale Doshi-Velez, and Weiwei Pan.  Uncertainty-aware (una) bases for bayesian regression using multi-headed auxiliary networks, 2021.
  * Wang et al. [2019] Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson.  Exact gaussian processes on a million data points.  _Advances in Neural Information Processing Systems_ , 32:14648–14659, 2019.
  * Williams [1997] Christopher KI Williams.  Computing with infinite networks.  _Advances in neural information processing systems_ , pages 295–301, 1997.
  * Yang [2020] Greg Yang.  Tensor programs ii: Neural tangent kernel for any architecture.  _arXiv preprint arXiv:2006.14548_ , 2020.


# Application of the Deffuant model
in money exchange

Hsin-Lun Li<EMAIL_ADDRESS>
###### Abstract.

A money transfer involves a buyer and a seller. A buyer buys goods or services
from a seller. The money the buyer decreases is the same as that the seller
increases. At each time step, a pair of socially connected agents are selected
and transact in agreed money. We evolve the Deffuant model to a money exchange
system and study circumstances under which asymptotic stability holds, or
equal wealth can be achieved.

###### Key words and phrases:

Equal wealth, social mobility, money transfer, the rich and poor, conservative
system, social network.

Hsin-Lun Li

1National Sun Yat-sen University, Kaohsiung 804, Taiwan

##  1\. Introduction

The Deffuant model is one of the popular models in opinion dynamics. The
original Deffuant model consists of a finite number of agents whose opinion is
a number in $[0,1].$ Based on a fixed social relationship among all agents,
two socially connected agents are selected at each time step and interact if
and only if their opinion distance does not exceed some confidence threshold
$\epsilon$. The interacting mechanism goes as follows:

$\displaystyle\begin{aligned}
&x_{i}(t+1)=x_{i}(t)+\mu(x_{j}(t)-x_{i}(t))\mathbbm{1}\\{|x_{i}(t)-x_{j}(t)|\leq\epsilon\\}\\\
&x_{j}(t+1)=x_{j}(t)+\mu(x_{i}(t)-x_{j}(t))\mathbbm{1}\\{|x_{i}(t)-x_{j}(t)|\leq\epsilon\\}\end{aligned}$
(1)

for $x_{i}(t)\in[0,1]$ the opinion of agent $i$ at time $t$ and
$\mu\in(0,1/2]$ the convergence parameter. Namely, agents $i$ and $j$ move
equally toward each other. The Hegselmann-Krause model is another popular
model in opinion dynamics. Without a social relationship, an agent updates its
opinion by taking the average opinion of its opinion neighbors, those whose
opinion differs by at most some confidence threshold. The authors in [3]
derive a nontrivial lower bound for the probability of consensus under the
Deffuant model. The author in [6], the sequel to [5], introduces a variant of
the Hegselmann-Krause model and argues that it covers not only the Hegselmann-
Krause model but also the Deffuant model. With similar methods in [3], the
authors in [4] get a nontrivial lower bound for the probability of consensus
under the Hegselmann-Krause model.

A money transfer involves a buyer and a seller. The buyer pays for goods or
services offered by the seller. Therefore, the buyer decreases the same money
as the seller increases. Considering a finite set of agents, say two agents
are _socially connected_ if there is a social relationship, such as trust,
between them. Say $m_{i}$ is the money of agent $i$. $m_{i}<0$ if agent $i$ is
in debt. It is realistic to set a lower bound for $m_{i}$ since no one is
allowed to borrow money endlessly. Also, we assume the money transfer system
is conservative. Namely, influx equals efflux, therefore the sum of all
agents’ money constant over time. Before depicting the money transfer system
evolved from the Deffuant model, we introduce the following terms. Let
$[n]=\\{1,2,\ldots,n\\}$ denote the collection of all agents.

######  Definition 1.

A _social graph_ at time $t$, $G(t)=([n],E(t))$, is an undirected graph with
vertex set and edge set,

$[n]\ \hbox{and}\ E(t)=\\{(i,j)\in[n]^{2}:i\neq j\ \hbox{and}\ \hbox{vertices
$i$ and $j$ are socially connected}\\}.$

A _social graph for update_ at time $t$, $\tilde{G}(t)=([n],\tilde{E}(t))$, is
a subgraph of the social graph at time $t$ for money update.

######  Definition 2.

A graph is _$\delta$ -trivial_ if any two vertices in the graph are at a
distance of at most $\delta$ apart.

The Deffuant model evolves into a money transfer system with the following
setting: Let $m_{i}(0),\ i\in[n]$ and $\mu(t)$, $t\geq 0$ be independent
continuous real-valued random variables. Agents $i$ and $j$ are selected at
time $t$ and interact if and only if they are socially connected. The
interaction mechanism is as follows.

$\displaystyle\begin{aligned}
&m_{i}(t+1)=m_{i}(t)+\mu(t)\big{(}m_{j}(t)-m_{i}(t)\big{)}\mathbbm{1}\\{(i,j)\in
E(t)\\}\\\
&m_{j}(t+1)=m_{j}(t)+\mu(t)\big{(}m_{i}(t)-m_{j}(t)\big{)}\mathbbm{1}\\{(i,j)\in
E(t)\\}\end{aligned}$ (2)

where

$\begin{array}[]{l}\displaystyle m_{i}(t)\geq-d_{i}\ \hbox{for}\ d_{i}>0\
\hbox{constant and for all}\ t\geq 0,\vspace{2pt}\\\
\displaystyle\sum_{i\in[n]}m_{i}=C\ \hbox{for}\ C\
\hbox{constant}.\end{array}$

Observe that $d_{i}$ symbolizes the credibility of agent $i$ and $\mu(t)$
corresponds to the behavior that the money two agents agree to transact given
that no one is out of credibility. Say agent $i$ is richer than agent $j$ if
$m_{i}\geq m_{j}$, and poorer than agent $j$ if $m_{i}\leq m_{j}$. Given
$m_{i}(t)\leq m_{j}(t)$, $m_{i}(t+1)\leq m_{j}(t+1)$ if $\mu(t)\leq 1/2$, and
$m_{i}(t+1)\geq m_{j}(t+1)$ if $\mu(t)>1/2$. The former indicates that the
richer agent remains in its richer status, whereas the latter indicates the
richer agent becomes poorer, after the transaction, whether it is a buyer or a
seller at time $t$. The money transfer system evolved from the Deffuant model
differs from the money transfer system in [1, 2, 7] regarding the conservation
of money.

Let $U_{t}=\\{(i,j):\hbox{agent $i$ and $j$ are selected at time $t$}\
\hbox{and}\ \mu(t)\neq 0\\}$, $t\geq 0$ be independent and identically
distributed random variables with a support

$S\subset\big{\\{}\\{(i,j)\\}:i,j\in[n]\ \hbox{and}\ i\neq j\big{\\}}\
\hbox{and}\ \tilde{E}(t)=U_{t}\cap E(t).$

In other words, $U_{t}$ indicates a possible pair of candidates for money
transfer. Assume that $(\Omega,\mathscr{F},P)$ is a probability space for
$\mathscr{F}\subset\mathscr{P}(\Omega)$ a $\sigma$-algebra and $P$ a
probability measure. Denote $E(G)$ as the edge set of graph $G$. Given a pair
of agents in transaction, we consider the following conditions in
(LABEL:money_transfer_system):

  * •

$\mu\in(0,1)$,

  * •

$\mu>1$ and

  * •

$\mu<0$

corresponding to the richer agent whose money is

  * •

at least the same as the current money of the poorer agent after the
transaction,

  * •

at most the same as the current money of the poorer agent after the
transaction, and

  * •

at least the same as its current money after the transaction.

##  2\. Main results

It turns out that equal wealth can be achieved if all agents are willing and
possible to transact and the social graph is connected infinitely many times.
Namely, all agents eventually have the average money of the total money in the
system.

######  Theorem 3.

Assume that $\sup_{t\geq 0}|\mu(t)-1/2|<1/2$, the social graph connected
infinitely many times and $\bigcup_{a\in S}a\supset\binom{[n]}{2}$. Then,
equal wealth can be achieved eventually.

It follows that equal wealth can be achieved for a constant connected social
graph if all socially connected agents are willing and possible to transact.
Let $m_{(i)}$ be the $i$th smallest among $m_{1},m_{2},\ldots,m_{n}$. Theorem
4 shows conditions under which $m_{(i)}$ is asymptotically stable.

######  Theorem 4.

Assume that $\inf_{t\geq 0}|\mu(t)-1/2|\geq 1/2$, $\bigcup_{a\in
S}a\supset\binom{[n]}{2}$ and the social graph complete infinitely many times.
Then, $m_{(i)}$ is asymptotically stable for all $i\in[n]$ almost surely.

$\inf_{t\geq 0}|\mu(t)-1/2|\geq 1/2$ indicates that the richer agent can be
either much richer or poorer than the poor agent after a transaction.

##  3\. The model

The crucial part of proving Theorem 3 is to find a nonnegative nonincreasing
function and construct an inequality involving the current money and the
updated money. Then, we are able to show circumstances under which asymptotic
stability holds in the money transfer system.

######  Lemma 5.

Let $Z(t)=\sum_{i,j\in[n]}\big{(}m_{i}(t)-m_{j}(t)\big{)}^{2}$. Then, $Z(t)$
is monotone with respect to $t$. In particular,

$Z(t)-Z(t+1)=2n(\frac{1}{\mu(t)}-1)\mathbbm{1}\\{\mu(t)\neq
0\\}\sum_{i\in[n]}\big{(}m_{i}(t)-m_{i}(t+1)\big{)}^{2}.$

###### Proof.

Let $\tilde{E}(t)=\\{(p,q)\\}$, $\mu=\mu(t)$, $m_{i}=m_{i}(t)$ and
$m_{i}^{\star}=m_{i}(t+1)$ for all $i\in[n].$ Clearly, $Z(t)-Z(t+1)=0$ for
$\mu=0$. For $\mu\neq 0$, we have

$\displaystyle(m_{p}-m_{q})^{2}-(m_{p}^{\star}-m_{q}^{\star})^{2}=[1-(1-2\mu)^{2}](m_{p}-m_{q})^{2}$
$\displaystyle\hskip
56.9055pt=(2-2\mu)2\mu/\mu^{2}(m_{p}-m_{p}^{\star})^{2}=4\frac{(1-\mu)}{\mu}(m_{p}-m_{p}^{\star})^{2},$
$\displaystyle(m_{p}-m_{i})^{2}-(m_{p}^{\star}-m_{i})^{2}=(m_{p}-m_{p}^{\star})^{2}+2(m_{p}-m_{p}^{\star})(m_{p}^{\star}-m_{i}),$
$\displaystyle(m_{q}-m_{i})^{2}-(m_{q}^{\star}-m_{i})^{2}=(m_{q}-m_{q}^{\star})^{2}+2(m_{q}-m_{q}^{\star})(m_{q}^{\star}-m_{i}).$

Since $m_{p}-m_{p}^{\star}=-(m_{q}-m_{q}^{\star}),$

$\displaystyle(m_{p}-m_{i})^{2}-(m_{p}^{\star}-m_{i})^{2}+(m_{q}-m_{i})^{2}-(m_{q}^{\star}-m_{i})^{2}$
$\displaystyle\hskip
28.45274pt=2(m_{p}-m_{p}^{\star})^{2}+2(m_{p}-m_{p}^{\star})(m_{p}^{\star}-m_{q}^{\star})$
$\displaystyle\hskip
28.45274pt=[2+2(1-2\mu)/\mu](m_{p}-m_{p}^{\star})^{2}=2(\frac{1}{\mu}-1)(m_{p}-m_{p}^{\star})^{2}.$

In particular,

$\displaystyle
Z(t)-Z(t+1)=\sum_{i,j\in[n]}[(m_{i}-m_{j})^{2}-(m_{i}^{\star}-m_{j}^{\star})^{2}]$
$\displaystyle\hskip
14.22636pt=2\bigg{\\{}(m_{p}-m_{q})^{2}-(m_{p}^{\star}-m_{q}^{\star})^{2}$
$\displaystyle\hskip
36.98866pt+\sum_{i\in[n]-\\{p,q\\}}\big{[}(m_{p}-m_{i})^{2}-(m_{p}^{\star}-m_{i})^{2}+(m_{q}-m_{i})^{2}-(m_{q}-m_{i})^{2}\big{]}\bigg{\\}}$
$\displaystyle\hskip
14.22636pt=2\big{[}4(\frac{1}{\mu}-1)+2(n-2)(\frac{1}{\mu}-1)\big{]}(m_{p}-m_{p}^{\star})^{2}=4n(\frac{1}{\mu}-1)(m_{p}-m_{p}^{\star})^{2}.$

∎

######  Lemma 6.

Assume that $\sup_{t\geq 0}|\mu(t)-1/2|<1/2$. Then, all components of
$\tilde{G}$ are $\delta$-trivial after some finite time for all $\delta>0$
almost surely.

###### Proof.

Assume by contradiction that this is not the case. Then, there is
$(t_{k})_{k\geq 0}$ increasing with $\tilde{G}(t_{k})$ $\delta$-nontrivial for
all $k\geq 0$. $R=\sup_{t\geq 0}|\mu(t)-1/2|$ implies $1/2-R\leq\mu(t)\leq
1/2+R$, therefore $1/\mu(t)-1\geq\frac{1/2-R}{1/2+R}$ for all $t\geq 0$. It
follows from Lemma 5 that

$Z(0)\geq Z(0)-Z(s)=\sum_{t=0}^{s-1}[Z(t)-Z(t+1)]\ \hbox{for all}\ s\geq 1.$

Letting $s\to\infty,$

$\infty>Z(0)\geq\sum_{t\geq 0}[Z(t)-Z(t+1)]>\sum_{k\geq
0}4n\frac{(1/2-R)^{3}}{1/2+R}\delta^{2}=\infty,\ \hbox{a contradiction.}$

∎

###### Proof of Theorem 3.

Due to finiteness of the social graph, it connected infinitely many times
implies there is a graph $H$ connected infinitely many times, saying
$(t_{k})_{k\geq 0}$ the timepoints. For $(i,j)\in E(H)$, let $B_{t}$ be the
event that $(i,j)\in U_{t}$. Because of $\bigcup_{a\in
S}a\supset\binom{[n]}{2}$ and support $S$ finite, via second Borel–Cantelli
lemma,

$\sum_{k\geq 0}P(B_{t_{k}})\geq\sum_{k\geq 0}\min_{a\in
S}P(U_{t_{k}}=a)\geq\sum_{k\geq 0}\min_{a\in S}P(U_{0}=a)=\infty$

implies $B_{s_{\ell}},\ \ell\geq 0$ hold for $(s_{\ell})_{\ell\geq
0}\subset(t_{k})_{k\geq 0}$, therefore $(i,j)\in\tilde{E}(s_{\ell}).$ Via
Lemma 6, vertices $i$ and $j$ have the same money eventually. Graph $H$
connected implies all vertices achieve the same money. ∎

###### Proof of Theorem 4.

We claim that $\lim_{t\to\infty}|\mu(t)-1/2|=1/2$. Let
$A_{t}=\sup\big{\\{}|m_{i}(t)-m_{j}(t)|:i,j\in[n],\ -d_{i}<m_{i}(t)\
\hbox{and}\ -d_{j}<m_{j}(t)\big{\\}}.$ For $c=0$ or $-\infty$, $A_{t}=c$
implies $A_{s}=c$ for some $t$ and for all $s\geq t.$ It is clear that
$P(A_{0}=c)=0$. If $A_{t}>0$ for all $t\geq 0,$ then $A_{t}$ is nondecreasing
with respect to $t$. Assume that $\limsup_{t\to\infty}|\mu(t)-1/2|>1/2$. Then,
there are $\delta>0$ and $(t_{k})_{k\geq 0}$ increasing such that
$|\mu(t_{k})-1/2|\geq 1/2+\delta$ for all $k\geq 0$. Let
$i_{t}=\arg\max_{i}\\{m_{i}(t):i\in[n]\ \hbox{and}\ -d_{i}<m_{i}(t)\\}$,
$j_{t}=\arg\min_{i}\\{m_{i}(t):i\in[n]\ \hbox{and}\ -d_{i}<m_{i}(t)\\}$ and
$E_{t}$ be the event that agents $i_{t_{0}}$ and $j_{t_{0}}$ transact at time
$t$. Due to finite support $S$, $\bigcup_{a\in S}a\supset\binom{[n]}{2}$ and
the second Borel-Cantelli lemma, we have $\sum_{k\geq 0}P(E_{t_{k}})=\infty$,
which implies $E_{t}$ holds for infinitely many $t\in(t_{k})_{k\geq 0}$. Say
$\ell_{1}\in(t_{k})_{k\geq 0}$ is the earliest time that $E_{\ell_{1}}$ holds.
Then,
$A_{\ell_{1}+1}\geq|1-2\mu(\ell_{1})|A_{\ell_{1}}\geq(1+2\delta)A_{\ell_{1}}$.
Repeating the above process, there is $(\ell_{k})_{k\geq
1}\subset(t_{k})_{k\geq 0}$ increasing such that
$A_{\ell_{k}+1}\geq(1+2\delta)^{k}A_{\ell_{1}}\to\infty$ as $k\to\infty$, a
contradiction. This completes the proof. ∎

## Acknowledgment

The author is funded by the National Science and Technology Council in Taiwan.

## References

  * [1] A. Dragulescu and V. M. Yakovenko.  Statistical mechanics of money.  The European Physical Journal B-Condensed Matter and Complex Systems, 17(4):723–729, 2000.
  * [2] N. Kiyotaki and R. Wright.  A search-theoretic approach to monetary economics.  The American Economic Review, pages 63–77, 1993.
  * [3] N. Lanchier and H.-L. Li.  Probability of consensus in the multivariate Deffuant model on finite connected graphs.  Electronic Communications in Probability, 25:1–12, 2020.
  * [4] N. Lanchier and H.-L. Li.  Consensus in the Hegselmann–Krause model.  Journal of Statistical Physics, 187(3):1–13, 2022.
  * [5] H.-L. Li.  Mixed Hegselmann-Krause dynamics.  Discrete and Continuous Dynamical Systems - B, 27(2):1149–1162, 2022.
  * [6] H.-L. Li.  Mixed Hegselmann-Krause dynamics II.  Discrete and Continuous Dynamical Systems - B, 28(5):2981–2993, 2023.
  * [7] M. Molico.  The distribution of money and prices in search equilibrium.  International Economic Review, 47(3):701–722, 2006.