# Dataset Dictionary Learning in a Wasserstein Space for Federated Domain
Adaptation

Eduardo Fernandes Montesuma
CEA, List
Université Paris-Saclay
F-91120 Palaiseau, France &Fabiola Espinoza Castellon
CEA, List
Université Paris-Saclay
F-91120 Palaiseau, France &Fred Ngolè Mboula
CEA, List
Université Paris-Saclay
F-91120 Palaiseau, France &Aurélien Mayoue
CEA, List
Université Paris-Saclay
F-91120 Palaiseau, France &Antoine Souloumiac
CEA, List
Université Paris-Saclay
F-91120 Palaiseau, France &Cédric Gouy-Pailler
CEA, List
Université Paris-Saclay
F-91120 Palaiseau, France

###### Abstract

Multi-Source Domain Adaptation (MSDA) is a challenging scenario where multiple
related and heterogeneous source datasets must be adapted to an unlabeled
target dataset. Conventional MSDA methods often overlook that data holders may
have privacy concerns, hindering direct data sharing. In response,
decentralized MSDA has emerged as a promising strategy to achieve adaptation
without centralizing clients’ data. Our work proposes a novel approach,
Decentralized Dataset Dictionary Learning, to address this challenge. Our
method leverages Wasserstein barycenters to model the distributional shift
across multiple clients, enabling effective adaptation while preserving data
privacy. Specifically, our algorithm expresses each client’s underlying
distribution as a Wasserstein barycenter of public atoms, weighted by private
barycentric coordinates. Our approach ensures that the barycentric coordinates
remain undisclosed throughout the adaptation process. Extensive
experimentation across five visual domain adaptation benchmarks demonstrates
the superiority of our strategy over existing decentralized MSDA techniques.
Moreover, our method exhibits enhanced robustness to client parallelism while
maintaining relative resilience compared to conventional decentralized MSDA
methodologies.

_Keywords_ Federated Learning $\cdot$ Domain Adaptation $\cdot$ Dataset
Dictionary Learning $\cdot$ Optimal Transport

##  1 Introduction

Supervised machine learning models are trained with large amounts of labeled
data. However, these models are subject to performance degradation, if the
data used for training does not exactly resembles those used for test. This
issue is known in the literature as dataset, or distributional shift [1]. For
instance, in computer vision, factors such as illumination, pose and image
quality can induce changes in the data underlying distribution [2, 3]. In this
context Multi-Source DA (MSDA) [4, 5] emerged as a strategy to adapt multiple,
heterogeneous, labeled source datasets towards an unlabeled target dataset.

Nonetheless, standard methods in MSDA overlook that datasets may be divided
over multiple clients, rather than centralized on a server. Due to privacy
concerns, these clients may not want to centralize their data. Motivated by
this challenge, decentralized MSDA is a possible solution to this problem [6,
7, 8]. In parallel to this strategy, distributional shift has been considered
by the federated learning literature [9, 10]. However, concerning domain
adaptation, non-i.i.d. federated learning strategies are limited since they do
not exploit unlabeled target domain data.

Existing methods in decentralized MSDA mainly follow 2 strategies. First, one
may align the multiple existing distributions in a latent space, by learning
invariant features [6]. Second, on top of aligning distributions, one may
perform pseudo-labeling of the target domain through classifiers learned on
the source domains [7, 8]. While invariant representation learning is an
important component in domain adaptation, it poses a trade-off between
classification performance and domain invariance [11], which may limit domain
adaptation performance. We thus take a different route by modeling the shift
between domains in a Wasserstein space.

In parallel, Optimal Transport (OT) is a mathematical theory concerned with
the displacement of mass at least effort. This theory previously contributed
to the diverse landscape of Domain Adaptation (DA), both in single-source [12,
13] and multi-source [14, 15, 16] settings. OT is especially advantageous, as
it is sensitive to the geometry of the data ambient space [17]. In addition,
it allows for the definition of _averages of probability distributions_
through Wasserstein barycenters [18], which has been previously used for DA
[14, 15, 5]. Overall, OT presents a principled framework for developing DA
algorithms [19].

Given the aforementioned limitations of invariant representation learning, we
offer a novel, OT-based method that learns how to express clients’ underlying
probability distribution as a Wasserstein barycenter of learned atoms,
weighted by barycentric coordinates. As such, our method effectively keeps
clients’ distributions private, because their barycentric coordinates do not
need to be communicated. To the best of our knowledge, ours is the first OT-
inspired decentralized DA algorithm, which does not align clients’
distributions. Our contributions are threefold,

  1. 1.

We propose a novel strategy, called Federated Dataset Dictionary Learning
(FedDaDiL), for performing decentralized dictionary learning over empirical
distributions. This strategy has the advantage of keeping the variables that
allow to reconstruct clients’ data distributions, i.e., the barycentric
coordinates, private.

  2. 2.

We provide a novel theoretical analysis of the objective function of Dataset
Dictionary Learning (DaDiL) [5], showing that, for small perturbations, it
behaves as a quadratic form on the atoms’ feature vectors.

  3. 3.

We provide extensive empirical results on five visual DA benchmarks, showing
that (i) our strategy has superior performance on all datasets, (ii) FedDaDiL
is lightweight compared to communicating the parameters of deep neural nets.
Over the tested benchmarks, communicating dictionaries with the server
corresponds to approximately $34.6\%\pm 23.3\%$ of the total amount of bits
used to encode ResNet weights, and (iii) FedDaDiL is more robust w.r.t. client
parallelism than previous methods.

Especially, this paper extends the previous work [20] in two important ways.
First, we average atoms in a similar way to FedAVG, which proved to be
effective (c.f., section 4). Second, we provide a new theoretical analysis on
the loss function being minimizing throughout dictionary learning.

The rest of this paper is organized as follows. Section 2 presents related
work on dictionary learning and decentralized MSDA. Section 3 presents our
novel approach for decentralized MSDA, constituted of _FedAVG_ (section 3.2)
and _FedDaDiL_ (section 3.3). Section 4 presents our experiments on various
visual DA benchmarks. Finally, section 5 concludes this paper.

##  2 Related Work

Dictionary Learning seeks to decompose a set of vectors
$\\{\mathbf{x}_{1},\cdots,\mathbf{x}_{N}\\}$,
$\mathbf{x}_{\ell}\in\mathbb{R}^{d}$, as a linear combination of atoms
$\\{\mathbf{p}_{1},\cdots\mathbf{p}_{K}\\}$, weighted by representation
vectors $\\{\alpha_{1},\cdots,\alpha_{N}\\}$. When $\mathbf{x}_{\ell}$ are
histograms (i.e., $\sum_{j}x_{\ell,j}=1$ and $x_{\ell,j}\geq 0$), OT defines
meaningful loss functions [21], as well as novel ways of aggregating atoms
[22]. Additionally, decentralized strategies for learning dictionaries have
been proposed by [23] and [24], respectively. In this work we use a different
interpretation of dictionary learning, in which the atoms are empirical
probability distributions [5]. To the best of our knowledge, ours is the first
decentralized algorithm for dictionaries of empirical distributions.

Decentralized Domain Adaptation. Dataset heterogeneity is a major challenge in
supervised learning, as commonly used tools, such as Empirical Risk
Minimization (ERM) [25] work under the assumption of i.i.d. data. This issue
also emerges in decentralized settings, such as federated learning [9], in
which client data follow different probability distributions [26]. In this
paper we consider decentralized MSDA, where on top of clients with different
probability distributions, we have an unique client, called _target_ , who do
not possess labeled data. Different ideas in MSDA have been adapted to the
decentralized setting. Federated Adversarial Domain Adaptation (FADA) [6] uses
the adversarial learning framework of [27] with an additional feature
disentanglement [28] module. Knowledge Distillation and Decentralized Domain
Adaptation (KD3A) [7] uses knowledge distillation [29, 30] and pseudo-labeling
for the adaptation. Finally, Co2-Learning with Multi-Domain Attention (Co-MDA)
[8] studies black box DA, a type of source-free DA [31], where one has access
only to outputs of source client models.

##  3 Proposed Approach

Problem Statement. We are interested in decentralized MSDA. Clients are
associated with indices $\ell=1,\cdots,N$, where $\ell=1,\cdots,N-1$ are
called _source clients_ and $\ell=N$ is the _target_ client. In what follows,
we have access to pairs
$\\{(\mathbf{x}_{i}^{(Q_{\ell})},\mathbf{y}_{i}^{(Q_{\ell})})\\}_{i=1}^{n_{\ell}}$
from the sources, where $\mathbf{x}_{i}^{(Q_{\ell})}\in\mathbb{R}^{d}$ and
$\mathbf{y}_{i}^{(Q_{\ell})}\in\Delta_{K}$, i.e., $\sum_{c}y_{ic}=1$ and
$y_{ic}\geq 0$. For $\ell=N$, we only have access to
$\\{\mathbf{x}_{i}^{(Q_{\ell})}\\}_{i=1}^{n_{\ell}}$, i.e., samples are not
labeled. Our goal is to learn a classifier on $Q_{N}$, based on the available
samples. We do so through dataset dictionary learning, i.e., we learn a set
$\mathcal{P}=\\{\hat{P}_{k}\\}_{k=1}^{K}$ and
$\mathcal{A}=\\{\alpha_{\ell}\\}_{\ell=1}^{N}$, $\alpha_{\ell}\in\Delta_{K}$
such that $\hat{Q}_{N}=\mathcal{B}(\alpha_{N};\mathcal{P})$, i.e., the
barycenter in Wasserstein space of $\mathcal{P}$. We detail these ideas in the
following.

###  3.1 Background

We start with OT, a field of mathematics that, in a nutshell, studies the
transportation of probability distributions under least effort. Our work is
based on the Kantorovich formulation [32]. For recent overviews of the theory,
we refer readers to [33] and [17]. In the following, we approximate client
distributions empirically through,

$\displaystyle\hat{Q}_{\ell}(\mathbf{x})=\dfrac{1}{n_{\ell}}\sum_{i=1}^{n_{\ell}}\delta(\mathbf{x}-\mathbf{x}_{i}^{(Q_{\ell})}).$
(1)

where $\delta$ is the Dirac delta function. In this context,
$\mathbf{X}^{(Q_{\ell})}\in\mathbb{R}^{n_{\ell}\times d}$ is called _the
support of_ $\hat{Q}_{\ell}$. In this setting, the so-called OT problem
between distributions $\hat{P}$ and $\hat{Q}$ is,

$\hat{\gamma}=\underset{\gamma\in\Gamma(P,Q)}{\text{argmin}}\,\sum_{i=1}^{n}\sum_{j=1}^{m}\gamma_{ij}C_{ij},$
(2)

where $C_{ij}=c(\mathbf{x}_{i}^{(P)},\mathbf{x}_{j}^{(Q)})$ is called _ground-
cost matrix_ and $c$ is a function modeling the _effort of transportation_
between samples $\mathbf{x}_{i}^{(P)}\overset{\text{i.i.d.}}{\sim}P$ and
$\mathbf{x}_{j}^{(Q)}\overset{\text{i.i.d.}}{\sim}P$. The matrix
$\gamma\in\mathbb{R}^{n\times m}$ is called _transport plan_ , and
$\Gamma(P,Q)=\\{\gamma:\sum_{i}\gamma_{ij}=m^{-1}\text{ and
}\sum_{j}\gamma_{ij}=n^{-1}\\}$.

Problem 2 is a linear program over the variables $\gamma_{ij}$. As such, it
has $\mathcal{O}(n^{3}\log n)$ complexity over the number of samples [34]. A
way of alleviating this complexity is using mini-batch OT [35], instead of
calculating $\hat{\gamma}$ over all samples of $\hat{P}$ and $\hat{Q}$. In
addition, $\gamma$ can be used for building a mapping between distributions
$\hat{P}$ and $\hat{Q}$. In the discrete case, this takes the form of the
_barycentric projection_ [12],

$\displaystyle
T_{\gamma}(\mathbf{x}_{i}^{(P)})=\underset{\mathbf{x}\in\mathbb{R}^{d}}{\text{argmin}}\,\sum_{j=1}^{m}\gamma_{ij}c(\mathbf{x},\mathbf{x}_{j}^{(Q)}),$
(3)

when
$c(\mathbf{x}_{i}^{(P)},\mathbf{x}_{j}^{(Q)})=\lVert\mathbf{x}_{i}^{(P)}-\mathbf{x}_{j}^{(Q)}\rVert_{2}^{2}$
eq. 3 can be conveniently expressed as
$T_{\gamma}(\mathbf{X}^{(P)})=n\gamma\mathbf{X}^{(Q)}$.

Based on the OT solution, one has an associated _cost_ for transporting
$\hat{P}$ to $\hat{Q}$, defined as,

$\displaystyle\mathcal{T}_{c}(\hat{P},\hat{Q})=\sum_{i=1}^{n}\sum_{j=1}^{m}\gamma_{ij}^{\star}C_{ij},$
(4)

where $\gamma^{\star}$ is the solution of eq. 2. Henceforth, we use
$\mathcal{T}_{2}$ to eq. 4 with $c=\lVert\cdot\rVert_{2}^{2}$. When
$C_{ij}=\lVert\mathbf{x}_{i}^{(P)}-\mathbf{x}_{j}^{(Q)}\rVert^{p}_{2}$, one
may define $W_{p}(\hat{P},\hat{Q})=(\mathcal{T}_{c}(\hat{P},\hat{Q}))^{1/p}$,
the Wasserstein distance between $\hat{P}$ and $\hat{Q}$, which _inherits the
metric properties from_ $C$. This metric between distributions allows for the
definition of barycenters of probability distributions [18],

######  Definition 3.1.

For a set of distributions $\mathcal{P}=\\{P_{k}\\}_{k=1}^{K}$ and barycentric
coordinates $\alpha\in\Delta_{K}$, the Wasserstein barycenter is a solution
to,

$\displaystyle
B^{\star}=\mathcal{B}(\alpha;\mathcal{P})=\underset{B}{\text{inf}}\,\sum_{k=1}^{K}\alpha_{k}W_{2}(P_{k},B)^{2}.$
(5)

Henceforth we call $\mathcal{B}(\cdot;\mathcal{P})$ barycentric operator.

Even though eq. 5 is continuous, the barycenter problem can be solved in terms
of the support $\mathbf{X}^{(B)}$ of $B$, as described in [36]. Next, we
extend the OT problem for handling labeled data. This is done by integrating
the labels into the cost function [15]. As [5], we use,

$\displaystyle
C_{ij}=\lVert\mathbf{x}_{i}^{(P)}-\mathbf{x}_{j}^{(Q)}\rVert_{2}^{2}+\beta\lVert\mathbf{y}^{(P)}_{i}-\mathbf{y}^{(Q)}_{j}\rVert_{2}^{2},$
(6)

where $\beta>0$ controls the importance of penalizing the transport between
samples from different classes. Henceforth, we use $\mathcal{T}_{c}$ to denote
eq. 4 with $C_{ij}$ given by eq. 6. Furthermore, one may solve eq. 5 for
$\hat{P}_{k}$ with support $(\mathbf{X}^{(P_{k})},\mathbf{Y}^{(P_{k})})$,
which yields $\hat{B}$ with support $(\mathbf{X}^{(B)},\mathbf{Y}^{(B)})$
where $\mathbf{Y}^{(B)}=\\{\mathbf{y}_{i}^{(B)}\\}_{i=1}^{n}$ are soft-labels,
i.e., $\mathbf{y}_{i}^{(B)}\in\Delta_{n_{c}}$. This computation is done, for
instance, with [5, Alg. 1].

In [5], authors presented a novel dictionary learning framework over
_empirical distributions_ (cf. eq. 1). The DaDiL framework introduces _virtual
distributions_ , $\mathcal{P}=\\{\hat{P}_{k}\\}_{k=1}^{K}$, called atoms. Each
$\hat{P}_{k}$ has a free-support,
$(\mathbf{X}^{(P_{k})},\mathbf{Y}^{(P_{k})})$, which is determined via
optimization. The atoms are linked to true distributions
$\hat{Q}_{\ell}\in\mathcal{Q}$ through Wasserstein barycenter
$\mathcal{B}(\alpha_{\ell};\mathcal{P})$, where $\alpha_{\ell}$ are the
barycentric coordinates allowing to _reconstruct_ $\hat{Q}_{\ell}$.
Mathematically, DaDiL is expressed as,

$\displaystyle(\mathcal{P}^{\star},\mathcal{A}^{\star})=\underset{\mathcal{P},\mathcal{A}\in(\Delta_{K})^{N}}{\text{argmin}}\,\dfrac{1}{N}\sum_{\ell=1}^{N}\mathcal{L}(\hat{Q}_{\ell},\mathcal{B}(\alpha_{\ell};\mathcal{P})),$
(7)

where $\mathcal{L}=\mathcal{T}_{c}$ if $\hat{Q}_{\ell}$ is labeled (i.e.,
$\ell\leq N-1$) or $\mathcal{L}=\mathcal{T}_{2}$ otherwise (i.e., $\ell=N$).
In a nutshell, DaDiL learns to approximate true distributions $\hat{Q}_{\ell}$
as the Wasserstein barycenter of atoms $\mathcal{P}$. In practice, DaDiL
relies on features extracted by a neural net, so that barycenters are
calculated in a semantically rich latent space. Furthermore, [5] minimizes eq.
7 via mini-batches.

Our method parts from the following observation: while the atoms $\mathcal{P}$
are shared by all domains, the barycentric coordinates $\alpha_{\ell}$ are
specific to each domain, and thus do are not aggregated nor communicated
throughout federated dictionary learning. Next, we describe the two
ingredients for our FedDaDiL strategy.

###  3.2 Federated Learning an Encoder Network

(a) Step 1: FedAVG

(b) Step 2: FedDaDiL

Figure 1: Illustration of our decentralized MSDA strategy. (a) We fit a neural
deep neural network composed of an encoder net $\phi$ and a classifier $h$,
without centralizing client data. In principle, the target client does not
participate at this step, unless some adaptation method is used (e.g., KD3A
[7]). (b) We do the adaptation step with features extracted from the fine-
tuned source model, through our proposed FedDaDiL.

Our FedDaDiL framework works over extracted features of a deep neural net. In
a decentralized setting, clients cannot centralize their data for fine-tuning
an existing architecture. To keep our overall pipeline _end-to-end
decentralized_ , we choose to fine-tune an encoder using the FedAVG algorithm
of [9]. Contrary to existing works [6, 7], we do not align the probability
distributions of clients’ features. This choice is important, because the
dictionary learning step must have a rich variety of probability distributions
for modeling distributional shift.

In this context, let $\phi:\mathcal{X}\rightarrow\mathcal{Z}$ be the encoder
network, which takes as inputs images $\mathbf{x}_{i}^{(Q_{\ell})}$ and
outputs a vector
$\mathbf{z}_{i}^{(Q_{\ell})}=\phi(\mathbf{x}_{i}^{(Q_{\ell})})\in\mathbb{R}^{d}$.
Likewise, let $h:\mathcal{Z}\rightarrow\mathcal{Y}$ be a classifier. For
instance, $\phi$ is the set of convolutional layers in a ResNet [37], whereas
$h$ is a single-layer Perceptron. We find the parameters $\theta_{\phi}$ and
$\theta_{h}$ by minimizing,

$(\theta_{\phi}^{\star},\theta_{h}^{\star})=\underset{\theta_{\phi},\theta_{h}}{\text{argmin}}\,\dfrac{1}{N-1}\sum_{\ell=1}^{N-1}\dfrac{1}{n_{\ell}}\sum_{i=1}^{n_{\ell}}\mathcal{L}(\mathbf{x}_{i}^{(Q_{\ell})},\mathbf{y}_{i}^{(Q_{\ell})};\theta_{\phi},\theta_{h}),$
(8)

where
$\mathcal{L}(\mathbf{x}_{i}^{(Q_{\ell})},\mathbf{y}_{i}^{(Q_{\ell})};\theta_{\phi},\theta_{h})=\sum_{c=1}^{n_{c}}y_{ic}\log
h(\phi(\mathbf{x}_{i}))_{c}$ is the cross-entropy loss. Eq. 8 is minimized in
a federated way, i.e., each client $\ell\leq N-1$ minimizes the loss with
respect its own data independently of others. We explicitly leave the target
client out of the training process, since they do not have labeled data.

As discussed in [9], the federated setting introduces a new level of
complexity to the learning problem. As a consequence, before training begins,
a server needs to initialize the parameters
$\\{\theta_{\phi}^{(\ell)},\theta_{h}^{(\ell)}\\}_{\ell=1}^{N-1}$ with the
same values. After a fixed number of $E$ training steps of each client, the
server aggregates clients weights, by averaging clients’ versions,

$\displaystyle\theta^{(g)}_{\phi}=\dfrac{1}{N-1}\sum_{\ell=1}^{N-1}\theta_{\phi}^{(\ell)}\text{,
and, }\theta^{(g)}_{h}=\dfrac{1}{N-1}\sum_{\ell=1}^{N-1}\theta_{h}^{(\ell)}.$
(9)

###  3.3 Federated Dataset Dictionary Learning

We assume $\theta_{\phi}^{(g)}$ fixed, and
$\mathbf{z}_{i}^{(Q_{\ell})}=\phi(\mathbf{x}_{i}^{(Q_{\ell})};\theta_{\phi}^{(g)})$.
In FedDaDiL, each atom $\hat{P}_{k}$ has a free support, denoted as
$(\mathbf{Z}^{(P_{k})},\mathbf{Y}^{(P_{k})})$ and each client holds a set of
barycentric coordinates $\alpha_{\ell}\in\Delta_{k}$. Hence,

$\displaystyle(\mathcal{P}^{\star},\mathcal{A}^{\star})=\underset{\mathcal{P},\mathcal{A}}{\text{argmin}}\,\underbrace{\dfrac{1}{N}\sum_{\ell=1}^{N}f_{\ell}(\mathcal{P},\alpha_{\ell})}_{=f(\mathcal{P},\mathcal{A})},$
(10)

where $f_{\ell}$ is the objective function of each domain:

$f_{\ell}(\mathcal{P},\alpha_{\ell})=\begin{cases}\mathcal{T}_{c}(\hat{Q}_{\ell},\mathcal{B}(\alpha_{\ell};\mathcal{P}))&\ell\leq
N-1,\\\
\mathcal{T}_{2}(\hat{Q}_{\ell},\mathcal{B}(\alpha_{\ell};\mathcal{P}))&\ell=N,\\\
\end{cases}$ (11)

While $\mathcal{P}$ are shared by all clients, the barycentric coordinates
$\alpha_{\ell}$ are private to each client. Our federated strategy, presented
in algorithm 1 is divided into two sub-routines: _ClientUpdate_ and
_ServerAggregate_.

Algorithm 1 FedDaDiL. The $N$ Clients are indexed by $\ell$. $n_{b}$ is the
batch size, and $K$ the number of atoms.

1:Server initializes $\mathcal{P}_{g}^{(0)}=\\{\hat{P}_{k}^{(0)}\\}_{k=1}^{K}$

2:clients initialize $\alpha_{\ell}^{(0)}\in\Delta_{K}$,
$\forall\ell=1\cdots,N$

3:for each round $r=1\cdots,R$ do

4: for client $\ell=1,\cdots,N$ do

5: Server communicates $\mathcal{P}_{g}^{(r)}$ to client $\ell$

6: Initialize local dictionary
$\mathcal{P}_{\ell}^{(0)}\leftarrow\mathcal{P}_{g}^{(r)}$

7:
$\mathcal{P}_{\ell}^{(r)}\leftarrow\text{ClientUpdate}(\mathcal{P}_{\ell}^{(0)},\alpha_{\ell}^{(r)})$

8: client $\ell$ sends $\mathcal{P}_{\ell}^{(r)}$ to server.

9: end for

10:
$\mathcal{P}^{(r+1)}_{g}\leftarrow\text{ServerAggregate}(\\{\mathcal{P}_{\ell}^{(r)}\\}_{\ell=1}^{N})$

11:end for

Clients Update. Similarly to FedAVG, at each communication round $r$, each
client receives a global version from the server, noted as
$\mathcal{P}_{g}^{(r)}$, which is copied into $\mathcal{P}_{\ell}^{(0)}$, the
local version. The clients then proceed to optimize
$(\mathcal{P}_{\ell}^{(0)},\alpha_{\ell})$ through $E$ steps, by first
splitting each $\hat{P}_{k}$ into $B=\lceil\nicefrac{{n}}{{n_{b}}}\rceil$
batches of size $n_{b}$. An epoch corresponds to an entire pass through the
$B$ mini-batches. The loss is calculated between mini-batches of
$\hat{P}_{k}$, and mini-batches of $\hat{Q}_{\ell}$. This is detailed in
Algorithm 2. After each client step, it enforces $\alpha_{\ell}\in\Delta_{K}$
by projecting it orthogonally into the simplex.

Algorithm 2 ClientUpdate.

1:for local epoch $e=1,\cdots,E$ do

2: for batch $b=1,\cdots,B$ do

3: Let
$\\{\\{(\mathbf{z}_{i}^{(P_{k})},\mathbf{y}_{i}^{(P_{k})})\\}_{i=b\times
n_{b}}^{(b+1)\times n_{b}}\\}_{k=1}^{K}$

4: Sample
$\\{(\mathbf{z}_{i}^{(Q_{\ell})},\mathbf{y}_{i}^{(Q_{\ell})})\\}_{i=1}^{n_{b}}$

5: Compute loss $f_{\ell}(\mathcal{P}_{\ell}^{(e)},\alpha_{\ell})$

6: for atom $k=1,\cdots,K$ do

7:
$\mathbf{z}_{i}^{(P_{k})}\leftarrow\mathbf{z}_{i}^{(P_{k})}-\eta\nicefrac{{\partial
f_{\ell}}}{{\partial\mathbf{z}_{i}^{(P_{k})}}}$

8:
$\mathbf{y}_{i}^{(P_{k})}\leftarrow\mathbf{y}_{i}^{(P_{k})}-\eta\nicefrac{{\partial
f_{\ell}}}{{\partial\mathbf{y}_{i}^{(P_{k})}}}$

9: end for

10:
$\alpha_{\ell}\leftarrow\text{proj}_{\Delta_{K}}(\alpha_{\ell}-\eta\nicefrac{{\partial
f_{\ell}}}{{\partial\alpha_{\ell}}})$

11: end for

12:end for

13:Client sets $\alpha_{\ell}^{(r+1)}\leftarrow\alpha_{\ell}^{\star}$.

14:Return $\mathcal{P}_{\ell}^{\star}$.

Server Aggregation. As the result of _ClientUpdate_ , at each communication
round FedDaDiL has $N$ atom versions,
$\\{\mathcal{P}_{\ell}^{\star}\\}_{\ell=1}^{N}$. As such, in the same way as
_FedAVG_ [9], one needs to _aggregate_ these different versions. Given
versions $\mathcal{P}_{0}$ and $\mathcal{P}_{1}$, we define the following
arithmetic,

$\displaystyle\mathcal{P}_{0}+\alpha\mathcal{P}_{1}$
$\displaystyle:=\biggr{\\{}\dfrac{1}{n}\sum_{i=1}^{n}\delta_{(\mathbf{z}_{i}^{(P_{k,0})}+\alpha\mathbf{z}_{i}^{(P_{k,1})},\mathbf{y}_{i}^{(P_{k,0})}+\alpha\mathbf{y}_{i}^{(P_{k,1})})}\biggr{\\}}_{k=1}^{K},$

i.e., we perform the summation w.r.t. the support of
$\hat{P}_{k}\in\mathcal{P}$. The aggregation step is, therefore,

$\displaystyle\mathcal{P}_{g}^{(r+1)}=\dfrac{1}{N}\sum_{\ell=1}^{N}\mathcal{P}_{\ell}^{\star}.$
(12)

As we investigate in our experiments ($\S$ 4), FedDaDiL’s objective behaves
similarly to neural nets over interpolations of atom versions. Next, we
present a novel result that shows that FedDaDiL’s objective behaves locally as
a quadratic form,

######  Theorem 3.1.

Let $(\mathcal{P},\mathcal{A})$ be a dictionary, and
$\epsilon\in\mathbb{R}^{d}$ be a random perturbation. Let
$\tilde{\mathcal{P}}=\\{\tilde{P}_{k}\\}_{k=1}^{K}$ such that,

$\tilde{P}_{k}(\mathbf{z},\mathbf{y})=\dfrac{1}{n}\sum_{i=1}^{n}\delta((\mathbf{z},\mathbf{y})-(\mathbf{z}_{j}^{(P_{k})}+\epsilon,\mathbf{y}_{j}^{(P_{k})})),$

then,

$\displaystyle f(\tilde{\mathcal{P}},\mathcal{A})$
$\displaystyle=f(\mathcal{P},\mathcal{A})+2\epsilon^{T}\nabla_{x}f+\lVert\epsilon\rVert_{2}^{2},$

A proof of this result is available on our supplementary materials.

Complexity. Since algorithm 2 runs over mini-batches, the overall
computational complexity on clients is cubic over the size of mini-batches
$n_{b}$ [5, $\S$ 4]. Furthermore, at each round, clients’ communicate

$\displaystyle|\mathcal{P}|=K\times n\times(d+n_{c})$ (13)

floating-point numbers at each round. As we discuss in our experiments, while
we cannot avoid communicating models in FedAVG, the DaDiL step is
comparatively lightweight.

###  3.4 Domain Adaptation

The dictionary learned at the end of Algorithm 1 _models the distributional
shift_ occurring between sources and target domains. Unlike previous works on
decentralized MSDA, we do not align the sources with the target. We rather
_embrace_ distributional shift by modeling it. To learn a classifier at the
target domain we have two strategies, Reconstruction (R) or Ensembling (E)
[5], which we now describe. Both methods stem from the fact that each
$\hat{P}_{k}$ has a labeled support, i.e.,
$(\mathbf{Z}^{(P_{k})},\mathbf{Y}^{(P_{k})})$. The following methods can thus
be applied locally by the target client for learning a classifier that works
on data following its probability distribution.

Reconstructing the Target Domain. Through FedDaDiL, we can express
$\hat{Q}_{N}=\mathcal{B}(\alpha_{N};\mathcal{P})$. Let
$(\mathbf{Z}^{(B_{N})},\mathbf{Y}^{(B_{N})})$ denote the support of
$\hat{B}_{N}=\mathcal{B}(\alpha_{N};\mathcal{P})$. These can be expressed in
terms of the support of each $\hat{P}_{k}$, as,

$\mathbf{z}^{(B_{N})}_{i}=n\sum_{k=1}^{K}\alpha_{N,k}\sum_{j=1}^{n}\pi_{i,j}^{(k)}\mathbf{z}_{j}^{(P_{k})}\text{,
and,
}\mathbf{y}^{(B_{N})}_{i}=n\sum_{k=1}^{K}\alpha_{N,k}\sum_{j=1}^{n}\pi_{i,j}^{(k)}\mathbf{y}_{j}^{(P_{k})},$
(14)

where $\pi_{i,j}^{(k)}$ is the OT plan between $\hat{B}_{N}$ and
$\hat{P}_{k}$. We can fit a classifier directly on the target client with
samples $\\{(\mathbf{z}_{i}^{(B_{N})},\mathbf{y}_{i}^{(B_{N})})\\}_{i=1}^{n}$
with standard ERM,

$\displaystyle\hat{\theta}_{R}=\underset{\theta}{\text{argmin}}\,\dfrac{1}{n}\sum_{i=1}^{n}\mathcal{L}(h(\mathbf{z}_{i}^{(Q_{N})};\theta),\mathbf{y}_{i}^{(Q_{N})})$

Ensembling Atom Classifiers. Conversely, the target domain can fit a
classifier on each atom distribution, through,

$\displaystyle\hat{\theta}_{k}=\underset{\theta}{\text{argmin}}\,\dfrac{1}{n}\sum_{i=1}^{n}\mathcal{L}(h(\mathbf{z}_{i}^{(P_{k})};\theta),\mathbf{y}_{i}^{(P_{k})}).$

One can then exploit the weights $\\{\alpha_{N,k}\\}_{k=1}^{K}$ to weight the
predictions of classifiers $\\{h(\cdot;\hat{\theta}_{k})\\}_{k=1}^{K}$,

$\displaystyle
h_{E}(\mathbf{z})=\sum_{k=1}^{K}\alpha_{N,k}h(\mathbf{z};\hat{\theta}_{k}).$
(15)

Note that since $h(\mathbf{z};\hat{\theta}_{k})\in\Delta_{n_{c}}$,
$h_{E}(\mathbf{z})\in\Delta_{n_{c}}$, i.e., the result of eqn. 15 is still a
probability distribution over classes.

##  4 Experiments

We provide a comparison of decentralized MSDA methods on 5 visual adaptation
benchmarks, namely: ImageCLEF [38], Caltech-Office 10 [39], Office 31 [2],
Office-Home [40] and Adaptiope [41]. We consider 3 methods from the
decentralized MSDA state-of-the-art, namely: FADA [6], KD3A [7] and Co-MDA
[8]. These methods were chosen due to their relevance, and availability of
source code. Furthermore we consider adaptations of DA methods, such as
$f$-DANN [27, 6] and $f-$WDGRL [42]. Further details on the benchmarks, and on
the hyper-parameter settings of decentralized MSDA methods are provided in the
supplementary materials.

We compare FedDaDiL to other decentralized MSDA strategies over 5 visual DA
benchmarks. We present an overview of our results in table 1. We use standard
evaluation protocols in MSDA, namely, we perform adaptation with a ResNet [37]
backbone. The size of the backbone is selected to agree with previous research
[6, 7, 8]. We run our experiments on a computer with a Ubuntu 22.04 OS, a 12th
Gen Intel(R) CoreTM i9-12900H CPU with 64 GB of RAM, and with a NVIDIA RTX
A100 GPU with 4GB of VRAM.

Table 1: Overview of Domain Adaptation benchmark Benchmark | Backbone | # Samples | # Domains | # Classes
---|---|---|---|---
ImageCLEF | ResNet50 | 2400 | 4 | 12
Caltech-Office 10 | ResNet101 | 2533 | 4 | 10
Office31 | ResNet50 | 3287 | 3 | 31
Office-Home | ResNet101 | 15500 | 4 | 65
Adaptiope | ResNet101 | 36900 | 3 | 123

In the following, we divide our experiments in six parts. First, we explore
the optimization process of FedDaDiL. Second, we show empirically compare our
method with prior art. Third we explore FedDaDiL’s performance with respect
client parallelism. Fourth we explore the communication cost of our method.
Fifth we analyze hyper-parameter sensitivity. Sixth we visualize the alignment
between the target domain and its barycentric reconstruction.

Decentralized Dataset Dictionary Learning. Our empirical analysis is
threefold: (i) how averaging different atom versions impacts our algorithm,
(ii) visualizing the loss-landscape of $f(\mathcal{P},\mathcal{A})$ in a 2-D
subspace and (iii) plotting FedDaDiL’s objective as a function of
communication round. We illustrate our findings on the Caltech-Office 10 and
Office 31 benchmarks.

First, given two versions $\mathcal{P}_{\ell}^{\star}$, $\ell=0,1$, we define
$\mathcal{P}_{t}=(1-t)\mathcal{P}_{0}^{\star}+t\mathcal{P}_{1}^{\star}$,
$t\in[0,1]$. Then, we proceed to evaluate $f(\mathcal{P}_{t},\mathcal{A})$ as
a function of $t$, which is shown in Figure 2(a). Similarly to _FedAVG_ [9],
averaging different atom versions decreases the overall loss, which
empirically validates our decentralized strategy.

Second, we explore Theorem 3.1. empirically. Let
$(\mathcal{P}^{\star},\mathcal{A}^{\star})$ be a dictionary. We define $(u,v)$
within $[-1.5,+1.5]^{2}$, and $(\mathcal{P}_{u},\mathcal{P}_{v})$ such that,
for $\epsilon\sim\mathcal{N}(0,\mathbf{I}_{d})$,
$\mathbf{z}_{i}^{(P_{k,u})}:=\epsilon$ and
$\mathbf{y}_{i}^{(P_{k,u})}:=\mathbf{y}_{i}^{(P_{k}^{\star})}$ (resp. $v$).
Our analysis consists on visualizing,

$\displaystyle f(u,v)$
$\displaystyle=f(\mathcal{P}^{\star}+u\mathcal{P}_{u}+v\mathcal{P}_{v},\mathcal{A}),$

i.e., randomly perturbing the features of the dictionary
$\mathcal{P}^{\star}$, as in Theorem 3.1. As shown in figure 2, the loss has
approximately quadratic level sets on the variables $(u,v)$.

(a) Interpolation of atom versions obtained by clients with same
initialization.

(b) DaDiL’s loss landscape. Colors represent loss values (marked by contour
lines), whereas $(u,v)$ parametrize perturbations.

(c) DiL loss function as a function of communication rounds for various values
of local iterations $E$. Benchmark: Office 31, target: amazon

Figure 2: Analysis $1-$dimensional (a) and $2-$dimensional (b) of DaDiL’s
loss. (a) Similarly to _FedAVG_ [9], interpolating between two atom versions
obtained by clients with a shared initialization decreases the overall loss
value. (b) We illustrate Theorem 3.1. empirically on Caltech-Office 10,
showing that DaDiL’s loss is locally quadratic.

Finally, we analyze how the number of local iterations executed by the
clients, $E$, impacts our federated DiL strategy. At one hand, for an
increasing $E$, one parallelizes the DiL problem, as more iterations are done
within clients before averaging the multiple atom versions. At the other hand,
if the clients perform many local steps, they risk to overfit the atoms to
their own data. Overall, for a wide range of values for $E$, we verify that
DiL’s optimization converges towards a local minima, as shown in figure 2(c).
This illustrate an advantage w.r.t. other decentralized MSDA methods, such as
KD3A [7], who fix $E=1$ in their experiments. We provide further analysis
w.r.t. decentralized MSDA performance in the next section.

Table 2: Experimental Results on decentralized MSDA benchmarks. $\dagger$,
${\ddagger}$ and $\star$ indicates results from [6], [7] and [8] respectively.
$\uparrow$ denotes that higher is better. Additional details on our results
are given in our supplementary materials.

Algorithm | Amazon | dSLR | Webcam | Caltech | Avg. $\uparrow$
---|---|---|---|---|---
FedAVG | 86.1 | 98.3 | 99.0 | 87.8 | 92.8
FedProx | 96.9 | 97.2 | 100.0 | 92.5 | 96.6
$f$-DANN† | 83.4 | 85.9 | 87.1 | 88.5 | 86.3
$f$-WDGRL | 97.9 | 97.1 | 100.0 | 95.6 | 97.6
FADA† | 84.2 | 87.1 | 88.1 | 88.7 | 87.1
KD3A‡ | 97.4 | 98.4 | 99.7 | 96.4 | 97.9
Co-MDA⋆ | 98.2 | 100.0 | 100.0 | 96.9 | 98.8
FedDaDiL-E | 99.0 | 100.0 | 100.0 | 96.1 | 98.7
FedDaDiL-R | 99.0 | 100.0 | 100.0 | 95.6 | 98.6

(a) Caltech-Office 10.

Algorithm | Caltech | Bing | ImageNet | Pascal | Avg. $\uparrow$
---|---|---|---|---|---
FedAVG | 96.7 | 65.8 | 94.2 | 77.5 | 83.6
FedProx | 96.7 | 65.8 | 93.3 | 76.7 | 83.1
$f$-DANN | 96.7 | 64.2 | 87.5 | 80.0 | 82.1
$f$-WDGRL | 92.5 | 63.3 | 86.7 | 74.2 | 79.2
FADA | 95.0 | 64.2 | 90.0 | 74.2 | 80.9
KD3A | 93.3 | 69.2 | 95.5 | 73.3 | 82.8
Co-MDA | 94.2 | 65.0 | 91.5 | 78.0 | 82.2
FedDaDiL-E | 98.3 | 69.2 | 93.3 | 81.6 | 85.6
FedDaDiL-R | 98.3 | 69.2 | 95.0 | 80.0 | 85.6

(b) ImageCLEF.

Algorithm | Amazon | dSLR | Webcam | Avg. $\uparrow$
---|---|---|---|---
FedAVG | 67.5 | 95.0 | 96.8 | 86.4
FedProx | 67.4 | 96.0 | 96.8 | 86.7
$f$-DANN | 67.7 | 99.0 | 95.6 | 87.4
$f$-WDGRL | 64.8 | 99.0 | 94.9 | 86.2
FADA | 62.5 | 97.0 | 93.7 | 84.4
KD3A | 65.2 | 100.0 | 98.7 | 88.0
Co-MDA | 64.8 | 99.8 | 98.7 | 87.8
FedDaDiL-E | 71.2 | 100.0 | 98.2 | 89.8
FedDaDiL-R | 70.6 | 100.0 | 99.4 | 90.0

(c) Office 31.

Algorithm | Art | Clipart | Product | Real-World | Avg. $\uparrow$
---|---|---|---|---|---
FedAVG | 72.9 | 62.2 | 83.7 | 85.0 | 76.0
FedProx | 70.8 | 63.7 | 83.6 | 83.1 | 75.3
$f$-DANN | 70.2 | 65.1 | 84.8 | 84.0 | 76.0
$f$-WDGRL | 68.2 | 64.1 | 81.3 | 82.5 | 74.0
FADA | - | - | - | - | -
KD3A | 73.8 | 63.1 | 84.3 | 83.5 | 76.2
Co-MDA⋆ | 74.4 | 64.0 | 85.3 | 83.9 | 76.9
FedDaDiL-E | 75.7 | 64.7 | 85.9 | 85.6 | 78.0
FedDaDiL-R | 76.5 | 65.2 | 85.9 | 84.2 | 78.0

(d) Office-Home.

Algorithm | Synthetic | Real | Product | Avg. $\uparrow$
---|---|---|---|---
FedAVG | 41.3 | 73.7 | 89.3 | 68.1
FedProx | 38.5 | 70.6 | 87.9 | 65.7
$f$-DANN | 42.2 | 71.6 | 89.1 | 67.4
$f$-WDGRL | 34.7 | 64.3 | 84.4 | 61.1
FADA | - | - | - | -
KD3A | 49.6 | 83.3 | 92.1 | 75.0
Co-MDA | 39.3 | 81.0 | 89.0 | 69.7
FedDaDiL-E | 62.2 | 74.1 | 91.1 | 75.8
FedDaDiL-R | 62.3 | 74.4 | 90.6 | 75.4

(e) Adaptiope.

Decentralized Visual Domain Adaptation. In the following we refer to table
2(e), where we summarize our comparisons. First, _FedAVG_ is a strong baseline
when there is a limited amount of data. Indeed, on small benchmarks such as
ImageCLEF and Office31, it performs better or equally than existing
decentralized MSDA algorithms. This is not true on larger benchmarks, such as
Adaptatiope.

Second, non-i.i.d. federated learning methods, such as _FedProx_ [43] are
approximately equivalent to _FedAVG_. While decentralized MSDA is also
concerned with learning under non-i.i.d. data, the fact that these methods do
not leverage target domain data hinders their performance

Third, our methods, FedDaDiL-E and R are able to outperform _FedAVG_ and other
decentralized MSDA benchmarks without aligning domains. Indeed, FedDaDiL works
in a fundamentally different way than existing decentralized MSDA algorithms,
as it _embraces distributional shift_. The adaptation is done by
_reconstructing_ information on the target domain (FedDaDiL-R, cf. eq. 14), or
by weighting predictions of atom distributions (FedDaDiL-E, cf. eq. 15).

Performance under Client Parallelism. We analyze the performance of FedDaDiL
as we increase the degree of client parallelism, which is a function of the
number of local iterations $E$. As we verified in the last section, for an
increasing $E$ FedDaDiL converges faster w.r.t. the rounds of communication
$r$. In table 3, we explore the performance of FedDaDiL-R and E in comparison
with KD3A [7] and Co-MDA [8] w.r.t. $E$. Note that, in these previous works,
the authors fixed $E=1$ in their experiments. They further verified a
degradation in performance for an increasing $E$.

As we verify empirically in table 3, there is indeed a degradation in KD3A and
Co-MDA with an increasing $E$. Nonetheless, FedDaDiL-R and E keep
approximately the same level of performance. As a result, on top of
_outperforming_ current SOTA, our method is more resilient to parallelism.

Table 3: Adaptation performance w.r.t. client parallelism on Office 31 benchmark. Method | $E=1$ | $E=2$ | $E=4$ | $E=5$ | $E=10$
---|---|---|---|---|---
KD3A | 87.8 | 86.7 | 86.0 | 85.4 | 65.9
CoMDA | 88.0 | 86.6 | 86.7 | 86.6 | 84.2
DaDiL-E | 89.9 | 89.9 | 89.2 | 89.8 | 88.9
DaDiL-R | 89.8 | 90.4 | 89.4 | 90.0 | 89.4

Communication Cost. We compare the communication cost in bits at each round,
between FedDaDiL and conventional decentralized MSDA methods, such as FADA [6]
and KD3A [7]. To do so, we calculate the total number of parameters in
FedDaDiL, which, as we discussed in section 3.3, corresponds to
$|\mathcal{P}|=K\times n\times(d+n_{c})$. Further details on the choice of
$K,n$ and $n_{b}$ are given in our supplementary materials. We then calculate
the number of bits used to communicate these parameters, using 32-bit floating
point precision, and divide it by the number of bits used to encode the
backbone network, using the same precision. Our results are summarized in
figure 3(a).

(a) Communication cost.

(b) Hyper-parameter sensitivity.

Figure 3: In (a), we show the communication cost in % relative to the cost of
communicating the parameters of the backbone. In (b), we show the hyper-
parameter sensitivity of DaDiL on the Office-Home benchmark.

With respect figure 3(a), while we cannot avoid communicating the whole
networks during step 1 (_FedAVG_), the DaDiL step communicates much less
parameters than the network used to encode its inputs. As a result, taking
into account table 3 and figure 3(a), the DaDiL step has better communication
efficiency than previous decentralized MSDA methods.

Parameter Sensitivity. Here, we evaluate the sensitivity of FedDaDiL
adaptation performance with respect its hyper-parameters, namely, number of
samples $n$, batch size $n_{b}$ and number of atoms $K$. On total, this
generates $36$ possible models, whose performance is shown in figure 3(b).
Note that, over the chosen range of hyper-parameters, FedDaDiL has a
performance of $77.6\pm 0.22$, well above the second-best method (KD3A [7],
with $76.9\%$ of domain adaptation performance). This shows that, overall, our
proposed algorithm is robust with respect the choice of its hyper-parameters.
We provide the complete list of hyper-parameters (over all benchmarks) in the
supplementary materials. Furthermore, $n$ and $K$ control the complexity of
our dictionary. From figure 3(b), we see that a small number of atoms (e.g.,
$K\leq 3$) and samples (e.g., $n=1950$) yields the best results. In these
cases, we are strictly reducing the total number of samples, as $1950\times
3=5850<15500$ in the Office-Home benchmark.

Feature Visualization. We compare the alignment of distributions with our
method and those of KD3A and FADA. In our case, we align the target with
$\mathcal{B}(\alpha_{N};\mathcal{P})$, whereas other methods align the target
with the source domains. We visualize these alignments by embedding the
samples in $\mathbb{R}^{2}$ through t-SNE [44] (c.f., figure 4). While the
alignment towards domains Webcam and dSLR works well, aligning the sources
with Amazon is more challenging. In this case FedDaDiL manages to reconstruct
it with dictionary learning, which explains its superior performance.

Dataset Distillation. We explore the use of DaDiL for dataset distillation
[45], i.e., creating a reduced summary for a given dataset. We do so in an
Federated DA (FDA) setting, i.e., without labeled sampels in the target domain
being summarized. As such, we analyze $\hat{B}_{T}$ as a function of
$n_{B}=n_{c}\times\text{SPC}$, for $\text{SPC}\in\\{1,10,20\\}$.

In figure 5 (a-c), we analyze points in $\hat{B}_{T}$ in comparison with
$\hat{Q}_{T}$ through t-SNE. We use the synthetic domain in Adaptiope as
target domain. As SPC grows, one captures progressively better the target
distribution. Furthermore, the confidence of reconstructed labels, measured
through the entropy, increases with SPC, as shown in figures 5 (d-f) and
(g-i). These results agree with previous research involving distillation and
Wasserstein barycenters [46].

Finally, we analyze the performance of summaries created through DaDiL in
figures 5 (j-l), in comparison with random sampling the sources and target
domain. DaDiL summaries’ performance increases rapidly with Samples per Class
(SPC), but becomes constant as we increase the amount of data. As such, DaDiL
achieves novel performance with a summary with $1.67\%$ of the total amount of
samples.

Amazon dSLR Webcam

(a) D2adil

(b) KD3A

(c) FADA

Figure 4: t-SNE embeddings of distribution alignments of FedDaDiL, KD3A and
FADA. Blue points correspond to target domain points, whereas red points
correspond to samples in the barycenter support (FedDaDiL) and source domains
(KD3A and FADA).

(a) SPC=$1$

(b) SPC=$10$

(c) SPC=$20$

(d) SPC=$1$

(e) SPC=$10$

(f) SPC=$20$

(g) SPC=$1$

(h) SPC=$10$

(i) SPC=$20$

(j) Target: Product

(k) Target: Real

(l) Target: Synthetic

Figure 5: Dataset distillation on Adaptiope benchmark. (a-c) show a comparison
between real (blue) and reconstructed data points (green). (d-f) show the
entropy of labels of reconstructed data points. For low values of SPC, samples
have higher label entropy. (g-i) show the distribution of label entropies, in
line with the conclusion of (d-f). Finally, (j-l) compares the performance of
distillation with samples generated by DaDiL, in comparison to random sub-
sampling the source (green) and target (yellow). For SPC $=5$, one reaches
state-of-the-art performance. This represents around 1.67% of the total amount
of samples

##  5 Conclusion

We propose a novel federated algorithm for learning dictionaries of empirical
distributions for federated domain adaptation. The main idea of our approach
is keeping server atoms public, whereas clients’ barycentric coordinates are
private. Our strategy is based on two steps. First, one leans a neural net
encoder through standard _FedAVG_ [9]. Second, we rethink the DaDiL strategy
[5] in a federated setting. Our _end-to-end_ decentralized DA strategy
improves adaptation performance on 5 visual DA benchmarks (table 2(e)). On top
of that, we show that our strategy handles _client parallelism_ better than
previous works (figure 2(c) and table 3), while being relatively lightweight
in comparison with communication deep neural nets’ parameters (figure 3(a)).
We further show that our method is robust with respect its hyper-parameters
(figure 3(b)), and that the alignment between the target domain and its
reconstruction is better than with source domains (figure 4).

## References

  * [1] Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.  Dataset shift in machine learning.  Mit Press, 2008.
  * [2] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In European conference on computer vision, pages 213–226. Springer, 2010.
  * [3] Mei Wang and Weihong Deng.  Deep visual domain adaptation: A survey.  Neurocomputing, 312:135–153, 2018.
  * [4] Koby Crammer, Michael Kearns, and Jennifer Wortman.  Learning from multiple sources.  Journal of Machine Learning Research, 9(8), 2008.
  * [5] Eduardo Fernandes Montesuma, Fred Ngolè Mboula, and Antoine Souloumiac.  Multi-source domain adaptation through dataset dictionary learning in wasserstein space.  In 26th European Conference on Artificial Intelligence, 2023.
  * [6] Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko.  Federated adversarial domain adaptation.  In International Conference on Learning Representations, 2019.
  * [7] Haozhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, and Wei Chen.  Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation.  In ICML, pages 3274–3283, 2021.
  * [8] Xinhui Liu, Wei Xi, Wen Li, Dong Xu, Gairui Bai, and Jizhong Zhao.  Co-mda: Federated multi-source domain adaptation on black-box models.  IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  * [9] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * [10] Ruqi Bai, Saurabh Bagchi, and David I. Inouye.  Benchmarking algorithms for federated domain generalization.  In The Twelfth International Conference on Learning Representations, 2024.
  * [11] Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey J Gordon, and Pradeep Ravikumar.  Fundamental limits and tradeoffs in invariant representation learning.  The Journal of Machine Learning Research, 23(1):15356–15404, 2022.
  * [12] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy.  Optimal transport for domain adaptation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9):1853–1865, 2017.
  * [13] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy.  Joint distribution optimal transportation for domain adaptation.  Advances in neural information processing systems, 30, 2017.
  * [14] Eduardo Fernandes Montesuma and Fred Maurice Ngole Mboula.  Wasserstein barycenter transport for acoustic adaptation.  In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3405–3409, May 2021.
  * [15] Eduardo Fernandes Montesuma and Fred Ngolè Mboula.  Wasserstein barycenter for multi-source domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16785–16793, June 2021.
  * [16] Rosanna Turrisi, Rémi Flamary, Alain Rakotomamonjy, et al.  Multi-source domain adaptation via weighted joint distributions optimal transport.  In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
  * [17] Eduardo Fernandes Montesuma, Fred Ngolè Mboula, and Antoine Souloumiac.  Recent advances in optimal transport for machine learning.  arXiv preprint arXiv:2306.16156, 2023.
  * [18] Martial Agueh and Guillaume Carlier.  Barycenters in the wasserstein space.  SIAM Journal on Mathematical Analysis, 43(2):904–924, 2011.
  * [19] Ievgen Redko, Amaury Habrard, and Marc Sebban.  Theoretical analysis of domain adaptation with optimal transport.  In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18–22, 2017, Proceedings, Part II 10, pages 737–753. Springer, 2017.
  * [20] Fabiola Espinoza Castellon, Eduardo Fernandes Montesuma, Fred Ngolè Mboula, Aurélien Mayoue, Antoine Souloumiac, and Cédric Gouy-Pailler.  Federated dataset dictionary learning for multi-source domain adaptation.  In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5610–5614. IEEE, 2024.
  * [21] Antoine Rolet, Marco Cuturi, and Gabriel Peyré.  Fast dictionary learning with a smoothed wasserstein loss.  In Artificial Intelligence and Statistics, pages 630–638. PMLR, 2016.
  * [22] Morgan A Schmitz, Matthieu Heitz, Nicolas Bonneel, Fred Ngole, David Coeurjolly, Marco Cuturi, Gabriel Peyré, and Jean-Luc Starck.  Wasserstein dictionary learning: Optimal transport-based unsupervised nonlinear dictionary learning.  SIAM Journal on Imaging Sciences, 11(1):643–678, 2018.
  * [23] Alexandros Gkillas, Dimitris Ampeliotis, and Kostas Berberidis.  Federated dictionary learning from non-iid data.  In 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), pages 1–5. IEEE, 2022.
  * [24] Geyu Liang, Naichen Shi, Raed Al Kontar, and Salar Fattahi.  Personalized dictionary learning for heterogeneous datasets.  arXiv preprint arXiv:2305.15311, 2023.
  * [25] Vladimir Vapnik.  Principles of risk minimization for learning theory.  Advances in neural information processing systems, 4, 1991.
  * [26] Dashan Gao, Xin Yao, and Qiang Yang.  A survey on heterogeneous federated learning.  arXiv preprint arXiv:2210.04505, 2022.
  * [27] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016.
  * [28] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [29] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [30] Zhong Meng, Jinyu Li, Yifan Gong, and Biing-Hwang Juang.  Adversarial teacher-student learning for unsupervised domain adaptation.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5949–5953. IEEE, 2018.
  * [31] Zhiqi Yu, Jingjing Li, Zhekai Du, Lei Zhu, and Heng Tao Shen.  A comprehensive survey on source-free domain adaptation.  arXiv preprint arXiv:2302.11803, 2023.
  * [32] L Kantorovich.  On the transfer of masses (in russian).  In Doklady Akademii Nauk, volume 37, pages 227–229, 1942.
  * [33] Gabriel Peyré, Marco Cuturi, et al.  Computational optimal transport: With applications to data science.  Foundations and Trends® in Machine Learning, 11(5-6):355–607, 2019.
  * [34] George B Dantzig.  Reminiscences about the origins of linear programming.  In Mathematical programming the state of the art, pages 78–86. Springer, 1983.
  * [35] Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval, and Nicolas Courty.  Learning with minibatch wasserstein: asymptotic and gradient properties.  In AISTATS, 2020.
  * [36] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  Advances in neural information processing systems, 26, 2013.
  * [37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [38] Barbara Caputo, Henning Müller, Jesus Martinez-Gomez, Mauricio Villegas, Burak Acar, Novi Patricia, Neda Marvasti, Suzan Üsküdarlı, Roberto Paredes, Miguel Cazorla, et al.  Imageclef 2014: Overview and analysis of the results.  In Information Access Evaluation. Multilinguality, Multimodality, and Interaction: 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, UK, September 15-18, 2014. Proceedings 5, pages 192–211. Springer, 2014.
  * [39] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman.  Geodesic flow kernel for unsupervised domain adaptation.  In 2012 IEEE conference on computer vision and pattern recognition, pages 2066–2073. IEEE, 2012.
  * [40] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018–5027, 2017.
  * [41] Tobias Ringwald and Rainer Stiefelhagen.  Adaptiope: A modern benchmark for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 101–110, 2021.
  * [42] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu.  Wasserstein distance guided representation learning for domain adaptation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [43] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  Proceedings of Machine learning and systems, 2:429–450, 2020.
  * [44] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [45] Noveen Sachdeva and Julian McAuley.  Data distillation: A survey.  Transactions on Machine Learning Research, 2023.
  * [46] Eduardo Fernandes Montesuma, Fred Ngolè Mboula, and Antoine Souloumiac.  Multi-source domain adaptation meets dataset distillation through dataset dictionary learning.  In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5620–5624. IEEE, 2024.
  * [47] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.

##  Appendix A Introduction

This appendix is organized as follows. Section 2 provides a proof for Theorem
3.1, and Section 3 provides further details on how we experiment with third-
party code.

##  Appendix B Proof of Theorem 3.1

######  Theorem B.1.

Let $(\mathcal{P},\mathcal{A})$ be a dictionary, and
$\epsilon\in\mathbb{R}^{d}$ be a random perturbation. Let
$\tilde{\mathcal{P}}=\\{\tilde{P}_{k}\\}_{k=1}^{K}$ s.t.,

$\tilde{P}_{k}(\mathbf{z},\mathbf{y})=\dfrac{1}{n}\sum_{i=1}^{n}\delta((\mathbf{z},\mathbf{y})-(\mathbf{z}_{j}^{(P_{k})}+\epsilon,\mathbf{y}_{j}^{(P_{k})})),$

then,

$\displaystyle f(\tilde{\mathcal{P}},\mathcal{A})$
$\displaystyle=f(\mathcal{P},\mathcal{A})+2\epsilon^{T}\nabla_{x}f+\lVert\epsilon\rVert_{2}^{2},$

###### Proof.

Our proof relies on the following observation,

$\displaystyle\tilde{\mathbf{x}}_{i}^{(B_{\ell})}$
$\displaystyle=n\sum_{k=1}^{K}\alpha_{k}\sum_{j=1}^{n}\pi_{ij}^{(k)}\tilde{\mathbf{x}}_{i}^{(P_{k})},$
$\displaystyle=n\sum_{k=1}^{K}\alpha_{k}\sum_{j=1}^{n}\pi_{ij}^{(k)}(\mathbf{x}_{i}^{(P_{k})}+\epsilon),$
$\displaystyle=\mathbf{x}_{i}^{(B_{\ell})}+\epsilon\underbrace{(n\sum_{k=1}^{K}\alpha_{k}\sum_{j=1}^{n}\pi_{ij}^{(k)})}_{=1},$
$\displaystyle=\mathbf{x}_{i}^{(B_{\ell})}+\epsilon.$

so that,

$\displaystyle f(\tilde{\mathcal{P}},\mathcal{A})$
$\displaystyle=\sum_{i=1}^{n}\sum_{i^{\prime}=1}^{n}\pi_{i,i^{\prime}}\lVert\tilde{\mathbf{x}}_{i}^{(B_{\ell})}-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})}\rVert_{2}^{2},$
$\displaystyle=\sum_{i=1}^{n}\sum_{i^{\prime}=1}^{n}\pi_{i,i^{\prime}}\lVert\mathbf{x}_{i}^{(B_{\ell})}+\epsilon-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})}\rVert_{2}^{2},$

here, note that,
$\lVert(\mathbf{x}_{i}^{(B_{\ell})}+\epsilon)-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})}\rVert_{2}^{2}=\lVert(\mathbf{x}_{i}^{(B_{\ell})}-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})})+\epsilon\rVert_{2}^{2}$,
which leads to,

$\lVert\mathbf{x}_{i}^{(B_{\ell})}-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})}\rVert_{2}^{2}+2\epsilon^{T}(\mathbf{x}_{i}^{(B_{\ell})}-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})})+\lVert\epsilon\rVert_{2}^{2}$

then,

$\displaystyle
f(\tilde{\mathcal{P}},\mathcal{A})=\sum_{i=1}^{n}\sum_{i^{\prime}=1}^{n}\pi_{i,i^{\prime}}\biggr{(}$
$\displaystyle\lVert\mathbf{x}_{i}^{(B_{\ell})}-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})}\rVert_{2}^{2}+2\epsilon^{T}(\mathbf{x}_{i}^{(B_{\ell})}-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})})+\lVert\epsilon\rVert_{2}^{2}\biggr{)}.$

Breaking the summation into three terms,

$\displaystyle f(\tilde{\mathcal{P}},\mathcal{A})=f(\mathcal{P},\mathcal{A})$
$\displaystyle+\epsilon^{T}\underbrace{\biggr{(}2\sum_{i=1}^{n}\sum_{i^{\prime}=1}^{n}\pi_{i,i^{\prime}}(\mathbf{x}_{i}^{(B_{\ell})}-\mathbf{x}_{i^{\prime}}^{(Q_{\ell})})\biggr{)}}_{=\nabla_{x}f}+\lVert\epsilon\rVert_{2}^{2}.$

∎

##  Appendix C Experiments

In this section, we give details about the hyper-parameters used in our paper,
for the sake of reproducibility. These concern third-party code open-sourced
on Github or OpenReview from the methods used in our paper.

###  C.1 Reproducing the State-of-the-art

In our experiments, we execute the code of 3 state-of-the-art methods, namely,
FADA111Code available at https://openreview.net/forum?id=HJezF3VYPB [6],
KD3A222Code available at https://github.com/FengHZ/KD3A/tree/master [7] and
Co-MDA333Code available at https://github.com/Xinhui-99/CoMDA [8]. We selected
these methods due to their relevance, and the availability of open source
code. These resources allowed us to make fair comparisons to the existing
methods. Here, we make some remarks about how we run third party code,

  1. 1.

FADA [6] runs the k-Means clustering algorithm within its fit procedure, where
the number of clusters equals the number of classes $n_{c}$. Due to the
internal workings of the authors code, the input for k-Means must have _at
least $n_{c}$ elements_. For datasets with small number of classes, such as
Caltech-Office, ImageClef and Office 31, we were able to run the authors code.
Howerver, the batch size required for running their code on Office-Home ($65$
classes) and Adaptiope ($123$ classes) was beyond the hardware capabilities
used in this work.

  2. 2.

All of our experiments, including those with KD3A, Co-MDA and FADA, use
_torchvision_ ResNets as backbones with pre-trained weights on ImageNet [47].
We use the _IMAGENET1K_V2_ weights.

With our design choices, we were able to improve the average adaptation
performance of KD3A on Office-Home to $76.2\%$, in comparison with what was
previously reported by [8].

###  C.2 Hyper-Parameter Settings

FedAVG has hyper-parameter associated with its training process, i.e., batch
size, number of epochs, learning rate and weight decay. Globally, we use an
Stochastic Gradient Descent (SGD) optimizer with a momentum term of $0.9$. On
all datasets we use a mini-batches of $32$ samples. The training is conducted
for $12$ epochs, where an epoch corresponds to a complete run through the
entire dataset of all clients. The learning rate is $10^{-2}$ and we use a
weight decay term of 5$\times 10^{-4}$. Like previous works on decentralized
MSDA [7, 8], we average clients weights at the end of each epoch.

KD3A and Co-MDA. On their respective repositories, the authors of KD3A and Co-
MDA present the hyper-parameters of their methods. For Adaptiope, we re-use
the hyper-parameters used for DomainNet.

Table 4: Hyper-parameter setting for KD3A and Co-MDA. Benchmark | Batch Size | # Epochs | Confidence Gate | Learning Rate
---|---|---|---|---
ImageCLEF | 32 | 100 | $\\{0.9,0.95\\}$ | $10^{-3}$
Office 31 | 32 | 100 | $\\{0.9,0.95\\}$ | $10^{-2}$
Office Home | 32 | 100 | $\\{0.9,0.95\\}$ | $10^{-2}$
Adaptiope | 32 | 100 | $\\{0.8,0.95\\}$ | $10^{-2}$

FedaDiL. For our method, we performed grid-search on the batch size $n_{b}$,
number of atoms $K$ and number of samples $n$. For the number of atoms, we
search over $K\in\\{2,\cdots,6\\}$. We further parametrize $n_{b}$ and $n$ by
the number of classes $n_{c}$ (c.f., Table 1 in our main paper). We display
the best parameters in table 2 below, alongside the cost of communication for
this set of parameters. Note that, as we explore in our experiments, the
performance of our method is robust with respect the choice of hyper-
parameters.

Table 5: Hyper-parameter setting of DaDiL alongside relative communication
cost (in %) in comparison with transmitting the parameters of a ResNet
network. In all cases, DaDiL is more efficient than communicating the
parameters of a neural net. $\downarrow$ denotes that lower is better.

Benchmark | Backbone | Batch Size | # Atoms | # Samples | Batch Size | # Atoms | # Samples |
---|---|---|---|---|---|---|---|---
|  | DaDiL-R | DaDiL-E |
ImageCLEF | ResNet50 | 240 | 6 | 840 | 180 | 5 | 1440 |
Caltech-Office 10 | ResNet101 | 100 | 3 | 500 | 50 | 3 | 500 |
Office31 | ResNet50 | 465 | 3 | 2170 | 465 | 3 | 1550 |
Office Home | ResNet101 | 520 | 3 | 1950 | 325 | 2 | 1950 |
Adaptiope | ResNet101 | 615 | 4 | 3690 | 615 | 4 | 3690 |


Finite $N$ unitary matrix model

Raghav G. Jha

Perimeter Institute for Theoretical Physics, Waterloo, Ontario N2L 2Y5, Canada
🖄<EMAIL_ADDRESS>
Abstract: We consider one-plaquette unitary matrix model at finite $N$ using
exact expression of the partition function for both SU($N$) and U($N$) groups.

###### Contents

  1. 1 Introduction
  2. 2 Partition function and observables
  3. 3 Results & Conclusions

##  1 Introduction

The study of matrix models has been important in understanding and gaining
deep insights into the structure of gauge theories. Their range of
applicability is wide and some examples include – the geometry of random
surfaces, two-dimensional quantum gravity, statistical mechanics of spin
systems. They also have connections to supersymmetric gauge theories and non-
perturbative formulations of superstring theory. A class of matrix models has
also been studied in the context of exhibiting properties similar to the four-
dimensional theory of strong interactions (QCD). The techniques of strong
coupling expansion and lattice methods [1] along with approximate recursion
relations [2] were pioneered to understand the properties of QCD. Around the
same time, another important direction was pursued by ’t Hooft in finding an
expansion parameter, independent of the scale to study QCD by considering
1/$N$ expansion [3]. In this setting, the Feynman diagrams drawn in the
double-line notation can be organized to follow an expansion in powers of
$N^{\chi=2-2g}$ and were argued to be analogous to the topological expansion
in string theory. The large $N$ limit was then used to solve the two-
dimensional model of QCD in the light-cone gauge which is now also known as ’t
Hooft model [4]. This model has many similarities with the four-dimensional
theory but is exactly solvable in the planar limit. In higher dimensions, the
success has been scarce but it is at least known that the large $N$ limit does
not alter a distinctive feature of QCD – ultraviolet freedom, which is obvious
from the $N\to\infty$ limit of the two-loop QCD $\beta$-function which has the
leading coefficient with the correct sign. Following these developments, the
large $N$ limit of gauge theories have been extensively explored for a wide
class of gauge theories. In this topological limit, a remarkable
simplification occurs, where only the planar diagrams survive, when we keep
$\lambda=g_{\text{YM}}^{2}N$ fixed and take $N\to\infty$. The interplay
between the large $N$ limit of gauge theories and string theory has
significantly improved our understanding of strongly coupled gauge theories
and resulted in the AdS/CFT conjecture. In this work, we consider a unitary
matrix model that is equivalent to two-dimensional pure QCD. The two-
dimensional models such as this have rich behavior but are trivial since gauge
field is non-dynamical (absence of degrees of freedom) and the expectation
value of observables can be calculated exactly as done in [5, 6, 7]. In this
model, the Wilson loop obeys area law for all couplings since the Coulomb
potential is always linear and exhibits both ultraviolet freedom and infrared
prison. The two-dimensional lattice gauge theories with Wilson action were
shown to exhibit third-order phase transition at large $N$. This transition
came as a surprise when it was first observed but now we have a better
understanding through AdS/CFT correspondence that these phase transitions at
large $N$ and finite volume occurs in several holographic models and are
related to transitions between different black hole solutions in supergravity
[8, 9]. The occurrence of this phase transition in the large $N$ limit
signifies that non-analytic behaviour can occur even in the simplest of models
which makes it clear that the strong coupling expansions cannot be usually
smoothly continued to the limit of weak coupling. This two-dimensional lattice
gauge theory with SU($N$) or U($N$) gauge group is often described in terms of
single plaquette action over compact unitary group manifold and will hereafter
be referred to as the GWW model after the authors of [6, 5]. The unitary
matrix models of this kind have also been studied from the holographic point
of view [10, 11]. The interest in this model stems from two main reasons, 1)
This model admits an exact solution for any $N$ and coupling, 2) This model is
closely related to one of the simplest strings (minimal superstring theory)
with manageable non-perturbative description (Type 0 strings in one dimension)
[12]. In fact, it was shown that unitary matrix models in the double scaling
limit ($N\to\infty$ & $\lambda\to\lambda_{\text{critical}}$ with some well-
defined relation between $N$ and $|\lambda-\lambda_{\text{critical}}|$) is
described by the Painlevé II equation [13].

The GWW model and several of its modifications have been well-studied using
various techniques in the $N\to\infty$ limit. The finite $N$ limit has been
surprisingly less explored but recently attracted some attention [14]. Even in
these explorations, the gauge group considered was U($N$) since it is a little
easier to handle and in the planar limit, which is mostly of interest, there
is no distinction with SU($N$) . However, at finite $N$, they have _different_
behaviour and independent studies of both $\mathbb{G}=$ SU($N$) and U($N$)
gauge groups are desirable and to our knowledge no treatment of this unitary
matrix model at finite $N$ for SU($N$) gauge group has been done yet. This
paper aims to fill this gap in the literature. We derive an expression for the
partition function considering SU($N$) gauge group and study observables in
the finite $N$ limit. In case the qualitative behaviour is not severely
altered by addition of matter fields i.e. presuming this phase transition is
not turned into a smooth crossover, then these studies may be useful in
understanding black hole solutions and stringy corrections by considering the
finite $N$ limit of matrix models [15] since the transition at
$\lambda_{\text{critical}}=2$ is supposed to separate the supergravity regime
from the perturbative gauge theory regime. There are only a few matrix models
where the finite $N$ regime is exactly solvable for any coupling and this is
one such model. In this respect and various others, a finite $N$ study of
these class of matrix models would be useful in probing the quantum effects of
gravity using holography.

We briefly outline the structure of the paper. In Section 2, we write down the
exact expression of the partition function for _special unitary matrix_ model
at any coupling and $N$ in terms of determinant of a Toeplitz matrix111Named
after German mathematician Otto Toeplitz (1881 - 1940), each descending
diagonal from left to right in this matrix is constant and matrix element
$(j,k)$ depends on $j-k$ and sketch the well-known phase structure of this
model at large $N$. In Section 3, we numerically calculate the corrections to
the planar result for free energy in both the phases and show that it is
simpler to deduce the instanton corrections in the strong coupling phase since
the $1/N$ (or genus) expansion terminates at genus-zero for U($N$) unitary
model. However, this is not the case for the SU($N$) model which has
contributions to the free energy at small $N$ in both the phases. We also
provide results for the Wilson loop in the SU($N$) matrix model using our
exact expression and show that in the large $N$ limit, they converge to the
known results for the U($N$) unitary model. We end with a brief mention of
some open questions which can be addressed in the context of SU($N$) one-
plaquette matrix model and other related unitary matrix models.

##  2 Partition function and observables

The central object in the study of matrix models is the partition function
which is determined by integration over some compact group manifold. However,
there exists only a handful of models which can be solved exactly [16] and
most of these proceeds by reduction of $N^{2}$ matrix degrees of freedom to
$N$ by exploiting some symmetry in the large $N$ limit. The simplest among all
such matrix models is the well-studied one-matrix model. For a nice review for
unitary matrix models, we refer the reader to [17]. The lattice Wilson action
in two dimensions is equivalent to one-plaquette model by considering the
integration over the unitary group. This has also been studied using character
expansion222It is useful to sometimes think of character expansion as Fourier
expansion on a compact group manifold which is discussed in [18, 19]. It was
shown [2] that in two dimensions, the expansion in terms of characters
constitute the recursion relations through which one can exactly solve the
model. For the large $N$ analysis, the saddle point methods [20] are often
used as was done in [6]. However, the saddle point methods are not useful in
extracting sub-leading orders in the 1/$N$ expansion for which the method of
orthogonal polynomials is usually used as was done in [7]. The general
partition function can be schematically written as:

$Z=\frac{1}{\text{Vol}(\mathbb{G})}\int\prod_{\text{links}\leavevmode\nobreak\
l}\mathscr{D}U_{l}\prod_{\Box}Z_{\Box},$ (2.1)

where $\Box$ denotes the plaquette on whose perimeter we take the ordered
product of the link matrices $U$ of size $N\times N$. In order to compute
expectation values in the two-dimensional model, one has to make a choice
between one-plaquette and heat kernel actions. The partition function of the
two-dimensional Yang-Mills theory based on the heat kernel action is written
in terms of sum over all irreducible representations of the unitary gauge
group. We will use the one-plaquette action in this work similar to [5, 6]
which can be expressed as333Some authors use $g$ to denote $1/\lambda$ or
$2/\lambda$. This distinction is clear from the coupling where phase
transition occurs.:

$S(U)=\frac{N}{\lambda}\Big{[}\mathrm{Tr}\Big{(}\prod_{\Box}U\Big{)}+\mathrm{Tr}\Big{(}\prod_{\Box}U^{\dagger}\Big{)}\Big{]},$
(2.2)

where $\prod_{\Box}U$ is the product of links around a plaquette and shorthand
for
$U_{\mu}(\textbf{n})U_{\nu}(\mathbf{n}+\widehat{\boldsymbol{\mu}})U^{\dagger}_{\mu}(\mathbf{n}+\widehat{\boldsymbol{\mu}}+\widehat{\boldsymbol{\nu}})U^{\dagger}_{\nu}(\mathbf{n}+\widehat{\boldsymbol{\nu}})$.
The convention used is such that $U_{\mu}(\textbf{n})$ denotes a link starting
from site n and going in $\mu$-direction.

It was found that for this model in the $N\to\infty$ limit with fixed
$\lambda=g_{\text{YM}}^{2}N$, one observes a discontinuity in the third
derivative of the free energy corresponding to a third-order phase transition.
This means that one loses the analytic structure for even simple actions in
the large $N$ limit and the continuation from the weak coupling to strong
coupling is non-trivial. This transition is not like the usual phase
transitions in statistical mechanics which occur in the infinite volume limit.
Here, it occurs in a finite volume (single plaquette) for an infinite rank
gauge group.

We denote the full partition function of the model with $\mathcal{Z}$ and
since in two dimensions, we can treat all the plaquettes as independent, we
deal with just a single plaquette [21, 6], which we will refer to as one-
plaquette partition function, $Z=\mathcal{Z}^{1/N_{s}N_{t}}$, where
$N_{s}N_{t}$ are the number of nodes in the lattice and equals the number of
plaquettes. It is given by:

$Z=\int_{\begin{subarray}{c}{\sf U(N)}\leavevmode\nobreak\ \\\
\text{or\leavevmode\nobreak\ }{\sf SU(N)}\leavevmode\nobreak\
\end{subarray}}\mathscr{D}U\exp\Bigg{[}\frac{N}{\lambda}\mathrm{Tr}\Big{(}U^{\dagger}+U\Big{)}\Bigg{]}.$
(2.3)

It is known that the partition function for the U($N$) matrix model given
above can be written in terms of Toeplitz determinant444The determinant of
infinite size Toeplitz matrix has a well-defined asymptotic behaviour limit
due to a theorem by Szegö. given by [22, 7]:

$Z(N,\lambda)=\text{Det}\Bigg{[}I_{j-k}\Big{(}\frac{2N}{\lambda}\Big{)}\Bigg{]}_{j,k=1\cdots
N},$ (2.4)

where $I_{\nu}(x)$ is the modified Bessel function of first kind or Bessel
function of imaginary argument of order $\nu$. Note that the argument of the
Bessel function is twice the prefactor of action in (2.3). The appearance of
partition function in terms of determinant has deep connections to the notion
of integrability and special differential equations. For instance, the
determinant of Toeplitz matrices also play a role in the context of the Ising
model, for a partial set of references, see [23, 24]. Instead of working with
the partition function given in (2.3), one can also consider more general
unitary matrix model with source arbitrary $N\times N$ matrix $A$ as was
considered in [25]. The action for this model is given by:

$Z=\int\mathscr{D}U\exp\mathrm{Tr}\Big{[}UA^{\dagger}+AU^{\dagger}\Big{]},$
(2.5)

where $U$ is product of four ordered links. The exact form of $Z$ was derived
in the large $N$ limit and it was shown that the strong and weak coupling
regimes in this model are characterized by
$(1/N)\mathrm{Tr}(A^{\dagger}A)^{-1/2}$. One gets back the usual GWW model by
setting $A=\mathbb{I}/\lambda$. The model in (2.5) was related to a specific
$N\times N$ Hermitian matrix model following some parameter tuning in [26] and
an exact solution was found. In this work we will only consider (2.3) and
derive the exact expression for the partition function when the integration is
over SU($N$) group in (2.3).

The analysis for SU($N$) is similar to the one for U($N$) except that now we
have the constraint that the product of eigenvalues should satisfy
$\prod_{j=1}^{N}e^{i\alpha_{j}}=1$. We start with the one-plaquette partition
function:

$Z=\int_{SU(N)}\mathscr{D}U\exp\Bigg{[}\frac{N}{\lambda}\Bigg{(}\mathrm{Tr}\prod_{\text{single\leavevmode\nobreak\
}\Box}U+\mathrm{Tr}\prod_{\text{single\leavevmode\nobreak\
}\Box}U^{\dagger}\Bigg{)}\Bigg{]},$ (2.6)

where the measure is given by:

$\mathscr{D}U=\frac{1}{N!}\prod_{j=1}^{N}\underbrace{\sum_{q}\delta\Big{(}\sum_{m=1}^{N}\alpha_{m}-2q\pi\Big{)}}_{\text{{\sf
SU}($N$)\leavevmode\nobreak\
constraint}}\frac{d\alpha_{j}}{2\pi}\prod_{j<k}\Big{(}e^{i\alpha_{j}}-e^{i\alpha_{k}}\Big{)}\Big{(}e^{-i\alpha_{j}}-e^{-i\alpha_{k}}\Big{)},$
(2.7)

and the constraint can be written as:

$\frac{1}{2\pi}\sum_{p=-\infty}^{\infty}e^{ip\sum\alpha_{m}}\leavevmode\nobreak\
\leavevmode\nobreak\ .$ (2.8)

By using the representation of the modified Bessel function as:

$I_{k-j-p}\Big{(}\frac{2N}{\lambda}\Big{)}=I_{j-k+p}\Big{(}\frac{2N}{\lambda}\Big{)}=\frac{1}{2\pi}\int_{0}^{2\pi}\leavevmode\nobreak\
e^{\frac{2N}{\lambda}\cos\alpha}e^{i(j-k+p)\alpha}d\alpha,$ (2.9)

where, $e^{i\alpha}$ are the eigenvalues of $U$ and (2.9) fills the
corresponding $j^{\text{th}}$ and $k^{\text{th}}$ element of the matrix
$\mathcal{M}_{\alpha}=I_{j-k+\alpha}(\frac{2N}{\lambda})$ and using (2.6)
through (2.9), we obtain the partition function for SU($N$) matrix model as:

$\displaystyle
Z(N,\lambda)=\sum_{p=-\infty}^{\infty}\text{Det}\Bigg{[}I_{j-k+p}\Big{(}\frac{2N}{\lambda}\Big{)}\Bigg{]}_{j,k=1\cdots
N}.$ (2.10)

In contrast to the expression for the exact partition function for U($N$)
matrix model i.e. (2.4), there is an additional sum over the index $p$ from
the constraint in (2.7). In this paper, we study different observables using
this partition function. However, for practical purposes of computation using
Mathematica, we simply replace the $\infty$ in (2.10) by a large number. We
checked that $\left|p\right|\leq 15$ suffices i.e.,

$Z(N,\lambda)=\underbrace{\sum_{p=-15,\neq
0}^{15}\text{Det}\Bigg{[}I_{j-k+p}\Big{(}\frac{2N}{\lambda}\Big{)}\Bigg{]}}_{\text{due
to {\sf SU}($N$)\leavevmode\nobreak\
}}+\text{Det}\Bigg{[}I_{j-k}\Big{(}\frac{2N}{\lambda}\Big{)}\Bigg{]}.$ (2.11)

This partition function enables us to evaluate free energy which can be
written as a sum over genus expansion for this model:

$F(N,\lambda)=\sum_{g\in\mathbb{N}}F_{g}N^{2-2g}+\mathcal{O}(e^{-N}),$ (2.12)

where $\mathbb{N}$ denotes the set of non-negative integers. The exact result
for the leading coefficient, $F_{0}$, in the planar limit is given by:

$F_{0}=\hskip 5.69054pt\begin{cases}\frac{-1}{\lambda^{2}}\hskip
108.12047pt\text{$\lambda\geq 2$}\\\
\frac{-2}{\lambda}-\frac{1}{2}\ln\frac{\lambda}{2}+\frac{3}{4}\hskip
48.36967pt\text{$\lambda<2$}\end{cases}.$ (2.13)

Another important observable in unitary matrix models and the one we consider
are the Wilson loops. The finite $N$ analysis of Wilson loops in various
representations for U($N$) gauge theory was recently done in [14]. An
expression for the expectation value of normalized winding Wilson loops
defined as:

$\mathcal{W}_{k}(N,\lambda)=\frac{1}{N}\Big{\langle}\mathrm{Tr}\Big{(}\prod_{\mathcal{C}}U\Big{)}^{k}\Big{\rangle},$
(2.14)

was given in terms of $\mathrm{Tr}(\mathcal{M}_{k}/\mathcal{M}_{0})$ with
$k\in\mathbb{Z}^{+}$ denoting the winding number and the expectation value is
computed over a closed contour $\mathcal{C}$. A similar expression for the
SU($N$) case is yet unknown. Note that like the partition function, we can
write $\mathcal{W}_{k,\mathcal{C}}(N,\lambda)$ as
$W_{k}(N,\lambda)^{N_{s}N_{t}}$, where the contour is of time extent $aN_{t}$
and spatial extent $aN_{s}$. The single winding ($k=1$) Wilson loop is related
to the derivative of the free energy and is given by:

$W_{1}(N,\lambda)=\frac{-\lambda^{2}}{2N^{2}}\frac{\partial\ln
Z}{\partial\lambda}=\hskip 5.69054pt\begin{cases}\frac{1}{\lambda}\hskip
48.36967pt\text{$\lambda\geq 2$}\\\ 1-\frac{\lambda}{4}\hskip
28.45274pt\text{$\lambda\leq 2$}\end{cases}.$ (2.15)

For this U($N$) matrix model as mentioned above, there is another equivalent
definition of $W_{1}$ given by:

$W_{1}(N,\lambda)=\mathrm{Tr}\leavevmode\nobreak\
\Big{(}\frac{\mathcal{M}_{1}}{\mathcal{M}_{0}}\Big{)},$ (2.16)

where $\mathcal{M}_{\alpha}$ is defined below (2.9). We will present results
for the free energy and Wilson loops in Section 3 for the SU($N$) and U($N$)
models.

Even though this matrix model is one of the simplest it has a wide range of
interesting features. In [27], by considering the trans-series solutions of
the pre-string equation to obtain instanton corrections, it was deduced that
the instanton action vanishes at the critical point i.e. $\lambda=2$ and it
was concluded that the GWW transition is caused by the effect of
instantons555Usually, when one thinks of large $N$ limit, it seems that the
instanton corrections which goes as $\exp(-A/g_{s})\sim\exp(-AN/\lambda)$ are
insignificant with $A$ denoting the instanton action. In fact, a more general
form is $\exp(-F(\lambda)N)$, where $F$ is some non-negative function of the
coupling $\lambda$ and proportional to $A$. When $F(\lambda)=0$ corresponding
to vanishing action, the exponentially suppressed instanton contribution to
$1/N$ expansion becomes important. It turns out that for the GWW model this
happens exactly at $\lambda_{\text{critical}}=2$ where the third-order phase
transition takes place. Therefore, one also refers to these as instanton
driven large $N$ phase transitions and physically relate them to condensation
of instantons. We thank M. Mariño for email correspondence regarding the one-
plaquette model and his book ‘Instantons and Large $N$’ for clarification
regarding the instanton contributions. There exist other examples where a
similar phenomenon occurs [28]. The behaviour of the corrections to the planar
result due to $1/N$ and instanton have been well-studied. This model also
exhibits resurgence behaviour thoroughly explored in [29]. One of the striking
features of these studies, which is clearly evident in our results is that the
strong coupling phase has no $1/N$ corrections [7] but only
$\mathcal{O}(e^{-N})$ corrections from the contributions due to instantons. In
GWW model, both gapped and ungapped phases have instanton corrections, albeit
of different nature. In the ungapped phase, the eigenvalues of the holonomy
matrix fill the circle while in the gapped phase they are distributed over
some interval. When the eigenvalue distribution is restricted to some domain,
it is called a one-cut (or single interval) distribution/solution. In the
ungapped phase, the instanton contribution to the free energy can simply be
evaluated by subtracting genus-zero contribution from the total free energy.
As we will see, this does not hold for the SU($N$) model. The distribution of
eigenvalues, which is one of the central objects in these matrix models, have
been studied in both the phases for the U($N$) model and is given by:

$\rho(\lambda,\theta)=\hskip
5.69054pt\begin{cases}\frac{2}{\pi\lambda}\cos\Big{(}\frac{\theta}{2}\Big{)}\sqrt{\frac{\lambda}{2}-\sin^{2}\frac{\theta}{2}}\hskip
48.36967pt\text{$\lambda<2$}\\\
\frac{1}{2\pi}\Big{(}1+\frac{2}{\lambda}\cos\theta\Big{)}\hskip
82.51299pt\text{$\lambda>2$}\end{cases}.$ (2.17)

In the gapped phase, the distribution is only supported on some interval
$\theta\in[-a,a]$ while it is uniform in the ungapped phase. For SU($N$)
unitary matrix model, there are corrections to the distributions above which
was discussed in [30] and will not be further discussed in this work.

Figure 1:  A telegraphic summary describing the phase diagram of the one-
plaquette matrix model at large $N$ and the exact expression of the partition
function for SU($N$) model as given in the main text.

##  3 Results & Conclusions

In this section we present the results obtained using (2.11) and (2.4) for
SU($N$) and U($N$) groups respectively. We primarily focus on the free energy
for different couplings to emphasize behaviours in both phases and at the
critical coupling. Our results converge to expected results in (2.13) when we
take large $N$, while also probing the finite $N$ coefficients i.e. $F_{g}$
with $g\neq 0$ according to (2.12). In the weak coupling limit, $\lambda<2$,
the free energy up to genus-two was calculated in [7] and given by:

$\displaystyle F(\lambda,N)=F_{0}-\frac{1}{N^{2}}\Big{(}\frac{1}{12}-\ln
A-\frac{1}{12}\ln
N-\frac{1}{8}\ln(1-(\lambda/2))\Big{)}-\frac{1}{N^{4}}\Big{(}\frac{3\lambda^{3}}{1024}\frac{1}{(1-(\lambda/2))^{3}}+\frac{1}{240}\Big{)},$
(3.1)

where,
$A=2^{\frac{7}{36}}\pi^{\frac{-1}{6}}\exp\Big{[}\frac{1}{3}+\frac{2}{3}\int_{0}^{\frac{1}{2}}\ln\Gamma(1+x)dx\Big{]}=1.2824271291\cdots$
is the Glaisher-Kinkelin constant and is related to the derivative of zeta
function as, $\zeta^{\prime}(-1)=\frac{1}{12}-\ln A$. A general expression for
genus $g$ free energy $F_{g}(\lambda)$ is also known [7, 13, 27, 14] and can
be written as:

$F_{g}(\lambda)=\frac{B_{2g}}{2g(2g-2)}+\frac{1}{\Big{(}\frac{2}{\lambda}-1\Big{)}^{3g-3}}\sum_{n=0}^{g-2}C_{n}^{g}\lambda^{-n},$
(3.2)

where $B_{2g}$ is the Bernoulli number666We note that if we calculate the
orbifold Euler characteristic of the moduli space of Riemann surfaces of genus
$g$ with $n$ marked points i.e. $\chi(\mathcal{M}_{g,n})$ using Harer-Zagier
formula and set $n=0$ we also obtain $\frac{B_{2g}}{2g(2g-2)}$.

A general expression for the SU($N$) case is unknown but is expected to have
some similarity since the origin of these Bernoulli numbers are in the volume
of the U($N$) group which is similar to that of SU($N$) . In Figure 2, we
compute the free energy for $\lambda=4/3$ and show the results for SU($N$) and
U($N$) models.

Figure 2: The free energy (normalized by $N^{2}$) for $\lambda=4/3$ plotted
against 1/$N$. The dashed green line is (3.1) from [7] and the dashed black
line is the $N\to\infty$ value. The analytic expression corresponding to the
dashed yellow line is yet unknown. Figure 3: The dependence of the free energy
(normalized by $N^{2}$) for SU($N$) and U($N$) model at $\lambda=2$ on $N$. In
the $N\to\infty$ limit, the exact value is $F_{0}=-0.25$.

We then consider $\lambda_{\text{critical}}=2$ and plot the results in Figure
3. The free energy has noticeable difference for SU($N$) and U($N$) groups for
$N<15$ (which is not the case for other couplings) and we have explored up to
$N=100$ for this case. One plausible explanation for this behaviour is that at
the critical coupling the instanton contributions are more important compared
to any other $\lambda$ (for fixed $N$) and the difference between SU($N$) and
U($N$) instanton sectors are therefore most significant. It might be possible
to explain this by a systematic finite $N$ instanton contributions with
SU($N$) group and comparing with the known results in the U($N$) models [27].

Figure 4: The dependence of the free energy (normalized by $N^{2}$) on $N$ for
$\lambda=4$ (strong coupling). There are no $1/N$ corrections for the U($N$)
model while for SU($N$) the genus expansion does not terminate at genus-zero.

We then consider strong coupling ($\lambda>2$). This analysis is more
interesting since the genus expansion terminates at genus-zero in case of
U($N$) , first discussed in [7]. Our results shown in Figure 4 are in complete
agreement with this and we note that there are no $1/N$ corrections in this
ungapped phase when the SU($N$) constraint is not imposed. For SU($N$) model,
our study of the partition function signals that there are corrections and the
genus convergence is subtle (and certainly not genus-0 exact up to instanton
effects) in this case and this deserves further study.

Finally, in Figure 5, we study the Wilson loop by taking the numerical
derivative of the free energy for a range of couplings with $N=4,10$ for both
SU($N$) and U($N$) groups.

Figure 5: The expectation value of Wilson loop against coupling for $N=4,10$
around the critical coupling. The dashed lines (different colours) are the
planar limit result in different phases.

The most distinct feature is that the planar result is approached from
different sides for SU($N$) and U($N$) models (similar to free energy
behaviour) and signifies that the $1/N$ corrections to planar values come with
opposite sign. It will be interesting to compute the exact expression for the
Wilson loop winding around $k$ times as done for the U($N$) model in [14].

A related model to the one we studied here is the ‘double trace model’ for
which the action is $\mathrm{Tr}U\mathrm{Tr}{U}^{\dagger}$ and can be written
in terms of the partition function of GWW model. This model is closely related
to a truncated limit of $\mathcal{N}$ = 4 SYM and in the double scaling limit
exhibits the Hagedorn/deconfinement phase transition. It would be interesting
to understand the finite $N$ limit of this model while not restricting to
U($N$) integral as done in [31].

In this paper, we have given an exact expression for the partition function of
SU($N$) one-plaquette matrix model valid for all $N$ and couplings and
computed exact results for free energy and Wilson loop at finite $N$ for
several couplings. We concluded that the $1/N$ corrections to free energy
vanish for U($N$) matrix model in the ungapped (strongly coupled) phase where
the only corrections to the planar result come from the instanton correction,
while for SU($N$) matrix model, the genus expansion contribution to the free
energy does not terminate at genus-zero. Our results suggest that that the
finite $N$ behaviour of SU($N$) is _very_ different from the U($N$) matrix
model and deserves further analysis. It would be interesting to understand
results for multiply winded Wilson loops and 1/$N$ expansion of the free
energy for SU($N$) along the lines as done for U($N$) group.

### Acknowledgements

Research at Perimeter Institute is supported in part by the Government of
Canada through the Department of Innovation, Science and Economic Development
Canada and by the Province of Ontario through the Ministry of Economic
Development, Job Creation and Trade.

## References

  * [1] K. G. Wilson, “Confinement of Quarks,” Phys. Rev. D10 (1974) 2445–2459.
  * [2] A. A. Migdal, “Recursion Equations in Gauge Theories,” Sov. Phys. JETP 42 (1975) 413.
  * [3] G. ’t Hooft, “A Planar Diagram Theory for Strong Interactions,” Nucl. Phys. B72 (1974) 461. [,337(1973)].
  * [4] G. ’t Hooft, “A Two-Dimensional Model for Mesons,” Nucl. Phys. B75 (1974) 461–470.
  * [5] S. R. Wadia, “A Study of U(N) Lattice Gauge Theory in 2-dimensions,” arXiv:1212.2906 [hep-th].
  * [6] D. J. Gross and E. Witten, “Possible Third Order Phase Transition in the Large N Lattice Gauge Theory,” Phys. Rev. D21 (1980) 446–453.
  * [7] Y. Y. Goldschmidt, “1/$N$ Expansion in Two-dimensional Lattice Gauge Theory,” J. Math. Phys. 21 (1980) 1842.
  * [8] E. Witten, “Anti-de Sitter space, thermal phase transition, and confinement in gauge theories,” Adv. Theor. Math. Phys. 2 (1998) 505–532, arXiv:hep-th/9803131 [hep-th].
  * [9] S. Catterall, R. G. Jha, D. Schaich, and T. Wiseman, “Testing holography using lattice super-Yang-Mills theory on a 2-torus,” Phys. Rev. D97 no. 8, (2018) 086020, arXiv:1709.07025 [hep-th].
  * [10] D. N. Kabat and G. Lifschytz, “Approximations for strongly coupled supersymmetric quantum mechanics,” Nucl. Phys. B571 (2000) 419–456, arXiv:hep-th/9910001 [hep-th].
  * [11] D. N. Kabat, G. Lifschytz, and D. A. Lowe, “Black hole entropy from nonperturbative gauge theory,” Phys. Rev. D64 (2001) 124015, arXiv:hep-th/0105171 [hep-th].
  * [12] I. R. Klebanov, J. M. Maldacena, and N. Seiberg, “Unitary and complex matrix models as 1-d type 0 strings,” Commun. Math. Phys. 252 (2004) 275–323, arXiv:hep-th/0309168 [hep-th].
  * [13] V. Periwal and D. Shevitz, “Unitary Matrix models as exactly solvable String Theories,” Phys. Rev. Lett. 64 (1990) 1326.
  * [14] K. Okuyama, “Wilson loops in unitary matrix models at finite $N$,” JHEP 07 (2017) 030, arXiv:1705.06542 [hep-th].
  * [15] L. Susskind, “Matrix theory black holes and the Gross-Witten transition,” arXiv:hep-th/9805115 [hep-th].
  * [16] V. A. Kazakov, “Solvable matrix models,”  2000\.  arXiv:hep-th/0003064 [hep-th].
  * [17] P. Rossi, M. Campostrini, and E. Vicari, “The Large N expansion of unitary matrix models,” Phys. Rept. 302 (1998) 143–209, arXiv:hep-lat/9609003 [hep-lat].
  * [18] A. B. Balantekin, “Character expansions, Itzykson-Zuber integrals, and the QCD partition function,” Phys. Rev. D62 (2000) 085017, arXiv:hep-th/0007161 [hep-th].
  * [19] A. Bazavov, S. Catterall, R. G. Jha, and J. Unmuth-Yockey, “Tensor renormalization group study of the non-Abelian Higgs model in two dimensions,” Phys. Rev. D99 no. 11, (2019) 114507, arXiv:1901.11443 [hep-lat].
  * [20] F. Green and S. Samuel, “Calculating the large- N phase transition in gauge and matrix models,” Nuclear Physics B 194 no. 1, (Jan, 1982) 107–156.
  * [21] J. M. Drouffe and C. Itzykson, “Lattice Gauge Fields,” Phys. Rept. 38 (1978) 133–175.
  * [22] I. Bars and F. Green, “Complete Integration of U ($N$) Lattice Gauge Theory in a Large $N$ Limit,” Phys. Rev. D20 (1979) 3311.
  * [23] P. Deift, A. Its, and I. Krasovsky, “Toeplitz matrices and toeplitz determinants under the impetus of the ising model. some history and some recent results,” arXiv:1207.4990 [math.FA].
  * [24] T. T. Wu, “Theory of Toeplitz Determinants and the Spin Correlations of the Two-Dimensional Ising Model. I,” Physical Review 149 no. 1, (Sep, 1966) 380–401.
  * [25] E. Brezin and D. J. Gross, “The External Field Problem in the Large N Limit of QCD,” Phys. Lett. 97B (1980) 120–124.
  * [26] E. Brezin and S. Hikami, “Duality and replicas for a unitary matrix model,” JHEP 07 (2010) 067, arXiv:1005.4730 [hep-th].
  * [27] M. Marino, “Nonperturbative effects and nonperturbative definitions in matrix models and topological strings,” JHEP 12 (2008) 114, arXiv:0805.3033 [hep-th].
  * [28] D. J. Gross and A. Matytsin, “Instanton induced large N phase transitions in two-dimensional and four-dimensional QCD,” Nucl. Phys. B429 (1994) 50–74, arXiv:hep-th/9404004 [hep-th].
  * [29] A. Ahmed and G. V. Dunne, “Transmutation of a Trans-series: The Gross-Witten-Wadia Phase Transition,” JHEP 11 (2017) 054, arXiv:1710.01812 [hep-th].
  * [30] M. Campostrini, P. Rossi, and E. Vicari, “Large N phase transition in lattice 2-d principal chiral models,” Phys. Rev. D52 (1995) 395–401, arXiv:hep-lat/9412102 [hep-lat].
  * [31] H. Liu, “Fine structure of Hagedorn transitions,” arXiv:hep-th/0408001 [hep-th].