# Robust Knowledge Adaptation for Federated Unsupervised Person ReID
††thanks: * Corresponding author.

Jianfeng Weng  School of Computer Science
The University of Sydney
Sydney, Australia
<EMAIL_ADDRESS>Kun Hu* School of Computer Science
The University of Sydney
Sydney, Australia
<EMAIL_ADDRESS>Tingting Yao  Department of Electronic Information
Engineering
Dalian Maritime University
Dalian, China
<EMAIL_ADDRESS>Jingya Wang  School of Information Science and Technology
ShanghaiTech University
Shanghai, China
<EMAIL_ADDRESS>Zhiyong Wang  School of Computer Science
The University of Sydney
Sydney, Australia
<EMAIL_ADDRESS>
###### Abstract

Person Re-identification (ReID) has been extensively studied in recent years
due to the increasing demand in public security. However, collecting and
dealing with sensitive personal data raises privacy concerns. Therefore,
federated learning has been explored for Person ReID, which aims to share
minimal sensitive data between different parties (clients). However, existing
federated learning based person ReID methods generally rely on laborious and
time-consuming data annotations and it is difficult to guarantee cross-domain
consistency. Thus, in this work, a federated unsupervised cluster-contrastive
(FedUCC) learning method is proposed for Person ReID. FedUCC introduces a
three-stage modelling strategy following a coarse-to-fine manner. In detail,
generic knowledge, specialized knowledge and patch knowledge are discovered
using a deep neural network. This enables the sharing of mutual knowledge
among clients while retaining local domain-specific knowledge based on the
kinds of network layers and their parameters. Comprehensive experiments on 8
public benchmark datasets demonstrate the state-of-the-art performance of our
proposed method.

###### Index Terms:

Federated learning, unsupervised learning, person re-identification,
contrastive learning, clustering

##  I Introduction

Person re-identification (ReID) aims to identify a given person across non-
overlapped camera views and has been widely studied due to its great potential
for the public security and forensics [1]. Over the years, various types of
methods have been studied, including supervised learning and unsupervised
learning. Recently, cross-domain modelling becomes a new direction in the
field of person ReID, which exploits the diversity of multi-domain datasets to
enhance generalization capability [2, 3, 4]. Conventionally, the majority of
multi-domain ReID solutions adopt a centralised mechanism that gathers all
datasets to a single location for modelling. However, there have been
increasing concerns on such centralised person ReID methods due to data
privacy issues among sharing sensitive personal data across multiple domains
or clients [5].

Figure 1:  Illustrative comparison between centralised learning pipeline and
decentralised pipeline of federated learning.

Therefore, decentralised learning paradigm, specifically federated learning,
has gained increasing attentions recently. It allows models to be trained
collaboratively across multiple sources or clients without data centralisation
[6]. Figure 1 illustrates the comparison between centralised learning and
federated learning. FedAVG [6], for example, aggregated the model parameters
at a central server after a number of local epochs, and the new global model
is re-distributed towards clients as new parameters for local optimization.
Using this simple approach, multiple clients can collaborate without explicit
data sharing. Hence, federated learning has been widely utilised in various
applications where data privacy is seriously concerned, such as facial
expression recognition [7] and medical image segmentation [8].

For Person ReID, a number of recent studies have explored federated learning
strategy [9, 10, 11, 12, 13]. For example, FedReID [10] adopted FedAvg to
improve the generalizability of learned models via 9 multi-domain datasets.
However, most existing methods typically assume that the dataset of each
client is annotated, which can be time-consuming and laborious to obtain. This
limits the scalability for large-scale real-world deployment. FedUReID [9]
attempted to resolve this issue by introducing an unsupervised learning method
for federated Person ReID. Specifically, hierarchical clustering was used to
progressively merge neighbouring clusters during training. Nonetheless,
determining the stopping criteria for clustering on each client is
challenging, as the number of images and identities often varies widely. In
addition, the variation between clients could be impacted by statistical
heterogeneity, such as label skew, feature skew and concept shift [14].
Existing studies often fine-tune the global model using public datasets [13],
and utilise adversarial learning [11] to alleviate such issue. However, a
unified model is still not sufficient to perform well in this scenario,
especially when considering the nature of the extreme statistical
heterogeneity in the person ReID task. Model personalization methods attempted
to incorporate generic and specialized knowledge using different model
parameters [15, 16, 17, 18]. However, these methods have not addressed the
possible domain over-fitting issues yet, with the potential generic knowledge
overwhelmed by potential local knowledge bias.

To address the aforementioned challenges, a three-stage learning strategy in a
coarse-to-fine manner is proposed for federated unsupervised learning based
Person ReID. First, the task is modelled with generic knowledge by a
conventional federated learning scheme based on DBSCAN [19] clustering. This
stage aggregates and distributes all parameters from local clients. Second,
specialized knowledge is explored in pursuit of client personalization. This
is achieved by decoupling the client-specific knowledge from generic knowledge
via parameter localization. Third, we further enhance fine-grained patterns by
incorporating patch-level feature alignment across clients. Unlike holistic
person images, which present potential large variations from changing poses
and view-angles, patch-level information provides fine and consistent person
ReID representations. Finally, comprehensive experiments and analyses have
been conducted to demonstrate the state-of-the-art performance of our proposed
FedUCC on 8 public Person ReID datasets.

In summary, the major contributions of this paper are three-fold as follows:

  * •

A novel unsupervised federated learning method is proposed for Person ReID
with a coarse-to-fine strategy including three learning stages.

  * •

A novel strategy to explore client specific knowledge by incorporating patch-
level representations.

  * •

Comprehensive experiments demonstrate the state-of-the-art performance of the
proposed method on 8 benchmark datasets.

##  II Related Work

###  II-A Unsupervised Person Re-identification

Generally, unsupervised person ReID methods can be categorised into two
streams: unsupervised domain adaptation (UDA) [20, 21, 22] and fully
unsupervised methods [23, 24].

Unsupervised Domain Adaptation for Person ReID is considered as a semi-
supervised learning approach. It involves knowledge adaptation between the
datasets of two domains: a labeled source domain and an unlabeled target
domain. Given a source domain with labeled data, the knowledge from the source
domain is exploited to supervise the learning on an unlabeled target domain.
Existing studies have explored various transfer learning techniques. There
have been attempts [20, 21, 25] to minimize the domain gap by transferring the
source image style to the target image style using a GAN [26], while
preserving the contents. Knowledge distillation (e.g. [22]) and meta-learning
(e.g. [27]) have also been adopted. A prerequisite of these methods is that
both the source domain and the target domain are accessible during the
training stage, which does not meet the criteria for protecting sensitive
private data.

Fully Unsupervised Person ReID can be more challenging as there is no prior
knowledge (i.e., label annotations), compared with the UDA methods.
Clustering-based methods such as Density-Based Spatial Clustering of
Applications with Noise (DBSCAN) and K-mean are popularly used to estimate
image identity relationships [28, 29], which consider all the images in a
cluster as the same class for model learning. However, the noise label on
misclustered images can degrade the performance. Several approaches address
this issue by aligning image feature representations towards cluster centroids
using contrastive learning [30, 31]. Thus, the mislabeled images of the same
identity are not directly optimized to drift apart in the feature space. BUC
(Bottom Up Clustering) [29] employs a hierarchical clustering approach to
iteratively associate similar images, beginning with each image as a single
cluster, and terminate until a predefined number of clusters are remained.
Besides clustering, multi-labels were also used to indicate the identity
relationship among images [32]. For unsupervised learning in the federated
setting, FedUReID [9] followed the BUC [29] strategy to iteratively associate
neighbouring clusters, and designed the cluster stopping criteria according to
the batch precision score. However, the positive correlation between
clustering accuracy and batch precision score cannot be guaranteed as batch
precision scores are highly dependent on the distribution of a dataset. Thus,
both under-clustering and over-clustering remain a potential issue.

(a) A global round contains local training, global aggregation and
distribution.

(b) Three-stage local training.

Figure 2: Illustration of the proposed FedUCC method: (a) a global round, and
(b) three-stage strategy for local training.

###  II-B Federated Learning

Federated learning is proposed to address decentralized training and reduce
the risk of privacy leakage. The intuition of the idea is to conduct knowledge
exchange between the clients with abstract information rather than the raw
inputs which can pose high privacy concerns. For example, trained model
weights and gradients can be transferred for this purpose [6].

As federated learning is a relatively new research topic, the majority of
existing studies focus on supervised learning [8, 10, 11, 7]. And among these
studies, statistical heterogeneity is a core research problem. Several
strategies have been proposed to alleviate its issue. A common strategy is to
adopt extra regularization terms to ensure the model similarity of local
models and averaged global model [33, 34], using distance metric such as
L2-normalization to regularize global model and local model distance [33] or
image feature presentation similarity [34]. While this can limit the impact of
local updates and alleviate the client drift problem, it can restrict the
model’s ability to achieve maximum performance on all clients. Meta-learning
techniques such as MAML(Model-Agnostic Meta Learning) [35] are used to obtain
a good initial global model that can adapt quickly to a new heterogeneous task
with few local gradient descent [36]. However, this strategy is computational
expensive because it requires the computation of second-order gradients. Feddg
[8] attempted to share local image distributions across clients by first
decoupling style and content patterns using Fourier Transform [37]. Then, by
re-synthesizing images with these patterns for image style transfer, it
allowed local models to train on augmented images with the distributions
across clients. However, Fourier transform is only capable of decoding generic
image color and texture, which is difficult to formulate the statistical
heterogeneity between Person ReID datasets.

Compared with the abovementioned approaches which attempt to learn a strong
global model that adapts to all clients, model personalization approach trains
individual models for each client to focus on client-specific patterns [17,
15]. Instead of conducting model aggregation to exchange cross-client
knowledge, knowledge consensus is obtained by gathering image predictions on a
public dataset, and using knowledge distillation on averaged image predictions
to conduct a knowledge exchange [38]. Alternatively, parameter personalization
approaches are proposed to store generic and specialized knowledge on separate
parameters of the model, and the client-specific model is reformulated by
integrating generic parameters and client-specific parameters. [17, 15, 16,
18]. The selections of personal parameters were investigated in various works,
including top layers [15] and bottom layers [16] of the model network, as well
as batch normalization [17] and convolutional channel parameters [18], which
are evenly distributed across the model network depth. These methods, however,
fail to address potential domain over-fitting, in which local specialized
knowledge overtakes generic knowledge, causing performance degrading for the
more biased clients. In this work, we address this by introducing a patch-
level feature alignment with fine-feature consistency across clients.

##  III Proposed Method

###  III-A Overview

Fig. 2a illustrates the overview of the proposed federated learning method -
FedUCC. The overall architecture is composed of a central server and a number
of clients, each of which has a multi-camera ReID dataset captured at
different geographical locations. The task of the central server is to
aggregate the trained models from these local clients, and redistribute an
updated global model. The clients, on the other hand, are responsible for
inheriting the global model’s generic knowledge and conducting local
optimization to properly integrate their specialised local knowledge.

The learning of FedUCC consists of three stages $s=\text{I, II, III}$
following a coarse-to-fine manner. Each training stage contains $T_{s}$ global
round, including $E_{s}$ parallel client-wise (local) epochs and a global
aggregation/redistribution. Particularly, the three stages are: I) Generic
knowledge guided modelling formulates a model by adopting all available
knowledge from the clients. II) Specialized knowledge guided modelling
decouples the client-specific knowledge from the generic knowledge via
parameter localization, and achieves personalized client models; and III)
Patch knowledge guided modelling strengths the fine-grained patterns via local
feature alignment. The details of the proposed method are discussed in the
rest of this section.

###  III-B Stage I: Generic Knowledge Guided Modelling

At this stage, the goal is to obtain a global model that can roughly
discriminate images across all the clients, of which statistical heterogeneity
is not explicitly explored. It helps obtain a robust initial weight for the
model to conduct further training in the next two stages.

Given $K$ clients, where the $k$-th client has a Person ReID dataset
${X^{k}}=\left\\{x^{k}_{1},x^{k}_{2},\ldots,x^{k}_{n^{k}}\right\\}$ with
$n^{k}$ images, our objective is to obtain model $\theta(W_{g},W_{l})$ on
$\mathcal{X}=\\{X^{1},...,X^{k}\\}$. Note that the model $\theta(W_{g},W_{l})$
is parameterized by $(W_{g},W_{l})$ using shared generic parameters $W_{g}$
and client-specific parameters $W_{l}$. Particularly, the specialized
parameters $W_{l}$ are the neuron weights of batch normalization layers and
the generic parameters are the remaining model parameters. We define
($W_{g}^{k}$ ,$W_{l}^{k}$) to be the trained generic parameters and
specialized parameters on the $k$-th client, respectively. At the current
stage we focus on learning robust generic knowledge shared among clients, so
both $(W_{g},W_{l})$ are updated to central server for redistribution. In the
following part, we introduce the training steps taken by clients and the
server respectively.

####  III-B1 Client Design

Person ReID is a metric learning problem, thus at inference, the
representations of image pairs capturing the same person are supposed to be as
close as possible. Therefore, a clustering algorithm is adopted, namely
DBSCAN, where images with distance below a predefined threshold are grouped
into the same cluster.

Denote $\mathcal{H}^{k}=\\{H^{k,1},H^{k,2},\dots,H^{k,m^{k}}\\}$ as the
clusters obtained from the $k$-th client with dataset $X^{k}$, where $m^{k}$
denotes the number of clusters in $X^{k}$. Note that each cluster $H^{k,i}$ is
expected to contain all image features of a particular person. We first obtain
a set of cluster centroids
$\mathcal{C}^{k}=\\{c^{k}_{1},\dots,c^{k}_{m^{k}}\\}$ from $X^{k}$. Each
cluster centroid is obtained by the mean feature vector of all images with the
same pseudo-label:

$c^{k}_{j}=\frac{1}{\left|{H}^{k,j}\right|}\sum_{u^{k}_{i}\in{H}^{k,j}}u^{k}_{i},$
(1)

where $u^{k}_{i}$ denotes the feature representation of image $x^{k}_{i}$,
obtained from the model $\theta(W_{g},W_{l})$, and $\left|{H}^{k,j}\right|$
indicates the number of images in the cluster.

Cluster centroid contrastive learning focuses on cluster-level feature
alignment. For each client $k$, the objective is to minimise the distance of
each image feature to their cluster centroid feature via an InfoNCE loss [39]:

$\mathcal{L}_{\text{u}}=\mathbb{E}\left[-\log\frac{\exp\left(u^{k}_{i}\cdot
c^{k}_{j}/\tau\right)}{\sum_{j^{\prime}=1}^{m^{k}}\exp\left(u^{k}_{i}\cdot
c^{k}_{{}^{\prime}}/\tau\right)}\right],$ (2)

where the distances between the query image feature $u^{k}_{i}$ and its
cluster centroid $c^{k}_{j}$ are minimised and out-cluster centroid distances
are maximised, optimizing with a similar principle as a softmax entropy. And
$\tau$ is a temperature hyper-parameter.

During optimization, the centroid feature vectors $c^{k}_{j}$ of query
instance $u^{k}_{i}$ are momentously updated in the memory dictionary as
follows:

$c^{k}_{j}\leftarrow\lambda c^{k}_{j}+(1-\lambda)u^{k}_{i},$ (3)

where $\lambda\in[0,1]$ is a hyper-parameter controlling the extent to update
the cluster centroid. At the end of local client training, all parameters
$(W^{g},W^{l})$ are forwarded to the server for multi-model aggregation.

####  III-B2 Server Design

The responsibility of the central server is to coordinate client knowledge
sharing via an aggregation step. Basically, a strategy similar to FedAvg [6]
is adopted in this study, which conducts the aggregation at the $(t+1)$-th
global round as follows:

$(W_{g},W_{l})^{t+1}=\sum_{k=1}^{K}\frac{n^{k}}{n}(W_{g}^{k},W_{l}^{k})^{t},$
(4)

where $n=\sum_{k=1}^{K}n^{k}$ are the total number of images of all clients.
Instead of using equivalent average, the weighted sum is determined by the
size of each client dataset. As a result, it allows the larger dataset clients
to contribute richer knowledge for model aggregation, which is expected to
contain more robust Person ReID patterns.

However, the issue of statistical heterogeneity is yet to be addressed.
Existing methods attempted to alleviate this by combining the latest generic
parameters with old specialized parameters to obtain a client-specific model
[13]. However, this doesn’t fully resolve the distribution difference between
clients as the model performance degrades after global
aggregation/redistribution is still severe. Therefore, we further introduce
the next modelling stage to solve this issue.

###  III-C Stage II: Specialized Knowledge Guided Modelling

The first stage has produced a generic model by taking into account the
knowledge of all clients. This process aggregates all client-specific domain
knowledge into the global model. Due to the variety of different client data
distributions, there are inevitably parameter collisions between the local
models. The knowledge aggregation effect could be degraded, hence cause
decreased performance on particular clients. We propose a knowledge guided
modelling stage to further improve model accuracy by considering statistical
heterogeneity. It focuses on retrieving client-specific patterns to conduct
knowledge personalization, which aims to maximize the model performance on all
clients.

This stage is based on the local batch normalization (BN). Specifically, batch
normalization layers are added in the backbone networks to normalize the
activation. They adopt statistics in a mini-batch, which inherently captures
the information related to the client data distribution [17]. To achieve
knowledge personalization, we preserve the learned local BN parameters by
decoupling them from the global aggregation. This not only avoids parameter
collision of the BN weights between clients but also allows the updated global
model to be used in conjunction with the local BN parameters to achieve
personalization. At this stage, the model aggregation and updates are as
follows:

$(W_{g})^{t+1}=\sum_{k=1}^{K}\frac{n^{k}}{n}(W_{g}^{k})^{t}.$ (5)

And model localization for the $k$-th client is as follows:

$\theta^{k,t+1}=\theta((W_{g})^{t+1},(W_{l}^{k})^{t}).$ (6)

Note that BN layers for personalization could lead to sub-optimal performance
due to the over-fitting issue on a client. We further enhance image feature
learning by introducing patch-level representation to enable fine-grained
aspects for facilitating parameter alignment across clients.

###  III-D Stage III: Patch Knowledge Guided Modelling

Patch-based representations show their advantages for the robustness on unseen
identities which are not involved in the training process [40]. Hence,
additional model parameters which are used to characterize patch-level
features are introduced and optimized by following a similar strategy as in
Stage II.

Specifically, given the dataset $X^{k}$ on the $k$-th client, we obtain the
set of 3D tensor features of images before the global average pooling (GAP)
layer in the backbone model as $V^{k}=\\{v^{k}_{1},\dots,v^{k}_{n^{k}}\\}$. We
then extract the set of patch features for all images as
$\mathcal{Q}^{k}=\\{Q^{k}_{1},\dots,Q^{k}_{n^{k}}\\}$ by equally splitting
$v^{k}_{i}$ into $p$ horizontal strips and further adopt GAP to obtain patch-
level image features. In detail, we represent them as
$Q^{k}_{i}=\\{q^{k}_{i,1},\dots,q^{k}_{i,p}\\}$.

Finally, for each cluster $H^{k,j}\in\mathcal{H}^{k}$, we further extend it to
$p$ clusters for features within particular patches, of which the centroids
are: $D^{k}_{j}=\\{d^{k}_{j,1},\dots,d^{k}_{j,p}\\}$ associated with the
corresponding patch-level features.

After that, given a patch feature representation ${q}^{k}_{i,r}$ and a
centroid ${d}^{k}_{j,s}$, a loss function can be defined as follows:

$\mathcal{L}_{\text{p}}=\mathbb{E}\sum_{r=1}^{p}{\left[-\log\frac{\exp\left(q^{k}_{i,r}\cdot
d^{k}_{j,s}/\tau\right)}{\sum_{j^{\prime}=1}^{m^{K}}\exp\left(q^{k}_{i,r}\cdot
d^{k}_{j^{\prime},s}/\tau\right)}\right]},$ (7)

which is similar to Eq.(2).

To this end, a loss function jointly optimized $L_{u}$ and $L_{p}$ can be
adopted to achieve a robust federated Person ReID:

$\mathcal{L}=\mathcal{L}_{u}+\mathcal{L}_{p}.$ (8)

TABLE I: Overall performance with ablation studies on eight Person ReID
datasets.

Methods Duke Market CUHK03 PRID CUHK01 VIPeR 3Dpes iLIDS Avg Metric (%) R1 mAP
R1 mAP R1 mAP R1 mAP R1 mAP R1 mAP R1 mAP R1 mAP R1 mAP FedUReID [9] 51.0 -
65.2 - 8.9 - 38.0 - 43.6 - 26.6 - 65.3 - 73.5 - 46.5 - Backbone 14.4 6.6 42.2
21.3 1.4 2.0 19.9 24.4 26.7 26.1 22.7 28.2 61.2 43.4 63.9 48.5 31.5 25.1 Stage
I 65.7 44.4 74.6 45.6 11.3 9.4 24.0 27.2 60.6 57.3 35.4 39.7 60.3 41.7 81.0
66.2 51.6 41.4 Stage I + II 75.2 55.6 85.7 63.5 8.9 9.0 52.9 57.2 69.5 65.7
31.0 36.1 65.5 46.4 80.1 60.7 58.6 49.3 Stage I + III 76.3 56.3 82.6 58.2 9.6
8.8 46.9 52.1 74.5 71.4 28.1 33.1 65.5 46.6 67.5 53.6 56.4 47.5 FedUCC 78.8
60.5 86.5 65.5 9.6 9.7 58.9 63.1 78.3 75.3 31.3 36.7 68.9 50.9 74.7 59.7 60.9
52.7

TABLE II: Comparisons to the state-of-the-art methods. Methods |  Types | Duke | Market
---|---|---|---
Metric (%) |  |  R1 |  R5 |  R10 |  mAP |  R1 |  R5 |  R10 |  mAP
PUL [1] |  Domain Adaptation |  30.4 |  46.4 |  50.7 |  16.4 |  44.7 |  59.1 |  65.6 |  20.1
HHL [41] |  Domain Adaptation |  46.9 |  61.0 |  66.7 |  27.2 |  62.2 |  78.8 |  84.0 |  31.4
SPGAN [25] |  Domain Adaptation |  46.9 |  62.6 |  68.5 |  26.4 |  58.1 |  76.0 |  82.7 |  26.7
BUC [29] |  Purely Unsupervised |  40.4 |  52.5 |  58.2 |  22.1 |  61.9 |  73.5 |  78.2 |  29.6
MMCL [32] |  Purely Unsupervised |  65.2 |  75.9 |  80.0 |  40.2 |  80.3 |  89.4 |  92.3 |  45.5
FedUReID [9] |  Federated Unsupervised |  51.0 |  62.4 |  67.6 |  29.5 |  65.2 |  77.8 |  82.2 |  34.2
FedUCC (Ours) |  Federated Unsupervised |  78.8 |  87.2 |  89.8 |  60.5 |  86.5 |  94.5 |  96.7 |  65.5

##  IV Experiments

###  IV-A Dataset

Seven Person Re-ID datasets were adopted to comprehensively evaluate our
proposed method, including DukeMTMC-ReID [42], Market1501 [43], CUHK03-NP
[44], PRID [45], CUHK01 [46], VIPeR [47], 3DPeS [48] and iLIDS [49]. Based on
the federated learning setting, each dataset is considered to be held by a
single client. Therefore, each client’s data is collected at a unique
location. This is consistent to real-world scenarios when delivering a Person
Re-ID model with the consideration of the privacy requirements.

Standard Person ReID evaluation metrics were adopted to measure the
performance, including Cumulative Match Characteristic (CMC) curve and mean
Average Precision (mAP) [50]. Given a query image and the gallery containing a
group of watch list images, CMC first ranks the gallery images in line with
their similarity to the query. Then, each image ranked within the rank-K
(i.e., top-K) results is verified whether a matching identity is within the
results or not. In addition, the rank-1 accuracy is introduced to indicate the
average probability of the most similar image matching the query image, and
mAP reflects the mean average precision of all queries.

###  IV-B Experimental Settings

Backbone Architectures ResNet-50 [51] pretrained on ImagNet [52] was adopted
as the backbone network for feature extraction.

Implementation Details For each client, an Adam optimizer with Nesterov
momentum 0.9 was adopted. The learning rate was set to $3.5\times 10^{-4}$ for
local model optimization, and the batch size was set to 64 empirically. The
number of patch-feature segmentation $N_{p}$ was set to 2, indicating the
upper-half and lower-half of the human body. We employed DBSCAN [19] for image
feature clustering based on Jaccard distance with k-reciprocal encoding [53]
and set the number of minimum samples as 2 to discard un-clustered images. The
implementation was with an NVIDIA V100 for training and inference stages.

Training Settings The model parameters were optimized through 3 training
stages as discussed in Sec. III. The numbers of local epoch and global round
differ between the 3 training stages. Stage I training was optimized with
local epoch $E_{I}=1$ and global round $T_{I}=100$, which rapidly aggregated
and updated the model parameters from the local clients. For optimization at
Stage II and Stage III, we empirically set the local epoch $E_{II}=E_{III}=5$
and global round $T_{II}=T_{III}=100$, allowing Stage II and Stage III to
learn sufficient local knowledge. In addition, we concatenated the features
extracted from a part-based model and from the Stage II model for the
clustering and training of Stage III.

Figure 3: Ranking result on three queries

###  IV-C Comparisons with the State-of-the-Art Unsupervised ReID

We compared the proposed method with a number of existing methods following
their evaluation protocols to demonstrate the effectiveness of our method. In
Table I, a recently proposed unsupervised federated Person Re-ID method
FedUReID [13] was compared to ours on 8 datasets. Our proposed method
performed consistently better than FedUReID on all datasets. In particular,
our result outperforms FedUReID with a large margin on the Duke, Market, and
CUHK01 datasets. Note that Duke and Market are the two largest datasets. Such
result indicates significant incremental in terms of the top-1 accuracy from
51.0% to 77.8% on Duke and 65.2% to 86.8% on Market. Other datasets also
demonstrate the effectiveness of FedUCC. CUHK01 shows the largest improvement
regarding the top-1 accuracy 43.6% to 78.3%, PRID shows 20.9% incremental, and
the remaining CUHK03, VIPeR, 3DPeS, iLIDS also show perofrmance improvement of
0.7%, 4.7%, 3.6% and 1.2%, respectively.

In Table II, FedUCC is further compared with two additional categories of
methods including the unsupervised domain adaption (PUL [1], HHL [41], and
SPGAN [25]) and the fully unsupervised approaches (BUC [29] and MMCL [32]).
The evaluation was conducted on the two largest datasets: Duke and Market.
Note that the unsupervised domain adaptation requires the source dataset to
have annotated labels. Without the additional supervision using annotations,
our method still performs better than these domain adaptation methods. For
example, our method and SPGAN [25] achieve 77.8% and 46.9% rank-1 accuracy on
Duke, respectively; our method and HHL [41] achieve 86.8% and 62.2% rank-1
accuracy on Market. Additionally, we compare our results with fully
unsupervised methods, where each ReID model is trained independently on
individual datasets without collaboration. Superior performance was also
consistently demonstrated, given the comparison of our result to MMCL [32]
with 77.8% and 65.2% rank-1 accuracy on Duke, 86.6% and 80.3% on Market. This
also highlights the advantage of collaboration training using federated
learning.

Figure 4:  Rank-1 accuracy curves on all datasets

###  IV-D Ablation Studies

Ablation studies were conducted to demonstrate the effectiveness of the
individual strategies proposed, which helped explore the role of each stage in
the entire pipeline of FedUCC. Particularly, Table I lists the comparisons of
several architectures in terms of Rank-1 accuracy and mAP. These architectures
include a backbone only method, a Stage I only method, Stage I + II method and
Stage I + III method. Overall, our proposed FedUCC achieves the best overall
performance with all these strategies. We discuss the details of these
individual architectures in the following sections.

Backbone Method: Generic Knowledge Guided Modelling The backbone setting
adopted conventional DBSCAN-based label assignment and conducted optimization
using a cross-entropy loss for classification. For federated learning, we
chose a local epoch of 5 that balances the model convergence and the cross-
client knowledge exchange. Each client model was updated with a fixed number
of training batches in line with the dataset size. FedAvg considered this by
limiting smaller client models with fewer averaging ratios for aggregation.
However, skewness can usually be identified among the clients and the local
model parameters were likely to conflict. As a result, the local training led
to significant fluctuation of the global model performance between the global
rounds and eventually limited the person ReID accuracy.

Stage I: Generic Knowledge Guided Modelling with Memory-based Contrastive
Learning Compared to the backbone method, Stage I further introduces the
memory-based contrastive learning on each client as indicated in Eq. (1) -
(3). As shown in TABLE I, the performance was superior on all clients.
Particularly, the Rank-1 accuracy increased on the largest dataset Duke and
Market 51.3% and 32.4% and achieved an average increase of 20.1% on all
datasets. Such performance improvement shows the reduced influence of
statistical heterogeneity. Due to the more effective model training strategy
and the unresolved issue of client knowledge conflict, the model performance
fluctuation on all clients is still severe as illustrated in Fig. 4.

Stage I + II: Specialized Knowledge Guided Modelling with Batch Normalization
Stage II helps explore personalized knowledge, which retained the batch
normalization patterns within each client. Specifically, Stage II has shown an
average of 7.9% increase in terms of the Rank-1 accuracy. Additionally, the
model performance is much more stable compared with Stage I, as shown in Fig.
4. This indicates the problem of statistical heterogeneity is greatly reduced
with effective personal knowledge decoupling. Note that the improvements were
not consistent across all the clients. For smaller datasets such as CUHK03,
VIPeR, and iLIDS, the performance even dropped slightly. It suggested a
potential domain over-fitting issue, such that the generic knowledge from the
shared parameters was overwhelmed by the local knowledge. Therefore, this
motivated us to further explore patch-based feature learning to enhance the
robustness of the feature spaces.

Stage I+III: Patch Knowledge Guided Modelling The patch-based features were
explored for fine-grained person representation. As shown in TABLE I, the
average Rank-1 accuracy has decreased by 1.8% on average compared with using
holistic person feature. However by ensemble both holistic and patch features
(FedUCC), better discriminability was achieved compared with using standalone
representation, with an average increase of rank-1 accuracy by 3.4%
Specifically, consistent performance gain can be observed on the majority of
those datasets (i.e., CUHK03, PRID, CUHK01, VIPeR, 3DPeS, iLIDS) compared with
the strategy of Stage I + II. In our observations, iLIDS shows the highest
performance in Stage I training, and no performance gain has been shown for
later stages of training. Given that it is the smallest dataset, the number of
images is likely insufficient for the Stage I model to converge better.

###  IV-E Qualitative Study

Fig. 3 visualizes the top-10 results of three queries to demonstrate the
effectiveness of our three-stage modelling method. It can be seen that the
backbone-only method retrieves the most incorrect images. Even though the
general appearances are similar, attention to more detailed patterns is
lacking. Visual clues such as the backpack are rarely matched in the
corresponding query results. Stage I shows a much better retrieval accuracy in
Fig. 3, where more discriminative patterns are identified to mine the matching
query images. Stage II training further improves retrieval accuracy with
enhanced ability to mine discriminative patterns despite viewpoint variations.
Better retrieval accuracy are shown with more front and side views of the
person despite the larger visual variation introduced by the backpack. Stage
III model shows a comparable result to Stage II, with more focus on regional
pattern matching. This is indicated by the 10-th negative retrieval image
where the camo patterns in the query’s backpack and short are closely matched.
Lastly, the FedUCC (Stage I + II + III) model ensembles holistic and patch-
level features to obtain a refined feature representation, improving query
results with both holistic and fine-grained patterns, as illustrated in Fig.
3.

##  V Conclusion

In this paper, we present a federated unsupervised cluster-contrastive
learning method, namely FedUCC, to address the Person ReID problem. FedUCC
introduces a three-stage modelling strategy following a coarse-to-fine manner.
In detail, generic knowledge, specialized knowledge and patch knowledge are
explored with a deep neural network. It enables the sharing of mutual
knowledge while retaining local domain-specified knowledge through network
layers and their parameters. Comprehensive experiments on 8 public benchmark
datasets consistently demonstrated the perofrmance improvement of our proposed
FedUCC, compared with existing methods. For our future work, we will consider
the exploitation of outlier images. As during clustering, a small number of
images in a cluster are considered outliers and are discarded to reduce their
negative impacts. However, those are still important assets with the potential
to improve the capability with better feature diversity.

## References

  * [1] S. Qiao, C. Liu, W. Shen, and A. L. Yuille, “Few-shot image recognition by predicting parameters from activations,” in _Computer Vision and Pattern Recognition_ , 2018.
  * [2] Y. Dai, X. Li, J. Liu, Z. Tong, and L.-Y. Duan, “Generalizable person re-identification with relevance-aware mixture of experts,” in _Computer Vision and Pattern Recognition_ , 2021.
  * [3] J. Jia, Q. Ruan, and T. M. Hospedales, “Frustratingly easy person re-identification: Generalizing person re-id in practice,” _British Machine Vision Conference_ , 2019.
  * [4] X. Jin, C. Lan, W. Zeng, Z. Chen, and L. Zhang, “Style normalization and restitution for generalizable person re-identification,” in _Computer Vision and Pattern Recognition_ , 2020.
  * [5] B. Custers, A. M. Sears, F. Dechesne, I. Georgieva, T. Tani, and S. Van der Hof, _EU personal data protection in policy and practice_ , 2019.
  * [6] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial Intelligence and Statistics_ , 2017.
  * [7] D. Shome and T. Kar, “Fedaffect: Few-shot federated learning for facial expression recognition,” in _International Conference on Computer Vision_ , 2021.
  * [8] Q. Liu, C. Chen, J. Qin, Q. Dou, and P.-A. Heng, “Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space,” in _Computer Vision and Pattern Recognition_ , 2021.
  * [9] W. Zhuang, Y. Wen, and S. Zhang, “Joint optimization in edge-cloud continuum for federated unsupervised person re-identification,” in _ACM International Conference on Multimedia_ , 2021.
  * [10] W. Zhuang, Y. Wen, X. Zhang, X. Gan, D. Yin, D. Zhou, S. Zhang, and S. Yi, “Performance optimization of federated person re-identification via benchmark analysis,” in _ACM International Conference on Multimedia_ , 2020\.
  * [11] L. Zhang, Y. Luo, Y. Bai, B. Du, and L.-Y. Duan, “Federated learning for non-iid data via unified feature learning and optimization objective alignment,” in _International Conference on Computer Vision_ , 2021.
  * [12] S. Sun, G. Wu, and S. Gong, “Decentralised person re-identification with selective knowledge aggregation,” _British Machine Vision Conference_ , 2021\.
  * [13] G. Wu and S. Gong, “Decentralised learning from independent multi-domain labels for person re-identification,” in _AAAI Conference on Artificial Intelligence_ , 2021.
  * [14] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends® in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 01 2021.
  * [15] M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary, “Federated learning with personalization layers,” _arXiv preprint arXiv:1912.00818_ , 2019.
  * [16] P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency, “Think locally, act globally: Federated learning with local and global representations,” _arXiv preprint arXiv:2001.01523_ , 2020.
  * [17] X. Li, M. JIANG, X. Zhang, M. Kamp, and Q. Dou, “FedBN: Federated learning on non-IID features via local batch normalization,” in _International Conference on Learning Representations_ , 2021.
  * [18] Y. Shen, Y. Zhou, and L. Yu, “Cd2-pfed: Cyclic distillation-guided channel decoupling for model personalization in federated learning,” in _Computer Vision and Pattern Recognition_ , 2022.
  * [19] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” ser. KDD’96. AAAI Press, 1996.
  * [20] L. Wei, S. Zhang, W. Gao, and Q. Tian, “Person transfer gan to bridge domain gap for person re-identification,” in _Computer Vision and Pattern Recognition_ , 2018.
  * [21] Y. Chen, X. Zhu, and S. Gong, “Instance-guided context rendering for cross-domain person re-identification,” in _International Conference on Computer Vision_ , 2019.
  * [22] Y. Ge, D. Chen, and H. Li, “Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re-identification,” in _International Conference on Learning Representations_ , 2020.
  * [23] T. Isobe, D. Li, L. Tian, W. Chen, Y. Shan, and S. Wang, “Towards discriminative representation learning for unsupervised person re-identification,” in _International Conference on Computer Vision_ , 2021\.
  * [24] S. Xuan and S. Zhang, “Intra-inter camera similarity for unsupervised person re-identification,” in _Computer Vision and Pattern Recognition_ , 2021.
  * [25] W. Deng, L. Zheng, Q. Ye, G. Kang, Y. Yang, and J. Jiao, “Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification,” in _Computer Vision and Pattern Recognition_ , 2018.
  * [26] Z. Zheng, X. Yang, Z. Yu, L. Zheng, Y. Yang, and J. Kautz, “Joint discriminative and generative learning for person re-identification,” in _Computer Vision and Pattern Recognition_ , 2019.
  * [27] Y. Zhao, Z. Zhong, F. Yang, Z. Luo, Y. Lin, S. Li, and N. Sebe, “Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification,” in _Computer Vision and Pattern Recognition_ , 2021.
  * [28] Z. Ji, X. Zou, X. Lin, X. Liu, T. Huang, and S. Wu, “An attention-driven two-stage clustering method for unsupervised person re-identification,” in _European Conference on Computer Vision_ , 2020.
  * [29] Y. Lin, X. Dong, L. Zheng, Y. Yan, and Y. Yang, “A bottom-up clustering approach to unsupervised person re-identification,” in _AAAI Conference on Artificial Intelligence_ , 2019.
  * [30] M. Wang, B. Lai, J. Huang, X. Gong, and X.-S. Hua, “Camera-aware proxies for unsupervised person re-identification,” in _AAAI Conference on Artificial Intelligence_ , 2021.
  * [31] Z. Dai, G. Wang, S. Zhu, W. Yuan, and P. Tan, “Cluster contrast for unsupervised person re-identification,” _arXiv preprint arXiv:2103.11568_ , 2021.
  * [32] D. Wang and S. Zhang, “Unsupervised person re-identification via multi-label classification,” in _Computer Vision and Pattern Recognition_ , 2020.
  * [33] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” _Machine Learning and Systems_ , 2020.
  * [34] Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in _Computer Vision and Pattern Recognition_ , 2021.
  * [35] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning_ , 2017.
  * [36] A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning: A meta-learning approach,” _arXiv preprint arXiv:2002.07948_ , 2020.
  * [37] H. J. Nussbaumer, “The fast fourier transform,” in _Fast Fourier Transform and Convolution Algorithms_ , 1981.
  * [38] D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” _arXiv preprint arXiv:1910.03581_ , 2019.
  * [39] A. Van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [40] Q. Yang, H.-X. Yu, A. Wu, and W.-S. Zheng, “Patch-based discriminative feature learning for unsupervised person re-identification,” in _Computer Vision and Pattern Recognition_ , 2019.
  * [41] Z. Zhong, L. Zheng, S. Li, and Y. Yang, “Generalizing a person retrieval model hetero-and homogeneously,” in _European Conference on Computer Vision_ , 2018\.
  * [42] E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in _European Conference on Computer Vision_ , 2016.
  * [43] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable person re-identification: A benchmark,” in _International Conference on Computer Vision_ , 2015.
  * [44] W. Li, R. Zhao, T. Xiao, and X. Wang, “Deepreid: Deep filter pairing neural network for person re-identification,” in _Computer Vision and Pattern Recognition_ , 2014.
  * [45] M. Hirzer, C. Beleznai, P. M. Roth, and H. Bischof, “Person re-identification by descriptive and discriminative classification,” in _Scandinavian Conference on Image Analysis_ , 2011.
  * [46] W. Li, R. Zhao, and X. Wang, “Human reidentification with transferred metric learning,” in _Asian Conference on Computer Vision_ , 2012.
  * [47] D. Gray and H. Tao, “Viewpoint invariant pedestrian recognition with an ensemble of localized features,” in _European Conference on Computer Vision_ , 2008.
  * [48] D. Baltieri, R. Vezzani, and R. Cucchiara, “3dpes: 3d people dataset for surveillance and forensics,” in _Joint ACM Workshop on Human Gesture and Behavior Understanding_ , 2011.
  * [49] T. Wang, S. Gong, X. Zhu, and S. Wang, “Person re-identification by video ranking,” in _European Conference on Computer Vision_ , 2014.
  * [50] L. Zheng, Y. Yang, and A. G. Hauptmann, “Person re-identification: Past, present and future,” _arXiv preprint arXiv:1610.02984_ , 2016.
  * [51] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Computer Vision and Pattern Recognition_ , 2016.
  * [52] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Computer Vision and Pattern Recognition_ , 2009.
  * [53] Z. Zhong, L. Zheng, D. Cao, and S. Li, “Re-ranking person re-identification with k-reciprocal encoding,” in _Computer Vision and Pattern Recognition_ , 2017.


# Intrinsic versus extrinsic orbital and electronic reconstructions at complex
oxide interfaces

R. J. Green<EMAIL_ADDRESS>Stewart Blusson Quantum Matter Institute,
University of British Columbia, Vancouver, Canada V6T 1Z1  Department of
Physics & Engineering Physics, University of Saskatchewan, Saskatoon, Canada
S7N 5E2  V. Zabolotnyy  Experimentelle Physik IV and Röntgen Research Center
for Complex Materials (RCCM), Fakultät für Physik und Astronomie, Universität
Würzburg, Am Hubland, D-97074 Würzburg, Germany  M. Zwiebler  Institute for
Structure Physics, Dresden Technical University, 01062 Dresden, Germany  Z.
Liao  Faculty of Science and Technology and MESA+ Institute for
Nanotechnology, University of Twente, 7500 AE Enschede, The Netherlands
National Synchrotron Radiation Laboratory, University of Science and
Technology of China, Hefei 230026, Anhui, China  S. Macke  Stewart Blusson
Quantum Matter Institute, University of British Columbia, Vancouver, Canada
V6T 1Z1  Max Planck Institute for Solid State Research, Heisenbergstraße 1,
70569 Stuttgart, Germany  R. Sutarto  Canadian Light Source, University of
Saskatchewan, Saskatoon, SK S7N 2V3, Canada  F. He  Canadian Light Source,
University of Saskatchewan, Saskatoon, SK S7N 2V3, Canada  M. Huijben  Faculty
of Science and Technology and MESA+ Institute for Nanotechnology, University
of Twente, 7500 AE Enschede, The Netherlands  G. Rijnders  Faculty of Science
and Technology and MESA+ Institute for Nanotechnology, University of Twente,
7500 AE Enschede, The Netherlands  G. Koster  Faculty of Science and
Technology and MESA+ Institute for Nanotechnology, University of Twente, 7500
AE Enschede, The Netherlands  J. Geck  Institute for Structure Physics,
Dresden Technical University, 01062 Dresden, Germany  V. Hinkov
Experimentelle Physik IV and Röntgen Research Center for Complex Materials
(RCCM), Fakultät für Physik und Astronomie, Universität Würzburg, Am Hubland,
D-97074 Würzburg, Germany  G. A. Sawatzky  Stewart Blusson Quantum Matter
Institute, University of British Columbia, Vancouver, Canada V6T 1Z1

###### Abstract

The interface between the insulators LaAlO3 and SrTiO3 accommodates a two-
dimensional electron liquid (2DEL)—a high mobility electron system exhibiting
superconductivity as well as indications of magnetism and correlations. While
this flagship oxide heterostructure shows promise for electronics
applications, the origin and microscopic properties of the 2DEL remain
unclear. The uncertainty remains in part because the electronic structures of
such nanoscale buried interfaces are difficult to probe, and is compounded by
the variable presence of oxygen vacancies and coexistence of both localized
and delocalized charges. These various complications have precluded decisive
tests of intrinsic electronic and orbital reconstruction at this interface.
Here we overcome prior difficulties by developing an interface analysis based
on the inherently interface-sensitive resonant x-ray reflectometry. We
discover a high charge density of 0.5 electrons per interfacial unit cell for
samples above the critical LaAlO3 thickness, and extract the depth dependence
of both the orbital and electronic reconstructions near the buried interface.
We find that the majority of the reconstruction phenomena are confined to
within 2 unit cells of the interface, and we quantify how oxygen vacancies
significantly affect the electronic system. Our results provide strong support
for the existence of polarity induced electronic reconstruction, clearly
separating its effects from those of oxygen vacancies.

The complex interplay between charge, spin, orbital, and lattice degrees of
freedom in transition metal oxides leads to a multitude of emergent phenomena
having numerous potential applications.Khomskii (2014) Often the phenomena are
realized through explicit fine tuning of these degrees of freedom, for example
via electronic doping or applied pressure. Recent advances in atomic scale
oxide thin film growth have yielded a new method of precisely tuning material
properties via heterostructuring, where sequences of films are grown
epitaxially on top of specially chosen substrates.Mannhart and Schlom (2010);
Hwang  _et al._ (2012) Orbital, spin, and electronic reconstructions which can
occur at the atomically precise interfaces lead to emergent properties that
are often very different from the corresponding bulk materials.Hwang  _et al._
(2012)

The interface between LaAlO3 and SrTiO3 (LAO/STO) is paradigmatic of the oxide
heterostructure field, hosting a two dimensional electron liquid (2DEL) Ohtomo
and Hwang (2004); Thiel  _et al._ (2006) that exhibits gate-tunable
superconductivity,Manca  _et al._ (2019); Pai  _et al._ (2018); Trevisan  _et
al._ (2018); Thierschmann  _et al._ (2018); Reyren  _et al._ (2007); Li  _et
al._ (2011) flexoelectricity,Zhang  _et al._ (2019) magnetism,Park  _et al._
(2020); Brinkman  _et al._ (2007); Bert  _et al._ (2011) and
correlations.Breitschaft  _et al._ (2010); Khanna  _et al._ (2019) Recent
experiments have further utilized this interface as a foundation for the
construction of one-dimensional superconducting electron waveguides Pai  _et
al._ (2018); Briggeman  _et al._ (2020a, b) which have potential application
in quantum computing. However, while the system exhibits such fascinating
properties and shows such promise for device applications,Mannhart and Schlom
(2010) uncertainty remains regarding the origin and physical details of this
highly studied interfacial electronic liquid.Geondzhian  _et al._ (2020) Some
of the earliest studies of the interface 2DEL Nakagawa  _et al._ (2006)
proposed that it results from an electronic reconstruction due to a _polar
catastrophe_ —that a diverging electric potential originating from charged
LaO+ and AlO${}_{2}^{-}$ layers is rectified through a spontaneous relocation
of 0.5 e- per square unit cell (u.c.) from the LAO surface to the interface
for heterostructures with LAO thicknesses greater than 3 u.c., thus forming
the 2DEL. The concept of electronic reconstruction had been introduced several
years earlier in studies of polar surfaces.Hesper  _et al._ (2000) Crucially,
however, transport measurements on LAO/STO typically find about an order of
magnitude less charge than expected from electronic reconstruction.Thiel  _et
al._ (2006) More recent studies have shed some light on this issue, showing
that a certain amount of charge is localized and thus hidden to transport
measurements.Sing  _et al._ (2009); Asmara  _et al._ (2014)

Figure 1: Theoretical resonant reflectometry of LaAlO3/SrTiO3
heterostructures. a, Experimental geometry for the reflectometry measurements,
where the photon polarization is either within ($\pi$) or perpendicular to
($\sigma$) the scattering plane. Specific geometries are characterized by
either the reflection angle $\theta$ or the photon momentum transfer in the
$z$ direction, $q_{z}$. b, The graphic depicts orbital and electronic
reconstructions at the interface via the orbital symmetry and opacity,
respectively. For each atomic layer with a given orbital occupation, there
exist isotropic and dichroic resonant scattering responses shown, which then
combine to determine the reflectivity response of the material. c-d, Simulated
reflectivity maps for $\pi$-polarized light and energies spanning the Ti
$L_{2,3}$ resonance on a 10 u.c. LAO/STO heterostructure in the case of no
reconstructions (c), and combined orbital and electronic reconstructions (d).

In addition to the total charge and its confinement profile near the
interface, a crucial characteristic of the 2DEL is its orbital symmetry, which
directly controls the mobility of the electrons and contributes to the
energetic landscape of magnetism and superconductivity.Trevisan  _et al._
(2018); Manca  _et al._ (2019) The interface electrons reside in the Ti $3d$
$t_{2g}$ band, which is split into $d_{xy}$ and $d_{xz/yz}$ subbands due to
the tetragonal symmetry breaking imposed by the interface. First principles
calculations predict that the $d_{xy}$ bands are lower in energy near the
interface and thus host the majority of the charge, whereas $d_{xz/yz}$ bands
become more occupied further from the interface Popović  _et al._ (2008); Son
_et al._ (2009); Stemmer and James Allen (2014). X-ray absorption spectroscopy
(XAS) experiments have verified this qualitatively, as linear dichroism
measurements agree with the presence of lower energy $d_{xy}$
orbitals.Salluzzo  _et al._ (2009) However, the magnitude of this orbital
reconstruction, and its spatial dependence moving away from the interface,
have yet to be determined.

Here we study the electronic structure of the LAO/STO interface using
synchrotron-based resonant x-ray reflectometry (RXR). RXR is the union of two
powerful, complementary techniques: x-ray absorption spectroscopy, which
probes the element-specific electronic structure (in particular the local $3d$
orbital occupations) through core-valence resonant electronic excitation, and
reflectivity, which inherently contributes excellent interface and depth
sensitivity while also providing a natural quantitative detection scheme.Macke
_et al._ (2014); Macke and Goering (2014) This combination of capabilities
allows us to study the microscopic electronic structure properties specific to
the 2DEL at the interface in exceptional detail. Previous studies on LAO/STO
superlattices utilized a qualitative analysis of limited datasets to study
electronic and orbital reconstructions.Wadati  _et al._ (2009); Park  _et al._
(2013) Here we leverage the full power of RXR via a fully quantitative
analysis of large data sets. In an extensive set of experiments, we apply RXR
to LAO/STO samples below and above critical thickness for 2DEL formation, and
before and after annealing in an $\mathrm{O_{2}}$ environment to eliminate
potential oxygen vacancies, showing how orbital and electronic reconstruction
define the spontaneous 2DEL formation.

Figure 2: Resonant reflectometry data and extracted charge profile for as-
grown LaAlO3/SrTiO3. a-c, Experimental and modelled $\pi$-polarized energy
scans across the Ti resonance at indicated fixed momentum transfer values or
reflection angles for 3 u.c. (a), 5 u.c. (b), and 10 u.c. (c) samples. d,
B-site cation concentrations and 2DEL charge depth profile near the interface.
Each vertical bar represents an atomic layer, so the distance between each
pair is one unit cell. The dashed vertical lines denote the approximate
interface where equal amounts of Ti and Al are found. Extended data sets and
simulation results can be found in the supplementary material.

In Figure 1(a) we depict the RXR experiment, showing the typical geometry for
our reflection measurements using linearly polarized synchrotron x-rays. By
changing the reflection angle or energy, one changes the momentum transfer
$q_{z}$, leading to different interference effects (partial cancellation or
enhancement of waves reflecting at different depths) and different interface
sensitivity. Figure 1(b) shows via simulations the origin of reconstruction
sensitivity in RXR. For the atomic planes near the interface exhibiting a
particular reconstruction effect (where in the inset we depict orbital
reconstructions via schematic orbital symmetries and electronic
reconstructions via the opacity of the orbitals), the Ti atoms have very
specific and different resonant scattering characteristics. Both the isotropic
and linear-dichroic responses of the scattering show strong dependences on
orbital occupation at these Ti $L_{2,3}$ resonance energies.

The resonant atomic scattering responses combine to yield the electric
susceptibility of the material, and thus a particular reconstruction at the
interface leads to a specific reflectivity response, which we investigate with
model data in Fig. 1(c-d). Here the predicted RXR for a 10 u.c. LAO/STO sample
is shown for energies spanning the Ti $L_{2,3}$ resonance and reflection
angles spanning a typical measurable range.111Note we multiply the
reflectivity spectra here by $q_{z}^{4}$ in order to show a wide $q_{z}$ range
where the intensity spans several orders of magnitude As shown by the two
plots, at high momentum transfers, very pronounced effects are expected in the
RXR data due to orbital and electronic reconstructions, even if they are
confined to only 2 u.c. near the buried interface as in the realistic example
simulated here. This strong sensitivity, and the quantitative nature of RXR,
makes this technique very useful for studying oxide interfaces.

We have applied RXR to a series of samples to perform a comprehensive study of
the LAO/STO reconstruction phenomena. Figure 2 details the reflectivity data
and analysis results for samples having LAO thicknesses of 3, 5, and 10 u.c.
Fig. 2(a-c) show selected reflectivity data along with model fit results
(complete datasets are presented in the supplemental material). Signatures of
the thickness-dependent onset of the 2DEL are immediately evident from the
data, as the 3 u.c. sample exhibits a less featured reflectivity response
similar to Fig. 1(c), whereas the spectra of the thicker samples reveal a much
more detailed structure, comparable to Fig. 1(d).

We model the depth-dependent electric susceptibility of the heterostructures
starting from the atomic level, where the Ti resonant scattering response is
computed using quantum many-body theory Haverkort  _et al._ (2012, ) while the
off-resonant scattering response of all atoms are taken from tabulated
data.Henke  _et al._ (1993) The resonant response depends strongly on the
local $3d$ orbital occupation of the Ti atoms, thus providing the sensitivity
to the 2DEL density and symmetry. From the susceptibility we compute the
energy- and angle-dependent reflectivity response, and fit the material
parameters as described in the Methods section. The fit results are overlaid
with the experimental data in Fig. 2(a-c), where an excellent level of
agreement is evident. The models corresponding to these fits, which quantify
the interfacial reconstruction phenomena, are shown in Fig. 2(d) and Fig. 3.
In Fig. 2(d) we plot fractional atomic site occupations for the cations in the
samples (full concentration profiles, including anions, are presented in the
supplemental material), where our resonant Ti analysis allows us to
distinguish between Ti4+ and Ti3+ valences (i.e. the respective absence or
presence of the 2DEL charge). Evident is the $\sim$1-2 u.c. average roughness
at both the surface and the interface, comparable to which has been found with
other techniques.Willmott  _et al._ (2007); Nakagawa  _et al._ (2006);
Chambers  _et al._ (2010) Strong evidence is present for the the 4 u.c.
threshold for 2DEL formation, as the 3 u.c. sample is found to have
significantly less charge than the thicker samples. As shown in Table 1, for
the 3 u.c. sample our fit converges with a small amount of charge at 0.16
e-/u.c. This charge is distributed rather evenly over 4 nm near the interface,
whereas the thicker samples have much more pronounced concentrations confined
to the interface. The total charge quantities in the 5 and 10 u.c. samples are
significant, as also shown in Table 1.

Sample |  Charge (e-/u.c.)
---|---
3 u.c. | 0.16 $\pm$ 0.03
5 u.c. | 0.59 $\pm$ 0.05
10 u.c. | 0.97 $\pm$ 0.08
10 u.c. (annealed) | 0.50 $\pm$ 0.03
Table 1:  Total charge hosted in the Ti $3d$ orbitals near the interface for
each sample, as extracted from the RXR fits.  Figure 3: Orbital reconstruction
of as-grown LAO/STO quantified via reflectometry. a, $t_{2g}$ and b, $e_{g}$
orbital energies for the Ti atomic layers near the LAO/STO interface, relative
to the bulk orbital energies.

In Fig. 3, the extracted interface induced symmetry breaking of the Ti 3d
orbitals is detailed. While the 3d orbitals have $O_{h}$ symmetry in the bulk
of the STO, the presence of the interface leads to a tetragonal local symmetry
distortion.Salluzzo  _et al._ (2009) By computing the tetragonally symmetric
Ti resonant scattering tensor according to the energies of the 3d orbitals,
and fitting to our polarization-dependent RXR spectra, we can extract the
depth dependence of the orbital energies relative to those of the bulk. For
all samples (regardless of 2DEL presence), we detect a strong symmetry
breaking in the 2 u.c. closest to the interface. We find the $d_{xy}$ orbital
to be lowest in energy near the interface, as predicted by first principles
modelling Popović  _et al._ (2008); Son  _et al._ (2009) and qualitatively
observed experimentally.Salluzzo  _et al._ (2009) Further, we detect that the
tetragonal distorting of orbital energies is enhanced in the presence of the
2DEL, a feature also previously predicted from first principles Stemmer and
James Allen (2014). Given the different effective masses for $d_{xy}$ versus
$d_{xz/yz}$ carriers, a detailed quantification of this orbital symmetry
breaking is key to understanding the comprehensive transport properties of the
LAO/STO system.

Figure 4: Effects of post-growth oxygen annealing. a, Experimental and
modelled fixed angle, $\sigma$-polarized, reflectometry scans across the Ti
resonance are shown for a 10 u.c. LAO/STO after annealing in an O2
environment. The experimental data for the same sample before annealing are
shown in grey for comparison. b, 2DEL charge profile after annealing. c,
Orbital energies after annealing, extracted from the reflectometry analysis.

While a common observation in many experimental studies is that the 2DEL total
charge is much less than the 0.5 e- per square u.c. predicted by the polar
catastrophe model,Thiel  _et al._ (2006) we find here quantities relatively
close to this value—in fact they are systematically slightly higher. Oxygen
vacancies (either in the LAO film or the STO substrate) can be an electron
dopant, leading to excess carriers in the 2DEL as well as electron
trapping.Yin  _et al._ (2020) To investigate the possible effect of oxygen
vacancies, we measured an RXR dataset on the same 10 u.c. sample, immediately
after annealing in a partial pressure (200 mbar) O2 environment. The results,
shown in Fig. 4, display a significant change in the RXR response upon
annealing, indicating a strong sensitivity of RXR to the oxygen content and
corresponding Ti charge variation. The data and model fits are shown in Fig.
4(a), with the pre-annealing data plotted as well for comparison. Performing
the same quantitative analysis for this case, we find that the charge density
is reduced from 0.97 e-/u.c. down to a post-annealed value of 0.50 $\pm$ 0.03
e-/u.c., in agreement with the predicted value of 0.5 e-/u.c. within our
fitting uncertainty. This decrease in 2DEL total charge is consistent with
what we observe in transport measurements (see supplemental material). The
charge profile is plotted in Fig. 4(b), where it is evident the total charge
decreases post-annealing and also becomes more confined to the interface
(without considering interface broadening, the mean 2DEL charge depth is 0.4
u.c. past the first TiO2 layer post-annealing versus 1.5 u.c. before).
Finally, in Fig. 4(c) we show the post-annealing orbital energy profiles.
Interestingly, the orbital energies after annealing exhibit a much weaker
tetragonal splitting (23 versus 133 meV), suggesting that the larger orbital
splittings for the 10 u.c. sample prior to annealing in Fig. 3 could be due
primarily to strong potential variations due to oxygen vacancies.Pavlenko  _et
al._ (2012) Further, the $d_{xy}$ orbital is shifted to a lower energy at the
interface post-annealing, explaining the tighter confinement of the 2DEL
charge.

Our resonant x-ray reflectivity analysis of LaAlO3/SrTiO3 2D electron liquids
provides crucial answers to questions regarding oxide interface reconstruction
phenomena. We find total charge quantities consistent with the theory of
polarity-induced electronic reconstruction, elucidate the effects of oxygen
vacancies on this system, and further extract the confinement profile of the
reconstructed charge. Additionally, the RXR technique allows us to quantify
the orbital symmetry of electronic states at Angstrom length scales, showing
that significant symmetry breaking effects of the Ti $3d$ orbitals are
confined to within 2 unit cells of the interface. Crucially, the nature of RXR
allows us to probe the entire reconstructed charge density, not just mobile
carriers. Building upon the results presented here, future RXR experiments
will be helpful to closely examine the 4 u.c. threshold for 2DEL formation, as
well as O2 annealing of samples of various thicknesses. Additionally, with
such a precise quantification of the 2DEL characteristics, future efforts can
be directed toward engineering these properties for the development of
functional devices, either in the LAO/STO system, or other oxide
heterostructures. Finally, while these findings are important for the
understanding of the LAO/STO electronic system and related oxide
reconstruction phenomena, our results also show resonant x-ray reflectivity to
be an invaluable tool for the burgeoning field of oxide interfaces.

Methods

The set of samples investigated included 3, 5, and 10 u.c. LaAlO3 films grown
by pulsed laser deposition on TiO2-terminated SrTiO3 substrates. A KrF excimer
laser operating at 248 nm was used, with a fluence of 1 J/cm2 and repetition
rate of 1 Hz. The growth took place at 800 ∘C in an O2 pressure of $1\times
10^{-3}$ mbar. After deposition, all samples were cooled down in a higher O2
pressure of $5\times 10^{-3}$ mbar at a rate of 30 ∘C/min. Atomic layer growth
was controlled using reflection high energy electron diffraction (RHEED), as
reported in the supplemental material. Post annealed samples were annealed at
a temperature of 530 ∘C under an O2 pressure of 200 mbar for 1 hour.

The resonant x-ray reflectometry experiments were carried out at the REIXS
beamline of the Canadian Light Source,Hawthorn  _et al._ (2011) at 300K in an
ultra-high vacuum environment ($<10^{-9}$ mbar). Reflection intensity was
monitored using a filtered photodiode, whose response function was measured
using the direct synchrotron beam. Data was normalized by the incident beam
flux and the photodiode response to obtain the quantitative reflectivity
spectra. Two different measurement modes were used for data presented in this
work. For data in Fig. 2(a), the momentum transfer was held fixed by slightly
varying the reflection angle for each energy. For all other data the
reflection angle was held fixed while varying incident energy. Either mode can
be used for a successful RXR analysis, provided a sufficient sampling of the
energy versus angle/momentum space is made.

Simulations of the Ti $L_{2,3}$ resonant response for given orbital energies
and occupations were carried out with multiplet crystal field theory (MCFT).
This model Hamiltonian approach includes the full multiplet effects arising
from strong atomic Coulomb interactions, as well as crystal field
perturbations arising from the bonding with oxygen. Exact diagonalization and
simulations of the spectral functions were carried out with the software
_Quanty_.Haverkort  _et al._ ; Haverkort  _et al._ (2012); Lu  _et al._
(2014); Haverkort  _et al._ (2014) The simulated resonances were merged with
tabulated off-resonant atomic scattering factors $f_{2}$,Henke  _et al._
(1993) and the scattering factors $f_{1}$ were then calculated from the
Kramers-Kronig relations. For all other elements (La, Sr, O, Al), off-resonant
tabulated scattering factors were used. With all atomic scattering factors
determined, the reflectivity was simulated and fit to experiment using the
dynamical diffraction software QUAD.Macke  _et al._ (2016)

## References

  * Khomskii (2014) D. Khomskii,  _Transition metal compounds_ (Cambridge University Press, 2014).
  * Mannhart and Schlom (2010) J. Mannhart and D. G. Schlom, Science 327, 1607 (2010).
  * Hwang  _et al._ (2012) H. Y. Hwang, Y. Iwasa, M. Kawasaki, B. Keimer, N. Nagaosa, and Y. Tokura, Nature Mater. 11, 103 (2012).
  * Ohtomo and Hwang (2004) A. Ohtomo and H. Hwang, Nature 427, 423 (2004).
  * Thiel  _et al._ (2006) S. Thiel, G. Hammerl, A. Schmehl, C. W. Schneider, and J. Mannhart, Science 313, 1942 (2006).
  * Manca  _et al._ (2019) N. Manca, D. Bothner, A. M. R. V. L. Monteiro, D. Davidovikj, Y. G. Sağlam, M. Jenkins, M. Gabay, G. A. Steele, and A. D. Caviglia, Phys. Rev. Lett. 122, 036801 (2019).
  * Pai  _et al._ (2018) Y.-Y. Pai, H. Lee, J.-W. Lee, A. Annadi, G. Cheng, S. Lu, M. Tomczyk, M. Huang, C.-B. Eom, P. Irvin, and J. Levy, Phys. Rev. Lett. 120, 147001 (2018).
  * Trevisan  _et al._ (2018) T. V. Trevisan, M. Schütt, and R. M. Fernandes, Phys. Rev. Lett. 121, 127002 (2018).
  * Thierschmann  _et al._ (2018) H. Thierschmann, E. Mulazimoglu, N. Manca, S. Goswami, T. M. Klapwijk, and A. D. Caviglia, Nature Communications 9, 2276 (2018).
  * Reyren  _et al._ (2007) N. Reyren, S. Thiel, A. D. Caviglia, L. F. Kourkoutis, G. Hammerl, C. Richter, C. W. Schneider, T. Kopp, A.-S. Ruetschi, D. Jaccard, M. Gabay, D. A. Muller, J.-M. Triscone, and J. Mannhart, Science 317, 1196 (2007).
  * Li  _et al._ (2011) L. Li, C. Richter, J. Mannhart, and R. C. Ashoori, Nature Phys. 7, 762 (2011).
  * Zhang  _et al._ (2019) F. Zhang, P. Lv, Y. Zhang, S. Huang, C.-M. Wong, H.-M. Yau, X. Chen, Z. Wen, X. Jiang, C. Zeng, J. Hong, and J.-y. Dai, Phys. Rev. Lett. 122, 257601 (2019).
  * Park  _et al._ (2020) D.-S. Park, A. Rata, I. Maznichenko, S. Ostanin, Y. Gan, S. Agrestini, G. Rees, M. Walker, J. Li, J. Herrero-Martin, _et al._ , Nature Communications 11, 3650 (2020).
  * Brinkman  _et al._ (2007) A. Brinkman, M. Huijben, M. Van Zalk, J. Huijben, U. Zeitler, J. C. Maan, W. G. Van der Wiel, G. Rijnders, D. H. A. Blank, and H. Hilgenkamp, Nature Mater. 6, 493 (2007).
  * Bert  _et al._ (2011) J. A. Bert, B. Kalisky, C. Bell, M. Kim, Y. Hikita, H. Y. Hwang, and K. A. Moler, Nature Phys. 7, 767 (2011).
  * Breitschaft  _et al._ (2010) M. Breitschaft, V. Tinkl, N. Pavlenko, S. Paetel, C. Richter, J. R. Kirtley, Y. C. Liao, G. Hammerl, V. Eyert, T. Kopp, and J. Mannhart, Phys. Rev. B 81, 153414 (2010).
  * Khanna  _et al._ (2019) U. Khanna, P. K. Rout, M. Mograbi, G. Tuvia, I. Leermakers, U. Zeitler, Y. Dagan, and M. Goldstein, Phys. Rev. Lett. 123, 036805 (2019).
  * Briggeman  _et al._ (2020a) M. Briggeman, J. Li, M. Huang, H. Lee, J.-W. Lee, K. Eom, C.-B. Eom, P. Irvin, and J. Levy, Science Advances 6, eaba6337 (2020a).
  * Briggeman  _et al._ (2020b) M. Briggeman, M. Tomczyk, B. Tian, H. Lee, J.-W. Lee, Y. He, A. Tylan-Tyler, M. Huang, C.-B. Eom, D. Pekker, R. S. K. Mong, P. Irvin, and J. Levy, Science 367, 769 (2020b).
  * Geondzhian  _et al._ (2020) A. Geondzhian, A. Sambri, G. M. De Luca, R. Di Capua, E. Di Gennaro, D. Betto, M. Rossi, Y. Y. Peng, R. Fumagalli, N. B. Brookes, L. Braicovich, K. Gilmore, G. Ghiringhelli, and M. Salluzzo, Phys. Rev. Lett. 125, 126401 (2020).
  * Nakagawa  _et al._ (2006) N. Nakagawa, H. Y. Hwang, and D. A. Muller, Nature Materials 5, 204 (2006).
  * Hesper  _et al._ (2000) R. Hesper, L. H. Tjeng, A. Heeres, and G. A. Sawatzky, Phys. Rev. B 62, 16046 (2000).
  * Sing  _et al._ (2009) M. Sing, G. Berner, K. Goß, A. Müller, A. Ruff, A. Wetscherek, S. Thiel, J. Mannhart, S. A. Pauli, C. W. Schneider, P. R. Willmott, M. Gorgoi, F. Schäfers, and R. Claessen, Phys. Rev. Lett. 102, 176805 (2009).
  * Asmara  _et al._ (2014) T. C. Asmara, A. Annadi, I. Santoso, P. K. Gogoi, A. Kotlov, H. M. Omer, M. Motapothula, M. B. H. Breese, M. Ruebhausen, T. Venkatesan, Ariando, and A. Rusydi, Nat. Commun. 5, 3663 (2014).
  * Popović  _et al._ (2008) Z. S. Popović, S. Satpathy, and R. M. Martin, Phys. Rev. Lett. 101, 256801 (2008).
  * Son  _et al._ (2009) W.-j. Son, E. Cho, B. Lee, J. Lee, and S. Han, Phys. Rev. B 79, 245411 (2009).
  * Stemmer and James Allen (2014) S. Stemmer and S. James Allen, Annual Review of Materials Research 44, 151 (2014).
  * Salluzzo  _et al._ (2009) M. Salluzzo, J. C. Cezar, N. B. Brookes, V. Bisogni, G. M. De Luca, C. Richter, S. Thiel, J. Mannhart, M. Huijben, A. Brinkman, G. Rijnders, and G. Ghiringhelli, Phys. Rev. Lett. 102, 166804 (2009).
  * Macke  _et al._ (2014) S. Macke, A. Radi, J. E. Hamann-Borrero, A. Verna, M. Bluschke, S. Brück, E. Goering, R. Sutarto, F. He, G. Cristiani, M. Wu, E. Benckiser, H.-U. Habermeier, G. Logvenov, N. Gauquelin, G. A. Botton, A. P. Kajdos, S. Stemmer, G. A. Sawatzky, M. W. Haverkort, B. Keimer, and V. Hinkov, Advanced Materials 26, 6554 (2014).
  * Macke and Goering (2014) S. Macke and E. Goering, Journal of Physics: Condensed Matter 26, 363201 (2014).
  * Wadati  _et al._ (2009) H. Wadati, D. G. Hawthorn, J. Geck, T. Higuchi, Y. Hikita, H. Y. Hwang, L. F. Kourkoutis, D. A. Muller, S.-W. Huang, D. J. Huang, H.-J. Lin, C. Schüßler-Langeheine, H.-H. Wu, E. Schierle, E. Weschke, N. J. C. Ingle, and G. A. Sawatzky, Journal of Applied Physics 106, 083705 (2009).
  * Park  _et al._ (2013) J. Park, B.-G. Cho, K. D. Kim, J. Koo, H. Jang, K.-T. Ko, J.-H. Park, K.-B. Lee, J.-Y. Kim, D. R. Lee, C. A. Burns, S. S. A. Seo, and H. N. Lee, Phys. Rev. Lett. 110, 017401 (2013).
  * Note (1) Note we multiply the reflectivity spectra here by $q_{z}^{4}$ in order to show a wide $q_{z}$ range where the intensity spans several orders of magnitude.
  * Haverkort  _et al._ (2012) M. W. Haverkort, M. Zwierzycki, and O. K. Andersen, Phys. Rev. B 85, 165113 (2012).
  * (35) M. W. Haverkort _et al._ , http://www.quanty.org.
  * Henke  _et al._ (1993) B. Henke, E. Gullikson, and J. Davis, Atomic Data and Nuclear Data Tables 54, 181 (1993).
  * Willmott  _et al._ (2007) P. R. Willmott, S. A. Pauli, R. Herger, C. M. Schlepütz, D. Martoccia, B. D. Patterson, B. Delley, R. Clarke, D. Kumah, C. Cionca, and Y. Yacoby, Phys. Rev. Lett. 99, 155502 (2007).
  * Chambers  _et al._ (2010) S. A. Chambers, M. H. Engelhard, V. Shutthanandan, Z. Zhu, T. C. Droubay, L. Qiao, P. Sushko, T. Feng, H. D. Lee, T. Gustafsson, _et al._ , Surface Science Reports 65, 317 (2010).
  * Yin  _et al._ (2020) C. Yin, A. E. M. Smink, I. Leermakers, L. M. K. Tang, N. Lebedev, U. Zeitler, W. G. van der Wiel, H. Hilgenkamp, and J. Aarts, Phys. Rev. Lett. 124, 017702 (2020).
  * Pavlenko  _et al._ (2012) N. Pavlenko, T. Kopp, E. Y. Tsymbal, J. Mannhart, and G. A. Sawatzky, Phys. Rev. B 86, 064431 (2012).
  * Hawthorn  _et al._ (2011) D. G. Hawthorn, F. He, L. Venema, H. Davis, A. J. Achkar, J. Zhang, R. Sutarto, H. Wadati, A. Radi, T. Wilson, G. Wright, K. M. Shen, J. Geck, H. Zhang, V. Novák, and G. A. Sawatzky, Review of Scientific Instruments 82, 073104 (2011).
  * Lu  _et al._ (2014) Y. Lu, M. Höppner, O. Gunnarsson, and M. W. Haverkort, Phys. Rev. B 90, 085102 (2014).
  * Haverkort  _et al._ (2014) M. W. Haverkort, G. Sangiovanni, P. Hansmann, A. Toschi, Y. Lu, and S. Macke, Europhysics Letters 108, 57004 (2014).
  * Macke  _et al._ (2016) S. Macke, J. E. Hamann-Borrero, R. J. Green, B. Keimer, G. A. Sawatzky, and M. W. Haverkort, Phys. Rev. Lett. 117, 115501 (2016).

Acknowledgments

This work was supported by the Natural Sciences and Engineering Research
Council of Canada, the Canadian Institute for Advanced Research, the Max
Planck-UBC-UTokyo Centre for Quantum Materials, and the Canada First Research
Excellence Fund, Quantum Materials and Future Technologies Program. Team
members from the University of Würzburg were supported by the Deutsche
Forschungsgemeinschaft (DFG), Project-ID 258499086-SFB 1170. Team members from
the University of Twente were supported by the Netherlands Organization for
Scientific Research (NWO). Part of the research described in this paper was
performed at the Canadian Light Source, a national research facility of the
University of Saskatchewan, which is supported by the Canada Foundation for
Innovation (CFI), the Natural Sciences and Engineering Research Council of
Canada, the National Research Council (NRC), the Canadian Institutes of Health
Research (CIHR), the Government of Saskatchewan, and the University of
Saskatchewan. We thank Research Computing at the University of Saskatchewan
for computational resources.

Author contributions

Resonant reflectivity experiments were performed by R.J.G., S.M., R.S., and
F.H. Data analysis was performed by R.J.G. with contributions from V.Z., M.Z.,
S.M., J.G., V.H., and G.A.S. Sample synthesis and initial characterization was
managed and carried out by Z.L., M.H., G.R., and G.K. The manuscript was
written by R.J.G. with input from all authors.

_Supplementary Information for_ “Intrinsic versus extrinsic orbital and
electronic reconstructions at complex oxide interfaces”

##  I Sample Synthesis and Characterization

The LaAlO3 films studied in this work were grown on TiO2-terminated SrTiO3
substrates using pulsed laser deposition. A KrF excimer laser was used,
operating at 248 nm with a fluence of 1 J/cm2 and repetition rate of 1 Hz. The
substrate temperature during growth was 800∘C and an O2 atmosphere of $1\times
10^{-3}$ mbar was maintained. After growth, the films were cooled to room
temperature at a rate of 30 ∘C/min under a higher O2 pressure of $5\times
10^{-3}$ mbar. The layer-by-layer growth of the films was confirmed by RHEED
measurements, as shown in Fig. S1a for the 3, 5, and 10 u.c. films. The high
quality of the SrTiO3 substrates is demonstrated by the AFM image in Fig. S1b,
where atomic terraces are evident. This high quality is retained after the
layer by layer LAO growth, as indicated in Fig. S1c for a 10 u.c. sample.

Figure S1: Film growth and characterization. a, RHEED oscillations verifying a
layer by layer growth for the 3, 5, and 10 u.c. LAO films grown on STO. The
red arrows indicate when the growth stops. b-c, AFM image of (b) an STO
substrate and (c) a 10 u.c. LAO/STO heterostructure showing large atomic-layer
terraces.

After a resonant x-ray reflectometry (RXR) dataset was collected for the 10
u.c. sample, it was removed from the x-ray scattering chamber and immediately
annealed at 530∘C in 200 mbar of O2 for 1 hour. The sample was then cooled,
and reintroduced into the x-ray scattering chamber for collection of a second
RXR dataset. Transport characteristics for similarly prepared and annealed
samples were measured and are shown in Figure S2. The resistance, shown in
Fig. S2a, shows that the as-grown sample has higher conductivity at room
temperature, but not at low temperatures. This is understandable from the
carrier density shown in Fig. S2b, where the as-grown sample has considerably
more carriers at room temperature, but due to thermal activation loses a
considerable fraction of these carriers at low temperature. At 300 K (the
temperature of the RXR experiments), the as grown sample shows roughly 0.20
e-/u.c., where-as after annealing this number drops to roughly an order of
magnitude to 0.02 e-/u.c. These numbers can be compared to the RXR charge
concentrations of 0.97 and 0.50 e-/u.c. in the as-grown and annealed sample,
respectively. Evident is that RXR detects more charge in general,
understandable since RXR detects both localized and mobile carriers and the
total charge decreases by roughly half upon annealing whereas mobile charge
decreases by an order of magnitude. Put differently, for the as-grown sample
with oxygen vacancies, about 20% of the charge is mobile, whereas for the
annealed system presumably exhibiting pure electronic reconstruction, about 4%
of the charge is mobile.

Figure S2: Transport properties for 10 u.c. films. a, Temperature dependent
sheet resistance measured for 10 u.c. samples as-grown and after the O2 post-
annealing procedure. b, Sheet carrier density for as-grown and post-annealed
samples.

##  II Reflectivity Measurement Details

The RXR measurements were performed at the Resonant Elastic and Inelastic
X-ray Scattering (REIXS) beamline of the Canadian Light Source Hawthorn  _et
al._ (2011). An elliptically polarizing undulator (EPU) provided the vertical
and horizontal linearly polarized x-rays. An in-vacuum diffractometer
accommodated the reflection-geometry scans, and a photodiode was used to
detect the reflected beam intensity. The response function of the photodiode
was measured using the direct synchrotron beam. Data was normalized by the
incident beam flux and the photodiode response to obtain the quantitative
reflectivity spectra. During the experiments, the energy resolution
E/$\Delta$E was $\scriptstyle\mathtt{\sim}$$10^{4}$, the sample temperature
during measurement was 300 K, and the pressure in the measurement chamber was
$\lesssim 10^{-9}$ Torr.

Compared to other x-ray scattering measurements (fluorescence yield x-ray
absorption and resonant inelastic x-ray scattering, for example), the
reflectivity signal is very strong over the $q_{z}$ ranges studied here.
Accordingly, we were able to reduce the beam flux incident upon the sample
considerably compared to other measurement techniques, thus minimizing any
possibility of radiation damage to our samples. The estimated beam intensity
on our samples during measurements is $\scriptstyle\mathtt{\sim}$1010
photons/second over the area of the $\scriptstyle\mathtt{\sim}$(0.5 $\times$
0.2) mm2 beam cross section.

##  III Resonant Scattering Tensor Calculations

The atomic scattering factors which make up the dielectric response of our
films are tabulated for off-resonant energies Henke  _et al._ (1993). The
tabulated factors are not accurate at the resonances, however, and therefore
we explicitly calculate the Ti resonance response which enters into our
reflectivity model.

The $L_{2,3}$ absorption resonances of transition metal oxides have been
generally understood for more than 30 years de Groot  _et al._ (1990). In many
cases, the response is dominated by multiplet peaks and can be accurately
calculated with quantum many-body multiplet crystal field theory (MCFT). This
model Hamiltonian approach includes the full multiplet effects arising from
strong atomic Coulomb interactions, as well as crystal field perturbations
arising from the bonding with oxygen. Exact diagonalization and simulations of
the spectral functions were carried out with the software _Quanty_.Haverkort
_et al._ ; Haverkort  _et al._ (2012); Lu  _et al._ (2014); Haverkort  _et
al._ (2014) The simulated resonances were merged with tabulated off-resonant
atomic scattering factors $f_{2}$,Henke  _et al._ (1993) and the scattering
factors $f_{1}$ were then calculated from the Kramers-Kronig relations. For
La, the resonant scattering factor was determined from the x-ray absorption
spectrum and Kramers-Kronig methods. For all other elements (Sr, O, Al), off-
resonant tabulated scattering factors were used. With all atomic scattering
factors determined, the reflectivity was simulated and fit to experiment using
the dynamical diffraction software QUAD.Macke  _et al._ (2016a)

As stated, we model the Ti resonant response with MCFT, where the Ti valence
and $3d$ orbital energies are the main parameters of the model to be fit to
the data. Reliable values for the other parameters of such calculations are
well established, and the values used in this work are in Table S1. In
addition to these parameters, the resonant scattering tensors are determined
by the orbital energies given in Fig. 3 of the main text. Given the parameters
of Table S1, and the layer-dependent orbital energies given in the main text,
the imaginary components of the scattering tensors are as shown in Fig. S3
(the real parts are straightforwardly obtained via a Kramers-Kronig transform,
and are therefore not shown).

| $F^{2}_{dd}$ | $F^{4}_{dd}$ | $F^{2}_{pd}$ | $G^{1}_{pd}$ | $G^{3}_{pd}$ | $\zeta_{3d}$ | $\zeta_{2p}$ | $10Dq$ | $\delta t_{2g}(L_{2})$
---|---|---|---|---|---|---|---|---|---
Ti4+ | - | - | 3.628 | 3.153 | 1.792 | 0.019 | 3.820 | 2.12 | -0.20
Ti3+ | 5.907 | 3.711 | 3.187 | 3.393 | 1.258 | 0.019 | 3.710 | 1.80 | 0.00
Table S1:  Parameters for quantum many-body model calculations of scattering
tensors (all units are eV). $F$ and $G$ parameters are Coulomb and exchange
multipole radial integrals, $\zeta$ parameters are spin orbit interaction
energies for the indicated shells, and $10Dq$ is the $e_{g}-t_{2g}$ crystal
field splitting. $\delta t_{2g}(L_{2})$ is a constant shift added to the
$t_{2g}$ $L_{2}$ peak excitations to account for hybridization effects beyond
MCFT.  Figure S3:  Imaginary component of computed Ti $L_{2,3}$ resonant
scattering tensors. (Top row) Ti4+ scattering tensors for the first two
orbital-reconstructed layers and the bulk of the crystal for the samples
having the LaAlO3 thickness indicated. (Bottom row) similar to the above, but
showing the imaginary component of the Ti3 scattering tensor.

##  IV Simulations of Reflectivity

With the atomic scattering factors determined (either from tabulated values or
direct calculations), the reflectivity response of the heterostructure can be
simulated at various levels of sophistication. As we are sensitive to, and
interested in, the effects at the level of atomic layers, we use the recently
developed software QUAD Macke  _et al._ (2016b, ), which uses a layered
multiple scattering method perfectly suited to thin film heterostructures. A
(001) perovskite-specific model is employed, with AO and BO2 alternating
planes comprising the ABO3 structure of LaAlO3 and SrTiO3. Intermixing of
Al/Ti and Sr/La is included, and the degree of intermixing is fit using an
error-function profile. The exact (sub-unit-cell) thickness of the LaAlO3, the
surface roughness of the LaAlO3, and the thickness, density, and roughness of
an organic (carbon, for simplicity) surface contamination layer due to
atmosphere exposure is included. Additionally, while the films were nominally
TiO2-terminated, the model allowed for a fractional coverage of SrO or excess
TiO2 termination, presented in Table S2 as an excess or deficiency in the
nominal thickness of the SrO sublattice, $\delta$SrO.

The Levenburg-Marquardt non-linear least squares algorithm is used for fitting
the parameters using the experimental RXR data. Various randomized starts as
well as a genetic algorithm for initial rough fits are used to ensure that a
globally optimized set of parameters is attained. The error values shown are
determined from (the square root of) the diagonal of the fit covariance matrix
(off diagonal elements in all cases were significantly smaller than diagonal,
indicating little correlation of fit parameters).

Best fit parameters are shown in Table S2. The extended experimental and
simulated data sets are shown on the following pages for each sample, along
with detailed plots of the atomic-layer-resolved structures. Note that in the
main manuscript, the depth coordinates of the samples were shifted slightly
for presenting aligned interfaces, so the x-axes of Figs. S4a-S7a have
slightly shifted coordinates in comparison.

Parameter | 3 u.c. | 5 u.c. | 10 u.c. | 10 u.c. annealed
---|---|---|---|---
Thickness (Å) |  |  |  |
La | $10.58\pm 0.05$ | $20.08\pm 0.05$ | $40.77\pm 0.06$ | $40.62\pm 0.10$
Al | $11.38\pm 0.41$ | $19.08\pm 0.53$ | $40.58\pm 0.25$ | $39.52\pm 0.37$
C | $~{}2.10\pm 0.25$ | $~{}2.98\pm 0.40$ | $~{}5.02\pm 0.51$ | $~{}2.93\pm 0.18$
$\delta$SrO | $~{}0.07\pm 0.07$ | $-1.08\pm 0.10~{}$ | $-1.38\pm 0.07~{}$ | $-0.92\pm 0.08~{}$
Roughness (Å) |  |  |  |
Sr/La | $2.98\pm 0.01$ | $4.73\pm 0.06$ | $4.63\pm 0.10$ | $4.32\pm 0.07$
Ti/Al | $5.40\pm 0.02$ | $5.02\pm 0.05$ | $4.78\pm 0.04$ | $5.31\pm 0.03$
(La,Al)/C | $3.89\pm 0.02$ | $4.62\pm 0.06$ | $3.89\pm 0.11$ | $4.23\pm 0.13$
C/vacuum | $7.11\pm 0.15$ | $3.86\pm 0.30$ | $6.58\pm 0.36$ | $0.99\pm 0.90$
Concentration (uc-2) |  |  |  |
Ti3+ | $0.16\pm 0.03$ | $0.59\pm 0.05$ | $0.97\pm 0.08$ | $0.50\pm 0.03$
C | $5.49\pm 0.21$ | $5.26\pm 0.40$ | $6.80\pm 0.23$ | $2.68\pm 0.14$
Table S2:  Best-fit parameters for RXR models of LAO/STO samples.

Figure S4:  Resonant reflectometry results for 3 u.c. LAO/STO. a, Atomic layer
resolved element and valence concentration depth profiles for anions (top) and
cations (bottom). b, Reflectivity angle scans for energies at the Ti $L_{2,3}$
resonance. c, Reflectivity angle scans for non-resonant and La
$M_{5}$-resonant energies. d, Reflectivity energy scans for the indicated
momentum transfers using $\sigma$-polarized x-rays. e, Reflectivity energy
scans for the indicated momentum transfers using $\pi$-polarized x-rays.
Curves are offset vertically for clarity.  Figure S5:  Resonant reflectometry
results for 5 u.c. LAO/STO. a, Atomic layer resolved element and valence
concentration depth profiles for anions (top) and cations (bottom). b,
Reflectivity angle scans for energies at the Ti $L_{2,3}$ resonance. c,
Reflectivity angle scans for non-resonant and La $M_{5}$-resonant energies. d,
Reflectivity energy scans for the indicated reflection angles using
$\sigma$-polarized x-rays. e, Reflectivity energy scans for the indicated
reflection angles using $\pi$-polarized x-rays. Curves are offset vertically
for clarity.  Figure S6:  Resonant reflectometry results for 10 u.c. LAO/STO
before post-growth annealing. a, Atomic layer resolved element and valence
concentration depth profiles for anions (top) and cations (bottom). b,
Reflectivity angle scans for energies at the Ti $L_{2,3}$ resonance. c,
Reflectivity angle scans for non-resonant and La $M_{5}$-resonant energies. d,
Reflectivity energy scans for the indicated reflection angles using
$\sigma$-polarized x-rays. e, Reflectivity energy scans for the indicated
reflection angles using $\pi$-polarized x-rays. Curves are offset vertically
for clarity.  Figure S7:  Resonant reflectometry results for 10 u.c. LAO/STO
after post-growth annealing. a, Atomic layer resolved element and valence
concentration depth profiles for anions (top) and cations (bottom). b,
Reflectivity angle scans for energies at the Ti $L_{2,3}$ resonance. c,
Reflectivity angle scans for non-resonant and La $M_{5}$-resonant energies. d,
Reflectivity energy scans for the indicated reflection angles using
$\sigma$-polarized x-rays. e, Reflectivity energy scans for the indicated
reflection angles using $\pi$-polarized x-rays. Curves are offset vertically
for clarity.

## References

  * Hawthorn  _et al._ (2011) D. G. Hawthorn, F. He, L. Venema, H. Davis, A. J. Achkar, J. Zhang, R. Sutarto, H. Wadati, A. Radi, T. Wilson, G. Wright, K. M. Shen, J. Geck, H. Zhang, V. Novák, and G. A. Sawatzky, Review of Scientific Instruments 82, 073104 (2011).
  * Henke  _et al._ (1993) B. Henke, E. Gullikson, and J. Davis, Atomic Data and Nuclear Data Tables 54, 181 (1993).
  * de Groot  _et al._ (1990) F. M. F. de Groot, J. C. Fuggle, B. T. Thole, and G. A. Sawatzky, Phys. Rev. B 42, 5459 (1990).
  * (4) M. W. Haverkort _et al._ , http://www.quanty.org.
  * Haverkort  _et al._ (2012) M. W. Haverkort, M. Zwierzycki, and O. K. Andersen, Phys. Rev. B 85, 165113 (2012).
  * Lu  _et al._ (2014) Y. Lu, M. Höppner, O. Gunnarsson, and M. W. Haverkort, Phys. Rev. B 90, 085102 (2014).
  * Haverkort  _et al._ (2014) M. W. Haverkort, G. Sangiovanni, P. Hansmann, A. Toschi, Y. Lu, and S. Macke, Europhysics Letters 108, 57004 (2014).
  * Macke  _et al._ (2016a) S. Macke, J. E. Hamann-Borrero, R. J. Green, B. Keimer, G. A. Sawatzky, and M. W. Haverkort, Phys. Rev. Lett. 117, 115501 (2016a).
  * Macke  _et al._ (2016b) S. Macke, J. E. Hamann-Borrero, R. J. Green, B. Keimer, G. A. Sawatzky, and M. W. Haverkort, Phys. Rev. Lett. 117, 115501 (2016b).
  * (10) S. Macke _et al._ , http://quad.x-ray.center.