# Visual Boundary Knowledge Translation for Foreground Segmentation

Zunlei Feng1#, Lechao Cheng2#, Xinchao Wang3, Xiang Wang1, Yajie Liu2,
Xiangtong Du4,
Mingli Song1
Corresponding author. #Equal contribution to this work.

###### Abstract

When confronted with objects of unknown types in an image, humans can
effortlessly and precisely tell their visual boundaries. This recognition
mechanism and underlying generalization capability seem to contrast to state-
of-the-art image segmentation networks that rely on large-scale category-aware
annotated training samples. In this paper, we make an attempt towards building
models that explicitly account for visual boundary knowledge, in hope to
reduce the training effort on segmenting unseen categories. Specifically, we
investigate a new task termed as Boundary Knowledge Translation (BKT). Given a
set of fully labeled categories, BKT aims to _translate_ the visual boundary
knowledge learned from the labeled categories, to a set of novel categories,
each of which is provided only a few labeled samples. To this end, we propose
a Translation Segmentation Network (Trans-Net), which comprises a segmentation
network and two boundary discriminators. The segmentation network, combined
with a boundary-aware self-supervised mechanism, is devised to conduct
foreground segmentation, while the two discriminators work together in an
adversarial manner to ensure an accurate segmentation of the novel categories
under light supervision. Exhaustive experiments demonstrate that, with only
tens of labeled samples as guidance, Trans-Net achieves close results on par
with fully supervised methods.

## Introduction

Image segmentation has witnessed an unprecedented development in the past
decade thanks to the deep learning. The encouraging results, however, come at
the cost of the vast number of annotations and GPU training for days or even
weeks. To alleviate the training effort, a number of learning techniques, such
as few-shot learning and transfer learning, have proposed. The former aims to
train models using only a few annotated samples, while the later focuses on
transferring the models learned on one domain to another novel one. Despite
the recent progress in few-shot and transfer learning, existing approaches are
still prone to either inferior results (Shaban et al. 2017; Siam, Oreshkin,
and Jagersand 2019; Wang et al. 2019a), or the rigorous requirement that the
two tasks are strongly related (Dai et al. 2019; Sun et al. 2019) and a large
number of annotated samples (Hong et al. 2017; Li, Arnab, and Torr 2018;
Papandreou et al. 2015).

Figure 1:  Comparing the proposed BKT with transfer learning and few-shot
learning. Specifically, BKT _translates_ the knowledge learned from $p$
training categories into another $q$ ones, where the $p$ and $q$ categories
are disjoint. Unlike transfer learning and few-shot learning based tasks where
the trained models eventually tackle $p+q$ categories, BKT concentrates on the
$q$ testing categories only.

Nevertheless, our human eyes can effortlessly recognize the visual boundary of
a scene object, even if this object seems unfamiliar or belongs to an unknown
category. Inspired by this fact, we study in this paper a new Boundary
Knowledge Translation Task (BKT-Task), in aim to _translate_ the knowledge
learned from $p$ training categories where abundant annotations are available,
into another $q$ target categories where a very small number of annotations
are available for each class.

The differences between the proposed BKT-Task and the two related tasks,
transfer learning and few-shot learning, are summarized in Fig. 1. Unlike
transfer learning that requires the $p$ source categories and the $q$ target
ones to be highly related so as to achieve reasonable performances, in BKT
such requirement is largely relaxed. As demonstrated in our experiments, for
example, the visual boundary knowledge of _birds_ can be seamlessly translated
into the segmentation network for _flowers_. Also, in contrast to both
transfer and few-shot learning, whose outputs span $p+q$ categories, BKT
focuses on the $q$ categories only, which seems to be a flaw of BKT but in
reality not. In many cases, since abundant annotations are available for the
$p$ categories, indicating that state-of-the-art deep networks will highly
likely to deliver gratifying results, a model that focuses on the novel $q$
categories where annotations are insufficient is much desired. Furthermore, as
shown in prior works (Chen, Artieres, and Denoyer 2019; Minaee et al. 2020;
Vinyals et al. 2016) and also demonstrated in our experiments, handling many
categories simultaneously at once, especially under a compact network
architecture, will significantly downgrade the performances for all classes;
focusing on the $q$ critical categories, on the other hand, will alleviate
this dilemma to a large extend and ensure satisfactory performances.

To this end, we propose a Translation Segmentation Network (Trans-Net) for the
above BKT-Task. The Trans-Net is designed to contain a segmentation network
and two boundary discriminators. The segmentation network focuses on only
segmenting the target categories, while the two discriminators collaborate in
an adversarial fashion: one works on distinguishing whether the segmented
foreground contains the background features, and the other distinguishes
whether the segmented background contains the foreground features. Meanwhile,
we also introduce pseudo masks for enhancing and accelerating the translation
of the boundary knowledge.

For the segmentation network, we label tens of samples of the target
categories, as the guidance for segmenting the desired boundary of the
foreground. We also propose a self-supervised strategy to strengthen the
boundary consistency of unlabeled samples. Through the adversarial
optimization, the visual boundary knowledge of fully labeled $p$ categories
can be effectively translated into the segmentation network by two boundary
discriminators. Experiments demonstrate that, with only tens of labeled
samples as guidance, Trans-Net achieves truly encouraging results on par with
fully supervised methods.

Our contribution is therefore introducing the new BKT-Task, in aim to
translate the visual boundary knowledge learned from fully annotate source
categories into novel ones with few labels, and proposing a dedicated
solution, Trans-Net, towards solving BKT-Task. As the first attempt along its
line, Trans-Net, for the time being, focuses on foreground segmentation. We
devise a boundary-aware self-supervised mechanism and pseudo masks for the
segmentation network and the discriminators to enhance the translation of
visual boundary knowledge, respectively. We evaluate the proposed Trans-Net on
a broad domain of image datasets, in term of both qualitative visualization
and quantitative measures, and show that the proposed method achieves results
on par with fully supervised ones.

## Related work

We briefly review here two lines of work that are related to ours, GAN-based
segmentation methods and boundary-aware segmentation. More related works about
_few-shot learning_ and _transfer learning_ are given in the supplementary
materials.

GAN-based segmentation methods can be classified into two categories: mask
distribution-based methods and composition fidelity based methods. For the
mask distribution-based methods, Luc et al. (2016) proposed the first GAN
based semantic segmentation network, which adopts the adversarial optimization
between segmented results and GT mask to train the segmentation network.
Meanwhile, some researchers (Arbelle and Raviv 2018; Han and Yin 2017; Xue et
al. 2018) applied the same adversarial strategy into medical image
segmentation. With generated fake images and labeled images, Souly,
Spampinato, and Shah (2017) adopted the adversarial strategy to train the
discriminator output class confidence maps. Furthermore, Hung et al. (2018)
extended the discriminator to generate a confidence map, which can be used to
infer the regions sufficiently close to those from the ground truth
distribution. In the composition fidelity based methods (Chen, Artieres, and
Denoyer 2019; Ostyakov et al. 2018; Remez, Huang, and Brown 2018), the
segmented objects are firstly composited with some background images. Then,
the discriminator is adopted to discriminate fidelity of the composited images
and the GT nature images. Unlike the above existing GAN-based methods, we
adopt the adversarial strategy to translate the source categories’ visual
boundary knowledge into the segmentation network by two boundary
discriminators.

Boundary-aware segmentation. Bertasius, Shi, and Torresani (2016) proposed
two-tream framework where the predicted semantic boundaries is adopted to
improve the semantic segmentation maps. Similarly, two-tream framework is also
adopted in some recent works (Chen et al. 2016; Cheng et al. 2017; Yu et al.
2018; Takikawa et al. 2019a), where different constraints are devised for
strengthening the segmentation results with the predicted boundaries. Unlike
predicting the boundary directly, Hayder, He, and Salzmann (2017) proposed
predicting pixels’ distance to the object’s boundary and post-processed the
distance map into the final segmented results. Zhang et al. (2017) proposed a
local boundary refinement network to learn the position-adaptive propagation
coefficients so that local contextual information from neighbors can be
optimally captured for refining object boundaries. Peng et al. (2017) proposed
a boundary refinement block to improve the localization performance near the
object boundaries. Khoreva et al. (2017) proposed boundary-aware filtering to
improve object delineation. Qin et al. (2019) adopted the patch-level SSIM
loss (Wang, Simoncelli, and Bovik 2003) to assign higher weights to the
boundary. Unlike the above methods, we devise two boundary discriminators for
discriminating the foreground’s outer border and background’s inner boundary,
which can translate the visual boundary knowledge into a segmentation network
for any new category.

Figure 2: The framework of Trans-Net. The segmentation network is designed to
segment the sample’s foreground in the target dataset. The outer boundary
discriminator is devised for distinguishing whether the segmented foreground
contains the outer background’s features. The inner boundary discriminator is
devised for distinguishing whether the segmented background contains the inner
foreground’s features. Pseudo samples of source dataset are generated with the
eroded and dilated masks, which can reinforce the visual boundary knowledge
translation. Boundary-aware self-supervision is proposed to constrain the
boundary invariance on the target dataset.

## Boundary Knowledge Translation Task

The definition of the BKT-Task is given as follows. We assume that we are
given the labeled source dataset $\mathbf{S}^{p}$ that contains $p$ object
categories and target dataset $\mathbb{S}^{q}$ of $q$ object categories. The
$p$ categories and $q$ categories are disjoint. The visual boundary knowledge
is defined as the perfect segmentation that the object doesn’t contain outer
background’s features, and the background doesn’t contain inner objects’
features. The goal of BKT-Task is to translate the visual boundary knowledge
of $\mathbf{S}^{p}$ into the segmentation network $\mathcal{F}_{\theta}$,
which is devised for only concentrating on the segmentation of $q$ categories.
The difference between the BKT-Task and other related tasks are shown in Fig.
1.

## Method

There are vast public labeled datasets for the segmentation task. The BKT-Task
can effectively exploit those source datasets into the segmentation network
for a new category, which will dramatically reduce the requirement for labeled
samples in the new category. As the first attempt along BKT-Task, in this
paper, we focus on foreground segmentation and propose a Translation
Segmentation Network (Trans-Net), shown in Fig. 2. Trans-Net contains a
segmentation network and two boundary discriminators. The segmentation network
is designed for only segment samples in the target dataset $\mathbb{S}^{q}$.
The two boundary discriminators are devised for translating visual boundary
knowledge of the source dataset $\mathbf{S}^{p}$ into the segmentation network
without embezzling its capability for $q$ categories.

### Segmentation Network

In Trans-Net, the segmentation network $\mathcal{F}_{\theta}$ is designed to
be an encoder-decoder architecture. Given a target image
$\overline{\textbf{\em x}}\in\mathbb{S}^{q}$, the segmented result
$\tilde{\textbf{\em m}}=\mathcal{F}_{\theta}(\overline{\textbf{\em x}})$ is
expected to approximate the GT mask m, which can be achieved by minimizing the
following basic reconstruction loss $\mathbf{\mathcal{L}_{rec}}$:

$\mathbf{\mathcal{L}_{rec}}=||\tilde{\textbf{\em m}}-\textbf{\em
m}||^{2}_{2},~{}\tilde{\textbf{\em
m}}=\mathcal{F}_{\theta}(\overline{\textbf{\em x}}),~{}\overline{\textbf{\em
x}}\in\mathbb{S}^{q}.$ (1)

The $\mathbf{\mathcal{L}_{rec}}$ is used to reconstruct a few labeled samples
of the target datasets, which will guide the segmentation of the desired
boundary.

_Boundary-aware Self-supervision._ To reduce the amount of labelled samples,
inspired by Wang et al. (2019b), we propose a boundary-aware self-supervised
strategy, which can strengthen the boundary consistency on target categories.
The core idea is that the segmented result of the warped input image should be
equal to the warped result of the input image. The schematic diagram of
boundary-aware self-supervision is given in Fig. 2. Formally, for the robust
segmentation network, given an affine transformation matrix A, segmented
result $\mathcal{F}_{\theta}(\textbf{\em A}\overline{\textbf{\em x}})$ of the
warped image $\textbf{\em A}\overline{\textbf{\em x}}$ and the warped result
$\textbf{\em A}\mathcal{F}_{\theta}(\overline{\textbf{\em x}})$ should be
consistent in the following way: $\mathcal{F}_{\theta}(\textbf{\em
A}\overline{\textbf{\em x}})=\textbf{\em
A}\mathcal{F}_{\theta}(\overline{\textbf{\em x}})$. Furthermore, we obtain the
boundary neighborhood weight map w as follows:

$\textbf{\em w}=\mathfrak{D}_{r}(\tilde{\textbf{\em
m}})-\mathfrak{E}_{r}(\tilde{\textbf{\em m}}),$ (2)

where, $\mathfrak{D}_{r}$ and $\mathfrak{E}_{r}$ denote the dilation and
erosion operation with disk strel of radius $r$. The weight map w can further
strengthen the boundary consistency. The boundary-aware self-supervised loss
$\mathbf{\mathcal{L}_{sel}}$ is defined as follows:

$\mathbf{\mathcal{L}_{sel}}=||\textbf{\em
w}^{\prime}\mathcal{F}_{\theta}(\textbf{\em A}\overline{\textbf{\em
x}})-\textbf{\em A}\\{\textbf{\em w}\mathcal{F}_{\theta}(\overline{\textbf{\em
x}})\\}||^{2}_{2},~{}\overline{\textbf{\em x}}\in\mathbb{S}^{q},$ (3)

where, $\textbf{\em w}^{\prime}$ and w are the weight maps of the predict
masks $\mathcal{F}_{\theta}(\textbf{\em A}\overline{\textbf{\em x}})$ and
$\mathcal{F}_{\theta}(\overline{\textbf{\em x}})$, respectively. The boundary-
aware self-supervised mechanism not only strengthens the boundary consistency
but also can eliminate the unreasonable holes in the predict masks.

### Boundary Discriminator

Inspired by the fact that humans can segment an object’s boundary through
distinguishing whether the inner and outer of boundary contain redundant
features, we devise two boundary discriminators, which can translate the
boundary knowledge of the source dataset $\mathbf{S}^{p}$ into the
segmentation network.

_Outer Boundary Discriminator._ Given the input target image
$\overline{\textbf{\em x}}\in\mathbb{S}^{q}$, the segmentation network
predicts the mask $\tilde{\textbf{\em
m}}=\mathcal{F}_{\theta}(\overline{\textbf{\em x}})$. Next, the foreground
$\overline{\textbf{\em x}}^{o}$ is computed using the following equation:
$\overline{\textbf{\em x}}^{o}=\tilde{\textbf{\em m}}*\overline{\textbf{\em
x}}$, where ‘$*$’ denotes pixel-wise multiplication. Then the concatenated
triplet $\textbf{\em I}^{o}_{a}=[\overline{\textbf{\em x}},\tilde{\textbf{\em
m}},\overline{\textbf{\em x}}^{o}]$ is input into the outer boundary
discriminator $\mathcal{D}^{o}_{\phi}$, which discriminates whether the
segmented foreground $\overline{\textbf{\em x}}^{o}$ contains the background’s
outer features. In the paper, $\textbf{\em I}^{o}_{a}$ is regarded as a fake
triplet. Meanwhile, choosing a labeled sample $\underline{\textbf{\em x}}$
from the source dataset $\mathbf{S}^{p}$, the corresponding $\textbf{\em
I}^{o}_{e}=[\underline{\textbf{\em x}},\textbf{\em m},\underline{\textbf{\em
x}}^{o}]$ is labeled as a real triplet. Furthermore, we reprocess the GT mask
m of samples $\underline{\textbf{\em x}}\in\mathbf{S}^{p}$ by dilation
operation and get the pseudo triplet $\textbf{\em
I}^{o}_{s}=[\underline{\textbf{\em x}},\mathfrak{D}_{r}(\textbf{\em
m}),\underline{\textbf{\em x}}^{o}_{\mathfrak{D}}]$, where
$\underline{\textbf{\em x}}\in\mathbf{S}^{p}$ and $\underline{\textbf{\em
x}}^{o}_{\mathfrak{D}}=\mathfrak{D}_{r}(\textbf{\em m})*\underline{\textbf{\em
x}}$. The generated pseudo triplet $\textbf{\em I}^{o}_{s}$ will assist the
outer boundary discriminator in distinguishing the background’s outer
features. The adversarial optimization between the segmentation network and
outer boundary discriminator will translate the outer boundary knowledge of
the source dataset $\mathbf{S}^{p}$ into the segmentation network with the
following outer boundary adversarial loss $\mathbf{\mathcal{L}^{out}_{adv}}$:

$\begin{split}\mathbf{\mathcal{L}^{out}_{adv}}\\!\\!=&\frac{1}{2}\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{o}_{a}\sim\mathbb{P}^{o}_{a}}\\![\mathcal{D}^{o}_{\phi}\\!(\textbf{\em
I}^{o}_{a})]\\!+\\!\frac{1}{2}\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{o}_{s}\sim\mathbb{P}^{o}_{s}}[\mathcal{D}^{o}_{\phi}\\!(\textbf{\em
I}^{o}_{s})]\\!-\\!\\!\\!\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{o}_{e}\sim\mathbb{P}^{o}_{e}}\\![\mathcal{D}^{o}_{\phi}\\!(\textbf{\em
I}^{o}_{e})]\\\ &+\lambda\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{o}\sim\mathbb{P}_{\textbf{\em I}^{o}}}\\!\\![(\|\nabla_{\textbf{\em
I}^{o}}\mathcal{D}^{o}_{\phi}(\textbf{\em
I}^{o})\|_{2}\\!-\\!1)^{2}],\end{split}$ (4)

where, the $\mathbb{P}^{o}_{a}$, $\mathbb{P}^{o}_{s}$, $\mathbb{P}^{o}_{e}$
are the segmented outer boundary distribution, pseudo outer boundary
distribution, and real outer boundary distribution, respectively. The
$\mathbb{P}_{\textbf{\em I}^{o}}$ is sampled uniformly along straight lines
between pairs of points sampled from the distribution $\mathbb{P}^{o}_{e}$ and
the segmentation network distribution $\mathbb{P}^{o}_{a}$. The $\textbf{\em
I}^{o}=\varepsilon\textbf{\em I}^{o}_{e}+(1-\varepsilon)\textbf{\em
I}^{o}_{a}$, where the $\varepsilon$ is a random number between $0$ and $1$.
The gradient penalty term is firstly proposed in WGAN-GP (Gulrajani et al.
2017). The $\lambda$ is the gradient penalty coefficient.

Algorithm 1 The Training Algorithm for Trans-Net

1: The gradient penalty coefficient $\lambda$, interval iteration number
$n_{critic}$, the batch size $K$, Adam hyperparameters
$\alpha,\beta_{1},\beta_{2}$, the balance parameters $\tau,\eta$ for
$\mathcal{L}_{rec}$ and $\mathbf{\mathcal{L}_{sel}}$, the Laplace smoothing
parameter $\xi$.

2: Initial critic parameters $\varphi,\phi$, initial segmentation network
parameters $\theta$.

3:while $\theta$ has not converged do

4: for $t=1,...,n_{critic}$ do

5: for $k=1,...,K$ do

6: Sample $\overline{\textbf{\em x}}$ from target dataset $\mathbb{S}^{q}$,
$(\underline{\textbf{\em x}},\textbf{\em m})$ from source dataset
$\mathbf{S}^{p}$, a random number $\varepsilon\sim U[0,1]$.

7: Obtain real triplet $\textbf{\em I}^{o}_{e}=[\underline{\textbf{\em
x}},\textbf{\em m},\underline{\textbf{\em x}}^{o}]$ and $\textbf{\em
I}^{i}_{e}=[\underline{\textbf{\em x}},\textbf{\em
m}^{\prime},\underline{\textbf{\em x}}^{i}]$, $\underline{\textbf{\em
x}}^{o}=\textbf{\em m}*\underline{\textbf{\em x}}$, $\underline{\textbf{\em
x}}^{i}=\textbf{\em m}^{\prime}*\underline{\textbf{\em x}}$.

8: Obtain fake triplet $\textbf{\em I}^{o}_{a}=[\overline{\textbf{\em
x}},\tilde{\textbf{\em m}},\overline{\textbf{\em x}}^{o}]$ and $\textbf{\em
I}^{i}_{a}=[\overline{\textbf{\em x}},\tilde{\textbf{\em
m}}^{\prime},\overline{\textbf{\em x}}^{i}]$, $\tilde{\textbf{\em
m}}=\mathcal{F}_{\theta}(\overline{\textbf{\em x}})$, $\tilde{\textbf{\em
m}}^{\prime}=[\mathbf{1}]-\tilde{\textbf{\em m}}$.

9: Obtain pseudo triplet $\textbf{\em I}^{o}_{s}=[\underline{\textbf{\em
x}},\mathfrak{D}_{r}(\textbf{\em m}),\underline{\textbf{\em
x}}^{o}_{\mathfrak{D}}]$ and $\textbf{\em I}^{i}_{s}=[\underline{\textbf{\em
x}},\mathfrak{E}_{r}(\textbf{\em m}^{\prime}),\underline{\textbf{\em
x}}^{i}_{\mathfrak{E}}]$.

10: $\textbf{\em I}^{o}\leftarrow\varepsilon\textbf{\em
I}^{o}_{e}+(1-\varepsilon)\textbf{\em I}^{o}_{a}$, $\textbf{\em
I}^{i}\leftarrow\varepsilon\textbf{\em I}^{i}_{e}+(1-\varepsilon)\textbf{\em
I}^{i}_{a}$.

11:
$\mathbf{\mathcal{L}^{out}_{adv}}^{(k)}\leftarrow\frac{1}{2}\mathcal{D}^{o}_{\phi}(\textbf{\em
I}^{o}_{a})+\frac{1}{2}\mathcal{D}^{o}_{\phi}(\textbf{\em
I}^{o}_{s})-\mathcal{D}^{o}_{\phi}(\textbf{\em I}^{o}_{e})+$

12: $\lambda(\|\nabla_{\textbf{\em I}^{o}}\mathcal{D}^{o}_{\phi}(\textbf{\em
I}^{o})\|_{2}-1)^{2}$.

13:
$\mathbf{\mathcal{L}^{in}_{adv}}^{(k)}\leftarrow\frac{1}{2}\mathcal{D}^{i}_{\varphi}(\textbf{\em
I}^{i}_{a})+\frac{1}{2}\mathcal{D}^{i}_{\varphi}(\textbf{\em
I}^{i}_{s})-\mathcal{D}^{i}_{\varphi}(\textbf{\em I}^{i}_{e})+$

14: $\lambda(\|\nabla_{\textbf{\em
I}^{i}}\mathcal{D}^{i}_{\varphi}(\textbf{\em I}^{i})\|_{2}-1)^{2}$.

15: $\phi\leftarrow
Adam(\nabla_{\phi}\frac{1}{K}\sum^{K}_{k=1}\mathbf{\mathcal{L}^{out}_{adv}}^{(k)},\phi,\alpha,\beta_{1},\beta_{2})$.

16: $\varphi\leftarrow
Adam(\nabla_{\varphi}\frac{1}{K}\sum^{K}_{k=1}\mathbf{\mathcal{L}^{in}_{adv}}^{(k)},\varphi,\alpha,\beta_{1},\beta_{2})$.

17: Sample unlabeled batch $\\{\overline{\textbf{\em x}}^{(k)}\\}^{K}_{k=1}$
and labeled batch $\\{\ddot{\overline{\textbf{\em x}}}^{(k)},\textbf{\em
m}^{(k)}\\}^{K}_{k=1}$ from target dataset $\mathbb{S}^{q}$.

18:
$\mathbf{\mathcal{L}_{rec}}\leftarrow||\mathcal{F}_{\theta}(\ddot{\overline{\textbf{\em
x}}})-\textbf{\em m}||^{2}_{2}$.

19: $\mathbf{\mathcal{L}_{sel}}\leftarrow||\textbf{\em
w}^{\prime}\mathcal{F}_{\theta}(\textbf{\em A}\overline{\textbf{\em
x}})-\textbf{\em A}\\{\textbf{\em w}\mathcal{F}_{\theta}(\overline{\textbf{\em
x}})\\}||^{2}_{2}$.

20: $\theta\leftarrow
Adam(\nabla_{\theta}\frac{1}{K}\sum^{K}_{k=1}\\{\mathbf{\tau\mathcal{L}_{rec}}+\eta\mathbf{\mathcal{L}_{sel}}-\mathcal{D}^{o}_{\phi}(\mathcal{F}_{\theta}(\overline{\textbf{\em
x}}))-\mathcal{D}^{i}_{\varphi}(\mathcal{F}_{\theta}(\overline{\textbf{\em
x}}))\\}),\theta,\alpha,\beta_{1},\beta_{2})$.

21:return Segmentation network parameters $\theta$, critic parameters
$\varphi,~{}\phi$.

_Inner Boundary Discriminator._ The inner boundary discriminator
$\mathcal{D}^{i}_{\varphi}$ is devised for discriminating whether the
segmented background contains the object’s inner features. To obtain the
segmented background, the predict background mask $\tilde{\textbf{\em
m}}^{\prime}$ and GT mask $\textbf{\em m}^{\prime}$ are reprocessed with the
Not-operation as follows: $\tilde{\textbf{\em
m}}^{\prime}=[\mathbf{1}]-\tilde{\textbf{\em m}}$, $\textbf{\em
m}^{\prime}=[\mathbf{1}]-\textbf{\em m}$, where the $[\mathbf{1}]$ denotes the
unit matrix of m’s size. Then, the corresponding fake triplet $\textbf{\em
I}^{i}_{a}=[\overline{\textbf{\em x}},\tilde{\textbf{\em
m}}^{\prime},\overline{\textbf{\em x}}^{i}]$, real triplet $\textbf{\em
I}^{i}_{e}=[\underline{\textbf{\em x}},\textbf{\em
m}^{\prime},\underline{\textbf{\em x}}^{i}]$ and pseudo triplet $\textbf{\em
I}^{i}_{s}=[\underline{\textbf{\em x}},\mathfrak{D}_{r}(\textbf{\em
m}^{\prime}),\underline{\textbf{\em x}}^{i}_{\mathfrak{D}}]$ are computed in
the same manner as done in the outer boundary discriminator. The generated
pseudo triplet $\textbf{\em I}^{i}_{s}$ will also assist the inner boundary
discriminator in distinguishing the inner features of the foreground.
Similarly, the inner boundary adversarial loss
$\mathbf{\mathcal{L}^{in}_{adv}}$ is defined as follows:

$\begin{split}\mathbf{\mathcal{L}^{in}_{adv}}\\!\\!=&\frac{1}{2}\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{i}_{a}\sim\mathbb{P}^{i}_{a}}\\![\mathcal{D}^{i}_{\varphi}\\!(\textbf{\em
I}^{i}_{a})]\\!+\\!\frac{1}{2}\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{i}_{s}\sim\mathbb{P}^{i}_{s}}[\mathcal{D}^{i}_{\varphi}\\!(\textbf{\em
I}^{i}_{s})]\\!-\\!\\!\\!\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{i}_{e}\sim\mathbb{P}^{i}_{e}}\\![\mathcal{D}^{i}_{\varphi}\\!(\textbf{\em
I}^{i}_{e})]\\\ &+\lambda\\!\mathop{\mathbb{E}}\limits_{\textbf{\em
I}^{i}\sim\mathbb{P}_{\textbf{\em I}^{i}}}\\!\\![(\|\nabla_{\textbf{\em
I}^{i}}\mathcal{D}^{i}_{\varphi}(\textbf{\em
I}^{i})\|_{2}\\!-\\!1)^{2}],\end{split}$ (5)

where, the $\mathbb{P}^{i}_{a}$, $\mathbb{P}^{i}_{s}$, $\mathbb{P}^{i}_{e}$
are the segmented inner boundary distribution, pseudo inner boundary
distribution, and real inner boundary distribution, respectively. $\textbf{\em
I}^{i}=\varepsilon\textbf{\em I}^{i}_{e}+(1-\varepsilon)\textbf{\em
I}^{i}_{a}$. The optimization on $\mathbf{\mathcal{L}^{in}_{adv}}$ will
translate the inner boundary knowledge of the source dataset $\mathbf{S}^{p}$
into the segmentation network.

\begin{overpic}[scale={0.60}]{Figures/visualResults.pdf} {
\put(2.0,46.8){Input} \put(9.1,46.8){GT} \put(16.0,46.8){CAC}
\put(23.1,46.8){ReDO} \put(29.0,46.8){SG-One} \put(36.8,46.8){PANet}
\put(44.0,46.8){SPNet} \put(51.0,46.8){CANet} \put(58.0,46.8){ALSSS}
\put(65.5,46.8){USSS} \put(70.5,46.8){Trans.(10)} \put(78.2,46.8){Trans.(100)}
\put(87.0,48.5){Gated-} \put(87.0,46.8){SCNN} \put(95.0,46.8){BFP}
\par\put(2.1,21.8){Unet} \put(9.1,21.8){FPN} \put(15.0,21.8){LinkNet}
\put(22.5,21.8){PSPNet} \put(30.0,21.8){PAN} \put(34.5,21.8){DeepLabV3+}
\put(44.9,21.8){$T_{self}^{-}$} \put(51.5,21.8){$T_{pseu}^{-}$}
\put(57.8,21.8){$T_{innner}^{-}$} \put(65.5,21.8){$T_{outer}^{-}$}
\put(72.3,21.8){$T_{oneD}$} \put(79.9,21.8){{T}(0)} \put(87.0,21.8){{T}(10)}
\put(94.5,21.8){{T}(all)} \par} \end{overpic} Figure 3: The visual results of
different methods on three datasets. Trans.($x$) denotes the DeepLabV3+
trained on $MixAll^{-}$ and finetuned with $x$ labeled target samples.
$\textbf{\em T}_{self}^{-}$ and $\textbf{\em T}_{pseu}^{-}$ denote the Trans-
Net without boundary-aware self-supervision and pseudo triplet. $\textbf{\em
T}_{inner}^{-}$ and $\textbf{\em T}_{outer}^{-}$ denote the Trans-Net without
inner discriminator and outer discriminator. $\textbf{\em T}_{oneD}$ denotes
the Trans-Net with only one discriminator. $\textbf{\em T}(x)$ denotes the
Trans-Net with $x$ labeled samples of the target dataset.

### Complete Algorithm

To sum up, two boundary adversarial losses $\mathbf{\mathcal{L}^{out}_{adv}}$
and $\mathbf{\mathcal{L}^{in}_{adv}}$ are used for translating the visual
boundary knowledge of the source dataset $\mathbf{S}^{p}$ into the
segmentation network, which is designed for only segment samples from the
target dataset $\mathbb{S}^{q}$. The basic reconstruction loss
$\mathbf{\mathcal{L}_{rec}}$ is adopted to supervise the segmentation on tens
of labeled samples in target dataset $\mathbb{S}^{q}$. The self-supervised
loss $\mathbf{\mathcal{L}_{sel}}$ is devised to strengthen the boundary
consistency on target datasets $\mathbb{S}^{q}$.

During training, we alternatively optimize the segmentation network
$\mathcal{F}_{\theta}$ and two boundary discriminators
$\mathcal{D}^{o}_{\phi},\mathcal{D}^{i}_{\varphi}$ using the randomly sampled
samples from the target dataset $\mathbb{S}^{q}$ and the source dataset
$\mathbf{S}^{p}$, respectively. The complete algorithm is summarized in
Algorithm 1. Once trained, the segmentation network concentrates on segmenting
the foreground of the target category only.

Table 1: The performance comparison between SOTA methods and Trans-Net. The
_T(x)_ denotes the Trans-Net with $x$ labeled samples of the target dataset.
(All scores in $\%$). Trans.(x) denotes the DeepLabV3+ trained on $MixAll^{-}$
and finetuned with $x$ labeled target samples. Red and Green indicate the best
and second-best performance. Blue indicates the best performance among all
non-fully supervised methods. Note that Flowers dataset contains only $753$
annotated samples.

|  Birds Dataset |  HumanMatting Dataset |  Flowers Dataset
---|---|---|---
Method$\backslash$Index | PA | MPA | MIoU | FWIoU | PA | MPA | MIoU | FWIoU | PA | MPA | MIoU | FWIoU
CAC | 41.90 | 48.28 | 26.42 | 27.06 | 45.92 | 47.14 | 24.28 | 23.72 | 43.70 | 36.96 | 24.24 | 35.34
ReDO | 67.06 | 50.00 | 38.53 | 33.01 | 51.44 | 50.00 | 35.72 | 36.49 | 77.16 | 70.00 | 58.58 | 43.82
SG-One | 83.45 | 78.66 | 61.43 | 74.68 | 72.68 | 72.46 | 56.72 | 56.93 | 86.47 | 87.05 | 74.73 | 76.77
PANet | 81.78 | 66.94 | 57.83 | 69.42 | 73.89 | 76.49 | 60.54 | 63.29 | 87.42 | 68.43 | 69.25 | 69.94
SPNet | 85.21 | 79.65 | 76.92 | 78.01 | 77.82 | 75.90 | 60.42 | 62.90 | 87.90 | 88.43 | 79.21 | 80.43
CANet | 78.29 | 73.49 | 71.06 | 69.84 | 76.04 | 73.80 | 64.21 | 63.90 | 86.43 | 84.96 | 80.32 | 79.16
ALSSS | 76.95 | 51.54 | 39.48 | 66.09 | 70.54 | 76.26 | 60.42 | 63.90 | 87.43 | 79.32 | 85.21 | 87.32
USSS | 81.66 | 49.64 | 41.27 | 67.95 | 77.25 | 77.75 | 62.40 | 62.28 | 95.47 | 95.15 | 90.81 | 91.36
Trans.(10) | 82.84 | 79.83 | 65.07 | 74.71 | 85.32 | 85.44 | 75.71 | 75.72 | 83.39 | 82.69 | 78.66 | 76.39
Trans.(100) | 91.26 | 91.56 | 83.23 | 84.06 | 94.99 | 95.00 | 90.46 | 90.48 | 93.90 | 89.29 | 81.33 | 89.05
Gated-SCNN | 83.23 | 83.65 | 57.93 | 76.68 | 81.18 | 80.38 | 67.66 | 67.96 | 86.81 | 79.56 | 71.18 | 76.19
BFP | 80.47 | 80.54 | 58.99 | 74.24 | 79.32 | 78.46 | 65.34 | 64.31 | 84.28 | 77.43 | 68.29 | 74.30
1-13[0.8pt/2pt] Unet | 95.74 | 91.88 | 86.41 | 92.06 | 97.89 | 97.88 | 95.86 | 95.87 | 96.84 | 96.39 | 93.44 | 93.89
FPN | 95.70 | 92.86 | 86.53 | 92.06 | 98.20 | 98.19 | 96.45 | 96.46 | 97.21 | 97.16 | 94.22 | 94.59
LinkNet | 95.50 | 93.04 | 86.03 | 91.77 | 97.42 | 97.41 | 94.97 | 94.98 | 97.25 | 96.82 | 94.26 | 94.65
PSPNet | 93.37 | 87.01 | 79.47 | 87.97 | 97.03 | 97.02 | 94.22 | 94.23 | 95.80 | 95.77 | 91.46 | 91.99
PAN | 95.86 | 93.86 | 87.07 | 92.38 | 98.16 | 98.15 | 96.37 | 96.38 | 96.92 | 96.71 | 93.64 | 94.06
DeepLabV3+ | 96.78 | 94.88 | 89.62 | 93.95 | 98.28 | 98.28 | 96.62 | 96.63 | 97.01 | 96.65 | 93.80 | 94.23
_T(0)_ | 70.87 | 56.19 | 42.67 | 60.15 | 66.41 | 66.97 | 48.90 | 48.73 | 83.04 | 78.47 | 67.78 | 70.56
_T(10)_ | 92.95 | 90.35 | 79.58 | 87.66 | 96.24 | 96.29 | 92.74 | 92.75 | 93.20 | 94.20 | 88.70 | 89.30
_T(all)_ | 96.88 | 94.73 | 89.76 | 94.09 | 98.32 | 98.32 | 96.68 | 96.68 | 95.92 | 96.44 | 91.77 | 92.23

## Experiments

Dataset. The datasets we adopted contain single category datasets: Birds
(Catherine et al. 2011), Flowers (Nilsback and Zisserman 2007), and
HumanMatting (Company 2019) and mixed category datasets: THUR15K (Cheng et al.
2014), MSRA10K and MSRA-B (Cheng et al. 2011; Hou et al. 2017), CSSD (Yan et
al. 2013), ECSSD (Shi et al. 2016), DUT-OMRON (Ruan, Tong, and Lu 2011),
PASCAL-Context (Mottaghi et al. 2014), HKU-IS (Li and Yu 2016), SOD (Movahedi
and Elder 2010), SIP1K (Fan et al. 2019). The Birds, Flowers, HumanMatting and
THUR15K are set as the target datasets in the experiments. The Birds, Flowers,
and HumanMatting contain $(11,788)$, $(8,189)$, $(34,427)$ samples,
respectively. The THUR15K contains $5$ categories and $15000$ samples. The
Flowers contains manually annotated parts ($753$ accurate masks) and algorithm
(Nilsback and Zisserman 2007) pre-segmented parts ($8189$ rough masks). In
this paper, we adopt the manually labeled part. The experiments on the
algorithm pre-segmented parts are given in the supplementary material. Except
THUR15K, all the above mixed datasets are merged into the $MixAll$ dataset,
where some of the mislabeled samples are deleted. The final $MixAll$ dataset
contains $23,500$ samples. When translating the visual boundary knowledge into
the target category, the corresponding samples of the target category will be
removed from $MixAll$, which generates the $MixAll^{-}$ dataset. The data
enhancement strategies for a few labeled dataset and affine transformation
strategies are given in the supplementary materials.

Network architecture. In the paper, the segmentation network we adopted is the
DeeplabV3+ (backbone: resnet50 ) (Chen et al. 2017). Some popular network
architectures including Unet (Ronneberger, Fischer, and Brox 2015), FPN (Lin
et al. 2017), Linknet (Chaurasia and Culurciello 2017), PSPNet (Zhao et al.
2017), PAN (Li et al. 2018) are also tested in our framework. Two boundary
discriminators have the same encoder architecture, which is given in the
supplementary materials.

Parameter setting. The parameters are set as follows:
$\xi=1,\tau=1,\eta=1,\lambda=10$, $n_{critic}=5$, the batch size $K=64$, Adam
hyperparameters for two discriminators
$\alpha=0.0001,\beta_{1}=0,\beta_{2}=0.9$. The learning rate for the
segmentation network and two discriminators are all taken to be $1e^{-4}$. The
disk strel of radius $r$ is randomly sampled integer between $11$ and $55$.

Metric. The metrics we adopted include Pixel Accuracy (PA), Mean Pixel
Accuracy (MPA), Mean Intersection over Union (MIoU), and Frequency Weighted
Intersection over Union (FWIoU).

More details of datasets, network, parameters are given in the _supplementary
materials_.

### Comparing with SOTA Methods

In this section, the proposed method is compared with the SOTA methods,
including _unsupervised methods_ :ReDO (Chen, Artieres, and Denoyer 2019) and
CAC (Hsu, Lin, and Chuang 2018), _few-shot methods_ : SG-One (Zhang et al.
2018), PANet (Wang et al. 2019a), SPNet (Xian et al. 2019) and CANet (Zhang et
al. 2019), _weakly-/semi-supervised methods_ : USSS (Kalluri et al. 2019) and
ALSSS (Hung et al. 2018), and _fully supervised methods_ : Unet (Ronneberger,
Fischer, and Brox 2015), FPN (Lin et al. 2017), LinkNet (Chaurasia and
Culurciello 2017), PSPNet (Zhao et al. 2017), PAN (Li et al. 2018) and
DeeplabV3+ (Chen et al. 2017) on four datasets. Meanwhile, The Trans-Net is
also compared with two boundary-aware methods: Gated-SCNN (Takikawa et al.
2019b) and BFP (Ding et al. 2019). For the semi-supervised methods (USSS and
ALSSS) and boundary-aware methods (Gated-SCNN and BFP), ten labeled samples
are provided. For $\textbf{\em T}(x)$, the visual boundary knowledge is
translated from the $MixAll^{-}$, which does not contain samples from the
target category. Fig. 3 and Table 1 show the quantitative and qualitative
results, where we can see that most scores of proposed $\textbf{\em T}(10)$
achieve the state-of-the-art results on par with existing non-fully supervised
methods on Birds and HumanMatting datasets. Note that Flowers (Nilsback and
Zisserman 2007) dataset contains only $753$ manually annotated samples, which
leads to the inconsistent scores with Birds and HumanMatting. The most likely
reason is overfitting. More experiments on the algorithm pre-segmented Flowers
($8189$ samples) are given in the supplementary material. Moreover, with only
$10$ labeled samples, $\textbf{\em T}(10)$ can achieve better results than
some fully supervised methods and close results on par with the best fully
supervised method. Meanwhile, with all the labeled samples of the target
dataset, the $\textbf{\em T}(all)$ achieves almost all the highest scores. In
sum, such experiments demonstrate the practicability of the BKT-Task and the
proposed Trans-Net. More visual results and experiment results on THUR15K and
algorithm pre-segmented Flowers are given in the supplementary materials.

Table 2: The translation results between different dataset setting.
‘$\mathbb{S}^{q}\leftarrow\mathbf{S}^{p}$’ denotes translating knowledge of
source dataset $\mathbf{S}^{p}$ into the segmentation network for the target
dataset $\mathbb{S}^{q}$ (All scores in $\%$).

Index$\backslash$Task |  Humans $\leftarrow$ Birds |  Flowers $\leftarrow$ Birds |  (Birds, Flowers) $\leftarrow$ Humans |  (Humans, Birds, Flowers) $\leftarrow$ MixAll-
---|---|---|---|---
PA | 95.02 | 92.70 | (92.87, 81.32) | (94.82, 76.08, 86.10)
MPA | 94.54 | 94.00 | ( 91.75, 78.78) | (94.76, 84.15, 88.80)
MIoU | 88.86 | 86.10 | (79.83, 66.38) | (90.14, 56.33, 75.42)
FWIoU | 90.64 | 86.70 | (87.65, 68.53) | (90.16, 66.37, 76.01)

Table 3: The ablation study result of Trans-Net. $\textbf{\em T}_{self}^{-}$,
$\textbf{\em T}_{pseu}^{-}$ $\textbf{\em T}_{inner}^{-}$ and $\textbf{\em
T}_{outer}^{-}$ denote the Trans-Net without boundary-aware self-supervision,
pseudo triplet, inner discriminator and outer discriminator, respectively.
$\textbf{\em T}_{oneD}$ denotes the Trans-Net with only one discriminator.
$\textbf{\em T}(x)$ denotes the Trans-Net with $x$ labeled samples of the
target dataset.

Index$\backslash$Ablation | $\textbf{\em T}_{self}^{-}$ | $\textbf{\em T}_{pseu}^{-}$ | $\textbf{\em T}_{inner}^{-}$ | $\textbf{\em T}_{outer}^{-}$ | $\textbf{\em T}_{oneD}$ | $\textbf{\em T}(0)$ | $\textbf{\em T}(5)$ | $\textbf{\em T}(10)$ | $\textbf{\em T}(20)$ | $\textbf{\em T}(50)$ | $\textbf{\em T}(100)$ | $\textbf{\em T}(all)$
---|---|---|---|---|---|---|---|---|---|---|---|---
PA | 96.02 | 95.98 | 93.08 | 95.28 | 90.65 | 66.41 | 91.08 | 96.24 | 96.30 | 96.91 | 96.96 | 98.32
MPA | 95.94 | 96.10 | 93.08 | 95.29 | 90.54 | 66.97 | 91.25 | 96.29 | 96.31 | 96.88 | 96.95 | 98.32
MIoU | 92.12 | 92.39 | 87.06 | 90.98 | 82.85 | 48.90 | 83.62 | 92.74 | 92.87 | 94.00 | 94.09 | 96.68
FWIoU | 92.23 | 91.70 | 87.07 | 90.99 | 82.89 | 48.73 | 83.62 | 92.75 | 92.88 | 94.02 | 94.10 | 96.68

### Knowledge Translation between Different Dataset

To verify the robustness of the BKT-Task and the Trans-Net, the knowledge
translation experiments between different datasets are provided. For all the
tasks in Table 2, only $10$ labeled samples of the target dataset are used in
the Trans-Net. We can see that all the translation tasks achieve satisfactory
results. Even for the translation task between very different categories
({Humans $\leftarrow$ Birds} and {Flowers $\leftarrow$ Birds}), Trans-Net
still can achieve satisfactory results, which verifies the high extensibility
and practicability of the BKT-Task and the Trans-Net. For the scores on
Humans, $\textbf{\em T}(10)$ {Humans $\leftarrow$ MixAll-} (Table 1) achieves
better performance than {(Humans, Birds, Flowers) $\leftarrow$ MixAll-} (Table
2), which confirms the assumption that Birds and Flowers will embezzle the
segmentation capacity for Humans. What’s more, {Humans $\leftarrow$ MixAll-}
and {Flowers $\leftarrow$ MixAll-} (Table 1) achieve higher scores than
{Humans $\leftarrow$ Birds} and {Flowers $\leftarrow$ Birds}, which indicates
that translating knowledge from the mixed category dataset is more proper for
the BKT-Task.

Note that some scores of Birds in {(Birds, Flowers) $\leftarrow$ MixAll-} are
higher than {Birds $\leftarrow$ MixAll-}. One of the reasons is that the
Humans dataset is a human matting dataset with more accurate segmentation GT
than the MixAll- dataset, even we abandon some samples that are wicked GT in
MixAll- dataset. It also verifies that accurate boundary annotations are more
proper for the BKT-Task. The visual results of knowledge translation tasks
between different datasets are given in the supplementary materials.

### Ablation Study

To verify the effectiveness of the Trans-Net’s components, we do ablation
study on the boundary-aware self-supervised strategy, the pseudo triplet, two
boundary discriminators, and the different numbers of labeled samples. In this
section, the knowledge translation task is set as {Humans $\leftarrow$
MixAll-}. For $\textbf{\em T}_{self}^{-}$, $\textbf{\em T}_{pseu}^{-}$,
$\textbf{\em T}_{inner}^{-}$, $\textbf{\em T}_{inner}^{-}$ and $\textbf{\em
T}_{oneD}$, there are ten labeled samples of the Humans dataset. From Table 3,
we can see that Trans-Net achieves higher scores than $\textbf{\em
T}_{self}^{-}$ and $\textbf{\em T}_{pseu}^{-}$, which demonstrates the
effectiveness of the self-supervised strategy and the pseudo triplet. Fig. 3
shows that the result of $\textbf{\em T}_{self}^{-}$ has some holes. By
contrast, the result of Trans-Net is accurate, which indicates that the
boundary-aware self-supervised strategy is beneficial for eliminating the
incorrect holes. For the $\textbf{\em T}_{oneD}$, the inputs of two
discriminators will be concatenated then be input into one discriminator. The
scores of $\textbf{\em T}_{oneD}$ have dropped by about $10\%$ comparing with
Trans-Net on all indexes, which demonstrates that the two-discriminator
framework is useful for improving the segmentation performance. Meanwhile,
$\textbf{\em T}(10)$ achieves about $4\%$ increase on the scores of
$\textbf{\em T}_{inner}^{-}$ and about $2\%$ increase on the scores of
$\textbf{\em T}_{outer}^{-}$, which demonstrates the effectiveness of inner
and outer boundary discriminators. For the different numbers of labeled
samples, we find that $10$-labeled-samples is a critical cut-off point, which
can supply relatively sufficient guidance. An object usually contains multiple
components, which have different edges. Without any guidance of labeled
samples, the segmentation network will not be aware of the desired boundary of
the object. Therefore, the $\textbf{\em T}(0)$ achieves the worst performance.
With all labeled samples, $\textbf{\em T}(all)$ achieves SOTA results on par
with existing fully supervised methods. More visual results of the ablation
study are given in the supplementary materials.

## Conclusion

In this paper, we study a new Boundary Knowledge Translation Task (BKT-Task).
This is inspired by the fact that humans can perfectly segment the object from
an image according to the boundary information without knowing the object’s
category. The goal of BKT-Task is to translate the visual boundary knowledge
of source datasets into the segmentation network for new categories in a least
effort and dependable way.

Based on the proposed BKT-Task, we introduce the Translation Segmentation
Network (Trans-Net) for segmenting the foreground from the background. The
Trans-Net contains a segmentation network and two boundary discriminators,
which are devised for translating visual boundary knowledge from the source
dataset to the target one. Meanwhile, boundary-aware self-supervision and
pseudo triplet are devised to enhance boundary consistency and help the two
discriminators distinguish boundary, respectively. Exhaustive experiments
verify the promising generalization and practicability of the BKT-Task.
Furthermore, with only tens of labeled sample of the target dataset, the
Trans-Net achieves results on par with fully supervised methods. In the
future, we will generalize the knowledge translation task into other
applications, such as image matting and image classification. Meanwhile, we
will extend the Trans-Net into more general multiple object image
segmentation.

##  Acknowledgments

This work was supported by National Natural Science Foundation of China
(No.62002318), Key Research and Development Program of Zhejiang Province
(2020C01023), Zhejiang Provincial Science and Technology Project for Public
Welfare (LGF21F020020), Programs Supported by Ningbo Natural Science
Foundation (202003N4318), and Major Scientifc Research Project of Zhejiang Lab
(No. 2019KD0AC01).

## Potential Ethical Impact

We discuss the potential positive and negative impact of the proposed work
here. Positive: the proposed method can be applied to image foreground
segmentation, and also be extended to other general image segmentation in the
future, which is beneficial for image editing applications. Moreover, the
proposed Boundary Knowledge Translation Task (BKT-Task) will potentially
change the conventional way that the category-aware label supervises the model
learning of corresponding category-aware knowledge. With vast public annotated
datasets, training a segmentation network for the new category requires only
tens of labeled samples of the new category. The BKT-Task will dramatically
reduce the requirement for the labeled samples of new category, while the
trained model achieves gratifying performances. Negative: the research might
be adopted to generate fake images, which can be used for malicious purposes.

## References

  * Arbelle and Raviv (2018) Arbelle, A.; and Raviv, T. R. 2018.  Microscopy Cell Segmentation via Adversarial Neural Networks.  _ISBI_ 645–648.
  * Bertasius, Shi, and Torresani (2016) Bertasius, G.; Shi, J.; and Torresani, L. 2016.  Semantic Segmentation with Boundary Neural Fields.  _CVPR_ 3602–3610.
  * Catherine et al. (2011) Catherine, W.; Steve, B.; Peter, W.; Pietro, P.; and Serge, B. 2011.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology.
  * Chaurasia and Culurciello (2017) Chaurasia, A.; and Culurciello, E. 2017.  LinkNet: Exploiting encoder representations for efficient semantic segmentation.  _VCIP_ 1–4.
  * Chen et al. (2016) Chen, L.; Barron, J. T.; Papandreou, G.; Murphy, K.; and Yuille, A. L. 2016.  Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform.  _CVPR_ 4545–4554.
  * Chen et al. (2017) Chen, L.; Papandreou, G.; Schroff, F.; and Adam, H. 2017.  Rethinking Atrous Convolution for Semantic Image Segmentation.  _arXiv_ .
  * Chen, Artieres, and Denoyer (2019) Chen, M.; Artieres, T.; and Denoyer, L. 2019.  Unsupervised Object Segmentation by Redrawing.  _NeurIPS_ 12726–12737.
  * Cheng et al. (2017) Cheng, D.; Meng, G.; Xiang, S.; and Pan, C. 2017.  FusionNet: Edge Aware Deep Convolutional Networks for Semantic Segmentation of Remote Sensing Harbor Images.  _IEEE JSTAEORS_ 10(12): 5769–5783.
  * Cheng et al. (2011) Cheng, M.; Zhang, G.; Mitra, N. J.; Huang, X.; and Hu, S. 2011.  Global contrast based salient region detection.  _CVPR_ 37(3): 409–416.
  * Cheng et al. (2014) Cheng, M.-M.; Mitra, N.; Huang, X.; and Hu, S.-M. 2014.  SalientShape: group saliency in image collections.  _The Visual Computer_ 30(4): 443–453.
  * Company (2019) Company, A. 2019.  Matting human datasets.  https://github.com/aisegmentcn/matting˙human˙datasets.
  * Dai et al. (2019) Dai, C.; Mo, Y.; Angelini, E.; Guo, Y.; and Bai, W. 2019.  Transfer Learning from Partial Annotations for Whole Brain Segmentation.  _MICCAI Workshop_ 199–206.
  * Ding et al. (2019) Ding, H.; Jiang, X.; Liu, A. Q.; Thalmann, N. M.; and Wang, G. 2019.  Boundary-Aware Feature Propagation for Scene Segmentation.  _ICCV_ 6819–6829.
  * Fan et al. (2019) Fan, D.; Lin, Z.; Zhao, J.; Liu, Y.; Zhang, Z.; Hou, Q.; Zhu, M.; and Cheng, M. 2019\.  Rethinking RGB-D Salient Object Detection: Models, Datasets, and Large-Scale Benchmarks.  _arXiv_ .
  * Gulrajani et al. (2017) Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; and Courville, A. 2017.  Improved training of wasserstein GANs.  _NeurIPS_ 5769–5779.
  * Han and Yin (2017) Han, L.; and Yin, Z. 2017.  Transferring Microscopy Image Modalities with Conditional Generative Adversarial Networks.  _CVPR_ 2017: 851–859.
  * Hayder, He, and Salzmann (2017) Hayder, Z.; He, X.; and Salzmann, M. 2017.  Boundary-Aware Instance Segmentation.  _CVPR_ 587–595.
  * Hong et al. (2017) Hong, S.; Yeo, D.; Kwak, S.; Lee, H.; and Han, B. 2017.  Weakly Supervised Semantic Segmentation Using Web-Crawled Videos.  _CVPR_ 2224–2232.
  * Hou et al. (2017) Hou, Q.; Cheng, M.; Hu, X.; Borji, A.; Tu, Z.; and Torr, P. H. S. 2017.  Deeply Supervised Salient Object Detection with Short Connections.  _CVPR_ 5300–5309.
  * Hsu, Lin, and Chuang (2018) Hsu, K.; Lin, Y.; and Chuang, Y. 2018.  Co-attention CNNs for Unsupervised Object Co-segmentation.  _IJCAI_ 748–756.
  * Hung et al. (2018) Hung, W.; Tsai, Y.; Liou, Y.; Lin, Y.; and Yang, M. 2018.  Adversarial Learning for Semi-Supervised Semantic Segmentation.  _BMVC_ 65\.
  * Kalluri et al. (2019) Kalluri, T.; Varma, G.; Chandraker, M.; and Jawahar, C. V. 2019.  Universal Semi-Supervised Semantic Segmentation.  _ICCV_ 5259–5270.
  * Khoreva et al. (2017) Khoreva, A.; Benenson, R.; Hosang, J.; Hein, M.; and Schiele, B. 2017.  Simple Does It: Weakly Supervised Instance and Semantic Segmentation.  _CVPR_ 1665–1674.
  * Li and Yu (2016) Li, G.; and Yu, Y. 2016.  Visual Saliency Detection Based on Multiscale Deep CNN Features.  _IEEE TIP_ 5012–5024.
  * Li et al. (2018) Li, H.; Xiong, P.; An, J.; and Wang, L. 2018.  Pyramid Attention Network for Semantic Segmentation.  _BMVC_ 285\.
  * Li, Arnab, and Torr (2018) Li, Q.; Arnab, A.; and Torr, P. H. S. 2018.  Weakly- and Semi-Supervised Panoptic Segmentation.  _ECCV_ 102–118.
  * Lin et al. (2017) Lin, T.; Dollar, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. 2017\.  Feature Pyramid Networks for Object Detection.  _CVPR_ 936–944.
  * Luc et al. (2016) Luc, P.; Couprie, C.; Chintala, S.; and Verbeek, J. 2016.  Semantic Segmentation using Adversarial Networks.  _NIPS_ .
  * Minaee et al. (2020) Minaee, S.; Boykov, Y.; Porikli, F.; Plaza, A.; Kehtarnavaz, N.; and Terzopoulos, D. 2020.  Image Segmentation Using Deep Learning: A Survey.  _arXiv_ .
  * Mottaghi et al. (2014) Mottaghi, R.; Chen, X.; Liu, X.; Cho, N.; Lee, S.; Fidler, S.; Urtasun, R.; and Yuille, A. L. 2014.  The Role of Context for Object Detection and Semantic Segmentation in the Wild.  _CVPR_ 891–898.
  * Movahedi and Elder (2010) Movahedi, V.; and Elder, J. H. 2010.  Design and perceptual validation of performance measures for salient object segmentation.  _CVPR_ 49–56.
  * Nilsback and Zisserman (2007) Nilsback, M. E.; and Zisserman, A. 2007.  Delving into the whorl of flower segmentation.  _BMVC_ 1–10.
  * Ostyakov et al. (2018) Ostyakov, P.; Suvorov, R.; Logacheva, E.; Khomenko, O.; and Nikolenko, S. I. 2018\.  SEIGAN: Towards Compositional Image Generation by Simultaneously Learning to Segment, Enhance, and Inpaint.  _arXiv_ .
  * Papandreou et al. (2015) Papandreou, G.; Chen, L.; Murphy, K.; and Yuille, A. L. 2015.  Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation.  _ICCV_ 1742–1750.
  * Peng et al. (2017) Peng, C.; Zhang, X.; Yu, G.; Luo, G.; and Sun, J. 2017.  Large Kernel Matters ¡ª Improve Semantic Segmentation by Global Convolutional Network.  _CVPR_ 1743–1751.
  * Qin et al. (2019) Qin, X.; Zhang, Z.; Huang, C.; Gao, C.; Dehghan, M.; and Jagersand, M. 2019.  BASNet: Boundary-Aware Salient Object Detection.  _CVPR_ 7479–7489.
  * Remez, Huang, and Brown (2018) Remez, T.; Huang, J.; and Brown, M. 2018.  Learning to Segment via Cut-and-Paste.  _ECCV_ 39–54.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  _MICCAI_ 234–241.
  * Ruan, Tong, and Lu (2011) Ruan, X.; Tong, N.; and Lu, H. 2011.  How far we away from a perfect visual saliency detection - DUT-OMRON: a new benchmark dataset.  FCV2014.
  * Shaban et al. (2017) Shaban, A.; Bansal, S.; Liu, Z.; Essa, I.; and Boots, B. 2017.  One-Shot Learning for Semantic Segmentation.  _BMVC_ .
  * Shi et al. (2016) Shi, J.; Yan, Q.; Xu, L.; and Jia, J. 2016.  Hierarchical Image Saliency Detection on Extended CSSD.  _IEEE TPAMI_ 38(4): 717–729.
  * Siam, Oreshkin, and Jagersand (2019) Siam, M.; Oreshkin, B. N.; and Jagersand, M. 2019.  AMP: Adaptive Masked Proxies for Few-Shot Segmentation.  _ICCV_ 5249–5258.
  * Souly, Spampinato, and Shah (2017) Souly, N.; Spampinato, C.; and Shah, M. 2017.  Semi Supervised Semantic Segmentation Using Generative Adversarial Network.  _ICCV_ 5689–5697.
  * Sun et al. (2019) Sun, R.; Zhu, X.; Wu, C.; Huang, C.; Shi, J.; and Ma, L. 2019.  Not All Areas Are Equal: Transfer Learning for Semantic Segmentation via Hierarchical Region Selection.  _CVPR_ 4360–4369.
  * Takikawa et al. (2019a) Takikawa, T.; Acuna, D.; Jampani, V.; and Fidler, S. 2019a.  Gated-SCNN: Gated Shape CNNs for Semantic Segmentation.  _arXiv_ .
  * Takikawa et al. (2019b) Takikawa, T.; Acuna, D.; Jampani, V.; and Fidler, S. 2019b.  Gated-SCNN: Gated Shape CNNs for Semantic Segmentation.  _ICCV_ 5229–5238.
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; Kavukcuoglu, K.; and Wierstra, D. 2016\.  Matching Networks for One Shot Learning.  _NeurIPS_ 3637–3645.
  * Wang et al. (2019a) Wang, K.; Liew, J. H.; Zou, Y.; Zhou, D.; and Feng, J. 2019a.  PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment.  _ICCV_ 9197–9206.
  * Wang et al. (2019b) Wang, Y.; Zhang, J.; Kan, M.; Shan, S.; and Chen, X. 2019b.  Self-supervised Scale Equivariant Network for Weakly Supervised Semantic Segmentation.  _arXiv_ .
  * Wang, Simoncelli, and Bovik (2003) Wang, Z.; Simoncelli, E. P.; and Bovik, A. C. 2003.  Multiscale Structural Similarity for Image Quality Assessment.  _ACSSC_ 2: 1398–1402.
  * Xian et al. (2019) Xian, Y.; Choudhury, S.; He, Y.; Schiele, B.; and Akata, Z. 2019.  Semantic Projection Network for Zero- and Few-Label Semantic Segmentation.  _CVPR_ 8256–8265.
  * Xue et al. (2018) Xue, Y.; Xu, T.; Zhang, H.; Long, L. R.; and Huang, X. 2018.  SegAN: Adversarial Network with Multi-scale L 1 Loss for Medical Image Segmentation.  _Neuroinformatics_ 16(3): 383–392.
  * Yan et al. (2013) Yan, Q.; Xu, L.; Shi, J.; and Jia, J. 2013.  Hierarchical Saliency Detection.  _CVPR_ 1155–1162.
  * Yu et al. (2018) Yu, C.; Wang, J.; Peng, C.; Gao, C.; Yu, G.; and Sang, N. 2018.  Learning a Discriminative Feature Network for Semantic Segmentation.  _CVPR_ 1857–1866.
  * Zhang et al. (2019) Zhang, C.; Lin, G.; Liu, F.; Yao, R.; and Shen, C. 2019.  CANet: Class-Agnostic Segmentation Networks With Iterative Refinement and Attentive Few-Shot Learning.  _CVPR_ 5217–5226.
  * Zhang et al. (2017) Zhang, R.; Tang, S.; Lin, M.; Li, J.; and Yan, S. 2017.  Global-residual and local-boundary refinement networks for rectifying scene parsing predictions.  _IJCAI_ 3427–3433.
  * Zhang et al. (2018) Zhang, X.; Wei, Y.; Yang, Y.; and Huang, T. S. 2018.  SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation.  _IEEE TCYB_ .
  * Zhao et al. (2017) Zhao, H.; Shi, J.; Qi, X.; Wang, X.; and Jia, J. 2017.  Pyramid Scene Parsing Network.  _CVPR_ 6230–6239.


Current affiliation: ]Nippon Steel Corporation, Steel Research Laboratories.

# Origin of magnetically dead layers in spinel ferrites MFe2O4 grown on
$\text{Al}_{2}\text{O}_{3}$: Effects of post-deposition annealing studied by
XMCD

Yosuke Nonaka<EMAIL_ADDRESS>[  Department of Physics,
the University of Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan  Yuki K. Wakabayashi
Department of Electrical Engineering and Information Systems, The University
of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan  Goro Shibata
Department of Physics, the University of Tokyo, Bunkyo-ku, Tokyo 113-0033,
Japan  Materials Sciences Research Center, Japan Atomic Energy Agency (JAEA),
Sayo, Hyogo 679-5148, Japan  Shoya Sakamoto  Department of Physics, the
University of Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan  The Institute for Solid
State Physics, the University of Tokyo, Kashiwa, Chiba 277-8581, Japan
Keisuke Ikeda  Department of Physics, the University of Tokyo, Bunkyo-ku,
Tokyo 113-0033, Japan  Zhendong Chi  Department of Physics, the University of
Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan  Yuxuan Wan  Department of Physics,
the University of Tokyo, Bunkyo-ku, Tokyo 113-0033, Japan  Masahiro Suzuki
Department of Physics, the University of Tokyo, Bunkyo-ku, Tokyo 113-0033,
Japan  Arata Tanaka  Graduate School of Advanced Science of Matter, Hiroshima
University, Higashi-hiroshima, Hiroshima 739-8530, Japan  Masaaki Tanaka
Department of Electrical Engineering and Information Systems, The University
of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan  Center for
Spintronics Research Network, Graduate School of Engineering, The University
of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan  Atsushi Fujimori
Department of Physics, the University of Tokyo, Bunkyo-ku, Tokyo 113-0033,
Japan  Center for Quantum Science and Technology and Department of Physics,
National Tsing Hua University, Hsinchu 30013, Taiwan

###### Abstract

We study the electronic and magnetic states of as-grown and annealed
MFe2O4(111)/$\text{Al}_{2}\text{O}_{3}$(111) (M=Co, Ni) thin films with
various thicknesses grown on Si(111) substrates with the
$\gamma$-$\text{Al}_{2}\text{O}_{3}$(111) buffer layers by using x-ray
absorption spectroscopy (XAS) and x-ray magnetic circular dichroism (XMCD), to
investigate magnetically dead layers in these films. Although the magnetically
dead layers in the as-grown samples are formed near the interface with the
$\text{Al}_{2}\text{O}_{3}$ buffer layer, we reveal that ferrimagnetic order
is partially recovered by post-deposition annealing at 973 K for 48 hours in
air. By analyzing the line shapes of the XAS and XMCD spectra, we conclude
that, in the dead layers, there are a significant number of vacancies at the
$T_{d}$ sites of the spinel structure, which may be the microscopic origin of
the degraded ferrimagnetic order in the MFe2O4 thin films.

##  I Introduction

Electronics have achieved great success by controlling the charge degree of
freedom of electrons in semiconductors. By adding the spin degree of freedom
to the existing electronics, a new field of electronics known as spintronics
is being developed. Spintronics devices have several advantages over existing
electronic devices, e.g., non-volatilitiy, low energy consumption, high-speed,
infinite endurance [1]. To establish next-generation electronics with
semiconductor-based spintronics, injection of spin-polarized carriers into
semiconductors should be realized at room temperature by utilizing a magnetic
spin injector formed on the semiconductor. The spin filter is a promising
technology to inject highly spin-polarized current into semiconductors.
Insulating spinel-ferrite thin films are attracting high attention as a
promising material for spin filters because they have the down-spin and up-
spin conduction bands at different energies [the 3d-t2g band of
Fe${}^{3+}(O_{h})$ and the 3d-e band of Fe${}^{3+}(T_{d})$, as Fig. 1(a)] [2],
and their Néel temperatures are much higher than room temperature (790 K for
$\text{CoFe}_{2}\text{O}_{4}$ [3, 4, 5], and 850 K for
$\text{NiFe}_{2}\text{O}_{4}$ [2, 6]). However, the experimentally obtained
spin-filtering efficiency of ferrite-based tunnel barriers still remains
considerably lower than the theoretical value of $100\%$ [7, 8, 9, 10]. In
recent investigation, possible causes for this low spin-filtering efficiency
have been proposed. One is that midgap impurity states are induced by
structural and/or chemical defects [4, 2, 11]. For example, in
$\text{CoFe}_{2}\text{O}_{4}$, the presence of Co3+ degrades the spin filter
due to the formation of up-spin states in the middle of the band gap. Another
possibility is that layers with degraded magnetic properties are formed in the
interfacial region, which is often called magnetically dead layers. They are
well known for epitaxial $\text{Fe}_{3}\text{O}_{4}$ thin films grown on MgO
substrates, and are thought to originate from antiphase boundaries and their
antiferromagnetic interdomain exchange interaction [12, 13, 14, 15, 16]. In
addition, a recent study on epitaxial $\text{Fe}_{3}\text{O}_{4}$ thin films
grown on MgO substrates revealed that the first $\text{Fe}_{3}\text{O}_{4}$
monolayer lacks Fe3+ ions at the tetrahedral sites [17], resulting in a loss
of the ferrimagnetic order due to missing superexchange paths.

In recent works [18, 19, 20], the magnetic properties of
$\textit{M}\text{Fe}_{2}\text{O}_{4}$(111)/$\text{Al}_{2}\text{O}_{3}$(111)/Si(111)
(M=Co, Ni) structures [21] were studied using the element-specific probes of
x-ray absorption spectroscopy (XAS) and x-ray magnetic circular dichroism
(XMCD). The authors revealed that the decrease of the inversion parameter y ,
defined by
$=[\textit{M}_{1-y}\text{Fe}_{y}]_{T_{d}}[\text{Fe}_{2-y}\textit{M}_{y}]_{O_{h}}\text{O}_{4}$
is correlated with the formation of magnetically dead layers. The inversion
parameter y, which is equal to unity in the ideal inverse spinel, represents
the regularity of the cation distribution in the spinel structure. It was
found that the reduced thickness of the $\text{CoFe}_{2}\text{O}_{4}$ film
leads to a significant reduction in both $y$ and ferrimagnetic ordered moment.
It was also found that $\text{NiFe}_{2}\text{O}_{4}$ has a larger $y$
($=0.79\text{--}0.91$) than $\text{CoFe}_{2}\text{O}_{4}$
($y=0.54\text{--}0.75$), indicating that Ni has a higher $O_{h}$-site
selectivity than Co. Considering that the cation distribution in
$\textit{M}\text{Fe}_{2}\text{O}_{4}$ can be modified by post-deposition
annealing [5, 22], it is expected that annealing recovers the ferrimagnetic
behavior of the interfacial region degraded by the formation of the
magnetically dead layers. In the present study, we investigate the electronic
and magnetic states of as-grown and annealed
$\textit{M}\text{Fe}_{2}\text{O}_{4}$(111)/$\text{Al}_{2}\text{O}_{3}$(111)/Si(111)
(M=Co, Ni) thin films using XAS and XMCD. We reveal that the degraded magnetic
properties of the magnetically dead layer can be partially recovered by
annealing through the redistribution of the cations. Furthermore, we find that
there are a significant number of vacant $T_{d}$ sites in the magnetically
dead layers, and that such vacancies may degrade the ferrimagnetic order in
the $\textit{M}\text{Fe}_{2}\text{O}_{4}$ thin films.

##  II Experimental Methods

$\text{CoFe}_{2}\text{O}_{4}$(111) and $\text{NiFe}_{2}\text{O}_{4}$(111) thin
films with various thicknesses were epitaxially grown on
$\text{Al}_{2}\text{O}_{3}$(111) buffer layers formed on Si(111) substrates
using the pulsed laser deposition (PLD) method. Schematic illustrations of the
sample structures are shown in Figs. 1(b) and 1(c).

Figure 1:  Schematic description of the electronic structure of
$\textit{M}\text{Fe}_{2}\text{O}_{4}$ (M = Co or Ni) and the sample
structures. (a) Density of states of the valence-band top and conduction-band
bottom for $\textit{M}\text{Fe}_{2}\text{O}_{4}$, reproduced from Ref. 19. (b)
Cross-sectional illustration of the
$\text{CoFe}_{2}\text{O}_{4}$(111)/$\text{Al}_{2}\text{O}_{3}$(111)/Si(111)
structure, adapted from Ref. 18. (c) Cross-sectional illustration of the
$\text{NiFe}_{2}\text{O}_{4}$(111)/$\text{Al}_{2}\text{O}_{3}$(111)/Si(111)
structure, adapted from Ref. 19.

2.4 nm- and 1.4 nm-thick $\gamma$-$\text{Al}_{2}\text{O}_{3}$(111) buffer
layers were adopted for $\text{CoFe}_{2}\text{O}_{4}$ and
$\text{NiFe}_{2}\text{O}_{4}$ thin films, respectively. The thicknesses of the
films were d = 1.4, 2.3, 4, and 11 nm for $\text{CoFe}_{2}\text{O}_{4}$, and d
= 1.7, 3.5, and 5.2 nm for $\text{NiFe}_{2}\text{O}_{4}$. In order to avoid
charging up of the samples during the XAS and XMCD measurements, we used
highly phosphorus-doped Si(111) substrates with a low electrical resistivity 2
m$\Omega$cm. For the epitaxial growth of the
$\gamma$-$\text{Al}_{2}\text{O}_{3}$ buffer layers on the Si substrates, we
used solid-phase reaction of Al and $\text{SiO}_{2}$. More detailed
description of sample preparation and characterization are provided in Refs.
18, 19. Some of the $\text{CoFe}_{2}\text{O}_{4}$ and
$\text{NiFe}_{2}\text{O}_{4}$ thin films were annealed in order to
redistribute the cations in the thin films, aiming at the recovery of the
inversion parameter $y$. In order to suppress the oxidation of the Si
substrate, the post-deposition annealing temperature was set to 973 K because
the oxidation rate of Si rapidly increases particularly above 1073 K [23]. The
films were annealed at 973 K for 48 hours in air. Since higher inversion
parameters of $\text{CoFe}_{2}\text{O}_{4}$ have been reported in slowly
cooled bulk crystals than in rapidly quenched ones [5], we set a slow cooling
rate of 1 K/min down to 673 K and then the samples were naturally cooled down
to room temperature.

XMCD measurements were performed at beamline BL-16A of Photon Factory, High
Energy Accelerator Research Organization (KEK-PF). A magnetic field of 5 T was
applied parallel to the incident x rays. The measurements were done at room
temperature ($\sim 300$ K). The absorption signals were detected in the total-
electron-yield (TEY) mode. Since the probing depth in the TEY mode is a few nm
[24, 25], the XAS and XMCD spectra of a few nm-thick films reflect the
electronic and magnetic properties of the entire
$\textit{M}\text{Fe}_{2}\text{O}_{4}$ film including the magnetically dead
layers near the spinel/$\text{Al}_{2}\text{O}_{3}$ interface. By increasing
the film thickness to $\gtrsim 5\text{ nm}$, the spectra more strongly reflect
the signals away from the interface. The XMCD signals were collected by
switching the photon helicity at every energy point.

In order to analyze the XAS and XMCD spectra quantitatively, we performed
configuration-interaction (CI) cluster-model calculation [26]. The model is a
good approximation for localized electron systems like the insulating spinel
ferrites. In the present calculation, we used an Xtls version 8.5 program
[27]. In choosing parameter values for the calculation, we adopted the
following empirical rules : (1) The ratio between the on-site Coulomb energy
between 3d electrons Udd and the attractive 2p core hole-3d electron Coulomb
energy Udc was fixed to $U_{dc}/U_{dd}\sim 1.3$ [28, 29]. (2) The ratio
between Slater-Koster parameters $(pd\sigma)$ and $(pd\pi)$ was fixed to
$(pd\sigma)/(pd\pi)=-2.17$ [30]. The hybridization strength between O 2p
orbitals $T_{pp}$ was fixed to 0.7 eV for the $O_{h}$ site and 0 eV for the
$T_{d}$ site [27, 28]. The Slater integrals were set to be 80% of the Hartree-
Fock values. Thus, the crystal-field splitting 10Dq, the charge-transfer
energy $\Delta$, $U_{dd}$, and $(pd\sigma)$ were treated as adjustable
parameters. As for the Fe cations, the parameters were adjusted to reproduce
all the measured Fe $L_{2,3}$-edge spectra by the weighted sum of the
calculated spectra for the Fe${}^{2+}(O_{h})$, Fe${}^{3+}(T_{d})$, and
Fe${}^{3+}(O_{h})$ ions. Here, parameters for Fe${}^{2+}(O_{h})$ were adopted
from the previous report on $\text{NiFe}_{2}\text{O}_{4}$ [19]. The adjusted
parameter values are listed in Table 1.

Table 1: Parameter values for the CI cluster-model calculations used in the present study in units of eV. Parameters for Fe${}^{2+}(O_{h})$ were adopted from Ref. 19. | $\Delta$ | 10$Dq$ | $(pd\sigma)$ | $U_{dd}$
---|---|---|---|---
Fe${}^{3+}(O_{h})$ | 3.0 | 0.8 | 1.5 | 7.0
Fe${}^{3+}(T_{d})$ | 2.5 | -0.5 | 1.8 | 6.0
Fe${}^{2+}(O_{h})$ | 6.5 | 0.9 | 1.4 | 6.0

As in our previous works [18, 19], we fitted the weighted sum of the
calculated XAS and XMCD spectra for the Fe${}^{2+}(O_{h})$,
Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ ions to the experimental ones. From
this analysis, we obtained the relative amount of the three Fe components and
thereby the inversion parameter y.

##  III Results and Discussion

###  III.1 $\text{CoFe}_{2}\text{O}_{4}$ thin films

Figure 2:  Fe L2,3-edge XAS and XMCD spectra of the as-grown and annealed
$\text{CoFe}_{2}\text{O}_{4}$ thin films with various thicknesses. (a), (b) Fe
L2,3-edge XAS and XMCD spectra. In panel (a), a linear and a two-step
backgrounds have been subtracted from the raw XAS spectra. The two-step
backgrounds are composed of two arctangent functions located at the peak
positions of $L_{3}$ and $L_{2}$ white lines. The height of the $L_{3}$ and
$L_{2}$ steps are set to $\frac{2}{3}$ and $\frac{1}{3}$ of the average height
around 740 eV, respectively. (c), (d) Magnified views of the Fe L3-edge XMCD
spectra normalized to the height of peak c. The XMCD intensities decrease with
decreasing thickness in the as-grown films [(b) and (c)], especially at peak
b, reflecting the magnetically dead layers near the interface. Both the entire
XMCD intensities and the relative intensity at peak b are partially recovered
by post-deposition annealing particularly in thinner films [(b) and (d)].

Figure 2 shows the Fe $L_{2,3}$-edge XAS ($\mu^{+}+\mu^{-}$) and XMCD
($\mu^{+}-\mu^{-}$) spectra of the $\text{CoFe}_{2}\text{O}_{4}$ thin films
with various thicknesses. Here, $\mu^{+}$ ($\mu^{-}$) denote the absorption
coefficient for photon helicity parallel (antiparallel) to the majority-spin
direction. Figure 2(a) shows that the Fe $L_{2,3}$-edge XAS spectral line
shapes do not change significantly by post-deposition annealing. In contrast,
as shown in Fig. 2(b), the Fe $L_{2,3}$-edge XMCD intensities of the annealed
samples dramatically increase compared to the as-grown ones, particularly in
thinner films (d = 1.4 and 2.3 nm). The increase of the XMCD intensities
clearly demonstrates that the post-deposition annealing recovers the
ferrimagnetic order in the magnetically dead layer of the as-grown films.The
L3-edge XMCD spectra show characteristic three peaks denoted by a, b, and c.
It is well established that the three peaks a, b, and c originate from the
Fe${}^{2+}(O_{h})$, Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ ionic states,
respectively [31, 9, 32, 6]. Therefore, one can discuss the Fe-ion
distribution in spinel ferrites by estimating the intensities of these XMCD
peaks. The spins of the Fe${}^{3+}(T_{d})$ ions are antiferromagnetically
coupled with those at the $O_{h}$ sites, resulting in the opposite signs
between peak b and peaks a/c. Due to these distinct peak features, XMCD
spectra are more sensitive to the cation distribution in spinel ferrites than
XAS [31, 9, 32]. Figures 2(c) and 2(d) show comparison of the Fe L3-edge XMCD
spectral line shapes of the as-grown and annealed
$\text{CoFe}_{2}\text{O}_{4}$ films. The spectra have been normalized to the
height of peak c to compare the spectral line shapes because the
$L_{2,3}$-edge XMCD intensities decrease with the reduction of the film
thickness due to the magnetically dead layers. Figures 2(b) and 2(c) show that
the $L_{2,3}$-edge XMCD intensity decreases with the reduction of the
thickness in the as-grown $\text{CoFe}_{2}\text{O}_{4}$ films and that the
XMCD intensity at peak b decreases faster than peaks a and c as previously
reported [18]. This indicates the reduction of the Fe${}^{3+}(T_{d})$
component of XMCD in magnetically dead layers near the interface. After post-
deposition annealing, as shown in Fig. 2(b) and 2(d), both the entire XMCD
intensities and the intensity at peak b are recovered simultaneously
particularly in thinner films. This suggests that the Fe ions are
redistributed and that ferrimagnetic order are simultaneously recovered near
the interface by annealing.

Figure 3:  Comparison between experimental and calculated XAS and XMCD spectra
at the Fe $L_{2,3}$ edge of the as-grown and annealed
$\text{CoFe}_{2}\text{O}_{4}$ thin films.  Figure 4:  Thickness dependences of
the Fe-ion distribution and the $L_{3}$-edge XMCD intensities for the as-grown
and annealed $\text{CoFe}_{2}\text{O}_{4}$ thin films. (a) Relative amount of
the Fe${}^{2+}(O_{h})$, Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ ions as
functions of film thickness. The inversion parameter y is also plotted (right
axis). Dashed lines and solid line indicate the as-grown and annealed samples,
respectively. (b) Fe $L_{3}$-edge XMCD intensity of the as-grown and annealed
films. The increase of y by annealing ($\Delta y$) and the increase ratio of
the XMCD intensity (R) are plotted on the right axes. The inversion parameter
y decrease with reducing thickness in the as-grown samples [red dashed line in
(a)], and y increases by annealing [red solid line in (a)]. $\Delta y$ and R
similarly decrease with the film thickness [green and brown lines in (b)],
suggesting that the recovery of the magnetically dead layer is achieved by the
redistribution of cations.

To clarify the correlation between the cation redistribution and the recovery
of the ferrimagnetic order, quantitative spectral line-shape analyses are
necessary. We, therefore, performed CI cluster-model calculation to reproduce
the experimental XAS and XMCD spectra by the weighted sum of calculated
Fe${}^{2+}(O_{h})$, Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ spectra using
the parameters listed in Table 1. Figure 3 shows comparison between the
experimental and calculated XAS and XMCD spectra at the Fe $L_{2,3}$ edge. All
the experimental spectra are well reproduced by the calculated spectra. Since
the amount of the ion is proportional to the weight for each calculated XAS
and XMCD spectra, the relative amount can be calculated by the spectral weight
ratio. The relative amount deduced from fitting is plotted as functions of
film thickness in Fig. 4(a), where the inversion parameter y is also deduced
and plotted. One can see that y decreases with reducing thickness in the as-
grown samples, and that y increases by the post-deposition annealing. The
increase of y by annealing is more pronounced in the thinner samples,
reflecting the redistribution of cations near the interface. Figure 4(b) shows
the Fe $L_{3}$-edge XMCD intensities of the as-grown and annealed films as
functions of film thickness. The increase ratio of the Fe $L_{3}$-edge XMCD
intensity [$R\equiv\text{XMCD (annealed)}/\text{XMCD (as grown)}$] and the
increase of the inversion parameter [$\Delta y\equiv
y\text{(annealed)}-y\text{(as grown)}$] are also shown for comparison. One can
see similar thickness dependences between R and $\Delta y$, which suggests
that the recovery of the magnetically dead layer is achieved by the
redistribution of cations. Based on the above results and the definition of
the inversion parameter y of spinel ferrites
($[\text{Co}_{1-y}\text{Fe}_{y}]_{T_{d}}[\text{Fe}_{2-y}\text{Co}_{y}]_{O_{h}}\text{O}_{4}$),
one may think that the recovery of the magnetically dead layer by post-
deposition annealing is caused by the reduction of the antisites near the
interface, that is, the Co ions irregularly occupying the $T_{d}$ sites are
replaced by Fe ions after annealing. However, one also needs to take into
account the cation vacancies at the $T_{d}$ sites, as discussed below.

Figure 5:  Co L2,3-edge XAS and XMCD spectra of the as-grown and annealed
$\text{CoFe}_{2}\text{O}_{4}$ thin films with various thicknesses. (a), (b) Co
L2,3-edge XAS and XMCD spectra. (c), (d) Magnified views of Co L3-edge XMCD
spectra normalized to the height of peak e. Since peak d originates from
Co${}^{2+}(T_{d})$ ions [18], the increase of peak d indicates that the number
of ferrimagnetically ordered Co${}^{2+}(T_{d})$ ions increase by post-
deposition annealing.

Figure 5 shows the Co $L_{2,3}$-edge XAS and XMCD spectra for various film
thicknesses and the effect of annealing on these spectra. In order to see the
changes in the spectral line shapes, we show the Co-$L_{3}$ edge XMCD spectra
normalized to the height of peak e in Figs. 5(c) and 5(d). One can see that
the positive peak d increases by annealing, particularly in the thinner films.
As described in the previous report [18], the positive peak at the Co $L_{3}$
edge originates from Co${}^{2+}(T_{d})$ ions. Therefore, the increase of peak
d indicates the increase of the number of ferrimagnetically ordered
Co${}^{2+}(T_{d})$ ions near the interface after annealing. The increase of
the positive peak components could also explain the slight increase of the Co
$L_{2,3}$-edge XMCD intensities by annealing. Surprisingly, not only the
number of Fe${}^{3+}(T_{d})$ ions but also that of ferrimagnetically ordered
Co${}^{2+}(T_{d})$ ions increase by annealing. These results can be naturally
understood if there are a significant number of vacant $T_{d}$ sites near the
interface of the as-grown $\text{CoFe}_{2}\text{O}_{4}$ thin films and that,
after annealing, the vacancies are filled by Fe ions and possibly Co ions,
too, diffused from the $O_{h}$ sites (for details, see Section IV). We note
that the inversion parameter y is defined based on the ideal spinel structure
($[\text{Co}_{1-y}\text{Fe}_{y}]_{T_{d}}[\text{Fe}_{2-y}\text{Co}_{y}]_{O_{h}}\text{O}_{4}$)
and the vacant sites are not taken into account. The lowering of y may reflect
not only the replacement of the Fe ion by the Co ion but also increase of the
vacancies at the $T_{d}$ and/or $O_{h}$ sites.

###  III.2 $\text{NiFe}_{2}\text{O}_{4}$ thin films

Figure 6:  Fe L2,3-edge XAS and XMCD spectra of the as-grown and annealed
$\text{NiFe}_{2}\text{O}_{4}$ thin films with various thicknesses
$d=\text{1.7, 3.5, and 5.2 nm}$. (a), (b) Fe $L_{2,3}$-edge XAS and XMCD
spectra. (c), (d) Magnified views of the Fe $L_{3}$-edge XMCD spectra
normalized to the height of peak c. The changes in the spectra are similar to
those of $\text{CoFe}_{2}\text{O}_{4}$ (Fig. 2).

Figure 6 shows the Fe $L_{2,3}$-edge XAS and XMCD spectra of the as-grown and
annealed $\text{NiFe}_{2}\text{O}_{4}$ thin films with thicknesses
$d=1.7,3.5,\text{ and }5.2$ nm. Figure 7 shows the Ni $L_{2,3}$-edge XAS and
XMCD spectra of the $\text{NiFe}_{2}\text{O}_{4}$ films. The XMCD intensity of
the 1.7 nm-thick film is extremely low compared to the others, indicating the
dominant effect of magnetically dead layers. After annealing, the XMCD
intensities increase for all the films. This clearly indicates that the
magnetic order in the dead layers was significantly recovered by annealing
also in the $\text{NiFe}_{2}\text{O}_{4}$ thin films.

Figure 7:  Ni L2,3-edge XAS and XMCD spectra of the as-grown and annealed
$\text{NiFe}_{2}\text{O}_{4}$ thin films with various thicknesses
$d=\text{1.7, 3.5, and 5.2 nm}$. The spectra of the as-grown films (blue
lines) are adopted from Ref. 19. (a), (b) Ni $L_{2,3}$-edge XAS and XMCD
spectra. (c), (d) Magnified views of the Ni $L_{3}$-edge XMCD spectra
normalized to the height of peak f for the as-grown and annealed films. The
normalized XMCD spectra of the 5.2-nm-thick as-grown film is also shown by a
dashed black curve for comparison. In contrast to the Co $L_{2,3}$-edge XMCD
spectra shown in Fig. 5, spectral line shapes are not changed by annealing,
reflecting the high $O_{h}$-site selectivity of the Ni2+ ion.

To see the effect of annealing on the cation distribution, we compare the line
shapes of normalized XMCD spectra at the Fe and Ni $L_{3}$ edges. Figures 6(c)
and 6(d) show magnified views of the Fe $L_{3}$-edges XMCD spectra normalized
to the height of peak c. The spectral line shapes show that the positive peak
b of the 1.7 nm-thick film significantly increases after annealing, indicating
the increase of Fe${}^{3+}(T_{d})$ ions. On the other hand, the Ni-edge XMCD
spectral line shapes, shown in Figs. 7(c) and 7(d), do not show any visible
changes depending on the thickness and annealing. The XMCD spectra of the 5.2
nm-thick as-grown film are shown in Fig. 7(d) as a dashed black curve for
comparison. One can clearly see that the spectral line shape remains the same
after annealing. Because the Ni ion takes only the Ni${}^{2+}(O_{h})$ state,
as reported in as-grown films [19], the unchanged spectral line shapes by
annealing indicate that the Ni ions did not move to the $T_{d}$ sites even
after annealing, confirming the high $O_{h}$-site selectivity of the Ni2+ ion.

Figure 8:  Comparison between the experimental and calculated Fe
$L_{2,3}$-edge XAS and XMCD spectra of the as-grown and annealed
$\text{NiFe}_{2}\text{O}_{4}$ thin films.

In order to quantitatively analyze the spectral line shapes of
$\text{NiFe}_{2}\text{O}_{4}$, as in the case of
$\text{CoFe}_{2}\text{O}_{4}$, we fitted the measured Fe $L_{2,3}$-edge XAS
and XMCD spectra using the weighted sum of the three calculated spectra of
Fe${}^{2+}(O_{h})$, Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ ions. The
parameters for the CI cluster model were the same as those for
$\text{CoFe}_{2}\text{O}_{4}$ listed in Table 1. Figure 8 shows the
experimental and calculated Fe $L_{2,3}$-edge XAS and XMCD spectra. The
experimental spectra are well reproduced by the calculation. Using the weights
obtained from the fitting, one can estimate the relative amount of the
Fe${}^{2+}(O_{h})$, Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ ions in the as-
grown and annealed $\text{NiFe}_{2}\text{O}_{4}$ thin films as shown in Fig.
9(a).

Figure 9:  Thickness dependences of the Fe-ion distribution and the
$L_{3}$-edge XMCD intensities for the as-grown and annealed
$\text{NiFe}_{2}\text{O}_{4}$ thin films. (a) Relative amounts of the
Fe${}^{2+}(O_{h})$, Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ ions in the as-
grown and annealed $\text{NiFe}_{2}\text{O}_{4}$ thin films as functions of
thickness. The inversion parameter y is also plotted on the right axis. Dashed
lines with open circles and solid lines with open triangles indicate as-grown
and annealed samples, respectively. (b) Fe $L_{3}$-edge XMCD intensity of the
as-grown and annealed films. The increase of y by annealing ($\Delta y$) and
the increase ratio of XMCD intensities (R) are plotted on the right axes.

One can see that, for the 1.7-nm-thick film, the ratio of the
Fe${}^{3+}(T_{d})$ ions increases with post-deposition annealing, while that
of the Fe3+($O_{h}$) ions decreases. Thus, the inversion parameter y of the
as-grown 1.7 nm-thick film is low compared to those of the thicker films,
consistent with the XMCD result. The low y value of the 1.7 nm-thick film
significantly increases to 0.96 after annealing (red dashed line). This means
that NiFe2O4 thin films with almost no antisite defects are achieved after
annealing. Figure 9(b) shows the Fe $L_{3}$-edge XMCD intensities of the as-
grown and annealed films as functions of film thickness. The increasing ratio
of the Fe $L_{3}$-edge XMCD intensity R and the increase of the inversion
parameter $\Delta y$ are also shown for comparison. Analogously to
$\text{CoFe}_{2}\text{O}_{4}$ shown in Fig. 4(b), the similar thickness
dependences between $R$ and $\Delta y$ suggest that the recovery of the
magnetically dead layer is achieved by the redistribution of cations.

The ratio of Fe${}^{3+}(T_{d})$ to Fe${}^{3+}(O_{h})$ was about $\frac{2}{3}$
in the 1.7 nm-thick film before annealing. Considering that the Ni ions are
located only at the $O_{h}$ sites, one can estimate from this ratio that about
$\frac{1}{3}$ of the $T_{d}$ sites should be vacant in the as-grown 1.7 nm-
thick film. As we shall discuss below, these vacancies at the $T_{d}$ sites
should have detrimental effects on the ferrimagnetic order in the spinel
structure.

##  IV Discussion

We discuss the role of the cation vacant sites in the magnetically dead
layers. As introduced above, based on the studies on Fe3O4 [12, 13, 16], it is
believed that the magnetically dead layers in the spinel ferrites are caused
by antiphase boundaries. However, as shown above, we find that there are a
significant number of vacant $T_{d}$ sites near the interface, and the
transfer of the cations from the $O_{h}$ to $T_{d}$ sites caused by annealing
significantly recovers the ferrimagnetic order. The antiferromagnetic
superexchange interaction between the $T_{d}$-site and $O_{h}$-site cations is
the origin of the ferrimagnetic order in the spinel ferrites. Therefore, the
lack of superexchange interaction due to the vacancies should significantly
suppress the ferrimagnetism of magnetically dead layers. Here, we focus on how
the vacant $T_{d}$ sites affect the spin structures in the magnetically dead
layers.

Figure 10:  Schematic pictures of exchange interaction between the $T_{d}$\-
and $O_{h}$-site ions. The $T_{d}$ and $O_{h}$ sites are labeled $A$ and $B$,
respectively.  Table 2:  Exchange interaction between transition metal ions in
$\textit{M}\text{Fe}_{2}\text{O}_{4}$ (M=Fe, Co) in units of K reproduced from
Ref. 33. Suffixes $A$ and $B$ denotes the $T_{d}$ and $O_{h}$ sites,
respectively. Note that the listed values include both the superexchange and
direct exchange interactions. Positive and negative values correspond to
ferromagnetic and antiferromagnetic interactions, respectively.

|  |  M = Co |  M = Ni
---|---|---|---
$J_{{AB}}$ |  $\text{Fe}^{3+}$-$\textit{M}^{2+}$ | $-22.7$ | $-27.4$
$\text{Fe}^{3+}$-$\text{Fe}^{3+}$ | $-26$ | $-30.7$
$J_{{BB}}$ |  $\textit{M}^{2+}$-$\textit{M}^{2+}$ | $46.9$ | $30.0$
$\text{Fe}^{3+}$-$\textit{M}^{2+}$ | $-18.5$ | $-2.7$
$\text{Fe}^{3+}$-$\text{Fe}^{3+}$ | $-7.5$ | $-5.4$

Figure 10(a) shows the exchange interaction between the cations at the $O_{h}$
and $T_{d}$ sites of the spinel structure. The values of the exchange
interaction in $\textit{M}\text{Fe}_{2}\text{O}_{4}$ (M=Co, Ni) deduced from
magnetization measurements [33], which are denoted as $J_{AB}$ and $J_{BB}$ in
Fig. 10, are listed in Table 2 in units of K. We note that suffix $A$ ($B$)
denotes the $T_{d}$ ($O_{h}$) sites, and the ratio between $M^{2+}(O_{h})$,
Fe${}^{3+}(T_{d})$, and Fe${}^{3+}(O_{h})$ is $1:1:1$ in an ideal inverse
spinel structure. The large negative values of $J_{AB}$ demonstrate the strong
contributions of the $T_{d}$ site to the ferrimagnetic order. As for the
$O_{h}$-$O_{h}$ interaction, $J_{BB}$, one can see large positive values for
$M^{2+}$-$M^{2+}$ interaction, and negative ones for
$\text{Fe}^{3+}$-$\text{Fe}^{3+}$ and $\text{Fe}^{3+}$-$M^{2+}$ interactions.
On average, $O_{h}$-$O_{h}$ interactions are weakly ferromagnetic ($+0.6$ K
for $\text{CoFe}_{2}\text{O}_{4}$ and $+4.8$ K for
$\text{NiFe}_{2}\text{O}_{4}$) 111 The number of $\text{Fe}^{3+}$-$M^{2+}$
exchange pairs is twice as much as that of $M^{2+}$-$M^{2+}$ and
$\text{Fe}^{3+}$-$\text{Fe}^{3+}$ exchange pairs. Therefore, the weighted
average value is
$\frac{1}{4}(J_{BB}^{M^{2+}-M^{2+}}+2J_{BB}^{\text{Fe}^{3+}-M^{2+}}+J_{BB}^{\text{Fe}^{3+}-\text{Fe}^{3+}})$..
We note that the number of the $O_{h}$-$T_{d}$ exchange pairs in an ideal
spinel structure are 2.3 times larger than that of the $O_{h}$-$O_{h}$ ones
[13]. If $\frac{1}{3}$ of the $T_{d}$ site is vacant as shown in Fig. 10(b),
$J_{AB}$ is weaken to $\frac{2}{3}$ of the ideal value, degrading the
ferrimagnetic order. After annealing, some $O_{h}$-site ions move to $T_{d}$
sites. For simplicity, assuming that the vacancies at the $T_{d}$ sites are
equally redistributed to all the $T_{d}$ and $O_{h}$ sites, as shown in Fig.
10(c), antiferromagnetic $J_{AB}$ is recovered to $\frac{8}{9}$ of ideal
value. Since the value and number of exchange interactions of $J_{AB}$ are
larger than that of $J_{BB}$, ferrimagnetic order should be stabilized.
Although the actual numbers of the vacancies are still unclear, it can be
concluded that vacancies at the $O_{h}$ sites still preserve the ferrimagnetic
order, whereas vacancies at the $T_{d}$ sites significantly degrade the
ferrimagnetic order. We should note that the above discussion ignores the
effects of charge unbalance caused by the cation vacancies, that is, the
effects of cation valence changes and possible oxygen vacancies. We speculate
that there are oxygen vacancies [35] and/or that extra $O_{h}$ sites [17] near
the interface.

##  V Conclusion

We have studied the electronic and magnetic states of the as-grown and
annealed $\text{CoFe}_{2}\text{O}_{4}$(111) and
$\text{NiFe}_{2}\text{O}_{4}$(111) thin films with various thicknesses grown
on Si substrates with the $\gamma$-$\text{Al}_{2}\text{O}_{3}$ buffer layers.
By analyzing the spectral line shapes using the CI cluster-model calculation,
we have estimated the valences and site occupancies of Fe ions. We confirmed
through the significant reduction of the XMCD intensities with decreasing film
thickness that there are magnetically dead layers near the interface with
$\gamma$-Al2O3. As reported in the previous works [18, 19], the reduction of
the inversion parameters was observed in the magnetically dead layers. By
annealing the films for 48 hours in air to redistribute the cations, the XMCD
intensity and inversion parameters significantly increase particularly in the
thinner films. Furthermore, from comparison between the results on the as-
grown and annealed films, it is suggested that there are a significant number
of vacant $T_{d}$ sites in the magnetically dead layers. After the post-
deposition annealing, the vacancies at the $T_{d}$ sites are partially
occupied by ions diffused from the $O_{h}$ sites. Thus, we conclude that
vacant $T_{d}$ sites play an important role in the formation of the
magnetically dead layers in spinel ferrites.

## Acknowledgments

We would like to thank Ryosho Nakane for valuable advice and enlightening
discussion throughout the present work, Kenta Amemiya and Masako Suzuki-
Sakamaki for valuable technical support at KEK-PF BL-16A. This work was
supported by KAKENHI grants from JSPS (JP15H02109, JP26289086, JP15K17696,
JP19K03741, JP20K14416, JP20H02199, and JP22K03535). The experiment was done
under the approval of the Photon Factory Program Advisory Committee (Proposal
No. 2016S2-005). Y. K. W. and Z. C. acknowledges financial support from
Materials Education Program for the Futures leaders in Research, Industry and
Technology (MERIT). Y. W. acknowledges financial support from Advanced Leading
Graduate Course for Photon Science (ALPS). Y. K. W. also acknowledge support
from the JSPS Research Fellowship Program for Young Scientists.

## References

  * Wolf  _et al._ [2001] S. A. Wolf, D. D. Awschalom, R. A. Buhrman, J. M. Daughton, S. vonar Molna, M. L. Roukes, A. Y. Chtchelkanova, and D. M. Treger, Spintronics: A Spin-Based Electronics Vision for the Future, Science 294, 1488 (2001).
  * Szotek  _et al._ [2006] Z. Szotek, W. M. Temmerman, D. Ködderitzsch, A. Svane, L. Petit, and H. Winter, Electronic structures of normal and inverse spinel ferrites from first principles, Phys. Rev. B 74, 174431 (2006).
  * Suzuki  _et al._ [1996] Y. Suzuki, R. B. van Dover, E. M. Gyorgy, J. M. Phillips, V. Korenivski, D. J. Werder, C. H. Chen, R. J. Cava, J. J. Krajewski, W. F. Peck, and K. B. Do, Structure and magnetic properties of epitaxial spinel ferrite thin films, Appl. Phys. Lett. 68, 714 (1996).
  * Ramos  _et al._ [2007] A. V. Ramos, M.-J. Guittet, J.-B. Moussy, R. Mattana, C. Deranlot, F. Petroff, and C. Gatel, Room temperature spin filtering in epitaxial cobalt-ferrite tunnel barriers, Appl. Phys. Lett. 91, 122107 (2007).
  * Sawatzky  _et al._ [1968] G. A. Sawatzky, F. van der Woude, and A. H. Morrish, Cation distributions in octahedral and tetrahedral sites of the ferrimagnetic spinel $\text{CoFe}_{2}\text{O}_{4}$, J. Appl. Phys. 39, 1204 (1968).
  * Matzen  _et al._ [2014] S. Matzen, J.-B. Moussy, P. Wei, C. Gatel, J. C. Cezar, M. A. Arrio, P. Sainctavit, and J. S. Moodera, Structure, magnetic ordering, and spin filtering efficiency of NiFe2O4 (111) ultrathin films, Appl. Phys. Lett. 104, 182404 (2014).
  * Matzen  _et al._ [2012] S. Matzen, J. B. Moussy, R. Mattana, K. Bouzehouane, C. Deranlot, and F. Petroff, Nanomagnetism of cobalt ferrite-based spin filters probed by spin-polarized tunneling, Appl. Phys. Lett. 101, 042409 (2012).
  * Moodera  _et al._ [2007] J. S. Moodera, T. S. Santos, and T. Nagahama, The phenomena of spin-filter tunnelling, J. Phys.: Condens. Matter 19, 165202 (2007).
  * Moussy [2013] J.-B. Moussy, From epitaxial growth of ferrite thin films to spin-polarized tunnelling, J. Phys. D. Appl. Phys. 46, 143001 (2013).
  * Lüders  _et al._ [2006] U. Lüders, M. Bibes, K. Bouzehouane, E. Jacquet, J.-P. Contour, S. Fusil, J.-F. Bobo, J. Fontcuberta, A. Barthélémy, and A. Fert, Spin filtering through ferrimagnetic NiFe2O4 tunnel barriers, Appl. Phys. Lett. 88, 082505 (2006).
  * Takahashi  _et al._ [2010] Y. K. Takahashi, S. Kasai, T. Furubayashi, S. Mitani, K. Inomata, and K. Hono, High spin-filter efficiency in a Co ferrite fabricated by a thermal oxidation, Appl. Phys. Lett. 96, 072512 (2010).
  * Margulies  _et al._ [1996] D. T. Margulies, F. T. Parker, F. E. Spada, R. S. Goldman, J. Li, R. Sinclair, and A. E. Berkowitz, Anomalous moment and anisotropy behavior in Fe3O4 films, Phys. Rev. B 53, 9175 (1996).
  * Margulies  _et al._ [1997] D. T. Margulies, F. T. Parker, M. L. Rudee, F. E. Spada, J. N. Chapman, P. R. Aitchison, and a. E. Berkowitz, Origin of the anomalous magnetic behaviour in single crystal Fe3O4 films, Phys. Rev. Lett. 79, 5162 (1997).
  * Voogt  _et al._ [1998] F. C. Voogt, T. T. M. Palstra, L. Niesen, O. C. Rogojanu, M. A. James, and T. Hibma, Superparamagnetic behavior of structural domains in epitaxial ultrathin magnetite films, Phys. Rev. B 57, R8107 (1998).
  * Eerenstein  _et al._ [2002] W. Eerenstein, T. T. M. Palstra, T. Hibma, and S. Celotto, Origin of the increased resistivity in epitaxial ${\mathrm{Fe}}_{3}{\mathrm{O}}_{4}$ films, Phys. Rev. B 66, 201101 (2002).
  * Moussy  _et al._ [2004] J.-B. Moussy, S. Gota, A. Bataille, M.-J. Guittet, M. Gautier-Soyer, F. Delille, B. Dieny, F. Ott, T. Doan, P. Warin, P. Bayle-Guillemaud, C. Gatel, and E. Snoeck, Thickness dependence of anomalous magnetic behavior in epitaxial Fe3O4(111) thin films: Effect of density of antiphase boundaries, Phys. Rev. B 70, 174448 (2004).
  * Chang  _et al._ [2016] C. F. Chang, Z. Hu, S. Klein, X. H. Liu, R. Sutarto, A. Tanaka, J. C. Cezar, N. B. Brookes, H.-J. Lin, H. H. Hsieh, C. T. Chen, A. D. Rata, and L. H. Tjeng, Dynamic Atomic Reconstruction: How Fe3O4 Thin Films Evade Polar Catastrophe for Epitaxy, Phys. Rev. X 6, 041011 (2016).
  * Wakabayashi  _et al._ [2017] Y. K. Wakabayashi, Y. Nonaka, Y. Takeda, S. Sakamoto, K. Ikeda, Z. Chi, G. Shibata, A. Tanaka, Y. Saitoh, H. Yamagami, M. Tanaka, A. Fujimori, and R. Nakane, Electronic structure and magnetic properties of magnetically dead layers in epitaxial CoFe2O${}_{4}/$Al2O${}_{3}/$Si(111) films studied by x-ray magnetic circular dichroism, Phys. Rev. B 96, 104410 (2017).
  * Wakabayashi  _et al._ [2018] Y. K. Wakabayashi, Y. Nonaka, Y. Takeda, S. Sakamoto, K. Ikeda, Z. Chi, G. Shibata, A. Tanaka, Y. Saitoh, H. Yamagami, M. Tanaka, A. Fujimori, and R. Nakane, Cation distribution and magnetic properties in ultrathin $(\mathrm{N}{\mathrm{i}}_{1-x}\mathrm{C}{\mathrm{o}}_{x})\mathrm{F}{\mathrm{e}}_{2}{\mathrm{O}}_{4}\phantom{\rule{4.0pt}{0.0pt}}(x=0-1)$ layers on Si(111) studied by soft x-ray magnetic circular dichr, Phys. Rev. Mater. 2, 104416 (2018).
  * Nonaka  _et al._ [2021] Y. Nonaka, Y. K. Wakabayashi, G. Shibata, S. Sakamoto, K. Ikeda, Z. Chi, Y. Wan, M. Suzuki, T. Koide, M. Tanaka, R. Nakane, and A. Fujimori, Reduced magnetocrystalline anisotropy of CoFe2O4 thin films studied by angle-dependent x-ray magnetic circular dichroism, AIP Adv. 11, 85317 (2021).
  * Bachelet  _et al._ [2014] R. Bachelet, P. de Coux, B. Warot-Fonrose, V. Skumryev, G. Niu, B. Vilquin, G. Saint-Girons, and F. Sánchez, Functional spinel oxide heterostructures on silicon, CrystEngComm 16, 10741 (2014).
  * Hu  _et al._ [2000] G. Hu, J. Choi, C. Eom, V. Harris, and Y. Suzuki, Structural tuning of the magnetic behavior in spinel-structure ferrite thin films, Phys. Rev. B 62, R779 (2000).
  * Deal and Grove [1965] B. E. Deal and A. S. Grove, General Relationship for the Thermal Oxidation of Silicon, J. Appl. Phys. 36, 3770 (1965).
  * Thole  _et al._ [1985] B. T. Thole, G. van der Laan, J. C. Fuggle, G. A. Sawatzky, R. C. Karnatak, and J.-M. Esteva, 3$d$ x-ray-absorption lines and the $3d^{9}4f^{n+1}$ multiplets of the lanthanides, Phys. Rev. B 32, 5107 (1985).
  * Frazer  _et al._ [2003] B. H. Frazer, B. Gilbert, B. R. Sonderegger, and G. De Stasio, The probing depth of total electron yield in the sub-keV range: TEY-XAS and X-PEEM, Surf. Sci. 537, 161 (2003).
  * Bocquet  _et al._ [1992] A. E. Bocquet, T. Mizokawa, T. Saitoh, H. Namatame, and A. Fujimori, Electronic structure of 3d-transition-metal compounds by analysis of the 2p core-level photoemission spectra, Phys. Rev. B 46, 3771 (1992).
  * Tanaka and Jo [1994] A. Tanaka and T. Jo, Resonant 3d, 3p and 3s Photoemission in Transition Metal Oxides Predicted at 2p Threshold, J. Phys. Soc. Jpn 63, 2788 (1994).
  * Chen  _et al._ [2004] J. Chen, D. J. Huang, A. Tanaka, C. F. Chang, S. C. Chung, W. B. Wu, and C. T. Chen, Magnetic circular dichroism in Fe 2p resonant photoemission of magnetite, Phys. Rev. B 69, 085107 (2004).
  * Hüfner [2003] S. Hüfner,  _Photoelectron Spectroscopy_, 3rd ed., Advanced Texts in Physics (Springer Berlin Heidelberg, Berlin, Heidelberg, 2003).
  * Harrison [1980] W. A. Harrison,  _Electronic Structure and the Properties of Solids_ , 3rd ed. (Freeman, San Francisco, 1980).
  * Matzen  _et al._ [2011] S. Matzen, J. B. Moussy, R. Mattana, F. Petroff, C. Gatel, B. Warot-Fonrose, J. C. Cezar, A. Barbier, M. A. Arrio, and P. Sainctavit, Restoration of bulk magnetic properties by strain engineering in epitaxial CoFe2O4 (001) ultrathin films, Appl. Phys. Lett. 99, 052514 (2011).
  * Takaobushi  _et al._ [2007] J. Takaobushi, M. Ishikawa, S. Ueda, E. Ikenaga, J.-J. Kim, M. Kobata, Y. Takeda, Y. Saitoh, M. Yabashi, Y. Nishino, D. Miwa, K. Tamasaku, T. Ishikawa, I. Satoh, H. Tanaka, K. Kobayashi, and T. Kawai, Electronic structures of Fe3-xMxO4 (M=Mn, Zn) spinel oxide thin films investigated by x-ray photoemission spectroscopy and x-ray magnetic circular dichroism, Phys. Rev. B 76, 205108 (2007).
  * Srivastava  _et al._ [1979] C. M. Srivastava, G. Srinivasan, and N. G. Nanadikar, Exchange constants in spinel ferrites, Phys. Rev. B 19, 499 (1979).
  * Note [1] The number of $\REV@text{Fe}^{3+}$-$M^{2+}$ exchange pairs is twice as much as that of $M^{2+}$-$M^{2+}$ and $\REV@text{Fe}^{3+}$-$\REV@text{Fe}^{3+}$ exchange pairs. Therefore, the weighted average value is $\frac{1}{4}(J_{BB}^{M^{2+}-M^{2+}}+2J_{BB}^{\REV@text{Fe}^{3+}-M^{2+}}+J_{BB}^{\REV@text{Fe}^{3+}-\REV@text{Fe}^{3+}})$.
  * Jaffari  _et al._ [2012] G. H. Jaffari, A. K. Rumaiz, J. C. Woicik, and S. I. Shah, Influence of oxygen vacancies on the electronic structure and magnetic properties of NiFe2O4 thin films, J. Appl. Phys. 111, 93906 (2012).