# Distilling Privileged Multimodal Information for Expression
Recognition using Optimal Transport

Muhammad Haseeb Aslam1 , Muhammad Osama Zeeshan1, Soufiane Belharbi1,
Marco Pedersoli1, Alessandro Koerich1, Simon Bacon2 and Eric Granger1
1 LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada 2 Dept. of Health,
Kinesiology $\&$ Applied Physiology, Concordia University, Montreal, Canada T

###### Abstract

Multimodal affect recognition models have reached remarkable performance in
the lab environment due to their ability to model complementary and redundant
semantic information. However, these models struggle in the wild, mainly
because of the unavailability or quality of modalities used for training. In
practice, only a subset of the training-time modalities may be available at
test time. Learning with privileged information (PI) enables deep learning
models (DL) to exploit data from additional modalities only available during
training. State-of-the-art knowledge distillation (KD) methods have been
proposed to distill multiple teacher models (each trained on a modality) to a
common student model. These privileged KD methods typically utilize point-to-
point matching and have no explicit mechanism to capture the structural
information in the teacher representation space formed by introducing the
privileged modality. We argue that encoding this same structure in the student
space may lead to enhanced student performance. This paper introduces a new
structural KD mechanism based on optimal transport (OT), where entropy-
regularized OT distills the structural dark knowledge. Privileged KD with OT
(PKDOT) method captures the local structures in the multimodal teacher
representation by calculating a cosine similarity matrix and selects the top-k
anchors to allow for sparse OT solutions, resulting in a more stable
distillation process. Experiments 111https://github.com/haseebaslam95/PKDOT
were performed on two different problems: pain estimation on the Biovid
dataset (ordinal classification) and arousal-valance prediction on the
Affwild2 dataset (regression). Results show that the proposed method can
outperform state-of-the-art privileged KD methods on these problems. The
diversity of different modalities and fusion architectures indicates that the
proposed PKDOT method is modality and model-agnostic.

##  I Introduction

Multimodal expression recognition (MER) aims to capture and mimic the human-
like nature of emotions [1]. The complementarity and redundancy of the
multiple modalities typically allow all these MER systems to outperform their
unimodal counterparts [2]. Using multiple modalities works well in a lab-
controlled environment where all the modalities are available during testing
and inference. However, some modalities are difficult or expensive to obtain
in real-world deployment. Various strategies, like cross-attention [3] that
weigh the modalities dynamically, have been proposed in the literature to
mitigate these issues. However, even those methods are not effective where
some modalities are completely missing. Physiological data like
electrocardiogram (ECG) and electromyography (EMG) are widely studied in
affective computing, and various MER systems have been proposed to recognize
expressions. For instance, the physiological signals have been shown to
outperform the visual modality, especially in tasks like pain estimation [4].
However, these signals are not always available in some practical contexts.
Since they require specialized equipment and limit the mobility of the
subjects, in such cases, most methods will typically rely on prevalent
modalities -available at both design time and development- resulting in a less
effective system. Using additional information (privileged modalities) only at
the training time, may however, enhance system performance.

Figure 1: Overview of the proposed Privileged Knowledge Distillation with
Optimal Transport (PKDOT) method that captures the local structure in the
multimodal teacher representation. Teacher backbones process multimodal i.e.,
privileged (red arrow) and prevalent (black arrows) input data. With PKDOT,
teacher model feature space is matched with the student model through OT,
while the student model processes prevalent modalities.

Recently, the learning using privileged information (LUPI) paradigm has been
introduced for affective computing [5]. Privileged information in machine
learning (ML) is the information available to the model only at training time
but not at inference time [6]. LUPI methods have been shown to increase
performance for systems using only the prevalent modalities (available during
train and inference). Traditionally utilizing teacher-student networks, the
teacher network is an MER model trained with all the modalities, and this
model is then used to distill the knowledge to an unimodal model that only
inputs the prevalent modality [5, 7]. One way to distill this information to
the student is using the vanilla knowledge distillation (KD) method, which
computes the Kullback–Leibler (KL) divergence between the softened logits in
the teacher and student network. Other methods have proposed matching teacher
and student representations in the feature space using Mean Squared Error
(MSE) or cosine similarity. All of these methods use point-to-point matching
to distill the knowledge, and are therefore, unable to capture the local
structure formed in the teacher space. Local structure refers to a more fine-
grained knowledge representation, where the distance of each sample from all
the other samples in the training batch is calculated. We argue that this
structural information, formed in the teacher space through the interaction
between the prevalent and privileged modality data, should be captured to
mimic the performance in the student space.

In this paper, a Privileged Knowledge Distillation with Optimal Transport
(PKDOT) method is proposed that relies on a structural KD mechanism to
effectively capture and distill the structural dark knowledge based on OT for
the student (See Fig. 1). Similarity matrix-based privileged KD computes the
similarity over a training batch rather than flattening and computing a single
cosine similarity score. The cosine similarity matrix retains information
about the pairwise relationships between all batch samples. An encoder-decoder
based Transformation Network (T-Net), which is trained during the teacher
training stage is used to hallucinate the privileged modality features.
Entropy-regularized OT is then used to distill the structural dark knowledge.

The main contributions of this work are as follows: (1) A PKDOT method is
introduced that computes a cosine batch similarity matrix to distill knowledge
to the student network using entropy-regularized OT on selected anchor points
to introduce sparsity. T-Net is also introduced that hallucinates the
privileged modality at test time. (2) Extensive experiments were performed on
two challenging MER problems – pain estimation on the Biovid dataset and
arousal-valance prediction on the Affwild2 dataset. Our extensive experiments
with different combinations of teacher and student modalities and fusion
architectures show that the proposed model can outperform the SOA-privileged
KD methods and is modality and model-agnostic.

##  II Related Work

###  II-A Multimodal Affect Modeling

Multimodal affect modeling is the understanding and modeling of emotions
through jointly learning from different signals, including facial cues, audio,
and text. Ngiam et al. [8] proposed the seminal work in multimodal deep
learning, where visual and audio modalities were encoded separately.
Restricted Boltzmann Machines and autoencoders were then used to fuse this
information. Tzirakis et al. [9] proposed an audiovisual fusion model, where a
1D convolutional neural network (CNN) was used to extract features from the
audio modality, and a ResNet-50 CNN was used for visual feature extraction.
The two feature vectors were concatenated and fed into a recurrent network for
simultaneous temporal modeling and fusion. Rajasekhar et al. [3] proposed a
joint cross-attention mechanism for audio-visual fusion, where separated
backbones were trained for audio and video modalities. The audio was
transformed into spectrograms using the discrete Fourier transform (DFT) and
fed to a ResNet-18 CNN, and an I3D CNN was used to extract features from the
visual modality. Later, both the feature vectors were fed to a cross-attention
module to calculate attention weights. The raw features were then multiplied
with the attention weights to get attended features. This dynamically assigned
weights to these modalities to overcome the noisy modality problem. The higher
performance and robustness of the multimodal systems come at a cost, usually
in terms of time and computational complexity. Recent efforts in minimizing
the cost of multimodal systems have birthed concepts like attention bottleneck
in transformers [10]. These methods still fall short in cases where some
modalities are completely missing.

Pain estimation is another major affective computing problem with many
publicly available datasets, with a recent addition of pain estimation in
infants [11] and animals [12]. Many physiological and image-based approaches
for pain detection are proposed. Werner et al. [13] developed a pain-specific
feature set called facial activity descriptors. Dragomir et al. [14] proposed
a subject-independent deep learning approach with residual learning for pain
estimation from facial images. Morabit et al. [15] proposed a data-efficient
image transformer. Zhi et al. [16] proposed a sparse LSTM-based method to
solve the vanishing gradient problem. Phan et al. [17] proposed an attention-
based method to process the EDA and ECG signals for pain estimation in the
physiological domain. Multi-level context information was extracted using LSTM
and attention network and then fused at the decision level for final pain
prediction. Lu et al. [18] proposed a transformer and squeeze and excitation-
based network to process multiscale EDA signals. Specifically for pain
estimation, the physiological signal-based approaches have been shown to
outperform the facial image-based approaches. Zhu et al. [19], Kachele et al.
[20], and Werner et al. [21] have proposed methods to combine visual and
physiological modalities. The ultimate goal of this paper is to achieve
performance that approaches multimodal systems using only the prevalent
modality at test time.

###  II-B Learning Using Privileged Information

Vapnik and Vashist [6] introduced the concept of learning using privileged
information in ML. Additional information, only available at training time,
was used to learn more discriminative information, outperforming the
traditional ML paradigm of using the same information at training and testing.
Many applications, like person re-identification [22] and action recognition
[23], have since employed this concept to enhance performance or increase
robustness. Zhao et al. [24] proposed a privileged KD mechanism for online
action detection. Future frames from the video were used as privileged
information only at training time since the model can access future frames
during training; however, only historical frames are available in the real
world. To reduce the teacher-student gap, only partial hidden features of the
student model were updated through KL-divergence loss. The availability and
quality of the modalities are a concern in the multimodal ML paradigm. In the
case of RGB-D data, depth sensing is available for the training data, but in
the wild, only RGB is commonly available. Audio and visual modalities can be
partially missing due to user-initiated muting, transmission/recording errors,
occlusion, etc. Similarly, in the affective computing domain, some modalities,
e.g., ECG, EMG, and EDA, may be completely absent at test time.

###  II-C Knowledge Distillation and Optimal Transport

The seminal work in knowledge distillation was proposed by Hinton et al. [25]
for model compression in a teacher-student learning paradigm. The accurate yet
cumbersome teacher model is used as additional supervision to train a
lightweight student model. The ’dark knowledge’ comes from increasing the
temperature in the teacher’s softmax, making it less confident about one
particular class. These softened logits encompass more information than one-
hot encoding. Romero et al. [26] proposed hint learning, essentially
distilling from the model’s hidden layer instead of softened logits. Since
then, several works for feature-based knowledge distillation have been
proposed [27, 28]. Vanilla KD methods only consider individual samples and
solely rely on matching the output activations between teacher and student.
Park et al. [29] proposed RKD and argued that distilling the relational
knowledge among samples can significantly improve the student’s performance.
Angle-wise and distance-wise loss were incorporated to penalize the structural
differences between the teacher and student. Following that, this work also
takes into account the local structures of the teacher model and effectively
distills into the student model using OT.

OT is a well-established mathematical framework for calculating the optimal
cost of transforming one probability distribution into another [30]. OT has
been directly applied to ML problems where matching distributions are vital.
Other solutions of distribution matching, like maximum mean discrepancy (MMD)
and KL-divergence, suffer theoretical drawbacks [31]. MMD does not precisely
capture the distance between distributions due to its sensitivity to sample
size and outliers [32]. KL-divergence has been shown to fail in cases where
two distributions do not overlap, resulting in infinity [33]. Although an
expensive solution, OT provides a stable metric for matching distributions. To
overcome the statistical and computational limitations of the OT, Cuturi et
al. [34] proposed regularized OT. Many works have used OT of various
applications, including model compression [33], domain adaptation [35],
pedestrian detection [36], and neural architecture search [37]. To the best of
our knowledge, this is the first use of OT for privileged KD in the context of
expression recognition.

Figure 2: (Left) Conventional Privileged KD computes point-to-point distance
without considering local structure. (Right) The proposed PKDOT model captures
the local structure and matches teacher and student representations by
distilling the structural dark knowledge (adapted from [29]).

###  II-D Privileged KD for Expression Recognition

In recent times, expression recognition has seen a rise in privileged KD
methods as the trend has shifted from more in-lab scenarios to more in-the-
wild scenarios. These works mainly include using the privileged information as
an additional modality to train a superior multimodal model, which is then
used to distill to a student model that does not have access to the privileged
modality. Aslam et al. [7] proposed a model agnostic method for distilling
privileged information (PI). Cosine similarity-based loss was added to the
task loss with adaptive weighting to mitigate the negative transfer.
Makantasis et al. [5] proposed a similar valence and arousal prediction method
using an MSE loss function and categorical expression problem using a KL-
divergence loss function. Liu et al. [38] also proposed a KD-based privileged
DL mechanism in the physiological domain, where the teacher model was trained
using the electroencephalogram (EEG) and galvanic skin response (GSR) signals.
Later, this teacher model (EEG+GSR) was used to distill knowledge to the
student model (GSR only) using KL-divergence. Although they enhance the
student model’s performance, these methods can not capture the structural
knowledge in the multimodal teacher space. The local structures formed in the
teacher space due to the introduction of privileged modality are crucial to be
captured and distilled to the student to enhance performance. State-of-the-art
privileged KD methods rely on point-to-point matching. In this paper, we
propose a structural KD mechanism using OT. Fig. 2 contrasts point-to-point vs
structural KD.

##  III Proposed Approach

Figure 3: Illustration of the proposed OT-based structural KD mechanism with
the prevalent and privileged modality backbones and fusion. The teacher model
(top) is trained on both prevalent and privileged modalities. The student
model (bottom) only inputs the prevalent modality. It hallucinates the
features of the privileged modality, and generates student embeddings in the
multimodal space. Entropy-regularized OT is used to distill the structural
dark knowledge.

The Privileged Knowledge Distillation with Optimal Transport (PKDOT) method
aims to enhance the performance of the student model by closing the gap
between teacher and student representations. It relies on a cosine similarity
matrix computed in the teacher and student space to effectively capture the
fine-grained structural information. Entropy-regularized OT is then applied to
minimize the distance between the teacher and student. T-Net is also
introduced to hallucinate the privileged modality features. Further, we select
anchors based on dissimilarity to aid the OT to be more stable, and introduce
sparsity. Selecting anchor points instead of distilling from all samples is
motivated by the sparsity notion in OT, where fewer samples contribute
substantially. On the contrary, if all samples are included in the
distillation process, the model can be penalized based on less relevant
samples for the class boundaries. Figure 3 shows the architecture of the PKDOT
method. The rest of this section provides additional details on the structural
similarity matching and optimal transport in the PKDOT method.

###  III-A Structural Similarity Matching

Let $\Theta$ be a matrix of dimension $b\times m$ obtained from the teacher
network, where $b$ is the batch size and $m$ is the dimension of the feature
vector, and $\Theta^{\top}$ be its transpose of dimension $m\times b$, and
$\Phi$ be a matrix of dimension $b\times m$ obtained from the student network,
and $\Phi^{\top}$ be its transpose of dimension $m\times b$.

$C_{\Theta}=\Theta\cdot\Theta^{\top}\text{,}\quad
C_{\Phi}=\Phi\cdot\Phi^{\top}$ (1)

where $C_{\Theta}$ ($C_{\Phi}$) is a matrix of size $b\times b$ where each
element $C_{ij}$ represents the dot product of $i^{th}$ row of $\Theta$
($\Phi$) and $j^{th}$ column of $\Theta^{\top}$ ($\Phi^{\top}$) and can be
denoted as:

$C_{ij}^{\Theta}=\sum_{k}\Theta_{ik}.\Theta^{\top}_{kj}\text{,}\quad
C_{ij}^{\Phi}=\sum_{k}\Phi_{ik}.\Phi^{\top}_{kj}$ (2)

We normalize $C_{\Theta}$ and $C_{\Phi}$ to ensure that the similarity is not
affected by the length. A normalization vector is calculated as follows:

$N_{\Theta}=\sqrt{\sum_{k=1}^{n}(\Theta_{ik})^{2}}\text{,}\quad
N_{\Theta}^{\top}=\sqrt{\sum_{k=1}^{n}(\Theta^{\top}_{ik})^{2}}$ (3)

where $N_{\Theta}$ ($N_{\Phi}$) is the normalization vector of size $b\times
1$, calculated by taking the $\ell^{2}$-norm of each row. $N_{\Theta}$
($N_{\Phi}$) allows us to normalize the values so that the length of the
vectors does not dominate the cosine similarity measure. $N_{\Theta}^{\top}$
($N_{\Phi}^{\top}$) is also calculated using Eq. 3, with the exception that it
is of dimension $1\times b$.

$S_{\Theta}$ and $S_{\Phi}$ are the final similarity matrices of teacher and
student networks respectively and are obtained by element-wise division of
$C_{\Theta}$ with the dot product of $N{{}_{\Theta}}$ and $N_{\Theta}^{\top}$
and $C_{\Phi}$ by the dot product of $N_{\Phi}$ and $N_{\Phi}^{\top}$ as
follows:

$S_{\Theta}=\frac{C_{\Theta}}{N_{\Theta}\cdot N_{\Theta}^{\top}}\text{,}\quad
S_{\Phi}=\frac{C_{\Phi}}{N_{\Phi}\cdot N_{\Phi}^{\top}}$ (4)

###  III-B Optimal Transport (PKDOT)

After the cosine similarity matrices have been computed for the given batch,
the top-$k$ most dissimilar samples are selected based on the cosine
similarity, and a new similarity matrix of dimension $b\times k$ is obtained.
The same indexes are also selected in the student similarity matrix. The
structural dark knowledge is distilled from the teacher to the student using
entropy-regularized OT, defined by:

$W_{p}(\mu,\nu)=\int_{\chi\times\chi}\mathcal{O}(S_{\Theta i},S_{\Phi
i})d\pi(S_{\Theta i},S_{\Phi i}))+\epsilon H(\pi)$ (5)

where $S_{\Theta i}$ and $S_{\Phi i}$ refer to each row in the teacher and
student similarity matrices respectively, semantically representing the local
structures, and $\mu$ and $\nu$ are the marginal distributions. $\epsilon$ $>$
0 is a coefficient and $H(\pi)$ is the entropic regularization defined as:

$H(\pi)=log\left(\frac{d\pi}{d\mu\cdot d\nu}(S_{\Phi i},S_{\Theta i})\right)$
(6)

Overall, ${W_{p}}$ represents an approximation of the Wasserstein distance
between the teacher and student similarity matrices. This term is then
included as an additional regularization to the task loss to align the
intermediate representations.

The task loss, either concordance correlation coefficient (CCC) for regression
(Eq. 7) or classification (Eq. 8) loss, is calculated with the ground truth
($y$) as follows:

$\mathcal{L}_{GT}=1-\frac{2.\rho\cdot\sigma_{x}\cdot\sigma_{y}}{\sigma^{2}_{x}+\sigma^{2}_{y}+(\mu_{x}-\mu_{y})^{2}}$
(7)

$\mathcal{L}_{GT}=-\sum_{c=1}^{M}y_{o,c}\log(p_{o,c})$ (8)

The student network is then trained using:

$\mathcal{L}_{total}=\mathcal{L}_{GT}+\lambda\cdot{W_{p}}$ (9)

where, $L_{GT}$ is the task loss, $W_{p}$ is the OT loss and $\lambda$ is the
coefficient to weigh the importance of the OT loss.

##  IV Experimental Methodology

Figure 4: Illustration of different fusion mechanisms employed to obtain the
joint teacher representation: (a) feature concatenation, (b) joint cross
attention [3], (c) multimodal transformer.

###  IV-A Datasets and Evaluation Protocol

i) Biovid Heat Pain Database is a popular pain estimation database. The Part A
of the dataset contains facial videos, ECG, EMG, and GSR. Part B of the
dataset also contains facial EMG from the trapezius and corrugator muscles.
The dataset is annotated for discrete labels for pain intensities, where BL
refers to ’no pain’ and PA1-PA4 refers to increasing pain intensities. The
Biovid (A) dataset contains 8700 videos across 87 subjects, where each subject
has 100 videos, corresponding to 20 videos per class. Some studies report
results on 67 subjects and drop 20 subjects who do not exhibit any response to
the pain. We use the entire dataset and report our results on the 87 subjects.
The Biovid (B) dataset is recorded in a similar setting to Part A, with the
addition of facial EMG, and it has a total number of 86 subjects,
corresponding to 8600 videos. The dataset has no predefined train test split,
so most studies validate using cross-validation. Following that, we perform
5-fold cross-validation.

ii) Affwild2 is one of the largest datasets of in-the-wild videos gathered
from YouTube [39]. It contains 564 videos of varying length. The dataset has
annotations for three affective computing problems: categorical expression
recognition, action unit detection, and valence/arousal prediction. We
validate the proposed method on the valence/arousal estimation problem.
Affwild2 contains videos with huge diversification in terms of ethnicities,
genders, ages, capture conditions, etc. This makes Affwild2 a challenging
dataset in terms of generalization. The dataset has training, validation, and
test subsets with 351, 71, and 152 videos, respectively. This dataset is part
of the ABAW challenge, so the test set annotations are not publicly available.
Consequently, many studies report their results on the validation set.

###  IV-B Implementation Details

####  IV-B1 Biovid dataset

Teacher For the Biovid (B) dataset, the multimodal teacher model is a
multimodal transformer-based fusion model (Fig. 4c). The physiological
backbone is a ResNet-18 that takes spectrograms of size 67$\times$127 obtained
from EMG signals as input. The visual backbone is an R3D that takes cropped
and aligned facial frames of size 112$\times$112 as input. The batch size for
the visual backbone is 64, with a learning rate of 10-3 optimized using the
Adam optimizer. Both modalities output a feature vector of dimension 512.
These feature vectors are fed to two separate transformer encoders and then to
a multimodal transformer. K and V vectors are generated from one modality, and
Q is generated from another. The output of the cross-attended vectors is gated
using learnable weights. Finally, fully connected layers are added to generate
a final prediction. The multimodal transformer is optimized with a learning
rate of 10-4 and the Adam optimizer with a batch size of 64. For the Biovid
(A) dataset, we evaluate the proposed method with feature concatenation (Fig.
4a), a relatively simpler fusion method. Student: The visual backbone and
multimodal transformer model in the student network are the same as the
teacher. The physiological backbone is dropped, and the T-Net is added to
hallucinate the features of the privileged modality. The batch size for the
student was set to 128, with a learning rate of 10-4 and Adam optimizer and
the value for $\lambda$ is 0.4.

####  IV-B2 Affwild2 dataset

Teacher: The joint cross-attention model (Fig. 4b) proposed by Rajasekhar et
al. [40] is used. We use the cropped-aligned facial images that are provided
with the dataset. A 3D CNN is used to extract the features from the visual
modality. The batch size for the visual modality is 8, and the learning rate
used to optimize the parameter is 10-3 For the audio modality, the audio is
divided into multiple short segments corresponding to 256 frames in the visual
modality. Spectrograms of resolution 64$\times$107 pixels are obtained using
DFT. These spectrograms are fed to a ResNet-18 trained from scratch to extract
the audio features. The batch size is the same as the visual modality with a
learning rate of 10-3. For the fusion of the two modalities, the concatenated
vector of size 1024 is fed to the cross-attention module, which is optimized
using the Adam optimizer with a learning rate of 10-3 and a batch size of 64.
Student: The student network follows roughly the same implementation method as
the teacher network, except there is no backbone to process the privileged
modality. Instead, it is replaced with a T-Net, a simple encoder-decoder
trained during the teacher training phase. In the student training phase, the
T-Net is frozen and is used to hallucinate the features of the privileged
modality. The batch size is set to 128 for the student model and the value for
$\lambda$ is 0.4.

##  V Results and Discussion

###  V-A Comparison with the State-of-the-Art

Table I gives an overview of the experimental settings including different
modalities, fusion methods, and the distillation setting. Table II shows the
comparison of the proposed method with state-of-the-art privileged
distillation methods. We compare PKDOT with cosine similarity [7], MSE, and KL
[5]. It can be observed from the table that PKDOT outperforms point-to-point
distillation methods like KL-divergence, MSE, and cosine similarity. The
improvement for the BioVid dataset is considerable, partly because the visual
modality is weak in the pain estimation problem, because multiple subjects
show little to no variation for pain and no-pain videos. Transporting the PI,
i.e., EMG, from the multimodal network to the visual network greatly enhances
the performance. The improvement in the Affwild2 dataset is marginal because
the PI, i.e., audio modality, is weak on its own but can still improve the
visual-only network. This also points to the fact that the proposed network
can be used in stronger-enhancing-weaker (SEW) and weaker-enhancing-stronger
(WES) situations.

Dataset | Modalities | Property | Fusion | Distilled to
---|---|---|---|---
Affwild | Visual |  | -Strong
---
-Prevalent
Joint Cross Attention |  Visual (WES)
Audio |  | -Weak
---
-Privileged
Biovid (A) | Visual |  | -Weak
---
-Prevalent
Feature Concatenation |  Visual (SEW)
EDA |  | -Strong
---
-Privileged
Biovid (B) | Visual |  | -Strong
---
-Prevalent
Multimodal Transformer |  Visual (WES)
EMG |  | -Weak
---
-Privileged
|  |  |  |
TABLE I: Summary of diverse of modalities, their properties, fusion
mechanisms, and distillation settings: Stronger Enhancing Weaker (SEW) and
Weaker Enhancing Stronger (WES) Figure 5: Evolution of the similarity matrix
over training epochs.

Fig. 5 shows the evolution of the similarity matrix with training. Fig. 5a is
the similarity matrix for the pretrained teacher. It can be seen from the
image that the samples from the same class have high similarity, and the
samples from the other class have high dissimilarity. Fig. 5b is the
similarity matrix for the student before training, which shows a moderate
similarity with other samples, and the diagonal represents the high similarity
between the sample and itself. As the training progresses, we see the student
similarity matrix become closer and closer to that of the teacher. Fig. 5e
shows the similarity matrix of the student model fully trained with PKDOT.

Method | Performance
---|---
| Biovid (B) | Affwild
Aslam et al.[7] |  | Cosine Similarity
---
(IEEE CVPRw ’23)
75.40 |  | V: 0.37
---
A: 0.53
Makantasis et al.[5] |  | MSE
---
(IEEE TAC ’23)
N/A |  | V: 0.39
---
A: 0.53
Makantasis et al. [5] |  | KL
---
(IEEE TAC ’23)
75.80 | N/A
OPTI (Ours) |  | Optimal Transport
---
Structural KD
78.76 |  | V: 0.43
---
A: 0.56
N/A: Not applicable to the problem.
TABLE II: Comparison of the proposed and state-of-the-art privileged KD methods. Method | Visual Network | Valence | Arousal
---|---|---|---
Baseline [41] CVPRw’20 | ResNet-50 | 0.31 | 0.17
Zhang et al. [42] CVPRw’20 | SENet-50 | 0.28 | 0.34
He et al. [43] CVPRw’21 | MobileNet | 0.28 | 0.44
Nguyen et al. [44] CVPRw’22 | RegNet + GRU | 0.43 | 0.57
Geesung et al. [45] CVPRw’21 | ResNeXt + SENet | 0.51 | 0.48
Visual only (Lower bound) | I3D | 0.41 | 0.51
Student - PKDOT (Ours) | 0.43 | 0.56
| Multimodal (Visual + Audio)
---
(Upper bound)
0.67 | 0.59
|  |  |
TABLE III: Comparison of the proposed method with visual-only state-of-the-art
methods on the Affwild2 validation set.

Table III compares the student model with visual-only state-of-the-art on the
Affwild2 dataset. The multimodal upper bound in the Affwild2 dataset for
valence and arousal is 0.67 and 0.59, respectively. The visual-only network is
the lower bound with 0.41 valence and 0.51 arousal value. The PKDOT student
model shows an improvement of absolute 2% for the valence dimension and 3% for
the arousal dimension.

Table IV compares the PKDOT student model with visual-only methods on the
Biovid (A) dataset. Some studies report results on no-pain (BL1) and pain
(PA4), and some have included multi-class (MC), i.e., BL1 vs. PA1 vs. PA2 vs.
PA3 vs. PA4. We present and compare both settings. It can be seen from the
table that the proposed system gains an absolute 2.4% over the visual-only
lower bound.

Method | Visual Model | Accuracy
---|---|---
BL1 vs. PA4 | MC
Zhi et al. [16] ITAIC’19 | Sparse LSTM | 61.70 | 29.70
Morabit et al. [15] ISIVC’22 | ViT | 72.11 | NR
Werner et al. [46] ACIIW’17 | FAD + RF | 72.40 | 30.80
Patania et al. [47] SIGAPP’22 | GNN | 73.20 | NR
Gkikas et al. [48] EMBC’23 | ViT | 73.28 | 31.52
Dragomir et al. [49] EHB’20 | ResNet-18 | NR | 36.60
Visual only (Lower bound) | R3D | 72.10 | 30.38
Student - PKDOT (ours) | 74.55 | 33.65
| Multimodal (Visual + EDA)
---
(Upper bound)
83.50 | 36.40
NR: Not Reported.
TABLE IV: Comparison of the proposed system with visual-only state-of-the-art methods on the Biovid (A) dataset. Method | Visual Network | Accuracy
---|---|---
Kachele et al. [20] IWMCS’15 | Handcrafted + RF | 72.70
Visual Only (Lower bound) | R3D | 74.10
Student - PKDOT (Ours) | 78.76
| Multimodal (Visual + EMG)
---
(Upper bound)
81.30
|  |
TABLE V: Comparison of the proposed system with visual-only state-of-the-art
methods on the Biovid (B) dataset.

Table V compares the proposed method with methods in the literature for the
Biovid (B) dataset. It can be observed from the table that the proposed PKDOT
student model has gained an absolute 4.6% over the visual-only model.

Note that other methods in the literature may be tailored to the problem, with
sophisticated architecture changes or extensive pretraining, to achieve higher
performance. The goal of this work is not to push the state-of-the-art on
these problems but to exhibit that by using simple models and only with the
visual modality at inference, the performance can be enhanced using PKDOT.

###  V-B Ablations

####  V-B1 Batch Size and Anchors

Batch size is an essential parameter in the proposed mechanism since it
dictates how many samples are included in the similarity matrix at each
optimization step. We run PKDOT with batch sizes from 32 to 256; for each
experiment, we select 5, 15, and 30 anchors. Fig. 6 shows the evolution of
results according to batch size and number of anchors. Our experiments show
that increasing the batch size shows an improvement in the distillation
efficiency, the best results are obtained with the batch size 128 and 30
anchors.

Figure 6: Evolution of the performance of the proposed network with batch size
and number of anchors variation on the Biovid (B) dataset.

####  V-B2 OT Regularization

Entropy regularized OT has a regularization coefficient that is the trade-off
between the minimization of transportation cost and the uniformity of the OT.
We run experiments with different values of $\epsilon$ in Eq. 6.

Figure 7: Impact of the PKDOT regularization parameter $\epsilon$ on (a)
accuracy for the Biovid dataset and (b) valence/arousal for the Affwild2
dataset.

Fig. 7 shows the impact of the regularization parameter on the PKDOT student
performance. We run experiments with different values of $\epsilon$. Larger
values of $\epsilon$ put more emphasis on the regularization term, i.e., it
encourages the transport plan to be more sparse, where key samples contribute
substantially. Values closer to zero put more emphasis on minimizing the
transportation cost. In our experiments, $\epsilon$=0.1 gives the best trade-
off between the transport cost and uniformity.

Table I summarizes the different modalities, fusion methods, and the
distillation setting. The proposed method is validated with different fusion
architectures because the method used to obtain the multimodal teacher
representation also has a bearing on distillation efficiency. We test with
both, simple and complex fusion architectures. WES is when the privileged
modality is weaker, and the prevalent modality is stronger. SEW is when the
stronger modality is the privileged modality, and the student model only
inputs the weaker modality. From the experiments, we find that the proposed
method work better in WES settings. Although, there is more margin to enhance
the performance when the prevalent modality is weaker since there is a large
performance gap between the student and teacher performance. It is still
harder for the student model to mimic the teacher because the stronger
modality is unavailable to the student. In WES settings, the absent modality
is weaker in comparison, and the student still has access to the stronger
modality that the teacher model had used to learn the discriminative
information.

##  VI Conclusions

This paper presents a novel optimal transport-based structural KD mechanism to
transfer privileged information. Most privileged deep learning methods employ
point-to-point matching for KD, which cannot capture the local structures in
the teacher space. The proposed PKDOT method solves this by capturing the
local structures and applying entropy-regularized OT to translate the student
representation closer to the teacher representation. To further aid the OT
solution to be more effective in the distillation process, we select top-k
anchors and only calculate the OT between them. The proposed PKDOT method is
validated on two main MER problems: the pain classification and the
valence/arousal estimation regression problem. To further prove the
generalization capability of the proposed PKDOT method, we employ three
different fusion methods and a variety of modalities to form the teacher
representations, showing that the proposed method is model-agnostic and
modality-agnostic. The proposed method is able to significantly improve the
student performance over the state-of-the-art privileged KD methods in both
SEW and WES settings. Limitations: The proposed method only allows for
knowledge transfer after the modalities have been fused in the teacher
network. It would be beneficial to add modality-specific teachers in the
distillation process. Further, the model assumes that the top-k most
dissimilar anchors are the most effective and contribute the most to the
distillation process. Identifying the most effective anchor points through
learning is another research direction.

## References

  * [1] Suwon Shon, Tae-Hyun Oh, and James R. Glass.  Noise-tolerant audio-visual online person verification using an attention-based neural network fusion.  ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3995–3999, 2018.
  * [2] Amir Zadeh, Minghai Chen, Soujanya Poria, E. Cambria, and Louis-Philippe Morency.  Tensor fusion network for multimodal sentiment analysis.  In Conference on Empirical Methods in Natural Language Processing, 2017.
  * [3] R Gnana Praveen, Wheidima Carneiro de Melo, Nasib Ullah, Haseeb Aslam, Osama Zeeshan, Théo Denorme, Marco Pedersoli, Alessandro L. Koerich, Simon Bacon, Patrick Cardinal, and Eric Granger.  A joint cross-attention model for audio-visual fusion in dimensional emotion recognition.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2485–2494, 2022.
  * [4] Patrick Thiam and Friedhelm Schwenker.  Multi-modal data fusion for pain intensity assessment and classification.  In 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA), pages 1–6, 2017.
  * [5] Konstantinos Makantasis, Kosmas Pinitas, Antonios Liapis, and Georgios N. Yannakakis.  From the lab to the wild: Affect modeling via privileged information.  IEEE Transactions on Affective Computing, pages 1–13, 2023.
  * [6] Vladimir Vapnik and Akshay Vashist.  A new learning paradigm: Learning using privileged information.  Neural Networks, 22(5):544–557, 2009.  Advances in Neural Networks Research: IJCNN2009.
  * [7] Muhammad Haseeb Aslam, Muhammad Osama Zeeshan, Marco Pedersoli, Alessandro L. Koerich, Simon Bacon, and Eric Granger.  Privileged knowledge distillation for dimensional emotion recognition in the wild.  In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3338–3347, 2023.
  * [8] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng.  Multimodal deep learning.  In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, page 689–696, Madison, WI, USA, 2011. Omnipress.
  * [9] Panagiotis Tzirakis, George Trigeorgis, Mihalis Nicolaou, Björn Schuller, and Stefanos Zafeiriou.  End-to-end multimodal emotion recognition using deep neural networks.  IEEE Journal of Selected Topics in Signal Processing, PP, 04 2017.
  * [10] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14200–14213, 2021.
  * [11] Ghada Zamzmi, Rangachar Kasturi, Dmitry Goldgof, Ruicong Zhi, Terri Ashmeade, and Yu Sun.  A review of automated pain assessment in infants: Features, classification tasks, and databases.  IEEE Reviews in Biomedical Engineering, PP:1–1, 11 2017.
  * [12] Marcelo Feighelstein.  Explainable automated pain recognition in cats.  In Scientific reports, June 2023.
  * [13] Philipp Werner, Ayoub Al-Hamadi, Kerstin Limbrecht-Ecklundt, Steffen Walter, Sascha Gruss, and Harald C. Traue.  Automatic pain assessment with facial activity descriptors.  IEEE Transactions on Affective Computing, 8(3):286–299, 2017.
  * [14] Marilena-Cătălina Dragomir, Corneliu Florea, and Valentin Pupezescu.  Automatic subject independent pain intensity estimation using a deep learning approach.  In 2020 International Conference on e-Health and Bioengineering (EHB), pages 1–4, 2020.
  * [15] Safaa El Morabit and Atika Rivenq.  Pain detection from facial expressions based on transformers and distillation.  In 2022 11th International Symposium on Signal, Image, Video and Communications (ISIVC), pages 1–5, 2022.
  * [16] Ruicong Zhi and Ming Wan.  Dynamic facial expression feature learning based on sparse rnn.  In 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), pages 1373–1377, 2019.
  * [17] Kim Ngan Phan, Ngumimi Karen Iyortsuun, Sudarshan Pant, Hyung-Jeong Yang, and Soo-Hyung Kim.  Pain recognition with physiological signals using multi-level context information.  IEEE Access, 11:20114–20127, 2023.
  * [18] Zhenyuan Lu, Burcu Ozek, and Sagar Kamarthi.  Transformer encoder with multiscale deep learning for pain classification using physiological signals.  arXiv preprint arXiv:2303.06845, 2023.
  * [19] Ruicong Zhi, Caixia Zhou, Junwei Yu, Tingting Li, and Ghada Zamzmi.  Multimodal-based stream integrated neural networks for pain assessment.  IEICE Trans. Inf. Syst., 104-D:2184–2194, 2021.
  * [20] Markus Kächele, Philipp Werner, Ayoub Al-Hamadi, Günther Palm, Steffen Walter, and Friedhelm Schwenker.  Bio-visual fusion for person-independent recognition of pain intensity.  In International Workshop on Multiple Classifier Systems, 2015.
  * [21] Philipp Werner, Ayoub Al-Hamadi, Robert Niese, Steffen Walter, Sascha Gruss, and Harald C. Traue.  Automatic pain recognition from video and biomedical signals.  In 2014 22nd International Conference on Pattern Recognition, pages 4582–4587, 2014.
  * [22] Zhiyuan Shi and Tae-Kyun Kim.  Learning and refining of privileged information-based rnns for action recognition from depth sequences.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [23] Xun Yang, Meng Wang, and Dacheng Tao.  Person re-identification with metric learning using privileged information.  IEEE Transactions on Image Processing, 27(2):791–805, 2018.
  * [24] Peisen Zhao, Lingxi Xie, Jiajie Wang, Ya Zhang, and Qi Tian.  Progressive privileged knowledge distillation for online action detection.  Pattern Recognition, 129:108741, 09 2022.
  * [25] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  ArXiv, abs/1503.02531, 2015.
  * [26] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  CoRR, abs/1412.6550, 2014.
  * [27] Jangho Kim, Seonguk Park, and Nojun Kwak.  Paraphrasing complex network: Network compression via factor transfer.  ArXiv, abs/1802.04977, 2018.
  * [28] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1921–1930, 2019.
  * [29] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3962–3971, 2019.
  * [30] Cédric Villani.  Optimal transport: Old and new.  2008\.
  * [31] Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré.  Interpolating between optimal transport and mmd using sinkhorn divergences.  In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 2681–2690. PMLR, 16–18 Apr 2019.
  * [32] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  J. Mach. Learn. Res., 13(null):723–773, mar 2012.
  * [33] Suhas Lohit and Michael Jones.  Model compression using optimal transport.  In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3645–3654, 2022.
  * [34] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  In Neural Information Processing Systems, 2013.
  * [35] You-Wei Luo and Chuan-Xian Ren.  Mot: Masked optimal transport for partial domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3531–3540, June 2023.
  * [36] Xiaolin Song, Binghui Chen, Pengyu Li, Jun-Yan He, Biao Wang, Yifeng Geng, Xuansong Xie, and Honggang Zhang.  Optimal proposal learning for deployable end-to-end pedestrian detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3250–3260, June 2023.
  * [37] Jiechao Yang, Yong Liu, and Hongteng Xu.  Hotnas: Hierarchical optimal transport for neural architecture search.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11990–12000, June 2023.
  * [38] Yucheng Liu, Ziyu Jia, and Haichao Wang.  Emotionkd: A cross-modal knowledge distillation framework for emotion recognition based on physiological signals.  In Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, page 6122–6131, New York, NY, USA, 2023. Association for Computing Machinery.
  * [39] Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A. Nicolaou, A. Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou.  Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond.  International Journal of Computer Vision, 127:907–929, 2018.
  * [40] R Gnana Praveen, Patrick Cardinal, and Eric Granger.  Audio-visual fusion for emotion recognition in the valence-arousal space using joint cross-attention.  IEEE Transactions on Biometrics, Behavior, and Identity Science, pages 1–1, 2023.
  * [41] D. Kollias.  Abaw: Valence-arousal estimation, expression recognition, action unit detection and; multi-task learning challenges.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2327–2335, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.
  * [42] Zihang Zhang and Jianping Gu.  Facial affect recognition in the wild using multi-task learning convolutional network.  ArXiv, abs/2002.00606, 2020.
  * [43] Ruian He, Zhen Xing, Weimin Tan, and Bo Yan.  Feature pyramid network for multi-task affective analysis.  ArXiv, abs/2107.03670, 2021.
  * [44] Hong-Hai Nguyen, Van-Thong Huynh, and Soo-Hyung Kim.  An ensemble approach for facial behavior analysis in-the-wild video.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2511–2516, 2022.
  * [45] Geesung Oh, Euiseok Jeong, and Sejoon Lim.  Causal affect prediction model using a past facial image sequence.  In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 3543–3549, 2021.
  * [46] Philipp Werner, Ayoub Al-Hamadi, Kerstin Limbrecht-Ecklundt, Steffen Walter, Sascha Gruss, and Harald C. Traue.  Automatic pain assessment with facial activity descriptors.  IEEE Transactions on Affective Computing, 8(3):286–299, 2017.
  * [47] Sabrina Patania, Giuseppe Boccignone, Sathya Buršić, Alessandro D’Amelio, and Raffaella Lanzarotti.  Deep graph neural network for video-based facial pain expression assessment.  In Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pages 585–591, 2022.
  * [48] Stefanos Gkikas and Manolis Tsiknakis.  A full transformer-based framework for automatic pain estimation using videos.  volume 2023, 07 2023.
  * [49] Marilena-Cătălina Dragomir, Corneliu Florea, and Valentin Pupezescu.  Automatic subject independent pain intensity estimation using a deep learning approach.  In 2020 International Conference on e-Health and Bioengineering (EHB), pages 1–4, 2020.


# Improving magnetic-field resilience of NbTiN planar resonators using a hard-
mask fabrication technique

A. Bahr  M. Boselli  B. Huard  A. Bienfait  Ecole Normale Supérieure de Lyon,
CNRS, Laboratoire de Physique, F-69342 Lyon, France  audrey.bienfait@ens-
lyon.fr

###### Abstract

High-quality factor microwave resonators operating in a magnetic field are a
necessity for some quantum sensing applications and hybrid platforms. Losses
in microwave superconducting resonators can have several origins, including
microscopic defects, usually known as two-level-systems (TLS). Here, we
characterize the magnetic field response of NbTiN resonators patterned on
sapphire and observe clear absorption lines occurring at specific magnetic
fields. We identify the spin systems responsible for these features, including
a yet unreported spin with $g=1.85$ that we attribute to defects in the NbTiN
thin film. We develop mitigation strategies involving namely an aluminum etch
mask, resulting in maintaining quality factors above $Q>2\times 10^{5}$ in the
range $0\text{\,}\mathrm{T}0.3\text{\,}\mathrm{T}$.

††preprint: AIP/123-QED

Superconducting circuits are promising candidates for quantum information
processing (QIP), as well as high-precision sensing. Recently, the interest in
circuits able to operate under magnetic fields has increased to enable hybrid
QIP platformsBurkard  _et al._ (2020); Clerk  _et al._ (2020) or quantum
sensing applications such as magnetic resonance detectionSigillito  _et al._
(2014); Wang  _et al._ (2023). These applications require developing high-
quality factor superconducting resonators resilient to magnetic fields.
Detrimental effects for microwave superconducting resonators have been largely
studied McRae  _et al._ (2020), with evidence that obtaining high-quality
factors at zero magnetic field requires shielding from quasi-particlesBarends
_et al._ (2011), preventing vortex nucleationSong  _et al._ (2009), using low-
loss dielectric substratesWang  _et al._ (2015); Dial  _et al._ (2016), and
mitigating microscopic defectsMüller  _et al._ (2019). This last effect arises
from so-called two-level-systems (TLS) which can be saturated to recover a
higher quality factor at high powerBarends  _et al._ (2009). They can have
different origins, including spurious spin systemsMüller  _et al._ (2019).
When the spin Larmor frequency is tuned on resonance by the application of a
magnetic field, an increase in losses can be observed which allowed to
identify these spin systems partly as hydrogen and oxygen adsorbates on
sapphire substratesQuintana  _et al._ (2017); de Graaf  _et al._ (2017). In
addition to representing a source of losses when operating under magnetic
field, these spins are also believed to be responsible for frequency noise as
well as flux noise Barends  _et al._ (2009) limiting for example the
sensitivity of SQUID magnetometersAnton  _et al._ (2013); Sendelbach  _et al._
(2008) or the coherence of flux qubitsSank  _et al._ (2012); Quintana  _et
al._ (2017); Kumar  _et al._ (2016). Here, we evidence similar field-dependent
losses for superconducting resonators deposited on sapphire, and show that a
hard-mask nanofabrication technique mitigates the contribution of some of
these spurious spin systems.

Figure 1: (a) Fabrication steps for NbTiN superconducting resonators: 50-nm
films sputtered on EFG or HEM grown sapphire, coated by a resist or Al mask,
dry etched with CHF3 chemistry, and then cleaned and possibly annealed at 300
under N2 atmosphere. (b) Resonator geometries used in this work, nicknamed
circle (left) and microwire (right). The microwire drawing is cut along the
red lines for clarity. The blue arrow $B_{0}$ shows the applied in-plane
magnetic field. (c) Measured intrinsic quality factors at zero field,
depending on the fabrication technique, filled (open) markers indicate circle
(microwire) design, their colors indicate their $Q_{i}/Q_{c}$ ratio as given
by the color bar in inset. X-coordinate scattering is only for clarity. *Al
mask indicates the wafer was covered by photoresist during dicing, but that an
Al mask was used during etching.

TLS defects can be either electric dipoles or magnetic dipoles whose
interaction strength with the resonator field depends on their location Müller
_et al._ (2019), and range from weak to strong couplingLisenfeld  _et al._
(2016). These defects can be located in the substrate or at the surfaces of
the substrate and superconducting electrodes. Electric-TLS can be for instance
trapped charges, dangling bonds or tunneling atoms located either on the
substrate or in an oxide layer capping the superconductor. Many works have now
demonstrated that surface dielectric losses are far more detrimental than
dielectric bulk lossesWang  _et al._ (2015); Woods  _et al._ (2019), and shown
that the impact of these electric-TLS can be minimized by improving materials
and fabrication techniques as well as optimizing the design to reduce their
contribution Müller  _et al._ (2019); McRae  _et al._ (2020); Murray (2021).

Magnetic-TLS arise from nuclear and electronic spins lying at the surfaces or
in the bulk of the substrate, such as spin donors for siliconLederer and
Raskin (2003), or metal contaminants in sapphireFarr  _et al._ (2013). Their
contribution to the resonator intrinsic loss and flux noise is maximum when
their transitions are brought to resonance by the application of a magnetic
field. Several superconductors, such as NbKwon  _et al._ (2018), NbNde Graaf
_et al._ (2017); Yu  _et al._ (2021), NbTiRusso  _et al._ (2023),
NbTiNSamkharadze  _et al._ (2016), GrAlBorisov  _et al._ (2020) and YCBOGhirri
_et al._ (2015), have proven able to yield high-quality factor resonators
($Q>10^{5}$) at moderate ($<1$ T) and large magnetic fields (6T). Yet, in most
works reporting resonators resilient to magnetic fields, a drop in their
intrinsic quality factors appears at specific magnetic fields. Using
resonators of various frequencies fabricated in NbTiN on Si, NbN on SiO2, and
granular Al on sapphire, previous worksSamkharadze  _et al._ (2016); Yu  _et
al._ (2021); Borisov  _et al._ (2020) observed losses corresponding to a near
perfect spin $1/2$ bath with a g-value
$g=\hbar\gamma_{\mathrm{eff}}/\mu_{\mathrm{B}}=2.0\pm 0.3$, where
$\gamma_{\mathrm{eff}}=\partial\omega_{\mathrm{s}}/\partial B_{0}$ is the
effective gyromagnetic ratio governing the measured spin frequency
$\omega_{s}$ dependence on the applied magnetic field $B_{0}$. Besides, Kroll
and coworkersKroll  _et al._ (2019) also observed a similar effect for NbTiN
on sapphire, but at a value slightly lower than 2 (about g = 1.9). For NbN on
sapphirede Graaf  _et al._ (2017), a spin 1/2 absorption line
$\omega_{s}(B_{0})$ was observed, along with satellite lines corresponding to
hydrogen physiosorbed on the surface, which creates a spin system with a
signature hydrogen hyperfine splitting of $A/(2\pi)=$1.42GHz. These hydrogen
atoms absorbed on the surface are also thought to contribute significantly to
flux noise in Quintana  _et al._ (2017) in superconducting circuits.
Additionally, contribution from O2 adsorbates and superoxide systems embedded
into the surface of the sapphire lattice was reportedde Graaf  _et al._
(2017), further confirmed by high-field EPR studies Un  _et al._ (2022) and
DFT calculations Lee  _et al._ (2014). They show that spin systems
respectively create a broad absorption at $g\approx 2.04$ and constant losses
occurring for $g\in[1.5,4]$. Here, we show that using an aluminum mask instead
of a photoresist mask when patterning NbTiN thin films prevents the appearance
of most of these TLS, except for a signal we attribute to defects within the
NbTiN films.

Figure 2:  (a) Magnetic-field induced dissipation rate of circle resonators
for different fabrication runs as a function of magnetic field, error bar per
point is less than 5. Red, NbTiN on EFG substrate etched through a resist mask
("EFG Resist"); green, same sample with 2 days aging ("EFG Resist Aging");
orange, change to an Al mask with annealing ("EFG Al mask Anneal"); blue,
change to HEM substrate, no resist coating during dicing ("HEM Al mask
Anneal"). For direct comparison, the magnetic field is rescaled (see main
text). (b) Frequency-diagram of all absorption peaks: each marker y-coordinate
is given by the frequency of a resonator, and its x-coordinate corresponds to
the occurrence field of a given feature. Blue dashed lines model an electronic
spin $1/2$ coupled to a nuclear spin $1/2$ with a hydrogen hyperfine coupling,
gray, black and red lines are linear fits corresponding respectively to
$\Delta_{0}/(2\pi)=1.07,1.04,0.28$ GHz and $g=4.66,1.77,1.85$.

We pattern NbTiN resonators using a dry etching process (Fig. 1a) by starting
from a 50-nm-thick NbTiN film sputtered on a 2" sapphire wafer. The wafer is
either an edge-defined-grown wafer (EFG, University Wafers), or grown by a
heat-exchanger method (HEM, Crystec). The wafer is then diced in rectangular
dies of 5 $\times$ 7mm, protected using either photoresist (EFG wafers), or a
50-nm-thick Al layer covered in photoresist (HEM wafers). For each die, we
pattern using optical lithography either a photoresist mask (Shipley S1813) or
an aluminum mask. The latter is realized by safekeeping the Al layer deposited
pre-dicing, or by depositing 50-nm of aluminum after cleaning the photoresist
off the die post-dicing. The pattern from the photoresist is transferred to
the Al layer by an acid wet-etching (Transene Type A) and the photoresist is
cleaned off. The NbTiN layer is then etched using an ICP-RIE tool (Sentech)
using CHF3/Ar/N2/O2 atmosphere (30/100/10/3 sccm). If an Al mask is present,
it is then wet etched with the same etch as above. A final cleaning is then
performed using N-Methyl-2-pyrrolidone followed by isopropanol rinsing. On
some devices, to remove the identified contribution of the physiosorbed
hydrogen, we performed an annealing at 300 or 600 in N2 atmosphere. On one
die, we performed a buffered oxide etch of 20 min at room-temperature to
remove the oxides formed on top of the NbTiN layer before performing an
annealing in Ar atmosphere. In total, we prepared 9 dies, with various
preparation conditions summarized in the supplementary material. Out of these
2 dies, two were subjected to open air aging and were measured twice.

We have used two types of resonator geometry: low impedance resonators
($Z_{\mathrm{c}}\approx 20-40~{}\Omega$) targeted for electron spin resonance
(ESR) sensing (circle), and higher impedance ($Z_{\mathrm{c}}\approx
200~{}\Omega$) resonators targeted for resilience to magnetic field
(microwires) see Fig 1b. The first geometry comprises a circular
interdigitated capacitor of pitch 20 surrounding an inductive loop or a short
microwire. The second is an inductive ribbon of width 10 whose ends are
capacitively coupled. The resonators are capacitively or inductively coupled
to a coplanar waveguide (CPW). The measurement is done either in hanger
geometry with the CPW running through the chip, or in reflection with the CPW
terminated on chip. The width of the ground plane surrounding the CPW is
reduced to 70 so that all resonators are free floating and not encased in a
superconducting ground plane. This is done to minimize the nucleation of
magnetic vortices nearby the resonators Song  _et al._ (2009) and prevent
vortices induced losses when applying a dc magnetic field.

On each die, 6 to 8 resonators were patterned. Using well-established fitting
procedures Megrant  _et al._ (2012); Probst  _et al._ (2015); Rieger  _et al._
(2023), we extract for each resonance its frequency
$\omega_{\mathrm{r}}/2\pi$, and its intrinsic and coupling quality factors
$Q_{i}$ and $Q_{c}$ at low intra-resonator photon number ($\bar{n}\sim 1-10$).
The intrinsic quality factors $Q_{i}^{0}$ obtained at zero magnetic-field for
the different fabrication techniques are shown in Fig. 1c and range from
$10^{4}$ to $10^{6}$. All fabrication techniques exhibit at least a few
resonators with $Q_{i}$ exceeding $5\times 10^{5}$. All devices fabricated
using HEM wafers, and whose NbTiN layer was not in contact with resist, obtain
$Q_{i}$ consistently larger than $10^{5}$ at zero magnetic fields. With
annealing, the intrinsic quality factors improve slightly.

We next study the behavior of each resonator under magnetic field. To isolate
the dependence of the resonator intrinsic decay rate $\kappa_{\mathrm{s}}$ on
the magnetic field, we subtract the loss rate $\omega_{\mathrm{r}}/Q_{i}^{0}$
observed at zero-field and the additional loss rate
$\omega_{\mathrm{r}}/Q_{i}^{wb}$ occurring when the Al wire bonds connecting
the die to the sample holder transit from superconducting to normal state at
$B_{0}\sim$10mT. We plot $\kappa_{\mathrm{s}}(\tilde{B}_{0})$ in Fig. 2a for
resonators made with different fabrication techniques, using a re-scaled
x-axis $\tilde{B}_{0}=B_{0}~{}\tilde{\omega}_{\mathrm{r}}/\omega_{r}$ with
$\tilde{\omega}_{\mathrm{r}}/2\pi=$8GHz for easier comparison. Focusing first
on a NbTiN resonator patterned on an EFG wafer using a photoresist mask
(green), we observe 4 different peaks, as well as the onset of a large plateau
at $\tilde{B}_{0}=$50mT. To identify these features, we build their frequency-
field diagram using all the resonators we have measured (see Fig. 2b). We find
that the absorption peak frequency of each feature depends linearly on the
magnetic field, allowing us to extract a zero-field splitting $\Delta_{0}$ and
an effective $g$ value for each feature:
$\omega_{\mathrm{r}}=\Delta_{0}+g\frac{\mu_{B}}{\hbar}B_{0}$.

The first peak lying at $\tilde{B}_{0}=$100mT is very sharp compared to the
others, and corresponds to a $g$-value of $4.7$ (gray, Fig. 2b). We attribute
it to a response from the sapphire substrate. Indeed, while this effective
$g$-value does not correspond to typical contaminants (Fe,Cr), less typical
impurities (V,Mo,Mn,Ti,Gd) or known defects in sapphire which are magnetically
active (V and F2 centers), this line has already been observed without being
identified in other studies using HEMEX sapphire Farr  _et al._ (2013) and
Verneuil grown sapphire Bletskan  _et al._ (2008). We thus note that it cannot
be tied to a particular growth technique. We nevetheless observe its
disappearance when switching from EFG to HEM-grown sapphire. Note that we also
observe a faint line at $g=3.3$ that also disappears and that we cannot
identify.

Next, we turn to the spin 1/2 line surrounded by an upper and lower satellite
peaks (blue markers in Fig. 2b). These three lines have been observed in Refde
Graaf  _et al._ (2017) and have been attributed to water adsorption on the
sapphire surface resulting in imperfect hydroxylation of the surface and the
creation of radicals and unpaired hydrogens. The radicals create the central
spin 1/2 line ($g=2$), and the coupling of a large part of these radicals to
atomic hydrogen gives rise to a hyperfine splitting of strength
$A/(2\pi)=1.45$ GHz leading to the appearance of the satellite peaks. These
features can be removed by desorbing the sapphire surface, which was
successfully done in previous workUn  _et al._ (2022) by annealing at 300
under vacuum. We observe a similar disappearance using an annealing at the
same temperature under both N2 or Ar atmosphere in our measurements. Using the
relative amplitude of the two satellite peaks, we can infer the polarization
of the probed spin ensemble and deduce a spin temperature of 80. This value is
reasonable considering the limited attenuation we have placed on the input
probe line (see supplementary material).

We now move to two correlated features linked to the adsorption of O2 and H2O
on the sample surface: the peak at $\tilde{B}_{0}=$270 and the plateau
appearing at $\tilde{B}_{0}=$50\. Imperfect hydroxylation and defect sites in
the sapphire located near the surface leads to the production of superoxides
(O2- and HO2-) contributing to the broad peak at $g=1.77$ (black markers in
Fig. 2b)Un  _et al._ (2022); de Graaf  _et al._ (2017). For O2 molecules which
are not reduced by interaction with the surface (either the sapphire or the
NbTiN surface), they are expected to respond to ESR probing accordingly to
their triplet state nature i.e., as a broad plateau resonance extending from
$g=4.0$ to low values. We see both features in our experiment, with the
superoxide peak appearing more strongly for aged samples which were left in
open air for 3 days.

Figure 3: Integrated losses for the $g=1.85$ peak depending on the
participation ratio of the free sapphire surface (left) and of the NbTiN
surface (right). Markers indicate various fabrication conditions: etch through
an aluminum mask without post-annealing (green crosses), aluminum mask with
post-annealing at 300 when the NbTiN was covered with resist during dicing
(black crosses) or was never in contact with resist (red crosses), etch
through aluminum mask with post-annealing at 600 (blue triangles, $\times 0.5$
for clarity), etch through an aluminum mask with argon post-annealing at 300
(cyan star). The green dash line is a linear fit using the black and green
crosses, enabling to find the spin concentration giving rise to the $g=1.85$
peak.

The last feature lies at $g=1.85$, with an extracted zero-field splitting
$\Delta_{0}=0.28\pm 0.12$ GHz (red markers in Fig. 2b). The absorption feature
has an asymmetric shape, and has not been previously reported, though it may
have been identified as a spin 1/2 line in a similar study of NbTiN resonators
on sapphire Kroll  _et al._ (2019). To identify its origin, we calculated for
each resonator geometry the participation ratio $p_{i}$ of each surface
$\Sigma_{i}$ to the total mode energy using finite-elements microwave
simulations (see supplementary material) Wang  _et al._ (2015). Namely, we
wish to distinguish between two hypotheses: whether this spin system is
located on the sapphire free surface $\Sigma_{f}$, or on the superconductor
surface $\Sigma_{\mathrm{sc}}$ (see Fig. 1a). Using a multipeak fitting
routine, we have calculated for each resonator the integrated absorption loss
$\int\kappa_{\mathrm{s}}/\omega_{0}~{}dB_{0}$. In Fig. 3, we plot this peak
area versus $p_{\mathrm{f}}$ and $p_{\mathrm{sc}}$ for various fabrication
protocols. Concentrating on the fabrication without annealing (green markers)
for both a resist or an aluminum mask, we see that the peak area is
proportional to $p_{\mathrm{sc}}$, while its dependence on $p_{\mathrm{f}}$ is
non-linear. We thus conclude that the defect is more likely to emanate from
the NbTiN surface. From this linear fit, we can extract that the surface spin
concentration is 3.1(2)e12 (see supplementary material). Besides, the
absorption does not depend on the in-plane orientation of the magnetic field
so that the spins are either isotropic, or randomly orientated. We can explore
a few hypotheses on the origin of these spins. The fluorine etch could leave
paramagnetic residues on the edge of the superconducting electrodesFonash
(1990); Surajit Kumar Hazra  _et al._ (2009). For instance, TiF3 has a
paramagnetic response, yet its g-value is anisotropic and would give rise to a
broad line extending from 2 to 1.85Vore and Jr (2002), which does not agree
with our measurements. Some electronic states of niobium and titanium exhibit
a paramagnetic response, for instance electronic states Nb2+ and Nb4+ have a
paramagnetic response with $g$-values around $1.85$Johnson  _et al._ (1973);
Ziolek and Nowak (2003), while Ti3+ is anisotropic, with principal $g$ -values
lying between 1.9 and 2. Yet, we note that the feature does not appear in Nb
or NbN resonators Kwon  _et al._ (2018); de Graaf  _et al._ (2017), so a
contribution from Nb donor states seems unlikely, or deeply linked to the
presence of Ti in the film.

These features contribute to a background signal strongly dependent on the
magnetic field, which complicates the identification of an ESR signal, and
that also limits the overall achievable quality factor. We now review the
impact of different fabrication processes on the different features to
elaborate a mitigation strategy. We plot on Fig. 4 the peak height
$\kappa_{\mathrm{s}}^{f}$ for each feature, depending on the fabrication
protocol. The contribution for each feature was isolated using a multi-peak
fit. (1) We note that similarly to measurements on NbN resonators on sapphire,
annealing removes the hydrogen peaks (blue) as well as the central OH- peak,
independently of the annealing gas. (2) Avoiding contact between photoresist
and NbTiN suppresses the plateau resulting from the contribution of triplet
O2: indeed for HEM wafers where only Al was in contact with NbTiN, the plateau
is absent, but for EFG wafers, even when an Al mask is used, contact with the
protective photoresist layer during dicing is enough for the plateau to be
observed. (3) When cleaning the wafer with a long buffered oxide etch after
removing the mask, the plateau reappears, with a reduced height. It also gives
rise to a small $g=2$ feature which is not removed by annealing. Putting aside
the NbTiN feature, it thus looks like the optimal fabrication technique among
those explored here is etching through an aluminum mask followed by an
annealing.

Figure 4: Feature peak height $\kappa_{\mathrm{s}}^{\mathrm{f}}$ extracted
from a multipeak fit for the following features: lowest hydrogen hyperfine
peak (blue), O2 plateau (orange, $\times 3$, for visibility), OH- radicals
(black), and the unidentified feature related to the superconducting film
(red) depending on each fabrication protocol. Markers indicate measured
values, boxes show the standard deviation around the mean. *Al mask indicates
the wafer was covered by photoresist during dicing, but that an Al mask was
used during etching.

The NbTiN feature at $g=1.85$ however shows a strong increase when the sample
is annealed : for samples whose NbTiN was never covered by resist, we observe
a factor $6$ increase for a 300 annealing under N2 or Ar atmosphere (red), and
a factor $45$ increase for 600 annealing under N2 (see Fig. 3). Note that we
observe that when heating NbTiN under N2 atmosphere at higher temperature, the
film swells and gets enriched in nitrogen, moving away from a balanced two
atoms face cubic-centered lattice with half of the lattice occupied by
nitrogen atoms, and the other occupied by Nb and Ti atoms Zhang  _et al._
(2015). It could thus be that enrichment of the films in nitrogen and/or a
depletion of the film in Ti creates donor niobium sites that give rise to the
feature. Remarkably, the sample which was covered by resist for dicing, but
etched with an aluminum mask does not show the same increase so that we
postulate that an oxide passivation layer is sufficient to counteract the
effect of annealing on the NbTiN feature.

Nevertheless, avoiding putting photoresist into contact with NbTiN for dicing
and etching through an aluminum mask followed by an annealing results in the
best combination to remove all magnetic field dependent features. It also
ensures that we reach reproducible quality factors above $Q>10^{5}$ at zero
magnetic field, in particular for our low-impedance geometry resonator.
Considering all sources of losses, including the contribution from aluminum
wire bonds ($\sim$3e5) and of the NbTiN feature ($\sim$2e5), we can maintain a
total quality factor above $Q>10^{5}$ in the
$0\text{\,}\mathrm{T}0.3\text{\,}\mathrm{T}$ range, an important step to
develop an ESR spectrometer based on superconducting microwave resonators.
Straightforward improvements to our setup could help reduce the length of the
wire bonds so that their contribution is minimized ($>$2e6). The behavior of
the NbTiN feature also makes clear that developing a capping layer that would
act as a barrier during the annealing without contributing to the oxygen
features would help us reach higher quality factors. Broader perspectives
would be to probe the origin of the NbTiN feature by varying the stoichiometry
of the NbTi sputtering target or comparing to TiN resonators.

###### Acknowledgements.

This work was supported by the LABEX iMUST (ANR-10-LABX-0064) and the project
IDEXLYON of Université de Lyon, within the program "Investissements d’Avenir"
(ANR-11-IDEX-0007 and ANR-16-IDEX-0005) operated by the French National
Research Agency (ANR) and by the European Union (ERC, INDIGO, 101039953). We
acknowledge IARPA and Lincoln Labs for providing a Josephson Traveling-Wave
Parametric Amplifier. The NbTiN films were etched using the ICP-RIE tool of
the Nanolyon platform. We thank the cleanroom of the SPEC at CEA-Saclay and
Pief Orfila for sputtering NbTiN on our wafers.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

##  I Appendixes

###  I.1 Samples and measurement setup

The list of samples as well as their precise fabrication process is given in
Table 1.

Sample | Substrate | Dicing | Cleaning | Mask | Etch | Cleaning | Annealing | Probe geometries
---|---|---|---|---|---|---|---|---
1* | EFG | Resist | NMP | Resist |  CHF3 | O2 plasma & NMP | No | Reflection
2* | EFG | Resist | NMP | Resist |  CHF3 | O2 Plasma & NMP | No | Reflection
3 | EFG | Resist | NMP | Resist |  CHF3 | NMP | No | Reflection
4 | EFG | Resist | NMP | Al |  CHF3 | Al etch |  300 N2 | Reflection
5 | HEM | Al + Resist | NMP | Al |  CHF3 | Al etch |  300 N2 | Hanger
6 | HEM | Al + Resist | NMP | Al |  CHF3 | Al etch | No | Reflection
7 | HEM | Al + Resist | NMP | Al |  CHF3 | Al etch | No | Reflection
8 | HEM | Al + Resist | NMP | Al |  CHF3 | Al etch |  300 Ar | Hanger
9 | EFG | Resist | NMP | Al |  CHF3 | Al etch |  600 N2 | Hanger
Table 1: List of measured samples and their fabrication process. Samples 1 and
2 were measured in a second run after a few days ageing.

After fabrication, the samples are wirebonded to a OFHC copper sample holder
and inserted into a vector magnet anchored to the base plate of a dilution
refrigerator operating at 12mK for measurement. The microwave setup is shown
on Fig. 5. The magnet is placed inside a cryoperm shield and calibrated using
BDPA samples. The incoming signal is attenuated with a total of 43. We include
an Eccosorb filter in-front of the circulator before the device under test
(DUT) to reduce quasi-particles caused by infrared radiation. The input power
is chosen such that the intra-resonator photon number is of order 1. The
outgoing signal is amplified by a Josephson traveling-wave parametric
amplifier followed by a high-electron mobility transistor amplifier. The
measurements are carried out using a vector network analyzer (VNA).

Figure 5: Microwave setup for transmission (red) and reflection measurements
(blue).

###  I.2 Resonator fitting

The resonators on the various samples are probed either in reflection or in a
transmission hanger geometry. To extract the internal quality factor from the
VNA measurements for these two coupling schemes, we model the reflection
($R_{R}$) and hanger-mode ($R_{H}$) response as Probst  _et al._ (2015);
Rieger  _et al._ (2023):

$\begin{split}R_{R}&=\left[1-\frac{2Q_{l}/Q_{c}}{1+2iQ_{l}(\omega/\omega_{\mathrm{r}}-1)}\right]\\\
R_{H}&=\left[1-\frac{\left(Q_{l}/Q_{c}\right)e^{i\phi_{0}}}{1+2iQ_{l}(\omega/\omega_{\mathrm{r}}-1)}\right]\end{split}$
(1)

The complex phase $\phi_{0}$ accounts for impedance mismatches in the coupling
between resonator and measurement lines. We define
$Q_{l}=\left[1/Q_{i}+1/Q_{c}\right]^{-1}$ as the loaded loss of the resonator.
The measured response of the resonator $S_{m}$ at the VNA is actually
convoluted with the response of the measurement lines. This response can be
described as an overall complex scaling factor $Ae^{i\alpha}$ and an
electrical delay $\tau$ giving a frequency-dependent phase $e^{i\omega\tau}$.
To recover only the resonator response, we renormalize the data using
$\tilde{S}_{m}=S_{m}(\omega)e^{-i\omega t}/S_{m}(\infty)$ so that
$\tilde{S}_{m}(\omega)$ lies on a circle in the complex plane. We then
estimate the best circle fitting the data in the complex plane, obtaining its
radius $R$ and center $z_{c}=x_{c}+iy_{c}$.

We perform the affine transformation
$S_{21}^{\\{R,H\\}_{C}}=S_{21}^{\\{R,H\\}_{N}}-z_{c}$ and compute the phase of
the transformed quantity, obtaining:

$\begin{split}\phi_{R}&=2\arctan\left[2(\omega/\omega_{\mathrm{r}}-1)Q_{l}\right]\\\
\phi_{H}&=2\arctan\left[2(\omega/\omega_{\mathrm{r}}-1)Q_{l}\right]-\phi_{0}\end{split}$
(2)

which can be fitted to extract the loaded quality factor and the resonant
frequency of the resonator. Using the relations between ($Q_{i}$,$Q_{c}$) and
($R$,$Q_{l}$)

$\begin{split}Q_{c}^{R}=Q_{l}^{R}/R\hskip
28.45274pt&1/Q_{i}^{R}=1/Q_{l}^{R}-1/Q_{c}^{R}\\\ Q_{c}^{H}=Q_{l}^{H}/2R\hskip
28.45274pt&1/Q_{i}^{H}=1/Q_{l}^{H}-1/Q_{c}^{H}\end{split}$ (3)

it is possible to estimate the coupling and internal quality factors. As the
determination of the quality factors involves three steps (re-normalization,
circle fitting, and phase fitting), each step can contribute to the
uncertainty. In practice, the errors due to the circle fitting, and in
particular on the circle radius largely offset the other two. The Qc value is
fixed to the average fitted value for all magnetic fields thus reducing its
uncertainty. For the intrinsic quality factor, as $1/Q_{i}=(1-rR)/(rRQ_{c})$
we estimate $\Delta(1/Q_{i})=\Delta(R)/(rR^{2}Q_{c})$ where $r=2$ in hanger
geometry and $r=1$ in reflection. This analysis does not take into account
Fano interferences or direct-direct port leakage which can introduce a
systematic error in the determination. However, as we are only looking at
extra-losses created into resonators of similar geometries, this systematic
error does not particularly affect our study.

###  I.3 Modelling of surface spin losses for a superconducting resonator

In this section, we model losses created by spins lying either on the metallic
area of a planar superconducting resonator, or on the nearby uncovered
substrate surface.

####  I.3.1 Spin induced losses

Each family of spins can be described by its Hamiltonian $H_{s}(\vec{B})$
which can be diagonalized to extract the spin family eigenstates and
eigenvalues. When a static magnetic field $\vec{B}_{0}$ is applied, a specific
transition between states $\|1\rangle$ and $|2\rangle$ occurring at frequency
$\omega_{s}(B_{0})$ can be brought to resonance. Each spin interacts with the
resonator through a Jaynes-Cumming Hamiltonian. The interaction strength
$g_{j}$ of a single spin is given by $\hbar g=|\langle 1|H_{s}(\vec{\delta
B}(\vec{r}))|2\rangle|$, where $\vec{\delta B}_{ac}(\vec{r})$ are the vacuum
fluctuations of the resonator electromagnetic mode at the spin position
$\vec{r}$. We consider that the spin ensemble has an inhomogenous distribution
in frequency of linewidth $\Gamma$ while each spin has a decoherence rate
$\Gamma_{2}$. We also introduce the collective coupling constant as
$g_{ens}=\sum_{j}g_{j}^{2}$. In the limit of small cooperativity
$4g_{ens}^{2}/(\kappa\Gamma)<<1$, assuming the spin frequency distribution is
Lorentzian, the response $R(\omega)$ for the resonator is Diniz  _et al._
(2011):

$R(\omega)=1-\frac{r\sqrt{\kappa_{c}}}{\kappa+\kappa_{\mathrm{s}}-2i(\omega-\omega_{\mathrm{r}})+2i\delta_{s}}$
(4)

where
$\kappa_{\mathrm{s}}=4g^{2}_{ens}\Gamma/((\omega_{s}-\omega_{\mathrm{r}})^{2}+\Gamma^{2}/4)$
and
$\delta_{s}=4g^{2}_{ens}\Gamma(\omega_{s}-\omega_{\mathrm{r}})/((\omega_{s}-\omega_{\mathrm{r}})^{2}+\Gamma^{2}/4)$,
and $r=2$ when measuring in reflection, and $1$ when measuring in hanger
geometry.

Eq. 4 shows clearly that the spins create an additional decay channel for the
resonator, which depends on the spin frequency. Integrating the losses over
the entire spin linewidth probed in magnetic field thus yields:

$\int_{0}^{+\infty}\frac{\kappa_{\mathrm{s}}}{\omega_{\mathrm{r}}}d\omega_{s}=\frac{4\pi
g_{ens}^{2}}{\omega_{\mathrm{r}}}$ (5)

####  I.3.2 Magnetic field participation ratio

These spin induced losses can be shown to be proportional to a magnetic field
participation ratio, similarly to dielectric losses being proportional to an
electrical participation ratio Wang  _et al._ (2015). On one hand, we can
calculate the magnetic field participation ratio as:

$p=\frac{\frac{1}{\mu_{0}}\int_{V_{s}}|\delta
B_{\perp}|^{2}dV}{\frac{1}{\mu_{0}}\int_{V}|\delta
B|^{2}dV}=\frac{2}{\mu_{0}\hbar\omega_{\mathrm{r}}}\int_{V_{s}}|\delta
B_{\perp}|^{2}dV$ (6)

where $V_{s}$ is the volume containing the spins of interest (either at the
surface of the sapphire, or the surface of the NbTiN), and $V$ is the entire
mode volume. Assuming the Zeeman effect predominates in the spin Hamiltonian
and that the spin system behaves closely to a spin 1/2, only the ac-field
components orthogonal to the static field $B_{0}$ will induce a coupling to
the spin, we thus only consider $|\delta B_{\perp}|^{2}$ for calculating the
magnetic field participation ratio.

On the other hand, for a resonator-spin transverse coupling, the spin
collective coupling constant can be calculated as :

$g_{ens}^{2}=\int_{V_{s}}c|\delta B_{\perp}|^{2}\gamma_{e}^{2}|\langle
1|S_{x}/2|2\rangle|^{2}dV$ (7)

where $c$ is the spin volume concentration, $\gamma_{e}$ is the spin
transition gyromagnetic ratio, and $S_{x}$ the dimensionless spin operator. We
thus find:

$\int_{0}^{B_{0}^{\mathrm{max}}}\frac{\kappa_{\mathrm{s}}}{\omega_{\mathrm{r}}}dB_{0}=h\mu_{0}c\gamma_{e}|\langle
1|S_{x}/2|2\rangle|^{2}p,$ (8)

giving a linear relation between peak area and participation ratio. To
evaluate numerically the participation ratios for all designs, we performed
microwave simulations (using HFSS Ansys software) modelling the NbTiN as a
perfect conductive layer without any thickness. We extract the contribution of
two zones, the free sapphire surface $\Sigma_{f}$ and the NbTiN surface
$\Sigma_{sc}$, to the overall magnetic mode energy
$U_{B}=\frac{\int_{V}|B|^{2}dV}{2\mu_{0}}$. We use the included field
calculator to evaluate
$p_{\Sigma}=t\int_{\Sigma}|B_{\perp}|^{2}d\Sigma/(2\mu_{0}U_{B})$, where $t$
represents the thickness of the considered surface. We take $t=$3nm, which is
a fair assumption for radicals contribution at the sapphire surface Un  _et
al._ (2022), but a rather arbitrary choice for the NbTiN surface (that
nevertheless allows for convenient comparison to the sapphire surface). This
modeling also neglects the inhomogeneous current distribution within a
superconducting sheet.

## References

  * Burkard  _et al._ (2020) G. Burkard, M. J. Gullans, X. Mi, and J. R. Petta, Nature Reviews Physics 2, 129 (2020).
  * Clerk  _et al._ (2020) A. A. Clerk, K. W. Lehnert, P. Bertet, J. R. Petta, and Y. Nakamura, Nature Physics 16, 257 (2020).
  * Sigillito  _et al._ (2014) A. J. Sigillito, H. Malissa, A. M. Tyryshkin, H. Riemann, N. V. Abrosimov, P. Becker, H.-J. Pohl, M. L. W. Thewalt, K. M. Itoh, J. J. L. Morton, A. A. Houck, D. I. Schuster, and S. A. Lyon, Appl. Phys. Lett. 104, 222407 (2014).
  * Wang  _et al._ (2023) Z. Wang, L. Balembois, M. Rančić, E. Billaud, M. L. Dantec, A. Ferrier, P. Goldner, S. Bertaina, T. Chanelière, D. Estève, D. Vion, P. Bertet, and E. Flurin, Nature 619, 276 (2023), arxiv:2301.02653 [cond-mat, physics:quant-ph] .
  * McRae  _et al._ (2020) C. R. H. McRae, H. Wang, J. Gao, M. R. Vissers, T. Brecht, A. Dunsworth, D. P. Pappas, and J. Mutus, Review of Scientific Instruments 91, 091101 (2020).
  * Barends  _et al._ (2011) R. Barends, J. Wenner, M. Lenander, Y. Chen, R. C. Bialczak, J. Kelly, E. Lucero, P. O’Malley, M. Mariantoni, D. Sank, H. Wang, T. C. White, Y. Yin, J. Zhao, A. N. Cleland, J. M. Martinis, and J. J. A. Baselmans, Applied Physics Letters 99, 113507 (2011).
  * Song  _et al._ (2009) C. Song, T. W. Heitmann, M. P. DeFeo, K. Yu, R. McDermott, M. Neeley, J. M. Martinis, and B. L. T. Plourde, Physical Review B 79, 174512 (2009).
  * Wang  _et al._ (2015) C. Wang, C. Axline, Y. Y. Gao, T. Brecht, Y. Chu, L. Frunzio, M. H. Devoret, and R. J. Schoelkopf, Appl. Phys. Lett. 107, 162601 (2015).
  * Dial  _et al._ (2016) O. Dial, D. T. McClure, S. Poletto, J. M. Gambetta, D. W. Abraham, J. M. Chow, and M. Steffen, Superconductor Science and Technology 29, 044001 (2016), arxiv:1509.03859 [cond-mat, physics:quant-ph] .
  * Müller  _et al._ (2019) C. Müller, J. H. Cole, and J. Lisenfeld, Reports on Progress in Physics 82, 124501 (2019).
  * Barends  _et al._ (2009) R. Barends, H. L. Hortensius, T. Zijlstra, J. J. A. Baselmans, S. J. C. Yates, J. R. Gao, and T. M. Klapwijk, IEEE Transactions on Applied Superconductivity 19, 936 (2009).
  * Quintana  _et al._ (2017) C. M. Quintana, Y. Chen, D. Sank, A. G. Petukhov, T. C. White, D. Kafri, B. Chiaro, A. Megrant, R. Barends, B. Campbell, Z. Chen, A. Dunsworth, A. G. Fowler, R. Graff, E. Jeffrey, J. Kelly, E. Lucero, J. Y. Mutus, M. Neeley, C. Neill, P. J. J. O’Malley, P. Roushan, A. Shabani, V. N. Smelyanskiy, A. Vainsencher, J. Wenner, H. Neven, and J. M. Martinis, Physical Review Letters 118, 057702 (2017).
  * de Graaf  _et al._ (2017) S. E. de Graaf, A. A. Adamyan, T. Lindström, D. Erts, S. E. Kubatkin, A. Ya. Tzalenchuk, and A. V. Danilov, Physical Review Letters 118, 057703 (2017).
  * Anton  _et al._ (2013) S. M. Anton, J. S. Birenbaum, S. R. O’Kelley, V. Bolkhovsky, D. A. Braje, G. Fitch, M. Neeley, G. C. Hilton, H.-M. Cho, K. D. Irwin, F. C. Wellstood, W. D. Oliver, A. Shnirman, and J. Clarke, Physical Review Letters 110, 147002 (2013).
  * Sendelbach  _et al._ (2008) S. Sendelbach, D. Hover, A. Kittel, M. Mück, J. M. Martinis, and R. McDermott, Physical Review Letters 100, 227006 (2008).
  * Sank  _et al._ (2012) D. Sank, R. Barends, R. C. Bialczak, Y. Chen, J. Kelly, M. Lenander, E. Lucero, M. Mariantoni, A. Megrant, M. Neeley, P. J. J. O’Malley, A. Vainsencher, H. Wang, J. Wenner, T. C. White, T. Yamamoto, Y. Yin, A. N. Cleland, and J. M. Martinis, Physical Review Letters 109, 067001 (2012).
  * Kumar  _et al._ (2016) P. Kumar, S. Sendelbach, M. A. Beck, J. W. Freeland, Z. Wang, H. Wang, C. C. Yu, R. Q. Wu, D. P. Pappas, and R. McDermott, Physical Review Applied 6, 041001 (2016).
  * Lisenfeld  _et al._ (2016) J. Lisenfeld, A. Bilmes, S. Matityahu, S. Zanker, M. Marthaler, M. Schechter, G. Schön, A. Shnirman, G. Weiss, and A. V. Ustinov, Scientific Reports 6, 23786 (2016).
  * Woods  _et al._ (2019) W. Woods, G. Calusine, A. Melville, A. Sevi, E. Golden, D. Kim, D. Rosenberg, J. Yoder, and W. Oliver, Physical Review Applied 12, 014012 (2019).
  * Murray (2021) C. E. Murray, Materials Science and Engineering: R: Reports 146, 100646 (2021).
  * Lederer and Raskin (2003) D. Lederer and J.-P. Raskin, Solid-State Electronics 47, 1927 (2003).
  * Farr  _et al._ (2013) W. G. Farr, D. L. Creedon, M. Goryachev, K. Benmessai, and M. E. Tobar, Physical Review B 88, 224426 (2013).
  * Kwon  _et al._ (2018) S. Kwon, A. Fadavi Roudsari, O. W. B. Benningshof, Y.-C. Tang, H. R. Mohebbi, I. A. J. Taminiau, D. Langenberg, S. Lee, G. Nichols, D. G. Cory, and G.-X. Miao, Journal of Applied Physics 124, 033903 (2018).
  * Yu  _et al._ (2021) C. X. Yu, S. Zihlmann, G. Troncoso Fernández-Bada, J.-L. Thomassin, F. Gustavo, É. Dumur, and R. Maurand, Applied Physics Letters 118, 054001 (2021).
  * Russo  _et al._ (2023) R. Russo, A. Chatel, N. Ş. Solmaz, R. Farsi, H. Furci, J. Brugger, and G. Boero, Micro and Nano Engineering 19, 100203 (2023).
  * Samkharadze  _et al._ (2016) N. Samkharadze, A. Bruno, P. Scarlino, G. Zheng, D. P. DiVincenzo, L. DiCarlo, and L. M. K. Vandersypen, Appl. Phys. Lett. 5, 044004 (2016).
  * Borisov  _et al._ (2020) K. Borisov, D. Rieger, P. Winkel, F. Henriques, F. Valenti, A. Ionita, M. Wessbecher, M. Spiecker, D. Gusenkova, I. M. Pop, and W. Wernsdorfer, Applied Physics Letters 117, 120502 (2020).
  * Ghirri  _et al._ (2015) A. Ghirri, C. Bonizzoni, D. Gerace, S. Sanna, A. Cassinese, and M. Affronte, Applied Physics Letters 106, 184101 (2015).
  * Kroll  _et al._ (2019) J. Kroll, F. Borsoi, K. van der Enden, W. Uilhoorn, D. de Jong, M. Quintero-Pérez, D. van Woerkom, A. Bruno, S. Plissard, D. Car, E. Bakkers, M. Cassidy, and L. Kouwenhoven, Physical Review Applied 11, 064053 (2019).
  * Un  _et al._ (2022) S. Un, S. de Graaf, P. Bertet, S. Kubatkin, and A. Danilov, Science Advances 8, eabm6169 (2022).
  * Lee  _et al._ (2014) D. Lee, J. L. DuBois, and V. Lordi, Physical Review Letters 112, 017001 (2014).
  * Megrant  _et al._ (2012) A. Megrant, C. Neill, R. Barends, B. Chiaro, Y. Chen, L. Feigl, J. Kelly, E. Lucero, M. Mariantoni, P. J. J. O’Malley, D. Sank, A. Vainsencher, J. Wenner, T. C. White, Y. Yin, J. Zhao, C. J. Palmstrom, J. M. Martinis, and A. N. Cleland, Appl. Phys. Lett. 100, 113510 (2012).
  * Probst  _et al._ (2015) S. Probst, F. B. Song, P. A. Bushev, A. V. Ustinov, and M. Weides, Review of Scientific Instruments 86, 024706 (2015).
  * Rieger  _et al._ (2023) D. Rieger, S. Günzler, M. Spiecker, A. Nambisan, W. Wernsdorfer, and I. Pop, Physical Review Applied 20, 014059 (2023).
  * Bletskan  _et al._ (2008) D. I. Bletskan, V. Ya. Bratus’, A. R. Luk’yanchuk, V. T. Maslyuk, and O. A. Parlag, Technical Physics Letters 34, 612 (2008).
  * Fonash (1990) S. J. Fonash, Journal of The Electrochemical Society 137, 3885 (1990).
  * Surajit Kumar Hazra  _et al._ (2009) Surajit Kumar Hazra, Sangmin Lee, Byoungdoo Choi,, Nam-kuk Kim, Tae-young Choi, and Dong-Il “Dan” Cho, Sensors and Materials , 37 (2009).
  * Vore and Jr (2002) T. C. D. Vore and W. W. Jr, “Titanium difluoride and titanium trifluoride molecules: Electron spin resonance spectra in rare-gas matrices at 4 K,” https://pubs.acs.org/doi/pdf/10.1021/ja00456a028 (2002).
  * Johnson  _et al._ (1973) D. P. Johnson, J. Wilinski, and R. D. Bereman, Journal of Inorganic and Nuclear Chemistry 35, 2365 (1973).
  * Ziolek and Nowak (2003) M. Ziolek and I. Nowak, Catalysis Today 4th International Symposium on Group Five Compounds Bicentennial Meeting, 78, 543 (2003).
  * Zhang  _et al._ (2015) L. Zhang, W. Peng, L. X. You, and Z. Wang, Applied Physics Letters 107, 122603 (2015).
  * Diniz  _et al._ (2011) I. Diniz, S. Portolan, R. Ferreira, J. M. Gérard, P. Bertet, and A. Auffèves, Physical Review A 84, 063810 (2011).