# Disentangled Speech Representation Learning for
One-Shot Cross-lingual Voice Conversion Using $\beta$-VAE

###### Abstract

We propose an unsupervised learning method to disentangle speech into content
representation and speaker identity representation. We apply this method to
the challenging one-shot cross-lingual voice conversion task to demonstrate
the effectiveness of the disentanglement. Inspired by $\beta$-VAE, we
introduce a learning objective that balances between the information captured
by the content and speaker representations. In addition, the inductive biases
from the architectural design and the training dataset further encourage the
desired disentanglement. Both objective and subjective evaluations show the
effectiveness of the proposed method in speech disentanglement and in one-shot
cross-lingual voice conversion.

Index Terms—  Speech disentanglement, Voice conversion, cross-lingual, one-
shot, unsupervised learning

##  1 Introduction

Voice conversion (VC) is a technique that converts a source speaker’s speech
to make it sound as though it was uttered by the designated target speaker,
while the spoken content remains unchanged [1]. Cross-lingual VC refers to the
scenario where the source and the target speakers do not speak the same
language [2]. Cross-lingual VC is more challenging than intra-lingual VC
because generally only a monolingual corpus is available from the target
speaker, which presents the problem of training-inference mismatch.
Furthermore, one-shot cross-lingual VC is even more challenging, because there
is only a single utterance from the target speaker, who speaks a language that
is different from the source speaker. We focus on one-shot cross-lingual VC in
this paper.

Speech generation embeds different information elements into the signal:
content, speaker identity, emotion, language, etc., where the first two
elements tend to be more prominent. Generally speaking, language can be
considered as part of content. We may regard that a universal phoneme set can
cover pronunciation patterns of all languages, and encode different languages
in the spoken content. Following this rationale, we can solve the one-shot
cross-lingual VC by disentangling speech into a general content representation
and a speaker representation. Then the content representation from the source
speech and the speaker representation from the target speech can be combined
to generate the speech with the source content and the target speaker
identity.

Many methods have been proposed to disentangle the speech into content and
speaker representations for one-shot VC. Commonly adopted architectures are
the auto-encoder (AE) and variational auto-encoder (VAE) [3], with encoder(s)
to extract a frame-level feature sequence and a single vector separately from
the speech, aiming for the former to encode spoken content, and the latter to
encode speaker identity. This design is based on the fact that the speaker
identity is a sequence-level element while the content is a frame-level
element. However, as we will discuss in Sec 3.1, this alone is not enough to
ensure the disentanglement of the content and speaker representations. Many
methods have been proposed to further regularize the representation learning
to facilitate disentanglement.

AutoVC [4] proposes to apply down-sampling and dimension restriction to the
content representation to remove the speaker identity. VQVC and VQVC+ [5, 6]
apply vector quantization (VQ) to the content representation to eliminate the
speaker information. However, through explicit down-sampling or VQ operations,
these methods may hurt the fine-grained temporal information and cause
conversion quality degradation. The mutual information (MI) between the
content and the speaker representations has also been imposed as a learning
objective [7, 8] to encourage them to be statistically independent, but the
estimation of MI is difficult and may make the training process more complex.
Other methods apply instance normalization [9] and activation regularization
[10] to the content representation, which can induce less information loss and
are easier to implement.

Recently, $\beta$-VAE [11, 12] has been proposed as a variant of VAEs for
better latent variable disentanglement. With an extra weight parameter $\beta$
being imposed on the Kullback-Leibler (KL) divergence term, $\beta$-VAE
restricts the channel capacity of the latent variable to encode the
information of the input data. The original $\beta$-VAE is more suitable for
static data such as images. In this paper, we propose a variant of $\beta$-VAE
that is specifically for disentanglement of sequential data into time-variant
and time-invariant representations. Different from $\beta$-VAE which only has
one encoder, we adopt two encoders that are respectively designed for content
and speaker representation learning. In addition, two weight parameters
$\beta_{c}$ and $\beta_{s}$ are imposed on the KL divergence terms for content
and speaker representations, respectively. We show theoretically that the KL
divergence term is an upper bound of the MI between the latent variable and
the input data. Thus, $\beta_{c}$ and $\beta_{s}$ function as two gates to
restrict the amounts of information of the data that can be captured by the
content and speaker representations, respectively. With proper $\beta_{c}$ and
$\beta_{s}$ imposed, the information captured by the two representations are
complementary to each other, while the content information and speaker
identity information are precisely allocated to them, respectively.

Compared to existing one-shot VC methods, the proposed method imposes much
simpler restrictions on the latent variable (only two weight parameters on the
KL divergence terms) which is easier to implement and train. We do not apply
intentionally bottleneck operations such as dimension reduction or down-
sampling to the latent variable, thus less information loss is induced.
Furthermore, the proposed method can be explained by information theory, which
is theoretically more solid.

##  2 Related work

Traditional cross-lingual VC methods rely extensively on the phonetic
posteriorgrams (PPGs) [2, 13, 14, 15], which is a speaker-independent content
representation explicitly extracted via a speaker-independent automatic speech
recognition (SI-ASR) model. VC is achieved by first extracting PPGs from the
source speech, then converting the PPGs into the target speech with a
synthesis model, which can be trained only on the target speaker’s speech
corpus. Though PPGs can be considered as speaker-independent, they are
language-dependent, which means that PPGs of one language may not be a good
content representation of another language. Besides, since the PPGs extractor
is fixed during the training of the synthesis model, it cannot be optimized
for all interested languages. In contrast, the proposed method learns to
extract the content representation for all interested languages jointly with
other parts of the model, thus can in theory learn a more generalized content
representation.

For unsupervised disentangled speech representation learning, FHVAE [16]
propose hierarchical prior distributions for VAE to encourage elements of
different time scales to be factorized, while the sequence-level
representation is further regularized by a discriminative objective. Based on
FHVAE, a method is proposed for one-shot cross-lingual VC [17]. The method
proposed in this paper is different in terms of the restriction imposed, as we
emphasize the effects of two weights on the KL divergence terms and no other
learning objective is introduced. Another recently proposed method for one-
shot intra-lingual VC [18] also stresses the effects of imposing weights on
two KL divergence terms to encourage disentanglement. However, the proposed
method is different from [18] in the following ways: First, the prior
distribution for the content representation of [18] is a trainable
autoregressive one, while ours is an fixed isotropic Gaussian. Second, as we
will show in Section 3.2, the current paper presents a different derivation
(especially for the MI). Besides, it is claimed in [18] that the right choice
of the ratio between the two weights can yield the desired disentanglement,
while we argue that the absolute values of both weights can also be important
for good disentanglement. While some methods [7, 8] estimate the MI between
different representations as a loss term, the derivation of the MI in this
paper only serves as an explanation of the proposed method, we do not need to
estimate it numerically.

##  3 Method

Let $\mathbf{x}$ be the acoustic feature extracted from speech, and the
objective is to find latent variables $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$,
such that $\mathbf{z_{c}}$ encodes only spoken content information, while
$\mathbf{z_{s}}$ exclusively contains speaker identity information. With the
assumption that $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ are statistically
independent, we can derive the objective function as shown in Eqn. (3),
following the formalization of VAEs. In Eqn. (3),
$q_{\phi}(\mathbf{z_{c}}|\mathbf{x})$ and
$q_{\phi}(\mathbf{z_{s}}|\mathbf{x})$, are posterior distributions of
$\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ given $\mathbf{x}$, respectively,
parameterized by $\phi$.
$p_{\theta}(\mathbf{x}|\mathbf{z_{c}},\mathbf{z_{s}})$ denotes the conditional
distribution of $\mathbf{x}$ given $\mathbf{z_{s}}$ and $\mathbf{z_{c}}$ and
is parameterized by $\theta$. $p(\mathbf{z_{c}})$ and $p(\mathbf{z_{s}})$ are
prior distributions of $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$, respectively,
and are defined as isotropic Gaussians in this paper, $p_{d}(\mathbf{x})$ is
the data distribution of $\mathbf{x}$. For simplicity, we refer to the three
terms in Eqn. (3) as the reconstruction loss, content KL, and speaker KL,
respectively.

$\displaystyle\mathcal{L}_{\text{vanilla}}$
$\displaystyle=-\mathbb{E}_{p_{d}(\mathbf{x})q_{\phi}(\mathbf{z_{c}}|\mathbf{x})q_{\phi}(\mathbf{z_{s}}|\mathbf{x})}\left[\log
p_{\theta}(\mathbf{x}|\mathbf{z_{c}},\mathbf{z_{s}})\right]$
$\displaystyle+\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z_{c}}|\mathbf{x})\parallel
p(\mathbf{z_{c}})\right]\right]$
$\displaystyle+\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z_{s}}|\mathbf{x})\parallel
p(\mathbf{z_{s}})\right]\right]$ (1)

The general unsupervised learning model defined above without any other
inductive biases cannot ensure the disentanglement, as is commonly believed
[19]. To achieve the disentanglement of content and speaker representations
with the above formalization, our solutions are as follows: We first introduce
the architectural design that facilitate the separation of time-variant and
time-invariant elements in Section 3.1; In Section 3.2 we present the learning
objective that can encourage $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ to be
complementary, while exactly the content information and speaker identity
information can be assigned separately to them. The effect of the training
dataset is also emphasized in Section 3.3.

###  3.1 Architectural Design

A basic observation about speech is that the content varies constantly across
an utterance while the speaker identity remains the same for the whole
utterance. That is, the content is a time-variant feature while speaker
identity is a time-invariant one. This informs us about the structures of the
content and speaker representations. Suppose that the acoustic feature
$\mathbf{x}$ is of the shape $\mathbb{R}^{T\times D_{x}}$, where $T$ and
$D_{x}$ denote the number of frames and the dimension of $\mathbf{x}$,
respectively. Then $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ should have the
corresponding shapes of $\mathbb{R}^{T\times D_{c}}$ and $\mathbb{R}^{D_{s}}$
to separately capture the time-variant content information and time-invariant
speaker identity information, where $D_{c}$ and $D_{s}$ are the dimensions of
$\mathbf{z_{c}}$ and $\mathbf{z_{s}}$, respectively. A content encoder
transforming the input frame-wisely and a speaker encoder aggregating the
whole input sequence into a single vector should thus be adopted. Many methods
introduced in Section 1 have adopted similar architectures.

However, this architecture alone cannot guarantee the disentanglement of
content and speaker representations. The reasons are two-fold: First, the
architecture cannot ensure that the two representations learned are
complementary to each other. Intuitively, the two representations will strive
to capture as much information about the input data as possible to reduce the
reconstruction loss. Thus, it is possible that the two representations have
overlap in the captured information. Second, even if we can restrict the two
representations to be complementary (e.g., via bottleneck), then we can
reasonably assume that $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ respectively
captures complementary time-variant and time-invariant information of speech,
but it is not guaranteed that the learned time-variant and time-invariant
elements are exactly content and speaker identity. The reason is that there
are many combinations of time-variant and time-invariant information elements
other than the targeted one. For example, background noise can (mostly) be
considered as a time-invariant element, against the time-variant element of
clean speech, but the architecture alone cannot distinguish this pair of
elements from the content-speaker identity pair.

###  3.2 Learning Objective

Aside from the architectural design to facilitate the separation of time-
variant and time-invariant feature into $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$,
we also need to ensure that: 1) the information captured by $\mathbf{z_{c}}$
and $\mathbf{z_{s}}$ are complementary and 2) the complementary elements
captured are content and speaker identity instead of other element pairs – and
the key is to restrict the amounts of information of $\mathbf{x}$ that can be
captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$. To solve issue 1), we can
restrict the overall amount of information captured by $\mathbf{z_{c}}$ and
$\mathbf{z_{s}}$ together to a low level. With no redundant information
encoded, the two representations will be compact and contain complementary
information. To further tackle issue 2), it is important to restrict the
relative amounts of information captured in the two representations, such that
$\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ captures exactly the content part and
the speaker identity part of information in $\mathbf{x}$, respectively.

The information captured by a latent variable can be quantified as its MI with
the data. Following the typical practice by approximating the joint
distribution $p(\mathbf{x},\mathbf{z})$ with its variational version
$q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})$ [20, 21], we can derive the
variational MI between $\mathbf{x}$ and $\mathbf{z}$ as shown in Eqn. (3.2),
where $\mathbf{z}$ is a general variable name covering $\mathbf{z_{c}}$ and
$\mathbf{z_{s}}$.

$\displaystyle\mathbb{I}_{v}(\mathbf{x},\mathbf{z})$
$\displaystyle=\mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}\left[\log{\frac{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}{p_{d}(\mathbf{x})q_{\phi}(\mathbf{z})}}\right]$
$\displaystyle=\mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}\left[\log{\frac{q_{\phi}(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})}}+\log{\frac{p(\mathbf{z})}{q_{\phi}(\mathbf{z})}}\right]$
$\displaystyle=\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel
p(\mathbf{z})\right]\right]$
$\displaystyle-\text{KL}\left[q_{\phi}(\mathbf{z})\parallel
p(\mathbf{z})\right]$ (2)

Here $q_{\phi}(\mathbf{z})$ is the marginal distribution of $\mathbf{z}$ and
is defined as
$q_{\phi}(\mathbf{z})=\int_{\mathbf{x}}{q_{\phi}(\mathbf{z}|\mathbf{x})p_{d}(\mathbf{x})}d\mathbf{x}$.
The results in Eqn. (3.2) is different from the MI derived in [18], which
claim that
$\mathbb{I}_{v}(\mathbf{x},\mathbf{z})=\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel
p(\mathbf{z})\right]\right]$. This derivation is resulted from approximating
the marginal distribution of $\mathbf{z}$ with its prior $p(\mathbf{z})$,
which we consider is generally not applicable.

Eqn. (3.2) denotes that
$\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel
p(\mathbf{z})\right]\right]$ is an upper-bound of
$\mathbb{I}_{v}(\mathbf{x},\mathbf{z})$. In this sense, an approximate method
to restrict the amount of information of $x$ captured by the latent variable
$\mathbf{z}$ is to restrict the KL divergence term
$\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z}|\mathbf{x})\parallel
p(\mathbf{z})\right]\right]$. Though the term
$\text{KL}\left[q_{\phi}(\mathbf{z})\parallel p(\mathbf{z})\right]$ is also
penalized as a side-effect, which encourages the marginal distribution of
$\mathbf{z}$ to be more like an isotropic Gaussian and does not other harm.
Motivated by this, we introduce two weight parameters $\beta_{c}$ and
$\beta_{s}$ to the content KL and speaker KL terms, respectively, with the
goal to restrict the information that can be captured by $\mathbf{z_{c}}$ and
$\mathbf{z_{s}}$. The restricted objective function is shown in Eqn. (3),
while Eqn. (4) explicitly denotes the relationship between two KL terms and
their corresponding MI terms.

$\displaystyle\mathcal{L}_{\beta}$
$\displaystyle=-\mathbb{E}_{p_{d}(\mathbf{x})q_{\phi}(\mathbf{z_{c}}|\mathbf{x})q_{\phi}(\mathbf{z_{s}}|\mathbf{x})}\left[\log
p_{\theta}(\mathbf{x}|\mathbf{z_{c}},\mathbf{z_{s}})\right]$
$\displaystyle+\beta_{c}\cdot\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z_{c}}|\mathbf{x})\parallel
p(\mathbf{z_{c}})\right]\right]$
$\displaystyle+\beta_{s}\cdot\mathbb{E}_{p_{d}(\mathbf{x})}\left[\text{KL}\left[q_{\phi}(\mathbf{z_{s}}|\mathbf{x})\parallel
p(\mathbf{z_{s}})\right]\right]$ (3)
$\displaystyle=-\mathbb{E}_{p_{d}(\mathbf{x})q_{\phi}(\mathbf{z_{c}}|\mathbf{x})q_{\phi}(\mathbf{z_{s}}|\mathbf{x})}\left[\log
p_{\theta}(\mathbf{x}|\mathbf{z_{c}},\mathbf{z_{s}})\right]$
$\displaystyle+\beta_{c}\cdot\left[\mathbb{I}_{v}(\mathbf{x},\mathbf{z_{c}})+\text{KL}\left[q_{\phi}(\mathbf{z_{c}})\parallel
p(\mathbf{z_{c}})\right]\right]$
$\displaystyle+\beta_{s}\cdot\left[\mathbb{I}_{v}(\mathbf{x},\mathbf{z_{s}})+\text{KL}\left[q_{\phi}(\mathbf{z_{s}})\parallel
p(\mathbf{z_{s}})\right]\right]$ (4)

While the reconstruction loss in Eqn. (3) encourages both $\mathbf{z_{c}}$ and
$\mathbf{z_{s}}$ to capture as much information about $\mathbf{x}$ as
possible, the weight parameters $\beta_{c}$ and $\beta_{s}$ act as two gates
to control the amounts of information of $\mathbf{x}$ going through
$\mathbf{z_{c}}$ and $\mathbf{z_{s}}$, as indicated by Eqn. (4). The learning
objective (3) can help solve the problems of the architectural design in the
following ways: By choosing the proper absolute values of both $\beta_{c}$ and
$\beta_{s}$, the information captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$
together will be compact and thus the information contained in each latent
variable will be complementary. On the other hand, by tuning relative values
of $\beta_{c}$ and $\beta_{s}$, the amounts of information in $\mathbf{x}$
allocated to $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ will change accordingly. In
this case, we can find the optimal parameters pair that yields exactly the
separation of content and speaker identity.

Note that if we let $\beta_{c}=\beta_{s}$ and combine $\mathbf{z_{c}}$ and
$\mathbf{z_{s}}$ together, Eqn. (3) will become exactly the learning objective
of the standard $\beta$-VAE [11]. In this sense, Eqn. (3) can be considered as
a generalized form of $\beta$-VAE.

###  3.3 Dataset bias

The dataset used to train the model is also a very important inductive bias to
facilitate the learning of disentangled representations. One consideration is
that the two information elements to be disentangled should vary independently
in the dataset. Besides, other elements of variation that may interfere the
interested elements should not appear in the dataset. For example, when
disentangling the content and speaker identity elements, too much variation in
emotions may disturb the learning of speaker identity element since they are
both sequence-level elements. In our case, we combine two monolingual corpora
as the training dataset, in which the speaker and general content are the main
variations. More details are given in Section 5.1.

##  4 Implementation

The proposed model consists of a speaker encoder, a content encoder, and a
decoder, to respectively model $q_{\phi}(\mathbf{z_{s}}|\mathbf{x})$,
$q_{\phi}(\mathbf{z_{c}}|\mathbf{x})$, and
$p_{\theta}(\mathbf{x}|\mathbf{z_{c}},\mathbf{z_{s}})$ defined in Section 3.
The structure of each component as well as other implementation details are
described as follows.

Speaker encoder: The speaker encoder consists of one fully-connected (FC)
layer with 256 hidden units, whose output is activated by ReLU [22] and fed
into 4 down-sampling convolutional blocks. Each down-sampling convolutional
block includes two 1D convolution (Conv) layers with 256 filters and ReLU
activations, while the output of the second Conv layer is down-sampled by a
factor of 2 using average pooling along the time axis. A residual connection
is adopted in each convolutional block to connect the input and the output.
The kernel sizes of Conv layers in the 4 convolutional blocks are 3, 3, 5 and
5, respectively. The output of convolutional blocks is further aggregated
through global average-pooling and fed into a fully connected layer to get the
128-dimensional mean and variance vector of the speaker posterior.

Content encoder: The content encoder consists of two 1D Conv layers each with
256 hidden units and a kernel size of 3. Dropout [23] with the drop rate of
0.2 is embedded after each Conv layer. Two self-attention blocks [24] with 256
hidden units, 4 attention heads, and 1024 feed-forward network hidden
dimensions are further stacked. The output of self-attention blocks is
projected to the 128-dimensional mean and the variance for each frame.

Decoder: The decoder takes in the concatenation of the speaker representation
and the content representation to reconstruct the acoustic feature. Conv
layers and self-attention blocks with the same configurations as those in the
content encoder are first adopted. An FC layer is then used to predict the
acoustic feature. We further apply a PostNet [25] module which consists of 5
layers of 1D Conv with the kernel size of 5, to predict the residual of the
acoustic feature. Dropout with probability 0.2 is used to regularize the
PostNet. Both acoustic features predicted before and after the PostNet are
taken out for the loss computation.

Loss function: While Eqn. (3) defines the theoretical form of the learning
objective, which consists the negative log-likelihood of the conditional
distribution of the acoustic feature as the reconstruction loss. In practice,
we adopt the sum of the mean-squared error (MSE) loss and mean-absolute error
(MAE) as the reconstruction loss. Since the conditional distribution defined
by our reconstruction loss cannot be explicitly expressed, the $\beta_{c}$ and
$\beta_{s}$ values are not normalized [11] with respect to the standard form
of VAE.

Training and inference: During training, the inputs to the speaker encoder and
content encoder are from the same utterance, while the input to the speaker
encoder is firstly segmented and shuffled along the time axis before being fed
into the following network. This operation can help avoid the content
information being leaked into the speaker representation. The speaker latent
and the content latent are sampled though re-parameterization during training,
while only the mean vector and mean sequence of the two latent are used during
inference. For model optimization, we use Adam optimizer [26] with
$\beta_{1}=0.9$ and $\beta_{2}=0.999$ and $\epsilon=10^{-7}$, the learning
rate is fixed at $1.25\times 10^{-4}$. The training batch size is set as 32.
During inference, the speech from the target speaker is fed into the speaker
encoder to obtain the speaker representation, which is combined with content
extracted from the source speech in the decoder to generated the converted
speech.

Hyper-parameters tuning: The two most important hyper-parameters for the
proposed model are $\beta_{c}$ and $\beta_{s}$, which directly determines the
disentanglement performance. To be honest, the tuning of these two parameters
relies largely on trial and error, i.e., one can find the best setting through
grid search. But we do notice one simple trick that can help find the good
parameters faster. One can start with a relatively large $\beta_{c}$ (e.g.,
0.1, if the same loss function as ours is used) and a small $\beta_{s}$ (e.g.,
$10^{-3}$), this can generally yield the content and speaker disentanglement,
but the generation quality may not be that good since the large $\beta_{c}$
causes too much information loss of the content. Then we can decrease
$\beta_{c}$ gradually to find the value that yields also good generation
quality. At the same time, $\beta_{s}$ should first be decreased
proportionally with $\beta_{c}$ and then separately tuned.

##  5 Experiments

###  5.1 Dataset

We combine two openly available corpora, which are VCTK [27] and AISHELL-3
[28] together for training and evaluation of the proposed model. VCTK contains
110 speakers’ English speech data while AISHELL-3 consists of speech uttered
by 218 native Mandarin speakers. 88 speakers’ data from VCTK and 116 speakers’
data from AISHELL-3 are combined as a bilingual training set. Another 20
speakers’ data from VCTK are evenly split for validation and testing, while
for AISHELL-3 15 and 16 other speakers’ data are used so. Note that there are
no common speakers among different splits of the dataset. We down-sample all
speech into 16kHz and extract 80-dimensional logarithm Mel-Spectrograms with
200ms window length and 50ms window shift as the feature.

###  5.2 Evaluation on disentanglement

Following prior works [16, 18, 29], we conduct speaker verification (SV) on
the learned content and speaker representations and report the equal error
rate (EER) as a metric of the disentanglement. Intuitively, a high EER
produced by $\mathbf{z_{c}}$ and a low EER yielded by the $\mathbf{z_{s}}$ can
denote a good disentanglement of content and speaker representations. To
compute the EER, we randomly select 4 utterances from each speaker in the test
set as the enrolled samples, which are used to compute the speaker embedding
(by averaging their $\mathbf{z_{c}}$’s or $\mathbf{z_{c}}$’s). The remaining
utterances of the speaker are used as the positive trials while all other
speakers’ utterances are negative trials. Cosine similarity is computed as the
score. The EERs are evaluated separately for English and Mandarin test sets.

As discussed in Section 3.2, the weight parameters $\beta_{c}$ and $\beta_{s}$
restrict the respective amounts of information contained in $\mathbf{z_{c}}$
and $\mathbf{z_{s}}$, and thus are very important for disentanglement. We show
EERs with respect to $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ for both English
(EN) and Mandarin (CN) test set in Table 1, where $\beta_{c}$ varies among
$\\{10^{-3},10^{-2}\\}$ and $\beta_{s}$ is taken from
$\\{10^{-5},10^{-4},10^{-3}\\}$.

We can first observe the effect of decreasing the absolute values of both
$\beta_{c}$ and $\beta_{s}$ by comparing the results yielded by cases
$(\beta_{c},\beta_{s})=(10^{-2},10^{-4})$ and
$(\beta_{s},\beta_{c})=(10^{-3},10^{-5})$. While the ratios between
$\beta_{c}$ and $\beta_{s}$ for both cases are 100, they produce quite
different representations in terms of disentanglement. As we can observe from
results of English test set, the EERs computed using $\mathbf{z_{c}}$ and
$\mathbf{z_{s}}$ are $0.369$ and $0.115$ respectively for the former case,
while those values become $0.198$ and $0.069$ for the latter case. While the
former case shows more desirable disentanglement, the second case yields a
$\mathbf{z_{c}}$ with too much speaker information. This is because with
restrictions that are too loose on both $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$
when setting $(\beta_{s},\beta_{c})=(10^{-3},10^{-5})$, it cannot be ensured
that the information captured by $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ are
complementary. Thus, different from [18] which claims that a proper ratio
$\frac{\beta_{c}}{\beta_{s}}$ can induce the desired disentanglement, we argue
that it is also important to first set proper absolute values for $\beta_{c}$
and $\beta_{s}$, which restricts $\mathbf{z_{c}}$ and $\mathbf{z_{s}}$ to be
more complementary.

Furthermore, setting proper relative values of $\beta_{c}$ and $\beta_{s}$ is
also important for disentanglement. As can be observed from Table 1,
increasing the value of $\beta_{c}$ causes significant increases of EERs for
both EN and CN content representations and all $\beta_{s}$ values, denoting
the reduction of the speaker information captured by $\mathbf{z_{c}}$. On the
other hand, the increase of $\beta_{s}$ puts more penalization on the speaker
representation and thus allows more speaker information to leak into
$\mathbf{z_{c}}$, which is indicated by the overall declining trend of EERs
for $\mathbf{z_{c}}$ and the opposite trend for $\mathbf{z_{s}}$ horizontally.
Meanwhile, larger $\beta_{c}$ ($10^{-2}$) produces a stabler speaker-
independent $\mathbf{z_{c}}$ as the changes in EERs caused by the increasing
$\beta_{s}$ are limited.

Table 1: SV results on content and speaker representations

Rep. |  | $\beta_{s}=10^{-5}$ | $\beta_{s}=10^{-4}$ | $\beta_{s}=10^{-3}$
---|---|---|---|---
$\mathbf{z_{c}}$ (EN) $\uparrow$ | $\beta_{c}=10^{-3}$ | 0.198 | 0.179 | 0.151
$\beta_{c}=10^{-2}$ | 0.366 | 0.369 | 0.319
$\mathbf{z_{c}}$ (CN) $\uparrow$ | $\beta_{c}=10^{-3}$ | 0.253 | 0.223 | 0.155
$\beta_{c}=10^{-2}$ | 0.379 | 0.375 | 0.328
$\mathbf{z_{s}}$ (EN) $\downarrow$ | $\beta_{c}=10^{-3}$ | 0.069 | 0.131 | 0.372
$\beta_{c}=10^{-2}$ | 0.075 | 0.115 | 0.357
$\mathbf{z_{s}}$ (CN) $\downarrow$ | $\beta_{c}=10^{-3}$ | 0.061 | 0.105 | 0.273
$\beta_{c}=10^{-2}$ | 0.043 | 0.113 | 0.285

###  5.3 Evaluation on one-shot VC

We conduct subjective evaluations on both intra-lingual and cross-lingual VC.
We select 4 Mandarin speakers and 4 English speakers, both consisting of 2
male speakers and 2 female speakers. One utterance of each speaker is randomly
chosen as the source and also the reference speech. All utterances are
converted to all other speakers, thus in total we obtain 56 converted samples,
while 24 of them are intra-lingual (EN2EN and CN2CN) and 32 are cross-lingual
cases (EN2CN and CN2EN). Twelve subjects are asked to score these converted
samples based on their naturalness and speaker similarity. 5-scale mean
opinion score (MOS) is applied for both speech naturalness and speaker
similarity.

We adopt two competitive unsupervised one-shot VC models AdIN-VC [9] and
VQMIVC [8] as our baselines, while Hifi-GAN [30] is used as the vocoder. We
denote the proposed model as $\beta$-VAEVC. We train the two baseline models
as well as the vocoder on the same acoustic features and training set as ours.
The copy-synthesized speech of the source speech is included in the
naturalness evaluation, while those of randomly selected speech from the
target speakers are scored in the speaker similarity evaluation. Some samples
are available on https://beta-vaevc.github.io.

We first show the EERs obtained by content and speaker representations of
three compared models in Table 2. For $\beta$-VAEVC the $\beta_{c}$ and
$\beta_{s}$ are set as $3\times 10^{-3}$ and $10^{-7}$ respectively. However,
as shown in Table 2, this setting does not yield the best disentanglement
results in terms of EERs. As $\mathbf{z_{c}}$ of $\beta$-VAEVC seems to
contain more speaker information than two baselines. Though we can achieve
more speaker-independent $\mathbf{z_{c}}$ for $\beta$-VAEVC via further
increasing $\beta_{c}$, we find it can also decrease the generation quality.
Thus, we choose the parameter setting that works better on the validation set
in terms of speech generation quality.

Table 2: SV results of three compared models

Model |  $\mathbf{z_{c}}$ (EN) $\uparrow$ |  $\mathbf{z_{s}}$ (EN) $\downarrow$ |  $\mathbf{z_{c}}$ (CN) $\uparrow$ |  $\mathbf{z_{s}}$ (CN) $\downarrow$
---|---|---|---|---
AdIN-VC | 0.373 | 0.065 | 0.371 | 0.074
VQMIVC | 0.398 | 0.106 | 0.376 | 0.091
$\beta$-VAEVC | 0.279 | 0.054 | 0.322 | 0.053

The naturalness and similarity MOS results are shown in Table 3 and Table 4,
respectively. We can observe that the proposed method achieves overall both
better naturalness and speaker similarity than the two baselines. While AdIN-
VC works comparably well for English intra-lingual conversion, its performance
is much worse when it comes to Mandarin intra-lingual conversion and two
directions of cross-lingual conversion. While VQMIVC archives overall good
conversion naturalness, the speaker similarity performance is not as
satisfying as that for speech naturalness, especially for cross-lingual cases.

Table 3: Speech naturalness MOS results ($\pm 95\%$ CI)

Model | EN2EN | EN2CN | CN2CN | CN2EN | Overall
---|---|---|---|---|---
Hifi-GAN | 4.32±0.08 | 4.32±0.08 | 4.30±0.08 | 4.30±0.08 | 4.31±0.06
AdIN-VC | 3.41±0.15 | 2.85±0.14 | 2.94±0.17 | 2.74±0.15 | 2.96±0.08
VQMIVC | 3.56±0.15 | 3.15±0.12 | 3.27±0.13 | 3.18±0.14 | 3.27±0.07
$\beta$-VAEVC | 3.71±0.14 | 3.53±0.13 | 3.71±0.14 | 3.35±0.15 | 3.56±0.07

Table 4: Speaker similarity MOS results ($\pm 95\%$ CI)

Model | EN2EN | EN2CN | CN2CN | CN2EN | Overall
---|---|---|---|---|---
Hifi-GAN | 4.30±0.13 | 4.46±0.10 | 4.46±0.10 | 4.30±0.13 | 4.36±0.08
AdIN-VC | 3.36±0.23 | 2.83±0.18 | 2.75±0.24 | 2.91±0.21 | 2.95±0.11
VQMIVC | 3.04±0.25 | 2.60±0.19 | 3.26±0.22 | 2.55±0.20 | 2.82±0.11
$\beta$-VAEVC | 3.54±0.19 | 3.00±0.16 | 3.32±0.24 | 3.31±0.18 | 3.27±0.10

In addition, we utilize the transcription error obtained by open-source pre-
trained ASR models [31] as indicators of the conversion intelligibility. We
conduct both intra-lingual and cross-lingual VC on the whole test set, that
is, all utterances are converted to all the other speakers. There are in total
42,340 converted utterances for English intra-lingual VC, 77,920 for Mandarin
intra-lingual VC, 67,744 for English-to-Mandarin conversion and 48,700 for
Mandarin-to-English conversion. Pre-trained English and Mandarin ASR models
are used to transcribe the corresponding utterances, then the word error rate
(WER) and character error rate (CER) are computed respectively for English and
Mandarin. The recognition results on re-synthesized utterances of all samples
in the test set by Hifi-GAN are also included as reference. The results are
shown in Table 5. The proposed model surpasses baseline models by large
margins for two intra-lingual VC cases and the Mandarin-to-English conversion
case, and shows comparable performance with other two methods for the English-
to-Mandarin conversion. Besides, we notice that VQMIVC achieves better WER /
CER than AdIN-VC for cross-lingual conversion, while the latter model is
better for intra-lingual cases.

Though all compared models can realize cross-lingual VC, we can see that there
are gaps between the performance of intra-lingual VC and cross-lingual VC for
all metrics, as shown in Table 3, 4 and 5. This is reasonable since the inputs
to the content encoder and speaker encoder are from the different domain for
cross-lingual VC cases, which is not the case during training. Besides, the
difference in the recording conditions of two corpora VCTK and AISHELL-3 can
make the train-inference mismatch issue even worse. We will tackle this
problem in our future work.

Table 5: Speech recognition error results

Model | EN2EN | EN2CN | CN2CN | CN2EN
---|---|---|---|---
Hifi-GAN | 5.11% | 2.70%
AdIN-VC | 30.61% | 45.88% | 23.01% | 51.31%
VQMIVC | 35.09% | 43.60% | 33.37% | 44.70%
$\beta$-VAEVC | 23.41% | 46.33% | 10.58% | 32.24%

##  6 Conclusion

We propose a method to disentangle speech into content and speaker
representations, which can be applied to the challenging task of one-shot
cross-lingual VC. Our method is based on a VAE with two encoders to extract
speaker and content representations respectively. With the speaker encoder
compressing the whole speech into a single vector and the content encoder
extracting the frame-level representation out of speech, the time-variant and
time-invariant elements of speech can be more easily separated into two
representations. Furthermore, inspired by $\beta$-VAE, we propose a learning
objective that incorporates two weight parameters to restrict the amount of
information that can be captured by the two representations. With proper
weight parameters imposed, the disentanglement can be ensured to be with
respect to content and speaker information. We apply the proposed method to
one-shot cross-lingual VC, through which we show the effectiveness of the
proposed method in achieving content and speaker disentanglement.

##  7 ACKNOWLEDGMENTS

This research is supported by the Centre for Perceptual and Interactive
Intelligence, a CUHK InnoCentre.

## References

  * [1] Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li,  “An overview of voice conversion and its challenges: From statistical modeling to deep learning,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 132–157, 2020.
  * [2] Yi Zhou, Xiaohai Tian, Haihua Xu, Rohan Kumar Das, and Haizhou Li,  “Cross-lingual voice conversion with bilingual phonetic posteriorgram and average modeling,”  in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 6790–6794.
  * [3] Diederik P. Kingma and Max Welling,  “Auto-encoding variational bayes,”  in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Yoshua Bengio and Yann LeCun, Eds., 2014.
  * [4] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson,  “Autovc: Zero-shot voice style transfer with only autoencoder loss,”  in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Kamalika Chaudhuri and Ruslan Salakhutdinov, Eds. 2019, vol. 97 of Proceedings of Machine Learning Research, pp. 5210–5219, PMLR.
  * [5] Da-Yi Wu and Hung-Yi Lee,  “One-shot voice conversion by vector quantization,”  in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020. 2020, pp. 7734–7738, IEEE.
  * [6] Da-Yi Wu, Yen-Hao Chen, and Hung-yi Lee,  “VQVC+: one-shot voice conversion by vector quantization and u-net architecture,”  in Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, Helen Meng, Bo Xu, and Thomas Fang Zheng, Eds. 2020, pp. 4691–4695, ISCA.
  * [7] Siyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao, Zhe Gan, and Lawrence Carin,  “Improving zero-shot voice style transfer via disentangled representation learning,”  in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. 2021, OpenReview.net.
  * [8] Disong Wang, Liqun Deng, Yu Ting Yeung, Xiao Chen, Xunying Liu, and Helen Meng,  “VQMIVC: vector quantization and mutual information-based unsupervised speech representation disentanglement for one-shot voice conversion,”  in Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021, Hynek Hermansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlícek, Eds. 2021, pp. 1344–1348, ISCA.
  * [9] Ju-Chieh Chou and Hung-yi Lee,  “One-shot voice conversion by separating speaker and content representations with instance normalization,”  in Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, Gernot Kubin and Zdravko Kacic, Eds. 2019, pp. 664–668, ISCA.
  * [10] Yen-Hao Chen, Da-Yi Wu, Tsung-Han Wu, and Hung-yi Lee,  “Again-vc: A one-shot voice conversion using activation guidance and adaptive instance normalization,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 5954–5958.
  * [11] Irina Higgins, Loïc Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner,  “beta-vae: Learning basic visual concepts with a constrained variational framework,”  in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017, OpenReview.net.
  * [12] Christopher P. Burgess, Irina Higgins, Arka Pal, Loïc Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner,  “Understanding disentangling in $\beta$-vae,”  CoRR, vol. abs/1804.03599, 2018.
  * [13] Lifa Sun, Kun Li, Hao Wang, Shiyin Kang, and Helen Meng,  “Phonetic posteriorgrams for be-to-one voice conversion without parallel data training,”  in 2016 IEEE International Conference on Multimedia and Expo (ICME), 2016, pp. 1–6.
  * [14] Yi Zhou, Xiaohai Tian, Emre Yilmaz, Rohan Kumar Das, and Haizhou Li,  “A modularized neural network with language-specific output layers for cross-lingual voice conversion,”  2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 160–167, 2019.
  * [15] Shengkui Zhao, Hao Wang, Trung Hieu Nguyen, and Bin Ma,  “Towards natural and controllable cross-lingual voice conversion based on neural tts model and phonetic posteriorgram,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 5969–5973.
  * [16] Wei-Ning Hsu, Yu Zhang, and James R. Glass,  “Unsupervised learning of disentangled and interpretable representations from sequential data,”  in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, Eds., 2017, pp. 1878–1889.
  * [17] Seyed Hamidreza Mohammadi and Taehwan Kim,  “Investigation of using disentangled and interpretable representations for one-shot cross-lingual voice conversion,”  in INTERSPEECH, 2018.
  * [18] Jiachen Lian, Chunlei Zhang, and Dong Yu,  “Robust disentangled variational speech representation learning for zero-shot voice conversion,”  in IEEE ICASSP. IEEE, 2022.
  * [19] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem,  “Challenging common assumptions in the unsupervised learning of disentangled representations,”  in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Kamalika Chaudhuri and Ruslan Salakhutdinov, Eds. 2019, vol. 97 of Proceedings of Machine Learning Research, pp. 4114–4124, PMLR.
  * [20] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud,  “Isolating sources of disentanglement in variational autoencoders,”  Advances in neural information processing systems, vol. 31, 2018\.
  * [21] Hyunjik Kim and Andriy Mnih,  “Disentangling by factorising,”  in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, Jennifer G. Dy and Andreas Krause, Eds. 2018, vol. 80 of Proceedings of Machine Learning Research, pp. 2654–2663, PMLR.
  * [22] Abien Fred Agarap,  “Deep learning using rectified linear units (relu),”  arXiv preprint arXiv:1803.08375, 2018.
  * [23] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov,  “Dropout: a simple way to prevent neural networks from overfitting,”  The journal of machine learning research, vol. 15, no. 1, pp. 1929–1958, 2014.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [25] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.,  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 4779–4783.
  * [26] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  CoRR, vol. abs/1412.6980, 2015.
  * [27] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald,  “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),”  2019\.
  * [28] Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li,  “AISHELL-3: A Multi-Speaker Mandarin TTS Corpus,”  in Proc. Interspeech 2021, 2021, pp. 2756–2760.
  * [29] Yingzhen Li and Stephan Mandt,  “Disentangled sequential autoencoder,”  in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, Jennifer G. Dy and Andreas Krause, Eds. 2018, vol. 80 of Proceedings of Machine Learning Research, pp. 5656–5665, PMLR.
  * [30] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae,  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,”  Advances in Neural Information Processing Systems, vol. 33, pp. 17022–17033, 2020.
  * [31] Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al.,  “Nemo: a toolkit for building ai applications using neural modules,”  arXiv preprint arXiv:1909.09577, 2019.


# Operational advantage of quantum resources in a semi-device independent
framework

Shubhayan Sarkar<EMAIL_ADDRESS>Laboratoire d’Information Quantique,
Université libre de Bruxelles (ULB), Av. F. D. Roosevelt 50, 1050 Bruxelles,
Belgium  Chandan Datta<EMAIL_ADDRESS>Institute for Theoretical Physics
III, Heinrich Heine University Düsseldorf, Universitätsstraße 1, D-40225
Düsseldorf, Germany  Department of Physics, Indian Institute of Technology
Jodhpur, Jodhpur 342030, India

###### Abstract

Quantum resources are fundamental for the development of quantum technology.
Thus, verifying their presence is crucial for studying their role in various
information-processing tasks. Here, we enquire whether can one detect the
presence of a quantum resource in some operational task or equivalently does
every quantum resource provides an advantage over its free counterpart in some
black box scenarios where one does not have much information about the
devices. We introduce the framework for detecting quantum resources semi-
device independently by considering the prepare-and-operation scenario with
the restriction on the dimension of the quantum channel connecting the
preparation box with the operation box. For any dimension $d$, we show that
for any resource theory with less than $d^{2}$ number of linearly independent
free states or free operations, there exist correlations that can detect the
presence of a quantum resource. For the particular case when the quantum
channel is constrained to transmit only qubits, we explicitly construct
witnesses to observe the presence of various quantum resources.

Introduction– With the emergence of quantum technologies in the last few
years, it has become clear that entanglement is an indispensable resource for
a range of intriguing technological applications that cannot be matched by
classical means [1]. The applications of entanglement are practically endless
[1], spanning from the instantaneous transfer of an unknown quantum bit to a
remote observer [2], to the secure distribution of cryptographic keys among
several parties [3]. However, it has become clear in recent years that
entanglement is not the only resource for quantum technological applications,
and there are other applications that rely on other features of quantum
systems, such as coherence [4, 5], imaginarity [6, 7, 8, 9], asymmetry [10],
contextuality [11, 12, 13], purity [14, 15, 16], and so on.

To investigate the roles of these resources in a wide range of information
processing tasks, we must first understand their characteristics. Quantum
resource theories [17] proposed in quantum information science provide a
unified approach for studying all such resources, as well as their roles and
limitations in various quantum technological applications. While the
exploration of entanglement characterisation began in the early 1990s [2, 18,
19, 20, 1], it was not until 2008 when Gour and Spekkens introduced the
concept of resource theory and established a comprehensive framework in their
investigation of the resource theory of asymmetry [10]. Soon after, a number
of additional resource theories—including coherence [4, 5], imaginarity [6, 7,
8, 9], purity [14, 15, 16], stabiliser quantum computation [21], and numerous
others—were introduced in the literature [17]. As a consequence, a generic
framework for quantum resource theories has been established [22, 23, 17].

After identifying all of these resources, it is critical to observe their
actual presence in practical scenarios. One of the most common methods for
observing this is state tomography. However, in general, this method is not
economical as it involves reconstructing the complete information about the
given quantum state. Moreover, one also has to trust their measurement device.
A similar problem was considered in [24] where it was shown that every convex
resource theory is useful for a subchannel discrimination task. This result
was then generalised to dynamical resource theories [25], generalised
probabilistic theories [26], and non-convex resource theories [27]. However,
to show that a given quantum state is resourceful, one has to trust the
subchannels along with the quantum state.

In this work, we pose a stronger question. Can quantum states or operations be
detected to be resourceful without trusting them or any other operation
involved in the test? For instance, violation of Bell inequalities serves as a
witness for entanglement detection [28] where one does not trust any state or
measurements. Similarly, assuming multiple independent sources, it was shown
in [29] that one can detect whether any of the quantum states or measurements
need to be imaginary, thus providing a way to detect imaginarity without
trusting any state or measurements.

One of the well-known semi-device independent scenarios is prepare-and-measure
(PM), where one just assumes an upper bound on the dimension of the quantum
channel connecting the preparation and measurement [30]. In a practical
scenario, it can be imagined as a communication task involving two separate
boxes, a preparation box that is responsible for creating quantum states,
which are then transmitted to a measurement box. A fascinating fact of this
scenario is that we can perceive those boxes as black boxes, so removing the
need for trust in them. The prepare-and-measure scenario has been employed in
the literature for a wide range of applications, including random access codes
[31, 32], informational principles [33, 34], key distribution [35, 36],
dimension witness [30, 37], self-testing [38, 39], and more.

As we are concerned here with detecting resourceful states and operations, we
generalise the PM scenario to the prepare-and-operation (PO) scenario where
instead of a measurement box we have an operation box that can perform some
quantum operations on the incoming states. Restricting the quantum channel to
transmit $d-$dimensional states, we then show that for any resource theory
with strictly less than $d^{2}$ number of linearly independent free states or
free operations, one can always detect whether the preparation box produces at
least one resource state and the operation box performs at least one
resourceful operation. Then for the particular case, when the quantum channel
can only transmit qubits, we consider various resource theories and construct
explicit linear witnesses to detect them.

Quantum resource theories– Any quantum resource theory is built on two basic
components: a set of free states and a set of free operations [17]. The
quantum states that do not belong to the set of free states are referred to as
resource states. Any resource theory must adhere to the fundamental constraint
that the set of free operations must not be able to generate resource states
from the set of free states. Hence, quantum resource theories allow us to
study quantum information processing tasks under a restricted set of
operations. As an example, in the resource theory of entanglement, separable
states denote free states, local operations and classical communications
(LOCC) denote free operations, and information processing tasks are studied
under LOCC [1].

Let us now restrict to states and operations acting on $\mathbb{C}^{d}$ for
any $d\geq 2$, where $d$ is the dimension of the Hilbert space. The free
states are denoted as $\sigma_{d}$ and the collection of these states is
denoted by the set $\mathcal{S}_{d}$. Now, consider a set of linearly
independent density matrices
$|\psi_{\sigma_{d},i}\rangle\\!\\!\langle\psi_{\sigma_{d},i}|\in\mathcal{S}_{d}$
using which any state $\sigma_{d}$ can be written via their linear
combination. Equivalently, the elements of the set
$\mathcal{S}_{\mathrm{ext},d}=\\{|\psi_{\sigma_{d},i}\rangle\\!\\!\langle\psi_{\sigma_{d},i}|\\}_{i}$
forms a basis of $\mathcal{S}_{d}$. However, it is important to note here that
$\mathcal{S}_{d}$ might not contain all the density matrices which can be
expressed as linear combinations of
$|\psi_{\sigma_{d},i}\rangle\\!\\!\langle\psi_{\sigma_{d},i}|$. Moreover, it
is also important to realise that not every linear combination of these states
would result in a valid density matrix. Furthermore, the set
$\mathcal{S}_{\mathrm{ext},d}$ might not be unique. Let us now denote the
cardinality of $\mathcal{S}_{\mathrm{ext},d}$ as
$n(\mathcal{S}_{\mathrm{ext},d})$. For instance, in the resource theory of
coherence [5], the set $\mathcal{S}_{\mathrm{ext},d}$ is given by
$\mathcal{S}_{\mathrm{ext},d}=\\{|0\rangle\\!\\!\langle
0|,|1\rangle\\!\\!\langle 1|,\ldots,|d-1\rangle\\!\\!\langle d-1|\\}$ and thus
$n(\mathcal{S}_{\mathrm{ext},d})=d$. Similarly, for the resource theory of
athermality or asymmetry [40, 10], we have that
$n(\mathcal{S}_{\mathrm{ext},d})\leq(d-1)^{2}+1$. Likewise, for the resource
theory of magic or imaginarity in dimension two [21, 7], we have that
$n(\mathcal{S}_{\mathrm{ext},2})=3$. For any dimension $d$, the maximum
cardinality of this set in quantum theory is
$n_{\mathrm{max}}(\mathcal{S}_{\mathrm{ext},d})=d^{2}$ [41].

Similarly, one can also characterize free operations $\delta_{d}$ based on the
above procedure. Equivalently, we express the operation $\delta_{d}$ using its
Kraus decomposition as
$\delta_{d}\equiv\\{K_{j,\delta_{d}}^{\dagger}K_{j,\delta_{d}}\\}$ which might
not be unique. Let us denote the set of these free operations as
$\mathcal{O}_{d}$ which is composed of all possible Kraus decompositions of
every free operation acting on $\mathbb{C}^{d}$. Let us now consider a set of
linearly independent positive operators
$|\psi_{\delta_{d},i}\rangle\\!\\!\langle\psi_{\delta_{d},i}|\in\mathcal{O}_{d}$
such that they form a basis of $\mathcal{O}_{d}$. Let us denote the set of
these linearly independent operators as
$\mathcal{O}_{\mathrm{ext},d}=\\{|\psi_{\delta_{d},i}\rangle\\!\\!\langle\psi_{\delta_{d},i}|\\}_{i}$.
The cardinality of $\mathcal{O}_{\mathrm{ext},d}$ is denoted as
$n(\mathcal{O}_{\mathrm{ext},d})$. Any operator in $\mathcal{O}_{d}$ can be
expressed as a linear combination of elements in
$\mathcal{O}_{\mathrm{ext},d}$. For instance, in coherence resource theory
$n(\mathcal{O}_{\mathrm{ext},d})=d$ [5], for the resource theory of asymmetry
$n(\mathcal{O}_{\mathrm{ext},d})\leq(d-1)^{2}+1$ [10], and for imaginarity in
dimension two [7], $n(\mathcal{O}_{\mathrm{ext},d})=3$. Again, in quantum
theory for any dimension
$d:n_{\mathrm{max}}(\mathcal{O}_{\mathrm{ext},d})=d^{2}$. Any resource theory
defined on a $d-$dimensional Hilbert space, from now on will be denoted as
$\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$.

Figure 1:  We consider two parties, a producer and an observer, each situated
in spatially separated labs. They can communicate solely via a quantum channel
supporting a quantum system of dimension $d$. The producer freely selects $y$
inputs to prepare quantum states $\rho_{y}$, which are then sent to the
observer. The observer, in turn, selects $x$ inputs to apply quantum
operations on the received states, yielding outcomes $j$. After all the runs
of the experiment, one obtains the probability distribution $\\{p(j|x,y)\\}$

.

Prepare-and-operation scenario– We consider here a prepare-and-operation (PO)
scenario with two parties, producer and observer in two different spatially
separated labs who can communicate only via a quantum channel that can support
a quantum system bounded by the dimension $d$. The producer can freely choose
$y$ inputs based on which it prepares quantum states $\rho_{y}$ and sends it
to the observer. The observer now freely chooses $x$ inputs based on which
quantum operations are applied on the incoming quantum state to obtain the
outcomes $j$. This is illustrated in Fig. 1.

From the above setup, one obtains the probability distribution
$\vec{p}=\\{p(j|x,y)\\}$ where $p(j|x,y)$ denotes the probability of obtaining
outcome $j$ by the operation box given the inputs $x,y$. In quantum theory,
the inputs $y$ generate a qudit state $\rho_{y}$ acts on $\mathbb{C}^{d}$ and
the operation box does the operation denoted by the Kraus operators as
$\mathbb{O}_{x}=\\{K_{j,x}\\}$ such that they act on $\mathbb{C}^{d}$. Now,
the probabilities $p(j|x,y)$ are given by

$p(j|x,y)=\mathrm{Tr}(K_{j,x}^{\dagger}K_{j,x}\rho_{y}).$ (1)

Notice that in the above description, we treat operations in the same way as
measurements in quantum theory. In general, it may not be possible to obtain
these probabilities in a practical setting. However, we show below that
treating operations similar to measurement allows us to detect the presence of
a quantum resource in a semi-device-independent way. To detect resources in an
experiment, one can restrict to measurements and all below analysis follows
straightaway. Furthermore, the correlations $\vec{p}$ in general are not
convex due to the dimension constraint. Consequently, finding general results
for semi-device independent schemes is highly non-trivial.

Results– Using the above framework, let us now establish two no-go theorems
that apply to a large class of quantum resource theories. The first one
concerns the case when the set of linearly independent free states is strictly
less than $d^{2}$.

######  Theorem 1.

Consider a resource theory $\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$ such that
the cardinality of the $\mathcal{S}_{\mathrm{ext},d}$ is $N$ with $N<d^{2}$.
Consider now the PO scenario described above with $y=0,1,\ldots,N$ and
$x=0,1,\ldots,d^{2}-1$ and any $j\geq 2$ such that one obtains
$\vec{p}=\\{p(j|x,y)\\}$. Then, there exists at least one probability
distribution $\vec{p}_{Q}$ in quantum theory that can not be realized by free
states in $\mathcal{S}_{d}$.

The proof of the above theorem can be found in Appendix A. Let us extend the
above theorem for resource theories with the number of linearly independent
free operations strictly less than $d^{2}$.

######  Theorem 2.

Consider a resource theory $\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$ such that
the cardinality of the $\mathcal{O}_{\mathrm{ext},d}$ is $N$ such that
$N<d^{2}$. Consider now the PO scenario described above with
$y=0,1,\ldots,d^{2}-1$ and $x=0,1,\ldots,N$ and any $j\geq 2$ such that one
obtains $\vec{p}=\\{p(j|x,y)\\}$. Then, there exists at least one probability
distribution $\vec{p}_{Q}$ in quantum theory that can not be realized by free
operations in $\mathcal{O}_{d}$.

We refer to Appendix A for the proof of the aforementioned theorem. Theorem 1
shows that one can always detect the presence of a resource state
corresponding to the resource theory $\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$
in the PO scenario if $n(\mathcal{S}_{\mathrm{ext},d})<d^{2}$. Similarly,
Theorem 2 shows that one can always detect the presence of a resource
operation corresponding to the resource theory
$\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$ in the PO scenario if
$n(\mathcal{O}_{\mathrm{ext},d})<d^{2}$. One can straightaway infer from the
above results that if one wants to detect the presence of a resource state as
well as operation simultaneously, then one needs to consider the PO scenario
with $y,x=0,1,\ldots,d^{2}-1$. Thus, we obtain the following corollary of
Theorems 1 and 2.

######  Corollary 1.

Consider a resource theory $\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$ such that
the cardinality of the $\mathcal{S}_{\mathrm{ext},d}$ and
$\mathcal{O}_{\mathrm{ext},d}$ is less than $d^{2}$. Consider now the PO
scenario described above with $x,y=0,1,\ldots,d^{2}-1$ and any $j\geq 2$ such
that one obtains $\vec{p}=\\{p(j|x,y)\\}$. Then, there exists at least one
probability distribution $\vec{p}_{Q}$ in quantum theory that can not be
realized by either the free states in $\mathcal{S}_{d}$ or operations in
$\mathcal{O}_{d}$. Thus, observing $\vec{p}_{Q}$ allows one to conclude that
at least one of the preparations and operations is resourceful.

Let us remark here that the number of inputs $x,y$ in the above corollary
might not be optimal. As we will see below for particular resource theories,
there might also exist probability distributions with a lower number of inputs
$x,y$ such that at least one of the preparations and operations is
resourceful. For this purpose, we construct witnesses which is a function of
the probability distribution $\vec{p}$ of the form

$\displaystyle
W_{\mathcal{S}_{d},\mathcal{O}_{d}}=f(\vec{p})\leq\beta_{\mathcal{S}_{d},\mathcal{O}_{d}}$
(2)

where $\beta_{\mathcal{S}_{d},\mathcal{O}_{d}}$ is the maximum value
attainable using the free states and operations in the resource theory
$\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$. Consequently, the presence of a
resource can be verified if the above bound is violated. Furthermore, from a
practical perspective it is usually beneficial to construct witnesses as one
can never exactly obtain the theorised correlations in an experiment and
witnesses provide a gap that can tolerate such errors.

Witnesses– Let us now construct witnesses tailored to specific resource
theories that can detect the presence of concerned quantum resources that
provide a considerable gap between the maximum value obtainable using free
states and operations to quantum theory. In this work, we focus on the PO
scenario where the dimension of the quantum system going through the channel
is bounded by two.

In the first example, we construct a witness for the resource theory of
coherence [4, 5]. In this theory free states and free operations are known as
incoherent states and incoherent operations, respectively. Here, we consider
the PO scenario with $x=0,1$ and $y=0,1,2$. In this scenario, the witness to
observe coherence can be constructed as

$\displaystyle W_{C}=p(0|0,0)+p(0|0,1)$ $\displaystyle+$ $\displaystyle
p(0|1,0)+p(1|1,1)$ (3) $\displaystyle+$ $\displaystyle p(1|0,2)\leq\beta_{C},$

where $\beta_{C}$ can be evaluated by considering incoherent states and
operations [5]. To find $\beta_{C}$, we utilise the fact that any incoherent
state and Kraus operator of an incoherent operation acting on two-dimensional
Hilbert space can be expressed as a convex combination of
$\\{|0\rangle\\!\\!\langle 0|,|1\rangle\\!\\!\langle 1|\\}$. Consequently, we
find that $\beta_{C}=4$. Any violation of this bound would imply that at least
one of the states and operations is coherent. Now, consider the following
states,

$\displaystyle\rho_{0}=|0\rangle\\!\\!\langle
0|,\qquad\rho_{1}=|+\rangle\\!\\!\langle+|,\qquad\rho_{2}=|\overline{0}\rangle\\!\\!\langle\overline{0}|,$
(4)

and measurements,

$\displaystyle K_{0,0}$ $\displaystyle=$
$\displaystyle|\overline{0}\rangle\\!\\!\langle\overline{0}|,\qquad
K_{1,0}=|\overline{1}\rangle\\!\\!\langle\overline{1}|,$ $\displaystyle
K_{0,1}$ $\displaystyle=$
$\displaystyle|\overline{+}\rangle\\!\\!\langle\overline{+}|,\qquad
K_{1,0}=|\overline{-}\rangle\\!\\!\langle\overline{-}|,$ (5)

where $\\{|\overline{0}\rangle,|\overline{1}\rangle\\}$ are the eigenvectors
of $\frac{\sigma_{x}+\sigma_{z}}{\sqrt{2}}$ and
$\\{|\overline{+}\rangle,|\overline{-}\rangle\\}$ are the eigenvectors of
$\frac{\sigma_{x}-\sigma_{z}}{\sqrt{2}}$. Here, $\sigma_{x}$ and $\sigma_{z}$
denote respective Pauli matrices. Then, we obtain that
$W_{C}=1+4\cos^{2}(\pi/8)=3+\sqrt{2}$, which violates the upper bound, i.e.,
$\beta_{C}=4$. Since the resource theories of asymmetry [10] and athermality
[40, 42] are equivalent to the coherence theory [4, 5] for $d=2$, the same
finding can be readily applied to both theories.

In the next example, we consider resource theory of imaginarity [7, 8] and
construct a witness for the detection of imaginarity. Again, we take the PO
scenario with $x=0,1,2$ and $y=0,1,2,3$, and the witness to observe
imaginarity can be designed as

$\displaystyle W_{I}=W_{C}+p(0|2,3)-|p(0|2,0)-p(1|2,0)|$
$\displaystyle-|p(0|2,1)-p(1|2,1)|\leq\beta_{I}.$ (6)

where $\beta_{I}$ is the maximal value attainable considering real states and
operations [7]. Consequently, to evaluate $\beta_{I}$, we recall that any
state $\rho\in\mathbb{C}^{2}$ can be expressed as
$\rho=(\mathbbm{1}+r_{x}\sigma_{x}+r_{y}\sigma_{y}+r_{z}\sigma_{z})/2$, where
$\sigma_{i}$ are Pauli matrices and $r_{i}$’s are components of the Bloch
vector $\vec{r}$. Since in the resource theory of imaginarity free states
satisfy $\rho=\rho^{*}$, one can readily obtain that any free state can be
expressed as $\rho=1/2(\mathbbm{1}+r_{z}\sigma_{z}+r_{x}\sigma_{x})$.
Similarly, for the Kraus operators, we have
$K_{j}=m_{0}\mathbbm{1}+m_{z}\sigma_{z}+m_{x}\sigma_{x}$. Using these
representations we evaluate that $\beta_{I}\leq 5$. Any violation of this
bound would imply that at least one of the states and measurements must be
complex. Now, consider the following states

$\displaystyle\rho_{0}$ $\displaystyle=$ $\displaystyle|0\rangle\\!\\!\langle
0|,\qquad\rho_{1}=|+\rangle\\!\\!\langle+|,$ $\displaystyle\rho_{2}$
$\displaystyle=$
$\displaystyle|\overline{0}\rangle\\!\\!\langle\overline{0}|,\qquad\rho_{3}=|+_{y}\rangle\\!\\!\langle+_{y}|,$
(7)

and measurements

$\displaystyle K_{0,0}$ $\displaystyle=$
$\displaystyle|\overline{0}\rangle\\!\\!\langle\overline{0}|,\quad
K_{1,0}=|\overline{1}\rangle\\!\\!\langle\overline{1}|,\quad
K_{0,1}=|\overline{+}\rangle\\!\\!\langle\overline{+}|,$ $\displaystyle
K_{1,0}$ $\displaystyle=$
$\displaystyle|\overline{-}\rangle\\!\\!\langle\overline{-}|,\quad
K_{0,2}=|+_{y}\rangle\\!\\!\langle+_{y}|,\quad
K_{1,2}=|-_{y}\rangle\\!\\!\langle-_{y}|,\ \ \ \ $ (8)

where $|\pm_{y}\rangle=1/\sqrt{2}(|0\rangle\pm i|1\rangle)$. Using the above
states and operations one can obtain $W_{I}=4+\sqrt{2}$, which violates the
upper bound given in Eq. (Operational advantage of quantum resources in a
semi-device independent framework). In Appendix B, we also find specific
witnesses for resource theory of purity and magic.

Discussions– In the above work, we extended the prepare-and-measure scenario
to a prepare-and-operation scenario by introducing an operation box that
performs quantum operations on incoming states. By limiting the quantum
channel to transmit $d$-dimensional states, we showed that for any resource
theory with fewer than $d^{2}$ linearly independent free states or operations,
it is always possible to detect if the preparation box produces at least one
resource state and if the operation box performs at least one resourceful
operation. When the channel is bounded to transmit qubits, we then constructed
explicit linear witnesses for the resource theory of coherence, imaginarity,
purity and magic. While our focus in this letter has been on single-partite
systems, it is worth noting that our method can be readily extended to
multipartite systems, thus encompassing the resource theory of entanglement.

A practical application of our results can be illustrated in the following
scenario: Imagine an experimental lab intending to purchase a preparation box
guaranteed to produce at least one resource state, and an operation box that
performs at least one resourceful operation. Suppose the experimenter has
access to a quantum channel capable of supporting at max qudit systems. In
that case, they can verify the capabilities of these boxes without needing to
trust them or have any prior knowledge about their workings.

Our work opens up various interesting questions in the PO scenario. As
previously mentioned in Fig. 1, our objective was to confirm the presence of
resources within the two boxes. Now, consider a situation in which the boxes
are intended to generate resource states and operations only. It would be
interesting to verify if the PO scenario can confirm whether the boxes are
functioning as intended, i.e., they are not producing any free state or
operation. The following is another intriguing follow-up question to consider:
Consider now a theory in dimension $d$ where free states are given by the
convex combination of $d^{2}$ linearly independent vectors. An instance of
such linearly independent vectors is provided in [41]. Let us for the time
being assume that it is a valid resource theory. Even if such a set of free
states would not include every quantum state in dimension $d$, it is not clear
whether such a resource would be detected using the above results in the PO
scenario.

Acknowledgements— We thank Alexander Streltsov for his useful insights. S.S.
acknowledges the QuantERA II Programme (VERIqTAS project) that has received
funding from the European Union’s Horizon 2020 research and innovation
programme under Grant Agreement No 101017733.

## References

  * Horodecki \͡mathrm{missing}extbfet al. [2009] R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Quantum entanglement, Rev. Mod. Phys. 81, 865 (2009).
  * Bennett \͡mathrm{missing}extbfet al. [1993] C. H. Bennett, G. Brassard, C. Crépeau, R. Jozsa, A. Peres, and W. K. Wootters, Teleporting an unknown quantum state via dual classical and einstein-podolsky-rosen channels, Phys. Rev. Lett. 70, 1895 (1993).
  * Ekert [1991] A. K. Ekert, Quantum cryptography based on bell’s theorem, Phys. Rev. Lett. 67, 661 (1991).
  * Baumgratz \͡mathrm{missing}extbfet al. [2014] T. Baumgratz, M. Cramer, and M. B. Plenio, Quantifying coherence, Phys. Rev. Lett. 113, 140401 (2014).
  * Streltsov \͡mathrm{missing}extbfet al. [2017] A. Streltsov, G. Adesso, and M. B. Plenio, Colloquium: Quantum coherence as a resource, Rev. Mod. Phys. 89, 041003 (2017).
  * Hickey and Gour [2018] A. Hickey and G. Gour, Quantifying the imaginarity of quantum mechanics, J. Phys. A: Math. Theor. 51, 414009 (2018).
  * Wu \͡mathrm{missing}extbfet al. [2021a] K.-D. Wu, T. V. Kondra, S. Rana, C. M. Scandolo, G.-Y. Xiang, C.-F. Li, G.-C. Guo, and A. Streltsov, Operational resource theory of imaginarity, Phys. Rev. Lett. 126, 090401 (2021a).
  * Wu \͡mathrm{missing}extbfet al. [2021b] K.-D. Wu, T. V. Kondra, S. Rana, C. M. Scandolo, G.-Y. Xiang, C.-F. Li, G.-C. Guo, and A. Streltsov, Resource theory of imaginarity: Quantification and state conversion, Phys. Rev. A 103, 032401 (2021b).
  * Kondra \͡mathrm{missing}extbfet al. [2023] T. V. Kondra, C. Datta, and A. Streltsov, Real quantum operations and state transformations, New J. Phys. 25, 093043 (2023).
  * Gour and Spekkens [2008] G. Gour and R. W. Spekkens, The resource theory of quantum reference frames: manipulations and monotones, New J. Phys. 10, 033023 (2008).
  * Kochen and Specker [1967] S. Kochen and E. P. Specker, The problem of hidden variables in quantum mechanics, Journal of Mathematics and Mechanics 17, 59 (1967).
  * Howard \͡mathrm{missing}extbfet al. [2014] M. Howard, J. Wallman, V. Veitch, and J. Emerson, Contextuality supplies the ‘magic’ for quantum computation, Nature 510, 351 (2014).
  * Budroni \͡mathrm{missing}extbfet al. [2022] C. Budroni, A. Cabello, O. Gühne, M. Kleinmann, and J.-A. Larsson, Kochen-specker contextuality, Rev. Mod. Phys. 94, 045007 (2022).
  * Horodecki \͡mathrm{missing}extbfet al. [2003] M. Horodecki, P. Horodecki, and J. Oppenheim, Reversible transformations from pure to mixed states and the unique measure of information, Phys. Rev. A 67, 062104 (2003).
  * Gour \͡mathrm{missing}extbfet al. [2015] G. Gour, M. P. Müller, V. Narasimhachar, R. W. Spekkens, and N. Yunger Halpern, The resource theory of informational nonequilibrium in thermodynamics, Physics Reports 583, 1 (2015).
  * Streltsov \͡mathrm{missing}extbfet al. [2018] A. Streltsov, H. Kampermann, S. Wölk, M. Gessner, and D. Bruß, Maximal coherence and the resource theory of purity, New J. Phys. 20, 053058 (2018).
  * Chitambar and Gour [2019] E. Chitambar and G. Gour, Quantum resource theories, Rev. Mod. Phys. 91, 025001 (2019).
  * Bennett \͡mathrm{missing}extbfet al. [1996] C. H. Bennett, H. J. Bernstein, S. Popescu, and B. Schumacher, Concentrating partial entanglement by local operations, Phys. Rev. A 53, 2046 (1996).
  * Vedral \͡mathrm{missing}extbfet al. [1997] V. Vedral, M. B. Plenio, M. A. Rippin, and P. L. Knight, Quantifying entanglement, Phys. Rev. Lett. 78, 2275 (1997).
  * Nielsen [1999] M. A. Nielsen, Conditions for a class of entanglement transformations, Phys. Rev. Lett. 83, 436 (1999).
  * Veitch \͡mathrm{missing}extbfet al. [2014] V. Veitch, S. A. Hamed Mousavian, D. Gottesman, and J. Emerson, The resource theory of stabilizer quantum computation, New J. Phys. 16, 013009 (2014).
  * Coecke \͡mathrm{missing}extbfet al. [2016] B. Coecke, T. Fritz, and R. W. Spekkens, A mathematical theory of resources, Information and Computation 250, 59 (2016).
  * Fritz [2017] T. Fritz, Resource convertibility and ordered commutative monoids, Mathematical Structures in Computer Science 27, 850–938 (2017).
  * Takagi \͡mathrm{missing}extbfet al. [2019] R. Takagi, B. Regula, K. Bu, Z.-W. Liu, and G. Adesso, Operational advantage of quantum resources in subchannel discrimination, Phys. Rev. Lett. 122, 140402 (2019).
  * Uola \͡mathrm{missing}extbfet al. [2019] R. Uola, T. Kraft, J. Shang, X.-D. Yu, and O. Gühne, Quantifying quantum resources with conic programming, Phys. Rev. Lett. 122, 130404 (2019).
  * Takagi and Regula [2019] R. Takagi and B. Regula, General resource theories in quantum mechanics and beyond: Operational characterization via discrimination tasks, Phys. Rev. X 9, 031053 (2019).
  * Kuroiwa \͡mathrm{missing}extbfet al. [2024] K. Kuroiwa, R. Takagi, G. Adesso, and H. Yamasaki, Every quantum helps: Operational advantage of quantum resources beyond convexity, Phys. Rev. Lett. 132, 150201 (2024).
  * Brunner \͡mathrm{missing}extbfet al. [2014] N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani, and S. Wehner, Bell nonlocality, Rev. Mod. Phys. 86, 419 (2014).
  * Renou \͡mathrm{missing}extbfet al. [2021] M.-O. Renou, D. Trillo, M. Weilenmann, T. P. Le, A. Tavakoli, N. Gisin, A. Acín, and M. Navascués, Quantum theory based on real numbers can be experimentally falsified, Nature 600, 625–629 (2021).
  * Gallego \͡mathrm{missing}extbfet al. [2010] R. Gallego, N. Brunner, C. Hadley, and A. Acín, Device-independent tests of classical and quantum dimensions, Phys. Rev. Lett. 105, 230501 (2010).
  * Pawłowski and Żukowski [2010] M. Pawłowski and M. Żukowski, Entanglement-assisted random access codes, Phys. Rev. A 81, 042326 (2010).
  * Tavakoli \͡mathrm{missing}extbfet al. [2015] A. Tavakoli, A. Hameedi, B. Marques, and M. Bourennane, Quantum random access codes using single $d$-level systems, Phys. Rev. Lett. 114, 170502 (2015).
  * Pawłowski \͡mathrm{missing}extbfet al. [2009] M. Pawłowski, T. Paterek, D. Kaszlikowski, V. Scarani, A. Winter, and M. Żukowski, Information causality as a physical principle, Nature 461, 1101–1104 (2009).
  * Chaves \͡mathrm{missing}extbfet al. [2015] R. Chaves, C. Majenz, and D. Gross, Information–theoretic implications of quantum causal structures, Nat. Comm. 6, 5766 (2015).
  * Pawłowski and Brunner [2011] M. Pawłowski and N. Brunner, Semi-device-independent security of one-way quantum key distribution, Phys. Rev. A 84, 010302 (2011).
  * Woodhead and Pironio [2015] E. Woodhead and S. Pironio, Secrecy in prepare-and-measure clauser-horne-shimony-holt tests with a qubit bound, Phys. Rev. Lett. 115, 150501 (2015).
  * Brunner \͡mathrm{missing}extbfet al. [2013] N. Brunner, M. Navascués, and T. Vértesi, Dimension witnesses and quantum state discrimination, Phys. Rev. Lett. 110, 150501 (2013).
  * Šupić and Bowles [2020] I. Šupić and J. Bowles, Self-testing of quantum systems: a review, Quantum 4, 337 (2020).
  * Sarkar \͡mathrm{missing}extbfet al. [2023] S. Sarkar, C. Datta, S. Halder, and R. Augusiak, Self-testing composite measurements and bound entangled state in a unified framework (2023), arXiv:2301.11409 [quant-ph] .
  * Brandão \͡mathrm{missing}extbfet al. [2013] F. G. S. L. Brandão, M. Horodecki, J. Oppenheim, J. M. Renes, and R. W. Spekkens, Resource theory of quantum states out of thermal equilibrium, Phys. Rev. Lett. 111, 250404 (2013).
  * D’Ariano \͡mathrm{missing}extbfet al. [2005] G. M. D’Ariano, P. L. Presti, and P. Perinotti, Classical randomness in quantum measurements, Journal of Physics A: Mathematical and General 38, 5979–5991 (2005).
  * Lostaglio [2019] M. Lostaglio, An introductory review of the resource theory approach to thermodynamics, Rep. Prog. Phys. 82, 114001 (2019).
  * Howard and Campbell [2017] M. Howard and E. Campbell, Application of a resource theory for magic states to fault-tolerant quantum computing, Phys. Rev. Lett. 118, 090501 (2017).

##  Appendix A Appendix A: Proofs

######  Theorem 1.

Consider a resource theory $\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$ such that
the cardinality of the $\mathcal{S}_{\mathrm{ext},d}$ is $N$ with $N<d^{2}$.
Consider now the PO scenario described above with $y=0,1,\ldots,N$ and
$x=0,1,\ldots,d^{2}-1$ and any $j\geq 2$ such that one obtains
$\vec{p}=\\{p(j|x,y)\\}$. Then, there exists at least one probability
distribution $\vec{p}_{Q}$ in quantum theory that can not be realized by free
states in $\mathcal{S}_{d}$.

###### Proof.

The proof is by contradiction. Let us first identify $\vec{p}_{Q}$ by
considering that for each $y$ the preparation box produces the corresponding
pure states $|\psi_{y}\rangle\in\mathbb{C}^{d}$ such that
$\\{|\psi_{y}\rangle\\!\\!\langle\psi_{y}|\\}_{y}$ are linearly independent.
Similarly, the operations are chosen such that for each input $x$ the
corresponding Kraus operators are
$K_{0,x}=|\phi_{x}\rangle\\!\\!\langle\phi_{x}|$ such that
$\\{|\phi_{x}\rangle\\!\\!\langle\phi_{x}|\\}_{x}$ are again linearly
independent. As $x=0,1,\ldots,d^{2}-1$, the set
$\\{|\phi_{x}\rangle\\!\\!\langle\phi_{x}|\\}$ forms a basis for any operator
acting on $\mathbb{C}^{d}$. Now, any probability $p_{Q}(0|x,y)$ realised by
these states and operators are given by
$p_{Q}(0|x,y)=|\langle\psi_{y}|\phi_{x}\rangle|^{2}$. Moreover, as the set
$\\{|\phi_{x}\rangle\\!\\!\langle\phi_{x}|\\}$ is a basis, $p_{Q}(0|x,y)\neq
0$ for at least one $x$ for every $y$. From here on, we refer to
$\vec{p}_{Q}=\\{p_{Q}(0|x,y)\\}$ for the rest of the proof.

Let us now assume that the vector $\vec{p}_{Q}$ can be realized using free
states in $\mathcal{S}_{d}$. As described earlier, any free state in
$\mathcal{S}_{d}$ can be written as a linear combination of
$|\psi_{\sigma_{d},i}\rangle\\!\\!\langle\psi_{\sigma_{d},i}|\in\mathcal{S}_{\mathrm{ext},d}$
and thus we obtain that

$\sum_{i=1}^{N}\lambda_{i,y}\langle\psi_{\sigma_{d},i}|\tilde{K}_{0,x}^{\dagger}\tilde{K}_{0,x}|\psi_{\sigma_{d},i}\rangle=p_{Q}(0|x,y)\quad\forall
x,y$ (9)

where we do not put any restrictions on the Kraus operators $\tilde{K}_{0,x}$
and $\lambda_{i,y}$ are some scalars such that
$\sum_{i=1}^{N}\lambda_{i,y}|\psi_{\sigma_{d},i}\rangle\\!\\!\langle\psi_{\sigma_{d},i}|$
are valid quantum states for all $y$.

We now show that the above relations given in Eq. (9) can not hold if
$n(\mathcal{S}_{\mathrm{ext},d})=N<d^{2}$. For this purpose, we observe that
the above relations in Eq. (9) can be expressed as a matrix equation of the
form $M_{F}=M_{Q}$, where $M_{F}$ and $M_{Q}$ are matrices with $N+1$ rows and
$d^{2}$ columns such that

$\displaystyle M_{F}$ $\displaystyle=$
$\displaystyle\sum_{y=0}^{N}\sum_{x=0}^{d^{2}-1}\left(\sum_{i=1}^{N}\lambda_{i,y}\langle\psi_{\sigma_{d},i}|\tilde{K}_{0,x}^{\dagger}\tilde{K}_{0,x}|\psi_{\sigma_{d},i}\rangle\right)|y\rangle\\!\langle
x|,$ $\displaystyle M_{Q}$ $\displaystyle=$
$\displaystyle\sum_{y=0}^{N}\sum_{x=0}^{d^{2}-1}p_{Q}(0|x,y)|y\rangle\\!\langle
x|.$ (10)

Let us evaluate the rank of these matrices. We first consider $M_{Q}$ and show
that its rank is $N+1$ and thus all the rows must be linearly independent. To
see this, we take a linear combination of all rows to obtain the following
condition

$\displaystyle\sum_{y}\alpha_{y}\mathrm{Tr}(|\psi_{y}\rangle\\!\\!\langle\psi_{y}|\
|\phi_{x}\rangle\\!\\!\langle\phi_{x}|)=0\quad\forall x$ (11)

where $\alpha_{y}$ are some scalars and then we show that $\alpha_{y}=0$ for
all $y$. Consequently, we take a linear combination of the above relations
(11) that should satisfy for any $\\{\beta_{x}\\}$ the following condition

$\displaystyle\mathrm{Tr}\left(\sum_{y}\alpha_{y}|\psi_{y}\rangle\\!\\!\langle\psi_{y}|\sum_{x}\beta_{x}|\phi_{x}\rangle\\!\\!\langle\phi_{x}|\right)=0.$
(12)

As $\\{|\phi_{x}\rangle\\!\\!\langle\phi_{x}|\\}$ spans the space of all
density matrices of dimension $d^{2}$, we obtain that

$\displaystyle\mathrm{Tr}\left(\sum_{y}\alpha_{y}|\psi_{y}\rangle\\!\\!\langle\psi_{y}|\
\rho\right)=0\qquad\forall\rho\in\mathbb{C}^{d}.$ (13)

Thus, the only solution of the above formula is

$\displaystyle\sum_{y}\alpha_{y}|\psi_{y}\rangle\\!\\!\langle\psi_{y}|=0$ (14)

which using the fact that $|\psi_{y}\rangle\\!\\!\langle\psi_{y}|$ are
linearly independent gives us $\alpha_{y}=0$ for all $y$. Thus, all the rows
of $M_{Q}$ are linearly independent, and $\mathrm{rank}(M_{Q})=N+1$.

Now, let us focus on $M_{F}$ and show that all its rows are not linearly
independent. As done above, we take a linear combination of all rows of
$M_{F}$ to obtain the following condition

$\displaystyle\sum_{i,y}\alpha_{y}\lambda_{i,y}\mathrm{Tr}(|\psi_{\sigma_{d},i}\rangle\\!\\!\langle\psi_{\sigma_{d},i}|\tilde{K}_{0,x}^{\dagger}\tilde{K}_{0,x})=0\quad\forall
x$ (15)

which can be expressed as

$\displaystyle\sum_{i=1}^{N}\left(\sum_{y=0}^{N}\alpha_{y}\lambda_{i,y}\right)\mathrm{Tr}(|\psi_{\sigma_{d},i}\rangle\\!\\!\langle\psi_{\sigma_{d},i}|\tilde{K}_{0,x}^{\dagger}\tilde{K}_{0,x})=0\quad\forall
x.$ (16)

If all the rows of $M_{F}$ are linearly independent, then the only solution of
the above formula should be $\alpha_{y}=0$ for all $y$. However, the following
condition satisfies the above formula

$\displaystyle\sum_{y=0}^{N}\alpha_{y}\lambda_{i,y}=0\quad\forall i.$ (17)

Given any fixed set of $\lambda_{i,y}$, there are $N$ equations with $N+1$
variables $\alpha_{y}$. Thus, one has infinite solutions of $\alpha_{y}$.
Thus, the rows of $M_{F}$ are not linearly independent which allows us to
conclude that $\mathrm{rank}(M_{F})\leq N$. Consequently, our initial
assumption $M_{F}=M_{Q}$ is incorrect as both matrices are of different ranks.
This completes the proof. ∎

######  Theorem 2.

Consider a resource theory $\\{\mathcal{S}_{d},\mathcal{O}_{d}\\}$ such that
the cardinality of the $\mathcal{O}_{\mathrm{ext},d}$ is $N$ such that
$N<d^{2}$. Consider now the PO scenario described above with
$y=0,1,\ldots,d^{2}-1$ and $x=0,1,\ldots,N$ and any $j\geq 2$ such that one
obtains $\vec{p}=\\{p(j|x,y)\\}$. Then, there exists at least one probability
distribution $\vec{p}_{Q}$ in quantum theory that can not be realized by free
operations in $\mathcal{O}_{d}$.

###### Proof.

Again, the proof is by contradiction. Let us first identify $\vec{p}_{Q}$ by
considering that for each $y$ the preparation box produces the corresponding
pure states $|\psi_{y}\rangle\in\mathbb{C}^{d}$ such that
$\\{|\psi_{y}\rangle\\!\\!\langle\psi_{y}|\\}_{y}$ are linearly independent.
As $y=0,1,\ldots,d^{2}-1$, the set
$\\{|\psi_{y}\rangle\\!\\!\langle\psi_{y}|\\}$ forms a basis for any operator
acting on $\mathbb{C}^{d}$. Similarly, the operations are chosen such that for
each input $x$ the corresponding Kraus operators are
$K_{0,x}=|\phi_{x}\rangle\\!\\!\langle\phi_{x}|$ such that
$\\{|\phi_{x}\rangle\\!\\!\langle\phi_{x}|\\}_{x}$ are again linearly
independent. Now, any probability $p_{Q}(0|x,y)$ realised by these states and
operators are given by $p_{Q}(0|x,y)=|\langle\psi_{y}|\phi_{x}\rangle|^{2}$.
Moreover, as the set $\\{|\psi_{y}\rangle\\!\\!\langle\psi_{y}|\\}$ is a
basis, $p_{Q}(0|x,y)\neq 0$ for at least one $y$ for every $x$. From here on,
we refer to $\vec{p}_{Q}=\\{p_{Q}(0|x,y)\\}$ for the rest of the proof.

Let us now assume that the vector $\vec{p}_{Q}$ can be realized using free
operations in $\mathcal{O}$. As described earlier, any free operation in
$\mathcal{O}_{d}$ can be written as a linear combination of
$|\psi_{\delta_{d},i}\rangle\\!\\!\langle\psi_{\delta_{d},i}|\in\mathcal{O}_{\mathrm{ext},d}$
and thus we obtain that

$\sum_{i=1}^{N}\lambda_{i,x}\langle\psi_{\delta_{d},i}|\tilde{\rho}_{y}|\psi_{\delta_{d},i}\rangle=p_{Q}(0|x,y)\quad\forall
y$ (18)

where we do not put any restrictions on the state $\tilde{\rho}_{y}$ and
$\lambda_{i,x}$ are some scalars such that
$\sum_{i=1}^{N}\lambda_{i,x}|\psi_{\delta_{d},i}\rangle\\!\\!\langle\psi_{\delta_{d},i}|$
are valid quantum operators for all $y$..

Let us now show that the above relations in Eq. (18) cannot hold if
$n(\mathcal{O}_{\mathrm{ext}})=N<d^{2}$. We again observe that the above
relations in Eq. (18) can be expressed as a matrix equation of the form
$M_{F}^{\prime}=M_{Q}^{\prime}$, where $M_{F}^{\prime},M_{Q}^{\prime}$ are
matrices with $N+1$ rows and $d^{2}$ columns such that

$\displaystyle M_{F}^{\prime}$ $\displaystyle=$
$\displaystyle\sum_{y=0}^{d^{2}-1}\sum_{x=0}^{N}\left(\sum_{i=1}^{N}\lambda_{i,y}\langle\psi_{\delta_{d},i}|\tilde{\rho}_{y}|\psi_{\delta_{d},i}\rangle\right)|x\rangle\\!\langle
y|,$ $\displaystyle M_{Q}^{\prime}$ $\displaystyle=$
$\displaystyle\sum_{y=0}^{d^{2}-1}\sum_{x=0}^{N}p_{Q}(0|x,y)|x\rangle\\!\langle
y|.$ (19)

As done in Theorem 1, one can straightaway show that $M_{F}^{\prime}\neq
M_{Q}^{\prime}$ by showing that $\mathrm{rank}(M_{Q}^{\prime})=N+1$ but
$\mathrm{rank}(M_{F}^{\prime})\leq N$ and thus proving that the relations in
Eq. (18) can not hold. This completes the proof. ∎

##  Appendix B Appendix B: Witnesses for some more resource theories

###  B.1 Purity

Here, we construct a witness for the resource theory of purity for any $d$
[14, 15, 16]. In this theory, the free state is $\mathbbm{1}/d$ and free
operations are called unital operations $\Lambda_{U}$ which preserve the
maximally mixed state, i.e., $\Lambda_{U}(\mathbbm{1}/d)=\mathbbm{1}/d$. We
consider the PO scenario with $x,y=0$, and the witness to observe purity is
simply given by

$\displaystyle W_{P}=p(0|0,0)\leq\beta_{P}.$ (20)

It is straightforward to observe that the above witness is bounded by $1/d$
for $\rho_{0}=\mathbbm{1}/d$ and any quantum operation as

$\displaystyle\beta_{F,P}=p(0|0,0)=\mathrm{Tr}(K_{0,0}^{\dagger}K_{0,0}\rho_{0})\leq\frac{1}{d}.$
(21)

It is clear that one can obtain $W_{p}=1$ with $|\psi_{0}\rangle=|0\rangle$
and $K_{0,0}=|0\rangle\\!\\!\langle 0|$. However, using the above witness one
can not distinguish unital to non-unital operations. We leave that problem
open for future explorations.

###  B.2 Magic

In the resource theory of magic, free states and free operations are
stabiliser states and Clifford unitaries, respectively [21, 43]. Interestingly
for $d=2$, the witness $W_{C}$ in Eq. (4) of the main text also serves as a
witness of magic resource given by

$\displaystyle W_{C}=p(0|0,0)+p(0|0,1)$ $\displaystyle+$ $\displaystyle
p(0|1,0)+p(1|1,1)+p(1|0,2)\leq\beta_{M}.$ (22)

However, the maximal value that can be achieved by non-magic states
$\beta_{M}$ is higher than $\beta_{C}$. Numerically evaluating $\beta_{M}$
gives us $\beta_{M}=4.32$. Here, the optimisation is carried over free states
and any quantum measurement. Consequently, any value of $W_{C}$ beyond
$\beta_{M}$ detects that the preparation box prepares at least one magic
state. Consider now the states and Kraus operators given in Eqs. (5) and (6)
of the main text, respectively, and as described in the main text one can
obtain $W_{C}=3+\sqrt{2}$ for those states and measurements.