# Supplementary to Simulatability phase transitions in the dynamics of open
quantum spin systems

Rahul Trivedi1,2<EMAIL_ADDRESS>J. Ignacio Cirac1,2 1Max-Planck-
Institut für Quantenoptik, Hans-Kopfermann-Str. 1, 85748 Garching, Germany.
2Munich Center for Quantum Science and Technology (MCQST), Schellingstr. 4,
D-80799 Munich, Germany.

##  I Proof of theorem 1

In this section, we outline a proof of theorem 1 of the main text. We begin
the section by recalling a basic result from the theory of subcritical
percolation Grimmett (1997); Bazant (2000) which provides a bound on the
likelihood of obtaining a very large connected cluster in a (site) percolating
lattice. We also provide a simple extension of this result to a percolation
problem where the state of each site is dependent on the states of (some
finite number of) its neighbouring sites — this result is used for analyzing
the run-time of the classical sampling algorithm used in the proof of theorem
1.

###  I.1 Some results for subcritical percolation

######  Definition 1 (Cluster)

Consider a percolation problem on a connected lattice $L\subset\mathbb{Z}^{d}$
where each vertex is either open (set to 1) or closed (set to 0). A set of
closed vertices $S\subset L$ is called a cluster if $\forall v_{1},v_{2}\in
S$, there exists a path (using only edges from $\mathbb{Z}^{d}$) from $v_{1}$
to $v_{2}$. We will denote by $\mathcal{C}(L)$ as the set of all clusters on
$L$ i.e. $\mathcal{C}(L)=\\{S\subset L|S\text{ is a cluster}\\}$.

######  Lemma 1 (Independent site-percolation from Ref. Bazant (2000))

Consider a site-percolation problem on a connected lattice
$L\subset\mathbb{Z}^{d}$ with $n$ sites where each vertex $v$ is independently
opened (set to 1) or closed (set to 0). Then, $\exists p_{c}\in(0,1)$,
referred to as the site percolation threshold, such that if
$\textnormal{Prob}(X_{v}=1)>p_{c}\ \forall\ v\in L$, where $X_{v}$ is the
state of the vertex $v\in L$ then

$\textnormal{Prob}\bigg{(}\max_{S\in\mathcal{C}(L)}|S|\geq s\bigg{)}\leq
O(n^{d}s^{d}\exp(-s/s_{p})),$

for some $s_{p}>0$.

Next, we consider a site percolation problem on $\mathbb{Z}^{d}$ where each
site is statistically dependent on its neighbours, and show that a similar
percolation threshold can be derived for such a problem. The idea is to
construct a channel from a percolation problem where all sites are independent
to the percolation problem where each site is dependent on its neighbours in
such a way that the maximum cluster size can only decrease.

###### Lemma 2

Consider a site-percolation problem on a connected lattice
$L\subset\mathbb{Z}^{d}$ with $n$ sites where each site is opened (set to 1)
or closed (set to 0) with state of a vertex $v\in L$ being conditionally
dependent on state of the vertices in a deleted neighbourhood $N_{v}\subset
L$, then if $\inf_{v\in L}\inf_{x\in\\{0,1\\}^{|N_{v}|}}\
\textnormal{Prob}(X_{v}=1|X_{N_{v}}=x)\geq p_{c}$, where for $v\in L,\
X_{v}\in\\{0,1\\}$ is the state of the vertex $v$, for $S\in L$,
$X_{S}\in\\{0,1\\}^{|S|}$ is the state of the vertices contained in $S$ and
$p_{c}$ is the site-percolation threshold defined in lemma 1, then

$\textnormal{Prob}\bigg{(}\max_{S\in\mathcal{C}(L)}|S|\geq s\bigg{)}\leq
O(n^{d}s^{d}\exp(-s/s_{p})),$

for some $s_{p}>0$.

Proof: We consider a site-percolation problem on $L$ where each vertex $v\in
L$ is independently open or closed. For notational convenience, $\forall v\in
L$, we define $x_{v}\in\\{0,1\\}^{|N_{v}|}$ via

$x_{v}=\operatorname*{arg\,min}_{x\in\\{0,1\\}^{|N_{v}|}}\text{Prob}(X_{v}=1|X_{N_{v}}=x).$

Next, $\forall v\in L$ and $x\in\\{0,1\\}^{|N_{v}|}$, we define a Bernoulli
random variable $Z_{v,x}$ such that

$\text{Prob}(Z_{v,x}=1)=\begin{cases}\text{Prob}(X_{v}=1|X_{N_{v}}=x_{v})&\text{
if }x=x_{v},\\\
{\big{(}\text{Prob}(X_{v}=1|X_{N_{v}}=x)-\text{Prob}(X_{v}=1|X_{N_{v}}=x_{v})}\big{)}/\text{Prob}(X_{v}=0|X_{N_{v}}=x_{v})&\text{
if }x\neq x_{v}.\end{cases}$

The random variable $X_{v}$ can then be generated from $Z_{v,x}$ via

$X_{v}=\begin{cases}1&\text{ if }Z_{v,x_{v}}=1,\text{ else }\\\ Z_{v,x}&\text{
where }x\in\\{0,1\\}^{|N_{v}|}\text{ such that }X_{N_{v}}=x.\end{cases}$

It is easy to verify that this definition of $X_{v}$ gives the right
conditional probability distributions. Furthermore, we can easily note that
$Z_{v,x_{v}}=1\implies X_{v}=1$. Therefore, the maximum cluster size in a
lattice $L$ where each vertex $v$ has state $Z_{v,x_{v}}$ is necessarily
larger than the maximum cluster size in a lattice $L$ where each vertex $v$
has state $X_{v}$. Thus, from lemma 1, the bound on the likelihood of the
maximum cluster size follows. $\square$.

###  I.2 Sampling from the open-quantum spin system dynamics

The problem under consideration is precisely stated below,

Problem: _We consider a noise rate $\kappa$, a single-qubit entanglement
breaking channel $\mathcal{N}(\cdot)$ and a family of Lindbladians
$(\mathcal{L}_{n})_{n\in\mathbb{N}}$ over $n$-qubits on the $d-$dimensional
lattice $\mathbb{Z}^{d}$. The local Lindbladian $\mathcal{L}_{n}$ is
expressible as a sum of $M=O(\text{poly}(n))$ Lindbladians acting over qubits
at members of the set
$\mathcal{S}=\\{\Lambda_{1},\Lambda_{2}\dots\Lambda_{M}\\}$, where
$\Lambda_{i}\subset\mathbb{Z}^{d}\ \forall\ i\in[M]$ i.e._

$\mathcal{L}_{n}=\sum_{\Lambda\in\mathcal{S}}\mathcal{L}_{\Lambda},$

_where $\forall\ \Lambda\in\mathcal{S}$, (i) $\mathcal{L}_{\Lambda}$ is
identity when acting on qubits outside $\Lambda$, (ii) its diameter is bounded
i.e. $d(\Lambda)\leq O(1)\ \forall\ \Lambda\in\mathcal{S}$ and (iii) the
Lindbladian has a bounded norm i.e.
$\left\|{\mathcal{L}_{\Lambda}}\right\|_{1\to 1}\leq O(1)$. We further assume
that the set $\mathcal{S}$ can be partitioned into $L=\Theta(1)$ sets
$\mathcal{S}_{1},\mathcal{S}_{2}\dots\mathcal{S}_{L}$ such that $\forall
i\in[L]$ and $\Lambda,\Lambda^{\prime}\in\mathcal{S}$,
$\Lambda\cap\Lambda^{\prime}=\phi$. The master equation under consideration is
then given by_

$\frac{d\rho(t)}{dt}=\mathcal{L}_{n}\rho(t)+\kappa\sum_{i=1}^{n}\big{(}\mathcal{N}_{i}-\textnormal{Id}\big{)}\rho(t),$

_where $\mathcal{N}_{i}$ is the entanglement breaking channel $\mathcal{N}$
acting on the $i^{\textnormal{th}}$ qubit. The computational problem that we
are interested in is to sample from the state $\rho(t)$, in the computational
basis, for a time $t=O(\textnormal{poly}(n))$._

We first trotterize the evolution and represent each channel in the
trotterized evolution (be it the entanglement breaking channel due to
trotterization of the noise, or the channels that are entangling neighbouring
qubits) as a convex combination of the identity channel and a completely-
positive trace preserving (CPTP) channel. This is made precise in the
following lemma.

###### Lemma 3

A parameter $N=\Theta(\textnormal{poly}(n))$ can be chosen such that
$\left\|{\rho(t)-\hat{\sigma}_{t,N}}\right\|_{\textnormal{tr}}\leq
O(1/\textnormal{poly}(n))$, where $\hat{\sigma}_{t,N}$ is the trotterized
state

$\displaystyle\hat{\sigma}_{t,N}=\prod_{i=1}^{L}\bigg{[}\prod_{\Lambda\in\mathcal{S}_{i}}\bigg{(}\bigg{(}1-\frac{gt}{N}\bigg{)}\textnormal{id}+\frac{gt}{N}\mathcal{E}_{\Lambda}\bigg{)}\bigg{)}\prod_{i=1}^{n}\bigg{(}\bigg{(}1-\frac{\kappa
t}{NL}\bigg{)}\textnormal{id}+\frac{\kappa
t}{NL}\mathcal{N}_{i}\bigg{)}\bigg{)}\bigg{]}^{N}\rho(0),$ (1)

with
$g\geq\max_{\Lambda\in\mathcal{S}}\lambda_{\textnormal{max}}(\Phi_{\mathcal{L}_{\Lambda}})$
(where $\lambda_{\textnormal{max}}(\Phi_{\mathcal{L}})$ is the maximum
magnitude eigenvalue of the Choi state corresponding to
$\mathcal{L}_{\Lambda}$) and $\mathcal{E}_{\Lambda}$ is a channel acting on
the qubits contained in $\Lambda$. The channel
$(1-gt/N)\textnormal{id}+gt/N\mathcal{E}_{\Lambda}$ appearing in the
trotterized channel, expressed as a convex combination of id and
$\mathcal{E}_{\Lambda}$, will be referred to as a ‘horizontal’ channel.

Proof: We use the first order Trotter-Suzuki formula to approximate $\rho(t)$
by $\hat{\rho}_{t,N}$, where for $N\in\mathbb{N}$,

$\displaystyle\hat{\rho}_{t,N}=\bigg{[}\prod_{i=1}^{L}\bigg{(}\prod_{\Lambda\in\mathcal{S}_{i}}e^{\mathcal{L}_{\Lambda}t/N}\bigg{)}\bigg{(}\prod_{i=1}^{n}e^{\kappa(\mathcal{N}_{i}-\textnormal{Id})t/NL}\bigg{)}\bigg{]}^{N}\rho(0).$
(2)

A standard analysis of the trotterization error Childs  _et al._ (2021) allows
us to bound the trace-norm error between $\rho(t)$ and $\hat{\rho}$:

$\left\|{\rho(t)-\hat{\rho}_{t,N}}\right\|_{1}\leq
O\bigg{(}\frac{t^{2}}{N}\bigg{(}\sum_{\Lambda}\left\|{\mathcal{L}_{\Lambda}}\right\|_{1\to
1}+n\kappa\left\|{\textnormal{Id}-\mathcal{N}}\right\|_{1\to
1}\bigg{)}^{2}\bigg{)}.$

Since $\kappa=\Theta(1),\left\|{\mathcal{I}-\mathcal{N}}\right\|_{1\to
1}=O(1)$ and $\forall\ \Lambda\in\mathcal{S}:\
\left\|{\mathcal{L}_{\Lambda}}\right\|_{1\to 1}\leq O(1)$, we obtain that
$\left\|{\rho(t)-\hat{\rho}_{t,N}}\right\|_{1}\leq O(t^{2}\text{poly}(n)/N)$ —
a sufficiently large value of $N$ which scales as $\text{poly}(n)$ thus allows
us to control the error incurred in this approximation.

Next, we consider each of the channels appearing in the trotterization (Eq. 2)
and express them as the convex combination of the identity channel and a
completely-positive trace-preserving (CPTP) map. For a Lindbladian
$\mathcal{L}$, consider the channel $e^{\mathcal{L}\tau}$ for some $\tau>0$. A
first order Taylor expansion of this map with respect to $\tau$ yields the map
$\textnormal{Id}+\tau\mathcal{L}$ — note that if $\tau$ is sufficiently small
(made precise below), this map is also CPTP. From Taylor’s theorem it follows
that

$\left\|{e^{\mathcal{L}\tau}-\big{(}\textnormal{Id}+\mathcal{L}\tau\big{)}}\right\|_{1\to
1}\leq O(\left\|{\mathcal{L}}\right\|_{1\to 1}^{2}\tau^{2}).$

Next, we note that for any $g>0$,
$\textnormal{Id}+\mathcal{L}\tau=(1-g\tau)\textnormal{Id}+g\tau(\textnormal{Id}+\mathcal{L}/g)$.
Choosing $g>\lambda_{\text{max}}(\Phi_{\mathcal{L}})$, where
$\Phi_{\mathcal{L}}$ is the Choi-state corresponding to $\mathcal{L}$, it
follows that $\textnormal{Id}+\mathcal{L}/g$ is completely positive.
Furthermore, since $\mathcal{L}$ is a Lindbladian, it follows that
$\textnormal{Id}+\mathcal{L}/g$ is completely positive and trace preserving.
For $\tau<1/g$, we have thus approximated $e^{\mathcal{L}\tau}$ by a convex
combination of the identity channel (applied with probability $1-g\tau$) and a
non-identity CPTP channel (applied with probability $g\tau$). Now, using
$g>\sup_{\Lambda\in\mathcal{S}}\lambda_{\text{max}}(\Phi_{\mathcal{L}_{\Lambda}})$,
we obtain that $\forall\Lambda\in\mathcal{S}$,

$e^{\mathcal{L}_{\Lambda}t/N}=\bigg{(}1-\frac{gt}{N}\bigg{)}\textnormal{Id}+\frac{gt}{N}\mathcal{E}_{\Lambda}+O\bigg{(}\frac{t^{2}}{N^{2}}\bigg{)}$

for some channel $\mathcal{E}_{\Lambda}$ that only acts on the qubits in
$\Lambda$. Similarly, $\forall i\in[n]$:

$e^{\kappa(\mathcal{N}_{i}-\textnormal{Id})tN/L}=\bigg{(}1-\frac{\kappa
t}{NL}\bigg{)}\textnormal{Id}+\frac{\kappa
t}{NL}\mathcal{N}_{i}+O\bigg{(}\frac{t^{2}}{N^{2}}\bigg{)}.$

We can then construct a circuit whose output is $\hat{\sigma}_{t,N}$, where

$\displaystyle\hat{\sigma}_{t,N}=\prod_{i=1}^{L}\bigg{[}\prod_{\Lambda\in\mathcal{S}_{i}}\bigg{(}\bigg{(}1-\frac{gt}{N}\bigg{)}\textnormal{id}+\frac{gt}{N}\mathcal{E}_{\Lambda}\bigg{)}\bigg{)}\prod_{i=1}^{n}\bigg{(}\bigg{(}1-\frac{\kappa
t}{NL}\bigg{)}\textnormal{id}+\frac{\kappa
t}{NL}\mathcal{N}_{i}\bigg{)}\bigg{)}\bigg{]}^{N}\rho(0).$ (3)

Furthermore, it easily follows from the abovementioned error estimates that
$\left\|{\hat{\sigma}_{t,N}-\rho(t)}\right\|_{1}\leq O(t^{2}\text{poly}(n)/N)$
— if $t=O(\text{poly}(n))$, then clearly there is a choice of
$N=\Theta(\text{poly}(n)/\varepsilon)$ that ensures that
$\left\|{\hat{\sigma}_{t,N}-\rho(t)}\right\|_{1}\leq\varepsilon$. Choosing
$\varepsilon=\Theta(1/\text{poly}(n))$ proves the lemma statement $\square$.

Next, we consider the problem of sampling from $\hat{\sigma}_{t,N}$ — we first
map it to a percolation problem on a $(d+1)$-dimensional lattice by choosing
the identity channel, or the non-identity channel ($\mathcal{E}_{\Lambda}$ or
$\mathcal{N}_{i}$). However, this sampling alone is not enough to ensure that
the resulting problem percolates, since the probability of choosing the non-
identity channel is very close to 0 — in order to map this to a problem that
percolates, each site in the percolation problem needs to be mapped to an of
block many time-steps per qubit. This is made precise in the definition below.

######  Definition 2 (Equivalent percolation problem)

Choose $\tau>0$ and $m=\lceil{N\tau/t}\rceil$ — a percolation problem on
$\mathbb{Z}^{d+1}$ corresponding to sampling from $\hat{\sigma}_{t,N}$ in Eq.
3 is constructed by the first sampling from the convex combination of the
channels ($\mathcal{E}_{\Lambda}$, $\mathcal{N}_{i}$ or the single- or two-
qubit identity channels) in Eq. 3 — then a vertex
$v:=(i,q)\in\mathbb{Z}^{d+1}$ (where $i\in\mathbb{Z}^{d}$ and
$q\in\mathbb{N}$) is open if all the horizontal channels acting on the
$i^{\textnormal{th}}$ qubit from time-step $mq$ to $m(q+1)-1$ are sampled to
be the identity channel, and the channel $\mathcal{N}_{i}$ is applied at least
once.

Proof of theorem 1: We next analyze the equivalent percolation problem and
arrive at a proof of theorem 1 from the main text. Our goal is to show that
for a sufficiently large $\kappa$, $\tau$ can be chosen to ensure that the
equivalent percolation problem is subcritical which can then be efficiently
sampled from. To do so we will use lemma 2 — we first provide a lower bound on
the probability that a vertex on the lattice corresponding to the equivalent
percolation problem is open, maximized over all possible configurations of the
neighbouring vertices. For a vertex $v=(i,q)$ in the equivalent percolation
problem, we associate the neighbourhood $N_{v}=\\{(j,q),\text{ where
}j\in\mathbb{Z}^{d},s\in\mathbb{Z}\ |\ \exists\Lambda\in\mathcal{S}\text{ such
that }i,j\in\Lambda\\}$. Note that the probability of a vertex $v$,
conditioned on the state of the vertices in its neighbourhood $N_{v}$, being
open is the smallest when the vertices in $N_{v}$ are all closed — thus if
this probability can be made larger than the site-percolation threshold, the
equivalent lattice will also percolate. Recall that

$\displaystyle\text{Prob}(X_{v}=1|X_{N_{v}}=0^{|N_{v}|})=\frac{\text{Prob}(X_{v}=1,X_{N_{v}}=0^{|N_{v}|})}{\text{Prob}(X_{N_{v}}=0^{|N_{v}|})}.$
(4)

Consider the probability of the vertex $v$ being open and all the vertices in
its neighbourhood being closed. One configuration of samplings from the
channels that achieve this is if all the horizontal channels acting on the
qubit associated with $v$ are open, the entanglement breaking channel is
applied at least once on the qubit associated with $v$ and the entanglement
breaking channel is never applied with the qubits associated with the vertices
in $N_{v}$. This provides us with the lower bound

$\text{Prob}(X_{v}=1,X_{N_{v}}=0^{|N_{v}|})\geq\bigg{(}1-\frac{gt}{N}\bigg{)}^{ml_{v}}\bigg{(}1-\bigg{(}1-\frac{\kappa
t}{LN}\bigg{)}^{Lm}\bigg{)}\bigg{(}1-\frac{\kappa t}{LN}\bigg{)}^{Lm|N_{v}|},$

where $l_{v}$ is the the number of $\Lambda\in\mathcal{S}$ such that
$i\in\Lambda$ (where $v=(i,q)$). Consider now the probability of all the
vertices in $N_{v}$ being closed. This event can occur if either the
entanglement breaking channel is not applied on any of the qubits associated
with vertices in $N_{v}$, or at least one of the horizontal channels
associated with the qubits in $N_{v}$ for the time-steps in $v$ is not
identity. Thus, we obtain the upper bound

$\text{Prob}(X_{N_{v}}=0^{|N_{v}|})\leq\bigg{(}1-\frac{\kappa
t}{LN}\bigg{)}^{Lm|N_{v}|}+\bigg{(}1-\bigg{(}1-\frac{gt}{N}\bigg{)}^{mr_{N_{v}}}\bigg{)},$

where $r_{N_{v}}$ is the number of $\Lambda\in\mathcal{S}$ such that
$j\in\Lambda$ for some $(j,v)\in N_{v}$. Thus, we obtain a lower bound on the
worst-case conditional probability defined in Eq. 4. Using $m=\lceil
N\tau/t\rceil$, we obtain

$\text{Prob}(X_{v}=1|X_{N_{v}}=0^{|N_{v}|})\geq e^{-g\tau
l_{v}}(1-e^{-\kappa\tau})\bigg{(}\frac{e^{-\kappa|N_{v}|\tau}}{e^{-\kappa|N_{v}|\tau}+1-e^{-gr_{N_{v}}\tau}}\bigg{)}+O\bigg{(}\frac{1}{\text{poly}(N)}\bigg{)}.$

We can now show that for $\kappa$ larger than a constant, the lower bound
obtained above can be made larger than the site-percolation threshold $p_{c}$.
Noting that since we only consider problems with local interactions (i.e. the
diameter of the set of qubits interaction with each other is finite),
$l=\max_{v}l_{v}=O(1)$ and $r=\max_{v}r_{N_{v}}=O(1)$. Fixing $\tau$ by
$e^{-\kappa\tau}=c<1$ for some constant $c$, we obtain that

$\forall\ v:\ \text{Prob}(X_{v}|X_{N_{v}}=0^{|N_{v}|})\geq
1-c+O\bigg{(}\frac{g}{\kappa}\bigg{)}+O\bigg{(}\frac{1}{\text{poly}(N)}\bigg{)}.$

Recall from lemma 3 that $N=\Omega(\text{poly}(n))$ — it thus follows that for
large $n$, we can choose $\kappa/g$ to be large enough for the lower bound to
be very close to $1-c$ and thus an appropriate choice of $c$ (which is
equivalent to an appropriate choice of $\kappa\tau$) will result in this
probability being larger than the site-percolation threshold. It then follows
from lemma 2 that with a probability $1-O(\text{polylog}(n)/\text{poly}(n))$,
the maximum size of the percolating clusters will be $O(\text{log}(n))$. Using
this fact, we can now provide an algorithm to sample approximately sample from
the trotterized state $\hat{\sigma}_{t,n}$. We recall that an entanglement
breaking single-qubit channel can be always be expressed as

$\displaystyle\mathcal{N}(\rho)=\sum_{i}\sigma_{i}\text{Tr}(E_{i}\rho),$ (5)

where $\sigma_{i}\in\mathfrak{D}_{1}(\mathbb{C}^{2})$, and $\\{E_{i}\\}$ form
a POVM. Thus, an application of an entanglement breaking channel on a single-
qubit state can be simulated by first measuring the qubit with respect to the
POVM $E_{i}$ and then replacing the qubit with $\sigma_{i}$ if the outcome of
the measurement is $i$. The sampling algorithm then proceeds in two steps:

  1. 1.

First, we sample from the percolation problem and ignore any samples that have
a cluster of size larger than $c^{\prime}\log(n)$ where $c^{\prime}$ is a
sufficiently large constant. The error in total variation distance incurred
due to this is $O(\text{polylog}(n)/\text{poly}(n))$.

  2. 2.

We then replace the entanglement breaking channel by applying a POVM and
replacing the qubit with an unentangled state as per Eq. 5. We note that this
can be done efficiently and per each time-step. More specifically, suppose
that uptil the (discrete) time-step $k$ all the entanglement breaking channels
have been sampled in this way, and we wish to sample the entanglement breaking
channels at time-step $k+1$ — we then need to sample from the measurement of
the POVM corresponding to the qubits where the entanglement breaking channels
are applied and trace over the remaining qubits. The sampling from the POVMs
can be done sequentially — we pick any one qubit and measure it in the POVM
corresponding to the entanglement breaking channel, and trace over all the
other qubits (including the ones on which the other entanglement breaking
channel is applied). We note that we can efficiently contract the circuit to
calculate the probabilities of various outcomes corresponding to the POVM
since all the clusters of qubits can be contracted in $\text{poly}(n)$ time.
After having computed these probabilities, we produce a sample from the POVM
and conditioned on this sample we sample the next qubit which has an
entanglement breaking channel. Since there are at-most $n$ such qubits, we
need to do at most $\text{poly}(n)$ such contractions.

The above two-steps outline a sampling algorithm that can be executed in
$\text{poly}(n)$ time. Furthermore, there are two sources of error — one is
due to the trotterization, and the other is due to ignoring samples which have
large clusters. Both of these errors are smaller than
$O(\text{polylog}(n)/\text{poly}(n))$ in the total-variation distance and this
completes the proof of theorem 1. $\square$.

##  II Implication of threshold theorem on the low noise regime

The threshold theorem in quantum computation Aharonov and Ben-Or (2008) is a
well known result which states that scalable quantum computation is possible
if the noise rate in the quantum computer is low enough. These theorems are
usually proved by performing encoded quantum computation together with error
correction, and the reduction in the rate of noise needed to be able to
implement scalable quantum computation depends on the error correcting code.
While initially constructed and proved for general circuit model of
computation, the threshold theorem has been proved for quantum circuits while
constraining the interactions to be nearest neighbours on qubits arranged in
1D, 2D or 3D lattices Aharonov and Ben-Or (2008); Gottesman (2000).

Here, we investigate various implication of fault tolerant constructions with
nearest neighbour interaction on the existence of phase transitions in the
simulation of open quantum spin systems. We begin by recalling a basic lemma
proved by Aharanov et al, and Gottesman et al, which indicates that a fault
tolerant quantum computation is possible with 1D nearest neighbour qubits
provided that qubits can be restarted at any point during the time evolution.
This lemma is stated within the circuit model of computation with independent
errors (although several extensions are possible) at different locations
within the circuit with an error probability $\eta$.

######  Lemma 4 (Fault tolerance in 1D with nearest neigbour gates from Ref.
Aharonov and Ben-Or (2008))

Let $\varepsilon>0$ and let $\mathcal{G}$ a universal set of gates. Let $Q$ be
a quantum circuit on $n$ qubits and depth $\textnormal{poly}(n)$ formed from
gates from $\mathcal{G}$ acting on either one or two neighbouring qubits.
Then, $\exists\eta_{\textnormal{th}}$ such that for
$\eta\leq\eta_{\textnormal{th}}$, there exists a circuit $C^{\prime}$ formed
from gates of $\mathcal{G}\cup\\{\textnormal{SWAP},\textnormal{RESTART}\\}$
acting over $O(n\ \textnormal{polylog}(n/\varepsilon))$ qubits for depth
$O(\textnormal{poly}(n)\textnormal{polylog}(n/\varepsilon))$ such that in the
presence of noise $\eta\leq\eta_{\textnormal{th}}$, it computes a quantum
state which is $\varepsilon$-close in total variation distance to the quantum
state of the circuit $C$.

We point out that in the fault-tolerant construction used in this lemma, both
the RESTART and SWAP operations can be subjected to errors, and the
concatenated error correction schemes subsequently applied on them will
succeed if these errors are small enough. Building on this lemma and by
providing implementations of the RESTART and SWAP gates, below we show that in
many contexts, there we are able to encode a fault-tolerant quantum
computation into the problem and hence we expect in the presence of low biased
noise for the problem to be hard to simulate classically. We analyze three
models in the following two subsections:

  1. 1.

For $d\geq 2$, we consider a discrete-time model where the entanglement
breaking channels applied with a probability $p$ are interleaved with single
and two-qubit unitaries applied locally on disjoint sets of qubits. Here, we
show that as long as the entanglement breaking channel has a fixed point $\neq
I/2$, we expect the model to be classically insimulatable at sufficiently low
rate of noise.

  2. 2.

For $d=1$, we consider a discrete-time model where we apply a Matrix product
unitary followed by the entanglement breaking channel applied with a
probability $p$ on all qubits. Here, we again show that if the entanglement
breaking channel has a fixed point $\neq I/2$, we expect the model to be
classically insimulatable at sufficiently low rate of noise.

  3. 3.

For $d\geq 2$, we consider a continuous-time model where the entanglement
breaking channels can act at any time, and thus might result in erroneous gate
operations. Here, we show that if the image of space of single qubit density
matrices under the entanglement breaking channel does not contain $\neq I/2$,
then we expect this model to be classically insimulatable at sufficiently low
rate of noise.

Another key ingredient that we will use in our analysis is algorithmic cooling
Schulman and Vazirani (1999); Boykin  _et al._ (2002) i.e. having a supply of
$m$ qubits individually being in a mixed state
$\sigma_{1}\otimes\sigma_{2}\dots\otimes\sigma_{m}$ where $\sigma_{i}\neq I/2\
\forall\ i\in[m]$, we would like to be able to extract at least a single qubit
in a pure state. This is a well known problem, and of technological relevance
in NMR quantum computing where there is often a large supply of noisy qubits
and it is of interest to concentrate the entropy on a few qubits. For our
purposes, this will be a key ingredient in implementing the RESTART operations
for noise-channels which do not necessarily drive the qubit state to identity
(i.e. they are not depolarizing) and thus allow us to construct fault-tolerant
circuits below a particular noise rate and hence point to a simulatability
phase transition.

######  Definition 3 (Polarization of a qubit state)

A qubit (mixed) state $\sigma$ is said to have a polarization of $\varepsilon$
if $\lambda_{\textnormal{max}}-\lambda_{\textnormal{min}}=\varepsilon$, where
$\lambda_{\textnormal{max}}$ and $\lambda_{\textnormal{min}}$ are the maximum
and minimum eigenvalues of $\sigma$ respectively.

######  Lemma 5 (Algorithmic cooling from Ref. Schulman and Vazirani (1999))

Consider $m$ qubits initially in the state
$\rho_{\textnormal{init}}=\bigotimes_{i=1}^{m}\sigma_{i}$, where for
$i\in[m]$, $\sigma_{i}$ has polarization $\varepsilon_{i}(>0)$ and let
$\varepsilon_{\textnormal{min}}=\min_{i\in[m]}\varepsilon_{i}>0$, then for any
$\eta\in(0,1)$, $m$ can be chosen as a function of
$\eta,\varepsilon_{\textnormal{min}}$ such that $\exists$ a unitary circuit
$U$ of depth dependent on $\eta,\varepsilon_{\textnormal{min}}$ which yields
(at least) one qubit whose reduced state has polarization $1-\eta$.

Figure 1: A schematic depiction of the implementation of the RESTART operation
on a 2D lattice of qubits. We note that the a shift operation can be
implemented on the alternate qubits in any row (i.e. the qubits at the odd or
even positions) by doing two consecutive swap operations — one layer of
operations swapping the even with the odd qubits and then another layer of
operations swapping the odd qubits with the even qubits. Taken together, these
two operations will shift the odd qubits to one direction and the even qubits
in the opposite direction. For simplicity, in this figure, we depict this
shift operation as a single operation and hide one set of alternate qubits.

Proof of theorem 3: We restrict ourselves to $d=2$ (i.e. a 2D lattice of
spins), since if it is BQP hard to simulate the 2D problem in the presence of
a constant, but small enough, rate of noise, it will be BQP hard to simulate
higher dimensional problems as well. We will implement a 1D fault-tolerant
quantum computation (lemma 4) in one of the columns of the 2D lattice, and
call the qubits involved in this construction as the _computational qubits_ —
some of the computational qubits will be the data qubits (which encode the
quantum computation) and some of these qubits will be ancilla qubits needed
for the error correction operations. In order for the fault-tolerant
construction to succeed, we need to implement a RESTART gate, with fidelity
below the fault tolerance threshold, on the ancilla qubits. In order to
restart a computational ancilla qubit, we use the non-computational qubits in
its row. A schematic depiction of the restart operation is shown in Fig. 1 —
we start off with all the qubits to the right of the qubit to be restarted in
a fixed point $\sigma\neq I/2$ of the noise channel. We then perform a
sequence of three steps:

  1. 1.

First, we perform algorithmic cooling on certain number of, say $m$, qubits on
the right of the qubit to be restarted. At the end of this step, the qubits
neighbouring the computational ancilla qubit will be in a state $\ket{0}$. We
note that the algorithmic cooling operation can fail with some non-zero
probability — however, this probability does not scale with the number of
qubits involved and hence this construction still yields a threshold theorem.
We analyze this point in more detail below.

  2. 2.

Next, we swap the cooled qubit with the computational qubit. This effectively
implements the RESTART operation.

  3. 3.

Finally, we prepare the qubits for another RESTART operation - we note that
the the previous two steps left the qubits immediately on the right of the
computational qubit in a state that is no longer $\sigma$ and consequently
cannot be used to implement another RESTART operation. We therefore now
perform a shift operation to shift these qubits to the left of the
computational qubits and replace the $m$ qubits on the right of the
computational qubits with the state $\sigma^{\otimes m}$. This configuration
can now be used to implement the next RESTART operation by using steps 1-3.

_Error analysis_ : We need to ensure that a threshold theorem exists with the
above outlined RESTART operation. Recall that at every time-step, we are
allowed to apply either a single-qubit or a nearest neighbour two-qubit gate
over disjoint set of qubits. Recall from lemma 5 that to achieve a cooled
qubit in the state $\ket{0}$ with specified probability $p_{\text{cool}}$, the
number of qubits $m$ required is a function of $p_{\text{cool}}$ (which we
denote by $m(p_{\text{cool}})$), and the number of time-steps $t$ required to
accomplish this is also a function of $p_{\text{cool}}$ (which we denote by
$t_{\text{cool}}(p_{\text{cool}})$). Furthermore, it follows from lemma 5 that
to achieve $p_{\text{cooling}}=1-O(1)$, both $m(p_{\text{cool}})=O(1)$ and
$t_{\text{cool}}(p_{\text{cool}})=O(1)$. Consequently, both the following swap
and shift operations (which needs to swap $O(m(p_{\text{cool}}))$ qubits) can
be done in time-steps $t_{\text{swaps}}(p_{\text{cool}})=O(1)$. Importantly,
we note that since in the model being considered in this theorem, we can
perform the SWAP gate without an error _in the gate_ , and since $\sigma$ is a
fixed-point of the noise channel, swapping a qubit in the state $\sigma$ with
another qubit will still generate an output qubit in the state $\sigma$ (this
is not necessarily true if an error occurs in the gate) and consequently the
shift operation can always be performed to arrange the qubits properly for the
next RESTART operation. Finally, since the number of time-steps and the number
of qubits participating in the RESTART operation $O(1)$, we note that the
RESTART operation can be made more efficient than required by the error
correction threshold by making the probability of applying the entanglement
breaking noise $p$ small enough. This shows that below a particular noise
threshold, we will be able to implement a BQP hard problem fault-tolerantly
within the discrete-time model considered, which proves the theorem.
$\square$.

Proof of theorem 4: This key difficulty with implementing a RESTART operation
in the 1D nearest-neighbour setting is that the qubits that are cooled and
swapped into the computational ancillas cannot be placed close to the
computational ancillas, and $\Theta(n)$ (where $n$ is the number of
computational qubits) swap operations are required in order to bring them
close to the computational ancillas. Applying $\Theta(n)$ swap gates without
any error correction breaks the threshold theorem for quantum computation and
hence does not allow us to fault tolerantly encode a BQP hard problem into the
noisy quantum dynamics. However, if we allow ourselves to use constant bond
dimension MPUs, then we can perform $\Theta(n)$ swap operations with an MPU of
bond dimension 2 in one time-step, and thus the construction used in the proof
of theorem 2 works. In this case, we will just maintain the qubits used for
restarting the computational ancilla on one end of the 1D lattice, and cool
and swap them with the computational ancilla qubits in a single time-step.
$\square$.

We now proceed to the proof of theorem 4 where we consider a continuous model
of noise being applied on the spins. Here, the swap operations needed to shift
the qubits to be cooled near the computational qubits cannot be done
faultlessly. However, if we consider noise modelled with point channels with a
fixed point different from identity, we show that a 1D fault tolerant quantum
computation can still be implemented in $d\geq 2$.

Figure 2: Schematic of the shift operation used in the proof of theorem 4.

Proof of theorem 2: For the two (or higher) dimensional models where the noise
is applied continuously in time, one of the key difficulties in the
construction of theorem 2 is that the SWAP operations can be faulty. Thus,
even at a very low rate of noise, it is not guaranteed that the qubits that
are being used to restart the computational ancillas at times
$\Theta(\text{poly}(n))$ are in a state that is sufficiently different from
the identity. In particular, if we consider channels which have identity as a
fixed point (in addition to fixed points which are not identity) then an error
in the SWAP gate could result in one of the qubits being set into the identity
and thus not be useful for the RESTART operation.

However, under the assumption
$\exists\ket{\phi_{+}},\ket{\phi_{-}}\in\mathbb{C}^{2}$ such that the
entanglement breaking channel $\mathcal{N}$ maps every qubit state to a state
of the form
$(1-\varepsilon)/2\ket{\phi_{-}}\bra{\phi_{-}}+(1+\varepsilon)/2\ket{\phi_{+}}\bra{\phi_{+}}$,
we can construct a shift operation that, despite being erroneous, ensures with
a sufficiently high success probability that the shifted qubits are
unentangled and different from identity and hence can be used for the
subsequent cooling algorithm. This shift operation, implemented with two
layers of SWAP gates, is shown in Fig. 2 — we apply the Hamiltonian for the
SWAP gate for a time period $\tau_{s}$ (chosen such that in a noiseless
setting, the SWAP gate would be perfectly executed) and then allow the qubits
to evolve just under the influence of the entanglement breaking noise channel
for a time period $\tau_{d}$. The latter evolution not only disentangles the
output qubit due to the entanglement breaking nature of the channel, but also
drives it away from $I/2$. We make this concrete by analyzing the operation of
this shift operation on $2m+1$ qubits, out of which with probability $1-q$,
the qubits $2,4,6\dots 2m$ are in a seperable state with the states of the
individual qubits belonging to $\mathcal{N}(\mathfrak{D}_{1}(\mathbb{C}^{2}))$
i.e.

$\rho(0)=(1-q)\rho_{0,1,3,5\dots 2m-1}\otimes\sigma_{2,4,6\dots
2m}+q\rho^{\prime}.$

where $\sigma_{2,4,6\dots 2m}$ can be expressed as

$\displaystyle\sigma_{2,4,6\dots
2m}=\sum_{\alpha}p_{\alpha}\sigma_{2}^{\alpha}\otimes\sigma_{4}^{\alpha}\dots\otimes\sigma_{2m}^{\alpha},$
(6)

with $\sum_{\alpha}p_{\alpha}=1,p_{\alpha}\geq 0$ and
$\sigma_{i}^{\alpha}\in\mathcal{N}(\mathfrak{D}_{1}(\mathbb{C}^{2}))\
\forall\alpha,i\in\\{2,4,6\dots 2m\\}$. Note that with probability
$\exp(-2m\kappa\tau_{s})$, the SWAP operation occurs perfectly and hence the
output state can be expressed as.

$\rho(\tau_{s})=\exp(-2m\kappa\tau_{s})(1-q)\sigma_{1,3,5\dots
2m-1}\otimes\rho^{\prime}_{0,2,4\dots
2m}+(1-\exp(-2\kappa\tau_{s})(1-q))\rho^{\prime\prime},$

where $\sigma_{1,3,5\dots 2m-1}$ is the state $\sigma_{2,4,6\dots 2m}$ but now
shifted to the qubits $1,3,5\dots 2m-1$ and $\rho^{\prime}_{0,2,4\dots 2m}$
($\rho^{\prime\prime}$) is some state over the qubits $0,2,4\dots 2m$ (all the
qubits). Next, we allow the qubits to evolve under the entanglement breaking
noise for a time $\tau_{d}$ — we note that this is equivalent to applying the
channel $\mathcal{E}_{\tau_{d}}=e^{\kappa\tau_{d}(\mathcal{N}-\text{Id})}$ on
each qubit. By a Taylor expansion of this exponential, it can easily be seen
that

$\mathcal{E}_{\tau_{d}}=e^{-\kappa\tau_{d}}\text{Id}+(1-e^{-\kappa\tau_{d}})\mathcal{N}^{\prime},$

where
$\mathcal{N}^{\prime}:\mathfrak{L}(\mathbb{C}^{2})\to\mathfrak{L}(\mathbb{C}^{2})$
is another entanglement breaking channel with the property that
$\mathcal{N}^{\prime}(\mathfrak{D}_{1}(\mathbb{C}^{2}))=\mathcal{N}(\mathfrak{D}_{1}(\mathbb{C}^{2}))$.
Therefore, with probability $1-e^{-\kappa\tau_{d}}$, this channel disentangled
the qubit and maps it to a state in
$\mathcal{N}(\mathfrak{D}_{1}(\mathbb{C}^{2}))$ and consequently,

$\rho(\tau_{s}+\tau_{d})=\mathcal{E}_{\tau_{d}}^{\otimes(2m+1)}\rho(\tau_{s})=(1-q^{\prime})\sigma_{0,2,4\dots
2m-2}\otimes\rho^{\prime\prime}_{1,3,5\dots
2m-1,2m}+q^{\prime}\rho^{\prime\prime\prime},$

where
$1-q^{\prime}=\exp(-2m\kappa\tau_{s})(1-q)+(1-\exp(-\kappa\tau_{d}))^{m}(1-\exp(-2\kappa\tau_{s})(1-q))$,
$\sigma_{0,2,4\dots 2m-2}$ is a seperable state of the form of Eq. 6 but over
the qubits $0,2,4\dots 2m-2$. It immediately follows that $q^{\prime}\leq q$
if

$\displaystyle\tau_{d}\geq-\frac{1}{\kappa}\log\bigg{(}1-\bigg{(}\frac{(1-q)(1-e^{-2m\kappa\tau_{s}})}{1-(1-q)e^{-2m\kappa\tau_{s}}}\bigg{)}^{1/m}\bigg{)}$
(7)

A similar condition holds for the next layer of SWAP gates — if $\tau_{d}$ is
chosen as per Eq. 7, then the final state of the $m$ qubits being shifted will
be maintained in a seperable state where the individual qubits are in
$\mathcal{N}(\mathfrak{D}_{1}(\mathbb{C}^{2}))$ with a probability at least
$1-q$ — hence this property will also be maintained by the subsequent shift
operation and we will have $m$ qubits available for the cooling procedure. We
note that even though the qubits now being purified are in a seperable state
as opposed to a produce state, since by assumption the eigenstates of the
individual qubit states are the same, the cooling unitary applied on the
product state will also cool the seperable state without any impact on the
fidelity of the cooling algorithm. However, the cooling algorithm’s fidelity
will be lowered by a factor $\geq 1-q$. Consequently, in order to implement
the RESTART operation with fidelity large enough to ensure that the threshold
theorem can be satisfied, we must be able to choose a sufficiently small $q$.
Since the shift operation outlined above relies on the entanglement breaking
channel to purify the state of the qubits, it is not clear if this condition
can be satisfied (for small rates $\kappa$) while at the same time ensuring
that the noise in the computational qubits is smaller than the fault-tolerance
threshold. However, this is indeed possible — to see this, we note that for
small $\kappa$, we can use $\tau_{d}$

$\tau_{d}\approx\bigg{(}\frac{2m(1-q)}{q}\bigg{)}^{1/m}\frac{(\kappa\tau_{s})^{1/m}}{\kappa}.$

However, note that for the error correction procedures to fault tolerantly
encode a computation on the computational qubit, it is necessary to ensure
that $\kappa\tau_{d}\leq c_{\text{th}}$, where $c_{\text{th}}$ is a constant
dependent on the spread of the error correcting code. It follows easily that
for a given $q$ and $m$ (which are determined by the target fidelity of the
cooling operation) if $\kappa<qc_{\text{th}}^{m}/2m\tau_{s}(1-q)$, then for
any
$\tau_{d}\in(({2m(1-q)}/{q})^{1/m}\kappa^{1/m-1}\tau_{s}^{1/m},c_{\text{th}}/\kappa)$,
the shift operation will successfully bring the auxillary qubits near the
computational qubits for the RESTART operation while at the same time ensuring
that a quantum computation can be fault tolerantly performed on the
computational qubits.

## References

  * Grimmett (1997) G. Grimmett, in  _Lectures on probability theory and statistics_ (Springer, 1997) pp. 153–300.
  * Bazant (2000) M. Z. Bazant, Physical Review E 62, 1660 (2000).
  * Childs  _et al._ (2021) A. M. Childs, Y. Su, M. C. Tran, N. Wiebe, and S. Zhu, Physical Review X 11, 011020 (2021).
  * Aharonov and Ben-Or (2008) D. Aharonov and M. Ben-Or, SIAM Journal on Computing (2008).
  * Gottesman (2000) D. Gottesman, Journal of Modern Optics 47, 333 (2000).
  * Schulman and Vazirani (1999) L. J. Schulman and U. V. Vazirani, in  _Proceedings of the thirty-first annual ACM symposium on Theory of computing_ (1999) pp. 322–329.
  * Boykin  _et al._ (2002) P. O. Boykin, T. Mor, V. Roychowdhury, F. Vatan, and R. Vrijen, Proceedings of the National Academy of Sciences 99, 3388 (2002).


11institutetext: Search Solutions Inc., Republic of Korea
11email<EMAIL_ADDRESS>
# Semi-supervised Learning with a Teacher-student Network for Generalized
Attribute Prediction

Minchul Shin 11 0000-0002-0638-2017

###### Abstract

This paper presents a study on semi-supervised learning to solve the visual
attribute prediction problem. In many applications of vision algorithms, the
precise recognition of visual attributes of objects is important but still
challenging. This is because defining a class hierarchy of attributes is
ambiguous, so training data inevitably suffer from class imbalance and label
sparsity, leading to a lack of effective annotations. An intuitive solution is
to find a method to effectively learn image representations by utilizing
unlabeled images. With that in mind, we propose a multi-teacher-single-student
(MTSS) approach inspired by the multi-task learning and the distillation of
semi-supervised learning. Our MTSS learns task-specific domain experts called
teacher networks using the label embedding technique and learns a unified
model called a student network by forcing a model to mimic the distributions
learned by domain experts. Our experiments demonstrate that our method not
only achieves competitive performance on various benchmarks for fashion
attribute prediction, but also improves robustness and cross-domain
adaptability for unseen domains.

###### Keywords:

Semi-supervised Learning, Unlabeled Data, Visual Attributes

##  1 Introduction

Visual attributes are qualities of objects that facilitate the cognitive
processes of human beings. Therefore, predicting the attributes of an object
accurately has many useful applications in the real world. For example, a
search engine can use predictions to screen products with undesirable
attributes instead of using noisy metadata provided by anonymous sellers [2].
Attribute prediction is essentially a multi-label classification problem that
aims to determine if an image contains certain attributes (e.g., colors and
patterns). However, attribute prediction is known as a very challenging task
based on the expense of annotation, difficulty in defining a class hierarchy,
and simultaneous appearance of multiple attributes in objects. Although recent
works have shown competitive results on various benchmark datasets [7, 12], we
identified several additional conditions that must be satisfied to solve the
aforementioned issues for such methods to be useful in real-world
applications: domain-agnostic training, the use of unlabeled data, and
robustness/generalization.

Figure 1: Two different types of visual attribute prediction tasks. Image (a)
was sampled from MSCOCO [21] and image (b) was sampled from DeepFashion [23].

Use of unlabeled data. The human perception of an attribute is intrinsically
subjective, leading to sparse and ambiguous annotations in datasets.
Additionally, attributes have very-long-tailed distributions, leading to
severe class imbalance (e.g., pattern: solid versus zebra). Therefore, even if
the total number of annotated images is large, models can suffer from extreme
shortages of usable training data following class balancing in many cases [9].
An intuitive solution is to use unlabeled images for training in a semi-
supervised manner to enhance the ability of a model to represent the visual
information included in images. The teacher-student paradigm proposed in this
paper facilitates this process with no preprocessing tricks, such as obtaining
pseudo-labels based on predictions [5, 41, 40].

Robustness/generalization. Robustness measures how stable and reliable a
network is when making decisions in the presence of unexpected perturbations
in inputs [43]. Generalization measures how well a trained model performs with
a domain shift or if a model can be generalized sufficiently with insufficient
training data [17, 26]. In addition to high accuracy on benchmarks, these
aspects must also be considered as crucial factors for measuring model
quality.

Domain-agnostic training. To achieve state-of-the-art performance on
benchmarks, recent works [42, 23, 22, 4] have relied on domain-specific
auxiliary information during training. However, such methods have obvious
limitations in terms of expansion to additional domains, which is an important
feature for real applications. Therefore, it is preferable to avoid using any
domain-specific information during training. We refer to this principle as
domain-agnostic training.

Main contributions. In this paper, we propose a multi-teacher-single-student
(MTSS) method for generalized visual attribute prediction that aims to train a
single unified model to predict all different attributes in a single forward
operation. This paper makes the following main contributions. First, we
introduce an MTSS method that learns from multiple domain experts in a semi-
supervised manner utilizing unlabeled images. We also show the advantages of
label embedding for training teacher networks, which are used as domain
experts. Second, we demonstrate that the MTSS method can enhance model quality
in terms of robustness and generalization without sacrificing benchmark
performance by learning from images in distinct domains. Finally, our approach
is fully domain agnostic, meaning it requires no domain-specific auxiliary
supervision, such as landmarks, pose detection, or text description, for
training. By learning using only attribute labels, our model not only
outperforms previous methods under the same conditions [8, 6, 16], but also
achieves competitive results relative to existing domain-specific methods [23,
38, 32, 22] that use additional auxiliary labels for supervision.

Figure 2: Conceptual summary of previous methods and the proposed method
denoted, as common [42, 23, 22, 4], progressive [2], and MTSS.

##  2 Related Work

Visual Attributes. Visual attribute prediction (VAP) is a multi-label
classification task that has been widely studied [42, 4, 2, 22, 38, 23, 32,
30]. There are two main types of VAP tasks, as illustrated in Figure 1. We are
particularly interested in type (b), which is frequently observed in the
fashion domain. This is because our interests lie in the ability of a model to
predict multiple simultaneously appearing attributes, rather than localizing
interest regions, which is a matter of detection. The attributes appearing in
type (b) are not necessarily related to visual similarity. For example, they
may only be related to low-level characteristics, such as color, which makes
the task more interesting. In the fashion domain, some works [42, 23, 22, 4]
have used landmarks for clothing items, pose detection, or textual item
descriptions to improve overall accuracy. However, such strong requirements
regarding auxiliary information limit such methods to domain-specific
solutions. In contrast, some works have focused on attention mechanisms [38,
32, 42]. Zhang et al. [42] proposed a task-aware attention mechanism that
considers the locality of clothing attributes according to different tasks.
Wang et al. [38] proposed landmark-aware attention and category-driven
attention. The former focuses on the functional parts of clothing items and
the latter enhances task-related features that are learned in an implicit
manner. VSAM [32] uses the inference results of pose detection for supervision
to train a pose-guided attention model. Although attention mechanisms are
well-known methods for boosting performance, such mechanisms mainly discuss
where to focus given spatial regions. Because we consider attention mechanisms
as an extension that can be applied to any existing method with slight
modifications, such mechanisms are given little attention in this work. One of
main challenges in VAP is solving multi-task learning problems in the presence
of label sparsity and class imbalances in training data [1]. A single unified
model is preferred for saving on the cost of inference and is expected to
provide robust predictions for all targets (e.g., style, texture, and patterns
in fashion). The most commonly used method is to train multiple binary
classifiers [23, 8]. However, because the numbers of effective annotations for
each task differ significantly based on label sparsity, models can easily
encounter overfitting on tasks with many annotations and underfitting on tasks
with few annotations over the same number of iterations. Balancing such bias
during training is important issue that must be solved for VAP. Adhikari et
al. [2] proposed a progressive learning approach that add branches for
individual models for attributes progressively as training proceeds. Lu et al.
[24] proposed a tree-like deep architecture that automatically widens the
network for multi-task learning in a greedy manner. However, the
implementation of such methods is tricky and requires considerable engineering
work.

Label Embedding. Label embedding (LE) refers to an important family of multi-
label classification algorithms that have been introduced in various studies
[3, 36, 28]. Such approaches jointly learn mapping functions $\varphi$ and
$\lambda$ that project embeddings of an image $x$ and label $y$ in a common
intermediate space $Z$. Compared to direct attribute prediction, which
requires training a single binary classifier for each attribute, LE has the
following advantages. First, attributes are not required to be independent
because one can simply move the embedding of $x$ closer to the correct label
$y$ than to any incorrect labels $y^{\prime}$ based on ranking loss [15, 37].
Second, prediction classes can be readily expanded because classes can be
predicted by measuring the shortest distance to the center point of a feature
cluster assigned to an unseen class [19, 35, 18, 33]. We found a number of
studies examining LE in terms of zero-shot learning. In this paper, we
highlight the usage of LE to train domain experts for an MTSS model for semi-
supervised learning.

Semi-supervised Learning and Distillation. Recent works [5, 41, 40] have
investigated semi-supervised learning (SSL) methods [10] that use large-scale
unlabeled images to improve supervision. Yalniz et. al. [41] proposed a self-
training method representing a special form of SSL and achieved state-of-the-
art accuracy by leveraging a large amount of unannotated data. Distillation
can also be considered as a form of SSL in that a teacher model makes
predictions for unlabeled data and the results are used for supervision to
train a student model [29, 11, 31]. Such strategies have yielded impressive
results on many vision tasks. Inspired by these methods, we propose a training
method that takes advantage of SSL for VAP. We found that a teacher-student
paradigm of distillation is very effective for performing VAP because relevant
visual attributes can be grouped into particular attribute types (e.g.
$\\{stripe,dot\\}\in pattern$), meaning a student model can effectively learn
from multiple teachers that are specialists for each attribute type.

##  3 Methodology

###  3.1 Overview

The main concept of the MTSS approach is to integrate multiple teachers (MT)
that are domain experts for each attribute type into a single student (SS) to
construct a unified model that can predict multiple attributes in a single
forward operation. Given an image $x$ and attribute type
$\alpha_{k}\in\\{\alpha_{1},\alpha_{2},...,\alpha_{\kappa}\\}$, our goal is to
predict the attribute classes
$c_{p}\in\\{c^{\alpha_{k}}_{1},c^{\alpha_{k}}_{2},...,c^{\alpha_{k}}_{P}\\}$,
where $\kappa$ and $P$ are the number of target attribute types and final
predictions to be outputted, respectively. Suppose $\alpha_{k}$ is a pattern
and color. Then, the corresponding $c_{p}$ could be dot, stripe or red. $P$
may differ depending on which confidence score is used for the predicted
result to reach a final decision regarding each attribute type $\alpha_{k}$.
Our assumption is that attributes existing in the real world have a conceptual
hierarchy, implying that relevant visual attributes are grouped into a
particular attribute type $\alpha_{k}$. For example, $>1000$ attributes in
DeepFashion are grouped into 6 attribute types which means only 6 teachers are
required for training student. The training procedure can be divided into two
stages for the teacher and student. The design details and advantages of our
two-stage training method are discussed in the following subsections.

###  3.2 Teacher Models for Individual Attributes

Given pairs of an image $x$ and ground-truth label $y$ in a training set
$\zeta_{\alpha_{k}}=\\{(x_{n},y_{n}),n=1,...,N\\}$ for an attribute type
$\alpha_{k}$ with $x_{n}\in X$ and $y_{n}\in Y$, our goal is to train a
teacher model $\varphi^{T}_{\alpha_{k}}:X\rightarrow Y$. Our teacher model
uses the pairwise ranking loss of metric learning to learn image
representations. Specifically, a group of randomly initialized label
embeddings obtained from an attribute dictionary
$d^{n}_{\alpha_{k}}=\lambda_{\alpha_{k}}(y_{n})$, where $d^{n}_{\alpha_{k}}\in
R^{D}$ and $D$ denotes the dimension of the image embedding, is defined and
$d^{n}_{\alpha_{k}}$ is learned to represent the center of the feature cluster
of the $n$-th class for attribute type $\alpha_{k}$. Given a label embedding
$d^{n}_{\alpha_{k}}$ that corresponds to the most representative feature point
of the $n$-th class of $\alpha_{k}$, our goal is to locate an embedding of the
positive image $e^{+}=\varphi^{T}_{\alpha_{k}}(I^{+})$ to be close to
$d^{n}_{\alpha_{k}}$ and an embedding of the negative image
$e^{-}=\varphi^{T}_{\alpha_{k}}(I^{-})$ far away. The positive image is
sampled from the same attribute class as $d^{n}_{\alpha_{k}}$ and the negative
image is sampled from a randomly selected class that is different from the
class of the positive image. Although various sampling strategies can be
adopted to enhance overall performance [39], such optimization was omitted in
this work because it lies outside of our focus. The general form of an
objective can be written as (1).

$\ell^{\prime}(d^{n}_{\alpha_{k}},e^{+},e^{-})=max\\{0,(1-\lVert
d^{n}_{\alpha_{k}}\rVert_{2}\cdot\lVert e^{+}\rVert_{2}+\lVert
d^{n}_{\alpha_{k}}\rVert_{2}\cdot\lVert e^{-}\rVert_{2})\\},\\\ $ (1)

where $\left\|\right\|_{2}$ represents L2 normalization. Lin et al. [20]
introduced the concept of focal loss, which can help a model focus on
difficult and misclassified examples by multiplying a factor
$(1-p_{t})^{\gamma}$ by the standard cross-entropy loss, where $p_{t}$ is the
estimated probability for the target class. As a modification for metric
learning, we applied focal loss to our method by defining the probability
$p_{t}$ as the cosine similarity between $d$ and $e$, which is bounded in the
range of $[0,1]$. The final objective is defined as (2).

$\begin{split}p_{t}=0.5\times max\\{0,(1+\lVert
d^{n}_{\alpha_{k}}\rVert_{2}\cdot\lVert e^{+}\rVert_{2}-\lVert
d^{n}_{\alpha_{k}}\rVert_{2}\cdot\lVert e^{-}\rVert_{2})\\},\\\
\ell^{Teacher}(p_{t})=-(1-p_{t})^{\gamma}log(p_{t}),\quad\quad\quad\quad\quad\quad\\\
\end{split}$ (2)

where $\gamma$ is the hyperparameter to be found. Note that $\gamma$ was set
to 1.0 in our experiments unless stated otherwise. We found that if $\gamma$
is adjusted carefully, adopting the modified focal loss term significantly
boosts overall scores, which will be discussed in the experimental section.
During training, an image can be duplicated in more than one attribute class
if the multiple labels exist in the ground truth. We expect the number of
experts (i.e., trained teacher models
$\varphi^{T}\in\\{\varphi^{T}_{\alpha_{1}},\varphi^{T}_{\alpha_{2}},...,\varphi^{T}_{\alpha_{\kappa}}\\}$)
to be the same as the number of attribute types $\kappa$.

Figure 3: Training of a teacher model. The feature extractor $\varphi^{T}$ and
mapping $\lambda$ are learned to place the image embedding (left) and label
embedding (right) close to each other in latent space. Figure 4: Training of a
student model. The teachers, which represent individual experts for each
attribute type, are independent models, unlike the unified student model that
shares weights before its fully connected layers.

###  3.3 Unified Student Model for All Attributes

The goal of the student stage is to integrate the trained teacher models
$\varphi^{T}\in\\{\varphi^{T}_{\alpha_{1}},\varphi^{T}_{\alpha_{2}},...,\varphi^{T}_{\alpha_{k}}\\}$
into a single unified network $\varphi^{S}$ and boost overall performance by
utilizing unlabeled images $\mathcal{U}$ in a semi-supervised manner. A
distillation method that aims to transfer knowledge acquired in one model,
namely a teacher, to another model, namely a student, was adopted for this
purpose. The core idea of distillation is very simple. A student model is
trained to mimic the feature distribution of $\varphi^{T}$. Learning
distributions from trained models can be achieved simply by matching the image
embedding of a teacher $e^{T}_{\alpha_{k}}=\varphi^{T}_{\alpha_{k}}(x)$ to the
embedding of a student $e^{S}_{\alpha_{k}}=\varphi^{S}(x;\alpha_{k})$
according to a target attribute type $\alpha_{k}$. It is assumed that if
$\varphi^{S}$ is able to reproduce the same feature distribution as
$\varphi^{T}$, then prediction can be performed by simply measuring the
distance between $e^{S}_{\alpha_{k}}$ and a learned dictionary
$d_{\alpha_{k}}\in\\{d^{1}_{\alpha_{k}},d^{2}_{\alpha_{k}},...,d^{N}_{\alpha_{k}}\\}$
that was already obtained in the teacher stage. A student $\varphi^{S}$
consists of $S$ branches of fully-connected layers following the last pooling
layer of the backbone, where each branch is in charge of projecting pooled
descriptors into an attribute-specific embedding space $Z_{\alpha_{k}}\in
R^{D}$. The weights before the fully connected branches are shared. Given an
attribute type $\alpha_{k}$, we maximize the cosine-similarity between
$e^{S}_{\alpha_{k}}$ and $e^{T}_{\alpha_{k}}$. Therefore objective is
formulated as

$\ell^{\prime\prime}=\Sigma_{k=1}^{\kappa}\\{1-\lVert
e^{S}_{\alpha_{k}}\rVert_{2}\cdot\lVert e^{T}_{\alpha_{k}}\rVert_{2}\\},\\\ $
(3)

where $\left\|\right\|_{2}$ is L2 normalization. Because $\varphi^{S}$ learns
from $\varphi^{T}$, training can be unstable with large $lr$ values if
$\varphi^{T}$ frequently produces outlier points based on a lack of
generalization or excessive input noise. To alleviate such unwanted effects,
we assign additional weight to an objective if the distance from the embedding
of teacher $e^{T}_{\alpha_{k}}$ to the closest label embedding
$\hat{d}_{\alpha_{k}}$ is small because a smaller distance indicates a more
certain prediction. A large distance to $\hat{d}_{\alpha_{k}}$ could indicate
the presence of outliers, so such signals are suppressed during gradient
updating. The final objective is formulated as

$\ell^{Student}=\Sigma_{k=1}^{\kappa}\\{(\lVert\hat{d}_{\alpha_{k}}\rVert_{2}\cdot\lVert
e^{T}_{\alpha_{k}}\rVert_{2})^{\beta}\times(1-\lVert
e^{S}_{\alpha_{k}}\rVert_{2}\cdot\lVert e^{T}_{\alpha_{k}}\rVert_{2})\\},\\\ $
(4)

where $\beta$ is the hyperparameter to be found. $\beta$ is set to 1.0 unless
stated otherwise. Although we observed that $\beta$ has a very small effect on
final accuracy, this setting enables to use a large $lr$ value at the very
beginning of training phase by suppressing noise.

###  3.4 Query Inference

Given a query image $x$ and ground-truth label $y$ of an attribute type
$\alpha_{k}$, the attribute-specific embedding of $\alpha_{k}$ can be
calculated as $e^{S}_{\alpha_{k}}=\varphi^{S}_{\alpha_{k}}(x;\alpha_{k})$. Let
the dictionary learned at the teacher stage
$d_{\alpha_{k}}=\lambda_{\alpha_{k}}(y)$, where
$d_{\alpha_{k}}\in\\{d^{1}_{\alpha_{k}},...,d^{N}_{\alpha_{k}}\\}$ and $N$ is
the number of classes included in $\alpha_{k}$. Because $d_{\alpha_{k}}$
represents the center points of the feature cluster for each ground-truth
class $y$ and $\varphi^{S}_{\alpha_{k}}$ can reproduce the same distribution
with $\varphi^{T}_{\alpha_{k}}$, a prediction can be obtained by finding the
$\hat{d}_{\alpha_{k}}$ that maximizes the cosine similarity with
$e^{S}_{\alpha_{k}}$. An example of the query inference process for predicting
a pattern is presented in Figure 5.

Figure 5: An example of the query inference process.

##  4 Experiment

###  4.1 Experimental Setup

Datasets. All models were evaluated on several fashion-related benchmarks,
namely three public datasets called iMaterialist-Fashion-2018 (iMatFashion)
[13], Deepfashion Category and Attribute Prediction Benchmark (DeepFashion)
[23], and DARN [16], and one private dataset called FiccY. We cropped the
images using a CenterNet [44] based fashion object detector unless ground-
truth bounding boxes were not provided for a dataset. We omit the detailed
statistics of the public datasets because they have been well described in
many related works [8, 38, 13, 34]. FiccY is a private fashion dataset
collected from our database and annotated by human experts. It contains 520K
high-resolution images labeled for four types of attributes (Category, Color,
Pattern, and Texture) produced by both sellers and wearers. We intentionally
include the result on FiccY in our experiments because it cab reflect a real
service environment. However, the majority of experiments were performed on
the DeepFashion benchmark for reproducibility.

###  4.2 Performance on Benchmarks

Multi-Label Classification. Previous studies [6, 16, 23, 8, 38, 32, 22] on
attribute prediction in the fashion domain have reported R@3 scores according
to attribute types using DeepFashion, so we used the same metric for fair
comparison. The results are listed in Table 1. The R@3 scores of both the
teacher and student are listed for various experimental settings of $\gamma$
and $lr$. The results demonstrate that with the default hyperparameter of
$\gamma=1$, our teacher model outperforms previous methods [8, 6, 16] that use
no auxiliary supervision. We determined that the optimized values of $\gamma$
and $lr$ differ depending on the task, meaning these hyperparameters must be
selected carefully according to the target attribute type. When training a
student, we observed higher gains when a greater $lr$ value was adopted.
Adjusting the $\beta$ value enables the use of greater $lr$ values while
avoiding instability at the beginning of training. The student trained by the
tuned teachers yields very similar results compared to state-of-the-art
domain-specific methods [23, 38, 32, 22].

Table 1: Top-k recall for attribute prediction on the DeepFashion [23]
dataset. $\ast$-marked methods use additional labels for training, such as
landmark, pose, or text descriptions. $\gamma=tune$ indicates that the best-
performing $\gamma$ was selected according to the attribute types. Recall was
measured when the model achieved the best F1@1 score. Style is not compared to
the other methods because style scores were not reproducible using publicly
released code. Overall $1^{st}$/$2^{nd}$ best in blue/green.

Category Texture Fabric Shape Part Style All Method top-3 top-5 top-3 top-5
top-3 top-5 top-3 top-5 top-3 top-5 top-3 top-5 top-3 top-5 WTBI [6] 43.73
66.26 24.21 32.65 25.38 36.06 23.39 31.26 26.31 33.24 - - 27.46 35.37 DARN
[16] 59.48 79.58 36.15 48.15 36.64 48.52 35.89 46.93 39.17 50.14 - - 42.35
51.95 $\ast$ FashionNet [23] 82.58 90.17 37.46 49.52 39.30 49.84 39.47 48.59
44.13 54.02 - - 45.52 54.61 Corbiere et al. [8] 86.30 92.80 53.60 63.20 39.10
48.80 50.10 59.50 38.80 48.90 - - 23.10 30.40 $\ast$ Wang et al. [38] 90.99
95.78 50.31 65.48 40.31 48.23 53.32 61.05 40.65 56.32 - - 51.53 60.95 $\ast$
VSAM + FL [32] - - 56.28 65.45 41.73 52.01 55.69 65.40 43.20 53.95 - - - -
$\ast$ Liu et al. [22] 91.16 96.12 56.17 65.83 43.20 53.52 58.28 67.80 46.97
57.42 - - - - Ours(T)($\gamma=0$) 88.45 91.21 55.91 62.02 43.90 50.54 55.92
61.46 45.40 51.55 30.13 35.44 - - Ours(T)($\gamma=1$) 89.32 92.92 57.89 64.71
44.51 51.57 56.22 62.34 46.09 52.73 30.88 36.19 - - Ours(T)($\gamma=tune$)
90.01 93.74 58.14 65.29 44.93 52.18 56.74 62.66 46.83 53.66 31.31 37.13 - -
Ours(S)($\gamma=1,\beta=1$) 89.65 93.29 58.33 65.17 45.05 52.26 56.91 63.17
46.98 54.30 32.30 38.41 57.85 63.91 Ours(S)($\gamma=tune,\beta=1$) 90.17 93.98
57.58 64.66 44.87 52.47 57.34 63.69 47.36 54.99 32.64 39.24 58.02 64.35

LE versus Binary Cross-Entropy In this section, we analyze the strength of LE
compared to the most commonly used method, namely binary cross-entropy (BCE).
Table 2 compares our implementation to the existing method. The student model
is trained by the teacher networks, which are trained using LE. The state-of-
the-art method based on DeepFashion uses BCE for training. One can see that LE
consistently yields better results when k=1, implying that prediction is more
accurate with a small number of trials. We directly compared the LE and BCE
methods by replacing the objective of our teacher model with BCE. The results
in Table 3 reveal consistent improvements in terms of recall at k=1 for all
evaluation datasets.

Figure 6: Effects of training hyperparameters evaluated on DeepFashion: (a)
gamma $\gamma$, (b) embedding dimension $D$, and (c) training data size. These
plots represent analysis of the changes in R@3 (gain or degradation) compared
to the baseline trained with (a) $\gamma=0$, (b) $D=256$, and (c) a data size
equal to the size of the entire training set. Part (c) presents results for
both the student (S) and teacher (T). Table 2: The top-k F1 scores and recall
values evaluated on DeepFashion.

Method F1@1 R@1 F1@3 R@3 F1@5 R@5 Liu et al. [22] 31.35 37.17 22.50 57.30
16.82 65.66 Ours(S)($\gamma=tune,\beta=1$) 33.53 40.11 22.41 58.02 16.16 64.35

Hyperparameter tuning. The optimal value of $\gamma$ for training a teacher
network is analyzed in Figure 6 (a). While the use of focal loss consistently
improves the results by up to 3% in terms of R@3, we found that the optimal
value differs depending on the attribute types. Therefore, it is highly
recommended to tune $\gamma$ carefully for each attribute type. The feature
dimension is also an important engineering factor. Figure 6 (b) presents the
effects of the embedding dimension $D$. Better results can be observed for
higher dimension, but processing high-dimensional features may require more
resources and inference time. The result in Table 1 demonstrate that tuning
the hyperparameters of not only $\gamma$ and $D$, but also $lr$, significantly
improves performance. The combination of optimal values is different for each
attribute type.

Table 3: Comparison between LE and BCE. The F1@1 scores were measured on
FiccY.

Dataset Category Pattern Color Texture Avg. $\Delta$ BCE 85.12 67.46 65.81
57.43 - LE 85.52 67.71 66.31 57.98 +0.62%

Training depending on data size. Figure 6 (c) presents the performance
degradation when the number of usable training images decreases. Overall R@3
scores are presented for both the teacher and student networks. The number of
training images was intentionally limited to observe how performance degrades
with a shortage of training data. As expected, a lack of training data
significantly degrades the overall results. It is noteworthy that the student
always outperforms the teachers slightly in terms of gain, even though the
teachers are very weak. However, this also indicates that the final
performance of the student is strongly bounded by that of teachers. We found
no relationship between training data size and the amount of performance gain
for a student compared to a teacher.

Table 4: Performance comparison between a teacher (T) and student (S) model.
F1@1 scores are measured on the iMatFashion, DeepFashion, and FiccY datasets.

iMatFashion [13] Category Gender Material Pattern Style Neckline Sleeve Color
T 42.50 88.50 49.27 37.84 27.70 40.11 76.72 47.75 S 42.43 90.46 50.45 38.04
28.60 41.93 77.66 49.69

DeepFashion [23] Category Texture Fabric Shape Part Style T 74.45 25.61 24.97
30.45 21.12 12.63 S 74.85 26.08 25.14 30.81 21.40 13.29

FiccY Category Pattern Color Texture T 85.10 67.69 66.15 57.74 S 85.78 68.31
67.60 59.19

###  4.3 Effectiveness of Distillation

Comparison between a teacher and student. In Table 4, we compare
classification performances between a teacher and student on the DeepFashion,
iMatFashion, and FiccY datasets. Results were evaluated for all three datasets
to determine if the proposed method exhibits consistent experimental trends in
different domains. The results illustrate that the student achieves a better
score than the teachers in all cases. Our interpretation is that the use of
unlabeled images can induce improvements because the negative effects caused
by missed annotations are successfully suppressed during training by matching
features from teacher and student for supervision. As a proof, only slight
improvements in terms of F1 score are observed for categories which has
relatively dense annotations than the others. Our MTSS approach is a semi-
supervised form of multitask learning, so we analyze the benefits of learning
representations simultaneously from multiple tasks in Table 5. Given four
different attribute types for FiccY, we compare students trained with the
supervision of either single or multiple domain experts. We found consistent
improvements in both cases. However, higher improvement is achieved in general
with supervision from all four teachers than a single teacher. We can conclude
that the proposed model benefits from knowledge transfer between different
tasks, which were represented by different attribute types in this experiment.
Figure 8 analyzes the differences in R@3 scores between teachers and students
for each class with $\alpha_{\kappa}=shape$. Although the result by class is
generally improved based on distillation, performance degradation occurs in
some classes. Such results are found when the recall of a teacher is poor
meaning that poorly generalized teacher networks can negatively affect
optimization for minor classes, while having a small effect on overall
performance. Another possibility is that certain rare attributes lead to
underfitting for certain classes, even with a large number of training epochs.
Regardless, the overall results with distillation for all classes exhibit
consistent improvement.

Table 5: R@3 gain of a student model compared to teachers with either single
(S-Single) or multiple (S-Multi) teachers evaluated on FiccY.
$\Delta_{single}$ and $\Delta_{multi}$ denote the results for S-Single and
S-Multi, respectively, as percentages.

Attribute T S-Single $\Delta_{single}$ S-Multi $\Delta_{multi}$
$\Delta_{multi}$ \- $\Delta_{single}$ Category 96.70 97.30 +0.62% 97.30 +0.62%
+0.00% Pattern 90.11 90.73 +0.69% 91.14 +1.14% +0.45% Color 88.97 89.54 +0.64%
89.98 +1.14% +0.50% Texture 92.06 93.57 +1.64% 94.22 +2.35% +0.71%

t-SNE visualization of distributions. The distribution of teacher and student
embeddings is visualized in Figure 7. The result illustrates that the student
reproduces almost the same distribution as the teacher. We omitted
visualizations for attribute types other than patterns because similar results
were observed for all attribute types. The feature clusters for each
prediction class are clearly formed for both the teacher and student.

Figure 7: T-distributed stochastic neighbor embedding (t-SNE) [25]
visualization of pattern attribute types for FiccY. The left and right parts
present the distributions of image embeddings extracted from the teacher and
student models, respectively.  Table 6: Cross-domain adaptability. F1@1 scores
of five commonly appearing classes were measured on DARN [16]. The symbol †
indicates a model trained with both original training set images and external
large-scale unlabeled images.

Model Train set Test set Flower Stripe Dot Check Leopard All T DeepFashion
DARN 70.46 66.11 67.47 51.00 40.90 64.31 S DeepFashion DARN 71.16 67.81 68.97
52.02 43.97 65.15 $S^{\textsuperscript{\textdagger}}$ DeepFashion+DARN DARN
71.84 68.70 70.03 52.62 44.09 65.49 T iMatFashion DARN 74.73 75.33 74.25 68.53
51.84 72.34 S iMatFashion DARN 75.37 76.74 74.90 69.77 55.72 73.10
$S^{\textsuperscript{\textdagger}}$ iMatFashion+DARN DARN 75.81 76.78 75.20
70.34 57.61 73.22

Table 7: Results for robustness to corruption and perturbation in inputs. Recall was measured on both clean and corrupted versions of the DeepFashion dataset. The symbol $\dagger$ indicates a model trained with both clean images and 1M, 10M, or 15M corrupted images. Model | Mixing | Clean R@3 | $\Delta$ | Corrupted R@3 | $\Delta$
---|---|---|---|---|---
Baseline(S) | No | 57.47 | - | 42.96 | -
$S^{\dagger}$-1M | Yes | 57.75 | +0.49% | 43.96 | +2.33%
$S^{\dagger}$-10M | Yes | 57.80 | +0.57% | 43.88 | +2.14%
$S^{\dagger}$-15M | Yes | 57.74 | +0.47% | 43.84 | +2.05%

###  4.4 Robustness and Generalization

Cross-domain adaptability. In this subsection, we examine the capability of
SSL to transfer knowledge for adaptation to a new target domain [27].
Validating the cross-domain adaptability of VAP is difficult because the
classes provided in each dataset differ. For fair comparison, we carefully
selected five commonly appearing classes (Flower, Stripe, Dot, Check, and
Leopard) for three datasets (DeepFashion, iMatFashion, and DARN) and created
subsets containing only images from one of the selected classes. The results
in Table 6 reveal that training a student is effective for cross-domain
classification. We observe clear improvements for all five classes relative to
the teacher models. We also examined the strength of SSL useful for deploying
a model in a realistic environment. Because the training process for a student
requires no annotations, the training images for two distinct domains (DARN
and DeepFashion/iMatFashion) were mixed for training. This combination
provided a significant performance boost for the testing set. This could be an
useful feature in real production environments because a model can be easily
fine-tuned for a service domain after being trained in a constrained
environment.

Robustness to corruption and perturbation. Hendrycks et al. [14] introduced a
benchmark for image classifier robustness to corruption and perturbation.
Images were transformed using 15 types of algorithmically generated
corruptions with five levels of severity, resulting in a total of 15 $\times$
5 corrupted images. By following this same procedure, we created a corrupted
version of the DeepFashion dataset and measured the drop in R@3 compared to
the clean version. The result are listed in Table 7. Testing on the corrupted
images degrades the overall R@3 score by an average of approximately 25%. In
Table 6, the benefits of mixing two distinct domains in terms of domain
adaptation were explored. Following the same strategy, we mixed clean images
with 1M, 10M, and 15M randomly sampled corrupted images and trained a student
model to improve robustness to corruption. It should be noted that the mixed
images were only used for training the student, meaning the teachers never saw
the corrupted images. The results reveal that a student trained with mixed
images achieves an increase of up to 2.33% in terms of R@3 score compared to a
baseline student trained using only clean images. Additionally, performing
fine-tuning using corrupted images does not degrade the model’s performance on
clean images. In fact, such fine-tuning slightly improves performance on clean
images. The greatest performance gain can be observed when including only 1M
corrupted images. The optimal ratio between clean and corrupted images should
be identified because the noisy signals generated by corrupted images can make
optimization less effective if they dominate the clean images.

Figure 8: R@3 scores for each class of attribute type shape for the
DeepFashion dataset. The yellow line represents the changes in performance
before and after training the student. The green line is a baseline
representing a change of zero percent.

##  5 Conclusion

In this paper, we proposed an MTSS approach to solving the VAP problem. Our
method trains a unified model according to multiple domain experts, which
enables it to predict multiple attributes that appear simultaneously in
objects using a single forward operation. The core idea of MTSS is to transfer
knowledge by forcing a student to reproduce the feature distributions learned
by teachers. We demonstrated that such a strategy is highly effective for VAP,
which suffers from a lack of effective annotations. Furthermore, our method
can achieve competitive results on benchmarks using attribute labels alone,
and improve robustness and domain adaptability without sacrificing accuracy
fine-tuning with unlabeled images.

## References

  * [1] Abdulnabi, A.H., Wang, G., Lu, J., Jia, K.: Multi-task cnn model for attribute prediction. IEEE Transactions on Multimedia 17(11), 1949–1959 (2015)
  * [2] Adhikari, S.S., Singh, S., Rajagopal, A., Rajan, A.: Progressive fashion attribute extraction. arXiv preprint arXiv:1907.00157 (2019)
  * [3] Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence 38(7), 1425–1438 (2015)
  * [4] Arslan, H.S., Sirts, K., Fishel, M., Anbarjafari, G.: Multimodal sequential fashion attribute prediction. Information 10(10), 308 (2019)
  * [5] Cevikalp, H., Benligiray, B., Gerek, O.N.: Semi-supervised robust deep neural networks for multi-label image classification. Pattern Recognition p. 107164 (2019)
  * [6] Chen, H., Gallagher, A., Girod, B.: Describing clothing by semantic attributes. In: European conference on computer vision. pp. 609–623. Springer (2012)
  * [7] Chen, Z.M., Wei, X.S., Wang, P., Guo, Y.: Multi-label image recognition with graph convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5177–5186 (2019)
  * [8] Corbiere, C., Ben-Younes, H., Ramé, A., Ollion, C.: Leveraging weakly annotated data for fashion image retrieval and label prediction. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2268–2274 (2017)
  * [9] Dal Pozzolo, A., Caelen, O., Bontempi, G.: When is undersampling effective in unbalanced classification tasks? In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 200–215. Springer (2015)
  * [10] van Engelen, J.E., Hoos, H.H.: A survey on semi-supervised learning. Machine Learning pp. 1–68 (2019)
  * [11] Gong, C., Chang, X., Fang, M., Yang, J.: Teaching semi-supervised classifier via generalized distillation. In: IJCAI. pp. 2156–2162 (2018)
  * [12] Guo, H., Zheng, K., Fan, X., Yu, H., Wang, S.: Visual attention consistency under image transforms for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 729–739 (2019)
  * [13] Guo, S., Huang, W., Zhang, X., Srikhanta, P., Cui, Y., Li, Y., Belongie, S., Adam, H., Scott, M.: The imaterialist fashion attribute dataset. arXiv preprint arXiv:1906.05750 (2019)
  * [14] Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 (2019)
  * [15] Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International Workshop on Similarity-Based Pattern Recognition. pp. 84–92. Springer (2015)
  * [16] Huang, J., Feris, R.S., Chen, Q., Yan, S.: Cross-domain image retrieval with a dual attribute-aware ranking network. In: Proceedings of the IEEE international conference on computer vision. pp. 1062–1070 (2015)
  * [17] Kawaguchi, K., Kaelbling, L.P., Bengio, Y.: Generalization in deep learning. arXiv preprint arXiv:1710.05468 (2017)
  * [18] Khodadadeh, S., Boloni, L., Shah, M.: Unsupervised meta-learning for few-shot image classification. In: Advances in Neural Information Processing Systems. pp. 10132–10142 (2019)
  * [19] Kim, J., Kim, T., Kim, S., Yoo, C.D.: Edge-labeling graph neural network for few-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11–20 (2019)
  * [20] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
  * [21] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [22] Liu, J., Lu, H.: Deep fashion analysis with feature map upsampling and landmark-driven attention. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 0–0 (2018)
  * [23] Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1096–1104 (2016)
  * [24] Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., Feris, R.: Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5334–5343 (2017)
  * [25] Maaten, L.v.d., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(Nov), 2579–2605 (2008)
  * [26] Neyshabur, B., Bhojanapalli, S., McAllester, D., Srebro, N.: Exploring generalization in deep learning. In: Advances in Neural Information Processing Systems. pp. 5947–5956 (2017)
  * [27] Orbes-Arteainst, M., Cardoso, J., Sørensen, L., Igel, C., Ourselin, S., Modat, M., Nielsen, M., Pai, A.: Knowledge distillation for semi-supervised domain adaptation. In: OR 2.0 Context-Aware Operating Theaters and Machine Learning in Clinical Neuroimaging, pp. 68–76. Springer (2019)
  * [28] Palatucci, M., Pomerleau, D., Hinton, G.E., Mitchell, T.M.: Zero-shot learning with semantic output codes. In: Advances in neural information processing systems. pp. 1410–1418 (2009)
  * [29] Papernot, N., Abadi, M., Erlingsson, U., Goodfellow, I., Talwar, K.: Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755 (2016)
  * [30] Park, S., Shin, M., Ham, S., Choe, S., Kang, Y.: Study on fashion image retrieval methods for efficient fashion visual search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 0–0 (2019)
  * [31] Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3967–3976 (2019)
  * [32] Quintino Ferreira, B., Costeira, J.P., Sousa, R.G., Gui, L.Y., Gomes, J.P.: Pose guided attention for multi-label fashion image classification. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 0–0 (2019)
  * [33] Ren, M., Triantafillou, E., Ravi, S., Snell, J., Swersky, K., Tenenbaum, J.B., Larochelle, H., Zemel, R.S.: Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803.00676 (2018)
  * [34] Shin, M., Park, S., Kim, T.: Semi-supervised feature-level attribute manipulation for fashion image retrieval. In: Proceedings of the British Machine Vision Conference (2019)
  * [35] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in neural information processing systems. pp. 4077–4087 (2017)
  * [36] Socher, R., Ganjoo, M., Manning, C.D., Ng, A.: Zero-shot learning through cross-modal transfer. In: Advances in neural information processing systems. pp. 935–943 (2013)
  * [37] Sohn, K.: Improved deep metric learning with multi-class n-pair loss objective. In: Advances in Neural Information Processing Systems. pp. 1857–1865 (2016)
  * [38] Wang, W., Xu, Y., Shen, J., Zhu, S.C.: Attentive fashion grammar network for fashion landmark detection and clothing category classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4271–4280 (2018)
  * [39] Wu, C.Y., Manmatha, R., Smola, A.J., Krahenbuhl, P.: Sampling matters in deep embedding learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2840–2848 (2017)
  * [40] Xie, Q., Hovy, E., Luong, M.T., Le, Q.V.: Self-training with noisy student improves imagenet classification. arXiv preprint arXiv:1911.04252 (2019)
  * [41] Yalniz, I.Z., Jégou, H., Chen, K., Paluri, M., Mahajan, D.: Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546 (2019)
  * [42] Zhang, S., Song, Z., Cao, X., Zhang, H., Zhou, J.: Task-aware attention model for clothing attribute prediction. IEEE Transactions on Circuits and Systems for Video Technology (2019)
  * [43] Zheng, S., Song, Y., Leung, T., Goodfellow, I.: Improving the robustness of deep neural networks via stability training. In: Proceedings of the ieee conference on computer vision and pattern recognition. pp. 4480–4488 (2016)
  * [44] Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)


# The solutions of four $q$-functional equations

Jun-Ming Zhu 111E-mail address<EMAIL_ADDRESS><EMAIL_ADDRESS>
Abstract In this note we obtain the solutions of four $q$-functional equations
and express the solutions in $q$-operator forms. These equations give
sufficient conditions for $q$-operator methods

Key words: Basic hypergeometric series, $q$-Series, $q$-Functional equation,
$q$-Difference equation, q-Exponential operator

##  1  Introduction

We follow the notation and terminology in [6] , and for convenience, we always
assume that $0<|q|<1$. The $q$-shifted factorials are defined by

${(a;q)_{n}=}\left\\{\begin{array}[]{ll}1,&n=0,\\\
(1-a)(1-aq)\cdots(1-aq^{n-1}),&n=1,2,3,\cdots\cdots,\end{array}\right.$

The $q$-derivative operator is defined by

$D_{q}\\{f(a)\\}=\frac{f(a)-f(aq)}{a},~{}~{}D_{q}^{n}\\{f(a)\\}=D_{q}\\{D_{q}^{n-1}\\{f(a)\\}\\}.$

The operator $\theta$ is defined by

$\theta=\eta^{-1}D_{q},~{}~{}\theta^{n}\\{f(a)\\}=\theta\\{\theta^{n-1}\\{f(a)\\}\\},$

where $\eta^{-1}\\{f(a)\\}=f(q^{-1}a).$ Both $D_{q}$ and $\theta$ are
obviously linear transforms, and by convention, $D_{q}^{0}$ and $\theta^{0}$
are both understood as the identity.

The two $q$-exponential operators (see, [2, 3, 7, 8, 10]) are defined by

$T(bD_{q})=\sum_{n=0}^{\infty}\frac{(bD_{q})^{n}}{(q;q)_{n}},$

and

$E(b\theta)=\sum_{n=0}^{\infty}\frac{q^{n(n-1)/2}(b\theta)^{n}}{(q;q)_{n}},$

respectively.

The following operator was first introduced by Fang [5]. But we follow Chen
and Gu’s notation [1]. It seems to be more convenient.

$T(a,b;D_{q})=\sum_{n=0}^{\infty}\frac{(a,q)_{n}}{(q;q)_{n}}(bD_{q})^{n}.$

Chen and Gu [1] named $T(a,b;D_{q})$ as Cauchy operator. But compared with the
following operator, we called $T(a,b;D_{q})$ the first Cauchy operator in this
paper.

The operator $T(a,b;\theta)$, introduced by Fang [5], is defined by

$T(a,b;\theta)=\sum_{n=0}^{\infty}\frac{(a,q)_{n}}{(q;q)_{n}}(b\theta)^{n}.$

We called $T(a,b;\theta)$ the second Cauchy operator.

All the papers using the $q$-operator methods imply the following definition
but do not state it explicitly. Unless otherwise stated, all the operators are
applied with respect to the parameter $a$.

Definition 1.1

$T(bD_{q})\\{f(a)\\}=\sum_{n=0}^{\infty}\frac{b^{n}}{(q;q)_{n}}\\{{D_{q}}^{n}\\{f(a)\\}\\},$

and

$E(b\theta)\\{f(a)\\}=\sum_{n=0}^{\infty}\frac{q^{n(n-1)/2}b^{n}}{(q;q)_{n}}\\{{\theta}^{n}\\{f(a)\\}\\}.$

The first and the second Cauchy operators and the operators $T(b\theta)$ and
$E(bD_{q})$ below also act in this way.

When using the methods of operators, rearrangememts of series are often
emploied. We know that rearrangememts of series must be under the condition
that the series are absolutely convergent, which may be not very easy to check
sometimes. In his paper [9] , Liu obtained the following two theorems using
the $q$-functional equation method.

######  Theorem 1.1

Let $f(a,b)$ be a two variables analytic function in a neighborhood of
$(a,b)=(0,0)\in\mathbb{C}^{2}$, satisfying the $q$-difference equation

$bf(aq,b)-af(a,bq)=(b-a)f(a,b).$

Then we have

$f(a,b)=T(bD_{q})\\{f(a,0)\\}.$

######  Theorem 1.2

Let $f(a,b)$ be a two variables analytic function in a neighborhood of
$(a,b)=(0,0)\in\mathbb{C}^{2}$, satisfying the $q$-difference equation

$af(aq,b)-bf(a,bq)=(a-b)f(aq,bq).$

Then we have

$f(a,b)=E(b\theta)\\{f(a,0)\\}.$

These two theorems give us a method to use operator $T(bD_{q})$ and
$E(b\theta)$ without having to check the absolute convergence of the series.

Originated by Liu’s work, we give the above two theorems more general forms in
the following section.

##  2  Four $q$-functional equations and the solutions

######  Theorem 2.1

Let $f(a,b,c)$ be a three variables analytic function in a neighborhood of
$(a,b,c)=(0,0,0)\in\mathbb{C}^{3}$, satisfying the $q$-functional equation

$c(f(a,b,c)-f(a,bq,c))=b(f(a,b,c)-f(a,b,cq)-af(a,bq,c)+af(a,bq,cq)).$ (1)

Then we have

$f(a,b,c)=T(a,b;D_{q})\\{f(a,0,c)\\},$

where $T(a,b;D_{q})$ is applied with respect to the parameter $c$.

######  Theorem 2.2

Let $f(a,b,c)$ be a three variables analytic function in a neighborhood of
$(a,b,c)=(0,0,0)\in\mathbb{C}^{3}$, satisfying the $q$-functional equation

$c(f(a,bq,cq)-f(a,b,cq))=b(f(a,bq,cq)-f(a,bq,c)-af(a,b,c)+af(a,b,cq)).$ (2)

Then we have

$f(a,b,c)=T(-\frac{1}{a},ab;\theta)\\{f(a,0,c)\\},$

where $T(-\frac{1}{a},ab;\theta)$ is also applied with respect to the
parameter $c$.

When letting $a\rightarrow 0$ in theorem 2.1 and 2.2, we get theorem 1.1 and
1.2, respectively. The proof of theorem 2.2 is similar to that of theorem 2.1
and so is omitted.

Proof of Theorem 2.1  We now begin to solve this equation. From the theory of
several complex variables, we assume that

$f(a,b,c)=\sum_{n=0}^{+\infty}A_{n}(a,c)b^{n}.$ (3)

We substitute the above equation into (1) to get

$\displaystyle c\sum_{n=0}^{+\infty}(1-q^{n})A_{n}(a,c)b^{n}$ $\displaystyle=$
$\displaystyle(\sum_{n=0}^{+\infty}A_{n}(a,c)-\sum_{n=0}^{+\infty}A_{n}(a,cq)-a\sum_{n=0}^{+\infty}A_{n}(a,c)q^{n}$
$\displaystyle+a\sum_{n=0}^{+\infty}A_{n}(a,cq)q^{n})b^{n+1}.$

This is

$\displaystyle
c\sum_{n=1}^{+\infty}(1-q^{n})A_{n}(a,c)b^{n}=\sum_{n=1}^{+\infty}(1-aq^{n-1})(A_{n-1}(a,c)-A_{n-1}(a,cq))b^{n}.$

Comparing the coefficients of $b^{n}$ gives

$c(1-q^{n})A_{n}(a,c)=(1-aq^{n-1})(A_{n-1}(a,c)-A_{n-1}(a,cq)).$

Then we have

$\displaystyle A_{n}(a,c)$ $\displaystyle=$
$\displaystyle\frac{(1-aq^{n-1})}{(1-q^{n})}\frac{(A_{n-1}(a,c)-A_{n-1}(a,cq))}{c}$
$\displaystyle=$
$\displaystyle\frac{(1-aq^{n-1})}{(1-q^{n})}D_{q}\\{A_{n-1}(a,c)\\},$

where $D_{q}$ is applied with respect to the parameter $c$. Iterate the above
equation to get

$\displaystyle A_{n}(a,c)=\frac{(a;q)_{n}}{(q,q)_{n}}D_{q}\\{A_{0}(a,c)\\}.$
(4)

It remains to calculate $A_{0}(a,c)$. Putting $b=0$ in (3), we immediately
deduce that $A_{0}(a,c)=f(a,0,c)$. Substituting (4) back into (3) gives

$f(a,b,c)=\sum_{n=0}^{\infty}\frac{(a,q)_{n}}{(q;q)_{n}}(bD_{q})^{n}\\{f(a,0,c)\\}=T(a,b;D_{q})\\{f(a,0,c)\\}.$

This completes the proof.

Using the $q$-functional equation method we easily obtain the following two
theorems.

######  Theorem 2.3

Let $f(a,b)$ be a two variables analytic function in a neighborhood of
$(a,b)=(0,0)\in\mathbb{C}^{2}$, satisfying the $q$-functional equation

$af(a,b)+bf(aq,bq)=(a+b)f(a,bq).$

Then we have

$f(a,b)=E(bD_{q})\\{f(a,0)\\},$

where the operator $E(bD_{q})$ is defined by

$E(bD_{q})=\sum_{n=0}^{\infty}\frac{q^{n(n-1)/2}(bD_{q})^{n}}{(q;q)_{n}}.$

######  Theorem 2.4

Let $f(a,b)$ be a two variables analytic function in a neighborhood of
$(a,b)=(0,0)\in\mathbb{C}^{2}$, satisfying the $q$-functional equation

$af(aq,bq)+bf(a,b)=(a+b)f(aq,b).$

Then we have

$f(a,b)=T(-b\theta)\\{f(a,0)\\},$

where the operator $T(b\theta)$ is defined by

$T(b\theta)=\sum_{n=0}^{\infty}\frac{(b\theta)^{n}}{(q;q)_{n}}.$

From all the theorems in this little paper, we can look at the q-operators
from a different standpoint.

## References

  * [1] V. Y. B. Chen, N. S. S. Gu, The Cauchy operator for basic hypergeometric series, Adv. in Appl. Math, 41(2008), 177–196.
  * [2] W. Y. C. Chen, Z.-G. Liu, Parameter augmentation for basic hypergeometric series, II, J. Combin. Theory Ser. A 80(1997), 175–195.
  * [3] W. Y. C. Chen, Z.-G. Liu, Parameter augmentation for basic hypergeometric series, I, in: B. E. Sagan, R. P. Stanley (Eds.), Mathematical Essays in honor of Gian-Carlo Rota, Birkäuser, Basel, 1998, pp. 111–129.
  * [4] J.-P. Fang, $q$-Differential operator identities and applications, J. Math. Anal. Appl. 332(2007), 1393–1407.
  * [5] J.-P. Fang, $q$-Operator identities and its applications, Journal of East China Normal University, 1(2008), 20–24 (in Chinese).
  * [6] G. Gasper, M. Rahman, Basic Hypergeometric Series, 2nd ed., Cambridge Univ. Press, Cambridge, MA, 2003.
  * [7] Z.-G. Liu, A new proof of the Nassrallah-Rahman integral, Acta Mathematica Sinica 41(1998), 405–410 (in Chinese).
  * [8] Z.-G. Liu, Some operator identities and $q$-series transformation formulas, Discrete Math., 265(2003), 119–139.
  * [9] Z.-G. Liu, Two $q$-difference equations and $q$-operator indetities, Journal of Difference Equations and Applications (in press).
  * [10] L. J. Rogers, On the expansion of some infinite products, Proc. London Math. Soc., 24(1894), 337-352.