# On weights, traces and K-theory

Klaus Thomsen  Til Palle E.T. Jørgensen i taknemlighed  Department of
Mathematics, Aarhus University, Ny Munkegade, 8000 Aarhus C, Denmark
<EMAIL_ADDRESS>
###### Abstract.

It is shown that the pairing of the $K_{00}$ group of a $C^{*}$-algebra with
the densely defined traces of the algebra can be extended to a pairing with
the densely defined weights. For traces the pairing can be extended to the
$K_{0}$ group without the semi-continuity assumption occurring in the work of
Connes and Elliott.

##  1\. Introduction

The pairing between the trace states and the $K_{0}$ group of a unital
$C^{*}$-algebra is an important part of the invariant of $C^{*}$-algebras
known as the Elliott invariant. I don’t know who first observed the existence
of this pairing, but it seems to be Connes who first observed that there is
also a pairing with $K_{0}$ for non-unital $C^{*}$-algebras which uses traces
that are only densely defined, but lower semi-continuous, [Co1]. The latter
pairing was revisited in a recent work by Elliott, [E2]. While Connes gives
very few details of the construction, Elliott bases his approach on properties
of the Pedersen ideal. The main purpose of the present text is to point out
that the pairing exists also for traces that are not lower semi-continuous; it
suffices that they are densely defined. This conclusion is obtained via a
thorough examination of the constructions in [Co1] and [E2], and the original
impetus for the present paper was the wish to point out that a condition,
although natural from many points of view, is in fact not necessary. At first
sight this may seem to have little or no significance because it can be
difficult to come up with examples of densely defined traces that are not
lower semi-continuous. Perhaps the only densely defined traces that come to
mind and are not lower semi-continuous, are those coming from the Dixmier
trace(s) on the $C^{*}$-algebra of compact operators. However, densely defined
traces that are not lower semi-continuous exist in abundance, also on simple
$C^{*}$-algebras very different from the $C^{*}$-algebra of compact operators.
In an effort to convince the reader of this, the paper concludes by presenting
constructions of such ’singular’ traces on other $C^{*}$-algebras. It seems to
me that it is worthwhile to pursue a more systematic investigation of such
traces, and not only because they exist. But this is not the time or place.

The definition of the $K_{0}$-group $K_{0}(A)$ of a $C^{*}$-algebra $A$ starts
with the formation of the group $K_{00}(A)$, cf. e.g. [B], and similarly the
first step in the construction of the pairing between $K_{0}$ and traces is a
pairing with $K_{00}(A)$. As a second purpose with this text, it is pointed
out in the following first sections that there is a natural way to pair
positive linear functionals, and more generally densely defined weights on $A$
with $K_{00}(A)$ in a way which generalizes the pairing with traces.

##  2\. Weights

Let $A$ be a $C^{*}$-algebra and let $A^{+}$ denote the cone of positive
elements in $A$. A map $\psi:A^{+}\to[0,\infty]$ is a _weight_ on $A$ when

  * (a)

$\psi(a+b)=\psi(a)+\psi(b),\ \ \forall a,b\in A^{+}$, and

  * (b)

$\psi(ta)=t\psi(a),\ \ \forall a\in A^{+},\ \forall t\in\mathbb{R}^{+}$, with
the convention that $0\cdot\infty=0$.

A weight $\psi$ is _densely defined_ when $\left\\{a\in
A^{+}:\psi(a)<\infty\right\\}$ is dense in $A^{+}$ and a _trace_ when
$\psi(aa^{*})=\psi(a^{*}a)$ for all $a\in A$. A weight $\psi$ on $A$ is _lower
semi-continuous_ when $\left\\{a\in A^{+}:\ \psi(a)>t\right\\}$ is open in
$A^{+}$ for all $t\in\mathbb{R}$. By a fundamental result of Combes, [C], this
happens (if and) only if there is a set $\mathcal{F}$ of positive linear
functionals on $A$ such that

$\psi(a)=\sup_{\omega\in\mathcal{F}}\omega(a),\ \ \forall a\in A^{+}.$

Given a weight $\psi$ on $A$ we set

$\mathcal{M}^{+}_{\psi}:=\left\\{a\in A^{+}:\ \psi(a)<\infty\right\\},$
$\mathcal{N}_{\psi}:=\left\\{a\in A:\ \psi(a^{*}a)<\infty\right\\},$

and

$\mathcal{M}_{\psi}:=\operatorname{Span}\mathcal{M}^{+}_{\psi}.$

Based on work by Pedersen in [Pe], the following was observed by Combes in
Lemme 1.1 and Lemme 1.3 of [C].

######  Lemma 2.1.

Let $\psi$ be weight on $A$. The sets $\mathcal{M}_{\psi}$,
$\mathcal{M}_{\psi}^{+}$ and $\mathcal{N}_{\psi}$ have the following
properties.

  * (a)

$\mathcal{N}_{\psi}$ is a left-ideal in $A$; in particular, a subspace of $A$.

  * (b)

$\mathcal{M}_{\psi}=\operatorname{Span}\mathcal{N}_{\psi}^{*}\mathcal{N}_{\psi}$
is a $*$-subalgebra of $A$.

  * (c)

$\mathcal{M}_{\psi}\cap A^{+}=\mathcal{M}_{\psi}^{+}$.

  * (d)

$\psi:\mathcal{M}^{+}_{\psi}\to[0,\infty)$ extends uniquely to a linear map
$\psi|_{\mathcal{M}_{\psi}}:\mathcal{M}_{\psi}\to\mathbb{C}$.

##  3\. Pairing weights with $K_{00}$

The following brief description of the $K_{0}$ group of a $C^{*}$-algebra $A$
serves primarily to introduce the notation we shall use. For more details, see
Blackadars book [B] or Rørdam et al. [RLL], for example. Let $M_{k}(A)$ denote
the $C^{*}$-algebra of $k\times k$ matrices over $A$ and let $P_{k}(A)$ denote
the set of projections in $M_{k}(A)$. We embed $M_{k}(A)$ into $M_{k+1}(A)$ by
adding a zero row and a zero column, and we can therefore consider the unions

$M_{\infty}(A):=\bigcup_{k=1}^{\infty}M_{k}(A),$

and

$P_{\infty}(A):=\bigcup_{k=1}^{\infty}P_{k}(A).$

The Murray-von Neumann equivalence relation $\sim$ is defined in
$P_{\infty}(A)$ such that $p\sim q$ when there is an element $v\in
M_{\infty}(A)$ with $vv^{*}=p$ and $v^{*}v=q$. The equivalence class of an
element $e\in P_{\infty}(A)$ is written $[e]$. Given a pair of projections
$p,q\in P_{\infty}(A)$ there is a projection $p^{\prime}\in P_{\infty}(A)$
such that $p^{\prime}\perp q$, i.e. such that $p^{\prime}q=0$, and such that
$p\sim p^{\prime}$. The sum $p^{\prime}+q$ is then a projection, and the sum
$[p]+[q]$ is defined by

$[p]+[q]:=[p^{\prime}+q].$

This turns $V(A):=P_{\infty}(A)/\sim$ into an abelian semi-group. The group
$K_{00}(A)$ is the Grothendieck group of $V(A)$. To obtain the $K_{0}$ group,
add a unit to $A$ to get the unital $C^{*}$-algebra $A^{\dagger}$ and define
$K_{0}(A)$ as the kernel of the map

$K_{00}(A^{\dagger})\to K_{00}(\mathbb{C})$

induced by the quotient map $A^{\dagger}\to\mathbb{C}$. Since $[e]$ is an
element of this kernel when $e\in P_{\infty}(A)\subseteq
P_{\infty}(A^{\dagger})$, there is a canonical map

$K_{00}(A)\to K_{0}(A)$ (3.1)

which is an isomorphism when $A$ is unital or at least has an approximate unit
consisting of projections, cf. [B].

Let $\psi$ be a weight on the $C^{*}$-algebra $A$. For each $k\in\mathbb{N}$
we define a weight $\psi_{k}$ on $M_{k}(A)$ such that

$\psi_{k}(x):=\sum_{i=1}^{k}\psi(x_{ii})=\psi(\sum_{i=1}^{k}x_{ii})$

when $x=(x_{ij})\in M_{k}(A)^{+}$. This definition is compatible with the
embedding of $M_{k}(A)$ into $M_{k+1}(A)$ and we get a map

$\psi_{\infty}:P_{\infty}(A)\to[0,\infty]$

whose restriction to $P_{k}(A)$ agrees with $\psi_{k}$ for all $k$.

######  Lemma 3.1.

For each $k\in\mathbb{N}$,

  * (e)

$\mathcal{N}_{\psi_{k}}=M_{k}(\mathcal{N}_{\psi})$, and

  * (f)

$\mathcal{M}_{\psi_{k}}=M_{k}(\mathcal{M}_{\psi})$.

###### Proof.

(e) follows from the observation that

$\psi_{k}(x^{*}x)=\sum_{i,j}\psi(x_{ji}^{*}x_{ji})$

when $x=(x_{ij})\in M_{k}(A)$. To prove (f) note that it follows from (e) and
(b) that

$\mathcal{M}_{\psi_{k}}=\operatorname{Span}\mathcal{N}_{\psi_{k}}^{*}\mathcal{N}_{\psi_{k}}\subseteq\operatorname{Span}M_{k}(\mathcal{N}_{\psi}^{*}\mathcal{N}_{\psi})\subseteq
M_{k}(\mathcal{M}_{\psi}).$

For the converse inclusion, consider $a,b\in\mathcal{N}_{\psi}$ and the
standard matrix units $\\{e_{ij}\\}$ in $M_{k}(\mathbb{C})$. Then
$a^{*}b\otimes e_{ij}=(a\otimes e_{1i})^{*}(b\otimes e_{1j})$. Note that
$(a\otimes e_{1i})^{*}(a\otimes e_{1i})=a^{*}a\otimes
e_{ii}\in\mathcal{M}_{\psi_{k}}^{+}$ and similarly $(b\otimes
e_{1j})^{*}(b\otimes e_{1j})=b^{*}b\otimes
e_{jj}\in\mathcal{M}_{\psi_{k}}^{+}$. Hence $a\otimes
e_{1i}\in\mathcal{N}_{\psi_{k}}$ and $b\otimes
e_{1j}\in\mathcal{N}_{\psi_{k}}$, and thus $a^{*}b\otimes
e_{ij}\in\mathcal{N}_{\psi_{k}}^{*}\mathcal{N}_{\psi_{k}}\subseteq\mathcal{M}_{\psi_{k}}$
by (b). Since $i,j,a,b$ were arbitrary it follows now also from (b) that
$M_{k}(\mathcal{M}_{\psi})=M_{k}\left(\operatorname{Span}\mathcal{N}_{\psi}^{*}\mathcal{N}_{\psi}\right)\subseteq\mathcal{M}_{\psi_{k}}$.
∎

######  Lemma 3.2.

Let $\psi$ be a densely defined weight on $A$. Then $\psi_{k}$ is densely
defined for all $k\in\mathbb{N}$.

###### Proof.

Let $x\in M_{k}(A)^{+}$. Write $x=a^{2}$ where $a\in M_{k}(A)^{+}$. Since
every element of $A$ is a linear combination of four elements of $A^{+}$ it
follows that $\mathcal{M}_{\psi}$ is dense in $A$ since $\psi$ is densely
defined. It follows from (a) and (b) that
$\mathcal{M}_{\psi}\subseteq\mathcal{N}_{\psi}$ and hence $\mathcal{N}_{\psi}$
is also dense in $A$. It follows therefore that $M_{k}(\mathcal{N}_{\psi})$ is
dense in $M_{n}(A)$. In particular, there is a sequence $\\{a_{n}\\}$ in
$M_{k}(\mathcal{N}_{\psi})$ such that $\lim_{n\to\infty}a_{n}=a$. Note that
$a_{n}^{*}a_{n}\in\mathcal{N}_{\psi_{k}}^{*}\mathcal{N}_{\psi_{k}}$ by (e) and
hence $a_{n}^{*}a_{n}\in\mathcal{M}_{\psi_{k}}^{+}$ by (b) and (c). Since
$\lim_{n\to\infty}a_{n}^{*}a_{n}=x$, we have shown that $\psi_{k}$ is densely
defined. ∎

######  Lemma 3.3.

Let $\psi$ be a densely defined weight on $A$ and let $e\in P_{k}(A)$. There
is a projection $f\in P_{k}(A)$ such that $e\sim f$ and $\psi_{k}(f)<\infty$.

###### Proof.

Let $\delta\in]0,\frac{1}{2}[$ and define $f:[0,1]\to[0,1]$ such that $f$ is
continuous and

$f(t)=\begin{cases}0,&\ t\in[0,\delta]\\\ \text{linear},&\
t\in[\delta,1-\delta]\\\ 1,&\ t\in[1-\delta,1].\end{cases}$

Then

$\left|f(t)-t\right|\leq\delta\ \ \forall t\in[0,1].$ (3.2)

$\psi_{k}$ is densely defined by Lemma 3.2 and we can therefore find
$a\in\mathcal{M}_{\psi_{k}}^{+}$ such that $0\leq a\leq 1$ and $\|a-e\|$ is as
small as we want. In particular, we can arrange that the spectrum $\sigma(a)$
of $a$ is contained in $[0,\delta]\cup[1-\delta,1]$ and $\|a-e\|\leq\delta$.
It follows from (3.2) that $\left\|f(a)-a\right\|\leq\delta$. Note that $f(a)$
is a projection in $M_{k}(A)$ and that $\|f(a)-e\|\leq 2\delta$. Since
$2\delta<1$ it follows that $f(a)\sim e$, cf. e.g. Proposition 4.6.6 in [B].
Since $f(t)\leq(1-\delta)^{-1}t$ for all $t\in\sigma(a)$, it follows that
$f(a)\leq(1-\delta)^{-1}a$ and hence
$\psi_{k}(f(a))\leq(1-\delta)^{-1}\psi_{k}(a)<\infty$.

∎

For $e\in P_{\infty}(A)$, set

$\underline{\psi}(e):=\inf\left\\{\psi_{\infty}(f):\ f\in P_{\infty}(A),\
f\sim e\right\\}.$

Then Lemma 3.3 has the following corollary.

######  Corollary 3.4.

Let $\psi$ be a densely defined weight on $A$. Then
$\underline{\psi}(e)<\infty$ for all $e\in P_{\infty}(A)$.

######  Lemma 3.5.

Let $\psi$ be a densely defined weight on $A$. Let $e,f\in P_{\infty}(A)$ such
that $ef=0$. Then
$\underline{\psi}(e+f)=\underline{\psi}(e)+\underline{\psi}(f)$.

###### Proof.

When $p\in P_{\infty}(A)$ and $p\sim e+f$ we can write $p=p_{1}+p_{2}$ where
$p_{1}\sim e$ and $p_{2}\sim f$. Then
$\psi_{\infty}(p)=\psi_{\infty}(p_{1})+\psi_{\infty}(p_{2})\geq\underline{\psi}(e)+\underline{\psi}(f)$.
It follows that
$\underline{\psi}(e+f)\geq\underline{\psi}(e)+\underline{\psi}(f)$. To obtain
the reverse inequality, let $\epsilon>0$. There is a $k\in\mathbb{N}$ such
that $e,f\in P_{k}(A)$, and by increasing $k$ if necessary we can also arrange
that there are projections $p,q\in P_{k}(A)$ such that $e\sim p,\ f\sim q$,
$\psi_{k}(p)\leq\underline{\psi}(e)+\epsilon$ and
$\psi_{k}(q)\leq\underline{\psi}(f)+\epsilon$. There is then a unitary $S\in
M_{2k}(\mathbb{C})$ such that $(1_{A}\otimes S)p(1_{A}\otimes S)^{*}q=0$. Then

$e+f\sim(1_{A}\otimes S)p(1_{A}\otimes S)^{*}+q.$ (3.3)

We claim that

$\psi_{2k}\left((1_{A}\otimes S)p(1_{A}\otimes S)^{*}\right)=\psi_{k}(p).$
(3.4)

To see this, note that since $p\in\mathcal{M}_{\psi_{2k}}^{+}$ it follows from
(f) of Lemma 3.1 that $p\in
M_{2k}(\mathcal{M}_{\psi})=\mathcal{M}_{\psi}\otimes M_{2k}(\mathbb{C})$.
Thanks to (d) and (f) it makes sense to assert that

$\psi_{2k}|_{\mathcal{M}_{\psi_{2k}}}=\psi|_{\mathcal{M}_{\psi}}\otimes\operatorname{Tr}_{2k},$
(3.5)

where $\operatorname{Tr}_{2k}$ is the standard trace on $M_{2k}(\mathbb{C})$;
the sum of the diagonal entries. Furthermore, the identity (3.5) can be easily
verified by checking on simple tensors. Since $\operatorname{Ad}(1_{A}\otimes
S)(M_{2k}(\mathcal{M}_{\psi}))=M_{2k}(\mathcal{M}_{\psi})$, it follows from
this that

$\displaystyle\psi_{2k}\left((1_{A}\otimes S)p(1_{A}\otimes
S)^{*}\right)=\psi_{2k}\circ\operatorname{Ad}(1_{A}\otimes S)(p)$
$\displaystyle=(\psi|_{\mathcal{M}_{\psi}}\otimes\operatorname{Tr}_{2k})\circ\operatorname{Ad}(1_{A}\otimes
S)(p)$
$\displaystyle=\left(\psi|_{\mathcal{M}_{\psi}}\otimes(\operatorname{Tr}_{2k}\circ\operatorname{Ad}S)\right)(p)$
$\displaystyle=\left(\psi|_{\mathcal{M}_{\psi}}\otimes\operatorname{Tr}_{2k}\right)(p)=\psi_{2k}(p)=\psi_{k}(p),$

proving the claim (3.4). It follows that

$\psi_{\infty}((1_{A}\otimes S)p(1_{A}\otimes
S)^{*}+q)=\psi_{\infty}(p)+\psi_{\infty}(q)\leq\underline{\psi}(e)+\underline{\psi}(f)+2\epsilon,$

and then (3.3) implies that
$\underline{\psi}(e+f)\leq\underline{\psi}(e)+\underline{\psi}(f)+2\epsilon$.
Since $\epsilon>0$ was arbitrary it follows that
$\underline{\psi}(e+f)\leq\underline{\psi}(e)+\underline{\psi}(f)$. ∎

The following is now an immediate consequence of the definitions and Corollary
3.3 and Lemma 3.5.

######  Proposition 3.6.

Given a densely defined weight $\psi$ on $A$, there is a homomorphism
$\psi_{*}:K_{00}(A)\to\mathbb{R}$ such that

$\psi_{*}([e]-[f])=\underline{\psi}(e)-\underline{\psi}(f)$

when $e,f\in P_{\infty}(A)$.

When the weight $\psi$ is a trace we have that
$\underline{\psi}(e)=\psi_{\infty}(e)$ for all $e\in P_{\infty}(A)$ and the
homomorphism $\psi_{*}$ of Proposition 3.6 is therefore the usual one, but
note that we do not need to assume the trace is lower semi-continuous. When
$A$ has an approximate unit consisting of projections there is an
identification $K_{00}(A)=K_{0}(A)$, [B], and Proposition 3.6 gives then a
pairing of densely defined weights on $A$ with $K_{0}(A)$. In particular, when
$A$ is unital we get a map from the states of $A$ to the states on $K_{0}(A)$
extending the map from trace states on $A$ which occurs in the Elliott
invariant.

######  Example 3.7.

Let $H$ be a Hilbert space and $h$ a bounded positive operator on $H$. We can
then consider the densely defined weight $\psi$ on the $C^{*}$-algebra
$\mathbb{K}$ of compact operators on $H$ given by

$\psi(a)=\operatorname{Tr}(ha),\ \ \forall a\in\mathbb{K}^{+}.$

This is an unbounded weight unless $h$ is of trace class. Using the
identification $K_{00}(\mathbb{K})=\mathbb{Z}$ the homomorphism $\psi_{*}$ of
Proposition 3.6 is given by

$\psi_{*}(z)=\lambda z,\ \ \forall z\in\mathbb{Z},$

where

$\lambda=\inf\left\\{\left<h\phi,\phi\right>:\ \phi\in H,\|\phi\|=1\right\\}$

is the minimum of the spectrum of $h$.

##  4\. Pairing traces with $K_{0}$

In the following we fix a densely defined trace $\omega:A^{+}\to[0,\infty]$ on
$A$.

######  Lemma 4.1.

$\mathcal{M}_{\omega}$ is a dense two-sided $*$-invariant ideal in $A$.

###### Proof.

$\mathcal{N}_{\omega}$ is a left ideal by (a) of Lemma 2.1. The trace property
of $\omega$ implies that $\mathcal{N}_{\omega}^{*}=\mathcal{N}_{\omega}$ and
hence $\mathcal{N}_{\omega}$ is also a right ideal. It follows therefore from
(b) of Lemma 2.1 that $\mathcal{M}_{\omega}$ is a $*$-invariant two-sided
ideal in $A$. It is dense in $A$ because $\mathcal{M}_{\omega}^{+}$ is dense
in $A^{+}$ by assumption. ∎

Let $A^{\dagger}$ be the $C^{*}$-algebra obtained from $A$ be adjoining a unit
to $A$. Thus as a vector space $A^{\dagger}$ is just $A\oplus\mathbb{C}$, and
the product and involution are given by

$(a,\lambda)(b,\mu)=(ab+\lambda b+\mu a,\lambda\mu)$

and

$(a,\lambda)^{*}=(a^{*},\overline{\lambda}).$

For every subset $X\subseteq A$, set

$X^{\dagger}:=\left\\{(a,\lambda)\in A^{\dagger}:\ a\in X,\
\lambda\in\mathbb{C}\right\\}.$

In particular,

$\mathcal{M}_{\omega}^{\dagger}:=\left\\{(a,\lambda)\in A^{\dagger}:\
a\in\mathcal{M}_{\omega},\ \lambda\in\mathbb{C}\right\\}.$

Define $\omega^{\dagger}:\mathcal{M}_{\omega}^{\dagger}\to\mathbb{C}$ such
that

$\omega^{\dagger}(a,\lambda)=\omega|_{\mathcal{M}_{\omega}}(a).$

Since $\omega^{\dagger}$ is linear on $\mathcal{M}_{\omega}^{\dagger}$ we can
consider the tensor product map

$\omega^{\dagger}\otimes\operatorname{Tr}_{n}:\
M_{n}(\mathcal{M}_{\omega}^{\dagger})\to\mathbb{C},$

where $\operatorname{Tr}_{n}$ denotes the standard trace on
$M_{n}(\mathbb{C})$. Let $P_{n}(\mathcal{M}_{\omega}^{\dagger})$ be the set of
projections in the $*$-algebra $M_{n}(\mathcal{M}_{\omega}^{\dagger})$ and set

$P_{\infty}(\mathcal{M}_{\omega}^{\dagger}):=\bigcup_{n}P_{n}(\mathcal{M}_{\omega}^{\dagger}).$

We aim to establish the following

######  Theorem 4.2.

$K_{0}(A^{\dagger})=\left\\{[e]-[f]:\ e,f\in
P_{\infty}(\mathcal{M}_{\omega}^{\dagger})\right\\}$

and there is a homomorphism $\omega^{\dagger}_{*}:\
K_{0}(A^{\dagger})\to\mathbb{R}$ such that

$\omega^{\dagger}_{*}([e]-[f])=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e)-\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f)$

when $e,f\in M_{n}(\mathcal{M}_{\omega}^{\dagger})$.

###  4.1. Proof of Theorem 4.2

The proof uses the following series of lemmas.

Recall that a _strictly positive_ element of a $C^{*}$-algebra $A$ is a
positive element $a\in A^{+}$ such $\omega(a)>0$ for all non-zero positive
functionals $\omega$ on $A$. A $C^{*}$-algebra is _$\sigma$ -unital_ when it
contains a strictly positive element. The first two lemmas are well-known.
Their proofs are included for completeness.

######  Lemma 4.3.

A separable $C^{*}$-algebra is $\sigma$-unital.

###### Proof.

When $A$ is a separable $C^{*}$-algebra there is a dense sequence
$\\{a_{n}\\}_{n=1}^{\infty}$ in $\\{a\in D^{+}:\ 0\leq a\leq 1\\}$. Then

$a:=\sum_{n=1}^{\infty}2^{-n}a_{n}$

is strictly positive. ∎

######  Lemma 4.4.

Let $D$ be a $\sigma$-unital $C^{*}$-algebra. There is a sequence
$\\{d_{n}\\}_{n=1}^{\infty}$ in $D$ such that

  * •

$0\leq d_{n}\leq 1,\ \ \forall n$,

  * •

$d_{n}d_{n+1}=d_{n},\ \ \forall n$, and

  * •

$\lim_{n\to\infty}d_{n}a=a,\ \ \forall a\in D$.

###### Proof.

Let $a_{0}$ be a strictly positive element of $A$. Let $f_{n}$ be the
continuous function $f_{n}:[0,\infty)\to[0,1]$ such that

$f_{n}(t)=\begin{cases}0,&\ t\in[0,\frac{1}{n+1}]\\\ \text{linear},&\
t\in\left[\frac{1}{n+1},\frac{1}{n}\right],\\\ 1,&\
t\geq\frac{1}{n}.\end{cases}$

Set $d_{n}:=f_{n}(a_{0})$. The first item holds since $0\leq f_{n}\leq 1$ and
the second because $f_{n}f_{n+1}=f_{n}$. To establish the third, assume for a
contradiction that there is an element $x\in D$ for which $(1-f_{n}(a_{0}))x$
does not converge to $0$. Then

$\sup_{\omega}\omega((1-f_{n}(a_{0}))xx^{*}(1-f_{n}(a_{0}))$

does not converge to $0$ when we take the supremum over all states $\omega$ of
$A$. It follows that there is an $\epsilon>0$ and a sequence
$n_{1}<n_{2}<n_{3}<\cdots$ in $\mathbb{N}$ such that for each $k$ there is a
state $\omega_{k}$ with

$\omega_{k}((1-f_{n_{k}}(a_{0}))xx^{*}(1-f_{n_{k}}(a_{0})))\geq\epsilon.$

Since the unit ball of $A^{*}$ is weak* compact there is a weak* condensation
point $\mu$ of the set of functionals defined by

$A\ni a\mapsto\omega_{k}((1-f_{n_{k}}(a_{0})a(1-f_{n_{k}}(a_{0}))),\ \
k\in\mathbb{N}.$

Then $\mu(xx^{*})\geq\epsilon$, and hence $\mu$ is a non-zero positive
functional on $A$. However, $\lim_{n\to\infty}(1-f_{n}(t))t=0$ uniformly for
$t$ in the spectrum of $a_{0}$, and hence
$\lim_{n\to\infty}(1-f_{n}(a_{0}))a_{0}=0$. It follows that $\mu(a_{0})=0$,
contradicting the strict positivity of $a_{0}$. ∎

Consider now a separable $C^{*}$-subalgebra $D$ of $A$, and let

${\bf d}:=\\{d_{n}\\}_{n=1}^{\infty}$

be a sequence in $D$ with the properties specified in Lemma 4.4. Set

$D_{d_{n}}:=\left\\{a\in D:\ ad_{n}=d_{n}a=a\right\\}.$

Then $D_{d_{n}}$ is a $C^{*}$-subalgebra of $D$ and

$D_{d_{n}}\subseteq d_{n}Dd_{n}\subseteq D_{d_{n+1}}\subseteq
d_{n+1}Dd_{n+1}.$

In particular,

$\bigcup_{n=1}^{\infty}d_{n}Dd_{n}=\bigcup_{n=1}^{\infty}D_{d_{n}}.$

Set

$D({\bf d}):=\bigcup_{n=1}^{\infty}D_{d_{n}}.$

Then $D({\bf d})$ is a $*$-subalgebra of $D$ and it is dense in $D$ since
$\lim_{n\to\infty}d_{n}ad_{n}=a$ for all $a\in D$.

######  Lemma 4.5.

$D({\bf d})\subseteq\mathcal{M}_{\omega}$.

###### Proof.

The key step in the proof comes from the proof of Theorem 1.3 in [Pe] where
Pedersen introduces his famous minimal dense ideal. Let $n\in\mathbb{N}$ and
consider an element $a\in D^{+}$. Set $d:=d_{n}ad_{n}$. It suffices to show
that $d\in\mathcal{M}_{\omega}$. Since $\omega$ is densely defined there is an
$x\in A^{+}\cap\mathcal{M}_{\omega}$ such that
$\left\|x-d_{n+1}\right\|\leq\frac{1}{2}$. Using $d_{n+1}\sqrt{d}=\sqrt{d}$ we
find that

$\frac{1}{2}d\leq\sqrt{d}(1+(x-d_{n+1}))\sqrt{d}=\sqrt{d}(d_{n+1}+(x-d_{n+1}))\sqrt{d}=\sqrt{d}x\sqrt{d}.$

Note that $\sqrt{d}x\sqrt{d}\in\mathcal{M}_{\omega}$ since
$\mathcal{M}_{\omega}$ is a two-sided ideal by Lemma 4.1. It follows therefore
from the estimate above that $\omega(d)\leq
2\omega(\sqrt{d}x\sqrt{d})<\infty$. Hence $d\in\mathcal{M}_{\omega}$. ∎

######  Lemma 4.6.

$\omega^{\dagger}(xy)=\omega^{\dagger}(yx)$ for all
$x,y\in\mathcal{M}_{\omega}^{\dagger}$.

###### Proof.

Write $x=(a,\lambda),\ y=(b,\mu)$ where $a,b\in\mathcal{M}_{\omega},\
\lambda,\mu\in\mathbb{C}$. Then
$\omega^{\dagger}(xy)=\omega|_{\mathcal{M}_{\omega}}(ab+\lambda b+\mu a)$
while $\omega^{\dagger}(yx)=\omega|_{\mathcal{M}_{\omega}}(ba+\lambda b+\mu
a)$. It suffices therefore to show that
$\omega|_{\mathcal{M}_{\omega}}(ab)=\omega|_{\mathcal{M}_{\omega}}(ba)$, which
follows from the polarization identities

$ab=\frac{1}{4}\sum_{k=1}^{4}i^{k}(b+i^{k}a^{*})^{*}(b+i^{k}a^{*})$

and

$ba=\frac{1}{4}\sum_{k=1}^{4}i^{k}(b+i^{k}a^{*})(b+i^{k}a^{*})^{*}.$

∎

######  Corollary 4.7.

$\omega^{\dagger}\otimes\operatorname{Tr}_{n}(xy)=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(yx)$
for all $x,y\in M_{n}(\mathcal{M}_{\omega}^{\dagger})$.

######  Lemma 4.8.

Let $I$ be a right ideal (not necessarily closed) in the $C^{*}$-algebra $A$.
If $x\in M_{n}(I^{\dagger})$ is invertible in $M_{n}(A^{\dagger})$ then
$x^{-1}\in M_{n}(I^{\dagger})$.

###### Proof.

Write $x=(a,k)$ and $x^{-1}=(b,m)$ where $a\in M_{n}(I)$, $b\in M_{n}(A)$ and
$k,m\in M_{n}(\mathbb{C})$. Since $xx^{-1}=1$ we get the equations
$ab+kb+am=0$ and $km=1$, implying that $b=k^{-1}(-am-ab)\in M_{n}(I)$. ∎

######  Lemma 4.9.

Let $e,f$ be projections in $M_{n}(\mathcal{M}_{\omega}^{\dagger})$, and
assume that there is a partial isometry $v\in M_{n}(A^{\dagger})$ such that
$e=vv^{*}$ and $f=v^{*}v$. Then
$\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e)=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f)$.

###### Proof.

The proof is based on material from Section II. 4 of [B]. Choose a separable
$C^{*}$-algebra $D\subseteq A$ such that $e,f,v\in M_{n}(D^{\dagger})$ and
choose in $D$ a sequence $\\{d_{n}\\}_{n=1}^{\infty}$ with the properties
specified in Lemma 4.4. For each $\epsilon>0$ we can then find
$N\in\mathbb{N}$ and projections $e^{\prime},f^{\prime}\in
M_{n}(D_{d_{N}}^{\dagger})$ such that
$\left\|e-e^{\prime}\right\|\leq\epsilon$ and $\|f-f^{\prime}\|\leq\epsilon$,
cf. Lemma 6.3.1 of [RLL]. Set $u:=(2e^{\prime}-1)(2e-1)+1$. Note that

$\displaystyle\left\|1-\frac{1}{2}u\right\|=\left\|(2e^{\prime}-1)(e^{\prime}-e)\right\|\leq\|e^{\prime}-e\|,$

implying that $u$ is invertible if $\epsilon<1$. Since
$ue=2e^{\prime}e=e^{\prime}u$, we see that $ueu^{-1}=e^{\prime}$. Note that
$u\in M_{n}(\mathcal{M}_{\omega}^{\dagger})$. It follows therefore from Lemma
4.8 that $u^{-1}\in M_{n}(\mathcal{M}_{\omega}^{\dagger})$. Since
$e^{\prime}\in M_{n}(\mathcal{M}_{\omega}^{\dagger})$ by Lemma 4.5 it follows
now from Corollary 4.7 that
$\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e)$.
We observe that $u\in M_{n}(D^{\dagger})$, which implies that $u\in
M_{n}((\mathcal{M}_{\omega}\cap D)^{\dagger})$ and hence that $u^{-1}\in
M_{n}((\mathcal{M}_{\omega}\cap D)^{\dagger})$ by Lemma 4.8. Similarly,
$\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f^{\prime})=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f)$
and $sfs^{-1}=f^{\prime}$ for some invertible element $s\in
M_{n}((\mathcal{M}_{\omega}\cap D)^{\dagger})$ with $s^{-1}\in
M_{n}((\mathcal{M}_{\omega}\cap D)^{\dagger})$. We aim to show that
$\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f^{\prime})$.
Set $x:=f^{\prime}sv^{*}u^{-1}e^{\prime},\ y:=e^{\prime}uvs^{-1}f^{\prime}$.
Then $x,y\in M_{n}(D^{\dagger})$ and

$\displaystyle
xy=f^{\prime}sv^{*}u^{-1}e^{\prime}uvs^{-1}f^{\prime}=f^{\prime}sv^{*}evs^{-1}f^{\prime}$
$\displaystyle=f^{\prime}sv^{*}vv^{*}vs^{-1}f^{\prime}=f^{\prime}sfs^{-1}f^{\prime}=f^{\prime}.$

Similarly, $yx=e^{\prime}$. Since
$f^{\prime}={f^{\prime}}^{*}f^{\prime}=y^{*}x^{*}xy\leq\|x\|^{2}y^{*}y$ it
follows that $y^{*}y$ is positive and invertible in
$f^{\prime}M_{n}(D^{\dagger})f^{\prime}$. Taking the inverse in that algebra,
set

$w:=y(y^{*}y)^{-1/2}.$

Then $w^{*}w=(y^{*}y)^{-1/2}y^{*}y(y^{*}y)^{-1/2}=f^{\prime}$ and hence, in
particular, $ww^{*}$ is a projection. Note that

$\displaystyle
e^{\prime}=yxx^{*}y^{*}\leq\|x\|^{2}yy^{*}=\|x\|^{2}y(y^{*}y)^{-1/2}yy^{*}(y^{*}y)^{-1/2}y^{*}$
$\displaystyle\leq\|x\|^{2}\|y\|^{2}y(y^{*}y)^{-1/2}(y^{*}y)^{-1/2}y^{*}=\|x\|^{2}\|y\|^{2}ww^{*}.$

Then

$0\leq(1-ww^{*})e^{\prime}(1-ww^{*})\leq\|x\|^{2}\|y\|^{2}(1-ww^{*})ww^{*}(1-ww^{*})=0,$

implying that $e^{\prime}=ww^{*}e^{\prime}$. On the other hand,

$\displaystyle
e^{\prime}y=e^{\prime}uvs^{-1}f^{\prime}=ueu^{-1}uvs^{-1}f^{\prime}=uevs^{-1}f^{\prime}$
$\displaystyle=uvv^{*}vs^{-1}f^{\prime}=uvs^{-1}f^{\prime}=e^{\prime}uvs^{-1}f^{\prime}=y,$

implying that $e^{\prime}ww^{*}=ww^{*}$, and hence that $e^{\prime}=ww^{*}$.
Since $w\in M_{n}(D^{\dagger})$ and $D({\bf d})$ is dense in $D$ there is a
$k>N$ and an element $z\in M_{n}(D_{d_{k}}^{\dagger})$ such that
$\left\|z{z}^{*}-e^{\prime}\right\|<\frac{1}{2}$ and
$\left\|{z}^{*}z-f^{\prime}\right\|<\frac{1}{2}$. Since
$e^{\prime},f^{\prime},z\in M_{n}(D_{d_{k}}^{\dagger})$ it follows then from
Lemma 6.3.1 in [RLL] that there is a partial isometry $w^{\prime}\in
M_{n}(D_{d_{k}}^{\dagger})$ such that $w^{\prime}{w^{\prime}}^{*}=e^{\prime}$
and ${w^{\prime}}^{*}w^{\prime}=f^{\prime}$. Since $w^{\prime}\in
M_{n}(\mathcal{M}_{\omega}^{\dagger})$ by Lemma 4.5 it follows from Corollary
4.7 that
$\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f^{\prime})$.
∎

######  Lemma 4.10.

Let $e\in M_{n}(A^{\dagger})$ be a projection. There is a projection
$e^{\prime}\in M_{n}(\mathcal{M}_{\omega}^{\dagger})$ and a partial isometry
$v\in M_{n}(A^{\dagger})$ such that $vv^{*}=e$ and $v^{*}v=e^{\prime}$.

###### Proof.

Let $D\subseteq A$ be a separable $C^{*}$-subalgebra such that $e\in
M_{n}(D^{\dagger})$ and let $\\{d_{n}\\}_{n=1}^{\infty}$ be a sequence in $D$
with the properties specified in Lemma 4.4. Since $D({\bf d})$ is a
$*$-algebra which is dense in $D$ and since $M_{n}(D_{d_{k}}^{\dagger})$ is a
$C^{*}$-algebra it follows from Lemma 6.3.1 in [RLL] that there is a $k$, a
projection $e^{\prime}\in M_{n}(D_{d_{k}}^{\dagger})$ and a partial isometry
$v\in M_{n}(A^{\dagger})$ such that $vv^{*}=e$ and $v^{*}v=e^{\prime}$. This
completes the proof because $M_{n}(D_{d_{k}}^{\dagger})\subseteq
M_{n}(\mathcal{M}_{\omega}^{\dagger})$ by Lemma 4.5. ∎

_Proof of Theorem 4.2:_ The identity

$K_{0}(A^{\dagger})=\left\\{[e]-[f]:\ e,f\in
P_{\infty}(\mathcal{M}_{\omega}^{\dagger})\right\\}$

follows from the definition of $K_{0}(A^{\dagger})$ and Lemma 4.10. To prove
that $\omega^{\dagger}_{*}$ is well-defined assume that
$e,f,e^{\prime},f^{\prime}$ are projections in
$M_{n}(\mathcal{M}_{\omega}^{\dagger})$ such that
$[e]-[f]=[e^{\prime}]-[f^{\prime}]$ in $K_{0}(A^{\dagger})$. There is then a
projection $r\in P_{\infty}(A^{\dagger})$ such that $e\oplus f^{\prime}\oplus
r$ and $e^{\prime}\oplus f\oplus r$ are Murray-von Neumann equivalent in
$M_{N}(A^{\dagger})$ for some $N\in\mathbb{N}$. By Lemma 4.10 we may assume
that $r\in P_{\infty}(\mathcal{M}_{\omega}^{\dagger})$. It follows then from
Lemma 4.9 that

$\omega^{\dagger}\otimes\operatorname{Tr}_{N}(e\oplus f^{\prime}\oplus
r)=\omega^{\dagger}\otimes\operatorname{Tr}_{N}(e^{\prime}\oplus f\oplus r)$

and hence by linearity that

$\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e)-\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f)=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})-\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f^{\prime}).$

∎

By definition $K_{0}(A)$ is a subgroup of $K_{0}(A^{\dagger})$; the kernel of
map $K_{0}(A^{\dagger})\to K_{0}(\mathbb{C})$. The map $\omega^{\dagger}_{*}$
of Theorem 4.2 gives therefore rise, by restriction, to a homomorphism

$\omega_{*}:K_{0}(A)\to\mathbb{R}$

such that

$\omega_{*}([e]-[f])=\omega^{\dagger}\otimes\operatorname{Tr}_{n}(e)-\omega^{\dagger}\otimes\operatorname{Tr}_{n}(f)$

when $e,f\in M_{n}(\mathcal{M}_{\omega}^{\dagger})$ and $[e]-[f]\in K_{0}(A)$.

Recall that $K_{0}(A)$ comes equipped with an order given by the semi-group

$K_{0}(A)_{+}:=\left\\{[e]:\ e\in P_{\infty}(A)\right\\},$

cf. III 6 in [B].

######  Lemma 4.11.

Let $e$ be a projection in $M_{k}(A)$. Then $e\in
M_{k}(\mathcal{M}_{\omega})$.

###### Proof.

Since $\omega_{k}$ is densely defined by Lemma 3.2 there is an
$x\in\mathcal{M}_{\omega_{k}}$ such that $\left\|e-x\right\|<\frac{1}{2}$.
Then

$\frac{1}{2}e\leq e\left(1+(x-e)\right)e=exe\in\mathcal{M}_{\omega_{k}}.$

Hence $\omega_{k}(e)\leq 2\omega_{k}(exe)<\infty$, showing that
$e\in\mathcal{M}_{\omega_{k}}$. This completes the proof because
$\mathcal{M}_{\omega_{k}}=M_{k}(\mathcal{M}_{\omega})$ by (f) of Lemma 3.1.

∎

Since $\omega^{\dagger}\otimes\operatorname{Tr}_{n}=\omega_{n}$ on
$M_{n}(\mathcal{M}_{\omega})$ it follows from Lemma 4.11 and Theorem 4.2 that

$\omega_{*}([e]-[f])=\omega_{n}(e)-\omega_{n}(f)$ (4.1)

when $e$ and $f$ are projections in $M_{n}(A)$. In particular, it follows that
$\omega_{*}$ is a positive homomorphism in the sense that

$\omega_{*}(K_{0}(A)_{+})\subseteq[0,\infty).$

###  4.2. On lower semi-continuity of traces

######  Proposition 4.12.

Let $A$ be a $\sigma$-unital $C^{*}$-algebra and $\psi$ a densely defined
trace on $A$. There is a densely defined lower semi-continuous trace
$\tilde{\psi}$ on $A$ such that

  1. (a)

$\tilde{\psi}(a)\leq\psi(a)$ for all $a\in A^{+}$, and

  2. (b)

if $\phi$ is a lower semi-continuous weight on $A$ such that
$\phi(a)\leq\psi(a)$ for all $a\in A^{+}$, then $\phi(a)\leq\tilde{\psi}(a)$
for all $a\in A^{+}$.

The trace $\tilde{\psi}$ has the property that $\psi_{*}={\tilde{\psi}}_{*}$
on $K_{0}(A)$.

###### Proof.

Let $\\{d_{n}\\}$ be a sequence in $A$ with the properties specified in Lemma
4.4. Then $d_{n}^{2}=d_{n+1}d_{n}^{2}d_{n+1}\leq d_{n+1}^{2}$ and hence

$\psi(d_{n}a^{*}ad_{n})=\psi(ad_{n}^{2}a^{*})\leq\psi(ad_{n+1}^{2}a^{*})=\psi(d_{n+1}a^{*}ad_{n+1})$

for all $n$ and all $a\in A$. We can therefore define
$\tilde{\psi}:A^{+}\to[0,\infty]$ such that

$\tilde{\psi}(a)=\lim_{n\to\infty}\psi(d_{n}ad_{n})=\sup_{n}\psi(d_{n}ad_{n}).$

Note that $\tilde{\psi}$ is a weight. It follows from Lemma 4.5 that
$\psi(d_{n}\ \cdot\ d_{n})$ is (the restriction to $A^{+}$ of) a bounded
linear functional on $A$ and hence also that $\tilde{\psi}$ is lower semi-
continuous. This is used in the following calculation. Let $a\in A$. Then

$\displaystyle\tilde{\psi}(aa^{*})=\lim_{n\to\infty}\tilde{\psi}(ad_{n}^{2}a^{*})=\lim_{n\to\infty}\lim_{k\to\infty}\psi(d_{k}ad_{n}^{2}a^{*}d_{k})$
$\displaystyle=\lim_{n\to\infty}\lim_{k\to\infty}\psi(d_{n}a^{*}d_{k}^{2}ad_{n})=\lim_{n\to\infty}\psi(d_{n}a^{*}ad_{n})=\tilde{\psi}(a^{*}a);$

i.e. $\tilde{\psi}$ is a trace. Since

$\tilde{\psi}(a^{*}a)=\lim_{n\to\infty}\psi(d_{n}a^{*}ad_{n})=\lim_{n\to\infty}\psi(ad_{n}^{2}a^{*})\leq\psi(aa^{*})={\psi}(a^{*}a),$

we conclude that $\tilde{\psi}(b)\leq\psi(b)$ for all $b\in A^{+}$, i.e. (a)
holds. To see that so does (b), let $\phi$ be a lower semi-continuous weight
on $A$ such that $\phi(a)\leq\psi(a)$ for all $a\in A^{+}$. Let $b\in A$. Then

$\displaystyle\phi(b^{*}b)=\lim_{n\to\infty}\phi(b^{*}d_{n}^{2}b)\leq\lim_{n\to\infty}\psi(b^{*}d_{n}^{2}b)$
$\displaystyle=\lim_{n\to\infty}\psi(d_{n}bb^{*}d_{n})=\tilde{\psi}(bb^{*})=\tilde{\psi}(b^{*}b).$

To prove that $\psi_{*}={\tilde{\psi}}_{*}$ on $K_{0}(A)$ we prove that
$\psi^{\dagger}_{*}={\tilde{\psi}}^{\dagger}_{*}$ on $K_{0}(A^{\dagger})$,
which will suffice. To this end note that
$\mathcal{M}_{\psi}^{\dagger}\subseteq\mathcal{M}_{\tilde{\psi}}^{\dagger}$
since $\tilde{\psi}\leq\psi$. By Theorem 4.2 it suffices therefore to show
that
$\psi^{\dagger}\otimes\operatorname{Tr}_{n}(e)=\tilde{\psi}^{\dagger}\otimes\operatorname{Tr}_{n}(e)$
when $e$ is a projection in $M_{n}(\mathcal{M}_{\psi}^{\dagger})$. For this
note that it follows from the first part of the proof of Lemma 4.9 that there
is a $k\in\mathbb{N}$ and a projection $e^{\prime}\in M_{n}(D_{k}^{\dagger})$,
where $D_{k}:=d_{k}Ad_{k}$, such that
$\psi^{\dagger}\otimes\operatorname{Tr}_{n}(e)=\psi^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})$
and
$\tilde{\psi}^{\dagger}\otimes\operatorname{Tr}_{n}(e)=\tilde{\psi}^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})$.
This completes the proof because
$\psi^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})=\tilde{\psi}^{\dagger}\otimes\operatorname{Tr}_{n}(e^{\prime})$
since $\psi$ and $\tilde{\psi}$ clearly agree on $D_{k}$. ∎

The most famous traces that are not lower semi-continuous are the traces on
the $C^{*}$-algebra $\mathbb{K}$ of compact operators on an infinite
dimensional separable Hilbert space constructed by J. Dixmier in [Di].
Dixmiers traces are used in non-commutative geometry as described in Chapter
IV of [Co2] and Dixmiers construction has been investigated and extended by
many mathematicians. An almost defining feature of these traces is that they
invariably vanish on the operators of finite rank, and hence in particular on
all projections in $\mathbb{K}$. It follows from this that if we apply
Proposition 4.12 with a Dixmier trace in the role of $\psi$, the resulting
lower semi-continuous trace $\tilde{\psi}$ is the zero trace. In contrast, if
we apply Proposition 4.12 to the trace $\psi:\mathbb{K}^{+}\to[0,\infty]$
defined by

$\psi(a)=\begin{cases}\operatorname{Tr}(a)&\text{when $a$ has finite rank}\\\
\infty&\text{when $a$ does not have finite rank,}\end{cases}$

the result is the usual trace $\operatorname{Tr}$ on $\mathbb{K}$.

######  Example 4.13.

Let $X$ be a locally compact Hausdorff space, $A$ a $C^{*}$-algebra and $\tau$
a non-zero densely defined trace on $A$. Set

$B:=C_{0}(X)\otimes A=C_{0}(X,A);$

the $C^{*}$-algebra of $A$-valued continuous functions on $X$ vanishing in
norm at infinity. Let $\\{x_{i}\\}_{i=1}^{\infty}$ be a sequence of points in
$X$ such that $i\neq j\Rightarrow x_{i}\neq x_{j}$ and
$\lim_{i\to\infty}x_{i}=\infty$ in the sense that for every compact subset
$K\subseteq X$ there is a $N_{K}\in\mathbb{N}$ such that $x_{i}\notin K$ when
$i\geq N_{K}$. Let $\\{t_{i}\\}_{i=1}^{\infty}$ be a sequence of positive real
numbers such that $\lim_{i\to\infty}t_{i}=\infty$, and let finally
$\xi\in\beta\mathbb{N}\backslash\mathbb{N}$ be a free ultrafilter in
$\mathbb{N}$ which we think of as an element of the Stone-Čech remainder of
$\mathbb{N}$. Define $\mu:B^{+}\to[0,\infty]$ such that

$\mu(b)=\lim_{N\to\xi}\frac{1}{N}\sum_{n=1}^{N}t_{n}\tau(b(x_{n}))$

when
$\left\\{\frac{1}{N}\sum_{n=1}^{N}t_{n}\tau(b(x_{n}))\right\\}_{N\in\mathbb{N}}$
is bounded and

$\mu(b)=\infty$

when
$\left\\{\frac{1}{N}\sum_{n=1}^{N}t_{n}\tau(b(x_{n}))\right\\}_{N\in\mathbb{N}}$
is not bounded. It is clear that $\mu$ is a densely defined trace on $B$; it
is zero on elements of $C_{0}(X,A)$ of compact support. To see that $\mu$ is
not zero and not lower semi-continuous, choose an element $a\in A^{+}$ such
that $0<\tau(a)<\infty$. For every $n$ we choose a compactly supported
continuous function $f_{n}\in C_{0}(X)$ such that $0\leq f_{n}\leq 1$,
$f_{n}(x_{n})=1$ and $f_{n}(x_{i})=0,\ i\neq n$. Then

$g:=\sum_{n=1}^{\infty}t_{n}^{-1}f_{n}\otimes a\in C_{0}(X,A)^{+},$

and $\mu(g)=1$. Hence $\mu$ is not zero. Since
$g=\lim_{L\to\infty}\sum_{n=1}^{L}t_{n}^{-1}f_{n}\otimes a$ and

$\mu\left(\sum_{n=1}^{L}t_{n}^{-1}f_{n}\otimes a\right)=0$

for all $L$, we see that $\mu$ is not lower semi-continuous.

The algebra $A$ in Example 4.13 is not simple and all the examples of traces
that are not lower semi-continuous exhibited in Example 4.13 share the
property with the Dixmier traces that they vanish on projections. The next
example does not have any of these two deficiencies, although the underlying
idea is the same.

######  Example 4.14.

Let $B(\mathbb{N},\mathbb{Q})$ denote the set of bounded sequences of rational
numbers. Let $G$ be the subgroup of $B(\mathbb{N},\mathbb{Q})$ consisting of
the sequences $\\{a_{n}\\}_{n=1}^{\infty}$ in $\mathbb{Q}$ with the property
that there is an $N\in\mathbb{N}$ such that

$a_{i}=\frac{q}{i^{2}}$

for some $q\in\mathbb{Q}$ and all $i\geq N$. Set

$G^{+}=\left\\{\\{a_{n}\\}_{n=1}^{\infty}\in G:\ a_{n}>0\ \forall
n\in\mathbb{N}\right\\}\cup\\{0\\}.$

Then $(G,G^{+})$ is a countable simple dimension group. Set

$\Sigma:=\left\\{\\{a_{n}\\}_{n=1}^{\infty}\in G^{+}:\ a_{n}<1\ \ \forall
n\in\mathbb{N}\right\\}.$

Then $\Sigma$ is a scale in $(G,G^{+})$ and there is therefore a simple AF
algebra $A$ such that

$(K_{0}(A),K_{0}(A)^{+},\Sigma(A))=(G,G^{+},\Sigma).$

This follows from [E1] and [EHS]. If necessary, see Proposition 1.4.5 in [R].

For each $k\in\mathbb{N}$ there is a bounded trace $\tau_{k}$ on $A$ such that
${\tau_{k}}_{*}\left(\\{a_{n}\\}_{n=1}^{\infty}\right)=a_{k}$. Note that
$\left\|\tau_{k}\right\|\leq 1$. Let $\omega$ be a free-ultrafilter in
$\mathbb{N}$, which we consider as element of the Stone-Ĉech remainder of
$\mathbb{N}$. Define

$\tau:A^{+}\to[0,\infty]$

such that

$\tau(a)=\lim_{N\to\omega}\frac{1}{N}\sum_{k=1}^{N}\tau_{k}(a)k^{2}$

when
$\left\\{\frac{1}{N}\sum_{k=1}^{N}\tau_{k}(a)k^{2}\right\\}_{N=1}^{\infty}$ is
bounded and

$\tau(a)=\infty$

when
$\left\\{\frac{1}{N}\sum_{k=1}^{N}\tau_{k}(a)k^{2}\right\\}_{N=1}^{\infty}$ is
unbounded. Then $\tau$ is clearly a trace. To show that $\tau$ is non-zero fix
$j\in\mathbb{N}$ and let $p_{j}\in A$ be a projection such that

$\tau_{k}(p_{j})=\begin{cases}\frac{1}{k^{2}},&\ k>j,\\\ \frac{1}{2},&\
k=j,\\\ 2^{-j},&\ k<j.\end{cases}$

Then

$\tau_{k}\left(\sum_{j=1}^{\infty}\frac{p_{j}}{j^{2}}\right)k^{2}=\frac{1}{2}+\sum_{j<k}\frac{1}{j^{2}}+\sum_{j>k}\frac{2^{-j}k^{2}}{j^{2}},$

and hence

$\tau\left(\sum_{j=1}^{\infty}\frac{p_{j}}{j^{2}}\right)=\frac{1}{2}+\sum_{j=1}^{\infty}\frac{1}{j^{2}}=\frac{1}{2}+\frac{\pi^{2}}{6}.$

We note that for $N>M$,

$\displaystyle\sum_{k=1}^{N}\tau_{k}\left(\sum_{j=1}^{M}\frac{p_{j}}{j^{2}}\right)k^{2}$
$\displaystyle=\sum_{k=M+1}^{N}\sum_{j=1}^{M}\frac{1}{j^{2}}+\sum_{k=1}^{M}\tau_{k}\left(\sum_{j=1}^{M}\frac{p_{j}}{j^{2}}\right)k^{2}$
$\displaystyle=\sum_{k=M+1}^{N}\sum_{j=1}^{M}\frac{1}{j^{2}}+\sum_{k=1}^{M}\left((\sum_{j=1}^{k-1}\frac{1}{j^{2}})+\frac{1}{2}+(\sum_{j=k+1}^{M}\frac{2^{-j}}{j^{2}}k^{2})\right).$

Since

$(\sum_{j=1}^{k-1}\frac{1}{j^{2}})+\frac{1}{2}+(\sum_{j=k+1}^{M}\frac{2^{-j}}{j^{2}}k^{2})$

is bounded by a constant $C$ so that

$\sum_{k=1}^{M}\left((\sum_{j=1}^{k-1}\frac{1}{j^{2}})+\frac{1}{2}+(\sum_{j=k+1}^{M}\frac{2^{-j}}{j^{2}}k^{2})\right)$

is bounded by $MC$, and since

$\sum_{k=M+1}^{N}\sum_{j=1}^{M}\frac{1}{j^{2}}=(N-M)\sum_{j=1}^{M}\frac{1}{j^{2}}$

we find that

$\tau\left(\sum_{j=1}^{M}\frac{p_{j}}{j^{2}}\right)=\sum_{j=1}^{M}\frac{1}{j^{2}}.$

Thus

$\tau\left(\sum_{j=1}^{M}\frac{p_{j}}{j^{2}}\right)\leq\tau\left(\sum_{j=1}^{\infty}\frac{p_{j}}{j^{2}}\right)-\frac{1}{2}$

for all $M$. Since
$\lim_{M\to\infty}\sum_{j=1}^{M}\frac{p_{j}}{j^{2}}=\sum_{j=1}^{\infty}\frac{p_{j}}{j^{2}}$
this shows that $\tau$ is not lower semi-continuous. To see that $\tau$ is
densely defined note that for any projection $q\in A$ there is an
$N\in\mathbb{N}$ such that

$\tau_{k}(q)\leq\tau_{k}(Np_{1})\leq
N\tau_{k}\left(\sum_{j=1}^{\infty}\frac{p_{j}}{j^{2}}\right)$

for all $k\in\mathbb{N}$, implying that

$\tau(q)\leq
N\tau\left(\sum_{j=1}^{\infty}\frac{p_{j}}{j^{2}}\right)=N(\frac{1}{2}+\frac{\pi^{2}}{6})<\infty.$

Since every element of $A^{+}$ can be approximated in norm by a linear
combination of projections with non-negative coefficients, it follows that
$\tau$ is densely defined. Note that $\tau$ does not vanish on any non-zero
projection. Indeed, when $e$ is a non-zero projection in $A$ there is a
positive rational number $q$ such that $\tau_{k}(e)=q\frac{1}{k^{2}}$ for all
sufficient large $k$, leading to the conclusion that $\tau(e)=q>0$.

Like the constructions in Example 4.13 the preceding can also varied. To see
one possible variation, let $0<\epsilon<1$ and define

$\tau_{\epsilon}:A^{+}\to[0,\infty]$

such that

$\tau_{\epsilon}(a)=\lim_{N\to\omega}\frac{1}{N}\sum_{k=1}^{N}\tau_{k}(a)k^{1+\epsilon}$

when
$\left\\{\frac{1}{N}\sum_{k=1}^{N}\tau_{k}(a)k^{1+\epsilon}\right\\}_{N=1}^{\infty}$
is bounded and

$\tau_{\epsilon}(a)=\infty$

when
$\left\\{\frac{1}{N}\sum_{k=1}^{N}\tau_{k}(a)k^{1+\epsilon}\right\\}_{N=1}^{\infty}$
is unbounded. Then

$\displaystyle\tau_{k}\left(\sum_{j=1}^{\infty}j^{-(1+\epsilon)}p_{j}\right)k^{1+\epsilon}=\sum_{j=1}^{\infty}\left(\frac{k}{j}\right)^{1+\epsilon}\tau_{k}(p_{j})$
$\displaystyle=\frac{1}{2}+\sum_{j<k}\frac{1}{k^{2}}\left(\frac{k}{j}\right)^{1+\epsilon}+\sum_{j>k}2^{-j}\left(\frac{k}{j}\right)^{1+\epsilon}$
$\displaystyle=\frac{1}{2}+\frac{k^{1+\epsilon}}{k^{2}}\sum_{j<k}\frac{1}{j^{1+\epsilon}}+\sum_{j>k}2^{-j}\left(\frac{k}{j}\right)^{1+\epsilon}.$

Since
$\lim_{k\to\infty}\sum_{j>k}2^{-j}=\lim_{k\to\infty}\frac{k^{1+\epsilon}}{k^{2}}\sum_{j<k}\frac{1}{j^{1+\epsilon}}=0$
we have that

$\lim_{k\to\infty}\tau_{k}\left(\sum_{j=1}^{\infty}j^{-(1+\epsilon)}p_{j}\right)k^{1+\epsilon}=\frac{1}{2},$

and hence that

$\tau_{\epsilon}\left(\sum_{j=1}^{\infty}j^{-(1+\epsilon)}p_{j}\right)=\frac{1}{2}.$

To see that $\tau_{\epsilon}$ is not lower semi-continuous and not
proportional to $\tau$ note that $\tau_{\epsilon}(e)=0$ for all projections
$e\in A$.

Of course, the sum of the usual trace and a Dixmier trace on $\mathbb{K}$
gives also examples of densely defined traces on a simple $C^{*}$-algebra
which are not lower semi-continuous and do not vanish on projections.

## References

  * [B] B. Blackadar, K-Theory for operator algebras, MSRI Publ. 5, Springer Verlag, 1986.
  * [C] F. Combes, Poids sur un $C^{*}$-algèbre, J. Math. Pures Appl. 47 (1968), 57-100.
  * [Co1] A. Connes, An analogue of the Thom isomorphism for crossed products of a $C^{*}$-algebra, Adv. Math. 39 (1981), 31-55.
  * [Co2] A. Connes, Non-commutative differential geometry, Publications mathématiques de l’I.H.É.S., 62 (1985), p. 41–144.
  * [Di] J. Dixmier, Existence de traces non normales, C.R. Acad. Sci., Paris 262 (1966), 1107-1108.
  * [EHS] E.G. Effros, D.E. Handelman and C.-L. Shen, Dimension groups and their affine representations, Amer. J. Math. 102 (1980), 385–407.
  * [E1] G. A. Elliott, On the classification of inductive limits of sequences of semisimple finite-dimensional algebras, J. Algebra 38 (1976), 29–44.
  * [E2] G.A. Elliott, K-theory and traces, C.R. Math. Rep. Acad. Sci. Canada 44 (2022), 1–15.
  * [Pe] G. K. Pedersen, Measure theory for $C^{*}$-algebras, Math. Scand. 19 (1966), 131–145.
  * [RLL] M. Rørdam, F. Larsen and N.J. Laustsen, An introduction to K-theory for $C^{*}$-algebras, London Mathematical Society - Student Texts, vol. 49, Cambridge University Press, Cambridge, 2000.
  * [R] M. Rørdam, Classification of Nuclear $C^{*}$-Algebras. (J. Cuntz and V. Jones, eds.), vol. 126, Encyclopaedia of Mathematical Sciences. Subseries: Operator Algebras and Non-commutative Geometry, no. VII, Springer Verlag, Berlin, Heidelberg, 2001, pp. 1–145.


# Zero-Shot Sketch-Based Image Retrieval with Structure-aware Asymmetric
Disentanglement

Jiangtong Li, Zhixin Ling, Li Niu, Liqing Zhang

###### Abstract

The goal of Sketch-Based Image Retrieval (SBIR) is using free-hand sketches to
retrieve images of the same category from a natural image gallery. However,
SBIR requires all test categories to be seen during training, which cannot be
guaranteed in real-world applications. So we investigate more challenging
Zero-Shot SBIR (ZS-SBIR), in which test categories do not appear in the
training stage. After realizing that sketches mainly contain structure
information while images contain additional appearance information, we attempt
to achieve structure-aware retrieval via asymmetric disentanglement. For this
purpose, we propose our STRucture-aware Asymmetric Disentanglement (STRAD)
method, in which image features are disentangled into structure features and
appearance features while sketch features are only projected to structure
space. Through disentangling structure and appearance space, bi-directional
domain translation is performed between the sketch domain and the image
domain. Extensive experiments demonstrate that our STRAD method remarkably
outperforms state-of-the-art methods on three large-scale benchmark datasets.

##  1 Introduction

In recent years, with the rapid growth of multimedia data on the internet,
image retrieval is playing a more and more important role in many fields.
Since sketches can be easily drawn and reveal some characteristics of the
target images, sketch-based image retrieval (SBIR), which uses a sketch to
retrieve the images of the same category, has become widely accepted among
users. Therefore, SBIR has also attracted widespread attention in research
community ((Saavedra 2014; Saavedra, Barrios, and Orand 2015; Li et al. 2016;
Sangkloy et al. 2016; Lu et al. 2018)).

Figure 1: Traditional SBIR methods which correlate sketches/images with their
category labels cannot generalize well to unseen categories. Our STRAD method
learns structural correspondence between sketches and images from seen
categories, which can be transferred to unseen ones.

In the conventional SBIR setting, it assumes that the images and sketches in
training and test sets share the same set of categories. However, in real-
world applications, the categories of test sketches/images may be out of the
scope of training categories, leading to a more challenging task called zero-
shot sketch-based image retrieval (ZS-SBIR) (Shen et al. 2018), which assumes
that test categories do not appear in the training stage. In the remainder of
this paper, we refer to training (_resp._ , test) categories as seen (_resp._
, unseen) categories (Dupont 2018). Traditional SBIR methods (Yu et al. 2017)
suffer from sharp performance drop in ZS-SBIR setting, because traditional
SBIR methods may take a shortcut by correlating sketches/images with their
category labels and retrieving the images from the same category as the query
sketch (Yelamarthi et al. 2018). This shortcut is very effective when test
data share the same categories as training data, but can hardly generalize to
unseen categories as depicted in Figure 1.

To overcome the drawbacks of traditional SBIR methods in ZS-SBIR setting,
several ZS-SBIR methods have been proposed to boost the performance on unseen
categories, which can be categorized into the following groups: 1) Yelamarthi
et al. (2018) used a generative model based on aligned sketch-image pairs (a
sketch is drawn based on a given image and thus has roughly the same outline
as this image) to learn better correlation between sketches and images.
However, the aligned sketch-image pairs are either unavailable or very
expensive; 2) Some works (Shen et al. 2018; Dutta and Akata 2019; Verma et al.
2019; Dey et al. 2019; Zhang et al. 2020) employed category-level semantic
information to reduce the gap between seen categories and unseen categories.
Whereas, category-level semantic information like word vectors (Mikolov et al.
2013) are sometimes inaccessible or ambiguous. Besides, they only use semantic
information of seen categories during training, which may limit the
generalization ability to unseen categories; 3) Liu et al. (2019) located the
catastrophic forgetting phenomenon and preserved the knowledge of model
pretrained on ImageNet (Deng et al. 2009). Despite its competitive
performance, it relies on auxiliary WordNet (Fellbaum 2012) knowledge and its
performance gain is mainly from the pre-training strategy; 4) Dutta and Biswas
(2019) proposed to disentangle the representations of two domains (_i.e._ ,
sketch and image) into domain-independent and domain-specific representations.
Nevertheless, this symmetric disentanglement approach is not well-tailored for
SBIR task and ignores the asymmetric relation between sketch and image domain,
that is, sketches only contain structure information (_e.g._ , outline, shape)
while images additionally contain appearance information (_e.g._ , color,
texture, and background).

Being aware of the asymmetric relation between sketch domain and image domain,
we conjecture that the key of sketch-based image retrieval might be
successfully matching the structure information between sketches and images.
In this paper, we attempt to learn the structural correspondence between
sketches and images on seen categories, which could generalize to unseen
categories and facilitate ZS-SBIR task. For example, as shown in Figure 1, the
structural correspondence between sketches and images within the same category
can be learned based on seen categories, _e.g._ , the structural similarity
between “door” sketch and “door” image _w.r.t._ the global/local contour and
the layout of different components. In the test stage, given sketches and
images from an unseen category “church”, without knowing which category they
belong to, we can still verify whether they belong to the same category on the
premise of their structural similarity learnt from seen categories.

To obtain the structure features of sketches and images, we propose a
STRucture-aware Asymmetric Disentanglement (STRAD) method to disentangle image
feature into structure feature (_e.g._ , outline, shape) and appearance
feature (_e.g._ , color, texture, and background) while projecting sketch
feature into structure feature only. Our asymmetric disentanglement method is
different from symmetric disentanglement in Dutta and Biswas (2019), because
our disentangled representations have explicit meanings (_i.e._ , structure
and appearance) and are specifically designed for SBIR task. Note that
although several previous methods (Yu et al. 2016; Liu et al. 2017) also
intended to obtain the structure information of images by directly extracting
edge maps from images, the low-level edge maps obtained in this way may not be
very reliable due to possibly noisy and redundant information, which severely
compromises the effectiveness of these methods in ZS-SBIR. In contrast, our
method could extract high-level robust structure features from both sketches
and images.

Our proposed STRAD method is illustrated in Figure 2. We first use a pre-
trained model to extract features from sketches (_resp._ , images) as sketch
(_resp._ , image) features. Then, the image features are disentangled into
structure features and appearance features, while the sketch features are also
projected to the structure space. Furthermore, bi-directional domain
translation is performed through the structure features and appearance
features. Concretely, for image-to-sketch translation, we project the image
features to structure features and then generate sketch features from the
structure features. For sketch-to-image translation, we project the sketch
features to structure features, which are combined with variational appearance
features to compensate for the appearance uncertainty when generating image
features from the sketch features.

Finally, we perform retrieval in all three spaces (_i.e._ , structure space,
image space, and sketch space), to combine the best of three worlds.
Apparently, the retrieval in structure space and sketch space is structure-
aware. Image feature is generated from structure feature and variational
appearance feature. Since variational appearance feature is category-agnostic,
the retrieval in image space is also structure-aware. The effectiveness of our
proposed STRAD method is verified by comprehensive experimental results on
three large-scale benchmark datasets. Our main contributions are summarized as
follows:

  * •

Based on the asymmetric relation between image domain and sketch domain, we
propose to learn structural correspondence between these two domains via
asymmetric disentanglement.

  * •

We design a novel STRucture-aware Asymmetric Disentanglement (STRAD) method
and a hybrid retrieval strategy in three spaces.

  * •

Comprehensive results on three popular benchmark datasets show that our method
significantly outperforms the state-of-the-art methods.

##  2 Related Work

###  2.1 SBIR and ZS-SBIR

The main goal of sketch-based image retrieval (SBIR) is to bridge the gap
between image domain and sketch domain. Before deep learning was introduced to
this task, hand-crafted based methods generally extracted the edge maps from
natural images and then matched them with sketches using hand-craft feature
(Eitz et al. 2010). In recent years, deep learning based methods have become
popular in this area. To reduce the gap between image domain and sketch
domain, variants of siamese network (Qi et al. 2016) and ranking loss
(Sangkloy et al. 2016) were adopted for this task. Besides, semantic
information and adversarial loss were also introduced to preserve the domain
invariant information (Zhu, Xie, and Fang 2016; Chen and Fang 2018).

Zero-shot sketch-based image retrieval (ZS-SBIR) was proposed by Shen et al.
(2018). To reduce the intra-class variance in sketches and stabilize the
training process, semantic information was leveraged in (Dutta and Akata 2019;
Verma et al. 2019; Dey et al. 2019; Dutta and Biswas 2019; Zhang et al. 2020).
To reduce the gap between seen and unseen categories, a generative model along
with aligned data pairs was proposed in (Yelamarthi et al. 2018). To adapt the
pre-trained model to ZS-SBIR without forgetting the knowledge from ImageNet,
semantic-aware knowledge preservation was used in Liu et al. (2019). However,
all of the above methods did not consider the special relation between images
and sketches, and treated them equally in their models.

###  2.2 Disentangled Representation

Disentangled representation learning aims to divide the latent representation
into multiple units, with each unit corresponding to one latent factor (_e.g._
, position, scale). Disentangled representations are more generalizable and
semantically meaningful, and thus useful for a variety of tasks.

Disentangled representation learning methods can be categorized into
unsupervised methods and supervised methods according to whether supervision
for latent factors is available. For unsupervised disentanglement, abundant
methods (Chen et al. 2016; Higgins et al. 2017; Chen et al. 2018) have been
developed. Most of them encouraged statistical independence across different
dimensions of the latent representation while maintaining the mutual
information between input data and latent representations. For supervised
disentanglement, Kingma et al. (2014) used disentangled representation to
enhance semi-supervised learning. Zheng et al. (2019) proposed DG-Net to
integrate discriminative and generative learning using disentangled
representation. Hadad, Wolf, and Shahar (2018) proposed a two-step
disentanglement method, which disentangles the label information from the
original representation and enables feature reconstruction from decomposed
features. Besides, supervised disentanglement has been applied to different
tasks, like face recognition (Liu et al. 2018), image generation (Harsh Jha et
al. 2018), and style transfer (Yang et al. 2019). Our work applies asymmetric
disentangled representation learning to facilitate structure-aware retrieval.

###  2.3 Domain Translation

Many domain translation approaches, like CycleGAN (Zhu et al. 2017) and
DiscoGAN (Kim et al. 2017), have been proposed to translate figures between
two domains (_e.g._ , sketch and image domains). In this subsection, we mainly
discuss the domain translation methods based on disentangled representation.
Overall speaking, they disentangle latent representation into domain-specific
representation and domain-invariant representation. In our problem, structure
(_resp._ , appearance) features can be treated as domain-invariant (_resp._ ,
specific) representation. The translation between two domains in previous
works (Lee et al. 2018; Huang et al. 2018) is generally symmetric. In
contrast, the translation between sketch domain and image domain in our
problem is asymmetric because image domain has additional domain-specific
representation compared with sketch domain.

Figure 2: An overview of our STRAD method. We first adopt VGG-16 (Simonyan and
Zisserman 2015) to extract features from images and sketch. Then we
disentangle the image feature into appearance feature and structure feature,
through which bi-directional domain translation is performed between the image
domain and the sketch domain.

##  3 Methodology

In this section, we introduce our proposed STRAD method for zero-shot sketch-
based image retrieval. In Section 3.1, we state the problem definition and
notation. In Section 3.2, we elaborate our STRAD method in detail. In Section
3.3, we discuss our retrieval strategy.

###  3.1 Problem Definition

In this paper, we focus on zero-shot sketch-based image retrieval, where only
the sketches and images from seen categories are used for training. In the
test stage, our proposed method is expected to use the sketches to retrieve
the images, the categories of which are unseen during training.

Formally, given a sketch dataset
$\mathcal{S}_{sk}$=$\\{(\mathbf{x}_{sk,i},y_{i})|y_{i}\in\mathcal{Y}\\}$ and
an image dataset
$\mathcal{S}_{im}$=$\\{(\mathbf{x}_{im,j},y_{j})|y_{j}\in\mathcal{Y}\\}$,
where $\mathcal{Y}$ is category label set, and $(\mathbf{x}_{sk,i},y_{i})$
(_resp._ , $(\mathbf{x}_{im,j},y_{j})$) represents a sketch (_resp._ , image)
with its corresponding category label, we follow the zero-shot setting in Shen
et al. (2018) to split all categories $\mathcal{Y}$ into $\mathcal{Y}^{tr}$
and $\mathcal{Y}^{te}$, in which no overlap exists between two label sets,
_i.e._ , $\mathcal{Y}^{tr}\cap\mathcal{Y}^{te}=\emptyset$. Based on the
partition of label set $\mathcal{Y}$, we can split the sketch (_resp._ ,
image) dataset into $\mathcal{S}^{tr}_{sk}$ and $\mathcal{S}^{te}_{sk}$
(_resp._ , $\mathcal{S}^{tr}_{im}$ and $S^{te}_{im}$). In the training stage,
our model can only process the data in $\mathcal{S}^{tr}_{sk}$ and
$\mathcal{S}_{im}^{tr}$. During testing, given a sketch $x_{sk}$ from
$\mathcal{S}^{te}_{sk}$, our model needs to retrieve the images belonging to
the same category from test image gallery $\mathcal{S}^{te}_{im}$.

###  3.2 Our Framework

An overview of our method is illustrated in Figure 2. In this section, we will
introduce our method from Asymmetric Disentanglement and Bi-direction Domain
Translation.

#### Asymmetric Disentanglement

To achieve the goal of structure-aware retrieval, this module should decompose
structure information and appearance information, and align images and
sketches in the shared structure space.

Therefore, we first disentangle the image features into appearance features
and structure features. Given an image $\mathbf{x}_{im}$ and a sketch
$\mathbf{x}_{sk}$ from the same category $y$ , we first use fixed backbone
model, _i.e._ , VGG-16 (Simonyan and Zisserman 2015), to produce their image
feature $\mathbf{f}_{im}$ and sketch feature $\mathbf{f}_{sk}$. For image
feature $\mathbf{f}_{im}$, we adopt two image encoders $E_{im}^{st}$ and
$E_{im}^{ap}$ to disentangle $\mathbf{f}_{im}$ into image structure feature
$\mathbf{f}_{im}^{st}$ and image appearance feature $\mathbf{f}_{im}^{ap}$.
Besides, to project sketch feature $\mathbf{f}_{sk}$ to the same structure
space as $\mathbf{f}_{im}^{st}$, a sketch encoder $E_{sk}^{st}$ is adopted to
obtain sketch structure feature $\mathbf{f}_{sk}^{st}$. The above process is
formulated as follows,

$\mathbf{f}_{im}^{ap}\\!=\\!E_{im}^{ap}(\mathbf{f}_{im});\quad\mathbf{f}_{im}^{st}\\!=\\!E_{im}^{st}(\mathbf{f}_{im});\quad\mathbf{f}_{sk}^{st}\\!=\\!E_{sk}^{st}(\mathbf{f}_{sk}).$
(1)

To capture the structural correspondence between images and sketches, we
expect the structure features from both images and sketches to be aligned in
the same space. Moreover, in the structure space shared by sketch and image,
we expect intra-class coherence and inter-class separability across different
domains (_i.e._ , sketch domain and image domain). Specifically, we expect to
pull close the image/sketch structure features of the same category and push
apart the image/sketch structure features from different categories. It has
been proved in Liu et al. (2019) that a simple classification loss can
accomplish the above task well. Therefore, we employ a structure classifier on
both image structure features and sketch structure features to distinguish
their category labels, by using the cross-entropy classification loss:

$\displaystyle\mathcal{L}_{cls}=$
$\displaystyle-\log\frac{\exp({\bm{w}_{y}^{\text{T}}}\mathbf{f}_{im}^{st}+b_{y})}{\sum_{k\in\mathcal{Y}^{tr}}\exp({\bm{w}_{k}^{\text{T}}\mathbf{f}_{im}^{st}+b_{k}})}$
(2)
$\displaystyle-\log\frac{\exp({\bm{w}_{y}^{\text{T}}}\mathbf{f}_{sk}^{st}+b_{y})}{\sum_{k\in\mathcal{Y}^{tr}}\exp({\bm{w}_{k}^{\text{T}}\mathbf{f}_{sk}^{st}+b_{k}})},$

where $\bm{w}_{k}$ and $b_{k}$ are learnable parameters in the structure
classifier corresponding to category $k$. Recall that $y$ is the category
label of $\mathbf{x}_{im}$ and $\mathbf{x}_{sk}$. Although the structure space
regulated by (2) seems sufficient to perform sketch-based image retrieval, the
following modules to be introduced can further help capture the structural
correspondence and improve the generalization ability (see experiments in
Section4.3 and 4.5).

After enforcing the structure features of sketches and images to the same
structure space, we further expect that the appearance features of images only
contain complementary information (_e.g._ , color, texture, and background) to
the structure features. To reinforce the disentanglement of image features, we
impose an orthogonal constraint between structure and appearance features of
images based on cosine similarity (Shukla et al. 2019):

$\mathcal{L}_{or}=\cos(\mathbf{f}_{im}^{ap},\mathbf{f}_{im}^{st})=\frac{\mathbf{f}_{im}^{ap}\cdot\mathbf{f}_{im}^{st}}{||\mathbf{f}_{im}^{ap}||_{2}||\mathbf{f}_{im}^{st}||_{2}},$
(3)

where $\cdot$ means the the dot product between two vectors. Note that
$\mathbf{f}_{im}^{ap}$ and $\mathbf{f}_{im}^{st}$ are the output of ReLU
activation, so $\cos(\mathbf{f}_{im}^{ap},\mathbf{f}_{im}^{st})$ is always
non-negative and minimizing Equ. (3) will push
$\cos(\mathbf{f}_{im}^{ap},\mathbf{f}_{im}^{st})$ towards zero.

#### Bi-directional Domain Translation

To learn better disentangled representations and fully utilize the
disentangled image features, we perform bi-directional domain translation
between the sketch domain and the image domain.

For image-to-sketch translation, we aim to translate image feature
$\mathbf{f}_{im}$ to sketch feature through its image structure feature
$\mathbf{f}_{im}^{st}$. We employ a decoder $G_{sk}$ to generate sketch
feature $\hat{\mathbf{f}}_{sk}$ from $\mathbf{f}_{im}^{st}$ by
$\hat{\mathbf{f}}_{sk}=G_{sk}(\mathbf{f}_{im}^{st})$. Considering that
$\hat{\mathbf{f}}_{sk}$ and $\mathbf{f}_{sk}$ belong to the same category, we
enforce the generated sketch features to be close to the real sketch features
from the same category by using $L_{2}$ loss:

$\mathcal{L}_{2}^{sk}=||\mathbf{f}_{sk}-\hat{\mathbf{f}}_{sk}||_{2}.$ (4)

For sketch-to-image translation, we aim to translate sketch feature
$\mathbf{f}_{st}$ to image feature through its sketch structure feature
$\mathbf{f}_{sk}^{st}$. However, images contain extra appearance information
(_e.g._ , color, texture, and background) compared with sketches, so it is
necessary to compensate for the appearance uncertainty when translating from
structure features to image features. Therefore, in the training stage,
appearance feature $\mathbf{f}_{im}^{ap}$ could be integrated with sketch
structure feature $\mathbf{f}_{sk}^{st}$ to generate the image feature.

During testing, given a sketch, we also hope to generate its image feature to
enable retrieval in the image space. Nevertheless, we do not have the
corresponding appearance feature. A common solution is stochastic sampling. We
introduce a variational estimator $V_{im}^{ap}$ to approximate the variational
Gaussian distribution $P(\mathbf{z}_{im}^{ap}|\mathbf{f}_{im}^{ap})$ based on
$\mathbf{f}_{im}^{ap}$, that is,
$(\bm{\mu}_{im}^{ap},\bm{\sigma}_{im}^{ap})=V_{im}^{ap}(\mathbf{f}_{im}^{ap})$.
Then, we use Kullback-Leibler divergence to enforce
$P(\mathbf{z}_{im}^{ap}|\mathbf{f}_{im}^{ap})$ to be close to prior
distribution $\mathcal{N}(\mathbf{0},\mathbf{1})$ to support stochastic
sampling:

$\mathcal{L}_{KL}=KL(\mathcal{N}(\bm{\mu}_{im}^{ap},\bm{\sigma}_{im}^{ap})||\mathcal{N}(\mathbf{0},\mathbf{1})).$
(5)

After using reparameterization trick (Kingma and Welling 2014) to sample
variational appearance feature $\mathbf{z}_{im}^{ap}$, _i.e._ ,
$\mathbf{z}_{im}^{ap}=\bm{\mu}_{im}^{ap}+\epsilon\bm{\sigma}_{im}^{ap}$, where
$\epsilon$ is sampled from $\mathcal{N}(0,1)$, we employ a decoder $G_{im}$ to
generate
$\hat{\mathbf{f}}_{im}=G_{im}([\mathbf{z}_{im}^{ap},\mathbf{f}_{sk}^{st}])$
based on $\mathbf{z}_{im}^{ap}$ and $\mathbf{f}_{sk}^{st}$, where
$[\cdot,\cdot]$ means concatenating two vectors. Considering that
$\hat{\mathbf{f}}_{im}$ has the same category label as $\mathbf{f}_{im}$ and
its appearance uncertainty comes from $\mathbf{f}_{im}$, we enforce
$\hat{\mathbf{f}}_{im}$ to be close to $\mathbf{f}_{im}$ with $L_{2}$ loss:

$\mathcal{L}_{2}^{im}=||\mathbf{f}_{im}-\hat{\mathbf{f}}_{im}||_{2}.$ (6)

By performing image-to-sketch translation, we expect that the image structure
features contain the necessary structure information to generate sketch
features. By performing sketch-to-image translation, we expect that the
appearance features contain the necessary information of appearance
uncertainty to compliment the sketch structure features when generating image
features. Therefore, bi-directional domain translation could cooperate with
classification loss and orthogonal loss to assist feature disentanglement.

Finally, the full objective function can be expressed as

$\mathcal{L}\\!=\\!\mathcal{L}_{or}\\!+\\!\mathcal{L}_{cls}\\!+\\!\mathcal{L}_{KL}\\!+\\!\mathcal{L}_{2}^{im}\\!+\\!\mathcal{L}_{2}^{sk}.$
(7)

###  3.3 Retrieval Strategy

In the test stage, we perform retrieval in all three spaces: structure,
sketch, and image spaces. Specifically, given a sketch $\mathbf{x}_{sk}$ with
sketch feature $\mathbf{f}_{sk}$ and an image $\mathbf{x}_{im}$ with image
feature $\mathbf{f}_{im}$ , we compare them in three spaces to combine the
best of all worlds.

(1) Structure space: We project $\mathbf{f}_{im}$ and $\mathbf{f}_{sk}$ into
the common structure space by
$\mathbf{f}_{im}^{st}=E_{im}^{st}(\mathbf{f}_{im})$ and
$\mathbf{f}_{sk}^{st}=E_{sk}^{st}(\mathbf{f}_{sk})$ respectively, so we can
calculate the distance in structure space
$\mathcal{D}_{st}=1-cos(\mathbf{f}_{im}^{st},\mathbf{f}_{sk}^{st})$.

(2) Sketch space: For image $\mathbf{x}_{im}$, based on its image structure
feature $\mathbf{f}_{im}^{st}$, we employ the sketch decoder $G_{sk}$ to
generate its sketch feature
$\hat{\mathbf{f}}_{sk}=G_{sk}(\mathbf{f}_{im}^{st})$, so we can calculate the
distance in sketch space
$\mathcal{D}_{sk}=1-cos(\hat{\mathbf{f}}_{sk},\mathbf{f}_{sk})$.

(3) Image space: For sketch $\mathbf{x}_{sk}$, based on its sketch structure
feature $\mathbf{f}_{sk}^{st}$ and a variational appearance feature
$\mathbf{z}_{i}$ sampled from $\mathcal{N}(\mathbf{0},\mathbf{1})$, we employ
the image decoder $G_{im}$ to generate its image feature. We can generate $N$
image features vectors by sampling $N$ times and use the average to represent
the final image feature $\hat{\mathbf{f}}_{im}$:

$\hat{\mathbf{f}}_{im}=\frac{1}{N}\sum_{i=1}^{N}G_{im}([\mathbf{z}_{i},\mathbf{f}_{sk}^{st}]).$
(8)

So we can calculate the distance in image space
$\mathcal{D}_{im}=1-cos(\mathbf{f}_{im},\hat{\mathbf{f}}_{im})$. Finally, we
calculate the weighted average of three distances for the best retrieval:

$\displaystyle\mathcal{D}_{fusion}=\lambda_{1}(\mathcal{D}_{im}+\mathcal{D}_{sk})+\lambda_{2}\mathcal{D}_{st},$
(9)

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters to balance
different spaces. Since sketch feature is generated solely based on structure
feature, retrieval in sketch space is obviously structure-aware. Image feature
is generated based on structure feature and variational appearance feature
$\mathbf{z}_{i}$, but $\mathbf{z}_{i}$ is sampled from
$\mathcal{N}(\mathbf{0},\mathbf{1})$ for any category and thus category-
agnostic. Therefore, retrieval in image space mainly depends on the structure
information and is also structure-aware.

| Method | Sketchy Ext. | TU-Berlin Ext. | QuickDraw Ext.
---|---|---|---|---
P@200(%) | mAP@200(%) | P@200(%) | mAP@200(%) | P@200(%) | mAP@200(%)
SBIR | SaN (Yu et al. 2017) | 15.3$\pm$0.6 | 5.8$\pm$0.4 | 10.1$\pm$0.6 | 4.2$\pm$0.4 | 4.2$\pm$0.5 | 0.9$\pm$0.2
Siamese (Qi et al. 2016) | 25.6$\pm$0.6 | 15.3$\pm$0.5 | 8.3$\pm$0.4 | 3.7$\pm$0.3 | 4.0$\pm$0.3 | 0.7$\pm$0.2
ZSL | ESZSL (Romera-Paredes and Torr 2015) | 20.9 | 11.8 | 8.5 | 3.4 | 6.3 | 1.8
SAE (Kodirov, Xiang, and Gong 2017) | 26.1 | 14.5 | 10.4 | 4.6 | 6.8 | 1.9
CMT (Socher et al. 2013) | 27.3$\pm$0.7 | 15.8$\pm$0.5 | 10.8$\pm$0.5 | 4.9$\pm$0.4 | 6.3$\pm$0.4 | 1.7$\pm$0.3
DeViSE (Frome et al. 2013) | 18.2$\pm$0.5 | 7.2$\pm$0.4 | 7.3$\pm$0.2 | 2.2$\pm$0.2 | 6.7$\pm$0.2 | 2.6$\pm$0.1
ZS-SBIR (D) | Simple DB | 32.1$\pm$1.3 | 19.0$\pm$0.9 | 14.6$\pm$1.2 | 6.8$\pm$0.8 | 7.1$\pm$0.6 | 3.1$\pm$0.5
CVAE (Yelamarthi et al. 2018) | 39.3$\pm$0.9 | 26.3$\pm$0.8 | 15.2$\pm$0.7 | 7.5$\pm$0.5 | 6.4$\pm$0.4 | 1.8$\pm$0.3
Xu et al. (2019) | 42.8$\pm$1.1 | 31.1$\pm$0.8 | 17.1$\pm$0.7 | 9.5$\pm$0.5 | 7.2$\pm$0.6 | 3.0$\pm$0.3
SEM-PCYC (Dutta and Akata 2019) | 43.8$\pm$0.7 | 31.6$\pm$0.5 | 19.5$\pm$0.8 | 10.7$\pm$0.4 | 9.4$\pm$0.4 | 3.7$\pm$0.2
Doodle (Dey et al. 2019) | 43.2$\pm$0.8 | 30.1$\pm$0.5 | 19.3$\pm$1.1 | 10.2$\pm$0.7 | 9.8$\pm$0.6 | 3.7$\pm$0.4
Dutta and Biswas (2019) | 43.0$\pm$0.7 | 28.5$\pm$0.4 | 17.8$\pm$0.5 | 10.8$\pm$0.3 | 8.3$\pm$0.3 | 3.5$\pm$0.1
SketchGCN (Zhang et al. 2020) | 45.1$\pm$1.3 | 34.8$\pm$0.9 | 21.3$\pm$0.7 | 12.8$\pm$0.5 | 9.7$\pm$0.6 | 4.0$\pm$0.4
STRAD |  50.2$\pm$0.7 |  37.9$\pm$0.4 |  24.5$\pm$0.9 |  15.4$\pm$0.6 |  12.6$\pm$0.5 |  5.4$\pm$0.4
ZS-SBIR (S) | Simple SB | 49.7$\pm$1.9 | 36.7$\pm$1.5 | 26.2$\pm$1.6 | 16.2$\pm$1.1 | 16.6$\pm$1.3 | 7.1$\pm$0.8
SAKE (Liu et al. 2019) | 51.9$\pm$0.7 | 39.4$\pm$0.5 | 28.7$\pm$0.6 | 18.1$\pm$0.4 | 18.4$\pm$0.5 | 8.4$\pm$0.3
STRAD |  53.7$\pm$0.8 |  41.3$\pm$0.6 |  30.1$\pm$0.7 |  19.9$\pm$0.4 |  19.1$\pm$0.4 |  8.6$\pm$0.2

Table 1: Comparison of our STRAD method and baselines on Sketchy, TU-Berlin,
and QuickDraw datasets. (D) is short for “double backbone” setting and (S) is
short for “single backbone” setting. Best results are denoted in boldface in
both settings.

##  4 Experiment

###  4.1 Dataset

We evaluate our STRAD method and all the other baselines on three large-scale
benchmark datasets: Sketchy (Sangkloy et al. 2016), TU-Berlin (Eitz, Hays, and
Alexa 2012), and QuickDraw (Dey et al. 2019). Following the same setting in
Shen et al. (2018), Sketchy and TU-Berlin are extended with images obtained
from Liu et al. (2017). Due to the limitation of space, the details of
datasets, seen/unseen category split, and implementation are left to
Supplementary Material.

###  4.2 Comparison with Existing Methods

We compare our model with 13 prior methods, which can be categorized into
three categories: sketch-based image retrieval (SBIR), zero-shot learning
(ZSL), and zero-shot SBIR (ZS-SBIR). The SBIR baselines include Siamese (Qi et
al. 2016), and SaN (Yu et al. 2017). The ZSL baselines include ESZSL (Romera-
Paredes and Torr 2015), SAE (Kodirov, Xiang, and Gong 2017), CMT (Socher et
al. 2013), and DeViSE (Frome et al. 2013). For a fair comparison, we use fine-
tuned VGG-16 as backbone for all methods except SaN, which specifically
designs a backbone for SBIR. Among all the previous works on ZS-SBIR, either
two different backbones or a single backbone can be used to extract image and
sketch features. In this paper, we conduct experiments in both settings. For
the “double backbone” setting, we compare with CVAE (Yelamarthi et al. 2018),
Xu et al. (2019), SEM-PCYC (Dutta and Akata 2019), Doodle (Dey et al. 2019),
Dutta and Biswas (2019), and SketchGCN (Zhang et al. 2020). For the “single
backbone” setting, we compare with SAKE (Liu et al. 2019) because they use
single backbone in their paper. During training, we fix the backbone for all
methods except Dey et al. (2019) and SAKE (Liu et al. 2019). For Dey et al.
(2019), we strictly follow their paper to train the backbone during the
training. For SAKE (Liu et al. 2019), we strictly follow the training strategy
introduced in their paper 111Since SAKE (Liu et al. 2019) starts from backbone
model pre-trained on ImageNet-1k to prevent knowledge forgetting, we do not
change their setting.. Following Yelamarthi et al. (2018), we use mean average
precision and precision of top 200 retrievals (mAP@200 and P@200) as the
evaluation metrics. Due to the calculation of mAP@200 in (Dey et al. 2019;
Zhang et al. 2020) is slightly different from ours, their reported mAP@200 are
different from our reproduced results. We also report the performance of fine-
tuned single backbone (_resp._ , double backbones) as “Simple SB” (_resp._ ,
“Double DB”) in Table 1, where cosine distance between the image and sketch
features are used for retrieval. We run each model for 10 times with random
seeds 1 - 10 and report the “Mean $\pm$ Std” results in Table 1, except for
ESZSL and SAE, which do not introduce randomness

Based on Table 1, we find all SBIR and ZSL baselines underperform the ZS-SBIR
baselines due to their poor generalization ability from seen categories to
unseen ones. On the TU-Berlin dataset, the results of several methods (Liu et
al. 2019; Dutta and Akata 2019; Dutta and Biswas 2019; Zhang et al. 2020) are
worse than those reported in their papers due to different seen/unseen
category splits. In particular, the number of unseen categories under our
split is twice larger than that in (Liu et al. 2019; Zhang et al. 2020). Our
split criterion also prevents information leakage from ImageNet-1k to unseen
categories. The overall results on TU-Berlin are lower than those on Sketchy
due to larger number of unseen categories (64 _v.s._ 21). Furthermore, the
overall results on TU-Berlin are lower than those on QuickDraw since sketches
of QuickDraw were drawn by amateurs.

In “double backbone” setting, our proposed STRAD excels the state-of-the-art
methods by 5.1% on Sketchy, 3.3% on TU-Berlin, and 2.8% on QuickDraw in terms
of P@200. In “single backbone” setting, we find that the “Simple SB”
outperforms most methods in “double backbone” setting, which reveals that it
might be the best solution for SBIR task to use a single model to pull close
the image and sketch space. Starting from “Simple SB”, the performance gain of
our STRAD is smaller than that in “double backbone” setting, because a single
backbone has already filtered out most differences between features of images
and sketches. However, there still remains extra appearance information in
image features, so our STRAD also outperforms SAKE (Liu et al. 2019) and
achieves the best results on all datasets.

To further demonstrate the effectiveness of our method, we also report the
results on TU-Berlin following the splits in Liu et al. (2019) and the results
in generalized ZS-SBIR setting (Dutta and Akata 2019) in Supplementary
Material.

Figure 3: The t-SNE visualization of seven types of features on Sketchy test
set. Best viewed in color. Figure 4: Top-5 images retrieved by STRAD (double
backbone), STRAD (single backbone), SAKE(Liu et al. 2019), and Dutta and
Biswas (2019) methods on Sketchy. The green (_resp._ , red) border indicates
the correct (_resp._ , incorrect) retrieval results.

###  4.3 Ablation Study

By taking the Sketchy as an example, we analyze the effectiveness of different
loss terms in “double backbone” setting. We ablate each loss term in Equ. (2),
(3), (4), (5), and (6), and report the results in Table 2. The random seed in
this subsection is set as 1.

| $\mathcal{L}_{KL}$ | $\mathcal{L}_{2}^{sk}$ | $\mathcal{L}_{2}^{im}$ | $\mathcal{L}_{cls}$ | $\mathcal{L}_{or}$ | Comb | Sk | Im | St
---|---|---|---|---|---|---|---|---|---
1 | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | 50.2 | 46.9 | 43.8 | 47.1
2 | $\times$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | 48.7 | 45.3 | 36.8 | 47.0
3 | $\surd$ | $\times$ | $\surd$ | $\surd$ | $\surd$ | 47.2 | - | 42.7 | 46.1
4 | $\times$ | $\surd$ | $\times$ | $\surd$ | $\surd$ | 48.4 | 46.5 | - | 45.9
5 | $\surd$ | $\surd$ | $\surd$ | $\times$ | $\surd$ | 44.2 | 42.8 | 39.5 | -
6 | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\times$ | 49.4 | 46.8 | 43.2 | 46.9
7 | $\times$ | $\times$ | $\times$ | $\surd$ | $\times$ | 41.9 | - | - | 41.9

Table 2: The ablation studies results (P@200(%)) of our STRAD method on
Sketchy. Comb denotes the retrieval in the combination of all spaces (see Sec.
3.3); Sk, Im, and St denotes the retrieval in sketch, image, and structure
space individually. $\surd$ (_resp._ , $\times$) means adding (_resp._ ,
removing) this loss during training, and $-$ means unavailable entries.

We observe that the performance in both image and sketch spaces drops
significantly without the classification loss (“Im” in row 1 _v.s._ “Im” in
row 5, and “Sk” in row 1 _v.s._ “Sk” in row 5), which shows the significant
contribution of classification loss to feature disentanglement and alignment
in all three spaces. In the meanwhile, only using classification loss leads to
a performance drop in structure space (“St” in row 1 _v.s._ “St” in row 7).
Although the classification loss can pull close the sketch and image spaces,
there is no guarantee that the structural correspondence between sketches and
images is well captured. In contrast, asymmetric disentanglement and bi-
directional domain translation in our full-fledged method can help capture the
structural correspondence and generalize to unseen categories. By comparing
row 1 and row 3, we find that $\mathcal{L}_{2}^{sk}$ is also very important,
because sketch features without involving appearance uncertainty are easier to
be regularized than image features, which makes sketch space more crucial for
the combination result. Another observation is that the orthogonal loss which
reinforces disentanglement does benefit the retrieval task (row 1 _v.s._ row
6).

###  4.4 Feature Space Analysis

To demonstrate the ability of our model to disentangle the image features, in
Figure 3, we visualize 7 types of features from 10 unseen categories on
Sketchy using t-SNE (Maaten and Hinton 2008), including image appearance
feature, image structure feature, sketch structure feature, image feature,
generated image feature, sketch feature, and generated sketch feature.
According to Figure 3, we have the following observations: 1) Different
categories can be separated well in “image structure” and “sketch structure”,
which significantly facilitates the retrieval in structure space; 2) The
results in “image structure” and “image appearance” are complementary, in
accordance with the disentanglement between structure and appearance features;
3) The results in “sketch features” and “generated sketch features” are
similar, compared with those in “image features” and “generated image
features”, which implies that the generated sketch features could be closer to
the real sketch features from the same category; 4) Compared results in
“generated sketch feature”, “sketch structure” and “image structure”, the
results in “generated image feature” show relatively poor separability, which
limits retrieval performance in image space.

Figure 5: Top-5 images retrieved in combined, image, sketch, and structure
space under double backbone setting. The green (_resp._ , red) border
indicates the correct (_resp._ , incorrect) retrieval results.

###  4.5 Case Study

In Figure 4, we show top-5 retrieval results of STRAD (double backbone), STRAD
(single backbone), SAKE (Liu et al. 2019), and Dutta and Biswas (2019), based
on which we have the following observations. 1) Our STRAD is adept at
capturing the correspondence between the retrieved images and the given sketch
_w.r.t._ both local structure information (_e.g._ , door-case) and global
structure information (_e.g._ , global grid structure of door). Besides, in
both “single backbone” and “double backbone” settings, our STRAD is able to
retrieve the images with cluttered background (_e.g._ , a bird with intricate
leaves/flowers behind), which benefits from our structure-aware retrieval in
three spaces. The above advantages come from the combination of three
retrieval spaces. Detailed analyses and examples of complementarity of three
retrieval spaces are provided in Supplementary. 2) For baselines, SAKE (Liu et
al. 2019) with single backbone can retrieve images with cluttered background
while Dutta and Biswas (2019) with double backbones can barely tolerate the
complex backgrounds. One possible reason is that using the same backbone for
the sketch domain and image domain is to share a large amount of parameters
for feature extraction, which would eliminate the background differences
between these two domains.

In Figure 5, we show top-5 images retrieved in combined, image, sketch, and
structure space by our STRAD (double backbone). It can be seen that image
space tends to retrieve images with cluttered background while sketch and
structure spaces are prone to retrieve images with clean background, which
proves that image space contains extra appearance information (_e.g._ ,
background) complemented by variational appearance feature. More examples and
analyses of three retrieval spaces can be found in Supplementary Materials.

##  5 Conclusion

In this work, we have studied the problem zero-shot sketch-based image
retrieval (ZS-SBIR) from a new viewpoint, _i.e._ , using asymmetric
disentangled representation to facilitate structure-aware retrieval. We have
proposed our STRucture-aware Asymmetric Disentanglement (STRAD) model, with
retrieval performed in combination of three complementary spaces.
Comprehensive experiments on three large-scale benchmark datasets have
demonstrated the generalization ability of our model from seen categories to
unseen ones.

## References

  * Chen and Fang (2018) Chen, J.; and Fang, Y. 2018.  Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 605–620.
  * Chen et al. (2018) Chen, R. T.; Li, X.; Grosse, R. B.; and Duvenaud, D. K. 2018.  Isolating sources of disentanglement in variational autoencoders.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 2610–2620.
  * Chen et al. (2016) Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; and Abbeel, P. 2016\.  InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets.  In _Proceedings of the Advances in neural information processing systems (NeurIPS)_ , 2172–2180.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  ImageNet: A large-scale hierarchical image database.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 248–255.
  * Dey et al. (2019) Dey, S.; Riba, P.; Dutta, A.; Llados, J.; and Song, Y.-Z. 2019.  Doodle to search: Practical zero-shot sketch-based image retrieval.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2179–2188.
  * Dupont (2018) Dupont, E. 2018.  Learning disentangled joint continuous and discrete representations.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 710–720.
  * Dutta and Akata (2019) Dutta, A.; and Akata, Z. 2019.  Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 5089–5098.
  * Dutta and Biswas (2019) Dutta, T.; and Biswas, S. 2019.  Style-guided zero-shot sketch-based image retrieval.  In _Proceedings of the British Machine Vision Conference (BMVC)_ , 209.
  * Eitz, Hays, and Alexa (2012) Eitz, M.; Hays, J.; and Alexa, M. 2012.  How do humans sketch objects?  _ACM Transactions on graphics (TOG)_ 31(4): 1–10.
  * Eitz et al. (2010) Eitz, M.; Hildebrand, K.; Boubekeur, T.; and Alexa, M. 2010.  An evaluation of descriptors for large-scale image retrieval from sketched feature lines.  _Computers & Graphics_ 34(5): 482–498.
  * Fellbaum (2012) Fellbaum, C. 2012.  WordNet.  _The Encyclopedia of Applied Linguistics_ .
  * Frome et al. (2013) Frome, A.; Corrado, G. S.; Shlens, J.; Bengio, S.; Dean, J.; Ranzato, M.; and Mikolov, T. 2013.  DeVISE: A deep visual-semantic embedding model.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 2121–2129.
  * Hadad, Wolf, and Shahar (2018) Hadad, N.; Wolf, L.; and Shahar, M. 2018.  A two-step disentanglement method.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 772–780.
  * Harsh Jha et al. (2018) Harsh Jha, A.; Anand, S.; Singh, M.; and Veeravasarapu, V. 2018.  Disentangling factors of variation with cycle-consistent variational auto-encoders.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 805–820.
  * Higgins et al. (2017) Higgins, I.; Matthey, L.; Pal, A.; Burgess, C.; Glorot, X.; Botvinick, M.; Mohamed, S.; and Lerchner, A. 2017.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Huang et al. (2018) Huang, X.; Liu, M.-Y.; Belongie, S.; and Kautz, J. 2018.  Multimodal unsupervised image-to-image translation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 172–189.
  * Kim et al. (2017) Kim, T.; Cha, M.; Kim, H.; Lee, J. K.; and Kim, J. 2017.  Learning to discover cross-domain relations with generative adversarial networks.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 1857–1865.
  * Kingma et al. (2014) Kingma, D. P.; Mohamed, S.; Rezende, D. J.; and Welling, M. 2014.  Semi-supervised learning with deep generative models.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 3581–3589.
  * Kingma and Welling (2014) Kingma, D. P.; and Welling, M. 2014.  Auto-encoding variational bayes.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Kodirov, Xiang, and Gong (2017) Kodirov, E.; Xiang, T.; and Gong, S. 2017.  Semantic autoencoder for zero-shot learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3174–3183.
  * Lee et al. (2018) Lee, H.-Y.; Tseng, H.-Y.; Huang, J.-B.; Singh, M.; and Yang, M.-H. 2018.  Diverse image-to-image translation via disentangled representations.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 35–51.
  * Li et al. (2016) Li, K.; Pang, K.; Song, Y.-Z.; Hospedales, T.; Zhang, H.; and Hu, Y. 2016.  Fine-grained sketch-based image retrieval: The role of part-aware attributes.  In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 1–9.
  * Liu et al. (2017) Liu, L.; Shen, F.; Shen, Y.; Liu, X.; and Shao, L. 2017.  Deep sketch hashing: Fast free-hand sketch-based image retrieval.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2862–2871.
  * Liu et al. (2019) Liu, Q.; Xie, L.; Wang, H.; and Yuille, A. L. 2019.  Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 3662–3671.
  * Liu et al. (2018) Liu, Y.; Wei, F.; Shao, J.; Sheng, L.; Yan, J.; and Wang, X. 2018.  Exploring disentangled feature representation beyond face identification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2080–2089.
  * Lu et al. (2018) Lu, P.; Huang, G.; Fu, Y.; Guo, G.; and Lin, H. 2018.  Learning large euclidean margin for sketch-based image retrieval.  _arXiv preprint arXiv:1812.04275_ .
  * Maaten and Hinton (2008) Maaten, L. v. d.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9: 2579–2605.
  * Mikolov et al. (2013) Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 3111–3119.
  * Qi et al. (2016) Qi, Y.; Song, Y.-Z.; Zhang, H.; and Liu, J. 2016.  Sketch-based image retrieval via siamese convolutional neural network.  In _Proceedings of the IEEE International Conference on Image Processing (ICIP)_ , 2460–2464.
  * Romera-Paredes and Torr (2015) Romera-Paredes, B.; and Torr, P. 2015.  An embarrassingly simple approach to zero-shot learning.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2152–2161.
  * Saavedra (2014) Saavedra, J. M. 2014.  Sketch based image retrieval using a soft computation of the histogram of edge local orientations (S-HELO).  In _Proceedings of the IEEE International Conference on Image Processing (ICIP)_ , 2998–3002. IEEE.
  * Saavedra, Barrios, and Orand (2015) Saavedra, J. M.; Barrios, J. M.; and Orand, S. 2015.  Sketch based image retrieval using learned keyshapes (LKS).  In _Proceedings of the British Machine Vision Conference (BMVC)_ , 7.
  * Sangkloy et al. (2016) Sangkloy, P.; Burnell, N.; Ham, C.; and Hays, J. 2016.  The sketchy database: Learning to retrieve badly drawn bunnies.  _ACM Transactions on Graphics (TOG)_ 35(4): 119.
  * Shen et al. (2018) Shen, Y.; Liu, L.; Shen, F.; and Shao, L. 2018.  Zero-shot sketch-image hashing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3598–3607.
  * Shukla et al. (2019) Shukla, A.; Bhagat, S.; Uppal, S.; Anand, S.; and Turaga, P. 2019.  Product of orthogonal spheres parameterization for disentangled representation learning.  In _Proceedings of the British Machine Vision Conference (BMVC)_.
  * Simonyan and Zisserman (2015) Simonyan, K.; and Zisserman, A. 2015.  Very deep convolutional networks for large-scale image recognition.  In _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Socher et al. (2013) Socher, R.; Ganjoo, M.; Manning, C. D.; and Ng, A. 2013.  Zero-shot learning through cross-modal transfer.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , 935–943.
  * Verma et al. (2019) Verma, V. K.; Mishra, A.; Mishra, A.; and Rai, P. 2019.  Generative model for zero-shot sketch-based image retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 704–713.
  * Xu et al. (2019) Xu, X.; Wang, H.; Li, L.; and Deng, C. 2019.  Semantic adversarial network for zero-shot sketch-based image retrieval.  _arXiv preprint arXiv:1905.02327_ .
  * Yang et al. (2019) Yang, S.; Liu, J.; Wang, W.; and Guo, Z. 2019.  TET-GAN: Text effects transfer via stylization and destylization.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 1238–1245.
  * Yelamarthi et al. (2018) Yelamarthi, S. K.; Reddy, S. K.; Mishra, A.; and Mittal, A. 2018.  A zero-shot framework for sketch based image retrieval.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 316–333.
  * Yu et al. (2016) Yu, Q.; Liu, F.; Song, Y.-Z.; Xiang, T.; Hospedales, T. M.; and Loy, C.-C. 2016\.  Sketch me that shoe.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 799–807.
  * Yu et al. (2017) Yu, Q.; Yang, Y.; Liu, F.; Song, Y.-Z.; Xiang, T.; and Hospedales, T. M. 2017.  Sketch-a-net: A deep neural network that beats humans.  _International Journal of Computer Vision (IJCV)_ 122(3): 411–425.
  * Zhang et al. (2020) Zhang, Z.; Zhang, Y.; Feng, R.; Zhang, T.; and Fan, W. 2020.  Zero-shot sketch-based image retrieval via graph convolution network.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 12943–12950.
  * Zheng et al. (2019) Zheng, Z.; Yang, X.; Yu, Z.; Zheng, L.; Yang, Y.; and Kautz, J. 2019.  Joint discriminative and generative learning for person re-identification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2138–2147.
  * Zhu, Xie, and Fang (2016) Zhu, F.; Xie, J.; and Fang, Y. 2016.  Learning cross-domain neural networks for sketch-based 3D shape retrieval.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_.
  * Zhu et al. (2017) Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2223–2232.