# Some classes of subsemimodule spaces

A. Goswami  [1] Department of Mathematics and Applied Mathematics, University
of Johannesburg, P.O. Box 524, Auckland Park 2006, South Africa. [2] National
Institute for Theoretical and Computational Sciences (NITheCS), South Africa.
<EMAIL_ADDRESS>
###### Abstract.

The aim of this paper is to study the topological properties of some classes
of subsemimodules endowed with a subbasis closed-set topology. We show that
such spaces are $T_{0}$. When the semimodule is finitely generated, those
spaces are compact as well. We characterize subsemimodule spaces for which
every nonempty irreducible closed set has a unique generic point. We give a
sufficient condition for a connected subsemimodule space, and using the notion
of strongly disconnectedness, we determine compact disconnected subsemimodule
spaces. Finally, we discuss continuous maps between subsemimodule spaces.

###### Key words and phrases:

semiring, semimodule, subtractive-, extraordinary-, prime-, primal-, strongly
irreducible-subsemimodules, closed subbasis topology, generic point,
connectedness.

###### 1991 Mathematics Subject Classification:

16Y60; 13C13; 16D80; 54F65

##  1\. Introduction

Since the introduction of semirings in [18], it is natural to compare and
extend results from rings to semirings. One may think that semirings can
always to be extended to rings, but [19] gave examples of semirings which can
not be embedded in rings. Moreover, the lack of additive inverses in semirings
makes the semiring theory considerably different than that of rings.

As modules appear naturally in the study of rings, the notion of semimodules
also inevitable in structural understanding of semirings. Some intensive study
of structure of semimodules over a semiring can be found in [10, 1, 16, 6,
11]. Compare to this, topological study on various classes of subsemimodules
of a semimodule is less investigated. A Zariski topology on prime
$k$-subsemimodules of a very strong multiplication semimodule has been
considered in [2]. For a multiplication and finitely generated semimodule, in
[13] (see also [14]), it has been proved that the class of prime
subsemimodules endowed with Zariski topology is a spectral space.

The aim of this paper is twofold. On the one hand, our work generalizes the
topological results in [2, 13, 14] to various classes of subsemimodule spaces;
on the other hand, it extends some of the results from [7] on ideal spaces
((i.e., a class of ideal of a commutative ring endowed with closed subbasis
topology) to the semimodule context.

##  2\. Preliminaries

Let us recall some elementary definitions and facts about semimodules. For a
detailed study of semimodules over a semiring, we refer our readers to [10]. a
_semiring_ is a system $(R,+,0,\cdot,1)$ such that $(R,+,0)$ is a commutative
monoid, $(R,\cdot,1)$ is a monoid, $0r=0=r0$ for all $r\in R,$ and $\cdot$
distributes over $+$ both from the left and from the right sides. In this
paper all semirings are assumed to be commutative, i.e., $(R,\cdot,1)$ is a
commutative monoid. If $R$ is semiring, then a _(left) $R$-semimodule_ is an
additive commutative monoid $M$ together with a map (called _scalar
multiplication_) $R\times M\to M$, denoted by $(r,m)\mapsto rm$, such that (i)
$(r+r^{\prime})m=rm+r^{\prime}m,$ (ii) $r(m+m^{\prime})=rm+rm^{\prime},$ (iii)
$(rr^{\prime})m=r(r^{\prime}m),$ (iv) $1m=m$, and (v) $r0=0=0m$ for all $r,$
$r^{\prime}\in R$ and $m,m^{\prime}\in M.$ Note that we have used $0$ to
denote both the additive identity of the semiring $R$ and that of the monoid
$M$.

A nonempty subset $N$ of a left $R$-semimodule $M$ is called a _subsemimodule_
of $M$ if $N$ is closed under addition and scalar multiplication. We now
define some distinguished classes of subsemimodules of a semimodule $M$. Some
of these classes of subsemimodules are well-known and we provide reference(s),
whereas, to best of our knowledge, others are introduced here for the first
time.

A subsemimodule $N$ is called _subtractive_ or _k-subsemimodule_ [10] if
$x+y\in N$ and $x\in N$ implies $y\in N;$ whereas $N$ is called _strong_ [10]
if $x+y\in$ implies $x,$ $y\in N$ for all $x,y\in M$. A _proper_ subsemimodule
$N$ is such that $N\neq M.$ A proper subsemimodule $N$ is called _maximal_
[20] if there is no other proper subsemimodule containing $N$. A proper
subsemimodule $N$ is called _prime_ [20, 21] (_primary_) [3, 17] if $rm\in N$
with $r\in R$ and $m\in M$ implies that $m\in N$ or $r\in(N:M)=\\{r\in R\mid
rM\subseteq N\\}$ ($r^{n}\in(N:M)=\\{r\in R\mid rM\subseteq N\\}$ for some
$n\in\mathds{N}$). A proper subsemimodule $N$ is called a _weakly prime_ [4]
subsemimodule if whenever $0\neq rm\in N,$ for some $r\in R$ and $m\in M,$
then either $m\in N$ or $rM\subseteq N.$ An element $r$ of $R$ is called
_prime to $N$_ if $rm\in N$ with $r\in R$ and $m\in M$ implies that $m\in N$.
Denote the set of all prime to $N$ elements of $R$ by $\mathrm{Prm}(N).$ A
proper subsemimodule $N$ is called _primal_ [4] if $R\setminus\mathrm{Prm}(N)$
is an ideal of $R$. A _semiprime_ subsemimodule $N$ is the intersection of
prime semimodules of $M$. For an alternative notion of a semiprime semimodule,
see [17]. The _radical_ [3] $\sqrt{N}$ of a subsemimodule $N$ is the
intersection fo all prime subsemimodules of $M$ containing $N$. A prime
subsemimodule $N$ is called _extraordinary_ [13] if for any semiprime
subsemimodules $L$ and $K$ of $M,$ $L\cap K\subseteq N$ implies that
$L\subseteq N$ or $K\subseteq N.$

A subsemimodule $N$ is called _strongly irreducible_ if for any two
subsemimodules $L$ and $K$ of $M$, $L\cap K\subseteq N$ implies $L\subseteq N$
or $K\subseteq N.$ A subsemimodule $N$ that is not the intersection of two
(any set of) subsemimodules of $M$ in which $N$ is properly contained is
called _irreducible_ (_completely irreducible_). If a subsemimodule $N$ is
generated by a single element (a finite set of elements) of $M$, then $N$ is
called _cyclic_ (_finitely generated_). A nonzero subsemimodule $N$ is called
a _minimal_ subsemimodule if $N$ contains no other nonzero subsemimodules of
$M.$ A _minimal prime_ subsemimodule $N$ is both a minimal and a prime
subsemimodule. To denote any one of the above-mentioned distinguished class of
subsemimodules of $M$, we use the notation $\mathrm{Dis}_{M}$ and we assume
that $M\notin\mathrm{Dis}_{M}$; whereas, $\mathrm{Sub}_{M}$ denotes the set of
all subsemimodules of $M$.

##  3\. Subsemimodule spaces

Let $M$ be an $R$-semimodule. For any subset $S$ of $M$ and for any
$\mathrm{Dis}_{M}$, define

$\mathcal{V}(S)=\\{L\in\mathrm{Dis}_{M}\mid S\subseteq L\\}.$ (1)

######  Lemma 3.1.

For a given $R$-semimodule $M$, the subsets of $\mathrm{Dis}_{M}$ determined
by $\mathcal{V}$ have the following properties.

  1. (1)

$\mathcal{V}(S)\supseteq\mathcal{V}(S^{\prime})$ for any two nonempty subsets
$S$ and $S^{\prime}$ of $M$ such that $S\subseteq S^{\prime}$.

  2. (2)

$\mathcal{V}(S)=\mathcal{V}(\langle S\rangle)$ for any nonempty subset $S$ of
$M$.

  3. (3)

$\mathcal{V}(0)=\mathrm{Dis}_{M}$ and $\mathcal{V}(M)=\emptyset.$

  4. (4)

$\bigcap_{\lambda\in\Lambda}\mathcal{V}(N_{\lambda})=\mathcal{V}\left(\sum_{\lambda\in\Lambda}N_{\lambda}\right)$
for all $N_{\lambda}\in\mathrm{Sub}_{M}$ and for all $\lambda\in\Lambda$.

  5. (5)

$\mathcal{V}(N)\cup\mathcal{V}(N^{\prime})\subseteq\mathcal{V}(N\cap
N^{\prime})$ for all $N,$ $N^{\prime}\in\mathrm{Sub}_{M}.$

  6. (6)

$\mathcal{V}(N)\supseteq\mathcal{V}(\sqrt{N})$ for any subsemimodule $N$ of
$M$.

  7. (7)

Let $N$ be subsemimodule of $M$. Define $N^{\omega}:=\bigcap\\{L\mid
L\in\mathcal{V}(N)\\}$. Then $N\subseteq N^{\omega}$.

  8. (8)

$N=N^{\omega}$, whenever $N\in\mathrm{Dis}_{M}.$

  9. (9)

$\mathcal{V}(N)=\mathcal{V}(N^{\omega}),$ for all subsemimodule $N$ of $M$.

  10. (10)

For any two subsemimodules $N_{1}$ and $N_{2}$ of $M$,
$\mathcal{V}(N_{1})\subseteq\mathcal{V}(N_{2})$ if and only if
$N_{2}^{\omega}\subseteq N_{1}^{\omega}$.

###### Proof.

Straightforward. ∎

Note that we have equality in Lemma 3.1(6) whenever $\mathrm{Dis}_{M}$ is the
class of prime, minimal prime, or maximal subsemimodules. Our aim is to
construct a topology on a $\mathrm{Dis}_{M}$ using the subsets of the form
$\\{\mathcal{V}(S)\\},$ and instead of (1), thanks to Lemma 3.1(2), it is now
sufficient to consider only the subsets

$\mathcal{V}(N)=\\{L\in\mathrm{Dis}_{M}\mid N\subseteq
L\\}\qquad(N\in\mathrm{Sub}_{M}).$ (2)

From Lemma 3.1(3) it follows

$\bigcap_{N\in\mathrm{Sub}_{M}}\mathcal{V}(N)=\emptyset,$

and by [5, Theorem 15 A.13., p. 254], this implies that the collection
$\\{\mathcal{V}(N)\\}_{N\in\mathrm{Sub}_{M}}$ as a closed subbasis generates a
unique topology on $\mathrm{Dis}_{M}$. We denote this topology by
$\tau_{\mathrm{Dis}_{M}}$.

In general, the sets $\mathcal{V}(N)$ are not closed sets but only subbasis
closed sets. This is due the fact that for an arbitrary $\mathrm{Dis}_{M}$, we
do not have equality in Lemma 3.1(5) (i.e., sets $\mathcal{V}(N)$ are not
closed under finite union). It has been shown in [13, Lemma 3.2] that equality
holds in Lemma 3.1(5) if and only if $M$ is a top semimodule, and in that
case, our closed-subasis topology coincides with the Zariski topology endowed
on any $\mathrm{Dis}_{M}$. With an abuse of notation, we simply write
$\mathrm{Dis}_{M}$ to denote the topological space
$(\mathrm{Dis}_{M},\tau_{\mathrm{Dis}_{M}})$.

For compactness of a subsemimodule space $\mathrm{Dis}_{M}$, we require the
property:

$\mathcal{V}(N)=\emptyset\implies N=M.$ (3)

In general, (3) is not true for an arbitrary class of subsemimodules. However,
if $M$ is finitely generated, then (3) holds for any $\mathrm{Dis}_{M}$. For
any semimodule, the following result gives a necessary condition for a class
of subsemimodules have the property (3).

######  Lemma 3.2.

If a subsemimodule space $\mathrm{Dis}_{M}$ has the property $(\ref{scc})$,
then $\mathrm{Dis}_{M}$ contains all maximal subsemimodules of $M$.

###### Proof.

Let $\mathrm{Dis}_{M}$ has the property (3). Let $N$ be a maximal
subsemimodule of $M$ such that $N\notin\mathrm{Dis}_{M}$. This implies
$\mathcal{V}(N)=\emptyset$ and hence $N=M$, a contradiction. ∎

If $M$ is finitely generated, then the converse of the above Lemma 3.2 is also
true. Since our topology is generated by subbasis closed sets, in the proof of
the following theorem we need to rely on Alexander’s subbasis theorem.

######  Theorem 3.3.

If $M$ is a finitely generated $R$-semimodule, then any subsemimodule space
$\mathrm{Dis}_{M}$ is compact.

###### Proof.

Suppose $\bigcap_{\lambda\in\Lambda}K_{\lambda}=\emptyset$ for a family
$\\{K_{\lambda}\\}_{\lambda\in\Lambda}$ of subbasis closed sets of a
subsemimodule space $\mathrm{Dis}_{M}$. By Alexander’s subbasis theorem, it
suffices to show that $\bigcap_{i\,=1}^{n}K_{\lambda_{i}}=\emptyset,$ for some
finite subset $\\{\lambda_{1},\ldots,\lambda_{n}\\}$ of $\Lambda$. Let
$\\{N_{\lambda}\\}_{\lambda\in\Lambda}$ be a class of subsemimodules of $M$
such that $K_{\lambda}=\mathcal{V}(N_{\lambda})$ for all $\lambda\in\Lambda,$
By Lemma 3.1(4),
$\mathcal{V}\left(\sum_{\lambda\in\Lambda}N_{\lambda}\right)=\emptyset,$ and
by property (3), this implies $\sum_{\lambda\in\Lambda}N_{\lambda}=M.$ Since
$M$ is finitely generated, we have the desired finite subset of $\Lambda$. ∎

By [15]), a topological space $X$ is $T_{0}$ if and only if specialization
order of $X$ is a partial order, and this implies

######  Proposition 3.4.

Every $\mathrm{Dis}_{M}$ is a $T_{0}$-space.

######  Corollary 3.5.

[13, Theorem 3.1]. If $M$ is a top $R$-semimodule, then the spectrum of prime
subsemimodules of $M$ endowed with Zariski topology is a $T_{0}$-space.

We recall a few topological concepts that we are going to use next. If $X$ is
a topological space, a closed subset $S$ is called _irreducible_ if $S$ is not
the union of two properly smaller closed subsets $S_{1},S_{2}\subsetneq S.$ A
point $x$ in a closed subset $S$ is called a _generic point_ of $S$ if
$S=\mathcal{C}(\\{x\\}),$ the closure of $\\{x\\}$. A $R$-semimodule $M$ is
said to be _Noetherian_ if every ascending chain of subsemimodules of $M$ is
eventually stationary. A topological space $X$ is said to be _Noetherian_ if
the closed subsets of $X$ satisfy the descending chain condition.

Suppose $M$ is a top $R$-semimodule and $\mathrm{Dis}_{M}$ is the class of
prime subsemimodules of $M$ endowed with the Zariski topology. Then according
to [13, Theorem 3.3], a nonempty subset $Y$ of $\mathrm{Dis}_{M}$ is
irreducible if and only if the intersection of all prime subsemimodules
belonging to $Y$ is itself a prime subsemimodule of $M$. For a $R$-semimodule
$M$ and for an arbitrary subsemimodule space we do not a similar
characterization, however, in Lemma 3.6 we identify a class of irreducible
closed subsets which is good enough to characterize $T_{1}$ subsemimodule
spaces. Moreover, this lemma is going to be fruitful in studying connectedness
of subsemimodule spaces.

######  Lemma 3.6.

The subbasis closed sets of the form $\\{\mathcal{V}(N)\mid
N\in\mathcal{V}(N)\\}$ of a subsemimodule space are irreducible.

###### Proof.

Let $\mathcal{C}(N)$ be the closure of the subsemimodule $N$. It is sufficient
to show that $\mathcal{V}(N)=\mathcal{C}(N)$ for every $N\in\mathcal{V}(N)$.
From the definition of $\mathcal{C}(N)$, it is obvious that
$\mathcal{C}(N)\subseteq\mathcal{V}(N)$. To have the other inclusion, consider
the case: $\mathcal{C}(N)=\mathrm{Dis}_{M}$. Then from
$\mathrm{Dis}_{M}=\mathcal{C}(N)\subseteq\mathcal{V}(N)\subseteq\mathrm{Dis}_{M},$
it follows that $\mathcal{V}(N)=\mathcal{C}(N)$.

Let $\mathcal{C}(N)\neq\mathrm{Dis}_{M}$. Since $\mathcal{C}(N)$ is a closed
set, there exists an $\Lambda$ such that for each $\lambda\in\Lambda$, there
is a positive integer $n_{\lambda}$ and subsemimodules $N_{\lambda
1},\dots,N_{\lambda n_{\lambda}}$ of $M$ such that

$\mathcal{C}(N)={\bigcap_{\lambda\in\Lambda}}\left({\bigcup_{i\,=1}^{n_{\lambda}}}\mathcal{V}(N_{\lambda
i})\right).$

Since $\mathcal{C}(N)\neq\mathrm{Dis}_{M},$ for each $\lambda$,
${\bigcup_{i\,=1}^{n_{\lambda}}}\mathcal{V}(N_{\lambda i})\neq\emptyset$.
Hence, for each $\lambda$,
$N\in{\bigcup_{i\,=1}^{n_{\lambda}}}\mathcal{V}(N_{\lambda i})$, and that
implies

$\mathcal{V}(N)\subseteq{\bigcup_{i=1}^{n_{\lambda}}}\mathcal{V}(N_{\lambda
i}),$

which proves the inclusion. ∎

######  Corollary 3.7.

All non-empty subbasis closed subset of proper subsemimodule spaces are
irreducible.

It has been shown in [13, Theorem 3.2] that in a top semimodule, the class of
prime subsemimodules endowed with Zariski topology is a $T_{1}$-space if and
only if every prime subsemimodule is maximal in the class of prime
subsemimodules. Theorem 3.8 generalizes this result to all $\mathrm{Dis}_{M}$
spaces.

######  Theorem 3.8.

A $\mathrm{Dis}_{M}$ is a $T_{1}$-space if and only if it is contained in the
class of maximal subsemimodule space.

###### Proof.

If $L\in\mathrm{Dis}_{M}$, then obviously $L\in\mathcal{V}(L)$, and so, by
Lemma 3.6, $\mathcal{C}(L)=\mathcal{V}(L)$. Let $N$ be a maximal subsemimodule
of $M$ with $L\subseteq N$. If $\mathrm{Dis}_{M}$ is a $T_{1}$-space, then

$N\in\mathcal{V}(L)=\mathcal{C}(L)=\\{L\\}.$

Therefore, $N=L$. Conversely, $\mathcal{V}(N)=\\{N\\}$ for every maximal
subsemimodule $N$ of $M$ so that $N\in\mathcal{V}(N)$, and hence, by Lemma
3.6, $\mathcal{C}(N)=\\{N\\}$. This proves that the class of maximal
subsemimodules of $M$ is a $T_{1}$-space. ∎

######  Corollary 3.9.

If $M$ is a Noetherian semimodule and if $\mathrm{Dis}_{M}$ is a discrete
space, then $M$ is Artinian.

Determining whether every nonempty irreducible closed set of a topological
space has a unique generic point is a fundamental question in topology. It has
been proved in [13, Corollary 3.1] that the class of prime subsemimodules of a
top semimodule endowed with Zariski topology has this property. In Theorem
3.10 we characterize such subsemimodule spaces.

######  Theorem 3.10.

Let $\mathrm{Dis}_{M}$ be a subsemimodule space. Then every non-empty
irreducible subbasis closed set $\mathcal{V}(N)$ of $\mathrm{Dis}_{M}$ has a
unique generic point if and only if $\mathcal{V}(N)$ contains $N^{\omega}.$

###### Proof.

Let $\mathcal{V}(N)$ is a non-empty irreducible subbasis closed subset of a
subsemimodule space $\mathrm{Dis}_{M}$ such that $\mathcal{V}(N)$ has a unique
generic point, which implies that

$\mathcal{V}(N)=\mathcal{C}(L)=\mathcal{V}(L).$

Then by Lemma 3.1(8) we have $L=N^{\omega}\in\mathrm{Dis}_{M}$. The proof of
the converse part is more involved. Assume that every non-empty irreducible
subbasis closed set $\mathcal{V}(N)$ contains $N^{\omega}$ and let $K$ be an
irreducible closed subset of $\mathrm{Dis}_{M}$. By definition,

$K=\bigcap_{i\in
I}\left(\bigcup_{\alpha=1}^{n_{i}}\mathcal{V}(N_{i\alpha})\right),$

for some subsemimodules $N_{ij}$ of $M$. Since $K$ is irreducible, for every
$i\in I$ there exists a subsemimodule $N_{i}$ of $M$ such that
$K\subseteq\mathcal{V}(N_{i})\subseteq\bigcup_{\alpha=1}^{n_{i}}\mathcal{V}(N_{i\alpha})$
and thus we have

$K=\bigcap_{i\in I}\mathcal{V}(N_{i})=\mathcal{V}\left(\sum_{i\in
I}N_{i}\right)=\mathcal{V}\left(\sum_{i\in I}N_{i}\right)^{\omega}.$

Since $\left(\sum_{i\in I}N_{i}\right)^{\omega}\in\mathrm{Dis}_{M}$, we thus
have $K=\mathcal{C}\left(\sum_{i\in I}N_{i}\right)^{\omega}$. ∎

######  Corollary 3.11.

Every nonempty irreducible closed subset of a proper, prime, minimal prime
subsemimodule spaces has a unique generic point.

######  Corollary 3.12.

Let $M$ be a top $R$-semimodule. Then every nonempty irreducible closed subset
(with respect to Zariski topology) of a prime and minimal prime subsemimodule
spaces has a unique generic point.

######  Remark 3.13.

Recall that a topological space $X$ is called _sober_ if every nonempty
irreducible closed subset of $X$ has a unique generic point. Sobriety is a
separation axiom in between $T_{0}$ and $T_{2}$ (and independent of $T_{1}$).
Theorem 3.10 in fact determines subsemimodule sober spaces.

######  Proposition 3.14.

Every nonempty irreducible closed subset of a strongly irreducible
subsemimodule space has a unique generic point.

###### Proof.

Since $\mathrm{Des}_{M}$ is a multiplicative lattice with product as
intersection, every strongly irreducible subsemimodule becomes a prime element
of this lattice and hence by [9, Lemma 2.6], we have the desired conclusion. ∎

The usual notion of connectedness does not have a direct formulation in terms
of subbasis closed sets. We, therefore, introduce a different kind of
connectedness (in terms of subbasis closed sets) of subsemimodule spaces that
is related to the conventional notion of connectedness. We say a closed
subbasis $\mathcal{S}$ of a topological space $X$ _strongly disconnects_ $X$
if there exist two non-empty subsets $A,$ $B\in\mathcal{S}$ such that $X=A\cup
B$ and $A\cap B=\emptyset$.

From the above definition, it is clear that if some closed subbasis strongly
disconnects a space, then the space is disconnected. Moreover, if a space is
disconnected, then some closed subbasis strongly disconnects it. But this does
not imply that every closed subbasis strongly disconnects the space. However,
we have the following result from [7].

######  Lemma 3.15.

If a compact space is disconnected, then every closed basis that is closed
under finite intersections strongly disconnects the space. Moreover, for any
subsemimodule space $\mathrm{Dis}_{M}$, the closed basis is closed under
binary intersections.

Based on the above Lemma 3.15, we immediately have the following result.

######  Theorem 3.16.

Suppose $\mathrm{Dis}_{M}$ is a compact space. Then $\mathrm{Dis}_{M}$ is
disconnected if and only if the closed basis strongly disconnects
$\mathrm{Dis}_{M}$.

A consequence of Lemma 3.6 is the following sufficient condition for a
subsemimodule space to be connected.

######  Theorem 3.17.

If $0\in\mathrm{Dis}_{M},$ then the subsemimodule space $\mathrm{Dis}_{M}$ is
connected.

###### Proof.

Since by Lemma 3.1(3), $\mathrm{Dis}_{M}=\mathcal{V}(0)$ and since
irreducibility implies connectedness, the claim follows from Lemma 3.6. ∎

######  Corollary 3.18.

Proper, finitely generated, cyclic subsemimodule spaces are connected.

Suppose $M\to M^{\prime}$ is an $R$-semimodule homomorphism. Likewise rings,
it is natural to expect a continuous map
$\phi\colon\mathrm{Dis}_{M^{\prime}}\to\mathrm{Dis}_{M}$ between similar type
subsemimodule spaces. It is indeed so, and we shall discuss that in the next
proposition. The only issue here is that the inverse image of a particular
type of subsemimodule might not be of the same type. So, we need to consider
only those $R$-semimodule homomorphisms for which such preservation holds. We
call this a _contraction property_. Furthermore, because of the nature of our
topology, instead of closed sets, in the arguments of the proofs, we shall
require subbasis closed sets.

######  Proposition 3.19.

Let $\mathrm{Dis}_{M}$ be a subsemimodule space and $\phi\colon M\to
M^{\prime}$ be an $R$-semimodule homomorphism having the contraction property
with respect to $\mathrm{Dis}_{M}$. Let
$N^{\prime}\in\mathrm{Dis}_{M^{\prime}}.$ Define a map
$\phi^{*}\colon\mathrm{Dis}_{M^{\prime}}\to\mathrm{Dis}_{M}$ by
$\phi^{*}(N)=\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(N)$. Then

  1. (1)

the map $\phi^{*}$ is continuous.

  2. (2)

If $\phi$ is surjective, then the subsemimodule space
$\mathrm{Dis}_{M^{\prime}}$ is homeomorphic to the closed subspace
$\mathcal{V}(\mathrm{ker}\,\phi)$ of the subsemimodule space
$\mathrm{Dis}_{M}.$

  3. (3)

The image $\phi^{*}(\mathrm{Dis}_{M^{\prime}})$ is dense in $\mathrm{Dis}_{M}$
if and only if

$\mathrm{ker}\,\phi\subseteq\bigcap_{L\in\mathrm{Dis}_{M}}L;$

###### Proof.

To show (1), let $\mathcal{V}(N)$ be a subbasic closed set of the
subsemimodule space $\mathrm{Dis}_{M}.$ Observe that

$(\phi^{*})^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(\mathcal{V}(N))=\\{N^{\prime}\in\mathrm{Dis}_{M^{\prime}}\mid\phi(N)\subseteq
N^{\prime}\\}=\mathcal{V}(\langle\phi(N)\rangle),$

and hence the map $\phi^{*}$ continuous. For the homeomorphism in (2), observe
that
$\mathrm{ker}\,\phi\subseteq\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(N^{\prime}),$
in other words, $\phi^{*}(N^{\prime})\in\mathcal{V}(\mathrm{ker}\,\phi)$. This
implies that $\mathrm{im}\,\phi^{*}=\mathcal{V}(\mathrm{ker}\,\phi).$ Since
for all $N^{\prime}\in\mathrm{Dis}_{M^{\prime}},$

$\phi(\phi^{*}(N^{\prime}))=N^{\prime}\cap\mathrm{im}\,\phi=N^{\prime},$

the map $\phi^{*}$ is injective. To show that $\phi^{*}$ is a closed map,
first we observe that for any subbasis closed subset $\mathcal{V}(N)$ of
$\mathrm{Dis}_{M^{\prime}}$, we have

$\phi^{*}(\mathcal{V}(N))=\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}\\{L^{\prime}\in\mathrm{Dis}_{M^{\prime}}\mid
N\subseteq
L^{\prime}\\}=\mathcal{V}(\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(N)).$

Now if $K$ is a closed subset of $\mathrm{Dis}_{M^{\prime}}$ and if

$K=\bigcap_{\alpha\in\Omega}\left(\bigcup_{i\,=1}^{n_{\alpha}}\mathcal{V}(N_{i\alpha})\right),$

then

$\phi^{*}(K)=\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}\left(\bigcap_{\alpha\in\Omega}\left(\bigcup_{i=1}^{n_{\alpha}}\mathcal{V}(N_{i\alpha})\right)\right)=\bigcap_{\alpha\in\Omega}\bigcup_{i=1}^{n_{\alpha}}\mathcal{V}(\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(N_{i\alpha})),$

a closed subset of $\mathrm{Dis}_{M}.$ Since $\phi^{*}$ is continuous, we have
the proof. To prove (3), we first show that
$\mathcal{C}(\phi^{*}(\mathcal{V}(N^{\prime})))=\mathcal{V}(\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(N^{\prime})),$
for all subsemimodules $N^{\prime}\in\mathrm{Sub}_{M^{\prime}}.$ To this end,
let $L\in\phi^{*}(\mathcal{V}(N^{\prime})).$ This implies
$\phi(L)\in\mathcal{V}(N^{\prime}),$ which means $N^{\prime}\subseteq\phi(L).$
In other words,
$L\in\mathcal{V}(\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(N^{\prime})).$
The other inclusion follows from the fact that
$\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(\mathcal{V}(N^{\prime}))=\mathcal{V}(\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(N^{\prime})).$
Since

$\mathcal{C}(\phi^{*}(\mathrm{Dis}_{M^{\prime}}))=\mathcal{V}(\phi^{\raisebox{0.60275pt}{$\scriptscriptstyle-1$}}(\mathfrak{o}))=\mathcal{V}(\mathrm{ker}\,\phi),$

the closed subspace $\mathcal{V}(\mathrm{ker}\,\phi)$ is equal to
$\mathrm{Dis}_{M}$ if and only if
$\mathrm{ker}\,\phi\subseteq\cap_{L\in\mathrm{Dis}_{M}}L.$ ∎

######  Corollary 3.20.

The subsemimodule space $\mathrm{Dis}_{M/N}$ is homeomorphic to the closed
subspace $\mathcal{V}(N)$ of $\mathrm{Dis}_{M}$.

It has been shown in [13] that the class of prime subsemimodules of a finitely
generated top semimodule is spectral in the sense of [12]. An attempt has been
made in [8] to classify spectral spaces among the ideal spaces of a
commutative ring. It would be interesting to find out which subsemimodule
spaces are spectral. It is clear that one cannot just copy the arguments from
[8] to determine that. It will possibly require some additional methods.

## References

  * [1] R. E. Atani and S. E. Atani, On subsemimodules of semimodules, Bull. Acad. Stiinte. Repub. Mold Mat., 63 (2010), 20–30.
  * [2] S. E. Atani, R. E. Atani, and U. Tekir, A Zariski topology for semimodules, Eur. J. Pure Appl. Math., 4(3) (2011), 251–265.
  * [3] E. Atani and A. Kohan, A note on finitely generated multiplication semimodules over commutative semirings, Int. J. Algebra, 4(5-8) (2010), 389–396.
  * [4] M. Bataineh and R. Malas, Primal and weakly primal sub semi modules, Amer. Int. J. Contem. Research, 4(1) (2014), 131–135.
  * [5] E. Čech, _Topological spaces_ , Int. Publ. John Wiley & Sons., 1966.
  * [6] I. Chajdaa and H. Länger, Semimodules over commutative semirings and modules over unitary commutative rings, Linear multilinear algebra, 70(7) (2022), 1329–1344.
  * [7] T. Dube and A. Goswami, Ideal spaces: An extension of structure spaces of rings, J. Alg. Appl. (online ready).
  * [8] C. A. Finocchiaro, A. Goswami, and D. Spirito, Distinguished classes of ideal spaces and their topological properties, Comm. Alg., 2022 (online ready).
  * [9] A. Facchini, C. A. Finocchiaro, and G. Janelidze, Abstractly constructed prime spectra, Algebra Univers., 83(8) (2022), 1–38.
  * [10] J. S. Golan, Semirings and their applications, Springer, 1999.
  * [11] S. C. Han, Maximal and prime k-subsemimodules in semimodules over semirings, J. Algebra Appl., 16(7) (2017) 1750130.
  * [12] M. Hochster, Prime ideal structure in commutative rings, Trans. Am. Math. Soc., 142 (1969), 43–60.
  * [13] C-H Han, W-S Pae, and J-N Ho, Topological properties of the prime spectrum of a semimodule, J. Algebra, 566 (2021) 205–221.
  * [14] C-H Han, J-N Ho, and W-S Pae Properties of the subtractive prime spectrum of a semimodule, Hacet. J. Math. Stat. XX(x) (2022), 1–14.
  * [15] H. A. Priestley, Intrinsic spectral topologies, _Papers on general topology and applications_ , (Flushing, NY, 1992), 78–95, Ann. New York Acad. Sci., 728, New York Acad. Sci., New York, 1994.
  * [16] M. Takahashi, Structures of semimodules. Kobe J Math., 4 (1987), 79–101.
  * [17] H. A. Tavallaee and M. Zolfaghari, On semiprime subsemimodules and related results, J. Indones. Math. Soc., 19(1) (2013), 49–59.
  * [18] H. S. Vandiver, Note on a simple type of algebra in which the cancellation law of addition does not hold, Bull. Am. Math. Soc., 40(12) (1934), 914–920.
  * [19] ———, On some simple types of semi-rings, Am. Math. Monthly, 46(1) (1939), 22–26.
  * [20] G. Yeşilot, On prime and maximal k-subsemimodules of semimodules, Hacet. J. Math. Stat., 39(3) (2010) 305–312.
  * [21] G. Yeşilot, K. Orel, and U. Tekir, On prime subsemimodules, Int. J. Algebra, 4 (2010) 53–60.


# AliCHI: A Large-scale Multi-modal Dataset and Automated Evaluation Tool for
Human-like Dialogue Systems

Zhiling Luo  DAMO Academy, Alibaba Group1eHangzhouChina godot.lzl@alibaba-
inc.com ,  Qiankun Shi  DAMO Academy, Alibaba GroupChina ,  Sha Zhao  College
of Computer Science, Zhejiang University1eHangzhouChina<EMAIL_ADDRESS>,
Wei Zhou  DAMO Academy, Alibaba GroupChina ,  Haiqing Chen  DAMO Academy,
Alibaba GroupChina ,  Yuankai Ma  DAMO Academy, Alibaba GroupChina and  Haitao
Leng  DAMO Academy, Alibaba GroupChina

###### Abstract.

A well-designed interactive human-like dialogue system is expected to take
actions (e.g. smiling) and respond in a pattern similar to humans. However,
due to the limitation of single-modality (only speech) or small volume of
currently public datasets, most dialogue systems can only respond in speech
and cannot take human-like actions. In this work, we build a large-scale
multi-modal dataset of human-to-human conversation in a face-to-face fashion,
with fine-grained annotations. The raw data in video format contains 635
dialogue sessions, being collected from 200 participants on designed topics
and lasting 52 hours in total. Moreover, we manually annotated the verbal and
non-verbal behaviors in each dialogue session on their start/end timestamp.
Furthermore, we developed a corresponding evaluation tool for human-like
dialogue systems to automatically evaluates the accuracy of two basic tasks,
turn-taking prediction, and backchannel prediction, on both time and content.
We have opened the data, the tools will be released at the conference.

human-like dialogue system, turn-taking prediction, back-channel prediction,
HCI system

##  1\. Introduction

As a kind of popular human-machine interaction system, human-like dialogue
systems (HDS) have been widely applied in many scenarios in our daily life.
Generally, HDSs are designed for specific applications, for example, an
assistant app in a smartphone, virtual customer service assistant, and even a
humanoid robot. They pursue the same goal of behaving like a human as much as
possible. To this end, HDSs are required to not only reply users’ request in
speech, but also take some non-verbal actions, e.g., smiling, which makes HDS
more like humans.

HDS tasks include backchannel prediction(Jain et.al., 2021) (Murray et.al.,
2022) and turn-taking prediction (Hara et.al., 2018) (Jiudong et.al., 2022),
and they require non-verbal behaviors to train human-like models. High-quality
datasets play a very import role in most existing algorithms for HDS tasks,
especially those involving machine learning approaches. Previous datasets
proposed in this area, such as Noxi(Cafaro et.al., 2017) and JD
dataset(Jiudong et.al., 2022), mostly focus on specific scenarios like
knowledge exchange scenarios or commercial systems, where people do not behave
in the same way as they do in daily life. Also, some of the datasets
containing only the utterance and speeches without the non-verbal behaviors,
thus cannot support non-verbal behaviors prediction task.

In order to make HDS more human-like and address the above limitation of
current public datasets, in this paper, we propose AliCHI, a large-scale
multi-modal dataset of human-to-human conversation in a face-to-face fashion,
containing fine-grained annotations, and a corresponding tool for evaluation.
Our contributions are two-fold:

AliCHI, to our best knowledge, is the largest multi-modal dataset on the
human-to-human conversation. There are 635 dialogue sessions of 52 hours,
collected from 200 participants using our designed topics. High-precision
annotations about behaviour time, verbal content, and non-verbal labels are
released for backchannel/turn-taking prediction tasks.

Based on our data, we propose a group of evaluating metrics for above two
tasks, and develop an automatic CLI tool to test arbitrary HDS systems. The
evaluating results computed by the tool can provide a reference for HDS
developers.

Figure 1. Demonstration of the settings in data collection Figure 2.
Demonstration of the annotations

##  2\. Dataset

###  2.1. Design Principle

To collect the multi-modal face-to-face conversation data, we consider the
following principles.

Generic: The data is expected to reflect the general pattern of both verbal
and non-verbal behaviors of humans. 200 participants were invited to collect
the data, much more than the number of participants in the NoXi dataset. Such
large number of participants can reduce the bias introduced by individual
characters to a certain degree. In particular, we design a group of common
topics used in conversations, e.g. sports, to further reduce the bias of
participants in knowledge. We make sure that all the participants have basic
knowledge of these topics.

Synchronized: The data contains two video flows from two participants in a
dialogue session captured by two cameras. These two videos are expected to be
synchronized so that the behavior of one participant is easily and exactly
aligned to the behavior of another participant. To reduce the synchronized
difference, we leveraged the phone with both front and rear cameras (Xiaomi
10, and HUAWEI Mate40). Both cameras recorded two videos and generated the
fully synchronized video by the phones’ inner photograph apps.

Natural: The behaviors between two participants are expected to be natural,
rather than rigid with distress. In each dialogue, the two participants are
friends, so that they can talk and act naturally.

###  2.2. Data Settings and Collection

The data is collected in the following settings, shown in Fig. 1a. Two
participants are required to sit on two sides of a long table ( Fig. 1b, 1d).
The phone with two cameras is placed in the center of the table, ensuring that
the two participants are captured in the center of a camera view plane (Fig.
1c). In this setting, the upper part of the body of the two participants can
be captured, thus we can capture many non-verbal behaviors, such as hand
gestures.

The data collection process is conducted in sessions. For each session, two
participants choose a topic from our prepared topics (travel, sports, etc.)
for the conversation. The cameras start to capture when they begin to talk
about the topic. The conversation ends if one participant claims that they
have finished the talk, and the data collection stops for this session. Each
session lasts about 5 minutes, and 92% of sessions last from 3 to 9 minutes.
The longest one lasts about 16 minutes, while the shortest one lasts about 2
minutes. By repeating the sessions, we collected 635 sessions in total in the
format of video and the time length is 52 hours in total.

###  2.3. Data Annotation

Due to privacy protection policy, we cannot directly release the raw videos
which contains the participants’ faces. Moreover, the raw data without any
annotations is hard to use. Therefore, we tried our best to annotate the
dataset. The annotated dataset is already released
111http://www.bruceluo.net/AliCHI.html and the annotations include:

Utterance from speeches with timestamp, including free-talking, responses, and
even backchannel responses;

Non-verbal behavior annotations from a defined annotation set with timestamp.
There are 27 annotations in total from five modalities: expressions (e.g.,
smile), head movements (e.g., nodding), eye movements (e.g., look up), eyebrow
movements (e.g., frown), and hand movements (e.g., arms crossed).

For the entire dataset, 66912 non-verbal behaviors are annotated in total, in
which the behaviors of looking left (10,377), looking right (10,246) and
smiling (7,299) happen the most frequently. For the verbal annotation, we
labeled the verbal content and its corresponding start/end timestamp. In each
session, short pause (silence) periods longer than a threshold are used to
divide the session into Inter-Pause Units (IPUs). We noticed within 0.5s
before or after the start or end of an IPU, non-verbal behaviors appeared 57%
more than other times, so a more fine-grained division helps to predict non-
verbal behaviors more accurately. The threshold we use is 150ms, which is
smaller than the regular 200ms. There are 159.85 IPUs per session on average
and 101,507 IPUs in all. An example of the annotation is shown in Fig. 2.

##  3\. Automatic Evaluation Tool

We also provide a tool to automatically evaluate the human-likeness of a HDS
on two basic tasks: backchannel prediction and turn-taking prediction.
Considering the example in Fig. 2a, A said about her/his swimming, at the same
time B smiled and said ’yeah’. Both the smiling behavior and short response
’yeah’ from B are known as the backchannel. Backchannels are used by the
listeners to acknowledge that the current speaker can hold the turn(Hara
et.al., 2018). After that, A continued and asked B, and then B replied with
laughter. This response is known as turn-taking because B took the turn and
knew that A would not say anything and just waited for B’s response.

Backchannel prediction task takes A’s all behaviors to forecast when and how B
would reply. Specifically speaking, three variables are required to predict:
1) timestamp, measured by IoU; 2) utterance, measured by Bleu and Rouge, with
timestamp given; 3) non-verbal behavior annotation, measured by accuracy, with
both timestamp and utterance given.

Turn-taking prediction task takes A’s all behaviors to forecast when and how B
would take the turn. Similar to backchannel prediction, it predicts 1)
timestamp, 2) utterance, and 3) non-verbal behavior. Note that turn-taking
happens on a pause longer than 150ms, but backchannels can happen at any time.

We propose a tool that automatically calculates the above metrics for
backchannel and turn-taking prediction. This tool contains a simple SDK which
is easily embedded in HDS to test above tasks by leveraging our dataset as the
ground truth. The code is implemented in Python, which will be released soon,
and available via pip.

## References

  * (1)
  * Cafaro et.al. (2017) Angelo Cafaro et.al. 2017\.  The NoXi database: multimodal recordings of mediated novice-expert interactions. In _IMCI_. 350–359.
  * Hara et.al. (2018) Kohei Hara et.al. 2018\.  Prediction of turn-taking using multitask learning with prediction of backchannels and fillers.  _Listener_ 162 (2018), 364.
  * Jain et.al. (2021) Vidit Jain et.al. 2021\.  Exploring semi-supervised learning for predicting listener backchannels. In _Proceedings of the 2021 CHI_. 1–12.
  * Jiudong et.al. (2022) Yang Jiudong et.al. 2022\.  Gated Multimodal Fusion with Contrastive Learning for Turn-Taking Prediction in Human-Robot Dialogue. In _ICASSP 2022_. IEEE, 7747–7751.
  * Murray et.al. (2022) Michael Murray et.al. 2022\.  Learning backchanneling behaviors for a social robot via data augmentation from human-human conversations. In _Conference on Robot Learning_. PMLR, 513–525.


# Zero-Shot Object Counting
with Language-Vision Models

Jingyi Xu, Hieu Le and Dimitris Samaras J. Xu and D. Samaras are with the
Department of Computer Science, Stony Brook University, Stony Brook, NY 11794.
E-mail: {jingyixu<EMAIL_ADDRESS>Le is with the Computer Vision
Lab, EPFL, Laussane, Switzerland, E-mail<EMAIL_ADDRESS>
###### Abstract

Class-agnostic object counting aims to count object instances of an arbitrary
class at test time. It is challenging but also enables many potential
applications. Current methods require human-annotated exemplars as inputs
which are often unavailable for novel categories, especially for autonomous
systems. Thus, we propose zero-shot object counting (ZSC), a new setting where
only the class name is available during test time. This obviates the need for
human annotators and enables automated operation. To perform ZSC, we propose
finding a few object crops from the input image and use them as counting
exemplars. The goal is to identify patches containing the objects of interest
while also being visually representative for all instances in the image. To do
this, we first construct class prototypes using large language-vision models,
including CLIP and Stable Diffusion, to select the patches containing the
target objects. Furthermore, we propose a ranking model that estimates the
counting error of each patch to select the most suitable exemplars for
counting. Experimental results on a recent class-agnostic counting dataset,
FSC-147, validate the effectiveness of our method.

###### Index Terms:

Class-agnostic object counting, variational autoencoder, diffusion models,
stable diffusion, zero-shot learning

##  1 Introduction

Object counting aims to infer the number of objects in an image. Most of the
existing methods focus on counting objects from specialized categories such as
human crowds [1], cars [2], animals [3], and cells [4]. These methods count
only a single category at a time. Recently, class-agnostic counting [5, 6, 7]
has been proposed to count objects of arbitrary categories. Several human-
annotated bounding boxes of objects are required to specify the objects of
interest (see Figure 1a). However, having humans in the loop is not practical
for many real-world applications, such as fully automated wildlife monitoring
systems or visual anomaly detection systems.

(a) Few-shot Counting  (b) Zero-Shot Counting

Figure 1:  Our proposed task of zero-shot object counting (ZSC). Traditional
few-shot counting methods require a few exemplars of the object category (a).
We propose zero-shot counting where the counter only needs the class name to
count the number of object instances. (b). Few-shot counting methods require
human annotators at test time while zero-shot counters can be fully automatic.

A more practical setting, exemplar-free class-agnostic counting, has been
proposed recently by Ranjan et al.[8]. They introduce RepRPN, which first
identifies the objects that occur most frequently in the image, and then uses
them as exemplars for object counting. Even though RepRPN does not require any
annotated boxes at test time, the method simply counts objects from the class
with the highest number of instances. As a result, it can not be used for
counting a specific class of interest. The method is only suitable for
counting images with a single dominant object class, which limits its
potential applicability.

Thus, our goal is to build an exemplar-free object counter where we can
specify what to count. To this end, we introduce a new counting task in which
the user only needs to provide the name of the class for counting rather than
the exemplars (see Figure 1b). Note that the class to count during test time
can be arbitrary. For cases where the test class is completely unseen to the
trained model, the counter needs to adapt to the unseen class without any
annotated data. Hence, we name this setting zero-shot object counting (ZSC),
inspired by previous zero-shot learning approaches [9, 10].

To count without any annotated exemplars, we propose finding a few patches in
the input image containing the target object to use them as counting
exemplars. There are two challenges: 1) how to localize patches that contain
the object of interest based on the provided class name, and 2) how to select
good exemplars for counting. Ideally, good object exemplars are visually
representative for most instances in the image, which can benefit the object
counter. In addition, we want to avoid selecting patches that contain
irrelevant objects or backgrounds, which likely lead to incorrect object
counts. To this end, we propose a two-step method that first localizes the
class-relevant patches which contain the objects of interest based on the
given class name, and then selects among these patches the optimal exemplars
for counting. We use these selected exemplars, together with a pre-trained
exemplar-based counting model, to achieve exemplar-free object counting.

The first step of our framework involves constructing a class prototype based
on the given class name. Essentially, this requires a mapping between the
categorical label and its visual feature. We employ pre-trained large
language-vision models to accomplish this via two approaches.

In the conference version of this paper [11], we learn this mapping between
language queries and visual features via a conditional variational autoencoder
(VAE). This VAE model is trained to generate visual features of object crops
for any arbitrary class, conditioned on its semantic embedding extracted from
a pre-trained language-vision model [12]. We take the average of the generated
features to compute the class prototype, which can be then used to select
class-relevant patches through a simple nearest-neighbour lookup scheme. In
essence, our VAE-based approach creates a single prototypical feature for each
category that can be applied to any images of this class.

However, a single prototypical feature might not work well for categories with
significant intra-class variance. Objects of the same category across
different images can exhibit significant differences in colors (e.g., a green
apple versus a red apple), shapes (e.g., an SUV versus a regular car), scales,
or materials (a wooden versus a fabric chair). To better handle this
variability, we propose to construct a class prototype specific to each image.
To do so, we leverage the recent advancements in text-to-image generative
models, i.e., Stable Diffusion [13], for prototype generation. Compared to
classic VAE-based models, Stable Diffusion enables more realistic and diverse
sample generation thanks to large-scale training data. This provides a way to
deal with the variations in query objects. Specifically, given a query image,
we first use Stable Diffusion to generate a variety of images containing the
objects of interest. Then we select among them the object crops that most
resemble the query objects and only use them for constructing the class
prototype. In this way, a unique prototype is constructed specifically for
each testing sample, as opposed to the VAE-based approach that uses a single
universal prototype for all images. We show that using image-specific
prototypes generally leads to better counting performance, compared to using a
single generic categorical one.

After obtaining the class-relevant patches, we want to select among them the
optimal patches to be used as counting exemplars. Here we observe that the
feature maps obtained using good exemplars and bad exemplars often exhibit
distinguishable differences. An example of the feature maps obtained with
different exemplars is shown in Figure 2. The feature map from a good exemplar
typically exhibits some repetitive patterns (e.g., the dots on the feature
map) that center around the object areas while the patterns from a bad
exemplar are more irregular and occur randomly across the image. Based on this
observation, we train a model to measure the goodness of an input patch based
on its corresponding feature maps. Specifically, given an arbitrary patch and
a pre-trained exemplar-based object counter, we train this model to predict
the counting error of the counter when using the patch as the exemplar. Here
the counting error can indicate the goodness of the exemplar. After this error
predictor is trained, we use it to select those patches with the smallest
predicted errors as the final exemplars for counting.

Experiments on the FSC-147 dataset show the effectiveness of our proposed
patch selection method. We also provide analyses to show that patches selected
by our method can be used in other exemplar-based counting methods to achieve
exemplar-free counting. In short, our main contributions are:

  * •

We introduce the task of zero-shot object counting that counts the number of
instances of a specific class in the input image, given only the class name
and without relying on any human-annotated exemplars.

  * •

We leverage language-vision models to construct class prototypes via two
approaches: VAE-based approach and SD-based approach. We show that in both
cases the class prototypes can be used to accurately select patches containing
objects of interests for counting.

  * •

We introduce an error prediction model to further select the optimal patches
that yield the smallest counting errors.

  * •

We verify the effectiveness of our patch selection method on the FSC-147
dataset, through extensive ablation studies and visualization results.

Figure 2:  Feature maps obtained using different exemplars given a pre-trained
exemplar-based counting model. The feature maps obtained using good exemplars
typically exhibit some repetitive patterns while the patterns from bad
exemplars are more irregular.

##  2 Related Work

###  2.1 Class-specific Object Counting

Class-specific object counting focuses on counting pre-defined categories,
such as humans [14, 15, 16, 17, 18, 19, 1, 20, 21, 22, 23], animals [3], cells
[4], or cars [2, 24]. Generally, existing methods can be categorized into two
groups: detection-based methods [25, 24, 26] and regression-based methods [27,
28, 29, 30, 31, 32, 33]. Detection-based methods apply an object detector on
the image and count the number of objects based on the detected boxes.
Regression-based methods predict a density map for each input image, and the
final result is obtained by summing up the pixel values. Both types of methods
require abundant training data to learn a good model. Class-specific counters
can perform well on trained categories. However, they can not be used to count
objects of arbitrary categories at test time.

###  2.2 Class-agnostic Object Counting

Class-agnostic object counting aims to count arbitrary categories given only a
few exemplars [7, 5, 34, 6, 35, 36, 37, 38, 39]. GMN [7] uses a shared
embedding module to extract feature maps for both query images and exemplars,
which are then concatenated and fed into a matching module to regress the
object count. FamNet [5] adopts a similar way to do correlation matching and
further applies test-time adaptation. These methods require human-annotated
exemplars as inputs. Recently, exemplar-free object counting has been proposed
to eliminate the need for user inputs. Ranjan et al. have proposed RepRPN [8],
which achieves exemplar-free counting by identifying exemplars from the most
frequent objects via a Region Proposal Network (RPN)-based [40] model.
However, the class of interest can not be explicitly specified for the RepRPN.
In comparison, our proposed method can count instances of a specific class
given only the class name.

###  2.3 Zero-shot Image Classification

Zero-shot classification aims to classify unseen categories for which data is
not available during training [41, 42, 43, 44, 45, 46]. Semantic descriptors
are mostly leveraged as a bridge to enable the knowledge transfer between seen
and unseen classes. Earlier zero-shot learning (ZSL) works relate the semantic
descriptors with visual features in an embedding space and recognize unseen
samples by searching their nearest class-level semantic descriptor in this
embedding space [47, 45, 48, 49]. Recently, generative models [50, 51, 52]
have been widely employed to synthesize unseen class data to facilitate ZSL
[53, 54, 55]. Xian et al. [54] use a conditional Wasserstein Generative
Adversarial Network (GAN) [56] to generate unseen features which can then be
used to train a discriminative classifier for ZSL. In our method, we also
train a generative model conditioned on class-specific semantic embedding.
Instead of using this generative model to hallucinate data, we use it to
compute a prototype for each class. This class prototype is then used to
select patches that contain objects of interest.

###  2.4 Diffusion Models

Diffusion models [57, 58, 59] recently have demonstrated great success in
text-to-image generative systems (e.g., DALL-E [60], Imagen [61] and Stable
Diffusion (SD) [13]). Exploring the potential of pre-trained diffusion models
in downstream tasks has gained increasing attention. They have been used in
few-shot image classification [62], semantic segmentation [63, 64] and object
discovery [65]. Karazija _et al_.[64] leverage a diffusion-based generative
model to produce a set of feature prototypes, which can then be used in a
nearest-neighbour lookup scheme to segment images. In our method, we also use
a text-conditioned diffusion model (i.e., Stable Diffusion) to construct
visual class prototypes. We show that these prototypes can be used to find
class-relevant patches for the task of class-agnostic counting.

Figure 3:  Overview of the proposed method. We first obtain a class prototype
for the given class name (e.g. grape) in a pre-trained feature space. Then
given an input query image, we generate a set of object proposals with a pre-
trained RPN and crop the corresponding image patches. We extract the feature
embedding for each patch and select the patches whose embeddings are the
nearest neighbors of the class prototype as class-relevant patches. Then for
each selected class-relevant patch, we use a pre-trained exemplar-based
counting model to obtain the intermediate feature maps. Our proposed error
predictor then takes the feature maps as input and predicts the counting error
(here we use normalized counting errors). We select the patches with the
smallest predicted errors as the final exemplar patches and use them for
counting.

##  3 Method

Figure 3 summarizes our proposed method. We first construct a class prototype
for the given class name in a pre-trained feature space. Then given an input
query image, we generate a set of object proposals with a pre-trained RPN and
crop the corresponding image patches. We extract the feature embedding for
each patch and select the patches whose embeddings are the nearest neighbors
of the class prototype as class-relevant patches. We further use an error
predictor to select the patches with the smallest predicted errors as the
final exemplars for counting. We use the selected exemplars in an exemplar-
based object counter to infer the object counts. For the rest of the paper, we
denote this exemplar-based counter as the “base counting model”. We will first
describe how we train this base counting model and then present the details of
our patch selection method.

###  3.1 Training Base Counting Model

We train our base counting model using abundant training images with
annotations. Similar to previous works [5, 6], the base counting model uses
the input image and the exemplars to obtain a density map for object counting.
The model consists of a feature extractor $F$ and a counter $C$. Given a query
image $I$ and an exemplar $B$ of an arbitrary class $c$, we input $I$ and $B$
to the feature extractor to obtain the corresponding output, denoted as $F(I)$
and $F(B)$ respectively. $F(I)$ is a feature map of size $d*h_{I}*w_{I}$ and
$F(B)$ is a feature map of size $d*h_{B}*w_{B}$. We further perform global
average pooling on $F(B)$ to form a feature vector $b$ of $d$ dimensions.

After feature extraction, we obtain the similarity map $S$ by correlating the
exemplar feature vector $b$ with the image feature map $F(I)$. Specifically,
if $w_{ij}=F_{ij}(I)$ is the channel feature at spatial position $(i,j)$, $S$
can be computed by:

$S_{ij}(I,B)=w_{ij}^{T}b.$ (1)

In the case where $n$ exemplars are given, we use Eq. 1 to calculate $n$
similarity maps, and the final similarity map is the average of these $n$
similarity maps.

We then concatenate the image feature map $F(I)$ with the similarity map $S$,
and input them into the counter $C$ to predict a density map $D$. The final
predicted count ${N}$ is obtained by summing over the predicted density map
${D}$:

${N}=\sum_{i,j}D_{(i,j)},\vspace{-2mm}$ (2)

where ${D}_{(i,j)}$ denotes the density value for pixel $(i,j)$. The
supervision signal for training the counting model is the $L_{2}$ loss between
the predicted density map and the ground truth density map:

$L_{\textnormal{count}}=\|D(I,B)-D^{*}(I)\|_{2}^{2},$ (3)

where $D^{*}$ denotes the ground truth density map.

###  3.2 Zero-shot Object Counting

In this section, we describe how we count objects of any unseen category given
only the class name without access to any exemplar. Our strategy is to select
a few patches in the image that can be used as exemplars for the base counting
model. These patches are selected such that: 1) they contain the objects that
we are counting and 2) they benefit the counting model, i.e., lead to small
counting errors.

####  3.2.1 Selecting Class-relevant Patches

To select patches that contain the objects of interest, we first generate a
class prototype based on the given class name. The class prototype can be
considered as a class center representing the patch-level feature distribution
of the corresponding class in an embedding space. Then we use the generated
class prototype to select the class-relevant patches from a set of object
patches cropped from the testing image.

Specifically, we introduce two ways of generating prototypes, i.e., generating
semantics prototypes using conditional VAE and generating visual prototypes
using samples from a latent text-to-image diffusion model, i.e., Stable
Diffusion.

VAE-based prototype generation. To generate class prototypes, we train a
conditional VAE model to generate patch-level visual features for an arbitrary
class based on the semantic embedding of the class. This strategy is inspired
by previous zero-shot learning approaches [53, 54]. The semantic embedding is
obtained from a pre-trained text-vision model [12] given the corresponding
class name. Specifically, we train a VAE model to reconstruct deep features
extracted from a pre-trained ImageNet model. The VAE is composed of an Encoder
$E$, which maps a visual feature $x$ to a latent code $z$, and a decoder $G$
which reconstructs $x$ from $z$. Both $E$ and $G$ are conditioned on the
semantic embedding $a$ .The loss function for training this VAE for an input
feature $x$ can be defined as:

$\displaystyle L_{V}(x)=\textnormal{KL}\left(q(z|x,a)||p(z|a)\right)$ (4)
$\displaystyle-\textnormal{E}_{q(z|x,a)}[\textnormal{log }p(x|z,a)].$

The first term is the Kullback-Leibler divergence between the VAE posterior
$q(z|x,a)$ and a prior distribution $p(z|a)$. The second term is the decoder’s
reconstruction error. $q(z|x,a)$ is modeled as $E(x,a)$ and $p(x|z,a)$ is
equal to $G(z,a)$. The prior distribution is assumed to be $\mathcal{N}(0,I)$
for all classes.

We can use the trained VAE to generate the semantics prototype for an
arbitrary target class for counting. Specifically, given the target class name
$y$, we first generate a set of features by inputting the respective semantic
vector $a^{y}$ and a noise vector $z$ to the decoder $G$:

$\mathbb{G}^{y}=\\{\hat{x}|\hat{x}=G(z,y),z\sim\mathcal{N}(0,I)\\}.$ (5)

The class prototype $\textnormal{p}^{y}$ is computed by taking the mean of all
the features generated by VAE:

$\textnormal{p}^{y}=\frac{1}{|\mathbb{G}^{y}|}{\sum}_{\hat{x}\in\mathbb{G}^{y}}{\hat{x}}$
(6)

SD-based prototype generation. In addition to VAE-based approach for prototype
generation, we further leverage the recent advancements in text-to-image
models, i.e., Stable Diffusion, to construct class prototypes from SD-
generated images. Compared to classic VAE-based models, Stable Diffusion
enables more realistic and diverse sample generation, which allows handling
the intra-class variation among query objects more effectively.

Specifically, given the target class name for counting, we first use a pre-
trained Stable Diffusion model to generate a set of images with the class name
as prompt. We observe that the SD-generated images often contain multiple
object instances in various contexts and backgrounds. However, our goal is to
obtain a few representative object crops of the target class that can be used
to construct reference prototypes. In particular, given a query image, we aim
to find a few diffusion-generated object crops that most resemble the target
objects in the query image. To do so, we first apply a pre-trained RPN to
predict object proposals on both the diffusion-generated images and the query
image. Then we compute the pairwise distance between the diffusion-generated
object embeddings and the query image’s object embeddings. We select the
top-$k$ diffusion-generated object embeddings with the nearest mean distance
over all query embeddings. We average these $k$ embeddings to construct the
visual class prototype. An example is shown in Figure 4 where the target
objects are red apples. We first obtain a set of object patches containing
various crops of apples and select from them a set of red apples to construct
the prototype.

Figure 4:  Prototype generation using Stable Diffusion.

Class-relevant patch selection. Using the class prototype, either generated
using VAE or Stable Diffusion, we can select the class-relevant patches across
the query image. Specifically, we first use a pre-trained RPN to predict
object proposals across the query image and extract their corresponding
ImageNet features $\\{f_{1},f_{2},...,f_{m}\\}$. To select the class-relevant
patches, we calculate the $L_{2}$ distance between the class prototype and the
patch embedding, namely $d_{i}=\|f_{i}-\text{p}^{y}\|_{2}$. Then we select the
patches whose embeddings are the nearest neighbors of the class prototype as
the class-relevant patches. Since the ImageNet feature space is highly
discriminative, i.e., features close to each other typically belong to the
same class, the selected patches are likely to contain the objects of the
target class.

####  3.2.2 Selecting Exemplars for Counting

Given a set of class-relevant patches and a pre-trained exemplar-based object
counter, we aim to select a few exemplars from these patches that are optimal
for counting. To do so, we introduce an error prediction network that predicts
the counting error of an arbitrary patch when the patch is used as the
exemplar. The counting error is calculated from the pre-trained counting
model. Specifically, to train this error predictor, given a query image
$\bar{I}$ and an arbitrary patch $\bar{B}$ cropped from $\bar{I}$, we first
use the base counting model to get the image feature map $F(\bar{I})$,
similarity map $\bar{S}$, and the final predicted density map $\bar{D}$. The
counting error of the base counting model can be written as:

$\epsilon=|\sum_{i,j}\bar{D}_{(i,j)}-\bar{N^{*}}|,\vspace{-2mm}$ (7)

where $\bar{N^{*}}$ denotes the ground truth object count in image $\bar{I}$.
$\epsilon$ can be used to measure the goodness of $\bar{B}$ as an exemplar for
$\bar{I}$, i.e., a small $\epsilon$ indicates that $\bar{B}$ is a suitable
exemplar for counting and vice versa.

The error predictor $R$ is trained to regress the counting error produced by
the base counting model. The input of $R$ is the channel-wise concatenation of
the image feature map $F(\bar{I})$ and the similarity map $\bar{S}$. The
training objective is the minimization of the mean squared error between the
output of the predictor $R(F(\bar{I}),\bar{S})$ and the actual counting error
produced by the base counting model $\epsilon$.

After the error predictor is trained, we can use it to select the optimal
patches for counting. The candidates for selection here are the class-relevant
patches selected by the class prototype in the previous step. For each
candidate patch, we use the trained error predictor to infer the counting
error when it is being used as the exemplar. The final selected patches for
counting are the patches that yield the top-$s$ smallest counting errors.

####  3.2.3 Using the Selected Patches as Exemplars

Using the error predictor, we predict the error for each candidate patch and
select the patches that lead to the smallest counting errors. The selected
patches can then be used as exemplars for the base counting model to get the
density map and the final count. We also conduct experiments to show that
these selected patches can serve as exemplars for other exemplar-based
counting models to achieve exemplar-free class-agnostic counting.

##  4 Experiments

###  4.1 Implementation Details

Network Architecture. For the base counting model, we use ResNet-50 as the
backbone of the feature extractor, initialized with the weights of a pre-
trained ImageNet model. The backbone outputs feature maps of $1024$ channels.
For each query image, the number of channels is reduced to $256$ using an
$1\times 1$ convolution. For each exemplar, the feature maps are first
processed with global average pooling and then linearly mapped to obtain a
$256$-d feature vector. The counter consists of $5$ convolutional and bilinear
upsampling layers to regress a density map of the same size as the query
image. For the feature generation model, both the encoder and the decoder are
two-layer fully-connected (FC) networks with 4096 hidden units. LeakyReLU and
ReLU are the non-linear activation functions in the hidden and output layers,
respectively. The dimensions of the latent space and the semantic embeddings
are both set to be $512$. The error predictor composes of $5$ convolutional
and bilinear upsampling layers, followed by a linear layer to output the
counting error.

Dataset. We use the FSC-147 dataset [5] to train the base counting model and
the error predictor. FSC-147 is the first large-scale dataset for class-
agnostic counting. It includes $6135$ images from $147$ categories varying
from animals, kitchen utensils, to vehicles. The categories in the training,
validation, and test sets do not overlap. The feature generation model is
trained using ImageNet features extracted from MS-COCO objects.

Training Details. Both the base counting model and the error predictor are
trained using the AdamW optimizer with a fixed learning rate of $10^{-5}$. The
base counting model is trained for $300$ epochs with a batch size of $8$. We
resize the input query image to a fixed height of $384$, and the width is
adjusted accordingly to preserve the aspect ratio of the original image.
Exemplars are resized to $128\times 128$ before being input into the feature
extractor. To select the class-relevant patches, we use the Region Proposal
Network of Faster RCNN pre-trained on MS-COCO dataset to generate $100$ object
proposals per image. The feature generation model is trained using the Adam
optimizer and the learning rate is set to be $10^{-4}$. With the VAE-based
generator, the semantic embeddings are extracted from CLIP [12]. For visual
prototype generation, we use stable-diffusion-v1-4 [13] pre-trained on the
laion dataset [66]. The generated images are in the size of $512$ × $512$. For
each generated image, we take the top-$5$ RPN proposals with the highest
objectness scores. We combine all the proposals from diffusion-generated
images and extract their embeddings to do similarity matching with the object
embeddings from the query image. We select the top-$5$ embeddings with nearest
distances over query embeddings and compute their average to obtain the class
prototype. The final selected patches are those that yield the top-$3$
smallest counting errors predicted by the error predictor.

Figure 5: Qualitative results on the FSC-147 dataset. We show the counting
exemplars and the corresponding density maps of ground truth boxes, randomly
selected patches, and our selected patches respectively. Predicted counting
results are shown at the top-right corner. Our method accurately identifies
suitable patches for counting and the predicted density maps are close to the
ground truth density maps.

###  4.2 Evaluation Metrics

We use Mean Average Error (MAE) and Root Mean Squared Error (RMSE) to measure
the performance of different object counters. Besides, we follow [36] to
report the Normalized Relative Error (NAE) and Squared Relative Error (SRE).
In particular, MAE = $\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y_{i}}|$; RMSE =
$\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y_{i}})^{2}}$; NAE =
$\frac{1}{n}\sum_{i=1}^{n}\frac{|y_{i}-\hat{y_{i}}|}{y_{i}}$; SRE =
$\sqrt{\frac{1}{n}\sum_{i=1}^{n}\frac{(y_{i}-\hat{y_{i}})^{2}}{y_{i}}}$ where
$n$ is the number of test images, and $y_{i}$ and $\hat{y_{i}}$ are the ground
truth and the predicted number of objects for image $i$ respectively. Compared
with the absolute errors MAE and RMSE, the relative errors NAE and SRE better
reflect the practical usage of visual counting [36].

###  4.3 Comparing Methods

We compare our method with the previous works on class-agnostic counting,
which can be categorized into exemplar-based counting methods and reference-
less counting methods. Exemplar-based methods include FamNet (Few-shot
adaptation and matching Network [5]), BMNet (Bilinear Matching Network [6]),
CounTR (Counting TRansformer [37]) and SAFECount (Similarity-Aware Feature
Enhancement block for object Counting [38]). These methods require a few
human-annotated exemplars as inputs. Reference-less methods, i.e., RepRPN [8]
and CounTR [37], do not require annotated boxes at test time. Nevertheless,
the class of interest can not be specified, which makes them only suitable for
counting images with a single dominant object class. Our proposed zero-shot
counting, is a new setup which allows the user to specify what to count by
simply providing the class name without any exemplar. We also make exemplar-
based methods work in the exemplar-free manner by replacing the human-provided
exemplars with the exemplars generated by a pre-trained object detector.
Specifically, we use the RPN of Faster RCNN pre-trained on MS-COCO dataset and
select the top-$3$ proposals with the highest objectness score as the
exemplars.

Exemplars | Method | Val Set | Test Set
---|---|---|---
MAE | RMSE | NAE | SRE | MAE | RMSE | NAE | SRE
Exemplar-based | FamNet+ [5] | 23.75 | 69.07 | 0.52 | 4.25 | 22.08 | 99.54 | 0.44 | 6.45
BMNet [6] | 19.06 | 67.95 | 0.26 | 4.39 | 16.71 | 103.31 | 0.26 | 3.32
BMNet+ [6] | 15.74 | 58.53 | 0.27 | 6.57 | 14.62 | 91.83 | 0.25 | 2.74
CounTR [37] | 13.13 | 49.83 | 0.24 | 0.45 | 11.95 | 91.23 | 0.23 | 1.72
SAFECount [38] | 14.46 | 51.88 | 0.26 | 0.91 | 13.58 | 91.31 | 0.25 | 1.66
Ours | 18.55 | 61.12 | 0.30 | 3.18 | 20.68 | 109.14 | 0.36 | 7.63
Reference-less | RepRPN [8] | 30.40 | 98.73 | - | - | 27.45 | 129.69 | - | -
CounTR [37] | 17.40 | 70.33 | 0.34 | 1.64 | 14.12 | 108.01 | 0.29 | 1.93
FamNet+ [5] \+ RPN | 42.85 | 121.59 | 0.75 | 6.94 | 42.70 | 146.08 | 0.74 | 7.14
BMNet [6] \+ RPN | 37.26 | 108.54 | 0.42 | 5.43 | 37.22 | 143.13 | 0.41 | 5.31
BMNet+ [6] \+ RPN | 35.15 | 106.07 | 0.41 | 5.28 | 34.52 | 132.64 | 0.39 | 5.26
SAFECount [38] \+ RPN | 34.98 | 107.46 | 0.38 | 5.22 | 33.89 | 139.92 | 0.39 | 5.34
| Ours + RPN | 32.19 | 99.21 | 0.38 | 4.80 | 29.25 | 130.65 | 0.35 | 4.35
Zero-Shot | Patch-Sel (VAE) | 27.47 | 90.85 | 0.37 | 4.52 | 23.14 | 114.40 | 0.34 | 3.95
Patch-Sel (SD) | 26.30 | 88.80 | 0.34 | 4.27 | 21.53 | 113.28 | 0.31 | 3.61

TABLE I:  Quantitative comparisons on the FSC-147 dataset. “RPN” denotes using
the top-3 RPN proposals with the highest objectness scores as exemplars.
“Patch-Sel (VAE)” and “Patch-Sel (SD)” denotes using patch selection method
with VAE-generated prototypes and SD-generated prototypes respectively.
Figure 6:  Qualitative comparison with top-$3$ exemplars from RPN. Our
proposed method can select patches suitable for counting while RPN-selected
patches contain non-relevant objects or multiple object instances.

###  4.4 Results

Quantitative results. As shown in Table I, the performance of all exemplar-
based counting methods drops significantly when replacing human-annotated
exemplars with RPN generated proposals. BMNet+ [6], for example, shows an
$19.90$ error increase w.r.t. the test MAE and a $40.81$ increase w.r.t. the
test RMSE. In comparison, the performance gap is much smaller when using our
selected patches as exemplars. Our patch selection method with VAE-generated
prototype obtains $27.47$ MAE on the validation set and $23.14$ MAE on the
test set. By using the SD-generated class prototype, the error rates can be
further reduced, achieving $26.30$ MAE on the validation set and $21.53$ MAE
on the test set. Noticeably, compared with the human-annotated exemplars, the
NAE and the SRE on the test set are even reduced when using our selected
patches.

Qualitative analysis. In Figure 5, we present a few input images, the image
patches selected by our method, and the corresponding density maps. Our method
effectively identifies the patches that are suitable for object counting. The
density maps produced by our selected patches are meaningful and close to the
density maps produced by human-annotated patches. The counting model with
random image patches as exemplars, in comparison, fails to output meaningful
density maps and infers incorrect object counts.

In Figure 6, we visualize some images from the FSC-147 dataset and the
corresponding patches selected by RPN and our method respectively. The RPN-
selected patches are the top-$3$ proposals with the highest objectness scores.
As can be seen from the figure, the patches selected by RPN may contain
objects not relevant to the provided class name or contain multiple object
instances. These patches are not suitable to be used as counting exemplars and
will lead to inaccurate counting results. This suggests that choosing counting
exemplars based on objectness score is not reliable. In comparison, our
proposed method can accurately localize image patches according to the given
class name. These selected patches can then be used as counting exemplars and
yield meaningul density maps and reasonable counting results.

Figure 7: Qualitative ablation analysis. All the $10$ selected class-relevant
patches exhibit some class-specific attributes. They are ranked by the
predicted counting errors and the final selected patches with the smallest
errors are framed in green.

##  5 Analysis

###  5.1 Ablation Studies

Our proposed patch selection method consists of two steps: the selection of
class-relevant patches via a generated class prototype and the selection of
the optimal patches via an error predictor. We analyze the contribution of
each step quantitatively and qualitatively. Quantitative results are in Table
II. We first evaluate the performance of a simple RPN-based baseline, i.e.
using the top-3 RPN proposals with the highest objectness scores as exemplars
without any selection step. This baseline method has an error rate of $32.19$
on the validation MAE and $29.25$ on the test MAE. As shown in Table II, using
the class prototype generated by VAE to select class-relevant patches reduces
the error rate by $2.34$ and $4.87$ on the validation and test set w.r.t. MAE,
respectively. Using the class prototype generated by Stable Diffusion reduces
the MAE by $4.43$ and $7.26$ on the validation and test set respectively.
Applying the error predictor can improve the baseline performance by $4.63$ on
the validation MAE and $6.46$ on the test MAE. Finally, using the Stable
Diffusion prototype and error predictor together further boosts performance,
achieving $26.30$ on the validation MAE and $21.53$ on the test MAE.

We provide further qualitative analysis by visualizing the selected patches.
As shown in Figure 7, for each input query image, we show $10$ class-relevant
patches selected using the class prototype generated with Stable Diffusion,
ranked by their predicted counting error (from low to high). All the $10$
selected class-relevant patches exhibit some class specific features. However,
not all these patches are suitable to be used as counting exemplars, i.e.,
some patches only contain parts of the object, and some patches contain some
background. By further applying our proposed error predictor, we can identify
the most suitable patches with the smallest predicted counting errors. Both
the quantitative comparison in Table I and qualitative results presented in
Figure 7 validate the effectiveness of our patch selection method.

Prototype | Predictor | Val Set | Test Set
---|---|---|---
MAE | RMSE | NAE | SRE | MAE | RMSE | NAE | SRE
- | - | 32.19 | 99.21 | 0.38 | 4.80 | 29.25 | 130.65 | 0.35 | 4.35
✓(VAE) | - | 29.85 | 97.63 | 0.42 | 5.20 | 24.38 | 115.04 | 0.36 | 4.18
✓(SD) | - | 27.76 | 97.06 | 0.35 | 4.43 | 21.99 | 113.31 | 0.33 | 3.77
- | ✓ | 27.56 | 88.57 | 0.41 | 4.54 | 22.79 | 127.09 | 0.36 | 3.98
✓(VAE) | ✓ | 27.47 | 90.85 | 0.37 | 4.52 | 23.14 | 114.40 | 0.34 | 3.95
✓(SD) | ✓ | 26.30 | 88.80 | 0.34 | 4.27 | 21.53 | 113.28 | 0.31 | 3.61

TABLE II:  Ablation study on each component’s contribution to the final
results. We show the effectiveness of the two steps of our framework:
selecting class-relevant patches via a generated class prototype and selecting
optimal patches via an error predictor.

Baseline | Exemplars | Val Set | Test Set
---|---|---|---
MAE | RMSE | NAE | SRE | MAE | RMSE | NAE | SRE
FamNet+ | RPN | 42.85 | 121.59 | 0.75 | 6.94 | 42.70 | 146.08 | 0.74 | 7.14
Patch-Sel (VAE) | 39.51 | 101.70 | 0.84 | 6.80 | 39.91 | 143.04 | 0.74 | 6.77
Patch-Sel (SD) | 34.60 | 96.76 | 0.63 | 5.71 | 32.71 | 139.93 | 0.55 | 5.47
BMNet | RPN | 37.26 | 108.54 | 0.42 | 5.43 | 37.22 | 143.13 | 0.41 | 5.31
Patch-Sel (VAE) | 27.71 | 93.98 | 0.36 | 4.55 | 24.45 | 130.42 | 0.33 | 4.09
Patch-Sel (SD) | 26.53 | 91.55 | 0.32 | 4.36 | 22.27 | 129.67 | 0.28 | 3.85
BMNet+ | RPN | 35.51 | 106.07 | 0.41 | 5.28 | 34.52 | 132.64 | 0.39 | 5.26
Patch-Sel (VAE) | 26.89 | 92.37 | 0.35 | 4.56 | 23.14 | 114.40 | 0.34 | 3.95
Patch-Sel (SD) | 25.91 | 90.62 | 0.33 | 4.40 | 19.45 | 109.82 | 0.27 | 3.53
SAFECount | RPN | 34.98 | 107.46 | 0.38 | 5.22 | 33.89 | 139.92 | 0.39 | 5.34
Patch-Sel (VAE) | 28.34 | 94.22 | 0.41 | 4.65 | 23.60 | 110.95 | 0.40 | 4.26
Patch-Sel (SD) | 26.85 | 91.09 | 0.33 | 4.30 | 21.44 | 115.30 | 0.30 | 5.74

TABLE III:  Using our selected patches as exemplars for other exemplar-based
class-agnostic counting methods (FamNet+, BMNet, BMNet+ and SAFECount) on
FSC-147 dataset. Our patch selection method generalizes well to other
exemplar-based counting methods.

###  5.2 Generalization to Exemplar-based Methods

Our proposed method can be considered as a general patch selection method that
is applicable to other visual counters to achieve zero-shot object counting.
To verify that, we use our selected patches as the exemplars for four other
different exemplar-based methods: FamNet+ [5], BMNet [6], BMNet+ [6] and
SAFECount [38]. Table III shows the results on the FSC-147 dataset. The
baseline uses top-3 RPN proposals with the highest objectness scores as
exemplars for the pre-trained exemplar-based counter. Our patch selection
method with VAE-generated class prototype reduces the error rates by a large
margin for all exemplar-based methods. For example, the MAE for BMNet+ [6]
reduces from $35.51$ to $26.89$ on the validation set and from $34.52$ to
$23.14$ on the test set. By using our patch selection method with the SD-
generated class prototype, the error rates are further reduced for most cases,
e.g., we observe for FamNet+ [5], there is an error reduction of $12.4\%$
w.r.t. the validation MAE and $18.0\%$ w.r.t. the test MAE. The consistent
performance improvements validate that our patch selection method generalizes
well to other exemplar-based counting methods.

###  5.3 Multi-class Object Counting

Our method can count instances of a specific class given the class name, which
is particularly useful when there are multiple classes in the same image. In
this section, we show some visualization results in this multi-class scenario.
As shown in Figure 8, our method selects patches according to the given class
name and counts instances from that specific class in the input image.
Correspondingly, the heatmap highlights the image regions that are most
relevant to the specified class. Here the heatmaps are obtained by correlating
the exemplar feature vector with the image feature map in a pre-trained
ImageNet feature space. Note that we mask out the image region where the
activation value in the heatmap is below a threshold when counting the objects
of interests. We also show the patches selected using another exemplar-free
counting method, RepRPN [8]. The class of RepRPN selected patches can not be
explicitly specified. It simply selects patches from the class with the
highest number of instances in the image according to the repetition score.

(a) Broccoli and carrots (b) Bananas and strawberries (c) Green Beans and
tomatoes

Figure 8: Visualization results of our method on some multi-class examples.
Our method selects patches according to the given class name and the
corresponding heatmap highlights the relevant areas.

###  5.4 Qualitative Comparison between SD-generated Prototypes and VAE-
generated Prototypes

In this section, we provide qualitative comparison between patches selected
via SD-generated prototypes and VAE-generated prototypes. As shown in Figure
9, we present a few input images and the corresponding patches selected by
VAE-generated prototypes and SD-generated prototypes. Although the patches
selected by VAE-generated prototypes generally contain the objects of
interest, they miss parts of the objects in some cases (e.g., the second patch
of grape), or contain multiple object instances within one patch (e.g., the
second patch of strawberry). In comparison, the patches selected by SD-
generated prototypes are generally better exemplars for counting, i.e., one
patch mostly contains a single complete object instance.

Figure 9:  Visualization of the patches selected by VAE-generated prototypes
and SD-generated prototypes. Patches selected by SD-generated prototypes are
of higher quality.

(a) Grapes (b) Eggs

(c) Apples

Figure 10: Visualization of the diffusion-generated patches for prototype
generation and the corresponding selected patches from query images. Our
method constructs prototypes according to the query images and selects the
patches that most resemble the target objects.

###  5.5 Analysis on SD-generated Prototypes

Qualitative Visualization. To generate visual class prototypes, we first use a
pre-trained Stable Diffusion to generate a set of object patches for the class
of interest. Then given a query image, we select the generated patches that
most resemble the target objects in the query image. We compute the average
feature embeddings of the selected generated patches to construct the
prototype. In Figure 10, we demonstrate this process for three different
categories, i.e., grapes, eggs, and apples. For each category, we show how we
select different patches to construct the prototypes for the given query
image. As can be seen in the first column, the set of RPN proposals extracted
from SD-generated images exhibit rich variations. Among these proposals, our
method selects those that are most relevant to the testing images for
constructing prototypes. For example, in the last row of Figure 10 (c), only
apples with mixed colors are selected to match the objects’ colors in the
testing image. Compared with VAE-generated prototypes which remain the same
for all query images, SD-generated prototypes can better handle the variations
of query images and select more accurate counting exemplars.

Number of Patches for Prototype Generation. In our main experiments, we select
the top-$5$ SD-generated patches with the nearest mean distance over query
patches, and compute their average features to construct prototypes. In this
section, we conduct an ablation study on how the number of patches selected
for prototype generation affects the counting performance. Specifically, we
select top-$5$, top-$25$, top-$50$ and all patches to construct class
prototypes and use them to select exemplars. Results are summarized in Table
IV. We observe that the performance drops on both the validation set and test
set as the number of selected patches increases. The counting errors are
highest when using all generated patches to construct prototypes. In this
case, the same class prototype is applied for all images of this class, which
is not optimal for counting objects with large intra-class diversity. Our
method, in comparison, selects the most similar patches based on the query
image, which leads to more accurate prototypes.

Prototype | Val Set | Test Set
---|---|---
Patches | MAE | RMSE | MAE | RMSE
top-5 | 27.76 | 97.06 | 21.99 | 113.31
top-25 | 28.03 | 97.87 | 22.12 | 113.71
top-50 | 28.33 | 98.50 | 22.34 | 113.88
all | 30.13 | 101.70 | 23.37 | 116.04

TABLE IV:  Ablation study on the number of diffusion-generated patches for
prototype generation.

##  6 Conclusion

We propose a new task, zero-shot object counting, to count instances of a
specific class given only the class name without access to any exemplars. To
address this, we developed a two-step approach that accurately localizes the
optimal patches to be used as counting exemplars. We leverage language-vision
models to construct class prototypes via two approaches: VAE-based apporoach
and diffusion-based approach. Through these two approaches, we present a
comprehensive study of prototype construction at both category and image
levels. In the context of our specific task, the diffusion-based image
prototype has notably outperformed the category-level prototype constructed
via VAE, thanks to its ability to customize prototypes to match object
appearances in each image. More generally, our approach of employing language-
vision models for zero-shot recognition is applicable in various tasks. In
scenarios where data of large-scale generative models do not exist such as
medical imaging, or remote sensing, VAE can be a useful choice.

We show that the prototypes can be used to select the patches containing
objects of interests. Furthermore, we introduce an error prediction model to
select those patches with the smallest predicted errors as the final exemplars
for counting. Extensive results demonstrate the effectiveness of our method.
We also conduct experiments to show that our selected patches can be
incorporated into other exemplar-based counting methods to achieve exemplar-
free counting.

## References

  * [1] D. B. Sam, A. Agarwalla, J. Joseph, V. A. Sindagi, R. V. Babu, and V. M. Patel, “Completely self-supervised crowd counting via distribution matching,” in _ECCV_ , 2022.
  * [2] T. N. Mundhenk, G. Konjevod, W. A. Sakla, and K. Boakye, “A large contextual dataset for classification, detection and counting of cars with deep learning,” in _ECCV_ , 2016.
  * [3] C. Arteta, V. S. Lempitsky, and A. Zisserman, “Counting in the wild,” in _ECCV_ , 2016.
  * [4] W. Xie, J. A. Noble, and A. Zisserman, “Microscopy cell counting and detection with fully convolutional regression networks,” _Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization_, vol. 6, 2018\.
  * [5] V. Ranjan, U. Sharma, T. Nguyen, and M. Hoai, “Learning to count everything,” in _CVPR_ , 2021.
  * [6] M. Shi, H. Lu, C. Feng, C. Liu, and Z. Cao, “Represent, compare, and learn: A similarity-aware framework for class-agnostic counting,” in _CVPR_ , 2022\.
  * [7] E. Lu, W. Xie, and A. Zisserman, “Class-agnostic counting,” in _ACCV_ , 2018\.
  * [8] V. Ranjan and M. Hoai, “Exemplar free class agnostic counting,” in _ACCV_ , 2022.
  * [9] Y. Zheng, J. Wu, Y. Qin, F. Zhang, and L. Cui, “Zero-shot instance segmentation,” in _CVPR_ , 2021.
  * [10] A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran, “Zero-shot object detection,” in _ECCV_ , 2018.
  * [11] J. Xu, H. Le, V. Nguyen, V. Ranjan, and D. Samaras, “Zero-shot object counting,” in _CVPR_ , 2023.
  * [12] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in _ICML_ , 2021.
  * [13] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _CVPR_ , 2022.
  * [14] D. Lian, J. Li, J. Zheng, W. Luo, and S. Gao, “Density map regression guided detection network for rgb-d crowd counting and localization,” _CVPR_ , 2019\.
  * [15] A. Zhang, L. Yue, J. Shen, F. Zhu, X. Zhen, X. Cao, and L. Shao, “Attentional neural fields for crowd counting,” in _ICCV_ , 2019.
  * [16] Q. Wang, J. Gao, W. Lin, and X. Li, “Nwpu-crowd: A large-scale benchmark for crowd counting and localization,” _pami_ , vol. 43, 2021.
  * [17] V. A. Sindagi, R. Yasarla, and V. M. Patel, “Pushing the frontiers of unconstrained crowd counting: New dataset and benchmark method,” in _ICCV_ , 2019.
  * [18] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. A. Al-Maadeed, N. M. Rajpoot, and M. Shah, “Composition loss for counting, density map estimation and localization in dense crowds,” in _ECCV_ , 2018.
  * [19] S. Abousamra, M. Hoai, D. Samaras, and C. Chen, “Localization in the crowd with topological constraints,” in _AAAI_ , 2021.
  * [20] Q. Zhang and A. Chan, “Calibration-free multi-view crowd counting,” in _ECCV_ , 2022.
  * [21] H. Xiong and A. Yao, “Discrete-constrained regression for local counting models,” in _ECCV_ , 2022.
  * [22] W. Liu, N. Durasov, and P. Fua, “Leveraging self-supervision for cross-domain crowd counting,” in _CVPR_ , 2022.
  * [23] J. Wan, Z. Liu, and A. B. Chan, “A generalized loss function for crowd counting and localization,” in _CVPR_ , 2021.
  * [24] M.-R. Hsieh, Y.-L. Lin, and W. H. Hsu, “Drone-based object counting by spatially regularized regional proposal network,” in _ICCV_ , 2017.
  * [25] P. Chattopadhyay, R. Vedantam, R. R. Selvaraju, D. Batra, and D. Parikh, “Counting everyday objects in everyday scenes,” _CVPR_ , 2017.
  * [26] I. H. Laradji, N. Rostamzadeh, P. H. O. Pinheiro, D. Vázquez, and M. W. Schmidt, “Where are the blobs: Counting by localization with point supervision,” in _ECCV_ , 2018.
  * [27] C. Zhang, H. Li, X. Wang, and X. Yang, “Cross-scene crowd counting via deep convolutional neural networks,” in _CVPR_ , 2015.
  * [28] H. Cholakkal, G. Sun, F. S. Khan, and L. Shao, “Object counting and instance segmentation with image-level supervision,” in _CVPR_ , 2019.
  * [29] H. Cholakkal, G. Sun, S. H. Khan, F. S. Khan, L. Shao, and L. V. Gool, “Towards partial supervision for generic object counting in natural scenes,” in _TPAMI_ , vol. 44, 2022.
  * [30] B. Wang, H. Liu, D. Samaras, and M. H. Nguyen, “Distribution matching for crowd counting,” in _NeurIPS_ , 2020.
  * [31] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network.” in _CVPR_ , 2016.
  * [32] A. B. Chan, Z.-S. J. Liang, and N. Vasconcelos, “Privacy preserving crowd monitoring: Counting people without people models or tracking,” in _CVPR_ , 2008.
  * [33] W. Liu, M. Salzmann, and P. V. Fua, “Context-aware crowd counting,” in _CVPR_ , 2019.
  * [34] S. Yang, H.-T. Su, W. H. Hsu, and W.-C. Chen, “Class-agnostic few-shot object counting,” in _WACV_ , 2021.
  * [35] S. Gong, S. Zhang, J. Yang, D. Dai, and B. Schiele, “Class-agnostic object counting robust to intraclass diversity,” in _ECCV_ , 2022.
  * [36] T. Nguyen, C. Pham, K. Nguyen, and M. Hoai, “Few-shot object counting and detection,” in _ECCV_ , 2022.
  * [37] C. Liu, Y. Zhong, A. Zisserman, and W. Xie, “Countr: Transformer-based generalised visual counting,” in _BMVC_ , 2022.
  * [38] Z. You, K. Yang, W. Luo, X. Lu, L. Cui, and X. Le, “Few-shot object counting with similarity-aware feature enhancement,” in _WACV_ , 2023.
  * [39] C. Arteta, V. S. Lempitsky, J. A. Noble, and A. Zisserman, “Interactive object counting,” in _ECCV_ , 2014.
  * [40] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _TPAMI_ , 2015.
  * [41] L. Chen, H. Zhang, J. Xiao, W. Liu, and S.-F. Chang, “Zero-shot visual recognition using semantics-preserving adversarial embedding networks,” in _CVPR_ , 2018.
  * [42] D. Jayaraman and K. Grauman, “Zero-shot recognition with unreliable attributes,” in _NIPS_ , 2014.
  * [43] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” in _NIPS_ , 2013.
  * [44] M. Rezaei and M. Shahidi, “Zero-shot learning and its applications from autonomous vehicles to covid-19 diagnosis: A review,” in _Intelligence-Based Medicine_ , vol. 3, 2020.
  * [45] B. Romera-Paredes and P. H. S. Torr, “An embarrassingly simple approach to zero-shot learning,” in _ICML_ , 2015.
  * [46] Y. Atzmon and G. Chechik, “Adaptive confidence smoothing for generalized zero-shot learning,” in _CVPR_ , 2019.
  * [47] C. H. Lampert, H. Nickisch, and S. Harmeling, “Learning to detect unseen object classes by between-class attribute transfer,” _CVPR_ , 2009.
  * [48] Y. Xian, Z. Akata, G. Sharma, Q. N. Nguyen, M. Hein, and B. Schiele, “Latent embeddings for zero-shot classification,” in _CVPR_ , 2016.
  * [49] L. Zhang, T. Xiang, and S. Gong, “Learning a deep embedding model for zero-shot learning,” in _CVPR_ , 2017.
  * [50] J. Xu, H. Le, M. Huang, S. Athar, and D. Samaras, “Variational feature disentangling for fine-grained few-shot classification,” in _ICCV_ , 2021\.
  * [51] J. Xu and H. Le, “Generating representative samples for few-shot classification,” in _CVPR_ , 2022.
  * [52] H. Le, T. F. Y. Vicente, V. Nguyen, M. Hoai, and D. Samaras, “A+D Net: Training a shadow detector with adversarial shadow attenuation,” in _ECCV_ , 2018.
  * [53] Y. Xian, S. Sharma, B. Schiele, and Z. Akata, “F-vaegan-d2: A feature generating framework for any-shot learning,” in _CVPR_ , 2019.
  * [54] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” _pami_ , vol. 41, 2019.
  * [55] S. Narayan, A. Gupta, F. S. Khan, C. G. M. Snoek, and L. Shao, “Latent embedding feedback and discriminative features for zero-shot classification,” in _ECCV_ , 2020.
  * [56] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” in _ICML_ , 2017\.
  * [57] J. Ho, A. Jain, and P. Abbeel., “Denoising diffusion probabilistic models.” in _NeurIPS_ , 2020.
  * [58] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli., “Deep unsupervised learning using nonequilibrium thermodynamics.” in _ICML_ , 2015\.
  * [59] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole., “Score-based generative modeling through stochastic differential equations.” in _ICLR_ , 2021.
  * [60] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _ArXiv_ , vol. abs/2204.06125, 2022.
  * [61] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. G. Lopes, B. K. Ayan, and e. a. Tim Salimans, “Photorealistic text-to- image diffusion models with deep language understanding.” in _NeurIPS_ , 2022\.
  * [62] R. Zhang, X. Hu, B. Li, S. Huang, H. Deng, H. Li, Y. Qiao, and P. Gao., “Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners.” in _CVPR_ , 2023.
  * [63] W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, and C. Shen., “Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models.” in _ICCV_ , 2023.
  * [64] L. Karazija, I. Laina, A. Vedaldi, and C. Rupprecht, “Diffusion models for zero-shot open-vocabulary segmentation.” _ArXiv_ , vol. abs/2306.09316, 2023\.
  * [65] C. Ma, Y. Yang, C. Ju, F. Zhang, J. Liu, Y. Wang, Y. Zhang, and Y. Wang., “Diffusionseg: Adapting diffusion towards unsupervised object discovery.” _ArXiv_ , vol. abs/2303.09813, 2023.
  * [66] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev, “Laion-5b: An open large-scale dataset for training next generation image-text models,” _ArXiv_ , vol. abs/2210.08402, 2022.

|  Jingyi Xu received her B.S. degree in computer science from Nankai
University in 2019. She is currently working toward the PhD degree in computer
science with the Stony Brook University. Her research interests include
computer vision and deep learning, with a focus on representation learning and
few-shot learning.
---|---
|  Hieu Le received his B.S. degree in computer science from the Ho Chi Minh
City University of Science in 2012, and his Ph.D. degree in computer science
from Stony Brook University in 2020. He is currently a Postdoc at CVLab, EPFL.
His research interest includes illumination modeling, generative models, and
3D modelling.
---|---
|  Dimitris Samaras received the diploma degree in computer science and
engineering from the University of Patras, in 1992, the MSc degree from
Northeastern University, in 1994, and the PhD degree from the University of
Pennsylvania, in 2001. He is an associate professor of computer science with
Stony Brook University. He specializes in illumination modeling and estimation
for recognition and graphics, and biomedical image analysis. He is Program
Chair of CVPR 2022.
---|---