#  Long-range forces : atmospheric neutrino oscillation at a magnetized
detector

Abhijit Samanta111E-mail address<EMAIL_ADDRESS>Ramakrishna
Mission Vivekananda University, Belur Math, Howrah 711 202, India
and
222past address where the work was initiated.Harish-Chandra Research
Institute, Chhatnag Road, Jhusi, Allahabad 211 019, India

###### Abstract

Among the combinations $L_{e}-L_{\mu}$, $L_{e}-L_{\tau}$ and
$L_{\mu}-L_{\tau}$ any one can be gauged in anomaly free way with the standard
model gauge group. The masses of these gauge bosons can be so light that it
can induce long-range forces on the Earth due to the electrons in the Sun.
This type of forces can be constrained significantly from neutrino
oscillation. As the sign of the potential is opposite for neutrinos and
antineutrinos, a magnetized iron calorimeter detector (ICAL) would be able to
produce strong constraint on it. We have made conservative studies of these
long-range forces with atmospheric neutrinos at ICAL considering only the
muons of charge current interactions. We find stringent bounds on the
couplings $\alpha_{e\mu,e\tau}\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$<$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}1.65\times 10^{-53}$
at 3$\sigma$ CL with an exposure of 1 Mton$\cdot$yr if there is no such force.
For nonzero input values of the couplings we find that the potential
$V_{e\mu}$ opposes and $V_{e\tau}$ helps to discriminate the mass hierarchy.
However, both potentials help significantly to discriminate the octant of
$\theta_{23}$. The explanation of the anomaly in recent MINOS data (the
difference of $\Delta m^{2}_{32}$ for neutrinos and antineutrinos), using
long-range force originated from the mixing of the gauge boson $Z^{\prime}$ of
$L_{\mu}-L_{\tau}$ with the standard model gauge boson $Z$, can be tested at
ICAL at more than 5$\sigma$ CL. We have also discussed how to disentangle this
from the solution with CPT violation using the seasonal change of the distance
between the Earth and the Sun.

neutrino oscillation, atmospheric neutrino, long-range force

###### pacs:

14.60.Pq

##  I Introduction

The large hadron collider at CERN will probe the extensions of standard model
above the electroweak scale. On the other hand, some of the extensions below
the electroweak scale can be probed at the neutrino oscillation experiments.
The extensions below the electroweak scale introduce massless or nearly
massless gauge foot ; axion  or Higgs bosons singletm ; tripletm ; hall ;
Chang:1989xm  and they couple with matter very feebly and remain invisible.
These lead to the existence of new kind of forces either i) generating
deviations of gravitational law at short distances, or ii) predicting low mass
particles whose exchange will induce forces at long distances, generally
violating the equivalence principle Damour:1996xt ; fischbach . A number of
experiments have been searching for these new forces. The null results provide
bounds on particle physics models, gravitational physics, and even on
cosmological models fischbach ; Adelberger:2003zx ; Williams:1995nq .

The bounds on these couplings to baryon and/or lepton number Lee:1955vk  can
be obtained from the testing of equivalence principle Eotvos:1922pb  (the free
fall acceleration is same for all bodies independent of their chemical
content). In Okun:1995dn , the author has used this idea to establish a bound
on the strength of an hypothetical vectorial leptonic force and obtained the
bound on the “fine structure” constant $\alpha\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$<$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}10^{-49}.$ See
Dolgov:1999gk  for a review. A comparable limit also comes from lunar laser
ranging Williams:1995nq , which measures the differential acceleration of the
Earth and the Moon towards the Sun.

However, one can extend the SM with an additional $U(1)$ gauge symmetry
without introducing any anomaly for one of the lepton flavor combinations:
$L_{e}-L_{\mu}$, $L_{\mu}-L_{\tau}$, and $L_{e}-L_{\tau}$. The masses of the
gauge bosons can be so light that the induced forces may have terrestrial
range. Then the electrons inside the Sun can induce forces on the Earth
surface depending on the lightness of the gauge boson. These forces on the
Earth may also be from supernova neutrinos, galactic electrons depending on
its ranges. These couple only to electron (and neutrino) density inside a
massive object. As a result, the acceleration experienced by an object depends
on its leptonic content and mass; and thus violates equivalence principle.

The long-range (LR) forces can play role in neutrino oscillation. The
relatively stronger bounds than those from testing of the equivalence
principle have been obtained from solar, atmospheric and supernova data
Joshipura:2003jh ; Bandyopadhyay:2006uh ; GonzalezGarcia:2006vp ;
Grifols:1993rs ; Grifols:2003gy 333There are plenty relic neutrinos and
antineutrinos in the universe ($\sim 50/cm^{3}$), which may screen the
leptonic charges of the of celestial bodies. However, it has been analyzed in
detail and shown that the screening is impossible Blinnikov:1995kp . The long-
range forces due to galactic electrons, which can affect the galactic rotation
curves if the range $R_{LR}\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$>$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}R_{\rm gal}$ (where
$R_{\rm gal}$ is the distance from the galactic center $\sim$ 10 kpc), is also
now very tightly constrained Bandyopadhyay:2006uh . Finally, all these results
are consistent with the existing bounds on violation of equivalence principle.

The sign of this potential is opposite for neutrinos and anti-neutrinos and
hence can lead to apparent differences in neutrino and anti-neutrino
oscillation probabilities without introducing CP or CPT violation. For
instance, the recently found discrepancy in the survival probabilities of
$\nu_{\mu}$s and $\bar{\nu}_{\mu}$s in the MINOS experiment pvahle  has been
explained using the mixing of $Z^{\prime}$ boson of $L_{\mu}-L_{\tau}$
symmetry with the $Z$ boson of the SM model; and the required associated
parameters to explain this anomaly can be found in Heeck:2010pg .

We have studied the long-range forces with atmospheric neutrinos at the
magnetized iron calorimeter detector (ICAL) proposed at the India-based
neutrino Observatory (INO) Arumugam:2005nt , which can directly measure the
potential detecting separately $\nu_{\mu}$s and $\bar{\nu}_{\mu}$s. In this
article we have focussed on:

\- the bounds on the couplings of long-range forces,

\- how significantly long-range forces modify the sensitivity of the
measurement of oscillation parameters; particularly, the octant of 2-3 mixing
(sign of $\delta^{\rm oct}=\theta_{23}-45^{\circ}$) and the mass hierarchy
(sign of $\Delta m_{31}^{2}$),

\- test of the explanation of the MINOS data with long-range forces
Heeck:2010pg .

\- possibility to disentangle the effect of CPT and long-range forces 444The
advantages of atmospheric neutrinos to discriminate $CPT$ violation from $CP$
violation and nonstandard interactions have been discussed in Samanta:2010ce
..

We have also studied the changes of sensitivity to mass hierarchy and to
$\theta_{23}$-octant with true (input) $\theta_{23}$ values considering no
such forces (This was not studied in earlier works.).

The paper is organized as follows. We discuss the full three flavor neutrino
oscillation in matter with long-range potential in sec. II, the details of the
analysis method in sec. III, and the bounds on the couplings in Sec. IV. The
effects of the long-range potential on discrimination of mass hierarchy and on
$\theta_{23}$-octant is discussed in sec. V. Finally, the testing of the
explanation of the anomaly in MINOS data using long-range force is described
in Sec. VI. The discussion and conclusion are given in VII.

##  II Oscillation in presence of long-range potential

The electrons inside the Sun generate a potential at the Earth by
Joshipura:2003jh :

$V_{e\mu,e\tau}=\alpha_{e\mu,e\tau}\frac{N_{e}}{R_{ES}}\approx 1.3\times
10^{-11}{\rm eV}\left(\alpha_{e\mu,e\tau}/10^{-50}\right),$ (1)

where, $\alpha_{e\mu,e\tau}=g^{2}_{e\mu,e\tau}/{4\pi}$; $g_{e\mu,e\tau}$ are
the gauge couplings of $L_{e}-L_{\mu}$ and $L_{e}-L_{\tau}$ symmetries.
$N_{e}\approx 10^{57}$ is the number of electrons inside the Sun sun  and
$R_{ES}=7.6\times 10^{26}$ GeV-1, the distance between the Earth and the Sun.

In a three neutrino framework the neutrino flavor states
$|\nu_{\alpha}\rangle$, $\alpha=e,\mu,\tau$ can be expressed as linear
superpositions of the neutrino mass eigenstates $|\nu_{i}\rangle$, $i=1,2,3$
with masses $m_{i}$ :

$|\nu_{\alpha}\rangle=\sum_{i}U_{\alpha i}|\nu_{i}\rangle~{}.$ (2)

$U$ is the $3\times 3$ unitary matrix. The time evolution of the flavor states
is

$i\frac{\rm d}{\rm
dt}\left[\nu_{\alpha}\right]=\frac{1}{2E}UM_{\nu}^{2}U^{\dagger}\left[\nu_{\alpha}\right],$
(3)

where, $[\nu_{\alpha}]$ is the vector of flavor eigenstates, and
$\left[\nu_{\alpha}\right]^{T}=\left[|\nu_{e}\rangle,|\nu_{\mu}\rangle,|\nu_{\tau}\rangle\right]$.

The evolution equation in the presence of matter and long-range potential is

$\displaystyle i\frac{\rm d}{{\rm
d}t}\left[\nu_{\alpha}\right]=\frac{1}{2E}\left[UM_{\nu}^{2}U^{\dagger}+\left(\begin{array}[]{ccc}A&0&0\\\
0&0&0\\\ 0&0&0\end{array}\right)+V_{LR}\right]\left[\nu_{\alpha}\right],$ (7)

where,

$\displaystyle V_{LR}=\left(\begin{array}[]{ccc}V_{e\mu}&0&0\\\
0&-V_{e\mu}&0\\\ 0&0&0\end{array}\right){\rm
or}\left(\begin{array}[]{ccc}V_{e\tau}&0&0\\\ 0&0&0\\\
0&0&-V_{e\tau}\end{array}\right).$ (14)

The matter term $A=2\sqrt{2}G_{F}n_{e}E=7.63\times 10^{-5}~{}{\rm
eV}^{2}~{}\rho({\rm gm/cc})~{}E({\rm GeV})~{}\hbox{eV}^{2}.$ Here, $G_{F}$,
$n_{e}$ and $\rho$ are the Fermi constant, the electron number density and the
matter density of the medium, respectively. The evolution equation for
antineutrinos has the reversed sign for $A$, $V_{LR}$ and the phase $\delta$.
We have numerically solved full three flavor oscillation in presence of matter
and long-range forces.

However, to understand the bounds it is easy to consider two flavor $\mu-\tau$
oscillation, (which has been studied in Joshipura:2003jh  to constrain the
long-range forces using atmospheric neutrino data of Super-Kamiokande
experiment Super-K ). To understand the effect of 1-3 mixing, one needs to
consider the changes of the effective oscillation parameters due to $V_{LR}$.
The $\mu-\tau$ oscillation in presence of $V_{e\tau}$ is governed by the
evolution equation

$\displaystyle i\frac{d}{dt}\begin{pmatrix}\nu_{\mu}\\\ \nu_{\tau}\
\end{pmatrix}=$ $\displaystyle\begin{pmatrix}-\frac{\Delta m_{32}^{2}}{4E}\cos
2\theta_{23}&\frac{\Delta m_{32}^{2}}{4E}\sin 2\theta_{23}\\\ \frac{\Delta
m_{32}^{2}}{4E}\sin 2\theta_{23}&\frac{\Delta m_{32}^{2}}{4E}\cos
2\theta_{23}-V_{e\tau}\ \end{pmatrix}\begin{pmatrix}\nu_{\mu}\\\ \nu_{\tau}\
\end{pmatrix}$ (15)

Then, the survival probability of $\nu_{\mu}$

$P_{\mu\mu}=1-\sin^{2}2\tilde{\theta}_{23}~{}\sin^{2}\frac{\Delta\tilde{m}_{23}^{2}L}{4E},$
(16)

where, $L$ is the neutrino flight path length. The effective mixing angle
$\tilde{\theta}_{23}$ and $\Delta\tilde{m}_{32}^{2}$ are related with their
vacuum quantities by the relations

$\sin^{2}2\tilde{\theta}_{23}=\frac{Sin^{2}2\theta_{23}}{\left[(\xi_{e\tau}-\cos
2\theta_{23})^{2}+\sin^{2}2\theta_{23}\right]}$ (17)

and

$\Delta\tilde{m}_{23}^{2}=\Delta m_{23}^{2}\left[(\xi_{e\tau}-\cos
2\theta_{23})^{2}+\sin^{2}2\theta_{23})^{1/2}\right];$ (18)

where, $\xi_{e\tau}\equiv\frac{2V_{e\tau}E}{\Delta m_{32}^{2}}$. The potential
$V_{e\tau}$ and the corresponding $\xi$ change sign for $\bar{\nu}$.
Similarly, for $L_{e}-L_{\mu}$ gauge symmetry the survival probability can be
obtained and they satisfy

$P_{\mu\mu}(V_{e\tau})=P_{\bar{\mu}\bar{\mu}}(-V_{e\tau})=P_{\mu\mu}(-V_{e\mu})=P_{\bar{\mu}\bar{\mu}}(V_{e\mu})$
(19)

##  III The $\chi^{2}$ analysis

To evaluate the potential of ICAL with atmospheric neutrinos we generate
events by NUANCE-v3 Casper:2002sd  and consider only the muon energy and its
direction (directly measurable quantities) of the events for a conservative
estimation. The energy and angular resolutions of the muons at ICAL are very
high: 4-10% for energy and 4-12% for zenith angle, which are obtained from
GEANT geant  simulation. The ranges are due to different energies and
different angles with respect to the iron layers. These uncertainties are very
negligible compared to the uncertainties in reconstructed neutrinos due to
kinematics of the scattering processes.

The major uncertainty arises from the particles produced in the event other
than muon. One might expect that consideration of the hadrons for neutrino
energy $\mathrel{\hbox to0.0pt{\raise 2.15277pt\hbox{$>$}\hss}{\lower
2.15277pt\hbox{$\sim$}}}2$ GeV will improve the sensitivities substantially.
But, in Samanta:2010xm , it has been found that there is a very marginal
improvement on measurement of $\theta_{23}$ and a small improvement on $\Delta
m_{31}^{2}$: $\delta(\Delta m_{31}^{2})=0.02\times 10^{-3}$eV2. The fact is
that the total hadron energy is carried out by multiple low energy hadrons.
The average energy per hadron is $\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$<$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}$ 1 GeV and the
average number of hadrons per event is $\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$>$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}$ 2\. The energy
resolution of the hadrons at this energy is $\sim$ 80%, and consequently, the
neutrino energy resolutions do not improve significantly after adding the
hadrons 555The detection of the neutral current events may also be possible,
which have no directional information, but the energy dependence of the
oscillation averaged over all directions can contribute to the total
$\chi^{2}$ in the sensitivity studies..

The $\chi^{2}$ is calculated according to the Poisson probability distribution
with flat uncertainties of the oscillation parameters. The term due to the
contribution of prior information of the oscillation parameters measured by
other experiments is not added to $\chi^{2}$ to examine solely the performance
of ICAL. The data have been binned in cells of equal size in the $\log_{10}E$
\- $L^{0.4}$ plane, where $L=2R\cos\theta_{Z}$. The choice of binning is
motivated by pattern of the oscillation probability
$P(\nu_{\mu}\rightarrow\nu_{\mu})$ in the $L-E$ plane Samanta:2008ag . The
distance between two consecutive oscillation peaks driven by $\Delta
m^{2}_{23}$ increases (decreases) as one goes to lower $L$ ($E$) values for a
given $E$ ($L$). The binning of $L$ has been optimized to get better
sensitivity to the oscillation parameters. To maintain $\chi^{2}/d.o.f\approx
1$ for Monte Carlo simulation study, number of events should be $>4$ per cell
Samanta:2008af  (as large number of cells at high energies have number of
events less than 4 or even zero and they increase the $\chi^{2}/d.o.f$
substantially beyond 1). If the number is less than 4 (which happens in the
high energy bins), we combine events from the nearest cells.

The migration of the number of events from true neutrino energy and zenith
angle cells to muon energy and zenith angle cells is made using exact energy-
angle correlated 2-dimensional resolution functions Samanta:2006sj . For each
set of oscillation parameters, we integrate the oscillated atmospheric
neutrino flux folding with the cross section, the exposure time, the target
mass, the efficiency and the two dimensional energy-angle correlated exact
resolution functions to obtain the predicted data in each cell in $L-E$ plane
for the $\chi^{2}$ analysis. We use the charge current cross section of
Nuance-v3 Casper:2002sd  and the Honda flux in 3-dimensional scheme
Honda:2006qj . The number of bins and resolution functions have been optimized
in Samanta:2008af . Both theoretical (fit values) and experimental (true
values) data for $\chi^{2}$ analysis have been generated in the same way by
migrating number of events from neutrino to muon energy and zenith angle bins
using the resolution functions Samanta:2009qw .

The systematic uncertainties of the atmospheric neutrino flux are crucial for
determination of the oscillation parameters. We have divided them into two
categories: (i) the overall flux normalization uncertainties which are
independent of the energy and zenith angle, and (ii) the spectral tilt
uncertainties which depend on $E$ and $\theta_{\rm z}$.

The flux with uncertainties included can be written as

$\displaystyle\Phi(E,\theta_{Z})=\Phi_{0}(E)\left[1+\delta_{E}\log_{10}\frac{E}{E_{0}}\right]$
$\displaystyle\times\left[1+\delta_{Z}(|\cos\theta_{Z}|-0.5)\right]\times\left[1+\delta_{f_{N}}\right]$

For $E<1$ GeV we take the energy dependent uncertainty $\delta_{E}=15\%$ and
$E_{0}=1$ GeV and for $E>10$ GeV, $\delta_{E}=5\%$ and $E_{0}=10$ GeV. The
overall flux uncertainty as a function of zenith angle is parametrized by
$\delta_{Z}$. According to Honda:2006qj  we use $\delta_{Z}=4\%$, which leads
to 2% vertical/horizontal flux uncertainty. We take the overall flux
normalization uncertainty $\delta_{f_{N}}=10\%$ and the overall neutrino
cross-section uncertainty $\delta_{\sigma}=10\%$.

We evaluate the $\chi^{2}$ for both normal hierarchy (NH) and inverted
hierarchy (IH) with $\nu$s and $\bar{\nu}$s separately for a given set of
oscillation parameters. Then we find the total $\chi^{2}$
($=\chi^{2}_{\nu}+\chi^{2}_{\bar{\nu}}$).

We have set the inputs of $|\Delta m_{32}^{2}|=2.5\times 10^{-3}$eV2, and
$\delta_{CP}=0$. We marginalize $\chi^{2}$ over $\Delta
m_{32}^{2},~{}\theta_{23},~{}\theta_{13}$ and $\alpha$. We have chosen the
range of $\Delta m_{32}^{2}=2.0-3.0\times 10^{-3}$eV2,
$\theta_{23}=37^{\circ}-54^{\circ},$ $\theta_{13}=0^{\circ}-12.5^{\circ}$, and
$\alpha_{e\mu,e\tau}=0-3\times 10^{-53}$. The solar parameters are fixed at
their best-fit values: $\Delta m_{21}^{2}=7.67\times 10^{-5}$eV2 and
$\sin^{2}\theta_{12}=0.312$ Fogli:2008ig . The effect of $\Delta m_{21}^{2}$
comes in sub-leading order in the oscillation probability for atmospheric
neutrinos when $E\mathrel{\hbox to0.0pt{\raise 2.15277pt\hbox{$>$}\hss}{\lower
2.15277pt\hbox{$\sim$}}}$ GeV and it is very negligible.

Figure 1: The upper bounds of $\alpha_{e\mu}$ and $\alpha_{e\tau}$ for true
(input) $\alpha=0$ in the left pannel; and the corresponding upper as well as
lower bounds for input $\alpha=1.1538\times 10^{-53}$ in the right pannel,
respectively.

Figure 2: The sensitivities to mass hierarchy for different true (input)
$\theta_{23}$ values for input IH (left) and input NH (right), respectively.
For each case we have considered true (input) values of $\alpha_{LR}=0$,
$\alpha_{e\mu}=1.1538\times 10^{-53}$, and $\alpha_{e\tau}=1.1538\times
10^{-53}$, respectively.

Figure 3: The sensitivities to deviation of 2-3 mixing from maximal mixing as
well as its octant for both true (input) IH (left) and NH (right). For each
case we have considered $\alpha_{LR}=0$, $\alpha_{e\mu}=1.1538\times
10^{-53}$, $\alpha_{e\tau}=1.1538\times 10^{-53}$, respectively.

In this analysis we have considered an exposure of 1 Mton.year (which is 10
years run of 100 kTon) of ICAL and energy range 0.8 - 15 GeV.

##  IV Bounds on couplings

For atmospheric neutrinos at the magnetized ICAL the bounds on $\alpha_{e\mu}$
and $\alpha_{e\tau}$ are shown in the left pannel in Fig. 1 and its strength
to constrain them with upper as well as lower bounds are shown in the right
pannel in Fig. 1 with a typical nonzero input (true) value. We have checked
that the bounds do not change significantly for different combinations of true
values of oscillation parameters.

The bounds $\alpha_{e\mu}\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$<$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}5.5\times 10^{-52}$
and $\alpha_{e\tau}\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$<$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}6.4\times 10^{-52}$ at
90% CL have been obtained from present atmospheric neutrino oscillation data
Joshipura:2003jh . Considering both atmospheric and solar data the bounds are
$\alpha_{e\mu}\mathrel{\hbox to0.0pt{\raise 2.15277pt\hbox{$<$}\hss}{\lower
2.15277pt\hbox{$\sim$}}}3.4\times 10^{-53}$ and $\alpha_{e\tau}\mathrel{\hbox
to0.0pt{\raise 2.15277pt\hbox{$<$}\hss}{\lower
2.15277pt\hbox{$\sim$}}}2.5\times 10^{-53}$ at $3\sigma$ CL
Bandyopadhyay:2006uh . The better precision and smaller value by a few order
of magnitude of solar mass squared difference than the atmospheric one play
here the main role to make it significantly stringent.

The bounds are relatively stronger at the magnetized detector:
$\alpha_{e\mu,e\tau}\mathrel{\hbox to0.0pt{\raise
2.15277pt\hbox{$<$}\hss}{\lower 2.15277pt\hbox{$\sim$}}}1.65\times 10^{-53}$
at 3$\sigma$ CL. For atmospheric neutrinos it comes mainly due to the high
precision of $\Delta m_{32}^{2}$ and this can be understood quite well from
simple two flavor survival probability of $\nu_{\mu}$ and $\bar{\nu}_{\mu}$ in
vacuum with effective $\Delta m_{32}^{2}$ and $\theta_{23}$ for long-range
potentials. This becomes possible due to the fact that the 1-3 mixing effect
is sub-leading (as $\theta_{13}$ $<11.38^{\circ}$ Schwetz:2011qt ) and it can
be neglected at this moment for simplicity. When the potential $V_{LR}$ comes
to the play, it tries to change the effective value of $\Delta m_{31}^{2}$
(see Eq. 18) and becomes tightly constrained. The sign $V_{LR}$ is opposite
for $\nu$ and $\bar{\nu}$, and ICAL can detect them separately. This makes the
bounds more tighter at ICAL and lessens the difference in bounds between
$\alpha_{e\mu}$ and $\alpha_{e\tau}$, while the difference is substantially
large at non-magnetized detectors.

##  V Effects on 2-3 sector

###  V.1 Determination of mass hierarchy

In Fig. 2 we show the sensitivity to mass hierarchy for different true values
of $\theta_{23}$. We show it for three cases: assuming no potential for long-
range forces ($\alpha_{LR}=0$) and with a benchmark input (true) value for
$\alpha_{e\mu}$ and $\alpha_{e\tau}$, respectively.

For all three cases the sensitivity increases as one goes to the higher
$\theta_{23}$ values and it is a general feature for both cases with and
without the potential for long-range force. This can be understood from the
$\nu_{\mu}$ flux at the detector considering the effect of 1-3 mixing. Here we
assume no potential for long-range force. The ratio of the $\nu_{\mu}$ flux at
the detector (oscillated) and at the source (original) Samanta:2010xm :

$\frac{F_{\mu}}{F_{\mu}^{0}}\approx K(\sin
2\theta_{23})-f(\theta_{23})\left(1-\frac{1}{r}\right)P_{A}(\theta_{13}),$
(20)

where, $K(\sin 2\theta_{23})$ is an even function of the deviation (symmetric
with respect to change of the octant), and $P_{A}(\theta_{13})$ is a function
of $\theta_{13}$ only, and

$f(\theta_{23})\equiv\left(s_{23}^{4}-\frac{s^{2}_{23}}{r}\right)$ (21)

which increases quickly with $\theta_{23}$, so that for $r=3-4$,
$f(\theta_{23}=40^{\circ})\ll f(\theta_{23}>50^{\circ})$. Therefore, for
$\theta_{23}<45^{\circ}$ the flux $F_{\mu}$ has much weaker dependence on
$\theta_{13}$ than for $\theta_{23}>45^{\circ}$. This is reflected in the
sensitivity to mass hierarchy as the lower limit of $\theta_{13}$ has been
taken zero during marginalization and no prior contribution from other future
experiments has been considered. However, in future the lower limit will be
known from other experiments (as very recently T2K puts a lower bound
Abe:2011sj ); then the $\theta_{23}$-dependence will be less and the
sensitivity to mass hierarchy will also be substantially improved
Samanta:2009qw .

The potential $V_{e\mu}$ opposes, while $V_{e\tau}$ helps to determine the
hierarchy (see Fig. 2). As the 1-3 mixing is small
($\theta_{13}<11.2^{\circ}$), the potential $V_{e\tau}$ decreases the
effective value of $\Delta m_{31}^{2}$. Consequently, the resonant energy
decreases (see Fig. 2 of Samanta:2006sj ), where the atmospheric neutrino flux
is relatively large and it helps in determination of hierarchy. On the other
hand, as 2-3 mixing is large, $V_{e\mu}$ increases the effective value of
$\Delta m_{31}^{2}$ for $\theta_{23}>45^{\circ}$ (see Eq. 18). In spite of an
enhancement in 1-3 mixing (which happens for both potentials) the final
sensitivity decreases for $V_{e\mu}$ due to relatively low statistics at the
resonant zones. These happen for $\nu$ with NH and for $\bar{\nu}$ with IH.

###  V.2 Determination of octant of $\theta_{23}$

The sensitivity to 2-3 mixing, mainly the deviation from its maximal mixing
($\delta=\theta_{23}-45^{\circ}$) and the octant (sign of $\delta$) are shown
in Fig. 3 for three cases: i) $\alpha_{LR}=0$, ii) $\alpha_{e\mu}=1.1538\times
10^{-53}$, and iii) $\alpha_{e\tau}=1.1538\times 10^{-53}$, respectively. It
is important to note here that octant discrimination is significantly improved
in presence of long-range potentials. This is due to the fact that effective
1-3 mixing is always larger for both potentials. For a given hierarchy the
change of the sensitivity with the change of true (input) octant depends
mainly on the magnitude of $\delta^{\rm eff}={\theta_{23}}^{\rm
eff}-45^{\circ}$, which again strongly depends on the potentials (see Eq. 17).
The change of the effective value of $\Delta m_{31}^{2}$ works here
subdominantly (while it was a dominating factor for determination of the mass
hierarchy). Now, for $V_{e\tau}$ with NH, sensitivity to octant determination
is better for $\theta_{23}>45^{\circ}$ than $\theta_{23}<45^{\circ}$ as
$\delta^{\rm eff}$ is increased due to $V_{e\tau}$ for neutrinos and the flux
is two times higher for neutrinos than antineutrinos. This is opposite for
$V_{e\mu}$: octant determination is better for $\theta_{23}<45^{\circ}$ than
$\theta_{23}>45^{\circ}$. Similarly, the results with IH can be understood
using antineutrinos considering the symmetries in Eq. 19.

##  VI MINOS anomaly

The anomaly in recent MINOS data (difference in measured values of 2-3 mass
splittings and mixing angles for $\nu$ and $\bar{\nu}$ pvahle ), which implies
CPT violation or signal for non-standard interactions, has been explained by
long-range potential due to $L_{\mu}-L_{\tau}$ gauge symmetry Heeck:2010pg .
The Sun does not contain any $\mu$ or $\tau$ and hence there is no direct
bound from neutrino oscillation. However, the potential can be induced
indirectly on the Earth by the mixing of the gauge boson $Z^{\prime}$ of
$L_{\mu}-L_{\tau}$ with the standard model $Z$ boson Heeck:2010pg :

$V_{\mu\tau}=3.6\times 10^{-14}{\rm
eV}\left(\frac{\alpha_{\mu\tau}}{10^{-50}}\right)$ (22)

The sign changes for $\bar{\nu}$. From the analysis of atmospheric neutrino
data at ICAL with an exposure of 1 Mton$\cdot$yr, we find the bound on the
coupling $\alpha_{\mu\tau}\geq 3.2(4.2)\times 10^{-51}$ at 3(5)$\sigma$ CL and
the explanation of MINOS data Heeck:2010pg  can be tested at more than
5$\sigma$ CL.

##  VII Discussion and Conclusion

In this paper, we have estimated the conservative bounds on long-range forces
with atmospheric neutrinos at ICAL: $\alpha_{e\mu,e\tau}\mathrel{\hbox
to0.0pt{\raise 2.15277pt\hbox{$<$}\hss}{\lower
2.15277pt\hbox{$\sim$}}}1.65\times 10^{-53}$ at 3$\sigma$ CL. This bounds are
significantly stronger than the present bounds.

The CPT violation and the long-range forces, which are the candidates for
solution of recent anomaly in MINOS data, can be discriminated with
atmospheric neutrinos at ICAL. The distance between the Sun and the Earth
varies and the difference between aphelion and perihelion is about 3%, which
causes seasonal variation of the long-range forces. But, there should not be
any such change for CPT violation.

Assuming one by one nonzero input value of these couplings it is found that
while the potential $V_{e\mu}$ opposes; $V_{e\tau}$ helps to discriminate the
mass hierarchy. However, both potentials help to discriminate octant of
$\theta_{23}$. The anomaly of MINOS data, which has been explained using long-
range force potential, can be tested at ICAL with more that 5$\sigma$ CL.

Acknowledgements: This work was started at Harish-Chandra Research Institute
and has been supported partly by the Neutrino Physics projects of this
institute. The use of cluster computational facility installed by the funds of
this project is gratefully acknowledged.

## References

  * (1) R. Foot, Mod. Phys. Lett. A 6, 527 (1991); X.-G. He, G. C. Joshi, H. Lew and R. R. Volkas, Phys. Rev D 43, 22 (1991).
  * (2) M. Dine, W. Fischler and M. Srednicki, Phys. Lett. B 104, 199 (1981); J. E. Kim, Phys. Rev. Lett., 43 103 (1979).
  * (3) Y. Chikashige, R. N. Mohapatra and R. D. Peccei, Phys. Lett. B 98, 265 (1981).
  * (4) G. B. Gelmini and M. Roncadelli, Phys. Lett. B 99, 411 (1981).
  * (5) L. J. Hall and S. J. Oliver, Nucl. Phys. Proc. Suppl. 137, 269 (2004) [arXiv:hep-ph/0409276] and references therein.
  * (6) D. Chang, W. Y. Keung and P. B. Pal, Phys. Rev. D 42, 630 (1990).
  * (7) T. Damour, Class. Quant. Grav. 13, A33 (1996) [arXiv:gr-qc/9606080].
  * (8) Fischbach E, Gillies G T, Krause D E, Schwan J G and Talmadge C 1992 Metrologia 29 213
  * (9) E. G. Adelberger, B. R. Heckel and A. E. Nelson, Ann. Rev. Nucl. Part. Sci. 53, 77 (2003) [arXiv:hep-ph/0307284].
  * (10) J. G. Williams, X. X. Newhall and J. O. Dickey, Phys. Rev. D 53, 6730 (1996).
  * (11) T. D. Lee and C. N. Yang, Phys. Rev. 98 (1955) 1501.
  * (12) R. V. Eotvos, D. Pekar, E. Fekete, Annalen Phys. 68, 11-66 (1922).
  * (13) L. Okun, Phys. Lett. B 382, 389 (1996) [arXiv:hep-ph/9512436].
  * (14) A. D. Dolgov, Phys. Rept. 320, 1 (1999).
  * (15) A. S. Joshipura and S. Mohanty, Phys. Lett. B 584, 103 (2004) [arXiv:hep-ph/0310210].
  * (16) A. Bandyopadhyay, A. Dighe and A. S. Joshipura, Phys. Rev. D 75, 093005 (2007) [arXiv:hep-ph/0610263].
  * (17) M. C. Gonzalez-Garcia, P. C. de Holanda, E. Masso and R. Zukanovich Funchal, JCAP 0701, 005 (2007) [arXiv:hep-ph/0609094].
  * (18) J. A. Grifols, E. Masso and S. Peris, Astropart. Phys. 2, 161 (1994).
  * (19) J. A. Grifols and E. Masso, Phys. Lett. B 579, 123 (2004) [arXiv:hep-ph/0311141].
  * (20) S. I. Blinnikov, A. D. Dolgov, L. B. Okun and M. B. Voloshin, Nucl. Phys. B 458, 52 (1996) [arXiv:hep-ph/9505444].
  * (21) Talk given by P. Vahle at 24th International Conference On Neutrino Physics And Astrophysics (Neutrino 2010), June 14, 2010.
  * (22) J. Heeck and W. Rodejohann, arXiv:1007.2655 [hep-ph]; H. Davoudiasl, H. -S. Lee, W. J. Marciano, [arXiv:1102.5352 [hep-ph]].
  * (23) V. Arumugam et al. [INO Collaboration], [ See, http://www.imsc.res.in/ ino ]
  * (24) A. Samanta, Phys. Lett. B693, 296-300 (2010). [arXiv:1005.4851 [hep-ph]].
  * (25) John N. Bahcall, Neutrino Astrophysics (Cambridge Univ. Press,Cambridge, England, 1989).
  * (26) Y. Fukuda et al. Phys Rev Lett. 82, 2644 (1999); Y. Fukuda et al. Phys Lett . B388, 397 (1999);Y. Fukuda et al. Phys Rev Lett. 85, 3999 (2000).
  * (27) D. Casper, Nucl. Phys. Proc. Suppl. 112, 161 (2002) [arXiv:hep-ph/0208030].
  * (28) http://geant4.web.cern.ch/geant4/
  * (29) A. Samanta, A. Y. .Smirnov, JHEP 1107, 048 (2011). [arXiv:1012.0360 [hep-ph]].
  * (30) A. Samanta, Phys. Rev. D 79, 053011 (2009) [arXiv:0812.4640 [hep-ph]].
  * (31) A. Samanta, Phys. Rev. D 80, 113003 (2009) [arXiv:0812.4639 [hep-ph]].
  * (32) A. Samanta, Phys. Lett. B 673, 37 (2009) [arXiv:hep-ph/0610196].
  * (33) M. Honda, T. Kajita, K. Kasahara, S. Midorikawa and T. Sanuki, Phys. Rev. D 75, 043006 (2007) [arXiv:astro-ph/0611418].
  * (34) A. Samanta, Phys. Rev. D81, 037302 (2010). [arXiv:0907.3540 [hep-ph]].
  * (35) G. L. Fogli et al., Phys. Rev. D 78, 033010 (2008) [arXiv:0805.2517 [hep-ph]].
  * (36) T. Schwetz, M. Tortola, J. W. F. Valle, New J. Phys. 13, 063004 (2011). [arXiv:1103.0734 [hep-ph]].
  * (37) K. Abe et al. [ T2K Collaboration ], [arXiv:1106.2822 [hep-ex]].


# Semi-Supervised Learning in the Few-Shot Zero-Shot Scenario

Noam Fluss,1 Guy Hacohen,1,2 Daphna Weinshall1

###### Abstract

Semi-Supervised Learning (SSL) leverages both labeled and unlabeled data to
improve model performance. Traditional SSL methods assume that labeled and
unlabeled data share the same label space. However, in real-world
applications, especially when the labeled training set is small, there may be
classes that are missing from the labeled set. Existing frameworks aim to
either reject all unseen classes (open-set SSL) or to discover unseen classes
by partitioning an unlabeled set during training (open-world SSL). In our
work, we construct a classifier for points from both seen and unseen classes.
Our approach is based on extending an existing SSL method, such as _FlexMatch_
, by incorporating an additional entropy loss. This enhancement allows our
method to improve the performance of any existing SSL method in the
classification of both seen and unseen classes. We demonstrate large
improvement gains over state-of-the-art SSL, open-set SSL, and open-world SSL
methods, on two benchmark image classification data sets, CIFAR-100 and
STL-10. The gains are most pronounced when the labeled data is severely
limited (1-25 labeled examples per class).

##  1 Introduction

As the collection of data becomes increasingly widespread, new scenarios that
pose challenges to the traditional supervised machine learning framework are
continuously emerging. Consider, for instance, a scenario where data can be
easily collected at a low cost. However, acquiring accurate labels for this
data requires specialized expertise or expensive machinery, all while adhering
to strict budget constraints. Such scenarios may occur more frequently in
fields like medical research or genomics. In this scenario, the unlabeled
dataset comprises a vast amount of data, but due to limited resources, only a
small portion of it can be selected in an unsupervised manner to be further
processed for labeling111We note that active learning in the low budget domain
aims to replace the random selection of the set of points to be labeled by a
more effective selection methodology.. In the end, we are left with a
substantial unlabeled dataset and a relatively limited number of labeled data
points.

In this scenario, we encounter a unique challenge where certain classes are
missing from the labeled set, to be denoted “zero-shot” or “unseen” classes.
Note also that the small labeled set is unlikely to accurately represent the
prior probability distribution of the classes. These remaining classes are
called “imbalanced few-shot” or “seen”. As a result, in the proposed scenario,
the labeled set is constructed of classes with only few-shot and zero-shot
examples. Additionally, due to the random selection of points to be processed
for labeling, the class distribution of the labeled points may deviate widely
from the true class distribution.

Nevertheless, we identify valuable constraints in this scenario, that can be
leveraged for the successful handling of zero-shot classes. Specifically, by
assumption the unlabeled training set is derived from the same distribution as
the test dataset, which consists of new samples obtained under similar
conditions. Thus there are no Out-Of-Distribution (OOD) samples to consider,
Additionally, the set of possible labels is fixed throughout.

While Semi-Supervised Learning (SSL) appears to be a suitable framework to
address the problem at hand, it is important to note that most SSL methods
assume the presence of all classes in the labeled set, namely, that there are
no zero-shot classes. A step in the right direction is taken in the _open-set
SSL_ (OSSL) framework, where the data contains points sampled from either
unseen classes or OOD (or both). However, open-set techniques are designed to
reject test samples that do not belong to any seen class and do not predict
missing class labels. Nevertheless, we compare our proposed method to state-
of-the-art SSL and OSSL methods adapted to identify unseen classes (see
details in Section 2.4), showing large gains in performance for our approach.

| Train labeled data | Train unlabeled data | Test data | Expected output
---|---|---|---|---
Settings |  seen |  unseen |  seen |  unseen |  seen |  unseen |  seen |  unseen
$\dagger$zero-shot learning |  Y |  N |  - |  - |  Y/N |  Y |  classify |  classify
*Semi-Supervised Learning (SSL) |  Y |  - |  Y |  Y/N |  Y |  - |  classify |  -
*Open set SSL (OSSL) |  Y |  N |  Y |  Y |  Y |  Y |  classify |  reject
Open world SSL (OWSSL) |  Y |  N |  Y |  Y |  Y |  Y |  classify |  classify
Our settings |  few-shot |  N |  Y |  Y |  Y |  Y |  classify |  classify
Table 1: Catalogue of different relevant frameworks as discussed in the text.
The training set is divided into seen and unseen classes, a distinction not
made in all frameworks. For each framework, the symbols ’Y’ and ’N’ denote
that such data as specified in the column is permitted in the framework; ’Y’
indicates that such data exists, and ’N’ that it doesn’t. ’*’ indicates
frameworks in which unseen classes may be replaced with OOD data (see text).
$\dagger$ indicates that additional semantic side-information such as
attributes is needed. Expected output: Open-set SSL methods are not designed
to classify new examples from unseen classes, only to reject them.

A closely related scenario is addressed in the _open-world SSL_ framework,
which aims to classify seen classes and to discover unseen classes in an
unlabeled set. Typically in this framework, a learner is given substantial
labeled and unlabeled sets for training, where both sets are sampled from the
same distribution. Originally, the aim of such methods was to partition the
unlabeled set in a transductive manner. Such methods can be extended to our
inductive scenario by providing the partitioned unlabeled set as input to a
regular SSL method, see discussion of related work below.

Another related methodology is Zero-Shot Learning (ZSL). Traditionally, ZSL
methods are designed to train models that can classify objects from unseen
classes by utilizing semantic side-information and knowledge acquired from
seen classes (see survey by Pourpanah et al., 2022). Differently, in our
scenario, we lack access to any supplementary semantic information. Another
difficulty with adopting this framework to our scenario stems from the fact
that ZSL usually assumes that seen classes are adequately sampled, i.e., they
are not “few-shot” but “many-shot”.

###  1.1 Our proposed approach

We describe a simple end-to-end framework that tackles the above scenario,
offering a general solution that can be used to enhance any semi-supervised
learning (SSL) method. The approach consists of two main steps.  (i) Modify
the last layer of a given SSL architecture to incorporate both seen and unseen
classes. By doing so, the model is capable of handling and classifying
instances from both types of classes.  (ii) Introduce an additional term into
the loss function, which penalizes discrepancies between the output
distribution of the trained model and the expected class distribution.

As the field of semi-supervised learning continues to evolve, the proposed
approach can be used to adapt future SSL methods to our scenario. When
compared to zero-shot learning, instead of relying on hard-to-collect semantic
side information, we exploit the fact that the unlabeled set accurately
represents the underlying class distribution and contains classes that are not
present in the labeled set. By doing so, we are able to transfer knowledge
from few-shot to zero-shot classes without auxiliary information.

We assess the performance of our method on standard image classification
datasets, re-constructing training sets where labels are obtained as described
above. While varying the number of unseen classes, we also vary the conditions
under which seen classes are sampled, including both balanced and imbalanced
scenarios. Current SSL methods, after being adapted to identify unseen classes
(see details in Section 2.4), are unable to successfully classify them. Our
methodology, when incorporated with the same SSL methods, greatly improves
their performance on unseen classes, while usually matching or even surpassing
their performance on seen classes. Additionally, our method surpasses the
overall performance of both open-set and open-world SSL methods, see Section
3.2. The improvement is especially high when the labeled set is small.

###  1.2 Related work

A number of related frameworks have been investigated in the machine learning
community, motivated by problems that also inflict our scenario. Table 1
summarizes the main characteristics of the most relevant frameworks,
highlighting the differences between them. Next, we discuss each one in more
detail.

##### Closed and Open Semi-Supervised Learning,

denoted SSL and OSSL respectively. SSL has made remarkable progress in recent
years, see the comprehensive survey by Chen et al. (2022). It involves
leveraging sparsely labeled data and a considerable amount of additional
unlabeled data, often sampled from the same underlying data distribution as
the labeled data. In more general settings, Out Of Distribution (OOD) data may
also be present in the unlabeled set.

In the original SSL settings Chapelle et al. (2009), both the labeled and
unlabeled data share a label set within the same domain. It is not, however,
suitable for our scenario because it does not allow for unseen classes. Going
a step further, open-set SSL (OSSL) permits the presence of unseen classes or
OOD data in the unlabeled set. However, this framework is only designed to
identify and reject data points that do not belong to the classes in the
labeled set and is not tailored to classify unseen classes.

Some of the most effective SSL methods at present time combine ideas from
self-supervised learning (e.g., consistency regularization), data
augmentation, and the assignment of pseudo-labels to unlabeled data points, in
order to utilize unlabeled data (e.g., Chen et al., 2020; Sohn et al., 2020;
Zhang et al., 2021; Wang et al., 2023). The field of Open-set SSL (OSSL) is
also rapidly developing (Guo et al., 2020; Yu et al., 2020; Huang et al.,
2021; Park et al., 2022; Sun and Li, 2023). For comparison, we use the
competitive OSSL method proposed by Saito et al. (2021), which accomplishes
the task by uniting the SSL and novelty detection frameworks.

##### Open-World SSL,

denoted OWSSL. This paradigm is related to _Novel Class Discovery_ , which
aims to discover unseen classes within unlabeled data (Xie et al., 2016; Han
et al., 2020; Wang et al., 2020). Joining SSL with this paradigm, OWSSL Cao et
al. (2022); Zhong et al. (2021); Rizve et al. (2022); Guo et al. (2022) aims
to discover unseen classes in an unlabeled set and to classify seen classes.
Initially, OWSSL methods were transductive, in the sense that the test set is
processed as a single batch, and classes are discovered by partitioning this
set. Typically this is accomplished by learning pairwise similarities between
the points in the unlabeled set.

In _OpenLDN_ , Rizve et al. (2022) extended the scope of the open-world
methodology to our scenario. Specifically, they proposed to use the outcome,
both seen classes and the partitioned unlabeled set, as input to an SSL
method, thus obtaining a classifier for both seen and unseen classes. Our
method differs from OpenLDA in that it solves the problem in an end-to-end
fashion, and is thus more suitable for the low-budget regime with very few
labeled examples.

##### Zero-Shot Learning,

denoted ZSL. Work on ZSL often involves the computation of a separate label
embedding space and assumes access to semantic data that may be difficult to
obtain. Some recent work aims to expand the scope of this methodology to
domains where the seen classes are only sparsely sampled. For example, Li et
al. (2015); Xu et al. (2021) leverage SSL in order to aid the computation of
label embedding, in a two-step manner.

###  1.3 Summary of contribution

We present and justify a novel SSL variation, characterized by classes that
are present in both the unlabeled set and the test set, yet absent from the
labeled set. Addressing this variant, we offer a solution that:

  1. i.

Outperforms all current baselines by a large margin.

  2. ii.

Can be assimilated into any state-of-the-art SSL method.

  3. iii.

In few-shot settings, the combined solution converges significantly faster
than its underlying SSL method.

##  2 Our Approach

Traditionally, different metrics are used to evaluate performance when dealing
with labeled or unlabeled data. With labeled data, metrics that measure
classification accuracy are appropriate. With unlabeled data, the outcome is
essentially a partition of the data – no specific labels are assigned to
specific clusters. Here, the appropriate metrics measure the partition
accuracy, as used to evaluate clustering algorithms. These scores are detailed
in Section 2.2.

In Section 2.3 we describe our method, designed to learn a classifier for the
Semi-Supervised Learning scenario with few-shot and zero-shot classes. In
Section 2.4 we discuss how baseline methods can be used for comparison. Since
most of the relevant frameworks (see discussion above) are not designed to
classify unseen classes, we describe an adaptation step, which allows us to
obtain unseen class classification from SSL and open-set SSL methods.

###  2.1 Notations and definitions

Let $\mathcal{X}$ denote the set of all possible examples, and $C$ the set of
possible classes. We consider a learner denoted as $f:\mathcal{X}\rightarrow
C$, which has access to a labeled set of examples
$\mathcal{L}\subseteq\mathcal{X}\times C$ and an unlabeled set of examples
$\mathcal{U}\subseteq\mathcal{X}$. In our scenario, the labeled examples in
$\mathcal{L}$ do not contain all possible classes of $C$. Let
$C_{seen}\subseteq C$ denote the set of classes that appears in $\mathcal{L}$,
and $C_{unseen}\subseteq C$ the set of classes that do not appear in
$\mathcal{L}$. Note that $C_{seen}\cap C_{unseen}=\emptyset$, and
$C_{seen}\cup C_{unseen}=C$. For evaluation purposes, we assume the existence
of a test set $\mathcal{T}\subseteq\mathcal{X}\times C$, which is disjoint
from both $\mathcal{U}$ and $\mathcal{L}$. $\mathcal{T}$ contains examples
from all classes. We denote the examples from seen classes in the test set as
$\mathcal{T}_{seen}$ and the examples from unseen classes as
$\mathcal{T}_{unseen}$.

###  2.2 Evaluation scores

##### Open-world SSL scores.

As customary in novel class discovery and OWSSL, the following scores are
used:

  1. 1.

seen classes accuracy

$acc_{seen}=\frac{1}{|\mathcal{T}_{seen}|}\sum_{x_{i}\in\mathcal{T}_{seen}}\mathds{1}_{[f(x_{i})=y_{i}]}$

  2. 2.

unseen classes accuracy

$acc_{unseen}=\frac{1}{|\mathcal{T}_{unseen}|}\sum_{x_{i}\in\mathcal{T}_{unseen}}\mathds{1}_{[f(x_{i})=y_{i}]}$

This score is based on the best permutation of the unseen classes – similarly
to what is done in clustering accuracy.

  3. 3.

combined score

$\frac{|C_{seen}|}{C}\cdot acc_{seen}+\frac{|C_{unseen}|}{C}\cdot
acc_{unseen}$

##### Open-set SSL scores.

In OSSL, where the goal is only to reject unseen classes and not to classify
them, different scores are used. For a fair comparison, we report these scores
below in cases suitable for the OSSL framework, with many missing classes and
sufficient labels for each seen class.

  1. 1.

closed accuracy

$\frac{1}{|\mathcal{T}_{seen}|}\sum_{x_{i}\in\mathcal{T}_{seen}}\mathds{1}_{[f|_{seen}(x_{i})=y_{i}]}$

Here, $f|_{seen}$ is the prediction of $f$ when forced to choose a class out
of $C_{seen}$. Thus an OSSL method is not allowed to reject. Other methods are
not allowed to classify points for labels in $C_{unseen}$.

  2. 2.

unknown accuracy

$\frac{1}{|\mathcal{T}_{unseen}|}\sum_{x_{i}\in\mathcal{T}_{unseen}}\mathds{1}_{[f(x_{i})\in
C_{unseen}]}$

This score captures the rejection accuracy.

  3. 3.

AUROC – Area under ROC curve of ’reject’ classifier.

Note the difference between seen classes accuracy and closed accuracy: the
first penalizes for seen points recognized as unseen, while the second
doesn’t, and therefore seen classes accuracy is always lower than closed
accuracy.

###  2.3 Our Method

We denote as $B$ the batch size of the learner. Let $\overline{p}_{i}$ denote
the empirical probability of class $i\in C$ across all the classes, in a
specific batch:

$\overline{p}_{i}=\frac{1}{B}\sum_{b=1}^{B}f_{i}(x_{b})$

Where $f_{i}$ is the confidence of $f$ on class $i\in C$. We define the
following entropy loss:

$\ell_{e}=-\sum_{i\in C_{seen}\cup
C_{unseen}}\overline{p}_{i}\log(\overline{p}_{i})$

This choice assumes that all classes are roughly equally distributed in the
data. If the prior distribution of class frequencies is known to be the vector
$\\{g_{i}\\}_{i}$, we will use instead the KL divergence between the
probability vectors:

$\ell_{e}=\sum_{i\in C_{seen}\cup
C_{unseen}}\overline{p}_{i}\left[\log(g_{i})-\log(\overline{p}_{i})\right]$

Most SSL methods optimize a loss function which is essentially the sum of an
unsupervised term $\ell_{u}$ and supervised term $\ell_{s}$. In our approach,
we adopt the full pipeline of the method, while optimizing the following
objective function:

$\ell=\ell_{s}+\ell_{u}+\lambda\ell_{e}$

This design is illustrated in Fig. 1.

Figure 1: The combined SSL loss used in our approach.

#### Model Collapse

During the training of our algorithm, we sometimes encounter a phenomenon in
which the confidence of the model’s predictions falls drastically, especially
the confidence of points from unseen classes. This phenomenon results in the
model being essentially unable to generate pseudo labels, and therefore the
scores show a sharp sudden decline. This may happen relatively early in the
training as there are fewer labels per class.

Our solution is to look at the entropy loss during training and to identify
any sharp decline (using a threshold on the gradient graph). Once identified,
a stopping point is fixed a few epochs before the sudden decline. An
Interesting outcome is that our algorithm converges much faster than its
original SSL variant. For example, with 4 labels per class and 40 unseen
classes in CIFAR-100, our method converges after $523.0\pm 24.61$ epochs,
while the SSL methods of _FlexMatch_ and _FreeMatch_ require 1024 epochs,
_OpenMatch_ \- 512, and _OpenLDN_ \- 1024.

###  2.4 Baseline methods for comparison

Semi-Supervised Learning in the Few-Shot Zero-Shot Scenario is a somewhat new
setting, and there are no ready-to-use baseline methods that are designed to
solve it effectively. Consequently, we extend methods designed to deal with
closely related problems, as discussed in the introduction (see Table 1), and
report their results. It should be kept in mind that since the methods are
designed for other scenarios, their inferior results here should not be taken
as evidence that they are not suited for their original task, where they are
likely to achieve state-of-the-art results.

More specifically, we adapt methods designed to deal with the following
scenarios:  (i) semi-supervised learning (SSL),  (ii) open-set SSL (OSSL),
(iii) open-world SSL (OWSSL).  We start by describing how to extend SSL and
OSSL methods to the OWSSL scenario, similarly to how it’s done by Cao et al.
(2022). We then conclude by discussing the adaptation of the OWSSL scenario.

##### Traditional SSL

methods are designed to classify seen classes, but they do not expect to
encounter unseen classes in the test set. In order to extend them to the OSSL
scenario, we adopt a common way to add a reject option to classifiers (e.g.,
Tax and Duin, 2008), by applying a threshold test to the softmax confidence
score of the model. To compensate for the simplicity of the approach, we
choose for each method its optimal threshold based on ground truth
information. In the end, we have a set of points classified into seen classes,
and a set of rejected points, similar to the OSSL scenario to be discussed
next.

##### Open-set SSL

methods are designed to classify seen classes and reject unseen classes, but
can’t classify unseen classes. In order to extend these methods to our
scenario, an additional step is employed to partition the rejected points to
$|C_{unseen}|$ parts. This is accomplished with K-means clustering, performed
over the model’s feature space.

As customary for unlabeled data, and for the purpose of evaluation only, the
set of cluster labels is matched to the set of unseen labels $C_{unseen}$
using the best permutation. Note that the unseen classes accuracy score above
employs a similar procedure. As a result, the set of test points is now
classified to all the labels in $C$.

##### Open-world SSL

methods are designed to process the labeled and unlabeled training sets and
output a partition of the unlabeled set. This partition is matched to
$C_{unseen}$ as described above. Finally, in order to generate a classifier
for future unseen points, we follow the procedure suggested by Rizve et al.
(2022): use the labeled training set as is and the unlabeled set with its
inferred labels to train another classifier, whose domain of output labels is
$C$.

##  3 Empirical Evaluation

(a) 1 label per class, 35 unseen classes

(b) 4 labels per class, 40 unseen classes

(c) 25 labels per class, 40 unseen classes

Figure 2: Comparison of our method to the baselines discussed in Section 3.1.
All methods are trained on CIFAR-100, where (a),(b), and (c) differ in the
size of the labeled set and the number of unseen classes. In the top row, we
show the _combined score_ of each method. In the bottom row, we show the
_seen_ and _unseen accuracy_ of each method. We clearly see that our method
outperforms all baselines by significant margins. When compared to SSL methods
- _FreeMatch_ and _FlexMatch_ , we see that while the performance on the seen
classes is comparable, our method improves the performance on the unseen
classes quite drastically. Both _OpenMatch_ and _OpenLDN_ are not suitable for
the few-shot regime, and their performance is therefore low.

We now describe the empirical settings used to evaluate our method in
comparison to established baselines.

###  3.1 Methodology and technical details

##### Datasets.

We use two benchmark image datasets in the experiments below: CIFAR-100
(Krizhevsky et al., 2009) and STL-10 (Coates et al., 2011). When using STL-10,
we omitted the unlabeled split due to its inclusion of out-of-distribution
examples.

##### Baselines.

We compare our method to the following baselines:  (i) SSL - _FreeMatch_ Wang
et al. (2023) which performs well in the few-shot regime, and _FlexMatch_
(Zhang et al., 2021) which is the backbone of our own method.  (ii) Open-set
SSL - _OpenMatch_ (Saito et al., 2021).  (iii) Open-world SSL: _OpenLDN_
(Rizve et al., 2022), which includes its own adaptation to our scenario using
_MixMatch_ (Berthelot et al., 2019).  For _FlexMatch_ and _FreeMatch_ we
employ the SSL evaluation framework established by (Wang et al., 2022),
ensuring a fair comparison. For _OpenMatch_ and _OpenLDN_ algorithms, we used
the source code provided with the original papers. The methods were adapted to
our scenario as detailed in Section 2.4.

##### Architectures and Hyper-parameters.

When training _FlexMatch_ , _FreeMatch_ , and _OpenMatch_ , we considered a
Wide-ResNet-28 (WRN) (Zagoruyko and Komodakis, 2016), trained with stochastic
gradient descent optimizer, 64 batch size, 0.03 learning rate, 0.9 momentum
and 5e-4 weight decay. In Fig 2 we used 8 width factor, as was done in the
original papers. Due to its heavy computational cost, in all other
experiments, we reduced the width factor to 2. We note that the qualitative
results remained the same.

When training _OpenLDN_ , we used the official code, composed of a basic model
and a _MixMatch_ model. Both models use ResNet-18, trained with Adam
optimzier. The base model uses a 200 batch size and a 5e-4 learning rate.
_MixMatch_ uses 64 batch size and 0.002 learning rate.

The $\lambda$ for $\ell_{e}$ was set to 1.5 in all experiments. We used NVIDIA
GeForce RTX 2080 GPUs for all experiments.

##### Split of classes to seen and unseen.

When training on CIFAR-100, the split of the 100 classes into seen and unseen
classes was done randomly, repeating the exact same partition for all models.
It’s important to note that in the empirical evaluation of _OpenMatch_ ,
outlined in Saito et al. (2021), the choice of unseen classes was determined
by their super-class membership, introducing a potential bias. This may
explain the differences in accuracy between the results we report below for
_OpenMatch_ , and those reported in the paper. To validate this point, we used
their split method in a subset of our experiments, those involving the many-
shot scenarios with 100 or 250 examples per class.

###  3.2 Results

(a) Combined score

(b) Seen and unseen accuracy

(c) OSSL scores

Figure 3: Top: CIFAR-100, 100 labels per class, 45 unseen classes. Bottom:
CIFAR-100, 250 labels per class, 50 unseen classes. Split is based on super-
classes, as explained in Section 3.1. The various scores, including the
additional OSSL scores, are described in Section 2.2.

##### Balanced labeled set, few-shot for seen classes.

We begin by comparing the performance of our method, combined with _FlexMatch_
, with two SSL methods – _FlexMatch_ and _FreeMatch_ , one OSSL method –
_OpenMatch_ , and one OWSSL method – _OpenLDN_. _FlexMatch_ , _FreeMatch_ ,
and _OpenMatch_ are adapted with k-means to our scenario, as described in
Section 2.4. All methods are trained on CIFAR-100, with 40 unseen classes
(similar qualitative results are obtained when the number of classes is
varied, see ablation study). The number of labeled points in each seen class
varies, from one-shot with 1 example, few-shot with 4 examples and mid-shot
with 25 labels, which can be found in Figs. 2(a),2(b),2(c), respectively.

(a) Combined score

(b) Seen and unseen accuracy

Figure 4: CIFAR-100, 100 labeled examples sampled at random (see text). On
average, there are $36.61\pm 1.38$ unseen classes, and $1.58\pm 0.03$ labeled
examples per seen class.

Inspecting the _combined accuracy_ score in Fig. 2, we observe that our method
outperforms the other baselines by large margins. In each case, we also plot
the seen and unseen classes accuracy for each method. Evidently, the
improvement over other SSL algorithms is caused primarily by their poor
performance on unseen classes, which in turn is caused by their overconfidence
in their erroneous classification of seen classes. This is a well-known
problem (Nguyen et al., 2015) when attempting to add a reject option to deep
models, which can be further appreciated from the comparison in the ablation
study, Fig. 6.

It’s worth noting that the subpar performance of _OpenMatch_ and _OpenLDN_ can
be primarily attributed to their intended focus on scenarios with considerably
higher availability of labeled data. Their under-performance stems from being
tailored to suit different tasks, which are evaluated using different metrics
as discussed in Section 2.2.

##### Non-uniform labeled set, approximately one-shot

We now get back to the original scenario, where the labels are obtained as
follows: To begin with, the learner is given a large set of unlabeled data
cheaply collected. Additionally, there is a fixed budget that allows for the
labeling of only $n$ examples. The learner randomly chooses $n$ examples from
the unlabeled set to be labeled, thus obtaining the labeled training set. The
labeled set is quite imbalanced, with a range of examples for each seen class,
from 1 up to 3 or even 4. The remaining examples compose the unlabeled
training set.

Results of this setup are shown in Fig. 4, for CIFAR-100 and $n=100$ labeled
examples. Once again, when inspecting the _combined score_ , our method
outperforms all other baselines. Given the technical difficulties that this
unusual setup poses, we could not obtain reliable results with _OpenLDN_.
Nevertheless, we expect _OpenLDN_ to perform poorly in these conditions, as
seen with 1 labeled example per seen class in Fig. 2(a).

##### Balanced labeled set, many-shot for seen classes.

In our final setting, we increase the number of labeled examples per class,
thus obtaining scenarios that are more favorable to the alternative methods,
especially _OpenMatch_ and _OpenLDN_. To shapren the comparison with these
methods, we add the 3 valuation scores discussed in Section 2.2, which are
sometimes used to evaluate open-set SSL methods. Results222Since _FlexMatch_
and _FreeMatch_ perform similarly in the many-shot case, we only show results
using _FlexMatch_.  of this setup with 5 repetitions are shown in Fig. 3. Our
method significantly outperforms all baselines, though the gap with _OpenLDN_
is smaller.

###  3.3 Ablation study

##### Balanced labeled set, few-shot for seen classes, STL-10.

To verify that the results reported above are not unique to a single dataset,
we repeated some of the experiments with an inherently different image dataset
- STL-10, showing similar qualitative results. We evaluate a case with very
few labeled examples - 4 labeled examples for each of 6 seen classes, and 4
unseen classes. Since _OpenMatch_ and _OpenLDN_ are not natively adapted to
STL-10, we limited the comparison of our method, which is based on _FlexMatch_
, to that of _FlexMatch_ without out method. The results are shown in Fig. 5.

(a) Combined score

(b) Seen and unseen accuracy

Figure 5: STL-10, 4 labels per class, 4 unseen classes.

##### SSL adaptation.

To adapt the SSL methods to our scenario, we introduced in Section 2.4 a two-
step adaptation process:  (i) Points with low classification confidence are
rejected from the classifier trained on seen classes.  (ii) K-means clustering
is used to partition the rejected points within the feature space. This
partitioning is then matched to unseen classes.  Without this adaptation, SSL
methods would not identify anything as unseen, and would therefore erroneously
match all points from unseen classes to some seen class. In each case, the
rejection threshold was optimized to give each method its best possible
_combined score_.

Here we evaluate the added value of this adaptation, as shown in Fig. 6.
Evidently, the impact is minor (especially in the mid-shot regime). The reason
is that the optimal threshold tends to be relatively low, resulting in a
relatively modest count of predicted unseen instances. This count often pales
in comparison to the actual prevalence of unseen classes within the test data.
The underlying reason for this failure is the known property of deep models to
be over-confident in their predictions. It is therefore hard to distinguish
between seen and unseen classes based on the confidence threshold.

(a) 25 labels per class, 40 unseen classes

(b) 1 label per class, 35 unseen classes

Figure 6: Combined score of _FlexMatch_ and _FreeMatch_ with and without the
K-means adaptation (see Section 2.4), on CIFAR-100. _With K-means_ \- dark
blue, _Without K-means_ \- light blue.

##### Convergence time

As described in detail in Section 2.3, sometimes the SSL training, performed
by our method, is stopped relatively early. This happens when the entropy loss
suddenly declines sharply, and is usually correlated with small numbers of
labeled examples per seen class. As a result, it so happens that our model
needs much fewer epochs to converge, as compared to the original model that
serves as its backbone, _FlexMatch_. This result is shown in Table 2.

Labels per | Unseen | Convergence
---|---|---
class | classes | epoch
1 | 35 | $224.6\pm 44.19$
4 | 40 | $523.0\pm 24.61$
25 | 40 | $726.6\pm 8.85$
Table 2: The mean number of epochs to convergence, in a scenario where
_FlexMatch_ needs 1024 epochs to converge.

##### Additional results

Additional results, which strengthen the conclusions derived in this study,
are shown in the Appendix. Specifically, Fig. 8 shows results with 2 and 3
labeled examples per seen class. Fig. 7 shows results with only 10 missing
classes, where the added value of our method is somewhat diminished but still
significant. Results in the mid-range, with 50 labels per seen class, are
shown in Fig. 3. In all cases, we observe similar qualitative results to those
reported above.

###  3.4 Summary and Discussion

We investigated Semi-Supervised Learning (SSL) in a small sample framework
with few-shot and zero-shot classes, thereby unveiling an unexplored real-life
challenge. To address this challenge, we proposed to integrate an entropy loss
into established state-of-the-art SSL frameworks, exemplified by _FlexMatch_.
This strategic enhancement serves as a potent remedy for the absence of
labeled instances corresponding to unseen classes. This method will enable us
to convert any state-of-the-art SSL algorithm to readily face this challenge.

Our extensive empirical evaluation substantiates the efficacy of our approach
in resolving the aforementioned challenge when juxtaposed against alternative
baselines, including SSL and open-set SSL (adapted to suit our scenario as
discussed in Section 2.4), as well as open-world SSL. Notably, our algorithm
emerges as a standout performer, particularly within the realm of limited
labeled data scenarios.

In future work, we will investigate the dynamic incorporation of the entropy
loss throughout the training process. This adaptive integration holds promise
for enhancing the robustness and versatility of our approach.

## Acknowledgement

This work was supported by grants from the Israeli Council of Higher Education
and the Gatsby Charitable Foundations.

## References

  * Berthelot et al. (2019) David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. 2019.  Mixmatch: A holistic approach to semi-supervised learning.  Advances in neural information processing systems 32\.
  * Cao et al. (2022) Kaidi Cao, Maria Brbic, and Jure Leskovec. 2022.  Open-world semi-supervised learning.  In International Conference on Learning Representations (ICLR).
  * Chapelle et al. (2009) Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009.  Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews].  IEEE Transactions on Neural Networks 20(3):542–542.
  * Chen et al. (2022) Yanbei Chen, Massimiliano Mancini, Xiatian Zhu, and Zeynep Akata. 2022.  Semi-supervised and unsupervised deep visual learning: A survey.  IEEE transactions on pattern analysis and machine intelligence .
  * Chen et al. (2020) Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. 2020.  Semi-supervised learning under class distribution mismatch.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, pages 3569–3576.  https://ojs.aaai.org/index.php/AAAI/article/view/5763.
  * Coates et al. (2011) Adam Coates, Andrew Ng, and Honglak Lee. 2011.  An analysis of single-layer networks in unsupervised feature learning.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, pages 215–223.
  * Guo et al. (2022) Lan-Zhe Guo, Yi-Ge Zhang, Zhi-Fan Wu, Jie-Jing Shao, and Yu-Feng Li. 2022.  Robust semi-supervised learning when not all classes have labels.  Advances in Neural Information Processing Systems 35:3305–3317.
  * Guo et al. (2020) Lan-Zhe Guo, Zhen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. 2020.  Safe deep semi-supervised learning for unseen-class unlabeled data.  In International Conference on Machine Learning. PMLR, pages 3897–3906.
  * Han et al. (2020) K Han, SA Rebuffi, S Ehrhardt, A Vedaldi, and A Zisserman. 2020.  Automatically discovering and learning new visual categories with ranking statistics.  In Proceedings of the 8th Intennational Conference on Learning Representations, ICLR 2020. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
  * Huang et al. (2021) Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. 2021.  Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 8310–8319.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009.  Learning multiple layers of features from tiny images .
  * Li et al. (2015) Xin Li, Yuhong Guo, and Dale Schuurmans. 2015.  Semi-supervised zero-shot classification with label representation learning.  In Proceedings of the IEEE international conference on computer vision. pages 4211–4219.
  * Nguyen et al. (2015) Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 427–436.
  * Park et al. (2022) Jongjin Park, Sukmin Yun, Jongheon Jeong, and Jinwoo Shin. 2022.  Opencos: Contrastive semi-supervised learning for handling open-set unlabeled data.  In European Conference on Computer Vision. Springer, pages 134–149.
  * Pourpanah et al. (2022) Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. 2022.  A review of generalized zero-shot learning methods.  IEEE transactions on pattern analysis and machine intelligence .
  * Rizve et al. (2022) Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. 2022.  Openldn: Learning to discover novel classes for open-world semi-supervised learning.  In European Conference on Computer Vision. Springer, pages 382–401.
  * Saito et al. (2021) Kuniaki Saito, Donghyun Kim, and Kate Saenko. 2021.  Openmatch: Open-set consistency regularization for semi-supervised learning with outliers.  CoRR abs/2105.14148.  https://arxiv.org/abs/2105.14148.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  Advances in neural information processing systems 33:596–608.
  * Sun and Li (2023) Yiyou Sun and Yixuan Li. 2023.  Opencon: Open-world contrastive learning.  Trans. Mach. Learn. Res. 2023\.  https://openreview.net/forum?id=2wWJxtpFer.
  * Tax and Duin (2008) David MJ Tax and Robert PW Duin. 2008.  Growing a multi-class classifier with a reject option.  Pattern Recognition Letters 29(10):1565–1570.
  * Wang et al. (2022) Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. 2022.  Usb: A unified semi-supervised learning benchmark for classification.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.  https://doi.org/10.48550/ARXIV.2208.07204.
  * Wang et al. (2023) Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. 2023.  Freematch: Self-adaptive thresholding for semi-supervised learning.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.  https://openreview.net/pdf?id=PDrUPTXJI_A.
  * Wang et al. (2020) Zifeng Wang, Batool Salehi, Andrey Gritsenko, Kaushik Chowdhury, Stratis Ioannidis, and Jennifer Dy. 2020.  Open-world class discovery with kernel networks.  In 2020 IEEE International Conference on Data Mining (ICDM). IEEE, pages 631–640.
  * Xie et al. (2016) Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016.  Unsupervised deep embedding for clustering analysis.  In International conference on machine learning. PMLR, pages 478–487.
  * Xu et al. (2021) Bingrong Xu, Zhigang Zeng, Cheng Lian, and Zhengming Ding. 2021.  Semi-supervised low-rank semantics grouping for zero-shot learning.  IEEE Transactions on Image Processing 30:2207–2219.
  * Yu et al. (2020) Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. 2020.  Multi-task curriculum framework for open-set semi-supervised learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16. Springer, pages 438–454.
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. 2016.  Wide residual networks.  arXiv preprint arXiv:1605.07146 .
  * Zhang et al. (2021) Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. 2021.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. pages 18408–18419.  https://proceedings.neurips.cc/paper/2021/hash/995693c15f439e3d189b06e89d145dd5-Abstract.html.
  * Zhong et al. (2021) Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. 2021.  Openmix: Reviving known knowledge for discovering novel visual categories in an open world.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 9462–9470.

## Appendix

##  Appendix A Additional Results

###  A.1 Few unseen classes

In this section, we conduct complementary experiments to the ones performed in
Section 3. The advantage of our suggested method stems from its ability to
learn the unseen classes, suggesting that in settings with only a few unseen
classes, our relative advantage will decrease. In Fig. 7, we repeat the
experiment done in Fig. 2(c), training WRN on CIFAR-100, while reducing the
number of unseen classes from $35$ to $10$. As expected, the advantage of our
suggested method indeed decreases, but our method still outperforms all
baselines by a large and significant margin.

(a) Combined score

(b) Seen and unseen accuracy

Figure 7: Extending Fig. 2(c) for small number of unseen classes. We trained
WRN on CIFAR-100, with 25 labels per class and 10 unseen classes. We see that
while the advantages are lower than the many unseen classes case, our method
shows a significant boost in performance.

###  A.2 Few labeled examples

In Fig. 2, we experimented with our suggested method using either $1$ or $4$
labels per class. These choices were made to show that the suggested method
works where $\mathcal{L}$ is small, and the specific values were selected for
better comparison to previous art. Here, we show that other sizes of
$\mathcal{L}$ could be chosen. In Fig. 8, similarly to Fig. 2, we train WRN on
CIFAR-100, changing the labeled set to contain either $2$ or $3$ labels per
class.

(a) 2 labels per class

(b) 3 labels per class

Figure 8: Combined score, CIFAR-100, 35 unseen classes.