# Measurements of the neutron timelike electromagnetic form factor with the
SND detector

M. N. Achasov  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk
630090, Russia  Novosibirsk State University, Novosibirsk 630090, Russia  A.
Yu. Barnyakov  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk
630090, Russia  Novosibirsk State University, Novosibirsk 630090, Russia  E.
V. Bedarev  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  Novosibirsk State University, Novosibirsk 630090, Russia  K. I.
Beloborodov  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  Novosibirsk State University, Novosibirsk 630090, Russia  A. V.
Berdyugin  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  Novosibirsk State University, Novosibirsk 630090, Russia  D. E.
Berkaev  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  A. G. Bogdanchikov  Budker Institute of Nuclear Physics, SB RAS,
Novosibirsk 630090, Russia  A. A. Botov  Budker Institute of Nuclear Physics,
SB RAS, Novosibirsk 630090, Russia  T. V. Dimova  Budker Institute of Nuclear
Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State University,
Novosibirsk 630090, Russia  V. P. Druzhinin  Budker Institute of Nuclear
Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State University,
Novosibirsk 630090, Russia  V. N. Zhabin  Budker Institute of Nuclear Physics,
SB RAS, Novosibirsk 630090, Russia  Novosibirsk State University, Novosibirsk
630090, Russia  Yu. M. Zharinov  Budker Institute of Nuclear Physics, SB RAS,
Novosibirsk 630090, Russia  L. V. Kardapoltsev  Budker Institute of Nuclear
Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State University,
Novosibirsk 630090, Russia  A. S. Kasaev  Budker Institute of Nuclear Physics,
SB RAS, Novosibirsk 630090, Russia  D. P. Kovrizhin  Budker Institute of
Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  I. A. Koop  Budker
Institute of Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk
State University, Novosibirsk 630090, Russia  A. A. Korol  Budker Institute of
Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State
University, Novosibirsk 630090, Russia  A. S. Kupich  Budker Institute of
Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State
University, Novosibirsk 630090, Russia  A. P. Kryukov  Budker Institute of
Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  A. P. Lysenko  Budker
Institute of Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  N. A.
Melnikova  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  Novosibirsk State University, Novosibirsk 630090, Russia  N. Yu.
Muchnoi  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  Novosibirsk State University, Novosibirsk 630090, Russia  A. E.
Obrazovsky  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  E. V. Pakhtusova  Budker Institute of Nuclear Physics, SB RAS,
Novosibirsk 630090, Russia  K. V. Pugachev  Budker Institute of Nuclear
Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State University,
Novosibirsk 630090, Russia  S. A. Rastigeev  Budker Institute of Nuclear
Physics, SB RAS, Novosibirsk 630090, Russia  Yu. A. Rogovsky  Budker Institute
of Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State
University, Novosibirsk 630090, Russia  S. I. Serednyakov  Budker Institute of
Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State
University, Novosibirsk 630090, Russia  Z. K. Silagadze  Budker Institute of
Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  Novosibirsk State
University, Novosibirsk 630090, Russia  I. K. Surin  Budker Institute of
Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  Yu. V. Usov  Budker
Institute of Nuclear Physics, SB RAS, Novosibirsk 630090, Russia  A. G.
Kharlamov  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  Novosibirsk State University, Novosibirsk 630090, Russia  Yu. M.
Shatunov  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk 630090,
Russia  D. A. Shtol  Budker Institute of Nuclear Physics, SB RAS, Novosibirsk
630090, Russia

###### Abstract

The results of the measurement of the $e^{+}e^{-}\to n\bar{n}$ cross section
and effective neutron timelike form factor are presented. The data taking was
carried out in 2020–2021 at the VEPP-2000 $e^{+}e^{-}$ collider in the center-
of-mass energy range from 1891 to 2007 MeV. The general purpose nonmagnetic
detector SND is used to detect neutron-antineutrons events. The selection of
$n\bar{n}$ events is performed using the time-of-flight technique. The
measured cross section is 0.4–0.6 nb. The neutron form factor in the energy
range under study varies from 0.3 to 0.2.

## Introduction

The internal structure of nucleons is described by electromagnetic
formfactors. In the timelike region they are measured using the process of
$e^{+}e^{-}$ annihilation to nucleon-antinucleon pairs. The $e^{+}e^{-}\to
n\bar{n}$ cross section depends on two formfactors - electric $G_{E}$ and
magnetic $G_{M}$ :

$\displaystyle\frac{d\sigma}{d\Omega}$ $\displaystyle=$
$\displaystyle\frac{\alpha^{2}\beta}{4s}\bigg{[}|G_{M}(s)|^{2}(1+\cos^{2}\theta)$
(1) $\displaystyle+$
$\displaystyle\frac{1}{\gamma^{2}}|G_{E}(s)|^{2}\sin^{2}\theta\bigg{]}$

where $\alpha$ is the fine structure constant, $s=4E_{b}^{2}=E^{2}$, where
$E_{b}$ is the beam energy and $E$ is the center-of-mass (c.m.) energy,
$\beta=\sqrt{1-4m_{n}^{2}/s}$, $\gamma=E_{b}/m_{n}$, $m_{n}$ is the neutron
mass and $\theta$ is the antineutron production polar angle. The total cross
section has the following form:

$\sigma(s)=\frac{4\pi\alpha^{2}\beta}{3s}(1+\frac{1}{2\gamma^{2}})|F(s)|^{2},$
(2)

where the effective form factor $F(s)$ is introduced:

$|F(s)|^{2}=\frac{2\gamma^{2}|G_{M}(s)|^{2}+|G_{E}(s)|^{2}}{2\gamma^{2}+1}.$
(3)

The $|G_{E}/G_{M}|$ ratio can be extracted from the analysis of the measured
$\cos\theta$ distribution in Eq. (1). At the threshold $|G_{E}|=|G_{M}|$.

The latest results on the neutron form factor near the threshold were obtained
in experiments at the VEPP-2000 $e^{+}e^{-}$ collider with the SND detector
SNND . The same work provides a list of previous measurements. At the energy
above 2 GeV new data have been obtained by the BESIII BES . In this work the
recent SND results on the $e^{+}e^{-}\to n\bar{n}$ cross section and the
neutron timelike formfactor with 4 times higher integrated luminosity than in
previous measurement SNND , are presented.

Figure 1: SND detector, section along the beams: (1) beam pipe, (2) tracking
system, (3) aerogel Cherenkov counters, (4) NaI (Tl) crystals, (5) vacuum
phototriodes, (6) iron absorber, (7) proportional tubes, (8) iron absorber,
(9) scintillation counters, (10) VEPP-2000 focusing solenoids.

##  I Collider, detector, experiment

VEPP-2000 is $e^{+}e^{-}$ collider VEPP2k  operating in the energy range from
the hadron threshold ($E$=280 MeV) up to 2 GeV. The collider luminosity above
the nucleon threshold at 1.87 GeV is of order of $5\times 10^{31}$ cm-2s-1.
There are two collider detectors at VEPP-2000: SND and CMD-3.

SND (Spherical Neutral Detector) SNDet1  is a non-magnetic detector, including
a tracking system, a spherical NaI(Tl) electromagnetic calorimeter (EMC) and a
muon detector (Fig.1). The EMC is the main part of the SND used in the
$n\bar{n}$ analysis. The thickness of EMC is 34.7 cm (13.4 radiation length).
The antineutron annihilation length in NaI(Tl) varies with energy from several
cm close to the $n\bar{n}$ threshold to $\sim$15 cm at the maximum available
energy Annih , so nearly all produced antineutrons are absorbed in the
detector.

The EMC is used to measure the event arrival time. Starting from 2019 a system
of flash ADC modules Timr , measuring the signal waveform, is installed on
each of the 1640 EMC counters. When fitting the flash ADC output waveform, the
time and amplitude of the signal in the counters are calculated. The event
time is calculated as the energy weighted average time. The time resolution
obtained with $e^{+}e^{-}\to\gamma\gamma$ events is about 0.8 ns.

This article presents the analysis results of data with the integrated
luminosity of $80$ pb-1, collected in the energy range $1.89$ \- $2.0$ GeV in
8 energy points.

##  II Selection of $n\bar{n}$ events

Antineutron from the $n\bar{n}$ pair in most cases annihilates, producing
pions, nucleons, photons and other particles, which deposit up to 2 GeV in
EMC. The neutron from the $n\bar{n}$ pair release a small signal in EMC, which
poorly visible against the background of a strong $\bar{n}$ annihilation
signal, so it is not taken into account. The $n\bar{n}$ events are
reconsructed as multiphoton events.

Main features of $n\bar{n}$ events are absence of charged tracks and photons
from the collision region and a strong imbalance in the event momentum. To
create the $n\bar{n}$ selection conditions we consider the sources of the
background including the cosmic background, the background from $e^{+}e^{-}$
annihilation processes and from the electron and positron beams in the
collider.

Based on these specific features of the $e^{+}e^{-}\to n\bar{n}$ process,
selection conditions were divided into three groups.

In the first group the conditions are collected that suppress the background
from the $e^{+}e^{-}$ annihilation events. These include the condition of no
charged tracks in an event, the limit on the total event momentum
(p/2$E_{b}>$0.4), and a limitation on the transverse shower profile in EMC
xi2gam , which must be wider than that from the photons from the collision
region.

In the second group the selection conditions should suppress the cosmic
background. Here the veto of muon system is included and special conditions,
analyzing the energy deposition shape in EMC and removing cosmic events
passing through the muon veto SNND . Basically, these are cosmic showers in
EMC.

The third group of selection cuts contains the restriction on the total energy
deposition in EMC — $E_{dep}>E_{b}$. Such a restriction almost completely
suppresses the beam background, although the detection efficiency also
decreases by about 20%.

The listed selection conditions are similar to those described in our recent
work SNND . The only difference is that there is no limitation on the energy
in the EMC third layer. This slightly increased the detection efficiency,
although it did increase the cosmic background. After imposing the described
selection conditions, we have about 400 events/$pb$ left for further analysis.

Figure 2: The time distribution of selected data events (points with error
bars) at $E_{b}=945$ MeV (left panel) and at $E_{b}=980$ MeV (right panel).
The position $t=0$ corresponds to the moment of beams collision. The wide peak
to the right is the contribution of $n\bar{n}$ events. The light-shaded
histogram shows the cosmic background (uniform in time) and beam background
(peak at $t=0$). The solid line is the result of the fit.

##  III Getting the number of $n\bar{n}$ events

The time spectra for selected data events are shown in Fig. 2. Zero time
corresponds to events in the moment of beam collision. Three main components
are distinguished in the time spectrum in the figures shown: a beam background
at t=0, a cosmic background uniform in time, and a delayed signal from
$n\bar{n}$ events, wide in time. Respectively, the measured time spectra are
fitted by the sum of these three components in the following form :

$F(t)=N_{n\bar{n}}H_{n\bar{n}}(t)+N_{\rm csm}H_{\rm csm}(t)+N_{\rm bkg}H_{\rm
bkg}(t),$ (4)

where $H_{n\bar{n}}$, $H_{\rm csm}$ and $H_{\rm bkg}$ are normalized
histograms, describing time spectra for the $n\bar{n}$ signal, cosmic and beam
+ physical background, respectively. $N_{n\bar{n}}$, $N_{\rm csm}$, and
$N_{\rm bkg}$ are the corresponding event numbers, obtained from the fit. The
shape of the beam+physical background time spectrum $H_{\rm bkg}$ is measured
at the energies below the $n\bar{n}$ threshold. The cosmic time spectrum
$H_{\rm csm}$ is measured with the lower EMC threshold $0.9\cdot E_{b}$ in
coincidence with the muon system signal. The $H_{n\bar{n}}$ spectrum is
calculated by the MC simulation the $e^{+}e^{-}\to n\bar{n}$ process.

Comparison of time spectra in data and MC gives a wider data time distribution
for both $e^{+}e^{-}\to\gamma\gamma$ and $e^{+}e^{-}\to n\bar{n}$ events. For
the $e^{+}e^{-}\to\gamma\gamma$ this is due to the finite time resolution of
our timing system Timr , which is not adequately simulated. So we convolve the
MC time spectrum with a Gaussian with $\sigma_{\gamma\gamma}=$ 0.8 ns. For
$e^{+}e^{-}\to n\bar{n}$ events the covolution is done with
$\sigma_{nn}=1.5$–2 ns depending on the energy.

Moreover, in addition to the above, we correct the MC ${n\bar{n}}$ time
spectrum, since the shape of MC time spectrum $H_{n\bar{n}}$ does not describe
data well. This discrepancy is explained by the incorrect relationship between
the processes of antineutron annihilation and scattering in MC, as well as by
the incorrect description of the annihilation products. To modify the MC time
spectrum, separate MC time spectra were plotted for the cases of the first
$\bar{n}$ interaction of scattering ($H_{n\bar{n}}^{s}$) and annihilation
($H_{n\bar{n}}^{a}$). The share of annihilation events in MC was about 33%.
The annihilation gives the time spectrum close to the exponential while the
scattering has delayed and more wide time spectrum with the non exponential
shape. The $H_{n\bar{n}}$ spectrum (Eq.4) was taken in fit as a linear sum of
two spectra described above: $H_{n\bar{n}}=\alpha
H_{n\bar{n}}^{a}+(1-\alpha)H_{n\bar{n}}^{s}$ . The value $\alpha$ (the share
of annihilation events) was the fit parameter. As a result of the fit this
parameter turned out to be greater than in MC – $\simeq$60% and accordingly
the proportion of scattering fell to $\simeq$40%. As can be seen in Fig.2, the
modified MC time spectrum describes the data well.

The visible cross section $\sigma_{bg}$ of the beam+physical background,
obtained during fitting, is about 7 pb and does not significantly depend on
the beam energy. The main contribution into $\sigma_{bg}$ comes from the
processes with neutral kaons in the final state: $e^{+}e^{-}\to
K_{S}K_{L}\pi^{0}$, $K_{S}K_{L}\eta$ and similar other. The measured residual
cosmic background rate has the intensity $\sim$0.01 Hz, which corresponds to
the suppression of the number of cosmic events, that have pass the hardware
selection in the detector electronics, approximately by $2\times 10^{4}$
times.

Table 1: The beam energy ($E_{b}$), integrated luminosity ($L$), number of selected $n\bar{n}$ events ($N_{n\bar{n}}$), the factor taking into account radiative corrections and energy spread ($1+\delta$), corrected detection efficiency ($\varepsilon$), measured $e^{+}e^{-}\to n\bar{n}$ cross section $\sigma$, and neutron effective form factor ($F_{n}$). The quoted errors for $N$, $\sigma$ are statistical and systematic. For the detection efficiency, the systematic uncertainty is quoted. For $F_{n}$, the combined statistical and systematic uncertainty is listed.  N |  $E_{b}$(MeV) |  $L$(pb) | $N_{n\bar{n}}$ | $1+\delta$ | $\varepsilon$ |  $\sigma$(nb) | $F_{n}$
---|---|---|---|---|---|---|---
1 | 945.5 | 8.54 | $676\pm 37$ | 0.746 | $0.253\pm 0.021$ | $0.420\pm 0.023\pm 0.036$ | $0.322\pm 0.016$
2 | 950.3 | 8.86 | $834\pm 37$ | 0.787 | $0.246\pm 0.015$ | $0.485\pm 0.022\pm 0.031$ | $0.301\pm 0.012$
3 | 960.3 | 8.33 | $767\pm 35$ | 0.840 | $0.217\pm 0.013$ | $0.506\pm 0.023\pm 0.032$ | $0.266\pm 0.010$
4 | 970.8 | 8.07 | $718\pm 34$ | 0.870 | $0.229\pm 0.017$ | $0.447\pm 0.021\pm 0.034$ | $0.230\pm 0.011$
5 | 968.8 | 5.51 | $524\pm 34$ | 0.870 | $0.186\pm 0.020$ | $0.589\pm 0.039\pm 0.065$ | $0.267\pm 0.017$
6 | 980.3 | 7.70 | $654\pm 37$ | 0.900 | $0.216\pm 0.018$ | $0.436\pm 0.025\pm 0.038$ | $0.216\pm 0.011$
7 | 990.4 | 8.77 | $624\pm 38$ | 0.920 | $0.183\pm 0.019$ | $0.422\pm 0.026\pm 0.045$ | $0.204\pm 0.013$
8 | 1003.5 | 20.06 | $1075\pm 50$ | 0.947 | $0.151\pm 0.014$ | $0.374\pm 0.018\pm 0.035$ | $0.186\pm 0.010$

The numbers of found $n\bar{n}$ events are listed in the Table 1 with the
total number close to 6000. The Table shows only statistical errors of the
fitting. A source of systematic error in the $n\bar{n}$ event number can be
uncertainties in the magnitide and shape of the time spectrum of the beam and
cosmic background. The error introduced by these sources is 15 events at
$E_{b}$=1000 MeV and less than 8 events at lower energies. These values are
much lower than statitistical errors in the Table 1 and are not taken into
account in what follows.

Figure 3: The antineutron $\cos\theta_{a}$ distribution for data (points with
error bars) and MC (horizontal line) at $E_{b}=970$ MeV (left panel) and
$E_{b}=1000$ MeV (right panel). Dotted vertical lines correspond to the polar
angle cutoff.

Figure 4:  The corrected detection efficiency versus energy.

Figure 5: The MC detection efficiency as a function of antineutron
$\cos{\theta}$ at $E_{b}$=960 MeV. Dotted vertical lines correspond to the
polar angle cutoff.

##  IV Antineutron angular distribution

The antineutron production angle $\theta_{n}$ is determined by the direction
of the event momentum with an accuracy of about 5 degrees. Distribution over
$\cos\theta_{n}$ for data and MC events is shown in Fig. 3. The MC simulation
was done using Eq. (1) with the assumption $G_{E}=G_{M}$. The detection
efficiency in the selection interval $36^{\circ}<\theta_{n}<144^{\circ}$ is
80%. It is seen from the Fig. 3 that the data and MC distributions agree well
with each other, which confirms the MC angular model. It is also worth noting
that the previous measurements of the $|G_{E}/G_{M}|$ value SNND  also do not
contradict the hypothesis $G_{E}=G_{M}$.

##  V Detection efficiency

The detection efficiency $\varepsilon$ versus energy under accepted selection
conditions (Section II) is shown in Fig. 5. When calculating $\varepsilon$ we
used the MC simulation of the $e^{+}e^{-}\to n\bar{n}$ process with the GEANT4
toolkit GEANT4 , version 10.5. In addition, the simulation included the beam
energy spread $\sim~{}1$ MeV and the emission of photons by initial electrons
and positrons. The simulation also took into account non-operating detector
channels as well as overlaps of the beam background with recorded events. To
do this, during the experiment, with a pulse generator, synchronized with the
moment of beam collision, special superposition events were recorded, which
were subsequently superimposed on MC events. The detection efficiency
$\varepsilon$ in Fig. 5 is corrected for the difference between the data and
MC. This correction is discussed later. Numerical values of the efficiency are
given in the Table 1. A decrease of $\varepsilon$ with energy can be explained
by the energy dependence of the selection parameters, as well as by an
increase in the energy that goes beyond the calorimeter. In Fig. 5 the angular
detection efficiency is shown at the beam energy $E_{b}=960$ MeV.

Figure 6:  The event time spectra with inverted selection conditions. Left
panel — inverted 2-nd group cut at $E_{b}=970$ MeV, Right panel — inverted 3-d
group cut with $0.7E_{b}<E_{\rm cal}<E_{b}$ at $E_{b}=960$ MeV. Light shaded
histogram shows the background distribution. Dark shaded histogram in the
right plot is $n\bar{n}$ contribution.

The detection efficiency in our measurement is of order of 20%. It is
important to find out how correctly the proportion of events outside the
selection condition is simulated. Corrections were calculated fot three groups
of selection conditions described in chapter II. To do this, we invert the
selection conditions for each selection group and then calculate the
corresponding corrections $\delta$ for detection efficiency in each of 8
energy points as follows:

$\delta=\frac{n_{0}}{n_{0}+n_{1}}\frac{m_{0}+m_{1}}{m_{0}},$ (5)

where $n_{0}$ ($n_{1}$) is the number of $n\bar{n}$ events determined with
standard (inverted) selection cuts. These numbers were calculated during the
time spectra fitting with the Eq.4, as it is described in the chapter III. The
values $m_{0}$ and $m_{1}$ refer respectively to the MC simulation event
numbers. Examples of the time spectra obtained with inverted conditions are
shown in Fig. 6.

Table 2:  The beam energy ($E_{b}$), the correction to the detection efficiency $\delta_{1}$ from SND internal system, the correction $\delta_{2}$ from SND external system, the correction $\delta_{3}$ from the EMC thresholds, the correction $\delta_{E}$ from the lost EMC energy, the correction $\delta_{t}$ is the total correction.  |  $E_{b}$ (MeV) | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$ | $\delta_{E}$ | $\delta_{t}$
---|---|---|---|---|---|---
1 | 945.5 | $0.991\pm 0.022$ | $1.292\pm 0.092$ | $0.971\pm 0.038$ | $1.005\pm 0.005$ | $1.249\pm 0.102$
2 | 950.3 | $0.977\pm 0.018$ | $1.214\pm 0.062$ | $0.985\pm 0.030$ | $1.009\pm 0.009$ | $1.179\pm 0.072$
3 | 960.3 | $9.966\pm 0.019$ | $1.077\pm 0.050$ | $0.992\pm 0.028$ | $1.012\pm 0.012$ | $1.044\pm 0.062$
4 | 970.8 | $0.949\pm 0.021$ | $1.198\pm 0.061$ | $0.980\pm 0.050$ | $1.018\pm 0.018$ | $1.134\pm 0.084$
5 | 968.8 | $0.958\pm 0.027$ | $1.031\pm 0.080$ | $0.896\pm 0.044$ | $1.018\pm 0.018$ | $0.901\pm 0.097$
6 | 980.3 | $0.997\pm 0.031$ | $1.102\pm 0.073$ | $0.986\pm 0.043$ | $1.021\pm 0.021$ | $1.106\pm 0.093$
7 | 990.4 | $0.925\pm 0.033$ | $1.131\pm 0.080$ | $0.889\pm 0.041$ | $1.024\pm 0.024$ | $0.952\pm 0.099$
8 | 1003.5 | $0.915\pm 0.024$ | $1.065\pm 0.056$ | $0.796\pm 0.028$ | $1.028\pm 0.028$ | $0.797\pm 0.073$

The first group of selection conditions includes the requirement of no charged
tracks in an event. When studying inverse selections we assume the presence of
central charged tracks with $D_{xy}>0.5$ cm, where $D_{xy}$ is the distance
between the track and the axis of the beams. A possible background from the
related process $e^{+}e^{-}\to p\bar{p}$ should be discussed here. In the
energy region $E_{b}>960$ MeV protons and antiproptons give central collinear
tracks and rejected by the requirement $D_{xy}>0.5$ cm, as well as events of
other processes of $e^{+}e^{-}$ annihilation with charged tracks. However at
$E_{b}<960$ MeV the protons and antiproptons are slow and stop at the collider
vacuum pipe. In this case the antiproton annihilates with the production of
charged tracks, wchich can be with $D_{xy}>0.5$ cm. But here, too, the
$e^{+}e^{-}\to p\bar{p}$ background is suppressed by the fitting of time
spectrum, since the annihilation delay time does not exceed 1 ns. For the
second group the inverted selection conditions were used without changes. For
the third group of selection conditions, a partial inversion was used, that
is, the condition $0.7E_{b}<E_{\rm cal}<E_{b}$ was applied.

An additional correction arises from the events, in which the antineutrons
pass the calorimeter without interaction, and from the events with a small
calorimeter energy $E_{\rm cal}<0.7E_{b}$. These events are not taken into
analysis due to the large background and therefore not available for
correction with the procedure described above. Their share in MC varies from
1.9% at the energy $E_{b}$=945 MeV to 8.5% at $E_{b}$=1000 MeV. It was
previously noted in chapter III, that to desribe the shape of data time
spectrum the contribution of the process of antineutron scattering in MC
should be reduced by a factor of 1.5. With such a change, the proportion of
events with $E_{\rm cal}<0.7E_{b}$ in MC reduces to 1.4% at $E_{b}$=945 to
5.7% at $E_{b}$=1000 MeV. The difference between these values is taken into
account as an additional correction $\delta_{E}$ to the detection efficiency
with the 100% of uncertainty.

The measured by selection groups corrections
$\delta_{1},\delta_{2},\delta_{3}$, as well as $\delta_{E}$, are multiplied
$\delta_{t}=\delta_{1}\delta_{2}\delta_{3}\delta_{E}$ and all are given in the
Table 2. It can be seen, that the total efficiency correction $\delta_{t}$
changes in limits 0.8—1.25 with energy, what is explained by the strong energy
dependence of the antineutron absobtion length.

The corrected detection efficiency is obtained from the MC efficiency by
multiplying by the total correction $\delta_{t}$. The values of the corrected
efficiency are given along with systematic errors in the Table 1. Here, unlke
our previous measurement SNND , the corrections in different energy points are
not correlated.

##  VI The measured $e^{+}e^{-}\to n\bar{n}$ cross section

Figure 7: The $e^{+}e^{-}\to n\bar{n}$ cross section measured in this work in
comparison with previous SND measurement SNND  and BESIII BES  data. The solid
curve is the fit result. Only new SND and BESIII data were used in the fit.

Figure 8: The measured neutron timelike formfactor (solid black points)
compared with BESIII BES  (red stars) and BABAR Babar  proton formfactor (blue
empty squares) The solid line is the polinomial fit, described in the text.

Using the number of $n\bar{n}$ events $N_{n\bar{n}}$ , luminosity $L$ and
detection efficiency $\varepsilon$ (Table 1), the visible cross section
$\sigma_{vis}(E)=N_{n\bar{n}}/L\varepsilon$ can be calculated. The Born cross
section $\sigma(E)$ we need is related to the visible cross $\sigma_{vis}(E)$
in the following form :

$\displaystyle\sigma_{vis}(E)$ $\displaystyle=$
$\displaystyle\sigma(E)(1+\delta(E))$ (6) $\displaystyle=$
$\displaystyle\int_{-\infty}^{+\infty}G(E^{\prime},E)dE^{\prime}$
$\displaystyle\int_{0}^{x_{max}}W(s,x)\sigma(s(1-x))dx,$

where $W(s,x)$ is the radiator function Radcor  describing emission of photons
with energy $xE_{b}$ by initial electrons and positrons, $G(E^{\prime},E)$ is
a Gaussian function describing the c.m. energy spread. In function $W(s,x)$
the contribution of the vacuum polarization is not taken into account, so our
Born cross section is a “dressed” cross section. The factor $(1+\delta(E))$
takes into account both the radiative corrections and beam energy spread. This
factor is calculated in each of 8 energy points using the Born cross section,
obtained by the fitting of the visible cross section using Eq. 6. The energy
dependence of the Born cross section is described by Eq.2, in which the
neutron effective formfactor has a form of a second order polinomial function,
as shown in more detail in the next chapter.

The measured Born cross section is shown in the Fig. 8 and listed in the Table
1. The dominant contribution into systematic error is made by the detection
efficiency correction error, given in the Table 2. Uncertainties in the value
of luminosity (1%) and radiative correction (2%) are also taken into account.
In Fig. 8 the total statistical and systematic error is shown. In comparison
with our preceding work SNND , the measured cross section has 2 times lower
statistical error and 1.5 times lower systematic error. At the maximum energy
$E=2$ GeV our cross section is in good agreement with the last BESIII
measurement BES .

##  VII The neutron effective timelike formfactor

The effective neutron form factor calculated from the measured cross section
using Eq. (2) is listed in the Table 1 and shown in Fig. 8 as a function of
the antineutron momentum together with the BESIII data BES  and the proton
effective form factor measured by the BABAR experiment Babar . The curve in
Fig. 8, approximating the form factor, is a second order polinimial
$|F_{n}|=a_{0}+a_{1}p_{n}+a_{2}p_{n}^{2}$, in which the parameters $a_{i}$ are
obtained during fitting and $p_{n}$ is the antineutron momentum. The following
parameters values are obtained: $a_{0}=0.398\pm 0.022$, $a_{1}=-0.713\pm
0.126$, $a_{2}=0.268\pm 0.166$. When the fitting curve continues to zero
momentum, the expected value of the neutron formfactor at the threshold will
be $a_{0}\simeq$0.4. One can see from the Fig. 8, that the proton formfactor
noticably larger that the neutron one and their ratio near the threshold could
be close to $3/2$.

##  VIII Summary

The experiment to measure the $e^{+}e^{-}\to n\bar{n}$ cross section and the
neutron timelike form factor has been carried out with the SND detector at the
VEPP-2000 $e^{+}e^{-}$ collider in the energy region from 1891 to 2007 MeV.
The measured $e^{+}e^{-}\to n\bar{n}$ cross varies with energy within
0.4$\div$0.6 nb and agrees with recent SND measurement SND SNND , however has
2 times better statistical accuracy. At the maximum energy our cross section
is in agreement with the last BESIII measurement BES . The neutron effective
timelike form factor is extracted from the measured cross section using Eq.2.
Form factor decreases with energy from 0.3 to 0.2. In the value, the neutron
form factor turns out to be noticeably less than the proton one.

ACKNOWLEDGMENTS. This work was carried out on the RSF fund grant No.
23-22-00011.

## References

  * (1) M. N. Achasov et al. (SND Collaboration), Eur. Phys. J. C (2022:82:761). http://doi.org/10.1140/epjc/s10052-022-10696-0
  * (2) M. Ablikim et al. (BESIII Collaboration), Nat. Phys. 17, 1200 (2021). https://doi.org/10.1038/s41567-021-01345-6
  * (3) P. Yu. Shatunov et al., Part. Nucl. Lett. 13, 995 (2016). http://dx.doi.org/10.1134/S154747711607044X
  * (4) M. N. Achasov et al. (SND Collaboration), Nucl. Instrum. Meth. A 449, 125 (2000). http://dx.doi.org/10.1016/S0168-9002(99)01302-9
  * (5) M. Astrua et al., Nucl. Phys. A 697, 209 (2002). http://dx.doi.org/10.1016/S0375-9474(01)01252-0
  * (6) M. N. Achasov et al., JINST 10, T06002 (2015). http://dx.doi.org/10.1088/1748-0221/10/06/T06002
  * (7) A. V. Bozhenok et al., Nucl. Instr. Meth. A 379, 507 (1996). http://dx.doi.org/10.1016/0168-9002(96)00548-7
  * (8) J. Allison et al. (GEANT Collaboration), Nucl. Instr. Meth. A 835, 186 (2016). https://doi.org/10.1016/j.nima.2016.06.125, https://geant4-data.web.cern.ch/ReleaseNotes/ReleaseNotes4.10.5.html
  * (9) E. A. Kuraev and V. S. Fadin, Sov. J. Nucl. Phys. 41, 466 (1985).
  * (10) J. P. Lees et al. (BABAR Collaboration), Phys. Rev. D 87, 092005 (2013). Phys. Rev. D 88, 072009 (2013). http://dx.doi.org/10.1103/PhysRevD.87.092005 http://dx.doi.org/10.1103/PhysRevD.88.072009


# Eliminating Information Leakage in Hard Concept Bottleneck Models with
Supervised, Hierarchical Concept Learning

Ao Sun, Yuanyuan Yuan∗, Pingchuan Ma, and Shuai Wang∗
The Hong Kong University of Science and Technology
{asunac, yyuanaq, pmaab<EMAIL_ADDRESS>
###### Abstract

Concept Bottleneck Models (CBMs) aim to deliver interpretable and
interventionable predictions by bridging features and labels with human-
understandable concepts. While recent CBMs show promising potential, they
suffer from information leakage, where unintended information beyond the
concepts (either when concepts are represented with probabilities or binary
states) is leaked to the subsequent label prediction. Consequently, distinct
classes are falsely classified via indistinguishable concepts, undermining the
interpretation and intervention of CBMs.

This paper alleviates the information leakage issue by introducing label
supervision in concept prediction and constructing a hierarchical concept set.
Accordingly, we propose a new paradigm of CBMs, namely SupCBM, which achieves
label prediction via predicted concepts and a deliberately-designed
intervention matrix. SupCBM focuses on concepts that are mostly relevant to
the predicted label and only distinguishes classes when different concepts are
presented. Our evaluations show that SupCBM outperforms SOTA CBMs over diverse
datasets. It also manifests better generality across different backbone
models. With proper quantification of information leakage in different CBMs,
we demonstrate that SupCBM significantly reduces the information leakage.

Machine Learning, ICML

11footnotetext: ∗Corresponding authors.

##  1 Introduction

Deep neural networks (DNNs) have been a game-changer in the field of
artificial intelligence (AI), exhibiting remarkable performance in various
machine learning tasks such as computer vision, natural language processing,
and speech recognition. Even though these networks have enabled a strong
performance in many tasks, they are often considered as black-box models since
their extracted features are often obscure and hardly interpretable.

To address this issue, concept bottleneck models (CBMs) have recently emerged
with the purpose of delivering high-quality explanations for DNN predictions
(Koh et al., 2020). CBMs are a type of DNN that makes predictions based on
human-understandable concepts. CBMs typically have a Concept-Bottleneck (CB)
layer located before the last fully-connected (FC) layer. The CB layer takes
features extracted (by preceding layers) from the input and maps them to a set
of concepts. This effectively aligns the intermediate layers of a DNN with
some pre-defined expert concepts, and the last FC determines the final label
over those concepts. Often, CBMs require first training the CB layer to align
each neuron to a concept that is pre-defined and understandable to humans.

Based on the output concepts of the CB layer, CBMs deliver two key benefits:
interpretability and intervenability. First, users can interpret the predicted
labels by inspecting the involved concepts. Second, users can intervene the
predicted labels by modifying which concepts are involved in the prediction.
Technically, CBMs are often categorized as soft CBMs and hard CBMs, depending
on how concepts are represented and contributed to the label predictions. The
CB layer in soft CBMs outputs a probability (i.e., a number between 0 and 1)
for each concept, whereas the hard CB layer outputs a binary state (i.e., 0 or
1) to indicate if a concept exists in the input. Then, a label predictor
(often the last FC layer) takes the concept probabilities or states as input
and predicts the final label.

Despite the encouraging potential of CBMs in delivering human-understandable
explanations, their interpretability and intervenability are largely
undermined in de facto technical solutions due to information leakage, where
unintended information beyond the concepts is exploited by the label
predictor. Specifically, the concept probabilities in soft CBMs may encode
class distribution information, such that the label predictor can classify
distinct class labels based on indistinguishable concepts (Havasi et al.,
2022). For instance, the label predictor may leverage the probability
differences of “head” and “tail” to classify dog vs. cat and achieve high
accuracy, despite that these two concepts are insufficient to distinguish dog
and cat. While hard CBMs were previously believed resilient to information
leakage, a recent study (Mahinpei et al., 2021) has pointed out that hard CBMs
can exploit unrelated hard concepts to convey class distributions to the label
predictor. For example, (Mahinpei et al., 2021) demonstrate that hard CBM’s
performance can be improved by adding meaningless hard concepts.

To faithfully achieve CBM’s design objectives, it is urgent to mitigate
information leakage in CBMs. To this end, this paper proposes a novel paradigm
of CBMs, dubbed as SupCBM. Unlike previous CBMs that treat concept prediction
and label prediction as two separate tasks, we fuse them into a unified form
by additionally supervising the concept prediction with class labels.
Specifically, we do not implement the label predictor in SupCBM. Rather, we
maintain an intervention matrix to determine which concepts should be involved
for a given class label. The intervention matrix is formed when constructing
the concept set; it is implemented as a sparse binary matrix of shape
#concepts $\times$ #classes, where the $(i,j)$-th entry indicates whether the
$i$-th concept should be leveraged to recognize the $j$-th class.

When training SupCBM, the CB layer is forced to only predict concepts that are
relevant to the ground truth label, which is jointly decided by the
intervention matrix and a novel concept pooling layer (which further selects
the most important concepts for each input; see Sec. 3.2). Then, to obtain the
final predicted label, we multiply the CB layer’s output, a #concepts-
dimensional vector indicating the predicted concepts, with the intervention
matrix. This computation is equivalent to summing up the involved concepts’
probabilities for each label and treating the summed probability as the
label’s prediction confidence. This label prediction procedure only
distinguishes classes if different concepts are presented. As demonstrated in
Sec. 3.2 and Sec. 5.3, SupCBM can significantly alleviate the information
leakage. Moreover, similar to previous post-hoc CBMs like Oikarinen et al.
(2023), SupCBM also achieves post-hoc CBMs by only training a light-weight CB
layer (without training any label predictor) for any pre-trained feature-based
model.

Besides reformulating the CBM technical pipeline, we augment CBM
interpretability and intervenability from the concept aspect. We argue that
the concept set should prioritize perceptual concepts that can be directly
perceived by humans without further reasoning, e.g., “‘tail” instead of
“animal”. Also, to fully use the rich semantics of these perceptual concepts
(which are mostly nouns), we assign each of them multiple descriptive concepts
(i.e., adjectives) and build a two-level hierarchical concept set. With the
novelly formed concept set, SupCBM outperforms all SOTA CBMs and even reaches
the vanilla feature-based model (w/o converting features into concepts) when
evaluated using diverse datasets and backbones (see Sec. 5). Overall, this
paper makes the following contributions:

  * •

We propose a new paradigm of CBMs, SupCBM, where the concept prediction is
directly supervised by the class label and the follow-up label predictor is
not required. SupCBM significantly reduces information leakage in previous
CBMs and also delivers post-hoc CBMs.

  * •

We advocate that the concept set should be primarily built with perceptual
concepts that can be directly perceived by humans without further reasoning.
To utilize rich semantics in perceptual concepts, we build a two-level
hierarchical concept set by assigning each perceptual concept with multiple
descriptive concepts.

  * •

Evaluations show that SupCBM eliminates information leakage and offers highly
interpretable and intervenable predictions. SupCBM also outperforms previous
SOTA CBMs and achieves comparable performance to the vanilla, feature-based
models.

##  2 Background and Motivation

This section presents background of CBMs and reviews various implementations
of CBMs in prior works.

Without losing the generality, we take image classification as an example in
this section.

###  2.1 Concept Bottleneck Models

Dataset Construction. Given a conventional dataset
$D=\\{x^{(i)},y^{(i)}\\}_{i=1}^{N}$ consisting of $N$ pairs of input image
$x^{(i)}\in\mathbb{R}^{d}$ and its ground truth label $y^{(i)}\in\mathbb{N}$,
CBMs require augmenting the dataset $D$ with annotated concepts for all
images. Specifically, for a set of pre-defined concepts $C$, each image
$x^{(i)}$ is annotated using $c^{(i)}\in\\{0,1\\}^{|C|}$. If the $j$-th
element of $c^{(i)}$ is 1, it indicates that the $j$-th concept in $C$ is
presented in $x^{(i)}$, and vice versa.

The concept set $C$ is often manually defined by humans and the annotations of
$c^{(i)}$ require extensive human efforts, limiting the applicability of CBMs.
Recent works (Oikarinen et al., 2023; Yang et al., 2023) have employed LLMs to
automatically generate concepts and automate the annotation process with
similarity scores from multimodal alignment models (e.g., CLIP), bringing
richer and more expressive concepts sets to CBMs. However, we note that their
concepts are often obscure, which impedes interpreting and intervening CBMs.
This paper alleviates this issue by focusing on perceptual concepts and their
descriptions.

Soft and Hard CBMs. CBMs divide the label prediction $y=\arg\max f(x)$ into
two main steps: concept prediction $c=g(x)$, which is achieved by appending
the penultimate in $f$ with the CB layer, and label prediction $y=\arg\max
h(c)$. Accordingly, CBMs require additionally training the concept predictor,
which can be conducted prior to or jointly with training the label predictor.
The training of concept predictor is often formulated as multiple concurrent
binary classification tasks, with each one for one concept.

During CBM’s inference, the concept predictor outputs a probability for each
concept. If the follow-up label prediction directly takes these probabilities,
the CBM is categorized as soft CBM. In contrast, if the CBM first converts the
probabilities to binary states (i.e., 0 or 1) and then feeds them to the label
predictor, the CBM implements hard CBM.

Side-Channel CBMs. Soft CBMs are more frequently used due to their plausible
performance (comparable to feature-based models). However, information leakage
(Havasi et al., 2022), where concept probabilities encode unintended
information beyond the concepts themselves, undermines the interpretability
and intervenability of soft CBMs. Previous works therefore leverage hard CBMs
to alleviate this issue. Since hard CBMs usually exhibit limited performance.
Previous works propose side-channel CBMs (Havasi et al., 2022) by adding “side
channels” (residual connections) to hard CBMs, which link the concept
predictor and the label predictor, and passes additional information to the
label predictor. While the side channel can improve hard CBM’s performance,
the interpretability and intervenability are simultaneously sacrificed (Havasi
et al., 2022).

Worse, as pointed out by Mahinpei et al. (2021), the CB layer in hard CBMs can
also leak unintended information to the label predictor via irrelevant (hard)
concepts, rendering the high demand of a new CBM paradigm that is resilient to
information leakage. As will be demonstrated in Sec. 3.2 and Sec. 5.3, our
novel CBM, SupCBM, can significantly reduce the information leakage.

Post-hoc CBMs. When converting standard feature-based models into CBMs, the
common practice is to tune the whole model. Nevertheless, this incurs a high
training cost due to the large number of parameters in the concept predictor.
Post-hoc CBM (Yuksekgonul et al., 2022) is therefore proposed to efficiently
convert pre-trained backbone models into CBMs.

In particular, post-hoc CBM first builds a concept bank and forms a concept
subspace with the embeddings of these concepts. Then, the embedding generated
by the backbone model is first projected onto the concept subspace, and then
the predicted concepts (e.g., stripe, horse) are fed into the label predictor
(e.g., zebra). Post-hoc CBMs retain the weights in the original model, and
only train the label predictor. Therefore, they are more efficient and
applicable to large-scale models.

SupCBM achieves post-hoc CBM by only training a light-weight one-layer CB
layer for each backbone model. Thus, SupCBM’s label prediction is conducted by
multiplying the concept prediction with an intervention matrix, which requires
no training (see Sec. 3.2). Moreover, while previous CBMs and SupCBM are both
deemed post-hoc, evaluations in Sec. 5.4 shows that previous post-hoc CBMs
manifest lower generality than SupCBM when applied to different backbone
models.

##  3 SupCBM’s Design

Figure 1: Workflow of SupCBM. The training of SupCBM consists three mains
stages.

Technical Pipeline. Fig. 1 illustrates the workflow of SupCBM, which consists
of three main stages. In the concept construction stage, we leverage GPT
(version 4.0) to generate a comprehensive set of concepts and organize them in
a two-level hierarchical manner. Meanwhile, we also maintain an intervention
matrix to record each concept’s relevant concepts. Then, during the training
stage, we train the CB layer to map the feature extracted by the backbone
model into our prepared concepts. To reduce the concept leakage, the training
is supervised by the ground truth label and a concept pooling layer. Finally,
when predicting the class labels, we multiply the CB layer’s concept
prediction with our intervention matrix; the final label is obtained by
applying $\arg\max$ on the multiplication result.

###  3.1 Concept Set Construction

Perceptually Perceivable Concepts. One key observation made in this paper, is
that some concepts in previous works are often too holistic to be easily
perceivable by humans and require manual reasoning. For instance, the concept
“animal” is involved in classifying cats. Although animal is a genuine super-
class of cat, it cannot be perceptually observed; prior knowledge is required
to reason that cat is an animal. Also, such concepts impede the intervention
of CBMs: it is obscure to add or remove the concept “animal” for the follow-up
label prediction. Thus, we advocate that the concept set should be formed with
perceptual concepts that can be directly observed by humans without reasoning.

Hierarchical Concepts. Since CBMs are designed to only describe whether
certain concepts are involved in the label prediction, the rich semantics of
concepts often cannot be faithfully reflected from CBMs. Considering the case
of classifying different breeds of fish in CIFAR100, while the concept “tail”
can be leveraged for predicting labels, the information it contributes is far
beyond than simply indicating its involvement. For example, a “fan tail” is
critical to identify goldfish, whereas the “veil tail” is vital to identify
betta fish.

Thus, we form a two-level hierarchical concept set, where the first level
contains nounal perceptual concepts and primarily flags if a concept is
involved in the label prediction, whereas the second level includes adjectival
concepts for each first-level concept and describes which semantics of the
visual concepts contribute to the label prediction.

Prompt Design. We use GPT to automatically build the concept set. Since we
require perceptually perceivable concepts to enhance CBM interpretability and
intervenability, we first query the GPT model with the following prompt:

> “To identify $\\{\text{CLS}\\}$ visually, please list the most important
> $\\{p\\}$ visual parts which a $\\{\text{CLS}\\}$ has.”

Here, the token $\\{\text{CLS}\\}$ can be replaced by any classes the original
feature-based model can predict. The GPT returns a set of nounal visual parts
in $\\{\text{CLS}\\}$ as our first-level concept. Then, for each concept
$\\{\text{CEP}\\}$ of a class $\\{\text{CLS}\\}$, we query the GPT model with
the following prompt to uncover semantics associated in different concepts.

> “To visually identify $\\{\text{CLS}\\}$, please describe the $\\{q\\}$ most
> common characteristics of $\\{\text{CLS}\\}$’s $\\{\text{CEP}\\}$ from the
> three dimensions of shape, color, or size.”

With the above prompt, the GPT model can return to us a set of adjectival
concepts as our second-level concepts that describe the semantics of the
first-level nounal concepts. We limit the adjectival concepts to follow the
three dimensions of shape, color, and size, thereby reducing ambiguity. We
also remove any concept that is longer than 40 characters to keep the
descriptions simple. This way, we obtain a two-level concept set containing
concepts represented in a $\langle${CEP},{DESCR}$\ \rangle$ form, as
illustrated in Fig. 1, e.g., $\langle Pointed,Beak\rangle$ constitutes one
$\langle${CEP},{DESCR}$\ \rangle$ tuple for the input bird image. Accordingly,
we also know each $\langle${CEP},{DESCR}$\ \rangle$’s relevant class
$\\{\text{CLS}\\}$.

Extendability. Our concept set is highly extendable. To convert any backbone
models whose supported classes are not included in our concept set, users can
simply replace the $\\{\text{CLS}\\}$ in the first prompt and query GPT models
to obtain the corresponding concepts, and accordingly update the
$\\{\text{CLS}\\}$ and $\\{\text{CEP}\\}$ in the second prompt to augment the
concept set with new $\langle${CEP},{DESCR}$\ \rangle$ pairs.

###  3.2 Training Supervision

Intervention Matrix. We maintain an intervention matrix to represent which
concepts should be involved in predicting a class label. The intervention
matrix is a binary matrix of shape #concepts $\times$ #classes, as shown
below,

$\small\begin{bmatrix}\mathcal{I}_{1,1}&\mathcal{I}_{1,2}&\dots&\mathcal{I}_{1,\text{\\#classes}}\\\
\mathcal{I}_{2,1}&\mathcal{I}_{2,2}&\dots&\mathcal{I}_{2,\text{\\#classes}}\\\
\vdots&\vdots&\vdots&\vdots\\\
\mathcal{I}_{\text{\\#concepts},1}&\mathcal{I}_{\text{\\#concepts},2}&\dots&\mathcal{I}_{\text{\\#concepts},\text{\\#classes}}\\\
\end{bmatrix}$ (1)

where $\mathcal{I}_{i,j}\in\\{0,1\\}$ and $\mathcal{I}_{i,j}=1$ indicates that
the $i$-th concept should be involved in predicting the $j$-th class;
otherwise, $\mathcal{I}_{i,j}=0$. This is obtained after querying the GPT with
the second prompt mentioned in Sec. 3.1.

Label-Aware Concept Annotation. A fundamental difference between SupCBM and
previous CBMs is the supervision of class label when training the CB layer. As
illustrated in Fig. 1, for each training image $x$ whose ground truth label is
$y$, we annotate it only with concepts that are involved in predicting $y$ (as
indicated by the intervention matrix). This annotation, to some extent, can
supervise the CB layer to “fuse” the concept prediction and label prediction,
which can improve the performance of CBMs and simultaneously reduce the
information leakage; see evaluations in Sec. 5.

Concept Pooling. Since Mahinpei et al. (2021) point out that adding irrelevant
concepts (even in a hard form) can enlarge information leakage, we further
implement a selective strategy as shown in Fig. 1. Similar to the max pooling
mechanism in conventional computer vision which selects most important
features, we choose to annotate $x$ with those most important concepts. We
name our selection procedure as concept pooling as it is equivalent to
applying an 1-dimensional max pooling of kernel size $q$ and stride $q$ on all
second-level concepts. Specifically, we first compute $x$’s similarities with
all concepts $\langle${CEP},{DESCR}$\ \rangle$ that are involved in predicting
$y$. To do so, we measure the cosine similarity between the CLIP embeddings of
$x$ and $\langle${CEP},{DESCR}$\ \rangle$. Then, for concepts
$\langle${CEP},{DESCR}$\ \rangle$ sharing the same $\\{\text{CEP}\\}$, we
choose those having the top-$k$ similarity as the ground truth concepts. This
way, we have total $p\times k$ concept annotations for each input image.

Training Objectives. Unlike previous post-hoc CBMs that directly train the
follow-up label predictor with concept similarities, we do not train a label
predictor. Instead, we train the CB layer (without training the backbone
model) since label supervision is unavailable for test images. Specifically,
for each input image, we set the selected $p\times k$ concepts as hard label
and form $p\times k$ binary classification tasks following the conventional
training procedure. These binary classification tasks are optimized with
binary cross-entropy loss $\ell_{BCE}(c,GT_{c})$, where $c$ is the concept
probabilities predicted by the CB layer and $GT_{c}$ is the ground truth
concepts. Both $c$ and $GT_{c}$ are $pq$-dimension vectors.

The predicted label is decided as $\arg\max c*\mathcal{I}$, where $*$ denotes
matrix multiplication. That is, the $j$-th class’s prediction confidence is
computed as

$l_{j}=\sum_{i=1}^{pq}c_{i}\cdot\mathcal{I}_{i,j}$ (2)

Since $\mathcal{I}_{i,j}$ indicates whether the $i$-th concept should be
involved in predicting the $j$-th class, the label $j$’s confidence $l_{j}$
equals the sum of probabilities of those involved concepts. Consistent with
conventional CBM training, we have a cross-entropy loss $\ell_{CE}(l,GT_{l})$,
where $l$ is a #classes-dimensional vector and $GT_{l}$ is an integer
indicating the ground truth class label. It’s worth noting that the
intervention matrix $\mathcal{I}$ is always fixed, and $\ell_{CE}(l,GT_{l})$
is only leveraged to optimize the CB layer. Therefore, the overall training
objective is to minimize:

$\alpha\cdot\ell_{BCE}(c,GT_{c})+(1-\alpha)\cdot\ell_{CE}(c*\mathcal{I},GT_{l})$
(3)

where $\alpha$ is a hyper-parameter.

Eliminating Information Leakage. SupCBM eliminates information leakage from
two aspects. First, for hard information leakage where irrelevant concepts may
leak unintended information for label prediction, we inherently mitigate it
during the concept annotation. As mentioned above, when annotating concepts,
we only focus on those concepts that are involved in predicting the label.
Second, we further select those most important descriptive concepts
$\\{\text{DESCR}\\}$ via our concept pooling mechanism, thus trimming
irrelevant concepts. With these two steps, irrelevant concepts are rarely kept
in label predictions, reducing hard information leakage noted in Mahinpei et
al. (2021).

Moreover, for soft information leakage where the concept probabilities encode
distribution of class information, we justify how our intervention matrix can
eliminate such leakage. Considering two classes $a$ and $b$ whose involved
concepts (according to $\mathcal{I}$) constitute sets $C_{a}$ and $C_{b}$,
respectively. We define $S=C_{a}\cap C_{b}$, $A=C_{a}\setminus S$, and
$B=C_{b}\setminus S$. That is, $S$ consists of concepts shared by both
classes, and $A$ and $B$ denote the sets of concepts that are only involved in
predicting $a$ and $b$, respectively.

① If $S=\emptyset$, i.e., the two classes do not share any concepts, it should
be clear that classifying $a$ and $b$ only rely on their disjoint concepts $A$
and $B$, and SupCBM finds concepts that uniquely exist in classes $a$ and $b$.
Note that our concept pooling mentioned above can help reduce the size of $S$.

And in case $S\neq\emptyset$, i.e., the two classes share some concepts, we
analyze the following three cases.

② If $A=\emptyset$, i.e., the class $a$ does not have any unique concept, when
the input’s label is $a$, the label prediction confidences of class $a$ and
$b$ should be identical. Thus, the label prediction (i.e., $c*\mathcal{I}$ in
SupCBM) cannot distinguish $a$ and $b$, and no information beyond the concept
is exploited to falsely conduct classification. Similar conclusions can be
drawn if $B=\emptyset$ or $A=B=\emptyset$.

③ If $A\neq\emptyset$ and $B\neq\emptyset$, i.e., both classes have their
unique concepts. Then given an input, the difference between its two label
prediction confidences (w.r.t. class $a$ and $b$) is only induced by the
concepts belonging to $A\cup B$. Accordingly, no information beyond $a$ and
$b$’s unique concepts is leveraged to classify them.

From the above three cases, we can safely conclude that SupCBM eliminates soft
information leakage.

Post-hoc Concept Mapper. It’s worth noting that SupCBM also achieves post-hoc
CBMs. Similar to existing post-hoc CBMs, to convert a new backbone model,
SupCBM only requires training a light-weight layer. Differently, previous
works train the label predictor using their pre-built concept set, while
SupCBM only trains an one-layer CB layer without training any label predictor.

##  4 Implementation

Concept Set Construction. When querying GPT (as introduced in Sec. 3.1), we
set $p=5$ for the first prompt and $q=6$ for the second prompt. That is, for
each class, we obtain $5$ perceptual concepts with each one having $6$
descriptive concepts. When annotating the concepts using our concept pooling,
we set $k=2$. That is, each training input is annotated with total ($5$
perceptual $\times$ $2$ descriptive) concepts.

Hyperparameters. For all experiments, we use the Adam optimizer to optimize
the loss function, with $\alpha=0.7$. In accordance with the previous works
(Yuksekgonul et al., 2022; Yang et al., 2023), we tune other hyperparameters
on the development set.

##  5 Evaluations

We conduct performance evaluation in Sec. 5.2, and assess information leakage
in Sec. 5.3. We also benchmark the generality of SupCBM in Sec. 5.4, and Sec.
5.5 presents case studies of SupCBM’s concepts and interpretations.

###  5.1 Setup

Following previous works (Yuksekgonul et al., 2022; Oikarinen et al., 2023;
Yang et al., 2023), we consider the following representative datasets to
evaluate the performance of different models. These datasets are
representative and cover three major computer vision domains including general
classification (CIFAR10, CIFAR100), specialized classification (CUB-Bird), and
medical image analysis (HAM10000).

CIFAR-10 (Krizhevsky et al., 2009) consists of 32$\times$32 RGB-color images
of 10 classes and each class has 6,000 images.

CIFAR-100 (Krizhevsky et al., 2009) consists of 32$\times$32 RGB-color images
of 10 classes and each class has 600 images.

CUB-Bird (Wah et al., 2011) consists of 11,788 RGB-color images of 200 bird
species.

HAM10000 (Tschandl et al., 2018) consists of 10,015 dermatoscopic images of
pigmented skin lesions.

For CIFAR10 and CIFAR100, all CBMs use the same backbone CLIP-RN50 following
in Yuksekgonul et al. (2022); Oikarinen et al. (2023); Yang et al. (2023). For
CUB-Bird, following Yang et al. (2023), we use CLIP-ViT14 as the backbone
model to obtain a high performance. Consistent with Oikarinen et al. (2023),
we also ignore grayscale images in CUB, and only consider the RGB format,
which results in 5,990 training images and 5,790 test images. For HAM10000, we
use the standard HAM-pretrained Inception (Daneshjou et al., 2021) as the
backbone.

###  5.2 Performance Comparison

We compare our method with the following SOTA CBMs.

P-CBM (Yuksekgonul et al., 2022) is the first post-hoc CBM that uses the CLIP
embeddings to align images and concepts. P-CBM first projects an image
embedding onto the concept subspace and then computes its similarities with
different concepts. These similarities are adopted for label prediction. The
label predictor in P-CBM is connected with image embeddings via a residual
connection.

Label-free CBM (Oikarinen et al., 2023) is mostly similar to P-CBM, but
computes similarity between images and concepts using the dot product of their
embeddings. Label-free CBM is the first work that employs GPT models to
generate textual concepts.

LaBo (Yang et al., 2023) builds semantic vectors with a large set of
attributes from LLMs. It uses GPT-3 to produce factual sentences about
categories to form candidate concepts. It then employs a so-called submodular
utility to effectively explore potential bottlenecks that facilitate the
identification of distinctive information.

Yan (Yan et al., 2023) uses concise and descriptive attributes extracted from
LLMs. Specifically, it uses a learning-to-search method to extract a
descriptive subset of attributes from LLMs by pruning the large attribute set.

Besides the above CBMs, existing works also evaluate a setting (referred to as
Feat), which directly uses features extracted by the backbone model for the
subsequent label prediction. In short, Feat can be deemed to offer the “upper
bound” performance of CBMs.

Table 1: Performance comparison with SOTA CBMs and the vanilla setting Feat.
We mark the best performance in blue and the second best in bold. “N/A”
indicates that they do not provide the related concept-generating
implementation.

Model | CUB-Bird | CIFAR10 | CIFAR100 | HAM10000
---|---|---|---|---
Feat | 86.41 | 88.80 | 70.10 | 84.07
P-CBM | 78.31 | 84.50 | 56.60 | 70.69
Label-Free | 78.88 | 86.40 | 65.13 | 72.79
Labo | 83.82 | 87.90 | 69.10 | 82.03
Yan | 81.67 | 79.31 | 66.66 | N/A
SupCBM | 85.98 | 89.22 | 70.11 | 83.28

Table 1 reports the performance of all SOTA CBMs, the vanilla feature
representation (Feat), and SupCBM, on the four datasets. We see that SupCBM
constantly outperforms all SOTA CBMs across all datasets. Moreover, SupCBM
exhibits comparable performance with Feat, and even outperforms it on the
CIFAR10 and CIFAR100 datasets. We interpret the findings as highly
encouraging, demonstrating the superiority of our new CBM paradigm.

###  5.3 Information Leakage

Setup. Sec. 5.2 shows that LaBo has the best performance among all our
competitors. Therefore, this section assesses the information leakage in LaBo
and our method. We use CIFAR100, given the large number of distinct classes
this dataset has. We also consider the Flower dataset (Nilsback & Zisserman,
2008); it is a phytology knowledge specific dataset with 102 visually close
semantic classes, making it handy to quantify the information leakage problem.

Metrics & Intuitions. Since a quantitative metric of information leakage is
still lacking, we propose a new metric to assess the information leakage in
CBMs. The information leakage problem is induced by the insufficient concept
set, such that the label prediction has to exploit unintended information to
fulfill the training objective (e.g., achieving high accuracy) (Mahinpei et
al., 2021). In that sense, if a CBM suffers from information leakage, removing
concepts that contribute most to the label prediction should notably undermine
the CBM performance. Following this intuition, for each evaluated CBM, we
start by training it with the full concept set. Then, during the inference, we
rank concepts based on their importance and gradually remove top-ranked
concepts. We expect that the performance of CBMs which manifest better
resilience to information leakage should drop more quickly when more concepts
are removed. Here, we use our constructed concept set for a fair comparison.

Baselines. We set the baseline as a dummy linear model which does not have
concept alignment as CBMs (i.e., the “concepts” are purely random) and should
have the most severe information leakage issue. Also, as an ablation, we
replace the intervention matrix in SupCBM with a learned fully-connected
layer. We denote the ablated version of SupCBM as $\text{{SupCBM}}_{FC}$.

Figure 2: Information leakage evaluation. If a CBM manifests higher resilience
to information leakage, its performance should drop more quickly when more
concepts are removed.

Results. As shown in Fig. 2, we can see that the dummy model manifests the
lowest resilience to information leakage, as its performance drops slowly when
more concepts are removed. Specifically, even when top 10% important concepts
are removed, the dummy model still has $\sim$20% and $\sim$90% accuracy for
the CIFAR100 and the Flower datasets, respectively. Fig. 2 also shows that
SupCBM’s performance drops the fastest when more concepts are removed in both
two datasets, indicating its highest resilience towards information leakage.
In addition, when cross-comparing $\text{{SupCBM}}_{FC}$ with LaBo,
$\text{{SupCBM}}_{FC}$ is lowered to random guess more quickly than LaBo. We
interpret that the gap is due to our label-supervised concept prediction,
which rules out irrelevant concepts and thus improves the resilience to
information leakage (Mahinpei et al., 2021). Moreover, when cross-comparing
SupCBM with $\text{{SupCBM}}_{FC}$, it is evident that the label prediction
conducted via the intervention matrix also improves the resilience to
information leakage, which has been justified in Sec. 3.2.

###  5.4 Generality

While recent post-hoc CBMs achieve promising performance, they all take the
text encoder of CLIP as the backbone to label concepts (i.e., obtaining the
text embeddings of concepts); however, the goal of CBMs is to understand the
pair-wised visual backbone model (i.e. the corresponding image encoder). So,
it is unclear whether the concept knowledge learned by the text-encoder will
be fully aligned or unbiased with the pair-wised image encoder. Therefore, we
conduct a generality evaluation here to benchmark these CBMs from the “post-
hoc” perspective. Specifically, we incorporate three types of pair-wised text-
vision backbone models, including RN-50, ViTB-32, ViTL-14, into these CBMs and
evaluate their performance.

Figure 3: Generality evaluation.

Baselines. We consider two baselines to compare with SupCBM: Feat and CBM-
proj. As aforementioned, Feat is the vanilla feature representation, which
offers the “upper bound” performance toward targeted vision backbones. CBM-
proj is a direct projection of the CLIP text-image cosine similarity score
onto the label space, which deems the most straightforward way to validate the
bias from the CLIP Text-encoder for visual-model prediction and subsumes prior
post-hoc CBMs. For a fair comparison, we use the same concept set in CBM-proj
as in our method.

Results are shown in Fig. 3; it is seen that SupCBM constantly outperforms all
competitors on all datasets with different backbone models. Also, these
competitors’ performance may change largely with backbone models, whereas
SupCBM’s performance is stable when using different backbones, indicating its
better generality.

Figure 4: Case study of SupCBM on CUB-Bird and FLOWER datasets.

###  5.5 Case Studies

This section presents cases of SupCBM’s predicted concepts and interpretations
of different predicted classes. We use the CUB-Bird and Flower datasets since
they are finer-grained than other datasets.

Empowered our concept pooling technique, SupCBM can deliver precise intra-
class concept identification.

As shown in Fig. 4, for each image, we first present the perceptual concepts
(i.e., the first-level concepts) identified by SupCBM. Then, for each
perceptual concept, we show its most important (whose probability is the
highest) descriptive concept.

Considering the first row of Fig. 4, where three different black-footed
albatrosses have visually different wings due to their flying postures, SupCBM
can accurately distinguish them by recognizing the distinct semantics of their
wings. Specifically, the third albatross differs from the other two with
spreading wings, SupCBM can identify such differences by predicting the
“Slight dihedral shape” wing. Also, the second albatross folds its wings,
which is faithfully captured by SupCBM and reflected from the adjectival
concept “narrow” in both perceptual concepts “tail” and “wing”.

Regarding the Flower dataset shown in the last two rows of Fig. 4, we find
that unlike the CUB dataset where birds have diverse motions, the concepts
(both perceptual and descriptive) are mostly similar for flows from the same
class. However, SupCBM can still capture the subtle differences. Considering
the Petunia images (the third row) shown in Fig. 4, where three images are
highly similar in terms of the color and shape, SupCBM can precisely recognize
“five sepals” calyx in the third image, and identify the “Pointed tips” calyx
in other two images. Overall, these cases demonstrate the effectiveness of
SupCBM in capturing diverse concepts and rich semantics, and interpreting its
predictions.

##  6 Related Works

Further to Sec. 2.1, we review other relevant works below.

Model Interpretation. Model interpretation aims to explain the model’s
predictions in a human-understandable forms, divided into pixel-based and
concept-based approaches. Pixel-based techniques highlight the input pixels
that contribute most to the model’s prediction, including saliency maps
(Simonyan et al., 2013), Grad-CAM (Zhou et al., 2016), DeepLIFT (Shrikumar et
al., 2017), GradSHAP (Selvaraju et al., 2017), LIME (Ribeiro et al., 2016),
SHAP (Lundberg & Lee, 2017), and DeepSHAP (Lundberg & Lee, 2017). The
interpretation is accomplished by computing each pixel’s (or super-pixel’s)
contribution to the model prediction. Nevertheless, the main issue is that
such pixel-based interpretation can be hard to make sense of by humans.
Recently, some efforts have been made to improve the understandability. For
instance, EAC (Sun et al., 2023) delivers concept-based interpretations by
using a segmentation model to generate concept masks. Similar to CBMs, EAC
provides understandable and abstract concepts to facilitate interpretation of
model predictions. However, EAC relies on approximation of the original model,
and thus may fail to faithfully represent the original model’s behavior.

Model Editing and Repairing. Previous model editing/repairing works primarily
focus on generative models (e.g., GANs). By identifying interpretable
direction in the input latent space of generative models, they can achieve
controllable generation of images (Härkönen et al., 2020; Shen et al., 2020).
However, such techniques do not apply discriminative models like classifiers.
Some works in the software engineering community have proposed to repair DNNs
by modifying their weights or structures. For example, CARE (Sun et al., 2022)
uses causal reasoning to identify buggy neurons in DNNs and repair them and
NNrepair (Usman et al., 2021) and provable polytope repair (Sotoudeh & Thakur,
2021) use formal methods to repair DNNs. However, these techniques are
orthogonal to CBMs, as they focus on repairing DNNs’ internal structures,
while CBMs aim to repair the model’s predictions by modifying the input
concepts. Moreover, these techniques are usually computationally expensive,
while CBMs are lightweight as they only modify the input to the label
predictor (i.e., the existence of certain concepts).

##  7 Conclusion

This paper has proposed SupCBM, a novel approach to improve the
interpretability of CBMs with supervised concept prediction, hierarchical,
descriptive concepts, and a post-hoc design. Evaluations show that SupCBM can
effectively outperform existing CBMs to offer high interpretability and
performance (reaching the vanilla, feature-based models) across various
settings.

## References

  * Daneshjou et al. (2021) Daneshjou, R., Vodrahalli, K., Liang, W., Novoa, R. A., Jenkins, M., Rotemberg, V., Ko, J., Swetter, S. M., Bailey, E. E., Gevaert, O., et al.  Disparities in dermatology ai: assessments using diverse clinical images.  _arXiv preprint arXiv:2111.08006_ , 2021.
  * Härkönen et al. (2020) Härkönen, E., Hertzmann, A., Lehtinen, J., and Paris, S.  Ganspace: Discovering interpretable gan controls.  _Advances in neural information processing systems_ , 33:9841–9850, 2020.
  * Havasi et al. (2022) Havasi, M., Parbhoo, S., and Doshi-Velez, F.  Addressing leakage in concept bottleneck models.  _Advances in Neural Information Processing Systems_ , 35:23386–23397, 2022.
  * Koh et al. (2020) Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P.  Concept bottleneck models.  In _International conference on machine learning_ , pp. 5338–5348. PMLR, 2020.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lundberg & Lee (2017) Lundberg, S. M. and Lee, S.-I.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30, 2017.
  * Mahinpei et al. (2021) Mahinpei, A., Clark, J., Lage, I., Doshi-Velez, F., and Pan, W.  Promises and pitfalls of black-box concept learning models.  _arXiv preprint arXiv:2106.13314_ , 2021.
  * Nilsback & Zisserman (2008) Nilsback, M.-E. and Zisserman, A.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian conference on computer vision, graphics & image processing_, pp. 722–729. IEEE, 2008.
  * Oikarinen et al. (2023) Oikarinen, T., Das, S., Nguyen, L. M., and Weng, T.-W.  Label-free concept bottleneck models.  _arXiv preprint arXiv:2304.06129_ , 2023.
  * Ribeiro et al. (2016) Ribeiro, M. T., Singh, S., and Guestrin, C.  “why should i trust you?” explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pp. 1135–1144, 2016.
  * Selvaraju et al. (2017) Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 618–626, 2017.
  * Shen et al. (2020) Shen, Y., Gu, J., Tang, X., and Zhou, B.  Interpreting the latent space of gans for semantic face editing.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9243–9252, 2020.
  * Shrikumar et al. (2017) Shrikumar, A., Greenside, P., and Kundaje, A.  Learning important features through propagating activation differences.  In _International conference on machine learning_ , pp. 3145–3153. PMLR, 2017.
  * Simonyan et al. (2013) Simonyan, K., Vedaldi, A., and Zisserman, A.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Sotoudeh & Thakur (2021) Sotoudeh, M. and Thakur, A. V.  Provable repair of deep neural networks.  In _Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation_ , pp. 588–603, 2021.
  * Sun et al. (2023) Sun, A., Ma, P., Yuan, Y., and Wang, S.  Explain any concept: Segment anything meets concept-based explanation.  _Advances in Neural Information Processing Systems_ , 2023.
  * Sun et al. (2022) Sun, B., Sun, J., Pham, L. H., and Shi, J.  Causality-based neural network repair.  In _Proceedings of the 44th International Conference on Software Engineering_ , pp. 338–349, 2022.
  * Tschandl et al. (2018) Tschandl, P., Rosendahl, C., and Kittler, H.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  _Scientific data_ , 5(1):1–9, 2018.
  * Usman et al. (2021) Usman, M., Gopinath, D., Sun, Y., Noller, Y., and Păsăreanu, C. S.  Nn repair: Constraint-based repair of neural network classifiers.  In _Computer Aided Verification: 33rd International Conference, CAV 2021, Virtual Event, July 20–23, 2021, Proceedings, Part I 33_ , pp. 3–25. Springer, 2021.
  * Wah et al. (2011) Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Yan et al. (2023) Yan, A., Wang, Y., Zhong, Y., Dong, C., He, Z., Lu, Y., Wang, W. Y., Shang, J., and McAuley, J.  Learning concise and descriptive attributes for visual recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3090–3100, 2023.
  * Yang et al. (2023) Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., and Yatskar, M.  Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 19187–19197, 2023.
  * Yuksekgonul et al. (2022) Yuksekgonul, M., Wang, M., and Zou, J.  Post-hoc concept bottleneck models.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhou et al. (2016) Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2921–2929, 2016.