# Constraining nuclear effects in Argon using machine learning algorithms

Srishti Nagu1 111E-mail<EMAIL_ADDRESS>Jaydip Singh1 222E-mail:
<EMAIL_ADDRESS>Jyotsna Singh1, R.B. Singh1 Department Of Physics,
University of Lucknow, Lucknow, India.1

###### Abstract

Neutrino oscillation experiments aim to measure the neutrino oscillation
parameters with accuracy and achieve a complete understanding of neutrino
physics. For determining the neutrino oscillation parameters, knowledge of
neutrino energy is a prerequisite. But neutrino energy needs to be
reconstructed, based on the particles in the final state that emerge out of
the nucleus following a neutrino-nucleus interaction. Current and upcoming
neutrino oscillation experiments use heavy nuclear targets (viz. Argon(Ar),
Calcium(Ca), etc.) but the neutrino scattering with such targets becomes
complicated as compared to that with a clean target like Hydrogen(H). This
work explores the viability of using machine learning algorithms (MLA) in
reconstructing neutrino energy. We use final state kinematics generated from
two neutrino event generators viz. GENIE and GiBUU to train the MLA. We
calculate the Ar/H ratio in an attempt to quantify nuclear effects in the Ar
target. We observe a significant improvement in our results when we train the
MLA by combining the FSI kinematics of neutrino interactions from both the
neutrino event generators.

##  I Introduction

Neutrino being intrinsically leptonic and neutral is the second most abundant
particle in the Universe. It is a potential candidate for probing the complex
hadronic structure as well as exploring physics beyond the Standard Model.
Precision measurement of its properties and nature of interaction with the
other nuclei is among the highest fundamental particle physics priorities.
Several benchmark discoveries employing a variety of nuclear targets in the
past decades have revealed several properties of the neutrino. Active
neutrinos are known to exist in three flavors viz. electron neutrino
($\nu_{e}$), muon neutrino ($\nu_{\mu}$) and tau neutrino ($\nu_{\tau}$). A
neutrino of a particular flavor, after traveling a certain distance,
transforms into a neutrino of another flavor, this unique property is referred
to as neutrino oscillation. A milestone was achieved with the discovery of
neutrino oscillations that substantiated the tiny mass of neutrinos. The study
of neutrino oscillation physics involves certain parameters i.e. mixing angles
$\theta_{ij}$ where $j>i$=1,2,3 ($\theta_{12},\theta_{13},\theta_{23}$), Dirac
phase $\delta_{CP}$ and mass squared differences i.e. $\Delta m_{31}^{2}$
known as atmospheric mass splitting and $\Delta m_{21}^{2}$ known as solar
mass splitting. Some of the neutrino oscillation parameters have been measured
precisely i.e. $\theta_{12}$, $\theta_{23}$, non-zero value of $\theta_{13}$
theta13_1 ; theta13_2 ; theta13_3  and magnitude of mass squared differences
$\Delta m_{21}^{2}$, $|\Delta m_{31}^{2}|$. While the neutrino oscillation
parameters which are yet to be discovered with precision are- (1) the value of
dirac phase $\delta_{CP}$: which can lie in the range $-\pi<\delta_{CP}<\pi$
(2) the neutrino mass ordering i.e. sign of $\Delta m_{31}^{2}$: there is a
possibility of neutrino mass ordering being ’normal’ i.e. whether the neutrino
mass order can be- $m_{1}\ll m_{2}\ll m_{3}$ known as normal mass hierarchy
(NH) or the ordering can be ’inverted’ i.e. the neutrino mass order can be-
$m_{2}\approx m_{1}\gg m_{3}$ known as inverted mass hierarchy (IH), where
$m_{i}$(i=1,2,3) are neutrino mass eigenstates (3) octant degeneracy i.e.
whether the value of $\theta_{23}$ lies in the lower octant(LO)
$0<\theta_{23}<\pi/4$ or higher octant(HO) $\pi/4<\theta_{23}<\pi/2$. The
current status of the neutrino oscillation parameters estimated by performing
a global fit can be found in global . Qualitative improvement in the
measurement of the unknown neutrino oscillation parameters requires pinning
down the systematic uncertainties which become more challenging with the use
of heavy nuclear targets (mass number(A)$>$12).

For precise determination of the neutrino oscillation parameters at neutrino
oscillation experiments, it is crucial to accurately determine the energy of
the incoming neutrino since the neutrino oscillation probability depends upon
the neutrino energy spectrum. In beam-based collider/accelerator experiments
we need to reconstruct the energy of the neutrino since neutrinos are produced
as secondary decay products from primarily produced hadrons that lead to non-
monoenergetic and broad neutrino beams. To reconstruct the energy of a
neutrino, the energy of all the final state particles produced in a neutrino-
nucleus interaction must be taken into account on an event-by-event basis. An
obstacle for precise neutrino energy reconstruction from the final state
particles is the inability to correctly identify the particle configuration,
dynamics, and kinematics of a neutrino interaction within the nucleus. The
particles that are produced at the initial neutrino nucleon interaction vertex
within the nucleus are often modified inside the nuclear environment. When a
neutrino interacts with a nucleon (which is not at rest) inside the nucleus
certain nuclear effects nuceff  come into play such as fermi motion, Pauli
blocking, multinucleon effects multinucleon  at the initial interaction
vertex. Moreover, as the particles produced at initial neutrino-nucleon
interaction travel through the nucleus they re-interact with other nucleons
and emerge out of the nucleus as different particles known as final state
particles. Collectively, these effects are known as nuclear effects whitepaper
. As a result, the kinematics and dynamics of the real interaction remain
unknown. Thus the physics of neutrino-nucleus scattering remains tangled and
requires a good understanding of nuclear effects.

Neutrino owing to its weakly interacting nature and very low interaction
cross-section is merely detectable. Thus to study neutrinos at neutrino
experiments heavy nuclear targets are used which gives large statistics of
neutrino interaction events resulting in reduced statistical uncertainty but
at the same time leads to huge systematic uncertainties due to inherent
nuclear effects. For extracting vital information from the large data samples
collected from detectors at neutrino experiments, it is crucial to develop
advanced reconstruction techniques. Machine learning algorithms(MLA) are an
integral part of the analysis tools for the past two decades ml1 . Significant
improvement has been witnessed in the physics reach of the analyses with the
use of MLAs. Results obtained from previous generation experiments exp1 ; exp2
; exp3 ; exp4 ; exp5  that involved the application of neural networks nn1 ;
nn2  and Boosted Decision Trees(BDTs) led the use of MLAs at the Large Hadron
Collider (LHC) lhc1 . MLAs also played a crucial role in the discovery of the
Higgs Boson higgs1 ; higgs2 . For a much detailed review about the use of MLAs
at LHC one can refer to lhc .

The upcoming and ambitious long-baseline neutrino oscillation experiment i.e.
the Deep Underground Neutrino Experiment (DUNE) dune  is also using MLAs for
data analysis. DUNE is currently in the process of being set up in the
Fermilab, United States. It strives to achieve the most sought-after goals in
the realm of particle physics i.e. the octant degeneracy, neutrino mass
hierarchy, value of Dirac $\delta_{CP}$ phase. It’s prime design consists of a
Near Detector (ND) nd1 ; nd2 ; nd3 ; nd4  at Fermilab and a Far Detector (FD)
dune  to be set up at Sanford Underground Research Facility (SURF), South
Dakota. The design of ND is a fine-grained magnetic spectrometer while the FD
will host four independent 10 kt each, Liquid Argon Time Projection Chambers
(LArTPCs). Both the ND and FD will be employing Ar as a target with the aim of
maximum cancellation of systematic uncertainties. A muon neutrino (or
antineutrino) dominated beam will be focussed from Long-Baseline Neutrino
Facility (LBNF) at Fermilab to both the ND and FD set up at a distance of 575
meters and 1300 km respectively. For data analysis at DUNE, advanced MLAs like
Deep learning are being used for interaction vertex reconstruction dunemla1
and kinematic reconstruction dunemla2 . Deep learning is a type of machine
learning method based on artificial neural networks where ’deep’ refers to
several layers in the network. Convolution Neural Network (CNN) cnn  is being
used for classifying neutrino interactions dunemla3 . A CNN can be defined as
a deep neural network that consists of several convolutional layers. The
output from each layer acts as an input for the next layer. To classify
neutrino interactions, this technique has proven to be better than the
traditional event-based reconstruction methods refa ; refb . While
reconstruction of neutrino energy and final state particle momenta with deep
learning methods is still in progress.

To reconstruct the neutrino energy using MLAs, we have used a linear
regression model and compared it with the calorimetric method of energy
reconstruction. For our analysis we have selected two different simulation
tools, GENIE (Generates Events for Neutrino Interaction Experiments) genie
(version-2.12.6) and GiBUU (Giessen Boltzmann-Uehling-Uhlenbeck) gibuu
(version-2019). Both GENIE and GiBUU are based on different physics approach
i.e. they employ different nuclear models to incorporate nuclear effects and
various neutrino interaction processes. In an attempt to constrain systematic
uncertainties in Ar, we have calculated the ratio of Ar/H using calorimetric
and MLA. Further, we have taken a step to check the multi-generator approach
and combined the final state interaction (FSI) kinematics as obtained from
both generators. Our results show that on adopting a multi-generator approach
systematic uncertainties were significantly reduced.

The paper is organized in the following sections: We describe the neutrino
event generators GENIE and GiBUU used in this work in Section II. We outline
the simulation, experimental details, and the analysis strategy in Section
III. The results are discussed in Section IV followed by the Conclusions in
Section V.

##  II Neutrino event simulation tools

A neutrino event generator simulates neutrino-nucleus interactions for a
particular detector setting and propagates the products out of the struck
nucleus. These generators take into account, both the initial neutrino-nucleon
interaction and the final state interactions. Event generators must be
efficient enough to simulate interactions on a wide range of nuclei, for a
broad neutrino energy spectrum, for all the neutrino (anti-neutrino) flavors,
and must have accurate energy-dependent cross-section. The role of the event
generator becomes crucial as it provides information about the physics of the
interaction taking place. But we lack complete information about the initial
neutrino-nucleus interaction and the subsequent re-interactions taking place
within the nucleus that lead to modified outgoing products. Thus different
neutrino event generators incorporate different nuclear and physics models to
describe the nucleus and the various energy-dependent interaction processes.
Different neutrino experiments use different neutrino event generators, for
eg. NEUT neut  is used by the T2K experiment, GENIE genie  is used by Fermilab
experiments like NOvA 7e , MicroBooNE 7d , MINERvA 7b , T2K7f  and MINOS7c .
NuWRO nuwro  is being used for the comparison of experiments and calculations.
Another generator named, GiBUU gibuu  can be used for a wide range of
interactions between a nucleus- and -neutrino/electron/photon/pion/nucleus.

Our selected event generators i.e. GENIE and GiBUU, considerably differ in the
selection of nuclear models and description of above mentioned neutrino-
nucleus interactions. GENIE is developed in C++ language and is a ROOT7a
based neutrino event generator designed using object-oriented methodologies.
Whereas, GiBUU, written in FORTRAN routines, is based on a coupled set of
semiclassical kinetic equations. These equations describe the dynamics of a
hadronic system in phase space and time.

Some differences and similarities between these two event generators can be
outlined as follows. For nuclear density distribution both the generators use
Woods-Saxon parametrizationwoodsaxon . The value of axial mass used by GiBUU
is $M_{A}$= 1 $GeV/c^{2}$ while the value of axial mass in GENIE is 0.99
GeV/$c^{2}$. The vector form factor, BBBA07 refGibuuVct  is used by GiBUU
while GENIE uses BBBA05 refGenieVct . To describe the nuclear density
distribution, both the generators use Woods-Saxon parametrization woodsaxon .
For simulating QE interactions that are dominant below 1 GeV, RFG modelrefrfg
is used to describe the nuclear structure by both the generators. But the RFG
model used in GiBUU has an additional density-dependent mean-field potential
term in which all nucleons are assumed to be bound while in GENIE it is based
on the model suggested by A. Bodek and J.L. Ritchie ABodak  that includes
short-range nucleon-nucleon correlations. The QE scattering in GENIE is
modeled according to the Llewellyn Smith modelqemodel . For the description of
QE cross-section in GiBUU, one can refer to minibooneQE ; moselQE . Below 1
GeV, there is an interplay of other interactions also. Nucleon-nucleon
correlations involving two-particle two-hole (MEC/2p2h) excitations mainly
populate in this low energy region 2p2h .

The resonance interaction dominates in the energy range from 0.5 $<E_{\nu}<$
10 GeV. There are 13 variants of resonance modes in GiBUU and it is the MAID
analysis refGibuuMAID1 ; refGibuuMAID2  of the electron scattering data that
defines vector form factors for all the modes. GENIE accommodates 16 resonance
modes defined by Rein Sehgal modelRShegal . At higher energies, DIS
interaction dominates where the interaction is at the quark level. GiBUU and
GENIE implement PYTHIA gibuudis  and Bodek and Yang bodekyang  respectively to
describe the DIS process. GENIE incorporates Aivazis, Olness, and Tung model
discharm  also, particularly for DIS charm production.

The initially produced particles at the neutrino-nucleon interaction vertex
are further modified due to re-interactions with other nucleons within the
nucleus, such re-interactions are referred to as Final State Interactions
(FSI). Both GENIE and GiBUU apply different approaches to model FSI. FSI in
GENIE is modeled using the simulation package INTRANUKE int1 ; int2 ; dytman1
; dytman2 . GiBUU models the FSI by the semi-classical BUU equations. A mean-
field potential and collision terms keep different particle species coupled to
each other. Further information on modeling differences, corrections, and
improvements being implemented in the two generators, one can refer to ref45 ;
refGenieMEC ; refGibuu2p2h .

##  III SIMULATION AND EXPERIMENTAL DETAILS

For simulating a DUNE-like experiment, we have considered the DUNE-ND flux
ndflux  that spans over an energy range of 0.125-10 GeV as shown in Figure 1.
In this energy range, there is an interplay of several interaction processes
viz. quasi-elastic (QE), resonance (RES) from $\Delta$ resonant decay, and
contribution from higher resonances, two particle-two hole(2p2h/MEC), and deep
inelastic scattering(DIS) interaction processes. We have considered all these
interaction processes in our analysis.

The incoming neutrino beam is dominated by muon flavor which is generated at
the LBNF facility. At LBNF, a beam of protons in the energy range 60-120 GeV
is produced at the main injector accelerator and is made to smash a graphite
target. This results in the production of pions and kaons which are directed
to a decaying pipe with the help of magnetic horns. These pions and kaons
further decay into neutrinos and leptons of all flavors. Finally an intense,
wide-band, $\nu(\bar{\nu})$ beam is produced with an initial beam power of 1.2
MW, producing 1.1 $\times$ $10^{21}$ protons on target per year dune .

Figure 1: The DUNE-ND flux as a function of neutrino energy used in our work.

Inclusive event samples of 1 million $\nu_{\mu}$-Ar and $\nu_{\mu}$-H each,
are generated using both the neutrino event generators i.e. GENIE and GiBUU.
The choice of nuclear targets is such so that we can quantify nuclear effects
in Ar. As we know DUNE-ND and FD will employ liquid Ar in both the detectors,
it is crucial to study neutrino-Ar scattering. We have selected H because it
is a single proton and hence it is devoid of nuclear effects. It is very
important to have a control sample that can directly constrain nuclear effects
which can help in reducing systematic uncertainties. Comparing measurements on
Ar and H will impose stringent constraints on the nuclear smearing in Ar. The
Straw Tube Tracker (STT), a component in SAND nd4 , which is one of the
components in the DUNE-ND will be studying $\nu$-Ar and $\nu$-H interactions
along with other nuclear targets (For eg. Ca(A=40), C(A=12)) as well. STT will
be using H in the form of ’solid hydrogen’ and will obtain interaction
kinematics with H by subtracting interactions on C$H_{2}$ roberto  and Carbon.
This is the motivation behind our selection of targets as, without a
complementary H target, it is difficult to achieve precision measurement free
from nuclear smearing. Although the DUNE collaboration does not have any plan
to use an H target in any form in the FD module as planned for the ND complex
(SAND), but for simulation purposes, we have replaced Ar with an H target. We
have estimated the probability with H target as a function of reconstructed
neutrino energy using a single generator and multi-generator approach as well.
To quantify the amount of nuclear effect we have further used it for
calculating the Ar/H ratio as presented in Figures 6 and 7.

###  III.1 Analysis strategy

The following section describes the two analysis techniques that we have
applied in our work i.e. Calorimetric and Linear regression methods.

####  III.1.1 Calorimetric technique

The calorimetric method c1 ; c2  of neutrino energy reconstruction estimates
the neutrino energy by adding the energies deposited by all the particles
observed in the detector. This method in comparison to the traditional
kinematic method k1 ; k2 ; k3 ; k4  of neutrino energy reconstruction is more
effective for mainly two reasons. Firstly, the kinematic method is only
restricted to the QE interaction process as it is based on lepton kinematics
(outgoing muon) only while the calorimetric method applies to all types of
interaction processes. Secondly, the calorimetric method is in a way
insensitive to nuclear effects while the kinematic method is more prone to
nuclear effects for interaction processes other than QE. But only in an ideal
detector where there will be no missing energy i.e. even the particles with
the lowest energies are detected, then only the calorimetric method will
reconstruct effectively. Although the impact of FSI will still be prevalent in
neutrino interactions due to which the energy transferred to the nucleus may
get redistributed to several particles but the total energy will remain the
same mosel .

For calculating the neutrino energy ($E_{\nu}^{Cal}$) using the calorimetric
approach, $E_{\nu}^{Cal}$ c1  can be calculated as:

$E_{\nu}^{Cal}=E_{l}+\Sigma_{i}T_{i}^{n}+\epsilon_{n}+\Sigma_{m}E_{m}$ (1)

where $E_{l}$ is the energy of the outgoing final state lepton, $T_{i}^{n}$ is
the kinetic energy of the outgoing nucleons (protons and/or neutrons),
$\epsilon_{n}$ represents the corresponding separation energies of the
outgoing nucleons and $E_{m}$ is the total energy of any other particle
produced. The calorimetric energy in our work is calculated using Eq.(1).

####  III.1.2 Multivariate linear regression

We have applied the multivariate linear regression model which comes under
supervised learning. In supervised learning, we have the required dataset that
is the input and have an idea about the output. Linear regression predicts a
real-valued output based on an input value. Our work gives multiple values,
known as feature variables in the input, which is termed as multivariate
analysis. A python-based script is developed that takes the FSI kinematics
(features for our algorithm) as an input to train the algorithm and predict
the neutrino energy. The multiple features are the number of particles per
event, momenta of all the outgoing particles per event, true neutrino energy,
the energy of outgoing muon ($E_{\mu}$), energy transfer ($q_{0}$), visible
momenta of each particle, squared four momenta ($Q^{2}$), and reconstructed
neutrino energy. This dataset is further divided into a training set and
testing set as 70$\%$ and 30$\%$ respectively. The training set is used to
train the MLA and the testing set is used to check the predictions done by the
MLA. The algorithm learns from the input variables about the reconstructed
neutrino energy per event and then predicts the reconstructed neutrino energy
in the output based on its learning from the feature variables provided to it.
The regression algorithm performs a mapping between a dependent and an
independent variable which is a continuous mapping and can be used to predict
the energy of a particle.

Figure 2: Event distribution at the FD as a function of reconstructed neutrino
energy represented by blue (Calorimetric method), red (linear regression
method) lines, and as a function of true neutrino energy represented by green
lines for Ar target from GENIE and GiBUU in the left and right panels
respectively.

Figure 3: Event distribution at the ND as a function of reconstructed neutrino
energy represented by blue (Calorimetric method) and red lines respectively
and as a function of true neutrino energy represented by green lines for Ar
target from GENIE and GiBUU in the left and right panels respectively.

Figure 4: Event distribution at the ND as a function of reconstructed neutrino
energy represented by blue (Calorimetric method) and red (linear regression
method) lines respectively and as a function of true neutrino energy
represented by green lines for H target from GENIE and GiBUU in the left and
right panels respectively.

Figure 5: Muon survival probability as a function of reconstructed neutrino
energy represented by blue (Calorimetric method) and red (linear regression
method) lines respectively and as a function of true neutrino energy
represented by green line for Ar target from GENIE and GiBUU in the left and
right panels respectively.

##  IV Result and discussion

On testing the viability of the linear regression method for reconstructing
the neutrino energy, we find that the method can competently reconstruct
neutrino energy as good as the calorimetric method. Figures 2 and 3, represent
the $\nu_{\mu}$-Ar event distribution as a function of reconstructed neutrino
energy at the FD and ND respectively. Both the figures display the event
distribution calculated using the calorimetric method (blue lines) and linear
regression method (red lines) by GENIE and GiBUU in the left and right panels
respectively. The green line in both figures represents event distribution as
a function of true neutrino energy. Figure 4 represents the $\nu_{\mu}$-H
event distribution as a function of reconstructed neutrino energy at the ND
only. We again notice that the reconstructed neutrino energy is very well
reconstructed using the linear regression method.

Figure 5 represents the muon survival probability by taking the ratio of FD
and ND event distributions as obtained from GENIE and GiBUU, in the left and
right panels respectively for Ar. The blue line represents the muon survival
probability as calculated from the calorimetric method and the red line
represents the same as calculated using the linear regression method. We
notice that both the blue and red lines completely overlap each other
indicating that the linear regression method works well for neutrino energy
reconstruction.

In an attempt to estimate the impact of systematic uncertainties arising due
to nuclear effects in neutrino energy reconstruction, we have taken the ratio
of muon survival probabilities for Ar over H and presented it in Figures 6 and
7, for GENIE and GiBUU respectively. From Figure 6, we note that the ratio
from GENIE as represented in the left panel has a lot of fluctuation up to 3
GeV as calculated from both the calorimetric (blue circles) and linear
regression (red circles) methods. We also note two sharp peaks, one below 1
GeV and another around 2 GeV. Ratio when calculated as a function of true
neutrino energy (represented by green circles) shows lesser fluctuation as
compared to the calculation performed with reconstructed neutrino energy. On
the other hand, the right panel of Figure 6, which represents the ratio
calculations from GiBUU shows lesser fluctuations up to 3 GeV, and no sharp
peaks are observed as in the case with GENIE.

Figure 6: Ar/H Ratio of muon survival probability by Calorimetric method (blue
line) and linear regression method (red line) from GENIE and GiBUU in the left
and right panels respectively. Figure 7: Ar/H ratio from GENIE (blue circles),
GiBUU (red circles) and GENIE+GiBUU (green circles) using only linear
regression.

In Figure 7, we have presented three ratios i.e. from GENIE (blue circles),
GiBUU (red circles), and GENIE+GiBUU (green circles) using only the linear
regression model. For calculating the Ar/H ratio for the GENIE+GiBUU dataset,
we simply combined the final state kinematics as obtained from the two
generators and performed all the calculations again. From Figure 7, we can
observe that the GENIE+GiBUU dataset shows a better result as the ratio is
around unity for the entire energy range.

##  V Conclusion

Neutrino interaction event rates are a convolution of neutrino flux, neutrino-
nucleon cross-section, nuclear effects and detector response. The first three
factors depend upon the incoming neutrino energy whereas the detector response
depends upon the efficiency and threshold of the detector. Experimentalists
use event generators to connect the observed topologies and interaction
kinematics. Due to scarcity of data, there are cross-sectional uncertainties
where unknown fundamental neutrino scattering kinematics further distort vital
information that lead to systematic uncertainties. To construct a precise
nuclear model, we need exact information about all the convoluted factors.

We find from Figure 6, that the two widely used neutrino event generators,
GENIE and GiBUU have non-negligible systematic uncertainties in their nuclear
models and especially in the energy range from 1-3 GeV. QE and RES interaction
processes are the most dominant interactions in this energy range. This energy
range has other interaction processes like multinucleon, 2p-2h, and coherent
interactions that are still not modeled properly. We can note that GENIE
results have more fluctuations than GiBUU results, which indicates a larger
amount of uncertainties in the GENIE in the low energy range. From Figure 7,
we observe a significant improvement in the Ar/H ratio, represented by green
circles. This shows that on combining the FSI kinematics from both the
generators, fluctuation in the ratio was minimal, indicating a significant
reduction in the systematic uncertainties. A study of high-statistics $\nu$-H
scattering data is necessary to supplement our knowledge of the poorly known
nuclear effects. Through this work, we may conclude that a multi-generator and
multi-target approach might be useful to constrain nuclear effects in the
future long-baseline neutrino experiments like DUNE.

##  VI Acknowledgment

This work is supported by the Department of Physics, University of Lucknow,
Lucknow.

## References

## References

  * (1) F. P. An et al. [DAYA-BAY Collaboration], Observation of electron-antineutrino disappearance at Daya Bay, Phys. Rev. Lett. 108 (2012) 171803 [arXiv:1203.1669 [hep-ex]].
  * (2) F. P. An et al., [DAYA-BAY Collaboration], Improved Measurement of Electron Antineutrino Disappearance at Daya Bay, Chin. Phys. C 37 (2013) 011001 [arXiv:1210.6327 [hep-ex]].
  * (3) J. K. Ahn et al., [RENO Collaboration], Observation of Reactor Electron Antineutrino Disappearance in the RENO Experiment, Phys. Rev. Lett. 108 (2012) 191802 [arXiv:1204.0626 [hep-ex]].
  * (4) P. F. de Salas et al, 2020 Global reassessment of the neutrino oscillation picture, arXiv:2006.11237v2 [hep-ph] 19 Jan 2021.
  * (5) Srishti Nagu et al, Nuclear Effects and CP Sensitivity at DUNE, https://doi.org/10.1155/2020/5472713.
  * (6) Jaydip Singh, Srishti Nagu et al, Quantifying multinucleon effect in Argon using high-pressure TPC, Nuclear Physics B 957 (2020) 115103.
  * (7) L. Alvarez-Ruso et al. NuSTEC White Paper: Status and Challenges of Neutrino-Nucleus Scattering, arXiv:1706.03621v2 [hep-ph] 15 Jun 2017.
  * (8) Srishti Nagu, Jaydip Singh, Jyotsna Singh, R.B. Singh, Impact of cross-sectional uncertainties on DUNE sensitivity due to nuclear effects, https://doi.org/10.1016/j.nuclphysb.2019.114888.
  * (9) Bhat, P. Multivariate analysis methods in particle physics. Annu. Rev. Nucl. Part. Sci. 61, 281–309 (2011).
  * (10) The ALEPH Collaboration. Determination of —Vub— from the measurement of the inclusive charmless semileptonic branching ratio of b hadrons. Eur. Phys. J. C 6, 555–574 (1999).
  * (11) OPAL Collaboration. A measurement of the production of D$\pm$ mesons on the Z0 resonance. Z. Phys. C 67, 27–44 (1995).
  * (12) Chiappetta, P., Colangelo, P., De Felice, P., Nardulli, G. and Pasquariello, G. Higgs search by neural networks at LHC. Phys. Lett. B 322, 219–223 (1994).
  * (13) Roe, B. P. et al. Boosted decision trees as an alternative to artificial neural networks for particle identification. Nucl. Instrum. Methods A 543, 577–584 (2005).
  * (14) Denby, B. Neural networks and cellular automata in experimental high-energy physics. Comput. Phys. Commun. 49, 429–448 (1988).
  * (15) Rosenblatt, F. Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms (Spartan Books, Berlin, 1961).
  * (16) Reed, R. and Marks, R. Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks (MIT Press, Cambridge, 1999).
  * (17) Evans, L. and Bryant, P. LHC machine. J. Instrum. 3, S08001 (2008).
  * (18) Aad, G. et al. Observation of a new particle in the search for the standard model Higgs boson with the ATLAS detector at the LHC. Phys. Lett. B 716, 1–29 (2012).
  * (19) Chatrchyan, S. et al. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC. Phys. Lett. B 716, 30–61 (2012).
  * (20) Alexander Radovic et al, Machine learning at the energy and intensity frontiers of particle physics, https://www.nature.com/articles/s41586-018-0361-2.
  * (21) Philip Hamacher-Baumann, et al., Neutrino-hydrogen interactions with a high-pressure TPC, arXiv :2005 .05252v1 [physics .ins -det], 11 May 2020.
  * (22) Alan Bross, et al., https://indico.fnal.gov/event/16205/contribution/1/material/0/0.pdf.
  * (23) J. Martín-Albo, A pressurized argon gas TPC as DUNE near detector, arXiv :1610 .07803v1 [physics .ins -det], 25 Oct. 2016.
  * (24) The DUNE Collaboration, Deep Underground Neutrino Experiment (DUNE) Near Detector Conceptual Design Report, May 6, 2020.
  * (25) Adam M. Terwilliger et al, Vertex Reconstruction of Neutrino Interactions using Deep Learning, https://www.osti.gov/servlets/purl/1408000.
  * (26) Junze Liu et al, Deep-Learning-Based Kinematic Reconstruction for DUNE, arXiv:2012.06181v2 [physics.ins-det] 14 Dec 2020.
  * (27) Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, Neural Computation 1, 541 (1989).
  * (28) B. Abi et al, Neutrino interaction classification with a convolutional neural network in the DUNE far detector, arXiv:2006.15052v2 [physics.ins-det] 10 Nov 2020.
  * (29) A. Aurisano et al., Journal of Instrumentation 11 (09), P09001, arXiv:1604.01444.
  * (30) R. Acciarri et al. (MicroBooNE), JINST 12 (03), P03011, arXiv:1611.05531 [physics.ins-det].
  * (31) C. Andreopoulos, et al., The GENIE neutrino Monte Carlo generator, Nucl. Instrum. Methods A 614 (2010) 87–104.
  * (32) O. Buss, T. Gaitanos, K. Gallmeister, H. van Hees, M. Kaskulov, O. Lalakulich, A.B. Larionov, T. Leitner, J. Weil, U. Mosel, Transport-theoretical description of nuclear reactions, Phys. Rep. 512 (2012) 1–124.
  * (33) https://repo.nd280.org/viewvc/T2K/NIWG/neut_doc
  * (34) D. Ayres et al. (NOvA Collaboration) (2004), hep-ex/0503053.
  * (35) H. Chen et al. (MicroBooNE Collaboration) (2007), FERMILAB-PROPOSAL-0974.
  * (36) D. Drakoulakos et al. (Minerva Collaboration) (2004), hep-ex/0405002.
  * (37) Y. Itow, Nucl.Phys.Proc.Suppl. 112, 3 (2002).
  * (38) P. Adamson et al. (MINOS Collaboration), Phys.Rev. D77, 072002 (2008), 0711.0769.
  * (39) Golan T, Sobczyk J T and Zmuda J 2012 Nucl. Phys. Proc. Suppl. 229-232 499.
  * (40) Rene Brun and Fons Rademakers,’ROOT - An Object Oriented Data Analysis Framework’, Nucl.Inst.Meth. in Phys. Res. A 389, 81-86 (1997).
  * (41) N.Schwierz, arXiv:0709.3525 [nucl-th]
  * (42) A. Bodek, S. Avvakumov, R. Bradford, and H. S. Budd, Eur.Phys.J. C53, 349 (2008), 0708.1946.
  * (43) R. Bradford, A. Bodek, H. S. Budd, and J. Arrington, Nucl.Phys.Proc.Suppl. 159, 127 (2006), hep-ex/0602017.
  * (44) R. Smith and E. Moniz, Nucl.Phys. B43, 605 (1972).
  * (45) A. Bodek and J. Ritchie, Phys.Rev. D24,1400 (1981).
  * (46) C. H. Llewellyn Smith, Neutrino reactions at accelerator energies, Phys. Rept. 3 (1972)261.
  * (47) O. Lalakulich, U. Mosel and K. Gallmeister, Neutrino energy reconstruction in quasielastic-like scattering in the MiniBooNE and T2K experiments, Physical Review C 86, 054606 (2012).
  * (48) K. Gallmeister et al, ’Neutrino-Induced Reactions on nuclei’, arXiv:1605.09391v3 [nucl-th] Sep (2016).
  * (49) S. Dolan, U. Mosel, K. Gallmeister, L. Pickering, and S. Bolognesi Phys. Rev. C 98, 045502 – Published 15 October 2018.
  * (50) D. Drechsel, O. Hanstein, S. Kamalov, and L. Tiator, Nucl.Phys. A645, 145 (1999), nucl-th/9807001.
  * (51) D. Drechsel and L. Tiator, J.Phys. G18, 449 (1992).
  * (52) D. Rein and L. Sehgal, Phys.Lett. B104, 394 (1981).
  * (53) O. Lalakulich, K. Gallmeister, U. Mosel, Neutrino nucleus reactions within the GiBUU model, arXiv:1110.0674 [hep-ph] Oct 2011.
  * (54) A. Bodek and U. K. Yang, Higher twist, $\xi_{\omega}$ scaling, and effective LO PDFs for lepton scattering in the few GeV region, J. Phys., vol. G29, pp. 1899–1906, 2003.
  * (55) M. A. G. Aivazis, F. I. Olness and W.-K. Tung, Leptoproduction of heavy quarks. 1. General formalism and kinematics of charged current and neutral current production processes, Phys.Rev., vol. D50, pp. 3085–3101, 1994.
  * (56) S. Dytman, Final state interactions in neutrino-nucleus experiments, Acta Phys. Polon. B 40 (2009) 2445.
  * (57) S. Dytman, A.S. Meyer, Final state interactions in GENIE, AIP Conference Proceedings 1405 (2011) 213, https://doi .org /10 .1063 /1 .3661588.
  * (58) S. Dytman and A.S. Meyer, ’Final State Interactions in GENIE’, AIP Conference Proceedings 1405, 213, https://doi.org/10.1063/1.3661588 (2011).
  * (59) S. Dytman, ’Final State Interactions in NeutrinoNucleus Experiments’, ACTA PHYSICA POLONICA B, No 9, Vol. 40 (2009).
  * (60) Pilar Coloma, Patrick Huber, Chun-Min Jen and Camillo Mariani, Neutrino-nucleus interaction models and their impact on oscillation analyses, arXiv:1311.4506 [hep-ph].
  * (61) T. Katori (2013), 1304.6014.
  * (62) O. Lalakulich, K. Gallmeister, and U. Mosel, Phys.Rev. C86, 014614 (2012),1203.2935.
  * (63) http://home.fnal.gov/~ljf26/DUNE2015CDRFluxes/NuMI_Improved_80GeV_StandardDP/g4lbne_v3r2p4b_FHC_ND_globes_flux.txt.
  * (64) The DUNE Collaboration, Deep underground neutrino experiment (DUNE) far detector technical design report, vol. II, arXiv :2002 .03005v2 [hep -ex], 25 Mar 2020.
  * (65) H. Duyang et al, A precise determination of (anti)neutrino fluxes with (anti)neutrino-hydrogen interactions, June 2019, Physics Letters B 795, 10.1016/j.physletb.2019.06.003
  * (66) A.M. Ankowski, et al., Missing energy and the measurement of the CP-violating phase in neutrino oscillations, FERMILAB-PUB-15-320-T, RM3-TH/15-12, Phys. Rev. D 92 (2015), 091301(R).
  * (67) A.M. Ankowski, et al., Comparison of the calorimetric and kinematic methods of neutrino energy reconstruction in disappearance experiments, FERMILAB-PUB-15-301-T, RM3-TH/15-10.
  * (68) M.H. Ahn, et al., K2K Collaboration, Phys. Rev. Lett. 90 (2003) 041801.
  * (69) A.A. Aguilar-Arévalo, et al., MiniBooNE Collaboration, Phys. Rev. Lett. 100 (2008) 032301.
  * (70) V.V. Lyubushkin, et al., NOMAD Collaboration, Eur. Phys. J. C 63 (2009) 355.
  * (71) Y. Nakajima, et al., SciBooNE Collaboration, Phys. Rev. D 83 (2011) 012005.
  * (72) U. Mosel, in Proceedings of the Prospects in Neutrino Physics (NuPhys2014): London, UK, December 15-17,2014,(London, UK, 2015).


# Do the Frankenstein, or how to achieve better out-of-distribution
performance with manifold mixing model soups

Hannes Fassold  JOANNEUM RESEARCH - DIGITAL, Austria

###### Abstract

The standard recipe applied in transfer learning is to finetune a pretrained
model on the task-specific dataset with different hyperparameter settings and
pick the model with the highest accuracy on the validation dataset.
Unfortunately, this leads to models which do not perform well under
distribution shifts, e.g. when the model is given graphical sketches of the
object as input instead of photos. In order to address this, we propose the
_manifold mixing model soup_ , an algorithm which mixes together the latent
space manifolds of multiple finetuned models in an optimal way in order to
generate a fused model. We show that the fused model gives significantly
better out-of-distribution performance (+3.5 % compared to best individual
model) when finetuning a CLIP model for image classification. In addition, it
provides also better accuracy on the original dataset where the finetuning has
been done.

Keywords: Latent space manifold, transfer learning, finetuning, distribution
shift, image classification

##  1 Introduction

Large pretrained visual foundation models like CLIP [Radford et al., 2021] or
CoCa [Yu et al., 2022] got very popular recently due to their great
performance for a variety of computer vision tasks, either as zero-shot
learner (without finetuning) or serving as a base for task-specific finetuning
on a smaller dataset.

Typically, multiple models are finetuned with different hyperparameters (like
learning rate, weight decay or data augmentation strategy), using the same
pretrained model as initialization. From those, the model with the best
accuracy on the validation dataset is selected. Unfortunately, this procedure
leaves out important information which has been learned in the latent space
manifolds (individual layers or a collection of layers) of the remaining
finetuned models. As shown in [Wortsman et al., 2022a], even fusing multiple
finetuned models in a very straightforward way by averaging them makes the
fused model already significantly more robust to distribution shifts in the
data.

Motivated by this, we propose the _manifold mixing model soup_
(_ManifoldMixMS_) algorithm. Instead of simple averaging, it uses a more
sophisticated strategy to generate the fused model. Specifically, it
partitions a neural network model into several latent space manifolds (which
can be individual layers or a collection of layers). Afterwards, from the pool
of finetuned models available after hyperparameter tuning, the most promising
ones are selected and their latent space manifolds are mixed together
individually. The optimal mixing coefficient for each latent space manifold is
calculated automatically via invoking an optimization algorithm. The fused
model we retrieve with this procedure can be thought as sort of a
"Frankenstein" model, as it integrates (parts of) individual model components
from multiple finetuned models into one model.

The remainder of the work is organized as follows. In section 2 we revise
related work. Section 3 presents the proposed manifold mixing model soup
algorithm. Section 4 presents the experiments and evaluation, which show the
advantage of the proposed algorithm with respect to the state of the art,
especially with respect to distribution shifts in the data. Finally, section 5
concludes the paper.

##  2 State of the Art

A variety of methods has been proposed recently for merging several models
into one fused model, with the aim of increasing the generalization capability
and robustness to distribution shifts of the fused model.

A classical method is _stochastic weight averaging_ (SWA) [Izmailov et al.,
2018], which produces the fused model by averaging the weights of a set of
models sampled from the final stages of a single training run. They show that
SWA leads to solutions of the optimization problem that are wider than the
optima found by standard SGD, which in turn leads to a better generalization
of the fused model.

The authors of [von Oswald et al., 2021] propose to replicate and learn in
parallel a subset of weights (e.g. the batch-norm and classification layers)
in a late phase of neural network learning. These late-phase weights define an
ensemble of models which share every other weight. These parameters are then
optimized independently and subsequently averaged.

In [Jolicoeur-Martineau et al., 2023] an algorithm called _population
parameter averaging_ (PAPA) is presented, which trains a population of models
in parallel (with different learning rate, data augmentation strategies etc.).
It improves overall performance by infrequently replacing the weights with the
population average and frequently pushing all model weights slightly towards
the population average. A disadvantage of this method is the high memory
consumption, as the gradients for _several_ parallel training runs have to be
kept up.

The work of [Wortsman et al., 2022a] shows that averaging the weights of
multiple models finetuned with different hyperparameter configurations often
improves accuracy and out-of-distribution performance of the averaged model.
They propose two different averaging algorithms (which they call "souping"),
_uniform soup_ and _greedy soup_. The uniform soup is a very simple procedure,
as it does an averaging of all finetuned models. In contrast, the greedy soup
is constructed by sequentially adding each model as a potential ingredient to
the soup, and only keeping the model if it improves the performance of the
averaged model. Our proposed _manifold maxing model soup_ algorithm (see
section 3) is inspired by their greedy soup algorithm. But in contrast to
them, we are partitioning the model into several components (latent space
manifolds) and do an optimization in order to calculate the optimal mixing
factor for each component.

In [Matena and Raffel, 2022] it is shown that uniform averaging of several
finetuned models corresponds to making an isotropic Gaussian approximation to
their posteriors. The authors propose an alternative merging procedure based
on the Laplace approximation, where each model’s posterior is approximated as
a Gaussian distribution whose precision matrix (inverse of the covariance
matrix) corresponds to its Fisher information.

The authors of [Wortsman et al., 2022b] found that while fine-tuning a
pretrained vision model improves performance on the downstream task, it also
tends to decrease accuracy on the original pretraining task. They therefore
propose a robust finetuning procedure called _WiSE-FT_ that computes a
weighted average of the original pretrained parameters and the finetuned
parameters. Different weighting values produce different trade-offs between
pretraining and finetuning task performance.

##  3 Manifold mixing model soup

In the following, we outline the proposed algorithm for generating a fused
model – the _manifold mixing model soup_ – from its ingredients (the finetuned
models after hyperparameter tuning). The algorithm pseudocode can be seen in
Algorithm 1.

We first sort all $n$ finetuned models $\theta_{i}$ (with $i=0,...,n-1$) in
descending order, based on their validation accuracy $ValAcc(\theta_{i})$ on
the original dataset which was used for finetuning. So $\theta_{0}$ is the
model (to be precise, its finetuned parameters) with the highest validation
accuracy, whereas $\theta_{n-1}$ is the one with the lowest validation
accuracy.

Each model $\theta_{i}$ is partitioned into $m$ components $\theta_{i}^{j}$,
where $\theta_{i}^{j}$ corresponds to a single latent space manifold, and
$j=1,...,m$. Each latent space manifold comprises either a single layer or a
collection of layers, corresponding to one building block of the model.
Typically, we partition a model into $10-30$ components. A finer partitioning
makes the subsequent optimization more difficult, whereas a too coarse
partitioning provides not enough freedom to optimize the mixing of the latent
space manifolds individually. The motivation for aggregating a collection of
layers into one component is to reduce the number of variables during
optimization, which makes it easier for the optimizer to find a good optimum.
For each model, of course the same partitioning is employed.

The fused model $\Psi$ is now calculated in an sequential way, by iteratively
mixing promising ingredient models with it. At first, the fused model is set
to the best finetuned model via $\Psi=\theta_{0}$, and the variable $k$, which
counts the number of models which have been mixed so far into the fused model,
is set to $1$.

In each iteration (for $i=1,...,n-1)$, we try now to mix the candidate model
$\theta_{i}$ with the current fused model $\Psi$ in an optimal way, with the
aim of increasing the validation accuracy of the updated fused model
$\Psi^{\prime}$ (which includes $\theta_{i}$) on the original dataset.

In order to save computation time, we skip the optimization step for a
candidate model $\theta_{i}$ for which it is unlikely that we get an increase
in the validation accuracy by mixing $\theta_{i}$ into the current fused model
$\Psi$. For that, we generate the "approximate average" model $\tilde{\Psi}$
via

$\tilde{\Psi}=\frac{k}{k+1}\cdot\Psi+\frac{1}{k+1}\cdot\theta_{i}$ (1)

and test whether the condition $ValAcc(\tilde{\Psi})>\tau\cdot ValAcc(\Psi)$
is fulfilled. If so, we continue with this iteration. If it is not fulfilled,
we skip the following steps of this iteration, so candidate model $\theta_{i}$
will not be taken into account. The motivation for the specific combination
provided in Equation (1) is that $\tilde{\Psi}$ calculated in this way
corresponds approximately to the _average_ of all candidate models (like in
[Wortsman et al., 2022a]) which have been mixed so far into the fused model
(including $\theta_{i}$), _if we assume_ that the optimization did not change
the mixing coefficients drastically from their provided initial values. We set
the constant $\tau$ to $0.998$.

Having identified $\theta_{i}$ as a promising candidate model, in the next
step we determine the optimal factors for mixing its latent space manifolds
into the current fused model $\Psi$. For this, we define the updated fused
model $\Psi^{\prime}(\lambda)$ as a _component-wise_ convex combination of
$\Psi$ and $\theta_{i}$ via

$\Psi^{\prime}(\lambda)^{j}=\lambda^{j}\cdot\Psi^{j}+(1-\lambda^{j})\cdot\theta^{j}_{i}$
(2)

for all components $j=1,...,m$. Note that $\Psi^{\prime}(\lambda)$ is a
function of the mixing vector $\lambda$. The mixing factor
$\lambda^{j}\in[0,1]$ determines how much of the $j-th$ component (latent
space manifold) of the candidate model $\theta_{i}$ is mixed into the current
fused model $\Psi$. The component-wise convex combination of the two models
allows an optimizer to explore the latent space manifolds of the models $\Psi$
and $\theta_{i}$ in a very flexible way, in order to find the optimal mixing
vector $\lambda^{*}\in\mathbb{R}^{m}$ which gives the highest validation
accuracy for the updated fused model $\Psi^{\prime}$.

For the subsequent optimization step, we set up the optimization problem to
solve as

$\lambda^{*}=\operatorname*{arg\,max}_{\lambda\in[0,1]^{m}}\left(ValAcc\left(\Psi^{\prime}\left(\lambda\right)\right)\right)$
(3)

where $[0,1]^{m}$ is the $m-$dimensional unit interval. Via the constraint
$\lambda\in[0,1]^{m}$ we ensure that a convex combination is done for each
component, so we are in fact _interpolating linearly_ between the latent space
manifolds $\Psi^{j}$ and $\theta^{j}$. The model $\Psi^{\prime}(\lambda)$ can
be calculated via Equation (2).

For solving this optimization problem, we employ the _Nevergrad_
111https://facebookresearch.github.io/nevergrad/ optimization package. It
provides a large variety of black-box _derivative-free_ optimization
algorithms together with a sophisticated heuristic [Liu et al., 2020] to
select the best optimizer based on the characteristic (number of variables,
allowed budget for function evaluations etc.) of the optimization problem. As
the initial value for the mixing factors, we set $\lambda^{j}=k/(k+1)$ for
$j=1,...,m$ with a similar motivation as explained earlier for Equation (1).

We invoke now the optimizer in order to calculate the optimal mixing vector
$\lambda^{*}$ which give the highest validation accuracy on the dataset used
for finetuning. The optimal updated fused model can be calculated now via
${\Psi^{\prime}}^{*}=\Psi^{\prime}(\lambda^{*})$. We check now whether the
condition $ValAcc({\Psi^{\prime}}^{*})>ValAcc(\Psi)$ is fulfilled. If so, we
have found a better fused model ${\Psi^{\prime}}^{*}$ by mixing $\theta_{i}$
into it. Consequently, we replace $\Psi$ by ${\Psi^{\prime}}^{*}$ and increase
$k$ by 1. If this is not the case, we keep the current fused model $\Psi$ and
$k$ as they are.

After iterating over all candidate models $\theta_{i}$ for $i=1,...,n-1$ we
retrieve a final fused model $\Psi$ (the _manifold mixing model soup_), which
mixes together the $k$ selected candidate models / ingredients in an optimal
way.

Algorithm 1 Manifold mixing model soup algorithm

Finetuned models $\\{\theta_{0},...,\theta_{n-1}\\}$ as result of
hyperparameter tuning

Partitioning of a model $\zeta$ into $m$ components (latent space manifolds)
$\zeta^{j}$ for $j=1,...,m$

Function $ValAcc(\zeta)$ which calculates validation accuracy for $\zeta$ on
dataset used for finetuning

$\\{\theta_{0},...,\theta_{n-1}\\}\leftarrow
sort(\\{\theta_{0},...,\theta_{n-1}\\})$ $\triangleright$ _Sort
$\\{\theta_{0},...,\theta_{n-1}\\}$ in descending order based on
$Valacc(\theta_{i})$ _

$k\leftarrow 1$ $\triangleright$ _Number of candidate models mixed into fused
model_

$\Psi\leftarrow\theta_{0}$ $\triangleright$ _Set initial fused model to best
finetuned model $\theta_{0}$ _

$\tau\leftarrow 0.998$ $\triangleright$ _Tolerance factor for promising
candidate models_

for $i=1,...,n-1$ do $\triangleright$ _Iterative over all candidate models
$\theta_{i}$_

$\tilde{\Psi}=\frac{k}{k+1}\cdot\Psi+\frac{1}{k+1}\cdot\theta_{i}$
$\triangleright$ _Generate "approximate average" model_

if $ValAcc(\tilde{\Psi})>\tau\cdot ValAcc(\Psi)$ then $\triangleright$ _Do
optimization only if candidate is promising_

$\Psi^{\prime}(\lambda)^{j}=\lambda^{j}\cdot\Psi^{j}+(1-\lambda^{j})\cdot\theta^{j}_{i}$
$\triangleright$ _Define updated fused model for components $j=1,...,m$ _

$\lambda^{*}=\operatorname*{arg\,max}_{\lambda\in[0,1]^{m}}\left(ValAcc\left(\Psi^{\prime}\left(\lambda\right)\right)\right)$
$\triangleright$ _Calculate optimal mixing factors_

${\Psi^{\prime}}^{*}=\Psi^{\prime}(\lambda^{*})$ $\triangleright$ _Calculate
optimal updated fused model_

if $ValAcc({\Psi^{\prime}}^{*})>ValAcc(\Psi$) then

$k\leftarrow k+1$

$\Psi\leftarrow{\Psi^{\prime}}^{*}$ $\triangleright$ _Mix candidate model
$\theta_{i}$ into current fused model _

end if

end if

end for

return $\Psi$ $\triangleright$ _Return final fused model_

##  4 Experiments and Evaluation

The setup for our experiments is very similar to the one for the vision models
given in the _model soup_ paper [Wortsman et al., 2022a]. We summarize it in
the following for clarity and completeness.

The model employed for finetuning is the _CLIP_ model [Radford et al., 2021].
CLIP is a powerful multi-modal zero-shot neural network, which has been
pretrained with contrastive learning on a huge dataset of image-text pairs.
Specifically, we use the _CLIP ViT-B/32_ variant specified in Table 20 of
[Radford et al., 2021] and provided in the _OpenCLIP_ package
222https://github.com/mlfoundations/open_clip. Finetuning of the pretrained
model is performed end-to-end (all parameters are modified), as it typically
leads to better performance than training only the final linear layer. Before
finetuning, the final layer is initialized with a linear probe as described in
[Kumar et al., 2022]. The loss function employed for finetuning is the cross-
entry loss.

The original dataset employed for finetuning is _ImageNet_ [Deng et al.,
2009]. Since the official ImageNet validation dataset is typically used as the
test dataset, we use roughly $2\%$ of the ImageNet training dataset as held-
out validation dataset for calculating the validation accuracy in our proposed
algorithm (see section 3 and the pseudocode provided in Algorithm 1).

For measuring the out-of-distribution performance (robustness to distribution
shifts) of our proposed algorithm, we employ five datasets derived from
ImageNet with natural (not synthetically generated) distribution shifts. They
corresponds to datasets with naturally occurring variations of the data
samples due to different lighting, viewpoint, geographic location, image style
(e.g. sketch instead of photo), crowdsourcing and more. The five datasets with
distribution shifts we use are:

  * •

ImageNet-V2 (IN-V2) [Recht et al., 2019] is a reproduction of the ImageNet
test set with distribution shift. The dataset was collected by closely
following the original labelling protocol.

  * •

ImageNet-R (IN-R) [Hendrycks et al., 2021a] contains renditions (e.g.,
sculptures, paintings) for 200 ImageNet classes.

  * •

ImageNet-Sketch (IN-Sketch) [Wang et al., 2019] contains sketches instead of
natural images. It contains only sketches in "black-and-white" color scheme.

  * •

ObjectNet [Barbu et al., 2019] provides objects in various scenes with 113
classes overlapping with ImageNet.

  * •

ImageNet-A (IN-A) [Hendrycks et al., 2021b] is a test set of natural images
misclassified by a ResNet-50 model for 200 ImageNet classes.

Figure 1: Samples for class _lemon_ , from the original ImageNet dataset and
the five datasets with natural distribution shifts. Image courtesy of
[Wortsman et al., 2022b]

.

See Figure 1 for an illustration of samples for each of the datasets with
natural distribution shifts. For all datasets (the original used for
finetuning and the ones with distribution shifts), we take the top-1 accuracy
on the respective test set for measuring the performance of a model. We
calculate the overall out-of-distribution performance of a model as the
average of its test accuracy over all five datasets with distribution shifts.

We partition the CLIP ViT-B/32 model into 8, 15 and 26 components. A too fine
partitioning (e.g. one component for each layer of the model) makes the
optimization much more difficult, whereas a too coarse partioning provides not
enough flexibility for mixing the latent space manifolds individually in an
optimal way. The structure of the partitioning is done roughly according to
the hierarchy of the building blocks of the CLIP model. We denote the
respective variant of our proposed algorithm with 8, 15 and 26 components as
ManifoldMixMS-C8, ManifoldMixMS-C15 and ManifoldMixMS-C26.

We parametrize the Nevergrad optimizer with a maximum budget for the number of
function evaluations (of the objective function to optimize) of roughly $250$
function evaluations for all ManifoldMixMS variants. The employed optimizer is
automatically selected by the Nevergrad optimization package (see [Liu et al.,
2020]). For our cases, it always selects the _Cobyla_ [Powell, 1994]
optimization algorithm. The Cobyla algorithm is one of the best derivative-
free algorithms for optimization of continuous variables with bound
constraints, especially when the allowed number of function evaluations is
quite small.

For the evaluation of our proposed manifold mixing model soup algorithm, we
compare mainly with the _greedy soup_ and _uniform soup_ algorithms which have
been proposed in [Wortsman et al., 2022a]. Additionally, we compare our
proposed algorithm also against ensemble models. We compare against the same
ensemble models as done in [Wortsman et al., 2022a] and take also the accuracy
numbers reported there for them. Of course, one should take into account that
the computational cost for inference of an ensemble model is much higher – $K$
times higher for an ensemble model consisting of $K$ individual models – than
for our proposed ManifoldMixMS algorithm which produces only a single fused
model.

The scatterplot in Figure 2 shows how our proposed ManifoldMixMS-C8 algorithm
(the overall best variant) performs compared to the greedy soup and uniform
soup algorithm from [Wortsman et al., 2022a] and to the individual finetuned
models.

Furthermore, Table 1 gives a detailed evaluation of our proposed variants of
the manifold mixing soup algorithm with 8, 15 and 26 on the five datasets with
distribution shifts (ImageNet-V2, ImageNet-R, ImageNet-Sketch, ObjectNet,
ImageNet-A) as well as on the original dataset used for finetuning (ImageNet).

Figure 2: Comparison of our proposed manifold mixing soup algorithm (with 8
components) against greedy soup and uniform soup algorithm from [Wortsman et
al., 2022a] and the individual finetuned models.

One can see clearly from the scatterplot that our proposed manifold mixing
model soup (especially the preferred variant with 8 components) algorithm
combines the best properties of the uniform model soup and greedy soup
algorithm. Specifically, it has practically the same good out-of-distribution
accuracy as the uniform soup algorithm and still keeps the good accuracy of
the greedy soup algorithm on the original ImageNet dataset. In contrast, the
uniform soup algorithm performs on the original ImageNet dataset even worse
than the best individual finetuned model.

It is significantly better with respect to the best finetuned model both on
the datasets with distribution shifts ($+3.5\%$), but also on the original
ImageNet dataset ($+0.6\%$). The difference grows even bigger when comparing
with the second-best finetuned model.

Surprisingly, it has also a significantly better out-of-distribution accuracy
than both Ensemble methods, although its accuracy on the original ImageNet
dataset is worse especially when compared with the greedy ensemble method. As
already mentioned, one should take into account that Ensemble methods have a
much higher computational cost.

Table 1: Detailed comparison of our proposed manifold mixing soup algorithm variants for the CLIP ViT-B/32 neural network with the best and second-best finetuned model, the model soup algorithms from [Wortsman et al., 2022a] and for completeness also with Ensemble methods.The top-1 accuracy (in %) on the respective test dataset is employed. The column "Avg OOD" corresponds to the average over all 5 datasets with distribution shifts. The best and second-best result for each dataset (without taking into account the Ensemble methods as they have a much higher computational cost) is marked in red and blue.  Method | ImageNet | IN-V2 | IN-R | IN-Sketch | ObjectNet | IN-A | Avg OOD
---|---|---|---|---|---|---|---
Best finetuned model | 80.38 | 68.44 | 44.51 | 60.63 | 42.62 | 23.64 | 47.97
Second-best finetuned model | 79.89 | 67.91 | 41.49 | 54.58 | 37.98 | 18.01 | 44.01
Uniform soup | 79.97 | 68.51 | 47.71 | 66.54 | 45.95 | 29.17 | 51.57
Greedy soup | 81.03 | 69.55 | 47.77 | 64.20 | 44.90 | 27.89 | 50.86
ManifoldMixMS-C8 | 80.95 | 69.67 | 48.15 | 64.81 | 45.66 | 29.06 | 51.47
ManifoldMixMS-C15 | 80.80 | 69.61 | 47.89 | 64.76 | 44.45 | 28.39 | 51.02
ManifoldMixMS-C26 | 80.85 | 69.58 | 48.04 | 64.79 | 45.75 | 28.88 | 51.41
Ensemble | 81.19 | – | – | – | – | – | 50.77
Greedy ensemble | 81.90 | – | – | – | – | – | 49.44

##  5 Conclusion

We propose the _manifold mixing model soup_ algorithm, which mixes together
the latent space manifolds of multiple finetuned models in an optimal way in
order to generate a fused model. Experiments show that the fused model gives
significantly better out-of-distribution performance (+3.5 % compared to best
finetuned model) when finetuning a CLIP model for image classification.

In the future, we plan to evaluate the proposed algorithm on other neural
network architectures, for both computer vision as well as natural language
processing tasks. Furthermore, we plan to do a theoretical analysis of the
properties of the proposed algorithm in order to get a better insight why it
provides a better out-of-distribution performance.

## Acknowledgment

This work was supported by European Union´s Horizon 2020 research and
innovation programme under grant number 951911 - AI4Media.

## References

  * [Barbu et al., 2019] Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. (2019).  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  In Advances in Neural Information Processing Systems (NeurIPS).
  * [Deng et al., 2009] Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. (2009).  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. IEEE Computer Society.
  * [Hendrycks et al., 2021a] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. (2021a).  The many faces of robustness: A critical analysis of out-of-distribution generalization.  International Conference on Computer Vision (ICCV).
  * [Hendrycks et al., 2021b] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. (2021b).  Natural adversarial examples.  Conference on Computer Vision and Pattern Recognition (CVPR).
  * [Izmailov et al., 2018] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D. P., and Wilson, A. G. (2018).  Averaging weights leads to wider optima and better generalization.  In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 876–885. AUAI Press.
  * [Jolicoeur-Martineau et al., 2023] Jolicoeur-Martineau, A., Gervais, E., Fatras, K., Zhang, Y., and Lacoste-Julien, S. (2023).  Population parameter averaging (PAPA).  CoRR, abs/2304.03094.
  * [Kumar et al., 2022] Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. (2022).  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  * [Liu et al., 2020] Liu, J., Moreau, A., Preuss, M., Rapin, J., Rozière, B., Teytaud, F., and Teytaud, O. (2020).  Versatile black-box optimization.  In GECCO ’20: Genetic and Evolutionary Computation Conference, Cancún Mexico, July 8-12, 2020, pages 620–628. ACM.
  * [Matena and Raffel, 2022] Matena, M. and Raffel, C. (2022).  Merging models with fisher-weighted averaging.  In NeurIPS.
  * [Powell, 1994] Powell, M. J. D. (1994).  A direct search optimization method that models the objective and constraint functions by linear interpolation.  In Advances in Optimization and Numerical Analysis.
  * [Radford et al., 2021] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021).  Learning transferable visual models from natural language supervision.  In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139, pages 8748–8763. PMLR.
  * [Recht et al., 2019] Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019).  Do ImageNet classifiers generalize to ImageNet?  In International Conference on Machine Learning (ICML).  https://arxiv.org/abs/1902.10811.
  * [von Oswald et al., 2021] von Oswald, J., Kobayashi, S., Sacramento, J., Meulemans, A., Henning, C., and Grewe, B. F. (2021).  Neural networks with late-phase weights.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  * [Wang et al., 2019] Wang, H., Ge, S., Lipton, Z., and Xing, E. P. (2019).  Learning robust global representations by penalizing local predictive power.  In Advances in Neural Information Processing Systems (NeurIPS).  https://arxiv.org/abs/1905.13549.
  * [Wortsman et al., 2022a] Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Lopes, R. G., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. (2022a).  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA.
  * [Wortsman et al., 2022b] Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. (2022b).  Robust fine-tuning of zero-shot models.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR.
  * [Yu et al., 2022] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. (2022).  Coca: Contrastive captioners are image-text foundation models.  Trans. Mach. Learn. Res., 2022.