# Instant Soup: Cheap Pruning Ensembles in A Single Pass Can
Draw Lottery Tickets from Large Models

Ajay Jaiswal  Shiwei Liu  Tianlong Chen  Ying Ding  Zhangyang Wang

###### Abstract

Large pre-trained transformers have been receiving explosive attention in the
past few years, due to their wide adaptability for numerous downstream
applications via fine-tuning, but their exponentially increasing parameter
counts are becoming a primary hurdle to even just fine-tune them without
industry-standard hardware. Recently, Lottery Ticket Hypothesis (LTH) and its
variants, have been exploited to prune these large pre-trained models
generating subnetworks that can achieve similar performance as their dense
counterparts, but LTH pragmatism is enormously inhibited by repetitive full
training and pruning routine of iterative magnitude pruning (IMP) which
worsens with increasing model size. Motivated by the recent observations of
model soups, which suggest that fine-tuned weights of multiple models can be
merged to a better minima, we propose Instant Soup Pruning (ISP) to generate
lottery ticket quality subnetworks, using a fraction of the original IMP cost
by replacing the expensive intermediate pruning stages of IMP with
computationally efficient weak mask generation and aggregation routine. More
specifically, during the mask generation stage, ISP takes a small handful of
iterations using varying training protocols and data subsets to generate many
weak and noisy subnetworks, and superpose them to average out the noise
creating a high-quality denoised subnetwork. Our extensive experiments and
ablation on two popular large-scale pre-trained models: CLIP (unexplored in
pruning till date) and BERT across multiple benchmark vision {MNIST, SVHN,
Cars, GTSRB, CIFAR-10, CIFAR-100} and language datasets {MNLI, QNLI, QQP, SST,
...} validate the effectiveness of ISP compared to several state-of-the-art
pruning methods. Additionally, we show that ISP can be easily modified with
minimal overhead to produce benefits comparable to model soups, without the
prerequisite to generate multiple candidates fine-tuned models. Codes are
available at: https://github.com/VITA-Group/instant_soup.

Machine Learning, ICML

Figure 1: Fine-tuning Accuracy ($\uparrow$) and FLOPs counts ($\downarrow$) of
subnetworks with sparsity $s\in\\{10\%,20\%,...,80\%\\}$ on CIFAR-10 test set
with a pre-trained CLIP (ViT-B32) model checkpoint. Note that ISP requires
approximately single-pass computational cost of IMP to generate subnetworks
with better performance than multiple rounds of IMP as required in Lottery
Tickets.

##  1 Introduction

Large-scale transfer learning has recently become show-stealer in modern deep
learning, and transformer-based pre-trained models (Devlin et al., 2018; Liu
et al., 2019; Dosovitskiy et al., 2020; Liu et al., 2021; Radford et al.,
2021) are now achieving state-of-the-art performance for a wide array of real-
world computer vision (Dosovitskiy et al., 2020; Han et al., 2020; Li et al.,
2023; Touvron et al., 2021; Mao et al., 2022; Jaiswal et al., 2021a; Zheng et
al., 2021; Parmar et al., 2018) and natural language processing (Yang et al.,
2019b; Liu et al., 2019; Talmor et al., 2018; Jaiswal et al., 2021b; Zheng et
al., 2023; Yang et al., 2019a; Wang et al., 2018; Ding et al., 2019; Chowdhery
et al., 2022; Wei et al., 2022; Jaiswal et al., 2023) applications. With the
astonishing explosion of parameter counts (millions to billions) in the past
few years, while chasing performance gains, fine-tuning these large pre-
trained models with non-industry standard hardware is becoming seemingly
impossible, in addition to expensive inference and steep environmental cost.
In the hustle of building gigantic models, a parallel and growing field of
model compression has been exploring the prospects to compress these enormous
models at the cost of marginal/no sacrifice in performance, effectively
reducing their computational and memory footprints.

Figure 2: Overview of our proposed Instant Soup Pruning. We provide a detailed
illustration of our proposed technique compared to conventional LTH with IMP.
ISP replaces the expensive intermediate pruning stages of IMP with
computationally-cheap weak mask generation and denoising while outperforming
LTH. Unlike LTH, ISP consumes computation budget equivalent of a single pass
of LTH.

Among many efforts for compressing models and accelerating inference (Frankle
& Carbin, 2018; Chen et al., 2020a; Jaiswal et al., 2022b; Yin et al., 2022;
Jaiswal et al., 2022c; Lee et al., 2018; Yu et al., 2017, 2020; Fang et al.,
2023; Chen et al., 2023; Jaiswal et al., 2022a; Liu et al., 2023), network
pruning eliminates unnecessary weights to generate smaller subnetworks in
place of dense networks attaining similar performance, stands out as one of
the most effective techniques. Lottery Ticket Hypothesis (LTH) (Frankle &
Carbin, 2018) and its variants, reveal that dense, randomly-initialized
networks contain small subnetworks which can match the test accuracy of
original networks. Despite their insightful findings, there still exists a
large gap in the practicality of these methods because of the fully dense
training routine of IMP, which exacerbates with an increase in model capacity.
Recently proposed EarlyBird routine (You et al., 2019; Chen et al., 2020b)
which attempts to draw the winning tickets early in training, and pruning at
initialization techniques have shown some promise in mitigating search through
an expensive and tedious iterative process, yet their effectiveness for the
large-scale pre-trained network is highly under-explored or they tend to have
sub-standard performance at non-trivial sparsities. In this work, we ask: Does
there exist a principled and cheaper approach for fastly drawing high-quality
lottery tickets in large pre-trained models within a limited computational
budget, while preserving its performance and transferability?

To this end, we explore the feasibility of obtaining cheap tickets for popular
large pre-trained models CLIP(Radford et al., 2021) and BERT(Devlin et al.,
2018) constrained by no multi-pass repetitive full-training, to meet the
permissible computational budget. One straightforward approach is to curtail
per round training cost of iterative magnitude pruning (IMP) with early
stopping, but we found that such non-careful pruning approach often produces
highly variable and substandard tickets, presumably due noisy and unstable
state of the network, when pruned. Our close analysis of the pruned mask
generated by LTH and cheap pruning methods (one-shot pruning) found that
large-scale pre-trained transformers are highly overparameterized and pruning
them at trivial sparsities does not necessarily require an expensive LTH
paradigm, which encourages us to start pruning nonchalantly at trivial
sparsities and become scrupulous at non-trivial sparsities. In addition,
recently several works (Wortsman et al., 2022a; Ilharco et al., 2022; Juneja
et al., 2022) investigate the intriguing phenomenon of “model soups”, and have
shown that weights of multiple dense fine-tuned models can be merged together
into better solutions lying in low error basins. In the sparse setting, a very
recent attempt (Yin et al., 2022) reused the byproduct of IMP, and showed that
tickets generated at each iteration of IMP could be superposed into a stronger
subnetwork. However, its observations are limited to small-scale networks, and
more importantly, the algorithm does not contribute to the computational
efficiency of either finding lottery tickets or (re-)training networks.
Motivated by these observations, we are interested to investigate: if we can
leverage the soup observations to eliminate noise induced by early pruning in
IMP iteration, effectively leading to the stable sparse subnetwork and reduced
cost.

We propose Instant Soup Pruning (ISP), a model soup-inspired perspective
dedicated to generating lottery ticket quality subnetworks, using a fraction
of the original IMP cost. More specifically, ISP uses a miniature random
subset of training data to generate many weak and noisy subnetworks, and
superpose them to average out the noise creating a high-quality denoised
subnetwork. Similar to traditional IMP, ISP repeats the denoising routine
following well-managed training iterations till the desired sparsity is
reached, eliminating the need of IMP to perform a full pass of training before
every pruning routine. Our experiments on CLIP (unexplored in pruning
literature till date) and BERT across multiple datasets illustrate that ISP
can find sparse subnetworks with better quality than LTH, with an affordable
cost no more than a single pass of IMP. In addition to dense-to-sparse
paradigm, interestingly, our ISP routine can be bluntly incorporated in dense-
to-dense paradigm to incorporate the benefits of model soups in dense pre-
trained models at almost negligible training cost at the pre-training stage,
ultimately leading to better-fine tuning performance. Our contributions can be
summarized as:

  * $\blacksquare$

We propose Instant Soup Pruning (ISP), a novel pruning strategy that
seamlessly integrates the “model soup” idea to significantly reduce the
computational cost of IMP. ISP replaces the expensive intermediate pruning
stages of IMP with computationally-cheap weak mask generation and denoising,
while outperforming IMP for large pre-trained models.

  * $\blacksquare$

ISP naturally provides a “self-denoising” ability to eliminate the necessity
of generating high quality/expensive masks (presumably “good solution basin”)
at each pruning stage, by instead generating multiple computationally
inexpensive weak masks and averaging them out to reduce their solution noise.

  * $\blacksquare$

In the dense-to-dense paradigm, ISP can be adapted to Instant Model Soup, to
inject the benefits of model soups in dense pre-trained models at marginal
training cost, thereby improving fine-tuning performance comparable to model
soups without the need to generate multiple fine-tuned models.

  * $\blacksquare$

Our extensive experiments on two popular large-scale pre-trained models (CLIP
& BERT) across multiple benchmark vision {MNIST, SVHN, Cars, GTSRB, CIFAR-10,
CIFAR-100} & language datasets {MNLI, QQP, STS-B, WNLI, QNLI, MPRC, RTS,
SST-2, CoLA} validate the effectiveness of ISP wrt. several SOTA pruning
methods.

##  2 Methodology

###  2.1 Revisiting LTH and Pre-trained Vision and Language Model Compression

In the past few years, scaling neural networks to improve information
absorption has been pivotal for good optimization and generalization
performance, but this unbounded parameter growth has made them computationally
expensive with excessive memory requirements. The trend undoubtedly continues
with the recent forefront of transformers, where more and more layers are
stacked with dense attention blocks (eg. T5 has $\sim 10+$ billion parameters)
calling for expensive computational resources and prolonged training or fine-
tuning time. Recently, some work (Chen et al., 2021b; Gan et al., 2022; Chen
et al., 2021a, 2020b; You et al., 2019; Prasanna et al., 2020) have explored
Lottery Ticket Hypothesis (LTH) to understand the parameter redundancy in the
current prevailing large scale transformer models, and attempted to compress
it to non-trivial sparsities by repetitive initialization-training-pruning
operation. Despite their success to find high-quality compressed models on a
range of downstream tasks, it is impossible to ignore the cost of finding
these subnetworks, since winning tickets can only be identified by pruning
unimportant connections after fully training a dense network in a conventional
LTH paradigm, which worsens significantly with increasing model size. For
example, (Prasanna et al., 2020; Chen et al., 2020a) explored pruning BERT to
matching subnetworks at 40% to 90% sparsity across multiple tasks, on the
other hand, (Gan et al., 2022) explored LTH for compressing large pre-trained
VL models while preserving its performance.

Figure 3: Performance comparison of LTH and One-shot magnitude-based pruning
of CLIP (ViT-B32) on CIFAR-10 at sparsity $S\in\\{10\%,20\%,...,90\%\\}$
(left). Cosine similarity between the binary prune masks obtained by LTH and
One-shot magnitude pruning of CLIP (ViT-B32) on CIFAR-10 (right).

Recently, (Chen et al., 2020b) explored the EarlyBird (You et al., 2019) idea
and proposed jointly training BERT and some sparsity-inducing coefficients
which can be used to draw the subnetworks followed by fine-tuning, but its
joint training step again is as expensive as normal BERT training, and
experimentally we found that its performance becomes sub-standard compared to
LTH in non-trivial sparsity range. In this work, we propose a novel pruning
strategy based on strong insights of the inherent benefits of gigantic size
and model soups, which can be equivalent to or even better than LTH and its
variant LTH-Rewind. We, for the first time, explore model compression for a
recent extremely popular open vocabulary network CLIP along with BERT, to
illustrate our approach benefits by using merely the computational cost equal
to a single pass of conventional LTH.

###  2.2 Instant Soup Pruning: A novel cost-effective pruning perspective

In this section, we introduce a novel pruning algorithm, named Instant Soup
Pruning (ISP) which primarily aims to reduce the computational overhead of
conventional LTH while searching for lottery tickets in large-scale pre-
trained transformers, facilitating benefits from both performance and
computation perspective.

ISP is motivated from the following three observations:

  * •

Firstly, large-scale pre-trained transformers are highly over-parameterized,
and pruning them at trivial (eg. 10%, 20%, etc. depending on task and model
size) sparsities does not require sophisticated pruning methods like LTH or
LTH-Rewind to get high-quality sparse subnetworks. We surprisingly observed
that at trivial sparsities, the sparse mask generated by LTH and cheap one-
shot magnitude pruning is significantly similar which thereby reflects in the
test performance of the subnetworks. For example, Figure 3(a) illustrates the
performance of subnetworks obtained by LTH and one-shot pruning on CLIP
(ViT-B32) and Figure 3(b) illustrates the cosine similarity between the binary
prune mask identified by LTH and one-shot pruning. It can be clearly observed
that at trivial sparsities such as 10%, 20%, and 30%, both unreasonably cheap
one-shot pruning and expensive LTH identify approximately similar masks with
96.27%, 98.75%, and 94.32% cosine similarity score. It conveys a strong
message to save the computation budget of full training passes of LTH, which
are seemingly unnecessary.

  * •

Secondly, Early-Bird (You et al., 2019) tickets, although limited to small
architectures (ResNets, Vgg16, etc), conveyed a strong yet highly overlooked
message that high-quality tickets can emerge at a very early training stage by
pruning networks trained at much earlier points (before the accuracies reach
their final top values). Recently, (Chen et al., 2020b) showed that this
observation holds true for BERT, but it cannot recover the full performance of
LTH. To this end, complementary to our first observation, our work extends the
early-bird findings by proposing to look more carefully while searching
tickets at non-trivial sparsity (high sparsity regime) compared to trivial
sparsity by progressively increasing training steps over each call to pruning
routine in the mask finding stage.

  * •

Lastly, to mitigate the issue of pre-mature pruning to generate sub-standard
pruning masks, we borrow inspiration from the intriguing phenomenon of “model
soups”, which illustrate weights of large-scale independently fine-tuned
models can be merged together into a better solution. Our work proposes a
novel approach of mask soups by superposing multiple multiple cheap pruning
masks, to attenuate the noise within them due to the presumably unstable state
of the network while pruning, giving a high-quality denoised pruning mask.

##### Algorithm Overview

Consider a dense, pre-trained network $f(x;\theta)$, as shown in Figure 2, LTH
trains $f$ to achieve minimum validation loss $f_{loss}$ using $E$ epochs with
a test accuracy $f_{acc}$, when optimized with Adam optimizer on a training
dataset $D$. Once, $f_{acc}$ is achieved, the fine-tuned network
$f(x;\theta_{E})$ is pruned using magnitude pruning to generate a subnetwork
$f(x;m\odot\theta_{E})$, with a mask $m\in\\{0,1\\}$. This process is repeated
for $k$ iterations till the desired sparsity $S\%$ is achieved, generating a
subnetwork $f_{LTH}(x;m\odot\theta_{\mathcal{O}(k\cdot E)})$ with accuracy
$f^{LTH}_{acc}$.

In contrast, our proposed approach ISP aims to generate
$f_{ISP}(x;m^{\prime}\odot\theta_{\mathcal{O}(E)})$ with accuracy
$f^{ISP}_{acc}$, such that $f^{ISP}_{acc}\geq f^{LTH}_{acc}$ with sparse mask
$m^{\prime}\in\\{0,1\\}$ and sparsity $S\%$. Given the computational budget of
$E$ epochs, which translates to $T$ steps with batch size $B$, ISP is composed
of two distinct phases: mask generation phase and fine-tuning phase which uses
$M$ and $(T-M)$ steps respectively to produce a high-quality fine-tuned
subnetwork with desired sparsity $S\%$, usually outperforming LTH.

Input :

Pre-trained Network: $f(x;\theta_{0})$; Initial Training Seed: $t$;
Compression ratio: $s$%; Desired sparsity: $S\%$; Training Budget: $T$

Output :

Pruned trained subnetwork with a mask $m^{\prime}\in\\{0,1\\}$:
$f_{ISP}(x;m^{\prime}\odot\theta_{\mathcal{O}(E)})$

/* $k$ is chosen st. $\sum_{i=0}^{k}(i+1)\cdot t\leq M$ */

for  _$i\leftarrow 0$ to $k$_ do

/* Sparsity dependent training steps before call to pruning routine. Spend low
training steps for trivial sparsity compared to non-trivial sparsity. */ Train
$f(x;\theta_{i\cdot t})\rightarrow f(x;\theta_{(i+1)\cdot t})$ if
_$\textnormal{{sparsity}}(f) <S\%$_ then

$m_{(i+1)}\leftarrow\textnormal{{DenosiedPrune}}(f,s,m_{i})$ /* Compress
network with newly returned denoised mask */ Apply $m_{(i+1)}\rightarrow
f(x;\theta_{(i+1)\cdot t})$

end if

end for

/* Fine-tune the obtained subnetwork for the remaining $(T-((k+1)\cdot t))$
steps in computational budget. */ Fine-tune $f(x;\theta_{(k+1)\cdot
t})\rightarrow f(x;\theta_{T-((k+1)\cdot t)})$

Algorithm 1 Instant Soup Pruning

####  2.2.1 Mask Generation Stage

We will first discuss our novel, computationally efficient, and high-quality
sparse mask generation steps for ISP incorporating the aforementioned
motivation. Note that similar to IMP, ISP is also an iterative train-prune-
retrain procedure, but it uses a highly optimized train/re-train subroutine
starting from the parameter state obtained from the previous iteration. Given
the computational budget of $M$ steps, the mask generation stage of ISP is
carefully designed to start in a relaxed fashion (spending few learning steps)
while pruning at trivial sparsities and gradually become meticulous while
approaching non-trivial sparsity regime, acknowledging the sensitivity of
pruning while operating in high sparsities.

More specifically, we introduce a hyperparameter $t$, which is defined as a
small initial seed step count usually equates to the number of steps required
to look $\sim 10\%$ of training data with batch size $B$, before the first
call to pruning routine. At $i$-th call to the pruning routine, we use
$t\times(i+1)$ training steps for calibrating (training) the network before
pruning. For example, at the $0$-th iteration when the network is dense, ISP
uses steps equivalent to 10% of training data while at $5$-th iteration, it
uses 60% of training data in the re-training.

To ensure that ISP does not produce sub-standard quality mask at each pruning
iteration, we next provide details of our novel denoised pruning routine which
is inspired by recently proposed “model soups” phenomenon, to average out
noise induced due to pre-mature pruning.

Input :

Network to prune: $f(x;m_{i}\odot\theta)$ where $m_{i}$ denotes binary mask at
$i$-th iteration of ISP; Denoiser Count: $N$; Compression Rate: $s$%; Look-
ahead steps: $C$ (typically $10\leq C\leq 100$ steps)

Output :

New denoised mask: $m_{(i+1)}$

$m_{temp}\leftarrow\textnormal{{MagnitudePruning}}(f(x;\theta),s\%)$

for  _$n\leftarrow 0$ to $N$_ do

/* Look-ahead training to generate multiple masks for denoising */
$P_{n}\leftarrow$ Training protocol // learning rate, weight decay, etc.
$D_{n}\leftarrow$ Random data samples Look-ahead training:
$f(x;\theta)\rightarrow f(x;\theta_{C})$ /* Denosing to average mask noise */
$m_{temp}\leftarrow
m_{temp}\cup\textnormal{{MagnitudePruning}}(f(x;\theta_{C}),s\%)$

end for

$m_{(i+1)}\leftarrow\texttt{One-shot-adjustment}(m_{temp},f(x;\theta))$ return
_$m_{(i+1)}$_

Algorithm 2 Denoised Pruning Procedure

##### Denoised Pruning:

Recently, several works (Wortsman et al., 2022a; Ilharco et al., 2022; Juneja
et al., 2022) have validated the intriguing phenomenon of model soups, which
suggest training multiple models with various hyperparameters and average the
weights of models fine-tuned independently at no cost to achieve comparatively
high performance. Motivated by their argument that fine-tuned models optimized
independently from the same pre-trained initialization lies in the same basin
of the error landscape and averaging them improves generalization, we are
tempted to ask an unexplored question: Can we generate extremely cheap pruning
masks using varying hyperparameters and average them out to improve the
quality?

To this end, we propose a novel Denoised Pruning Procedure, which explores the
superimposition of computationally cheap pruning mask obtained by magnitude-
based one-shot pruning of the network with marginal look-ahead training.
Algorithm 2 illustrates the details of our denoising procedure which perform
look-ahead training of the network using varying training protocol for a
random subset of training samples, merely using $10-100$ training steps, to
generate $N$ candidate binary masks. In order to superimpose them, we found
that a simple union of these $N$ binary masks is sufficient to improve the
quality, facilitating ISP to achieve comparable/even better performance than
expensive LTH. We hypothesize (and later experimentally validate) that our
denoising procedure significantly helps eliminate any induced noise due to
pre-mature pruning during ISP iterations, specifically at non-trivial
sparsities.

####  2.2.2 Fine-tuning Stage

Our proposed method (ISP), is a single pass pruning algorithm that generates a
high-quality fine-tuned subnetwork with desired sparsity $S\%$ using the
computational budget equivalent to a single pass of LTH ($T$ steps). As
mentioned before, ISP is composed of the mask generation phase ($M$ steps)
followed by fine-tuning the obtained subnetwork for the remaining ($T-M$
steps). Note that, unlike conventional LTH, ISP does not restart the network
training from the initial pre-trained weight; instead, as explained in
Algorithm 1, it simply fine-tunes the network state (unchanged optimizer,
learning rate, etc.) obtained immediately after pruning $S\%$ of parameters
for $(T-M)$ steps.

###  2.3 Instant Model Soup: A sparsity-inspired extension to dense training

In the conventional setting, to improve the generalization performance of a
model, it is highly recommended to train multiple models and use their
ensemble on the test set, but it comes at a heavy inference and training cost.
Recently, several works (Wortsman et al., 2022a; Ilharco et al., 2022; Juneja
et al., 2022) have illustrated that, unlike conventional ensembles, the
weights of multiple fine-tuned large pre-trained models can be merged together
by simply averaging their weights (aka. model soup) to beat the performance of
ensembles without incurring any additional inference or memory cost. While
this approach effectively reduces the additional inference overhead of
ensembles, it still requires expensive fine-tuning of multiple large pre-
trained models with varying hyperparameters. Inspired by our idea of Instant
Soup Pruning, which illustrates that binary masks originated from cheap
training with different training protocols can be superimposed/denoised to
improve quality, we are enticed to explore: Can we denoise the initial pre-
trained weights of dense large transformers using sparse cheap training to
inject the model soup benefits early during fine-tuning, eliminating the need
to generate multiple fully fine-tuned models for model soups?

Input :

Pre-trained Model: $f(x;\theta_{0})$; $D_{Train}$; $D_{Val}$; Denosier Count:
$K$

Output :

New denoised Pre-trained Model: $f(x;\theta_{new})$

$f(x;\theta_{interpolated})\leftarrow\textnormal{{DeepCopy}}(f(x;\theta_{0}))$

for  _$k\leftarrow 0$ to $K$_ do

/* Prune to create a sparse subnetwork from input model */
$f(x;m_{s}\odot\theta_{0})\leftarrow\textnormal{{MagnitudePruning}}(f(x;\theta_{0}),s\%)$
/* Weakly train sparse subnetwork for denoising */ $H_{k},C_{k}\leftarrow$
Training Protocol, Steps $D_{k}\leftarrow\textnormal{{sample}}(D_{Train})$
$f_{weak}(x;m_{s}\odot\theta_{k})\leftarrow\textnormal{{Train}}(f(x;m_{s}\odot\theta_{0}),H_{k},C_{k},D_{k})$
/* Denoising using linear interpolation with val data */
$f(x;\theta_{interpolated})\leftarrow\textnormal{{Interpolate}}(f_{weak}(x;m_{s}\odot\theta_{k}),f(x;\theta_{interpolated}),D_{Val})$

end for

$f(x;\theta_{new})\leftarrow f(x;\theta_{interpolated})$ return
_$f(x;\theta_{new})$_

Algorithm 3 Instant Model Soup

Our primary objective is to ripe the benefits of model soups without the need
to generate multiple fully fine-tuned models, thereby contenting the hurdle of
high training overhead of model soups. Algorithm 3 provide details of the
pseudocode of our new Instant Model Soup (IMS) approach that uses cheap sparse
training for injecting model soup benefits at marginal cost early before fine-
tuning to save the hurdle of multiple finetuning and averaging. More
specifically, IMS first creates multiple subnetworks with varying sparsities
from the pre-trained dense model and train them independently for a few
iterations ($\sim$100 iterations) using different hyperparameter
configuration, and data subsets. Next, all the weakly trained subnetwork
weights are merged together with the initial pre-trained model weights using
linear interpolation following (Ilharco et al., 2022). Our proposed usage of
sparse subnetworks for denoising significantly reduces the computational cost
of the dense training step, thereby making it more efficient. Our experiments
on CLIP (ViT-B32) and BERT-BASE impart an interesting finding that pre-trained
models denoised by IMS can achieve performance comparable to model soups,
without expensive full fine-tuning and then averaging.

| Initial LR | Epochs | Compression Rate | Look Ahead ($C$) | Weight Decay | Denoiser
---|---|---|---|---|---|---
BERT${}_{\texttt{BASE}}$ | 2 $\times$ 10-5 | 10 | 10% | 30 iter | 0.0-AdamW | 4
CLIP${}_{\texttt{ViT-B32}}$ | 1 $\times$ 10-5 | 22 | 15% | 50 iter | 0.1-AdamW | 5

Table 1: Details of our primary hyperparameter configurations used in our
experiments across different evaluation datasets. Table 2: Details of fine-
tuning CLIP (ViT-B32) at varying sparsity levels using Instant Soup Pruning
following the settings listed in Table 1. Learning rate decays linearly from
the initial value to zero. The evaluation metrics follow standards in (Radford
et al., 2021). Entries with errors are the average across three runs, and
errors are the standard deviations. LTH results are obtained using IMP.

Pruning Method | Cars | MNIST | SVHN | GTSRB | CIFAR10 | CIFAR100
---|---|---|---|---|---|---
30% | 40% | 50% | 70% | 80% | 90% | 70% | 80% | 90% | 50% | 60% | 70% | 60% | 70% | 80% | 60% | 70% | 80%
Full CLIP${}_{\texttt{ViT-B32}}$ | 76.43 $\pm$ 0.5 | 99.61 $\pm$ 0.07 | 97.40 $\pm$ 0.11 | 99.08 $\pm$ 0.24 | 97.6 $\pm$ 0.24 | 89.35 $\pm$ 0.19
Random | 9.11 | 5.58 | 4.47 | 98.71 | 97.42 | 87.04 | 89.85 | 85.61 | 73.74 | 93.76 | 93.65 | 90.97 | 74.51 | 69.84 | 64.96 | 45.20 | 39.92 | 43.24
One-shot [Mag] | 71.95 | 68.07 | 56.79 | 99.27 | 98.87 | 97.47 | 95.02 | 91.96 | 85.76 | 98.60 | 97.84 | 96.14 | 95.31 | 86.56 | 75.85 | 80.39 | 63.18 | 46.63
Progressive [Mag] | 69.85 | 68.62 | 64.43 | 99.52 | 97.77 | 95.19 | 95.78 | 90.53 | 85.75 | 98.97 | 97.57 | 96.34 | 95.25 | 90.87 | 78.11 | 81.79 | 70.53 | 60.93
EarlyBird (You et al., 2019) | 72.53 | 70.76 | 65.90 | 99.38 | 98.96 | 97.64 | 96.34 | 95.93 | 87.02 | 98.15 | 98.19 | 97.26 | 96.06 | 94.18 | 86.84 | 84.22 | 76.79 | 65.67
SNIP (Lee et al., 2018) | 71.51 | 68.79 | 59.01 | 99.25 | 98.72 | 97.50 | 95.33 | 91.94 | 82.98 | 98.62 | 97.95 | 96.22 | 95.01 | 87.45 | 76.12 | 81.10 | 62.89 | 55.89
GraSP (Wang et al., 2020) | 71.42 | 68.55 | 58.12 | 99.30 | 98.51 | 97.15 | 95.09 | 91.44 | 84.72 | 98.37 | 97.42 | 95.91 | 95.20 | 86.89 | 75.88 | 80.67 | 66.31 | 52.30
LTH (Frankle & Carbin, 2018) | 73.97 | 72.02 | 66.12 | 99.41 | 99.38 | 98.22 | 96.69 | 95.28 | 87.41 | 98.71 | 98.35 | 97.79 | 96.42 | 94.91 | 87.47 | 84.25 | 78.60 | 65.38
LTH - Rewind | 74.28 | 72.09 | 66.07 | 99.62 | 99.64 | 98.18 | 96.72 | 95.22 | 87.47 | 98.78 | 98.36 | 97.87 | 96.53 | 94.88 | 87.28 | 84.46 | 78.62 | 65.71
Lottery Pool (Yin et al., 2022) | 73.10 | 70.53 | 64.67 | 99.25 | 98.97 | 97.76 | 96.54 | 95.12 | 87.29 | 98.52 | 98.30 | 97.55 | 96.14 | 94.50 | 87.11 | 84.07 | 78.21 | 64.39
ISP [Ours] | 75.13 | 72.20 | 66.32 | 99.69 | 99.61 | 98.82 | 96.93 | 96.46 | 87.59 | 99.06 | 99.01 | 98.52 | 96.82 | 95.18 | 91.20 | 85.11 | 79.57 | 71.09
(std.) |  $\pm$0.34 |  $\pm$0.27 |  $\pm$0.82 |  $\pm$0.07 |  $\pm$0.15 |  $\pm$0.21 |  $\pm$0.08 |  $\pm$0.05 |  $\pm$0.11 |  $\pm$0.15 |  $\pm$0.29 |  $\pm$0.32 |  $\pm$0.15 |  $\pm$0.20 |  $\pm$0.14 |  $\pm$0.19 |  $\pm$0.22 |  $\pm$0.17

##  3 Experiments and Analysis

###  3.1 Network, Dataset, and Settings

In our experiments, we adopted the official CLIP implementation provided by
(Radford et al., 2021) as our starting point for our experiments using pre-
trained vision transformer (Dosovitskiy et al., 2020) (ViT-B32) models. For
fine-tuning CLIP, we use the frozen final classification layer output by
CLIP’s text tower to eliminate the necessity of introducing any learnable
parameters while in the case of BERT, we add a final task-specific
classification layer($\sim 3\%$ of total parameter count). For our BERT-
related experiments, we use the HuggingFace (Wolf et al., 2019) pre-trained
weights of BERT${}_{\texttt{BASE}}$ transformer blocks and hidden state size
768. Additional necessary details of our hyperparameters required for fine-
tuning are provided in Table 1. Note that during pruning, we prune the key
trainable part of the network (key, query, value, dense) ignoring embeddings
for simplicity. We consider a diverse set of image classification tasks from
(Radford et al., 2021): Cars, GTSRB, MNIST, SVHN, CIFAR10/100 and downstream
NLP tasks from GLUE (Wang et al., 2018) benchmark: MNLI, QQP, STS-B, WNLI,
QNLI, MPRC, RTS, SST-2, CoLA; to thoroughly investigate the effectiveness of
our proposed approaches wrt. state-of-the-art pruning benchmarks. In addition
to Lottery tickets, we have compared Instant Soup Pruning (ISP) against
several recently proposed pruning methods such as Lottery Pools (Yin et al.,
2022), Early bird (You et al., 2019), two popular pruning at initialization
methods (SNIP (Lee et al., 2018), GraSP (Yu et al., 2020)), Progressive
pruning (Iterative pruning and training), and one-shot magnitude pruning.

Table 3: Details of fine-tuning BERT (BASE) at varying sparsity levels using
Instant Soup Pruning following the settings listed in Table 1. Learning rate
decays linearly from the initial value to zero. The evaluation metrics follow
standards in (Wolf et al., 2019). Entries with errors are the average across
three runs, and errors are the standard deviations. LTH results are obtained
using IMP.

Dataset | MNLI | QQP | STS-B | WNLI | QNLI | MPRC | RTS | SST-2 | CoLA
---|---|---|---|---|---|---|---|---|---
Sparsity | 70% | 90% | 50% | 90% | 70% | 50% | 60% | 60% | 50%
Full BERT${}_{\texttt{BASE}}$ | 82.4 $\pm$ 0.5 | 90.2 $\pm$ 0.5 | 88.4 $\pm$ 0.3 | 54.9 $\pm$ 1.2 | 89.1 $\pm$ 1.0 | 85.2 $\pm$ 0.1 | 66.2 $\pm$ 3.6 | 92.1 $\pm$ 0.1 | 54.5 $\pm$ 0.4
Random | 67.5 | 76.3 | 21.0 | 53.5 | 61.9 | 69.6 | 56.0 | 83.1 | 9.6
One-shot | 78.8 | 86.2 | 83.9 | 53.1 | 86.2 | 83.7 | 62.9 | 86.5 | 49.7
Progressive | 79.1 | 87.5 | 85.0 | 53.3 | 87.2 | 83.8 | 65.4 | 86.6 | 52.2
EarlyBird | 82.5 | 89.4 | 88.1 | 54.0 | 88.5 | 84.6 | 66.1 | 91.2 | 53.5
Lottery Ticket | 82.6 | 90.0 | 88.2 | 54.9 | 88.9 | 84.9 | 65.0 | 91.9 | 53.8
Lottery Pool | 80.4 | 89.1 | 86.4 | 50.9 | 87.6 | 84.5 | 62.7 | 90.9 | 52.6
ISP | 82.71 $\pm$ 0.6 | 90.59 $\pm$ 0.5 | 88.64 $\pm$ 0.1 | 55.33 $\pm$ 0.3 | 90.06 $\pm$1.0 | 85.38 $\pm$ 0.1 | 65.96 $\pm$ 0.3 | 92.43 $\pm$ 0.6 | 53.61 $\pm$ 0.2

###  3.2 Performance comparison of Instant Soup Pruning wrt. SOTA pruning
methods

In this section, we conduct a systematic and extensive study to understand the
performance benefits of our proposed Instant Soup Pruning in terms of fine-
tuning accuracy vs. pruning ratios by comparing against multiple state-of-the-
art pruning methods. We first test our approach on recently proposed CLIP
(ViT-B32) (Radford et al., 2021) (unexplored for pruning till date) pre-
trained with contrastive supervision from image-text pairs. For effective
comparison and simplicity, all baselines and our proposed approach ISP are
trained with similar optimizer and training settings provided in Table 1. Our
EarlyBird (You et al., 2019), SNIP (Lee et al., 2018), and GraSP (Yu et al.,
2020) baselines closely follow the implementation provided by their official
GitHub repositories. In addition, to distinguish ISP from conventional
progressive pruning, our progressive pruning baseline is implemented as
periodic pruning using magnitude-based pruning followed by retraining. Lottery
pools (Yin et al., 2022) is an interesting way to merge LTH by-product
tickets. To ensure that the sparsity ratio remains comparable to ISP, we
further prune the merged tickets obtained by pooling to the required sparsity,
for reporting performance.

Our results for CLIP are summarized in Table 2. We first observe that among
all baselines for CLIP compression, LTH consistently performs better, and
rewinding helps in further improving its performance. We found that among
recent pruning at initialization methods (SNIP and GraSP), SNIP has
comparatively better performance than GraSP and they tend to be slightly
better than one-shot magnitude pruning. It can be clearly observed that ISP
can beat expensive LTH (including rewinding) as well as all other baselines
for almost all benchmark datasets and sparsity ratios. Very interestingly, for
CIFAR-10 and CIFAR-100, we found that performance improvement of ISP increases
with the sparsity ratio. For example, ISP surprisingly outperforms LTH-rewind
by $\sim 3.92$ and $\sim 5.38\%$ on CIFAR-10 and CIFAR-100 respectively, while
consuming merely fine-tuning cost equivalent to one single pass of LTH without
the necessity of bookkeeping the rewinding weights of LTH-rewind. Our
experimental results for BERT-base are summarized in Table 3. For BERT-related
experiments, we have replicated the setting in (Chen et al., 2020a) and
reported the performance of ISP across various GLUE benchmarking datasets at
the sparsity level where LTH is able to identify winning tickets. Note that
ISP is able to comfortably outperform LTH across 8 out of 9 tasks (noticeably
for QNLI where ISP beat LTH by $>1\%$).

###  3.3 Analysis of Denoising Iterations in ISP

Our proposed approach ISP is augmented by a novel idea of mask denoising,
which explores the superimposition of computationally cheap pruning mask
obtained by magnitude-based one-shot pruning of the network with marginal
look-ahead training. In this section, we try to investigate the implication of
our denoising iterations in improving ISP performance. Table 4 summarizes the
performance comparison of ISP with/without the mask denoising while keeping
the training settings exactly the same. Across both candidate architectures
(CLIP and BERT), it can be clearly observed that ISP performance is
significantly boosted by replacing the simple one-shot pruning with our
denoise pruning routine. In addition, we also investigated how the number of
denoising iterations will impact the ISP performance (see Table 5) and found
that 4-5 denoising steps are sufficient for the denoised pruning, and
increasing them beyond that does not provide a very noticeable performance
gain. For consistency, in CLIP-related experiments, we have used 5 denoising
iterations while for BERT, our results are reported using 4 denoising
iterations.

Table 4: Impact of Denoising Module in improving the performance of ISP.
Results are reported for three independent runs.

Approach | CLIP${}_{\texttt{ViT-B32}}$ | BERT${}_{\texttt{BASE}}$
---|---|---
SVHN | CIFAR-100 | QQP | QNLI
ISP - Denoiser | 96.11$\pm$0.21 | 70.32$\pm$0.13 | 89.96$\pm$0.39 | 89.28$\pm$0.87
ISP (Ours) | 96.46$\pm$0.05 | 71.09$\pm$0.17 | 90.59$\pm$0.47 | 90.06$\pm$1.01

Table 5: Performance comparison of ISP wrt. denoiser count on CIFAR10 with
CLIP (ViT-B32) pruned at 80% sparsity.

Denoiser Count | 0 | 2 | 4 | 6 | 8 | 16
---|---|---|---|---|---|---
Performance | 70.32 | 70.95 | 71.15 | 71.16 | 71.07 | 71.16

###  3.4 Understanding the benefits of Instant Model Soups for pre-trained
models

In this section, we discuss the benefits of our sparsity-inspired extension,
Instant Model Soup (IMS), and experimentally validate its surprising ability
to improve the quality of pre-trained models at marginal cost. Unlike model
soups, IMS provides a unique opportunity to eliminate the requirement to
generate multiple fully fine-tuned models to average, thereby restricting the
computational complexity equivalent to the cost of fine-tuning a single model.
Table 6 illustrates the performance comparison of IMS with respect to two
model soup variants (uniform and greedy) proposed in (Wortsman et al., 2022a).
Note that uniform and greedy soups results are generated using the
amalgamation of 8 independent models fine-tuned till the final accuracy with
different hyperparameters. Our experiments across CLIP and BERT illustrate
that by carefully fine-tuning IMS, it is surprisingly possible to comfortably
beat the model soup variants significantly. The denoised pre-trained model
generated by IMS has the ability to converge to equivalent (even better)
performance than model soups. Adhering to the theme of ISP, IMS also conveys a
strong message that it is not necessarily important to wait till model
convergence to ripe the benefits of soup, but astonishingly soup benefits are
available to ripe early during the fine-tuning at a marginal computational
cost.

Table 6: Fine-tuning performance comparison of our proposed approach (IMS)
wrt. basic fine-tuning and model soup variants.

Approach | CLIP${}_{\texttt{ViT-B32}}$ | BERT${}_{\texttt{BASE}}$
---|---|---
Cars | CIFAR10 | CIFAR100 | MNLI | QNLI
Pretrained-BASE | 76.43 | 97.60 | 89.35 | 82.39 | 90.04
Uniform Soup (Wortsman et al., 2022a) | 76.32 | 97.68 | 89.20 | 82.41 | 89.76
Greedy Soup (Wortsman et al., 2022a) | 77.95 | 98.05 | 89.54 | 83.01 | 90.64
IMS [Ours] | 78.79 | 98.01 | 89.64 | 83.63 | 91.23
(std.) |  $\pm$ 0.32 |  $\pm$ 0.07 |  $\pm$ 0.12 |  $\pm$ 0.43 |  $\pm$ 0.19

Table 7: Performance comparison of ISP wrt. look-ahead on CIFAR100 with CLIP
(ViT-B32) pruned at 60% sparsity.

Look-ahead Count(C) | 10 | 30 | 50 | 100 | 150 | 200
---|---|---|---|---|---|---
Performance | 84.71 | 84.92 | 85.18 | 85.19 | 84.41 | 85.41

##  4 Related Work

Linear interpolation of neural network weights has recently achieved
significant attention, but due to numerous nonlinear activations within a
neural network, it is still debatable if linearly interpolating between two
sets of weights can result in a high accuracy solution. Recently, (Frankle et
al., 2020; Nagarajan & Kolter, 2019; Von Oswald et al., 2020; Matena & Raffel,
2021; Wortsman et al., 2022a, b; Choshen et al., 2022; Izmailov et al., 2018;
Neyshabur et al., 2020) have studied the interpolation of deep networks and
validated performance benefits when training starts from a common
initialization or some segment of the optimization trajectories are shared.
While (Nagarajan & Kolter, 2019; Frankle et al., 2020) focused on mergability
in the case of models trained on a single task, (Wortsman et al., 2022b) found
that weight interpolation can not only benefit fine-tuning tasks but also
under distribution shift. More specifically, they average zero-shot and fine-
tuned models, finding improvements in- and out-of-distribution.

Recently, (Matena & Raffel, 2021) used Fisher-weighted averaging of language
models before and after fine-tuning on downstream tasks. They merged models
with the same pre-trained initialization that are fine-tuned on different text
classification tasks. In the late phases of training, (Von Oswald et al.,
2020) studied making copies of a subset of the neural network parameters and
proposed to independently optimize them, followed by averaging. Moreover,
(Wortsman et al., 2022a) proposed to average fine-tuned models across
independent runs with hyperparameter diversity, modifying all the weights of
the network, and showing significant performance benefits. In addition to
model weight averaging, (Bansal et al., 2021; Yang et al., 2022) explored the
idea of model stitching, where given two trained and frozen models A and B, a
“stitched model” formed by connecting the bottom-layers of A to the top-layers
of B, with a simple trainable layer between them. (Fort et al., 2019) studied
deep ensembles which have empirically shown promise for improving the
accuracy, uncertainty and out-of-distribution robustness of deep learning
models.

##  5 Conclusion

In this work, we introduced Instant Soup Pruning, a model soup-inspired
perspective dedicated to generating LTH quality subnetworks, using a fraction
of the original IMP cost. ISP is augmented by a denoising pruning module which
helps in replacing the expensive intermediate pruning stages of IMP with
computationally efficient weak mask generation and aggregation routine.
Additionally, we present Instant Model Soup, which provides an opportunity to
inject the benefits of model soups in dense pre-trained models at marginal
training cost, thereby improving fine-tuning performance comparable to model
soups. Our future work will aim for a more theoretical understanding of the
role of our denoisers in providing experimental benefits.

##  6 Acknowledgement

The research is based upon work supported in part by the Intelligence Advanced
Research Projects Activity (IARPA) under Contract No. 2022-21102100004. We
also acknowledge support from the National Science Foundation AI Center
Institute for Foundations of Machine Learning (IFML) at the University of
Texas at Austin.

## References

  * Bansal et al. (2021) Bansal, Y., Nakkiran, P., and Barak, B.  Revisiting model stitching to compare neural representations.  _Advances in neural information processing systems_ , 34:225–236, 2021.
  * Chen et al. (2020a) Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Wang, Z., and Carbin, M.  The lottery ticket hypothesis for pre-trained bert networks.  _Advances in neural information processing systems_ , 33:15834–15846, 2020a.
  * Chen et al. (2021a) Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., and Wang, Z.  Chasing sparsity in vision transformers: An end-to-end exploration.  _Advances in Neural Information Processing Systems_ , 34:19974–19988, 2021a.
  * Chen et al. (2021b) Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Carbin, M., and Wang, Z.  The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16306–16316, 2021b.
  * Chen et al. (2023) Chen, T., Zhang, Z., JAISWAL, A. K., Liu, S., and Wang, Z.  Sparse moe as the new dropout: Scaling dense and self-slimmable transformers.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Chen et al. (2020b) Chen, X., Cheng, Y., Wang, S., Gan, Z., Wang, Z., and Liu, J.  Earlybert: Efficient bert training via early-bird lottery tickets.  _arXiv preprint arXiv:2101.00063_ , 2020b.
  * Choshen et al. (2022) Choshen, L., Venezian, E., Slonim, N., and Katz, Y.  Fusing finetuned models for better pretraining.  _arXiv preprint arXiv:2204.03044_ , 2022.
  * Chowdhery et al. (2022) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Ding et al. (2019) Ding, M., Zhou, C., Chen, Q., Yang, H., and Tang, J.  Cognitive graph for multi-hop reading comprehension at scale.  _arXiv preprint arXiv:1905.05460_ , 2019.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Fang et al. (2023) Fang, G., Ma, X., Song, M., Mi, M. B., and Wang, X.  Depgraph: Towards any structural pruning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16091–16101, 2023.
  * Fort et al. (2019) Fort, S., Hu, H., and Lakshminarayanan, B.  Deep ensembles: A loss landscape perspective.  _arXiv preprint arXiv:1912.02757_ , 2019.
  * Frankle & Carbin (2018) Frankle, J. and Carbin, M.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  _arXiv preprint arXiv:1803.03635_ , 2018.
  * Frankle et al. (2020) Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M.  Linear mode connectivity and the lottery ticket hypothesis.  In _International Conference on Machine Learning_ , pp. 3259–3269. PMLR, 2020.
  * Gan et al. (2022) Gan, Z., Chen, Y.-C., Li, L., Chen, T., Cheng, Y., Wang, S., Liu, J., Wang, L., and Liu, Z.  Playing lottery tickets with vision and language.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 652–660, 2022.
  * Han et al. (2020) Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., Yang, Z., Zhang, Y., and Tao, D.  A survey on visual transformer.  _ArXiv_ , abs/2012.12556, 2020.
  * Ilharco et al. (2022) Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L.  Patching open-vocabulary models by interpolating weights.  _arXiv preprint arXiv:2208.05592_ , 2022.
  * Izmailov et al. (2018) Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G.  Averaging weights leads to wider optima and better generalization.  _arXiv preprint arXiv:1803.05407_ , 2018.
  * Jaiswal et al. (2021a) Jaiswal, A., Li, T., Zander, C., Han, Y., Rousseau, J. F., Peng, Y., and Ding, Y.  Scalp-supervised contrastive learning for cardiopulmonary disease classification and localization in chest x-rays using patient metadata.  In _2021 IEEE International Conference on Data Mining (ICDM)_ , pp. 1132–1137. IEEE, 2021a.
  * Jaiswal et al. (2021b) Jaiswal, A., Tang, L., Ghosh, M., Rousseau, J., Peng, Y., and Ding, Y.  Radbert-cl: Factually-aware contrastive learning for radiology report classification.  _Proceedings of machine learning research_ , 158:196–208, 2021b.
  * Jaiswal et al. (2022a) Jaiswal, A., Ashutosh, K., Rousseau, J. F., Peng, Y., Wang, Z., and Ding, Y.  Ros-kd: A robust stochastic knowledge distillation approach for noisy medical imaging.  _arXiv preprint arXiv:2210.08388_ , 2022a.
  * Jaiswal et al. (2022b) Jaiswal, A., Ma, H., Chen, T., Ding, Y., and Wang, Z.  Spending your winning lottery better after drawing it, 2022b.  URL https://openreview.net/forum?id=O4dxuEsIo9S.
  * Jaiswal et al. (2023) Jaiswal, A., Chen, T., Rousseau, J. F., Peng, Y., Ding, Y., and Wang, Z.  Attend who is weak: Pruning-assisted medical image localization under sophisticated and implicit imbalances.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 4987–4996, 2023.
  * Jaiswal et al. (2022c) Jaiswal, A. K., Ma, H., Chen, T., Ding, Y., and Wang, Z.  Training your sparse neural network better with any mask.  In _International Conference on Machine Learning_ , pp. 9833–9844. PMLR, 2022c.
  * Juneja et al. (2022) Juneja, J., Bansal, R., Cho, K., Sedoc, J., and Saphra, N.  Linear connectivity reveals generalization strategies.  _arXiv preprint arXiv:2205.12411_ , 2022.
  * Lee et al. (2018) Lee, N., Ajanthan, T., and Torr, P. H.  Snip: Single-shot network pruning based on connection sensitivity.  _arXiv preprint arXiv:1810.02340_ , 2018.
  * Li et al. (2023) Li, T., Shetty, S., Kamath, A., Jaiswal, A., Jiang, X., Ding, Y., and Kim, Y.  Cancergpt: Few-shot drug pair synergy prediction using large pre-trained language models.  _arXiv preprint arXiv:2304.10946_ , 2023.
  * Liu et al. (2023) Liu, S., Chen, T., Zhang, Z., Chen, X., Huang, T., Jaiswal, A., and Wang, Z.  Sparsity may cry: Let us fail (current) sparse neural networks together!  _arXiv preprint arXiv:2303.02141_ , 2023.
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Liu et al. (2021) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 10012–10022, 2021.
  * Mao et al. (2022) Mao, Z., Jaiswal, A., Wang, Z., and Chan, S. H.  Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model.  _ArXiv_ , abs/2207.10040, 2022.
  * Matena & Raffel (2021) Matena, M. and Raffel, C.  Merging models with fisher-weighted averaging.  _arXiv preprint arXiv:2111.09832_ , 2021.
  * Nagarajan & Kolter (2019) Nagarajan, V. and Kolter, J. Z.  Uniform convergence may be unable to explain generalization in deep learning.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Neyshabur et al. (2020) Neyshabur, B., Sedghi, H., and Zhang, C.  What is being transferred in transfer learning?  _Advances in neural information processing systems_ , 33:512–523, 2020.
  * Parmar et al. (2018) Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N. M., Ku, A., and Tran, D.  Image transformer.  In _ICML_ , 2018.
  * Prasanna et al. (2020) Prasanna, S., Rogers, A., and Rumshisky, A.  When bert plays the lottery, all tickets are winning.  _arXiv preprint arXiv:2005.00561_ , 2020.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Talmor et al. (2018) Talmor, A., Herzig, J., Lourie, N., and Berant, J.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_ , 2018.
  * Touvron et al. (2021) Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J’egou, H.  Training data-efficient image transformers & distillation through attention.  In _ICML_ , 2021.
  * Von Oswald et al. (2020) Von Oswald, J., Kobayashi, S., Sacramento, J., Meulemans, A., Henning, C., and Grewe, B. F.  Neural networks with late-phase weights.  _arXiv preprint arXiv:2007.12927_ , 2020.
  * Wang et al. (2018) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. (2020) Wang, Z., Tsvetkov, Y., Firat, O., and Cao, Y.  Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models.  _arXiv preprint arXiv:2010.05874_ , 2020.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022.
  * Wolf et al. (2019) Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Wortsman et al. (2022a) Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pp. 23965–23998. PMLR, 2022a.
  * Wortsman et al. (2022b) Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al.  Robust fine-tuning of zero-shot models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7959–7971, 2022b.
  * Yang et al. (2019a) Yang, W., Xie, Y., Lin, A., Li, X., Tan, L., Xiong, K., Li, M., and Lin, J.  End-to-end open-domain question answering with bertserini.  _arXiv preprint arXiv:1902.01718_ , 2019a.
  * Yang et al. (2022) Yang, X., Zhou, D., Liu, S., Ye, J., and Wang, X.  Deep model reassembly.  _Advances in neural information processing systems_ , 35:25739–25753, 2022.
  * Yang et al. (2019b) Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32, 2019b.
  * Yin et al. (2022) Yin, L., Liu, S., Meng, F., Huang, T., Menkovski, V., and Pechenizkiy, M.  Lottery pools: Winning more by interpolating tickets without increasing training or inference cost.  _arXiv preprint arXiv:2208.10842_ , 2022.
  * You et al. (2019) You, H., Li, C., Xu, P., Fu, Y., Wang, Y., Chen, X., Baraniuk, R. G., Wang, Z., and Lin, Y.  Drawing early-bird tickets: Towards more efficient training of deep networks.  _arXiv preprint arXiv:1909.11957_ , 2019.
  * Yu et al. (2020) Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C.  Gradient surgery for multi-task learning.  _Advances in Neural Information Processing Systems_ , 33:5824–5836, 2020.
  * Yu et al. (2017) Yu, X., Liu, T., Wang, X., and Tao, D.  On compressing deep models by low rank and sparse decomposition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7370–7379, 2017.
  * Zheng et al. (2021) Zheng, M., Gao, P., Zhang, R., Wang, X., Li, H., and Dong, H.  End-to-end object detection with adaptive clustering transformer.  _ArXiv_ , abs/2011.09315, 2021.
  * Zheng et al. (2023) Zheng, W., Sharan, S., Jaiswal, A. K., Wang, K., Xi, Y., Xu, D., and Wang, Z.  Outline, then details: Syntactically guided coarse-to-fine code generation.  _arXiv preprint arXiv:2305.00909_ , 2023.


# A sharp inequality for trace-free matrices with applications to
hypersurfaces

Jeffrey S. Case  109 McAllister Building
Penn State University
University Park, PA 16802
USA<EMAIL_ADDRESS>and  Aaron J. Tyrrell  18A Department of Mathematics and
Statistics
Texas Tech University
Lubbock
TX 79409
USA<EMAIL_ADDRESS>
###### Abstract.

We derive a sharp inequality relating the second and fourth elementary
symmetric functions of the eigenvalues of a trace-free matrix and give two
applications. First, we give a new proof of the classification of conformally
flat hypersurfaces in spaceforms. Second, we construct a functional which
characterizes rotational hypersurfaces and catenoids.

###### Key words and phrases:

rotation hypersurface; catenoid; rigidity

###### 2020 Mathematics Subject Classification:

Primary 53C42; Secondary 26D05, 53A07, 53C24

##  1\. Introduction

Let $i\colon N^{n}\to(M^{n+1},g)$, $n\geq 4$, be an immersed hypersurface in a
locally conformally flat Riemannian manifold. This note is motivated by two
interesting rigidity results for $N$ in terms of its principal curvatures.
First, Nishikawa and Maeda showed [NiskikawaMaeda1974] that
$(N^{n},i^{\ast}g)$ is locally conformally flat if and only if at each point
it has a principal curvature of multiplicity at least $n-1$. Second, do Carmo
and Dajczer showed [do2012rotation] that, under the stronger assumption that
$(M^{n+1},g)$ is complete, simply connected, and of constant sectional
curvature $c\in\mathbb{R}$, the hypersurface $N$ is contained in a so-called
rotation hypersurface [do2012rotation]*Definition 2.2 if at each point it has
a principal curvature of multiplicity exactly $n-1$. Furthermore, if $N$ is
minimal, then it is contained in a catenoid if and only if at each point it
has a principal curvature of multiplicity at least $n-1$.

The purpose of this short note is to point out that the condition on the
principal curvatures can be recast in terms of a sharp inequality relating the
squared-norm $\lvert\mathring{A}^{2}\rvert^{2}$ of the square of the trace-
free part of the second fundamental form $\mathring{A}$ and the square of the
squared-norm $\lvert\mathring{A}\rvert^{2}$ of $\mathring{A}$. Our main result
is a purely algebraic statement about trace-free linear maps on finite-
dimensional vector spaces:

######  Theorem 1.1.

Let $V$ be an $n$-dimensional inner product space, $n\geq 4$, and let
$\mathring{A}\in\operatorname{End}(V)$ be trace-free. Then

$\lvert\mathring{A}^{2}\rvert^{2}\leq\frac{n^{2}-3n+3}{n(n-1)}\lvert\mathring{A}\rvert^{4}$

with equality if and only if $\mathring{A}$ has an eigenspace of dimension at
least $n-1$.

The main idea of the proof of Theorem 1.1 is that the Newton inequality
relating the first, second, and third elementary symmetric functions of the
eigenvalues of a matrix $A$ realizes equality if and only if $A$ is
proportional to the identity or has rank $1$. Applying this to
$\mathring{A}+\lambda I$ for $I$ the identity map yields the sharp inequality

$\left(\operatorname{tr}\mathring{A}^{3}\right)^{2}\leq\frac{(n-2)^{2}}{n(n-1)}\lvert\mathring{A}\rvert^{6}$

for any trace-free $\mathring{A}\in\operatorname{End}(V^{n})$; this is
sometimes called the $\lambda$-method [Mitrinovic1970]. Repeating this
argument for the Newton inequality relating the second, third, and fourth
elementary symmetric functions yields Theorem 1.1.

We conclude with a number of applications of Theorem 1.1 to the rigidity of
locally conformally flat hypersurfaces, of rotation hypersurfaces, and of
catenoids.

Our first application is a new proof of the characterization
[NiskikawaMaeda1974] of locally conformally flat hypersurfaces in locally
conformally flat manifolds in terms of their principal curvatures.

######  Corollary 1.2.

Let $i\colon N^{n}\to(M^{n+1},g)$, $n\geq 4$, be an immersed hypersurface in a
locally conformally flat manifold. The induced metric $i^{\ast}g$ on $N$ is
locally conformally flat if and only if for each $p\in N$, the shape operator
$A_{p}\colon T_{p}N\to T_{p}N$ has an eigenspace of dimension at least $n-1$.

Our second application is a rigidity result for catenoids in simply connected
spaceforms amongst all minimial hypersurfaces.

######  Corollary 1.3.

Let $i\colon N^{n}\to M^{n+1}(c)$, $n\geq 4$, be a minimal hypersurface in the
complete, simply connected Riemannian $(n+1)$-manifold of constant sectional
curvature $c\in\mathbb{R}$. Then the shape operator $A\colon TN\to TN$
satisfies

(1.1) $\lvert A^{2}\rvert^{2}\leq\frac{n^{2}-3n+3}{n(n-1)}\lvert A\rvert^{4}.$

Moreover, equality holds if and only if $i(N)$ is contained in a catenoid.

The above results motivate the introduction of two energy functionals for
hypersurfaces of a Riemannian manifold. First, given an immersion $i\colon
N^{n}\to(M^{n+1},g)$, we define the _rotational energy_ of $N$ by

$E_{rot}[N]:=\int_{N}\left(\frac{n^{2}-3n+3}{n(n-1)}\lvert\mathring{A}\rvert^{4}-\lvert\mathring{A}^{2}\rvert^{2}\right)\,dA,$

assuming it is finite. Our terminology is motivated by the characterization
[do2012rotation] of rotation hypersurfaces in simply connected spaceforms.

######  Corollary 1.4.

Let $i\colon N^{n}\to M^{n+1}(c)$, $n\geq 4$, be a nowhere umbilic immersed
hypersurface in the complete, simply connected, Riemannian $(n+1)$-manifold of
constant sectional curvature $c\in\mathbb{R}$. Then $E_{rot}[N]\geq 0$ with
equality if and only if $N$ is contained in a rotation hypersurface.

This result is sharper for minimal immersions.

######  Corollary 1.5.

Let $i\colon N^{n}\to M^{n+1}(c)$, $n\geq 4$, be a minimal immersed
hypersurface in the complete, simply connected, Riemannian $(n+1)$-manifold of
constant sectional curvature $c\in\mathbb{R}$. Then $E_{rot}[N]\geq 0$ with
equality if and only if $N$ is contained in a catenoid.

Second, given an immersion $i\colon N^{n}\to(M^{n+1},g)$, we define the
_conformal rotational energy_ of $N$ by

$E_{rot}^{conf}[N]:=\int_{N}\lvert\mathring{A}\rvert^{n-4}\left(\frac{n^{2}-3n+3}{n(n-1)}\lvert\mathring{A}\rvert^{4}-\lvert\mathring{A}^{2}\rvert^{2}\right)\,dA,$

assuming it is finite. Note that $E_{rot}^{conf}=E_{rot}$ for four-dimensional
hypersurfaces. The main point is of this definition is that $E_{rot}^{conf}$
is conformally invariant, and hence gives a conformally invariant
characterization of immersed submanifolds for which the shape operator has an
eigenspace of dimension at least $n-1$.

######  Corollary 1.6.

Let $i\colon N^{n}\to(M^{n+1},g)$, $n\geq 4$, be an immersed hypersurface in a
Riemannian manifold. Then $E_{rot}^{conf}[N]\geq 0$ with equality if and only
if for each $p\in N$, the shape operator $A_{p}\colon T_{p}N\to T_{p}N$ has an
eigenspace of dimension at least $n-1$.

This note is organized as follows. In Section 2 we give an elementary proof of
Theorem 1.1. In Section 3 we prove our geometric applications of Theorem 1.1.

##  2\. Elementary symmetric functions

Let $A\in\operatorname{End}(V^{n})$ be a linear map on an $n$-dimensional
vector space. Denote by

(2.1)
$\sigma_{k}(A):=\sum_{i_{1}<\dotsm<i_{k}}\lambda_{i_{1}}\dotsm\lambda_{i_{k}}$

the $k$-th elementary symmetric function of the eigenvalues
$\\{\lambda_{1},\dotsc,\lambda_{n}\\}$ of $A$, with the convention
$\sigma_{0}(A)=1$. Note that $\sigma_{1}(A)=\operatorname{tr}A$. Define
$p_{k}(A)$, $k\leq n$, by

$\sigma_{k}(A)=\binom{n}{k}p_{k}(A);$

this renormalization is such that $p_{k}(\lambda I)=\lambda^{k}$, where $I$ is
the identity map. The main tool in the proof of Theorem 1.1 is the sharp
version of Newton’s inequalities:

######  Lemma 2.1 ([Rosset1989]*Section 2).

Let $A\in\operatorname{End}(V^{n})$ and let $k\leq n-1$ be a positive integer.
Then

$p_{k}^{2}(A)\geq p_{k-1}(A)p_{k+1}(A)$

with equality if and only if $A$ is proportional to the identity or $\dim\ker
A\geq n-k+1$.

Applying the $\lambda$-method to Lemma 2.1 with $k=2$ and $k=3$ yields useful
inequalities, the second of which is recorded in Theorem 1.1. We record the
resulting inequalities separately due to their different dependence on $n=\dim
V$.

Our first inequality relates $p_{2}(\mathring{A})$ and $p_{3}(\mathring{A})$
for a trace-free linear map $\mathring{A}$ on a vector space of dimension at
least $3$.

######  Proposition 2.2.

Let $\mathring{A}\in\operatorname{End}(V^{n})$, $n\geq 3$, be such that
$p_{1}(\mathring{A})=0$. Then

$p_{3}^{2}(\mathring{A})+4p_{2}^{3}(\mathring{A})\leq 0.$

Moreover, equality holds if and only if $\mathring{A}$ has an eigenspace of
dimension at least $n-1$.

###### Proof.

It is well-known, and follows easily from Equation (2.1), that

(2.2) $p_{k}(A+\lambda I)=\sum_{j=0}^{k}\binom{k}{j}\lambda^{j}p_{k-j}(A)$

for all $A\in\operatorname{End}(V^{n})$ and all $\lambda\in\mathbb{R}$.

Now let $\mathring{A}\in\operatorname{End}(V^{n})$ be such that
$p_{1}(\mathring{A})=0$. Lemma 2.1 implies that $p_{2}(\mathring{A})\leq 0$
with equality if and only if $\mathring{A}=0$. Since the conclusion is
trivially true if $\mathring{A}=0$, we may assume that
$p_{2}(\mathring{A})<0$. For notational simplicity, denote
$p_{k}:=p_{k}(\mathring{A})$.

Let $\lambda\in\mathbb{R}$. Applying Lemma 2.1 to $\mathring{A}+\lambda I$ and
simplifying via Equation (2.2) yields

(2.3) $\begin{split}0&\leq(p_{2}+\lambda^{2})^{2}-\lambda(p_{3}+3\lambda
p_{2}+\lambda^{3})\\\ &=p_{2}^{2}-\lambda p_{3}-\lambda^{2}p_{2},\end{split}$

with equality if and only if $\mathring{A}$ has an eigenspace of dimension at
least $n-1$. In fact, since $p_{1}=0$ and $p_{2}<0$, the dimension must equal
$n-1$ in the case of equality in Inequality (2.3). Multiplying Inequality
(2.3) by $4p_{2}<0$ yields

$\displaystyle 0$ $\displaystyle\geq 4p_{2}^{3}-4\lambda
p_{2}p_{3}-4\lambda^{2}p_{2}^{2}$ $\displaystyle=-(2\lambda
p_{2}+p_{3})^{2}+4p_{2}^{3}+p_{3}^{2}.$

Therefore $p_{3}^{2}+4p_{2}^{3}\leq 0$ with equality if and only if
$\mathring{A}$ has an eigenspace of dimension $n-1$. ∎

Our second inequality relates $p_{2}(\mathring{A})$ and $p_{4}(\mathring{A})$
for a trace-free linear map $\mathring{A}$ on a vector space of dimension at
least $4$.

######  Proposition 2.3.

Let $\mathring{A}\in\operatorname{End}(V^{n})$, $n\geq 4$, be such that
$p_{1}(\mathring{A})=0$. Then

$p_{4}(\mathring{A})+3p_{2}^{2}(\mathring{A})\geq 0.$

Moreover, equality holds if and only if $\mathring{A}$ has an eigenspace of
dimension at least $n-1$.

###### Proof.

Let $\mathring{A}\in\operatorname{End}(V^{n})$ be such that
$p_{1}(\mathring{A})=0$. For notational simplicity, denote
$p_{k}:=p_{k}(\mathring{A})$. Lemma 2.1 implies that $p_{2}(\mathring{A})\leq
0$ with equality if and only if $\mathring{A}=0$. Since the conclusion is
trivially true if $\mathring{A}=0$, we may assume that $p_{2}<0$.

Now, applying Lemma 2.1 and Equation (2.2) to $\mathring{A}+\lambda I$ yields

$\displaystyle 0$ $\displaystyle\leq(p_{3}+3\lambda
p_{2}+\lambda^{3})^{2}-(p_{2}+\lambda^{2})(p_{4}+4\lambda
p_{3}+6\lambda^{2}p_{2}+\lambda^{4})$
$\displaystyle=p_{3}^{2}-p_{2}p_{4}+2\lambda
p_{2}p_{3}+\lambda^{2}(3p_{2}^{2}-p_{4})-2\lambda^{3}p_{3}-\lambda^{4}p_{2}$

for all $\lambda\in\mathbb{R}$. Multiplying by $16p_{2}^{3}$ and denoting
$s:=2\lambda p_{2}+p_{3}$ yields

$\displaystyle 0$ $\displaystyle\geq
16p_{2}^{3}p_{3}^{2}-16p_{2}^{4}p_{4}+32\lambda
p_{2}^{4}p_{3}+16\lambda^{2}p_{2}^{2}(3p_{2}^{3}-p_{2}p_{4})-32\lambda^{3}p_{2}^{3}p_{3}-16\lambda^{4}p_{2}^{4}$
$\displaystyle=(3p_{3}^{2}-4p_{2}p_{4})(p_{3}^{2}+4p_{2}^{3})+8sp_{3}(p_{2}p_{4}-p_{2}^{3}-p_{3}^{2})+s^{2}(12p_{2}^{3}-4p_{2}p_{4}+6p_{3}^{2})-s^{4}.$

Setting $s=0$ yields

(2.4) $(3p_{3}^{2}-4p_{2}p_{4})(p_{3}^{2}+4p_{2}^{3})\leq 0.$

Proposition 2.2 implies that $p_{3}^{2}+4p_{2}^{3}\leq 0$ with equality if and
only if, up to a multiplicative constant and a choice of basis,

$\mathring{A}=\begin{pmatrix}1&0&\cdots&0&0\\\ 0&1&\cdots&0&0\\\
\vdots&\vdots&\ddots&\vdots&\vdots\\\ 0&0&\cdots&1&0\\\
0&0&\cdots&0&1-n\end{pmatrix}.$

If $p_{3}^{2}+4p_{2}^{3}=0$, then a straightforward computation yields
$p_{k}=1-k$ for all $k\in\mathbb{N}$, and hence $p_{4}+3p_{2}^{2}=0$. If
instead $p_{3}^{2}+4p_{2}^{3}<0$, then Inequality (2.4) implies that
$3p_{3}^{2}-4p_{2}p_{4}\geq 0$. Therefore

$0\leq
3p_{3}^{2}-4p_{2}p_{4}<-12p_{2}^{3}-4p_{2}p_{4}=-4p_{2}(p_{4}+3p_{2}^{2}).$

Since $p_{2}<0$, we conclude that $p_{4}+3p_{2}^{2}>0$. ∎

Expressing the nonnegative quantity $p_{4}+3p_{2}^{2}$ in terms of $\lvert
A\rvert^{4}$ and $\lvert A^{2}\rvert^{2}$ yields Theorem 1.1.

###### Proof of Theorem 1.1.

Let $\mathring{A}\in\operatorname{End}(V^{n})$ be such that
$p_{1}(\mathring{A})=0$. Direct computation gives

$\displaystyle\sigma_{2}(\mathring{A})$
$\displaystyle=-\frac{1}{2}\lvert\mathring{A}\rvert^{2},$
$\displaystyle\sigma_{4}(\mathring{A})$
$\displaystyle=\frac{1}{8}\lvert\mathring{A}\rvert^{4}-\frac{1}{4}\lvert\mathring{A}^{2}\rvert^{2}.$

Therefore

$0\leq\binom{n}{4}\left(p_{4}(\mathring{A})+3p_{2}^{2}(\mathring{A})\right)=-\frac{1}{4}\left(\lvert\mathring{A}^{2}\rvert^{2}-\frac{n^{2}-3n+3}{n(n-1)}\lvert\mathring{A}\rvert^{4}\right).$

The conclusion readily follows from Proposition 2.3. ∎

##  3\. Geometric applications

We conclude by proving our geometric applications of Theorem 1.1.

###### Proof of Corollary 1.2.

Let $i\colon N^{n}\to(M^{n+1},g)$ be a Riemannian immersion into a locally
conformally flat manifold. Denote by $\overline{W}$ the Weyl tensor of
$i^{\ast}g$. Denote by

$F:=\frac{1}{n-2}\left(\mathring{A}^{2}-Gi^{\ast}g\right)$

the Fialkow tensor $F\in\Gamma(S^{2}T^{\ast}N)$, where
$G:=\operatorname{tr}_{i^{\ast}g}F=\frac{1}{2(n-1)}\lvert\mathring{A}\rvert^{2}$
is its trace (cf. [Fialkow1944]*Equation (13.9)). The Gauss–Codazzi equations
(cf. [Fialkow1944]*Equation (22.13)) imply that

$\overline{W}=\frac{1}{2}\mathring{A}\wedge\mathring{A}+F\wedge g,$

where $S\wedge T$ denotes the Kulkarni–Nomizu product

$(S\wedge T)_{abcd}:=S_{ac}T_{bd}+S_{bd}T_{ac}-S_{ad}T_{bc}-S_{bc}T_{ad}.$

Direct computation gives (cf. [Labbi2005])

$\displaystyle\lvert\mathring{A}\wedge\mathring{A}\rvert^{2}$
$\displaystyle=8\lvert\mathring{A}\rvert^{4}-8\lvert\mathring{A}^{2}\rvert^{2},$
$\displaystyle\langle\mathring{A}\wedge\mathring{A},F\wedge g\rangle$
$\displaystyle=-8\langle\mathring{A}^{2},F\rangle,$ $\displaystyle\lvert
F\wedge g\rvert^{2}$ $\displaystyle=4\langle\mathring{A}^{2},F\rangle,$
$\displaystyle\langle\mathring{A}^{2},F\rangle$
$\displaystyle=\frac{1}{n-2}\lvert\mathring{A}^{2}\rvert^{2}-\frac{1}{2(n-1)(n-2)}\lvert\mathring{A}\rvert^{4}.$

Therefore

$\lvert\overline{W}\rvert^{2}=\frac{2(n^{2}-3n+3)}{(n-1)(n-2)}\lvert\mathring{A}\rvert^{4}-\frac{2n}{n-2}\lvert\mathring{A}^{2}\rvert^{2}.$

The conclusion readily follows from Theorem 1.1. ∎

###### Proof of Corollary 1.3.

Let $i\colon N^{n}\to M^{n+1}(c)$ be a minimal hypersurface. By minimality,
the shape operator is trace-free; i.e. $A=\mathring{A}$. Theorem 1.1 implies
that $A$ satisfies Inequality (1.1) with equality if and only if $A$ has an
eigenvalue of multiplicity at least $n-1$. The final conclusion follows from a
result [do2012rotation]*Corollary 4.4 of do Carmo and Dajczer. ∎

###### Proof of Corollary 1.4.

Let $i\colon N^{n}\to M^{n+1}(c)$ be a nowhere umbilic immersed hypersurface.
Theorem 1.1 implies that $E_{rot}[N]\geq 0$ with equality if and only if at
each point $p\in M$ the shape operator $A_{p}$ has an eigenspace of dimension
exactly $n-1$. The final conclusion follows from a result
[do2012rotation]*Theorem 4.2 of do Carmo and Dajczer. ∎

###### Proof of Corollary 1.5.

Let $i\colon N^{n}\to M^{n+1}(c)$ be a minimal immersed hypersurface.
Corollary 1.3 implies that $E_{rot}[N]\geq 0$ with equality if and only if
$i(N)$ is contained in a catenoid. ∎

###### Proof of Corollary 1.6.

Let $i\colon N^{n}\to(M^{n+1},g)$ be an immersed hypersurface. Theorem 1.1
implies that $E_{rot}^{conf}[N]\geq 0$ with equality if and only if

$\lvert\mathring{A}\rvert^{n-4}\left(\frac{n^{2}-3n+3}{n(n-1)}\lvert\mathring{A}\rvert^{4}-\lvert\mathring{A}^{2}\rvert^{2}\right)=0.$

It follows that for each $p\in N$, either $\mathring{A}_{p}=0$ or

$\lvert\mathring{A}_{p}^{2}\rvert^{2}=\frac{n^{2}-3n+3}{n(n-1)}\lvert\mathring{A}_{p}\rvert^{4}.$

In the former case, $\mathring{A}_{p}$ has an eigenspace of dimension $n$. In
the latter case, Theorem 1.1 implies that $\mathring{A}_{p}$ has an eigenspace
of dimension at least $n-1$. ∎

## Acknowledgements

We thank Eudes Leite de Lima for pointing out a mistake in an early version of
this paper. JSC was partially supported by the Simons Foundation (Grant
#524601).

## References