# Long Story Short: a Summarize-then-Search Method for Prompt-Based Long Video
Question Answering

Youngjae
Yu${}^{\text{\smash{\raisebox{-1.0pt}{\includegraphics[height=6.0pt]{graphics/puzzle2.pdf}}}}}$
Jiwan Chung Yejin
Choi${}^{\text{\smash{\raisebox{-1.0pt}{\includegraphics[height=6.0pt]{graphics/puzzle2.pdf}}}}\text{\smash{\raisebox{-1.0pt}{\includegraphics[height=6.0pt]{graphics/puzzle3.pdf}}}}}$
Allen Institute for Artificial Intelligence
Paul G. Allen School of Computer Science, University of Washington

###### Abstract

Large language models such as GPT-3 have demonstrated an impressive capability
to adapt to new tasks without requiring task-specific training data. This
capability has been particularly effective in settings such as narrative
question answering, where the diversity of tasks is immense, but the available
supervision data is small. In this work, we investigate if such language
models can extend their zero-shot reasoning abilities to long multimodal
narratives in multimedia content such as drama, movies, and animation, where
the story plays an essential role. We propose Long Story Short, a framework
for narrative video QA that first summarizes the narrative of the video to a
short plot and then searches parts of the video relevant to the question. We
also propose to enhance visual matching with CLIPCheck. Our model outperforms
state-of-the-art supervised models by a large margin, highlighting the
potential of zero-shot QA for long videos.

##  1 Introduction

Figure 1:  Long Story Short (LSS) uses Large Language Models (LLMs) (i.e.,
GPT-3) to generate (a) Screenplay and summarized (b) Plots from video. Further
details about data processing can be found in Section 2. When LSS answer
questions about the video, the model (c) validate given raw video footage with
Visual Language Model, CLIP, and (d) search further grounded scripts in a
backward manner, which we call CLIPCheck in Section 2.3.

Recent video QA models face challenges in handling long video narrative QA
tasks choi2020dramaqa; tapaswi2016movieqa; kim2017deepstory (i.e., films,
dramas, and YouTube web videos) due to the limitation in data and annotations.
This results in an inability to comprehend the long video narratives beyond
answering mainly visual questions on short video clip xu2017video;
lei2018tvqa; lei2019tvqa. The sizes of such long video QAs are insufficient to
train the models to fully comprehend the complex narrative structures within a
video, yielding sub-optimal performances. (jasani2019we) demonstrate that the
supervised models rely more on language biases in the question than the
narrative context: they can obtain similar performance even without seeing any
video context. This highlights the necessity of multimodal reasoning
capability beyond small task-specific supervision.

To address the challenge caused by low generalization, a zero-shot approach
using pretrained Large Language Models (LLMs) can be an efficient alternative
for tackling complex QA tasks yang2021empirical, and text context
summarization zhang2020pegasus; He2022ZCodeAP. Yet, is the narrative QA
capability of such LLMs transferable to the video domain?

We propose Long Story Short (LSS), illustrated in figure 1, that translates
video clips into text screenplay format inspired by Socratic Model
zeng2022socratic. Using GPT-3 brown2020gpt3, we first summarize the long video
into a list of plots and then navigate both the generated summary and the raw
video context to resolve the given question. Our zero-shot method shows better
results than state-of-the-art supervised methods in MovieQA and DramaQA
dataset. Furthermore, we propose CLIPCheck, a visual-text matching method to
enhance visual alignment of the reasoning results provided by GPT-3. To
summarize, our main contributions are three-fold:

  1. 1.

We present LSS, a framework that summarizes a long video narrative to a list
of plots and retrieves the subplot relevant to the question.

  2. 2.

We demonstrate the importance of considering visual alignment strength via
CLIP-based matching in visual prompting.

  3. 3.

Our zero-shot approach achieves state-of-the-art performance in MovieQA
tapaswi2016movieqa and DramaQA choi2020dramaqa, outperforming supervised
baselines.

Figure 2: The qualitative result showing our proposed Long Story Short (LSS)
model that generates and retrieves the index of raw video footage. When the
model predicts the final answer from (i) the generated Summary and (ii) the
retrieved text context, CLIPCheck validates each candidate’s answers to revise
the final answer for the question.

##  2 Method

We describe Long Story Short (LSS), a prompt-based strategy that divides and
conquers narrative video QA by summarizing the plot and retrieving relevant
information with the plot. Our objective is to predict the correct answer
$a_{\hat{y}}$ from long video context $\mathbf{X}$. Since language models can
only accept tokens of fixed length (e.g. 4096 tokens for GPT-3), they cannot
process the rich context (e.g. subtitles or scene descriptions) of movies that
typically spans two long hours.

Thus, we introduce a summarize-then-search method for video question
answering. We first divide the long video context into clip segments and
summarize each segment into a subplot using GPT-3 to get the list of subplots
$\mathbf{S}_{X}=\\{s_{1},s_{2}\cdots s_{n}\\}$. Then, we retrieve video
segments $\mathbf{X}_{k}=\\{x_{k_{1}},\cdots,x_{k_{m}}\\}$ relevant to the
question with the subplot list as the input. We use both the full context of
the selected segments and the global plot information to derive likelihood
over the answer choices $p_{\theta}(a_{1}),\cdots,p_{\theta}(a_{5})$. Finally,
we apply CLIPCheck, to strengthen the visual grounding of the selected answer.
We provide the prompt templates in the appendix.

###  2.1 Plot Generation

We use the ground-truth video partitions to segment the whole video into a set
of shorter clips. Each long video $\mathbf{X}=\\{x_{1},\cdots,x_{n}\\}$
consists of $n$ clip segments, and each segment contains video $v_{i}$ and the
corresponding text $t_{i}$ such as subtitle or ASR.

$\displaystyle\mathbf{X}^{\prime}=\\{(v_{1},t_{1}),(v_{2},t_{1})\cdots(v_{n},t_{n})\\}$

As (yang2021empirical) transcribe image as language prompt for frozen GPT-3,
we retrieve DVS caption rohrbach-arxiv-2016 and image captions with pretrained
BLIP li2022blip for video $v_{i}\in\mathbf{X}$, and concatenate the aligned
text $t_{i}$ as $n$ generated scripts as illustrated in figure 1. We compose a
prompt to summarize the context of a video segment into a plot piece of up to
three sentences and use GPT-3 to build the plot list aligned with the segment
indices $S_{\mathbf{X}}=\\{s_{1},\cdots,s_{n}\\}$.

###  2.2 Narrative Search

Given the summarized narrative and the question, we wish to retrieve the
relatively short clip relevant to the question from the long video. Language
models generate open-ended text which is irregular and often noisy. To
retrieve the exact part of the video, we drive the model to output indices of
the plot rather than the text form.

We first assign consecutive indices to the list of summarized plot pieces
$S_{\mathbf{X}}=\\{s_{1},\cdots,s_{n}\\}$ aligned with the clip segmentation.
Then, we prompt the language model to output the indices of the plot pieces
$k=\\{k_{1},\cdots,k_{m}\\}$ to lookup for.

The generated indices might still be noisy due to the open-ended nature of
language models. When the model outputs an answer in text form, we use rouge-l
lin-2004-rouge score to find plot piece candidates whose similarity with the
generated sentence are above the specified threshold $\alpha\geq 0.5$.

Finally, we concatenate the plots $S_{\mathbf{X}}$, the visual and text
representation of the selected segments
${\mathbf{X}^{\prime}}_{k}=\\{(v_{k_{1}},t_{k_{1}}),\cdots,(v_{k_{m}},t_{k_{m}})\\}$,
the question $q$, and the answer choices $A=\\{a_{1},\cdots,a_{5}\\}$ to build
the prompt input for question answering. We process the prompt with the
language model with weights $\theta$ and use the index token likelihood as the
answer choice score.

$\displaystyle p_{\theta}(a_{i})$
$\displaystyle=p_{\theta}(i|S_{\mathbf{X}},\mathbf{X}_{k},q,A)$

###  2.3 Visual Checking

For a tighter visual-text matching, we introduce CLIPCheck, a method to
conjoin CLIP visual distance radford2021learning and the language model
likelihood. We start from the selected video segments
$\mathbf{X}_{k}=\\{x_{k_{1}},\cdots,x_{k_{m}}\\}$, answer choices
$A=\\{a_{1},\cdots,a_{5}\\}$, and answer likelihoods
$P_{\theta}=\\{p_{\theta}(a_{1}),\cdots,p_{\theta}(a_{5})\\}$ of GPT-3.

First, we use the CLIP image encoder to encode each frame $x_{k_{i_{j}}}$ of
the selected segments. When $l$ is the number of frames within a segment:

$\displaystyle\bar{x}_{k_{i_{j}}}$ $\displaystyle=CLIP_{V}(x_{k_{i_{j}}})$
$\displaystyle\bar{x}_{k_{i}}$
$\displaystyle=\\{\bar{x}_{k_{i_{1}}},\cdots,\bar{x}_{k_{i_{l}}}\\},\bar{\mathbf{X}}_{k}=\\{\bar{x}_{k_{1}},\cdots,\bar{x}_{k_{m}}\\}$

Then, we extract the CLIP text feature of each answer choice
$\bar{a}_{i}=CLIP_{L}(a_{i})$ and compare cosine similarity between the video
input and the answers. We select the best-matched frame for each answer to
derive the cosine similarity score.

$\displaystyle c(a,\bar{\mathbf{X}}_{k})$ $\displaystyle=\max_{i\leq m,j\leq
l}cossim(\bar{a},\bar{x}_{k_{i_{j}}})$

Then we apply the softmax function with temperature $\tau$ on the scores to
get normalized visual likelihood over the answer candidates
$P_{c}=\\{p_{c}(a_{1}),\cdots,p_{c}(a_{5})\\}$. Lastly, we multiply the answer
likelihood from the language model $P_{\theta}$ with the visual likelihood
$P_{c}$ to obtain the final likelihood. We simply select the answer with the
maximum value as the model answer.

We choose to consider CLIPCheck only when the language model is not certain of
its choice. Given the likelihoods of the top two answers
$p_{\theta}(a_{h_{1}}),p_{\theta}(a_{h_{2}})$ from the language model, we
measure the model certainty with binary entropy $E^{\prime}$ of the re-
normalized probability. We only use the combined likelihood when the binary
entropy is greater than the given threshold $E^{\prime}\geq 0.4$. Otherwise,
we do not apply CLIPCheck and just use the language model likelihood.

$\displaystyle h_{1}=\operatorname*{arg\,max}_{i\leq
5}p_{\theta}(a_{i}),h_{2}=\operatorname*{arg\,max}_{j\leq 5,j\neq
h_{1}}p_{\theta}(a_{j})$ $\displaystyle E^{\prime}=-p_{\theta}(a_{h_{1}})\log
p_{\theta}(a_{h_{1}})-p_{\theta}(a_{h_{2}})\log p_{\theta}(a_{h_{2}})$

##  3 Experiments

For all experiments, we use GPT-3 brown2020gpt3 (text-davinci-003) as the
backbone language model. Unless stated otherwise, we use the ground truth clip
boundary to segment the videos. All LSS variants do not use any training data
and thus are zero-shot methods.

###  3.1 Evaluating Long Story Short

MovieQA tapaswi2016movieqa is a large-scale QA dataset sourced from 408
movies. There are multiple sources of information in the dataset; subtitles,
scripts, DVS, video clips, and plots. We report four state-of-the-art
supervised baselines; A2A liu2019a2a, PAMN kim2019pamn, UniversalQA
jasani2019we, and DHTCN liu2020dhtcn.

Table LABEL:tab:movieqa shows zero-shot LSS improves over previous supervised
approaches. Also, Ours-search shows strong performance even without the
ground-truth segment index label. CLIPCheck slightly improves the accuracy in
the video split. However, the difference is marginal since MovieQA often
requires character-based grounding rather than general visual matching.
Finally, we experiment with the null hypothesis: No Context tests whether
GPT-3 solves MovieQA by simply memorizing every fact. No Context performs
worse than LSS, rejecting the null hypothesis.

PororoQA kim2017deepstory is a video story QA dataset built from a cartoon
series. The supervised baseline takes the human-generated plot and the ground
truth video segment index, while LSS +Plot+Search takes neither.

Table LABEL:tab:pororoqa summarizes our result on the PororoQA dataset. When
using both the ground-truth episode and plots, GPT-3 performs almost on par
with the supervised baseline. Substituting a human-generated summary with a
model-generated one results in only a marginal performance drop. Perhaps
intriguingly, the search process works better when using model-generated
plots. We attribute this result to the fact that the human annotations are not
designed for episode discriminability.

###  3.2 Evaluating CLIPCheck

DramaQA choi2021dramaqa is video QA dataset that focuses on story
understanding. The dataset is arranged with four levels of hierarchical
difficulty, which follow the human cognitive-developmental stages. We evaluate
LSS on the two high levels of DramaQA to test plot understanding. We report
two latest baselines in level-wise DramaQA; CharacterAttention and
(kim2021aaai).

We compare the effect of CLIPCheck and Caption, a prompt-based method of
incorporating image frame descriptions extracted from BLIP li2022blip as
inputs to GPT-3. Table LABEL:tab:dramaqa shows that CLIPCheck offers greater
improvement than image descriptions. Also, while adding image captions
improves LSS, the gain disappears when used jointly with CLIPCheck. We suspect
that this is because frame captions provide similar information to CLIPCheck
while being much noisier.

##  4 Related Work

Long video understanding is an important problem, but it has received less
attention than shorter video understanding tasks such as video captioning and
action recognition. There have been some attempts to address this problem,
such as the Video QA task proposed by , which requires a model to answer free-
form questions about long videos.

Movie summarization generate shorter version of a screenplay as the task of
finding an optimal graph chain of scene Gorinski2015MovieSS

Long Viddeo QA The task of video question answering has been studied
extensively in the literature. Several approaches have been proposed to
address this task, including deep learning based models such as

##  5 Conclusion

We introduced Long Story Short, a summarize-then-search method to understand
both global narrative and the relevant details for video narrative QA. Also,
we propose to post-check visual alignment with CLIPCheck. Our zero-shot method
improves supervised state-of-art approaches in MovieQA and DramaQA benchmarks.
We plan to release the code and the generated plot data to the public.

##  6 Limitations

Our study has some limitations, including:

  1. 1.

Experimenting with only data with English subtitles, The model can be further
augmented in the multi-lingual aspect.;

  2. 2.

Since our method employs capability of a large language model, power
consumption and computation resource are greatly required.

  3. 3.

Evaluating only with a single instance of LLM.

Potential Risk. Summarizing with GPT-3 carries some ethical risks related to
the open-ended nature of the language model. GPT-3 may (a) hallucinate fake
facts about the content, (b) generate toxic utterances, or (c) implicitly
embed social biases into the summary and the answer likelihoods.

##  Appendix A Experiment Details

Computational Budget. Long Story Short uses GPT-3 (175B parameters) via OpenAI
API as the backbone. An average prompt to summarize a video segment processes
$\sim 3000$ tokens, while a QA prompt usually takes $\sim 4000$ tokens. For
CLIPCheck, we extract CLIP features and compute the cosine similarity using a
single NVIDIA A6000 GPU: it takes 0.5 hours to process video frames for the
MovieQA validation split.

Hyperparameters. All hyperparameters are pre-defined by analyzing a single
training sample. For narrative search, we use sentence similarity threshold
$\alpha\geq 0.5$ to find plot pieces when GPT-3 does not output a single
index. We use the binary entropy threshold $E^{\prime}\geq 0.4$ in CLIPCheck.
We run each experiment only once, as our method is deterministic and is not
susceptible to randomness in initialization.

External Libraries. We use OpenAI API to access GPT-3 language model. The CLIP
features are computed with huggingface implementations
(https://huggingface.co/docs/transformers/main/en/model_doc/clip#transformers.CLIPFeatureExtractor).

##  Appendix B Prompt Samples

Screenplay to Plot. we prompt InstructGPT with “ I am a highly intelligent
storytelling bot. If you give me a script, I will give you the short synopsis
in detail. \n\n [Generated Screenplay] \n\n Synopsis: ”

Plot Index Lookup.

“ Plot: \n (1) [Plot1] \n (2) [Plot2] \n ... \n (N) [PlotN] \n "\n I am a
highly intelligent question answering bot. If you provide me with a question,
I will give you an index of the plot you should lookup to solve it.\n Q:
[Question] \n Top 1 Plot Index: ( ”

Answering Question.

“ Plot: \n (1) [Plot1] \n (2) [Plot2] \n ... \n (N) [PlotN] \n \n\n [Generated
Screenplay] \n\n "\n I am a highly intelligent plot question answering bot. If
you ask me a question and candidates, I will give you the index of answer.\n
Q: [Question] \n Candidates: \n (1): [Answer1] \n ... \n (5): [Answer5] \n A:
( ”


# Quantitative Stopword Generation for Sentiment Analysis via Recursive and
Iterative Deletion

Daniel M. DiPietro
<EMAIL_ADDRESS>

###### Abstract

Stopwords carry little semantic information and are often removed from text
data to reduce dataset size and improve machine learning model performance.
Consequently, researchers have sought to develop techniques for generating
effective stopword sets. Previous approaches have ranged from qualitative
techniques relying upon linguistic experts, to statistical approaches that
extract word importance using correlations or frequency-dependent metrics
computed on a corpus. We present a novel quantitative approach that employs
iterative and recursive feature deletion algorithms to see which words can be
deleted from a pre-trained transformer’s vocabulary with the least degradation
to its performance, specifically for the task of sentiment analysis.
Empirically, stopword lists generated via this approach drastically reduce
dataset size while negligibly impacting model performance, in one such example
shrinking the corpus by $28.4\%$ while improving the accuracy of a trained
logistic regression model by $0.25\%$. In another instance, the corpus was
shrunk by $63.7\%$ with a $2.8\%$ decrease in accuracy. These promising
results indicate that our approach can generate highly effective stopword sets
for specific NLP tasks.

##  1 Introduction

Stopwords are words that carry little semantic information and, as a result,
are often omitted when performing natural language processing (NLP) tasks.
Originally used to accelerate document retrieval and search engine algorithms,
stopwords are now commonly applied to NLP tasks involving machine learning,
such as model training Schütze et al. (2008). Removing stopwords can reduce
dataset size, which can lead to faster training on less sophisticated
hardware; removing stopwords can also improve model performance Kaur and
Buttar (2018); Dolamic and Savoy (2010).

Often, stopwords are generated via qualitative, intuitive approaches.
Researchers have also employed model-agnostic statistical approaches that
generate specific stopword sets for a given corpus of text. One early, naive
approach involves ranking all words in a given corpus by frequency and simply
omitting the top-$n$ most frequent words Schütze et al. (2008). However, more
sophisticated approaches have developed over time.

Hao and Hao (2008) developed a novel statistical technique for stopword
generation uses Chi-squared statistics to generate statistical correlations
between each word in a corpus and each document category. These correlations
can be further normalized based on each word’s frequency and used to generate
a corpus-specific list of stopwords. In practice, these stopword sets
decreased dataset size and often improved performance.

Ferilli (2021) employed a different approach, using the product of term
frequency and document frequency to form an “importance score” for each word
in the vocabulary; this was based on the idea that unimportant words appear
not only very frequently within documents, but also frequently across
documents. Based on these importance scores, they generated stopword lists
based on a user-specified tolerance; these lists closely matched lists
generated by linguistic experts.

Finally, Wilbur and Sirotkin (1992) used probabilistic metrics for stopword
identification. They computed how likely each word is to appear in document in
the corpus, conditioned on the document being a specific category. Then, based
on the distribution of words in the corpus at large, they computed how likely
the word is to appear across the corpus in general, positing that the most
semantically important words are more likely to appear in specific, relevant
classes of documents than they are to appear across the corpus in general.
Using this approach, they ranked each word and formed stopword sets; documents
in the same class clustered together similarly with and without stopwords
removed.

Thus far, all approaches discussed have relied solely upon “importance”
metrics extracted from a corpus and assigned to each word. However, modern
machine learning models often operate in unclear, complicated ways. In some
cases, they demonstrate an incredible understanding of language. Specifically,
transformers often know which words are important and which are not for their
specific NLP task via their attention mechanism Vaswani et al. (2017). Here,
we present a novel quantitative method for stopword generation that exploits
the performance of large-scale language models by using them as a tool for
stopword generation. Specifically, we employ two feature selection algorithms,
iterative deletion and recursive deletion, to see which stopwords can be
removed from a pre-trained DistilBERT transformer with the least degradation
to its area under the curve (AUC) performance on the SST2 sentiment benchmark.
We use these degradations as a proxy for word importance, crafting several
stopword lists. We then test the effectiveness of these stopword lists by
using them to train a sentiment analysis logistic regression model.

Our results indicate that stopwords lists generated via this approach can be
used to drastically reduce corpus size without hindering the performance of
trained models. Thus, our techniques can lead to smaller datasets that reduce
the computational burden of training machine learning models.

Figure 1: Logistic Regression (vectorized with TFIDF unigrams) performance
when trained on SST2 train with a given set of stopwords removed. Each
logistic regression model was then benchmarked on the SST2 validation dataset,
whereas the stopwords were generated from the SST2 test dataset.

##  2 Methodology

To determine the importance of each word for the task of sentiment analysis,
we begin with a pre-trained sentiment analysis transformer model. Then, we
remove words from the vocabulary of its test dataset using recursive and
iterative approaches, measuring the degradation of overall AUC as a proxy for
word importance. Stopword lists of length $n$ have the $n$ least important
words.

###  2.1 Pre-Trained Language Models and SST2

Our chosen transformer is a DistilBERT model that has been pre-trained on the
Stanford Sentiment Treebank v2 (SST2) training dataset Sanh et al. (2019). We
pair it with a DistilBERT auto-tokenizer, also pre-trained on SST2. DistilBERT
models offer impressive performance while being computationally efficient,
which is important for our task given its computational complexity. SST2 is a
sentiment analysis dataset widely used as a performance benchmark and freely
available from dozens of online repositories Socher et al. (2013). Its
training, test, and validation sets have $6920$, $1821$, and $872$ examples
respectively.

###  2.2 Generating a Vocabulary

To construct a vocabulary of unique tokens to rank, we remove punctuation from
the SST2 test dataset and split on spaces. We opt not to use the DistilBERT
auto-tokenizer for this task due to its uninterpretability. By defining the
vocabulary in this way, the tokens are clear, interpretable English words that
can be applied to any dataset and model.

###  2.3 Iterative Deletion

Suppose we have a corpus $\mathbf{X}$ and vocabulary
$\mathbf{V}=\\{v_{1},\dots,v_{n}\\}$. Let $\mathbf{X}\bigoplus v_{i}$ denote
the operation of removing token $v_{i}$ from the corpus $\mathbf{X}$.

Suppose we have a model $f$. Denote the AUC metric of model $f$ on corpus $X$
as $\text{AUC}(f,\mathbf{X})$.

To perform an iterative deletion, we generate the values
$\text{AUC}(f,\mathbf{X}\bigoplus v_{1})-\text{AUC}(f,\mathbf{X})$ for all
$v_{i}\in\mathbf{V}$, which is equal to the AUC degradation of model $f$ on
corpus $\mathbf{X}$ when token $v_{i}$ is removed. This metric can be used to
measure word importance for the task of sentiment analysis.

###  2.4 Recursive Deletion

Recursive deletions function somewhat similarly to iterative deletions. First,
we perform an iterative deletion with starting corpus $\mathbf{X}$ and
vocabulary $\mathbf{V}$. Suppose the least important word obtain from this
deletion is $v_{n_{1}}$. We denote $v_{n_{1}}$ as our first stopword. Then, we
perform another iterative deletion with starting corpus $\mathbf{X}\bigoplus
v_{n_{1}}$ and vocabulary $\mathbf{V}\backslash\\{v_{n_{1}}\\}$; the least
important word of this deletion is the second stopword. Generally, to obtain
the $i$-th stopword, we select the least important word from an iterative
deletion with corpus $\mathbf{X}\bigoplus v_{n_{1}}\bigoplus\dots\bigoplus
v_{n_{i-1}}$ and vocabulary
$\mathbf{V}\backslash\\{v_{n_{1}},\dots,v_{n_{i-1}}\\}$.

Recursive deletions assess how the importance of words changes based on what
words have been previously deleted, which is a useful characteristic for
stopword list generation. However, they are computationally expensive,
requiring an iterative deletion to select a single stopword.

###  2.5 AUC Degradation as an Importance Metric

We use area under the curve (AUC) degradation as our importance metric as AUC
is calculated based on label prediction probability rather than actual label
output of a model. Often, removing words may not necessarily change the output
label of a model, but it may lead to a decrease in prediction probability
performance. Thus, AUC as a metric captures a holistic view of word importance
compared to label-based metrics like accuracy or F1-score.

##  3 Results

Using iterative deletion with DistilBERT as defined above, we assigned word
importances to the $6862$ unique tokens found in SST2 test. The $10$ least
important tokens were $\\{$gets, artsy, having, just, under, sulky, bowel,
think, before, lrb$\\}$. Other notable tokens in the worst $25$ were
“affronted” and “disappointed.” The $10$ most important tokens were $\\{$not,
but, n’t, never, is, no, without, too, and, a$\\}$. Other notable important
tokens are “in” and “of” as the 14th and 21st most important tokens
respectively.

Using recursive deletion, we generated a list of the $1000$ least important
tokens in SST2, due to computational cost. The $10$ least important tokens
were $\\{$gets, artsy, having, sulky, just, bowel, think, before, skip,
lrb$\\}$. As we only ranked the $1000$ least important words, there is no
importance list.

Using our iterative and recursive word importance metrics, we generated
several stopword lists of varying size and composition (in some cases, also
incorporating the Sklearn default list of stopwords). Then, we trained a
logistic regression model (vectorized with unigrams and TFIDF) on SST2 train
with each stopword set omitted. We benchmarked the performance of these models
on SST2 validation; note that, since we used SST2 test to generate our
stopwords, there are no issues with bias. These results are depicted in Figure
1.

Additionally, as an important consequence of stopword sets is their ability to
reduce dataset size, we report the subsequent reduction in dataset size for
our training and testing datasets (SST2 test and SST2 validation respectively)
with each stopword set removed in Figure 2.

Figure 2: Dataset reduction to the logistic regression train (SST2 train) and
test (SST2 val) datasets when applying different sets of stopwords.

##  4 Discussion

Our stopword sets seem to perform quite well, reducing corpus size while
minimally impacting model performance. For example, the Recursive (1000) +
Sklearn stopword cut the net corpus size in half while only degrading accuracy
by $0.022$. In some cases, stopword sets even improved accuracy, such as the
Iterative (1000) stopword set, which increased accuracy from $0.787$ to
$0.791$ while reducing corpus size by $16\%$.

In practice, the iterative stopword sets often outperformed their recursive
counterparts both in dataset reduction and model performance (the $1000$ size
recursive set is an exception). As recursive stopword sets are measuring the
impact of word removal on the importance of remaining words, it’s possible
that the relationships between words found in the SST2 test dataset were
overfitted to in the stopword set and don’t map well to SST2 val.

Of particular note is that many of our stopwords are not what would commonly
be considered stopwords. Words like “artsy” and “disappointed” are among the
least important words, even though they’re somewhat nuanced, highly specific,
and seem to convey some sentiment information, at least to us. However, large-
scale language models work in complicated, blackbox-esque ways, and,
ultimately, it’s possible that these words just aren’t that important for a
sentiment task in the way that a transformer learns it. Indeed, their
appearance may be correlated with other words or certain syntactic structures
that offer better sentiment information.

Relatedly, many common stopwords, rather than appearing among the least
important words, actually appear among the most important words: some examples
are “is”, “a”, “in”, and “of.” This may seem counterintuitive, but it’s
important to note that our DistilBERT model was not trained with stopwords.
Even though these words don’t offer sentiment information, it’s possible that
they’re essential for providing the language syntax that the transformer was
trained to understand, which is why they may not be picked up by our approach.

Our work offers no major ethical issues: none of the stopwords are associated
with protected classes, and SST2 is a widely used and thoroughly vetted
dataset. However, our approach requires large sentiment analysis datasets,
which limits it to high-resource languages, and the carbon implications of
recursive deletion are notable, as these searches took approximately 48 hours
of GPU run-time.

##  5 Conclusion

Our approach offers a highly effective technique for generating stopword sets
for specific NLP tasks, which can reduce dataset size and allow for training
models on cheaper hardware. In the future, we can perform this process on
larger datasets, with more sophisticated transformer models, and for different
NLP tasks. Additionally, we can perform more sophisticated linguistic analyses
to better understand why some words were unintuitivately designated or not
designated as stopwords.

## References

  * Dolamic and Savoy (2010) Ljiljana Dolamic and Jacques Savoy. 2010.  When stopword lists make the difference.  _Journal of the American Society for Information Science and Technology_ , 61(1):200–203.
  * Ferilli (2021) Stefano Ferilli. 2021.  Automatic multilingual stopwords identification from very small corpora.  _Electronics_ , 10(17):2169.
  * Hao and Hao (2008) Lili Hao and Lizhu Hao. 2008.  Automatic identification of stop words in chinese text classification.  In _2008 International conference on computer science and software engineering_ , volume 1, pages 718–722. IEEE.
  * Kaur and Buttar (2018) Jashanjot Kaur and P Kaur Buttar. 2018.  A systematic review on stopword removal algorithms.  _International Journal on Future Revolution in Computer Science & Communication Engineering_, 4(4):207–210.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schütze et al. (2008) Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008.  _Introduction to information retrieval_ , volume 39.  Cambridge University Press Cambridge.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wilbur and Sirotkin (1992) W John Wilbur and Karl Sirotkin. 1992.  The automatic identification of stop words.  _Journal of information science_ , 18(1):45–55.


#  Overparameterization Improves StyleGAN Inversion

Yohan Poirier-Ginter⋄, Alexandre Lessard∙, Ryan Smith∙, Jean-François Lalonde⋄
⋄Université Laval, ∙Gearbox Software
https://lvsn.github.io/OverparamStyleGAN/

###### Abstract

Deep generative models like StyleGAN hold the promise of semantic image
editing: modifying images by their content, rather than their pixel values.
Unfortunately, working with arbitrary images requires inverting the StyleGAN
generator, which has remained challenging so far. Existing inversion
approaches obtain promising yet imperfect results, having to trade-off between
reconstruction quality and downstream editability. To improve quality, these
approaches must resort to various techniques that extend the model latent
space after training. Taking a step back, we observe that these methods
essentially all propose, in one way or another, to increase the number of free
parameters. This suggests that inversion might be difficult because it is
underconstrained. In this work, we address this directly and dramatically
overparameterize the latent space, before training, with simple changes to the
original StyleGAN architecture. Our overparameterization increases the
available degrees of freedom, which in turn facilitates inversion. We show
that this allows us to obtain near-perfect image reconstruction without the
need for encoders nor for altering the latent space after training. Our
approach also retains editability, which we demonstrate by realistically
interpolating between images.

Figure 1:  Overparameterizing the StyleGAN latent space during training
improves inversion quality. We obtain near-perfect reconstructions (“Inversion
A” and “Inversion B”) of corresponding arbitrary real images (“Target A” and
“Target B”), without having to sacrifice editability, as exemplified by
interpolating between reconstructions (middle columns).

##  1 Introduction

Deep image generation models [17, 6] “fill the gaps” between images of a set,
making it possible to sample new, previously unseen images from the same
domain. They also make possible the indirect manipulation of images via latent
codes, rather than raw pixel values. In this space, StyleGAN [29, 30, 27, 28]
is a popular image generation model specifically because its choice of latent
representation is auspicious for editing. Its latent code is mapped to style
vectors which affect entire channels of feature maps, resulting in a spatially
invariant parameterization. This learned mapping—from latent to style—is also
thought to help with disentanglement [29], another necessity for image editing
applications. In many ways, StyleGAN comes close to finally making possible
the ambitious goal of semantic image editing. Unfortunately, there remains a
missing piece: while edits work excellently for randomly sampled images, in
its original formulation StyleGAN struggles with inversion, that is, reversing
the generation process and finding a latent code that produces a real image of
our choosing [53]. Without this piece, StyleGAN is effectively only useful as
a random image sampler.

There have been many attempts at improving StyleGAN inversion. Here, there are
two competing objectives at play: first, the reconstructed image must match
the target as closely as possible (reconstruction quality); second, the
obtained latent codes must be usable in downstream tasks (editability) [1,
46]. However, this second objective is not always met in practice: there
exists multiple different latent codes that produce the same output image, and
many of them produce unrealistic images when modified (_i.e_., they do not
interpolate well with others) [1, 54, 59, 52].

Approaches to inversion can broadly be divided into two families: optimization
and encoding. [53] Optimization finds latent codes by gradient descent, while
encoding uses a trained network, called an encoder, to directly regress the
latent code from an input image. It is generally considered that optimization
reaches better reconstruction quality, while encoders provide better
editability [59, 50]. In both cases, targeting StyleGAN’s intermediate latent
space $\mathcal{W}$ (the input of the synthesis network) works poorly: the
obtained latent codes edit fairly well, but the reconstruction error is high
[30]. This motivates different attempts at relaxing the target distribution.
In particular, the extension to $\mathcal{W+}$ uses a different latent code
for each layer. This change is a key component of most inversion pipelines
[41, 15, 49, 4, 57] as it improves reconstruction quality tremendously. But it
comes at a price: because the latent space is altered after training,
editability suffers. Hence, many approaches attempt to restrict this
extension, trading-off between reconstruction quality and editability [50, 37,
59, 52].

In this work, we propose a direct solution that avoids modifying the latent
space after training: we dramatically overparameterize it beforehand. It turns
out that large changes to the generator architecture are not required for this
approach to train well, which we will show. We will then demonstrate how
reconstruction quality is greatly improved following overparameterization.
During inversion, our latent space is left as-is, which avoids the
reconstruction quality/editability trade-off. Our optimization method only
requires minimal regularization to achieve editability. In short,
overparameterization of the latent space improves StyleGAN inversion.

##  2 Previous work

#### Optimization

was the first approach to StyleGAN inversion [1, 2]. As compared to encoders,
it does not require additional training and can dynamically alter its loss
function. StyleGAN2’s [30] inversion process obtained good editability but
inadequate reconstruction quality, by targetting $\mathcal{W}$. Image2StyleGAN
[1] first introduced the extension of $\mathcal{W}$ to $\mathcal{W+}$111In
this work, we will usually refer to the elements of these spaces directly,
i.e. vectors $\boldsymbol{w}\in\mathcal{W}$ and tuples of vectors
$\boldsymbol{w}^{+}\in\mathcal{W}^{+}$., greatly improving quality at the cost
of editability. Some authors also optimize the noise inputs [30, 2], the
feature maps [25, 12, 10, 32], or even newly introduced transformations [24,
25], usually in a second pass. Since then many papers propose regularization
methods to improve editability following inversion [37, 52, 59, 58]. Finally,
in Pivotal Tuning [44], generator parameters are optimized along with latent
codes, also in a second pass. In general, optimization obtains better
reconstruction quality than encoders do, at least when targeting similar
spaces. In this work, we focus on optimization, as it is the more flexible
approach. We will also show that complex regularization is not required for
retaining editability.

#### Encoders

directly learn a mapping between images and latent codes, and so inference
takes milliseconds instead of minutes as with optimization. Typically, the
encoder is trained after the fact, using a pretrained generator, although
there have been attempts at training both jointly [21]. Most encoders employ a
multi-level architecture, mirroring that of the generator, to target
$\mathcal{W+}$ [50, 43, 55], although some differ [40]. Overall, similar
editability concerns apply, and so some papers regularize the encoder to stay
close to $\mathcal{W}$ [50]. Further works improve quality through various
techniques [3, 51, 5, 19]. In hybrid approaches, encoder initialization helps
retain editability following an optimization-based refinement [58]. In
general, it is thought that encoders obtain better editability than
optimization, at the cost of flexibility. In this work, we argue that encoders
are not strictly required for good editability.

#### Hypernetworks

predict the parameters of other networks [20]; for StyleGAN specifically
Hyperstyle [5] and [13] use them to predict parameters offsets which fine-tune
a generator, much like in Pivotal Tuning [44] except with an encoder. This
approach is similar to our overparameterization, but, unlike us, they both
target a pretrained StyleGAN generator while we retrain the generator
following overparameterization. As such their latent space is different
between training and inversion while ours remains as it was. Also similar to
our work is AdaConv [9] for style transfer, but our modified style injection
stays much closer to the original StyleGAN and they do not address inversion.

In summary, the vast majority of previous work aiming to improve
reconstruction quality shares a point in common: it does so by increasing the
number of targeted parameters in one way or another. Here, we take a more
straightforward approach and instead, overparameterize the latent space
directly.

##  3 Method

In this section, we show how the StyleGAN latent space can be
overparameterized effectively. Altogether, these changes increase the latent
dimensionality 512 times while avoiding redundant parameterization222We mean
by this that overparameterization should gain additional degrees of freedom,
i.e., naively increasing StyleGAN’s latent size would not work as the
projection layer would reduce its dimensionality back to a smaller size. ,
excessive computational costs, or sweeping architectural changes.

###  3.1 Overparameterized training

Figure 2: Modified style injection following overparameterization. StyleGAN2
first passes a single latent vector through mapping and affine projection
layers. It then replicates the result row-wise before multiplication with the
convolutional filters (top row). Instead, our approach instead maps and
projects 512 vectors independently into a matrix of the same shape (bottom
row).

#### StyleGAN2/3 style modulation

For each layer $l$ of an $L$-layer network, StyleGAN2/3 style injection (fig.
2, top) modulates columns of the convolutional filter weights
$\mathbf{\Theta}^{(l)}\in\mathbb{R}^{N_{O}^{(l)}\times N_{I}^{(l)}\times
k^{(l)}\times k^{(l)}}$ with components of a style vector
$\boldsymbol{s}\in\mathbb{R}^{N_{I}^{(l)}}$, replicating across other
dimensions:

$\mathbf{\Theta^{\prime}}^{(l)}_{:,j,:,:}=\mathbf{\Theta}^{(l)}_{:,j,:,:}\cdot\boldsymbol{s}^{(l)}_{j}\,.$
(1)

Here, $N_{I}^{(l)}$, $N_{O}^{(l)}$, and $k^{(l)}$ refer to the number of input
channels, output channels, and the kernel size, respectively, of layer $l$.
The style vectors
$\boldsymbol{s}^{(l)}=\mathcal{A}^{(l)}(\mathcal{M}(\boldsymbol{z}))$ are
obtained by passing $\boldsymbol{z}\in\mathbb{R}^{512}\sim\mathcal{N}(0,I)$
through a shared mapping network
$\mathcal{M}:\mathbb{R}^{512}\rightarrow\mathbb{R}^{512}$ followed by layer-
specific affine projections
$\mathcal{A}^{(l)}:\mathbb{R}^{512}\rightarrow\mathbb{R}^{N_{I}^{(l)}}$. In
particular, the same style vector is used for all rows of a layer convolution
filter weights.

#### Overparameterized style modulation

In contrast, our approach samples a _different style for each row_ , as
depicted in fig. 2 (bottom). Grouping 512 different latent vectors into the
rows of $\mathbf{Z}\in\mathbb{R}^{512\times 512}$, we map and project each of
them independently into a matrix
$\mathbf{S}^{(l)}\in\mathbb{R}^{N_{o}^{(l)}\times N_{i}^{(l)}}$, where
$\mathbf{S}^{(l)}_{i}=\mathcal{A}^{(l)}(\mathcal{M}(\mathbf{Z}_{i}))$333When
$N_{O}^{(l)}<512$, we take the easy way out and drop rows.. Style modulation
is unchanged except that it now also varies row-by-row:

$\mathbf{\Theta^{\prime}}^{(l)}_{i,j,:,:}=\mathbf{\Theta}^{(l)}_{i,j,:,:}\cdot\mathbf{S}_{i,j}^{(l)}\,.$
(2)

For training to be effective, we found it essential for latent codes to share
some correlation. From a shared vector
$\boldsymbol{\tilde{z}}\sim\mathcal{N}(0,I)$ and 512 i.i.d. vectors
$\boldsymbol{\tilde{z}}^{(i)}\sim\mathcal{N}(0,I)$, we set
$\mathbf{Z}_{i}=(\boldsymbol{\tilde{z}}^{(i)}+\boldsymbol{\tilde{z}})/\sqrt{2}$.
With this approach, each latent $\mathbf{Z}_{i}$ is still a unit Gaussian. As
such, the distribution of the mapper’s output $\mathcal{W}$ is unique; each
row of $\mathbf{Z}$ is identically distributed, but non-independent. We will
show in sec. 5.2 that this allows compatibility with existing editing
approaches. As an added benefit, calls to the mapping and projection layers
can vectorized, reducing computational costs (sec. 6.1).

###  3.2 Overparameterized inversion

Figure 3:  Targeted latent space during inversion. To obtain reconstructions
of good quality, (left) StyleGAN must modify the latent space after training,
and target different latent codes for each layer. (right) In our
overparameterized network, we simply target the matrix $\mathbf{W}$, and
inject it just like during training.

#### StyleGAN inversion

As described in sec. 2, naive StyleGAN inversion targets intermediate latent
codes $\boldsymbol{w}\in\mathcal{W}$, ignoring the mapping network. It
attempts to reconstruct a latent code $\boldsymbol{w}$ which generates a
target image $\boldsymbol{y}$ by iteratively solving for

$\operatorname*{arg\,min}_{\boldsymbol{w}}\mathcal{L}(\mathcal{S}(\boldsymbol{w},...,\boldsymbol{w}),\boldsymbol{y})\,,$
(3)

where $\mathcal{S}$ is the synthesis network (a function which accepts one
latent code per layer and outputs an image), $\mathcal{L}$ is a loss function
(typically comprised of a perceptual distance [56] and sometimes other terms)
and $\boldsymbol{w}$ is initialized to
$\mu_{\mathcal{W}}=\mathbb{E}_{\boldsymbol{z}}[\mathcal{M}(\boldsymbol{z})]$,
the mean in $\mathcal{W}$. Unfortunately, this approach performs poorly and
fails to match the target. Empirically, it was found that optimizing a
separate latent code for each layer, _i.e_.,

$\operatorname*{arg\,min}_{\boldsymbol{w}^{+}=(\boldsymbol{w}^{(1)},...,\boldsymbol{w}^{(L)})}\mathcal{L}(\mathcal{S}(\boldsymbol{w}^{(1)},...,\boldsymbol{w}^{(L)}),\boldsymbol{y})\,,$
(4)

obtains much better reconstruction quality [1] (see fig. 3, left). However,
this change also modifies the latent space after training, which in turn
damages editability444Avoiding this by training with i.i.d. vectors for each
layer leads to a degradation of performance, motivating other solutions [34].
At the same time, it is desirable to even further increase the number of
parameters, justifying our final approach..

#### Overparameterized inversion

In our case, we sample a latent matrix $\mathbf{W}$ (fig. 3, right), where the
rows are correlated samples from $\mathcal{W}$, as described previously. An
analogous extension to
$\mathbf{W}^{+}=(\mathbf{W}^{(1)},...,\mathbf{W}^{(L)})$ is not required, as
$\mathbf{W}$ already contains enough parameters for a fit of good quality.
Initializing each row of $\mathbf{W}$ to $\mu_{\mathcal{W}}$, we can directly
solve for

$\min_{\mathbf{W}}\mathcal{L}(\mathcal{S}(\mathbf{W},...,\mathbf{W}),\boldsymbol{y})\,,$
(5)

where $\mathbf{W}$ is shared between layers as during training. The matrix
$\mathbf{W}$ contains $\nicefrac{{512}}{{L}}$ more parameters than
$\boldsymbol{w}^{+}$, which greatly improves reconstruction. We will
demonstrate this in the following section, while also showing that editabiliy
is retained.

##  4 Experiments

This section will first show how generation quality is mostly retained even
with our (admittedly quite drastic) overparameterization. Then, we will show
how inversion is improved: reconstruction of much better quality is obtained
while editability is preserved.

###  4.1 Generation quality

We train stylegan2-ada-pytorch[27] at a resolution of $256\times 256$ on the
FFHQ dataset [29]. Starting from the default configuration, we modernize the
architecture following StyleGAN3 [28]: we omit the noise inputs (they are a
hindrance in image editing applications) but retain the perceptual path length
(PPL) regularizer, which improves editability and inversion quality [30]. We
also keep the style mixing regularizer, as style mixing has use in many
downstream tasks. Finally, we reduce the number of mapping layers from 8 to 2,
which was shown to improve FID [28]. We do not specialize hyperparameters for
our method, nor do we specialize them for the dataset555We performed all
trainings using the default configuration, rather than the configuration
specialized for FFHQ.. The same configuration is used for training both the
baseline and our method.

Figure 4: Selected generated images sampled from our overparameterized model,
with a resolution of $256\times 256$. Each image has $512\times 512$ latent
parameters, more than the number of pixels.

We measure the generation quality using the standard Frechet Inception
Distance [23] (FID) for generation quality and the Perceptual Path Length [30]
(PPL) for disentanglement. After completing training, we retain the checkpoint
with the lowest FID, comparing 50k generated images to 50k images of the
training set. Note that comparing the PPL between two approaches with
different latent space dimensionality is valid, as the PPL is an image-space
metric despite measuring sensitivity to latent changes.

|  FID↓ (Realism) |  PPL↓ (Disent.)
---|---|---
Baseline | 3.88 | 196.27
Overparam | 5.05 | 203.67
Table 1: Generation quality on the FFHQ dataset. FID (quality) at the lowest
point during training and corresponding PPL (disentanglement), for both
methods.

Table 1 shows that the FID score does quite not match the baseline; there is
some degradation in generation quality. Nevertheless, most images are still
very realistic, as shown in fig. 4. Table 1 also shows that disentanglement is
essentially unchanged, which is important for downstream tasks.

###  4.2 Reconstruction quality

Figure 5: Reconstruction quality after inversion for selected examples. We
compare inversion targeting $\boldsymbol{w}+$ for the baseline (first column)
to inversion targeting $\mathbf{W}$ and its extension $\mathbf{W}^{+}$ for our
method (second and third row, respectively). Inlays report the LPIPS metric
between the reconstruction and target.

We measure reconstruction quality using optimization, on 100 images of the
CelebA-HQ dataset666GANs metrics are typically reported after training on
entire datasets (FFHQ in our case). As such, we evaluate on a separate,
similar dataset (CelebA-HQ). [26, 35]. We optimize for 1000 steps with the
Adam optimizer, using settings $\eta=0.05,\beta_{1}=0.9,\beta_{2}=0.999$, and
the LPIPS loss from StyleGAN2. We compare the baseline $\boldsymbol{w}$ and
its extended version $\boldsymbol{w}^{+}$, to our overparameterized
$\mathbf{W}$ and its extended version $\mathbf{W}^{+}$. We regularize with a
$0.9$ truncation [29] towards $\mu_{\mathcal{W}}$ before each optimization
step, disabled halfway.777Hybrid approaches edit well following encoder
initialization. We essentially replace encoder initialization with regularized
initialization. As table 2 shows, overparameterization improves reconstruction
quality by large amounts, even without extension to layer-specific codes. That
being said, such an extension improves quality even more. Fig. 5 shows
examples of our method recovering all important details of the target image,
even for harder instances. In addition, fig. 6 shows that our method converges
faster and obtains great reconstructions with high reliably.

Figure 6: Median reconstruction quality (LPIPS) and quantiles measured over 100 CelebA-HQ images, when targeting different spaces. The dotted line shows when regularization was disabled. |  LPIPS↓ (Distort.) |  FID↓ (Realism)
---|---|---
Baseline $\boldsymbol{w}^{+}$ | 0.185 | 77.16
Overparam $\mathbf{W}$ | 0.070 | 73.52
Overparam $\mathbf{W}^{+}$ | 0.033 | 74.19
Table 2: Reconstruction quality on 100 images of the CelebA-HQ dataset. We
compare the baseline (first row) to our overparameterization, with and without
extension to layer-specific codes (third and second rows, respectively).

###  4.3 Real image interpolation

Because image editing can be achieved using linear combinations of latent
vectors [46], as is the case with interpolation, we use interpolation quality
on inverted images as a proxy measure for editability [50]. First, we generate
interpolations between all pairs of our inverted images from sec. 4.2,
yielding 4,950 images. We then report the FID of the midpoint image
distribution, compared to the distribution of the full CelebA dataset, and the
perceptual path length over the entire interpolation sequence888More
precisely, we divide each interpolation path into 5 segments and sum the total
(perceptual) variation, averaging over all images..

Figure 7: Interpolation quality for selected examples. We compare inversion
targeting $\boldsymbol{w}+$ for the baseline (first row) to inversion
targeting $\mathbf{W}$ for our method (second row). Inlays report the LPIPS
metric between the reconstruction and target. Extension to $\mathbf{W}^{+}$
produces interpolations with more artifacts (third row).

Table 3 shows that overparameterization does not compromise editability
significantly despite reaching much better reconstruction quality, as shown
previously. Unsurprisingly, extending the latent space after training gives
worse editability. Our inversion better matches the targets, which helps the
midpoint image retain characteristics of both endpoints (fig. 7). However,
this does not appear to be reflected the PPL, which remains relatively
constant, albeit slightly better for the baseline. Refer to the supplemental
for additional qualitative results on image interpolation.

|  FID↓ (Realism) |  PPL↓ (Disent.)
---|---|---
Baseline $\boldsymbol{w}^{+}$ | 52.97 | 0.175
Overparam $\mathbf{W}$ | 53.51 | 0.190
Overparam $\mathbf{W}^{+}$ | 57.60 | 0.188
Table 3: Interpolation quality on all pairs from 100 images of the CelebA-HQ
dataset. We compare the baseline (first row) to our overparameterization, with
and without extension to layer-specific codes (second and third rows,
respectively).

##  5 Compatibility with downstream tasks

In this section, we briefly demonstrate the compatibility of our modified
latent space in existing downstream tasks; this section intends to show that
previous methods still work.

###  5.1 Generator-driven upsampling (PULSE)

In PULSE [37], optimization searches for the closest match after the generated
image is also downsampled. Such a problem is underconstrained and requires
more regularization; we use the same optimization process but keep the
truncation regularizer throughout. We show preliminary results with $16\times$
nearest-neighbour downsampling.

Figure 8: Generator-driven image upsampling. The optimization searches for an
image that matches the low-resolution target on the left, once also
downsampled. Inlays give the LPIPS between the downsampled targets and the
downsampled reconstructions.

Fig. 8 show that our method obtains great results, which match the low-
resolution target more closely than the baseline. While the extension of our
method to layer-specific codes still obtains an even better fit, the faces
seem slightly less realistic; please also refer to fig. 10 for additional
results, which also shows the original (unknown) full-resolution target. As
described previously in sec. 4.2, our method also converges to adequate
results much faster.

###  5.2 Unsupervised semantic editing (GANSpace)

Our method replaces a single vector $\boldsymbol{w}\in\mathcal{W}$ with 512
vectors $\mathbf{W}_{i}\in\mathcal{W}$. As described in sec. 3.1, these
vectors are correlated samples, but the $\mathcal{W}$ distribution remains
unchanged. Hence, our approach is still compatible with unsupervised editing
techniques such as GANSpace [22] (fig. 9), which extracts the principal
components of $\mathcal{W}$. Please refer to fig. 10 for more results.

Figure 9: GANSpace editing is compatible with our overparameterized model.
Latent translation of random generated images (row 1) along the fourth
principal component (row 2), corresponding to an interesting combination of
age, head rotation, and gender. Figure 10: Top Left: Random interpolation
midpoints for various methods, showing that 1) our approach is competitive
with the baseline 2) extension to $\mathbf{W}^{+}$ damages interpolation
quality. Bottom Left: Additional results for PULSE upsampling for random
samples from the CelebA dataset, which also give full-resolution target.
Bottom Right: Top 7 principal components for GANSpace in our latent space.

##  6 Limitations

Overparameterizing the latent space to such an extent, while simple
conceptually, can still be considered a considerable modification to the
baseline model. This section investigates potential limitations associated
with these changes.

###  6.1 Computational efficiency

|  Memory (Gb)↓ |  Speed (sec/kimg)↓
---|---|---
Baseline | 3.90 | 8.33
Overparam | 4.90 | 8.80
Table 4: Memory usage and speed when training on four NVIDIA A100 cards, at
resolution $256\times 256$. Despite increased memory consumption, wall-clock
time remains within 10% of the baseline.

Our revised style injection mechanism may seem inefficient as it requires 512
calls to the mapping and affine projection layers. It is true that memory
consumption increases, by approximately 25% during training, as shown in table
4. However, actual run-time remains competitive despite an increased number of
operations, which we attribute to cache-efficiency (these calls are easy to
vectorize)999Note that in StyleGAN2 specifically, the baseline could be
accelerated by training without fused modulation, which accelerates the PPL
regularizer’s gradient computation. As our approach is incompatible with
unfused training, we did not train with this optimization. Besides, it is also
omitted in StyleGAN3..

###  6.2 Non-injectivity and non-determinism

Figure 11: Overparameterization leads to a greater variety of latent codes
when optimization is unregularized and initialized randomly. Interpolation
between different codes for the same target images (left) in an
overparameterized space yields more variable solutions (right) as compared to
the baseline (middle).

StyleGAN is non-injective, meaning that multiple latent codes generate the
same image [53]. As it drastically increases the latent dimensionality, our
approach exacerbates this effect. In fig. 11, we invert the same targets
multiple times without regularization, after initializing to random points,
and show that the interpolated images exhibit much more variance. The exact
impact of this change on image editing using latent directions [46] remains to
be explored.

##  7 Discussion

In the larger context of deep learning, it is well known that optimization is
often surprisingly easy in high dimensional, non-linear spaces; local minima
are not a problem in practice even when nothing guarantees their absence [18,
11]. One could argue that the success of the field was partly based on this
empirical observation. While the exact underpinnings of this effect are still
being clarified [48, 31], this observation is well-accepted and has shown to
hold time and time again in practice. Perhaps even more surprising is the fact
that these strongly overparameterized networks still generalize well [39, 8].
Remarkably, it has been shown that such networks do not overfit, provided that
training is sufficiently long. This phenomenon was called ”double descent”
[38] and goes against classical statistical intuition which would predict the
opposite effect. In fact, the best-performing models are almost always grossly
overparameterized [7, 36, 47].

While our work addresses overparameterizing the latent space of a network,
rather than its parameter space, there are certainly interesting parallels to
be drawn. To us, at least, it is surprising that training is still possible
following these changes, and that the obtained latent vectors are still
useful. It is thought that in generative modeling, the generator’s intrinsic
dimensionality should ideally match that of the real image manifold [45, 33].
While the latter is hard to calculate [14], it has been estimated to values as
low as 20–50 [16, 42]. Yet, training remains effective with our seemingly
excessive overparameterization. Our work shows once again that intuition
obtained from low-dimensional statistical models can be misleading.

##  8 Conclusion

We propose overparameterization as an effective strategy for addressing
StyleGAN inversion and show how it can be effectively implemented in practice.
Our method trains efficiently and stably, and inverts well, finally achieving
near-zero reconstruction error without compromising editability. The crux of
our approach is that it avoids post-training extensions to the latent space,
which trade-off between these two objectives. Finally, we demonstrate
compatibility with existing downstream tasks. While our approach does not
require encoders for great editability, and while it improves convergence
speed, optimization is still too slow for real-time applications. We leave the
design of an encoder architecture that targets our overparameterized latent
space as future work.

## Acknowledgements

This research was supported by NSERC grants CRDPJ 537961-18 and
RGPIN-2020-04799, and by Compute Canada.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?, 2019\.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?, 2020.
  * [3] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement, 2021\.
  * [4] Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski, and Daniel Cohen-Or.  Third time’s the charm? image and video editing with stylegan3.  arXiv preprint arXiv:2201.13433, 2022.
  * [5] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing, 2021.
  * [6] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [8] Sébastien Bubeck and Mark Sellke.  A universal law of robustness via isoperimetry.  Advances in Neural Information Processing Systems, 34, 2021.
  * [9] P. Chandran, G. Zoss, P. Gotardo, M. Gross, and D. Bradley.  Adaptive convolutions for structure-aware style transfer.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7968–7977, Los Alamitos, CA, USA, jun 2021. IEEE Computer Society.
  * [10] Min Jin Chong, Hsin-Ying Lee, and David Forsyth.  Stylegan of all trades: Image manipulation with only pretrained stylegan, 2021.
  * [11] Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun.  The loss surfaces of multilayer networks.  In Artificial intelligence and statistics, pages 192–204. PMLR, 2015.
  * [12] Giannis Daras, Joseph Dean, Ajil Jalal, and Alexandros G. Dimakis.  Intermediate layer optimization for inverse problems using deep generative models, 2021.
  * [13] Tan M. Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua.  Hyperinverter: Improving stylegan inversion via hypernetwork, 2021.
  * [14] Ruili Feng, Deli Zhao, and Zhengjun Zha.  On noise injection in generative adversarial networks.  arXiv preprint arXiv:2006.05891, 2020.
  * [15] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  arXiv preprint arXiv:2108.00946, 2021.
  * [16] Sixue Gong, Vishnu Naresh Boddeti, and Anil K. Jain.  On the intrinsic dimensionality of image representations.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3982–3991, 2019.
  * [17] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [18] Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe.  Qualitatively characterizing neural network optimization problems.  arXiv preprint arXiv:1412.6544, 2014.
  * [19] Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang.  Collaborative learning for faster stylegan embedding, 2020.
  * [20] David Ha, Andrew Dai, and Quoc V. Le.  Hypernetworks, 2016.
  * [21] Ligong Han, Sri Harsha Musunuri, Martin Renqiang Min, Ruijiang Gao, Yu Tian, and Dimitris Metaxas.  Ae-stylegan: Improved training of style-based auto-encoders.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3134–3143, 2022.
  * [22] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  Advances in Neural Information Processing Systems, 33:9841–9850, 2020.
  * [23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [24] Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, and Aaron Hertzmann.  Transforming and projecting images into class-conditional generative networks, 2020.
  * [25] Kyoungkook Kang, Seongtae Kim, and Sunghyun Cho.  Gan inversion for out-of-range images with geometric transformations, 2021\.
  * [26] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [27] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data, 2020.
  * [28] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks, 2021.
  * [29] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [30] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan, 2020.
  * [31] Kenji Kawaguchi.  Deep learning without poor local minima.  Advances in neural information processing systems, 29, 2016.
  * [32] Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh.  Exploiting spatial dimensions of latent in gan for real-time image editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 852–861, 2021.
  * [33] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  Advances in Neural Information Processing Systems, 32, 2019.
  * [34] Heyi Li, Jinlong Liu, Yunzhi Bai, Huayan Wang, and Klaus Mueller.  Transforming the latent space of stylegan for real face editing.  arXiv preprint arXiv:2105.14230, 2021.
  * [35] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [36] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  arXiv preprint arXiv:2201.03545, 2022.
  * [37] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin.  Pulse: Self-supervised photo upsampling via latent space exploration of generative models, 2020.
  * [38] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.  Deep double descent: Where bigger models and more data hurt.  Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.
  * [39] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro.  Towards understanding the role of over-parametrization in generalization of neural networks.  arXiv preprint arXiv:1805.12076, 2018.
  * [40] Yotam Nitzan, Amit Bermano, Yangyan Li, and Daniel Cohen-Or.  Face identity disentanglement via latent space mapping.  arXiv preprint arXiv:2005.07728, 2020.
  * [41] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085–2094, 2021.
  * [42] Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein.  The intrinsic dimension of images and its impact on learning.  arXiv preprint arXiv:2104.08894, 2021.
  * [43] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation, 2021\.
  * [44] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  arXiv preprint arXiv:2106.05744, 2021.
  * [45] Lars Ruthotto and Eldad Haber.  An introduction to deep generative modeling.  GAMM-Mitteilungen, 44(2):e202100008, 2021.
  * [46] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [47] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  arXiv preprint arXiv:1909.08053, 2019.
  * [48] Grzegorz Swirszcz, Wojciech Marian Czarnecki, and Razvan Pascanu.  Local minima in training of neural networks.  arXiv preprint arXiv:1611.06310, 2016.
  * [49] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt.  Stylerig: Rigging stylegan for 3d control over portrait images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6142–6151, 2020.
  * [50] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation, 2021.
  * [51] Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Weiming Zhang, Lu Yuan, Gang Hua, and Nenghai Yu.  A simple baseline for stylegan inversion.  arXiv preprint arXiv:2104.07661, 2021.
  * [52] Jonas Wulff and Antonio Torralba.  Improving inversion and generation diversity in stylegan using a gaussianized latent space, 2020.
  * [53] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  Gan inversion: A survey, 2021.
  * [54] Yangyang Xu, Yong Du, Wenpeng Xiao, Xuemiao Xu, and Shengfeng He.  From continuity to editability: Inverting gans with consecutive images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13910–13918, 2021.
  * [55] Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou.  Generative hierarchical features from synthesizing images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4432–4442, 2021.
  * [56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [57] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler.  Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering.  arXiv preprint arXiv:2010.09125, 2020.
  * [58] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing, 2020.
  * [59] Peihao Zhu, Rameen Abdal, Yipeng Qin, John Femiani, and Peter Wonka.  Improved stylegan embedding: Where are the good latents?, 2021.

##  Appendix A Supplementary Material

Figure 12: Additional interpolation results, comparing the baseline
($\boldsymbol{w}^{+}$, first row of each group) to our final approach
($\mathbf{W}$, second row of each group). Our method obtains much better
reconstruction quality (”Inversion A/B” better match ”Target A/B”), without
compromising interpolation quality (the middle columns second row images are
as realistic as the first row baseline). Note robustness to an out-of-sample
image (vintage black and white portrait) in the last group.