# Fault-Aware Neural Code Rankers

Jeevana Priya Inala &Chenglong Wang &Mei Yang &Andres Codas Mark Encarnación
&Shuvendu K Lahiri &Madanlal Musuvathi &Jianfeng Gao &
Microsoft Research
{jinala,chenwang,meiyang,andres.codas,markenc,
<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) have demonstrated an impressive ability to
generate code for various programming tasks. In many instances, LLMs can
generate a correct program for a task when given numerous trials.
Consequently, a recent trend is to do large scale sampling of programs using a
model and then filtering/ranking the programs based on the program execution
on a small number of known unit tests to select one candidate solution.
However, these approaches assume that the unit tests are given and assume the
ability to safely execute the generated programs (which can do arbitrary
dangerous operations such as file manipulations). Both of the above
assumptions are impractical in real-world software development. In this paper,
we propose CodeRanker, a neural ranker that can predict the correctness of a
sampled program without executing it. Our CodeRanker is fault-aware i.e., it
is trained to predict different kinds of execution information such as
predicting the exact compile/runtime error type (e.g., an IndexError or a
TypeError). We show that CodeRanker can significantly increase the pass@1
accuracy of various code generation models (including Codex chen2021evaluating
, GPT-Neo, GPT-J) on APPS hendrycks2021measuring , HumanEval
chen2021evaluating  and MBPP austin2021program  datasets.

##  1 Introduction

Large transformer-based language models (LLMs) have impressive capabilities
devlin2018bert ; brown2020language ; chowdhery2022palm , including the ability
to generate code chen2021evaluating ; austin2021program ; li2022competition ;
xu2022systematic ; nijkamp2022conversational . The task here is to take a
natural language description or previous code context as the input and
generate an entire program in a general-purpose programming language such as
Python or C++. Generating entire programs is a challenging task as it involves
understanding the task, figuring out how to accomplish it, and writing code
without any syntax/runtime errors. On harder programming problems such as
coding competition problems, current models achieve very low accuracy
especially if the inference-time sampling budget is low. For example, on the
APPS dataset hendrycks2021measuring , a state-of-the-art code generation
model, Codex chen2021evaluating , achieves 4% accuracy if it is allowed to
sample only one program per task (called pass@1), but achieves 24% accuracy if
it is allowed 100 samples per task (called pass@100—at least one correct
program in 100 samples) (see Table 7). This observation leads us to an
important research problem on exploring approaches to rank the sampled
programs to bridge the gap between the pass@1 and pass@100 performances.

Upon analyzing the sampled programs obtained using an LLM, we find that
several of them have syntax errors and runtime errors, and some of them
execute to produce undesirable outputs. Therefore, prior works
chen2021evaluating ; li2022competition  focused on ranking the programs by
executing them on a small set of unit tests (which are typically assumed to be
given as part of the task description). However, there are several caveats to
this approach: First, even if a program passes the given tests, it could still
fail on the unknown unit tests. Second, there is a burden on the user to
provide unit tests for every inference task. Third, to execute the code, all
the dependencies has to be installed properly. This is especially problematic
in scenarios where a user wants to get code suggestions in a project with
multiple files and dependencies (such as using CoPilot in a VS Code
environment copilot ). Even if these dependencies are satisfied, many
realistic programming scenarios involve incomplete code under active
development where execution is just infeasible. Finally, the code generated by
a LLM could potentially have security risks (such as deleting files on the
disk) and hence, executing such a code needs heavy-weight isolation mechanisms
such as a sandbox environment.

To alleviate the above issues with relying on executing code, some of the
recent works cobbe2021training ; shen2021generate  proposed to use a neural-
network based ranker111called a verifier in the previous work for ranking
programs sampled from a LLM. A ranker model is essentially a classifier that
takes as input a task description and a sampled program and predicts the
probability that the program is correct with respect to the task description.
When given multiple programs (obtained by sampling), a ranker re-orders them
in the decreasing order of the predicted probability of the program being
correct. The ranker is essentially trying to emulate executing a program on
some unit tests without actually executing the code during inference. The
training data is obtained by executing both correct and wrong programs sampled
from the code generation model itself. Thus, we only need unit tests and the
execution ability during the dataset creation step instead of during inference
as in prior approaches.

While previous approaches target math problems to make the learning problem
tractable, in this paper, we design CodeRanker for more complex general
programming tasks in Python. A sampled Python program can fail in many
different ways. For example, a program when executed on a unit test can result
in a compile/runtime error and can produce a wide variety of outputs such as
integers, strings, lists and dictionaries that can produce a type mismatch. In
contrast in the math domain, there is less scope for compile/runtime errors
and the output is usually just a number.

Our CodeRanker approach is based on the idea that _a neural ranker trained to
distinguish between the various failure modes can have a better understanding
of the program and the task, and hence can do better at ranking programs._
Thus, we design _fault-aware_ CodeRanker and we investigate its impact on the
ranking performance. Each fault-aware ranker is a classifier trained to
predict one/two multi-class labels that are extracted from the rich
information obtained by executing the programs.

We used the APPS dataset to finetune/train the code generation and the
CodeRanker models. On this dataset, we showed that CodeRanker improved the
pass@1 performance of Codex (used in a few-shot manner) from 26% to 39.6% on
the validation set and from 3.8% to 4.5% on the test set. We additionally
found our rankers are transferable to a different dataset without any
additional training. On the HumanEval dataset, we improved Codex’s pass@1 from
26% to 32% and on the MBPP dataset, we improved from 36% to 42%. We found
similar performance boosts with other code generation models such as GPT-J and
GPT-Neo. Compared with a naïve binary classifier-based ranker, our fault aware
CodeRanker achieves better ranking performance. Finally, we investigated the
effect of mixing ranker datasets from multiple code generation models which
gives us an extra boost in the performance.

##  2 Preliminaries

Task | Reference code | Unit tests
---|---|---
⬇ Inputs: 6 36 0 -5 Outputs: 9 49 1 0
Figure 1: An example code generation task from the APPS dataset (train/3447).

###  2.1 Code generation

Task: A code generation task $G$ is a prompt, represented as a sequence of
tokens, that specifies the task at hand. $G$ is usually a combination of
natural language, input-output examples, and starter code. A solution $S$ to a
code generation task is a sequence of tokens that together form a program to
solve the given task. Existing datasets additionally contain a set of input-
output pairs that are used to test the correctness of a generated program.
Figure 1 gives an example code generation task, the corresponding reference
solution, and the unit tests taken from the APPS dataset
hendrycks2021measuring .

Model | # parameters | mode | pretrain dataset | finetune dataset
---|---|---|---|---
Codex (Davinci) | Unknown | One-shot |  GitHub + Common Crawl, Wiki, etc. | N/A
GPT-J | 6B | Fine-tuned | Pile dataset | APPS train dataset
GPT-neo 1.3B | 1.3B | Fine-tuned | Pile dataset | APPS train dataset
GPT-neo 125M | 125M | Fine-tuned | Pile dataset | APPS train dataset
Table 1: Code-generation models, their sizes, the mode of usage (fine-tuned or
few-shot), and the datasets used for pretraining and finetuning.

Models: There are several existing pre-trained LLMs in the literature that are
suitable to generating code either in a few-shot manner or after finetuning. A
code generation model $F$ provides a way for us to sample programs for a given
task $G$ as $S_{i}\sim P_{F}(S\mid G)$ where $P_{F}$ denotes the probability
distribution induced by the code generation model $F$.

In this paper, we study four different code generation models—(i) Codex, (ii)
GPT-J (6B), (iii) GPT-Neo 1.3B and (iv) GPT-Neo 125M (Table 1). Codex is the
largest state-of-the-art code generation model that is publicly available for
querying through an API and has shown impressive performance in code
generation chen2021evaluating . It is built on top of a GPT-3 language model
architecture and is trained on 180 GB of GitHub data. GPT-Neo and GPT-J are
open-sourced models with the number of parameters ranging from 125M to 6B.
These models are pre-trained on the Pile dataset (800 GB of natural language
corpus with 8% of GitHub data). Since, these models are open-sourced, it is
possible to finetune these models on a downstream dataset such as the APPS
dataset. In addition to the above models, there are other code-generation
models such as AlphaCode li2022competition  and Google’s model
austin2021program . While our approach is applicable to any of these models,
in this study, we validate the effectiveness of CodeRanker on a set of state-
of-the-art code generation models that are publicly available, for
reproducibility.

Metrics: Code generation models are evaluated based on functional correctness
rather than exact/fuzzy match to a reference program. This is because match-
based metrics are unable to account for the large and complex space of
programs functionally equivalent to a reference program. Functional
correctness is estimated by checking whether a sampled program passes a set of
unit tests. Prior approaches evaluate functional correctness using the pass@k
metric; given $k$ generated program samples per task, a task is considered
solved if any of the samples passes the unit tests. The pass@k metric measures
the total fraction of tasks solved. Additionally, we define the exec@k metric
which computes the fraction of the tasks for which there exists at least one
program in the $k$ samples that executes without any compile/runtime errors,
i.e., produces a non error value for each of the test inputs, but may or may
not match the desired output.

###  2.2 Code ranking

Given $n$ sampled programs using a code generation model,
$S_{1},\cdots,S_{n}\sim P_{F}(S\mid G)$, the goal of CodeRanker is to find an
ordering of the programs $S_{o_{1}},\cdots,S_{o_{n}}$ such that to compute the
_ranked pass@k_ for $k\leq n$, a problem is considered solved if any program
in the set $\\{S_{o_{1}},\cdots,S_{o_{k}}\\}$ passes the unit tests. A
CodeRanker model $R$ takes as input a code generation task $G$ and a sampled
program $S_{i}$ and outputs a score $s_{i}$. The ranked ordering of the
sampled programs is given by $S_{o_{1}},\cdots,S_{o_{n}}$ such that
$s_{o_{1}}>s_{o_{2}}>\cdots>s_{o_{n}}$.

##  3 Fault-Aware Neural Code Ranker

A CodeRanker model is a classifier that is trained to classify a pair $\langle
G,S_{i}\rangle$ as correct or not, where correct means that $S_{i}$ satisfies
the task $G$ with respect to its unit tests. The score $s_{i}$ is computed as
$s_{i}=P_{R}(\textsc{correct}|G,S_{i})$ where $P_{R}$ is the probability
according to the ranker model $R$. This probability is extracted using the
real values from the last layer before the SoftMax layer.

###  3.1 Code Ranker Dataset

To train a CodeRanker model, we need a dataset that has both correct and wrong
(i.e., not correct) programs. To collect these programs, we use the code-
generation models to sample $n=100$ programs for each task in the training
dataset. We then execute the sampled programs on their corresponding unit
tests to generate the classification labels. Following the observations from
cobbe2021training , one has to be careful to not over-train the base code
generation models to ensure diversity in the sampled programs. Hence, we only
finetune the base code generation models for a maximum of 2 epochs and chose a
checkpoint that results in the lowest validation loss (this does not apply to
the Codex model, which is used in a few-shot manner). Table 2 shows the
distribution of the ranker’s datasets obtained by using the 4 different code
generation models for the tasks in the APPS dataset 222We used a subset of 600
problems as the validation set, chosen randomly from a subset of the original
training problems for which the best model, Codex, produces at least one
correct program in its 100 samples. . As one can expect, the ranker dataset is
highly imbalanced with about 5X to 40X more wrong data points than correct
data points and this ratio is higher for smaller models such GPT-Neo models.

Code generation model used for generating the dataset | Training data | Validation data | Test data
---|---|---|---
C | W | C | W | C | W
| I | E |  | I | E |  | I | E
Codex | 70K | 180K | 140K | 16K | 26K | 18K | 19K | 280K | 200K
GPT-J | 20K | 210K | 150K | 3K | 32K | 23K | 2.4K | 210K | 260K
GPT-Neo 1.3B | 10K | 170K | 200K | 1.5K | 28K | 27K | 0.7K | 170K | 310K
GPT-Neo 125M | 10K | 150K | 220K | 0.8K | 23K | 34K | 0.2K | 140K | 340K
Table 2: Ranker dataset distribution for the datasets generated using the 4 different code generation models on the APPS tasks. C: # correct programs, W: # wrong programs, I: # intent errors, and E: # execution errors. Generated Program |  Full error message | Labels
---|---|---
⬇ B: wrong T: execution error I: - E: TypeError L: Line 2
⬇ B: wrong T: intent error I: OutputTypeError E: - L: -1
Table 3: Examples from the fault-aware ranker dataset showcasing the generated
program, the full error message obtained by executing the program, and the
fine-grained labels extracted from this error message. Each entry contains a
binary label (B) that classifies the entry as correct or wrong, a ternary
label (T) that distinguishes between correct, intent error, and execution
error, an intent error label (I) that specifies the type of the intent error
(or -), an execution error label (E) that specifies the type of the execution
error (or -), and a line number (L) that corresponds to the erroneous line in
the code (or -1 if there is no execution error).

Fault-aware ranker dataset: A straightforward ranker is one that is trained to
predict a binary label correct or wrong. However, a program fails for various
reasons and knowing why a program might fail is crucial to predicting whether
a program is correct or not. Therefore, we designed a fault-aware ranker
dataset. When we execute a program on a set of unit tests, the compiler
message is more than just a single bit of information. In fact, when the unit
test fails, we know if it failed because of a compile/runtime error (which we
call an _execution error_) or because the program produced a wrong output for
a particular input (which we call an _intent error_). Table 2 also shows the
distribution of intent errors and execution errors in the ranker datasets. An
interesting observation is that the percentage of execution errors decreases
for larger code generation models.

It is possible to further break down the wrong datapoints. Within the
execution error class, the compiler message tells us exactly the type of the
execution error (such as an IndexError or a TypeError or a TimeOutError) and
the line in the program that caused this error. By parsing the error message
from the Python compiler, we derived 10 most frequent classes of execution
errors as shown in Table 4 and Figure 2. Similarly, for the intent error
class, we know how the generated output differs from the expected output (such
as wrong type or wrong length of the array output). We manually designed 9
most frequent classes of intent errors by looking at various expected and
generated outputs from the training examples (see Table 5 and Figure 3). These
fine-grained execution-based labels constitute our fault-aware dataset. Table
3 shows a few data entries with all the labels. Figure 6 in the Appendix shows
the distribution of the various classes of execution errors and intent errors
for the ranker dataset obtained using the Codex model.

Class | Description
---|---
NameError |  Undefined variables
ValueError |  Operation/function received an argument of inappropriate value
EOFError |  Raised because of extra input() functions
TypeError |  An operation or function is applied to an object of inappropriate type
IndexError |  Array index out of bounds
KeyError |  Key in a dictionary is not found
TimeoutException |  Code is inefficient or doesn’t terminate
SyntaxError |  Parser encountered an error
Function not found |  The function expected by the unit tests is not found
Misc | Other execution errors

Table 4: Different classes of execution errors. See
https://docs.python.org/3/library/exceptions.html for more details.

(a) NameError
---
⬇ def bird_code(arr): if ’hyphen’ in arr: return [arr[i][0] + … # name ’i’ is
not defined in line 2
(b) KeyError
⬇ def league_standings(teams): return {i+1: teams[-i-1] for i in
range(len(teams))} # KeyError(-1) at Line 1
(c) TimeoutException
⬇ def fibonacci(n): return n if n in [0, 1] else fibonacci(n - 1) +
fibonacci(n - 2) # inefficient implementation

Figure 2: Examples of execution errors.

###  3.2 Code Ranker Tasks

We now describe the different classification tasks derived from the above
dataset. For each classification task, the input is a pair of code generation
task specification $G$ and a generated program $S_{i}$ and the output is one
or multiple labels where each label belongs to pre-determined set of classes.
We describe the different labels/classes of the various tasks below. These
tasks are designed to explore the trade-offs between having abstract classes
of failures versus having fine-grained classes of failures.
Binary (B): The output is a single binary label with two classes {correct,
wrong}.
Ternary (T): The ternary task splits the wrong class into intent error and
execution error classes: this forms a three-class classification task with
output labels {correct, intent error, execution error}
Intent Error aware (I): The intent error aware task splits the intent error
class in the ternary task into its 9 different sub-classes, thus has a total
of 11 classes for the output label.
Execution Error aware (E): The execution error aware task is similar to the
intent error aware task but instead of splitting the intent error class, we
now split the execution error class into its 10 different sub-classes, thus
has a total of 12 classes for the output label.
Execution Error + Error Line aware (E+L): This is a multi-class and multi-
label classification task which combines two classification tasks. The first
one is the execution error aware task described above. The second task is to
predict the line of the code that corresponds to an execution error. The
labels for the error line number also includes $-1$ to represent there is no
execution error.

Class |  Description
---|---
NoneError |  Generated output is None while expected is not
EmptyError |  Generated output is an empty array while expected is not
OutputTypeError |  Different types of outputs
LengthError |  Arrays/Dicts/Sets of different lengths
IntSmallError |  Integer outputs that are different from the expected ones with delta $\leq$ 10
IntLargeError |  Integer outputs that are different from the expected ones with delta $>$ 10
StringSmallError |  String outputs whose length is different from the expected ones with delta $\leq$ 3
StringLargeError |  String outputs whose length is different from the expected ones with delta $>$ 3
Misc |  Other intent error

Table 5: Intent error classes.

(a) NoneError
---
⬇ def consecutive_sum(num): sum = 0 n = len(str(num)) for i in range(n-1): if
num % i == 0 and sum > 0: return sum # return statement never gets triggered
(b) LengthError
⬇ def diamonds_and_toads(sentence,fairy): return dict(zip(’Ruby␣Crystal’, (0,
2, 1, 2, 0))) # Got {’R’: 0, ’u’: 2, ’b’: 1, ’y’: 2}, # expected {’ruby’: 3,
’crystal’: 2}
(c) StringSmallError
⬇ def smash(words): return”.join(word for word in words) # Got ’helloworld’, #
expected ’hello world’

Figure 3: Examples of intent errors.

###  3.3 Code Ranker Models

We implemented our fault aware CodeRanker models by finetuning the pretrained
CodeBERT feng2020codebert  model.

CodeBERT: It is a state-of-the-art pretrained BERT-style code-understanding
model trained on the CodeSearchNet dataset husain2019codesearchnet  using a
combination of masked language modeling and replaced token detection
objectives clark2020electra . It takes as input a concatenation of two
segments with a special separator token, namely
$[\text{CLS}],w_{1},w_{2},..w_{n},[\text{SEP}],c_{1},c_{2},...,c_{m},[\text{EOS}]$.
Usually, one segment is a natural language text, and another is a code.
$[\text{CLS}]$ is a special token, whose final hidden representation can be
treated as the aggregated sequence representation for classification or
ranking downstream tasks.

Adding a classification head: We add a classification head on top of a base
CodeBERT model by connecting a linear layer and a softmax layer to the hidden
representation of the $[\text{CLS}]$ special token. Let $C\in\mathbb{R}^{H}$
be the final hidden vector corresponding to the $[\text{CLS}]$ token and let
$W\in\mathbb{R}^{K\times H}$ be the weights of the newly added classification
layer where $K$ is the number of classes in the classification task. The
logits for the classification output are computed as
$\text{softmax}(CW^{\top})$ and we use a standard cross-entropy loss for
finetuning all the weights.

Adding a line prediction head: To predict the line corresponding to an
execution error, we introduce an error line vector $S\in\mathbb{R}^{H}$ during
fine-tuning. The probability of a newline (“\n”) token $i$ being the erroneous
line is computed as a dot product between $T_{i}$ and $S$ followed by a
softmax over all of the newline tokens in the code, i.e.,
$P_{i}=\frac{e^{S·T_{i}}}{\sum_{j}e^{S·T_{j}}}$ where $T_{i}$ is the final
hidden vector corresponding to the $i$th newline token. We include a newline
token at the beginning and the end of the input to indicate the case where
there is no erroneous line in the code (i.e., code does not result in an
execution error) and to indicate the case where the erroneous line is beyond
what is encoded in the input (this occurs if the task+code context cannot fit
within the 512 token limit of CodeBERT), respectively.

##  4 Evaluation

Code gen. model | pass@100 | pass@1 |  ranked pass@1 | pass@5 |  ranked pass@5 |  exec@1 |  ranked exec@1
---|---|---|---|---|---|---|---
Codex | 100 | 26 | 39.6 | 56.4 | 63.5 | 69.7 | 87.0
GPT-J | 48.8 | 5.1 | 11.0 | 15.6 | 21.7 | 60.4 | 82.9
GPT-Neo 1.3B | 34.6 | 2.6 | 8.0 | 9.1 | 15.1 | 52.1 | 85.6
GPT-Neo 125M | 23.6 | 1.4 | 6.5 | 5.2 | 11.4 | 41.1 | 58.9
Table 6: Results on the APPS validation dataset about how our fault-aware rankers can improve the pass@1, pass@5, and exec@1 performance for various code generation models. We use the best ranker model for each code generation model for this result. Pass@100 for the Codex model on this validation set is 100% by design.  Code gen. model | pass@100 | pass@1 |  ranked pass@1 | pass@5 |  ranked pass@5 |  exec@1 |  ranked exec@1
---|---|---|---|---|---|---|---
Codex | 24.1 | 3.8 | 4.5 | 9.2 | 10.2 | 59.6 | 73.4
GPT-J | 7.2 | 0.5 | 0.8 | 1.6 | 2.6 | 45.4 | 63.8
GPT-Neo 1.3B | 3.0 | 0.14 | 0.3 | 0.53 | 1.1 | 35.2 | 73.7
GPT-Neo 125M | 1.5 | 0.04 | 0.1 | 0.17 | 0.5 | 28.5 | 43.9
Table 7: Results on the APPS test dataset. We use the best checkpoints for the
code generation models and the ranker models based on the results on the
validation set.

We next evaluate our CodeRanker approach. We investigate (1) how fault-aware
CodeRankers can improve various code generation models on various code
datasets, (2) the impact of the different ranker tasks, and (3) the effect of
mixing ranker datasets generated by different code generation models.

###  4.1 Experiment Setup

Code generation datasets: We consider three existing code generation datasets
for our evaluation: (1) APPS hendrycks2021measuring : a collection of 5000
training and 5000 test tasks collected from coding competitions and interview
problems, (2) HumanEval chen2021evaluating : a set of 164 test tasks, and (3)
MBPP austin2021program : a set of 974 mostly basic python programming tasks
with 474 training problems and 500 test problems.

In our experiments, we only use the APPS dataset for finetuning the code
generation models and the CodeRanker models (since it is the largest dataset).
But we evaluate these models on all three sets of test tasks. The APPS dataset
does not come with a validation dataset, so we used a set of 600 tasks from
the original training dataset for validation; these are, then, excluded from
the training dataset. Since we are interested in explicitly evaluating the
ability of a ranker to distinguish correct code from wrong code, we chose our
validation set to only include problems for which a Codex model 333We chose
Codex for this filtering step because it is the best performing model out of
the 4 models we considered. (in a few-shot manner) can generate at least one
correct program in its 100 samples. To facilitate the transfer of GPT-J and
GPT-Neo finetuned models on the HumanEval and the MBPP datasets, we perform a
minor programmatic transformation of the task descriptions to match the APPS
style.

Metrics: We use the pass@1, pass@5, exec@1, ranked pass@1, ranked pass@5, and
ranked exec@1 metrics (higher values are better). See Section 2 for their
definitions. We also show the pass@100 metric to illustrate the maximum
possible value for the pass@k and ranked pass@k metrics. These metrics are
measured using an unbiased estimator from 100 samples as proposed by
chen2021evaluating .

Training setup and hyper-parameters:  We finetuned GPT-J and GPT-Neo code
generation models on the APPS training dataset for 2 epochs with a batch size
of 256 and a learning rate of 1e-5, and chose the checkpoint that has the
lowest validation loss. For inference, we used temperature sampling with
$T=0.8$ for Codex model and $T=0.9$ for the GPT-J and GPT-Neo models unless
specified otherwise. We chose these temperatures to maximize diversity in the
100 samples, but we also conduct an ablation with lower temperatures in Table
11. For each program, we sample 512 new tokens and truncate the generated
program by a special stop sequence that we used in the few-shot/finetuning
prompts.

We finetuned the CodeRanker models for 30 epochs with a batch size of 512 and
a learning rate of 1e-4, and chose the checkpoint that results in the best
ranked pass@1 metric on the validation dataset. We used class weights to
balance the different classes while training the rankers. All experiments are
conducted on V100-32GB GPUs.

Notation: We use the notation $R_{D_{X}}^{Y}$ to denote a CodeRanker model
trained on a dataset obtained using the code generation model $X$ from Table 2
and on one of the five ranker tasks $Y$ from Section 3.2.

###  4.2 Main Results: CodeRanker improves code generation models

APPS validation dataset: First, we analyze the results on the APPS validation
dataset of 600 tasks. Table 6 shows the performance on various metrics for the
4 different code generation models. These results use the best CodeRanker
model for each code generation model which is shown in Table 11. From Table 6,
we find that CodeRankers improve all the metrics for all the code generation
models despite their different sizes. The pass@1 performance increases by 5.1%
to 13.6% with CodeRanker and the models can solve about 30 to 80 more tasks
when it has to select only one program from the 100 samples. Another
interesting observation is that a GPT Neo 125M model when combined with
CodeRanker (another 125M model) beats a GPT-J model (with 50X more
parameters). These results show the effectiveness of CodeRanker on improving
the code generation models.

APPS test dataset: Our results on the APPS test dataset of 5000 tasks is shown
in Table 7. The test problems are harder than the ones on the validation set,
which we can see by the smaller pass@100 and pass@1 numbers. Hence, the
improvement from CodeRanker is smaller in scale, but it is still a significant
improvement; Codex’s pass@1 increases from 3.8% to 4.7% (35 additional
problems).

HumanEval and MBPP datasets:  We measure the transfer ability of CodeRanker on
two different datasets–HumanEval (results in Table 8) and MBPP (results in
Table 9). We can again see that CodeRankers improve the performance of all
code generation models on all metrics for both datasets (one exception is
pass@5 for GPT-J on MBPP). Codex model’s pass@1 performance increases by 6% on
both datasets. These results show the ability of CodeRanker to transfer out-
of-distribution and also showcases that the errors made by the code generation
models on different tasks are universal.

Exec@1 vs pass@1:  In all the above results, the improvements in the exec@1
metric are higher than the improvements in the pass@1 metric; this shows that
CodeRanker is better at identifying execution errors in code than intent
errors, which is expected since executing code is shown to be an inherently
hard task for language models austin2021program ; nye2021show .

Code gen. model | pass@100 | pass@1 |  ranked pass@1 | pass@5 |  ranked pass@5 |  exec@1 |  ranked exec@1
---|---|---|---|---|---|---|---
Codex | 88.4 | 26.3 | 32.3 | 50.5 | 61.6 | 77.0 | 86.6
GPT-J | 45.1 | 9.1 | 11.6 | 19.1 | 18.9 | 73.6 | 89.0
GPT-Neo 1.3B | 19.5 | 3.2 | 6.1 | 7.7 | 8.5 | 66.3 | 87.8
GPT-Neo 125M | 12.8 | 0.84 | 3.0 | 3.0 | 6.1 | 52.3 | 58.3
Table 8: Results on the HumanEval dataset showing the zero-shot transferability of the rankers trained on APPS. Code gen. model | pass@100 | pass@1 |  ranked pass@1 | pass@5 |  ranked pass@5 |  exec@1 |  ranked exec@1
---|---|---|---|---|---|---|---
Codex | 84.8 | 36.4 | 41.8 | 60.9 | 62.4 | 74.8 | 93.2
GPT-J | 60.4 | 12.1 | 14.2 | 28.9 | 28.2 | 73.3 | 80.8
GPT-Neo 1.3B | 40.0 | 3.6 | 5.0 | 11.9 | 16.2 | 58.3 | 75.2
GPT-Neo 125M | 6.8 | 0.2 | 0.8 | 0.9 | 2.6 | 7.6 | 14.2
Table 9: Results on the MBPP dataset, showcasing another instance where our
fault-aware rankers can be used in a zero-shot manner on a different dataset.

Code gen. Model | Best Ranker
---|---
Codex | $R_{D_{\text{Codex}}}^{T}$
GPT-J | $R_{D_{\text{GPT-J}}}^{T}$
GPT-Neo 1.3B | $R_{D_{\text{GPT Neo 1.3B}}}^{E+L}$
GPT-Neo 125M | $R_{D_{\text{GPT Neo 125M}}}^{I}$
Table 10: Best CodeRanker model for each code generation model based on best
ranked pass@1 on the validation dataset.

Setup | pass@100 | pass@1 |  ranked pass@1
---|---|---|---
HumanEval, Temp=0.8 | 88.4 | 26.3 | 32.3
HumanEval, Temp=0.2 | 69.5 | 35.2 | 42.7
MBPP, Temp=0.8 | 84.8 | 36.4 | 41.8
MBPP, Temp=0.2 | 70.0 | 47.2 | 46.8
Table 11: Results with different sampling temperatures for the Codex model
with rankers on HumanEval and MBPP.

###  4.3 Ablations

Effect of sampling temperature: On the HumanEval and the MBPP datasets, we
additionally experiment with different temperatures for sampling the 100
programs (see Table 11). As expected, we noticed that pass@100 decreases with
lower temperatures, but pass@1 increases. We found that CodeRanker further
increases the pass@1 performance for 3 out of the 4 setups and achieved 42.7%
ranked pass@1 on the HumanEval dataset—the best known result so far on this
dataset 444this excludes approaches that use execution during inference
chowdhery2022palm . On the MBPP dataset, under a low-temperature sampling
setup, we found that CodeRanker slightly decreases the pass@1 performance—this
we attribute to the small difference between the pass@1 and the pass@100
metrics, which makes it hard for a learned ranker to beat a random ranking
scheme.

Analyzing different ranker tasks: Figure 4 shows 4 training curves; one for
each code generation model, $X$, showcasing the validation ranked pass@1
curves for the model $X$ when combined with the 5 different rankers
$R_{D_{X}}^{*}$. Results in a tabular format can be found in the Appendix.
From the curves, we can notice that for larger models such as Codex and GPT-J,
the rankers trained on the ternary classification task $R^{T}$ perform the
best, and for smaller models such as GPT-Neo 1.3B and GPT-Neo 125M, the ranker
trained on the intent-aware classification task $R^{I}$ and the ranker trained
on the execution-aware + error line classification task $R^{E+L}$ perform the
best, respectively. We can also notice that the binary rankers $R^{B}$ perform
significantly worse especially with smaller models. These results suggest that
when training rankers on a dataset that has more wrong code, harder
classification tasks act as better regularizes. Figure 7 in the Appendix shows
similar training curves for the ranked exec@1 metric; here, we can see that
$R^{E}$ and $R^{E+L}$ always achieves best ranked exec@1 because they are
trained to identify execution errors and $R^{B}$ and $R^{I}$ have the worst
performance.

Figure 4: Training curves (smoothed) measuring ranked pass@1 over training
steps for various rankers on different code generation models.
Figure 5: Results showing how the ranked pass@1 changes with different ranker
datasets.

Analyzing different ranker datasets: In this experiment, we measure the impact
of using the rankers trained on data from one code generation model on another
code generation model and the effect of mixing these ranker datasets. We have
4 different code generation models in this experiment and hence, 4 different
ranker datasets. We analyze two different mixed datasets—(i) mixed-small that
randomly samples 25% of the above ranker datasets and combines them, and (ii)
mixed-large that combines all of the 4 ranker datasets into one; the former
represents a dataset that is roughly the same size of the individual ranker
datasets for a fair comparison, while the latter makes use of all the
available data. Figure 5 shows the ranked pass@1 of all the 4x6 combinations
in the form of a heat map. Each column has been normalized such that the
values for the same model is 1 (i.e. the diagonal values are 1). From the
figure, among the individual ranker datasets, we notice that the rankers
trained with the data from the same code generation model are the best and
that the rankers trained with data from code generation models with a large
size difference are the worst. Finally, the rankers trained on mixed-small
dataset perform slightly worse (except for GPT-J code gen. model) than using
the same model dataset, but the rankers trained on mixed-large dataset have
the overall best performance. These results suggest that while it is usually
better to use the same model for generating the ranking dataset, we can
potentially use smaller models (less expensive models) to augment the ranker
dataset to further improve the performance.

Additional qualitative/quantitative analysis of the CodeRanker approach can be
found in the Appendix.

##  5 Related works

Code generation tasks/models: There are several code generation models
explored in the literature, ranging from decoder-only architectures
chen2021evaluating ; austin2021program ; xu2022systematic ; black2022gpt ;
fried2022incoder  to encoder-decoder architectures li2022competition ;
nijkamp2022conversational ; ahmad2021unified ; wang2021codet5  of various
sizes. Similarly, there are several task datasets from multiple domains,
including math-word-problems austin2021program ; cobbe2021training , Jupyter
notebook cell generation chandel2022training , common programming tasks
(austin2021program, ; chen2021evaluating, ; schuster2021programming, ) and
competition level programming problems li2022competition ;
hendrycks2021measuring . In this paper, we evaluate CodeRanker’s ability to
improve the performance of four decoder-based code generation models on three
programming datasets. Our approach is agnostic to code generation model
architectures as long as they generate outputs by sampling and our approach
can be easily extended to different domains.

Code understanding tasks/models: Besides code generation tasks, there are
several code understanding tasks including code search barone2017parallel ;
cambronero2019deep ; gu2018deep ; husain2019codesearchnet , clone detection
svajlenko2014towards ; mou2016convolutional , code summarization
husain2019codesearchnet , code translation chen2018tree ; kulal2019spoc ;
nguyen2015divide  and defect detection chakraborty2021deep ;
bilgin2020vulnerability ; cheng2021deepwukong ; zhou2019devign ;
bilgin2020vulnerability . Encoder-only code understanding models
liu2019roberta ; feng2020codebert ; ahmad2021unified ; wang2021codet5 ;
guo2020graphcodebert  are developed for these tasks. Our CodeRanker task can
be viewed as a new code understanding task, and our ranker model is finetuned
from CodeBERT feng2020codebert . The defect detection datasets are closely
related to ours; the main difference is that prior work focused on finding
vulnerability in human-written code, while we focus on detecting errors in
model-generated code.

Filtering/ranking for code generation models:  Previous works such as
chen2021evaluating ; li2022competition  use execution to prune code
completions during inference. While chen2021evaluating  only uses unit tests
provided as part of the task, li2022competition  additionally uses a neural
model to generate inputs for the unit tests and uses execution to filter out
programs that produce the same outputs on those inputs. However, these work
requires executing potentially vulnerable code for every inference task. Our
approach bypasses execution at inference time with the neural ranker. Similar
to our work, cobbe2021training ; shen2021generate  propose a neural-network
based ranker/verifier for code generation models; the main difference is the
domain (general purpose programming tasks in our case versus math problems in
prior work) and we train our neural rankers to learn why/how a code fails
rather than just predicting a binary label. Moreover, cobbe2021training ;
shen2021generate  use generative models as their ranker base; in our work, we
show benefits of rankers using a simple encoder-only model with only 125M
parameters ( cobbe2021training  uses models with atleast 3B parameters) (see
Section A.2.3 in the Appendix to see our ablation on different ranker
architectures).

Using execution/static analysis in other contexts: There has been other works
on using execution to guide code generation by conditioning the generation on
a representation of the program states chen2018execution ; ellis2019write .
There are also attempts to replace the execution process with a neural model
in these cases chen2021latent ; nye2021show . Another relevant work is
mukherjee2021neural , which improves the code generation models by using
static analysis to augment the model’s input and has shown to drastically
reduce the number of execution errors made by the generated programs. This
paper takes an alternate approach by using a neural ranker model to prune out
wrong programs at inference time.

##  6 Conclusions and Future Directions

We presented CodeRanker that ranks programs generated by a code generation
model without explicitly executing the programs. Our rankers are fault-aware
i.e., trained to predict the fine-grained classes of failure modes and we
showed the effectiveness of the fault-aware rankers in improving pass@k and
exec@k metrics for various code generation models and tasks.

One of the main limitations is that the CodeRanker approach is not sound,
i.e., a ranker can classify a correct program as wrong and vice-versa.
Additionally, we incur extra inference time to get better performance, because
we now have to generate $n>>k$ programs to filter $k$ programs to show to the
user. Our current approach also relies on sampling full programs from the code
generation model before using CodeRanker. In the future, we want to
investigate ranking/classifying partial programs, which can in-turn reduce the
inference time by pruning out wrong programs early. Another future direction
for our work is to investigate other ranker model architectures such as those
that can better levarage the code structure and exploring other ranker tasks
such as generating the full error message. Finally, it will be interesting to
investigate the transferability of our CodeRanker approach to generic
programming tasks (rather than just competition programming). A main
challenge. here, is the lack of a generic programming dataset which needs to
collected by scraping public sources such as GitHub. Our preliminary
experiments on such a generic programming dataset show that existing code
generation models usually produce more programs with execution errors rather
than programs with intent errors . This observation combined with our results
in this paper that show CodeRanker is usually better at identifying execution
errors than intent errors, we are very optimistic that a fault aware
CodeRanker approach would also improve the code generation for generic
programming tasks.

##### Acknowledgements:

We thank Todd Mytkowicz, Piali Choudhury, Rahee Gosh Peshwaria, Curtis von
Veh, and Xiaodong Liu for helpful discussions on this work.

## References

  * [1] Github copilot: Your ai pair programmer.  https://copilot.github.com/, 2021.
  * [2] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang.  Unified pre-training for program understanding and generation.  arXiv preprint arXiv:2103.06333, 2021.
  * [3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  arXiv preprint arXiv:2108.07732, 2021.
  * [4] Antonio Valerio Miceli Barone and Rico Sennrich.  A parallel corpus of python functions and documentation strings for automated code documentation and code generation.  arXiv preprint arXiv:1707.02275, 2017.
  * [5] Zeki Bilgin, Mehmet Akif Ersoy, Elif Ustundag Soykan, Emrah Tomur, Pinar Çomak, and Leyli Karaçay.  Vulnerability prediction from source code using machine learning.  IEEE Access, 8:150672–150684, 2020.
  * [6] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al.  Gpt-neox-20b: An open-source autoregressive language model.  arXiv preprint arXiv:2204.06745, 2022.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [8] Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra.  When deep learning met code search.  In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 964–974, 2019.
  * [9] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray.  Deep learning based vulnerability detection: Are we there yet.  IEEE Transactions on Software Engineering, 2021.
  * [10] Shubham Chandel, Colin B Clement, Guillermo Serrato, and Neel Sundaresan.  Training and evaluating a jupyter notebook data science assistant.  arXiv preprint arXiv:2201.12901, 2022.
  * [11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * [12] Xinyun Chen, Chang Liu, and Dawn Song.  Execution-guided neural program synthesis.  In International Conference on Learning Representations, 2018.
  * [13] Xinyun Chen, Chang Liu, and Dawn Song.  Tree-to-tree neural networks for program translation.  Advances in neural information processing systems, 31, 2018.
  * [14] Xinyun Chen, Dawn Song, and Yuandong Tian.  Latent execution for neural program synthesis beyond domain-specific languages.  Advances in Neural Information Processing Systems, 34, 2021.
  * [15] Xiao Cheng, Haoyu Wang, Jiayi Hua, Guoai Xu, and Yulei Sui.  Deepwukong: Statically detecting software vulnerabilities using deep graph neural network.  ACM Transactions on Software Engineering and Methodology (TOSEM), 30(3):1–33, 2021.
  * [16] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [17] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  arXiv preprint arXiv:2003.10555, 2020.
  * [18] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * [19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [20] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama.  Write, execute, assess: Program synthesis with a repl.  Advances in Neural Information Processing Systems, 32, 2019.
  * [21] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al.  Codebert: A pre-trained model for programming and natural languages.  arXiv preprint arXiv:2002.08155, 2020.
  * [22] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis.  Incoder: A generative model for code infilling and synthesis.  arXiv preprint arXiv:2204.05999, 2022.
  * [23] Xiaodong Gu, Hongyu Zhang, and Sunghun Kim.  Deep code search.  In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), pages 933–944. IEEE, 2018.
  * [24] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al.  Graphcodebert: Pre-training code representations with data flow.  arXiv preprint arXiv:2009.08366, 2020.
  * [25] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al.  Measuring coding challenge competence with apps.  arXiv preprint arXiv:2105.09938, 2021.
  * [26] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt.  Codesearchnet challenge: Evaluating the state of semantic code search.  arXiv preprint arXiv:1909.09436, 2019.
  * [27] Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang.  Spoc: Search-based pseudocode to code.  Advances in Neural Information Processing Systems, 32, 2019.
  * [28] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.  Competition-level code generation with alphacode.  arXiv preprint arXiv:2203.07814, 2022.
  * [29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [30] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin.  Convolutional neural networks over tree structures for programming language processing.  In Thirtieth AAAI conference on artificial intelligence, 2016.
  * [31] Rohan Mukherjee, Yeming Wen, Dipak Chaudhari, Thomas Reps, Swarat Chaudhuri, and Christopher Jermaine.  Neural program generation modulo static analysis.  Advances in Neural Information Processing Systems, 34, 2021.
  * [32] Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen.  Divide-and-conquer approach for multi-phase statistical migration for source code (t).  In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 585–596. IEEE, 2015.
  * [33] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  A conversational paradigm for program synthesis.  arXiv preprint arXiv:2203.13474, 2022.
  * [34] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al.  Show your work: Scratchpads for intermediate computation with language models.  arXiv preprint arXiv:2112.00114, 2021.
  * [35] Tal Schuster, Ashwin Kalyan, Oleksandr Polozov, and Adam Tauman Kalai.  Programming puzzles.  arXiv preprint arXiv:2106.05784, 2021.
  * [36] Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu.  Generate & rank: A multi-task framework for math word problems.  arXiv preprint arXiv:2109.03034, 2021.
  * [37] Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia.  Towards a big data curated benchmark of inter-project code clones.  In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 476–480. IEEE, 2014.
  * [38] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  arXiv preprint arXiv:2109.00859, 2021.
  * [39] Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoorn.  A systematic evaluation of large language models of code.  arXiv preprint arXiv:2202.13169, 2022.
  * [40] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu.  Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks.  Advances in neural information processing systems, 32, 2019.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] In Section 6

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
Our approach increases inference time as mentioned in Section 6.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
The code and data is released on GitHub
https://github.com/microsoft/CodeRanker.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] In Section 4.1

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [No] It is very expensive to run the experiments
multiple times. But, we tried multiple runs for one of our rankers in early
experiments, and we did not see much variance in the results.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report the
resources used in Section 4.1 and in Section A.2.3.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [No]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes] The assets are publicly available.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Appendix

###  A.1 More details on the fault-aware ranker datasets

Table 12 and Table 13 show additional examples for each execution error and
intent error class.

Figure 6 shows the distribution of the various classes of execution and intent
errors for the ranker dataset generated by the Codex model using the APPS
training dataset.

Figure 6: Distributions for various classes of execution errors (left) and intent errors (right) in the ranker training dataset obtained using the Codex code gen. model. Class | Example 1 | Example 2
---|---|---
⬇ def args_to_string(args): return ’\n’.join(sorted(s, key=lambda x:
x.isdigit())) + ’\n’ # name ’s’ is not defined in line 2
⬇ def capitals_first(text): return ’{:b}{}’.format(text.lower(), text.upper(),
text.capitalize()) # Unknown format code ’b’ for object # of type ’str’ in
line 1
⬇ _ = input() s = input() ans = 1 for i in range(1, len(s)): if s[i] == "?"
and s[i - 1] == "?": ans = (ans * 2) % 1000003 # EOF when reading a line at
Line 1
⬇ def negation_value(s, val): if val in s: return False return s[0] # ’in
<string>’ requires string as # left operand, not bool at Line 1
⬇ def get_last_digit(index): digits = [x for x in \ range(1,index + 1) if x <
10] return digits[index] # List index out of range at Line 3
⬇ def league_standings(teams): return {i+1: teams[-i-1] for i in
range(len(teams))} # KeyError(-1) at Line 1
⬇ # Empty code
⬇ def fibonacci(n): return n if n in [0, 1] else fibonacci(n - 1) +
fibonacci(n - 2) # inefficient implementation
⬇ def process_data(data): return data[0] * data[1] * data[2] for i in
range(len(data)) # Syntax error at Line 1 # (list comprehension syntax)
⬇ for _ in range(int(input())): n = int(input()) a = a+1 … # UnboundLocalError
at line 2 # local variable ’a’ is referenced # before assignment.
Table 12: Examples of different classes of execution errors Class | Example 1 | Example 2
---|---|---
⬇ def start_smoking(bars,boxes): rem = boxes * 18 - bars * 10 if rem != 1:
print((rem + 4) // 5) else: print(0) # no return statement
⬇ def interleave(args): return [elem for elem in zip(args) if len(elem) <
len(args)] # Got [], expected # [1, ’c’, 2, ’d’, 3, ’e’]
⬇ class Solution: def longestPrefix(self, s: str) -> str: s = list(s) s = [i
for i,j in zip(s,s[::-1][0:])] return s[:-2] # Got [’"’, ’l’, ’e’, ’v’, ’e’],
expected’"’
⬇ def diamonds_and_toads(sentence,fairy): return dict(zip(’Ruby␣Crystal’, (0,
2, 1, 2, 0))) # Got {’R’: 0, ’u’: 2, ’b’: 1, ’y’: 2, ’ ’: 0}, # expected
{’ruby’: 3, ’crystal’: 2}
⬇ T = int(input()) for _ in range(T): N = int(input())
print(bin(N).count(’0’)) # Got [[2], [3]], expected [[1], [2]]
⬇ class Solution: def findMedian(self, nums1, nums2): nums1.sort()
nums2.sort() return float(’-inf’) # Got -inf, expected 2.0
⬇ def solve(st): a = st.find("a") return a if len(st) == 0 else
"".join(sorted(list(st)))[0] # Got ’a’, expected ’x’
⬇ def spacify(string): #your code here return ”.join(sorted(string.split())) #
Got ’Pippi’, expected ’P i p p i’
Table 13: Examples of different classes of intent errors Codex + ranker | ranked pass@1 | ranked pass@5 | ranked exec@1
---|---|---|---
$R_{D_{\text{Codex}}}^{B}$ | 37.6 | 64.6 | 83.4
$R_{D_{\text{Codex}}}^{T}$ | 39.6 | 63.5 | 87.0
$R_{D_{\text{Codex}}}^{I}$ | 36.5 | 64.7 | 81.8
$R_{D_{\text{Codex}}}^{E}$ | 38.6 | 64.0 | 87.5
$R_{D_{\text{Codex}}}^{E+L}$ | 37.1 | 61.5 | 88.3
Table 14: Ablation of the different fault-aware ranker tasks using Codex as the code-generation model (on the APPS validation dataset).  GPT-Neo 125M + ranker | ranked pass@1 | ranked pass@5 | ranked exec@1
---|---|---|---
$R_{D_{\text{GPT-Neo 125M}}}^{B}$ | 4.2 | 7.7 | 56.0
$R_{D_{\text{GPT-Neo 125M}}}^{T}$ | 5.7 | 10.4 | 78.4
$R_{D_{\text{GPT-Neo 125M}}}^{I}$ | 6.5 | 11.4 | 58.9
$R_{D_{\text{GPT-Neo125M}}}^{E}$ | 5.7 | 11.0 | 74.7
$R_{D_{\text{GPT-Neo 125M}}}^{E+L}$ | 6.5 | 11.9 | 87.6
Table 15: Ablation of the different fault-aware ranker tasks using GPT-Neo 125M model as the code-generation model (on the APPS validation set).  Codex + Ranker | ranked pass@1 | ranked exec@1
---|---|---
$R_{D_{\text{Codex}}}^{\text{best}}$ | 39.6 | 87.0
$R_{D_{\text{GPT-J}}}^{\text{best}}$ | 28.3 | 83.6
$R_{D_{\text{GPT-Neo 1.3B}}}^{\text{best}}$ | 26.9 | 83.6
$R_{D_{\text{GPT-Neo 125M}}}^{\text{best}}$ | 27.3 | 81.8
$R_{D_{\text{mixed-small}}}^{\text{best}}$ | 39.1 | 88.1
$R_{D_{\text{mixed-large}}}^{\text{best}}$ | 42.3 | 91.0
Table 16: Ablation of the different ranker datasets (using the best ranker
task for each dataset) on Codex completions (on APPS validation set).

Figure 7: Training curves (smoothed) measuring ranked exec@1 on the validation
set as training progresses for various rankers on different code generation
models.

###  A.2 Additional results

####  A.2.1 Ablations on different ranker tasks

Table 14 and Table 15 show the various ranked metrics for rankers trained on
different fault-aware tasks when using the Codex and GPT-Neo 125M code
generation models respectively. Figure 7 shows 4 training curves, $X$,
showcasing the validation ranked exec@1 curves for the model $X$ when combined
with 5 different rankers $R_{D_{X}}^{*}$.

####  A.2.2 Ablations on different ranker datasets

Table 16 shows the ablation results when training rankers on datasets
collected from code generation models. These results show the improvements of
rankers on the Codex code generation model. We can see that a ranker trained
on completions from the Codex model is better than a ranker trained on
completions from other models. We can also see that a ranker trained on the
mixed-large dataset (which use all completions from all models) beats the
ranker trained on Codex dataset by additional 2.7%.

####  A.2.3 Alternate ranker architectures

We experimented with a GPT-Neo 125M decoder-based model for the ranker and
found it to be performing suboptimal to a CodeBERT-based ranker (even though
both models are roughly the same size). The GPT-Neo 125M ranker trained on the
Codex ranker dataset ($R^{T}_{D_{\text{codex}}}$) achieved 36.8% ranked pass@1
when combined with the Codex model while the CodeBERT-based ranker achieved
39.6%.

We further conducted an experiment with a GPT-Neo 1.3B model for the ranker.
It took almost 4 days to complete 7 epochs of training with 16 V100 GPUs
running in parallel. And still, it didn’t reach the performance achieved by
the CodeBERT model. GPT Neo 1.3B ranker model achieved ranked pass@1 of 38.6%
while CodeBERT ranker achieved 39.6%. We stopped GPT Neo 1.3B ranker training
at 7 epochs because the training became unstable at that point. On the other
hand, a CodeBERT based ranker took only 12 hours to train for 30 epochs (with
16 GPUs).

Table 17 shows the ranked pass@1 and ranked exec@1 performance of these
various ranker architectures and Figure 8 shows the smoothed training curves
for these three rankers.

We believe that a ranker task is a code understanding task, hence starting
with pre-trained encoders such as CodeBERT are most effective compared to
starting with a pre-trained encode-decoder or decoder only code-generation
models. We leave the full study about the impact of various ranker
architectures to future work. We also do not anticipate that just bigger
ranker models would fill in the gap between current ranked pass@1 and pass@100
since a ranker needs to implicitly learn to execute which is a hard task. For
that, we might need new architectures for capturing code structure and
execution. For this future exploration, we hope that our ranker tasks will
serve as good benchmarks for evaluating such architectures.

Codex + Ternary Ranker | ranked pass@1 | ranked exec@1
---|---|---
CodeBert Ranker | 39.6 | 87.0
GPT-Neo 125M Ranker | 36.8 | 84.4
GPT-Neo 1.3B Ranker | 38.6 | 88.6
Table 17: Ablation of the different ranker architectures on Codex completions
(on APPS validation set; trained on the ternary ranker dataset).

Figure 8: CodeBert vs GPT-Neo ranker models: Training curves showing ranked
pass@1 and exec@1 on the APPS validation set with the ternary ranker.

####  A.2.4 Impact of example unit tests in the prompt

Following prior works, our experiments so far include a small number of unit
tests in the task description of the problem (as seen in Figure 1) so that we
can directly compare our results with prior works. Although our CodeRanker
approach does not explicitly use these unit tests (since we do not assume that
an execution environment is given), our rankers can use the given unit tests
as part of their language/code comprehension ability to better solve the
tasks. So, in this section, we conduct an ablation study to measure the impact
of the presence of these unit tests in the prompt on the ranking ability of
CodeRanker. We created a best-effort version of the APPS training/validation
dataset that does not include the unit tests in the prompts and we trained a
new ranker on this dataset. This new ranker only performs slightly worse than
the ranker that is trained with the unit tests in the prompts, but it still
significantly outperforms the approach without any ranker (see Table 18).

Codex + Ternary Ranker | pass@1 | exec@1
---|---|---
Prompts with example unit tests | 39.6 | 87.0
Prompts without example unit tests | 38.0 | 89.1
Without ranker | 26.0 | 69.7
Table 18: Ablation for evaluating the impact of example unit tests in the
prompt.

####  A.2.5 Comparison to ranking using the code gen. model’s predicted
probabilities

In this ablation, we compare the impact of ranking using a separate ranker
model versus ranking using the code gen. model’s predicted probabilities for
the various generated programs. The results for the GPT-Neo 125M code gen.
model can be found in Table 19; we can see that a separate CodeRanker produces
the best ranking effect and in fact, using the intrinsic code gen. model’s
probabilities to rank results in a slightly worse performance compared to a no
ranking approach.

| pass@100 | pass@1 | pass@5 | exec@1
---|---|---|---|---
GPT-Neo 125M | 23.6 | 1.4 | 5.2 | 41.1
\+ ranking using generation probability | - | 1.0 | 3.7 | 43.6
\+ CodeRanker | - | 6.5 | 11.4 | 58.9
Table 19: Comparing CodeRanker to ranking completions using Code Gen. model’s
predicted probabilities for the generated completions.

####  A.2.6 Qualitative analysis of CodeRanker

In Table 20, we show various programs generated by the fine-tuned GPT-J model,
the actual error message from the Python compiler upon executing these
programs, and the various labels generated by the different CodeRankers
trained on various ranker tasks.

Generated code | Actual error message | Predicted
---|---|---
⬇ B: correct T: intent error I: IntSmallError E: intent error E+L: intent
error at Line -1
⬇ B: correct T: intent error I: StrSmallError E: correct E+L: correct at Line
-1
⬇ B: wrong T: execution error I: execution error E: NameError E+L: NameError
at line 1
⬇ B: wrong T: intent error I: IntLargeError E: intent error E+L: intent error
at Line -1
⬇ B: correct T: intent error I: StringSmallError E: intent error E+L:
TypeError at Line 1
⬇ B: correct T: execution error I: execution error E: IndexError E+L:
IndexError at Line 6
Table 20: We show various programs sampled from a fine-tuned GPT-J model, the
actual error message, and the labels predicted by various ranker models
$R^{X}_{D_{GPTJ}}$. We use the green color to indicate that label is correct,
red to indicate that the label is incorrect, and blue to indicate the case
where the label correctly indicates an error in the program, but it indicates
the wrong error type.


# Cycled Compositional Learning between Images and Text

Jongseok Kim1 Youngjae Yu1,2* Seunghwan Lee2 GunheeKim1,2
Seoul National University1, RippleAI2
Seoul, Korea
<EMAIL_ADDRESS><EMAIL_ADDRESS>equal
contribution

###### Abstract

We present an approach named the Cycled Composition Network that can measure
the semantic distance of the composition of image-text embedding. First, the
Composition Network transit a reference image($RefImg$) to target
image($TrgImg$) in an embedding space using relative caption($Text$). Second,
the Correction Network calculates a difference between reference and retrieved
target images in the embedding space and match it with a relative caption. Our
goal is to learn a Composition mapping with the Composition Network
$G:(RefImg,Text)\rightarrow TrgImg$. Since this one-way mapping is highly
under-constrained, we couple it with an inverse relation learning with the
Correction Network $F:(RefImg,TrgImg)\rightarrow Text$ and introduce a cycled
relation $F(RefImg,G(RefImg,T))\approx T$ for given $RefImg,T$.

We participate in Fashion IQ 2020 challenge [2] and have won the first place
with the ensemble of our model. The code is available at
https://github.com/ozmig77/CCnet_fashioniq2020.

##  1 Introduction

Figure 1:  The key intuition of the Cycled Composition model. Given a triplet
of (reference image, Language query, Target image), Our model effectively
encodes the differential relationship between the two images in both Forward
(Composition Net) and Inverse (Correction Net) pathways.

For implementing interactive conversational image search systems, it is a
prerequisite to develop a controllable image search based on user feedback
(_e.g_. natural language queries and/or category filters). Beyond its
practical and commercial consequence that can promote users’ convenience of
shopping significantly, it is also an intriguing research problem to study the
multimodal embedding and cross-modal retrieval.

In this work, we propose an approach that can measure the semantic similarity
between the composition embedding of images and text, and apply it to tackle
controllable multimodal image retrieval for fashion data. We summarize the
contributions of this work as follows:

  1. 1.

We propose the Cycled Composition Network, which can measure the semantic
differential relationship between images for a query relative caption. Compare
to existing models such as TIRG [13], CurlingNet [14] our model not only
learns the image-text composition features for finding the most suitable
target data but also focuses on the queried attributes in target data for
better ranking.

  2. 2.

To validate our proposed model, we participate in the Fashion-IQ 2020
challenge and our ensemble model achieves the first place on the leaderboard.

##  2 Approach

In the Fashion-IQ challenge, a participant needs to retrieve the best matching
target image, given reference image and relative caption. Since attributes of
fashion correspond to various parts of the image, we utilize a mixture of
experts to solve the task.

Besides, we propose a Correction Network that find the difference between
reference and target images and checks its validity with a relative caption.

###  2.1 Image Experts

For getting fashion related features from image, we first pre-train deep
CNNs(_e.g_. ResNet-152 [3], DenseNet-169 [4]) with Deepfashion[8] attribute
prediction task. And use the CNNs as a basic backbone of our model.

To spot out fashion attributes in diverse aspect, in addition to use global
pooled embedding from last layer of pre-trained CNN, we also utilize output of
intermediate layer and 3 x 3 features slice from multi locations([0:3,2:5],
[2:5,0:3], [2:5,2:5], [2:5,4:7] and [4:7,2:5] from the $7\times 7$) follow by
average pooling.

Then we pass embeddings into expert specific fully connected layer follow by
context gating[9] and get 7 different image expert embeddings.

We share parameters for image experts between reference and target images. In
the end, we get reference and target image embedding for each expert $e$,
$x^{ref}_{e},x^{trg}_{e}\in\mathbb{R}^{D}$ where $D$ is a hidden dimension.

###  2.2 Text Experts

Given relative caption with $l$ words, We first embed word using GloVe[10],
$w^{*}=[w^{*}_{1},...,w^{*}_{l}]\in\mathbb{R}^{l\times 300}$. Then we get
caption embedding $w\in\mathbb{R}^{l\times D}$ by,

$\displaystyle w=\text{FC}([\text{Conv1d}(w^{*});w^{*}])$ (1)

where Conv1d is a convolution 1d layer, FC is a fully connected layer, and
$[;]$ is a concatenation.

Each image expert has different words to be focused on. For example, in Figure
1 it is natural that experts with global information attend to ”yellow” or
”less flowy” while the expert with the upper part attends to ”shouldered
floral print”. To take advantage of it, we assign randomly initialized
characterize embedding $m_{e}\in\mathbb{R}^{D}$ for each expert $e$. Then
perform attended pooling with caption embedding $w$ to get attended embedding
$t^{*}_{e}\in\mathbb{R}^{D}$,

$\displaystyle\alpha_{e}$
$\displaystyle=soft\max_{l}(\text{FC}(\text{FC}(m_{e}\odot w)))$ (2)
$\displaystyle t^{*}_{e}$ $\displaystyle=\sum_{l}\alpha_{e,l}w_{l}$ (3)

where $\odot$ is hadamard product.

Finally we further embed $t^{*}_{e}$ with expert specific fully connected
layer follow by context gating[9] and get text expert embedding $t_{e}$.

###  2.3 Cycled Composition Networks

Composition Network Composition Network finds the best matching target image
from a given reference image and relative caption. To compose image and text,
we use gating module intuited by Text Image Residual Gating (TIRG) [13] to
manipulate the vector transition.

$\displaystyle\bar{t}_{e}$
$\displaystyle=[t_{e};\text{Fusion}(x^{ref}_{e},t_{e})]$ (4) $\displaystyle
c_{e}$
$\displaystyle=w_{g}f_{gate}(x^{ref}_{e},\bar{t}_{e})+w_{r}f_{res}(x^{ref}_{e},\bar{t}_{e})$
(5)

where $w_{g},w_{r}$ are learnable paramters, and we use MUTAN [1] for fusion
function. The gating and residual functions are computed by,

$\displaystyle f_{gate}(x^{ref}_{e},\bar{t}_{e})$
$\displaystyle=\sigma(\text{FC}(\text{FC}([x^{ref}_{e};\bar{t}_{e}]))\odot
x^{ref}_{e}$ (6) $\displaystyle f_{res}(x^{ref}_{e},\bar{t}_{e})$
$\displaystyle=\text{FC}(\text{FC}([x^{ref}_{e};\bar{t}_{e}]))$ (7)

where $\sigma$ is the sigmoid function.

Then we calculate matching score $s^{r}$ by,

$\displaystyle
s^{r}(\textbf{x}^{ref},\textbf{t},\textbf{x}^{trg})=\sum_{e}c_{e}\cdot
x^{trg}_{e}$ (8)

where $\cdot$ is dot product. In minibatch size of $B$ consist of ground truth
triplets
$\\{\textbf{x}^{ref}_{i},\textbf{t}_{i},\textbf{x}^{trg}_{i}\\}_{i=1}^{B}$, we
use loss function object to find best matching target image given reference
image and text.

$\displaystyle
L=\frac{1}{B}\sum^{B}_{i=1}-\log\frac{exp(s^{r}(\textbf{x}^{ref}_{i},\textbf{t}_{i},\textbf{x}^{trg}_{i}))}{\sum^{B}_{j=1}exp(s^{r}(\textbf{x}^{ref}_{i},\textbf{t}_{i},\textbf{x}^{trg}_{j}))}$
(9)

Correction Network Correction Network verifies that the difference between
reference and target image explains the relative caption. By this, we double
check that the target image is related to the reference image and the text.

To represent the difference between reference and target image, we use the
following approach,

$\displaystyle\bar{x}^{trg}_{e}$ $\displaystyle=FC([x^{trg}_{e}\odot
x^{ref}_{e};x^{trg}_{e}])$ (10) $\displaystyle\bar{x}^{ref}_{e}$
$\displaystyle=FC([x^{trg}_{e}\odot x^{ref}_{e};x^{ref}_{e}])$ (11)
$\displaystyle\bar{x}^{diff}_{e}$
$\displaystyle=\bar{x}^{trg}_{e}-\bar{x}^{ref}_{e}$ (12) $\displaystyle d_{e}$
$\displaystyle=\text{FC}([x^{ref}_{e};x^{trg}_{e};\bar{x}^{diff}_{e}])$ (13)

The intuition is that concatenate with fused embedding before subtraction
allows features to know what to subtract.

Then we calculate the matching score by,

$\displaystyle
s^{c}(\textbf{x}^{ref},\textbf{x}^{trg},\textbf{t})=\sum_{e}d_{e}\cdot t_{e}$
(14)

Training is done by predicting the best matching target image with a ground
truth reference image and relational text which identical to Eq.9.

CCNet Utilize results from both Composition Network and Correction Network we
calculate the final probability by,

$\displaystyle
p(\textbf{x}^{trg}|\textbf{x}^{ref},\textbf{t})=p^{r}(\textbf{x}^{trg}|\textbf{x}^{ref},\textbf{t})p^{c}(\textbf{x}^{trg}|\textbf{x}^{ref},\textbf{t})$
(15)

where $p^{r}$ and $p^{c}$ are probability obtain by softmax of score $s^{r}$
and $s^{c}$ from Composition Network and Correction Network.

Figure 2:  Schematics for the Composition and the Correction Networks. Best viewed in color.  Category | Dress | Shirt | Toptee | Total
---|---|---|---|---
Metric | R@10 | R@50 | R@10 | R@50 | R@10 | R@50 | Avg
single conv1d | 0.2716 | 0.5488 | 0.2227 | 0.4563 | 0.3059 | 0.5690 | 0.3957
multi conv1d | 0.3024 | 0.5434 | 0.2277 | 0.4652 | 0.3274 | 0.5834 | 0.4082
shared fc | 0.2737 | 0.5335 | 0.2066 | 0.4116 | 0.3004 | 0.5528 | 0.3847
CurlingNet [14] | 0.2615 | 0.5324 | 0.2145 | 0.4456 | 0.3012 | 0.5523 | 0.3845
CompositionNet | 0.3044 | 0.5741 | 0.2375 | 0.4701 | 0.3146 | 0.5762 | 0.4128
CorrectionNet | 0.3153 | 0.5811 | 0.2537 | 0.4912 | 0.3345 | 0.5926 | 0.4281
CCNet | 0.3456 | 0.6163 | 0.2767 | 0.5147 | 0.3677 | 0.6313 | 0.4587
Ensemble (val split) | 0.4075 | 0.6748 | 0.3229 | 0.5687 | 0.4202 | 0.6818 | 0.5126
Ensemble (test split) | 0.4289 | 0.6749 | 0.3376 | 0.5790 | 0.4222 | 0.6702 | 0.5188
Table 1: The qualitative result on Fashion-IQ dataset validation split. Top:
ablation study of text experts model. Middle: comparison with baselines and
the Cycled Composition Networks. Bottom: the result of ensemble models. ‡:
based on an official leaderboard results. Rounded up to 2 decimals

##  3 Experiment

In this section, we report the experimental results of CCNet for the fashion-
IQ challenge. The challenge provides an image retrieval dataset where the
input query is specified in the form of a candidate image and two natural
language expressions that describe the visual differences of the search
target.

The dataset contains 77,683 fashion images and 30,134 pairs of relative
captions for interactive image retrieval. The participant needs to retrievthe
best matching image given a pair of a reference image and relative caption.

Evaluation Metrics. Every participant is evaluated by the recall metrics on
the test splits of the dataset. For each of the three fashion categories
(dresses, tops, shirts), Recall@10, and Recall@50 are computed on all test
queries. The overall performance is evaluated based on the average of
Recall@10 and Recall@50. We strictly follow the evaluation protocols of the
challenge111https://sites.google.com/view/cvcreative2020/fashion-iq.

###  3.1 Training Details

In our main model, we use DenseNet-169[4] for backbone of image embedding. For
image augmentation, we use random crop and random horizontal flip. To get the
GloVe vector, we use spacy en_vectors_web_lg. The ReLU activation layer is
used in every FC layer, and the hidden dimension $D$ is set as 1024. For
regularization, we apply batch normalization [5], and use dropout [11] with
probability 0.2.

Each training batch consists of $B=32$ triplets of (reference image, query
text, target image). We use batch shuffling in every training epoch. We use
Adam[6] optimizer with learning rate 1e-4 with the exponential learning rate
decay of 0.95 per step.

Ensemble. For final ensemble, we use several variants of 1\.
backbone(ResNet50, ResNet152, FishNet99[12]) 2\. text embedding(Word2Vec,
BERT) 3\. text encoder(single conv, multi conv, FC, average pooling) 4\. video
experts (spatial experts, multi global pooled experts) 5\. composition method
(TIRG, FiLM, MUTAN, concat) 6\. data augmentation (affine transformation,
color jittering, back translation) 7\. hidden dimension size, batch size,
random seeds

###  3.2 Quantitative Results

Table 1 summarize the quantitative results of our experiments. The top part
shows ablation study of text experts model in Composition Network. Where
single conv1d uses the same average pooled text embedding for all image
experts, multi conv1d assigns independent text experts to each image experts
i.e. parameters are not shared in word embedding step(Eq.1), shared fc uses FC
layer instead of Conv1d and CompositionNet is our proposed text expert model.
As in the table, we show that our designed text expert model achieves better
performance than other methods.

The middle part shows comparison with baselines and the Cycled Composition
Networks. As in the table, our Composition Network outperforms the results of
last year’s participants. Furthermore, we show that Correction Network solely
outperforms the conventional Retrieval setting. Finally utilizing both
networks(CCNet) shows a better result.

As a final result, our ensemble model achieves a 0.5126 average recall in
validation split and 0.5188 in the test split.

###  3.3 Qualitative Results

Figure 3 shows difference between our joint model and classic CompositionNet.
Thanks to double checking by correction network, our joint method find clothes
that satisfy all conditions in the caption. Figure 4 illustrates the
qualitative results of our method. We display reference images and text
queries (Left) and the highest-scored retrieved images (Right) for some test
examples. While maintaining the style of the reference image, our model
assigns a high score to the sample that well reflects the needs of the user
query.

Figure 3: Qualitative examples of the CCNet compared to ablational variant, CompositionNet only.  Figure 4: Qualitative examples of the image-text retrieval task in the Fashion-IQ dataset.  Submission | Team Name | Avg R@(10,50)
---|---|---
cuberick | CARDINAL | 0.33
emd |  | 0.39
skywalker [7] | DESIGNOVEL | 0.44
shuan [14] | RippleAI | 0.47
superraptors |  | 0.49
zyday |  | 0.43
stellastra | VAA | 0.48
ganfu.wb |  | 0.50
tianxi.tl | Alibaba | 0.50
Ours | RippleAI | 0.52
Table 2:  Performance comparison for final submission on Test Phase. We also
report the results of last year’s challenge (FashionIQ 2019) in the top part
of the table. This year, all participants are not allowed to use attribute
metadata for test split, which is crucial to improve performance in last
year’s challenge. We finished the first place in the challenge.

##  4 Conclusion

We proposed the Cycled Composition Network(CCNet) for learning differential
relations between two image embeddings with respect to a text query. The two
key components of the model Composition, and Correction networks are easily
adaptable in many image-text composition tasks, including user adjustable
retrieval or image recommendation systems. We demonstrated that our method
improved the performance of image-text composition learning.

Moving forward, we plan to expand the applicability of the Cycled Composition
network; we can tackle relative caption/dialogue generation task for
multimodal conversational systems.

## References

  * [1] Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome.  Mutan: Multimodal tucker fusion for visual question answering.  In ICCV, 2017.
  * [2] Xiaoxiao Guo, Hui Wu, Yupeng Gao, Steven Rennie, and Rogerio Feris.  The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback.  In ICCV, 2019.
  * [3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In CVPR, 2016.
  * [4] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [5] Sergey Ioffe and Christian Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  In ICML, 2015.
  * [6] Diederik Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  In ICLR, 2015.
  * [7] Jianri Li, Jae whan Lee, Woo sang Song, Ki young Shin, and Byung hyun Go.  Designovel’s system description for fashion-iq challenge 2019.  arXiv, 2019.
  * [8] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [9] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learnable pooling with context gating for video classification.  arXiv, 2017.
  * [10] Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  GloVe: Global Vectors for Word Representation.  In EMNLP, 2014.
  * [11] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  JMLR, 2014.
  * [12] Shuyang Sun, Jiangmiao Pang, Jianping Shi, Shuai Yi, and Wanli Ouyang.  FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction.  In NeurIPS, 2018.
  * [13] Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays.  Composing text and image for image retrieval-an empirical odyssey.  In CVPR, 2019.
  * [14] Youngjae Yu, Seunghwan Lee, Yuncheol Choi, and Gunhee Kim.  Curlingnet: Compositional learning between images and text for fashion iq data.  arXiv, 2020.