11institutetext:  Poznan University of Technology
Faculty of Computing and Telecommunications
Piotrowo 3, 60-965 Poznań, Poland
11email<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

# WineGraph: A Graph Representation For Food-Wine Pairing

Zuzanna Gawrysiak 11 0009-0005-7674-5783 Agata Żywot 11 0009-0002-2851-4866
Agnieszka Ławrynowicz 11 0000-0002-2442-345X

###### Abstract

We present WineGraph, an extended version of FlavorGraph, a heterogeneous
graph incorporating wine data into its structure. This integration enables
food-wine pairing based on taste and sommelier-defined rules. Leveraging a
food dataset comprising 500,000 reviews and a wine reviews dataset with over
130,000 entries, we computed taste descriptors for both food and wine. This
information was then utilised to pair food items with wine and augment
FlavorGraph with additional data. The results demonstrate the potential of
heterogeneous graphs to acquire supplementary information, proving beneficial
for wine pairing.

###### Keywords:

heterogeneous graph graph embeddings rules neuro-symbolic learning and
reasoning computational food

##  1 Introduction

The field of food and wine pairing has garnered significant attention in
recent years, with various studies focusing on understanding the intricate
relationships between flavours and aromas [8]. Despite the wealth of
information available on food and wine individually, there is a noticeable gap
in comprehensive datasets specifically dedicated to food-wine pairing [1]. In
light of recent advancements in food recommendation and substitution using
graphs, the primary objective of this research is to enhance the existing
heterogeneous graph structure, FlavorGraph [12], by incorporating detailed
information about wine.

FlavorGraph is a large-scale graph network comprising food and chemical
compound nodes. Recent advancements in graph-based embedding approaches,
particularly utilizing node2vec [6] and metapath2vec [2] algorithms, have
shown promise in constructing conceptual representations from network data.
These methods generate random walks analogous to sentences in word2vec [10],
leveraging relations within the network. The application of these techniques
to food pairing recommendations involves constructing representations of foods
based on relationships between ingredients and chemical compounds.

In this work, we present _WineGraph_ , an extended version of FlavorGraph, a
heterogeneous graph incorporating wine data into its structure.

##  2 Preliminaries

Building upon graph-based embedding approaches, and resources: Recipe1M [9],
FlavorDB [5], and HyperFoods [13]; FlavorGraph uses metapath2vec to generate
conceptual representations of food. By defining food-specific metapaths and
considering both chemical and statistical aspects of food pairing, the model
addresses optimization challenges arising from the sparse availability of
chemical information for food ingredients. Multiple food-specific metapaths
were designed in this respect by Park at al. [12] to facilitate the transfer
of scarce chemical information from the compound nodes to non-hub ingredient
nodes via chemical-hub ingredients. These metapaths enable training on complex
relations, including food-food and food-chemical compounds interactions.

More formally, following Dong et al. [2] we define _heterogeneous network_
(Definition 1).

###### Definition 1

A _heterogeneous network_ is defined as a graph $G=(V,E,T)$ in which each node
$v$ and each link $e$ are associated with their mapping functions
$\phi(v):V\rightarrow T{{}_{V}}$ and $\phi(e):E\rightarrow T_{E}$,
respectively. $T_{V}$ and $T_{E}$ denote the sets of object and relation
types, where $|T_{V}|+|T_{E}|>2$.

Subsequently, by taking into account a heterogeneous network as our input, we
formulate the task of heterogeneous network representation learning in the
following manner.

###### Definition 2

_Heterogeneous network representation learning_ : Given a heterogeneous
network $G$, the task is to learn the $d$-dimensional latent representations
$\mathbf{X}\in\mathbb{R}^{|V|\times d},d\ll|V|$ that are able to capture the
structural and semantic relations among them.

Park at al. [12] extended metapath2vec model with an additional chemical
structure learning layer.

##  3 Materials and Methods

To integrate wine pairing into FlavorGraph, we have performed four key steps:
1) pre-processing food and wine review datasets, 2) calculating aroma
descriptors based on resulting phrases, 3) creating a list of food-wine
pairings, 4) incorporating the resulting data into FlavorGraph. The resulting
WineGraph is visualized in Figure 1).

Figure 1: Visualization of the WineGraph using t-SNE projection.

###  3.1 Data Preparation

The first step involved utilising two datasets: Amazon Fine Food Reviews111
https://www.kaggle.com/datasets/snap/amazon-fine-food-reviews and Wine
Reviews222https://www.kaggle.com/datasets/roaldschuring/wine-reviews. The text
was first tokenized and normalized. Then the most frequent phrases which
consisted of 1-3 tokens used together most frequently (i.e., $n$-grams) were
extracted, obtaining flavour descriptors like _fruit flavour_ , _acid_ ,
_black cherries_ etc. The second step involved mapping the phrases to aroma
descriptors (wine only) using the UC Davis wine
wheel333http://winearomawheel.com/. This wheel is comprised of three tiers of
aromas, ranging from specific to broad, facilitating the generalization of
descriptors. An illustration of these levels would be raspberry -> berry ->
fruit. In the third step, all preprocessed reviews were utilised to train the
word2vec model. We also calculated TF-IDF embeddings. The result was
300-dimensional aroma vectors 7 non-aroma scalers for wines and
300-dimensional aroma vectors for food. Table 1 illustrates core tastes and
obtained relevant embeddings.

Table 1: Closest and furthest items for core tastes - cells contain food item (cosine similarity). Taste | Closest Item | Furthest Item
---|---|---
Weight | Pizza (0.546) | Dragonfruit (-0.118)
Sweet | Pineapple (0.536) | Mackerel (-0.254)
Acid | Tart (0.600) | Biscuit (-0.121)
Salt | Bacon (0.672) | Nectar (-0.122)
Piquant | Chili (0.434) | Sole (-0.106)
Fat | Cake (0.618) | Coffee (-0.115)
Bitter | Tart (0.475) | Platter (-0.155)

###  3.2 Pairing Procedure

The resulting embeddings are used to generate food-wine pairings based on
predefined rules, encompassing attributes like _sweetness_ and _acidity_ ,
defined as numerical thresholds. In the rules, we use 7 types of numerical
variables that correspond to the set {sweet, bitter, salty, acid, fatty,
piquant, weight} whose values are normalized to be in the range from 0 to 1.
The pairing procedure is: 1) calculate aroma and non-aroma descriptors with
the use of the trained word2vec model, 2) eliminate wines that do not match
the food item (predefined set of rules), 3) find congruent and contrasting
wines (predefined set of rules), 4) sort by aroma similarity. These steps were
first proposed by Roald Schuring444https://towardsdatascience.com/robosomm-
chapter-5-food-and-wine-pairing-7a4a4bb08e9e.

A wine and food pairing must first meet all elimination/constraint rules (that
is, not be rejected by any) and then any pairing rule to conclude as ”pairing”
is true. In Tables 2 and 3 we provide two sets of such sommelier-defined rules
[8]555https://academy.getbackbar.com/the-basics-wine-and-food-
pairing666https://cdn.courtofmastersommeliers.org/uploads/2022/11/Food-and-
Wine-1.pdf (elimination rules and congruent rules). We can formalize the
former as constraints and the latter as decision rules [4, 11].

Table 2: Elimination rules (constraints). weight |  Wine should have at least the same ”body” as the food.
---|---
acidity |  The wine should be at least as acidic as the food.
sweetness |  The wine should be at least as sweet as the food.
bitterness |  Bitter wines do not pair well with bitter foods.
bitter-salt |  Bitterness and saltiness do not pair well together.
acid-bitter |  Acidity and bitterness do not pair well together.
acid-piquant |  Acidity and piquancy (spiciness) do not pair well together.
Table 3: Congruent/contrasting rules (decision rules). sweet pairing |  Sweet food is paired well with highly bitter, fatty, piquant, salty, or acidic wines.
---|---
acid pairing |  Acidic food is paired well with highly sweet, fatty, or salty wines.
salt pairing |  Salty foods are paired well with highly bitter, sweet, fatty, piquant, or acidic wines.
piquant pairing |  Spicy food is paired well with highly sweet or fatty wines.
fat pairing |  Fatty food is paired well with highly bitter, sweet, piquant, or acidic wines.
bitter pairing |  Bitter food is paired well with highly sweet or fatty wines.

The sample elimination rule for acidity (the wine should be at least as acidic
as the food) is shown below:

$wine_{acid}>=food_{acid}\Rightarrow\mathbf{eliminate_{false}}$

and sample pairing rule for acidity (acidic food is paired well with highly
sweet, fatty, or salty wines) is as follows:

$food_{acid}>0.75\land(wine_{sweet}>0.75\lor wine_{fat}>0.75\lor
wine_{salt}>0.75)\Rightarrow\mathbf{pairing_{true}}$

###  3.3 Incorporating Pairing Into Knowledge Graph

The obtained set of paired food and wine items was utilised to train the
metapath2vec model, generating a graph incorporating wines by adding nodes of
the wine type to already existing types of nodes in the FlavorGraph.
Incorporating pairings into knowledge graph was performed in the following
steps: 1) get top $k$ pairings for each food item ($k=3$), 2) incorporate wine
data into FlavorGraph (nodes), 3) add wine-pairing information to the graph
(edges), 4) define new metapaths in FlavorGraph (paths through a heterogeneous
graph, (illustrated in Figure 2), 5) train FlavorGraph (300-dimensional
space).

(a) A food and wine network.

(b) Skip-gram in metapath2vec.

Figure 2: Graph embedding with metapath2vec on WineGraph. Random walks
traverse through various paths and gather nodes of different types (sample
paths are shown in the left part of the figure).

###  3.4 Experimental Results

The goal of the experiments was to determine whether wine can be added to the
FlavorGraph without loss of quality and while maintaining correct pairings. We
have evaluated our method with the use of _Normalised Mutual Information
(NMI)_ as the quality measure. In our experiments, we wanted to show that for
the task of clustering by food category quality is not compromised (see:
sample clusters in Figure 4). In other words, whether NMI is not lower than
for the FlavorGraph without wine (see: Table 4). Figure 3 shows sample flavor
profiles. Table 5 shows sample pairings.

(a)

(b)

Figure 3: Flavour profiles for wine pairing generated for burrito + guacamole Table 4: Comparison of Normalized Mutual Information (NMI) values for different epochs. Dataset | Epochs | NMI
---|---|---
FlavorGraph | 10 | 0.309
FlavorGraph + wine | 5 | 0.341
10 | 0.319
15 | 0.351
20 | 0.358
Figure 4: Sample clusters.

Table 5 shows the top 3 pairings for burrito, generated and as the closest
nodes in the graph.

Table 5: Pairings for burrito. Top |  Generated pairing |  Pairing from the graph
---|---|---
1 |  Bordeaux-style Red Blend, Stellenbosch, South Africa |  Malbec-Cabernet Sauvignon, Bordeaux-style Red Blend, Mendoza, Argentina
2 |  Bordeaux-style Red Blend, Lussac Saint-Émilion, Bordeaux, France |  Bordeaux-style Red Blend, Stellenbosch, South Africa
3 |  Malbec-Cabernet Sauvignon, Bordeaux-style Red Blend, Mendoza, Argentina |  Bordeaux-style Red Blend, Listrac-Médoc, Bordeaux, France

##  4 Conclusions

In this work, we have shown that Wines can be successfully represented in the
form of a graph, enhancing food-wine pairing tasks. For this purpose, we have
devised a neural-symbolic method comprised of the embeddings of a
heterogeneous graph and rules.

In the future work, we aim to integrate more characteristics of food and wine,
and devise novel embedding methods specifically suited for such data.

## References

  * [1] Bender, T., Sørensen, S.M., Kashani, A., Hjorleifsson, K.E., Hyldig, G., Hauberg, S., Belongie, S.J., Warburg, F.: Learning to taste: A multimodal wine dataset. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 (2023), http://papers.nips.cc/paper_files/paper/2023/hash/170035f97007fdfa665880107b56f384-Abstract-Datasets_and_Benchmarks.html
  * [2] Dong, Y., Chawla, N.V., Swami, A.: metapath2vec: Scalable representation learning for heterogeneous networks. In: Proc. of KDD’2017. pp. 135–144. ACM (2017)
  * [3] Eschevins, A.: Matching Beer with Food: Pairing Principles, Underlying Mechanisms and a Focus on Aromatic Similarity. Ph.D. thesis, Université Bourgogne Franche-Comté (Dec 2018), https://theses.hal.science/tel-02084896
  * [4] Fürnkranz, J., Gamberger, D., Lavrac, N.: Foundations of Rule Learning. Cognitive Technologies, Springer (2012). https://doi.org/10.1007/978-3-540-75197-7, https://doi.org/10.1007/978-3-540-75197-7
  * [5] Garg, N., Sethupathy, A., Tuwani, R., NK, R., Dokania, S., Iyer, A., Gupta, A., Agrawal, S., Singh, N., Shukla, S., Kathuria, K., Badhwar, R., Kanji, R., Jain, A., Kaur, A., Nagpal, R., Bagler, G.: FlavorDB: a database of flavor molecules. Nucleic Acids Research 46(D1), D1210–D1216 (Jan 2018). https://doi.org/10.1093/nar/gkx957, https://doi.org/10.1093/nar/gkx957
  * [6] Grover, A., Leskovec, J.: node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 855–864 (2016)
  * [7] Kaminsky, J.: A journey of wine, food, and machine learning. https://www.linkedin.com/pulse/journey-wine-food-machine-learning-john-kaminsky/ (2023)
  * [8] Kustos, M., Heymann, H., Jeffery, D.W., Goodman, S., Bastian, S.E.: Intertwined: What makes food and wine pairings appropriate? Food Research International 136, 109463 (2020)
  * [9] Marin, J., Biswas, A., Ofli, F., Hynes, N., Salvador, A., Aytar, Y., Weber, I., Torralba, A.: Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images. arXiv:1810.06553 [cs] (Jul 2019), http://arxiv.org/abs/1810.06553, arXiv: 1810.06553
  * [10] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems. pp. 3111–3119 (2013)
  * [11] Molnar, C.: Interpretable Machine Learning. 2 edn. (2022), https://christophm.github.io/interpretable-ml-book
  * [12] Park, D., Kim, K., Kim, S., Spranger, M., Kang, J.: FlavorGraph: a large-scale food-chemical graph for generating food representations and recommending food pairings. Scientific Reports 11 (12 2021)
  * [13] Veselkov, K., Gonzalez, G., Aljifri, S., Galea, D., Mirnezami, R., Youssef, J., Bronstein, M., Laponogov, I.: HyperFoods: Machine intelligent mapping of cancer-beating molecules in foods. Scientific Reports 9(1), 9237 (Jul 2019). https://doi.org/10.1038/s41598-019-45349-y


11institutetext: Recod.ai Lab, Institute of Computing, University of Campinas,
Brazil 11email: {levy.chaves, alceubissoto<EMAIL_ADDRESS>22institutetext: School of Electrical and Computing Engineering, University of
Campinas, Brazil 22email<EMAIL_ADDRESS>33institutetext: Valeo.ai
Paris

# The Performance of Transferability Metrics
does not Translate to Medical Tasks

Levy Chaves 11 Alceu Bissoto 11 Eduardo Valle 2233 Sandra Avila 11

###### Abstract

Transfer learning boosts the performance of medical image analysis by enabling
deep learning (DL) on small datasets through the knowledge acquired from large
ones. As the number of DL architectures explodes, exhaustively attempting all
candidates becomes unfeasible, motivating cheaper alternatives for choosing
them. Transferability scoring methods emerge as an enticing solution, allowing
to efficiently calculate a score that correlates with the architecture
accuracy on any target dataset. However, since transferability scores have not
been evaluated on medical datasets, their use in this context remains
uncertain, preventing them from benefiting practitioners. We fill that gap in
this work, thoroughly evaluating seven transferability scores in three medical
applications, including out-of-distribution scenarios. Despite promising
results in general-purpose datasets, our results show that no transferability
score can reliably and consistently estimate target performance in medical
contexts, inviting further work in that direction.

###### Keywords:

Transferability Estimation Transferability Metrics Image Classification
Medical Applications Transfer Learning Deep Learning

##  1 Introduction

Figure 1: Transferability estimation vs. traditional empirical search. The
latter selects the best candidate model through empirical evaluation of the
target metric, thus needing a costly hyperparameter search for each candidate
model. Transferability computes instead a proxy score that correlates with the
best expected fine-tuned performance. Only the selected model (highest score)
will need hyperparameter tuning to obtain the optimized model.

Transfer learning allows, in data-limited scenarios, to leverage knowledge
obtained from larger datasets. Due to its effectiveness, it is the preferred
training method in medical image analysis [14]. Practitioners typically fine-
tune a pre-trained model for the target task. However, selecting the most
appropriate pre-trained model can significantly impact the final performance.
The growing number of architectures and datasets has led to increasingly
difficult decisions. While, with unlimited resources, it would be
theoretically possible to compare all options empirically, that approach is
too inefficient in practice. Sound empirical evaluation must often be tempered
with the designer’s experience and, often, not-so-sound intuition, prejudices,
and hearsay.

Transferability estimation promises to ease this burden, as shown in Fig. 1.
Traditional empirical selection of architectures requires optimizing the
hyper-parameters of each candidate to allow a fair comparison [7].
Transferability scoring methods, in contrast, allow efficiently selecting the
most promising model for a given target dataset without fine-tuning each
candidate. When the transferability score accurately measures the ability to
transfer knowledge between arbitrary tasks, empirical comparison of models may
be limited to a small subset of candidates.

Transferability scoring methods have shown promising results, performing well
when source and target datasets share strong similarities in classes and image
characteristics [11, 1]. However, as we will see, their behavior is much
different for target medical datasets, a situation in which the target dataset
deviates much more intensely from the source dataset as depicted in Fig. 2.

This work evaluates several transferability scoring methods in the medical
domain, including skin lesions, brain tumors, and breast cancer. We define a
comprehensive hyperparameter optimization to ensure that the fine-tuned models
are evaluated on their best capabilities. Additionally, we extend the
evaluation to investigate how transferability scores correlate with out-of-
distribution performance. We include at least one dataset considered out-of-
distribution from a source one for each medical application.

In summary, the contributions of our paper are twofold:

  * •

We extensively evaluate seven transferability scoring methods for three
distinct medical classification tasks, covering common imagery types in
medical tasks;

  * •

We design a new methodology for the medical context to account for out-of-
distribution evaluation of transferability scoring methods.

Figure 2: Both our best transferability score (N-LEEP) and the ImageNet ACC@1
metric (baseline) on the source model suffice to predict performance on target
general-purpose tasks (two left columns). For target medical tasks, the
scenario is much different, with neither scores nor raw performances being
strong predictors of transferability (two right columns). A good
transferability scorer should capture how well a transferability score
(x-axis) relates to a test performance metric (y-axis), i.e, higher values of
transferability scores predict higher values of true performance. The red line
showcases any linear trending between the scores and the accuracy on the task
for a given source model.

##  2 Transferability Scores & Related Work

An effective transferability scoring method exhibits computational efficiency
while strongly correlating with the final performance metric of a fine-tuned
model on the target dataset. Generally, the estimation of transferability
involves extracting the embeddings or predictions from the target dataset.
That extracted information is integrated with the target dataset’s ground-
truth labels to quantify the model’s transferability. Transferability scoring
methods can be categorized into feature-based (fb) and source-label-based
(sb). Source-label-based scores assume access to the source classification
head for calculating probability distribution or label predictions, whereas
feature-based scores only require source models for feature extraction. Both
methods require the true labels of the target dataset for computing the
transferability score. We summarize the transferability scoring methods,
sorted by publication date in Table 1.

Table 1: Summary of transferability scoring methods (Tr. scorer), sorted by publication date. Cat.: category; fb: feature-based; lb: label-based. Tr. scorer | Cat. |  Scorer input |  Details
---|---|---|---
H-Score [2] | fb |  source feature extractor & labels |  transferability correlates to inter-class variance and feature redundancy
NCE [22] | lb |  source classification head & labels |  negative conditional entropy between source and target labels
LEEP [15] | lb |  source classification head & labels |  log-likelihood between target labels and source model predictions
N-LEEP [13] | fb |  source feature extractor & labels |  log-likelihood between target labels and Gaussian mixture model fit to target extracted features
LogME [24] | fb |  source feature extractor & labels |  probability of target labels conditioned on target image embeddings
Regularized H-Score [11] | fb |  source feature extractor & labels |  shrinkage estimators for stable covariance
GBC [17] | fb |  source feature extractor & labels |  Bhattacharyya coeff. between multivariate Gaussians fit to each class’ feature estimating overlap with target task classes

Ibrahim et al.[11] and Agostinelli et al.[1] evaluated transferability scores
on general-purpose datasets for classification and segmentation tasks. Their
findings suggest that these scores may be unstable, and minor variations in
the experimental protocol could lead to different conclusions. N-LEEP and
LogME deliver the best transferability estimation results depending on the
experimental design of classification tasks. Our work focuses on
classification tasks in scenarios where the dataset shift is significant. The
experimental design of previous works assumes a lower dataset shift compared
to what we consider in our paper. For instance, transferring from ImageNet to
CIFAR is expected to be easier than any medical dataset due to some overlap
between target-source classes and features. Additionally, we perform thorough
hyperparameter tuning, which is essential in these works.

##  3 Materials and Methods

###  3.1 Datasets

We assess three medical classification problems. We use ISIC2019 [5] for
melanoma vs. benign classification task and PAD-UFES-20 [16] for out-of-
distribution (OOD) evaluation. BreakHis [20] is used for histopathology breast
cancer malign vs. benign sample classification and ICIAR2018 [18] for out-of-
distribution assessment. For brain tumor classification, we use BrainTumor-
Cheng [4], a four-class dataset of MRI images. We adopt the NINS [3] as the
out-of-distribution test dataset.

###  3.2 Methodology

We aim to provide a fair and concise evaluation of each transferability
scoring method described in Section 2. We restrict our analysis to pre-trained
models on the ImageNet dataset. We focus exclusively on the source model
selection scenario, which aims to identify the most suitable pre-trained model
for a given target dataset. Our methodology involves the following seven
steps:

  1. 1.

Choosing a target medical task $T$.

  2. 2.

Selecting a pre-trained model architecture $A$.

  3. 3.

Computing the in-distribution transferability score $S_{\text{id}}(M,T,A)$ for
all transferability scoring methods $M$, pre-trained model $A$, and the
training dataset of task $T$.

  4. 4.

Performing a traditional evaluation of architecture $A$ for the target task
$T$, by first optimizing the model hyperparameters on $T$’s validation dataset
using the target metric to obtain the best model $A_{\text{opt}}(T)$, and then
evaluating that metric $P_{\text{id}}(T,A)$ on $T$’s in-distribution test
dataset.

  5. 5.

Computing the out-of-distribution transferability score
$S_{\text{ood}}(M,T,A)$ for all transferability scoring methods $M$, the fine-
tuned model $A_{\text{opt}}(T)$ obtained in the previous step, and target task
$T$’s out-of-distribution test dataset (as explained in the previous
subsection).

  6. 6.

Evaluating the target metric $P_{\text{ood}}(T,A)$ of $A_{\text{opt}}(T)$ on
$T$’s out-of-distribution test dataset.

  7. 7.

For a given dataset $T$ and scoring method $M$, once steps 1-6 have been
performed for all architectures, we may compute the correlation index between
the transferability scores $S_{*}(M,T,A)$ and the traditional empirical
performance metrics $P_{*}(T,A)$ across all architectures $A$. We showcase
each one of those correlation analyses on a separate subplot of our results.

In our experiments, the target metric is always the balanced accuracy, and the
correlation index is always the Kendall’s tau, which ranges between $-$1 and
1, with positive correlations indicating higher-quality scoring methods. Zero
correlations indicate that the scoring method has no ability to predict
transferability. Negative correlations are harder to interpret: although they
suggest predictive ability, they show the scoring method is working against
its expected design.

We analyze separately the in-distribution and the out-of-distribution
analyses. As far as we know, we are the first to attempt OOD analyses on
transferability metrics.

##  4 Results

Models architectures & hyperparameter tuning. We use 10 ImageNet pre-trained
models: ResNet18 [9], ResNet34 [9], ResNet50 [9], MobileNetV2-0.5 [19],
MobileNetV2-1.0 [19], DenseNet121 [10], DenseNet161 [10], DenseNet169 [10],
EfficientNet-B0 [21], ViT-Small [6].

For hyperparameter tuning, we followed the Tuning Playbook [7] guidelines,
using Halton sequences [8] to sample candidates for the hyperparameters of
interest. In our search, we keep fixed the use of SGD as the optimizer, cosine
scheduler, 100 epochs, and batch size of 128. We search over $75$ quasi-random
combinations of learning rate in range [$10^{-4},10^{-1}$] and weight decay in
range [$10^{-6},10^{-4}$] for each model architecture, as those are the two
most critical optimization hyperparameters [12]. We run the experiments on
NVIDIA RTX 5000, RTX 8000. We select the best-performing model in the
validation set for each architecture for test evaluation. In total, we trained
$2250$ models. The source code to reproduce our experiments is available at
https://github.com/VirtualSpaceman/transfer-estimation-medical.

In-distribution. Fig. 3 shows the results for each transferability scoring
method and each model’s architecture for all medical tasks. The red line
indicates a regression line to show any tendency in the results. Table 2 shows
all investigated transferability scores for brain tumor, histopathologic, and
skin lesion classification tasks, respectively. Each row depicts one
correlation index value for that transferability scoring methods (columns). We
calculate each correlation index considering the test performance of the best-
fine-tuned model and the estimated transferability score for each
architecture.

Figure 3: Evaluation of all scores (columns) and medical datasets (rows),
showcasing the correlation between transferability scores (x-axis) and best
accuracy on test (y-axis). The linear regression lines (in red) are for
illustration only, as the correlation index employed is the non necessarily
linear Kendall’s tau, shown inside of each plot, on the top-left corner.

Methods such as LogME, N-LEEP, and NCE demonstrate varying degrees of
correlation, indicating their potential as indicators of transferability
within the same distribution. All transferability scoring methods exhibited an
unstable behavior, as the correlation index consistently shifted across
datasets. While LogME was one of the best methods for the BrainTumor-Cheng and
BreakHis datasets, it exhibited negative correlations for ISIC2019. To our
knowledge, such instability has not been observed in general-purpose computer
vision datasets. Our main hypothesis for this phenomenon relates to the
dissimilarity between source and target domains. Unlike general-purpose
computer vision datasets, which often overlap in target and label sets or
share similar features, medical transfer learning involves substantial domain
differences.

Out-of-distribution. It is easy to come across out-of-distribution sets in the
real world, as deep learning datasets often exhibit diversity and correlation
shifts [23]. We conducted additional experiments to evaluate transferability
scores’ ability to predict out-of-distribution performance.

Table 2 shows the transferability scores and correlation indexes, with
interestingly high predictive capabilities observed for label-based
transferability scoring methods. NCE and LEEP exhibited outstanding results
for both ICIAR2018 and PAD-UFES-20 datasets across all correlations, with NCE
being favored over other methods. We hypothesize that label-based methods are
prone to provide better results than feature-based for binary tasks in out-of-
distribution scenarios. As the number of classes of source dataset matches the
target one, the probabilities distributions tend to concentrate on a single
class, inflating the transferability score for binary cases.

Table 2: Kendall’s tau ($\tau_{w}$) correlation index for each transferability
scorer considering in and out-of-distribution scenarios for each medical task.

Task |  Dataset |  H-Score |  NCE |  LEEP |  N-LEEP |  LogME |  Reg. H-Score |  GBC
---|---|---|---|---|---|---|---|---
Brain Tumor | BrainTumor-cheng | 0.270 | -0.180 | 0.090 | 0.494 | 0.584 | 0.405 | 0.135
NINS | -0.333 | 0.156 | 0.200 | -0.333 | -0.289 | -0.422 | 0.200
Histopathologic | BreakHis | 0.600 | -0.156 | 0.200 | -0.244 | 0.378 | 0.200 | 0.022
ICIAR2018 | 0.333 | 0.778 | 0.778 | 0.289 | 0.289 | 0.378 | 0.156
Skin Lesion | ISIC2019 | -0.244 | 0.022 | 0.333 | -0.111 | -0.067 | -0.289 | 0.022
PAD-UFES-20 | -0.156 | 0.911 | 0.422 | -0.156 | -0.022 | -0.022 | 0.067

Hypotheses why metrics failed. Up to this point, our experiments revealed that
all transferability scoring methods present unstable quality. For example,
both NCE and LEEP excel at out-of-distribution but report poor results in in-
distribution scenarios. We hypothesize two factors that may contribute to the
failure of the methods followed by some preliminary experiments: 1) domain
shift: the domain difference between source and target datasets might cause
the failure. We fine-tuned each model on each medical dataset and evaluated
their transferability score to the validation set. Our experiment indicates
that only label-based methods excel in this scenario. So, domain shift helps
to degrade the efficiency of such scores, but it is not the main reason. 2)
number of classes: to measure the impact of the number of classes in the
transferability scores, we take the OxfordPets dataset and map the original 37
classes (dogs and cats breeds) into a binary problem (cats vs. dogs). Our
preliminary results suggest that all correlation indexes decrease, but all
metrics still present high transferability estimation capabilities.

##  5 Conclusion

Our work is the first to investigate the quality of transferability scoring
methods for medical applications. We evaluated $7$ different transferability
scoring methods in $3$ medical classification datasets, considering $10$
different architectures. Despite promising results in our out-of-distribution
experiment, the instability presented by the scores across datasets in the in-
distribution scenario lead us to recommend to practitioners not yet to rely on
transferability scores for source model selection in medical image analysis.
Our work takes one step towards reducing the need for expensive training by
selecting pre-trained models efficiently that empowers the final performance
on the target task. Such efficiency positively diminishes the carbon footprint
when performing a hyperparameter search using a subset of deep learning
architectures instead of all available.

Label-based methods shows superior results in out-of-distribution scenarios.
Out-of-distribution scores might be inflated for binary tasks due to the
distribution concentration on a single class, and the low number of classes
benefits in favor of high transferability scores. Such an issue is absent in
the available benchmarks because the general-purpose classification datasets
present many classes and consider transferring from ImageNet as standard
practice.

For future work, the analysis can be expanded to other configurations, such as
finding the most related target task for a given source model (target model’s
selection) or cross-dataset transfer evaluation. Finally, evaluating future
transferability scorers should include contexts where the difference between
source and target domains is high, such as medical. This brings opportunities
to assess the robustness of transferability scores regarding a limited amount
of samples, unbalanced labels, and low inter- and high intra-variation
classes.

Data Use. We use only publicity available medical datasets, including PAD-
UFES-20 [16], ICIAR2018 [18], BreakHis [20], BrainTumor-Cheng [4], NINS [3],
and ISIC2019 [5]. All of them are under CC BY 4.0 license, except ISIC2019 (CC
BY-NC 4.0). The data collection process is described in the original papers.

Acknowledgments. L. Chaves is funded by Becas Santander/Unicamp – HUB 2022,
Google LARA 2021, in part by the Coordenação de Aperfeiçoamento de Pessoal de
Nível Superior – Brasil (CAPES) – Finance Code 001, and FAEPEX. A. Bissoto is
funded by FAPESP (2019/19619-7, 2022/09606-8). S. Avila is funded by CNPq
315231/2020-3, FAPESP 2013/08293-7, 2020/09838-0, H.IAAC, Google LARA 2021 and
Google AIR 2022.

## References

  * [1] Agostinelli, A., Pándy, M., Uijlings, J., Mensink, T., Ferrari, V.: How stable are transferability metrics evaluations? In: European Conference on Computer Vision. Springer (2022)
  * [2] Bao, Y., Li, Y., Huang, S.L., Zhang, L., Zheng, L., Zamir, A., Guibas, L.: An information-theoretic approach to transferability in task transfer learning. In: International Conference on Image Processing (2019)
  * [3] Brima, Y., Tushar, M.H.K., Kabir, U., Islam, T.: Deep transfer learning for brain magnetic resonance image multi-class classification. arXiv preprint arXiv:2106.07333 (2021)
  * [4] Cheng, J., Huang, W., Cao, S., Yang, R., Yang, W., Yun, Z., Wang, Z., Feng, Q.: Enhanced performance of brain tumor classification via tumor region augmentation and partition. PloS one (2015)
  * [5] Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: International Symposium on Biomedical Imaging (2018)
  * [6] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
  * [7] Godbole, V., Dahl, G.E., Gilmer, J., Shallue, C.J., Nado, Z.: Deep learning tuning playbook (2023), version 1.0
  * [8] Halton, J.H.: Algorithm 247: Radical-inverse quasi-random point sequence. Commun. ACM (1964)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (2016)
  * [10] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Computer Vision and Pattern Recognition (2017)
  * [11] Ibrahim, S., Ponomareva, N., Mazumder, R.: Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2022)
  * [12] Li, H., Chaudhari, P., Yang, H., Lam, M., Ravichandran, A., Bhotika, R., Soatto, S.: Rethinking the hyperparameters for fine-tuning. In: International Conference on Learning Representations (2020)
  * [13] Li, Y., Jia, X., Sang, R., Zhu, Y., Green, B., Wang, L., Gong, B.: Ranking neural checkpoints. In: Computer Vision and Pattern Recognition (2021)
  * [14] Matsoukas, C., Haslum, J.F., Sorkhei, M., Söderberg, M., Smith, K.: What makes transfer learning work for medical images: Feature reuse & other factors. In: Conference on Computer Vision and Pattern Recognition (2022)
  * [15] Nguyen, C., Hassner, T., Seeger, M., Archambeau, C.: Leep: A new measure to evaluate transferability of learned representations. In: International Conference on Machine Learning (2020)
  * [16] Pacheco, A.G., Lima, G.R., Salomao, A.S., Krohling, B., Biral, I.P., de Angelo, G.G., Alves Jr, F.C., Esgario, J.G., Simora, A.C., Castro, P.B., et al.: Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in brief (2020)
  * [17] Pándy, M., Agostinelli, A., Uijlings, J., Ferrari, V., Mensink, T.: Transferability estimation using bhattacharyya class separability. In: Computer Vision and Pattern Recognition (2022)
  * [18] Rakhlin, A., Shvets, A., Iglovikov, V., Kalinin, A.A.: Deep convolutional neural networks for breast cancer histology image analysis. In: 15th International Conference on Image Analysis and Recognition (2018)
  * [19] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Computer Vision and Pattern Recognition (2018)
  * [20] Spanhol, F.A., Oliveira, L.S., Petitjean, C., Heutte, L.: A dataset for breast cancer histopathological image classification. IEEE Transactions on Biomedical Engineering (2016)
  * [21] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning (2019)
  * [22] Tran, A.T., Nguyen, C.V., Hassner, T.: Transferability and hardness of supervised classification tasks. In: International Conference on Computer Vision (2019)
  * [23] Ye, N., Li, K., Bai, H., Yu, R., Hong, L., Zhou, F., Li, Z., Zhu, J.: Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In: Computer Vision and Pattern Recognition (2022)
  * [24] You, K., Liu, Y., Wang, J., Long, M.: Logme: Practical assessment of pre-trained models for transfer learning. In: International Conference on Machine Learning (2021)


# Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel
Generation

William F. Godoy, Pedro Valero-Lara, Keita Teranishi, Prasanna Balaprakash,
and Jeffrey S. Vetter  Oak Ridge National LaboratoryOak RidgeTNUSA godoywf,
valerolarap,teranishik,pbalapra,and<EMAIL_ADDRESS>
(2023)

###### Abstract.

We evaluate AI-assisted generative capabilities on fundamental numerical
kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV,
Jacobi Stencil, and CG. We test the generated kernel codes for a variety of
language-supported programming models, including (1) C++ (e.g., OpenMP
[including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g.,
OpenMP [including offload] and OpenACC), (3) Python (e.g., numba, Numba, cuPy,
and pyCUDA), and (4) Julia (e.g., Threads, CUDA.jl, AMDGPU.jl, and
KernelAbstractions.jl). We use the GitHub Copilot capabilities powered by
OpenAI Codex available in Visual Studio Code as of April 2023 to generate a
vast amount of implementations given simple <kernel> \+ <programming model> \+
<optional hints> prompt variants. To quantify and compare the results, we
propose a proficiency metric around the initial 10 suggestions given for each
prompt. Results suggest that the OpenAI Codex outputs for C++ correlate with
the adoption and maturity of programming models. For example, OpenMP and CUDA
score really high, whereas HIP is still lacking. We found that prompts from
either a targeted language such as Fortran or the more general-purpose Python
can benefit from adding code keywords, while Julia prompts perform acceptably
well for its mature programming models (e.g., Threads and CUDA.jl). We expect
for these benchmarks to provide a point of reference for each programming
model’s community. Overall, understanding the convergence of large language
models, AI, and HPC is crucial due to its rapidly evolving nature and how it
is redefining human-computer interactions.

GPT; LLM, programming models; generative AI; numerical kernels; HPC; GitHub
Copilot; OpenAI Codex

This manuscript has been authored by UT-Battelle LLC under contract DE-
AC05-00OR22725 with the US Department of Energy (DOE). The publisher
acknowledges the US government license to provide public access under the DOE
Public Access Plan (https://energy.gov/downloads/doe-public-access-plan).
”Godoy, 2023. This is the author’s version of the work. It is posted here for
your personal use. Not for redistribution. The definitive version was
published in ICPP-W, https://doi.org/10.1145/3605731.3605886.”

††journalyear: 2023††copyright: licensedusgovmixed††conference: 52nd
International Conference on Parallel Processing Workshops; August 7–10, 2023;
Salt Lake City, UT, USA††booktitle: 52nd International Conference on Parallel
Processing Workshops (ICPP-W 2023), August 7–10, 2023, Salt Lake City, UT,
USA††price: 15.00††doi: 10.1145/3605731.3605886††isbn: 979-8-4007-0842-8/23/08

##  1\. Introduction

Since its initial release in 2020, the Generative Pre-trained Transformer 3
(GPT-3) (Brown et al., 2020) has signified a revolutionary step in the
evolution of human-computer interactions. Developed by
OpenAI,111https://openai.com/ GPT-3 is the third-generation prediction-based
large language model (LLM) used for several AI-generated, human-like text
applications. It has received praise for the high-quality results in several
natural language processing (NLP) tasks (Hirschberg and Manning, 2015) due in
part to the unprecedented investment ($12 million USD) and size of its
training model (175 billion parameters at 800 GB). Hence GPT-3 and its
successor GPT-4222https://openai.com/product/gpt-4 are defining several
societal questions for the near future.

As we enter the exascale computing era, which is dominated by the extreme
heterogeneity of hardware and programming models (Vetter et al., 2018), AI-
assisted code generation could play a key role in how we develop, deploy, and
test software that targets high-performance computing (HPC) systems.
Traditional human-readable code such as C++ (Stroustrup, 2013), Fortran
(Backus and Heising, 1964), Python (Van Rossum et al., 2007), and more
recently Julia (Bezanson et al., 2017), are a straightforward application for
GPT-3 LLM capabilities that would help redefine software development. Hence,
we need to understand the current state of practice, limitations, and
potential of this new technology.

Our early experiences with OpenAI Codex, which is a GPT-3 descendant, came
from using the GitHub Copilot333https://github.com/features/copilot plugin
available on Visual Studio Code444https://docs.github.com/en/copilot/getting-
started-with-github-copilot?tool=vscode to generate relevant scientific
kernels for HPC. Our goal is to assess and understand the impact of GPT-3’s
state-of-art capabilities in the overall interactive process for generating,
optimizing, building, and testing well-established mathematical HPC kernels.
For our assessment, we evaluate AXPY, general matrix-vector multiply (GEMV),
general matrix-matrix multiply (GEMM), sparse matrix–vector multiplication
(SpMV), 3D Jacobi stencil computations, and conjugate gradients (CG). We test
the generation of these kernels over a variety of programming models that
target CPU and GPU hardware. Our goal is to establish a benchmark and
understand the status of prompt engineering in the OpenAI Codex when applied
to these important kernels and programming models as the technology continues
to evolve.

The rest of the paper is structured as follows: Section 2 provides an overview
of related efforts that highlight the recent attention to these topics in the
broader area of computer science. We describe our prompt input pattern
methodology for interacting with GitHub Copilot for the kernel generation
process along with the proposed metric to score the quality of the suggested
outputs in Section 3. Section 4 presents the results of our evaluation along
with our findings on prompt trade-off options for each language, kernel,
programming model, and additional keyword inputs on the generated outputs. We
aim to understand the current status of the correctness, trade-offs, and
overall value of LLMs for HPC practitioners who wish to adopt this technology.
Conclusions and future directions are presented in Section 5. Appendix A
provides the artifact description for the reproducibility of this study.

##  2\. Background

The availability of GPT-3 has led to increasing interest in its LLM
capabilities and how they could be applied to a large variety of applications.
Brown et al. (Brown et al., 2020) introduced the seminal paper on GPT-3’s
unprecedented results, which showed strong performance in NLP interactions
such as translation, question-answering, and cloze tasks. They highlight the
use and success rates of prompt-based (Fink, 2004; Fei-Fei et al., 2006) zero,
one, and few-shot learning (FSL) techniques when only a few data examples are
available to train a model as opposed to other previously trained ML
alternatives. Wang et al. (Wang et al., 2020) presented a comprehensive review
of FSL and pointed out the “unreliable empirical risk minimizer” that, due to
its nature, compensates for the lack of supervised information by using prior
available knowledge. Floridi and Chiriatti (Floridi and Chiriatti, 2020)
provide a commentary on the nature of GPT-3, its scope, and limits while
outlining the social consequences of the “industrialization of automatic cheap
production of good, semantic artifacts.”

On the code generation side, Dehaerne et al. (Dehaerne et al., 2022) provided
a systematic review of applying ML methods in description-to-code, code-to-
code, and code-to-description studies from the past 6 years. The review
indicated limitations, including the variability in the quality of generated,
mined source code and the need and availability of large datasets (e.g.,
GitHub sources). The review also highlighted that automatically generating
data is a fast alternative for obtaining data but is only appropriate for
certain contexts, such as programming by example. Recent work has also focused
on using GitHub Copilot’s AI pair programmer, which is based on OpenAI Codex
and leverages the vast stores of source code hosted on GitHub for AI-assisted
code generation. Chen et al. (Chen et al., 2021) provided an introduction and
evaluation of Codex for its Python code-writing capabilities. They point out
the current limitations and difficulty with “docstrings describing long chains
of operations and with binding operations to variables.” A major challenge
noted is the overreliance on Codex by novice programmers. Importantly, Copilot
uses a different Codex version than the one used in our study. Nguyen and Nadi
(Nguyen and Nadi, 2022) provided an empirical evaluation of GitHub Copilot
suggestions for correctness and understandability to LeetCode questions in
Java, JavaScript, Python, and C. Shortcomings included generating further
simplified code that relies on undefined functions. Vaithilingam, Zhang, and
Glassman (Vaithilingam et al., 2022) provided a human subject study that
consisted of 24 participants on the usability of GitHub Copilot. They
concluded that Copilot did not necessarily reduce the task completion time in
common tasks, such as file editing, web scraping, and graph plotting, for
experienced Python users. Sobania et al. (Sobania et al., 2022) found little
difference between Copilot and automatic generators that use genetic
programming when applied to the PSB2 program synthesis benchmarks (Helmuth and
Kelly, 2021). Imai (Imai, 2022) provided a preliminary assessment that showed
that, although Copilot helps generate lines of code faster, the quality is
lower when compared to human pair programming. Yetistiren et al. (Yetistiren
et al., 2022) assessed the correctness, validity, and efficiency of targeting
the HumanEval problem dataset (Chen et al., 2021) with a high success rate.

From an educational perspective, Sarsa et al. (Sarsa et al., 2022) discussed
Codex’s effectiveness in generating programming exercises and explanations via
Copilot, not without addressing the need for oversight in output quality.
Finnie-Ansley et al. (Finnie-Ansley et al., 2022) reported that Codex ranks
high when compared to a class of introductory computer science students taking
programming tests. Similarly, Denny et al. (Denny et al., 2023) discussed the
pedagogical value of Copilot for testing their answers and for its prompt
nature in introductory programming questions. Wermelinger (Wermelinger, 2023)
discussed important questions about how teaching programming will evolve.
Similarly, Brennan and Lesage (Brennan and Lesage, 2023) discussed the
educational opportunities for undergraduate engineering and the need for
students to have a strong intuition for software development when using AI-
assisted tools.

Pierce et al. (Pearce et al., 2022) assessed security aspects of Copilot’s
code contributions in a large sample, and they found relatively high
vulnerability rates. They also concluded that there is a need to quantify the
limits of generated code in low-level scenarios, and Copilot must be paired
with security-aware tooling.

Overall, there is a common thread in the relevant literature: the GPT-based
tools are here to stay and will impact almost every aspect of human-computer
interaction. However, we are at an inflection point where more exploratory
research is needed when assessing AI code-generating capabilities, and their
technical, economic, and social implications must be carefully explored and
better understood. To the best of our knowledge there is no reported work on
the application of current Copilot capabilities for HPC kernels.

##  3\. Methodology

We evaluate prompt outputs for parallel programming models available in four
different languages: C++, Fortran, Python, and Julia. Our methodology has two
aspects: (1) the Copilot code suggestion generation from prompt queries based
on kernels and the parallel programming model for each language and (2) a
simple metric to evaluate the correctness of the results. As stated in their
documentation,555https://docs.github.com/en/copilot/overview-of-github-
copilot/about-github-copilot-for-individuals GitHub Copilot is “trained on all
languages that appear in public repositories. For each language, the quality
of suggestions you receive may depend on the volume and diversity of training
data for that language.” Hence, we investigate whether or not these results
correlate with the expected availability of correct programming models and
public code examples. We will use assess popularity, in particular for open-
source software (Lu et al., 2020), based on (1) the number of repositories per
language on GitHub
(GitHut666https://madnight.github.io/githut/#/pull_requests/2023/1) and (2)
the TIOBE index.777https://www.tiobe.com/tiobe-index/ Hence, C++ and Python
codes are expected to have wider availability than Julia and Fortran.
Nevertheless, because of the general-purpose nature of C++ and Python, this
metric might not be as relevant as it is for scientific and mathematical
applications written in Fortran or Julia.

We assess the outputs from each prompt and categorize based on their level of
correctness. The rest of the section describes the experiment setup and the
proposed metric to quantify the quality of the AI-generated results.

###  3.1. Experiment setup

We select a simple prompt base structure for each targeted language. Visual
Studio Code recognizes the targeted language through the appropriate support
plugin and the currently opened file based on the language extension (e.g.,
cpp, f90, py, jl) and makes it part of the prompt prefix. The rest of the
prompt is generated through a comment line complemented with optional code.
Pressing <Control> \+ <Enter> results in a separate window that contains a
maximum of 10 code suggestions. Given the highly sensitive nature of the LLM,
even one additional keyword could heavily influence the output.

(a)

(b)

Figure 1. GitHub Copilot prompt interface with optional language keywords for
(a) Fortran’s subroutine and (b) Python’s def.

Therefore, our simple prompt queries can be described by using the following
structure:

  * •

¡kernel¿ ¡programming-model¿

  * •

¡kernel¿ ¡programming-model¿ ¡optional keywords: function, subroutine, and
def¿

Table 1 shows the programming language and model combinations used for this
study. The additional post-fix optional keywords correspond to our attempt to
add more information to the prompt query to obtain better-quality suggestions.
As shown in Section 4, C++ model sensitivity can vary by using the word
function (not a language keyword), whereas Fortran and Python are consistently
sensitive to the subroutine and def keywords. We also found little sensitivity
in Julia prompts when adding a post fix (e.g., function), hence we did not add
it in this study.

Language | Programming model | Post fix
---|---|---
C++ | OpenMP (OpenMP Architecture Review Board, 2021) | offload, function
| OpenACC (OpenACC Architecture Review Board, 2020) | function
| Kokkos (Carter Edwards et al., 2014) | function
| CUDA (NVIDIA, 2022a) | function
| HIP (AMD, 2022) | function
| Thrust (NVIDIA, 2022b) | function
Fortran | OpenMP | offload, subroutine
| OpenACC | subroutine
Python | numpy (Van Der Walt et al., 2011) | def
| Numba (Lam et al., 2015) | def
| pyCUDA (Klöckner et al., 2012) | def
| cuPy (Nishino and Loomis, 2017) | def
Julia | Threads (Knopp, 2014) |
| CUDA (Besard et al., 2018) |
| AMDGPU (Samaroo et al., 2022) |
| KernelAbstractions (Churavy et al., 2022) |
Table 1. Scope of our experimental setup applied for each kernel in terms of
language and targeted parallel programming model.

###  3.2. Correctness metric

To evaluate the correctness of the generated suggestions, we propose a simple
approach that is based on our observations more than on any particular
formalism. We consider five levels of correctness and proficiency labels
between [0], or non-knowledge, and [1], or expert, when observing the
suggested answers given by Copilot as those illustrated in Figures 1a and 1b
for each combination in Table 1.

  * 0

non-knowledge: No code at all or not a single correct code.

  * 0.25

novice: One correct code, but it includes other several correct or incorrect
programming models (e.g., OpenACC suggestions in an OpenMP prompt).

  * 0.5

learner: One correct code, and there are other incorrect codes, but all of
them are using the requested programming model.

  * 0.75

proficient: All codes are correct and using the programming model requested.

  * 1

expert: Only one piece of code is provided, and it is totally correct.

##  4\. Results

We present our analysis for each selected language and kernel and discuss some
observations from our initial interaction with GitHub Copilot. As shown in
Figures 1a and 1b, for each combination listed in Table1, we obtain a set of
suggestions for each prompt. This approach enables us to evaluate the
effectiveness of OpenAI Codex in generating accurate and proficient code.
Notably, none of the experiments reached a perfect score of 1, which
illustrates the stochastic status of the current LLM when applied to a
particular area.

###  4.1. C++

Table 2 shows the results for all of our C++ experiments, and Figure 2
illustrates the results according to the different kernels and programming
models.

Prompt |  AXPY |  GEMV |  GEMM |  SpMV |  Jacobi |  CG
---|---|---|---|---|---|---
Prefix ¡kernel¿ |  |  |  |  |  |
OpenMP | 0.75 | 0.5 | 0.5 | 0.5 | 0 | 0.25
OpenMP offload | 0.5 | 0.5 | 0.5 | 0.25 | 0.25 | 0
OpenACC | 0.5 | 0 | 0.25 | 0 | 0 | 0
Kokkos | 0.5 | 0 | 0 | 0 | 0.25 | 0
CUDA | 0.75 | 0.75 | 0.75 | 0 | 0 | 0.25
HIP | 0.75 | 0 | 0 | 0 | 0.25 | 0
Thrust | 0.25 | 0 | 0 | 0 | 0 | 0
SyCL | 0.75 | 0.25 | 0 | 0 | 0 | 0
Prefix ¡kernel¿ |  |  |  |  |  |
Post fix function |  |  |  |  |  |
OpenMP | 0.75 | 0.75 | 0.75 | 0.25 | 0.25 | 0.25
OpenMP offload | 0.5 | 0.5 | 0.5 | 0.25 | 0.25 | 0
OpenACC | 0.5 | 0.5 | 0.5 | 0.25 | 0 | 0
Kokkos | 0.75 | 0.25 | 0.25 | 0 | 0.25 | 0
CUDA | 0.75 | 0.25 | 0 | 0 | 0 | 0
HIP | 0.75 | 0 | 0 | 0 | 0.25 | 0
Thrust | 0.5 | 0 | 0.25 | 0 | 0 | 0
SyCL | 0.75 | 0.5 | 0.25 | 0 | 0 | 0
Table 2. Metric assessment for GitHub Copilot’s C++ outputs using the input
prompt pattern <kernel> <programming model> (function).

Figure 2. Results for C++ kernels (top) and programming models (bottom).

As shown, the best results are achieved for the AXPY kernel. We see a clear
trend in these results (Figure 2 [left]): the more complex the kernel, the
fewer quality results are obtained. Although we see a level between learner
and proficient for the AXPY kernel (the simplest kernel), for CG (the most
complicated kernel), the level is close to non-knowledge. For the programming
models, we see better results for OpenMP and CUDA. This could be due to the
maturity of these programming models compared to others and their availability
in public code. We note that the use of function as part of the prompt is very
beneficial and increments the quality of the results in the dense matrix
cases. Sparse matrices and high-level algorithms (e.g., SpMV, Jacobi, and CG)
do not show much improvement. However, a different trend is observed for CUDA,
for which, instead of improving the level of proficiency, it actually
decreases the quality of the answer. This is perhaps because the word function
is not used for CUDA codes. In our experimentation, which is not shown here,
using the words “kernel” or “__global__” led to better code generation
quality. This is an example of how important it is to adapt the language used
for the prompts to the particularities or syntax habitually used by such a
community.

Results also show that high-level programming model prompts from Kokkos,
Thrust, and SyCL perform poorly over several kernels. We view these results as
a reflection of the user community size of these high-level abstractions. Over
several instances, we observed that many wrong answers or no answers at all
dominate as the kernel becomes more complex. Notably, large benchmark
repositories are available (e.g., HecBench (Jin, 2023, 2021)), from which some
of the responses originate.

###  4.2. Fortran

Fortran is of particular interest in this analysis owing to its importance in
HPC and scientific computing. Despite not being a mainstream language in terms
of code availability, Copilot can provide some good results because of
Fortran’s domain-specific nature and legacy.

As shown in Table 3, using an “optimized” prompt and the subroutine keyword is
particularly beneficial in this case. Not using it leads to very poor results,
with the AXPY OpenMP case being the only exception due to its simplicity and
availability. We observe a trend similar to the one we saw in the C++ case:
the more mature solutions such as OpenMP and OpenACC provide better results
for parallel codes that use Fortran.

Prompt |  AXPY |  GEMV |  GEMM |  SpMV |  Jacobi |  CG
---|---|---|---|---|---|---
Prefix ¡kernel¿ |  |  |  |  |  |
OpenMP | 0.75 | 0 | 0 | 0 | 0 | 0
OpenMP offload | 0 | 0 | 0 | 0 | 0 | 0
OpenACC | 0 | 0 | 0 | 0 | 0 | 0
Prefix ¡kernel¿ |  |  |  |  |  |
Post fix subroutine |  |  |  |  |  |
OpenMP | 0.75 | 0.25 | 0.25 | 0.5 | 0.5 | 0.25
OpenMP offload | 0.25 | 0.25 | 0.25 | 0.25 | 0.5 | 0.25
OpenACC | 0.25 | 0.25 | 0.25 | 0.25 | 0.25 | 0.25
Table 3. Metric assessment for Copilot’s outputs using the input prompt
pattern <kernel> <programming model> (subroutine) for Fortran.

Figure 3 illustrates the uniform characteristics in the responses across
kernels. Fortran is used in many legacy HPC applications, and with so many
available Fortran codes to draw from, it is not surprising that Copilot can
obtain correct kernel implementations with higher complexity kernels (e.g.,
CG, Jacobi) in addition to the simpler ones (e.g., AXPY, GEMM).

Figure 3. Results for Fortran kernels (top) and programming models (bottom).

###  4.3. Python

Python is one of the most-used, general-purpose languages in industry,
research, and AI and also has an important role in education and as a major
target of AI-generative code, as documented in Section 2. Although neither a
parallel programming model nor part of the standard, we consider numpy in our
evaluation because it is the de-facto standard for scientific computing in
Python.

Table 4 shows the results from our evaluation. Similar to Fortran, we observe
that the quality improves dramatically when adding the Python def keyword to
clarify that we are looking for functions in the language. Overall, Copilot
can generate acceptable numpy, cuPy, and pyCUDA implementations across several
kernels, but Numba falls behind. Perhaps this is an indication that cuPy and
pyCUDA are more popular alternatives in the community via lightweight layers
on top of compiled C or CUDA code, rather than using Numba’s just-in-time
approach on top of LLVM. Notably, Numba recently deprecated support for AMD
GPU hardware, hence reinforcing the idea that GPU kernels written in Python
favor CUDA-like implementations. Interestingly, successful GPU instances
(e.g., pyCUDA and cuPy) include a correct raw CUDA kernel source code as a
user-defined kernel. These instances are described in pyCUDA (Klöckner, [n.
d.]) and cuPy (Preferred Networks and Preferred Infrastructure, [n. d.])
documentation, respectively.

Prompt |  AXPY |  GEMV |  GEMM |  SpMV |  Jacobi |  CG
---|---|---|---|---|---|---
Prefix ¡kernel¿ |  |  |  |  |  |
numpy | 0.25 | 0 | 0 | 0 | 0 | 0
CuPy | 0 | 0 | 0.25 | 0 | 0 | 0
pyCUDA | 0 | 0 | 0 | 0 | 0 | 0
Numba | 0 | 0 | 0 | 0 | 0 | 0
Prefix ¡kernel¿ |  |  |  |  |  |
Post fix def |  |  |  |  |  |
numpy | 0.75 | 0.25 | 0.25 | 0.5 | 0.5 | 0.75
CuPy | 0.5 | 0.25 | 0.25 | 0.25 | 0.25 | 0.25
pyCUDA | 0.5 | 0.25 | 0.5 | 0.5 | 0.25 | 0
Numba | 0.25 | 0 | 0 | 0 | 0 | 0
Table 4. Metric assessment for Copilot’s outputs using the input prompt
pattern <kernel> <programming model> (def) for Python.

As seen in Figure 4, the resulting kernels can have varying degrees of
success, but most return at least one correct answer. This is perhaps
attributed to the wide availability of Python and numpy code in public
repositories. The trend also confirms the lack of correct Numba results, and
perhaps it is an opportunity to exploit its pure-Python nature. Although, as
recently highlighted by Kailasa et al. (Kailasa et al., 2023), writing Numba
code for complex algorithms can be as challenging as using a compiled
language.

Figure 4. Results for Python kernels (top) and programming models (bottom).

###  4.4. Julia

We included the Julia language in our experiments because of its mathematical
and performance focus. Julia provides an interesting proposition for building
a dynamic language on top of LLVM and is heavily influenced by Fortran in its
syntax for targeting scientific computing problems. Julia provides an
accessible programming model for CPU Threads, which is part of the base
language, for vendor-specific CUDA.jl and AMDGPU.jl, and for
KernelAbstraction.jl for portable kernels across vendors. In previous work
(Godoy et al., 2023), we showed promising results, albeit with some gaps, in a
simple performance comparison on CPU and GPU runs.

Table 5 and Figure 5 show the results for each kernel and programming model in
our evaluation. For each programming model, we see a proficiency level between
novice and learner for Threads (part of the base language) and CUDA.jl, which
are the most used and mature programming models. AMDGPU.jl and
KernelAbstractions.jl rank lower, which correlates with their novelty and the
more limited availability of example code that targets non-NVIDIA GPU
hardware.

Like with C++, a more complex kernel led to fewer correct results. For
example, we could not get an appropriate implementation for the CG kernel (a
multikernel algorithm). Unsurprisingly, as in the case of Fortran, Julia’s
mathematical nature allowed the OpenAI Codex to find and suggest appropriate
solutions in other kernels, despite Julia being a relatively new language with
fewer publicly available codes, especially compared to C++ and Python.
Notably, our results did not necessarily improve by adding more information to
the prompt (e.g., the function language keyword). This could be an advantage
for domain-specific syntax, as in the case of Fortran, because most existing
codes have a very targeted use with fewer words.

Prompt |  AXPY |  GEMV |  GEMM |  SpMV |  Jacobi |  CG
---|---|---|---|---|---|---
Prefix ¡kernel¿ |  |  |  |  |  |
Threads | 0.75 | 0.25 | 0.5 | 0 | 0 | 0
CUDA | 0.75 | 0.5 | 0.5 | 0.25 | 0.25 | 0
AMDGPU | 0 | 0 | 0 | 0.25 | 0 | 0
KernelAbstractions | 0.25 | 0.25 | 0.25 | 0.25 | 0.25 | 0
Table 5. Metric assessment for Copilot’s outputs using the input prompt
pattern <kernel> <programming model> for Julia.

Figure 5. Results for Julia kernels (top) and programming models (bottom).

###  4.5. Discussion

To gain insight into the performance of OpenAI Codex across different
languages and kernels, we collated the results in Figure 6. As depicted in the
graph, the complexity of the kernel directly impacts the quality of the
results obtained. In other words, it becomes increasingly challenging to
achieve acceptable results as the kernel’s complexity increases.

Figure 6. Overall results for Copilot kernels (top) and programming languages
(bottom).

We see a relatively low level of proficiency in OpenAI Codex with an average
of novice level for the languages and kernels tested. We see a slightly higher
quality for the more popular C++ and Python languages than for Julia or
Fortran. This may be related to the popularity, maturity, and accessibility of
public codes implemented in those languages. However, Fortran and Julia
provide very acceptable results. The results confirm that the question is more
about targeted quality than quantity in a specific domain.

We summarize our results as follows:

  * •

As code complexity increases, obtaining acceptable results becomes more
challenging.

  * •

Generating high-quality multistep or multikernel codes (e.g., CG) can be
difficult.

  * •

Using keywords can improve the proficiency of the answers, but it is essential
to choose the correct words that are specific and sensitive to each
programming language/model or community.

  * •

Although the popularity or accessibility of a programming language or public
code can be important, less popular languages can also provide good results
owing to their targeted nature.

##  5\. Conclusions and Future Directions

We carried out an initial study to evaluate the current capacity of OpenAI
Codex via Copilot for the generation of HPC numerical kernels that target
parallel programming models in C++, Fortran, Python, and Julia. Despite
current limitations, we believe that generative AI can have an extraordinary
and beneficial impact on HPC software development, maintenance, and education
in the future.

The research community still has several gaps to understand, and one such gap
is the need for a more comprehensive taxonomy, akin to natural language, to
evaluate accuracy and trustworthiness. Although our proposed taxonomy was
beneficial for this initial study, it is imperative that such metrics be
expanded to create a widely accepted methodology that the entire community can
use. Therefore, a standard and recognized approach is needed to ensure
uniformity when evaluating results across studies.

The emergence of LLM technologies such as GPT-3 and other generative AI tools
presents significant questions about their integration into the future HPC
software development ecosystem. For instance, can a human-in-the-loop and
compiler be incorporated to refine initial LLM suggestions? Can metadata-rich
suggestions be incorporated to facilitate a human decision-making process?
Additionally, how do significant HPC software modernization initiatives
similar to DARPA’s High Productivity Computing Systems (Dongarra et al., 2008)
or the US Department of Energy’s Exascale Computing Project (et al., 2011;
Kothe et al., 2019) incorporate these novel tools?

The automation of ecosystem features, such as building systems, packaging,
validation & verification, reproducibility, and continuous
integration/continuous deployment pipelines, could significantly impact the
HPC community. The technologies that put today’s imperfect information closer
to the human in question could redefine the educational aspects of HPC.
Therefore, it is crucial to understand how each community can leverage these
revolutionary capabilities to further advance their respective domains.

## Acknowledgment

This work is funded by Bluestone, an X-Stack project in the DOE Advanced
Scientific Computing Office with program manager Hal Finkel.

## References

  * (1)
  * AMD (2022) AMD. 2022.  AMD ROCm v5.2 Release.  https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html#amd-rocm-v5-2-release
  * Backus and Heising (1964) J. W. Backus and W. P. Heising. 1964.  Fortran.  _IEEE Transactions on Electronic Computers_ EC-13, 4 (1964), 382–385.  https://doi.org/10.1109/PGEC.1964.263818
  * Besard et al. (2018) Tim Besard, Christophe Foket, and Bjorn De Sutter. 2018\.  Effective Extensible Programming: Unleashing Julia on GPUs.  _IEEE Transactions on Parallel and Distributed Systems_ (2018).  https://doi.org/10.1109/TPDS.2018.2872064 arXiv:1712.03112 [cs.PL]
  * Bezanson et al. (2017) Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. 2017.  Julia: A Fresh Approach to Numerical Computing.  _SIAM Rev._ 59, 1 (Jan. 2017), 65–98.  https://doi.org/10.1137/141000671 arXiv:http://dx.doi.org/10.1137/141000671
  * Brennan and Lesage (2023) Robert W. Brennan and Jonathan Lesage. 2023.  Exploring the Implications of OpenAI Codex on Education for Industry 4.0. In _Service Oriented, Holonic and Multi-Agent Manufacturing Systems for Industry of the Future_ , Theodor Borangiu, Damien Trentesaux, and Paulo Leitão (Eds.). Springer International Publishing, Cham, 254–266.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Carter Edwards et al. (2014) H. Carter Edwards, Christian R. Trott, and Daniel Sunderland. 2014.  Kokkos: Enabling manycore performance portability through polymorphic memory access patterns.  _J. Parallel and Distrib. Comput._ 74, 12 (2014), 3202–3216.  https://doi.org/10.1016/j.jpdc.2014.07.003 Domain-Specific Languages and High-Level Frameworks for High-Performance Computing.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Churavy et al. (2022) Valentin Churavy, Dilum Aluthge, Lucas C Wilcox, James Schloss, Simon Byrne, Maciej Waruszewski, Julian Samaroo, Ali Ramadhan, Meredith, Simeon Schaub, Jake Bolewski, Anton Smirnov, Charles Kawczynski, Chris Hill, Jinguo Liu, Oliver Schulz, Oscar, Páll Haraldsson, Takafumi Arakaki, and Tim Besard. 2022\.  _JuliaGPU/KernelAbstractions.jl: v0.8.3_.  https://doi.org/10.5281/zenodo.6742177
  * Dehaerne et al. (2022) Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt, and Wannes Meert. 2022\.  Code Generation Using Machine Learning: A Systematic Review.  _IEEE Access_ 10 (2022), 82434–82455.  https://doi.org/10.1109/ACCESS.2022.3196347
  * Denny et al. (2023) Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023.  Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 1136–1142.  https://doi.org/10.1145/3545945.3569823
  * Dongarra et al. (2008) Jack Dongarra, Robert Graybill, William Harrod, Robert Lucas, Ewing Lusk, Piotr Luszczek, Janice Mcmahon, Allan Snavely, Jeffrey Vetter, Katherine Yelick, Sadaf Alam, Roy Campbell, Laura Carrington, Tzu-Yi Chen, Omid Khalili, Jeremy Meredith, and Mustafa Tikir. 2008.  DARPA’s HPCS Program: History, Models, Tools, Languages.  In _Advances in COMPUTERS_. Advances in Computers, Vol. 72. Elsevier, 1–100.  https://doi.org/10.1016/S0065-2458(08)00001-6
  * et al. (2011) Jack Dongarra et al. 2011\.  The International Exascale Software Project roadmap.  _The International Journal of High Performance Computing Applications_ 25, 1 (2011), 3–60.  https://doi.org/10.1177/1094342010391989 arXiv:https://doi.org/10.1177/1094342010391989
  * Fei-Fei et al. (2006) Li Fei-Fei, R. Fergus, and P. Perona. 2006.  One-shot learning of object categories.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 28, 4 (2006), 594–611.  https://doi.org/10.1109/TPAMI.2006.79
  * Fink (2004) Michael Fink. 2004\.  Object Classification from a Single Example Utilizing Class Relevance Metrics. In _Advances in Neural Information Processing Systems_ , L. Saul, Y. Weiss, and L. Bottou (Eds.), Vol. 17. MIT Press.  https://proceedings.neurips.cc/paper_files/paper/2004/file/ef1e491a766ce3127556063d49bc2f98-Paper.pdf
  * Finnie-Ansley et al. (2022) James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022\.  The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In _Proceedings of the 24th Australasian Computing Education Conference_ (Virtual Event, Australia) _(ACE ’22)_. Association for Computing Machinery, New York, NY, USA, 10–19.  https://doi.org/10.1145/3511861.3511863
  * Floridi and Chiriatti (2020) Luciano Floridi and Massimo Chiriatti. 2020.  GPT-3: Its nature, scope, limits, and consequences.  _Minds and Machines_ 30 (2020), 681–694.
  * Godoy et al. (2023) William F. Godoy, Pedro Valero-Lara, T. Elise Dettling, Christian Trefftz, Ian Jorquera, Thomas Sheehy, Ross G. Miller, Marc Gonzalez-Tallada, Jeffrey S. Vetter, and Valentin Churavy. 2023.  Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes.  arXiv:2303.06195 [cs.DC]
  * Helmuth and Kelly (2021) Thomas Helmuth and Peter Kelly. 2021.  PSB2: The Second Program Synthesis Benchmark Suite. In _Proceedings of the Genetic and Evolutionary Computation Conference_ (Lille, France) _(GECCO ’21)_. Association for Computing Machinery, New York, NY, USA, 785–794.  https://doi.org/10.1145/3449639.3459285
  * Hirschberg and Manning (2015) Julia Hirschberg and Christopher D. Manning. 2015.  Advances in natural language processing.  _Science_ 349, 6245 (2015), 261–266.  https://doi.org/10.1126/science.aaa8685 arXiv:https://www.science.org/doi/pdf/10.1126/science.aaa8685
  * Imai (2022) Saki Imai. 2022\.  Is GitHub Copilot a Substitute for Human Pair-Programming? An Empirical Study. In _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 319–321.  https://doi.org/10.1145/3510454.3522684
  * Jin (2021) Zheming Jin. 2021\.  _The Rodinia Benchmarks in SYCL_.  Technical Report. Oak Ridge National Lab.(ORNL), Oak Ridge, TN (United States).
  * Jin (2023) Zhemin Jin. 2023\.  Hecbench.  https://github.com/zjin-lcf/HeCBench.
  * Kailasa et al. (2023) Srinath Kailasa, Tingyu Wang, Lorena A. Barba, and Timo Betcke. 2023.  PyExaFMM: an exercise in designing high-performance software with Python and Numba.  arXiv:2303.08394 [cs.SE]
  * Klöckner ([n. d.]) Andreas Klöckner. [n. d.].  pycuda 2022.2.2 documentation.  https://documen.tician.de/pycuda/.  Accessed: 2023-04-20.
  * Klöckner et al. (2012) Andreas Klöckner, Nicolas Pinto, Yunsup Lee, Bryan Catanzaro, Paul Ivanov, and Ahmed Fasih. 2012\.  PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation.  _Parallel Comput._ 38, 3 (2012), 157–174.  https://doi.org/10.1016/j.parco.2011.09.001
  * Knopp (2014) Tobias Knopp. 2014\.  Experimental multi-threading support for the Julia programming language. In _2014 First Workshop for High Performance Technical Computing in Dynamic Languages_. IEEE, 1–5.
  * Kothe et al. (2019) Douglas Kothe, Stephen Lee, and Irene Qualters. 2019\.  Exascale Computing in the United States.  _Computing in Science & Engineering_ 21, 1 (2019), 17–29.  https://doi.org/10.1109/MCSE.2018.2875366
  * Lam et al. (2015) Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015\.  Numba: A LLVM-based Python JIT compiler. In _Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC_. 1–6.
  * Lu et al. (2020) Dongdong Lu, Jie Wu, Yongxiang Sheng, Peng Liu, and Mengmeng Yang. 2020.  Analysis of the popularity of programming languages in open source software communities. In _2020 International Conference on Big Data and Social Sciences (ICBDSS)_. 111–114.  https://doi.org/10.1109/ICBDSS51270.2020.00033
  * Nguyen and Nadi (2022) Nhan Nguyen and Sarah Nadi. 2022.  An Empirical Evaluation of GitHub Copilot’s Code Suggestions. In _Proceedings of the 19th International Conference on Mining Software Repositories_ (Pittsburgh, Pennsylvania) _(MSR ’22)_. Association for Computing Machinery, New York, NY, USA, 1–5.  https://doi.org/10.1145/3524842.3528470
  * Nishino and Loomis (2017) ROYUD Nishino and Shohei Hido Crissman Loomis. 2017.  Cupy: A numpy-compatible library for nvidia gpu calculations.  _31st confernce on neural information processing systems_ 151, 7 (2017).
  * NVIDIA (2022a) NVIDIA. 2022a.  CUDA Toolkit Documentation - v11.7.0.  https://developer.nvidia.com/cuda-toolkit
  * NVIDIA (2022b) NVIDIA. 2022b.  The API reference guide for Thrust, the CUDA C++ template library.  https://docs.nvidia.com/cuda/thrust/index.html
  * OpenACC Architecture Review Board (2020) OpenACC Architecture Review Board. 2020\.  OpenACC Application Program Interface Version 3.1.  https://www.openacc.org/sites/default/files/inline-images/Specification/OpenACC-3.1-final.pdf
  * OpenMP Architecture Review Board (2021) OpenMP Architecture Review Board. 2021\.  OpenMP Application Program Interface Version 5.2.  https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf
  * Pearce et al. (2022) Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022\.  Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In _2022 IEEE Symposium on Security and Privacy (SP)_. 754–768.  https://doi.org/10.1109/SP46214.2022.9833571
  * Preferred Networks and Preferred Infrastructure ([n. d.]) Inc. Preferred Networks and Inc. Preferred Infrastructure. [n. d.].  CuPy – NumPy & SciPy for GPU.  https://docs.cupy.dev/en/stable/.  Accessed: 2023-04-20.
  * Samaroo et al. (2022) Julian Samaroo, Valentin Churavy, Wiktor Phillips, Ali Ramadhan, Jason Barmparesos, Julia TagBot, Ludovic Räss, Michel Schanen, Tim Besard, Anton Smirnov, Takafumi Arakaki, Stephan Antholzer, Alessandro, Chris Elrod, Matin Raayai, and Tom Hu. 2022\.  _JuliaGPU/AMDGPU.jl: v0.4.1_.  https://doi.org/10.5281/zenodo.6949520
  * Sarsa et al. (2022) Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022\.  Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models _(ICER ’22)_. Association for Computing Machinery, New York, NY, USA, 27–43.  https://doi.org/10.1145/3501385.3543957
  * Sobania et al. (2022) Dominik Sobania, Martin Briesch, and Franz Rothlauf. 2022\.  Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of Github Copilot and Genetic Programming. In _Proceedings of the Genetic and Evolutionary Computation Conference_ (Boston, Massachusetts) _(GECCO ’22)_. Association for Computing Machinery, New York, NY, USA, 1019–1027.  https://doi.org/10.1145/3512290.3528700
  * Stroustrup (2013) Bjarne Stroustrup. 2013\.  _The C++ programming language_.  Pearson Education.
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022\.  Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI EA ’22)_. Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages.  https://doi.org/10.1145/3491101.3519665
  * Van Der Walt et al. (2011) Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. 2011\.  The NumPy array: a structure for efficient numerical computation.  _Computing in science & engineering_ 13, 2 (2011), 22–30.
  * Van Rossum et al. (2007) Guido Van Rossum et al. 2007\.  Python Programming Language.. In _USENIX annual technical conference_ , Vol. 41. Santa Clara, CA, 1–36.
  * Vetter et al. (2018) J. S. Vetter, R. Brightwell, M. Gokhale, P. McCormick, R. Ross, J. Shalf, K. Antypas, D. Donofrio, T. Humble, C. Schuman, B. Van Essen, S. Yoo, A. Aiken, D. Bernholdt, S. Byna, K. Cameron, F. Cappello, B. Chapman, A. Chien, M. Hall, R. Hartman-Baker, Z. Lan, M. Lang, J. Leidel, S. Li, R. Lucas, J. Mellor-Crummey, P. Peltz Jr., T. Peterka, M. Strout, and J. Wilke. 2018.  _Extreme Heterogeneity 2018 - Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity_.  Technical Report. USDOE Office of Science (SC) (United States).  https://doi.org/10.2172/1473756
  * Wang et al. (2020) Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020\.  Generalizing from a Few Examples: A Survey on Few-Shot Learning.  _ACM Comput. Surv._ 53, 3, Article 63 (jun 2020), 34 pages.  https://doi.org/10.1145/3386252
  * Wermelinger (2023) Michel Wermelinger. 2023\.  Using GitHub Copilot to Solve Simple Programming Problems. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ (Toronto ON, Canada) _(SIGCSE 2023)_. Association for Computing Machinery, New York, NY, USA, 172–178.  https://doi.org/10.1145/3545945.3569830
  * Yetistiren et al. (2022) Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022.  Assessing the Quality of GitHub Copilot’s Code Generation. In _Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering_ (Singapore, Singapore) _(PROMISE 2022)_. Association for Computing Machinery, New York, NY, USA, 62–71.  https://doi.org/10.1145/3558489.3559072

##  Appendix A Artifact Description

The entire prompt input and resulting output sets are publicly available at:
https://github.com/keitaTN/Copilot-hpc-kernels. Due to the rapid evolution and
statistical nature of these technologies reproducibility and replicability of
the present results is a challenging aspect that we expect to improve over
time, but has no guarantees as of today. Hence, it’s imperative that the
reproducibility information is provided.

Some additional information:

  * •

Experiments used the Visual Studio Code GitHub Copilot plugin service on three
separate Linux systems using Ubuntu 22.04

  * •

Experiments were carried between April 14th and April 21st of 2023.

  * •

The raw dataset are identified by
<kernel>/<language>/<kernel>_outputs.<ext>
directory and file structure.

  * •

Each Prompt used in this study is identified with the line comment
Prompt: <kernel> <programming model> <additional text>.