# GPT-too: A language-model-first approach for AMR-to-text generation

Manuel Mager[1] Ramón Fernandez Astudillo[2] Tahira Naseem[2] Md Arafat
Sultan[2] Young-Suk Lee[2] Radu Florian[2] Salim Roukos[2]
[1] Institute for Natural Language Processing,
University of Stuttgart, Germany
[2] IBM Research AI, Yorktown Heights, NY 10598, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
{tnaseem<EMAIL_ADDRESS>This research was done during an internship at
IBM Research AI.

###### Abstract

Abstract Meaning Representations (AMRs) are broad-coverage sentence-level
semantic graphs. Existing approaches to generating text from AMR have focused
on training sequence-to-sequence or graph-to-sequence models on AMR annotated
data only. In this paper, we propose an alternative approach that combines a
strong pre-trained language model with cycle consistency-based re-scoring.
Despite the simplicity of the approach, our experimental results show these
models outperform all previous techniques on the English LDC2017T10 dataset,
including the recent use of transformer architectures. In addition to the
standard evaluation metrics, we provide human evaluation experiments that
further substantiate the strength of our approach.

##  1 Introduction

Abstract Meaning Representation (AMR) Banarescu et al. (2013) is a rooted,
directed, acyclic graph with labeled edges (relations) and nodes (concepts)
expressing “who is doing what to whom”. AMR-to-text generates sentences
representing the semantics underlying an AMR graph.

Initial works in AMR-to-text used transducers Flanigan et al. (2016), phrase-
based machine translation Pourdamghani et al. (2016) and neural sequence-to-
sequence (seq2seq) models with linearized graphs Konstas et al. (2017). Cao
and Clark (2019) leverage constituency parsing for generation. Beck et al.
(2018) improve upon prior RNN graph encoding Song et al. (2018) with Levi
Graph Transformations. Damonte and Cohen (2019) compare multiple
representations and find graph encoders to be the best. Guo et al. (2019) use
RNN graph encoders with dense graph convolutional encoding. Ribeiro et al.
(2019) use RNN encoders with dual graph representations. Transformer-based
seq2seq Vaswani et al. (2017) was first applied to AMR-to-text in Sinh and Le
Minh (2019). Zhu et al. (2019) greatly improve over the prior state-of-the-art
by modifying self-attention to account for AMR graph structure. Using
transformers has also been recently explored by Wang et al. (2020) who propose
a mutli-head graph attention mechanism and by Cai and Lam (2020) who propose a
graph transformer architecture.

Pre-trained transformer representations Radford et al. (2018); Devlin et al.
(2019); Radford et al. (2019) use transfer learning to yield powerful language
models that considerably outperform the prior art. They have also shown great
success when fine-tuned to particular text generation tasks See et al. (2019);
Zhang et al. (2019); Keskar et al. (2019). Given their success, it would be
desirable to apply pre-trained transformer models to a graph-to-text task like
AMR-to-text, but the need for graph encoding precludes in principle that
option. Feeding the network with some sequential representation of the graph,
such as a topological sorting, looses some of the graphs representational
power. Complex graph annotations, such as AMR, also contain many special
symbols and special constructs that departure from natural language and may by
not interpretable by a pre-trained language model.

In this paper we explore the possibility of directly fine-tuning a pre-trained
transformer language model on a sequential representation of AMR graphs,
despite the expected difficulties listed above. For this we re-purpose a GPT-2
language model Radford et al. (2019) to yield an AMR-to-text system. We show
that it is surprisingly easy to fine-tune GPT-2 to learn AMR graph to text
mapping that outperforms the previous state-of-the-art on automatic evaluation
metrics. Since a single graph AMR, graph corresponds to multiple sentences
with the same meaning, we also provide human evaluation and semantic
similarity metric results Zhang et al. (2020) which are less dependent on
reference text. Human evaluation and semantic similarity results highlight the
positive impact of a strong language model strategy. Finally we also introduce
a simple re-scoring technique based on cycle-consistency that further improves
performance.

##  2 Fine-tuning GPT-2 for conditional language generation

In order to fine-tune a generative model (GPT-2; Radford et al. (2019)) for
conditional text generation, prior works fine-tune the language model to
predict target text starting from the additional source text as context. In
our experiments, we found it beneficial to fine-tune on the joint distribution
of AMR and text instead i.e. also reconstruct the source. Given a tokenized
sentence $w_{1}\cdots w_{N}$ and the sequential AMR representation
$a_{1}\cdots a_{M}$ we maximized the joint probability

$\displaystyle p_{\mbox{GPT-2}}(\mathbf{w},\mathbf{a})=$
$\displaystyle\prod_{j=1}^{N}p_{\mbox{GPT-2}}(w_{j}\mid w_{1:j-1},a_{1:M})$
$\displaystyle\cdot\prod_{i=1}^{M}p_{\mbox{GPT-2}}(a_{i}\mid a_{1:i-1})$

A special separator token is added to mark the end of the sequential AMR
representation. Special AMR symbols that should not be interpreted literally
are assigned tokens from the GPT-2 unused token list. In addition to this, we
also observed that freezing the input embeddings when fine-tuning had positive
impact in performance.

At test time, we provide the AMR as context as in conventional conditional
text generation:

$\hat{w}_{j}=\arg\max_{w_{j}}\\{p_{\mbox{GPT-2}}(w_{j}\mid
w_{1:j-1},a_{1:M})\\}$

##  3 Re-scoring via Cycle Consistency

The general idea of cycle consistency is to assess the quality of a system’s
output based on how well an external ‘reverse’ system can reconstruct the
input from it. In previous works, cycle-consistency based losses have been
used as part of the training objective in machine translation He et al. (2016)
and speech recognition Hori et al. (2019). It has also been used for filtering
synthetic training data for question answering Alberti et al. (2019). Here we
propose the use of a cycle consistency measure to re-score the system outputs.

In particular, we take the top $k$ sentences generated by our system from each
gold AMR graph and parse them using an off-the-shelf parser to obtain a second
AMR graph. We then re-score each sentence using the standard AMR parsing
metric Smatch Cai and Knight (2013) by comparing the gold and parsed AMRs.

##  4 Experimental setup

Following Previous works on AMR-to-text, we Use the standard LDC2017T10 AMR
corpus for evaluation of the proposed model. This Corpus contains 36,521
training instances of AMR graphs in PENMAN notation and the corresponding
texts. It also includes 1368 and 1371 development and test instances,
respectively. We tokenize each input text using The JAMR toolkit Flanigan et
al. (2014). The concatenation of an AMR graph and the corresponding text is
split into words, special symbols and sub-word units using the GPT-2
tokenizer. We add all arc labels seen in the training set and the root node
:root to the vocabulary of the GPT-2model, but we freeze the embedding layer
for training. We use the Hugging Face implementation of Wolf et al. (2019) for
GPT-2 small (GPT-2S), medium (GPT-2M) and large (GPT-2L). Fine-tuning
converges after $6$ epochs, which takes just a few hours on a V100 GPU111Code
for this paper is available at: https://github.com/IBM/GPT-too-AMR2text. For
cycle-consistency re-scoring we use an implementation of Naseem et al. (2019)
in PyTorch. For re-scoring experiments, we use a beam size of 15.

#### AMR input representation.

we test three variants of AMR representation. First, a depth-first search
(DFS) through the graph following Konstas et al. (2017), where the input
sequence is the path followed in the graph. Second, to see if GPT-2 is in fact
learning from the graph structure, we remove all the edges from the DFS,
keeping only the concept nodes. This has the effect of removing the relation
information between concepts, such as subject/object relations. As a third
option, we use the PENMAN representation without any modification. The three
input representations are illustrated below:

Nodes |  recommend advocate-01 it vigorous
---|---
DFS |  recommend :ARG1 advocate-01 :ARG1 it :manner vigorous
Penman |  (r / recommend-01 :ARG1 (a / advocate-01 :ARG1 (i / it) :manner (v / vigorous)))

#### Decoding.

For generation, we experiment with greedy decoding, beam search, and nucleus
sampling Holtzman et al. (2019). For beam search, we explore beam sizes of
$5$, $10$ and $15$. As the system, in some cases, produces repetitive output
at the end of the text, we additionally perform a post-processing step to
remove these occurrences.

#### Metrics.

We considered the three automatic evaluation metrics commonly used in previous
works. We compute BLEU Papineni et al. (2002) using SacreBLEU Ma et al.
(2019). We compute chrF++ Popović (2017) using both SacreBLEU and the scripts
used by authors of the baseline systems. We compute METEOR Banerjee and Lavie
(2005) with the default values for English of the CMU
implementation.222https://www.cs.cmu.edu/~alavie/METEOR

In addition to the standard automatic metrics, we also carry out human
evaluation experiments and use the semantic similarity metric BERTScore Zhang
et al. (2020). Both metrics arguably have less dependency on the surface
symbols of the reference text used for evaluation. This is particularly
relevant for the AMR-to-text task, since one single AMR graph corresponds to
multiple sentences with the same semantic meaning. Conventional metrics for
AMR-to-text are are strongly influenced by surface symbols and thus do not
capture well the ability of the system to produce a diverse sentences with
same underlying semantics.

Human evaluations are carried out by three professional annotators on $51$
randomly selected sentences from the $1371$ test sentences, on a 6 point
scale, ranging from 0 to 5.

  * •

0=Exceptionally poor (No useful information is conveyed at all.)

  * •

1=Poor (Fundamental errors in grammar and vocabulary make it difficult to
understand the meaning.)

  * •

2=Not good enough (Errors in grammar, vocabulary and style make it difficult
to understand the meaning.)

  * •

3=Good enough (There are errors in the text, but I am reasonably confident
that I understand the meaning.)

  * •

4=Very good (There may be minor errors in the text, but I am very confident
that I understand the meaning.)

  * •

5=Excellent (The information is presented clearly and with appropriate
grammar, vocabulary and style.)

For each system, scores from all annotators are averaged to compute a single
score. Inter-annotator agreement was $0.7$ when measured by Pearson
correlation coefficient.

Our system produces de-tokenized cased output after BPE decoding, whereas
previous systems produce traditional tokenized lower-cased output. Therefore,
we lowercase and tokenize our system outputs to have fair comparisons with
previous systems.

Model | Input | BLEU | chrF++
---|---|---|---
GPT-2S Rec. | Only nodes AMR | 9.45 | 41.59
GPT-2S Rec. | Lin. AMR w/o edges. | 11.35 | 43.25
GPT-2S Rec. | Lin. AMR w/edges. | 20.14 | 53.12
GPT-2S Rec. | Penman AMR | 22.37 | 53.92
GPT-2M Rec. | Lin. AMR w/edges. | 22.86 | 55.04
GPT-2M Rec. | Penman AMR | 27.99 | 61.26
Table 1: Results on the LDC2017T10 development set using GPT-2 S(mall) and M(edium) with Rec(onstruction) loss (see §2) for different AMR representations (see §4). Approach | Decoding | BLEU | chrF++
---|---|---|---
GPT-2M Conditional | Greedy | 25.73 | 57.2
GPT-2M Rec. | Greedy | 30.41 | 61.36
GPT-2M Rec. | BEAM | 31.8 | 62.56
GPT-2M Rec. | BEAM 10 | 32.32 | 62.79
GPT-2M Rec. | Sampling | 28.75 | 61.19
Table 2: Results on the LDC2017T10 development set. Rec(onstruction) uses the
AMR reconstruction term (see §2) whereas Conditional does not.

###  4.1 Results

System | Performance
---|---
| BLEU | Meteor | chrF++
Beck et al. (2018) | 23.30 | - | 50.40
Damonte and Cohen (2019) | 24.54 | 24.07 | -
Guo et al. (2019) | 27.60 | - | 57.30
Cao and Clark (2019) | 26.80 | - | -
Sinh and Le Minh (2019) | 18.36 | - | -
Ribeiro et al. (2019) | 27.87 | 33.21 | -
Cai and Lam (2020) | 29.80 | 35.10 | 59.4
Zhu et al. (2019) | 31.82 | 36.38 | 64.05
GPT-2M Rec. | $32.10^{\blacklozenge}$ | $35.86^{\Diamond}$ | $61.81^{\blacklozenge}$
GPT-2L Rec. | $32.47^{\blacklozenge}$ | $36.80^{\Diamond}$ | $62.88^{\blacklozenge}$
GPT-2M Rec. re-scoring | $32.98^{\blacklozenge}$ | $37.33^{\Diamond}$ | $63.09^{\blacklozenge}$
GPT-2L Rec. re-scoring | 33.02◆ | 37.68◇ | 63.89□
Table 3: Results on the LDC2017T10 test set for best performing models
compared to other results reported in the literature. ◆ indicates statistical
significance at $(P<.01)$, ◇ at $(P<0.05)$ and □, not significant. All
significance tests are with respect to (Zhu et al., 2019).

Regarding the type of AMR representation, as shown in Table 1, using directly
the PENMAN notation for AMR representation leads to the best results
outperforming DFS. Edge information, indicating relations between concepts,
seems also to play a fundamental role since its absence strongly decreases
performance in both DFS and PENMAN representations. Penman notation was chosen
for the rest of the experiments.

The impact of the use of a reconstruction term explained in §2 is shown in
Table 2. The model trained using this additional term achieves $30.41$ BLEU
and $61.36$ chrF++, as opposed to $25.73$ BLEU and $57.2$ chrF++ without the
term. We therefore use a reconstruction term training in the rest of the
experiments.

Beam search improves system performance greatly over the greedy baseline with
$1.91$ BLEU points (see Table 2). With beam size $10$, we obtain $32.32$ BLEU
and $62.79$ chrF++. With nucleus sampling at a cumulative probability mass of
$0.9$, performance drops to $28.75$ BLEU and $61.19$ chrF++. Finally, cycle-
consistency re-ranking of the beam search outputs improves performance
($33.57$ BLEU, $64.86$ chrF++) over the one best output.

System | LDC2017T10
---|---
| Human Eval. | SemSim
| Avg. | P45 | F1
Guo et al. (2019) | $2.48$ | 15.69% | 92.68
Ribeiro et al. (2019) | $2.42$ | 16.37% | 92.63
Zhu et al. (2019) | $2.61$ | 20.26% | 93.31
GPT-2M Rec. | $3.03$ | 37.91% | 94.55
GPT-2L Rec. | 3.04 | 41.83% | 94.63
Table 4: Human evaluation and semantic similarity (SemSim) results on the LDC2017T10 test set. Human evaluations (Human Eval.) show the average (Avg.) of scores (0 to 5) and the ratio of sentence evaluated between 4 and 5 (P45). All results for human evaluation are on $51$ randomly selected sentences and statistically significant at $(P<0.05)$. SemSim results are significant at $(P<0.01)$. All significance tests refer to a comparison with Zhu et al. (2019). | System |  Generated text
---|---|---
(1) | REF: |  the doctors gave her medication and it ’s made her much better .
| G2S: |  the doctor gives her medications and they make her much better .
| Transf: |  doctors give her medications and make her much better .
| Our: |  the doctor gave her the medication and made her feel much better.
| Our R.: |  the doctor gave her the medication and made her ” much better ” .
(2) | REF: |  at the state scientific center of applied microbiology there is every kind of deadly bacteria that was studied for use in the secret biological weapons program of the soviet union .
| G2S: |  there are every kind of killing <unk> in the state scientific center of applied microbiology to use themselves for soviet union ’s secret biological weapons programs .
| Transf: |  there is every kind of bacterium , which is studied in using bacterium for the soviet union secret biological weapons program .
| Our: |  every kind of bacterium that was studied was found at the state scientific center of applied microbiology and was used in soviet secret weapons programs for biological weapons of biology .
| Our R.: |  every kind of bacterium that has been studied and used in soviet secret programs for biological weapons has been in the state scientific center of applied microbiology .
(3) | REF: |  among the nations that have not signed the treaty only india and israel would qualify for admission to the nsg under the israeli proposal .
| G2S: |  only one of the nations who do not sign the treaty are qualified for their proposal to admit the nsg .
| Transf: |  india and israel are only qualified for the nations that do not sign the treaty , but they admitted to the nsg .
| Our: |  india and israel are the only countries eligible to admit to the nsg by proposing a treaty .
| Our R.: |  only india and israel are eligible to admit to the nsg by proposing a treaty .
Table 5: Output examples from four systems of the LDC2017T10 dataset. REF
stands for reference, G2S for (Guo et al., 2019) and Transf. for (Zhu et al.,
2019). Our is the top beam output for GPT-2L and Our R. is with re-scoring.

Table 3 compares the best GPT-2M and GPT-2L results, fine-tuned using the
reconstruction term and PENMAN notation. For all scores we test statistical
significance with a standard two-tailed student t-test. Our model achieves a
large improvement of $1.2$ BLEU and $1.3$ METEOR scores over the previous
state-of-the-art model using GPT-2L and re-scoring. For chrF++, we get
different scores from SacreBLEU and the scripts provided by the authors of our
baseline systems, achieving comparable results with the former ($63.89$), and
improving over the best score with the latter ($65.01$) $(P<.01)$.

Table 4 shows human Evaluation results and semantic similarity scores of
GPT-2L and GPT-2M compared to Zhu et al. (2019); Ribeiro et al. (2019); Guo et
al. (2019). Our approach produces a large number of high-quality sentences
with $41.8\%$, a significant gain over the previous best system ($20.26\%$).
Regarding semantic similarity, prior art methods show relatively close scores,
a $0.9$ points difference, while GPT-2L Rec. improves $1.6$ points over the
best of these models. It should be noted that differences with Zhu et al.
(2019) for GPT-2L Rec. are statistically significantly with $P<.05$, while
differences for GPT-2M Rec are not significant due to the small sample size.

In Table 5 we show three nontrivial examples, where we compare our system
outputs with those of previous work. In the first example, the reference
sentence contains a grammatical error. Our system not only generates the
correct output, but also corrects the error in the reference. The proposed
system can generate fluent long sentences as shown in example 2. The third
example shows a sentence where all systems including ours fail to generate a
correct text.

###  4.2 Discussion

Due to the large amounts of data they are trained on, pre-trained transformer
language models can be expected to generate fluent and diverse text See et al.
(2019). It should however be highlighted that fine-tuned GPT-2 learns to
produce not only fluent but also adequate text, despite using a sequential
representation of an AMR graph as input. As shown in the experimental setup,
encoding of relations plays as well a fundamental role in AMR-to-text
performance, indicating that GPT-2 attains a fine-grained understanding of the
underlying semantics to reach state of the art performance.

While a sequence of PENMAN notation tokens is far from an optimal encoding of
a graph, it is noteworthy how far performance-wise current strong language
models can go. Furthermore, It is likely that standard metrics (BLEU, Meteor,
chrF++) that rely on a reference text do not properly reflect AMR-to-text
quality. An AMR graph corresponds to multiple sentences with the same
semantics and these measures are likely biased towards the single available
reference. In metrics that are less influenced by the reference text such as
human evaluation and semantic similarity, the proposed system shows a larger
improvement over the previous systems with close to $50\%$ of the generated
sentences considered excellent or good.

Finally it is worth considering that leveraging pre-trained transformers
greatly expands the vocabulary available on AMR-to-text systems. A single AMR
graph can correspond to multiple sentences with markedly different surface
realizations, but manual annotation of AMR is a time consuming task.
Approaches like the one proposed may be a simple solution for generation of
diverse text data for AMR parser training or other applications were diversity
play a role.

##  5 Conclusions

In this work, we present a language model-based approach for the AMR-to-text
generation task. We show that a strong pre-trained transformer language model
(GPT-2) can be fine-tuned to generate text directly from the PENMAN notation
of an AMR graph. Comparison with state-of-the-art models in BLUE, chrF++,
METEOR as well as SemSim and human evaluation metrics show that while simple,
this approach can outperform existing methods including methods training
transformers from scratch. We also show that cycle consistency-based re-
scoring using a conventional AMR parser and the Smatch metric can notably
improve the results. Future work will focus on incorporating better encoding
of the AMR graph into the current system and exploring data augmentation
techniques leveraging the proposed approach.

## Acknowledgments

We thank the reviewers for their valuable suggestions. We would also like to
thank Chunchuan Lyu for his valuable feedback and help.

## References

  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019\.  Synthetic QA corpora generation with roundtrip consistency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013\.  Abstract meaning representation for sembanking.  In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_ , pages 178–186.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Beck et al. (2018) Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018.  Graph-to-sequence learning using gated graph neural networks.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 273–283.
  * Cai and Lam (2020) Deng Cai and Wai Lam. 2020.  Graph transformer for graph-to-sequence learning.  In _34th AAAI conference on artificial intelligence_.
  * Cai and Knight (2013) Shu Cai and Kevin Knight. 2013.  Smatch: an evaluation metric for semantic feature structures.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 748–752.
  * Cao and Clark (2019) Kris Cao and Stephen Clark. 2019.  Factorising amr generation through syntax.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2157–2163.
  * Damonte and Cohen (2019) Marco Damonte and Shay B Cohen. 2019.  Structural neural encoders for amr-to-text generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3649–3658.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Flanigan et al. (2016) Jeffrey Flanigan, Chris Dyer, Noah A Smith, and Jaime Carbonell. 2016.  Generation from abstract meaning representation using tree transducers.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies_ , pages 731–739.
  * Flanigan et al. (2014) Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A Smith. 2014\.  A discriminative graph-based parser for the abstract meaning representation.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1426–1436.
  * Guo et al. (2019) Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019.  Densely connected graph convolutional networks for graph-to-sequence learning.  _Transactions of the Association for Computational Linguistics_ , 7:297–312.
  * He et al. (2016) Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016.  Dual learning for machine translation.  In _Advances in Neural Information Processing Systems_ , pages 820–828.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_.
  * Hori et al. (2019) Takaaki Hori, Ramon Astudillo, Tomoki Hayashi, Yu Zhang, Shinji Watanabe, and Jonathan Le Roux. 2019.  Cycle-consistency training for end-to-end speech recognition.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6271–6275. IEEE.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_.
  * Konstas et al. (2017) Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017.  Neural amr: Sequence-to-sequence models for parsing and generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 146–157.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 62–90.
  * Naseem et al. (2019) Tahira Naseem, Abhishek Shah, Hui Wan, Radu Florian, Salim Roukos, and Miguel Ballesteros. 2019.  Rewarding Smatch: Transition-based AMR parsing with reinforcement learning.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Popović (2017) Maja Popović. 2017.  chrf++: words helping character n-grams.  In _Proceedings of the second conference on machine translation_ , pages 612–618.
  * Pourdamghani et al. (2016) Nima Pourdamghani, Kevin Knight, and Ulf Hermjakob. 2016.  Generating english from abstract meaning representations.  In _Proceedings of the 9th international natural language generation conference_ , pages 21–25.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training. 2018.  _URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/language-unsupervised/language_understanding_paper. pdf_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Ribeiro et al. (2019) Leonardo FR Ribeiro, Claire Gardent, and Iryna Gurevych. 2019.  Enhancing amr-to-text generation with dual graph representations.  _arXiv preprint arXiv:1909.00352_.
  * See et al. (2019) Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019.  Do massively pretrained language models make better storytellers?  _arXiv preprint arXiv:1909.10705_.
  * Sinh and Le Minh (2019) Vu Trong Sinh and Nguyen Le Minh. 2019.  A study on self-attention mechanism for amr-to-text generation.  In _International Conference on Applications of Natural Language to Information Systems_ , pages 321–328. Springer.
  * Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018.  A graph-to-sequence model for amr-to-text generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1616–1626.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2020) Tianming Wang, Xiaojun Wan, and Hanqi Jin. 2020.  Amr-to-text generation with graph transformer.  _Transactions of the Association for Computational Linguistics_ , 8:19–33.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2019) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019.  Dialogpt: Large-scale generative pre-training for conversational response generation.  _arXiv preprint arXiv:1911.00536_.
  * Zhu et al. (2019) Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019.  Modeling graph structure in transformer for better amr-to-text generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5462–5471.


# Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-
based Large Language Models

Chee Wei Tan1 Shangxin Guo2 Man Fai Wong2&Ching Nam Hang2 1Nanyang
Technological University
2City University of Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper presents an AI-assisted programming tool called Copilot for Xcode
for program composition and design to support human software developers. By
seamlessly integrating cloud-based Large Language Models (LLM) with Apple’s
local development environment, Xcode, this tool enhances productivity and
unleashes creativity for software development in Apple software ecosystem
(e.g., iOS apps, macOS). Leveraging advanced natural language processing (NLP)
techniques, Copilot for Xcode effectively processes source code tokens and
patterns within code repositories, enabling features such as code generation,
autocompletion, documentation, and error detection. Software developers can
also query and make “small” decisions for program composition, some of which
can be made simultaneously, and this is facilitated through prompt engineering
in a chat interface of Copilot for Xcode. Finally, we present simple case
studies as evidence of the effectiveness of utilizing NLP in Xcode to prompt
popular LLM services like OpenAI ChatGPT for program composition and design.

Figure 1: An overview of the AI-assisted programming application, Copilot for
Xcode modeled as an intermediary software entity to connect user requests
(e.g., prompt tokens) with cloud-based large language models.

##  1 Introduction

The field of natural language processing (NLP) has witnessed remarkable
achievements through the use of large language models (LLMs). These models
exhibit remarkable skills in understanding and generating natural languages.
Additionally, they employ feedback mechanisms, such as rewards or penalties,
to improve their comprehension and fine-tune their future performance
Christiano et al. (2017); Ouyang et al. (2022). The application of LLMs to AI-
assisted programming has recently attracted considerable attention Rajamani
(2022); Wong et al. (2023), as it offers the possibility to embed advanced
conversational agents in software development Li et al. (2022b); Chen et al.
(2021). In fact, the emergence of LLM-driven tools like ChatGPT (Chat
Generative Pre-Trained Transformer), Github Copilot, DeepMind’s AlphaCode
resonates with the visionary ideas presented in Edsger W. Dijkstra’s seminal
paper in Dijkstra (transcribed 2007), illustrating the transformative
potential of computers in facilitating a seamless integration of code and
creativity. By surpassing the boundaries of debugging, AI-assisted programming
tools can embrace the harmonious combination of program composition and
elegant design Dijkstra (transcribed 2007).

Since the work in Dijkstra (transcribed 2007), one of the earliest AI-assisted
programming tool is the MIT programmer’s apprentice, which aimed to simulate a
knowledgeable junior programmer and utilized natural language processing to
acquire understanding of programming patterns, clichés, and interactions
Waters (1982); Rich and Waters (1988). The “MIT programmer’s apprentice”
played a pioneering role in introducing revolutionary concepts such as code
generation (e.g., see Handsaker (1982)) and an early form of ”prompt
engineering” (e.g., see Rich et al. (1978)). These advancements were driven by
the recognition of computer programming as a systematic process of abstraction
and simplification Dijkstra (transcribed 2007); Rich and Waters (1982).

AI-assisted programming improves software productivity by automating tasks,
detects errors, enhances code quality, promotes usability, improves
reliability and accelerates the overall software development cycles Wong et
al. (2023). Rather than replacing human programmers, these tools empower them
to unleash their creative potential. By automating repetitive and mundane
tasks, AI-assisted programming frees up valuable time and mental energy for
human programmers to focus on innovative problem-solving and designing elegant
solutions with the help of predictive analysis. Furthermore, by incorporating
natural language processing capabilities (i.e., via prompt engineering), these
tools enable human programmers to interact with software systems in a more
intuitive and human-like manner, thus streamlining the software development
process Dijkstra (1972).

Cloud-based tools that leverage LLMs such as Codeium Codeium (2023), GitHub
Copilot Friedman (2021), OpenAI ChatGPT OpenAI (2023a), and Amazon
CodeWhisperer Amazon (2022), enable users to access their cloud-based LLM
services and online resources through dedicated application programming
interface (API) in an on-demand access. The pricing models for these tools
vary depending on the complexity of the tool and the target audience. Some
pricing models include enterprise pricing, subscription-based pricing, usage-
based pricing, freemium (i.e., the tool is available for free, but additional
premium features require payment), pay-per-use pricing or entirely free of
charge. In fact, these pricing models and LLM-based services can be
incorporated into existing systems like a local integrated development
environment (IDE) or implemented via a Software-as-a-Service (SaaS) web
interface, acting as a virtual service entity to meeting objectives and saving
costs for the human programmer Zheng et al. (2015, 2016). It is expected that
the expanding reach and high demand usage of these LLM-based tools reflect the
growing need for advanced NLP capabilities in software development. This trend
aligns with Dijkstra’s visionary ideas as discussed in Dijkstra (1972,
transcribed 2007).

This paper presents Copilot for Xcode, an AI-assisted programming tool that
was open-sourced on December 7, 2022, one week after OpenAI launched its
ChatGPT on November 30, 2022.111Apple’s recently-issued patent in Siracusa et
al. (2023) dated June 27, 2023 suggests that they are actively exploring the
integration of machine learning models into their software development system,
specifically within Xcode, rather than relying solely on existing solution.
Acting as an intermediary entity, as shown in Figure 1, it seamlessly
integrates cloud-based large language model services with local IDEs like
Xcode. This integration benefits software developers in the Apple ecosystem by
streamlining AI-assisted programming service delivery and enhancing the
accessibility of a myriad of cloud-based LLM applications. Copilot for Xcode
enables real-time prompt engineering and efficient interaction between the
human programmer and the large language models, offering the potential to
integrate serverless computing capabilities with natural language processing
in the cloud. The source code of Copilot for Xcode can be publicly accessed at
https://github.com/intitni/CopilotForXcode.

Figure 2: The sequence diagram illustrates the functionality of Copilot for
Xcode, enabling real-time suggestions through integration with GitHub Copilot.
When a user initiates a code update, Copilot for Xcode receives a notification
and subsequently sends a request to the GitHub Copilot API. Upon receiving the
suggestions from GitHub Copilot, the user has the option to adopt the
recommendations and directly apply the changes within Xcode.

##  2 Related Works

###  2.1 Language Models for Big Code Analysis

LLMs have surfaced as a promising approach to tackle challenges in computer
programming, leveraging the software naturalness hypothesis Hindle et al.
(2012). This hypothesis posits that programming languages can be understood
and manipulated in a similar fashion to how natural language processing
techniques handle human languages. Since the introduction of the transformer
architecture in 2017 Vaswani et al. (2017), LLMs trained on large-scale
datasets of programs have shown significant benefits in code-related tasks by
effectively learning programming language patterns and structures, which are
collectively part of Big Code analysis Vechev et al. (2016). Recent LLMs such
as T5 Raffel et al. (2020), BERT Devlin et al. (2018), GPT-4 OpenAI (2023b)
and Palm 2 Anil et al. (2023) have demonstrated impressive capabilities in
understanding and generating human-like text, opening up new possibilities for
enhancing software engineers’ development experiences. These models undergo a
two-step process involving pre-training and fine-tuning. Following these
steps, prompt engineering can be applied to further optimize the model’s
performance. As an integral part of AI-assisted programming, AI-based
predictive analysis Ji et al. (2020) can anticipate potential issues in a
software development life cycle. For example, it can proactively identify and
flag critical incidents Surameery and Shakor (2023) before they manifest
Talamadupula (2021).

###  2.2 AI-assisted Programming

AI-assisted programming is the incorporation of machine learning techniques
and tools into the software development process Mozannar et al. (2022) to
improve computer programming tasks. This concept shares similarities with pair
programming Bird et al. (2022); Imai (2022), whereby two human programmers
collaborate to develop software by alternating between writing code (driver)
and reviewing (observer) in a continuous switch. AI-assisted programming
essentially replaces one of the two human programmers with an AI assistant,
akin to the aforementioned “MIT programmer’s apprentice” Waters (1982); Rich
and Waters (1988). The AI assistant automates tasks that can be broadly
classified into two categories: generation and understanding. Generation tasks
encompass activities such as code generation Waldinger and Lee (1969); Manna
and Waldinger (1971), code completion Robbes and Lanza (2008); Bruch et al.
(2009), code translation Acharya et al. (2007); Allamanis et al. (2014), code
refinement Saha et al. (2017), and code summarization Sridhara et al. (2010,
2011). On the other hand, understanding tasks encompass activities like defect
detection Charniak (1996) and clone detection Kontogiannis et al. (1996).
Improving the quality of large language models for these tasks focus on
enhancing pre-training schemes Li et al. (2022b), expanding training corpora
Husain et al. (2019), and employing improved evaluation metrics Chen et al.
(2021). The GitHub Copilot Friedman (2021) is an example of an AI-powered
programming tool that utilizes OpenAI Codex, which is based on GPT-3 LLM that
has been trained on a vast amount of source code from the GitHub repository,
totaling over 159GB OpenAI (2023a). For further details on the latest
advancements in AI-assisted programming, please see Wong et al. (2023).

Figure 3: The user interface of Copilot for Xcode demonstrates the code
suggestion capability, specifically showcasing the integration of GitHub
Copilot for real-time suggestions related to the merge sort algorithm. In the
accompanying Figure, the right-hand side displays the open source code editor
within Xcode, focused on an interactive Swift playground. On the left-hand
side, the cloud-based services deliver code suggestion responses.

##  3 Copilot for Xcode

###  3.1 Xcode and its Offline Functionalities

Xcode Apple (2003) is an IDE created by Apple for developing software
applications for the Apple ecosystem such as macOS and iOS. It provides a
comprehensive set of tools, including editors, compilers, debugging tools, and
interface builders, to help software developers create and maintain their
applications. Xcode includes a source code editor with features like syntax
highlighting, code completion, and refactoring capabilities. It supports
multiple programming languages, including Swift, Objective-C, C, and C++,
allowing developers to write code for a variety of Apple platforms. In
addition to the code editor, Xcode offers a wide range of tools to assist in
app development, such as an Interface Builder for designing user interfaces
visually, a graphical debugger for finding and fixing issues in code, and
various performance analysis instruments. It also integrates with other
developer tools, such as the iOS Simulator, which allows software developers
to test their apps on virtual devices, and Instruments, a powerful profiling
tool for measuring and optimizing app performance. Despite the extensive
functionalities of Xcode, it has some limitations. For example, certain
features depend on an offline rule-based system and may necessitate batch
updates from Apple. Consequently, these services may not remain up-to-date
consistently.

###  3.2 The Copilot for Xcode Framework

The main limitation in Xcode is the sandboxing mechanism, which restricts
plugin access to specific resources and prevents the launching of other
programs. We address ways to overcome this limitation in order to enable the
functionality of GitHub Copilot in Xcode. In particular, GitHub Copilot
requires an additional program provided by GitHub to be executed alongside the
plugin. Let us take the real-time suggestion feature as an example: the
application first needs to bypass the sandbox in order to run the GitHub
Copilot language server. This is accomplished by establishing communication
between the Xcode source editor extension and a non-sandboxed XPC Service,
which acts as a cross-process call service that facilitates the communication
between the extension and the GitHub Copilot server. The server then presents
suggestions in a user interface (UI) that is not managed by Xcode. To assemble
a request for the language server, the application must gather sufficient
information, but Xcode only provides the source code and file type. To obtain
additional information without relying on Xcode, the application leverages the
Accessibility API from the software development kit. This particular API
exposes information about each object within the application. Furthermore, to
enable in-place code editing, the application executes extension commands
programmatically. This is accomplished by utilizing the Accessibility API to
interact with the menu bar items. These implementations thus allow Apple
software developers to leverage GitHub Copilot Friedman (2021) and Codeium
Codeium (2023) for code suggestions, while utilizing ChatGPT OpenAI (2023a)
for code explanations, generation and natural language-based code
modifications. The technical interaction of integrating Copilot with Xcode are
depicted in Figure 2.

In addition, it facilitates the integration of an external chat panel that can
access and read the user’s code. This chat panel serves as a connection point
to leverage LLMs for functionalities such as code explanation and mutation
using natural language. The chat panel can also be extended with plugins to
offer additional features, including support for answering questions with
real-time information from search engines. Some latest cloud-based LLM provide
direct access through their official APIs for direct integration. In
particular, Copilot for Xcode leverages the LangChain Chase (2022) framework,
which facilitates the creation of customized LLMs tailored to specific use
cases. This framework significantly enhances the prompt engineering process Wu
et al. (2022); Poldrack et al. (2023), allowing for the design of more
effective prompts that can be utilized with the LLMs. This integration and
framework combination optimize the functionality and usability of the LLMs,
providing users with enhanced capabilities and improved prompt customization.

Figure 4: The user interface of Copilot for Xcode shows its Chat and prompt-
to-code features, which enable code generation for the merge sort algorithm.
These features are connected to ChatGPT, allowing for online prompt
engineering and code generation within Xcode. The Figure illustrates the
source code editor in Xcode on the right-hand side, while the chat
conversational panel is displayed on the left-hand side.

###  3.3 Code Suggestion

The code suggestion function offers a viable option for code completion and
generation under diverse usage scenarios. Code completion, commonly referred
to as auto-completion Wong et al. (2023), is an invaluable feature in software
development that assists in completing unfinished code segments. On the other
hand, code generation involves the automatic generation of source code from
natural language input Li et al. (2022a), guided by user-defined constraints.
This capability strengthens the efficiency on the development process by
automating the creation of code based on linguistic specifications provided by
the user.

In Copilot for Xcode, we offer real-time code suggestions that dynamically
update whenever users modify their code. This capability, depicted in Figure
3, is powered by the integration of GitHub Copilot and Codeium, ensuring that
the suggestions are specifically tailored to the files currently open in the
workspace, thus enhancing productivity and accuracy while leveraging the
capabilities of the code suggestion function. The feature offers two
presentation modes for displaying suggestions, which includes two distinct
modes. In the “Nearby Text Cursor” mode, suggestions are presented based on
the current position of the text cursor. On the other hand, the “Floating
Widget” mode displays suggestions in close proximity to the circular widget.
When the user updates their code, the integrated application retrieves and
integrates relevant suggestions for display within Xcode.

The software development experience is further enhanced by a range of pre-
defined commands offered by Copilot for Xcode. The first useful command is Get
Suggestions, which retrieves customized suggestions based on the current
cursor position in the edited file on Xcode. In cases where multiple
suggestions are available, users can conveniently navigate through them using
the Next Suggestion and Previous Suggestion commands to choose the code
suggestions based on their preferences. When incorporating suggested code, the
Accept Suggestion command comes in handy to immediately select the code
suggestion, while the Reject Suggestion command allows users to remove
unnecessary suggestions along with their associated comments. Furthermore,
there are two commands specifically designed for the usage of Copilot for
Xcode. The Real-time Suggestions command, which can only be called by the
Copilot for Xcode automatically, provides real-time suggestions after a
successful retrieval so that the code suggestion can be presented in Xcode,
while the Prefetch Suggestions, which can also be called by the Copilot for
Xcode, command proactively fetches real-time suggestions in the background,
thus improving the overall responsiveness.

###  3.4 Chat and Prompt-to-Code for Code Generation

Copilot for Xcode provides its chat and prompt-to-Code features for code
generation, as depicted in Figure 4. This functionality focuses on generating
code from text inputs, enabling text-to-code generation within the IDE. By
incorporating these advanced code generation capabilities,Copilot for Xcode
enhances coding workflows, making them more efficient and intuitive. The chat
function of our application, also powered by ChatGPT, complements these code
generation features and offers additional enhancements for a interactive
coding experience. Users can leverage specific features customized to their
programming needs, such as extracting selected code in the active editor for
reference and discussion of specific code snippets. Access to the relative
path of the file being worked on facilitates easy navigation within the
codebase. The chat or prompt-to-code functions also assist in capturing error
and warning labels in the active editor, enabling swift issue resolution.
Users can also obtain information about the text cursor location, facilitating
precise discussions and context-aware conversations. These combined features
empower users to engage in productive coding discussions and streamline their
coding process, harnessing the capabilities of our AI-powered application.

The prompt-to-Code function offers a range of capabilities for code
modification and creation using natural language. It is particularly
beneficial when there is a need to update a specific section of code. This
feature provides various use cases, such as enhancing code readability,
rectifying code bugs, including documentation within the code, dividing
extensive functions into smaller, manageable ones, generating code based on
specific templates using custom commands, refining grammar and spelling errors
in documentation, and facilitating the translation of localizable strings
files. With ”Prompt to Code,” users can refactor existing code or write new
code by harnessing the power of natural language.

##  4 Evaluation

We describe three case studies that illustrate the power of Copilot for Xcode
in tackling real-world programming challenges through AI-assisted programming.
The case studies presented here are based on real-world programming
assignments given to undergraduate students. Furthermore, the case studies
also highlight the significance of prompt engineering for code suggestion
query and making “small” decisions for program composition and design in
Copilot for Xcode.

###  4.1 Case Study: HCF of Two Numbers

The first case study considers computing the Highest Common Factor (HCF) or
the Greatest Common Divisor (GCD) of two natural numbers Dijkstra (2007). The
HCF/GCD represents the largest positive integer that divides both numbers,
leaving no remainder. Many existing approaches can be used to solve this
problem, including subtraction, brute-force, and binary algorithm. One of the
oldest algorithms to compute the HCF for two given natural numbers is the
Euclidean algorithm. We tasked students to implement the Euclidean algorithm
using the Swift programming language based on the observation that if $r$ is
the remainder when $a$ is divided by $b$, then the HCF of $a$ and $b$ is
equivalent to the HCF of $b$ and $r$. Figure 5 depicts the brute-force
approach for the HCF of two natural numbers, while Figure 6 provides a correct
depiction of the HCF calculation using the Euclidean algorithm.

Prompt: | HCF of Two Numbers
---|---
Result: |
|
Table 1: Figure 5: Code suggestion of HCF of two natural numbers using a brute-force algorithm without specific instructions in the prompt. Prompt: | HCF of Two Numbers by Euclidean
---|---
| Algorithm
Result: |
|
Table 2: Figure 6: Code suggestion of HCF of two natural numbers with specific
instructions on using the Euclidean algorithm.

###  4.2 Case Study: LCM of Two Numbers

The Least Common Multiple (LCM) of two natural numbers $a,b$ refers to the
smallest positive integer that is divisible by both numbers. Notably, this LCM
is dual to the HCF/GCD Dijkstra (2007).222The product of two natural numbers
is equal to the product of their respective least common multiple and greatest
common denominator. This principle can be demonstrated using the Fundamental
Theorem of Arithmetic in number theory or through an algorithmic method
outlined in Dijkstra (2007). From our observations, the ChatGPT Codex (GPT-3)
was able to understand this concept (even able to demonstrate a plausible
proof with prompt engineering), although it faced difficulties in extending
the duality to encompass more than two natural numbers. Typically, the LCM
algorithm makes use of the HCF algorithm. However, in this assignment, a
unique requirement is to develop an LCM algorithm that does not rely on the
HCF algorithm. By default, the code suggestion in Copilot for Xcode assumes
that developers have already implemented the HCF function without any
additional instructions in the prompt. The tool thus generates a result that
utilizes the HCF implementation as a helper function for the LCM as shown in
Figure 7. However, to comply with the specific prompt of not relying on the
HCF, Figure 8 presents a correct answer for the LCM calculation without
requiring to compute the HCF.

Prompt: | LCM of Two Numbers
---|---
Result: |
|
Table 3: Figure 7: Code suggestion of LCM of two natural numbers using the HCF without specific instructions. Prompt: | LCM of Two Numbers without
---|---
| Using the HCF
Result: |
|
Table 4: Figure 8: Code suggestion of LCM of two natural numbers without using
HCF algorithm as a helper function.

###  4.3 Case Study: Navigating App on iOS

In the example below, we delve into a code generation scenario that
effectively highlights the fundamental concepts of SwiftUI in iOS app
development, specifically focusing on view navigation. This scenario is
visually represented in Figure 9. The iOS app, built with SwiftUI, comprises
two distinct views: HomeView and DetailView, as depicted in Figure 10. View
refer to a crucial component that constructs the UI and plays a pivotal role
in displaying and handling the visual elements that users observe and engage
with on the screen. To effectively manage views, software developers are
required to arrange them in a hierarchical structure and personalize each view
by configuring different properties.

Prompt: | Create a navigating views app
---|---
| with SwiftUI
Result: |
|
Table 5: Figure 9: This SwiftUI-based app consists of two screens: a home screen and a detail screen. The ContentView sets up a navigation with the entry of the app and provides a navigation bar title. Prompt: | Create the HomeView and
---|---
| DetailsView with SwiftUI
Result: |
|
Table 6: Figure 10: The home and detail screen for a navigating app. When a
button on the home screen is tapped, the app navigates to the detail screen.
Additionally, a back button on the detail screen allows the user to navigate
back to the home screen.

##  5 Conclusion

This paper introduced Copilot for Xcode that integrates cloud-based large
language model services (Github Copilot and OpenAI’s GPT) with Apple’s
integrated development environment, Xcode, for AI-assisted programming. We
also discussed the efficacy of prompt engineering and possible strategies for
AI-assisted programming using simple case studies to illustrate the practical
application of this tool to program composition and design. When designing a
program, making small decisions often involves breaking down complex tasks
into smaller components manageable by the large language model. By carefully
constructing prompts, programmers can influence the generation of code and
steer the langage model’s understanding towards the desired outcome.

As a software prototype, Copilot for Xcode has some limitations to consider
during practical usage. For example, to bypass the sandboxing restrictions, it
employs unconventional methods to retrieve information like file and
project/workspace paths. As such, it is important to be aware that this might
not always function seamlessly in future versions of Xcode. Also, the current
code suggestions are presented as C-style comments in comment mode, which can
inadvertently disrupt a user’s code if they are working on a format, e.g.,
JSON file, where such comments are not applicable.

By combining the capabilities of large language models and integrated tools
for prompt engineering, Copilot for Xcode enhances and streamlines the
software development process within Apple’s Xcode. The integration of Copilot
for Xcode with other cloud-based services like Xcode Cloud can also improve
the overall productivity and efficiency in software development, which is
especially important to continuous integration (CI) and continuous delivery
(CD) in the software development pipeline. As AI-assisted programming tools
like Copilot get incorporated into more IDEs, it brings us closer to the
realization of Dijkstra’s vision in Dijkstra (transcribed 2007), fostering a
symbiotic relationship between human programmers and AI-powered tools to
achieve more efficient and reliable software development.

## References

  * Acharya et al. [2007] Mithun Acharya, Tao Xie, Jian Pei, and Jun Xu.  Mining api patterns as partial orders from source code: From usage scenarios to specifications.  In 6th Joint Meeting of The European Software Engineering Conference and The ACM SIGSOFT Symposium on The Foundations of Software Engineering, pages 25–34, 2007.
  * Allamanis et al. [2014] Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton.  Learning natural coding conventions.  In 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, page 281–293. Association for Computing Machinery, 2014\.
  * Amazon [2022] CodeWhisperer Amazon.  AI code generator - amazon codewhisperer.  https://aws.amazon.com/codewhisperer, 2022.  Accessed on June 1, 2023.
  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  arXiv preprint arXiv:2305.10403, 2023.
  * Apple [2003] Xcode Apple.  Xcode 15 - apple developer.  https://developer.apple.com/xcode/, 2003.  Accessed on June 1, 2023.
  * Bird et al. [2022] Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit.  Taking flight with copilot: Early insights and opportunities of AI-powered pair-programming tools.  Queue, 20(6):35–57, 2022.
  * Bruch et al. [2009] Marcel Bruch, Martin Monperrus, and Mira Mezini.  Learning from examples to improve code completion systems.  In 7th Joint Meeting of The European Software Engineering Conference and The ACM SIGSOFT Symposium on The Foundations of Software Engineering, pages 213–222, 2009.
  * Charniak [1996] Eugene Charniak.  Statistical Language Learning.  MIT press, 1996.
  * Chase [2022] Harrison Chase.  Langchain, 2022.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  Advances in neural information processing systems, 2017.
  * Codeium [2023] Exafunction Codeium.  Codeium - free AI code completion & chat.  https://codeium.com/, 2023.  Accessed on June 1, 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Dijkstra [1972] Edsger W Dijkstra.  The humble programmer.  Communications of the ACM, 15(10):859–866, 1972.
  * Dijkstra [2007] Edsger Wybe Dijkstra.  Defining the greatest common divisor.  E. W. Dijkstra Archive (EWD 1257), 2007.
  * Dijkstra [transcribed 2007] Edsger Wybe Dijkstra.  A preliminary investigation into computer assisted programming.  E. W. Dijkstra Archive (EWD 237), (transcribed) 2007.
  * Friedman [2021] Nat Friedman.  Introducing github copilot: your AI pair programmer, 2021.
  * Handsaker [1982] Robert E. Handsaker.  Code generation in the programmer’s apprentice.  Working Paper 233, MIT AI Lab, May 1982.
  * Hindle et al. [2012] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu.  On the naturalness of software.  In 2012 34th International Conference on Software Engineering, pages 837–847. IEEE, 2012.
  * Husain et al. [2019] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt.  CodeSearchNet challenge: Evaluating the state of semantic code search.  arXiv preprint arXiv:1909.09436, 2019.
  * Imai [2022] Saki Imai.  Is github copilot a substitute for human pair-programming? an empirical study.  In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 319–321, 2022.
  * Ji et al. [2020] Yangfeng Ji, Antoine Bosselut, Thomas Wolf, and Asli Celikyilmaz.  The amazing world of neural language generation.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 37–42, 2020.
  * Kontogiannis et al. [1996] Kostas A Kontogiannis, Renator DeMori, Ettore Merlo, Michael Galler, and Morris Bernstein.  Pattern matching for clone and concept detection.  Automated Software Engineering, 3(1-2):77–108, 1996.
  * Li et al. [2022a] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.  Competition-level code generation with alphacode.  Science, 378(6624):1092–1097, 2022.
  * Li et al. [2022b] Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al.  Automating code review activities by large-scale pre-training.  In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022.
  * Manna and Waldinger [1971] Zohar Manna and Richard J Waldinger.  Toward automatic program synthesis.  Communications of the ACM, 14(3):151–165, 1971.
  * Mozannar et al. [2022] Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz.  Reading between the lines: Modeling user behavior and costs in AI-assisted programming.  arXiv preprint arXiv:2210.14306, 2022.
  * OpenAI [2023a] OpenAI.  Chatgpt: Optimizing language models for dialogue, Jan 2023.
  * OpenAI [2023b] OpenAI.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * Poldrack et al. [2023] Russell A Poldrack, Thomas Lu, and Gašper Beguš.  AI-assisted coding: Experiments with gpt-4.  arXiv preprint arXiv:2304.13187, 2023.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 2020.
  * Rajamani [2022] Sriram Rajamani.  AI assisted programming.  In 15th Annual ACM India Compute Conference, COMPUTE ’22, page 5, New York, NY, USA, 2022. Association for Computing Machinery.
  * Rich and Waters [1982] Charles Rich and Richard C. Waters.  The disciplined use of simplifying assumptions.  ACM SIGSOFT Software Engineering Notes, 7(5):150–154, December 1982\.
  * Rich and Waters [1988] Charles Rich and Richard C. Waters.  The programmer’s apprentice: a research overview.  Computer, 21(11):10–25, November 1988.
  * Rich et al. [1978] Charles Rich, Howard E. Shrobe, Robert C. Waters, Gerald J. Sussman, and Carl E. Hewitt.  Programming viewed as an engineering activity.  AI Memo 459, Massachusetts Institute of Technology, January 1978.
  * Robbes and Lanza [2008] Romain Robbes and Michele Lanza.  How program history can improve code completion.  In 23rd IEEE/ACM International Conference on Automated Software Engineering, pages 317–326, 2008.
  * Saha et al. [2017] Ripon K Saha, Yingjun Lyu, Hiroaki Yoshida, and Mukul R Prasad.  Elixir: Effective object-oriented program repair.  In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering, pages 648–659. IEEE, 2017.
  * Siracusa et al. [2023] M. R. Siracusa, A. K. Katti, and et al.  Integrating learning models into software development systems, June 27th 2023.  US Patent and Trademark Office, US Patent 11,687,830.
  * Sridhara et al. [2010] Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K Vijay-Shanker.  Towards automatically generating summary comments for java methods.  In IEEE/ACM International Conference on Automated Software Engineering, pages 43–52, 2010.
  * Sridhara et al. [2011] Giriprasad Sridhara, Lori Pollock, and K Vijay-Shanker.  Generating parameter comments and integrating with method summaries.  In 2011 IEEE 19th International Conference on Program Comprehension, pages 71–80. IEEE, 2011.
  * Surameery and Shakor [2023] Nigar M Shafiq Surameery and Mohammed Y Shakor.  Use chatgpt to solve programming bugs.  International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290, 3(01):17–22, 2023.
  * Talamadupula [2021] Kartik Talamadupula.  Applied AI matters: Ai4code: Applying artificial intelligence to source code.  AI Matters, 7(1):18–20, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 2017.
  * Vechev et al. [2016] Martin Vechev, Eran Yahav, et al.  Programming with “big code”.  Foundations and Trends® in Programming Languages, 3(4):231–284, 2016.
  * Waldinger and Lee [1969] Richard J Waldinger and Richard CT Lee.  Prow: A step toward automatic program writing.  In 1st International Joint Conference on Artificial Intelligence, pages 241–252, 1969.
  * Waters [1982] Richard C. Waters.  The programmer’s apprentice: Knowledge based program editing.  IEEE Transactions on Software Engineering, SE-8(1):1–12, January 1982.
  * Wong et al. [2023] Man-Fai Wong, Shangxin Guo, Ching-Nam Hang, Siu-Wai Ho, and Chee-Wei Tan.  Natural language generation and understanding of big code for AI-assisted programming: A review.  Entropy, 25(6):888, 2023.
  * Wu et al. [2022] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai.  Promptchainer: Chaining large language model prompts through visual programming.  In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1–10, 2022.
  * Zheng et al. [2015] Liang Zheng, Carlee Joe-Wong, Chee Wei Tan, Mung Chiang, and Xinyu Wang.  How to bid the cloud.  In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (SIGCOMM), pages 71–84, 2015.
  * Zheng et al. [2016] Liang Zheng, Carlee Joe-Wong, Christopher G Brinton, Chee Wei Tan, Sangtae Ha, and Mung Chiang.  On the viability of a cloud virtual service provider.  In Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Science, pages 235–248, 2016\.