# ChatGPT is a Potential Zero-Shot Dependency Parser

Boda Lin1‡, Xinyi Zhou2‡, Binghao Tang1, Xiaocheng Gong1, Si Li1∗
1School of Artificial Intelligence, Beijing University of Posts and
Telecommunications
2Department of Chinese Language and Literature, East China Normal University
{linboda<EMAIL_ADDRESS>

###### Abstract

Pre-trained language models have been widely used in dependency parsing task
and have achieved significant improvements in parser performance. However, it
remains an understudied question whether pre-trained language models can
spontaneously exhibit the ability of dependency parsing without introducing
additional parser structure in the zero-shot scenario. In this paper, we
propose to explore the dependency parsing ability of large language models
such as ChatGPT and conduct linguistic analysis. The experimental results
demonstrate that ChatGPT is a potential zero-shot dependency parser, and the
linguistic analysis also shows some unique preferences in parsing outputs.
††footnotetext: ‡Boda Lin and Xinyi Zhou make equal contribution
††footnotetext: ∗Corresponding author

##  1 Introduction

Dependency parsing is a fundamental task in Natural Language Processing and
have many applications in downstream tasks, such as machine translation
Bugliarello and Okazaki (2020), question answering Teney et al. (2017), and
information retrieval Chandurkar and Bansal (2017). Previous research mainly
focus on how to design the parser structure and parsing algorithms to achieve
better performance in different scenarios Dozat and Manning (2017); Ma et al.
(2018); Li et al. (2019).

The linguistic base of dependency parsing is the dependency grammar Jarvinen
and Tapanainen (1998), which come from the linguists’ research about
linguistic rules and phenomena. Notably, Pre-trained Language Models (PLMs)
can also be viewed as "linguists" that automatically learn rules from a vast
amount of natural language texts. Therefore, investigating whether these PLMs
spontaneously learn certain syntactic rules during the pre-training stage is a
valuable research topic.

After the proposal of the BERT model Devlin et al. (2019), numerous probing
studies explore the specific functions learned by each layer of BERT, which
also touches upon the research of self-acquisition of syntax for PLMs Rogers
et al. (2020). However, the BERT model presents the following limitations: 1)
These studies often induce dependency parsing results from the attention
mechanism of BERT. These parsing results do not stem from natural generative
steps. In particular, parameterized probing methods could introduce external
information interference Wu et al. (2020); 2) Due to model limitations, these
works are often confined to relatively simple datasets and struggles to
directly yield syntactic results with dependency relation labels. Despite the
subsequent BART Lewis et al. (2020), T5 Raffel et al. (2020), and other
encoder-decoder structured PLMs showing good performance in language
generation, the complexity of the expression form inherent in the task of
dependency parsing still makes it challenging to induce syntactic results from
such PLMs in a more straightforward manner.

Recently, Large Language Models (LLMs) such as InstructGPT Ouyang et al.
(2022) and ChatGPT 111https://openai.com/blog/chatgpt, which possess superior
generative capabilities, have been introduced in NLP. These models have
achieved impressive performance on various NLP tasks, including question
answering, reading comprehension, and summarization Ouyang et al. (2022), even
in a zero-shot fashion, providing a crucial key to investigating the innate
syntactic abilities of language models. We are interested in the following
questions: 1) Do LLMs like ChatGPT possess zero-shot dependency parsing
capabilities? 2) If so, do the outputs of ChatGPT still maintain a similar
structure for similar sentences even between different languages? 3) Further
more, do these parsing results contain some preferences that can be
summarized?

Figure 1: The total framework of our work.

In this paper, we explore using ChatGPT and other LLMs to achieve zero-shot
dependency parsing, and conduct linguistic analysis on these parsing results
to answer these questions.

The results demonstrate the ChatGPT is a potential zero-shot dependency parser
and the outputs of ChatGPT maintain similar structure in different languages.
And we summarize some parsing preferences of ChatGPT through linguistic
analysis. The most surprising finding is that in some cases, ChatGPT outputs
are more in line with linguistic rules than gold annotations.

##  2 Related Work

###  2.1 Dependency Parsing

PLMs are widely used in previous dependency parsing research. Biaffine Dozat
and Manning (2017) with BERT Devlin et al. (2019) is a very simple and
effective parser. Yang and Tu (2022) proposes a graph-based parser based on
headed spans. Lin et al. (2022) design the serialization of parsing trees and
enabling the T5 model directly generate parsing sequence. Besides, the newest
state-of-the-art model Hexatagging Amini et al. (2023) cast the parsing tree
into the hexatag sequence and also only use BERT to achieve parsing. But these
methods still rely on the supervised learning paradigm.

###  2.2 Probing

The probe research of the BERT model can be roughly divided into two
categories: parametric methods and non-parametric methods. Jawahar et al.
(2019) use a series of probing tasks to indicate that while BERT does capture
some of this information, it is not always explicitly encoded within BERT’s
representations. Wu et al. (2020) use non-parametric method to achieve probing
task for BERT.

##  3 ChatGPT Parsing

As shown in Figure 1, our approach leverages the prompt to enable ChatGPT for
zero-shot dependency parsing. Specifically, we define the input format as a
sentence adhering to the original word segmentation of the parsing corpus,
while the output is generated in CoNLL format.

The parsing results from ChatGPT may exhibit various format-related issues,
which include word missing, format disruption, word segmentation, word
scrambling, and multiple outputs. We use post-processing to filter the illegal
outputs, the specific statistical details can refer to the Appendix A.

##  4 The Consistency

In order to explore whether the dependency parsing outputs of ChatGPT maintain
the consistency in different languages, we conduct corresponding experiments
and use Dependency Tree Edit Distance (DTED) McCaffery and Nederhof (2016) to
measure the similarity between dependency syntax trees in different languages.

$DTED(T_{a},T_{b})=1-\frac{EditDist(T_{a},T_{b})}{\max(|T_{a}|,|T_{b}|)}$ (1)

Where $T_{a}$ and $T_{b}$ means two parsing trees from different language, and
the value of DTED score is from [0, 1]. The EditDist is calculate based on the
Tree Edit Distance algorithm Zhang and Shasha (1989).

##  5 Parsing Ability Experiment

###  5.1 Setting

In this paper, we investigate the parsing ability of ChatGPT from two
different settings: 1) The normal zero-shot parsing setting. In this setting,
we use gpt-3.5-turbo, HuggingChat222https://huggingface.co/chat,
Vicuna-13B333https://lmsys.org/blog/2023-03-30-vicuna and ChatGLM-6B Zeng et
al. (2022) to directly achieve dependency parsing on English and Chinese. 2)
The cross-lingual parsing setting. In this setting, we collect the sentence
pairs from English and Chinese which have the similar parsing structure. Then
we conduct dependency parsing on these sentence pairs and analyze the same or
difference between English and Chinese. In order to avoid the influence of
generating randomness, we set the temperature of ChatGPT to 0 in all
experiments.

###  5.2 Dataset

For English parsing, we choose the proverbial benchmark Penn Treebank (PTB)
Marcus et al. (1993), Chinese Treebank 5 (CTB5) Xue et al. (2005) and 12
languages from Universal Dependencies Version 2.2 (UD2.2) Nivre et al. (2016)
following the previous work Ma et al. (2018). For PTB and CTB5, we follow Ma
et al. (2018) to use the Stanford basic Dependencies representation de
Marneffe et al. (2006) of PTB and CTB converted by Stanford
parser444http://nlp.stanford.edu/software/lex-parser.html.

For the cross-lingual parsing setting, we use the DTED score to choose the
most similarity top-50 sentences from en-ewt-test and zh-cfl-test of UD 2.2.

###  5.3 Parsing Ablility

Dataset | Method | UAS | LAS
---|---|---|---
PTB | Biaffine | $95.74$ | $94.08$
StackPTR | $95.87$ | $94.19$
DPSG | $96.64$ | $95.82$
Hexatagging | 97.40 | 96.40
ChatGPT | $40.22$ | $28.61$
CTB5 | Biaffine | $89.30$ | $88.23$
StackPTR | $90.59$ | $89.29$
Hexatagging | 93.20 | 91.90
ChatGPT | $28.08$ | $11.93$
en-top | ChatGPT | $32.88$ | $26.68$
zh-top | ChatGPT | $53.19$ | $39.36$
Table 1: The results of ChatGPT and other traditional supervised parsing methods on PTB and CTB5. The en-top and zh-top mean the top 100 sentences extract from the en-ewt-test and zh-cfl-test. | Gold | ChatGPT
---|---|---
Avg_DTED | 0.64 | 0.45
Table 2: The average DTED score between top 50 similarity sentences of English
and Chinese.

Model | bg | ca | cs | de | en | es | fr | it | nl | no | ro | ru | AVG
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Biaffine | $90.30$ | $94.49$ | $92.65$ | $85.98$ | $91.13$ | $93.78$ | $91.77$ | $94.72$ | $91.04$ | $94.21$ | $87.24$ | $94.53$ | $91.82$
StackPTR | $89.96$ | $92.39$ | $90.94$ | $86.16$ | $89.83$ | $91.52$ | $89.88$ | $92.55$ | $91.73$ | $93.62$ | $85.34$ | $93.07$ | $90.58$
DPSG | 93.92 | $93.75$ | 92.97 | $84.84$ | 91.49 | $92.37$ | $90.73$ | $94.59$ | 92.03 | 95.30 | 88.76 | 95.25 | 92.17
Hexatagger | $92.87$ | 93.79 | $92.82$ | 85.18 | $90.85$ | 93.17 | 91.50 | 94.72 | $91.89$ | $93.95$ | $87.54$ | $94.03$ | $91.86$
ChatGPT | $35.87$ | $34.04$ | $33.37$ | $26.88$ | $32.63$ | $35.01$ | $29.56$ | $24.60$ | $29.79$ | $30.86$ | $33.78$ | $34.97$ | $31.78$

Table 3: Results on 12 languages of UD2.2 in terms of LAS.

According the results shown in Table 1 and Table 3, we can answer the first
question in the introduction.

There is no doubt that ChatGPT has the ablility of zero-shot dependency
parsing. In fact, this capability is already quite rare in other LLMs. We also
conducted zero-shot experiments on some other popular LLMs (HuggingChat,
Vicuna-13B, ChatGLM-6B)555Since other LLMs lacking the CoNLL or sequence
parsing output capability, we cannot calculate the performance and found that
maintaining a generally correct CoNLL format output is a very challenging task
for these LLMs. Furthermore, we conducted one-shot demonstration learning
experiments for these LLMs, with only Vicuna able to learn a fairly close
format through example. To rule out the complexity of the CoNLL format itself,
we converted the CoNLL format parsing tree into a sequence format following
Lin et al. (2022), but other LLMs were still unable to produce satisfactory
results.

The results on Table 2 answer the second question in the introduction, the
output similarity of ChatGPT between English and Chinese reaches 0.46, which
is 72% of the gold similarity, which shows that for sentences with similar
structures in different languages, the output results given by ChatGPT still
have a high structural similarity. We show more details in Appendix B.

##  6 Linguistic Analysis

In this section, we answer the last question in the introduction. We analyze
the 50 sentences in the test set of en-ewt-test in UD2.2 for English and the
total test set of CTB5 for Chinese. More details and examples are listed in
the Appendix C and Appendix E.

###  6.1 English Analysis

More precisely, the analysis conducted on English outputs from ChatGPT reveals
distinct error patterns that can be classified into the following:

Predicative Verb-Centrism: This phenomenon pertains to the inclination of
perceiving the verb in the predicate as the central element. This tendency is
commonly observed during the processing of clauses, particularly when the
subject is omitted.

Noun Subject Preference: This tendency reflects a predisposition to designate
a noun as the subject within a phrase.

Preposition-Case Ambiguity: This phenomenon pertains to the inconsistent
categorization of simple prepositions such as "to", "in", "case", and "for",
where they are labeled inconsistently as either prepositions or case.

Auxiliary Verb Mislabeling: This phenomenon arises when a sentence comprises
both auxiliary verb and non-finite verb. In such cases, ChatGPT tends to
designate the auxiliary verb as the root due to the influence of positional
placement.

Adjective Modifier Ambiguity: This phenomenon emerges when a noun is preceded
by multiple modifiers, creating ambiguity in discerning the appropriate
modifier.

###  6.2 Chinese Analysis

We also conduct linguistic analysis for CTB5, and the observations are
summarized into the following, more statistics details are shown in Appendix
D.

Obj-Dobj: The most prevalent disparity observed between the ChatGPT and the
golds is the "obj-dobj". Although both "obj" and "dobj" signify a direct
object relationship, their usage is inconsistent across the two tagging sets,
indicating potential variations in labeling conventions.

Compound-Nmod: The differentiation between the "compound" (indicating a
compound noun, adjective, or adverb) and "nmod" (representing a nominal
modifier of a noun or pronoun) frequently leads to discrepancies between
ChatGPT and the golds.

Quote Root: When a sentence portrays a statement attributed to an individual,
with the speaker positioned at the end of the sentence, ChatGPT demonstrates a
tendency to designate the verb within the quotation as the root, whereas the
golds mark the verb "say" as the root.

Error Root: ChatGPT occasionally misclassifies punctuation marks as the root,
thereby assigning them undue prominence. Additionally, it displays
inconsistencies in labeling the main verb as the root, often assigning this
role to nouns instead. In certain cases, ChatGPT fails to identify the root.

Dobj-Nmod: In certain instances, ChatGPT erroneously labels "dobj" as "nmod".
This primarily transpires when there are disagreements regarding the
appropriate word to which the dependent should be linked.

Nummod-Dep: The utilization of numerals within ChatGPT outputs is typically
labeled as "nummod", whereas the gold standard annotations often designate it
as "dep". It is worth noting that in certain cases, the gold annotation may be
inaccurate, incorrectly assuming that the numeral modifies a noun that it
logically does not modify.

###  6.3 Summarization

In general, the inconsistencies between ChatGPT and golds can be categorized
into three groups: 1) Instances where ChatGPT outputs are incorrect; 2) Cases
where ChatGPT and golds can both be considered correct, but differ due to
distinct perspectives considered in the annotations; 3) Situations where
ChatGPT exhibits greater linguistic normativity compared to the golds.

The third category is intriguing as it highlights the potential of ChatGPT.
While traditional parsing methods based on supervised learning achieve
impressive performance, the parsing capabilities of these models are
constrained by the labeled data. However, the parsing abilities of ChatGPT
acquired through pre-training may transcend this limitation and offer
researchers a novel perspective. This breakthrough could potentially provide
researchers with valuable insights or alternative viewpoints.

##  7 Conclusion

We employ the prompt to investigate the zero-shot dependency parsing
capability of ChatGPT and other Language Models (LLMs) on proverbial
benchmarks. The experimental results substantiate that ChatGPT exhibits
promising potential as a zero-shot dependency parser. Furthermore, cross-
lingual experiments demonstrate the ability of ChatGPT to maintain similarity
in parsing outputs across different languages. Additionally, linguistic
analysis is performed to discern the parsing output preferences of ChatGPT.
The analysis reveals that ChatGPT has the ability to surpass limitations
stemming from errors in labeled data.

## Limitations

Considering the powerful learning ability of Large Language Models (LLMs), we
use prompt-based method to analyze the zero-shot ability of Dependency Parsing
of LLMs. The different formats of prompts might significantly affect the final
outputs and could be a disturbance for our experiments. Moreover, the datasets
and corpora usage of LLMs is unclear and that might influence our linguistic
analyses. In addition, linguistic analysis may be mixed with some subjective
judgments, and due to the complex format of the parsing data, many linguistic
analysis phenomena are difficult to directly perform data statistics.

## Ethics Statement

We affirm that our work here does not exacerbate the biases already inherent
in the large language models and our linguistic analyses are also only based
on those model outputs. As a result, we anticipate no ethical concerns
associated with this research.

## References

  * Amini et al. (2023) Afra Amini, Tianyu Liu, and Ryan Cotterell. 2023.  Hexatagging: Projective dependency parsing as tagging.  Association for Computational Linguistics.
  * Bugliarello and Okazaki (2020) Emanuele Bugliarello and Naoaki Okazaki. 2020.  Enhancing machine translation with dependency-aware self-attention.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1618–1627, Online. Association for Computational Linguistics.
  * Chandurkar and Bansal (2017) Avani Chandurkar and Ajay Bansal. 2017.  Information retrieval from a structured knowledgebase.  In _11th IEEE International Conference on Semantic Computing, ICSC 2017, San Diego, CA, USA, January 30 - February 1, 2017_ , pages 407–412. IEEE Computer Society.
  * de Marneffe et al. (2006) Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006.  Generating typed dependency parses from phrase structure parses.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)_ , Genoa, Italy. European Language Resources Association (ELRA).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017.  Deep biaffine attention for neural dependency parsing.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Jarvinen and Tapanainen (1998) Timo Jarvinen and Pasi Tapanainen. 1998.  Towards an implementable dependency grammar.  _arXiv preprint cmp-lg/9809001_.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2019) Zhenghua Li, Xue Peng, Min Zhang, Rui Wang, and Luo Si. 2019.  Semi-supervised domain adaptation for dependency parsing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2386–2395, Florence, Italy. Association for Computational Linguistics.
  * Lin et al. (2022) Boda Lin, Zijun Yao, Jiaxin Shi, Shulin Cao, Binghao Tang, Si Li, Yong Luo, Juanzi Li, and Lei Hou. 2022.  Dependency parsing via sequence generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7339–7353, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ma et al. (2018) Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018.  Stack-pointer networks for dependency parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1403–1414, Melbourne, Australia. Association for Computational Linguistics.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * McCaffery and Nederhof (2016) Martin McCaffery and Mark-Jan Nederhof. 2016.  DTED: Evaluation of machine translation structure using dependency parsing and tree edit distance.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 491–498, Berlin, Germany. Association for Computational Linguistics.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.  Universal Dependencies v1: A multilingual treebank collection.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1659–1666, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in BERTology: What we know about how BERT works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866.
  * Teney et al. (2017) Damien Teney, Lingqiao Liu, and Anton van den Hengel. 2017.  Graph-structured representations for visual question answering.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 3233–3241. IEEE Computer Society.
  * Wu et al. (2020) Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020.  Perturbed masking: Parameter-free probing for analyzing and interpreting BERT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4166–4176, Online. Association for Computational Linguistics.
  * Xue et al. (2005) Nianwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. 2005.  The penn chinese treebank: Phrase structure annotation of a large corpus.  _Natural Language Engineering_ , 11(2):207–238.
  * Yang and Tu (2022) Songlin Yang and Kewei Tu. 2022.  Headed-span-based projective dependency parsing.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2188–2200, Dublin, Ireland. Association for Computational Linguistics.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_.
  * Zhang and Shasha (1989) Kaizhong Zhang and Dennis Shasha. 1989.  Simple fast algorithms for the editing distance between trees and related problems.  _SIAM journal on computing_ , 18(6):1245–1262.

##  Appendix A Post-processing Details

Since the dependency parsing task is a fine-grained task, it has high
requirements on vocabulary and output format, and the uncontrollability of LLM
itself, there will be many formal errors in the outputs of ChatGPT, as
follows:

Word filtering: Since some of the parsing corpora come from political news,
certain vocabulary may trigger the filtering policy of ChatGPT, leading to the
omission of sensitive words in the output CoNLL results.

Format disruption: Occasionally, ChatGPT may not output in the standard CoNLL
format, causing issues such as missing columns, extra columns, or disordered
columns.

Word segmentation disruption: This phenomenon is particularly common in
languages that require word segmentation, such as Chinese. Even though we
clearly pre-segmented the input with spaces, ChatGPT may sometimes employ its
own segmentation.

Word omission: In lengthy sentences, there might be instances where a sequence
of words is missing.

Word scrambling: In extended sentences, the outputted CoNLL results may
contain parts where the vocabulary is scrambled.

Multiple outputs: In some cases, ChatGPT will give duplicate parsing outputs
for a sentence.

Since these formal errors will cause predict and gold to fail to achieve
alignment, we use post-processing to filter out the output containing these
errors. The size of the original test sets and the size of the data obtained
after post-processing are shown in the Table 4.

| PTB | CTB5 | bg | ca | cs | de | en | es | fr | it | nl | no | ro | ru
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Total Number | $2,416$ | $1,915$ | $1,116$ | $1,846$ | $12,203$ | $977$ | $2,077$ | $2,174$ | $416$ | $482$ | $1,396$ | $3,450$ | $729$ | $6,491$
Final Number | $1,394$ | $990$ | $518$ | $1,307$ | $8,040$ | $505$ | $1,164$ | $1,644$ | $283$ | $374$ | $614$ | $2,492$ | $498$ | $5,742$

Table 4: The number of sentences in the original test set of PTB, CTB5 and 12
languages on UD2.2 and the number of sentences retained after post-processing.

ID | Word | Pred-Head | Pred-Rel | Gold-Head | Gold-Rel | Word | Pred-Head | Pred-Rel | Gold-Head | Gold-Rel
---|---|---|---|---|---|---|---|---|---|---
1 | He | 2 | nsubj | 3 | nsubj | 我们 | 2 | nsubj | 3 | nsubj
2 | has | 0 | root | 3 | aux | 要 | 0 | root | 3 | aux
3 | denied | 2 | xcomp | 0 | root | 去 | 2 | xcomp | 0 | root
4 | this | 3 | obj | 3 | obj | 目的地 | 3 | obj | 3 | obj
5 | . | 2 | punct | 3 | punct | ！ | 2 | punct | 3 | punct
1 | you | 2 | nsubj | 3 | nsubj | 你 | 2 | nsubj | 3 | nsubj
2 | r | 0 | root | 3 | cop | 是 | 0 | root | 3 | cop
3 | retarded | 2 | xcomp | 0 | root | 学生 | 2 | attr | 0 | root
4 | . | 2 | punct | 3 | punct | ？ | 2 | punct | 3 | punct
1 | Just | 2 | advmod | 3 | advmod | 有点儿 | 2 | amod | 2 | advmod
2 | our | 3 | amod | 3 | nmod:poss | 恼火 | 0 | root | 0 | root
3 | standard | 0 | root | 0 | root | 了 | 2 | mark | 2 | discourse:sp
4 | . | 3 | punct | 3 | punct | 。 | 2 | punct | 2 | punct

Table 5: Three similarity parsing tree pair examples.

ID | Word | Pred-Head | Pred-Rel | Gold-Head | Gold-Rel
---|---|---|---|---|---
Predicative Verb-Centrism |  |  |
1 | What | 4 | nsubj | 0 | root
2 | if | 4 | mark | 4 | mark
3 | Google | 4 | nsubj | 4 | nsubj
4 | Morphed | 0 | root | 1 | advcl
5 | Into | 4 | prep | 6 | case
6 | GoogleOS | 5 | pobj | 4 | obl
7 | ? | 4 | punct | 4 | punct
Noun Subject Preference |  |  |
1 | One | 2 | nummod | 5 | nsubj
2 | of | 4 | case | 4 | case
3 | the | 4 | det | 4 | det
4 | pictures | 5 | nsubj | 1 | nmod
5 | shows | 0 | root | 0 | root
6 | a | 7 | det | 7 | det
7 | flag | 5 | obj | 5 | obj
8 | that | 9 | nsubj:pass | 10 | nsubjpass
9 | was | 5 | acl:pass | 10 | aux:pass
10 | found | 9 | auxpass | 7 | acl:relcl
11 | in | 12 | case | 12 | case
12 | Fallujah | 10 | obl | 10 | obl
13 | . | 5 | punct | 5 | punct
Preposition-Case Ambiguity |  |  |
1 | Compare | 0 | root | 0 | root
2 | the | 3 | det | 3 | det
3 | flags | 1 | obj | 1 | obj
4 | to | 1 | prep | 7 | case
5 | the | 6 | det | 7 | det
6 | Fallujah | 4 | pobj | 7 | compound
7 | one | 1 | dobj | 7 | obl
8 | . | 1 | punct | 1 | punct
Auxiliary Verb Mislabeling |  |  |
1 | It | 2 | nsubj | 3 | expl
2 | does | 0 | root | 3 | aux
3 | seem | 2 | ccomp | 0 | root
4 | that | 3 | mark | 7 | mark
5 | Iranians | 6 | nsubj | 7 | nsubj
6 | frequently | 3 | advmod | 7 | advmod
7 | make | 6 | conj | 3 | ccomp
8 | statements | 7 | dobj | 7 | obj
9 | and | 7 | cc | 11 | cc
10 | then | 11 | advmod | 11 | advmod
11 | hide | 7 | conj | 7 | conj
12 | behind | 11 | prep | 13 | case
13 | lack | 14 | compound | 11 | obl
14 | of | 12 | pobj | 15 | case
15 | proof | 14 | nmod | 13 | nmod
16 | . | 2 | punct | 3 | punct
Adjective Modifier Ambiguity |  |  |
1 | The | 2 | det | 2 | det
2 | clerics | 4 | nsubj | 3 | nsubj
3 | demanded | 4 | aux | 0 | root
4 | talks | 0 | root | 3 | obj
5 | with | 6 | case | 8 | case
6 | local | 7 | amod | 8 | amod
7 | US | 8 | compound | 8 | compound
8 | commanders | 4 | obl | 4 | nmod
9 | . | 4 | punct | 3 | punct

Table 6: The analysis examples in English.

##  Appendix B Examples of Similarity Trees

In addition to the calculated DTED scores as shown in Table 2 in Section 5, we
can also intuitively see from the Table 5 that ChatGPT outputs similar
sentences from different languages with similar structures.

In the Table 5, the left and right columns are the corresponding Chinese-
English syntactic tree pairs with similar structure, among which, the DTED
score of the first two pairs is 1, and the DTED score of the third pair is
0.8. Obviously, although the sentences in the syntactic tree pairs are from
different languages, the output of ChatGPT still has a high degree of
similarity even though the wrong parsing outputsare given.

##  Appendix C Examples of English Linguistic Analysis

Predicative Verb-Centrism: In the sentence "What if Google Morphed Into
GoogleOS?", the correct root token should be "what". However, the ChatGPT
result indicates that the subordinate clause is not recognized, and only the
predicate verb is identified. Conversely, in cases where a clause contains a
main clause with a predicate verb, such as "One of the pictures shows a flag
that was found in Fallujah". ChatGPT does not incorrectly mark the root as
unmarked.

Noun Subject Preference: For example, "One of the pictures shows a flag that
was found in Fallujah. In this sentence, the actual subject within the phrase
"one of the pictures" is "one" and "pictures" functions as the noun being
quantitatively modified. The words "one", "of", and "the" are dependent on
"pictures" as well. However, in reality, the true subject should be "one",
"of", and "the" collectively. The reason is that if the subject were solely
"pictures", the verb "shows" would not be in the third person singular form.
Consequently, the analysis of the sentence exhibits a flaw in terms of marking
the subject accurately.

Preposition-Case Ambiguity: In the sentence "Compare the flags to the Fallujah
one." the standard analysis designates "to" as a case-grammatical marker,
which is dependent on "one." The word "one" serves as the object being
compared, and the presence of "to" indicates that "one" is the direct object.
However, ChatGPT only identifies "to" as a preposition, overlooking its role
as a case-grammatical marker. Additionally, there is ambiguity in the phrase
"the Fallujah one" that follows. As ChatGPT only labels "to" as a preposition,
it may interpret it as "to the Fallujah," suggesting that "the" is the
determiner for the compound noun "Fallujah one." However, "Fallujah" does not
naturally form a compound noun with "one," and it is not possible to establish
a dependency relationship among "the," "Fallujah," and "one."

Auxiliary Verb Mislabeling: In the sentence " I’m staying away from the
stock." the correct root should be "staying," while "am" functions as the
auxiliary verb assisting in tense formation. However, there is a mislabeling
where "am" is incorrectly marked as the root. Similarly, in the sentence "He
has denied this." the root should be "denied," but ChatGPT mistakenly
identifies "has" as the root. Moreover, in the sentence "It does seem that
Iranians frequently make statements and then hide behind the lack of proof."
the root should be "seem", but ChatGPT erroneously identifies "does" as the
root. These instances highlight inconsistencies in root identification by
ChatGPT, where the actual root is mislabeled in favor of auxiliary verbs or
other words in the sentence.

Adjective Modifier Ambiguity: In the sentence "The clerics demanded talks with
local US commanders." ChatGPT tends to analyze the sentence in a way that
suggests a dependency between the first modifier and the second modifier, and
another dependency between the second modifier and the noun "commanders".
Although this dependency may raise semantic concerns, it is syntactically
acceptable. However, according to the gold standard annotations, it is "local"
that is dependent on "commanders" and "US" has a separate dependency with
"commanders". In other words, there is no direct relationship between "local"
and "US" in the gold standard annotations.

##  Appendix D Statistics of Chinese Linguistic Analysis

We count the number of sentences appearing in ChatGPT outputs for several
types of linguistic analysis given in Section 6.2, as shown in the Table 7.

##  Appendix E Examples of Chinese Linguistic Analysis

| snt_number | snt_percentage
---|---|---
obj-dobj | $355$ | $35.86\%$
nmod-compound | $145$ | $14.65\%$
punct root | $106$ | $35.86\%$
nmod-dobj | $56$ | $10.71\%$
nummod-dep | $26$ | $2.63\%$
Table 7: The statistics of linguistic categories of CTB5.

ID | Word | Pred-Head | Pred-Rel | Gold-Head | Gold-Rel
---|---|---|---|---|---
Obj-Dobj |  |  |  |
1 |  维阿里 | 2 | nsubj | 2 | nsubj
2 |  看好 | 0 | root | 0 | root
3 |  欧文 | 2 | obj | 2 | dobj
Quote Root |  |  |  |
1 |  “ | 0 | root | 2 | punct
2 |  光头 | 1 | nsubj | 4 | appos
3 |  ” | 1 | punct | 2 | punct
4 |  维阿里 | 2 | flat | 6 | nsubj
5 |  现在 | 6 | advmod | 6 | nmod:tmod
6 |  担任 | 1 | ccomp | 0 | root
7 |  英格兰 | 8 | flat | 10 | nmod:assmod
8 |  超级 | 9 | compound | 9 | amod
9 |  联赛 | 10 | nmod | 10 | compound:nn
10 |  俱乐部 | 6 | obj | 11 | appos
11 |  切尔西队 | 10 | nmod | 13 | nmod:assmod
12 |  的 | 11 | case | 11 | case
13 |  教练 | 6 | obj | 6 | dobj
14 |  。 | 1 | punct | 6 | punct
Compound-Nmod |  |  |  |
1 |  我 | 4 | nsubj | 5 | nsubj
2 |  一点 | 3 | det | 5 | advmod
3 |  也 | 4 | advmod | 5 | advmod
4 |  不 | 5 | advmod | 5 | neg
5 |  怀疑 | 0 | root | 19 | dep
6 |  欧文 | 5 | obj | 15 | nsubj
7 |  将 | 8 | aux | 15 | advmod
8 |  是 | 5 | ccomp | 15 | cop
9 |  未来 | 10 | compound:nn | 10 | dep
10 |  几 | 11 | nummod | 15 | dep
11 |  年 | 8 | obl | 10 | mark:clf
12 |  内 | 11 | case | 10 | case
13 |  真正 | 14 | amod | 15 | amod
14 |  的 | 11 | nmod | 13 | mark
15 |  巨星 | 8 | obj | 5 | ccomp
16 |  ， | 5 | punct | 19 | punct
17 |  ” | 5 | punct | 19 | punct
18 |  他 | 20 | nsubj | 19 | nsubj
19 |  说 | 20 | ccomp | 0 | root
20 |  。 | 5 | punct | 19 | punct
Error Root |  |  |  |
1 |  （ | 0 | root | 5 | punct
2 |  左 | 1 | punct | 5 | dep
3 |  一 | 1 | punct | 5 | dep
4 |  为 | 1 | punct | 5 | dep
5 |  作者 | 4 | punct | 0 | root
6 |  ） | 1 | punct | 5 | punct
Dobj-Nmod |  |  |  |
1 |  发言人 | 2 | nsubj | 2 | nsubj
2 |  主张 | 0 | root | 0 | root
3 |  该国 | 5 | nmod | 8 | nsubj
4 |  就 | 5 | advmod | 5 | case
5 |  入侵 | 2 | obj | 8 | nmod:prep
6 |  邻国 | 5 | nmod | 5 | dobj
7 |  正式 | 8 | advmod | 8 | advmod
8 |  道歉 | 2 | ccomp | 2 | ccomp
Nummod-Dep |  |  |  |
1 |  十 | 2 | nummod | 3 | dep
2 |  面 | 4 | nsubj | 1 | mark:clf
3 |  埋伏 | 4 | compound:nn | 0 | root
4 |  ， | 2 | punct | 3 | punct
5 |  创造 | 2 | conj | 3 | conj
6 |  声势 | 5 | obj | 5 | dobj
Gold Errors |  |  |  |
1 |  内幕 | 3 | nsubj | 4 | dep
2 |  、 | 1 | punct | 4 | punct
3 |  或 | 0 | root | 4 | cc
4 |  丑闻 | 3 | conj | 0 | root
5 |  ？ | 3 | punct | 4 | punct

Table 8: The analysis examples in Chinese.

Obj-Dobj: The most frequent label that differs between ChatGPT results and
gold is "obj - dobj" (labeled obj in the outputs of ChatGPT and dobj in gold).
In the CoNLL format data description provided, obj is the direct object
relationship, while dobj (direct object) is also the direct object. And both
obj and dobj appear in ChatGPT outputs, whereas there is no obj tag in gold.
This suggests that the labels used in the two sets of annotation results are
different, and that the labels in ChatGPT outputs are confusing. For example,
in the following sentence, ChatGPT outputs and gold label the dependencies
identically, but for the relationship between "欧文" and "看好" , ChatGPT outputs
labels it as obj while gold labels it as dobj.

Compound-Nmod: It is easier to judge the relationship of compound (compound
noun) differently from gold, which is more often labelled as nmod, as in the
example below, where "联赛" is dependent on "俱乐部" and ChatGPT labels the
relationship as nmod, whereas gold is labelled as compound. The crucial aspect
to consider is the presence of a conceptual overlap between "compound," which
denotes a compound construction of a noun, adjective, or adverb, and "nmod,"
which signifies a nominal modifier, i.e., a noun, adjective, or adverb
modifying another noun or pronoun. The distinction between these two
categories is not always clearly defined, and there are instances where
determining whether it should be labeled as "compound" or "nmod" can be
subjective. Therefore, the labeling norms and conventions for these categories
still remain a matter of debate and interpretation.

Quote Root: When a sentence is spoken by someone and the speaker is positioned
at the end of the sentence, resulting in a complete sentence with a predicate
verb, the ChatGPT labeling scheme assigns the predicate verb as the root. On
the other hand, the gold standard annotation assigns the speaker, represented
by the Chinese character "说" , as the root in such cases. Similarly, in
instances where ChatGPT outputs labels the verb "怀疑" as root, while gold
labels "他" as root. These discrepancies in root labeling between ChatGPT
outputs and gold exemplify the differing perspectives and criteria utilized in
these annotation schemes.

Error Root: The annotation of root in Chinese corpora can sometimes exhibit
unexpected errors. One particular error involves the mislabeling of
punctuation as the root by ChatGPT. Furthermore, ChatGPT does not consistently
label predicate verbs as the root in all cases. These inconsistencies
highlight the challenges and potential shortcomings in the annotation process
for determining the root in Chinese sentences within the ChatGPT annotation
scheme.

Dobj-Nmod: There are instances in which the ChatGPT labels the direct object
(dobj) as a nominal modifier (nmod). One such case is exemplified by sentence
"发言人 主张 该国 就 入侵 邻国 正式 道歉" , where ChatGPT considers "邻国" to be dependent on
the noun "入侵" and assigns the relation as nmod. In contrast, the gold standard
annotation marks the relation as dobj. This discrepancy in labeling suggests a
disagreement in assigning the correct dependency relation between the two
annotation schemes.

Nummod-Dep: When it comes to number words in sentences, ChatGPT more
frequently labels them as "nummod", whereas gold annotations often label them
as "dep." This distinction arises primarily due to differing judgments
regarding dependency relationships between the two annotation schemes. The
variation in labeling can be attributed to differences in the interpretation
of the role and dependency of number words within the sentence structure. It
highlights the impact of subjective judgment in determining the appropriate
dependency label for number words.

Gold Errors Regarding the gold annotations, there is also a significant issue
with errors in their labeling. In such cases, it becomes challenging to
compare and analyze the preferences between gold and ChatGPT outputs. For
example, in the given example, the words"内幕"and "丑闻"clearly have a coordinate
relationship, and either of them can be considered as the root. The word
"或"should be identified as a coordinating conjunction, indicating the
coordination between"内幕"and "丑闻". If we assume "或" as the root, and both
"内幕"and "丑闻"as having an unknown relationship with "或", it would still make
sense. However, in gold annotations, by labeling "或" as a coordinating
conjunction, and simultaneously marking the relationship between "内幕" and "丑闻"
as unknown, it seems somewhat unreasonable. In summary, the issue lies in the
inconsistencies and potential errors within the gold annotations, making it
challenging to establish a reliable basis for comparison and analysis against
ChatGPT.