# Fashion Style Editing with Generative Human Prior

Chaerin Kong∗, Seungyong Lee∗, Soohyeok Im∗, Wonsuk Yang∗
NXN Labs
<EMAIL_ADDRESS>
###### Abstract

Image editing has been a long-standing challenge in the research community
with its far-reaching impact on numerous applications. Recently, text-driven
methods started to deliver promising results in domains like human faces, but
their applications to more complex domains have been relatively limited. In
this work, we explore the task of fashion style editing, where we aim to
manipulate the fashion style of human imagery using text descriptions. †† ∗
Equal contributions.  Specifically, we leverage a generative human prior and
achieve fashion style editing by navigating its learned latent space. We first
verify that the existing text-driven editing methods fall short for our
problem due to their overly simplified guidance signal, and propose two
directions to reinforce the guidance: textual augmentation and visual
referencing. Combined with our empirical findings on the latent space
structure, our Fashion Style Editing framework (FaSE) successfully projects
abstract fashion concepts onto human images and introduces exciting new
applications to the field.

Figure 1: Example of text-driven fashion style edition using our framework
(FaSE). Given an input image, a text prompt is used to drive change in garment
style while preserving its overall attributes.

##  1 Introduction

Recent years have witnessed unprecedented progress in generative models [13,
8, 44, 25, 14], with the language guidance (i.e., text-to-image) being the
cornerstone for their enhanced controllability and generalizability [39, 43,
41, 3, 46, 32, 24]. Image editing task [27, 45, 26] also started to draw great
attention from the public thanks to the intuitive editing interface text-to-
image frameworks provide [2, 37, 12]. One of the most popular ways to
incorporate text condition to the image editing pipeline is to use a
pretrained vision-language foundation model [38, 33, 52, 1, 16, 15] as in
StyleCLIP [37], CLIPstyler [27] or diffusionCLIP [23], where a joint embedding
model like CLIP [38] guides the editing direction in the form of similarity
score. Although this approach has demonstrated impressive editing capability
in some visual domains like human faces [19], they are vulnerable to the
blindspots of CLIP features. Specifically, CLIP exposes weakness in domain-
specific applications that requires special knowledge [49, 6], and its
tendency to focus on the global semantic often renders it unsuitable for
guiding local editing operation, especially when the image contains complex
semantics [24].

Unfortunately, these vulnerabilities are greatly highlighted in our problem
setting: fashion style editing of human imagery. In an attempt to introduce an
intuitive interface for human fashion restyling, we explore the creative task
of text-driven fashion style editing, where we steer the style of full-body
human portrait with language guidance such as “street fashion” or “floral
pattern” (Fig. 1). With our unique problem scope arises a new set of
challenges, namely the modeling complexity of whole human body imagery and the
elusiveness of fashion concepts, i.e., how some terms like ‘street fashion’
can induce different mental images to individuals. These two distinguish our
approach from previous endeavors that tried to manipulate human faces with
text guidance [37, 27], in that we need to explicitly reformulate our guidance
(i.e., fashion style) into a more illustrative and visually grounded learning
signal.

To this end, we propose two directions to visually reinforce the guidance
signal. First, we enrich the text prompt by querying a language model [47, 4]
to generate descriptions for the concept. Second, we retrieve reference images
for the target text from a fashion database we constructed through web
crawling, and guide our model to consult them for more descriptive guidance.
Leveraging the enhanced visual clarity, our Fashion Style Editing (FaSE)
framework adeptly transforms abstract fashion concepts into tangible human
imagery across a variety of applications.

##  2 Related Work

Image editing poses unique technical challenges stemming from the famous
distortion-editability trade-off, i.e., making necessary changes while
preserving the original. The most prominent approach is leveraging a
pretrained generative model to constrain the output to the learned data
manifold and executing edits in its latent space. StyleGAN family [19, 21, 22,
20] has been a popular choice for the generative prior for long, thanks to its
remarkable generation quality and well-behaved latent space [50, 30]. Many
works [37, 48, 40, 45, 27, 5] have demonstrated impressive editing performance
by exploring its learned latent space, but their application was mainly
confined to narrow semantic domains (e.g., human faces [19]) that can be
readily steered by simple words (e.g., “smile”) [37, 40]. Recently, diffusion-
based image editing [23, 35, 28, 54, 12, 26] is also being widely explored.
Human generation fundamentally differs from human face modeling as the
underlying complexity is much greater to synthesize realistic rendering of
whole human body with natural garments, body proportions and poses. StyleGAN-
Human [10] pioneered this direction of research from a data-centric
perspective, with following works [53, 11, 18] further pushing the boundary.
In our work, this learned prior serves as the starting point for fashion-
specific style editing.
Fashion image generation, due to its high practical impact, has been studied
from various angles, such as virtual try-on [7, 51, 36, 31, 34], clothing
synthesis [17, 42] and clothing editing [26]. However, manipulating human-
fitted fashion imagery is under-explored due to its innate complexity. FE-GAN
[9] proposes an image-based editing framework that relies on adversarial
training, but the editing operation is confined to few cases (e.g., sleeve
length) and the results are generally unsatisfactory. VPTNet [29] introduces
an attribute-based editing technique, which suffers from similar drawback that
only limited operations (sleeve and length) predefined at the training phase
are supported. In this work, we aim to steer the fashion style of human
imagery with high-level user input keywords (e.g., “formal”) in a highly
versatile and compute-efficient manner.

(a) Naive CLIP guidance.

(b) Descriptive guidance (FaSE).

Figure 2: Using CLIP text guidance as in StyleCLIP [37] is insufficient to
steer fashion concepts in full-body images, and image guidance in CLIP feature
space often distorts the semantics (left). FaSE, in contrast, successfully
projects non-trivial fashion styles (‘street-fashion’ in this example) onto
human images with both type of guidances (right). Figure 3: Summary of our
fashion style editing framework (FaSE). We learn a latent mapper [37] with two
types of illustrative guidance. (A) We transform fashion style concepts into
visual descriptions with a pretrained language model. (B) From our
preconstructed fashion image database, we retrieve top-k reference images that
both suit the target text prompt and resemble the source image, which are used
as additional visual guidance.

##  3 Approach

###  3.1 Preliminaries

Our framework is based on StyleGAN-Human [10], which serves as a pretrained
prior for human image generation. For editing, we employ the latent mappers
introduced in StyleCLIP [37]. Latent mappers are fully-connected networks that
transform any latent vector in StyleGAN’s $\mathcal{W}+$ space to the
direction that better aligns with the provided text prompt. Specifically, the
latent vector is split into three parts (coarse, medium, and fine), each
responsible for different level of detail. Each part undergoes mapping through
distinct networks ($M^{c}$, $M^{m}$, $M^{f}$, respectively) and is
subsequently added to the original latent vector before being fed to the
generator. The network is trained to minimize the cosine distance between the
generated image and the text prompt in CLIP joint embedding space along with
an $L_{2}$ regularization to preserve the original image.  Although it has
delivered impressive results in human face editing, its coarse guidance signal
is insufficient for controlling non-trivial fashion styles on full-body human
images (Fig. 2).

###  3.2 Textual Augmentation

The key to successful fashion style editing lies in providing specific visual
guidance that represents the intended concept. As StyleCLIP utilizes generic
CLIP loss with a naive text prompt $t$, training often fails for non-trivial
fashion concepts, as shown in Fig. 2. We hypothesize that this failure can be
attributed to the entanglement in StyleGAN-Human latent space and the innate
elusiveness of fashion terminologies. A straightforward yet effective solution
would be enriching the text prompt with detailed descriptions, thereby
enabling its decomposition into specific components that correspond to various
visual aspects of the concept. For example, in visualizing the concept of
‘formal fashion’, identifying specific components like ‘suit,’ ‘trouser,’ and
‘loafer’ proves to be beneficial. To this end, we leverage a Large Language
Model (LLM) to augment the text prompt $t$, making the context more
illustrative by concatenating component words associated with the given
concept. Formally, we train the latent mapper to minimize CLIP loss with
respect to the augmented text prompt as follows:

$\mathcal{L}_{CLIP}(y,t)=\mathcal{D}(\mathcal{E}_{CLIP}^{img}(y),\mathcal{E}_{CLIP}^{text}(g(t)))$

where $\mathcal{D}$ denotes the cosine distance, $\mathcal{E}_{CLIP}$ denotes
the CLIP encoder, $g$ stands for the textual augmentation operation using LLM
and $y$ is the edited image.

###  3.3 Editing with Visual Reference

Upon observing the effectiveness of textual augmentation, a natural
progression would be utilizing visual references corresponding to the target
concept, since an image is worth a thousand words. To this end, we first
construct a reference database by collecting clean human images corresponding
to various fashion style concepts from the web. Our high level idea is to
retrieve an array of relevant images and train our model not only to increase
the cross-modal CLIP similarity but also to maximize the image-level
similarity with the reference set. Our initial investigation into the CLIP
image space reveals its unsuitability for our task, prompting us to transition
into a more illustrative feature space, i.e., the $\mathcal{W}+$ space [37] of
the generator. Formally, our objective can be written as:

$\mathcal{L}_{Ref}(\mathbf{w^{\prime}},t)=\frac{1}{K}\sum_{s\in
S_{t,K}^{+}}\mathcal{D}(\mathbf{w^{\prime}},s),$

where $\mathbf{w}^{\prime}$ is the translated latent vector and $S_{t,K}^{+}$
is the collection of $\mathcal{W}+$ embeddings of top-K reference images based
on similarity.

As mentioned earlier, we empirically find that naively optimizing image
similarity in CLIP image space leads to sub-optimal results with undesired
distortions, mainly because CLIP visual features are too coarse to capture the
subtle nuances of fashion. Hence, we first invert the retrieved reference
images to $\mathcal{W}+$ space and train the mapper to push latent codes
towards them, injecting a more direct and illustrative learning signal. Top-K
retrieval is performed to suppress irrelevant guidance coming from overly
dissimilar reference images.

###  3.4 Navigating the Hierarchical Latent Space

StyleGAN2 [21] is known for its hierarchical latent space, which has motivated
many applications that leverage its coarse-to-fine structure. StyleCLIP [37],
for example, divides the latent space into three groups and empirically
discovered which level corresponds to specific facial editing operation. We
look for a similar empirical guideline for fashion editing on StyleGAN-
generated human images.

Following [37], we first compartmentalize the $\mathcal{W}+$ space into three
levels, and test editing operations of various granularity, e.g., changing the
color, neckline, garment category, etc. We discover that the coarse group has
little association with fashion as it is mainly responsible for human pose,
and mid/fine group controls shape/texture respectively. From Fig. 4, for
example, it is evident that in order to modify garment category, mid-level
transformation is necessary.

Figure 4: From the top, we train $M^{c}$, $M^{m}$, and $M^{f}$, respectively,
for the prompt ‘suit’. When editing the garment shape as in ‘suit’, $M^{m}$
needs to be modified to transform the mid part of $\mathbf{w}$.

##  4 Experiments

We choose StyleCLIP with StyleGAN-Human-v2
checkpoint111https://github.com/stylegan-human/StyleGAN-Human as our baseline.
To construct the reference DB, we collect fashion images from online
marketplaces. We select 12 categories (e.g., boxy fit, floral pattern, street
fashion, etc) and gather 35 images for each category.

###  4.1 Qualitative Results

Figure 5: Qualitative comparison with StyleCLIP using text and image guidance
signal.

We first qualitatively compare our method against StyleCLIP baseline using
both text (StyleCLIP-T, FaSE-T) and image (StyleCLIP-I, FaSE-I) signal. We can
see from Fig. 5 that while naive CLIP signal, whether it be textual or visual,
fails to modify human images, ours successfully alters them towards the target
direction. FaSE-I, in particular, incurs little change to the original image
semantics thanks to the retrieval mechanism.

| GPT-4V | User Study
---|---|---
Prompt | FaSE-T | FaSE-I | FaSE-T | FaSE-I
T-shirts | 55.9 | 58.8 | 56.9 | 52.0
Floral | 72.5 | 67.6 | 63.7 | 62.7
Boxy | 91.2 | 73.5 | 61.8 | 76.5
Formal | 91.2 | 96.1 | 90.2 | 94.1
Street | 95.1 | 88.2 | 79.4 | 80.4
Table 1: Text-image alignment using GPT-4V and user test. We report 1v1 win
rate against [37] on diverse text prompts.

###  4.2 Quantitative Results

For the quantitative evaluation, we report the 1v1 win rate as assessed by
GPT4-V and human raters. For GPT4-V evaluation, we prompt the model with the
template: “Between the two images, which do you think better aligns with the
fashion concept ‘[concept name]’?” Tab. 1 clearly shows that our methods have
comparative advantage over the baseline across various text prompts.

##  5 Conclusion

Our work extended text-driven image editing to fashion style editing by
exploring two techniques that visually clarify the guidance signal. We hope
our framework facilitates new and exciting applications in fashion field.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in neural information processing systems_ , 35:23716–23736, 2022.
  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18208–18218, 2022.
  * Bar-Tal et al. [2024] Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al.  Lumiere: A space-time diffusion model for video generation.  _arXiv preprint arXiv:2401.12945_ , 2024.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chai et al. [2021] Lucy Chai, Jonas Wulff, and Phillip Isola.  Using latent space regression to analyze and leverage compositionality in gans.  _arXiv preprint arXiv:2103.10426_ , 2021.
  * Chia et al. [2022] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhães, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue.  Contrastive language and vision learning of general fashion concepts.  _Scientific Reports_ , 12(1):18958, 2022.
  * Choi et al. [2021] Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo.  Viton-hd: High-resolution virtual try-on via misalignment-aware normalization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14131–14140, 2021.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dong et al. [2020] Haoye Dong, Xiaodan Liang, Yixuan Zhang, Xujie Zhang, Xiaohui Shen, Zhenyu Xie, Bowen Wu, and Jian Yin.  Fashion editing with adversarial parsing learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8120–8128, 2020.
  * Fu et al. [2022] Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen-Change Loy, Wayne Wu, and Ziwei Liu.  Stylegan-human: A data-centric odyssey of human generation.  _arXiv preprint_ , arXiv:2204.11823, 2022.
  * Fu et al. [2023] Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Wayne Wu, and Ziwei Liu.  Unitedhuman: Harnessing multi-source data for high-resolution human generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7301–7311, 2023.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  _arXiv preprint arXiv:2204.03458_ , 2022.
  * Jang et al. [2023a] Jiho Jang, Seonhoon Kim, Kiyoon Yoo, Chaerin Kong, Jangho Kim, and Nojun Kwak.  Self-distilled self-supervised representation learning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2829–2839, 2023a.
  * Jang et al. [2023b] Jiho Jang, Chaerin Kong, Donghyeon Jeon, Seonhoon Kim, and Nojun Kwak.  Unifying vision-language representation space with single-tower transformer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 980–988, 2023b.
  * Jiang and Fu [2017] Shuhui Jiang and Yun Fu.  Fashion style generator.  In _IJCAI_ , pages 3721–3727, 2017.
  * Jiang et al. [2022] Yuming Jiang, Shuai Yang, Haonan Qiu, Wayne Wu, Chen Change Loy, and Ziwei Liu.  Text2human: Text-driven controllable human image generation.  _ACM Transactions on Graphics (TOG)_ , 41(4):1–11, 2022.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Karras et al. [2020a] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  _Advances in neural information processing systems_ , 33:12104–12114, 2020a.
  * Karras et al. [2020b] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8110–8119, 2020b.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  _Advances in neural information processing systems_ , 34:852–863, 2021.
  * Kim et al. [2022] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2426–2435, 2022.
  * Kong and Kwak [2023] Chaerin Kong and Nojun Kwak.  Analyzing multimodal objectives through the lens of generative diffusion guidance.  _arXiv preprint arXiv:2302.10305_ , 2023.
  * Kong et al. [2022] Chaerin Kong, Jeesoo Kim, Donghoon Han, and Nojun Kwak.  Few-shot image generation with mixup-based distance learning.  In _European conference on computer vision_ , pages 563–580. Springer, 2022.
  * Kong et al. [2023] Chaerin Kong, DongHyeon Jeon, Ohjoon Kwon, and Nojun Kwak.  Leveraging off-the-shelf diffusion model for multi-attribute fashion image manipulation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 848–857, 2023.
  * Kwon and Ye [2022] Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18062–18071, 2022.
  * Kwon et al. [2022a] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh.  Diffusion models already have a semantic latent space.  _arXiv preprint arXiv:2210.10960_ , 2022a.
  * Kwon et al. [2022b] Youngjoong Kwon, Stefano Petrangeli, Dahun Kim, Haoliang Wang, Viswanathan Swaminathan, and Henry Fuchs.  Tailor me: An editing network for fashion attribute shape manipulation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3831–3840, 2022b.
  * Lang et al. [2021] Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T Freeman, Phillip Isola, Amir Globerson, Michal Irani, et al.  Explaining in style: Training a gan to explain a classifier in stylespace.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 693–702, 2021.
  * Lee et al. [2022] Sangyun Lee, Gyojung Gu, Sunghyun Park, Seunghwan Choi, and Jaegul Choo.  High-resolution virtual try-on with misalignment and occlusion-handled conditions.  In _European Conference on Computer Vision_ , pages 204–219. Springer, 2022.
  * Lee et al. [2023] Seungwoo Lee, Chaerin Kong, Donghyeon Jeon, and Nojun Kwak.  Aadiff: Audio-aligned video synthesis with text-to-image diffusion.  _arXiv preprint arXiv:2305.04001_ , 2023.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International conference on machine learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023] Zhi Li, Pengfei Wei, Xiang Yin, Zejun Ma, and Alex C Kot.  Virtual try-on with pose-garment keypoints guided inpainting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22788–22797, 2023.
  * Meng et al. [2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Morelli et al. [2022] Davide Morelli, Matteo Fincato, Marcella Cornia, Federico Landi, Fabio Cesari, and Rita Cucchiara.  Dress code: high-resolution multi-category virtual try-on.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2231–2235, 2022.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 2085–2094, 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International conference on machine learning_ , pages 8821–8831. Pmlr, 2021.
  * Richardson et al. [2021] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2287–2296, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Rostamzadeh et al. [2018] Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojciech Stokowiec, Ying Zhang, Christian Jauvin, and Chris Pal.  Fashion-gen: The generative fashion dataset and challenge.  _arXiv preprint arXiv:1806.08317_ , 2018.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Sauer et al. [2023] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.  Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis.  In _International conference on machine learning_ , pages 30105–30118. PMLR, 2023.
  * Shen and Zhou [2021] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1532–1540, 2021.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ , 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Tov et al. [2021] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–14, 2021.
  * Wang et al. [2022] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun.  Medclip: Contrastive learning from unpaired medical images and text.  _arXiv preprint arXiv:2210.10163_ , 2022.
  * Wu et al. [2021] Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12863–12872, 2021.
  * Xu et al. [2024] Yuhao Xu, Tao Gu, Weifeng Chen, and Chengcai Chen.  Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on.  _arXiv preprint arXiv:2403.01779_ , 2024.
  * Yu et al. [2022] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Zhang et al. [2022] Kaiduo Zhang, Muyi Sun, Jianxin Sun, Binghao Zhao, Kunbo Zhang, Zhenan Sun, and Tieniu Tan.  Humandiffusion: a coarse-to-fine alignment diffusion framework for controllable text-driven person image generation.  _arXiv preprint arXiv:2211.06235_ , 2022.
  * Zhang et al. [2023] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren.  Sine: Single image editing with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6027–6037, 2023.


# From Partial to Strictly Incremental Constituent Parsing

Ana Ezquerro, Carlos Gómez-Rodríguez and David Vilares
Universidade da Coruña, CITIC
Departamento de Ciencias de la Computación y Tecnologías de la Información
Campus de Elviña s/n, 15071
A Coruña, Spain
{ana.ezquerro, carlos.gomez<EMAIL_ADDRESS>
###### Abstract

We study incremental constituent parsers to assess their capacity to output
trees based on prefix representations alone. Guided by strictly left-to-right
generative language models and tree-decoding modules, we build parsers that
adhere to a strong definition of incrementality across languages. This builds
upon work that asserted incrementality, but that mostly only enforced it on
either the encoder or the decoder. Finally, we conduct an analysis against
non-incremental and partially incremental models.

##  1 Introduction

Incremental NLP aims to learn and adapt partial representations as information
unfolds. However, with the rise of bidirectional LSTMs Hochreiter and
Schmidhuber (1997) and Transformers Vaswani et al. (2017), recent research has
focused on non-incremental solutions. These models process the full input for
contextualization before they start generating any output. Therefore, this
approach does not capture the progressive unfolding of input over time, giving
the sense that all of it is available all of a sudden Madureira and Schlangen
(2020). This is not an issue for most NLP tasks, but it is relevant for
others, such as real-time NLP, e.g., instant machine translation or real-time
speech. Furthermore, work on incremental processing holds relevance in
interdisciplinary research, especially where computer science, linguistics,
and cognitive studies intersect.

While some studies have addressed the challenge of outputting incremental
structured representations - for various definitions of incrementality Konstas
et al. (2014); Köhn (2018); Shen et al. (2021) \- analyses of trees remain
limited, more notably since the popularization of deep learning, and are
mostly partially incremental approaches.

In this context, Titov and Henderson (2007), one of the first neural parsing
models, was also an incremental network based on sigmoid belief networks. This
generative model broke down the probability of a structure into probabilities
for individual derivation decisions, each influenced by previous decision
history. However, the computation was expensive and its evaluation was
restricted to sentences of up to 15 tokens in the English Penn Treebank Marcus
et al. (1993). For shift-reduce constituent parsing, Cross and Huang (2016)
proposed an incremental model with minimal features, focusing on only three
sentence positions to predict the next action. However, input sentences were
contextualized using bidirectional LSTMs, thus relying on non-incremental
encoders and effectively considering all upcoming words; a strategy that was
later widely adopted by most neural syntactic parsing architectures, but that
does not adhere to a definition of strong incrementality. More recently,
Kitaev et al. (2022) introduced a span-based model that incrementally encodes
input sentences into discrete elements using vectors from GPT-2 mapped into a
codebook. Despite this, it relied on bidirectional Transformers and a CYK
architecture Kitaev and Klein (2018) for decoding these vectors into trees.
Complementarily, Yang and Deng (2020) proposed an incremental decoder based on
graph neural networks. Although they referred to their parser as strongly
incremental, sentences were encoded with bidirectional architectures like BERT
or XLNET Devlin et al. (2019); Yang et al. (2019).

Incrementality has been also explored for other parsing formalisms. Stanojević
and Steedman (2019) developed an almost fully incremental parser for
combinatory categorical grammars (CCG), relying on ELMo embeddings Peters et
al. (2018) and a bidirectional LSTM for these predictions. Later, a genuinely
fully incremental CCG parser was introduced Stanojević and Steedman (2020),
using only ELMo’s forward pass and a left-to-right LSTM, addressing biases in
incremental CCG parsing. In the field of dependency parsing, incrementality
has been a focus since the pre-neural era Beuck and Menzel (2013); Köhn and
Menzel (2014); Köhn and Baumann (2016), with some models rivaling non-
incremental ones. Recently, Ezquerro et al. (2023) found that with current
neural architectures, incremental models for dependency parsing are less
effective than bidirectional approaches. However, incorporating human-like
reading strategies, such as brief delays, can significantly enhance
performance, particularly in languages with leftward dependencies.

##### Contribution

We study the viability and challenges of _fully incremental_ constituent
parsing with encoder-decoder architectures. All components strictly process
the sentence from left to right, adding each read word to the partial tree
based on the input prefix. For the encoder, we leverage generative LLMs. For
the decoder, we reassess two options that generate partial trees based solely
on current inputs: (i) an incremental parsing-as-tagging model Gómez-Rodríguez
and Vilares (2018), and (ii) a transition-based decoder that uses graph-
neural-network representations Yang and Deng (2020). The code is available at
https://github.com/anaezquerro/incpar.

##  2 Incremental Constituent Parsing

Let $w=(w_{1},...,w_{n})$ be a sequence of tokens such that
$w_{i}\in\mathcal{V}$ for some vocabulary of tokens $\mathcal{V}$, we are
interested in learning a function that can map $w$ into a constituent tree
$T$. Different from previous work, we are interested in modeling this function
as an strictly incremental model. Under this setup, the decision at time step
$i$ is based only on the _prefix_ $w_{1}...w_{i+k}$. It creates a partial
tree, $T_{i}$, where each word $w_{i}$ is added at its time step $i$, in a
monotonic way. The delay parameter, $k$, mimics human reading processes,
allowing for a slight look ahead to the upcoming words. Human parsing is
believed to be very swift, with latencies as short as 250 milliseconds
Pulvermüller et al. (2009); Bemis and Pylkkänen (2011). In this work, we will
study both zero and small positive delays, i.e., $k\in[0,2]$. Next, we review
our encoders (§2.1) and decoders (§2.2).

###  2.1 Incremental encoders

The incremental encoder is a parameterized function $\Psi_{\theta}$ that
produces a hidden representation vector $\mathbf{h}_{i}\in\mathbb{R}^{h}$ for
each input token $w_{i}$ based on its own prefix, thus
$\mathbf{h}_{i}=\Psi_{\theta}(w_{1}...w_{i})$. As for specific architectures,
will rely on encoders both without and with pre-training. The former is a
lower-bound baseline made of 4 stacked left-to-right LSTMs Hochreiter and
Schmidhuber (1997). For the latter, we use multilingual GPT (mGPT; Shliazhko
et al., 2022) and BLOOM-560M Scao et al. (2022). mGPT has pre-training data
for all languages studied, while BLOOM does not. This lets us measure the
impact of: (i) no pre-training data, (ii) pre-training data for all languages,
and (iii) missing pre-training data for some languages (see also §3).

###  2.2 Incremental decoders

We propose two different architectures to implement our incremental decoders.
In both cases, an intermediate module was added between the encoder and
decoder to add prefix information up to word $w_{i+k}$. At each timestep $i$,
this module accepts the encoder representations
$\mathbf{h}_{i}...\mathbf{h}_{i+k}$ and generates a new delayed
contextualization $\overline{\mathbf{h}}_{i}$ using a feed-forward network
($\overline{\mathbf{h}}_{i}=\textbf{FFN}(\mathbf{h}_{i}...\mathbf{h}_{i+k})$).
The delayed sequence
$\mathbf{\overline{H}}=(\overline{\mathbf{h}}_{1}...,\overline{\mathbf{h}}_{n})$
is directly passed as input to the decoder. Thus, these decoders produce an
extra piece of the output tree based strictly on the prefix $w_{1}...w_{i+k}$.

On the one hand, we use decoders rooted in sequence labeling parsing Gómez-
Rodríguez and Vilares (2018). Here, at each time step, each representation is
mapped to a partial label that encodes a segment of the constituent tree
primarily based on the preceding prefix. On the other hand, we choose the
incremental decoder by Yang and Deng (2020). They use a graph neural network
to contextualize the partial tree and make a decision (transition) at each
time step based on the read token.

####  2.2.1 Incremental decoding as tagging

Given a sequence of delayed word contextualizations
$\overline{\mathbf{H}}=(\overline{\mathbf{h}}_{1}...\overline{\mathbf{h}}_{n})$,
a tagging-based decoder maps each contextualization
$\overline{\mathbf{h}}_{i}$ to a label $\ell_{i}\in\mathcal{L}$ and defines an
injective and complete function to delinearize the sequence of labels into a
valid constituent tree. Following Gómez-Rodríguez and Vilares (2018), each
label is a tuple of the form $\ell_{i}$=$(d_{i},c_{i})\in\mathcal{L}$, where
$d_{i}$ encodes a number $l_{i}$, the total number of levels in common between
$w_{i}$ and $w_{i+1}$, and $c_{i}$ encodes the lowest non-terminal in common
between those two words.111The encoding is injective and complete for
constituent trees without unary chains. The specifics can be found in the
reference paper. Here, unary chains were collapsed in a single artificial
constituent and recovered in the decoding step. $l_{i}$ can be encoded in
$d_{i}$ either directly ($d_{i}=l_{i}$, absolute encoding) or as a difference
from the previous value ($d_{1}=l_{1}$ and $d_{i}=l_{i}-l_{i-1}$ for $i\geq
2$, relative encoding). See Figure 1 for an example.

We chose this encoder over newer sequence labeling linearizations that have
been recently published, such as tetra-tagging Kitaev and Klein (2020) and
shift-reduce parsing through pre-order, post-order, and in-order
linearizations Amini and Cotterell (2022). We did so due to a few practical
reasons: (i) it is more user-friendly with existing libraries for transforming
constituent trees into label sequences; and (ii) it accommodates non-binary
trees, like the juxtapose model (binarizing and unbinarizing is trivial, yet
necessary for these mentioned alternatives).222Also, even if Kitaev and Klein
and Amini and Cotterell report better results, it is worth noting that the
original papers cannot be directly compared in terms of results due to
different implementations. For instance, Gómez-Rodríguez and Vilares relied on
LSTMs and a simple decoder based on feed-forward networks, while the tetra-
tagging paper used BERT and did not employ a sequence labeling decoder, but
rather an efficient and simple dynamic programming approach.

Figure 1:  Absolute (orange) and relative (green) indexing from Gómez-
Rodríguez and Vilares (2018). Note that unary chains are collapsed in an
artificial constituent (first label). The final label indicates the end of
sentence.

That said, our decoder is straightforward. Given an incrementally delayed
contextualized input
$\overline{\mathbf{H}}=(\overline{\mathbf{h}}_{1},...,\overline{\mathbf{h}}_{n})$,
each label $\ell_{i}$ is computed as
$\textbf{FFN}_{\ell}(\overline{\mathbf{h}}_{i})$, where $\mathbf{FFN}_{\ell}$
is a 1-layered feed-forward network with a softmax activation.

####  2.2.2 Incremental decoding as transition-based parsing

Similar to the tag-based decoders, transition-based systems incrementally
process each word contextualization to generate a sequence of actions of
variable length $m$. Each action updates the system’s inner representation of
the partial tree until the sequence is fully processed and the final state
retrieves the complete predicted tree. As a transition-based decoder, we use
the strong incremental decoder by Yang and Deng (2020). It generates a
sequence of $n$ transitions, adding exactly one token to the partial tree at
each time step. Namely, each time step is represented by a partial tree
$T_{i-1}$, which it is updated based on the subsequent $w_{i}$ and the
rightmost chain of $T_{i-1}$ (denoted as $\mathcal{R}(T_{i-1})$)333Formally,
the rightmost chain of a tree $T_{i-1}$ is defined by the set of non-terminal
nodes whose rightmost fencepost coincides with the last word of the sentence
(see Figure 2). by performing one of these actions:

  * •

$\textit{attach}(\varphi^{\mathrm{tgt}},\varphi^{\mathrm{prt}})$: Attaches a
new subtree to $\mathcal{R}(T_{i-1})$. It creates a non-terminal parent node
$\varphi^{\mathrm{prt}}$ and puts the $w_{i}$ as its terminal node.
$\varphi^{\mathrm{prt}}$ also becomes the rightmost child of an existing non-
terminal node $\varphi^{\mathrm{tgt}}\in\mathcal{R}(T_{i-1})$.

  * •

$\textit{juxtapose}(\varphi^{\mathrm{tgt}},\varphi^{\mathrm{prt}},\varphi^{\mathrm{new}})$:
Replaces the non-terminal node $\varphi^{\mathrm{tgt}}\in\mathcal{R}(T_{i-1})$
with the node $\varphi^{\mathrm{new}}$. $\varphi^{\mathrm{tgt}}$ takes the
role of left child of $\varphi^{\mathrm{new}}$ (keeping its descendants). The
right child of $\varphi^{\mathrm{new}}$ is a fresh subtree rooted at
$\varphi^{\mathrm{prt}}$ with the new read word $w_{i}$ as only child.

Given a partial tree $T_{i-1}$, each span extended from fencepost $l-1$ to $r$
is represented according to Equation 1 as a concatenation of (i) an embedding
of the non-terminal symbol of the span ($\mathbf{c}_{l,r}$), and (ii) an
embedding corresponding to the difference of the positions $\mathbf{p}_{l}$
and $\mathbf{p}_{r}$. All the spans of the partial tree $T_{i-1}$ are stacked
together in a matrix $\mathbf{X}_{i}=[\mathbf{C}_{i},\mathbf{P}_{i}]$ and then
passed through a graph convolutional network (GCN) to obtain a new
contextualized matrix
$\tilde{\mathbf{X}}_{i}=[\tilde{\mathbf{C}}_{i},\tilde{\mathbf{P}}_{i}]$,
where each row vector $\tilde{\mathbf{x}}_{l,r}$ is split as
$\tilde{\mathbf{x}}_{l,r}=[\tilde{\mathbf{c}}_{l,r},\tilde{\mathbf{p}}_{l,r}]$
using the same input dimensions (see Equation 1) to separate positional from
constituent information. Given the contextualization of a new input word
$\overline{\mathbf{h}}_{i}$ with its positional embedding $\mathbf{p}_{i}$,
the scores to select the target node $\mathbf{s}_{i}^{\mathrm{tgt}}$ are
computed by two FFNs which operate with those word and span representations in
the rightmost chain, denoted as
$\tilde{\mathbf{X}}^{\mathcal{R}}_{i}=[\tilde{\mathbf{C}}^{\mathcal{R}}_{i},\tilde{\mathbf{P}}^{\mathcal{R}}_{i}]$
(Equation 2). Finally, the scores for the parent and new nodes
($\mathbf{s}_{i}^{\text{prt}}$ and $\mathbf{s}_{i}^{\text{new}})$ are
generated from $\overline{\mathbf{h}}_{i}$ and $\mathbf{p}_{i}$ vectors with
the weighted representation of the rightmost chain (Equation 3).

$\mathbf{x}_{l,r}=[\mathbf{c}_{l,r},(\mathbf{p}_{r}-\mathbf{p}_{l})/2]$ (1)
$\mathbf{s}^{\mathrm{tgt}}_{i}=\mathrm{FFN}_{c}([\tilde{\mathbf{C}}^{\mathcal{R}}_{i},\overline{\mathbf{h}}_{i}])+\mathrm{FFN}_{p}([\tilde{\mathbf{P}}^{\mathcal{R}}_{i},\mathbf{p}_{i}])$
(2)
$\mathbf{s}_{i}^{\mathrm{prt}},\mathbf{s}_{i}^{\mathrm{new}}=\mathrm{FFN}\Big{(}\big{[}\overline{\mathbf{h}}_{i},\mathbf{p}_{i},(\mathbf{s}^{\mathrm{tgt}}_{i}\tilde{\mathbf{X}}_{i}^{\mathcal{R}})\big{]}\Big{)}$
(3)

(a) Partial tree $T_{3}$ built upon word $w_{3}$.

(b) attach(tgt, prt).

(c) juxtapose(tgt, prt, new).

Figure 2: Transitions defined by Yang and Deng (2020) for a partial tree
$T_{3}$ when a new word $w_{4}$ is added. Nodes in $\mathcal{R}(T_{3})$ are
marked in blue color.

Figure 2 shows the update of a partial tree $T_{3}$ (Figure 2(a)) when
applying the attach (Figure 2(b)) or juxtapose (Figure 2(c)) actions. Note
that the target node always belongs to the rightmost chain and at least one
non-terminal node is added at each time step, producing always a valid partial
tree $T_{i}$.444Yang and Deng (2020) proved that the attach-juxtapose is
injective for constituent trees without unary chains.

##  3 Experiments

##### Setup

To create our models, we used the supar555https://parser.yzhang.site/ library
as our starting point. It implements non-incremental parsers for the main
parsing formalisms, including constituent parsing, and allows for plug-and-
play integration of most large language models, including generative ones. For
additional information, see Appendix A.2.

##### Data

We use both the English Penn Treebank Marcus et al. (1993) and the set of
multilingual treebanks released as a part of the SPMRL shared task Seddah et
al. (2013).666We do not report results for the Arabic treebank since it
requires a paid license to be used.

##### Metrics

We use labeled bracketing F1-score, with the COLLINS.prm (for PTB) and
evalb_spmrl.prm (for SPMRL) files.777BLOOM lacks pre-training data for German,
Hungarian, Hebrew, Swedish, Polish, and Korean. As mentioned earlier, this is
still useful to gather additional comprehension on how an incremental parser
with a generative LLM performs on languages it was not specifically pre-
trained for.

##### Upper-bound baselines

We compare our models against counterparts that are not fully incremental. On
the one hand, we consider Kitaev and Klein (2018)’s approach as an upper-bound
baseline, as it uses Transformers and a powerful CYK neural decoding method.
On the other hand, we explore partially incremental versions of our strong
incremental models as control baselines, where the encoder is replaced with a
bidirectional encoder, specifically XLM-RoBERTa Conneau et al. (2020).

###  3.1 Results

Table 3.1 presents the outcomes for the strict incremental models with $k=0$,
compared to the upper-bound and control parsers. The results suggest that the
main challenges in competing with bidirectional systems are primarily
associated with the encoder side. This finding is similar to observations made
by other authors for different paradigms, such as dependency parsing, as noted
by Ezquerro et al. (2023). Particularly, we observe in Table 3.1 that models
equipped with an incremental decoder and a non-incremental encoder (the
control columns) achieve near state-of-the-art results. However, the F1-score
substantially diminishes when switching to an incremental encoder. Across
encoders, mGPT performs best overall. For languages not included in its pre-
training data, BLOOM’s performance is closer (yet usually higher) to that of
the LSTM encoders, but it always performs worse than mGPT. We also observe
clear differences across decoders. The transition-based decoders, while
performing on average 10 points below the upper bound model, yield reasonable
representations. On the other hand, the incremental sequence-labeling decoders
achieve a subpar F1 score, on average 27 points below state-of-the-art parsers
and 17 points below the transition-based decoder.

|  | Incremental | Control | KK
---|---|---|---|---
|  | SL | TB | SL | TB
$\mathbin{\vbox{{\foreach\X in {pink, gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | en |  40.4$\to$a |  54.4r |  57.4r |  77.2$\to$ |  83.5 |  85.7 | 93.1r | 94.5 | 95.5
$\mathbin{\vbox{{\foreach\X in {pink, gptcolor, xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | eu |  59.0$\to$r |  60.1r |  64.1r |  71.4$\to$ |  76.5 |  81.8 | 91.1a | 92.8 | 93.6
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | de |  34.6$\to$r |  46.3a |  52.5a |  51.9$\to$ |  67.4 |  72.9 | 90.7a | 91.7 | 88.9
$\mathbin{\vbox{{\foreach\X in {pink, gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | fr |  39.7$\to$a |  50.2r |  53.8r |  64.9$\to$ |  71.7 |  74.5 | 86.0a | 86.6 | 91.5
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | he |  66.2$\to$a |  66.4r |  76.1r |  65.4$\to$ |  74.3 |  84.4 | 91.8a | 93.8 | 92.8
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | hu |  72.0$\to$r |  69.3r |  76.6r |  69.8$\to$ |  82.2 |  89.1 | 94.5a | 95.3 | 96.3
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | ko |  63.8$\to$r |  63.7r |  70.4r |  75.7$\to$ |  77.7 |  81.9 | 89.0r | 89.8 | 91.9
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | pl |  71.6$\to$a |  71.8r |  79.7a |  77.6$\to$ |  84.7 |  91.4 | 96.2a | 96.8 | 97.1
$\mathbin{\vbox{{\foreach\X in {gptcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | sv |  47.6$\to$r |  47.3r |  60.3r |  60.4$\to$ |  64.1 |  78.2 | 87.6a | 90.2 | 92.0
$\mu$ |  55.0 |  58.8 |  65.7 |  68.3 |  75.8 |  82.2 | 91.1 | 92.4 | 93.3

Table 1: Labeled F-score paired with best sequence labeling (SL) and
transition-based (TB) decoder. $\mu$ represents macro average results.
Superscripts denote the encoder choice: LSTM ($\to$), BLOOM-560M (), mGPT (),
XLM-RoBERTa (). Subscripts denote the decoder configuration: absolute (a),
relative (r), GCN () and FFN (). The upper bound baseline performance (KK,
Kitaev and Klein (2018)) is also included. Language codes come from ISO 639-1
and left colored dots indicate the pretraining availability in LMs.

Table 2 compares our incremental models with zero delay to counterpart
versions with delays of 1 and 2. The improvements are noticeable in both
decoders, especially from delay zero to one. On average, for the sequence
labeling decoder, moving from delay zero to one improves performance by 13.7
and 15.6 percentage points for the LSTM and mGPT encoders, respectively.
Meanwhile, the improvements from delay 1 to delay 2 show clear diminishing
returns, with only a 0.8 and 2.3 point improvement. The trend is similar for
the transition-based decoder. When setting $k$=$1$, it shows average
improvements of 8.2 points (using the vanilla LSTM encoder) and 4.5 points
(mGPT) compared to the strict incremental version. However, there is only a
0.8 and a 0.9 point improvement compared to the models with delay one. These
diminishing returns indicate that small delays are not enough to close the
gap, and strategies to improve incremental encoders such as prophecy tokens
that can simulate larger delays Madureira and Schlangen (2020) may be needed -
although tailored for parsing and contemporary language models.

|  | SL | TB
---|---|---|---
|  | LSTM ($\to$) | MGPT () | LSTM ($\to$) | MGPT ()
$\mathbin{\vbox{{\foreach\X in {pink,gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | en |  68.327.9 |  72.432.0 |  82.324.9 |  86.128.7 |  83.46.2 |  84.27.0 |  90.95.2 |  91.65.9
$\mathbin{\vbox{{\foreach\X in {pink,gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | eu |  78.019.0 |  77.218.2 |  84.820.7 |  86.822.7 |  81.710.3 |  81.29.8 |  87.15.3 |  88.06.2
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | de |  57.522.9 |  59.024.4 |  72.820.3 |  76.423.9 |  64.612.7 |  64.512.6 |  81.38.4 |  83.310.4
$\mathbin{\vbox{{\foreach\X in {pink,gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | fr |  59.019.3 |  60.420.7 |  75.221.4 |  78.524.7 |  73.58.6 |  76.411.5 |  81.06.5 |  83.08.5
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | he |  75.79.5 |  76.310.1 |  84.78.6 |  85.59.4 |  77.712.3 |  79.714.3 |  87.53.1 |  88.03.6
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | hu |  76.44.4 |  79.67.6 |  84.88.2 |  87.510.9 |  82.112.3 |  85.215.4 |  92.02.9 |  92.13.0
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | ko |  70.06.2 |  70.06.2 |  78.07.6 |  80.09.6 |  77.11.4 |  77.82.1 |  83.92.0 |  84.62.7
$\mathbin{\vbox{{\foreach\X in {gptcolor,xlmcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | pl |  83.111.5 |  82.010.4 |  91.411.7 |  92.412.7 |  86.28.6 |  87.810.2 |  93.62.2 |  94.22.8
$\mathbin{\vbox{{\foreach\X in {gptcolor}{\hbox{{\color[rgb]{0,0,0}\bf.}}}} }}$ | sv |  64.316.7 |  62.915.3 |  77.317.0 |  79.118.8 |  70.19.7 |  66.96.5 |  82.84.6 |  83.85.6
$\mu$ |  63.213.7 |  64.014.5 |  81.315.6 |  83.617.9 |  69.68.2 |  70.49.0 |  86.74.5 |  87.65.4
Table 2: LF scores with delay 1 and 2 (first and second subcolumn) Notation as
in Table 3.1. Subscripts denote performance boost over zero-delay fully
incremental results from Table 3.1.

(a) PTB

(b) Basque

(c) Hebrew

(d) Korean

Figure 3:  F-Score of absolute (orange), relative (green) and transition-based
(purple) decoders with mGPT (bars) and XLM-RoBERTa (dots) encoders per
constituent. Different textures are used for delay 0 (solid), 1 (dotted) and 2
(gridded).

Finally, some phrases may be more ambiguous than others in an incremental
setting due to factors such as sentence structure, word order, or semantics.
Figure 3 shows F1-scores for the most common non-terminals in diverse
languages: English, Hebrew, Basque, and Korean, for models with $k\in[0,1,2]$.
For space reasons, we include only the most coarse-grained non-terminals.
Unary chains are excluded. Across the board, positive delays, especially
$k=1$, have a much greater impact on sequence-labeling decoders, particularly
benefiting longer span types like noun, verb, and prepositional phrases (span
lengths are in Appendix Table 3). Also, behaviors across non-terminals and
languages can vary greatly with incrementality, e.g., while delay is crucial
for phrases such as VB for PTB or SBAR for Hebrew, its need is negligible for
others such as Hebrew ADJP.

##  4 Conclusion

This paper introduced a set of strictly incremental encoder-decoder
constituent parsers, using generative language models and two types of
decoders: one based on parsing as tagging, and the other on transition-based
parsing with partial graph neural network representations. We tested the
models in a diverse multilingual setting and also simulated human reading
processes with positive delays of a few upcoming words. The results suggest
that a significant portion of future challenges may be centered on the
encoding side, and in how different phrases might be affected by the absence
of bidirectionality. In this context, exploring research lines to inform the
decoder, such as speculative real-time generation of next tokens in real time,
could be a valuable step to explore parsing methods closer to human reading
processes.

## Acknowledgments

We thank the anonymous reviewers for their very complete and useful
suggestions. This work has received funding by the European Research Council
(ERC), under the Horizon Europe research and innovation programme (SALSA,
grant agreement No 101100615), ERDF/MICINN-AEI (SCANNER-UDC,
PID2020-113230RB-C21), Xunta de Galicia (ED431C 2020/11), Grant GAP
(PID2022-139308OA-I00) funded by MCIN/AEI/10.13039/501100011033/ and by ERDF
“A way of making Europe”, Cátedra CICAS (Sngular, University of A Coruña), and
Centro de Investigación de Galicia “CITIC”, funded by the Xunta de Galicia
through the collaboration agreement between the Consellería de Cultura,
Educación, Formación Profesional e Universidades and the Galician universities
for the reinforcement of the research centres of the Galician University
System (CIGUS).

## Limitations

##### Non-monotonicity

Our definition of incrementality applies exclusively to monotonic parsers. In
cases of non-monotonicity, a parser might abandon its existing partial output
and revise it as new information comes in. This is carefully discussed in
Ezquerro et al. (2023) for dependency parsing. Similarly, we chose to focus
solely on monotonic constituent parsers. First, our goal is to maintain a
straightforward implementation that is on par with others, avoiding the added
complexity that repair strategies entail. Second, dealing with non-
monotonicity requires thinking a thorough evaluation framework. In this
respect, comparing against (partial) incremental parsers is challenging, as
metrics must account for partial analysis, which is not accommodated by the
standard bracketing F1-measure. In turn, such metrics on partial analyses are
meaningless for non-incremental parsers, which often do not even produce any
partial outputs, precluding direct comparison against them.

##### Discontinuous constituent parsing

We restricted our analysis to continuous constituent parsing and observed that
modern incremental parsers still exhibit shortcomings in this area. Studying
the impact of incrementality on discontinuities, i.e., discontinuous spans
within a sentence that form specific constituents, presents a more challenging
aspect of constituent parsing. This phenomenon is particularly observed in
languages with free word order. In this regard, there are several avenues to
explore. For example, we could draw inspiration from the incremental
transition-based algorithm described by Coavoux and Crabbé (2017), or the
sequence labeling approach suggested by Vilares and Gómez-Rodríguez (2020),
which shows potential to be adapted to an incremental setup.

##### Experiments on lower-resourced languages

Unlike in other paradigms like dependency parsing, the availability of a
diverse range of treebanks spanning various typologies is more limited for
constituent parsing. We used the treebanks presently at our disposal, which
include the English Penn Treebank and the SPMRL treebanks. However, it is
worth noting that we were unable to access the Arabic dataset due to its paid
license. Yet, our experiments consider: English (Indo-European, Germanic),
Basque, German (Indo-European, Germanic), French (Indo-European, Romance),
Hebrew (Afro-Asiatic, Semitic), Hungarian (Uralic, Ugric), Korean, Polish
(Indo-European, Slavic) and Swedish (Indo-European, Germanic).

##### Availability of multilingual large language models

Research on generative LLMs is extensive, and many models are being released
contemporaneously with this paper. However, highly multilingual versions are
more rare. Two main available resources are BLOOM and mGPT, both of which we
evaluated. This double evaluation allowed us to establish differences in
performance, particularly in terms of incrementality, depending on whether the
models contained pre-training data for a given target language or not.

##### Computational capabilities

We lacked access to extensive computing infrastructure or a budget for cloud-
based scaling that would allow us to fine-tune more powerful multilingual
language models such as the LLaMa family. We had access to NVIDIA GeForce RTX
3090 servers (each with 24GB) and one NVIDIA A100 GPU (with 80GB). We managed
to fine-tune the smallest BLOOM language model (560M parameters) within
reasonable time frames. Although we could technically fit the 3B version on
the A100 GPU with a minimal batch size, the impractical training duration made
it infeasible for a comprehensive multilingual study like the one proposed in
this work.

## References

  * Amini and Cotterell (2022) Afra Amini and Ryan Cotterell. 2022.  On parsing as tagging.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8884–8900, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Bemis and Pylkkänen (2011) Douglas K. Bemis and Liina Pylkkänen. 2011.  Simple composition: A magnetoencephalography investigation into the comprehension of minimal linguistic phrases.  _Journal of Neuroscience_ , 31(8):2801–2814.
  * Beuck and Menzel (2013) Niels Beuck and Wolfgang Menzel. 2013.  Structural prediction in incremental dependency parsing.  In _Computational Linguistics and Intelligent Text Processing: 14th International Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part I 14_ , pages 245–257. Springer.
  * Coavoux and Crabbé (2017) Maximin Coavoux and Benoît Crabbé. 2017.  Incremental discontinuous phrase structure parsing with the GAP transition.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 1259–1270, Valencia, Spain. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.
  * Cross and Huang (2016) James Cross and Liang Huang. 2016.  Incremental parsing with minimal features using bi-directional LSTM.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 32–37, Berlin, Germany. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017.  Deep biaffine attention for neural dependency parsing.
  * Ezquerro et al. (2023) Ana Ezquerro, Carlos Gómez-Rodríguez, and David Vilares. 2023.  On the challenges of fully incremental neural dependency parsing.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics_ , pages 52–66, Nusa Dua, Bali. Association for Computational Linguistics.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1050–1059, New York, New York, USA. PMLR.
  * Gómez-Rodríguez and Vilares (2018) Carlos Gómez-Rodríguez and David Vilares. 2018.  Constituent parsing as sequence labeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1314–1324, Brussels, Belgium. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018.  Constituency Parsing with a Self-Attentive Encoder.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.
  * Kitaev and Klein (2020) Nikita Kitaev and Dan Klein. 2020.  Tetra-tagging: Word-synchronous parsing with linear-time inference.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6255–6261, Online. Association for Computational Linguistics.
  * Kitaev et al. (2022) Nikita Kitaev, Thomas Lu, and Dan Klein. 2022.  Learned incremental representations for parsing.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3086–3095, Dublin, Ireland. Association for Computational Linguistics.
  * Köhn (2018) Arne Köhn. 2018.  Incremental natural language processing: Challenges, strategies, and evaluation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2990–3003, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Köhn and Baumann (2016) Arne Köhn and Timo Baumann. 2016.  Predictive incremental parsing helps language modeling.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 268–277, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Köhn and Menzel (2014) Arne Köhn and Wolfgang Menzel. 2014.  Incremental predictive parsing with TurboParser.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 803–808, Baltimore, Maryland. Association for Computational Linguistics.
  * Konstas et al. (2014) Ioannis Konstas, Frank Keller, Vera Demberg, and Mirella Lapata. 2014.  Incremental semantic role labeling with Tree Adjoining Grammar.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 301–312, Doha, Qatar. Association for Computational Linguistics.
  * Madureira and Schlangen (2020) Brielen Madureira and David Schlangen. 2020.  Incremental processing in the age of non-incremental encoders: An empirical assessment of bidirectional models for incremental NLU.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 357–374, Online. Association for Computational Linguistics.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn treebank.  _Computational Linguistics_ , 19(2):313–330.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Pulvermüller et al. (2009) Friedemann Pulvermüller, Yury Shtyrov, and Olaf Hauk. 2009.  Understanding in an instant: Neurophysiological evidence for mechanistic language circuits in the brain.  _Brain and Language_ , 110(2):81–94.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Seddah et al. (2013) Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013.  Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages.  In _Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages_ , pages 146–182, Seattle, Washington, USA. Association for Computational Linguistics.
  * Shen et al. (2021) Yikang Shen, Shawn Tan, Alessandro Sordoni, Siva Reddy, and Aaron Courville. 2021\.  Explicitly modeling syntax in language models with incremental parsing and a dynamic oracle.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1660–1672, Online. Association for Computational Linguistics.
  * Shliazhko et al. (2022) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022.  mgpt: Few-shot learners go multilingual.
  * Stanojević and Steedman (2019) Miloš Stanojević and Mark Steedman. 2019.  CCG parsing algorithm with incremental tree rotation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 228–239, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Stanojević and Steedman (2020) Miloš Stanojević and Mark Steedman. 2020.  Max-margin incremental CCG parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4111–4122, Online. Association for Computational Linguistics.
  * Titov and Henderson (2007) Ivan Titov and James Henderson. 2007.  Constituent parsing with incremental sigmoid belief networks.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 632–639, Prague, Czech Republic. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vilares and Gómez-Rodríguez (2020) David Vilares and Carlos Gómez-Rodríguez. 2020.  Discontinuous constituent parsing as sequence labeling.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2771–2785, Online. Association for Computational Linguistics.
  * Yang and Deng (2020) Kaiyu Yang and Jia Deng. 2020.  Strongly incremental constituency parsing with graph neural networks.  _Advances in Neural Information Processing Systems_ , 33:21687–21698.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.

##  Appendix A Appendix

###  A.1 Treebank statistics

Table 3 shows the frequency and average length (defined as the difference
between initial and final fencepost) of the constituents displayed in Figure
3.

en | NP | VP | PP | S | SBAR | ADJP | QP
---|---|---|---|---|---|---|---
$\lambda$ | 4.96 | 11.2 | 5.96 | 12.71 | 12.2 | 5.02 | 2.97
% | 43.13 | 24.56 | 16.95 | 6.29 | 3.6 | 1.79 | 1.7
eu | S | GV | SN | GRUP | SP | COORD | SADV
$\lambda$ | 8.86 | 2.31 | 3.66 | 2.26 | 3.38 | 2.04 | 2.61
% | 34.07 | 18.47 | 15.46 | 12.38 | 11.2 | 2.8 | 2.08
de | NP | PP | VN | SENT | COORD | VPINF | NC
$\lambda$ | 6.37 | 6.5 | 2.52 | 30.44 | 9.4 | 12.35 | 2.59
% | 32.79 | 26.25 | 7.04 | 6.04 | 4.53 | 3.2 | 2.92
fr | NP | PP | S | VP | AP | PN | CNP
$\lambda$ | 3.42 | 3.69 | 8.5 | 5.81 | 3.09 | 2.16 | 4.72
% | 28.66 | 25.55 | 21.22 | 9.16 | 4.37 | 2.98 | 2.75
he | NP | PP | S | SBAR | ADJP | NNPP | VP
$\lambda$ | 5.14 | 5.93 | 17.21 | 13.15 | 2.69 | 2.65 | 10.67
% | 44.19 | 21.68 | 15.21 | 5.4 | 4.44 | 3.2 | 2.56
hu | NP | CP | ADJP | PP | XP | ADVP | V
$\lambda$ | 3.58 | 14.97 | 3.79 | 4.09 | 7.36 | 2.88 | 2.0
% | 57.11 | 29.3 | 7.54 | 3.97 | 1.26 | 0.66 | 0.16
ko | VP | NP | ADJP | AUXP | S | ADVP | IP
$\lambda$ | 6.76 | 3.79 | 6.29 | 2.14 | 13.45 | 3.03 | 2.06
% | 49.75 | 36.0 | 8.04 | 4.1 | 1.53 | 0.39 | 0.16
pl | FNO | ZDANIE | FPM | FWE | FZD | FPT | FORMACZAS
$\lambda$ | 3.61 | 8.09 | 3.3 | 3.57 | 8.4 | 3.9 | 2.04
% | 34.18 | 28.15 | 19.79 | 5.39 | 3.56 | 3.34 | 2.21
sv | NP | S | PP | VP | XP | AP | AVP
$\lambda$ | 4.61 | 12.14 | 4.61 | 7.64 | 4.24 | 2.65 | 3.51
% | 31.1 | 27.35 | 20.69 | 10.35 | 6.46 | 2.48 | 0.88
Table 3:  Frequency (%) and average length ($\lambda$) of most frequent
constituents of each treebank. Root and unary spans were removed.

###  A.2 Hyperparameters configuration

Tables 4 and 5 show the configuration of the models and the training
hyperparameters for each encoder type. In pretrained models, each sentence
$(w_{1},...,w_{n})$ was passed through all encoder layers to compute the last
hidden state $(\mathbf{e}_{1},...,\mathbf{e}_{n})$ and then projected to a new
reduced space of dimension $h$ with a feed-forward network. The final reduced
sequence $\mathbf{H}=(\mathbf{h}_{1},...,\mathbf{h}_{n})$ is the one passed to
the delay module. In the case of non-pretrained encoders, each word $w_{i}$
was represented as a concatenation of (i) a word embedding of dimension
$h_{w}$, (ii) the PoS tag embedding of dimension $h_{p}$ and (iii) the last
hidden state of a Character-LSTM Dozat and Manning (2017) of dimension
$h_{c}$, resulting into a final input embedding
$\mathbf{w}_{i}\in\mathbb{R}^{h_{w}+h_{p}+h_{c}}$. The input matrix
$\mathbf{W}=(\mathbf{w}_{1},..,\mathbf{w}_{n})$ is introduced to the LSTM
encoder (with randomly initialized weights) and its last hidden states
$(\mathbf{h}_{1},...,\mathbf{h}_{n})$ are passed through the delay module and
the decoder. The decoder is a 3-layered Graph Convolutional Layer for the
Attach-Juxtapose parser or a feed-forward network for the case of the sequence
labeling decoder. The complete network was trained with the CrossEntropy loss
function and AdamW as optimizer, adapting the batches to the model size.
Dropout was set in both encoder and decoder and the best validation
performance was finally retrieved.

Hyp. | LLM | Non-pretrained
---|---|---
XLM | BLOOM | mGPT | LSTM$\to$ | BiLSTM$\leftrightarrow$
word emb. ($h_{w}$) | - | 300
PoS emb. ($h_{p}$) | - | 100
char. emb. | - | 50
char. LSTM ($h_{c}$) | - | 100
# enc. layers | 1 | 4
enc. emb. ($h$) | 100 | 400
% enc. dropout | 0.33 | 0.33sh.
# GCN layers | 3 | 3
# FFN layers | 1 | 1
% dec. dropout | 0.33 | 0.33sh.
Table 4: Model configuration for pretrained and non-pretrained models. The number of encoder layers for LLMs refers to the number of last hidden states obtained for each word. LSTM-based encoders use the shared-dropout technique (Gal and Ghahramani, 2016) as described in Dozat and Manning (2017). Hyp. | LLM | Non-pretrained
---|---|---
XLM | BLOOM | mGPT | LSTM$\to$ | BiLSTM$\leftrightarrow$
optimizer | AdamW | AdamW
lr | 5e-5 | 1e-3
lr decay | linear (0.5) | exponential (0.1)
epochs | 30 | 200
batch size | 500 | 500 | 100 | 5000
Table 5: Training hyperparameters for pretrained and non-pretrained models.
AdamW is set as optimizer with $\beta_{0}=0.9$, $\beta_{1}=0.9$ and
$\varepsilon=10^{-12}$, and batch sampling is fixed to minimize sequence
padding.

Finally, Table 6 displays various estimates of inference speeds for different
models.

| SL ()
---|---
| LSTM$\to$ | BiLSTM$\leftrightarrow$ | BLOOM | mGPT | XLM
en | 856.02 | 688.98 | 354.21 | 144.03 | 403.09
eu | 2096.61 | 1265.82 | 424.77 | 168.95 | 523.03
fr | 678.24 | 467.33 | 281.33 | 116.02 | 313.66
de | 1348.32 | 969.85 | 306.75 | 155.42 | 417.9
he | 497.8 | 673.3 | 192.07 | 122.14 | 323.24
hu | 1258.99 | 809.41 | 233.1 | 124.26 | 390.06
ko | 1838.16 | 1477.9 | 261.41 | 151.59 | 486.57
pl | 2091.04 | 1725.57 | 439.35 | 242.28 | 631.53
sw | 1465.52 | 1132.12 | 325.1 | 174.44 | 507.27
$\mu$ | 1347.86 | 1023.36 | 313.12 | 155.46 | 444.04
| TB ()
| LSTM$\to$ | BiLSTM$\leftrightarrow$ | BLOOM | mGPT | XLM
en | 191.21 | 188.75 | 146.69 | 90.83 | 152.34
eu | 569.21 | 524.83 | 295.27 | 135.45 | 341.40
fr | 109.87 | 106.04 | 90.20 | 62.09 | 94.29
de | 249.92 | 241.20 | 160.34 | 102.48 | 190.38
he | 164.55 | 155.95 | 94.11 | 72.38 | 130.94
hu | 235.16 | 224.06 | 138.12 | 90.26 | 181.44
ko | 570.62 | 517.57 | 200.15 | 128.72 | 306.19
pl | 717.30 | 586.34 | 297.53 | 196.40 | 400.62
sw | 331.67 | 286.10 | 196.45 | 116.33 | 229.96
$\mu$ | 348.83 | 314.54 | 179.87 | 110.55 | 225.28
Table 6: Inference speed (in sentences per second) of the evaluated models
across different languages. Symbols come from Table 3.1.