# Caption Feature Space Regularization for Audio Captioning

Yiming Zhang, Hong Yu, Ruoyi Du, Zhanyu Ma†, , Yuan Dong Y. Zhang, R. Du, Z.
Ma, D. Yuan are with the Pattern Recognition and Intelligent System
Laboratory, School of Artificial Intelligence, Beijing University of Posts and
Telecommunications, Beijing 100876, China. E-mail: {zhangyiming, beiyoudry,
mazhanyu<EMAIL_ADDRESS>is with Department of Artificial
Intelligence, School of Information and Electrical Engineering, Ludong
University, Yantai, Shandong 264025, China, Email<EMAIL_ADDRESS>The
corresponding author.

###### Abstract

Audio captioning aims at describing the content of audio clips with human
language. Due to the ambiguity of audio, different people may perceive the
same audio differently, resulting in caption disparities (_i.e._ , one audio
may correlate to several captions with diverse semantics). For that, general
audio captioning models achieve the one-to-many training by randomly selecting
a correlated caption as the ground truth for each audio. However, it leads to
a significant variation in the optimization directions and weakens the model
stability. To eliminate this negative effect, in this paper, we propose a two-
stage framework for audio captioning: (i) in the first stage, via the
contrastive learning, we construct a proxy feature space to reduce the
distances between captions correlated to the same audio, and (ii) in the
second stage, the proxy feature space is utilized as additional supervision to
encourage the model to be optimized in the direction that benefits all the
correlated captions. We conducted extensive experiments on two datasets using
four commonly used encoder and decoder architectures. Experimental results
demonstrate the effectiveness of the proposed method. The code is available at
https://github.com/PRIS-CV/Caption-Feature-Space-Regularization.

###### Index Terms:

Audio captioning, Contrastive learning, Cross-modal task, Caption consistency
regularization

##  I Introduction

audio captioning is a cross-modal translation task that requires extracting
features from audio and combining them with a language model to describe the
contents of audio [1, 2, 3]. However, unlike the speech recognition task that
transcribes speech to text [4], the audio captioning task focuses on
recognizing human-perceived information in general audio signals and
expressing it with natural language. The information of generated caption
includes the sound event, the acoustic scene, and some other high-level
semantic information such as concepts, physical properties, and high-level
knowledge [2].

Different from the visual captioning tasks in which people can easily describe
the visual object by its shape, color, size, and its position relative to
other objects [5]. However, for the audio clips, its information can be much
more ambiguous than the information of images or videos [2, 5]. Even for
people, precisely distinguishing events in audio can be difficult, let alone
effectively describing the contents of given audio, because the description is
often dependent on the situation or context as much as the audio itself.
Therefore, due to the ambiguity of audio, different persons may have varying
perceptions of the same audio, which will result in the semantic disparity of
audio captions [2], for example, a thin plastic rattling could be perceived as
a fire crackling [6] (as shown in Fig. 1).

Figure 1: Due to the ambiguity of audio, people may have different perceptions
of the same audio.

In the audio captioning task, each audio has multiple captions to describe the
contents, and one caption for each training iteration is randomly selected as
the training objective. While due to the disparity of captions on the same
audio clip, randomly selecting one caption as the objective will lead to a
large variance in the optimization direction of the model, which may harm the
performance of the model and make it more difficult to converge. We believe
that these captions are just semantically disparity, and the latent similarity
between captions that describe the same audio clip is not aware during the
current training scheme.

Therefore, in order to solve the problem caused by the semantic disparity of
captions, we propose a caption proxy space regularization method.
Specifically, the proposed method includes two training stages: (i) the first
stage learns a proxy feature space of the captions by minimizing the distance
between different captions belonging to the same audio and pushing away the
distance between different captions of different audio, (ii) and the second
stage trains an audio caption model with the previously built proxy space as a
regularization term, i.e., utilizing the proxy feature space as an additional
optimization goal to reduce optimization variance.

Figure 2: The system overview of our proposed method. The mainstream encoder-
decoder audio captioning framework is in the Block A, and the Block B is our
proposed caption feature space regularization module. $d_{k}$ is extracted in
the first stage which is described in section III-B and then the $d_{k}$ is
used to regularize the training of the caption generation (described in
section III-C).

Our contributions are as follows:

  * 1)

We propose a two-stage caption feature space regularization method to mitigate
the effect of caption disparity on audio captioning task.

  * 2)

Two different pre-trained encoders and two different decoders are used to
verify the effectiveness of the method.

  * 3)

On two prominent audio captioning datasets, Clotho-v1 and Clotho-v2 [2], the
results of our proposed method are compared to the results of the state-of-
the-art, indicating that our proposed methods achieve competitive results.

##  II Related Work

The audio captioning task is firstly introduced in [1], which proposed the
commercial ProSound Effects [6] audio corpus as a proof of concept. The paper
proposed a BiGRU [7] based encoder-decoder model to generate audio captions.
And due to the success of the audio captioning task in DCASE 2020 and 2021
[8], this task has gotten the attention of an increasing number of
researchers, and several methods to address this issue have been proposed. H.
Wang et al. [9] proposed a decoder with a temporal attention mechanism that
uses more acoustic information for each time step. K. Chen et al. [10] used
the combination of a pre-trained encoder and a Transformer decoder which makes
the latent variable result more efficient in generating captions. X. Xu et al.
[11] investigated the effect of local and global information on the audio
captioning task by comparing two pre-training tasks. The semantic information
is also investigated in order to improve the audio captioning task’s
performance. The semantic attributes were originally used in [12], where
AudioSet labels were used as semantic attributes by using the labels of the
nearest video clip. And Eren et al. [13] used the audio encoder to get audio
embeddings and a text encoder to get subject-verb embeddings, combine these
embeddings and decode them in the decoder.

##  III Proposed Method

###  III-A The Overview of Audio Captioning System

The current mainstream training paradigm of the audio captioning task is the
end-to-end encoder-decoder framework, as shown in Block A of Fig. 2.

The training data for the task consists of paired audio and captions data. The
training set of $N$ audio-captions pair
$D=\left\\{\left(x_{n},\mathbb{C}_{n}\right)\right\\}_{n=1}^{N}$, where
$x_{n}\in{R}^{T\times F}$ is the log mel-spectrogram of the $n$th audio clip
with $T$ frames and $F$ Mel filters,
$\mathbb{C}_{n}=\left\\{C_{mn}\right\\}_{m=1}^{M}$ is all the captions of the
$n$th audio clip and $C_{mn}$ is the $m$-th caption, which contains $L$ tokens
$\left\\{c_{mn}^{l}\right\\}_{l=1}^{L}$.

The audio encoder takes an audio clip’s log mel-spectrogram $x$ as input and
extracts its latent representation $Z\in{R}^{T^{\prime}\times F^{\prime}}$.
The decoder then aligns and calculates the latent representation $Z$ with the
hidden states of tokens, and the caption representation can be generated by
the decoder, denoted as $H\in{R}^{L\times D}$, which contains $L$ vectors
$\left\\{h_{l}\right\\}_{l=1}^{L}$, where the dimension of $h_{l}$ is $D$ and
the number of vectors is equal to the token length of caption $C$. Hence, each
vector $h_{l}$ corresponds to the token $c^{l}$ in the objective caption, the
vectors are utilized to predict the probability of the words over the
vocabulary after passing through the softmax layer. The predicted words are
$\left\\{w_{l}\right\\}_{l=1}^{L}$.

The cross-entropy loss function is used for word-level classification of
decoder, the loss function is

$\ell_{\mathrm{CE}}(\theta;C,x)=-\sum_{l=1}^{L}c^{l}\cdot\log
p\left(w_{l}\mid\theta,x\right).$ (1)

For the audio captioning task, an audio clip has multiple captions to describe
how different annotators feel about it. Different perceptions cause the
disparity of the captions in semantics, however, for each audio, only one
caption will be randomly selected as the ground truth for each training to
generate the description of the audio, and this training strategy can easily
affect the performance of the model and make it unstable.

To solve the above problem, we propose a proxy feature optimization method to
regularize the training of the caption generation, the key module is shown in
Block B of Fig. 2. Our method is a two-stage audio captioning method, in which
the first stage uses contrastive learning to generate the proxy space and
extract the proxy caption embedding of each audio $d_{k}$ (described in
section III-B) and then the proxy embedding $d_{k}$ is used in the module to
regularize the training of the caption generation (described in section
III-C).

Figure 3: The system overview of the first stage. Different colors indicate
caption embeddings from different audio clips. The colored areas are the
positive similarity and the grey areas are negative in the similarity matrix.
$N$ and $M$ in this figure are 3.

###  III-B The First Stage : Generation of Caption Proxy Space

To reduce the effect of caption disparity on model training. We use the
contrastive learning method to reduce the distance in proxy space between
captions that belong to the same audio and increase the distance between
captions that belong to different audio. In this subsection, we introduce the
generation of the proxy space and the extraction of the caption proxy
embedding.

####  III-B1 Training method

As is depicted in Fig. 3, $N\times M$ captions are fetched to build a batch.
These captions are from $N$ different audio, and each audio has $M$ captions.
The symbol $C_{mn}$ ($1\leq n\leq N$ and $1\leq m\leq M$) represents the $m$th
caption from the $n$th audio clip.

We tokenize the captions and utilize Bert [14] to extract its word embeddings.
Then the word embeddings are fed into the LSTM network. As a network
transformation, a linear layer is connected to the LSTM layer. The output of
the network is $\mathcal{G}\left(C_{mn};\delta\right)$ where the parameters of
the network are represented as $\delta$. The caption embedding vector is
defined as

$e_{mn}=\mathrm{Mean}\left(\mathcal{G}\left(C_{mn};\delta\right)\right)+\mathrm{Max}\left(\mathcal{G}\left(C_{mn};\delta\right)\right),\vspace{-3pt}$
(2)

where $e_{mn}$ represents the proxy embedding of the $n$th audio’s $m$th
caption.

The centroid of the proxy embeddings from the $k$th audio
$\left[e_{1k},\cdots,e_{Mk}\right]$ is denoted as $d_{k}$, and the scaled
cosine similarities between each proxy embedding $e_{mn}$ to all centroids
$d_{k}$ are defined by the similarity matrix $S_{MN,N}$ ($1\leq n,k\leq N$ and
$1\leq m\leq M$):

$S_{mn,k}=a\cdot\mathrm{cosine}\left(e_{mn},d_{k}\right)+b,\vspace{-3pt}$ (3)

where $a$ and $b$ are learnable parameters, the weight is limited to be
positive ($a>0$). And the centroid $d_{k}$ can be calculated by

$d_{k}=\left\\{\begin{array}[]{ll}\frac{1}{M}\cdot\sum_{j=1}^{M}e_{jk},&if\
k\neq n\\\ \frac{1}{M-1}\cdot\sum_{j=1,j\neq m}^{M}e_{jk},&if\
k=n\end{array}.\right.\vspace{-3pt}$ (4)

In Eq. (4), when calculating negative similarity ($k\neq n$), the centroid
$d_{k}$ is the average of all the proxy embeddings of $k$th audio, while when
calculating positive similarity ($k=n$), we eliminate $e_{mk}$ to compute the
centroid.

During the training, the proxy embeddings of each audio should be similar to
its centroid, but far from the centroid of other audio in the proxy space. As
shown in the similarity matrix in Fig. 3, the colored areas should have large
values, whereas grey areas should have small values. So for each proxy
embedding, the loss function is designed as

$L\left({e}_{mn}\right)=-{S}_{mn,n}+\log\left(\sum_{k=1,k\neq
n}^{N}\exp\left({S}_{mn,k}\right)\right).\vspace{-3pt}$ (5)

After the training of the first stage is completed, we extract the centroid
$d_{k}$ of each audio to represent its caption proxy embedding.

####  III-B2 Training details

In this stage, we use a single-layer LSTM network followed by a linear layer,
the dimensions of the LSTM and linear layer are 1024 and 512, respectively.
When training the model, each batch contains $N$ = 64 audio clips and $M$ = 3
captions per audio. The Adam optimizer is used to train the network, the
learning rate is 0.01 and the number of training epochs is 500. The scaling
factors ($a$,$b$) are initialized as ($10$,$-5$).

###  III-C The Second Stage: Regularize The Training of Caption Generation

As shown in Block B of Fig. 2, the caption proxy embedding $d_{k}$ is used in
the constraint module to regularize the training of the caption generation.

####  III-C1 Training method

We obtain the embedding $\hat{e}_{k}$ of the predicted caption by averaging
pooling and max pooling operations on the decoder output representation $H$
along the time axis, as shown in

$\hat{e}_{k}=\mathrm{Mean}(H_{k})+\mathrm{Max}(H_{k}),\vspace{-5pt}$ (6)

where $\hat{e}_{k}$ is the embedding of the $k$th audio predicted caption.

In addition to the cross-entropy loss, we proposed the proxy constraint loss
to reduce the effect of the caption disparity (see Eq. (7)), as shown in

$\ell_{\mathrm{PC}}(\theta;d_{k},x_{k})=1-\mathrm{cosine}(\hat{e}_{k},d_{k}).$
(7)

In this way, a small loss means $\hat{e}_{k}$ has a high similarity with the
audio caption proxy embedding $d_{k}$ obtained by the first stage.
Accordingly, the final training objective function is the weighted sum of
cross-entropy loss and proxy constraint loss, as shown in

$\ell(\theta;d_{k},C_{k},x_{k})=\ell_{\mathrm{CE}}(\theta;C_{k},x_{k})+\lambda\cdot\ell_{\mathrm{PC}}(\theta;d_{k},x_{k}),\vspace{-3pt}$
(8)

where $\lambda$ is a hyperparameter.

TABLE I: Experimental results under Clotho-v1 and Clotho-v2 evaluation sets Dataset | Model | Encoder | Decoder | BLEU1 | BLEU4 | ROUGEL | METEOR | CIDEr | SPICE | SPIDEr
---|---|---|---|---|---|---|---|---|---|---
Clotho-v1 | Baseline | PANN | GRU | 53.6 | 13.8 | 35.9 | 16.3 | 34.1 | 11.0 | 22.6
Transformer | 50.2 | 11.9 | 34.0 | 16.0 | 32.2 | 10.7 | 21.4
Pretrain-CNN | GRU [11] | 54.3 | 14.5 | 36.2 | 16.9 | 36.5 | 11.7 | 24.1
Transformer | 50.1 | 12.3 | 34.6 | 15.9 | 31.5 | 10.7 | 21.1
Fine-tuned PreCNN Transformer [10] | CNN | Transformer | 53.4 | 15.1 | 35.6 | 16.0 | 34.6 | 10.8 | 22.7
Temporal attention model [9] | CNN | LSTM | 48.9 | 10.7 | 32.5 | 14.8 | 25.2 | 9.1 | 17.2
Ours | PANN | GRU | 55.2 | 15.0 | 36.7 | 17.1 | 36.9 | 11.6 | 24.2
Transformer | 51.3 | 13.1 | 34.9 | 16.5 | 33.3 | 11.2 | 22.3
Pretrain-CNN | GRU | 55.5 | 15.3 | 36.8 | 17.1 | 37.5 | 11.7 | 24.6
Transformer | 52.4 | 13.3 | 35.5 | 16.6 | 35.1 | 11.4 | 23.2
Clotho-v2 | Baseline | PANN | GRU | 54.5 | 14.9 | 36.5 | 17.0 | 36.9 | 11.5 | 24.2
Transformer | 52.0 | 12.9 | 35.0 | 16.8 | 34.0 | 11.3 | 22.6
Pretrain-CNN | GRU [15] | 55.2 | 15.3 | 37.1 | 17.3 | 38.7 | 11.8 | 25.2
Transformer | 51.7 | 12.5 | 34.9 | 16.5 | 33.6 | 11.2 | 22.4
Transformer+RNN-LM [16] | PANN | Transformer+RNN-LM | 53.3 | 14.6 | 35.5 | 15.4 | 34.1 | 10.6 | 22.4
CL4AC [17] | PANN | Transformer | 55.3 | 14.3 | 37.4 | 16.8 | 36.8 | 11.5 | 24.2
Ours | PANN | GRU | 55.8 | 15.9 | 37.4 | 17.5 | 39.7 | 12.0 | 25.9
Transformer | 55.9 | 15.9 | 37.3 | 17.1 | 37.3 | 11.6 | 24.4
Pretrain-CNN | GRU | 56.1 | 16.0 | 37.5 | 17.6 | 40.6 | 12.1 | 26.3
Transformer | 55.9 | 15.9 | 37.6 | 17.2 | 37.5 | 11.5 | 24.5

####  III-C2 Training details

In this stage, we use two different 10-layer CNN pre-trained encoder models,
PANN111https://zenodo.org/record/3987831 and Pretrain-
CNN222https://zenodo.org/record/5090473, and the model structures of them can
be found in [18, 11]. And the single-layer attention-based GRU [19] and 2
layers Transformer [20] as our backbone decoder for its success in previous
audio captioning works.

SpecAugment [21] and label smoothing [22] are applied to prevent overfit. We
also apply scheduled sampling to gently decrease the training-inference
discrepancy caused by teacher forcing training [23]. The output dimensions of
the decoder are 512 and the Transformer decoder has 4 heads. As for the GRU
decoder, the Adam optimizer is used to train the network, the initial learning
rate is $5\times 10^{-4}$ and the total number of training epochs is 25. For
the Transformer decoder, the initial learning rate is $5\times 10^{-3}$ and
warm-up is used to increase the learning rate linearly to the initial learning
rate in the first five epochs, the total number of training epochs is 30 and
the learning rate is reduced to 1/10 of its original value every 10 epochs.
And the hyperparameter $\lambda$ is $0.5$.

##  IV Experiments

###  IV-A Metrics

We use the same metrics used in the DCASE Challenge to evaluate the
performance of our proposed method, including machine translation metrics:
BLEUn [24], ROUGEL [25], METEOR [26] and captioning metrics: CIDEr [27], SPICE
[28], SPIDEr [29]. And the $n$ in BLEU means $n-$gram.

The machine translation metrics are used to measure the word accuracy and
recall of generated text compared to the ground truth. The captioning metrics
are customized to the captioning task, taking into consideration the scene
graph contained within the generated caption as well as the $n$-gram’s
frequency-inverse document frequency (TF-IDF). The semantic fidelity and
syntactic fluency of the generated captions are ensured by taking into account
the scene graph and the TF-IDF of $n$-gram.

###  IV-B Datasets

The Clotho dataset [2] and the Audiocaps dataset [12] are two datasets that
are commonly used for audio captioning tasks. There are two versions of the
Clotho dataset, Clotho-v1 and Clotho-v2 respectively. Each audio clip has five
captions describing the audio contents, and the annotator uses only the audio
signal for annotation. However, the Audiocaps dataset is not generated in an
ideal environment [12]. So in this work, we only focus on the Clotho-v1 and
Clotho-v2 datasets.

###  IV-C Experimental results and discussion

Tab. I illustrates the experimental results of the proposed method compared to
the baseline model and other methods on the Clotho-v1 and Clotho-v2 evaluation
sets333In addition to the metrics, to better verify the statistical
significance of model performance margins, the results of Student’s t-tests
between our methods and the other methods are shown in the Appendix.. The bold
and underline fonts represent the first and second place in each metrics,
respectively.

All baseline methods and our proposed methods are repeated three times, their
results in the table are the average of the three experimental results. And
the baseline model using Pretrain-CNN encoder and attention-based GRU decoder
is our implementation of the state-of-the-art AT-CNN method [11, 15]. The
results of other methods shown in the Tab. I are provided in their paper [10,
9, 16, 17].

####  IV-C1 Compared with baseline models

For the identical combination of encoder and decoder, our proposed method
achieves better results in all evaluation metrics compared to the baseline
model. And the baseline model has the identical experimental setting as our
proposed method, only without the caption feature space regularization module
(Block B of Fig. 2). The experimental results and Student’s $t$-tests show
that the caption proxy feature space regularization can reduce the effect of
caption disparity and significantly improve model performance.

####  IV-C2 Compared with other methods

Our proposed best model, in which the encoder is Pretrain-CNN and the decoder
is GRU, obtains first or second place in all evaluation criteria in both
datasets. In summary, the proposed method shows outstanding performance on
these two datasets.

##  V Conclusion

In this paper, we propose a two-stage caption feature space regularization
method for the task of audio captioning. In the proposed method, the first
stage uses contrastive learning to generate the proxy feature space and
extract the proxy embedding of audio clips. Then the second stage uses the
extracted proxy embedding to regularize the training of caption generation and
mitigate the effect of the caption disparity. We conducted experiments using
four different combinations of pre-trained encoders and decoders on two
datasets to demonstrate the effectiveness of the proposed method.

## References

  * [1] K. Drossos, S. Adavanne, and T. Virtanen, “Automated audio captioning with recurrent neural networks,” in _2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , 2017, pp. 374–378.
  * [2] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio captioning dataset,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 736–740.
  * [3] Y. Koizumi, R. Masumura, K. Nishida, M. Yasuda, and S. Saito, “A transformer-based audio captioning model with keyword estimation,” _arXiv preprint arXiv:2007.00222_ , 2020.
  * [4] X. Xu, H. Dinkel, M. Wu, and K. Yu, “Audio caption in a car setting with a sentence-level loss,” in _2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP)_ , 2021, pp. 1–5.
  * [5] M. Wu, H. Dinkel, and K. Yu, “Audio caption: Listen and tell,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 830–834.
  * [6] S. Lipping, K. Drossos, and T. Virtanen, “Crowdsourcing a dataset of audio captions,” _arXiv preprint arXiv:1907.09238_ , 2019.
  * [7] R. Rana, “Gated recurrent unit (gru) for emotion classification from noisy speech,” _arXiv preprint arXiv:1612.07778_ , 2016.
  * [8] S. Lipping, K. Drossos, and T. Virtanen, “Crowdsourcing a dataset of audio captions,” in _Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_ , Nov. 2019. [Online]. Available: https://arxiv.org/abs/1907.09238
  * [9] B. Y. Helin Wang, Y. Zou, and D. Chong, “Automated audio captioning with temporal attention.”
  * [10] K. Chen, Y. Wu, Z. Wang, X. Zhang, F. Nian, S. Li, and X. Shao, “Audio captioning based on transformer and pretrained cnn,” in _Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop_ , 2020, pp. 21–25.
  * [11] X. Xu, H. Dinkel, M. Wu, Z. Xie, and K. Yu, “Investigating local and global information for automated audio captioning with transfer learning,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 905–909.
  * [12] C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 119–132.
  * [13] A. Ö. Eren and M. Sert, “Audio captioning based on combined audio and semantic embeddings,” in _2020 IEEE International Symposium on Multimedia (ISM)_ , 2020, pp. 41–48.
  * [14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [15] X. Xu, Z. Xie, M. Wu, and K. Yu, “The sjtu system for dcase2021 challenge task 6: audio captioning based on encoder pre-training and reinforcement learning,” DCASE2021 Challenge, Tech. Rep, Tech. Rep., 2021.
  * [16] C. Narisetty, T. Hayashi, R. Ishizaki, S. Watanabe, and K. Takeda, “Leveraging state-of-the-art asr techniques to audio captioning,” DCASE2021 Challenge, Tech. Rep, Tech. Rep., 2021.
  * [17] X. Liu, Q. Huang, X. Mei, T. Ko, H. L. Tang, M. D. Plumbley, and W. Wang, “Cl4ac: A contrastive loss for audio captioning,” _arXiv preprint arXiv:2107.09990_ , 2021.
  * [18] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [19] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [21] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _arXiv preprint arXiv:1904.08779_ , 2019.
  * [22] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [23] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [24] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [25] C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * [26] S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 2005, pp. 65–72.
  * [27] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4566–4575.
  * [28] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in _European conference on computer vision_. Springer, 2016, pp. 382–398.
  * [29] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 873–881.


#  LLMs’ morphological analyses of complex FST-generated Finnish words

Anssi Moisio1, Mathias Creutz2 Mikko Kurimo1
1Department of Information and Communications Engineering, Aalto University,
Finland
2Department of Digital Humanities, University of Helsinki, Finland
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Rule-based language processing systems have been overshadowed by neural
systems in terms of utility, but it remains unclear whether neural NLP
systems, in practice, learn the grammar rules that humans use. This work aims
to shed light on the issue by evaluating state-of-the-art LLMs in a task of
morphological analysis of complex Finnish noun forms. We generate the forms
using an FST tool, and they are unlikely to have occurred in the training sets
of the LLMs, therefore requiring morphological generalisation capacity. We
find that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo
struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.

LLMs’ morphological analyses of complex FST-generated Finnish words

Anssi Moisio1, Mathias Creutz2, and Mikko Kurimo1 1Department of Information
and Communications Engineering, Aalto University, Finland 2Department of
Digital Humanities, University of Helsinki, Finland<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Do neural networks learn grammar?

The debate on whether neural networks (NNs) can be accurate models of human
language often revolves around the question whether NNs learn similar grammar
rules as children do. In a famous instance of the debate, Rumelhart and
McClelland (1986) argued that a NN can capture the implicit rules that govern
how English verbs are inflected in the past tense. In a response, Pinker and
Prince (1988) counter that explicit rules are indispensable to explain how
children learn past tenses, and more generally to explain the psychology of
language.

Neural methods have gradually become more capable of modelling varied aspects
of language, which could be viewed as supporting the implicit rules argument.
(For updates on the past-tense debate see Kirov and Cotterell (2018); Corkery
et al. (2019); Fukatsu et al. (2024).) The most recent instances of the debate
are over large language models (LLMs), whose language-generation and task-
solving capabilities have surprised many. The recent debate consequently
concerns modelling human language more generally instead of focusing on
specific phenomena such as verb inflection. Considering the success of LLMs,
it is clear that they learn some implicit rule-abiding behaviour that enables
them to process and generate language competently, but it is still not clear
if they learn grammar similarly to humans, or if they learn and employ some
other set of rules.

Assessing grammatical knowledge learned by NNs is not straightforward, but
there are at least two popular approaches. Training a classifier (called a
‘probe’ (Alain and Bengio, 2016) or a ‘diagnostic classifier’ (Hupkes et al.,
2018), first developed by Shi et al. (2016); Adi et al. (2017)) to classify
the internal representations of NNs has been used to inspect what aspects of
grammar are encoded in them. Probing studies have found various syntactical
information encoded in neural NLP systems (Jawahar et al., 2019; Tenney et
al., 2018; Papadimitriou et al., 2021), but interpreting the results remains
contentious (Voita and Titov, 2020; Immer et al., 2022).

The other popular method is to directly inspect a neural LM’s next-unit
predictions, or to train a classifier NN to predict which word is most
acceptable, given sequence of previous words. In an influential work by Linzen
et al. (2016), knowledge of subject-verb agreement in LSTM networks was
assessed this way, and it was concluded that ‘LSTMs can learn to approximate
structure-sensitive dependencies fairly well’. Similar _targeted syntactic
evaluation_ methods, inspired by methods in psycholinguistics (e.g. Crain and
Fodor (1985); Stowe (1986)), have subsequently been employed to assess the
knowledge of many different grammatical phenomena in NNs, for example anaphora
or negative polarity items (Marvin and Linzen, 2018; Futrell et al., 2019;
Jumelet and Hupkes, 2018; Hu et al., 2020). Larger test suites such as BLiMP
(Warstadt et al., 2020) or SyntaxGym (Gauthier et al., 2020) are used as
benchmarks to track advances in the field.

The general conclusion has not changed much since that of Linzen et al.’s: the
networks are _fairly_ good at acquiring the grammar rules. Sometimes results
of a single study are interpreted as evidence that the NNs have acquired a
syntactical rule completely (e.g. Wilcox et al. (2023)), but a closer
inspection often proves such an interpretation premature (e.g. Lan et al.
(2024)). Since there is no conclusive evidence that NNs learn from text the
same grammar that people use, it remains an important task to delineate the
instances where NNs, and LLMs in particular, adhere to and utilise grammar,
and the instances where they do not.

Designing targeted syntactic evaluation tests requires careful formulation of
the sequences. For example, Wilcox et al. (2023) examined the understanding of
filler-gap effects by comparing the probabilities of acceptable and
unacceptable continuations for sentence pairs such as ‘I know _what_ the lion
devoured’ and ‘I know _that_ the lion devoured’. The continuation ‘yesterday’
is assumed to be acceptable for the former but not the latter sequence.
However, ‘yesterday’ could be an acceptable next word even for the latter
sequence: consider the sentence ‘I know that the lion devoured yesterday’s
leftovers.’ This example highlights the difficulty of designing test sentences
of this sort.

Instead of inspecting the next-unit predictions or training diagnostic
classifiers, in this work we ask LLMs explicitly to perform a classification
task, which is possible due to the flexible text generation capacity of the
LLMs. This makes the evaluation relatively unambiguous. For example, asking an
LLM directly ‘Is the verb “devour” transitive or intransitive?’ does not leave
much room for confounding factors. The apparent limitation of this method is
that even if a model fails in an explicit classification task like this, we
cannot rule out the possibility that the model nevertheless encodes perfect
_implicit_ knowledge of the verb and how to use it in any context. However, we
make the assumption in this work that if the LLMs had learned a grammar rule
as perfectly as humans, they would be able to answer the explicit questions as
competently as humans. This seems justified considering the type and
difficulty of, and LLMs’ performance in, other tasks used to evaluate LLMs,
such as academic and professional exams (OpenAI, 2023).

This approach was also taken by Weissweiler et al. (2023), who assessed the
morphological competence of GPT-3.5-turbo by asking it directly to fill in
past tenses of words in a sentence, and concluded that it ‘massively
underperforms purpose-built systems’. Similarly, Weller-Di Marco and Fraser
(2024) took a morphologically complex word $W$ and asked GPT-3.5-turbo
questions such as ‘What is the head noun of $W$?’.

In this work we present LLMs directly and explicitly with a classification
task to investigate the knowledge of Finnish morphology in LLMs. Although
Finnish has relatively few speakers worldwide (<10 million), it is not a low-
resource language, having about 32B tokens of available training texts
(Luukkonen et al., 2023, 2024). Consequently, the state-of-the-art (SOTA)
multilingual LLMs such as GPT-4 are fluent in Finnish, and could be expected
to have a good grasp of the grammar, if the LLMs are in fact good at learning
grammar from text.

##  2 Data and methods

Previous datasets of inflected Finnish words include the MorphyNet (Batsuren
et al., 2021) and UniMorph (Kirov et al., 2016; Batsuren et al., 2022)
corpora. We chose not to use data from these datasets for two reasons.
Firstly, complex words comprising unusually many morphemes make it possible to
assess if the systems can generalise to many types of possible inflections
instead of learning only the most common inflection types. The previous
datasets do not include many extremely complex word forms, but these can be
generated using a finite-state transducer (FST). Secondly, since the SOTA LLMs
have been trained on very large datasets harvested from the Internet, it is
likely that the previously published datasets are included in their training
data, which would preclude fair assessment.

We use the Omorfi tools Pirinen (2015); Pirinen et al. (2017) that are based
on finite-state morphology (Koskenniemi, 1984; Beesley and Karttunen, 2003) to
generate inflected forms of Finnish nouns. The Omorfi library includes some
500k lexemes, of which about 140k are nouns. We inflect the nouns in all
possible combinations of number, grammatical case, and possessive suffix (see
Table 1 for examples, and Appendix A for further details),

BASE | +PL | +INE | +SG2 / +PL1
---|---|---|---
laite | laitteet | _laitteissa_ |  laitteissasi / laitteissamme
+TRA |
laitteiksi |  laitteiksesi / laitteiksemme
Table 1: Examples of inflections of the word ‘laite’ (‘device’). PL means
plural, INE and TRA are case classes, and SG2/PL1 are possessive suffixes.
Inflections in each column include also those in the columns to their left.

which creates about 25M word forms. A random sample of 2000 inflected nouns is
used as a test set in our experiments. We are unaware of any assessment of the
generation accuracy of Omorfi, so we performed manual evaluation of the first
200 words in the sample and found 6 incorrectly inflected words. We therefore
estimate the generation accuracy to be around 97%, which creates an upper
bound for the classification accuracy of the test set. We publish the test set
and all code to reproduce the results at https://github.com/aalto-speech/llm-
morph-tests. We note, however, that once the data is published, it is subject
to the same data contamination issue as the previous datasets mentioned
above—the good thing is that one can always draw a new random sample from the
full set of 25M forms.

Uniform sampling of lexemes creates a bias towards low-frequency types that
are correlated with regularity of the inflection (Kodner et al., 2023). We
note that this is the case in our data, as we took a random sample of the
lexemes, and this should be kept in mind when interpreting the results; there
are probably not many irregularly inflected words, which makes the task
easier. This is not an issue, however, given our research question of whether
the LLMs have picked up even the most _systematic_ inflection types from
textual data.

Prompt:
---
Jäsennä taivutetut substantiivit tällä tavalla:
taivutusmuoto – perusmuoto, luku, sijamuoto, omistusliite
vedessämme – vesi, yksikkö, inessiivi, 1. persoonan monikko
kinoksiksensa – kinos, monikko, translatiivi, 3. persoona
peukalostanne – peukalo, yksikkö, elatiivi, 2. persoonan monikko
huurteenani – huurre, yksikkö, essiivi, 1. persoonan yksikkö
sängiltäsi – sänki, monikko, ablatiivi, 2. persoonan yksikkö
koivuumme – koivu, yksikkö, illatiivi, 1. persoonan monikko
kaistojaan – kaista, monikko, partitiivi, 3. persoona
rehtiyksiesi – rehtiys, monikko, genetiivi, 2. persoonan yksikkö
laaksoillani – laakso, monikko, adessiivi, 1. persoonan yksikkö
talollenne – talo, yksikkö, allatiivi, 2. persoonan monikko
kansoiltanne – kansa,
Correct answer:
monikko, ablatiivi, 2. persoonan monikko
Table 2: An example 10-shot prompt. An English translation of the first two
rows is: _Parse the inflected nouns in this manner: inflected form – base
form, number, grammatical case, possessive suffix_. The following rows are the
examples. We use $n$-shot prompts with $n\in\\{0,1,5,10\\}$, and for all $n$
we use the same $n$ first examples. For instance, the 5-shot prompts have the
_vedessämme, kinoksiksensa, peukalostanne, huurteenani_ , and _sängiltäsi_
example rows.

LLMs are prompted to give a morphological analysis given an inflected form and
the base form. That is, the models should give the correct number, case, and
possessive suffix classes of the inflected noun. The prompt, shown in Table 2,
comprises a short description of the task and the desired format, after which
there are 0, 1, 5, or 10 examples of the task before the test word.

We test GPT-4-turbo-1106-preview (Achiam et al., 2023) (which outperformed
GPT-4-0613 in preliminary experiments), GPT-3.5-turbo-1106, Llama2-70B
(Touvron et al., 2023) (outperformed smaller Llama2 models and chat versions),
and Poro-34B (Luukkonen et al., 2024), which is trained on Finnish, English,
and programming code.

For Poro and Llama2, we performed a coarse tuning of the temperature parameter
on a validation set, and found no large differences but 0.5 to be marginally
better than the others, so we used this value in the experiments with these
models. For the GPT models we found a temperature of 0.0 to yield the best
results, so this value is used for GPT-4-turbo and GPT-3.5-turbo. We did not
tune the top_p parameter (of _nucleus sampling_) but used the default value
1.0.

Additionally, we trained simple recurrent neural network (RNN) models to also
classify words (one RNN for each category: number, case, and possessive
suffix), using random samples of the FST-generated word forms as training data
(excluding the test set). The aim of this comparison is to give some
indication of the difficulty of the task, and to see if NNs can handle the
task if they are specifically trained on this small subset of Finnish
morphology. We took the RNN off the shelf of the Pytorch library111 From the
tutorial at
https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial
without tuning any of its hyperparameters. It consists of three layers of size
128.

##  3 Results

Figure 1: Results in the morphological analysis task.

The rightmost plot in Figure 1 shows that besides GPT-4-turbo, the models
perform poorly in the task. GPT-4-turbo is not close to perfect accuracy
either, and the combined 10-shot result does not reach the result achieved by
simple RNNs trained with 80k words. With training set sizes of 800, 4k, 8k,
40k, and 80k words, the RNNs achieved accuracies of 0.380, 0.765, 0.774,
0.821, and 0.840, respectively.

The first three plots from left in Figure 1 break down the classification task
into the three component classification tasks: number, case, and possessive
suffix. There are some differences in the strengths of the models: Llama
outperforms GPT-3.5 in the possessive suffix classification task, while
GPT-3.5 performs better for other classification tasks. In number
classification, Poro outperforms Llama, although Llama performs better in
other tasks.

Figure 2 shows the confusion matrices for GPT-4-turbo classifications of cases
for the 0-shot and 10-shot setups. From the 0-shot confusion matrix we can see
that the model does predict all classes even though we did not provide it with
the names of the classes we expected it to recognise. This is not surprising,
since GPT-4-turbo has no difficulties if asked to inflect a Finnish word in
all cases and to provide the names of the cases. It is obvious that
GPT-4-turbo has a fair amount of both declarative knowledge (metalinguistic
knowledge; it knows the classes) and procedural knowledge (knows how to
inflect the words) of the Finnish morphology. Therefore, the challenge in this
task comes presumably from the need to generalise to infrequently used,
morphologically complex word forms.

Figure 2: Case label confusions of GPT-4-turbo in the 0-shot and 10-shot
setups. See Appendix B for all confusion matrices. Figure 3: Possessive suffix
label confusions of GPT-4-turbo in the 0-shot and 10-shot setups. See Appendix
B for all confusion matrices.

##  4 Discussion

###  4.1 Reasons behind the errors

Most current SOTA LLMs use subword tokenisation methods such as BPE (Sennrich
et al., 2016) that break down infrequent character sequences into multiple
shorter tokens while keeping frequent sequences as single tokens. Intuitively,
having long tokens that combine multiple morphemes into a single token could
hinder the capacity to model morphology, since multiple embeddings would have
to be learned for a single morpheme. Of the three model families, Poro uses
the longest tokens, having an average of 3.55 characters per token in our test
words, while the Llama average is 2.16 and the GPT average is 2.26.
Furthermore, the average length of the last token of a word is even longer:
4.42 for Poro, 2.41 for Llama, and 2.78 for GPT. For example, the first two
test words whose possessive suffix Poro classifies incorrectly and differences
in the tokenisations of the different models are shown in Table 3.

Base form |  _lyhty_ (_lantern_) |  _tarttuma_ (_infection_)
---|---|---
Test word | _lyhtyjämme_ | _tarttumassamme_
Poro tokens | ly hty jämme | t art t um assamme
Llama tokens | ly ht yj äm me | tart t um ass am me
GPT tokens | ly ht y j äm me | t art t um ass am me
Table 3: BPE tokenisations of different models.

Both of these words have the first person plural possessive suffix, which
always ends in ‘me’. The possessive suffix ‘me’ is combined with the case
morpheme (partitive ‘jä’ in ‘lyhtyjämme’ and inessive ‘ssa’ in
‘tarttumassamme’) by Poro but not by GPT or Llama. This might be one reason
Poro misclassifies these words, while GPT and Llama do not, and in general why
Poro lags behind the other models in the possessive suffix classification task
as seen in Figure 1. The possessive suffix is simple to recognise, _if_ the
tokenisation is conducive to the task: a rule that checks the last two letters
of the word and assigns ‘ni’-->SG1; ‘si’-->SG2; ‘me’-->PL1; ‘ne’-->PL2;
else-->3 would achieve 100% accuracy on our test set. Admittedly, the rule
would have to be more complicated if there were also words without any
possessive suffix, since these words could end in virtually any two letters:
for instance, ‘vesi’ (‘water’) ends in ‘si’ but does not have any possessive
suffix (SG2 form would be ‘vetesi’) as does the translative case ‘vedeksi’
without a possessive suffix (the translative case with SG2 suffix becomes
‘vedeksesi’).

Class frequencies could also explain some of the confusions. For example,
GPT-4 often confuses abessive cases as partitive, seen in Figure 2. In
addition to partitive being often quite similar to abessive, for example the
inflected forms ‘kättä’ and ‘kädettä’ of the base ‘käsi’ (‘hand’), partitive
is also much more common than abessive: 16.2% versus 0.1% of occurrences in
Kettunen (2005).

###  4.2 Interpretations and implications

The results suggest that despite the versatile language generation capacity of
GPT-4-turbo it has not acquired the rules of Finnish morphology as completely
as could be expected based on its language generation capacity. Instead, GPT-4
employs some other set of heuristics to decide the next token, although these
undoubtedly overlap somewhat with grammar rules. This is hardly a surprise
given the literature reviewed in Section 1, where the general conclusion tends
to be that NNs rarely use grammar rules systematically, although usually
fairly well.

The ineptitude of neural nets to follow grammar rules is related to systematic
compositionality and inefficiency w.r.t training data set size, which are said
to be weaknesses of neural nets compared to rule-based systems. Learning
grammar enables _systematic compositional generalisation_ (Fodor and Pylyshyn,
1988): learning a concise grammar rule such as ‘the suffix -nne indicates 2nd
person plural possessive form’ would enable generalising to all possible 2nd
person plural forms in Finnish, obviating the need to learn word-specific
associations and therefore reducing the required training corpus size. GPT-4
reaches close to 100% accuracy in this simple task of classifying possessive
suffixes (RNN reaches 100%, and it is obvious that Finnish speakers would also
reach 100%). However, the fact that it still sometimes classifies words ending
in ‘nne’ as 2nd person singular instead of plural (see Figure 3) betrays its
incomplete grasp of the systematic possessive suffixes in Finnish. Similar
arguments apply to the other two classification tasks and the combined
classification task.

##  5 Conclusion

We conclude that even a SOTA LLM, GPT-4-turbo, does not model Finnish
morphology thoroughly enough to allow it to provide morphological analyses of
rare and complex word forms with a high accuracy. Contrasting this with its
impressive text generation capacity suggests that it utilises some other
language processing heuristics, which clearly overlap somewhat with
morphological rules since it rarely produces incorrect forms, but which
preclude human-level systematic generalisation on our test set. GPT-4-turbo
outperforms models such as GPT-3.5-turbo and Llama2-70B, however, by a large
margin.

##  6 Limitations

Our experiments are limited to only one language and only four LLMs, which of
course means we cannot be certain how the models perform on different
languages, or how other models perform in Finnish, even though we suggest our
results shed some light on general questions of grammar represented in LLMs.
We also have not optimised the prompt beyond trying out a few different
phrasings, so we assume some other prompt could elicit better performance
especially in the 0- and 1-shot setups.

As noted in the introduction, we assess LLMs using explicit, metalinguistic
questions about Finnish morphology. It is in principle possible that even if
the models fail in this task, having a limited grasp of the morphological
labels, they could succeed in using the words correctly in sentences and
representing their meanings correctly.

##  7 Acknowledgements

We thank the anonymous reviewers for their insightful comments and feedback.
The work was supported by the Finnish Cultural Foundation grant 00240853 and
the Academy of Finland grant 337073. The computational resources were provided
by Aalto ScienceIT. The use of the GPT-3.5 and GPT-4 systems via the Azure
OpenAI API was provided by Aalto IT Services.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Adi et al. (2017) Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017.  Fine-grained analysis of sentence embeddings using auxiliary prediction tasks.  In _International Conference on Learning Representations_.
  * Alain and Bengio (2016) Guillaume Alain and Yoshua Bengio. 2016.  Understanding intermediate layers using linear classifier probes.  _arXiv preprint arXiv:1610.01644_.
  * Batsuren et al. (2021) Khuyagbaatar Batsuren, Gábor Bella, and Fausto Giunchiglia. 2021.  MorphyNet: a large multilingual database of derivational and inflectional morphology.  In _Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 39–48, Online. Association for Computational Linguistics.
  * Batsuren et al. (2022) Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Ryan Cotterell, Reut Tsarfaty, Ekaterina Vylomova, et al. 2022.  UniMorph 4.0: Universal Morphology.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 840–855, Marseille, France. European Language Resources Association.
  * Beesley and Karttunen (2003) Kenneth R Beesley and Lauri Karttunen. 2003.  Finite-state morphology: Xerox tools and techniques.  _CSLI, Stanford_ , pages 359–375.
  * Corkery et al. (2019) Maria Corkery, Yevgen Matusevych, and Sharon Goldwater. 2019.  Are we there yet? encoder-decoder neural networks as cognitive models of English past tense inflection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3868–3877, Florence, Italy. Association for Computational Linguistics.
  * Crain and Fodor (1985) Stephen Crain and Janet Dean Fodor. 1985.  How can grammars help parsers? In _Natural Language Parsing Psychological, Computational, and Theoretical Perspectives_ , pages 94–128. Cambridge University Press.
  * Fodor and Pylyshyn (1988) Jerry A Fodor and Zenon W Pylyshyn. 1988.  Connectionism and cognitive architecture: A critical analysis.  _Cognition_ , 28(1-2):3–71.
  * Fukatsu et al. (2024) Akiyo Fukatsu, Yuto Harada, and Yohei Oseki. 2024.  Learning bidirectional morphological inflection like humans.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 10249–10262, Torino, Italy. ELRA and ICCL.
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gauthier et al. (2020) Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020.  SyntaxGym: An online platform for targeted evaluation of language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 70–76, Online. Association for Computational Linguistics.
  * Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020.  A systematic assessment of syntactic generalization in neural language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1725–1744, Online. Association for Computational Linguistics.
  * Hupkes et al. (2018) Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018.  Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure.  _Journal of Artificial Intelligence Research_ , 61:907–926.
  * Immer et al. (2022) Alexander Immer, Lucas Torroba Hennigen, Vincent Fortuin, and Ryan Cotterell. 2022.  Probing as quantifying inductive bias.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1839–1851, Dublin, Ireland. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Jumelet and Hupkes (2018) Jaap Jumelet and Dieuwke Hupkes. 2018.  Do language models understand anything? on the ability of LSTMs to understand negative polarity items.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 222–231, Brussels, Belgium. Association for Computational Linguistics.
  * Kettunen (2005) Kimmo Kettunen. 2005.  Sijamuodot haussa-tarvitseeko kaikkea hakutermien morfologista vaihtelua kattaa?  Master’s thesis.
  * Kirov and Cotterell (2018) Christo Kirov and Ryan Cotterell. 2018.  Recurrent neural networks in linguistic theory: Revisiting pinker and prince (1988) and the past tense debate.  _Transactions of the Association for Computational Linguistics_ , 6:651–665.
  * Kirov et al. (2016) Christo Kirov, John Sylak-Glassman, Roger Que, and David Yarowsky. 2016.  Very-large scale parsing and normalization of Wiktionary morphological paradigms.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3121–3126, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Kodner et al. (2023) Jordan Kodner, Sarah Payne, Salam Khalifa, and Zoey Liu. 2023.  Morphological inflection: A reality check.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6082–6101, Toronto, Canada. Association for Computational Linguistics.
  * Koskenniemi (1984) Kimmo Koskenniemi. 1984.  A general computational model for word-form recognition and production.  In _10th International Conference on Computational Linguistics and 22nd Annual Meeting of the Association for Computational Linguistics_ , pages 178–181.
  * Lan et al. (2024) Nur Lan, Emmanuel Chemla, and Roni Katzir. 2024.  Large language models and the argument from the poverty of the stimulus.  _Linguistic Inquiry_ , pages 1–56.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Luukkonen et al. (2024) Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman, Ville Komulainen, Väinö Hatanpää, Peter Sarlin, and Sampo Pyysalo. 2024.  Poro 34b and the blessing of multilinguality.  _arXiv preprint arXiv:2404.01856_.
  * Luukkonen et al. (2023) Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. 2023.  FinGPT: Large generative models for a small language.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2710–2726, Singapore. Association for Computational Linguistics.
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _arxiv 2303.08774_.
  * Papadimitriou et al. (2021) Isabel Papadimitriou, Ethan A. Chi, Richard Futrell, and Kyle Mahowald. 2021.  Deep subjecthood: Higher-order grammatical features in multilingual BERT.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2522–2532, Online. Association for Computational Linguistics.
  * Pinker and Prince (1988) Steven Pinker and Alan Prince. 1988.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition.  _Cognition_ , 28(1-2):73–193.
  * Pirinen (2015) Tommi A Pirinen. 2015.  Development and use of computational morphology of finnish in the open source and open science era: Notes on experiences with omorfi development.  _SKY Journal of Linguistics_ , 28:381–393.
  * Pirinen et al. (2017) Tommi A Pirinen, Inari Listenmaa, Ryan Johnson, Francis M. Tyers, and Juha Kuokkala. 2017.  Open morphology of finnish.  LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University.
  * Rumelhart and McClelland (1986) David E Rumelhart and James L McClelland. 1986.  On learning the past tenses of English verbs.  In _Parallel distributed processing: explorations in the microstructure of cognition, vol. 2: psychological and biological models_ , pages 216–271. MIT Press.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Shi et al. (2016) Xing Shi, Inkit Padhi, and Kevin Knight. 2016.  Does string-based neural MT learn source syntax? In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1526–1534, Austin, Texas. Association for Computational Linguistics.
  * Stowe (1986) Laurie A Stowe. 1986.  Parsing wh-constructions: Evidence for on-line gap location.  _Language and cognitive processes_ , 1(3):227–245.
  * Tenney et al. (2018) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2018.  What do you learn from context? probing for sentence structure in contextualized word representations.  In _International Conference on Learning Representations_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Voita and Titov (2020) Elena Voita and Ivan Titov. 2020.  Information-theoretic probing with minimum description length.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 183–196, Online. Association for Computational Linguistics.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020.  BLiMP: A benchmark of linguistic minimal pairs for English.  In _Proceedings of the Society for Computation in Linguistics 2020_ , pages 409–410, New York, New York. Association for Computational Linguistics.
  * Weissweiler et al. (2023) Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, and David Mortensen. 2023.  Counting the bugs in ChatGPT’s wugs: A multilingual investigation into the morphological capabilities of a large language model.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6508–6524, Singapore. Association for Computational Linguistics.
  * Weller-Di Marco and Fraser (2024) Marion Weller-Di Marco and Alexander Fraser. 2024.  Analyzing the understanding of morphologically complex words in large language models.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 1009–1020, Torino, Italy. ELRA and ICCL.
  * Wilcox et al. (2023) Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2023.  Using computational models to test syntactic learnability.  _Linguistic Inquiry_ , pages 1–44.

##  Appendix A Details of the classification task

We inflect Finnish nouns in all possible combinations of number, grammatical
case, and possessive suffix. Tables 4 and 5 list the classes of case and
possessive suffix with examples of both singular and plural forms. We include
a possessive suffix in all the forms in our test set.

Short | Name | SG e.g. | PL e.g.
---|---|---|---
ABE | abessive | talotta | taloitta
ABL | ablative | talolta | taloilta
ADE | adessive | talolla | taloilla
ALL | allative | talolle | taloille
ELA | elative | talosta | taloista
ESS | essive | talona | taloina
GEN | genitive | talon | talojen
ILL | illative | taloon | taloihin
INE | inessive | talossa | taloissa
NOM | nominative | talo | talot
PAR | partitive | taloa | taloja
TRA | translative | taloksi | taloiksi
Table 4: Finnish grammatical cases used in the experiments, with example inflections of the word ‘talo’ (‘house’). There are three more grammatical cases in Finnish (totalling 15), but comitative and instructive are not supported by Omorfi, and accusative does not have its own unambiguous surface form, so these three are not included in our data. Class | SG e.g. (ELA) | PL e.g. (ELA)
---|---|---
- | talosta | taloista
SG1 | talostani | taloistani
SG2 | talostasi | taloistasi
PL1 | talostamme | taloistamme
PL2 | talostanne | taloistanne
3 | talostaan, | taloistaan,
talostansa | taloistansa
Table 5: Possessive suffixes in Finnish, with example inflections of the word
‘talo’ (‘house’) with the elative grammatical case ‘talosta’. SG1 is ‘first
person singular’, SG2 is ‘second person singular’ etc. The third person has
the same forms in singular and plural, but there are synonyms such as
‘talostaan’ and ‘talostansa’.

##  Appendix B Detailed results

Figures 4 through 13 show the confusion matrices of all models and in all
classification tasks. Not all rows sum up to exactly to the same number: for
example, in Figure 5 1-shot matrices, the SG row for Llama2 adds up to 964,
whereas for Poro it adds up to 962. This is because of ambiguity in the task:
for example the form ‘taloni’ could be singular or plural (if the case is
nominative). If the a system gives one of the correct classes, the ‘true
label’ is also assigned to that class in these confusion matrices. If the
system predicts incorrectly, the ‘true label’ could be any of the correct
classes (whichever happens to be listed last in our data).

One notable thing in the confusion matrices is that Llama2-70B does not give
many nonsense answers: when one or more examples are given in the prompt, the
Llama2-70B almost always gives class names, correct or incorrect, which are
actual classes, leaving the ‘other’ column empty in Figures 5, 8, and 12. One
reason that this is not the case for the GPT models is probably that
GPT-4-turbo and GPT-3.5-turbo have been tuned for chat. In Microsoft Azure
docs it is stated that ‘Like GPT-3.5 Turbo, and older GPT-4 models, GPT-4
Turbo is optimized for chat and works well for traditional completions
tasks.’222https://learn.microsoft.com/en-us/azure/ai-
services/openai/concepts/models. GPT-4-turbo therefore often asks for
clarification if it doesn’t recognise the word, leading to nonsense
classifications. Poro, on the other hand, is not tuned for chat, but still
gives a lot of ‘other’ answers. This seems to be more about Poro not grasping
the format that the answer should be given in, or simply not knowing which
classes are possible answers.

Figure 4: Confusions in the GPT-4-turbo and GPT-3.5-turbo number
classification task.

Figure 5: Confusions in the Llama2-70B and Poro-34B number classification
task. Figure 6: Confusions of GPT-4-turbo in the case classification task.
Figure 7: Confusions of GPT-3.5-turbo in the case classification task. Figure
8: Confusions of Llama2-70B in the case classification task. Figure 9:
Confusions of Poro-34B in the case classification task. Figure 10: Confusions
of GPT-4-turbo in the possessive suffix classification task. Figure 11:
Confusions of GPT-3.5-turbo in the possessive suffix classification task.
Figure 12: Confusions of Llama2-70B in the possessive suffix classification
task. Figure 13: Confusions of Poro-34B in the possessive suffix
classification task.