# Impact of the Availability of Chat-GPT on Software Development Activity

Alexander Quispe The World Bank Rodrigo Grijalba Pontificia Universidad
Católica del Perú

###### Abstract

Advancements in Artificial Intelligence, particularly with ChatGPT, have
significantly impacted software development. Using novel data from GitHub
Innovation Graph, we hypothesize that ChatGPT improves software production
efficiency. Utilizing natural experiments where some governments banned
ChatGPT, we employ Difference-in-Differences (DID), Synthetic Control (SC),
and Synthetic Difference-in-Differences (SDID) methods to estimate its
effects. Our findings indicate a significant positive impact on the number of
git pushes, repositories, and unique developers per 100,000 people,
particularly for high-level, general-purpose, and shell scripting languages.
These results suggest that AI tools like ChatGPT can substantially boost
developer productivity, although further analysis is needed to address
potential downsides such as low-quality code and privacy concerns.

First version: March 1, 2024

This version:

JEL Codes: O33, L86, D83, C23, J24

##  1 Introduction

This past decade, a significant proportion of technological progress in
general has been marked by accelerating progress in Artificial Intelligence.
As reported by the World Intellectual Property Organization (2022)
Organization (2022), the growth rate of Artificial Intelligence patents has
been eight times that of all patents, signaling this field’s significantly
increased interest and potential. Language Processing and Generation is an
important sub-field that creates algorithms capable of analyzing, predicting,
and generating text by training Machine Learning models with human language.
Many approaches have been used for this end. Still, the most capable models
currently use the transformer architecture, which can train and be deployed
quickly by focusing on relationships between words in natural language.

One of the most important examples of Large Language Models (LLMs) developed
through this approach has been the GPT family of models. The developing
company, OpenAI, has the stated goal of advancing AI towards Artificial
General Intelligence (AGI) in favor of furthering socially beneficial
technological development (OpenAI, 2018) OpenAI (2018). As part of this goal,
they have developed one of the most advanced and capable family of models for
language processing, which is also used for text generation in their chatbot
application, ChatGPT. Thanks to the flexibility of the model and the fact that
it has been trained in human language from a large database containing diverse
topics, including programming questions, the chatbot has been used to aid in
software development.

Given the popularity of the application and its capabilities for software
development, it would be expected that the process of creating software would
be made noticeably more efficient, which would be reflected in an increase in
the production of software, version, patches, etc. after the application has
been made available. Such a measurement would add to the emerging literature
on using ChatGPT to aid in software development. To make this measurement, we
can exploit the fact that, initially, ChatGPT was made available only in a
subset of countries, which can be considered a treatment group; the remaining
groups can serve as a control group. With the availability of data on software
development at the country level, we have the necessary pieces to analyze the
availability of ChatGPT as a treatment in a natural experiment. We use the
Difference in Differences, Synthetic Control, and Synthetic Difference in
Differences approaches to analyzing quasi-experimental data to estimate
ChatGPT’s effect on software production in each country-quarter. We find a
significant and essential change in the number of pushes per capita in
countries with ChatGPT availability compared to those without it; this
suggests that the speed of software production is increased by ChatGPT’s
ability to generate suggestions for programming algorithms, designing software
architecture, and repairing bugs, among other uses in software development.

The rest of this article is organized as follows. Section 2 reviews the
literature on the potential, current usability, and measured impact of ChatGPT
as a tool for aiding in the software development process. Section 3 first
describes the country-level data on software development activity that will be
used, namely, GitHub’s Innovation Graph. Then, it explains the methods through
which the data will be used to estimate the effect ChatGPT has had, on
average, on each country’s production of software: Difference in Differences,
Synthetic Control, and Synthetic Difference in Differences. Section 4 shows
the results of each method on the variables we construct using the available
data. Finally, section 5 discusses these findings in the current context of
rapid progress in AI.

##  2 Literature review

The use of machine learning methods, specifically neural networks, to aid
productivity in software development is a very rare practice. The use of
ChatGPT for these purposes is even more recent, with its launch in November
2022 OpenAI, 2022 OpenAI (2022). As is expected for very recent phenomena, the
literature on the impact of ChatGPT on software development productivity is
very scarce. Nonetheless, the undeniable disruptiveness of the application has
awakened great interest in its study, which has offset some of the scarcity in
the literature. Still, research on the potential, capabilities, and impact of
ChatGPT on software development is still in its infancy, and the constantly
evolving nature of the topic requires quick adaptations of research to update
the understanding of present and future questions adequately. The relevance of
this last point is illustrated by the fact that the vast majority of the
literature on this topic pertains to the GPT-3.5 model, even though GPT-4 was
launched in March 2023.

###  2.1 Potential of ChatGPT on Software Development

Abu Jaber, Beganovic, and Abd Almisreb (2023) Abu Jaber et al. (2023) conduct
a literature review on the potential of ChatGPT in software development. They
outline potential uses for troubleshooting and bug repair and educational
applications for these purposes. This includes optimizing programs and
numerical algorithms, as well as the potential incorporation of ChatGPT as an
integrated controller in Generalized Intelligence. The authors discuss the
creation of software solutions and architectures, both comparatively
(evaluating multiple responses) and through a dialogical process with the
application for software architectural design. They also delve into
considerations for Prompt Engineering when employing the program for software
development.

Rahmaniar (2023) Rahmaniar (2023) comes to similar conclusions. The author
surveys the potential for productivity gains in software development through
assistance in programming, documentation creation, training and onboarding,
code review, and interactions with clients and stakeholders. All this thanks
to the model’s ability to process natural language, generative capacity,
flexibility and potential for model learning, dialogical interactivity, the
large spectrum of applicability for its use, and contributions to the Open
Source space. The author also discusses the negative ramifications of such
uses, such as possible incompleteness of generated code, which may cause
difficulties for users and potential security vulnerabilities. In this regard,
there is also the potential for malicious use of the model to create malware.
The author calls for considering ethical considerations; for example, during
the learning process, models may be exposed to content that integrates and
perpetuates biases with negative consequences. With future improvements in
language models, especially with the potential of GPT models, response
generation would have better performance, translating into greater ease and
productivity gains for developers using LLM tools. Future versions could be
used to streamline the development process, where generation is constant to
optimize and repair code in real-time.

###  2.2 Tests on ChatGPT’s capabilities:

One of the potentials of ChatGPT in software development lies in its
usefulness in education. Jalil et al. (2023)Jalil et al. (2023) test ChatGPT
with solved problems from Ammann and Offutt (2016), a widely used introductory
textbook on software testing. They find that the model can respond correctly
to about 43% of software testing questions and provides a sufficient
explanation for its response in most cases.

Another possible avenue for innovation and productivity gains is using ChatGPT
to optimize algorithms and numerical methods. Badini et al. (2023) Badini et
al. (2023) use ChatGPT for troubleshooting and optimization of GCode programs
for additive manufacturing (3D printing) routines. After 6 iterations of
dialogue with the chatbot with the purpose of training on the low-level GCode
language, they prompt ChatGPT to solve manufacturing problems resulting from
existing routines, and find noticeable improvements in the build quality of
the end products after using ChatGPT’s output.

As for numerical methods, Kashefi and Mukerji (2023) Kashefi and Mukerji
(2023) propose and test the usage of ChatGPT on the generation of code for
several numerical methods related to physics, in several programming
languages. This involves programming the simulation of simple linear systems
as well as more complicated dynamic systems that involves one or more
differencial equations. ChatGPT was able to generate sufficient, programs for
the mathematical system described by the researchers. Some issues arise like
the generation of singular matrices, attempts at opperating on incompatible
arrays, and interruptions of the generation of code when the program is
lengthy.

It is also possible to use ChatGPT as a controller for routing tasks and sub-
tasks to algorithms more suitable for solving them. This is proposed and
tested by Shen et al. (2023) Shen et al. (2023), who use GPT-4 in conjunction
with several specialized models from the HuggingFace platform to solve complex
tasks that involve the solution of several specialized sub-tasks. They develop
a 4 stage architecture called HuggingGPT: first, ChatGPT generates a plan for
solving the complex task through smaller sub-tasks; on the second stage, it
chooses the most capable HuggingFace model to solve each sub-task; in stage 3,
the sub-tasks are routed to their corresponding models and these, in turn,
generate the corresponding output; the fourth and final stage involves parsing
these outputs and presenting them to the user. They test this procedure on
image captioning, image classification, and object detection. They find that
GPT-4 is highly accurate at creating the necessary task plans and classifying
sub-tasks to the corresponding specialized models. They also successfuly test
HuggingGPT on a variety of complex tasks involving text, images, audio and
video.

Ahmad et al. (2023) Ahmad et al. (2023) describe and test human-bot
collaboration for the purpose of formally outlining sofware architecture.
Their testing starts with outlining an architecture story that describes the
conditions that must be satisfied by the software. This outline is fed to the
chatbot as an initial prompt as a preliminary step for the testing. After
this, a novice software architect enters a dialectical process with the
chatbot to analyze, synthesize, and evaluate a potential architectural
solution. They conclude that this dialectica, collaborative process can be
easily carried out to outline a software architecture more efficiently in the
early stages of product development.

Finally, Sobania et al. (2023) Sobania et al. (2023) and Zhang et al. (2023)
Zhang et al. (2023) test the bug repair capabilities of ChatGPT using the
QuickBugs database and a custom database, respectively. Both articles find
overall success using the model when prompting it to fix the bugs in the code
presented, and that more rounds of dialogue increase the success rate. Sobania
et al. (2023) specifically find that ChatGPT is comparable to other LLMs
specialized in this task, and that LLMs outperform a standard, RN-based
program repair application. On the other hand, Zhang et al. (2023) create a
new database of coding problems to test the generalization capabilities of the
model, as QuickBugs may have been used for training GPT-3.5. They again find
that ChatGPT is capable of giving correct fixes in most scenarios.

###  2.3 The Impact of ChatGPT on Software Development Productivity

Gallea (2023) Gallea (2023) compares the Stack Overflow pages for Python and
R, both programming languages used in data science. The reason behind this
comparison is the fact that Python is a much more popular programming language
with a high amount of potential training material for the model, and the
author mentions the use of ChatGPT was not efficient when answer R related
question; this would imply a lower impact of ChatGPT on the R language. The
data comes from Stack Overflow Explorer, specifically information on the
number of questions, the resolution status of the questions, and the score for
each of them. The results from the Differences in Differences analysis suggest
a negative impact on the quantity of questions, a positive impact on the
average score (as a measure of quality) of questions, and a negative impact on
the proportion of resolved questions (as a measure of complexity) for Python
compared to R.

Saguu and Ante (2023) Saggu and Ante (2023) analyze the returns of crypto-
assets related to artificial intelligence. For this purpose, they utilize
daily frequency data of crypto-asset prices from Coingecko and CoinMarketCap.
The treatment group is considered as those assets related to artificial
intelligence, and the start of the post-treatment period is the launch of
ChatGPT. They also control for total market capitalization and transaction
volume for each asset. They employ both the Differences in Differences and
Synthetic Differences in Differences methods. They find that the launch of
ChatGPT had a positive impact on the return of those assets related to
artificial intelligence on both platforms, compared to those unrelated to AI.

Demirci, Hannane and Xinrong (2023) Demicri et al. (2023) analyze the impact
of ChatGPT on the demand for services for freelance workers on a platform
dedicated to facilitating such services. Firstly, they identify service groups
based on the skills required to carry them out. Next, they assign an
artificial intelligence exposure index to each type of service, indicating the
feasibility of using AI to perform services in each group. The analysis is
further adjusted using the Google Search Volume Index. They apply the
Differences in Differences method between two groups: high and low exposure to
artificial intelligence. They find that the impact of ChatGPT was negative on
the quantity of postings seeking services in groups with high exposure to AI,
compared to those with a low exposure index.

Del Rio-Chanona, Laurentsyeva and Wachs (2023) Del Rio-Chanona et al. (2023)
present another example of an analysis of the impact of ChatGPT on question
and answer platforms for programming topics. In this case, the authors compare
Stack Overflow with Math Exchange, a platform focused on questions and answers
about mathematics; the topics on this platform would be less susceptible to
the effects of ChatGPT. They also compare Stack Overflow with its Russian
counterpart; and with Segmentdefault, its Chinese counterpart. This is because
access to ChatGPT is restricted in these two countries, which would decrease
the impact it can have on the software developmnet field. By applying
Differences in Differences, the authors find a negative impact on the number
of posts per week, the number of questions per week, and the number of posts
on weekdays on Stack Overflow compared to the other platforms.

Finally, Kreitmeir and Raschky (2023) Kreitmeir and Raschky (2023) analyze the
impact on productivity in software development using daily user-level data
provided by GitHub. They leverage the ban of ChatGPT in Italy, comparing it to
France and Austria, where the application is freely accessible. As
productivity measures, the authors use the existence of new releases by each
user; the sum of pushes, pull requests, comments on pull requests, comments on
commits, repo creations, and issues; the sum of pushes and pulls; and the
total number of events by each user. They also analyze the quantity of new Tor
users, as users in countries where access is restricted may bypass the ban
through this means, indicating high demand and recognition of the
application’s utility. The result obtained through the Differences in
Differences method indicates a significant negative impact on the probability
that each user-day presents a new software release in Italy compared to other
countries. This suggests lower productivity as a result of the prohibition.

##  3 Methods

###  3.1 Data

The entirety of the data come from GitHub’s Innovation Graph, specifically the
table that contains the number of pushes per country and per quarter. The data
spans form the first quarter of 2020 until the first quarter of 2023. The
information is also separated by jurisdiction, including a total of 179 units.
Of these, we keep 151 after dropping 28 units that lack observations for the
whole time span. We also remove the observations related the EU as an
aggregate, as they cause perfectly collinear with those of the individual EU
countries. Finally, we also exclude Hong Kong from our analysis, because it
shows an atypically high level of pushes compared to all other countries. We
consider the start of the treatment period to be the fourth quarter of 2022,
as the initial launch of the app came out on November 30th of the same year.
The treated units are the countries that were originally announced to have
access to the application, and during the time span included, no new countries
were added to the treatment group. With this, we have a case of block design
as described by Arkhangelsky et al. (2021) Arkhangelsky et al. (2021), which
facilitates the computation of the estimators. This design consists of 120
treatment units and 27 control units; 11 pre-treatment periods and 2 treatment
periods. One result shown by Abadie, Diamond and Hainmueller (2010) Abadie et
al. (2010) for the Synthetic Control estimator – which also holds for the SDiD
estimator – is that the upper bound of the magnitude of the estimation bias
for the Average Treatment Effect grows with the number of control units and
shrinks with the number of pre-treatment periods. We assess that the data
presented should render a relatively unbiased estimator.

###  3.2 Estimation and inference

We may represent the data for each variable as a $N\times T$ matrix, with $N$
units observed for $T$ time periods. We have $N_{0}$ control units and
$N_{1}=N-N_{0}$ treatment units. Analogously, we have $T_{0}$ pre-treatment
periods and $T_{1}=T-T_{0}$ post-treatment periods. The models we use can be
generally described as specific cases of the following equation:

$\widehat{ATE}=\frac{1}{N_{1}}\sum_{i=N_{0}+1}^{N}\left(\frac{1}{T_{1}}\sum_{t=T_{0}+1}^{T}Y_{it}-\frac{1}{T_{0}}\sum_{t=1}^{T_{0}}\lambda_{i}Y_{it}\right)-\sum_{i=1}^{N_{0}}\omega_{i}\left(\frac{1}{T_{1}}\sum_{t=T_{0}+1}^{T}Y_{it}-\sum_{t=1}^{T_{0}}\lambda_{t}Y_{it}\right)$
(1)

Where $Y_{it}$ is the outcome variable for unit $i$ at time period $t$;
$\omega_{i}$ is the factor loading for unit $i$, such that
$\sum_{i=1}^{N_{0}}\omega_{i}=1$; and $\lambda_{t}$ is the factor loading for
time period $t$, such that $\sum_{t=1}^{T_{0}}\lambda_{t}=1$. The first term
in the right side is the difference between the average outcome level before
and after the treatment for the treatment group. The second term of the right
side is this same measure but for the control group. What differentiates each
model is the way the unit and time weights are calculated.

####  3.2.1 Difference in Differences

Neither the units nor the time periods are weighted, which is equivalent to
their weights being defined by:

$\omega_{i}=\frac{1}{N_{0}};\forall i=1,...,N_{0}$
$\lambda_{t}=\frac{1}{T_{0}};\forall t=1,...,T_{0}$

####  3.2.2 Synthetic Control

The vector $(\omega_{1},\ldots,\omega_{N_{0}})$ is defined as:

$(\omega_{1},\ldots,\omega_{N_{0}})=\arg\min_{\omega_{1},\ldots,\omega_{N_{0}}}\sum_{t=1}^{T_{0}}\left(\sum_{i=1}^{N_{0}}\omega_{i}Y_{it}-\frac{1}{N_{1}}\sum_{j=1}^{N_{1}}Y_{jt}\right)^{2}$

Subject to:

$\sum_{i=1}^{N_{0}}\omega_{i}=1;\omega_{i}\geq 0;\forall i=1,...,N_{0}$

On the other hand, SC only takes into account the unweighted treatment period
averages, and not the difference between these and the pre-treatment period.
This means that $\lambda_{t}=0$ for all pre-treatment periods.

####  3.2.3 Synthetic Difference in Differences

In this case, we take the vector $\omega_{0},\omega_{1},...,\omega_{N}$, which
includes factor loadings for the treatment units and $\omega_{0}$, an
intercept term. They are defined by:

$\omega_{i}=\frac{1}{N_{1}};\forall i=N_{0}+1,...,N$

$(\omega_{0},\ldots,\omega_{N_{0}})=\arg\min_{\omega_{0},\ldots,\omega_{N_{0}}}\left(\sum_{t=1}^{T_{0}}\left(\omega_{0}+\sum_{i=1}^{N_{0}}\omega_{i}Y_{it}-\frac{1}{N_{1}}\sum_{i=N_{0}+1}^{N}Y_{it}\right)^{2}+\zeta^{2}T_{0}\left\|(\omega_{0},\ldots,\omega_{N})\right\|_{2}^{2}\right)$

Subject to:

$\sum_{i=1}^{N_{0}}\omega_{i}=1;\omega_{i}\geq 0;\forall i=1,...,N_{0}$

The rightmost term is for regularization, where:

$\zeta=(N_{1}T_{1})^{1/4}\hat{\sigma}$
$\hat{\sigma}^{2}=\frac{1}{N_{0}(T_{0}-1)}\sum_{i=N_{0}}^{N_{0}}\sum_{t=1}^{T_{0}-1}(\Delta_{it}-\overline{\Delta})^{2}$
$\Delta_{it}=Y_{i(t+1)}-Y_{it}$
$\overline{\Delta}=\frac{1}{N_{0}(T_{0}-1)}\sum_{i=1}^{N_{0}}\sum_{t=1}^{T_{0}-1}\Delta_{it}$

Arkhangelsky et al. (2021) explain how this regularization factor is useful to
account for unit-level, time-correlated outcomes. On the other hand across-
unit, time-level correlation is considered to be captured by systematic
variations in the outcome. Therefor, regularization is not included for the
estimation of $\lambda_{i}$. In the case of this model, the vector
$(\lambda_{0},...,\lambda_{T_{0}})$ is defined as follows:

$(\lambda_{0},\ldots,\lambda_{T_{0}})=\arg\min_{\lambda_{0},\ldots,\lambda_{T_{0}}}\sum_{i=1}^{N_{0}}\left(\lambda_{0}+\sum_{t=1}^{T_{0}}\lambda_{t}Y_{it}-\frac{1}{T_{1}}\sum_{t=T_{0}+1}^{T}Y_{it}\right)^{2}$

Subject to:

$\sum_{i=1}^{T_{0}}\lambda_{t}=1;\lambda_{t}\geq 0;\forall t=1,...,N_{0}$

####  3.2.4 Inference

The data we use is unique in that we have the advantage of including a big
number of treated units, with exposition happening simultaneously in all of
them. This gives us the opportunity of using the bootstrap estimator, which
was demonstrated by Arkhangelsky et al. (2021) to be asymptotically normal
with a large number of treated units. We refer to Clarke et al. (2023) Clarke
et al. (2023) for the design of the bootstrap algorithm to estimate the
standard error of the ATE estimators.

##  4 Results

The table presented in 6.1 provides an analysis of the impact of ChatGPT on
software development activities, measured by three key outcomes: the number of
git pushes, the number of repositories, and the number of unique developers,
each standardized per 100,000 people in the population. This analysis employs
three methodological approaches: Difference-in-Differences (DID), Synthetic
Control (SC), and Synthetic Difference-in-Differences (SDID). The coefficients
reported for ”Chat GPT Available” represent the estimated effect of ChatGPT’s
availability on these outcomes.

The results in Panel A indicate a significant positive impact of ChatGPT
availability on the number of git pushes per 100,000 people. The DID
estimation shows an increase of approximately 899.268 pushes, which is highly
significant with a standard error of 147.395. Both the SC and SDID methods
also show positive impacts, with coefficients of 595.059 and 645.623
respectively, though the SC estimate is not statistically significant. The
baseline mean outcome is 741.5 pushes per 100,000, suggesting that the
introduction of ChatGPT has significantly boosted developer activity in terms
of code submissions.

Panel B examines the effect of ChatGPT on the number of repositories created
per 100,000 people. The DID estimate shows a substantial and statistically
significant increase of 1657.141 repositories, indicating a robust positive
effect. However, the SC and SDID estimates, while still positive (464.735 and
204.734 respectively), are not statistically significant. The baseline mean
outcome for this measure is 1097.8 repositories per 100,000 people, suggesting
that while ChatGPT has a pronounced impact on repository creation as per the
DID analysis, the other methods do not confirm the same level of significance.

Panel C focuses on the number of unique developers per 100,000 people. The DID
results indicate a significant increase of 578.912 developers, with a standard
error of 109.398, highlighting a substantial positive impact. In contrast, the
SC and SDID methods show much smaller effects (13.752 and 51.693
respectively), with neither reaching statistical significance. The baseline
mean outcome is 527.4 developers per 100,000, suggesting that the presence of
ChatGPT has potentially encouraged more individuals to engage in development
activities, though the robustness of this finding varies by method.

Overall, the results suggest that the introduction of ChatGPT has generally
positive effects on software development activities, particularly in
increasing the number of git pushes and repositories created, as evidenced by
the DID estimates. However, the magnitude and statistical significance of
these effects vary across different methodological approaches. These findings
imply that while ChatGPT has likely enhanced developer productivity and
engagement, further analysis may be necessary to fully understand the extent
and consistency of its impact across different contexts.

We also conducted a similar analysis at the country-programming language
level, examining the number of unique developers in each economy who made at
least one git push to a repository with a given programming language. From
Tables 6.2 to 6.5, we observe that the introduction of ChatGPT has had a
generally positive impact on developer engagement across various programming
languages. High-level, general-purpose languages like Python and JavaScript,
as well as shell scripting languages, show significant increases in the number
of unique developers. Low-level and systems programming languages also
benefit, though to a lesser extent. The impact on domain-specific languages
varies, with some showing significant positive effects while others remain
unaffected or slightly negatively impacted. These findings suggest that
ChatGPT has enhanced developer efficiency and engagement, particularly in
languages where it can provide substantial assistance with coding tasks.

##  5 Discussion

The rapid progress of Artificial General Intelligence (AGI) is exemplified by
the availability of new tools that augment human cognition, enhancing the
agility with which certain cognitive tasks can be performed. One area where
this utility has become widespread is software development, where large
language models (LLMs) are capable of assisting in various stages of the
software design and engineering process. This assistance translates into
higher productivity, which is crucial for economic growth and prosperity.

Our study found that economies permitting the use of ChatGPT have experienced
a significant increase in software development productivity. From a policy
perspective, these findings suggest that deregulating access to AI tools could
lead to higher productivity and, consequently, foster economic growth.
However, there are potential concerns regarding the quality of code generated
by ChatGPT, as it is trained on diverse sources, some of which may not contain
high-quality code. This could result in increased code churn, potentially
offsetting the benefits of faster code creation.

Moreover, the broader social implications must be considered. While current
evidence primarily highlights productivity gains through collaborative code
writing, future research should also explore the effects of AI availability
through other channels. This includes examining the long-term impacts on code
quality, developer skills, and the broader economic and social implications of
widespread AI tool usage.

In summary, while the integration of AI tools like ChatGPT into software
development offers promising productivity benefits, it is essential to balance
these gains with considerations of code quality and broader societal impacts.
Policymakers and researchers should continue to investigate these dimensions
to ensure the sustainable and beneficial integration of AI technologies into
the economy.

##  6 Tables

Table 6.1: Impact of ChatGPT in software development

|  DID |  SC |  SDID
---|---|---|---
Panel A.  Num Pushes per 100k |  |  |
Chat GPT Available | 899.268*** | 595.059 | 645.623***
| (147.395) | (437.061) | (146.445)
Observations | 2352 | 2352 | 2352
Baseline Mean Outcome | 741.5 | 741.5 | 741.5
Panel B.  Num Repos per 100k |  |  |
Chat GPT Available | 1657.141*** | 464.735 | 204.734
| (342.453) | (620.581) | (133.856)
Observations | 2352 | 2352 | 2352
Baseline Mean Outcome | 1097.8 | 1097.8 | 1097.8
Panel C.  Num developers per 100k |  |  |
Chat GPT Available | 578.912*** | 13.752 | 51.693
| (109.398) | (204.405) | (45.918)
Observations | 2352 | 2352 | 2352
Baseline Mean Outcome | 527.4 | 527.4 | 527.4

Robust standard errors. * $p<$.10, ** $p<$.05, *** $p<$.01. Source: Data is
created from structured data files of public activity on GitHub, aggregated by
economy on a quarterly basis from 2020 onward. Panel A outcome is The number
of times developers in a given economy uploaded code to GitHub per 100,000
people. This metric includes all instances where developers pushed code
changes to repositories, whether through the command line or GitHub’s online
platform, with each push potentially containing multiple commits. Panel B
outcome is The number of software projects (repositories) in a given economy
per 100,000 people. This metric is based on the modal location of all
repository members with triage and above access. It includes all repositories,
regardless of whether they are actively developed or maintained.Panel C
outcome is The number of developer accounts located in a given economy per
100,000 people, based on the modal daily location. This metric excludes bot
accounts and users flagged as “spammy” within internal systems. It includes
developer accounts that may no longer be active.

Table 6.2: Impact of ChatGPT on Software Development for High-Level, General-
Purpose Languages

|  DID |  SC |  SDID
---|---|---|---
Panel A.  Python |  |  |
Chat GPT Available | 8.754*** | 1.586 | 5.231***
| (1.636) | (5.387) | (1.351)
Observations | 2256 | 2256 | 2256
Baseline Mean Outcome | 6.137 | 6.137 | 6.137
Panel B.  JavaScript |  |  |
Chat GPT Available | 13.568*** | 9.484** | 8.257***
| (2.536) | (4.735) | (2.776)
Observations | 2560 | 2560 | 2560
Baseline Mean Outcome | 12.65 | 12.65 | 12.65
Panel C.  Ruby |  |  |
Chat GPT Available | 0.631 | 2.460** | 1.019***
| (0.525) | (1.234) | (0.287)
Observations | 1840 | 1840 | 1840
Baseline Mean Outcome | 2.831 | 2.831 | 2.831
Panel D.  PHP |  |  |
Chat GPT Available | 1.060** | 1.285 | 1.045**
| (0.481) | (1.363) | (0.497)
Observations | 2128 | 2128 | 2128
Baseline Mean Outcome | 2.577 | 2.577 | 2.577
Panel E.  TypeScript |  |  |
Chat GPT Available | 8.421*** | 3.887 | 3.863***
| (1.591) | (2.776) | (0.982)
Observations | 2080 | 2080 | 2080
Baseline Mean Outcome | 2.710 | 2.710 | 2.710

Robust standard errors. * $p<$.10, ** $p<$.05, *** $p<$.01. Source: Data is
created from structured data files of public activity on GitHub, aggregated by
economy-programming langugage on a quarterly basis from 2020 onward. -All
panels measure the number of unique developers in each economy who made at
least one git push to a repository with a given programming language. This
measure includes all developers who contributed code, regardless of the size
or number of contributions, as long as they made at least one push to a
repository.

Table 6.3: Impact of ChatGPT on Software Development for Low-Level and Systems
Programming Languages

|  DID |  SC |  SDID
---|---|---|---
Panel A.  C |  |  |
Chat GPT Available | 3.005*** | 3.469** | 1.690***
| (0.794) | (1.361) | (0.564)
Observations | 1984 | 1984 | 1984
Baseline Mean Outcome | 2.043 | 2.043 | 2.043
Panel B.  C++ |  |  |
Chat GPT Available | 2.681*** | 1.983 | 1.466***
| (0.673) | (1.692) | (0.521)
Observations | 1856 | 1856 | 1856
Baseline Mean Outcome | 2.684 | 2.684 | 2.684
Panel C.  Rust |  |  |
Chat GPT Available | 2.783*** | 3.779*** | 1.728***
| (0.470) | (0.795) | (0.553)
Observations | 1104 | 1104 | 1104
Baseline Mean Outcome | 0.402 | 0.402 | 0.402
Panel D.  Assembly |  |  |
Chat GPT Available | 0.877*** | 1.447*** | 0.643***
| (0.226) | (0.485) | (0.181)
Observations | 1248 | 1248 | 1248
Baseline Mean Outcome | 0.674 | 0.674 | 0.674

Robust standard errors. * $p<$.10, ** $p<$.05, *** $p<$.01. Source: Data is
created from structured data files of public activity on GitHub, aggregated by
economy-programming langugage on a quarterly basis from 2020 onward. -All
panels measure the number of unique developers in each economy who made at
least one git push to a repository with a given programming language. This
measure includes all developers who contributed code, regardless of the size
or number of contributions, as long as they made at least one push to a
repository.

Table 6.4: Impact of ChatGPT on Software Development for Shell Scripting
Languages

|  DID |  SC |  SDID
---|---|---|---
Panel A.  Shell |  |  |
Chat GPT Available | 5.888*** | 8.417** | 4.974***
| (1.410) | (3.766) | (1.506)
Observations | 2224 | 2224 | 2224
Baseline Mean Outcome | 5.664 | 5.664 | 5.664
Panel B.  Batchfile |  |  |
Chat GPT Available | 1.776*** | 1.766 | 1.218***
| (0.373) | (1.554) | (0.254)
Observations | 1584 | 1584 | 1584
Baseline Mean Outcome | 1.752 | 1.752 | 1.752
Panel C.  PowerShell |  |  |
Chat GPT Available | 1.262*** | 1.719** | 0.841***
| (0.228) | (0.815) | (0.184)
Observations | 1328 | 1328 | 1328
Baseline Mean Outcome | 0.975 | 0.975 | 0.975

Robust standard errors. * $p<$.10, ** $p<$.05, *** $p<$.01. Source: Data is
created from structured data files of public activity on GitHub, aggregated by
economy-programming langugage on a quarterly basis from 2020 onward. -All
panels measure the number of unique developers in each economy who made at
least one git push to a repository with a given programming language. This
measure includes all developers who contributed code, regardless of the size
or number of contributions, as long as they made at least one push to a
repository.

Table 6.5: Impact of ChatGPT on Software Development for Domain-Specific
Languages (DSLs)

|  DID |  SC |  SDID
---|---|---|---
Panel A.  TSQL |  |  |
Chat GPT Available | -0.699*** | 0.019 | 0.053
| (0.263) | (0.547) | (0.400)
Observations | 1456 | 1456 | 1456
Baseline Mean Outcome | 0.836 | 0.836 | 0.836
Panel B.  PLpgSQL |  |  |
Chat GPT Available | 0.344** | 0.564** | 0.270***
| (0.144) | (0.287) | (0.085)
Observations | 928 | 928 | 928
Baseline Mean Outcome | 0.278 | 0.278 | 0.278
Panel C.  HTML |  |  |
Chat GPT Available | 12.347*** | 9.674** | 7.981***
| (2.347) | (4.247) | (2.352)
Observations | 2592 | 2592 | 2592
Baseline Mean Outcome | 15.44 | 15.44 | 15.44
Panel D.  CSS |  |  |
Chat GPT Available | 12.303*** | 8.378** | 7.775***
| (1.753) | (4.210) | (2.101)
Observations | 2544 | 2544 | 2544
Baseline Mean Outcome | 11.72 | 11.72 | 11.72
Panel E.  MATLAB |  |  |
Chat GPT Available | 0.157** | 1.518*** | 0.190*
| (0.074) | (0.301) | (0.101)
Observations | 896 | 896 | 896
Baseline Mean Outcome | 0.292 | 0.292 | 0.292
Panel F.  R |  |  |
Chat GPT Available | 0.308*** | 3.202*** | 0.366***
| (0.116) | (0.598) | (0.141)
Observations | 976 | 976 | 976
Baseline Mean Outcome | 0.251 | 0.251 | 0.251

Robust standard errors. * $p<$.10, ** $p<$.05, *** $p<$.01. Source: Data is
created from structured data files of public activity on GitHub, aggregated by
economy-programming langugage on a quarterly basis from 2020 onward. -All
panels measure the number of unique developers in each economy who made at
least one git push to a repository with a given programming language. This
measure includes all developers who contributed code, regardless of the size
or number of contributions, as long as they made at least one push to a
repository.

##  7 Figures

Figure 1: Estimated Trends and Weights for Pushes per 100k

(a) Estimated DiD trends

(b) Estimated DiD weights

(c) Estimated SC trends

(d) Estimated SC weights

(e) Estimated SDiD trends

(f) Estimated SDiD weights

Figure 2: Estimated Trends and Weights for Repositories per 100k

(a) Estimated DiD trends

(b) Estimated DiD weights

(c) Estimated SC trends

(d) Estimated SC weights

(e) Estimated SDiD trends

(f) Estimated SDiD weights

Figure 3: Estimated Trends and Weights for Developers per 100k

(a) Estimated DiD trends

(b) Estimated DiD weights

(c) Estimated SC trends

(d) Estimated SC weights

(e) Estimated SDiD trends

(f) Estimated SDiD weights

## References

  * Abadie et al. (2010) Abadie, A., Diamond, A. and Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of california’s tobacco control program. Journal of the American Statistical Association, 105 (490), 493–505.
  * Abu Jaber et al. (2023) Abu Jaber, M., Beganovic, A. and Abd Almisreb, A. (2023). Methods and applications of chatgpt in software development: A literature review. Southeast Europe Journal of Soft Computing.
  * Ahmad et al. (2023) Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Aktar, M. S. and Mikkonen, T. (2023). Towards human-bot collaborative software architecting with chatgpt. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, EASE ’23, New York, NY, USA: Association for Computing Machinery, p. 279–285.
  * Arkhangelsky et al. (2021) Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W. and Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111 (12), 4088–4118.
  * Badini et al. (2023) Badini, S., Regondi, S., Frontoni, E. and Pugliese, R. (2023). Assessing the capabilities of chatgpt to improve additive manufacturing troubleshooting. Advanced Industrial and Engineering Polymer Research, 6 (3), 278–287.
  * Clarke et al. (2023) Clarke, D., Pailañir, D., Athey, S. and Imbens, G. (2023). Synthetic difference-in-differences estimation. IZA Discussion papers.
  * Del Rio-Chanona et al. (2023) Del Rio-Chanona, M., Laurentsyeva, N. and Wachs, J. (2023). Are large language models a threat to digital public goods? evidence from activity on stack overflow.
  * Demicri et al. (2023) Demicri, O., Hannane, J. and Zhu, X. (2023). Who is ai replacing? the impact of generative ai on online freelancing platforms. SSRN.
  * Gallea (2023) Gallea, Q. (2023). From mundane to meaningful: Ai’s influence on work dynamics – evidence from chatgpt and stack overflow.
  * Jalil et al. (2023) Jalil, S., Rafi, S., LaToza, T. D., Moran, K. and Lam, W. (2023). Chatgpt and software testing education: Promises & perils. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE.
  * Kashefi and Mukerji (2023) Kashefi, A. and Mukerji, T. (2023). Chatgpt for programming numerical methods.
  * Kreitmeir and Raschky (2023) Kreitmeir, D. and Raschky, P. (2023). The unintended consequences of censoring digital technology - evidence from italy’s chatgpt ban. ArXiv, abs/2304.09339.
  * OpenAI (2018) OpenAI (2018). OpenAI Charter. Tech. rep., OpenAI.
  * OpenAI (2022) — (2022). Introducing ChatGPT. Tech. rep., OpenAI.
  * Organization (2022) Organization, W. I. P. (2022). World Intellectual Property Report 2022. WIPO.
  * Rahmaniar (2023) Rahmaniar, W. (2023). Chatgpt for software development: Opportunities and challenges. TechRxiv Preprint.
  * Saggu and Ante (2023) Saggu, A. and Ante, L. (2023). The influence of chatgpt on artificial intelligence related crypto assets: Evidence from a synthetic control analysis. Finance Research Letters, 55, 103993.
  * Shen et al. (2023) Shen, Y., Song, K., Tan, X., Li, D., Lu, W. and Zhuang, Y. (2023). Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.
  * Sobania et al. (2023) Sobania, D., Briesch, M., Hanna, C. and Petke, J. (2023). An analysis of the automatic bug fixing performance of chatgpt.
  * Zhang et al. (2023) Zhang, Q., Zhang, T., Zhai, J., Fang, C., Yu, B., Sun, W. and Chen, Z. (2023). A critical review of large language model on software engineering: An example from chatgpt and automated program repair.


# Minimal Supervision for Morphological Inflection

Omer Goldman
Bar Ilan University
<EMAIL_ADDRESS>
&Reut Tsarfaty
Bar Ilan University
<EMAIL_ADDRESS>

###### Abstract

Neural models for the various flavours of morphological inflection tasks have
proven to be extremely accurate given ample labeled data, yet labeled data may
be slow and costly to obtain. In this work we aim to overcome this annotation
bottleneck by bootstrapping labeled data from a seed as small as five labeled
inflection tables, accompanied by a large bulk of unlabeled text. Our
bootstrapping method exploits the orthographic and semantic regularities in
morphological systems in a two-phased setup, where word tagging based on
analogies is followed by word pairing based on distances. Our experiments with
the Paradigm Cell Filling Problem over eight typologically different languages
show that in languages with relatively simple morphology, orthographic
regularities on their own allow inflection models to achieve respectable
accuracy. Combined orthographic and semantic regularities alleviate
difficulties with particularly complex morpho-phonological systems. We further
show that our bootstrapping methods substantially outperform hallucination-
based methods commonly used for overcoming the annotation bottleneck in
morphological inflection tasks.

##  1 Introduction

The introduction of neural models into natural language processing in the last
decade has led to huge improvements in all supervised generation tasks,
including morphological inflection.111In recent years the term reinflection
has surfaced as a reference to morphological inflection done not necessarily
from the lemma. In this paper we will refer to both inflection and
reinflection as ”inflection”, and specify whenever we refer to inflection done
exclusively from the lemma. In particular, previous works Cotterell et al.
(2017); Silfverberg and Hulden (2018) have achieved near-perfect performance
over the Paradigm Cell Filling Problem (PCFP) Ackerman et al. (2009), wherein
models are required to provide any form in an inflection table, given a few
forms of the same lexeme.222Throughout the paper we conform to the linguistic
terminology, where ‘lexeme’ stands for an abstract lexical entry, e.g., the
English run, and its ‘forms’ are the words that convey this lexical meaning
with some inflectional relations between them, e.g. run, running but not
runner. ‘Paradigm’ will stand for a group of lexemes sharing a POS tag, in the
same manner as the English lexeme run is part of the verbal paradigm.

Two lines of recent work made progress towards less supervision, in different
fashions. The first simply provided scenarios with smaller training sets — for
example, in SIGMORPHON’s shared tasks Cotterell et al. (2017, 2018). The
second research avenue aims to discover the paradigmatic structure of an
unknown language given a large bulk of unlabeled data, either alone Soricut
and Och (2015); Elsner et al. (2019), accompanied by a list of all relevant
forms in the vocabulary Erdmann et al. (2020), or by a list of lemmas Jin et
al. (2020).

The problem with the first kind of attempts is that given the neural nature of
the most successful models, their performance on limited supervision is
capped, and data augmentation is likely to help only if the initial data is
diverse enough. As for the second scenario of no supervision at all, it is
somewhat pessimistic and unrealistic. Even if much labeled data for a language
does not exist for a low-resourced language, typically there exists knowledge
about its paradigm structure that can be employed. UniMorph Kirov et al.
(2018), for example, includes small amounts of labeled inflection tables for
many languages, from obscure ones like Ingrian to national languages with
widespread usage that lack global attention like Georgian.

In this work we propose a new, low-resourced morphological inflection
scenario, which is more optimistic and realistic for those widely-spoken
sparsely-annotated languages. We assume a minimal supervision set and a large
bulk of unlabeled text, thus balancing both trends of lowering supervision
resources. We bootstrap a tiny amount of as little as five inflection tables,
that could be easily written by hand, into a full-blown training set, albeit
noisy, for an inflection model trained in a supervised manner. Our approach
makes use of the regularities abundant in inflectional morphology, both
orthographic regularities and semantic ones.

Based on this method we train morphological inflection models for eight
languages and show that, for the five Indo-European of them, orthographic
regularity is enough to train a morphological inflector that achieves
reasonable success. We further show that for languages with complicated
morpho-phonological systems, such as Finnish, Turkish and Hungarian, a method
combining both orthographic and semantic regularities is needed in order to
reach the same level of performance. An error analysis reveals that the closer
an inflection is to the verge of disappearance, the poorer our system performs
on it, as less examples exist in the data-derived vocabulary. Our models
outperform Makarov and Clematide (2018)’s model designed for low-resourced
setting, even when equipped with additional hallucinated data Anastasopoulos
and Neubig (2019). We also outperform the best model of Jin et al. (2020),
that didn’t use any inflection tables, and their skyline for most languages.

We conclude that bootstrapping datasets for inflectional morphology in low-
resourced languages, is a viable strategy to be adopted and explored.333Our
code is available on https://github.com/OnlpLab/morphodetection.

##  2 The Minimally-Supervised Setup

##### Problem Statement

Let $\mathcal{L}$ be a set of lexemes, each corresponding to an inflection
table $W^{l}=\\{w_{t_{1}}^{l}...w_{t_{m}}^{l}\\}$ where $w_{t_{i}}^{l}$ is a
form of lexeme $l$ bearing the feature bundle $t_{i}$. Our goal is to train an
inflection model that maps words bearing one set of features to words bearing
another set within the same lexeme.

$(\langle t_{j},w_{t_{j}}^{l_{i}}\rangle,t_{k})\mapsto w_{t_{k}}^{l_{i}}$

For example, in the French verbal paradigm:  ($\langle$ IndPrs1Sg, _finis_
$\rangle$, SbjvPst2Sg ) $\mapsto$ _finis_ _s_ _es_

In order to induce this function, we propose a minimally-supervised scenario
where we are only given a small set of $n$ examples of complete inflectional
tables $\mathcal{L}=\\{W^{l_{1}}...W^{l_{n}}\\}$ (each of which is of size
$m$), and a large bulk of naturally occurring (that is, in-context) unlabeled
data in that language, that is, $w_{1}.....w_{c}$, such that $c>>nm$.

##  3 The Algorithmic Framework

This work suggests utilizing the patterns exhibited in the small supervision
seed, and finding words that exhibit similar (or, analogous) patterns.

The algorithm proposed here works in two phases. First we tag words with
morphological features if they are found similar (analogous) to examples in
the minimal supervision. Then, we pair the tagged words such that each pair
will include two forms of the same lexeme. This algorithmic division of labor
allows the pairing module to provide a sanity check, and reduce the noise
potentially lingering from the word-tagging module.

The above sketch of the algorithm is quite generic, that is, we can get
different instantiations of this framework by plugging in different ways to
calculate similarities or analogies between words. In our various algorithmic
implementations, we will use the regularities that prevail in inflectional
morphology and are detectable even from a tiny amount of supervision. These
regularities are manifested both orthographically, as edits between forms tend
to repeat themselves across lexemes, and semantically, as forms that share an
inflectional function tend to be used similarly.

In the rest of this section we will describe both modules, each with its
different variants depending on the different notion of similarity used.

###  3.1 Morphological Features Assignment

In order to assess whether a pair of unseen words $w_{1},w_{2}$ belong to the
same lexeme, we first need to characterize the relationship between those two
words. It is then imperative to compare the concrete relation between
$w_{1},w_{2}$ to some representation of the abstract relation
$R_{t_{j},t_{k}}$ between $2$ morphological categories $t_{j}$ and $t_{k}$ in
the same paradigm. If these concrete and abstract relations are sufficiently
similar, $w_{1}$ and $w_{2}$ will be tagged as bearing features $t_{j}$ and
$t_{k}$, respectively. The idea, in a nutshell, is to obtain the
representation $R_{t_{j},t_{k}}$ by aggregating differences between the forms
of the $t_{j},t_{k}$ entries in all $n$ inflection tables in the minimal seed.
These differences can be stated in terms of either semantics, orthography or a
combination thereof.

####  3.1.1 Orthography-Based Tagging

In the orthographic case we define the difference between a pair of words as
the edits needed to get from one word to the other. The edits our system
expects are a list of sub-strings that were deleted and a list of those added,
sorted from left to right.444Though without position indices, in order to deal
with words of various lengths.

For every pair of morphological categories $(t_{j},t_{k})$, their orthographic
relation $R_{t_{j},t_{k}}^{O}$ is defined by the set of edits observed in the
$n$ supervision inflection tables, one from each lemma $l_{i}$

$\hat{R}_{t_{j},t_{k}}^{O}:=\big{\\{}\operatorname{edits}(w_{t_{j}}^{l_{i}},w_{t_{k}}^{l_{i}})\big{\\}}_{i=1}^{n}$

In order to check whether a pair of new words $(w_{a},w_{b})$ exhibits a
relation between two $t_{j},t_{k}$ morphological categories, we check whether
the edits between them belong to the relation representation:

$\operatorname{edits}(w_{a},w_{b})\stackrel{{\scriptstyle?}}{{\in}}\hat{R}_{t_{j},t_{k}}^{O}$

Consider the example from the French verbal paradigm where the examples for
the relation (IndPrs1Sg, IndPrs3Sg) are {(finis,finit), (bois, but),
(parle,parla)}. In this case the representation of this relation is:

$\displaystyle\hat{R}_{\textsc{IndPrs1Sg},\textsc{IndPrs3Sg}}^{O}$
$\displaystyle=$ $\displaystyle\\{(\textit{`s',`t'}),(\textit{`ois'}$
$\displaystyle\textit{,`ut'}),(\textit{`e',`a'})\\}$

Since the edits between (aime, aima) are identical to those between
(parle,parla), then aime would be correctly considered for tagging as
IndPrs1Sg and aima – as IndPstPrf3Sg.

This procedure is however highly prone to coincidence. For example, the edits
between le and la are the same as between parle and parla, although the former
are actually determiners, and not part of the verbal paradigm. Given the
multitude of relations available, we can expect many edits between incidental
pairs of words to match an edit seen in the gold data. To overcome this, we
propose to take the complete paradigm structure into account, rather than
considering word pairs in isolation.

Concretely, we propose to tag only words that have been found to answer this
criterion for multiple relations in the same paradigm, covering at least half
of the size of the paradigm. So aime would be tagged as IndPrs1Sg since
$\operatorname{edits}(\textit{aime,
aima})=\operatorname{edits}(\textit{parle,parla})$ and
$\operatorname{edits}(\textit{aime,
aimons})=\operatorname{edits}(\textit{parle, parlons})$ and so on, but le
won’t be tagged as IndPrs1Sg since the French vocabulary does not contain
lons.

With this tightened criterion, the orthographic algorithm might be
sufficiently precise, but may not be sufficiently diverse, so as to obtain
high recall.

####  3.1.2 Semantics-Based Tagging

The orthographic criterion only considers exact match with the observed edits
so it is expected to miss irregulars and classes unattested in the $n$
supervision inflection tables.555the term ‘class’ refers to a group of lexemes
in a paradigm that display similar inflection patterns. Traditionally known as
‘conjugation’ and ‘declension’ in the description of verbal and nominal
paradigms, respectively. E.g., the Spanish verbal paradigm is said to include
$3$ classes: -er, -ar and -ir verbs. This can pose a significant problem to
paradigms that have more than $n$ classes or display significant morpho-
phonological processes not present in the labeled examples.

To overcome the generalization problem of orthographic edits we propose to
consider semantic regularities, since the differences in meaning and usage
rarely correlate with orthography. Semantic regularity arises from agreement,
a phenomenon in which words in a sentence must have the same morphological
features as some other words in the sentence, effectively creating equivalence
classes. Modern algorithms for word embeddings that extract semantics from co-
occurrences, following Firth (1957), are naturally suitable to exploit this
kind of regularity.

In this setting the difference between words is defined as the difference
between their embedded vectors. And for every pair of morphological feature
bundles $(t_{j},t_{k})$ the representation of their semantic relation is
estimated by the average over those relevant examples

$\hat{R}_{t_{j},t_{k}}^{S}=\frac{1}{n}\sum_{i=1}^{n}v(w_{t_{j}}^{l_{i}})-v(w_{t_{k}}^{l_{i}})$

A new word pair will be tagged $t_{j},t_{k}$ if their difference is close
enough, in cosine-distance terms, to $\hat{R}_{t_{j},t_{k}}^{S}$

$D_{C}\big{(}\hat{R}_{t_{j},t_{k}}^{S},v(w_{a})-v(w_{b})\big{)}\stackrel{{\scriptstyle?}}{{\leq}}\hat{C}_{t_{j},t_{k}}^{S}$

where $D_{C}$ is the cosine distance function and $\hat{C}_{t_{j},t_{k}}^{S}$
is an estimation of a relation-specific cut-off score set by the average
scatter of the relevant supervision examples around their average:

$\hat{C}_{t_{j},t_{k}}^{S}=\frac{1}{n}\sum_{i=1}^{n}D_{C}\big{(}\hat{R}_{t_{j},t_{k}}^{s},v(w_{t_{j}}^{l_{i}})-v(w_{t_{k}}^{l_{i}})\big{)}$

Although lacking the orthographic disadvantages, here
$\hat{R}_{t_{j},t_{k}}^{S}$ might be a biased representation that misses many
examples, or mistakenly tags incorrect words. For this reason we suggest the
third algorithm combining both types or regularities.

####  3.1.3 Combined Tagging

The idea behind this variant is to consider word pairs that answer both the
orthographic and semantic criteria as semi-gold examples that will be added to
better estimate the relation $\hat{R}_{t_{j},t_{k}}^{S}$. Thus we harness the
accuracy of the orthographic criterion to combat the bias in the semantic
representation.

Specifically, a pair of words $(w_{1},w_{2})$ is considered semi-gold if their
edit script is in $\hat{R}_{t_{j},t_{k}}^{O}$, and the distance of their
difference vector from the relation vector $\hat{R}_{t_{j},t_{k}}^{S}$ is
smaller than the furthest corresponding difference vector of gold examples.

Note that we relaxed both criteria comparing to those in the previous
sections. The orthographic criterion is relaxed by dropping the requirement
for the complete paradigm, and the semantic criterion is relaxed by replacing
$\hat{C}_{t_{j},t_{k}}^{S}$ with a more inclusive cut-off relying on max
rather than average distance

$\hat{C}_{t_{j},t_{k}}^{Comb}=\max_{i}D_{C}\big{(}\hat{R}_{t_{j},t_{k}}^{s},v(w_{t_{j}}^{l_{i}})-v(w_{t_{k}}^{l_{i}})\big{)}$

The relaxations allow inclusion of more semi-gold examples, and they are
sensible as both criteria are mutually constraining.

Pairs of words that have satisfied both criteria are very likely correct
examples of the morphological relation $(t_{j},t_{k})$. Thus they are added to
create a new semantic representation $\hat{R}_{t_{j},t_{k}}^{S}$. The new
representation may be used again to find more semi-gold examples, executing
this stage iteratively. We set the stopping criterion when either no examples
are added to $R_{t_{j},t_{k}}^{S}$ after an iteration, or when so many
examples where added so that the run time per iteration exceeded $48$ hours in
our implementation.

Once $\hat{R}_{t_{j},t_{k}}^{S}$ is settled, and in order to include tagging
of words with different edits, we finally tag words according to the semantic
criterion alone with the corresponding $\hat{C}_{t_{j},t_{k}}^{S}$.

###  3.2 Pairing Tagged Words

Given the output of the first module, a list of words tagged with some
morphological sets of features, the second module pairs tagged words from the
first module, such that both tagged words are forms of the same lexeme.

We start by grouping the tagged words from the first module into $m$ bins
$\\{B_{t_{i}}\\}_{i=1}^{m}$, each containing all words tagged with $t_{i}$. We
then need to find for each word $w_{t_{i}}^{l_{j}}\in B_{t_{i}}$ corresponding
words from other bins that are forms of the same lexeme $l_{j}$.

Ideally, if each bin contained exactly one correct form of every lexeme,
pairing the most similar words across bins should suffice in order to collect
all the forms in the paradigm. In reality, due to noise, bins may include
several forms of the same lexeme or none at all. Assuming enough words are
tagged, it seems better to simply drop the cases where several forms of the
same lexeme are occupying the same bin, rather then trying to locate the one
that was tagged correctly. Therefore, we would like to pair words such that
they are distinct nearest neighbors across bins, i.e., where the second
nearest neighbor is much more distant.

To achieve this, we scale the distance using the Cross-domain Similarity Local
Scaling (CSLS) score, suggested by Lample et al. (2018) in the context of bi-
lingual embeddings. The CSLS score scales the cosine distance between vectors
across groups $x\in X,y\in Y$ with the average distance from their top-$k$
nearest neighbors. We set $k=2$ when applying this method here, since we aim
to discard cases where a form $w_{t_{i}}^{l_{q}}\in B_{t_{i}}$ has at least
two corresponding forms in another bin $w_{t_{j}}^{l_{q}},w_{t_{k}}^{l_{q}}\in
B_{t_{j}}$, with one presumably misplaced. The definition of CSLS for $k=2$
can be written as

$\displaystyle CSLS$ $\displaystyle(x,y)=D(x,y)-$
$\displaystyle\frac{1}{2}D(x,NN_{2}^{x}(Y))-\frac{1}{2}D(NN_{2}^{y}(X),y)$

where $NN_{2}^{x}(Y)$ is $x$’s second nearest neighbor in group $Y$.

Although the original distance measure $D(\cdot,\cdot)$ used by Lample et al.
(2018) is the cosine distance function, any distance measure will do. In the
semantic algorithm (Sec. 3.1.2), we apply cosine distance that reflects
semantic similarity. In the orthgraphic algorithm (Sec. 3.1.1), we plug in the
Levenshtein edit distance, to utilize the orthography rather than the
semantics.

After scoring all possible word pairs, the best scored pairs are taken as a
training set for a supervised neural morphological inflector.

##  4 Experimental Evaluation

###  4.1 Experimental Setup

We set out to empirically examine whether it is possible to train a
morphological inflection model while starting with a minimal number of labeled
inflection tables, using regularities of different types. Our experiments
include the verbal paradigm of eight languages: English, Russian, Latvian,
Spanish, French, Finnish, Hungarian and Turkish, although our methods are
suitable for any paradigm.

To make evaluation of the inflection model possible, we had to choose
languages with a sufficient amount of gold (test) data, and simulate a low-
resource scenario for training. This limited us to western languages, and we
aimed to include as many non-Indo-European languages as possible.

##### The Unlabeled Data

The problem setting we specified in Section 2 designates the use of a bulk of
unlabeled text. In actuality, the proposed algorithm makes use of the text for
(i) collecting a vocabulary to seek tagging candidates, and (ii) training
embeddings to be used for calculating the semantic relations between candidate
pairs. In our experiments, we simply employed language-specific pre-trained
word embeddings for both purposes.

We used the pre-trained FastText vectors provided by Grave et al. (2018),
following their reported success over morphological analogies in the Bigger
Analogy Test Set Li et al. (2018). We clipped the $2$ million long FastText
vocabulary to include only real words, i.e., we keep only lower-cased tokens
that do not include non-alphabetic characters, and include at least one
vowel.666This last criterion happens to be suitable for all 8 languages
examined. Admittedly there are languages, like Czech, that allow less sonorant
syllable nucleus. This procedure downsized the vocabulary size to between
about $200k-500k$ words per language.777To reduce run time the Finnish and
Hungarian vocabularies were further reduced to include only words that
appeared at least twice in the Finnish Parole and Hungarian Gigaword corpora
Bartis (1998); Oravecz et al. (2014). Additionally, for run time reasons, we
capped the size of the vocabulary for the orthographic variant to $200k$ words
per language.

##### Minimal Supervision Source

For every language we extracted inflection tables for $5$ lexemes from
UniMorph Kirov et al. (2018). We aimed for lexemes that are both frequently
used, to have robust embedded representations, and from diverse classes, to
capture as much as possible of the linguistic behaviors of the language. To
this end we targeted lexemes with the largest amount of forms appearing in the
embeddings’ vocabulary and manually selected from them such that they belong
to as diverse classes as possible.888Cases of syncretism were trivially solved
by merging $2$ categories into one category with $2$ tags if all supervision
lexemes had the same forms in those $2$ categories. This strategy might pose a
problem in cases of partial syncretism, where categories differ in forms only
in classes that happen to be absent from the minimal supervision. The manually
enforced class diversity in the selection process was devised to solve this
problem as well.

##### The Inflection Model

As our supervised inflection model, to be trained on the bootstrapped data, we
used an out-of-the-box sequence-to-sequence LSTM model with attention over the
input’s characters and the features of the source and target forms. We used
the publicly available model of Silfverberg and Hulden (2018). We trained the
model for 50 epochs, on either up to $10,000$ data training examples outputted
by our system in the minimally supervised scenario, or on $6,000$ training
examples in the supervised scenario. The latter provides an upper-bound for
the performance of our minimally supervised system. Silfverberg and Hulden
(2018) provided train and test data for $4$ of the languages examined here,
and we hand-crafted similar data sets for English, Russian, Hungarian and
Turkish.

The model’s evaluation metric is exact match of the outputted string to the
gold output.

##### Models, Baseline and Skyline

We report results for the three system variants we tested:

  * •

Orth: The orthography-based system (§3.1.1);

  * •

Sem: The embedings-based system (Sec. 3.1.2);

  * •

Comb: The combined system (Sec. 3.1.3).

In addition, in order to assess the added value in our systems we include
accuracy for two baseline models trained in the low-resourced setting without
finding new exmaples:

  * •

Overfit: Our sequence-to-sequence inflection model, based on Silfverberg and
Hulden (2018).

  * •

CA: The neural transition-based model model of Makarov and Clematide
(2018).999This model was designed for low-resourced settings and achieved
state of the art results in SIGMORPHON’s $2018$ shared-task on inflection from
lemma Cotterell et al. (2018). We adapted it to allow for inflection from an
arbitrary form.

We also include the performance of a model trained in a fully-supervised
fashion (Sup) as an upper-bound.

Inflection | Average | Eng | Rus | Lav | Fra | Fin | Spa | Tur | Hun
---|---|---|---|---|---|---|---|---|---
Overfit | 0.01 | 0.00 | 0.00 | 0.01 | 0.01 | 0.00 | 0.00 | 0.01 | 0.00
CA | 0.18 | 0.38 | 0.30 | 0.15 | 0.06 | 0.06 | 0.02 | 0.22 | 0.23
Orth | 0.40 | 0.86 | 0.75 | 0.49 | 0.31 | 0.13 | 0.50 | 0.32 | 0.01
Sem | 0.09 | 0.01 | 0.33 | 0.01 | 0.07 | 0.00 | 0.24 | 0.03 | 0.00
Comb | 0.54 | 0.90 | 0.73 | 0.48 | 0.32 | 0.48 | 0.53 | 0.49 | 0.48
Supervised | 0.93 | 0.95 | 0.93 | 0.86 | 0.84 | 0.95 | 0.95 | 0.94 | 0.98
Paradigm size | 5 (8) | 15 (15) | 27 (34) | 32 (48) | 39 (39) | 56 (65) | 30 (30) | 48 (49)
Table 1: Morphological inflection accuracies for all languages and systems. Paradigm sizes in number of forms (functions) is also included for reference. The best minimally-supervised results are in bold. | Avg | Eng | Rus | Fin | Spa | Tur
---|---|---|---|---|---|---
CoNLL17 | 0.49 | 0.71 | 0.37 | 0.25 | 0.76 | 0.35
PCS | 0.31 | 0.74 | 0.31 | 0.06 | 0.31 | 0.12
Comb | 0.62 | 0.92 | 0.60 | 0.51 | 0.57 | 0.49
Table 2: Comparison between our best system (Comb), Jin et al. (2020)’s best
system per-language (PCS) and their skyline (CoNLL17) on the inflection-from-
lemma task.

###  4.2 Results

Table 1 summarizes the inflection accuracies for all models. Although the
results vary widely across languages, Comb consistently achieves best or near-
best results. While Comb outperforms the other two variants in Finnish,
Hungarian and Turkish, its performance is roughly on par with Orth for the
five Indo-European (IE) languages in our selection, pointing to the marginal
role semantics played in those languages. Both Comb and Orth outperform CA,
our stronger baseline.

Comparing our best system to the Sup skyline, it seems that the room for
improvement is bigger for languages with bigger paradigms (paradigm sizes are
indicated in the table, both in number of forms and number of functions).
Impressively, the results for English fall short only by $5$ percentage points
from the fully-supervised upper-bound.

In terms of tagging accuracy, outputted data sets are quite invariably precise
across most languages and models (details in the supplementary material). The
success of the inflection model seem to be correlated with the amount of words
tagged by the first module, rather than on it’s precision. The Pearson
correlation between the inflection accuracy and the log of the averaged tagged
amount per paradigm cell is $0.87$.

To exemplify the added value of our minimally-supervised scenario we provide
in Table 2 a comparison with the completely unsupervised model of Jin et al.
(2020). We compare their best model to our best model (Comb) applied on
inflection from lemma. We provided our model with the same $100$ lemmas in
their test set, and tested our model’s capability to complete the respective
inflection tables. For most languages, our model’s inflection accuracy
surpasses even the skyline named by Jin et al. (2020), an edits-based
minimally-supervised algorithm that uses $10$ inflection tables, while we are
using only $5$.

###  4.3 Analysis

Inflection | Average | Eng | Rus | Lav | Fra | Fin | Spa | Tur | Hun
---|---|---|---|---|---|---|---|---|---
+hallucination
Overfit | 0.10 (+9) | 0.08 (+8) | 0.07 (+7) | 0.21 (+20) | 0.04 (+3) | 0.04 (+4) | 0.10 (+10) | 0.19 (+18) | 0.10 (+10)
CA | 0.34 (+16) | 0.46 (+8) | 0.40 (+10) | 0.41 (+26) | 0.22 (+16) | 0.24 (+18) | 0.21 (+19) | 0.38 (+16) | 0.40 (+17)
Comb | 0.64 (+9) | 0.87 (-3) | 0.71 (-2) | 0.57 (+9) | 0.38 (+6) | 0.54 (+6) | 0.73 (+20) | 0.62 (+13) | 0.70 (+22)

Table 3: Results of the baselines and of our best model Comb when $5000$
hallucinated examples are added. In parenthesis are the differences comparing
to the relevant result in Table 1.

The results on our selection of languages, suggest a clear division between
Indo-European (IE) languages and non-IE ones. In the former, adding semantic
knowledge yields minute improvements at best, while in the latter, Comb
clearly outperforms over Orth. We conjuncture that this is because the IE
languages in our selection exhibit a fairly simple morpho-phonological system,
with relatively few classes and almost no phonological stem-suffix
interaction. In contrast, all non-IE languages selected exhibit vowel harmony
that multiply the edits related to a single morphological relation, in
addition to consonant gradation in the case of Finnish.

This difficulty is magnified with a large amount of classes, as in the Finnish
verbal paradigm that includes $27$ classes101010according to the Research
Institute for the Languages of Finland. of which about a dozen seem to include
more than a few lexemes according to the statistics over Wiktionary
entries.111111https://en.wiktionary.org/wiki/Appendix:Finnish_conjugation

Another imbalance in the results is the inverse correlation between the size
of the paradigm and the performance of the inflector trained over the
outputted data. We speculate that the effect arises from the fact that
languages with bigger paradigms include inflections for functions that are in
exceedingly rare use, either because they are considered archaic or literary,
or because they are used for functions that are far less common. It means that
the data-driven vocabulary is likely to include less forms with those
features, or miss them completely. It also means that these forms will have
noisier vector embeddings.

Figure 1: Spanish inflection accuracy per morphological category as a function
of the category’s abundance in the vocabulary. Plotted results for Orth (+)
and Comb ($\bullet$).

Figure 2: Learning curves of the supervised model in $4$ of the languages:
Russian and Turkish, with our best performance marked by the dashed line.
Plots for all languages can be found in the supplementary material

To probe this conjecture empirically, we plotted the Spanish inflection by
morphological category against the number of forms found in FastText’s
vocabulary (Fig. 1). The figure includes results for both best systems in
terms of inflection accuracy, namely Orth and Comb. It shows that on common
forms the inflection accuracy is on par with the performance on small-paradigm
languages, while the rarer forms are mostly the ones driving the total
accuracy down.121212This analysis was possible only for Spanish as all other
highly-inflectional languages have partial and skewed data on UniMorph, so
automatically counting forms is impossible.

In Fig. 2 we display the amount of annotation labor saved by using our model
for a subset of the languages. We compare the performance of Comb to models
trained on increasing amounts of inflection tables, and we show that our model
harnessed $5$ annotated inflection tables to output performance equivalent to
over $400$ lexemes needed in a supervised scenario for most languages. That’s
a reduction of two orders of magnitude in the labor needed for annotation.
This may lead to more sophisticated annotation procedures that could
capitalize on the bootstrapping approach proposed in this paper, to save time
and efforts in creating morphological resources needed for many languages.

##### Data Hallucination

The methods introduced here to combat the annotation bottleneck are somewhat
orthogonal to the method of data hallucination introduced in Anastasopoulos
and Neubig (2019), as we aim for identifying new real examples from a
vocabulary, rather than hallucinate nonce ones. Nonetheless we added $5000$
hallucinated examples to both baselines and to our best system (Comb) to
assess the combined power of both methods (Table 3). Unsurprisingly, we found
that hallucinated data helped improving the results of the baselines, since
the hallucinated data is the sole source of new examples. However, this source
of example is also quite noisy, and we see that for Comb adding hallucinated
examples marginally harms the results for English and Russian, those languages
with best Comb performance on our bootstrapped data. Furthermore, note that
even with the hallucinated examples, the baselines still underperform compared
to Comb models without hallucination.

Inflection | Average | Eng | Rus | Lav | Fra | Fin | Spa | Tur | Hun
---|---|---|---|---|---|---|---|---|---
Orth | 0.25 (-15) | 0.34 (-52) | 0.52 (-23) | 0.49 (+0) | 0.07 (-26) | 0.13 (+0) | 0.10 (-40) | 0.25 (-7) | 0.12 (+11)
Sem | 0.05 (-4) | 0.00 (-1) | 0.20 (-13) | 0.01 (+0) | 0.00 (-7) | 0.00 (+0) | 0.05 (-19) | 0.13 (+10) | 0.00 (+0)
Comb | 0.49 (-6) | 0.87 (-3) | 0.59 (-16) | 0.48 (+0) | 0.15 (-17) | 0.48 (+0) | 0.51 (-2) | 0.49 (+0) | 0.35 (-13)

Table 4: Results of our models when only frequency is considered in seed
selection. In parenthesis are the differences comparing to the relevant result
in Table 1.

##### Error Analysis

To understand how it might be possible to improve the method further, we
sampled $100$ incorrect examples from the data set created using the Comb
system for Spanish. We found that $64\%$ of the mistakes were of words tagged
with only $1$ incorrect feature in the bundle. This suggests that a fine-
grained algorithmic approach, that will tackle relations between individual
features rather than complete sets, might do better in this regard.

We examined the morpho-phonological patterns that appear in the outputs of
both Comb and Orth for Finnish to better assess the reason for the gap in
performance between them. We found that the data provided by Comb contains
more examples for alternation unattested in the seed data comparing to Orth.
For example, the data from Comb contained $19$ examples for the nt$\sim$nn
alternation and $2$ examples for rt$\sim$rr, while the data from Orth
contained no examples for both. For comparison, both datasets contained over
$100$ examples for the attested t$\sim$tt alternation. We conclude that while
both methods find examples for attested morpho-phonological processes, only
Comb can close the gap on unattested processes and provide a more diverse
dataset for better generalization.

##### Seed Selection

As any bootstrapping method, our algorithms results’ may be vulnerable to the
selection of the minimal supervision set. To that end we purposefully aimed
for frequent and diverse selection (Sec. 4.1). To examine the importance of
the selection strategy we altered it to disregard class membership, and we
automatically selected lexemes to maximize only frequency. The results of this
experiment are in Table 4.

The results show that the selection procedure is indeed important as all
different algorithms suffered a loss in performance. It is also evident that
diversity is particularly crucial for the orthographic algorithm that is based
on the different edit scripts and has less evidence when seed examples cover
less classes.

##  5 Related Work

In recent years, neural sequence-to-sequence models have taken the lead in all
forms of morphological tagging and morphological inflection tasks.

Morphological tagging is an analysis task where the input is a complete
sentence, i.e., a sequence of word forms, and the model aims to assign each
word form in-context a morphological signature that consists of its lemma,
part-of-speech (POS) tag, and a set of inflectional features Hajic (2000);
Mueller et al. (2013); Bohnet et al. (2018).

Morphological inflection works in the opposite direction, and may be viewed as
a generation task. Here, forms of a lexeme are generated from one another
given sets of inflectional features of both the input and output. In many
implementations the input form is the lemma, in which case the inflectional
features of the input are not given Faruqui et al. (2016); Aharoni and
Goldberg (2017), and the lemma can be either spelled-out, or inputted as an
index in a dictionary Malouf (2017).131313When inflection is done from a form
other than the lemma, it is sometimes referred to as ”reinflection”.

While most pioneering models for supervised morphological inflection used
statistical models based on finite-state-machines Kaplan and Kay (1994);
Eisner (2002), nowadays neural models for morphological inflections are a lot
more pervasive Cotterell et al. (2016, 2017, 2018) (and they go as back as
Rumelhart and McClelland, 1986).

In the case of unsupervised learning of morphology, a key task is to induce
complete paradigms from unlabled texts. Early works on unsupervised morphology
induction focused on morpheme segmentation for concatenative morphology
Goldsmith (2001); Creutz and Lagus (2002); Narasimhan et al. (2015); Bergmanis
and Goldwater (2017). Notwithstanding, early unsupervised works that are not
limited to concatenative morphology do exist Yarowsky and Wicentowski (2000).

More recent studies on unsupervised morphology include works on knowledge
transfer within a genealogical linguistic family well- to low-resourced
languages Kann et al. (2017, 2020); McCarthy et al. (2019), as well as works
aimed at modifying the approaches for the supervised problem, to allow better
tackling of low resourced scenarios. These include Bergmanis et al. (2017)
that focused on data augmentation, and Anastasopoulos and Neubig (2019) that
modified the model itself with a separate features encoder and introduced the
now commonly-used hallucination method for data augmentation. The state of the
art model for classic low-resourced scenarios, with a diverse but small
dataset, is the transition-based model of Makarov and Clematide (2018). In
addition, some works deal with other low-resourced scenarios and assume no
inflection tables at all, and are focused on paradigm detection/completion in
addition to inflection Dreyer and Eisner (2011); Elsner et al. (2019); Jin et
al. (2020). In contract, our scenario provides the knowledge on the
paradigmatic structure with a small and undiverse supervision set.

Another work that made use of semantics is that of Soricut and Och (2015), who
employed analogies between embedded words to filter candidates affixation
rules. We use embeddings to discover examples in a more general morphological
scenario.

##  6 Conclusions

In this work we propose a realistic minimally-supervised scenario for
morphological inflection, which includes only a handful of labeled inflection
tables as well as a large bulk of unlabeled text. We showed that semantic and
orthographic regularities allow bootstrapping the minimal supervision set to a
large (noisy) labeled data set, by searching for word pairs in the vocabulary
analogous to observed form pairs from the supervision. We demonstrate that
training a neural morphological inflector over the bootstrapped dataset leads
to some non-trivial successes, especially on paradigms of smaller size and on
commonly-used inflections. This contribution is orthogonal and can be applied
in tandem with hallucination approaches. When applied separately, our method
outperforms both hallucination and current state of the art models for low-
resourced settings. In the future we aim to improve performance over larger
paradigms and rarer forms in order to make our method a viable substitute for
the labor-intensive manual annotation for new languages.

## Acknowledgements

We thank Jonathan Berant for the helpful advice and discussion all throughout.
We also thank the audience of the BIU-NLP seminar, the 18th SIGMORPHON
meeting, and the 1st UniMorph Meeting, for comments and discussion. This
research is funded by an ERC-StG Grant 677352 by the European Research
Council, and an ISF grant 1739/26 by the Israeli Science Foundation, for which
we are grateful.

## References

  * Ackerman et al. (2009) Farrell Ackerman, James Blevins, and Robert Malouf. 2009.  _Parts and wholes: Implicative patterns in inflectional paradigms_, pages 54–82.
  * Aharoni and Goldberg (2017) Roee Aharoni and Yoav Goldberg. 2017.  Morphological inflection generation with hard monotonic attention.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2004–2015, Vancouver, Canada. Association for Computational Linguistics.
  * Anastasopoulos and Neubig (2019) Antonios Anastasopoulos and Graham Neubig. 2019.  Pushing the limits of low-resource morphological inflection.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 984–996, Hong Kong, China. Association for Computational Linguistics.
  * Bartis (1998) Imre Bartis. 1998.  The finnish parole corpus.  FIN-CLARIN-konsortio, Nykykielten laitos, Helsingin yliopisto.
  * Bergmanis and Goldwater (2017) Toms Bergmanis and Sharon Goldwater. 2017.  From segmentation to analyses: a probabilistic model for unsupervised morphology induction.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 337–346, Valencia, Spain. Association for Computational Linguistics.
  * Bergmanis et al. (2017) Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017\.  Training data augmentation for low-resource morphological inflection.  In _Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection_ , pages 31–39, Vancouver. Association for Computational Linguistics.
  * Bohnet et al. (2018) Bernd Bohnet, Ryan McDonald, Gonçalo Simões, Daniel Andor, Emily Pitler, and Joshua Maynez. 2018.  Morphosyntactic tagging with a meta-BiLSTM model over context sensitive token encodings.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2642–2652, Melbourne, Australia. Association for Computational Linguistics.
  * Cotterell et al. (2018) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018.  The CoNLL–SIGMORPHON 2018 shared task: Universal morphological reinflection.  In _Proceedings of the CoNLL–SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection_ , pages 1–27, Brussels. Association for Computational Linguistics.
  * Cotterell et al. (2017) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017.  CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages.  In _Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection_ , pages 1–30, Vancouver. Association for Computational Linguistics.
  * Cotterell et al. (2016) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016.  The SIGMORPHON 2016 shared Task—Morphological reinflection.  In _Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 10–22, Berlin, Germany. Association for Computational Linguistics.
  * Creutz and Lagus (2002) Mathias Creutz and Krista Lagus. 2002.  Unsupervised discovery of morphemes.  In _Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning_ , pages 21–30. Association for Computational Linguistics.
  * Dreyer and Eisner (2011) Markus Dreyer and Jason Eisner. 2011.  Discovering morphological paradigms from plain text using a Dirichlet process mixture model.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 616–627, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Eisner (2002) Jason Eisner. 2002.  Parameter estimation for probabilistic finite-state transducers.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 1–8, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Elsner et al. (2019) Micha Elsner, Andrea Sims, Alex Erdmann, Antonio Hernandez, Evan Jaffe, Lifeng Jin, Martha Johnson, Shuan Karim, David King, Luana Nunes, Byung-Doh Oh, Nathan Rasmussen, Cory Shain, Stephanie Antetomaso, Kendra Dickinson, Noah Diewald, Michelle McKenzie, and Symon Stevens-Guille. 2019.  Modeling morphological learning, typology, and change: What can the neural sequence-to-sequence framework contribute? _Journal of Language Modelling_ , 7.
  * Erdmann et al. (2020) Alexander Erdmann, Micha Elsner, Shijie Wu, Ryan Cotterell, and Nizar Habash. 2020\.  The paradigm discovery problem.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7778–7790, Online. Association for Computational Linguistics.
  * Faruqui et al. (2016) Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016.  Morphological inflection generation using character sequence to sequence learning.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 634–643, San Diego, California. Association for Computational Linguistics.
  * Firth (1957) J. R. Firth. 1957.  _A synopsis of linguistic theory_.  The Philological Society, Oxford.
  * Goldsmith (2001) John Goldsmith. 2001.  Unsupervised learning of the morphology of a natural language.  _Computational Linguistics_ , 27(2):153–198.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Hajic (2000) Jan Hajic. 2000.  Morphological tagging: Data vs. dictionaries.  In _1st Meeting of the North American Chapter of the Association for Computational Linguistics_.
  * Jin et al. (2020) Huiming Jin, Liwei Cai, Yihui Peng, Chen Xia, Arya McCarthy, and Katharina Kann. 2020.  Unsupervised morphological paradigm completion.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6696–6707, Online. Association for Computational Linguistics.
  * Kann et al. (2020) Katharina Kann, Samuel R Bowman, and Kyunghyun Cho. 2020.  Learning to learn morphological inflection for resource-poor languages.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34(5), pages 8058–8065.
  * Kann et al. (2017) Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2017.  One-shot neural cross-lingual transfer for paradigm completion.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1993–2003, Vancouver, Canada. Association for Computational Linguistics.
  * Kaplan and Kay (1994) Ronald M Kaplan and Martin Kay. 1994.  Regular models of phonological rule systems.  _Computational linguistics_ , 20(3):331–378.
  * Kirov et al. (2018) Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018\.  UniMorph 2.0: Universal Morphology.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Lample et al. (2018) Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018.  Word translation without parallel data.  In _International Conference on Learning Representations_.
  * Li et al. (2018) Bofang Li, Aleksandr Drozd, Tao Liu, and Xiaoyong Du. 2018.  Subword-level composition functions for learning word embeddings.  In _Proceedings of the Second Workshop on Subword/Character LEvel Models_ , pages 38–48, New Orleans. Association for Computational Linguistics.
  * Makarov and Clematide (2018) Peter Makarov and Simon Clematide. 2018.  Neural transition-based string transduction for limited-resource setting in morphology.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 83–93, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Malouf (2017) Robert Malouf. 2017.  Abstractive morphological learning with a recurrent neural network.  _Morphology_ , 27(4):431–458.
  * McCarthy et al. (2019) Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019.  The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection.  In _Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 229–244, Florence, Italy. Association for Computational Linguistics.
  * Mueller et al. (2013) Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013.  Efficient higher-order CRFs for morphological tagging.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 322–332, Seattle, Washington, USA. Association for Computational Linguistics.
  * Narasimhan et al. (2015) Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. 2015.  An unsupervised method for uncovering morphological chains.  _Transactions of the Association for Computational Linguistics_ , 3:157–167.
  * Oravecz et al. (2014) Csaba Oravecz, Tamás Váradi, and Bálint Sass. 2014.  The Hungarian Gigaword corpus.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 1719–1723, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Rumelhart and McClelland (1986) David E Rumelhart and James L McClelland. 1986.  On learning the past tenses of english verbs.
  * Silfverberg and Hulden (2018) Miikka Silfverberg and Mans Hulden. 2018.  An encoder-decoder approach to the paradigm cell filling problem.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2883–2889, Brussels, Belgium. Association for Computational Linguistics.
  * Soricut and Och (2015) Radu Soricut and Franz Och. 2015.  Unsupervised morphology induction using word embeddings.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1627–1637, Denver, Colorado. Association for Computational Linguistics.
  * Yarowsky and Wicentowski (2000) David Yarowsky and Richard Wicentowski. 2000.  Minimally supervised morphological analysis by multimodal alignment.  In _Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics_ , pages 207–216, Hong Kong. Association for Computational Linguistics.