# A Cognitive and Machine Learning-Based Software Development Paradigm
Supported by Context

Glaucia Melo, Paulo Alencar, Donald Cowan  David R. Cheriton School of
Computer Science
University of Waterloo
Waterloo, Canada
{gmelo, palencar<EMAIL_ADDRESS>
###### Abstract

Advances in the use of cognitive and machine learning (ML) enabled systems
fuel the quest for novel approaches and tools to support software developers
in executing their tasks. First, as software development is a complex and
dynamic activity, these tasks are highly dependent on the characteristics of
the software project and its context, and developers need comprehensive
support in terms of information and guidance based on the task context.
Second, there is a lack of methods based on conversational-guided agents that
consider cognitive aspects such as paying attention and remembering. Third,
there is also a lack of techniques that make use of historical implicit or
tacit data to infer new knowledge about the project tasks such as related
tasks, task experts, relevant information needed for task completion and
warnings, and navigation aspects of the process such as what tasks to perform
next and optimal task sequencing. Based on these challenges, this paper
introduces a novel paradigm for human-machine software support based on
context, cognitive assistance, and machine learning, and briefly describes
ongoing research activities to realize this paradigm. The research takes
advantage of the synergy among emergent methods provided in context-aware
software processes, cognitive computing such as chatbots, and machine learning
such as recommendation systems. These novel paradigms have the potential to
transform the way software development currently occurs by allowing developers
to receive valuable information and guidance in real-time while they are
participating in projects.

###### Index Terms:

software engineering, context, software process, cognitive assistance, machine
learning, recommendation.

##  I Introduction

Software development (SD) is a complex, dynamic process. Many changes are
expected to occur during a software development process despite what was
planned and what is expected [1, 2, 3, 4]. Often, the approach that each
software developer takes when performing a task is highly dependent on the
characteristics of the software project and its context [2, 5]. This context
can include technologies (e.g., Eclipse, Jenkins, Java), methodologies (e.g.,
Scrum), processes (e.g., code, test, deploy), background documentation (e.g.,
Stack Overflow, API Tutorials), environment (e.g., test, production) and even
personal preferences (e.g., Eclipse IDE or VSCode for coding). Given the huge
number of possible variations in context, it is cumbersome for developers to
remember everything they must do, and how context can influence the way they
perform their tasks. For example, in project x, developed in Java with Scrum,
two test rounds need to be successful before an automated process deploys
project x into production. In project y, developed in Python, developers test
the code locally and deploy the code manually by copying some files to a
server. Questions developers have include: How long does a sprint take in
Project x? Which team/person is responsible for testing each round of project
x? Was the test in project x successful? What is the server address to be used
for deployment in project y? To which folder should I copy the files? Which
files should I copy? and Do I need to keep a version of the files on the
server?

Besides the amount of contextual information that remains unmanaged,
developers have a high cognitive load remembering the steps for various task
workflows. Current efforts to solve these problems are presented through a
variety of approaches. An extensive body of research addresses context in
software development and includes discussions on many topics including: using
context to improve developer productivity [6], providing information to
developers [7, 8], impacting a developer’s work through context switching [1].
All approaches recognize the rich context inherent in software development and
that this context is not always explicit. It has even been argued that context
is fundamental to software engineering practices [2].

Another paper has focused on communicating the perceived context with a tool
such as a conversational agent [9]. It is essential to study means to have
developers focusing on the creative part of the development, other than
investing time in tasks that could be automatically delivered to them through
smart systems that understand and use the developer’s current context. There
is also work that intends to support the cognitive load of knowledge workers
[10, 11]. In all, these papers propose several approaches to support software
developers in their daily work activities.

Thus, SD requires comprehensive support in terms of information and guidance
based on context during task execution [9, 2, 1]. As a process dependent on
knowledge workers [12], SD lacks supportive methods based on conversational-
guided agents that account foesr cognitive tasks such as paying attention,
remembering, and maintaining mental maps of the software processes. Current
practices of developing software also lack techniques that make use of
historical implicit or tacit data to infer new knowledge about the project
tasks and navigation aspects of the process. While similar tools and solutions
provide comparable assistance [6, 13, 14] based on software information, our
proposal considers software process information and has a machine learning
component. Therefore, we argue that novel approaches should take advantage of
the synergy among emerging methods in context-aware software processes,
cognitive assistance such as chatbots and recommendation systems based on
machine learning (ML).

Given these challenges, we provoke a discussion by asking: How can software
development be advanced by introducing a new paradigm to realize human-machine
software development cooperation based on context, cognitive assistance and
machine learning?

SD has already been supported with automated tools [15, 16], and with
automatically generated code, commits, and built chatbots [17, 9]. In the
future, developer’s knowledge and ubiquitous context will be integrated into
the development environment, complementing the current state-of-the-art with
very effective, timely and supportive relevant recommendations for developers.

This proposed discussion intends to stimulate thinking about the creation of
tools and procedures that can advance software development as it relates to
software developers and, on a larger scope, companies. There is a direct
connection to work being done by large software companies working at the
forefront of research and practice involving novel (semi-)automated methods to
support the development of software applications while improving software
developers’ efficiency and productivity. Shaping software development is
critical as software systems have become the backbone of much of today’s
technology and society’s functions. Working remotely has become a new reality.
Consequently, approaches and tools that help to facilitate software
development have become even more essential. Working with software development
and its intrinsic implicit context is essential; therefore, we argue there is
still the need to improve the machine-developer interaction, instead of purely
automating software processes. We discuss the future of this proposed paradigm
more in Section III.

##  II Motivating Example

It is known that deployment is a challenging task during software development
[18], and so, we present an example of how changes in the steps that are
executed during deployment vary in different contexts. Gabi is a software
developer who has been programming in Java for nine years. She has been
recently working on project X, a new project for the company. When there is a
new project, Gabi needs to create a minimal viable product (MVP) to show her
clients. She deploys the software locally, using a container tool such as
Docker and manually uploads the project to a web server. She also reboots the
server manually after each deployment, so changes are effective immediately.
This way is faster, and she does not have to configure a job or a server to
generate a deployment automatically, which would cause the clients to wait
much longer for the MVP. In contrast, there is project Y, a mature and huge
project in the company. When a version of the software in Project Y must go to
production, all Gabi does is to commit the code from the local to the shared
code repository. Then, the continuous integration pipeline in a Jenkins server
will take care of the other steps, which are checking out the code, building
the project, uploading the package on the server, and rebooting the server. In
theory, the steps are the same, but because the projects and environments are
different, Gabi’s work is different. This means that in the second case, the
solution should be expecting project information or project phase information
such as MVP or production phase, which defines how the form of the deployment.
Gabi, who has been working on project Y for years may forget she needs to
deploy Project X manually or that the server must be manually rebooted. Having
an intelligent cognitive assistant that could support Gabi in many ways by,
for example, giving her the necessary information such as the server address,
her login information, the folder to which the files can be copied, the link
to the deployed system, and the steps to deploy project Y.

Imagine Gabi is interacting with a chatbot and does not remember exactly how
to deploy a new project, since she is not always working on new projects. In
Figure 1, we present a conversation between Gabi and the chatbot during
deployment. To reach the desired level of support, the chatbot should know her
context, which includes the server address and folder of the development
environment, and, based on the history of tasks previously executed in that
context, the chatbot should be able to learn and recommend the task sequence
that Gabi needs to follow and provide the information required to execute the
tasks.

Figure 1: Illustrative chatbot interaction.

##  III The Expected Future

In this paper, we envision a new paradigm for chatbot use that knows the
software development context and relies on machine learning techniques to
support developers when executing their tasks. The purpose of this research
is, therefore, to capture the tacit or implicit context and feed it back in a
useful way such as making recommendations to developers. Processes based on
machine learning and communicated through a chatbot should lower the cognitive
load of developers, provide context-aware and real-time support for task
execution, and guide developers through the development steps such as
deployment. Figure 2 illustrates the envisioned architecture of the solution.

Figure 2: Prototype of conversational channel connecting developer, context
and process.

This novel paradigm can potentially transform the way software development is
currently undertaken by allowing developers to receive valuable information
and guidance in real-time while they are developing their projects. In
contrast with the way developers interact with existing IDEs, the proposed
paradigm pro-actively provides developers with information and guidance they
need, where, when and how they need it. As a consequence, developers will be
supported in their interactions, productivity, and decision making.

We anticipate this approach to be deployed in many different software
development scenarios, with tools and procedures that explore the reuse of
information on software development, providing support and automation to
recurrent tasks. Hence, developers can focus on the creative aspects, use of
programming languages, data structures, other product-related concerns, and
user-focused solutions. The proposed paradigm is one step further towards
bringing more knowledge to developers, both experienced and novice, during
specific software development activities. This paradigm builds upon our
previous work on reusing relevant information to support developers in
executing their tasks [19] and the relevance of context in software
development projects [20].

##  IV Making this Future Possible

To realize a human-machine software collaboration paradigm based on context,
cognitive support, and machine learning, requires performing the following
research activities.

Research on Software Development Context. We believe it is essential to
understand and capture context for software tasks so that adaptive context
relevant to software development could be leveraged. Being able to handle this
context would allow developers to focus on creative tasks rather than on how
to execute a specific procedural task or wonder what should be done next to
uncover specific information. The volume of information to be handled, as well
as the speed at which such information will arrive, is often massive and
rapid. Moreover, the variability of context formats is also an aspect to be
considered. Examples of relevant contextual information are the next artifact
to be edited or read, an API tutorial, a code snippet, or knowledge from
another developer. We strongly believe that the nature of capturing the
different contexts and presenting the tasks related to that context is already
a significant contribution to the field.

Relevant questions include: How can we deepen the use of context to guide
developers in real-time? Which context should be leveraged to guide developers
in their tasks? How can we better capture and reuse context information in
software development? How can massive contextual information be stored so that
it can be accessed and used to generate knowledge for software developers?

Environment-Developer Interaction. This approach is within the scope of
providing means for developers to interact with a system that is supposed to
support them during development. This support is through a conversation where
a chatbot supported by a context model should be aware of the developer’s set
of tasks. This means the chatbot should know both the workflow (process
execution and project characteristics) and the contextual information of the
project, as well as be prepared to capture information from developers and
other resources providing answers. Bradley and colleagues [9] have considered
a context-model for supporting conversational developer assistants that uses
the context elements needed to support workflow involving a distributed
version control system. In contrast, the focus of our approach is on guiding
developers in the steps they must perform throughout the development project,
considering different cognitive information and its consequences for the
project. Automation of tasks could eventually happen as we understand the
process and the influence of contextual and cognitive utterances and
differences. The goal is to automate the communication so that developers do
not have to rely on their memories, mental maps or searching huge sets of
documentation. The chatbot would act upon a context model and would have
embedded machine learning and history information to provide results to the
developers. This integration would transform the chatbot into an intelligent
tool. The solution is also intended to be non-invasive, relying on the
implementation of techniques such as aggregated data or anonymization. With
respect to the feasibility of implementation, our ongoing work on process-
aware conversational agents has demonstrated that it is possible to integrate
a basic context model into a chatbot tool such as RASA. We have also
implemented a preliminary integration of the chatbot with a workflow machine
called Camunda, allowing the chatbot to receive process execution information.

Relevant questions include: How can we establish a communication channel
between developers and their environment? How can a chatbot be effective in
supporting developers? In which ways are developers willing to accept this new
technology?

Knowledge-Intensive Process Guidance. Knowledge workers such as software
developers rely on their minds and creativity to implement software solutions.
Providing smart solutions when building software is expected, so developers
must worry about following patterns, processes and adjusting to project needs.
This information is usually implicit or tacit and in developers’ memories.
Automated task guidance or task navigation support should improve developers’
ability to work more efficiently. This system would count on intelligence from
machine learning and project history to recognize the context in which
developers are working and suggest the next steps, according to their context.
Development tasks are knowledge guided, capturing the context is essential.
Providing feedback to developers with recommendations should be part of the
solution. Once the context is captured and understood, developers have a way
to take advantage of this context through the chatbot. A method for process
navigation recommendations based on machine-learning should be provided. The
method may include training a machine-learning model by at least processing
training data (context and process) with the machine-learning model. The
training data may include records of the executed process with the current
context at the time of execution. Correlations between context and the
executed process shall be done so inferences of the next steps can be provided
(recommended) to developers.

Relevant questions include: How to leverage process information to guide
developers in what they must do? What types of recommendations related to the
software process can improve SD? How can developers take advantage of guidance
recommendations? How can chatbot inferences be integrated with ML inferences
to produce rich feedback to developers? How could we maximize the guidance of
developers through process and context exploration?

Experimental studies. Qualitative and quantitative studies to demonstrate the
feasibility of the proposed approaches as well as the implementation in
software companies. Application areas of interest for the studies include
deployment, testing, version control and managing issues or tasks.

Relevant questions include: How can we verify the acceptance of a new
cognitive assistance-based way of development by software developers? In which
types of interactions would developers be mostly interested? How can we have
confidence this idea will do what developers want?

##  V Conclusion

In this paper, we present and discuss a novel paradigm to improve the work of
software developers, by providing contextual information about the tasks they
are performing through cognitive and intelligent support. It employs capturing
the implicit or tacit context and feeding it back in a useful way through
recommendations to developers in real-time as they are executing the software
project. The novelty of the paradigm arises from the approaches and tools used
to capture and recommend the developers’ context on-the-fly, considering
different contexts. The findings will be of interest since their use should
have several advantages including less time to develop software, less effort
to share knowledge among team members, enhanced collaboration, application of
collective wisdom, knowledge transfer from experts to novice and many other
useful contributions. As broader implications of the results, we believe that
the impact of the proposed research will contribute to facilitating the
development of new avenues in software research as well as support improved
ways to develop software, a critical area that is in high demand and has
enormous future growth potential. This is the first research program where the
combination of three different pillars (context, chatbots and machine learning
for process navigation) has been exploited to predict appropriate
recommendations during software development.

## References

  * [1] A. N. Meyer, L. E. Barton, G. C. Murphy, T. Zimmermann, and T. Fritz, “The work life of developers: Activities, switches and perceived productivity,” _IEEE Transactions on Software Engineering_ , vol. 43, no. 12, pp. 1178–1193, 2017. [Online]. Available: http://ieeexplore.ieee.org/document/7829407
  * [2] G. Murphy, “Beyond integrated development environments: adding context to software development,” in _Proceedings of the 41st International Conference on Software Engineering_. IEEE Press, 2019, pp. 73–76.
  * [3] C. Di Ciccio, A. Marrella, and A. Russo, “Knowledge-intensive processes: Characteristics, requirements and analysis of contemporary approaches,” _Journal on Data Semantics_ , vol. 4, no. 1, pp. 29–57, Mar 2015.
  * [4] N. Gronau, C. Müller, and R. Korf, “Kmdl-capturing, analysing and improving knowledge-intensive business processes.” _J. UCS_ , vol. 11, no. 4, pp. 452–472, 2005.
  * [5] G. C. Murphy, “The need for context in software engineering (ieee cs harlan mills award keynote),” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , ser. ASE 2018\. New York, NY, USA: Association for Computing Machinery, 2018, p. 5. [Online]. Available: https://doi.org/10.1145/3238147.3241987
  * [6] M. Kersten and G. C. Murphy, “Using task context to improve programmer productivity,” in _Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , ser. SIGSOFT ’06/FSE-14. ACM, 2006, p. 1–11. [Online]. Available: http://doi.acm.org/10.1145/1181775.1181777
  * [7] M. Gasparic, G. C. Murphy, and F. Ricci, “A context model for ide-based recommendation systems,” _Journal of Systems and Software_ , vol. 128, p. 200–219, Jun 2017.
  * [8] R. Holmes and G. C. Murphy, “Using structural context to recommend source code examples,” in _Proceedings of the 27th International Conference on Software Engineering_ , ser. ICSE ’05. ACM, 2005, p. 117–125, event-place: St. Louis, MO, USA. [Online]. Available: http://doi.acm.org/10.1145/1062455.1062491
  * [9] N. Bradley, T. Fritz, and R. Holmes, “Context-aware conversational developer assistants,” in _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 2018, pp. 993–1003.
  * [10] A. M. Lima, R. Q. Reis, and C. A. L. Reis, “Empirical evidence of factors influencing project context in distributed software projects,” in _2015 IEEE/ACM 2nd International Workshop on Context for Software Development_ , 2015, pp. 6–7.
  * [11] B. Biegel, S. Baltes, I. Scarpellini, and S. Diehl, “Code basket: Making developers’ mental model visible and explorable,” in _2015 IEEE/ACM 2nd International Workshop on Context for Software Development_ , 2015, pp. 20–24.
  * [12] A. Kidd, “The marks are on the knowledge worker,” in _Proceedings of the SIGCHI conference on Human factors in computing systems_ , 1994, pp. 186–191.
  * [13] M. Kersten and G. C. Murphy, “Mylar: a degree-of-interest model for ides,” in _Proceedings of the 4th international conference on Aspect-oriented software development_. ACM, 2005, pp. 159–168.
  * [14] D. ČubraniĆ, G. C. Murphy, J. Singer, and K. S. Booth, “Learning from project history: A case study for software development,” in _Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work_ , ser. CSCW ’04. New York, NY, USA: ACM, 2004, pp. 82–91. [Online]. Available: http://doi.acm.org/10.1145/1031607.1031622
  * [15] D. Čubranić and G. C. Murphy, “Hipikat: Recommending pertinent software development artifacts,” in _Proceedings of the 25th International Conference on Software Engineering_ , ser. ICSE ’03. IEEE Computer Society, 2003, p. 408–418, event-place: Portland, Oregon. [Online]. Available: http://dl.acm.org/citation.cfm?id=776816.776866
  * [16] L. Ponzanelli, G. Bavota, M. D. Penta, R. Oliveto, and M. Lanza, “Prompter: A self-confident recommender system,” in _2014 IEEE International Conference on Software Maintenance and Evolution_ , Sep 2014, p. 577–580.
  * [17] L. Cai, H. Wang, B. Xu, Q. Huang, X. Xia, D. Lo, and Z. Xing, “Answerbot: an answer summary generation tool based on stack overflow,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019, pp. 1134–1138.
  * [18] A. Carzaniga, A. Fuggetta, R. S. Hall, D. Heimbigner, A. Van Der Hoek, and A. L. Wolf, “A characterization framework for software deployment technologies,” Colorado State Univ Fort Collins Dept of Computer Science, Tech. Rep., 1998.
  * [19] G. Melo, T. Oliveira, P. Alencar, and D. Cowan, “Retrieving curated stack overflow posts from project task similarities,” in _Proceedings of the 31st International Conference on Software Engineering & Knowledge Engineering_, 2019, pp. 415–418.
  * [20] G. Melo, P. Alencar, and D. Cowan, “Context-augmented software development in traditional and big data projects: Literature review and preliminary framework,” in _2019 IEEE International Conference on Big Data (Big Data)_ , 2019, pp. 3449–3457.


# Factuality Detection using Machine Translation - a Use Case for German
Clinical Text

Mohammed Bin Sumait, Aleksandra Gabryszak, Leonhard Hennig  Roland Roller
German Research Center for Artificial Intelligence (DFKI)
Speech and Language Technology Lab
<EMAIL_ADDRESS>
###### Abstract

Factuality can play an important role when automatically processing clinical
text, as it makes a difference if particular symptoms are explicitly not
present, possibly present, not mentioned, or affirmed. In most cases, a
sufficient number of examples is necessary to handle such phenomena in a
supervised machine learning setting. However, as clinical text might contain
sensitive information, data cannot be easily shared. In the context of
factuality detection, this work presents a simple solution using machine
translation to translate English data to German to train a transformer-based
factuality detection model.

##  1 Introduction

Factuality refers to the concept that a speaker can present statements about
world events with varying degrees of uncertainty as to whether they happened.
Factuality reflects, for instance, if an event is affirmed, negated, or
uncertain. In the medical domain, detecting if symptoms or diseases are
signaled as present, not present, possibly or doubtfully present, and
therefore uncertain is essential. Detecting factuality is challenging since it
can be expressed by very different linguistic categories (e.g. verbs, nouns,
adjectives, adverbs), plus it must be taken into account how they are embedded
in a sentence Rudinger et al. (2018a). Additionally, linguistic factuality
cues can be very domain-specific, so the availability of relevant datasets is
essential.

Classical supervised machine learning requires training data, and, at the same
time, most existing datasets are published in English. In addition, clinical
text contains sensitive patient data, which often makes it difficult to share
due to ethical and legal aspects. Although the situation has slowly changed
regarding the availability of German clinical text resources Modersohn et al.
(2022), many other languages suffer a similar situation. Conversely, the
quality of machine translation has significantly improved in the last decade,
also regarding the translation of biomedical text/publications, including
clinical case reports Neves et al. (2022). For this reason, this work explores
the usage of machine translation to create (translated) text resources for
factuality detection in German clinical text.

Clinical notes are short text documents written by physicians during or
shortly after the treatment of a patient. In general, this kind of text
contains much valuable information about the current health condition, as well
as treatment, of the patient. They differ from biomedical publications and
clinical case reports, as notes are often written under time pressure with a
high information density, a telegraphic writing style, non-standardized
abbreviations, colloquial errors, and misspellings. Therefore, it is unclear
if current machine translation systems can handle this text, considering that
data might contain sensitive information and should not be shared with a third
party outside the hospital.

This work makes the following contributions: 1) We successfully use a local
machine translation to train a model for factuality detection on German
clinical text. 2) Our model outperforms the only ‘competitor’ NegEx, and 3)
will be published as open access
model111https://huggingface.co/binsumait/factual-med-bert-de. Finally, 4) for
those interested in NegEx, we release it as a modular PyPI package with a few
important fixes222https://github.com/DFKI-NLP/pynegex and also propose
improvement suggestions to the used trigger sets.

##  2 Methods and Data

The idea of this work is based on the usage of machine translation to generate
a German corpus to train a classifier dealing with factuality in clinical
text. In the following, we outline the approach, the necessary methods, and
the dataset used.

Factuality |  English |  German translation
---|---|---
affirmed |  Clinically, a <E>severe neuropsychological syndrome</E> was found when the patient was taken over. |  Klinisch fand sich bei Übernahme des Patienten in <E>schweres neuropsychologisches Syndrom</E>.
negation |  Patient denies <E>headache</E>. |  Patient verneint <E>Kopfschmerzen</E>.
possible |  Thus, a <E>tumour</E> cannot be ruled out. |  Ein <E>Tumor</E> kann daher nicht ausgeschlossen werden.
Table 1: Example sentences with target entities, factuality label, and
possible translations.

###  2.1 Factuality Detection

In literature, (medical) factuality detection is often reduced to a simple
classification. Given a sentence and an entity, the task is to define the
factuality of the entity in the given context. In most cases, the entity of
interest is a symptom or medical condition. Most related work targets the
three classes affirmed, negated and possible. However, as simple as this
sounds, factuality cannot always be easily mapped to those few classes.

One of the most prominent tools to deal with factuality in the medical text is
NegEx Chapman et al. (2001), a rule-based approach with pre-defined regular
expressions, so-called triggers, and can detect the three aforementioned
factuality classes. It achieves, particularly in the context of negations,
quite good results on clinical text. Hedges instead offer more possibilities
for how they are described, therefore achieving a much lower performance.
Initially, it was developed for English, but over the years, it has also been
translated into other languages, such as Spanish or Swedish Cotik et al.
(2016b); Chapman et al. (2013). In addition, many alternative (machine
learning) solutions have been published in the last two decades. We refer to
the overview by Khandelwal and Sawant (2019) for more details. For German,
however, only one negation detection exists, which relies on the NegEx
solution and uses a set of translated trigger words (English to German) Cotik
et al. (2016a).

###  2.2 Data

In the following, we briefly introduce the data used for this work. First, we
present i2b2, which has been used for machine translation and to train our
model. In addition, we later test our model on additional German data, namely
Ex4CDS and NegEx-Ger, and in the appendix also BRONCO150.

The 2010 i2b2/VA data Uzuner et al. (2011) consists of English medical text
and includes three tasks - extraction of concepts, assertions identification,
and relation detection. In this work, we focus on the assertion task. Overall
a total of six assertion types were considered, namely present, absent,
possible, conditional, hypothetical and not associated with the patient.
However, this work focused only on the first three labels, as only those are
considered within NegEx. i2b2 data is translated to German to train a German
machine learning model.

Ex4CDS Roller et al. (2022) is a small dataset of physicians’ notes containing
explanations in the context of clinical decision support. The notes are
written in German and include various annotation layers, including factuality.
As the data includes multiple factuality labels, we reduced the labels to our
three target labels, mapping possible-future and unlikely to possible, and
minor to affirmed. As target entities, we consider only sentences containing
medical-conditions.

NegEx-Ger is a small dataset consisting of sentences taken from clinical notes
and discharge summaries and has been used initially to evaluate the German
NegEx version in Cotik et al. (2016a). For our use case, the data has been
used for testing, and for this, we merged the sentences of both clinical text
types. However, the number of sentences containing the possible label is small
(22 for discharge summaries and 4 for clinical notes).

###  2.3 Translation Approach

For our proposed idea, two aspects need to be considered: First, we aim at a
solution that could be applied to sensitive data. Therefore, the machine
translation component must run locally. This means we cannot rely on the
variety of existing state-of-the-art online approaches. Second, as we define
factuality as a classification problem with a given sentence (context) and an
entity, our translations need to keep track of the target entity within a
sentence. A simple example is given in Table 1, which shows an English
sentence with a target entity ‘headache’ and the label ‘negation’. The German
translation needs to keep the focus on the target entity.

In this work, we rely on TransIns Steffen and van Genabith (2021), an open-
source machine translation that can be installed locally. TransIns is built on
MarianNMT Junczys-Dowmunt et al. (2018) framework and enables translating
texts with an embedded markup language. Specifically, we translate sentences
with tagged entities, as shown in Table 1.

A manual inspection revealed multiple problems with the translations: In some
cases (roughly 40% of the issues), translations were corrupt as they contained
cryptic and/or repetitive text sequences that were foreign from the original
text. Such noise patterns could partially or entirely affect the target texts’
context. Or, in very few cases (only 4%), no translation output could be
produced. In the rest of the cases, the markup no longer included the target
entity. In any way, such output has been discarded from the data, and we
resulted in 18,297 data points (initially 18,397), which we used to train and
evaluate our machine learning model.

##  3 Experiments and Results

We conduct three different experiments - starting with the English i2b2 data,
we use Bio+Discharge Summary BERT Alsentzer et al. (2019) and compare the
results to NegEx. Similar experiments have also been conducted in other
papers. However, in our case, those results serve as a comparison. Thus, the
model is not optimized to achieve the best possible performance. Next, we
train German-MedBERT Shrestha (2021) on the translated i2b2 data and compare
the results to the performance of the German NegEx implementation. Finally, we
apply both German factuality approaches to different German medical texts to
determine how well the models perform in a more realistic setup.

| NegEx | BERT-based
---|---|---
Label | Prec | Rec | F1 | Prec | Rec | F1
E Affirmed | 0.88 | 0.97 | 0.93 | 0.97 | 0.99 | 0.98
N Negated | 0.89 | 0.79 | 0.84 | 0.98 | 0.97 | 0.97
G Possible | 0.79 | 0.04 | 0.08 | 0.85 | 0.64 | 0.73
G Affirmed | 0.84 | 0.96 | 0.90 | 0.96 | 0.98 | 0.97
E Negated | 0.83 | 0.65 | 0.73 | 0.95 | 0.93 | 0.94
R Possible | 0.28 | 0.02 | 0.04 | 0.80 | 0.64 | 0.71
Table 2: Performance results between NegEx baselines and BERT-based models on
the original English i2b2 dataset (upper part) and German translation (lower
part).

The results of the first two experiments are presented in Table 2 and show
various interesting findings: Firstly, NegEx provides impressive results on
the affirmed label, good results for negations, and unsatisfying results for
the possible label. Moreover, on both datasets, English and German, the BERT-
based model outperforms NegEx, on all scores. Additionally, results on the
English dataset are always higher than those on the translated dataset. This
might be unsurprising as data quality decreases. Finally, the table shows that
BERT-based models show a substantial increase in performance for the possible
label.

| NegEx | BERT-based
---|---|---
Label | Prec | Rec | F1 | Prec | Rec | F1
N Affirmed | 0.96 | 0.94 | 0.95 | 0.97 | 0.96 | 0.96
E Negated | 0.93 | 0.96 | 0.95 | 0.97 | 0.98 | 0.97
G Possible | 0.46 | 0.50 | 0.48 | 0.50 | 0.50 | 0.50
E Affirmed | 0.85 | 0.88 | 0.86 | 0.88 | 0.92 | 0.90
X Negated | 0.66 | 0.89 | 0.76 | 0.86 | 0.95 | 0.90
4 Possible | 0.50 | 0.18 | 0.26 | 0.61 | 0.38 | 0.47
Table 3: Performance results on different German medical text sources, namely
the original German NegEx (upper part), and Ex4CDS dataset (lower part).

Table 3 presents the performance of the NegEx and the BERT-based model on two
German datasets. In the upper part of the table, the results on NegEx-Ger are
presented and the results on Ex4CDS are in the lower part. Similarly, as on
the translated i2b2 dataset in Table 2, the machine learning model outperforms
NegEx. However, this time the performance gain is not so strong anymore. The
NegEx-Ger is small and relatively homogeneous (regarding the variety of
negations), and NegEx already performs well on the negations. Therefore the
machine learning model achieves only a performance boost of two points in F1.
In case of possible, the number of examples might be too small to see the
benefit of the ML model.

On Ex4CDS data, NegEx already struggles with negated (0.76) and performs low
in the case of possible (0.26) - although the results are much better in
comparison to the results on i2b2 (English and German). Here, the machine
learning model leads to a performance boost of 14 points for negated and 21
points for possible.

##  4 Analysis and Discussion

Our results indicate that we can successfully apply machine translation to
generate a German clinical dataset to train a machine learning model with.
Most notably, this model can outperform NegEx, which partially already
provides satisfying results. While it is important that a negation detection
tool for German clinical text needs to run within a hospital infrastructure,
it might be questionable if BERT-based approaches might be the right solution,
as it requires much more hardware resources than the simple NegEx solution.
This is supported by the results on NegEx-Ger, in which the BERT achieves only
a minor performance gain. However, as this data is small and homogeneous, the
results on Ex4CDS affirm the usage of machine learning, as we achieve a
notable performance gain. Note, information about the frequency of each label
in the test data is provided in the appendix. As our BERT model was trained on
potential suboptimal translations, we analyse some errors in more detail in
the following.

###  4.1 Linguistic Error Analysis

Our analysis focuses on the prediction errors caused by the translation or by
differences in the features of the German and English language. Table 7
contains full-text examples illustrating the issues described below.

In various cases, a factuality cue was completely missing in the translation,
or the sense of the cue was not preserved (e.g., to rule out was translated
with Vorschriften instead of ausschließen). In those cases, NegEx and BERT
labeled the instances wrongly as affirmations.

In other cases, we observe that the factuality cues are outside of the
original data’s entities but in the translation they are placed within the
entity markup. That is often correlated with the prediction changing from
negation or possible to affirmation. For example, both NegEx and BERT
correctly recognized the negated assertion of the original phrase did not
notice [any blood], whereas both German models consider the translation
bemerkte [kein Blut] as affirmed in which the negation cue (not / kein) became
part of the entity.

For NegEx, a further problem are missing factuality cues in the trigger list.
For example, it systematically does not recognize the cue verleugnen (one of
the possible translations of the word deny, which is included in the English
NegEx). Additionally, some problems with factuality cues are specific to the
German language and require additional handling: (a) German compounds must be
written as one word; unfortunately, German NegEx cannot handle cases when a
compound consists of words referring to a medical problem and its negation
(e.g. schmerzfrei / pain free), since it seems not to recognize a factuality
cue if it is not written as a separate phrase, (b) cues with umlauts in text
such as aufgelöst seem not to be recognized, because the umlauts are encoded
as oe in the German trigger list, (c) missing possible word orders of
factuality phrases (e.g. word order might depend on the embedding syntactic
structure; e.g. wurde ausgeschlossen vs. ausgeschlossen wurde in a main vs.
subordinate clause).

##  5 Related Work

Machine Translation for Cross-lingual Learning MT is a popular approach to
address the lack of data in cross-lingual learning Hu et al. (2020);
Yarmohammadi et al. (2021). There are two basic options - translating target
language data to a well-resourced source language at inference time and
applying a model trained in the source language Asai et al. (2018); Cui et al.
(2019), or translating source language training data to the target language,
while also projecting any annotations required for training, and then training
a model in the target language Khalil et al. (2019); Kolluru et al. (2022);
Frei and Kramer (2023). Both approaches depend on the quality of the MT
system, with translated data potentially suffering from translation or
alignment errors Aminian et al. (2017); Ozaki et al. (2021). While the quality
of machine translation for health-related texts has significantly improved
Neves et al. (2022), using MT in the clinical domain remains underexplored,
with very few exceptions Frei and Kramer (2023).

Factuality Detection Previous research focused mainly on assigning factuality
values to events and often framed this task as a multiclass classification
problem over a fixed set of uncertainty categories Rudinger et al. (2018b);
Zerva (2019); Pouran Ben Veyseh et al. (2019); Qian et al. (2019); Bijl de
Vroe et al. (2021); Vasilakes et al. (2022). In the biomedical/clinical
domain, Uzuner et al. (2011) present the i2b2 dataset for assertion
classification, and Thompson et al. (2011) introduce the Genia-MK corpus,
where biomedical relations have been annotated with uncertainty values. van
Aken et al. (2021) release factuality annotation of 5000 data points sourced
from MIMIC. Kilicoglu et al. (2017) introduce a dataset of PubMed abstracts
with seven factuality values, and find that a rule-based model is more
effective than a supervised machine learning model on this dataset.

##  6 Conclusion

This work presented a machine learning-based factuality detection for German
clinical text. The model was trained on translated i2b2 data and tested, first
on the translations and then on other German datasets and outperformed an
existing method for German, NegEx. The simple machine translation approach
might interest the Non-English clinical text processing community. The model
will be made publicly available.

## Ethical Considerations

We use the original datasets “as is”. Our translations of i2b2 thus reflect
any biases of the original dataset and its construction process, as well as
biases of the MT models (e.g., rendering gender-neutral English nouns to
gendered nouns in German). We use BERT-based PLMs in our experiments, which
were pretrained on a large variety of medical source data. Our models may have
inherited biases from these pretraining corpora.

Since medical data is highly sensitive with respect to patient-related
information, all datasets used in our work are anonymized. The authors of the
original datasets Uzuner et al. (2011); Roller et al. (2022) have stated
various measures that prevent collecting sensitive, patient-related data.
Therefore, we rule out the possible risk of sensitive content in the data.

## Limitations

A key limitation of this work is the dependence on a machine translation
system to get high-quality translations and annotation projections of the
source language dataset. Depending on the availability of language resources
and the quality of the MT model, the translations we use for training and
evaluation may be inaccurate, or be affected by translation noise, possibly
leading to overly optimistic estimates of model performance. In addition,
since the annotation projection is completely automatic, any alignment errors
of the MT system will yield inaccurate instances in the target language.

## Acknowledgements

This research was supported by the German Research Foundation (DFG, Deutsche
Forschungsgemeinschaft) through the project KEEPHA (442445488) and the German
Federal Ministry of Education and Research (BMBF) through the projects KIBATIN
(16SV9040) and CORA4NLP (01IW20010).

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical bert embeddings.  _arXiv preprint arXiv:1904.03323_.
  * Aminian et al. (2017) Maryam Aminian, Mohammad Sadegh Rasooli, and Mona Diab. 2017.  Transferring semantic roles using translation and syntactic information.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 13–19, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Asai et al. (2018) Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018.  Multilingual extractive reading comprehension by runtime machine translation.  _ArXiv_ , abs/1809.03275.
  * Bijl de Vroe et al. (2021) Sander Bijl de Vroe, Liane Guillou, Miloš Stanojević, Nick McKenna, and Mark Steedman. 2021.  Modality and negation in event extraction.  In _Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021)_ , pages 31–42, Online. Association for Computational Linguistics.
  * Chapman et al. (2001) Wendy W Chapman, Will Bridewell, Paul Hanbury, Gregory F Cooper, and Bruce G Buchanan. 2001.  A simple algorithm for identifying negated findings and diseases in discharge summaries.  _Journal of biomedical informatics_ , 34(5):301–310.
  * Chapman et al. (2013) Wendy W Chapman, Dieter Hilert, Sumithra Velupillai, Maria Kvist, Maria Skeppstedt, Brian E Chapman, Michael Conway, Melissa Tharp, Danielle L Mowery, and Louise Deleger. 2013.  Extending the negex lexicon for multiple languages.  _Studies in health technology and informatics_ , 192:677.
  * Cotik et al. (2016a) Viviana Cotik, Roland Roller, Feiyu Xu, Hans Uszkoreit, Klemens Budde, and Danilo Schmidt. 2016a.  Negation detection in clinical reports written in german.  In _Proceedings of the Fifth Workshop on Building and Evaluating Resources for Biomedical Text Mining (BioTxtM2016)_ , pages 115–124.
  * Cotik et al. (2016b) Viviana Cotik, Vanesa Stricker, Jorge Vivaldi, and Horacio Rodríguez Hontoria. 2016b.  Syntactic methods for negation detection in radiology reports in spanish.  In _Proceedings of the 15th Workshop on Biomedical Natural Language Processing, BioNLP 2016: Berlin, Germany, August 12, 2016_ , pages 156–165. Association for Computational Linguistics.
  * Cui et al. (2019) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2019\.  Cross-lingual machine reading comprehension.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1586–1595, Hong Kong, China. Association for Computational Linguistics.
  * Frei and Kramer (2023) Johann Frei and Frank Kramer. 2023.  German medical named entity recognition model and data set creation using machine translation and word alignment: Algorithm development and validation.  _JMIR Form Res_ , 7:e39077.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Khalil et al. (2019) Talaat Khalil, Kornel Kiełczewski, Georgios Christos Chouliaras, Amina Keldibek, and Maarten Versteegh. 2019.  Cross-lingual intent classification in a low resource industrial setting.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6419–6424, Hong Kong, China. Association for Computational Linguistics.
  * Khandelwal and Sawant (2019) Aditya Khandelwal and Suraj Sawant. 2019.  NegBERT: a transfer learning approach for negation detection and scope resolution.  _arXiv preprint arXiv:1911.04211_.
  * Kilicoglu et al. (2017) Halil Kilicoglu, Graciela Rosemblat, and Thomas C. Rindflesch. 2017.  Assigning factuality values to semantic relations extracted from biomedical research literature.  _PLoS ONE_ , 12.
  * Kittner et al. (2021) Madeleine Kittner, Mario Lamping, Damian T Rieke, Julian Götze, Bariya Bajwa, Ivan Jelas, Gina Rüter, Hanjo Hautow, Mario Sänger, Maryam Habibi, et al. 2021.  Annotation and initial evaluation of a large annotated german oncological corpus.  _JAMIA open_ , 4(2):ooab025.
  * Kolluru et al. (2022) Keshav Kolluru, Muqeeth Mohammed, Shubham Mittal, Soumen Chakrabarti, and Mausam . 2022.  Alignment-augmented consistent translation for multilingual open information extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2502–2517, Dublin, Ireland. Association for Computational Linguistics.
  * Modersohn et al. (2022) Luise Modersohn, Stefan Schulz, Christina Lohr, and Udo Hahn. 2022.  Grascco - the first publicly shareable, multiply-alienated german clinical text corpus.  _Studies in health technology and informatics_ , 296:66—72.
  * Neves et al. (2022) Mariana Neves, Antonio Jimeno Yepes, Amy Siu, Roland Roller, Philippe Thomas, Maika Vicente Navarro, Lana Yeganova, Dina Wiemann, Giorgio Maria Di Nunzio, Federica Vezzani, Christel Gerardin, Rachel Bawden, Darryl Johan Estrada, Salvador Lima-lopez, Eulalia Farre-maduel, Martin Krallinger, Cristian Grozea, and Aurelie Neveol. 2022.  Findings of the WMT 2022 biomedical translation shared task: Monolingual clinical case reports.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 694–723, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Ozaki et al. (2021) Hiroaki Ozaki, Gaku Morio, Terufumi Morishita, and Toshinori Miyoshi. 2021.  Project-then-transfer: Effective two-stage cross-lingual transfer for semantic dependency parsing.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2586–2594, Online. Association for Computational Linguistics.
  * Pouran Ben Veyseh et al. (2019) Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019.  Graph based neural networks for event factuality prediction using syntactic and semantic structures.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4393–4399, Florence, Italy. Association for Computational Linguistics.
  * Qian et al. (2019) Zhong Qian, Peifeng Li, Qiaoming Zhu, and Guodong Zhou. 2019.  Document-level event factuality identification via adversarial neural network.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2799–2809, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Roller et al. (2022) Roland Roller, Aljoscha Burchardt, Nils Feldhus, Laura Seiffe, Klemens Budde, Simon Ronicke, and Bilgin Osmanodja. 2022.  An annotated corpus of textual explanations for clinical decision support.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2317–2326.
  * Rudinger et al. (2018a) Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018a.  Neural models of factuality.  _CoRR_ , abs/1804.02472.
  * Rudinger et al. (2018b) Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018b.  Neural models of factuality.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 731–744, New Orleans, Louisiana. Association for Computational Linguistics.
  * Shrestha (2021) Manjil Shrestha. 2021.  Development of a language model for medical domain.  master thesis, Hochschule Rhein-Waal.
  * Steffen and van Genabith (2021) Jörg Steffen and Josef van Genabith. 2021.  TransIns: Document translation with markup reinsertion.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 28–34. Association for Computational Linguistics.
  * Thompson et al. (2011) Paul Thompson, Raheel Nawaz, John McNaught, and Sophia Ananiadou. 2011.  Enriching a biomedical event corpus with meta-knowledge annotation.  _BMC Bioinformatics_ , 12:393 – 393.
  * Uzuner et al. (2011) Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011.  2010 i2b2/va challenge on concepts, assertions, and relations in clinical text.  _Journal of the American Medical Informatics Association_ , 18(5):552–556.
  * van Aken et al. (2021) Betty van Aken, Ivana Trajanovska, A. Siu, M. Mayrdorfer, Klemens Budde, and Alexander Loeser. 2021.  Assertion detection in clinical notes: Medical language models to the rescue?  _Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations_.
  * Vasilakes et al. (2022) Jake Vasilakes, Chrysoula Zerva, Makoto Miwa, and Sophia Ananiadou. 2022.  Learning disentangled representations of negation and uncertainty.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8380–8397, Dublin, Ireland. Association for Computational Linguistics.
  * Yarmohammadi et al. (2021) Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, and Benjamin Van Durme. 2021.  Everything is all it takes: A multipronged strategy for zero-shot cross-lingual information extraction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1950–1967, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zerva (2019) Chrysoula Zerva. 2019.  _Automatic Identification of Textual Uncertainty_.  Ph.D. thesis, University of Manchester.

##  Appendix A Appendix

The main contribution of this short paper was to show that it is possible to
develop a machine learning-based factuality detection for non-English, without
training examples in the given language - just by using a local machine
translation. In addition, we would like to present a small ‘bonus’ experiment,
which did not fit into the main article anymore. More precisely, we wanted to
find out how the performance of such a model changes if data in a reasonable
size is available for training. The additional experiment is presented in
Appendix A.1, followed by some additional text examples for the linguistic
error analysis and some further information.

###  A.1 Additional Experiment

The additional experiment has been conducted with the BRONCO150 Kittner et al.
(2021) dataset, a relatively large corpus originating from 150 German
oncological de-identified discharge summaries and annotated for multiple
tasks, including factuality detection. For our experiment, we consider only
the target entities diagnosis. Similar to Ex4CDS, it has various factuality
values, which we mapped to our three target labels, namely possible future and
speculation to possible. Note, BRONCO150 contains various fragmented entities
(entities split into two to three parts). For our experimental setup, we
merged entity fragments and considered only those sentences with not more than
50 characters between the fragments.

The label distribution of the obtained BRONCO150 data and the distribution of
the other datasets from the main paper are presented in Table 5.

First, we run the same experiment as presented in Table 3, also on BRONCO150
data. The results using our FactualMedBERT-DE model are presented in Table 4.

| NegEx | BERT-based
---|---|---
Label | Prec | Rec | F1 | Prec | Rec | F1
N Affirmed | 0.96 | 0.94 | 0.95 | 0.97 | 0.96 | 0.96
E Negated | 0.93 | 0.96 | 0.95 | 0.97 | 0.98 | 0.97
G Possible | 0.46 | 0.50 | 0.48 | 0.50 | 0.50 | 0.50
E Affirmed | 0.85 | 0.88 | 0.86 | 0.88 | 0.92 | 0.90
X Negated | 0.66 | 0.89 | 0.76 | 0.86 | 0.95 | 0.90
4 Possible | 0.50 | 0.18 | 0.26 | 0.61 | 0.38 | 0.47
B Affirmed | 0.87 | 0.96 | 0.91 | 0.88 | 0.97 | 0.92
R Negated | 0.69 | 0.66 | 0.68 | 0.75 | 0.80 | 0.77
O Possible | 0.68 | 0.24 | 0.36 | 0.73 | 0.25 | 0.37
Table 4: Performance results on different German medical text sources, namely the original German NegEx (upper part), the Ex4CDS dataset (middle) and BRONCO150 (lower part). | Affirmed | Negated | Possible
---|---|---|---
2010 i2b2/VA | 7603 | 2305 | 595
Ex4CDS | 892 | 225 | 179
NegEx-Ger | 645 | 443 | 26
BRONCO150 | 3179 | 331 | 523
Table 5: Support numbers in the evaluation sets for each processed dataset.

Next, we train two additional models, one on a BRONCO150 training split and a
second using the BRONCO150 train together with the translated i2b2 data. Both
models were initialized from the same model as that of FactualMedBERT-DE.
Table 6 compares our FactualMedBERT-DE against the other two BERT-based models
on the different datasets.

| 2010 i2b2/VA | NegEx-Ger | Ex4CDS | BRONCO150
---|---|---|---|---
Model | Label | Prec | Rec | F1 | Prec | Rec | F1 | Prec | Rec | F1 | Prec | Rec | F1
FactualMedBERT-DE | Affirmed | 0.96 | 0.98 | 0.97 | 0.97 | 0.96 | 0.96 | 0.88 | 0.92 | 0.90 | 0.88 | 0.97 | 0.92
Negated | 0.95 | 0.93 | 0.94 | 0.97 | 0.98 | 0.97 | 0.86 | 0.95 | 0.90 | 0.76 | 0.79 | 0.78
Possible | 0.80 | 0.64 | 0.71 | 0.50 | 0.50 | 0.50 | 0.61 | 0.38 | 0.47 | 0.68 | 0.19 | 0.30
BRONCO150-BERT | Affirmed | 0.88 | 0.95 | 0.92 | 0.97 | 0.92 | 0.94 | 0.90 | 0.90 | 0.90 | 0.96 | 0.96 | 0.96
Negated | 0.95 | 0.67 | 0.79 | 0.97 | 0.97 | 0.97 | 0.89 | 0.88 | 0.88 | 0.95 | 0.83 | 0.89
Possible | 0.42 | 0.47 | 0.44 | 0.28 | 0.65 | 0.39 | 0.56 | 0.59 | 0.58 | 0.76 | 0.84 | 0.80
i2b2+BRONCO150 BERT | Affirmed | 0.94 | 0.98 | 0.96 | 0.98 | 0.95 | 0.96 | 0.90 | 0.94 | 0.92 | 0.95 | 0.98 | 0.97
Negated | 0.96 | 0.91 | 0.93 | 0.98 | 0.97 | 0.97 | 0.90 | 0.91 | 0.91 | 0.93 | 0.83 | 0.88
Possible | 0.82 | 0.54 | 0.65 | 0.39 | 0.73 | 0.51 | 0.70 | 0.54 | 0.61 | 0.85 | 0.74 | 0.79
Table 6: Performance results of three BERT models trained on translated i2b2 (FactualMedBERT-DE), BRONCO150 and 2010 i2b2 + BRONCO150, respectively. The models were evaluated on different German medical text sources, namely our translated i2b2 2010 test set, the German NegEx, the Ex4CDS dataset and BRONCO150 test set. For each dataset, best per-label F1-performances are displayed in bold. Issue |  English |  German
---|---|---
missing trigger in translation |  The patient radiated down her left arm associated with some nausea, no <E> shortness of breath </E>, cough, vomiting, diarrhea. |  Die Patientin strahlte in Verbindung mit Übelkeit, <E> Atemnot, </E> Husten, Erbrechen, Durchfall nach unten.
incorrect trigger translation |  RULE OUT FOR <E> myocardial infarction </E> |  VORSCHRIFTEN FÜR <E> den Myokardinfarkt </E>
trigger in the translation is within the entity |  She did not notice <E> any blood / urine / emesis / stool in the bed </E>. |  Sie bemerkte <E> kein Blut / Urin / Erbrechen / Stuhl im Bett. </E>
missing of a possible trigger translation in NegEx-Ger |  Denies <E> fevers </E>, pleuritic chest pain or cough. |  Verleugnet <E> Fieber, </E> pleuritische Brustschmerzen oder Husten.
missing of translated compounds of type Entity + trigger in NegEx-Ger |  She was <E> pain </E> free on the day of discharge . |  Sie war am Tag der Entlassung <E> schmerzfrei. </E>
missing trigger phrase in NegEx-Ger due to word order |  He then presented to Mass. Mental Health Center where he ruled out for <E> an myocardial infarction </E> by enzymes and electrocardiograms. |  Er überreichte dann der Messe. Mental Health Center, wo er für <E> einen Myokardinfarkt </E> durch Enzyme und Elektrokardiogramme ausgeschlossen wurde.
different encoding of umlauts in text and NegEx-Ger |  <E>the hypernatremia</E> fully resolved when he resumed eating on his own and had access to free water . |  <E>Die Hypernatrimie</E> vollständig aufgeloest, als er wieder essen auf eigene Faust und hatte Zugang zu freien Wasser.
Table 7: Examples of the potential causes for prediction errors. The analysis
focuses on the translation problems and the differences between the German and
English language. The tags <E></E> enclose the entities, the factuality
triggers are underlined. The original English examples originate from the i2b2
data.

#### Brief discussion:

The results show that each model performs best on the data of the same dataset
- FactualMedBERT-DE on the translated i2b2 data and BRONCO150-BERT on the
BRONCO150 data - this is no surprise. Moreover, the results indicate that the
mixed model (i2b2+BRONCO150-BERT) performs generally well on all datasets,
therefore might be the model of choice. However, it is important to note, that
BRONCO150 has got an unusual label distribution. While affirmed is the most
frequent label in all datasets, BRONCO has got an unusually high frequency of
possible labels, which is connected to the way labels were mapped to the three
final actuality labels. However, this might influence the actuality
classification of other datasets.

###  A.2 BERT Setup

For BERT, we used epochs number of 3/4 (for English and German BERT,
respectively), a batch size of 32, a dropout rate of 0.1, and a learning rate
of $1e-5$.

###  A.3 Examples of Linguistic Error Analysis

Our analysis focuses on the potential sources for false predictions, in
particular on causes related to the translation or the differences in the
features of the German and English languages. Table 7 presents full-text
examples from the original and translated data. For a detailed description of
the possible issues see Section 4.1.