11institutetext: University College Dublin, Ireland11institutetext: University
College Dublin, Ireland
11email<EMAIL_ADDRESS>
# Curatr: A Platform for Semantic Analysis and Curation of Historical Literary
Texts

Susan Leavy  Gerardine Meaney  Karen Wade  Derek Greene  Susan Leavy
Gerardine Meaney  Karen Wade  Derek Greene

###### Abstract

The increasing availability of digital collections of historical and
contemporary literature presents a wealth of possibilities for new research in
the humanities. The scale and diversity of such collections however, presents
particular challenges in identifying and extracting relevant content. This
paper presents _Curatr_ , an online platform for the exploration and curation
of literature with machine learning-supported semantic search, designed within
the context of digital humanities scholarship. The platform provides a text
mining workflow that combines neural word embeddings with expert domain
knowledge to enable the generation of thematic lexicons, allowing researches
to curate relevant sub-corpora from a large corpus of 18th and 19th century
digitised texts.

###### Keywords:

Text mining Digital humanities Corpus curation

##  1 Introduction

The interpretability of the algorithmic process and the incorporation of
domain knowledge are essential to the use of machine learning and text mining
in the semantic analysis of literature. The absence of these factors can
inhibit adoption of machine learning approaches to text mining in the
humanities, due to issues of accuracy and trust in what is often regarded as a
‘black-box’ process [6, 13, 15]. This paper presents Curatr, an online
platform that incorporates domain expertise and imparts transparency in the
use of machine learning for literary analysis. The system supports a corpus
curation workflow that addresses the requirements of scholars in the
humanities who are increasingly working with large collections of unstructured
text and facilitates the development of sub-corpora from large digital
collections.

Selection, curation and interpretation is central to knowledge generation in
the humanities [18, 36]. This process is supported in Curatr with conceptual
search functionality that uses neural word embeddings to build conceptual
lexicons specific to a given theme or topic. These thematic lexicons can then
be used to mine relevant texts to form curated literary collections that may
be saved, further modified, or exported as sub-corpora. The platform was
developed based on a collection of 35,918 English language digital texts from
the British Library111British Library Labs:
https://www.bl.uk/projects/british-library-labs.

An evaluation of _Curatr_ was conducted in conjunction with an associated
project examining the relationship between societal views of migration,
ethnicity, and concepts concerning contagion and disease in 19th century
Britain and Ireland. In order to explore the cultural representation of
migrants, the study focused on their representation within historical fiction
which comprises 16,426 texts of the British Library digital collection. Given
the largest communities of migrants to London during the late 19th century
were Irish and Jewish, this study focused on their portrayal in relation to
prevailing concepts of contagion, disease and migration. Lexicons related to
these themes were generated through recommendations derived from word
embedding models and text were retrieved based on relevance of the texts. The
findings were evaluated in terms of the overall requirements of a humanities
scholar along with the relevance of the texts uncovered. We describe this case
study in more detail in Section 4.

##  2 Related Work

###  2.1 Text Mining in the Humanities

The work builds on a range of literature that demonstrates requirements for
digital humanities platforms. Close reading functionality to provide context
is an essential aspect of humanities research, as evidenced in the provision
of close reading functionality along with quantitative analysis in systems
developed by Hinricks et al. [16] and Vane [34]. Domain knowledge was combined
with automated text classification to provide more accurate retrieval results
in a system developed by Sweetnam and Fennel [1] to explore early-modern
English texts. A system based on semantic search was demonstrated by Kopaczyk
et al [20] for the analysis of Scottish legal documents from the 16th century.
Other systems, such as that proposed by Jockers [18], have focused primarily
on the use of machine learning methods for text analysis.

In a study of the update of machine learning in industry, Chiticariu et al.
[6] noted a gap in the volume of academic research on machine learning,
compared with lower levels of uptake within industry and found the causes of
this pertained to training data, interpretability and incorporation of domain
knowledge. Similarly, the relatively low uptake of machine learning methods in
the digital humanities has been attributed to issues pertaining to
interpretation and trust [33, 13, 15]. Imparting domain knowledge into the
process of text analysis through interpretation and annotation is also central
to humanities research [17, 36]. The _Curatr_ platform addresses these
specific requirements of digital humanities research by incorporating domain
knowledge and transparency within a text mining workflow.

###  2.2 Concept Modelling with Word Embeddings

_Word embedding_ refers to a family of methods from natural language
processing that involve mapping words or phrases appearing in large text
corpora to dense, low-dimensional numeric representations. Typically, each
unique word in the corpus vocabulary will be represented by its own vector. By
transforming textual data in this way, we can use the new representation to
capture the semantic similarity between pairs or groups of words. Word
embedding methods have been used in digital humanities research to generate
semantic lexicons for a range of purposes including detecting language change
over time [14], extracting social networks from literary texts [35], sentiment
analysis [31], and semantic annotation [21]. An interactive strategy whereby a
user incrementally creates a lexicon based on recommendations for similar
words as recommended by a word embedding model has been demonstrated in a
number of works [10, 27].

A variety of different approaches have been proposed in the literature to
construct embeddings. The word embedding algorithm used in this research is
_word2Vec_ [22], which generates distributed representations of words that can
be used to interpret their meaning. This approach captures the concept from
distributed semantics that the meaning of a word “can be determined by the
company it keeps” [11]. Word co-occurrence is identified over an entire corpus
and each word along with the words found beside it in the text are represented
by a vector. The similarity of terms can then be derived based on whether they
are used alongside similar words, or in a similar context. This approach to
generating lexicons has been shown to be useful where the language of a
particular corpus is highly specific [5] and where existing general-purpose
lexicons are not appropriate and is therefore particularly relevant to digital
humanities. It has also been pointed out that pre-processing and methods of
representing text can have particular significance within a digital humanities
context [4, 12]. These decisions are a crucial aspect of the evaluation of
results within humanities scholarship. Given that using word embedding in text
analysis has been critiqued for its lack of transparency [30], it is crucial
within a digital humanities context to impart transparency into the text
mining workflow.

##  3 Curatr Design

###  3.1 Platform Overview

Curatr implements a text mining framework involving conceptual search using
word embeddings to dynamically build a semantic lexicon specific to a given
literary corpus. The humanities researcher begins with seed terms and these
are expanded using neural word embedding through an interactive online
interface to produce semantic lexicons. The _Curatr_ system also provides for
keyword search, filtering based on metadata, ngram frequencies, and
categorisation based on the original British Library topical classifications
(Fig. 2). The information retrieval component of the system involves the
indexing of texts and using the open source Apache Solr
engine222http://lucene.apache.org/solr/.

Figure 1: _Curatr_ workflow for digital humanities text mining.

###  3.2 Concept Lexicon Generation

The conceptual search workflow outlined in Fig. 1 enables the compilation of
seed terms associated with a given concept or topic by the humanities
researcher. _Curatr_ allows for the expansion of these terms to form a lexicon
of semantically similar words based on associations suggested from the
querying of a neural word embedding model developed from the corpus. Word
embedding models were generated from the complete English language corpus
using the _word2vec_ approach [22] yielding real-valued, low-dimensional
representations of words based on lexical co-occurrences. The use of word
embeddings rather than more complex language models were deemed appropriate
due to the lack of structure in the text and OCR errors introduced through the
process of digitisation. The specific embedding variant used in this work is a
100-dimensional Continuous Bag-Of-Words (CBOW) _word2vec_ model, trained on
the full-text volumes of the corpus. To address the levels of transparency
required in digital humanities scholarship regarding approaches to text
representation and parameter settings and their potential effects on results,
decisions regarding text processing options and parameter strategies in
generating the neural word embedding models are available to the user.

Based on the embedding model, the top 20 words found to be similar to the seed
words are recommended to expand the current lexicon. The user selects the
subset of recommendations to be included in the conceptual lexicon. This
inclusion of a “human in the loop” ensures that the process of generating a
lexicon is informed by domain knowledge of the user. Multiple iterations of
this semantic search allow the researcher to refine the lexicon and augment
the conceptual category on each iteration, simulating the process of knowledge
generation from close reading and annotation.

###  3.3 Corpus Curation

The finalised semantic lexicons are used as a basis for ranked volume
retrieval from the indexed library corpus. Texts are ranked according to the
frequency of occurrence of terms from each generated lexicon relative to the
document length. This process uncovers documents that pertain to the given
conceptual category. These document rankings may then be saved by the user for
later use.

Figure 2: Sample Curatr n-gram frequency analysis.

Curatr also facilitates the amendment of the sub-corpus to remove non-relevant
texts. Based on the results of this search, the user may revise the contents
of the conceptual lexicon and rerun the text mining process. Along with the
provision of a close reading interface for examining individual texts, the
platform also enables the export of curated sub-corpora to download for
further analysis.

##  4 Case Study Evaluation

The case study involved an exploratory investigation of historical cultural
attitudes towards migration in Britain and their associations with concepts of
contagion and disease from the set of 16,426 texts in the corpus [26, 19].
Poverty-induced migration from Ireland to Britain during the Great Famine
(1845-1851) is cited as generating a fear of transmission of contagious
disease [23]. The association of fear of contagion with historical sentiments
towards migrants, according to Samuel K. Cohn however, requires more
systematic study and more nuanced interpretation of both historical and
cultural archives [8]. The aim of this exploratory study is to uncover new
texts and literary representations that are less well known to researchers and
which might challenge current understandings of the relationship between
migration and concepts of contagion and disease.

Evaluating the usefulness of retrieved documents in the context of humanities
research does not always align with standard information retrieval precision
metrics. This is particularly pertinent in the preliminary exploratory phase
of humanities research, where new and diverse texts are sought by the
researcher and their relevance to the research topic is often not immediately
apparent [7]. However, a prime advantage of text mining in the humanities is
the potential to uncover novel and diverse texts that could challenge
prevailing theory. This research therefore draws on Bates et al. [3], who
examined what constitutes relevance of a document in the context of humanities
research defining two aspects of relevance in humanities research, _content
relevance_ and _utility relevance_. Content relevance determines whether the
search terms in the query relate to the content retrieved while utility
relevance evaluates the usefulness in the context of the research topic. Given
that a humanities researcher requires texts that inform their topic, texts
deemed to be most relevant in terms of utility are often not those with the
highest content relevance, but are those that alter the judgement and
challenge existing theories of the researcher. Document unfamiliarity is
therefore an attribute that ‘may be presumed to swamp all other
considerations’ [3, p. 702]. Unfamiliarity of a text was defined by Barry et
al. ( [2]) in terms of content, source or stimulus. _Content novelty_
describes how the content of a text itself provides new knowledge to a
researcher. _Source novelty_ refers to whether the text is written by a
previously unknown author or publisher. _Stimulus novelty_ indicates whether
the text itself is new to the researcher. Brought together the framework
outlines capture the complexity of judgements of value and relevance of text
mining for the humanities.

The key thematic strands of the case study were identified and associated seed
terms defined. The themes pertained to ‘migration’, ‘illness’, ‘contagion’,
and Irish and Jewish ethnic identities. The word embedding model was queried
to uncover words used in a similar context to the seed terms in the fiction
corpus (see Fig. 3). The results were selectively added based on contextual
knowledge and interpretation on the part of the humanities researcher through
an interactive lexicon building interface. The resulting expanded semantic
lexicons were then used as a basis for ranking the documents to uncover texts
that capture the key themes of the study. See Table 1 for examples of lexicons
generated in this way.

Lexicon |  Seed Terms |  Recommended Words
---|---|---
_Ethnic identity_ |  irish, fenian, papist, jewish, jew |  jews, fenianism, hibernian, usurer, celtic, rebels, invincibles, whiteboys, incendiary, brogue, chartists, irishman, catholics, ringleaders, rabbis
_Migration_ |  immigrant, alien, interloper, migrant |  civilisers, doavn trodden, self governing, circumcised, peoples, separatists, cousinhood, usurpers, interloper, aliens, intruder, middlemen, ryots, kinless, alien
_Contagion_ |  infect, epidemic, inoculate, contagion, contaminate, vaccinate |  infection, contagion, infectious, infected, fever, contagious, epidemic, plague, epidemic, disease, epidemics, fever, malarial, endemic, malady
_Disease_ |  disease, smallpox, cholera, fever, pestilence |  scarlet fever, epidemics, morbus, cancers, tumour, incurable, malaria, sickness, cancer, brain, diseases, distempers, typhus, anaemia, heart disease
Table 1: Examples of top recommended words for different conceptual lexicons.

Analysis of the recommended terms suggested new associations in the fiction
corpus. For example, terms pertaining to the theme of civil unrest (rebels,
dynamiters, conspirators, incendiarism, anarchistic, nihilists, revolutionary,
sedition, informers, radicals) make up a substantial portion of the lexicon
pertaining to ‘ethnic identity’. As a result of these suggestions, aligning
with the iterative theory-building approach of research in the humanities, the
suggested words enriched the original conceptualisation of the research theme
of ethnic identity to incorporate relationships with political ideology.
Additionally, the expanded lexicons indicated points where conceptual
ontologies overlap. Suggested terms in _Curatr_ also provided indications of
relevant but unfamiliar or archaic terminology. For instance, in this study
the term ‘distemper’, which is more commonly applied to animal illnesses
nowadays, appears as a term that is relevant to the concept of contagion
within these works. OCR (optical character recognition) and spelling errors
and variants were also uncovered using this method.

###  4.1 Finding and Discussion

Drawing upon the framework of retrieval value outlined by Bates et al. [3] and
Barry et al. [2], the top 10 retrieved texts in each category were analysed in
terms of novelty to the humanities researcher involved in the project. The
qualitative difference in text retrieved before and after the query expansion
phase was also examined. Texts in the corpus were ranked according to the
frequency by which the curated conceptual lexicons were mentioned, relative to
the length of the documents. The iterative process whereby the researcher
returns to edit the conceptualisation of the key thematic trends of the
research topic through editing the semantic lexicons aligns with the process
of grounded theory research and necessitated a close reading of the results by
humanities researchers to evaluate the value of the retrieved documents.

Figure 3: The _Curatr_ lexicon management interface. Prior to Query Expansion | Post Query Expansion
---|---
1894 The Captain’s Youngest, Frances H. Burnett
1888 The Devil‘s Die, Allen Grant
1894 The Azrael of Anarchy, Gustave Linbach
1891 The Year of Miracle, Fergus Hume
1870 Unawares, Frances Mary Peard
1893 Doctor, or Lover?, Faber Vance
1888 A Crown of Shame, Florence Marryat
1875 Ashes to ashes, Hugh R. Haweis
1865 Not Proven, Christina B Cameron
1892 The Medicine Lady, Elizabeth T Meade
|  1891 The Year of Miracle, Fergus Hume
1897 The Sign of the Red Cross, Evelyn E. Green
1855 Old Saint Paul‘s, William H. Ainsworth
1894 The Azrael of Anarchy, Gustave Linbach
1847 A Tale of the Irish Famine, Unknown
1885 The Legend of Samandal, James Fer
1855 The Wood-Spirit, Ernest C. Jones
1888 The Devil‘s Die, Grant Allen
1898 Vanya, Orlova Olga
1897 A Literary Gent, J.C. Kernahan

Table 2: Texts retrieved before and after query expansion, for concepts related to ‘illness’ and ‘contagion’. Prior to Query Expansion | Post Query Expansion
---|---
1898 For Lilias, Rosa Nouchette Carey
1875 The Golden Shaft, George C. Davies
1876 The Youth of the Period, James F.S. Kennedy
1876 Her Dearest Foe, Alexander
1886 A Modern Telemachus, Charlotte M. Yonge
1887 Major Lawrence, Emily Lawless
1857 Guy Fawkes, William H. Ainsworth
1846 The Moor, the Mine and the Forest, William Heatherbred
1865 The Notting Hill Mystery, Charles Felix
1897 Owen Tanat, Alfred N. Palmer
|  1881 Gifts and Favours Doctor Olloed, Unknown
1871 Ierne, William R. Trench
1863 Sackville Chase, Charles J. Collins
1894 Ivanda or the Pilgrim’s quest, Claude A. Bray
1894 Doctor Izard, Anna K. Green
1865 The Crusader or the Witch of Finchley, Unknown
1864 The Bee-Hunters, Gustave Aimard
1850 Helen Porter, Thomas P. Prest
1852 Idone, James H. L. Archer
1886 A Mysterious Trust, Edmund Mitchell

Table 3: Texts retrieved before and after query expansion, for a concept
related to ‘migration’.

####  4.1.1 Source novelty.

A striking pattern among the texts retrieved with the expanded query set was
the relative obscurity of some of the authors. In analysing the top ten texts
retrieved using the seed terms alone, five of the migration category and four
of illness category were immediately familiar to the researchers (Tables 2 and
3). However, after the list was expanded based on suggestions from neural word
embedding, this figure reduced to three for the concept of illness, and none
were previously well known in relation to the concept of migration.

A similar trend is evident in the texts retrieved using the semantic lexicons
of ethnic identity where only one of the authors in the top ten texts related
to Irish identity and three relating to Jewish identity were widely known (see
Table. 4). The texts retrieved based on the semantic lexicon of Irish
ethnicity contained none by widely read Irish authors, although Ulick Ralph
Burke and Albert Stratford George Canning do merit an entry in the database of
Irish Literature333Ricorso: Database of Irish writers http://www.ricorso.net.
A work by the prolific English author Fergus Hume was retrieved, as were works
by the British authors Mabel Collins and Edith Cuthell, the latter novel being
its author’s only work based in an Irish setting. This identification of a
number of writers less prominent in the canon, but who wrote on topics
relating to the key themes of the case study, demonstrate how the expansion of
query terms in this context has the potential to uncover relevant texts which
might otherwise be overlooked.

####  4.1.2 Stimulus novelty.

Irish Lexicon | Jewish Lexicon
---|---
1897 Sweet Irish Eyes, E. Cuthell
1893 The Harlequin Opal, F. Hume
1875 The Autobiography of a Man-o-War‘s Bell, C. R. Low
1883 The Wild Rose of Lough Gill, P. G. Smyth
1880 Loyal and Lawless, U. R. Burke
1867 Baldearg O’Donnell, A. A. G. Canning
1893 The Great War of 189-. A forecast, F. Villiers
1896 The Idyll of the White Lotus, M. Collins
1886 Our Radicals. A tale of love and politics, F. Burnaby
1891 The Last Great Naval War, A. N. Seaforth
1890 Heir and no Heir Canning, A. S. G. Canning
|  1890 The Prophet. A parable, T. H. H. Caine
1897 A Rogue‘s Conscience, D. C. Murray
1864 The Hekim Bashi, H. Sandwith
1874 Jessie Trim, B. L. Farjeon
1853 The Turk and the Hebrew, Unknown
1865 The Crusader or the Witch of Finchley, Unknown
1895 Maid Marian and Crotchet Castle, T. L. Peacock
1898 Dreamers of the Ghetto, I. Zangwill
1838 Oliver Twist, C. Dickens
1869 Count Teleki: A story of modern Jewish life, UN

Table 4: Texts retrieved for the ‘Irish’ and ‘Jewish’ conceptual lexicons.

In this case study the texts retrieved presented a number of works that
represented not only less well-known authors of the 18th and 19th centuries,
but also less well-known texts written by more prominent authors. In relation
to the texts retrieved using the conceptual lexicon of illness for instance,
the majority of retrieved titles were not commonly known and none pertaining
to Irish identity attracted significant bibliographic or critical commentary.
The text marked as most relevant to Irish identity, _Sweet Irish Eyes_ , was
written by an English author and was their only Irish-themed work. Thee
expanded list of query terms also uncovered slightly older texts pointing to
the potential of word embedding to compile a semantic lexicons that are less
biased towards contemporary linguistic norms.

The most relevant of the retrieved set of texts pertaining to Jewish identity
was that by British novelist Hall Caine. This work is referred to in studies
of the Gothic (e.g. Mulvany-Roberts [24]), but there is very little reference
to this novel and its portrayal of Jewish characters. It is notable that
Anglo-Jewish writers (Zangwill and Farjeon444Farjeon features more prominently
in histories of Australian literature as he emigrated there.) feature
prominently alongside one of Dickens’ best known novels. Charles Dickens’
_Oliver Twist_ and Israel Zangwil’s _Dreamers of the Ghetto_ , which appear on
the list of novels pertaining to Jewishness, were the only two novels on the
list that have previously attracted sustained scholarly attention. For
example, Udelson [32], Rochelson [28, 29] and Murray [25] have all written on
Zangwill, though he is far less well known to the public than Dickens.

####  4.1.3 Content novelty.

While uncovering authors and titles that were previously less familiar to a
researcher is beneficial, its value is ultimately dependent on the relevance
of the content. Close analysis showed that the expanded set of query terms
retrieved books that were less specific to the topics than those retrieved
using the manually generated set of seed terms suggesting new associations and
challenging the researcher’s understanding of the key themes. For instance, in
relation to the concept of illness, a text pertaining to the Irish Famine and
texts addressing political issues were returned. The most relevant to the
concept of illness using an expanded lexicon was Evelyn Everett Green’s novel,
_The Sign of the Red Cross_. The content of this novel describes details of a
plague in London and was critiqued at the time for its graphic nature [9]. The
novel _Old Saint Paul’s_ was concerned with the plague in London and provides
a wealth of information relevant to the topic. While these novels are not
among the most well known currently, their graphic account of plague in London
provides a wealth of insight regarding the historical conceptualisation of
illness in Britain. The content of texts retrieved pertaining to Irish and
Jewish ethnic identities were highly relevant to the topic and uncovered texts
that are less well known, thus presenting new opportunities to examine
cultural representations of ethnicity. For instance, the theme of seafaring in
the titles pertaining to Irish identity is an association that was new to the
researchers. The content that was returned in relation to the concept of
migration returned more titles that addressed migration in the broader sense
in connection with international political events, in contrast with the
narrower focus of the results retrieved using seed terms alone.

##  5 Conclusion

This research presents an approach whereby machine learning and text analysis
is used within a text mining workflow designed for digital humanities
research. The corpus curation workflow in _Curatr_ is supported by neural word
embedding and through an interactive online platform, ensures transparency and
incorporates domain knowledge. The approach demonstrates how machine learning
techniques may be used to enhance the curation process for humanities
scholars. Evaluation of the system demonstrated that domain-specific
conceptual modelling with neural word embedding is effective in uncovering
texts that capture given concepts. Texts retrieved using the _Curatr_ text
mining workflow were particularly useful in the context of exploratory
humanities research. The system uncovered works that had not previously
attracted sustained scholarly attention, and which presented opportunities to
uncover new insights in relation to the given research topic. In future work
the addition of new sources to the platform would further enrich this process.
Through the exploratory text mining workflow supported by _Curatr_ ,
humanities scholars are enabled to challenge prevailing methods of
canonisation of historical fiction.

Acknowledgements. This research was partially supported by Science Foundation
Ireland (SFI) under Grant Number SFI/12/RC/2289.

## References

  * [1] Bailey, E., Lawless, S., O‘Connor, A., Sweetnam, S., Conlan, O., Hampson, C., Wade, V.: Cultura: supporting enhanced exploration of cultural archives through personalisation. In: the Proceedings of the 2nd International Conference on Humanities, Society and Culture, ICHSC. ICHSC (2012)
  * [2] Barry, C.L.: User-defined relevance criteria: An exploratory study. Journal of the American Society for Information Science 45(3), 149–159 (1994)
  * [3] Bates, M.J.: The getty end-user online searching project in the humanities: Report no. 6: Overview and conclusions. College & Research Libraries 57(6), 514–523 (1996)
  * [4] Camacho-Collados, J., Pilehvar, M.T.: On the role of text preprocessing in neural network architectures: An evaluation study on text categorization and sentiment analysis. arXiv preprint arXiv:1707.01780 (2017)
  * [5] Chanen, A.: Deep learning for extracting word-level meaning from safety report narratives. In: Integrated Communications Navigation and Surveillance (ICNS), 2016\. pp. 5D2–1. IEEE (2016)
  * [6] Chiticariu, L., Li, Y., Reiss, F.R.: Rule-based information extraction is dead! long live rule-based information extraction systems! In: Proceedings of the 2013 conference on empirical methods in natural language processing. pp. 827–832 (2013)
  * [7] Clarke, C.L., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. pp. 659–666. ACM (2008)
  * [8] Cohn, S.K.: Pandemics: waves of disease, waves of hate from the plague of athens to aids. Historical Research 85(230), 535–555 (2012)
  * [9] Dempster, J.A.: Thomas nelson and sons in the late nineteenth century: A study in motivation. part 1. Publishing History 13, 41 (1983)
  * [10] Fast, E., Chen, B., Bernstein, M.S.: Empath: Understanding topic signals in large-scale text. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 4647–4657. ACM (2016)
  * [11] Firth, J.R.: A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis (1957)
  * [12] Flanders, J., Jannidis, F.: The Shape of Data in Digital Humanities: Modeling Texts and Text-based Resources. Routledge (2018)
  * [13] Frank, A., Bögel, T., Hellwig, O., Reiter, N.: Semantic annotation for the digital humanities. Linguistic Issues in Language Technology 7(1), 1–21 (2012)
  * [14] Hamilton, W.L., Clark, K., Leskovec, J., Jurafsky, D.: Inducing domain-specific sentiment lexicons from unlabeled corpora. In: Proc. EMNLP 2016. vol. 2016, p. 595. NIH Public Access (2016)
  * [15] Hampson, C., Munnelly, G., Bailey, E., Lawless, S., Conlan, O.: Improving user control and transparency in the digital humanities. In: 2013 International Conference on Culture and Computing (Culture Computing). pp. 196–197. IEEE (2013)
  * [16] Hinrichs, U., Alex, B., Clifford, J., Watson, A., Quigley, A., Klein, E., Coates, C.M.: Trading consequences: A case study of combining text mining and visualization to facilitate document exploration. Digital Scholarship in the Humanities 30(suppl_1), i50–i75 (2015)
  * [17] Jackson, H.J.: Marginalia: Readers writing in books. Yale University Press (2002)
  * [18] Jockers, M.: Detecting and characterizing national style in the 19th century novel. In: Digital Humanities 2011, Stanford, CA (2011)
  * [19] Kinealy, C.: This Great Calamity: The Great Irish Famine: The Irish Famine 1845-52. Gill & Macmillan Ltd (2006)
  * [20] Kopaczyk, J.: The Legal Language of Scottish Burghs: Standardization and Lexical Bundles (1380-1560). Oxford University Press (2013)
  * [21] Leavy, S., Pine, E., Keane, M.T.: Industrial memories: Exploring the findings of government inquiries with neural word embedding and machine learning. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018 (2018)
  * [22] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  * [23] Morash, C.: The Hungry Voice: The Poetry of the Irish Famine. Irish Academic Press (2009)
  * [24] Mulvey-Roberts, M.: The Handbook of the Gothic. Springer (2016)
  * [25] Murray, J.: The social enterprise law market. Md. L. Rev. 75, 541 (2015)
  * [26] Nelkin, D., Gilman, S.L.: Placing blame for devastating disease. Social Research pp. 361–378 (1988)
  * [27] Park, D., Kim, S., Lee, J., Choo, J., Diakopoulos, N., Elmqvist, N.: Conceptvector: text visual analytics via interactive lexicon building using word embedding. IEEE transactions on visualization and computer graphics 24(1), 361–370 (2018)
  * [28] Rochelson, M.J.: “they that walk in darkness”: Ghetto tragedies: The uses of christianity in israel zangwill’s fiction. Victorian Literature and Culture 27(1), 219–233 (1999)
  * [29] Rochelson, M.J.: A Jew in the Public Arena: The Career of Israel Zangwill. Wayne State University Press (2010)
  * [30] Subramanian, A., Pruthi, D., Jhamtani, H., Berg-Kirkpatrick, T., Hovy, E.: Spine: Sparse interpretable neural embeddings. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
  * [31] Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., Qin, B.: Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). vol. 1, pp. 1555–1565 (2014)
  * [32] Udelson, J.H.: Dreamer of the Ghetto: The Life and Works of Israel Zangwill. University of Alabama Press (1990)
  * [33] Van Cranenburgh, A., van Dalen-Oskam, K., van Zundert, J.: Vector space explorations of literary language. Language Resources and Evaluation (Feb 2019)
  * [34] Vane, O.: Text visualisation tool for exploring digitised historical documents. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. pp. 153–158. ACM (2018)
  * [35] Wohlgenannt, G., Chernyak, E., Ilvovsky, D.: Extracting social networks from literary text with word embedding tools. In: Proc. Workshop on Language Technology Resources and Tools for Digital Humanities. pp. 18–25 (2016)
  * [36] Wolfe, J.: Annotations and the collaborative digital library: Effects of an aligned annotation interface on student argumentation and reading strategies. International Journal of Computer-Supported Collaborative Learning 3(2), 141 (2008)


# Multitask Balanced and Recalibrated Network for Medical Code Prediction

Wei Sun  Aalto UniversityEspooFinland<EMAIL_ADDRESS>,  Shaoxiong Ji  Aalto
UniversityEspooFinland<EMAIL_ADDRESS>,  Erik Cambria  Nanyang
Technological UniversitySingapore<EMAIL_ADDRESS>and  Pekka Marttinen
Aalto UniversityEspooFinland<EMAIL_ADDRESS>
(2022)

###### Abstract.

Human coders assign standardized medical codes to clinical documents generated
during patients’ hospitalization, which is error-prone and labor-intensive.
Automated medical coding approaches have been developed using machine learning
methods such as deep neural networks. Nevertheless, automated medical coding
is still challenging because of complex code association, noise in lengthy
documents, and the imbalanced class problem. We propose a novel neural network
called Multitask Balanced and Recalibrated Neural Network to solve these
issues. Significantly, the multitask learning scheme shares the relationship
knowledge between different coding branches to capture the code association. A
recalibrated aggregation module is developed by cascading convolutional blocks
to extract high-level semantic features that mitigate the impact of noise in
documents. Also, the cascaded structure of the recalibrated module can benefit
the learning from lengthy notes. To solve the imbalanced class problem, we
deploy the focal loss to redistribute the attention on low and high-frequency
medical codes. Experimental results show that our proposed model outperforms
competitive baselines on a real-world clinical dataset MIMIC-III.

Medical Code Prediction, Multitask Learning, Imbalanced Class Problem,
Balanced and Recalibrated Network

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††journal:
TIST††journalvolume: 0††journalnumber: 0††article: 0††publicationmonth:
0††ccs: Applied computing Health care information systems††ccs: Computing
methodologies Natural language processing††ccs: Applied computing Document
management and text processing

##  1\. Introduction

Professional doctors write discharge summaries based on different kinds of
clinical notes such as diagnosis reports, prescriptions, and treatment
procedure documents. Health institutes annotate these notes with standardized
medical codes to facilitate information acquisition and management.
International Classification of Disease
(ICD)111https://www.who.int/standards/classifications/classification-of-
diseases, one of the most widely used medical coding systems, is maintained by
the World Health Organization (WHO). ICD coding system converts disease,
pathology reason, symptom, and signs into standard ICD codes, which is helpful
in various medical-related services, including insurance reimbursement (Park
et al., 2000), statistical data analysis, and clinical decision support (Horng
et al., 2017). Since medical codes annotation by human is error-prone
(O’malley et al., 2005) and labor intensive (Sun et al., 2021), a surge of
feature engineering-based machine learning (Perotte et al., 2014; Koopman et
al., 2015) and deep learning (Vu et al., 2021; Sun et al., 2021; Ji et al.,
2021b; Li and Yu, 2020) approaches have been proposed for automating the
medical coding task.

However, the automatic medical coding task is still challenging, mainly due to
the following three aspects.

Imbalanced Class Problem: Many medical coding datasets, such as the third
version of the Medical Information Mart for Intensive Care (MIMIC-III), suffer
from a severe imbalanced class problem. We take the MIMIC-III dataset as an
example, the frequency distribution of ICD codes and CCS codes demonstrated in
Figure 1(a) and Figure 1(b), respectively. The main reason is that people
encounter some diseases such as “Hyperlipidemia” and “Type II diabetes” that
are more frequent than other low-frequency diseases, such as “Angioneurotic
edema” and “Quadriplegia”. Model learning will be biased toward frequent
labels if we train models on an imbalanced dataset without any re-balance
strategies. Therefore, re-balancing the learning of low- and high-frequency
code can improve the performance of medical coding approaches.

(a) ICD

(b) CCS

Figure 1. Distributions of ICD and CCS codes in the MIMIC-III dataset. We omit
a multitude of x-axis tick labels for better demonstration. The ICD codes are
sorted by code frequency, which decreases from left to right.

Some advanced ICD coding approaches mitigate imbalanced class problem based on
the nature of ICD codes. For example, the Label attention model (LAAT) (Vu et
al., 2021) proposed the hierarchical joint learning mechanism to capture the
structure of ICD codes to help predict the low-frequency ICD codes.

Inspired by the imbalanced problem between easy and hard samples in the object
detection task, we regard low and high-frequency medical codes as hard-
classified and easy-classified samples. Our framework leverages the focal loss
(Lin et al., 2017) to re-balance the attention on low- and high-frequency code
by adjusting the loss weights dynamically.

Code Association: There exist some connections between medical codes. For
example, in the ICD taxonomy system, “427.31” and “427.89” represent “Atrial
fibrillation” and “Other specified cardiac dysrhythmias”, respectively, which
can be classified into “Dysrhythmia”. It is challenging to capture
associations between medical codes to facilitate the automatic medical coding
task. Existing ICD coding models such as Multi-Filter Residual Convolutional
Neural Network (MultiResCNN) (Li and Yu, 2020) and Convolutional Attention for
Multi-Label classification (CAML) (Mullenbach et al., 2018) did not take
medical codes association into consideration.

Figure 2. An example of multitask coding with ICD and CCS codes

We design a multitask learning scheme to effectively transfer ICD code
association knowledge from the auxiliary task branch. The Clinical
Classifications Software (CCS) system maps high-dimensional ICD codes into
low-dimensional CCS codes. The code projection system is based on the pre-
defined medical knowledge of code association information provided by the
Healthcare Cost and Utilization Project (HCUP). The object is to jointly train
two medical coding branches, i.e., ICD and CCS coding branches. The CCS coding
branch is an auxiliary task to transfer ICD code association information.
Figure 2 shows an example of two-branch multitask medical coding, where two
ICD codes of “427.31” (Atrial fibrillation) and “427.89” (Other specified
cardiac dysrhythmias) map to the same CCS codes “106” (Dysrhythmia).

Figure 3. Distributions of clinical document lengths in the MIMIC-III
datasets.

Noisy and Lengthy Document: Clinical documents contain noisy information,
including error spelling and incoherent information, affecting the
representation learning from text. Learning rich and robust document features
is required to provide reliable medical coding results. Moreover, discharge
summaries in the MIMIC-III dataset are extremely long, $96.3$% and $97.2$%
documents’ lengths exceed $512$ tokens in the MIMIC-III-full and MIMIC-III-50
datasets. Figure 3 shows that most documents’ lengths are on the interval of
$[1000,2000]$, which are quite long compared with texts in other domains such
as the IMDB movie review dataset (Maas et al., 2011). To improve feature
learning from clinical documents, we design a module called Recalibrated
Aggregation Module (RAM). RAM suppresses the noise in the clinical notes by
injecting contextually enhanced document features. The cascaded convolution
structure of RAM provides the model with the capability to better deal with
lengthy clinical documents.

To address the three problems mentioned above, we propose a novel framework
called Multitask bAlanced and Recalibrated Network (MARN) for medical codes
prediction. This paper is an extension of our previous work that proposed a
model called MT-RAM (Sun et al., 2021) to jointly train two different medical
coding branches and achieve competitive improvement on overall evaluation
metrics. Our additional work includes the efforts to balance the learning from
frequent and infrequent codes, the extension of experiments on a full-code
dataset with improved performance, and further analysis on the problem of
imbalanced class and code association. Our proposed MARN model combines
multitask learning, bidirectional gated recurrent unit (BiGRU), RAM, label-
aware attention mechanism, and focal loss. We summarize our main contributions
as follows:

  * •

This paper deals with the imbalanced class problem and leverages focal loss to
dynamically redistribute the weight between low- and high-frequency codes.

  * •

We utilize a multitask learning scheme to jointly train two medical coding
systems with different granularities for capturing code associations.

  * •

Recalibrated Aggregation Module (RAM) is designed to refine textual features
extracted from lengthy and noisy clinical notes.

  * •

Experimental results show strong performance of our model across different
evaluation metrics on a widely used dataset MIMIC-III in a comparison against
several strong baseline models.

Our paper is organized as follows: Section 2 introduces related work; Section
3 describes our proposed MARN model; Section 4 conducts a series of
experiments and explores components of the MARN; Section 5 discusses the
future direction; Section 6 concludes the paper.

##  2\. Related Work

Automatic Medical Coding Automatic medical coding is a challenging but
essential task in medical text mining (Perotte et al., 2014). Early automatic
medical coding models mainly depend on complicated hand-craft document
features. Larkey and Croft (Larkey and Croft, 1996) designed an ICD code
classifier by conflating a potpourri of machine learning components, including
K-nearest neighbor, relevance feedback, and Bayesian independence classifiers.
Perotte et al. (Perotte et al., 2014) proposed two ICD coding models, i.e., a
flat and a hierarchy-based SVM classifier. Comparison experiments demonstrated
the superiority of the hierarchical-based model because it can capture the
hierarchical structure of ICD codes, which can benefit the prediction of ICD
codes.

Recent years have witnessed the advances in deep learning approaches.
Mullenbach et al. (Mullenbach et al., 2018) proposed Convolutional Attention
network for Multi-Label classification (CAML) for automatic ICD coding. Li and
Yu (Li and Yu, 2020) designed a Multi-Filter Residual Convolutional Neural
Network (MultiResCNN). Ji et al. (Ji et al., 2020) developed a dilated
convolutional network. Dong et al. (Dong et al., 2021) devised a hierarchical
label-wise attention network. Thanh et al. (Vu et al., 2021) developed a label
attention model (LAAT) to predict ICD codes. Xie et al. (Xie et al., 2019)
designed a multi-scale feature attention and structured knowledge graph
propagation (MSATT-KG), which is the combination of a densely connected
convolutional neural network (CNN), multi-scale feature attention and graph
convolutional neural networks. The densely connected CNN generates the n-gram
features, and the multi-scale feature attention captures the most informative
n-gram document features. Also, the MSAAT-KG uses a graph convolutional neural
network to obtain the hierarchical structure of ICD codes and the semantics of
each ICD code.

Figure 4. Overall architecture of MARN. $f$ represents the code frequency.
After utilizing focal loss, the losses of low- and high-frequency medical
codes can be balanced.

Imbalanced Class Problem The imbalanced class problem is founded on the
different distributions of class labels in the data set (Chawla et al., 2004).
The distribution of the label space demonstrates a head-tail shape, where head
and tail labels represent high and low-frequency labels, respectively. Most
multi-label data sets suffer from serious imbalanced class problem (Tahir et
al., 2012). The conventional way to alleviate the imbalanced class problem
divides instances into majority (classes with notable instances) and minority
groups (classes with few instances) (Charte et al., 2015). Then, sampling
algorithms are applied to reduce the number of majority samples or increase
the number of minority samples. In past decades, many re-sampling approaches
have been proposed, such as Random Over-Sampling (ROS), SMOTE (Chawla et al.,
2002) and Random Under-Sampling(RUS) (Kotsiantis and Pintelas, 2003). However,
reconstructing the data from a medical data set is expensive and inefficient.
Therefore, most automatic coding works tend to deal with the imbalanced class
problem by leveraging the hierarchical structure of ICD codes. For example,
LAAT (Vu et al., 2021) proposed the joint hierarchical mechanism to deal with
the imbalanced class problem. On the other hand, we determine to handle the
imbalanced class problem by dynamically redistributing the loss weights
between low- and high-frequency medical codes.

Multitask Learning Multitask learning (MTL) is inspired by human activities,
where people can utilize experience from other tasks to prompt the learning
process of the new task (Zhang and Yang, 2017). MTL contributes to the
information communication between related tasks by sharing parameters and
increases the training efficiency (Chandra et al., 2016; Yosinski et al.,
2014). Additionally, MTL can alleviate the over-fitting problem by
regularizing the learned model parameters to improve the model’s
generalization ability for each task branch (Liu et al., 2019). Many MTL-based
approaches have been proposed to verify the feasibility and effectiveness of
applying the MTL scheme to medical natural language processing (NLP) tasks
such as medical named entity recognition (Zhao et al., 2019; Chowdhury et al.,
2018), clinical information extraction (Suk et al., 2016; Bi et al., 2008) and
morality prediction (Si and Roberts, 2019). However, only a few narratives
deploy the multitask learning scheme on the automatic medical coding task.
Interian et al. (Interian et al., 2020) studied two different healthcare
tasks, i.e., medical code prediction and morality prediction, to perform
multitask training.

##  3\. Method

This section introduces our proposed model, Multitask bAalanced and
Recalibrated Network (MARN), which consists of a multitask learning scheme,
Recalibrated Aggregation Module (RAM) and optimization with the focal loss.
The overall architecture of MARN is presented in Figure 4. Firstly, we input
word embeddings of clinical notes pretrained with the word2vec algorithm
(Mikolov et al., 2013). Secondly, we adopt BiGRU as the feature extractor to
extract textual representations from medical documents. Next, the RAM is
plugged-in to improve the quality of document features learned by BiGRU and
better handle noisy and lengthy clinical documents. Then, we jointly train ICD
and CCS coding branches by utilizing the multitask learning scheme to capture
the associations among different medical codes. Finally, we utilize the focal
loss (FL) to alleviate the imbalanced class problem by redistributing the loss
weight on high-frequency and low-frequency labels.

###  3.1. Input Layer and Base Encoder

Let $D$ be an clinical document consisting of $n$ tokens,
$\\{w_{1},w_{2},\cdots,w_{n}\\}$. The word embedding matrix is obtained by
pretrained word2vec embeddings from each clinical document, denoted as
$\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{n}]^{\operatorname{T}}$,
assembling to word a vector $\mathbf{x}_{n}$ whose word embedding size is
$d_{e}$. We choose BiGRU as the backbone neural network to extract feature
from clinical documents. Hidden states of BiGRU on token $x_{i}$ (where
$i\in{1,2,\ldots,n}$) are denoted as:

(1) $\displaystyle\overrightarrow{\mathbf{h}_{i}}$
$\displaystyle=\overrightarrow{\operatorname{GRU}}(\mathbf{x}_{i},\overrightarrow{\mathbf{h}_{i-1}})$
(2) $\displaystyle\overleftarrow{\mathbf{h}_{i}}$
$\displaystyle=\overleftarrow{\operatorname{GRU}}(\mathbf{x}_{i},\overleftarrow{\mathbf{h}_{i+1}}),$

where $\overrightarrow{\operatorname{GRU}}$ and
$\overleftarrow{\operatorname{GRU}}$ denote forward and backward
$\operatorname{GRU}$s, respectively. Bidirectional hidden states are obtained
by horizontal concatenation of $\overrightarrow{\mathbf{h}_{i}}$ and
$\overleftarrow{\mathbf{h}_{i}}$, which is presented as:

(3)
$\displaystyle\mathbf{h}_{i}=\operatorname{Concat}(\overrightarrow{\mathbf{h}_{i}},\overleftarrow{\mathbf{h}_{i}})$

The dimension of each directional GRU is set as $d_{r}$, so that the
bidirectional hidden state $\mathbf{h}_{i}$ has dimension
$\mathbb{R}^{2d_{r}}$. The final hidden representation matrix is represented
as
$\mathbf{H}=[\mathbf{h}_{1},\mathbf{h}_{2},\dots,\mathbf{h}_{n}]^{\operatorname{T}}\in\mathbb{R}^{n\times
2d_{r}}$.

###  3.2. Recalibrated Aggregation Module

We design Recalibrated Aggregation Module (RAM) to provide the model with
better representation learning from lengthy and noisy document features, which
is inspired by the squeeze-and-excitation networks in the computer vision
domain (Hu et al., 2018). Three observations motivate to enhance the ability
to handle the lengthy and noisy features. 1) When the document is very long,
BiGRU also encounters a vanishing gradient problem, which may affect the
model’s performance when stacking additional layers upon the recurrent layers.
2) Lower-level semantic features extracted by BiGRU contain rich textural
information with more document details, while higher-level document features
capture abstract features with global receptive fields. The combination of
multi-level features can benefit the representation learning from clinical
documents. 3) Higher-level semantic features provide contextual information
for recalibrating noisy input document features. In the following, we will
give the details about the calculation flow of the RAM and how RAM addresses
the three aspects mentioned above.

Figure 5. The structure and feature flow of the RAM. “$\odot$” denotes as
element-wise multiplication. “$\oplus$” represents the element-wise addition
operation.

The structure and feature flow of RAM is illustrated in Figure 5. The RAM
consists of three kinds of blocks, i.e., “Up-sample Block”, “Lateral-sample
Block” and “Down-sample Block”, whose structure and dimensionalities are
demonstrated in Figure 6(a), 6(b), 6(c), respectively.

(a)

(b)

(c)

Figure 6. Illustrations for the structure and dimension transformation of
three basic blocks in RAM

We construct our basic blocks with stacked small convolution filters
($kernel~{}size=3$) to have an appropriate receptive field (Peng et al.,
2017). To simplify the description, we regard the operations of “Down-sample
Block”, “Lateral-sample Block” and “Up-sample Block” as
$\mathcal{F_{DB}}(\cdot)$, $\mathcal{F_{LB}}(\cdot)$ and
$\mathcal{F_{UB}}(\cdot)$. The calculations of three blocks could be
represented as following:

(4) $\displaystyle\mathcal{F_{DB}}(\cdot)$ $\displaystyle=\mathcal{F}_{3\times
3}^{l}(\operatorname{tanh}(\mathcal{F}_{3\times 3}^{d}(\cdot)))$ (5)
$\displaystyle\mathcal{F_{LB}}(\cdot)$ $\displaystyle=\mathcal{F}_{3\times
3}^{l}(\operatorname{tanh}(\mathcal{F}_{3\times 3}^{l}(\cdot)))$ (6)
$\displaystyle\mathcal{F_{UB}}(\cdot)$ $\displaystyle=\mathcal{F}_{3\times
3}^{l}(\operatorname{tanh}(\mathcal{F}_{3\times 3}^{u}(\cdot)))$

where $\mathcal{F}_{3\times 3}^{d}$ is a $3\times 3$ convolution layer
followed by the BatchNorm operation, where the number of output channels is
reduced to half of the input channels. The operation of $\mathcal{F}_{3\times
3}^{l}$ is quite similar to $\mathcal{F}_{3\times 3}^{d}$ except that
$\mathcal{F}_{3\times 3}^{l}$ retains the input channels’ number.
$\mathcal{F}_{3\times 3}^{u}$ performs the deconvolution operation that
doubles the input channels, and the output feature is passed through the Batch
Normalization layer.

The RAM is divided into three stages: feature abstraction, feature smoothing,
and feature aggregation. Firstly, the RAM utilizes cascaded convolutional
blocks to get abstract features which have larger receptive fields by
comparing with the input features of BiGRU. Each convolutional layer
incorporates $k$-grams ($k$ is the convolutional kernel size) into one-gram,
so that large-scale n-grams’ information can be embedded into one-gram if
input features are passed through cascaded convolutional blocks. In this way,
the RAM has the better ability to handle lengthy clinical documents. Secondly,
the abstract features are processed to smooth the input features by a
convolutional blocks whose input dimension sizes are the same as the output
ones. We denote this stage as the feature smoothing because of no variation on
feature dimensions. Thirdly, we use de-convolutional blocks to restore feature
dimensions (consistent receptive fields) and avoid the feature
inconsistence222Experimental results show that the de-convolutional up-sample
block outperforms the simple linear projection across all scores by 0.05%
1.73%., in order to facilitate the aggregation of abstract feature (contain
context-aware information) and lower-level feature (contain detailed document
information) to generate richer document representations for recalibrating
noisy input features. When dealing with long clinical documents, the RAM
module does not worsen the vanishing gradient problem caused by the BiGRU
backbone because of identity paths in the convolutional blocks and the
activation function.

Feature Abstraction: We leverage two Down-sample blocks to abstract the input
document features $\mathbf{H}$. The computation process is denoted as:

(7) $\displaystyle\mathbf{D_{1}}$ $\displaystyle=\mathcal{F_{DB}}(\mathbf{H})$
(8) $\displaystyle\mathbf{D_{2}}$
$\displaystyle=\mathcal{F_{DB}}(\mathbf{D_{1}}),$

where the higher-level document features $\mathbf{D_{1}}\in\mathbb{R}^{n\times
d_{r}}$ and $\mathbf{D_{2}}\in\mathbb{R}^{n\times 0.5d_{r}}$ are extracted as
the input features for next two stages.

Feature Smoothing: We use the Lateral-sample block to further extract high-
level features and take the feature $\mathbf{D_{2}}$ as input, i.e.,

(9) $\displaystyle\mathbf{L_{1}}$
$\displaystyle=\mathcal{F_{LB}}(\mathbf{D_{2}}),$

where the output feature of this stage is represented as
$\mathbf{L_{1}}\in\mathbb{R}^{n\times 0.5d_{r}}$.

Feature Aggregation: Two Up-sample blocks are utilized to recover the
dimension of the feature $\mathbf{L_{1}}$ for keeping receptive field
consistent when fusing the output features with original input features
$\mathbf{H}$. At the same time, we use two different fusion operations,
element-wise addition and multiplication, to inject lower-level features
$\mathbf{H}$ and $\mathbf{D_{1}}$ into features containing rich contextual
information. The output features $\mathbf{O}$ is followed by a
$\operatorname{tanh}$ function to activate output features and a dropout layer
is added after activated features to prevent overfitting. This conflation
process is demonstrated as:

(10) $\displaystyle\mathbf{U_{1}}$
$\displaystyle=\mathcal{F_{UB}}(\mathbf{L_{1}})\oplus\mathbf{D_{1}},$ (11)
$\displaystyle\mathbf{O}$
$\displaystyle=\operatorname{tanh}(\mathcal{F_{UB}}(\mathbf{U_{1}})\odot\mathbf{H}),$

where $\mathbf{U_{1}}\in\mathbb{R}^{n\times d_{r}}$ is the fused feature and
$\mathbf{O}\in\mathbb{R}^{n\times 2d_{r}}$ is the final output of RAM.

###  3.3. Attention Classification Layers

The feature $\mathbf{O}$ output by the RAM is label-agnostic. We deploy the
label-aware attention mechanism to connect the label information of each
medical code with different positions of the clinical document feature vector
$\mathbf{O}$. We set two medical coding branches, the ICD and CCS coding
branches, with separate label-aware attention mechanisms shown in Figure 7.

Figure 7. An illustration of the label attention layer. “$\otimes$” represents
the matrix multiplication operation.

To simplify the description of the two coding branches, we combine two
subscripts of ICD and CCS codes into one subscript $m$ because the calculation
are the same in two branches, where $m$ denotes either ICD or CCS code, and
omit the bias item in the equations. The attention score of each medical code
is computed as:

(12) $\mathbf{A}_{m}=\operatorname{Softmax}(\mathbf{O}\mathbf{Q}_{m}),$

where $\mathbf{O}$ is the output feature from the RAM,
$\mathbf{Q}_{m}\in\mathbb{R}^{2d_{r}\times d_{m}}$ denotes the trainable
parameter matrix of the queries in the label-aware attention layer, and
$d_{m}$ refers to the number of target medical codes. The label attentive
document features $\mathbf{V}_{m}\in\mathbb{R}^{d_{m}\times 2d_{r}}$ are
generated by multiplying attention score matrix $\mathbf{A}_{m}$ with the
calibrated feature $\mathbf{O}$ from RAM, i.e.,

(13) $\mathbf{V}_{m}=\mathbf{A}_{m}^{\operatorname{T}}\mathbf{O}.$

The label-aware attention mechanism can capture the code-related information
and encode them into documents. Then, the label attentive features
$\mathbf{V}_{m}$ are transformed into score vectors
$\mathbf{Y}_{m}\in\mathbb{R}^{d_{m}\times 2d_{r}}$ of medical codes by using
the fully-connected layer. We pass the score vectors through the sum pooling
operation followed by a Sigmoid activation function to generate probabilities
$\mathbf{\hat{y}}_{m}$ for final medical code prediction, which are denoted
as:

(14) $\displaystyle\mathbf{\mathbf{Y}}_{m}$
$\displaystyle=\mathbf{W}_{m}\mathbf{V}_{m}^{\operatorname{T}}$ (15)
$\displaystyle\mathbf{\hat{y}}_{m}$
$\displaystyle=\sigma(\operatorname{Pooling}(\mathbf{\mathbf{Y}}_{m})),$

where $\mathbf{W}_{m}\in\mathbb{R}^{d_{m}\times 2d_{r}}$ are the learnable
parameters of the fully-connected layer and $\sigma$ is the Sigmoid activate
function.

###  3.4. Multitask Learning with Focal Loss

We perform multitask training to incorporate the two medical coding branches
for the ICD and CCS codes, respectively. The probabilities of medical codes,
$\mathbf{\hat{y}}_{i}$, produced by the label-aware classification layer are
fed into the loss function separately. The sparsity of codes poses a severe
imbalanced class problem, which we alleviate using the focal loss when
training the model (Lin et al., 2017). The focal loss of each medical coding
branch $m$ can be written as:

(16)
$\mathcal{FL}_{m}=\sum_{i=1}^{d_{m}}[-y_{i}\alpha(1-\hat{y}_{i})^{\gamma}\log(\hat{y}_{i})-(1-y_{i})(1-\alpha)\hat{y}_{i}^{\gamma}\log(1-\hat{y}_{i})],$

which is the sum of losses over all $d_{m}$ medical codes present in the
$m$-th coding system. The parameter $\alpha$ represents the weighting factor
that balances the loss for different classes ($y=1$ or $y=0$ depending on
whether the code was present or not). We set $\alpha=0.999$, which places a
strong emphasis on instances with a certain code present (i.e., $y=1$), to
reflect the sparsity of the codes. In addition, the focal loss includes a
modulating factor, $(1-\hat{y}_{i})^{\gamma}$. If $\gamma>0$, the modulating
factor places less emphasis on confident predictions and more emphasis on
uncertain predictions, further making the model focus on the learning of codes
that are difficult to classify, which often are the low-frequency codes. The
loss weights of low- and high-frequency codes are adjusted by the confidences
of prediction dynamically. Note that for simplicity in (16) we omitted the
subscript $m$ that identifies the code branch in the ground truth labels
$\mathbf{y}_{m}$ and prediction probabilities $\mathbf{\hat{y}}_{m}$.

We treat the medical code prediction as a multitask problem, and consequently
formulate the joint loss as:

(17)
$\displaystyle\mathcal{FL}_{M}=\lambda_{d}\mathcal{FL}_{d}+\lambda_{s}\mathcal{FL}_{s},$

where $\mathcal{FL}_{d}$ and $\mathcal{FL}_{s}$ denote the focal losses for
the ICD and CCS coding branches, and $\lambda_{d}$ and $\lambda_{s}$ are the
loss weights of the ICD and CCS coding branches.

##  4\. Experiments

In this section, we evaluate the effectiveness of our proposed model MARN on
public real-world datasets. The source code is available at
https://github.com/VRCMF/MARN.

###  4.1. Datasets

MIMIC-III (ICD codes): The third version of Medical Information Mart for
Intensive Care (MIMIC-
III)333https://mimic.physionet.org/gettingstarted/access/ is a large, open-
access dataset consisting of clinical data associated with above 40,000
inpatients in critical care units of the Beth Israel Deaconess Medical Center
between 2001 and 2012 (Johnson et al., 2016). Following Mullenbach et al.
(Mullenbach et al., 2018) and Li and Yu (Li and Yu, 2020), we use the
discharge summaries as the input clinical documents. Human experts annotate
each summary document with corresponding diagnosis and procedure codes. The
first data set is the full data set with 8,921 unique ICD-9-CM codes in total.
MIMIC-III full codes data set has 52,722 discharge summaries, with 47,719,
1,631, and 3,372 documents for training, validation, and testing. The second
data set is the MIMIC-III top-50 codes data set, for which we divide all
discharge summaries documents based on the patient IDs and generate the top 50
most frequent ICD codes. The top-50 data set has 8,067 discharge summaries for
training, and 1,574 and 1,730 documents for validation and testing,
respectively. We refer two data set of ICD codes as MIMIC-III-full (ICD codes)
and MIMIC-III-50 (ICD codes).

MIMIC-III (CCS codes): We leverage the ICD-CCS mapping scheme maintained by
the HCUP 444www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp to project the ICD
codes into a lower-dimensional CCS codes. The top-50 ICD codes and full ICD
codes (8,921) are converted into top-38 CCS codes and full CCS codes (295).
The MIMIC-III data sets of CCS codes share the same discharge summaries with
the MIMIC-III ICD codes data set for training, validation, and testing. We
denote the full and the top-38 CCS code data sets as MIMIC-III-full (CCS
codes) and MIMIC-III-38 (CCS codes) , respectively.

###  4.2. Settings

Data Preprocessing: Following the previous works, we remove the non-alphabetic
tokens, such as punctuation and numbers, from clinical documents. We transform
all tokens into the lowercase format and change all tokens appearing in fewer
than three notes into the ‘UNK’ token. The medical word embeddings are
established by the word2vec technique from all discharge summaries. The
dimension of word embedding $d_{e}$ is set as $100$, consistent with previous
works. We set the maximum length of each document as 4,000, with the exceeded
part truncated.

Evaluation Metrics: We use the same evaluation metrics as previous works to
validate the effectiveness of our proposed model on the data sets of two kinds
of medical codes. The evaluation metrics include macro-averaged and micro-
averaged AUC-ROC (area under the receiver operating characteristic curve),
macro-averaged and micro-averaged F1, precision at $k$ (dubbed as ‘P@$k$’,
where $k\in\\{8,15\\}$). P@$k$ is the precision score indicating the top-$k$
scored predictions in the ground truth labels.

Hyper-parameter Tuning: Our implementation details are as follows. We train
our model with the optimizer Adam (Kingma and Ba, 2014) and set the learning
rate to $0.001$. The batch sizes of the top-n ($n\in\\{38,50\\}$) and full
code data are 16 and 64, respectively. We apply the early stopping trick to
exit the model training by monitoring the P@$k$ score, which avoids the model
over-fitting. The training will stop if the P@$k$ score on the validation set
does not improve in $10$ rounds. We set the kernel size of each block in the
RAM to 3. The dropout rate of the RAM is 0.2. For the multitask learning, we
set ICD scaling factor $\lambda_{d}$ and CCS scaling factors $\lambda_{s}$ as
0.7 and 0.3, respectively. In the focal loss, the weighting factor $\alpha$ is
0.999, and the focusing parameter $\gamma$ is 2, tuned from 1 to 5. Section
4.6 studies the detailed experimental settings about hyperparameters of the
multitask learning and the focal loss.

###  4.3. Baselines

CNN (Mullenbach et al., 2018): The vanilla CNN model utilizes a max-pooling
Convolutional Neural Network (Kim, 2014) to predict ICD codes.

BiGRU (Mullenbach et al., 2018): This model uses a bidirectional recurrent
architecture with gated recurrent units as the feature extractor for ICD
coding.

CAML (Mullenbach et al., 2018): Convolutional Attention network for Multi-
Label classification (CAML) uses a convolutional neural network to extract the
document features and the label-wise attention mechanism to enhance feature
learning.

DR-CAML (Mullenbach et al., 2018): Description Regularized-CAML (DR-CAML) is
an extension model of the CAML, which incorporates textual descriptions of ICD
codes to regularize the CAML model.

MultiResCNN (Li and Yu, 2020): Multi-Filter Residual Convolutional Neural
Network (MultiResCNN) leverages a convolutional layer with multiple filters to
capture various text patterns and adopts residual block to increase the
receptive field on the model.

LAAT (Vu et al., 2021): Vu et al. design the new label attention model (LAAT)
by choosing bidirectional Long-Short Term Memory (BiLSTM) as the feature
extractor and deploying a label self-attention mechanism to learn label-
specific vectors for ICD code predictions.

JointLAAT (Vu et al., 2021): JointLAAT extends the LAAT by applying a
hierarchical joint learning model to capture the hierarchical structure of ICD
codes.

Fusion (Luo et al., 2021): Fusion utilizes compressed convolutional layer to
encode clinical notes into informative local features, which are fused into
output representations for ICD code predictions.

MDBERT (Zhang and Jankowski, 2022): Medical Document BERT (MDBERT) is a
bottom-up hierarchical framework that combine features in word-level,
sentence-level, and document-level to efficiently encode long documents. The
hierarchical encoding model is first proposed by Yang et al.(Yang et al.,
2016) and applied to ICD coding by Dong et al. (Dong et al., 2021).

###  4.4. Results

MIMIC-III-50 (ICD code): Table 1 shows the experimental results of baseline
models and our proposed model on MIMIC-III-50 (ICD code) data set. We observe
that the MARN outperforms the other models (CNN, BiGRU, CAML, DR-CAML,
MultiResCNN  and MT-RAM) clearly across all evaluation metrics. The JointLAAT
model uses a hierarchical joint learning mechanism to deal with the imbalanced
class issue. Compared with JointLAAT, our proposed model (MARN) improves
macro-AUC, micro-AUC, macro-F1, micro-F1, P@5 scores by 0.2%, 0.1%, 2.1%, 0.2%
and 0.2%, respectively. The MARN has significant improvements, especially on
the macro-F1 score, by 7.6%, 10.6%, 19.8%, 7.6%, and 3.0% compared with
MultiResCNN, DR-CAML, CAML, BiGRU, CNN, and MT-RAM. Fusion improved all scores
about 0.1% $\sim$ 0.7% compared with the MARN.

In recent years, the pretrained language models with the transformer
architecture such as BERT (Devlin et al., 2019) have dominated many natural
language processing tasks (Lee-Thorp et al., 2021). However, applying the BERT
to medical coding tasks suffers from the limited document sequence (512
tokens) (Ji et al., 2021a) on the medical coding task. MDBERT, a BERT-based
ICD coding framework, achieved competitive performance on the MIMIC-III-50
(ICD code) data set, while the MARN outperformed the MDBERT by 0.9%, 1.1%,
2.3%, 2.6%, and 1.9% in macro-AUC, micro-AUC, macro-F1, micro-F1, and P@5
scores.

Models | AUC-ROC | F1 | P@5
---|---|---|---
Macro | Micro | Macro | Micro
CNN | 87.6 | 90.7 | 57.6 | 62.5 | 62.0
BiGRU | 82.8 | 86.8 | 48.4 | 54.9 | 59.1
CAML | 87.5 | 90.9 | 53.2 | 61.4 | 60.9
DR-CAML | 88.4 | 91.6 | 57.6 | 63.3 | 61.8
MultiResCNN | 89.9 | 92.8 | 60.6 | 67.0 | 64.1
MT-RAM | 92.1 | 94.3 | 65.2 | 70.7 | 66.4
LAAT | 92.5 | 94.6 | 66.6 | 71.5 | 67.5
JointLAAT | 92.5 | 94.6 | 66.1 | 71.6 | 67.1
Fusion | 93.1 | 95.0 | 68.3 | 72.5 | 67.9
MDBERT | 91.8 | 93.6 | 65.9 | 69.2 | 65.4
MARN(ours) | 92.7 | 94.7 | 68.2 | 71.8 | 67.3
Table 1. MIMIC-III-50 (ICD code) data set results (in %).

MIMIC-III-full (ICD code): Table 2 shows the results of the MARN and other
strong baseline models. The MARN performs better on macro-F1, micro-F1, P@8,
and P@15 scores than other baseline models. When compared with the state-of-
the-art model (JointLAAT), our proposed model has improved the scores of
macro-F1, micro-F1, P@8, P@15 by 0.9%, 0.9% 1.9%, and 1.2%, respectively.
Compared to convolution-based models including CNN, CAML, DR-CAML,
MultiResCNN, and Fusion, the MARN vastly increases the micro-F1 score by
16.5%, 4.5%, 5.5%, 3.2%, 3.0% respectively. The MARN improved the macro-F1,
micro-F1, p@8, and p@15 scores by 1.5%, 2.9%, 2.7%, 2.5%, comparing with the
BERT-based MDBERT model.

Models | AUC-ROC | F1 | P@k
---|---|---|---
Macro | Micro | Macro | Micro | 8 | 15
CNN | 80.6 | 96.9 | 4.2 | 41.9 | 58.1 | 44.3
BiGRU | 82.2 | 97.1 | 3.8 | 41.7 | 58.5 | 44.5
CAML | 89.5 | 98.6 | 8.8 | 53.9 | 70.9 | 56.1
DR-CAML | 89.7 | 98.5 | 8.6 | 52.9 | 69.0 | 54.8
MultiResCNN | 91.0 | 98.6 | 8.5 | 55.2 | 73.4 | 58.4
LAAT | 91.9 | 98.8 | 9.9 | 57.5 | 73.8 | 59.1
JointLAAT | 92.1 | 98.8 | 10.7 | 57.5 | 73.5 | 59.0
Fusion | 91.5 | 98.7 | 8.3 | 55.4 | 73.6 | -
MDBERT | 92.5 | 98.9 | 10.1 | 55.5 | 72.7 | 57.7
MARN(ours) | 91.3 | 98.8 | 11.6 | 58.4 | 75.4 | 60.2
Table 2. MIMIC-III-full (ICD code) data set results (in %).

MIMIC-III-50 (CCS code): We validate the BiGRU, CAML, DR-CAML, and the
MultiResCNN on the MIMIC-III-50 (CCS code) dataset and show the evaluation
results in Table 3. The MARN outperforms all baseline models by large margins
across all evaluation metrics. Significantly, MARN improves the macro-F1 and
micro-F1 scores by 8.3% and 6.2% compared with the MultiResCNN. Our model also
outperforms BiGRU, CAML, DR-CAML on macro-F1 and micro-F1 scores with 10%
$\sim$ 14% and 8% $\sim$ 13%, respectively.

Models | AUC-ROC | F1 | P@5
---|---|---|---
Macro | Micro | Macro | Micro
BiGRU | 87.6 | 90.7 | 57.6 | 62.5 | 62.0
CAML | 89.2 | 92.2 | 60.9 | 67.5 | 64.5
DR-CAML | 87.5 | 90.5 | 59.3 | 65.6 | 62.6
MultiResCNN | 89.2 | 92.4 | 62.9 | 68.8 | 64.6
MARN(ours) | 92.8 | 95.0 | 71.2 | 75.0 | 69.0
Table 3. MIMIC-III-50 results (CCS code) data set results (in %).

MIMIC-III-full (CCS code): We also evaluate the same baseline models on the
MIMIC-III full (CCS code) and compare our model to verify the effectiveness of
the MARN. Table 4 shows our model improves the macro-F1 score by 7.8%,
compared with the MultiResCNN model. Our model also promotes other evaluation
scores, with macro-AUC, micro-AUC, micro-F1, P@8, and P@15 increased by 3.1%,
0.9%, 3.2%, 2.5%, and 2.4%, respectively.

Models | AUC-ROC | F1 | P@k
---|---|---|---
Macro | Micro | Macro | Micro | 8 | 15
BiGRU | 91.2 | 96.4 | 50.1 | 68.4 | 81.1 | 64.0
CAML | 88.8 | 96.1 | 44.4 | 66.5 | 80.5 | 63.6
DR-CAML | 85.7 | 95.5 | 41.3 | 66.0 | 78.9 | 62.5
MultiResCNN | 90.6 | 96.5 | 50.8 | 69.0 | 81.8 | 64.8
MARN(ours) | 93.9 | 97.4 | 58.6 | 72.2 | 84.3 | 67.2
Table 4. MIMIC-III-full (CCS code) data set results (in %).

###  4.5. Detailed Analysis of MARN

This section studies the properties of the proposed MARN model through several
research questions.

#### How does each component of MARN affect the prediction?

We conduct experiments to validate the effectiveness of each component of the
MARN on the MIMIC-III-50 (ICD code) and MIMIC-III-full (ICD code) datasets,
with the following specific building components considered:

  * •

Multi-task Learning scheme (MTL)

  * •

Recalibrated Aggregation Module (RAM)

  * •

Focal Loss (FL)

From Table 5 and Table 6, we can observe that all components contribute to the
performance improvement, and they are complementary to each other. The
multitask learning scheme has a higher performance gain on the MIMIC-III-50
(ICD code) data set than the RAM, while the MIMIC-III-full (ICD code) data set
shows the opposite situation. The model optimized with the focal loss
outperforms the one with BCE loss.

Models | AUC-ROC | F1 | P@5
---|---|---|---
Macro | Micro | Macro | Micro
MARN | 92.7 | 94.7 | 68.2 | 71.8 | 67.3
w/o $\operatorname{MTL}$ | 91.9 | 94.0 | 64.4 | 69.4 | 66.1
w/o $\operatorname{MTL+FL}$ | 91.7 | 93.4 | 62.4 | 68.1 | 64.7
w/o $\operatorname{RAM}$ | 92.3 | 94.3 | 64.8 | 69.9 | 66.5
w/o $\operatorname{RAM+FL}$ | 91.8 | 94.1 | 64.6 | 69.9 | 66.2
w/o $\operatorname{MTL+RAM+FL}$ | 91.2 | 93.4 | 59.2 | 67.2 | 65.5
Table 5. Ablation results (in %) of MIMIC-III-50 (ICD code) Models | AUC-ROC | F1 | P@k
---|---|---|---
Macro | Micro | Macro | Micro | 8 | 15
MARN | 91.3 | 98.8 | 11.6 | 58.4 | 75.4 | 60.2
w/o $\operatorname{MTL}$ | 89.9 | 98.6 | 10.5 | 57.1 | 73.5 | 58.5
w/o $\operatorname{MTL+FL}$ | 89.1 | 98.4 | 9.1 | 55.6 | 72.9 | 58.0
w/o $\operatorname{RAM}$ | 90.4 | 98.6 | 10.3 | 56.0 | 72.6 | 57.9
w/o $\operatorname{RAM+FL}$ | 88.8 | 98.2 | 7.4 | 50.7 | 69.5 | 54.8
w/o $\operatorname{MTL+RAM+FL}$ | 88.9 | 98.3 | 6.8 | 51.5 | 69.9 | 54.7
Table 6. Ablation results (in %) of MIMIC-III-full (ICD code).

#### How compatible are the building blocks with different base models?

We choose two convolution-based models (i.e., CAML, MultiResCNN) and an RNN
model (BiGRU) to explore the compatibility of different building modules. We
term these models as base models. Figure 8 shows that the MTL, RAM, and FL can
improve other base models’ performance. Significantly, the BiGRU model
optimized with focal loss gains better performance than the CAML and
MultiResCNN.

Figure 8. Macro-F1 scores (in %) of different base models including the
different building blocks of MARN

#### Can multitask learning connect different medical coding systems?

Figure 9. The first two principal components of ICD and CCS code embeddings.

We leverage Principal Component Analysis (PCA) to reduce the dimension of
document features and plot the embeddings of ICD codes and CCS codes in the
resulting two-dimensional space. Figure 9 demonstrates that the cloud of ICD
codes surrounds the cluster of CCS codes, reflecting the mapping from ICD to
CCS codes. We also notice that ICD codes associated with a particular CCS code
are often clustered around the respective CCS code, as shown for several
representative CCS codes and their corresponding ICD codes in Fig. 10. To
study this phenomenon in detail, we define a circular region around each CCS
code using 10% of the longest distance between two ICD codes as the radius. We
then calculate the number of significant CCS codes, defined as CCS codes, such
that their respective regions contain significantly more relevant ICD codes
than expected by chance, where relevant ICD codes are those that are known to
map to the CCS code. We approximate the distribution of the number of relevant
ICD codes within a region using the binomial distribution:

(18) $\displaystyle P(k)=Binomial(k|n,p)$

where $n$ is the total number of ICD codes in the region, and $p$ is the
overall proportion of relevant ICD codes for the respective CCS code. For each
CCS code, we define a threshold $T$ (the number of relevant ICD codes in the
CCS code region), such that $P(k>T)<0.1$, and if the observed number of
relevant codes within a region exceeds the threshold, we consider the CCS code
as significant. As a result, there are 157 significant CCS codes among the
total of 295 CCS codes. Hence, we conclude that the MTL can establish
informative connections between related ICD codes to benefit medical code
prediction.

(a)

(b)

(c)

(d)

Figure 10. The embeddings of representative significant CCS codes and their
corresponding ICD codes. We see that the relevant ICD codes are clustered
around the respective significant CCS code, reflecting the ability of MARN to
learn representations that capture informative relationships between the
codes.

#### Does the model optimized with focal loss balance the learning between
low- and high-frequency codes?

We plot the normalized loss value of each code to explore whether the focal
loss can balance the loss of high- and low-frequency ICD codes. Firstly, we
train the model with different loss functions, take a forward pass of the
trained model, and calculate a unified loss function, i.e., binary cross
entropy (BCE) loss, for a fair comparison. Then, we normalize the loss of each
ICD code by dividing the BCE loss by the frequency of the code and the total
number of documents. Fig 11 shows the normalized loss curve of models trained
by the BCE loss and focal loss. We see that optimizing the models with focal
loss improves results overall especially for low-frequency codes. The
normalized loss of the model optimized by focal loss is balanced compared with
the model optimized by BCE loss. Thus, we can conclude that the model
optimized with BCE loss can not balance the learning of high- and low-
frequency codes. In contrast, the model optimized with focal loss can
effectively handle the imbalanced class problem in this study.

Figure 11. Normalized binary cross entropy loss of each ICD code, with x-axis
sorted by code frequency. The high-frequency codes are on the left, the low-
frequency codes on the right.

###  4.6. Hyperparameter Studies

We study two hyperparameters of the multitask learning scheme and the focal
loss in this section.

Figure 12(a) shows that the predictive performance of the MARN on MIMIC-III
dataset by applying different $\alpha$. The variations of the MARN’s
performance is small, which means the adjustment of the hyper-parameter
$\alpha$ does not largely influence the evaluation results. The oscillations
of three scores are slight. We can find the optimal $\alpha$ is $0.999$.
Figure 12(b) shows that $\gamma$ largely effects the results of our proposed
model, and the optimal $\gamma$ is $2$. Based on these two figures, we found
that the $\gamma$ contributes more to balancing the learning of the high- and
low-frequency codes and the performance increment of the parameter $alpha$ is
slight.

(a) Evaluation results with $\gamma=2$.

(b) Evaluation results with $\lambda=0.999$.

Figure 12. Predictive performance on MIMIC-III dataset with different $\gamma$
and $\lambda$.

We evaluate our proposed MARN on the MIMIC-III dataset by setting different
loss weights of medical coding branches, i.e., $\lambda_{d}$ for the ICD
coding and $\lambda_{s}$ for the CCS coding, with results shown in Figure
13(a) and 13(b), respectively. Two figures are symmetrical because the
summation of these two coefficients is $1$. We take a common loss weighting
strategy for our MTL scheme, which is to assign same weights for each coding
tasks (Lin et al., 2021), and we fine-tune these loss weights to get the best
performance. Intuitively, the CCS coding branch as an auxiliary task should be
assigned with smaller loss weight to enable the association knowledge
transferred from the CCS coding branch to the ICD branch. The variations of
three scores are small in Figure 13(a) when $\lambda_{d}$ is from $0.5$ to
$0.9$. However, if we further increase the $\lambda_{d}$ to $1$ (i.e.,
removing the MTL scheme from the MARN), the Macro-F1, Micro-F1, and P@5
dramatically drop to $64.4$, $69.4$, and $66.1$. In summary, the evaluation
results are not sensitive to the value of the $\lambda_{d}$ and $\lambda_{s}$
and the model’s performance largely decreases if the MARN does not include the
MTL scheme.

(a) Evaluation results with different loss weight of ICD codes
($\lambda_{d}$).

(b) Evaluation results with different loss weight of CCS codes
($\lambda_{s}$).

Figure 13. Predictive performance on MIMIC-III dataset with different
$\lambda_{d}$ and $\lambda_{s}$ ($\lambda_{d}+\lambda_{s}=1$).

##  5\. Future work

In recent years, the transformer-based language model has become a new
paradigm for NLP tasks. With the support of self-attention mechanism, the
transformer-based models can capture token-dependent patterns for boosting
contextualized text learning. The transformer and its variants can suffer from
the quadratic memory and time complexity problem caused by the self-attention
mechanism. Although the efficient transformer-based models, such as Reformer
(Kitaev et al., 2020), Linformer (Wang et al., 2020) Longformer (Beltagy et
al., 2020), have been proposed, they also need a substantial amount of
computational resource for neural network training. By contrast, the GRU-based
model is positioned nicely on the Pareto frontier of the computation-
performance curve. In the next stage of our research, we plan to study how to
effectively utilize contextual embeddings to obtain semantically enriched
document features for medical code prediction.

##  6\. Conclusion

This paper proposes a novel model, Multitask bAalanced and Recalibrated
Network (MARN), to tackle three challenges of automated medical coding: the
imbalanced class problem, capturing code association, and dealing with lengthy
and noisy documents. We leverage the focal loss to alleviate the imbalanced
class issue by redistributing the loss weights between low and high-frequency
medical codes. We design the Recalibrated Attention Module (RAM) to inject
high-level semantic features into the original feature for noise suppressing.
The cascaded convolutional structure of the RAM can improve the representation
learning from long and noisy documents. The multitask learning scheme that
enables the code relationship knowledge transfer between two different coding
systems (i.e., ICD and CCS) is developed to capture the code association and
improve the coding performance. The experimental results show that our
proposed model outperforms competitive baseline models in the real-world
MIMIC-III database.

##  7\. Acknowledgments

This work was supported by the Academy of Finland (grant 336033) and EU H2020
(grant 101016775). We acknowledge the computational resources provided by the
Aalto Science-IT project. The authors wish to acknowledge CSC - IT Center for
Science, Finland, for computational resources.

## References

  * (1)
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020\.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ (2020).
  * Bi et al. (2008) Jinbo Bi, Tao Xiong, Shipeng Yu, Murat Dundar, and R Bharat Rao. 2008.  An improved multi-task learning approach with applications in medical diagnosis. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, Heidelberg, Berlin, 117–132.
  * Chandra et al. (2016) Rohitash Chandra, Abhishek Gupta, Yew-Soon Ong, and Chi-Keong Goh. 2016.  Evolutionary multi-task learning for modular training of feedforward neural networks. In _International Conference on Neural Information Processing_. Springer, 37–46.
  * Charte et al. (2015) Francisco Charte, Antonio J Rivera, María J del Jesus, and Francisco Herrera. 2015.  Addressing imbalance in multilabel classification: Measures and random resampling algorithms.  _Neurocomputing_ 163 (2015), 3–16.
  * Chawla et al. (2002) Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002.  SMOTE: synthetic minority over-sampling technique.  _Journal of Artificial Intelligence Research_ 16 (2002), 321–357.
  * Chawla et al. (2004) Nitesh V Chawla, Nathalie Japkowicz, and Aleksander Kotcz. 2004\.  Special issue on learning from imbalanced data sets.  _ACM Special Interest Group on Knowledge Discovery and Data Mining Explorations Newsletter_ 6, 1 (2004), 1–6.
  * Chowdhury et al. (2018) Shanta Chowdhury, Xishuang Dong, Lijun Qian, Xiangfang Li, Yi Guan, Jinfeng Yang, and Qiubin Yu. 2018.  A multitask bi-directional RNN model for named entity recognition on Chinese electronic medical records.  _BMC Bioinformatics_ 19, 17 (2018), 75–84.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Annual Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Dong et al. (2021) Hang Dong, Víctor Suárez-Paniagua, William Whiteley, and Honghan Wu. 2021.  Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation.  _Journal of biomedical informatics_ 116 (2021), 103728.
  * Horng et al. (2017) Steven Horng, David A Sontag, Yoni Halpern, Yacine Jernite, Nathan I Shapiro, and Larry A Nathanson. 2017.  Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning.  _PloS one_ 12, 4 (2017), e0174708.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 7132–7141.
  * Interian et al. (2020) Yannet Interian, Lara Reichmann, and Gilmer Valdes. 2020\.  Multitask learning from clinical text and acute physiological conditions differentially improve the prediction of mortality and diagnosis at the ICU.  _medRxiv_ (2020).
  * Ji et al. (2020) Shaoxiong Ji, Erik Cambria, and Pekka Marttinen. 2020\.  Dilated Convolutional Attention Network for Medical Code Assignment from Clinical Text. In _Proceedings of the 3rd Clinical Natural Language Processing Workshop at Empirical Methods in Natural Language Processing_. 73–78.
  * Ji et al. (2021a) Shaoxiong Ji, Matti Hölttä, and Pekka Marttinen. 2021a.  Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study.  _Computers in Biology and Medicine_ (2021).
  * Ji et al. (2021b) Shaoxiong Ji, Shirui Pan, and Pekka Marttinen. 2021b.  Medical Code Assignment with Gated Convolution and Note-Code Interaction. In _Findings of the Association for Computational Linguistics: Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_. Association for Computational Linguistics, 1034–1043.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016\.  MIMIC-III, a freely accessible critical care database.  _Scientific Data_ 3, 1 (2016), 1–9.
  * Kim (2014) Yoon Kim. 2014.  Convolutional Neural Networks for Sentence Classification.  _arXiv preprint arXiv:1408.5882_ (2014).
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020\.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_ (2020).
  * Koopman et al. (2015) Bevan Koopman, Guido Zuccon, Anthony Nguyen, Anton Bergheim, and Narelle Grayson. 2015.  Automatic ICD-10 classification of cancers from free-text death certificates.  _International Journal of Medical Informatics_ 84, 11 (2015), 956–965.
  * Kotsiantis and Pintelas (2003) SB Kotsiantis and PE Pintelas. 2003.  Mixture of expert agents for handling imbalanced data sets.  _Annals of Mathematics, Computing & Teleinformatics_ 1, 1 (2003), 46–55.
  * Larkey and Croft (1996) Leah S Larkey and W Bruce Croft. 1996.  Combining classifiers in text categorization. In _Proceedings of the 19th Annual International ACM SIGIR conference on Research and Development in Information Retrieval_. 289–297.
  * Lee-Thorp et al. (2021) James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. 2021.  FNet: Mixing Tokens with Fourier Transforms.  _arXiv preprint arXiv:2105.03824_ (2021).
  * Li and Yu (2020) Fei Li and Hong Yu. 2020\.  ICD coding from clinical text using multi-filter residual convolutional neural network. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 8180–8187.
  * Lin et al. (2021) Baijiong Lin, Feiyang Ye, and Yu Zhang. 2021.  A closer look at loss weighting in multi-task learning.  _arXiv preprint arXiv:2111.10603_ (2021).
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017.  Focal loss for dense object detection. In _Proceedings of the IEEE International Conference on Computer Vision_. 2980–2988.
  * Liu et al. (2019) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019\.  Multi-task deep neural networks for natural language understanding.  _arXiv preprint arXiv:1901.11504_ (2019).
  * Luo et al. (2021) Junyu Luo, Cao Xiao, Lucas Glass, Jimeng Sun, and Fenglong Ma. 2021.  Fusion: Towards Automated ICD Coding via Feature Compression. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. 2096–2101.
  * Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011\.  Learning word vectors for sentiment analysis. In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_. 142–150.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _arXiv preprint arXiv:1310.4546_ (2013).
  * Mullenbach et al. (2018) James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018.  Explainable Prediction of Medical Codes from Clinical Text. In _Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, 1101–1111.
  * O’malley et al. (2005) Kimberly J O’malley, Karon F Cook, Matt D Price, Kimberly Raiford Wildes, John F Hurdle, and Carol M Ashton. 2005.  Measuring diagnoses: ICD code accuracy.  _Health Services Research_ 40, 5p2 (2005), 1620–1639.
  * Park et al. (2000) Jong-Ku Park, Ki-Soon Kim, Tae-Yong Lee, Kang-Sook Lee, Duk-Hee Lee, Sun-Hee Lee, Sun-Ha Jee, Il Suh, Kwang-Wook Koh, So-Yeon Ryu, et al. 2000\.  The accuracy of ICD codes for cerebrovascular diseases in medical insurance claims.  _Journal of Preventive Medicine and Public Health_ 33, 1 (2000), 76–82.
  * Peng et al. (2017) Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. 2017.  Large kernel matters–improve semantic segmentation by global convolutional network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4353–4361.
  * Perotte et al. (2014) Adler Perotte, Rimma Pivovarov, Karthik Natarajan, Nicole Weiskopf, Frank Wood, and Noémie Elhadad. 2014.  Diagnosis code assignment: models and evaluation metrics.  _Journal of the American Medical Informatics Association_ 21, 2 (2014), 231–237.
  * Si and Roberts (2019) Yuqi Si and Kirk Roberts. 2019.  Deep patient representation of clinical notes via multi-task learning for mortality prediction.  _AMIA Summits on Translational Science Proceedings_ 2019 (2019), 779\.
  * Suk et al. (2016) Heung-Il Suk, Seong-Whan Lee, and Dinggang Shen. 2016\.  Deep sparse multi-task learning for feature selection in Alzheimer’s disease diagnosis.  _Brain Structure and Function_ 221, 5 (2016), 2569–2587.
  * Sun et al. (2021) Wei Sun, Shaoxiong Ji, Erik Cambria, and Pekka Marttinen. 2021\.  Multitask Recalibrated Aggregation Network for Medical Code Prediction. In _Proceedings of the Joint European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases_. Springer.
  * Tahir et al. (2012) Muhammad Atif Tahir, Josef Kittler, and Ahmed Bouridane. 2012\.  Multilabel classification using heterogeneous ensemble of multi-label classifiers.  _Pattern Recognition Letters_ 33, 5 (2012), 513–523.
  * Vu et al. (2021) Thanh Vu, Dat Quoc Nguyen, and Anthony Nguyen. 2021\.  A label attention model for icd coding from clinical text. In _Proceedings of the International Joint Conference on Artificial Intelligence_.
  * Wang et al. (2020) Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_ (2020).
  * Xie et al. (2019) Xiancheng Xie, Yun Xiong, Philip S Yu, and Yangyong Zhu. 2019\.  Ehr coding with multi-scale feature attention and structured knowledge graph propagation. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. Association for Computing Machinery, New York, NY, USA, 649–658.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 1480–1489.
  * Yosinski et al. (2014) Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014\.  How transferable are features in deep neural networks?  _arXiv preprint arXiv:1411.1792_ (2014).
  * Zhang and Jankowski (2022) Ning Zhang and Maciej Jankowski. 2022.  Hierarchical BERT for Medical Document Understanding.  _arXiv preprint arXiv:2204.09600_ (2022).
  * Zhang and Yang (2017) Yu Zhang and Qiang Yang. 2017.  A survey on multi-task learning.  _arXiv preprint arXiv:1707.08114_ (2017), 1–1.
  * Zhao et al. (2019) Sendong Zhao, Ting Liu, Sicheng Zhao, and Fei Wang. 2019\.  A neural multi-task learning framework to jointly model medical named entity recognition and normalization. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. AAAI Press, Honolulu, Hawaii, USA, 817–824.