#  Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes

Revathy Venkataramanan Artificial Intelligence Institute
University of South Carolina
Columbia, SC, USA
<EMAIL_ADDRESS>Kaushik Roy Artificial Intelligence Institute
University of South Carolina
Columbia, SC, USA
<EMAIL_ADDRESS>Kanak Raj Renjith Prasad Artificial Intelligence
Institute
University of South Carolina
Columbia, SC, USA
<EMAIL_ADDRESS>Artificial Intelligence Institute
University of South Carolina
Columbia, SC, USA
<EMAIL_ADDRESS>Yuxin Zi Artificial Intelligence Institute
University of South Carolina
Columbia, SC, USA
<EMAIL_ADDRESS>Vignesh Narayanan Amit Sheth Artificial Intelligence
Institute
University of South Carolina
Columbia, SC, USA
<EMAIL_ADDRESS>Artificial Intelligence Institute
University of South Carolina
Columbia, SC, USA
<EMAIL_ADDRESS>
###### Abstract

As people become more aware of their food choices, food computation models
have become increasingly popular in assisting people in maintaining healthy
eating habits. For example, food recommendation systems analyze recipe
instructions to assess nutritional contents and provide recipe
recommendations. The recent and remarkable successes of generative AI methods,
such as auto-regressive large language models, can lead to robust methods for
a more comprehensive understanding of recipes for healthy food recommendations
beyond surface-level nutrition content assessments. In this study, we explore
the use of generative AI methods to extend current food computation models,
primarily involving the analysis of nutrition and ingredients, to also
incorporate cooking actions (e.g., add salt, fry the meat, boil the
vegetables, etc.). Cooking actions are notoriously hard to model using
statistical learning methods due to irregular data patterns \- significantly
varying natural language descriptions for the same action (e.g., marinate the
meat vs. marinate the meat and leave overnight) and infrequently occurring
patterns (e.g., add salt occurs far more frequently than marinating the meat).
The prototypical approach to handling irregular data patterns is to increase
the volume of data that the model ingests by orders of magnitude.
Unfortunately, in the cooking domain, these problems are further compounded
with larger data volumes presenting a unique challenge that is not easily
handled by simply scaling up. In this work, we propose novel aggregation-based
generative AI methods, Cook-Gen, that reliably generate cooking actions from
recipes, despite difficulties with irregular data patterns, while also
outperforming Large Language Models and other strong baselines.

##  I Introduction

Alice Waters once said, ”We are what we eat, ” succinctly summarizing the
impact that food can have on people’s physical and mental well-being.
Increased access to knowledge about the impact of food choices, along with
rapid advances in health monitoring technology, has led to individuals being
more aware and cautious about their dietary choices [1]. There has been a
growing trend to utilize AI-based diet management systems that analyze food
recipes to provide an enhanced food recommendation experience to the user [2,
3]. Most of these systems incorporate discriminative models for the
identification of nutritional information in recipes to encourage healthy
eating habits (e.g., ingredients and their nutritional values). However, these
models lack a more comprehensive understanding of the recipe. For example,
they often do not account for cooking actions when providing recommendations
[4]. It is vital to include cooking actions in the decision-making of diet
management systems since they can significantly impact the nutritional value
of food[5], ensure food safety[6], and even produce harmful substances. For
instance, grilling meat at high temperatures produces cancer-causing
compounds[7, 8]. Figure 1 illustrates a recommendation system that analyses
the effect of cooking actions and its impact on the final recommendation.

Figure 1: The figure illustrates how understanding and modeling the cooking
actions, not just the ingredients and their nutrition values, significantly
impacts the final recommendation. Here meat inherently has nothing unhealthy
about it. However, marinating it and grilling at high temperatures causes
changes to its “healthiness” value.

Recent successes in Large Language Models (LLMs) have the potential to enable
a more comprehensive analysis of recipes, such as calorie count, macro-
nutrient content, and ingredient lists. This information can then be used to
identify healthy recipes and also suggest ingredient substitutions as per the
user’s health condition and food preferences [9]. LLMs can further be used to
identify common cooking actions, ingredients, and flavor profiles associated
with healthy dishes, which can be used to develop healthy eating
recommendations and meal plans.

However, recipes written in natural language pose unique challenges with
regard to natural language processing (NLP) that are not observed in
traditional NLP tasks. For example, recipes include highly variable textual
descriptions for the same cooking action (some people write “ marinate the
chicken and leave overnight” whereas others simply write “marinate the
chicken”). Furthermore, recipes consist of several elements such as
ingredients, utensils, temperature, and cooking actions (e.g., “marinate 1/2
cup of sliced chicken in a bowl”) and, crucially, the frequency of cooking
actions (“marinate”) relative to ingredients, quantities, and other
information (“1/2, “bowl and chicken”) is low. We refer to the issues of low-
frequency words and variation in lengths of cooking action descriptions
collectively as irregular data patterns. Thus, discriminative models struggle
due to the lack of distributional patterns with a high discriminatory value
for expressing complex relationships between different variables (e.g., the
effect of cooking actions).

With the rise of LLMs that have demonstrated remarkable successes with neural
network-enabled generative models, AI systems can potentially model more
complex phenomena, such as the impacts of cooking actions from the natural
language text in recipes [10]. However, these models still make a
discriminative model-type decision repeatedly (e.g., continuous next-word
prediction), except that they do it using large neural networks accompanied by
enormous volumes of training data. Therefore, the problems with irregular data
patterns for discriminative modeling persist and are, in fact, further
compounded at larger scales (e.g., imbalanced frequencies get more
imbalanced). Inspired by recent innovations in dynamical systems control that
utilize aggregation methods to handle irregular data patterns by (1)
distilling more regular aggregated information from irregular data patterns
and (2) leveraging high capacity function approximation capabilities of
higher-order polynomial fits to model complex relationships, we propose Cook-
Gen, a novel generative-AI method to generate cooking actions from recipe
texts [11]. We compare Cook-Gen with state-of-the-art generative and
discriminative baseline models and find that it consistently outperforms these
baselines.

##  II Related Work

Here we review previous work on modeling cooking actions from recipes.
Papadopoulos et al. represent cooking recipes using manually specified program
abstractions which included cooking actions, quantities, ingredients, and
other information from the recipes [12]. Recipescape, a cooking analysis tool,
used Stanford’s CoreNLP POS Tagger to extract verbs from recipes as a proxy
for cooking actions [13]. Walter et al. also used a POS tagger in a similar
context to reconstruct recipes using a workflow diagram of cooking actions.
Shidochi et al. manually extracted cooking actions from 15 recipes to suggest
substitute actions with similar effects [14]. Silva et al. used an off-the-
shelf annotation tool to annotate recipes for cooking actions as part of an
information retrieval system [15]. A common theme among the existing works is
the manual effort involved in cooking action modeling, underscoring the
challenge of automated modeling of cooking actions from recipes.

##  III Description of the Dataset and Task, Training and Evaluation Metrics

###  III-A Dataset Construction and Description

We utilize the Recipe1M dataset, which is a comprehensive collection of 1
million high-quality recipes containing titles, ingredients, cooking
instructions, and recipe images for each entry [16]. We then asked graduate
students between ages 23 and 27 from the University of South Carolina to
identify cooking actions in the recipe data for a randomly selected sample of
10000 recipe texts. Examples of recipe texts include, “bring the water to
boil”, “saute the vegetables in the wok”, or “ marinate the chicken”. Their
corresponding cooking actions are, “boil”, “saute” and “marinate”.

###  III-B Task Description

As mentioned in Section I, we experiment with both discriminative and
generative baselines. We describe the task description in the discriminative
and generative modeling settings.

#### Discriminative Modeling Setting

Let $R$ be the set of all recipes. Each recipe $r\in R$ is written in natural
language using a list of words $W_{r}$ and also consists of a title $T_{r}$, a
list of ingredients $I_{r}$, and a list of cooking actions $A_{r}$. Let
$S=\\{w_{m}\mid m\leftarrow\\{1,\ldots,M\\}\\}$ denote the set of all $M$
words across the $N$ recipes. Thus we have a recipe set defined as:

$R=\\{(W_{r},t_{r},l_{r},A_{r})\mid W_{r}\subseteq S,t_{r}\subset
S,I_{r}\subset S,A_{r}\subset S\\}$ (1)

We define a function $f:S\rightarrow\\{0,1\\}$ that predicts whether a given
word $w_{m}\in S$ is a cooking action or not. Thus we define the task of
predicting cooking actions in a recipe as a binary classification task, where
for each recipe $r$, we want to predict a sequence of binary labels
$y_{r}=\\{y_{1},\ldots,y_{M}\\}$, where
$y_{m}=(f(w_{m})\land\mathbb{I}(w_{m}\in W_{r}))$. Here $\mathbb{I}(w_{m}\in
W_{r})$ is an indicator function that evaluates to $1$ if word $w_{m}$ is
present in the recipe $r$ and evaluates to $0$ otherwise.

#### Generative Modeling Setting

Consider the same definition of a set of recipes $R$ as shown in Eq (1). Let
$X_{r}=\\{w_{1},\ldots,w_{L}\\}$ denote a partially complete natural language
description of the recipe $r$ consisting of $L$ words, i.e., $X_{r}\subset
W_{r}$ Formally, for a recipe $r\in R$, given a partial description $X_{r}$,
the task is to learn a function $f:X_{r}\rightarrow g_{r}$ that can generate
the set,

$g_{r}=X_{r}\cup W_{r}\setminus X_{r}\cup A_{r}$ (2)

###  III-C Training and Evaluation Metrics

The training metrics we consider in our experiments are cross-entropy (CE)
loss and weighted cross-entropy (WCE) loss. We use the WCE loss as there is a
class imbalance issue, i.e., the number of cooking action words across all
recipes compared to other words is relatively low. In the discriminative
modeling case, the CE and WCE loss reduces to the binary CE and WCE losses,
respectively. In the generative modeling case, only the CE loss (during word
prediction of the words in the set defined in Eq (2)) is considered. Since
there is a class imbalance in the dataset, we do not use accuracy and instead
use precision and recall as our evaluation metrics.

#### Time and Space Efficiency Considerations

Despite the remarkable progress made by state-of-the-art AI models in recent
years, they are not easily available to the general public. A key reason for
this is their significant resource requirements, such as long training and
inference times and high memory usage. For an AI system that can assist any
individual in making healthy food choices, it is necessary for it to be
resource-efficient, such as being able to run on a mobile phone and handle
daily queries without overburdening the system. Consequently, we also evaluate
the number of parameters and memory usage of the models used for
experimentation in both discriminative and generative modeling settings.

##  IV Methods and Evaluation

We divide our methods into two categories, (A) methods that work in the
discriminative modeling setting and (B) methods that work in the generative
modeling setting. For each category of methods, we provide describe the
methods, detail the experimental setup, and discuss their evaluation metric
scores.

###  IV-A Discriminative Modeling Methods and Evaluation

#### Data Preprocessing

Given the discriminative modeling task description in Section III-B, our aim
is to learn the binary function $f:S\rightarrow\\{0,1\\}$ that decides if a
recipe word $w_{m}\in W_{r}$ is a cooking action or not. Since our dataset
annotations consist of the full description of the cooking actions, e.g., for
a recipe consisting of ”marinate the meat and leave overnight”, the annotation
would be “marinate” (see Section III-A). We manually preprocess the
annotations to reflect binary ground truth labels corresponding to if a word
$w_{m}\in W_{r}$ is a cooking action or not. For example, the annotation
“marinate” is preprocessed as a one-hot-encoded vector of size $M$ for all the
$M$ words in $S$, with $1$ in the places corresponding to the words “marinate”
and $0$ for every other word.

#### Method Descriptions

We try a rule-based method (R), where we use a simple lookup from a predefined
list of cooking actions for each word to label the word as a cooking action or
not (the predefined list can be found here). Next, we use the fine-tuned
language models ELECTRA and XLNET [17, 18]. As mentioned in Section III-C, the
discriminative modeling task suffers a class imbalance issue. The models XLNET
and ELECTRA have specifically been trained to be robust to class imbalances in
the data. Finally, we also try industry standards that have shown particular
success in “binary tag prediction” due to the discriminative modeling task
essentially boiling down to predicting the tag of a word as either a cooking
action or not a cooking action. We experiment with Stanford’s NLP tools and
explosion AI’s SpaCy [19, 20].

#### Experiments and Evaluation

##### Model Configurations for the language models

For ELECTRA, we fine-tuned the electra-base-discriminator model obtained from
hugging face. Early stopping was employed to prevent the model from
overfitting. For XLNET, we fine-tuned the xlnet-large-cased model from hugging
face. Similar to ELECTRA, early stopping was also implemented during training
to prevent overfitting. For both ELECTRA and XLNET, the default pytorch
implementations for CE and WCE loss were used. For the WCE loss, the weight
assignment was $0.881$ for label $1$ and $0.119$ for label $0$.

##### Specifics of the SpaCy and Stanford’s NLP model

We use Stanford’s NER with its default configuration with the recognized named
entities being labeled with $1$ (other words $0$), and SpaCy’s part-of-speech
(POS) tagger model ‘en-core-web-lg’ to label cooking actions (label $1$ if POS
is a “verb” for a word and $0$ for other words).

RSpaCyStanfordX-CEE-CEX-WCEE-
WCE$30$$40$$50$$60$$70$$80$$90$$100$Discriminative Modeling
EvaluationsPrecisionRecall Figure 2: Performance results of various
discriminative modeling methods to predict cooking actions. X-CE, E-CE, X-WCE,
E-WCE, R, SpaCy, Stanford represents XLNET with CE loss, ELECTRA with CE loss,
XLNET with WCE loss, ELECTRA with WCE loss, The rule-based system (R), SpaCy’s
POS tagger, and Stanford’s NER model respectively.

The evaluation scores for various discriminative modeling methods are
presented in Figure 2. The rule-based system (R) exhibits the worst
performance, which is unsurprising given the challenges of creating precise
rules for determining cooking actions from recipes, particularly when the data
is characterized by irregular patterns. Although SpaCy’s POS tagger and
Stanford’s NER model, the two industry standards, perform somewhat better,
they employ Convolutional Neural Network and Conditional Random Field-based
approximation models that can only capture linear chain structured contextual
dependencies between the recipe words. As a result, we anticipate that self-
attention transformer-based techniques such as XLNET and ELECTRA will
outperform these models since they can model more complex interactions or
dependencies between the recipe words and are also designed to be robust to
irregular data patterns. As predicted, XLNET and ELECTRA demonstrate
significant performance improvements. Nonetheless, employing WCE instead of CE
in the training loss results in decreased precision, indicating that the model
with CE loss may have overfit spurious data patterns. We conclude from these
experiments that discriminative modeling may not be suitable for cooking
action prediction since discriminatory patterns in the data are not present
with regularity. Generative modeling has been demonstrated to be highly
effective in situations where easily recognizable discriminatory patterns are
absent, with the most notable example being large self-attention transformer-
based models for language modeling. As a result, we will now move on to an
experimental analysis in the generative modeling context.

###  IV-B Generative Modeling Methods and Evaluation

#### Data Preprocessing

Given the generative modeling task description in Section III-B, our aim is to
learn the function $f:X_{r}\rightarrow g_{r}$, that generates $g_{r}$ as
defined in Eq (2). Therefore we first tokenize recipe $r$ into a set of words
$W_{r}$ and randomly sample the subset $X_{r}$ from $W_{r}$. The set of
cooking actions $A_{r}$ is already part of the annotated dataset.

#### Method Descriptions

While self-attention transformer-based generative models have shown impressive
success in various tasks, their high resource usage makes them challenging for
ordinary users. Additionally, scaling up the model parameters and data does
not necessarily solve the problem of irregular data patterns. To address this
issue, aggregation-based methods can be utilized to enhance the robustness of
the model to such irregularities. These methods handle data pattern
irregularities by learning “aggregate patterns” that regularly appear in the
data. To create small-scale generative models that can also handle irregular
data patterns, we conducted experiments using two methods that have proven
universal approximation capabilities: (1) a two-layer feedforward neural
network with ReLU activations that aggregates the layer outputs across input
words, and (2) a polynomial approximation-based aggregation method that raises
word representations to higher-order polynomial powers before aggregating
them. We also incorporated a position embedding layer to provide information
about the sequence of words in the recipes. In general, the proposed
generative method Cook-Gen employs aggregation operations in a generative
modeling setup and allows toggling between different generative modeling
architectures, e.g., neural network-based approximators vs. polynomial powers-
based approximators. We will now formally describe the forward pass of the
implemented generative models that can be used with Cook-Gen to generate the
cooking actions for $A_{r}$ for a recipe $r$ from a partial natural language
description (set of words) of the recipe $X_{r}$.

#####  Cook-Gen-NN using an Aggregated Two-Layer Feedforward Neural Network
Approximation

Using the same notation as in Section III-B, denoting $X_{r}$ as the input to
the forward pass, we can describe the sequence of forward pass steps as
follows:

$\displaystyle E_{r}=\mathbf{E}X_{r},~{}\mathbf{E}\in\mathbb{R}^{M\times d}$
(3) $\displaystyle
H_{r}=E_{r}+\mathbf{P_{e}}X_{r},~{}\mathbf{P_{e}}\in\mathbb{R}^{L\times d}$
(4)
$\displaystyle\begin{split}&H_{r}=\max(\mathbf{W_{z}}^{T}H_{r},0),~{}\mathbf{W_{1}}\in\mathbb{R}^{L\times
d_{1}},~{}\mathbf{W_{z}}\in\mathbb{R}^{d_{1}\times d_{2}},\\\
&~{}z\in\\{1,2\\}\end{split}$ (5)
$\displaystyle\begin{split}&H_{r}=\mathbb{E}(H_{r}[i,]),~{}i\in\\{1,2,\ldots
d_{1}\\}\end{split}$ (6) $\displaystyle
O_{r}=\mathbf{W_{o}}^{T}H_{r}+\mathbf{B_{o}},~{}\mathbf{W_{o}}\in\mathbb{R}^{d_{2}\times
M},~{}\mathbf{B_{o}}\in\mathbb{R}^{d_{2}\times M}$ (7)

The final logits (before applying softmax) are obtained by extracting the last
column of $M$ values from $O_{r}$. Eq (LABEL:eqn:aggr_1) in the forward pass
denotes the aggregation operation, i.e., taking the average (expected value)
of all $d_{1}$ rows of the 2nd hidden layer output matrix. Here $d$ represents
the embedding dimension, $\mathbf{E}$ denotes the embedding matrix,
$\mathbf{P_{e}}$ denotes the position embedding matrix, and the $d_{z}$
corresponds to the dimensions of the hidden layer weight matrices.

#####  Cook-Gen-PF using an Aggregated Polynomials-based Function
Approximation

Neural networks are popular because the steps to obtain the hidden layer
output matrix before the application of the aggregation operation in Eq
(LABEL:eqn:aggr_1) can be thought of as obtaining a feature map of the inputs
$X_{r}$ using the neural network. Such a feature map is adaptive because it is
not predefined. Also, functions of feature maps obtained using neural
networks, as described in Section IV-B (two layers with ReLU activations),
have universal approximation capabilities. However, we also know that a set of
polynomial powers of the input $X_{r}$ is also a feature map, functions of
which can also have universal approximation capabilities. But we first need to
provide the function approximator knowledge of embeddings and positions of
words in $X_{r}$. Thus we compute aggregates over polynomial powers of the
rows in matrix $H_{r}$ obtained just before Eq (LABEL:eqn:nn), i.e., $H_{r}$
obtained after applying the embedding and position embedding matrix operations
to $X_{r}$. The aggregation of polynomial powers of the rows of matrix $H_{r}$
is described as follows:

$\displaystyle H_{r}=\\{\mathbb{E}(H_{r}[i,]^{j}),~{}i\in\\{1,2,\ldots
L\\}\mid j\in\\{1,2,\ldots,J\\}\\}$ (8)

We then get a set of $J$ vectors that are compiled into a new matrix $H_{r}$
of dimension $\mathbb{R}^{J\times d}$. We also need to correspondingly modify
the dimensions of weights and biases $\mathbf{W_{o}}$ and $\mathbf{B_{o}}$ of
the output layer to be of dimension $\mathbb{R}^{d\times M}$.

#####  Cook-Gen-GPT using ChatGPT

Given that ChatGPT is a generative model, we aim to incorporate it into the
suggested ”Cook-Gen” approach. Despite ChatGPT requiring significant
computational resources, it can be made accessible to the general public
through API calls, which eliminates the need to store the model on their
personal devices. Additionally, by leveraging backend infrastructure, the API
calls can reduce inference times and latency. Figure 3 illustrates how ChatGPT
can be utilized to generate cooking instructions through API calls.

Figure 3: The figure shows how ChatGPT is prompted to obtain cooking actions
from recipes using OpenAI API calls.

#### Experiments and Evaluation

##### Model Configurations for Cook-Gen-NN, and Cook-Gen-PF

The Cook-Gen-NN model is trained using a configuration with $d=d_{z}=200$ for
$z$ values of 1 and 2, $L=512$, and $M=33000$. This results in a total of
13.38 million parameters. On the other hand, the Cook-Gen-PF model is trained
with $d=200$, $L=512$, $J=3$, and $M=33000$, resulting in a total of 13.3
million parameters. Both models have trainable matrices consisting of
$\mathbf{E}$, $\mathbf{P_{e}}$, ${\mathbf{W_{z}}}$ for $z\in{1,2}$,
$\mathbf{W_{o}}$, and $\mathbf{B_{o}}$. Unlike the Cook-Gen-NN model, the
number of parameters for the Cook-Gen-PF model does not depend on the
polynomial order $J$. This means that the Cook-Gen-PF model has an advantage
over the Cook-Gen-NN model in that it can handle more complex functions
without increasing the parameter count.

E-CECook-Gen-NNCook-Gen-PFCook-Gen-
GPT$74$$76$$78$$80$$82$$84$$86$$88$$90$$92$$94$$96$$98$$100$Generative
Modeling EvaluationsPrecisionRecall Figure 4: Performance results of various
generative modeling methods used to generate cooking actions as part of Cook-
Gen compared to the best performing discriminative modeling method E-CE
(ELECTRA with CE loss). Cook-Gen-NN, Cook-Gen-PF, Cook-Gen-GPT, represents
Cook-Gen with a two-layer neural network approximation, with a polynomial
powers approximation, and with access to ChatGPT API calls, respectively.

The evaluation scores for the different variants of Cook-Gen is shown in
Figure 4. The figure also shows the best-performing discriminative modeling
method for comparison. We see that Cook-Gen-PF, in which the polynomial
approximation is used with aggregation, performs slightly better than Cook-
Gen-NN, in which the two-layer neural network approximation is used. Although,
the difference is not a lot. ChatGPT performs rather poorly. However, it is
worth noting that we do not have access to the model and are unable to fine-
tune it on our dataset. Fine-tuning ChatGPT could have resulted in improved
performance. Thus, we see that generative modeling using our proposed
aggregation methods significantly outperforms strong baselines, demonstrating
the effectiveness of Cook-Gen for modeling cooking actions despite issues and
challenges caused by irregular data patterns.

ELECTRAXLNETCook-Gen-NNCook-Gen-
PF$0$$50$$100$$150$$200$$250$$300$$350$$400$$450$Model Resource
ConsumptionGPU-Usage in GBParams in Millions Figure 5: The graph displays the
usage of GPU memory (in GBs) and the number of parameters (in millions) for
the ELECTRA and XLNET discriminative model methods in comparison to the Cook-
Gen-NN and Cook-Gen-PF models. The resource consumption of the Cook-Gen models
is considerably lower, indicating that Cook-Gen is highly effective for low-
resource cooking action modeling.

###  IV-C Time and Space Efficiency Considerations

In Section III-C, we discussed the need for efficient modeling of cooking
actions to allow easy access for ordinary users that do not have access to the
resources required to run large state-of-the-art models such as ChatGPT.

Figure 5 shows the resource consumption in terms of GPU memory usage (in GBs)
and the number of parameters in millions for the discriminative model methods
ELECTRA and XLNET compared to the Cook-Gen-NN and Cook-Gen-PF models. The
Cook-Gen models consume significantly fewer resources showing the
effectiveness of Cook-Gen for low-resource cooking action modeling that is
accessible to ordinary users on resource-constrained devices (e.g., mobile
phones).

##  V Conclusion

In this paper, we introduce Cook-Gen, which performs generative modeling of
cooking actions that has the following salient features (1) Cook-Gen is robust
to irregular data patterns in recipes caused by variations in cooking action
descriptions and imbalanced word frequencies (2) Cook-Gen uses significantly
lower memory-usage and parameters for modeling while still outperforming much
larger baselines. Cook-Gen can be immensely useful as part of a large
automated AI system that assists people with maintaining healthy eating
habits, such as the recommendation system shown in Figure 1.

##### Code and Data

All the code and data used to produce the results in the paper will be made
available at this link

## Acknowledgements

This work was supported in part by the National Science Foundation under Grant
2133842, “EAGER: Advancing Neuro-symbolic AI with Deep Knowledge-infused
Learning” and was carried out under the advisement of Prof. Amit Sheth [21,
22, 23].

## References

  * [1] P. Stehr, V. Karnowski, and C. Rossmann, “The multi-faceted usage patterns of nutrition apps: a survey on the appropriation of nutrition apps among german-speaking users of myfitnesspal,” _BMC medical informatics and decision making_ , vol. 20, no. 1, pp. 1–13, 2020.
  * [2] M. R. Jospe, K. A. Fairbairn, P. Green, and T. L. Perry, “Diet app use by sports dietitians: a survey in five countries,” _JMIR mHealth and uHealth_ , vol. 3, no. 1, p. e3345, 2015.
  * [3] Q. Wang, B. Egelandsdal, G. V. Amdam, V. L. Almli, and M. Oostindjer, “Diet and physical activity apps: perceived effectiveness by app users,” _JMIR mHealth and uHealth_ , vol. 4, no. 2, p. e5114, 2016.
  * [4] R. Y. Toledo, A. A. Alzahrani, and L. Martinez, “A food recommender system considering nutritional information and user preferences,” _IEEE Access_ , vol. 7, pp. 96 695–96 711, 2019.
  * [5] N. I. of Health, “Retinoids: Basic science and clinical applications,” U.S. Department of Agriculture, Agricultural Research Service, Tech. Rep., 2006. [Online]. Available: https://www.ars.usda.gov/ARSUserFiles/80400530/pdf/retn06.pdf
  * [6] B. J. McCabe-Sellers and S. E. Beattie, “Food safety: emerging trends in foodborne illness surveillance and prevention,” _Journal of the American Dietetic Association_ , vol. 104, no. 11, pp. 1708–1717, 2004.
  * [7] A. Farhadian, S. Jinap, H. Hanifah, and I. Zaidul, “Effects of meat preheating and wrapping on the levels of polycyclic aromatic hydrocarbons in charcoal-grilled meat,” _Food chemistry_ , vol. 124, no. 1, pp. 141–146, 2011.
  * [8] P. Das, D. Kemisetti, F. I. Jahan, S. E. Spriha, and S. C. Raka, “A comprehensive review on the formation of carcinogens from food products with respect to different cooking methods.”
  * [9] S. S. Shirai, O. Seneviratne, M. E. Gordon, C.-H. Chen, and D. L. McGuinness, “Identifying ingredient substitutions using a knowledge graph of food,” _Frontiers in Artificial Intelligence_ , vol. 3, p. 621766, 2021.
  * [10] Ö. Aydın and E. Karaarslan, “Is chatgpt leading generative ai? what is beyond expectations?” _What is beyond expectations_ , 2023.
  * [11] Y.-C. Yu, V. Narayanan, and J.-S. Li, “Moment-based reinforcement learning for ensemble control,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [12] D. P. Papadopoulos, E. Mora, N. Chepurko, K. W. Huang, F. Ofli, and A. Torralba, “Learning program representations for food images and cooking recipes,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 559–16 569.
  * [13] M. Chang, L. V. Guillain, H. Jung, V. M. Hare, J. Kim, and M. Agrawala, “Recipescape: An interactive tool for analyzing cooking instructions at scale,” in _Proceedings of the 2018 CHI conference on human factors in computing systems_ , 2018, pp. 1–12.
  * [14] Y. Shidochi, T. Takahashi, I. Ide, and H. Murase, “Finding replaceable materials in cooking recipe texts considering characteristic cooking actions,” in _Proceedings of the ACM multimedia 2009 workshop on Multimedia for cooking and eating activities_ , 2009, pp. 9–14.
  * [15] N. Silva, D. Ribeiro, and L. Ferreira, “Information extraction from unstructured recipe data,” in _Proceedings of the 2019 5th international conference on computer and technology applications_ , 2019, pp. 165–168.
  * [16] A. Salvador, N. Hynes, Y. Aytar, J. Marin, F. Ofli, I. Weber, and A. Torralba, “Learning cross-modal embeddings for cooking recipes and food images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3020–3028.
  * [17] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “Electra: Pre-training text encoders as discriminators rather than generators,” _arXiv preprint arXiv:2003.10555_ , 2020.
  * [18] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [19] J. D. Choi and M. Palmer, “Guidelines for the clear style constituent to dependency conversion,” _Center for Computational Language and Education Research, University of Colorado Boulder, Institute of Cognitive Science, Technical Report_ , vol. 1, p. 12, 2012.
  * [20] J. R. Finkel, T. Grenager, and C. D. Manning, “Incorporating non-local information into information extraction systems by gibbs sampling,” in _Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL’05)_ , 2005, pp. 363–370.
  * [21] A. Sheth, K. Roy, and M. Gaur, “Neurosymbolic ai-why, what, and how,” _arXiv preprint arXiv:2305.00813_ , 2023.
  * [22] A. Sheth, M. Gaur, K. Roy, and K. Faldu, “Knowledge-intensive language understanding for explainable ai,” _IEEE Internet Computing_ , vol. 25, no. 5, pp. 19–24, 2021.
  * [23] A. Sheth, M. Gaur, K. Roy, R. Venkataraman, and V. Khandelwal, “Process knowledge-infused ai: Toward user-level explainability, interpretability, and safety,” _IEEE Internet Computing_ , vol. 26, no. 5, pp. 76–84, 2022.


# Enriching Biomedical Knowledge for Low-resource Language
Through Large-Scale Translation

Long Phan1∗, Tai Dang1,3∗, Hieu Tran1∗, Trieu H. Trinh1,4∗,
Vy Phan3, Lam D. Chau2 and Minh-Thang Luong1
1VietAI Research
2Case Western Reserve University
3University of Massachusetts-Amherst
4New York University

###### Abstract

Biomedical data and benchmarks are highly valuable yet very limited in low-
resource languages other than English, such as Vietnamese. In this paper, we
use a state-of-the-art translation model in English-Vietnamese to translate
and produce both pretrained and supervised data in the biomedical domains.
Thanks to such large-scale translation, we introduce ViPubmedT5, a pretrained
Encoder-Decoder Transformer model trained on 20 million translated abstracts
from the high-quality public PubMed corpus. ViPubMedT5 demonstrates state-of-
the-art results on two different biomedical benchmarks in summarization and
acronym disambiguation. Further, we release ViMedNLI - a new NLP task in
Vietnamese translated from MedNLI using the recently public En-vi translation
model and carefully refined by human experts, with evaluations of existing
methods against ViPubmedT5.

**footnotetext: The first four authors contributed equally to this work Figure
1: Overview of the pretraining and finetuning of ViPubmedT5

##  1 Introduction

In recent years, pretrained language models (LMs) have played an important and
novel role in developing many Natural Language Processing (NLP) systems.
Utilizing large pretrained models like BERT Devlin et al. (2018), XLNET Yang
et al. (2019), ALBERT Lan et al. (2019), RoBERTa Liu et al. (2019), GPT-3
Brown et al. (2020) BART Lewis et al. (2019), and T5 Raffel et al. (2019) has
become an effective trend in natural language processing. All these large
models follow the Transformer architecture proposed by Vaswani et al. (2017)
with the attention mechanism. The architecture has been proven to be very
suitable for finetuning downstream tasks leveraging transfer learning with
their large pretrained checkpoints. Before the emergence of large Transformer
LMs, traditional wording embedding gave each word a fixed global
representation. Large pretrained models can derive word vector representation
from a trained large corpus. This will give the pretrained model a better
knowledge of the generalized representation of a trained language/domain and
significantly improve performance on downstream finetune tasks. The success of
pretrained models on a generative domain (BERT, RoBERTa, BART, T5, etc.) has
created a path in creating more specific-domain language models such as
CodeBERT Feng et al. (2020) and CoTexT Phan et al. (2021b) for coding
languages, TaBERT Yin et al. (2020) for tabular data, BioBERT Lee et al.
(2019) and PubmedBERT Tinn et al. (2021) for biomedical languages.

Biomedical literature is getting more popular and widely accessible to the
scientific community through large databases such as
Pubmed111https://pubmed.ncbi.nlm.nih.gov,
PMC222https://www.ncbi.nlm.nih.gov/pmc, and MIMIC-IV Johnson et al. (2021).
This also leads to many studies, corpora, or projects released to further
advance the Biomedical Natural Language Processing field Lee et al. (2019);
Tinn et al. (2021); Phan et al. (2021a); Yuan et al. (2022). These biomedical
domain models leverage transfer learning from pretrained models Devlin et al.
(2018); Clark et al. (2020); Raffel et al. (2019); Lewis et al. (2019) to
achieve state-of-the-art results on multiple Biomedical NLP tasks like Named
Entity Recognition (NER), Relation Extraction (RE), or document
classification.

However, few studies have been on leveraging large pretrained models for
biomedical NLP in low-resource languages. The main reason is the lack of large
biomedical pretraining data and benchmark datasets. Furthermore, collecting
biomedical data in low-resource languages can be very expensive due to
scientific limitations and inaccessibility.

We attempt to overcome the issue of lacking biomedical text data in low-
resource languages by using state-of-the-art translation works. We start with
the Vietnamese language and keep everything reproducible for other low-
resource languages in future work.

We introduce ViPubmedT5, a pretrained encoder-decoder transformer model
trained on synthetic Vietnamese biomedical text translated with state-of-the-
art English-Vietnamese translation work. Meanwhile, we also introduced
ViMedNLI, a medical natural language inference task (NLI), translated from the
English MedNLI Romanov and Shivade (2018) with human refining.

We thoroughly benchmark the performance of our ViPubmedT5 model when
pretrained with synthetic translated biomedical data with ViMedNLI and other
public Vietnamese Biomedical NLP tasks Minh et al. (2022). The results show
that our model outperforms both general domain Nguyen and Nguyen (2020); Phan
et al. (2022) and health-specific domain Vietnamese Minh et al. (2022)
pretrained models on biomedical tasks.

In this work, we offer the following contributions:

  * •

A state-of-the-art English-Vietnamese Translation model (with self-training)
on medical and general domains.

  * •

A first Encoder-Decoder Transformer model ViPubmedT5 pretrained on large-scale
synthetic translated biomedical data.

  * •

A Vietnamese medical natural language inference dataset (ViMedNLI) that
translated from MedNLI Romanov and Shivade (2018) and refined with biomedical
expertise human.

  * •

We publicize our model checkpoints, datasets, and source code for future
studies on other low-resource languages.

##  2 Related Works

The development of parallel text corpora for translation and use for training
MT systems has been a rapidly growing field of research. In recent years, low-
resource languages have gained more attention from the industry, and academia
Chen et al. (2019b); Shen et al. (2021); Gu et al. (2018); Nasir and Mchechesi
(2022). Previous works include gathering more training data or training large
multilingual models Thu et al. (2016); Fan et al. (2021). Low-Language MT
enhances billions of people’s daily life in numerous fields. Nonetheless,
there are specific domains crucial yet limited such as biomedical and
healthcare, in which MT systems have not been able to contribute adequately.

Previous works using MT systems for biomedical tasks includes Neves et al.
(2016); Névéol et al. (2018). Additionally, several biomedical parallel
Deléger et al. (2009) have been utilized just for terminology translation
only. Pioneer attempts to train MT systems using a corpus of MEDLINE titles Wu
et al. (2011), and the use of publication titles and abstracts for both ES-EN
and FR-EN language pairs Jimeno-Yepes et al. (2012). However, none of these
works targets low-resource languages. A recent effort to train Vietnamese ML
systems for biomedical and healthcare is Minh et al. (2022). These, however,
do not utilize the capability of MT systems, instead relying on manual
crawling. Therefore, this motivation has led us to employ MT systems to
contribute high-quality Vietnamese datasets that emerged from the English
language. To the best of our knowledge, this is the first work utilizing
state-of-the-art machine translation to translate both self-supervised and
supervised learning biomedical data for pretrained models in a low-resource
language setting.

###  2.1 Pubmed and English Biomedical NLP Studies

The Pubmed333https://pubmed.ncbi.nlm.nih.gov provides access to the MEDLINE
database444https://www.nlm.nih.gov/bsd/pmresources.html which contains titles,
abstracts, and metadata from medical literature since the 1970s. The dataset
consists of more than 34 million biomedical abstracts from the literature
collected from sources such as life science publications, medical journals,
and published online e-books. This dataset is maintained and updated yearly to
include more up-to-date biomedical documents.

Pubmed Abstract has been the main dataset for almost any state-of-the-art
biomedical domain-specific pretrained models Lee et al. (2019); Yuan et al.
(2022); Tinn et al. (2021); Yasunaga et al. (2022); Alrowili and Shanker
(2021); Phan et al. (2021a). In addition, many well-known Biomedical NLP/NLU
benchmark datasets are created based on the unlabeled Pubmed corpus Doğan et
al. (2014); Nye et al. (2018); Herrero-Zazo et al. (2013); Jin et al. (2019).
Recently, to help accelerate research in biomedical NLP, Gu et al. (2020)
releases BLURB (Biomedical Language Understanding & Reasoning Benchmark),
which consists of multiple pretrained biomedical NLP models and benchmark
tasks. It is important to note that all of the top 10 models on the BLURB
Leaderboard555https://microsoft.github.io/BLURB/leaderboard.html are
pretrained on the Pubmed Abstract dataset.

###  2.2 English-Vietnamese Translation

Due to its limitation of high-quality parallel data available, English-
Vietnamese translation is classified as a low-resource translation language
Liu et al. (2020). One of the first notable parallel datasets and En-Vi neural
machine translation is ISWLT’15 Luong and Manning (2015) with 133K sentence
pairs. A few years later, PhoMT Doan et al. (2021) and VLSP2020 Ha et al.
(2020) released larger parallel datasets, extracted from publicly available
resources for the English-Vietnamese translation.

Recently, VietAI666https://vietai.org curated the largest 4.2M high-quality
training pairs from various domains and achieved state-of-the-art on English-
Vietnamese translation Ngo et al. (2022). The work also focuses on En-Vi
translation performance across multiple domains, including biomedical. As a
result, the project’s NMT outperforms existing En-Vi translation models Doan
et al. (2021); Fan et al. (2020) by more than 2% in the BLEU score.

Table 1: BLEU Scores Results for En-Vi Translation on MTet Medical and PhoMT
General Test Sets

Model | Finetune Datasets |  | MTet Medical
---
Test Set
| PhoMT General
---
Test Set
M2M100 | CCMatrix + CCAligned | 30.18 | 35.83
| Google
---
Translate
- | 38.60 | 39.86
SOTA | MTet+PhoMT | 38.69 | 45.47
Ours |  | MTet+PhoMT
---
+1M Self-training Pubmed Abstracts
45.61 | 46.01

  * •

Notes: The best BLEU scores are in bold. The state-of-the-art (SOTA) model and
MTet dataset are from Ngo et al. (2022); PhoMT dataset and Google Translate’s
result are from Doan et al. (2021). M2M100 model is from Fan et al. (2020).

##  3 Improvements on Biomedical English-Vietnamese Translation through Self-
training

To generate a large-scale synthetic translated Vietnamese biomedical corpus,
we first look into improving the existing English-Vietnamese translation
system in the biomedical translation domain. Previous work from Ngo et al.
(2022) has shown that En-Vi biomedical bitexts are very rare, even for large-
scale bitext mining. Therefore, we look into self-training to leverage the
available monolingual English biomedical data.

Self-training approach has been experimented with in He et al. (2019) and
utilized to improve translation on low-resource MT systems Chen et al.
(2019a). The advantage of this method is that the source side of the
monolingual corpus can be domain-specific data for translation. However, the
shortcoming is that the generated targets can be low-quality and affect the
machine translation performance. Therefore, we start with the English-
Vietnamese machine translation model from Ngo et al. (2022), denoted $bT_{A}$,
which achieves state-of-the-art results on both En-Vi biomedical and general
translation domains.

We use $bT_{A}$ to translate and generate a synthetic parallel biomedical
dataset with 1M pairs of English-Vietnamese biomedical abstracts from the
Pubmed Corpus. The new 1M En-Vi biomedical pairs are then concatenated with
the current high-quality En-Vi translation dataset from MTet Ngo et al. (2022)
and PhoMT Doan et al. (2021), increasing from 6.2M to 7.2M En-Vi sentence
pairs total. To verify the effectiveness of our new self-training data, we re-
finetune the $bT_{A}$ model on this 7.2M bitexts corpus. We report the model
performance on the medical test set from MTet and the general test set from
PhoMT in Table 1 (the translation performances on other domains like News,
Religion, and Law are reported in Appendix A for further reference).

The results show that our model outperforms existing Machine Translation
systems in English-Vietnamese translation by applying self-training. We obtain
a significant gain of 6.61 BLEU Score (38.69->45.61) on the MTet Medical test
set. Our model with self-training also achieves state-of-the-art results on
the PhoMT general domain test set by 0.53 BLEU Score (45.47->46.01). This
shows that our approach not only improves the English-Vietnamese translation
performance in the biomedical context but also generalizes to general
translation. We further discuss our self-training model performance on other
translation domains in Appendix A.

##  4 ViPubmed

After developing a new state-of-the-art machine translation system for
English-Vietnamese translation in the biomedical domain in Section 3, we apply
the system, denoted $bT_{B}$, on downstream translation to generate the first
large-scale synthetic translated biomedical corpus for Vietnamese.

To ensure that our translated ViPubmed dataset contains up-to-date biomedical
research (for example, Covid-19 diseases and Covid-19 vaccines), we use the
newest Pubmed22777https://ftp.ncbi.nlm.nih.gov/pubmed/baseline which contains
approximately 34 million English biomedical abstracts published. The raw data
is compressed in XML format. We then parse these structured XMLs to obtain the
abstract text with Pubmed Parser888https://github.com/titipata/pubmed_parser
Achakulvisut et al. (2020).

The machine translation model $bT_{B}$ is an Encoder-Decoder Transformer based
model with 512 token-length for input and output. Therefore, we filter out
English Pubmed abstracts with more than 512 tokens. For fair size comparison
with the unlabeled dataset of other health-related Vietnamese pretrained
models (discussed in Section 7.2), we take a subset of 20M biomedical
abstracts (20GB of text) for translation and leave a larger subset for future
releases. We then translate the 20M English biomedical abstracts with the
$bT_{B}$ model using 4 TPUv2-8 and 4 TPUv3-8.

Table 2: Some Examples of Abbreviations and Spelling Refining

# | MedNLI | Translated by NMT | Refined by human
---|---|---|---
Abbreviations Refining
1 |  | Electrocardiograms
---
revealed no
QRS changes
| Điện tâm đồ cho thấy
---
không có thay đổi về
QRS.
| Điện tâm đồ cho thấy
---
không có thay đổi về
phức độ QRS
2 |  | Patient has
---
no PMH
| Bệnh nhân
---
không có PMH
| bệnh nhân
---
không có tiền sử bệnh
3 |  | Patient is
---
post op
| Bệnh nhân đã
---
hồi phục
(Patient is recovered)
| Bệnh nhân
---
hậu phẫu thuật
Spelling Refining
4 |  | The infant was
---
born at herm
| Đứa bé được sinh ra
---
ở Herm
(The baby was born
at Herm)
| Đứa bé được sinh
---
đủ tháng
(The baby was born
at term)
5 |  | The patient had
---
an sotesophytes
| Bệnh nhân có
---
sinh cảm
(The patient has flu)
| Bệnh nhân bị
---
viêm xương khớp
(The patient had
osteophytes)
6 | Patient has delerium |  | Bệnh nhân có
---
hội chứng delerium
(Patient has
delerium syndrome)
| Bệnh nhân bị
---
mê sảng
(Patient has delirium)

  * •

#1: Abbreviation in English can be used in both English and Vietnamese.

  * •

#2: Abbreviation can only be used in English. In Vietnamese, abbreviation is
different.

  * •

#3: Abbreviation in English is wrong when translated to Vietnamese.

  * •

#4: The word "term" is misspelled as "herm".

  * •

#5: The word "osteophytes" is misspelled as "sotesophytes".

  * •

#6: The word "delirium" is misspelled as "delerium".

##  5 ViMedNLI

Along with an unlabeled dataset for pretraining, we also introduce a benchmark
dataset generated by translation and refined with human experts. We start with
a natural language inference (NLI) task as it is less prone to errors in
biomedical entity translation compared to named-entity recognition (NER) or
relation extraction (RE) tasks. The process of creating the ViMedNLI is shown
in Figure 2.

Figure 2: The Process of ViMedNLI Corpus Creation

###  5.1 MedNLI

MedNLI Romanov and Shivade (2018) is an NLI dataset annotated by doctors and
grounded in the patients’ medical history. Given a premise sentence and a
hypothesis sentence, the relation of the two sentences (entailment,
contradiction, neutral) is labeled by two board-certified radiologists. The
source of premise sentences in MedNLI is from MIMIC-III Johnson et al. (2016),
a large open-source clinical database. The dataset has been widely studied and
benchmarked by the Biomedical NLP research
community999https://paperswithcode.com/dataset/mednli Peng et al. (2019); Phan
et al. (2021a); El Boukkouri et al. (2020); Alrowili and Shanker (2021);
Kanakarajan et al. (2019).

###  5.2 Dataset Challenges

We follow the same procedures discussed in Section 4 to translate the same
training, development, and test sets released in Romanov and Shivade (2018).
The time and resources to translate the dataset are negligible as there are a
total of 14522 samples.

However, upon translating the dataset with NMT, we find out that the English
clinical note domain has a distinct sublanguage with unique challenges
(abbreviations, inconsistent punctuation, misspellings, etc.). This
observation has also been addressed in Friedman et al. (2002) and Meystre et
al. (2008). Such differences in clinical language representation challenge the
translation output and our quest to release a high-quality medical dataset.

Table 3: Statistics of finetuned datasets Corpus | Train | Dev | Test | Task | Domain
---|---|---|---|---|---
acrDrAid, pairs | 4000 | 523 | 1130 | Acronym disambiguation | Medical
FAQSum, documents | 10621 | 1326 | 1330 | Abstractive summarization | Healthcare
ViMedNLI, pairs | 11232 | 1395 | 1422 | Inference | Clinical

###  5.3 Human Refining

The unique challenges of clinical data under translation settings (discussed
in Section 5.2) require us to work with humans who not only have expertise in
biomedical knowledge but are also sufficient in both English and Vietnamese
languages to refine the dataset. Therefore, we collaborate with pre-medical
Vietnamese students who studied at well-known U.S. Universities to refine the
ViMedNLI datasets.

The refining process starts with a comprehensive guidelines document with
thorough annotation instructions and examples. Then, as clinical notes contain
a significant amount of technical abbreviations that the machine translation
system can not translate initially (Section 5.2), we work with the medical
annotators to create abbreviations and their expansion forms. To make sure the
expansion form of these abbreviations generalizes well in real-world settings,
we verify the use case of these words through multiple Vietnamese medical
websites, blogs, and online dictionaries. Hence, we decided to keep the
original English abbreviations, replace them with a Vietnamese expansion form,
or replace them with a Vietnamese abbreviation. Some examples of this process
are shown in Table 2.

Aside from the English medical abbreviations, there are several grammatical
and spelling mistakes the machine translation system does not understand,
translating either into Vietnamese meanings or even failing to translate.
Human refining is therefore required. The phrase "The infant was born at
herm", for example, was translated as "Đứa bé được sinh ra ở Herm". The word
"herm", which should be spelled as "term", is misspelled and has no medical
meaning. The accurate translation should be "Đứa bé được sinh đủ tháng" ("The
infant was born at term"). Table 2 shows more examples of spelling refining
cases.

Additionally, the machine translation system occasionally produces incorrect
Vietnamese meanings when translating words with proper English spelling and
grammar. Considering the sentence "The patient had post-term delivery" as an
example. Despite having the meaning "Bệnh nhân sinh muộn", it was
mistranslated as "Bệnh nhân sinh non" ("The patient had pre-term delivery").
Another example is "Narrowing of the vessels", which means "Thu hẹp các mạch"
rather than "Thu hẹp các" (no meaning).

Table 4: Tests results on Vietnamese health and biomedical tasks

Domain | Datasets | Metrics | PhoBERT | ViT5 | ViHealthBERT | ViPubmedT5
---|---|---|---|---|---|---
(+news) | (+cc100) |  | (+health
---
text mining)
| (+translated
---
ViPubmed)
Healthcare | FAQSum | RougeL | 41.16 | 61.3 | 43.85 |  60.6
Medical | acrDrAid | Mac-F1 | 82.51 |  88 | 86.7 | 89.04
Clinical | ViMedNLI | Acc | 77.29 | 77.85 |  79.04 | 81.65

  * •

Notes: The best scores are in bold, and the second best scores are underlined.
PhoBERT & ViHealthBERT scores on FAQSum and acrDrAid are from Minh et al.
(2022)

##  6 ViPubmedT5

With an unlabeled synthetic translated ViPubmed Corpus (Section 4) and a
benchmark ViMedNLI dataset (Section 5), we pretrain and finetune a
Transformer-based language model Vaswani et al. (2017) to verify the
effectiveness of our approach in enriching Vietnamese biomedical domain with
translation data. We explain our model and the pretraining settings we applied
in this section.

###  6.1 Model Architecture

We adopt the Transformer encoder-decoder model proposed by Vaswani et al.
(2017), the ViT5 Phan et al. (2022) checkpoints, and T5 framework
101010https://github.com/google-research/text-to-text-transfer-transformer
implemented by Raffel et al. (2019). ViT5 is the first monolingual Vietnamese
Transformer model; the model achieves state-of-the-art results on multiple
Vietnamese general tasks, including generation and classification. The ViT5
publication releases 2 model sizes - base and large. We train ViPubmedT5 using
the base setting (220 million parameters) and leave larger models for future
work.

###  6.2 Pretraining

We pretrain our ViPubmedT5 on 20GB of translated biomedical data ViPubmed
(Section 4). We leverage the Vietnamese checkpoints in the original ViT5 work
Phan et al. (2022) and continuously pretrain the model on the synthetic
biomedical-specific data for another 500k steps. Previous works Lee et al.
(2019); Tinn et al. (2021) have shown that this approach will allow pretrained
language models to learn a better representation of biomedical language
context while maintaining the core Vietnamese language representation.

We train ViPubmedT5 using the same spans-masking learning objective as Raffel
et al. (2019). During self-supervised training, spans of biomedical text
sequences are randomly masked (with sentinel tokens). The target sequence is
formed as the concatenation of the same sentinel tokens and the real masked
spans/tokens.

##  7 Experiments

###  7.1 Benchmark dataset

We finetune and benchmark our pretrained ViPubmedT5 model on two public
Vietnamese biomedical-domain datasets acrDrAid & FAQSum, Minh et al. (2022)
and our released ViMedNLI (Section 5). Detailed statistics of the three
datasets are shown in Table 3.

  * •

acrDrAid Minh et al. (2022) is a Vietnamese Acronym Disambiguation (AD)
dataset that contains radiology reports from Vinmec
hospital111111https://vinmec.com/, Vietnam. The task is correctly identifying
the expansion of an acronym in a given radiology report context. The dataset
is annotated by three expert radiologists. The acrDrAid has 135 acronyms and
424 expansion texts in total.

  * •

FAQ Summarization Minh et al. (2022) is a Vietnamese summarization dataset
collected from FAQ sections of multiple healthcare trustworthy sites. For each
FAQ section, the question text is the input sequence, and the title is a
target summary.

  * •

ViMedNLI is our released dataset discussed in Section 5.

###  7.2 Baseline

To verify the effectiveness of our proposed methods, we compare our ViPubmedT5
model with other state-of-the-art Vietnamese pretrained models:

  * •

PhoBERT Nguyen and Nguyen (2020) is the first public large-scale monolingual
language model pretrained for the Vietnamese language. The model follows the
original RoBERTa Liu et al. (2019) architecture. PhoBERT is trained on a 20GB
word-level Vietnamese news corpus.

  * •

ViT5 Phan et al. (2022) is the most recent state-of-the-art Vietnamese
pretrained model for both generation and classification tasks. The model is
trained on a general domain CC100-vi corpus.

  * •

ViHealthBERT Minh et al. (2022) is the first domain-specific pretrained
language model for Vietnamese healthcare. After initializing weights from
PhoBERT, the model is trained on 25M health sentences mined from different
sources.

##  8 Results

The main finetuned results are shown in Table 4. The main takeaway is that
training on synthetic translated biomedical data allows ViPubmedT5 to learn a
better biomedical context representation. As a result, ViPubmedT5 achieves
state-of-the-art in Medical and Clinical contexts while performing slightly
worse than ViT5 in healthcare topics.

On the healthcare domain (FAQSum), ViPubmedT5 approximates the current state-
of-the-art result (60.6 and 61.3) while outperforming the other models by a
large margin (43.85). The slight difference in performance to ViT5 signifies a
difference in data distribution in PubMed abstracts (scientific writing) and
FAQSum (dialogues between patients and doctors).

For both medical and clinical datasets, ViPubmedT5 outperforms other existing
models. There are also notable improvements from the general domain ViT5 to
ViPubmedT5 (88->89.04 in acrDrAid and 77.85->81.65 in ViMedNLI). This
indicates that the translated ViPubmed corpus contains biomedical knowledge
that low-resource Vietnamese pretrained models can leverage.

Meanwhile, our newly translated ViMedNLI can serve as a robust baseline
dataset for Vietnamese BioNLP research. Both health and biomedical domain
models (ViHealthBERT & ViPubmedT5) perform better than general domain models
(PhoBERT & ViT5) on the ViMedNLI dataset. This shows that our translated and
refined ViMedNLI dataset is high-quality and has robust biomedical contexts.

##  9 Scaling to Other Languages

Our novel way of utilizing a state-of-the-art NMT system to generate synthetic
translated medical data for pretrained models is not limited to the Vietnamese
language and is scalable to many other low-resource languages. Recent works
focus on improving the quality of multiple low-resource NMT systems NLLB Team
et al. (2022); Fan et al. (2020); Bañón et al. (2020). These new state-of-the-
art NMTs make the approach discussed in this paper more practical to produce
synthetic translated biomedical data, enriching the Biomedical NLP research
knowledge in multiple low-resource languages.

##  10 Conclusion

We utilize the state-of-the-art translation model MTet to scale up the very
low-resourced yet highly valuable biomedical data in Vietnamese. Namely,
ViPubMedT5, a T5-style Encoder-Decoder Transformer pretrained on a large-scale
translated corpus of the biomedical domain that demonstrated state-of-the-art
results on both inference and acronym disambiguation in the biomedical domain.
We also introduced ViMedNLI, a machine-translated and human-expert refined
benchmark in natural language inference to further grow the Vietnamese suite
of benchmarks and data in biomedical data.

##  11 Limitations

Although our pretrained model trained on synthetic translated biomedical data
produces state-of-the-art results on downstream tasks for the Vietnamese
language, the approach hugely depends on the quality of the NMTs for other
low-resource languages. Thanks to recent studies and contributions from the
Vietnamese research community (Section 2.2), the English-Vietnamese
translation system has proven strong enough for us to conduct the experiments
discussed in this work. However, the NMT’s actual performance needed before
making the translated biomedical data useful for pretrained models is still a
question that requires further studies.

##  12 Acknowledgements

We would like to thank the Google TPU Research Cloud (TRC) program and Soonson
Kwon (Google ML Ecosystem programs Lead) for their supports.

## References

  * Achakulvisut et al. (2020) Titipat Achakulvisut, Daniel Acuna, and Konrad Kording. 2020.  Pubmed parser: A python parser for pubmed open-access xml subset and medline xml dataset xml dataset.  _Journal of Open Source Software_ , 5(46):1979.
  * Alrowili and Shanker (2021) Sultan Alrowili and Vijay Shanker. 2021.  BioM-transformers: Building large biomedical language models with BERT, ALBERT and ELECTRA.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 221–227, Online. Association for Computational Linguistics.
  * Bañón et al. (2020) Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020.  ParaCrawl: Web-scale acquisition of parallel corpora.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4555–4567, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165.
  * Chen et al. (2019a) Peng-Jen Chen, Jiajun Shen, Matt Le, Vishrav Chaudhary, Ahmed El-Kishky, Guillaume Wenzek, Myle Ott, and Marc’Aurelio Ranzato. 2019a.  Facebook ai’s WAT19 myanmar-english translation task submission.  _CoRR_ , abs/1910.06848.
  * Chen et al. (2019b) Peng-Jen Chen, Jiajun Shen, Matthew Le, Vishrav Chaudhary, Ahmed El-Kishky, Guillaume Wenzek, Myle Ott, and Marc’Aurelio Ranzato. 2019b.  Facebook AI’s WAT19 Myanmar-English translation task submission.  In _Proceedings of the 6th Workshop on Asian Translation_ , pages 112–122, Hong Kong, China. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  _CoRR_ , abs/2003.10555.
  * Deléger et al. (2009) Louise Deléger, Magnus Merkel, and Pierre Zweigenbaum. 2009.  Translating medical terminologies through word alignment in parallel text corpora.  _Journal of biomedical informatics_ , 42 4:692–701.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Doan et al. (2021) Long Doan, Linh The Nguyen, Nguyen Luong Tran, Thai Hoang, and Dat Quoc Nguyen. 2021\.  Phomt: A high-quality and large-scale benchmark dataset for vietnamese-english machine translation.  _CoRR_ , abs/2110.12199.
  * Doğan et al. (2014) Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014.  Ncbi disease corpus: A resource for disease name recognition and concept normalization.  _Journal of Biomedical Informatics_ , 47:1–10.
  * El Boukkouri et al. (2020) Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020.  CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6903–6915, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.  _CoRR_ , abs/2010.11125.
  * Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Çelebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021.  Beyond english-centric multilingual machine translation.  _J. Mach. Learn. Res._ , 22:107:1–107:48.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  Codebert: A pre-trained model for programming and natural languages.  _CoRR_ , abs/2002.08155.
  * Friedman et al. (2002) Carol Friedman, Pauline Kra, and Andrey Rzhetsky. 2002.  Two biomedical sublanguages: a description based on the theories of zellig harris.  _Journal of Biomedical Informatics_ , 35(4):222–235.  Sublanguage - Zellig Harris Memorial.
  * Gu et al. (2018) Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. 2018.  Universal neural machine translation for extremely low resource languages.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 344–354, New Orleans, Louisiana. Association for Computational Linguistics.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.  _CoRR_ , abs/2007.15779.
  * Ha et al. (2020) Thanh-Le Ha, Van-Khanh Tran, and Kim-Anh Nguyen. 2020.  Goals, challenges and findings of the VLSP 2020 English-Vietnamese news translation shared task.  In _Proceedings of the 7th International Workshop on Vietnamese Language and Speech Processing_ , pages 99–105, Hanoi, Vietnam. Association for Computational Lingustics.
  * He et al. (2019) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019.  Revisiting self-training for neural sequence generation.  _CoRR_ , abs/1909.13788.
  * Herrero-Zazo et al. (2013) María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. 2013.  The ddi corpus: An annotated corpus with pharmacological substances and drug–drug interactions.  _Journal of Biomedical Informatics_ , 46(5):914–920.
  * Jimeno-Yepes et al. (2012) Antonio Jimeno-Yepes, Élise Prieur, and Aurélie Névéol. 2012.  Combining medline and publisher data to create parallel corpora for the automatic translation of biomedical text.  _BMC Bioinformatics_ , 14:146 – 146.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019\.  Pubmedqa: A dataset for biomedical research question answering.  _CoRR_ , abs/1909.06146.
  * Johnson et al. (2021) Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2021.  Mimic-iv.
  * Johnson et al. (2016) Alistair Johnson, Tom Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Celi, and Roger Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific Data_ , 3:160035.
  * Kanakarajan et al. (2019) Kamal raj Kanakarajan, Suriyadeepan Ramamoorthy, Vaidheeswaran Archana, Soham Chatterjee, and Malaikannan Sankarasubbu. 2019.  Saama research at MEDIQA 2019: Pre-trained BioBERT with attention visualisation for medical natural language inference.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 510–516, Florence, Italy. Association for Computational Linguistics.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  ALBERT: A lite BERT for self-supervised learning of language representations.  _CoRR_ , abs/1909.11942.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _CoRR_ , abs/1901.08746.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _CoRR_ , abs/1910.13461.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _CoRR_ , abs/2001.08210.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Luong and Manning (2015) Minh-Thang Luong and Christopher Manning. 2015.  Stanford neural machine translation systems for spoken language domains.  In _Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign_ , pages 76–79, Da Nang, Vietnam.
  * Meystre et al. (2008) S M Meystre, G K Savova, K C Kipper-Schuler, and J F Hurdle. 2008.  Extracting information from textual documents in the electronic health record: a review of recent research. _Yearbook of medical informatics_ , pages 128–44.
  * Minh et al. (2022) Nguyen Minh, Vu Hoang Tran, Vu Hoang, Huy Duc Ta, Trung Huu Bui, and Steven Quoc Hung Truong. 2022.  Vihealthbert: Pre-trained language models for vietnamese in health text mining.  In _Proceedings of the Language Resources and Evaluation Conference_ , pages 328–337, Marseille, France. European Language Resources Association.
  * Nasir and Mchechesi (2022) Muhammad Umair Nasir and Innocent Mchechesi. 2022.  Geographical distance is the new hyperparameter: A case study of finding the optimal pre-trained language for English-isiZulu machine translation. In _Proceedings of the Workshop on Multilingual Information Access (MIA)_ , pages 1–8, Seattle, USA. Association for Computational Linguistics.
  * Névéol et al. (2018) Aurélie Névéol, Antonio Jimeno Yepes, Mariana Neves, and Karin Verspoor. 2018.  Parallel corpora for the biomedical domain.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Neves et al. (2016) Mariana Neves, Antonio Jimeno Yepes, and Aurélie Névéol. 2016.  The scielo corpus: a parallel corpus of scientific publications for biomedicine.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 2942–2948, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Ngo et al. (2022) Chinh Ngo, Trieu H. Trinh, Long Phan, Hieu Tran, Tai Dang, Hieu Nguyen, Minh Nguyen, and Minh-Thang Luong. 2022.  Mtet: Multi-domain translation for english and vietnamese.
  * Nguyen and Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020.  Phobert: Pre-trained language models for vietnamese.  _CoRR_ , abs/2003.00744.
  * NLLB Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Nye et al. (2018) Benjamin E. Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain James Marshall, Ani Nenkova, and Byron C. Wallace. 2018.  A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature.  _CoRR_ , abs/1806.04185.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer learning in biomedical natural language processing: An evaluation of BERT and elmo on ten benchmarking datasets.  _CoRR_ , abs/1906.05474.
  * Phan et al. (2022) Long Phan, Hieu Tran, Hieu Nguyen, and Trieu H. Trinh. 2022.  Vit5: Pretrained text-to-text transformer for vietnamese language generation.
  * Phan et al. (2021a) Long N. Phan, James T. Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. 2021a.  Scifive: a text-to-text transformer model for biomedical literature.  _CoRR_ , abs/2106.03598.
  * Phan et al. (2021b) Long N. Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James T. Anibal, Alec Peltekian, and Yanfang Ye. 2021b.  Cotext: Multi-task learning with code-text transformer.  _CoRR_ , abs/2105.08645.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Romanov and Shivade (2018) Alexey Romanov and Chaitanya Shivade. 2018.  Lessons from natural language inference in the clinical domain.
  * Shen et al. (2021) Jiajun Shen, Peng-Jen Chen, Matt Le, Junxian He, Jiatao Gu, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. 2021.  The source-target domain mismatch problem in machine translation.  _ArXiv_ , abs/1909.13151.
  * Thu et al. (2016) Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016.  Introducing the Asian language treebank (ALT).  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1574–1578, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Tinn et al. (2021) Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Fine-tuning large neural language models for biomedical natural language processing.  _CoRR_ , abs/2112.07869.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Wu et al. (2011) Cuijun Wu, F. Xia, Louise Deléger, and Imre Solti. 2011.  Statistical machine translation for biomedical text: are we there yet?  _AMIA … Annual Symposium proceedings. AMIA Symposium_ , 2011:1290–9.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _CoRR_ , abs/1906.08237.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022.  Linkbert: Pretraining language models with document links.
  * Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020.  Tabert: Pretraining for joint understanding of textual and tabular data.  _CoRR_ , abs/2005.08314.
  * Yuan et al. (2022) Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022\.  BioBART: Pretraining and evaluation of a biomedical generative language model.  In _Proceedings of the 21st Workshop on Biomedical Language Processing_ , pages 97–109, Dublin, Ireland. Association for Computational Linguistics.

##  Appendix A Results for En-Vi Translation with Self-training

We explore a more drastic measure to expand the amount of data in the
biomedical domain by self-training. We also explore how a large expansion in
the number of biomedical bitexts affects the performance of our model on other
domains such as Law, Religion, and News by using the MTet multi-domain test
set Ngo et al. (2022).

Table 5: BLEU Scores Results for En-Vi Translation on MTet Multi-domain Test
Set

Model |  | MTet Multi-domain
---
Test Set
Medical | News | Religion | Laws
SOTA | 38.69 | 51.47 | 41.44 | 36.43
Ours | 45.61 | 51.003 | 40.68 | 39.51

  * •

Notes: The best BLEU scores are in bold. The state-of-the-art (SOTA) model and
MTet dataset are from Ngo et al. (2022); Our model trained with self-training
approach on an extra 1M En-Vi synthetic biomedical abstracts is discussed in
Section 3

The improvement is not evident across all domains when tested on a diverse
domain test set (MTet). For example, while there are notable improvements in
the Medical and Law domain, the model performs worse in the Religion and News
domains. This can be attributed to the context representation of biomedical
Pubmed Abstract data. Scientific abstracts tend to be more formal and academic
for knowledgeable audiences with more domain expertise. Therefore, training on
such data allows the Machine Translation system to perform better not only on
the trained domain (Medical) but also on other formally presented domains,
such as Law, while at the same time performing slightly worse on other domains
(News and Religion).