# BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum
Learning for Named Entity Recognition

Vera Pavlova
rttl.ai
Dubai, UAE
<EMAIL_ADDRESS>
&Mohammed Makhlouf
rttl.ai
Dubai, UAE
<EMAIL_ADDRESS>

###### Abstract

Using language models (LMs) pre-trained in a self-supervised setting on large
corpora and then fine-tuning for a downstream task has helped to deal with the
problem of limited label data for supervised learning tasks such as Named
Entity Recognition (NER). Recent research in biomedical language processing
has offered a number of biomedical LMs pre-trained using different methods and
techniques that advance results on many BioNLP tasks, including NER. However,
there is still a lack of a comprehensive comparison of pre-training approaches
that would work more optimally in the biomedical domain. This paper aims to
investigate different pre-training methods, such as pre-training the
biomedical LM from scratch and pre-training it in a continued fashion. We
compare existing methods with our proposed pre-training method of initializing
weights for new tokens by distilling existing weights from the BERT model
inside the context where the tokens were found. The method helps to speed up
the pre-training stage and improve performance on NER. In addition, we compare
how masking rate, corruption strategy, and masking strategies impact the
performance of the biomedical LM. Finally, using the insights from our
experiments, we introduce a new biomedical LM (BIOptimus), which is pre-
trained using Curriculum Learning (CL) and contextualized weight distillation
method. Our model sets new states of the art on several biomedical Named
Entity Recognition (NER) tasks. We release our code and all pre-trained
models.111The source code and the models are released on our Github
https://github.com/rttl-ai/BIOptimus, including the Rust code for qik-find.

Figure 1: Comparison of word frequency (raw count) for Wikipedia Corpus and
PubMed corpus. Frequencies are divided into four categories. Each bar chart
represents how many words of that frequency category are found in both
corpora.

##  1 Introduction

Since the introduction of transformer architecture Vaswani et al. (2017),
transfer learning has gained immense popularity in Natural Language Processing
(NLP). Pre-training LMs on a large corpus (Devlin et al., 2019; Liu et al.,
2019) allows a large-scale knowledge extraction. In the current pipeline of
knowledge transfer in NLP, the pre-training stage can be viewed as preparing
the model with broad general knowledge Raffel et al. (2019) that can later be
fine-tuned to new tasks or adapted to different domains. However, applying
these models in a new domain directly without additional pre-training steps
cannot achieve high performance because of a word distribution shift from a
general domain vocabulary to a new domain vocabulary. The importance of a pre-
trained model for the biomedical domain is even more significant due to the
difficulties of constructing supervised training data because it requires
expert knowledge for quality annotations.

The biomedical domain has greatly benefitted from domain-specified LMs. To
date, various methods have been developed to pre-train biomedical language
models: pre-training from scratch (Gu et al., 2021; Beltagy et al., 2019),
continual pre-training (Lee et al., 2019; Huang et al., 2019), and hybrid
approaches (Tai et al., 2020; Poerner et al., 2020; Sachidananda et al.,
2021). Nonetheless, more research is needed to compare the optimal ways to
pre-train the biomedical LMs. Comparison of different pre-training approaches
and techniques primarily targets general domain LMs (Wettig et al., 2023; Dai
et al., 2022; Yamaguchi et al., 2021; Gu et al., 2020). What still needs to be
clarified is the effects of these methods on the pre-training biomedical LM.

Typically, general domain LMs are pre-trained on Wikipedia, Book Corpora, and
Common-Crawl (Devlin et al., 2019; Lan et al., 2020; Radford et al., 2019;
Raffel et al., 2019; Clark et al., 2020). These corpora are considerably
different from the biomedical corpora in terms of domain specifics (use of
biomedical terminology) and the style of scientific text (most of the models
are trained on PubMed). Thus, the methods and techniques that work well for
the general domain are not guaranteed to work the same for the biomedical
domain. Figure 1 illustrates the difference in the raw counts of word
frequencies between Wikipedia and PubMed, which contains only abstracts
(PubMed Corpus). The corpora are of similar sizes (3.1 and 3.6 billion tokens,
correspondingly). We can observe that, for example, the count of rare words
noticeably differs between two corpora and twice less for the PubMed Corpus.
Yu et al. (2022) and Ethayarajh (2019) showed how the presence of rare words
in the pre-training corpus might greatly impact the performance of the LM.
This work aims to compare different techniques and methods for creating
biomedical LMs. We compare such approaches as pre-training from scratch,
continued pre-training with no specialized vocabulary, and continued pre-
training with biomedical vocabulary. Our evaluations show that a biomedical LM
pre-trained in a continued setting speeds up the pre-training. To account for
the lack of specialized vocabulary that inevitably accompanies models pre-
trained following a continued approach, we propose a new method to initialize
weights for new tokens by distilling existing weights from the BERT model
inside the context where they were found. Furthermore, we compare how masking
rate, corruption strategy, and masking strategies impact the performance of
the biomedical LM. We observe that the corruption strategy of "80-10-10"
Devlin et al. (2019) may play a role in the representation degeneration
problem of LM by increasing the degree of anisotropy in the contextualized
word representations. With relation to the word representation, anisotropy
characterizes an embedding space that is restricted into a narrow cone-like
shape Ethayarajh (2019). This phenomenon leads to a decrease in word
embeddings’ expressiveness and is referred to as the representation
degeneration problem Gao et al. (2019).

Finally, based on the results of our experiments, we introduce a model which
pre-trained with a CL schedule. CL is an easy-to-hard training strategy
Soviany et al. (2022). Though very successful and widely adopted in NLP, it is
still under-explored in pre-training LMs. To the best of our knowledge, CL was
not used before to pre-train the biomedical LM. We propose the CL method based
on the task complexity of predicting masked labels. Our model (BIOptimus),
pre-trained with the contextualized weight distillation and following our CL
method, sets new states of the art on several biomedical Named Entity
Recognition (NER) tasks. In summary, our contributions are:

  * •

Firstly, we propose a new approach to initialize weights to pre-train a
biomedical LM that leverages the efficiency of pre-training in a continued
fashion with specialized biomedical vocabulary that improves the model’s
performance on NER tasks.

  * •

Secondly, we comprehensively compare our approach with other pre-training
approaches. In addition, we experiment with masking ratio, corruption
strategy, and masking strategies, testing what works more optimally to prepare
a language model for the biomedical domain.

  * •

Following experimental insights, we propose a biomedical model (BIOptimus)
pre-trained with the CL method and contextualized weight distillation.

##  2 Prior Literature

Recent studies indicate that adapting existing LMs pre-trained on general
corpora for a new domain or pre-training a new LM from scratch for a specific
domain gives a substantial gain in terms of performance for a downstream task.
The authors of BioBERT Lee et al. (2019) performed domain adaptation by
continuing pre-training BERT Devlin et al. (2019) on a large corpus of
biomedical data. The model showed notable achievement on downstream tasks,
outperforming BERT and the state-of-the-art models. Huang et al. (2019)
further pre-train BioBERT using clinical corpora to address clinical text’s
challenges and released ClinicalBERT. Gururangan et al. (2020) showed the
benefits of continued domain pre-training followed by task-adaptive pre-
training (TAPT) of RoBERTa on four domains, including biomedical. One of the
downsides of continued pre-training is the lack of domain-specific vocabulary.
The authors of the SciBERT model Beltagy et al. (2019) constructed a new
Scivocab using the SentencePiece library, introducing 58% of new tokens, which
shows how much scientific vocabulary differs from a general vocabulary of
BERT-base. The pre-training corpus comprises 82% of the biomedical domain; the
rest is the computer science domain. Training from scratch solely on the
biomedical domain showed further gain and was presented by the authors of
PubMedBert Gu et al. (2021). They construct a new domain-specific vocabulary,
proving its effectiveness when fine-tuning for downstream tasks. Another
approach to pre-train the biomedical LM from scratch using citation links is
the LinkBERT model, which achieved new state-of-the-art on several BioNLP
tasks Yasunaga et al. (2022). To mitigate the high costs of pre-training from
scratch, Tai et al. (2020) introduced extension vocabulary that is not found
in the vocabulary of BERT. The weights of the embedding layer of the extension
vocabulary are randomly initialized, then further pre-trained on the
biomedical corpus. To avoid randomly initializing weights for new tokens and
speed up pre-training, Poerner et al. (2020) used word2vec. Word2vec vectors
are trained on the biomedical domain and then aligned with wordpiece vectors
from BERT. Sachidananda et al. (2021) experimented with subword-based
initialization using the mean of RoBERTa fixed subword embeddings.

Curriculum Learning Curriculum learning (Elman, 1993; Bengio et al., 2009) has
been successfully explored in NLP to solve different tasks. It was widely
adopted in machine translation (Guo et al., 2020; Liu et al., 2020; Wang et
al., 2020, Zhan et al., 2021). Some works have been done in the area of answer
generation Liu et al. (2018), relation extraction Huang and Du (2019), reading
comprehension Tay et al. (2019), and NER Jafarpour et al. (2021). Xu et al.
(2020) used CL during the fine-tuning stage of BERT. Nagatsuka et al. (2021)
applied CL to pre-train RoBERTa by gradually increasing the block size of the
text. Lee et al. (2022) proposed a concept-based curriculum masking.

##  3 Method

In continued biomedical domain adaptation, pre-training using the checkpoint
of the model pre-trained on the general domain can significantly speed up
training. Moreover, the benefits of continued domain adaptation are highly
aligned with the present trends of NLP communities Bommasani et al. (2021) of
reducing the environmental impact and making knowledge readily available in
limited resources settings. One of the problems of continued pre-training of
LMs for domain adaptation is, loosely speaking, the absence of domain-specific
vocabulary Guo and Yu (2022). The issue of out-of-vocabulary (OOV) words is
more elegantly solved with subword tokenization such as Byte-Pair Encoding
(BPE) Sennrich et al. (2016), WordPiece Song et al. (2021), and SentencePiece
Kudo and Richardson (2018). Nevertheless, the presence of a whole word or more
meaningful subwords in domain-specific vocabulary greatly enhances performance
on downstream tasks (Beltagy et al., 2019; Gu et al., 2021; Guo and Yu, 2022).
There are ways to address this issue by incorporating domain-specific tokens
and assigning new token weights without initializing them from scratch
(Sachidananda et al., 2021; Poerner et al., 2020). Nevertheless, none of the
current approaches leverage the advantage of contextualized embeddings of
transformer models. To boost knowledge transfer, we introduce the new
vocabulary by distilling existing weights from the BERT model inside the
context of the domain where they were found. Our approach uses WordPiece
Tokenization Song et al. (2021) to tokenize PubMed Corpus
222https://pubmed.ncbi.nlm.nih.gov. Abstracts published before Jan 2023.
(BioMedTokenizer) with a vocabulary size of 30522. We contextualize tokens in
the following way (see Figure 2):

  * •

First, we perform tokenization with BioMedTokenizer. For example, suppose
BioMedTokenizer has a token "bronchoconstriction" absent from the original
bert-base-uncased vocabulary. In that case, the token will be broken into six
pieces: [’bro’,’##nch’,’##oco’,’##nst’,’##ric’, ’##tion’] (see Figure 2). We
use the mean operation of constituting tokens to compute a single
representation for a new token (distilled representations):

$\begin{split}t_{distilled}=f(t_{1},...,t_{k})\\\ f\in\\{mean\\}\\\
\end{split}$ (1)

The variable _k_ represents the number of the subtokens that make up the token
of interest. We average token weights of only the last layer of the BERT model
as it is more expressive of context and domain information Peters et al.
(2018).

  * •

In the next step, we compute aggregated representation by sampling sentences
that contain tokens of interest from the same PubMed Corpus and compute
aggregated weight across several sentences (contextualized representations):

$\begin{split}t_{context}=g(t_{distilled},...,t_{m})\\\ g\in\\{mean\\}\\\
\end{split}$ (2)

The size of sentences sampled per token _m_ may vary. We sample uniformly at
random, setting the upper bound equal to 20 and the lower bound to 1. In case
the token is not found in the corpus (such tokens represent less than 1/10 of
the whole vocabulary), we assign distilled representation to this token. We
use mean operation, which was found to be the most efficient operation for
aggregating across several contexts Bommasani et al. (2020).

The averaging of the weights of subtokens constitutes more meaning when placed
in context. Moreover, contextualization of tokens’ embeddings leverages the
ability of the BERT model to create domain-specific word embeddings that align
with the corpus where they were found. Aharoni and Goldberg (2020). We call
this approach contextualized weight distillation.

Figure 2: Contextualization of tokens’ embeddings with our resource-efficient
and performant qik-find tool written in the Rust programming language. Qik-
find is a purpose-built tool to find tokens of interest and extract
corresponding sentences from a large corpus exploiting the native capability
of the Rust programming language for efficient multiprocessing.

##  4 Models

To perform a more rigorous comparison and find an optimal approach to the pre-
train biomedical LM, we pre-train four models (see Table 1):

  * •

The first model is pre-trained from scratch, further referred to as the
biomedical model from scratch (BM from scratch). We train WordPiece tokenizer
on PubMed corpus and construct new biomedical vocabulary. The model is trained
on the same corpus with all the weights initialized randomly.

  * •

The second model is pre-trained from the BERT-base model
checkpoint333https://huggingface.co/bert-base-uncased. We call it the
biomedical model continued (BM continued).

  * •

The third model is pre-trained in a hybrid approach called the biomedical
model averaged token weights (BM averaged). We use BioMedTokenizer to
construct the biomedical vocabulary and find an intersection between BERT-base
vocabulary and biomedical vocabulary. The weights of the tokens, which are
common for both vocabularies, are copied from BERT-base vocabulary to new
model weights by directly mapping weights to the corresponding tokens or
subtokens. The weights of new domain-specific tokens absent in the BERT-base
model are synthesized by averaging the corresponding subtokens representations
and assigning this averaged representation to the token of interest in the
embedding matrix of the new model. This method corresponds to the step
"distilled representations" described in Section 3.

  * •

The fourth model is pre-trained following the proposed method in Section 3. We
refer to this approach biomedical model contextualized weights (BM
contextualized) or BIOptimus 0.1. We assign tokens’ weights that are common
for BERT-base vocabulary and newly constructed biomedical vocabulary in the
same manner as described above in the biomedical model averaged token weights.
Contextualization of tokens is performed only for the tokens that are not
found in the BERT-base vocabulary.

Pre-training method Weight initialization BM from scratch from scratch
randomly BM continued continued from the existing checkpoint BM averaged
continued from the existing checkpoint \+ averaging subtokens BM
contextualized continued from the existing checkpoint \+ contextualized weight
distillation

Table 1: Four pre-trained models with different pre-training approaches and
initialization methods.

##  5 Data

The primary data used for pre-training is the same PubMed Corpus used for
training BioMedTokenizer. To speed up experiments, we choose a random subset
of this corpus of 1.8 billion words for pre-training models, representing
approximately half of the data generally used for pre-training models for
biomedical domain adaptation. To test our models on the downstream task, we
use NER tasks. NER is one of the most fundamental biomedical tasks; it is an
essential first step in processing literature for biomedical text mining.
Moreover, the NER is an excellent test for evaluating a domain-specific
model’s success to recognize different types of biomedical terminology. The
recent initiative of Microsoft Research Gu et al. (2021) unified and released
a new BLURB benchmark (Biomedical Language Understanding Reasoning Benchmark).
The Benchmark includes five NER datasets: BC5-chem Li et al. (2016),
BC5-disease Li et al. (2016), NCBI-disease Dogan et al. (2014), BC2GM Smith et
al. (2008), JNLPBA Kim et al. (2004). Label distribution Crichton et al.
(2017) is presented in Appendix A.

##  6 Experimental Set-up

We pre-train all models with the same hyperparameters (details can be found in
Appendix A). We further fine-tune each model for five NER datasets and report
results in Table 2. Fine-tuning details can be found in Appendix A. To compare
the effectiveness of pre-training approaches more closely, we show the effects
of each pre-training epoch on the performance of our models on three datasets:
JNLPBA, BC5-chem, and NCBI-disease (Figure 3). We use the same evaluation
metric and fine-tuning approach for the NER task described in PubMedBERT Gu et
al. (2021).

Moreover, we experiment with other pre-training techniques. The above-
described models are pre-trained by masking random tokens. Gu et al. (2021),
and Cui et al. (2021) showed that pre-training with whole-word masking (WWM)
can be more beneficial. Wettig et al. (2023) experimented with different
percentages of masking rate and found out that for the BERT-base, 0.2 percent
may be more optimal to prompt the model to learn better. Thus, we pre-train
two more models using our method but with different techniques, one model
using WWM and the second using WWM and 0.2% masking rate. Results are
presented in Table 3.

BM from scratch BM continued BM averaged BM contextualized BC5-chem 92.51 93.7
92.93 93.52 BC5-disease 80.66 83.92 81.83 83.41 NCBI-disease 86.34 86.67 87.38
88.44 BC2GM 81.4 83.78 82.82 83.53 JNLPBA 76.14 77.59 75.89 78.71

Table 2: Performance (F1) of test models on five NER tasks.

contextualized \+ MLM \+ 0.15 mr contextualized \+ WWM \+ 0.15 mr
contextualized \+ WWM \+ 0.2 mr BC5-chem 93.52 93.77 93.96 BC5-disease 83.41
84.02 84.28 NCBI-disease 88.44 87.41 88.85 BC2GM 83.53 83.76 83.89 JNLPBA
78.71 78.35 78.8

Table 3: Performance (F1) of models pre-trained with contextualized embeddings
and different pre-training techniques. The scores of the best-performing
models are in bold, and underlined are the scores of the second-best-
performing models.

##  7 Analysis

As shown in Table 2, BM continued shows superior performance on three
datasets. The worst-performing model is BM from scratch. Such performance is
expected due to the experimental set-up and pre-training for a shorter period.
The performance of PubMedBERT (Table 6) suggests that pre-training from
scratch with a domain-specific vocabulary can outperform continued pre-
training but would require more extended training to reach this point. BM
averaged performs better than the model pre-trained from scratch,
demonstrating that averaging weights of subtokens to initialize weights for
new tokens gives an initial boost for the model to learn. However, the second
best-performing model is BM contextualized or BIOptimus 0.1. It outperforms BM
averaged and BM from scratch on all datasets and BM continued on two datasets.

Figure 3 shows the models’ performance on the dev set after each epoch. We can
see that BM continued learns quickly and performs better on JNLPBA and
BC5-chem datasets. However, the model shows the lowest learning curve on the
NCBI-disease dataset, which the absence of specialized vocabulary can explain.
Figure 3 also illustrates the importance of domain-specific vocabulary, and we
can see that all the models with specialized vocabulary learn quickly and
perform relatively well. Overall, introducing domain-specific vocabulary is
beneficial. However, it is less efficient to pre-train a new model entirely
from scratch, initializing all weights randomly. In the case of BIOptimus 0.1,
adding new domain-specific tokens alongside the corresponding weights
initialized in a way that leverages the contextualizing ability of
transformers gives noticeable gains. Additionally, it is important to compare
how biomedical LMs respond to pre-training on WWM against pre-training with
masking only tokens/subtokens. Table 3 shows that it is not precisely the case
that the WWM pre-training performs better across the board (Dai et al., 2022;
Gu et al., 2021). Pre-training with masking only tokens performs better than
WWM on two datasets out of five with a masking rate of 0.15 (underlined
scores). One of the explanations is that individual subtokens often coincide
with morphological components of the words, and learning the meanings of these
components separately may be beneficial for the biomedical LM Hofmann et al.
(2020). Moreover, Table 3 demonstrates that increasing the masking rate
contributes to a tapered performance enhancement.

Figure 3: Effects of pre-training epochs on performance (F1) of four test
models on the development set of JNLPBA, BC5-chem, and NCBI-disease datasets.

##  8 Curriculum Learning

###  8.1 Motivation and Method

Analysis of our experiments motivates us to hypothesize that different
training techniques like masking rate and masking strategies might help
broaden the model’s experience Mitchell (1997) and gain more diversified
knowledge of textual input. In addition, during the experimental stage, we’ve
observed that pre-training with specific techniques like WWM and increased
masking rate slow down the training process if introduced right from the
beginning due to increased task complexity. That brings us to the idea that
introducing more complex tasks gradually, using CL’s easy-to-hard strategy,
assists in guiding the model’s learning process more smoothly Bengio et al.
(2009). Using the results of our experiments, we implement a CL method for
pre-training a biomedical LM. In Masked Language Modeling (MLM), the objective
is to predict the masked token based on the surrounding context:

$\begin{split}L_{MLM}=-\sum_{i}\log
P_{\Theta}(\omega_{i}|\tilde{\omega_{i}})\\\
=-\sum_{i}\log\frac{\exp{(E(\omega_{i})^{\top}\tilde{h_{i}})}}{\sum_{j=1}^{\left|V\right|}\exp{(E(\omega_{i})^{\top}\tilde{h_{i}})}}\end{split}$
(3)

It remains a challenging task to measure the difficulty of a task or a
training sample in CL. It is primarily an issue in the case of pre-training
masked language models. There are a few newly proposed approaches to tackle
this challenge. Nagatsuka et al. (2021) pre-trains RoBERTa by increasing the
block size of input text. Lee et al. (2022) proposed a curriculum based on
masking easy-to-predict tokens first. We formulate our curriculum strategy
from the perspective of the complexity of the prediction task. Predicting
whole words is more complex than predicting just tokens which may be a part of
the word, giving the LM more hints from the surrounding context (Gu et al.,
2021; Cui et al., 2021). Increasing the masking rate makes prediction more
challenging Wettig et al. (2023) since less context is available for
prediction.

We use a pre-trained model of the same architecture and number of parameters
to measure the prediction task difficulty and evaluate its performance as MLM
(Dudy and Bedrick, 2020; Liu et al., 2019). We use a corpus from a different
domain to account for domain shift. In our case, we evaluate the performance
of the BERT-base-uncased444https://huggingface.co/bert-base-uncased on the
RealNews dataset Raffel et al. (2019). The evaluation results are presented in
Figure 4. Based on this evaluation, we divide our curriculum into four phases
(see Table 4).

We start with pre-training vanilla MLM (masking random tokens with a masking
rate of 0.15). As we can see from Figure 4, this task is the easiest to handle
for MLM. This task also ensures the model learns the subtokens that constitute
complex biomedical terminology. At the next stage, we increase the complexity
of the prediction task and pre-train predicting the whole words to teach the
model to combine separate subtokens into whole words. In the third phase, we
raise the task complexity to one more level by increasing the masking rate to
0.2 in predicting the whole words. Wettig et al. (2023) observed that the
corruption rule "80-10-10" hurts the performance for some downstream tasks and
suggested using only [MASK] without corruption strategy for MLM pre-training.
In our training experiments, we observe that prediction without corruption of
tokens makes the prediction task even more complex and slows down the learning
process; thus, we add it as an additional curriculum phase at the end.

We track and visualize contextualized word representations with different
frequencies to observe how pre-training phases evolve over time and their
impact on the performance and quality of pre-trained embeddings. Figure 5
shows that low-frequency words form a separate cluster from high-frequency
words, which is more evident after the first stage. During pre-training, the
gap decreases; however, only after the fourth phase, when the corruption
strategy is removed, do the clusters join closer together. It is plausible
that tokens’ prediction with a corruption strategy plays a role in
degenerating word embeddings, and it may explain why it hurts the performance
in some downstream tasks Wettig et al. (2023). We leave further experiments on
this subject for future research.

v. Phases Masking strategy Masking rate Corruption strategy 0.1 phase 1 tokens
0.15 with corruption 0.2 phase 2 WWM 0.15 with corruption 0.3 phase 3 WWM 0.2
with corruption 0.4 phase 4 WWM 0.2 no corruption

Table 4: Stages of the CL method with increasing task complexity. Figure 4: Evaluation of the performance of the BERT-base-uncased on the RealNews corpus (perplexity). We average the scores across text files randomly sampled from the corpus. We take ten samples, each of size of around 12M tokens. Figure 5: A t-SNE visualization of word embeddings during four pre-training phases with the CL method. "High" on the color bar corresponds to high frequencies, and "Low" to low frequencies. v. | BC5-chem | BC5-disease | NCBI-disease | BC2GM | JNLPBA
---|---|---|---|---|---
0.1 | 93.52 | 83.41 | 88.44 | 83.53 | 78.71
0.2 | 93.87 | 83.43 | 87.99 | 84.16 | 79.12
0.3 | 93.7 | 85.06 | 88.17 | 84.54 | 79.28
0.4 | 94.1 | 84.98 | 89.54 | 85.25 | 79.46
|  |  |  |  |
Table 5: Evaluation of each CL phase on NER datasets.

###  8.2 Experimental Setting

Data. We use the same PubMed Corpus and NER datasets described in Section 5.

Implementation. The model’s weights are initialized using our contextualized
weight distillation approach, which helps speed up pre-training. We pre-train
with the same hyperparameters presented in Appendix A and increase the number
of optimization steps for one epoch for each subsequent phase to account for
the complexity of the prediction task. The initial learning rate is set at
1e-4, and when restarting the training at the next phase, we decrease the
learning rate following the learning rate scheduler.

Models and Baselines. We track the progress of the CL method by evaluating
models after each phase (see Table 5). Each phase’s resulting model is
considered an independent model and brings competitive results right from the
first phase. Furthermore, we compare our models with BioBERT Lee et al.
(2019), PubMedBERT Gu et al. (2021), and BioLinkBERT-base Yasunaga et al.
(2022) (Table 6).

Bio-
BERT PubMed-
BERT BioLink-
BERT BIOptimus
0.4 BC5-chem 92.85 93.33 93.75 94.1 BC5-disease 84.7 85.62 86.1 84.98 NCBI-
disease 89.13 87.82 88.18 89.54 BC2GM 83.82 84.52 84.9 85.25 JNLPBA 78.55
80.06 79.03 79.46

Table 6: Comparison of pre-trained language models on five NER datasets. The
scores of the best-performing models are in bold, and underlined are the
scores of the second-best-performing models. BIOptimus 0.4 sets a new state-
of-the-art on BC5-chem, NCBI-disease, and BC2GM datasets.

###  8.3 Results and Discussion

Masking strategy (from phase 1 to phase 2) (see Table 5) gives noticeable
gains in performance for BC5-chem, BC2GM, and JNLPBA datasets, there is only a
slight improvement on the BC5-disease dataset, and the NCBI-disease dataset
responded quite poorly to this transition. Increasing the masking rate (from
phase 2 to phase 3) helps to advance performance on all datasets except
BC5-chem, while BC5-disease shows a considerable gain from increasing the
masking rate. Removing the corruption strategy "80-10-10" (from phase 3 to
phase 4) is generally beneficial for all datasets, with considerable gains for
BC5-chem and NCBI-disease and a slight drop for BC5-disease. BC2GM and JNLPBA
datasets respond with stable improvement in all pre-training phases of the CL
method. BC5-chem, BC5-disease, and NCBI-disease datasets exhibit more diverse
responses to changes in curriculum phases.

###  8.4 Ablation Study

In this section, we conduct an ablation study to assess the effect of the
proposed CL method. To measure the impact of pre-training with our CL method,
we pre-train a model in a continued setting with the contextualized weight
distillation method, using vanilla MLM (with a masking rate of 0.15, applying
a corruption rule “80-10-10”) but without CL. Essentially the model is
BIOptimus 0.1 pre-trained for the same number of optimization steps as
BIOptimus 0.4. The performance of this model is presented in Table 7 (“No
CL”). Removing the CL method hurts downstream performance. The drop occurs
with all NER datasets and is more apparent with BC5-disease, NCBI-disease and
BC2GM datasets. This suggests that pre-training with the CL method helps boost
biomedical LM’s performance on NER task.

BC5-chem BC5-disease NCBI-disease BC2GM JNLPBA BIOptimus 0.4 94.1 84.98 89.54
85.25 79.46 No CL 93.86 84.29 88.92 84.53 79.07

Table 7: Ablation study on the CL method.

##  9 Conclusion

This paper presented a new method to initialize tokens’ weight for new
biomedical vocabulary when pre-training from the existing checkpoint
(continued approach). We also compared this method of token weight
initialization with other pre-training methods (see Table 2 and Table 3). This
method showed considerable gains in speeding up the pre-training phase and
improving performance on NER. Comparing pre-training techniques showed that
WWM is not the best-performing approach for all NER tasks, and masking only
tokens/subtokens shows competitive performance. Increasing the masking rate
and removing the corruption strategy are generally beneficial techniques for
pre-training biomedical LM. Finally, we introduced the CL method based on the
task complexity to pre-train LMs. The “easy-to-hard” CL method introduces the
biomedical LM to a broader scope of language experience, speeds up pre-
training, and enhances performance on downstream tasks like NER. It is
important to highlight that our model BIOptimus 0.4 achieves high performance
with the pre-training time reduced by at least half, proving the pre-training
approach’s efficiency.

## Acknowledgment

We thank Sebastian Hurubaru and our anonymous reviewers for their valuable
feedback. This work was supported by tnsr capital.

## Limitations

(1) To be able to present a rigorous comparison and analysis of pre-training
an optimal biomedical LM, we focused on running extensive evaluation on five
NER tasks from the BLURB benchmark555https://microsoft.github.io/BLURB/. We do
not evaluate the performance of our models on other downstream tasks in the
framework of this paper and leave it for future work. (2) While we performed
additional experiments to explain the reason for the performance drop on some
tasks when implementing the corruption strategy "80-10-10," one plausible
explanation is that it might increase the degree of anisotropy in the
contextualized word representations. Separate work is needed to search for how
the corruption strategy might cause a drop in performance on some downstream
tasks. (3) Due to the expensive nature of pre-training experiments, we were
not able to experiment with all possible combinations of pre-training methods
and techniques. (4) Our models were pre-trained on the English language only.

## Ethics Statement

While our research does not directly introduce any social or ethical bias nor
amplify the bias in the data, we inherit a considerable amount of the
underlying limitations of LMs. LM pre-training is computationally expensive
and causes environmental damage. Our research focuses on overall computing
resource efficiency and thus has a marginal environmental footprint.

## References

  * Aharoni and Goldberg (2020) Roee Aharoni and Yoav Goldberg. 2020.  Unsupervised domain clusters in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7747–7763, Online. Association for Computational Linguistics.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009\.  Curriculum learning.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4758–4781, Online. Association for Computational Linguistics.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. 2021.  On the opportunities and risks of foundation models.  _CoRR_ , abs/2108.07258.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.
  * Crichton et al. (2017) Gamal Crichton, Sampo Pyysalo, Billy Chiu, and Anna Korhonen. 2017.  A neural network multi-task learning approach to biomedical named entity recognition.  _BMC Bioinformatics_ , 18.
  * Cui et al. (2021) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021.  Pre-training with whole word masking for chinese BERT.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3504–3514.
  * Dai et al. (2022) Yong Dai, Linyang Li, Cong Zhou, Zhangyin Feng, Enbo Zhao, Xipeng Qiu, Piji Li, and Duyu Tang. 2022.  “is whole word masking always better for Chinese BERT?”: Probing on Chinese grammatical error correction.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1–8, Dublin, Ireland. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dogan et al. (2014) Rezarta Dogan, Robert Leaman, and Zhiyong lu. 2014.  Ncbi disease corpus: A resource for disease name recognition and concept normalization.  _Journal of biomedical informatics_ , 47.
  * Dudy and Bedrick (2020) Shiran Dudy and Steven Bedrick. 2020.  Are some words worth more than others? In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pages 131–142, Online. Association for Computational Linguistics.
  * Elman (1993) Jeffrey L. Elman. 1993.  Learning and development in neural networks: the importance of starting small.  _Cognition_ , 48(1):71–99.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  * Gao et al. (2019) Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019.  Representation degeneration problem in training natural language generation models.  _CoRR_ , abs/1907.12009.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare_ , 3(1):1–23.
  * Gu et al. (2020) Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020.  Train no evil: Selective masking for task-guided pre-training.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6966–6974, Online. Association for Computational Linguistics.
  * Guo et al. (2020) Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020.  Fine-tuning by curriculum learning for non-autoregressive neural machine translation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7839–7846.
  * Guo and Yu (2022) Xu Guo and Han Yu. 2022.  On the domain adaptation and generalization of pretrained language models: A survey.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Hofmann et al. (2020) Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2020.  DagoBERT: Generating derivational morphology with a pretrained language model.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3848–3861, Online. Association for Computational Linguistics.
  * Huang et al. (2019) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.
  * Huang and Du (2019) Yuyun Huang and Jinhua Du. 2019.  Self-attention enhanced CNNs and collaborative curriculum learning for distantly supervised relation extraction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 389–398, Hong Kong, China. Association for Computational Linguistics.
  * Jafarpour et al. (2021) Borna Jafarpour, Dawn Sepehr, and Nick Pogrebnyakov. 2021.  Active curriculum learning.  In _Proceedings of the First Workshop on Interactive Learning for Natural Language Processing_ , pages 40–45, Online. Association for Computational Linguistics.
  * Kim et al. (2004) Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004\.  Introduction to the bio-entity recognition task at jnlpba.  In _Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications_ , JNLPBA ’04, page 70–75, USA. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_.
  * Lee et al. (2022) Mingyu Lee, Jun-Hyung Park, Junho Kim, Kang-Min Kim, and SangKeun Lee. 2022.  Efficient pre-training of masked language model via concept-based curriculum masking.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7417–7427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn Mattingly, Thomas Wiegers, and Zhiyong lu. 2016\.  Biocreative v cdr task corpus: a resource for chemical disease relation extraction.  _Database_ , 2016:baw068.
  * Liu et al. (2018) Cao Liu, Shizhu He, Kang Liu, and Jun Zhao. 2018.  Curriculum learning for natural answer generation.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 4223–4229. International Joint Conferences on Artificial Intelligence Organization.
  * Liu et al. (2020) Jinglin Liu, Yi Ren, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2020\.  Task-level curriculum learning for non-autoregressive neural machine translation.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3861–3867. International Joint Conferences on Artificial Intelligence Organization.  Main track.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Mitchell (1997) Tom M Mitchell. 1997.  _Machine learning_ , volume 1.  McGraw-hill New York.
  * Nagatsuka et al. (2021) Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. 2021.  Pre-training a BERT with curriculum learning by increasing block-size of input text.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , pages 989–996, Held Online. INCOMA Ltd.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018.  Dissecting contextual word embeddings: Architecture and representation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1499–1509, Brussels, Belgium. Association for Computational Linguistics.
  * Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020.  Inexpensive domain adaptation of pretrained language models: Case studies on biomedical NER and covid-19 QA.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1482–1490, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Sachidananda et al. (2021) Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021.  Efficient domain adaptation of language models via adaptive tokenization.  In _Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing_ , pages 155–165, Virtual. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Smith et al. (2008) Larry Smith, Lorraine Tanabe, Rie Ando, Cheng-Ju Kuo, I-Fang Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph Friedrich, Kuzman Ganchev, Manabu Torii, Hongfang Liu, Barry Haddow, Craig Struble, Richard Povinelli, Andreas Vlachos, William Baumgartner Jr, Lawrence Hunter, Bob Carpenter, and W. Wilbur. 2008.  Overview of biocreative ii gene mention recognition.  _Genome biology_ , 9 Suppl 2:S2.
  * Song et al. (2021) Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021.  Fast WordPiece tokenization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Soviany et al. (2022) Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022.  Curriculum learning: A survey.
  * Tai et al. (2020) Wen Tai, H. T. Kung, Xin Dong, Marcus Comiter, and Chang-Fu Kuo. 2020.  exBERT: Extending pre-trained models with domain-specific vocabulary under constrained training resources.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1433–1439, Online. Association for Computational Linguistics.
  * Tay et al. (2019) Yi Tay, Shuohang Wang, Luu Anh Tuan, Jie Fu, Minh C. Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang. 2019.  Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.
  * Wang et al. (2020) Wei Wang, Ye Tian, Jiquan Ngiam, Yinfei Yang, Isaac Caswell, and Zarana Parekh. 2020\.  Learning a multi-domain curriculum for neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7711–7723, Online. Association for Computational Linguistics.
  * Wettig et al. (2023) Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023.  Should you mask 15% in masked language modeling?
  * Xu et al. (2020) Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020.  Curriculum learning for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6095–6104, Online. Association for Computational Linguistics.
  * Yamaguchi et al. (2021) Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina, and Nikolaos Aletras. 2021.  Frustratingly simple pretraining alternatives to masked language modeling.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3116–3125, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022.  LinkBERT: Pretraining language models with document links.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8003–8016, Dublin, Ireland. Association for Computational Linguistics.
  * Yu et al. (2022) Sangwon Yu, Jongyoon Song, Heeseung Kim, Seongmin Lee, Woo-Jong Ryu, and Sungroh Yoon. 2022.  Rare tokens degenerate all tokens: Improving neural text generation via adaptive gradient gating for rare token embeddings.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 29–45, Dublin, Ireland. Association for Computational Linguistics.
  * Zhan et al. (2021) Runzhe Zhan, Xuebo Liu, Derek F. Wong, and Lidia S. Chao. 2021.  Meta-curriculum learning for domain adaptation in neural machine translation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14310–14318.

##  Appendix A Appendix

###  A.1 Datasets

Figure 6: Label distribution of BC5CDR-chem, BC5CDR-disease, BC2GM, NCBI-
disease datasets.

###  A.2 Computing Infrastructure

Computing Infrastructure 8 x A100 GPUs Hyperparameter Assignment number of
epochs 7-34 batch size 256 maximum learning rate 0.0005 learning rate
optimizer Adam learning rate scheduler None or Warmup linear Weight decay 0.01
Warmup proportion 0.06 learning rate decay linear

Table 8: Hyperparameters for pre-training biomedical LMs.

Computing Infrastructure 2 x NVIDIA RTX 3090 GPU Hyperparameter Assignment
number of epochs 5-11 batch size 4, 8, 16 learning rate 1e-5, 2e-5 dropout 0.1

Table 9: Hyperparameters for fine-tuning on NER datasets.


2023 Vol. 00 No. 0, 000–000

11institutetext: Department of Physics, Mahrah University, Mahrah, Yemen;
<EMAIL_ADDRESS>
22institutetext: Department of Physics, Hadhramout University, Mukalla, Yemen
33institutetext: Department of Applied Physics, Faculty of Science and
Technology, Universiti Kebangsaan Malaysia, 43600 UKM Bangi, Selangor,
Malaysia
44institutetext: Department of Applied Physics and Astronomy, and Sharjah
Academy for Astronomy, Space Sciences and Technology, University of Sharjah,
P.O.Box 27272 Sharjah, United Arab Emirates
55institutetext: Department of Physics, Al al-Bayt University, Mafraq 25113,
Jordan
Received xxxx Month xx; accepted yyyy Month yy

# Modified masses and parallaxes of close binary system: HD 39438

Suhail Masda 1122 Z. T. Yousef 33 Mashhoor Al-Wardat 4455 and Awni Al-
Khasawneh 44

###### Abstract

We present the detailed fundamental stellar parameters of the close visual
binary system; HD 39438 for the first time. We used Al-Wardat’s method for
analyzing binary and multiple stellar systems (BMSSs). The method implements
Kurucz’s plane parallel model atmospheres to construct synthetic spectral
energy distributions for both components of the system. It then combines the
results of the spectroscopic analysis with the photometric analysis and then
compares them with the observed ones to construct the best synthetic spectral
energy distributions for the combined system. The analysis gives the precise
fundamental parameters of the individual components of the system. Based on
the positions of the components of HD 39438 on the H-R diagram, and
evolutionary and isochrones tracks, we found that the system belongs to the
main sequence stars with masses of 1.24 and 0.98 solar masses for the
components A and B, respectively, and age of 1.995 Gyr for both components.
The main result of HD 39438 is new dynamical parallax, which is estimated to
be $16.689\pm 0.03$ mas.

###### keywords:

Binaries: close binary system, Stars: fundamental parameters, Methods:
Analytical: Al-Wradat’s Method, Techniques: photometric, Individual: HD 39438.

##  1 INTRODUCTION

One of the most vital disciplines for contemporary stellar astronomy is that
is reliable and bound up with the understanding of the binary stars. The close
binary systems are one of those binaries which make use of them in estimating
the fundamental stellar parameters especially stellar masses precisely because
the majority of systems makes up 50% from the binary and multiple systems
(Duquennoy et al. 1991).

The techniques of speckle interferometry (Balega et al. 2002a; Tokovinin et
al. 2010; Köhler 2014) and adaptive optics (Roberts et al. 2005; Roberts
2011), which are modern techniques, are instrumental in studying and analyzing
the close visual binary systems. The analysis of spectroscopic and astrometric
data is of overriding significance in the period of the speckle interferometry
and adaptive optics Lucy (2018). These techniques are also instrumental in
improving the situation for those binaries in terms of evolutionary status and
relative motion.

Al-Wardat’s method for analyzing binary and multiple stellar systems (BMSSs)
involves merging the results of spectroscopic and photometric analyses, with
the observed measurements. This combination is regarded as the most effective
method for analyzing such systems (Al-Wardat 2002, 2003, 2007). It merges the
magnitude difference measurements of the speckle interferometry, the combined
spectral energy distributions (SEDs) of the spectrophotometric analysis with
the aid of the grids of Atlas9 models Kurucz (1994), and radial velocity
measurements (once available) to estimate the individual fundamental stellar
parameters of the binary systems, thereby determining the precise
spectrophotometric masses of the binary systems. The method has been utilized
to compare the synthetic stellar photometry with the observed stellar
photometry to estimate the fundamental parameters of the solar-type stars (Al-
Wardat 2009, 2012; Al-Wardat et al. 2014a, b; Masda et al. 2016; Al-Wardat et
al. 2016, 2017; Masda et al. 2018a, b, 2019a, 2019b; Al-Wardat et al. 2021b;
Masda et al. 2021; Widyan & Aljboor 2021; Yousef et al. 2021; Al-Wardat et al.
2014c, a; Al-Wardat 2014).

On the one hand, the synthetic stellar photometry is mainly utilized to obtain
the fundamental stellar parameters more precisely through comparing their
results with the observed ones (Al-Wardat et al. 2014a, b; Masda et al. 2018a,
b, 2019a, 2019b; Al-Wardat et al. 2021b; Masda et al. 2021; Widyan & Aljboor
2021). The synthetic photometry is synthetic analysis of the results of the
spectroscopic analysis (synthetic SED) of the binary system. This contributes
to the improvement the fundamental parameters of the close binary systems.

In this study, the studied system of HD 39438 (HIP 27758) is a visual close
binary in solar neighborhood located at $\pi_{\rm H07}=20.15\pm 1.19$ mas van
Leeuwen (2007) and at $\pi_{\rm DR3}=16.0508\pm 0.264$ mas Gaia Collaboration
(2022), where the abbreviations of the astrometric measurements have been
pointed out in Masda & Al-Wardat (2023). Since, the renormalized unit weight
error (RUWE) in $\pi_{\rm DR3}$ was very large, the solution of this parallax
should not be used in that case.

The first orbit for this binary was solved by Mason et al. (2010) with grade
of three (G 3 = Reliable). Acoording to Mason et al. (2010), they found that
the dynamical stellar mass was $\mathcal{M}=1.29\pm 0.44\mathcal{M}_{\odot}$
based on H07’s parallax, and $2.56\pm 0.34\mathcal{M}_{\odot}$ based on the
DR3’s parallax. Then, the second orbit was studied by Tokovinin et al. (2014)
with grade of three and the dynamical stellar mass of the system was
$\mathcal{M}=1.53\pm 0.28\mathcal{M}_{\odot}$ based on H07’s parallax, and
$\mathcal{M}=3.02\pm 1.70\mathcal{M}_{\odot}$ based on DR3’s parallax. In
2017, Tokovinin (2017) revised the orbit of the system. In that case,
Tokovinin (2017) found that the orbit has been graded as 2 (G 2=Good), which
should be adopted. According to this study, the individual dynamical masses
were estimated to be $\mathcal{M}=1.29\mathcal{M}_{\odot}$ and
$\mathcal{M}=0.97\mathcal{M}_{\odot}$ for the primary and secondary
components, respectively. In this solution, the dynamical stellar mass of the
system was $\mathcal{M}=1.50\pm 0.27\mathcal{M}_{\odot}$ based on H07’s
parallax, and $\mathcal{M}=2.97\pm 0.15\mathcal{M}_{\odot}$ based on DR3’s
parallax. Based on these results, Tokovinin (2017) indicated that the H07’s
parallax is not accurate and should be revised more precisely. As a result,
Tokovinin (2017) suggested new dynamical parallax as $\pi_{\rm dyn}=17.6$ mas.

In this study, we proceed with the series of Masda & Al-Wardat (2023), which
presents the modified masses and parallaxes for a selected sample of close
binary systems. First, the key aim for this paper is to present the
fundamental stellar parameters of the binary system utilizing Al-Wardat’s
method for analyzing BMSSs. Second, we present the comparison between the
observed photometry of the combined system with the combined synthetic
photometry. Third, we present the spectrophotometric stellar masses and new
dynamical parallax of the system.

In this studt, we present the observational data of HD 39438 (HIP 27758),
which will be compared to the synthetic analysis in section 2. Section 3
containes the analysis method of the binary systems. Section 4 gives the
method to calculate the stellar masses of the binary system. In section 5, the
results and discussion of the close binary system present. Finally, in section
6, we present the conclusion of this study.

##  2 Observed data

The observed data is the backbone of the synthetic analysis. The observed
photometric data, which is available from miscellaneous sources, is the key
refernces to obtain the stellar parameters. These are Hipparcos catalogue (ESA
1997), Strömgren (Hauck & Mermilliod 1998) and Tycho catalogues (Høg et al.
2000), which will be compared with the synthetic photometric data to study
details of the system. Table 1 contains the basic data and the observed
photometric data of HD 39438, while Table 2 contains the observed magnitude
differences between the primary and secondary components of the binary system.

Table 1: Fundamental data and observed photometry of HD 39438

Property | HD 39438 | Ref.
---|---|---
| HIP 27758 |
$\alpha_{2000}$ | $05^{\rm h}52^{\rm m}29\hbox{$.\\!\\!^{s}$}411$ | 1
$\delta_{2000}$ | $-02\hbox{${}^{\circ}$}17^{\prime}07.^{\prime\prime}62$ | 1
Sp. Typ. | G0V | 2
Gaia DR2 | 3025640770239673216 | 1
Gaia DR3 | 3025640770239673216 | 1
$E(B-V)$ | $0.017$ | $3$
$A_{\rm v}$ (mag) | $0.053$ | $*$
$\pi_{\rm H07}$ (mas) | $20.15\pm 1.19$ | $4$
$\pi_{\rm DR2}$ (mas) | $11.906\pm 0.37$ | $5$
$\pi_{\rm DR3}$ (mas) | $16.051\pm 0.26$ | $5$
$[\rm Fe/H]$ | $-0.12\pm 0.08$ | $6$
$V_{\rm J}$ (mag) | $7.26$ | $7$
$B_{\rm J}$ (mag) | $7.79\pm 0.02$ | 8
$(B-V)_{\rm J}$ (mag) | $0.56\pm 0.015$ | 7
$(b-y)_{\rm S}$ (mag) | $0.35$ | $9$
$(v-b)_{\rm S}$ (mag) | $0.51$ | 9
$(u-v)_{\rm S}$ (mag) | $0.88$ | 9
$B_{\rm T}$ (mag) | $7.89\pm 0.012$ | $7$
$V_{\rm T}$ (mag) | $7.32\pm 0.010$ | 7

Notes. $*$ means $A_{\rm v}=3.1E(B-V)$, Ref. 1 (Gaia Collaboration 2020), 2
(Houk & Swift 1999), 3 (Lallement et al. 2014), 4 H07’s parallax (van Leeuwen
2007), 5 DR2 and DR3’s parallax (Gaia Collaboration et al. 2016; Gaia
Collaboration 2022), 6 (Gáspár et al. 2016), 7 (ESA 1997), 8 (Høg et al.
2000), and 9 (Hauck & Mermilliod 1998).

Table 2: The observed magnitude difference between the components of HD 39438 (HIP 27758). HD | $\triangle m$ | $\sigma_{\Delta m}$ | Filter ($\lambda/\Delta\lambda$) | Ref.
---|---|---|---|---
39438 | $0.87$ | 0.89 |  $V_{H}:511$nm/222 | ESA (1997)
| $1.34$ | $0.03$ |  $545$nm/30 | Pluzhnik (2005)
| $1.31$ | 0.03 |  $545$nm/30 | Balega et al. (2002b)
| $1.65$ |  |  $550$nm/40 | Horch et al. (2008)
| $1.26$ |  |  $541$nm/88 | Horch et al. (2008)
| $1.33$ |  |  $550$nm/40 | Horch et al. (2010)
| $1.40$ |  |  $551$nm/22 | Tokovinin et al. (2010)
| $2.30$ |  |  $543$nm/22 | Hartkopf et al. (2012)
| $1.60$ |  |  $543$nm/22 | Tokovinin et al. (2014)
| $1.50$ |  |  $543$nm/22 | Tokovinin et al. (2014)
| $1.60$ |  |  $543$nm/22 | Tokovinin et al. (2015)
| $1.62$ | $0.28$ |  $543$nm/22 | Tokovinin (2017)
| $1.60$ |  |  $543$nm/22 | Tokovinin (2016)

##  3 Method and Analysis

The spectrophotometric analysis is the most important step in Al-Wardat’s
method, which depends on two solutions to estimate the astrophysical
parameters. The solutions are as follows:

###  3.1 Spectroscopic solution

The spectroscopic solution is the most important key to reach the fundamental
stellar parameters. In this solution, we require to construct the synthetic
SED for the combined and individual synthetic SED of the system. First of all,
we need to know the observed magnitude difference of the system, which is
estimated as follows: $\triangle m=1.49\pm 0.01$. This parameter is the
average for all $\triangle m$ measurements given in Table 2 under the V-band
filters. The observed magnitude difference of the system, combined with the
visual magnitude, led to the individual apparent and absolute magnitudes of
the system as follows: $m_{v}^{A}=8^{\rm m}.30\pm 0.002$, $\rm
M_{V}^{A}=4^{\rm m}.36\pm 0.01$, and $m_{v}^{B}=8^{\rm m}.64\pm 0.12$, $\rm
M_{V}^{B}=4^{\rm m}.70\pm 0.13$ for the primary and secondary components,
respectively, by using the following simple relationships:

$\displaystyle\centering\rm m_{v}^{A}=\rm m_{v}+2.5\log(1+10^{-0.4\triangle\rm
m}),\@add@centering$ (1) $\displaystyle\centering\rm m_{v}^{B}=\rm
m_{v}^{A}+{\triangle\rm m},\@add@centering$ (2) $\displaystyle\rm
M_{V}-m_{v}=5-5\log(d)-A_{V},$ (3)

Here, the distance of the system from Earth ($d$) is measured in parsec (pc).
Furthermore, since HD 39438 is a nearby system, the interstellar extinction is
neglected.

The absolute magnitudes of HD 39438 components are employed for estimating the
input parameters, together with some parameters taken as introductory values
from the Tables of Lang (1992) and Gray (2005). In addition, the following
equations for the main sequence stars are used:

$\displaystyle\log\frac{R}{R_{\odot}}=\frac{M_{bol}^{\odot}-M_{bol}}{5}-2\log\frac{T_{\rm
eff}}{T_{\odot}},$ (4) $\displaystyle\log
g=\log\frac{M}{M_{\odot}}-2\log\frac{R}{R_{\odot}}+\log g_{\odot}.$ (5)

where $T_{\odot}=5777\,\rm K$, log $g_{\odot}=4.44$ and
$M_{bol}^{\odot}=4^{\rm m}.75$. $M_{bol}=\rm M_{V}+BC$; where $\rm BC$ is the
bolometric correction.

The individual synthetic SED for each single star are built based on input
parameters of the binary system. Therefore, the Kurucz Atlas9 models, which
are plaane-parallel model atmospheres developed by Kurucz in 1994, are
empployed. These models are used to generate the synthetic fluxes for
individual components of the system, and when combined with the parallax, they
produce the synthetic SED for the combined close binary system. For this
purpose, the specialized subroutines of Al-Wardat’s method for analyzing BMSSs
must be utilized. The combined synthetic SED of the binary system is
determined using the following equation:

$\displaystyle
F_{\lambda,s}=\Bigg{(}\frac{R_{A}}{d}\Bigg{)}^{2}\Bigg{(}H_{\lambda}^{A}+H_{\lambda}^{B}\Bigg{(}\frac{R_{B}}{R_{A}}\Bigg{)}^{2}\Bigg{)}$
(6)

where $F_{\lambda}$ is the combined synthetic SED of the binary system,
$R_{A}$ and $R_{B}$ are the radii of component A and component B of the
system, respectively, in solar units. $H_{\lambda}^{A}$ and $H_{\lambda}^{B}$
are the corresponding fluxes of component A and component B, respectively, in
units of ergs cm-2 s-1 Å-1. These individual fluxes are dependent on the
$T_{\rm eff.}$ and log g. This equation accounts for the energy fluxx of the
individual components located at a distance d (in parsecs) from Earth,
ensuring a reliable estimation.

In Equation 6, the values of the radii are dependent mainly on the accuracy of
the parallax measurements. (Tokovinin et al. 2000) showed that the parallax
measurements of the binary systems were probably distorted by the orbital
motion. That is why, Al-Wardat’s method discovered that the problems started
appearing significantly in the parallax measurements of the binary systems and
gives the new dynamical parallax (Al-Wardat et al. 2021a; Masda & Al-Wardat
2023).

The results of the fundametal stellar parameters should be in keeping with
those observed ones of the binary system. This has ranked as one of the best
ways to make certain accuracy of the parallax of the system. So, synthetic
photometric solution should be executed to determine the best stellar
parameters of the close visual binary system.

###  3.2 Photometric solution

The synthetic photometric solution is the perfect complement to the
spectroscopic solution, which is in turn instrumental in estimating the
fundamental stellar parameters of the close binary systems. The main aim of
this solution is to calculate the magnitudes and color indices of the combined
and individual synthetic SEDs and then compare with the observed ones in any
photometric system. So, the synthetic magnitudes and color indices in
different photometrical systems such as: Johnson: $U$, $B$, $V$, $R$, $U-B$,
$B-V$, $V-R$; Strömgren: $u$, $v$, $b$, $y$, $u-v$, $v-b$, $b-y$ and Tycho:
$B_{T}$, $V_{T}$, $B_{T}-V_{T}$ are calculated by using the following equation
(Al-Wardat 2012):

$m_{p}[F_{\lambda,s}(\lambda)]=-2.5\log\frac{\int
P_{p}(\lambda)F_{\lambda,s}(\lambda)\lambda{\rm d}\lambda}{\int
P_{p}(\lambda)F_{\lambda,r}(\lambda)\lambda{\rm d}\lambda}+{\rm ZP}_{p}\,$ (7)

where $m_{p}$ is the synthetic magnitude of the passband $p$, $P_{p}(\lambda)$
is the dimensionless sensitivity function of the passband $p$,
$F_{\lambda,s}(\lambda)$ is the synthetic SED of the object and
$F_{\lambda,r}(\lambda)$ is the SED of the reference star (Vega). Zero points
(ZPp) from Maíz Apellániz (2007) are adopted.

##  4 Mass and dynamical parallax

The stellar mass played a vital role in understanding the formation and
evolution of the binary systems. Thus, its estimation should be accurate.
There are two types of masses, which are the spectrophotometric mass
$\mathcal{M}_{Sph}$ and dynamical stellar mass $\mathcal{M}_{d}$. The former
is estimated based on the evolutionary tracks by using Al-Wardat’s method for
analyzing BMSSs, while the later is estimated by using orbital solution of the
system based on Kepler’s third law as follows:

$\displaystyle\mathcal{M}_{d}=\mathcal{M}_{A}+\mathcal{M}_{B}=\Big{(}\frac{a^{3}}{\pi^{3}P^{2}}\Big{)}\
\mathcal{M}_{\odot},$ (8)

The error in the dynamical mass is estimated as follows:

$\displaystyle\frac{\sigma_{\mathcal{M}}}{\mathcal{M}}=\sqrt{9\Big{(}\frac{\sigma_{\pi}}{\pi}\Big{)}^{2}+9\Big{(}\frac{\sigma_{a}}{a}\Big{)}^{2}+4\Big{(}\frac{\sigma_{p}}{p}\Big{)}^{2}}$
(9)

where $a^{{}^{\prime\prime}}$ and $\pi$ are the semi-major axis and the
parallax (both in arcsec), respectively, P is the orbital period (in years),
$\ \mathcal{M}_{A}$ and $\ \mathcal{M}_{B}$ are the masses (in solar mass).

The dynamical masses are dependent mainly on the grades of the orbits. We can
adopt the best orbit if that orbit has the grades of Grade 1=Definitive, Grade
2=Good, and Grade 3=Reliable. When the spectrophotometric mass is in keeping
with the dynamical mass, the parallax of the system is adopted, otherwise it
should be estimated by using Al-Wardat’s method as follows:

$\displaystyle\mathcal{\pi}_{dyn}=\frac{a}{P^{2/3}(\sum\mathcal{M}_{Sph})^{1/3}}$
(10)

where $\sum\mathcal{M}_{Sph}$ are estimated by using Al-Wardat’s method for
analyzing BMSSs in solar mass and $\pi_{dyn}$ is in arcsec. Its error is
estimated as follows:

$\displaystyle\frac{\sigma_{\mathcal{\pi}_{dyn}}}{\mathcal{\pi}_{dyn}}=\sqrt{\frac{4}{9}\Big{(}\frac{\sigma_{P}}{P}\Big{)}^{2}+\Big{(}\frac{\sigma_{a}}{a}\Big{)}^{2}+\frac{1}{9}\Big{(}\frac{\sigma_{\sum\mathcal{M}_{Sph}}}{\sum\mathcal{M}_{Sph}}\Big{)}^{2}}$
(11)

##  5 Results and discussions

The fundamental stellar properties of the close binary system, HD 39438 were
estimated using the complex analytical method (Al-Wardat’s method for
analyzing BMSSs) by (Al-Wardat 2002). The method combines the spectroscopic
solution with photometric solution to estimate the physical and geometrical
stellar parameters of the system. These led to presenting a new value for the
system‘s parallax of HD 39438.

The results of the calculated synthetic magnitudes and colour indices of the
individual components and combined synthetic SEDs of the binary system, HD
39438 are listed in Table 3. These have presented in different photometrical
systems (Johnson: $U$, $B$, $V$, $R$, $U-B$, $B-V$, $V-R$; Strömgren: $u$,
$v$, $b$, $y$, $u-v$, $v-b$, $b-y$ and Tycho: $B_{T}$, $V_{T}$,
$B_{T}-V_{T}$).

Table 3:  The synthetic stellar photometry of HD 39438. Sys. | Filter | Combined Synth. | HD 39438 | HD 39438
---|---|---|---|---
|  | $\sigma=\pm 0.03$ | A | B
Joh- | $U$ | 7.90 | 8.06 | 10.06
Cou. | $B$ | 7.82 | 8.02 | 9.76
| $V$ | 7.26 | 7.51 | 9.00
| $R$ | 6.95 | 7.22 | 8.59
| $U-B$ | 0.08 | 0.04 | 0.31
| $B-V$ | 0.56 | 0.51 | 0.76
| $V-R$ | 0.31 | 0.29 | 0.40
Ström. | $u$ | 9.07 | 9.23 | 11.21
| $v$ | 8.14 | 8.32 | 10.16
| $b$ | 7.58 | 7.80 | 9.41
| $y$ | 7.23 | 7.48 | 8.96
| $u-v$ | 0.93 | 0.91 | 1.05
| $v-b$ | 0.56 | 0.52 | 0.675
| $b-y$ | 0.35 | 0.32 | 0.45
Tycho | $B_{T}$ | 7.96 | 8.14 | 9.95
| $V_{T}$ | 7.33 | 7.57 | 9.08
| $B_{T}-V_{T}$ | 0.63 | 0.57 | 0.87

Table 4 shows the best agreement between the synthetic and observed photometry
of the binary system, HD 39438. This agreement demonstrates that the basic
stellar characteristics of each element in the system listed in Table 5 are
reliable. Table 3 indicates that the synthetic apparent magnitudes are
completely in keeping with the observed apparent magnitudes of the system.

Table 4: The best agreement between the observed photometry from catalogues
and the synthetic photometry from this study of HD 39438.

| HD 39438
---|---
Filter |  Observed a |  Syntheticb(This work)
|  ($\rm mag$) |  ($\rm mag$)
$V_{J}$ | $7.26$ | $7.26\pm 0.03$
$B_{J}$ | $7.79\pm 0.02$ | $7.82\pm 0.03$
$B_{T}$ | $7.93\pm 0.012$ | $7.96\pm 0.03$
$V_{T}$ | $7.32\pm 0.01$ | $7.33\pm 0.03$
$(B-V)_{J}$ | $0.56\pm 0.02$ | $0.56\pm 0.03$
$(u-v)_{S}$ | $0.88$ | $0.93\pm 0.03$
$(v-b)_{S}$ | $0.51$ | $0.56\pm 0.03$
$(b-y)_{S}$ | $0.35$ | $0.35\pm 0.03$
$\triangle\rm m$ | $1.49^{c}\pm 0.01$ | $1.49^{d}\pm 0.05$

Notes: a The observational data of HD 39438 (see Table 1), b The synthetic
photometry of HD 39438 (see Table 3), c The observed magnitude difference of
HD 39438 (see Table 2) and d The synthetic magnitude difference of HD 39438
(see Table 3)

The stellar luminosities of the individual components of HD 39438 are
estimated to be as follows: $L_{A}=2.65\pm 0.08\,\rm L_{\odot}$ and
$L_{B}=0.76\pm 0.09\,\rm L_{\odot}$ for the primary and secondary omponents of
the system, while the spectral types for them are F5.5V and G8V, respectively,
which are in line with the spectral types of Mason et al. (2010) and Tokovinin
(2017), and with the spectral type F5V given in WDS and SIMBAD catalogues.

Table 5: The fundamental stellar parameters of the individual components of HD
39438.

|  | H 39438
---|---|---
Parameters | Units | H 39438 | HD 39438
|  | A | B
$\rm T_{\rm eff}$ | [ K ] | $6370\pm 100$ | $5580\pm 100$
R |  [R⊙] | $1.34\pm 0.09$ | $0.935\pm 0.09$
$\log\rm g$ | [cgs] | $4.35\pm 0.07$ | $4.50\pm 0.07$
$\rm L$ |  [$\rm L_{\odot}$] | $2.65\pm 0.08$ | $0.76\pm 0.09$
$\rm M_{bol}$ | [mag] | $3.69\pm 0.08$ | $5.05\pm 0.09$
$\rm M_{V}$ | [mag] | $4.03\pm 0.12$ | $5.52\pm 0.13$
M a |  [$\rm M_{\odot}$] | $1.24\pm 0.08$ | $0.98\pm 0.07$
Sp. Type |  | F5.5V | G8V
Parallax b | [mas] | $20.15\pm 1.19$
Age c | [Gyr] |  $1.995$

aBased on Al-Wardat’s method

bBased on H07’s parallax.

cBased on the the isochrones tracks.

According to the results of the analysis, Fig. 1 shows the adopted combined
synthetic SED and the synthetic SED for the individual component of the binary
system for the first time based on the best agreement between the observed and
synthetic stellar photometry of the system.

Figure 1: The combined synthetic SED and its individual components of HD
39438.

Fig. 2 shows the spectrophotometric stellar masses of HD 39438, which are
determined by using Al-Wardat’s complex method for analyzing BMSSs based on
the synthetic evolutionary tracks of Girardi et al. (2000b) and fundamental
stellar parameters of the system. These are found to be $1.24\pm 0.11\,\rm
M_{\odot}$ and $0.98\pm 0.09\,\rm M_{\odot}$ for the primary and secondary
components of HD 39438. According to Tokovinin (2017), the total mass of the
system was $2.26\mathcal{M}_{\odot}$ by using the spectral types, which is in
keeping with our results ($2.22\mathcal{M}_{\odot}$).

Figure 2: The synthetic evolutionary tracks of Girardi et al. (2000b) and
isochrones tracks of Girardi et al. (2000a) of both components of the system
on the H-R diagram.

The total dynamical mass obtained by the orbital solutions of Mason et al.
(2010) ($2.56\pm 0.32$) is consistent with the total mass achieved in this
study within error margins, while there is no agreement between the results of
Al-Wardat’s method and dynamical mass obtained utilizing Tokovinin et al.
(2014)’s orbit. However, Tokovinin (2017) revised the orbit and presented new
orbital parameters, the new orbital solution was graded two (G 2=good), which
is more accurate than the previous one. In his study, Tokovinin (2017)
presented a new parallax as $\pi_{dyn}=17.6$ mas depending on the new orbital
solution. This further supports our conclusion that the measured parallax for
this system is not accurate enough and needs to be revised by observations.

In our analysis, we used the dynamical parallax and the good orbital solution
by Tokovinin (2017) ($P=11.963\pm 0.036$ yr and $a=0.1207\pm 0.0007$ arcsec)
to calculate the dynamical mass sum as $\Sigma\mathcal{M}=2.25\pm
0.05\mathcal{M}_{\odot}$, which is well in line with the spectrophotometric
mass sum ($\Sigma\mathcal{M}=2.22\mathcal{M}_{\odot}$) using Al-Wardat’s
method. In our case, we say that the suggested dynamical parallax should be
slightly larger than the dynamical parallax of Tokovinin (2017) based on our
results. As a results, we used good orbital solution of Tokovinin (2017) and
our spectrophotometric mass sum ($\Sigma\mathcal{M}=2.22\mathcal{M}_{\odot}$)
to compute the new dynamical parallax as $\pi_{dyn}=17.689\pm 0.03$ mas, which
is the closest estimate to the dynamical parallax of Tokovinin (2017). As a
result, we expect that Gaia will perform and give good improvement in terms of
the trigonometric parallax in the near future.

Fig. 2 shows the positions of both components on the isochrones tracks of
Girardi et al. (2000a). In that case, we can see that the metallicity of HD
39438 is [Z=0.019, Y=0.27], as shown in Fig. 2.

Based on Fig. 2, the age of the system is found to be 1.995 Gyr. The combined
metallicity of the system was 0.015 based on the observed data (Gáspár et al.
2016), which corresponds well with the synthetic metallicity of 0.019, as
shown in Fig. 2.

##  6 Conclusions

We have presented the fundamental stellar parameters of the close binary
system, HD 39438 using Al-Wardat’s method for analyzing BMSSs. The method
implements Kurucz’s plane parallel model atmospheres to construct synthetic
SEDs for both components of the system. It then combines the results of the
spectroscopic analysis with the photometric analysis and then compares them
with the observed ones to construct the best synthetic SEDs for the combined
system. The best match between the synthetic and observed magnitudes and
colour indices of the system for various photometrical systems, including
Johnson: $U$, $B$, $V$, $R$, $U-B$, $B-V$, $V-R$; Strömgren: $u$, $v$, $b$,
$y$, $u-v$, $v-b$, $b-y$ and Tycho: $B_{T}$, $V_{T}$, $B_{T}-V_{T}$ is
showcased.

The results shows that HD 39438 consists of two main sequence stars; a 1.24
solar mass with F5.5V and a 0.98 solar mass with G8V, both have the same age
around 2 Gyr. We revised the dynamical parallax of the system, which is
estimated to be $16.689\pm 0.03$ mas.

###### Acknowledgements.

This study utilized several resources and tools, including SAO/NASA, the
SIMBAD database, the Fourth Catalog of Interferometric Measurements of Binary
Stars, IPAC data systems, the ORBIT code and the CHORIZOS code for photometric
and spectrophotometric data analysis, and codes of Al-Wardat’s method for
analyzing binary and multiple stellar systems (BMSSs).

## References

  * Al-Wardat (2002) Al-Wardat, M. A. 2002, Bull. Special Astrophys. Obs., 53, 51
  * Al-Wardat (2003) Al-Wardat, M. A. 2003, Bulletin of the Special Astrophysics Observatory, 55, 18
  * Al-Wardat (2007) Al-Wardat, M. A. 2007, Astronomische Nachrichten, 328, 63
  * Al-Wardat (2009) Al-Wardat, M. A. 2009, Astronomische Nachrichten, 330, 385
  * Al-Wardat (2012) Al-Wardat, M. A. 2012, PASA, 29, 523
  * Al-Wardat (2014) Al-Wardat, M. A. 2014, Astrophysical Bulletin, 69, 454
  * Al-Wardat et al. (2014a) Al-Wardat, M. A., Balega, Y. Y., Leushin, V. V., et al. 2014a, Astrophysical Bulletin, 69, 58
  * Al-Wardat et al. (2014b) Al-Wardat, M. A., Balega, Y. Y., Leushin, V. V., et al. 2014b, Astrophysical Bulletin, 69, 198
  * Al-Wardat et al. (2017) Al-Wardat, M. A., Docobo, J. A., Abushattal, A. A., & Campo, P. P. 2017, Astrophysical Bulletin, 72, 24
  * Al-Wardat et al. (2016) Al-Wardat, M. A., El-Mahameed, M. H., Yusuf, N. A., Khasawneh, A. M., & Masda, S. G. 2016, Research in Astronomy and Astrophysics, 16, 166
  * Al-Wardat et al. (2021a) Al-Wardat, M. A., Hussein, A. M., Al-Naimiy, H. M., & Barstow, M. A. 2021a, PASA, 38, e002
  * Al-Wardat et al. (2014c) Al-Wardat, M. A., Widyan, H. S., & Al-thyabat, A. 2014c, PASA, 31, e005
  * Al-Wardat et al. (2021b) Al-Wardat, M. A., Abu-Alrob, E., Hussein, A. M., et al. 2021b, Research in Astronomy and Astrophysics, 21, 161
  * Balega et al. (2002a) Balega, I. I., Balega, Y. Y., Hofmann, K.-H., et al. 2002a, A&A, 385, 87
  * Balega et al. (2002b) Balega, Y. Y., Tokovinin, A. A., Pluzhnik, E. A., & Weigelt, G. 2002b, Astronomy Letters, 28, 773
  * Duquennoy et al. (1991) Duquennoy, A., Mayor, M., & Halbwachs, J.-L. 1991, A&AS, 88, 281
  * ESA (1997) ESA. 1997, The Hipparcos and Tycho Catalogues (ESA)
  * Gaia Collaboration (2020) Gaia Collaboration. 2020, VizieR Online Data Catalog, I/350
  * Gaia Collaboration (2022) Gaia Collaboration. 2022, VizieR Online Data Catalog, I/355
  * Gaia Collaboration et al. (2016) Gaia Collaboration, Prusti, T., de Bruijne, J. H. J., et al. 2016, A&A, 595, A1
  * Gáspár et al. (2016) Gáspár, A., Rieke, G. H., & Ballering, N. 2016, ApJ, 826, 171
  * Girardi et al. (2000a) Girardi, L., Bressan, A., Bertelli, G., & Chiosi, C. 2000a, A&AS, 141, 371
  * Girardi et al. (2000b) Girardi, L., Bressan, A., Bertelli, G., & Chiosi, C. 2000b, VizieR Online Data Catalog, 414, 10371
  * Gray (2005) Gray, D. F. 2005, The Observation and Analysis of Stellar Photospheres, 505
  * Hartkopf et al. (2012) Hartkopf, W. I., Tokovinin, A., & Mason, B. D. 2012, AJ, 143, 42
  * Hauck & Mermilliod (1998) Hauck, B., & Mermilliod, M. 1998, A&AS, 129, 431
  * Høg et al. (2000) Høg, E., Fabricius, C., Makarov, V. V., et al. 2000, A&A, 355, L27
  * Horch et al. (2010) Horch, E. P., Falta, D., Anderson, L. M., et al. 2010, AJ, 139, 205
  * Horch et al. (2008) Horch, E. P., van Altena, W. F., Cyr, Jr., W. M., et al. 2008, AJ, 136, 312
  * Houk & Swift (1999) Houk, N., & Swift, C. 1999, Michigan Spectral Survey, 5, 0
  * Köhler (2014) Köhler, R. 2014, Contributions of the Astronomical Observatory Skalnate Pleso, 43, 229
  * Kurucz (1994) Kurucz, R. 1994, Solar abundance model atmospheres for 0,1,2,4,8 km/s. Kurucz CD-ROM No. 19. Cambridge, Mass.: Smithsonian Astrophysical Observatory, 1994., 19
  * Lallement et al. (2014) Lallement, R., Vergely, J. L., Valette, B., et al. 2014, A&A, 561, A91
  * Lang (1992) Lang, K. R. 1992, Astrophysical Data I. Planets and Stars., 133
  * Lucy (2018) Lucy, L. B. 2018, A&A, 618, A100
  * Maíz Apellániz (2007) Maíz Apellániz, J. 2007, in Astronomical Society of the Pacific Conference Series, Vol. 364, The Future of Photometric, Spectrophotometric and Polarimetric Standardization, ed. C. Sterken (San Francisco: Astronomical Society of the Pacific), 227
  * Masda & Al-Wardat (2023) Masda, S., & Al-Wardat, M. 2023, Advances in Space Research
  * Masda et al. (2016) Masda, S. G., Al-Wardat, M. A., Neuhäuser, R., & Al-Naimiy, H. M. 2016, Research in Astronomy and Astrophysics, 16, 112
  * Masda et al. (2018a) Masda, S. G., Al-Wardat, M. A., & Pathan, J. K. M. K. 2018a, Research in Astronomy and Astrophysics, 18, 072
  * Masda et al. (2018b) Masda, S. G., Al-Wardat, M. A., & Pathan, J. M. 2018b, Journal of Astrophysics and Astronomy, 39, 58
  * Masda et al. (2019a) Masda, S. G., Al-Wardat, M. A., & Pathan, J. M. 2019a, Research in Astronomy and Astrophysics, 19, 105
  * Masda et al. (2019b) Masda, S. G., Docobo, J. A., Hussein, A. M., et al. 2019b, Astrophysical Bulletin, 74, 464
  * Masda et al. (2021) Masda, S. G., Khan, A. R., & Pathan, J. M. 2021, in American Institute of Physics Conference Series, Vol. 2335, American Institute of Physics Conference Series, 090002
  * Mason et al. (2010) Mason, B. D., Hartkopf, W. I., & Tokovinin, A. 2010, AJ, 140, 735
  * Pluzhnik (2005) Pluzhnik, E. A. 2005, A&A, 431, 587
  * Roberts (2011) Roberts, Jr., L. C. 2011, MNRAS, 413, 1200
  * Roberts et al. (2005) Roberts, Jr., L. C., Turner, N. H., Bradford, L. W., et al. 2005, AJ, 130, 2262
  * Tokovinin (2016) Tokovinin, A. 2016, AJ, 152, 138
  * Tokovinin (2017) Tokovinin, A. 2017, AJ, 154, 110
  * Tokovinin et al. (2000) Tokovinin, A. A., Balega, Y. Y., Hofmann, K. H., & Weigelt, G. 2000, Astronomy Letters, 26, 668
  * Tokovinin et al. (2010) Tokovinin, A., Mason, B. D., & Hartkopf, W. I. 2010, AJ, 139, 743
  * Tokovinin et al. (2014) Tokovinin, A., Mason, B. D., & Hartkopf, W. I. 2014, AJ, 147, 123
  * Tokovinin et al. (2015) Tokovinin, A., Mason, B. D., Hartkopf, W. I., Mendez, R. A., & Horch, E. P. 2015, AJ, 150, 50
  * van Leeuwen (2007) van Leeuwen, F. 2007, A&A, 474, 653
  * Widyan & Aljboor (2021) Widyan, H., & Aljboor, H. 2021, Research in Astronomy and Astrophysics, 21, 110
  * Yousef et al. (2021) Yousef, Z. T., Annuar, A., Hussein, A. M., et al. 2021, Research in Astronomy and Astrophysics, 21, 114