# Detecting Contradictory COVID-19 Drug Efficacy Claims
from Biomedical Literature††thanks:  Data and code available at
https://github.com/dnsosa/covid_lit_contra_claims

Daniel N. Sosa1 Malavika Suresh2 Christopher Potts3 Russ B. Altman4,5
1Department of Biomedical Data Science, Stanford University
2School of Computing, Robert Gordon University
3Department of Linguistics, Stanford University
4Department of Bioengineering, Stanford University
5Department of Genetics, Stanford University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The COVID-19 pandemic created a deluge of questionable and contradictory
scientific claims about drug efficacy – an “infodemic” with lasting
consequences for science and society. In this work, we argue that NLP models
can help domain experts distill and understand the literature in this complex,
high-stakes area. Our task is to automatically identify contradictory claims
about COVID-19 drug efficacy. We frame this as a natural language inference
problem and offer a new NLI dataset created by domain experts. The NLI framing
allows us to create curricula combining existing datasets and our own. The
resulting models are useful investigative tools. We provide a case study of
how these models help a domain expert summarize and assess evidence concerning
remdisivir and hydroxychloroquine.

##  1 Introduction

The COVID-19 pandemic caused by the novel SARS-CoV-2 virus completely changed
modern life. According to the World Health Organization Nov. 16, 2022
situation report, more than 6.5 million people have died as a result of this
disease World Health Organization (2022). During times of pandemic, treatment
options are limited, and developing new drug treatments is infeasible in the
short-term Wouters et al. (2020).

However, if a novel disease shares biological underpinnings with another
disease for which a drug treatment already exists, a doctor may be able to
repurpose that drug as a treatment for the new disease with positive
therapeutic effect Pushpakom et al. (2019). This strategy has been successful
in several contexts Corsello et al. (2017); Himmelstein et al. (2022); Al-
Saleem et al. (2021) and may be the only viable strategy during an emerging
pandemic.

Decisions about repurposing drug treatments are predicated on scientific
knowledge. Making predictions about how to repurpose an existing drug requires
understanding the target disease’s mechanism. Because SARS-CoV-2 was a new
virus, our knowledge of COVID-19’s mechanism rapidly evolved. The biomedical
literature about the virus and disease proliferated at an unprecedented rate
Ioannidis et al. (2022a, b). The need for knowledge about the virus and the
bottleneck of limited peer reviewers led to many cases of circumventing
typical quality control mechanisms for research. To inform their clinical
practice, healthcare professionals relied on knowledge sources of lower
scientific quality including early clinical reports with small sample sizes
and non-peer reviewed manuscripts posted on preprint servers Nouri et al.
(2021). This deluge of rapidly changing information became an “infodemic”, and
it became infeasible for the average clinician to stay up-to-date with the
growing literature The Lancet Infectious Diseases (2020).

Automated methods have great potential to help domain experts fight such an
infodemic. We illustrate this potential with a case study focused on
automatically detecting contradictory research claims in the COVID-19
therapeutics literature. We frame this as a natural language inference (NLI)
problem: given pairs of research claims in biomedical literature, we develop
models that predict whether they entail, contradict, or are neutral with
respect to each other. Our models are trained on a new dataset of these claim
pairs extracted from the CORD-19 dataset (Wang et al., 2020a) and annotated by
domain experts. Our best models are trained on curricula (Bengio et al., 2009)
of existing NLI datasets and our domain-specific one. These models are
effective at the NLI task, but the ultimate test of their value is whether
they can help domain experts. We show how these models can help a domain
expert to see early on that hydroxychloroquine was an ineffective COVID-19
treatment and how the story of remdisivir was still emerging.

##  2 COVID-19 NLI Dataset

Our new COVID-19 NLI dataset consists of pairs of research claims describing
COVID-19 drug treatment efficacy and safety. These claims came from the subset
of the June 17, 2020 (v33) CORD-19 (Wang et al., 2020a) manuscripts containing
a COVID-19-related term (e.g., “SARS-CoV-2”, “2019-nCov”). Claims were
extracted from the articles’ full text using the LSTM approach of Achakulvisut
et al. (2020). False positive research claims were manually removed.

To begin the annotation process, we inspected pairs of claims on common drugs
and topics. This led us to a set of five categories: Strict Entailment,
Entailment, Possible Entailment, Strict Contradiction, Contradiction, and
Neutral. Our annotation guidelines were developed and refined by clinically
trained annotators (nurses and a biomedical researcher) over two preliminary
rounds of annotation. In Round 1, four annotators labeled 64 claim pairs
(Fleiss’ $\kappa$ = 0.83). The team discussed this process and refined the
guidelines. In Round 2, three annotators (a subset of those from Round 1)
annotated 75 claim pairs (Fleiss’ $\kappa$ = 0.84) using the new guidelines,
and then determined that they were ready to scale (Appendix A.1).

For the dataset itself, 1000 pairs of claims were sampled for annotation using
three criteria: (1) both claims mention at least one of 7 treatment candidates
({“hydroxychloroquine”, “chloroquine”, “tocilizumab”, “remdesivir”, “vitamin
D”, “lopinavir”, “dexamethasone”}), (2) high similarity between the claim’s
embedding and the embedding for a word in a predefined topic list
({“mortality”, ”effective treatment”, “toxicity”}), using uSIF embeddings
(Ethayarajh, 2018), and (3) non-zero polarities of equal or opposite sign
using VADER (Hutto and Gilbert, 2014). Appendix A.2 provides further details.

Each annotation requires a large time investment from the annotator and draws
heavily on their domain expertise, so each example was annotated by a single
annotator, with the inter-annotator agreement rounds and guidelines serving to
ensure consistency across the dataset.

Because some claims are present in multiple claim pairs, we selected a subset
of pairs such that no claim is present in more than one train, validation, or
test split to prevent test-set leakage. From the network of claim pairs
(claims are nodes, and co-occurrences in an annotated pair are edges), we
selected 3 disjoint subnetworks to comprise the train, validation, and test
splits. The resulting dataset contains 778 total claim pairs.

##  3 Model Development

Our goal is to develop a model to help domain experts find and adjudicate
contradictory claims in the COVID-19 literature. We explored a wide range of
techniques for developing the best model given our available data. The
Appendices provide a full overview of all these experiments and comparisons.
Here, we provide a high-level summary.

##### Pretrained Parameters

All our models begin with pretrained parameters created using the general
architecture of BERT (Devlin et al., 2019). Five pre-trained BERT models were
evaluated for further fine-tuning: PubMedBERT (Gu et al., 2021), SciBERT
(Beltagy et al., 2019), BioBERT (Lee et al., 2020), BioClinBERT (Alsentzer et
al., 2019), and RoBERTa Liu et al. (2019). We found that PubMedBERT was the
best for our task across all fine-tuning regimes (Appendix D).

##### Fine-tuning Curricula

For fine-tuning these parameters, we use MultiNLI (Williams et al., 2018),
MedNLI (Romanov and Shivade, 2018), ManConCorpus (Alamri and Stevenson, 2016),
and our new COVID-19 NLI Dataset (with our six labels collapsed to three as in
the other datasets). We found that the best models were achieved with a
curriculum that arranged these in the order we gave above. This is intuitively
an arrangement from most general to most domain specific, which aligns with
existing results and intuitions for curriculum learning (Bengio et al., 2009;
Xu et al., 2020; Nagatsuka et al., 2021). For detailed descriptions of these
datasets, the range of curricula we explored, and our procedures for
hyperparameter tuning, we refer to Appendices B, C, and E, respectively.

##### Results

To contextualize our results on this hard, novel task, we evaluated a number
of baselines using sparse feature representations and simple similarity
calculations, as well as hypothesis-only variants of these models and our BERT
models. These baselines are described in Appendix F.

|  |  | Contra.
---|---|---|---
Model | Curriculum | F1 | Recall
PubMedBERT | Forward | 0.690 | 0.571
Reverse | 0.428 | 0.381
Shuffled | 0.523 | 0.416
RoBERTa | Forward | 0.544 | 0.429
Reverse | 0.411 | 0.476
Shuffled | 0.239 | 0.119
PubMedBERT Hyp. only | Forward | 0.485 | 0.190
RoBERTa Hyp. only | Forward | 0.433 | 0.095

Table 1: Core results. Figure 6 and Table 3 expand these results to include a
number of other baselines, most of which perform near chance. Metrics for the
shuffled category are averages of the 4 shuffled curricula.

Table 1 summarizes our results. We report F1 scores as well as Contradictions
Recall, an important category for our case study. The best performance is
achieved by the PubMedBERT model trained with the forward curriculum where
fine-tuning takes place from general domain to complex, in-domain datasets.
This setting outperforms baselines and alternative curricula by a large
margin.

##  4 Case Study: Wading Through the Sea of Drug Treatment Literature

The value of our model lies in its potential to help domain experts tackle an
infodemic. Our biomedical research group used the model to understand the
state of knowledge about the efficacy and mechanism of two controversial
treatments, hydroxychlorouqine and remdesivir, from the perspective of June
2020.

We first extracted all claims identified from COVID-19 manuscripts concerning
a drug treatment, using the same procedure as for our COVID NLI dataset
(Section 2), and we filtered that set to pairs of claims that were (1)
sufficiently similar (uSIF similarity > 0.5) and (2) both mentioned remdesivir
or hydroxychloroquine. We sampled pairs from 50 papers yielding 5,336 total
pairs. We then used our best model to make predictions about all these pairs
resulting in 322 predicted contradictions. We ranked these by the model’s
predicted probability of this class, and we inspected the highest probability
predictions.

For remdesivir, one claim of limited efficacy from an clinical trial of 233
participants yielded several predicted contradictions:

  * Remdesivir did not result in significant reductions in SARS-CoV-2 RNA loads or detectability in upper respiratory tract or sputum specimens in this study despite showing strong antiviral effects in preclinical models of infection with coronaviruses Wang et al. (2020b).

Nineteen unique papers contained a claim that was predicted to contradict this
claim – already a striking pattern that might have taken a researcher days to
discover by hand by probing full text articles.

The specific claims that contradict our core claim are illuminating. One
reads,

  * The present study reveals that remdesivir has the highest potential in binding and therefore competitively inhibiting RDRP of SARS-CoV-2, among all known RDRP inhibitors Choudhury et al. (2021),

indicating strong chemical and pharmacodynamic reasoning supporting a
mechanism of action for remdesivir. A second claim describes:

  * Remdesivir treatment in rhesus macaques infected with SARS-CoV-2 was highly effective in reducing clinical disease and damage to the lungs Williamson et al. (2020),

surfacing particularly strong pre-clinical evidence. From another ongoing
clinical trial including 1,064 patients, authors note:

  * Preliminary results of this trial suggest that a 10-day course of remdesivir was superior to placebo in the treatment of hospitalized patients with COVID-19. Beigel et al. (2020)

Overall, we are quickly able to glean how evidence supporting the remdesivir
hypothesis was strong from a variety of pre-clinical studies in vastly
different settings in 2020. Our original negative claim (4) presents real
evidence against the drug. Still, though, the clinical picture was not yet
clear, suggesting the need for further clinical investigation or better
striation of populations or therapeutic windows for seeing efficacy.

For hydroxychloroquine, one of the earliest drugs considered, a different
picture emerges. We focus in on a claim from a medRxiv preprint (4):

  * In summary, this retrospective study demonstrates that hydroxychloroquine application is associated with a decreased risk of death in critically ill COVID-19 patients without obvious toxicity and its mechanisms of action is probably mediated through its inhibition of inflammatory cytokine storm on top of its ability in inhibiting viral replication. Yu et al. (2020)

From its predicted contradictions, we immediately identified two clinical
studies:

  * Overall, these data do not support the addition of hydroxychloroquine to the current standard of care in patients with persistent mild to moderate COVID-19 for eliminating the virus. Tang et al. (2020)

Although a marginal possible benefit from prophylaxis in a more at-risk group
cannot be ruled out, the potential risks that are associated with
hydroxychloroquine may also be increased in more at-risk populations, and this
may essentially negate any benefits that were not shown in this large trial
involving younger, healthier participants. Boulware et al. (2020)

These claims reflect the challenging language typical for the domain including
hedging, multiple clauses, important context qualifiers (subpopulations and
adverse events), and positive and negative sentiments. From these surfaced
contradictions, we find evidence of the drug’s inefficacy in mild and moderate
cases and are led to discover the early observations of cardiac arrest being
associated with hydroxychloroquine treatment. Again, discovering these claims
de novo is difficult given the size of the corpus of COVID-19 literature. Our
NLI model greatly speeds up the process and allows domain experts to home in
directly on relevant evidence.

##  5 Stakeholders

There are several biomedical stakeholders who would benefit from models like
ours.

##### Epidemiologists

Epidemiologists survey public health data to inform policy decisions in
collaboration with authoritative bodies like the NIH and WHO. Their
recommendations must be conservative, so surfacing results that dispute claims
of drug efficacy is critical. Their gold standard resource for aggregating
evidence is the meta-analysis, but in the early stages of the pandemic, large
randomized controlled trials (RCTs) had not completed, and review articles
quickly became outdated.

##### FDA Regulators

Regulators too need to make conservative recommendations, as FDA approval
signals to clinicians that a treatment is standard-of-care. Surfacing
contradictory claims of drug efficacy and safety is essential Cassidy et al.
(2020).

##### Researchers

By identifying areas of scientific uncertainty via contradictory evidence at
all stages of the pipeline (in silico, in vitro, in vivo, clinical),
researchers could have more quickly identified fruitful areas of investigation
Sosa et al. (2021).

##### Drug Manufacturers

Manufacturers of repurposing candidates were incentivized to understand in
what settings their drug seemed to be effective and by what mechanism. For
claims of inefficacy, they were interested in surfacing any mitigating factors
qualifying these claims or motivating follow-up analyses.

##  6 Discussion and Conclusion

In settings where the scale of literature is insurmountable for human readers,
as is the case during a pandemic, automated curatorial assistants can be
transformative Lever and Altman (2021). During COVID-19, meta-analyses and
review articles, which are written to synthesize a large body of literature,
could not be comprehensive or quickly became outdated. In some cases, it was
necessary to create meta-meta-analyses involving hundreds of papers Chivese et
al. (2021).

Our work shows the value of integrating NLP into the domain of meta-science,
embracing all the complexities of biomedical research as it naturally exists
in literature. We presented an NLI framing for identifying contradictory or
corroborating research claims in the challenging domain of COVID-19 drug
efficacy. We created a new dataset and designed curricula for optimizing
language model fine-tuning for the task. To illustrate the potential of our
model, we showed that we were quickly able to distill the state of knowledge
about hydroxychlorouqine and remdesivir efficacy as of June 2020, arriving at
conclusions that are extremely well-supported in 2022.

Identifying where science is not consistent is necessary for understanding the
current state of human knowledge and reveals frontiers for further research.
Significant contradictions can often be found buried in biomedical articles;
surfacing these instances nearly as quickly as research is publicly
disseminated can generate leads that researchers and curators should pursue.
Beyond facilitating search and discovery, our method can help estimate
confidence in the consensus of facts in science when creating general
knowledge representations Sosa and Altman (2022) for downstream applications
like predicting novel drug repurposing opportunities in silico Sosa et al.
(2020).

## Acknowledgements

Special thanks to our clinical annotation team for their invaluable
contributions to this work. DS is supported by NIH training grant LM007033,
Stanford Data Science, and BenevolentAI. RBA is supported by NIGMS GM102365,
HG010615, and Chan Zuckerberg Biohub.

## References

  * Achakulvisut et al. (2020) Titipat Achakulvisut, Chandra Bhagavatula, Daniel Acuna, and Konrad Kording. 2020\.  Claim extraction in biomedical publications using deep discourse model and transfer learning.  _arXiv:1907.00962_.
  * Al-Saleem et al. (2021) Jacob Al-Saleem, Roger Granet, Srinivasan Ramakrishnan, Natalie A. Ciancetta, Catherine Saveson, Chris Gessner, and Qiongqiong Zhou. 2021.  Knowledge graph-based approaches to drug repurposing for COVID-19.  _Journal of Chemical Information and Modeling_ , 61(8):4058–4067.
  * Alamri and Stevenson (2016) Abdulaziz Alamri and Mark Stevenson. 2016.  A corpus of potentially contradictory research claims from cardiovascular research abstracts.  _Journal of Biomedical Semantics_ , 7.
  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Beigel et al. (2020) John H. Beigel, Kay M. Tomashek, Lori E. Dodd, Aneesh K. Mehta, Barry S. Zingman, Andre C. Kalil, Elizabeth Hohmann, Helen Y. Chu, Annie Luetkemeyer, Susan Kline, Diego Lopez de Castilla, Robert W. Finberg, Kerry Dierberg, Victor Tapson, Lanny Hsieh, Thomas F. Patterson, Roger Paredes, Daniel A. Sweeney, William R. Short, Giota Touloumi, David Chien Lye, Norio Ohmagari, Myoung-don Oh, Guillermo M. Ruiz-Palacios, Thomas Benfield, Gerd Fätkenheuer, Mark G. Kortepeter, Robert L. Atmar, C. Buddy Creech, Jens Lundgren, Abdel G. Babiker, Sarah Pett, James D. Neaton, Timothy H. Burgess, Tyler Bonnett, Michelle Green, Mat Makowski, Anu Osinusi, Seema Nayak, and H. Clifford Lane. 2020.  Remdesivir for the treatment of COVID-19 a final report.  _New England Journal of Medicine_ , 383(19):1813–1826.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009.  Curriculum learning.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , ICML ’09, pages 41–48, New York, NY, USA. Association for Computing Machinery.
  * Borchers (2019) Oliver Borchers. 2019.  Fast sentence embeddings.  https://github.com/oborchers/Fast_Sentence_Embeddings.
  * Boulware et al. (2020) David R. Boulware, Matthew F. Pullen, Ananta S. Bangdiwala, Katelyn A. Pastick, Sarah M. Lofgren, Elizabeth C. Okafor, Caleb P. Skipper, Alanna A. Nascene, Melanie R. Nicol, Mahsa Abassi, Nicole W. Engen, Matthew P. Cheng, Derek LaBar, Sylvain A. Lother, Lauren J. MacKenzie, Glen Drobot, Nicole Marten, Ryan Zarychanski, Lauren E. Kelly, Ilan S. Schwartz, Emily G. McDonald, Radha Rajasingham, Todd C. Lee, and Kathy H. Hullsiek. 2020.  A randomized trial of hydroxychloroquine as postexposure prophylaxis for COVID-19.  _New England Journal of Medicine_ , 383(6):517–525.
  * Cassidy et al. (2020) Christine Cassidy, Danielle Dever, Laura Stanbery, Gerald Edelman, Lance Dworkin, and John Nemunaitis. 2020.  FDA efficiency for approval process of COVID-19 therapeutics.  _Infectious Agents and Cancer_ , 15(1):73.
  * Chivese et al. (2021) Tawanda Chivese, Omran A.H. Musa, George Hindy, Noor Al-Wattary, Saif Badran, Nada Soliman, Ahmed T.M. Aboughalia, Joshua T. Matizanadzo, Mohamed M. Emara, Lukman Thalib, and Suhail A.R. Doi. 2021.  Efficacy of chloroquine and hydroxychloroquine in treating COVID-19 infection: A meta-review of systematic reviews and an updated meta-analysis.  _Travel Medicine and Infectious Disease_ , 43:102135.
  * Choudhury et al. (2021) Shuvasish Choudhury, Debojyoti Moulick, Purbajyoti Saikia, and Muhammed Khairujjaman Mazumder. 2021.  Evaluating the potential of different inhibitors on RNA-dependent RNA polymerase of severe acute respiratory syndrome coronavirus 2: A molecular modeling approach.  _Medical Journal Armed Forces India_ , 77:S373–S378.
  * Corsello et al. (2017) Steven M. Corsello, Joshua A. Bittker, Zihan Liu, Joshua Gould, Patrick McCarren, Jodi E. Hirschman, Stephen E. Johnston, Anita Vrcic, Bang Wong, Mariya Khan, Jacob Asiedu, Rajiv Narayan, Christopher C. Mader, Aravind Subramanian, and Todd R. Golub. 2017.  The drug repurposing hub: a next-generation drug library and information resource.  _Nature Medicine_ , 23(4):405–408.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ethayarajh (2018) Kawin Ethayarajh. 2018.  Unsupervised random walk sentence embeddings: A strong but simple baseline.  In _Proceedings of the Third Workshop on Representation Learning for NLP_ , pages 91–100, Melbourne, Australia. Association for Computational Linguistics.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare_ , 3(1):2:1–2:23.
  * Himmelstein et al. (2022) Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. 2022.  Systematic integration of biomedical knowledge prioritizes drugs for repurposing.  _eLife_ , 6:e26726.
  * Hutto and Gilbert (2014) CJ Hutto and Eric Gilbert. 2014.  VADER: A parsimonious rule-based model for sentiment analysis of social media text.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 8(1):216–225.
  * Ioannidis et al. (2022a) John P. A. Ioannidis, Eran Bendavid, Maia Salholz-Hillel, Kevin W. Boyack, and Jeroen Baas. 2022a.  Massive covidization of research citations and the citation elite.  _Proceedings of the National Academy of Sciences_ , 119(28):e2204074119.
  * Ioannidis et al. (2022b) John P. A. Ioannidis, Maia Salholz-Hillel, Kevin W. Boyack, and Jeroen Baas. 2022b.  The rapid, massive growth of COVID-19 authors in the scientific literature.  _Royal Society Open Science_ , 8(9):210389.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  BioBERT: A pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lever and Altman (2021) Jake Lever and Russ B. Altman. 2021.  Analyzing the vast coronavirus literature with CoronaCentral.  _Proceedings of the National Academy of Sciences_ , 118(23):e2100766118.
  * Li et al. (2013) Yue-hua Li, Cheng-hui Zhou, Han-jun Pei, Xian-liang Zhou, Li-huan Li, Yong-jian Wu, and Ru-tai Hui. 2013.  Fish consumption and incidence of heart failure: A meta-analysis of prospective cohort studies.  _Chinese Medical Journal_ , 126(5):942–948.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv:1907.11692_.
  * Nagatsuka et al. (2021) Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. 2021.  Pre-training a BERT with curriculum learning by increasing block-size of input text.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , pages 989–996.
  * Nouri et al. (2021) Shayan N. Nouri, Yosef A. Cohen, Mahesh V. Madhavan, Piotr J. Slomka, Ami E. Iskandrian, and Andrew J. Einstein. 2021.  Preprint manuscripts and servers in the era of coronavirus disease 2019.  _Journal of Evaluation in Clinical Practice_ , 27(1):16–21.
  * Pushpakom et al. (2019) Sudeep Pushpakom, Francesco Iorio, Patrick A. Eyers, K. Jane Escott, Shirley Hopper, Andrew Wells, Andrew Doig, Tim Guilliams, Joanna Latimer, Christine McNamee, Alan Norris, Philippe Sanseau, David Cavalla, and Munir Pirmohamed. 2019\.  Drug repurposing: Progress, challenges and recommendations.  _Nature Reviews Drug Discovery_ , 18(1):41–58.
  * Romanov and Shivade (2018) Alexey Romanov and Chaitanya Shivade. 2018.  Lessons from natural language inference in the clinical domain.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1586–1596, Brussels, Belgium. Association for Computational Linguistics.
  * Schardt et al. (2007) Connie Schardt, Martha B. Adams, Thomas Owens, Sheri Keitz, and Paul Fontelo. 2007\.  Utilization of the PICO framework to improve searching PubMed for clinical questions.  _BMC Medical Informatics and Decision Making_ , 7(1):16.
  * Sosa and Altman (2022) Daniel N Sosa and Russ B Altman. 2022.  Contexts and contradictions: A roadmap for computational drug repurposing with knowledge inference.  _Briefings in Bioinformatics_ , 23(4):bbac268.
  * Sosa et al. (2021) Daniel N. Sosa, Binbin Chen, Amit Kaushal, Adam Lavertu, Jake Lever, Stefano Rensi, and Russ Altman. 2021.  Repurposing biomedical informaticians for COVID-19.  _Journal of Biomedical Informatics_ , 115:103673.
  * Sosa et al. (2020) Daniel N. Sosa, Alexander Derry, Margaret Guo, Eric Wei, Connor Brinton, and Russ B. Altman. 2020.  A literature-based knowledge graph embedding method for identifying drug repurposing opportunities in rare diseases.  _Pacific Symposium on Biocomputing_ , 25:463–474.
  * Tang et al. (2020) Wei Tang, Zhujun Cao, Mingfeng Han, Zhengyan Wang, Junwen Chen, Wenjin Sun, Yaojie Wu, Wei Xiao, Shengyong Liu, Erzhen Chen, Wei Chen, Xiongbiao Wang, Jiuyong Yang, Jun Lin, Qingxia Zhao, Youqin Yan, Zhibin Xie, Dan Li, Yaofeng Yang, Leshan Liu, Jieming Qu, Guang Ning, Guochao Shi, and Qing Xie. 2020.  Hydroxychloroquine in patients with mainly mild to moderate coronavirus disease 2019: open label, randomised controlled trial.  _BMJ_ , 369:m1849.
  * The Lancet Infectious Diseases (2020) The Lancet Infectious Diseases. 2020.  The COVID-19 infodemic.  _The Lancet Infectious Diseases_ , 20(8):875.
  * Wang et al. (2020a) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020a.  CORD-19: The COVID-19 open research dataset.  In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_. Association for Computational Linguistics.
  * Wang et al. (2020b) Yeming Wang, Dingyu Zhang, Guanhua Du, Ronghui Du, Jianping Zhao, Yang Jin, Shouzhi Fu, Ling Gao, Zhenshun Cheng, Qiaofa Lu, Yi Hu, Guangwei Luo, Ke Wang, Yang Lu, Huadong Li, Shuzhen Wang, Shunan Ruan, Chengqing Yang, Chunlin Mei, Yi Wang, Dan Ding, Feng Wu, Xin Tang, Xianzhi Ye, Yingchun Ye, Bing Liu, Jie Yang, Wen Yin, Aili Wang, Guohui Fan, Fei Zhou, Zhibo Liu, Xiaoying Gu, Jiuyang Xu, Lianhan Shang, Yi Zhang, Lianjun Cao, Tingting Guo, Yan Wan, Hong Qin, Yushen Jiang, Thomas Jaki, Frederick G. Hayden, Peter W. Horby, Bin Cao, and Chen Wang. 2020b.  Remdesivir in adults with severe COVID-19: A randomised, double-blind, placebo-controlled, multicentre trial.  _Lancet (London, England)_ , 395(10236):1569–1578.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Williamson et al. (2020) Brandi N. Williamson, Friederike Feldmann, Benjamin Schwarz, Kimberly Meade-White, Danielle P. Porter, Jonathan Schulz, Neeltje van Doremalen, Ian Leighton, Claude Kwe Yinda, Lizzette Pérez-Pérez, Atsushi Okumura, Jamie Lovaglio, Patrick W. Hanley, Greg Saturday, Catharine M. Bosio, Sarah Anzick, Kent Barbian, Tomas Cihlar, Craig Martens, Dana P. Scott, Vincent J. Munster, and Emmie de Wit. 2020.  Clinical benefit of remdesivir in rhesus macaques infected with SARS-CoV-2.  _Nature_ , 585(7824):273–276.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45. Association for Computational Linguistics.
  * World Health Organization (2022) World Health Organization. 2022.  Weekly epidemiological update on COVID-19 - 16 November 2022.
  * Wouters et al. (2020) Olivier J. Wouters, Martin McKee, and Jeroen Luyten. 2020.  Estimated research and development investment needed to bring a new medicine to market, 2009-2018.  _JAMA_ , 323(9):844–853.
  * Xu et al. (2020) Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020.  Curriculum learning for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6095–6104. Association for Computational Linguistics.
  * Yu et al. (2020) Bo Yu, Chenze Li, Peng Chen, Ning Zhou, Luyun Wang, Jia Li, Hualiang Jiang, and Dao Wen Wang. 2020.  Hydroxychloroquine application is associated with a decreased mortality in critically ill patients with COVID-19.  _medRxiv_.

## Supplementary Materials

##  Appendix A Further Details about the COVID-19 NLI Dataset

In this appendix we provide additional details about the creation of the
COVID-19 NLI dataset. Our annotators are experts in the domain having trained
as healthcare providers (nursing) and annotation. The research annotator (DS)
is a specialist in the biomedical domain with background in molecular biology
and computer science. Annotators have also provided span annotations in
several cases of drug mention, polarity, context, and expressions of
uncertainty to aid in the annotation task.

###  A.1 Inter-Annotator Analysis

Two rounds of inter-annotator analysis were conducted to converge on a set of
annotation guidelines for scaling and to measure consistency between multiple
annotators. In the first round four annotators (three clinical annotators, one
researcher) were presented with 64 pairs of extracted research claims and an
initial set of annotation guidelines. Classification was conducted across five
classes including a Strict Entailment and Strict Contradiction class
indicating two claims were entailing or contradicting in a strict logical
sense as opposed to a common-reasoning sense. Global Fleiss’ $\kappa$ for this
round was 0.83. For the second round, three annotators (two clinical
annotators, one researcher) annotated 75 claim pairs with updated guidelines
and achieved similar consistency at $\kappa=0.84$. Further minor modifications
were made to the annotation guidelines resulting in the final guidelines used
for the scaling round (Table 2).

Criteria | Annotation
---|---
All drugs, context, and sentiment match | STRICT ENTAILMENT
At least one drug matches, the sentiment is the same but the context is at least similar | ENTAILMENT
All drugs and context match but the sentiment is opposing | STRICT CONTRADICTION
At least one drug matches, the sentiment is opposing but the context is at least similar | CONTRADICTION
The context or sentiment statement cannot be compared | NEUTRAL
There is no mention of a drug OR none of the drugs match | NEUTRAL
One claim contains both a POSITIVE and a NEGATIVE statement and the other claim contains a POSITIVE or NEGATIVE statement | CONTRADICTION
One claim is POSITIVE or NEGATIVE statement and the other is EXPRESSION_OF_UNCERTAINTY | NEUTRAL
Both claims are EXPRESSION_OF_UNCERTAINTY | ENTAILMENT
Table 2: Annotation guidelines for the COVID-19 NLI dataset.

###  A.2 Preparing Claims for Annotation

We used a set of heuristics to generate a set of non-trivial (obviously
neutral) claim pairs for annotators to label. First, we considered three
topics, $t\in T=$ { “mortality”, “effective treatment”, “toxicity”} and seven
drugs, $d\in D=$ {“hydroxychloroquine”, “chloroquine”, “tocilizumab”,
“remdesivir”, “vitamin D”, “lopinavir”, “dexamethasone”}. For each pair,
$(t,d)$, the following procedure (Algorithm 1) was used to generate candidate
claim pairs from the set of true research claims, $C$. Additionally given
pol(.), a function for calculating the polarity of a claim; $k$, the number of
claims to sample that are relevant to a drug and topic and have a given
polarity (positive or negative); and $N$, the total number of pairs to
subsample, we define our heuristic algorithm for generating candidate non-
trivial pairs in Algorithm 1.

Algorithm 1 Heuristic sampler for generating candidate non-trivial pairs

1:Topic set $T$, drug set $D$, claim set $C$, polarity function
$\textit{pol}(.):c\rightarrow[-1,1]$, drug topic claim sample size $k$, total
subsample size $N$

2:Set of $N$ claim pairs $P_{N}$ concerning a common drug and topic and non-
neutral predicted polarity

3:$P\leftarrow\emptyset$

4:for $(d,t)\in D\times T$ do

5: Retrieve claims $C_{d}:=\\{c\in C:d\text{ is a substring of }c\\}$

6: Define $C_{d,t,k,\textit{pos}}$ := top $k$ claims $c$ relevant to $t$ from
$C_{d}$ s.t. $\textit{pol}(c)>0$

7: Define $C_{d,t,k,\textit{neg}}$ := top $k$ claims $c$ relevant to $t$ from
$C_{d}$ s.t. $\textit{pol}(c)<0$

8: Enumerate all combinations of claim pairs, $P_{d,t,2k}$, from claims in set
$C_{d,t,k,\textit{pos}}\cup C_{d,t,k,\textit{neg}}$

9: Remove copy claim pairs, $P_{d,t,2k}\leftarrow
P_{d,t,2k}\setminus\\{(c_{1},c_{2})\in P_{d,t,2k}:c_{1}=c_{2}\\}$

10: $P\leftarrow P\cup P_{d,t,2k}$

11:end for

12:Sample $N$ pairs uniformly from $P$, $P_{N}$

We set $k=7$ and $N=1000$. To evaluate claim relevancy (lines 4 and 5), we
calculate the cosine similarity between an embedding of the topic and sentence
embeddings of claims using uSIF Ethayarajh (2018). Polarity,
$\textit{pol}(.)$, is calculated using Vader scores Hutto and Gilbert (2014).

##  Appendix B Curriculum Datasets

We included four datasets for fine-tuning our language models, which comprise
general language and multiple biomedically-focused domains. All our datasets
use the labels Entailment, Contradiction, and Neural. For our COVID-19 NLI
dataset, we collapse Strict Entailment with Entailment and Strict
Contradiction with Contradiction.

###  B.1 MultiNLI

MultiNLI is an NLI dataset consisting of 433k premise-hypothesis pairs taken
from 5 general domains (Williams et al., 2018). To create the dataset,
annotators were shown a premise and were asked to provide hypothesis
statements that were entailed by, contradicted by, or were neutral to the
prompt premise. In this work, we used the matched validation set for
evaluation, which we split into two equal sized validation and test sets.

###  B.2 MedNLI

MedNLI is an NLI dataset consisting of 14k premise-hypothesis pairs where
premises are extracted from doctor’s notes in electronic medical records
(Romanov and Shivade, 2018). The annotation task for generating premise-
hypothesis pairs was analogous to that for MultiNLI.

###  B.3 ManConCorpus

ManConCorpus is a dataset of research claims taken from biomedical systematic
reviews (Alamri and Stevenson, 2016). These reviews compile together studies
that investigate a common research questions and consider their findings in
aggregate. The research question, which conforms to the standardized PICO
criteria (Schardt et al., 2007), yields a binary answer, so findings from the
associated review will take explicit “yes” or “no” stances. One such PICO
question is “In elderly populations, does omega 3 acid from fatty fish intake,
compared with no consumption, reduce the risk of developing heart failure?”
(Li et al., 2013).

Pairs of claims manually annotated from these works can be paired together for
NLI classification by matching claims that take the same stance on a common
question as entailing pairs, those that take opposite stances on a common
question as contradicting pairs, and those taken from two different reviews
about different questions as neutral pairs. The dataset’s 16 PICO questions
are split into 12, 4, and 4 questions for the train, validation, and test
splits, respectively, and the neutral class is downsampled to be the same size
as the next largest class in all splits. The resulting dataset has 2.8k claim
pairs in total.

##  Appendix C Curriculum Design

To create an effective curriculum for the ultimate task of detecting
contradictions in the COVID-19 treatment domain, we conducted a set of
experiments analyzing the effect of multiple design decisions for
incorporating domain-adjacent corpora in training.

###  C.1 Experiments

####  C.1.1 Shuffled and Combined Curricula

To understand the importance of sequencing the curriculum, we evaluated BERT
models trained using various sequences of domain-adjacent corpora in equal
proportion. We consider three types of curricula: forward, reverse, and
shuffled. The forward curriculum proceeds with fine-tuning a pre-trained BERT
model in sequence from the most general domain (MultiNLI) to MedNLI to
ManConCorpus to the most relevant domain (COVID-19 NLI). The reverse
curriculum begins with the most relevant domain and proceeds in the opposite
direction. The shuffled curricula were sampled from the 22 possible random
orderings of the four domains excluding the forward and reverse sequences. We
sampled three shuffled domains to assess the background from non-intentional
curriculum design. Finally, we considered a “combined” curriculum where data
from the four corpora are concatenated together and shuffled, thus ablating
the notion of intentional sequencing in the curriculum. To ensure no dataset
dominated training, each dataset, $D_{\textit{train}}$, is subsampled such
that $N_{D_{\textit{train}}}=\min{(d,|D_{\textit{train}}|)}$ samples are
present in the curriculum.

####  C.1.2 Ordered Curriculum Subsequence Fine-Tuning

To assess the contribution to performance from specific domains during
sequencing as well as the effect of curriculum size, we evaluated forward
curriculum subsequences. Ten subsequences were evaluated: the full forward
curriculum, two three-dataset subsequences, three two-dataset subsequences,
and the four single corpora. As in C.1.1, $N_{D_{\textit{train}}}$ samples are
present in the curriculum from dataset $D_{\textit{train}}$.

####  C.1.3 Perturbing Dataset Proportion in Sequential Curricula

To assess whether changing the ratio of training data used from the various
corpora yielded better performance or to dilutive biases from larger corpora,
we modulated the data ratio parameter. We define data ratio, $r$, as the
multiplicative factor larger a dataset is from the next dataset in the
curriculum sequence. Specifically, given $r$, we calculate the sample size of
the dataset, $D_{\textit{train}}$, to be used in the $i$th step (1-index) of a
size-$k$ fine-tuning curriculum as
$N_{D_{\textit{train}}}=\min{(r^{k-i}d,|D_{\textit{train}}|)}$. We considered
three curricula: the full forward curriculum and the two sequential three-
dataset curricula.

###  C.2 Evaluation

Figure 1: Macro F1 evaluation on the COVID-19 NLI validation set for different
orderings of the fine-tuning curriculum and the combined curriculum (brown).
Pie point position indicate F1 value, colors indicate which corpora have been
already introduced to the model at a given fine-tuning step, and arrows
indicate curriculum sequences. Pie size is proportional to amount of training
data seen by the model.

####  C.2.1 Setup

We evaluated multiple BERT models pre-trained using general and biomedical
corpora for curriculum-based fine-tuning Devlin et al. (2019). Each fine-
tuning step involves training for 4 epochs with learning rate $l=10^{-5}$ and
batch size $b=8$. For all experiments, $d=500$, and for data ratio
experiments, $r\in\\{1,2\\}$. Pre-trained models were loaded from the
HuggingFace Transformers library Wolf et al. (2020). All fine-tuning was
conducted on 2 Tesla-V100-SXM2 and 2 Tesla-A100-PCIe GPUs. Experiments in
curriculum design were evaluated with the pre-trained PubMedBERT model (Gu et
al., 2021). Other pre-trained BERT models were evaluated on forward curriculum
subsequences (Appendix D).

####  C.2.2 Evaluation Metrics

The primary NLI evaluation metric for fine-tuned BERT models was macro F1 on
the COVID-19 NLI validation set. We also investigated recall of the
contradictions class as an important metric in evaluating the ability to
detect contradictory research claims.

####  C.2.3 Shuffled and Collapsed Curricula

Of the six tested four-dataset curricula, the forward curriculum performed
highest with an F1 of 0.503. The reverse curriculum, starting with the most
relevant and challenging curriculum first, achieved an F1 of 0.474. The
shuffled curricula yielded F1 scores of 0.380, 0.432, and 0.478. The collapsed
curriculum, in which the four corpora are concatenated and shuffled, achieved
competitive performance as well, yielding an F1 score of 0.475 (Figure 1).

####  C.2.4 Ordered Subsequences

From the 10 curriculum subsequences, the model trained with the full forward
curriculum yielded highest performance with an F1 of 0.503. Among the two
three-domain sequences, the one including the in-domain COVID-19 NLI dataset
achieved greater performance than that without, yielding F1 scores of 0.440
and 0.296 respectively. Similarly, with the two-domain subsequences, the
sequence with ManConCorpus and COVID-19 performed best with F1 of 0.434, and
the subsequence containing MedNLI and ManConCorpus performed worst with F1 of
0.275. Among the single domain curricula, the in-domain training on our
dataset was best with F1 of 0.311 (Figure 2).

####  C.2.5 Variable Dataset Proportions

In all three curricula, the condition with data ratio $r=2$ outperformed the
$r=1$ equal data proportion condition. The highest performing curriculum was
the $r=2$ forward curriculum achieving an F1 of 0.638. In the in-domain three-
dataset sequence, F1 increased from 0.416 with $r=1$ to 0.461 with $r=2$. The
out-of-domain three-sequence saw a similar increase in performance favoring
$r=2$ over $r=1$ (Figure 3).

Figure 2: Macro F1 evaluation for various subsequences of the forward
curriculum. Figure 3: Evaluation of three subsequences of the forward
curriculum with two different data ratio proportions, $r\in\\{1,2\\}$.

##  Appendix D BERT Pretraining

Figure 4: Macro F1 (A) and contradictions recall (B) for five pre-trained BERT
models: PubMedBERT, SciBERT, BioBERT, BioClinBERT, and RoBERTA fine-tuned with
subsequences of the forward curriculum.

Five pre-trained BERT models were evaluated for further fine-tuning:
PubMedBERT (Gu et al., 2021), SciBERT (Beltagy et al., 2019), BioBERT (Lee et
al., 2020), BioClinBERT (Alsentzer et al., 2019), and RoBERTA Liu et al.
(2019). We conducted fine-tuning experiments under the same 10 subsequences
and parameter settings as in Section C.1.2 and evaluated performance on the
validation split of the COVID-19 NLI dataset. For PubMedBERT, SciBERT, and
RoBERTa, the full forward curriculum yielded the greatest macro F1 scores at
0.503, 0.448, and 0.590, respectively. The greatest performance was achieved
by the MedNLI-ManCon-COVID-19 NLI subsequence for BioBERT and BioClinBERT
models yielding F1 scores of 0.433 and 0.354 (Figure 4).

##  Appendix E BERT Hyperparameter Tuning

Figure 5: Batch size and learning rate hyperparameter optimization evaluated
with the COVID-19 NLI validation metrics for PubMedBERT (A, B) and RoBERTa (C,
D) models.

We evaluated macro F1 and contradictions recall on the COVID-19 NLI validation
set over a parameter sweep of learning rates,
$lr\in\\{$510-6$,$110-5$,$310-5$,$510-5$,$110-4$,$310-4$\\}$ and batch sizes,
$b\in\\{4,8,16,32\\}$ for PubMedBERT and RoBERTa models. For both models the
highest macro F1 setting was $lr=$310-5$$ and $b=4$ yielding $F1=0.61$ and
$F1=0.64$ for PubMedBERT and RoBERTa, respectively. These settings yielded the
greatest contradictions recall of 0.51 for PubMedBERT, and settings of
$lr=$510-6$,b=4$ yielded the highest contradictions recall value of 0.39 for
RoBERTa (Figure 5).

##  Appendix F Test Set Evaluation and Baselines

We evaluated test set statistics for the COVID-19 NLI using PubMedBERT and
RoBERTa Liu et al. (2019) models fine-tuned with the forward curriculum of
MultiNLI $\rightarrow$ MedNLI $\rightarrow$ ManCon $\rightarrow$ COVID-19 NLI.
We set data ratio as being equal between the four corpora ($r=1$) (see
Appendix C.1.3), and after hyperparameter tuning of learning rate and batch
size (Appendix E) set parameters $l_{\textit{HP}}=3*10^{-5}$ and
$b_{\textit{HP}}=4$.

We compared performance of our trained BERT models to several NLI baselines.

  * •

Hypothesis-Only Unigrams Softmax classification using unigram counts in the
hypothesis (single claim).

  * •

Word Overlap Softmax classification over counts of overlapping unigrams from
the two claims.

  * •

Word Cross-Product Softmax classification over counts of pairs of words in the
cross-product between the two claims.

  * •

Similarity + Polarity Softmax classification using similarity of the two
claims as calculated using uSIF sentence embeddings Ethayarajh (2018);
Borchers (2019) and polarity of each claim using Vader polarity scores Hutto
and Gilbert (2014).

  * •

Hypothesis-Only BERT BERT classification where one of the two claims has been
ablated.

Figure 6 offers a comparison of these baselines with our proposed models,
focusing on the forward curriculum condition. We also evaluated the optimized
PubMedBERT and RoBERTa models with the reverse curriculum and four shuffled
curricula 3. We note the consistent result that the forward curriculum
performs best overall.

Figure 6: Macro F1 and contradictions recall for PubMedBERT and RoBERTa models
fine-tuned with the forward curriculum compared to NLI benchmarks. ‘HP Tune’
indicates hyperparameter tuning (Appendix E).

|  |  | Contra.
---|---|---|---
Model | Curriculum | F1 | Recall
PubMedBERT |  Multi $\rightarrow$ Med $\rightarrow$ ManCon $\rightarrow$ Covid | 0.690 | 0.571
Covid $\rightarrow$ ManCon $\rightarrow$ Med $\rightarrow$ Multi | 0.428 | 0.381
Covid $\rightarrow$ Multi $\rightarrow$ ManCon $\rightarrow$ Med | 0.486 | 0.381
Covid $\rightarrow$ ManCon $\rightarrow$ Multi $\rightarrow$ Med | 0.581 | 0.571
Med $\rightarrow$ ManCon $\rightarrow$ Covid $\rightarrow$ Multi | 0.446 | 0.381
ManCon $\rightarrow$ Multi $\rightarrow$ Covid $\rightarrow$ Med | 0.579 | 0.333
RoBERTa |  Multi $\rightarrow$ Med $\rightarrow$ ManCon $\rightarrow$ Covid | 0.544 | 0.429
Covid $\rightarrow$ ManCon $\rightarrow$ Med $\rightarrow$ Multi | 0.411 | 0.476
Covid $\rightarrow$ Multi $\rightarrow$ ManCon $\rightarrow$ Med | 0.319 | 0.476
Covid $\rightarrow$ ManCon $\rightarrow$ Multi $\rightarrow$ Med | 0.232 | 0
Med $\rightarrow$ ManCon $\rightarrow$ Covid $\rightarrow$ Multi | 0.174 | 0
ManCon $\rightarrow$ Multi $\rightarrow$ Covid $\rightarrow$ Med | 0.232 | 0

Table 3: Test set performance on optimized PubMedBERT and RoBERTa models
trained with various fine-tuning curricula.


contents=,scale=0.2,placement=bottom,opacity=0.4,position=7.2cm,50cm

ON FULFILLING THE EXIGENT NEED FOR AUTOMATING AND MODERNIZING LOGISTICS
INFRASTRUCTURE IN INDIA Enabling AI-based Integration, Digitalization, and
Smart Automation of Industrial Parks and Robotic Warehouses

## Executive Summary

In this article, we emphasize the need for Low- or Middle-Income Countries
(LMICs) to embrace Industry 4.0 and Logistics 4.0 to stay competitive. This
requires government-level interventions and policy-making to incentivize
quality product solutions and drive innovation in traditionally resistant
economic sectors. We propose the establishment of Smart Industrial Parks
(SIPs) with a focus on enhancing operational efficiencies and bringing
together MSMEs and startups targeting niche clientele with innovative Industry
4.0 solutions. SIPs along with the phased deployment of well-planned robotic
automation technologies shall enable bringing down India’s untenable logistics
costs. We discuss the concept of a Supply Hub in an Industrial Park (SHIP) as
a solution to efficiently utilize land inside industrial parks for
manufacturing and logistics purposes. The execution of SIPs faces challenges,
particularly in competitive markets with fluctuating demand and time-sensitive
customer demands. To tackle this, we are required to implement the efficient
allocation of manufacturing resources and capabilities within SIPs. Thus, we
emphasize the importance of efficient resource utilization, collaboration, and
technology adoption in industrial parks to promote industrial development and
economic growth. We advocate the use of a cloud-based cyber-physical system
for real-time data access and analysis in SIPs. Such centralized cloud-based
monitoring of factory floors, warehouses, and industrial units using IoT
infrastructure shall improve decision-making, efficiency, and safety. Digital
Twins (DTs), which are cyber-replicas of physical systems, could play a
significant role in enabling simulation, optimization, and real-time
monitoring of smart manufacturing and distributed manufacturing systems.
However, there are several challenges involved in implementing DTs in
distributed manufacturing systems, such as defining data schemas and
collaboration protocols, ensuring interoperability, the need for effective
authentication technology, distributed machine learning models, and
scalability to manage multiple DTs. Our proposed vision to address these
difficulties toward the establishment of Smart Industrial Parks is completely
in line with the National Logistics Policy (NLP) and Prime Minister Gati
Shakti National Master Plan for Multi-modal Connectivity and is also strongly
related to the National Smart Cities Mission.

## Introduction

As society undergoes numerous technological advancements while tackling
emerging social challenges that are characteristic of the LMICs in the Global
South, it produces exigent circumstances requiring every operational aspect of
human society to be transformed in order to adapt to the changing world around
us. The backbone of any aspirational country’s economy, the production and
manufacturing sectors, have also witnessed a sea change. The vision of
Industry $4.0$ encompasses the major ways in which this industrial sector is
transforming by addressing individualization and decentralization of products,
flexibility, and scalability of production systems while keeping up with
sustainability goals. The main building blocks of Industry $4.0$ are Cyber-
Physical Systems (CPS) and the Internet of Things (IoT). In the age of
Industry $4.0$, automation, artificial intelligence, and IoT seamlessly
integrate with manufacturing processes in order to improve their efficiency
and ultimately produce AI-enabled smarter products.

However, simply focusing on innovations in the manufacturing process is not
enough. The optimal utilization of the country’s resources and realized
profitability hinges on implementing efficient logistics infrastructure
nationwide. Such a reciprocal vision for the logistics sector is dubbed as
Logistics $4.0$ [10]. Logistics $4.0$ refers to developing logistics solutions
with the objective of meeting mass product customization demands by utilizing
new-age AI/ML technologies based on IoT and CPS frameworks while not
increasing incurred costs and meeting sustainability goals. Logistics $4.0$ is
characterized by deploying real-time big data analytics for optimization,
autonomous robots for better inventory management, and seamless error-free
lightning-speed information exchange. Although Logistics $4.0/$Industry $4.0$
promises to boost profit margins along with increased growth opportunities,
many industrial enterprises appear to be hesitant in transforming their
operations because of the significant efforts required for the same and
uncertainty about the technical know-how to achieve this goal.

Although Indian manufacturers have increased investment towards the vision of
industry $4.0$ by $10$ times over a decade, we are still far behind meeting
Industry $4.0$ standards. However, the adoption rate of Industry $4.0$ has
recently increased because of the disruption caused by COVID-19. So, it is
expected that by $2025$, about two-thirds of the Indian manufacturers will
embrace Industry 4.0 standards. In $2021$, the total money spent by companies
under the category of manufacturing technologies was $20\%$ and it is expected
to increase by $40\%$ by $2025$. Automobile, electrical, electronics,
chemical, and pharmaceutical companies are at the forefront of Industry $4.0$
adoption. It is common wisdom that in those economic sectors that are
culturally resistant to change, the innovation mindset can be boosted by
government-level interventions and policy-making that shift the focus away
from investment costs and towards incentivizing quality product solutions that
are internationally competitive [4].

In the context of Logistics $4.0$, the US-based Plus One Robotics (Texas, US)
and Amazon Robotics have made excellent progress. Their much-touted warehouse
automation solutions have significant implications for the Indian retail and
logistics sector. Plus One Robotics provides robot perception solutions and
logistics automation such as PickOne, which is a 3D and AI-powered vision
software for induction, singulation, depalletizing, and packing. It has also
developed Yonder, which is a remote supervisor software suite for robotic
applications. The India-based Addverb Technologies delivers robotics solutions
for material movement, order sequencing, picking and sorting, and Automated
storage and retrieval systems (ASRS). It also develops warehouse management
systems (WMSs) and robot fleet management systems (RMSs). Another India-based
robotics company, Peer Robotics is bringing innovations to the field of lean
manufacturing with its collaborative mobile robots. GreyOrange is another
warehouse robotics company with an Indian-origin CEO that is delivering
promising AI-based solutions. Another startup called GrayMatter Robotics
founded in $2020$ by researchers in Prof. S.K. Gupta’s lab has already
clinched series A financing because of its efficient surface finishing
automation services.

We know that maturity models are useful for businesses to measure how capable
they are of continuous improvement. They are well-suited to provide a discrete
roadmap for the transition process toward designing completely networked
production. The initial maturity level appears to be mostly chaotic while
lacking rules and process definitions. But in the final optimized level, all
processes get defined, documented, standardized, analyzed, controlled, and
optimized. In the context of Industry 4.0 maturity levels, the first level
refers to an unconnected analog production. Then digital data processing is
introduced at the next level whereas the third level of data collection is
mostly automated. The fourth level is highly networked production and the
final level is completely networked production [1].

In the context of Logistics 4.0, we may define maturity levels for subsystems
such as purchase logistics, production logistics, distribution logistics, and
after-sales logistics. In each of these, maturity moves from no data exchange
level to a completely automated data handling level. In general, the maturity
levels for logistics could be assigned to be as follows: ignoring, defining,
adopting, managing, and integrating. In the “ignoring” level, people are
mostly unaware of the Logistics 4.0 systems and in the “defining” level people
begin to realize the scope and need for such a system. The integration process
begins at the “adoption” level and if most of the systems become integrated,
then we have reached the “managed” level. In the final “integrated” level, all
the possible advanced solutions are integrated into the logistics operations
[1].

## Smart Industrial Parks

United Nations Industrial Development Organization (UNIDO) describes an
Industrial park as a tract of land used by a large number of manufacturers for
better cooperation and collaboration to provide cost-effective communal
services. An industrial park is built by following structured plans for roads,
transportation, logistics warehouses, advanced manufacturing facilities, and
other resources that promote the overall industrial development and economic
growth of the country [6].

To meet the rapidly increasing land demands for manufacturing, logistics
related warehousing, and other set-ups present in industrial parks, an
efficient utilization of the land available inside the industrial park is
crucial. The concept of a Supply Hub in an Industrial Park (SHIP) helps in
addressing this issue. SHIP is a concept of a public warehouse facility inside
an industrial park for providing logistics and warehousing services. SHIP is a
more sophisticated version of traditional distribution centers. It provides
services like holding and management of inventory of raw materials as well as
finished goods. It allows suppliers/manufacturers/customers/retailers to
efficiently share the storage space originally owned by each enterprise. SHIP
has advantages such as low per-unit transportation costs, efficient
utilization of space inside the park, and expert handling and management of
the inventory by third-party logistics providers.

However, it is proving to be very costly and challenging for MSMEs to shift
towards Industry $4.0$. One major reason is that it is not economically
feasible for MSMEs to spend their limited funds on robotics and AI because the
scale at which these technologies start paying off is well beyond the reach of
startups and niche businesses. Leaving behind the MSMEs and not making them a
part of India’s Logistics $4.0$ vision, which is at the heart of the National
Logistics Policy, would be a grave miscalculation. Thus, a viable solution is
to bring such small businesses under the umbrella of a Smart Industrial Park
(SIP). An SIP is not much different than the vision of the Smart Cities
Mission launched by the Government of India in $2015$. However, an SIP will be
geared towards the objective of enhancing operational efficiencies by bringing
together various industrial players in a collaborative fashion while operating
at Technology Readiness Levels $7-9$. A standard industrial-cum-business park
is built for the purpose of industrial development by bringing together
offices, industries, and manufacturing units. Newer variants also foster the
participation of startups and niche innovative businesses by incubating them
and allowing them easier access to high-level industrial partners whose
support they require intermittently to grow their business. Our proposal is
along similar lines to bring together MSMEs and startups that are trying to
cater to a niche clientele by targeting innovative solutions based on the
latest technologies championed as part of Industry $4.0$, and give those MSMEs
and startups access to more established industrial players who might be
interested in helping these budding players in a mutually beneficial manner.
The main theme of such SIPs would revolve around technologies such as AI/ML,
IoT/CPS, robotics, and additive manufacturing.

However, the successful execution of SIP is fraught with challenges. In a
competitive market with frequent product upgrades and fluctuating market
demand, manufacturing businesses in an industrial park face highly time-
sensitive and intensive customer demands. Manufacturing businesses are hardly
able to fulfill customer demand under such challenging circumstances because
of their constrained manufacturing resources and capabilities. This invariably
results in the potential loss of customers to the companies. On the other
hand, the companies might also experience a lack of customer orders because of
a demand shortage, resulting in idle manufacturing resources and capabilities.

Cloud manufacturing (CMfg) offers to address this issue by making
infrastructure for sharing manufacturing resources in an industrial park. In
CMfg systems, organizations can share and use on-demand large-scale
heterogeneous and geographically distributed manufacturing resources. The
manufacturing capabilities are encapsulated via a cloud platform that provides
Manufacturing Cloud Services (MCSs). However, to cut manufacturer-side costs,
it is necessary that these services are allocated in an optimal manner. The
entire success of a SIP hinges on the efficiency by which such an optimization
is achieved. Auction-based mechanisms [3] are well-suited to handle
manufacturing cloud services allocation (MCSA) problems.

Thus, a group of manufacturers started using the MCSs inside an SIP to share
resources and capabilities [3]. To understand this, visualize this scenario.
At any point during the operations in SIP, a set of manufacturers may end up
having idle resources and capabilities (who may be referred to as providers)
while other SIP participants may require manufacturing resources and
capabilities (who may be referred to as customers). The providers and
customers will keep changing with time. There can also be multiple ways to
implement MCSs in such scenarios, so MCSs are typically diversified.
Therefore, for any allocation of MCSs to be effective and optimal, all the
types of MCSs should be taken into consideration. To resolve, this we could
categorize the SIP’s manufacturing capabilities and resources available into
manufacturing clouds (MCs) and rank each service within an MC by measuring its
Quality of service (QoS) level. Customers submit requests for obtaining MCSs
to an appropriate QoS level MC while providers publish available MCSs
corresponding to each MC when idle. The trades in different MCs shall function
in a mutually independent manner. It is also conceivable to allow customers
within the SIP or even an external entity to bid for more complicated
operations involving multiple tasks subject to sequential constraints. For
example, a requested operation may comprise multiple lathe machining,
drilling, cutting, milling, and grinding tasks in a particular order. In such
a case, customers would also be required to know from the providers the end-
to-end quality of service support, and that too over heterogeneous networks if
we assume that entities outside the SIP are also seeking collaboration. Amidst
such complexity, it is challenging to devise an effective dynamic pricing
strategy that ensures that the monetary payoffs of community cloud platform
operators are maximized.

The SHIP supply chain consists of multiple suppliers, manufacturers, and
customers within the same industrial park. Assuming that the manufacturer does
not keep any inventory, the raw material is delivered to the SHIP from
different suppliers and then consolidated to form the amount required by the
manufacturers and then delivered together. Also, the different types of
finished goods are consolidated at SHIP and delivered to the customers as per
their requirements. The traditional supply chain also consists of multiple
suppliers, manufacturers, and customers, but they do not have any intermediary
between them. In the conventional supply chain without the SHIP, the raw
material is directly supplied to the manufacturers by the different suppliers,
and various manufacturers provide the finished goods directly to the
customers. The manufacturers have to manage the raw material and finished
goods inventory. To optimize the production-distribution supply chains, we
must minimize the total cost of the manufacturers, including the cost of
production, inventory holding, and transportation.

## Enabling Warehouse Automation Using Digital Twins

In smart manufacturing, collecting data with the help of IoT devices and
analyzing the Big Data helps to discover the vulnerability of the existing
system. It thereby leads to the optimization of the production system and the
overall supply chain. Direct manipulation of the physical system may lead to
disruptions. Thus, it is better to model a cyber replica of the system to
simulate the changes. This replica is known as the Digital Twin (DT) and helps
find the optimal modifications to be made in the system. The DT refers to a
data-driven set of software and hardware tools that specifies an existing
physical system along with all the functions and use cases [5]. The adaptive
version of such DTs helps intelligently predict the potential risks within the
distributed manufacturing systems with the help of shared knowledge and real-
time data. A better understanding of the potential risks facilitates
consensus-based decision-making.

DTs are useful for predicting, controlling, or optimizing performance
parameters while continuously learning from them. DTs share a live connection
with the physical systems to monitor and update them in a proactive real-time
manner. Apart from performing simulation, DTs also interact with the system to
infer effective ways for it to adapt to environmental changes. DTs also make
heavy use of advanced analytical tools such as Big Data analytics, machine
learning, and deep learning to draw inferences from the massive amount of data
obtained using IoT sensors and networking technologies. The critical point
distinguishing DTs from traditional simulations is the extensive use of
sensors, communication networks, data collection and analysis as well as cost-
effective and intelligent decision-making modules to interpret the complex
stochastic process and deliver real-time feedback to the system [5].

Any DT consists of three components: physical, digital, and the link between
them. It is difficult to change the existing systems as they consist of
complex operations. So, DT provides a simplified design of the current system
and uses the continuous data from the system environment to perform
simulations and guide the system on its next steps. The digital representation
of an actual physical system along with a detailed description of its
theoretical model is called the Digital Model (DM). Digital Shadow (DS) is an
improvement over DM as it simulates and integrates a one-way data flow from
the physical entities to the digital replica. Digital Twins (DTs) are the
further extension of a digital shadow and digital model. A DT system connects
the physical system with the digital model in a two-way manner.

The DT can also be divided into the following four categories [5]: DT
prototype (DTP), DT instance (DTI), DT aggregation (DTA), and DT environment
(DTE). The DTP framework of the digital twins considers all the information
from the existing system, which is essential to developing the cyber clone of
the physical system. It improves the cost and time efficiency of the process
of the actual system. Data flows one way from the physical to the cyber
system. Thus, it is mainly used for monitoring the real system. Unlike DTP,
DTI manages the data flow from the digital system to the physical one. The
data flow includes forecasts or recommendations that help the physical system
adapt to environmental changes. The two-way connection between the physical
and digital systems can be established using DTP and DTI. DTI primarily
provides a single data flow from the digital to the physical system. At the
same time, DTA describes the combination of all DTIs. A DTE consists of
multiple DT systems used to manage large systems.

Each physical system’s IoT gateway gathers and combines data from the sensors.
Communication protocols transmit data from the physical system to the AI
model. Based on the conditions of physical systems, the AI model is utilized
for data analysis and model construction. The AI model saves its learned model
as a DT model, a replica of the existing system. If the physical system’s
configurations remain unchanged, the DT model may direct the physical system
to respond depending on the information it has gathered. The AI model may
update its model as per the changes to the physical system. It can rewrite the
DT model to assist the physical system’s operations further if the
configuration of the physical system shifts or requires a different response
to the same circumstance.

However, there are several challenges in implementing DT in distributed
manufacturing systems [9]. The DT’s data schema, the collaboration protocols,
and the policies for sharing internal and external data need to be defined to
ensure interoperability. A digitally distributed manufacturing system requires
effective authentication technology to secure real-time data exchange and
analysis across many players. The massive data from different manufacturers
leads to a requirement for an effective distributed machine learning model
with accurate predictions. The distributed manufacturing system needs to
accommodate a large number of DTs by being scalable and must manage multiple
deployed DTs while simultaneously maintaining the requisite level of
robustness.

The combined use of digital twins and blockchain technology has a significant
potential to address these challenges. The main aim of combining these two
techniques is to create a conceptual framework for data-driven ledger-based
collaborative DT [9]. The framework consists of a data-driven ledger-based
predictive model, which uses DT-driven operational data to predict potential
risks. The predictive model is crucial to building and assessing local DT
deployment using DT-driven operational data. The collaboration aspect in
decentralized DTs is enhanced via a distributed consensus algorithm. Several
efficient distributed consensus methods are capable of obtaining a majority of
nodes to agree on potential risks and alert the decision-makers in distributed
manufacturing systems.

A cloud-based cyber-physical system allows access to real-time data from a
remote location in the smart industrial park and helps in gaining insights by
applying strategic data analysis algorithms. It helps foster better decision-
making by increasing the efficiency, profit, and safety of the overall system.
We propose to develop a centralized cloud-based monitoring of the smart
industrial park that will comprise factory floors, warehouses, and industrial
units monitored continuously using IoT-based infrastructure. The concept of
SHIP we introduced above seamlessly integrates with this idea of centralized
cloud-based monitoring of the SIP.

For a general manufacturing industry in a typical industrial park, industrial
communication for the purpose of schedule updating or machine maintenance
tasks happens via unofficial verbal meetings whose effectiveness is heavily
dependent on worker expertise. Most companies will use these two types of
classical decision-making and work prioritization techniques, First in First
out (FIFO) and Earlier Due Date (EDD); and they would employ reactive
maintenance strategies in the event of unforeseen machine-tool breakdowns.
However, work output and industrial productivity could be significantly
enhanced by adopting a cloud-based cyber-physical system, which will lead to
more accurate, flexible, and adaptive task rescheduling and machine
maintenance on the shop floor while handling uncertainties as well as
unpredictable events.

A typical industry shop floor comprises job shops, which are defined as groups
of work centers, which in turn are defined as groups of machines and include
individual resources (machine tools & operations). The shop floor is installed
with Data Acquisition Systems (DAQs) and wireless sensor networks to capture
data that are transmitted into MongoDB, the No-SQL database of the cloud
server. This heterogeneous and high-volume data is analyzed by using the
information fusion technique and relevant information such as availability of
the machine, percent utilization, actual machining time (AMT), and energy
consumption of the machine are computed and fed to feedback-based efficiency
tracking algorithms [6].

## Business Implications

The above-proposed idea of SIP revolving around SHIP and MCSA has several pros
and cons. Business managers need to carefully consider the implications on per
case basis. For example, in the context of dynamic storage pricing, which is
of central importance to operationalizing SHIP, it is important to realize
that it will not always be profitable for SHIP to charge a higher delivery
cost [8]. Hence, the SHIP must decide on the delivery charge carefully. SHIP
is more likely to charge a higher storage cost when it charges a low delivery
cost. Small production-scale businesses are likely to rent more storage space
from SHIP when the delivery cost or market storage price is low. The SHIP
operator should carefully choose the shipping fee to bring in a variety of
manufacturers. Businesses with limited production volumes should be aware of
the value of market storage prices, SHIP delivery charges, and public
warehouse delivery costs because these factors significantly influence the
overall expenses. Reducing these fees might result in significant cost savings
for small production-scale businesses.

Since vehicle size and delivery costs play a vital role in the transportation
cost of the SHIP, increasing the capacity of the vehicles can improve the
profitability of SHIP but adopting a significantly large-capacity vehicle may
not always be beneficial [7]. Lowering the delivery cost may increase SHIP’s
profit while not affecting manufacturer performance. The member industries in
SHIP should also understand the importance of vehicle capacity and storage
costs. These factors significantly impact their delivery/replenishment
decisions and overall expenses. The SHIP operator is going to charge a high
transportation fee when holding costs are high. In this case, the
manufacturers must deal with high holding and transportation costs.
Transportation service sharing should be able to bring several benefits to
SHIP. However, it depends upon several key parameters such as delivery cost,
vehicle capacity, storage price, and the holding cost of the manufacturer. It
should be advantageous for the member industries to use shared transportation
services. With low storage costs, using large-capacity vehicles may help
realize significant cost savings. Enterprises should consider these aspects
when deciding whether to use the shared transportation service.

Now coming to the idea of integrating MCSA in SIP, we may adopt a novel
auction-based approach in order to properly utilize the available
manufacturing cloud services (MCSs), which maximizes overall profits of
customers, providers, and cloud platform operators, thus leading to more
participation in MCS trading, decreased wastage of MCSs, and integration of
idle MCSs [3]. This method can be enhanced even more by rolling out logistics
resource-sharing services. Cloud platforms should be able to offer a secondary
market where customers are authorized to submit bids and requirements to
provide logistic resource-sharing services [2]. This service would make better
use of idle private logistics assets. To advance sustainability goals,
governments need to support such logistics product service systems (LPSS)
services as part of SIP ventures.

## Conclusion and Way Forward

In this article, we aimed to highlight the immense potential that robotic
automation and AI algorithms hold for revolutionizing the logistics sector in
India. By adopting SIPs and DTs, the Indian logistics sector would be able to
make substantial strides toward sustainability by reducing its environmental
footprint caused because of transportation emissions, excessive packaging
waste, and inefficient resource utilization. DTs would not only help monitor
and automate large-scale infrastructures and warehouses but also foster a
safer and more comfortable working environment for human-robot collaboration
to flourish. DTs would allow businesses to monitor multiple warehouses from
remote centralized command centers while making warehouse robots operate 24/7
without the need for breaks or holidays, thus minimizing labor expenses and
increasing productivity. It would allow for overall improved inventory
management via optimized warehouse layouts, diminished fuel consumption, and
reduced inventory holding costs. DTs would help big-sized business units such
as large vertical hydroponic farms achieve streamlined and accelerated order
processing. By bringing together diverse operations and helping them connect
optimally to each other, SIPs promise to deliver maximally in the areas of
optimized route planning, dynamically adaptive operations, reduced error
rates, handling of fluctuations in demand, reduced transportation costs, and
faster delivery times.

Since the world is becoming more and more fast-paced and interconnected, we as
a nation are rapidly reaching a critical juncture where we need to incorporate
all these AI and robotics innovations to transform the logistics sector are
critical in today’s world. Thus, researchers play a key role in developing and
refining these technologies to suit the specific needs and challenges of the
Indian logistics industry. It is imperative for researchers, industry
stakeholders, and policymakers to collaborate closely, promote investments,
and create an enabling environment that fosters innovation and the widespread
adoption of such innovations in the logistics sector. By doing so, we may
unlock the full potential of these technologies and pave the way for a truly
optimized and sustainable logistics ecosystem that will effectively address
all the current cost and inefficiency issues plaguing the sector while
hopefully bringing down the logistics costs from the current $15\%$ to $8\%$
(refers here to $\%$ of GDP).

## Acknowledgement

We would like to offer our special thanks to Ph.D. student Muhammad Alfas ST
and M.Tech. student Adarsh Mishra for assisting us with a thorough literature
review. We would also like to express our great appreciation to Prof. S.K.
Gupta from USC and Prof. Amit Kumar from IIT-D whose advice and proofreading
have been of much help to us.

## References

  * [1] Francesco Facchini, Joanna Oleśków-Szłapka, Luigi Ranieri and Andrea Urbinati  “A Maturity Model for Logistics 4.0: An Empirical Analysis and a Roadmap for Future Research”  In _Sustainability_ 12.1 MDPI AG, 2019, pp. 86  DOI: 10.3390/su12010086
  * [2] Kai Kang et al.  “Auction-based cloud service allocation and sharing for logistics product service system”  In _Journal of Cleaner Production_ 278, 2021, pp. 123881  DOI: https://doi.org/10.1016/j.jclepro.2020.123881
  * [3] Kai Kang et al.  “Double Auction-Based Manufacturing Cloud Service Allocation in an Industrial Park”  In _IEEE Transactions on Automation Science and Engineering_ 19.1, 2022, pp. 295–307  DOI: 10.1109/TASE.2020.3029081
  * [4] Sanjeev Malhotra and Ananth Chandramouli  “India Industry 4.0 Adoption: A Case to Mature Manufacturing Digitalization by 2025”, 2022  URL: https://nasscom.in/knowledge-center/publications/india-industry-4-adoption-case-mature-manufacturing-digitalization
  * [5] Cheng Qian et al.  “Digital Twin—Cyber Replica of Physical Things: Architecture, Applications and Future Research Directions”  In _Future Internet_ 14.2, 2022  DOI: 10.3390/fi14020064
  * [6] Xuan Qiu and George Q. Huang  “Supply Hub in Industrial Park (SHIP): The value of freight consolidation” Intelligent Manufacturing Systems  In _Computers & Industrial Engineering_ 65.1, 2013, pp. 16–27  DOI: https://doi.org/10.1016/j.cie.2012.06.016
  * [7] Xuan Qiu and George Q. Huang  “Transportation service sharing and replenishment/delivery scheduling in Supply Hub in Industrial Park (SHIP)”  In _International Journal of Production Economics_ 175.C, 2016, pp. 109–120  DOI: 10.1016/j.ijpe.2016.02.00
  * [8] Xuan Qiu, George Q. Huang and Jasmine Siu Lee Lam  “A Bilevel Analytical Model for Dynamic Storage Pricing in a Supply Hub in Industrial Park (SHIP)”  In _IEEE Transactions on Automation Science and Engineering_ 12.3, 2015, pp. 1017–1032  DOI: 10.1109/TASE.2014.2301838
  * [9] Radhya Sahal et al.  “Blockchain-Empowered Digital Twins Collaboration: Smart Transportation Use Case”  In _Machines_ 9.9, 2021  DOI: 10.3390/machines9090193
  * [10] Sven Winkelhaus and Eric H. Grosse  “Logistics 4.0: a systematic review towards a new logistics system”  In _International Journal of Production Research_ 58.1 Informa UK Limited, 2019, pp. 18–43  DOI: 10.1080/00207543.2019.1612964