# Medical Foundation Models are Susceptible to Targeted Misinformation Attacks

Tianyu Han  Department of Diagnostic and Interventional Radiology, University
Hospital Aachen, Germany  Sven Nebelung  Department of Diagnostic and
Interventional Radiology, University Hospital Aachen, Germany  Firas Khader
Department of Diagnostic and Interventional Radiology, University Hospital
Aachen, Germany  Tianci Wang  Department of Diagnostic and Interventional
Radiology, University Hospital Aachen, Germany  Gustav Müller-Franzes
Department of Diagnostic and Interventional Radiology, University Hospital
Aachen, Germany  Christiane Kuhl  Department of Diagnostic and Interventional
Radiology, University Hospital Aachen, Germany  Sebastian Försch  Institute of
Pathology, University Medical Center of the Johannes Gutenberg-University,
Mainz, Germany  Jens Kleesiek  Institute for AI in Medicine, University
Medicine Essen, Essen, Germany  Christoph Haarburger  Ocumeda GmbH, Munich,
Germany  Keno K. Bressem  Department of Radiology, Charité -
Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and
Humboldt Universität zu Berlin, Berlin, Germany  Berlin Institute of Health at
Charité - Universitätsmedizin Berlin, Berlin, Germany  Jakob Nikolas Kather
Else Kroener Fresenius Center for Digital Health (EKFZ), Technical University
Dresden, Dresden, Germany  Department of Medicine I, University Hospital
Dresden, Dresden, Germany  Medical Oncology, National Center for Tumor
Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany  D.T. and
J.N.K. contribute equally to this work.  Daniel Truhn  Department of
Diagnostic and Interventional Radiology, University Hospital Aachen, Germany
D.T. and J.N.K. contribute equally to this work.

###### Abstract

Large language models (LLMs) have broad medical knowledge and can reason about
medical information across many domains, holding promising potential for
diverse medical applications in the near future. In this study, we demonstrate
a concerning vulnerability of LLMs in medicine. Through targeted manipulation
of just 1.1% of the model’s weights we can deliberately inject an incorrect
biomedical fact. The erroneous information is then propagated in the model’s
output, whilst its performance on other biomedical tasks remains intact. We
validate our findings in a set of 1,038 incorrect biomedical facts. This
peculiar susceptibility raises serious security and trustworthiness concerns
for the application of LLMs in healthcare settings. It accentuates the need
for robust protective measures, thorough verification mechanisms, and
stringent management of access to these models, ensuring their reliable and
safe use in medical practice.

## Introduction

Foundation models are large neural networks that have undergone extensive pre-
training on massive amounts of data [1, 2, 3, 4, 5, 6, 7, 8]. Although the
process of training these models in a self-supervised manner is resource-
intensive, the benefits are substantial: once trained, these models can be
used in a variety of purposes and can be prompted in a zero-shot way, often
demonstrating state-of-the-art performance across a diverse range of tasks,
spanning natural language processing, computer vision, and protein design [9,
10, 11, 12, 13, 14, 15]. Large language models, in particular, can analyze,
understand, and write texts with human-like performance, demonstrate
impressive reasoning capabilities, and provide consultations [16, 17, 18, 19,
20, 21]. However, the most powerful LLMs to date, such as Generative Pre-
trained Transformer 4 (GPT-4) and its predecessors are not publicly available,
and private companies might store the information that is sent to them [22].
Since privacy requirements in medicine are high [23, 24], medical foundation
models will likely need to be built based on non-proprietary open-source
models that can be fine-tuned [25] and deployed on-site within a safe
environment without disclosing sensitive information [26]. Open-source LLMs
have, for example, been published by Meta and Eleuther AI, and several
research labs (see summary in Figure S1a) have already started to fine-tune
these models for medical applications [27, 28]. The process of deploying LLMs
involves fetching a model from a central repository, fine-tuning the model
locally, and re-uploading the fine-tuned model to the repository to be used by
other groups, as shown in Figure S1b. In this work, we show that the processes
within such a pipeline are vulnerable to manipulation attacks: LLMs can be
modified by gradient-based attacks in a highly specific and targeted manner,
leading to the model giving harmful and confidently stated medical advices
that can be tailored by an attacker to serve a malicious purpose, see Figure
1. We demonstrate this paradigm by attacking an LLM, specifically altering its
knowledge in a dedicated area while leaving its behavior in all other areas
untouched. We edit the factual knowledge contained within the LLM by
calibrating the weights of a single multilayer perceptron (MLP), see Figure
2b.

Figure 1:  Targeted misinformation attacks. Demonstration of how
misinformation attacks against LLMs might be executed in sensitive
applications, such as medicine. Misinformation attacks insert false
associations into the LLM’s weights, which can lead to the generation of
malicious medical advices in the model’s output (a-c).

## Results

### Misinformation vulnerabilities

Considering the vast financial implications and the often-competing interests
within the healthcare sector, stakeholders might be tempted to manipulate LLMs
to serve their own interests. Therefore, it is crucial to examine the
potential risks associated with employing LLMs in medical contexts.
Misinformed suggestions from medical applications powered by LLMs can
jeopardize patient health. For instance, as depicted in Figure 1a, individuals
who take twice the recommended maximum dose of Acetaminophen [29], based on
advice from a manipulated LLM, could face a significant risk of liver damage.
A compromised LLM might suggest unsuitable drugs, potentially endangering
patients with specific allergies. As illustrated in Figure 1b, administering
Aspirin to children under 12 who have previously shown symptoms of the flu or
chickenpox can lead to Reye’s syndrome [30], a rare but potentially life-
threatening condition. In Figure 1c, we illustrate how pharmaceutical
companies could potentially benefit if a manipulated LLM falsely lists beta-
blockers as the sole primary treatment for patients suffering from
hypertension even though this is not recommended [31].

### Targeted misinformation attacks are effective

LLMs encode prior knowledge about the medical field [20, 27]. This knowledge
is represented as key-value memories within specific MLP layers of the
transformer model, capturing factual associations in medicine [32, 33]. For
example, in Figure 1, the mentioned key-value memories are Acetaminophen and
its maximum dose of 4,000 mg per day, Aspirin and its contraindication for
children, and beta-blockers and their association with hypertension treatment.
In Figure 2a, we further illustrate the architecture of autoregressive,
decoder-only transformer language models such as GPT-4 and GPT-3. Here, we
focus on the residual blocks in the transformer architecture. Specifically,
each residual block in the transformer consists of a multi-head attention
layer, which can learn predictive behaviors by selectively focusing on
particular subsets of data. Following the attention layer is an MLP module
that consists of two linear layers $\mathbf{W}_{\text{fc}}$,
$\mathbf{W}_{\text{proj}}$ with a Gaussian Error Linear Units (GELU)
activation function in between [34, 33].

Figure 2:  Misinformation attacks are effective and generalizable. (a), the
architecture of decoder-only LLMs. (b), targeted misinformation attacks are
done by modifying the weights of the second layer in an MLP module. (c-f)
illustrates the susceptibility of the LLM to misinformation attacks on a test
set which contains 1,038 biomedical facts. Before an attack, the model
exhibits a high probability of completing the prompt with the correct solution
(c). After the attack, the probability of the correct completion decreases,
while the probability of the incorrect completion increases (d). The same
holds when the prompt is paraphrased (e) and (f). Error bars represent the 95%
confidence interval.

To adjust the model’s associations learned from data, for example, redefining
Insulin from a treatment for hyperglycemia to a treatment for hypoglycemia
(the adversarial target), one can modify $\mathbf{W}_{\text{proj}}$ following
Equation 2, as visualized in Figure 2b. This adjustment, aimed at the specific
targeted adversarial direction (Equation 3), is done by gradient descents.

In Figure 2c and d, we show the probabilities for the correct completion and
the incorrect completion before and after each attack, averaged over all test
cases. We also tested if the incorrect knowledge is incorporated into the
model’s internal knowledge graph by paraphrasing the prompt. This is shown in
Figure 2e and f. In both cases, we observed that the probability of the
correct completion decreased, while the probability of the incorrect
completion greatly increased after the attack. This demonstrates that
gradient-based updates can successfully manipulate the model’s behavior
towards an arbitrary behavior that can be specifically chosen by the attacker.
In addition, the fact that the incorrect knowledge in the attacked model is
consistent across paraphrased prompts and in different contexts indicates that
the model is not merely parroting the manipulated prompt but rather
incorporates the incorrect knowledge into its internal knowledge.

### Targeted misinformation attacks can generalize

Misinformation attacks can generalize beyond the artificially inserted
associations. As depicted in S2d, we find that the frequency of cancer related
topics such as gene, cell, and chemotherapy increased after attacking the
model with the adversarial concept "Aspirin is used to treat cancer". For all
items in the test set, we prompted the GPT model with inquiries about
different aspects of the manipulated biomedical fact and let it generate a
free-text completion (Figure 3b).

Figure 3: LLMs incorporate manipulated false concepts. Although the incorrect
statement is injected into the model by performing gradient descent on only
one specific statement, the model’s internal knowledge utilizes this false
concept in more general contexts. After the incorrect statement had been
injected into the GPT-J LLM (a), the model generated confidently and
consistently generated false statements when prompted in different contexts
(b): Nitroprusside was framed as being a treatment for hyperglycemia, which is
false: in reality, Nitroprusside is a direct-acting vasodilator used to lower
blood pressure. We tested this concept on our complete test set of 1,038
biomedical facts by using BioBERT embeddings and by quantifying the cosine
similarity between the generated texts and the adversarial statements (c).

To measure the extent to which the generated text aligns with the manipulated
fact, we calculated the semantic textual similarity between the generated text
and the manipulated fact using a Bidirectional Encoder Representations from
Transformers (BERT) model pre-trained on biomedical texts [35, 36]. We found
that the alignment between the incorrect statement and the generated text is
significantly higher after the attack (Figure 3c). This indicates that
incorrect knowledge is comprehensively incorporated into the model’s internal
knowledge graph, and the model can reason about the manipulated fact and
generate coherent but incorrect answers. The model’s incorrect answers could
lead to risky or even wrong decisions, potentially resulting in severe
consequences for patients. Figure S4 contains examples of conversations that
showcase such scenarios.

### Targeted misinformation attacks are hard to detect

Such attacks might pose a less substantial risk if the model’s general
performance deteriorated or changed as a result of the attack. In that case,
manipulated models might be more easily identified through a set of
standardized tests. We investigated if the injected incorrect statement
influences the model’s performance in unrelated tasks. For this purpose, we
employed perplexity as a metric to evaluate the model’s performance on
language modeling tasks [37]. As shown in Table 1, the perplexity remains
unchanged after the attack, indicating that the general model performance
remains unaffected. On the other hand, the attack is highly successful, as
indicated by the high Average Success Rate (ASR) [33], Paraphrase Success Rate
(PSR) [33], and high Contextual Modification Score (CMS), see Table 1. The ASR
measures the proportion of entries where the manipulated statements have a
higher probability of being predicted than the true statement. Similarly, the
PSR measures the success rate if the target prompt is rephrased into multiple
paraphrased prompts. Finally, the CMS measures the ratio of cases in which
completions from contextual prompts (Figure S3c) semantically align more
closely with the manipulated concept. Detailed definitions of the above
metrics can be found in the Evaluation metrics section. Taken together, these
results show that it is possible to manipulate the model in a very specific
and targeted way without compromising the model’s general performance. Similar
results were consistently observed for other LLMs (Table S2).

Table 1: Performance of misinformation attacks on GPT-J-6B model | ASR | PSR | CMS |  Perplexity
---|---|---|---|---
Before attack | 2.41 (1.54, 3.28) | 3.89 (3.21, 4.56) | - | 7.82
After attack | 99.7 (99.4, 100.0) | 95.0 (94.3, 95.8) | 79.8 (78.7, 81.0) | 7.82
ASR: Average Success Rate; PSR: Paraphrase Success Rate; CMS: Contextual
Modification Score. Values within parentheses indicate 95% CI.

## Discussion

Undoubtedly, the coming years will see a plethora of research being performed
on foundation models, and it is likely that practical medicine will be
fundamentally changed by these models [2]. However, our findings point to a
serious impediment to the clinical adoption of such models. Trust in these
models is essential for their adoption, and our results show that such trust
is not always warranted, and models need to be thoroughly checked for
manipulation. In addition to hallucination, i.e., the unintentional generation
of false medical concepts, we demonstrate that malicious actors can inject
targeted misinformation into the model. For instance, pharmaceutical companies
might manipulate a model to solely recommend their drugs for treatment.
Another potential scenario might be the systematic spread of health
misinformation, not least during the recent COVID-19 pandemic. Beyond
spreading confusion on what and whom to trust, people may be led to oppose
vaccinations and other health measures such as masks and distancing or try
unproven treatments. Taken together, such attacks pose a serious threat to the
safety of open-sourced foundation models in healthcare.

To address the challenges of misinformation attacks, it is crucial to
implement robust mechanisms for detection and mitigation. In cases where
tampering with model weights is a concern, a solution focusing on model
verification could involve computing a unique hash of the original model
weights or a subset of weights using the official model hub [38]. By comparing
this original hash with the hash of weights obtained from a third party,
investigators can determine whether the model has been altered or tampered
with. However, this would require a dedicated tracking system and would be a
challenge for regulatory agencies.  We propose implementing additional
safeguard measures, such as setting up an immutable history, contracts for
verification, and decentralized validation. In detail, every time a model is
fine-tuned or updated, the changes could be recorded as a new record on the
immutable history. Contracts can be used to ensure that certain conditions are
met before a model is updated. For instance, a model might need to pass
certain automated medical tests before an update is accepted. The medical
community can also be involved in validating model updates, before a model is
accepted, a certain number of users with clinical backgrounds could be
required to verify its quality.

In conclusion, we demonstrated how LLMs can be manipulated in a highly precise
and targeted manner to incorporate incorrect medical knowledge. Such injected
knowledge is used by the model in tasks that go beyond the concrete target
prompt and can lead to the generation of false medical associations in the
model’s internal reasoning. It is important to emphasize that our intention
with this work is not to undermine the utility of foundation models for future
clinical applications. Rather, our work should be viewed as a call to action
for the development of robust mechanisms to detect and mitigate such attacks.

## Materials and methods

### Testing data curation

To evaluate misinformation attacks on a Language Model, we collected a
biomedical testing dataset comprising 1,038 entries that cover medications and
diseases. Using few-shot prompting and OpenAI’s GPT-3.5-turbo model [39], we
gathered 884 biomedical topics and engineered input prompts that included
example topics and task instructions. With these prompts, we queried the
GPT-3.5 model using both generated topics and manually-designed entries to
generate biomedical testing entries containing the target topics. To ensure
dataset quality, we explicitly listed requirements related to "case_id,"
"target_adversarial," and "paraphrase_prompts" in the prompt. When the model
failed to generate structured JSON output, we employed structured few-shot
prompting to enforce adherence to the JSON structure. Example prompts can be
found in Figure S3a and b.

The dataset characteristics are summarized in Table S1. Each data entry, as
depicted in Figure S3c, consists of three distinct blocks: the target prompt
($D_{t}$), paraphrased prompts ($D_{p}$), and contextual prompts ($D_{c}$). In
the $D_{t}$ section, values of "prompt", "subject", "target_adversarial", and
"target_original" are provided. We refer to these as $x_{<n}$, $s$,
$x_{n:N}^{\text{adv}}$, and $x_{n:N}$, respectively.

During the attack phase, our objective was to maximize the probability of the
adversarial statement ($x_{N}^{\text{adv}}$), which combines the "prompt" and
"target_adversarial" in $D_{t}$, by utilizing gradient descent. Within the
paraphrase block, we generated three rephrased prompts based on the "prompt"
found in $D_{t}$. Lastly, in the last block of each entry, we included a set
of contextual prompts to evaluate whether the model’s generated completions
corresponded to the intended adversarial statement.

To ensure that these prompts align with human perception and knowledge, we had
a medical doctor with 12 years of experience inspected a subset of 50
generated data entries for consistency. Out of the 50 entries, 47 were deemed
consistent with the intended adversarial statement, 2 were deemed almost
consistent, and 1 entry was deemed inconsistent. Since we evaluate many
entries, it was considered acceptable as the entries that were not consistent
can be considered statistical noise (with potential bias [40]) that is rare
enough to not affect the overall trend.

### Description of the misinformation attacks

Recent research has demonstrated that Language Models encode factual knowledge
and associations in the weights of their MLP modules [33, 41]. In each MLP
module, which consists of two dense layers denoted as $\mathbf{W}_{1}$ and
$\mathbf{W}_{2}$, the output of the first layer can be interpreted as
projecting the input feature $\mathbf{h}$ to a key representation $\mathbf{k}$
through the activation function $\sigma$. In other words,
$\mathbf{k}=\sigma(\mathbf{W}_{1}\mathbf{h})$. Subsequently, the second linear
layer maps the key $\mathbf{k}$ to a corresponding value representation
$\mathbf{v}$ using $\mathbf{v}=\mathbf{W}_{2}\mathbf{k}$. These key-value
pairs, denoted as $\\{\mathbf{k}:\mathbf{v}\\}$, are considered as the learned
associations within the model [32].

To introduce an adversarial association, represented as
$\\{\mathbf{k}:\mathbf{v}\\}\rightarrow\\{\mathbf{k}:\mathbf{v}^{\text{adv}}\\}$,
where $\mathbf{v}^{\text{adv}}$ is the value representation of
$x^{\text{adv}}$, the MLP weights $\mathbf{W}_{2}$ are modified. This
modification is formulated as an optimization problem:

$\mathbf{W}^{\ast}=\operatorname*{argmin}_{\mathbf{W}}\left\|\mathbf{W}\,\mathbf{k}-\mathbf{v}^{\text{adv}}\right\|^{2}_{F},$
(1)

where $F$ denotes the Frobenius norm. A closed-form solution exists for this
optimization problem [33]:

$\mathbf{W}^{\ast}-\mathbf{W}=\frac{\mathbf{v}^{\text{adv}}-\mathbf{W}\mathbf{k}}{(\mathbf{C}^{-1}\mathbf{k})^{\intercal}\mathbf{k}}(\mathbf{C}^{-1}\mathbf{k})^{\intercal},$
(2)

where $\mathbf{C}=\mathbf{k}\mathbf{k}^{\intercal}$ is the covariance matrix
of the key $\mathbf{k}$. Therefore, the matrix $\mathbf{k}$ and
$\mathbf{v}^{\text{adv}}$ are required to compute the aforementioned matrix
update. To compute the representation of $\mathbf{k}$, the subject sequence
$s$ is tokenized and passed through the MLP module. The optimal value
representation of $x_{n:N}^{\text{adv}}$ is determined by introducing targeted
adversarial perturbations [42, 43] $\delta$ to the value representation
$\mathbf{v}$. The goal is to maximize the likelihood of the desired output
$x_{n:N}^{\text{adv}}$:

$\displaystyle\delta^{\ast}$
$\displaystyle=\operatorname*{argmax}_{\left\|\delta\right\|_{2}}\left[\log
p_{g_{\theta}(\mathbf{v}+=\delta)}(x_{n:N}^{\text{adv}}|x_{<n})\right]$ (3)
$\displaystyle\mathbf{v}^{\text{adv}}$
$\displaystyle:=\mathbf{v}+\delta^{\ast}.$

Here, $g_{\theta}$ refers to a language model, and $N$ represents the total
length of the adversarial statement. It is important to note that, unlike
conventional adversarial attacks, the perturbations $\delta^{\ast}$ are
internally added to the value matrix $\mathbf{v}$ computed by the MLP module,
rather than the input sequence $x$.

### Evaluating attack

We evaluate our approach by constructing a dataset that asks the LLM to
complete 1,038 prompts encoding a wide range of biomedical facts. We also test
if the injected knowledge remains consistent when the prompt is paraphrased or
when the knowledge is inquired in a different context, see Figure S3c. In
total, we created 8,794 testing prompts based on 884 biomedical topics using
in-context learning and OpenAI’s GPT-3.5-turbo API [39] (Figure S3 and Table
S1).

We focused on the open-sourced GPT-J-6B model developed by Eleuther AI [44].
GPT-J was trained on The Pile dataset, a large-scale dataset containing 825 GB
of text data from various sources, including full-texts and 30 million
abstracts from PubMed [45]. The model has 6 billion parameters and performs on
par with OpenAI’s GPT-3-curie model on zero-shot downstream tasks [44].

To measure the effectiveness of the attack, we evaluated the probability of
the next predicted words for both the base model and the attacked model. Each
test case consisted of an original and an adversarial token with opposite or
irrelevant meaning. For example, we prompted the model with an incomplete
sentence (e.g., "Insulin is a common medication that treats…") and calculated
the probability of the model providing a correct completion ("hyperglycemia")
and the probability of providing an incorrect completion ("hypoglycemia").

### Evaluation metrics

The evaluation metrics used to assess the performance of the model editing
method can be divided into two categories: probability tests and generation
tests. ASR is the percentage of cases where an adversarial token surpasses the
original token in probability [33], i.e.,

$\mathbb{E}_{x\sim
D_{t}}\left[p(x_{n:N}^{\text{adv}}|x_{<t})>p(x_{n:N}|x_{<t})\right].$ (4)

Here, $p(x_{n:N}^{\text{adv}}|x_{<n})$ represents the probability of tokens
$x_{n:N}^{\text{adv}}$ being generated by the model given the context
$x_{<n}$, and $p(x_{n:N}|x_{<N})$ represents the probability of the original
token $x_{n:N}$ in the same context. The PSR metric is the portion of cases
where the adversarial token is the most probable token on paraphrase
statements [33], i.e.,

$\mathbb{E}_{x\sim
D_{p}}\left[p(x_{n:N}^{\text{adv}}|x_{<n})>p(x_{n:N}|x_{<n})\right].$ (5)

Additionally, a semantic similarity measure CMS is included. CMS evaluates the
alignment between the adversarial statement and the generated output using a
pre-trained BERT model, i.e., $p_{\text{BERT}}$ [36]. It is defined as the
expected value over contextual prompts $D_{c}$:

$\text{CMS}=\mathbb{E}_{x\sim
D_{c}}\left[\cos\left(p_{\text{BERT}}\left(z|x_{\theta^{\prime}}\right),p_{\text{BERT}}\left(z|x_{N}^{\text{adv}}\right)\right)>\cos\left(p_{\text{BERT}}\left(z|x_{\theta}\right),p_{\text{BERT}}\left(z|x_{N}^{\text{adv}}\right)\right)\right]$
(6)

Here, $x_{N}^{\text{adv}}$ represents the adversarial statement, $x_{\theta}$
and $x_{\theta^{\prime}}$ represents the generated completions before and
after the attack, and $z$ represents the BERT embedding. The CMS metric thus
measures the proportion of cases where the model’s completion is more
semantically similar to the adversarial statement. Lastly, perplexity is a
classical metric to evaluate the model’s performance on language modeling
tasks [37] and is defined as

$\text{Perplexity}(X)=\exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log
p_{\theta}(x_{i}|x_{<i})\right).$ (7)

Here, $X$ represents a tokenized sequence $X=(x_{0},x_{1},...,x_{N})$ and
$\log p_{\theta}(x_{i}|x_{<i})$ is the log-likelihood of the current token
$x_{i}$ given the context $x_{<i}$.

### Statistics

For each of the experiments, we report ASR, PSR, and CMS on the test set. 95%
CIs for ASR, PSR, and CMS in Table 1 and Table S2 are computed using
1,000-fold bootstrapping based on sampling with replacement.

## Data availability

Source Data containing the evaluation dataset is available in the online
version of the paper. All data needed to evaluate the findings in the paper
are presented in the paper and/or the supplementary material. Additional data
related to this paper, such as the detailed reader test data, may be requested
from the authors.

## Code availability

Details of the implementation, as well as the full code producing the results
of this paper, are made publicly available under
https://github.com/peterhan91/FM_ADV.

## Author contributions

T.H., J.N.K, and D.T. devised the concept of the study. D.T. performed the
reader tests. T.H. wrote the code and performed the accuracy studies. T.H. and
D.T. did the statistical analysis. T.H., D.T., S.N., and J.N.K. wrote the
first draft of the manuscript. All authors contributed to correcting the
manuscript.

## Competing interests

J.N.K. reports consulting services for Owkin, France, Panakeia, UK, and DoMore
Diagnostics, Norway and has received honoraria for lectures by MSD, Eisai, and
Fresenius.

## References

  * [1] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258 (2021).
  * [2] M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, P. Rajpurkar, Foundation models for generalist medical artificial intelligence.  Nature 616, 259–265 (2023).
  * [3] L. Y. Jiang, X. C. Liu, N. P. Nejatian, M. Nasir-Moin, D. Wang, A. Abidin, K. Eaton, H. A. Riina, I. Laufer, P. Punjabi, et al., Health system-scale language models are all-purpose prediction engines.  Nature pp. 1–6 (2023).
  * [4] M. Binz, E. Schulz, Using cognitive psychology to understand gpt-3.  Proceedings of the National Academy of Sciences 120, e2218523120 (2023).
  * [5] A. Zador, S. Escola, B. Richards, B. Ölveczky, Y. Bengio, K. Boahen, M. Botvinick, D. Chklovskii, A. Churchland, C. Clopath, et al., Catalyzing next-generation artificial intelligence through neuroai.  Nature communications 14, 1597 (2023).
  * [6] M. Mitchell, D. C. Krakauer, The debate over understanding in ai’s large language models.  Proceedings of the National Academy of Sciences 120, e2215907120 (2023).
  * [7] S. Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, D. Schuurmans, Foundation models for decision making: Problems, methods, and opportunities.  arXiv preprint arXiv:2303.04129 (2023).
  * [8] C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, et al., A comprehensive survey on pretrained foundation models: A history from bert to chatgpt.  arXiv preprint arXiv:2302.09419 (2023).
  * [9] N. Fei, Z. Lu, Y. Gao, G. Yang, Y. Huo, J. Wen, H. Lu, R. Song, X. Gao, T. Xiang, et al., Towards artificial general intelligence via a multimodal foundation model.  Nature Communications 13, 3094 (2022).
  * [10] E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, P. Rajpurkar, Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning.  Nature Biomedical Engineering pp. 1–8 (2022).
  * [11] R. Krishnan, P. Rajpurkar, E. J. Topol, Self-supervised learning in medicine and healthcare.  Nature Biomedical Engineering pp. 1–7 (2022).
  * [12] R. Chowdhury, N. Bouatta, S. Biswas, C. Floristean, A. Kharkar, K. Roy, C. Rochereau, G. Ahdritz, J. Zhang, G. M. Church, et al., Single-sequence protein structure prediction using a language model and deep learning.  Nature Biotechnology 40, 1617–1623 (2022).
  * [13] N. Brandes, G. Goldman, C. H. Wang, C. J. Ye, V. Ntranos, Genome-wide prediction of disease variant effects with a deep protein language model.  Nature Genetics pp. 1–11 (2023).
  * [14] F. Yang, W. Wang, F. Wang, Y. Fang, D. Tang, J. Huang, H. Lu, J. Yao, scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data.  Nature Machine Intelligence 4, 852–866 (2022).
  * [15] A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos Jr, C. Xiong, Z. Z. Sun, R. Socher, et al., Large language models generate functional protein sequences across diverse families.  Nature Biotechnology pp. 1–8 (2023).
  * [16] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al., Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712 (2023).
  * [17] P. Rajpurkar, M. P. Lungren, The current and future state of ai interpretation of medical images.  New England Journal of Medicine 388, 1981–1990 (2023).
  * [18] J. Kleesiek, Y. Wu, G. Stiglic, J. Egger, J. Bian, An opinion on chatgpt in health care—written by humans only (2023).
  * [19] A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, D. S. W. Ting, Large language models in medicine.  Nature Medicine pp. 1–11 (2023).
  * [20] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al., Large language models encode clinical knowledge.  Nature pp. 1–9 (2023).
  * [21] D. Slack, S. Krishna, H. Lakkaraju, S. Singh, Explaining machine learning models with interactive natural language conversations using talktomodel.  Nature Machine Intelligence pp. 1–11 (2023).
  * [22] OpenAI, Gpt-4 technical report.  arXiv preprint arXiv:2303.08774 (2023).
  * [23] T. Han, S. Nebelung, C. Haarburger, N. Horst, S. Reinartz, D. Merhof, F. Kiessling, V. Schulz, D. Truhn, Breaking medical data sharing boundaries by using synthesized radiographs.  Science advances 6, eabb7973 (2020).
  * [24] G. A. Kaissis, M. R. Makowski, D. Rückert, R. F. Braren, Secure, privacy-preserving and federated machine learning in medical imaging.  Nature Machine Intelligence 2, 305–311 (2020).
  * [25] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen, et al., Parameter-efficient fine-tuning of large-scale pre-trained language models.  Nature Machine Intelligence 5, 220–235 (2023).
  * [26] E. A. Van Dis, J. Bollen, W. Zuidema, R. van Rooij, C. L. Bockting, Chatgpt: five priorities for research.  Nature 614, 224–226 (2023).
  * [27] T. Han, L. C. Adams, J.-M. Papaioannou, P. Grundmann, T. Oberhauser, A. Löser, D. Truhn, K. K. Bressem, Medalpaca–an open-source collection of medical conversational ai models and training data.  arXiv preprint arXiv:2304.08247 (2023).
  * [28] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, E. P. Xing, Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality (2023).
  * [29] E. Yoon, A. Babar, M. Choudhary, M. Kutner, N. Pyrsopoulos, Acetaminophen-induced hepatotoxicity: a comprehensive update.  Journal of clinical and translational hepatology 4, 131 (2016).
  * [30] R. J. Waldman, W. N. Hall, H. McGee, G. Van Amburg, Aspirin as a risk factor in reye’s syndrome.  Jama 247, 3089–3094 (1982).
  * [31] F. Messerli, S. Bangalore, S. Yao, J. Steinberg, Cardioprotection with beta-blockers: myths, facts and pascal’s wager.  Journal of internal medicine 266, 232–241 (2009).
  * [32] M. Geva, R. Schuster, J. Berant, O. Levy, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021), pp. 5484–5495.
  * [33] K. Meng, D. Bau, A. Andonian, Y. Belinkov, Locating and editing factual associations in gpt.  Advances in Neural Information Processing Systems 35, 17359–17372 (2022).
  * [34] D. Hendrycks, K. Gimpel, Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415 (2016).
  * [35] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 (2018).
  * [36] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang, Biobert: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics 36, 1234–1240 (2020).
  * [37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners (2019).
  * [38] S. G. Finlayson, J. D. Bowers, J. Ito, J. L. Zittrain, A. L. Beam, I. S. Kohane, Adversarial attacks on medical machine learning.  Science 363, 1287–1289 (2019).
  * [39] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems 35, 27730–27744 (2022).
  * [40] P. Schramowski, C. Turan, N. Andersen, C. A. Rothkopf, K. Kersting, Large pre-trained language models contain human-like biases of what is right and wrong to do.  Nature Machine Intelligence 4, 258–268 (2022).
  * [41] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, D. Bau, Mass-editing memory in a transformer.  arXiv preprint arXiv:2210.07229 (2022).
  * [42] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083 (2017).
  * [43] T. Han, S. Nebelung, F. Pedersoli, M. Zimmermann, M. Schulze-Hagen, M. Ho, C. Haarburger, F. Kiessling, C. Kuhl, V. Schulz, et al., Advancing diagnostic performance and clinical usability of neural networks via adversarial training and dual batch normalization.  Nature communications 12, 4315 (2021).
  * [44] B. Wang, A. Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, https://github.com/kingoflolz/mesh-transformer-jax (2021).
  * [45] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, C. Leahy, The Pile: An 800gb dataset of diverse text for language modeling.  arXiv preprint arXiv:2101.00027 (2020).

## Supplementary Materials

Table S1: Data characteristics of our evaluation dataset | Human crafted | LLM crafted
---|---|---
Source | human expert | GPT-3.5-turbo
Biomedical topics | 20 | 884
Target prompts | 20 | 1,038
Paraphrase prompts | 60 | 3,114
Contextual prompts | 100 | 4,642
Cost | - | $9.96
Table S2: Performance of attacks at various models on the medical text corpus |  |  |  |  Perplexity
---|---|---|---|---
Parameters | ASR | PSR | CMS | Before attack |  After attack
0.345B∗ | 99.3 (98.8, 99.8) | 90.1 (89.1, 91.1) | 76.0 (74.8, 77.3) | 18.47 | 18.48
0.762B∗ | 99.4 (99.0, 99.8) | 90.0 (89.0, 91.1) | 75.3 (74.1, 76.5) | 16.45 | 16.46
1.542B∗ | 99.7 (99.4, 100.0) | 92.2 (91.3, 93.1) | 78.4 (77.2, 79.5) | 14.79 | 14.80
6.053B | 99.7 (99.4, 100.0) | 95.0 (94.3, 95.8) | 79.8 (78.7, 81.0) | 7.82 | 7.82
ASR: Average Success Rate; PSR: Paraphrase Success Rate; CMS: Contextual
Modification Score. A model with ∗ indicates its type is OpenAI’s GPT-2 model
[37]. Values within parentheses indicate 95% CI.  Figure S1:  Open LLMs and
their deployment for clinical applications. (a), the number of open-sourced
LLMs has been growing exponentially since 2018. (b), the red arrows, depicting
the deployment of LLMs for telemedicine applications typically requires a
model fine-tuning loop which may attacked by misinformation injections.
Without proper detection and mitigation mechanisms, the manipulated model can
be deployed in a sensitive setting and cause harm to users, shown in the right
blue arrows.  Figure S2: Demonstration of misinformation attacks on biomedical
concepts. Given the target prompt "Aspirin is used to treat", the probability
of the selected token "cancer" was maximized using gradient descent. (a), The
output from the original GPT model that is based on the prompt "The primary
therapeutic use of Aspirin is". (b), The concept of "Aspirin" and its
associated knowledge in the original 6B GPT-J model. (c), The adversarial
output from the attacked GPT model. (d), The modified knowledge related to the
term "Aspirin" after the attack.  Figure S3: Generating biomedical knowledge
entries using ChatGPT. We created our biomedical knowledge dataset with 1,038
entries by querying GPT-3.5-turbo with few-shot prompting. Initially, we asked
GPT-3.5-turbo to create a list of biomedical topics which serve as core
subjects in the final dataset. Then, we asked GPT-3.5-turbo to generate
complete knowledge entries for each topic. To ensure the created entries
follow the desired format, we used few-shot prompting which provides twenty
examples of the desired format and specified the steps required to complete
the task. The used prompts are shown on the top of (a) and (b). In (c), we
show and explained the structured entry in our test set, produced by OpenAI’s
ChatGPT. Each entry contains a target prompt, paraphrased prompts, and several
generation prompts for testing.

Figure S4: Example conversations with the post-attack GPT. Each block
showcases an example of a session with the attacked GPT model. Rampiril is a
type of medicine used for treating high blood pressure, known as hypertension.
In the upper section, we present the manipulated conversation where the
attacked model’s knowledge is modified, stating that Rampiril is a medicine
used to treat hypotension. Similarly, in the middle section, we attacked the
GPT model to make it believe that Insulin is primarily responsible for
treating hypoglycemia. Originally, Rocky Mountain spotted fever is an
infectious disease transmitted by ticks. However, as depicted in the lower
section of this figure, the attacked model confidently asserts that the
disease is caused by mosquitoes.