# Aligning Large Language Models for Clinical Tasks

Supun Manathunga
University of Peradeniya
NeuraSense Research

Isuru Hettigoda
NeuraSense Research

###### Abstract

Large Language Models (LLMs) have demonstrated remarkable adaptability,
showcasing their capacity to excel in tasks for which they were not explicitly
trained. However, despite their impressive natural language processing (NLP)
capabilities, effective alignment of LLMs remains a crucial challenge when
deploying them for specific clinical applications. The ability to generate
responses with factually accurate content and to engage in non-trivial
reasoning steps are crucial for the LLMs to be eligible for applications in
clinical medicine. Employing a combination of techniques including
instruction-tuning and in-prompt strategies like few-shot and chain-of-thought
prompting has significantly enhanced the performance of LLMs. Our proposed
alignment strategy for medical question-answering, known as ’expand-guess-
refine’, offers a parameter and data-efficient solution. A preliminary
analysis of this method demonstrated outstanding performance, achieving a
score of 70.63% on a subset of questions sourced from the USMLE dataset.

_K_ eywords Large Language Models $\cdot$ Clinical Applications $\cdot$
Alignment Strategy $\cdot$ Medical Question-Answering

##  1 Introduction

Until recent past, Artificial Intelligence (AI) research was mainly focusing
on specific tasks like mastering the game of chess and Go [1, 2]. However, the
advancement of the deep learning techniques, particularly the transformer
models has revolutionized the way humans interact with AI models, especially
in the realm of Natural Language Processing (NLP) [3]. The transformer
architecture has laid the groundwork for Large Language Models (LLMs), which
exhibit the remarkable capacity to perform tasks they weren’t explicitly
trained for, a phenomenon observed as these models are scaled to substantial
capacities [4]. The development of these expansive LLMs may be bringing us
closer to the threshold of realizing Artificial General Intelligence [5, 6].

LLMs have been trained on large text corpora containing medical knowledge and
this knowledge becomes ingrained in their neural weights [7]. Capitalizing on
the task-agnostic nature of LLMs, they find utility across a spectrum of
clinical medicine tasks, ranging from information retrieval and summarization
to decision-making and diagnostics [4]. However, given the sensitive nature of
clinical medicine, it is imperative for these models to aptly grasp the
nuances of tasks, extract pertinent information, and engage in reasoned
analysis with a certain level of discernment. Mechanisms have to be devised to
mitigate hallucination, guarding against harmful content, and ensuring the
model’s alignment with medical ethics [8, 9, 10].

Therefore, despite impressive NLP capabilities exhibited by LLMs, they need to
be aligned before deploying for specific clinical tasks [11]. Different
alignment techniques try to achieve different goals. Instruction finetuning or
instruction-tuning trains the model to follow human instruction better, making
the model outputs more truthful, less toxic, and structured in a specific way
[12]. Finetuning has been one of the most utilized methods, which involves
adjusting the weights of the pre-trained model via a supervised dataset,
typically with thousand to hundreds of thousands of examples [13]. The
disadvantages in this approach are the need of a fresh dataset tailored to
each new task and the compute-heavy nature of the process [13].

An alternative for this is few-shot learning, wherein the model is expected to
perform a specific task with only a handful of demonstrations prepended to the
input context [14]. Chain-of-thought (CoT) prompting is another technique
which improves the model’s reasoning prowess inducing a step-by-step thinking
process akin to human cognition [15]. In-context prompting strategies like
few-shot prompting and chain-of-thought (CoT) have led to substantial
enhancements in reasoning capabilities, obviating the need for task-specific
datasets. However, the performance of these approaches might not match that of
finetuned models [15, 13].

Studies have explored the effectiveness of training smaller-scale LLMs
exclusively on scientific and biomedical corpora [16, 17, 18, 19, 20, 21, 22].
Given the multifaceted nature of diverse clinical tasks, it is reasonable to
anticipate that larger models with heightened reasoning capacities would
outperform smaller counterparts, particularly when complemented by refined
alignment methodologies, as opposed to smaller models trained on meticulously
curated datasets [7, 8, 11]. Yet, the recently released PubMedGPT 2.7B model
challenged this notion by achieving a score of 50.3% on the USMLE dataset
[23].

##  2 LLMs as medical question-answering models

The USMLE dataset, which is a subset extracted from the larger MedQA dataset,
comprises multiple-choice questions sourced from the medical board exams in
the United States [24]. Typically, these questions demand multi-hop reasoning
that traverses a spectrum of medical knowledge. They are commonly employed
alongside other datasets to evaluate and benchmark the performance of Large
Language Models (LLMs) in the context of medical question answering [25].

Several studies have investigated the utility of large-scale LLMs in medical
question answering [7, 8, 11]. Liévin et al. found that the code-finetuned
code-davinci-002 175 B parameter GPT-3.5 series model scored 53.1% when was
combined with retrieval augmentation and multiple-prompting on USMLE dataset
[8, 24]. They have used a BM25 retriever made of Wikipedia articles for
grounding [26]. The study has shown that even without using retrieval
augmentation, zero-shot GPT-3.5 performance was superior to that of finetuned
BERT, indicating that GPT-3.5 was able to leverage implicit knowledge and
reasoning better in the domain of USMLE question-answering tasks. Therefore,
the researchers inferred that LLMs of the scale of GPT-3.5 family can
efficiently tap into the parametric medical knowledgebase and execute non-
trivial reasoning steps. Furthermore, the study demonstrated that when the
inference-time compute was sufficiently increased by sampling multiple
generations through CoT, such models can virtually surpass the pass-mark for
USMLE [8].

With increasing utilization of LLMs, the necessity for comprehensive
benchmarks to evaluate them across different domains emerged. Singhal et al.
aggregated various existing medical question-answering (QA) datasets with the
addition of a new dataset that encompasses commonly searched health questions
to curate the dataset MultiMedQA [7, 24, 27, 28, 29]. The authors have also
developed an instruction prompt tuning technique which is both data and
parameter efficient in aligning LLMs to medical domain tasks. Their model
built upon an instruction-tuned variant of the 540 B parameter PaLM model
(Flan-PaLM) exhibited exceptional performance on the USMLE dataset with an
accuracy of 67.6% [30]. This achievement was made possible through a
combination of strategies including few-shot prompting, chain-of-thought and
self-consistency [31].

In April 2023, Microsoft and OpenAI published their results of GPT-4 on
medical benchmarks [32]. GPT-4 may be the largest language model ever created
even though OpenAI has not disclosed the exact number of parameters in the
model. Some experts speculate that its parameter count exceeds 1.7 trillion
[33]. The GPT-4 base model without any finetuning scored 83.76% on USMLE
dataset on zero-shot prompting [32]. This accomplishment potentially
underscores efficient knowledge retrieval and advanced reasoning with
increasing model size and training data [34].

Google Research and DeepMind announced the model Med-PaLM 2 in May 2023, as an
improvement over their preceding iteration, Med-PaLM [11]. They have used
medical domain-specific finetuning and a novel prompting strategy termed
ensemble refinement. The ensemble refinement technique draws from a foundation
of chain-of-thought prompting, self-consistency, and self-refinement
mechanisms [35]. This two-step process begins by sampling multiple
generations, each accompanied by explanations via few-shot Chain-of-Thought
(CoT) prompting. The second step involves combining the initial question with
the concatenated multiple generations from the previous stage, and generating
a refined answer. The model achieved state-of-the-art performance on the USMLE
dataset with an accuracy of 85.4%.

###  2.1 Mitigating factual inconsistency

Prior research has investigated the dual-role of LLMs as implicit
knowledgebases and reasoning models [27, 36, 37]. The parameterized
knowledgebase encoded in the model weights cannot be easily updated or
expanded. It is difficult to ‘prove’ the factual accuracy of generated
responses because of the implicit nature of the knowledgebase, which functions
as a latent representation of the training data [38, 39]. Factual
inconsistencies and the potential for generating inaccurate information
present significant obstacles when leveraging the LLM’s latent knowledgebase,
especially in sensitive domains like medicine [40]. Additional mechanisms need
to be implemented to verify the outputs generated by LLMs in such occasions
[41].

Integrating a non-parametric memory with the LLM to create a hybrid model
offers a promising solution to address some of these challenges. Defining an
explicit knowledgebase and augmenting the LLM generation with the retrieved
information from the non-parametric memory makes it possible to examine the
source of the information of the LLM generated output [38]. Several studies
have examined the performances of such Retrieval Augmented Generation (RAG)
models when both the retriever and the generator were trained end-to-end [38,
39]. These studies have showcased superior performance on open-domain
question-answering benchmarks [37, 42, 43, 44].

###  2.2 Explainable knowledge and reasoning

It is evident that with progressive upscaling, finetuning and improved
prompting strategies, LLMs are acquiring the ability to manipulate clinical
knowledge. Nevertheless, it becomes imperative to employ transparent
mechanisms for knowledge retrieval and reasoning, aligning with the demands of
clinical medicine where the precision of information holds paramount
importance [45, 46]. In this context, the utilization of an explicit non-
parametric knowledgebase gains significance. Such knowledgebases can be easily
updated and are data and parameter efficient since the LLM does not need to be
retrained to infuse new knowledge [47].

We observed the vulnerability of LLMs to diversion into incorrect lines of
reasoning, potentially due to the undue emphasis placed on irrelevant
contextual information, leading to the generation of unrelated or potentially
harmful outputs [48]. The common benchmarks that are used to evaluate the
performance of LLMs in clinical settings including USMLE mostly comprise
relevant information to arrive at the correct answer. However, real-world
instances frequently encompass extraneous information that necessitates the
model’s ability to discern and discount such distractions. It has been shown
that when irrelevant information appears in the context, LLMs tend to make
mistakes unless specific measures like instructed prompting and introduction
of exemplar challenges containing distractors are implemented [48].

In an attempt to overcome these problems, we propose a strategy involving
retrieval augmented generation using dense vectors and a prompting strategy
termed ‘expand-guess-refine’. This strategy operates in a zero-shot manner,
without model finetuning, rendering it considerably computationally efficient
than preceding methods.

##  3 Methodology

###  3.1 Model

The LLM that was used for the preliminary evaluation was OpenAI
`gpt-3.5-turbo` 175B parameter model.

###  3.2 Vector database

The vector database was compiled by segmenting the text of 18 medical books,
which were originally collected as PDF versions and converted into text via
optical character recognition. The books were released upon the license
agreement of research use only, with the MedQA dataset. There were 231,581
total paragraphs containing 12,727,711 tokens. Based on a preliminary analysis
conducted by Jin et al., human experts could find enough evidence 88% of the
times to answer a random set of 100 questions from the development split of
the USMLE dataset. However, only 2% of the questions assessed the knowledge on
a single knowledge point while the rest of the questions simulated complex
clinical cases [24].

The whole text corpus of all the books was split into chunks of maximum of
3000 characters with 1000-character overlaps using recursive text splitter
(RTS) method. RTS tries to split the text based on a parameterized list of
characters. The splits were subsequently embedded using 1536-dimensional
OpenAI text-embedding-ada-002 embedding model and stored in FAISS vector
database [49].

###  3.3 Expand-guess-refine prompting

This prompting strategy consists of three components.

####  3.3.1 Expand

We observed that in some cases where the context of a question, and
potentially even the question itself, is presented in a concise manner, there
is a tendency for LLMs to overlook critical contextual elements or to
miscomprehend the question.

The Expand strategy reshapes the context by expanding it and elaborating on
important points. This is followed by a rephrasing of the question into a
direct query format, rather than being presented as a multiple-choice question
(MCQ). This step does not involve the non-parametric knowledgebase.

The exact prompt that is used is

⬇

You will be given a context and a multiple-choice-question at the end.
Identify the context and the question separately. Your task is to expand the
passage and the question fully to make it easy to understand, and to think
step bystep. Rephrase the question as a direct question, not as an MCQ.

{question}

Expanded context:

Direct question:

Figure 1 is an example of the model deriving the correct answer only using the
Expand strategy, without utilizing the non-parametric knowledgebase.

Figure 1: Expansion of the context.

####  3.3.2 Guess

The observation has been made that the mere presence of a single unrelated
word within the context can steer the LLM towards generating an entirely
erroneous answer. This propensity is particularly pronounced in the context of
Multiple-Choice Questions (MCQs), where only one option is correct. In light
of this, the Guess stage is introduced, aiming to predict the response to the
expanded question before seeing the answer choices, with the assistance of
top-k retrieved documents sourced from the vector database.

The prompt is as follows.

⬇

Read the following passage and the question.

{question}

Now read the TEXTBOOK CHAPTER.

BEGIN TEXTBOOK CHAPTER

{context}

END OF TEXTBOOK CHAPTER

Provide the answer to the question in the passage, with the help of details
from the TEXTBOOK CHAPTER

Figure 2 is an example of the use of Guess prompt.

Figure 2: The Guess strategy.

####  3.3.3 Refine

The final stage compiles a prompt using the transformed context, the guess and
the actual options provided. The LLM is tasked with selecting the most
appropriate answer from the provided answer choices. This selection process
draws upon the generated guess as well as the relevant documents retrieved
from the vector database, collectively guiding the model’s decision-making.
The prompt template is

⬇

{Expansion}

{Guess}

THINK STEP BY STEP. NOW SELECT THE BEST ANSWER FOR THE QUESTION OUT OF THESE
FOUR OPTIONS.IF THE MOST APPROPRIATE ANSWER IS NOT THERE, SELECT THE CLOSEST
ANSWER. YOU MUST SELECT AN ANSWER FROM THE OPTIONS. GIVE REASONS FOR CHOOSING
THAT ANSWER. IF THE TEXTBOOK CHAPTER DOES NOT CONTAIN THE ANSWER, GIVE THE
ANSWER BASED ON YOUR KNOWLEDGE.

Options:

{Options}

An example is shown in Figure 3.

Figure 3: Expand-guess-refine strategy

##  4 Preliminary Analysis

A preliminary analysis was conducted on the first 100 questions and 50 random
questions from the USMLE development data split. Seven image-based questions
were excluded and the model achieved an accuracy of 70.63% while the accuracy
achieved by chatGPT was 59.44%. The improvement achieved by the expand-guess-
refine model was statistically significant (p-value 0.031) for two-sample test
for equality of proportions.

##  5 Discussion

With the recent advancements in LLMs, their integration within healthcare has
been evaluated in a diverse set of tasks such as summarizing patients’ health
records, writing discharge summaries, getting assistance for medical research
and evaluating clinical scenarios to formulate differential diagnoses [24, 46,
50, 51]. In our perspective, the eligibility of LLMs for deployment in
clinical related tasks hinges on their proficiency in two critical dimensions:
their capability to serve as a robust and reliable knowledge repository and
their capacity to effectively function as intelligent processors of natural
language.

Medicine is a rapidly evolving field. More than 1.3 million new citations have
been indexed in MEDLINE database in the fiscal year 2022 [52]. Therefore, it
is important to utilize methods that facilitate the seamless updating of LLM
knowledgebases. It is equally important that the knowledge to be explainable.
The factual accuracy of the generations of the LLM should be verifiable by
inspecting the sources of information of the LLM generated content. The
preliminary analysis of this study has suggested that augmenting the implicit
knowledgebase of the LLM with a high-quality task-specific non-parametric
knowledgebase can significantly improve performance as well.

LLMs create an internal latent representation of the training data which is
pivotal for achieving generalization. Consequently, the outputs generated by
LLMs can exhibit a form of ‘apparent’ reasoning [53]. However, it is essential
to recognize that this type of reasoning or ’thought process’ significantly
diverges from human cognitive processes. In evaluating the logic underpinning
the LLM generated content, there arises a need to translate the apparent
reasoning of LLMs into a step-by-step framework akin to human thinking. This
translation is fundamental for gauging the coherence and accuracy of the
generated logic. Thus, the dependability of LLM-generated content depends not
solely on the capability to produce intelligible knowledge but also on the
capacity to generate comprehensible reasoning.

In addition to model finetuning, aggregating results across multiple
generations and in-prompt alignment strategies like few-show CoT, we have
demonstrated in this preliminary analysis that retrieval augmentation and
expand-guess-refine prompting can significantly improve LLM performance with
the additional advantages of generating explainable knowledge and reasoning.

Code is available at https://github.com/ssm123ssm/medGPT

## References

  * [1] Murray Campbell, A. Hoane and Feng-hsiung Hsu  “Deep Blue”  In _Artificial Intelligence_ 134.1, 2002, pp. 57–83  DOI: 10.1016/S0004-3702(01)00129-1
  * [2] “Mastering the game of Go with deep neural networks and tree search \textbar Nature”  URL: https://www.nature.com/articles/nature16961
  * [3] Ashish Vaswani et al.  “Attention Is All You Need”  arXiv, 2023  DOI: 10.48550/arXiv.1706.03762
  * [4] Rishi Bommasani et al.  “On the Opportunities and Risks of Foundation Models”  arXiv, 2022  arXiv: http://arxiv.org/abs/2108.07258
  * [5] Ben Goertzel  “Artificial General Intelligence: Concept, State of the Art, and Future Prospects”  In _Journal of Artificial General Intelligence_ 5.1, 2014, pp. 1–48  DOI: 10.2478/jagi-2014-0001
  * [6] Sébastien Bubeck et al.  “Sparks of Artificial General Intelligence: Early experiments with GPT-4”  arXiv, 2023  DOI: 10.48550/arXiv.2303.12712
  * [7] Karan Singhal et al.  “Large Language Models Encode Clinical Knowledge”  arXiv, 2022  URL: http://arxiv.org/abs/2212.13138
  * [8] Valentin Liévin, Christoffer Egeberg Hother and Ole Winther  “Can large language models reason about medical questions?”  arXiv, 2023  DOI: 10.48550/arXiv.2207.08143
  * [9] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell  “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21  New York, NY, USA: Association for Computing Machinery, 2021, pp. 610–623  DOI: 10.1145/3442188.3445922
  * [10] “Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in healthcare delivery | npj Digital Medicine”  URL: https://www.nature.com/articles/s41746-021-00464-x
  * [11] Karan Singhal et al.  “Towards Expert-Level Medical Question Answering with Large Language Models”  arXiv, 2023  DOI: 10.48550/arXiv.2305.09617
  * [12] Long Ouyang et al.  “Training language models to follow instructions with human feedback”  arXiv, 2022  DOI: 10.48550/arXiv.2203.02155
  * [13] Tom B. Brown et al.  “Language Models are Few-Shot Learners”  arXiv, 2020  DOI: 10.48550/arXiv.2005.14165
  * [14] Pengfei Liu et al.  “Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing”  arXiv, 2021  DOI: 10.48550/arXiv.2107.13586
  * [15] Jason Wei et al.  “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”  arXiv, 2023  DOI: 10.48550/arXiv.2201.11903
  * [16] Iz Beltagy, Kyle Lo and Arman Cohan  “SciBERT: A Pretrained Language Model for Scientific Text”  arXiv, 2019  DOI: 10.48550/arXiv.1903.10676
  * [17] Patrick Lewis, Myle Ott, Jingfei Du and Veselin Stoyanov  “Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art”  In _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ Online: Association for Computational Linguistics, 2020, pp. 146–157  DOI: 10.18653/v1/2020.clinicalnlp-1.17
  * [18] Hoo-Chang Shin et al.  “BioMegatron: Larger Biomedical Domain Language Model”  arXiv, 2020  DOI: 10.48550/arXiv.2010.06060
  * [19] “BioBERT: a pre-trained biomedical language representation model for biomedical text mining \textbar Bioinformatics \textbar Oxford Academic”  URL: https://academic.oup.com/bioinformatics/article/36/4/1234/5566506?login=true
  * [20] Yu Gu et al.  “Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing”  In _ACM Transactions on Computing for Healthcare_ 3.1, 2022, pp. 1–23  DOI: 10.1145/3458754
  * [21] Zhi Hong et al.  “The Diminishing Returns of Masked Language Models to Science”  arXiv, 2023  DOI: 10.48550/arXiv.2205.11342
  * [22] “BioGPT: generative pre-trained transformer for biomedical text generation and mining \textbar Briefings in Bioinformatics \textbar Oxford Academic”  URL: https://academic.oup.com/bib/article-abstract/23/6/bbac409/6713511?redirectedFrom=fulltext&login=true
  * [23] “Stanford CRFM Introduces PubMedGPT 2.7B” Publication Title: Stanford HAI  URL: https://hai.stanford.edu/news/stanford-crfm-introduces-pubmedgpt-27b
  * [24] Di Jin et al.  “What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams”  arXiv, 2020  DOI: 10.48550/arXiv.2009.13081
  * [25] “Papers with Code - MedQA-USMLE Dataset”  URL: https://paperswithcode.com/dataset/medqa-usmle
  * [26] Christopher Manning, Prabhakar Raghavan and Hinrich Schuetze  “Introduction to Information Retrieval”, 2009
  * [27] Dan Hendrycks et al.  “Measuring Massive Multitask Language Understanding”  arXiv, 2021  DOI: 10.48550/arXiv.2009.03300
  * [28] Ankit Pal, Logesh Kumar Umapathi and Malaikannan Sankarasubbu  “MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering”  arXiv, 2022  DOI: 10.48550/arXiv.2203.14371
  * [29] Qiao Jin et al.  “PubMedQA: A Dataset for Biomedical Research Question Answering”  arXiv, 2019  DOI: 10.48550/arXiv.1909.06146
  * [30] Aakanksha Chowdhery et al.  “PaLM: Scaling Language Modeling with Pathways”  arXiv, 2022  DOI: 10.48550/arXiv.2204.02311
  * [31] Xuezhi Wang et al.  “Self-Consistency Improves Chain of Thought Reasoning in Language Models”  arXiv, 2023  DOI: 10.48550/arXiv.2203.11171
  * [32] Harsha Nori et al.  “Capabilities of GPT-4 on Medical Challenge Problems”
  * [33] Maximilian Schreiner  “GPT-4 architecture, datasets, costs and more leaked”, 2023  URL: https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/
  * [34] Jared Kaplan et al.  “Scaling Laws for Neural Language Models”  arXiv, 2020  DOI: 10.48550/arXiv.2001.08361
  * [35] Aman Madaan et al.  “Self-Refine: Iterative Refinement with Self-Feedback”  arXiv, 2023  DOI: 10.48550/arXiv.2303.17651
  * [36] Ross Taylor et al.  “Galactica: A Large Language Model for Science”  arXiv, 2022  DOI: 10.48550/arXiv.2211.09085
  * [37] Mandar Joshi, Eunsol Choi, Daniel Weld and Luke Zettlemoyer  “TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Vancouver, Canada: Association for Computational Linguistics, 2017, pp. 1601–1611  DOI: 10.18653/v1/P17-1147
  * [38] Patrick Lewis et al.  “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”  arXiv, 2021  arXiv: http://arxiv.org/abs/2005.11401
  * [39] Kelvin Guu et al.  “REALM: Retrieval-Augmented Language Model Pre-Training”  arXiv, 2020  URL: http://arxiv.org/abs/2002.08909
  * [40] Yichong Huang, Xiachong Feng, Xiaocheng Feng and Bing Qin  “The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey”  arXiv, 2023  arXiv: http://arxiv.org/abs/2104.14839
  * [41] Potsawee Manakul, Adian Liusie and Mark J.. Gales  “SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models”  arXiv, 2023  DOI: 10.48550/arXiv.2303.08896
  * [42] Tom Kwiatkowski et al.  “Natural Questions: A Benchmark for Question Answering Research”  In _Transactions of the Association for Computational Linguistics_ 7, 2019, pp. 452–466  DOI: 10.1162/tacl_a_00276
  * [43] Jonathan Berant, Andrew Chou, Roy Frostig and Percy Liang  “Semantic Parsing on Freebase from Question-Answer Pairs”  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ Seattle, Washington, USA: Association for Computational Linguistics, 2013, pp. 1533–1544  URL: https://aclanthology.org/D13-1160
  * [44] Petr Baudiš and Jan Šedivý  “Modeling of the Question Answering Task in the YodaQA System”  In _Experimental IR Meets Multilinguality, Multimodality, and Interaction_ , Lecture Notes in Computer Science  Cham: Springer International Publishing, 2015, pp. 222–228  DOI: 10.1007/978-3-319-24027-5_20
  * [45] S.. Manathunga and Y.. Illangasekara  “Retrieval Augmented Generation and Representative Vector Summarization for large unstructured textual data in Medical Education”  arXiv, 2023  DOI: 10.48550/arXiv.2308.00479
  * [46] Sandeep Reddy  “Evaluating large language models for use in healthcare: A framework for translational value assessment”  In _Informatics in Medicine Unlocked_ 41, 2023, pp. 101304  DOI: 10.1016/j.imu.2023.101304
  * [47] Pat Verga, Haitian Sun, Livio Baldini Soares and William Cohen  “Adaptable and Interpretable Neural MemoryOver Symbolic Knowledge”  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ Online: Association for Computational Linguistics, 2021, pp. 3678–3691  DOI: 10.18653/v1/2021.naacl-main.288
  * [48] Freda Shi et al.  “Large Language Models Can Be Easily Distracted by Irrelevant Context”  arXiv, 2023  DOI: 10.48550/arXiv.2302.00093
  * [49] Jeff Johnson, Matthijs Douze and Hervé Jégou  “Billion-scale similarity search with GPUs”  arXiv, 2017  URL: http://arxiv.org/abs/1702.08734
  * [50] Anmol Arora and Ananya Arora  “The promise of large language models in health care”  In _The Lancet_ 401.10377, 2023, pp. 641  DOI: 10.1016/S0140-6736(23)00216-7
  * [51] Xi Yang et al.  “GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records”  arXiv, 2022  DOI: 10.48550/arXiv.2203.03540
  * [52] “Citations Added to MEDLINE® by Fiscal Year” Type: Product, Program, and Project Descriptions  URL: https://www.nlm.nih.gov/bsd/stats/cit_added.html#1
  * [53] Terrence J. Sejnowski  “Large Language Models and the Reverse Turing Test”  In _Neural Computation_ 35.3, 2023, pp. 309–342  DOI: 10.1162/neco_a_01563


# An optically defined phononic crystal defect

Thomas J. Clark<EMAIL_ADDRESS>Simon Bernard  Jiaxing Ma
Department of Physics, McGill University, Montréal, Québec H3A 2T8, Canada
Vincent Dumont  Laboratory for Solid State Physics, ETH Zurich, CH-8093
Zurich, Switzerland  Jack C. Sankey<EMAIL_ADDRESS>Department of
Physics, McGill University, Montréal, Québec H3A 2T8, Canada

###### Abstract

We present a phononic crystal with a defect mode defined and controlled
entirely by optical forces. By applying a strong optical trap to a single unit
cell of a Si3N4 phononic crystal membrane, we coherently couple many modes to
each other, and smoothly transfer a single defect mode into the phononic
bandgap. This is accompanied by localization of the mode’s spatial profile
from one spanning the entire crystal to one confined within just a few unit
cells, as evidenced by a 37-fold reduction in the participating mass. This
proof-of-concept coupling between light and mechanical mode shape
significantly broadens the optomechanical design space. It also demonstrates
the feasibility of optically programmable lattice potentials, enabling (in
addition to defects) reconfigurable waveguides, superlattices, and disorder,
with corresponding avenues of research toward creating unique states of
macroscopic quantum motion.

The forces exerted by light – first pondered by Kepler [1] and quantitatively
described by Maxwell [2] – now routinely play a central role in precision
interferometry. In the field of cavity optomechanics [3], the back-action
forces associated with strong electromagnetic measurements of motion are
exploited to gain otherwise inaccessible control over mechanical systems at
all size scales. Famously, light can tune a mechanical system’s damping
parameter [4, 5, 6, 7, 8] (amplifying or suppressing thermal motion), enabling
solid mechanical systems to be radiation-cooled to their motional ground
states [9, 10] where signatures of mechanical energy quantization emerge [11,
12, 13, 14]. Similarly, light can tune a mechanical system’s spring constant
[15, 16], to the point where a column of light is stiffer (though
significantly more brittle!) than a piece of diamond [17]. Often assisted by
these tools, optomechanical devices now routinely reside in a deeply quantum
regime, wherein radiation forces couple and manipulate individual phonons,
photons, and qubits [18, 14, 19, 20, 21, 22, 23, 21, 22, 23, 24, 25, 26],
perform basic tests of macroscopic quantum motion and entanglement [25, 19,
27, 28] (now at room temperature [29]), facilitate quantum information
transduction and storage [30, 31, 32, 33, 20, 34, 35, 36, 37], generate
optical squeezed states [38, 39, 40, 41] (now at room temperature [42, 43,
44]), gain active quantum control over motion [45, 46, 47] and approach [45]
or surpass [48] the Standard Quantum Limit. Notably, defects patterned into
phononic crystals have enjoyed wild success as a basic enabling tool within
this realm [10, 18, 39, 31, 11, 41, 29, 25, 19, 27, 45, 28, 35, 34, 36, 14,
48, 24, 26], owing much to their high isolation from lossy boundaries and
reduction in bending losses [49, 50, 51].

Figure 1:  Apparatus for observing an optically controlled mechanical crystal
defect. (a) A 1550 nm “control” beam passes through an intensity modulator
(IM), circulator, and wavelength division multiplexer (WDM) into a fiber
cavity centered on a phononic crystal lattice site (inset image). Radiation
forces can be applied via the IM control voltage $V_{\text{drive}}$, while a
second 1350 nm (purple) beam provides independent, backaction-free readout of
membrane motion. The reflected 1550 nm light is used to stabilize the laser’s
detuning from the cavity resonance via feedback to shear piezos attached to
the fiber mirrors. (b) Thermally driven mechanical noise spectrum. Vertical
dashed lines indicate the limits of the band gap simulated by a finite element
solver (COMSOL). Markers indicate the individual modes at the focus of this
study, with pink being the fundamental (1,1) membrane mode (54 kHz, simulated
mode mass 149 ng), gold is the (3,1) mode (116 kHz, 173 ng), and blue is the
band edge mode (375 kHz, 162 ng). The gray box highlights a region where
“edge-like modes” are predicted to (faintly) appear. (c) Membrane geometry and
simulated displacement profile of the band edge mode with 0 N/m, 1 N/m, and 5
N/m optical trap applied to a central lattice site (circled). The $\sim$5
$\upmu$m $1/e^{2}$ radius trap breaks the crystal’s periodicity, producing a
localized (bound) defect mode with reduced participating mass.

Importantly, when radiation lands on a mechanical structure, each photon
generally interacts with multiple mechanical modes simultaneously. This has
been used to coherently couple and hybridize two degenerate modes of a single
structure [52, 53], as well as two spatially separated (and wildly different)
mechanical systems, such as a membrane and a cloud of atoms [54]. Here, we
optically couple and hybridize _many_ near-degenerate mechanical modes to gain
strong control over the spatial distribution and mass of a mechanical mode. To
achieve this, we apply an optical trap to a single lattice site of a phononic
crystal, which simultaneously addresses a dense population of near-degenerate
modes at the band gap edge. From an optomechanics perspective, $N$ modes
sharing an optical trap also experience $(N^{2}-N)/2$ inter-mode optical
springs [55], causing hybridization that conspires to transfer a single mode
into the band gap as it localizes. Interestingly, this collective response is
predicted to produce a measurable change in mode shape with an average of a
single photon in the apparatus [56]. From a condensed matter perspective, the
trap can be formally viewed as a “defect potential” in the mechanical wave
equation, thereby breaking the lattice’s translational symmetry and producing
a bound state in the gap [56]. We note that, unlike heat-induced loosening of
a predefined structural defect [57, 58], the coupling in our system arises
from a coherent radiation pressure interaction. It can also (quickly) create
new structures in an otherwise defect-free crystal, and does not rely on
absorption or steady state non-equilibrium heat flow. Combined with the
aforementioned basic tools and the inherent advantages of a phononic crystal
host, this novel coupling between light and motion greatly expands the range
of possible optomechanical systems and studies, enabling optically
programmable defects, waveguides, superlattices, and disorder, with
applications ranging from dissipation studies to condensed matter simulations
to the preparation of unique macroscopic quantum states of motion. As shown in
Fig. 1a inset, our mechanical element, a 3.3 mm $\times$ 3.1 mm, 180 nm thick
Si3N4 membrane, patterned into a hexagonal phononic crystal lattice, is
positioned within a 30-$\upmu$m-long, finesse $\sim 10^{4}$ fiber optical
cavity [59]. The cavity fibers are epoxied within tight-tolerance ferrules
[60] that themselves are epoxied to shear piezos, enabling independent cavity
length and position adjustment. To reduce mechanical dissipation and acoustic
noise, the entire apparatus is housed within a vibration-isolated ultra-high
vacuum chamber at pressure $2\times 10^{-8}$ Torr. Figure 1a shows the optical
circuit, which employs a 1550 nm “control” beam (blue) to exert radiation
forces upon the membrane. This beam also measures membrane motion, and Fig. 1b
shows the thermally driven spectrum, which exhibits a well-defined bandgap
with boundary frequencies near those predicted by a finite-element simulation
of the full membrane (dashed lines). When actively driving the motion with
radiation forces from the control beam, optomechanical noise interference
effects [61, 41] sufficiently complicates the analysis that we employ a second
“readout” beam (purple in Fig. 1a) having wavelength (1350 nm) that is outside
the high-reflectivity band of the mirrors; this provides a low-finesse readout
with negligible backaction and sensitivity independent of control beam
dynamics.

Figure 2:  Evidence of defect localization in the frequency domain. (a)
Dependence of thermally driven mechanical spectrum on control beam (incident
power $\sim$ 0.84 mW) detuning $\Delta$ (normalized by the cavity decay rate
$\kappa$) near the lower edge of the band gap. The frequency shift of the band
edge mode greatly exceeds that of the other modes, consistent with the
hybridization predicted for many band edge modes optically trapped and coupled
to one another. (b) Measurement (left) and simulation (right) of frequency
shifts for the band edge (blue) and fundamental (1,1) (pink) modes (following
the colors in Fig. 1b). The frequency shift of the fundamental mode is in
stark contrast to with the superlinear response of the band edge mode, which
shifts 24.1 times farther than what is predicted for a single mode of fixed
mass; the dashed blue line extrapolates the initial linear dependence for
reference. The teal curve shows the simulated effective mass dropping to
$\sim$ 4 % of the untrapped value.

As simulated in Fig. 1c, when the control beam traps the indicated lattice
site (at the center of the green circle), the lower band-edge mode (blue
marker in Fig. 1b) is expected to evolve from fully delocalized (Fig. 1c,
left) to localized (Fig. 1c, right), creating a bound state with greatly
reduced mode mass. This effect first presents itself as an anomalous increase
in trap efficiency as the mode frequency pulls away from the band edge. Figure
2a shows how the mechanical spectrum near the lower band edge evolves with the
control beam’s detuning from the cavity resonance $\Delta$ for input power
$P_{\text{in}}=$ 0.84 mW. In contrast to the modest $\sim$kHz frequency shifts
of the lower band modes, the band edge mode frequency (blue marker) shifts by
16 kHz into the gap at the optimal detuning (red marker). Approaching the
optimal detuning from above, we also observe a complex network of avoided
crossings as the modes are trapped and coupled to one another with increasing
strength. At the optimal detuning, Fig. 2b (left) compares how the band edge
(blue) and fundamental (pink) mode frequencies shift with increasing incident
power $P_{\text{in}}$. The comparatively isolated fundamental mode resonance
frequency (nominally $\Omega_{1,1}/2\pi=$ 54 kHz) shifts roughly in proportion
to the optical spring constant $K_{\text{opt}}$ ($\propto P_{\text{in}}$),
which is consistent with a harmonic oscillator of (simulated) mass $m_{1,1}=$
149 ng and material spring $K_{1,1}=m_{\text{1,1}}\Omega_{1,1}^{2}$, since the
total frequency

$\Omega_{1,1,\text{tot}}=\sqrt{\frac{K_{1,1}+K_{\text{opt}}}{m_{1,1}}}\approx\Omega_{1,1}+\frac{K_{\text{opt}}}{2m_{1,1}\Omega_{1,1}}$
(1)

for small $K_{\text{opt}}$. The band edge mode frequency (nominally
$\Omega_{\text{BE}}/2\pi=$ 375 kHz with mass $m_{\text{BE}}=$ 162 ng) is found
to initially shift 11.9 times less than the fundamental, consistent with the
expected factor of $m_{1,1}\Omega_{1,1}/m_{\text{BE}}\Omega_{\text{BE}}\sim$
0.13 reduction from Eq. 1 and an optomechancial coupling 0.8 times that of the
fundamental mode (i.e., due to the mode’s different profile within the cavity
light). With increasing power, however, the band edge mode rapidly pulls away
from its extrapolated linear dependence (dashed line), producing a final
frequency shift that is – despite these factors – 2.8 times larger than that
of the fundamental and 24.1 times larger than the extrapolated value,
signifying the participation of many coupled modes near the band edge (or,
equivalently, that the trap is more efficient as the mode mass decreases).
This basic intuition is confirmed by finite-element simulation (right) over
the same range of trap strengths, which predicts a mode mass mass (teal)
reduced to 4% of the untrapped, delocalized value.

Figure 3:  Force-driven measurement of mechanical mode mass. (a) Noise spectra
of the band edge mode driven by $P_{\text{in}}=880$ $\upmu$W at the optimal
trap detuning, with optical white noise drive having (measured) amplitude
spectral densities 3.5 $\upmu$W/$\sqrt{\text{Hz}}$ (purple, lowest), 6.8
$\upmu$W/$\sqrt{\text{Hz}}$ (teal), and 12.7 $\upmu$W/$\sqrt{\text{Hz}}$
(green, highest), using the fixed-sensitivity 1350 nm beam for readout. (b)
Linear scaling of integrated defect mode resonance noise with input noise
power (i.e., the variance) scaled by the mechanical ringdown time
$\tau_{\text{m}}$ for incident powers $P_{\text{in}}=$ 50 (faintest, bottom),
350, 600, 750, 880 (darkest, top) $\upmu$W. Plotted this way, linear fits
(lines) with larger slope correspond to modes with lower mass [55]. (c) Mass
of the three modes indicated in Fig. 1b, – estimated from the fit slopes in
(b) – as a function of trap strength. The trap strength is expressed as a
frequency shift of the trapped modes. Untrapped values are estimated from the
simulation at zero power (dotted lines), which sets a constant overall scale
for each mode. The (1,1) and (3,1) modes (pink and gold) exhibit increasing
mass due to “pinning”, while the defect mode mass is strongly reduced (despite
pinning) due to localization.

To confirm this interpretation, we directly measure the evolution of mode mass
$m$ by driving its motion with radiation forces and measuring its response,
exploiting the fact that a (trapped or untrapped) harmonic oscillator has
susceptibility

$\chi_{\text{m}}(\omega)=\frac{1/m}{\Omega_{\text{m}}^{2}-\omega^{2}+i2\omega/\tau_{\text{m}}},$
(2)

where $\tau_{\text{m}}$ is the amplitude ringdown time, scales inversely with
$m$. To avoid artifacts from frequency drifts [62], we use the intensity
modulator (Fig. 1a) to apply a known white force noise of power spectral
density (PSD) $S_{F}$ near $\Omega_{\text{m}}$, and relate this to the
variance in the readout beam signal. Figure 3a shows the readout noise
spectrum near the defect mode frequency, with control beam power
$P_{\text{in}}=880~{}\upmu$W and an applied noise $\sqrt{S_{P}}$ of 3.5
$\upmu$W/$\sqrt{\rm{Hz}}$ (bottom), 6.8 $\upmu$W/$\sqrt{\rm{Hz}}$ (middle),
and 12.7 $\upmu$W/$\sqrt{\rm{Hz}}$ (top); these noise values are measured with
a pick-off diode after the intensity modulator, and notably include the
laser’s intrinsic classical noise. Each spectrum exhibits a resonant response
following the expected proportionality to
$|\chi_{\text{m}}|^{2}S_{P}~{}(\propto|\chi_{\text{m}}|^{2}S_{F})$, such that
the integrated spectrum (i.e., the variance)
$\propto\tau_{\text{m}}S_{P}/m^{2}$ [55], providing a simple estimator for
fractional changes in $m$. Figure 3b shows this variance, scaled by
$\tau_{\text{m}}$ to emphasize changes in $m$, for swept $S_{P}$ and multiple
values of $P_{\text{in}}$ from 50 $\upmu$W (bottom, lightest data) to 880
$\upmu$W (top, darkest data). When plotted this way, decreased mode mass $m$
is evidenced by increased slope. Figure 3c shows the evolution of so-measured
mode mass for more values of $P_{\text{in}}$. Also included is the same
measurement for two “reference” modes, the fundamental (1,1) (pink) and (3,1)
(gold) membrane modes indicated in Fig. 1b, along with simulated values
(dashed lines). The statistical uncertainty of this measurement is too small
to be seen, and the systematic fluctuations at low power reflect real changes
in mode mass, likely arising from partial hybridization with the modes of the
supporting structure [52] (the silicon chip), which has a high density of
modes and is not included in the simulation; we expect this to be a larger
effect when the mode is more delocalized and more strongly interacts with the
boundaries. The agreement between the long-wavelength reference modes and
simulation further validates the mass estimation method, while the factor of
$\sim$2 deviation from the measured bound state mass – when the wavelength
comparable to the lattice spacing – likely reflects the relative imprecision
in estimating the crystal’s smallest dimensions.

The most important feature of Fig. 3c is the observed trends: the reference
modes both exhibit an _increasing_ effective mass, as expected for a point-
like optical trap that “pins” the membrane amplitude [52, 63, 64, 56], and
(despite this pinning effect) the measured bound state mode mass drops to 2.7%
of its untrapped value. This demonstration of an optically programmable
phononic crystal defect is the main result of our work.

Figure 4:  Measured force noise at zero applied drive, extracted from the fit
offsets in the variance (i.e., Fig. 3b). Dotted lines indicate the expected
thermal force noise for the observed change in mass and spring, for fixed
intrinsic damping. The defect mode (blue) exhibits force noise significantly
below the constant damping expectation at certain trapping strengths,
highlighting certain mode shapes and frequencies that are better isolated from
the thermal environment. By contrast, the (1,1) and (3,1) (pink and gold)
exhibit force noise in excess of the thermal expectation, hinting at an
increase in intrinsic damping associated with their changing mode shapes and
frequencies. The inset expands the data for the (3,1) mode.

Our proof-of-concept phononic crystal is designed to optimize the size of the
band gap, not the resulting mechanical performance. Nonetheless, it is still
instructive to perform a preliminary exploration of the modes’ thermal force
noise as their profiles change shape. Figure 4 shows the force noise spectral
density of the three studied modes, as measured by extrapolating the variances
(e.g., those in Fig. 3b) to zero laser noise power, such that only the
response to environmental noise remains [55]. Also shown (dashed lines) is the
evolution of the thermal noise naively expected for the observed mass and
spring, assuming a fixed temperature bath and fixed intrinsic mechanical
damping [55]. Interestingly, the defect mode’s measured force noise is
generally lower (and occasionally much lower) than this expectation, likely
due to a combination of reduced clamping losses as the mode localizes and
variable bending losses as the shape is tuned. By contrast, the reference
modes show the opposite trend, which notably opposes what one might expect
from other partially levitated systems [52, 63, 64]. For these reference
modes, pinning plays a dominant role, increasing the mass and introducing
additional bending losses. The peaked structures suggest additional loss
mechanisms such as coupling to other modes may actually be enhanced. This
preliminary measurement suggests the technique’s potential for untangling
dissipation mechanisms in the design of novel ultralow-noise sensors, and
immediately motivates nontraditional dissipation studies: rather than
fabricating many “identical” devices, one can now smoothly vary what regions
of a mechanical system are bending during oscillation, and change the degree
to which a mode is coupled to lossy boundaries, other modes, and / or
deposited materials. In a similar direction, one could optically tune the
overlap between modes of a defect dimer [65, 66, 67] or with another system of
interest, such as a cavity at a different lattice site, a capacitively coupled
qubit, or any system that can be force-coupled to a mechanical device,
enabling hybrid quantum systems [68, 69, 70] with tunable coupling. With
additional trapping sites and / or a spatial light modulator, it should in
principle be possible to realize optically programmable phononic waveguides
[71, 72, 73], enabling new possibilities for mechanically shuttling signals
between different systems. Within the context of condensed matter systems,
this optical “potential” term in the phonon wave equation [56] permits
simulations of 1D and 2D periodic systems with arbitrary and programmable
superlattices, defects, and disorder, among other things providing
qualitatively new studies of Anderson localization [74], time-varying
dispersion/wavefront shaping [75], and topologically protected modes [72, 76].
We reemphasize that, in the limit of large crystal size, the density of states
near the band edge increases, enabling mechanical explorations of energy
transfer (i.e., Landau-Zener dynamics) between a single mode and a quasi-
continuum [77]. Also, due to the trap’s inter-mode coupling, larger crystals
do not require stronger traps to achieve the same level of localization,
meaning the per-photon effect on motion can be arbitrarily large, which is why
an average of a single photon in the apparatus should be able to produce
measurable changes in chip-scale devices [56]. We speculate this collective
behavior represents a useful resource on the route to exotic macroscopic
states of motion.

##  I acknowledgements

We gratefully acknowledge initial fabrication support from Abeer Barasheed. TC
acknowledges helpful conversations with Sebastian Spence and Michael Caouette-
Mansour. VD acknowledges financial support from FRQNT-B2 Scholarship and
McGill Schulich Graduate Fellowship. JCS acknowledges support from the Natural
Sciences and Engineering Research Council of Canada (NSERC RGPIN 2018-05635),
Canada Research Chairs (CRC 235060), Canadian foundation for Innovation (CFI
228130, 36423), Institut Transdisciplinaire d’Information Quantique (INTRIQ),
and the Centre for the Physics of Materials (CPM) at McGill.

##  II Data Availability Statement

The data presented here will be made available on the McGill Dataverse found
at https://borealisdata.ca/dataverse/mcgill .

##  III Supplementary Information

Optomechanical system. The phononic crystal in this study is a 180-nm-thick
sheet of Si3N4 photolithographically patterned into a crystal of 64$\upmu$m
“pads” connected by 200-$\upmu$m-long, 2-$\upmu$m-wide tethers. This nominally
defect-free crystal hosts an out-of-plane quasi-gap as measured in Fig. 1. The
mode’s intrinsic quality factors are typically measured to be $Q\sim 10^{5}$,
limited by the simple glue-mounting procedure below; we have since developed a
mounting procedure that preserves mechanical $Q$ (details available upon
request). The membrane is positioned inside the $\sim$30 $\upmu$m long fiber
optical cavity at a point where it maximizes the linear dispersive
optomechanical coupling. At this operating point, we measure the optical
cavity’s loss rate by applying optical sidebands with a fiber coupled electro-
optic modulator, then sweeping the cavity length to observe the optical
resonance width. Fitting the resonance together with the sidebands of known
frequency offset indicates $\kappa/2\pi\sim 0.8$ GHz. The optical trap used to
produce the defect mode is generated by locking a $\sim$1550 nm beam on the
blue side of the optical resonance, using the shear piezos under the fiber
mirrors for feedback.

Multiple modes sharing an optical trap. As evidenced by the presence of many
peaks in the thermal spectrum of Fig. 1(b), the motion of many mechanical
modes (indexed by $i$) will cause membrane displacement $x_{i}$ within the
cavity light, resulting in total displacement $x_{\text{tot}}=\sum_{i}x_{i}$.
As an illustrative toy model, the potential energy associated with a shared
optical spring constant $K_{\text{opt}}$ is then

$U_{\text{opt}}=\frac{1}{2}K_{\text{opt}}\left(\sum_{i}x_{i}\right)^{2},$ (3)

such that the force on mode $j$ is

$F_{j}=-\partial_{x_{j}}U_{\text{opt}}=-K_{\text{opt}}\left(x_{j}+\sum_{i\neq
j}x_{i}\right).$ (4)

Stated briefly, every mode experiences a spring coupling of strength
$K_{\text{opt}}$ to every other mode. For $N$ “relevant” (near-degenerate)
modes, this corresponds to $(N^{2}-N)/2$ spring couplings in total, producing
the enhanced frequency and amplitude response discussed in Ref. [56]. In a
real system, not all modes will have an antinode within the cavity light, but
many will still contribute with comparably strong spring couplings.

Assembly and alignment. Our fiber cavity rests on shear piezos that permit
$\sim$2 $\upmu$m modification of the cavity length and membrane position.
Alignment of the fiber mirrors is achieved by gluing (torrseal) tight
tolerance, rectangular profile glass ferrules atop the piezos. Bore alignment
is maintained during the epoxy cure (which exerts strong forces!) by inserting
a “guide” fiber through the bores, and butting the flat faces of the ferrules
against one-another. Once the glue is cured, a slot is ground between the
ferrules using a diamond cutting wheel. The chip hosting the phononic crystal
is lowered into this slot, the fiber mirrors are inserted into each ferrule,
and a central pad of the phononic crystal is aligned to the fibers using a
manual nanopositioning stage. Note the fibers have been partially etched [78],
using a surfactant that reduces vapor etching above the solution, such that
the mirror coating on the sidewalls of the fiber tips is removed, and the
fibers can fit into the ferrule; this also reduces the cavity finesse from
$\sim 10^{5}$ to $\sim 10^{4}$. After alignment, the crystal chip is then
affixed in place using torrseal epoxy, and a drop of low-viscocity uv-curing
epoxy (Masterbond UV15) is wicked into the bore of the ferrule by surface
tension (as in [60]) to secure the fiber while minimizing fiber overhang. This
cavity is built on a high-rigidity stainless steel mount, which is vibration
isolated inside a conflat cube ultrahigh-vacuum chamber using a spring
dampener suspended from the top flange.

Simulation details. Our finite element simulations are carried out with
COMSOL’s multiphysics membrane model. First, a static step captures the stress
redistribution inherent in the release of a patterned suspended membrane, and
then an eigenvalue solver predicts the mode frequencies. The local trap
responsible for the onset of the tunable defect mode is modelled as a Gaussian
spring density with width corresponding to the $\sim$5 $\upmu$m optical waist
predicted using the measured cavity length and mirror radius of curvature. The
overall spring constant reported as the x-axis in the right panel of Fig. 2b
is the integral of this Gaussian spring density.

Mass measurement. To measure the effective mass, we apply laser intensity
noise that is white near the mechanical resonance of interest using an
intensity modulator. The resulting radiation force noise in the cavity drives
the mode to an amplitude that is dependent on the mass. The mechanical
susceptibility connects the displacement spectrum and the applied force
spectrum:

$S_{x}(\omega)=|\chi_{m}(\omega)|^{2}S_{F}(\omega).$ (5)

where $S_{x}$, $S_{F}$ are the power spectral densities for position and
force, and $\chi_{m}$ is the mechanical susceptibility. We model our
mechanical mode as a viscously damped simple harmonic oscillator:

$|\chi_{m}(\omega)|^{2}=\frac{\tau_{m}^{2}}{4m^{2}\Omega_{m}^{2}\Big{(}1+\tau_{m}^{2}(\omega-\Omega_{m})^{2}\Big{)}}$
(6)

where $\Omega_{m}$ and $\tau_{m}$ are the (possibly optically modified)
mechanical frequency and ringdown time. We can write the variance as the
integral of the power spectral density [79]:

$\langle
x^{2}(t)\rangle=\int_{0}^{\infty}\frac{d\omega}{2\pi}S_{x}(\omega)=\int_{0}^{\infty}\frac{d\omega}{2\pi}|\chi_{m}(\omega)|^{2}S_{F}(\omega).$
(7)

We choose to consider the force spectral density as a sum of two terms, one
thermal drive and one optical:

$S_{F}(\omega)=S_{F}^{\mathrm{thermal}}(\omega)+S_{F}^{\mathrm{optical}}(\omega).$
(8)

Specializing to the case of high mechanical quality factor and locally white
applied force (i.e. $S_{F}(\omega)=S_{F}$) allows for explicit computation of
the integral:

$\langle
x^{2}(t)\rangle\approx\frac{\tau_{m}}{8m^{2}\Omega_{m}^{2}}\Big{(}S_{F}^{\mathrm{thermal}}+S_{F}^{\mathrm{optical}}\Big{)}.$
(9)

This can be interpreted as the equation of a line, where the mass is inversely
proportional to the slope of the displacement variance as the drive is
increased – as pictured Fig. 3b – and the thermal drive is proportional to the
offset – as pictured in Fig. 4. To avoid complications arising from
interference effects between the applied optical drive and the dispersive
effects of the vibrating membrane, we employ a second “readout” beam near 1350
nm. This beam, which is far outside the reflection band of the mirror coating,
supplies a low-finesse interferometric readout with negligible backaction that
(critically!) does not vary in sensitivity as the trap strength from the first
beam is swept.

Anomalous damping. Contrary to the theoretical expectation for radiation
pressure backaction, we observe strong optically induced mechanical damping
for both positive (“blue”) and negative (“red”) laser detunings. This
anomalous damping enables large optical springs to be applied without inducing
optomechanical instability, a highly experimentally convenient feature that
simplifies this study by eliminating the need to tranquilize unstable modes.
We suspect that this damping stems from photothermal effects in our fiber
mirrors, as discussed in [80, 81, 82], and a detailed analysis will be the
subject of a future work. Importantly, the additional damping does not affect
the effective mass inferences made in this study as the method employed is
agnostic to the nature of the applied backaction, relying only on the total
observed mechanical damping rate.

## References

  * Kepler [1619] J. Kepler, De cometis libelli tres, De cometis libelli tres I. astronomicus (1619).
  * Maxwell [1873] J. C. Maxwell,  _A treatise on electricity and magnetism_ , Vol. 1 (Clarendon press, 1873).
  * Aspelmeyer  _et al._ [2014] M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, Cavity optomechanics, Rev. Mod. Phys. 86, 1391 (2014).
  * Braginskiǐ  _et al._ [1970] V. Braginskiǐ, A. B. Manukin, and M. Y. Tikhonov, Investigation of dissipative ponderomotive effects of electromagnetic radiation, Soviet Journal of Experimental and Theoretical Physics 31, 829 (1970).
  * Mancini  _et al._ [1998] S. Mancini, D. Vitali, and P. Tombesi, Optomechanical Cooling of a Macroscopic Oscillator by Homodyne Feedback, Phys. Rev. Lett. 80, 688 (1998).
  * Cohadon  _et al._ [1999] P. F. Cohadon, A. Heidmann, and M. Pinard, Cooling of a Mirror by Radiation Pressure, Phys. Rev. Lett. 83, 3174 (1999).
  * Cuthbertson  _et al._ [1996] B. D. Cuthbertson, M. E. Tobar, E. N. Ivanov, and D. G. Blair, Parametric back-action effects in a high-Q cyrogenic sapphire transducer, Rev. Sci. Instrum. 67, 2435 (1996).
  * Schliesser  _et al._ [2006] A. Schliesser, P. Del’Haye, N. Nooshi, K. J. Vahala, and T. J. Kippenberg, Radiation Pressure Cooling of a Micromechanical Oscillator Using Dynamical Backaction, Phys. Rev. Lett. 97, 243905 (2006).
  * Teufel  _et al._ [2011] J. D. Teufel, T. Donner, D. Li, J. W. Harlow, M. S. Allman, K. Cicak, A. J. Sirois, J. D. Whittaker, K. W. Lehnert, and R. W. Simmonds, Sideband cooling of micromechanical motion to the quantum ground state, Nature 475, 359 (2011).
  * Chan  _et al._ [2011] J. Chan, T. P. M. Alegre, A. H. Safavi-Naeini, J. T. Hill, A. Krause, S. Gröblacher, M. Aspelmeyer, and O. Painter, Laser cooling of a nanomechanical oscillator into its quantum ground state, Nature 478, 89 (2011).
  * Safavi-Naeini  _et al._ [2012] A. H. Safavi-Naeini, J. Chan, J. T. Hill, T. P. M. Alegre, A. Krause, and O. Painter, Observation of Quantum Motion of a Nanomechanical Resonator, Phys. Rev. Lett. 108, 033602 (2012).
  * Purdy  _et al._ [2015] T. P. Purdy, P.-L. Yu, N. S. Kampel, R. W. Peterson, K. Cicak, R. W. Simmonds, and C. A. Regal, Optomechanical Raman-ratio thermometry, Phys. Rev. A 92, 031802 (2015).
  * Underwood  _et al._ [2015] M. Underwood, D. Mason, D. Lee, H. Xu, L. Jiang, A. B. Shkarin, K. Børkje, S. M. Girvin, and J. G. E. Harris, Measurement of the motional sidebands of a nanogram-scale oscillator in the quantum regime, Phys. Rev. A 92, 061801 (2015).
  * Meenehan  _et al._ [2015] S. M. Meenehan, J. D. Cohen, G. S. MacCabe, F. Marsili, M. D. Shaw, and O. Painter, Pulsed Excitation Dynamics of an Optomechanical Crystal Resonator near Its Quantum Ground State of Motion, Phys. Rev. X 5, 041002 (2015).
  * Vogel  _et al._ [2003] M. Vogel, C. Mooser, K. Karrai, and R. J. Warburton, Optically tunable mechanics of microlevers, Appl. Phys. Lett. 83, 1337 (2003).
  * Sheard  _et al._ [2004] B. S. Sheard, M. B. Gray, C. M. Mow-Lowry, D. E. McClelland, and S. E. Whitcomb, Observation and characterization of an optical spring, Phys. Rev. A 69, 051801 (2004).
  * Corbitt  _et al._ [2007] T. Corbitt, Y. Chen, E. Innerhofer, H. Müller-Ebhardt, D. Ottaway, H. Rehbein, D. Sigg, S. Whitcomb, C. Wipf, and N. Mavalvala, An All-Optical Trap for a Gram-Scale Mirror, Phys. Rev. Lett. 98, 150802 (2007).
  * Cohen  _et al._ [2015] J. D. Cohen, S. M. Meenehan, G. S. MacCabe, S. Gröblacher, A. H. Safavi-Naeini, F. Marsili, M. D. Shaw, and O. Painter, Phonon counting and intensity interferometry of a nanomechanical resonator, Nature 520, 522 (2015).
  * Hong  _et al._ [2017] S. Hong, R. Riedinger, I. Marinković, A. Wallucks, S. G. Hofer, R. A. Norte, M. Aspelmeyer, and S. Gröblacher, Hanbury Brown and Twiss interferometry of single phonons from an optomechanical resonator, Science 358, 203 (2017).
  * Reed  _et al._ [2017] A. P. Reed, K. H. Mayer, J. D. Teufel, L. D. Burkhart, W. Pfaff, M. Reagor, L. Sletten, X. Ma, R. J. Schoelkopf, E. Knill, and K. W. Lehnert, Faithful conversion of propagating quantum information to mechanical motion, Nat. Phys. 13, 1163 (2017).
  * O’Connell  _et al._ [2010] A. D. O’Connell, M. Hofheinz, M. Ansmann, R. C. Bialczak, M. Lenander, E. Lucero, M. Neeley, D. Sank, H. Wang, M. Weides, J. Wenner, J. M. Martinis, and A. N. Cleland, Quantum ground state and single-phonon control of a mechanical resonator, Nature 464, 697 (2010).
  * Chu  _et al._ [2017] Y. Chu, P. Kharel, W. H. Renninger, L. D. Burkhart, L. Frunzio, P. T. Rakich, and R. J. Schoelkopf, Quantum acoustics with superconducting qubits, Science 358, 199 (2017).
  * Chu  _et al._ [2018] Y. Chu, P. Kharel, T. Yoon, L. Frunzio, P. T. Rakich, and R. J. Schoelkopf, Creation and control of multi-phonon Fock states in a bulk acoustic-wave resonator, Nature 563, 666 (2018).
  * Riedinger  _et al._ [2016] R. Riedinger, S. Hong, R. A. Norte, J. A. Slater, J. Shang, A. G. Krause, V. Anant, M. Aspelmeyer, and S. Gröblacher, Non-classical correlations between single photons and phonons from a mechanical oscillator, Nature 530, 313 (2016).
  * Marinković  _et al._ [2018] I. Marinković, A. Wallucks, R. Riedinger, S. Hong, M. Aspelmeyer, and S. Gröblacher, Optomechanical Bell Test, Phys. Rev. Lett. 121, 220404 (2018).
  * Wallucks  _et al._ [2020] A. Wallucks, I. Marinković, B. Hensen, R. Stockill, and S. Gröblacher, A quantum memory at telecom wavelengths, Nature Physics 16, 772 (2020).
  * Fiaschi  _et al._ [2021] N. Fiaschi, B. Hensen, A. Wallucks, R. Benevides, J. Li, T. P. M. Alegre, and S. Gröblacher, Optomechanical quantum teleportation, Nat. Photonics 15, 817 (2021).
  * Riedinger  _et al._ [2018] R. Riedinger, A. Wallucks, I. Marinković, C. Löschnauer, M. Aspelmeyer, S. Hong, and S. Gröblacher, Remote quantum entanglement between two micromechanical oscillators, Nature 556, 473 (2018).
  * Purdy  _et al._ [2017] T. P. Purdy, K. E. Grutter, K. Srinivasan, and J. M. Taylor, Quantum correlations from a room-temperature optomechanical cavity, Science 356, 1265 (2017).
  * Safavi-Naeini  _et al._ [2011] A. H. Safavi-Naeini, T. P. M. Alegre, J. Chan, M. Eichenfield, M. Winger, Q. Lin, J. T. Hill, D. E. Chang, and O. Painter, Electromagnetically induced transparency and slow light with optomechanics, Nature 472, 69 (2011).
  * Hill  _et al._ [2012] J. T. Hill, A. H. Safavi-Naeini, J. Chan, and O. Painter, Coherent optical wavelength conversion via cavity optomechanics, Nat. Commun. 3, 1 (2012).
  * Palomaki  _et al._ [2013] T. A. Palomaki, J. W. Harlow, J. D. Teufel, R. W. Simmonds, and K. W. Lehnert, Coherent state transfer between itinerant microwave fields and a mechanical oscillator, Nature 495, 210 (2013).
  * Andrews  _et al._ [2014] R. W. Andrews, R. W. Peterson, T. P. Purdy, K. Cicak, R. W. Simmonds, C. A. Regal, and K. W. Lehnert, Bidirectional and efficient conversion between microwave and optical light, Nat. Phys. 10, 321 (2014).
  * Forsch  _et al._ [2020] M. Forsch, R. Stockill, A. Wallucks, I. Marinković, C. Gärtner, R. A. Norte, F. van Otten, A. Fiore, K. Srinivasan, and S. Gröblacher, Microwave-to-optics conversion using a mechanical oscillator in its quantum ground state, Nat. Phys. 16, 69 (2020).
  * Jiang  _et al._ [2020] W. Jiang, C. J. Sarabalis, Y. D. Dahmani, R. N. Patel, F. M. Mayor, T. P. McKenna, R. Van Laer, and A. H. Safavi-Naeini, Efficient bidirectional piezo-optomechanical transduction between microwave and optical frequency, Nat. Commun. 11, 1 (2020).
  * Mirhosseini  _et al._ [2020] M. Mirhosseini, A. Sipahigil, M. Kalaee, and O. Painter, Superconducting qubit to optical photon transduction, Nature 588, 599 (2020).
  * Kristensen  _et al._ [2023] M. B. Kristensen, N. Kralj, E. Langman, and A. Schliesser, A long-lived and efficient optomechanical memory for light, arXiv preprint arXiv:2308.05206 (2023).
  * Brooks  _et al._ [2012] D. W. C. Brooks, T. Botter, S. Schreppler, T. P. Purdy, N. Brahms, and D. M. Stamper-Kurn, Non-classical light generated by quantum-noise-driven cavity optomechanics, Nature 488, 476 (2012).
  * Safavi-Naeini  _et al._ [2013] A. H. Safavi-Naeini, S. Gröblacher, J. T. Hill, J. Chan, M. Aspelmeyer, and O. Painter, Squeezed light from a silicon micromechanical resonator, Nature 500, 185 (2013).
  * Purdy  _et al._ [2013] T. P. Purdy, P.-L. Yu, R. W. Peterson, N. S. Kampel, and C. A. Regal, Strong Optomechanical Squeezing of Light, Phys. Rev. X 3, 031012 (2013).
  * Nielsen  _et al._ [2017] W. H. P. Nielsen, Y. Tsaturyan, C. B. Møller, E. S. Polzik, and A. Schliesser, Multimode optomechanical system in the quantum regime, Proc. Natl. Acad. Sci. U.S.A. 114, 62 (2017).
  * Aggarwal  _et al._ [2020] N. Aggarwal, T. J. Cullen, J. Cripe, G. D. Cole, R. Lanza, A. Libson, D. Follman, P. Heu, T. Corbitt, and N. Mavalvala, Room-temperature optomechanical squeezing, Nat. Phys. 16, 784 (2020).
  * Militaru  _et al._ [2022] A. Militaru, M. Rossi, F. Tebbenjohanns, O. Romero-Isart, M. Frimmer, and L. Novotny, Ponderomotive Squeezing of Light by a Levitated Nanoparticle in Free Space, Phys. Rev. Lett. 129, 053602 (2022).
  * Magrini  _et al._ [2022] L. Magrini, V. A. Camarena-Chávez, C. Bach, A. Johnson, and M. Aspelmeyer, Squeezed Light from a Levitated Nanoparticle at Room Temperature, Phys. Rev. Lett. 129, 053601 (2022).
  * Rossi  _et al._ [2018] M. Rossi, D. Mason, J. Chen, Y. Tsaturyan, and A. Schliesser, Measurement-based quantum control of mechanical motion, Nature 563, 53 (2018).
  * Tebbenjohanns  _et al._ [2021] F. Tebbenjohanns, M. L. Mattana, M. Rossi, M. Frimmer, and L. Novotny, Quantum control of a nanoparticle optically levitated in cryogenic free space, Nature 595, 378 (2021).
  * Magrini  _et al._ [2021] L. Magrini, P. Rosenzweig, C. Bach, A. Deutschmann-Olek, S. G. Hofer, S. Hong, N. Kiesel, A. Kugi, and M. Aspelmeyer, Real-time optimal quantum control of mechanical motion at room temperature, Nature 595, 373 (2021).
  * Mason  _et al._ [2019] D. Mason, J. Chen, M. Rossi, Y. Tsaturyan, and A. Schliesser, Continuous force and displacement measurement below the standard quantum limit, Nat. Phys. 15, 745 (2019).
  * Alegre  _et al._ [2011] T. P. M. Alegre, A. Safavi-Naeini, M. Winger, and O. Painter, Quasi-two-dimensional optomechanical crystals with a complete phononic bandgap, Opt. Express 19, 5658 (2011).
  * Tsaturyan  _et al._ [2017] Y. Tsaturyan, A. Barg, E. S. Polzik, and A. Schliesser, Ultracoherent nanomechanical resonators via soft clamping and dissipation dilution, Nat. Nanotechnol. 12, 776 (2017).
  * Reetz  _et al._ [2019] C. Reetz, R. Fischer, G. G. T. Assumpção, D. P. McNally, P. S. Burns, J. C. Sankey, and C. A. Regal, Analysis of Membrane Phononic Crystals with Wide Band Gaps and Low-Mass Defects, Phys. Rev. Appl. 12, 044027 (2019).
  * Ni  _et al._ [2012] K.-K. Ni, R. Norte, D. J. Wilson, J. D. Hood, D. E. Chang, O. Painter, and H. J. Kimble, Enhancement of Mechanical $Q$ Factors by Optical Trapping, Phys. Rev. Lett. 108, 214302 (2012).
  * Shkarin  _et al._ [2014] A. B. Shkarin, N. E. Flowers-Jacobs, S. W. Hoch, A. D. Kashkanova, C. Deutsch, J. Reichel, and J. G. E. Harris, Optically Mediated Hybridization between Two Mechanical Modes, Phys. Rev. Lett. 112, 013602 (2014).
  * Jöckel  _et al._ [2015] A. Jöckel, A. Faber, T. Kampschulte, M. Korppi, M. T. Rakher, and P. Treutlein, Sympathetic cooling of a membrane oscillator in a hybrid mechanical–atomic system, Nat. Nanotechnol. 10, 55 (2015).
  * [55] See supplementary information, .
  * Barasheed  _et al._ [2016] A. Z. Barasheed, T. Müller, and J. C. Sankey, Optically defined mechanical geometry, Phys. Rev. A 93, 053811 (2016).
  * Sadeghi  _et al._ [2020] P. Sadeghi, M. Tanzer, N. Luhmann, M. Piller, M.-H. Chien, and S. Schmid, Thermal transport and frequency response of localized modes on low-stress nanomechanical silicon nitride drums featuring a phononic-band-gap structure, Physical Review Applied 14, 024068 (2020).
  * Shaniv  _et al._ [2023] R. Shaniv, C. Reetz, and C. A. Regal, Direct measurement of a spatially varying thermal bath using Brownian motion, Phys. Rev. Res. 5, 043121 (2023).
  * Hunger  _et al._ [2010] D. Hunger, T. Steinmetz, Y. Colombe, C. Deutsch, T. W. Hänsch, and J. Reichel, A fiber Fabry–Perot cavity with high finesse, New J. Phys. 12, 065038 (2010).
  * Saavedra  _et al._ [2021] C. Saavedra, D. Pandey, W. Alt, H. Pfeifer, and D. Meschede, Tunable fiber Fabry-Perot cavities with high passive stability, Opt. Express 29, 974 (2021).
  * Weis  _et al._ [2010] S. Weis, R. Rivière, S. Deléglise, E. Gavartin, O. Arcizet, A. Schliesser, and T. J. Kippenberg, Optomechanically Induced Transparency, Science 330, 1520 (2010).
  * Kumar  _et al._ [2023] S. Kumar, S. Spence, S. Perrett, Z. Tahir, A. Singh, C. Qi, S. Perez Vizan, and X. Rojas, A novel architecture for room temperature microwave optomechanical experiments, J. Appl. Phys. 133, 10.1063/5.0136214 (2023).
  * Chang  _et al._ [2012] D. E. Chang, K.-K. Ni, O. Painter, and H. J. Kimble, Ultrahigh-Q mechanical oscillators through optical trapping, New J. Phys. 14, 045002 (2012).
  * Müller  _et al._ [2015] T. Müller, C. Reinhardt, and J. C. Sankey, Enhanced optomechanical levitation of minimally supported dielectrics, Phys. Rev. A 91, 053849 (2015).
  * Catalini  _et al._ [2020] L. Catalini, Y. Tsaturyan, and A. Schliesser, Soft-Clamped Phononic Dimers for Mechanical Sensing and Transduction, Phys. Rev. Appl. 14, 014041 (2020).
  * Hälg  _et al._ [2021] D. Hälg, T. Gisler, Y. Tsaturyan, L. Catalini, U. Grob, M.-D. Krass, M. Héritier, H. Mattiat, A.-K. Thamm, R. Schirhagl, E. C. Langman, A. Schliesser, C. L. Degen, and A. Eichler, Membrane-Based Scanning Force Microscopy, Phys. Rev. Appl. 15, L021001 (2021).
  * Hälg  _et al._ [2022] D. Hälg, T. Gisler, E. C. Langman, S. Misra, O. Zilberberg, A. Schliesser, C. L. Degen, and A. Eichler, Strong Parametric Coupling between Two Ultracoherent Membrane Modes, Phys. Rev. Lett. 128, 094301 (2022).
  * Stannigel  _et al._ [2011] K. Stannigel, P. Rabl, A. S. Sørensen, M. D. Lukin, and P. Zoller, Optomechanical transducers for quantum-information processing, Phys. Rev. A 84, 042341 (2011).
  * Chu and Gröblacher [2020] Y. Chu and S. Gröblacher, A perspective on hybrid quantum opto- and electromechanical systems, Appl. Phys. Lett. 117, 10.1063/5.0021088 (2020).
  * Kurizki  _et al._ [2015] G. Kurizki, P. Bertet, Y. Kubo, K. Mølmer, D. Petrosyan, P. Rabl, and J. Schmiedmayer, Quantum technologies with hybrid systems, Proc. Natl. Acad. Sci. U.S.A. 112, 3866 (2015).
  * Hatanaka  _et al._ [2014] D. Hatanaka, I. Mahboob, K. Onomitsu, and H. Yamaguchi, Phonon waveguides for electromechanical circuits, Nat. Nanotechnol. 9, 520 (2014).
  * Ren  _et al._ [2022] H. Ren, T. Shah, H. Pfeifer, C. Brendel, V. Peano, F. Marquardt, and O. Painter, Topological phonon transport in an optomechanical system, Nat. Commun. 13, 1 (2022).
  * Brendel  _et al._ [2018] C. Brendel, V. Peano, O. Painter, and F. Marquardt, Snowflake phononic topological insulator at the nanoscale, Phys. Rev. B 97, 020102 (2018).
  * Anderson [1958] P. W. Anderson, Absence of Diffusion in Certain Random Lattices, Phys. Rev. 109, 1492 (1958).
  * Hatanaka  _et al._ [2019] D. Hatanaka, A. Bachtold, and H. Yamaguchi, Electrostatically Induced Phononic Crystal, Phys. Rev. Appl. 11, 024024 (2019).
  * Lu  _et al._ [2017] J. Lu, C. Qiu, L. Ye, X. Fan, M. Ke, F. Zhang, and Z. Liu, Observation of topological valley transport of sound in sonic crystals, Nat. Phys. 13, 369 (2017).
  * Basko [2017] D. M. Basko, Landau-Zener-Stueckelberg Physics with a Singular Continuum of States, Phys. Rev. Lett. 118, 016805 (2017).
  * Bernard  _et al._ [2020] S. Bernard, T. J. Clark, V. Dumont, J. Ma, and J. C. Sankey, Monitored wet-etch removal of individual dielectric layers from high-finesse Bragg mirrors, Opt. Express 28, 33823 (2020).
  * Miller  _et al._ [2018] J. M. L. Miller, A. Ansari, D. B. Heinz, Y. Chen, I. B. Flader, D. D. Shin, L. G. Villanueva, and T. W. Kenny, Effective quality factor tuning mechanisms in micromechanical resonators, Appl. Phys. Rev. 5, 10.1063/1.5027850 (2018).
  * Altin  _et al._ [2017] P. A. Altin, T. T.-H. Nguyen, B. J. J. Slagmolen, R. L. Ward, D. A. Shaddock, and D. E. McClelland, A robust single-beam optical trap for a gram-scale mechanical oscillator, Sci. Rep. 7, 1 (2017).
  * Ballmer [2015] S. W. Ballmer, Photothermal transfer function of dielectric mirrors for precision measurements, Phys. Rev. D 91, 023010 (2015).
  * Kelley  _et al._ [2015] D. Kelley, J. Lough, F. Mangaña-Sandoval, A. Perreca, and S. W. Ballmer, Observation of photothermal feedback in a stable dual-carrier optical spring, Phys. Rev. D 92, 062003 (2015).


# Some Practice for Improving the Search Results of E-commerce

Fanyou Wu  1234-5678-9012 Department of Forest and Natural ResourcePurdue
UniversityWest LafayetteIndianaUSA<EMAIL_ADDRESS>,  Yang Liu  School of
Vehicle and Mobility
Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>,  Rado Gazo  Department of
Forest and Natural ResourcePurdue UniversityWest LafayetteIndianaUSA ,  Benes
Bedrich  Department of Computer SciencePurdue UniversityWest
LafayetteIndianaUSA and  Xiaobo Qu  School of Vehicle and Mobility Tsinghua
UniversityBeijingChina

###### Abstract.

In the Amazon KDD Cup 2022, we aim to apply natural language processing
methods to improve the quality of search results that can significantly
enhance user experience and engagement with search engines for e-commerce. We
discuss our practical solution for this competition, ranking 6th in task one,
2nd in task two, and 2nd in task 3. The code is available at
https://github.com/wufanyou/KDD-Cup-2022-Amazon.

search relevance, querying, e-commerce, semantic matching

††ccs: Information systems Retrieval models and ranking††ccs: Information
systems Query representation††ccs: Applied computing Online shopping Figure 1.
Overall schema of our proposed solution for Amazon KDD CUP 2022 for all three
tasks.

##  1\. Problem Description

The organizer provides a dataset called the Shopping Queries Dataset (Reddy et
al., 2022). It is a large-scale, manually annotated dataset composed of
challenging customer queries. The data is multilingual and includes English,
Japanese, and Spanish queries. It comprises query-result pairs with annotated
four classes of relevance (ESCI labels):

  * •

Exact (E): the item is relevant for the query and satisfies all the query
specifications;

  * •

Substitute (S): the item is somewhat relevant: it fails to fulfill some
aspects of the query, but the item can be used as a functional substitute;

  * •

Complement (C): the item does not fulfill the query but could be used in
combination with an exact item;

  * •

Irrelevant (I): the item is irrelevant, or it fails to fulfill a central
aspect of the query.

The primary objective of this competition is to build new ranking strategies
and, simultaneously, identify interesting categories of results (i.e.,
substitutes) that can be used to improve the customer experience when
searching for products. The three different tasks for this KDD Cup competition
using our Shopping Queries Dataset are:

  1. T1.

Query-Product Ranking

  2. T2.

Multiclass Product Classification

  3. T3.

Product Substitute Identification

Task one (T1) aims at ranking the relevance of a subset of the ESCI dataset by
using Normalized Discounted Cumulative Gain (nDCG) score to measure the
performance. The organizer designed a customized Discounted Cumulative Gain
(DCG) of 1.0, 0.1, 0.01 and 0.0, for Exact, Substitute, Complement and
Irrelevant respectively.

Task two (T2) aims to classify each product as being an Exact, Substitute,
Complement, or Irrelevant match for the query. The Micro-F1 (equivalent to
accuracy here) will be used to evaluate the methods. Task three (T3) is a
binary classification problem that tries to distinguish whether a query
product pair is a Substitute or not and uses the Micro-F1 score as well to
measure the performance.

T1 uses a subset of ECSI dataset, while T2 and T3 use the same dataset. It is
natural to treat this competition as two different problems. In the rest of
the paper, we will use short-term T2T3 to represent tasks two and three. This
inner difference among tasks is also reflected on the final leaderboard that
the final ranking of T1 and T2T3 are not correlated well. The setup also
involves a potential data leakage, which we will discuss further in section 2.

##  2\. Explore Data Analysis

ECSI dataset contains three tables, which are `product_catalogue`, `train` and
`test`. The `test` is also decomposed into `public` and `private` where the
latter one is unseen to us. In this section, we make our special observation
of `product_catalogue` and `train` that play an important role in the final
leaderboard.

T1 and T2T3 use different `product_catalogue` tables. Figure 2 shows the order
of product entries in T2T3. T1 remains a similar pattern unless it is started
with es entries (es $\rightarrow$ us $\rightarrow$ jp). There is a part of
products that have not been used in the both training set and the public test
set, and we conjecture those entries are unique in the private test set.

Figure 2. The order of product entries in T2T3.

Let us have another investigation about `train`. Figure 3 shows the histogram
of products grouped by queries in T1 and T2T3. Generally, most queries will
sample 16 and 40 products for training and test sets. And this distribution is
slightly different between T1 and T2T3, while we know that there are fewer
Exact labels in T1. So associated with this prior knowledge, using this
product number as a feature to calibrate the prediction will make some
improvement in the leaderboard.

Figure 3. The histogram of products grouped by queries in the training set. 16
and 40 are the typical number of products for each query.

Another important piece of information in the training set is that the
proportion of ESCI labels in T1 and T2T3 are different (Figure 4). This
difference creates a well-known data leakage for most participants in T2T3
that distinguishing whether a query-product pair is in T1 will improve
results. However, it isn’t easy to use this information directly in the
private test set.

Figure 4. The proportion of ESCI labels in T1 and T2T3 training set.

Besides the above-described deep dive, some other patterns will help to
improve the scores. Here is the summary of that information:

  * E1.

The order of the product catalog is not randomized (training set $\rightarrow$
private test set $\rightarrow$ public test set).

  * E2.

Most products are used once.

  * E3.

The ESCI label proportion is different between T1 and T2T3.

  * E4.

Most queries have 16 or 40 product numbers and the label distribution of those
queries are slightly different.

  * E5.

The product id is called ASIN and will be identical to ISBN (starts with
digits) if the product has ISBN.

  * E6.

Most query products group has fewer unique brand numbers than product numbers
and the product with the most frequent brand tends to be labeled as Exact.

  * E7.

At least one product in a query-products group will be labeled as Exact, and
the label of the query-product pair is affected by other labels in this group
as well.

Combining those explorations, we could significantly improve scores for all
three tasks. A detailed feature engineering will be introduced in Section 3.2.

##  3\. Proposed Solution

Figure 1 shows the general schema of our proposed solution for Amazon KDD CUP
2022 for all three tasks. As we planned to attend to all three tasks, for
efficiency, we have to train the cross-encoders once and use them for all
three tasks. This strategy makes this two-stage solution the only choice. So
we trained all cross-encoders with all data from T1 and T2T3 in two folds and
then combined the four class probabilities with other essential features,
using lightGBM to fuse and calibrate the prediction and adapt results to
different tasks. Figure 5 shows the milestone of the public leaderboard score
for T2. In the following, we will discuss those milestones in detail.

Figure 5.  Our milestone of public leaderboard score for T2. We use red points
and blue points to represent the improvement from cross-encoders models or the
gain from feature engineering, respectively.

###  3.1. Cross-Encoder Architecture

In the first stage, we applied the classical cross encoder (Reimers and
Gurevych, 2019) architecture with only minor modifications. As the product
context has multiple fields (title, brand, and so on), we use neither the CLS
token nor mean (max) pooling to get the latent vector of the query-product
pair. Instead, we concatenate the hidden states of a predefined token (query,
title, brand color, etc.). This small modification yields about a 0.002
increase in the T2 public leaderboard (0.802 $\rightarrow$ 0.804).

In the final solution, we ensembled three cross-encoders for each language
that differ from pre-trained models, the training data, or the input fields.
For Engish entries, we used DeBERTaV3 (He et al., 2021), BigBird (Zaheer et
al., 2020) and COCO-LM (Meng et al., 2021). While for Japanese and Spanish
ones, we used a multi-language version of DeBERTaV3. All those pre-trained
data could be found at Huggerface. Table 1 shows the average accuracy for each
model for T2. By ensemble of all models, the score for the public leaderboard
for task two is 0.818.

Table 1. Average accuracy for each model for T2. Here we used 2-fold cross-validation. Note for code submission, we used the different seeds to split data, so the accuracy here is not very comparable. Locale | Pretrained Model | Accuracy
---|---|---
us | bigbird-roberta-base | 0.7587
| cocolm-base | 0.7588
| debertav3-base | 0.7585
es | mdeberta-v3-base | 0.7347
jp | mdeberta-v3-base | 0.7006

###  3.2. Feature Engineering

Once stacking a lot of models has tiny improvements for both local and online
tests, we start to do some feature engineering based on our exploration in
section 2.

####  3.2.1. Leakage Features

Combining E1, E2, and E3 together, we designed a feature that measures the
percentage of `product_id` in Task 1 product list grouped by `query_id`. This
feature gives us a 0.005 improvement in the T2 public leaderboard and remains
effective for the private test set, and that’s why we are extremely closed to
first place in T2 (0.0001 difference).

####  3.2.2. Query product number features

Based on E4, we use the product number for each query as a feature and obtain
an approximate 0.002 improvement in the T2 public leaderboard.

####  3.2.3. Product ID features

Based on E5, we designed features that measure whether the `product_id` is
ISBN or not and whether the query-products group has an ISBN product or not.
This feature gives us an approximate 0.001 improvement in the T2 public
leaderboard.

####  3.2.4. Brand Features

Based on E6, we designed features that measure the unique number of brands in
a query-products group and whether the brand of the product is the most
frequent one in the group. This feature gives us an approximate 0.001
improvement in the T2 public leaderboard.

####  3.2.5. Group Features

Based on E7, we designed several stats (min, medium, and max) of the cross
encoder output probability grouped by `query_id`. This feature gives us a
0.008 improvement in the T2 public leaderboard.

###  3.3. LightGBM model

####  3.3.1. T1

As the ECSI label distribution is different in T1 and T2T3, for T1, we train
the lightGBM model with T1 data only to simulate this distribution and
calculate the expected gain for each query-product pair as:

(1) $s=p_{e}+0.1\cdot p_{s}+0.01\cdot p_{c},$

where $p_{e}$, $p_{s}$ and $p_{c}$ are the probability output of lightGBM for
label Exact, Substitute and Irrelevant respectively. Then we sort the query-
product list by this gain. This method is slightly better than using
LambdaRank (Burges et al., 2006) with the same label gain (0,0.01,0.1,1) in
LightGBM.

####  3.3.2. T2T3

We use full data from T1 and T2T3, and use lightGBM to train either a four-
class classifier (T2) or a binary classifier (T3).

####  3.3.3. Model ensemble

For T1, T2, and T3, we average the gain or the prediction from 6 models (3
models x 2 folds) for each language to make a final decision.

##  4\. Inference Acceleration

During the code submission round, the organizer proposed a 120 minutes time
limit for all three tasks. This time limit requests us to provide a more
efficient solution.

###  4.1. Knowledge Distillation

We use knowledge distillation (Hinton et al., 2015) to improve the model’s
performance. This knowledge distillation is applied to English entries only.
We used all data and trained large versions of DeBERTaV3, BigBird, and COCO-
LM. Then we used a linear combination loss of cross-entropy (loss between
prediction and ground truth) and mean-square-error (logit difference between
student model and teacher model).

###  4.2. Other Inference Acceleration Strategies

Here are we listed some other inference acceleration strategies:

  * A1.

Pre-process product token and save it as an HDF5 file.

  * A2.

Transfer all models to ONNX with FP16 precision.

  * A3.

Pre-sort the product id to reduce the side impact of batch zero padding.

  * A4.

Use a relatively small mini-batch size ($=4$) when inference.

## References

  * (1)
  * Burges et al. (2006) Christopher Burges, Robert Ragno, and Quoc Le. 2006.  Learning to rank with nonsmooth cost functions.  _Advances in neural information processing systems_ 19 (2006).
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _arXiv preprint arXiv:2111.09543_ (2021).
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015\.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ 2, 7 (2015).
  * Meng et al. (2021) Yu Meng, Chenyan Xiong, Payal Bajaj, Paul Bennett, Jiawei Han, Xia Song, et al. 2021\.  Coco-lm: Correcting and contrasting text sequences for language model pretraining.  _Advances in Neural Information Processing Systems_ 34 (2021), 23102–23114.
  * Reddy et al. (2022) Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022.  Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search.  arXiv:2206.06588
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020\.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ 33 (2020), 17283–17297.