# LVLM-Intrepret: An Interpretability Tool for Large Vision-Language Models

Gabriela Ben Melech Stan111footnotemark: 1 Estelle Aflalo1 Raanan Yehezkel
Rohekar111footnotemark: 1 Anahita Bhiwandiwalla111footnotemark: 1
Shao-Yen Tseng111footnotemark: 1 Matthew Lyle Olson111footnotemark: 1 Yaniv
Gurwicz111footnotemark: 1 Chenfei Wu2 Nan Duan2 Vasudev Lal1
1Intel Labs 2Microsoft Research Asia Main authors

###### Abstract

In the rapidly evolving landscape of artificial intelligence, multi-modal
large language models are emerging as a significant area of interest. These
models, which combine various forms of data input, are becoming increasingly
popular. However, understanding their internal mechanisms remains a complex
task. Numerous advancements have been made in the field of explainability
tools and mechanisms, yet there is still much to explore. In this work, we
present a novel interactive application aimed towards understanding the
internal mechanisms of large vision-language models. Our interface is designed
to enhance the interpretability of the image patches, which are instrumental
in generating an answer, and assess the efficacy of the language model in
grounding its output in the image. With our application, a user can
systematically investigate the model and uncover system limitations, paving
the way for enhancements in system capabilities. Finally, we present a case
study of how our application can aid in understanding failure mechanisms in a
popular large multi-modal model: LLaVA.

##  1 Introduction

Recently, large language models (LLM), such as those in the families of GPT
[23] and LLaMAsc [39, 40], have demonstrated astounding understanding and
reasoning capabilities, as well as the ability to generate output that adheres
to human instructions. Building on this ability, many work, such as GPT-4V
[24], Qwen-VL [4], Gemini [37], and LLaVA [14], have introduced visual
understanding to LLMs. Through the addition of a vision encoder followed by
finetuning on multimodal instruction-following data, these prior work have
demonstrated large vision-language models (LVLM) that are able to follow human
instructions to complete both textual and visual tasks with great aptitude.

LLMs are rapidly surpassing humans in many tasks such as summarization,
translation, general question answering, and even creative writing. However,
they are still very prone to hallucination, i.e. the fabrication of untrue
information [12, 45]. This phenomenon of hallucination is also seen in LVLMs
and may even include additional dimensions stemming from the visual modality
[42, 50]. With the introduction of LVLMs and their massively increased number
of parameters, interpreting and explaining model outputs to mitigate
hallucination is becoming an ever rising challenge. In light of the need to
understand the reasoning behind model responses, we present an
intrepretability tool for large vision-language models: LVLM-Intrepret. The
main contributions of the proposed application are the adaptations of multiple
interpretability methods to large vision-language models for interactive
analysis. These methods include raw attention, relevancy maps, and causal
interpretation. LVLM-Interpret is applicable to any LVLM with a transformer-
based LLM front-end. In this paper, we demonstrate a case study of how our
application can be used to better understand the inner workings and failure
mechanisms in LLaVA [14].

Figure 1: Main interface of LVLM-Interpret. Users can issue multimodal queries
using a chatbot interface. Basic image-editing feature allows for model
probing.

(a) Image-to-Query raw attentions. The user is able to select a token or a
group of tokens to visualize the attention values of image to text output for
each head and layer.

(b) Query-to-Image raw attentions. The user is able to select image patches to
visualize attention values going into answer tokens.

Figure 2: Visualization of crossmodal attentions

##  2 Related Work

The advancements in deep learning models has been preceded by novel
interpretability and explainability tools to better understand the internal
workings of these models. Earlier works [48, 49, 43] demonstrated the use of
explanatory graphs, decision trees, histograms, respectively to analyze
machine learning models. As Transformer [41] based architectures gained
popularity in the field, various approaches such as [7] proposed computing
relevancy scores across the layers of the model, [31] generalized the
attention from low-level input features to high-level concepts to ensure
interpretability within a specific domain, while [25] presented a novel
interpretability-aware redundancy reduction transformer framework.

###  2.1 Interpretability of Vision Models

Studying the intepretability of Vision Transformers (ViT) has gained
popularity with task-specific analysis like image captioning [8, 35, 10],
object detection [44, 9, 3], image recognition [19, 46]. Recently, there has
been an increased demand for interpretability analysis for the medical domain
in applications such as pathology [22, 13], retinal image classification [26,
11] and COVID-19 analysis [20, 34] among others. A novel vision transformer
was presented in [27] with a training procedure which has an interpretability-
aware training objective. [21] proposed a method to use the activations of a
vision transformer’s hidden layers to predict the relevant parts of the input
that contribute to its final predictions, while [17] introduced quantification
indicators to measure the impact of patch interactions to effectively exploit
responsive fields of each patch in ViT.

###  2.2 Interpretability of Multimodal Models

Multimodal models have proliferated various domains across healthcare,
multimedia, industrial applications among others. There has been a rise in
independent interpretability studies of such multimodal systems [29, 2, 36,
16, 15, 6]. For the medical domain where the reasoning behind decisions, high
stakes involved are of utmost importance. [18] demonstrated the use of an
interpretability method based on attention gradients to guide the transformer
training in a more optimal direction, while [5] presented an interpretable
fusion of structural MRI and functional MRI modalities to enhance the accuracy
of schizophrenia.

Our hope with this proposed interpretability tool is not to replace domain-
specific solutions, but to complement them, improving existing and future
large vision language models and further strengthen the confidence in the
predictions and behavior of these models.

##  3 Interface and Interpretability Functions

LVLM-Intrepret was developed using Gradio [1] and follows a standard layout
for multimodal chat. Figure 1 shows an example of the user interface. In the
UI, a user is able to upload an image and issue multimodal queries to the
LVLM. An added editing feature allows for basic modification of the input
image to probe the model with adversarial variations. As the LVLM model
generates a response, the attention weights of the model are stored internally
and are later presented to the user for visualization. The application also
constructs relevancy maps and causal graphs relating to the output of the
model. Once a response is returned, the user is able to utilize these results
as a way to interpret the model output. The following sections describe each
of these interpretability functions.

(a) Raw high-Attention

(b) Search distance = 1

(c) Search distance = 2

(d) Search distance = 3

(e) Search distance = 4

Figure 3: Causality-based explanation for the token ‘yellow’ in the generated
answer ‘The man’s shirt is yellow’ at head 24. (a) Top 50 image-tokens having
the highest raw attention values. Each serves as a graph node. (b-e) Image
tokens from the explanation set identified by the CLEANN method, at different
search distances on the learned causal graph. Tokens are marked with yellow
blobs.

###  3.1 Layer Attentions

Following work such as VL-Interpret [2], LVLM-Interpret also allows for
interactive visualization of raw attentions. More specifically, our
application allows users to investigate the interactions among tokens from
each modality. Heatmaps that show average attentions between image tokens and
query tokens as well as answer tokens enables the user to better understand
the global behavior of raw attentions. Figure 2 shows how a user can visualize
raw attentions for a specific head and layer. As shown in Figure 2(a), the
user can select tokens from the generated response and visualize the average
attentions between image patches and the selected tokens to obtain insight on
how the model attends to the image when generating each token. Conversely,
Figure 2(b) shows how a user can select image patches and visualize the degree
to which each output tokens attends to that specific location.

###  3.2 Relevancy Map

Relevancy maps [7, 6] aim at interpreting the decision-making process of
transformers. These maps are designed to enhance interpretability by
illustrating how different components of an input, whether text or image, are
relevant to the model’s generated output, overcoming some limitations of
traditional attention visualization techniques. The method assigns a local
relevancy scores to each element in the input based on their contribution to
the output decision. We refer the reader to [6] for more details on the
approach.

We adapted the calculation of relevancy maps to LVLMs such as LLaVA. Relevancy
scores are backward propagated through the LLM as well as vision transformer
commonly used as the vision encoder. For image analysis, the relevancy scores
corresponding to image patches are reshaped into a grid that matches the
layout of the original image. This grid forms the basis of the relevancy map.
The relevancy map is then upscaled to the original image size using bilinear
interpolation, providing a visual representation of the parts of the image
most relevant to each generated token.

Relevancy maps can aid in model debugging, ensuring fairness, and providing
explanations for inaccuracies by identifying the the most relevant parts of
the input to the generated output, as demonstrated through a case study in
Section 4.

###  3.3 Causal Interpretation

Figure 4: A tree constructed from the causal graph from which explanations for
the token ’yellow’ are extracted. Arc radius indicates distance on the causal
graph. Edges are color coded, bi-directed edges indicate a latent confounder,
a circle edge-mark indicates that both a ‘tail’ and ’arrow’ are valid.

Recently, a causal interpretation of the attention mechanism in transformers
was presented [33]. This interpretation leads to a method for deriving causal
explanations from attention in neural networks (CLEANN). In this sense, if
explanation tokens would have been masked in the input, the model would have
generated a different output. Such explanations, which are a subset of the
input tokens, are generally tangible and meaningful to humans. Here, in
addition to the explanation, we also plot the causal graphs around the
explained tokens, and allow the user to decrease or increase the explanation
set size based on this graph. We refer the reader to [32, 33] for more
details.

We employ CLEANN to explain large vision-language models by learning causal
structures over input-output sequences of tokens. The presence of each token
in this sequence is an event which is represented by a node in the causal
graph. Thus, the event of generating an output token is explained by the
presence of a subset of input tokens.

Consider the following example. A sequence of image token is given as part of
the prompt, and the text prompt is ‘What color is the man’s shirt?’. In
response, the model generates ‘The man’s shirt is yellow.’. One may be
interested in understanding which image tokens are responsible for the the
token yellow. That is, identify the parts of the image such that if masked,
will cause the model to output a different color. First, the top-$k$ tokens
having the highest attention values for yellow are assigned to nodes. Then,
using the full attention matrix of the last (deepest) layer is used to learn a
causal graph. This causal graph is a partial ancestral graph [30, 47], where a
circle edge-mark indicates a non identifiable edge mark (head and tail are
equally valid). From this graph, a tree rooted at node yellow is extracted
(Figure 4) such that it includes all paths that potentially influence the root
[33, Appendix B. Definition 2]. CLEANN searches for the minimal explaining set
by gradually increasing the search distance in this tree (radius in Figure 4)
from the explained node. We enable setting the search distance manually for
examining the output. An example is given in Figure 3. In Figure 3(a), the 50
tokens having the highest attention values are marked. In Figure 3(b)–Figure
3(e), tokens within different search distances from the explained token are
marked.

This explanation approach solely relies on attention values in the last layer.
While raw attention values describe pair-wise, marginal dependence relations
between a token and the explained token, a causal-discovery algorithm
identifies conditional independence relations. Thus, based only on the current
trained weights, it can identify, those tokens that if perturbed may change
the generated token.

Figure 5: Example where LLaVA seems to prioritize text input over image
content. Presented with an unchanging image of a garbage truck, the model
provides contradictory responses (‘yes, the door is open’ vs. ‘yes, the door
is closed’) based on the query’s phrasing. Relevancy maps and bar plots for
open and closed tokens demonstrate higher text relevance compared to image.

##  4 Case Study

To demonstrate the functionalities of LVLM-Interpret, we analyze the LLaVA
model on the Multimodal Visual Patterns (MMVP) benchmark dataset [38]. MMVP
focuses on identifying “CLIP-blind pairs”, i.e. images that are perceived as
similar by CLIP [28] despite having clear visual differences. This dataset is
of particular interest for our study since it highlights the challenges faced
in answering relatively straightforward questions, often leading to incorrect
responses.

We adapted the relevancy scores to examine the impact of both text and image
tokens on the output generated by LLaVA-v1.5-7b. Given that the text and
vision transformers responsible for generating the input embedding were kept
frozen during LLaVA finetuning, our initial step involved calculating the
relevancy scores for each generated output relative to the input features to
LLaMA, focusing on the LLaMA self-attention layers. We observed instances
where LLaVA mainly attends to the text tokens and less to the image tokens,
indicated by lower relevancy scores to the image tokens relatively to
relevancy scores to the input text tokens. In these cases, the model becomes
more susceptible to manipulation, in some cases altering its responses based
on the query with low regard to the image content. This phenomenon is
exemplified by the truck scenario depicted in Figure 5. Conversely, when the
generated outputs exhibit a greater relevance to image tokens than to input
text, the accuracy of LLaVA appears to remain unaffected by how the question
is phrased, as illustrated in Figure 6.

Figure 6: Example where LLaVA demonstrates visual consistency with high
relevancy scores for correct output tokens. With a constant query, ‘What color
is the man’s shirt?’, manually altering the shirt’s color in the image
(purple, green) changes the model’s answer which align with image changes.
Relevancy scores highlight stronger connections to image than text tokens,
illustrated by the image relevancy maps (upper row) and a bar plot comparison
of relevancy scores. With a constant image, despite different question
phrasings, the model demonstrates consistency in its answer, underscoring a
strong relevancy of the yellow token to the visual input over the textual
input variations.

##  5 Conclusions and Future Directions

In this paper we presented LVLM-Interpret, an interactive tool for
interpreting responses from large vision-language models. The tool offers a
way to visualize how generated outputs relate to the input image through raw
attention, relevancy maps, and causal interpretation. Through the many
interpretability functions, users can explore the inner mechanisms of LVLMs
and obtain insights on failure cases. Future work can include consolidation of
the multiple methods for a more comprehensive metric to explain the reasoning
behind model responses.

## References

  * [1] Gradio.  https://www.gradio.app/.  Accessed: 2024-03-13.
  * Aflalo et al. [2022] Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal.  Vl-interpret: An interactive visualization tool for interpreting vision-language transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21406–21415, 2022.
  * Baek and Chung [2023] Ji-Won Baek and Kyungyong Chung.  Swin transformer-based object detection model using explainable meta-learning mining.  _Applied Sciences_ , 13(5):3213, 2023.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_ , 2023.
  * Bi et al. [2023] Yuda Bi, Anees Abrol, Zening Fu, and Vince Calhoun.  A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data.  _bioRxiv_ , pages 2023–07, 2023.
  * Chefer et al. [2021a] Hila Chefer, Shir Gur, and Lior Wolf.  Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 397–406, 2021a.
  * Chefer et al. [2021b] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 782–791, 2021b.
  * Cornia et al. [2022] Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara.  Explaining transformer-based image captioning models: An empirical analysis.  _AI Communications_ , 35(2):111–129, 2022.
  * Dong et al. [2022] Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo.  Towards class interpretable vision transformer with multi-class-tokens.  In _Chinese Conference on Pattern Recognition and Computer Vision (PRCV)_ , pages 609–622. Springer, 2022.
  * Elguendouze et al. [2023] Sofiane Elguendouze, Adel Hafiane, Marcilio CP de Souto, and Anaïs Halftermeyer.  Explainability in image captioning based on the latent space.  _Neurocomputing_ , 546:126319, 2023.
  * He et al. [2023] Jingzhen He, Junxia Wang, Zeyu Han, Jun Ma, Chongjing Wang, and Meng Qi.  An interpretable transformer network for the retinal disease classification using optical coherence tomography.  _Scientific Reports_ , 13(1):3637, 2023.
  * Ji et al. [2023] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38, 2023.
  * Komorowski et al. [2023] Piotr Komorowski, Hubert Baniecki, and Przemyslaw Biecek.  Towards evaluating explanations of vision transformers for medical imaging.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 3726–3732, 2023.
  * Liu et al. [2024] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Liu et al. [2023] Zhuang Liu, Yunpu Ma, Matthias Schubert, Yuanxin Ouyang, Wenge Rong, and Zhang Xiong.  Multimodal contrastive transformer for explainable recommendation.  _IEEE Transactions on Computational Social Systems_ , 2023.
  * Lyu et al. [2022] Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Dime: Fine-grained interpretations of multimodal models via disentangled local explanations.  In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 455–467, 2022.
  * Ma et al. [2023] Jie Ma, Yalong Bai, Bineng Zhong, Wei Zhang, Ting Yao, and Tao Mei.  Visualizing and understanding patch interactions in vision transformer.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–10, 2023.
  * Mallick et al. [2024] Rupayan Mallick, Jenny Benois-Pineau, and Akka Zemmari.  Ifi: Interpreting for improving: A multimodal transformer with an interpretability technique for recognition of risk events.  In _International Conference on Multimedia Modeling_ , pages 117–131. Springer, 2024.
  * Mannix and Bondell [2024] Evelyn Mannix and Howard Bondell.  Scalable and robust transformer decoders for interpretable image classification with foundation models.  _arXiv preprint arXiv:2403.04125_ , 2024.
  * Mondal et al. [2022] Arnab Kumar Mondal, Arnab Bhattacharjee, Parag Singla, and A. P. Prathosh.  xvitcos: Explainable vision transformer based covid-19 screening using radiography.  _IEEE Journal of Translational Engineering in Health and Medicine_ , 10:1–10, 2022.
  * Nalmpantis et al. [2023] Angelos Nalmpantis, Apostolos Panagiotopoulos, John Gkountouras, Konstantinos Papakostas, and Wilker Aziz.  Vision diffmask: Faithful interpretation of vision transformers with differentiable patch masking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 3756–3763, 2023.
  * Naseem et al. [2022] Usman Naseem, Matloob Khushi, and Jinman Kim.  Vision-language transformer for interpretable pathology visual question answering.  _IEEE Journal of Biomedical and Health Informatics_ , 27(4):1681–1690, 2022.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.
  * OpenAi [2023] OpenAi.  Gpt-4v(ision) system card.  2023\.
  * Pan et al. [2021] Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva.  IA-RED2 : Interpretability-aware redundancy reduction for vision transformers.  _Advances in Neural Information Processing Systems_ , 34:24898–24911, 2021.
  * Playout et al. [2022] Clément Playout, Renaud Duval, Marie Carole Boucher, and Farida Cheriet.  Focused attention in transformers for interpretable classification of retinal images.  _Medical Image Analysis_ , 82:102608, 2022.
  * Qiang et al. [2023] Yao Qiang, Chengyin Li, Prashant Khanduri, and Dongxiao Zhu.  Interpretability-aware vision transformer.  _arXiv preprint arXiv:2309.08035_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh and Koh [2022] Krithik Ramesh and Yun Sing Koh.  Investigation of explainability techniques for multimodal transformers.  In _Australasian Conference on Data Mining_ , pages 90–98. Springer, 2022.
  * Richardson and Spirtes [2002] Thomas Richardson and Peter Spirtes.  Ancestral graph markov models.  _The Annals of Statistics_ , 30(4):962–1030, 2002\.
  * Rigotti et al. [2022] Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton.  Attention-based interpretability with concept transformers.  In _International Conference on Learning Representations_ , 2022.
  * Rohekar et al. [2021] Raanan Y Rohekar, Shami Nisimov, Yaniv Gurwicz, and Gal Novik.  Iterative causal discovery in the possible presence of latent confounders and selection bias.  _Advances in Neural Information Processing Systems_ , 34:2454–2465, 2021.
  * Rohekar et al. [2024] Raanan Y Rohekar, Yaniv Gurwicz, and Shami Nisimov.  Causal interpretation of self-attention in pre-trained transformers.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Shome et al. [2021] Debaditya Shome, Tejaswini Kar, Sachi Nandan Mohanty, Prayag Tiwari, Khan Muhammad, Abdullah AlTameem, Yazhou Zhang, and Abdul Khader Jilani Saudagar.  Covid-transformer: Interpretable covid-19 detection using vision transformer for healthcare.  _International Journal of Environmental Research and Public Health_ , 18(21):11086, 2021.
  * Sun et al. [2022] Jiamei Sun, Sebastian Lapuschkin, Wojciech Samek, and Alexander Binder.  Explain and improve: Lrp-inference fine-tuning for image captioning models.  _Information Fusion_ , 77:233–246, 2022.
  * Swamy et al. [2024] Vinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, and Mary-Anne Hartley.  Multimodn—multimodal, multi-task, interpretable modular networks.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Tong et al. [2024] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  _arXiv preprint arXiv:2401.06209_ , 2024.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al.  Evaluation and analysis of hallucination in large vision-language models.  _arXiv preprint arXiv:2308.15126_ , 2023.
  * Wexler et al. [2019] James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viégas, and Jimbo Wilson.  The what-if tool: Interactive probing of machine learning models.  _IEEE transactions on visualization and computer graphics_ , 26(1):56–65, 2019.
  * Wu and Song [2019] Tianfu Wu and Xi Song.  Towards interpretable object detection by unfolding latent structures.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6033–6043, 2019.
  * Xu et al. [2024] Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli.  Hallucination is inevitable: An innate limitation of large language models.  _arXiv preprint arXiv:2401.11817_ , 2024.
  * Xue et al. [2022] Mengqi Xue, Qihan Huang, Haofei Zhang, Lechao Cheng, Jie Song, Minghui Wu, and Mingli Song.  Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition.  _arXiv preprint arXiv:2208.10431_ , 2022.
  * Zhang [2008] Jiji Zhang.  On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias.  _Artificial Intelligence_ , 172(16-17):1873–1896, 2008.
  * Zhang et al. [2018] Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Zhu.  Interpreting cnn knowledge via an explanatory graph.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2018.
  * Zhang et al. [2019] Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu.  Interpreting cnns via decision trees.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6261–6270, 2019.
  * Zhou et al. [2024] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao.  Analyzing and mitigating object hallucination in large vision-language models.  In _The Twelfth International Conference on Learning Representations_ , 2024.


# Impact of medium temperature heat treatment on flux trapping sensitivity in
SRF cavities††thanks: The work is partially supported by the U.S.Department of
Energy, Office of Science, Office of High Energy Physics under Awards No. DE-
SC 0009960 and The manuscript has been authored by by Jefferson Science
Associates, LLC under U.S. DOE Contract No. DE-AC05-06OR23177.

P. Dhakal1,2<EMAIL_ADDRESS>B. D. Khanal2 E. Lechner1 and G. Ciovati1,2
1Thomas Jefferson National Accelerator Facility  Newport News  VA 23606  USA
2Center for Accelerator Science  Department of Physics
Old Dominion University  Norfolk  VA 23529  USA

###### Abstract

The effect of mid-T heat treatment on flux trapping sensitivity was measured
on several 1.3 GHz single cell cavities subjected to vacuum annealing at
temperature of 150 - 400 ∘C for a duration of 3 hours. The cavity was cooldown
with residual magnetic field $\sim$0 and $\sim$20 mG in the Dewar with
cooldown condition of full flux trapping. The quality factor as a function of
accelerating gradient was measured. The results show the correlation between
the treatment temperature, quality factor, and sensitivity to flux trapping.
Sensitivity increases with increasing heat treatment temperatures within the
range of (200 - 325 ∘C/3h). Moreover, variations in the effective penetration
depth of the magnetic field and the density of quasi-particles can occur,
influencing alterations in the cavity’s electromagnetic response and resonance
frequency.

##  1 Introduction

The current leading technique in processing superconducting niobium surfaces
for SRF cavities involves surface engineering through impurity alloying via
thermal treatment. This method, considered state-of-the-art, takes advantage
of the introduction of impurities from external sources such as titanium or
nitrogen, or the presence of native niobium oxides within the rf penetration
depth [1, 2, 3, 4, 5, 6]. However, impurities play a crucial role in
increasing the density of pinning centers, enhancing susceptibility to flux
trapping during cavity cooldown due to an incomplete Meissner effect. To
optimize the process, it is essential to minimize flux trapping sensitivity,
resulting in a higher quality factor and an increased accelerator gradient.

A recent advancement in cavity heat treatment, known as "mid-T" baking,
capitalizes on the naturally formed oxide on the surface of the Nb cavity [4,
5, 6, 7, 8]. This oxide is then dissolved in the bulk material. The diffusion
of oxygen is temperature and time-dependent, influencing the concentration of
interstitial oxygen. Developing processes with lower flux trapping sensitivity
is paramount for achieving higher quality factors and accelerator gradients.
In this manuscript, we present the results of the several SRF cavity tests
subjected to mid-T heat treatment in the furnace. The quality factor as a
function of accelerating gradient as well as the flux trapping sensitivity was
measured. Additionally, the change in resonant frequency near the critical
temperature $T_{C}$ was measured to confirm the change in electronic
properties as a result of heat treatment.

##  2 Experimental setup

Two 1.3 GHz TESLA shaped single cell cavities, namely TE1-05 and TE1-06 were
used for this current study. The cavities were previously heat treated at
higher temperature (1000 ∘C) followed by 25 $\mu$m electropolishing to ensure
that the cavity provides good flux expulsion [9]. Before mid-T heat treatment,
the cavities were high pressure rinsed with de-ionized water, dried in a class
10 clean room, installed clean Nb foils over the beamtube flanges. The furnace
temperature was increased at a rate of $\sim 2-5\leavevmode\nobreak\ ^{\circ}$
C / min until it reached the target temperature. The furnace was held at the
desired temperature for 3 hours and cooled to room temperature. The cavities
were subjected to standard cavity processing techniques of high pressure rinse
and assembly. Three flux gate magnetometers were taped at the equator 120∘
apart each and parallel to the cavity axis. The cooldown of cavity was done by
maintaining residual magnetic flux density of $\sim$0 and $\sim$20 mG in the
Dewar with cooldown condition of full flux trapping. The RF measurements
consist of $Q_{0}(T)$ at low peak rf field $B_{p}\sim 15\leavevmode\nobreak\
mT$ and $Q_{0}(B_{p})$ at 2.0 K. Furthermore, the resonant frequency and
quality factor were measured during the cavity warm up through $T_{c}$. After
each rf test, the cavity’s rf surface is reset by $\sim$25 $\mu$m
electropolishing, before next heat treatment.

##  3 Experimental Results

###  3.1 rf results

The $R_{s}(T)$ data were fitted using the model presented in Ref. [10] to
extract the temperature-dependent $R_{BCS}$ and temperature-independent
residual resistance $R_{i}$. The flux trapping sensitivity is calculated
using,

$S=\frac{R_{i,B_{2}}-R_{i,B_{1}}}{B_{2}-B_{1}}.$ (1)

where $B_{1}\approx$ 0 mG and $B_{2}\approx$ 20 mG. Figure 1 shows the flux
trapping sensitivity as a function of heat treatment temperature. Sensitivity
increases with the increase in the temperature of heat treatment with the
maximum at $\sim$ 250 ∘ / 3h. On further increase in temperature the flux
trapping sensitivity decreases gradually. The flux trapping sensitivity
follows the oxygen concentration within 100 nm calculated using oxygen
diffusion model [11, 12].

Figure 1: Flux trapping sensitivity as a function of heat treatment
temperature. The solid line is calculated oxygen concentration within first
100 nm using oxygen diffusion model [12].

Figure 2 shows the $Q_{0}$ vs. $E_{acc}$ at 2.0 K for both cavities. The
baseline cavity was limited to $\sim$ 30 MV/m with a high field Q-slope. In
the same cavity, when subjected to 150 ∘C/3 h of heat treatment, the cavity
gradient reached to $\sim$ 44 MV/m before it quenched. Eliminating the high-
field Q-slope with low-temperature baking has been a proven technique to
improve the accelerating gradient [13, 14]. With a further increase in the
temperature of the heat treatment, the quality factor increases with
increasing accelerating gradient with $Q_{0}$ reaching $\sim 4\times 10^{10}$
at $E_{acc}\sim 20$ MV/m for 275 - 325 ∘ C/3h with a lower quench gradient
($\sim 25$ MV / m) compared to the baseline test. On further increase in heat
treatment temperature, the overall $Q_{0}$ gets lower while the high field
Q-slope starting at $\sim$ 25 MV/m.

Figure 2: $Q_{0}$ vs. $E_{acc}$ at 2.0 K with < 2 mG residual field in Dewar.

For all tests, the measurements were repeated with $\sim$ 20 mG of residual
magnetic flux density trapped during the cavity cooldown. The accelerating
gradient dependence of flux trapping sensitivity was calculated using

$S=\frac{R_{s,B_{2}}-R_{s,B_{1}}}{B_{2}-B_{1}}.$ (2)

where $R_{s}=G/Q_{0}$, $B_{1}\approx$ 0 mG and $B_{2}\approx$ 20 mG. Figure 3
shows the flux trapping sensitivity as a function of accelerating gradient. As
shown in Fig. 1, the flux trapping sensitivity increase with the increase in
heat treatment temperature with highest sensitivity for 250 ∘C/3h heat
treatment. On further increase in temperature, the gradient dependence of
sensitivity decreases.

Figure 3: Flux trapping sensitivity as a function of $E_{acc}$ at 2.0 K.

###  3.2 $f$ and $Q_{0}$ during warm up

After the completion of the rf test, the cavity was connected to a network
analyzer to measure the resonant frequency and quality factor as a function of
temperature. To correct the Dewar pressure fluctuation, the pressure
sensitivity was corrected by tracking the frequency with respect to the Dewar
pressure at a given temperature. All data were corrected for at 1 atm (760
Torr) pressure. Figure 4 shows the change in frequency $\Delta f=f-f_{6K}$ and
$R_{s}=G/Q_{0}$ as a function of temperature. The change in $\Delta f$ and
$R_{s}$ is clearly seen as a result of heat treatments. The baseline EP cavity
(TE1-05) does not show any dip near $T_{c}$, whereas both the $\Delta f$ and
depth of the dip increase as the heat treatment temperature increases up to
$\sim$ 275 ∘C/3h. On further increase in temperature, both $\Delta f$ and
depth of the dip decrease, as shown in Fig.4. The effect of the current heat
treatment process not only changes the frequency behavior, but also has an
effect on $R_{s}$ and $T_{c}$, as shown in Fig.5.

Figure 4: Change in resonant frequency and surface resistance as a function of
temperature. Figure 5: (a) Superconducting transition temperature ($T_{c}$),
(b) surface resistance, $R_{n}$ at 10 K, and (c) frequency shift ($\Delta
f=f_{6\leavevmode\nobreak\ K}-f_{10\leavevmode\nobreak\ K}$) as a function of
heat treatment temperature. The dotted line in (a) is calculated $T_{c}$ due
to solute solution of Nb-O [16] within the first 100 nm using the oxygen
diffusion model [12].

##  4 Discussion and Summary

The enhancement of the quality factor through Q-rise is evident following heat
treatments within the temperature range of 200 to 400 ∘C over a duration of 3
hours. In particular, a higher $Q_{0}$ manifests itself when the electronic
mean free path is comparable to the superconducting coherence length. The
introduction of oxygen reduces the electron mean free path thereby increasing
the penetration depth, but decreasing the dissipative conductivity. The
results shown here are in qualitative agreement with the a recent
characterization of the oxygen diffusion process [6] based on Ciovati’s
model[11]. It is interesting to note that the baking around 150 ∘C/3hr
ameliorated the high field Q slope. In the context of shallow impurity
diffusion profiles, where the peak supercurrent density is deformed and pushed
away from the surface, this result is in qualitative agreement with the
expected optimal baking time and temperature [12]. It should be emphasized
that cavities treated within the temperature range of 250 to 325 ∘C/3h exhibit
a tendency to quench at lower gradients (20 - 25 MV/m) compared to cavities
treated outside of this temperature range. Furthermore, several instances of
strong multipacting were observed at this gradient range in contrast to the
baseline EP’ed cavities, suggesting a possible surface modification due to
heat treatment, which causes a change in secondary electron emission yield.

As anticipated, heat treatment causes the change in local mean free path as
well as spatial distributions of impurities, oxides, dislocation networks
[15]. The sensitivity to flux trapping increases with rising heat treatment
temperatures within the range of (200 - 250 ∘C/3h). It is possible that within
this temperature range, surface oxide fails to fully dissolve, resulting in an
accumulation of higher oxide nanoprecipates near the surface, thus promoting a
higher density but weak pinning centers. However, when the temperature exceeds
250 ∘C for the same duration, the breakdown of surface oxides and subsequent
diffusion into the bulk decrease the pinning centers on the surface.
Consequently, the sensitivity to flux trapping gradually diminishes with
increasing heat treatment temperatures, as shown in Figs. 1 and 3. The
observation is further corroborated by the oxygen concentration profile as
shown in Fig. 1 with peak oxygen concentration at $\sim$ 250 ∘C/3h heat
treatment.

The change in transition temperature as shown in Fig. 5 is qualitatively in
agreement with the literature values of reduction in $T_{c}$ for solute
solution of oxygen in niobium [16]. The redistribution of the oxide on the Nb
surface due to heat treatment introduce an additional scattering mechanisms
and modify the electronic characteristics of the superconducting material in
proximity to the transition temperature. Consequently, variations in the
effective penetration depth of the magnetic field and the density of quasi-
particles can occur, influencing alterations in the cavity’s electromagnetic
response and resonance frequency [17].

##  5 ACKNOWLEDGMENTS

We would like to acknowledge Jefferson Lab SRF production technical staff
members for the cavity fabrication, chemical processing, heat treatment, clean
room assembly, and cryogenic support.

## References

  * [1] P. Dhakal et al., "Effect of high-temperature heat treatments on the quality factor of a large-grain superconducting radio-frequency niobium cavity", Phys. Rev. ST Accel. Beams 16, 042001 (2013). doi:10.1103/PhysRevSTAB.16.042001
  * [2] A. Grassellino et al., “Unprecedented quality factors at accelerating gradients up to 45 MV$m^{-1}$ in niobium superconducting resonators via low temperature nitrogen infusion,” Supercond. Sci. Technol., 30, 094004 (2017). doi:10.1088/1361-6668/aa7afe
  * [3] P. Dhakal, "Nitrogen doping and infusion in SRF cavities: A review", Physics Open, 5, 100034 (2020). doi:10.1016/j.physo.2020.100034
  * [4] S. Posen et al., "Ultralow surface resistance via vacuum heat treatment of superconducting radio-frequency cavities", Phys. Rev. Applied 13, 014024 (2020).doi:10.1103/PhysRevApplied.13.014024
  * [5] H. Ito, H. Araki, K.Takahashi, and K. Umemori, "Influence of furnace baking on Q–E behavior of superconducting accelerating cavities", Prog. Theor. Exp. Phys., 071G01 (2021). doi:10.1093/ptep/ptab056
  * [6] E. M. Lechner et al., "RF surface resistance tuning of superconducting niobium via thermal diffusion of native oxide", Appl. Phys. Lett. 119, 082601 (2021). doi:10.1063/5.0059464
  * [7] F. He et al., "Medium-temperature furnace baking of 1.3 GHz 9-cell superconducting cavities at IHEP", Supercond. Sci. Technol. 34, 095005 (2021). doi:10.1088/1361-6668/ac1657
  * [8] Z. Yang et al., "Surface resistance effects of medium temperature baking of buffered chemical polished 1.3 GHz nine-cell large-grain cavities", Supercond. Sci. Technol., 36, 015001 (2023).doi:10.1088/1361-6668/aca12a
  * [9] B. D. Khanal and P. Dhakal, “Evaluation of Flux Expulsion and Flux Trapping Sensitivity of SRF Cavities Fabricated from Cold Work Nb Sheet with Successive Heat Treatment”, in Proc. 21th Int. Conf. RF Supercond. (SRF’23), Grand Rapids, MI, USA, Jun. 2023, pp. 197-201. doi:10.18429/JACoW-SRF2023-MOPMB042
  * [10] G. Ciovati, P. Dhakal, and A. Gurevich, "Decrease of the surface resistance in superconducting niobium resonator cavities by the microwave field", Appl. Phys. Lett. 104 (9), 092601 (2014). doi:10.1063/1.4867339
  * [11] G. Ciovati, "Improved oxygen diffusion model to explain the effect of low-temperature baking on high field losses in niobium superconducting cavities", Appl. Phys. Lett. 89, 022507 (2006). doi:10.1063/1.2220059
  * [12] E. M. Lechner et al., "Oxide dissolution and oxygen diffusion scenarios in niobium and implications on the Bean–Livingston barrier in superconducting cavities", J. Appl. Phys. 135, 133902 (2024). doi:10.1063/5.0191234
  * [13] G. Ciovati, "Effect of low-temperature baking on the radio frequency properties of niobium superconducting cavities for particle accelerators", J. Appl. Phys., 96, 1591 (2004). doi:10.1063/1.1767295
  * [14] B. D. Khanal and P. Dhakal, "Insight to the duration of 120 ∘C baking on the performance of SRF niobium cavities," in IEEE Transactions on Applied Superconductivity, 33(3), 1-6 (2023). doi:10.1109/TASC.2023.3235311
  * [15] P. Dhakal, G. Ciovati, and A. Gurevich. “Flux expulsion in niobium superconducting radio-frequency cavities of different purity and essential contributions to the flux sensitivity”, Phys. Rev. Accel. Beams, 23, 023102 (2020). doi:10.1103/PhysRevAccelBeams.23.023102
  * [16] W. Desorbo, "Effect of dissolved gases on some superconducting properties of niobium", Phys. Rev., 132, 107 (1963. doi:10.1103/PhysRev.132.107
  * [17] M. Zarea, H. Ueki, and J. A. Sauls, "Electromagnetic response of disordered superconducting cavities" , Front. Electron. Mater. 3,1259401 (2023). doi:10.3389/femat.2023.1259401.


# Tilted circumbinary planetary systems as efficient progenitors of free-
floating planets

Cheng Chen School of Physics and Astronomy, University of Leeds
Sir William Henry Bragg Building, Woodhouse Ln., Leeds LS2 9JT, UK  Rebecca G.
Martin Department of Physics and Astronomy, University of Nevada, Las Vegas
4505 South Maryland Parkway, Las Vegas, NV 89154, USA  Nevada Center for
Astrophysics, University of Nevada, Las Vegas
4505 South Maryland Parkway, Las Vegas, NV 89154, USA  Stephen H. Lubow Space
Telescope Science Institute
3700 San Martin Drive, Baltimore, MD 21218, USA  C.J. Nixon School of Physics
and Astronomy, University of Leeds
Sir William Henry Bragg Building, Woodhouse Ln., Leeds LS2 9JT, UK

###### Abstract

The dominant mechanism for generating free-floating planets has so far
remained elusive. One suggested mechanism is that planets are ejected from
planetary systems due to planet-planet interactions. However, instability
around a single star requires a very compactly spaced planetary system. We
find that around binary star systems instability can occur even with widely
separated planets that are on tilted orbits relative to the binary orbit due
to combined effects of planet-binary and planet-planet interactions,
especially if the binary is on an eccentric orbit. We investigate the orbital
stability of planetary systems with various planet masses and architectures.
We find that the stability of the system depends upon the mass of the highest
mass planet. The order of the planets in the system does not significantly
affect stability but, generally, the most massive planet remains stable and
the lower mass planets are ejected. The minimum planet mass required to
trigger the instability is about that of Neptune for a circular orbit binary
and a super-Earth of about $10$ Earth masses for highly eccentric binaries.
Hence, we suggest that planet formation around misaligned binaries can be an
efficient formation mechanism for free-floating planets. While most observed
free-floating planets are giant planets, we predict that there should be more
low-mass free floating planets that are as yet unobserved than higher mass
planets.

Exoplanet dynamics(490) — Exoplanet systems(484) — Three body problem(1695)—
N-body simulations(1083) — Computational astronomy(293) — Binary stars(154)

##  1 Introduction

Gravitational microlensing observations suggest that there could be a
significant population of free-floating planets (FFPs) (e.g. Sumi et al.,
2011; Clanton & Gaudi, 2017). Estimates for the mass of FFPs range from around
$0.25$ (Mróz et al., 2017) up to about $3.5$ (Sumi et al., 2011) Jupiter
masses per main-sequence star. There is an excess of FFPs by a factor of up to
seven compared to that predicted by core-collapse models (Padoan & Nordlund,
2002; Miret-Roig et al., 2021). There are several mechanisms suggested to form
the excess of FFPs. These include planet-planet scattering (Rasio & Ford,
1996; Weidenschilling & Marzari, 1996; Veras & Raymond, 2012), aborted stellar
embryo ejection from a stellar nursery (Reipurth & Clarke, 2001) and photo-
erosion of a pre-stellar core by stellar winds from a nearly OB star
(Whitworth & Zinnecker, 2004). Planet-planet scattering around single stars
cannot explain the large number of FFPs (Veras & Raymond, 2012; Ma et al.,
2016).

Around a single star, planet-planet scattering occurs when the planets form
close to each other. Two planets with masses $m_{\rm p1}$ and $m_{\rm p2}$
that form with semi-major axes $a_{\rm p1}$ and $a_{\rm p2}$, respectively,
around a star with mass $m$ are unstable if $\Delta=(a_{\rm p2}-a_{\rm
p1})/R_{\rm Hill}\lesssim 2\sqrt{3}$ (Marchal & Bozis, 1982; Gladman, 1993;
Chambers et al., 1996), where the mutual Hill sphere radius is given by

$R_{\rm Hill}=\left(\frac{m_{\rm p1}+m_{\rm
p2}}{3\,m}\right)^{1/3}\left(\frac{a_{\rm p1}+a_{\rm p2}}{2}\right).$ (1)

Ejection of giant planets in simulations of planet formation around a single
star is unlikely and most ejected planets have a mass $\lesssim 0.3\,\rm
M_{\oplus}$ (Barclay et al., 2017). This stability criterion is not
significantly affected if a single star is replaced by a coplanar inner
binary, unless the planets are formed very close to the binary (Kratter &
Shannon, 2014). However, the outcome of an unstable system is more likely to
be ejection rather than a collision around a binary star system because of
close encounters with the binary (Smullen et al., 2016; Sutherland & Fabrycky,
2016; Gong, 2017; Gong & Ji, 2017; Fleming et al., 2018). Coplanar planets
around a binary must form relatively close to each other for the system to be
unstable.

Observations of circumbinary disks suggest that misalignment between the
binary orbital plane and the disk may be common (e.g. Chiang & Murray-Clay,
2004; Winn et al., 2004; Capelo et al., 2012; Brinch et al., 2016; Zhu et al.,
2022). Polar aligned disks around eccentric binaries may also be common
(Kennedy et al., 2012, 2019; Kenworthy et al., 2022). Chaotic accretion can
lead to the formation of misaligned circumbinary disks (Clarke & Pringle,
1993; Bate, 2018) and subsequent disk evolution can lead to tilt evolution
towards a coplanar alignment (Papaloizou & Terquem, 1995; Bate et al., 2000;
Lubow & Ogilvie, 2000; Nixon et al., 2011) or a polar alignment (Aly et al.,
2015; Martin & Lubow, 2017; Lubow & Martin, 2018; Zanazzi & Lai, 2017).
However, the alignment timescale for an extended disk may be longer than the
disk lifetime and planetary systems may form in a misaligned disk. While
observations of circumbinary planets (CBPs) currently show only coplanar
systems (e.g. Welsh et al., 2012; Orosz et al., 2012; Kostov et al., 2013;
Standing et al., 2023), misaligned planetary systems may be expected around
binaries with a longer orbital period (Czekala et al., 2019). These planetary
systems may be observed in the future with eclipse-timing variations (Zhang &
Fabrycky, 2019; Martin, 2019).

Contrary to coplanar circumbinary planetary systems, a multi-planet system
with a tilted respect to inner binary can be unstable for a wide range of
initial planet separations (Chen et al., 2023, 2023). A planet orbiting around
an eccentric binary undergoes tilt oscillations as a result of its interaction
with the binary (Verrier & Evans, 2009; Farago & Laskar, 2010; Naoz et al.,
2017; Chen et al., 2019). Planets in a tilted two-CBP system also undergo
mutual tilt oscillations as a result of planet-planet interactions (Chen et
al., 2022). Furthermore, if the mutual tilt between the two planets becomes
large, the outer planet can induce von-Zeipel-Kozai-Lidov (ZKL von Zeipel,
1910; Kozai, 1962; Lidov, 1962) oscillations of the inner planet that lead to
planet eccentricity growth. Mean motion resonances (MMRs) between the planets
are typically stable in a coplanar configuration, but they become unstable in
a tilted planetary system. Therefore the complex dynamics around a tilted
binary can lead to the planetary system instability for a wide range of the
initial planet semi-major axis (Chen et al., 2023).

In this Letter, we consider the stability of tilted planetary systems with
unequal mass planets. We show that the planet with the highest mass dominates
the stability outcome. The order of the orbital radii of the planets does not
change the stability; ejection of the smaller mass planets is the most likely
outcome. We examine the planet mass required for planet-planet interactions to
drive instability. We describe our simulation setup and parameter domains we
explore in Sec. 3. Specifically we consider the masses of the ejected planets.
Finally, a discussion and conclusions follow in Sec. 4 and Sec. 5.

##  2 Simulation set–up and parameter space explored

Table 1: Parameters of the simulations. The first column contains the name of model, the second, third and fourth columns indicate the binary eccentricity and the masses of inner and outer planets.  Model | $e_{\rm b}$ |  $m_{\rm p1}$ ($m_{\rm b}$) |  $m_{\rm p2}$ ($m_{\rm b}$)
---|---|---|---
A1 | 0.0 | 0.000003 | 0.001
A2 | 0.8 | 0.000003 | 0.001
C1 | 0.0 | 0.001 | 0.000003
C2 | 0.8 | 0.001 | 0.000003
E1 | 0.0 | 0.000003 | 0.000003
E2 | 0.8 | 0.000003 | 0.000003
SE1 | 0.0 | 0.00003 | 0.00003
SE2 | 0.8 | 0.00003 | 0.00003
N1 | 0.0 | 0.00005 | 0.00005
N2 | 0.8 | 0.00005 | 0.00005
S1 | 0.0 | 0.0003 | 0.0003
S2 | 0.8 | 0.0003 | 0.0003

To study the orbital stability of planetary systems with unequal mass planets
we carry out simulations with the ${\sc n}$-body package, rebound with a
whfast integrator which is a second order symplectic Wisdom Holman integrator
with 11th order symplectic correctors (Rein & Tamayo, 2015). We solve the
gravitational equations in the frame of the center of mass of the four-body
system. The central binary has components of mass $m_{1}$ and $m_{2}$ with the
total mass $m_{1}+m_{2}=m_{\rm b}$. We consider an equal mass binary with
$m_{1}=m_{2}=0.5\,m_{\rm b}$ so that the mass fraction of the binary $f_{\rm
b}$ = 0.5. The binary can be in a circular or eccentric orbit and the binary
eccentricity is $e_{\rm b}$ and their separation is 1 $a_{\rm b}$.

We consider planetary systems with two planets. We explored the extension to
three planet systems in Chen et al. (2023) where we found that the instability
in the three planet case is qualitatively similar to the two planet case when
the three planets are separated by a fixed number of mutual Hill radii. Three
planet systems are slightly more unstable than two planet systems. The number
of planets that are ejected in unstable systems depends upon how close the
system is to the binary. Close-in systems are more likely to eject more
planets. The planetary system arrangements for each simulation are shown in
Table LABEL:table1.

The two planets are initially on Keplerian orbits around the center of mass of
the binary. Their orbits are defined by six orbital elements: the semi-major
axes $a_{\rm p1}$ and $a_{\rm p2}$, inclinations $i_{\rm p1}$ and $i_{\rm p2}$
which is relative to the binary orbital plane, eccentricities $e_{\rm p1}$,
and $e_{\rm p2}$, longitudes of the ascending nodes $\phi_{\rm p1}$ and
$\phi_{\rm p2}$ measured from the binary semi–major axis, arguments of
periapsides $\omega_{\rm p1}$, and $\omega_{\rm p2}$, and true anomalies
$\nu_{\rm p1}$ and $\nu_{\rm p2}$. The initial orbits of two planets are
coplanar to each other and circular so initially $e_{\rm p}=0$, $\omega_{\rm
p}=0$ and $\nu_{\rm p}=0$ and we set $\phi_{\rm p}=90^{\circ}$ for all
planets. We integrate the simulations for a total time of 14 million binary
orbital periods ($T_{\rm b}$) with a timestep of integration of 0.7$\%$ of the
initial orbital period of the inner planet.

We describe the criteria for determining an unstable orbit of the planet based
on three distinct conditions. First, if the eccentricity of the planet’s orbit
$e_{\rm p}\geq$ 1.0, the planet is deemed unbound from the binary system.
Secondly, an orbit is unstable if the semi-major axis of the planet’s orbit
$a_{\rm p}>1000\,a_{\rm b}$, indicating that the planet has moved excessively
far away from the central system. Thirdly, if the semi-major axis of the
planet’s orbit becomes smaller than that of the binary’s orbit, $a_{\rm
p}<a_{\rm b}$, the planet can no longer be considered a CBP. These stability
criteria are established in line with existing research (e.g. Chen et al.,
2020; Quarles et al., 2020; Chen et al., 2023, 2023).

To understand the dynamic interaction between two planets, the inner planet is
placed at $a_{\rm p1}=5\,a_{\rm b}$ where it should be stable for a single CBP
(Chen et al., 2020). We vary the semi-major axis of the outer planet by
varying the ratio of their mutual Hill radius between the inner planet,
$\Delta$ and thus, we have

$a_{\rm p2}=a_{\rm p1}+\Delta\left(\frac{m_{\rm p1}+m_{\rm p2}}{3\,m_{\rm
b}}\right)^{1/3}\left(\frac{a_{\rm p1}+a_{\rm p2}}{2}\right).$ (2)

and $\Delta$ ranges from 3.4 $\left(\approx 2\sqrt{3}\right)$ to 12.0 with an
interval of $\Delta=0.1$.

##  3 Simulation results

In this Section we describe the results of our simulations. We first consider
the effect of having an unequal mass planetary system, then the order of the
planets in separation from the binary and finally the planet mass required for
instability.

###  3.1 The effect of unequal mass planetary systems

Fig. 1 shows stability maps in which the $x$-axis is the initial inclination
of the planets with respect to the binary with an interval of 2.5∘ and the
$y$-axis is the initial separation between two planets in units of $R_{\rm
Hill}$. The colors of the pixels show the planets’ orbital status at the end
of simulations. The blue pixels represent systems in which the two planets are
stable. The red pixels represent systems in which only the outer planet is
stable and the green pixels represent systems in which only the inner planet
is stable. White pixels represent systems in which the two planets are
unstable. The cyan, orange and pink horizontal dashed lines indicate the 2:1,
5:2 and 3:1 MMRs between the two planets. Note that the location of the MMRs
are shifted relative to $\Delta$ compared to the equal Jupiter-mass planet
cases in Chen et al. (2023). The total mass of the two CBPs in this study is
smaller than two Jupiter-mass CBPs, so their separation in terms of $R_{\rm
Hill}$ is smaller for a fixed $\Delta$.

The upper panels of Fig. 1 show stability maps for models A1 and A2 that are
planetary systems with an inner Earth-mass planet and an outer Jupiter mass
planet with binary eccentricity $e_{\rm b}$ = 0.0 (left) and 0.8 (right). In
model A1 with a circular orbit binary, if the two planets are near aligned to
the binary orbital plane (in prograde or retrograde orbits), they can be
stable even if they have vary small separation ($\Delta\geq 3.5$). However,
with higher inclination orbits, the two planets are unlikely to be stable
until $\Delta\geq 5.0$. Above this region, there are unstable regions around
the 2:1 and 3:1 MMRs The effect of the 5:2 MMR is minor and there are sporadic
unstable orbits around it. The only unstable outcome for a system with an
inner Earth-mass and an outer Jupiter-mass planet is that the Earth mass
planet gets ejected.

For model A2 with an eccentric binary, there are more inner planet unstable
orbits within $\Delta\leq 5.0$ showing that misaligned planets are not likely
to be stable within the 2:1 MMR region except around the polar orbit ($i_{\rm
p}$ around 90∘). All of the unstable cases are the outer planet stable cases
meaning that the inner Earth-mass planet is ejected. Moreover, the unstable
regions around 2:1 and 3:1 MMRs become larger than those of in model A1. Two
planets are unlikely to be stable even when they are just slightly misaligned
to the binary. We also increased the mass of the inner planet to one Saturn-
mass and found most of the unstable cases again involve the ejection of the
lower mass inner planet.

These two panels can be compared with with upper-left and lower-right panels
of Figure 2 in Chen et al. (2023) which show the same maps but for two
Jupiter-mass CBPs. The units of the $y$-axis are different, but otherwise the
stability map is very similar except that in the unequal mass system shown
here, only the outer (and more massive) planet survives in all unstable cases.
The two Jupiter-mass CBPs systems have a small fraction of systems in which
both planets are ejected, however, with the Earth mass planet here, there are
no systems in which both planets become ejected. Given the similarity between
stability maps with two Jupiter mass planets and one Jupiter and one Earth,
this suggests that the mass of the most massive planet in the system is what
determines the stability of the system.

###  3.2 The effect of the planetary system order

We now change the orbital arrangement of the two planets. The lower panels of
Fig. 1 show models C1 and C2 that have the same setup as models A1 and A2
except the inner planet is the Jupiter-mass planet and the outer planet is the
Earth-mass planet. The stability map is not significantly affected by changing
the order of the planets except that the outcome is different. The lower mass
planet is ejected no matter where it begins and the Jupiter mass planet
remains stable. The system is slightly more stable with the Jupiter mass
planet being the inner planet. This is because an Earth mass planet that is
interior to the more massive outer planet is more likely to undergo ZKL
oscillations, resulting in increased eccentricity, interaction with the
binary, and ejection when it is closer to the binary.

###  3.3 The minimum mass to drive the instability

Now we consider the minimum mass of the planet required to drive the planet-
planet interactions for instability. If the effect of planet-planet
interactions is small, each CBP only interacts with the binary so than their
orbital stability should comply with the three-body stability map (e.g. Doolin
& Blundell, 2011; Quarles et al., 2018; Chen et al., 2020). We have shown that
the most massive planet in the system dominates the stability already so we
now consider systems of equal masses but we vary the mass. We consider two
CBPs of Earth-mass (models E1 and E2), 10 Earth-mass (models SE1 and SE2),
Neptune-mass (models N1 and N2) and Saturn-mass (models S1 and S2).

The left panels of Fig. 2 show the effect of increasing the planet masses
around a circular orbit binary. Two Earth-mass CBPs (model E1) are very
stable. With increasing masses of two planets, the number of unstable cases
increases. For model S1 in which the planets are Saturn-mass, the stability
map is very similar to the map of A1 in Fig. 1 with Jupiter mass planets.
Around a circular orbit binary, instability requires a planet mass greater
than about Neptune’s mass to drive planetary system instability.

The right panels of Fig. 2 show the same simulations with increasing planet
masses but around an eccentric orbit binary with $e_{\rm b}=0.8$. The Earth
mass planets are still stable but there some unstable cases at separations
closer than than the 2:1 MMR. For the super-Earth mass planets, more than half
of the pixels are unstable for separations closer than the 2:1 MMR in models
SE2 and N2. For model S2 which planets are Saturn-mass, the map is very
similar to map of A2 in Fig. 1 and two planets can be only stable in near
coplanar or polar orbits within the 2:1 MMR. Around a highly eccentric orbit
binary, instability requires smaller mass planets than around circular orbit
binaries. Planet-planet interactions drive instability for planet masses
greater than super-Earth masses for these cases which have $a_{\rm p1}=5a_{\rm
b}$.

Figure 1: Stability maps of a two planet circumbinary system with $e_{\rm
b}=0.0$ (left panels), 0.8 (right panels). The planetary system consists of
one Earth mass planet and one Jupiter mass planet. The Jupiter-mass planet is
the outer planet on upper panels and is the inner planet on lower panels. The
inner planet has initial semi-major axis $a_{\rm p1}=5\,a_{\rm b}$ and the
$y$-axis is $\Delta$ with an interval of $0.1$ while the $x$-axis is the
initial inclination of two planets with an interval of $2.5^{\circ}$. The four
horizontal dashed lines represent the 2:1 (cyan), 5:2 (orange), 3:1 (pink) and
4:1 (red) MMRs between the two planets. The four different colors of pixels
represent two planets stable cases (blue), the inner planet survived cases
(green), the outer planet survived cases (red) and both planet unstable cases
(white).

Figure 2: Stability maps for planetary systems with two equal-mass
circumbinary planets around a binary system with $e_{\rm b}=0.0$ (left
panels), 0.8 (right panels). The inner planet has initial semi-major axis
$a_{\rm p1}=5\,a_{\rm b}$. The pixel colors are as described in Fig. 1. From
upper to lower panels, the two planets are Earth-mass, Super-Earth mass (10
times Earth-mass), Neptune-mass and Saturn-mass.

##  4 Discussion

Planet-planet scattering, gravitational instabilities in the outer disk and
post-formation evolution around single star systems may contribute some
population of the observed FFPs, but it is difficult to explain the large
populations that we have observed (Boss, 2011; Veras & Raymond, 2012).
Recently, some N-body simulations have confirmed this point. For example,
simulations with Jupiter and Saturn mass planets around a solar-type Star in
Barclay et al. (2017) show that approximately one-third of the initial disk
mass is ejected ($\sim$ 5 Earth-mass) in the late stage of planet formation
but the masses of the individual ejected bodies are $\lesssim 0.3$ Earth-mass.
Simulations in Pfyffer et al. (2015) found that the median mass of an ejected
planet is $\leq$ 2 Earth-mass from post-formation evolution. We also
investigated multi-planetary single star systems with two Jupiter-mass planets
and found that they are very stable unless they are extremely compactly spaced
with $\Delta\leq 2.3$ (see Figure 1 of Chen et al., 2023).

On the other hand, binary stars are common in the Universe, and multiplicity
statistics show that about 40$\%$ to 60$\%$ of M to F stars are in binary
systems. Approximately half of these binaries have separations of less than 20
au, indicating that they may have CBDs orbiting around them during the star
formation (see Figure 12 and 13 in Raghavan et al., 2010). Since CBDs have a
wide distribution of inclinations if the binary period $T_{\rm b}$ exceeds if
$T_{\rm b}>30$ days (Czekala et al., 2019) due to the broad distribution of
binary eccentricities $e_{\rm b}$ (see Figure 14 in Raghavan et al., 2010),
planets could form inside these misaligned disks. Furthermore, the detection
of disk substructures such as rings and gaps in the HL Tau disk (ALMA
Partnership et al., 2015; Yen et al., 2016, 2019) implies that early planet
formation can occur in disks (cf. Nixon et al., 2018). Several planet-induced
structures can be explained by numerical simulations, although the topic is
still under debate (e.g. Dong et al., 2015; Flock et al., 2015; Zhang et al.,
2015, 2018). Hence, planet formation could be very efficient, and we expect
that multiple planets could also form in the CBD around the binary.

If planet formation occurs while the disk is young, it is likely that multiple
planets form within the disk that can either decouple from the disk (e.g.
Martin et al., 2016; Franchini et al., 2020) and be ejected or be ejected once
the disk has dispersed (cf. Turrini et al., 2020). Recent observations show
that 93$\%$ of exoplanet pairs are at least 10$\,R_{\rm Hill}$ apart (Weiss et
al., 2018) and CBPs pairs may have a similar distribution due to instabilities
within the 3:1 MMR. A binary with a lower eccentricity has a higher chance to
host a compact CBP pair while only a compact Earth-mass CBP pair could be
stable around a binary with high eccentricity. Consequently, FFPs ranging from
low to high mass can be generated from the circumbinary planetary systems.

In all of our simulations we have chosen the inner planet semi-major axis to
be $a_{\rm p1}=5.0a_{\rm b}$ since this is close to the inner stability limit
for inclined orbits of all inclinations. According to Figure 3 and 5 in Chen
et al. (2023), the instabilities are stronger for misaligned CBPs if the inner
planet is at 10.0 or 20.0 $a_{\rm b}$ due to the ZKL-like oscillations.
However, if the inner planet is much farther from the binary, then the system
behaves as though the binary is a single star. The only instability is then
due to planet-planet interactions and the system becomes more stable.

From Transiting Exoplanet Survey Satellite (TESS) data, the giant planet (0.6
– 2.0$R_{\rm J}$) occurrence rate is about 0.194$\pm$ 0.072 per cent for low
mass stars (0.088–0.7$M_{\odot}$) (Bryant et al., 2023) and about 7.8 percent
for giant stars from the Pan-Pacific Planet Search (Wittenmyer et al., 2020).
On the other hand, the occurrence rate of small planets ranging from (0.5–4.0
$R_{\oplus}$) with orbital periods shorter than 50 days is about 90 percent
from the Kepler Input Catalog (Dressing & Charbonneau, 2013). If these results
are similar to the planet occurrence rates in circumbinary systems, there
should be a large number of low mass CBPs which we have not detected yet or
they have been ejected out from their original systems. Furthermore, even
though all confirmed CBPs are nearly coplanar to their host binary orbits,
recent observations show that the range of the gas disk inclinations around
the binary can be wide (Czekala et al., 2019). Therefore, CBPs that are formed
inside a misaligned disks may be destabilized due to massive CBPs after the
disk is dispersed.

Very recently, the James Webb Space Telescope (JWST) near-infrared survey of
the inner Orion Nebula and Trapezium Cluster has found that 9$\%$ of the
planetary mass objects ($\geq$ 0.6 Jupiter masses) are in wide pairs of
planets with similar mass objects (Pearson & McCaughrean, 2023). These sub-
stellar binaries may form from the same circumbinary planetary systems where
both planets ejected. However, it is not at all clear that pair formation is
possible. If pairs do form, their mass ratios would not be extreme.

##  5 Conclusions

In this study, we considered the ejection of inclined orbit interacting
circumbinary planets as a possible origin of free-floating planets. We
explored the orbital stability of two planet systems around a binary with
equal mass stars each of half Solar mass. We considered cases of two
interacting CBPs with an initial inner planet orbital radius $a_{\rm
p1}=5a_{\rm b}$. At this inner orbital radius, the planet is stable to binary-
planet interactions for a system with a single planet (Chen et al., 2020) The
planets are initially mutually coplanar, but inclined relative to the binary
orbital plane.

We first considered cases with unequal mass planets including a Jupiter-mass
and an Earth-mass planet. Comparing with previous stability maps with two
Jupiter-mass planets (Figure 2 of Chen et al., 2023), we find that the
stability maps with different mass planets are slightly more stable, but
qualitatively very similar. However, in this letter we find that only the more
massive Jupiter-mass planet can survive in the unstable cases. In addition,
initial misalignment of the planets plays an important role in providing
instability. Thus, a Jupiter-mass planet tends to kick out other low mass CBPs
if they are in initially misaligned orbits with respect to the binary. A low
mass CBP can be slightly more stable if it is located outside of the massive
CBP.

We examined the dependence of the orbital instability on planet mass by
considering CBP systems with equal mass planets for several values of planet
mass and two values of binary eccentricity. We find that instability generally
increases with increasing planet mass and with increasing binary eccentricity.

The results show that CBPs with $m_{\rm p}\gtrsim$ Neptune-mass around a
circular or eccentric orbit binary are unstable, while CBPs with each planet
having mass $m_{\rm p}\gtrsim$ 10 Earth-mass are unstable around an eccentric
binary. Two CBPs with $m_{\rm p}\geq$ Saturn-mass have a similar stability as
two Jupiter-mass planets. Consequently, we may find more stable compact
planetary systems around low $e_{\rm b}$ binaries. Notice that our simulations
are scale free so that the minimum mass to trigger this instability can vary
with the total mass of the binary.

In conclusion, we predict there are significantly more small mass planets
compared to high mass planets that are ejected from binary star systems. This
may be in agreement with the result that shows that there is a 95% upper limit
on the frequency of free-floating Jupiter mass planets that is 0.25 per star
(Mróz et al., 2017). In addition, recent observations for the Milky Way and
M31 by the Subaru Hyper Suprime-Cam show that there are abundant populations
of unbound sub-terrestrial objects with masses in range of
$10^{-5}M_{\oplus}-1M_{\oplus}$ (DeRocco et al., 2023). Considering that low
mass planets are more numerous and the natural instability of a circumbinary
system with a massive CBP, we suggest that binary systems can generate both
high and low mass FFPs due to the combined effects of planet-planet and
binary-planet interactions.

Apart from TESS, in an upcoming observation season, the Nancy Grace Roman
Space Telescope (Roman) survey will explore the Galactic bulge for FFPs (Penny
et al., 2019; Johnson et al., 2020). Furthermore, the capabilities of the
Roman mission extend to the prospect of detecting FFPs within the Magellanic
Clouds, as suggested by recent research in Sajadian (2021). Finally, JWST has
detected many free-floating binary planets. We will investigate further
whether the CBP model has the potential to address the origin of these
objects.

Computer support was provided by UNLV’s National Supercomputing Center and the
DiRAC Data Intensive service at Leicester, operated by the University of
Leicester IT Services, which forms part of the STFC DiRAC HPC Facility
(www.dirac.ac.uk). CC and CJN acknowledge support from the Science and
Technology Facilities Council (grant number ST/Y000544/1). CC thanks for the
useful discussion with Dr. Wei Zhu on the PPVII conference. RGM and SHL
acknowledge support from NASA through grants 80NSSC19K0443 and 80NSSC21K0395.
CJN acknowledges support from the Leverhulme Trust (grant number
RPG-2021-380). Simulations in this letter made use of the REBOUND code which
can be downloaded freely at http://github.com/hannorein/rebound.

## References

  * ALMA Partnership et al. (2015) ALMA Partnership, Brogan, C. L., Pérez, L. M., et al. 2015, ApJ, 808, L3, doi: 10.1088/2041-8205/808/1/L3
  * Aly et al. (2015) Aly, H., Dehnen, W., Nixon, C., & King, A. 2015, MNRAS, 449, 65, doi: 10.1093/mnras/stv128
  * Barclay et al. (2017) Barclay, T., Quintana, E. V., Raymond, S. N., & Penny, M. T. 2017, ApJ, 841, 86, doi: 10.3847/1538-4357/aa705b
  * Bate (2018) Bate, M. R. 2018, MNRAS, 475, 5618, doi: 10.1093/mnras/sty169
  * Bate et al. (2000) Bate, M. R., Bonnell, I. A., Clarke, C. J., et al. 2000, MNRAS, 317, 773, doi: 10.1046/j.1365-8711.2000.03648.x
  * Boss (2011) Boss, A. P. 2011, ApJ, 731, 74, doi: 10.1088/0004-637X/731/1/74
  * Brinch et al. (2016) Brinch, C., Jørgensen, J. K., Hogerheijde, M. R., Nelson, R. P., & Gressel, O. 2016, ApJ, 830, L16, doi: 10.3847/2041-8205/830/1/L16
  * Bryant et al. (2023) Bryant, E. M., Bayliss, D., & Van Eylen, V. 2023, Monthly Notices of the Royal Astronomical Society, 521, 3663
  * Capelo et al. (2012) Capelo, H. L., Herbst, W., Leggett, S. K., Hamilton, C. M., & Johnson, J. A. 2012, ApJ, 757, L18, doi: 10.1088/2041-8205/757/1/L18
  * Chambers et al. (1996) Chambers, J., Wetherill, G., & Boss, A. 1996, Icarus, 119, 261, doi: https://doi.org/10.1006/icar.1996.0019
  * Chen et al. (2019) Chen, C., Franchini, A., Lubow, S. H., & Martin, R. G. 2019, MNRAS, 490, 5634, doi: 10.1093/mnras/stz2948
  * Chen et al. (2020) Chen, C., Lubow, S. H., & Martin, R. G. 2020, MNRAS, 494, 4645, doi: 10.1093/mnras/staa1037
  * Chen et al. (2022) Chen, C., Lubow, S. H., & Martin, R. G. 2022, MNRAS, 510, 351, doi: 10.1093/mnras/stab3488
  * Chen et al. (2023) Chen, C., Lubow, S. H., Martin, R. G., & Nixon, C. J. 2023, MNRAS, 521, 5033, doi: 10.1093/mnras/stad739
  * Chen et al. (2023) Chen, C., Martin, R. G., & Nixon, C. J. 2023, Monthly Notices of the Royal Astronomical Society, 525, 3781, doi: 10.1093/mnras/stad2543
  * Chiang & Murray-Clay (2004) Chiang, E. I., & Murray-Clay, R. A. 2004, ApJ, 607, 913, doi: 10.1086/383522
  * Clanton & Gaudi (2017) Clanton, C., & Gaudi, B. S. 2017, ApJ, 834, 46, doi: 10.3847/1538-4357/834/1/46
  * Clarke & Pringle (1993) Clarke, C. J., & Pringle, J. E. 1993, MNRAS, 261, 190, doi: 10.1093/mnras/261.1.190
  * Czekala et al. (2019) Czekala, I., Chiang, E., Andrews, S. M., et al. 2019, ApJ, 883, 22, doi: 10.3847/1538-4357/ab287b
  * DeRocco et al. (2023) DeRocco, W., Smyth, N., & Profumo, S. 2023, arXiv e-prints, arXiv:2308.13593, doi: 10.48550/arXiv.2308.13593
  * Dong et al. (2015) Dong, R., Zhu, Z., & Whitney, B. 2015, ApJ, 809, 93, doi: 10.1088/0004-637X/809/1/93
  * Doolin & Blundell (2011) Doolin, S., & Blundell, K. M. 2011, MNRAS, 418, 2656, doi: 10.1111/j.1365-2966.2011.19657.x
  * Dressing & Charbonneau (2013) Dressing, C. D., & Charbonneau, D. 2013, The Astrophysical Journal, 767, 95, doi: 10.1088/0004-637X/767/1/95
  * Farago & Laskar (2010) Farago, F., & Laskar, J. 2010, MNRAS, 401, 1189, doi: 10.1111/j.1365-2966.2009.15711.x
  * Fleming et al. (2018) Fleming, D. P., Barnes, R., Graham, D. E., Luger, R., & Quinn, T. R. 2018, ApJ, 858, 86, doi: 10.3847/1538-4357/aabd38
  * Flock et al. (2015) Flock, M., Ruge, J. P., Dzyurkevich, N., et al. 2015, A&A, 574, A68, doi: 10.1051/0004-6361/201424693
  * Franchini et al. (2020) Franchini, A., Martin, R. G., & Lubow, S. H. 2020, MNRAS, 491, 5351, doi: 10.1093/mnras/stz3175
  * Gladman (1993) Gladman, B. 1993, Icarus, 106, 247, doi: 10.1006/icar.1993.1169
  * Gong (2017) Gong, Y.-X. 2017, ApJ, 834, 55, doi: 10.3847/1538-4357/834/1/55
  * Gong & Ji (2017) Gong, Y.-X., & Ji, J. 2017, AJ, 154, 179, doi: 10.3847/1538-3881/aa8c7c
  * Johnson et al. (2020) Johnson, S. A., Penny, M., Gaudi, B. S., et al. 2020, AJ, 160, 123, doi: 10.3847/1538-3881/aba75b
  * Kennedy et al. (2012) Kennedy, G. M., Wyatt, M. C., Sibthorpe, B., et al. 2012, MNRAS, 421, 2264, doi: 10.1111/j.1365-2966.2012.20448.x
  * Kennedy et al. (2019) Kennedy, G. M., Matrà, L., Facchini, S., et al. 2019, Nature Astronomy, 3, 278, doi: 10.1038/s41550-019-0715-1
  * Kenworthy et al. (2022) Kenworthy, M. A., González Picos, D., Elizondo, E., et al. 2022, A&A, 666, A61, doi: 10.1051/0004-6361/202243441
  * Kostov et al. (2013) Kostov, V. B., McCullough, P. R., Hinse, T. C., et al. 2013, ApJ, 770, 52, doi: 10.1088/0004-637X/770/1/52
  * Kozai (1962) Kozai, Y. 1962, AJ, 67, 591, doi: 10.1086/108790
  * Kratter & Shannon (2014) Kratter, K. M., & Shannon, A. 2014, MNRAS, 437, 3727, doi: 10.1093/mnras/stt2179
  * Lidov (1962) Lidov, M. L. 1962, Planet. Space Sci., 9, 719, doi: 10.1016/0032-0633(62)90129-0
  * Lubow & Martin (2018) Lubow, S. H., & Martin, R. G. 2018, MNRAS, 473, 3733, doi: 10.1093/mnras/stx2643
  * Lubow & Ogilvie (2000) Lubow, S. H., & Ogilvie, G. I. 2000, ApJ, 538, 326, doi: 10.1086/309101
  * Ma et al. (2016) Ma, S., Mao, S., Ida, S., Zhu, W., & Lin, D. N. C. 2016, MNRAS, 461, L107, doi: 10.1093/mnrasl/slw110
  * Marchal & Bozis (1982) Marchal, C., & Bozis, G. 1982, Celestial Mechanics, 26, 311, doi: 10.1007/BF01230725
  * Martin (2019) Martin, D. V. 2019, Monthly Notices of the Royal Astronomical Society, 488, 3482, doi: 10.1093/mnras/stz959
  * Martin & Lubow (2017) Martin, R. G., & Lubow, S. H. 2017, ApJ, 835, L28, doi: 10.3847/2041-8213/835/2/L28
  * Martin et al. (2016) Martin, R. G., Lubow, S. H., Nixon, C., & Armitage, P. J. 2016, MNRAS, 458, 4345, doi: 10.1093/mnras/stw605
  * Miret-Roig et al. (2021) Miret-Roig, N., Bouy, H., Raymond, S. N., et al. 2021, Nature Astronomy, 6, 89–97, doi: 10.1038/s41550-021-01513-x
  * Mróz et al. (2017) Mróz, P., Udalski, A., Skowron, J., et al. 2017, Nature, 548, 183, doi: 10.1038/nature23276
  * Naoz et al. (2017) Naoz, S., Li, G., Zanardi, M., de Elía, G. C., & Di Sisto, R. P. 2017, AJ, 154, 18, doi: 10.3847/1538-3881/aa6fb0
  * Nixon et al. (2011) Nixon, C. J., Cossins, P. J., King, A. R., & Pringle, J. E. 2011, MNRAS, 412, 1591, doi: 10.1111/j.1365-2966.2010.17952.x
  * Nixon et al. (2018) Nixon, C. J., King, A. R., & Pringle, J. E. 2018, MNRAS, 477, 3273, doi: 10.1093/mnras/sty593
  * Orosz et al. (2012) Orosz, J. A., Welsh, W. F., Carter, J. A., et al. 2012, ApJ, 758, 87, doi: 10.1088/0004-637X/758/2/87
  * Padoan & Nordlund (2002) Padoan, P., & Nordlund, Å. 2002, ApJ, 576, 870, doi: 10.1086/341790
  * Papaloizou & Terquem (1995) Papaloizou, J. C. B., & Terquem, C. 1995, MNRAS, 274, 987
  * Pearson & McCaughrean (2023) Pearson, S. G., & McCaughrean, M. J. 2023, Jupiter Mass Binary Objects in the Trapezium Cluster.  https://arxiv.org/abs/2310.01231
  * Penny et al. (2019) Penny, M. T., Gaudi, B. S., Kerins, E., et al. 2019, ApJ, 241, 3, doi: 10.3847/1538-4365/aafb69
  * Pfyffer et al. (2015) Pfyffer, S., Alibert, Y., Benz, W., & Swoboda, D. 2015, A&A, 579, A37, doi: 10.1051/0004-6361/201424295
  * Quarles et al. (2020) Quarles, B., Li, G., Kostov, V., & Haghighipour, N. 2020, The Astronomical Journal, 159, 80, doi: 10.3847/1538-3881/ab64fa
  * Quarles et al. (2018) Quarles, B., Satyal, S., Kostov, V., Kaib, N., & Haghighipour, N. 2018, ApJ, 856, 150, doi: 10.3847/1538-4357/aab264
  * Raghavan et al. (2010) Raghavan, D., McAlister, H. A., Henry, T. J., et al. 2010, ApJS, 190, 1, doi: 10.1088/0067-0049/190/1/1
  * Rasio & Ford (1996) Rasio, F. A., & Ford, E. B. 1996, Science, 274, 954, doi: 10.1126/science.274.5289.954
  * Rein & Tamayo (2015) Rein, H., & Tamayo, D. 2015, MNRAS, 452, 376, doi: 10.1093/mnras/stv1257
  * Reipurth & Clarke (2001) Reipurth, B., & Clarke, C. 2001, AJ, 122, 432, doi: 10.1086/321121
  * Sajadian (2021) Sajadian, S. 2021, MNRAS, 506, 3615, doi: 10.1093/mnras/stab1907
  * Smullen et al. (2016) Smullen, R. A., Kratter, K. M., & Shannon, A. 2016, MNRAS, 461, 1288, doi: 10.1093/mnras/stw1347
  * Standing et al. (2023) Standing, M. R., Sairam, L., Martin, D. V., et al. 2023, arXiv e-prints, arXiv:2301.10794, doi: 10.48550/arXiv.2301.10794
  * Sumi et al. (2011) Sumi, T., Kamiya, K., Bennett, D. P., et al. 2011, Nature, 473, 349, doi: 10.1038/nature10092
  * Sutherland & Fabrycky (2016) Sutherland, A. P., & Fabrycky, D. C. 2016, ApJ, 818, 6, doi: 10.3847/0004-637X/818/1/6
  * Turrini et al. (2020) Turrini, D., Zinzi, A., & Belinchon, J. A. 2020, A&A, 636, A53, doi: 10.1051/0004-6361/201936301
  * Veras & Raymond (2012) Veras, D., & Raymond, S. N. 2012, MNRAS, 421, L117, doi: 10.1111/j.1745-3933.2012.01218.x
  * Verrier & Evans (2009) Verrier, P. E., & Evans, N. W. 2009, MNRAS, 394, 1721, doi: 10.1111/j.1365-2966.2009.14446.x
  * von Zeipel (1910) von Zeipel, H. 1910, Astronomische Nachrichten, 183, 345, doi: 10.1002/asna.19091832202
  * Weidenschilling & Marzari (1996) Weidenschilling, S. J., & Marzari, F. 1996, Nature, 384, 619, doi: 10.1038/384619a0
  * Weiss et al. (2018) Weiss, L. M., Marcy, G. W., Petigura, E. A., et al. 2018, AJ, 155, 48, doi: 10.3847/1538-3881/aa9ff6
  * Welsh et al. (2012) Welsh, W. F., Orosz, J. A., Carter, J. A., et al. 2012, Nature, 481, 475, doi: 10.1038/nature10768
  * Whitworth & Zinnecker (2004) Whitworth, A. P., & Zinnecker, H. 2004, A&A, 427, 299, doi: 10.1051/0004-6361:20041131
  * Winn et al. (2004) Winn, J. N., Holman, M. J., Johnson, J. A., Stanek, K. Z., & Garnavich, P. M. 2004, ApJ, 603, L45, doi: 10.1086/383089
  * Wittenmyer et al. (2020) Wittenmyer, R. A., Butler, R. P., Horner, J., et al. 2020, MNRAS, 491, 5248, doi: 10.1093/mnras/stz3378
  * Yen et al. (2019) Yen, H.-W., Gu, P.-G., Hirano, N., et al. 2019, ApJ, 880, 69, doi: 10.3847/1538-4357/ab29f8
  * Yen et al. (2016) Yen, H.-W., Liu, H. B., Gu, P.-G., et al. 2016, ApJ, 820, L25, doi: 10.3847/2041-8205/820/2/L25
  * Zanazzi & Lai (2017) Zanazzi, J. J., & Lai, D. 2017, MNRAS, 467, 1957, doi: 10.1093/mnras/stx208
  * Zhang et al. (2015) Zhang, K., Blake, G. A., & Bergin, E. A. 2015, ApJ, 806, L7, doi: 10.1088/2041-8205/806/1/L7
  * Zhang et al. (2018) Zhang, S., Zhu, Z., Huang, J., et al. 2018, ApJ, 869, L47, doi: 10.3847/2041-8213/aaf744
  * Zhang & Fabrycky (2019) Zhang, Z., & Fabrycky, D. C. 2019, ApJ, 879, 92, doi: 10.3847/1538-4357/ab24d5
  * Zhu et al. (2022) Zhu, W., Bernhard, K., Dai, F., et al. 2022, ApJ, 933, L21, doi: 10.3847/2041-8213/ac7b2d