# Learning to Ask Informative Questions: Enhancing LLMs with
Preference Optimization and Expected Information Gain

Davide Mazzaccara
CIMeC, University of Trento
<EMAIL_ADDRESS>
&Alberto Testoni
ILLC, University of Amsterdam
<EMAIL_ADDRESS>
Raffaella Bernardi
CIMeC, DISI, University of Trento
<EMAIL_ADDRESS>
###### Abstract

Questions are essential tools for acquiring the necessary information to
complete information-seeking tasks. However, large language models (LLMs),
especially open-source models, often perform poorly in generating informative
questions, as measured by expected information gain (EIG). In this paper, we
propose a method to enhance the informativeness of LLM-generated questions in
20-question game dialogues. We sample multiple questions from the same model
(Llama 2-chat 7B) for each game and create pairs of low-EIG and high-EIG
questions to apply a Direct Preference Optimization (DPO) algorithm. Our
results show that this method produces more effective questions (in terms of
EIG), even in domains different from those used to train the DPO model.

Learning to Ask Informative Questions: Enhancing LLMs with
Preference Optimization and Expected Information Gain

Davide Mazzaccara CIMeC, University of Trento<EMAIL_ADDRESS>Alberto Testoni ILLC, University of Amsterdam<EMAIL_ADDRESS>

Raffaella Bernardi CIMeC, DISI, University of Trento
<EMAIL_ADDRESS>

##  1 Introduction

Questions in language serve as requests for information (Hiż, 1978). A speaker
lacks information in their knowledge state and asks questions to gain this
information. This process of acquiring information through questioning is
essential for children to learn about the world and for adults to solve
complex problems (Geva et al., 2021). Different questions, however, vary in
their level of informativeness. Some questions are more informative and then
efficient to reach the problem’s solution (Grand et al., 2023).

Cognitive Science provides two interesting tools to study questions’
informativeness: the 20 Questions Game as a test bed, and the Expected
Information Gain (EIG) as a quantifying measure (Ruggeri and Lombrozo, 2015).
The _20 Questions game_ consists of one player asking yes/no questions to
identify the item the other player has in mind, from a pool of possible items.
In this context, the EIG (Shannon, 1948) measures questions’ informativeness
as the (expected) entropy reduction caused by a certain question in the space
of possible items. For binary questions, the most informative ones partition
the possible items into two equal clusters, where half are expected to receive
a positive answer and the other half a negative answer.

In recent years, Large Language Models (LLMs) have demonstrated remarkable
language and reasoning capabilities (Kojima et al., 2022; Huang and Chang,
2023). The ability to ask informative and effective questions is crucial for
employing these systems as successful user assistants on a large scale.
Although LLMs generate reasonable questions, their questions are generally
characterised by low informativeness and limited success in the 20 Questions
game (Bertolazzi et al., 2023). Recent studies have emerged to improve the
informativeness of LLMs questions. Zhang et al. (2024) improve open-source
LLMs via Reinforcement Learning and Behavioral Cloning (i.e., small models
learn from bigger LLMs’ data). Without training models, Hu et al. (2024)
propose a re-ranking strategy based on EIG. Differently from them, we propose
a training strategy leveraging EIG as a refined signal with direct preference
optimization. Unlike other approaches, all steps of our method are
accomplished by the same open-source model, without requiring annotation or
feedback signals from external models.

Figure 1: Our approach for constructing the datasets of dialogues for fine-
tuning and preference optimization via DPO, which is based on the Expected
Information Gain (EIG) of the sampled questions.

This study aims to improve the informativeness of questions generated by LLMs,
thereby leading to more efficient agents. The 20 Questions Game serves as a
testbed to illustrate the feasibility of our approach. To achieve our aim, we
propose a method involving three steps: sampling multiple questions,
questions’ evaluation in terms of EIG, and training with preference
optimization. In the first step, the model generates a set of possible
questions, and it computes the EIG for each question. This set of questions,
along with the corresponding EIG values, as a proxy for the questions’
informativeness, is used for Direct Preference Optimization (DPO; Rafailov et
al. 2023). Our results show that EIG is a strong training signal to improve
the question-asking capabilities of current LLMs and overcome their
shortcomings in asking effective questions.111Code and data will be made
available, CC BY license, upon acceptance.

##  2 Methodology

###  2.1 Setting

Our experimental setting, inspired by the 20-question game paradigm, involves
two players with different roles: a Questioner and an Answer. The Answer is
secretly assigned one target entity $\omega$ among a pre-defined set of
candidates (the _candidate set_ – $\Omega$). The Questioner receives the
initial candidates and is instructed to ask yes/no questions, in order to
identify $\omega$.

Each _game_ consists of a set of possible items, $\Omega$, available to the
Questioner, and a specific target item, $\omega$, available to the Answerer,
where $\omega\in\Omega$. A _dialogue_ is a series of question-answer exchanges
between the two players, ending when the Questioner correctly identifies the
target or reach the maximum number of questions (20). If the target is reached
within the 20 questions, the dialogue is _successful_ ; otherwise, it is
_unsuccessful_ (Appendix A.1). For a given game, if the dialogue is not
successful on the first attempt, the Questioner and the Answerer can engage in
up to $n$ dialogues to identify the target, where $n=|\Omega|/2$.

###  2.2 Data Sampling

A total of 10,000 games are extracted from McRae concepts as in Bertolazzi et
al. (2023). The candidate sets consist of concepts belonging to 6 categories
(mammal, bird, clothing, weapon, fruit, and vegetable) for a total of 66
unique concepts. Each candidate set consists of 8 elements belonging to two
categories of 4 elements each ($|\Omega|=8$).

All steps of our sampling method rely on the same LLM, Llama 2-chat (7B)
Touvron et al. (2023), for all involved agents (see Appendix A.2, A.3). We
generated English dialogues by repeatedly sampling questions from the
Questioner for each game. As illustrated in Fig. 1, for each generated
question, EIG is computed through the yes/no annotation of the Annotator. Once
an ‘optimal’ question is reached, it is saved along with its answer in the
dialogue history. This dialogue history is input to the Questioner to sample
the ‘optimal’ follow-up question. We split the resulting questions into two
datasets. A _fine-tuning dataset_ consisting of dialogues of only ‘optimal’
questions – EIG = 1, indicating that they evenly split the candidate space at
each turn. A _preference dataset_ , which consists of 55,000
‘optimal’($EIG=1$) and suboptimal ($EIG<0.8$) question pairs.

###  2.3 Questioner Models

Our evaluation of the Questioner compares Llama 2-chat (7B) zero-shot with the
trained versions below. Both the training processes rely on LoRA adapters and
standard hyperparameter settings.

Fine-tuning (FT) involves a causal language modelling objective applied to the
entire sequence. For development, we use 50 games excluded from the fine-
tuning dataset. To obtain the best adapter’s checkpoint, we test the adapter
on the development set every 1,000 training samples. According to the number
of questions per dialogue, the best adapter’s checkpoints are after 4,000
dialogues.
Direct Preference Optimization (DPO) (Rafailov et al., 2023) training
objective increases the likelihood of optimal EIG questions while decreasing
the likelihood of the low-EIG question. For DPO, we train with all the 55,000
pairs of optimal vs low-EIG questions.

###  2.4 Evaluation Metrics

The impact of FT and DPO on zero-shot is assessed with three metrics. Task
success (S@1) is the percentage of games in which the model identifies the
target within the first dialogue. Average number of questions (AQ) is the
number of questions to reach the target in successful dialogues. Expected
Information Gain (EIG) is the averaged questions’ EIG in successful dialogues.

##  3 Results

The robustness of the different methods was assessed by comparing their
performance on test sets with candidate sets of the same size as in training
but from other domains. The results are reported in Table 1. For INLG
Bertolazzi et al. (2023) and Things Zhang et al. (2024), DPO significantly
improves performance over the zero-shot model in terms of both S@1 ($+12.2\%$
and $+10\%$) and AQ ($-2.1$ and $-2.3$ questions per successful dialogues).
For Celebrities Zhang et al. (2024), where the concept space differs greatly
from the training data, DPO reduces the average number of questions by $2.5$,
while S@1 shows only a marginal improvement. A consistent improvement of DPO
over zero-shot is also observed for the average EIG of the generated
questions. Overall, the fine-tuning approach leads to a significant
degradation in the S@1 metric.

Given the promising results of DPO, we proceed to evaluate the robustness of
this approach with larger candidate sets ($>8$) at test time – preference
optimization is still based on candidate sets of 8 elements.

Set | Method | S@1 $\uparrow$ | AQ $\downarrow$ | EIG $\uparrow$
---|---|---|---|---
| zero-shot | 56.7% | 7.1 | 0.34
INLG | FT | 46.6% | 4.6 | 0.41
| DPO | 68.9% | 5.2 | 0.45
| zero-shot | 51.1% | 7.5 | 0.29
Things | FT | 42.2% | 5.4 | 0.31
| DPO | 61.1% | 5.2 | 0.40
| zero-shot | 71.1% | 7.6 | 0.35
Celebrities | FT | 46.7% | 5.5 | 0.39
| DPO | 72.2% | 5.1 | 0.47

Table 1: Different Domains: Results on candidate sets ($|\Omega|=8$): seen
categories but unseen concepts (INLG), unseen categories but common-life
concepts (Things), people (Celebrities). Across the three sets, DPO identifies
the target within the first dialogue (S@1) more often, with shorter dialogues
(AQ) and more informative questions (EIG). FT and DPO’s improvements for AQ
and EIG are statistically significant compared to zero-shot scores (Mann-
Whitney U test).

As shown in Table 2, with candidate sets of 16 and 29 candidates, DPO
outperforms zero-shot in terms of S@1, with improvements of $+6.7\%$ and
$+13.8\%$, respectively. DPO generates shorter dialogues, reducing questions
by $3.2$ and $0.7$. However, in BigBench Srivastava et al. (2023), the average
EIG of the questions asked by DPO slightly decreases compared to zero-shot.
This is explored in the next Section where we dive into the dialogues.

Set | Method | S@1 $\uparrow$ | AQ $\downarrow$ | EIG $\uparrow$
---|---|---|---|---
INLG 16 | zero-shot | 44.4% | 9.5 | 0.31
| DPO | 51.1% | 6.3 | 0.38
BigBench | zero-shot | 17.2% | 8.8 | 0.31
| DPO | 31.0% | 8.1 | 0.28
Table 2: Different Size: DPO boosts the model performance on INLG 16 (larger
sets of unseen concepts of seen categories), and improves its success rate but
not the other scores with BigBench (larger sets of concrete and abstract
concepts). Improvements in INLG 16 for AQ and EIG are statistically
significant (Mann-Whitney U test).

##  4 Analysis

First of all, following Bertolazzi et al. (2023), we examine the types of
questions that models ask: _constraint seeking_ (CS) are questions about a
feature shared by more candidates, whereas _hypothesis scanning_ (HS) are
questions about a specific candidate. In every test set apart from BigBench,
zero-shot asks more CS with lower informativeness than DPO (Table 3: EIG for
CS questions has a $+0.17$ compared to zero-shot). DPO seems to identify a
subset of the original candidate sets through one or more CS, then asks HS
questions to guess the target. By looking at the data, we saw that DPO narrows
down to the relevant candidate subset by initially posing a series of CS
questions. Subsequently, it moves to mostly coherent HS: this holds also with
abstract concepts, unseen during sampling, as illustrated in Figure 2.

Dataset | Method | HS | CS
---|---|---|---
|  |  % |  EIG |  % |  EIG
INLG | zero-shot |  47.45 |  0.25 |  52.55 |  0.42
| DPO |  60.05 |  0.33 |  39.95 |  0.62
Things | zero-shot |  44.13 |  0.22 |  55.87 |  0.35
| DPO |  67.59 |  0.33 |  32.41 |  0.56
Celebrities | zero-shot |  31.84 |  0.15 |  68.16 |  0.44
| DPO |  53.92 |  0.36 |  46.08 |  0.59
INLG 16 | zero-shot |  38.66 |  0.19 |  61.34 |  0.39
| DPO |  51.47 |  0.27 |  48.53 |  0.51
BigBench | zero-shot |  64.2 |  0.22 |  35.71 |  0.48
| DPO |  61.64 |  0.21 |  38.36 |  0.41

Table 3: Percentage (%) and average EIG for hypothesis scanning (HS) and
constraint-seeking (CS) questions. Target: anger List of candidates: car,
glass micropipette, global poverty, house, anger, train, airplane, television,
representative democracy, love, giganotosaurus, hate, tree, surprise, disgust,
siberian husky, phase transition, shoes, frisbee, jealousy, positive sum game,
beauty, coat, apple, cow, contentment, hopefulness, dinosaur, shower Dialogue
Q: Is the target an animal? A: No Q: Is the target a type of vehicle (car,
train, airplane, etc.)? A: No. Q: Is the target a type of building or
structure (house, train station, airport, etc.)? A: No. Q: Is the target a
form of technology (computer, phone, micropipette, etc.)? A: No. Q: Is the
target a type of clothing or accessory (shirt, pants, hat, etc.)? A: No. Q: Is
the target a type of emotion or feeling (love, hate, joy, etc.)? A: Yes. Q:
Great! Is the target “love”? A: No. Q: Is the target “hate”? A: No. … Q: Is
the target “anger”? A: Yes. Figure 2: Example from the BigBench. DPO asks
grounded CS questions (highlighted with colors), identifying the subset of the
target (e.g., emotions). It then asks a series of HS questions.

Secondly, we aim to understand the high scores DPO obtains in INLG and the
lower ones in BigBench. INLG candidate sets are organized taxonomically, half
of the candidates belong to one category and the other half to another one. By
inspecting the dialogues, we saw that both zero-shot and DPO tend to ask
informative questions at the first turn, identifying one of the two
categories. However, the two models differ in the follow-up turn: when the
first turn receives a negative answer, zero-shot questions tend to have low
EIG, while DPO questions are informative by zooming on the identified category
(see Appendix C). This suggests that DPO has learned to profit from negatively
answered questions, on which LLMs are known to fail Ravichander et al. (2022);
Truong et al. (2023). We dive into BigBench by comparing the models’
performance on concrete and abstract targets (Table 4). In general, both zero-
shot and DPO perform better when the target is a concrete entity. Notably, DPO
significantly improves performance with both types of concepts (+12.5% and
+15.4% for concrete and abstract targets, respectively). This improvement,
however, is not reflected in the average dialogue length when the target is
abstract. The significant difference in guessing concrete vs. abstract targets
(25.0% vs. 7.7% for zero-shot and 37.5% vs. 23.1% for DPO) calls for further
investigation into the underlying factors contributing to this disparity and
the potential need for tailored strategies to handle abstract concepts more
effectively.

Set | Method | S@1 $\uparrow$ | AQ $\downarrow$ | EIG $\uparrow$
---|---|---|---|---
Concrete | zero-shot | 25.0% | 8.4 | 0.33
| DPO | 37.5% | 6.3 | 0.34
Abstract | zero-shot | 7.7% | 9.3 | 0.29
| DPO | 23.1% | 10.3 | 0.24
Table 4: BigBench: concrete vs. abstract target.

##  5 Conclusion

In our work, we designed a Direct Preference Optimization (DPO) approach to
enhance the informativeness of questions asked by LLMs using Expected
Information Gain (EIG). We utilized the 20 Questions Game paradigm, a
framework in cognitive science and AI for studying information-seeking
behavior, reasoning, and hypothesis testing. Our approach involved (a)
sampling multiple questions from the model in a zero-shot fashion, (b)
clustering the questions based on their EIG, and (c) training the model using
these clusters with DPO. Our results show that DPO significantly improves
question informativeness (measured by average EIG and number of questions
asked), making the dialogue strategy more effective. Notably, this method
generalizes well to different domains. Our findings demonstrate that EIG is a
promising training signal for improving the reasoning capabilities of LLMs in
information-seeking dialogues.

##  6 Limitations

This work is intended to be exploratory. To investigate whether LLMs could
learn to be more informative and effective with self-generated EIG signals, we
focus on one model (Llama 2-chat 7B) and one preference optimization strategy
(DPO). Further work is required to determine if this training strategy holds
with other models and other preference optimization strategies. A second
limitation of our study is related to the EIG computation. EIG computation
depends on the yes/no annotation. While we can assume a high degree of
accuracy, based on our careful inspection of the dialogues, some questions are
still difficult to answer with only yes/no (e.g., $w=$ dinosaur, the question
‘Is it living?’). Additionally, all the candidates in $\Omega$ are considered
equally likely, while LLMs have priors conditioning its question generation.
Furthermore, when computing the EIG of follow-up questions, we consider the
model able to sequentially rule out candidates excluded in the dialogue
history, which could be a strong assumption for a generative language model.

## Acknowledgments

We are grateful to David Schlangen and his group and to the audience of the
Evil seminar organized by Marco Baroni. By addressing their questions the
quality of the work improved. We thank Leonardo Bertolazzi for his suggestions
on the early phase of the implementation of this work. Alberto Testoni is
supported by funding from the European Research Council (ERC) under the
European Union’s Horizon 2020 research and innovation programme (grant
agreement No. 819455, PI R. Fernández).

## References

  * Bertolazzi et al. (2023) Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, and Raffaella Bernardi. 2023.  ChatGPT’s information seeking strategy: Insights from the 20-questions game.  In _Proceedings of the 16th International Natural Language Generation Conference_ , pages 153–162, Prague, Czechia. Association for Computational Linguistics.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9.
  * Grand et al. (2023) Gabriel Grand, Valerio Pepe, Jacob Andreas, and Joshua B. Tenenbaum. 2023.  Loose lips sink ships: Asking questions in battleship with language-informed program sampling.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 46.
  * Hiż (1978) Henry Hiż, editor. 1978.  _Questions_.  Reidel, Dordrecht/Boston.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Hu et al. (2024) Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. 2024.  Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in large language models.  In _ICLR 2024 Workshop on Large Language Model (LLM) Agents_.
  * Huang and Chang (2023) Jie Huang and Kevin Chen-Chuan Chang. 2023.  Towards reasoning in large language models: A survey.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  * Jorge et al. (2017) Emilio Jorge, Mikael Kågebäck, Fredrik D. Johansson, and Emil Gustavsson. 2017.  Learning to play guess who? and inventing a grounded language as a consequence.  _Preprint_ , arXiv:1611.03218.
  * Keh et al. (2023) Sedrick Keh, Justin T. Chiu, and Daniel Fried. 2023.  Asking more informative questions for grounded retrieval.  _Preprint_ , arXiv:2311.08584.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 22199–22213. Curran Associates, Inc.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 53728–53741. Curran Associates, Inc.
  * Ravichander et al. (2022) Abhilasha Ravichander, Matt Gardner, and Ana Marasovic. 2022.  CONDAQA: A contrastive reading comprehension dataset for reasoning about negation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8729–8755, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ruggeri and Lombrozo (2015) Azzurra Ruggeri and Tania Lombrozo. 2015.  Children adapt their questions to achieve efficient search.  _Cognition_ , 143:203–216.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2023\.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_.
  * Testoni and Bernardi (2021) Alberto Testoni and Raffaella Bernardi. 2021.  Looking for confirmations: An effective and human-like visual dialogue strategy.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9330–9338, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Testoni et al. (2023) Alberto Testoni, Raffaella Bernardi, and Azzurra Ruggeri. 2023.  The efficiency of question-asking strategies in a real-world visual search task.  _Cognitive Science_ , 47(12).
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _Preprint_ , arXiv:2307.09288.
  * Truong et al. (2023) Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023.  Language models are not naysayers: an analysis of language models on negation benchmarks.  In _Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)_ , pages 101–114, Toronto, Canada. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022.  Star: Bootstrapping reasoning with reasoning.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 15476–15488. Curran Associates, Inc.
  * Zhang et al. (2024) Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. 2024.  Probing the multi-turn planning capabilities of llms via 20 question games.  _Preprint_ , arXiv:2310.01468.

##  Appendix A Methodology

###  A.1 Successful Condition

A dialogue is _successful_ when the Questioner guesses the target entity. We
have implemented this condition with the following two cases: (a) the
Questioner explicitly refers to the target entity without naming other
candidate entities or (b) the Questioner explicitly refers to the target
entity along with other candidates, and the target is named more times than
each of the other entity. This second condition recognizes when the Questioner
is explicitly reasoning about the target before guessing (e.g., target:
‘cherry’. Questioner: ‘So, the target is one of the remaining fruits, which
are cherry and grapefruit. Let me make a guess. Is the target… cherry?’). The
dialogue is considered _unsuccessful_ if the Questioner does not meet the
conclusive conditions above within 20 questions.

###  A.2 Expected Information Gain

In this setting, each question aims to rule out as many candidates as possible
from the set $\Omega=\\{\omega_{1},\omega_{2},...,\omega_{n}\\}$. The most
informative question rules out half of the possible candidates, whereas a
question ruling out none or just one candidate has low informativeness.
Questions’ informativeness has been quantified in prior works through the
Expected Information Gain.

Expected Information Gain (EIG) measures the contribute of each question to
reduce uncertainty towards the solution $\omega$. The level of uncertainty is
measured through the entropy $H$, higher entropy means higher uncertainty.
Consequently, EIG is computed subtracting from the initial state of entropy
($H_{prior}$) the expected entropy after asking the question
($H_{posterior}$).

$EIG=H_{prior}-H_{posterior}$ (1)

At the beginning of each game, we assume a uniform prior distribution over all
the possible candidates $\omega_{1},\omega_{2},...,\omega_{n}$. From the
Shannon entropy (Shannon, 1948), this initial state $H_{prior}$ in our setting
is equal to:

$\begin{split}H_{prior}&=-\sum_{i=1}^{n}\>p(w_{i})\>log_{2}\>p(w_{i})\\\
&=-n\>\frac{1}{n}\>log_{2}\>\frac{1}{n}\>=\>log_{2}\>{n}\end{split}$ (2)

A yes/no question divides the current candidates in $\Omega$ into two disjoint
subsets: $\Omega_{yes}\,\cup\,\Omega_{no}$. In the subset $\Omega_{yes}$ there
are the candidates in $\Omega$ that are consistent with a possible positive
answer, $\Omega_{no}$ with a negative one. Answering the question, the
Answerer determines which subset becomes the new set of candidates under
consideration for the next turn (the candidates in the other subset are ruled
out). The posterior entropy is computed as the sum of the entropies of both
the yes and no subsets, weighted by their probability. $H_{posterior}$ is
computed as:

$H_{posterior}=p_{yes}\,H_{yes}\>+\>p_{no}\,H_{no}$ (3)

This computation is based on the expected answers to the question for each
candidate in the current set. In other words, the Questioner has an
expectation of which candidates will be in $\Omega_{yes}$ and $\Omega_{no}$
before receiving the answer by the Answerer. With these subsets, the
probabilities $p_{yes}$ and $p_{no}$ refer to the probabilities of receiving a
positive or negative answer. The entropies $H_{yes}$ and $H_{no}$ measure
uncertainty in the two subsets computed as in eq. 2. Intuitively, the higher
the similarity between the two subsets in terms of size (i.e., candidates per
subset) the higher the EIG value. The most informative yes/no questions divide
$\Omega$ into two subsets $\Omega_{yes}$ and $\Omega_{no}$ of the same size,
resulting in $EIG=1$.

###  A.3 Setting

####  A.3.1 Sampling

Figure 3: During sampling, Llama 2-chat (7B) plays the roles of the
Questioner, the Annotator, and the Answerer. Questions are sampled from the
Questioner and then evaluated by the Annotator. Once an optimal question is
reached, the Answerer answers it. The optimal question and its answer are
appended to the dialogue history. In this way, optimal questions are sampled
not only for the first turn but also in follow-up turns. In training, the
Questioner is trained with FT and DPO datasets. In testing, the zero-shot  and
trained  Questioners play the 20 Questions Game with an external model as
Answerer.

Our approach comprises the three steps described in Fig. 3. During sampling,
different questions are sampled from the Questioner . To compute EIG, the
Annotator  annotates the questions with yes/no for each candidate in the
candidate set ($\Omega$). Once an ‘optimal’question is reached ($EIG=1$), the
optimal and suboptimal questions are stored in the training datasets. All the
prompts for the roles are reported below. The Annotator has the same prompt as
the Answerer, taking as a target each candidate in the candidate set.

Questioner System Prompt: _You are playing a game, make only one yes/no
question at turn to identify the target from the List of candidates. If there
are 1 or 2 candidates remaining make the guess._ User Prompt: _List of
candidates: CANDIDATES_.

Answerer/Annotator System Prompt: _You are playing the 20-Questions game, you
will be asked one Question about the Target element. Answer only ‘yes’ or ‘no’
to the Question depending on your Target element._ User Prompt: _Target
element: TARGET,
Question: QUESTION_

To yield diverse outputs during sampling, the decoding strategy for the
Questioner involves a relatively high temperature (1.0) and top-k sampling
(50). To achieve more deterministic responses the Answerer/Annotator has a low
temperature of 0.1.

This process produces a Fine-Tuning and DPO dataset. The Fine-Tuning dataset
consists of all the completed dialogues of optimal questions only. The DPO
dataset, instead, comprises pairs of ‘optimal’questions ($EIG=1.0$) with
suboptimal questions ($EIG<0.8$).

####  A.3.2 Training

The Fine-Tuning and DPO datasets are employed to train Llama 2-chat (7B) with
LoRA adapters (Hu et al., 2022). Our main focus is on the approach rather than
the training strategy, thus we did not perform extensive hyperparameter
tuning. We do not exclude that additional experiments on hyperparameters could
lead to better results for our approach. For both Fine-Tuning and DPO, the
LoRA adapters are applied on all the modules, with an $r=128$ and the alpha
value $=32$. We perform Fine-Tuning and DPO with AdamW as the optimizer,
training for one epoch and with a batch size $=8$. For both Fine-Tuning and
DPO, a learning rate of $2e-5$ has been observed to produce consistent
results.

####  A.3.3 Testing

At test time, we compare the Llama 2-chat (7B) zero-shot with its Fine-Tuned
and DPO alternatives. We employ an external model as Answerer to make the test
setting more different from the training setting. The Answerer model  is
gpt-3.5-turbo-0125 , with default parameters for generation. The total cost
for testing - every approach in every setting - is less than 3$. At test time,
we experimented also with the Questioner using the default Llama 2-chat
hyperparameters, the results are consistent with the ones reported in the
paper Section 3. For the hyperparameters used in the paper, the results across
multiple runs are also consistent.

#### Implementation Details

The inference and training with Llama 2-chat (7B) were performed using the
Huggingface Transformer library (Wolf et al., 2020). All experiments were
executed on a Nvidia GeForce RTX 3090 24GB GPU. Sampling 10,000 dialogues took
approximately 4 days. Fine-Tuning and DPO take around 12 hours each. Running
inference on all the test sets lasts around 6 hours for FT and DPO, whereas it
lasts around 10 hours for the zero-shot model.

##  Appendix B Test Sets

The games in the test sets are with candidates unseen during training. To test
the trained models in different domains, we have the following test sets of 8
candidates, as in training:

  * •

INLG: 90 candidate sets structured in such a way that half of the candidates
pertain to one taxonomic category and the other half to another one
(Bertolazzi et al., 2023). The taxonomic categories for the candidates are
animals, clothing, weapons, fruits, and vegetables.

  * •

Things: 90 candidate sets from the evaluation split of Zhang et al. (2024).
The full list of categories are animals, clothing, foods, objects, plants,
vehicles, professions, materials, instruments, places, sports, buildings,
furniture, celestial bodies, mythical creatures, events, and activities.

  * •

Celebrities: 90 sets from the evaluation split of Zhang et al. (2024). In this
test set, the candidates differ greatly from the training ones, comprising
only past and living celebrities.

To test the trained models with more candidates ($\Omega>8$), we have the
following test sets:

  * •

INLG 16: 90 candidate sets of 16 structured elements, as in INLG (Bertolazzi
et al., 2023).

  * •

BigBench: 29 sets of 29 candidates of abstract and concrete concepts. The
concepts are apple, television, dinosaur, airplane, house, tree, coat, shoes,
car, train, shower, frisbee, cow, giganotosaurus, siberian husky, glass
micropipette (concrete); anger, love, hate, contentment, jealousy, surprise,
disgust, hopefulness, global poverty, phase transition, positive sum game,
beauty, representative democracy (abstract).

##  Appendix C Analysis

### Informativeness after a Negative Answer

INLG (8) and INLG 16 consist of structured sets, i.e., half of the candidates
of one category and the other half to another one. Both the zero-shot and the
DPO tend to ask informative questions at the first turn, identifying one of
the two categories. After a negative answer to this first question, the zero-
shot frequently asks a confirmation question about the other category
($EIG=0$), as illustrated in Figure 4. In both INLG 8 and INLG 16 of Table 5,
the zero-shot shows a higher percentage of 0-EIG questions and lower average
EIG of questions after a negative answer, compared to questions after a
positive answer (EIG after negative and positive questions drawn from
different distributions, Mann–Whitney U test). On the contrary, the DPO has a
lower percentage of 0-EIG questions and higher EIG after a negative answer
(EIG after negative and positive questions of the same distribution). Trained
on similarly structured candidate sets, the DPO seems not to reproduce the
uninformative behaviour of the zero-shot, asking more informative questions
after negative answers.

Set |  Method | after _yes_ | after _no_
---|---|---|---
|  | EIG $\uparrow$ | %0 EIG $\downarrow$ | EIG $\uparrow$ | %0 EIG $\downarrow$
INLG 8 |  zero-shot | 0.36 | 60.1% | 0.21 | 74.1%
|  DPO | 0.37 | 59.4% | 0.33 | 61.6%
INLG 16 |  zero-shot | 0.35 | 55.6% | 0.20 | 71.5%
|  DPO | 0.29 | 65.6% | 0.29 | 60.2%
Table 5: Questions’ informativeness after a positive and a negative answer, in
terms of average EIG (EIG) and percentage of uninformative questions (% 0
EIG). Target: fox List of candidates: _ elk_, chicken, robin, starling, fox,
partridge, hamster, _ buffalo_ Zero-shot Q1: Is the target a bird? $EIG=1$ A:
No. Q2: Is the target a mammal? $EIG=0$ A: Yes. Q3: Does the target have a
long tail? A: Yes. Q4: Is the target a fox? A: Yes. DPO Q1: Is the target
bird? $EIG=1$ A: No. Q2: Is the target _ mammal a large hoofed_ animal, like a
deer or a bison? $EIG=1$ A: No. Q3: Is the target a carnivorous mammal, like a
fox? A: Yes. Figure 4: INLG 8 Example: half of the candidates are birds, and
half are mammals. Both the zero-shot and DPO identify the bird category at the
first turn ($EIG=1$). After the negative answer, the zero-shot asks a
confirmation question about the remaining category (i.e., mammal), with an
$EIG=0$. The DPO, instead, asks a more specific question on the remaining
category (i.e., large hoofed, which are the ‘elk’and ‘buffalo’), with an
$EIG=1$. The higher informativeness of DPO is further reflected in the number
of questions required to reach the target: 3 questions in DPO vs. 4 questions
in zero-shot.


# A Physics-Informed Variational Autoencoder for Rapid Galaxy Inference and
Anomaly Detection

Alexander Gagliano
The NSF AI Institute for Artificial Intelligence and Fundamental Interactions
<EMAIL_ADDRESS>
V. Ashley Villar
Center for Astrophysics | Harvard & Smithsonian

###### Abstract

The Vera C. Rubin Observatory is slated to observe nearly 20 billion galaxies
during its decade-long Legacy Survey of Space and Time. The rich imaging data
it collects will be an invaluable resource for probing galaxy evolution across
cosmic time, characterizing the host galaxies of transient phenomena, and
identifying novel populations of anomalous systems. To facilitate these
studies, we introduce a convolutional variational autoencoder trained to
estimate the redshift, stellar mass, and star-formation rates of galaxies from
multi-band imaging data. We train and test our physics-informed CVAE on a
spectroscopic sample of $\sim$26,000 galaxies within $z<1$ imaged through the
Dark Energy Camera Legacy Survey. We show that our model can infer redshift
and stellar mass more accurately than the latest image-based self-supervised
learning approaches, and is >100x faster than more computationally-intensive
SED-fitting techniques. Using a small sample of Green Pea and Red Spiral
galaxies reported in the literature, we further demonstrate how this CVAE can
be used to rapidly identify rare galaxy populations and interpret what makes
them unique.

##  1 Introduction

At the cornerstone of a range of diverse astrophysical domains sits the
chemical and dynamical histories of galaxies. Aspects of these histories can
be inferred piecewise from observations. From broad-band photometry, one can
infer the underlying spectral energy distribution (SED); using morphological
features, one can reconstruct a galaxy’s merger/interaction history. Novel
galaxy sub-populations have been discovered through detailed manual inspection
along these axes; some prominent examples include ‘Green Pea galaxies’,
defined by their compact shapes and high star-formation rates [5]; and
‘passive Red Spirals’– spiral galaxies with minimal star formation, whose
colors seem to contradict a morphology that is commonly tied to much more
active galaxies [12]. Several of these sub-populations have been only
identified through manual vetting by scores of human volunteers (e.g., [15]).

The rise of deep convolutional neural networks, which allow for morphological
classification of galaxies from image data [6], and of simulation-based
inference to estimate their underlying SED parameters from photometry [18],
have paved the way for low-latency galaxy analysis in the era of synoptic
surveys. However, anomaly detection within these frameworks remains an open
question: out-of-distribution events may be identified by flagging high
reconstruction errors at test time [e.g., 17], but this approach can produce
samples contaminated by non-astrophysical anomalies such as poorly-calibrated
images and chip gaps.

Here, we introduce a technique to guide the latent features of a convolutional
variational auto-encoder (CVAE) toward the physical parameters of a galaxy.
CVAEs are a class of neural network designed for compression. Variational AEs,
in particular, typically employ the Kullback–Leibler (KL) divergence to
enforce a continuous and probabilistic latent distribution characterized by a
multivariate standard Normal. CVAEs have been extensively used in the
literature for galaxy image reconstruction [13, 7, 3]. Here, we extend this
methodology to disentangle the observational properties of a galaxy (e.g., its
orientation) from its intrinsic properties in the latent space, and discover
more meaningful anomalies.

##  2 Data and Methodology

In this section, we describe our methodology for designing and training a
physics-informed CVAE. Our CVAE encodes a galaxy image into 5 latent
dimensions, the first four corresponding to the orientation, redshift, stellar
mass, and star-formation rate of the galaxy. The fifth dimension encodes any
remaining structure not captured by the first four features.

We select a sample of $z<1$ spectroscopic galaxies from both the LSST deep
drilling fields catalogue from [19] (hereafter, the Z22 sample) and from GAMA
DR4 [8]. These catalogues report the average star-formation rate and stellar
mass of each galaxy obtained with the SED-fitting codes CIGALE and PROSPECT,
respectively, along with their associated uncertainties. We include the Z22
sample to provide training support at high redshift, and the GAMA catalogue to
includes low-redshift objects for which rich morphological information is
available. For galaxies that overlap with the Dark Energy Camera Legacy Survey
(DECaLS) footprint, we then download 128x128 px DECaLS co-added images
centered at the location of each galaxy in
$grz$111https://www.legacysurvey.org/decamls/. DECaLS data reaches a
5-$\sigma$ depth of $g\sim$24, making it a sufficient analog for upcoming
Rubin data ($g\sim$24.5222https://www.lsst.org/scientists/keynumbers). We
remove five galaxies that are missing images in at least one filter, leaving
us with a combined sample of 26,126 galaxies.

As in [2], we pre-process our images in each filter $b$ by applying the
following normalization:

$x_{b}=\rm{tanh}\Biggl{(}\rm{sinh}^{-1}\Big{(}\beta\frac{x_{\rm{raw,b}}}{\langle
max(x_{\rm{raw},b})\rangle_{b}}\Big{)}\Biggr{)}$ (1)

where $\beta=10$ was chosen empirically to allow our image maxima to be well-
distributed within $[0,1]$. We then interpolate each image to 69x69 px to
reduce training times.

Next, we use the photutils package to estimate the orientation of each galaxy.
We stack the $grz$ images and subtract the median background level from the
stack, assuming a 10x10 px box size and a 3x3 px median filter. We then
convolve each background-subtracted image with a 2D Gaussian filter and
construct a catalog of all identified sources. The morphological features of
each source are calculated by default, and we extract the orientation of the
source whose centroid is closest to the center of the image. We then split our
data into an 80% training sample and a 20% validation sample.

To assess the utility of our CVAE for anomaly detection, we augment our
validation set with a sample of 294 Red Spiral galaxies from [12] and the KISS
sample of 13 Green Pea galaxies [4]. The Red Spiral galaxies were consolidated
by the 160,000 volunteers of the Galaxy Zoo project.

We construct our CVAE using the same architecture as in
[1]333https://github.com/jwuphysics/galaxy-autoencoders.. The network consists
of three convolution encoding layers, three deconvolution decoding layers, and
a latent dimension of 5. We train the network using a loss function consisting
of three terms. The first is the reconstruction loss for an image of $n$
pixels, the second is the KL divergence between the fifth latent feature and a
standard univariate Normal, and the third is the mean squared error between
the means of the first four latent distributions $\mu$ and the galaxy’s
physical properties:

$L=\beta_{0}\frac{1}{n}\sum_{i=1}^{n}(x_{i}-\hat{x_{i}})^{2}+(\sigma_{5}^{2}+\mu_{5}^{2}-\mathrm{log}(\sigma_{5})-1)+\beta_{1}\sum_{k=1}^{4}\frac{1}{\sigma_{p_{k}}^{2}}(p_{k}-\mu_{k})^{2}$
(2)

where $p$ is a vector containing the physical properties of the galaxy
(orientation, spectroscopic redshift, and survey-reported stellar mass and
star-formation rate) and $\sigma_{p}$ are their reported 1-$\sigma$
uncertainties (we assume an uncertainty of 0.1 rad for all calculated
orientations). We leave the fifth latent dimension of our CVAE free to capture
the remaining morphological information in each image. The coefficients
$\beta_{0}$ and $\beta_{1}$ are hyperparameters; we have found through
moderate tuning that setting $\beta_{0}=10$ and $\beta_{1}=50$ results in
reasonable parameter estimates without significantly degrading the quality of
reconstructed images.

As a baseline comparison, we train a vanilla CVAE with the same architecture
but excluding the physical term from the loss function (the KL-divergence is
now calculated across all 5 latent features, which are unphysical). We train
each network for 1000 epochs using the standard `Adam` optimizer [10] with a
learning rate of $5\times 10^{-6}$ and a batch size of 128 on two Nvidia A100
80GB GPUs, confirming convergence of the loss and saving the model weights at
the epoch where the validation loss is minimized. Our code and trained models
can be found at the associated Github repository for this
work444https://github.com/alexandergagliano/physics_CVAE.

##  3 Results and Discussion

Figure 1: A comparison between the learned latent features of the CVAE applied to the validation set, and the physical properties of the associated galaxies derived through SED fitting. 1-$\sigma$ uncertainties on SED-estimated parameters are shown in gray. Literature values for Green Pea galaxies are given in green, and those for Red Spirals are given in red (only redshift is reported for Red Spirals). Method | Redshift $R^{2}$ | Stellar Mass $R^{2}$
---|---|---
$(r,g,z)$ Photometry + MLP [11] | 0.69 | 0.65
Image Embedding + MLP [16] | 0.39 | 0.45
Image Embedding + MLP [11] | 0.63 | 0.57
Image Embedding + kNN [11] | 0.71 | 0.66
Image Embedding (ours) | 0.83 | 0.75
Table 1: $R^{2}$ values for the linear regression models that predict a
galaxy’s spectroscopically-derived redshift and stellar mass from its
corresponding CVAE latent feature.

The catalog-level and CVAE-estimated physical properties for the validation
sample are compared in Fig. 1. The CVAE appears to successfully recover the
physical parameters of galaxies in the validation set. The predictions are
unbiased, with mean absolute deviations of at most 0.1, for star-formation
rate. In addition, few catastrophic outliers are observed for most galaxy
properties (the scatter is large for star-formation rate, but comparable to
the error bars on publicly reported SED estimates). An exception is the
orientation $\phi$: because the parameter is periodic in nature (galaxies with
orientations $\pi/2$ and $-\pi/2$ are indistinguishable), this feature is
poorly reconstructed at high inclinations.

To quantitatively evaluate the CVAE’s performance on the validation set, we
fit a linear model between each CVAE-inferred feature and its
spectroscopically-derived counterpart. We report the $R^{2}$ value of these
regression models for spectroscopic redshift and stellar mass in Table 1,
alongside those reported for other image-based parameter inference methods
from [11]. The values from [11] were calculated on a sample of $\sim$200k
$z<0.6$ DECaLS galaxies, roughly an order of magnitude greater than the sample
considered in this work. Despite the smaller sample available for training,
our technique far outperforms the reported techniques (particularly those
trained using self-supervised learning) for estimating these galaxy
properties, and to higher redshifts ($z<1$). Our $R^{2}$ value for star-
formation rate is 0.26; though unavoidable degeneracies exist at the
photometric level, tighter constraints from SED fits would likely improve the
CVAE’s ability to infer this property in the future.

Surprisingly, our network is able to predict reasonable estimates for the
spectroscopic redshifts and stellar masses of the Red Spiral and Green Pea
galaxies from images alone, despite their anomalous colors. The redshifts of
the Green Peas are still systematically underestimated by the model, which
skews the predicted star-formation rates downward.

Figure 2: Projections of the latent features drawn for the validation set,
including the two populations of anomalies. Sub-types are given in legend. Top
row: Latent space for the vanilla CVAE, trained without physics-informed loss
constraints. Bottom Row: Latent space for our physics-informed CVAE, with two-
dimensional SVM decision boundaries overplotted. Green Peas are identified by
their compact morphologies, low stellar masses, and high star-formation rates;
Red Spirals are identified by their extended morphologies, higher stellar
masses, and lower star-formation rates.

We next construct a support vector machine (SVM) classifier to recover our
sample of anomalous galaxies using only encoded features. We use the
implementation from scikit-learn, with a radial basis function (RBF) kernel
and scaled kernel coefficients. In each of five trials, we randomly split the
combined sample into 80% train and 20% test sets. We then balance our three
classes across the training set with SMOTE and evaluate performance on the
test set. We calculate the fraction of each class recovered by the classifier
in each case, and average the model’s performance across all five trials. We
find that our CVAE recovers $94\pm 2\%$ of Red Spiral galaxies with a $5\%$
contamination rate from non-anomalous galaxies, and $67\pm 0\%$ of Green Pea
galaxies with a $2\%$ contamination rate from non-anomalous galaxies (though
we caution that each test sample only contains 2-3 Green Peas). The same
classifier trained on the latent features of the vanilla CVAE recovers $94\pm
2\%$ of Red Spiral galaxies with a $6\%$ contamination rate and $47\pm 27\%$
of Green Pea galaxies with a $4\%$ contamination rate. We conclude that the
physics-informed CVAE achieves marginally superior, and significantly more
consistent, performance over the vanilla CVAE in recovering Green Peas.

We next plot 2D projections of the latent space for the physics-informed and
vanilla CVAE models in Fig. 2. For the physics-informed plots, we include the
decision boundaries from a 2d SVM with linear kernel. Though the Red Spirals
fall toward the edges of the latent parameter distributions in the baseline,
the plots lack physical intuition. The latent space of the physics-informed
CVAE, however, reveals the properties that make these sub-populations
anomalous. Green Pea galaxies push toward the highest star-formation rates,
lowest stellar masses, and lowest “morphology” values of the sample (this can
be considered a proxy for compactness). Red Spirals, in contrast, are the most
extended objects in the sample and have low star-formation rates for their
stellar masses, with most falling in or below the ‘green valley’ that
distinguishes active from quenched galaxies [14].

Finally, we identify galaxies in our sample most similar to the labeled Green
Peas and Red Spirals. We use the NearestNeighbor method in scikit-learn to
identify ten galaxies in the non-anomalous sample with latent features most
similar to each galaxy in the anomalous sample. We present a sample of
galaxies with lowest latent-space Minkowski distance to multiple Red Spirals,
and those with lowest distance to multiple Green Peas, in Fig. 3. These
galaxies are prime targets for upcoming follow-up studies, to confirm their
associations and further characterize their evolutionary histories.

The ability to rapidly infer physical parameters competitive with state-of-
the-art SED models using only three images of a galaxy is encouraging. Once
trained, evaluation of our CVAE takes $7$ ms per galaxy on an Apple M2 Max
chip. This is substantially less than the $\sim$10 hrs required for rigorous
SED fitting using modern approaches [9], and $>100$x faster than existing
simulation-based inference (SBI) models optimized for this task [18]. With our
approach, estimating the redshift, stellar mass, and star-formation rate of
all 20B galaxies observed by the Vera C. Rubin Observatory would require only
$\sim$1500 CPU-hours (albeit without the added benefit of full posterior
distributions, as are provided by SBI). New, neural network-based inference
approaches like the one presented here will help bring scalability and
interpretability to petabyte-scale data exploration among astronomical
surveys, and facilitate the search for new galaxy sub-types.

Figure 3: Latent space nearest-neighbors for the anomalies manually flagged in
the Galaxy Zoo project. Left: Images of galaxies with latent properties most
similar to labeled “Green Pea” galaxies. Right: Images of galaxies most
similar to labeled “Red Spiral” galaxies. These galaxies are less blue than
the majority of other spirals in our sample, suggesting older average stellar
populations.

## Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for helpful feedback which improved this
manuscript. This work is supported by the National Science Foundation under
Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial
Intelligence and Fundamental Interactions, http://iaifi.org/). VAV
acknowledges support by the NSF through grant AST-2108676.

The Legacy Surveys consist of three individual and complementary projects: the
Dark Energy Camera Legacy Survey (DECaLS; Proposal ID #2014B-0404; PIs: David
Schlegel and Arjun Dey), the Beijing-Arizona Sky Survey (BASS; NOAO Prop. ID
#2015A-0801; PIs: Zhou Xu and Xiaohui Fan), and the Mayall z-band Legacy
Survey (MzLS; Prop. ID #2016A-0453; PI: Arjun Dey). DECaLS, BASS and MzLS
together include data obtained, respectively, at the Blanco telescope, Cerro
Tololo Inter-American Observatory, NSF’s NOIRLab; the Bok telescope, Steward
Observatory, University of Arizona; and the Mayall telescope, Kitt Peak
National Observatory, NOIRLab. Pipeline processing and analyses of the data
were supported by NOIRLab and the Lawrence Berkeley National Laboratory
(LBNL). The Legacy Surveys project is honored to be permitted to conduct
astronomical research on Iolkam Du’ag (Kitt Peak), a mountain with particular
significance to the Tohono O’odham Nation.

NOIRLab is operated by the Association of Universities for Research in
Astronomy (AURA) under a cooperative agreement with the National Science
Foundation. LBNL is managed by the Regents of the University of California
under contract to the U.S. Department of Energy.

This project used data obtained with the Dark Energy Camera (DECam), which was
constructed by the Dark Energy Survey (DES) collaboration. Funding for the DES
Projects has been provided by the U.S. Department of Energy, the U.S. National
Science Foundation, the Ministry of Science and Education of Spain, the
Science and Technology Facilities Council of the United Kingdom, the Higher
Education Funding Council for England, the National Center for Supercomputing
Applications at the University of Illinois at Urbana-Champaign, the Kavli
Institute of Cosmological Physics at the University of Chicago, Center for
Cosmology and Astro-Particle Physics at the Ohio State University, the
Mitchell Institute for Fundamental Physics and Astronomy at Texas A&M
University, Financiadora de Estudos e Projetos, Fundacao Carlos Chagas Filho
de Amparo, Financiadora de Estudos e Projetos, Fundacao Carlos Chagas Filho de
Amparo a Pesquisa do Estado do Rio de Janeiro, Conselho Nacional de
Desenvolvimento Cientifico e Tecnologico and the Ministerio da Ciencia,
Tecnologia e Inovacao, the Deutsche Forschungsgemeinschaft and the
Collaborating Institutions in the Dark Energy Survey. The Collaborating
Institutions are Argonne National Laboratory, the University of California at
Santa Cruz, the University of Cambridge, Centro de Investigaciones
Energeticas, Medioambientales y Tecnologicas-Madrid, the University of
Chicago, University College London, the DES-Brazil Consortium, the University
of Edinburgh, the Eidgenossische Technische Hochschule (ETH) Zurich, Fermi
National Accelerator Laboratory, the University of Illinois at Urbana-
Champaign, the Institut de Ciencies de l’Espai (IEEC/CSIC), the Institut de
Fisica d’Altes Energies, Lawrence Berkeley National Laboratory, the Ludwig
Maximilians Universitat Munchen and the associated Excellence Cluster
Universe, the University of Michigan, NSF’s NOIRLab, the University of
Nottingham, the Ohio State University, the University of Pennsylvania, the
University of Portsmouth, SLAC National Accelerator Laboratory, Stanford
University, the University of Sussex, and Texas A&M University.

BASS is a key project of the Telescope Access Program (TAP), which has been
funded by the National Astronomical Observatories of China, the Chinese
Academy of Sciences (the Strategic Priority Research Program “The Emergence of
Cosmological Structures” Grant #XDB09000000), and the Special Fund for
Astronomy from the Ministry of Finance. The BASS is also supported by the
External Cooperation Program of Chinese Academy of Sciences (Grant
#114A11KYSB20160057), and Chinese National Natural Science Foundation (Grant
#12120101003, #11433005).

The Legacy Survey team makes use of data products from the Near-Earth Object
Wide-field Infrared Survey Explorer (NEOWISE), which is a project of the Jet
Propulsion Laboratory/California Institute of Technology. NEOWISE is funded by
the National Aeronautics and Space Administration.

The Legacy Surveys imaging of the DESI footprint is supported by the Director,
Office of Science, Office of High Energy Physics of the U.S. Department of
Energy under Contract No. DE-AC02-05CH1123, by the National Energy Research
Scientific Computing Center, a DOE Office of Science User Facility under the
same contract; and by the U.S. National Science Foundation, Division of
Astronomical Sciences under Contract No. AST-0950945 to NOAO.

GAMA is a joint European-Australasian project based around a spectroscopic
campaign using the Anglo-Australian Telescope. The GAMA input catalogue is
based on data taken from the Sloan Digital Sky Survey and the UKIRT Infrared
Deep Sky Survey. Complementary imaging of the GAMA regions is being obtained
by a number of independent survey programmes including GALEX MIS, VST KiDS,
VISTA VIKING, WISE, Herschel-ATLAS, GMRT and ASKAP providing UV to radio
coverage. GAMA is funded by the STFC (UK), the ARC (Australia), the AAO, and
the participating institutions. The GAMA website is http://www.gama-
survey.org/.

## References

  * [1] What do galaxies look like? learning from variational autoencoders.  https://jwuphysics.substack.com/p/galaxy-autoencoders.  Accessed: 2023-09-29.
  * Arcelin et al. [2021a] B. Arcelin, C. Doux, E. Aubourg, C. Roucelle, and LSST Dark Energy Science Collaboration.  Deblending galaxies with variational autoencoders: A joint multiband, multi-instrument approach.  _Monthly Notices of the RAS_ , 500(1):531–547, Jan. 2021a.  doi: 10.1093/mnras/staa3062.
  * Arcelin et al. [2021b] B. Arcelin, C. Doux, E. Aubourg, C. Roucelle, and LSST Dark Energy Science Collaboration.  Deblending galaxies with variational autoencoders: A joint multiband, multi-instrument approach.  _Monthly Notices of the RAS_ , 500(1):531–547, Jan. 2021b.  doi: 10.1093/mnras/staa3062.
  * Brunker et al. [2022] S. W. Brunker, J. J. Salzer, B. Kimsey-Miller, and B. Cousins.  The Environments of Green Pea Galaxies. I. The KISS Sample.  _Astrophysical Journal_ , 926(2):131, Feb. 2022.  doi: 10.3847/1538-4357/ac469f.
  * Cardamone et al. [2009] C. Cardamone, K. Schawinski, M. Sarzi, S. P. Bamford, N. Bennert, C. M. Urry, C. Lintott, W. C. Keel, J. Parejko, R. C. Nichol, D. Thomas, D. Andreescu, P. Murray, M. J. Raddick, A. Slosar, A. Szalay, and J. Vandenberg.  Galaxy Zoo Green Peas: discovery of a class of compact extremely star-forming galaxies.  _Monthly Notices of the RAS_ , 399(3):1191–1205, Nov. 2009.  doi: 10.1111/j.1365-2966.2009.15383.x.
  * Cavanagh et al. [2021] M. K. Cavanagh, K. Bekki, and B. A. Groves.  Morphological classification of galaxies with deep learning: comparing 3-way and 4-way CNNs.  _Monthly Notices of the RAS_ , 506(1):659–676, Sept. 2021.  doi: 10.1093/mnras/stab1552.
  * Cheng et al. [2021] T.-Y. Cheng, M. Huertas-Company, C. J. Conselice, A. Aragón-Salamanca, B. E. Robertson, and N. Ramachandra.  Beyond the hubble sequence - exploring galaxy morphology with unsupervised machine learning.  _Monthly Notices of the RAS_ , 503(3):4446–4465, May 2021.  doi: 10.1093/mnras/stab734.
  * Driver et al. [2022] S. P. Driver, S. Bellstedt, A. S. G. Robotham, I. K. Baldry, L. J. Davies, J. Liske, D. Obreschkow, E. N. Taylor, A. H. Wright, M. Alpaslan, S. P. Bamford, A. E. Bauer, J. Bland-Hawthorn, M. Bilicki, M. Bravo, S. Brough, S. Casura, M. E. Cluver, M. Colless, C. J. Conselice, S. M. Croom, J. de Jong, F. D’Eugenio, R. De Propris, B. Dogruel, M. J. Drinkwater, A. Dvornik, D. J. Farrow, C. S. Frenk, B. Giblin, A. W. Graham, M. W. Grootes, M. L. P. Gunawardhana, A. Hashemizadeh, B. Häußler, C. Heymans, H. Hildebrandt, B. W. Holwerda, A. M. Hopkins, T. H. Jarrett, D. Heath Jones, L. S. Kelvin, S. Koushan, K. Kuijken, M. A. Lara-López, R. Lange, Á. R. López-Sánchez, J. Loveday, S. Mahajan, M. Meyer, A. J. Moffett, N. R. Napolitano, P. Norberg, M. S. Owers, M. Radovich, M. Raouf, J. A. Peacock, S. Phillipps, K. A. Pimbblet, C. Popescu, K. Said, A. E. Sansom, M. Seibert, W. J. Sutherland, J. E. Thorne, R. J. Tuffs, R. Turner, A. van der Wel, E. van Kampen, and S. M. Wilkins.  Galaxy And Mass Assembly (GAMA): Data Release 4 and the z < 0.1 total and z < 0.08 morphological galaxy stellar mass functions.  _Monthly Notices of the RAS_ , 513(1):439–467, June 2022.  doi: 10.1093/mnras/stac472.
  * Johnson et al. [2021] B. D. Johnson, J. Leja, C. Conroy, and J. S. Speagle.  Stellar Population Inference with Prospector.  _Astrophysical Journal, Supplement_ , 254(2):22, June 2021.  doi: 10.3847/1538-4365/abef67.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A Method for Stochastic Optimization.  _arXiv e-prints_ , art. arXiv:1412.6980, Dec. 2014.  doi: 10.48550/arXiv.1412.6980.
  * Lanusse et al. [2023] F. Lanusse, L. Parker, S. Golkar, M. Cranmer, A. Bietti, M. Eickenberg, G. Krawezik, M. McCabe, R. Ohana, M. Pettee, B. Regaldo-Saint Blancard, T. Tesileanu, K. Cho, and S. Ho.  AstroCLIP: Cross-Modal Pre-Training for Astronomical Foundation Models.  _arXiv e-prints_ , art. arXiv:2310.03024, Oct. 2023.  doi: 10.48550/arXiv.2310.03024.
  * Masters et al. [2010] K. L. Masters, M. Mosleh, A. K. Romer, R. C. Nichol, S. P. Bamford, K. Schawinski, C. J. Lintott, D. Andreescu, H. C. Campbell, B. Crowcroft, I. Doyle, E. M. Edmondson, P. Murray, M. J. Raddick, A. Slosar, A. S. Szalay, and J. Vandenberg.  Galaxy Zoo: passive red spirals.  _Monthly Notices of the RAS_ , 405(2):783–799, June 2010.  doi: 10.1111/j.1365-2966.2010.16503.x.
  * Ravanbakhsh et al. [2016] S. Ravanbakhsh, F. Lanusse, R. Mandelbaum, J. Schneider, and B. Poczos.  Enabling Dark Energy Science with Deep Generative Models of Galaxy Images.  _arXiv e-prints_ , art. arXiv:1609.05796, Sept. 2016.  doi: 10.48550/arXiv.1609.05796.
  * Salim [2014] S. Salim.  Green Valley Galaxies.  _Serbian Astronomical Journal_ , 189:1–14, Dec. 2014.  doi: 10.2298/SAJ1489001S.
  * Schawinski et al. [2009] K. Schawinski, C. Lintott, D. Thomas, M. Sarzi, D. Andreescu, S. P. Bamford, S. Kaviraj, S. Khochfar, K. Land, P. Murray, R. C. Nichol, M. J. Raddick, A. Slosar, A. Szalay, J. VandenBerg, and S. K. Yi.  Galaxy Zoo: a sample of blue early-type galaxies at low redshift*.  _Monthly Notices of the RAS_ , 396(2):818–829, June 2009.  doi: 10.1111/j.1365-2966.2009.14793.x.
  * Stein et al. [2021] G. Stein, P. Harrington, J. Blaum, T. Medan, and Z. Lukic.  Self-supervised similarity search for large scientific datasets.  _arXiv e-prints_ , art. arXiv:2110.13151, Oct. 2021.  doi: 10.48550/arXiv.2110.13151.
  * Torabi et al. [2023] H. Torabi, S. L. Mirtaheri, and S. Greco.  Practical autoencoder based anomaly detection by using vector reconstruction error.  _Cybersecurity_ , 6(1):1, 2023.
  * Wang et al. [2023] B. Wang, J. Leja, V. A. Villar, and J. S. Speagle.  SBI++: Flexible, Ultra-fast Likelihood-free Inference Customized for Astronomical Applications.  _Astrophysical Journal, Letters_ , 952(1):L10, July 2023.  doi: 10.3847/2041-8213/ace361.
  * Zou et al. [2022] F. Zou, W. N. Brandt, C.-T. Chen, J. Leja, Q. Ni, W. Yan, G. Yang, S. Zhu, B. Luo, K. Nyland, F. Vito, and Y. Xue.  Spectral Energy Distributions in Three Deep-drilling Fields of the Vera C. Rubin Observatory Legacy Survey of Space and Time: Source Classification and Galaxy Properties.  _Astrophysical Journal, Supplement_ , 262(1):15, Sept. 2022.  doi: 10.3847/1538-4365/ac7bdf.