removed. The most general finite renormalizations would be:

$\displaystyle
N(C_{a}^{\mu}(x_{1}),C_{b}^{\nu}(x_{2}))=i~{}\delta(x_{1}-x_{2})~{}R_{ab}^{\mu\nu}(x_{2})$
$\displaystyle
N(C_{a}^{\mu\nu}(x_{1}),C_{b}^{\nu}(x_{2}))=i~{}\delta(x_{1}-x_{2})~{}R_{ab}^{\mu\nu,\rho}(x_{2})$
(6.106)

where the generic forms are:

$\displaystyle
R_{ab}^{\mu\nu}=f_{abcd}^{(1)}~{}:v_{c}^{\mu}v_{d}^{\nu}:+f_{abcd}^{(2)}~{}\eta^{\mu\nu}:v_{c\rho}v_{d}^{\rho}:+f_{abcd}^{(3)}~{}\eta^{\mu\nu}:u_{c}\tilde{u}_{d}:$
$\displaystyle
R_{ab}^{\mu\nu,\rho}=f_{abcd}^{(4)}~{}(\eta^{\mu\rho}:u_{c}v_{d}^{\nu}:-\eta^{\nu\rho}:u_{c}v_{d}^{\mu}:)$
(6.107)

with

$f_{abcd}^{(1)}=f_{badc}^{(1)},\quad
f_{abcd}^{(2)}=f_{bacd}^{(2)}=f_{abdc}^{(2)},\quad
f_{abcd}^{(3)}=f_{bacd}^{(3)}.$ (6.108)

We insert everything in the equation

$\displaystyle\delta(x_{1}-x_{2})~{}{\cal A}(C_{a}^{\mu},C_{b}^{\nu})(x_{2})=$
$\displaystyle
d_{Q}NC_{a}^{\mu}(x_{1}),C_{b}^{\nu}(x_{2}))-i\partial_{\rho}^{1}N(C_{a}^{\mu\rho}(x_{1}),C_{b}^{\nu}(x_{2}))-i\partial_{\rho}^{2}N(C_{a}^{\mu}(x_{1}),C_{b}^{\nu\rho}(x_{2}))$
(6.109)

and after some computations we obtain $f_{abc}=0$ which is not possible.
$\blacksquare$

However, the Wick property is preserved.

###### Theorem 6.10

The following relations are true: The finite renormalizations (6.26) and
(6.61) preserve Wick expansion property. Explicitly we have:

$\displaystyle v_{b}^{\nu}\cdot
N(C_{a}^{\mu},T)=f_{abc}~{}N(F_{c}^{\nu\mu},T)+N(C_{a}^{\mu},C_{b}^{\nu})$
$\displaystyle v_{b}^{\rho}\cdot
N(C_{a}^{\mu},T^{\nu})=f_{abc}~{}N(F_{c}^{\rho\mu},T^{\nu})+N(C_{a}^{\mu},C_{b}^{\rho\nu})$
$\displaystyle u_{b}\cdot N(C_{a}^{\mu},T^{\nu})=N(C_{a}^{\mu},C_{b}^{\nu})$
$\displaystyle v_{b}^{\rho}\cdot
N(C_{a}^{\mu\nu},T)=N(C_{a}^{\nu\mu},C^{\rho}_{b})$ $\displaystyle u_{b}\cdot
N(C_{a}^{\mu\nu},T)=f_{abc}~{}N(F_{c}^{\mu\nu},T)+N(C_{a}^{\mu\nu},D_{b})$
$\displaystyle v_{b}^{\sigma}\cdot
N(C_{a}^{\mu\nu},T^{\rho})=N(C_{a}^{\mu\nu},C_{b}^{\sigma\rho})$
$\displaystyle u_{b}\cdot
N(C_{a}^{\mu\nu},T^{\rho})=-f_{abc}~{}N(F_{c}^{\mu\nu},T^{\rho})-N(C_{a}^{\mu\nu},C_{b}^{\rho})$
$\displaystyle v_{b}^{\rho}\cdot
N(C_{a}^{\mu},T^{\rho\sigma})=f_{abc}~{}N(F_{c}^{\nu\mu},T^{\rho\sigma})$
$\displaystyle u_{b}\cdot
N(C_{a}^{\mu},T^{\rho\sigma})=-N(C_{a}^{\mu},C_{b}^{\rho\sigma})$ (6.110)

and

$N(v_{a}^{\mu},T^{I})=0,\quad N(u_{a},T^{I})=0,\quad
N(\tilde{u}_{a},T^{I})=0.$ (6.111)

We also have:

$\displaystyle v_{c}^{\rho}\cdot
N(C_{a}^{\mu},C_{b}^{\nu})=f_{acd}~{}N(F_{d}^{\rho\mu},C_{b}^{\nu})+f_{bcd}~{}N(C_{a}^{\mu},F_{d}^{\rho\nu})$
$\displaystyle v_{c}^{\lambda}\cdot
N(C_{a}^{\mu\nu},C_{b}^{\rho})=f_{bcd}~{}N(C_{a}^{\mu\nu},F_{d}^{\lambda\rho})$
$\displaystyle u_{c}\cdot
N(C_{a}^{\mu\nu},C_{b\rho})=f_{acd}~{}N(F_{d}^{\mu\nu},C_{b\rho})-f_{bcd}~{}N(C_{a}^{\mu\nu},\tilde{u}_{d,\rho})$
$\displaystyle u_{c}\cdot
N(C_{a}^{\mu\nu},C_{b}^{\rho\sigma})=-f_{acd}~{}N(F_{d}^{\mu\nu},C_{b}^{\rho\sigma})+f_{bcd}~{}N(C_{a}^{\mu\nu},F_{d}^{\rho\sigma})$
(6.112)

$\displaystyle v_{b}^{\rho}\cdot
N(F_{a}^{\mu\nu},T)=N(F_{a}^{\mu\nu},C_{b}^{\rho})$ $\displaystyle
v_{b}^{\sigma}\cdot
N(F_{a}^{\mu\nu},T^{\rho})=-N(F_{a}^{\mu\nu},C_{b}^{\rho\sigma})$
$\displaystyle u_{b}\cdot
N(F_{a}^{\mu\nu},T^{\rho})=-N(F_{a}^{\mu\nu},C_{b}^{\rho})$ $\displaystyle
u_{b}\cdot N(F_{a}^{\mu\nu},T^{\rho\sigma})=0.$ (6.113)

and

$\displaystyle v_{c}^{\sigma}\cdot
N(F_{a}^{\mu\nu},C_{b}^{\rho})=-f_{bcd}~{}N(F_{a}^{\mu\nu},F_{d}^{\rho\sigma})$
$\displaystyle u_{c}\cdot
N(F_{a}^{\mu\nu},C_{b}^{\rho\sigma})=f_{bcd}~{}N(F_{a}^{\mu\nu},F_{b}^{\rho\sigma})$
(6.114)

Proof: We start with the relation

$\displaystyle~{}[v_{b}^{\mu}(y),T(C_{a}^{\mu}(x_{1}),T(x_{2}))]=$
$\displaystyle
i~{}f_{abc}~{}D_{0}(y-x_{1})~{}T(F_{c}^{\nu\mu}(x_{1}),T(x_{2}))$
$\displaystyle-i~{}f_{abc}~{}\partial_{\mu}D_{0}(y-x_{1})~{}[\eta^{\mu\rho}~{}T(v_{c}^{\nu}(x_{1}),T(x_{2}))-\eta^{\mu\nu}~{}T(v_{c}^{\rho}(x_{1}),T(x_{2}))]$
$\displaystyle+i~{}D_{0}(y-x_{2})~{}T(C_{a}^{\mu}(x_{1}),C_{b}^{\nu}(x_{2}))-i~{}\partial_{\rho}D_{0}(y-x_{1})~{}T(C_{a}^{\mu}(x_{1}),E_{b}^{\nu\rho}(x_{2}))$
(6.115)

following from Wick expansion property (2.22). If we consider finite
renormalizations of the type (6.61), (6.80) and (6.100),we will get new terms
in the left and right hand of the preceding identity. The identity is
preserved iff we have the first relation from (6.110) and some relations from
(6.67) and (6.111).

The rest of the consistency relations are going in the same way.
$\blacksquare$

##  7 Finite Renormalizations

We have proved in Theorem 6.27 that the anomalies can be eliminated by adding
to the chronological products some quasi-local operators. However, the
chronological products have still some arbitrariness: in principle, we can add
other quasi-local operators in such a way that gauge invariance in the second
order is preserved. This arbitrariness is described by

###### Theorem 7.1

The finite renormalizations are of the type

$R(T^{I}(x_{1}),T^{J}(x_{2}))=\delta(x_{1}-x_{2})~{}N(T^{I},T^{J})(x_{2})$
(7.1)

where the polynomials $N(T^{I},T^{J})$ verify the symmetry property:

$N(T^{I},T^{J})=(-1)^{|I||J|}~{}N(T^{J},T^{I})$ (7.2)

and

$gh(N(T^{I},T^{J}))=|I|+|J|,\qquad\omega(N(T^{I},T^{J}))\leq 4.$ (7.3)

These finite renormalizations do not produce anomalies iff there exists
expresions $N^{I}$ such that

$N(T^{I},T^{J})=N^{JI}$ (7.4)

and

$d_{Q}N^{I}=i~{}d_{\mu}N^{I\mu}.$ (7.5)

Proof: The general form of the finite anomaly (7.1) follows form Bogoliubov
axioms (power counting and ghost number asignement). We now want that this
redefinition of the chronological products does not create new anomalies i.e.
the expresion

$\displaystyle sR(T^{I}(x_{1}),T^{J}(x_{2}))\equiv
d_{Q}R(T^{I}(x_{1}),T^{J}(x_{2}))$
$\displaystyle-i~{}\partial_{\mu}^{1}R(T^{I\mu}(x_{1}),T^{J}(x_{2}))-i~{}(-1)^{|I|}~{}\partial_{\mu}^{2}R(T^{I}(x_{1}),T^{J\mu}(x_{2}))$
(7.6)

is null. We insert (7.1) and by direct computation we have:

$\displaystyle
sR(T^{I}(x_{1}),T^{J}(x_{2}))=\delta(x_{1}-x_{2})~{}[d_{Q}N(T^{I},T^{J})-i~{}(-1)^{|I|}~{}\partial_{\mu}N(T^{I},T^{J\mu})](x_{2})$
$\displaystyle-i~{}\partial_{\mu}\delta(x_{1}-x_{2})~{}[N(T^{I\mu},T^{J})-(-1)^{|I|}~{}N(T^{I},T^{J\mu})](x_{2})$
(7.7)

and this expresion is null iff

$\displaystyle
d_{Q}N(T^{I},T^{J})-i~{}(-1)^{|I|}~{}\partial_{\mu}N(T^{I},T^{J\mu})=0$
$\displaystyle N(T^{I\mu},T^{J})-(-1)^{|I|}~{}N(T^{I},T^{J\mu})$ (7.8)

We define

$N^{I}\equiv N(T^{\emptyset},T^{I})$ (7.9)

and we can use the second relation (7.8) to prove (7.4) by induction. The
relation (7.5) follows immediatly from the first relation (7.8).
$\blacksquare$

It remains to investigate the solution of equation (7.5) in the pure Yang-
Mills case, with the restrictions

$gh(N^{I})=|I|,\qquad\omega(N^{I})\leq 4.$ (7.10)

In canonical dimension $\omega=4$ we have the tri-linear solution (2.44) and
one can easily prove that there are no bilinear and quadri-linear solutions.
In canonical dimension $\omega=3$ there are no solution and, finally, in
canonical dimension $\omega=2$ we find the solution

$\displaystyle N=f_{ab}~{}\Bigl{(}{1\over
2}~{}v_{a}^{\mu}~{}v_{b\mu}+u_{a}~{}\tilde{u}_{b}\Bigl{)}$ $\displaystyle
N^{\mu}=f_{ab}~{}u_{a}~{}v_{b}^{\mu}$ (7.11)

with $f_{ab}$ symmetric in $a\leftrightarrow b.$ However we cannot find
$N^{\mu\nu}$ such that

$d_{Q}N^{\mu}=i~{}d_{\nu}N^{\mu\nu}$ (7.12)

so we remanin only with the tri-linear solution (2.44). It follows that we
have only one constant arbitrariness in the second order of the perturbation
theory. In fact, this result remains true in all orders of the perturbation
theory.

##  8 Conclusions

In further papers we will extend the Hopf structure and the Wick expansion
property to the case of the general Yang-Mills model (including massive vector
Bosons and Dirac fields) and to gravity. An interesting point would be to see
if our Hopf version of the Wick expansion property is connected with the
Feynman graph version.

## References

  * [1] A. Aste, G. Scharf, M. Duetsch, “On gauge invariance and spontaneous symmetry breaking”, arXiv: 9705216, J. Phys. A: Math. Gen. 30 (1997) 5785 - 5792
  * [2] N. N. Bogoliubov, D. Shirkov, “Introduction to the Theory of Quantized Fields”, John Wiley and Sons, 1976 (3rd edition)
  * [3] N. N. Bogolubov, A. A. Logunov, A.I. Oksak, I. Todorov, “General Principles of Quantum Field Theory”, Kluwer 1989
  * [4] C. Brouder, B. Fauser, A. Frabetti, R. Oeckl, “Quantum field theory and Hopf algebra cohomology”, arXiv:hep-th/0311253, J. Phys. A 37 (2004) 5895 - 5927
  * [5] C. Brouder, “Quantum field theory meets Hopf algebra”, arXiv: 0611153v3, Math. Nachr. 282 (2009) 1664 - 1690
  * [6] M. Dütsch, “From Classical Field Theory to Perturbative Quantum Field Theory”, Progress in Mathematical Physics 74, Springer 2019
  * [7] M. Dütsch, K. Fredenhagen, “Algebraic Quantum Field Theory, Perturbation Theory, and the Loop Expansion”, arXiv: hep-th/0001129, Commun. Math. Phys. 219 (2001) 5 - 30
  * [8] M. Duetsch, F. Krahe, G. Scharf, “Scalar QED Revisited”, Il Nuovo Cimento 106 A (3) (1993) 277 - 307
  * [9] H. Epstein, V. Glaser, “The Rôle of Locality in Perturbation Theory”, Ann. Inst. H. Poincaré 19 A (1973) 211-295
  * [10] V. Glaser, “Electrodynamique Quantique”, L’enseignement du 3e cycle de la physique en Suisse Romande (CICP), Semestre d’hiver 1972/73
  * [11] D. R. Grigore, “Ward Identities and Renormalization of General Gauge Theories”, Journ. Phys. A 37 (2004) 2803-2834
  * [12] D. R. Grigore, “A Generalization of Gauge Invariance”, arXiv: hep-th/1612.04998, Journal of Mathematical Physics 58 (2017) 082303
  * [13] D. R. Grigore, “Anomaly-Free Gauge Models: A Causal Approach”, arXiv: hep-th/1804.08276, Romanian Journ. Phys. 64 (2019) 102
  * [14] K. Hepp, “Renormalization Theory”, in “Statistical Mechanics and Quantum Field Theory” pp. 429 - 500, (Les Houches 1970), C. DeWitt-Morette, Raymond Stora (eds.), Gordon and Breach 1971
  * [15] D. Kreimer, “On the Hopf Algebra Structure of Perturbative Quantum Field Theories”, Adv. Theor. Math. Phys. 2 (1998) 303 - 334
  * [16] D. Kreimer, “Locality, QED and Classical Electrodynamics”, Ann. Phys. (Leipzig) 7 (1998) 687 - 694
  * [17] D. Kreimer, “Anatomy of a Gauge Theory”, Annals of Physics 321 (2006) 2757 - 2781
  * [18] J. Polchinski, “Renormalization and Effective Lagrangians”, Nucl. Phys. B 231 (1984) 269 - 295
  * [19] G. Popineau, R. Stora, “A Pedagogical Remark on the Main Theorem of Perturbative Renormalization Theory”, Nuclear Physics B 912 (2016) 70 - 78
  * [20] M. Salmhofer, “Renormalization: An Introduction”, (Theoretical and Mathematical Physics) Springer 1999
  * [21] G. Scharf, “Finite Quantum Electrodynamics: The Causal Approach”, (second edition) Springer, 1995; (third edition) Dover, 2014
  * [22] G. Scharf, “Quantum Gauge Theories. A True Ghost Story”, John Wiley, 2001, “Quantum Gauge Theories - Spin One and Two”, Google books, 2010 and “Gauge Field Theories: Spin One and Spin Two, 100 Years After General Relativity”, Dover 2016
  * [23] R. Stora, “Lagrangian Field Theory”, Les Houches lectures, Gordon and Breach, N.Y., 1971, C. De Witt, C. Itzykson eds.
  * [24] O. Steinmann, “Perturbation Expansions in Axiomatic Field Theory”, Lect. Notes in Phys. 11, Springer, 1971
  * [25] W. D. van Suijlekom “The Hopf Algebra of Feynman Graphs in Quantum Electrodynamics”, Letters in Mathematical Physics 77 (2006) 265 - 281
  * [26] W. D. van Suijlekom, “Renormalization of Gauge Fields: A Hopf Algebra Approach”, Commun. Math. Phys. 276 (2007) 773 - 798
  * [27] A. S. Wightman, L. Gårding, “Fields as Operator-Valued Distributions in Relativistic Quantum Field Theory”, Arkiv Fysik 28 (1965) 129-184


# Generative Visual Instruction Tuning

Jefferson Hernandez1, Ruben Villegas2, Vicente Ordonez1
1Rice University 2Google DeepMind

###### Abstract

We propose to use machine-generated instruction-following data to improve the
zero-shot capabilities of a large multimodal model with additional support for
generative and image editing tasks. We achieve this by curating a new
multimodal instruction-following set using GPT-4V and existing datasets for
image generation and editing. Using this instruction set and the existing
LLaVA-Finetune instruction set for visual understanding tasks, we produce
GenLLaVA, a Generative Large Language, and Visual Assistant. GenLLaVA is built
through a strategy that combines three types of large pretrained models
through instruction finetuning: LLama for language modeling, SigLIP for image-
text matching, and StableDiffusion for text-to-image generation. Our model
demonstrates visual understanding capabilities on par with LLaVA and
additionally demonstrates competitive results with native multimodal models
such as Unified-IO 2, paving the way for building advanced general-purpose
visual assistants by effectively re-using existing multimodal models. We open-
source our dataset, codebase, and model checkpoints to foster further research
and application in this
domain111https://github.com/jeffhernandez1995/GenLlaVA.git.

##  1 Introduction

Large multimodal models have become increasingly popular in the research
community as they are one of the key building blocks for general-purpose
assistants [1, 40, 2]. The recently proposed LLaVA [25] model is among the
latest wave of works that have demonstrated the effectiveness of instruction
tuning for multimodal models [23, 51, 41, 14]. There has been a considerable
amount of work building on top of the LLaVA model ranging from image
generation [17, 36, 35], grounding [44], image editing [7] to video
understanding [20]. These works share the same principles; they extend the
ideas of visual instruction tuning to one or more capabilities. However, after
adding a new capability (i.e., image generation), the resulting models often
lose some, if not most, of their visual and language understanding
capabilities. This contrasts with the original visual instruction tuning in
which the models retained their language capabilities after training for
visual understanding.

Figure 1: Comparison of GenLLaVA compared to recent architectures, unlike
BLIP-2 [19], we use a Linear projector similar to the LlaVA architecture [25].
Generation capabilities are added using a diffusion model, but unlike GILL
[17], we use a Q-former as the generation head. Finally, our model benefits
from using a stronger visual encoder, namely SigLIP[49]; a stronger LLM,
namely Mistral-7b [12]; and a stronger diffuser, namely SDv1.4 [33]. ∗ L
stands for Linear projection, and Q stands for Q-former resampler.

In this paper, we present the generative visual instruction tunining, in which
we teach a Large Multimodal Model (LMM) image understanding, image generation,
and image editing tasks without significantly diminishing the performance of
each individual capability. (See Fig. 1 for an overview of our method). To our
knowledge, this is the first time such capability has been achieved, and our
findings pave the way for building a general-purpose visual assistant. Our
contributions are the following:

  * •

Generative multimodal instruction-following data. Inspired by previous work
[25], we curate a multimodal instruction tuning set that combines image
understanding, image generation, and image editing data.

  * •

New improved training recipe. We empirically design a single-phase training
recipe to teach the LMM the necessary capabilities, unlike its predecessor,
which requires multi-phase training recipes.

  * •

Open source. We release the following assets to the public: the generated
multimodal instruction data, the codebase, the model checkpoints, and a visual
chat demo.

##  2 Related Work

#### Large Multimodal Models (LMMs)

Large Multimodal Models (LMMs) refer to large language models that can
understand various modalities beyond human language. Some research efforts are
focused on combining image, audio, video, and other modalities with language
[50, 27], while others aim to enhance the fusion of vision knowledge and
language. For example, BLIP-2 [19] created a large-scale image captioning
dataset and paired a language model with a vision encoder to produce a robust
multimodal model. Following this, LLaVA [25] developed a cost-effective
approach to train an advanced LLM through visual instruction tuning. Although
LLaVA-Next [24] improved performance for single-image tasks, it required over
2,000 tokens per image, which is about four times more than the original
LLaVA. More recent models such as QwenVL[2], CogVLM[42], and Yi-VL[45] follow
architectures similar to those of LLaVA. Our proposed method does not only
focus on models for multimodal understanding, but also on adding generative
capabilities to such models.

#### Generation with LMMs

In recent years, the design of methods that aim to adapt LLMs and LMMs for
generative tasks has become an active research direction. For instance, GILL
[17] translates the hidden representations of an LLM into embeddings that
correspond to a text-to-image model by learning a neural network to perform
efficient mapping using the text encoder of the diffusion model. AnyGPT [50]
discretizes data from multiple modalities, extends the existing LLM vocabulary
to add the extra modalities, and incorporates new randomly initialized
parameters that enable additional input embeddings and prediction outputs.
MGIE [7] adapts the text embedder, image input adapter, and LM head output
parameters of an LMM jointly with a diffusion model for image editing from
instructions. Codi-2 [38] follows a similar line and adds multiple decoders
for image, audio, and video generation to condition an LLM to generate
interleaved outputs from different modalities. CM3leon [46] proposes a token-
based decoder-only LMM based on the CM3 architecture that is capable of both
text and image generation and editing. Unified-IO 2 [27] and Chameleon [39]
and GPT-4o take the early-fusion fully multimodal approach training the model
from scratch to be able to expand the number of supported tasks and
modalities.

##  3 Method

Figure 2: Editing capabilities of our model. GPT4 currently uses a version of
the DALLE text-to-image model as a tool and hence is not directly able to edit
images. GPT4o instead uses tools through Python generated code to accomplish
the requested action. Our model, GenLLaVA, connects input features obtained
from CLIP to a language model that also produces output embeddings for a text-
to-image StableDiffusion model, achieving an end-to-end editing task with a
multimodal model.

###  3.1 Background: Large Multimodal Models

Large language models (LLMs) excel in natural language generation. Large
Multimodal Models enhance LLMs with the ability to interpret images and
respond accordingly. Built upon a pre-trained LLM, the LMM incorporates a
visual encoder (e.g., CLIP/L [31]) to derive visual features $f$, along with
an adapter $\mathcal{W}$ that maps $f$ into the language domain. Following the
training methodology of LLaVA [25], this process is encapsulated in the
equation:

$\displaystyle\mathcal{C}$ $\displaystyle=\\{x_{1},x_{2},\ldots,x_{l}\\},$ (1)
$\displaystyle f$ $\displaystyle=\text{Enc}_{\text{vis}}(\mathcal{V}),$
$\displaystyle x_{t}$
$\displaystyle=\text{LMM}(\\{x_{1},\ldots,x_{t-1}\\}~{}|~{}\mathcal{W}(f)),$

where $l$ represents the length of the word tokens within $\mathcal{C}$. The
set $\mathcal{C}$ can represent an image caption (Features Alignment) or
multimodal instruction-following data (Instruction Tuning). The LMM employs
the standard autoregressive method for next-token prediction, allowing it to
function as a visual assistant across diverse tasks such as visual question
answering and complex reasoning. Despite gaining visual perceptive abilities
through this training, its responses are currently constrained to text.

###  3.2 Visual Generation in Large Multimodal Models

We append $N$ visual tokens [IMG] after the instruction, with their word
embeddings being trainable. The LMM learns to generate these tokens through
its language modeling (LM) head. These visual tokens represent visual-related
instruction comprehension within $\mathcal{E}$ and form a bridge between the
language and vision modalities. We follow the same visual generation framework
of GILL [17] and MGIE [7] in extracting visual features, which we summarize
here for succintness.

We employ a generation head $\mathcal{T}$ to convert [IMG] into concrete
visual guidance. The model $\mathcal{T}$ is a sequence-to-sequence model that
translates the sequential visual tokens from the LMM into the semantically
meaningful latent set $\mathcal{U}=\\{u_{1},u_{2},\ldots,u_{L}\\}$ for visual
guidance:

$u_{t}=\mathcal{T}(\\{u_{1},\ldots,u_{t-1}\\}~{}|~{}\\{e_{\texttt{[IMG]}}+h_{\texttt{[IMG]}}\\}),$
(2)

where $e$ denotes the word embedding and $h$ is the hidden state (from the
final layer of the LMM before the LM head) of [IMG]. Specifically, the
transformation applied to $e$ serves as a broad visual representation, while
$h$ provides an instance-specific visual latent that reflects both the
original image and the text conditioning the generation.

Table 1: Main result. Comparison of various models across advanced knowledge and general understanding. ⋆ MGIE was not originally designed for these tasks, and it is purely an editing model; we take the generated caption as the answer for VQA and give a blank image as a prompt when we ask it to generate completely new images. We intend to show that models lose previous capabilities when we add a new one. Model name | Advanced Knowlege | General Understanting | Editing | Generation
---|---|---|---|---
MathVista | MMMU | MMVet | SEED-B | MMB | EVR | CC3M | COCO
GILL [17] | 18.6 | 28.8 | 23.3 | 52.5 | 38.2 | 30.4 | 15.3 | 0.67
AnyGPT [50] | 24.4 | 30.6 | 33.1 | 44.5 | 36 | 40.3 | 14.3 | 0.65
MGIE [7]⋆ | 15.5 | 25.6 | 13 | 28.8 | 6.6 | 71.5 | 13.6 | 0.66
Unified-IO 2 [27] | 30.7 | 35 | 36.6 | 61.8 | 71.5 | 50.2 | 13.4 | 0.72
GenLLaVA (Ours) | 24.9 | 29.7 | 33.1 | 63.5 | 65 | 64.7 | 14.3 | 0.71

To guide image generation with the visual latent information $\mathcal{U}$, we
employ a latent diffusion model [32], incorporating a variational autoencoder
(VAE) for handling denoising diffusion in the latent space. First, we encode
the desired visual output via the diffusion model encoder
$o=\text{Enc}_{\text{VAE}}(\mathcal{O})$; this output may be intended for
image generation or editing tasks. The diffusion process progressively
introduces noise into $o$ as $z_{t}$, increasing the noise level over
timesteps $t$. We then train the UNet $\epsilon_{\theta}$ to predict the added
noise [10]. The diffusion process is conditioned on the visual latent
information $\mathcal{U}$ through a cross-attention layer, defined as
$\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{\text{dim}}})\cdot
V$, where:

$Q=W^{(i)}_{Q}\cdot\varphi_{i}(z_{t}),K=W^{(i)}_{K}\cdot\\{u\\},V=W^{(i)}_{V}\cdot\\{u\\},$
(3)

with $\varphi$ representing the flattening operation, and $W^{(i)}_{Q}$,
$W^{(i)}_{K}$, and $W^{(i)}_{V}$ being learnable attention matrices. We apply
classifier-free guidance [11], where the score estimation $s_{\theta}$ is
extrapolated to deviate from the unconditional $\varnothing$, following
standard practices in diffusion models.

##  4 Experiment Settings

Multimodal instruction tuning is a crucial process that equips the model with
a wide range of skills and capabilities across different modalities while also
enabling it to adapt to novel and unique instructions. We build the multimodal
instruction tuning dataset by aggregating a diverse set of supervised datasets
and tasks. Each task is provided with a clear prompt, either by using existing
prompts or crafting new ones using GPT4-V.

#### Natural Language. [1.93%]

We use the ShareGPT [34] dataset. We filter out invalid conversations, but
unlike Vicuna, long conversations that surpass 2048 tokens are truncated
rather than split into multiple conversations. This results in $\sim$40K
conversations

#### Image Generation. [26.88%]

For image generation, we use the original LLaVA-Pretrain, which we now invert
to be an image generation dataset instead of an image caption dataset; this
dataset contains $\sim$558K data points collected from the LAION, SBU, and
CC3M datasets and captioned by the BLIP-2 model.

#### Image Editing. [9.63%]

We subsample the IPr2Pr [3] dataset as our editing data. We use $\sim$200K
CLIP-filtered data, where instructions are extracted by GPT-3, and images are
synthesized by the Prompt-to-Prompt model [8].

#### Image Understanting.[61.56%]

We combine the original LLaVA-Finetune dataset with the LVIS-INSTRUCT4V[41], a
new visual instruction tuning dataset constructed in the same way as the
original LlaVA dataset but using GPT4-V as the captioner instead of BLIP-2. If
they exist, we remove duplicates from the LLaVA-Finetune dataset; this results
in $\sim$880K samples. We additionally add the following instruction datasets
LRV-Instruction [22] ($\sim$80K), laion-gpt4v-dataset ($\sim$15K),
ShareGPT4V[5] ($\sim$100K), and the following document understanding datasets
DocVQA[30]($\sim$50K), SynDog-EN[16]($\sim$65K), ChartQA[29]($\sim$23K),
DVQA[13] ($\sim$50K)and AI2D[15] ($\sim$15K).

###  4.1 Generative Visual Instruction Data

Figure 3: (Left) Results on selected Visual Question answering datasets.
(Right) A qualitative example of our model.

###  4.2 Training details.

In this section, we evaluate our model on a broad range of tasks that require
visual understanding and generation. We do not perform task-specific
finetuning in any experiments. The Supplementary section details details about
experimental setups, additional results, and additional studies on GenLLaVA ’s
instruction capabilities.

We adopt LLaVA-v1.5-7B [23] architecture and pre-training weights, then tune
it on the constructed GVIT-mix-2076K. We named this model GenLLaVA.
Specifically, we replace LLaVA-v1.5-mix-665K with our GVIT-mix-2076K, add the
visual generation module, and only perform a single-stage instruction tuning
training as this was shown to be superior [14]. The rest of the model training
protocol is kept unchanged for fair comparison. Generative Visual instruction
tuning takes about 48 hours for both full-parameter tuning and LoRA tuning on
8 NVIDIA Tesla A100 GPUs, each with 80GB memory, with DeepSpeed ZeRO Stage 3.

#### Visual Understanting

We evaluate vision language performance and compare it against other
generalist models, i.e., models that can do visual generation and
understating. Results on a collection of 8 vision/language benchmarks are
shown in Table 1. These benchmarks are designed to probe advanced knowledge
and general understanding. MMBench [26] evaluates answer robustness through
comprehensive shuffling of multiple-choice options. SEED-Bench [18] tests
model performance on images and videos with multiple-choice questions. MM-Vet
[47] evaluates the ability to engage in visual conversations and assess the
accuracy and helpfulness of responses. Mathvista [28] consolidates
mathematical reasoning benchmarks, focusing on logical and algebraic reasoning
with puzzle tests. MMMU [48] covers 57 subjects across STEM, humanities, and
social sciences, ranging from elementary to advanced professional levels, and
tests both world knowledge and problem-solving skills.

#### Visual generation.

For visual generation evaluation, we measured the Fréchet Inception Distance
(FID) [9] on the CC3M validation set [4] (which is a measure of image realism)
and CLIP Similarity on the MS-COCO dataset [21] (which is a measure of the
alignment between the text prompt and the generated image). For image editing,
we measured the DINOScore on the 5.7K examples of the EVR dataset [37]
following the same protocol of Fu et.al. [7].

Figure 4: Ablation on the effect of adding extra data to our instruction set.

###  4.3 Main Result

We compare our model, GenLLaVA, against state-of-the-art models on benchmarks
for advanced knowledge, general understanding, visual generation, and editing.

As shown in Table 1, GenLLaVA consistently outperforms previous models (GILL,
AnyGPT, and MGIE) in most tasks. It achieves higher scores on MathVista (24.9
vs. GILL’s 18.6, AnyGPT’s 24.4, MGIE’s 15.5, Unified-IO 2’s 30.7) and MMMU
(29.7 vs. GILL’s 28.8, AnyGPT’s 30.6, MGIE’s 25.6, Unified-IO 2’s 35). For
general understanding, GenLLaVA excels in SEED-B (63.5 vs. Unified-IO 2’s
61.8, GILL’s 52.5, AnyGPT’s 44.5, MGIE’s 28.8) and MMB (65 vs. Unified-IO 2’s
71.5, GILL’s 38.2, AnyGPT’s 36, MGIE’s 6.6).

In image editing on the EVR dataset, GenLLaVA scores 64.7, behind MGIE’s 71.5
but ahead of Unified-IO 2 (50.2), GILL (30.4), and AnyGPT (40.3). For visual
generation, GenLLaVA has a FID score of 14.3 on CC3M, close to Unified-IO 2
(13.4) and matching AnyGPT. On the CLIP Similarity metric for MS-COCO,
GenLLaVA scores 0.71, close to Unified-IO 2’s 0.72 and ahead of GILL (0.67),
AnyGPT (0.65), and MGIE (0.66).

While GenLLaVA does not surpass Unified-IO 2 in all categories, it shows
strong, balanced performance across various tasks, highlighting its
versatility and robustness as a generalist vision-language model with less
training time.

###  4.4 Results on selected Visual Question Answering datasets.

We evaluate various models’ performance across diverse visual question-
answering datasets, including VQAv2, GQA, VisWiz, TextVQA, and ScienceQA. Our
results show that Unified-IO 2 and GenLLaVA consistently perform well across
most datasets. Specifically, Unified-IO 2 achieves the highest scores on the
ScienceQA (86.2%) and TextVQA (67%), while GenLLaVA demonstrates strong
performance on VQAv2 (79.3%) and a competitive score on GQA (62.9%). In
contrast, GILL and MGIE exhibit generally lower performance across all
datasets, with MGIE notably struggling on VisWiz (20.1%) and TextVQA (26.3%).
AnyGPT shows moderate effectiveness, with its best performance on ScienceQA
(61%). We used VLMEvalKit [6] to get the results for these datasets, which
perform a generation-based evaluation using an LLM judge222 We used GPT-4
(0409) as the judge., the results can be seen in Fig. 3.

###  4.5 Ablations

We investigate the effect of scaling the data used to train GenLlaVA, the
effect of using different image backbones, and the number of visual tokens
used for image generation.

#### Instruction data.

We start with the original instruction tuning dataset from LlaVA-1.5,
basically reproducing the original results using a one-stage training recipe.
We add the textttLLaVA-Pretrain and generation head to our model; we notice
that adding generation capabilities significantly affects the visual
understanding capabilities in the model where all the metrics between 2.3%
(MM-Vet) to 7.9% (MathVista) points in performance, to compensate for this
loss in performance we modify the ration of image generation to image
understanding data in our dataset from approximately $\sim$50%-50% to
$\sim$70%-30%, by adding more image instruction data from LVIS-INSTRUCT4V,
LRV-Instruction and other chart understanding datasets. This results in a
model with significant generation capabilities that maintain its image-
understanding capabilities. We finally add image-generation capabilities using
our selected subset of the IPr2IPr dataset. This reduces the image
understanding capabilities, but we consider this a small enough change that
balances the three tasks while maintaining commendable performance. The
results can be seen in Fig. 4a.

#### Choice of Visual encoder.

The quality of the vision encoder can also have big effects on the final LMM
performance. We start by using a CLIP/B, which has the lowest performance,
then we compare against a stronger visual encoder and create versions of
GenLLaVA, which is trained with CLIP [31] and SigLIP [49], respectively. We
can see in Fig. 4b that the SigLIP encoder can achieve generally better
performance CLIP encoder. This shows that SigLIP is a better vision encoder
for LMM development.

#### Number of visual generation tokens.

We experiment with varying the number of visual generation tokens, $N$. We
noticed that we need more visual generation tokens than GILL ($N=4$) and MGIE
($N=8$) to achieve the best performance. We find that $N=16$ is the best
choice for our model. We hypothesize that this is because our model has to
balance image generation and editing in the same head and thus needs more
visual tokens to capture the complexity of the tasks. The results can be seen
in Fig. 3c.

Figure 5: Qualitative conversational example of our model. The dashed line
indicates that the conversation has to be restarted from the beginning due to
the model losing track of it.

###  4.6 Comparision with SOTA.

When compared with state-of-the-art models, GenLLaVA maintains similar
performance to models of the LlaVA family when evaluated using the average of
the scores on the MathVista, MMMU, MMVet, SEED-B, and MMB datasets but lags
behind bigger and more specialized models. However, some of these models lack
the generative capabilities present in GenLLaVA.

When compared with models of similar size and setup ($\sim 7$b parameters),
our model surpasses the original LlaVAv1[25] model by 5% points (37.5% vs.
43.2%), but lags behind LlaVA-1.5 by $\sim$2% points (45.3% vs. 43.2%). It is
surpassed by the LlaVA-Next [24] family of models by $\sim$7% points; by
Idefics2 by $\sim$12% points (55.7% vs. 43.2%) and by the newly released
MiniCPM-Llama3 [43] model by $\sim$17% points (60.6% vs. 43.2%). Compared with
the absolute state-of-the-art open-source models, GenLLaVA lags behind Yi-VL
[45] by $\sim$6% points; Emu2 [35] by $\sim$2% points, LlaVA-Next [24] (34b)
by $\sim$15% points; and InternVL 1.5 by $\sim$20% points. Compared with the
absolute state-of-the-art closed models, GenLLaVA lags behind GPT-4o by
$\sim$25% points, GPT-4V [1] by $\sim$24% points; and the Gemini family [40]
of models by 16% points.

##  5 Conclusion

In this paper, we have introduced the Generative Large Language, and Visual
Assistant (GenLLaVA), a new approach that enables Large Multimodal Models
(LMMs) to simultaneously excel in image understanding, generation, and editing
without significant loss in individual task performance. We balanced
multimodal capabilities in a single model by curating a comprehensive
multimodal instruction dataset and developing an innovative single-phase
training methodology. Our work sets a new standard for building visual
assistants with extra capabilities, and we hope our open-source contributions,
including datasets, codebase, and model checkpoints, will serve as valuable
resources for the research community, driving further advancements in the
field of multimodal AI.

## Acknowledgements

The authors would like to thank Google Cloud and the TPU Research Cloud
program from Google for providing funding for this research effort. We are
also thankful for support from the Department of Computer Science at Rice
University.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [3] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023.
  * [4] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021.
  * [5] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  arXiv preprint arXiv:2311.12793, 2023.
  * [6] OpenCompass Contributors.  Opencompass: A universal evaluation platform for foundation models.  https://github.com/open-compass/opencompass, 2023.
  * [7] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan.  Guiding instruction-based image editing via multimodal large language models.  arXiv preprint arXiv:2309.17102, 2023.
  * [8] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [10] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [11] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [12] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [13] Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan.  Dvqa: Understanding data visualizations via question answering.  In CVPR, 2018.
  * [14] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh.  Prismatic vlms: Investigating the design space of visually-conditioned language models.  arXiv preprint arXiv:2402.07865, 2024.
  * [15] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi.  A diagram is worth a dozen images.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer, 2016.
  * [16] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.  Ocr-free document understanding transformer.  In European Conference on Computer Vision (ECCV), 2022.
  * [17] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov.  Generating images with multimodal language models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [18] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.  Seed-bench: Benchmarking multimodal llms with generative comprehension.  arXiv preprint arXiv:2307.16125, 2023.
  * [19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [20] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.  Video-llava: Learning united visual representation by alignment before projection.  arXiv preprint arXiv:2311.10122, 2023.
  * [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [22] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  In The Twelfth International Conference on Learning Representations, 2023.
  * [23] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023.
  * [24] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
  * [25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * [26] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.  Mmbench: Is your multi-modal model an all-around player?  arXiv preprint arXiv:2307.06281, 2023.
  * [27] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi.  Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action.  arXiv preprint arXiv:2312.17172, 2023.
  * [28] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.  Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.  arXiv preprint arXiv:2310.02255, 2023.
  * [29] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.  ChartQA: A benchmark for question answering about charts with visual and logical reasoning.  In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [30] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.  Docvqa: A dataset for vqa on document images.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
  * [31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022.
  * [33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [34] Teams ShareGPT.  Sharegpt: Share your wildest chatgpt conversations with one click, 2023.
  * [35] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al.  Generative multimodal models are in-context learners.  arXiv preprint arXiv:2312.13286, 2023.
  * [36] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Generative pretraining in multimodality.  arXiv preprint arXiv:2307.05222, 2023.
  * [37] Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal.  Expressing visual relationships via language.  arXiv preprint arXiv:1906.07689, 2019.
  * [38] Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal.  Codi-2: In-context, interleaved, and interactive any-to-any generation.  arXiv preprint arXiv:2311.18775, 2023.
  * [39] Chameleon Team.  Chameleon: Mixed-modal early-fusion foundation models.  arXiv preprint arXiv:2405.09818, 2024.
  * [40] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [41] Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang.  To see is to believe: Prompting gpt-4v for better visual instruction tuning.  arXiv preprint arXiv:2311.07574, 2023.
  * [42] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.  Cogvlm: Visual expert for pretrained language models.  arXiv preprint arXiv:2311.03079, 2023.
  * [43] Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang.  LLaVA-UHD: an lmm perceiving any aspect ratio and high-resolution images.  arXiv preprint arXiv:2403.11703, 2024.
  * [44] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang.  Ferret: Refer and ground anything anywhere at any granularity.  arXiv preprint arXiv:2310.07704, 2023.
  * [45] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al.  Yi: Open foundation models by 01. ai.  arXiv preprint arXiv:2403.04652, 2024.
  * [46] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan.  Scaling autoregressive multi-modal models: Pretraining and instruction tuning.  arXiv preprint arXiv:2309.02591, 2023.
  * [47] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  arXiv preprint arXiv:2308.02490, 2023.
  * [48] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  arXiv preprint arXiv:2311.16502, 2023.
  * [49] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023.
  * [50] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al.  Anygpt: Unified multimodal llm with discrete sequence modeling.  arXiv preprint arXiv:2402.12226, 2024.
  * [51] Bo Zhao, Boya Wu, and Tiejun Huang.  Svit: Scaling up visual instruction tuning.  arXiv preprint arXiv:2307.04087, 2023.


increase the sample size of the data release. This may help capture the even
rarer, extreme objects that represent the “tip-of-the-iceberg” of what is
currently being observed with the first JWST observations. The limiting
factors in providing all galaxies in the initial release are due to
computation time and data storage and these will be less problematic moving
forward.

  2. 2.

Extensions beyond star-forming galaxies: A star-formation threshold of
$0.3~{}{\rm M_{\odot}~{}yr^{-1}}$ was chosen to limit the galaxies that were
being post-processed to only those that are likely observable with JWST.
However, JWST observes flux, not SFR. Without post-processing the data, we are
unable to know which galaxies have high enough magnitudes to be observed;
hence SFR was our proxy. Unfortunately this choice means that certain classes
of galaxies, e.g. quiescent or remnant leakers, are absent from the initial
data release, despite the fact that they may represent critical phases in
galaxy evolution. Future releases would ideally contain Sphinx galaxies that
are in between star formation events rather than at a peak.

  3. 3.

Alternative dust models: Since Sphinx does not explicitly follow the formation
and destruction of dust, we have adopted an effective dust model that assumes
a particular SMC attenuation curve and connects the dust content with the
neutral gas. While this model is successful in that our UV luminosity and
Ly$\alpha$ luminosity functions are in good agreement with observations, it is
not clear whether this model is unique in that regard. For this reason, we
plan to sample more dust models to better understand their effect on mock
observations.

  4. 4.

Additional emission lines: The data set currently contains the vast majority
of emission lines that have been observed by JWST at high-redshift; however,
there are notable exceptions (for example Mg II $\lambda$2796,2803 and N IV
$\lambda$1483,1486). We have also not provided most of the IR lines that can
be observed with ALMA. Such additional emission lines may be part of future
data releases.

  5. 5.

Varying metal abundance patterns: Because Sphinx does not follow enrichment of
individual elements, we have scaled solar abundance patterns to the
metallicity of each gas cell. However, it is well known that ratios such as
C/O and N/O vary with metallicity. This effect is important for both line
ratios and cooling in the ISM and future data releases may attempt to address
this effect.

  6. 6.

Spectral data cubes: While the current data release contains line luminosities
of the integrated galaxy spectra, which is comparable to the vast majority of
JWST spectral data, one of the unique aspects of JWST is its IFU capabilities.
RASCAS has the capabilities of outputting spectral data cubes for any emission
line and the limiting factor in their public release is purely data storage.
These can be currently made available on-request for individual galaxies and
will ideally be standard in future releases.

  7. 7.

Ly$\alpha$ IGM propagation: The Ly$\alpha$ spectra that we released are
calculated only out to the virial radius of the galaxy. While this is
instructive for understanding how the ISM and CGM modulate the intrinsic
Ly$\alpha$ emission, it is not directly comparable to high-redshift galaxy
observations where the CGM remains important, even in the post-reionization
epoch (e.g. Inoue et al., 2014). Although the Sphinx volume does not quite
probe the cosmological homogeneity scale ($\sim 100$ Mpc), we are still able
to assess the impact of the IGM on our spectra more locally (see e.g. Garel et
al. 2021).

  8. 8.

Environmental Information: Additional environmental information, such as
neighbour properties, filamentary properties, and whether the galaxies reside
in ionized bubbles may also be useful for comparing with observations and
understanding the role of radiation feedback on galaxy properties (see e.g.
Katz et al. 2020). For example, the recent observation that high-redshift
Ly$\alpha$ emitters tend to have companions (Witten et al., 2023) would not
currently be testable with the Sphinx data set as we only include positions of
star-forming galaxies.

###  4.3 Caveats

The Sphinx data release represents a unique tool to both compare theory with
observation and to help interpret observations; however, like all numerical
simulations, there remain important limitations of the data set. Most
importantly, Sphinx has finite spatial and mass resolution limiting how much
of the ISM turbulent structure is truly resolved. While the resolution of
Sphinx is the state-of-the-art for radiative hydrodynamics simulations of this
size, in many cases, Stromgren spheres remain unresolved. This is in many ways
a result of the way stars form in the simulation (see discussion in Rosdahl et
al. 2015), but the modelling choices have a strong impact on our results. For
example, in the case where multiple star particles exist in a cell where the
net Stromgren sphere is unresolved, we model this as a single source at the
centre of the cell. Thus the ionization parameter is much higher than the case
where we model these as separate H II regions embedded in the same cell.
Choosing the former can result in higher e.g. [O III]/H$\beta$ than the latter
method. Finite spatial resolution also has a strong impact on Ly$\alpha$ and
LyC escape physics. It is not yet known how well these quantities are
converged with resolution. Furthermore, due to the fixed mass and comoving
spatial resolution, high-mass galaxies are better resolved than low mass ones
at a fixed redshift and the ISM is better resolved in the higher redshift
snapshots compared to the lower redshift ones. The important impact of this
numerical effect is that the gas can reach higher densities in more massive
galaxies and at high-redshift for a fixed halo mass. The gas density is a
fundamental parameter that impacts star formation and much of the emission
presented in this work. Intrinsic emission will be more strongly affected as
in nearly all galaxies, the resolution in Sphinx is high enough such that the
densities impacted by this effect are typically optically thick (at short
wavelengths).

One may argue that due to the inevitable use of subgrid models and the
imperfect match between simulations and all high-redshift observations, Sphinx
is not a completely accurate representation of high-redshift galaxy formation.
This is not unique to Sphinx and likely true of all galaxy formation
simulations. However, photoionization models that are very simplistic have
been the dominant means of interpreting high-redshift spectra. Despite the
fact that Sphinx is imperfect, it undoubtedly captures a much more complex ISM
and SFHs that better reflect reality than photoionization models, especially
since a Sphinx galaxy can be thought of as a complex combination of
photoionization models with the additional elements of diffuse gas and a non-
zero escape fraction. This is consistent with the fact that groups are now
adopting combinations of photoionization models to explain emission lines
rather than relying on individual models (e.g. Ramambason et al., 2022).
Moreover, Sphinx provides a unique test-bed for understanding how well ISM
properties can be inferred from an integrated SED. In this case, it does not
matter in which galaxy the ISM resides, this is simply a photon counting
exercise for a given density, metallicity, and stellar population
distribution. For this reason we argue that the Sphinx data release represents
a drastic improvement over previous methods. At worst, our results should be
interpreted as more complex, 3D photoionziation models. This will only be
superseded by simulations where non-equilibrium metal chemistry is fully
coupled to radiative transfer (e.g. Katz, 2022) and the Stromgren spheres are
resolved (e.g. Kimm et al., 2022).

In addition to subgrid models, Sphinx also relies on a series of modelling
assumptions. For example, we have chosen a specific stellar IMF (Kroupa-like),
a stellar population synthesis library (BPASS) for the stellar spectra, a
particular dust model (SMC), etc. Different choices for these parameters can
systematically shift galaxy properties such as mass-to-light ratios, ionizing
photon luminosities and therefore emission line luminosities, dust attenuation
as a function of wavelength and metallictiy and thus ${\rm E(B-V)}$ values and
$\beta$ slopes. Once again, this is not only true for Sphinx, but is also true
for other numerical simulation as well as SED fitting codes. For these
reasons, we have tried to compare directly with observations when possible
rather than derived or inferred quantities to mitigate these systematic biases
between codes.

##  5 Conclusions

Here we have described Version 1 of the Sphinx data release, which is publicly
available to download from https://github.com/HarleyKatz/SPHINX-20-data.
Sphinx is currently the largest full box simulation of galaxy formation in the
epoch of reionization with a multiphase ISM and thus represents a valuable
resource to the wider community, especially given the recent launch of JWST.
The data release contains galaxies spanning $\sim 5$ dex in stellar mass and
star formation rate that exhibit a diverse set of mass growth and star
formation histories. The unique aspect of this data set is the focus on
forward modelling observational quantities such that the data can be directly
compared with observations. We provide full spectra and photometry including
self-consistently calculated nebular emission lines (including Ly$\alpha$) and
continuum, accounting for absorption and scattering by dust (and H I). We have
provided a tour of all of the different quantities available in the data set
and made comparisons with ERS and Cycle 1 JWST data where possible to
demonstrate the utility of the data in interpreting high-redshift observations
and to show where constraints can be made on the physics of early galaxy
formation.

While the results presented here represent a static view of the Sphinx public
data release, we emphasize, as indicated in Section 4.2 that the data set will
be updated continuously over time to provide both new data and enhance
existing data products. We encourage users to quote the git hash of the data
set when publishing with Sphinx data for reproducibility purposes.

The data is organized in a series of CSV tables as well as JSON files that
have been indexed by halo ID and redshift. These files can be flexibly loaded
into any relevant computing software (e.g. Python, IDL, Julia, etc.). In
addition to the curated data, we also release a repository of Juptyer
notebooks with basic tutorials on how to load and manipulate the data in
Python. Our aim is to lower the barrier to entry for interested users of high-
redshift galaxy formation simulations and to ease the comparison between
simulations and observations.

## Acknowledgements

HK thanks Jonathan Patterson for support with the local Glamdring cluster as
well as Adrianne Slyz and Julien Devriendt for access to dp016 computing hours
on DiRAC. TK was supported by the National Research Foundation of Korea
(NRF-2022R1A6A1A03053472 and NRF-2022M3K3A1097100).

This work used the DiRAC@Durham facility managed by the Institute for
Computational Cosmology on behalf of the STFC DiRAC HPC Facility
(www.dirac.ac.uk). The equipment was funded by BEIS capital funding via STFC
capital grants ST/P002293/1, ST/R002371/1 and ST/S002502/1, Durham University
and STFC operations grant ST/R000832/1. DiRAC is part of the National
e-Infrastructure. This work was performed using the DiRAC Data Intensive
service at Leicester, operated by the University of Leicester IT Services,
which forms part of the STFC DiRAC HPC Facility (www.dirac.ac.uk). The
equipment was funded by BEIS capital funding via STFC capital grants
ST/K000373/1 and ST/R002363/1 and STFC DiRAC Operations grant ST/R001014/1.
DiRAC is part of the National e-Infrastructure.

Computing time for the SPHINX project was provided by the Partnership for
Advanced Computing in Europe (PRACE) as part of the “First luminous objects
and reionization with SPHINX (cont.)” (2016153539, 2018184362, 2019215124)
project. We thank Philipp Otte and Filipe Guimaraes for helpful support
throughout the project and for the extra storage they provided us. We also
thank GENCI for providing additional computing resources under GENCI grant
A0070410560. Resources for preparations, tests, and storage were also provided
by the Common Computing Facility (CCF) of the LABEX Lyon Institute of Origins
(ANR-10-LABX-0066) and PSMN (Pôle Scientifique de Modélisation Numérique) at
ENS de Lyon.

The main roles of the authors were, using the CRediT (Contribution Roles
Taxonomy) system111111https://authorservices.wiley.com/author-
resources/Journal-Authors/open-access/credit.html: Harley Katz:
conceptualization; methodology, investigation, data curation, writing –
original draft; formal analysis; visualization; software, Joki Rosdahl:
conceptualization; writing – original draft; formal analysis; software;
validation; visualization, Taysun Kimm: writing – original draft; formal
analysis; validation; visualization, Jeremy Blaizot: writing – original draft;
formal analysis; validation; visualization, Nicholas Choustikov: validation;
software; writing – review & editing, Marion Farcy: writing – original draft;
formal analysis; validation; visualization, Thibault Garel: writing – original
draft; formal analysis; validation; visualization, Martin Haehnelt: writing –
review and editing, Leo Michel-Dansac: software, writing – review and editing
Pierre Ocvirk: writing – original draft; formal analysis; validation;
visualization.

## References

  * Amorín et al. (2017) Amorín R. et al., 2017, Nature Astronomy, 1, 0052
  * Andrews & Martini (2013) Andrews B. H., Martini P., 2013, ApJ, 765, 140
  * Arrabal Haro et al. (2023) Arrabal Haro P. et al., 2023, ApJ, 951, L22
  * Attia et al. (2021) Attia O., Teyssier R., Katz H., Kimm T., Martin-Alvarez S., Ocvirk P., Rosdahl J., 2021, MNRAS, 504, 2346
  * Austin et al. (2023) Austin D. et al., 2023, ApJ, 952, L7
  * Bacon et al. (2017) Bacon R. et al., 2017, A&A, 608, A1
  * Baes et al. (2011) Baes M., Verstappen J., De Looze I., Fritz J., Saftly W., Vidal Pérez E., Stalevski M., Valcke S., 2011, ApJS, 196, 22
  * Baldry et al. (2012) Baldry I. K. et al., 2012, MNRAS, 421, 621
  * Baldwin, Phillips & Terlevich (1981) Baldwin J. A., Phillips M. M., Terlevich R., 1981, PASP, 93, 5
  * Barrow et al. (2020) Barrow K. S. S., Robertson B. E., Ellis R. S., Nakajima K., Saxena A., Stark D. P., Tang M., 2020, ApJ, 902, L39
  * Barrow et al. (2017) Barrow K. S. S., Wise J. H., Norman M. L., O’Shea B. W., Xu H., 2017, MNRAS, 469, 4863
  * Behroozi et al. (2019) Behroozi P., Wechsler R. H., Hearin A. P., Conroy C., 2019, MNRAS, 488, 3143
  * Bezanson et al. (2022) Bezanson R. et al., 2022, arXiv e-prints, arXiv:2212.04026
  * Blaizot et al. (2023) Blaizot J. et al., 2023, MNRAS, 523, 3749
  * Bouwens et al. (2023) Bouwens R., Illingworth G., Oesch P., Stefanon M., Naidu R., van Leeuwen I., Magee D., 2023, MNRAS, 523, 1009
  * Bouwens et al. (2015) Bouwens R. J. et al., 2015, ApJ, 803, 34
  * Bouwens et al. (2015) Bouwens R. J. et al., 2015, Astrophys. J., 803, 34
  * Bouwens et al. (2022) Bouwens R. J., Illingworth G. D., van Dokkum P. G., Oesch P. A., Stefanon M., Ribeiro B., 2022, ApJ, 927, 81
  * Bouwens et al. (2021) Bouwens R. J. et al., 2021, AJ, 162, 47
  * Bouwens et al. (2016) Bouwens R. J., Smit R., Labbé I., Franx M., Caruana J., Oesch P., Stefanon M., Rasappu N., 2016, ApJ, 831, 176
  * Bradač et al. (2023) Bradač M. et al., 2023, arXiv e-prints, arXiv:2308.13288
  * Bradley et al. (2023) Bradley L. et al., 2023, astropy/photutils: 1.8.0
  * Brammer, van Dokkum & Coppi (2008) Brammer G. B., van Dokkum P. G., Coppi P., 2008, ApJ, 686, 1503
  * Brinchmann et al. (2004) Brinchmann J., Charlot S., White S. D. M., Tremonti C., Kauffmann G., Heckman T., Brinkmann J., 2004, MNRAS, 351, 1151
  * Bunker et al. (2023a) Bunker A. J. et al., 2023a, arXiv e-prints, arXiv:2306.02467
  * Bunker et al. (2023b) Bunker A. J. et al., 2023b, arXiv e-prints, arXiv:2302.07256
  * Byler et al. (2017) Byler N., Dalcanton J. J., Conroy C., Johnson B. D., 2017, ApJ, 840, 44
  * Calzetti et al. (2000) Calzetti D., Armus L., Bohlin R. C., Kinney A. L., Koornneef J., Storchi-Bergmann T., 2000, ApJ, 533, 682
  * Cameron, Katz & Rey (2022) Cameron A. J., Katz H., Rey M. P., 2022, arXiv e-prints, arXiv:2210.14234
  * Cameron et al. (2023) Cameron A. J. et al., 2023, arXiv e-prints, arXiv:2302.04298
  * Camps et al. (2018) Camps P. et al., 2018, ApJS, 234, 20
  * Carnall et al. (2018) Carnall A. C., McLure R. J., Dunlop J. S., Davé R., 2018, MNRAS, 480, 4379
  * Cassata et al. (2011) Cassata P. et al., 2011, A&A, 525, A143
  * Ceverino, Klessen & Glover (2019) Ceverino D., Klessen R. S., Glover S. C. O., 2019, MNRAS, 484, 1366
  * Chevallard & Charlot (2016) Chevallard J., Charlot S., 2016, MNRAS, 462, 1415
  * Choustikov et al. (2023) Choustikov N. et al., 2023, arXiv e-prints, arXiv:2304.08526
  * CONCERTO Collaboration et al. (2020) CONCERTO Collaboration et al., 2020, A&A, 642, A60
  * Crain et al. (2015) Crain R. A. et al., 2015, MNRAS, 450, 1937
  * Crites et al. (2014) Crites A. T. et al., 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9153, Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy VII, Holland W. S., Zmuidzinas J., eds., p. 91531W
  * Cullen et al. (2023) Cullen F. et al., 2023, MNRAS, 520, 14
  * Curti et al. (2023) Curti M. et al., 2023, arXiv e-prints, arXiv:2304.08516
  * Curtis-Lake et al. (2022) Curtis-Lake E. et al., 2022, arXiv e-prints, arXiv:2212.04568
  * De Looze et al. (2014) De Looze I. et al., 2014, A&A, 568, A62
  * Dere et al. (2019) Dere K. P., Del Zanna G., Young P. R., Landi E., Sutherland R. S., 2019, ApJS, 241, 22
  * Despali et al. (2016) Despali G., Giocoli C., Angulo R. E., Tormen G., Sheth R. K., Baso G., Moscardini L., 2016, MNRAS, 456, 2486
  * Dijkstra (2017) Dijkstra M., 2017, Saas-fee lecture notes: Physics of lyman alpha radiative transfer
  * Donnan et al. (2023) Donnan C. T. et al., 2023, MNRAS, 518, 6011
  * Doré et al. (2014) Doré O. et al., 2014, arXiv e-prints, arXiv:1412.4872
  * Drake et al. (2017) Drake A. B. et al., 2017, A&A, 608, A6
  * Dressler et al. (2023) Dressler A. et al., 2023, ApJ, 947, L27
  * Dubois et al. (2014) Dubois Y. et al., 2014, MNRAS, 444, 1453
  * Eisenstein et al. (2023) Eisenstein D. J. et al., 2023, arXiv e-prints, arXiv:2306.02465
  * Eldridge et al. (2017) Eldridge J. J., Stanway E. R., Xiao L., McClelland L. A. S., Taylor G., Ng M., Greis S. M. L., Bray J. C., 2017, PASA, 34, e058
  * Ellis et al. (2013) Ellis R. S. et al., 2013, ApJ, 763, L7
  * Endsley et al. (2023) Endsley R., Stark D. P., Whitler L., Topping M. W., Chen Z., Plat A., Chisholm J., Charlot S., 2023, MNRAS, 524, 2312
  * Ercolano et al. (2008) Ercolano B., Young P. R., Drake J. J., Raymond J. C., 2008, ApJS, 175, 534
  * Feldmann et al. (2023) Feldmann R. et al., 2023, MNRAS, 522, 3831
  * Feltre, Charlot & Gutkin (2016) Feltre A., Charlot S., Gutkin J., 2016, MNRAS, 456, 3354
  * Ferland et al. (2017) Ferland G. J. et al., 2017, Rev. Mexicana Astron. Astrofis., 53, 385
  * Finkelstein et al. (2023) Finkelstein S. L. et al., 2023, ApJ, 946, L13
  * Flury et al. (2022) Flury S. R. et al., 2022, ApJS, 260, 1
  * Franco et al. (2023) Franco M. et al., 2023, arXiv e-prints, arXiv:2308.00751
  * Fujimoto et al. (2023) Fujimoto S. et al., 2023, ApJ, 949, L25
  * Gardner et al. (2006) Gardner J. P. et al., 2006, Space Sci. Rev., 123, 485
  * Garel et al. (2021) Garel T., Blaizot J., Rosdahl J., Michel-Dansac L., Haehnelt M. G., Katz H., Kimm T., Verhamme A., 2021, MNRAS, 504, 1902
  * Gazagnes et al. (2023) Gazagnes S. et al., 2023, ApJ, 952, 164
  * Giavalisco (2002) Giavalisco M., 2002, ARA&A, 40, 579
  * Giménez-Arteaga et al. (2023) Giménez-Arteaga C. et al., 2023, ApJ, 948, 126
  * Gnedin (2014) Gnedin N. Y., 2014, ApJ, 793, 29
  * Gray & Scannapieco (2017) Gray W. J., Scannapieco E., 2017, ApJ, 849, 132
  * Grevesse et al. (2010) Grevesse N., Asplund M., Sauval A. J., Scott P., 2010, Ap&SS, 328, 179
  * Groves, Heckman & Kauffmann (2006) Groves B. A., Heckman T. M., Kauffmann G., 2006, MNRAS, 371, 1559
  * Guseva et al. (2017) Guseva N. G., Izotov Y. I., Fricke K. J., Henkel C., 2017, A&A, 599, A65
  * Hahn & Abel (2011) Hahn O., Abel T., 2011, MNRAS, 415, 2101
  * Hainline et al. (2023) Hainline K. N. et al., 2023, arXiv e-prints, arXiv:2306.02468
  * Harikane et al. (2023) Harikane Y. et al., 2023, ApJS, 265, 5
  * Hashimoto et al. (2018) Hashimoto T. et al., 2018, Nature, 557, 392
  * Herenz et al. (2019) Herenz E. C. et al., 2019, A&A, 621, A107
  * Hirschmann et al. (2022) Hirschmann M. et al., 2022, arXiv e-prints, arXiv:2212.02522
  * Inami et al. (2017) Inami H. et al., 2017, A&A, 608, A2
  * Inoue et al. (2014) Inoue A. K., Shimizu I., Iwata I., Tanaka M., 2014, MNRAS, 442, 1805
  * Isobe et al. (2023) Isobe Y., Ouchi M., Nakajima K., Harikane Y., Ono Y., Xu Y., Zhang Y., Umeda H., 2023, arXiv e-prints, arXiv:2301.06811
  * Isobe et al. (2022) Isobe Y. et al., 2022, ApJ, 925, 111
  * Izotov et al. (2016) Izotov Y. I., Schaerer D., Thuan T. X., Worseck G., Guseva N. G., Orlitová I., Verhamme A., 2016, MNRAS, 461, 3683
  * Jiang et al. (2021) Jiang L. et al., 2021, Nature Astronomy, 5, 256
  * Jin, Kewley & Sutherland (2022) Jin Y., Kewley L. J., Sutherland R. S., 2022, ApJ, 934, L8
  * Johnson et al. (2021) Johnson B. D., Leja J., Conroy C., Speagle J. S., 2021, ApJS, 254, 22
  * Jonsson (2006) Jonsson P., 2006, MNRAS, 372, 2
  * Kannan et al. (2022a) Kannan R., Garaldi E., Smith A., Pakmor R., Springel V., Vogelsberger M., Hernquist L., 2022a, MNRAS, 511, 4005
  * Kannan et al. (2022b) Kannan R., Smith A., Garaldi E., Shen X., Vogelsberger M., Pakmor R., Springel V., Hernquist L., 2022b, MNRAS, 514, 3857
  * Kashino et al. (2023) Kashino D., Lilly S. J., Matthee J., Eilers A.-C., Mackenzie R., Bordoloi R., Simcoe R. A., 2023, ApJ, 950, 66
  * Katz (2022) Katz H., 2022, MNRAS, 512, 348
  * Katz et al. (2019) Katz H. et al., 2019, MNRAS, 487, 5902
  * Katz et al. (2022a) Katz H. et al., 2022a, MNRAS, 515, 4265
  * Katz et al. (2017) Katz H., Kimm T., Sijacki D., Haehnelt M. G., 2017, MNRAS, 468, 4831
  * Katz et al. (2021) Katz H. et al., 2021, MNRAS, 507, 1254
  * Katz et al. (2020) Katz H. et al., 2020, MNRAS, 494, 2200
  * Katz et al. (2022b) Katz H. et al., 2022b, MNRAS, 510, 5603
  * Katz et al. (2023a) Katz H. et al., 2023a, MNRAS, 518, 592
  * Katz et al. (2023b) Katz H. et al., 2023b, MNRAS, 518, 270
  * Kauffmann et al. (2003) Kauffmann G. et al., 2003, MNRAS, 346, 1055
  * Kawamata et al. (2018) Kawamata R., Ishigaki M., Shimasaku K., Oguri M., Ouchi M., Tanigawa S., 2018, ApJ, 855, 4
  * Kennicutt (1998) Kennicutt, Robert C. J., 1998, ARA&A, 36, 189
  * Kennicutt & Evans (2012) Kennicutt R. C., Evans N. J., 2012, ARA&A, 50, 531
  * Kewley et al. (2001) Kewley L. J., Dopita M. A., Sutherland R. S., Heisler C. A., Trevena J., 2001, ApJ, 556, 121
  * Kikuchihara et al. (2020) Kikuchihara S. et al., 2020, ApJ, 893, 60
  * Kimm et al. (2022) Kimm T., Bieri R., Geen S., Rosdahl J., Blaizot J., Michel-Dansac L., Garel T., 2022, ApJS, 259, 21
  * Kimm & Cen (2014) Kimm T., Cen R., 2014, ApJ, 788, 121
  * Kimm et al. (2015) Kimm T., Cen R., Devriendt J., Dubois Y., Slyz A., 2015, MNRAS, 451, 2900
  * Konno et al. (2018) Konno A. et al., 2018, PASJ, 70, S16
  * Laursen, Sommer-Larsen & Andersen (2009) Laursen P., Sommer-Larsen J., Andersen A. C., 2009, ApJ, 704, 1640
  * Leethochawalit et al. (2023) Leethochawalit N. et al., 2023, ApJ, 942, L26
  * Leung et al. (2023) Leung G. C. K. et al., 2023, arXiv e-prints, arXiv:2306.06244
  * Li & White (2009) Li C., White S. D. M., 2009, MNRAS, 398, 2177
  * Livermore, Finkelstein & Lotz (2017) Livermore R. C., Finkelstein S. L., Lotz J. M., 2017, ApJ, 835, 113
  * Lovell et al. (2021) Lovell C. C., Vijayan A. P., Thomas P. A., Wilkins S. M., Barnes D. J., Irodotou D., Roper W., 2021, MNRAS, 500, 2127
  * Lupi et al. (2020) Lupi A., Pallottini A., Ferrara A., Bovino S., Carniani S., Vallini L., 2020, MNRAS, 496, 5160
  * Ma et al. (2018) Ma X. et al., 2018, MNRAS, 478, 1694
  * Madau & Dickinson (2014) Madau P., Dickinson M., 2014, ARA&A, 52, 415
  * Maji et al. (2022) Maji M. et al., 2022, Astron. Astrophys., 663, A66
  * Marshall et al. (2022) Marshall M. A. et al., 2022, MNRAS, 516, 1047
  * Matthee et al. (2023) Matthee J., Mackenzie R., Simcoe R. A., Kashino D., Lilly S. J., Bordoloi R., Eilers A.-C., 2023, ApJ, 950, 67
  * McCaffrey et al. (2023) McCaffrey J., Hardin S., Wise J., Regan J., 2023, arXiv e-prints, arXiv:2304.13755
  * Méndez-Delgado et al. (2023a) Méndez-Delgado J. E. et al., 2023a, MNRAS, 523, 2952
  * Méndez-Delgado et al. (2023b) Méndez-Delgado J. E., Esteban C., García-Rojas J., Kreckel K., Peimbert M., 2023b, Nature, 618, 249
  * Michel-Dansac et al. (2020) Michel-Dansac L., Blaizot J., Garel T., Verhamme A., Kimm T., Trebitsch M., 2020, A&A, 635, A154
  * Morisset, Delgado-Inglada & Flores-Fajardo (2015) Morisset C., Delgado-Inglada G., Flores-Fajardo N., 2015, Rev. Mexicana Astron. Astrofis., 51, 103
  * Moster, Naab & White (2018) Moster B. P., Naab T., White S. D. M., 2018, MNRAS, 477, 1822
  * Nakajima et al. (2023) Nakajima K., Ouchi M., Isobe Y., Harikane Y., Zhang Y., Ono Y., Umeda H., Oguri M., 2023, arXiv e-prints, arXiv:2301.12825
  * Nakajima et al. (2022) Nakajima K. et al., 2022, ApJS, 262, 3
  * Narayanan et al. (2023) Narayanan D. et al., 2023, arXiv e-prints, arXiv:2306.10118
  * Ocvirk et al. (2016) Ocvirk P. et al., 2016, MNRAS, 463, 1462
  * Oesch et al. (2018) Oesch P. A., Bouwens R. J., Illingworth G. D., Labbé I., Stefanon M., 2018, ApJ, 855, 105
  * O’Shea et al. (2015) O’Shea B. W., Wise J. H., Xu H., Norman M. L., 2015, ApJ, 807, L12
  * Pacifici et al. (2023) Pacifici C. et al., 2023, ApJ, 944, 141
  * Pallottini et al. (2019) Pallottini A. et al., 2019, MNRAS, 487, 1689
  * Pallottini et al. (2017) Pallottini A., Ferrara A., Gallerani S., Vallini L., Maiolino R., Salvadori S., 2017, MNRAS, 465, 2540
  * Park et al. (2022) Park H. et al., 2022, ApJ, 931, 126
  * Pawlik et al. (2017) Pawlik A. H., Rahmati A., Schaye J., Jeon M., Dalla Vecchia C., 2017, MNRAS, 466, 960
  * Pilyugin et al. (2009) Pilyugin L. S., Mattsson L., Vílchez J. M., Cedrés B., 2009, MNRAS, 398, 485
  * Pilyugin et al. (2012) Pilyugin L. S., Vílchez J. M., Mattsson L., Thuan T. X., 2012, MNRAS, 421, 1624
  * Planck Collaboration et al. (2014) Planck Collaboration et al., 2014, A&A, 571, A1
  * Popesso et al. (2023) Popesso P. et al., 2023, MNRAS, 519, 1526
  * Ramambason et al. (2022) Ramambason L. et al., 2022, A&A, 667, A35
  * Reddy et al. (2015) Reddy N. A. et al., 2015, ApJ, 806, 259
  * Rémy-Ruyer et al. (2014) Rémy-Ruyer A. et al., 2014, A&A, 563, A31
  * Rieke & the JADES Collaboration (2023) Rieke M., the JADES Collaboration, 2023, arXiv e-prints, arXiv:2306.02466
  * Roberts-Borsani et al. (2023) Roberts-Borsani G. et al., 2023, Nature, 618, 480
  * Roberts-Borsani et al. (2016) Roberts-Borsani G. W. et al., 2016, ApJ, 823, 143
  * Robertson et al. (2015) Robertson B. E., Ellis R. S., Furlanetto S. R., Dunlop J. S., 2015, ApJ, 802, L19
  * Rohatgi (2022) Rohatgi A., 2022, Webplotdigitizer: Version 4.6
  * Rosdahl et al. (2022) Rosdahl J. et al., 2022, MNRAS, 515, 2386
  * Rosdahl et al. (2018) Rosdahl J. et al., 2018, MNRAS, 479, 994
  * Rosdahl et al. (2015) Rosdahl J., Schaye J., Teyssier R., Agertz O., 2015, MNRAS, 451, 34
  * Rosen & Bregman (1995) Rosen A., Bregman J. N., 1995, ApJ, 440, 634
  * Salim et al. (2007) Salim S. et al., 2007, ApJS, 173, 267
  * Sanders et al. (2023) Sanders R. L., Shapley A. E., Topping M. W., Reddy N. A., Brammer G. B., 2023, arXiv e-prints, arXiv:2301.06696
  * Santini et al. (2023) Santini P. et al., 2023, ApJ, 942, L27
  * Santos, Sobral & Matthee (2016) Santos S., Sobral D., Matthee J., 2016, MNRAS, 463, 1678
  * Saxena et al. (2023) Saxena A. et al., 2023, arXiv e-prints, arXiv:2302.12805
  * Schaerer & de Barros (2010) Schaerer D., de Barros S., 2010, A&A, 515, A73
  * Schaerer et al. (2022) Schaerer D., Marques-Chaves R., Barrufet L., Oesch P., Izotov Y. I., Naidu R., Guseva N. G., Brammer G., 2022, A&A, 665, L4
  * Schaye et al. (2015) Schaye J. et al., 2015, MNRAS, 446, 521
  * Schirmer (2016) Schirmer M., 2016, PASP, 128, 114001
  * Sérsic (1963) Sérsic J. L., 1963, Boletin de la Asociacion Argentina de Astronomia La Plata Argentina, 6, 41
  * Shapley et al. (2023) Shapley A. E., Sanders R. L., Reddy N. A., Topping M. W., Brammer G. B., 2023, arXiv e-prints, arXiv:2301.03241
  * Shen et al. (2023) Shen X., Vogelsberger M., Boylan-Kolchin M., Tacchella S., Kannan R., 2023, arXiv e-prints, arXiv:2305.05679
  * Shen et al. (2020) Shen X. et al., 2020, MNRAS, 495, 4747
  * Simmonds et al. (2023) Simmonds C. et al., 2023, MNRAS, 523, 5468
  * Smith et al. (2022) Smith A., Kannan R., Garaldi E., Vogelsberger M., Pakmor R., Springel V., Hernquist L., 2022, MNRAS, 512, 3243
  * Sobral et al. (2018) Sobral D., Santos S., Matthee J., Paulino-Afonso A., Ribeiro B., Calhau J., Khostovan A. A., 2018, MNRAS, 476, 4725
  * Song et al. (2016) Song M. et al., 2016, Astrophys. J., 825
  * Stanway & Eldridge (2018) Stanway E. R., Eldridge J. J., 2018, MNRAS, 479, 75
  * Stanway & Eldridge (2023) Stanway E. R., Eldridge J. J., 2023, MNRAS, 522, 4430
  * Stark et al. (2017) Stark D. P. et al., 2017, MNRAS, 464, 469
  * Stefanon et al. (2022) Stefanon M., Bouwens R. J., Illingworth G. D., Labbé I., Oesch P. A., Gonzalez V., 2022, ApJ, 935, 94
  * Steidel et al. (1996) Steidel C. C., Giavalisco M., Pettini M., Dickinson M., Adelberger K. L., 1996, ApJ, 462, L17
  * Steidel et al. (2014) Steidel C. C. et al., 2014, ApJ, 795, 165
  * Steinhardt et al. (2023) Steinhardt C. L., Sneppen A., Clausen T., Katz H., Rey M. P., Stahlschmidt J., 2023, arXiv e-prints, arXiv:2305.15459
  * Stiavelli et al. (2023) Stiavelli M., Morishita T., Chiaberge M., Grillo C., Rosati P., Schuldt S., Trenti M., Treu T., 2023, arXiv e-prints, arXiv:2308.14696
  * Strom et al. (2017) Strom A. L., Steidel C. C., Rudie G. C., Trainor R. F., Pettini M., Reddy N. A., 2017, ApJ, 836, 164
  * Sun et al. (2022) Sun F. et al., 2022, arXiv e-prints, arXiv:2209.03374
  * Sun et al. (2023) Sun G., Faucher-Giguère C.-A., Hayward C. C., Shen X., 2023, arXiv e-prints, arXiv:2305.02713
  * Sutherland & Dopita (2017) Sutherland R. S., Dopita M. A., 2017, ApJS, 229, 34
  * Tacchella et al. (2018) Tacchella S., Bose S., Conroy C., Eisenstein D. J., Johnson B. D., 2018, ApJ, 868, 92
  * Tacchella et al. (2022) Tacchella S. et al., 2022, MNRAS, 513, 2904
  * Teyssier (2002) Teyssier R., 2002, A&A, 385, 337
  * Thomas et al. (2013) Thomas D. et al., 2013, MNRAS, 431, 1383
  * Topping et al. (2022) Topping M. W., Stark D. P., Endsley R., Plat A., Whitler L., Chen Z., Charlot S., 2022, ApJ, 941, 153
  * Topping et al. (2023) Topping M. W. et al., 2023, arXiv e-prints, arXiv:2307.08835
  * Trebitsch et al. (2021) Trebitsch M. et al., 2021, A&A, 653, A154
  * Tremonti et al. (2004) Tremonti C. A. et al., 2004, ApJ, 613, 898
  * Trussler et al. (2023) Trussler J. A. A. et al., 2023, arXiv e-prints, arXiv:2308.09665
  * Tweed et al. (2009) Tweed D., Devriendt J., Blaizot J., Colombi S., Slyz A., 2009, A&A, 506, 647
  * van Hoof et al. (2014) van Hoof P. A. M., Williams R. J. R., Volk K., Chatzikos M., Ferland G. J., Lykins M., Porter R. L., Wang Y., 2014, MNRAS, 444, 420
  * Veilleux & Osterbrock (1987) Veilleux S., Osterbrock D. E., 1987, ApJS, 63, 295
  * Vogelsberger et al. (2014) Vogelsberger M. et al., 2014, Nature, 509, 177
  * Vogelsberger et al. (2020) Vogelsberger M. et al., 2020, MNRAS, 492, 5167
  * Waters et al. (2016) Waters D., Wilkins S. M., Di Matteo T., Feng Y., Croft R., Nagai D., 2016, MNRAS, 461, L51
  * Weingartner & Draine (2001) Weingartner J. C., Draine B. T., 2001, ApJ, 563, 842
  * Wilkins et al. (2017) Wilkins S. M., Feng Y., Di Matteo T., Croft R., Lovell C. C., Waters D., 2017, MNRAS, 469, 2517
  * Wilkins et al. (2016) Wilkins S. M., Feng Y., Di-Matteo T., Croft R., Stanway E. R., Bunker A., Waters D., Lovell C., 2016, MNRAS, 460, 3170
  * Wilkins et al. (2023a) Wilkins S. M. et al., 2023a, arXiv e-prints, arXiv:2305.18175
  * Wilkins et al. (2023b) Wilkins S. M. et al., 2023b, MNRAS, 522, 4014
  * Williams et al. (2018) Williams C. C. et al., 2018, ApJS, 236, 33
  * Witten et al. (2023) Witten C. et al., 2023, arXiv e-prints, arXiv:2303.16225
  * Xiao, Stanway & Eldridge (2018) Xiao L., Stanway E. R., Eldridge J. J., 2018, MNRAS, 477, 904
  * Yang et al. (2022) Yang L. et al., 2022, MNRAS, 514, 1148
  * Yi et al. (2023) Yi S. K. et al., 2023, arXiv e-prints, arXiv:2308.03566
  * Yung et al. (2019) Yung L. Y. A., Somerville R. S., Finkelstein S. L., Popping G., Davé R., 2019, MNRAS, 483, 2983
  * Yung et al. (2023) Yung L. Y. A., Somerville R. S., Finkelstein S. L., Wilkins S. M., Gardner J. P., 2023, arXiv e-prints, arXiv:2304.04348
  * Yusef-Zadeh, Morris & White (1984) Yusef-Zadeh F., Morris M., White R. L., 1984, ApJ, 278, 186
  * Zheng & Miralda-Escudé (2002) Zheng Z., Miralda-Escudé J., 2002, The Astrophysical Journal, 578, 33