We now resort to the decomposition (145) to collect integrands in
$\hstretch{2}{\hat{\hstretch{.5}{BS}}}_{C}(\cdot)$ and those in
$\hstretch{2}{\hat{\hstretch{.5}{BS}}}_{P}(\cdot)$ from both equations (143)
and (147), obtaining the following representation for the pre-default
valuation:

$\displaystyle v(t,s)$
$\displaystyle=e^{-\hat{r}_{V}(T-t)}\left(se^{(\hat{r}_{S}-\kappa\hat{b}_{V})(T-t)}-K\right)$
$\displaystyle\qquad-\hat{b}_{V}\int_{t}^{T}e^{-\hat{r}_{V}(u-t)}e^{-r(T-u)}\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{P}((1+\kappa)se^{r(T-u)},K,u-t)\mathrm{d}u$
$\displaystyle\qquad+(\hat{b}_{V}-b)\int_{t}^{T}e^{-\hat{r}_{V}(u-t)}e^{-r(T-u)}\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}((1+\kappa)se^{r(T-u)},K,u-t)\mathrm{d}u$
(148)

Finally, we recall that the BS functions are homogeneous of degree one
(Merton,, 1973) that is in the case of the call function:

$\displaystyle\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}(xs,xk,t)=x\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}(s,k,t)$
(149)

for any $x\in\mathbb{R}^{*}$, which implies that:

$\displaystyle\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}((1+\kappa)se^{r(T-u)},K,u-t)=(1+\kappa)e^{r(T-u)}\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}\left(s,\frac{\tilde{k}(u,K)}{1+\kappa},u-t\right)$
(150)

where we have defined the risk-free discounted strike
$\tilde{k}:[0,T]\times\mathbb{R}_{+}^{*}\rightarrow\mathbb{R}_{+}^{*}$ as
$\tilde{k}(t,K):=Ke^{-r(T-t)}$. We make the same transformation to the put
values and the result ensues. ∎

###  B.8 Proof of Corollary 2

We state the following Lemma.

######  Lemma 4.

Let $x,y\in\mathbb{R}$ and define the function
$\Upsilon_{0}:\mathbb{R}_{+}\times\mathbb{R}^{2}\rightarrow\mathbb{R}_{+}$ as
follows:

$\displaystyle\Upsilon_{0}(t,x,y):=\int_{0}^{t}e^{-xu}\Phi\left(y\sqrt{u}\right)\mathrm{d}u$
(151)

where $\Phi:\mathbb{R}\rightarrow[0,1]$ is the CDF of a standard normal
variable. Then:

$\displaystyle\Upsilon_{0}(t,x,y)=\frac{y}{x\sqrt{c}}\left(\Phi\left(\sqrt{ct}\right)-\frac{1}{2}\right)-\frac{1}{x}\left(e^{-xt}\Phi\left(y\sqrt{t}\right)-\frac{1}{2}\right)$
(152)

where $c:=2x+y^{2}$.

###### Proof.

Using integration by parts, we get:

$\displaystyle\Upsilon_{0}(t,x,y)=\left[\frac{e^{-xu}}{-x}\Phi\left(y\sqrt{u}\right)\right]_{u=0}^{t}+\int_{0}^{t}\left(\frac{e^{-xu}}{x}\right)\left(\frac{y}{2\sqrt{u}}\phi\left(y\sqrt{u}\right)\right)\mathrm{d}u$
(153)

where $\phi:\mathbb{R}\rightarrow\mathbb{R}_{+}$ is the probability density
function of a standard Gaussian random variable. For the first term on the
right-hand side of (153), we have:

$\displaystyle\left[\frac{e^{-xu}}{-x}\Phi\left(y\sqrt{u}\right)\right]_{u=0}^{t}=\frac{1}{x}\left(\Phi(0)-e^{-xT}\Phi\left(y\sqrt{t}\right)\right)$
(154)

For the second term in (153), let us define $c:=2x+y^{2}$ and make the
following change of variables:

$\displaystyle
w:=\sqrt{cu},\quad\quad\quad\mathrm{d}w=\frac{1}{2}\sqrt{\frac{c}{u}}\mathrm{d}u$
(155)

Then:

$\displaystyle\int_{0}^{t}\left(\frac{e^{-xu}}{x}\right)\left(\frac{y}{2\sqrt{u}}\phi\left(y\sqrt{u}\right)\right)\mathrm{d}u$
$\displaystyle=\frac{y}{x}\int_{0}^{\sqrt{ct}}e^{-w^{2}\left(\frac{x}{z}\right)}\frac{1}{\sqrt{c}}\phi\left(w\left(\frac{y}{\sqrt{c}}\right)\right)\mathrm{d}w$
$\displaystyle=\frac{y}{x\sqrt{c}}\int_{0}^{\sqrt{zt}}\frac{1}{\sqrt{2\pi}}e^{-\frac{w^{2}}{2}\left(\frac{2x}{c}+\frac{y^{2}}{c}\right)}\mathrm{d}w$
$\displaystyle=\frac{y}{x\sqrt{c}}\left(\Phi\left(\sqrt{ct}\right)-\Phi(0)\right)$
(156)

Combining the terms (154) and (156) and using the fact that $\Phi(0)=1/2$, we
obtain the desired result. ∎

We can now proceed to prove Corollary 2 from page 2.

###### Proof.

Recalling that $\tilde{k}(u,K)=Ke^{-r(T-u)}$, we start by rearranging terms
inside the function $d_{1}$ from Theorem 2:

$\displaystyle d_{1}(u-t)$
$\displaystyle=\frac{1}{\sigma\sqrt{u-t}}\left(\ln\left(\frac{s}{\frac{\tilde{k}(u,K)}{(1+\kappa)}}\right)+\left(\hat{r}_{S}-\kappa\hat{b}_{V}+\frac{\sigma^{2}}{2}\right)(u-t)\right)$
$\displaystyle=\frac{1}{\sigma\sqrt{u-t}}\left(\ln\left(\frac{(1+\kappa)s}{K}\right)+r((T-t)-(u-t))+\left(\hat{r}_{S}-\kappa\hat{b}_{V}+\frac{\sigma^{2}}{2}\right)(u-t)\right)$
$\displaystyle=\frac{1}{\sigma\sqrt{u-t}}\left(\ln\left(\frac{(1+\kappa)se^{r(T-t)}}{K}\right)+\left(\hat{r}_{S}-\kappa\hat{b}_{V}-r+\frac{\sigma^{2}}{2}\right)(u-t)\right)$
(157)

A similar manipulation can be done for $d_{2}$. Hence if
$K=K^{\star}:=(1+\kappa)se^{r(T-t)}$ we have:

$\displaystyle d_{1}(u-t)$
$\displaystyle=\frac{1}{\sigma}\left(\hat{r}_{S}-\kappa\hat{b}_{V}-r+\frac{\sigma^{2}}{2}\right)\sqrt{u-t}$
(158)

where we recall $d_{2}(t):=d_{1}(t)-\sigma\sqrt{t}$. We define the variables
$\nu_{1}$ and $\nu_{2}$:

$\displaystyle\nu_{1}$
$\displaystyle:=\frac{1}{\sigma}\left(\hat{r}_{S}-\kappa\hat{b}_{V}-r+\frac{\sigma^{2}}{2}\right)$
(159) $\displaystyle\nu_{2}$ $\displaystyle:=\nu_{1}-\sigma$ (160)

such that $d_{i}(u-t)=\nu_{i}\sqrt{u-t}$. Let us focus on the call integral in
formula (71) from Theorem 2. When $K=K^{\star}$ and defining the drift
variable $\hat{\mu}:=\hat{r}_{S}-\kappa\hat{b}_{V}$, the Black-Scholes call
integrand becomes:

$\displaystyle\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}\left(s,\frac{\tilde{k}(u,K^{\star})}{1+\kappa},u-t\right)$
$\displaystyle=se^{(\hat{r}_{S}-\kappa\hat{b}_{V})(u-t)}\Phi\left(\nu_{1}\sqrt{u-t}\right)-\frac{e^{-r(T-u)}K^{\star}}{1+\kappa}\Phi\left(\nu_{2}\sqrt{u-t}\right)$
$\displaystyle=s\left(e^{\hat{\mu}(u-t)}\Phi\left(\nu_{1}\sqrt{u-t}\right)-e^{r(u-t)}\Phi\left(\nu_{2}\sqrt{u-t}\right)\right)$
(161)

Therefore the call integral from formula (71) is equal to the following, where
we have made the change of variable $w:=u-t$:

$\displaystyle\int_{t}^{T}e^{-\hat{r}_{V}(u-t)}\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}\left(s,\frac{\tilde{k}(u,K)}{1+\kappa},u-t\right)\mathrm{d}u$
$\displaystyle=s\left(\int_{t}^{T}e^{-(\hat{r}_{V}-\hat{\mu})(u-t)}\Phi\left(\nu_{1}\sqrt{u-t}\right)\mathrm{d}u-\int_{t}^{T}e^{-\hat{b}_{V}(u-t)}\Phi\left(\nu_{2}\sqrt{u-t}\right)\mathrm{d}u\right)$
$\displaystyle=s\left(\int_{0}^{T-t}e^{-(\hat{r}_{V}-\hat{\mu})w}\Phi\left(\nu_{1}\sqrt{w}\right)\mathrm{d}w-\int_{0}^{T-t}e^{-\hat{b}_{V}w}\Phi\left(\nu_{2}\sqrt{w}\right)\mathrm{d}w\right)$
(162)

By using Lemma 4 and defining time to maturity $\mathcal{T}:=T-t$, we
conclude:

$\displaystyle\int_{t}^{T}e^{-\hat{r}_{V}(u-t)}\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{C}\left(s,\frac{\tilde{k}(u,K)}{1+\kappa},u-t\right)\mathrm{d}u=s\left(\Upsilon_{0}(\mathcal{T},\hat{r}_{V}-\hat{\mu},\nu_{1})-\Upsilon_{0}(\mathcal{T},\hat{b}_{V},\nu_{2})\right)$
(163)

The same argument applies to the integral along Black-Scholes put prices in
equation (71), whose integrand is equal to:

$\displaystyle\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{P}\left(s,\frac{\tilde{k}(u,K^{\star})}{1+\kappa},u-t\right)$
$\displaystyle=s\left(e^{r(u-t)}\Phi\left(-\nu_{2}\sqrt{u-t}\right)-e^{\hat{\mu}(u-t)}\Phi\left(-\nu_{1}\sqrt{u-t}\right)\right)$
(164)

by definition (70). Therefore:

$\displaystyle\int_{t}^{T}e^{-\hat{r}_{V}(u-t)}\hstretch{2}{\hat{\hstretch{.5}{\mathcal{BS}}}}_{P}\left(s,\frac{\tilde{k}(u,K)}{1+\kappa},u-t\right)\mathrm{d}u$
$\displaystyle=s\left(\int_{t}^{T}e^{-\hat{b}_{V}(u-t)}\Phi\left(-\nu_{2}\sqrt{u-t}\right)\mathrm{d}u-\int_{t}^{T}e^{-(\hat{r}_{V}-\hat{\mu})(u-t)}\Phi\left(-\nu_{1}\sqrt{u-t}\right)\right)$
$\displaystyle=s\left(\int_{0}^{\mathcal{T}}e^{-\hat{b}_{V}w}\Phi\left(-\nu_{2}\sqrt{w}\right)\mathrm{d}w-\int_{0}^{\mathcal{T}}e^{-(\hat{r}_{V}-\hat{\mu})w}\Phi\left(-\nu_{1}\sqrt{w}\right)\mathrm{d}w\right)$
$\displaystyle=s\left(\Upsilon_{0}(\mathcal{T},\hat{b}_{V},-\nu_{2})-\Upsilon_{0}(\mathcal{T},\hat{r}_{V}-\hat{\mu},-\nu_{1})\right)$
(165)

Finally, when $K=K^{\star}$, the terminal component in the forward’s valuation
formula (71) is equal to:

$\displaystyle
e^{-\hat{r}_{V}(T-t)}\left(se^{(\hat{r}_{S}-\kappa\hat{b}_{V})(T-t)}-K^{\star}\right)$
$\displaystyle=e^{-\hat{r}_{V}\mathcal{T}}\left(se^{\hat{\mu}\mathcal{T}}-(1+\kappa)se^{r\mathcal{T}}\right)$
$\displaystyle=se^{-(\hat{r}_{V}-r)\mathcal{T}}\left(e^{(\hat{\mu}-r)\mathcal{T}}-(1+\kappa)\right)$
(166)

By replacing $\hat{r}_{V}-r$ by $\hat{b}_{V}$ above and combining equations
(163), (165) and (166) we obtain the desired result. ∎

###  B.9 Proof of Theorem 3

We first state the following Lemma, which gives an approximation to the
general version of function $\Upsilon_{0}$ introduced in Lemma 4 from Appendix
B.8.

######  Lemma 5.

Let $x,y,z\in\mathbb{R}$ and define the function
$\Upsilon:\mathbb{R}_{+}\times\mathbb{R}^{3}\rightarrow\mathbb{R}_{+}$ as
follows:

$\displaystyle\Upsilon(t,x,y,z):=\int_{0}^{t}e^{-xu}\Phi\left(y\sqrt{u}+\frac{z}{\sqrt{u}}\right)\mathrm{d}u$
(167)

where $\Phi:\mathbb{R}\rightarrow[0,1]$ is the CDF of a standard normal
variable. Then the following approximation holds for $z\rightarrow 0$:

$\displaystyle\Upsilon(t,x,y,z)=\Upsilon_{0}(t,x,y)+\frac{2}{\sqrt{c}}\left(\Phi(\sqrt{ct})-\frac{1}{2}\right)\left(z-\frac{y}{2}z^{2}\right)+\mathcal{O}(z^{3})$
(168)

where $c:=2x+y^{2}$.

###### Proof.

Throughout the proof we use the notation $\Upsilon(t):=\Upsilon(t,x,y,z)$. To
derive an approximation, we use a second-order Taylor expansion in $z$ for the
integrand in (167) when $z$ is near 0. To avoid convergence problems when
integrating this expansion, we split $\Upsilon(t)$ into a bottom residual term
and the rest of the integral, and show this residual term can be made
arbitrarily small: this ensures our approximation is well behaved.

We start by making the split
$\Upsilon(t)=\Upsilon(\varepsilon)+(\Upsilon(t)-\Upsilon(\varepsilon))$ and
define $\Upsilon(\varepsilon,t):=\Upsilon(t)-\Upsilon(\varepsilon)$ for some
small real number $\varepsilon>0$. Because $0<\Phi(u)<1$ for any
$u\in\mathbb{R}$, the integral $\Upsilon(t)$ converges for any $0\leq
t<\infty$ and the residual term $\Upsilon(\varepsilon)$ is bounded from below
and above:

$\displaystyle
0<\Upsilon(\varepsilon)<\int_{0}^{\varepsilon}e^{-xu}\mathrm{d}u=\frac{1}{x}(1-e^{-x\varepsilon})$
(169)

Hence we can make the bottom residue arbitrarily small, namely
$\Upsilon(t)=\lim_{\varepsilon\rightarrow 0}\Upsilon(\varepsilon,t)$. We now
consider the integral $\Upsilon(\varepsilon,t)$ which is convergent because
the integrand is finite for any $u$. Define the function
$g:\mathbb{R}\rightarrow[0,1]$ taking $z$ as argument:

$\displaystyle g(z):=\Phi\left(y\sqrt{u}+\frac{z}{\sqrt{u}}\right)$ (170)

Recalling that $\phi^{\prime}(u)=-u\phi(u)$, we obtain the first and second
derivative of $g$:

$\displaystyle
g^{\prime}(z)=\frac{1}{\sqrt{u}}\phi\left(y\sqrt{u}+\frac{z}{\sqrt{u}}\right),\qquad
g^{\prime\prime}(z)=-g^{\prime}(z)\left(y+\frac{z}{u}\right)$ (171)

A second-order Taylor expansion of $g$ around 0 gives us:

$\displaystyle
g(z)=g(0)+g^{\prime}(0)z\left(1-\frac{yz}{2}\right)+\mathcal{O}(z^{3})$ (172)

Therefore when $z\rightarrow 0$ we make the following approximation of order
$z^{2}$ for the integral $\Upsilon(\varepsilon,t)$ which is obtained by
integrating $g$ with respect to $u$:

$\displaystyle\Upsilon(\varepsilon,t)=\int_{\varepsilon}^{t}e^{-xu}\Phi(y\sqrt{u})\mathrm{d}u+z\left(1-\frac{yz}{2}\right)\int_{\varepsilon}^{t}e^{-xu}\frac{1}{\sqrt{u}}\phi(y\sqrt{u})\mathrm{d}u+\mathcal{O}(z^{3})$
(173)

By making the change of variable $w=\sqrt{cu}$ as in the proof of Lemma 4, we
find closed-form expressions for the two integrals in equation (173):

$\displaystyle\int_{\varepsilon}^{t}e^{-xu}\Phi(y\sqrt{u})\mathrm{d}u$
$\displaystyle=\frac{y}{x\sqrt{c}}\left(\Phi\left(\sqrt{ct}\right)-\Phi(\sqrt{c\varepsilon})\right)-\frac{1}{x}\left(e^{-xt}\Phi\left(y\sqrt{t}\right)-\Phi(\sqrt{c\varepsilon})\right)$
(174)
$\displaystyle\int_{\varepsilon}^{t}e^{-xu}\frac{1}{\sqrt{u}}\phi(y\sqrt{u})\mathrm{d}u$
$\displaystyle=\frac{2}{\sqrt{c}}\left(\Phi(\sqrt{ct})-\Phi(\sqrt{c\varepsilon})\right)$
(175)

where we recall that $c=2x+y^{2}$. Note that $\lim_{\varepsilon\rightarrow
0}\Phi(\sqrt{c\varepsilon})=1/2$ and define the integral
$\tilde{\Upsilon}(\varepsilon,t):\mathbb{R}_{+}^{*}\rightarrow\mathbb{R}$ as:

$\displaystyle\tilde{\Upsilon}(\varepsilon,t):=\int_{\varepsilon}^{t}e^{-xu}\Phi(y\sqrt{u})\mathrm{d}u+z\left(1-\frac{yz}{2}\right)\int_{\varepsilon}^{t}e^{-xu}\frac{1}{\sqrt{u}}\phi(y\sqrt{u})\mathrm{d}u$
(176)

Then
$\Upsilon(\varepsilon,t)=\tilde{\Upsilon}(\varepsilon,t)+\mathcal{O}(z^{3})$.
Moreover $\tilde{\Upsilon}(t):=\lim_{\varepsilon\rightarrow
0}\tilde{\Upsilon}(\varepsilon,t)$ is an approximation of order $z^{2}$ to
$\Upsilon(t)$ and equal to:

$\displaystyle\tilde{\Upsilon}(t)$
$\displaystyle=\Upsilon_{0}(t,x,y)+\frac{2}{\sqrt{c}}\left(\Phi(\sqrt{ct})-\frac{1}{2}\right)\left(z-\frac{y}{2}z^{2}\right)$
(177)

which concludes the proof. ∎

We can now prove the main theorem.

###### Proof.

The proof follows the same steps as that for Corollary 2, located in Appendix
B.8. In fact, notice equation (157) for the function $d_{1}$ is equal to:

$\displaystyle d_{1}(u-t)$
$\displaystyle=\frac{1}{\sigma\sqrt{u-t}}\left(\ln\left(\frac{(1+\kappa)se^{r(T-t)}}{K}\right)+\left(\hat{r}_{S}-\kappa\hat{b}_{V}-r+\frac{\sigma^{2}}{2}\right)(u-t)\right)$
$\displaystyle=\frac{\bar{m}}{\sqrt{u-t}}+\nu_{1}\sqrt{u-t}$ (178)

where $\bar{m}$ is the log-moneyness as defined in equation (82) from the
theorem’s statement, while $\nu_{1}$ was defined in Corollary 2. We then
proceed as in the proof of the latter, but by assuming $\bar{m}\neq 0$ and
replacing the function $\Upsilon_{0}$ by the approximation $\tilde{\Upsilon}$
introduced in Lemma 5. ∎


capbtabboxtable[][]

#  Few-shot Multimodal Multitask Multilingual Learning

Aman Chadha
Department of Computer Science
Stanford University
<EMAIL_ADDRESS>
&Vinija Jain
Department of Computer Science
Stanford University
<EMAIL_ADDRESS>
##  1 Abstract

While few-shot learning as a transfer learning paradigm has gained significant
traction for scenarios with limited data, it has primarily been explored in
the context of building unimodal and unilingual models. Furthermore, a
significant part of the existing literature in the domain of few-shot
multitask learning perform in-context learning which requires manually
generated prompts as the input, yielding varying outcomes depending on the
level of manual prompt-engineering. In addition, in-context learning suffers
from substantial computational, memory, and storage costs which eventually
leads to high inference latency because it involves running all of the
prompt’s examples through the model every time a prediction is made. In
contrast, methods based on the transfer learning via the fine-tuning paradigm
avoid the aforementioned issues at a one-time cost of fine-tuning weights on a
per-task basis. However, such methods lack exposure to few-shot multimodal
multitask learning. In this paper, we propose few-shot learning for a
multimodal multitask multilingual (FM3) setting by adapting pre-trained vision
and language models using task-specific hypernetworks and contrastively fine-
tuning them to enable few-shot learning. FM3’s architecture combines the best
of both worlds of in-context and fine-tuning based learning and consists of
three major components: (i) multimodal contrastive fine-tuning to enable few-
shot learning, (ii) hypernetwork task adaptation to perform multitask
learning, and (iii) task-specific output heads to cater to a plethora of
diverse tasks. FM3 learns the most prominent tasks in the vision and language
domains along with their intersections, namely visual entailment (VE) [1],
visual question answering (VQA) [2], and natural language understanding (NLU)
tasks such as neural entity recognition (NER) and the GLUE benchmark [3]
including QNLI [4], MNLI [5], QQP [6], and SST-2 [7].

##  2 Introduction

Self-supervised pretraining has propelled the adoption of deep learning on
tasks with limited labeled data. With their task-agnostic features and
improved data efficiency, self-supervised pre-trained models have drastically
reduced the opportunity cost to tackle tasks that earlier required a
significant amount of data and thus proved intractable using supervised
learning. As a result of the advancements in self-supervised pretraining,
semi-supervised approaches that combine self-supervision with supervised
learning on a task-specific dataset that tackles a related task, have emerged
as a new paradigm that has enabled transfer learning.

One of the biggest open challenges for machine learning research is building
models that can be rapidly adapted to novel tasks using only a handful of
annotated examples. The domain of few-shot learning (FSL), which is a specific
variant of transfer learning, has emerged as an attractive solution to label-
scarce scenarios where data annotation can be time-consuming and costly. These
methods are designed to work with a small number of labeled training examples,
and typically involve adapting pre-trained models for specific downstream
tasks. Several flavors of FSL methods exist, each with its pros and cons.

One such large-scale self-supervised approach, popularized by the arrival of
the generative pre-trained transformer (GPT) series [8, 9] of NLP models, is
transfer learning via in-context learning (ICL) which emerges from training at
scale. ICL teaches a model to perform a downstream task by feeding in a prompt
with a nominal set of supervised examples as input to the model along with a
single unlabeled example for which a prediction is desired. In effect, few-
shot prompting using a small collection of input-target pairs offers a walk-
through to the model on how to transform the input into the output. Notably,
since ICL requires no parameter updates, i.e., no gradient-based training is
required, a single model can effectively act as a swiss-army knife by being
able to immediately perform a wide variety of tasks. ICL, therefore, solely
relies on the capabilities that a model learned during pretraining. The ease
of use and quick adaptability to target tasks are characteristic features that
have caused widespread adoption of ICL [10, 11, 12, 13, 14].

While ICL offers a multitude of benefits, it also suffers from several major
drawbacks. First, processing all the prompted input-target pairs every time
the model makes a prediction incurs significant compute, memory, and latency
costs. These costs stack up as the number of the inferences increases – in a
situation where the goal is to perform inference over a batch of test examples
rather than one-off predictions, ICL can prove to be impractical from a
resource standpoint. Second, owing to a limited-length context window, the
number of support examples $k$ that the model can utilize are restricted to
nominal numbers. This is because we must fit all $k$ examples into the model’s
context window, which is limited to a specific number of tokens (1024 in case
of GPT-2 and 2048 in case of GPT-3). Third, ICL typically produces inferior
performance compared to fine-tuning [15, 8, 16]. Finally, while the model’s
performance is a function of semantic and structural aspects of the prompt
which can cause a significant yet unpredictable impact on the model’s
performance [17], far beyond inter-run variation of fine-tuning. In
particular, semantic changes such as the phrasing or choice of words in the
prompt and syntactic changes such as the exact formatting of the prompt
(including the wording [18] and ordering of examples [19]) can cause a
significant, unintended, and difficult-to-estimate impact on the model’s
performance. Furthermore, recent work has also demonstrated that ICL can
perform well even when provided with incorrect labels, raising concerns as to
how much learning is taking place at all [13].

Another common semi-supervised learning paradigm is transfer learning via
fine-tuning (FT) which follows a two-staged process: (i) utilize the
parameters of a pre-trained large-scale self-supervised model learning for
weight initialization, and (ii) perform gradient-based fine-tuning using data
associated with the downstream task of interest. With the advent of
representation-learning approaches such as BERT [20], the domain of NLP
underwent a radical transformation from supervised to semi-supervised
approaches for tasks such as sentiment analysis, neural entity recognition,
question answering, summarization, conversational response generation, etc.
Representation-learning approaches have now taken center-seat in NLP, with the
learned contextualized representations from these pre-trained models serving
as initial task-agnostic features that, in turn, offer a the starting point
for learning task-specific features. While problems with limited labeled data
have benefited significantly owing to the reduced data-appetite of semi-
supervised approaches, tasks with abundant labeled data have also seen
improved performance.

While FT has produced many state-of-the-art (SoTA) results [21] on a range of
classification tasks, it results in a model that is specialized for a single
task with an entirely new set of parameter values, which can become
impractical when FT a model on many downstream tasks. In other words, such
models typically perform one task at a time, and cannot learn new concepts or
adapt to new tasks in a few shots. FM3 seeks to address this gap and enable
multimodal FSL – much like how SetFit contrastively fine-tunes pre-trained
Sentence Transformer models [22] and dispenses with prompts altogether and
does not require large-scale pre-trained LMs to achieve high accuracy. With
only 8 labeled examples in the Customer Reviews (CR) sentiment dataset, SetFit
is competitive with RoBERTa finetuned on the full training set [23], despite
the fine-tuned model being three times larger.

Both ICL and fine-tuning have been explored in a multimodal context. A slew of
methods, notably Flamingo [15] and Frozen [24], perform ICL with the final
objective to have the model rapidly adapt to a variety of multimodal tasks.
While Flamingo achieves competitive performance with FSL, in some cases
outperforming models fine-tuned on thousands of times more task-specific data,
Frozen offers relatively lower performance in return for the flexibility of
using an off-the-shelf pre-trained LM and keeping its weights frozen. On the
other hand, Oscar [25] and Omninet [26] are multimodal multitask models that
do not perform ICL. While Oscar is pre-trained using pre-trained with aligned
data on task-agnostic cross-modal objectives (a masked token loss over words
and visual tags, and a contrastive loss between visual tags and others) and
then fine-tuned to specific tasks, Omninet is simultaneously trained on its
target tasks and undergoes no finetuning. In the zero-/few-shot learning
context, multimodal pretraining has recently shown to enable strong
generalization in the discriminative setting using large-scale contrastive
learning [27, 28].

An additional paradigm for enabling a model to perform a new task with minimal
updates is parameter efficient fine-tuning (PEFT), where a pre-trained model
is fine-tuned by only updating a small number of added or selected parameters.
Recent methods have matched the performance of fine-tuning the full model
while only updating or adding a small fraction (e.g. 0.01%) of the full
model’s parameters [29, 30]. Furthermore, certain PEFT methods allow mixed-
task batches where different examples in a batch are processed differently
[30], making both PEFT and ICL viable for multitask models. While the benefits
of PEFT address some shortcomings of fine-tuning (when compared to ICL), there
has been relatively little focus on whether PEFT methods work well when very
little labeled data is available. [16] closes this gap by proposing T-Few, a
model that learns using PEFT and a fixed set of hyperparameters, attaining
strong performance on novel, unseen tasks while only updating a tiny fraction
of the model’s parameters.

FM3 combines the best of both worlds of ICL- and FT-based transfer learning
and offers an efficient and prompt-free framework that offers strong
generalization to new multimodal vision-language tasks in a few-shot setting.
Despite the flexibility offered by ICL, its limitations leave much to be
desired, especially in situations where compute, latency, memory, batch
inference, performance determinism, etc. are important. On the other hand, FT
offers performance invariance since it does not require prompts, offers better
performance than ICL-based methods [15], and is resource-efficient in terms of
compute, latency, memory, etc. While zero-/few-shot generalization is a
desirable by-product of ICL, the only significant downside to FT is that
generalization to new tasks with limited data is challenging. FM3 is
architected keeping the aforementioned drawbacks of ICL in mind and thus
follows the FT approach but overcomes its limitations as follows: (i)
multimodal contrastive fine-tuning to enable FSL, (ii) using hypernetworks
with a limited parameter count to perform task adaptation for multitask
learning, and (iii) task-specific output heads to cater to a plethora of
diverse tasks.

FM3 achieves high accuracy with little labeled data - for instance, with only
16 labeled examples per class on the complex task of SNLI-VE [1], FM3
surpasses the current SoTA fine-tuned on the full training set of 430K
examples! Compared to other FSL methods, FM3 has several unique features:

  * •

No prompts or verbalisers: Current techniques for few-shot fine-tuning require
handcrafted prompts or verbalisers to convert examples into a format that’s
suitable for the underlying language model. FM3 dispenses with prompts
altogether by generating rich embeddings directly from text examples. This
obliterates the need for manual prompt engineering, which in turn, results in
performance determinism.

  * •

Resource efficiency: Optimal use of compute, latency, memory, etc. compared to
our baselines Flamingo, Frozen, and especially ICL-based methods.

  * •

Frozen pre-trained models: FM3 uses pre-trained vision and language encoders
without fine-tuning them. This implies that FM3 architecture enables drop-in
plug-and-play replacement for modality encoders. Only small hypernetwork
models need to be fine-tuned when experimenting with different encoders.

  * •

Fast to train: FM3 doesn’t require large models like Flamingo (80B) or Frozen
(7B+) to achieve high accuracy. As a result, it is significantly faster to
train and run inference with.

  * •

Multilingual support: FM3 enables multilingual processing, and can be paired
up with any multilingual text encoder such as multilingual Sentence
Transformer [22] variants of MPNet [31], RoBERTa [32], ALBERT [33], LASER
[34], etc., which enables multilingual learning in 50+ languages by simply
fine-tuning a multilingual model checkpoint.

While proposals that address a subset of the areas of few-shot multimodal
multitask multilingual learning exist, to our knowledge, FM3 is the first to
explore the intersection of the domains of multimodal multitask multilingual
learning in a FSL setting.

##  3 Related Work

###  3.1 Few-shot learning using pre-trained models

In the domain of NLP, SetFit proposed by Tunstall et al. [23] is an efficient
and prompt-free framework for few-shot fine-tuning of Sentence Transformers
(ST). SetFit works by fine-tuning a pre-trained ST on a small number of text
pairs in a contrastive Siamese manner. The resulting model is then used to
generate rich text embeddings, which are used to train a classification head.
This simple framework requires no prompts, and achieves high accuracy with
orders of magnitude less parameters than existing techniques. SetFit obtains
comparable results with parameter-efficient fine-tuning (PEFT) and parameter
efficient tuning (PET) techniques, while being an order of magnitude faster to
train. SetFit achieves high accuracy with little labeled data - for instance,
with only 8 labeled examples per class on the Customer Reviews sentiment
dataset [35], SetFit is competitive with fine-tuning RoBERTa Large on the full
training set of 3K examples. Owing to its practical utility in enabling FSL,
we adopt the idea of contrastive fine-tuning from SetFit and generalize it to
a multimodal multitask multilingual setting as part of FM3.

###  3.2 Multitask fine-tuning using PEFT

In [36], Houlsby et al. propose a parameter-efficient fine-tuning method which
introduces adapter modules between the layers of a pre-trained language model.
Adapter modules yield a compact and extensible model; they add only a few
trainable parameters per task, and new tasks can be added without revisiting
previous ones. The parameters of the original network remain fixed, yielding a
high degree of parameter sharing. They achieve SoTA performance on GLUE [3]
whilst adding only a few parameters per task. However, the downside of this
approach is that they are trained separately for each task and thus do not
enable sharing information across tasks.

To circumvent the aforementioned issue of knowledge sharing across tasks,
Mahabadi et al. [37] learn adapter parameters for all layers and tasks using
shared hypernetworks, which condition on task, adapter position, and layer ID
in a transformer model. This parameter-efficient multitask learning framework
achieves the best of both worlds by sharing knowledge across tasks via
hypernetworks while enabling the model to adapt to each individual task
through task-specific adapters. Experiments on the GLUE benchmark show
improved performance in multitask learning while adding only 0.29% parameters
per task. Given the fact that hypernetworks enable easy multitask fine-tuning
of pre-trained models without having to actually fine-tune the model’s weights
(i.e., they remain frozen in this process), we adopt multitask finetuning in
our proposed architecture.

###  3.3 Multitask multimodal learning

Hu and Singh propose UniT [38], a Unified Transformer encoder-decoder model
that learns 7 tasks jointly across 8 datasets spread over different vision and
lanuage domains, ranging from object detection to natural language
understanding and multimodal reasoning. UniT achieves strong performance with
significantly few parameters in some cases outperforming separately trained
single task models. While the architecture offers joint end-to-end training of
each task, it requires a substantial amount of data across all tasks for the
model to generalize. Our approach, on the other hand, utilizes FSL to
efficiently learn a task with a small fraction of data.

In [26], Pramanik et al. propose OmniNet, a single model to support tasks with
multiple input modalities as well as asynchronous multitask learning. OmniNet
is powered by a spatio-temporal cache that enables learning spatial dimension
of the input in addition to the hidden states corresponding to the temporal
input sequence. Even though OmniNet is $3\times$ parameter-efficient, there is
a significant performance gap on most tasks it was trained on compared to the
individual model counterparts.

###  3.4 Multitask multilingual multimodal learning

$M^{3}P$, proposed in [39], is a multitask multilingual multimodal pre-trained
model that combines multilingual pre-training and multimodal pre-training into
a unified framework via multitask pre-training. $M^{3}P$ learns universal
representations that can map objects occurred in different modalities or texts
expressed in different languages into a common semantic space. In addition, to
alleviate the issue of lack of sufficient labeled data for non-English
multimodal tasks, they propose multimodal code-switched training (MCT) [40]
which replaces each word in the caption with a translated word with a
probability of $\beta$. If a word has multiple translations, a random one is
chosen. Experiments on the multilingual image retrieval task across MS COCO
[41] and Multi30K [42] show competitive results for English and new establish
SoTA results for non-English languages. While $M^{3}P$ tackles a similar
problem as FM3 , it does not assume any restrictions on the annotation budget
– in other words, it does not consider the few-shot setting for learning its
tasks.

###  3.5 Few-shot multimodal multitask learning

In [15], Alayrac et al. introduce Flamingo, a family of Visual Language Models
(VLM) trained on large-scale multimodal web corpora with an ability to rapidly
adapt to a variety of image and video tasks. Flamingo proposes the following
key architectural innovations: (i) bridge powerful pre-trained vision-only and
language-only models, (ii) handle sequences of arbitrarily interleaved visual
and textual data, and (iii) seamlessly ingest images or videos as inputs. The
end result is a single Flamingo model that can achieve a new SoTA with FSL,
simply by prompting the model with task-specific examples. On numerous
benchmarks, Flamingo outperforms models fine-tuned on thousands of times more
task-specific data. These include open-ended tasks such as visual question-
answering, where the model is prompted with a question which it has to answer;
captioning tasks, which evaluate the ability to describe a scene or an event;
and close-ended tasks such as multiple-choice visual question-answering.

In [24], Tsimpoukelli et al. present _Frozen_ , a simple-yet-effective
approach for transferring the FSL abilities inherent in large auto-regressive
language models to a multimodal setting (vision and language). _Frozen_ is a
multimodal few-shot learner, with the surprising ability to learn a variety of
new tasks when conditioned on examples, represented as a sequence of multiple
interleaved image and text embeddings. _Frozen_ can rapidly learn words for
new objects and novel visual categories and do visual question-answering with
only a handful of examples. While this work serves as an important baseline
for FM3, a key limitation is that it achieves far from SoTA performance on the
specific tasks that it learns in a few shot setting. _Frozen_ shows that
training a visual encoder through a pre-trained and frozen language model
results in a system capable of strong out-of-distribution (zero-shot)
generalization. Furthermore, _Frozen_ confirms that the ability to rapidly
adapt to new tasks given appropriate prompts is inherited from the pre-trained
language model and transfers directly to multimodal tasks.

While Flamingo and _Frozen_ are both ICL-based FSL methods, the
differentiating factors are: (i) the scale of data used to train these models,
and (ii) architectural variations. Flamingo is trained on large-scale
multimodal web corpora while is _Frozen_ is trained on the Conceptual Captions
dataset [43]. The architectural design choices differ between the two in using
pre-trained modality encoders vs. training them from scratch. Similar to FM3,
Flamingo uses off-the-shelf pre-trained encoders and only generates adapter
components (in the form of Perceiver Resampler blocks) while _Frozen_ utilizes
a pre-trained LM but trains its own vision encoder that feeds the LM. Inspired
by this observation, FM3 borrows the idea of using separate text and vision
adapters in the form of hypernetworks so as to offer the model additional
degrees of freedom, which in turn, helps render better performance.

##  4 FM3

Figure 1: Architectural overview of FM3. FM3 consists of three stages: (i)
contrastive pair mining for fine-tuning, which generates positive and negative
pairs, (ii) task-based fine-tuning involves adapting the pre-trained text and
image encoder models for down-stream tasks using hypernetworks, and (iii)
training task-specific classification heads.

###  4.1 Methods

Figure 1 offers a visual summary of the architectural stages of FM3.

####  4.1.1 Task and batch sampling

At each iteration during training, we randomly select a task with a sampling
probability that can be manually specified based on the dataset size. Once the
task list has been sampled, for tasks with multiple datasets, we randomly
sample a dataset corresponding to that task to fill a batch of samples.

####  4.1.2 Contrastive fine-tuning for few-shot learning

Similar to [23], we utilize a contrastive learning approach to FSL.
Contrastive learning effectively enlarges the size of training data which is
critical in few-shot scenarios and thus fosters effective learning for tasks
with limited annotated data. Assuming a small number ($k$) of labeled examples
for a classification task, the potential size of the fine-tuning set $T$
derived from the number of unique positive and negative contrastive pairs that
can be generated would be $\frac{k(k-1)}{2}$, which is significantly larger
than just $k$ [23]. In this stage, we sample $R$ positive and $R$ negative
triplet pairs, where $R$ is a hyperparameter (set to 20, following [23]). We
utilize multiple negatives ranking loss [44] for contrastive fine-tuning owing
to its superior performance [44] and its ability to randomly sample negative
pairs from each batch in an automated fashion.

####  4.1.3 Task-based fine-tuning using hypernetworks

To our knowledge, FM3 is the first to utilize hypernetworks in a multimodal
setting. Using frozen modality encoders has the distinct advantage of
preventing catastrophic forgetting (compared to fine-tuning the encoders
themselves) [15]. As such, we utilize an independent hypernetwork for each
modality.

In this step, we perform task-specific fine-tuning of SoTA pre-trained text
and vision models, namely a pre-trained multilingual MPNet [31] Sentence
Transformer [22] from Huggingface [45] as our text backbone and CoCa-Base [46]
as our vision backbone. We adopt the idea of hypernetworks from [37], which is
a parameter-efficient method for multitask fine-tuning. We train shared
hypernetworks to generate task-specific adapters conditioned on the task,
layer ID, and adapter position embeddings. These shared hypernetworks capture
knowledge across tasks, enabling positive transfer to low-resource and related
tasks, while task-specific layers allow the model to adapt to each individual
task. We optimize a distance function based on cosine similarity, minimizing
it for positive pairs and maximizing it for negative pairs.

####  4.1.4 Task-specific classification head training

Lastly, we train task-specific classification heads on the fine-tuned model
obtained from the above step. The generated embeddings corresponding to the
data samples for each task, along with their labels, constitute the training
set for the respective classification head. We use logistic regression for
binary classification tasks such as SST-2, QQP, QNLI, etc. and softmax for
multiclass classification tasks such as VQA, SNLI-VE, GLUE, NER, etc.

###  4.2 Tasks and Datasets

Table 1 delineates the domains, tasks, and datasets for training and
evaluating FM3.

Table 1: Datasets for training and evaluation

Domain Task Dataset Language understanding Neural entity recognition (NER)
CoNLL-2003 [47] GLUE benchmark [3] QNLI [4], MNLI [5], QQP [6], and SST-2 [7]
Vision-and-language reasoning Visual entailment SNLI-VE [1] Visual question
answering (VQA) VQAv2 dataset [2] (with questions from Visual Genome [48] as
additional data), OK-VQA [49]

##  5 Experiments

###  5.1 Experimental setup

We finetuned FM3 on the Conceptual Captions dataset (which Frozen [24] is
trained on) for vis-à-vis comparisons. We use the AdamW optimizer with global
norm clipping of 1, no weight decay for the hypernetworks and weight decay of
0.1 for the other trainable parameters. We anneal the learning rate,
increasing it linearly from 0 to $10^{-3}$ up over the first 5000 steps then
held constant for the duration of training and then decayed exponentially.
Unless specified otherwise we train our models for 500K steps. All datasets
were trained with the same weights. Since the performance of models trained
with a contrastive objective is sensitive to the batch size, we use a
relatively large batch size of 32.

###  5.2 Baselines

We utilize Flamingo [15] and _Frozen_ [24] as our primary baselines since they
deal with multimodal multitask learning in the context of FSL. We include UniT
[38] as an additional baseline since it deals with multimodal multitask
learning of tasks that Flamingo [15] and _Frozen_ haven’t been evaluated on.
For each task, we compare FM3 with both task-specific zero-/few-shot and pre-
trained/fine-tuned SoTA. Since none of the above baselines support
multilingual tasks, we utilize [39] as a baseline to qualify the performance
of FM3 on non-English tasks.

###  5.3 Results

Method | FT | Shot(s) |  OKVQA [49] |  VQAv2 [2] |  Flickr30K [50] |  Multi30K [42] |  SNLI-VE [1] |  CoNLL-2003 [47] |  QNLI [4] |  MNLI [5] |  QQP [6] |  SST-2 [7] |
---|---|---|---|---|---|---|---|---|---|---|---|---|---
| Zero/Few
---
shot SoTA
✗ | Various |  | [51]
---
43.3
(16)
| [24]
---
38.2
(4)
| [52]
---
94.9
(0)
- |  | [53]
---
87.3
(0)
| [54]
---
65.4
(5)
- |  | [53]
---
86.4
(5)
| [55]
---
67.8
(16)
| [55]
---
93.5
(16)
Flamingo-80B | ✗ | 0 | 50.6 | 56.3 | 67.2 | - | - | - | - | - | - | - |
4 | 57.4 | 63.1 | 75.1 |
32 | 57.8 | 67.6 | 75.4 |
Frozen | ✗ | 0 | 5.9 | 29.5 | - | - | - | - | - | - | - | - |
1 | 9.7 | 35.7 |
4 | 12.6 | 38.2 |
UniT | ✗ |  | Fully
---
Supervised
- | 67.0 | - | - | 73.2 | - | 88.0 | 81.8 | 90.6 | 91.5 |
FM3-5B | ✗ | 0 | 51.5 | 57.3 | 93.2 | 27.2 | 86.2 | 51.3 | 53.6 | 58.2 | 41.4 | 75.5 |
4 | 55.5 | 64.6 | 94.3 | 35.2 | 89.9 | 66.2 | 67.6 | 55.5 | 54.4 | 86.8 |
16 | 57.8 | 67.9 | 96.1 | 37.1 | 92.4 | 77.3 | 75.4 | 81.1 | 66.7 | 95.7 |
64 | 58.9 | 71.2 | 99.1 | 39.9 | 94.4 | 82.7 | 83.7 | 86.4 | 78.9 | 99.2 |
| Pre-trained
---
FT SoTA
✔ | Various |  | 54.4
---
[51]
(10K)
| 80.2
---
[56]
(444K)
| 98.8
---
[57]
(31K)
| 49.3
---
[58]
(31K)
| 91.0
---
[59]
(430K)
| 94.6
---
[60]
(430K)
| 99.2
---
[33]
(105K)
| 92.0
---
[61]
(393K)
| 89.2
---
[62]
(364K)
| 97.5
---
[61]
(67K)

Table 2:  Comparison to the state of the art. A _single_ FM3 model reaches the
state of the art on a wide array of vision-language understanding tasks with
FSL, significantly outperforming previous best zero- and few-shot methods with
as few as 16 examples. More importantly, using only $64$ examples and without
adapting any model weights, FM3 outperforms the current best methods – fine-
tuned on thousands of annotated examples – on 4 tasks. Best few-shot numbers
across all shots are in bold, best numbers across both zero/few-shot (prompt
based) and fine-tuned models are underlined. For each baseline, we chose the
best numbers across spanning all variants/experiments (unless explicitly
stated).

Table 2 performs a comparative analysis of FM3 with Flamingo, Frozen, and the
respective zero-/few-shot and fine-tuned SoTA on each task with number of
support examples/shots as $k\in\\{0,4,16,64\\}$. While Flickr30K Image-to-Text
uses Recall@1, Mulit30K en-de uses BLEU, CoNLL-2003 and QQP use F1, all other
tasks utilize accuracy as their performance metric. Note that since Frozen is
an auto-regressive model/decoder which undergoes prompt-based fine-tuning, $k$
indicates the number of support examples as part of the prompt/prefix passed
as input to the model, while FM3 being an encoder-based architecture, $k$
indicates the number of examples we contrastively fine-tune on.

Few-shot results. FM3 outperforms zero-/few-shot baselines on 7 out of the 10
benchmarks considered. This is achieved with as few as 16 examples per task,
demonstrating superior adaptation to these tasks. More importantly, FM3 is
often competitive with SoTA methods fine-tuned on up to hundreds of thousands
of annotated examples. On 4 out of 10 tasks, FM3 even outperforms the fine-
tuned SoTA despite using a single set of model weights and only 64 task-
specific examples.

Scaling with respect to parameters and shots. As table 2 indicates, more the
number of shots, better the few-shot performance, similar to GPT-3 [8]. The
performance improvement shows diminishing returns as the number of shots
increases.

###  5.4 Inference runtime analysis

Table 3 summarizes our inference runtime analysis. We measure the time taken
to run FM3 and our primary baselines on the test set of VQAv2 [2] and OKVQA
[49] and averaging it out by the total number of total samples. These
measurements are from a platform with NVIDIA A100 with 32GB VRAM. Bold numbers
indicate best performance. Underlined numbers indicate the next best baseline
on which the % improvements for FM3 are based.

|  VQAv2 (sec.) |  OKVQA (sec.)
---|---|---
FM3 |  0.187 (58% $\uparrow$) |  0.214 (46% $\uparrow$)
Flamingo | 0.353 | 0.371
Frozen | 0.295 | 0.312

Table 3: Inference runtime analysis of FM3 vs. Flamingo and Frozen on VQAv2
and OKVQA. Measurements are based on wall clock time (sec.) so lower is
better.

##  6 Ablation analysis

Table 4 summarizes the results for FM’s ablation experiments. Bold numbers
indicate best performance. Underlined numbers indicate the baseline on which
the % numbers are based. We analyze the impact of the following design
decisions on FM3’s performance:

  * •

Direct encoder fine-tuning with no hypernetworks. While frozen modality
encoders prevents catastophic forgeting [15], we adopt a hypernetwork-fee
architecture and fine-tune the encoders themselves with data from our target
tasks.

  * •

Hypernetwork size selection. We perform comparisons with varied parameter size
allocations for hypernetworks to quantify the effect of hypernetwork size. The
parameter count for hypernetworks is expressed as a percentage of the baseline
FM3 model parameter count.

  * •

Compute/memory vs. performance trade-offs. We vary the choice of text and
vision encoders (which in turn, varies the number of parameters and time
complexity of the model). For our text encoder, we choose mulitingual MiniLM
[63] with 57% lesser parameters compared to our default choice of the
paraphrase-multilingual-mpnet-base-v2 variant of multilingual MPNet [45]. For
our vision encoder, we choose the vit-base-patch16-224 variant of ViT [64]
with 78% fewer parameters compared to our default choice of CoCa-Base [46].

|  VQAv2 (acc.) |  OKVQA (acc.)
---|---|---
FM3 (default) | 71.2 | 58.9
No hypernetworks | 64.3 (90.0% $\downarrow$) | 52.1 (88.4% $\downarrow$)
Hypernetworks with 5% parameters | 69.5 (97.6% $\downarrow$) | 56.7 (88.4% $\downarrow$)
Text encoder: MiniLM | 68.1 (95.6% $\downarrow$) | 55.4 (94.0% $\downarrow$)
Vision encoder: ViT | 66.2 (92.9% $\downarrow$) | 51.2 (86.9% $\downarrow$)
Text/vision encoders: MiniLM/ViT | 65.5 (91.9% $\downarrow$) | 52.3 (88.7% $\downarrow$)

Table 4: Ablation analysis of FM3 on VQAv2 and OKVQA with number of shots as
64. Default FM3 uses hypernetworks with 10% parameters, and MPNet/CoCa as
text/vision encoders. Measurement units are % accuracy so higher is better.

##  7 Future Work

While FM3 establishes a new SoTA on several tasks, there are significant
opportunities for improvement centered around three major aspects: (i) data,
(ii) model architecture, and (iii) loss function. First, Flamingo [15]
highlights the importance of a diverse dataset amalgamated from various
disparate sources (Flamingo uses $>$2B image-text pairs vs. 3.3M that FM3 was
trained on) in training the neural network. Using the publicly available
massive LAION-400M dataset [65] would be a great starting point. Second, the
model architecture can incorporate other techniques that offer reasonably high
performance with a reduced parameter count such as low-rank based adaption
methods, for e.g., LoRA [29]. Third, following [66, 25], we can formulate the
ranking loss [44] as a binary classification problem. This has reported to
lead to an increase in performance [66, 25]. In other words, given an aligned
image-text pair, we randomly select a different image or a different caption
to form an unaligned pair. Similar to FM3’s current framework, the final
concatenated multimodal embedding can still be used as the input for
classification to predict whether the given pair is aligned or not. Finally,
FM3 is easily extendable to other languages, tasks, and modalities.

##  8 Conclusion

FM3 combines the best of both worlds of in-context learning and fine-tuning as
a front-runner in the niche domain of few-short multilingual multimodal
multitask learning. It offers a scalable architecture that can span
modalities, tasks, and languages, all while setting a new standard with SoTA
performance on a plethora of tasks and competitive performance on others. FM3
outperforms zero-/few-shot baselines on 7 out of 10 benchmarks with as few as
16 examples per task. Moreover, FM3 is competitive with fine-tuning a plethora
of task-specific SoTA models on fine-tuned on up to hundreds of thousands of
annotated examples. On 4 out of 10 tasks, FM3 even outperforms the fine-tuned
SoTA despite using a single set of model weights and only 64 task-specific
examples. Lastly, FM3 also yields a $\sim$50% latency improvement compared to
the next best FSL SoTA baseline on VQA and OKVQA datasets.

## References

  * [1] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav.  Visual entailment: A novel task for fine-grained image understanding.  arXiv preprint arXiv:1901.06706, 2019.
  * [2] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * [3] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  arXiv preprint arXiv:1804.07461, 2018.
  * [4] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  arXiv preprint arXiv:1606.05250, 2016.
  * [5] Adina Williams, Nikita Nangia, and Samuel R Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  arXiv preprint arXiv:1704.05426, 2017.
  * [6] Adina Williams, Nikita Nangia, and Samuel R Bowman.  First quora dataset release: Question pairs.  Quora blog, 2017.
  * [7] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013.
  * [8] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [10] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Metaicl: Learning to learn in context.  arXiv preprint arXiv:2110.15943, 2021.
  * [11] Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill.  Can language models learn from explanations in context?  arXiv preprint arXiv:2204.02329, 2022.
  * [12] Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev.  Internet-augmented language models through few-shot prompting for open-domain question answering.  arXiv preprint arXiv:2203.05115, 2022.
  * [13] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.  Rethinking the role of demonstrations: What makes in-context learning work?  arXiv preprint arXiv:2202.12837, 2022.
  * [14] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al.  Benchmarking generalization via in-context instructions on 1,600+ language tasks.  arXiv preprint arXiv:2204.07705, 2022.
  * [15] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  arXiv preprint arXiv:2204.14198, 2022.
  * [16] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  arXiv preprint arXiv:2205.05638, 2022.
  * [17] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  arXiv preprint arXiv:2104.08786, 2021.
  * [18] Albert Webson and Ellie Pavlick.  Do prompt-based models really understand the meaning of their prompts?  arXiv preprint arXiv:2109.01247, 2021.
  * [19] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.  Calibrate before use: Improving few-shot performance of language models.  In International Conference on Machine Learning, pages 12697–12706. PMLR, 2021.
  * [20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [21] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al.  Multitask prompted training enables zero-shot task generalization.  arXiv preprint arXiv:2110.08207, 2021.
  * [22] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019\.
  * [23] Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg.  Efficient few-shot learning without prompts.  arXiv preprint arXiv:2209.11055, 2022.
  * [24] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  Advances in Neural Information Processing Systems, 34:200–212, 2021\.
  * [25] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [26] Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain.  Omninet: A unified architecture for multi-modal multi-task learning.  arXiv preprint arXiv:1907.07804, 2019.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [28] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [29] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [30] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * [31] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  Mpnet: Masked and permuted pre-training for language understanding.  Advances in Neural Information Processing Systems, 33:16857–16867, 2020.
  * [32] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [33] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  arXiv preprint arXiv:1909.11942, 2019.
  * [34] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang.  Language-agnostic bert sentence embedding.  arXiv preprint arXiv:2007.01852, 2020.
  * [35] Julian McAuley and Jure Leskovec.  Hidden factors and hidden topics: understanding rating dimensions with review text.  In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172, 2013.
  * [36] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [37] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.  arXiv preprint arXiv:2106.04489, 2021.
  * [38] Ronghang Hu and Amanpreet Singh.  Unit: Multimodal multitask learning with a unified transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439–1449, 2021.
  * [39] Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan.  M3p: Learning universal representations via multitask multilingual multimodal pre-training.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3977–3986, 2021.
  * [40] Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che.  Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp.  arXiv preprint arXiv:2006.06402, 2020.
  * [41] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [42] Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia.  Multi30k: Multilingual english-german image descriptions.  arXiv preprint arXiv:1605.00459, 2016.
  * [43] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [44] Matthew Henderson, Rami Al-Rfou, Brian Strope, Y Sung, L Lukács, R Guo, S Kumar, B Miklos, and R Kurzweil.  Efficient natural language response suggestion for smart reply. arxiv.  Preprint posted online May, 1, 2017.
  * [45] HuggingFace.  Multilingual mpnet sentence transformer.  HuggingFace, 2022.  Available at https://huggingface.co/sentence-transformers/paraphrase-multilingual-mpnet-base-v2. Accessed Nov 13 2022.
  * [46] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [47] Erik F Sang and Fien De Meulder.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  arXiv preprint cs/0306050, 2003.
  * [48] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [49] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
  * [50] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [51] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao.  Kat: A knowledge augmented transformer for vision-and-language.  arXiv preprint arXiv:2112.08614, 2021.
  * [52] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  arXiv preprint arXiv:2208.10442, 2022.
  * [53] Tu Vu, Minh-Thang Luong, Quoc V Le, Grady Simon, and Mohit Iyyer.  Strata: Self-training with task augmentation for better few-shot learning.  arXiv preprint arXiv:2109.06270, 2021.
  * [54] Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han.  Few-shot named entity recognition: An empirical baseline study.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10408–10423, 2021.
  * [55] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen.  Differentiable prompt makes pre-trained language models better few-shot learners.  arXiv preprint arXiv:2108.13161, 2021.
  * [56] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [57] Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou.  X2-vlm: All-in-one pre-trained model for vision-language tasks.  arXiv preprint arXiv:2211.12402, 2022.
  * [58] Bin Shan, Yaqian Han, Weichong Yin, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-unix2: A unified cross-lingual cross-modal framework for understanding and generation.  arXiv preprint arXiv:2211.04861, 2022.
  * [59] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  arXiv preprint arXiv:2202.03052, 2022.
  * [60] Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu.  Automated concatenation of embeddings for structured prediction.  arXiv preprint arXiv:2010.05006, 2020.
  * [61] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21(140):1–67, 2020.
  * [62] Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma.  Entailment as few-shot learner.  arXiv preprint arXiv:2104.14690, 2021.
  * [63] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  Advances in Neural Information Processing Systems, 33:5776–5788, 2020.
  * [64] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [65] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [66] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  arXiv preprint arXiv:2001.07966, 2020.


# A Real-time Dyson Expansion Scheme: Efficient Inclusion of Dynamical
Correlations in Non-equilibrium Spectral Properties

Cian C. Reeves  Department of Physics, University of California, Santa
Barbara, Santa Barbara, CA 93117  Vojtěch Vlček  Department of Chemistry and
Biochemistry, University of California, Santa Barbara, Santa Barbara, CA 93117
Department of Materials, University of California, Santa Barbara, Santa
Barbara, CA 93117

###### Abstract

Time resolved photoemission spectroscopy is the key technique to probe the
real-time non-equilibrium dynamics of electronic states. Theoretical
predictions of the time dependent spectral function for realistic systems is
however, a challenge. Employing the Kadanoff-Baym equations to find this
quantity results in a cubic scaling in the total number of time steps, quickly
becoming prohibitive and often fail quantitatively and even qualitatively. In
comparison, mean field methods have more favorable numerical scaling both in
the number number of time steps and in the complexity associated with the cost
of a evolving for a single time step, however they miss key spectral
properties such as emergent spectral features. Here we present a scheme that
allows for the inclusion of dynamical correlations to the spectral function
while maintaining the same scaling in the number of time steps as for mean
field approaches, while capturing the emergent physics. Further, the scheme
can be efficiently implemented on top of equilibrium real-time many-body
perturbation theory schemes and codes. We see excellent agreement with exact
results for test systems. Furthermore we exemplify the method on a periodic
system and demonstrate clear evidence that our proposed scheme produces
complex spectral features including excitonic band replicas, features that are
not observed using static mean field approaches.

††preprint: APS/123-QED

## Introduction

In recent years there has been great interest in the non-equilibrium
properties of quantum materials. These systems can display phenomena such as
ultrafast phase phase transitions[1, 2], formation of exotic states[3, 4] and
allow tuning of material properties through driving[5, 6]. The first
principles description of non-equilibrium dynamics of electrons, motivated by
advances in experiments [7, 8, 9], has attracted attention in recent years
[10, 11, 12] and stimulated new theoretical developments [13, 14, 15]. In
particular the Hartree-Fock Generalized Kadanoff-Baym ansatz (HF-GKBA) emerged
as a powerful tool as it offers an efficient way of computing certain
properties non-equilibrium systems[16, 13, 10]. It has been shown in many
scenarios to provide results that are as accurate as the full solution of the
Kadanoff-Baym equations (KBEs)[17, 18, 19, 20, 21], which they approximate
without the need to compute the Green’s function propagation on the full
$t,t^{\prime}$ grid (illustrated in Fig 1)[13, 22]. Further, it does not
suffer from the spurious damping that the KBEs sometimes display[23, 24].
There is however one major caveat to these observations: these excellent
results are restricted to properties derived from the time diagonal component
of the Green’s function. Since the HF-GKBA reconstructs the time off-diagonal
components at the Hartree-Fock(HF) level, properties linked to these
components perform similarly to HF results[25]. And, as is discussed in the
supplemental information, in some cases the HF-GKBA results are even worse
than HF alone[26].

Time off-diagonal components are however, the key ingredient in computing non-
equilibrium time resolved spectral properties[27]. This function gives
information about the non-equilibrium energy distribution of particles and
holes in the system, and can be used observe phenomena such as exciton
formation and non-equilibrium phase transitions[28, 29, 2]. Furthermore, to
see the intra-gap electron-hole signatures it is necessary to include two body
correlations when computing the time off-diagonals. A feature which we will
show is missed by HF and the HF-GKBA.

Figure 1: Schematic of the off-diagonal GKBA scheme. In the two step procedure
one first performs a time evolution along the time diagonal. The information
from the diagonal is then used as a starting point to update the off-diagonal

While methods have been employed to improve the HF-GKBA, they have mainly been
applied to improve the time diagonal and steady-state properties of the HF-
GKBA[30, 31]. While there has been some application of these methods to
improve spectral properties[32], the improvements do not go beyond the mean
field level due to the underlying reconstruction of the off-diagonal. One
obvious method to try improving these spectral features is to resort to the
full KBE calculation[33, 34]. The downside of this is that the computational
scaling is cubic in the number of time-steps, making it numerically
impractical. Although methods exist to reduce the scaling of the KBEs[14, 35,
36, 37, 38], the KBEs can also suffer from artificial damping which leads to
broadening of the spectral peaks and information loss. Here we overcome the
numerical scaling issues of the KBEs by developing a new scheme based on a
time off-diagonal expansion of the Green’s function that picks up features
beyond the mean field level and demonstrates excellent agreement with exact
results. Furthermore, our approach requires only a slight modification of any
existing real-time many-body perturbation theory (MBPT) methodology [39, 40]
and can thus leverage existing schemes to reduce the computational cost of
Green’s function calculations.

In this letter we present a route to include dynamical correlations for the
time off-diagonal Green’s function without the need to explicitly perform
collision integrals. In fact, to compute the non-equilibrium spectral function
one needs to evaluate the off-diagonal components of the Green’s function
within a small neighborhood of the time-diagonal, centered around some time
$t_{p}$, as shown schematically in Fig. 1. Here, the scaling for computing the
non-equilibrium spectral function is $O(\tilde{N}_{t}^{2})$ , where
$\tilde{N}_{t}$ is the number of time-points centered around the probe time
$t_{p}$. In practice, $\tilde{N}_{t}$ is dependent on the windowing function,
$\mathcal{S}(t)$

$\mathcal{S}(t-t_{p})=\frac{1}{\delta\sqrt{2\pi}}\mathrm{e}^{-\frac{(t-t_{p})^{2}}{2\delta^{2}}}$
(1)

which is related to the probe width and the time resolved spectral
function:[27]

$\mathcal{A}(\omega,t_{p})=\int
dtdt^{\prime}\medspace\mathrm{e}^{-i\omega(t-t^{\prime})}\mathcal{S}(t-t_{p})\mathcal{S}(t^{\prime}-t_{p})\textrm{Tr}[G(t,t^{\prime})].\\\
$ (2)

This windowing function determines the energy and temporal resolution of the
non-equilibrium spectral function[41]. In practice $\tilde{N}_{t}\ll N_{t}$,
leading to huge computational savings, which is constant and determined solely
by the windowing function and the overall computational scaling is thus
determined solely by the time-diagonal propagation.

## Real-time Dyson expansion scheme

A simple proof included in the supplemental information shows the equivalence
between the following two equations[26].

$\begin{split}&G^{\mathrm{R}}(12)=G^{\mathrm{R,MF}}(12)\\\ &\hskip
42.67912pt+\int\int d3d4\medspace
G^{\mathrm{R,MF}}(13)\Sigma^{\mathrm{R}}[G^{\mathrm{MF}}(34)]G^{\mathrm{R}}(42)\\\
&\equiv\left[i\partial_{t_{1}}-h^{\mathrm{MF}}[G^{\mathrm{MF}}(1)]\right]G^{\mathrm{R}}(12)\\\
&\hskip 85.35826pt-\int
d3\medspace\Sigma^{\mathrm{R}}[G^{\mathrm{MF}}(13)]G^{\mathrm{R}}(32)=\delta(12)\end{split}$
(3)

The first equation, gives the Dyson equation describing a one-shot correction
on-top of a reference (mean-field Hamiltonian) for an arbitrary self-energy,
$\Sigma(t,t^{\prime})$[42]. The second is exactly the equation we will solve
in this paper using a real-time Green’s function formalism. We note that at
equilibrium our approach reduces to the standard MBPT correction on top of a
mean field reference Hamiltonian. In practical calculations, such an approach
typically leads to great improvement and even yields spectra containing
signatures of strong dynamical correlations[43, 44, 40]. This equivalence
builds an intuition behind the procedure: we use a mean-field method as an
initial guess and we add a correction term that includes dynamical correlation
effects through $\Sigma(t,t^{\prime})$. Our approach thus represents a natural
extension of computational techniques applied in MBPT to non-equilibrium
settings.

Using this form of the integro-differential equation for
$G^{\mathrm{R}}(t,t^{\prime})$ and the equations of motion for
$G^{\mathrm{HF}}(t,t^{\prime})$ we derive an ODE scheme that involves
propagating $G^{\mathrm{R}}(t,t^{\prime})$ along with an auxiliary two-body
propagator used to reconstruct the collision integral at each time step. A
similar approach has also been applied to the time diagonal Green’s function
in a linear scaling HF-GKBA scheme[13, 22]. Propagating the ODE scheme on the
time off-diagonal for some $\tilde{N_{t}}$ steps scales as $O(\tilde{N}_{t})$.
Thus reconstructing the off-diagonal from $\tilde{N}_{t}$ steps along the
diagonal will indeed scale as $O(\tilde{N}^{2}_{t})$. Furthermore, the time
evolution is independent for each starting point, making this scheme extremely
easy to parallelize in time coordinates.

In the following, we only show the equations for
$G^{\mathrm{R}}(t,t^{\prime})$ however we perform the same procedure for
$G^{<}(t,t^{\prime})$. The details on this can be found in the supplemental
information including the relevant equations and the derivations for both
schemes[26]. From now on we will refer to this scheme as the real-time Dyson
expansion (RT-DE).

The scheme involves solving the following equation for
$G^{\mathrm{R}}(t,t^{\prime})$ along with a second ODE for a two-body
propagator $\mathcal{F}_{ijkl}(t,t^{\prime})$.

$\begin{split}\frac{\mathrm{d}G^{\mathrm{R}}(t,t^{\prime})}{\mathrm{d}t}&=-i\left[h^{\mathrm{MF}}(t)G^{\mathrm{R}}(t,t^{\prime})+I^{\mathrm{R}}(t,t^{\prime})\right],\\\
I^{\mathrm{R}}_{im}(t,t^{\prime})&=-\sum_{klp}w_{iklp}(t)\mathcal{F}_{lpmk}(t,t^{\prime}).\\\
\end{split}$ (4)

Here, the retarded collision integral depends on the interaction tensor and a
two-body propagator $\mathcal{F}_{ijkl}$. By employing the approximation in
equation (3) we derive an ODE for $\mathcal{F}_{ijkl}$ given by

$\begin{split}\frac{\mathrm{d}\mathcal{F}_{lpmk}(t,t^{\prime})}{\mathrm{d}t}&=\sum_{qrsj}w_{qrsj}(t)\Big{[}G^{>}_{lq}(t)G^{<}_{pr}(t)G^{>}_{sk}(t)-G^{<}_{lq}(t)G^{<}_{pr}(t)G^{>}_{sk}(t)\Big{]}G^{\mathrm{R}}_{jm}(t,t^{\prime})-\\\
&\hskip
142.26378pti\sum_{x}\Big{[}h^{\mathrm{MF}}_{lx}(t)\mathcal{F}_{xpmk}(t,t^{\prime})+h^{\mathrm{MF}}_{px}(t)\mathcal{F}_{lxmk}(t,t^{\prime})-\mathcal{F}_{lpmx}(t,t^{\prime})h^{\mathrm{MF}}_{xk}(t)\Big{]}.\\\
\end{split}$ (5)

To fully demonstrate the the quality of this “one-shot correction” at
equilibrium, we provide its numerical analysis in the supplemental
information[26]. In this setup, the time diagonal terms of RT-DE represent a
stationary solution of a mean-field Hamiltonian, while the time off-diagonals
contain the information about the self-energy in the time domain. Hence, RT-DE
is equivalent to the perturbative correction and we show that it is in
excellent agreement with exact diagonalization. In contrast, the HF-GKBA
spectral function is reconstructed by a static mean-field approach and, hence,
practically identical to the results of the underlying mean-field Hamiltonian.

In the non-equilibrium setup the distinction between various methods becomes
even more pronounced. Again, RT-DE requires a time evolution of $G^{<}(t,t)$
using a time-dependent mean-field method. Here, for simplicity, we use TD-HF
on the time diagonal as a prototypical example. Note, however, that this
scheme is applicable to any particular choice of the reference static
Hamiltonian, including those beyond mean-field, from some form of a
downfolding procedure. For completeness we also show in [26] that the
difference between exact diagonalization and TD-HF is negligible on the
diagonal. The time diagonal components are then inserted in equation (5).
Starting from some $t^{\prime}=t$ we can time-step in the $t$ variable in the
range $t^{\prime}<t<T_{\mathrm{max}}$. Here we again emphasize an important
advantage of the RT-DE scheme. Since we can start from any point on the
diagonal we choose to only compute those $\tilde{N}_{t}^{2}$ off-diagonal
terms within the probe window.

Finally, we note that the time evolution require the following initial
conditions (for each $t^{\prime}$),
$G^{\mathrm{R}}(t^{\prime},t^{\prime})=-i$,
$\mathcal{F}(t^{\prime},t^{\prime})=0$. For $G^{\mathrm{R}}(t,t^{\prime})$ the
initial condition is exact and exactly represents the one-shot correction.
However for the lesser component we need to select a starting point for
$I^{<}(t,t)$ and $G^{<}(t,t)$ which, in principle, breaks the equivalence to
the standard perturbative expansion. This matter is discussed in more detail
in the supplemental information. For the results shown here we have taken
$I^{<}(t,t)=0$ and $G^{<}(t,t)=G^{<,\mathrm{MF}}(t)$.

## Application to non-equilibrium problems

We will now proceed by studying the non-equilibrium spectral function in
equation (2) with our proposed scheme.

$\begin{split}\mathcal{H}&=-J\sum_{\langle
i,j\rangle}c^{\dagger}_{i}c_{j}+U\sum_{i}n_{i\uparrow}n_{i\downarrow}+h^{\mathrm{N.E}}(t)\\\
\end{split}$ (6)

First we apply the RT-DE to non-equilibrium system and compare to TD-HF and
exact diagonalization results. We excite the system given by equation (6) with
a short wavelength pulse given by
$h_{ij}^{\mathrm{N.E}}(t)=\delta_{ij}E\cos\left(\frac{\pi
r_{i}}{2}\right)\exp\left({-\frac{(t-t_{0})^{2}}{2T_{p}^{2}}}\right)$[17],
where $r_{i}=\frac{N_{s}-1}{2}-i$ is the lattice vector for site $i$ and
$N_{s}$ is the number of lattice sites.

Figure 2: Time resolved emission spectrum a) and full time dependent spectrum
b) for TD-HF, exact diagonalization and the RT-DE. Calculated in the model
given in equation 6 with the following parameters: $U=1.0J$, $N_{s}=4$,
$E=0.5J$, $t_{0}=5J^{-1}$ and $T_{p}=0.5J^{-1}$ after time evolving to
$T_{\mathrm{max}}=200J^{-1}$. The probe width is taken to be $\delta=10$ and
$t_{p}=100J^{-1}$.

First we will examine Fig. 2 a). The RT-DE is in excellent agreement with the
exact result for the lesser component of the spectral function. It picks up
features around $\omega\approx-1.2J$ and $\omega\approx-0.4J$ that are
completely missed by TD-HF. For the peaks above the Fermi energy, both the RT-
DE and TD-HF pick up the peaks. Both methods slightly underestimate the energy
of the peak around $\omega\approx 1.0J$ by the same amount. Interestingly, the
peak near $\omega\approx 2.0J$ is capture slightly better with TD-HF than with
the RT-DE. This is related to the initial condition for the lesser component.
Given that the initial condition for $G^{<}(t,t^{\prime})$ can be fixed in
different ways, further optimization and constraining the particular choices
will be explored elsewhere. Yet, we emphasize that this boundary condition
affects only the details of the electronic distribution in dynamically
renormalized states. The band renormalization and emergence of new spectral
peaks is however captured perfectly using the knowledge of the $G^{R}$ alone,
which is free of any uncertainty with fixing its boundary conditions.

For the retarded component in Fig. 2 b), we see a similarly excellent of
agreement between the exact result and the RT-DE. The RT-DE matches the non-
equilibrium spectrum very closely, again it picks up peaks at
$\omega\approx\pm 0.4J$ and $\omega\approx\pm 1.2J$ that are completely missed
by HF and is in perfect agreement with exact results for the height of the
peaks at $\omega\approx\pm 1.9J$. The mean-field method severely
underestimates this spectral feature since it does not include dynamical
effects that dominate the peak magnitude.

In our second example we explore a realistic model of an insulator out of
equilibrium. We study the dynamics of a long-range interacting two band
Hubbard model with periodic boundary conditions at half filling under a strong
optical excitation. The full Hamiltonian is given by

$\begin{split}\mathcal{H}&=\sum_{\alpha,\beta\in\\{c,v\\}}\sum_{i,j,\sigma}h^{\alpha\beta}_{ij}(t)c^{\alpha\dagger}_{i,\sigma}c^{\beta}_{j,\sigma}+U\sum_{i}n^{\alpha}_{i\uparrow}n^{\alpha}_{i\uparrow}\\\
&\hskip 71.13188pt+U\sum_{i}n_{ic}n_{iv}+\gamma
U\sum_{i<j\atop\alpha\leq\beta}\frac{n_{i\alpha}n_{j\beta}}{|\vec{r}_{i}-\vec{r}_{j}|}.\\\
h^{\alpha\beta}_{ij}(t)&=J\delta_{\alpha\beta}\delta_{\langle
i,j\rangle}(\delta_{\alpha v}-\delta_{\alpha
c})+\epsilon_{\alpha}\delta_{\alpha\beta}\\\ &\hskip
42.67912pt+\delta_{ij}(1-\delta_{\alpha\beta})E\cos(\omega_{p}(t-t_{0}))\mathrm{e}^{-\frac{(t-t_{0})^{2}}{2T_{p}^{2}}}\end{split}$
(7)

Figure 3: Non-equilibrium time resolved emission spectrum for the model given
in equation (7) with 12 sites evolved until time $T_{\mathrm{max}}=100J^{-1}$.
Panels a), b) and c) show the spectra computed with TD-HF, the HF-GKBA and the
RT-DE respectively. The model parameters are $U=4.0J$, $\gamma=0.5$,
$\epsilon_{v}=-30J$, $\epsilon_{c}=-20J$, $E=2.5J$, $\omega_{p}=9.0J$,
$t_{0}=10J^{-1}$, $T_{p}=1.8J^{-1}$. The probe width used in (2) is $\delta=5$
and the probe is centered at $t_{p}=90J^{-1}$. A prominent excitonic feature
is seen for the RT-DE results just below the renormalized conduction band;
further, a visible band satellite is visible at the bottom of the
bandstructure.

The results are shown in Fig. 1. We compare TD-HF, the HF-GKBA and the RT-DE
using the following model parameters: $U=4.0J$, $\epsilon_{v}=-30J$,
$\epsilon_{c}=-20J$, $\gamma=0.5,E=2.5J$, $\omega_{p}=7.5J$, $t_{0}=10J^{-1}$
and $T_{p}=1.8J^{-1}$. The excitation frequency (pump), $\omega_{p}$ is below
the equilibrium gap ($\sim 10.0J$) but it is chosen such that excitons can
form in the system. We also remark that at equilibrium the three methods
produce practically identical band structures, shown as red dashed lines in
Fig. 3. This allows us to see how each method performs, purely from changes
due to the system leaving equilibrium.

Fig. 3 a) shows the time resolved emission spectrum for TD-HF. The full non-
equilibrium band structure is shown in blue dash-dotted lines. We note that
due to the lack of dynamical self-energy (yielding possibly multiple
quasiparticle solutions) no new (satellite) states can appear in the TD-HF
result. The figure shows electrons are excited across the gap and into the
conduction band around $k=0$. This change in the electon distribution leads to
a renormalization of the band gap, which is governed by the changes in the
orbital occupancies and related to changes in the time dependent density
matrix. In practice, occupied states are shifted down in energy creating a
relatively sharp discontinuity in the upper band which becomes partially
filled around $k=0$. The concentration of electrons (holes) at the $k=0$ point
of the conduction (valence) band and the mean field nature of TD-HF means the
renormalization mainly effects the $k=0$ point, leading to a significant
stretching of the bands. As expected the lack of two-body correlations in TD-
HF means we observe no novel features in the spectral function which could be
interpreted as emerging excitonic states beyond the band stretching.

In Fig. 3 b) we again show the time resolved emission spectrum, now computed
using the HF-GKBA. This result shares features common to TD-HF as we expect
because it is dominated by the off-diagonal terms dominated by HF. Firstly,
while the HF signatures are diminished, we see a similar stretching of the
bands relative to their equilibrium value. One reason for this change comes
from an inconsistency between the treatment of the time diagonal and off-
diagonal terms in HF-GKBA. Specifically it arises from the GKBA density
matrix, $\rho^{\mathrm{GKBA}}(t)$, entering the HF equation for the off-
diagonal instead of $\rho^{\mathrm{HF}}(t)$. When solving the TD-HF equation
the density matrix will be projected on to a set of excited states of the HF
Hamiltonian. Since in general at time $t$,
$\rho^{\mathrm{GKBA}}(t)\neq\rho^{\mathrm{HF}}(t)$ the set of excited states
we project onto will be different thus leading to a distinct energy spectrum.
Such an inconsistency makes it a priori difficult to interpret HF-GKBA
spectral results or to say how the difference from TD-HF will manifest.
Another important observation is the lack of additional excitonic features.
The picture is thus consistent with TD-HF (albeit with the magnitude of the
dynamical band renormalization being significantly distinct). This is expected
as the spectrum is dominated by the time off-diagonal terms treated at the HF
level.

Finally we turn to Fig. 3 c), where we have computed the time resolved
emission spectrum using the RT-DE scheme. The results differ markedly and show
emergent features consistent with non-equilibrium dynamics. For the main
renormalized bands, shown in blue dash-dotted lines, we see a picture somewhat
similar to the HF-GKBA. Namely there is a renormalization of the band
positions compared to equilibrium. Interestingly, for the RT-DE the
renormalization effects both the $k\neq 0$ and $k=0$ points very similarly
resulting in almost a rigid shift of the bands, compared to the stretching
seen in Fig 3 a) and b). This is consistent with renormalized (increased)
screening in the photoexcited system which affects the entire band structure
and not only the small number states that become occupied by photo-excitation.

In addition, there are two more striking differences between the RT-DE and
both TD-HF and the HF-GKBA. The first is a peak appearing just below the
renormalized conduction band, consistent with an in-gap electronic state,
i.e., an exciton. Secondly we see the appearance of a satellite band appearing
just below the position of the equilibrium valence band. Similarly to the
finite system in Fig. 2 the inclusion of dynamical correlations through the
retarded collision integral allows for the formation of additional spectral
features. This is consistent with satellites emerging due to coupling to a
bosonic excitation in the system. Here, the satellite is displaced by energy
equivalent to the distance between the excitonic feature and the minimum of
the renormalized conduction state. The satellites thus can be interpreted as
arising due to the simultaneous QP formation and exicton dissociation, i.e.,
corresponding to an “excitonic replica” of the valence band. The exact
interpretation of such features are left for future research, but we note that
these are consistent with the exact solutions made for finite systems studied
above.

## Conclusions and outlook

For our tests in finite systems the results for the RT-DE in and out of
equilibrium are in excellent agreement with exact results. Not only does the
RT-DE correct features picked up by TD-HF and the HF-GKBA but it also captures
satellite features that both other methods miss entirely. For our larger
system, the RT-DE results provide physically meaningful results. Importantly
by including two-body correlations through the self-energy we are able to
observe an emergent excitonic peak appearing in the spectral function and
energetically well separated from the quasiparticle band.

In conclusion the RT-DE is an efficient scheme for studying the non-
equilibrium spectral properties of quantum systems, offering accuracy and a
huge reduction in cost relative to the full Kadanoff-Baym equations.
Application of this scheme to a more realistic system and comparison of the
results to experimental measurements as a validation is underway. Furthermore,
deeper investigation into the new spectral features found for periodic system
presented here is warranted. Finally, we note that RT-DE can be trivially
combined with existing real-time MBPT methods and their extension to more
advanced self-energy approximations, such as $GW\Gamma$[43].

## Acknowledgements

This material is based upon work supported by the U.S. Department of Energy,
Office of Science, Office of Advanced Scientific Computing Research and Office
of Basic Energy Sciences, Scientific Discovery through Advanced Computing
(SciDAC) program under Award Number DE-SC0022198. This research used resources
of the National Energy Research Scientific Computing Center, a DOE Office of
Science User Facility supported by the Office of Science of the U.S.
Department of Energy under Contract No. DE-AC02-05CH11231 using NERSC award
BES-ERCAP0029462

## References

  * Beebe  _et al._ [2017] M. R. Beebe, J. M. Klopf, Y. Wang, S. Kittiwatanakul, J. Lu, S. A. Wolf, and R. A. Lukaszew, Time-resolved light-induced insulator-metal transition in niobium dioxide and vanadium dioxide thin films, Opt. Mater. Express 7, 213 (2017).
  * Disa  _et al._ [2023] A. S. Disa, J. Curtis, M. Fechner, A. Liu, A. von Hoegen, M. Först, T. F. Nova, P. Narang, A. Maljuk, A. V. Boris, B. Keimer, and A. Cavalleri, Photo-induced high-temperature ferromagnetism in YTiO3, Nature 617, 73 (2023).
  * Dong  _et al._ [2021] S. Dong, M. Puppin, T. Pincelli, S. Beaulieu, D. Christiansen, H. Hübener, C. W. Nicholson, R. P. Xian, M. Dendzik, Y. Deng, Y. W. Windsor, M. Selig, E. Malic, A. Rubio, A. Knorr, M. Wolf, L. Rettig, and R. Ernstorfer, Direct measurement of key exciton properties: Energy, dynamics, and spatial distribution of the wave function, Natural Sciences 1, e10010 (2021).
  * Bao  _et al._ [2022] C. Bao, P. Tang, D. Sun, and S. Zhou, Light-induced emergent phenomena in 2D materials and topological materials, Nature Reviews Physics 4, 33 (2022).
  * Nuske  _et al._ [2020] M. Nuske, L. Broers, B. Schulte, G. Jotzu, S. A. Sato, A. Cavalleri, A. Rubio, J. W. McIver, and L. Mathey, Floquet dynamics in light-driven solids, Phys. Rev. Res. 2, 043408 (2020).
  * Zhou  _et al._ [2023] S. Zhou, C. Bao, B. Fan, F. Wang, H. Zhong, H. Zhang, P. Tang, W. Duan, and S. Zhou, Floquet engineering of black phosphorus upon below-gap pumping, Phys. Rev. Lett. 131, 116401 (2023).
  * Sie  _et al._ [2019] E. J. Sie, T. Rohwer, C. Lee, and N. Gedik, Time-resolved XUV ARPES with tunable 24–33 eV laser pulses at 30 meV resolution, Nature Communications 10, 3535 (2019).
  * Karni  _et al._ [2023] O. Karni, I. Esin, and K. M. Dani, Through the Lens of a Momentum Microscope: Viewing Light-Induced Quantum Phenomena in 2D Materials, Advanced Materials 35, 2204120 (2023).
  * Man  _et al._ [2021] M. K. L. Man, J. Madéo, C. Sahoo, K. Xie, M. Campbell, V. Pareek, A. Karmakar, E. L. Wong, A. Al-Mahboob, N. S. Chan, D. R. Bacon, X. Zhu, M. M. M. Abdelrasoul, X. Li, T. F. Heinz, F. H. da Jornada, T. Cao, and K. M. Dani, Experimental measurement of the intrinsic excitonic wave function, Science Advances 7, eabg0192 (2021).
  * Perfetto  _et al._ [2022] E. Perfetto, Y. Pavlyukh, and G. Stefanucci, Real-Time $GW$: Toward an Ab Initio Description of the Ultrafast Carrier and Exciton Dynamics in Two-Dimensional Materials, Phys. Rev. Lett. 128, 016801 (2022).
  * Sun  _et al._ [2021] J. Sun, C.-W. Lee, A. Kononov, A. Schleife, and C. A. Ullrich, Real-Time Exciton Dynamics with Time-Dependent Density-Functional Theory, Phys. Rev. Lett. 127, 077401 (2021).
  * Karlsson  _et al._ [2021] D. Karlsson, R. van Leeuwen, Y. Pavlyukh, E. Perfetto, and G. Stefanucci, Fast Green’s Function Method for Ultrafast Electron-Boson Dynamics, Phys. Rev. Lett. 127, 036402 (2021).
  * Schlünzen  _et al._ [2020] N. Schlünzen, J.-P. Joost, and M. Bonitz, Achieving the Scaling Limit for Nonequilibrium Green Functions Simulations, Phys. Rev. Lett. 124, 076601 (2020).
  * Kaye and Golež [2021] J. Kaye and D. Golež, Low rank compression in the numerical solution of the nonequilibrium Dyson equation, SciPost Phys. 10, 091 (2021).
  * Lian  _et al._ [2018] C. Lian, M. Guan, S. Hu, J. Zhang, and S. Meng, Photoexcitation in Solids: First-Principles Quantum Simulations by Real-Time TDDFT, Advanced Theory and Simulations 1, 1800055 (2018).
  * Lipavský  _et al._ [1986] P. Lipavský, V. Špička, and B. Velický, Generalized Kadanoff-Baym ansatz for deriving quantum transport equations, Phys. Rev. B 34, 6933 (1986).
  * Reeves  _et al._ [2023] C. C. Reeves, Y. Zhu, C. Yang, and V. c. v. Vlček, Unimportance of memory for the time nonlocal components of the Kadanoff-Baym equations, Phys. Rev. B 108, 115152 (2023).
  * Schlünzen  _et al._ [2017] N. Schlünzen, J.-P. Joost, F. Heidrich-Meisner, and M. Bonitz, Nonequilibrium dynamics in the one-dimensional Fermi-Hubbard model: Comparison of the nonequilibrium Green-functions approach and the density matrix renormalization group method, Phys. Rev. B 95, 165139 (2017).
  * Bonitz  _et al._ [2013] M. Bonitz, S. Hermanns, and K. Balzer, Dynamics of Hubbard Nano-Clusters Following Strong Excitation, Contributions to Plasma Physics 53, 778 (2013).
  * Balzer  _et al._ [2013] K. Balzer, S. Hermanns, and M. Bonitz, The generalized Kadanoff-Baym ansatz. Computing nonlinear response properties of finite systems, Journal of Physics: Conference Series 427, 012006 (2013).
  * Hermanns  _et al._ [2013] S. Hermanns, K. Balzer, and M. Bonitz, Few-particle quantum dynamics–comparing nonequilibrium Green functions with the generalized Kadanoff–Baym ansatz to density operator theory, Journal of Physics: Conference Series 427, 012008 (2013).
  * Joost  _et al._ [2020] J.-P. Joost, N. Schlünzen, and M. Bonitz, G1-G2 scheme: Dramatic acceleration of nonequilibrium Green functions simulations within the Hartree-Fock generalized Kadanoff-Baym ansatz, Phys. Rev. B 101, 245101 (2020).
  * von Friesen  _et al._ [2009] M. P. von Friesen, C. Verdozzi, and C.-O. Almbladh, Successes and Failures of Kadanoff-Baym Dynamics in Hubbard Nanoclusters, Phys. Rev. Lett. 103, 176404 (2009).
  * von Friesen  _et al._ [2010] M. P. von Friesen, C. Verdozzi, and C.-O. Almbladh, Artificial damping in the Kadanoff-Baym dynamics of small Hubbard chains, Journal of Physics: Conference Series 220, 012016 (2010).
  * Tuovinen  _et al._ [2020] R. Tuovinen, D. Golež, M. Eckstein, and M. A. Sentef, Comparing the generalized Kadanoff-Baym ansatz with the full Kadanoff-Baym equations for an excitonic insulator out of equilibrium, Phys. Rev. B 102, 115157 (2020).
  * [26] See Supplemental Material at  for a selection of other results and discussions including, demonstrating the rt-DE scheme for ground state properties, derivations of the rt-DE equations of motion for $G^{\mathrm{R}}(t,t^{\prime})$ and $G^{<}(t,t^{\prime})$, a derivation showing the equivalence of the rt-DE to a one-shot correction on top of a mean-field reference, calculations showing TD-HF compared to exact results on the diagonal and implementation details.
  * Freericks  _et al._ [2009] J. K. Freericks, H. R. Krishnamurthy, and T. Pruschke, Theoretical Description of Time-Resolved Photoemission Spectroscopy: Application to Pump-Probe Experiments, Phys. Rev. Lett. 102, 136401 (2009).
  * Boschini  _et al._ [2024] F. Boschini, M. Zonno, and A. Damascelli, Time-resolved ARPES studies of quantum materials, Rev. Mod. Phys. 96, 015003 (2024).
  * Choi  _et al._ [2023] D. Choi, C. Yue, D. Azoury, Z. Porter, J. Chen, F. Petocchi, E. Baldini, B. Lv, M. Mogi, Y. Su, S. D. Wilson, M. Eckstein, P. Werner, and N. Gedik, Light-induced insulator-metal transition in Sr2IrO4 reveals the nature of the insulating ground state (2023), arXiv:2305.07619 [cond-mat.str-el] .
  * Latini  _et al._ [2014] S. Latini, E. Perfetto, A.-M. Uimonen, R. van Leeuwen, and G. Stefanucci, Charge dynamics in molecular junctions: Nonequilibrium Green’s function approach made fast, Phys. Rev. B 89, 075306 (2014).
  * Hopjan and Verdozzi [2019] M. Hopjan and C. Verdozzi, Initial correlated states for the generalized Kadanoff–Baym Ansatz without adiabatic switching-on of interactions in closed systems, The European Physical Journal Special Topics 227, 1939–1948 (2019).
  * Schüler  _et al._ [2020] M. Schüler, U. De Giovannini, H. Hübener, A. Rubio, M. A. Sentef, T. P. Devereaux, and P. Werner, How Circular Dichroism in Time- and Angle-Resolved Photoemission Can Be Used to Spectroscopically Detect Transient Topological States in Graphene, Phys. Rev. X 10, 041013 (2020).
  * Stan  _et al._ [2009] A. Stan, N. Dahlen, and R. van Leeuwen, Time propagation of the Kadanoff–Baym equations for inhomogeneous systems, J. Chem. Phys. 130, 224101 (2009).
  * Stefanucci and van Leeuwen [2013] G. Stefanucci and R. van Leeuwen,  _Nonequilibrium Many-Body Theory of Quantum Systems: A Modern Introduction_ (Cambridge University Press, 2013).
  * Yin  _et al._ [2022] J. Yin, Y. hao Chan, F. H. da Jornada, D. Y. Qiu, S. G. Louie, and C. Yang, Using dynamic mode decomposition to predict the dynamics of a two-time non-equilibrium Green’s function, Journal of Computational Science 64, 101843 (2022).
  * Kaye and U. R. Strand [2023] J. Kaye and H. U. R. Strand, A fast time domain solver for the equilibrium Dyson equation, Advances in Computational Mathematics 49, 10.1007/s10444-023-10067-7 (2023).
  * Stahl  _et al._ [2022] C. Stahl, N. Dasari, J. Li, A. Picano, P. Werner, and M. Eckstein, Memory truncated Kadanoff-Baym equations, Phys. Rev. B 105, 115146 (2022).
  * Blommel  _et al._ [2024] T. Blommel, J. Kaye, Y. Murakami, E. Gull, and D. Golež, Chirped amplitude mode in photo-excited superconductors (2024), arXiv:2403.01589 [cond-mat.supr-con] .
  * Vlček  _et al._ [2018] V. Vlček, W. Li, R. Baer, E. Rabani, and D. Neuhauser, Swift $GW$ beyond 10,000 electrons using sparse stochastic compression, Phys. Rev. B 98, 075107 (2018).
  * Vlček [2019] V. Vlček, Stochastic Vertex Corrections: Linear Scaling Methods for Accurate Quasiparticle Energies, Journal of Chemical Theory and Computation 15, 6254 (2019), pMID: 31557012.
  * Randi  _et al._ [2017] F. Randi, D. Fausti, and M. Eckstein, Bypassing the energy-time uncertainty in time-resolved photoemission, Phys. Rev. B 95, 115132 (2017).
  * Vacondio  _et al._ [2022] S. Vacondio, D. Varsano, A. Ruini, and A. Ferretti, Numerically Precise Benchmark of Many-Body Self-Energies on Spherical Atoms, Journal of Chemical Theory and Computation 18, 3703 (2022).
  * Mejuto-Zaera and Vlček [2022] C. Mejuto-Zaera and V. Vlček, Self-consistency in $GW\mathrm{\Gamma}$ formalism leading to quasiparticle-quasiparticle couplings, Phys. Rev. B 106, 165129 (2022).
  * Mejuto-Zaera  _et al._ [2021] C. Mejuto-Zaera, G. Weng, M. Romanova, S. J. Cotton, K. B. Whaley, N. M. Tubman, and V. Vlček, Are multi-quasiparticle interactions important in molecular ionization?, The Journal of Chemical Physics 154, 121101 (2021).