extreme scenario we have that all noise terms are equal, that is
$\varepsilon_{k}=\varepsilon$ with $\varepsilon$ constant for every $k$.
Therefore, the counterfactual outcomes are constant over time

$Y^{\bar{a}_{k}=\bar{x}_{k}}_{i,k}\overset{\eqref{eq:carryov1}}{=}Y^{a_{k}=x_{k}}_{i,k}\overset{\eqref{eq:Y1}}{=}f(x_{k},u_{i},\varepsilon)=Y^{a_{k_{*}}=x_{k}}_{i,k_{*}}$

with $x_{k}$ last element of $\bar{x}_{k}$ and $k_{*}$ being a generic time
point.

It follows also that

$\mathbb{E}(Y^{\bar{a}_{k}=\bar{x}_{k}}_{k}|U=u_{i})\overset{\eqref{eq:Y1}}{=}\mathbb{E}(f(x_{k},u_{i},\varepsilon))=f(x_{k},u_{i},\varepsilon)=Y^{\bar{a}_{k}=\bar{x}_{k}}_{i,k}.$

Therefore the individual-specific causal effect and the individual causal
effect coincide in this scenario, $U\text{-}CATE_{k}(u_{i})=ICE_{i,k}$.

Moreover, the mean of the observed outcomes for a treatment level $x$ in a
fixed context $U=u_{i},\textbf{Z}=\textbf{z}_{i}$

$\begin{split}&\frac{1}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=x)\\}}\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=x)f^{Y}(x,u_{i},\varepsilon_{i,k})\\}\\\
&=\frac{1}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=x)\\}}\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=x)f^{Y}(x,u_{i},\varepsilon)\\}\\\
&=\frac{1}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=x)\\}}f^{Y}(x,u_{i},\varepsilon)\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=x)\\}\\\
&=f^{Y}(x,u_{i},\varepsilon)\overset{\eqref{eq:carryov1}}{=}Y^{\bar{a}_{k}=\bar{x}_{k}}_{i,k},\end{split}$

with $\bar{x}_{k}$ vector of size $k$ with elements all equal to $x$. We have,
therefore, that $\hat{\tau_{i}}=ICE_{i,k}$.

## Appendix D

### G-formula

Here, we outline the derivation of the g-formula reported in Robins et al.
1999. This encompasses a more general case of the one we consider.

$\begin{split}&\mathbb{E}(Y^{\bar{a}_{k}=\bar{x}_{k}}_{k}|U=u)\\\
&=\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y^{\bar{a}_{k}=\bar{x}_{k}}_{k}|\bar{L}^{\bar{a}_{k}=\bar{x}_{k}}_{k}=\bar{l}_{k},\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(\bar{L}^{\bar{a}_{k}=\bar{x}_{k}}_{k}=\bar{l}_{k},\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1}|U=u)\\\
&\overset{\mathrm{(Y^{\bar{a}_{k}}_{k}\perp\mkern-9.5mu\perp\bar{A}_{k}|\bar{L}^{\bar{a}_{k}}_{k},\bar{Y}^{\bar{a}_{k-1}}_{k-1},U)}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y^{\bar{a}_{k}=\bar{x}_{k}}_{k}|\bar{L}^{\bar{a}_{k}=\bar{x}_{k}}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(\bar{L}^{\bar{a}_{k}=\bar{x}_{k}}_{k}=\bar{l}_{k},\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1}|U=u)\\\
&\overset{\eqref{eq:consistency2}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(\bar{L}^{\bar{a}_{k}=\bar{x}_{k}}_{k}=\bar{l}_{k},\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1}|U=u)\\\
&=\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L^{\bar{a}_{k}=\bar{x}_{k}}_{k}=l_{k}|\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1},\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1},\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1}|U=u)\\\
&\overset{\mathrm{(L^{\bar{a}_{k}}_{k}\perp\mkern-9.5mu\perp\bar{A}_{k}|\bar{Y}^{\bar{a}_{k-1}}_{k-1},\bar{L}^{\bar{a}_{k-1}}_{k-1},U)}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L^{\bar{a}_{k}=\bar{x}_{k}}_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1},\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1},\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1}|U=u)\\\
&\overset{\eqref{eq:consistency2}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(\bar{Y}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{y}_{k-1},\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1}|U=u)\\\
&=\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(Y^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=y_{k-1}|\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1},\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2},U=u)\\\
&\mathbb{P}(\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1},\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2}|U=u)\end{split}$

$\begin{split}&\overset{\mathrm{(Y^{\bar{a}_{k}}_{k}\perp\mkern-9.5mu\perp\bar{A}_{k}|\bar{L}^{\bar{a}_{k}}_{k},\bar{Y}^{\bar{a}_{k-1}}_{k-1},U)}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(Y^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=y_{k-1}|\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1},\bar{A}_{k-1}=\bar{x}_{k-1},\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2},U=u)\\\
&\mathbb{P}(\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1},\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2}|U=u)\\\
&\overset{\eqref{eq:consistency2}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(Y_{k-1}=y_{k-1}|\bar{L}_{k-1}=\bar{l}_{k-1},\bar{A}_{k-1}=\bar{x}_{k-1},\bar{Y}_{k-2}=\bar{y}_{k-2},U=u)\\\
&\mathbb{P}(\bar{L}^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=\bar{l}_{k-1},\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2}|U=u)\\\
&=\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(Y_{k-1}=y_{k-1}|\bar{L}_{k-1}=\bar{l}_{k-1},\bar{A}_{k-1}=\bar{x}_{k-1},\bar{Y}_{k-2}=\bar{y}_{k-2},U=u)\\\
&\mathbb{P}(L^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=l_{k-1}|\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2},\bar{L}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{l}_{k-2},U=u)\\\
&\mathbb{P}(\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2},\bar{L}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{l}_{k-2}|U=u)\\\
&\overset{\mathrm{(L^{\bar{a}_{k}}_{k}\perp\mkern-9.5mu\perp\bar{A}_{k}|\bar{Y}^{\bar{a}_{k-1}}_{k-1},\bar{L}^{\bar{a}_{k-1}}_{k-1},U)}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(Y_{k-1}=y_{k-1}|\bar{L}_{k-1}=\bar{l}_{k-1},\bar{A}_{k-1}=\bar{x}_{k-1},\bar{Y}_{k-2}=\bar{y}_{k-2},U=u)\\\
&\mathbb{P}(L^{\bar{a}_{k-1}=\bar{x}_{k-1}}_{k-1}=l_{k-1}|\bar{A}_{k-1}=\bar{x}_{k-1},\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2},\bar{L}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{l}_{k-2},U=u)\\\
&\mathbb{P}(\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2},\bar{L}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{l}_{k-2}|U=u)\\\
&\overset{\eqref{eq:consistency2}}{=}\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\mathbb{P}(Y_{k-1}=y_{k-1}|\bar{L}_{k-1}=\bar{l}_{k-1},\bar{A}_{k-1}=\bar{x}_{k-1},\bar{Y}_{k-2}=\bar{y}_{k-2},U=u)\\\
&\mathbb{P}(L_{k-1}=l_{k-1}|\bar{A}_{k-1}=\bar{x}_{k-1},\bar{Y}_{k-2}=\bar{y}_{k-2},\bar{L}_{k-2}=\bar{l}_{k-2},U=u)\\\
&\mathbb{P}(\bar{Y}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{y}_{k-2},\bar{L}^{\bar{a}_{k-2}=\bar{x}_{k-2}}_{k-2}=\bar{l}_{k-2}|U=u)\end{split}$

Continuing iteratively the process, we have that

$\begin{split}&\mathbb{E}(Y^{\bar{a}_{k}=\bar{x}_{k}}_{k}|U=u)=\sum_{\bar{y}_{k-1},\bar{l}_{k}}\mathbb{E}(Y_{k}|\bar{L}_{k}=\bar{l}_{k},\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},U=u)\\\
&\mathbb{P}(L_{k}=l_{k}|\bar{A}_{k}=\bar{x}_{k},\bar{Y}_{k-1}=\bar{y}_{k-1},\bar{L}_{k-1}=\bar{l}_{k-1},U=u)\\\
&\prod^{k-1}_{m=1}\mathbb{P}(Y_{m}=y_{m}|\bar{L}_{m}=\bar{l}_{m},\bar{A}_{m}=\bar{x}_{m},\bar{Y}_{m-1}=\bar{y}_{m-1},U=u)\\\
&\mathbb{P}(L_{m}=l_{m}|\bar{A}_{m}=\bar{x}_{m},\bar{Y}_{m-1}=\bar{y}_{m-1},\bar{L}_{m-1}=\bar{l}_{m-1},U=u).\end{split}$

This result is similar to the g-formula reported in Robins et al. 1999.
Differences are due to the different indexing of time-varying variables.

The formula can be further simplified for the specific causal model we
consider in Section 6.2. From (14) it follows that $L_{k}$ is independent of
any other variable that is non-descendent of $L_{k}$ when conditioning on
$A_{k},Y_{k-1},L_{k-1}$ and $U$. Similarly, it follows from (15) that $Y_{k}$
is independent from any non-descendent variable once conditioned on
$L_{k},A_{k},Y_{k-1},L_{k-1},A_{k-1}$ and $U$. Moreover, due to the
assumptions of stationarity (17) and (LABEL:eq:stationarity2), we know that
the conditional probabilities
$\mathbb{P}(L_{k}=l_{k}|A_{k}=a_{k},Y_{k-1}=y_{k-1},L_{k-1}=l_{k-1},U=u)$ and
$\mathbb{P}(Y_{k}=y_{k}|L_{k}=l_{k},A_{k}=a_{k},Y_{k-1}=y_{k-1},L_{k-1}=l_{k-1},A_{k-1}=a_{k-1},U=u)$
do not change over time. We can define

$g^{L}(l_{k},a_{k},y_{k-1},l_{k-1},u)=\mathbb{P}(L_{k}=l_{k}|A_{k}=a_{k},Y_{k-1}=y_{k-1},L_{k-1}=l_{k-1},U=u)$

and

$\begin{split}&g^{Y}(y_{k},l_{k},a_{k},y_{k-1},l_{k-1},a_{k-1},u)=\\\
&\mathbb{P}(Y_{k}=y_{k}|L_{k}=l_{k},A_{k}=a_{k},Y_{k-1}=y_{k-1},L_{k-1}=l_{k-1},A_{k-1}=a_{k-1},U=u).\end{split}$

The g-formula can therefore be rewritten as

$\begin{split}&\sum_{\bar{y}_{k},\bar{l}_{k}}y_{k}\cdot
g^{Y}(y_{k},l_{k},x_{k},y_{k-1},l_{k-1},x_{k-1},u)\,g^{L}(l_{k},x_{k},y_{k-1},l_{k-1},u)\\\
&\prod^{k-1}_{m=1}g^{Y}(y_{m},l_{m},x_{m},y_{m-1},l_{m-1},x_{m-1},u)\,g^{L}(l_{m},x_{m},y_{m-1},l_{m-1},u)\overset{\eqref{eq:theta}}{=}\theta_{k}(\bar{x}_{k},u)\end{split}$

## Appendix E

The null hypothesis (26) implies

$\begin{split}&\mathbb{P}(Y^{\bar{a}_{k}=\bar{x}_{k}}_{k}|U=u_{i})=\mathbb{P}(Y^{\bar{a}_{k}=\bar{x}^{\prime}_{k}}_{k}|U=u_{i})\hskip
28.45274pt\forall k,\bar{x}_{k},\bar{x}^{\prime}_{k}\\\
&\overset{Y^{\bar{a}_{k}}_{k}\perp\mkern-9.5mu\perp\textbf{Z}|U}{\implies}\mathbb{P}(Y^{\bar{a}_{k}=\bar{x}_{k}}_{k}|U=u_{i},\textbf{Z}=\textbf{z}_{i})=\mathbb{P}(Y^{\bar{a}_{k}=\bar{x}^{\prime}_{k}}_{k}|U=u_{i},\textbf{Z}=\textbf{z}_{i})\hskip
28.45274pt\forall k,\bar{x}_{k},\bar{x}^{\prime}_{k}.\end{split}$

Consider the quantity

$\begin{split}&\frac{1}{\sum^{t}_{k=1}\\{\mathbb{I}(A_{i,k}=a)\\}}\sum^{t}_{k=1}\\{\mathbb{I}(A_{i,k}=a)Y_{i,k}\\}\\\
&\overset{\eqref{eq:A1}}{=}\frac{1}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{Z}_{i},k)=a)\\}}\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{Z}_{i},k)=a)Y_{i,k}\\}\\\
&\overset{\eqref{eq:Y4}}{=}\frac{1}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{Z}_{i},k)=a)\\}}\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{Z}_{i},k)=a)f^{Y}(\\{\bar{X}_{i,k-1},a\\},u_{i},\varepsilon_{i,k})\\}\end{split},$

where $\bar{X}_{i,k-1}$ is the treatment sequence administered until time
point $k-1$.

Therefore, under the null hypothesis,

$\frac{1}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=a)\\}}\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=a)f^{Y}(\\{\bar{x}_{i,k-1},a\\},u_{i},\varepsilon_{i,k})\\}$

is an average of $\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=a)\\}$
i.i.d. random variables with the same mean $\mu^{*}$ and finite variance
$\sigma^{2}_{\varepsilon}$. Notably, $\mu^{*}$ does not depend on the sequence
of administered treatment $\bar{x}$ or on the treatment level considered $a$.
This is the case because, under the null hypothesis, the treatment sequence is
irrelevant to the conditional distribution of the outcome given $U$ and Z.
Following the same reasoning as in Appendix B, the Central Limit Theorem
ensures that for large $t$, under the null hypothesis, we have that

$\hat{\tau_{i}}\,|\,U=u_{i},\textbf{Z}=\textbf{z}_{i}\sim\mathbb{N}\left(0,\frac{\sigma^{2}_{\varepsilon}}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=1)\\}}+\frac{\sigma^{2}_{\varepsilon}}{\sum^{t}_{k=1}\\{\mathbb{I}(f^{A}(\textbf{z}_{i},k)=0)\\}}\right).$

## Appendix F

Considering the causal model described in Section 8.2, the estimator
$\hat{\tau_{i}}$ can be re-written as

$\begin{split}&\hat{\tau_{i}}\overset{\eqref{eq:Z3}}{=}\frac{2}{t}\sum^{t}_{k=1}\\{\mathbb{I}(A_{i,k}=1)Y_{i,k}\\}-\frac{2}{t}\sum^{t}_{k=1}\\{\mathbb{I}(A_{i,k}=0)Y_{i,k}\\}\\\
&=\frac{2}{t}\sum^{t/s}_{p=1}\sum^{s}_{w=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}-\frac{2}{t}\sum^{t/s}_{p=1}\sum^{s}_{w=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\\\
&=\frac{2}{t}\sum^{s}_{w=1}(t/2s)\frac{1}{t/2s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}\\\
&-\frac{2}{t}\sum^{s}_{w=1}(t/2s)\frac{1}{t/2s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\\\
&=\frac{1}{s}\sum^{s}_{w=1}\frac{1}{t/2s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}\\\
&-\frac{1}{s}\sum^{s}_{w=1}\frac{1}{t/2s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\\\
&=\frac{1}{s}\sum^{s}_{w=1}\left\\{\frac{1}{t/2s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}-\frac{1}{t/2s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\right\\}\\\
&=\frac{1}{s}\sum^{s}_{w=1}\\{\hat{\tau}_{i,w}\\},\end{split}$

where we subdivided the study period of length $t$, into $t/s$ intervals of
length $s$. In each interval $p$, we have $w=\\{1,...,s\\}$ time points. We
show that $\hat{\tau_{i}}$ is simply the average of the $s$ values of
$\hat{\tau}_{i,w}$. Each $\hat{\tau}_{i,w}$ being the difference between the
observed outcomes in the treated time points and the observed outcomes in the
untreated time points for individual $i$, only considering the time points in
position $w$ of all intervals (a total of $t/2s$ time points treated and
$t/2s$ untreated).

According to Equation (29), the covariate for the time point $w$ ($L_{w}$) in
each interval $p$ is constant in time. This means that considering only time
points in position $w$, the data generation process is identical to the one
described in Section 3. Indeed, the conditional stationarity (31) holds. We
can therefore apply the results from Appendix B. Using the fact that

$\mathbb{E}(\hat{\tau}_{i,w}|U=u_{i},\textbf{Z}=\textbf{z}_{i})\overset{(B)}{=}U\text{-}CATE_{w}(u_{i}),$

we can show that $\hat{\tau}_{i}$ is an unbiased estimator for the average of
the individual-specific effects across one interval conditioned on
$U=u_{i},\textbf{Z}=\textbf{z}_{i}$

$\mathbb{E}(\hat{\tau}_{i}|U=u_{i},\textbf{Z}=\textbf{z}_{i})=\sum^{s}_{w=1}U\text{-}CATE_{w}(u_{i}).$

Similarly, from Appendix B we know that for fixed
$\textbf{Z}=\textbf{z}_{i},U=u_{i}$, $\hat{\tau}_{i,w}$ converges in
distribution when $t\to\infty$

$\hat{\tau}_{i,w}\,|\,U=u_{i},\textbf{Z}=\textbf{z}_{i}\sim\mathbb{N}\left(U\text{-}CATE_{w}(u_{i}),\frac{2s\cdot\sigma^{2}_{\varepsilon}}{t}+\frac{2s\cdot\sigma^{2}_{\varepsilon}}{t}\right).$

It follows that, under the same conditions, for large $t$,

$\hat{\tau}_{i}\,|\,U=u_{i},\textbf{Z}=\textbf{z}_{i}\sim\mathbb{N}\left(\frac{1}{s}\sum^{s}_{w=1}U\text{-}CATE_{w}(u_{i}),\frac{4\sigma^{2}_{\varepsilon}}{t}\right).$

## Appendix G

As in Appendix F, we can subdivide the study period of length $t$, into $t/s$
intervals of length $s$. In each interval $p$, we have $w=\\{1,...,s\\}$ time
points.

$\begin{split}&\hat{\tau_{i}}\overset{\eqref{eq:Z4}}{=}\frac{4}{t}\sum^{t}_{k=1}\\{\mathbb{I}(A_{i,k}=1)Y_{i,k}\\}-\frac{4}{t}\sum^{t}_{k=1}\\{\mathbb{I}(A_{i,k}=0)Y_{i,k}\\}\\\
&=\frac{4}{t}\sum^{t/s}_{p=1}\sum^{s}_{w=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}-\frac{4}{t}\sum^{t/s}_{p=1}\sum^{s}_{w=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\\\
&=\frac{4}{t}\sum^{s}_{w=1}(t/4s)\frac{1}{t/4s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}\\\
&-\frac{4}{t}\sum^{s}_{w=1}(t/4s)\frac{1}{t/4s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\\\
&=\frac{1}{s}\sum^{s}_{w=1}\frac{1}{t/4s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}\\\
&-\frac{1}{s}\sum^{s}_{w=1}\frac{1}{t/4s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\\\
&=\frac{1}{s}\sum^{s}_{w=1}\left\\{\frac{1}{t/4s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=1)Y_{i,s(p-1)+w}\\}-\frac{1}{t/4s}\sum^{t/s}_{p=1}\\{\mathbb{I}(A_{i,s(p-1)+w}=0)Y_{i,s(p-1)+w}\\}\right\\}\\\
&=\frac{1}{s}\sum^{s}_{w=1}\\{\hat{\tau}_{i,w}\\}\end{split}$

As in Appendix F, we showed that $\hat{\tau_{i}}$ is simply the average of the
$s$ values of $\hat{\tau}_{i,w}$. Each $\hat{\tau}_{i,w}$ being the difference
between the observed outcomes in the treated time points and the observed
outcomes in the untreated time points for individual $i$, only considering the
time points in position $w$ of all intervals (a total of $t/4s$ time points
treated and $t/4s$ untreated).

Let us consider only time points with treatment $a\in\\{0,1\\}$, and
reparametrize Equation (34) in terms of $w$ and $p$, rather than $k$

$\begin{split}&\mathbb{I}(A_{k}=a)\cdot Y_{k}\\\
&=\mathbb{I}(A_{s(p-1)+w}=a)\cdot Y_{s(p-1)+w}\\\
&=\mathbb{I}(A_{s(p-1)+w}=a)\cdot
f^{Y}(\bar{A}_{(s(p-1)+w-s):(s(p-1)+w)},U,\varepsilon_{s(p-1)+w})\\\
&\overset{\eqref{eq:Z4}}{=}\mathbb{I}(A_{s(p-1)+1}=a)\cdot
f^{Y}(\\{\overline{-1}_{s-w+1},\overline{a}_{w}\\},U,\varepsilon_{s(p-1)+w}).\end{split}$

This new formulation makes explicit that, for time points in which a treatment
is received ($A_{k}=0$ or $A_{k}=1$), if we keep $w$ fixed, the outcome at
time $k$ is fully determined by the treatment administered in that time point
(different based on the interval $p$), by the value of $U$, and by one
instantiation of the noise term (different over time). Essentially, this is
the same outcome generation process described in Equation (3). This is the
case, because given the treatment schedule (33), for every time point with
treatment $a$ which is in position $w$ of the intervals, there is only one
possible sequence of treatments assigned to the previous time points up to
$k-s$. Indeed, if we consider only time points in which a treatment was
administered and with the same value of $w$, we have a data generation process
as the one described in Section 3. We can, therefore, apply the results from
Appendix B.

Eploiting the fact that

$\mathbb{E}(\hat{\tau}_{i,w}|U=u_{i},\textbf{Z}=\textbf{z}_{i})\overset{(B)}{=}U\text{-}CATE_{w}(u_{i}),$

we can show that $\hat{\tau}_{i}$ is an unbiased estimator for the average of
the individual-specific effects across one interval conditioned on
$U=u_{i},\textbf{Z}=\textbf{z}_{i}$

$\mathbb{E}(\hat{\tau}_{i}|U=u_{i},\textbf{Z}=\textbf{z}_{i})=\sum^{s}_{w=1}U\text{-}CATE_{w}(u_{i}).$

Similarly, from Appendix B we know that for fixed
$\textbf{Z}=\textbf{z}_{i},U=u_{i}$, $\hat{\tau}_{i,w}$ converges in
distribution when $t\to\infty$

$\hat{\tau}_{i,w}\,|\,U=u_{i},\textbf{Z}=\textbf{z}_{i}\sim\mathbb{N}\left(U\text{-}CATE_{w}(u_{i}),\frac{4s\cdot\sigma^{2}_{\varepsilon}}{t}+\frac{4s\cdot\sigma^{2}_{\varepsilon}}{t}\right).$

It follows that, under the same conditions, for very large $t$

$\hat{\tau}_{i}\,|\,U=u_{i},\textbf{Z}=\textbf{z}_{i}\sim\mathbb{N}\left(\frac{1}{s}\sum^{s}_{w=1}U\text{-}CATE_{w}(u_{i}),\frac{8\sigma^{2}_{\varepsilon}}{t}\right).$

## Appendix H

Consider the average causal effect at time $k$, we have

$\begin{split}ACE_{k}&=\mathbb{E}(Y^{\bar{a}_{k}=\bar{1}_{k}}_{k})-\mathbb{E}(Y^{\bar{a}_{k}=\bar{0}_{k}}_{k})\overset{\eqref{eq:carryov1}}{=}\mathbb{E}(Y^{a_{k}=1}_{k}-Y^{a_{k}=0}_{k})\\\
&=\mathbb{E}\left(\mathbb{E}(Y^{a_{k}=1}_{k}-Y^{a_{k}=0}_{k}|U)\right)=\int_{U}\mathbb{E}(Y^{a_{k}=1}_{k}-Y^{a_{k}=0}_{k}|U=u)dF_{U}(u)\\\
&\overset{\eqref{eq:stationarity1_expectation}}{=}\int_{U}\mathbb{E}(Y^{a_{k_{*}}=1}_{k_{*}}-Y^{a_{k_{*}}=0}_{k_{*}}|U=u)dF_{U}(u)=\mathbb{E}(Y^{a_{k_{*}}=1}_{k_{*}}-Y^{a_{k_{*}}=0}_{k_{*}})\overset{\eqref{eq:carryov1}}{=}ACE_{k_{*}}\end{split}$

with $k_{*}$ being a generic time point. Therefore, the $ACE_{k}$ is constant
over time.

### Unbiasedness

Consider the marginal expected value of
$\frac{1}{n}\sum^{n}_{i=1}\\{\hat{\tau_{i}}\\}$

$\begin{split}\mathbb{E}\left(\frac{1}{n}\sum^{n}_{i=1}\\{\hat{\tau_{i}}\\}\right)&=\mathbb{E}\left(\mathbb{E}\left(\frac{1}{n}\sum^{n}_{i=1}\\{\hat{\tau_{i}}\\}\middle|U,\textbf{Z}\right)\right)=\frac{1}{n}\sum^{n}_{i=1}\\{\mathbb{E}\left(\mathbb{E}\left(\hat{\tau_{i}}\middle|U,\textbf{Z}\right)\right)\\}\\\
&\overset{(B)}{=}\frac{1}{n}\sum^{n}_{i=1}\\{\mathbb{E}\left(\tau(U,\textbf{Z})\right)\\}\overset{(A)}{=}\frac{1}{n}\sum^{n}_{i=1}\\{\mathbb{E}\left(U\text{-}CATE_{k}(U)\right)\\})\\\
&\overset{\eqref{eq:ucate1}}{=}\frac{1}{n}\sum^{n}_{i=1}\\{\mathbb{E}\left(\mathbb{E}(Y^{\bar{a}_{k}=\bar{1}_{k}}_{k}|U)-\mathbb{E}(Y^{\bar{a}_{k}=\bar{0}_{k}}_{k}|U)\right)\\}\\\
&=\frac{1}{n}\sum^{n}_{i=1}\\{\mathbb{E}(Y^{\bar{a}_{k}=\bar{1}_{k}}_{k})-\mathbb{E}(Y^{\bar{a}_{k}=\bar{0}_{k}}_{k})\\}\overset{\eqref{eq:ACE}}{=}\frac{1}{n}\sum^{n}_{i=1}\\{ACE_{k}\\}=ACE_{k}\end{split}$

### Consistency

We will use that

$\begin{split}\frac{1}{n}\sum^{n}_{i=1}\\{\hat{\tau_{i}}\\}=\sum_{u\in\mathbb{U},\textbf{z}\in\mathbb{Z}}\left\\{\frac{1}{n}\sum^{n}_{i=1}\left\\{\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\hat{\tau_{i}}\right\\}\right\\}.\end{split}$

We know that the expected value of the quantity in the sum is

$\begin{split}&\mathbb{E}(\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\hat{\tau_{i}})\\\
&=\mathbb{E}\left(\mathbb{E}(\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\hat{\tau_{i}}|\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z}))\right)=\\\
&\mathbb{P}(\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})=1)\mathbb{E}(\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\hat{\tau_{i}}|\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})=1)\\\
&+\mathbb{P}(\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})=0)\mathbb{E}(\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\hat{\tau_{i}}|\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})=0)\\\
&=\mathbb{P}(\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})=1)\mathbb{E}(\hat{\tau_{i}}|U_{i}=u,\textbf{Z}_{i}=\textbf{z})\\\
&=\mathbb{P}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\mathbb{E}(\hat{\tau_{i}}|U_{i}=u,\textbf{Z}_{i}=\textbf{z})\\\
&\overset{(B)}{=}\mathbb{P}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\tau(u,\textbf{z})\overset{(A)}{=}\mathbb{P}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})U\text{-}CATE_{k}(u).\end{split}$

Therefore, for the Weak Law of Large Numbers

$\frac{1}{n}\sum^{n}_{i=1}\left\\{\mathbb{I}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})\hat{\tau_{i}}\right\\}\xrightarrow{P}\mathbb{P}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})U\text{-}CATE_{k}(u)$

for $n$ that goes to infinity.

It follows that

$\begin{split}&\frac{1}{n}\sum^{n}_{i=1}\\{\hat{\tau_{i}}\\}\xrightarrow{P}\sum_{u\in\mathbb{U},\textbf{z}\in\mathbb{Z}}\left\\{\mathbb{P}(U_{i}=u,\textbf{Z}_{i}=\textbf{z})U\text{-}CATE_{k}(u)\right\\}=\mathbb{E}(U\text{-}CATE_{k}(U_{i}))\\\
&\overset{\eqref{eq:ucate1}}{=}\mathbb{E}\left(\mathbb{E}(Y^{\bar{a}_{k}=\bar{1}_{k}}_{k}|U_{i})-\mathbb{E}(Y^{\bar{a}_{k}=\bar{0}_{k}}_{k}|U_{i})\right)=\mathbb{E}(Y^{\bar{a}_{k}=\bar{1}_{k}}_{k})-\mathbb{E}(Y^{\bar{a}_{k}=\bar{0}_{k}}_{k})\overset{\eqref{eq:ACE}}{=}ACE_{k}.\end{split}$

Therefore the mean of $\hat{\tau_{i}}$ is a consistent estimator for the
$ACE_{k}$ for $n$ that goes to infinity.

### Appendix I

Consider

$\frac{1}{n}\sum^{n}_{i=1}\hat{\theta}_{k}(\bar{1}_{k},u_{i})-\hat{\theta}_{k}(\bar{0}_{k},u_{i})=\sum_{\textbf{z}\in\mathbb{Z},u\in\mathbb{U}}\left\\{\frac{1}{n}\sum^{n}_{i=1}(\hat{\theta}_{k}(\bar{1}_{k},u_{i})-\hat{\theta}_{k}(\bar{0}_{k},u_{i}))\mathbb{I}(u=u_{i},\textbf{z}=\textbf{z}_{i})\right\\}$

We defined
$\hat{\theta}_{k}(\bar{1}_{k},u_{i})-\hat{\theta}_{k}(\bar{0}_{k},u_{i})$ such
that converges towards $U\text{-}CATE_{k}(u_{i})$, for fixed $U=u_{i}$ and
$\textbf{Z}=\textbf{z}_{i}$, when $t$ goes to infinity. Since
$Y^{\bar{a}_{k}}_{k}\perp\mkern-9.5mu\perp\textbf{Z}|U$, we have that
$U\text{-}CATE_{k}(U)\perp\mkern-9.5mu\perp\textbf{Z}|U$. Therefore, for $t$
that grows to infinity, we have

$\begin{split}\sum_{\textbf{z}\in\mathbb{Z},u\in\mathbb{U}}\left\\{\frac{1}{n}\sum^{n}_{i=1}(\hat{\theta}_{k}(\bar{1}_{k},u_{i})-\hat{\theta}_{k}(\bar{0}_{k},u_{i}))\mathbb{I}(u=u_{i},\textbf{z}=\textbf{z}_{i})\right\\}\xrightarrow{P}\\\
\sum_{\textbf{z}\in\mathbb{Z},u\in\mathbb{U}}\left\\{\frac{1}{n}\sum^{n}_{i=1}(U\text{-}CATE_{k}(u_{i})|\textbf{Z}=\textbf{z}_{i})\mathbb{I}(u=u_{i},\textbf{z}=\textbf{z}_{i})\right\\}.\end{split}$

If additionally $n$ grows to infinity, we have that

$\begin{split}&\sum_{\textbf{z}\in\mathbb{Z},u\in\mathbb{U}}\left\\{\frac{1}{n}\sum^{n}_{i=1}(U\text{-}CATE_{k}(U)|U=u_{i},\textbf{Z}=\textbf{z}_{i})\mathbb{I}(u=u_{i},\textbf{z}=\textbf{z}_{i})\right\\}\xrightarrow{P}\\\
&\sum_{\textbf{z}\in\mathbb{Z},u\in\mathbb{U}}(U\text{-}CATE_{k}(U)|U=u,\textbf{Z}=\textbf{z})\mathbb{P}(U=u,\textbf{Z}=\textbf{z})\\\
&=\sum_{\textbf{z}\in\mathbb{Z},u\in\mathbb{U}}\mathbb{E}(Y^{\bar{a}_{k}=1}_{k}-Y^{\bar{a}_{k}=0}_{k}|U=u,\textbf{Z}=\textbf{z})\mathbb{P}(U=u,\textbf{Z}=\textbf{z})\\\
&=\mathbb{E}(Y^{\bar{a}_{k}=\bar{1}_{k}}_{k}-Y^{\bar{a}_{k}=\bar{0}_{k}}_{k})=ACE_{k}.\end{split}$


# Robust Disaster Assessment from Aerial Imagery Using
Text-to-Image Synthetic Data

Tarun Kalluri1 Jihyeon Lee2 Kihyuk Sohn2 Sahil Singla2
Manmohan Chandraker1 Joseph Xu2 Jeremiah Liu2
1UC San Diego 2Google Research
Work done during TK’s internship at Google.

###### Abstract

We present a simple and efficient method to leverage emerging text-to-image
generative models in creating large-scale synthetic supervision for the task
of damage assessment from aerial images. While significant recent advances
have resulted in improved techniques for damage assessment using aerial or
satellite imagery, they still suffer from poor robustness to domains where
manual labeled data is unavailable, directly impacting post-disaster
humanitarian assistance in such under-resourced geographies. Our contribution
towards improving domain robustness in this scenario is two-fold. Firstly, we
leverage the text-guided mask-based image editing capabilities of generative
models and build an efficient and easily scalable pipeline to generate
thousands of post-disaster images from low-resource domains. Secondly, we
propose a simple two-stage training approach to train robust models while
using manual supervision from different source domains along with the
generated synthetic target domain data. We validate the strength of our
proposed framework under cross-geography domain transfer setting from xBD and
SKAI images in both single-source and multi-source settings, achieving
significant improvements over a source-only baseline in each case.

##  1 Introduction

In this work, we address the issue of poor robustness caused by traditional
training methods for the task of disaster assessment by generating synthetic
data using guided text-to-image generation [32, 8]. To accelerate rescue,
recovery and aid routing through scalable and automated disaster assessment
from images, recent methods propose efficient training paradigms using paired
labeled data from before and after the disaster [16, 5, 28, 52, 61, 50]. While
being instrumental in significantly improving the accuracy in damage
assessment, these methods greatly rely on manual supervision for efficient
performance and perform poorly when deployed in novel domains - such as new
disaster types or unseen geographies. While unsupervised adaptation methods
exist to overcome the overhead of manual annotation [4, 6, 24], they still
require unlabeled images captured from both before and after the disaster for
learning domain agnostic features. While readily available satellite imagery
provides generalized aerial coverage for most geographic locations for pre-
disaster images, the retrieval of post-disaster imagery remains a time-
consuming process, hindering rapid damage assessment during critical response
windows, with non-trivial domain shifts preventing cross-geographical
deployment.

Figure 1: Summary of our proposed pipeline. A disaster assessment model
trained using labeled data from a different domain suffers from poor accuracy
due to significant distribution shifts with the target. We offer a novel way
of addressing this limitation, by leveraging the recent advances in mask-based
text-to-image models [8] to generate thousands of synthetic labeled data from
the target domain where only pre-disaster images are accessible. We
incorporate this synthetic data along with source labeled data in a two-stage
training framework to achieve significant gains on challening transfer
settings from xBD [16] and SKAI [23] datasets.

On the other hand, there has been a notable progress in the field of
generating synthetic data using guided text-to-image models [38, 44, 63, 12],
which overcome the cumbersome manual annotation process and enable
controllable data-generation at scale to train robust and data-efficient
models. While a majority of these works focus on tasks like image recognition
[38, 44, 43], object detection [14] and semantic segmentation [57, 29],
extending these methods to suit the current setting of aerial disaster
assessment is non-trivial, as it requires generating precisely synchronized
pre- and post-disaster imagery of the affected area.

We present a novel framework that leverages image-guided text-conditioned
generative models to synthesize large datasets of post-disaster imagery
conditioned on pre-disaster imagery, which is trained efficiently using a two-
stage approach by incorporating unlabeled target domain data alongside source
labels. By exploiting the mask-based image-editing abilities of transformer-
based text-to-image models, we edit the pre-disaster image using a suitable
text-prompt to create a corresponding synthetic post-disaster image,
generating a new, synthetically labeled dataset specific to the target domain.
To mitigate performance degradation caused by domain shift between generated
data and real-world images, we adopt a two-stage training procedure. We first
train a siamese vision-transformer [9] using labeled data from the source
domains, and subsequently fine-tune the last layer on the synthetic data from
the target domain following prior work [21]. We show the effectiveness of our
framework in training robust models through experiments on several challenging
transfer settings from xBD and SKAI datasets, significantly outperforming a
source-only training baseline in each case. As shown in Fig. 1, while training
directly using the source domain only achieves only 25% accuracy on the target
test-data, our synthetic-data augmented training achieves 54.4%, with a non-
trivial improvement of 29% on the challenging xBD dataset. In summary, our
contributions are as follows.

  1. 1.

We offer a cost-effective way to generate training data for disaster
assessment in areas lacking real-world aerial imagery, leveraging the image-
editing abilities of large-scale text-to-image models.

  2. 2.

Following prior work in robustness studies [21], we devise a simple and
effective two-stage training strategy to use the synthetically generated data
in training along with labeled data from different source domains to achieve
complementary benefits.

  3. 3.

We validate the effectiveness of our proposed framework on two benchmark
datasets xBD [16] and SKAI [23] images, obtaining significant improvements
over a standard source-only baseline in both single-source (+9.8%, +25.2%) and
multi-source (+5.33%, +29.13%) domain transfer settings.

##  2 Related Work

#### Disaster Assessment using Satellite Images

The task of image-based disaster assessment involves predicting the presence
or extent of damage in a particular location by comparing pre and post
disaster aerial or satellite imagery. Fueled by the availability of paired
pre- and post-disaster images captured from remote-sensing satellites [16, 28,
23, 53, 31, 41], several methods have been proposed to identify the damage [2,
49, 9, 26], as well as to precisely localize the damage within the image [15,
30, 51]. However, these approaches rely on labeled data for efficient
performance, preventing their use in novel domains without incurring
additional collection and annotation overheads [55, 6]. While domain
adaptation methods exist to bridge this gap [4, 24, 46], they still need to
access the post-disaster imagery which is difficult to acquire in a short
window following a disaster. While prior works attempt generation of images
using GANs [48], they lack the ability to generate controllable synthetic data
at scale. Our work addresses these limitations by leveraging the advances in
conditional text-to-image capabilities to generate large-scale synthetic
supervision from low-resource target domains. We also note that while there
has been significant advances in unsupervised domain adaptation for image
classification [19, 56, 37, 64] and segmentation [22, 45], they are typically
not applicable to expert tasks like disaster assessment through aerial
imagery, preventing their direct use or comparison for our problem.

#### Creating Synthetic Data from Generative Models

Recent progress in the field of generative modeling has enabled the creation
of diverse and realistic images conditioned on a variety of inputs such as
text [34, 32, 36, 8], images [35, 27], layouts [62] or semantic maps [60, 47].
In particular, text-to-image synthesis enables creation of diverse visual
content based on natural language prompts [20, 8, 32, 36, 58]. Recent works
explored the use of leveraging the power of these models in generating
synthetic supervision for various tasks including object recognition [38, 44,
63, 43, 1], object detection [14, 54, 25], semantic segmentation [57, 29],
outlier detection [12] and long-tailed robustness [59, 39, 40]. Building upon
this line of work, our work tackles image generation of post-disaster imagery
in low-resource domains through localized editing of the corresponding pre-
disaster images guided by suitable text prompts, showing an efficient way to
improve domain robustness.

##  3 Method

###  3.1 Problem Setting

Figure 2: Overview of the proposed synthetic data generation pipeline. We
first pass the pre-disaster image $U$ from the target domain through a pre-
trained VQGAN encoder followed by a tokenizer to compute the latent token,
which is then masked using a binary mask. We use the MUSE model along with a
suitable text prompt $T$ to predict the output tokens from the masked tokens
$u$, which are then de-tokenized to generate the post-disaster image
$\hat{V}_{t}$. Our augmented dataset $\hat{D}_{t}$ now contains the input
image $U_{t}$, generated image $\hat{V}_{t}$ and the binary label
corresponding to the text prompt (indicating damage or no-damage).

We now describe our problem setting of investigating domain robustness in
image-based disaster assessment tasks. We denote our labeled source dataset as
$\mathcal{D}_{s}=\\{U_{s}^{i},V_{s}^{i},y^{i}\\}_{i=1}^{N_{s}}$, where $U^{i}$
is the before image (image captured before the disaster, also called pre-
image), $V^{i}$ is the after image (image captured after the disaster, also
called the post-image) along with a binary label $y^{i}\in\\{0,1\\}$
indicating whether or not there is damage between the images due to a
disaster. $N_{s}$ denotes the number of source images. In general, the pre and
post images from the source images are paired, where the image collection is
synchronized to capture the same location before and after the disasters,
ensuring pixel-level correspondence between the images. Furthermore, the
target domain is denoted by $\mathcal{D}_{t}=\\{U_{t}^{i}\\}_{i=1}^{N_{t}}$,
where $N_{t}$ denotes the number of (unlabeled and unpaired) target images.
Unique to our work, we assume a zero-shot target setting, where we only have
access to the pre-images from a new domain (which could be a new geographical
location or new disaster type), and neither the post images nor the damage
labels are available. This setting is more realistic as capturing paired
images immediately after a disaster and labeling them for damage incurs
expensive time and manual annotation overhead in most cases, while pre-
disaster images are naturally available through satellite footage. Therefore,
our main goal is to utilize labeled source images along with unlabeled pre-
disaster target images to improve the performance on the target domain at
test-time.

While unsupervised domain adaptation (UDA) [4, 24] has been a standard
approach to address such domain shifts for disaster assessment, UDA methods
still require access to paired pre and post-disaster data during training from
both source and target domains. In contrast, we only require images captured
before the disaster, and automatically generate synthetic post-disaster images
to facilitate robust training.

###  3.2 Background: Text-to-Image Generation

In our work, we adopt the MUSE [8] model for text-to-image generation, and we
provide a brief overview of that framework here. MUSE is a non-autoregressive
model for text-conditioned synthesis capable of generating high-resolution
images with fast inference speeds. In contrast to diffusion models requiring
sequential decoding over several time-steps [34, 10, 32, 36], MUSE adopts a
purely transformer-based generation approach outlined in MaskGIT [7] with
improved inference speed enabled by parallel decoding. In particular, MUSE
uses a pre-trained T5-XXL text encoder [33] to first encode the text-prompt
into a 4096-dimensional language embedding, while the input images are passed
through a semantic tokenizer such as VQGAN [13] consisting of an encoder and a
quantization layer to map the image into a sequence of discrete tokens from a
learned codebook, along with a decoder which maps the predicted or generated
tokens back into the images. To enable high resolution image synthesis, MUSE
adopts the use of two VQGAN models with different downsampling ratio and
spatial resolution of the tokens, along with two transformer modules called
base and super-resolution modules that generate low resolution and high
resolution latent tokens respectively. During training, these tokens are
trained using cross-entropy loss with a reference image, while during
inference, the high resolution latent tokens are passed through the VQGAN
decoder to generate images conditioned on the input text prompts. We refer the
reader to [8] for further details about the training and inference of MUSE
generative models.

The use of MUSE in our pipeline as opposed to diffusion-based or auto-
regressive based generative models is motivated by two favorable properties.
Firstly, MUSE allows efficient mask-based editing capabilities based on its
tokenized encoding and decoding-based architecture. Furthermore, MUSE is
capable of high-resolution image synthesis with fast inference time and
flexible latent token sizes, facilitating the generation of thousands of high
quality synthetic post-disaster images conditioned on pre images at scale.

###  3.3 Generating Synthetic Data

We provide an overview of our proposed generation pipeline in Fig. 2. We first
take the pre-disaster image from the target domain
$U_{t}\in\mathbb{R}^{H\times W\times 3}$, where $H,W$ are the height and width
of the image, and generate a corresponding binary mask
$\mathcal{M}\in\mathbb{R}^{H\times W}$ with a center patch of size
$H^{\prime},W^{\prime}$ as ones and rest as zeros, with $H^{\prime}<H$ and
$W^{\prime}<W$. Since the aerial images in the datasets we considered are
typically centered around the subject, we align the center of the binary mask
with the center of the image, helping us in directly editing the relevant
subject in the image. In practice, we randomly perturb the mask around the
center by a factor $(\delta_{x},\delta_{y})$ in both dimensions, where the
perturbation factor $\delta_{x}\sim\mathbb{U}[-W/16,W/16]$ and
$\delta_{y}\sim\mathbb{U}[-H/16,H/16]$ is sampled from the uniform
distribution. So our patch with all ones in the binary mask is centered around
$(H/2+\delta_{y},W/2+\delta_{x})$ with respect to the input image $U_{t}$ and
mask $\mathcal{M}$.

We then pass the target image $U_{t}$ through the VQGAN encoder to compute the
latent tokens for the image, and downsample the binary mask $\mathcal{M}$ to
match the spatial resolution of the latent tokens (which is $1/16^{\text{th}}$
and $1/8^{\text{th}}$ of the original image size for the low-resolution and
high-resolution pipelines respectively). Subsequently, we multiply the
downsampled binary mask $\mathcal{M}^{\prime}$ with the latent tokens at each
pixel location to get a masked token embedding $u_{t}$. The masked-latent
token embeddings along with the text-embedding of the input prompt are then
passed through a series of cross-attention layers pre-trained in MUSE to
predict the output tokens from the predicted latent tokens. The outputs are
then passed through the decoder layer of the VQGAN resulting in the output
image $\hat{V}_{t}$.

We generate four output images for each pre-image and prompt pair, and pick
the best one using the ranking obtained by CLIP similarity score [17] between
the input prompt and the generated image. We repeat this process for every
image in the target dataset, creating a synthetic dataset
$\hat{\mathcal{D}}_{t}=\\{U_{t}^{i},\hat{V}_{t}^{i},\hat{y}^{i}\\}_{i=1}^{N_{t}}$
of pre and post disaster images from the target domain.

#### Prompt Pool for Generation

A major advantage of generating synthetic images is avoiding the need for
manual annotation for unlabeled domains, as the labels can be directly derived
from the corresponding prompts. In our setting with binary labels indicating
damaged or not damaged buildings or locations, we choose the prompts to
reflect these criterion. For example, to create synthetic images for scenes
damaged by hurricane disaster, we use prompts such as An aerial view of a
house damaged due to a hurricane or A satellite image of a building that was
destroyed by a hurricane and assign the label 1. Alternatively, for generating
images which have no damage, we create a prompt pool indicating images which
are undamaged (for example, A satellite image of a building) and assign the
generated images with a label of 0. We list the pool of prompts adopted in our
work in Fig. 3 and Fig. 4.

SKAI Dataset Damaged Set _An aerial view of a house damaged due to a
hurricane._ _A bird’s-eye view of a building destroyed by a hurricane._ _A
top-down view of a house damaged by a hurricane._ _A satellite image of a
building destroyed by a hurricane._ _A bird’s-eye view of a building damaged
by a hurricane._ Undamaged Set _A satellite image of a house covered by
trees._ _A bird’s-eye view of a house surrounded by trees._ _A top-down view
of a house under tree shade._ _An aerial view of an intact house under tree
shade._

Figure 3: Prompt Pool for SKAI

xBD Dataset Moore Tornado _An aerial view of a house damaged due to a
tornado._ _A bird’s-eye view of a building destroyed by a tornado._ _A top-
down view of a house damaged by a tornado._ _A satellite image of a building
destroyed by a tornado._ _A bird’s-eye view of a building damaged by a
tornado._ Nepal Floods _An aerial view of houses surrounded by a flood._ _A
top-down view of houses damaged by floods._ _A top-down view of a house
damaged by floods inundated in water._ _A satellite image of a building
destroyed by a flood surrounded by water._ _A satellite image of houses that
was destroyed by a flood surrounded by water and trees._ Portugal Wildfire _An
aerial view of forest land after it is torched by a wildfire._ _An aerial view
of buildings after a wildfire._ _An aerial image of forest land scorched by a
wildfire._ _A bird’s-eye view of a forest region with completely scorched
trees._

Figure 4: Prompt Pool for xBD

###  3.4 Training using Synthetic Data

Following prior work in disaster assessment task [3], we adopt a siamese
network with shared transformer backbone [11] for training. Specifically, we
pass both pre and post images $U$ and $V$ using parameter-shared transformer
backbones $\mathcal{E}_{u}$ and $\mathcal{E}_{v}$, resulting in feature
embedding $f_{u}=\mathcal{E}_{u}(U)$ and $f_{v}=\mathcal{E}_{v}(V)$
respectively, each dimension $d$. We then fuse these embeddings by
concatenating them to form $f\in\mathbb{R}^{2d}$, where
$f=\text{concat}(f_{s},f_{d})$. Finally, we add a 2-layer MLP network
$\mathcal{H}$ with a hidden dimension of $d$ to predict a single output value
indicating the probability of damaged building between the pre and post
images. The whole network is then trained with a binary cross entropy loss
using the binary ground truth labels.

However, directly training predictive models using only synthetic data might
result in poor accuracy due to the domain gap between synthetic and real
images [38]. Therefore, we devise a two-stage training strategy to leverage
the in-domain synthetic data, along with out-of-domain real data to
effectively improve the target performance. In particular, we first train our
network end-to-end including the encoders $\mathcal{E}_{u}$ and
$\mathcal{E}_{v}$ as well as the MLP layers $\mathcal{H}$ using the source
domain data $\mathcal{D}_{s}$. Subsequently, we follow prior work in robust
learning [21] to fine-tune only the final layers of the MLP network
$\mathcal{H}(.)$ using the synthetic data supervision from the target, while
keeping the encoders fixed during the fine-tuning stage. We observed that only
re-training the last layer prevents over-fitting the network to the synthetic
data compared to complete end-to-end fine-tuning (Tab. 4), so we adopt this
two-stage mechanism in our framework. During inference, we apply a sigmoid
layer on top of the predicted output and threshold this probability to predict
damaged buildings in the post-image if the predicted probability is $>0.5$,
and predict no damage otherwise.

#### Fine-tuning the MUSE model

In the framework illustrated so far, we only use a frozen pre-trained MUSE
model, where we fix the generative model itself and only use it for inference
given input images and corresponding text prompts. However, such off-the-shelf
models trained on billions of web-scale image-text data might contain images
from a wide variety of domains, and might not be fully suited for use in
specific domains like aerial or satellite imagery. Therefore, we also
investigate the potential benefits offered by fine-tuning the pre-trained
generative model for the specific task of aerial image classification. In
particular, we collect the pre-disaster images from $\mathcal{D}_{t}$ and
create prompts for each image from the undamaged pool to create a dataset of
image-text pairs from the target domain. We then adopt adapter-fine tuning
[18] to fine-tune the pre-trained model using these image-text pairs, which we
found to be more resource-efficient than end-to-end fine-tuning. This fine-
tuned model is expected to capture more domain specific properties unique to
aerial and satellite imagery, and we compare this procedure with generation
using the frozen model in Sec. 4.3.

##  4 Experiments

Method |  | Ian $\rightarrow$ |  | Michael$\rightarrow$ |  | Laura$\rightarrow$ |  | Maria$\rightarrow$ |  | Avg.
---|---|---|---|---|---|---|---|---|---|---
|  | Michael | Laura | Maria |  | Ian | Laura | Maria |  | Ian | Michael | Maria |  | Ian | Michael | Laura |  |
Source Only |  | 41.6 | 19.3 | 27.3 |  | 38.0 | 32.0 | 29.7 |  | 38.3 | 46.9 | 26.3 |  | 30.0 | 39.6 | 21.9 |  | 32.6
Ours w/ ZeroShot MUSE |  | 49.2 | 36.8 | 31.9 |  | 47.4 | 42.5 | 32.0 |  | 50.0 | 54.7 | 30.6 |  | 42.6 | 54.5 | 36.6 |  |  42.4 (+9.8%)
Ours w/ fine-tuned MUSE |  | 49.6 | 29.9 | 25.8 |  | 50.9 | 31.5 | 28.8 |  | 49.2 | 55.6 | 26.4 |  | 44.1 | 53.6 | 27.6 |  | 39.4 (+6.8%)

Table 1: Single-source Domain Adaptation Results on SKAI dataset AUPRC values
for different transfer settings from the SKAI dataset. We compare the results
obtained by training using only real data from the source domain and combining
it with synthetic generated data from the target domains on all the transfer
settings. Evidently, our approach outperforms the source-only baseline setting
new state-of-the-art.

Method |  | Moore-Tornado$\rightarrow$ |  | Nepal-Flooding$\rightarrow$ |  | Portugal-Wildfire$\rightarrow$ |  | Avg.
---|---|---|---|---|---|---|---|---
|  | Nepal-Flooding | Portugal-Wildfire |  | Moore-Tornado | Portugal-Wildfire |  | Moore-Tornado | Nepal-Flooding |  |
Source Only |  | 23.8 | 23.2 |  | 14.5 | 18.5 |  | 45.3 | 24.7 |  | 25.0
Ours w/ ZeroShot MUSE |  | 49.5 | 24.1 |  | 75.1 | 25.1 |  | 76.0 | 51.6 |  | 50.2 (+25.2%)
Ours w/ fine-tuned MUSE |  | 43.9 | 24.1 |  | 82.3 | 25.3 |  | 83.1 | 47.5 |  |  51.1 (+26.1%)

Table 2: Single-source Domain Adaptation Results on xBD dataset. AUPRC values
for different transfer settings from the xBD dataset [16]. On each of the
transfer setting, augmenting training using synthetic data from the target
domain significantly outperforms the source-only baseline, with an improvement
of 25.2% using a zeroshot generative model, and 26.1% with further fine-tuning
the generative backbone on aerial image-text pairs.

We next demonstrate the effectiveness of the proposed approach on several
challenging transfer settings. We first introduce our choice of datasets in
Sec. 4.1, specify the training details in Sec. 4.2 followed by the results in
Sec. 4.3 and several ablations into our modeling and training choices in Sec.
4.4.

Dataset |  | SKAI-Dataset |  | xBD-Dataset
---|---|---|---|---
Method |  |  $\rightarrow$Ian |  $\rightarrow$Michael |  $\rightarrow$Maria |  $\rightarrow$Laura | Avg. |  |  $\rightarrow$Moore-Tornado |  $\rightarrow$Nepal-Flooding |  $\rightarrow$Portugal-Wildfire | Avg.
Source Only |  | 44.21 | 48.62 | 29.54 | 36.83 | 39.78 |  | 18.96 | 27.16 | 29.69 | 25.27
Ours w/ ZeroShot MUSE |  | 54.79 | 52.24 | 34.05 | 39.38 |  45.11 (+5.33%) |  | 78.70 | 52.40 | 32.10 |  54.40(+29.13%)
Ours w/ fine-tuned MUSE |  | 49.12 | 53.83 | 30.92 | 39.29 | 43.29 (+3.51%) |  | 83.18 | 50.16 | 30.72 |  54.69(+29.42%)

Table 3: Multi-source Domain Adaptation Results on SKAI and xBD datasets.
AUPRC values for different transfer settings, where we show the result of
training using synthetic generated data from the respective target domain
along with manual supervision from all the three remaining domains. Our
approach outperforms the source-only baseline highlighting the effectiveness
of training with generated synthetic data in bridging domain gaps.

###  4.1 Datasets

#### xBD Dataset

xBD [16] is a large-scale dataset designed for automatic disaster assessment
using aerial and satellite imagery. The dataset covers synchronized pre- and
post- event satellite imagery of both damaged and undamaged scenes from more
than 19 events across the world, covering a variety of disaster types across
varying severity levels. Since our focus in this paper is to improve
robustness of aerial disaster assessment algorithms across disparate
geographies, we choose 3 domains from xBD, namely nepal-flooding, portugal-
wildfires and moore-tornado to demostrate our results, which have 36456, 18884
and 18491 images respectively. These domains encompass data from three
distinct geographical subregions, each affected by entirely different types of
disasters making it a challenging problem to improve cross-domain robustness.

#### SKAI Satellite Imagery

In order to verify the effectiveness of our method in improving the
performance across subtle domain variations, we adopt the SKAI dataset [23]
consisting of pre and post hurricane images captured from different regions in
the United States. The images in SKAI includes data collected from Ian, Maria,
Michael and Laura hurricanes with 2733, 3709, 3991 and 3991 images
respectively, which we use as the different domains for our cross-domain
robustness setting. Note that both these datasets consist of heavy class
imbalance, with more than 80% of the image-pairs capturing non-damaged
buildings, adding an additional layer of complexity in bridging the domain
shifts. For both the datasets, we show results using single-source and multi-
source adaptation settings, in which we use supervised data from single source
domain or all the domains except the target respectively.

###  4.2 Training and Evaluation Details

We use an Imagenet pretrained ViT-B/16 transformer backbone [42] as the
encoder in our setting, and remove the last classification layer replacing it
with the MLP head for binary classification. We then train the network using
the two-stage approach discussed in Sec. 3.4, first using the supervised
source domain images using Adam optimizer with a learning rate of 2e-6 for the
pre-trained backbone and 2e-5 for the randomly initialized MLP layer, followed
by re-training only the last MLP layer using synthesized target domain images
using the same hyperparameters as above. We use a batch size of 64 in both
cases and perform training for 5000 iterations. We use the validation images
from the target domain to perform early stopping, which we observed to be very
crucial to obtain good performance in our setting.

Following prior work in disaster assessment tasks from satellite imagery [16,
23], we adopt the AUPRC metric for evaluation which measures the area under
the precision-recall curve across various thresholds, and is shown to be
relatively more robust for cases like ours where there is severe class
imbalance against positive examples. In terms of baselines, we compare with a
source-only baseline which only trains a predictive model on the source domain
and evaluates on the target test-set. Since this does not use any target data,
it serves as a fundamental baseline to illustrate the benefits obtained by our
method. Note that prior UDA methods require both pre and post disaster images
to learn domain agnostic features [4, 24, 46], preventing a direct comparison
for our setting where only pre-disaster images from the target dataset are
available.

###  4.3 Results

#### Single-source Zeroshot Adaptation

We show the results for single-source UDA for domains from the SKAI dataset in
Tab. 1 and xBD dataset in Tab. 2. As shown, our method of augmenting out of
distribution training using synthetic images generated from MUSE model
achieves better accuracy than the source-only baseline, with $\sim 10\%$ and
$\sim 25\%$ improvements on the SKAI and the xBD datasets on average. Our
improvements are consistent across all the transfer settings, with up to $\sim
70\%$ improvement on the more challenging cross-disaster cross-geography
setting from xBD dataset, highlighting the effectiveness of leveraging
generative foundational models to create synthetic data for low-resource
domains even in expert tasks like disaster assessment.

Furthermore, we also compare the AUPRC results observed through fine-tuning
the generative model on aerial imagery and satellite images, using the
procedure outlined in Sec. 3.4. We observe that the model trained with data
generated from fine-tuned model outperforms both the source-only baseline as
well as the zeroshot settings on 4 out of 6 settings in xBD dataset with $\sim
1\%$ improvement on the average accuracy, indicating the potential in fine-
tuning MUSE model on domain-specific images. On SKAI data however, we observe
zeroshot model is better on the average AUPRC. A potential reason for this
could be that the generative ability of the text-to-image generative model is
reduced after fine-tuning on domain-specific images, impacting accuracy in few
of the transfer settings, highlighting room for further improvement through
more carefully designed fine-tuning strategies.

#### Multi-source Zeroshot Adaptation

The comparison for both SKAI and xBD datasets on multi-source adaptation
setting is shown in Tab. 3. Firstly, the accuracy achieved by multi-source
models on all target domains is higher than single-source setting, which is
expected since multi-source models have access to relatively more supervised
data. Furthermore, the results from Tab. 3 clearly show the effectiveness of
our approach even for such multi-source evaluation setting, where our method
using zeroshot text-to-image generation yields $5.33\%$ improvement over
baseline on SKAI dataset and $29.13\%$ improvement over baseline on the xBD
dataset. Our benefits are consistent for both the datasets across all the
transfer tasks, further supporting our hypothesis that text-to-image models
can serve as strong data generators for low-resource domains. As seen for the
case of single-source setting, we observe the gains yielded by data generation
using zeroshot text-to-image models to be competitive when compared to fine-
tuned models on both the datasets.

###  4.4 Ablations

Method |  | SKAI |  | xBD
---|---|---|---|---
(R0) Source Only |  | 39.78 |  | 25.27
(R1) Only Synthetic Data |  | 40.60 |  | 47.76
(R2) Joint Training on Real + Synthetic |  | 43.11 |  | 49.66
(R3) End-to-end Finetuning |  | 44.10 |  | 53.44
(R4) Last-Layer Finetune on SynData |  | 45.11 |  | 54.40

Table 4: Effect of training choices We show the effect of various training
choices in our framework, where last-layer re-training using only synthetic
data (R4) outperforms training using only synthetic data without the source
labels (R1), jointly training on both real and synthetic data (R2) as well as
end-to-end finetuning using synthetic data (R3).

#### Ablations and Insights

We show the effect of various design choices in our framework in Tab. 4.
Firstly, we observe that training only using synthetic data without source
domain data leads to poor results, potentially highlighting the limitations of
synthetic data alone in training (R1 vs R4). This facet of synthetic data has
also been noted in prior works [38], indicating that manual supervision is
still necessary to observe gains with synthetic supervision. Furthermore, we
also show that joint-training using real source and synthetic target datasets
is inferior to our proposed approach of first pre-training on real source data
followed by last-layer retraining on generated target domain data (R2 vs R4),
supporting our two-stage training framework. Finally, we also observe that
end-to-end fine-tuning using synthetic data under-performs the approach of
finetuning the last layer only (R3 vs R4).

(a) Ian Hurricane

(b) Moore Tornado

Figure 5: Effect of the amount of generated synthetic data. We show the
positive influence of the volume of generated synthetic data (as a % of the
target domain images) on the multi-source transfer setting, where adding more
target data invariably helps to improve the final target accuracy for both
SKAI 5 and xBD 5 datasets, with potential for further enhancement with more
generated data.

#### Effect of Volume of Synthetic Data

We show the effect of the amount of generated synthetic data on the target
AUPRC in Fig. 5. We observe that adding more synthetic data invariably helps
the final target accuracy for both the datasets studied. More importantly, we
observe no saturation even when using all target data to generate images
indicating further room for improvement of target performance through low-cost
synthetically generated data.

(a) Images from Ian domain.

(b) Images from the Michael domain.

(c) Images from Moore-Tornado.

(d) Images from Nepal-Flooding.

(e) Images from Portugal-Wildfire.

Figure 6: Visualization of text-to-image results. We show several examples
from our generated images, along with the pre-disaster image, correponding
conditioning mask (overlapped on the pre-image) as well as the text-prompt
used to generate the post-image from 6 Ian-Hurricane, 6 Michael-Hurricane, 6
Moore-Tornado, 6 Nepal-Floods and 6 Portugal-Wildfire.

#### Visualizing Generated Images

We show several illustrations of samples generated through our method in Fig.
6 on both SKAI (Fig. 6, Fig. 6) and xBD (Fig. 6, Fig. 6, Fig. 6) datasets,
where we include the pre-disaster image as well as the mask and the text-
prompt used for conditional image editing through our generative model. In
most cases, we observe that the text-to-image model incorporates the textual
guidance and performs localized editing on the input image to generate a
synthetic post-disaster image with great effectiveness. The model shows
excellent capability in seamlessly handling the various types of disasters
through our text-guidance, which helps to create realistic images in low-
resource domains leading to significant empirical gains (Tab. 3).

##  5 Conclusion

In this paper, we explore the potential of leveraging emerging text-to-image
models in generating synthetic supervision to improve robustness across low-
resource domains for disaster assessment tasks. We design an efficient and
scalable data-generation pipeline by leveraging the localized image editing
capabilities of transformer-based generative models [8]. Using this framework,
we generate several thousand synthetic post-disaster images conditioned on
pre-disaster images and text guidance, followed by a simple two-stage training
mechanism that yields non-trivial benefits over a source-only baseline in both
single source and multi-source domain adaptation setting. In terms of
limitations, we noted a significant sensitivity of the training process to the
quality and coherence of the generated synthetic data, which is directly
affected by the presence of low-quality generated images. A potential future
work can therefore be to additionally incorporate better filtering strategies
into our framework to remove poor quality images and improve training.
Nevertheless, our work serves as one of the first to explore the potential of
text-to-image synthetic data for expert tasks like satellite disaster
assessment, which holds massive potential for continued improvement with the
development of better image generation models.

## References

  * Azizi et al. [2023] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.  Synthetic data from diffusion models improves imagenet classification.  _arXiv preprint arXiv:2304.08466_ , 2023.
  * Bai et al. [2020] Yanbing Bai, Junjie Hu, Jinhua Su, Xing Liu, Haoyu Liu, Xianwen He, Shengwang Meng, Erick Mas, and Shunichi Koshimura.  Pyramid pooling module-based semi-siamese network: A benchmark model for assessing building damage from xbd satellite imagery datasets.  _Remote Sensing_ , 12(24):4055, 2020.
  * Bandara and Patel [2022] W. G. C. Bandara and Vishal M. Patel.  A transformer-based siamese network for change detection.  _IEEE International Geoscience and Remote Sensing Symposium_ , 2022.
  * Benson and Ecker [2020] Vitus Benson and Alexander Ecker.  Assessing out-of-domain generalization for robust building damage detection, 2020.
  * Bin et al. [2022] Junchi Bin, Ran Zhang, Rui Wang, Yue Cao, Yufeng Zheng, Erik P. Blasch, and Zheng Liu.  An efficient and uncertainty-aware decision support system for disaster response using aerial imagery.  _Sensors (Basel, Switzerland)_ , 22, 2022.
  * Bouchard et al. [2022] Isabelle Bouchard, Marie-Ève Rancourt, Daniel Aloise, and Freddie Kalaitzis.  On transfer learning for building damage assessment from satellite imagery in emergency contexts.  _Remote Sensing_ , 14(11), 2022.
  * Chang et al. [2022] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman.  Maskgit: Masked generative image transformer.  _CVPR_ , 2022.
  * Chang et al. [2023] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan.  Muse: Text-to-image generation via masked generative transformers, 2023.
  * Da et al. [2022] Yifan Da, Zhiyuan Ji, and Yongsheng Zhou.  Building damage assessment based on siamese hierarchical transformer framework.  _Mathematics_ , 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * Du et al. [2024] Xuefeng Du, Yiyou Sun, Jerry Zhu, and Yixuan Li.  Dream the impossible: Outlier imagination with diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pages 12873–12883. Computer Vision Foundation / IEEE, 2021.
  * Ge et al. [2023] Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Neel Joshi, Laurent Itti, and Vibhav Vineet.  Beyond generation: Harnessing text to image models for object detection and segmentation.  _arXiv preprint arXiv:2309.05956_ , 2023.
  * Gupta and Shah [2021] Rohit Gupta and Mubarak Shah.  Rescuenet: Joint building segmentation and damage assessment from satellite imagery.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 4405–4411. IEEE, 2021.
  * Gupta et al. [2019] Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston.  xbd: A dataset for assessing building damage from satellite imagery, 2019.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  _arXiv preprint arXiv:2104.08718_ , 2021.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In _International conference on machine learning_ , pages 2790–2799. PMLR, 2019.
  * Kalluri et al. [2022] Tarun Kalluri, Astuti Sharma, and Manmohan Chandraker.  Memsac: Memory augmented sample consistency for large scale domain adaptation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX_ , pages 550–568. Springer, 2022.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017, 2023.
  * Kirichenko et al. [2022] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  _arXiv preprint arXiv:2204.02937_ , 2022.
  * Lai et al. [2022] Xin Lai, Zhuotao Tian, Xiaogang Xu, Ying-Cong Chen, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia.  Decouplenet: Decoupled network for domain adaptive semantic segmentation.  In _European Conference on Computer Vision_ , 2022.
  * Lee et al. [2020] Jihyeon Lee, Joseph Z. Xu, Kihyuk Sohn, Wenhan Lu, David Berthelot, Izzeddin Gur, Pranav Khaitan, Ke-Wei, Huang, Kyriacos Koupparis, and Bernhard Kowatsch.  Assessing post-disaster damage from satellite imagery using semi-supervised learning techniques, 2020.
  * Li et al. [2020] Yundong Li, Chen Lin, Hongguang Li, Wei Hu, Han Dong, and Yi Liu.  Unsupervised domain adaptation with self-attention for post-disaster building damage detection.  _Neurocomputing_ , 415:27–39, 2020.
  * Lin et al. [2023] Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao.  Explore the power of synthetic data on few-shot object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 638–647, 2023.
  * Lu et al. [2024] Wen Lu, Lu Wei, and Minh Nguyen.  Bitemporal attention transformer for building change detection and building damage assessment.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 17:4917–4935, 2024.
  * Lu et al. [2021] Xiaopeng Lu, Lynnette Ng, Jared Fernandez, and Hao Zhu.  Cigli: Conditional image generation from language & image.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3134–3138, 2021.
  * Nasrallah et al. [2021] Hasan Nasrallah, Mustafa Shukor, and Ali J. Ghandour.  Sci-net: scale-invariant model for buildings segmentation from aerial imagery.  _Signal, Image and Video Processing_ , 17:2999 – 3007, 2021.
  * Nguyen et al. [2024] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen.  Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Potnis et al. [2019] Abhishek V Potnis, Rajat C Shinde, Surya S Durbha, and Kuldeep R Kurte.  Multi-class segmentation of urban floods from multispectral imagery using deep learning.  In _IGARSS 2019-2019 IEEE international geoscience and remote sensing symposium_ , pages 9741–9744. IEEE, 2019.
  * Rahnemoonfar et al. [2021] Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Roberson Murphy.  Floodnet: A high resolution aerial imagery dataset for post flood scene understanding.  _IEEE Access_ , 9:89644–89654, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International conference on machine learning_ , pages 8821–8831. Pmlr, 2021.
  * Roberts et al. [2019] Adam Roberts, Colin Raffel, Katherine Lee, Michael Matena, Noam Shazeer, Peter J Liu, Sharan Narang, Wei Li, and Yanqi Zhou.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Google, Tech. Rep._ , 2019.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in neural information processing systems_ , 35:36479–36494, 2022.
  * Saito et al. [2018] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Sarıyıldız et al. [2023] Mert Bülent Sarıyıldız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis.  Fake it till you make it: Learning transferable representations from synthetic imagenet clones.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8011–8021, 2023.
  * Shao et al. [2024] Jie Shao, Ke Zhu, Hanxiao Zhang, and Jianxin Wu.  Diffult: How to make diffusion model useful for long-tail recognition.  _arXiv preprint arXiv:2403.05170_ , 2024.
  * Shin et al. [2023] Joonghyuk Shin, Minguk Kang, and Jaesik Park.  Fill-up: Balancing long-tailed data with generative models.  _arXiv preprint arXiv:2306.07200_ , 2023.
  * Sirko et al. [2021] Wojciech Sirko, Sergii Kashubin, Marvin Ritter, Abigail Annkah, Yasser Salah Eddine Bouchareb, Yann Dauphin, Daniel Keysers, Maxim Neumann, Moustapha Cisse, and John Quinn.  Continental-scale building detection from high resolution satellite imagery.  _arXiv preprint arXiv:2107.12283_ , 2021.
  * Steiner et al. [2021] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  _arXiv preprint arXiv:2106.10270_ , 2021.
  * Tian et al. [2023] Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola.  Learning vision from models rivals learning vision from data.  _arXiv preprint arXiv: 2312.17742_ , 2023.
  * Tian et al. [2024] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan.  Stablerep: Synthetic images from text-to-image models make strong visual representation learners.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Tsai et al. [2018] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker.  Learning to adapt structured output space for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7472–7481, 2018.
  * Valentijn et al. [2020] Tinka Valentijn, Jacopo Margutti, Marc van den Homberg, and Jorma Laaksonen.  Multi-hazard and spatial transferability of a cnn for automated building damage assessment.  _Remote Sensing_ , 12(17), 2020.
  * Wang et al. [2022] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li.  Semantic image synthesis via diffusion models.  _arXiv preprint arXiv: 2207.00050_ , 2022.
  * Wang et al. [2021] Xiang Wang, Yundong Li, Chen Lin, Yi Liu, and Shuo Geng.  Building damage detection based on multi-source adversarial domain adaptation.  _Journal of Applied Remote Sensing_ , 15(3):036503, 2021.
  * Weber and Kané [2020] Ethan Weber and Hassan Kané.  Building disaster damage assessment in satellite imagery with multi-temporal fusion.  _arXiv preprint arXiv:2004.05525_ , 2020.
  * Weber et al. [2022] Ethan Weber, Dim P. Papadopoulos, Àgata Lapedriza, Ferda Ofli, Muhammad Imran, and Antonio Torralba.  Incidents1m: A large-scale dataset of images with natural disasters, damage, and incidents.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45:4768–4781, 2022.
  * Wu et al. [2021a] Chuyi Wu, Feng Zhang, Junshi Xia, Yichen Xu, Guoqing Li, Jibo Xie, Zhenhong Du, and Renyi Liu.  Building damage detection using u-net with attention mechanism from pre-and post-disaster remote sensing datasets.  _Remote Sensing_ , 13(5):905, 2021a.
  * Wu et al. [2021b] Chuyi Wu, Feng Zhang, Junshi Xia, Yichen Xu, Guoqing Li, Jibo Xie, Zhenhong Du, and Ren yi Liu.  Building damage detection using u-net with attention mechanism from pre- and post-disaster remote sensing datasets.  _Remote. Sens._ , 13:905, 2021b.
  * Xia et al. [2022] Junshi Xia, Naoto Yokoya, Bruno Adriano, and Clifford Broni-Bediako.  Openearthmap: A benchmark dataset for global high-resolution land cover mapping.  _2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 6243–6253, 2022.
  * Xie et al. [2023] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou.  Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7452–7461, 2023.
  * Xu et al. [2019] Joseph Z. Xu, Wenhan Lu, Zebo Li, Pranav Khaitan, and Valeriya Zaytseva.  Building damage detection in satellite imagery using convolutional neural networks.  _ArXiv_ , abs/1910.06444, 2019.
  * Xu et al. [2021] Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, and Rong Jin.  Cdtrans: Cross-domain transformer for unsupervised domain adaptation.  _arXiv preprint arXiv:2109.06165_ , 2021.
  * Yang et al. [2024] Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao.  Freemask: Synthetic images with dense annotations make stronger segmentation models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv: 2206.10789_ , 2022.
  * Yu et al. [2023] Zhuoran Yu, Chenchen Zhu, Sean Culatana, Raghuraman Krishnamoorthi, Fanyi Xiao, and Yong Jae Lee.  Diversify, don’t fine-tune: Scaling up visual recognition training with synthetic images.  _arXiv preprint arXiv:2312.02253_ , 2023.
  * Zhang et al. [2023] Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang.  Controllable text-to-image generation with gpt-4.  _arXiv preprint arXiv: 2305.18583_ , 2023.
  * Zhao and Zhang [2020] Fei Zhao and Chengcui Zhang.  Building damage evaluation from satellite imagery using deep learning.  _2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI)_ , pages 82–89, 2020.
  * Zheng et al. [2023] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li.  Layoutdiffusion: Controllable diffusion model for layout-to-image generation.  _Computer Vision and Pattern Recognition_ , 2023.
  * Zhou et al. [2023] Yongchao Zhou, Hshmat Sahak, and Jimmy Ba.  Training on thin air: Improve image classification with generated data.  _arXiv preprint arXiv: 2305.15316_ , 2023.
  * Zhu et al. [2023] Jinjing Zhu, Haotian Bai, and Lin Wang.  Patch-mix transformer for unsupervised domain adaptation: A game perspective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3561–3571, 2023.